Linux MM tree latest commits
 help / color / mirror / Atom feed
From: Andrew Morton <akpm@linux-foundation.org>
To: mm-commits@vger.kernel.org,ziy@nvidia.com,zhongyuan@hygon.cn,
	ying.huang@linux.alibaba.com,xhao@linux.alibaba.com,wujianyong@hygon.cn,
	wangfengyu@hygon.cn,matthew.brost@intel.com,liuyibin@hygon.cn,
	joshua.hahnjy@gmail.com,huangsj@hygon.cn,gourry@gourry.net,geshifei@hygon.cn,
	david@kernel.org,dave.hansen@linux.intel.com,byungchul@sk.com,
	apopple@nvidia.com,qiuzhiteng@hygon.cn,akpm@linux-foundation.org
Subject: + mm-migrate-dont-batch-the-tlb-flush-when-there-is-nothing-to-batch.patch added to mm-unstable branch
Date: Sun, 22 Nov 2026 09:16:54 -0800	[thread overview]
Message-ID: <20261122171655.654BA1F00A3C@smtp.kernel.org> (raw)
In-Reply-To: <20261112010357.2739389-1-qiuzhiteng@hygon.cn>


The patch titled
     Subject: mm/migrate: don't batch the TLB flush when there is nothing to batch
has been added to the -mm mm-unstable branch.  Its filename is
     mm-migrate-dont-batch-the-tlb-flush-when-there-is-nothing-to-batch.patch

This patch will shortly appear at
     https://git.kernel.org/pub/scm/linux/kernel/git/akpm/25-new.git/tree/patches/mm-migrate-dont-batch-the-tlb-flush-when-there-is-nothing-to-batch.patch

This patch will later appear in the mm-unstable branch at
    git://git.kernel.org/pub/scm/linux/kernel/git/akpm/mm

Before you just go and hit "reply", please:
   a) Consider who else should be cc'ed
   b) Prefer to cc a suitable mailing list as well
   c) Ideally: find the original patch on the mailing list and do a
      reply-to-all to that, adding suitable additional cc's

*** Remember to use Documentation/process/submit-checklist.rst when testing your code ***

The -mm tree is included into linux-next via various
branches at git://git.kernel.org/pub/scm/linux/kernel/git/akpm/mm
and is updated there most days

------------------------------------------------------
From: Zhiteng Qiu <qiuzhiteng@hygon.cn>
Subject: mm/migrate: don't batch the TLB flush when there is nothing to batch
Date: Thu, 12 Nov 2026 09:03:57 +0800

NUMA balancing migrates one folio at a time, but since commit
7e12beb8ca2a ("migrate_pages: batch flushing TLB") it still unmaps
with TTU_BATCH_FLUSH like any other MIGRATE_ASYNC migration.  When the
batch contains one small folio mapped only once, there is nothing to
batch, and the batched unmap can leave mm->tlb_flush_batched pending,
so the NUMA scanner's next change_pte_range() can call flush_tlb_mm()
before changing the PTE table.  With migrations running concurrently,
that state keeps being refilled, and on this workload the scanner
often flushes the whole mm as it walks the PTE tables.

This slows scanning and keeps mm->tlb_flush_pending raised longer.  On
x86, a migration that unmaps a PROT_NONE PTE while
mm->tlb_flush_pending is set gets true from pte_accessible() and needs
a flush too; with batching, the resulting arch_tlbbatch_flush() is a
full flush.  The two feed each other.  In one run, 33% of the
migrations overlapped a scan pass, although passes covered only 9% of
the time.

On a 2-socket, 384-CPU x86 machine without INVLPGB, with v7.3, a
120 GiB anonymous working set on 4 KiB pages and 384 threads (three
balancing on/off pairs per kernel):

                                     v7.3          patched
  throughput loss from balancing     13.0-16.5%    0.5-6.3%
  TLB interrupts per migrated page   202-210       12.7-13.6
  average scanner mmu_gather time    ~90 ms        ~6.7 ms

The mmu_gather times were measured in separate instrumented runs.

So don't ask for a batched flush when the batch holds a single small
folio that is not mapped shared; try_to_migrate_one() then uses
ptep_clear_flush() for each PTE it encounters.  Large folios retain
their existing flush behavior.  migrate_pages_batch() computes the
flags, since it knows how many folios it is migrating.  PMD-mapped
THPs are not affected, since set_pmd_migration_entry() flushes them
regardless of TTU_BATCH_FLUSH.

Link: https://patch.msgid.link/20261112010357.2739389-1-qiuzhiteng@hygon.cn
Signed-off-by: Zhiteng Qiu <qiuzhiteng@hygon.cn>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Cc: David Hildenbrand <david@kernel.org>
Cc: Ying Huang <ying.huang@linux.alibaba.com>
Cc: Zi Yan <ziy@nvidia.com>
Cc: Matthew Brost <matthew.brost@intel.com>
Cc: Joshua Hahn <joshua.hahnjy@gmail.com>
Cc: Byungchul Park <byungchul@sk.com>
Cc: Gregory Price <gourry@gourry.net>
Cc: Alistair Popple <apopple@nvidia.com>
Cc: Xin Hao <xhao@linux.alibaba.com>
Cc: Dave Hansen <dave.hansen@linux.intel.com>
Cc: Yuan Zhong <zhongyuan@hygon.cn>
Cc: Shifei Ge <geshifei@hygon.cn>
Cc: Jianyong Wu <wujianyong@hygon.cn>
Cc: Shijie Huang <huangsj@hygon.cn>
Cc: Yibin Liu <liuyibin@hygon.cn>
Cc: Fengyu Wang <wangfengyu@hygon.cn>
---
tl;dr: A patch-specific A/B test confirmed that this patch avoids a
deferred TLB flush for a single, privately mapped 4 KiB folio.

On A, asynchronous NUMA migration selected TTU_BATCH_FLUSH, deferred
the PTE clear, and left a pending flush that a scanner of the same mm
performed. With this patch alone, migration selected ttu=0, cleared
the PTE directly, and the scanner found no corresponding pending
flush. Migration completed with the page contents intact.

Additional tests with the complete patch set reached single-folio
migration through DAMON physical-address monitoring and direct
compaction; both changed from deferred to direct PTE clearing.
Multi-folio, intact THP, and shared-folio migrations retained batching,
as intended, and their checked contents remained intact.

A forced THP split exposed a limit of this patch alone: the final
single child still selected batching because the decision used the
original list size. The accompanying (fix) patch corrected that case.

One 30-second run per kernel recorded two scanner flushes on A and
none with the complete patch set. The runs are too small to establish
a throughput improvement.

 mm/migrate.c |   29 +++++++++++++++++++++++++----
 1 file changed, 25 insertions(+), 4 deletions(-)

--- a/mm/migrate.c~mm-migrate-dont-batch-the-tlb-flush-when-there-is-nothing-to-batch
+++ a/mm/migrate.c
@@ -1229,11 +1229,28 @@ static void migrate_folio_done(struct fo
 		folio_put(src);
 }
 
+/*
+ * Batching the TLB flush only helps when the rmap walk can clear more than
+ * one PTE.  A single small folio mapped once has nothing to batch, and the
+ * batched unmap would leave mm->tlb_flush_batched pending.
+ */
+static enum ttu_flags migrate_ttu_flags(struct folio *folio,
+					enum migrate_mode mode,
+					bool single_folio)
+{
+	if (mode != MIGRATE_ASYNC)
+		return 0;
+	if (single_folio && !folio_test_large(folio) &&
+	    !folio_maybe_mapped_shared(folio))
+		return 0;
+	return TTU_BATCH_FLUSH;
+}
+
 /* Obtain the lock on page, remove all ptes. */
 static int migrate_folio_unmap(new_folio_t get_new_folio,
 		free_folio_t put_new_folio, unsigned long private,
 		struct folio *src, struct folio **dstp, enum migrate_mode mode,
-		struct list_head *ret)
+		enum ttu_flags ttu, struct list_head *ret)
 {
 	struct folio *dst;
 	int rc = -EAGAIN;
@@ -1355,7 +1372,7 @@ static int migrate_folio_unmap(new_folio
 		/* Establish migration ptes */
 		VM_BUG_ON_FOLIO(folio_test_anon(src) &&
 			       !folio_test_ksm(src) && !anon_vma, src);
-		try_to_migrate(src, mode == MIGRATE_ASYNC ? TTU_BATCH_FLUSH : 0);
+		try_to_migrate(src, ttu);
 		old_folio_state |= FOLIO_WAS_MAPPED;
 	}
 
@@ -1846,9 +1863,11 @@ static int migrate_pages_batch(struct li
 	LIST_HEAD(unmap_folios);
 	LIST_HEAD(dst_folios);
 	bool nosplit = (reason == MR_NUMA_MISPLACED);
+	bool single_folio = list_is_singular(from);
+	enum ttu_flags ttu;
 
 	VM_WARN_ON_ONCE(mode != MIGRATE_ASYNC &&
-			!list_empty(from) && !list_is_singular(from));
+			!list_empty(from) && !single_folio);
 
 	for (pass = 0; pass < nr_pass && retry; pass++) {
 		retry = 0;
@@ -1935,8 +1954,10 @@ static int migrate_pages_batch(struct li
 				continue;
 			}
 
+			ttu = migrate_ttu_flags(folio, mode, single_folio);
 			rc = migrate_folio_unmap(get_new_folio, put_new_folio,
-					private, folio, &dst, mode, ret_folios);
+					private, folio, &dst, mode, ttu,
+					ret_folios);
 			/*
 			 * The rules are:
 			 *	0: folio will be put on unmap_folios list,
_

Patches currently in -mm which might be from qiuzhiteng@hygon.cn are

mm-migrate-dont-batch-the-tlb-flush-when-there-is-nothing-to-batch.patch


           reply	other threads:[~2026-11-22 17:16 UTC|newest]

Thread overview: expand[flat|nested]  mbox.gz  Atom feed
 [parent not found: <20261112010357.2739389-1-qiuzhiteng@hygon.cn>]

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=20261122171655.654BA1F00A3C@smtp.kernel.org \
    --to=akpm@linux-foundation.org \
    --cc=apopple@nvidia.com \
    --cc=byungchul@sk.com \
    --cc=dave.hansen@linux.intel.com \
    --cc=david@kernel.org \
    --cc=geshifei@hygon.cn \
    --cc=gourry@gourry.net \
    --cc=huangsj@hygon.cn \
    --cc=joshua.hahnjy@gmail.com \
    --cc=liuyibin@hygon.cn \
    --cc=matthew.brost@intel.com \
    --cc=mm-commits@vger.kernel.org \
    --cc=qiuzhiteng@hygon.cn \
    --cc=wangfengyu@hygon.cn \
    --cc=wujianyong@hygon.cn \
    --cc=xhao@linux.alibaba.com \
    --cc=ying.huang@linux.alibaba.com \
    --cc=zhongyuan@hygon.cn \
    --cc=ziy@nvidia.com \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox