All of lore.kernel.org
 help / color / mirror / Atom feed
From: "Barry Song (Xiaomi)" <baohua@kernel.org>
To: zippermonkey@icloud.com
Cc: akpm@linux-foundation.org, axelrasmussen@google.com,
	baohua@kernel.org, baolin.wang@linux.alibaba.com,
	baoquan.he@linux.dev, bruzzhang@tencent.com, david@kernel.org,
	hannes@cmpxchg.org, kasong@tencent.com,
	linux-kernel@vger.kernel.org, linux-mm@kvack.org, ljs@kernel.org,
	mhocko@kernel.org, qi.zheng@linux.dev, shakeel.butt@linux.dev,
	weixugc@google.com, yuanchu@google.com
Subject: [PATCH v6 0/5] mm: batch TLB flushing for dirty folios in vmscan
Date: Mon, 16 Nov 2026 07:02:31 +0800	[thread overview]
Message-ID: <20261115230231.21194-1-baohua@kernel.org> (raw)
In-Reply-To: <20261031-batch-tlb-flush-v6-0-190539839eec@tencent.com>

On Sat, Oct 31, 2026 at 6:35 PM Zhang Peng <zippermonkey@icloud.com> wrote:
>
> This series introduces batch TLB flushing optimization for dirty folios
> during memory reclaim, aiming to reduce IPI overhead on multi-core systems.
>
> Background
> ----------
> Reclaim calls try_to_unmap_flush_dirty() before writing out each dirty
> folio. When writable TLB entries are pending, this triggers a flush.
> On systems that use IPIs for TLB shootdowns, frequent flushes can add
> significant overhead.
>
> Approach
> --------
> Collect dirty folios into a folio_batch, releasing their locks while
> collecting the batch so swap faults can progress. Before writeout, try
> locking each folio again and recheck writeback, mappings and DMA pins.
> Keep eligible folios locked, flush pending writable TLB entries once,
> then page them out. Process a final partial batch at the end of the scan.
>
> Changes
> -------
> Patch 1: Extract the folio activation block at activate_locked into
>          folio_activate_locked().
> Patch 2: Extract the folio-freeing path (buffer release, lazyfree,
>          __remove_mapping, folio_batch drain) into folio_try_reclaim_free().
>          Group retained folios, deferred frees and the reclaimed count
>          in reclaim_folio_state, local to each shrink_folio_list() call.
> Patch 3: Extract the pageout() dispatch state machine into folio_try_pageout().
> Patch 4: Extract the TTU setup and try_to_unmap() block into folio_try_unmap().
> Patch 5: Implement batch TLB flushing logic. Dirty folios are accumulated in
>          batches and a single TLB flush is performed for each batch before
>          calling pageout.
>
> Testing
> -------
> The benchmark script uses stress-ng to compare TLB shootdown behavior before and
> after this patch. It constrains a stress-ng workload via memcg to force reclaim
> through shrink_folio_list(), reporting TLB shootdowns and IPIs.
>
> Core benchmark command: stress-ng --vm 16 --vm-bytes 2G --vm-keep --timeout 60
>
> ==========================================================================
>                  batch_dirty_tlb_flush Benchmark Results
> ==========================================================================
>   Kernel: 7.0.0-rc1+   CPUs: 16
>   MemTotal: 31834M   SwapTotal: 8191M
>   memcg limit: 512M   alloc: 2G   workers: 16   duration: 60s
> --------------------------------------------------------------------------
> Metric                 Before        After             Delta (abs / %)
> --------------------------------------------------------------------------
> bogo ops/s             28238.63      35833.97          +7595.34 (+26.9%)
> TLB shootdowns         55428953      17621697          -37807256 (-68.2%)
> Function call IPIs     34073695      14498768          -19574927 (-57.4%)
> pgscan_anon (pages)    52856224      60252894          7396670 (+14.0%)
> pgsteal_anon (pages)   29004962      34054753          5049791 (+17.4%)
> --------------------------------------------------------------------------
>
> Suggested-by: Kairui Song <kasong@tencent.com>
> Signed-off-by: Zhang Peng <bruzzhang@tencent.com>
> ---

Hi Peng,

Just a quick update: this patch is still on my radar. I ran a
quick perf test while building the kernel. Without your patch,
try_to_unmap_flush_dirty() accounted for 1.75% of CPU cycles.

I guess there could be workloads where try_to_unmap_flush_dirty()
accounts for an even larger share of CPU cycles.

# Children      Self  Command  Shared Object      Symbol                      
# ........  ........  .......  .................  ............................
#

     1.75%     0.01%  cc1              [kernel.kallsyms]  [.] try_to_unmap_flush_dirty
            |          
             --1.74%--try_to_unmap_flush_dirty
                       try_to_unmap_flush (inlined)
                       |          
                        --1.73%--arch_tlbbatch_flush
                                  |          
                                   --1.70%--flush_tlb_multi (inlined)
                                             __flush_tlb_multi (inlined)
                                             |          
                                              --1.69%--native_flush_tlb_multi
                                                        |          
                                                         --1.68%--on_each_cpu_cond_mask
                                                                   |          
                                                                    --1.62%--smp_call_function_many_cond
                                                                              |          
                                                                               --1.23%--csd_lock_wait (inlined)


With your patch, the CPU overhead drops by roughly 7x:


# Children      Self  Command  Shared Object      Symbol                      
# ........  ........  .......  .................  ............................
#
     0.25%     0.00%  cc1      [kernel.kallsyms]  [.] try_to_unmap_flush_dirty
            |          
             --0.24%--try_to_unmap_flush_dirty
                       try_to_unmap_flush (inlined)
                       |          
                        --0.24%--arch_tlbbatch_flush
                                  |          
                                   --0.24%--flush_tlb_multi (inlined)
                                             __flush_tlb_multi (inlined)
                                             |          
                                              --0.24%--native_flush_tlb_multi
                                                        |          
                                                         --0.23%--on_each_cpu_cond_mask
                                                                   |          
                                                                    --0.21%--smp_call_function_many_cond
                                                                              |          
                                                                               --0.14%--csd_lock_wait (inlined)

     0.00%     0.00%  fixdep   [kernel.kallsyms]  [.] try_to_unmap_flush_dirty


Overall, I think this is a good approach. I'll be traveling soon
and will return to this patchset for more testing and a thorough
review in about two and a half weeks.

Thanks
Barry


  parent reply	other threads:[~2026-11-15 23:02 UTC|newest]

Thread overview: 12+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2026-10-31 10:35 [PATCH v6 0/5] mm: batch TLB flushing for dirty folios in vmscan Zhang Peng
2026-10-31 10:35 ` [PATCH v6 1/5] mm/vmscan: introduce folio_activate_locked() helper Zhang Peng
2026-10-31 10:35 ` [PATCH v6 2/5] mm/vmscan: extract folio reclaim freeing from shrink_folio_list() Zhang Peng
2026-10-31 10:35 ` [PATCH v6 3/5] mm/vmscan: extract folio pageout " Zhang Peng
2026-10-31 10:35 ` [PATCH v6 4/5] mm/vmscan: extract folio unmap logic into folio_try_unmap() Zhang Peng
2026-10-31 10:35 ` [PATCH v6 5/5] mm/vmscan: batch dirty TLB flushes before pageout Zhang Peng
2026-11-01  2:51   ` sashiko-bot
2026-11-15 23:02 ` Barry Song (Xiaomi) [this message]
2026-11-15 23:27   ` [PATCH v6 0/5] mm: batch TLB flushing for dirty folios in vmscan Barry Song
2026-11-16 13:00   ` Zhang Peng
2026-11-16 12:10 ` David Hildenbrand (Arm)
2026-11-16 12:47   ` Zhang Peng

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=20261115230231.21194-1-baohua@kernel.org \
    --to=baohua@kernel.org \
    --cc=akpm@linux-foundation.org \
    --cc=axelrasmussen@google.com \
    --cc=baolin.wang@linux.alibaba.com \
    --cc=baoquan.he@linux.dev \
    --cc=bruzzhang@tencent.com \
    --cc=david@kernel.org \
    --cc=hannes@cmpxchg.org \
    --cc=kasong@tencent.com \
    --cc=linux-kernel@vger.kernel.org \
    --cc=linux-mm@kvack.org \
    --cc=ljs@kernel.org \
    --cc=mhocko@kernel.org \
    --cc=qi.zheng@linux.dev \
    --cc=shakeel.butt@linux.dev \
    --cc=weixugc@google.com \
    --cc=yuanchu@google.com \
    --cc=zippermonkey@icloud.com \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.