From: Byungchul Park <byungchul@sk.com>
To: Dave Hansen <dave.hansen@intel.com>
Cc: linux-kernel@vger.kernel.org, linux-mm@kvack.org,
kernel_team@skhynix.com, akpm@linux-foundation.org,
ying.huang@intel.com, namit@vmware.com, xhao@linux.alibaba.com,
mgorman@techsingularity.net, hughd@google.com,
willy@infradead.org, david@redhat.com, peterz@infradead.org,
luto@kernel.org, tglx@linutronix.de, mingo@redhat.com,
bp@alien8.de, dave.hansen@linux.intel.com
Subject: Re: [v3 0/3] Reduce TLB flushes under some specific conditions
Date: Tue, 31 Oct 2023 11:37:04 +0900 [thread overview]
Message-ID: <20231031023704.GA39338@system.software.com> (raw)
In-Reply-To: <08c82a91-87d1-42c7-93c4-4028f3725340@intel.com>
On Mon, Oct 30, 2023 at 10:55:07AM -0700, Dave Hansen wrote:
> On 10/30/23 00:25, Byungchul Park wrote:
> > I'm suggesting a mechanism to reduce TLB flushes by keeping source and
> > destination of folios participated in the migrations until all TLB
> > flushes required are done, only if those folios are not mapped with
> > write permission PTE entries at all. I worked Based on v6.6-rc5.
>
> There's a lot of common overhead here, on top of the complexity in general:
>
> * A new page flag
> * A new cpumask_t in task_struct
> * A new zone list
> * Extra (temporary) memory consumption
>
> and the benefits are ... "performance improved a little bit" on one
> workload. That doesn't seem like a good overall tradeoff to me.
I tested it under limited conditions to get stable results e.g. not to
use hyper-thread, dedicate cpu times to the test and so on. However, I'm
conviced that this patch set is more worth developing than you think it
is. Lemme share the results I've just got after changing # of CPUs
participated in the test, 16 -> 80, in the system with 80 CPUs. This is
just for your information - not that stable tho.
Byungchul
---
Architecture - x86_64
QEMU - kvm enabled, host cpu
Numa - 2 nodes (80 CPUs 1GB, no CPUs 8GB)
Linux Kernel - v6.6-rc5, numa balancing tiering on, demotion enabled
Benchmark - XSBench -p 50000000 (-p option makes the runtime longer)
mainline kernel
===============
The 1st try)
=====================================
Threads: 64
Runtime: 233.118 seconds
=====================================
numa_pages_migrated 758334
pgmigrate_success 1724964
nr_tlb_remote_flush 305706
nr_tlb_remote_flush_received 18598543
nr_tlb_local_flush_all 19092
nr_tlb_local_flush_one 4518717
The 2nd try)
=====================================
Threads: 64
Runtime: 221.725 seconds
=====================================
numa_pages_migrated 633209
pgmigrate_success 2156509
nr_tlb_remote_flush 261977
nr_tlb_remote_flush_received 14289256
nr_tlb_local_flush_all 11738
nr_tlb_local_flush_one 4520317
mainline kernel + migrc
=======================
The 1st try)
=====================================
Threads: 64
Runtime: 212.522 seconds
====================================
numa_pages_migrated 901264
pgmigrate_success 1990814
nr_tlb_remote_flush 151280
nr_tlb_remote_flush_received 9031376
nr_tlb_local_flush_all 21208
nr_tlb_local_flush_one 4519595
The 2nd try)
=====================================
Threads: 64
Runtime: 204.410 seconds
====================================
numa_pages_migrated 929260
pgmigrate_success 2729868
nr_tlb_remote_flush 166722
nr_tlb_remote_flush_received 8238273
nr_tlb_local_flush_all 13717
nr_tlb_local_flush_one 4519582
prev parent reply other threads:[~2023-10-31 2:37 UTC|newest]
Thread overview: 27+ messages / expand[flat|nested] mbox.gz Atom feed top
2023-10-30 7:25 [v3 0/3] Reduce TLB flushes under some specific conditions Byungchul Park
2023-10-30 7:25 ` [v3 1/3] mm/rmap: Recognize non-writable TLB entries during TLB batch flush Byungchul Park
2023-10-30 7:52 ` Nadav Amit
2023-10-30 10:26 ` Byungchul Park
2023-10-30 7:25 ` [v3 2/3] mm: Defer TLB flush by keeping both src and dst folios at migration Byungchul Park
2023-10-30 8:00 ` David Hildenbrand
2023-10-30 9:58 ` Byungchul Park
2023-11-01 3:06 ` Huang, Ying
2023-10-30 8:50 ` Nadav Amit
2023-10-30 12:51 ` Byungchul Park
2023-10-30 15:58 ` Nadav Amit
2023-10-30 22:40 ` Byungchul Park
2023-11-08 4:12 ` Byungchul Park
2023-11-09 10:16 ` Nadav Amit
2023-11-10 1:02 ` Byungchul Park
2023-11-10 3:13 ` Byungchul Park
2023-11-10 22:18 ` Nadav Amit
2023-11-15 5:48 ` Byungchul Park
2023-11-09 5:35 ` Byungchul Park
2023-10-30 7:25 ` [v3 3/3] mm, migrc: Add a sysctl knob to enable/disable MIGRC mechanism Byungchul Park
2023-10-30 8:51 ` Nadav Amit
2023-10-30 10:36 ` Byungchul Park
2023-10-30 17:55 ` [v3 0/3] Reduce TLB flushes under some specific conditions Dave Hansen
2023-10-30 18:32 ` Nadav Amit
2023-10-30 22:55 ` Byungchul Park
2023-10-31 8:46 ` David Hildenbrand
2023-10-31 2:37 ` Byungchul Park [this message]
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20231031023704.GA39338@system.software.com \
--to=byungchul@sk.com \
--cc=akpm@linux-foundation.org \
--cc=bp@alien8.de \
--cc=dave.hansen@intel.com \
--cc=dave.hansen@linux.intel.com \
--cc=david@redhat.com \
--cc=hughd@google.com \
--cc=kernel_team@skhynix.com \
--cc=linux-kernel@vger.kernel.org \
--cc=linux-mm@kvack.org \
--cc=luto@kernel.org \
--cc=mgorman@techsingularity.net \
--cc=mingo@redhat.com \
--cc=namit@vmware.com \
--cc=peterz@infradead.org \
--cc=tglx@linutronix.de \
--cc=willy@infradead.org \
--cc=xhao@linux.alibaba.com \
--cc=ying.huang@intel.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).