linux-mm.kvack.org archive mirror
 help / color / mirror / Atom feed
From: Feng Tang <feng.tang@intel.com>
To: Vlastimil Babka <vbabka@suse.cz>,
	Andrew Morton <akpm@linux-foundation.org>,
	Christoph Lameter <cl@linux.com>,
	Pekka Enberg <penberg@kernel.org>,
	David Rientjes <rientjes@google.com>,
	Joonsoo Kim <iamjoonsoo.kim@lge.com>,
	Roman Gushchin <roman.gushchin@linux.dev>,
	Hyeonggon Yoo <42.hyeyoo@gmail.com>,
	linux-mm@kvack.org, linux-kernel@vger.kernel.org
Cc: Feng Tang <feng.tang@intel.com>
Subject: [RFC Patch 1/3] mm/slub: increase the maximum slab order to 4 for big systems
Date: Tue,  5 Sep 2023 22:13:46 +0800	[thread overview]
Message-ID: <20230905141348.32946-2-feng.tang@intel.com> (raw)
In-Reply-To: <20230905141348.32946-1-feng.tang@intel.com>

There are reports about severe lock contention for slub's per-node
'list_lock' in 'hackbench' test, [1][2], on server systems. And
similar contention is also seen when running 'mmap1' case of
will-it-scale on big systems. As the trend is one processor (socket)
will have more and more CPUs (100+, 200+), the contention could be
much more severe and becomes a scalability issue.

One way to help reducing the contention is to increase the maximum
slab order from 3 to 4, for big systems.

Unconditionally increasing the order could  bring trouble to client
devices with very limited size of memory, which may care more about
memory footprint, also allocating order 4 page could be harder under
memory pressure. So the increase will only be done for big systems
like servers, which usually are equipped with plenty of memory and
easier to hit lock contention issues.

Following is some performance data:

will-it-scale/mmap1
-------------------
Run will-it-scale benchmark's 'mmap1' test case on a 2 socket Sapphire
Rapids server (112 cores / 224 threads) with 256 GB DRAM, run 3
configurations with parallel test threads of 25%, 50% and 100% of
number of CPUs, and the data is (base is vanilla v6.5 kernel):

		     base                      base+patch
wis-mmap1-25%	    223670           +33.3%     298205        per_process_ops
wis-mmap1-50%	    186020           +51.8%     282383        per_process_ops
wis-mmap1-100%       89200           +65.0%     147139        per_process_ops

Take the perf-profile comparasion of 50% test case, the lock contention
is greatly reduced:

      43.80           -30.8       13.04       pp.self.native_queued_spin_lock_slowpath
      0.85            -0.2        0.65        pp.self.___slab_alloc
      0.41            -0.1        0.27        pp.self.__unfreeze_partials
      0.20 ±  2%      -0.1        0.12 ±  4%  pp.self.get_any_partial

hackbench
---------

Run same hackbench testcase  mentioned in [1], use same HW/SW as will-it-scale:

		     base                      base+patch
hackbench	    759951           +10.5%     839601        hackbench.throughput

perf-profile diff:
     22.20 ±  3%     -15.2        7.05        pp.self.native_queued_spin_lock_slowpath
      0.82            -0.2        0.59        pp.self.___slab_alloc
      0.33            -0.2        0.13        pp.self.__unfreeze_partials

[1]. https://lore.kernel.org/all/202307172140.3b34825a-oliver.sang@intel.com/
[2]. ttps://lore.kernel.org/lkml/ZORaUsd+So+tnyMV@chenyu5-mobl2/
Signed-off-by: Feng Tang <feng.tang@intel.com>
---
 mm/slub.c | 51 ++++++++++++++++++++++++++++++++++++++-------------
 1 file changed, 38 insertions(+), 13 deletions(-)

diff --git a/mm/slub.c b/mm/slub.c
index f7940048138c..09ae1ed642b7 100644
--- a/mm/slub.c
+++ b/mm/slub.c
@@ -4081,7 +4081,7 @@ EXPORT_SYMBOL(kmem_cache_alloc_bulk);
  */
 static unsigned int slub_min_order;
 static unsigned int slub_max_order =
-	IS_ENABLED(CONFIG_SLUB_TINY) ? 1 : PAGE_ALLOC_COSTLY_ORDER;
+	IS_ENABLED(CONFIG_SLUB_TINY) ? 1 : 4;
 static unsigned int slub_min_objects;
 
 /*
@@ -4134,6 +4134,26 @@ static inline unsigned int calc_slab_order(unsigned int size,
 	return order;
 }
 
+static inline int num_cpus(void)
+{
+	int nr_cpus;
+
+	/*
+	 * Some architectures will only update present cpus when
+	 * onlining them, so don't trust the number if it's just 1. But
+	 * we also don't want to use nr_cpu_ids always, as on some other
+	 * architectures, there can be many possible cpus, but never
+	 * onlined. Here we compromise between trying to avoid too high
+	 * order on systems that appear larger than they are, and too
+	 * low order on systems that appear smaller than they are.
+	 */
+	nr_cpus = num_present_cpus();
+	if (nr_cpus <= 1)
+		nr_cpus = nr_cpu_ids;
+
+	return nr_cpus;
+}
+
 static inline int calculate_order(unsigned int size)
 {
 	unsigned int order;
@@ -4151,19 +4171,17 @@ static inline int calculate_order(unsigned int size)
 	 */
 	min_objects = slub_min_objects;
 	if (!min_objects) {
-		/*
-		 * Some architectures will only update present cpus when
-		 * onlining them, so don't trust the number if it's just 1. But
-		 * we also don't want to use nr_cpu_ids always, as on some other
-		 * architectures, there can be many possible cpus, but never
-		 * onlined. Here we compromise between trying to avoid too high
-		 * order on systems that appear larger than they are, and too
-		 * low order on systems that appear smaller than they are.
-		 */
-		nr_cpus = num_present_cpus();
-		if (nr_cpus <= 1)
-			nr_cpus = nr_cpu_ids;
+		nr_cpus = num_cpus();
 		min_objects = 4 * (fls(nr_cpus) + 1);
+
+		/*
+		 * If nr_cpus >= 32, the platform is likely to be a server
+		 * which usually has much more memory, and is easier to be
+		 * hurt by scalability issue, so enlarge it to reduce the
+		 * possible contention of the per-node 'list_lock'.
+		 */
+		if (nr_cpus >= 32)
+			min_objects *= 2;
 	}
 	max_objects = order_objects(slub_max_order, size);
 	min_objects = min(min_objects, max_objects);
@@ -4361,6 +4379,13 @@ static void set_cpu_partial(struct kmem_cache *s)
 	else
 		nr_objects = 120;
 
+	/*
+	 * Give larger system more buffer to reduce scalability issue, like
+	 * the handling in calculate_order().
+	 */
+	if (num_cpus() >= 32)
+		nr_objects *= 2;
+
 	slub_set_cpu_partial(s, nr_objects);
 #endif
 }
-- 
2.27.0



  reply	other threads:[~2023-09-05 14:08 UTC|newest]

Thread overview: 11+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2023-09-05 14:13 [RFC Patch 0/3] mm/slub: reduce contention for per-node list_lock for large systems Feng Tang
2023-09-05 14:13 ` Feng Tang [this message]
2023-09-12  4:52   ` [RFC Patch 1/3] mm/slub: increase the maximum slab order to 4 for big systems Hyeonggon Yoo
2023-09-12 15:52     ` Feng Tang
2023-09-05 14:13 ` [RFC Patch 2/3] mm/slub: double per-cpu partial number for large systems Feng Tang
2023-09-05 14:13 ` [RFC Patch 3/3] mm/slub: setup maxim per-node partial according to cpu numbers Feng Tang
2023-09-12  4:48   ` Hyeonggon Yoo
2023-09-14  7:05     ` Feng Tang
2023-09-15  2:40       ` Lameter, Christopher
2023-09-15  5:05         ` Feng Tang
2023-09-15 16:13           ` Lameter, Christopher

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=20230905141348.32946-2-feng.tang@intel.com \
    --to=feng.tang@intel.com \
    --cc=42.hyeyoo@gmail.com \
    --cc=akpm@linux-foundation.org \
    --cc=cl@linux.com \
    --cc=iamjoonsoo.kim@lge.com \
    --cc=linux-kernel@vger.kernel.org \
    --cc=linux-mm@kvack.org \
    --cc=penberg@kernel.org \
    --cc=rientjes@google.com \
    --cc=roman.gushchin@linux.dev \
    --cc=vbabka@suse.cz \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).