[PATCH v11 17/19] tools headers: Synchronize prctl.h ABI header

[PATCH v11 17/19] tools headers: Synchronize prctl.h ABI header

[Sebastian Andrzej Siewior]

Synchronize prctl.h with current uapi version after adding
PR_FUTEX_HASH.

Cc: "Liang, Kan" <kan.liang@linux.intel.com>
Cc: Adrian Hunter <adrian.hunter@intel.com>
Cc: Alexander Shishkin <alexander.shishkin@linux.intel.com>
Cc: Arnaldo Carvalho de Melo <acme@kernel.org>
Cc: Ian Rogers <irogers@google.com>
Cc: Ingo Molnar <mingo@redhat.com>
Cc: Jiri Olsa <jolsa@kernel.org>
Cc: Mark Rutland <mark.rutland@arm.com>
Cc: Namhyung Kim <namhyung@kernel.org>
Cc: linux-perf-users@vger.kernel.org
Signed-off-by: Sebastian Andrzej Siewior <bigeasy@linutronix.de>
---
 tools/include/uapi/linux/prctl.h | 43 +++++++++++++++++++++++++++++++-
 1 file changed, 42 insertions(+), 1 deletion(-)

diff --git a/tools/include/uapi/linux/prctl.h b/tools/include/uapi/linux/prctl.h
index 35791791a879b..3b93fb906e3c5 100644
--- a/tools/include/uapi/linux/prctl.h
+++ b/tools/include/uapi/linux/prctl.h
@@ -230,7 +230,7 @@ struct prctl_mm_map {
 # define PR_PAC_APDBKEY			(1UL << 3)
 # define PR_PAC_APGAKEY			(1UL << 4)
 
-/* Tagged user address controls for arm64 */
+/* Tagged user address controls for arm64 and RISC-V */
 #define PR_SET_TAGGED_ADDR_CTRL		55
 #define PR_GET_TAGGED_ADDR_CTRL		56
 # define PR_TAGGED_ADDR_ENABLE		(1UL << 0)
@@ -244,6 +244,9 @@ struct prctl_mm_map {
 # define PR_MTE_TAG_MASK		(0xffffUL << PR_MTE_TAG_SHIFT)
 /* Unused; kept only for source compatibility */
 # define PR_MTE_TCF_SHIFT		1
+/* RISC-V pointer masking tag length */
+# define PR_PMLEN_SHIFT			24
+# define PR_PMLEN_MASK			(0x7fUL << PR_PMLEN_SHIFT)
 
 /* Control reclaim behavior when allocating memory */
 #define PR_SET_IO_FLUSHER		57
@@ -328,4 +331,42 @@ struct prctl_mm_map {
 # define PR_PPC_DEXCR_CTRL_CLEAR_ONEXEC	0x10 /* Clear the aspect on exec */
 # define PR_PPC_DEXCR_CTRL_MASK		0x1f
 
+/*
+ * Get the current shadow stack configuration for the current thread,
+ * this will be the value configured via PR_SET_SHADOW_STACK_STATUS.
+ */
+#define PR_GET_SHADOW_STACK_STATUS      74
+
+/*
+ * Set the current shadow stack configuration.  Enabling the shadow
+ * stack will cause a shadow stack to be allocated for the thread.
+ */
+#define PR_SET_SHADOW_STACK_STATUS      75
+# define PR_SHADOW_STACK_ENABLE         (1UL << 0)
+# define PR_SHADOW_STACK_WRITE		(1UL << 1)
+# define PR_SHADOW_STACK_PUSH		(1UL << 2)
+
+/*
+ * Prevent further changes to the specified shadow stack
+ * configuration.  All bits may be locked via this call, including
+ * undefined bits.
+ */
+#define PR_LOCK_SHADOW_STACK_STATUS      76
+
+/*
+ * Controls the mode of timer_create() for CRIU restore operations.
+ * Enabling this allows CRIU to restore timers with explicit IDs.
+ *
+ * Don't use for normal operations as the result might be undefined.
+ */
+#define PR_TIMER_CREATE_RESTORE_IDS		77
+# define PR_TIMER_CREATE_RESTORE_IDS_OFF	0
+# define PR_TIMER_CREATE_RESTORE_IDS_ON		1
+# define PR_TIMER_CREATE_RESTORE_IDS_GET	2
+
+/* FUTEX hash management */
+#define PR_FUTEX_HASH			78
+# define PR_FUTEX_HASH_SET_SLOTS	1
+# define PR_FUTEX_HASH_GET_SLOTS	2
+
 #endif /* _LINUX_PRCTL_H */
-- 
2.49.0

[PATCH v11 18/19] tools/perf: Allow to select the number of hash buckets.

[Sebastian Andrzej Siewior]

Add the -b/ --buckets argument to specify the number of hash buckets for
the private futex hash. This is directly passed to
    prctl(PR_FUTEX_HASH, PR_FUTEX_HASH_SET_SLOTS)

and must return without an error if specified. The size of the private
hash is verified with PR_FUTEX_HASH_GET_SLOTS.
If the PR_FUTEX_HASH_GET_SLOTS failed then it is assumed that an older
kernel was used without the support and that the global hash is used.

Cc: "Liang, Kan" <kan.liang@linux.intel.com>
Cc: Adrian Hunter <adrian.hunter@intel.com>
Cc: Alexander Shishkin <alexander.shishkin@linux.intel.com>
Cc: Arnaldo Carvalho de Melo <acme@kernel.org>
Cc: Ian Rogers <irogers@google.com>
Cc: Ingo Molnar <mingo@redhat.com>
Cc: Jiri Olsa <jolsa@kernel.org>
Cc: Mark Rutland <mark.rutland@arm.com>
Cc: Namhyung Kim <namhyung@kernel.org>
Cc: linux-perf-users@vger.kernel.org
Signed-off-by: Sebastian Andrzej Siewior <bigeasy@linutronix.de>
---
 tools/perf/bench/Build                 |  1 +
 tools/perf/bench/futex-hash.c          |  6 +++
 tools/perf/bench/futex-lock-pi.c       |  4 ++
 tools/perf/bench/futex-requeue.c       |  5 +++
 tools/perf/bench/futex-wake-parallel.c |  8 +++-
 tools/perf/bench/futex-wake.c          |  3 ++
 tools/perf/bench/futex.c               | 58 ++++++++++++++++++++++++++
 tools/perf/bench/futex.h               |  4 ++
 8 files changed, 88 insertions(+), 1 deletion(-)
 create mode 100644 tools/perf/bench/futex.c

diff --git a/tools/perf/bench/Build b/tools/perf/bench/Build
index 279ab2ab4abe4..b558ab98719f9 100644
--- a/tools/perf/bench/Build
+++ b/tools/perf/bench/Build
@@ -3,6 +3,7 @@ perf-bench-y += sched-pipe.o
 perf-bench-y += sched-seccomp-notify.o
 perf-bench-y += syscall.o
 perf-bench-y += mem-functions.o
+perf-bench-y += futex.o
 perf-bench-y += futex-hash.o
 perf-bench-y += futex-wake.o
 perf-bench-y += futex-wake-parallel.o
diff --git a/tools/perf/bench/futex-hash.c b/tools/perf/bench/futex-hash.c
index b472eded521b1..c843bd8543c74 100644
--- a/tools/perf/bench/futex-hash.c
+++ b/tools/perf/bench/futex-hash.c
@@ -18,9 +18,11 @@
 #include <stdlib.h>
 #include <linux/compiler.h>
 #include <linux/kernel.h>
+#include <linux/prctl.h>
 #include <linux/zalloc.h>
 #include <sys/time.h>
 #include <sys/mman.h>
+#include <sys/prctl.h>
 #include <perf/cpumap.h>
 
 #include "../util/mutex.h"
@@ -50,9 +52,11 @@ struct worker {
 static struct bench_futex_parameters params = {
 	.nfutexes = 1024,
 	.runtime  = 10,
+	.nbuckets = -1,
 };
 
 static const struct option options[] = {
+	OPT_INTEGER( 'b', "buckets", &params.nbuckets, "Specify amount of hash buckets"),
 	OPT_UINTEGER('t', "threads", &params.nthreads, "Specify amount of threads"),
 	OPT_UINTEGER('r', "runtime", &params.runtime, "Specify runtime (in seconds)"),
 	OPT_UINTEGER('f', "futexes", &params.nfutexes, "Specify amount of futexes per threads"),
@@ -118,6 +122,7 @@ static void print_summary(void)
 	printf("%sAveraged %ld operations/sec (+- %.2f%%), total secs = %d\n",
 	       !params.silent ? "\n" : "", avg, rel_stddev_stats(stddev, avg),
 	       (int)bench__runtime.tv_sec);
+	futex_print_nbuckets(&params);
 }
 
 int bench_futex_hash(int argc, const char **argv)
@@ -161,6 +166,7 @@ int bench_futex_hash(int argc, const char **argv)
 
 	if (!params.fshared)
 		futex_flag = FUTEX_PRIVATE_FLAG;
+	futex_set_nbuckets_param(&params);
 
 	printf("Run summary [PID %d]: %d threads, each operating on %d [%s] futexes for %d secs.\n\n",
 	       getpid(), params.nthreads, params.nfutexes, params.fshared ? "shared":"private", params.runtime);
diff --git a/tools/perf/bench/futex-lock-pi.c b/tools/perf/bench/futex-lock-pi.c
index 0416120c091b2..40640b6744279 100644
--- a/tools/perf/bench/futex-lock-pi.c
+++ b/tools/perf/bench/futex-lock-pi.c
@@ -41,10 +41,12 @@ static struct stats throughput_stats;
 static struct cond thread_parent, thread_worker;
 
 static struct bench_futex_parameters params = {
+	.nbuckets = -1,
 	.runtime  = 10,
 };
 
 static const struct option options[] = {
+	OPT_INTEGER( 'b', "buckets", &params.nbuckets, "Specify amount of hash buckets"),
 	OPT_UINTEGER('t', "threads", &params.nthreads, "Specify amount of threads"),
 	OPT_UINTEGER('r', "runtime", &params.runtime, "Specify runtime (in seconds)"),
 	OPT_BOOLEAN( 'M', "multi",   &params.multi, "Use multiple futexes"),
@@ -67,6 +69,7 @@ static void print_summary(void)
 	printf("%sAveraged %ld operations/sec (+- %.2f%%), total secs = %d\n",
 	       !params.silent ? "\n" : "", avg, rel_stddev_stats(stddev, avg),
 	       (int)bench__runtime.tv_sec);
+	futex_print_nbuckets(&params);
 }
 
 static void toggle_done(int sig __maybe_unused,
@@ -203,6 +206,7 @@ int bench_futex_lock_pi(int argc, const char **argv)
 	mutex_init(&thread_lock);
 	cond_init(&thread_parent);
 	cond_init(&thread_worker);
+	futex_set_nbuckets_param(&params);
 
 	threads_starting = params.nthreads;
 	gettimeofday(&bench__start, NULL);
diff --git a/tools/perf/bench/futex-requeue.c b/tools/perf/bench/futex-requeue.c
index aad5bfc4fe188..0748b0fd689e8 100644
--- a/tools/perf/bench/futex-requeue.c
+++ b/tools/perf/bench/futex-requeue.c
@@ -42,6 +42,7 @@ static unsigned int threads_starting;
 static int futex_flag = 0;
 
 static struct bench_futex_parameters params = {
+	.nbuckets = -1,
 	/*
 	 * How many tasks to requeue at a time.
 	 * Default to 1 in order to make the kernel work more.
@@ -50,6 +51,7 @@ static struct bench_futex_parameters params = {
 };
 
 static const struct option options[] = {
+	OPT_INTEGER( 'b', "buckets", &params.nbuckets, "Specify amount of hash buckets"),
 	OPT_UINTEGER('t', "threads",  &params.nthreads, "Specify amount of threads"),
 	OPT_UINTEGER('q', "nrequeue", &params.nrequeue, "Specify amount of threads to requeue at once"),
 	OPT_BOOLEAN( 's', "silent",   &params.silent, "Silent mode: do not display data/details"),
@@ -77,6 +79,7 @@ static void print_summary(void)
 	       params.nthreads,
 	       requeuetime_avg / USEC_PER_MSEC,
 	       rel_stddev_stats(requeuetime_stddev, requeuetime_avg));
+	futex_print_nbuckets(&params);
 }
 
 static void *workerfn(void *arg __maybe_unused)
@@ -204,6 +207,8 @@ int bench_futex_requeue(int argc, const char **argv)
 	if (params.broadcast)
 		params.nrequeue = params.nthreads;
 
+	futex_set_nbuckets_param(&params);
+
 	printf("Run summary [PID %d]: Requeuing %d threads (from [%s] %p to %s%p), "
 	       "%d at a time.\n\n",  getpid(), params.nthreads,
 	       params.fshared ? "shared":"private", &futex1,
diff --git a/tools/perf/bench/futex-wake-parallel.c b/tools/perf/bench/futex-wake-parallel.c
index 4352e318631e9..6aede7c46b337 100644
--- a/tools/perf/bench/futex-wake-parallel.c
+++ b/tools/perf/bench/futex-wake-parallel.c
@@ -57,9 +57,12 @@ static struct stats waketime_stats, wakeup_stats;
 static unsigned int threads_starting;
 static int futex_flag = 0;
 
-static struct bench_futex_parameters params;
+static struct bench_futex_parameters params = {
+	.nbuckets = -1,
+};
 
 static const struct option options[] = {
+	OPT_INTEGER( 'b', "buckets", &params.nbuckets, "Specify amount of hash buckets"),
 	OPT_UINTEGER('t', "threads", &params.nthreads, "Specify amount of threads"),
 	OPT_UINTEGER('w', "nwakers", &params.nwakes, "Specify amount of waking threads"),
 	OPT_BOOLEAN( 's', "silent",  &params.silent, "Silent mode: do not display data/details"),
@@ -218,6 +221,7 @@ static void print_summary(void)
 	       params.nthreads,
 	       waketime_avg / USEC_PER_MSEC,
 	       rel_stddev_stats(waketime_stddev, waketime_avg));
+	futex_print_nbuckets(&params);
 }
 
 
@@ -291,6 +295,8 @@ int bench_futex_wake_parallel(int argc, const char **argv)
 	if (!params.fshared)
 		futex_flag = FUTEX_PRIVATE_FLAG;
 
+	futex_set_nbuckets_param(&params);
+
 	printf("Run summary [PID %d]: blocking on %d threads (at [%s] "
 	       "futex %p), %d threads waking up %d at a time.\n\n",
 	       getpid(), params.nthreads, params.fshared ? "shared":"private",
diff --git a/tools/perf/bench/futex-wake.c b/tools/perf/bench/futex-wake.c
index 49b3c89b0b35d..a31fc1563862e 100644
--- a/tools/perf/bench/futex-wake.c
+++ b/tools/perf/bench/futex-wake.c
@@ -42,6 +42,7 @@ static unsigned int threads_starting;
 static int futex_flag = 0;
 
 static struct bench_futex_parameters params = {
+	.nbuckets = -1,
 	/*
 	 * How many wakeups to do at a time.
 	 * Default to 1 in order to make the kernel work more.
@@ -50,6 +51,7 @@ static struct bench_futex_parameters params = {
 };
 
 static const struct option options[] = {
+	OPT_INTEGER( 'b', "buckets", &params.nbuckets, "Specify amount of hash buckets"),
 	OPT_UINTEGER('t', "threads", &params.nthreads, "Specify amount of threads"),
 	OPT_UINTEGER('w', "nwakes",  &params.nwakes, "Specify amount of threads to wake at once"),
 	OPT_BOOLEAN( 's', "silent",  &params.silent, "Silent mode: do not display data/details"),
@@ -93,6 +95,7 @@ static void print_summary(void)
 	       params.nthreads,
 	       waketime_avg / USEC_PER_MSEC,
 	       rel_stddev_stats(waketime_stddev, waketime_avg));
+	futex_print_nbuckets(&params);
 }
 
 static void block_threads(pthread_t *w, struct perf_cpu_map *cpu)
diff --git a/tools/perf/bench/futex.c b/tools/perf/bench/futex.c
new file mode 100644
index 0000000000000..8109d6bf3ede2
--- /dev/null
+++ b/tools/perf/bench/futex.c
@@ -0,0 +1,58 @@
+// SPDX-License-Identifier: GPL-2.0
+#include <err.h>
+#include <stdio.h>
+#include <stdlib.h>
+#include <linux/prctl.h>
+#include <sys/prctl.h>
+
+#include "futex.h"
+
+void futex_set_nbuckets_param(struct bench_futex_parameters *params)
+{
+	int ret;
+
+	if (params->nbuckets < 0)
+		return;
+
+	ret = prctl(PR_FUTEX_HASH, PR_FUTEX_HASH_SET_SLOTS, params->nbuckets);
+	if (ret) {
+		printf("Requesting %d hash buckets failed: %d/%m\n",
+		       params->nbuckets, ret);
+		err(EXIT_FAILURE, "prctl(PR_FUTEX_HASH)");
+	}
+}
+
+void futex_print_nbuckets(struct bench_futex_parameters *params)
+{
+	char *futex_hash_mode;
+	int ret;
+
+	ret = prctl(PR_FUTEX_HASH, PR_FUTEX_HASH_GET_SLOTS);
+	if (params->nbuckets >= 0) {
+		if (ret != params->nbuckets) {
+			if (ret < 0) {
+				printf("Can't query number of buckets: %d/%m\n", ret);
+				err(EXIT_FAILURE, "prctl(PR_FUTEX_HASH)");
+			}
+			printf("Requested number of hash buckets does not currently used.\n");
+			printf("Requested: %d in usage: %d\n", params->nbuckets, ret);
+			err(EXIT_FAILURE, "prctl(PR_FUTEX_HASH)");
+		}
+		if (params->nbuckets == 0)
+			ret = asprintf(&futex_hash_mode, "Futex hashing: global hash");
+		else
+			ret = asprintf(&futex_hash_mode, "Futex hashing: %d hash buckets",
+				       params->nbuckets);
+	} else {
+		if (ret <= 0) {
+			ret = asprintf(&futex_hash_mode, "Futex hashing: global hash");
+		} else {
+			ret = asprintf(&futex_hash_mode, "Futex hashing: auto resized to %d buckets",
+				       ret);
+		}
+	}
+	if (ret < 0)
+		err(EXIT_FAILURE, "ENOMEM, futex_hash_mode");
+	printf("%s\n", futex_hash_mode);
+	free(futex_hash_mode);
+}
diff --git a/tools/perf/bench/futex.h b/tools/perf/bench/futex.h
index ebdc2b032afc1..dd295d27044ac 100644
--- a/tools/perf/bench/futex.h
+++ b/tools/perf/bench/futex.h
@@ -25,6 +25,7 @@ struct bench_futex_parameters {
 	unsigned int nfutexes;
 	unsigned int nwakes;
 	unsigned int nrequeue;
+	int nbuckets;
 };
 
 /**
@@ -143,4 +144,7 @@ futex_cmp_requeue_pi(u_int32_t *uaddr, u_int32_t val, u_int32_t *uaddr2,
 					val, opflags);
 }
 
+void futex_set_nbuckets_param(struct bench_futex_parameters *params);
+void futex_print_nbuckets(struct bench_futex_parameters *params);
+
 #endif /* _FUTEX_H */
-- 
2.49.0

[PATCH v11 19/19] futex: Allow to make the private hash immutable.

[Sebastian Andrzej Siewior]

My initial testing showed that
	perf bench futex hash

reported less operations/sec with private hash. After using the same
amount of buckets in the private hash as used by the global hash then
the operations/sec were about the same.

This changed once the private hash became resizable. This feature added
a RCU section and reference counting via atomic inc+dec operation into
the hot path.
The reference counting can be avoided if the private hash is made
immutable.
Extend PR_FUTEX_HASH_SET_SLOTS by a fourth argument which denotes if the
private should be made immutable. Once set (to true) the a further
resize is not allowed (same if set to global hash).
Add PR_FUTEX_HASH_GET_IMMUTABLE which returns true if the hash can not
be changed.
Update "perf bench" suite.

For comparison, results of "perf bench futex hash -s":
- Xeon CPU E5-2650, 2 NUMA nodes, total 32 CPUs:
  - Before the introducing task local hash
    shared  Averaged 1.487.148 operations/sec (+- 0,53%), total secs = 10
    private Averaged 2.192.405 operations/sec (+- 0,07%), total secs = 10

  - With the series
    shared  Averaged 1.326.342 operations/sec (+- 0,41%), total secs = 10
    -b128   Averaged   141.394 operations/sec (+- 1,15%), total secs = 10
    -Ib128  Averaged   851.490 operations/sec (+- 0,67%), total secs = 10
    -b8192  Averaged   131.321 operations/sec (+- 2,13%), total secs = 10
    -Ib8192 Averaged 1.923.077 operations/sec (+- 0,61%), total secs = 10
    128 is the default allocation of hash buckets.
    8192 was the previous amount of allocated hash buckets.

- Xeon(R) CPU E7-8890 v3, 4 NUMA nodes, total 144 CPUs:
  - Before the introducing task local hash
    shared   Averaged 1.810.936 operations/sec (+- 0,26%), total secs = 20
    private  Averaged 2.505.801 operations/sec (+- 0,05%), total secs = 20

  - With the series
    shared   Averaged 1.589.002 operations/sec (+- 0,25%), total secs = 20
    -b1024   Averaged    42.410 operations/sec (+- 0,20%), total secs = 20
    -Ib1024  Averaged   740.638 operations/sec (+- 1,51%), total secs = 20
    -b65536  Averaged    48.811 operations/sec (+- 1,35%), total secs = 20
    -Ib65536 Averaged 1.963.165 operations/sec (+- 0,18%), total secs = 20
    1024 is the default allocation of hash buckets.
    65536 was the previous amount of allocated hash buckets.

Cc: "Liang, Kan" <kan.liang@linux.intel.com>
Cc: Adrian Hunter <adrian.hunter@intel.com>
Cc: Alexander Shishkin <alexander.shishkin@linux.intel.com>
Cc: Arnaldo Carvalho de Melo <acme@kernel.org>
Cc: Ian Rogers <irogers@google.com>
Cc: Ingo Molnar <mingo@redhat.com>
Cc: Jiri Olsa <jolsa@kernel.org>
Cc: Mark Rutland <mark.rutland@arm.com>
Cc: Namhyung Kim <namhyung@kernel.org>
Cc: linux-perf-users@vger.kernel.org
Signed-off-by: Sebastian Andrzej Siewior <bigeasy@linutronix.de>
---
 include/linux/futex.h                  |  2 +-
 include/uapi/linux/prctl.h             |  1 +
 kernel/futex/core.c                    | 42 ++++++++++++++++++++++----
 kernel/sys.c                           |  2 +-
 tools/include/uapi/linux/prctl.h       |  1 +
 tools/perf/bench/futex-hash.c          |  1 +
 tools/perf/bench/futex-lock-pi.c       |  1 +
 tools/perf/bench/futex-requeue.c       |  1 +
 tools/perf/bench/futex-wake-parallel.c |  1 +
 tools/perf/bench/futex-wake.c          |  1 +
 tools/perf/bench/futex.c               |  8 +++--
 tools/perf/bench/futex.h               |  1 +
 12 files changed, 51 insertions(+), 11 deletions(-)

diff --git a/include/linux/futex.h b/include/linux/futex.h
index ee48dcfbfe59d..96c7229856d97 100644
--- a/include/linux/futex.h
+++ b/include/linux/futex.h
@@ -80,7 +80,7 @@ void futex_exec_release(struct task_struct *tsk);
 
 long do_futex(u32 __user *uaddr, int op, u32 val, ktime_t *timeout,
 	      u32 __user *uaddr2, u32 val2, u32 val3);
-int futex_hash_prctl(unsigned long arg2, unsigned long arg3);
+int futex_hash_prctl(unsigned long arg2, unsigned long arg3, unsigned long arg4);
 
 #ifdef CONFIG_FUTEX_PRIVATE_HASH
 int futex_hash_allocate_default(void);
diff --git a/include/uapi/linux/prctl.h b/include/uapi/linux/prctl.h
index 3b93fb906e3c5..21f30b3ded74b 100644
--- a/include/uapi/linux/prctl.h
+++ b/include/uapi/linux/prctl.h
@@ -368,5 +368,6 @@ struct prctl_mm_map {
 #define PR_FUTEX_HASH			78
 # define PR_FUTEX_HASH_SET_SLOTS	1
 # define PR_FUTEX_HASH_GET_SLOTS	2
+# define PR_FUTEX_HASH_GET_IMMUTABLE	3
 
 #endif /* _LINUX_PRCTL_H */
diff --git a/kernel/futex/core.c b/kernel/futex/core.c
index 5b8609c8729e7..44bb9eeb0a9c1 100644
--- a/kernel/futex/core.c
+++ b/kernel/futex/core.c
@@ -70,6 +70,7 @@ struct futex_private_hash {
 	struct rcu_head	rcu;
 	void		*mm;
 	bool		custom;
+	bool		immutable;
 	struct futex_hash_bucket queues[];
 };
 
@@ -139,12 +140,16 @@ static inline bool futex_key_is_private(union futex_key *key)
 
 bool futex_private_hash_get(struct futex_private_hash *fph)
 {
+	if (fph->immutable)
+		return true;
 	return rcuref_get(&fph->users);
 }
 
 void futex_private_hash_put(struct futex_private_hash *fph)
 {
 	/* Ignore return value, last put is verified via rcuref_is_dead() */
+	if (fph->immutable)
+		return;
 	if (rcuref_put(&fph->users))
 		wake_up_var(fph->mm);
 }
@@ -284,6 +289,8 @@ struct futex_private_hash *futex_private_hash(void)
 		if (!fph)
 			return NULL;
 
+		if (fph->immutable)
+			return fph;
 		if (rcuref_get(&fph->users))
 			return fph;
 	}
@@ -1558,7 +1565,7 @@ static bool futex_hash_less(struct futex_private_hash *a,
 	return false; /* equal */
 }
 
-static int futex_hash_allocate(unsigned int hash_slots, bool custom)
+static int futex_hash_allocate(unsigned int hash_slots, unsigned int immutable, bool custom)
 {
 	struct mm_struct *mm = current->mm;
 	struct futex_private_hash *fph;
@@ -1572,7 +1579,7 @@ static int futex_hash_allocate(unsigned int hash_slots, bool custom)
 	 */
 	scoped_guard(rcu) {
 		fph = rcu_dereference(mm->futex_phash);
-		if (fph && !fph->hash_mask) {
+		if (fph && (!fph->hash_mask || fph->immutable)) {
 			if (custom)
 				return -EBUSY;
 			return 0;
@@ -1586,6 +1593,7 @@ static int futex_hash_allocate(unsigned int hash_slots, bool custom)
 	rcuref_init(&fph->users, 1);
 	fph->hash_mask = hash_slots ? hash_slots - 1 : 0;
 	fph->custom = custom;
+	fph->immutable = !!immutable;
 	fph->mm = mm;
 
 	for (i = 0; i < hash_slots; i++)
@@ -1678,7 +1686,7 @@ int futex_hash_allocate_default(void)
 	if (current_buckets >= buckets)
 		return 0;
 
-	return futex_hash_allocate(buckets, false);
+	return futex_hash_allocate(buckets, 0, false);
 }
 
 static int futex_hash_get_slots(void)
@@ -1692,9 +1700,22 @@ static int futex_hash_get_slots(void)
 	return 0;
 }
 
+static int futex_hash_get_immutable(void)
+{
+	struct futex_private_hash *fph;
+
+	guard(rcu)();
+	fph = rcu_dereference(current->mm->futex_phash);
+	if (fph && fph->immutable)
+		return 1;
+	if (fph && !fph->hash_mask)
+		return 1;
+	return 0;
+}
+
 #else
 
-static int futex_hash_allocate(unsigned int hash_slots, bool custom)
+static int futex_hash_allocate(unsigned int hash_slots, unsigned int immutable, bool custom)
 {
 	return -EINVAL;
 }
@@ -1703,21 +1724,30 @@ static int futex_hash_get_slots(void)
 {
 	return 0;
 }
+
+static int futex_hash_get_immutable(void)
+{
+	return 0;
+}
 #endif
 
-int futex_hash_prctl(unsigned long arg2, unsigned long arg3)
+int futex_hash_prctl(unsigned long arg2, unsigned long arg3, unsigned long arg4)
 {
 	int ret;
 
 	switch (arg2) {
 	case PR_FUTEX_HASH_SET_SLOTS:
-		ret = futex_hash_allocate(arg3, true);
+		ret = futex_hash_allocate(arg3, arg4, true);
 		break;
 
 	case PR_FUTEX_HASH_GET_SLOTS:
 		ret = futex_hash_get_slots();
 		break;
 
+	case PR_FUTEX_HASH_GET_IMMUTABLE:
+		ret = futex_hash_get_immutable();
+		break;
+
 	default:
 		ret = -EINVAL;
 		break;
diff --git a/kernel/sys.c b/kernel/sys.c
index d446d8ecb0b33..adc0de0aa364a 100644
--- a/kernel/sys.c
+++ b/kernel/sys.c
@@ -2822,7 +2822,7 @@ SYSCALL_DEFINE5(prctl, int, option, unsigned long, arg2, unsigned long, arg3,
 		error = posixtimer_create_prctl(arg2);
 		break;
 	case PR_FUTEX_HASH:
-		error = futex_hash_prctl(arg2, arg3);
+		error = futex_hash_prctl(arg2, arg3, arg4);
 		break;
 	default:
 		trace_task_prctl_unknown(option, arg2, arg3, arg4, arg5);
diff --git a/tools/include/uapi/linux/prctl.h b/tools/include/uapi/linux/prctl.h
index 3b93fb906e3c5..21f30b3ded74b 100644
--- a/tools/include/uapi/linux/prctl.h
+++ b/tools/include/uapi/linux/prctl.h
@@ -368,5 +368,6 @@ struct prctl_mm_map {
 #define PR_FUTEX_HASH			78
 # define PR_FUTEX_HASH_SET_SLOTS	1
 # define PR_FUTEX_HASH_GET_SLOTS	2
+# define PR_FUTEX_HASH_GET_IMMUTABLE	3
 
 #endif /* _LINUX_PRCTL_H */
diff --git a/tools/perf/bench/futex-hash.c b/tools/perf/bench/futex-hash.c
index c843bd8543c74..fdf133c9520f7 100644
--- a/tools/perf/bench/futex-hash.c
+++ b/tools/perf/bench/futex-hash.c
@@ -57,6 +57,7 @@ static struct bench_futex_parameters params = {
 
 static const struct option options[] = {
 	OPT_INTEGER( 'b', "buckets", &params.nbuckets, "Specify amount of hash buckets"),
+	OPT_BOOLEAN( 'I', "immutable", &params.buckets_immutable, "Make the hash buckets immutable"),
 	OPT_UINTEGER('t', "threads", &params.nthreads, "Specify amount of threads"),
 	OPT_UINTEGER('r', "runtime", &params.runtime, "Specify runtime (in seconds)"),
 	OPT_UINTEGER('f', "futexes", &params.nfutexes, "Specify amount of futexes per threads"),
diff --git a/tools/perf/bench/futex-lock-pi.c b/tools/perf/bench/futex-lock-pi.c
index 40640b6744279..5144a158512cc 100644
--- a/tools/perf/bench/futex-lock-pi.c
+++ b/tools/perf/bench/futex-lock-pi.c
@@ -47,6 +47,7 @@ static struct bench_futex_parameters params = {
 
 static const struct option options[] = {
 	OPT_INTEGER( 'b', "buckets", &params.nbuckets, "Specify amount of hash buckets"),
+	OPT_BOOLEAN( 'I', "immutable", &params.buckets_immutable, "Make the hash buckets immutable"),
 	OPT_UINTEGER('t', "threads", &params.nthreads, "Specify amount of threads"),
 	OPT_UINTEGER('r', "runtime", &params.runtime, "Specify runtime (in seconds)"),
 	OPT_BOOLEAN( 'M', "multi",   &params.multi, "Use multiple futexes"),
diff --git a/tools/perf/bench/futex-requeue.c b/tools/perf/bench/futex-requeue.c
index 0748b0fd689e8..a2f91ee1950b3 100644
--- a/tools/perf/bench/futex-requeue.c
+++ b/tools/perf/bench/futex-requeue.c
@@ -52,6 +52,7 @@ static struct bench_futex_parameters params = {
 
 static const struct option options[] = {
 	OPT_INTEGER( 'b', "buckets", &params.nbuckets, "Specify amount of hash buckets"),
+	OPT_BOOLEAN( 'I', "immutable", &params.buckets_immutable, "Make the hash buckets immutable"),
 	OPT_UINTEGER('t', "threads",  &params.nthreads, "Specify amount of threads"),
 	OPT_UINTEGER('q', "nrequeue", &params.nrequeue, "Specify amount of threads to requeue at once"),
 	OPT_BOOLEAN( 's', "silent",   &params.silent, "Silent mode: do not display data/details"),
diff --git a/tools/perf/bench/futex-wake-parallel.c b/tools/perf/bench/futex-wake-parallel.c
index 6aede7c46b337..ee66482c29fd1 100644
--- a/tools/perf/bench/futex-wake-parallel.c
+++ b/tools/perf/bench/futex-wake-parallel.c
@@ -63,6 +63,7 @@ static struct bench_futex_parameters params = {
 
 static const struct option options[] = {
 	OPT_INTEGER( 'b', "buckets", &params.nbuckets, "Specify amount of hash buckets"),
+	OPT_BOOLEAN( 'I', "immutable", &params.buckets_immutable, "Make the hash buckets immutable"),
 	OPT_UINTEGER('t', "threads", &params.nthreads, "Specify amount of threads"),
 	OPT_UINTEGER('w', "nwakers", &params.nwakes, "Specify amount of waking threads"),
 	OPT_BOOLEAN( 's', "silent",  &params.silent, "Silent mode: do not display data/details"),
diff --git a/tools/perf/bench/futex-wake.c b/tools/perf/bench/futex-wake.c
index a31fc1563862e..8d6107f7cd941 100644
--- a/tools/perf/bench/futex-wake.c
+++ b/tools/perf/bench/futex-wake.c
@@ -52,6 +52,7 @@ static struct bench_futex_parameters params = {
 
 static const struct option options[] = {
 	OPT_INTEGER( 'b', "buckets", &params.nbuckets, "Specify amount of hash buckets"),
+	OPT_BOOLEAN( 'I', "immutable", &params.buckets_immutable, "Make the hash buckets immutable"),
 	OPT_UINTEGER('t', "threads", &params.nthreads, "Specify amount of threads"),
 	OPT_UINTEGER('w', "nwakes",  &params.nwakes, "Specify amount of threads to wake at once"),
 	OPT_BOOLEAN( 's', "silent",  &params.silent, "Silent mode: do not display data/details"),
diff --git a/tools/perf/bench/futex.c b/tools/perf/bench/futex.c
index 8109d6bf3ede2..bed3b6e46d109 100644
--- a/tools/perf/bench/futex.c
+++ b/tools/perf/bench/futex.c
@@ -14,7 +14,7 @@ void futex_set_nbuckets_param(struct bench_futex_parameters *params)
 	if (params->nbuckets < 0)
 		return;
 
-	ret = prctl(PR_FUTEX_HASH, PR_FUTEX_HASH_SET_SLOTS, params->nbuckets);
+	ret = prctl(PR_FUTEX_HASH, PR_FUTEX_HASH_SET_SLOTS, params->nbuckets, params->buckets_immutable);
 	if (ret) {
 		printf("Requesting %d hash buckets failed: %d/%m\n",
 		       params->nbuckets, ret);
@@ -38,11 +38,13 @@ void futex_print_nbuckets(struct bench_futex_parameters *params)
 			printf("Requested: %d in usage: %d\n", params->nbuckets, ret);
 			err(EXIT_FAILURE, "prctl(PR_FUTEX_HASH)");
 		}
+		ret = prctl(PR_FUTEX_HASH, PR_FUTEX_HASH_GET_IMMUTABLE);
 		if (params->nbuckets == 0)
 			ret = asprintf(&futex_hash_mode, "Futex hashing: global hash");
 		else
-			ret = asprintf(&futex_hash_mode, "Futex hashing: %d hash buckets",
-				       params->nbuckets);
+			ret = asprintf(&futex_hash_mode, "Futex hashing: %d hash buckets %s",
+				       params->nbuckets,
+				       ret == 1 ? "(immutable)" : "");
 	} else {
 		if (ret <= 0) {
 			ret = asprintf(&futex_hash_mode, "Futex hashing: global hash");
diff --git a/tools/perf/bench/futex.h b/tools/perf/bench/futex.h
index dd295d27044ac..9c9a73f9d865e 100644
--- a/tools/perf/bench/futex.h
+++ b/tools/perf/bench/futex.h
@@ -26,6 +26,7 @@ struct bench_futex_parameters {
 	unsigned int nwakes;
 	unsigned int nrequeue;
 	int nbuckets;
+	bool buckets_immutable;
 };
 
 /**
-- 
2.49.0

Re: [PATCH v11 19/19] futex: Allow to make the private hash immutable.

[Sebastian Andrzej Siewior]

On 2025-04-07 17:57:42 [+0200], To linux-kernel@vger.kernel.org wrote:
> - Xeon CPU E5-2650, 2 NUMA nodes, total 32 CPUs:
>   - Before the introducing task local hash
>     shared  Averaged 1.487.148 operations/sec (+- 0,53%), total secs = 10
>     private Averaged 2.192.405 operations/sec (+- 0,07%), total secs = 10
> 
>   - With the series
>     shared  Averaged 1.326.342 operations/sec (+- 0,41%), total secs = 10
>     -b128   Averaged   141.394 operations/sec (+- 1,15%), total secs = 10
>     -Ib128  Averaged   851.490 operations/sec (+- 0,67%), total secs = 10
>     -b8192  Averaged   131.321 operations/sec (+- 2,13%), total secs = 10
>     -Ib8192 Averaged 1.923.077 operations/sec (+- 0,61%), total secs = 10
>     128 is the default allocation of hash buckets.
>     8192 was the previous amount of allocated hash buckets.
> 
> - Xeon(R) CPU E7-8890 v3, 4 NUMA nodes, total 144 CPUs:
>   - Before the introducing task local hash
>     shared   Averaged 1.810.936 operations/sec (+- 0,26%), total secs = 20
>     private  Averaged 2.505.801 operations/sec (+- 0,05%), total secs = 20
> 
>   - With the series
>     shared   Averaged 1.589.002 operations/sec (+- 0,25%), total secs = 20
>     -b1024   Averaged    42.410 operations/sec (+- 0,20%), total secs = 20
>     -Ib1024  Averaged   740.638 operations/sec (+- 1,51%), total secs = 20
>     -b65536  Averaged    48.811 operations/sec (+- 1,35%), total secs = 20
>     -Ib65536 Averaged 1.963.165 operations/sec (+- 0,18%), total secs = 20
>     1024 is the default allocation of hash buckets.
>     65536 was the previous amount of allocated hash buckets.

On EPYC 7713, 2 NUMA nodes, 256 CPUs:
				ops/sec		buckets
 -t 250                     25.393 (+- 1,51%)     1024
 -t 250 -b 1024 -I       1.645.327 (+- 0,34%)     1024
 -t 250 -b 65536            25.445 (+- 2,41%)    65536
 -t 250 -b 65536 -I      1.733.745 (+- 0,36%)    65536
 -t 250 -b 0             1.745.242 (+- 0,21%)    32768 * 2

Sebastian

Re: [PATCH v11 19/19] futex: Allow to make the private hash immutable.

[Shrikanth Hegde]

Hi Sebastian.

On 4/7/25 21:27, Sebastian Andrzej Siewior wrote:
> My initial testing showed that
> 	perf bench futex hash
> 
> reported less operations/sec with private hash. After using the same
> amount of buckets in the private hash as used by the global hash then
> the operations/sec were about the same.
> 
> This changed once the private hash became resizable. This feature added
> a RCU section and reference counting via atomic inc+dec operation into
> the hot path.
> The reference counting can be avoided if the private hash is made
> immutable.
> Extend PR_FUTEX_HASH_SET_SLOTS by a fourth argument which denotes if the
> private should be made immutable. Once set (to true) the a further
> resize is not allowed (same if set to global hash).
> Add PR_FUTEX_HASH_GET_IMMUTABLE which returns true if the hash can not
> be changed.
> Update "perf bench" suite.
> 

It would be good option for the application to decide if it needs this.
Using this option makes the perf regression goes away using previous number of buckets.

Acked-by: Shrikanth Hegde <sshegde@linux.ibm.com>

base:
./perf bench futex hash
Averaged 1556023 operations/sec (+- 0.08%), total secs = 10  <<-- 1.5M

with series:
./perf bench futex hash -b32768
Averaged 126499 operations/sec (+- 0.41%), total secs = 10   <<-- .12M

./perf bench futex hash -Ib32768
Averaged 1549339 operations/sec (+- 0.08%), total secs = 10  <<-- 1.5M


> For comparison, results of "perf bench futex hash -s":
> - Xeon CPU E5-2650, 2 NUMA nodes, total 32 CPUs:
>    - Before the introducing task local hash
>      shared  Averaged 1.487.148 operations/sec (+- 0,53%), total secs = 10
>      private Averaged 2.192.405 operations/sec (+- 0,07%), total secs = 10
> 
>    - With the series
>      shared  Averaged 1.326.342 operations/sec (+- 0,41%), total secs = 10
>      -b128   Averaged   141.394 operations/sec (+- 1,15%), total secs = 10
>      -Ib128  Averaged   851.490 operations/sec (+- 0,67%), total secs = 10
>      -b8192  Averaged   131.321 operations/sec (+- 2,13%), total secs = 10
>      -Ib8192 Averaged 1.923.077 operations/sec (+- 0,61%), total secs = 10
>      128 is the default allocation of hash buckets.
>      8192 was the previous amount of allocated hash buckets.
> 
> - Xeon(R) CPU E7-8890 v3, 4 NUMA nodes, total 144 CPUs:
>    - Before the introducing task local hash
>      shared   Averaged 1.810.936 operations/sec (+- 0,26%), total secs = 20
>      private  Averaged 2.505.801 operations/sec (+- 0,05%), total secs = 20
> 
>    - With the series
>      shared   Averaged 1.589.002 operations/sec (+- 0,25%), total secs = 20
>      -b1024   Averaged    42.410 operations/sec (+- 0,20%), total secs = 20
>      -Ib1024  Averaged   740.638 operations/sec (+- 1,51%), total secs = 20
>      -b65536  Averaged    48.811 operations/sec (+- 1,35%), total secs = 20
>      -Ib65536 Averaged 1.963.165 operations/sec (+- 0,18%), total secs = 20
>      1024 is the default allocation of hash buckets.
>      65536 was the previous amount of allocated hash buckets.
> 
> Cc: "Liang, Kan" <kan.liang@linux.intel.com>
> Cc: Adrian Hunter <adrian.hunter@intel.com>
> Cc: Alexander Shishkin <alexander.shishkin@linux.intel.com>
> Cc: Arnaldo Carvalho de Melo <acme@kernel.org>
> Cc: Ian Rogers <irogers@google.com>
> Cc: Ingo Molnar <mingo@redhat.com>
> Cc: Jiri Olsa <jolsa@kernel.org>
> Cc: Mark Rutland <mark.rutland@arm.com>
> Cc: Namhyung Kim <namhyung@kernel.org>
> Cc: linux-perf-users@vger.kernel.org
> Signed-off-by: Sebastian Andrzej Siewior <bigeasy@linutronix.de>
> ---
>   include/linux/futex.h                  |  2 +-
>   include/uapi/linux/prctl.h             |  1 +
>   kernel/futex/core.c                    | 42 ++++++++++++++++++++++----
>   kernel/sys.c                           |  2 +-
>   tools/include/uapi/linux/prctl.h       |  1 +
>   tools/perf/bench/futex-hash.c          |  1 +
>   tools/perf/bench/futex-lock-pi.c       |  1 +
>   tools/perf/bench/futex-requeue.c       |  1 +
>   tools/perf/bench/futex-wake-parallel.c |  1 +
>   tools/perf/bench/futex-wake.c          |  1 +
>   tools/perf/bench/futex.c               |  8 +++--
>   tools/perf/bench/futex.h               |  1 +
>   12 files changed, 51 insertions(+), 11 deletions(-)
> 

nit: Does it makes sense to split this patch into futex and perf?

Re: [PATCH v11 19/19] futex: Allow to make the private hash immutable.

[Sebastian Andrzej Siewior]

On 2025-04-10 20:22:08 [+0530], Shrikanth Hegde wrote:
> Hi Sebastian.
Hi Shrikanth,

> It would be good option for the application to decide if it needs this.

You mean to have it as I introduced it here or something else?

> Using this option makes the perf regression goes away using previous number of buckets.

Okay, good to know. You test this on on ppc64le?

> Acked-by: Shrikanth Hegde <sshegde@linux.ibm.com>
> 
> base:
> ./perf bench futex hash
> Averaged 1556023 operations/sec (+- 0.08%), total secs = 10  <<-- 1.5M
> 
> with series:
> ./perf bench futex hash -b32768
> Averaged 126499 operations/sec (+- 0.41%), total secs = 10   <<-- .12M
> 
> ./perf bench futex hash -Ib32768
> Averaged 1549339 operations/sec (+- 0.08%), total secs = 10  <<-- 1.5M
Thank you for testing.
…
> nit: Does it makes sense to split this patch into futex and perf?

First I wanted to figure if we really do this. I have no idea if this
regression would show up in real world use case or just here as part of
the micro benchmark.
And if we do this, it would probably make sense to have one perf patch
which introduces -b & -I.  And then figure out if the additional option
to prctl should be part of the resize patch or not. Probably we should
enforce 0/1 of arg4 from the beginning so maybe folding this in makes
sense.

Sebastian

Re: [PATCH v11 19/19] futex: Allow to make the private hash immutable.

[Shrikanth Hegde]

On 4/10/25 20:58, Sebastian Andrzej Siewior wrote:
> On 2025-04-10 20:22:08 [+0530], Shrikanth Hegde wrote:
>> Hi Sebastian.
> Hi Shrikanth,
> 
>> It would be good option for the application to decide if it needs this.
> 
> You mean to have it as I introduced it here or something else?
> 

as you have introduced it here.

>> Using this option makes the perf regression goes away using previous number of buckets.
> 
> Okay, good to know. You test this on on ppc64le?

Yes.

> 
>> Acked-by: Shrikanth Hegde <sshegde@linux.ibm.com>
>>
>> base:
>> ./perf bench futex hash
>> Averaged 1556023 operations/sec (+- 0.08%), total secs = 10  <<-- 1.5M
>>
>> with series:
>> ./perf bench futex hash -b32768
>> Averaged 126499 operations/sec (+- 0.41%), total secs = 10   <<-- .12M
>>
>> ./perf bench futex hash -Ib32768
>> Averaged 1549339 operations/sec (+- 0.08%), total secs = 10  <<-- 1.5M
> Thank you for testing.
> …
>> nit: Does it makes sense to split this patch into futex and perf?
> 
> First I wanted to figure if we really do this. I have no idea if this
> regression would show up in real world use case or just here as part of
> the micro benchmark.
> And if we do this, it would probably make sense to have one perf patch
> which introduces -b & -I.  And then figure out if the additional option
> to prctl should be part of the resize patch or not. Probably we should
> enforce 0/1 of arg4 from the beginning so maybe folding this in makes
> sense.
>

ok.

> Sebastian