From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 1C6711F874C for ; Sun, 14 Dec 2025 23:35:59 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1765755360; cv=none; b=eUy0rAwpuBU0bMcy/i4TI7n8SfX+1orjxioew9wyt6EmE1xrTObelHaeezVl3GWp72AIuVcuLvfSEqT2F+R9xtlmSxEbG7hUo3DgxXSy3PFA5v8BSmEGjjfoiD19eaLv1mZL01Ejl4oSQ+qVGiTYvAj8uNcFg9/WUJj0aJNBzac= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1765755360; c=relaxed/simple; bh=CNnmSV59/1IAJD9yUZe4wSgby2b/jUREY5zIPUH0FbE=; h=Date:To:From:Subject:Message-Id; b=aqE5vWA/zI3jfamY16/EYpg7eajWpDZ5ETARkyq9qNV8HxasiRxRkIpW1WINGHCH4YvaW4k9/TQjTN93njZ5+VJ5lJOkxgshUzXeXAW0Daa8EpBlmiJVlVEvHS63qxEP8Hsp57oVWhLgGRTZOVTiykOzoeC5/iTv6BBV1ve0W1E= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=linux-foundation.org header.i=@linux-foundation.org header.b=Y0BxJW/l; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=linux-foundation.org header.i=@linux-foundation.org header.b="Y0BxJW/l" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 6CDB4C4CEF1; Sun, 14 Dec 2025 23:35:59 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=linux-foundation.org; s=korg; t=1765755359; bh=CNnmSV59/1IAJD9yUZe4wSgby2b/jUREY5zIPUH0FbE=; h=Date:To:From:Subject:From; b=Y0BxJW/lpTZIn3to4tesBhMNqv5074P0E7M2nvNW6SkMOCce3iy4g34yD8+tr92aE QI6XsEnsP05/VNyoDhx1/EbnL/AcE3c8q2Yf/1r/JLdd7g/2MH/SXyhcbuwhRouGe0 1dssWqCeYLM4+KnHb+4saxvHv+w2MbGirMuRQW38= Date: Sun, 14 Dec 2025 15:35:58 -0800 To: mm-commits@vger.kernel.org,yuzhao@google.com,willy@infradead.org,viro@zeniv.linux.org.uk,vbabka@suse.cz,tj@kernel.org,sweettea-kernel@dorminy.me,surenb@google.com,sj@kernel.org,shakeel.butt@linux.dev,rppt@kernel.org,rostedt@goodmis.org,roman.gushchin@linux.dev,rientjes@google.com,richard.weiyang@gmail.com,paulmck@kernel.org,mjguzik@gmail.com,mhocko@suse.com,mhiramat@kernel.org,lorenzo.stoakes@oracle.com,liumartin@google.com,linmiaohe@huawei.com,liam.howlett@oracle.com,hannes@cmpxchg.org,dennis@kernel.org,david@redhat.com,cl@linux.com,christian.koenig@amd.com,brauner@kernel.org,baolin.wang@linux.alibaba.com,aboorvad@linux.ibm.com,mathieu.desnoyers@efficios.com,akpm@linux-foundation.org From: Andrew Morton Subject: + lib-introduce-hierarchical-per-cpu-counters.patch added to mm-new branch Message-Id: <20251214233559.6CDB4C4CEF1@smtp.kernel.org> Precedence: bulk X-Mailing-List: mm-commits@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: The patch titled Subject: lib: introduce hierarchical per-cpu counters has been added to the -mm mm-new branch. Its filename is lib-introduce-hierarchical-per-cpu-counters.patch This patch will shortly appear at https://git.kernel.org/pub/scm/linux/kernel/git/akpm/25-new.git/tree/patches/lib-introduce-hierarchical-per-cpu-counters.patch This patch will later appear in the mm-new branch at git://git.kernel.org/pub/scm/linux/kernel/git/akpm/mm Note, mm-new is a provisional staging ground for work-in-progress patches, and acceptance into mm-new is a notification for others take notice and to finish up reviews. Please do not hesitate to respond to review feedback and post updated versions to replace or incrementally fixup patches in mm-new. Before you just go and hit "reply", please: a) Consider who else should be cc'ed b) Prefer to cc a suitable mailing list as well c) Ideally: find the original patch on the mailing list and do a reply-to-all to that, adding suitable additional cc's *** Remember to use Documentation/process/submit-checklist.rst when testing your code *** The -mm tree is included into linux-next via the mm-everything branch at git://git.kernel.org/pub/scm/linux/kernel/git/akpm/mm and is updated there every 2-3 working days ------------------------------------------------------ From: Mathieu Desnoyers Subject: lib: introduce hierarchical per-cpu counters Date: Sat, 13 Dec 2025 13:56:06 -0500 Patch series "mm: Fix OOM killer inaccuracy on large many-core systems", v10. Introduce hierarchical per-cpu counters and use them for RSS tracking to fix the per-mm RSS tracking which has become too inaccurate for OOM killer purposes on large many-core systems. The following rss tracking issues were noted by Sweet Tea Dorminy [1], which lead to picking wrong tasks as OOM kill target: Recently, several internal services had an RSS usage regression as part of a kernel upgrade. Previously, they were on a pre-6.2 kernel and were able to read RSS statistics in a backup watchdog process to monitor and decide if they'd overrun their memory budget. Now, however, a representative service with five threads, expected to use about a hundred MB of memory, on a 250-cpu machine had memory usage tens of megabytes different from the expected amount -- this constituted a significant percentage of inaccuracy, causing the watchdog to act. This was a result of commit f1a7941243c1 ("mm: convert mm's rss stats into percpu_counter") [1]. Previously, the memory error was bounded by 64*nr_threads pages, a very livable megabyte. Now, however, as a result of scheduler decisions moving the threads around the CPUs, the memory error could be as large as a gigabyte. This is a really tremendous inaccuracy for any few-threaded program on a large machine and impedes monitoring significantly. These stat counters are also used to make OOM killing decisions, so this additional inaccuracy could make a big difference in OOM situations -- either resulting in the wrong process being killed, or in less memory being returned from an OOM-kill than expected. The approach proposed here is to replace this by the hierarchical per-cpu counters, which bounds the inaccuracy based on the system topology with O(N*logN). Notable change for v10: The new patch 3/3 changes the implementation of the oom killer task selection to a 2-pass algorithm, where the first pass uses the fast approximation provided by the hierarchical percpu counters, and the second pass does a precise sum for all tasks which have badness values within the range of the approximation accuracy. I've done moderate testing of this series on a 256-core VM with 128GB RAM. Figuring out whether this indeed helps solve issues with real-life workloads will require broader feedback from the community. The one request I did not have time to fulfill yet is to port the tests from the librseq feature branch implementation (userspace) to the kernel selftests. This patch (of 3): * Motivation The purpose of this hierarchical split-counter scheme is to: - Minimize contention when incrementing and decrementing counters, - Provide fast access to a sum approximation, - Provide a sum approximation with an acceptable accuracy level when scaling to many-core systems. - Provide approximate and precise comparison of two counters, and between a counter and a value. It aims at fixing the per-mm RSS tracking which has become too inaccurate for OOM killer purposes on large many-core systems [1]. * Design The hierarchical per-CPU counters propagate a sum approximation through a N-way tree. When reaching the batch size, the carry is propagated through a binary tree which consists of logN(nr_cpu_ids) levels. The batch size for each level is twice the batch size of the prior level. Example propagation diagram with 8 cpus through a binary tree: Level 0: 0 1 2 3 4 5 6 7 | / | / | / | / | / | / | / | / | / | / | / | / Level 1: 0 1 2 3 | / | / | / | / | / | / Level 2: 0 1 | / | / | / Level 3: 0 For a binary tree, the maximum inaccuracy is bound by: batch_size * log2(nr_cpus) * nr_cpus which evolves with O(n*log(n)) as the number of CPUs increases. For a N-way tree, the maximum inaccuracy can be pre-calculated based on the the N-arity of each level and the batch size. Link: https://lkml.kernel.org/r/20251213185608.3418096-1-mathieu.desnoyers@efficios.com Link: https://lore.kernel.org/lkml/20250331223516.7810-2-sweettea-kernel@dorminy.me/ # [1] Link: https://lkml.kernel.org/r/20251213185608.3418096-2-mathieu.desnoyers@efficios.com Signed-off-by: Mathieu Desnoyers Cc: "Paul E. McKenney" Cc: Steven Rostedt Cc: Masami Hiramatsu Cc: Mathieu Desnoyers Cc: Dennis Zhou Cc: Tejun Heo Cc: Christoph Lameter Cc: Martin Liu Cc: David Rientjes Cc: Christan König Cc: Shakeel Butt Cc: SeongJae Park Cc: Michal Hocko Cc: Johannes Weiner Cc: Sweet Tea Dorminy Cc: Lorenzo Stoakes Cc: "Liam R . Howlett" Cc: Mike Rapoport Cc: Suren Baghdasaryan Cc: Vlastimil Babka Cc: Christian Brauner Cc: Wei Yang Cc: David Hildenbrand Cc: Miaohe Lin Cc: Al Viro Cc: Yu Zhao Cc: Roman Gushchin Cc: Mateusz Guzik Cc: Matthew Wilcox Cc: Baolin Wang Cc: Aboorva Devarajan Signed-off-by: Andrew Morton --- include/linux/percpu_counter_tree.h | 242 ++++++++ init/main.c | 2 lib/Makefile | 1 lib/percpu_counter_tree.c | 705 ++++++++++++++++++++++++++ 4 files changed, 950 insertions(+) diff --git a/include/linux/percpu_counter_tree.h a/include/linux/percpu_counter_tree.h new file mode 100644 --- /dev/null +++ a/include/linux/percpu_counter_tree.h @@ -0,0 +1,242 @@ +/* SPDX-License-Identifier: GPL-2.0+ OR MIT */ +/* SPDX-FileCopyrightText: 2025 Mathieu Desnoyers */ + +#ifndef _PERCPU_COUNTER_TREE_H +#define _PERCPU_COUNTER_TREE_H + +#include +#include +#include + +#ifdef CONFIG_SMP + +struct percpu_counter_tree_level_item { + atomic_t count; /* + * Count the number of carry fort this tree item. + * The carry counter is kept at the order of the + * carry accounted for at this tree level. + */ +} ____cacheline_aligned_in_smp; + +struct percpu_counter_tree { + /* Fast-path fields. */ + unsigned int __percpu *level0; /* Pointer to per-CPU split counters (tree level 0). */ + unsigned int level0_bit_mask; /* Bit mask to apply to detect carry propagation from tree level 0. */ + union { + unsigned int *i; /* Approximate sum for single-CPU topology. */ + atomic_t *a; /* Approximate sum for SMP topology. */ + } approx_sum; + int bias; /* Bias to apply to counter precise and approximate values. */ + + /* Slow-path fields. */ + struct percpu_counter_tree_level_item *items; /* Array of tree items for levels 1 to N. */ + unsigned int batch_size; /* + * The batch size is the increment step at level 0 which + * triggers a carry propagation. The batch size is required + * to be greater than 1, and a power of 2. + */ + /* + * The tree approximate sum is guaranteed to be within this accuracy range: + * (precise_sum - approx_accuracy_range.under) <= approx_sum <= (precise_sum + approx_accuracy_range.over). + * This accuracy is derived from the hardware topology and the tree batch_size. + * The "under" accuracy is larger than the "over" accuracy because the negative range of a + * two's complement signed integer is one unit larger than the positive range. This delta + * is summed for each tree item, which leads to a significantly larger "under" accuracy range + * compared to the "over" accuracy range. + */ + struct { + unsigned int under; + unsigned int over; + } approx_accuracy_range; +}; + +size_t percpu_counter_tree_items_size(void); +int percpu_counter_tree_init_many(struct percpu_counter_tree *counters, struct percpu_counter_tree_level_item *items, + unsigned int nr_counters, unsigned int batch_size, gfp_t gfp_flags); +int percpu_counter_tree_init(struct percpu_counter_tree *counter, struct percpu_counter_tree_level_item *items, + unsigned int batch_size, gfp_t gfp_flags); +void percpu_counter_tree_destroy_many(struct percpu_counter_tree *counter, unsigned int nr_counters); +void percpu_counter_tree_destroy(struct percpu_counter_tree *counter); +void percpu_counter_tree_add(struct percpu_counter_tree *counter, int inc); +int percpu_counter_tree_precise_sum(struct percpu_counter_tree *counter); +int percpu_counter_tree_approximate_compare(struct percpu_counter_tree *a, struct percpu_counter_tree *b); +int percpu_counter_tree_approximate_compare_value(struct percpu_counter_tree *counter, int v); +int percpu_counter_tree_precise_compare(struct percpu_counter_tree *a, struct percpu_counter_tree *b); +int percpu_counter_tree_precise_compare_value(struct percpu_counter_tree *counter, int v); +void percpu_counter_tree_set(struct percpu_counter_tree *counter, int v); +void percpu_counter_tree_approximate_accuracy_range(struct percpu_counter_tree *counter, + unsigned int *under, unsigned int *over); +int percpu_counter_tree_subsystem_init(void); + +/** + * percpu_counter_tree_approximate_sum() - Return approximate counter sum. + * @counter: The counter to sum. + * + * Querying the approximate sum is fast, but it is only accurate within + * the bounds delimited by percpu_counter_tree_approximate_accuracy_range(). + * This is meant to be used when speed is preferred over accuracy. + * + * Return: The current approximate counter sum. + */ +static inline +int percpu_counter_tree_approximate_sum(struct percpu_counter_tree *counter) +{ + unsigned int v; + + if (!counter->level0_bit_mask) + v = READ_ONCE(*counter->approx_sum.i); + else + v = atomic_read(counter->approx_sum.a); + return (int) (v + (unsigned int)READ_ONCE(counter->bias)); +} + +#else /* !CONFIG_SMP */ + +struct percpu_counter_tree_level_item; + +struct percpu_counter_tree { + atomic_t count; +}; + +static inline +size_t percpu_counter_tree_items_size(void) +{ + return 0; +} + +static inline +int percpu_counter_tree_init_many(struct percpu_counter_tree *counters, struct percpu_counter_tree_level_item *items, + unsigned int nr_counters, unsigned int batch_size, gfp_t gfp_flags) +{ + for (unsigned int i = 0; i < nr_counters; i++) + atomic_set(&counters[i].count, 0); + return 0; +} + +static inline +int percpu_counter_tree_init(struct percpu_counter_tree *counter, struct percpu_counter_tree_level_item *items, + unsigned int batch_size, gfp_t gfp_flags) +{ + return percpu_counter_tree_init_many(counter, items, 1, batch_size, gfp_flags); +} + +static inline +void percpu_counter_tree_destroy_many(struct percpu_counter_tree *counter, unsigned int nr_counters) +{ +} + +static inline +void percpu_counter_tree_destroy(struct percpu_counter_tree *counter) +{ +} + +static inline +int percpu_counter_tree_precise_sum(struct percpu_counter_tree *counter) +{ + return atomic_read(&counter->count); +} + +static inline +int percpu_counter_tree_precise_compare(struct percpu_counter_tree *a, struct percpu_counter_tree *b) +{ + int count_a = percpu_counter_tree_precise_sum(a), + count_b = percpu_counter_tree_precise_sum(b); + + if (count_a == count_b) + return 0; + if (count_a < count_b) + return -1; + return 1; +} + +static inline +int percpu_counter_tree_precise_compare_value(struct percpu_counter_tree *counter, int v) +{ + int count = percpu_counter_tree_precise_sum(counter); + + if (count == v) + return 0; + if (count < v) + return -1; + return 1; +} + +static inline +int percpu_counter_tree_approximate_compare(struct percpu_counter_tree *a, struct percpu_counter_tree *b) +{ + return percpu_counter_tree_precise_compare(a, b); +} + +static inline +int percpu_counter_tree_approximate_compare_value(struct percpu_counter_tree *counter, int v) +{ + return percpu_counter_tree_precise_compare_value(counter, v); +} + +static inline +void percpu_counter_tree_set(struct percpu_counter_tree *counter, int v) +{ + atomic_set(&counter->count, v); +} + +static inline +void percpu_counter_tree_approximate_accuracy_range(struct percpu_counter_tree *counter, + unsigned int *under, unsigned int *over) +{ + *under = 0; + *over = 0; +} + +static inline +void percpu_counter_tree_add(struct percpu_counter_tree *counter, int inc) +{ + atomic_add(inc, &counter->count); +} + +static inline +int percpu_counter_tree_approximate_sum(struct percpu_counter_tree *counter) +{ + return percpu_counter_tree_precise_sum(counter); +} + +static inline +int percpu_counter_tree_subsystem_init(void) +{ + return 0; +} + +#endif /* CONFIG_SMP */ + +/** + * percpu_counter_tree_approximate_sum_positive() - Return a positive approximate counter sum. + * @counter: The counter to sum. + * + * Return an approximate counter sum which is guaranteed to be greater + * or equal to 0. + * + * Return: The current positive approximate counter sum. + */ +static inline +int percpu_counter_tree_approximate_sum_positive(struct percpu_counter_tree *counter) +{ + int v = percpu_counter_tree_approximate_sum(counter); + return v > 0 ? v : 0; +} + +/** + * percpu_counter_tree_precise_sum_positive() - Return a positive precise counter sum. + * @counter: The counter to sum. + * + * Return a precise counter sum which is guaranteed to be greater + * or equal to 0. + * + * Return: The current positive precise counter sum. + */ +static inline +int percpu_counter_tree_precise_sum_positive(struct percpu_counter_tree *counter) +{ + int v = percpu_counter_tree_precise_sum(counter); + return v > 0 ? v : 0; +} + +#endif /* _PERCPU_COUNTER_TREE_H */ --- a/init/main.c~lib-introduce-hierarchical-per-cpu-counters +++ a/init/main.c @@ -104,6 +104,7 @@ #include #include #include +#include #include #include @@ -1063,6 +1064,7 @@ void start_kernel(void) vfs_caches_init_early(); sort_main_extable(); trap_init(); + percpu_counter_tree_subsystem_init(); mm_core_init(); maple_tree_init(); poking_init(); --- a/lib/Makefile~lib-introduce-hierarchical-per-cpu-counters +++ a/lib/Makefile @@ -181,6 +181,7 @@ obj-$(CONFIG_TEXTSEARCH_KMP) += ts_kmp.o obj-$(CONFIG_TEXTSEARCH_BM) += ts_bm.o obj-$(CONFIG_TEXTSEARCH_FSM) += ts_fsm.o obj-$(CONFIG_SMP) += percpu_counter.o +obj-$(CONFIG_SMP) += percpu_counter_tree.o obj-$(CONFIG_AUDIT_GENERIC) += audit.o obj-$(CONFIG_AUDIT_COMPAT_GENERIC) += compat_audit.o diff --git a/lib/percpu_counter_tree.c a/lib/percpu_counter_tree.c new file mode 100644 --- /dev/null +++ a/lib/percpu_counter_tree.c @@ -0,0 +1,705 @@ +// SPDX-License-Identifier: GPL-2.0+ OR MIT +// SPDX-FileCopyrightText: 2025 Mathieu Desnoyers + +/* + * Split Counters With Tree Approximation Propagation + * + * * Propagation diagram when reaching batch size thresholds (± batch size): + * + * Example diagram for 8 CPUs: + * + * log2(8) = 3 levels + * + * At each level, each pair propagates its values to the next level when + * reaching the batch size thresholds. + * + * Counters at levels 0, 1, 2 can be kept on a single byte ([-128 .. +127] range), + * although it may be relevant to keep them on 32-bit counters for + * simplicity. (complexity vs memory footprint tradeoff) + * + * Counter at level 3 can be kept on a 32-bit counter. + * + * Level 0: 0 1 2 3 4 5 6 7 + * | / | / | / | / + * | / | / | / | / + * | / | / | / | / + * Level 1: 0 1 2 3 + * | / | / + * | / | / + * | / | / + * Level 2: 0 1 + * | / + * | / + * | / + * Level 3: 0 + * + * * Approximation accuracy: + * + * BATCH(level N): Level N batch size. + * + * Example for BATCH(level 0) = 32. + * + * BATCH(level 0) = 32 + * BATCH(level 1) = 64 + * BATCH(level 2) = 128 + * BATCH(level N) = BATCH(level 0) * 2^N + * + * per-counter global + * accuracy accuracy + * Level 0: [ -32 .. +31] ±256 (8 * 32) + * Level 1: [ -64 .. +63] ±256 (4 * 64) + * Level 2: [-128 .. +127] ±256 (2 * 128) + * Total: ------ ±768 (log2(nr_cpu_ids) * BATCH(level 0) * nr_cpu_ids) + * + * Note that the global accuracy can be calculated more precisely + * by taking into account that the positive accuracy range is + * 31 rather than 32. + * + * ----- + * + * Approximate Sum Carry Propagation + * + * Let's define a number of counter bits for each level, e.g.: + * + * log2(BATCH(level 0)) = log2(32) = 5 + * + * nr_bit value_mask range + * Level 0: 5 bits v 0 .. +31 + * Level 1: 1 bit (v & ~((1UL << 5) - 1)) 0 .. +63 + * Level 2: 1 bit (v & ~((1UL << 6) - 1)) 0 .. +127 + * Level 3: 25 bits (v & ~((1UL << 7) - 1)) 0 .. 2^32-1 + * + * Note: Use a full 32-bit per-cpu counter at level 0 to allow precise sum. + * + * Note: Use cacheline aligned counters at levels above 0 to prevent false sharing. + * If memory footprint is an issue, a specialized allocator could be used + * to eliminate padding. + * + * Example with expanded values: + * + * counter_add(counter, inc): + * + * if (!inc) + * return; + * + * res = percpu_add_return(counter @ Level 0, inc); + * orig = res - inc; + * if (inc < 0) { + * inc = -(-inc & ~0b00011111); // Clear used bits + * // xor bit 5: underflow + * if ((inc ^ orig ^ res) & 0b00100000) + * inc -= 0b00100000; + * } else { + * inc &= ~0b00011111; // Clear used bits + * // xor bit 5: overflow + * if ((inc ^ orig ^ res) & 0b00100000) + * inc += 0b00100000; + * } + * if (!inc) + * return; + * + * res = atomic_add_return(counter @ Level 1, inc); + * orig = res - inc; + * if (inc < 0) { + * inc = -(-inc & ~0b00111111); // Clear used bits + * // xor bit 6: underflow + * if ((inc ^ orig ^ res) & 0b01000000) + * inc -= 0b01000000; + * } else { + * inc &= ~0b00111111; // Clear used bits + * // xor bit 6: overflow + * if ((inc ^ orig ^ res) & 0b01000000) + * inc += 0b01000000; + * } + * if (!inc) + * return; + * + * res = atomic_add_return(counter @ Level 2, inc); + * orig = res - inc; + * if (inc < 0) { + * inc = -(-inc & ~0b01111111); // Clear used bits + * // xor bit 7: underflow + * if ((inc ^ orig ^ res) & 0b10000000) + * inc -= 0b10000000; + * } else { + * inc &= ~0b01111111; // Clear used bits + * // xor bit 7: overflow + * if ((inc ^ orig ^ res) & 0b10000000) + * inc += 0b10000000; + * } + * if (!inc) + * return; + * + * atomic_add(counter @ Level 3, inc); + */ + +#include +#include +#include +#include +#include +#include +#include + +#define MAX_NR_LEVELS 5 + +/* + * The counter configuration is selected at boot time based on the + * hardware topology. + */ +struct counter_config { + unsigned int nr_items; /* + * nr_items is the number of items in the tree for levels 1 + * up to and including the final level (approximate sum). + * It excludes the level 0 per-CPU counters. + */ + unsigned char nr_levels; /* + * nr_levels is the number of hierarchical counter tree levels. + * It excludes the final level (approximate sum). + */ + unsigned char n_arity_order[MAX_NR_LEVELS]; /* + * n-arity of tree nodes for each level from + * 0 to (nr_levels - 1). + */ +}; + +static const struct counter_config per_nr_cpu_order_config[] = { + [0] = { .nr_items = 0, .nr_levels = 0, .n_arity_order = { 0 } }, + [1] = { .nr_items = 1, .nr_levels = 1, .n_arity_order = { 1 } }, + [2] = { .nr_items = 3, .nr_levels = 2, .n_arity_order = { 1, 1 } }, + [3] = { .nr_items = 7, .nr_levels = 3, .n_arity_order = { 1, 1, 1 } }, + [4] = { .nr_items = 7, .nr_levels = 3, .n_arity_order = { 2, 1, 1 } }, + [5] = { .nr_items = 11, .nr_levels = 3, .n_arity_order = { 2, 2, 1 } }, + [6] = { .nr_items = 21, .nr_levels = 3, .n_arity_order = { 2, 2, 2 } }, + [7] = { .nr_items = 21, .nr_levels = 3, .n_arity_order = { 3, 2, 2 } }, + [8] = { .nr_items = 37, .nr_levels = 3, .n_arity_order = { 3, 3, 2 } }, + [9] = { .nr_items = 73, .nr_levels = 3, .n_arity_order = { 3, 3, 3 } }, + [10] = { .nr_items = 149, .nr_levels = 4, .n_arity_order = { 3, 3, 2, 2 } }, + [11] = { .nr_items = 293, .nr_levels = 4, .n_arity_order = { 3, 3, 3, 2 } }, + [12] = { .nr_items = 585, .nr_levels = 4, .n_arity_order = { 3, 3, 3, 3 } }, + [13] = { .nr_items = 1173, .nr_levels = 5, .n_arity_order = { 3, 3, 3, 2, 2 } }, + [14] = { .nr_items = 2341, .nr_levels = 5, .n_arity_order = { 3, 3, 3, 3, 2 } }, + [15] = { .nr_items = 4681, .nr_levels = 5, .n_arity_order = { 3, 3, 3, 3, 3 } }, + [16] = { .nr_items = 4681, .nr_levels = 5, .n_arity_order = { 4, 3, 3, 3, 3 } }, + [17] = { .nr_items = 8777, .nr_levels = 5, .n_arity_order = { 4, 4, 3, 3, 3 } }, + [18] = { .nr_items = 17481, .nr_levels = 5, .n_arity_order = { 4, 4, 4, 3, 3 } }, + [19] = { .nr_items = 34953, .nr_levels = 5, .n_arity_order = { 4, 4, 4, 4, 3 } }, + [20] = { .nr_items = 69905, .nr_levels = 5, .n_arity_order = { 4, 4, 4, 4, 4 } }, +}; + +static const struct counter_config *counter_config; /* Hierarchical counter configuration for the hardware topology. */ +static unsigned int nr_cpus_order, /* Order of nr_cpu_ids. */ + accuracy_multiplier; /* Calculate accuracy for a given batch size (multiplication factor). */ + +static +int __percpu_counter_tree_init(struct percpu_counter_tree *counter, + unsigned int batch_size, gfp_t gfp_flags, + unsigned int __percpu *level0, + struct percpu_counter_tree_level_item *items) +{ + /* Batch size must be greater than 1, and a power of 2. */ + if (WARN_ON(batch_size <= 1 || (batch_size & (batch_size - 1)))) + return -EINVAL; + counter->batch_size = batch_size; + counter->bias = 0; + counter->level0 = level0; + counter->items = items; + if (!nr_cpus_order) { + counter->approx_sum.i = per_cpu_ptr(counter->level0, 0); + counter->level0_bit_mask = 0; + } else { + counter->approx_sum.a = &counter->items[counter_config->nr_items - 1].count; + counter->level0_bit_mask = 1UL << get_count_order(batch_size); + } + /* + * Each tree item signed integer has a negative range which is + * one unit greater than the positive range. + */ + counter->approx_accuracy_range.under = batch_size * accuracy_multiplier; + counter->approx_accuracy_range.over = (batch_size - 1) * accuracy_multiplier; + return 0; +} + +/** + * percpu_counter_tree_init_many() - Initialize many per-CPU counter trees. + * @counters: An array of @nr_counters counters to initialize. + * Their memory is provided by the caller. + * @items: Pointer to memory area where to store tree items. + * This memory is provided by the caller. + * Its size needs to be at least @nr_counters * percpu_counter_tree_items_size(). + * @nr_counters: The number of counter trees to initialize + * @batch_size: The batch size is the increment step at level 0 which triggers a + * carry propagation. + * The batch size is required to be greater than 1, and a power of 2. + * @gfp_flags: gfp flags to pass to the per-CPU allocator. + * + * Initialize many per-CPU counter trees using a single per-CPU + * allocator invocation for @nr_counters counters. + * + * Return: + * * %0: Success + * * %-EINVAL: - Invalid @batch_size argument + * * %-ENOMEM: - Out of memory + */ +int percpu_counter_tree_init_many(struct percpu_counter_tree *counters, struct percpu_counter_tree_level_item *items, + unsigned int nr_counters, unsigned int batch_size, gfp_t gfp_flags) +{ + void __percpu *level0, *level0_iter; + size_t counter_size, items_size = 0; + void *items_iter; + unsigned int i; + int ret; + + counter_size = ALIGN(sizeof(*counters), __alignof__(*counters)); + level0 = __alloc_percpu_gfp(nr_counters * counter_size, + __alignof__(*counters), gfp_flags); + if (!level0) + return -ENOMEM; + if (nr_cpus_order) { + items_size = percpu_counter_tree_items_size(); + memset(items, 0, items_size * nr_counters); + } + level0_iter = level0; + items_iter = items; + for (i = 0; i < nr_counters; i++) { + ret = __percpu_counter_tree_init(&counters[i], batch_size, gfp_flags, level0_iter, items_iter); + if (ret) + goto free_level0; + level0_iter += counter_size; + if (nr_cpus_order) + items_iter += items_size; + } + return 0; + +free_level0: + free_percpu(level0); + return ret; +} + +/** + * percpu_counter_tree_init() - Initialize one per-CPU counter tree. + * @counter: Counter to initialize. + * Its memory is provided by the caller. + * @items: Pointer to memory area where to store tree items. + * This memory is provided by the caller. + * Its size needs to be at least percpu_counter_tree_items_size(). + * @batch_size: The batch size is the increment step at level 0 which triggers a + * carry propagation. + * The batch size is required to be greater than 1, and a power of 2. + * @gfp_flags: gfp flags to pass to the per-CPU allocator. + * + * Initialize one per-CPU counter tree. + * + * Return: + * * %0: Success + * * %-EINVAL: - Invalid @batch_size argument + * * %-ENOMEM: - Out of memory + */ +int percpu_counter_tree_init(struct percpu_counter_tree *counter, struct percpu_counter_tree_level_item *items, + unsigned int batch_size, gfp_t gfp_flags) +{ + return percpu_counter_tree_init_many(counter, items, 1, batch_size, gfp_flags); +} + +/** + * percpu_counter_tree_destroy_many() - Destroy many per-CPU counter trees. + * @counters: Array of counters trees to destroy. + * @nr_counters: The number of counter trees to destroy. + * + * Release internal resources allocated for @nr_counters per-CPU counter trees. + */ + +void percpu_counter_tree_destroy_many(struct percpu_counter_tree *counters, unsigned int nr_counters) +{ + free_percpu(counters->level0); +} + +/** + * percpu_counter_tree_destroy() - Destroy one per-CPU counter tree. + * @counter: Counter to destroy. + * + * Release internal resources allocated for one per-CPU counter tree. + */ +void percpu_counter_tree_destroy(struct percpu_counter_tree *counter) +{ + return percpu_counter_tree_destroy_many(counter, 1); +} + +static +int percpu_counter_tree_carry(int orig, int res, int inc, unsigned int bit_mask) +{ + if (inc < 0) { + inc = -(-inc & ~(bit_mask - 1)); + /* + * xor bit_mask: underflow. + * + * If inc has bit set, decrement an additional bit if + * there is _no_ bit transition between orig and res. + * Else, inc has bit cleared, decrement an additional + * bit if there is a bit transition between orig and + * res. + */ + if ((inc ^ orig ^ res) & bit_mask) + inc -= bit_mask; + } else { + inc &= ~(bit_mask - 1); + /* + * xor bit_mask: overflow. + * + * If inc has bit set, increment an additional bit if + * there is _no_ bit transition between orig and res. + * Else, inc has bit cleared, increment an additional + * bit if there is a bit transition between orig and + * res. + */ + if ((inc ^ orig ^ res) & bit_mask) + inc += bit_mask; + } + return inc; +} + +/* + * It does not matter through which path the carry propagates up the + * tree, therefore there is no need to disable preemption because the + * cpu number is only used to favor cache locality. + */ +static +void percpu_counter_tree_add_slowpath(struct percpu_counter_tree *counter, int inc) +{ + unsigned int level_items, nr_levels = counter_config->nr_levels, + level, n_arity_order, bit_mask; + struct percpu_counter_tree_level_item *item = counter->items; + unsigned int cpu = raw_smp_processor_id(); + + WARN_ON_ONCE(!nr_cpus_order); /* Should never be called for 1 cpu. */ + + n_arity_order = counter_config->n_arity_order[0]; + bit_mask = counter->level0_bit_mask << n_arity_order; + level_items = 1U << (nr_cpus_order - n_arity_order); + + for (level = 1; level < nr_levels; level++) { + atomic_t *count = &item[cpu & (level_items - 1)].count; + unsigned int orig, res; + + res = atomic_add_return_relaxed(inc, count); + orig = res - inc; + inc = percpu_counter_tree_carry(orig, res, inc, bit_mask); + if (likely(!inc)) + return; + item += level_items; + n_arity_order = counter_config->n_arity_order[level]; + level_items >>= n_arity_order; + bit_mask <<= n_arity_order; + } + atomic_add(inc, counter->approx_sum.a); +} + +/** + * percpu_counter_tree_add() - Add to a per-CPU counter tree. + * @counter: Counter added to. + * @inc: Increment value (either positive or negative). + * + * Add @inc to a per-CPU counter tree. This is a fast-path which will + * typically increment per-CPU counters as long as there is no carry + * greater or equal to the counter tree batch size. + */ +void percpu_counter_tree_add(struct percpu_counter_tree *counter, int inc) +{ + unsigned int bit_mask = counter->level0_bit_mask, orig, res; + + res = this_cpu_add_return(*counter->level0, inc); + orig = res - inc; + inc = percpu_counter_tree_carry(orig, res, inc, bit_mask); + if (likely(!inc)) + return; + percpu_counter_tree_add_slowpath(counter, inc); +} + + +static +int percpu_counter_tree_precise_sum_unbiased(struct percpu_counter_tree *counter) +{ + unsigned int sum = 0; + int cpu; + + for_each_possible_cpu(cpu) + sum += *per_cpu_ptr(counter->level0, cpu); + return (int) sum; +} + +/** + * percpu_counter_tree_precise_sum() - Return precise counter sum. + * @counter: The counter to sum. + * + * Querying the precise sum is relatively expensive because it needs to + * iterate over all CPUs. + * This is meant to be used when accuracy is preferred over speed. + * + * Return: The current precise counter sum. + */ +int percpu_counter_tree_precise_sum(struct percpu_counter_tree *counter) +{ + return percpu_counter_tree_precise_sum_unbiased(counter) + READ_ONCE(counter->bias); +} + +static +int compare_delta(int delta, unsigned int accuracy_neg, unsigned int accuracy_pos) +{ + if (delta >= 0) { + if (delta <= accuracy_pos) + return 0; + else + return 1; + } else { + if (-delta <= accuracy_neg) + return 0; + else + return -1; + } +} + +/** + * percpu_counter_tree_approximate_compare - Approximated comparison of two counter trees. + * @a: First counter to compare. + * @b: Second counter to compare. + * + * Evaluate an approximate comparison of two counter trees. + * This approximation comparison is fast, and provides an accurate + * answer if the counters are found to be either less than or greater + * than the other. However, if the approximated comparison returns + * 0, the counters respective sums are found to be within the two + * counters accuracy range. + * + * Return: + * * %0 - Counters @a and @b do not differ by more than the sum of their respective + * accuracy ranges. + * * %-1 - Counter @a less than counter @b. + * * %1 - Counter @a is greater than counter @b. + */ +int percpu_counter_tree_approximate_compare(struct percpu_counter_tree *a, struct percpu_counter_tree *b) +{ + return compare_delta(percpu_counter_tree_approximate_sum(a) - percpu_counter_tree_approximate_sum(b), + a->approx_accuracy_range.over + b->approx_accuracy_range.under, + a->approx_accuracy_range.under + b->approx_accuracy_range.over); +} + +/** + * percpu_counter_tree_approximate_compare_value - Approximated comparison of a counter tree against a given value. + * @counter: Counter to compare. + * @v: Value to compare. + * + * Evaluate an approximate comparison of a counter tree against a given value. + * This approximation comparison is fast, and provides an accurate + * answer if the counter is found to be either less than or greater + * than the value. However, if the approximated comparison returns + * 0, the value is within the counter accuracy range. + * + * Return: + * * %0 - The value @v is within the accuracy range of the counter. + * * %-1 - The value @v is less than the counter. + * * %1 - The value @v is greater than the counter. + */ +int percpu_counter_tree_approximate_compare_value(struct percpu_counter_tree *counter, int v) +{ + return compare_delta(v - percpu_counter_tree_approximate_sum(counter), + counter->approx_accuracy_range.under, + counter->approx_accuracy_range.over); +} + +/** + * percpu_counter_tree_precise_compare - Precise comparison of two counter trees. + * @a: First counter to compare. + * @b: Second counter to compare. + * + * Evaluate a precise comparison of two counter trees. + * As an optimization, it uses the approximate counter comparison + * to quickly compare counters which are far apart. Only cases where + * counter sums are within the accuracy range require precise counter + * sums. + * + * Return: + * * %0 - Counters are equal. + * * %-1 - Counter @a less than counter @b. + * * %1 - Counter @a is greater than counter @b. + */ +int percpu_counter_tree_precise_compare(struct percpu_counter_tree *a, struct percpu_counter_tree *b) +{ + int count_a = percpu_counter_tree_approximate_sum(a), + count_b = percpu_counter_tree_approximate_sum(b); + unsigned int accuracy_a, accuracy_b; + int delta = count_a - count_b; + int res; + + res = compare_delta(delta, + a->approx_accuracy_range.over + b->approx_accuracy_range.under, + a->approx_accuracy_range.under + b->approx_accuracy_range.over); + /* The values are distanced enough for an accurate approximated comparison. */ + if (res) + return res; + + /* + * The approximated comparison is within the accuracy range, therefore at least one + * precise sum is needed. Sum the counter which has the largest accuracy first. + */ + if (delta >= 0) { + accuracy_a = a->approx_accuracy_range.under; + accuracy_b = b->approx_accuracy_range.over; + } else { + accuracy_a = a->approx_accuracy_range.over; + accuracy_b = b->approx_accuracy_range.under; + } + if (accuracy_b < accuracy_a) { + count_a = percpu_counter_tree_precise_sum(a); + res = compare_delta(count_a - count_b, + b->approx_accuracy_range.under, + b->approx_accuracy_range.over); + if (res) + return res; + /* Precise sum of second counter is required. */ + count_b = percpu_counter_tree_precise_sum(b); + } else { + count_b = percpu_counter_tree_precise_sum(b); + res = compare_delta(count_a - count_b, + a->approx_accuracy_range.over, + a->approx_accuracy_range.under); + if (res) + return res; + /* Precise sum of second counter is required. */ + count_a = percpu_counter_tree_precise_sum(a); + } + if (count_a - count_b < 0) + return -1; + if (count_a - count_b > 0) + return 1; + return 0; +} + +/** + * percpu_counter_tree_precise_compare_value - Precise comparison of a counter tree against a given value. + * @counter: Counter to compare. + * @v: Value to compare. + * + * Evaluate a precise comparison of a counter tree against a given value. + * As an optimization, it uses the approximate counter comparison + * to quickly identify whether the counter and value are far apart. + * Only cases where the value is within the counter accuracy range + * require a precise counter sum. + * + * Return: + * * %0 - The value @v is equal to the counter. + * * %-1 - The value @v is less than the counter. + * * %1 - The value @v is greater than the counter. + */ +int percpu_counter_tree_precise_compare_value(struct percpu_counter_tree *counter, int v) +{ + int count = percpu_counter_tree_approximate_sum(counter); + int res; + + res = compare_delta(v - count, + counter->approx_accuracy_range.under, + counter->approx_accuracy_range.over); + /* The values are distanced enough for an accurate approximated comparison. */ + if (res) + return res; + + /* Precise sum is required. */ + count = percpu_counter_tree_precise_sum(counter); + if (v - count < 0) + return -1; + if (v - count > 0) + return 1; + return 0; +} + +static +void percpu_counter_tree_set_bias(struct percpu_counter_tree *counter, int bias) +{ + WRITE_ONCE(counter->bias, bias); +} + +/** + * percpu_counter_tree_set - Set the counter tree sum to a given value. + * @counter: Counter to set. + * @v: Value to set. + * + * Set the counter sum to a given value. It can be useful for instance + * to reset the counter sum to 0. Note that even after setting the + * counter sum to a given value, the counter sum approximation can + * return any value within the accuracy range around that value. + */ +void percpu_counter_tree_set(struct percpu_counter_tree *counter, int v) +{ + percpu_counter_tree_set_bias(counter, + v - percpu_counter_tree_precise_sum_unbiased(counter)); +} + +/** + * percpu_counter_tree_approximate_accuracy_range - Query the accuracy range for a counter tree. + * @counter: Counter to query. + * @under: Pointer where to store the to the accuracy range below the approximation. + * @over: Pointer where to store the to the accuracy range above the approximation. + * + * Query the accuracy range limits for the counter. + * Because of two's complement binary representation, the "under" range is typically + * slightly larger than the "over" range. + * Those values are derived from the hardware topology and the counter tree batch size. + * They are invariant for a given counter tree. + * Using this function should not be typically required, see the following functions instead: + * * percpu_counter_tree_approximate_compare(), + * * percpu_counter_tree_approximate_compare_value(), + * * percpu_counter_tree_precise_compare(), + * * percpu_counter_tree_precise_compare_value(). + */ +void percpu_counter_tree_approximate_accuracy_range(struct percpu_counter_tree *counter, + unsigned int *under, unsigned int *over) +{ + *under = counter->approx_accuracy_range.under; + *over = counter->approx_accuracy_range.over; +} + +/* + * percpu_counter_tree_items_size - Query the size required for counter tree items. + * + * Query the size of the memory area required to hold the counter tree + * items. This depends on the hardware topology and is invariant after + * boot. + * + * Return: Size required to hold tree items. + */ +size_t percpu_counter_tree_items_size(void) +{ + if (!nr_cpus_order) + return 0; + return counter_config->nr_items * sizeof(struct percpu_counter_tree_level_item); +} + +static void __init calculate_accuracy_topology(void) +{ + unsigned int nr_levels = counter_config->nr_levels, level; + unsigned int level_items = 1U << nr_cpus_order; + unsigned int batch_size = 1; + + for (level = 0; level < nr_levels; level++) { + unsigned int n_arity_order = counter_config->n_arity_order[level]; + + /* + * The accuracy multiplier is derived from a batch size of 1 + * to speed up calculating the accuracy at tree initialization. + */ + accuracy_multiplier += batch_size * level_items; + batch_size <<= n_arity_order; + level_items >>= n_arity_order; + } +} + +int __init percpu_counter_tree_subsystem_init(void) +{ + nr_cpus_order = get_count_order(nr_cpu_ids); + if (WARN_ON_ONCE(nr_cpus_order >= ARRAY_SIZE(per_nr_cpu_order_config))) { + printk(KERN_ERR "Unsupported number of CPUs (%u)\n", nr_cpu_ids); + return -1; + } + counter_config = &per_nr_cpu_order_config[nr_cpus_order]; + calculate_accuracy_topology(); + return 0; +} _ Patches currently in -mm which might be from mathieu.desnoyers@efficios.com are lib-introduce-hierarchical-per-cpu-counters.patch mm-fix-oom-killer-inaccuracy-on-large-many-core-systems.patch mm-implement-precise-oom-killer-task-selection.patch