* [PATCH v4 2/2] mm: make minimum slab alignment a runtime property
2022-04-26 20:32 [PATCH v4 1/2] printk: stop including cache.h from printk.h Peter Collingbourne
@ 2022-04-26 20:32 ` Peter Collingbourne
2022-04-27 3:25 ` [PATCH v4 1/2] printk: stop including cache.h from printk.h kernel test robot
2022-04-27 9:25 ` kernel test robot
2 siblings, 0 replies; 4+ messages in thread
From: Peter Collingbourne @ 2022-04-26 20:32 UTC (permalink / raw)
To: Andrey Konovalov, Hyeonggon Yoo, Andrew Morton, Catalin Marinas
Cc: Peter Collingbourne, Linux ARM, Linux Memory Management List,
Linux Kernel Mailing List, vbabka, penberg, roman.gushchin,
iamjoonsoo.kim, rientjes, Herbert Xu, Andrey Ryabinin,
Alexander Potapenko, Dmitry Vyukov, kasan-dev, Eric Biederman,
Kees Cook
When CONFIG_KASAN_HW_TAGS is enabled we currently increase the minimum
slab alignment to 16. This happens even if MTE is not supported in
hardware or disabled via kasan=off, which creates an unnecessary
memory overhead in those cases. Eliminate this overhead by making
the minimum slab alignment a runtime property and only aligning to
16 if KASAN is enabled at runtime.
On a DragonBoard 845c (non-MTE hardware) with a kernel built with
CONFIG_KASAN_HW_TAGS, waiting for quiescence after a full Android
boot I see the following Slab measurements in /proc/meminfo (median
of 3 reboots):
Before: 169020 kB
After: 167304 kB
Link: https://linux-review.googlesource.com/id/I752e725179b43b144153f4b6f584ceb646473ead
Signed-off-by: Peter Collingbourne <pcc@google.com>
Reviewed-by: Andrey Konovalov <andreyknvl@gmail.com>
Reviewed-by: Hyeonggon Yoo <42.hyeyoo@gmail.com>
Tested-by: Hyeonggon Yoo <42.hyeyoo@gmail.com>
Acked-by: David Rientjes <rientjes@google.com>
Reviewed-by: Catalin Marinas <catalin.marinas@arm.com>
---
v4:
- add a dependent patch to fix the build with CONFIG_JUMP_LABEL disabled
v3:
- go back to ARCH_SLAB_MINALIGN
- revert changes to fs/binfmt_flat.c
- update arch_slab_minalign() comment to say that it must be a power of two
v2:
- use max instead of max_t in flat_stack_align()
arch/arm64/include/asm/cache.h | 17 ++++++++++++-----
include/linux/slab.h | 12 ++++++++++++
mm/slab.c | 7 +++----
mm/slab_common.c | 3 +--
mm/slob.c | 6 +++---
5 files changed, 31 insertions(+), 14 deletions(-)
diff --git a/arch/arm64/include/asm/cache.h b/arch/arm64/include/asm/cache.h
index a074459f8f2f..22b22dc1b1b5 100644
--- a/arch/arm64/include/asm/cache.h
+++ b/arch/arm64/include/asm/cache.h
@@ -6,6 +6,7 @@
#define __ASM_CACHE_H
#include <asm/cputype.h>
+#include <asm/mte-def.h>
#define CTR_L1IP_SHIFT 14
#define CTR_L1IP_MASK 3
@@ -49,16 +50,22 @@
*/
#define ARCH_DMA_MINALIGN (128)
+#ifndef __ASSEMBLY__
+
+#include <linux/bitops.h>
+#include <linux/kasan-enabled.h>
+
#ifdef CONFIG_KASAN_SW_TAGS
#define ARCH_SLAB_MINALIGN (1ULL << KASAN_SHADOW_SCALE_SHIFT)
#elif defined(CONFIG_KASAN_HW_TAGS)
-#define ARCH_SLAB_MINALIGN MTE_GRANULE_SIZE
+static inline size_t arch_slab_minalign(void)
+{
+ return kasan_hw_tags_enabled() ? MTE_GRANULE_SIZE :
+ __alignof__(unsigned long long);
+}
+#define arch_slab_minalign() arch_slab_minalign()
#endif
-#ifndef __ASSEMBLY__
-
-#include <linux/bitops.h>
-
#define ICACHEF_ALIASING 0
#define ICACHEF_VPIPT 1
extern unsigned long __icache_flags;
diff --git a/include/linux/slab.h b/include/linux/slab.h
index 373b3ef99f4e..2c7190db4cc0 100644
--- a/include/linux/slab.h
+++ b/include/linux/slab.h
@@ -209,6 +209,18 @@ void kmem_dump_obj(void *object);
#define ARCH_SLAB_MINALIGN __alignof__(unsigned long long)
#endif
+/*
+ * Arches can define this function if they want to decide the minimum slab
+ * alignment at runtime. The value returned by the function must be a power
+ * of two and >= ARCH_SLAB_MINALIGN.
+ */
+#ifndef arch_slab_minalign
+static inline size_t arch_slab_minalign(void)
+{
+ return ARCH_SLAB_MINALIGN;
+}
+#endif
+
/*
* kmalloc and friends return ARCH_KMALLOC_MINALIGN aligned
* pointers. kmem_cache_alloc and friends return ARCH_SLAB_MINALIGN
diff --git a/mm/slab.c b/mm/slab.c
index 0edb474edef1..97b756976c8b 100644
--- a/mm/slab.c
+++ b/mm/slab.c
@@ -3009,10 +3009,9 @@ static void *cache_alloc_debugcheck_after(struct kmem_cache *cachep,
objp += obj_offset(cachep);
if (cachep->ctor && cachep->flags & SLAB_POISON)
cachep->ctor(objp);
- if (ARCH_SLAB_MINALIGN &&
- ((unsigned long)objp & (ARCH_SLAB_MINALIGN-1))) {
- pr_err("0x%px: not aligned to ARCH_SLAB_MINALIGN=%d\n",
- objp, (int)ARCH_SLAB_MINALIGN);
+ if ((unsigned long)objp & (arch_slab_minalign() - 1)) {
+ pr_err("0x%px: not aligned to arch_slab_minalign()=%d\n", objp,
+ (int)arch_slab_minalign());
}
return objp;
}
diff --git a/mm/slab_common.c b/mm/slab_common.c
index 2b3206a2c3b5..33cc49810a54 100644
--- a/mm/slab_common.c
+++ b/mm/slab_common.c
@@ -154,8 +154,7 @@ static unsigned int calculate_alignment(slab_flags_t flags,
align = max(align, ralign);
}
- if (align < ARCH_SLAB_MINALIGN)
- align = ARCH_SLAB_MINALIGN;
+ align = max_t(size_t, align, arch_slab_minalign());
return ALIGN(align, sizeof(void *));
}
diff --git a/mm/slob.c b/mm/slob.c
index 40ea6e2d4ccd..3bd2669bd690 100644
--- a/mm/slob.c
+++ b/mm/slob.c
@@ -478,7 +478,7 @@ static __always_inline void *
__do_kmalloc_node(size_t size, gfp_t gfp, int node, unsigned long caller)
{
unsigned int *m;
- int minalign = max_t(size_t, ARCH_KMALLOC_MINALIGN, ARCH_SLAB_MINALIGN);
+ int minalign = max_t(size_t, ARCH_KMALLOC_MINALIGN, arch_slab_minalign());
void *ret;
gfp &= gfp_allowed_mask;
@@ -555,7 +555,7 @@ void kfree(const void *block)
sp = virt_to_folio(block);
if (folio_test_slab(sp)) {
- int align = max_t(size_t, ARCH_KMALLOC_MINALIGN, ARCH_SLAB_MINALIGN);
+ int align = max_t(size_t, ARCH_KMALLOC_MINALIGN, arch_slab_minalign());
unsigned int *m = (unsigned int *)(block - align);
slob_free(m, *m + align);
} else {
@@ -584,7 +584,7 @@ size_t __ksize(const void *block)
if (unlikely(!folio_test_slab(folio)))
return folio_size(folio);
- align = max_t(size_t, ARCH_KMALLOC_MINALIGN, ARCH_SLAB_MINALIGN);
+ align = max_t(size_t, ARCH_KMALLOC_MINALIGN, arch_slab_minalign());
m = (unsigned int *)(block - align);
return SLOB_UNITS(*m) * SLOB_UNIT;
}
--
2.36.0.rc2.479.g8af0fa9b8e-goog
^ permalink raw reply related [flat|nested] 4+ messages in thread* Re: [PATCH v4 1/2] printk: stop including cache.h from printk.h
2022-04-26 20:32 [PATCH v4 1/2] printk: stop including cache.h from printk.h Peter Collingbourne
2022-04-26 20:32 ` [PATCH v4 2/2] mm: make minimum slab alignment a runtime property Peter Collingbourne
@ 2022-04-27 3:25 ` kernel test robot
2022-04-27 9:25 ` kernel test robot
2 siblings, 0 replies; 4+ messages in thread
From: kernel test robot @ 2022-04-27 3:25 UTC (permalink / raw)
To: Peter Collingbourne, Andrey Konovalov, Hyeonggon Yoo,
Andrew Morton, Catalin Marinas
Cc: llvm, kbuild-all, Linux Memory Management List,
Peter Collingbourne, Linux ARM, Linux Kernel Mailing List, vbabka,
penberg, roman.gushchin, iamjoonsoo.kim, rientjes, Herbert Xu,
Andrey Ryabinin, Alexander Potapenko, Dmitry Vyukov, kasan-dev,
Eric Biederman, Kees Cook
Hi Peter,
Thank you for the patch! Yet something to improve:
[auto build test ERROR on vbabka-slab/for-next]
[also build test ERROR on arm64/for-next/core linus/master v5.18-rc4 next-20220426]
[cannot apply to dennis-percpu/for-next]
[If your patch is applied to the wrong git tree, kindly drop us a note.
And when submitting patch, we suggest to use '--base' as documented in
https://git-scm.com/docs/git-format-patch]
url: https://github.com/intel-lab-lkp/linux/commits/Peter-Collingbourne/printk-stop-including-cache-h-from-printk-h/20220427-043357
base: git://git.kernel.org/pub/scm/linux/kernel/git/vbabka/slab.git for-next
config: arm-randconfig-r025-20220425 (https://download.01.org/0day-ci/archive/20220427/202204271135.P05x34Pe-lkp@intel.com/config)
compiler: clang version 15.0.0 (https://github.com/llvm/llvm-project 1cddcfdc3c683b393df1a5c9063252eb60e52818)
reproduce (this is a W=1 build):
wget https://raw.githubusercontent.com/intel/lkp-tests/master/sbin/make.cross -O ~/bin/make.cross
chmod +x ~/bin/make.cross
# install arm cross compiling tool for clang build
# apt-get install binutils-arm-linux-gnueabi
# https://github.com/intel-lab-lkp/linux/commit/edcb0f592304f7849a39586f9e3fe0d8f6e6c6b9
git remote add linux-review https://github.com/intel-lab-lkp/linux
git fetch --no-tags linux-review Peter-Collingbourne/printk-stop-including-cache-h-from-printk-h/20220427-043357
git checkout edcb0f592304f7849a39586f9e3fe0d8f6e6c6b9
# save the config file
mkdir build_dir && cp config build_dir/.config
COMPILER_INSTALL_PATH=$HOME/0day COMPILER=clang make.cross W=1 O=build_dir ARCH=arm SHELL=/bin/bash
If you fix the issue, kindly add following tag as appropriate
Reported-by: kernel test robot <lkp@intel.com>
All errors (new ones prefixed by >>):
In file included from kernel/bpf/bpf_lru_list.c:8:
>> kernel/bpf/bpf_lru_list.h:36:21: error: expected ';' at end of declaration list
raw_spinlock_t lock ____cacheline_aligned_in_smp;
^
;
1 error generated.
vim +36 kernel/bpf/bpf_lru_list.h
3a08c2fd763450a Martin KaFai Lau 2016-11-11 29
3a08c2fd763450a Martin KaFai Lau 2016-11-11 30 struct bpf_lru_list {
3a08c2fd763450a Martin KaFai Lau 2016-11-11 31 struct list_head lists[NR_BPF_LRU_LIST_T];
3a08c2fd763450a Martin KaFai Lau 2016-11-11 32 unsigned int counts[NR_BPF_LRU_LIST_COUNT];
0ac16296ffc638f Qiujun Huang 2020-04-03 33 /* The next inactive list rotation starts from here */
3a08c2fd763450a Martin KaFai Lau 2016-11-11 34 struct list_head *next_inactive_rotation;
3a08c2fd763450a Martin KaFai Lau 2016-11-11 35
3a08c2fd763450a Martin KaFai Lau 2016-11-11 @36 raw_spinlock_t lock ____cacheline_aligned_in_smp;
3a08c2fd763450a Martin KaFai Lau 2016-11-11 37 };
3a08c2fd763450a Martin KaFai Lau 2016-11-11 38
--
0-DAY CI Kernel Test Service
https://01.org/lkp
^ permalink raw reply [flat|nested] 4+ messages in thread* Re: [PATCH v4 1/2] printk: stop including cache.h from printk.h
2022-04-26 20:32 [PATCH v4 1/2] printk: stop including cache.h from printk.h Peter Collingbourne
2022-04-26 20:32 ` [PATCH v4 2/2] mm: make minimum slab alignment a runtime property Peter Collingbourne
2022-04-27 3:25 ` [PATCH v4 1/2] printk: stop including cache.h from printk.h kernel test robot
@ 2022-04-27 9:25 ` kernel test robot
2 siblings, 0 replies; 4+ messages in thread
From: kernel test robot @ 2022-04-27 9:25 UTC (permalink / raw)
To: Peter Collingbourne, Andrey Konovalov, Hyeonggon Yoo,
Andrew Morton, Catalin Marinas
Cc: kbuild-all, Linux Memory Management List, Peter Collingbourne,
Linux ARM, Linux Kernel Mailing List, vbabka, penberg,
roman.gushchin, iamjoonsoo.kim, rientjes, Herbert Xu,
Andrey Ryabinin, Alexander Potapenko, Dmitry Vyukov, kasan-dev,
Eric Biederman, Kees Cook
Hi Peter,
Thank you for the patch! Yet something to improve:
[auto build test ERROR on vbabka-slab/for-next]
[also build test ERROR on arm64/for-next/core linus/master v5.18-rc4 next-20220427]
[cannot apply to dennis-percpu/for-next]
[If your patch is applied to the wrong git tree, kindly drop us a note.
And when submitting patch, we suggest to use '--base' as documented in
https://git-scm.com/docs/git-format-patch]
url: https://github.com/intel-lab-lkp/linux/commits/Peter-Collingbourne/printk-stop-including-cache-h-from-printk-h/20220427-043357
base: git://git.kernel.org/pub/scm/linux/kernel/git/vbabka/slab.git for-next
config: csky-randconfig-r031-20220425 (https://download.01.org/0day-ci/archive/20220427/202204271721.kgeFN450-lkp@intel.com/config)
compiler: csky-linux-gcc (GCC) 11.3.0
reproduce (this is a W=1 build):
wget https://raw.githubusercontent.com/intel/lkp-tests/master/sbin/make.cross -O ~/bin/make.cross
chmod +x ~/bin/make.cross
# https://github.com/intel-lab-lkp/linux/commit/edcb0f592304f7849a39586f9e3fe0d8f6e6c6b9
git remote add linux-review https://github.com/intel-lab-lkp/linux
git fetch --no-tags linux-review Peter-Collingbourne/printk-stop-including-cache-h-from-printk-h/20220427-043357
git checkout edcb0f592304f7849a39586f9e3fe0d8f6e6c6b9
# save the config file
mkdir build_dir && cp config build_dir/.config
COMPILER_INSTALL_PATH=$HOME/0day COMPILER=gcc-11.3.0 make.cross W=1 O=build_dir ARCH=csky prepare
If you fix the issue, kindly add following tag as appropriate
Reported-by: kernel test robot <lkp@intel.com>
All errors (new ones prefixed by >>):
In file included from include/linux/compiler_types.h:73,
from <command-line>:
>> arch/csky/include/asm/processor.h:19:13: error: 'SMP_CACHE_BYTES' undeclared here (not in a function); did you mean 'L1_CACHE_BYTES'?
19 | } __aligned(SMP_CACHE_BYTES);
| ^~~~~~~~~~~~~~~
include/linux/compiler_attributes.h:33:68: note: in definition of macro '__aligned'
33 | #define __aligned(x) __attribute__((__aligned__(x)))
| ^
make[2]: *** [scripts/Makefile.build:120: arch/csky/kernel/asm-offsets.s] Error 1
make[2]: Target '__build' not remade because of errors.
make[1]: *** [Makefile:1194: prepare0] Error 2
make[1]: Target 'prepare' not remade because of errors.
make: *** [Makefile:219: __sub-make] Error 2
make: Target 'prepare' not remade because of errors.
vim +19 arch/csky/include/asm/processor.h
e9564df753fd547 Guo Ren 2018-09-05 16
e9564df753fd547 Guo Ren 2018-09-05 17 struct cpuinfo_csky {
e9564df753fd547 Guo Ren 2018-09-05 18 unsigned long asid_cache;
e9564df753fd547 Guo Ren 2018-09-05 @19 } __aligned(SMP_CACHE_BYTES);
e9564df753fd547 Guo Ren 2018-09-05 20
--
0-DAY CI Kernel Test Service
https://01.org/lkp
^ permalink raw reply [flat|nested] 4+ messages in thread