From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 1BD8AECAAA1 for ; Thu, 27 Oct 2022 12:13:01 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:In-Reply-To:MIME-Version:References: Message-ID:Subject:Cc:To:From:Date:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=8WIdHpFuzYe+l37DNhsab20ypmRs8Yrduqqhe+zsjVk=; b=wtQPQRwAbzX1ck 1bA7irIIHr5DHJtoFqnhyXoD4oqFPth+33BsEo/DM0Dl4olgxV5c0I1c0HKRfpIkgV5OvDEMYkBNb t9+SrjItiKzheVIeCAzVSScJUf1J4XOaR697BknJoaRdC7uB5XJ7/iRyb94cb5Wt6jB3c6v1ktNVn qJH8oxO+ZWdoE1Brf1jsGpTRUXpm6iuJU4YfCurlNsw5PxtnKas5usTQl1B22DXOhR8MtHQYckvzW Mty7pNdZfQN+CHZQzNwAsKndRgVFksvGb3c0O9HHSs3rjSM1LANWdouPdkXgmijZ2phucv/ufVqVX XGBg/cKjMJP6Ok4eBc5g==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1oo1jv-00D8Y9-IR; Thu, 27 Oct 2022 12:12:03 +0000 Received: from mail-pj1-x1036.google.com ([2607:f8b0:4864:20::1036]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1oo1js-00D8Wb-4g for linux-arm-kernel@lists.infradead.org; Thu, 27 Oct 2022 12:12:01 +0000 Received: by mail-pj1-x1036.google.com with SMTP id d59-20020a17090a6f4100b00213202d77e1so6116049pjk.2 for ; Thu, 27 Oct 2022 05:11:58 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:from:date:from:to:cc:subject:date:message-id:reply-to; bh=F6Ci2e1GctkVuBTZcZj2eOapSo4Y9VSQb5VfOkL4ECQ=; b=I1BrW8L4ftDKz7+Ha5iaHWSl0PQkJLe9nd7F4aZFl6znTYgH3oUmR8iLi7IUccm7uR H3lVXMkh7opU7Ylg8CssHwAvXjVsgkxssw0XM+B2P3vucup510f9Z8Rh3ChdIbSjJyjQ nyyfpHvv6/Vx4koh31Dg9bKDaC3o6tLmUO4lMcqoKhJYKweBXPDeB5+Mxc7ul5DGQ+qT vYEtva9OUTJQOyxzkK7WwxvsgqhXY15uMNhcI2XgCk9GgkIHi8m1id0YRwB8tzHGWEES 8kh5tO6+oUHZIJ2RcRZTzGF315rgUGJXQSg7TNPmIAu2/p72aFq5Rr/VobYc1mMSgigS pnPg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:from:date:x-gm-message-state:from:to:cc:subject:date :message-id:reply-to; bh=F6Ci2e1GctkVuBTZcZj2eOapSo4Y9VSQb5VfOkL4ECQ=; b=GgofFFhFqUjZQ6VjjkEw4AvNjrsQFSQvOCS9cf5xddkkGDWhJnVa+4eZehupN+lmSt 3xfc4fXFUaDtKP2qBdvaYmAU6VGqCUXyx/0ReRuNQeiqTtTXFr7N4jiUihy7QwsKujoH dAiMAI8hpGkNPdUtDkGec/Dr52W37Z23XNeooeYyuj0V9swJLpHbwhnQ3GR2GhjG1gE0 a8oNnbTO+S/VcHX6Cr6SXIk/EKrKFEfBv7oZjRDiIzoAyyaZzucOu0++pQLnZVYHc85u 6zXL5fZlUseIi1kazhZxeThQ6lcBlwv3NDY9kjEcfM5tyEtkvMR1wo8yozFsJRYHpxip dHHg== X-Gm-Message-State: ACrzQf1ji58GcIf65nzByPJi/37RN/yMYaf50/K/VTNX6yKJ7ld3ZoEx zkSnWZSpzX1tPzW+O/mda1c= X-Google-Smtp-Source: AMsMyM4nffcpt1vkSfU7BlBDrYyw1AhchiK11XevHkvIemdpwdIBMJbqBGjSR5GoiwCY0WaDKWNOrg== X-Received: by 2002:a17:903:2596:b0:186:a395:c4bd with SMTP id jb22-20020a170903259600b00186a395c4bdmr22258036plb.60.1666872717473; Thu, 27 Oct 2022 05:11:57 -0700 (PDT) Received: from hyeyoo ([114.29.91.56]) by smtp.gmail.com with ESMTPSA id q4-20020aa79824000000b0056bb7d90f0fsm1063961pfl.182.2022.10.27.05.11.52 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 27 Oct 2022 05:11:56 -0700 (PDT) Date: Thu, 27 Oct 2022 21:11:49 +0900 From: Hyeonggon Yoo <42.hyeyoo@gmail.com> To: Catalin Marinas Cc: Linus Torvalds , Arnd Bergmann , Will Deacon , Marc Zyngier , Greg Kroah-Hartman , Andrew Morton , Herbert Xu , Ard Biesheuvel , Christoph Hellwig , Isaac Manjarres , Saravana Kannan , linux-mm@kvack.org, Vlastimil Babka , David Rientjes , Christoph Lameter , Roman Gushchin , linux-arm-kernel@lists.infradead.org Subject: Re: [PATCH v2 1/2] mm: slab: Introduce __GFP_PACKED for smaller kmalloc() alignments Message-ID: References: <20221025205247.3264568-1-catalin.marinas@arm.com> <20221025205247.3264568-2-catalin.marinas@arm.com> MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: <20221025205247.3264568-2-catalin.marinas@arm.com> X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20221027_051200_222980_94D78E47 X-CRM114-Status: GOOD ( 32.79 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org On Tue, Oct 25, 2022 at 09:52:46PM +0100, Catalin Marinas wrote: > By default kmalloc() returns objects aligned to ARCH_KMALLOC_MINALIGN. > This can be somewhat large on architectures defining ARCH_DMA_MINALIGN > (e.g. 128 on arm64) and significant memory is wasted through small > kmalloc() allocations. > > Reduce the minimum alignment for kmalloc() to the default > KMALLOC_MIN_SIZE (8 for slub, 32 for slab) but align the > requested size to the bigger ARCH_KMALLOC_MINALIGN unless a newly added > __GFP_PACKED flag is passed. > > With this gfp flag, the alignment is > reduced to KMALLOC_PACKED_ALIGN, at least sizeof(unsigned long long). > > There's no slob support. > Thank you for pushing it forward! > Signed-off-by: Catalin Marinas > --- > include/linux/gfp_types.h | 10 ++++++++-- > include/linux/slab.h | 22 ++++++++++++++++++---- > mm/slab_common.c | 3 ++- > 3 files changed, 28 insertions(+), 7 deletions(-) > > diff --git a/include/linux/gfp_types.h b/include/linux/gfp_types.h > index d88c46ca82e1..305cb8cb6f8b 100644 > --- a/include/linux/gfp_types.h > +++ b/include/linux/gfp_types.h > @@ -55,8 +55,9 @@ typedef unsigned int __bitwise gfp_t; > #define ___GFP_SKIP_KASAN_UNPOISON 0 > #define ___GFP_SKIP_KASAN_POISON 0 > #endif > +#define ___GFP_PACKED 0x8000000u > #ifdef CONFIG_LOCKDEP > -#define ___GFP_NOLOCKDEP 0x8000000u > +#define ___GFP_NOLOCKDEP 0x10000000u > #else > #define ___GFP_NOLOCKDEP 0 > #endif > @@ -243,6 +244,10 @@ typedef unsigned int __bitwise gfp_t; > * > * %__GFP_SKIP_KASAN_POISON makes KASAN skip poisoning on page deallocation. > * Typically, used for userspace pages. Only effective in HW_TAGS mode. > + * > + * %__GFP_PACKED returns a pointer aligned to the possibly smaller > + * KMALLOC_PACKED_ALIGN rather than ARCH_KMALLOC_MINALIGN. Useful for small > + * object allocation on architectures that define large ARCH_DMA_MINALIGN. > */ > #define __GFP_NOWARN ((__force gfp_t)___GFP_NOWARN) > #define __GFP_COMP ((__force gfp_t)___GFP_COMP) > @@ -251,12 +256,13 @@ typedef unsigned int __bitwise gfp_t; > #define __GFP_SKIP_ZERO ((__force gfp_t)___GFP_SKIP_ZERO) > #define __GFP_SKIP_KASAN_UNPOISON ((__force gfp_t)___GFP_SKIP_KASAN_UNPOISON) > #define __GFP_SKIP_KASAN_POISON ((__force gfp_t)___GFP_SKIP_KASAN_POISON) > +#define __GFP_PACKED ((__force gfp_t)___GFP_PACKED) > > /* Disable lockdep for GFP context tracking */ > #define __GFP_NOLOCKDEP ((__force gfp_t)___GFP_NOLOCKDEP) > > /* Room for N __GFP_FOO bits */ > -#define __GFP_BITS_SHIFT (27 + IS_ENABLED(CONFIG_LOCKDEP)) > +#define __GFP_BITS_SHIFT (28 + IS_ENABLED(CONFIG_LOCKDEP)) > #define __GFP_BITS_MASK ((__force gfp_t)((1 << __GFP_BITS_SHIFT) - 1)) > > /** > diff --git a/include/linux/slab.h b/include/linux/slab.h > index 90877fcde70b..0f59585b5fbf 100644 > --- a/include/linux/slab.h > +++ b/include/linux/slab.h > @@ -223,8 +223,6 @@ void kmem_dump_obj(void *object); > */ > #if defined(ARCH_DMA_MINALIGN) && ARCH_DMA_MINALIGN > 8 > #define ARCH_KMALLOC_MINALIGN ARCH_DMA_MINALIGN > -#define KMALLOC_MIN_SIZE ARCH_DMA_MINALIGN > -#define KMALLOC_SHIFT_LOW ilog2(ARCH_DMA_MINALIGN) > #else > #define ARCH_KMALLOC_MINALIGN __alignof__(unsigned long long) > #endif > @@ -310,6 +308,11 @@ static inline unsigned int arch_slab_minalign(void) > #define KMALLOC_MIN_SIZE (1 << KMALLOC_SHIFT_LOW) > #endif > > +/* > + * This alignment should be at least sizeof(unsigned long long). > + */ > +#define KMALLOC_PACKED_ALIGN (KMALLOC_MIN_SIZE) > + I think __assume_kmalloc_alignment should be changed as well, to avoid compiler making wrong decision. > /* > * This restriction comes from byte sized index implementation. > * Page size is normally 2^12 bytes and, in this case, if we want to use > @@ -382,6 +385,17 @@ static __always_inline enum kmalloc_cache_type kmalloc_type(gfp_t flags) > return KMALLOC_CGROUP; > } > > +/* > + * Align the size to ARCH_KMALLOC_MINALIGN unless __GFP_PACKED is passed. > + */ > +static __always_inline size_t kmalloc_size_align(size_t size, gfp_t flags) > +{ > + if (ARCH_KMALLOC_MINALIGN > KMALLOC_PACKED_ALIGN && > + !(flags & __GFP_PACKED)) > + size = ALIGN(size, ARCH_KMALLOC_MINALIGN); > + return size; > +} > + > /* > * Figure out which kmalloc slab an allocation of a certain size > * belongs to. > @@ -568,7 +582,7 @@ static __always_inline __alloc_size(1) void *kmalloc(size_t size, gfp_t flags) > if (size > KMALLOC_MAX_CACHE_SIZE) > return kmalloc_large(size, flags); > #ifndef CONFIG_SLOB > - index = kmalloc_index(size); > + index = kmalloc_index(kmalloc_size_align(size, flags)); > > if (!index) > return ZERO_SIZE_PTR; > @@ -590,7 +604,7 @@ static __always_inline __alloc_size(1) void *kmalloc_node(size_t size, gfp_t fla > if (size > KMALLOC_MAX_CACHE_SIZE) > return kmalloc_large_node(size, flags, node); > > - index = kmalloc_index(size); > + index = kmalloc_index(kmalloc_size_align(size, flags)); > > if (!index) > return ZERO_SIZE_PTR; > > diff --git a/mm/slab_common.c b/mm/slab_common.c > index 33b1886b06eb..0e4ea396cd4f 100644 > --- a/mm/slab_common.c > +++ b/mm/slab_common.c > @@ -627,7 +627,7 @@ void __init create_boot_cache(struct kmem_cache *s, const char *name, > unsigned int useroffset, unsigned int usersize) > { > int err; > - unsigned int align = ARCH_KMALLOC_MINALIGN; > + unsigned int align = KMALLOC_PACKED_ALIGN; > > s->name = name; > s->size = s->object_size = size; > @@ -720,6 +720,7 @@ struct kmem_cache *kmalloc_slab(size_t size, gfp_t flags) > { > unsigned int index; > > + size = kmalloc_size_align(size, flags); > > > if (size <= 192) { > if (!size) > return ZERO_SIZE_PTR; > -- Thanks, Hyeonggon _______________________________________________ linux-arm-kernel mailing list linux-arm-kernel@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-arm-kernel