From: Hyeonggon Yoo <42.hyeyoo@gmail.com>
To: John Thomson <lists@johnthomson.fastmail.com.au>
Cc: Feng Tang <feng.tang@intel.com>, Vlastimil Babka <vbabka@suse.cz>,
Andrew Morton <akpm@linux-foundation.org>,
Christoph Lameter <cl@linux.com>,
Pekka Enberg <penberg@kernel.org>,
David Rientjes <rientjes@google.com>,
Joonsoo Kim <iamjoonsoo.kim@lge.com>,
Roman Gushchin <roman.gushchin@linux.dev>,
Dmitry Vyukov <dvyukov@google.com>,
Jonathan Corbet <corbet@lwn.net>,
Andrey Konovalov <andreyknvl@gmail.com>,
"Hansen, Dave" <dave.hansen@intel.com>,
"linux-mm@kvack.org" <linux-mm@kvack.org>,
"linux-kernel@vger.kernel.org" <linux-kernel@vger.kernel.org>,
"kasan-dev@googlegroups.com" <kasan-dev@googlegroups.com>,
Robin Murphy <robin.murphy@arm.com>,
John Garry <john.garry@huawei.com>,
Kefeng Wang <wangkefeng.wang@huawei.com>,
Thomas Bogendoerfer <tsbogend@alpha.franken.de>,
John Crispin <john@phrozen.org>,
Matthias Brugger <matthias.bgg@gmail.com>,
linux-mips@vger.kernel.org
Subject: Re: [PATCH v6 1/4] mm/slub: enable debugging memory wasting of kmalloc
Date: Tue, 1 Nov 2022 19:42:23 +0900 [thread overview]
Message-ID: <Y2D4D52h5VVa8QpE@hyeyoo> (raw)
In-Reply-To: <29271a2b-cf19-4af9-bfe5-5bcff8a23fda@app.fastmail.com>
On Tue, Nov 01, 2022 at 10:33:32AM +0000, John Thomson wrote:
> On Tue, 1 Nov 2022, at 09:31, Hyeonggon Yoo wrote:
> > On Tue, Nov 01, 2022 at 09:20:21AM +0000, John Thomson wrote:
> >> On Tue, 1 Nov 2022, at 07:57, Feng Tang wrote:
> >> > Hi Thomson,
> >> >
> >> > Thanks for testing!
> >> >
> >> > + mips maintainer and mail list. The original report is here
> >> >
> >> > https://lore.kernel.org/lkml/becf2ac3-2a90-4f3a-96d9-a70f67c66e4a@app.fastmail.com/
> >>
> >> I am guessing my issue comes from __kmem_cache_alloc_lru accessing s->object_size when (kmem_cache) s is NULL?
> >> If that is the case, this change is not to blame, it only exposes the issue?
> >>
> >> I get the following dmesg (note very early NULL kmem_cache) with the below change atop v6.1-rc3:
> >>
> >> transfer started ......................................... transfer ok, time=2.02s
> >> setting up elf image... OK
> >> jumping to kernel code
> >> zimage at: 80B842A0 810B4EFC
> >>
> >> Uncompressing Linux at load address 80001000
> >>
> >> Copy device tree to address 80B80EE0
> >>
> >> Now, booting the kernel...
> >>
> >> [ 0.000000] Linux version 6.1.0-rc3+ (john@john) (mipsel-buildroot-linux-gnu-gcc.br_real (Buildroot 2021.11-4428-g6b6741b) 12.2.0, GNU ld (GNU Binutils) 2.39) #61 SMP Tue Nov 1 18:04:13 AEST 2022
> >> [ 0.000000] slub: kmem_cache_alloc called with kmem_cache: 0x0
> >> [ 0.000000] slub: __kmem_cache_alloc_lru called with kmem_cache: 0x0
> >> [ 0.000000] SoC Type: MediaTek MT7621 ver:1 eco:3
> >> [ 0.000000] printk: bootconsole [early0] enabled
> >> [ 0.000000] CPU0 revision is: 0001992f (MIPS 1004Kc)
> >> [ 0.000000] MIPS: machine is MikroTik RouterBOARD 760iGS
> >>
> >> normal boot
> >>
> >>
> >> diff --git a/mm/slub.c b/mm/slub.c
> >> index 157527d7101b..10fcdf2520d2 100644
> >> --- a/mm/slub.c
> >> +++ b/mm/slub.c
> >> @@ -3410,7 +3410,13 @@ static __always_inline
> >> void *__kmem_cache_alloc_lru(struct kmem_cache *s, struct list_lru *lru,
> >> gfp_t gfpflags)
> >> {
> >> - void *ret = slab_alloc(s, lru, gfpflags, _RET_IP_, s->object_size);
> >> + void *ret;
> >> + if (IS_ERR_OR_NULL(s)) {
> >> + pr_warn("slub: __kmem_cache_alloc_lru called with kmem_cache: %pSR\n", s);
> >> + ret = slab_alloc(s, lru, gfpflags, _RET_IP_, 0);
> >> + } else {
> >> + ret = slab_alloc(s, lru, gfpflags, _RET_IP_, s->object_size);
> >> + }
> >>
> >> trace_kmem_cache_alloc(_RET_IP_, ret, s, gfpflags, NUMA_NO_NODE);
> >>
> >> @@ -3419,6 +3425,8 @@ void *__kmem_cache_alloc_lru(struct kmem_cache *s, struct list_lru *lru,
> >>
> >> void *kmem_cache_alloc(struct kmem_cache *s, gfp_t gfpflags)
> >> {
> >> + if (IS_ERR_OR_NULL(s))
> >> + pr_warn("slub: kmem_cache_alloc called with kmem_cache: %pSR\n", s);
> >> return __kmem_cache_alloc_lru(s, NULL, gfpflags);
> >> }
> >> EXPORT_SYMBOL(kmem_cache_alloc);
> >> @@ -3426,6 +3434,8 @@ EXPORT_SYMBOL(kmem_cache_alloc);
> >> void *kmem_cache_alloc_lru(struct kmem_cache *s, struct list_lru *lru,
> >> gfp_t gfpflags)
> >> {
> >> + if (IS_ERR_OR_NULL(s))
> >> + pr_warn("slub: __kmem_cache_alloc_lru called with kmem_cache: %pSR\n", s);
> >> return __kmem_cache_alloc_lru(s, lru, gfpflags);
> >> }
> >> EXPORT_SYMBOL(kmem_cache_alloc_lru);
> >>
> >>
> >> Any hints on where kmem_cache_alloc would be being called from this early?
> >> I will start looking from /init/main.c around pr_notice("%s", linux_banner);
> >
> > Great. Would you try calling dump_stack(); when we observed s == NULL?
> > That would give more information about who passed s == NULL to these
> > functions.
> >
>
> With the dump_stack() in place:
>
> Now, booting the kernel...
>
> [ 0.000000] Linux version 6.1.0-rc3+ (john@john) (mipsel-buildroot-linux-gnu-gcc.br_real (Buildroot 2021.11-4428-g6b6741b) 12.2.0, GNU ld (GNU Binutils) 2.39) #62 SMP Tue Nov 1 19:49:52 AEST 2022
> [ 0.000000] slub: __kmem_cache_alloc_lru called with kmem_cache ptr: 0x0
> [ 0.000000] CPU: 0 PID: 0 Comm: swapper Not tainted 6.1.0-rc3+ #62
> [ 0.000000] Stack : 810fff78 80084d98 80889d00 00000004 00000000 00000000 80889d5c 80c90000
> [ 0.000000] 80920000 807bd380 8089d368 80923bd3 00000000 00000001 80889d08 00000000
> [ 0.000000] 00000000 00000000 807bd380 8084bd51 00000002 00000002 00000001 6d6f4320
> [ 0.000000] 00000000 80c97ce9 80c97d14 fffffffc 807bd380 00000000 00000003 00000dc0
> [ 0.000000] 00000000 a0000000 80910000 8110a0b4 00000000 00000020 80010000 80010000
> [ 0.000000] ...
> [ 0.000000] Call Trace:
> [ 0.000000] [<80008260>] show_stack+0x28/0xf0
> [ 0.000000] [<8070cdc0>] dump_stack_lvl+0x60/0x80
> [ 0.000000] [<801c1428>] kmem_cache_alloc+0x5c0/0x740
> [ 0.000000] [<8092856c>] prom_soc_init+0x1fc/0x2b4
> [ 0.000000] [<80928060>] prom_init+0x44/0xf0
> [ 0.000000] [<80929214>] setup_arch+0x4c/0x6a8
> [ 0.000000] [<809257e0>] start_kernel+0x88/0x7c0
> [ 0.000000]
> [ 0.000000] SoC Type: MediaTek MT7621 ver:1 eco:3
setup_arch() is too early to use slab allocators.
I think slab received NULL pointer because kmalloc is not initialized.
It seems arch/mips/ralink/mt7621.c is using slab too early.
>
>
> Now, booting the kernel...
>
> [ 0.000000] Linux version 6.1.0-rc3+ (john@john) (mipsel-buildroot-linux-gnu-gcc.br_real (Buildroot 2021.11-4428-g6b6741b) 12.2.0, GNU ld (GNU Binutils) 2.39) #62 SMP Tue Nov 1 19:49:52 AEST 2022
> [ 0.000000] slub: __kmem_cache_alloc_lru called with kmem_cache ptr: 0x0
> [ 0.000000] CPU: 0 PID: 0 Comm: swapper Not tainted 6.1.0-rc3+ #62
> [ 0.000000] Stack : 810fff78 80084d98 80889d00 00000004 00000000 00000000 80889d5c 80c90000
> [ 0.000000] 80920000 807bd380 8089d368 80923bd3 00000000 00000001 80889d08 00000000
> [ 0.000000] 00000000 00000000 807bd380 8084bd51 00000002 00000002 00000001 6d6f4320
> [ 0.000000] 00000000 80c97ce9 80c97d14 fffffffc 807bd380 00000000 00000003 00000dc0
> [ 0.000000] 00000000 a0000000 80910000 8110a0b4 00000000 00000020 80010000 80010000
> [ 0.000000] ...
> [ 0.000000] Call Trace:
> [ 0.000000] show_stack (/mnt/pool_ssd/code/linux/linux-stable-mt7621/./arch/mips/include/asm/stacktrace.h:43 /mnt/pool_ssd/code/linux/linux-stable-mt7621/arch/mips/kernel/traps.c:223)
> [ 0.000000] dump_stack_lvl (/mnt/pool_ssd/code/linux/linux-stable-mt7621/lib/dump_stack.c:107 (discriminator 1))
> [ 0.000000] kmem_cache_alloc (/mnt/pool_ssd/code/linux/linux-stable-mt7621/mm/slub.c:3318 /mnt/pool_ssd/code/linux/linux-stable-mt7621/mm/slub.c:3406 /mnt/pool_ssd/code/linux/linux-stable-mt7621/mm/slub.c:3418 /mnt/pool_ssd/code/linux/linux-stable-mt7621/mm/slub.c:3430)
> [ 0.000000] prom_soc_init (/mnt/pool_ssd/code/linux/linux-stable-mt7621/arch/mips/ralink/mt7621.c:106 /mnt/pool_ssd/code/linux/linux-stable-mt7621/arch/mips/ralink/mt7621.c:177)
> [ 0.000000] prom_init (/mnt/pool_ssd/code/linux/linux-stable-mt7621/arch/mips/ralink/prom.c:64)
> [ 0.000000] setup_arch (/mnt/pool_ssd/code/linux/linux-stable-mt7621/arch/mips/kernel/setup.c:786)
> [ 0.000000] start_kernel (/mnt/pool_ssd/code/linux/linux-stable-mt7621/init/main.c:279 /mnt/pool_ssd/code/linux/linux-stable-mt7621/init/main.c:477 /mnt/pool_ssd/code/linux/linux-stable-mt7621/init/main.c:960)
> [ 0.000000]
> [ 0.000000] SoC Type: MediaTek MT7621 ver:1 eco:3
>
>
> I have not found it yet.
>
>
> Cheers,
> --
> John Thomson
--
Thanks,
Hyeonggon
next prev parent reply other threads:[~2022-11-01 10:42 UTC|newest]
Thread overview: 48+ messages / expand[flat|nested] mbox.gz Atom feed top
2022-09-13 6:54 [PATCH v6 0/4] mm/slub: some debug enhancements for kmalloc Feng Tang
2022-09-13 6:54 ` [PATCH v6 1/4] mm/slub: enable debugging memory wasting of kmalloc Feng Tang
2022-09-23 11:43 ` Vlastimil Babka
2022-09-24 7:08 ` Feng Tang
2022-10-30 19:23 ` John Thomson
2022-10-30 21:30 ` Vlastimil Babka
2022-10-31 2:36 ` Feng Tang
2022-10-31 10:05 ` John Thomson
2022-10-31 11:36 ` Hyeonggon Yoo
2022-10-31 11:42 ` Feng Tang
2022-11-01 0:18 ` John Thomson
2022-11-01 2:41 ` John Thomson
2022-11-01 7:57 ` Feng Tang
2022-11-01 9:20 ` John Thomson
2022-11-01 9:31 ` Hyeonggon Yoo
2022-11-01 10:33 ` John Thomson
2022-11-01 10:42 ` Hyeonggon Yoo [this message]
2022-11-01 13:55 ` Feng Tang
2022-11-01 19:39 ` John Thomson
2022-11-02 6:08 ` Feng Tang
2022-11-02 7:16 ` Hyeonggon Yoo
2022-11-03 7:18 ` Feng Tang
2022-11-03 7:45 ` John Thomson
2022-11-03 8:16 ` Feng Tang
2022-11-02 8:22 ` Vlastimil Babka
2022-11-03 5:54 ` Feng Tang
2022-11-03 8:33 ` Vlastimil Babka
2022-11-03 14:16 ` Feng Tang
2022-11-03 14:36 ` Hyeonggon Yoo
2022-11-03 16:57 ` Vlastimil Babka
2022-11-03 17:35 ` Vlastimil Babka
2022-11-04 3:52 ` Feng Tang
2022-09-13 6:54 ` [PATCH v6 2/4] mm/slub: only zero the requested size of buffer for kzalloc Feng Tang
2022-09-26 19:11 ` Andrey Konovalov
2022-09-26 20:15 ` Kees Cook
2022-09-27 1:22 ` Feng Tang
2022-09-27 2:42 ` Feng Tang
2022-10-13 14:00 ` Andrey Konovalov
2022-10-14 5:59 ` Feng Tang
2022-09-13 6:54 ` [PATCH v6 3/4] mm: kasan: Add free_meta size info in struct kasan_cache Feng Tang
2022-09-20 19:20 ` Andrey Konovalov
2022-09-21 12:02 ` Feng Tang
2022-09-24 18:05 ` Andrey Konovalov
2022-09-25 11:26 ` Feng Tang
2022-09-25 16:31 ` Andrey Konovalov
2022-09-27 3:03 ` Feng Tang
2022-09-13 6:54 ` [PATCH v6 4/4] mm/slub: extend redzone check to extra allocated kmalloc space than requested Feng Tang
2022-09-13 8:53 ` Hyeonggon Yoo
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=Y2D4D52h5VVa8QpE@hyeyoo \
--to=42.hyeyoo@gmail.com \
--cc=akpm@linux-foundation.org \
--cc=andreyknvl@gmail.com \
--cc=cl@linux.com \
--cc=corbet@lwn.net \
--cc=dave.hansen@intel.com \
--cc=dvyukov@google.com \
--cc=feng.tang@intel.com \
--cc=iamjoonsoo.kim@lge.com \
--cc=john.garry@huawei.com \
--cc=john@phrozen.org \
--cc=kasan-dev@googlegroups.com \
--cc=linux-kernel@vger.kernel.org \
--cc=linux-mips@vger.kernel.org \
--cc=linux-mm@kvack.org \
--cc=lists@johnthomson.fastmail.com.au \
--cc=matthias.bgg@gmail.com \
--cc=penberg@kernel.org \
--cc=rientjes@google.com \
--cc=robin.murphy@arm.com \
--cc=roman.gushchin@linux.dev \
--cc=tsbogend@alpha.franken.de \
--cc=vbabka@suse.cz \
--cc=wangkefeng.wang@huawei.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.