* [PATCH 0/2] SLUB memory debugging improvements. @ 2011-06-27 23:17 greearb 2011-06-27 23:17 ` [PATCH 1/2] slub: Enable backtrace for create/delete points greearb 2011-06-27 23:17 ` [PATCH 2/2] slub: Add method to verify memory is not deleted greearb 0 siblings, 2 replies; 9+ messages in thread From: greearb @ 2011-06-27 23:17 UTC (permalink / raw) To: linux-kernel; +Cc: Ben Greear From: Ben Greear <greearb@candelatech.com> The idea is to grab a useful sized stack trace for creation/deletion points instead of just a single method. A second patch adds a command to check if memory should be in use. This is in an attempt to track down some use-after-free bugs I'm seeing in nfs/rpc. However, I did not exactly hit the bug I was expecting, but got this output instead. I'm not sure if my debug patches are not working right, or if this warning is just another symptom of the problem: ============================================================================= BUG kmalloc-64: Wrong object count. Counter is 2 but counted were 1 ----------------------------------------------------------------------------- INFO: Slab 0xffffea0001e19fd0 objects=20 used=20 fp=0x (null) flags=0x20000000004080 Pid: 2349, comm: kworker/1:2 Not tainted 3.0.0-rc4+ #4 Call Trace: [<ffffffff81105484>] slab_err+0x8a/0x98 [<ffffffff81077c0b>] ? register_lock_class+0x1e/0x336 [<ffffffff81078c91>] ? mark_lock+0x2d/0x22d [<ffffffff81105783>] on_freelist+0x179/0x1ad [<ffffffffa0290601>] ? __rpc_execute+0x1ce/0x24b [sunrpc] [<ffffffff81106849>] verify_mem_not_deleted+0x5f/0xb3 [<ffffffffa02906c0>] ? rpc_execute+0x42/0x42 [sunrpc] [<ffffffffa02981b5>] rpcb_getport_done+0x23/0x126 [sunrpc] [<ffffffffa02900ba>] rpc_exit_task+0x3f/0x6d [sunrpc] [<ffffffffa02904ab>] __rpc_execute+0x78/0x24b [sunrpc] [<ffffffffa02906c0>] ? rpc_execute+0x42/0x42 [sunrpc] [<ffffffffa02906d0>] rpc_async_schedule+0x10/0x12 [sunrpc] [<ffffffff810611b7>] process_one_work+0x230/0x41d [<ffffffff81061102>] ? process_one_work+0x17b/0x41d [<ffffffff81063613>] worker_thread+0x133/0x217 [<ffffffff810634e0>] ? manage_workers+0x191/0x191 [<ffffffff81066e10>] kthread+0x7d/0x85 [<ffffffff81485924>] kernel_thread_helper+0x4/0x10 [<ffffffff8147eb18>] ? retint_restore_args+0x13/0x13 [<ffffffff81066d93>] ? __init_kthread_worker+0x56/0x56 [<ffffffff81485920>] ? gs_change+0x13/0x13 FIX kmalloc-64: Object count adjusted. ============================================================================= BUG kmalloc-64: Wrong object count. Counter is 1 but counted were 20 ----------------------------------------------------------------------------- INFO: Slab 0xffffea0001e19fd0 objects=20 used=1 fp=0x (null) flags=0x20000000004081 Pid: 2349, comm: kworker/1:2 Not tainted 3.0.0-rc4+ #4 Call Trace: [<ffffffff81105484>] slab_err+0x8a/0x98 [<ffffffff81078c91>] ? mark_lock+0x2d/0x22d [<ffffffff8147e4c2>] ? _raw_spin_unlock_irqrestore+0x42/0x79 [<ffffffff81041b91>] ? get_parent_ip+0x11/0x41 [<ffffffff81105783>] on_freelist+0x179/0x1ad [<ffffffffa029889c>] ? rpcb_map_release+0x3f/0x44 [sunrpc] [<ffffffffa02906c0>] ? rpc_execute+0x42/0x42 [sunrpc] [<ffffffff81106d4b>] free_debug_processing+0x5c/0x18e [<ffffffffa02906c0>] ? rpc_execute+0x42/0x42 [sunrpc] [<ffffffff811073b0>] __slab_free+0x57/0x150 [<ffffffffa029889c>] ? rpcb_map_release+0x3f/0x44 [sunrpc] [<ffffffffa029889c>] ? rpcb_map_release+0x3f/0x44 [sunrpc] [<ffffffff81107e9c>] kfree+0x107/0x13a [<ffffffffa029889c>] rpcb_map_release+0x3f/0x44 [sunrpc] [<ffffffffa028fa5a>] rpc_release_calldata+0x12/0x14 [sunrpc] [<ffffffffa028fc41>] rpc_free_task+0x59/0x61 [sunrpc] [<ffffffffa028fccb>] rpc_final_put_task+0x82/0x8a [sunrpc] [<ffffffffa029066f>] __rpc_execute+0x23c/0x24b [sunrpc] [<ffffffffa02906c0>] ? rpc_execute+0x42/0x42 [sunrpc] [<ffffffffa02906d0>] rpc_async_schedule+0x10/0x12 [sunrpc] [<ffffffff810611b7>] process_one_work+0x230/0x41d [<ffffffff81061102>] ? process_one_work+0x17b/0x41d [<ffffffff81063613>] worker_thread+0x133/0x217 [<ffffffff810634e0>] ? manage_workers+0x191/0x191 [<ffffffff81066e10>] kthread+0x7d/0x85 [<ffffffff81485924>] kernel_thread_helper+0x4/0x10 [<ffffffff8147eb18>] ? retint_restore_args+0x13/0x13 [<ffffffff81066d93>] ? __init_kthread_worker+0x56/0x56 [<ffffffff81485920>] ? gs_change+0x13/0x13 FIX kmalloc-64: Object count adjusted. Ben Greear (2): slub: Enable backtrace for create/delete points. slub: Add method to verify memory is not deleted. include/linux/slab.h | 9 ++++++ mm/slub.c | 73 +++++++++++++++++++++++++++++++++++++++++++++----- 2 files changed, 75 insertions(+), 7 deletions(-) -- 1.7.3.4 ^ permalink raw reply [flat|nested] 9+ messages in thread
* [PATCH 1/2] slub: Enable backtrace for create/delete points. 2011-06-27 23:17 [PATCH 0/2] SLUB memory debugging improvements greearb @ 2011-06-27 23:17 ` greearb 2011-06-27 23:17 ` [PATCH 2/2] slub: Add method to verify memory is not deleted greearb 1 sibling, 0 replies; 9+ messages in thread From: greearb @ 2011-06-27 23:17 UTC (permalink / raw) To: linux-kernel; +Cc: Ben Greear From: Ben Greear <greearb@candelatech.com> This patch attempts to grab a backtrace for the creation and deletion points of the slub object. When a fault is detected, we can then get a better idea of where the item was deleted. Signed-off-by: Ben Greear <greearb@candelatech.com> --- :100644 100644 35f351f... 14d0135... M mm/slub.c mm/slub.c | 39 ++++++++++++++++++++++++++++++++------- 1 files changed, 32 insertions(+), 7 deletions(-) diff --git a/mm/slub.c b/mm/slub.c index 35f351f..14d0135 100644 --- a/mm/slub.c +++ b/mm/slub.c @@ -191,8 +191,10 @@ static LIST_HEAD(slab_caches); /* * Tracking user of a slab. */ +#define TRACK_ADDRS_COUNT 16 struct track { - unsigned long addr; /* Called from address */ + unsigned long caddr; + unsigned long addrs[TRACK_ADDRS_COUNT]; /* Called from address */ int cpu; /* Was running on cpu */ int pid; /* Pid context */ unsigned long when; /* When did the operation occur */ @@ -420,7 +422,24 @@ static void set_track(struct kmem_cache *s, void *object, struct track *p = get_track(s, object, alloc); if (addr) { - p->addr = addr; + struct stack_trace trace; + int i; + + trace.nr_entries = 0; + trace.max_entries = TRACK_ADDRS_COUNT; + trace.entries = p->addrs; + trace.skip = 3; + save_stack_trace(&trace); + + /* See rant in lockdep.c */ + if (trace.nr_entries != 0 && + trace.entries[trace.nr_entries - 1] == ULONG_MAX) + trace.nr_entries--; + + for (i = trace.nr_entries; i<TRACK_ADDRS_COUNT; i++) + p->addrs[i] = 0; + + p->caddr = addr; p->cpu = smp_processor_id(); p->pid = current->pid; p->when = jiffies; @@ -439,11 +458,17 @@ static void init_tracking(struct kmem_cache *s, void *object) static void print_track(const char *s, struct track *t) { - if (!t->addr) + int i; + if (!t->caddr) return; printk(KERN_ERR "INFO: %s in %pS age=%lu cpu=%u pid=%d\n", - s, (void *)t->addr, jiffies - t->when, t->cpu, t->pid); + s, (void *)t->caddr, jiffies - t->when, t->cpu, t->pid); + for (i = 0; i < TRACK_ADDRS_COUNT; i++) + if (t->addrs[i]) + printk(KERN_ERR "\t%pS\n", (void *)t->addrs[i]); + else + break; } static void print_tracking(struct kmem_cache *s, void *object) @@ -3721,7 +3746,7 @@ static int add_location(struct loc_track *t, struct kmem_cache *s, break; caddr = t->loc[pos].addr; - if (track->addr == caddr) { + if (track->caddr == caddr) { l = &t->loc[pos]; l->count++; @@ -3744,7 +3769,7 @@ static int add_location(struct loc_track *t, struct kmem_cache *s, return 1; } - if (track->addr < caddr) + if (track->caddr < caddr) end = pos; else start = pos; @@ -3762,7 +3787,7 @@ static int add_location(struct loc_track *t, struct kmem_cache *s, (t->count - pos) * sizeof(struct location)); t->count++; l->count = 1; - l->addr = track->addr; + l->addr = track->caddr; l->sum_time = age; l->min_time = age; l->max_time = age; -- 1.7.3.4 ^ permalink raw reply related [flat|nested] 9+ messages in thread
* [PATCH 2/2] slub: Add method to verify memory is not deleted. 2011-06-27 23:17 [PATCH 0/2] SLUB memory debugging improvements greearb 2011-06-27 23:17 ` [PATCH 1/2] slub: Enable backtrace for create/delete points greearb @ 2011-06-27 23:17 ` greearb 2011-06-27 23:28 ` David Rientjes 1 sibling, 1 reply; 9+ messages in thread From: greearb @ 2011-06-27 23:17 UTC (permalink / raw) To: linux-kernel; +Cc: Ben Greear From: Ben Greear <greearb@candelatech.com> This is for tracking down suspect memory usage. Signed-off-by: Ben Greear <greearb@candelatech.com> --- :100644 100644 ad4dd1c... 35a13d2... M include/linux/slab.h :100644 100644 14d0135... b3d7680... M mm/slub.c include/linux/slab.h | 9 +++++++++ mm/slub.c | 34 ++++++++++++++++++++++++++++++++++ 2 files changed, 43 insertions(+), 0 deletions(-) diff --git a/include/linux/slab.h b/include/linux/slab.h index ad4dd1c..35a13d2 100644 --- a/include/linux/slab.h +++ b/include/linux/slab.h @@ -302,6 +302,15 @@ extern void *__kmalloc_node_track_caller(size_t, gfp_t, int, unsigned long); #endif /* CONFIG_NUMA */ +/** Calling this on deleted objects will print some + * SLUB debugging information. + */ +#if defined(CONFIG_SLUB) && defined(CONFIG_SLUB_DEBUG) +extern bool verify_mem_not_deleted(const void *x); +#else +#define verify_mem_not_deleted(x) +#endif + /* * Shortcuts */ diff --git a/mm/slub.c b/mm/slub.c index 14d0135..b3d7680 100644 --- a/mm/slub.c +++ b/mm/slub.c @@ -2953,6 +2953,40 @@ size_t ksize(const void *object) } EXPORT_SYMBOL(ksize); +#if defined(CONFIG_SLUB_DEBUG) +bool verify_mem_not_deleted(const void *x) +{ + struct page *page; + void *object = (void *)x; + struct kmem_cache *s; + unsigned long flags; + bool rv = false; + + if (unlikely(ZERO_OR_NULL_PTR(x))) + false; + + local_irq_save(flags); + + page = virt_to_head_page(x); + if (unlikely(!PageSlab(page))) { + BUG_ON(!PageCompound(page)); + goto out_unlock; + } + + s = page->slab; + + if (on_freelist(s, page, object)) { + object_err(s, page, object, "Object is on fee-list"); + goto out_unlock; + } + rv = true; +out_unlock: + local_irq_restore(flags); + return rv; +} +EXPORT_SYMBOL(verify_mem_not_deleted); +#endif + void kfree(const void *x) { struct page *page; -- 1.7.3.4 ^ permalink raw reply related [flat|nested] 9+ messages in thread
* Re: [PATCH 2/2] slub: Add method to verify memory is not deleted. 2011-06-27 23:17 ` [PATCH 2/2] slub: Add method to verify memory is not deleted greearb @ 2011-06-27 23:28 ` David Rientjes 2011-06-27 23:46 ` Ben Greear 0 siblings, 1 reply; 9+ messages in thread From: David Rientjes @ 2011-06-27 23:28 UTC (permalink / raw) To: Ben Greear; +Cc: linux-kernel, Pekka Enberg, Christoph Lameter On Mon, 27 Jun 2011, greearb@candelatech.com wrote: > From: Ben Greear <greearb@candelatech.com> > > This is for tracking down suspect memory usage. > Several things wrong with this: - I have no idea where patch 1/2 is. - the subject line is ambiguous, when you say memory is "deleted," I thought at first you were talking about hot-removed, but it seems like you're talking about "freed." - what "suspect memory usage" are you adding functionality to catch and how are you doing it? - you didn't cc the slab maintainers, Pekka Enberg and Christoph Lameter (I added them). > Signed-off-by: Ben Greear <greearb@candelatech.com> > --- > :100644 100644 ad4dd1c... 35a13d2... M include/linux/slab.h > :100644 100644 14d0135... b3d7680... M mm/slub.c > include/linux/slab.h | 9 +++++++++ > mm/slub.c | 34 ++++++++++++++++++++++++++++++++++ > 2 files changed, 43 insertions(+), 0 deletions(-) > > diff --git a/include/linux/slab.h b/include/linux/slab.h > index ad4dd1c..35a13d2 100644 > --- a/include/linux/slab.h > +++ b/include/linux/slab.h > @@ -302,6 +302,15 @@ extern void *__kmalloc_node_track_caller(size_t, gfp_t, int, unsigned long); > > #endif /* CONFIG_NUMA */ > > +/** Calling this on deleted objects will print some > + * SLUB debugging information. > + */ Ambiguous as to what it will be printing and violation of the comment style used in the kernel (see Documentation/CodingStyle). > +#if defined(CONFIG_SLUB) && defined(CONFIG_SLUB_DEBUG) > +extern bool verify_mem_not_deleted(const void *x); > +#else > +#define verify_mem_not_deleted(x) This will surely break if anybody isn't using slub or CONFIG_SLUB_DEBUG and it testing verify_mem_not_deleted(). You probably want static inline bool verify_mem_not_deleted(const void *x) { return false; } > +#endif > + > /* > * Shortcuts > */ > diff --git a/mm/slub.c b/mm/slub.c > index 14d0135..b3d7680 100644 > --- a/mm/slub.c > +++ b/mm/slub.c > @@ -2953,6 +2953,40 @@ size_t ksize(const void *object) > } > EXPORT_SYMBOL(ksize); > > +#if defined(CONFIG_SLUB_DEBUG) #ifdef CONFIG_SLUB_DEBUG > +bool verify_mem_not_deleted(const void *x) > +{ > + struct page *page; > + void *object = (void *)x; > + struct kmem_cache *s; > + unsigned long flags; > + bool rv = false; > + > + if (unlikely(ZERO_OR_NULL_PTR(x))) > + false; > + Did you even compile-test this? > + local_irq_save(flags); > + > + page = virt_to_head_page(x); > + if (unlikely(!PageSlab(page))) { > + BUG_ON(!PageCompound(page)); Why is there a BUG_ON() here if we didn't pass a pointer to a slab object? > + goto out_unlock; > + } > + > + s = page->slab; > + > + if (on_freelist(s, page, object)) { > + object_err(s, page, object, "Object is on fee-list"); s/fee/free/ > + goto out_unlock; > + } > + rv = true; > +out_unlock: > + local_irq_restore(flags); > + return rv; > +} > +EXPORT_SYMBOL(verify_mem_not_deleted); > +#endif > + ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: [PATCH 2/2] slub: Add method to verify memory is not deleted. 2011-06-27 23:28 ` David Rientjes @ 2011-06-27 23:46 ` Ben Greear 2011-06-28 0:18 ` Ben Greear 2011-06-28 0:19 ` David Rientjes 0 siblings, 2 replies; 9+ messages in thread From: Ben Greear @ 2011-06-27 23:46 UTC (permalink / raw) To: David Rientjes; +Cc: linux-kernel, Pekka Enberg, Christoph Lameter On 06/27/2011 04:28 PM, David Rientjes wrote: > On Mon, 27 Jun 2011, greearb@candelatech.com wrote: > >> From: Ben Greear<greearb@candelatech.com> >> >> This is for tracking down suspect memory usage. >> > > Several things wrong with this: > > - I have no idea where patch 1/2 is. It was sent to lkml...should show up soon. > - the subject line is ambiguous, when you say memory is "deleted," I > thought at first you were talking about hot-removed, but it seems like > you're talking about "freed." Ok, I can fix that. I'm talking about memory that was freed. > > - what "suspect memory usage" are you adding functionality to catch and > how are you doing it? I saw a case where xprt was 0x6b6b6b6b. I'm trying to figure out what freed it. I am not certain if this is a bug I introduced in some nfs changes I am testing, or if it is in a standard kernel. I am unable to reproduce it in standard kernel, but that is partly because my test case depends on my nfs patches. The code below contains my debugging hacks, definitely not for kernel inclusion as is. /* * Rpcbind child task calls this callback via tk_exit. */ static void rpcb_getport_done(struct rpc_task *child, void *data) { struct rpcbind_args *map = data; struct rpc_xprt *xprt = map->r_xprt; int status = child->tk_status; verify_mem_not_deleted(map); verify_mem_not_deleted(xprt); BUG_ON((unsigned int)(map) == (unsigned int)(0x6b6b6b6b)); if ((unsigned int)(xprt) == (unsigned int)(0x6b6b6b6b)) { printk("xprt: %p is invalid, which means map: %p is likely deleted already.\n", xprt, map); printk("map: r_owner: %p r_prog: %u r_status: %u\n", map->r_owner, map->r_prog, map->r_status); printk("child: %p status: %i\n", child, status); BUG_ON(1); } BUG_ON((unsigned int)(status) == (unsigned int)(0x6b6b6b6b)); > - you didn't cc the slab maintainers, Pekka Enberg and Christoph Lameter > (I added them). Thanks for that. >> +/** Calling this on deleted objects will print some >> + * SLUB debugging information. >> + */ > > Ambiguous as to what it will be printing and violation of the comment > style used in the kernel (see Documentation/CodingStyle). > >> +#if defined(CONFIG_SLUB)&& defined(CONFIG_SLUB_DEBUG) >> +extern bool verify_mem_not_deleted(const void *x); >> +#else >> +#define verify_mem_not_deleted(x) > > This will surely break if anybody isn't using slub or CONFIG_SLUB_DEBUG > and it testing verify_mem_not_deleted(). You probably want > > static inline bool verify_mem_not_deleted(const void *x) > { > return false; > } Ok. > >> +#endif >> + >> /* >> * Shortcuts >> */ >> diff --git a/mm/slub.c b/mm/slub.c >> index 14d0135..b3d7680 100644 >> --- a/mm/slub.c >> +++ b/mm/slub.c >> @@ -2953,6 +2953,40 @@ size_t ksize(const void *object) >> } >> EXPORT_SYMBOL(ksize); >> >> +#if defined(CONFIG_SLUB_DEBUG) > > #ifdef CONFIG_SLUB_DEBUG > >> +bool verify_mem_not_deleted(const void *x) >> +{ >> + struct page *page; >> + void *object = (void *)x; >> + struct kmem_cache *s; >> + unsigned long flags; >> + bool rv = false; >> + >> + if (unlikely(ZERO_OR_NULL_PTR(x))) >> + false; >> + > > Did you even compile-test this? Well yes, but obviously I'm having a bad day! Aside from the lame mistakes in my patch..is the general approach usable? >> + local_irq_save(flags); >> + >> + page = virt_to_head_page(x); >> + if (unlikely(!PageSlab(page))) { >> + BUG_ON(!PageCompound(page)); > > Why is there a BUG_ON() here if we didn't pass a pointer to a slab object? Just copying the kfree code. Just return 'true' in this case? Thanks, Ben -- Ben Greear <greearb@candelatech.com> Candela Technologies Inc http://www.candelatech.com ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: [PATCH 2/2] slub: Add method to verify memory is not deleted. 2011-06-27 23:46 ` Ben Greear @ 2011-06-28 0:18 ` Ben Greear 2011-06-28 0:19 ` David Rientjes 1 sibling, 0 replies; 9+ messages in thread From: Ben Greear @ 2011-06-28 0:18 UTC (permalink / raw) To: David Rientjes; +Cc: linux-kernel, Pekka Enberg, Christoph Lameter On 06/27/2011 04:46 PM, Ben Greear wrote: >> Did you even compile-test this? > > Well yes, but obviously I'm having a bad day! > > Aside from the lame mistakes in my patch..is the general approach > usable? I was missing slab_lock() around the on_freelist() call, that is probably why I got the printouts about object count being off. Fixing that and will let it cook overnight. Thanks, Ben -- Ben Greear <greearb@candelatech.com> Candela Technologies Inc http://www.candelatech.com ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: [PATCH 2/2] slub: Add method to verify memory is not deleted. 2011-06-27 23:46 ` Ben Greear 2011-06-28 0:18 ` Ben Greear @ 2011-06-28 0:19 ` David Rientjes 2011-06-28 0:24 ` Ben Greear 1 sibling, 1 reply; 9+ messages in thread From: David Rientjes @ 2011-06-28 0:19 UTC (permalink / raw) To: Ben Greear; +Cc: linux-kernel, Pekka Enberg, Christoph Lameter On Mon, 27 Jun 2011, Ben Greear wrote: > I saw a case where xprt was 0x6b6b6b6b. I'm trying to figure out > what freed it. And echo 1 > /sys/kernel/slab/<cache>/store_user doesn't help? ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: [PATCH 2/2] slub: Add method to verify memory is not deleted. 2011-06-28 0:19 ` David Rientjes @ 2011-06-28 0:24 ` Ben Greear 2011-06-28 3:45 ` Ben Greear 0 siblings, 1 reply; 9+ messages in thread From: Ben Greear @ 2011-06-28 0:24 UTC (permalink / raw) To: David Rientjes; +Cc: linux-kernel, Pekka Enberg, Christoph Lameter On 06/27/2011 05:19 PM, David Rientjes wrote: > On Mon, 27 Jun 2011, Ben Greear wrote: > >> I saw a case where xprt was 0x6b6b6b6b. I'm trying to figure out >> what freed it. > > And echo 1> /sys/kernel/slab/<cache>/store_user doesn't help? That gives one method, right? The rpc stuff is freed at the bottom of a complicated callback chain, and the interesting stuff is what caused the memory to be freed, not the actually free method. In previous network (ath9k) hacking I had the same trouble..I think for most cases you need a full or mostly full stack to make use of the slub debug logic. Thanks, Ben -- Ben Greear <greearb@candelatech.com> Candela Technologies Inc http://www.candelatech.com ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: [PATCH 2/2] slub: Add method to verify memory is not deleted. 2011-06-28 0:24 ` Ben Greear @ 2011-06-28 3:45 ` Ben Greear 0 siblings, 0 replies; 9+ messages in thread From: Ben Greear @ 2011-06-28 3:45 UTC (permalink / raw) To: David Rientjes; +Cc: linux-kernel, Pekka Enberg, Christoph Lameter On 06/27/2011 05:24 PM, Ben Greear wrote: > On 06/27/2011 05:19 PM, David Rientjes wrote: >> On Mon, 27 Jun 2011, Ben Greear wrote: >> >>> I saw a case where xprt was 0x6b6b6b6b. I'm trying to figure out >>> what freed it. >> >> And echo 1> /sys/kernel/slab/<cache>/store_user doesn't help? > > That gives one method, right? The rpc stuff is freed at the bottom of a > complicated callback chain, and the interesting stuff is what caused the memory > to be freed, not the actually free method. > > In previous network (ath9k) hacking I had the same trouble..I think for most > cases you need a full or mostly full stack to make use of the slub > debug logic. Hit my bug after a while...here's some example output with the slub patches applied: ============================================================================= BUG kmalloc-64: Object is on free-list ----------------------------------------------------------------------------- INFO: Allocated in rpcb_getport_async+0x39c/0x5a5 [sunrpc] age=381 cpu=3 pid=3750 __slab_alloc+0x348/0x3ba kmem_cache_alloc_trace+0x67/0xe7 rpcb_getport_async+0x39c/0x5a5 [sunrpc] call_bind+0x70/0x75 [sunrpc] __rpc_execute+0x78/0x24b [sunrpc] rpc_execute+0x3d/0x42 [sunrpc] rpc_run_task+0x79/0x81 [sunrpc] rpc_call_sync+0x3f/0x60 [sunrpc] rpc_ping+0x42/0x58 [sunrpc] rpc_create+0x4aa/0x527 [sunrpc] nfs_create_rpc_client+0xb1/0xf6 [nfs] nfs_init_client+0x3b/0x7d [nfs] nfs_get_client+0x453/0x5ab [nfs] nfs_create_server+0x10b/0x437 [nfs] nfs_fs_mount+0x4ca/0x708 [nfs] mount_fs+0x6b/0x152 INFO: Freed in rpcb_map_release+0x3f/0x44 [sunrpc] age=30 cpu=2 pid=29049 __slab_free+0x57/0x150 kfree+0x107/0x13a rpcb_map_release+0x3f/0x44 [sunrpc] rpc_release_calldata+0x12/0x14 [sunrpc] rpc_free_task+0x59/0x61 [sunrpc] rpc_final_put_task+0x82/0x8a [sunrpc] __rpc_execute+0x23c/0x24b [sunrpc] rpc_async_schedule+0x10/0x12 [sunrpc] process_one_work+0x230/0x41d worker_thread+0x133/0x217 kthread+0x7d/0x85 kernel_thread_helper+0x4/0x10 INFO: Slab 0xffffea00029aa470 objects=20 used=9 fp=0xffff8800be7830d8 flags=0x20000000004081 INFO: Object 0xffff8800be7830d8 @offset=4312 fp=0xffff8800be7827a8 Bytes b4 0xffff8800be7830c8: 87 a8 96 00 01 00 00 00 5a 5a 5a 5a 5a 5a 5a 5a .�......ZZZZZZZZ Object 0xffff8800be7830d8: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Object 0xffff8800be7830e8: 6b 6b 6b 6b 01 08 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkk..kkkkkkkkkk Object 0xffff8800be7830f8: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Object 0xffff8800be783108: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b a5 kkkkkkkkkkkkkkk� Redzone 0xffff8800be783118: bb bb bb bb bb bb bb bb �������� Padding 0xffff8800be783258: 5a 5a 5a 5a 5a 5a 5a 5a ZZZZZZZZ Pid: 29049, comm: kworker/2:2 Not tainted 3.0.0-rc4+ #8 Call Trace: [<ffffffff811055c3>] print_trailer+0x131/0x13a [<ffffffff81105601>] object_err+0x35/0x3e [<ffffffff8110746f>] verify_mem_not_deleted+0x7a/0xb7 [<ffffffffa02851b5>] rpcb_getport_done+0x23/0x126 [sunrpc] [<ffffffffa027d0ba>] rpc_exit_task+0x3f/0x6d [sunrpc] [<ffffffffa027d4ab>] __rpc_execute+0x78/0x24b [sunrpc] [<ffffffffa027d6c0>] ? rpc_execute+0x42/0x42 [sunrpc] [<ffffffffa027d6d0>] rpc_async_schedule+0x10/0x12 [sunrpc] [<ffffffff810611b7>] process_one_work+0x230/0x41d [<ffffffff81061102>] ? process_one_work+0x17b/0x41d [<ffffffff81063613>] worker_thread+0x133/0x217 [<ffffffff810634e0>] ? manage_workers+0x191/0x191 [<ffffffff81066e10>] kthread+0x7d/0x85 [<ffffffff81485924>] kernel_thread_helper+0x4/0x10 [<ffffffff8147eb18>] ? retint_restore_args+0x13/0x13 [<ffffffff81066d93>] ? __init_kthread_worker+0x56/0x56 [<ffffffff81485920>] ? gs_change+0x13/0x13 general protection fault: 0000 [#1] PREEMPT SMP DEBUG_PAGEALLOC CPU 2 Modules linked in: xt_addrtype xt_TPROXY nf_tproxy_core xt_socket nf_defrag_ipv6 xt_set ip_set nfnetlink xt_connlimit 8021q garp ip6table_filter ip6_tables macvlan ebtable_nat ebtables fuse pktgen iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi stp llc nfs lockd fscache auth_rpcgss nfs_acl sunrpc ipv6 kvm_intel kvm uinput i5k_amb i5000_edac edac_core iTCO_wdt e1000e iTCO_vendor_support ioatdma microcode pcspkr i2c_i801 shpchp dca floppy radeon ttm drm_kms_helper drm hwmon i2c_algo_bit i2c_core [last unloaded: xt_connmark] Pid: 29049, comm: kworker/2:2 Not tainted 3.0.0-rc4+ #8 Supermicro X7DBU/X7DBU RIP: 0010:[<ffffffff81105eae>] [<ffffffff81105eae>] virt_to_head_page+0x1e/0x2c RSP: 0000:ffff8801021d3ce0 EFLAGS: 00010003 RAX: 0177e39bf7f7f7d0 RBX: ffff8800be7830d8 RCX: ffff8800be7830d8 RDX: ffffea0000000000 RSI: ffff8801021d3f58 RDI: 6b6b6b6b6b6b6b6b RBP: ffff8801021d3ce0 R08: ffff8800be7830d8 R09: ffff8801021d3990 R10: 0000000000000001 R11: 0000000000000078 R12: 6b6b6b6b6b6b6b6b R13: 0000000000000202 R14: 0000000000000001 R15: ffffffffa027d6c0 FS: 0000000000000000(0000) GS:ffff88012fc80000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b CR2: 000000311421b080 CR3: 0000000001a03000 CR4: 00000000000006e0 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Process kworker/2:2 (pid: 29049, threadinfo ffff8801021d2000, task ffff8801288ab3f0) Stack: ffff8801021d3d10 ffffffff81107431 ffff8800be7830d8 0000000000000000 ffff8800c7635d40 6b6b6b6b6b6b6b6b ffff8801021d3d40 ffffffffa02851bd ffff8800c7635d40 ffff8800c7635db0 0000000000000001 0000000000000000 Call Trace: [<ffffffff81107431>] verify_mem_not_deleted+0x3c/0xb7 [<ffffffffa02851bd>] rpcb_getport_done+0x2b/0x126 [sunrpc] [<ffffffffa027d0ba>] rpc_exit_task+0x3f/0x6d [sunrpc] [<ffffffffa027d4ab>] __rpc_execute+0x78/0x24b [sunrpc] [<ffffffffa027d6c0>] ? rpc_execute+0x42/0x42 [sunrpc] [<ffffffffa027d6d0>] rpc_async_schedule+0x10/0x12 [sunrpc] [<ffffffff810611b7>] process_one_work+0x230/0x41d [<ffffffff81061102>] ? process_one_work+0x17b/0x41d [<ffffffff81063613>] worker_thread+0x133/0x217 [<ffffffff810634e0>] ? manage_workers+0x191/0x191 [<ffffffff81066e10>] kthread+0x7d/0x85 [<ffffffff81485924>] kernel_thread_helper+0x4/0x10 [<ffffffff8147eb18>] ? retint_restore_args+0x13/0x13 [<ffffffff81066d93>] ? __init_kthread_worker+0x56/0x56 [<ffffffff81485920>] ? gs_change+0x13/0x13 Code: 02 00 00 3d 00 02 00 00 0f 4f c2 c9 c3 55 48 89 e5 e8 eb e9 f2 ff 48 c1 e8 0c 48 ba 00 00 00 00 00 ea ff ff 48 6b c0 38 48 01 d0 8b 10 66 85 d2 79 04 48 8b 40 10 c9 c3 55 48 89 e5 41 57 41 RIP [<ffffffff81105eae>] virt_to_head_page+0x1e/0x2c RSP <ffff8801021d3ce0> ---[ end trace 4cb7eac5b28823fd ]--- Thanks, Ben -- Ben Greear <greearb@candelatech.com> Candela Technologies Inc http://www.candelatech.com ^ permalink raw reply [flat|nested] 9+ messages in thread
end of thread, other threads:[~2011-06-28 3:47 UTC | newest] Thread overview: 9+ messages (download: mbox.gz follow: Atom feed -- links below jump to the message on this page -- 2011-06-27 23:17 [PATCH 0/2] SLUB memory debugging improvements greearb 2011-06-27 23:17 ` [PATCH 1/2] slub: Enable backtrace for create/delete points greearb 2011-06-27 23:17 ` [PATCH 2/2] slub: Add method to verify memory is not deleted greearb 2011-06-27 23:28 ` David Rientjes 2011-06-27 23:46 ` Ben Greear 2011-06-28 0:18 ` Ben Greear 2011-06-28 0:19 ` David Rientjes 2011-06-28 0:24 ` Ben Greear 2011-06-28 3:45 ` Ben Greear
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox