From: paulmck@linux.ibm.com (Paul E. McKenney)
Subject: v5.0-rc2 and NVMeOF
Date: Mon, 11 Feb 2019 13:08:08 -0800 [thread overview]
Message-ID: <20190211210808.GS4240@linux.ibm.com> (raw)
In-Reply-To: <1549905891.19311.5.camel@acm.org>
On Mon, Feb 11, 2019@09:24:51AM -0800, Bart Van Assche wrote:
> On Wed, 2019-01-16@17:16 -0800, Sagi Grimberg wrote:
> > On 1/15/19 11:07 AM, Bart Van Assche wrote:
> > > Hello,
> > >
> > > With Linus' kernel v5.0-rc2 the blktests nvmeof-mp tests trigger the
> > > complaint shown below. Is this a known issue?
> >
> > Seems like ns remove is racing with ns revalidate again..
> >
> > Wasn't this related to: eb4c2382272a ("srcu: Lock srcu_data structure in
> > srcu_gp_start()") ?
>
> (+Paul)
>
> I'm not sure. Paul, are you perhaps aware of any open issues in the RCU
> infrastructure? If I run the following test:
The last one I knew of is present in v5.0-rc1 eb4c2382272a ("srcu:
Lock srcu_data structure in srcu_gp_start()").
> git clone https://github.com/osandov/blktests.git
> cd blktests
> ./check -q nvmeof-mp
>
> then the following appears on the console:
>
> BUG: KASAN: use-after-free in srcu_invoke_callbacks+0x209/0x290
> Read of size 8 at addr ffff8881126b6df0 by task kworker/2:94/26747
> CPU: 2 PID: 26747 Comm: kworker/2:94 Not tainted 5.0.0-rc5-dbg+ #5
> Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.10.2-1 04/01/2014
> Workqueue: rcu_gp srcu_invoke_callbacks
> Call Trace:
> dump_stack+0x86/0xca
> print_address_description+0x71/0x239
> kasan_report.cold.3+0x1b/0x3e
> __asan_load8+0x54/0x90
> srcu_invoke_callbacks+0x209/0x290
> process_one_work+0x4f1/0xa40
> worker_thread+0x67/0x5b0
> kthread+0x1cf/0x1f0
> ret_from_fork+0x24/0x30
The usual way that something like this happens is by invoking call_srcu()
twice in a row on the same object, similar to double-kfree() but with
call_srcu() instead of kfree(). One way to check for this sort of thing
is to reproduce in a kernel built with CONFIG_DEBUG_OBJECTS_RCU_HEAD=y.
Thanx, Paul
> Allocated by task 955:
> save_stack+0x43/0xd0
> __kasan_kmalloc.constprop.9+0xcb/0xd0
> kasan_kmalloc+0x9/0x10
> kmem_cache_alloc_trace+0x14c/0x340
> nvme_validate_ns+0xada/0x1170
> nvme_scan_work+0x299/0x4c8
> process_one_work+0x4f1/0xa40
> worker_thread+0x67/0x5b0
> kthread+0x1cf/0x1f0
> ret_from_fork+0x24/0x30
>
> Freed by task 55:
> save_stack+0x43/0xd0
> __kasan_slab_free+0x139/0x190
> kasan_slab_free+0xe/0x10
> kfree+0x103/0x320
> nvme_free_ns+0x198/0x1a0
> nvme_ns_remove+0x1c5/0x240
> nvme_remove_namespaces+0x1b3/0x210
> nvme_delete_ctrl_work+0x7d/0xe0
> process_one_work+0x4f1/0xa40
> worker_thread+0x367/0x5b0
> kthread+0x1cf/0x1f0
> ret_from_fork+0x24/0x30
>
> The buggy address belongs to the object at ffff8881126b6c00
> which belongs to the cache kmalloc-1k of size 1024
> The buggy address is located 496 bytes inside of
> 1024-byte region [ffff8881126b6c00, ffff8881126b7000)
> The buggy address belongs to the page:
> page:ffffea000449ac00 count:1 mapcount:0 mapping:ffff88811b002a00 index:0xffff8881126b1f80 compound_mapcount: 0
> flags: 0x2fff000000010200(slab|head)
> raw: 2fff000000010200 ffffea00042bcc08 ffffea000457b808 ffff88811b002a00
> raw: ffff8881126b1f80 00000000001c0011 00000001ffffffff 0000000000000000
> page dumped because: kasan: bad access detected
>
> Memory state around the buggy address:
> ffff8881126b6c80: fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb
> ffff8881126b6d00: fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb
> >ffff8881126b6d80: fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb
> ^
> ffff8881126b6e00: fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb
> ffff8881126b6e80: fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb
>
next prev parent reply other threads:[~2019-02-11 21:08 UTC|newest]
Thread overview: 35+ messages / expand[flat|nested] mbox.gz Atom feed top
2019-01-15 19:07 v5.0-rc2 and NVMeOF Bart Van Assche
2019-01-17 1:16 ` Sagi Grimberg
2019-02-11 17:24 ` Bart Van Assche
2019-02-11 21:08 ` Paul E. McKenney [this message]
2019-02-11 22:27 ` Bart Van Assche
2019-02-12 1:24 ` Paul E. McKenney
2019-02-12 16:47 ` Bart Van Assche
2019-02-12 17:47 ` Paul E. McKenney
2019-02-12 19:15 ` Paul E. McKenney
2019-02-13 0:44 ` Bart Van Assche
2019-02-13 1:10 ` Paul E. McKenney
2019-02-13 15:19 ` Paul E. McKenney
2019-02-13 15:24 ` Paul E. McKenney
2019-02-13 18:36 ` Bart Van Assche
2019-02-13 18:48 ` Paul E. McKenney
2019-02-13 19:12 ` Bart Van Assche
2019-02-13 19:30 ` Paul E. McKenney
2019-02-13 19:52 ` Paul E. McKenney
2019-02-13 21:00 ` Bart Van Assche
2019-02-13 22:09 ` Paul E. McKenney
2019-02-13 23:07 ` Paul E. McKenney
2019-02-14 0:21 ` Bart Van Assche
2019-02-14 1:02 ` Paul E. McKenney
2019-02-26 17:35 ` Paul E. McKenney
2019-02-26 17:47 ` Bart Van Assche
2019-02-26 18:12 ` Paul E. McKenney
2019-02-26 18:40 ` Bart Van Assche
2019-02-26 19:20 ` Paul E. McKenney
2019-02-26 23:48 ` Bart Van Assche
2019-02-27 16:04 ` Paul E. McKenney
2019-02-27 16:25 ` Bart Van Assche
2019-02-27 18:22 ` Paul E. McKenney
2019-02-13 19:13 ` Paul E. McKenney
2019-02-13 0:47 ` Bart Van Assche
2019-02-13 1:07 ` Paul E. McKenney
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20190211210808.GS4240@linux.ibm.com \
--to=paulmck@linux.ibm.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.