From: Davidlohr Bueso <dave@stgolabs.net>
To: Dave Chinner <david@fromorbit.com>
Cc: Ming Lei <ming.lei@canonical.com>,
linux-kernel@vger.kernel.org,
"Peter Zijlstra (Intel)" <peterz@infradead.org>,
Jason Low <jason.low2@hp.com>,
Linus Torvalds <torvalds@linux-foundation.org>,
Michel Lespinasse <walken@google.com>,
"Paul E. McKenney" <paulmck@linux.vnet.ibm.com>,
Tim Chen <tim.c.chen@linux.intel.com>,
Ingo Molnar <mingo@kernel.org>, "Theodore Ts'o" <tytso@mit.edu>
Subject: Re: [PATCH -next] locking/rwsem: don't spin in heavy contention
Date: Fri, 06 Mar 2015 14:24:23 -0800 [thread overview]
Message-ID: <1425680663.19505.65.camel@stgolabs.net> (raw)
In-Reply-To: <20150306215417.GD13958@dastard>
On Sat, 2015-03-07 at 08:54 +1100, Dave Chinner wrote:
> On Fri, Mar 06, 2015 at 11:13:10PM +0800, Ming Lei wrote:
> > Before commit b3fd4f03ca0b995(locking/rwsem: Avoid deceiving lock
> > spinners), rwsem_spin_on_owner() returns false if the owner is changed.
> > This commit just returns true under the situation, then kernel
> > softlock can be triggered easily in xfstest.
> >
> > So this patch recovers to previous behaviour, and it should be
> > reasonable to stop spining in case of heavy contention.
> >
> > The soft lockup can be reproduced easily in xfstests(generic/299)
> > over ext4:
> >
> > [ 236.417011] NMI watchdog: BUG: soft lockup - CPU#5 stuck for 23s! [kworker/5:80:3288]
> > [ 236.417011] Modules linked in: nbd ipv6 kvm_intel kvm serio_raw
> > [ 236.417011] CPU: 5 PID: 3288 Comm: kworker/5:80 Not tainted 4.0.0-rc1-next-20150303+ #69
> > [ 236.417011] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS Bochs 01/01/2011
> > [ 236.417011] Workqueue: dio/sda dio_aio_complete_work
> > [ 236.417011] task: ffff8800b87c0000 ti: ffff8800b703c000 task.ti: ffff8800b703c000
> > [ 236.417011] RIP: 0010:[<ffffffff81083c20>] [<ffffffff81083c20>] __rcu_read_unlock+0x47/0x55
> > [ 236.417011] RSP: 0018:ffff8800b703fb98 EFLAGS: 00000246
> > [ 236.417011] RAX: 0000000000000000 RBX: ffff8800b703fb48 RCX: 000000000003b080
> > [ 236.417011] RDX: fffffffe00000001 RSI: ffff880231f03a20 RDI: ffff8800bb755568
> > [ 236.417011] RBP: ffff8800b703fba8 R08: ffff880227908078 R09: ffff8800b87c0000
> > [ 236.417011] R10: 0000000000000001 R11: 0000000000000020 R12: ffff8800b703c000
> > [ 236.417011] R13: ffff8800b87c0000 R14: 000000000000000f R15: 0000000000000101
> > [ 236.417011] FS: 0000000000000000(0000) GS:ffff88023eca0000(0000) knlGS:0000000000000000
> > [ 236.417011] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
> > [ 236.417011] CR2: 00007f549369f948 CR3: 00000000ba891000 CR4: 00000000000007e0
> > [ 236.417011] Stack:
> > [ 236.417011] fffffffe00000001 ffff8800bb755568 ffff8800b703fbc8 ffffffff81073917
> > [ 236.417011] ffff8800bb755568 ffff8800bb755584 ffff8800b703fc48 ffffffff814d1ba4
> > [ 236.417011] ffff8800b703fbe8 ffff8800b87c0000 ffff8800b703fc78 ffffffff811cf51b
> > [ 236.417011] Call Trace:
> > [ 236.417011] [<ffffffff81073917>] rwsem_spin_on_owner+0x2b/0x79
> > [ 236.417011] [<ffffffff814d1ba4>] rwsem_down_write_failed+0xc0/0x2f1
> > [ 236.417011] [<ffffffff811cf51b>] ? start_this_handle+0x494/0x4bd
> > [ 236.417011] [<ffffffff810d1149>] ? trace_preempt_on+0x12/0x2f
> > [ 236.417011] [<ffffffff812ae8f3>] call_rwsem_down_write_failed+0x13/0x20
> > [ 236.417011] [<ffffffff814d1623>] ? down_write+0x24/0x33
> > [ 236.417011] [<ffffffff81199404>] ext4_map_blocks+0x236/0x3cb
> > [ 236.417011] [<ffffffff811bb407>] ? ext4_convert_unwritten_extents+0xd2/0x19c
> > [ 236.417011] [<ffffffff811bcae4>] ? __ext4_journal_start_sb+0x77/0xb8
> > [ 236.417011] [<ffffffff811bb42e>] ext4_convert_unwritten_extents+0xf9/0x19c
> > [ 236.417011] [<ffffffff8119e214>] ext4_put_io_end+0x3a/0x5d
> > [ 236.417011] [<ffffffff81197268>] ext4_end_io_dio+0x2a/0x2c
> > [ 236.417011] [<ffffffff8116418c>] dio_complete+0x97/0x12d
> > [ 236.417011] [<ffffffff81164333>] dio_aio_complete_work+0x21/0x23
>
> If you're getting stuff there, I'd be looking for a bug in ext4, not
> the rwsem code. There's no way there should be enough unwritten
> extent conversion pending to lock up the system for that length of
> time. Especially considering the test has concurrent truncates
> running which should drain the entire IO queue every couple of
> seconds at worst....
FYI, this issue is being handled here:
https://lkml.org/lkml/2015/3/6/811
Thanks,
Davidlohr
next prev parent reply other threads:[~2015-03-06 22:24 UTC|newest]
Thread overview: 4+ messages / expand[flat|nested] mbox.gz Atom feed top
2015-03-06 15:13 [PATCH -next] locking/rwsem: don't spin in heavy contention Ming Lei
2015-03-06 21:54 ` Dave Chinner
2015-03-06 22:24 ` Davidlohr Bueso [this message]
2015-03-07 11:47 ` Ming Lei
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=1425680663.19505.65.camel@stgolabs.net \
--to=dave@stgolabs.net \
--cc=david@fromorbit.com \
--cc=jason.low2@hp.com \
--cc=linux-kernel@vger.kernel.org \
--cc=ming.lei@canonical.com \
--cc=mingo@kernel.org \
--cc=paulmck@linux.vnet.ibm.com \
--cc=peterz@infradead.org \
--cc=tim.c.chen@linux.intel.com \
--cc=torvalds@linux-foundation.org \
--cc=tytso@mit.edu \
--cc=walken@google.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox