From: Trond Myklebust <trondmy@primarydata.com>
To: "tj@kernel.org" <tj@kernel.org>
Cc: "bfields@fieldses.org" <bfields@fieldses.org>,
"linux-kernel@vger.kernel.org" <linux-kernel@vger.kernel.org>,
"lorenzo.pieralisi@arm.com" <lorenzo.pieralisi@arm.com>,
"jlayton@poochiereds.net" <jlayton@poochiereds.net>,
"linux-nfs@vger.kernel.org" <linux-nfs@vger.kernel.org>,
"jiangshanlai@gmail.com" <jiangshanlai@gmail.com>,
"anna.schumaker@netapp.com" <anna.schumaker@netapp.com>
Subject: Re: net/sunrpc: v4.14-rc4 lockdep warning
Date: Tue, 10 Oct 2017 16:48:57 +0000 [thread overview]
Message-ID: <1507654135.4442.4.camel@primarydata.com> (raw)
In-Reply-To: <20171010140336.GI3301751@devbig577.frc2.facebook.com>
T24gVHVlLCAyMDE3LTEwLTEwIGF0IDA3OjAzIC0wNzAwLCB0akBrZXJuZWwub3JnIHdyb3RlOg0K
PiBIZWxsbywgVHJvbmQuDQo+IA0KPiBPbiBNb24sIE9jdCAwOSwgMjAxNyBhdCAwNjozMjoxM1BN
ICswMDAwLCBUcm9uZCBNeWtsZWJ1c3Qgd3JvdGU6DQo+ID4gT24gTW9uLCAyMDE3LTEwLTA5IGF0
IDE5OjE3ICswMTAwLCBMb3JlbnpvIFBpZXJhbGlzaSB3cm90ZToNCj4gPiA+IEkgaGF2ZSBydW4g
aW50byB0aGUgbG9ja2RlcCB3YXJuaW5nIGJlbG93IHdoaWxlIHJ1bm5pbmcgdjQuMTQtDQo+ID4g
PiByYzMvcmM0DQo+ID4gPiBvbiBhbiBBUk02NCBkZWZjb25maWcgSnVubyBkZXYgYm9hcmQgLSBy
ZXBvcnRpbmcgaXQgdG8gY2hlY2sNCj4gPiA+IHdoZXRoZXINCj4gPiA+IGl0IGlzIGEga25vd24v
Z2VudWluZSBpc3N1ZS4NCj4gPiA+IA0KPiA+ID4gUGxlYXNlIGxldCBtZSBrbm93IGlmIHlvdSBu
ZWVkIGZ1cnRoZXIgZGVidWcgZGF0YSBvciBuZWVkIHNvbWUNCj4gPiA+IHNwZWNpZmljIHRlc3Rz
Lg0KPiA+ID4gDQo+ID4gPiBbICAgIDYuMjA5Mzg0XQ0KPiA+ID4gPT09PT09PT09PT09PT09PT09
PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09DQo+ID4gPiBbICAgIDYuMjE1NTY5
XSBXQVJOSU5HOiBwb3NzaWJsZSBjaXJjdWxhciBsb2NraW5nIGRlcGVuZGVuY3kNCj4gPiA+IGRl
dGVjdGVkDQo+ID4gPiBbICAgIDYuMjIxNzU1XSA0LjE0LjAtcmM0ICM1NCBOb3QgdGFpbnRlZA0K
PiA+ID4gWyAgICA2LjIyNTUwM10gLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0t
LS0tLS0tLS0tLS0tLS0NCj4gPiA+IC0tLS0NCj4gPiA+IFsgICAgNi4yMzE2ODldIGt3b3JrZXIv
NDowSC8zMiBpcyB0cnlpbmcgdG8gYWNxdWlyZSBsb2NrOg0KPiA+ID4gWyAgICA2LjIzNjgzMF0g
ICgoJnRhc2stPnUudGtfd29yaykpeysuKy59LCBhdDoNCj4gPiA+IFs8ZmZmZjAwMDAwODBlNjRj
Yz5dDQo+ID4gPiBwcm9jZXNzX29uZV93b3JrKzB4MWNjLzB4M2YwDQo+ID4gPiBbICAgIDYuMjQ1
NDcyXSANCj4gPiA+ICAgICAgICAgICAgICAgIGJ1dCB0YXNrIGlzIGFscmVhZHkgaG9sZGluZyBs
b2NrOg0KPiA+ID4gWyAgICA2LjI1MTMwOV0gICgieHBydGlvZCIpeysuKy59LCBhdDogWzxmZmZm
MDAwMDA4MGU2NGNjPl0NCj4gPiA+IHByb2Nlc3Nfb25lX3dvcmsrMHgxY2MvMHgzZjANCj4gPiA+
IFsgICAgNi4yNTkxNThdIA0KPiA+ID4gICAgICAgICAgICAgICAgd2hpY2ggbG9jayBhbHJlYWR5
IGRlcGVuZHMgb24gdGhlIG5ldyBsb2NrLg0KPiA+ID4gDQo+ID4gPiBbICAgIDYuMjY3MzQ1XSAN
Cj4gPiA+ICAgICAgICAgICAgICAgIHRoZSBleGlzdGluZyBkZXBlbmRlbmN5IGNoYWluIChpbiBy
ZXZlcnNlIG9yZGVyKQ0KPiA+ID4gaXM6DQo+IA0KPiAuLg0KPiA+IEFkZGluZyBUZWp1biBhbmQg
TGFpLCBzaW5jZSB0aGlzIGxvb2tzIGxpa2UgYSB3b3JrcXVldWUgbG9ja2luZw0KPiA+IGlzc3Vl
Lg0KPiANCj4gSXQgbG9va3MgYSBiaXQgY3J5cHRpYyBidXQgaXQncyB3YXJuaW5nIGFnYWluc3Qg
dGhlIGZvbGxvd2luZyBjYXNlLg0KPiANCj4gMS4gTWVtb3J5IHByZXNzdXJlIGlzIGhpZ2ggYW5k
IHJlc2N1ZXIga2lja3MgaW4gZm9yIHRoZSB4cHJ0aW9kDQo+ICAgIHdvcmtxdWV1ZS4gIFRoZXJl
IGFyZSBubyBvdGhlciBrd29ya2VycyBzZXJ2aW5nIHRoZSB3b3JrcXVldWUuDQo+IA0KPiAyLiBU
aGUgcmVzY3VlciBydW5zIHRoZSB4cHRyX2Rlc3Ryb3kgcGF0aCBhbmQgZW5kcyB1cCBjYWxsaW5n
DQo+ICAgIGNhbmNlbF93b3JrX3N5bmMoKSBvbiBhIHdvcmsgaXRlbSB3aGljaCBpcyBxdWV1ZWQg
b24geHBydGlvZC4NCj4gDQo+IDMuIFRoZSB3b3JrIGl0ZW0gaXMgcGVuZGluZyBvbiB0aGUgc2Ft
ZSB3b3JrcXVldWUgYW5kIGFzc3VtaW5nIHRoYXQNCj4gICAgbWVtb3J5IHByZXNzdXJlIGRvZXNu
J3QgbGV0IG9mZiAobGV0J3Mgc2F5IHJlY2xhaW0gaXMgdHJ5aW5nIHRvDQo+ICAgIGtpY2sgb2Zm
IG5mcyBwYWdlcyksIHRoZSBvbmx5IHdheSBpdCBjYW4gZ2V0IGV4ZWN1dGVkIGlzIGJ5IHRoZQ0K
PiAgICByZXNjdWVyIHdoaWNoIGlzIHdhaXRpbmcgZm9yIHRoZSB3b3JrIGl0ZW0gLSBhbiBBLUIt
QSBkZWFkbG9jay4NCj4gDQoNCkhpIFRlanVuLA0KDQpUaGFua3MgZm9yIHRoZSBleHBsYW5hdGlv
bi4gV2hhdCBJJ20gbm90IHJlYWxseSB1bmRlcnN0YW5kaW5nIGhlcmUNCnRob3VnaCwgaXMgaG93
IHRoZSB3b3JrIGl0ZW0gY291bGQgYmUgcXVldWVkIGF0IGFsbC4gV2UgaGF2ZSBhDQp3YWl0X29u
X2JpdF9sb2NrKCkgaW4geHBydF9kZXN0cm95KCkgdGhhdCBzaG91bGQgbWVhbiB0aGUgeHBydC0N
Cj50YXNrX2NsZWFudXAgd29yayBpdGVtIGhhcyBjb21wbGV0ZWQgcnVubmluZywgYW5kIHRoYXQg
aXQgY2Fubm90IGJlDQpyZXF1ZXVlZC4NCg0KSXMgdGhlcmUgYSBwb3NzaWJpbGl0eSB0aGF0IHRo
ZSBmbHVzaF9xdWV1ZSgpIG1pZ2h0IGJlIHRyaWdnZXJlZA0KZGVzcGl0ZSB0aGUgd29yayBpdGVt
IG5vdCBiZWluZyBxdWV1ZWQ/DQoNCi0tIA0KVHJvbmQgTXlrbGVidXN0DQpMaW51eCBORlMgY2xp
ZW50IG1haW50YWluZXIsIFByaW1hcnlEYXRhDQp0cm9uZC5teWtsZWJ1c3RAcHJpbWFyeWRhdGEu
Y29tDQo=
WARNING: multiple messages have this Message-ID (diff)
From: Trond Myklebust <trondmy@primarydata.com>
To: "tj@kernel.org" <tj@kernel.org>
Cc: "bfields@fieldses.org" <bfields@fieldses.org>,
"linux-kernel@vger.kernel.org" <linux-kernel@vger.kernel.org>,
"lorenzo.pieralisi@arm.com" <lorenzo.pieralisi@arm.com>,
"jlayton@poochiereds.net" <jlayton@poochiereds.net>,
"linux-nfs@vger.kernel.org" <linux-nfs@vger.kernel.org>,
"jiangshanlai@gmail.com" <jiangshanlai@gmail.com>,
"anna.schumaker@netapp.com" <anna.schumaker@netapp.com>
Subject: Re: net/sunrpc: v4.14-rc4 lockdep warning
Date: Tue, 10 Oct 2017 16:48:57 +0000 [thread overview]
Message-ID: <1507654135.4442.4.camel@primarydata.com> (raw)
In-Reply-To: <20171010140336.GI3301751@devbig577.frc2.facebook.com>
On Tue, 2017-10-10 at 07:03 -0700, tj@kernel.org wrote:
> Hello, Trond.
>
> On Mon, Oct 09, 2017 at 06:32:13PM +0000, Trond Myklebust wrote:
> > On Mon, 2017-10-09 at 19:17 +0100, Lorenzo Pieralisi wrote:
> > > I have run into the lockdep warning below while running v4.14-
> > > rc3/rc4
> > > on an ARM64 defconfig Juno dev board - reporting it to check
> > > whether
> > > it is a known/genuine issue.
> > >
> > > Please let me know if you need further debug data or need some
> > > specific tests.
> > >
> > > [ 6.209384]
> > > ======================================================
> > > [ 6.215569] WARNING: possible circular locking dependency
> > > detected
> > > [ 6.221755] 4.14.0-rc4 #54 Not tainted
> > > [ 6.225503] --------------------------------------------------
> > > ----
> > > [ 6.231689] kworker/4:0H/32 is trying to acquire lock:
> > > [ 6.236830] ((&task->u.tk_work)){+.+.}, at:
> > > [<ffff0000080e64cc>]
> > > process_one_work+0x1cc/0x3f0
> > > [ 6.245472]
> > > but task is already holding lock:
> > > [ 6.251309] ("xprtiod"){+.+.}, at: [<ffff0000080e64cc>]
> > > process_one_work+0x1cc/0x3f0
> > > [ 6.259158]
> > > which lock already depends on the new lock.
> > >
> > > [ 6.267345]
> > > the existing dependency chain (in reverse order)
> > > is:
>
> ..
> > Adding Tejun and Lai, since this looks like a workqueue locking
> > issue.
>
> It looks a bit cryptic but it's warning against the following case.
>
> 1. Memory pressure is high and rescuer kicks in for the xprtiod
> workqueue. There are no other kworkers serving the workqueue.
>
> 2. The rescuer runs the xptr_destroy path and ends up calling
> cancel_work_sync() on a work item which is queued on xprtiod.
>
> 3. The work item is pending on the same workqueue and assuming that
> memory pressure doesn't let off (let's say reclaim is trying to
> kick off nfs pages), the only way it can get executed is by the
> rescuer which is waiting for the work item - an A-B-A deadlock.
>
Hi Tejun,
Thanks for the explanation. What I'm not really understanding here
though, is how the work item could be queued at all. We have a
wait_on_bit_lock() in xprt_destroy() that should mean the xprt-
>task_cleanup work item has completed running, and that it cannot be
requeued.
Is there a possibility that the flush_queue() might be triggered
despite the work item not being queued?
--
Trond Myklebust
Linux NFS client maintainer, PrimaryData
trond.myklebust@primarydata.com
next prev parent reply other threads:[~2017-10-10 16:49 UTC|newest]
Thread overview: 10+ messages / expand[flat|nested] mbox.gz Atom feed top
2017-10-09 18:17 net/sunrpc: v4.14-rc4 lockdep warning Lorenzo Pieralisi
2017-10-09 18:32 ` Trond Myklebust
2017-10-09 18:32 ` Trond Myklebust
2017-10-10 14:03 ` tj
2017-10-10 16:48 ` Trond Myklebust [this message]
2017-10-10 16:48 ` Trond Myklebust
2017-10-10 17:19 ` tj
2017-10-11 17:49 ` Trond Myklebust
2017-10-11 17:49 ` Trond Myklebust
2017-10-16 13:34 ` Jan Glauber
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=1507654135.4442.4.camel@primarydata.com \
--to=trondmy@primarydata.com \
--cc=anna.schumaker@netapp.com \
--cc=bfields@fieldses.org \
--cc=jiangshanlai@gmail.com \
--cc=jlayton@poochiereds.net \
--cc=linux-kernel@vger.kernel.org \
--cc=linux-nfs@vger.kernel.org \
--cc=lorenzo.pieralisi@arm.com \
--cc=tj@kernel.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.