From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from us-smtp-delivery-194.mimecast.com ([216.205.24.194]:58506 "EHLO us-smtp-delivery-194.mimecast.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S932298AbdJJQtE (ORCPT ); Tue, 10 Oct 2017 12:49:04 -0400 From: Trond Myklebust To: "tj@kernel.org" CC: "bfields@fieldses.org" , "linux-kernel@vger.kernel.org" , "lorenzo.pieralisi@arm.com" , "jlayton@poochiereds.net" , "linux-nfs@vger.kernel.org" , "jiangshanlai@gmail.com" , "anna.schumaker@netapp.com" Subject: Re: net/sunrpc: v4.14-rc4 lockdep warning Date: Tue, 10 Oct 2017 16:48:57 +0000 Message-ID: <1507654135.4442.4.camel@primarydata.com> References: <20171009181738.GA30680@red-moon> <1507573931.3516.3.camel@primarydata.com> <20171010140336.GI3301751@devbig577.frc2.facebook.com> In-Reply-To: <20171010140336.GI3301751@devbig577.frc2.facebook.com> MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Sender: linux-nfs-owner@vger.kernel.org List-ID: T24gVHVlLCAyMDE3LTEwLTEwIGF0IDA3OjAzIC0wNzAwLCB0akBrZXJuZWwub3JnIHdyb3RlOg0K PiBIZWxsbywgVHJvbmQuDQo+IA0KPiBPbiBNb24sIE9jdCAwOSwgMjAxNyBhdCAwNjozMjoxM1BN ICswMDAwLCBUcm9uZCBNeWtsZWJ1c3Qgd3JvdGU6DQo+ID4gT24gTW9uLCAyMDE3LTEwLTA5IGF0 IDE5OjE3ICswMTAwLCBMb3JlbnpvIFBpZXJhbGlzaSB3cm90ZToNCj4gPiA+IEkgaGF2ZSBydW4g aW50byB0aGUgbG9ja2RlcCB3YXJuaW5nIGJlbG93IHdoaWxlIHJ1bm5pbmcgdjQuMTQtDQo+ID4g PiByYzMvcmM0DQo+ID4gPiBvbiBhbiBBUk02NCBkZWZjb25maWcgSnVubyBkZXYgYm9hcmQgLSBy ZXBvcnRpbmcgaXQgdG8gY2hlY2sNCj4gPiA+IHdoZXRoZXINCj4gPiA+IGl0IGlzIGEga25vd24v Z2VudWluZSBpc3N1ZS4NCj4gPiA+IA0KPiA+ID4gUGxlYXNlIGxldCBtZSBrbm93IGlmIHlvdSBu ZWVkIGZ1cnRoZXIgZGVidWcgZGF0YSBvciBuZWVkIHNvbWUNCj4gPiA+IHNwZWNpZmljIHRlc3Rz Lg0KPiA+ID4gDQo+ID4gPiBbICAgIDYuMjA5Mzg0XQ0KPiA+ID4gPT09PT09PT09PT09PT09PT09 PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09DQo+ID4gPiBbICAgIDYuMjE1NTY5 XSBXQVJOSU5HOiBwb3NzaWJsZSBjaXJjdWxhciBsb2NraW5nIGRlcGVuZGVuY3kNCj4gPiA+IGRl dGVjdGVkDQo+ID4gPiBbICAgIDYuMjIxNzU1XSA0LjE0LjAtcmM0ICM1NCBOb3QgdGFpbnRlZA0K PiA+ID4gWyAgICA2LjIyNTUwM10gLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0t LS0tLS0tLS0tLS0tLS0NCj4gPiA+IC0tLS0NCj4gPiA+IFsgICAgNi4yMzE2ODldIGt3b3JrZXIv NDowSC8zMiBpcyB0cnlpbmcgdG8gYWNxdWlyZSBsb2NrOg0KPiA+ID4gWyAgICA2LjIzNjgzMF0g ICgoJnRhc2stPnUudGtfd29yaykpeysuKy59LCBhdDoNCj4gPiA+IFs8ZmZmZjAwMDAwODBlNjRj Yz5dDQo+ID4gPiBwcm9jZXNzX29uZV93b3JrKzB4MWNjLzB4M2YwDQo+ID4gPiBbICAgIDYuMjQ1 NDcyXSANCj4gPiA+ICAgICAgICAgICAgICAgIGJ1dCB0YXNrIGlzIGFscmVhZHkgaG9sZGluZyBs b2NrOg0KPiA+ID4gWyAgICA2LjI1MTMwOV0gICgieHBydGlvZCIpeysuKy59LCBhdDogWzxmZmZm MDAwMDA4MGU2NGNjPl0NCj4gPiA+IHByb2Nlc3Nfb25lX3dvcmsrMHgxY2MvMHgzZjANCj4gPiA+ IFsgICAgNi4yNTkxNThdIA0KPiA+ID4gICAgICAgICAgICAgICAgd2hpY2ggbG9jayBhbHJlYWR5 IGRlcGVuZHMgb24gdGhlIG5ldyBsb2NrLg0KPiA+ID4gDQo+ID4gPiBbICAgIDYuMjY3MzQ1XSAN Cj4gPiA+ICAgICAgICAgICAgICAgIHRoZSBleGlzdGluZyBkZXBlbmRlbmN5IGNoYWluIChpbiBy ZXZlcnNlIG9yZGVyKQ0KPiA+ID4gaXM6DQo+IA0KPiAuLg0KPiA+IEFkZGluZyBUZWp1biBhbmQg TGFpLCBzaW5jZSB0aGlzIGxvb2tzIGxpa2UgYSB3b3JrcXVldWUgbG9ja2luZw0KPiA+IGlzc3Vl Lg0KPiANCj4gSXQgbG9va3MgYSBiaXQgY3J5cHRpYyBidXQgaXQncyB3YXJuaW5nIGFnYWluc3Qg dGhlIGZvbGxvd2luZyBjYXNlLg0KPiANCj4gMS4gTWVtb3J5IHByZXNzdXJlIGlzIGhpZ2ggYW5k IHJlc2N1ZXIga2lja3MgaW4gZm9yIHRoZSB4cHJ0aW9kDQo+ICAgIHdvcmtxdWV1ZS4gIFRoZXJl IGFyZSBubyBvdGhlciBrd29ya2VycyBzZXJ2aW5nIHRoZSB3b3JrcXVldWUuDQo+IA0KPiAyLiBU aGUgcmVzY3VlciBydW5zIHRoZSB4cHRyX2Rlc3Ryb3kgcGF0aCBhbmQgZW5kcyB1cCBjYWxsaW5n DQo+ICAgIGNhbmNlbF93b3JrX3N5bmMoKSBvbiBhIHdvcmsgaXRlbSB3aGljaCBpcyBxdWV1ZWQg b24geHBydGlvZC4NCj4gDQo+IDMuIFRoZSB3b3JrIGl0ZW0gaXMgcGVuZGluZyBvbiB0aGUgc2Ft ZSB3b3JrcXVldWUgYW5kIGFzc3VtaW5nIHRoYXQNCj4gICAgbWVtb3J5IHByZXNzdXJlIGRvZXNu J3QgbGV0IG9mZiAobGV0J3Mgc2F5IHJlY2xhaW0gaXMgdHJ5aW5nIHRvDQo+ICAgIGtpY2sgb2Zm IG5mcyBwYWdlcyksIHRoZSBvbmx5IHdheSBpdCBjYW4gZ2V0IGV4ZWN1dGVkIGlzIGJ5IHRoZQ0K PiAgICByZXNjdWVyIHdoaWNoIGlzIHdhaXRpbmcgZm9yIHRoZSB3b3JrIGl0ZW0gLSBhbiBBLUIt QSBkZWFkbG9jay4NCj4gDQoNCkhpIFRlanVuLA0KDQpUaGFua3MgZm9yIHRoZSBleHBsYW5hdGlv bi4gV2hhdCBJJ20gbm90IHJlYWxseSB1bmRlcnN0YW5kaW5nIGhlcmUNCnRob3VnaCwgaXMgaG93 IHRoZSB3b3JrIGl0ZW0gY291bGQgYmUgcXVldWVkIGF0IGFsbC4gV2UgaGF2ZSBhDQp3YWl0X29u X2JpdF9sb2NrKCkgaW4geHBydF9kZXN0cm95KCkgdGhhdCBzaG91bGQgbWVhbiB0aGUgeHBydC0N Cj50YXNrX2NsZWFudXAgd29yayBpdGVtIGhhcyBjb21wbGV0ZWQgcnVubmluZywgYW5kIHRoYXQg aXQgY2Fubm90IGJlDQpyZXF1ZXVlZC4NCg0KSXMgdGhlcmUgYSBwb3NzaWJpbGl0eSB0aGF0IHRo ZSBmbHVzaF9xdWV1ZSgpIG1pZ2h0IGJlIHRyaWdnZXJlZA0KZGVzcGl0ZSB0aGUgd29yayBpdGVt IG5vdCBiZWluZyBxdWV1ZWQ/DQoNCi0tIA0KVHJvbmQgTXlrbGVidXN0DQpMaW51eCBORlMgY2xp ZW50IG1haW50YWluZXIsIFByaW1hcnlEYXRhDQp0cm9uZC5teWtsZWJ1c3RAcHJpbWFyeWRhdGEu Y29tDQo= From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S932449AbdJJQtG (ORCPT ); Tue, 10 Oct 2017 12:49:06 -0400 Received: from us-smtp-delivery-194.mimecast.com ([216.205.24.194]:60212 "EHLO us-smtp-delivery-194.mimecast.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S932296AbdJJQtE (ORCPT ); Tue, 10 Oct 2017 12:49:04 -0400 From: Trond Myklebust To: "tj@kernel.org" CC: "bfields@fieldses.org" , "linux-kernel@vger.kernel.org" , "lorenzo.pieralisi@arm.com" , "jlayton@poochiereds.net" , "linux-nfs@vger.kernel.org" , "jiangshanlai@gmail.com" , "anna.schumaker@netapp.com" Subject: Re: net/sunrpc: v4.14-rc4 lockdep warning Thread-Topic: net/sunrpc: v4.14-rc4 lockdep warning Thread-Index: AQHTQSrsNS+3n/veiEmJminQ1Tz/9qLb14GAgAFHSgCAAC4wgA== Date: Tue, 10 Oct 2017 16:48:57 +0000 Message-ID: <1507654135.4442.4.camel@primarydata.com> References: <20171009181738.GA30680@red-moon> <1507573931.3516.3.camel@primarydata.com> <20171010140336.GI3301751@devbig577.frc2.facebook.com> In-Reply-To: <20171010140336.GI3301751@devbig577.frc2.facebook.com> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-originating-ip: [68.49.162.121] x-ms-publictraffictype: Email x-microsoft-exchange-diagnostics: 1;DM5PR11MB0073;20:BfzJJ4RXqdqxnO86Jbg43NOjkm6R38XX+QYKudVJF7RTkCjDvpnY6i1XW4WW4ABVHBQlGE8/IRlbnNsoCRU31MZREyH8vPziUsYyQGcQ9TFlC+4NQhtqgzugJHaBBLiLgMnRjvd5WFKtN9Tt+IgEdc9VDJV93aJqjG6z/+QXFtE= x-ms-exchange-antispam-srfa-diagnostics: SSOS; x-ms-office365-filtering-correlation-id: 75524de3-8d41-42f3-055c-08d50ffeccd4 x-microsoft-antispam: UriScan:;BCL:0;PCL:0;RULEID:(22001)(2017030254152)(2017082002075)(2017052603199)(201703131423075)(201702281549075);SRVR:DM5PR11MB0073; x-ms-traffictypediagnostic: DM5PR11MB0073: x-exchange-antispam-report-test: UriScan:(211171220733660); x-microsoft-antispam-prvs: x-exchange-antispam-report-cfa-test: BCL:0;PCL:0;RULEID:(100000700101)(100105000095)(100000701101)(100105300095)(100000702101)(100105100095)(6040450)(2401047)(5005006)(8121501046)(3002001)(10201501046)(100000703101)(100105400095)(93006095)(93001095)(6041248)(20161123555025)(20161123560025)(20161123558100)(2016111802025)(20161123562025)(20161123564025)(201703131423075)(201702281528075)(201703061421075)(201703061406153)(6072148)(6043046)(201708071742011)(100000704101)(100105200095)(100000705101)(100105500095);SRVR:DM5PR11MB0073;BCL:0;PCL:0;RULEID:(100000800101)(100110000095)(100000801101)(100110300095)(100000802101)(100110100095)(100000803101)(100110400095)(100000804101)(100110200095)(100000805101)(100110500095);SRVR:DM5PR11MB0073; x-forefront-prvs: 04569283F9 x-forefront-antispam-report: SFV:NSPM;SFS:(10019020)(6009001)(39830400002)(376002)(346002)(377424004)(189002)(24454002)(199003)(51914003)(105586002)(53936002)(33646002)(2351001)(5640700003)(2900100001)(316002)(36756003)(3660700001)(3280700002)(2906002)(68736007)(2950100002)(54906003)(101416001)(50986999)(4001150100001)(76176999)(14454004)(66066001)(305945005)(54356999)(106356001)(6916009)(3846002)(25786009)(189998001)(97736004)(5660300001)(2501003)(102836003)(6116002)(81156014)(1730700003)(7736002)(81166006)(8676002)(99286003)(77096006)(86362001)(6436002)(6246003)(4326008)(6486002)(8936002)(6506006)(478600001)(39060400002)(6512007)(103116003)(229853002);DIR:OUT;SFP:1102;SCL:1;SRVR:DM5PR11MB0073;H:DM5PR11MB0075.namprd11.prod.outlook.com;FPR:;SPF:None;PTR:InfoNoRecords;MX:1;A:1;LANG:en; spamdiagnosticoutput: 1:99 spamdiagnosticmetadata: NSPM Content-ID: <7896EE6AEFCF734395EBEE2C9C1DCE21@namprd11.prod.outlook.com> MIME-Version: 1.0 X-OriginatorOrg: primarydata.com X-MS-Exchange-CrossTenant-originalarrivaltime: 10 Oct 2017 16:48:57.4100 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: 03193ed6-8726-4bb3-a832-18ab0d28adb7 X-MS-Exchange-Transport-CrossTenantHeadersStamped: DM5PR11MB0073 X-MC-Unique: I8c_H95JPN6d0M3bqapWMg-1 Content-Type: text/plain; charset=UTF-8 Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Transfer-Encoding: 8bit X-MIME-Autoconverted: from base64 to 8bit by nfs id v9AGnFsS028426 On Tue, 2017-10-10 at 07:03 -0700, tj@kernel.org wrote: > Hello, Trond. > > On Mon, Oct 09, 2017 at 06:32:13PM +0000, Trond Myklebust wrote: > > On Mon, 2017-10-09 at 19:17 +0100, Lorenzo Pieralisi wrote: > > > I have run into the lockdep warning below while running v4.14- > > > rc3/rc4 > > > on an ARM64 defconfig Juno dev board - reporting it to check > > > whether > > > it is a known/genuine issue. > > > > > > Please let me know if you need further debug data or need some > > > specific tests. > > > > > > [ 6.209384] > > > ====================================================== > > > [ 6.215569] WARNING: possible circular locking dependency > > > detected > > > [ 6.221755] 4.14.0-rc4 #54 Not tainted > > > [ 6.225503] -------------------------------------------------- > > > ---- > > > [ 6.231689] kworker/4:0H/32 is trying to acquire lock: > > > [ 6.236830] ((&task->u.tk_work)){+.+.}, at: > > > [] > > > process_one_work+0x1cc/0x3f0 > > > [ 6.245472] > > > but task is already holding lock: > > > [ 6.251309] ("xprtiod"){+.+.}, at: [] > > > process_one_work+0x1cc/0x3f0 > > > [ 6.259158] > > > which lock already depends on the new lock. > > > > > > [ 6.267345] > > > the existing dependency chain (in reverse order) > > > is: > > .. > > Adding Tejun and Lai, since this looks like a workqueue locking > > issue. > > It looks a bit cryptic but it's warning against the following case. > > 1. Memory pressure is high and rescuer kicks in for the xprtiod > workqueue. There are no other kworkers serving the workqueue. > > 2. The rescuer runs the xptr_destroy path and ends up calling > cancel_work_sync() on a work item which is queued on xprtiod. > > 3. The work item is pending on the same workqueue and assuming that > memory pressure doesn't let off (let's say reclaim is trying to > kick off nfs pages), the only way it can get executed is by the > rescuer which is waiting for the work item - an A-B-A deadlock. > Hi Tejun, Thanks for the explanation. What I'm not really understanding here though, is how the work item could be queued at all. We have a wait_on_bit_lock() in xprt_destroy() that should mean the xprt- >task_cleanup work item has completed running, and that it cannot be requeued. Is there a possibility that the flush_queue() might be triggered despite the work item not being queued? -- Trond Myklebust Linux NFS client maintainer, PrimaryData trond.myklebust@primarydata.com