From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S932449AbdJJQtG (ORCPT ); Tue, 10 Oct 2017 12:49:06 -0400 Received: from us-smtp-delivery-194.mimecast.com ([216.205.24.194]:60212 "EHLO us-smtp-delivery-194.mimecast.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S932296AbdJJQtE (ORCPT ); Tue, 10 Oct 2017 12:49:04 -0400 From: Trond Myklebust To: "tj@kernel.org" CC: "bfields@fieldses.org" , "linux-kernel@vger.kernel.org" , "lorenzo.pieralisi@arm.com" , "jlayton@poochiereds.net" , "linux-nfs@vger.kernel.org" , "jiangshanlai@gmail.com" , "anna.schumaker@netapp.com" Subject: Re: net/sunrpc: v4.14-rc4 lockdep warning Thread-Topic: net/sunrpc: v4.14-rc4 lockdep warning Thread-Index: AQHTQSrsNS+3n/veiEmJminQ1Tz/9qLb14GAgAFHSgCAAC4wgA== Date: Tue, 10 Oct 2017 16:48:57 +0000 Message-ID: <1507654135.4442.4.camel@primarydata.com> References: <20171009181738.GA30680@red-moon> <1507573931.3516.3.camel@primarydata.com> <20171010140336.GI3301751@devbig577.frc2.facebook.com> In-Reply-To: <20171010140336.GI3301751@devbig577.frc2.facebook.com> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-originating-ip: [68.49.162.121] x-ms-publictraffictype: Email x-microsoft-exchange-diagnostics: 1;DM5PR11MB0073;20:BfzJJ4RXqdqxnO86Jbg43NOjkm6R38XX+QYKudVJF7RTkCjDvpnY6i1XW4WW4ABVHBQlGE8/IRlbnNsoCRU31MZREyH8vPziUsYyQGcQ9TFlC+4NQhtqgzugJHaBBLiLgMnRjvd5WFKtN9Tt+IgEdc9VDJV93aJqjG6z/+QXFtE= x-ms-exchange-antispam-srfa-diagnostics: SSOS; x-ms-office365-filtering-correlation-id: 75524de3-8d41-42f3-055c-08d50ffeccd4 x-microsoft-antispam: UriScan:;BCL:0;PCL:0;RULEID:(22001)(2017030254152)(2017082002075)(2017052603199)(201703131423075)(201702281549075);SRVR:DM5PR11MB0073; x-ms-traffictypediagnostic: DM5PR11MB0073: x-exchange-antispam-report-test: UriScan:(211171220733660); x-microsoft-antispam-prvs: x-exchange-antispam-report-cfa-test: BCL:0;PCL:0;RULEID:(100000700101)(100105000095)(100000701101)(100105300095)(100000702101)(100105100095)(6040450)(2401047)(5005006)(8121501046)(3002001)(10201501046)(100000703101)(100105400095)(93006095)(93001095)(6041248)(20161123555025)(20161123560025)(20161123558100)(2016111802025)(20161123562025)(20161123564025)(201703131423075)(201702281528075)(201703061421075)(201703061406153)(6072148)(6043046)(201708071742011)(100000704101)(100105200095)(100000705101)(100105500095);SRVR:DM5PR11MB0073;BCL:0;PCL:0;RULEID:(100000800101)(100110000095)(100000801101)(100110300095)(100000802101)(100110100095)(100000803101)(100110400095)(100000804101)(100110200095)(100000805101)(100110500095);SRVR:DM5PR11MB0073; x-forefront-prvs: 04569283F9 x-forefront-antispam-report: SFV:NSPM;SFS:(10019020)(6009001)(39830400002)(376002)(346002)(377424004)(189002)(24454002)(199003)(51914003)(105586002)(53936002)(33646002)(2351001)(5640700003)(2900100001)(316002)(36756003)(3660700001)(3280700002)(2906002)(68736007)(2950100002)(54906003)(101416001)(50986999)(4001150100001)(76176999)(14454004)(66066001)(305945005)(54356999)(106356001)(6916009)(3846002)(25786009)(189998001)(97736004)(5660300001)(2501003)(102836003)(6116002)(81156014)(1730700003)(7736002)(81166006)(8676002)(99286003)(77096006)(86362001)(6436002)(6246003)(4326008)(6486002)(8936002)(6506006)(478600001)(39060400002)(6512007)(103116003)(229853002);DIR:OUT;SFP:1102;SCL:1;SRVR:DM5PR11MB0073;H:DM5PR11MB0075.namprd11.prod.outlook.com;FPR:;SPF:None;PTR:InfoNoRecords;MX:1;A:1;LANG:en; spamdiagnosticoutput: 1:99 spamdiagnosticmetadata: NSPM Content-ID: <7896EE6AEFCF734395EBEE2C9C1DCE21@namprd11.prod.outlook.com> MIME-Version: 1.0 X-OriginatorOrg: primarydata.com X-MS-Exchange-CrossTenant-originalarrivaltime: 10 Oct 2017 16:48:57.4100 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: 03193ed6-8726-4bb3-a832-18ab0d28adb7 X-MS-Exchange-Transport-CrossTenantHeadersStamped: DM5PR11MB0073 X-MC-Unique: I8c_H95JPN6d0M3bqapWMg-1 Content-Type: text/plain; charset=UTF-8 Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Transfer-Encoding: 8bit X-MIME-Autoconverted: from base64 to 8bit by nfs id v9AGnFsS028426 On Tue, 2017-10-10 at 07:03 -0700, tj@kernel.org wrote: > Hello, Trond. > > On Mon, Oct 09, 2017 at 06:32:13PM +0000, Trond Myklebust wrote: > > On Mon, 2017-10-09 at 19:17 +0100, Lorenzo Pieralisi wrote: > > > I have run into the lockdep warning below while running v4.14- > > > rc3/rc4 > > > on an ARM64 defconfig Juno dev board - reporting it to check > > > whether > > > it is a known/genuine issue. > > > > > > Please let me know if you need further debug data or need some > > > specific tests. > > > > > > [ 6.209384] > > > ====================================================== > > > [ 6.215569] WARNING: possible circular locking dependency > > > detected > > > [ 6.221755] 4.14.0-rc4 #54 Not tainted > > > [ 6.225503] -------------------------------------------------- > > > ---- > > > [ 6.231689] kworker/4:0H/32 is trying to acquire lock: > > > [ 6.236830] ((&task->u.tk_work)){+.+.}, at: > > > [] > > > process_one_work+0x1cc/0x3f0 > > > [ 6.245472] > > > but task is already holding lock: > > > [ 6.251309] ("xprtiod"){+.+.}, at: [] > > > process_one_work+0x1cc/0x3f0 > > > [ 6.259158] > > > which lock already depends on the new lock. > > > > > > [ 6.267345] > > > the existing dependency chain (in reverse order) > > > is: > > .. > > Adding Tejun and Lai, since this looks like a workqueue locking > > issue. > > It looks a bit cryptic but it's warning against the following case. > > 1. Memory pressure is high and rescuer kicks in for the xprtiod > workqueue. There are no other kworkers serving the workqueue. > > 2. The rescuer runs the xptr_destroy path and ends up calling > cancel_work_sync() on a work item which is queued on xprtiod. > > 3. The work item is pending on the same workqueue and assuming that > memory pressure doesn't let off (let's say reclaim is trying to > kick off nfs pages), the only way it can get executed is by the > rescuer which is waiting for the work item - an A-B-A deadlock. > Hi Tejun, Thanks for the explanation. What I'm not really understanding here though, is how the work item could be queued at all. We have a wait_on_bit_lock() in xprt_destroy() that should mean the xprt- >task_cleanup work item has completed running, and that it cannot be requeued. Is there a possibility that the flush_queue() might be triggered despite the work item not being queued? -- Trond Myklebust Linux NFS client maintainer, PrimaryData trond.myklebust@primarydata.com