From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S932072AbdJIScU (ORCPT ); Mon, 9 Oct 2017 14:32:20 -0400 Received: from us-smtp-delivery-194.mimecast.com ([63.128.21.194]:59191 "EHLO us-smtp-delivery-194.mimecast.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1755125AbdJIScS (ORCPT ); Mon, 9 Oct 2017 14:32:18 -0400 From: Trond Myklebust To: "linux-kernel@vger.kernel.org" , "lorenzo.pieralisi@arm.com" , "linux-nfs@vger.kernel.org" , "jiangshanlai@gmail.com" , "tj@kernel.org" CC: "bfields@fieldses.org" , "anna.schumaker@netapp.com" , "jlayton@poochiereds.net" Subject: Re: net/sunrpc: v4.14-rc4 lockdep warning Thread-Topic: net/sunrpc: v4.14-rc4 lockdep warning Thread-Index: AQHTQSrsNS+3n/veiEmJminQ1Tz/9qLb14GA Date: Mon, 9 Oct 2017 18:32:13 +0000 Message-ID: <1507573931.3516.3.camel@primarydata.com> References: <20171009181738.GA30680@red-moon> In-Reply-To: <20171009181738.GA30680@red-moon> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-originating-ip: [68.49.162.121] x-ms-publictraffictype: Email x-microsoft-exchange-diagnostics: 1;DM5PR11MB0076;20:cKpVpGkJkw6Hf/9f8s4rj9WhmA21LF72i9OFN57TH7f27EkS09/EQtKINC3mPLxHXA6RIjXi4uOnyA1A3CrX7SzCwz0gmURQOYssYSLjFDQBFBZGS/+2KmVl0bnVWvHS22BlDn00JTdiqCwC44nj+2PHQebaawNzZj/dFIuI8/8= x-ms-exchange-antispam-srfa-diagnostics: SSOS; x-ms-office365-filtering-correlation-id: e0b41de4-36d1-4a57-32ca-08d50f440f4e x-microsoft-antispam: UriScan:;BCL:0;PCL:0;RULEID:(22001)(2017030254152)(2017082002075)(2017052603199)(201703131423075)(201702281549075);SRVR:DM5PR11MB0076; x-ms-traffictypediagnostic: DM5PR11MB0076: x-exchange-antispam-report-test: UriScan:(211171220733660); x-microsoft-antispam-prvs: x-exchange-antispam-report-cfa-test: BCL:0;PCL:0;RULEID:(100000700101)(100105000095)(100000701101)(100105300095)(100000702101)(100105100095)(6040450)(2401047)(8121501046)(5005006)(10201501046)(93006095)(93001095)(100000703101)(100105400095)(3002001)(6041248)(20161123560025)(20161123564025)(20161123555025)(2016111802025)(201703131423075)(201702281528075)(201703061421075)(201703061406153)(20161123562025)(20161123558100)(6072148)(6043046)(201708071742011)(100000704101)(100105200095)(100000705101)(100105500095);SRVR:DM5PR11MB0076;BCL:0;PCL:0;RULEID:(100000800101)(100110000095)(100000801101)(100110300095)(100000802101)(100110100095)(100000803101)(100110400095)(100000804101)(100110200095)(100000805101)(100110500095);SRVR:DM5PR11MB0076; x-forefront-prvs: 045584D28C x-forefront-antispam-report: SFV:NSPM;SFS:(10019020)(6009001)(376002)(39830400002)(346002)(24454002)(199003)(189002)(51234002)(377424004)(105586002)(5660300001)(106356001)(36756003)(229853002)(8676002)(81156014)(81166006)(2950100002)(6486002)(305945005)(77096006)(25786009)(7736002)(66066001)(2501003)(97736004)(3660700001)(86362001)(316002)(2201001)(54356999)(76176999)(3280700002)(54906003)(6436002)(50986999)(6512007)(3846002)(33646002)(102836003)(6116002)(6246003)(39060400002)(6506006)(14454004)(68736007)(99286003)(110136005)(2900100001)(103116003)(2906002)(8936002)(478600001)(101416001)(53936002)(4326008)(189998001);DIR:OUT;SFP:1102;SCL:1;SRVR:DM5PR11MB0076;H:DM5PR11MB0075.namprd11.prod.outlook.com;FPR:;SPF:None;PTR:InfoNoRecords;A:1;MX:1;LANG:en; spamdiagnosticoutput: 1:99 spamdiagnosticmetadata: NSPM Content-ID: <5A4652DF6A1ACB48B44B75DE8B667307@namprd11.prod.outlook.com> MIME-Version: 1.0 X-OriginatorOrg: primarydata.com X-MS-Exchange-CrossTenant-originalarrivaltime: 09 Oct 2017 18:32:13.0519 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: 03193ed6-8726-4bb3-a832-18ab0d28adb7 X-MS-Exchange-Transport-CrossTenantHeadersStamped: DM5PR11MB0076 X-MC-Unique: DLQxZAE0PP6qlUv7k0pCIg-1 Content-Type: text/plain; charset=UTF-8 Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Transfer-Encoding: 8bit X-MIME-Autoconverted: from base64 to 8bit by nfs id v99IWRpW002789 On Mon, 2017-10-09 at 19:17 +0100, Lorenzo Pieralisi wrote: > Hi, > > I have run into the lockdep warning below while running v4.14-rc3/rc4 > on an ARM64 defconfig Juno dev board - reporting it to check whether > it is a known/genuine issue. > > Please let me know if you need further debug data or need some > specific tests. > > Thanks, > Lorenzo > > [ 6.209384] ====================================================== > [ 6.215569] WARNING: possible circular locking dependency detected > [ 6.221755] 4.14.0-rc4 #54 Not tainted > [ 6.225503] ------------------------------------------------------ > [ 6.231689] kworker/4:0H/32 is trying to acquire lock: > [ 6.236830] ((&task->u.tk_work)){+.+.}, at: [] > process_one_work+0x1cc/0x3f0 > [ 6.245472] > but task is already holding lock: > [ 6.251309] ("xprtiod"){+.+.}, at: [] > process_one_work+0x1cc/0x3f0 > [ 6.259158] > which lock already depends on the new lock. > > [ 6.267345] > the existing dependency chain (in reverse order) is: > [ 6.274836] > -> #1 ("xprtiod"){+.+.}: > [ 6.279903] lock_acquire+0x6c/0xb8 > [ 6.283914] flush_work+0x188/0x270 > [ 6.287926] __cancel_work_timer+0x120/0x198 > [ 6.292720] cancel_work_sync+0x10/0x18 > [ 6.297081] xs_destroy+0x34/0x58 > [ 6.300917] xprt_destroy+0x84/0x90 > [ 6.304927] xprt_put+0x34/0x40 > [ 6.308589] rpc_task_release_client+0x6c/0x80 > [ 6.313557] rpc_release_resources_task+0x2c/0x38 > [ 6.318786] __rpc_execute+0x9c/0x210 > [ 6.322971] rpc_async_schedule+0x10/0x18 > [ 6.327504] process_one_work+0x240/0x3f0 > [ 6.332036] worker_thread+0x48/0x420 > [ 6.336222] kthread+0x12c/0x158 > [ 6.339972] ret_from_fork+0x10/0x18 > [ 6.344068] > -> #0 ((&task->u.tk_work)){+.+.}: > [ 6.349920] __lock_acquire+0x12ec/0x14a8 > [ 6.354451] lock_acquire+0x6c/0xb8 > [ 6.358462] process_one_work+0x22c/0x3f0 > [ 6.362994] worker_thread+0x48/0x420 > [ 6.367180] kthread+0x12c/0x158 > [ 6.370929] ret_from_fork+0x10/0x18 > [ 6.375025] > other info that might help us debug this: > > [ 6.383038] Possible unsafe locking scenario: > > [ 6.388962] CPU0 CPU1 > [ 6.393493] ---- ---- > [ 6.398023] lock("xprtiod"); > [ 6.401080] lock((&task- > >u.tk_work)); > [ 6.407444] lock("xprtiod"); > [ 6.413024] lock((&task->u.tk_work)); > [ 6.416863] > *** DEADLOCK *** > > [ 6.422789] 1 lock held by kworker/4:0H/32: > [ 6.426972] #0: ("xprtiod"){+.+.}, at: [] > process_one_work+0x1cc/0x3f0 > [ 6.435258] > stack backtrace: > [ 6.439618] CPU: 4 PID: 32 Comm: kworker/4:0H Not tainted 4.14.0- > rc4 #54 > [ 6.446325] Hardware name: ARM Juno development board (r2) (DT) > [ 6.452252] Workqueue: xprtiod rpc_async_schedule > [ 6.456959] Call trace: > [ 6.459406] [] dump_backtrace+0x0/0x3c8 > [ 6.464810] [] show_stack+0x14/0x20 > [ 6.469866] [] dump_stack+0xb8/0xf0 > [ 6.474922] [] print_circular_bug+0x224/0x3a0 > [ 6.480849] [] check_prev_add+0x304/0x860 > [ 6.486426] [] __lock_acquire+0x12ec/0x14a8 > [ 6.492177] [] lock_acquire+0x6c/0xb8 > [ 6.497406] [] process_one_work+0x22c/0x3f0 > [ 6.503156] [] worker_thread+0x48/0x420 > [ 6.508560] [] kthread+0x12c/0x158 > [ 6.513528] [] ret_from_fork+0x10/0x18 > Adding Tejun and Lai, since this looks like a workqueue locking issue. Cheers Trond -- Trond Myklebust Linux NFS client maintainer, PrimaryData trond.myklebust@primarydata.com