From mboxrd@z Thu Jan 1 00:00:00 1970 From: Joonas Lahtinen Subject: Re: [PATCH] [RFC] kernel/cpu: Use lockref for online CPU reference counting Date: Tue, 16 Feb 2016 10:49:36 +0200 Message-ID: <1455612576.4977.11.camel@linux.intel.com> References: <1455539803-13913-1-git-send-email-joonas.lahtinen@linux.intel.com> <20160215141755.GG6357@twins.programming.kicks-ass.net> <20160215170618.GL6375@twins.programming.kicks-ass.net> Mime-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: base64 Return-path: Received: from mga09.intel.com (mga09.intel.com [134.134.136.24]) by gabe.freedesktop.org (Postfix) with ESMTP id B98616E25F for ; Tue, 16 Feb 2016 08:49:40 +0000 (UTC) In-Reply-To: <20160215170618.GL6375@twins.programming.kicks-ass.net> List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: intel-gfx-bounces@lists.freedesktop.org Sender: "Intel-gfx" To: Peter Zijlstra Cc: "Gautham R. Shenoy" , Intel graphics driver community testing & development , Linux kernel development , David Hildenbrand , "Paul E. McKenney" , Ingo Molnar List-Id: intel-gfx@lists.freedesktop.org SGksCgpPbiBtYSwgMjAxNi0wMi0xNSBhdCAxODowNiArMDEwMCwgUGV0ZXIgWmlqbHN0cmEgd3Jv dGU6Cj4gT24gTW9uLCBGZWIgMTUsIDIwMTYgYXQgMDM6MTc6NTVQTSArMDEwMCwgUGV0ZXIgWmlq bHN0cmEgd3JvdGU6Cj4gPiBPbiBNb24sIEZlYiAxNSwgMjAxNiBhdCAwMjozNjo0M1BNICswMjAw LCBKb29uYXMgTGFodGluZW4gd3JvdGU6Cj4gPiA+IEluc3RlYWQgb2YgaW1wbGVtZW50aW5nIGEg Y3VzdG9tIGxvY2tlZCByZWZlcmVuY2UgY291bnRpbmcsIHVzZSBsb2NrcmVmLgo+ID4gPiAKPiA+ ID4gQ3VycmVudCBpbXBsZW1lbnRhdGlvbiBsZWFkcyB0byBhIGRlYWRsb2NrIHNwbGF0IG9uIElu dGVsIFNLTCBwbGF0Zm9ybXMKPiA+ID4gd2hlbiBsb2NrZGVwIGRlYnVnZ2luZyBpcyBlbmFibGVk Lgo+ID4gPiAKPiA+ID4gVGhpcyBpcyBkdWUgdG8gZmV3IG9mIENQVWZyZXEgZHJpdmVycyAoaW5j bHVkaW5nIEludGVsIFAtc3RhdGUpIGhhdmluZyB0aGlzOwo+ID4gPiBwb2xpY3ktPnJ3c2VtIGlz IGxvY2tlZCBkdXJpbmcgZHJpdmVyIGluaXRpYWxpemF0aW9uIGFuZCB0aGUgZnVuY3Rpb25zIGNh bGxlZAo+ID4gPiBkdXJpbmcgaW5pdCB0aGF0IGFjdHVhbGx5IGFwcGx5IENQVSBsaW1pdHMgdXNl IGdldF9vbmxpbmVfY3B1cyAoYmVjYXVzZSB0aGV5Cj4gPiA+IGhhdmUgb3RoZXIgY2FsbGluZyBw YXRocyB0b28pLCB3aGljaCB3aWxsIGJyaWVmbHkgbG9jayBjcHVfaG90cGx1Zy5sb2NrIHRvCj4g PiA+IGluY3JlYXNlIGNwdV9ob3RwbHVnLnJlZmNvdW50Lgo+ID4gPiAKPiA+ID4gT24gbGF0ZXIg Y2FsbGluZyBwYXRoLCB3aGVuIGRvaW5nIGEgc3VzcGVuZCwgd2hlbiBjcHVfaG90cGx1Z19iZWdp bigpIGlzIGNhbGxlZAo+ID4gPiBpbiBkaXNhYmxlX25vbmJvb3RfY3B1cygpLCBjYWxsYmFja3Mg dG8gQ1BVZnJlcSBmdW5jdGlvbnMgZ2V0IGNhbGxlZCBhZnRlciwKPiA+ID4gd2hpY2ggd2lsbCBs b2NrIHBvbGljeS0+cndzZW0gYW5kIGNwdV9ob3RwbHVnLmxvY2sgaXMgYWxyZWFkeSBoZWxkIGJ5 Cj4gPiA+IGNwdV9ob3RwbHVnX2JlZ2luKCkgYW5kIHdlIGRvIGhhdmUgYSBwb3RlbnRpYWwgZGVh ZGxvY2sgc2NlbmFyaW8gcmVwb3J0ZWQgYnkKPiA+ID4gb3VyIENJIHN5c3RlbSAodGhvdWdoIGl0 IGlzIGEgdmVyeSB1bmxpa2VseSBvbmUpLiBTZWUgdGhlIEJ1Z3ppbGxhIGxpbmsgZm9yIG1vcmUK PiA+ID4gZGV0YWlscy4KPiA+IAo+ID4gSSd2ZSBiZWVuIG1lYW5pbmcgdG8gY2hhbmdlIHRoZSB0 aGluZyBpbnRvIGEgcGVyY3B1LXJ3c2VtLCBJIGp1c3QKPiA+IGhhdmVuJ3QgaGFkIHRpbWUgdG8g bG9vayBpbnRvIHRoZSBsb2NrZGVwIHNwbGF0IHRoYXQgZ2VuZXJhdGVkLgo+IAo+IAo+IFRoZSBi ZWxvdyBoYXMgcGxlbnR5IGxvY2tkZXAgaXNzdWVzIGJlY2F1c2UgcGVyY3B1LXJ3c2VtIGlzCj4g cmVhZGVyLXdyaXRlciBmYWlyIChsaWtlIHRoZSByZWd1bGFyIHJ3c2VtKSwgc28gaXQgZG9lcyB0 aHJvdyB1cCBhIGZhaXIKPiBudW1iZXIgb2YgdmVyeSBpY2t5IGlzc3Vlcy4KPiAKCkkgb3JpZ2lu YWxseSB0aG91Z2h0IG9mIGltcGxlbWVudGluZyB0aGlzIG1vcmUgc2ltaWxhciB0byB3aGF0IHlv dQpzcGVjaWZ5LCBidXQgdGhlbiBJIGNhbWUgYWNyb3NzIGEgZGlzY3Vzc2lvbiBpbiB0aGUgbWFp bGluZyBsaXN0IHdoZXJlCml0IHdhcyBOQUtlZCBhZGRpbmcgbW9yZSBtZW1iZXJzIHRvIHRhc2tf c3RydWN0OwoKaHR0cDovL2NvbW1lbnRzLmdtYW5lLm9yZy9nbWFuZS5saW51eC5rZXJuZWwvOTcw MjczCgpBZGRpbmcgcHJvcGVyIHJlY3Vyc2lvbiAodGhlIHdheSBteSBpbml0aWFsIGltcGxlbWVu dGF0aW9uIHdhcyBnb2luZykKZ290IHVnbHkgd2l0aG91dCBtb2RpZnlpbmcgdGFza19zdHJ1Y3Qg YmVjYXVzZcKgZ2V0X29ubGluZV9jcHVzKCkgaXMgYQpzcGVlZCBjcml0aWNhbCBjb2RlIHBhdGgu CgpTbyBJJ20gYWxsIGZvciBmaXhpbmcgdGhlIGN1cnJlbnQgY29kZSBpbiBhIGRpZmZlcmVudCB3 YXkgaWYgdGhhdCB3aWxsCnRoZW4gYmUgbWVyZ2VkLgoKUmVnYXJkcywgSm9vbmFzCgo+IElmIGF0 IGFsbCBwb3NzaWJsZSwgSSdkIHJlYWxseSByYXRoZXIgZml4IHRob3NlIGFuZCBoYXZlIGEgJ3Nh bmVyJwo+IGhvdHBsdWcgbG9jaywgcmF0aGVyIHRoYW4gbXVkZGxlIG9uIHdpdGggb3Blbi1jb2Rl ZCBob3Jyb3IgbG9jayB3ZSBoYXZlCj4gbm93Lgo+IAo+IAoKPFNOSVA+CgotLSAKSm9vbmFzIExh aHRpbmVuCk9wZW4gU291cmNlIFRlY2hub2xvZ3kgQ2VudGVyCkludGVsIENvcnBvcmF0aW9uCl9f X19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fCkludGVsLWdmeCBt YWlsaW5nIGxpc3QKSW50ZWwtZ2Z4QGxpc3RzLmZyZWVkZXNrdG9wLm9yZwpodHRwczovL2xpc3Rz LmZyZWVkZXNrdG9wLm9yZy9tYWlsbWFuL2xpc3RpbmZvL2ludGVsLWdmeAo= From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1754558AbcBPItl (ORCPT ); Tue, 16 Feb 2016 03:49:41 -0500 Received: from mga11.intel.com ([192.55.52.93]:56900 "EHLO mga11.intel.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1754291AbcBPItk (ORCPT ); Tue, 16 Feb 2016 03:49:40 -0500 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.22,454,1449561600"; d="scan'208";a="915834851" Message-ID: <1455612576.4977.11.camel@linux.intel.com> Subject: Re: [PATCH] [RFC] kernel/cpu: Use lockref for online CPU reference counting From: Joonas Lahtinen To: Peter Zijlstra Cc: Intel graphics driver community testing & development , Linux kernel development , Ingo Molnar , David Hildenbrand , "Paul E. McKenney" , "Gautham R. Shenoy" , Chris Wilson , Daniel Vetter Date: Tue, 16 Feb 2016 10:49:36 +0200 In-Reply-To: <20160215170618.GL6375@twins.programming.kicks-ass.net> References: <1455539803-13913-1-git-send-email-joonas.lahtinen@linux.intel.com> <20160215141755.GG6357@twins.programming.kicks-ass.net> <20160215170618.GL6375@twins.programming.kicks-ass.net> Organization: Intel Finland Oy - BIC 0357606-4 - Westendinkatu 7, 02160 Espoo Content-Type: text/plain; charset="UTF-8" X-Mailer: Evolution 3.18.4 (3.18.4-1.fc23) Mime-Version: 1.0 Content-Transfer-Encoding: 8bit Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Hi, On ma, 2016-02-15 at 18:06 +0100, Peter Zijlstra wrote: > On Mon, Feb 15, 2016 at 03:17:55PM +0100, Peter Zijlstra wrote: > > On Mon, Feb 15, 2016 at 02:36:43PM +0200, Joonas Lahtinen wrote: > > > Instead of implementing a custom locked reference counting, use lockref. > > > > > > Current implementation leads to a deadlock splat on Intel SKL platforms > > > when lockdep debugging is enabled. > > > > > > This is due to few of CPUfreq drivers (including Intel P-state) having this; > > > policy->rwsem is locked during driver initialization and the functions called > > > during init that actually apply CPU limits use get_online_cpus (because they > > > have other calling paths too), which will briefly lock cpu_hotplug.lock to > > > increase cpu_hotplug.refcount. > > > > > > On later calling path, when doing a suspend, when cpu_hotplug_begin() is called > > > in disable_nonboot_cpus(), callbacks to CPUfreq functions get called after, > > > which will lock policy->rwsem and cpu_hotplug.lock is already held by > > > cpu_hotplug_begin() and we do have a potential deadlock scenario reported by > > > our CI system (though it is a very unlikely one). See the Bugzilla link for more > > > details. > > > > I've been meaning to change the thing into a percpu-rwsem, I just > > haven't had time to look into the lockdep splat that generated. > > > The below has plenty lockdep issues because percpu-rwsem is > reader-writer fair (like the regular rwsem), so it does throw up a fair > number of very icky issues. > I originally thought of implementing this more similar to what you specify, but then I came across a discussion in the mailing list where it was NAKed adding more members to task_struct; http://comments.gmane.org/gmane.linux.kernel/970273 Adding proper recursion (the way my initial implementation was going) got ugly without modifying task_struct because get_online_cpus() is a speed critical code path. So I'm all for fixing the current code in a different way if that will then be merged. Regards, Joonas > If at all possible, I'd really rather fix those and have a 'saner' > hotplug lock, rather than muddle on with open-coded horror lock we have > now. > > -- Joonas Lahtinen Open Source Technology Center Intel Corporation