From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from pandora.armlinux.org.uk (pandora.armlinux.org.uk [78.32.30.218]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 119807BB11; Wed, 13 Mar 2024 17:13:51 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=78.32.30.218 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1710350034; cv=none; b=ePBNxK8DSfDwv5CLekJc2YZkN6rgeJvWjWdwc5ZHv2lKdNdKzVoIcAu0+huD/ve5+yeL2Tc6Al/45vy6d8AGgf3akggf8jqtwqIhRVta6pm+SzEvZuSXetTckJeWUkvInt5plgn6iUWxJoyfA+NhqJ9pNttlvxt6fbCnzWyykv0= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1710350034; c=relaxed/simple; bh=GUQXLTXTZ/RC3EIvgtVKiqNodwEKbwS4Cu2U8qXFU/w=; h=Date:From:To:Cc:Subject:Message-ID:References:MIME-Version: Content-Type:Content-Disposition:In-Reply-To; b=u8IuPHKfZ3k6qosQYp8noz9b3W2LQ4YEY7tpdWC0xxH3eU5bn3U42cespeDtX4TF40svG+NqI9w5DqdtXtXLJ95oz+VIgnzpzBeZhrUmXiRWm3Fa8cqbbNobbfAPZ1MVvIL7RjXqHaJ6hBaggNcIA5QA3O9f9pBgD7jNBUp/QAI= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=armlinux.org.uk; spf=none smtp.mailfrom=armlinux.org.uk; dkim=pass (2048-bit key) header.d=armlinux.org.uk header.i=@armlinux.org.uk header.b=gaE78Dgv; arc=none smtp.client-ip=78.32.30.218 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=armlinux.org.uk Authentication-Results: smtp.subspace.kernel.org; spf=none smtp.mailfrom=armlinux.org.uk Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=armlinux.org.uk header.i=@armlinux.org.uk header.b="gaE78Dgv" DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=armlinux.org.uk; s=pandora-2019; h=Sender:In-Reply-To: Content-Transfer-Encoding:Content-Type:MIME-Version:References:Message-ID: Subject:Cc:To:From:Date:Reply-To:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Id: List-Help:List-Unsubscribe:List-Subscribe:List-Post:List-Owner:List-Archive; bh=PplELY1KFkbbainlrPdk4FaZv2dPnxhGL2R5dvxZJcc=; b=gaE78DgvKAJxF5LfQGZm9rbZnq Z6g9f1MSGKltQmW+YWLoqsnkeGJUXUGPP1sjsV3PidbKBCS5V9wNQjTXOrlwwmwm5bbV/BxWMAr2x sTjZF0F6PR9AYHWc9+auHzQVw4pZRi7IIZ5xMlKxmn9VcoawshgDsEDKz0zSmdCCE+ZXVUKwRHXLY bdvmNEyOB6vNM01wor0GwCEy0QZ8WoBSHJ97KjFOXz7i368MBndTTPTtjn75tFwxUgqqU3n7bkAHc 6CzEW+lQ1fE9lpgzIG4VkIQ8bJ/vG3Egqoref58CZpLPcRcoFZWM9kQqNYrJ/5MPmamzjixkUvUvW RHj+UJcA==; Received: from shell.armlinux.org.uk ([fd8f:7570:feb6:1:5054:ff:fe00:4ec]:45220) by pandora.armlinux.org.uk with esmtpsa (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.96) (envelope-from ) id 1rkSAf-0007y7-0R; Wed, 13 Mar 2024 17:13:41 +0000 Received: from linux by shell.armlinux.org.uk with local (Exim 4.94.2) (envelope-from ) id 1rkSAX-0004eW-SY; Wed, 13 Mar 2024 17:13:33 +0000 Date: Wed, 13 Mar 2024 17:13:33 +0000 From: "Russell King (Oracle)" To: Marek Szyprowski Cc: Sudeep Holla , Catalin Marinas , "Christoph Lameter (Ampere)" , Mark Rutland , "linux-pm@vger.kernel.org" , "Rafael J. Wysocki" , Viresh Kumar , Will Deacon , Jonathan.Cameron@huawei.com, Matteo.Carlini@arm.com, Valentin.Schneider@arm.com, akpm@linux-foundation.org, anshuman.khandual@arm.com, Eric Mackay , dave.kleikamp@oracle.com, linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, robin.murphy@arm.com, vanshikonda@os.amperecomputing.com, yang@os.amperecomputing.com, Nishanth Menon , Stephen Boyd Subject: Re: [PATCH v3] ARM64: Dynamically allocate cpumasks and increase supported CPUs to 512 Message-ID: References: <37099a57-b655-3b3a-56d0-5f7fbd49d7db@gentwo.org> <9352f410-9dad-ac89-181a-b3cfc86176b8@linux.com> <432c1980-b00f-4b07-9e24-0bec52ccb5d6@samsung.com> Precedence: bulk X-Mailing-List: linux-pm@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Disposition: inline Content-Transfer-Encoding: 8bit In-Reply-To: <432c1980-b00f-4b07-9e24-0bec52ccb5d6@samsung.com> Sender: Russell King (Oracle) On Wed, Mar 13, 2024 at 05:22:33PM +0100, Marek Szyprowski wrote: > On 13.03.2024 15:35, Sudeep Holla wrote: > > On Tue, Mar 12, 2024 at 05:55:49PM +0000, Catalin Marinas wrote: > >> On Tue, Mar 12, 2024 at 10:06:06AM -0700, Christoph Lameter (Ampere) wrote: > >>> On Mon, 11 Mar 2024, Christoph Lameter (Ampere) wrote: > >>> > >>>> This could be an issue in the ARM64 arch code itself where there maybe > >>>> an assumption elsewhere that a cpumask can always store up to NR_CPU > >>>> cpus and not only nr_cpu_ids as OFFSTACK does. > >>>> > >>>> How can I exercise the opp driver in order to recreate the problem? > >>>> > >>>> I assume the opp driver is ARM specific? x86 defaults to OFFSTACK so if > >>>> there is an issue with OFFSTACK in opp then it should fail with kernel > >>>> default configuration on that platform. > >>> I checked the ARM64 arch sources use of NR_CPUS and its all fine. > >>> > >>> Also verified in my testing logs that CONFIG_PM_OPP was set in all tests. > >>> > >>> No warnings in the kernel log during those tests. > >>> > >>> How to reproduce this? > >> I guess you need a platform with a dts that has an "operating-points-v2" > >> property. I don't have any around. > >> > >> Sudeep was trying to trigger this code path earlier, not sure where he > >> got to. > > I did try to trigger this on FVP by adding OPPs + some hacks to add dummy > > clock provider to successfully probe this driver. I couldn't hit the issue > > reported 🙁. It could be that with the hardware clock/regulator drivers, it > > take a different path in OPP core. > > I can fully reproduce this issue on Khadas VIM3 and Odroid-N2 boards. > Both Meson A311D SoC based. So, if I'm reading the OPP code and the DTS* files for Khadas VIM3 correctly, these use operating-points-v2, which is parsed by the opp layer. If the opp layer is unable to parse any operating points, it should print "no supported OPPs" and remove the table (thereby preventing the code in question being reached.) So, I wonder whether what you're seeing is a latent bug which is being tickled by the presence of the CPU masks being off-stack changing the kernel timing. I would suggest the printk debug approach may help here to see when the OPPs are begun to be parsed, when they're created etc and their timing relationship to being used. Given the suspicion, it's possible that the mere addition of printk() may "fix" the problem, which again would be another semi-useful data point. -- RMK's Patch system: https://www.armlinux.org.uk/developer/patches/ FTTP is here! 80Mbps down 10Mbps up. Decent connectivity at last! From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 0DC67C54791 for ; Wed, 13 Mar 2024 17:16:22 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:In-Reply-To:MIME-Version:References: Message-ID:Subject:Cc:To:From:Date:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=+SFiGi9pZLRj01gtCA8cXuHSc0u8/WNXXCaJtj3xiGw=; b=h/T8QIYPPL0rdo LfkJggn8zr1FfhvGz8sDaHEnQ3ssLbAfqogACwm+R5yyoukIV7hLLkgwvQTtjHpEXqPSM9Qnw9LD8 79JFY8vh+PDm8WB+HPPcsDZ2mc/wiEdA+Q1/ZuEc4jJfSlkVg7VrQcZ2BPjkt5q43vegrC/4B2j7g 7Utlx309fLHwmBbNvnxlvwXcwcHux43Y9tg1IW4C+lHvxeAzpire0LYfnySBaEU8MlUDPR438HvV1 jKAvGTqwZivMvbd357W+8QwgyUVZkYMbjrgoE4RubHPa8Gp8ausHB6V4sZ4sh2sfgJdfJZVCvsBun RiYTGolkaJ7lg3dKRcPw==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.97.1 #2 (Red Hat Linux)) id 1rkSD4-0000000AytT-3H0A; Wed, 13 Mar 2024 17:16:10 +0000 Received: from pandora.armlinux.org.uk ([2001:4d48:ad52:32c8:5054:ff:fe00:142]) by bombadil.infradead.org with esmtps (Exim 4.97.1 #2 (Red Hat Linux)) id 1rkSD1-0000000AyAw-1hoF for linux-arm-kernel@lists.infradead.org; Wed, 13 Mar 2024 17:16:08 +0000 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=armlinux.org.uk; s=pandora-2019; h=Sender:In-Reply-To: Content-Transfer-Encoding:Content-Type:MIME-Version:References:Message-ID: Subject:Cc:To:From:Date:Reply-To:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Id: List-Help:List-Unsubscribe:List-Subscribe:List-Post:List-Owner:List-Archive; bh=PplELY1KFkbbainlrPdk4FaZv2dPnxhGL2R5dvxZJcc=; b=gaE78DgvKAJxF5LfQGZm9rbZnq Z6g9f1MSGKltQmW+YWLoqsnkeGJUXUGPP1sjsV3PidbKBCS5V9wNQjTXOrlwwmwm5bbV/BxWMAr2x sTjZF0F6PR9AYHWc9+auHzQVw4pZRi7IIZ5xMlKxmn9VcoawshgDsEDKz0zSmdCCE+ZXVUKwRHXLY bdvmNEyOB6vNM01wor0GwCEy0QZ8WoBSHJ97KjFOXz7i368MBndTTPTtjn75tFwxUgqqU3n7bkAHc 6CzEW+lQ1fE9lpgzIG4VkIQ8bJ/vG3Egqoref58CZpLPcRcoFZWM9kQqNYrJ/5MPmamzjixkUvUvW RHj+UJcA==; Received: from shell.armlinux.org.uk ([fd8f:7570:feb6:1:5054:ff:fe00:4ec]:45220) by pandora.armlinux.org.uk with esmtpsa (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.96) (envelope-from ) id 1rkSAf-0007y7-0R; Wed, 13 Mar 2024 17:13:41 +0000 Received: from linux by shell.armlinux.org.uk with local (Exim 4.94.2) (envelope-from ) id 1rkSAX-0004eW-SY; Wed, 13 Mar 2024 17:13:33 +0000 Date: Wed, 13 Mar 2024 17:13:33 +0000 From: "Russell King (Oracle)" To: Marek Szyprowski Cc: Sudeep Holla , Catalin Marinas , "Christoph Lameter (Ampere)" , Mark Rutland , "linux-pm@vger.kernel.org" , "Rafael J. Wysocki" , Viresh Kumar , Will Deacon , Jonathan.Cameron@huawei.com, Matteo.Carlini@arm.com, Valentin.Schneider@arm.com, akpm@linux-foundation.org, anshuman.khandual@arm.com, Eric Mackay , dave.kleikamp@oracle.com, linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, robin.murphy@arm.com, vanshikonda@os.amperecomputing.com, yang@os.amperecomputing.com, Nishanth Menon , Stephen Boyd Subject: Re: [PATCH v3] ARM64: Dynamically allocate cpumasks and increase supported CPUs to 512 Message-ID: References: <37099a57-b655-3b3a-56d0-5f7fbd49d7db@gentwo.org> <9352f410-9dad-ac89-181a-b3cfc86176b8@linux.com> <432c1980-b00f-4b07-9e24-0bec52ccb5d6@samsung.com> MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: <432c1980-b00f-4b07-9e24-0bec52ccb5d6@samsung.com> X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20240313_101607_469631_A8AAB743 X-CRM114-Status: GOOD ( 27.33 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: base64 Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org T24gV2VkLCBNYXIgMTMsIDIwMjQgYXQgMDU6MjI6MzNQTSArMDEwMCwgTWFyZWsgU3p5cHJvd3Nr aSB3cm90ZToKPiBPbiAxMy4wMy4yMDI0IDE1OjM1LCBTdWRlZXAgSG9sbGEgd3JvdGU6Cj4gPiBP biBUdWUsIE1hciAxMiwgMjAyNCBhdCAwNTo1NTo0OVBNICswMDAwLCBDYXRhbGluIE1hcmluYXMg d3JvdGU6Cj4gPj4gT24gVHVlLCBNYXIgMTIsIDIwMjQgYXQgMTA6MDY6MDZBTSAtMDcwMCwgQ2hy aXN0b3BoIExhbWV0ZXIgKEFtcGVyZSkgd3JvdGU6Cj4gPj4+IE9uIE1vbiwgMTEgTWFyIDIwMjQs IENocmlzdG9waCBMYW1ldGVyIChBbXBlcmUpIHdyb3RlOgo+ID4+Pgo+ID4+Pj4gVGhpcyBjb3Vs ZCBiZSBhbiBpc3N1ZSBpbiB0aGUgQVJNNjQgYXJjaCBjb2RlIGl0c2VsZiB3aGVyZSB0aGVyZSBt YXliZQo+ID4+Pj4gYW4gYXNzdW1wdGlvbiBlbHNld2hlcmUgdGhhdCBhIGNwdW1hc2sgY2FuIGFs d2F5cyBzdG9yZSB1cCB0byBOUl9DUFUKPiA+Pj4+IGNwdXMgYW5kIG5vdCBvbmx5IG5yX2NwdV9p ZHMgYXMgT0ZGU1RBQ0sgZG9lcy4KPiA+Pj4+Cj4gPj4+PiBIb3cgY2FuIEkgZXhlcmNpc2UgdGhl IG9wcCBkcml2ZXIgaW4gb3JkZXIgdG8gcmVjcmVhdGUgdGhlIHByb2JsZW0/Cj4gPj4+Pgo+ID4+ Pj4gSSBhc3N1bWUgdGhlIG9wcCBkcml2ZXIgaXMgQVJNIHNwZWNpZmljPyB4ODYgZGVmYXVsdHMg dG8gT0ZGU1RBQ0sgc28gaWYKPiA+Pj4+IHRoZXJlIGlzIGFuIGlzc3VlIHdpdGggT0ZGU1RBQ0sg aW4gb3BwIHRoZW4gaXQgc2hvdWxkIGZhaWwgd2l0aCBrZXJuZWwKPiA+Pj4+IGRlZmF1bHQgY29u ZmlndXJhdGlvbiBvbiB0aGF0IHBsYXRmb3JtLgo+ID4+PiBJIGNoZWNrZWQgdGhlIEFSTTY0IGFy Y2ggc291cmNlcyB1c2Ugb2YgTlJfQ1BVUyBhbmQgaXRzIGFsbCBmaW5lLgo+ID4+Pgo+ID4+PiBB bHNvIHZlcmlmaWVkIGluIG15IHRlc3RpbmcgbG9ncyB0aGF0IENPTkZJR19QTV9PUFAgd2FzIHNl dCBpbiBhbGwgdGVzdHMuCj4gPj4+Cj4gPj4+IE5vIHdhcm5pbmdzIGluIHRoZSBrZXJuZWwgbG9n IGR1cmluZyB0aG9zZSB0ZXN0cy4KPiA+Pj4KPiA+Pj4gSG93IHRvIHJlcHJvZHVjZSB0aGlzPwo+ ID4+IEkgZ3Vlc3MgeW91IG5lZWQgYSBwbGF0Zm9ybSB3aXRoIGEgZHRzIHRoYXQgaGFzIGFuICJv cGVyYXRpbmctcG9pbnRzLXYyIgo+ID4+IHByb3BlcnR5LiBJIGRvbid0IGhhdmUgYW55IGFyb3Vu ZC4KPiA+Pgo+ID4+IFN1ZGVlcCB3YXMgdHJ5aW5nIHRvIHRyaWdnZXIgdGhpcyBjb2RlIHBhdGgg ZWFybGllciwgbm90IHN1cmUgd2hlcmUgaGUKPiA+PiBnb3QgdG8uCj4gPiBJIGRpZCB0cnkgdG8g dHJpZ2dlciB0aGlzIG9uIEZWUCBieSBhZGRpbmcgT1BQcyArIHNvbWUgaGFja3MgdG8gYWRkIGR1 bW15Cj4gPiBjbG9jayBwcm92aWRlciB0byBzdWNjZXNzZnVsbHkgcHJvYmUgdGhpcyBkcml2ZXIu IEkgY291bGRuJ3QgaGl0IHRoZSBpc3N1ZQo+ID4gcmVwb3J0ZWQg8J+ZgS4gSXQgY291bGQgYmUg dGhhdCB3aXRoIHRoZSBoYXJkd2FyZSBjbG9jay9yZWd1bGF0b3IgZHJpdmVycywgaXQKPiA+IHRh a2UgYSBkaWZmZXJlbnQgcGF0aCBpbiBPUFAgY29yZS4KPiAKPiBJIGNhbiBmdWxseSByZXByb2R1 Y2UgdGhpcyBpc3N1ZSBvbiBLaGFkYXMgVklNMyBhbmQgT2Ryb2lkLU4yIGJvYXJkcy4gCj4gQm90 aCBNZXNvbiBBMzExRCBTb0MgYmFzZWQuCgpTbywgaWYgSSdtIHJlYWRpbmcgdGhlIE9QUCBjb2Rl IGFuZCB0aGUgRFRTKiBmaWxlcyBmb3IgS2hhZGFzIFZJTTMKY29ycmVjdGx5LCB0aGVzZSB1c2Ug b3BlcmF0aW5nLXBvaW50cy12Miwgd2hpY2ggaXMgcGFyc2VkIGJ5IHRoZSBvcHAKbGF5ZXIuCgpJ ZiB0aGUgb3BwIGxheWVyIGlzIHVuYWJsZSB0byBwYXJzZSBhbnkgb3BlcmF0aW5nIHBvaW50cywg aXQgc2hvdWxkCnByaW50ICJubyBzdXBwb3J0ZWQgT1BQcyIgYW5kIHJlbW92ZSB0aGUgdGFibGUg KHRoZXJlYnkgcHJldmVudGluZwp0aGUgY29kZSBpbiBxdWVzdGlvbiBiZWluZyByZWFjaGVkLikK ClNvLCBJIHdvbmRlciB3aGV0aGVyIHdoYXQgeW91J3JlIHNlZWluZyBpcyBhIGxhdGVudCBidWcg d2hpY2ggaXMKYmVpbmcgdGlja2xlZCBieSB0aGUgcHJlc2VuY2Ugb2YgdGhlIENQVSBtYXNrcyBi ZWluZyBvZmYtc3RhY2sKY2hhbmdpbmcgdGhlIGtlcm5lbCB0aW1pbmcuCgpJIHdvdWxkIHN1Z2dl c3QgdGhlIHByaW50ayBkZWJ1ZyBhcHByb2FjaCBtYXkgaGVscCBoZXJlIHRvIHNlZSB3aGVuCnRo ZSBPUFBzIGFyZSBiZWd1biB0byBiZSBwYXJzZWQsIHdoZW4gdGhleSdyZSBjcmVhdGVkIGV0YyBh bmQgdGhlaXIKdGltaW5nIHJlbGF0aW9uc2hpcCB0byBiZWluZyB1c2VkLiBHaXZlbiB0aGUgc3Vz cGljaW9uLCBpdCdzIHBvc3NpYmxlCnRoYXQgdGhlIG1lcmUgYWRkaXRpb24gb2YgcHJpbnRrKCkg bWF5ICJmaXgiIHRoZSBwcm9ibGVtLCB3aGljaCBhZ2Fpbgp3b3VsZCBiZSBhbm90aGVyIHNlbWkt dXNlZnVsIGRhdGEgcG9pbnQuCgotLSAKUk1LJ3MgUGF0Y2ggc3lzdGVtOiBodHRwczovL3d3dy5h cm1saW51eC5vcmcudWsvZGV2ZWxvcGVyL3BhdGNoZXMvCkZUVFAgaXMgaGVyZSEgODBNYnBzIGRv d24gMTBNYnBzIHVwLiBEZWNlbnQgY29ubmVjdGl2aXR5IGF0IGxhc3QhCgpfX19fX19fX19fX19f X19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fXwpsaW51eC1hcm0ta2VybmVsIG1haWxp bmcgbGlzdApsaW51eC1hcm0ta2VybmVsQGxpc3RzLmluZnJhZGVhZC5vcmcKaHR0cDovL2xpc3Rz LmluZnJhZGVhZC5vcmcvbWFpbG1hbi9saXN0aW5mby9saW51eC1hcm0ta2VybmVsCg==