From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id ABFF728387; Wed, 8 May 2024 23:33:04 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1715211184; cv=none; b=R/flGOXWLd1H+bcl36bSe2hySg2GGFSkZnF8K7NlDm5a9PCi2HLIB5DdA08idHm12e610V5JLm5ALfdAAZbTQaHOs+YRYZSSOA/GX0WcARjqVGnW5gRhuhZZp64OqDH8xQFaUb+ISZZUxSXAnaUp0H90IVIYA4OwdVT9mdsWlqQ= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1715211184; c=relaxed/simple; bh=rymFRL9RenzD7dHMXH9iTV136lK+n6YZqLAw1D9IOc8=; h=Date:From:To:Cc:Subject:Message-ID:MIME-Version:Content-Type: Content-Disposition:In-Reply-To; b=Q8tp2p/EKI90FMk+xWyk7sbWfEd7EINrCzbcGXmnEBsXCMvorb4inypj8U1WoTkqdW7BNeVkx684Hbz/NSGj0Md7rIbRs1OqDaQSe3AzWTEuN1sfVaABjipFlx24GSoH+ESiKqWiab3KmghCJTkmDvhrWoWI0T9/RdG5GGRWv5Y= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=CxhTdRwk; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="CxhTdRwk" Received: by smtp.kernel.org (Postfix) with ESMTPSA id EF799C113CC; Wed, 8 May 2024 23:33:03 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1715211184; bh=rymFRL9RenzD7dHMXH9iTV136lK+n6YZqLAw1D9IOc8=; h=Date:From:To:Cc:Subject:In-Reply-To:From; b=CxhTdRwkIoIHjNCan0EzKcUVMCxw2dMCjbzheFWQvigKFzGghLCSSOemAlAC6Dxud nAUdQmMOva5iragqXcCT1EAjDe/Pav1c+wCI7LgXhsMbNw4PCZZViYm11GvPQgxHQK D7qkFBVzHRof0oRjK/vPCsAJ7bKUqr6pCvTnVlKtwi86s7qOfzUcD83Er0C2omZGFT y2JQKTGOnFJr+eyZp4BFMVUmUVoCgca9SdCGzU77Dkopr+6PvgysAvQZr+++Zg2umX neNtqGlXZ/E5+JeX73T2EactCiUOM4/f7eBwAGUFtvK8LdEbPPbPI+ZYtVrIinc5v3 oZ3FthdxurO3A== Date: Wed, 8 May 2024 18:33:02 -0500 From: Bjorn Helgaas To: Jim Quinlan Cc: linux-pci@vger.kernel.org, Nicolas Saenz Julienne , Bjorn Helgaas , Lorenzo Pieralisi , Cyril Brulebois , Phil Elwell , bcm-kernel-feedback-list@broadcom.com, Florian Fainelli , Jim Quinlan , Lorenzo Pieralisi , Krzysztof =?utf-8?Q?Wilczy=C5=84ski?= , Rob Herring , "moderated list:BROADCOM BCM2711/BCM2835 ARM ARCHITECTURE" , "moderated list:BROADCOM BCM2711/BCM2835 ARM ARCHITECTURE" , open list Subject: Re: [PATCH v9 4/4] PCI: brcmstb: Configure HW CLKREQ# mode appropriate for downstream device Message-ID: <20240508233302.GA1792067@bhelgaas> Precedence: bulk X-Mailing-List: linux-pci@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Disposition: inline Content-Transfer-Encoding: 8bit In-Reply-To: On Wed, May 08, 2024 at 01:55:24PM -0400, Jim Quinlan wrote: > On Mon, May 6, 2024 at 7:20 PM Bjorn Helgaas wrote: > ... > > As a user, how do I determine which setting to use? > > Using the "safe" mode will always work. In fact I considered making > this the default mode. > As I said, I cannot enumerate all of the reasons why one mode works > and one does not for a particular device+board+connector combo. The > HW folks have not really been forthcoming on the reasons as well. > > > Trial and error? If so, how do I identify the errors? > > Either PCIe link-up is not happening, or it is happening but the > device driver is non-functional and boot typically hangs. What I'm hearing is that it's trial and error. If we can't tell users how to figure out which mode to use, I think we have to explicitly say "try the modes in this order until you find one that works." That sucks, but if it's all we can do, I guess we don't have much choice, and we should just own up to it. There's no point in telling users "if your card drives CLKREQ# use X, but if not and it can tolerate out-of-spec T_CLRon timing, use Y" because nobody knows how to figure that out. And we can say which features are enabled in each mode so they aren't surprised, e.g., something like this: "default" -- The Root Port supports ASPM L0s, L1, L1 Substates, and Clock Power Management. This provides the best power savings but some devices may not work correctly because the Root Port doesn't comply with T_CLRon timing required for PCIe Mini Cards [1]. "no-l1ss" -- The Root Port supports ASPM L0s, L1 (but not L1 Substates), and Clock Power Management. [I assume there's some other Root Port defect that causes issues with some devices in this mode; I dunno. If we don't know exactly what it is, I guess we can't really say anything.] "safe" -- The Root Port supports ASPM L0, L1, L1 Substates, but not Clock Power Management. All devices should work in this mode. [1] PCIe Mini CEM r2.1, sec 3.2.5.2.2 (I'm not sure which features are *actually* enabled in each mode, I just guessed.) Bjorn From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id C82AEC04FFE for ; Wed, 8 May 2024 23:33:27 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:In-Reply-To:MIME-Version:Message-ID: Subject:Cc:To:From:Date:Reply-To:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:References: List-Owner; bh=L3zhj21KSdUy7Zy9t1K2uC1Mtj/ujtyBC48G8T4ri48=; b=S6VfntMq6oRiNA ygmyi/c+o3MTiR3985deuFrx/NfJgGQGuIEVbhkf5tm8DzAikj0uOXnAazh6Fur3hrngZqkqV66jh InxmF5H3UPMULpDhGOGWvKM/Fc1N78GiEF+JksymCSmBKk6Ed3DLbxEuGHUiMCmeHnLUwuP8yi/2s tV9RWc1uNdl8IlervqKvNONxfST1NpVfUHvUtc6vrCCDPLjIkNWlNSVd1ivU+ZI7HFylEaG5l8cvK lrXoSPqOOakleerG6FPq6x6C6pIyfCritOxXRjqTl0c6ql7t6tCOxe2TJyMx/gmI1mqwZleQzfKxj quN3C9zX9CmGSkQ7M7OQ==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.97.1 #2 (Red Hat Linux)) id 1s4qmc-0000000HHCd-0TK2; Wed, 08 May 2024 23:33:10 +0000 Received: from sin.source.kernel.org ([145.40.73.55]) by bombadil.infradead.org with esmtps (Exim 4.97.1 #2 (Red Hat Linux)) id 1s4qmY-0000000HHBm-3KXe; Wed, 08 May 2024 23:33:08 +0000 Received: from smtp.kernel.org (transwarp.subspace.kernel.org [100.75.92.58]) by sin.source.kernel.org (Postfix) with ESMTP id 03CE6CE1A72; Wed, 8 May 2024 23:33:05 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id EF799C113CC; Wed, 8 May 2024 23:33:03 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1715211184; bh=rymFRL9RenzD7dHMXH9iTV136lK+n6YZqLAw1D9IOc8=; h=Date:From:To:Cc:Subject:In-Reply-To:From; b=CxhTdRwkIoIHjNCan0EzKcUVMCxw2dMCjbzheFWQvigKFzGghLCSSOemAlAC6Dxud nAUdQmMOva5iragqXcCT1EAjDe/Pav1c+wCI7LgXhsMbNw4PCZZViYm11GvPQgxHQK D7qkFBVzHRof0oRjK/vPCsAJ7bKUqr6pCvTnVlKtwi86s7qOfzUcD83Er0C2omZGFT y2JQKTGOnFJr+eyZp4BFMVUmUVoCgca9SdCGzU77Dkopr+6PvgysAvQZr+++Zg2umX neNtqGlXZ/E5+JeX73T2EactCiUOM4/f7eBwAGUFtvK8LdEbPPbPI+ZYtVrIinc5v3 oZ3FthdxurO3A== Date: Wed, 8 May 2024 18:33:02 -0500 From: Bjorn Helgaas To: Jim Quinlan Cc: linux-pci@vger.kernel.org, Nicolas Saenz Julienne , Bjorn Helgaas , Lorenzo Pieralisi , Cyril Brulebois , Phil Elwell , bcm-kernel-feedback-list@broadcom.com, Florian Fainelli , Jim Quinlan , Lorenzo Pieralisi , Krzysztof =?utf-8?Q?Wilczy=C5=84ski?= , Rob Herring , "moderated list:BROADCOM BCM2711/BCM2835 ARM ARCHITECTURE" , "moderated list:BROADCOM BCM2711/BCM2835 ARM ARCHITECTURE" , open list Subject: Re: [PATCH v9 4/4] PCI: brcmstb: Configure HW CLKREQ# mode appropriate for downstream device Message-ID: <20240508233302.GA1792067@bhelgaas> MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20240508_163307_214075_F76BB3A2 X-CRM114-Status: GOOD ( 20.34 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: base64 Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org T24gV2VkLCBNYXkgMDgsIDIwMjQgYXQgMDE6NTU6MjRQTSAtMDQwMCwgSmltIFF1aW5sYW4gd3Jv dGU6Cj4gT24gTW9uLCBNYXkgNiwgMjAyNCBhdCA3OjIw4oCvUE0gQmpvcm4gSGVsZ2FhcyA8aGVs Z2Fhc0BrZXJuZWwub3JnPiB3cm90ZToKPiAuLi4KCj4gPiBBcyBhIHVzZXIsIGhvdyBkbyBJIGRl dGVybWluZSB3aGljaCBzZXR0aW5nIHRvIHVzZT8KPgo+IFVzaW5nIHRoZSAic2FmZSIgbW9kZSB3 aWxsIGFsd2F5cyB3b3JrLiAgSW4gZmFjdCBJIGNvbnNpZGVyZWQgbWFraW5nCj4gdGhpcyB0aGUg ZGVmYXVsdCBtb2RlLgoKPiBBcyBJIHNhaWQsIEkgY2Fubm90IGVudW1lcmF0ZSBhbGwgb2YgdGhl IHJlYXNvbnMgd2h5IG9uZSBtb2RlIHdvcmtzCj4gYW5kIG9uZSBkb2VzIG5vdCBmb3IgYSBwYXJ0 aWN1bGFyIGRldmljZStib2FyZCtjb25uZWN0b3IgY29tYm8uICBUaGUKPiBIVyBmb2xrcyBoYXZl IG5vdCByZWFsbHkgYmVlbiBmb3J0aGNvbWluZyBvbiB0aGUgcmVhc29ucyBhcyB3ZWxsLgo+IAo+ ID4gVHJpYWwgYW5kIGVycm9yPyAgSWYgc28sIGhvdyBkbyBJIGlkZW50aWZ5IHRoZSBlcnJvcnM/ Cj4KPiBFaXRoZXIgUENJZSBsaW5rLXVwIGlzIG5vdCBoYXBwZW5pbmcsIG9yIGl0IGlzIGhhcHBl bmluZyBidXQgdGhlCj4gZGV2aWNlIGRyaXZlciBpcyBub24tZnVuY3Rpb25hbCBhbmQgYm9vdCB0 eXBpY2FsbHkgIGhhbmdzLgoKV2hhdCBJJ20gaGVhcmluZyBpcyB0aGF0IGl0J3MgdHJpYWwgYW5k IGVycm9yLiAKCklmIHdlIGNhbid0IHRlbGwgdXNlcnMgaG93IHRvIGZpZ3VyZSBvdXQgd2hpY2gg bW9kZSB0byB1c2UsIEkgdGhpbmsgd2UKaGF2ZSB0byBleHBsaWNpdGx5IHNheSAidHJ5IHRoZSBt b2RlcyBpbiB0aGlzIG9yZGVyIHVudGlsIHlvdSBmaW5kIG9uZQp0aGF0IHdvcmtzLiIKClRoYXQg c3Vja3MsIGJ1dCBpZiBpdCdzIGFsbCB3ZSBjYW4gZG8sIEkgZ3Vlc3Mgd2UgZG9uJ3QgaGF2ZSBt dWNoCmNob2ljZSwgYW5kIHdlIHNob3VsZCBqdXN0IG93biB1cCB0byBpdC4KClRoZXJlJ3Mgbm8g cG9pbnQgaW4gdGVsbGluZyB1c2VycyAiaWYgeW91ciBjYXJkIGRyaXZlcyBDTEtSRVEjIHVzZSBY LApidXQgaWYgbm90IGFuZCBpdCBjYW4gdG9sZXJhdGUgb3V0LW9mLXNwZWMgVF9DTFJvbiB0aW1p bmcsIHVzZSBZIgpiZWNhdXNlIG5vYm9keSBrbm93cyBob3cgdG8gZmlndXJlIHRoYXQgb3V0LgoK QW5kIHdlIGNhbiBzYXkgd2hpY2ggZmVhdHVyZXMgYXJlIGVuYWJsZWQgaW4gZWFjaCBtb2RlIHNv IHRoZXkgYXJlbid0CnN1cnByaXNlZCwgZS5nLiwgc29tZXRoaW5nIGxpa2UgdGhpczoKCiAgImRl ZmF1bHQiIC0tIFRoZSBSb290IFBvcnQgc3VwcG9ydHMgQVNQTSBMMHMsIEwxLCBMMSBTdWJzdGF0 ZXMsIGFuZAogICAgQ2xvY2sgUG93ZXIgTWFuYWdlbWVudC4gIFRoaXMgcHJvdmlkZXMgdGhlIGJl c3QgcG93ZXIgc2F2aW5ncyBidXQKICAgIHNvbWUgZGV2aWNlcyBtYXkgbm90IHdvcmsgY29ycmVj dGx5IGJlY2F1c2UgdGhlIFJvb3QgUG9ydCBkb2Vzbid0CiAgICBjb21wbHkgd2l0aCBUX0NMUm9u IHRpbWluZyByZXF1aXJlZCBmb3IgUENJZSBNaW5pIENhcmRzIFsxXS4KCiAgIm5vLWwxc3MiIC0t IFRoZSBSb290IFBvcnQgc3VwcG9ydHMgQVNQTSBMMHMsIEwxIChidXQgbm90IEwxCiAgICBTdWJz dGF0ZXMpLCBhbmQgQ2xvY2sgUG93ZXIgTWFuYWdlbWVudC4gIFtJIGFzc3VtZSB0aGVyZSdzIHNv bWUKICAgIG90aGVyIFJvb3QgUG9ydCBkZWZlY3QgdGhhdCBjYXVzZXMgaXNzdWVzIHdpdGggc29t ZSBkZXZpY2VzIGluCiAgICB0aGlzIG1vZGU7IEkgZHVubm8uICBJZiB3ZSBkb24ndCBrbm93IGV4 YWN0bHkgd2hhdCBpdCBpcywgSSBndWVzcwogICAgd2UgY2FuJ3QgcmVhbGx5IHNheSBhbnl0aGlu Zy5dCgogICJzYWZlIiAtLSBUaGUgUm9vdCBQb3J0IHN1cHBvcnRzIEFTUE0gTDAsIEwxLCBMMSBT dWJzdGF0ZXMsIGJ1dCBub3QKICAgIENsb2NrIFBvd2VyIE1hbmFnZW1lbnQuICBBbGwgZGV2aWNl cyBzaG91bGQgd29yayBpbiB0aGlzIG1vZGUuCgpbMV0gUENJZSBNaW5pIENFTSByMi4xLCBzZWMg My4yLjUuMi4yCgooSSdtIG5vdCBzdXJlIHdoaWNoIGZlYXR1cmVzIGFyZSAqYWN0dWFsbHkqIGVu YWJsZWQgaW4gZWFjaCBtb2RlLCBJCmp1c3QgZ3Vlc3NlZC4pCgpCam9ybgoKX19fX19fX19fX19f X19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX18KbGludXgtYXJtLWtlcm5lbCBtYWls aW5nIGxpc3QKbGludXgtYXJtLWtlcm5lbEBsaXN0cy5pbmZyYWRlYWQub3JnCmh0dHA6Ly9saXN0 cy5pbmZyYWRlYWQub3JnL21haWxtYW4vbGlzdGluZm8vbGludXgtYXJtLWtlcm5lbAo=