From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id E135AD3EE75 for ; Thu, 22 Jan 2026 15:29:16 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender:List-Subscribe:List-Help :List-Post:List-Archive:List-Unsubscribe:List-Id:In-Reply-To:Content-Type: MIME-Version:Message-ID:Subject:Cc:To:From:Date:Reply-To: Content-Transfer-Encoding:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:References: List-Owner; bh=XbLFktTgUVhlXS7hsHZ44jYzUaVrVObn3A/c5wttcmw=; b=YenkUHUUQo4t6w F5hA90Ic5rfcvsPHOCYa6RCxXkC5n0TtTgFp5uWYRezinY0H0VT6XKOLM+4Z/a+J5+Gv+HWQGJwJX UkWwLNdrAatyOODys5163gjMgBAm+1b+Bm5SWn97uRGrTu5+1zU4+OvRFM8LMPSK35QZuk6q/nKsn nx3N2vAJm42Dc49YDFerXlWSfyAqeTMZobp38+JEmpHxVBBnaOScKlZ5lvBOEv2uqZc+59klRZ5Wk lfdfCxG2mJuTx0+q2j0FCuvfsIPeSD1PwU+mBkSJrtiRmBFcxq3kDii5NuzfoaSE6i/73lzrLi6EU 4r7ZMCEN3ZJHsPQDR1HQ==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.98.2 #2 (Red Hat Linux)) id 1viwcO-00000007L5l-3v75; Thu, 22 Jan 2026 15:29:08 +0000 Received: from tor.source.kernel.org ([172.105.4.254]) by bombadil.infradead.org with esmtps (Exim 4.98.2 #2 (Red Hat Linux)) id 1viwcN-00000007L5e-2jj6 for linux-nvme@lists.infradead.org; Thu, 22 Jan 2026 15:29:07 +0000 Received: from smtp.kernel.org (transwarp.subspace.kernel.org [100.75.92.58]) by tor.source.kernel.org (Postfix) with ESMTP id 6F9C460053; Thu, 22 Jan 2026 15:29:05 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id EE824C116C6; Thu, 22 Jan 2026 15:29:04 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1769095745; bh=vYjhezx8RBYrZaQh5Oid2wtb4ZoiGXx6HNPQ9eDro9w=; h=Date:From:To:Cc:Subject:In-Reply-To:From; b=r9RIThb7pS8xgBhPwF6j7rujxOlpGK87EDpxtnKQWiX/kMFD7/1HzsMUNgEf8eVpS Z97YIo9I3WxQWVAAi+Ntp2bI5zcFcogD46jybFOBK2thytrqmLuoJHF0at0Lpkktvd C50JSZHao1pYE8mN7r725zBWUSidfJXAWE8qHrl3+WMLIf52IQHjz/Q5IPn0UTr2uk m5AEBZbfcfeMVkjyZ72imyixCHrppf8FWodV1EZ5pWGvNeGt/9u1DmXpvk8tbf8T0y e3rtCDPK//aK9ftaHcGYC6h9++sOAmDxrAWTiTJz7+jxx95PZPT2xgUkst+wL9+Kv8 MI+3qjq6ef4Nw== Date: Thu, 22 Jan 2026 09:29:03 -0600 From: Bjorn Helgaas To: Jon Hunter Cc: manivannan.sadhasivam@oss.qualcomm.com, Bjorn Helgaas , Manivannan Sadhasivam , Lorenzo Pieralisi , Krzysztof =?utf-8?Q?Wilczy=C5=84ski?= , Rob Herring , linux-pci@vger.kernel.org, linux-kernel@vger.kernel.org, linux-arm-msm@vger.kernel.org, "David E. Box" , Kai-Heng Feng , "Rafael J. Wysocki" , Heiner Kallweit , Chia-Lin Kao , "linux-tegra@vger.kernel.org" , Keith Busch , Jens Axboe , Christoph Hellwig , Sagi Grimberg , linux-nvme@lists.infradead.org Subject: Re: [PATCH v2 1/2] PCI/ASPM: Override the ASPM and Clock PM states set by BIOS for devicetree platforms Message-ID: <20260122152903.GA1247682@bhelgaas> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <7306256a-b380-489b-8248-b774e6d3d80e@nvidia.com> X-BeenThere: linux-nvme@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "Linux-nvme" Errors-To: linux-nvme-bounces+linux-nvme=archiver.kernel.org@lists.infradead.org [+cc NVMe folks] On Thu, Jan 22, 2026 at 12:12:42PM +0000, Jon Hunter wrote: > ... > Since this commit was added in Linux v6.18, I have been observing a suspend > test failures on some of our boards. The suspend test suspends the devices > for 20 secs and before this change the board would resume in about ~27 secs > (including the 20 sec sleep). After this change the board would take over 80 > secs to resume and this triggered a failure. > > Looking at the logs, I can see it is the NVMe device on the board that is > having an issue, and I see the reset failing ... > > [ 945.754939] r8169 0007:01:00.0 enP7p1s0: Link is Up - 1Gbps/Full - > flow control rx/tx > [ 1002.467432] nvme nvme0: I/O tag 12 (400c) opcode 0x9 (Admin Cmd) QID > 0 timeout, reset controller > [ 1002.493713] nvme nvme0: 12/0/0 default/read/poll queues > [ 1003.050448] nvme nvme0: ctrl state 1 is not RESETTING > [ 1003.050481] OOM killer enabled. > [ 1003.054035] nvme nvme0: Disabling device after reset failure: -19 > > From the above timestamps the delay is coming from the NVMe. I see this > issue on several boards with different NVMe devices and I can workaround > this by disabling ASPM L0/L1 for these devices ... > > DECLARE_PCI_FIXUP_HEADER(0x15b7, 0x5011, quirk_disable_aspm_l0s_l1); > DECLARE_PCI_FIXUP_HEADER(0x15b7, 0x5036, quirk_disable_aspm_l0s_l1); > DECLARE_PCI_FIXUP_HEADER(0x1b4b, 0x1322, quirk_disable_aspm_l0s_l1); > DECLARE_PCI_FIXUP_HEADER(0xc0a9, 0x540a, quirk_disable_aspm_l0s_l1); > > I am curious if you have seen any similar issues? > > Other PCIe devices seem to be OK (like the realtek r8169) but just > the NVMe is having issues. So I am trying to figure out the best way > to resolve this? For context, "this commit" refers to f3ac2ff14834, modified by df5192d9bb0e: f3ac2ff14834 ("PCI/ASPM: Enable all ClockPM and ASPM states for devicetree platforms") df5192d9bb0e ("PCI/ASPM: Enable only L0s and L1 for devicetree platforms") The fact that this suspend issue only affects NVMe reminds me of the code in dw_pcie_suspend_noirq() [1] that bails out early if L1 is enabled because of some NVMe expectation: dw_pcie_suspend_noirq() { ... /* * If L1SS is supported, then do not put the link into L2 as some * devices such as NVMe expect low resume latency. */ if (dw_pcie_readw_dbi(pci, offset + PCI_EXP_LNKCTL) & PCI_EXP_LNKCTL_ASPM_L1) return 0; ... That suggests there's some NVMe/ASPM interaction that the PCI core doesn't understand yet. [1] https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/drivers/pci/controller/dwc/pcie-designware-host.c?id=v6.18#n1146