From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 26C43C76195 for ; Mon, 27 Mar 2023 18:31:50 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230075AbjC0Sbt (ORCPT ); Mon, 27 Mar 2023 14:31:49 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:41614 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229606AbjC0Sbq (ORCPT ); Mon, 27 Mar 2023 14:31:46 -0400 X-Greylist: delayed 900 seconds by postgrey-1.37 at lindbergh.monkeyblade.net; Mon, 27 Mar 2023 11:31:45 PDT Received: from mx2.n90.eu (mx.n90.eu [65.21.251.117]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 21E26271D; Mon, 27 Mar 2023 11:31:45 -0700 (PDT) Received: by mx2.n90.eu (Postfix, from userid 182) id 6277A1027DB45; Mon, 27 Mar 2023 18:06:10 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=n90.eu; s=default; t=1679940370; bh=OGbOFJayhgRcXq5FMeS6bRDyUolQeucd8waNYOur4j4=; h=References:From:To:Cc:Subject:Date:In-reply-to; b=nPD85CB4IjQubR+FqhGXUYhBxV7bbwiHtN9kDpUsMfvJ5VKjUsWV88RDwd2C5oy4/ 9tlJmIwPVi3rnu2uF27A0Xua2/i08lUSZ8Yll0MXwtsw+rld4069lz2IM8kPcwxggO phjBZkPfcy0QaShZxkSowFm5huqTrA2RmJl4unv3BE9WpLw5cHrd/vU7l+6slZ5l9/ p7m9eSvV87wcasXYp44lZW2pwDpGg8ecJrJGUayyE9ySjUesRCJHONIfC7Dbqvhl94 Nfe8MHZJvedo8D6w1a4cBrMOGJeF/f9TiWznHy5d8o12ZS2UBdgtiZxheA/fa2BBpK PygFgKG/87wsA== Received: from spica (unknown [172.20.188.202]) by mx2.n90.eu (Postfix) with ESMTP id CD5411025BB5A; Mon, 27 Mar 2023 18:06:07 +0000 (UTC) References: <20230327143359.GA2834753@bhelgaas> User-agent: mu4e 1.8.14; emacs 30.0.50 From: Aleksander Trofimowicz To: Keith Busch Cc: Bjorn Helgaas , Jens Axboe , Christoph Hellwig , Sagi Grimberg , Lukas Wunner , linux-pci@vger.kernel.org, linux-nvme@lists.infradead.org, linux-kernel@vger.kernel.org Subject: Re: [bugzilla-daemon@kernel.org: [Bug 217251] New: pciehp: nvme not visible after re-insert to tbt port] Date: Mon, 27 Mar 2023 17:43:18 +0000 In-reply-to: X-Mailer: boring 1.0 Message-ID: <871qlank6o.fsf@n90.eu> MIME-Version: 1.0 Content-Type: text/plain Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Keith Busch writes: > On Mon, Mar 27, 2023 at 09:33:59AM -0500, Bjorn Helgaas wrote: >> Forwarding to NVMe folks, lists for visibility. >> >> ----- Forwarded message from bugzilla-daemon@kernel.org ----- >> >> https://bugzilla.kernel.org/show_bug.cgi?id=217251 >> ... >> >> Created attachment 304031 >> --> https://bugzilla.kernel.org/attachment.cgi?id=304031&action=edit >> the tracing of nvme_pci_enable() during re-insertion >> >> Hi, >> >> There is a JHL7540-based device that may host a NVMe device. After the first >> insertion a nvme drive is properly discovered and handled by the relevant >> modules. Once disconnected any further attempts are not successful. The device >> is visible on a PCI bus, but nvme_pci_enable() ends up calling >> pci_disable_device() every time; the runtime PM status of the device is >> "suspended", the power status of the 04:01.0 PCI bridge is D3. Preventing the >> device from being power managed ("on" -> /sys/devices/../power/control) >> combined with device removal and pci rescan changes nothing. A host reboot >> restores the initial state. >> >> I would appreciate any suggestions how to debug it further. > > Sounds the same as this report: > > http://lists.infradead.org/pipermail/linux-nvme/2023-March/038259.html > > The driver is bailing on the device because we can't read it's status register > out of the remapped BAR. There's nothing we can do about that from the nvme > driver level. Memory mapped IO has to work in order to proceed. > Thanks. I can confirm it is the same problem: a) the platform is Intel Alderlake b) readl(dev->bar + NVME_REG_CSTS) in nvme_pci_enable() fails c) reading BAR0 via setpci gives 0x00000004 -- at