From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-4.0 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS, URIBL_BLOCKED autolearn=no autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 10220C55178 for ; Thu, 29 Oct 2020 19:45:06 +0000 (UTC) Received: from merlin.infradead.org (merlin.infradead.org [205.233.59.134]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 8117620782 for ; Thu, 29 Oct 2020 19:45:05 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=lists.infradead.org header.i=@lists.infradead.org header.b="hYiQmNs8"; dkim=fail reason="signature verification failed" (1024-bit key) header.d=kernel.org header.i=@kernel.org header.b="LGGw3FyN" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 8117620782 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=kernel.org Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=merlin.20170209; h=Sender:Content-Transfer-Encoding: Content-Type:Cc:List-Subscribe:List-Help:List-Post:List-Archive: List-Unsubscribe:List-Id:In-Reply-To:MIME-Version:Message-ID:Subject:To:From: Date:Reply-To:Content-ID:Content-Description:Resent-Date:Resent-From: Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:References:List-Owner; bh=T5oFeEvoN5ti4oHSrpM3VXWfxKKY6BsPtgw5ulSvv8g=; b=hYiQmNs8+vZVZxl1O2dDOlknh LZQEs5PcB8kvRj1SwAEXiAf8xxNXlLs2EN86Oz56j4fR/1e4MMpufKkCOffjK6NuJXrnmV3vd/kjb guFepVZNYgzCBfjyPqQKcm/qAe9k5xSHjAO+PJi6+ws+UGsFFM6K3kEpLqShSfvsClsyeb6nukJCz TAuKexsc63CrDNixf05vkZnla7miZruhSFhYbEjMSc4oLaEJ1HatUZSoYJMyYsiMt+qQCuSVr1yJq lcaTIW6JEg4xHeN13HtJbBctJSN22uIwHqXFgFjU8rYh3d47ohnu4X6lWHJ2TDdrV1cYtdssR8k6j a03GThSLw==; Received: from localhost ([::1] helo=merlin.infradead.org) by merlin.infradead.org with esmtp (Exim 4.92.3 #3 (Red Hat Linux)) id 1kYDqL-0005cX-Td; Thu, 29 Oct 2020 19:44:18 +0000 Received: from mail.kernel.org ([198.145.29.99]) by merlin.infradead.org with esmtps (Exim 4.92.3 #3 (Red Hat Linux)) id 1kYDcw-00087d-ES for linux-arm-kernel@lists.infradead.org; Thu, 29 Oct 2020 19:30:27 +0000 Received: from localhost (230.sub-72-107-127.myvzw.com [72.107.127.230]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPSA id ED0D3204FD; Thu, 29 Oct 2020 19:30:23 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=default; t=1603999824; bh=MKfSe1ErAg1ensKCCgvIbXqQFpX3QbKj0Wf0wExX180=; h=Date:From:To:Cc:Subject:In-Reply-To:From; b=LGGw3FyNE/xfEtfU/rXpitDOkY0ZfMTLXTYs/a8KLT1a8GuhRBUxbDvZUbaXUNegd NRlvRDcdC9bYCHInrsKlOyoDyO3dynfK5Y3UJ8M0PNjvQkHz2fM44z1yUnZ2gaznkx MReo3sAVrr2UfSvJOXuUpVz/KGYsPDG/m9vWKv2I= Date: Thu, 29 Oct 2020 14:30:22 -0500 From: Bjorn Helgaas To: Toke =?iso-8859-1?Q?H=F8iland-J=F8rgensen?= Subject: Re: PCI trouble on mvebu (Turris Omnia) Message-ID: <20201029193022.GA476048@bjorn-Precision-5520> MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: <871rhhmgkq.fsf@toke.dk> X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20201029_153026_754647_B79F5F84 X-CRM114-Status: GOOD ( 23.13 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Rob Herring , Jason Cooper , Pali =?iso-8859-1?Q?Roh=E1r?= , Ilias Apalodimas , Marek =?iso-8859-1?Q?Beh=FAn?= , Thomas Petazzoni , linux-pci@vger.kernel.org, vtolkm@gmail.com, linux-arm-kernel@lists.infradead.org Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org On Thu, Oct 29, 2020 at 12:12:21PM +0100, Toke H=F8iland-J=F8rgensen wrote: > Pali Roh=E1r writes: > > I have been testing mainline kernel on Turris Omnia with two PCIe > > default cards (WLE200 and WLE900) and it worked fine. But I do not know > > if I had ASPM enabled or not. > > > > So it is working fine for you when CONFIG_PCIEASPM is disabled and whole > > issue is only when CONFIG_PCIEASPM is enabled? > = > Yup, exactly. And I'm also currently testing with the default WLE200/900 > cards... I just tried sticking an MT76-based WiFi card into the third > PCI slot, and that doesn't come up either when I enable PCIEASPM. Huh. So IIUC, the following cases all try to retrain the link and it fails to come up again: - aardvark + WLE900VX (see commit 43fc679ced18) - mvebu + WLE200 - mvebu + WLE900 - mvebu + MT76 In all these cases, Linux was able to enumerate the NIC, which means the link was up when firmware handed it off. I think Linux decided the Common Clock Configuration was wrong, so it tried to fix it and retrain the link, and the link didn't come back up. I don't have "lspci -vv" output from all of them, but in vtolkm's case, the firmware handed off with: 00:02.0 Root Port to [bus 02] SlotClk+ CommClk+ 02:00.0 QCA986x/988x NIC SlotClk+ CommClk- Per spec (PCIe r5, sec 7.5.3.7), SlotClk is HwInit and CommClk is RW and should power up as 0. If I'm reading the implementation note correctly, if SlotClk is set on both ends of the link, software should set CommClk, so the config above *does* look wrong, and CommClk+ on the Root Port suggests that firmware set it. I think both the aardvark and mvebu systems probably use U-Boot. I don't know U-Boot at all, but I don't see anything in it that touches Link Control. I'm curious what happens if you put one of these cards in a PC. If anybody tries it, please collect the "sudo lspci -vv" and dmesg output. We could quirk these NICs to avoid the retrain, but since aardvark and mvebu have no obvious connection and WLE200/WLE900 and MT76 have no obvious connection, I doubt there's a simple hardware defect that explains all these. = Maybe we're doing something wrong in the retrain, but obviously the link came up in the first place. AFAIK the only thing we're changing is the CommClk setting, and that looks legitimate per spec. Another experiment: build kernel without CONFIG_PCIEASPM, set $ROOT and $NIC appropriately, and try the following: # Set $ROOT and $NIC (update to match your system): # ROOT=3D00:02.0 # NIC=3D02:00.0 # Dump the Root Port and NIC Link registers: # setpci -s$ROOT CAP_EXP+0xc.l # Link Capabilities # setpci -s$ROOT CAP_EXP+0x10.w # Link Control # setpci -s$ROOT CAP_EXP+0x12.w # Link Status # setpci -s$NIC CAP_EXP+0xc.l # Link Capabilities # setpci -s$NIC CAP_EXP+0x10.w # Link Control # setpci -s$NIC CAP_EXP+0x12.w # Link Status # Retrain the link: # setpci -s$ROOT CAP_EXP+0x10.w=3D0x0020 # Link Control Retrain Li= nk # sleep 1 # setpci -s$ROOT CAP_EXP+0x12.w # Link Status # setpci -s$NIC CAP_EXP+0x12.w # Link Status # Set CommClk+ and retrain the link: # setpci -s$NIC CAP_EXP+0x10.w=3D0x0040 # Link Control Common Clo= ck # setpci -s$ROOT CAP_EXP+0x10.w=3D0x0040 # Link Control Common Clo= ck # setpci -s$ROOT CAP_EXP+0x10.w=3D0x0060 # Link Control RL + CC # sleep 1 # setpci -s$ROOT CAP_EXP+0x12.w # Link Status # setpci -s$NIC CAP_EXP+0x12.w # Link Status _______________________________________________ linux-arm-kernel mailing list linux-arm-kernel@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-arm-kernel