From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from phobos.denx.de (phobos.denx.de [85.214.62.61]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id D0913C433EF for ; Sat, 15 Jan 2022 12:37:55 +0000 (UTC) Received: from h2850616.stratoserver.net (localhost [IPv6:::1]) by phobos.denx.de (Postfix) with ESMTP id 8403D834D2; Sat, 15 Jan 2022 13:37:48 +0100 (CET) Authentication-Results: phobos.denx.de; dmarc=none (p=none dis=none) header.from=konsulko.com Authentication-Results: phobos.denx.de; spf=pass smtp.mailfrom=u-boot-bounces@lists.denx.de Authentication-Results: phobos.denx.de; dkim=pass (1024-bit key; unprotected) header.d=konsulko.com header.i=@konsulko.com header.b="qbe0Wyrb"; dkim-atps=neutral Received: by phobos.denx.de (Postfix, from userid 109) id C137883496; Sat, 15 Jan 2022 13:37:46 +0100 (CET) Received: from mail-qk1-x72b.google.com (mail-qk1-x72b.google.com [IPv6:2607:f8b0:4864:20::72b]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits)) (No client certificate requested) by phobos.denx.de (Postfix) with ESMTPS id 68C6B83461 for ; Sat, 15 Jan 2022 13:37:42 +0100 (CET) Authentication-Results: phobos.denx.de; dmarc=none (p=none dis=none) header.from=konsulko.com Authentication-Results: phobos.denx.de; spf=pass smtp.mailfrom=trini@konsulko.com Received: by mail-qk1-x72b.google.com with SMTP id j85so13792152qke.2 for ; Sat, 15 Jan 2022 04:37:42 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=konsulko.com; s=google; h=date:from:to:cc:subject:message-id:references:mime-version :content-disposition:in-reply-to; bh=BPTNu8slu8FP/OKim2OrhNhOEw0S3HEotzqlibM1kd0=; b=qbe0WyrbbYISaraHzgJTETphCJT8EhybJOTEz6xULc6yGohILUnaHr4SiHIB9oglY7 Pv6Gqdh7+jUlLDkpEe5xDsJymqvvRL/qQoXFv+jNhxul2kZCu5Ko7jqV8hPTtqtNa3bQ sDZ39JVCZdDRdujTzvT/QT1i9hkY6nRqpiIZc= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:date:from:to:cc:subject:message-id:references :mime-version:content-disposition:in-reply-to; bh=BPTNu8slu8FP/OKim2OrhNhOEw0S3HEotzqlibM1kd0=; b=cynZDWGRJIflni57M323pTFnoLQGk2cFOT95zwEj68nVXgiGhUJaFAT2MYGxtPwOvY o2nzZSrViHcI5MHPS7t2TvJ/2J8AaxbvAk/qsJEa9dqco/SyVkKfuJmAGdZ3jbuTuVUx 9oSYo9hsvb+0WFjf4V3UzPGJRmdUMn5mvuI4/5N0EJSHJ9HywxwHvX85yO/o4d1qX1Mc oT7eRjzFi63ht8jKsEOkYz1fOhqHCV3NspH4NoRkDjCJ+gXbt/ktrm7zgxGsYtb9ruLt vbXu18Qji5NsXNtSbwVc/wpwGiO4ScH8S9EAHylTzh7nnrBFFO40Pd3yRSa1oc9Fpsrb CKbQ== X-Gm-Message-State: AOAM532rI8hNuwpbIcwCgmCsnzkHv8i6mkoAwGo8jsa9ICI8powC1ZLY STUky138ez8ls6TS8OKQYl4jBQ== X-Google-Smtp-Source: ABdhPJzCD3Fht80fArgFtenoRU56GTdv2r7CU7qcdIqDQoGWxjMg3KiMOfu8v5lUJq7SMaJt03xQdA== X-Received: by 2002:a05:620a:8dc:: with SMTP id z28mr4415041qkz.723.1642250261156; Sat, 15 Jan 2022 04:37:41 -0800 (PST) Received: from bill-the-cat (2603-6081-7b01-cbda-9533-0c51-a891-732b.res6.spectrum.com. [2603:6081:7b01:cbda:9533:c51:a891:732b]) by smtp.gmail.com with ESMTPSA id b11sm5914904qtx.85.2022.01.15.04.37.39 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sat, 15 Jan 2022 04:37:40 -0800 (PST) Date: Sat, 15 Jan 2022 07:37:38 -0500 From: Tom Rini To: "Maciej W. Rozycki" Cc: u-boot@lists.denx.de, Stefan Roese , Simon Glass , Pali =?iso-8859-1?Q?Roh=E1r?= , Phil Sutter , Vladimir Oltean , Bin Meng , Tim Harvey , Jim Wilson , David Abdurachmanov Subject: Re: [PATCH v3] pci: Work around PCIe link training failures Message-ID: <20220115123738.GW9207@bill-the-cat> References: MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha512; protocol="application/pgp-signature"; boundary="bDTAc/+PmGfS/M5J" Content-Disposition: inline In-Reply-To: X-Clacks-Overhead: GNU Terry Pratchett X-BeenThere: u-boot@lists.denx.de X-Mailman-Version: 2.1.39 Precedence: list List-Id: U-Boot discussion List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: u-boot-bounces@lists.denx.de Sender: "U-Boot" X-Virus-Scanned: clamav-milter 0.103.2 at phobos.denx.de X-Virus-Status: Clean --bDTAc/+PmGfS/M5J Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Sat, Nov 20, 2021 at 11:03:30PM +0000, Maciej W. Rozycki wrote: > Attempt to handle cases with a downstream port of a PCIe switch where > link training never completes and the link continues switching between=20 > speeds indefinitely with the data link layer never reaching the active=20 > state. >=20 > It has been observed with a downstream port of the ASMedia ASM2824 Gen 3= =20 > switch wired to the upstream port of the Pericom PI7C9X2G304 Gen 2=20 > switch, using a Delock Riser Card PCI Express x1 > 2 x PCIe x1 device,=20 > P/N 41433, wired to a SiFive HiFive Unmatched board. In this setup the= =20 > switches are supposed to negotiate the link speed of preferably 5.0GT/s,= =20 > falling back to 2.5GT/s. >=20 > However the link continues oscillating between the two speeds, at the=20 > rate of 34-35 times per second, with link training reported repeatedly=20 > active ~84% of the time, e.g.: >=20 > 02:03.0 PCI bridge [0604]: ASMedia Technology Inc. ASM2824 PCIe Gen3 Pack= et Switch [1b21:2824] (rev 01) (prog-if 00 [Normal decode]) > [...] > Bus: primary=3D02, secondary=3D05, subordinate=3D05, sec-latency=3D0 > [...] > Capabilities: [80] Express (v2) Downstream Port (Slot+), MSI 00 > [...] > LnkSta: Speed 5GT/s (downgraded), Width x1 (ok) > TrErr- Train+ SlotClk+ DLActive- BWMgmt+ ABWMgmt- > [...] > LnkCtl2: Target Link Speed: 8GT/s, EnterCompliance- SpeedDis+, Selectab= le De-emphasis: -3.5dB > Transmit Margin: Normal Operating Range, EnterModifiedCompliance- Com= plianceSOS- > Compliance De-emphasis: -6dB > [...] >=20 > Forcibly limiting the target link speed to 2.5GT/s with the upstream=20 > ASM2824 device makes the two switches communicate correctly however: >=20 > 02:03.0 PCI bridge [0604]: ASMedia Technology Inc. ASM2824 PCIe Gen3 Pack= et Switch [1b21:2824] (rev 01) (prog-if 00 [Normal decode]) > [...] > Bus: primary=3D02, secondary=3D05, subordinate=3D09, sec-latency=3D0 > [...] > Capabilities: [80] Express (v2) Downstream Port (Slot+), MSI 00 > [...] > LnkSta: Speed 2.5GT/s (downgraded), Width x1 (ok) > TrErr- Train- SlotClk+ DLActive+ BWMgmt- ABWMgmt- > [...] > LnkCtl2: Target Link Speed: 2.5GT/s, EnterCompliance- SpeedDis+, Select= able De-emphasis: -3.5dB > Transmit Margin: Normal Operating Range, EnterModifiedCompliance- Com= plianceSOS- > Compliance De-emphasis: -6dB > [...] >=20 > and then: >=20 > 05:00.0 PCI bridge [0604]: Pericom Semiconductor PI7C9X2G304 EL/SL PCIe2 = 3-Port/4-Lane Packet Switch [12d8:2304] (rev 05) (prog-if 00 [Normal decode= ]) > [...] > Bus: primary=3D05, secondary=3D06, subordinate=3D09, sec-latency=3D0 > [...] > Capabilities: [c0] Express (v2) Upstream Port, MSI 00 > [...] > LnkSta: Speed 2.5GT/s (downgraded), Width x1 (downgraded) > TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt- > [...] > LnkCtl2: Target Link Speed: 5GT/s, EnterCompliance- SpeedDis- > Transmit Margin: Normal Operating Range, EnterModifiedCompliance- Com= plianceSOS- > Compliance De-emphasis: -6dB > [...] >=20 > Make use of this observation then and attempt to detect the inability to= =20 > negotiate the link speed automatically, and then handle it by hand. Use= =20 > the Data Link Layer Link Active status flag as the primary indicator of= =20 > successful link speed negotiation, but given that the flag is optional=20 > by hardware to implement (the ASM2824 does have it though), resort to=20 > checking for the mandatory Link Bandwidth Management Status flag showing= =20 > that the link speed or width has been changed in an attempt to correct=20 > unreliable link operation (the ASM2824 does set it too). >=20 > If these checks indicate that link may not operate correctly, then poll= =20 > the Data Link Layer Link Active status flag along with the Link Training= =20 > flag for the duration of 200ms to see if the link has stabilised, that=20 > is either that the Data Link Layer Link Active status flag has been set= =20 > or that Link Training has been inactive during at least the second half= =20 > of the interval. >=20 > If that has indicated failure, restrict the target speed to 2.5GT/s,=20 > request a link retrain and check again if the link has stabilised. If=20 > that does not work either, then restore the original speed setting and=20 > claim defeat, otherwise we are done. >=20 > NB interestingly enough with the ASM2824 vs PI7C9X2G304 configuration=20 > referred above asking the ASM2824 to retrain with a higher target link=20 > speed once the 2.5GT/s speed has been negotiated makes the two devices=20 > successfully negotiate 5.0GT/s. Lifting the 2.5GT/s speed restriction=20 > would however prevent our workaround from working with an OS that issues= =20 > a reset and that is unaware of the problem. This is because the devices= =20 > would then try to negotiate a higher link speed from scratch and fail,=20 > while the sticky property of the Target Link Speed setting will keep the= =20 > 2.5GT/s speed restriction across a reset. >=20 > Keep the 2.5GT/s speed restriction then, conservatively, if functional=20 > once applied. >=20 > Signed-off-by: Maciej W. Rozycki > Reviewed-by: Stefan Roese Applied to u-boot/master, thanks! --=20 Tom --bDTAc/+PmGfS/M5J Content-Type: application/pgp-signature; name="signature.asc" -----BEGIN PGP SIGNATURE----- iQGzBAABCgAdFiEEGjx/cOCPqxcHgJu/FHw5/5Y0tywFAmHiwBIACgkQFHw5/5Y0 tyytegwAgZPQ6JCFgIWKSK1vIt+WkT0KbHXkD3zaCpQcd6J94SZIfwsEpB0MZxHa AmhsZCaK/OKMY0I5YOR/eq1yf5H1crtDDGUAa8s3V6YiIVDqihmZ7xsp6EfneN8+ oXZz0CTLD/3CNlG8YcvikgxtmHwYd1bd+BwQ+GEESC5KL4IzD0QXlJnU52IDmb8c KaOulU5L9ZbuEJRixFVEnwPjO5cwd5gjvjRnCO8ZXvuhxuTcXx9oeEqS3sjaZPPB K9B/PlmFQFGPdpTFrZDZJthu0shm3QNEXK3YkkRoxnge59j7timpSaCu0tDBc7bU jbYUB2VMcxIcBWAhEBWUleTH0SmPRz/Fir7LeKstDYrUUTyTHQLR6J3wekv3SAdN By1aIjuQ0lBW++HmB4397IXDLl4oZp1Jqt+XnCjpJiwym0ws3WoTfRWJszALqsms x1sItSidoJWqbLK9ip4pYlGzEZFT5Dho3wQ1NNA57iFgDC+Ca6MXQyOZH3uCJFen RNS8noKF =TuoS -----END PGP SIGNATURE----- --bDTAc/+PmGfS/M5J--