From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id DDC45C433F5 for ; Tue, 8 Feb 2022 01:20:55 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:In-Reply-To:MIME-Version:References: Message-ID:Subject:Cc:To:From:Date:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=WO5mSCHRnrMNBvMkPdCqIbnLgr1QzufSM4iMnZ/EN5U=; b=dXnZNee+PJdGJd pdF8PSogtwmmjH4byJRqmbUkK0t4nmuYeJGwHcjDrmfNvlSZyMoF7GOInDF9AfWnzQX9/BhauCAP6 PycYtpK/jNbNYJrdnAoZVazyk789MNVU6MXwUFklDI3Dm6O29vLpvhzRxpBxg4KyiyjYz+P5TFLW6 5AYn52z3NSAooYkUoiWqa5apkeFss483uaQr8Mq4PBZ4+lu13vc6njTi5Ga5P+ynDtmHG/aQgf014 uZpRxRtu9rPUVygm0Z+Qzx08MfADScJRgW8NnhdlgG9/VY/7GoH2buxPQAt06m9Gc4euC8PT4Qi4D 4ubkSZqgY0jmDf8DslUw==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1nHFAM-00C8wl-6M; Tue, 08 Feb 2022 01:19:34 +0000 Received: from smtp-relay-internal-0.canonical.com ([185.125.188.122]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1nHFAI-00C8vW-F2 for linux-arm-kernel@lists.infradead.org; Tue, 08 Feb 2022 01:19:32 +0000 Received: from mail-io1-f70.google.com (mail-io1-f70.google.com [209.85.166.70]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by smtp-relay-internal-0.canonical.com (Postfix) with ESMTPS id 5FFD73F32B for ; Tue, 8 Feb 2022 01:19:22 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=canonical.com; s=20210705; t=1644283162; bh=bR+KnBD3V9VfQQwZt2Z/wkPMJzyvsWJ3UhlANeJa8dA=; h=Date:From:To:Cc:Subject:Message-ID:References:MIME-Version: Content-Type:In-Reply-To; b=MXNNMIOHLah8Au890gGzYH4JOMn8Geyph9D1Sy0pE8bzDCjjR8VrMe54r7MeBncup lcrPHtFX2H5oQo/YbJ6iejVN8nvqWQYPlNuiwd7IZWPXYM+NAjFjPKKuAXut4Aqr/D c0BF4PUobDYzqvbIW+EFYJGGKQUmGxQzHVaGT1uKuZfRX+Y84I7KKbuznFxXFWeWMT 4CHvPhhqPUvqMhJTMNwVaaU+uDieoZf77bWBM7knmiUy5zAtChdZX+nL9eM4lDIQna wh5VA4bIrF8qB0ODDz093tkrLHBFvd50gi+fn/OO7n6YDvDnjMHJ9R8RCxzZcpINyJ M/QgMT7mOgrLg== Received: by mail-io1-f70.google.com with SMTP id g7-20020a5ec747000000b00612ad6f568cso10368223iop.8 for ; Mon, 07 Feb 2022 17:19:22 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:date:from:to:cc:subject:message-id:references :mime-version:content-disposition:content-transfer-encoding :in-reply-to; bh=bR+KnBD3V9VfQQwZt2Z/wkPMJzyvsWJ3UhlANeJa8dA=; b=31edZRhrPOPE5Dbhk7HlcqAaATkN4RLfPYj5iugsCB9njflSxkI7PWUyTkj0WiOFj2 Mo3bj2108l1lKmy9Z6gDOA35nRflRVbvjeQC1RXrZqmRi8jjoKwBDsD/C6TIts2cFnmt G0xaxeeH0L8KJ6Pwu85H9RNK6mgZM928ndggT9nWKexkn1+wXYiemp6RhOdWluA1qM1h CJALWdZB1oEQttTqLNZW6kwOy8OgLcS9dFEuHhmn/2LsMdPaCo+ggDZKwGZpVgdmfx9F Y2DHoyaAqb5LG7QhbEvlk856zSMZrsU+SndhOB0Kvqa8hrbb0J7CToNuJssZt/5l3SSr 0qJA== X-Gm-Message-State: AOAM532TXd05gOdY1hzUvXlpYyefLdScZj6e1jEZzaoi/bIFZykH3acf sVpmD/74klNhM021Qu9tFlgp0M74QmWU87LTXZYl2fogeWqN63XqnBuch7mPQoIqIgTUnvNMicl kZgYtFqNUiBeZsYngME4ELhwptmzvuEhd6QjpnFiUlhArQcxb0P8f X-Received: by 2002:a05:6638:10ea:: with SMTP id g10mr1085376jae.79.1644283159934; Mon, 07 Feb 2022 17:19:19 -0800 (PST) X-Google-Smtp-Source: ABdhPJxgE+zqaGNjhqInu2uD4JORuhXtZ5+ThLHHgAdHtKjBMTdOQNSqcTIspI6mC//tHBlpexBZmA== X-Received: by 2002:a05:6638:10ea:: with SMTP id g10mr1085369jae.79.1644283159579; Mon, 07 Feb 2022 17:19:19 -0800 (PST) Received: from xps13.dannf (c-71-196-238-11.hsd1.co.comcast.net. [71.196.238.11]) by smtp.gmail.com with ESMTPSA id ay35sm7126949iob.3.2022.02.07.17.19.17 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 07 Feb 2022 17:19:18 -0800 (PST) Date: Mon, 7 Feb 2022 18:19:16 -0700 From: dann frazier To: Rob Herring Cc: Toan Le , Lorenzo Pieralisi , Krzysztof =?utf-8?Q?Wilczy=C5=84ski?= , Bjorn Helgaas , Andrew Murray , =?iso-8859-1?Q?St=E9phane?= Graber , stable , PCI , linux-arm-kernel , "linux-kernel@vger.kernel.org" Subject: Re: [PATCH] PCI: xgene: Fix IB window setup Message-ID: References: <20211129173637.303201-1-robh@kernel.org> MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20220207_171930_760407_32A3BFE0 X-CRM114-Status: GOOD ( 45.66 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org On Mon, Feb 07, 2022 at 10:09:31AM -0600, Rob Herring wrote: > On Sat, Feb 5, 2022 at 3:13 PM dann frazier = wrote: > > > > On Sat, Feb 5, 2022 at 9:05 AM Rob Herring wrote: > > > > > > On Fri, Feb 4, 2022 at 5:01 PM dann frazier wrote: > > > > > > > > On Mon, Nov 29, 2021 at 11:36:37AM -0600, Rob Herring wrote: > > > > > Commit 6dce5aa59e0b ("PCI: xgene: Use inbound resources for setup= ") > > > > > broke PCI support on XGene. The cause is the IB resources are now= sorted > > > > > in address order instead of being in DT dma-ranges order. The res= ult is > > > > > which inbound registers are used for each region are swapped. I d= on't > > > > > know the details about this h/w, but it appears that IB region 0 > > > > > registers can't handle a size greater than 4GB. In any case, limi= ting > > > > > the size for region 0 is enough to get back to the original assig= nment > > > > > of dma-ranges to regions. > > > > > > > > hey Rob! > > > > > > > > I've been seeing a panic on HP Moonshoot m400 cartridges (X-Gene1) - > > > > only during network installs - that I also bisected down to commit > > > > 6dce5aa59e0b ("PCI: xgene: Use inbound resources for setup"). I was > > > > hoping that this patch that fixed the issue on St=E9phane's X-Gene2 > > > > system would also fix my issue, but no luck. In fact, it seems to j= ust > > > > makes it fail differently. Reverting both patches is required to ge= t a > > > > v5.17-rc kernel to boot. > > > > > > > > I've collected the following logs - let me know if anything else wo= uld > > > > be useful. > > > > > > > > 1) v5.17-rc2+ (unmodified): > > > > http://dannf.org/bugs/m400-no-reverts.log > > > > Note that the mlx4 driver fails initialization. > > > > > > > > 2) v5.17-rc2+, w/o the commit that fixed St=E9phane's system: > > > > http://dannf.org/bugs/m400-xgene2-fix-reverted.log > > > > Note the mlx4 MSI-X timeout, and later panic. > > > > > > > > 3) v5.17-rc2+, w/ both commits reverted (works) > > > > http://dannf.org/bugs/m400-both-reverted.log > > > > > > The ranges and dma-ranges addresses don't appear to match up with any > > > upstream dts files. Can you send me the DT? > > > > Sure: http://dannf.org/bugs/fdt > = > The first fix certainly is a problem. It's going to need something > besides size to key off of (originally it was dependent on order of > dma-ranges entries). > = > The 2nd issue is the 'dma-ranges' has a second entry that is now ignored: > = > dma-ranges =3D <0x42000000 0x40 0x00 0x40 0x00 0x40 0x00 0x00>, <0x00 > 0x79000000 0x00 0x79000000 0x00 0x800000>; > = > Based on the flags (3rd addr cell: 0x0), we have an inbound config > space which the kernel now ignores because inbound config space > accesses make no sense. But clearly some setup is needed. Upstream, in > contrast, sets up a memory range that includes this region, so the > setup does happen: > = > <0x42000000 0x00 0x00000000 0x00 0x00000000 0x80 0x00000000> > = > Minimally, I suspect it will work if you change dma-ranges 2nd entry to: > = > <0x42000000 0x79000000 0x00 0x79000000 0x00 0x800000> Thanks for looking into this Rob. I tried to test that theory, but it didn't seem to work. This is what I tried: --- m400.dts 2022-02-07 20:16:44.840475323 +0000 +++ m400.dts.dmaonly 2022-02-08 00:17:54.097132000 +0000 @@ -446,7 +446,7 @@ reg =3D <0x00 0x1f2b0000 0x00 0x10000 0xe0 0xd0000000 0x00 0x200000 0x0= 0 0x79e00000 0x00 0x2000000 0x00 0x79000000 0x00 0x800000>; reg-names =3D "csr\0cfg\0msi_gen\0msi_term"; ranges =3D <0x1000000 0x00 0x00 0xe0 0x10000000 0x00 0x10000 0x2000000 = 0x00 0x30000000 0xe1 0x30000000 0x00 0x80000000>; - dma-ranges =3D <0x42000000 0x40 0x00 0x40 0x00 0x40 0x00 0x00 0x00 0x79= 000000 0x00 0x79000000 0x00 0x800000>; + dma-ranges =3D <0x42000000 0x40 0x00 0x40 0x00 0x40 0x00 0x00 0x4200000= 0 0x79000000 0x00 0x79000000 0x00 0x800000>; ib-ranges =3D <0x42000000 0x40 0x00 0x40 0x00 0x40 0x00 0x00 0x00 0x790= 00000 0x00 0x79000000 0x00 0x800000>; ib-ranges-ep =3D <0x2000000 0x00 0x00 0x00 0x00 0x00 0x400000 0x2000000= 0x00 0x00 0x00 0x00 0x00 0x400000 0x2000000 0x00 0x79000000 0x00 0x7900000= 0 0x00 0x100000>; interrupts =3D <0x00 0x10 0x04>; @@ -471,7 +471,7 @@ reg =3D <0x00 0x1f2c0000 0x00 0x10000 0xd0 0xd0000000 0x00 0x200000 0x0= 0 0x79e00000 0x00 0x2000000 0x00 0x79000000 0x00 0x800000>; reg-names =3D "csr\0cfg\0msi_gen\0msi_term"; ranges =3D <0x1000000 0x00 0x00 0xd0 0x10000000 0x00 0x10000 0x2000000 = 0x00 0x30000000 0xd1 0x30000000 0x00 0x80000000>; - dma-ranges =3D <0x42000000 0x40 0x00 0x40 0x00 0x40 0x00 0x00 0x00 0x79= 000000 0x00 0x79000000 0x00 0x800000>; + dma-ranges =3D <0x42000000 0x40 0x00 0x40 0x00 0x40 0x00 0x00 0x4200000= 0 0x79000000 0x00 0x79000000 0x00 0x800000>; ib-ranges =3D <0x42000000 0x40 0x00 0x40 0x00 0x40 0x00 0x00 0x00 0x790= 00000 0x00 0x79000000 0x00 0x800000>; ib-ranges-ep =3D <0x2000000 0x00 0x00 0x00 0x00 0x00 0x400000 0x2000000= 0x00 0x00 0x00 0x00 0x00 0x400000 0x2000000 0x00 0x79000000 0x00 0x7900000= 0 0x00 0x100000>; interrupts =3D <0x00 0x10 0x04>; @@ -496,7 +496,7 @@ reg =3D <0x00 0x1f2d0000 0x00 0x10000 0x90 0xd0000000 0x00 0x200000 0x0= 0 0x79e00000 0x00 0x2000000 0x00 0x79000000 0x00 0x800000>; reg-names =3D "csr\0cfg\0msi_gen\0msi_term"; ranges =3D <0x1000000 0x00 0x00 0x90 0x10000000 0x00 0x10000 0x2000000 = 0x00 0x30000000 0x91 0x30000000 0x00 0x80000000>; - dma-ranges =3D <0x42000000 0x40 0x00 0x40 0x00 0x40 0x00 0x00 0x00 0x79= 000000 0x00 0x79000000 0x00 0x800000>; + dma-ranges =3D <0x42000000 0x40 0x00 0x40 0x00 0x40 0x00 0x00 0x4200000= 0 0x79000000 0x00 0x79000000 0x00 0x800000>; ib-ranges =3D <0x42000000 0x40 0x00 0x40 0x00 0x40 0x00 0x00 0x00 0x790= 00000 0x00 0x79000000 0x00 0x800000>; ib-ranges-ep =3D <0x2000000 0x00 0x00 0x00 0x00 0x00 0x400000 0x2000000= 0x00 0x00 0x00 0x00 0x00 0x400000 0x2000000 0x00 0x79000000 0x00 0x7900000= 0 0x00 0x100000>; interrupts =3D <0x00 0x10 0x04>; @@ -522,7 +522,7 @@ reg =3D <0x00 0x1f500000 0x00 0x10000 0xa0 0xd0000000 0x00 0x200000 0x0= 0 0x79e00000 0x00 0x2000000 0x00 0x79000000 0x00 0x800000>; reg-names =3D "csr\0cfg\0msi_gen\0msi_term"; ranges =3D <0x2000000 0x00 0x30000000 0xa1 0x30000000 0x00 0x80000000>; - dma-ranges =3D <0x42000000 0x40 0x00 0x40 0x00 0x40 0x00 0x00 0x00 0x79= 000000 0x00 0x79000000 0x00 0x800000>; + dma-ranges =3D <0x42000000 0x40 0x00 0x40 0x00 0x40 0x00 0x00 0x4200000= 0 0x79000000 0x00 0x79000000 0x00 0x800000>; ib-ranges =3D <0x42000000 0x40 0x00 0x40 0x00 0x40 0x00 0x00 0x00 0x790= 00000 0x00 0x79000000 0x00 0x800000>; ib-ranges-ep =3D <0x2000000 0x00 0x00 0x00 0x00 0x00 0x400000 0x2000000= 0x00 0x00 0x00 0x00 0x00 0x400000 0x2000000 0x00 0x79000000 0x00 0x7900000= 0 0x00 0x100000>; interrupts =3D <0x00 0x10 0x04>; @@ -547,7 +547,7 @@ reg =3D <0x00 0x1f510000 0x00 0x10000 0xc0 0xd0000000 0x00 0x200000 0x0= 0 0x79e00000 0x00 0x2000000 0x00 0x79000000 0x00 0x800000>; reg-names =3D "csr\0cfg\0msi_gen\0msi_term"; ranges =3D <0x1000000 0x00 0x00 0xc0 0x10000000 0x00 0x10000 0x2000000 = 0x00 0x30000000 0xc1 0x30000000 0x00 0x80000000>; - dma-ranges =3D <0x42000000 0x40 0x00 0x40 0x00 0x40 0x00 0x00 0x00 0x79= 000000 0x00 0x79000000 0x00 0x800000>; + dma-ranges =3D <0x42000000 0x40 0x00 0x40 0x00 0x40 0x00 0x00 0x4200000= 0 0x79000000 0x00 0x79000000 0x00 0x800000>; ib-ranges =3D <0x42000000 0x40 0x00 0x40 0x00 0x40 0x00 0x00 0x00 0x790= 00000 0x00 0x79000000 0x00 0x800000>; ib-ranges-ep =3D <0x2000000 0x00 0x00 0x00 0x00 0x00 0x400000 0x2000000= 0x00 0x00 0x00 0x00 0x00 0x400000 0x2000000 0x00 0x79000000 0x00 0x7900000= 0 0x00 0x100000>; interrupts =3D <0x00 0x10 0x04>; And that failed to boot with a 5.17-rc3. Since dma-ranges was previously identical to ib-ranges, I also tried making the same change to ib-ranges, but with no success. > While we shouldn't break existing DTs, the moonshot DT doesn't use > what's documented upstream. There are multiple differences compared to > what's documented. Is upstream supposed to support upstream DTs, > downstream DTs, and ACPI for XGene which is an abandoned platform with > only a handful of users? That's a fair question, though it's one of a policy, and I feel I'd be overstepping by weighing in. I suppose one option I have is to try and create and upstream a dts for these systems and modify our boot.scr to always load that over the one provided by firmware. While we do have some of these systems in production, they are being retired and replaced with newer kit over time, and it's possible we'll never need to upgrade them to a modern kernel. -dann _______________________________________________ linux-arm-kernel mailing list linux-arm-kernel@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-arm-kernel