From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 57A49C433F5 for ; Fri, 11 Feb 2022 02:18:11 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:In-Reply-To:MIME-Version:References: Message-ID:Subject:Cc:To:From:Date:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=PhEn8NfCWgEmY6uCA9MijQacpv33/X+nSbsSBk7Mtzc=; b=T+aS5SgSTTgi8N rQdYjnJPJuXUvE3Pt/VdcWg4f/mC8nMLX4ijkg1GW6+6R6lSqmWYT1Nl2DavegBr6IuK9lKyTwOTT ec4h7kHU4IEKAWFhKF+gxjHI/mhisg8tGOnkCtaBi/sKVadarMSKeIxQRfT87MildW5u7EYU90nnz xUObpK47cNNVWPp4ZyIuvdsVQJ0rzOWu8YJ+eiZj68qfGYRo3cBTUHVvDJobSFwk/BDBOrY/jRIlB peRPXqF8Wy4wwYiK2znrWgb653NIzBvl95JUjMFDWQ4QOggay62cSRruXEYwDhV0SR7RcfAw4vjVF mi4IHg0NSVcrjU18G23A==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1nILUF-005VOn-1o; Fri, 11 Feb 2022 02:16:39 +0000 Received: from smtp-relay-internal-1.canonical.com ([185.125.188.123]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1nILUA-005VM3-VE for linux-arm-kernel@lists.infradead.org; Fri, 11 Feb 2022 02:16:37 +0000 Received: from mail-io1-f71.google.com (mail-io1-f71.google.com [209.85.166.71]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by smtp-relay-internal-1.canonical.com (Postfix) with ESMTPS id 66E394029C for ; Fri, 11 Feb 2022 02:16:26 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=canonical.com; s=20210705; t=1644545786; bh=wr917FetGRB+Dwa6hoOdwfA0a1yOjjIVMUk1tc6Vh2Q=; h=Date:From:To:Cc:Subject:Message-ID:References:MIME-Version: Content-Type:In-Reply-To; b=F+ZotKceKixZ+v5cOnDdxdsHZZaw2fwxBche2Al7Kz6h0e3dx0l3F41c/5V7ItQFf wkPrcX4eLqnCVSVqS+TZG+PdzfdQRfQnf5fjAbD5n+5GdecNrsLUbxbSuMbS0vdWk6 Z0I5rkf6NM3vlCJTAM43jLOWKKP4DetL0Poi5nls4QqSmnrFbQlvEsLb6xDtspSu6z 0Kycwl3UjeovzaHoVhVtZVZzdW9l34o4OwwB44wS2Zmop/5wWDkj+E+3PIcHZDMMud q3rRS+6MghJsZhxgrBz6owyabx4zVqwuPkDrb/JWmBPs0jWUTaakU9xeGG4V6JBpx+ WwqD2ZBnkpW9A== Received: by mail-io1-f71.google.com with SMTP id y23-20020a5e8717000000b0060fd92726cbso5333651ioj.22 for ; Thu, 10 Feb 2022 18:16:26 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:date:from:to:cc:subject:message-id:references :mime-version:content-disposition:content-transfer-encoding :in-reply-to; bh=wr917FetGRB+Dwa6hoOdwfA0a1yOjjIVMUk1tc6Vh2Q=; b=YhheE2iouIDwJN7J0wCvAoXNzoSlZ/RYqMrmgv9o3Ztwghi4SCE4fr3dpmh8y0rhwE iTpGCEInu9yGZ5n9yvCt25CUs7t61W2HjP1DeGHEcfNC/USetiMQddM8W8/354G7EwSY VBKI0pkZN1c73UVxdSwn22jDRxTozX3iw6oQLpirRHPIg1e/F+RsSJ8F3kV0iN6eohKI 3uWZHVN1T4W/V7ZXJl/PcFTXXufufUpa9PIFWegHsQXdAMQkPb858Xaud3oVIHBrpDCV VxQu+HQmVXjOnxvVScH+MiJ+ykeQ7ntKfg/S9+xCGB+vu3j7BYd6rEEI1UQY3DvWCrn5 KWxg== X-Gm-Message-State: AOAM5312mgjyrFvSeJ7gI7dsh0Q4zzt5i88GgKB9L5XgZ48SLn2xhdic eV0JX71njRh48ad7d5EWblz3kF8W+Nw50j7HdrUyv7xJedjtzMGkzGw3M3UENy+YNjhblTr9/1z rrpoU7naSbfK5AbrJo7OttdKjtSAgWThmPPoB/YlobZtMDQHBRMYN X-Received: by 2002:a02:c8c8:: with SMTP id q8mr5221357jao.243.1644545783560; Thu, 10 Feb 2022 18:16:23 -0800 (PST) X-Google-Smtp-Source: ABdhPJyMHhPZZCj8/aMYnFzDF+/epemaAS8yzBkanyfCuHI/zd+71O1cwnboJ+jJrLUY3cVTfwXTzA== X-Received: by 2002:a02:c8c8:: with SMTP id q8mr5221351jao.243.1644545783301; Thu, 10 Feb 2022 18:16:23 -0800 (PST) Received: from xps13.dannf (c-71-196-238-11.hsd1.co.comcast.net. [71.196.238.11]) by smtp.gmail.com with ESMTPSA id d2sm4867105iog.42.2022.02.10.18.16.21 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 10 Feb 2022 18:16:22 -0800 (PST) Date: Thu, 10 Feb 2022 19:16:19 -0700 From: dann frazier To: Rob Herring Cc: Toan Le , Lorenzo Pieralisi , Krzysztof =?utf-8?Q?Wilczy=C5=84ski?= , Bjorn Helgaas , Andrew Murray , =?iso-8859-1?Q?St=E9phane?= Graber , stable , PCI , linux-arm-kernel , "linux-kernel@vger.kernel.org" Subject: Re: [PATCH] PCI: xgene: Fix IB window setup Message-ID: References: <20211129173637.303201-1-robh@kernel.org> MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20220210_181635_306308_DEBEDF5B X-CRM114-Status: GOOD ( 56.56 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org On Tue, Feb 08, 2022 at 08:34:45AM -0600, Rob Herring wrote: > On Mon, Feb 7, 2022 at 7:19 PM dann frazier = wrote: > > > > On Mon, Feb 07, 2022 at 10:09:31AM -0600, Rob Herring wrote: > > > On Sat, Feb 5, 2022 at 3:13 PM dann frazier wrote: > > > > > > > > On Sat, Feb 5, 2022 at 9:05 AM Rob Herring wrote: > > > > > > > > > > On Fri, Feb 4, 2022 at 5:01 PM dann frazier wrote: > > > > > > > > > > > > On Mon, Nov 29, 2021 at 11:36:37AM -0600, Rob Herring wrote: > > > > > > > Commit 6dce5aa59e0b ("PCI: xgene: Use inbound resources for s= etup") > > > > > > > broke PCI support on XGene. The cause is the IB resources are= now sorted > > > > > > > in address order instead of being in DT dma-ranges order. The= result is > > > > > > > which inbound registers are used for each region are swapped.= I don't > > > > > > > know the details about this h/w, but it appears that IB regio= n 0 > > > > > > > registers can't handle a size greater than 4GB. In any case, = limiting > > > > > > > the size for region 0 is enough to get back to the original a= ssignment > > > > > > > of dma-ranges to regions. > > > > > > > > > > > > hey Rob! > > > > > > > > > > > > I've been seeing a panic on HP Moonshoot m400 cartridges (X-Gen= e1) - > > > > > > only during network installs - that I also bisected down to com= mit > > > > > > 6dce5aa59e0b ("PCI: xgene: Use inbound resources for setup"). I= was > > > > > > hoping that this patch that fixed the issue on St=E9phane's X-G= ene2 > > > > > > system would also fix my issue, but no luck. In fact, it seems = to just > > > > > > makes it fail differently. Reverting both patches is required t= o get a > > > > > > v5.17-rc kernel to boot. > > > > > > > > > > > > I've collected the following logs - let me know if anything els= e would > > > > > > be useful. > > > > > > > > > > > > 1) v5.17-rc2+ (unmodified): > > > > > > http://dannf.org/bugs/m400-no-reverts.log > > > > > > Note that the mlx4 driver fails initialization. > > > > > > > > > > > > 2) v5.17-rc2+, w/o the commit that fixed St=E9phane's system: > > > > > > http://dannf.org/bugs/m400-xgene2-fix-reverted.log > > > > > > Note the mlx4 MSI-X timeout, and later panic. > > > > > > > > > > > > 3) v5.17-rc2+, w/ both commits reverted (works) > > > > > > http://dannf.org/bugs/m400-both-reverted.log > > > > > > > > > > The ranges and dma-ranges addresses don't appear to match up with= any > > > > > upstream dts files. Can you send me the DT? > > > > > > > > Sure: http://dannf.org/bugs/fdt > > > > > > The first fix certainly is a problem. It's going to need something > > > besides size to key off of (originally it was dependent on order of > > > dma-ranges entries). > > > > > > The 2nd issue is the 'dma-ranges' has a second entry that is now igno= red: > > > > > > dma-ranges =3D <0x42000000 0x40 0x00 0x40 0x00 0x40 0x00 0x00>, <0x00 > > > 0x79000000 0x00 0x79000000 0x00 0x800000>; > > > > > > Based on the flags (3rd addr cell: 0x0), we have an inbound config > > > space which the kernel now ignores because inbound config space > > > accesses make no sense. But clearly some setup is needed. Upstream, in > > > contrast, sets up a memory range that includes this region, so the > > > setup does happen: > > > > > > <0x42000000 0x00 0x00000000 0x00 0x00000000 0x80 0x00000000> > > > > > > Minimally, I suspect it will work if you change dma-ranges 2nd entry = to: > > > > > > <0x42000000 0x79000000 0x00 0x79000000 0x00 0x800000> > > > > Thanks for looking into this Rob. I tried to test that theory, but it > > didn't seem to work. This is what I tried: > > > > --- m400.dts 2022-02-07 20:16:44.840475323 +0000 > > +++ m400.dts.dmaonly 2022-02-08 00:17:54.097132000 +0000 > > @@ -446,7 +446,7 @@ > > reg =3D <0x00 0x1f2b0000 0x00 0x10000 0xe0 0xd0= 000000 0x00 0x200000 0x00 0x79e00000 0x00 0x2000000 0x00 0x79000000 0x00 0x= 800000>; > > reg-names =3D "csr\0cfg\0msi_gen\0msi_term"; > > ranges =3D <0x1000000 0x00 0x00 0xe0 0x10000000= 0x00 0x10000 0x2000000 0x00 0x30000000 0xe1 0x30000000 0x00 0x80000000>; > > - dma-ranges =3D <0x42000000 0x40 0x00 0x40 0x00 = 0x40 0x00 0x00 0x00 0x79000000 0x00 0x79000000 0x00 0x800000>; > > + dma-ranges =3D <0x42000000 0x40 0x00 0x40 0x00 = 0x40 0x00 0x00 0x42000000 0x79000000 0x00 0x79000000 0x00 0x800000>; > > ib-ranges =3D <0x42000000 0x40 0x00 0x40 0x00 0= x40 0x00 0x00 0x00 0x79000000 0x00 0x79000000 0x00 0x800000>; > > ib-ranges-ep =3D <0x2000000 0x00 0x00 0x00 0x00= 0x00 0x400000 0x2000000 0x00 0x00 0x00 0x00 0x00 0x400000 0x2000000 0x00 0= x79000000 0x00 0x79000000 0x00 0x100000>; > > interrupts =3D <0x00 0x10 0x04>; > > @@ -471,7 +471,7 @@ > > reg =3D <0x00 0x1f2c0000 0x00 0x10000 0xd0 0xd0= 000000 0x00 0x200000 0x00 0x79e00000 0x00 0x2000000 0x00 0x79000000 0x00 0x= 800000>; > > reg-names =3D "csr\0cfg\0msi_gen\0msi_term"; > > ranges =3D <0x1000000 0x00 0x00 0xd0 0x10000000= 0x00 0x10000 0x2000000 0x00 0x30000000 0xd1 0x30000000 0x00 0x80000000>; > > - dma-ranges =3D <0x42000000 0x40 0x00 0x40 0x00 = 0x40 0x00 0x00 0x00 0x79000000 0x00 0x79000000 0x00 0x800000>; > > + dma-ranges =3D <0x42000000 0x40 0x00 0x40 0x00 = 0x40 0x00 0x00 0x42000000 0x79000000 0x00 0x79000000 0x00 0x800000>; > > ib-ranges =3D <0x42000000 0x40 0x00 0x40 0x00 0= x40 0x00 0x00 0x00 0x79000000 0x00 0x79000000 0x00 0x800000>; > > ib-ranges-ep =3D <0x2000000 0x00 0x00 0x00 0x00= 0x00 0x400000 0x2000000 0x00 0x00 0x00 0x00 0x00 0x400000 0x2000000 0x00 0= x79000000 0x00 0x79000000 0x00 0x100000>; > > interrupts =3D <0x00 0x10 0x04>; > > @@ -496,7 +496,7 @@ > > reg =3D <0x00 0x1f2d0000 0x00 0x10000 0x90 0xd0= 000000 0x00 0x200000 0x00 0x79e00000 0x00 0x2000000 0x00 0x79000000 0x00 0x= 800000>; > > reg-names =3D "csr\0cfg\0msi_gen\0msi_term"; > > ranges =3D <0x1000000 0x00 0x00 0x90 0x10000000= 0x00 0x10000 0x2000000 0x00 0x30000000 0x91 0x30000000 0x00 0x80000000>; > > - dma-ranges =3D <0x42000000 0x40 0x00 0x40 0x00 = 0x40 0x00 0x00 0x00 0x79000000 0x00 0x79000000 0x00 0x800000>; > > + dma-ranges =3D <0x42000000 0x40 0x00 0x40 0x00 = 0x40 0x00 0x00 0x42000000 0x79000000 0x00 0x79000000 0x00 0x800000>; > > ib-ranges =3D <0x42000000 0x40 0x00 0x40 0x00 0= x40 0x00 0x00 0x00 0x79000000 0x00 0x79000000 0x00 0x800000>; > > ib-ranges-ep =3D <0x2000000 0x00 0x00 0x00 0x00= 0x00 0x400000 0x2000000 0x00 0x00 0x00 0x00 0x00 0x400000 0x2000000 0x00 0= x79000000 0x00 0x79000000 0x00 0x100000>; > > interrupts =3D <0x00 0x10 0x04>; > > @@ -522,7 +522,7 @@ > > reg =3D <0x00 0x1f500000 0x00 0x10000 0xa0 0xd0= 000000 0x00 0x200000 0x00 0x79e00000 0x00 0x2000000 0x00 0x79000000 0x00 0x= 800000>; > > reg-names =3D "csr\0cfg\0msi_gen\0msi_term"; > > ranges =3D <0x2000000 0x00 0x30000000 0xa1 0x30= 000000 0x00 0x80000000>; > > - dma-ranges =3D <0x42000000 0x40 0x00 0x40 0x00 = 0x40 0x00 0x00 0x00 0x79000000 0x00 0x79000000 0x00 0x800000>; > > + dma-ranges =3D <0x42000000 0x40 0x00 0x40 0x00 = 0x40 0x00 0x00 0x42000000 0x79000000 0x00 0x79000000 0x00 0x800000>; > > ib-ranges =3D <0x42000000 0x40 0x00 0x40 0x00 0= x40 0x00 0x00 0x00 0x79000000 0x00 0x79000000 0x00 0x800000>; > > ib-ranges-ep =3D <0x2000000 0x00 0x00 0x00 0x00= 0x00 0x400000 0x2000000 0x00 0x00 0x00 0x00 0x00 0x400000 0x2000000 0x00 0= x79000000 0x00 0x79000000 0x00 0x100000>; > > interrupts =3D <0x00 0x10 0x04>; > > @@ -547,7 +547,7 @@ > > reg =3D <0x00 0x1f510000 0x00 0x10000 0xc0 0xd0= 000000 0x00 0x200000 0x00 0x79e00000 0x00 0x2000000 0x00 0x79000000 0x00 0x= 800000>; > > reg-names =3D "csr\0cfg\0msi_gen\0msi_term"; > > ranges =3D <0x1000000 0x00 0x00 0xc0 0x10000000= 0x00 0x10000 0x2000000 0x00 0x30000000 0xc1 0x30000000 0x00 0x80000000>; > > - dma-ranges =3D <0x42000000 0x40 0x00 0x40 0x00 = 0x40 0x00 0x00 0x00 0x79000000 0x00 0x79000000 0x00 0x800000>; > > + dma-ranges =3D <0x42000000 0x40 0x00 0x40 0x00 = 0x40 0x00 0x00 0x42000000 0x79000000 0x00 0x79000000 0x00 0x800000>; > > ib-ranges =3D <0x42000000 0x40 0x00 0x40 0x00 0= x40 0x00 0x00 0x00 0x79000000 0x00 0x79000000 0x00 0x800000>; > > ib-ranges-ep =3D <0x2000000 0x00 0x00 0x00 0x00= 0x00 0x400000 0x2000000 0x00 0x00 0x00 0x00 0x00 0x400000 0x2000000 0x00 0= x79000000 0x00 0x79000000 0x00 0x100000>; > > interrupts =3D <0x00 0x10 0x04>; > > > > And that failed to boot with a 5.17-rc3. Since dma-ranges was > > previously identical to ib-ranges, I also tried making the same change > > to ib-ranges, but with no success. > = > Failed to boot at all or just PCIe still didn't work causing boot to > eventually fail? Sorry, I mean PCIe still didn't work, here's the log: http://dannf.org/bugs/m400-tweaked_dtb.log (unmodified kernel source w/ above dtb change) > 'ib-ranges' is unknown to the kernel, so the firmware > is using it somehow? > = > You also need to revert the first fix for PCIe to work. Oh, OK. I misunderstood. I tried reverting commit 6dce5aa59e0b "PCI: xgene: Use inbound resources for setup" along with a dtb with the dma-ranges change in the diff above, but PCIe still didn't work. Here's the log: http://dannf.org/bugs/m400-6dce5aa5_reverted+tweaked_dtb.log -dann = > = > > > While we shouldn't break existing DTs, the moonshot DT doesn't use > > > what's documented upstream. There are multiple differences compared to > > > what's documented. Is upstream supposed to support upstream DTs, > > > downstream DTs, and ACPI for XGene which is an abandoned platform with > > > only a handful of users? > > > > That's a fair question, though it's one of a policy, and I feel I'd be > > overstepping by weighing in. I suppose one option I have is to try > > and create and upstream a dts for these systems and modify our > > boot.scr to always load that over the one provided by firmware. While > > we do have some of these systems in production, they are being retired > > and replaced with newer kit over time, and it's possible we'll never > > need to upgrade them to a modern kernel. > > > > -dann _______________________________________________ linux-arm-kernel mailing list linux-arm-kernel@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-arm-kernel