From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-6.8 required=3.0 tests=BAYES_00, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE, SPF_PASS,URIBL_BLOCKED autolearn=no autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 1C51CC433E2 for ; Mon, 7 Sep 2020 15:01:20 +0000 (UTC) Received: from whitealder.osuosl.org (smtp1.osuosl.org [140.211.166.138]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id B3C1221481 for ; Mon, 7 Sep 2020 15:01:19 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org B3C1221481 Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=suse.de Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=iommu-bounces@lists.linux-foundation.org Received: from localhost (localhost [127.0.0.1]) by whitealder.osuosl.org (Postfix) with ESMTP id 54997867F7; Mon, 7 Sep 2020 15:01:19 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from whitealder.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id EAnuJ9oaNEjM; Mon, 7 Sep 2020 15:01:18 +0000 (UTC) Received: from lists.linuxfoundation.org (lf-lists.osuosl.org [140.211.9.56]) by whitealder.osuosl.org (Postfix) with ESMTP id 75D70867F0; Mon, 7 Sep 2020 15:01:18 +0000 (UTC) Received: from lf-lists.osuosl.org (localhost [127.0.0.1]) by lists.linuxfoundation.org (Postfix) with ESMTP id 5F23CC0052; Mon, 7 Sep 2020 15:01:18 +0000 (UTC) Received: from silver.osuosl.org (smtp3.osuosl.org [140.211.166.136]) by lists.linuxfoundation.org (Postfix) with ESMTP id 211A8C0051 for ; Mon, 7 Sep 2020 15:01:17 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by silver.osuosl.org (Postfix) with ESMTP id 0DA4820517 for ; Mon, 7 Sep 2020 15:01:17 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from silver.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id RuluZuZmWBeP for ; Mon, 7 Sep 2020 15:01:15 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.7.6 Received: from mx2.suse.de (mx2.suse.de [195.135.220.15]) by silver.osuosl.org (Postfix) with ESMTPS id 2A69720513 for ; Mon, 7 Sep 2020 15:01:15 +0000 (UTC) X-Virus-Scanned: by amavisd-new at test-mx.suse.de Received: from relay2.suse.de (unknown [195.135.221.27]) by mx2.suse.de (Postfix) with ESMTP id C97A1B6CF; Mon, 7 Sep 2020 15:01:13 +0000 (UTC) Message-ID: <34aa0d6094e7d6fb3492d2cda0fec8ecc04790ed.camel@suse.de> Subject: Re: [PATCH v11 07/11] device-mapping: Introduce DMA range map, supplanting dma_pfn_offset From: Nicolas Saenz Julienne To: Jim Quinlan , Nathan Chancellor , Christoph Hellwig Date: Mon, 07 Sep 2020 17:01:08 +0200 In-Reply-To: References: <20200824193036.6033-1-james.quinlan@broadcom.com> <20200824193036.6033-8-james.quinlan@broadcom.com> <20200902215314.GA881878@ubuntu-n2-xlarge-x86> <20200902223852.GA1786990@ubuntu-n2-xlarge-x86> <6922bc0b-1849-2f2f-ec2f-fe9f0124dcfc@gmail.com> <20200903005240.GA1118@Ryzen-9-3900X.localdomain> User-Agent: Evolution 3.36.5 MIME-Version: 1.0 Cc: Rich Felker , "open list:SUPERH" , David Airlie , "open list:PCI NATIVE HOST BRIDGE AND ENDPOINT DRIVERS" , Hanjun Guo , "open list:REMOTE PROCESSOR REMOTEPROC SUBSYSTEM" , "open list:DRM DRIVERS FOR ALLWINNER A10" , Julien Grall , "H. Peter Anvin" , Will Deacon , Christoph Hellwig , "open list:STAGING SUBSYSTEM" , Jean-Philippe Brucker , Florian Fainelli , Yoshinori Sato , Bartosz Golaszewski , Frank Rowand , "maintainer:X86 ARCHITECTURE 32-BIT AND 64-BIT" , Russell King , "open list:ACPI FOR ARM64 ACPI/arm64" , Chen-Yu Tsai , Ingo Molnar , "maintainer:BROADCOM BCM7XXX ARM ARCHITECTURE" , Alan Stern , Len Brown , Ohad Ben-Cohen , "open list:OPEN FIRMWARE AND FLATTENED DEVICE TREE" , Mauro Carvalho Chehab , Maxime Ripard , Rob Herring , Borislav Petkov , Yong Deng , Santosh Shilimkar , Bjorn Helgaas , Dan Williams , Andy Shevchenko , "moderated list:ARM PORT" , Felipe Balbi , Saravana Kannan , Greg Kroah-Hartman , "open list:USB SUBSYSTEM" , "Rafael J. Wysocki" , open list , Paul Kocialkowski , "open list:IOMMU DRIVERS" , Thomas Gleixner , Stefano Stabellini , Daniel Vetter , Sudeep Holla , "open list:ALLWINNER A10 CSI DRIVER" , Robin Murphy X-BeenThere: iommu@lists.linux-foundation.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: Development issues for Linux IOMMU support List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Content-Type: multipart/mixed; boundary="===============8120415809466771585==" Errors-To: iommu-bounces@lists.linux-foundation.org Sender: "iommu" --===============8120415809466771585== Content-Type: multipart/signed; micalg="pgp-sha256"; protocol="application/pgp-signature"; boundary="=-cTysW+6S+4/l7Qusohcm" --=-cTysW+6S+4/l7Qusohcm Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable Hi Jim, sorry I'm a little late to the party, but was on vacation. On Thu, 2020-09-03 at 13:32 -0400, Jim Quinlan wrote: > On Wed, Sep 2, 2020 at 8:52 PM Nathan Chancellor > wrote: > > On Wed, Sep 02, 2020 at 05:36:29PM -0700, Florian Fainelli wrote: > > >=20 > > > On 9/2/2020 3:38 PM, Nathan Chancellor wrote: > > > [snip] > > > > > Hello Nathan, > > > > >=20 > > > > > Can you tell me how much memory your RPI has and if all of it is > > > >=20 > > > > This is the 4GB version. > > > >=20 > > > > > accessible by the PCIe device? Could you also please include the= DTS > > > > > of the PCIe node? IIRC, the RPI firmware does some mangling of t= he > > > > > PCIe DT before Linux boots -- could you describe what is going on > > > > > there? > > > >=20 > > > > Unfortunately, I am not familiar with how to get this information. = If > > > > you could provide some instructions for how to do so, I am more tha= n > > > > happy to. I am not very knowleagable about the inner working of the= Pi, > > > > I mainly use it as a test platform for making sure that LLVM does n= ot > > > > cause problems on real devices. > > >=20 > > > Can you bring the dtc application to your Pi root filesystem, and if = so, can > > > you run the following: > > >=20 > > > dtc -I fs -O dtb /proc/device-tree -f > /tmp/device.dtb > >=20 > > Sure, the result is attached. > >=20 > > > or cat /sys/firmware/fdt > device.dtb > > >=20 > > > and attach the resulting file? > > >=20 > > > > > Finally, can you attach the text of the full boot log? > > > >=20 > > > > I have attached a working and broken boot log. Thank you for the qu= ick > > > > response! > > >=20 > > > Is it possible for you to rebuild your kernel with CONFIG_MMC_DEBUG b= y any > > > chance? > >=20 > > Of course. A new log is attached with the debug output from that config= . >=20 > Hi Nicolas, >=20 > Can you please help us out here? It appears that your commit It's dma_offset_from_dma_addr() that's causing trouble. It goes over all th= e dma regions and, if no region matches the phys address range, it returns 0. This isn't acceptable as is. In the past, we used to pass the offset nonetheless, which would make the phys address grow out of the dma memory a= rea and fail the dma_capable() test. For example, RPi4 devices attached to the old interconnect see phys [0x0 0x3fffffff] at [0xc0000000 0xffffffff]. So say you're trying to convert physical address 0xfa000000, you'll get 0 from dma_offset_from_phys_addr() (since your dma range only covers the first GB) to then test if 0xfa000000 = is dma_capable(), which it is, but for the wrong reasons. Causing DMA issues further down the line. I don't have a proper suggestion on how to solve this but here's the soluti= on I used: diff --git a/include/linux/dma-mapping.h b/include/linux/dma-mapping.h index 4c4646761afe..40fe3c97f2be 100644 --- a/include/linux/dma-mapping.h +++ b/include/linux/dma-mapping.h @@ -217,11 +217,19 @@ static inline u64 dma_offset_from_phys_addr(struct de= vice *dev, { const struct bus_dma_region *m =3D dev->dma_range_map; =20 - if (m) + if (m) { for (; m->size; m++) if (paddr >=3D m->cpu_start && paddr - m->cpu_start < m->size) return m->offset; + + /* + * The physical address doesn't fit any of the DMA regions, + * return an impossible to fulfill offset. + */ + return -(1ULL << 46); + } + return 0; } I didn't catch this on my previous tests as I was using a newer bcm2711 SoC revision which expanded emmc2's DMA capabilities (can do 32 bit DMA as oppo= sed to 30 bit). Regards, Nicolas --=-cTysW+6S+4/l7Qusohcm Content-Type: application/pgp-signature; name="signature.asc" Content-Description: This is a digitally signed message part Content-Transfer-Encoding: 7bit -----BEGIN PGP SIGNATURE----- iQEzBAABCAAdFiEErOkkGDHCg2EbPcGjlfZmHno8x/4FAl9WSzQACgkQlfZmHno8 x/7bxQgAq+hxWcUTGOm78RMY+sJ/vmapTRt7oyJb7Lveh5vUDXmv6qDBblJRUU7M uhwz/DHwnoJS8qsUKpmkuz5thA96uATLTDNBDRQ3TSAfM1L62TrgIklSBL8QJhov djYxt05Roimhkzx2OIzIjx52N+G4t/unl4Pcfm837cYCCwA4DKuDF5eYLriboTHg 8osj9aYS4X3f75bWGuNRgMJk16v/Cz5h+DoXHPWbarzBDH1xBV30pUzftEc9bW/0 8O2Rihk2HT8p1DOz2jkNttE6OW0bD+tpDcU0dBUJ/areTb/Bdxh/OcYMWqybtMV7 7rUzo5M4M9oHMwPQf6KN2p/9cpQfTQ== =08KD -----END PGP SIGNATURE----- --=-cTysW+6S+4/l7Qusohcm-- --===============8120415809466771585== Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Disposition: inline _______________________________________________ iommu mailing list iommu@lists.linux-foundation.org https://lists.linuxfoundation.org/mailman/listinfo/iommu --===============8120415809466771585==--