From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-0.6 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,HTML_MESSAGE,MAILING_LIST_MULTI, SPF_HELO_NONE,SPF_PASS autolearn=no autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id D88B2C433DB for ; Sun, 21 Mar 2021 17:31:09 +0000 (UTC) Received: from smtp3.osuosl.org (smtp3.osuosl.org [140.211.166.136]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 5913561946 for ; Sun, 21 Mar 2021 17:31:09 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 5913561946 Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=lists.linux-foundation.org Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=iommu-bounces@lists.linux-foundation.org Received: from localhost (localhost [127.0.0.1]) by smtp3.osuosl.org (Postfix) with ESMTP id 155B2605F0; Sun, 21 Mar 2021 17:31:09 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from smtp3.osuosl.org ([127.0.0.1]) by localhost (smtp3.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id hk7hbXdzwNus; Sun, 21 Mar 2021 17:31:07 +0000 (UTC) Received: from lists.linuxfoundation.org (lf-lists.osuosl.org [140.211.9.56]) by smtp3.osuosl.org (Postfix) with ESMTP id 9CFEA605D1; Sun, 21 Mar 2021 17:31:07 +0000 (UTC) Received: from lf-lists.osuosl.org (localhost [127.0.0.1]) by lists.linuxfoundation.org (Postfix) with ESMTP id 7D049C000A; Sun, 21 Mar 2021 17:31:07 +0000 (UTC) Received: from smtp1.osuosl.org (smtp1.osuosl.org [IPv6:2605:bc80:3010::138]) by lists.linuxfoundation.org (Postfix) with ESMTP id 97C3FC0001 for ; Sun, 21 Mar 2021 17:31:05 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by smtp1.osuosl.org (Postfix) with ESMTP id 86F2183560 for ; Sun, 21 Mar 2021 17:31:05 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Authentication-Results: smtp1.osuosl.org (amavisd-new); dkim=pass (2048-bit key) header.d=svenpeter.dev header.b="VokPC+Ab"; dkim=pass (2048-bit key) header.d=messagingengine.com header.b="QlE9cJva" Received: from smtp1.osuosl.org ([127.0.0.1]) by localhost (smtp1.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id Jg3gg2OWHTvY for ; Sun, 21 Mar 2021 17:31:02 +0000 (UTC) X-Greylist: from auto-whitelisted by SQLgrey-1.8.0 Received: from new2-smtp.messagingengine.com (new2-smtp.messagingengine.com [66.111.4.224]) by smtp1.osuosl.org (Postfix) with ESMTPS id AC8278354D for ; Sun, 21 Mar 2021 17:31:02 +0000 (UTC) Received: from compute3.internal (compute3.nyi.internal [10.202.2.43]) by mailnew.nyi.internal (Postfix) with ESMTP id 62B8058034F; Sun, 21 Mar 2021 13:22:37 -0400 (EDT) Received: from imap21 ([10.202.2.71]) by compute3.internal (MEProxy); Sun, 21 Mar 2021 13:22:37 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=svenpeter.dev; h=mime-version:message-id:in-reply-to:references:date:from:to :cc:subject:content-type; s=fm1; bh=Sh7yO4YyIwW3dq3G9kmAp+S/GX5l VHJLYLmDSBaxRpA=; b=VokPC+AbTb3wsAJKrhvDy0/kYnLFuusAVxfwIEuQytXc zjOAY49NZ2ayj+lW6Tth8P8JcA7a3mgdFxNel1e9dZI9Hr/VLFLfymtHKiKcgwK/ aQIaN+S002w5ZAWEmU8YJ22NxM6ZleyzYouQtXlWdOxQcKgInQXWkB2xLYxxP6PG yeNlDf6UnJSOBlhU5WTf1SOLQG/rVYBK5DXCygun9+YcefnWtzmI3v5uUSNEthdF o4Aig3sgO2pLVHQukNiGA45VklyUWfxgMR8XQlwkHvOAACK4eM0e/PrctukQfw1f FW92R4GVlPWVzcGr749T9MYygKxZN92l88ebKSU+Tw== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:content-type:date:from:in-reply-to :message-id:mime-version:references:subject:to:x-me-proxy :x-me-proxy:x-me-sender:x-me-sender:x-sasl-enc; s=fm2; bh=Sh7yO4 YyIwW3dq3G9kmAp+S/GX5lVHJLYLmDSBaxRpA=; b=QlE9cJvaenJVfCfiXVElRO vNcEORF42pyNZaQS9CTSc2KyDNZMaTR76/T5hSjn08G5nlNOKaUbBtA0HMJs3+Qo /NqVS6XPnyPPa8/IlrR/CZglPNCWMUIa7r/qaAPEgeVtoCXt90czNVFC9NBQkqjs D34/dnsMwMefgD52Vcx5hJI00/eWPrmKCl1MF2N9QRjmd2YRlyTWahsw8PbsxQTI ko+za7i8jbPUgF99qIqm9n/4VpdwQALmui5uBypAeIHGISAQ1Rbg2xegZaY41NHX Wpi08poMCZVgqOHFG4G+NZ9EG6EqO+oXydCNtmM05euALTd7tqo8wmddXrX4X8aw == X-ME-Sender: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgeduledrudegvddguddttdcutefuodetggdotefrod ftvfcurfhrohhfihhlvgemucfhrghsthforghilhdpqfgfvfdpuffrtefokffrpgfnqfgh necuuegrihhlohhuthemuceftddtnecusecvtfgvtghiphhivghnthhsucdlqddutddtmd enucfjughrpefofgggkfgjfhffhffvufgtsegrtderreerredtnecuhfhrohhmpedfufhv vghnucfrvghtvghrfdcuoehsvhgvnhesshhvvghnphgvthgvrhdruggvvheqnecuggftrf grthhtvghrnhepvdetgfejuddvheeigedtfefhvdffiefhiedvgedtudeijedvgffftedv lefgueejnecuvehluhhsthgvrhfuihiivgeptdenucfrrghrrghmpehmrghilhhfrhhomh epshhvvghnsehsvhgvnhhpvghtvghrrdguvghv X-ME-Proxy: Received: by mailuser.nyi.internal (Postfix, from userid 501) id DE3F451C005E; Sun, 21 Mar 2021 13:22:35 -0400 (EDT) X-Mailer: MessagingEngine.com Webmail Interface User-Agent: Cyrus-JMAP/3.5.0-alpha0-206-g078a48fda5-fm-20210226.001-g078a48fd Mime-Version: 1.0 Message-Id: <8360b3b3-296c-450d-abc3-bb47159bf4e1@www.fastmail.com> In-Reply-To: References: <20210320151903.60759-1-sven@svenpeter.dev> Date: Sun, 21 Mar 2021 18:22:15 +0100 To: "Mark Kettenis" Subject: Re: [PATCH 0/3] Apple M1 DART IOMMU driver Cc: arnd@kernel.org, devicetree@vger.kernel.org, will@kernel.org, marcan@marcan.st, linux-kernel@vger.kernel.org, iommu@lists.linux-foundation.org, robh+dt@kernel.org, maz@kernel.org, mohamed.mediouni@caramail.com, robin.murphy@arm.com, linux-arm-kernel@lists.infradead.org, stan@corellium.com X-BeenThere: iommu@lists.linux-foundation.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: Development issues for Linux IOMMU support List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , From: Sven Peter via iommu Reply-To: Sven Peter Content-Type: multipart/mixed; boundary="===============5813216942538423323==" Errors-To: iommu-bounces@lists.linux-foundation.org Sender: "iommu" --===============5813216942538423323== Content-Type: multipart/alternative; boundary=a51a6ec388fd4cb2ab5292471f389a96 --a51a6ec388fd4cb2ab5292471f389a96 Content-Type: text/plain Hi Mark, > On 21. Mar 2021, at 17:00, Mark Kettenis wrote: > > I don't think the first option is going to work for PCIe. PCIe > devices will have to use "iommu-map" properties to map PCI devices to > the right iommu, and the currently implementation seems to assume that > #iommu-cells = <1>. The devictree binding[1] doesn't explicitly state > that it relies on #iommu-cells = <1>, but it isn't clear how the > rid-base to iommu-base mapping mechanism would work when that isn't > the case. > > Now the PCIe DARTs are simpler and seem to have only one "instance" > per DART. So if we keep #iommu-cells = <1> for those, you'd still be > fine using the first approach. Good point, I guess that only leaves option two for now then. Having some DARTs use cells = <1> and others <2> sounds confusing to me. > > As I mentioned before, not all DARTs support the full 32-bit aperture. > In particular the PCIe DARTs support a smaller address-space. It is > not clear whether this is a restriction of the PCIe host controller or > the DART, but the Apple Device Tree has "vm-base" and "vm-size" > properties that encode the base address and size of the aperture. > These single-cell properties which is probably why for the USB DARTs > only "vm-base" is given; since "vm-base" is 0, a 32-bit number > wouldn't be able to encode the full aperture size. We could make them > 64-bit numbers in the Linux device tree though and always be explicit > about the size. Older Sun SPARC machines used a single "virtual-dma" > property to encode the aperture. We could do someting similar. You > would use this property to initialize domain->geometry.aperture_start > and domain->geometry.aperture_end in diff 3/3 of this series. > > I think it would make sense to include this in this series, as this > would make adding support for PCIe very easy, and PCIe gives you > aupport for network (both wired and wireless) and the type-A USB ports > on the mini. Agreed, I'd ideally like to converge on a device tree binding that won't have to change in the near future. I've tried to use an address space larger than 32bit and that seems to work for parts of the dwc3 controller but breaks for the xhci parts because the upper lines don't seem to be connected there (e.g. if xhci tries to use <0x1 0xffff0000> I get a fault for <0 0xffff0000>). Looking at other iommu drivers I have found the following two similar bindings: qcom uses a ranges property with a 64bit address and 32 bit size [1] apps_iommu: iommu@1e20000 { ... ranges = <0 0x1e20000 0x40000>; ... }; and tegra seems to support a dma-window property with 32bit address and size [2] smmu { [...] dma-window = <0 0x40000000>; /* IOVA start & length */ [...] }; I believe there already is of_get_dma_window to handle parsing this in the common iommu code [3] but I can't find any place using it. It's a little bit more complex that we need since it allows to specify the number of cells for both the address and the size but it should allow us to express all possible configurations: usb_dart { [ ... ] #dma-address-cells = <1>; #dma-size-cells = <2>; dma-window = <0 0x1 0x0>; [ ... ] }; pcie_dart { [ ... ] #dma-address-cells = <1>; #dma-size-cells = <1>; dma-window = <0x100000 0x3fe00000>; [ ... ] }; where #dma-address-cells and #dma-size-cells default to #address-cells and #size-cells respectively if I understand the code correctly. That way we could also just always use a 64bit address and size in the DT, e.g. pcie_dart { [ ... ] dma-window = <0 0x100000 0 0x3fe00000>; [ ... ] }; Best, Sven [1] Documentation/devicetree/bindings/iommu/qcom,iommu.txt [2] Documentation/devicetree/bindings/iommu/nvidia,tegra30-smmu.txt [3] drivers/iommu/of_iommu.c --a51a6ec388fd4cb2ab5292471f389a96 Content-Type: text/html Content-Transfer-Encoding: quoted-printable

= Hi Mark,
<= /div>

On 21. Mar 20= 21, at 17:00, Mark Kettenis <mark.kettenis@xs4all.nl> wrote:

I= don't think the first option is going to work for PCIe.  PCIe
<= /div>
devices will have to use "iommu-map" properties to map PCI dev= ices to
the right iommu, and the currently implementation = seems to assume that
#iommu-cells =3D <1>.  The= devictree binding[1] doesn't explicitly state
that it rel= ies on #iommu-cells =3D <1>, but it isn't clear how the
<= div>rid-base to iommu-base mapping mechanism would work when that isn't<= br>
the case.

Now the PCIe DARTs = are simpler and seem to have only one "instance"
per DART.=  So if we keep #iommu-cells =3D <1> for those, you'd still b= e
fine using the first approach.

Good point, I guess that only leaves option two for now= then.
Having some DARTs use cells =3D <1> and other= s <2> sounds confusing to me.



As I mentioned before, = not all DARTs support the full 32-bit aperture.
In particu= lar the PCIe DARTs support a smaller address-space.  It is
not clear whether this is a restriction of the PCIe host controlle= r or
the DART, but the Apple Device Tree has "vm-base" and= "vm-size"
properties that encode the base address and siz= e of the aperture.
These single-cell properties which is p= robably why for the USB DARTs
only "vm-base" is given; sin= ce "vm-base" is 0, a 32-bit number
wouldn't be able to enc= ode the full aperture size.  We could make them
64-bi= t numbers in the Linux device tree though and always be explicit
about the size.  Older Sun SPARC machines used a single "vir= tual-dma"
property to encode the aperture.  We could = do someting similar.  You
would use this property to = initialize domain->geometry.aperture_start
and domain-&= gt;geometry.aperture_end in diff 3/3 of this series.

<= /div>
I think it would make sense to include this in this series, as= this
would make adding support for PCIe very easy, and PC= Ie gives you
aupport for network (both wired and wireless)= and the type-A USB ports
on the mini.



Agreed, I'd id= eally like to converge on a device tree binding
that won't= have to change in the near future.

I've tr= ied to use an address space larger than 32bit and that seems to
work for parts of the dwc3 controller but breaks for the xhci part= s because
the upper lines don't seem to be connected there= (e.g. if xhci tries to
use <0x1 0xffff0000> I get a= fault for <0 0xffff0000>).

Looking a= t other iommu drivers I have found the following two similar
bindings:

qcom uses a ranges property wi= th a 64bit address and 32 bit size [1]

&nbs= p; apps_iommu: iommu@1e20000 {
      ...
      ranges =3D <0 0x1e20000 0x40000>;=
      ...
  };

and tegra seems to support a dma-window property wi= th 32bit address
and size [2]

  smmu {
          [...]
          dma-window =3D <0 0x40= 000000>;    /* IOVA start & length */
&nb= sp;         [...]
  };
<= div>
I believe there already is of_get_dma_window to handl= e parsing this
in the common iommu code [3] but I can't fi= nd any place using it.
It's a little bit more complex that= we need since it allows to specify the
number of cells fo= r both the address and the size but it should allow us to
= express all possible configurations:

 = usb_dart {
      [ ... ]
&nb= sp;     #dma-address-cells =3D <1>;
 =     #dma-size-cells =3D <2>;
   = ;   dma-window =3D <0 0x1 0x0>;
    &= nbsp; [ ... ]
  };
  pcie_dart {
      [ ... ]
    &n= bsp; #dma-address-cells =3D <1>;
     = ; #dma-size-cells =3D <1>;
      dma-= window =3D <0x100000 0x3fe00000>;
    &nbs= p; [ ... ]
  };

where #d= ma-address-cells and #dma-size-cells default to
#address-c= ells and #size-cells respectively if I understand
the code= correctly. That way we could also just always use
a 64bit= address and size in the DT, e.g.

  pc= ie_dart {
      [ ... ]
 = ;     dma-window =3D <0 0x100000 0 0x3fe00000>;
      [ ... ]
  };


Best,

= Sven


[1] Documentation/= devicetree/bindings/iommu/qcom,iommu.txt
[2] Documentation= /devicetree/bindings/iommu/nvidia,tegra30-smmu.txt
[3] dri= vers/iommu/of_iommu.c

--a51a6ec388fd4cb2ab5292471f389a96-- --===============5813216942538423323== Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Disposition: inline _______________________________________________ iommu mailing list iommu@lists.linux-foundation.org https://lists.linuxfoundation.org/mailman/listinfo/iommu --===============5813216942538423323==--