From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 4023AC433F5 for ; Fri, 1 Oct 2021 03:28:23 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 1DFBB6134F for ; Fri, 1 Oct 2021 03:28:23 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1351795AbhJADaE (ORCPT ); Thu, 30 Sep 2021 23:30:04 -0400 Received: from verein.lst.de ([213.95.11.211]:33481 "EHLO verein.lst.de" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230283AbhJADaD (ORCPT ); Thu, 30 Sep 2021 23:30:03 -0400 Received: by verein.lst.de (Postfix, from userid 2407) id C1AB068AFE; Fri, 1 Oct 2021 05:28:16 +0200 (CEST) Date: Fri, 1 Oct 2021 05:28:16 +0200 From: "hch@lst.de" To: Jason Gunthorpe Cc: Jean-Philippe Brucker , "Tian, Kevin" , "kvm@vger.kernel.org" , "jasowang@redhat.com" , "kwankhede@nvidia.com" , "hch@lst.de" , "Jiang, Dave" , "Raj, Ashok" , "corbet@lwn.net" , "parav@mellanox.com" , Alex Williamson , "lkml@metux.net" , "david@gibson.dropbear.id.au" , "dwmw2@infradead.org" , "Tian, Jun J" , "linux-kernel@vger.kernel.org" , "lushenming@huawei.com" , "pbonzini@redhat.com" , "robin.murphy@arm.com" Subject: Re: [RFC 10/20] iommu/iommufd: Add IOMMU_DEVICE_GET_INFO Message-ID: <20211001032816.GC16450@lst.de> References: <20210923112716.GE964074@nvidia.com> <20210923122220.GL964074@nvidia.com> <20210929123630.GS964074@nvidia.com> <20210930220446.GF964074@nvidia.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20210930220446.GF964074@nvidia.com> User-Agent: Mutt/1.5.17 (2007-11-01) Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On Thu, Sep 30, 2021 at 07:04:46PM -0300, Jason Gunthorpe wrote: > > On Arm cache coherency is configured through PTE attributes. I don't think > > PCI No_snoop should be used because it's not necessarily supported > > throughout the system and, as far as I understand, software can't discover > > whether it is. > > The usage of no-snoop is a behavior of a device. A generic PCI driver > should be able to program the device to generate no-snoop TLPs and > ideally rely on an arch specific API in the OS to trigger the required > cache maintenance. Well, it is a combination of the device, the root port and the driver which all need to be in line to use this. > It doesn't make much sense for a portable driver to rely on a > non-portable IO PTE flag to control coherency, since that is not a > standards based approach. > > That said, Linux doesn't have a generic DMA API to support > no-snoop. The few GPUs drivers that use this stuff just hardwired > wbsync on Intel.. Yes, as usual the GPU folks come up with nasty hacks instead of providing generic helper. Basically all we'd need to support it in a generic way is: - a DMA_ATTR_NO_SNOOP (or DMA_ATTR_FORCE_NONCOHERENT to fit the Linux terminology) which treats the current dma_map/unmap/sync calls as if dev_is_dma_coherent was false - a way for the driver to discover that a given architecture / running system actually supports this > What I don't really understand is why ARM, with an IOMMU that supports > PTE WB, has devices where dev_is_dma_coherent() == false ? Because no IOMMU in the world can help that fact that a periphal on the SOC is not part of the cache coherency protocol.