From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 57DB7C54798 for ; Tue, 27 Feb 2024 12:43:39 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:In-Reply-To:MIME-Version:References: Message-ID:Subject:Cc:To:From:Date:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=cqzHQ1MishPstjZbyoREkGevT6sHGqkUHnrtIu5oYN8=; b=F2wES+9vHvjosv /Kjm9rg8fH/el9gnhSMaMj+4CAGD1swZzgIT1dWiYEUQhl2Mm2BOYRnOL+RsY12e8TFgkbCPuEaYD Eoh99bmyadN+i/9lOmkRTrSzyrEcnE443LxK6Sfe7hiIT+KJZ/c7LumJHU8a9fBmoc6aVNuKHm5Ry kMeqevgXuxEDC9m0ESmPh9+uGQYLVDR4fzFX0YjKfb/+wxNw4QK2Tkg7YblU75WLNWl7oCk1jI7pQ SrRfBO2Ee9hLUmeJRP/hKiM46SRJl+N1jb0XEeNhd9wEYAEB/25zn2xe+rH3ka0YVHBBqxvFIiEw0 1+WzTpyAO1EQfES8CnoQ==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.97.1 #2 (Red Hat Linux)) id 1rewnx-00000005Edk-2GCd; Tue, 27 Feb 2024 12:43:29 +0000 Received: from dfw.source.kernel.org ([139.178.84.217]) by bombadil.infradead.org with esmtps (Exim 4.97.1 #2 (Red Hat Linux)) id 1rewnu-00000005EcL-1OCM for linux-arm-kernel@lists.infradead.org; Tue, 27 Feb 2024 12:43:27 +0000 Received: from smtp.kernel.org (transwarp.subspace.kernel.org [100.75.92.58]) by dfw.source.kernel.org (Postfix) with ESMTP id CACE361241; Tue, 27 Feb 2024 12:43:25 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 8B72EC433F1; Tue, 27 Feb 2024 12:43:22 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1709037805; bh=Gdt77ksnx11y5DS0VA+3N+NAJ38/cUuc0MumnJGbAs8=; h=Date:From:To:Cc:Subject:References:In-Reply-To:From; b=CIhDQNRyhsuk12fqmDhRrBXuSjcniVMPQPDRa6jNCYpJ/iGkiW+Esxopeq0PjOh+T H46XCwmJUMFYcNhvxqxN9ImZOcxqq/oKlYqSnWmI11XDPOC7zadLmtGKko7KIthAsl y3d9wZnQJu5BnHPGucjL6e6Xz2m/X/9YYi7zmBhigkI5Nhk6KuysEobc0IAfirczce XR7P5GMomohVwtJaCg0px+ccRlI8lLDp7d3gpcZWFv9WOxPOMgRscee3TailKEt7Aq 1jHIklenbdRc1cWt8rz/4k2ZOWg/OvuGTbG4TNYDE+9z76Jh7/aLIXadu1XwYIUKDb u++P2Rsp58R2A== Date: Tue, 27 Feb 2024 12:43:18 +0000 From: Will Deacon To: Jason Gunthorpe Cc: iommu@lists.linux.dev, Joerg Roedel , linux-arm-kernel@lists.infradead.org, Robin Murphy , Lu Baolu , Jean-Philippe Brucker , Joerg Roedel , Moritz Fischer , Moritz Fischer , Michael Shavit , Nicolin Chen , patches@lists.linux.dev, Shameer Kolothum , Mostafa Saleh , Zhangfei Gao Subject: Re: [PATCH v5 01/17] iommu/arm-smmu-v3: Make STE programming independent of the callers Message-ID: <20240227124318.GA14089@willie-the-truck> References: <0-v5-cd1be8dd9c71+3fa-smmuv3_newapi_p1_jgg@nvidia.com> <1-v5-cd1be8dd9c71+3fa-smmuv3_newapi_p1_jgg@nvidia.com> <20240215134952.GA690@willie-the-truck> <20240215160135.GL1088888@nvidia.com> <20240221134923.GA7362@willie-the-truck> <20240221140818.GA2635804@nvidia.com> <20240222174346.GB9488@willie-the-truck> <20240223151841.GH13330@nvidia.com> MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: <20240223151841.GH13330@nvidia.com> User-Agent: Mutt/1.10.1 (2018-07-13) X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20240227_044326_482883_1D2469A6 X-CRM114-Status: GOOD ( 32.10 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org On Fri, Feb 23, 2024 at 11:18:41AM -0400, Jason Gunthorpe wrote: > On Thu, Feb 22, 2024 at 05:43:46PM +0000, Will Deacon wrote: > > On Wed, Feb 21, 2024 at 10:08:18AM -0400, Jason Gunthorpe wrote: > > > On Wed, Feb 21, 2024 at 01:49:23PM +0000, Will Deacon wrote: > > > > > > > Very roughly, yes, although I'd go further and just return a bitmap of > > > > used qwords instead of tracking these bits. Basically, we could have some > > > > #defines saying which qwords are used by which configs, > > > > > > I don't think this will work well for CD's EPD0 case.. > > > > > > static void arm_smmu_get_cd_used(const __le64 *ent, __le64 *used_bits) > > > { > > > used_bits[0] = cpu_to_le64(CTXDESC_CD_0_V); > > > if (!(ent[0] & cpu_to_le64(CTXDESC_CD_0_V))) > > > return; > > > memset(used_bits, 0xFF, sizeof(struct arm_smmu_cd)); > > > > > > /* EPD0 means T0SZ/TG0/IR0/OR0/SH0/TTB0 are IGNORED */ > > > if (ent[0] & cpu_to_le64(CTXDESC_CD_0_TCR_EPD0)) { > > > used_bits[0] &= ~cpu_to_le64( > > > CTXDESC_CD_0_TCR_T0SZ | CTXDESC_CD_0_TCR_TG0 | > > > CTXDESC_CD_0_TCR_IRGN0 | CTXDESC_CD_0_TCR_ORGN0 | > > > CTXDESC_CD_0_TCR_SH0); > > > used_bits[1] &= ~cpu_to_le64(CTXDESC_CD_1_TTB0_MASK); > > > } > > > } > > > > Please can you explain more about the issue here? I know what EPDx are, > > but I'm not understanding why they're problematic. This presumably > > involves a hitless transition to/from an aborting CD? > > When a process using SVA exits uncleanly the MM is released so the > SMMU HW must stop chasing the page table pointers since all that > memory will be freed. > > However, in an unclean exit we can't control the order of shutdown so > something like uacce or RDMA may not have quieted the DMA device yet. > > So there is a period during shutdown where the mm has been released > and the device is doing DMA, the desire is that the DMA continue to be > handled as a PRI and the SW will return failure for all PRI requests. > > Specifically we do not want to trigger any dmesg log events during > this condition. Curious, but why is it problematic to log events? As you say, it's an "unclean" exit, so it doesn't seem that unreasonable to me. > Jean-Philippe came up with this solution where we hitlessly use EPD0 > in release to allow the mm to release the page table while continuing > to use the PRI flow. > > So it is going from a "SVA domain with a page table" to a "SVA domain > without a page table but EPD0 set", hitlessly. Ok, and so the reason this adds complexity is because the set of used bits/qwords changes based on something other than the cfg? I think it's a pretty weak argument for field vs qwords, but it's a good counter-example to my naive approach of per-config masks, so thanks. > BTW Michael's self test won't be in part 1 because it needs the ops to > be restored in order to work (now done in part 2), and has a few other > more minor dependencies on part 2 and 3. That's a pity, but fair enough. Will _______________________________________________ linux-arm-kernel mailing list linux-arm-kernel@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-arm-kernel