From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from mail-qk1-f178.google.com (mail-qk1-f178.google.com [209.85.222.178]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 0897C31E826 for ; Thu, 23 Apr 2026 14:23:29 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.222.178 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1776954211; cv=none; b=ZjoC04PuWZaHFUATp8dww+IO6t08ardaBPk67/aNDmpvzxLkWjaA/T9zi38bJLEIUsNBfVwTURLNlWhXOJC32Xzw2YZkHQ2x53HZtzGFSJyUKgGM6asDGPehFqqfqIPcZsD4t/QSTwJP+qPn9QDUQpIQ1ykh0gGzyy62WoV24Ns= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1776954211; c=relaxed/simple; bh=m9/+Adp7uzzinYLOPcOlOBCorYgZVbqbRNXtl6rb6hk=; h=Date:From:To:Cc:Subject:Message-ID:References:MIME-Version: Content-Type:Content-Disposition:In-Reply-To; b=a6SL1UrHjGxwDSV30mmZZ5ddyQySiW9A592MWQ8qX3RZbKr1W+Y0+/58LB5b1qzkhhkiEpYFKsBIYKyThxb7H28fJtxzl9YLbUyQcjbAqNOUvXZunC5/LH4DWh8emDcsjO0kdhybgyBdHsAh81D+MPqxKviT1IDfzKsxYX4Uk/w= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=ziepe.ca; spf=pass smtp.mailfrom=ziepe.ca; dkim=pass (2048-bit key) header.d=ziepe.ca header.i=@ziepe.ca header.b=kT/AW257; arc=none smtp.client-ip=209.85.222.178 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=ziepe.ca Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=ziepe.ca Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=ziepe.ca header.i=@ziepe.ca header.b="kT/AW257" Received: by mail-qk1-f178.google.com with SMTP id af79cd13be357-8e8c0c2d2bcso130956985a.1 for ; Thu, 23 Apr 2026 07:23:29 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ziepe.ca; s=google; t=1776954208; x=1777559008; darn=vger.kernel.org; h=in-reply-to:content-transfer-encoding:content-disposition :mime-version:references:message-id:subject:cc:to:from:date:from:to :cc:subject:date:message-id:reply-to; bh=9qVBvXxlgya89SxaZAr5N0EnIlllRggjnL98TRto1Gs=; b=kT/AW257sbG8kiqEWZi04oZ/09mfYqeyoaK/tMo1zWGQ8hQqOR9kSfTUJ2HUFP1HjX g0ETsCxbHTd6WSo+f/0M7vt1JEiWhIpC7JgvU8mYz4i8/Nd4YfKHj/Tj5OmoBWkRDbrG Z2c2JV/py0affOuGW5Pe3SY97exOmsne1xb/LQSD2LpLUcCfSta8oBATMFl1XbB8NYxz IiwURO0GHFZzmM/M9aaLC+Louix1wsmdEwp6/T+1fzHAS/Zylv3gZFftdgwdxLnYnER+ WhvsTRmyiINKWcLwllqn8ChefTxbQn5wuv4kCH000DX8bo0iVJQjYx1jWA4Pj9p96FZy 2teg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1776954208; x=1777559008; h=in-reply-to:content-transfer-encoding:content-disposition :mime-version:references:message-id:subject:cc:to:from:date:x-gm-gg :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=9qVBvXxlgya89SxaZAr5N0EnIlllRggjnL98TRto1Gs=; b=eaUDXRdlXlAVEuc7VaygCjr8eCqRC34voR7ViPOJcrkeqDPQNsQMWjstfgbxZMejdX GAvNYeSwu9+iDvTYAic8TrUoP+U7dWuqJ8M1CEJtQYw+oXWzNgnHKnkqp7NbQR6ncM2H C+XWPC7ND8u/k1WaU0SBSvkDkoQmsiJQzFdf7V84WSLn6EpZvZ7DspB7HLGKMlTPnN3u A2wq7asYO+mJCuiiEm0CeaH1ygcaJF0cfqcN39wCv+SjkG8l5LlLZX0YNUaHxB7LrKZY iq2tV4VY3J0V5FEVPm//YKjmbsoE5rgqF16BPwtIGy9UK7K91LoQff1AJEzKmodtJy6l 57wQ== X-Forwarded-Encrypted: i=1; AFNElJ8iEjoI4N5RpfmUsBQNY3bdrfWuCzLk5VUBvcWcqyz9MOzAS4IoMb/CwBEJ3Kp+LlTiEvnse5lpADrPcEc=@vger.kernel.org X-Gm-Message-State: AOJu0Yym93AyfHzOdCIvy0t/SxCCvIhaSJQQVggBqHngtiwgd9cVHIR0 rohzWNYq84Y9WbbWPIkQz6uuU55PN0+7oPrJsR4kWgpqVTP/R09oAMT9OjQe/tRojkI= X-Gm-Gg: AeBDievxY5MzlBLBRi4SFuQ0GGb66Ae00FN7Q9kbT/YFtyPTn5PgL1gFbNqHMK+8u71 ZmhZRviYs0WsuLSpfWygdQFzbcv7jo8ibgGNIHqEnOHZHyYWgxB9/X1/4m18Do2EzFS9lkVpSao 3VSsJm7/WxekIn1ToCVuloMzCg80lgYQ8zdVcv2C1Iigtr6WX8vZ7M/wkfe5WsOx+SMi41h6xzD RhEOmES7NlFORkEvrxYWBLrJAsAei8d0ZpySk10SqyjPLtWdfhOh/pI6Nc/nVOgV5zktnPS+z9b yzG9H0I0MgtnaBuUaeT7mVFqOxJ43XbnooivNHMPjmr3GtZkK9ZEgz54HXJppq8My3EZmji/BeK gLk9JZ0gr3LtNyIbn+nj1NjLXnfJN/uQioFFTXWIlUAgu5ZJchxKy/WJiIvwtq22pKWUwFXAYkV CZkrfQxxGNuH+qEOvobi9bl+WsyguJTFVwp93NUEXXJnDC4PsZOmOoo36oNd62jm2TCRcauqvXC gU+RFm7ARa0d20uwq1WFNAG/es= X-Received: by 2002:a05:620a:649b:b0:8ea:b4d3:b19c with SMTP id af79cd13be357-8eab4d3b2d6mr2477211985a.28.1776954208170; Thu, 23 Apr 2026 07:23:28 -0700 (PDT) Received: from ziepe.ca (crbknf0213w-47-54-130-67.pppoe-dynamic.high-speed.nl.bellaliant.net. [47.54.130.67]) by smtp.gmail.com with ESMTPSA id af79cd13be357-8eb9becc6c3sm1028305785a.39.2026.04.23.07.23.27 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 23 Apr 2026 07:23:27 -0700 (PDT) Received: from jgg by wakko with local (Exim 4.97) (envelope-from ) id 1wFuxi-0000000EEOw-3RUE; Thu, 23 Apr 2026 11:23:26 -0300 Date: Thu, 23 Apr 2026 11:23:26 -0300 From: Jason Gunthorpe To: Will Deacon Cc: Evangelos Petrongonas , Robin Murphy , Joerg Roedel , Nicolin Chen , Pranjal Shrivastava , Lu Baolu , linux-arm-kernel@lists.infradead.org, iommu@lists.linux.dev, linux-kernel@vger.kernel.org, nh-open-source@amazon.com, Zeev Zilberman Subject: Re: [PATCH] iommu/arm-smmu-v3: Allow disabling Stage 1 translation Message-ID: <20260423142326.GP3611611@ziepe.ca> References: <20260420123221.20801-1-epetron@amazon.de> <20260420124032.GO2577880@ziepe.ca> <20260422064431.GA49867@dev-dsk-epetron-1c-1d4d9719.eu-west-1.amazon.com> <20260422162351.GK3611611@ziepe.ca> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Disposition: inline Content-Transfer-Encoding: 8bit In-Reply-To: On Thu, Apr 23, 2026 at 10:47:49AM +0100, Will Deacon wrote: > On Thu, Apr 23, 2026 at 10:44:08AM +0100, Will Deacon wrote: > > On Wed, Apr 22, 2026 at 01:23:51PM -0300, Jason Gunthorpe wrote: > > > On Wed, Apr 22, 2026 at 06:44:31AM +0000, Evangelos Petrongonas wrote: > > > > The motivation is live update of the hypervisor: we want to kexec into a > > > > new kernel while keeping DMA from passthrough devices flowing, which > > > > means the SMMU's translation state has to survive the handover. The Live > > > > Update Orchestrator work [1] and the in-progress  "iommu: Add live > > > > update state preservation" series [2] are building exactly this plumbing > > > > on top of KHO; [2]'s cover letter calls out Arm SMMUv3 support as future > > > > work, and an earlier RFC from Amazon [3] sketched the same idea for > > > > iommufd. > > > > > > It would be appropriate to keep this patch with the rest of that out > > > of tree pile, for example in the series that enables s2 only support > > > in smmuv3. > > > > > > > For this use case, Stage 2 is materially easier to persist than Stage 1, > > > > for structural rather than performance reasons: > > > > > > I don't think so. The driver needs to know each and every STE that > > > will survive KHO. The ones that don't survive need to be reset to > > > abort STEs. From that point it is trivial enough to include the CD > > > memory in the preservation. > > > > > > It would help to send a preparation series to switch the ARM STE and > > > CD logic away from dma_alloc_coherent and use iommu-pages instead, > > > since we only expect iommu-pages to support preservation.. > > > > Does iommu-pages provide a mechanism to map the memory as non-cacheable > > if the SMMU isn't coherent? No, it has to use CMOs today. It looks like all the stuff dma_alloc_coherent does to make a non-cached mapping are pretty arch specific. I don't know if there is a way we could make more general code get a struct page into an uncached KVA and meet all the arch rules? I also think dma_alloc_coherent is far to complex, with pools and more, to support KHO. > > I really don't want to entertain CMOs for > the queues. > > Sorry, I said "queues" here but I was really referring to any of the > current dma_alloc_coherent() allocations and it's the CDs that matter > in this thread. queues shouldn't change they are too performance sensitive > The rationale being that: > > 1. A cacheable mapping is going to pollute the cache unnecessarily. > 2. Reasoning about atomicity and ordering is a lot more subtle with CMOs. The page table suffers from all of these draw backs, and the STE/CD is touched alot less frequently. It is kind of odd to focus on these issues with STE/CD when page table is a much bigger problem. STE/CD is pretty simple now, there is only one place to put the CMO and the ordering is all handled with that shared code. We no longer care about ordering beyond all the writes must be visible to HW before issuing the CMDQ invalidation command - which is the same environment as the pagetable. > 3. It seems like a pretty invasive driver change to support live update, > which isn't relevant for a lot of systems. That's sort of the whole story of live update.. Trying to keep it small means using the abstractions that support it like iommu-pages. IMHO live update is OK to require coherent only, so at worst it could use iommu-pages on coherent systems and keep using the dma_alloc_coherent() for others. I also don't like this "lot of systems thing". I don't want these powerful capabilities locked up in some giant CSP's proprietary kernel. I want all the companies in the cloud market to have access to the same feature set. That's what open source is supposed to be driving toward. I have several interesting use cases for this functionality already. It will run probably $50-100B of AI cloud servers at least, I think that is enough justification. Jason