Re: [PATCH] iommu/arm-smmu-v3: Allow disabling Stage 1 translation

[Jason Gunthorpe <jgg@ziepe.ca>]

On Thu, Apr 23, 2026 at 10:47:49AM +0100, Will Deacon wrote:
> On Thu, Apr 23, 2026 at 10:44:08AM +0100, Will Deacon wrote:
> > On Wed, Apr 22, 2026 at 01:23:51PM -0300, Jason Gunthorpe wrote:
> > > On Wed, Apr 22, 2026 at 06:44:31AM +0000, Evangelos Petrongonas wrote:
> > > > The motivation is live update of the hypervisor: we want to kexec into a
> > > > new kernel while keeping DMA from passthrough devices flowing, which
> > > > means the SMMU's translation state has to survive the handover. The Live
> > > > Update Orchestrator work [1] and the in-progress  "iommu: Add live
> > > > update state preservation" series [2] are building exactly this plumbing
> > > > on top of KHO; [2]'s cover letter calls out Arm SMMUv3 support as future
> > > > work, and an earlier RFC from Amazon [3] sketched the same idea for
> > > > iommufd.
> > > 
> > > It would be appropriate to keep this patch with the rest of that out
> > > of tree pile, for example in the series that enables s2 only support
> > > in smmuv3.
> > > 
> > > > For this use case, Stage 2 is materially easier to persist than Stage 1,
> > > > for structural rather than performance reasons: 
> > > 
> > > I don't think so. The driver needs to know each and every STE that
> > > will survive KHO. The ones that don't survive need to be reset to
> > > abort STEs. From that point it is trivial enough to include the CD
> > > memory in the preservation.
> > > 
> > > It would help to send a preparation series to switch the ARM STE and
> > > CD logic away from dma_alloc_coherent and use iommu-pages instead,
> > > since we only expect iommu-pages to support preservation..
> > 
> > Does iommu-pages provide a mechanism to map the memory as non-cacheable
> > if the SMMU isn't coherent? 

No, it has to use CMOs today.

It looks like all the stuff dma_alloc_coherent does to make a
non-cached mapping are pretty arch specific. I don't know if there is
a way we could make more general code get a struct page into an
uncached KVA and meet all the arch rules?

I also think dma_alloc_coherent is far to complex, with pools and
more, to support KHO.

> > I really don't want to entertain CMOs for > the queues.
> 
> Sorry, I said "queues" here but I was really referring to any of the
> current dma_alloc_coherent() allocations and it's the CDs that matter
> in this thread.

queues shouldn't change they are too performance sensitive

> The rationale being that:
> 
> 1. A cacheable mapping is going to pollute the cache unnecessarily.
> 2. Reasoning about atomicity and ordering is a lot more subtle with CMOs.

The page table suffers from all of these draw backs, and the STE/CD is
touched alot less frequently. It is kind of odd to focus on these
issues with STE/CD when page table is a much bigger problem.

STE/CD is pretty simple now, there is only one place to put the CMO
and the ordering is all handled with that shared code. We no longer
care about ordering beyond all the writes must be visible to HW before
issuing the CMDQ invalidation command - which is the same environment
as the pagetable.

> 3. It seems like a pretty invasive driver change to support live update,
>    which isn't relevant for a lot of systems.

That's sort of the whole story of live update.. Trying to keep it
small means using the abstractions that support it like iommu-pages.

IMHO live update is OK to require coherent only, so at worst it could
use iommu-pages on coherent systems and keep using the
dma_alloc_coherent() for others.

I also don't like this "lot of systems thing". I don't want these
powerful capabilities locked up in some giant CSP's proprietary
kernel.  I want all the companies in the cloud market to have access
to the same feature set. That's what open source is supposed to be
driving toward. I have several interesting use cases for this
functionality already.

It will run probably $50-100B of AI cloud servers at least, I think
that is enough justification.

Jason