Re: [RFC PATCH 3/5] iommu/arm-smmu-v3: Implement pm_runtime & system sleep ops

[Pranjal Shrivastava <praan@google.com>]

On Tue, Apr 15, 2025 at 03:28:58PM -0700, Nicolin Chen wrote:
> On Tue, Apr 15, 2025 at 08:47:03PM +0000, Pranjal Shrivastava wrote:
> > On Mon, Apr 14, 2025 at 02:26:19PM -0700, Nicolin Chen wrote:
> > > > On a system with vCMDQs, there are currently one standard SMMU
> > > > CMDQ and two vCMDQs, i.e. totally 3 cmdqs that could be picked
> > > > in this context. Perhaps SMMU might need a list of cmdqs that
> > > > any new allocated cmdq must be added to, so we can iterate all
> > > > the cmdqs in the list?
> > > 
> > > Another tricky thing: if a vcmdq is assigned to a guest VM, it
> > > won't be added to the "list" above. Should we wait for them or
> > > simply rely on VMM sending a suspend signal to all its VMs?
> > > 
> > 
> > Hmm... I haven't looked at the vCMDQ series (of the vIOMMU) yet but I'm
> > assuming that's what we're talking about here?
> > 
> > I'm not sure if sending a signal to the VMM would help if we have some
> > stalled transactions? I guess we'll need to somehow figure out if
> > there's some PCI device assigned that's using that vIOMMU/vCMDQ ? 
> 
> If all VMs go into suspend prior to the host machine and all
> guest-owned vcmdqs are ensured to be empty, we would be fine.
> 
> But this is difficult to tell, since a VMM might not signal
> a suspend to all guests VMs. And even if it does, not all the
> guest kernels might have implemented a suspend routine in the
> guest vcmdq driver draining all guest-owned vcmdqs.
> 

I skimmed through the vCMDQ series and I have some context now. (Btw,
nicely written cover letter :) )

The issue is that we may have dedicated CMDQs (like the Tegra vCMDQ)
assigned to the guest and we may not know what's their status..and the
whole point of this is to avoid VM_exits for cache invalidations..

Since we're mmap-ing an mmio region for passing it through to the guest,
I asssume the host has access to the prod/cons registers for the vCMDQ?
If yes, then we could have a notifier tell the host driver that a vCMDQ
has been assigned to a VM and the host driver could simply poll on those
indices till it's drained?

Also, this would mean that we'll have to take care if the Guest Kernel
ends up touching the mmio region while the SMMU is suspended?

Let me go through the entire series and the tegra-vcmdq driver once.. if
at all the Guest kernel ends up calling the host-kernel vCMDQ driver, in
that case I think we should be able to handle this well... otherwise
we'd need to disable power management for cases where vCMDQs are
assigned..

I'll just go through the vCMDQ series as well..

> > Maybe we can smartly flush only the PCIe ATC inv and pri resp commands?
> 
> Yes, I suspect the host driver will have to do something like
> that for the reason that I mentioned.
> 
> Thanks
> Nicolin

Thanks
Praan