Re: [RFC PATCH 3/5] iommu/arm-smmu-v3: Implement pm_runtime & system sleep ops

[Pranjal Shrivastava <praan@google.com>]

On Wed, Apr 16, 2025 at 10:07:11AM -0300, Jason Gunthorpe wrote:
> On Wed, Apr 16, 2025 at 12:52:28PM +0000, Pranjal Shrivastava wrote:
> > On Wed, Apr 16, 2025 at 09:42:52AM -0300, Jason Gunthorpe wrote:
> > > On Wed, Apr 16, 2025 at 12:29:05PM +0000, Pranjal Shrivastava wrote:
> > > > On Wed, Apr 16, 2025 at 09:02:51AM -0300, Jason Gunthorpe wrote:
> > > > > On Wed, Apr 16, 2025 at 10:24:52AM +0000, Pranjal Shrivastava wrote:
> > > > > 
> > > > > > Also, this would mean that we'll have to take care if the Guest Kernel
> > > > > > ends up touching the mmio region while the SMMU is suspended?
> > > > > 
> > > > > I think this is approaching it backwards.
> > > > > 
> > > > > If power management is supported then the power should be on unless
> > > > > the VM has activated its own virtual power management. VM virtual
> > > > > power mangement would flush the cmdq from the VM side then signal the
> > > > > host that it is OK to power down the SMMU, which the host may or may
> > > > > not do
> > > > > 
> > > > > It doesn't make alot of sense to power down the SMMU while a VM is
> > > > > running...
> > > > > 
> > > > 
> > > > Exactly, that's what I meant to convey..
> > > > So.. can't we simply take a ref as soon as a VM is created (IOMMU is
> > > > assigned to it), maybe from somewhere like vfio_change_dma_owner and
> > > > keep things powered on? That way we keep the IOMMU powered ON if it's
> > > > controlled by the userspace.
> > > > 
> > > > This would also allow us to only care about the host-managed queues
> > > > while suspening because we'll be sure that if we are suspending the
> > > > IOMMU, no VM is active.
> > > 
> > > I thought we agreed VFIO already would need to do something to keep
> > > the power on as it can do DMA at any time?
> > > 
> > 
> > Yes.. but if that's the case why are the Guest-assigned vCMDQs coming
> > into the picture?
> 
> Don't know.
> 
> But the vcmdq's are used by the host too..
> 

Yea..I agree that the host-owned ones would need to be drained.

> > I assumed that there's a situation where the VMM doesn't use
> > VFIO and only uses iommufd to assign/configure the IOMMU, maybe we'd
> > need to get a pm ref there as well? Is that the case?
> 
> Right now you can't reach the iommu HW without a VFIO, so that path is
> closed off.
> 

Ack. 

> > Please tell me if I'm missing something? 
> 
> I think the issue here is the host operation of the vcmdq's
> itself. They need to be quieted just like the normal command queue
> 

YEs, I agree we'll need to handle them as well.

> cmdqs that are assigned to userspace can be ignored, but the upstream
> kernel does not support this yet.

Ack.

> 
> Jason