Re: [RFC PATCH 3/5] iommu/arm-smmu-v3: Implement pm_runtime & system sleep ops

[Pranjal Shrivastava <praan@google.com>]

On Tue, Apr 15, 2025 at 03:13:20PM -0700, Nicolin Chen wrote:
> On Tue, Apr 15, 2025 at 08:37:45PM +0000, Pranjal Shrivastava wrote:
> > On Mon, Apr 14, 2025 at 10:57:32AM -0700, Nicolin Chen wrote:
> > > On Wed, Mar 19, 2025 at 12:42:52AM +0000, Pranjal Shrivastava wrote:
> > > > +static int __maybe_unused arm_smmu_runtime_suspend(struct device *dev)
> > > > +{
> > > > +	struct arm_smmu_device *smmu = dev_get_drvdata(dev);
> > > > +
> > > > +	/* We might get the vcmdq */
> > > > +	struct arm_smmu_cmdq_ent cmd = {
> > > > +		.opcode	= smmu->features & ARM_SMMU_FEAT_E2H ?
> > > > +			CMDQ_OP_TLBI_EL2_VA : CMDQ_OP_TLBI_NH_VA,
> > > > +	};
> > > > +
> > > > +	struct arm_smmu_cmdq *cmdq = arm_smmu_get_cmdq(smmu, &cmd);
> > > > +	struct arm_smmu_ll_queue *llq = &cmdq->q.llq;
> > > > +
> > > > +	/*
> > > > +	 * Since suspend is invoked when all clients have been
> > > > +	 * we don't expect more commands to be added to the cmdq.
> > > > +	 * Thus, wait for all existing commands to complete.
> > > > +	 */
> > > > +	arm_smmu_cmdq_shared_lock(cmdq);
> > > > +	arm_smmu_cmdq_poll_until_empty(smmu, cmdq, llq);
> > > > +	arm_smmu_cmdq_shared_unlock(cmdq);
> > > 
> > > Hmm, I just realized this: with an SMMU having multiple CMDQs
> > > (currently with vCMDQs and potentially with ECMDQs), should we
> > > make sure all cmdqs (not only the cmdq picked in this function
> > > by the arm_smmu_get_cmdq call above) to be empty?
> > > 
> > > On a system with vCMDQs, there are currently one standard SMMU
> > > CMDQ and two vCMDQs, i.e. totally 3 cmdqs that could be picked
> > > in this context. Perhaps SMMU might need a list of cmdqs that
> > > any new allocated cmdq must be added to, so we can iterate all
> > > the cmdqs in the list?
> > > 
> > 
> > Well.. I was thinking to only drain the cmdq if we have ats/pri enabled
> > because the only case where we'd wanna drain the cmdq is when:
> > 
> > 1. There's a pending ATC invalidation
> > 2. There's a pending CMD_RESUME / PRI_RESP
> > 
> > Since for any other invalidations, we'll clear the TLBs & config caches
> > on resume. Does the same hold true for the commands supported by
> > tegra-vcmdq?
> > 
> > Is there a spec I could read about tegra vCMDQ and the commands
> > supported by it?
> 
> I don't have a link on hand. But the concept is very simple.
> 
> Think of vCMDQs as additional CPU cores to CPU0. They will be
> able to execute the same ATC command as the main SMMU CMDQ.
> 
> But by doing arm_smmu_get_cmdq(), it would only get one of the
> CMDQs, which can be a vcmdq (out of several) or the SMMU CMDQ.
> And that specific CMDQ might not have a command pending, while
> one of the other vcmdqs or the SMMU CMDQ is executing commands.
> 

Ack. So, the commands are still the same it's just a diff queue.

> So, the point here is to iterate all the host-managed CMDQs to
> wait for all of their completions of ATC/RESUME/PRI or etc.
> 
> We can do a list of cmdqs, and add each to the list in the
> arm_smmu_init_one_queue(). Alternatively, we could add a new
> impl op to defer to the vcmdq driver for a proper waiting,
> if we do need to handle guest-owned cmdqs as well.
> 

Ack. I'd prefer adding a new impl op for the vcmdq driver to handle it

> Thanks
> Nicolin

Thanks,
Praan