Re: [RFC PATCH 3/5] iommu/arm-smmu-v3: Implement pm_runtime & system sleep ops

[Nicolin Chen <nicolinc@nvidia.com>]

On Tue, Apr 15, 2025 at 08:37:45PM +0000, Pranjal Shrivastava wrote:
> On Mon, Apr 14, 2025 at 10:57:32AM -0700, Nicolin Chen wrote:
> > On Wed, Mar 19, 2025 at 12:42:52AM +0000, Pranjal Shrivastava wrote:
> > > +static int __maybe_unused arm_smmu_runtime_suspend(struct device *dev)
> > > +{
> > > +	struct arm_smmu_device *smmu = dev_get_drvdata(dev);
> > > +
> > > +	/* We might get the vcmdq */
> > > +	struct arm_smmu_cmdq_ent cmd = {
> > > +		.opcode	= smmu->features & ARM_SMMU_FEAT_E2H ?
> > > +			CMDQ_OP_TLBI_EL2_VA : CMDQ_OP_TLBI_NH_VA,
> > > +	};
> > > +
> > > +	struct arm_smmu_cmdq *cmdq = arm_smmu_get_cmdq(smmu, &cmd);
> > > +	struct arm_smmu_ll_queue *llq = &cmdq->q.llq;
> > > +
> > > +	/*
> > > +	 * Since suspend is invoked when all clients have been
> > > +	 * we don't expect more commands to be added to the cmdq.
> > > +	 * Thus, wait for all existing commands to complete.
> > > +	 */
> > > +	arm_smmu_cmdq_shared_lock(cmdq);
> > > +	arm_smmu_cmdq_poll_until_empty(smmu, cmdq, llq);
> > > +	arm_smmu_cmdq_shared_unlock(cmdq);
> > 
> > Hmm, I just realized this: with an SMMU having multiple CMDQs
> > (currently with vCMDQs and potentially with ECMDQs), should we
> > make sure all cmdqs (not only the cmdq picked in this function
> > by the arm_smmu_get_cmdq call above) to be empty?
> > 
> > On a system with vCMDQs, there are currently one standard SMMU
> > CMDQ and two vCMDQs, i.e. totally 3 cmdqs that could be picked
> > in this context. Perhaps SMMU might need a list of cmdqs that
> > any new allocated cmdq must be added to, so we can iterate all
> > the cmdqs in the list?
> > 
> 
> Well.. I was thinking to only drain the cmdq if we have ats/pri enabled
> because the only case where we'd wanna drain the cmdq is when:
> 
> 1. There's a pending ATC invalidation
> 2. There's a pending CMD_RESUME / PRI_RESP
> 
> Since for any other invalidations, we'll clear the TLBs & config caches
> on resume. Does the same hold true for the commands supported by
> tegra-vcmdq?
> 
> Is there a spec I could read about tegra vCMDQ and the commands
> supported by it?

I don't have a link on hand. But the concept is very simple.

Think of vCMDQs as additional CPU cores to CPU0. They will be
able to execute the same ATC command as the main SMMU CMDQ.

But by doing arm_smmu_get_cmdq(), it would only get one of the
CMDQs, which can be a vcmdq (out of several) or the SMMU CMDQ.
And that specific CMDQ might not have a command pending, while
one of the other vcmdqs or the SMMU CMDQ is executing commands.

So, the point here is to iterate all the host-managed CMDQs to
wait for all of their completions of ATC/RESUME/PRI or etc.

We can do a list of cmdqs, and add each to the list in the
arm_smmu_init_one_queue(). Alternatively, we could add a new
impl op to defer to the vcmdq driver for a proper waiting,
if we do need to handle guest-owned cmdqs as well.

Thanks
Nicolin