Re: [RFC PATCH 3/5] iommu/arm-smmu-v3: Implement pm_runtime & system sleep ops

[Nicolin Chen <nicolinc@nvidia.com>]

On Wed, Mar 19, 2025 at 12:42:52AM +0000, Pranjal Shrivastava wrote:
> +static int __maybe_unused arm_smmu_runtime_suspend(struct device *dev)
> +{
> +	struct arm_smmu_device *smmu = dev_get_drvdata(dev);
> +
> +	/* We might get the vcmdq */
> +	struct arm_smmu_cmdq_ent cmd = {
> +		.opcode	= smmu->features & ARM_SMMU_FEAT_E2H ?
> +			CMDQ_OP_TLBI_EL2_VA : CMDQ_OP_TLBI_NH_VA,
> +	};
> +
> +	struct arm_smmu_cmdq *cmdq = arm_smmu_get_cmdq(smmu, &cmd);
> +	struct arm_smmu_ll_queue *llq = &cmdq->q.llq;
> +
> +	/*
> +	 * Since suspend is invoked when all clients have been
> +	 * we don't expect more commands to be added to the cmdq.
> +	 * Thus, wait for all existing commands to complete.
> +	 */
> +	arm_smmu_cmdq_shared_lock(cmdq);
> +	arm_smmu_cmdq_poll_until_empty(smmu, cmdq, llq);
> +	arm_smmu_cmdq_shared_unlock(cmdq);

Hmm, I just realized this: with an SMMU having multiple CMDQs
(currently with vCMDQs and potentially with ECMDQs), should we
make sure all cmdqs (not only the cmdq picked in this function
by the arm_smmu_get_cmdq call above) to be empty?

On a system with vCMDQs, there are currently one standard SMMU
CMDQ and two vCMDQs, i.e. totally 3 cmdqs that could be picked
in this context. Perhaps SMMU might need a list of cmdqs that
any new allocated cmdq must be added to, so we can iterate all
the cmdqs in the list?

Thanks
Nicolin