Re: [RFC PATCH 2/5] iommu/arm-smmu-v3: Add a helper to wait till cmdq drains

[Pranjal Shrivastava <praan@google.com>]

On Thu, Mar 20, 2025 at 10:30:23PM +0000, Mostafa Saleh wrote:
> On Wed, Mar 19, 2025 at 12:42:51AM +0000, Pranjal Shrivastava wrote:
> > Before we suspend SMMU, we want to ensure that all commands have been
> > consumed by the SMMU's command queue, i.e. the cmdq is empty. Add a
> > helper function that waits till the cmdq is empty using `queue_empty`.
> > 
> > Signed-off-by: Pranjal Shrivastava <praan@google.com>
> > ---
> >  drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c | 24 +++++++++++++++++++++
> >  1 file changed, 24 insertions(+)
> > 
> > diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
> > index 80362d9f293b..f65d9bca0392 100644
> > --- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
> > +++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
> > @@ -754,6 +754,30 @@ static int arm_smmu_cmdq_poll_until_sync(struct arm_smmu_device *smmu,
> >  	return __arm_smmu_cmdq_poll_until_consumed(smmu, cmdq, llq);
> >  }
> >  
> > +/*
> > + * Wait until the SMMU cmdq is empty
> > + * Must be called with the cmdq lock held in some capacity.
> > + */
> > +static int arm_smmu_cmdq_poll_until_empty(struct arm_smmu_device *smmu,
> > +					  struct arm_smmu_cmdq *cmdq,
> > +					  struct arm_smmu_ll_queue *llq)
> > +{
> > +	struct arm_smmu_queue_poll qp;
> > +	int ret = 0;
> > +
> > +	queue_poll_init(smmu, &qp);
> > +	llq->val = READ_ONCE(cmdq->q.llq.val);
> > +	do {
> > +		if (queue_empty(llq))
> > +			break;
> > +
> > +		ret = queue_poll(&qp);
> > +		llq->cons = readl(cmdq->q.cons_reg);
> 
> If the command queue is not empty, that means some other thread is already
> polling it, so I guess there is no need to do any MMIO here, just poll
> the SW view of prod/cons, something similar to arm_smmu_cmdq_poll_until_not_full()
> 

That's a good insight. But, I don't understand how we can avoid MMIO
here? The commands that are inserted without (sync == true) won't update
the llq.cons... in which case if we keep polling on the SW copies of
prod and cons we'd never know if the queue was drained? I believe we
will have to poll on the cons register.

> Thanks,
> Mostafa

Thanks,
Praan