Re: [PATCH v5 00/10] iommu/arm-smmu-v3: Implement Runtime/System Sleep ops

[Pranjal Shrivastava <praan@google.com>]

On Fri, Feb 06, 2026 at 09:18:06PM +0530, Ashish Mhetre wrote:
> 
> 
> On 1/26/2026 8:41 PM, Pranjal Shrivastava wrote:
> > External email: Use caution opening links or attachments
> > 
> > 
> > As arm-smmu-v3 rapidly finds its way into SoCs designed for hand-held
> > devices, power management capabilities, similar to its predecessors, are
> > crucial for these applications. This series introduces power management
> > support for the arm-smmu-v3 driver.
> > 
> > Design
> > =======
> > The arm-smmu-v3 primarily operates with in-memory data structures
> > through HW registers pointing to these data structures. The proposed design
> > makes use of this fact for implementing suspend and resume ops, centered
> > around a driver-specific biased usage counter.
> > 
> > 1. Biased Usage Counter
> > To safely manage runtime PM state, this series introduces an atomic
> > `nr_cmdq_users` counter. This counter is "biased" by being initialized to
> > 1, representing an "idle but active" state.
> > 
> > The suspend callback waits for this counter to be exactly 1 and then
> > atomically transitions it to 0, signifying a "suspended" state. Any
> > other value indicates that the command queue is in use, and the suspend
> > operation is aborted.
> > 
> > Code paths that submit commands to the hardware (the "fast path") use
> > `atomic_inc_not_zero()` to acquire a reference. This operation only
> > succeeds if the counter is not zero (i.e., not suspended), effectively
> > gating hardware access. The reference is dropped using
> > `atomic_dec_return_release()` after the hardware interaction is
> > complete.
> > 
> > 2. Suspend / Resume Flow
> > The "suspend" op now polls for a short period (100us) for the usage
> > counter to become 1. Once successful, it configures the GBPA register
> > to abort new transactions and disables the SMMU, EVTQ, and PRIQ, leaving
> > only the CMDQ enabled to drain any in-flight commands.
> > 
> > The "resume" operation uses the `arm_smmu_device_reset` function, which
> > re-initializes the HW using the SW-copies maintained by the driver
> > (e.g., prod/cons pointers, queue base addresses) and clears all cached
> > configurations.
> > 
> > 3. Guarding Hardware Access
> > Instead of eliding operations, the driver now ensures the SMMU is active
> > before any hardware access. This is managed by `arm_smmu_rpm_get()` and
> > `arm_smmu_rpm_put()` helpers, which wrap the standard runtime PM calls.
> > These wrappers are now used throughout the driver in interrupt handlers,
> > TLB invalidation paths, and any other function that touches hardware
> > registers. This ensures that operations are implicitly queued until the
> > device resumes.
> > 
> > 4. Interrupt Re-config
> > a. Wired irqs: The series refactors the `arm_smmu_setup_irqs` to be
> > able to enable/disable irqs and install their handlers separately to
> > help with the re-initialization of the interrupts correctly.
> > 
> > b. MSIs: The series relies on caching the `msi_msg` and retrieving it
> > through a newly introduced helper `arm_smmu_resume_msis()` which
> > re-configures the *_IRQ_CFGn registers by writing back the cached msi_msg.
> > 
> > Call for review
> > Any insights/comments on the proposed changes are appreciated,
> > especially in areas related to locking, atomic contexts, and potential
> > optimizations.
> > 
> > Note: The series isn't tested with MSIs and is weakly tested for PCIe
> > clients. The same holds true for tegra241_cmdv changes. Any help in
> > reviewing and testing these parts is much appreciated.
> > 
> > [v5]
> >   - Refactored GERROR handling into a helper function and invoked it during
> >     runtime suspend after disabling the SMMU to capture any late-breaking
> >     gerrors as suggested by Jason.
> >   - Updated `arm_smmu_page_response` to be power-state aware and drop
> >     page faults received while suspended.
> >   - Included a patch from Ashish to correctly restore PROD and CONS
> >     indices for tegra241-cmdqv after a hardware reset.
> >   - Collected Reviewed-bys from Mostafa and Nicolin.
> > 
> > [v4]
> >   - https://lore.kernel.org/all/20251117191433.3360130-1-praan@google.com/
> >   - Dropped the `pm_runtime_get_if_not_suspended()` API in favor of a
> >     simpler, driver-specific biased counter (`nr_cmdq_users`) to manage
> >     runtime PM state.
> >   - Reworked the suspend callback to poll on the biased counter before
> >     disabling the SMMU.
> >   - Addressed comments for the MSI refactor.
> > 
> > [v3]
> >   - https://lore.kernel.org/all/20250616203149.2649118-1-praan@google.com/
> >   - Introduced `pm_runtime_get_if_not_suspended` API to avoid races due
> >     to bouncing RPM states while eliding TLBIs as pointed out by Daniel.
> >   - Addressed Nicolin's comments regarding msi_resume and CMDQV flush
> >   - Addressed Daniel's comments about CMDQ locking and draining
> >   - Addressed issues related to draining the evtq and priq
> >   - Dropped the code to identify and track user-space attachments
> > 
> > [v2]
> >   - https://lore.kernel.org/all/20250418233409.3926715-1-praan@google.com/
> >   - Introduced `arm_smmu_rpm_get_if_active` for eliding TLBIs & CFGIs
> >   - Updated the rpm helper invocation strategy.
> >   - Drained all queues in suspend callback (including tegra241-cmdv)
> >   - Cache and restore msi_msg instead of free-ing realloc-ing on resume
> >   - Added support to identify and track user-space attachments
> >   - Fixed the setup_irqs as per Nicolin & Mostafa's suggestions
> >   - Used force_runtime_suspend/resume instead as per Mostafa's suggestion.
> >   - Added "Reviewed-by" line from Mostafa on an unchanged patch
> > 
> > [v1]
> >   - https://lore.kernel.org/all/20250319004254.2547950-1-praan@google.com/
> > 
> > Ashish Mhetre (1):
> >    iommu/tegra241-cmdqv: Restore PROD and CONS after resume
> > 
> > Pranjal Shrivastava (9):
> >    iommu/arm-smmu-v3: Refactor arm_smmu_setup_irqs
> >    iommu/arm-smmu-v3: Add a helper to drain cmd queues
> >    iommu/tegra241-cmdqv: Add a helper to drain VCMDQs
> >    iommu/arm-smmu-v3: Cache and restore MSI config
> >    iommu/arm-smmu-v3: Add a usage counter for cmdq
> >    iommu/arm-smmu-v3: Implement pm_runtime & system sleep ops
> >    iommu/arm-smmu-v3: Handle gerror during suspend
> >    iommu/arm-smmu-v3: Enable pm_runtime and setup devlinks
> >    iommu/arm-smmu-v3: Invoke pm_runtime before hw access
> > 
> >   .../arm/arm-smmu-v3/arm-smmu-v3-iommufd.c     |  10 +-
> >   drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c   | 429 ++++++++++++++++--
> >   drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h   |  13 +
> >   .../iommu/arm/arm-smmu-v3/tegra241-cmdqv.c    |  29 ++
> >   4 files changed, 445 insertions(+), 36 deletions(-)
> > 
> > --
> > 2.52.0.457.g6b5491de43-goog
> > 
> 
> Hi Pranjal,
> 
> I have pulled this series and tested on Tegra264 with CMDQV enabled
> and 16 vcmdqs assigned to guest. I ran multiple iterations of system
> suspend and resume with each iteration following by SMMU tests for
> multiple client devices. Everything worked as expected and tests
> were passing.
> 
> Tested-by: Ashish Mhetre <amhetre@nvidia.com>
> 

Thanks a lot for helping test this, Ashish, especially on Tegra CMDQV!

Regards,
Praan