[PATCH 2/2] iommufd: Periodically reschedule when unmapping

[Aaron Lewis <aaronlewis@google.com>]

Unmapping large DMA regions can be slow and cause a CPU core to be
unresponsive for a long period of time, leading to kernel scheduler
warnings.

For example:

[ 1389.148910] vfio-pci 0007:01:01.3: resetting
[ 1389.252814] vfio-pci 0007:01:01.3: reset done
[ 1392.943771] sched: CPU 49 need_resched set for > 100000053 ns (100 ticks) without schedule
[ 1392.943776] CPU: 49 UID: 0 PID: 32912 Comm: vfio_dma_mappin Kdump: loaded Not tainted 7.1.0-smp-DEV #1 PREEMPTLAZY
[ 1392.943780] Call trace:
[ 1392.943796]  resched_latency_warn+0x7c/0x88
[ 1392.943800]  sched_tick+0x18c/0x208
[ 1392.943804]  update_process_times+0xa8/0xd8
[ 1392.943808]  tick_nohz_handler+0x120/0x1c8
[ 1392.943810]  __hrtimer_run_queues+0x124/0x3a0
[ 1392.943833]  arm_lpae_iova_to_phys+0x80/0xf0 (P)
[ 1392.943835]  arm_smmu_iova_to_phys+0x24/0x38
[ 1392.943837]  iommu_iova_to_phys+0x38/0x58
[ 1392.943839]  batch_from_domain+0x16c/0x218
[ 1392.943840]  __iopt_area_unfill_domain+0x2f0/0x5d8
[ 1392.943841]  iopt_area_unfill_domains+0x9c/0x1c8
[ 1392.943842]  iopt_unmap_iova_range+0xec/0x218
[ 1392.943844]  iopt_unmap_iova+0x38/0x50
[ 1392.943846]  iommufd_ioas_unmap+0xac/0x1f8
[ 1392.943847]  iommufd_fops_ioctl+0x200/0x348

Note that in order for this warning to fire, the kernel must be built
without preemption, or... for testing, the code in batch_from_domain()
can explicitly disable preemption.

This warning was produced on ARM using the VFIO selftest
vfio_dma_mapping_perf_test:

  ./vfio_dma_mapping_perf_test -p -b 256G -a "-v iommufd_anonymous_hugetlb_1gb"

To profile the unmap flow, bpftrace was used with the tooling
described at:
https://lore.kernel.org/kvm/20260630141152.3757722-2-aaronlewis@google.com/

The results show:

  iommufd_ioas_unmap, thread_time = 1086.52ms, count = 1
    __iopt_area_unfill_domain, thread_time = 1086.51ms, count = 1
      batch_from_domain, thread_time = 1086.48ms, count = 1

This shows that the execution time is spent entirely inside
batch_from_domain(). Furthermore, the function is entered only once.
Therefore, to avoid the scheduler warning and allow the core to remain
responsive, add a cond_resched() call to the loop in
batch_from_domain(). Also, limit the cond_resched() call to once
per 1GB to rate limit it.

Additional context on what consumes time within the inner loop can be
seen in this sample perf profile:

  22.96%  [vfio_dma_mapping_perf_test] ([vfio_dma_mapping_perf_test])
  22.96%  vfs_ioctl (ioctl.c) Inlined
  22.96%  iommufd_fops_ioctl (main.c)
  19.38%  iommufd_ioas_unmap (ioas.c)
  19.38%  iopt_unmap_iova (io_pagetable.c)
  19.38%  iopt_unmap_iova_range (io_pagetable.c)
  19.38%  iopt_area_unfill_domains (pages.c)
  19.38%  iopt_area_unfill_domain (pages.c) Inlined
  19.38%  __iopt_area_unfill_domain (pages.c)
  19.38%  iopt_area_unpin_domain (pages.c) Inlined
  18.84%  batch_from_domain (pages.c)
  15.06%  iommu_iova_to_phys (iommu.c)
  14.27%  arm_smmu_iova_to_phys (arm-smmu-v3.c)
  13.29%  arm_lpae_iova_to_phys (io-pgtable-arm.c)
  10.60%  __arm_lpae_iopte_walk (io-pgtable-arm.c)
   7.70%  io_pgtable_visit (io-pgtable-arm.c) Inlined
   4.71%  __arm_lpae_iopte_walk (io-pgtable-arm.c)
   2.31%  io_pgtable_visit (io-pgtable-arm.c) Inlined

Signed-off-by: Aaron Lewis <aaronlewis@google.com>
---
 drivers/iommu/iommufd/pages.c | 4 ++++
 1 file changed, 4 insertions(+)

diff --git a/drivers/iommu/iommufd/pages.c b/drivers/iommu/iommufd/pages.c
index 9bdb2945afe1..e478771c0ccf 100644
--- a/drivers/iommu/iommufd/pages.c
+++ b/drivers/iommu/iommufd/pages.c
@@ -412,6 +412,7 @@ static void batch_from_domain(struct pfn_batch *batch,
 	unsigned int page_offset = 0;
 	unsigned long iova;
 	phys_addr_t phys;
+	int i = 0;
 
 	iova = iopt_area_index_to_iova(area, start_index);
 	if (start_index == iopt_area_index(area))
@@ -428,6 +429,9 @@ static void batch_from_domain(struct pfn_batch *batch,
 		iova += PAGE_SIZE - page_offset;
 		page_offset = 0;
 		start_index++;
+
+		if ((++i % BIT(PUD_ORDER)) == 0)
+			cond_resched();
 	}
 }
 
-- 
2.55.0.141.g00534a21ce-goog