[PATCH 0/2] vfio/iommufd: Prevent scheduler warnings when unmapping large regions

[PATCH 0/2] vfio/iommufd: Prevent scheduler warnings when unmapping large regions

[Aaron Lewis]

Both the Type1 IOMMU and IOMMUFD can be guilty of not playing nice with
the scheduler during DMA unmapping on large regions (e.g., 256GB). Add
rate-limited calls to cond_resched() to ensure this doesn't happen.

This series was tested using the selftest proposed here:
 - https://lore.kernel.org/kvm/20260701203311.326798-1-aaronlewis@google.com/

And profiled using the bpftrace tooling proposed here:
 - https://lore.kernel.org/kvm/20260630141152.3757722-2-aaronlewis@google.com/

Note that the kernel must be built with preemption disabled and have
LATENCY_WARN enabled in order to trigger the warnings. With those
conditions met, the issue should reproduce as described in the
changelogs. 

Aaron Lewis (2):
  vfio/type1: Periodically try rescheduling when unmapping
  iommufd: Periodically reschedule when unmapping

 drivers/iommu/iommufd/pages.c   | 4 ++++
 drivers/vfio/vfio_iommu_type1.c | 3 +++
 2 files changed, 7 insertions(+)

-- 
2.55.0.141.g00534a21ce-goog

[PATCH 1/2] vfio/type1: Periodically try rescheduling when unmapping

[Aaron Lewis]

Unmapping large DMA regions can be slow and cause a CPU core to be
unresponsive for a long period of time, leading to kernel scheduler
warnings.

For example:

[ 1404.810673] sched: CPU 19 need_resched set for > 100000026 ns (100 ticks) without schedule
[ 1404.810679] CPU: 19 UID: 0 PID: 34615 Comm: vfio_dma_mappin Kdump: loaded Not tainted 7.1.0-smp-DEV #2 PREEMPTLAZY
[ 1404.810682] Call trace:
[ 1404.810700]  resched_latency_warn+0x7c/0x88
[ 1404.810704]  sched_tick+0x18c/0x208
[ 1404.810708]  update_process_times+0xa8/0xd8
[ 1404.810711]  tick_nohz_handler+0x120/0x1c8
[ 1404.810713]  __hrtimer_run_queues+0x124/0x3a0
[ 1404.810716]  hrtimer_interrupt+0xdc/0x2c0
[ 1404.810718]  arch_timer_handler_phys+0x3c/0x58
[ 1404.810721]  handle_percpu_devid_irq+0x144/0x1b8
[ 1404.810723]  generic_handle_domain_irq+0x44/0x70
[ 1404.810725]  gic_handle_irq+0x1b8/0x320
[ 1404.810726]  call_on_irq_stack+0x30/0x48
[ 1404.810727]  do_interrupt_handler+0x54/0x80
[ 1404.810734]  __arm_lpae_iopte_walk+0x15c/0x1b8 (P)
[ 1404.810736]  arm_lpae_iova_to_phys+0x60/0xf0
[ 1404.810737]  arm_smmu_iova_to_phys+0x24/0x38
[ 1404.810739]  iommu_iova_to_phys+0x38/0x58
[ 1404.810752]  vfio_unmap_unpin+0x8/0x1c0
[ 1404.810753]  vfio_remove_dma+0x38/0x140
[ 1404.810757]  vfio_iommu_type1_ioctl+0x8/0xc80

Note that in order for this warning to fire, the kernel must be built
without preemption, or... for testing, the code around the for loop
can explicitly disable preemption.

This warning was produced on ARM using the VFIO selftest
vfio_dma_mapping_perf_test:

  ./vfio_dma_mapping_perf_test -p -b 256G -a "-v vfio_type1_iommu_anonymous_hugetlb_1gb"

To profile the unmap flow, bpftrace was used with the tooling
described at:
https://lore.kernel.org/kvm/20260630141152.3757722-2-aaronlewis@google.com/

To isolate parts of the runtime, two temporary helper functions were
added to vfio_unmap_unpin() during testing: vfio_try_unpin(), which
encapsulated the outer while loop, and vfio_get_largest_chunk(), which
encapsulated the inner for loop.  Doing this helped identify where the
latency issues were coming from and how the code flowed.

The results show:

  vfio_iommu_type1_unmap_dma, thread_time = 1124.85ms, count = 1
    vfio_unmap_unpin, thread_time = 1124.84ms, count = 1
      vfio_try_unpin, thread_time = 1124.80ms, count = 1
        vfio_get_largest_chunk, thread_time = 1124.79ms, count = 1

This shows that the execution time is spent entirely inside the
inner for loop. Furthermore, the loop is entered only once. Therefore,
To avoid the scheduler warning and allow the system to remain responsive,
add a cond_resched() to the inner for loop in vfio_unmap_unpin().
Also, limit the cond_resched() to once per 1GB to rate limit
it.

Additional context on what consumes time within the inner loop can be
seen in this sample profile from perf:

  24.13%  [vfio_dma_mapping_perf_test] ([vfio_dma_mapping_perf_test])
  24.12%  vfs_ioctl (ioctl.c) Inlined
  24.12%  vfio_fops_unl_ioctl (container.c)
  24.12%  vfio_iommu_type1_ioctl (vfio_iommu_type1.c)
  19.66%  vfio_iommu_type1_unmap_dma (vfio_iommu_type1.c) Inlined
  19.66%  vfio_dma_do_unmap (vfio_iommu_type1.c) Inlined
  19.66%  vfio_remove_dma (vfio_iommu_type1.c)
  19.66%  vfio_unmap_unpin (vfio_iommu_type1.c)
  19.65%  vfio_try_unpin (vfio_iommu_type1.c)
  19.10%  vfio_get_largest_chunk (vfio_iommu_type1.c)
  15.91%  iommu_iova_to_phys (iommu.c)
  15.18%  arm_smmu_iova_to_phys (arm-smmu-v3.c)
  14.09%  arm_lpae_iova_to_phys (io-pgtable-arm.c)
  11.59%  __arm_lpae_iopte_walk (io-pgtable-arm.c)
   8.51%  io_pgtable_visit (io-pgtable-arm.c) Inlined
   5.27%  __arm_lpae_iopte_walk (io-pgtable-arm.c)
   2.64%  io_pgtable_visit (io-pgtable-arm.c) Inlined

Signed-off-by: Aaron Lewis <aaronlewis@google.com>
---
 drivers/vfio/vfio_iommu_type1.c | 3 +++
 1 file changed, 3 insertions(+)

diff --git a/drivers/vfio/vfio_iommu_type1.c b/drivers/vfio/vfio_iommu_type1.c
index c8151ba54de3..e78bc07d9b6e 100644
--- a/drivers/vfio/vfio_iommu_type1.c
+++ b/drivers/vfio/vfio_iommu_type1.c
@@ -1153,6 +1153,7 @@ static long vfio_unmap_unpin(struct vfio_iommu *iommu, struct vfio_dma *dma,
 	int unmapped_region_cnt = 0;
 	long unlocked = 0;
 	size_t pos = 0;
+	int i = 0;
 
 	if (!dma->size)
 		return 0;
@@ -1196,6 +1197,8 @@ static long vfio_unmap_unpin(struct vfio_iommu *iommu, struct vfio_dma *dma,
 			next = iommu_iova_to_phys(domain->domain, iova + len);
 			if (next != phys + len)
 				break;
+			if ((++i % BIT(PUD_ORDER)) == 0)
+				cond_resched();
 		}
 
 		/*
-- 
2.55.0.141.g00534a21ce-goog

[PATCH 2/2] iommufd: Periodically reschedule when unmapping

[Aaron Lewis]

Unmapping large DMA regions can be slow and cause a CPU core to be
unresponsive for a long period of time, leading to kernel scheduler
warnings.

For example:

[ 1389.148910] vfio-pci 0007:01:01.3: resetting
[ 1389.252814] vfio-pci 0007:01:01.3: reset done
[ 1392.943771] sched: CPU 49 need_resched set for > 100000053 ns (100 ticks) without schedule
[ 1392.943776] CPU: 49 UID: 0 PID: 32912 Comm: vfio_dma_mappin Kdump: loaded Not tainted 7.1.0-smp-DEV #1 PREEMPTLAZY
[ 1392.943780] Call trace:
[ 1392.943796]  resched_latency_warn+0x7c/0x88
[ 1392.943800]  sched_tick+0x18c/0x208
[ 1392.943804]  update_process_times+0xa8/0xd8
[ 1392.943808]  tick_nohz_handler+0x120/0x1c8
[ 1392.943810]  __hrtimer_run_queues+0x124/0x3a0
[ 1392.943833]  arm_lpae_iova_to_phys+0x80/0xf0 (P)
[ 1392.943835]  arm_smmu_iova_to_phys+0x24/0x38
[ 1392.943837]  iommu_iova_to_phys+0x38/0x58
[ 1392.943839]  batch_from_domain+0x16c/0x218
[ 1392.943840]  __iopt_area_unfill_domain+0x2f0/0x5d8
[ 1392.943841]  iopt_area_unfill_domains+0x9c/0x1c8
[ 1392.943842]  iopt_unmap_iova_range+0xec/0x218
[ 1392.943844]  iopt_unmap_iova+0x38/0x50
[ 1392.943846]  iommufd_ioas_unmap+0xac/0x1f8
[ 1392.943847]  iommufd_fops_ioctl+0x200/0x348

Note that in order for this warning to fire, the kernel must be built
without preemption, or... for testing, the code in batch_from_domain()
can explicitly disable preemption.

This warning was produced on ARM using the VFIO selftest
vfio_dma_mapping_perf_test:

  ./vfio_dma_mapping_perf_test -p -b 256G -a "-v iommufd_anonymous_hugetlb_1gb"

To profile the unmap flow, bpftrace was used with the tooling
described at:
https://lore.kernel.org/kvm/20260630141152.3757722-2-aaronlewis@google.com/

The results show:

  iommufd_ioas_unmap, thread_time = 1086.52ms, count = 1
    __iopt_area_unfill_domain, thread_time = 1086.51ms, count = 1
      batch_from_domain, thread_time = 1086.48ms, count = 1

This shows that the execution time is spent entirely inside
batch_from_domain(). Furthermore, the function is entered only once.
Therefore, to avoid the scheduler warning and allow the core to remain
responsive, add a cond_resched() call to the loop in
batch_from_domain(). Also, limit the cond_resched() call to once
per 1GB to rate limit it.

Additional context on what consumes time within the inner loop can be
seen in this sample perf profile:

  22.96%  [vfio_dma_mapping_perf_test] ([vfio_dma_mapping_perf_test])
  22.96%  vfs_ioctl (ioctl.c) Inlined
  22.96%  iommufd_fops_ioctl (main.c)
  19.38%  iommufd_ioas_unmap (ioas.c)
  19.38%  iopt_unmap_iova (io_pagetable.c)
  19.38%  iopt_unmap_iova_range (io_pagetable.c)
  19.38%  iopt_area_unfill_domains (pages.c)
  19.38%  iopt_area_unfill_domain (pages.c) Inlined
  19.38%  __iopt_area_unfill_domain (pages.c)
  19.38%  iopt_area_unpin_domain (pages.c) Inlined
  18.84%  batch_from_domain (pages.c)
  15.06%  iommu_iova_to_phys (iommu.c)
  14.27%  arm_smmu_iova_to_phys (arm-smmu-v3.c)
  13.29%  arm_lpae_iova_to_phys (io-pgtable-arm.c)
  10.60%  __arm_lpae_iopte_walk (io-pgtable-arm.c)
   7.70%  io_pgtable_visit (io-pgtable-arm.c) Inlined
   4.71%  __arm_lpae_iopte_walk (io-pgtable-arm.c)
   2.31%  io_pgtable_visit (io-pgtable-arm.c) Inlined

Signed-off-by: Aaron Lewis <aaronlewis@google.com>
---
 drivers/iommu/iommufd/pages.c | 4 ++++
 1 file changed, 4 insertions(+)

diff --git a/drivers/iommu/iommufd/pages.c b/drivers/iommu/iommufd/pages.c
index 9bdb2945afe1..e478771c0ccf 100644
--- a/drivers/iommu/iommufd/pages.c
+++ b/drivers/iommu/iommufd/pages.c
@@ -412,6 +412,7 @@ static void batch_from_domain(struct pfn_batch *batch,
 	unsigned int page_offset = 0;
 	unsigned long iova;
 	phys_addr_t phys;
+	int i = 0;
 
 	iova = iopt_area_index_to_iova(area, start_index);
 	if (start_index == iopt_area_index(area))
@@ -428,6 +429,9 @@ static void batch_from_domain(struct pfn_batch *batch,
 		iova += PAGE_SIZE - page_offset;
 		page_offset = 0;
 		start_index++;
+
+		if ((++i % BIT(PUD_ORDER)) == 0)
+			cond_resched();
 	}
 }
 
-- 
2.55.0.141.g00534a21ce-goog

Re: [PATCH 1/2] vfio/type1: Periodically try rescheduling when unmapping

[sashiko-bot]

Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [Medium] The use of `BIT(PUD_ORDER)` to rate-limit `cond_resched()` evaluates to incorrectly large values on architectures with large page sizes, failing to prevent the scheduler stall.
--

--- Patch [1]: [PATCH 1/2] vfio/type1: Periodically try rescheduling when unmapping ---
commit 2d6c55cffed789bb39f187cfc28ac1de6edd70b2
Author: Aaron Lewis <aaronlewis@google.com>

vfio/type1: Periodically try rescheduling when unmapping

This commit adds a cond_resched() to the inner loop of vfio_unmap_unpin()
to prevent CPU stalls during the unmapping of large DMA regions. It
attempts to rate limit the reschedule check to once per 1GB using
BIT(PUD_ORDER).

> diff --git a/drivers/vfio/vfio_iommu_type1.c b/drivers/vfio/vfio_iommu_type1.c
> --- a/drivers/vfio/vfio_iommu_type1.c
> +++ b/drivers/vfio/vfio_iommu_type1.c
[ ... ]
> @@ -1196,6 +1197,8 @@ static long vfio_unmap_unpin(struct vfio_iommu *iommu, struct vfio_dma *dma,
>  			next = iommu_iova_to_phys(domain->domain, iova + len);
>  			if (next != phys + len)
>  				break;
> +			if ((++i % BIT(PUD_ORDER)) == 0)
> +				cond_resched();

[Severity: Medium]
Will this rate limit fail to prevent scheduler stalls on architectures with
large page sizes?

On an ARM64 system configured with 64KB pages, PUD_SHIFT is 42 and PAGE_SHIFT
is 16, resulting in a PUD_ORDER of 26. BIT(26) evaluates to 67,108,864 loop
iterations.

Since each iteration processes PAGE_SIZE (64KB), the loop would process 4TB
of contiguous memory before calling cond_resched(). As noted in the commit
message, 67 million loop iterations can take over 1 second, which vastly
exceeds the 100ms scheduler tick limit.

Could this use a constant size limit instead, such as (SZ_1G >> PAGE_SHIFT),
to ensure the interval remains consistent across different architectures and
page sizes?

-- 
Sashiko AI review · https://sashiko.dev/#/patchset/20260714210303.3967981-1-aaronlewis@google.com?part=1

Re: [PATCH 2/2] iommufd: Periodically reschedule when unmapping

[sashiko-bot]

Thank you for your contribution! Sashiko AI review found 2 potential issue(s) to consider:
- [High] The `cond_resched()` rate limiter is circumvented for fragmented memory because the loop counter `i` is local and resets on every batch, leaving the scheduler warning unresolved.
- [High] Using `BIT(PUD_ORDER)` to rate-limit `cond_resched()` to 1GB is unportable and scales incorrectly on architectures with larger page sizes, preventing rescheduling.
--

--- Patch [2]: [PATCH 2/2] iommufd: Periodically reschedule when unmapping ---
commit 32d4b71fc4dce755ccc1bb27abe57a685f82b576
Author: Aaron Lewis <aaronlewis@google.com>

iommufd: Periodically reschedule when unmapping

This patch adds a cond_resched() call to batch_from_domain() to avoid
scheduler warnings during large DMA unmaps. It attempts to rate-limit
the yields to approximately once per 1GB of processed memory.

> diff --git a/drivers/iommu/iommufd/pages.c b/drivers/iommu/iommufd/pages.c
> index 03c8379bbc347..e7abb3350b235 100644
> --- a/drivers/iommu/iommufd/pages.c
> +++ b/drivers/iommu/iommufd/pages.c
> @@ -412,6 +412,7 @@ static void batch_from_domain(struct pfn_batch *batch,
>  	unsigned int page_offset = 0;
>  	unsigned long iova;
>  	phys_addr_t phys;
> +	int i = 0;
>  
>  	iova = iopt_area_index_to_iova(area, start_index);
>  	if (start_index == iopt_area_index(area))
> @@ -428,6 +429,9 @@ static void batch_from_domain(struct pfn_batch *batch,
>  		iova += PAGE_SIZE - page_offset;
>  		page_offset = 0;
>  		start_index++;
> +
> +		if ((++i % BIT(PUD_ORDER)) == 0)
> +			cond_resched();

[Severity: High]
Does this regression mean the counter fails to trigger cond_resched() when
operating on fragmented memory?

Because i is a local variable, it resets to 0 on every call to
batch_from_domain(). For highly fragmented memory, batch_add_pfn() will
fill its array and return false early (limited to roughly 5,461 pages by
TEMP_MEMORY_LIMIT).

This early return causes the outer loops, such as iopt_area_unpin_domain(),
to repeatedly call batch_from_domain(). This continuously resets i to 0
before it can ever reach the threshold, meaning cond_resched() is never
called and scheduler lockups can still occur.

[Severity: High]
Is this a regression on architectures where BIT(PUD_ORDER) does not equal 1GB?

The commit message mentions limiting the cond_resched() call to once per 1GB.
While this holds true on architectures like x86_64, it scales incorrectly
on architectures with larger page sizes.

For example, on ARM64 with 64KB pages and 3-level page tables, PUD_ORDER
is 26. BIT(26) iterations of 64KB pages corresponds to 4TB of memory,
not 1GB. The loop will only yield once every 4TB, which effectively defeats
the mitigation on these architectures.

>  	}
>  }

-- 
Sashiko AI review · https://sashiko.dev/#/patchset/20260714210303.3967981-1-aaronlewis@google.com?part=2