* [PATCH v7 5/8] drm/nouveau: Use hmm_range_fault_unlocked_timeout() for SVM faults
From: Stanislav Kinsburskii @ 2026-07-07 19:47 UTC (permalink / raw)
To: airlied, akhilesh, akpm, corbet, dakr, david, decui, haiyangz,
jgg, kees, kys, leon, liam, lizhi.hou, ljs, longli, lyude,
maarten.lankhorst, mamin506, mhocko, mripard, nouveau, ogabbay,
oleg, rppt, shuah, simona, skhan, skinsburskii, surenb,
tzimmermann, vbabka, wei.liu, skinsburskii
Cc: dri-devel, linux-mm, linux-doc, linux-hyperv, linux-kernel,
linux-kselftest, linux-rdma
In-Reply-To: <178345345668.660027.2952911919681614557.stgit@skinsburskii>
nouveau_range_fault() takes mmap_read_lock() only to call
hmm_range_fault(). It also keeps a single HMM_RANGE_DEFAULT_TIMEOUT
deadline across both HMM -EBUSY retries and post-fault
mmu_interval_read_retry() retries.
Use hmm_range_fault_unlocked_timeout() instead. The HMM helper now owns
the mmap lock and refreshes range->notifier_seq for its internal retries.
Nouveau keeps its existing absolute deadline in the outer loop and passes
the remaining jiffies to the helper for each fault attempt, so retries
caused by mmu_interval_read_retry() do not reset the overall retry budget.
Nouveau still validates the interval notifier sequence while holding
svmm->mutex before programming the GPU mapping.
Signed-off-by: Stanislav Kinsburskii <skinsburskii@gmail.com>
---
drivers/gpu/drm/nouveau/nouveau_svm.c | 11 +++--------
1 file changed, 3 insertions(+), 8 deletions(-)
diff --git a/drivers/gpu/drm/nouveau/nouveau_svm.c b/drivers/gpu/drm/nouveau/nouveau_svm.c
index dcc92131488e..ba93273341af 100644
--- a/drivers/gpu/drm/nouveau/nouveau_svm.c
+++ b/drivers/gpu/drm/nouveau/nouveau_svm.c
@@ -683,15 +683,10 @@ static int nouveau_range_fault(struct nouveau_svmm *svmm,
goto out;
}
- range.notifier_seq = mmu_interval_read_begin(range.notifier);
- mmap_read_lock(mm);
- ret = hmm_range_fault(&range);
- mmap_read_unlock(mm);
- if (ret) {
- if (ret == -EBUSY)
- continue;
+ ret = hmm_range_fault_unlocked_timeout(&range,
+ max_t(long, timeout - jiffies, 1UL));
+ if (ret)
goto out;
- }
mutex_lock(&svmm->mutex);
if (mmu_interval_read_retry(range.notifier,
^ permalink raw reply related
* [PATCH v7 4/8] mshv: Use hmm_range_fault_unlocked_timeout() for region faults
From: Stanislav Kinsburskii @ 2026-07-07 19:47 UTC (permalink / raw)
To: airlied, akhilesh, akpm, corbet, dakr, david, decui, haiyangz,
jgg, kees, kys, leon, liam, lizhi.hou, ljs, longli, lyude,
maarten.lankhorst, mamin506, mhocko, mripard, nouveau, ogabbay,
oleg, rppt, shuah, simona, skhan, skinsburskii, surenb,
tzimmermann, vbabka, wei.liu, skinsburskii
Cc: dri-devel, linux-mm, linux-doc, linux-hyperv, linux-kernel,
linux-kselftest, linux-rdma
In-Reply-To: <178345345668.660027.2952911919681614557.stgit@skinsburskii>
MSHV currently faults movable memory regions by taking mmap_read_lock()
around hmm_range_fault(). That prevents the fault path from handling VMAs
whose fault handlers need to drop mmap_lock, such as userfaultfd-backed
mappings.
Use hmm_range_fault_unlocked_timeout() instead. Passing a timeout of 0
preserves MSHV's existing unbounded retry behavior while letting the HMM
helper own mmap_lock acquisition and refresh range->notifier_seq internally
before walking the range. After the fault succeeds, MSHV still takes
mreg_mutex and checks mmu_interval_read_retry() before installing the pages
into the region, so the existing invalidation synchronization is preserved.
Fold the small fault-and-lock helper into mshv_region_range_fault(), since
the remaining retry path is just the standard "fault, take the driver lock,
check the interval notifier sequence" pattern.
Signed-off-by: Stanislav Kinsburskii <skinsburskii@gmail.com>
---
drivers/hv/mshv_regions.c | 54 ++++++++-------------------------------------
1 file changed, 10 insertions(+), 44 deletions(-)
diff --git a/drivers/hv/mshv_regions.c b/drivers/hv/mshv_regions.c
index 6d65e5b42152..dddaade31b5d 100644
--- a/drivers/hv/mshv_regions.c
+++ b/drivers/hv/mshv_regions.c
@@ -381,46 +381,6 @@ int mshv_region_get(struct mshv_mem_region *region)
return kref_get_unless_zero(®ion->mreg_refcount);
}
-/**
- * mshv_region_hmm_fault_and_lock - Handle HMM faults and lock the memory region
- * @region: Pointer to the memory region structure
- * @range: Pointer to the HMM range structure
- *
- * This function performs the following steps:
- * 1. Reads the notifier sequence for the HMM range.
- * 2. Acquires a read lock on the memory map.
- * 3. Handles HMM faults for the specified range.
- * 4. Releases the read lock on the memory map.
- * 5. If successful, locks the memory region mutex.
- * 6. Verifies if the notifier sequence has changed during the operation.
- * If it has, releases the mutex and returns -EBUSY to match with
- * hmm_range_fault() return code for repeating.
- *
- * Return: 0 on success, a negative error code otherwise.
- */
-static int mshv_region_hmm_fault_and_lock(struct mshv_mem_region *region,
- struct hmm_range *range)
-{
- int ret;
-
- range->notifier_seq = mmu_interval_read_begin(range->notifier);
- mmap_read_lock(region->mreg_mni.mm);
- ret = hmm_range_fault(range);
- mmap_read_unlock(region->mreg_mni.mm);
- if (ret)
- return ret;
-
- mutex_lock(®ion->mreg_mutex);
-
- if (mmu_interval_read_retry(range->notifier, range->notifier_seq)) {
- mutex_unlock(®ion->mreg_mutex);
- cond_resched();
- return -EBUSY;
- }
-
- return 0;
-}
-
/**
* mshv_region_range_fault - Handle memory range faults for a given region.
* @region: Pointer to the memory region structure.
@@ -452,13 +412,19 @@ static int mshv_region_range_fault(struct mshv_mem_region *region,
range.start = region->start_uaddr + page_offset * HV_HYP_PAGE_SIZE;
range.end = range.start + page_count * HV_HYP_PAGE_SIZE;
- do {
- ret = mshv_region_hmm_fault_and_lock(region, &range);
- } while (ret == -EBUSY);
-
+again:
+ ret = hmm_range_fault_unlocked_timeout(&range, 0);
if (ret)
goto out;
+ mutex_lock(®ion->mreg_mutex);
+
+ if (mmu_interval_read_retry(range.notifier, range.notifier_seq)) {
+ mutex_unlock(®ion->mreg_mutex);
+ cond_resched();
+ goto again;
+ }
+
for (i = 0; i < page_count; i++)
region->mreg_pages[page_offset + i] = hmm_pfn_to_page(pfns[i]);
^ permalink raw reply related
* [PATCH v7 3/8] selftests/mm: add HMM tests for mmap lock-dropping faults
From: Stanislav Kinsburskii @ 2026-07-07 19:47 UTC (permalink / raw)
To: airlied, akhilesh, akpm, corbet, dakr, david, decui, haiyangz,
jgg, kees, kys, leon, liam, lizhi.hou, ljs, longli, lyude,
maarten.lankhorst, mamin506, mhocko, mripard, nouveau, ogabbay,
oleg, rppt, shuah, simona, skhan, skinsburskii, surenb,
tzimmermann, vbabka, wei.liu, skinsburskii
Cc: dri-devel, linux-mm, linux-doc, linux-hyperv, linux-kernel,
linux-kselftest, linux-rdma
In-Reply-To: <178345345668.660027.2952911919681614557.stgit@skinsburskii>
Add test_hmm coverage for the HMM lock-dropping fault path. The test module
gets new HMM_DMIRROR_READ_UNLOCKED and HMM_DMIRROR_READ_UNLOCKED_TIMEOUT
ioctls that both call hmm_range_fault_unlocked_timeout(). The unlocked
ioctl passes a timeout of 0 to exercise the unbounded retry mode, while the
timeout ioctl passes HMM_RANGE_DEFAULT_TIMEOUT converted to jiffies.
Add a userfaultfd_read selftest that registers an anonymous mapping with
UFFDIO_REGISTER_MODE_MISSING, services the faults from a handler thread
with UFFDIO_COPY, and verifies that HMM can read back the data supplied by
the handler. This exercises the path where handle_mm_fault() drops
mmap_lock and hmm_range_fault_unlocked_timeout() restarts the walk
internally.
Add a userfaultfd_timeout selftest for the finite-timeout mode. It
registers a missing-mode userfaultfd range without a data-supplying
handler, then closes the userfaultfd after the timeout window so the
blocked fault can unwind and hmm_range_fault_unlocked_timeout() can return
-EBUSY.
Assisted-by: GitHub-Copilot:claude-opus-4.6
Signed-off-by: Stanislav Kinsburskii <skinsburskii@gmail.com>
---
lib/test_hmm.c | 111 ++++++++++++++++
lib/test_hmm_uapi.h | 3
tools/testing/selftests/mm/hmm-tests.c | 220 ++++++++++++++++++++++++++++++++
3 files changed, 334 insertions(+)
diff --git a/lib/test_hmm.c b/lib/test_hmm.c
index 45c0cb992218..02a98887c290 100644
--- a/lib/test_hmm.c
+++ b/lib/test_hmm.c
@@ -389,6 +389,67 @@ static int dmirror_range_fault(struct dmirror *dmirror,
return ret;
}
+static int dmirror_range_fault_unlocked(struct dmirror *dmirror,
+ struct hmm_range *range,
+ unsigned long timeout)
+{
+ int ret;
+
+ while (true) {
+ ret = hmm_range_fault_unlocked_timeout(range, timeout);
+ if (ret)
+ goto out;
+
+ mutex_lock(&dmirror->mutex);
+ if (mmu_interval_read_retry(range->notifier,
+ range->notifier_seq)) {
+ mutex_unlock(&dmirror->mutex);
+ continue;
+ }
+ break;
+ }
+
+ ret = dmirror_do_fault(dmirror, range);
+
+ mutex_unlock(&dmirror->mutex);
+out:
+ return ret;
+}
+
+static int dmirror_fault_unlocked(struct dmirror *dmirror,
+ unsigned long start,
+ unsigned long end, bool write,
+ unsigned long timeout)
+{
+ struct mm_struct *mm = dmirror->notifier.mm;
+ unsigned long addr;
+ unsigned long pfns[32];
+ struct hmm_range range = {
+ .notifier = &dmirror->notifier,
+ .hmm_pfns = pfns,
+ .pfn_flags_mask = 0,
+ .default_flags =
+ HMM_PFN_REQ_FAULT | (write ? HMM_PFN_REQ_WRITE : 0),
+ .dev_private_owner = dmirror->mdevice,
+ };
+ int ret = 0;
+
+ if (!mmget_not_zero(mm))
+ return -EFAULT;
+
+ for (addr = start; addr < end; addr = range.end) {
+ range.start = addr;
+ range.end = min(addr + (ARRAY_SIZE(pfns) << PAGE_SHIFT), end);
+
+ ret = dmirror_range_fault_unlocked(dmirror, &range, timeout);
+ if (ret)
+ break;
+ }
+
+ mmput(mm);
+ return ret;
+}
+
static int dmirror_fault(struct dmirror *dmirror, unsigned long start,
unsigned long end, bool write)
{
@@ -488,6 +549,48 @@ static int dmirror_read(struct dmirror *dmirror, struct hmm_dmirror_cmd *cmd)
return ret;
}
+static int dmirror_read_unlocked(struct dmirror *dmirror,
+ struct hmm_dmirror_cmd *cmd,
+ unsigned long timeout)
+{
+ struct dmirror_bounce bounce;
+ unsigned long start, end;
+ unsigned long size = cmd->npages << PAGE_SHIFT;
+ int ret;
+
+ start = cmd->addr;
+ end = start + size;
+ if (end < start)
+ return -EINVAL;
+
+ ret = dmirror_bounce_init(&bounce, start, size);
+ if (ret)
+ return ret;
+
+ while (1) {
+ mutex_lock(&dmirror->mutex);
+ ret = dmirror_do_read(dmirror, start, end, &bounce);
+ mutex_unlock(&dmirror->mutex);
+ if (ret != -ENOENT)
+ break;
+
+ start = cmd->addr + (bounce.cpages << PAGE_SHIFT);
+ ret = dmirror_fault_unlocked(dmirror, start, end, false, timeout);
+ if (ret)
+ break;
+ cmd->faults++;
+ }
+
+ if (ret == 0) {
+ if (copy_to_user(u64_to_user_ptr(cmd->ptr), bounce.ptr,
+ bounce.size))
+ ret = -EFAULT;
+ }
+ cmd->cpages = bounce.cpages;
+ dmirror_bounce_fini(&bounce);
+ return ret;
+}
+
static int dmirror_do_write(struct dmirror *dmirror, unsigned long start,
unsigned long end, struct dmirror_bounce *bounce)
{
@@ -1572,6 +1675,14 @@ static long dmirror_fops_unlocked_ioctl(struct file *filp,
dmirror->flags = cmd.npages;
ret = 0;
break;
+ case HMM_DMIRROR_READ_UNLOCKED:
+ ret = dmirror_read_unlocked(dmirror, &cmd, 0);
+ break;
+ case HMM_DMIRROR_READ_UNLOCKED_TIMEOUT:
+ ret = dmirror_read_unlocked(dmirror, &cmd,
+ msecs_to_jiffies(
+ HMM_RANGE_DEFAULT_TIMEOUT));
+ break;
default:
return -EINVAL;
diff --git a/lib/test_hmm_uapi.h b/lib/test_hmm_uapi.h
index f94c6d457338..249ab2d9e3b2 100644
--- a/lib/test_hmm_uapi.h
+++ b/lib/test_hmm_uapi.h
@@ -38,6 +38,9 @@ struct hmm_dmirror_cmd {
#define HMM_DMIRROR_CHECK_EXCLUSIVE _IOWR('H', 0x06, struct hmm_dmirror_cmd)
#define HMM_DMIRROR_RELEASE _IOWR('H', 0x07, struct hmm_dmirror_cmd)
#define HMM_DMIRROR_FLAGS _IOWR('H', 0x08, struct hmm_dmirror_cmd)
+#define HMM_DMIRROR_READ_UNLOCKED _IOWR('H', 0x09, struct hmm_dmirror_cmd)
+#define HMM_DMIRROR_READ_UNLOCKED_TIMEOUT \
+ _IOWR('H', 0x0a, struct hmm_dmirror_cmd)
#define HMM_DMIRROR_FLAG_FAIL_ALLOC (1ULL << 0)
diff --git a/tools/testing/selftests/mm/hmm-tests.c b/tools/testing/selftests/mm/hmm-tests.c
index 2f2b9879d100..dc031d2e2aae 100644
--- a/tools/testing/selftests/mm/hmm-tests.c
+++ b/tools/testing/selftests/mm/hmm-tests.c
@@ -29,6 +29,10 @@
#include <sys/mman.h>
#include <sys/ioctl.h>
#include <sys/time.h>
+#include <sys/syscall.h>
+#include <sys/eventfd.h>
+#include <linux/userfaultfd.h>
+#include <poll.h>
/*
* This is a private UAPI to the kernel test module so it isn't exported
@@ -2949,4 +2953,220 @@ TEST_F_TIMEOUT(hmm, benchmark_thp_migration, 120)
&thp_results, ®ular_results);
}
}
+/*
+ * Test that HMM can fault in pages backed by userfaultfd using the
+ * hmm_range_fault_unlocked_timeout() path with no timeout. This exercises
+ * the lock-drop retry logic in the HMM framework.
+ */
+struct uffd_thread_args {
+ int uffd;
+ int stop_fd;
+ void *page_buffer;
+ unsigned long page_size;
+};
+
+static void *uffd_handler_thread(void *arg)
+{
+ struct uffd_thread_args *args = arg;
+ struct uffd_msg msg;
+ struct uffdio_copy copy;
+ struct pollfd pollfd[2];
+ int ret;
+
+ pollfd[0].fd = args->uffd;
+ pollfd[0].events = POLLIN;
+ pollfd[1].fd = args->stop_fd;
+ pollfd[1].events = POLLIN;
+
+ while (1) {
+ ret = poll(pollfd, 2, -1);
+ if (ret <= 0)
+ break;
+ if (pollfd[1].revents)
+ break;
+ if (!(pollfd[0].revents & POLLIN))
+ break;
+
+ ret = read(args->uffd, &msg, sizeof(msg));
+ if (ret != sizeof(msg))
+ break;
+
+ if (msg.event != UFFD_EVENT_PAGEFAULT)
+ break;
+
+ /* Fill the page with a known pattern */
+ memset(args->page_buffer, 0xAB, args->page_size);
+
+ copy.dst = msg.arg.pagefault.address & ~(args->page_size - 1);
+ copy.src = (unsigned long)args->page_buffer;
+ copy.len = args->page_size;
+ copy.mode = 0;
+ copy.copy = 0;
+
+ ret = ioctl(args->uffd, UFFDIO_COPY, ©);
+ if (ret < 0)
+ break;
+ }
+
+ return NULL;
+}
+
+struct uffd_close_args {
+ int uffd;
+ unsigned int delay_us;
+};
+
+static void *uffd_close_thread(void *arg)
+{
+ struct uffd_close_args *args = arg;
+
+ usleep(args->delay_us);
+ close(args->uffd);
+ return NULL;
+}
+
+TEST_F(hmm, userfaultfd_read)
+{
+ struct hmm_buffer *buffer;
+ struct uffd_thread_args uffd_args;
+ unsigned long npages;
+ unsigned long size;
+ unsigned long i;
+ unsigned char *ptr;
+ pthread_t thread;
+ int uffd;
+ int stop_fd;
+ int ret;
+ struct uffdio_api api;
+ struct uffdio_register reg;
+ uint64_t stop = 1;
+ ssize_t nwrite;
+
+ npages = 4;
+ size = npages << self->page_shift;
+
+ /* Create userfaultfd */
+ uffd = syscall(__NR_userfaultfd, O_CLOEXEC | O_NONBLOCK);
+ if (uffd < 0)
+ SKIP(return, "userfaultfd not available");
+
+ api.api = UFFD_API;
+ api.features = 0;
+ ret = ioctl(uffd, UFFDIO_API, &api);
+ ASSERT_EQ(ret, 0);
+
+ buffer = malloc(sizeof(*buffer));
+ ASSERT_NE(buffer, NULL);
+
+ buffer->fd = -1;
+ buffer->size = size;
+ buffer->mirror = malloc(size);
+ ASSERT_NE(buffer->mirror, NULL);
+
+ /* Create anonymous mapping */
+ buffer->ptr = mmap(NULL, size,
+ PROT_READ | PROT_WRITE,
+ MAP_PRIVATE | MAP_ANONYMOUS,
+ -1, 0);
+ ASSERT_NE(buffer->ptr, MAP_FAILED);
+
+ /* Register the region with userfaultfd */
+ reg.range.start = (unsigned long)buffer->ptr;
+ reg.range.len = size;
+ reg.mode = UFFDIO_REGISTER_MODE_MISSING;
+ ret = ioctl(uffd, UFFDIO_REGISTER, ®);
+ ASSERT_EQ(ret, 0);
+
+ /* Set up the handler thread */
+ uffd_args.uffd = uffd;
+ stop_fd = eventfd(0, EFD_CLOEXEC);
+ ASSERT_GE(stop_fd, 0);
+ uffd_args.stop_fd = stop_fd;
+ uffd_args.page_buffer = malloc(self->page_size);
+ ASSERT_NE(uffd_args.page_buffer, NULL);
+ uffd_args.page_size = self->page_size;
+
+ ret = pthread_create(&thread, NULL, uffd_handler_thread, &uffd_args);
+ ASSERT_EQ(ret, 0);
+
+ /*
+ * Use the unlocked read path which allows the mmap lock to be
+ * dropped during the fault, enabling userfaultfd resolution.
+ */
+ ret = hmm_dmirror_cmd(self->fd, HMM_DMIRROR_READ_UNLOCKED,
+ buffer, npages);
+ ASSERT_EQ(ret, 0);
+ ASSERT_EQ(buffer->cpages, npages);
+
+ /* Verify the device read the data filled by the uffd handler */
+ ptr = buffer->mirror;
+ for (i = 0; i < size; ++i)
+ ASSERT_EQ(ptr[i], (unsigned char)0xAB);
+
+ nwrite = write(stop_fd, &stop, sizeof(stop));
+ ASSERT_EQ(nwrite, sizeof(stop));
+ pthread_join(thread, NULL);
+ close(stop_fd);
+ free(uffd_args.page_buffer);
+ close(uffd);
+ hmm_buffer_free(buffer);
+}
+
+TEST_F(hmm, userfaultfd_timeout)
+{
+ struct uffd_close_args close_args;
+ struct hmm_buffer *buffer;
+ unsigned long npages;
+ unsigned long size;
+ pthread_t thread;
+ int uffd;
+ int ret;
+ struct uffdio_api api;
+ struct uffdio_register reg;
+
+ npages = 4;
+ size = npages << self->page_shift;
+
+ uffd = syscall(__NR_userfaultfd, O_CLOEXEC | O_NONBLOCK);
+ if (uffd < 0)
+ SKIP(return, "userfaultfd not available");
+
+ api.api = UFFD_API;
+ api.features = 0;
+ ret = ioctl(uffd, UFFDIO_API, &api);
+ ASSERT_EQ(ret, 0);
+
+ buffer = malloc(sizeof(*buffer));
+ ASSERT_NE(buffer, NULL);
+
+ buffer->fd = -1;
+ buffer->size = size;
+ buffer->mirror = malloc(size);
+ ASSERT_NE(buffer->mirror, NULL);
+
+ buffer->ptr = mmap(NULL, size,
+ PROT_READ | PROT_WRITE,
+ MAP_PRIVATE | MAP_ANONYMOUS,
+ -1, 0);
+ ASSERT_NE(buffer->ptr, MAP_FAILED);
+
+ reg.range.start = (unsigned long)buffer->ptr;
+ reg.range.len = size;
+ reg.mode = UFFDIO_REGISTER_MODE_MISSING;
+ ret = ioctl(uffd, UFFDIO_REGISTER, ®);
+ ASSERT_EQ(ret, 0);
+
+ close_args.uffd = uffd;
+ close_args.delay_us = 1500 * 1000;
+ ret = pthread_create(&thread, NULL, uffd_close_thread, &close_args);
+ ASSERT_EQ(ret, 0);
+
+ ret = hmm_dmirror_cmd(self->fd, HMM_DMIRROR_READ_UNLOCKED_TIMEOUT,
+ buffer, npages);
+ pthread_join(thread, NULL);
+ ASSERT_EQ(ret, -EBUSY);
+
+ hmm_buffer_free(buffer);
+}
+
TEST_HARNESS_MAIN
^ permalink raw reply related
* [PATCH v7 2/8] mm/hmm: add hmm_range_fault_unlocked_timeout() for mmap lock-drop support
From: Stanislav Kinsburskii @ 2026-07-07 19:47 UTC (permalink / raw)
To: airlied, akhilesh, akpm, corbet, dakr, david, decui, haiyangz,
jgg, kees, kys, leon, liam, lizhi.hou, ljs, longli, lyude,
maarten.lankhorst, mamin506, mhocko, mripard, nouveau, ogabbay,
oleg, rppt, shuah, simona, skhan, skinsburskii, surenb,
tzimmermann, vbabka, wei.liu, skinsburskii
Cc: dri-devel, linux-mm, linux-doc, linux-hyperv, linux-kernel,
linux-kselftest, linux-rdma
In-Reply-To: <178345345668.660027.2952911919681614557.stgit@skinsburskii>
hmm_range_fault() requires the caller to hold the mmap read lock for the
duration of the call. This is incompatible with mappings whose fault
handler may release the mmap lock, notably userfaultfd-managed regions,
where handle_mm_fault() can return VM_FAULT_RETRY or VM_FAULT_COMPLETED
after dropping the lock. Drivers that need to populate device page tables
for such mappings have no way to do so today.
Add hmm_range_fault_unlocked_timeout() for callers that do not need to hold
mmap_lock across any work outside the HMM fault itself. The helper takes
mmap_read_lock_killable() internally, calls the common HMM fault
implementation, and releases the lock before returning if it is still held.
The timeout is specified in jiffies; passing 0 retries indefinitely, while
a non-zero timeout makes the helper return -EBUSY when the retry budget
expires.
When handle_mm_fault() drops mmap_lock, or when the range is invalidated,
hmm_range_fault_unlocked_timeout() refreshes range->notifier_seq and
retries the walk internally. The caller only needs to perform the usual
post-success mmu_interval_read_retry() check while holding its update lock
before consuming the pfns. If mmap_lock acquisition is interrupted or a
fatal signal is pending during retry handling, -EINTR is returned instead.
The common implementation conditionally sets FAULT_FLAG_ALLOW_RETRY and
FAULT_FLAG_KILLABLE only for hmm_range_fault_unlocked_timeout(). The
existing hmm_range_fault() path still passes no locked state, does not
allow handle_mm_fault() to drop mmap_lock, and remains a thin wrapper
preserving the existing API contract for current callers.
The previous refactor that moved page fault handling out of the page-table
walk callbacks is what makes this change small. Faults now run after
walk_page_range() has unwound, with only mmap_lock held, so dropping it
does not interact with the walker's pte spinlock or hugetlb_vma_lock.
Hugetlb regions therefore participate in the unlocked path uniformly with
PTE- and PMD-level mappings; no special case is required.
Documentation/mm/hmm.rst is updated with a description of the new API and
the recommended caller pattern.
Signed-off-by: Stanislav Kinsburskii <skinsburskii@gmail.com>
---
Documentation/mm/hmm.rst | 63 +++++++++++++++++++
include/linux/hmm.h | 2 +
mm/hmm.c | 152 +++++++++++++++++++++++++++++++++++++---------
3 files changed, 188 insertions(+), 29 deletions(-)
diff --git a/Documentation/mm/hmm.rst b/Documentation/mm/hmm.rst
index 7d61b7a8b65b..70885f153d03 100644
--- a/Documentation/mm/hmm.rst
+++ b/Documentation/mm/hmm.rst
@@ -208,6 +208,69 @@ invalidate() callback. That lock must be held before calling
mmu_interval_read_retry() to avoid any race with a concurrent CPU page table
update.
+Dropping the mmap lock during page faults
+=========================================
+
+Some VMAs have fault handlers that need to release the mmap lock while
+servicing a fault (for example, regions managed by ``userfaultfd``).
+``hmm_range_fault()`` cannot be used on such mappings because it must hold the
+mmap lock for the duration of the call. Drivers that need to support them
+should call::
+
+ int hmm_range_fault_unlocked_timeout(struct hmm_range *range,
+ unsigned long timeout);
+
+The caller must not hold ``mmap_read_lock`` before the call.
+``hmm_range_fault_unlocked_timeout()`` takes the mmap read lock internally and
+allows ``handle_mm_fault()`` to drop it during fault handling. If the mmap lock
+is dropped or the range is invalidated, the function refreshes
+``range->notifier_seq`` and restarts the walk internally. ``-EINTR`` is
+returned if mmap lock acquisition is interrupted or a fatal signal is pending
+during retry handling.
+
+The timeout is specified in jiffies; passing ``0`` means retry indefinitely. If
+the timeout expires while the function is retrying after ``-EBUSY``,
+``-EBUSY`` is returned to the caller.
+
+A typical caller looks like this::
+
+ int driver_populate_range_unlocked(...)
+ {
+ struct hmm_range range;
+ unsigned long timeout;
+ ...
+
+ timeout = msecs_to_jiffies(HMM_RANGE_DEFAULT_TIMEOUT);
+ range.notifier = &interval_sub;
+ range.start = ...;
+ range.end = ...;
+ range.hmm_pfns = ...;
+
+ if (!mmget_not_zero(interval_sub.mm))
+ return -EFAULT;
+
+ again:
+ ret = hmm_range_fault_unlocked_timeout(&range, timeout);
+ if (ret)
+ goto out_put;
+
+ take_lock(driver->update);
+ if (mmu_interval_read_retry(&interval_sub, range.notifier_seq)) {
+ release_lock(driver->update);
+ goto again;
+ }
+
+ /* Use pfns array content to update device page table,
+ * under the update lock */
+
+ release_lock(driver->update);
+ ret = 0;
+
+ out_put:
+ mmput(interval_sub.mm);
+ return ret;
+ }
+
Leverage default_flags and pfn_flags_mask
=========================================
diff --git a/include/linux/hmm.h b/include/linux/hmm.h
index db75ffc949a7..6f04e3932f5b 100644
--- a/include/linux/hmm.h
+++ b/include/linux/hmm.h
@@ -123,6 +123,8 @@ struct hmm_range {
* Please see Documentation/mm/hmm.rst for how to use the range API.
*/
int hmm_range_fault(struct hmm_range *range);
+int hmm_range_fault_unlocked_timeout(struct hmm_range *range,
+ unsigned long timeout);
/*
* HMM_RANGE_DEFAULT_TIMEOUT - default timeout (ms) when waiting for a range
diff --git a/mm/hmm.c b/mm/hmm.c
index 2129b1ee4c35..15110409be00 100644
--- a/mm/hmm.c
+++ b/mm/hmm.c
@@ -32,6 +32,7 @@
struct hmm_vma_walk {
struct hmm_range *range;
+ int *locked;
unsigned long last;
unsigned long end;
unsigned int required_fault;
@@ -44,6 +45,14 @@ struct hmm_vma_walk {
*/
#define HMM_FAULT_PENDING -EAGAIN
+/*
+ * Internal sentinel returned by hmm_do_fault() when handle_mm_fault()
+ * completes a page fault with the mmap lock dropped. hmm_do_fault() sets
+ * *locked = 0; the outer loop consumes the sentinel and never propagates it
+ * to the caller.
+ */
+#define HMM_FAULT_UNLOCKED -ENOLCK
+
enum {
HMM_NEED_FAULT = 1 << 0,
HMM_NEED_WRITE_FAULT = 1 << 1,
@@ -73,9 +82,9 @@ static int hmm_pfns_fill(unsigned long addr, unsigned long end,
*
* Called by the walk callbacks when they discover that part of the range
* needs a page fault. The callback records what to fault and returns
- * HMM_FAULT_PENDING; the outer loop in hmm_range_fault() drops back out of
- * walk_page_range() and invokes handle_mm_fault() from a context where no
- * page-table or hugetlb_vma_lock is held.
+ * HMM_FAULT_PENDING; the outer loop in hmm_range_fault_locked() drops
+ * back out of walk_page_range() and invokes handle_mm_fault() from a context
+ * where no page-table or hugetlb_vma_lock is held.
*/
static int hmm_record_fault(unsigned long addr, unsigned long end,
unsigned int required_fault,
@@ -624,7 +633,7 @@ static const struct mm_walk_ops hmm_walk_ops = {
/*
* hmm_do_fault - fault in a range recorded by a walk callback
*
- * Called from the outer loop in hmm_range_fault() after a callback
+ * Called from the outer loop in hmm_range_fault_locked() after a callback
* returned HMM_FAULT_PENDING. At this point we hold only mmap_lock;
* the page-table spinlock and any hugetlb_vma_lock acquired by the walk
* framework have already been released by the unwind.
@@ -641,6 +650,9 @@ static int hmm_do_fault(struct mm_struct *mm,
unsigned int fault_flags = FAULT_FLAG_REMOTE;
struct vm_area_struct *vma;
+ if (hmm_vma_walk->locked)
+ fault_flags |= FAULT_FLAG_ALLOW_RETRY | FAULT_FLAG_KILLABLE;
+
vma = vma_lookup(mm, addr);
if (!vma)
return -EFAULT;
@@ -651,37 +663,33 @@ static int hmm_do_fault(struct mm_struct *mm,
fault_flags |= FAULT_FLAG_WRITE;
}
- for (; addr < end; addr += PAGE_SIZE)
- if (handle_mm_fault(vma, addr, fault_flags, NULL) &
- VM_FAULT_ERROR)
- return -EFAULT;
+ for (; addr < end; addr += PAGE_SIZE) {
+ vm_fault_t ret;
+
+ ret = handle_mm_fault(vma, addr, fault_flags, NULL);
+
+ if (ret & (VM_FAULT_COMPLETED | VM_FAULT_RETRY)) {
+ *hmm_vma_walk->locked = 0;
+ return HMM_FAULT_UNLOCKED;
+ }
+
+ if (ret & VM_FAULT_ERROR) {
+ int err = vm_fault_to_errno(ret, 0);
+
+ if (err)
+ return err;
+ BUG();
+ }
+ }
return -EBUSY;
}
-/**
- * hmm_range_fault - try to fault some address in a virtual address range
- * @range: argument structure
- *
- * Returns 0 on success or one of the following error codes:
- *
- * -EINVAL: Invalid arguments or mm or virtual address is in an invalid vma
- * (e.g., device file vma).
- * -ENOMEM: Out of memory.
- * -EPERM: Invalid permission (e.g., asking for write and range is read
- * only).
- * -EBUSY: The range has been invalidated and the caller needs to wait for
- * the invalidation to finish.
- * -EFAULT: A page was requested to be valid and could not be made valid
- * ie it has no backing VMA or it is illegal to access
- *
- * This is similar to get_user_pages(), except that it can read the page tables
- * without mutating them (ie causing faults).
- */
-int hmm_range_fault(struct hmm_range *range)
+static int hmm_range_fault_locked(struct hmm_range *range, int *locked)
{
struct hmm_vma_walk hmm_vma_walk = {
.range = range,
+ .locked = locked,
.last = range->start,
};
struct mm_struct *mm = range->notifier->mm;
@@ -704,8 +712,14 @@ int hmm_range_fault(struct hmm_range *range)
* returns -EBUSY so the loop re-walks and picks up the
* now-present entries.
*/
- if (ret == HMM_FAULT_PENDING)
+ if (ret == HMM_FAULT_PENDING) {
ret = hmm_do_fault(mm, &hmm_vma_walk);
+ if (ret == HMM_FAULT_UNLOCKED) {
+ if (fatal_signal_pending(current))
+ return -EINTR;
+ return -EBUSY;
+ }
+ }
/*
* When -EBUSY is returned the loop restarts with
* hmm_vma_walk.last set to an address that has not been stored
@@ -715,8 +729,88 @@ int hmm_range_fault(struct hmm_range *range)
} while (ret == -EBUSY);
return ret;
}
+
+/**
+ * hmm_range_fault - try to fault some address in a virtual address range
+ * @range: argument structure
+ *
+ * Returns 0 on success or one of the following error codes:
+ *
+ * -EINVAL: Invalid arguments or mm or virtual address is in an invalid vma
+ * (e.g., device file vma).
+ * -ENOMEM: Out of memory.
+ * -EPERM: Invalid permission (e.g., asking for write and range is read
+ * only).
+ * -EBUSY: The range has been invalidated and the caller needs to wait for
+ * the invalidation to finish.
+ * -EFAULT: A page was requested to be valid and could not be made valid
+ * ie it has no backing VMA or it is illegal to access
+ *
+ * This is similar to get_user_pages(), except that it can read the page tables
+ * without mutating them (ie causing faults).
+ *
+ * The mmap lock must be held by the caller and will remain held on return.
+ * For a variant that allows the mmap lock to be dropped during faults (e.g.,
+ * for userfaultfd support), see hmm_range_fault_unlocked_timeout().
+ */
+int hmm_range_fault(struct hmm_range *range)
+{
+ return hmm_range_fault_locked(range, NULL);
+}
EXPORT_SYMBOL(hmm_range_fault);
+/**
+ * hmm_range_fault_unlocked_timeout - fault in a range with a retry timeout
+ * @range: argument structure
+ * @timeout: timeout in jiffies for internal -EBUSY retries, or 0 to retry
+ * indefinitely
+ *
+ * This is similar to hmm_range_fault(), except the caller must not hold the
+ * mmap lock. The function takes the mmap read lock internally and allows
+ * handle_mm_fault() to drop it during faults. If the mmap lock is dropped or
+ * the range is invalidated, the function refreshes range->notifier_seq and
+ * restarts the walk internally. Passing 0 for @timeout retries indefinitely;
+ * otherwise, if @timeout expires while retrying -EBUSY, -EBUSY is returned to
+ * the caller.
+ *
+ * Returns 0 on success or one of the error codes documented for
+ * hmm_range_fault(). -EINTR is returned if mmap_lock acquisition is
+ * interrupted or a fatal signal is pending during retry handling.
+ */
+int hmm_range_fault_unlocked_timeout(struct hmm_range *range,
+ unsigned long timeout)
+{
+ struct mm_struct *mm = range->notifier->mm;
+ unsigned long deadline = 0;
+ int locked, ret;
+
+ if (timeout)
+ deadline = jiffies + timeout;
+
+ do {
+ if (fatal_signal_pending(current))
+ return -EINTR;
+
+ if (timeout && time_after(jiffies, deadline))
+ return -EBUSY;
+
+ range->notifier_seq =
+ mmu_interval_read_begin(range->notifier);
+
+ ret = mmap_read_lock_killable(mm);
+ if (ret)
+ return ret;
+
+ locked = 1;
+ ret = hmm_range_fault_locked(range, &locked);
+ if (locked)
+ mmap_read_unlock(mm);
+ } while (ret == -EBUSY);
+
+ return ret;
+}
+EXPORT_SYMBOL(hmm_range_fault_unlocked_timeout);
+
/**
* hmm_dma_map_alloc - Allocate HMM map structure
* @dev: device to allocate structure for
^ permalink raw reply related
* [PATCH v7 1/8] mm/hmm: move page fault handling out of walk callbacks
From: Stanislav Kinsburskii @ 2026-07-07 19:46 UTC (permalink / raw)
To: airlied, akhilesh, akpm, corbet, dakr, david, decui, haiyangz,
jgg, kees, kys, leon, liam, lizhi.hou, ljs, longli, lyude,
maarten.lankhorst, mamin506, mhocko, mripard, nouveau, ogabbay,
oleg, rppt, shuah, simona, skhan, skinsburskii, surenb,
tzimmermann, vbabka, wei.liu, skinsburskii
Cc: dri-devel, linux-mm, linux-doc, linux-hyperv, linux-kernel,
linux-kselftest, linux-rdma
In-Reply-To: <178345345668.660027.2952911919681614557.stgit@skinsburskii>
hmm_range_fault() currently triggers page faults from inside the page-table
walk callbacks: hmm_vma_walk_pmd(), hmm_vma_walk_pud(),
hmm_vma_walk_hugetlb_entry() and the pte-level helper all call
hmm_vma_fault(), which in turn calls handle_mm_fault() while the walker
still holds nested locks. The pte spinlock is dropped explicitly by each
caller, and the hugetlb path manually drops and retakes
hugetlb_vma_lock_read around the fault to dodge a deadlock against the walk
framework's unconditional unlock.
This layering does not extend cleanly to fault handlers that may release
mmap_lock (VM_FAULT_RETRY, VM_FAULT_COMPLETED). If the lock is dropped
while walk_page_range() is mid-traversal, the VMA can be freed before the
walk framework's matching hugetlb_vma_unlock_read(), turning that unlock
into a use-after-free.
Split the responsibilities the way get_user_pages() does. Walk callbacks
become inspect-only: when they detect a range that needs to be faulted in,
they record it in struct hmm_vma_walk and return a private sentinel
(HMM_FAULT_PENDING). The outer loop in hmm_range_fault() then drops out of
walk_page_range(), invokes a new helper hmm_do_fault() that calls
handle_mm_fault() with only mmap_lock held, and restarts the walk so the
now-present entries are collected into hmm_pfns.
No functional change for existing callers. As a side effect the hugetlb
callback no longer needs the hugetlb_vma_{un}lock_read dance, and every
fault-path exit from the callbacks now releases the pte spinlock on a
single, common path. This refactor is also a precursor for adding an
unlockable variant of hmm_range_fault() in a follow-up patch.
Signed-off-by: Stanislav Kinsburskii <skinsburskii@gmail.com>
---
mm/hmm.c | 118 +++++++++++++++++++++++++++++++++++++++-----------------------
1 file changed, 75 insertions(+), 43 deletions(-)
diff --git a/mm/hmm.c b/mm/hmm.c
index 4f3f627d2b47..2129b1ee4c35 100644
--- a/mm/hmm.c
+++ b/mm/hmm.c
@@ -33,8 +33,17 @@
struct hmm_vma_walk {
struct hmm_range *range;
unsigned long last;
+ unsigned long end;
+ unsigned int required_fault;
};
+/*
+ * Internal sentinel returned by walk callbacks when they need a page fault.
+ * The callback stores end/required_fault in hmm_vma_walk; the outer loop
+ * consumes the sentinel and never propagates it to the caller.
+ */
+#define HMM_FAULT_PENDING -EAGAIN
+
enum {
HMM_NEED_FAULT = 1 << 0,
HMM_NEED_WRITE_FAULT = 1 << 1,
@@ -60,37 +69,25 @@ static int hmm_pfns_fill(unsigned long addr, unsigned long end,
}
/*
- * hmm_vma_fault() - fault in a range lacking valid pmd or pte(s)
- * @addr: range virtual start address (inclusive)
- * @end: range virtual end address (exclusive)
- * @required_fault: HMM_NEED_* flags
- * @walk: mm_walk structure
- * Return: -EBUSY after page fault, or page fault error
+ * hmm_record_fault() - record a range that needs to be faulted in
*
- * This function will be called whenever pmd_none() or pte_none() returns true,
- * or whenever there is no page directory covering the virtual address range.
+ * Called by the walk callbacks when they discover that part of the range
+ * needs a page fault. The callback records what to fault and returns
+ * HMM_FAULT_PENDING; the outer loop in hmm_range_fault() drops back out of
+ * walk_page_range() and invokes handle_mm_fault() from a context where no
+ * page-table or hugetlb_vma_lock is held.
*/
-static int hmm_vma_fault(unsigned long addr, unsigned long end,
- unsigned int required_fault, struct mm_walk *walk)
+static int hmm_record_fault(unsigned long addr, unsigned long end,
+ unsigned int required_fault,
+ struct mm_walk *walk)
{
struct hmm_vma_walk *hmm_vma_walk = walk->private;
- struct vm_area_struct *vma = walk->vma;
- unsigned int fault_flags = FAULT_FLAG_REMOTE;
WARN_ON_ONCE(!required_fault);
hmm_vma_walk->last = addr;
-
- if (required_fault & HMM_NEED_WRITE_FAULT) {
- if (!(vma->vm_flags & VM_WRITE))
- return -EPERM;
- fault_flags |= FAULT_FLAG_WRITE;
- }
-
- for (; addr < end; addr += PAGE_SIZE)
- if (handle_mm_fault(vma, addr, fault_flags, NULL) &
- VM_FAULT_ERROR)
- return -EFAULT;
- return -EBUSY;
+ hmm_vma_walk->end = end;
+ hmm_vma_walk->required_fault = required_fault;
+ return HMM_FAULT_PENDING;
}
static unsigned int hmm_pte_need_fault(const struct hmm_vma_walk *hmm_vma_walk,
@@ -174,7 +171,7 @@ static int hmm_vma_walk_hole(unsigned long addr, unsigned long end,
return hmm_pfns_fill(addr, end, range, HMM_PFN_ERROR);
}
if (required_fault)
- return hmm_vma_fault(addr, end, required_fault, walk);
+ return hmm_record_fault(addr, end, required_fault, walk);
return hmm_pfns_fill(addr, end, range, 0);
}
@@ -209,7 +206,7 @@ static int hmm_vma_handle_pmd(struct mm_walk *walk, unsigned long addr,
required_fault =
hmm_range_need_fault(hmm_vma_walk, hmm_pfns, npages, cpu_flags);
if (required_fault)
- return hmm_vma_fault(addr, end, required_fault, walk);
+ return hmm_record_fault(addr, end, required_fault, walk);
pfn = pmd_pfn(pmd) + ((addr & ~PMD_MASK) >> PAGE_SHIFT);
for (i = 0; addr < end; addr += PAGE_SIZE, i++, pfn++) {
@@ -328,7 +325,7 @@ static int hmm_vma_handle_pte(struct mm_walk *walk, unsigned long addr,
fault:
pte_unmap(ptep);
/* Fault any virtual address we were asked to fault */
- return hmm_vma_fault(addr, end, required_fault, walk);
+ return hmm_record_fault(addr, end, required_fault, walk);
}
#ifdef CONFIG_ARCH_SUPPORTS_PMD_SOFTLEAF
@@ -371,7 +368,7 @@ static int hmm_vma_handle_absent_pmd(struct mm_walk *walk, unsigned long start,
npages, 0);
if (required_fault) {
if (softleaf_is_device_private(entry))
- return hmm_vma_fault(addr, end, required_fault, walk);
+ return hmm_record_fault(addr, end, required_fault, walk);
else
return -EFAULT;
}
@@ -517,7 +514,7 @@ static int hmm_vma_walk_pud(pud_t *pudp, unsigned long start, unsigned long end,
npages, cpu_flags);
if (required_fault) {
spin_unlock(ptl);
- return hmm_vma_fault(addr, end, required_fault, walk);
+ return hmm_record_fault(addr, end, required_fault, walk);
}
pfn = pud_pfn(pud) + ((addr & ~PUD_MASK) >> PAGE_SHIFT);
@@ -564,21 +561,8 @@ static int hmm_vma_walk_hugetlb_entry(pte_t *pte, unsigned long hmask,
required_fault =
hmm_pte_need_fault(hmm_vma_walk, pfn_req_flags, cpu_flags);
if (required_fault) {
- int ret;
-
spin_unlock(ptl);
- hugetlb_vma_unlock_read(vma);
- /*
- * Avoid deadlock: drop the vma lock before calling
- * hmm_vma_fault(), which will itself potentially take and
- * drop the vma lock. This is also correct from a
- * protection point of view, because there is no further
- * use here of either pte or ptl after dropping the vma
- * lock.
- */
- ret = hmm_vma_fault(addr, end, required_fault, walk);
- hugetlb_vma_lock_read(vma);
- return ret;
+ return hmm_record_fault(addr, end, required_fault, walk);
}
pfn = pte_pfn(entry) + ((start & ~hmask) >> PAGE_SHIFT);
@@ -637,6 +621,44 @@ static const struct mm_walk_ops hmm_walk_ops = {
.walk_lock = PGWALK_RDLOCK,
};
+/*
+ * hmm_do_fault - fault in a range recorded by a walk callback
+ *
+ * Called from the outer loop in hmm_range_fault() after a callback
+ * returned HMM_FAULT_PENDING. At this point we hold only mmap_lock;
+ * the page-table spinlock and any hugetlb_vma_lock acquired by the walk
+ * framework have already been released by the unwind.
+ *
+ * Returns -EBUSY on success (all pages faulted, caller should re-walk).
+ * Returns a negative errno on failure.
+ */
+static int hmm_do_fault(struct mm_struct *mm,
+ struct hmm_vma_walk *hmm_vma_walk)
+{
+ unsigned long addr = hmm_vma_walk->last;
+ unsigned long end = hmm_vma_walk->end;
+ unsigned int required_fault = hmm_vma_walk->required_fault;
+ unsigned int fault_flags = FAULT_FLAG_REMOTE;
+ struct vm_area_struct *vma;
+
+ vma = vma_lookup(mm, addr);
+ if (!vma)
+ return -EFAULT;
+
+ if (required_fault & HMM_NEED_WRITE_FAULT) {
+ if (!(vma->vm_flags & VM_WRITE))
+ return -EPERM;
+ fault_flags |= FAULT_FLAG_WRITE;
+ }
+
+ for (; addr < end; addr += PAGE_SIZE)
+ if (handle_mm_fault(vma, addr, fault_flags, NULL) &
+ VM_FAULT_ERROR)
+ return -EFAULT;
+
+ return -EBUSY;
+}
+
/**
* hmm_range_fault - try to fault some address in a virtual address range
* @range: argument structure
@@ -674,6 +696,16 @@ int hmm_range_fault(struct hmm_range *range)
return -EBUSY;
ret = walk_page_range(mm, hmm_vma_walk.last, range->end,
&hmm_walk_ops, &hmm_vma_walk);
+ /*
+ * When HMM_FAULT_PENDING is returned a walk callback
+ * recorded a range that needs handle_mm_fault();
+ * hmm_do_fault() runs the fault outside walk_page_range()
+ * (so no page-table or hugetlb_vma_lock is held) and
+ * returns -EBUSY so the loop re-walks and picks up the
+ * now-present entries.
+ */
+ if (ret == HMM_FAULT_PENDING)
+ ret = hmm_do_fault(mm, &hmm_vma_walk);
/*
* When -EBUSY is returned the loop restarts with
* hmm_vma_walk.last set to an address that has not been stored
^ permalink raw reply related
* [PATCH v7 0/8] mm/hmm: Add mmap lock-drop support for userfaultfd-backed mappings
From: Stanislav Kinsburskii @ 2026-07-07 19:46 UTC (permalink / raw)
To: airlied, akhilesh, akpm, corbet, dakr, david, decui, haiyangz,
jgg, kees, kys, leon, liam, lizhi.hou, ljs, longli, lyude,
maarten.lankhorst, mamin506, mhocko, mripard, nouveau, ogabbay,
oleg, rppt, shuah, simona, skhan, skinsburskii, surenb,
tzimmermann, vbabka, wei.liu, skinsburskii
Cc: dri-devel, linux-mm, linux-doc, linux-hyperv, linux-kernel,
linux-kselftest, linux-rdma
This series extends the HMM framework to support userfaultfd-backed memory
by allowing the mmap read lock to be dropped during hmm_range_fault().
Some page fault handlers — most notably userfaultfd — require the mmap lock
to be released so that userspace can resolve the fault. The current HMM
interface never sets FAULT_FLAG_ALLOW_RETRY, making it impossible to fault
in pages from userfaultfd-registered regions.
This series follows the established int *locked pattern from
get_user_pages_remote() in mm/gup.c. A new helper function,
hmm_range_fault_locked(), accepts an int *locked parameter. When the
mmap lock is dropped during fault resolution (VM_FAULT_RETRY or
VM_FAULT_COMPLETED), the function returns 0 with *locked = 0, signalling
the caller to restart its walk. The existing hmm_range_fault() is
refactored into a thin wrapper that passes NULL, preserving current
behavior for all existing callers.
Possible approaches to lift this limitation are documented in
Documentation/mm/hmm.rst.
Changes in v7:
- Replaced the unlocked HMM API with
hmm_range_fault_unlocked_timeout(). The helper now takes a timeout in
jiffies, with 0 meaning retry indefinitely.
- Moved -EBUSY retry handling into the HMM helper for the unlocked path.
The helper refreshes range->notifier_seq internally before each retry.
- Switched the unlocked path to mmap_read_lock_killable() and return
-EINTR if mmap lock acquisition is interrupted or a fatal signal is
pending during retry handling.
- Removed the redundant non-timeout hmm_range_fault_unlocked() interface.
- Updated Documentation/mm/hmm.rst and kernel-doc to describe the timeout API
and the intended caller pattern.
- Updated the HMM selftests to use hmm_range_fault_unlocked_timeout()
only, including coverage for the finite-timeout path.
- Added in-tree users of the new helper:
- mshv
- nouveau
- RDMA/umem
- amdxdna
- drm/gpusvm
- Preserved each converted driver’s existing timeout convention:
- unbounded retry where the old code retried indefinitely,
- HMM_RANGE_DEFAULT_TIMEOUT where the old code used that budget,
- existing driver-specific timeout return values such as -ETIME.
- Used max_t(long, timeout - jiffies, 1) when passing remaining time from
absolute jiffies deadlines to avoid unsigned underflow while keeping a
minimum one-jiffy retry window.
- Left callers on hmm_range_fault() when they already need to hold
mmap_lock across surrounding work, such as drm_gpusvm_check_pages().
Changes in v6:
- Reworked the new API from the external int *locked pattern to
hmm_range_fault_unlocked(), which owns mmap_read_lock() internally.
- Changed the dropped-lock contract: hmm_range_fault_unlocked() now returns
-EBUSY when the mmap lock is dropped, and callers restart with a fresh
mmu_interval_read_begin() sequence.
- Kept hmm_range_fault() as the locked variant for existing users, preserving
its caller-held mmap lock contract.
- Added an in-tree user by converting the MSHV region fault path to
hmm_range_fault_unlocked().
- Updated Documentation/mm/hmm.rst and kernel-doc to describe the unlocked
helper and retry pattern.
- Updated commit messages to match the new API and return semantics.
- Kept the userfaultfd HMM selftest using the test_hmm unlocked read ioctl
path.
Changes in v5:
- Rework hmm_range_fault_unlockable() retry handling to retry
VM_FAULT_RETRY internally with FAULT_FLAG_TRIED set, matching the
fixup_user_fault() pattern and avoiding repeated first-retry lock drops.
- Distinguish VM_FAULT_RETRY from VM_FAULT_COMPLETED: retry faults now
reacquire the mmap lock internally, while completed faults return to the
caller with *locked = 0 so the caller can restart with a fresh notifier
sequence.
- Document the two *locked return states, including the -EINTR case when a
fatal signal is pending after the mmap lock has already been dropped.
- Update comments around HMM_FAULT_UNLOCKED and the HMM fault loop to match
the current hmm_range_fault_unlockable() implementation.
Changes in v4:
- Rebased on 7.2-rc1
Changes in v3:
- Return -EFAULT from dmirror_fault_unlockable() when the mirrored mm can
no longer be pinned.
- Add an eventfd stop signal for the userfaultfd handler thread to avoid
waiting for the poll timeout on successful test completion.
Changes in v2:
- Split into a preparatory refactor (new patch 1) that moves
handle_mm_fault() out of the walk callbacks, plus a smaller feature
patch on top. Suggested by David Hildenbrand.
- Hugetlb regions are now supported on the unlockable path; the v1
-EFAULT short-circuit and the hugetlb_vma_lock_read drop/retake
dance are gone.
- Distinct internal sentinels for "needs fault" (HMM_FAULT_PENDING)
and "lock dropped" (HMM_FAULT_UNLOCKED).
- Outer loop now re-walks after a successful internal fault so the
faulted pfns end up in range->hmm_pfns.
- Kernel-doc on hmm_range_fault_unlockable() and the
Documentation/mm/hmm.rst example match the implementation.
- Dropped the mshv driver conversion (v1 patch 2); will post
separately.
- Selftest converted to drive the path through test_hmm with a
userfaultfd handler (new HMM_DMIRROR_READ_UNLOCKABLE ioctl).
---
Stanislav Kinsburskii (8):
mm/hmm: move page fault handling out of walk callbacks
mm/hmm: add hmm_range_fault_unlocked_timeout() for mmap lock-drop support
selftests/mm: add HMM tests for mmap lock-dropping faults
mshv: Use hmm_range_fault_unlocked_timeout() for region faults
drm/nouveau: Use hmm_range_fault_unlocked_timeout() for SVM faults
RDMA/umem: Use hmm_range_fault_unlocked_timeout() for ODP faults
accel/amdxdna: Use hmm_range_fault_unlocked_timeout() for range population
drm/gpusvm: Use hmm_range_fault_unlocked_timeout() for range faults
Documentation/mm/hmm.rst | 63 ++++++++
drivers/accel/amdxdna/aie2_ctx.c | 17 --
drivers/gpu/drm/drm_gpusvm.c | 52 +------
drivers/gpu/drm/nouveau/nouveau_svm.c | 11 -
drivers/hv/mshv_regions.c | 54 +------
drivers/infiniband/core/umem_odp.c | 18 +-
include/linux/hmm.h | 2
lib/test_hmm.c | 111 ++++++++++++++
lib/test_hmm_uapi.h | 3
mm/hmm.c | 246 ++++++++++++++++++++++++--------
tools/testing/selftests/mm/hmm-tests.c | 220 +++++++++++++++++++++++++++++
11 files changed, 613 insertions(+), 184 deletions(-)
--
Signature
^ permalink raw reply
* Re: [PATCH v2 3/4] iommu/hyperv: Add para-virtualized IOMMU support for Hyper-V guest
From: Jacob Pan @ 2026-07-07 14:48 UTC (permalink / raw)
To: Mukesh R
Cc: Yu Zhang, linux-kernel, linux-hyperv, iommu, linux-pci,
linux-arch, wei.liu, kys, haiyangz, decui, longli, joro, will,
robin.murphy, bhelgaas, kwilczynski, lpieralisi, mani, robh, arnd,
jgg, mhklinux, tgopinath, easwar.hariharan, jacob.pan
In-Reply-To: <1f57d1d9-ccbb-ff4e-9680-d45adc7821ea@linux.microsoft.com>
Hi Mukesh,
On Mon, 6 Jul 2026 17:08:50 -0700
Mukesh R <mrathor@linux.microsoft.com> wrote:
> From: Mukesh R <mrathor@linux.microsoft.com>
> To: Yu Zhang <zhangyu1@linux.microsoft.com>,
> linux-kernel@vger.kernel.org, linux-hyperv@vger.kernel.org,
> iommu@lists.linux.dev, linux-pci@vger.kernel.org,
> linux-arch@vger.kernel.org Cc: wei.liu@kernel.org, kys@microsoft.com,
> haiyangz@microsoft.com, decui@microsoft.com, longli@microsoft.com,
> joro@8bytes.org, will@kernel.org, robin.murphy@arm.com,
> bhelgaas@google.com, kwilczynski@kernel.org, lpieralisi@kernel.org,
> mani@kernel.org, robh@kernel.org, arnd@arndb.de, jgg@ziepe.ca,
> mhklinux@outlook.com, jacob.pan@linux.microsoft.com,
> tgopinath@linux.microsoft.com, easwar.hariharan@linux.microsoft.com
> Subject: Re: [PATCH v2 3/4] iommu/hyperv: Add para-virtualized IOMMU
> support for Hyper-V guest Date: Mon, 6 Jul 2026 17:08:50 -0700
> User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:91.0) Gecko/20100101
> Thunderbird/91.13.1
>
> On 7/2/26 09:05, Yu Zhang wrote:
> > Add a para-virtualized IOMMU driver for Linux guests running on
> > Hyper-V. This driver implements stage-1 IO translation within the
> > guest OS. It integrates with the Linux IOMMU core, utilizing
> > Hyper-V hypercalls for:
> > - Capability discovery
> > - Domain allocation, configuration, and deallocation
> > - Device attachment and detachment
> > - IOTLB invalidation
> >
> > The driver constructs x86-compatible stage-1 IO page tables in the
> > guest memory using consolidated IO page table helpers. This allows
> > the guest to manage stage-1 translations independently of vendor-
> > specific drivers (like Intel VT-d or AMD IOMMU).
> >
> > Hyper-V consumes this stage-1 IO page table when a device domain is
> > created and configured, and nests it with the host's stage-2 IO page
> > tables, therefore eliminating the VM exits for guest IOMMU mapping
> > operations. For unmapping operations, VM exits to perform the IOTLB
> > flush are still unavoidable.
> >
> > To identify a device in its hypercall interface, the driver looks
> > up the logical device ID prefix registered for the device's PCI
> > domain (see the logical device ID registry in hv_common.c) and
> > combines it with the PCI function number of the endpoint device.
> >
> > Co-developed-by: Wei Liu <wei.liu@kernel.org>
> > Signed-off-by: Wei Liu <wei.liu@kernel.org>
> > Co-developed-by: Easwar Hariharan
> > <easwar.hariharan@linux.microsoft.com> Signed-off-by: Easwar
> > Hariharan <easwar.hariharan@linux.microsoft.com> Signed-off-by: Yu
> > Zhang <zhangyu1@linux.microsoft.com> ---
> > arch/x86/hyperv/hv_init.c | 4 +
> > arch/x86/include/asm/mshyperv.h | 4 +
> > drivers/iommu/Kconfig | 1 +
> > drivers/iommu/hyperv/Kconfig | 16 +
> > drivers/iommu/hyperv/Makefile | 1 +
> > drivers/iommu/hyperv/iommu.c | 620
> > ++++++++++++++++++++++++++++++++ drivers/iommu/hyperv/iommu.h |
> > 51 +++ 7 files changed, 697 insertions(+)
> > create mode 100644 drivers/iommu/hyperv/Kconfig
> > create mode 100644 drivers/iommu/hyperv/iommu.c
>
> Hey Jacob,
>
> You had suggested I rename iommu.c to iommu-root.c (I called it
> hv-iommu-root.c eventually), so this needs to be renamed also,
> right?
yes, I agree. I feel it is clearer to name it hv-iommu-guest.c since
this is a guest only driver.
^ permalink raw reply
* Re: [PATCH v2 3/4] iommu/hyperv: Add para-virtualized IOMMU support for Hyper-V guest
From: Mukesh R @ 2026-07-07 0:08 UTC (permalink / raw)
To: Yu Zhang, linux-kernel, linux-hyperv, iommu, linux-pci,
linux-arch
Cc: wei.liu, kys, haiyangz, decui, longli, joro, will, robin.murphy,
bhelgaas, kwilczynski, lpieralisi, mani, robh, arnd, jgg,
mhklinux, jacob.pan, tgopinath, easwar.hariharan
In-Reply-To: <20260702160518.311234-4-zhangyu1@linux.microsoft.com>
On 7/2/26 09:05, Yu Zhang wrote:
> Add a para-virtualized IOMMU driver for Linux guests running on Hyper-V.
> This driver implements stage-1 IO translation within the guest OS.
> It integrates with the Linux IOMMU core, utilizing Hyper-V hypercalls
> for:
> - Capability discovery
> - Domain allocation, configuration, and deallocation
> - Device attachment and detachment
> - IOTLB invalidation
>
> The driver constructs x86-compatible stage-1 IO page tables in the
> guest memory using consolidated IO page table helpers. This allows
> the guest to manage stage-1 translations independently of vendor-
> specific drivers (like Intel VT-d or AMD IOMMU).
>
> Hyper-V consumes this stage-1 IO page table when a device domain is
> created and configured, and nests it with the host's stage-2 IO page
> tables, therefore eliminating the VM exits for guest IOMMU mapping
> operations. For unmapping operations, VM exits to perform the IOTLB
> flush are still unavoidable.
>
> To identify a device in its hypercall interface, the driver looks up the
> logical device ID prefix registered for the device's PCI domain (see the
> logical device ID registry in hv_common.c) and combines it with the PCI
> function number of the endpoint device.
>
> Co-developed-by: Wei Liu <wei.liu@kernel.org>
> Signed-off-by: Wei Liu <wei.liu@kernel.org>
> Co-developed-by: Easwar Hariharan <easwar.hariharan@linux.microsoft.com>
> Signed-off-by: Easwar Hariharan <easwar.hariharan@linux.microsoft.com>
> Signed-off-by: Yu Zhang <zhangyu1@linux.microsoft.com>
> ---
> arch/x86/hyperv/hv_init.c | 4 +
> arch/x86/include/asm/mshyperv.h | 4 +
> drivers/iommu/Kconfig | 1 +
> drivers/iommu/hyperv/Kconfig | 16 +
> drivers/iommu/hyperv/Makefile | 1 +
> drivers/iommu/hyperv/iommu.c | 620 ++++++++++++++++++++++++++++++++
> drivers/iommu/hyperv/iommu.h | 51 +++
> 7 files changed, 697 insertions(+)
> create mode 100644 drivers/iommu/hyperv/Kconfig
> create mode 100644 drivers/iommu/hyperv/iommu.c
Hey Jacob,
You had suggested I rename iommu.c to iommu-root.c (I called it
hv-iommu-root.c eventually), so this needs to be renamed also,
right?
Thanks,
-Mukesh
> create mode 100644 drivers/iommu/hyperv/iommu.h
>
> diff --git a/arch/x86/hyperv/hv_init.c b/arch/x86/hyperv/hv_init.c
> index 55a8b6de2865..094f9f7ddb72 100644
> --- a/arch/x86/hyperv/hv_init.c
> +++ b/arch/x86/hyperv/hv_init.c
> @@ -578,6 +578,10 @@ void __init hyperv_init(void)
> old_setup_percpu_clockev = x86_init.timers.setup_percpu_clockev;
> x86_init.timers.setup_percpu_clockev = hv_stimer_setup_percpu_clockev;
>
> +#ifdef CONFIG_HYPERV_PVIOMMU
> + x86_init.iommu.iommu_init = hv_iommu_init;
> +#endif
> +
> hv_apic_init();
>
> x86_init.pci.arch_init = hv_pci_init;
> diff --git a/arch/x86/include/asm/mshyperv.h b/arch/x86/include/asm/mshyperv.h
> index f64393e853ee..20d947c2c758 100644
> --- a/arch/x86/include/asm/mshyperv.h
> +++ b/arch/x86/include/asm/mshyperv.h
> @@ -313,6 +313,10 @@ static inline void mshv_vtl_return_hypercall(void) {}
> static inline void __mshv_vtl_return_call(struct mshv_vtl_cpu_context *vtl0) {}
> #endif
>
> +#ifdef CONFIG_HYPERV_PVIOMMU
> +int __init hv_iommu_init(void);
> +#endif
> +
> #include <asm-generic/mshyperv.h>
>
> #endif
> diff --git a/drivers/iommu/Kconfig b/drivers/iommu/Kconfig
> index 6e07bd69467a..0d128f377929 100644
> --- a/drivers/iommu/Kconfig
> +++ b/drivers/iommu/Kconfig
> @@ -195,6 +195,7 @@ config MSM_IOMMU
> source "drivers/iommu/amd/Kconfig"
> source "drivers/iommu/arm/Kconfig"
> source "drivers/iommu/intel/Kconfig"
> +source "drivers/iommu/hyperv/Kconfig"
> source "drivers/iommu/iommufd/Kconfig"
> source "drivers/iommu/riscv/Kconfig"
>
> diff --git a/drivers/iommu/hyperv/Kconfig b/drivers/iommu/hyperv/Kconfig
> new file mode 100644
> index 000000000000..8b6abbaaf9b8
> --- /dev/null
> +++ b/drivers/iommu/hyperv/Kconfig
> @@ -0,0 +1,16 @@
> +# SPDX-License-Identifier: GPL-2.0-only
> +# HyperV paravirtualized IOMMU support
> +config HYPERV_PVIOMMU
> + bool "Microsoft Hypervisor para-virtualized IOMMU support"
> + depends on X86_64 && HYPERV
> + select IOMMU_API
> + select GENERIC_PT
> + select IOMMU_PT
> + select IOMMU_PT_X86_64
> + select IOMMU_IOVA
> + default HYPERV
> + help
> + Para-virtualized IOMMU driver for Linux guests running on
> + Microsoft Hyper-V. Provides DMA remapping and IOTLB
> + flush support to enable DMA isolation for devices
> + assigned to the guest.
> diff --git a/drivers/iommu/hyperv/Makefile b/drivers/iommu/hyperv/Makefile
> index 6ef0ef97f3dd..fefb409d976b 100644
> --- a/drivers/iommu/hyperv/Makefile
> +++ b/drivers/iommu/hyperv/Makefile
> @@ -1,2 +1,3 @@
> # SPDX-License-Identifier: GPL-2.0
> obj-$(CONFIG_IRQ_REMAP) += hv-irq-remap-x86.o
> +obj-$(CONFIG_HYPERV_PVIOMMU) += iommu.o
> diff --git a/drivers/iommu/hyperv/iommu.c b/drivers/iommu/hyperv/iommu.c
> new file mode 100644
> index 000000000000..254136946404
> --- /dev/null
> +++ b/drivers/iommu/hyperv/iommu.c
> @@ -0,0 +1,620 @@
> +// SPDX-License-Identifier: GPL-2.0
> +
> +/*
> + * Hyper-V IOMMU driver.
> + *
> + * Copyright (C) 2019, 2024-2026 Microsoft, Inc.
> + */
> +
> +#define pr_fmt(fmt) "Hyper-V pvIOMMU: " fmt
> +#define dev_fmt(fmt) pr_fmt(fmt)
> +
> +#include <linux/iommu.h>
> +#include <linux/pci.h>
> +#include <linux/dma-map-ops.h>
> +#include <linux/generic_pt/iommu.h>
> +#include <linux/pci-ats.h>
> +
> +#include <asm/iommu.h>
> +#include <asm/hypervisor.h>
> +#include <asm/mshyperv.h>
> +
> +#include "iommu.h"
> +#include "../iommu-pages.h"
> +
> +struct hv_iommu_dev *hv_iommu_device;
> +
> +/*
> + * Identity and blocking domains are static singletons: identity is a 1:1
> + * passthrough with no page table, blocking rejects all DMA. Neither holds
> + * per-IOMMU state, so one instance suffices even with multiple vIOMMUs.
> + */
> +static const struct iommu_domain_ops hv_iommu_identity_domain_ops;
> +static const struct iommu_domain_ops hv_iommu_blocking_domain_ops;
> +static struct iommu_ops hv_iommu_ops;
> +
> +static struct hv_iommu_domain hv_identity_domain = {
> + .domain = {
> + .type = IOMMU_DOMAIN_IDENTITY,
> + .ops = &hv_iommu_identity_domain_ops,
> + .owner = &hv_iommu_ops,
> + },
> +};
> +static struct hv_iommu_domain hv_blocking_domain = {
> + .domain = {
> + .type = IOMMU_DOMAIN_BLOCKED,
> + .ops = &hv_iommu_blocking_domain_ops,
> + .owner = &hv_iommu_ops,
> + },
> +};
> +
> +static inline bool hv_iommu_present(u64 cap)
> +{
> + return cap & HV_IOMMU_CAP_PRESENT;
> +}
> +
> +static inline bool hv_iommu_s1_domain_supported(u64 cap)
> +{
> + return cap & HV_IOMMU_CAP_S1;
> +}
> +
> +static inline bool hv_iommu_5lvl_supported(u64 cap)
> +{
> + return cap & HV_IOMMU_CAP_S1_5LVL;
> +}
> +
> +static inline bool hv_iommu_ats_supported(u64 cap)
> +{
> + return cap & HV_IOMMU_CAP_ATS;
> +}
> +
> +static int hv_create_device_domain(struct hv_iommu_domain *hv_domain, u32 domain_stage)
> +{
> + int ret;
> + u64 status;
> + unsigned long flags;
> + struct hv_input_create_device_domain *input;
> +
> + ret = ida_alloc_range(&hv_iommu_device->domain_ids,
> + hv_iommu_device->first_domain, hv_iommu_device->last_domain,
> + GFP_KERNEL);
> + if (ret < 0)
> + return ret;
> +
> + hv_domain->device_domain.partition_id = HV_PARTITION_ID_SELF;
> + hv_domain->device_domain.domain_id.type = domain_stage;
> + hv_domain->device_domain.domain_id.id = ret;
> + hv_domain->hv_iommu = hv_iommu_device;
> +
> + local_irq_save(flags);
> +
> + input = *this_cpu_ptr(hyperv_pcpu_input_arg);
> + memset(input, 0, sizeof(*input));
> + input->device_domain = hv_domain->device_domain;
> + input->create_device_domain_flags.forward_progress_required = 1;
> + input->create_device_domain_flags.inherit_owning_vtl = 0;
> + status = hv_do_hypercall(HVCALL_CREATE_DEVICE_DOMAIN, input, NULL);
> +
> + local_irq_restore(flags);
> +
> + if (!hv_result_success(status)) {
> + pr_err("HVCALL_CREATE_DEVICE_DOMAIN failed, status %lld\n", status);
> + ida_free(&hv_iommu_device->domain_ids, hv_domain->device_domain.domain_id.id);
> + }
> +
> + return hv_result_to_errno(status);
> +}
> +
> +static void hv_delete_device_domain(struct hv_iommu_domain *hv_domain)
> +{
> + u64 status;
> + unsigned long flags;
> + struct hv_input_delete_device_domain *input;
> +
> + local_irq_save(flags);
> +
> + input = *this_cpu_ptr(hyperv_pcpu_input_arg);
> + memset(input, 0, sizeof(*input));
> + input->device_domain = hv_domain->device_domain;
> + status = hv_do_hypercall(HVCALL_DELETE_DEVICE_DOMAIN, input, NULL);
> +
> + local_irq_restore(flags);
> +
> + if (!hv_result_success(status))
> + pr_err("HVCALL_DELETE_DEVICE_DOMAIN failed, status %lld\n", status);
> +
> + ida_free(&hv_domain->hv_iommu->domain_ids, hv_domain->device_domain.domain_id.id);
> +}
> +
> +static bool hv_iommu_capable(struct device *dev, enum iommu_cap cap)
> +{
> + switch (cap) {
> + case IOMMU_CAP_CACHE_COHERENCY:
> + return true;
> + case IOMMU_CAP_DEFERRED_FLUSH:
> + return true;
> + default:
> + return false;
> + }
> +}
> +
> +static void hv_flush_device_domain(struct hv_iommu_domain *hv_domain)
> +{
> + u64 status;
> + unsigned long flags;
> + struct hv_input_flush_device_domain *input;
> +
> + local_irq_save(flags);
> +
> + input = *this_cpu_ptr(hyperv_pcpu_input_arg);
> + memset(input, 0, sizeof(*input));
> + input->device_domain = hv_domain->device_domain;
> + status = hv_do_hypercall(HVCALL_FLUSH_DEVICE_DOMAIN, input, NULL);
> +
> + local_irq_restore(flags);
> +
> + if (!hv_result_success(status))
> + pr_err("HVCALL_FLUSH_DEVICE_DOMAIN failed, status %lld\n", status);
> +}
> +
> +static int hv_iommu_attach_dev(struct iommu_domain *domain, struct device *dev,
> + struct iommu_domain *old)
> +{
> + u64 status;
> + u32 prefix;
> + unsigned long flags;
> + struct pci_dev *pdev;
> + struct hv_input_attach_device_domain *input;
> + struct hv_iommu_endpoint *vdev = dev_iommu_priv_get(dev);
> + struct hv_iommu_domain *hv_domain = to_hv_iommu_domain(domain);
> + int ret;
> +
> + if (vdev->hv_domain == hv_domain)
> + return 0;
> +
> + pdev = to_pci_dev(dev);
> + dev_dbg(dev, "attaching to domain %d\n",
> + hv_domain->device_domain.domain_id.id);
> +
> + ret = hv_iommu_lookup_logical_dev_id(pci_domain_nr(pdev->bus), &prefix);
> + if (ret) {
> + dev_err(&pdev->dev, "no IOMMU registration for vPCI bus\n");
> + return ret;
> + }
> +
> + local_irq_save(flags);
> +
> + input = *this_cpu_ptr(hyperv_pcpu_input_arg);
> + memset(input, 0, sizeof(*input));
> + input->device_domain = hv_domain->device_domain;
> + input->device_id.as_uint64 = (u64)prefix | PCI_FUNC(pdev->devfn);
> + status = hv_do_hypercall(HVCALL_ATTACH_DEVICE_DOMAIN, input, NULL);
> +
> + local_irq_restore(flags);
> +
> + if (!hv_result_success(status))
> + pr_err("HVCALL_ATTACH_DEVICE_DOMAIN failed, status %lld\n", status);
> + else
> + vdev->hv_domain = hv_domain;
> +
> + return hv_result_to_errno(status);
> +}
> +
> +static int hv_iommu_blocking_attach_dev(struct iommu_domain *domain,
> + struct device *dev,
> + struct iommu_domain *old)
> +{
> + int ret = hv_iommu_attach_dev(domain, dev, old);
> +
> + /*
> + * Attaching to the blocking domain only asks the hypervisor to
> + * disable translation and IOPF for the device, so it cannot fail
> + * unless there is a driver or hypervisor bug. Return the hypercall
> + * status rather than 0 so that a failure on the DMA ownership claim
> + * path (VFIO/iommufd) fails the claim instead of leaving the device
> + * unblocked. WARN since such a failure indicates a bug.
> + */
> + WARN_ON(ret);
> + return ret;
> +}
> +
> +static int hv_iommu_get_logical_device_property(struct device *dev,
> + u32 code,
> + struct hv_output_get_logical_device_property *property)
> +{
> + u64 status;
> + u32 prefix;
> + unsigned long flags;
> + int ret;
> + struct pci_dev *pdev = to_pci_dev(dev);
> + struct hv_input_get_logical_device_property *input;
> + struct hv_output_get_logical_device_property *output;
> +
> + ret = hv_iommu_lookup_logical_dev_id(pci_domain_nr(pdev->bus), &prefix);
> + if (ret)
> + return ret;
> +
> + local_irq_save(flags);
> +
> + input = *this_cpu_ptr(hyperv_pcpu_input_arg);
> + output = (struct hv_output_get_logical_device_property *)(input + 1);
> + memset(input, 0, sizeof(*input));
> + input->partition_id = HV_PARTITION_ID_SELF;
> + input->logical_device_id = (u64)prefix | PCI_FUNC(pdev->devfn);
> + input->code = code;
> + status = hv_do_hypercall(HVCALL_GET_LOGICAL_DEVICE_PROPERTY, input, output);
> + *property = *output;
> +
> + local_irq_restore(flags);
> +
> + if (!hv_result_success(status))
> + pr_err("HVCALL_GET_LOGICAL_DEVICE_PROPERTY failed, status %lld\n", status);
> +
> + return hv_result_to_errno(status);
> +}
> +
> +static struct iommu_device *hv_iommu_probe_device(struct device *dev)
> +{
> + struct pci_dev *pdev;
> + struct hv_iommu_endpoint *vdev;
> + struct hv_output_get_logical_device_property device_iommu_property = {0};
> +
> + if (!dev_is_pci(dev))
> + return ERR_PTR(-ENODEV);
> +
> + pdev = to_pci_dev(dev);
> +
> + if (hv_iommu_get_logical_device_property(dev,
> + HV_LOGICAL_DEVICE_PROPERTY_PVIOMMU,
> + &device_iommu_property) ||
> + !(device_iommu_property.device_iommu & HV_DEVICE_IOMMU_ENABLED))
> + return ERR_PTR(-ENODEV);
> +
> + vdev = kzalloc_obj(*vdev, GFP_KERNEL);
> + if (!vdev)
> + return ERR_PTR(-ENOMEM);
> +
> + vdev->dev = dev;
> + vdev->hv_iommu = hv_iommu_device;
> + dev_iommu_priv_set(dev, vdev);
> +
> + if (hv_iommu_ats_supported(hv_iommu_device->cap) &&
> + pci_ats_supported(pdev))
> + pci_enable_ats(pdev, __ffs(hv_iommu_device->pgsize_bitmap));
> +
> + return &vdev->hv_iommu->iommu;
> +}
> +
> +static void hv_iommu_release_device(struct device *dev)
> +{
> + struct hv_iommu_endpoint *vdev = dev_iommu_priv_get(dev);
> + struct pci_dev *pdev = to_pci_dev(dev);
> +
> + if (pdev->ats_enabled)
> + pci_disable_ats(pdev);
> +
> + dev_iommu_priv_set(dev, NULL);
> +
> + kfree(vdev);
> +}
> +
> +static struct iommu_group *hv_iommu_device_group(struct device *dev)
> +{
> + if (dev_is_pci(dev))
> + return pci_device_group(dev);
> +
> + WARN_ON_ONCE(1);
> + return generic_device_group(dev);
> +}
> +
> +static int hv_configure_device_domain(struct hv_iommu_domain *hv_domain, u32 domain_type)
> +{
> + u64 status;
> + unsigned long flags;
> + struct pt_iommu_x86_64_hw_info pt_info;
> + struct hv_input_configure_device_domain *input;
> +
> + local_irq_save(flags);
> +
> + input = *this_cpu_ptr(hyperv_pcpu_input_arg);
> + memset(input, 0, sizeof(*input));
> + input->device_domain = hv_domain->device_domain;
> + input->settings.flags.blocked = (domain_type == IOMMU_DOMAIN_BLOCKED);
> + /*
> + * Clearing translation_enabled bypasses translation (DMA uses the GPA
> + * directly), which only suits identity. The hypervisor requires paging
> + * and blocked domains to keep it set.
> + */
> + input->settings.flags.translation_enabled = (domain_type != IOMMU_DOMAIN_IDENTITY);
> +
> + if (domain_type & __IOMMU_DOMAIN_PAGING) {
> + pt_iommu_x86_64_hw_info(&hv_domain->pt_iommu_x86_64, &pt_info);
> + input->settings.page_table_root = pt_info.gcr3_pt;
> + input->settings.flags.first_stage_paging_mode =
> + pt_info.levels == 5;
> + }
> + status = hv_do_hypercall(HVCALL_CONFIGURE_DEVICE_DOMAIN, input, NULL);
> +
> + local_irq_restore(flags);
> +
> + if (!hv_result_success(status))
> + pr_err("HVCALL_CONFIGURE_DEVICE_DOMAIN failed, status %lld\n", status);
> +
> + return hv_result_to_errno(status);
> +}
> +
> +static int __init hv_initialize_static_domains(void)
> +{
> + int ret;
> + struct hv_iommu_domain *hv_domain;
> +
> + /* Default stage-1 identity domain */
> + hv_domain = &hv_identity_domain;
> +
> + ret = hv_create_device_domain(hv_domain, HV_DEVICE_DOMAIN_TYPE_S1);
> + if (ret)
> + return ret;
> +
> + ret = hv_configure_device_domain(hv_domain, IOMMU_DOMAIN_IDENTITY);
> + if (ret)
> + goto delete_identity_domain;
> +
> + /* Default stage-1 blocked domain */
> + hv_domain = &hv_blocking_domain;
> +
> + ret = hv_create_device_domain(hv_domain, HV_DEVICE_DOMAIN_TYPE_S1);
> + if (ret)
> + goto delete_identity_domain;
> +
> + ret = hv_configure_device_domain(hv_domain, IOMMU_DOMAIN_BLOCKED);
> + if (ret)
> + goto delete_blocked_domain;
> +
> + return 0;
> +
> +delete_blocked_domain:
> + hv_delete_device_domain(&hv_blocking_domain);
> +delete_identity_domain:
> + hv_delete_device_domain(&hv_identity_domain);
> + return ret;
> +}
> +
> +/* x86 architectural MSI address range */
> +#define INTERRUPT_RANGE_START (0xfee00000)
> +#define INTERRUPT_RANGE_END (0xfeefffff)
> +static void hv_iommu_get_resv_regions(struct device *dev,
> + struct list_head *head)
> +{
> + struct iommu_resv_region *region;
> +
> + region = iommu_alloc_resv_region(INTERRUPT_RANGE_START,
> + INTERRUPT_RANGE_END - INTERRUPT_RANGE_START + 1,
> + 0, IOMMU_RESV_MSI, GFP_KERNEL);
> + if (!region)
> + return;
> +
> + list_add_tail(®ion->list, head);
> +}
> +
> +static void hv_iommu_flush_iotlb_all(struct iommu_domain *domain)
> +{
> + hv_flush_device_domain(to_hv_iommu_domain(domain));
> +}
> +
> +static void hv_iommu_iotlb_sync(struct iommu_domain *domain,
> + struct iommu_iotlb_gather *iotlb_gather)
> +{
> + hv_flush_device_domain(to_hv_iommu_domain(domain));
> +
> + iommu_put_pages_list(&iotlb_gather->freelist);
> +}
> +
> +static void hv_iommu_paging_domain_free(struct iommu_domain *domain)
> +{
> + struct hv_iommu_domain *hv_domain = to_hv_iommu_domain(domain);
> +
> + /* Free all remaining mappings */
> + pt_iommu_deinit(&hv_domain->pt_iommu);
> +
> + hv_delete_device_domain(hv_domain);
> +
> + kfree(hv_domain);
> +}
> +
> +static const struct iommu_domain_ops hv_iommu_identity_domain_ops = {
> + .attach_dev = hv_iommu_attach_dev,
> +};
> +
> +static const struct iommu_domain_ops hv_iommu_blocking_domain_ops = {
> + .attach_dev = hv_iommu_blocking_attach_dev,
> +};
> +
> +static const struct iommu_domain_ops hv_iommu_paging_domain_ops = {
> + .attach_dev = hv_iommu_attach_dev,
> + IOMMU_PT_DOMAIN_OPS(x86_64),
> + .flush_iotlb_all = hv_iommu_flush_iotlb_all,
> + .iotlb_sync = hv_iommu_iotlb_sync,
> + .free = hv_iommu_paging_domain_free,
> +};
> +
> +static struct iommu_domain *hv_iommu_domain_alloc_paging(struct device *dev)
> +{
> + int ret;
> + struct hv_iommu_domain *hv_domain;
> + struct pt_iommu_x86_64_cfg cfg = {};
> +
> + hv_domain = kzalloc_obj(*hv_domain, GFP_KERNEL);
> + if (!hv_domain)
> + return ERR_PTR(-ENOMEM);
> +
> + ret = hv_create_device_domain(hv_domain, HV_DEVICE_DOMAIN_TYPE_S1);
> + if (ret)
> + goto err_free;
> +
> + hv_domain->pt_iommu.nid = dev_to_node(dev);
> +
> + cfg.common.hw_max_vasz_lg2 = hv_iommu_device->max_iova_width;
> + cfg.common.hw_max_oasz_lg2 = 52;
> + cfg.top_level = (hv_iommu_device->max_iova_width > 48) ? 4 : 3;
> +
> + ret = pt_iommu_x86_64_init(&hv_domain->pt_iommu_x86_64, &cfg, GFP_KERNEL);
> + if (ret)
> + goto err_delete_domain;
> +
> + /* Constrain to page sizes the hypervisor supports */
> + hv_domain->domain.pgsize_bitmap &= hv_iommu_device->pgsize_bitmap;
> +
> + hv_domain->domain.ops = &hv_iommu_paging_domain_ops;
> +
> + ret = hv_configure_device_domain(hv_domain, __IOMMU_DOMAIN_PAGING);
> + if (ret)
> + goto err_pt_deinit;
> +
> + return &hv_domain->domain;
> +
> +err_pt_deinit:
> + pt_iommu_deinit(&hv_domain->pt_iommu);
> +err_delete_domain:
> + hv_delete_device_domain(hv_domain);
> +err_free:
> + kfree(hv_domain);
> + return ERR_PTR(ret);
> +}
> +
> +static struct iommu_ops hv_iommu_ops = {
> + .capable = hv_iommu_capable,
> + .domain_alloc_paging = hv_iommu_domain_alloc_paging,
> + .probe_device = hv_iommu_probe_device,
> + .release_device = hv_iommu_release_device,
> + .device_group = hv_iommu_device_group,
> + .get_resv_regions = hv_iommu_get_resv_regions,
> + .owner = THIS_MODULE,
> + .identity_domain = &hv_identity_domain.domain,
> + .blocked_domain = &hv_blocking_domain.domain,
> + .release_domain = &hv_blocking_domain.domain,
> +};
> +
> +static int hv_iommu_detect(struct hv_output_get_iommu_capabilities *hv_iommu_cap)
> +{
> + u64 status;
> + unsigned long flags;
> + struct hv_input_get_iommu_capabilities *input;
> + struct hv_output_get_iommu_capabilities *output;
> +
> + local_irq_save(flags);
> +
> + input = *this_cpu_ptr(hyperv_pcpu_input_arg);
> + output = (struct hv_output_get_iommu_capabilities *)(input + 1);
> + memset(input, 0, sizeof(*input));
> + input->partition_id = HV_PARTITION_ID_SELF;
> + status = hv_do_hypercall(HVCALL_GET_IOMMU_CAPABILITIES, input, output);
> + *hv_iommu_cap = *output;
> +
> + local_irq_restore(flags);
> +
> + if (!hv_result_success(status))
> + pr_err("HVCALL_GET_IOMMU_CAPABILITIES failed, status %lld\n", status);
> +
> + return hv_result_to_errno(status);
> +}
> +
> +static void __init hv_init_iommu_device(struct hv_iommu_dev *hv_iommu,
> + struct hv_output_get_iommu_capabilities *hv_iommu_cap)
> +{
> + ida_init(&hv_iommu->domain_ids);
> +
> + hv_iommu->cap = hv_iommu_cap->iommu_cap;
> + hv_iommu->max_iova_width = hv_iommu_cap->max_iova_width;
> + if (!hv_iommu_5lvl_supported(hv_iommu->cap) &&
> + hv_iommu->max_iova_width > 48) {
> + pr_info("5-level paging not supported, limiting iova width to 48.\n");
> + hv_iommu->max_iova_width = 48;
> + }
> +
> + hv_iommu->geometry = (struct iommu_domain_geometry) {
> + .aperture_start = 0,
> + .aperture_end = (((u64)1) << hv_iommu->max_iova_width) - 1,
> + .force_aperture = true,
> + };
> +
> + hv_iommu->first_domain = HV_DEVICE_DOMAIN_ID_DEFAULT + 1;
> + hv_iommu->last_domain = HV_DEVICE_DOMAIN_ID_NULL - 1;
> + hv_iommu->pgsize_bitmap = hv_iommu_cap->pgsize_bitmap;
> + hv_iommu_device = hv_iommu;
> +}
> +
> +int __init hv_iommu_init(void)
> +{
> + int ret = 0;
> + struct hv_iommu_dev *hv_iommu = NULL;
> + struct hv_output_get_iommu_capabilities hv_iommu_cap = {0};
> +
> + if (no_iommu || iommu_detected)
> + return -ENODEV;
> +
> + if (!hv_is_hyperv_initialized())
> + return -ENODEV;
> +
> + ret = hv_iommu_detect(&hv_iommu_cap);
> + if (ret) {
> + pr_err("HVCALL_GET_IOMMU_CAPABILITIES failed: %d\n", ret);
> + return -ENODEV;
> + }
> +
> + if (!hv_iommu_present(hv_iommu_cap.iommu_cap) ||
> + !hv_iommu_s1_domain_supported(hv_iommu_cap.iommu_cap)) {
> + pr_err("IOMMU capabilities not sufficient: cap=0x%llx\n",
> + hv_iommu_cap.iommu_cap);
> + return -ENODEV;
> + }
> +
> + /*
> + * The page table code only maps x86 page sizes (4K/2M/1G); require the
> + * hypervisor to advertise a non-empty subset of exactly those.
> + */
> + if (!hv_iommu_cap.pgsize_bitmap ||
> + (hv_iommu_cap.pgsize_bitmap & ~(u64)(SZ_4K | SZ_2M | SZ_1G))) {
> + pr_err("unsupported page sizes: pgsize_bitmap=0x%llx\n",
> + hv_iommu_cap.pgsize_bitmap);
> + return -ENODEV;
> + }
> +
> + iommu_detected = 1;
> + pci_request_acs();
> +
> + hv_iommu = kzalloc_obj(*hv_iommu, GFP_KERNEL);
> + if (!hv_iommu)
> + return -ENOMEM;
> +
> + hv_init_iommu_device(hv_iommu, &hv_iommu_cap);
> +
> + ret = hv_initialize_static_domains();
> + if (ret) {
> + pr_err("static domains init failed: %d\n", ret);
> + goto err_free;
> + }
> +
> + ret = iommu_device_sysfs_add(&hv_iommu->iommu, NULL, NULL, "%s", "hv-iommu");
> + if (ret) {
> + pr_err("iommu_device_sysfs_add failed: %d\n", ret);
> + goto err_delete_static_domains;
> + }
> +
> + ret = iommu_device_register(&hv_iommu->iommu, &hv_iommu_ops, NULL);
> + if (ret) {
> + pr_err("iommu_device_register failed: %d\n", ret);
> + goto err_sysfs_remove;
> + }
> +
> + pr_info("successfully initialized\n");
> + return 0;
> +
> +err_sysfs_remove:
> + iommu_device_sysfs_remove(&hv_iommu->iommu);
> +err_delete_static_domains:
> + hv_delete_device_domain(&hv_blocking_domain);
> + hv_delete_device_domain(&hv_identity_domain);
> +err_free:
> + kfree(hv_iommu);
> + return ret;
> +}
> diff --git a/drivers/iommu/hyperv/iommu.h b/drivers/iommu/hyperv/iommu.h
> new file mode 100644
> index 000000000000..3a9f40fa2403
> --- /dev/null
> +++ b/drivers/iommu/hyperv/iommu.h
> @@ -0,0 +1,51 @@
> +/* SPDX-License-Identifier: GPL-2.0 */
> +
> +/*
> + * Hyper-V IOMMU driver.
> + *
> + * Copyright (C) 2024-2025, Microsoft, Inc.
> + *
> + */
> +
> +#ifndef _HYPERV_IOMMU_H
> +#define _HYPERV_IOMMU_H
> +
> +struct hv_iommu_dev {
> + struct iommu_device iommu;
> + struct ida domain_ids;
> +
> + /* Device configuration */
> + u8 max_iova_width;
> + u8 max_pasid_width;
> + u64 cap;
> + u64 pgsize_bitmap;
> +
> + struct iommu_domain_geometry geometry;
> + u64 first_domain;
> + u64 last_domain;
> +};
> +
> +struct hv_iommu_domain {
> + union {
> + struct iommu_domain domain;
> + struct pt_iommu pt_iommu;
> + struct pt_iommu_x86_64 pt_iommu_x86_64;
> + };
> + struct hv_iommu_dev *hv_iommu;
> + struct hv_input_device_domain device_domain;
> + u64 pgsize_bitmap;
> +};
> +
> +PT_IOMMU_CHECK_DOMAIN(struct hv_iommu_domain, pt_iommu, domain);
> +PT_IOMMU_CHECK_DOMAIN(struct hv_iommu_domain, pt_iommu_x86_64.iommu, domain);
> +
> +struct hv_iommu_endpoint {
> + struct device *dev;
> + struct hv_iommu_dev *hv_iommu;
> + struct hv_iommu_domain *hv_domain;
> +};
> +
> +#define to_hv_iommu_domain(d) \
> + container_of(d, struct hv_iommu_domain, domain)
> +
> +#endif /* _HYPERV_IOMMU_H */
^ permalink raw reply
* Re: [PATCH v6 4/4] mshv: Use hmm_range_fault_unlocked() for region faults
From: sashiko-bot @ 2026-07-06 18:21 UTC (permalink / raw)
To: Stanislav Kinsburskii; +Cc: linux-hyperv
In-Reply-To: <178336052192.504354.1841795575701703197.stgit@skinsburskii>
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
Pre-existing issues:
- [High] Missing check for fatal signals in retry loop handling HMM faults.
--
commit 4f92a5b1f4b0160351016ede4e0c8d47d308cbfa
Author: Stanislav Kinsburskii <skinsburskii@gmail.com>
mshv: Use hmm_range_fault_unlocked() for region faults
Convert mshv_region_hmm_fault_and_lock() to use hmm_range_fault_unlocked()
instead of taking mmap_read_lock() around hmm_range_fault() directly.
This allows the MSHV fault path to handle mappings whose fault handlers may
drop mmap_lock.
> diff --git a/drivers/hv/mshv_regions.c b/drivers/hv/mshv_regions.c
> index 6d65e5b421529..04676f06c5c74 100644
> --- a/drivers/hv/mshv_regions.c
> +++ b/drivers/hv/mshv_regions.c
> @@ -388,13 +388,11 @@ int mshv_region_get(struct mshv_mem_region *region)
> *
> * This function performs the following steps:
> * 1. Reads the notifier sequence for the HMM range.
> - * 2. Acquires a read lock on the memory map.
> - * 3. Handles HMM faults for the specified range.
> - * 4. Releases the read lock on the memory map.
> - * 5. If successful, locks the memory region mutex.
> - * 6. Verifies if the notifier sequence has changed during the operation.
> + * 2. Handles HMM faults for the specified range.
> + * 3. If successful, locks the memory region mutex.
> + * 4. Verifies if the notifier sequence has changed during the operation.
> * If it has, releases the mutex and returns -EBUSY to match with
> - * hmm_range_fault() return code for repeating.
> + * hmm_range_fault_unlocked() so the caller retries the range fault.
[Severity: High]
This isn't a bug introduced by this patch, but since we are modifying the
fault retry logic, does the retry loop in mshv_region_range_fault() need to
check for fatal signals?
When mmu_interval_read_retry() detects a collision, this function returns
-EBUSY, which bubbles up to the caller in drivers/hv/mshv_regions.c:
mshv_region_range_fault() {
...
do {
ret = mshv_region_hmm_fault_and_lock(region, &range);
} while (ret == -EBUSY);
...
}
If a userspace program continuously invalidates the memory range concurrently
(for example, via madvise(MADV_DONTNEED)), mmu_interval_read_retry() can
repeatedly return true.
Without a check for fatal_signal_pending(current) inside the loop or before
returning -EBUSY, could this cause the kernel thread to spin indefinitely and
ignore fatal signals like SIGKILL?
> *
> * Return: 0 on success, a negative error code otherwise.
> */
> @@ -404,9 +402,7 @@ static int mshv_region_hmm_fault_and_lock(struct mshv_mem_region *region,
> int ret;
>
> range->notifier_seq = mmu_interval_read_begin(range->notifier);
> - mmap_read_lock(region->mreg_mni.mm);
> - ret = hmm_range_fault(range);
> - mmap_read_unlock(region->mreg_mni.mm);
> + ret = hmm_range_fault_unlocked(range);
> if (ret)
> return ret;
>
--
Sashiko AI review · https://sashiko.dev/#/patchset/178336023903.504354.7500950448226027718.stgit@skinsburskii?part=4
^ permalink raw reply
* Re: [PATCH v6 4/4] mshv: Use hmm_range_fault_unlocked() for region faults
From: Jason Gunthorpe @ 2026-07-06 18:16 UTC (permalink / raw)
To: Stanislav Kinsburskii
Cc: Liam.Howlett, akpm, david, corbet, leon, ljs, mhocko, rppt, shuah,
skhan, surenb, vbabka, kys, haiyangz, wei.liu, decui, longli,
linux-doc, linux-kernel, linux-kselftest, linux-mm, linux-hyperv
In-Reply-To: <178336052192.504354.1841795575701703197.stgit@skinsburskii>
On Mon, Jul 06, 2026 at 10:55:21AM -0700, Stanislav Kinsburskii wrote:
> Convert mshv_region_hmm_fault_and_lock() to use
> hmm_range_fault_unlocked() instead of taking mmap_read_lock() around
> hmm_range_fault() directly.
Please convert all the trivial users too, thanks
Jason
^ permalink raw reply
* Re: [PATCH v6 2/4] mm/hmm: add hmm_range_fault_unlocked() for mmap lock-drop support
From: sashiko-bot @ 2026-07-06 18:16 UTC (permalink / raw)
To: Stanislav Kinsburskii; +Cc: linux-hyperv
In-Reply-To: <178336050736.504354.6052508493877924780.stgit@skinsburskii>
Thank you for your contribution! Sashiko AI review found 2 potential issue(s) to consider:
- [High] `hmm_range_fault_unlocked()` uses uninterruptible `mmap_read_lock()` inside a retry sequence, which can delay or block fatal signals.
- [High] Unconditional dereference of `hmm_vma_walk->locked` in `hmm_do_fault()` can cause a kernel panic if a buggy fault handler returns `VM_FAULT_RETRY` or `VM_FAULT_COMPLETED` without `FAULT_FLAG_ALLOW_RETRY`.
--
commit 0808f6917e681bb06bd68894103530cfbabe07b9
Author: Stanislav Kinsburskii <skinsburskii@gmail.com>
mm/hmm: add hmm_range_fault_unlocked() for mmap lock-drop support
This commit adds hmm_range_fault_unlocked() for callers that do not
need to hold mmap_lock across any work outside the HMM fault itself.
It takes mmap_read_lock() internally and allows handle_mm_fault() to
drop it during fault handling.
> diff --git a/mm/hmm.c b/mm/hmm.c
> index 2129b1ee4c35b..316116994209c 100644
> --- a/mm/hmm.c
> +++ b/mm/hmm.c
[ ... ]
> @@ -651,37 +663,33 @@ static int hmm_do_fault(struct mm_struct *mm,
> fault_flags |= FAULT_FLAG_WRITE;
> }
>
> - for (; addr < end; addr += PAGE_SIZE)
> - if (handle_mm_fault(vma, addr, fault_flags, NULL) &
> - VM_FAULT_ERROR)
> - return -EFAULT;
> + for (; addr < end; addr += PAGE_SIZE) {
> + vm_fault_t ret;
> +
> + ret = handle_mm_fault(vma, addr, fault_flags, NULL);
> +
> + if (ret & (VM_FAULT_COMPLETED | VM_FAULT_RETRY)) {
> + *hmm_vma_walk->locked = 0;
[Severity: High]
Can this result in a NULL pointer dereference?
When the legacy hmm_range_fault() is called, it passes NULL for the
locked parameter, meaning hmm_vma_walk->locked will be NULL.
If a buggy fault handler incorrectly returns VM_FAULT_RETRY even when
FAULT_FLAG_ALLOW_RETRY is not set, it seems this would unconditionally
dereference the NULL pointer here.
> + return HMM_FAULT_UNLOCKED;
> + }
[ ... ]
> +int hmm_range_fault_unlocked(struct hmm_range *range)
> +{
> + struct mm_struct *mm = range->notifier->mm;
> + int locked = 1;
> + int ret;
> +
> + mmap_read_lock(mm);
[Severity: High]
Should this use mmap_read_lock_killable() instead of the uninterruptible
mmap_read_lock()?
Since this function is designed to be called inside a retry loop, if the
mmap lock is highly contended, it appears a task could enter an
uninterruptible sleep here and block fatal signals indefinitely. This might
defeat the purpose of the FAULT_FLAG_KILLABLE flag that is set during the
actual fault.
> + ret = hmm_range_fault_locked(range, &locked);
> + if (locked)
> + mmap_read_unlock(mm);
> +
> + return ret;
> +}
> +EXPORT_SYMBOL(hmm_range_fault_unlocked);
--
Sashiko AI review · https://sashiko.dev/#/patchset/178336023903.504354.7500950448226027718.stgit@skinsburskii?part=2
^ permalink raw reply
* Re: [PATCH v6 2/4] mm/hmm: add hmm_range_fault_unlocked() for mmap lock-drop support
From: Jason Gunthorpe @ 2026-07-06 18:02 UTC (permalink / raw)
To: Stanislav Kinsburskii
Cc: Liam.Howlett, akpm, david, corbet, leon, ljs, mhocko, rppt, shuah,
skhan, surenb, vbabka, kys, haiyangz, wei.liu, decui, longli,
linux-doc, linux-kernel, linux-kselftest, linux-mm, linux-hyperv
In-Reply-To: <178336050736.504354.6052508493877924780.stgit@skinsburskii>
On Mon, Jul 06, 2026 at 10:55:07AM -0700, Stanislav Kinsburskii wrote:
> +The caller must not hold ``mmap_read_lock`` before the call.
> +``hmm_range_fault_unlocked()`` takes the mmap read lock internally and allows
> +``handle_mm_fault()`` to drop it during fault handling. If the mmap lock is
> +dropped, the function returns ``-EBUSY``. The caller must then restart the
> +walk from ``range->start`` with a fresh notifier sequence. ``-EINTR`` is
> +returned if a fatal signal is pending during retry handling.
It doesn't make sense to me to expose this to the caller, the caller
isn't holding the lock and there is no reason to redo the mmu_interval
at this point, so why not just loop inside hmm_range_fault_unlocked ?
Jason
^ permalink raw reply
* Re: [PATCH v5 01/51] x86/apic: Provide helpers to set local APIC timer period in hz and khz
From: Sean Christopherson @ 2026-07-06 18:00 UTC (permalink / raw)
To: Michael Kelley
Cc: Jonathan Corbet, Paolo Bonzini, Thomas Gleixner, Ingo Molnar,
Borislav Petkov, Dave Hansen, x86@kernel.org, Kiryl Shutsemau,
Rick Edgecombe, K. Y. Srinivasan, Haiyang Zhang, Wei Liu,
Dexuan Cui, Long Li, Ajay Kaher, Alexey Makhalov, Jan Kiszka,
Andy Lutomirski, Peter Zijlstra, Juergen Gross, Daniel Lezcano,
John Stultz, Shuah Khan, H. Peter Anvin, Vitaly Kuznetsov,
Broadcom internal kernel review list, Boris Ostrovsky,
Stephen Boyd, linux-doc@vger.kernel.org, kvm@vger.kernel.org,
linux-kernel@vger.kernel.org, linux-coco@lists.linux.dev,
linux-hyperv@vger.kernel.org, virtualization@lists.linux.dev,
xen-devel@lists.xenproject.org, Tom Lendacky, Nikunj A Dadhania,
David Woodhouse, David Woodhouse, Thomas Gleixner
In-Reply-To: <SN6PR02MB41578D3C34AB283B892C27A4D4F52@SN6PR02MB4157.namprd02.prod.outlook.com>
On Thu, Jul 02, 2026, Michael Kelley wrote:
> > @@ -796,6 +796,16 @@ bool __init apic_needs_pit(void)
> > return lapic_timer_period == 0;
> > }
> >
> > +void apic_set_timer_period_khz(u64 period_khz, const char *source)
> > +{
> > + lapic_timer_period = mul_u64_u32_div(period_khz, 1000, HZ);
> > +}
> > +
> > +void apic_set_timer_period_hz(u64 period_hz, const char *source)
> > +{
> > + lapic_timer_period = div_u64(period_hz, HZ);
> > +}
>
> A string "source" argument is passed in, but not used. Is there an
> envisioned future use? Also, this function doesn't output a pr_info()
> message like the existing Hyper-V and VMware code does.
It was a complete goof on my part (Sashiko also pointed out the oddity[*]). I
fully intended to log a message and provide equivalent Hyper-V/VMware behavior,
and totally spaced it.
[*] https://lore.kernel.org/all/20260701194621.4BD691F000E9@smtp.kernel.org
> I don't know that the message is all that useful, though I do remember one
> case where I was debugging some clock/timer issue when I looked at it.
^ permalink raw reply
* [PATCH v6 4/4] mshv: Use hmm_range_fault_unlocked() for region faults
From: Stanislav Kinsburskii @ 2026-07-06 17:55 UTC (permalink / raw)
To: Liam.Howlett, akpm, akpm, david, jgg, corbet, leon, ljs, mhocko,
rppt, shuah, skhan, surenb, vbabka, skinsburskii, kys, haiyangz,
wei.liu, decui, longli
Cc: linux-doc, linux-kernel, linux-kernel, linux-kselftest, linux-mm,
linux-hyperv
In-Reply-To: <178336023903.504354.7500950448226027718.stgit@skinsburskii>
Convert mshv_region_hmm_fault_and_lock() to use
hmm_range_fault_unlocked() instead of taking mmap_read_lock() around
hmm_range_fault() directly.
This lets the HMM helper own the mmap read lock and allows the MSHV fault
path to handle mappings whose fault handlers may drop mmap_lock, including
userfaultfd-backed VMAs. The existing caller already retries on -EBUSY
after refreshing the mmu_interval_notifier sequence, so no control flow
change is needed beyond using the unlocked helper.
Signed-off-by: Stanislav Kinsburskii <skinsburskii@gmail.com>
---
drivers/hv/mshv_regions.c | 14 +++++---------
1 file changed, 5 insertions(+), 9 deletions(-)
diff --git a/drivers/hv/mshv_regions.c b/drivers/hv/mshv_regions.c
index 6d65e5b42152..04676f06c5c7 100644
--- a/drivers/hv/mshv_regions.c
+++ b/drivers/hv/mshv_regions.c
@@ -388,13 +388,11 @@ int mshv_region_get(struct mshv_mem_region *region)
*
* This function performs the following steps:
* 1. Reads the notifier sequence for the HMM range.
- * 2. Acquires a read lock on the memory map.
- * 3. Handles HMM faults for the specified range.
- * 4. Releases the read lock on the memory map.
- * 5. If successful, locks the memory region mutex.
- * 6. Verifies if the notifier sequence has changed during the operation.
+ * 2. Handles HMM faults for the specified range.
+ * 3. If successful, locks the memory region mutex.
+ * 4. Verifies if the notifier sequence has changed during the operation.
* If it has, releases the mutex and returns -EBUSY to match with
- * hmm_range_fault() return code for repeating.
+ * hmm_range_fault_unlocked() so the caller retries the range fault.
*
* Return: 0 on success, a negative error code otherwise.
*/
@@ -404,9 +402,7 @@ static int mshv_region_hmm_fault_and_lock(struct mshv_mem_region *region,
int ret;
range->notifier_seq = mmu_interval_read_begin(range->notifier);
- mmap_read_lock(region->mreg_mni.mm);
- ret = hmm_range_fault(range);
- mmap_read_unlock(region->mreg_mni.mm);
+ ret = hmm_range_fault_unlocked(range);
if (ret)
return ret;
^ permalink raw reply related
* [PATCH v6 3/4] selftests/mm: add userfaultfd test for HMM unlocked path
From: Stanislav Kinsburskii @ 2026-07-06 17:55 UTC (permalink / raw)
To: Liam.Howlett, akpm, akpm, david, jgg, corbet, leon, ljs, mhocko,
rppt, shuah, skhan, surenb, vbabka, skinsburskii, kys, haiyangz,
wei.liu, decui, longli
Cc: linux-doc, linux-kernel, linux-kernel, linux-kselftest, linux-mm,
linux-hyperv
In-Reply-To: <178336023903.504354.7500950448226027718.stgit@skinsburskii>
Add a selftest that exercises hmm_range_fault_unlocked() with a
userfaultfd-backed mapping. The test:
1. Creates an anonymous mmap region
2. Registers it with userfaultfd (UFFDIO_REGISTER_MODE_MISSING)
3. Spawns a handler thread that responds to page faults by filling
pages with a known pattern (0xAB) via UFFDIO_COPY
4. Issues HMM_DMIRROR_READ_UNLOCKED to the test_hmm driver, which
calls hmm_range_fault_unlocked() internally
5. Verifies the device read back the data provided by the userfaultfd
handler
This requires changes to the test_hmm kernel module:
- New dmirror_range_fault_unlocked() that uses the new HMM API
- New dmirror_fault_unlocked() and dmirror_read_unlocked() wrappers
- New HMM_DMIRROR_READ_UNLOCKED ioctl (0x09)
Assisted-by: GitHub-Copilot:claude-opus-4.6
Signed-off-by: Stanislav Kinsburskii <skinsburskii@gmail.com>
---
lib/test_hmm.c | 114 ++++++++++++++++++++++++
lib/test_hmm_uapi.h | 1
tools/testing/selftests/mm/hmm-tests.c | 149 ++++++++++++++++++++++++++++++++
3 files changed, 264 insertions(+)
diff --git a/lib/test_hmm.c b/lib/test_hmm.c
index 45c0cb992218..dcc3a23bb983 100644
--- a/lib/test_hmm.c
+++ b/lib/test_hmm.c
@@ -389,6 +389,76 @@ static int dmirror_range_fault(struct dmirror *dmirror,
return ret;
}
+static int dmirror_range_fault_unlocked(struct dmirror *dmirror,
+ struct hmm_range *range)
+{
+ unsigned long timeout =
+ jiffies + msecs_to_jiffies(HMM_RANGE_DEFAULT_TIMEOUT);
+ int ret;
+
+ while (true) {
+ if (time_after(jiffies, timeout)) {
+ ret = -EBUSY;
+ goto out;
+ }
+
+ range->notifier_seq = mmu_interval_read_begin(range->notifier);
+ ret = hmm_range_fault_unlocked(range);
+ if (ret) {
+ if (ret == -EBUSY)
+ continue;
+ goto out;
+ }
+
+ mutex_lock(&dmirror->mutex);
+ if (mmu_interval_read_retry(range->notifier,
+ range->notifier_seq)) {
+ mutex_unlock(&dmirror->mutex);
+ continue;
+ }
+ break;
+ }
+
+ ret = dmirror_do_fault(dmirror, range);
+
+ mutex_unlock(&dmirror->mutex);
+out:
+ return ret;
+}
+
+static int dmirror_fault_unlocked(struct dmirror *dmirror,
+ unsigned long start,
+ unsigned long end, bool write)
+{
+ struct mm_struct *mm = dmirror->notifier.mm;
+ unsigned long addr;
+ unsigned long pfns[32];
+ struct hmm_range range = {
+ .notifier = &dmirror->notifier,
+ .hmm_pfns = pfns,
+ .pfn_flags_mask = 0,
+ .default_flags =
+ HMM_PFN_REQ_FAULT | (write ? HMM_PFN_REQ_WRITE : 0),
+ .dev_private_owner = dmirror->mdevice,
+ };
+ int ret = 0;
+
+ if (!mmget_not_zero(mm))
+ return -EFAULT;
+
+ for (addr = start; addr < end; addr = range.end) {
+ range.start = addr;
+ range.end = min(addr + (ARRAY_SIZE(pfns) << PAGE_SHIFT), end);
+
+ ret = dmirror_range_fault_unlocked(dmirror, &range);
+ if (ret)
+ break;
+ }
+
+ mmput(mm);
+ return ret;
+}
+
static int dmirror_fault(struct dmirror *dmirror, unsigned long start,
unsigned long end, bool write)
{
@@ -488,6 +558,47 @@ static int dmirror_read(struct dmirror *dmirror, struct hmm_dmirror_cmd *cmd)
return ret;
}
+static int dmirror_read_unlocked(struct dmirror *dmirror,
+ struct hmm_dmirror_cmd *cmd)
+{
+ struct dmirror_bounce bounce;
+ unsigned long start, end;
+ unsigned long size = cmd->npages << PAGE_SHIFT;
+ int ret;
+
+ start = cmd->addr;
+ end = start + size;
+ if (end < start)
+ return -EINVAL;
+
+ ret = dmirror_bounce_init(&bounce, start, size);
+ if (ret)
+ return ret;
+
+ while (1) {
+ mutex_lock(&dmirror->mutex);
+ ret = dmirror_do_read(dmirror, start, end, &bounce);
+ mutex_unlock(&dmirror->mutex);
+ if (ret != -ENOENT)
+ break;
+
+ start = cmd->addr + (bounce.cpages << PAGE_SHIFT);
+ ret = dmirror_fault_unlocked(dmirror, start, end, false);
+ if (ret)
+ break;
+ cmd->faults++;
+ }
+
+ if (ret == 0) {
+ if (copy_to_user(u64_to_user_ptr(cmd->ptr), bounce.ptr,
+ bounce.size))
+ ret = -EFAULT;
+ }
+ cmd->cpages = bounce.cpages;
+ dmirror_bounce_fini(&bounce);
+ return ret;
+}
+
static int dmirror_do_write(struct dmirror *dmirror, unsigned long start,
unsigned long end, struct dmirror_bounce *bounce)
{
@@ -1572,6 +1683,9 @@ static long dmirror_fops_unlocked_ioctl(struct file *filp,
dmirror->flags = cmd.npages;
ret = 0;
break;
+ case HMM_DMIRROR_READ_UNLOCKED:
+ ret = dmirror_read_unlocked(dmirror, &cmd);
+ break;
default:
return -EINVAL;
diff --git a/lib/test_hmm_uapi.h b/lib/test_hmm_uapi.h
index f94c6d457338..ea9b0ec404fb 100644
--- a/lib/test_hmm_uapi.h
+++ b/lib/test_hmm_uapi.h
@@ -38,6 +38,7 @@ struct hmm_dmirror_cmd {
#define HMM_DMIRROR_CHECK_EXCLUSIVE _IOWR('H', 0x06, struct hmm_dmirror_cmd)
#define HMM_DMIRROR_RELEASE _IOWR('H', 0x07, struct hmm_dmirror_cmd)
#define HMM_DMIRROR_FLAGS _IOWR('H', 0x08, struct hmm_dmirror_cmd)
+#define HMM_DMIRROR_READ_UNLOCKED _IOWR('H', 0x09, struct hmm_dmirror_cmd)
#define HMM_DMIRROR_FLAG_FAIL_ALLOC (1ULL << 0)
diff --git a/tools/testing/selftests/mm/hmm-tests.c b/tools/testing/selftests/mm/hmm-tests.c
index 2f2b9879d100..71765c4d80f3 100644
--- a/tools/testing/selftests/mm/hmm-tests.c
+++ b/tools/testing/selftests/mm/hmm-tests.c
@@ -29,6 +29,10 @@
#include <sys/mman.h>
#include <sys/ioctl.h>
#include <sys/time.h>
+#include <sys/syscall.h>
+#include <sys/eventfd.h>
+#include <linux/userfaultfd.h>
+#include <poll.h>
/*
* This is a private UAPI to the kernel test module so it isn't exported
@@ -2949,4 +2953,149 @@ TEST_F_TIMEOUT(hmm, benchmark_thp_migration, 120)
&thp_results, ®ular_results);
}
}
+/*
+ * Test that HMM can fault in pages backed by userfaultfd using the
+ * hmm_range_fault_unlocked() path. This exercises the lock-drop retry
+ * logic in the HMM framework.
+ */
+struct uffd_thread_args {
+ int uffd;
+ int stop_fd;
+ void *page_buffer;
+ unsigned long page_size;
+};
+
+static void *uffd_handler_thread(void *arg)
+{
+ struct uffd_thread_args *args = arg;
+ struct uffd_msg msg;
+ struct uffdio_copy copy;
+ struct pollfd pollfd[2];
+ int ret;
+
+ pollfd[0].fd = args->uffd;
+ pollfd[0].events = POLLIN;
+ pollfd[1].fd = args->stop_fd;
+ pollfd[1].events = POLLIN;
+
+ while (1) {
+ ret = poll(pollfd, 2, -1);
+ if (ret <= 0)
+ break;
+ if (pollfd[1].revents)
+ break;
+ if (!(pollfd[0].revents & POLLIN))
+ break;
+
+ ret = read(args->uffd, &msg, sizeof(msg));
+ if (ret != sizeof(msg))
+ break;
+
+ if (msg.event != UFFD_EVENT_PAGEFAULT)
+ break;
+
+ /* Fill the page with a known pattern */
+ memset(args->page_buffer, 0xAB, args->page_size);
+
+ copy.dst = msg.arg.pagefault.address & ~(args->page_size - 1);
+ copy.src = (unsigned long)args->page_buffer;
+ copy.len = args->page_size;
+ copy.mode = 0;
+ copy.copy = 0;
+
+ ret = ioctl(args->uffd, UFFDIO_COPY, ©);
+ if (ret < 0)
+ break;
+ }
+
+ return NULL;
+}
+
+TEST_F(hmm, userfaultfd_read)
+{
+ struct hmm_buffer *buffer;
+ struct uffd_thread_args uffd_args;
+ unsigned long npages;
+ unsigned long size;
+ unsigned long i;
+ unsigned char *ptr;
+ pthread_t thread;
+ int uffd;
+ int stop_fd;
+ int ret;
+ struct uffdio_api api;
+ struct uffdio_register reg;
+ uint64_t stop = 1;
+ ssize_t nwrite;
+
+ npages = 4;
+ size = npages << self->page_shift;
+
+ /* Create userfaultfd */
+ uffd = syscall(__NR_userfaultfd, O_CLOEXEC | O_NONBLOCK);
+ if (uffd < 0)
+ SKIP(return, "userfaultfd not available");
+
+ api.api = UFFD_API;
+ api.features = 0;
+ ret = ioctl(uffd, UFFDIO_API, &api);
+ ASSERT_EQ(ret, 0);
+
+ buffer = malloc(sizeof(*buffer));
+ ASSERT_NE(buffer, NULL);
+
+ buffer->fd = -1;
+ buffer->size = size;
+ buffer->mirror = malloc(size);
+ ASSERT_NE(buffer->mirror, NULL);
+
+ /* Create anonymous mapping */
+ buffer->ptr = mmap(NULL, size,
+ PROT_READ | PROT_WRITE,
+ MAP_PRIVATE | MAP_ANONYMOUS,
+ -1, 0);
+ ASSERT_NE(buffer->ptr, MAP_FAILED);
+
+ /* Register the region with userfaultfd */
+ reg.range.start = (unsigned long)buffer->ptr;
+ reg.range.len = size;
+ reg.mode = UFFDIO_REGISTER_MODE_MISSING;
+ ret = ioctl(uffd, UFFDIO_REGISTER, ®);
+ ASSERT_EQ(ret, 0);
+
+ /* Set up the handler thread */
+ uffd_args.uffd = uffd;
+ stop_fd = eventfd(0, EFD_CLOEXEC);
+ ASSERT_GE(stop_fd, 0);
+ uffd_args.stop_fd = stop_fd;
+ uffd_args.page_buffer = malloc(self->page_size);
+ ASSERT_NE(uffd_args.page_buffer, NULL);
+ uffd_args.page_size = self->page_size;
+
+ ret = pthread_create(&thread, NULL, uffd_handler_thread, &uffd_args);
+ ASSERT_EQ(ret, 0);
+
+ /*
+ * Use the unlocked read path which allows the mmap lock to be
+ * dropped during the fault, enabling userfaultfd resolution.
+ */
+ ret = hmm_dmirror_cmd(self->fd, HMM_DMIRROR_READ_UNLOCKED,
+ buffer, npages);
+ ASSERT_EQ(ret, 0);
+ ASSERT_EQ(buffer->cpages, npages);
+
+ /* Verify the device read the data filled by the uffd handler */
+ ptr = buffer->mirror;
+ for (i = 0; i < size; ++i)
+ ASSERT_EQ(ptr[i], (unsigned char)0xAB);
+
+ nwrite = write(stop_fd, &stop, sizeof(stop));
+ ASSERT_EQ(nwrite, sizeof(stop));
+ pthread_join(thread, NULL);
+ close(stop_fd);
+ free(uffd_args.page_buffer);
+ close(uffd);
+ hmm_buffer_free(buffer);
+}
+
TEST_HARNESS_MAIN
^ permalink raw reply related
* [PATCH v6 2/4] mm/hmm: add hmm_range_fault_unlocked() for mmap lock-drop support
From: Stanislav Kinsburskii @ 2026-07-06 17:55 UTC (permalink / raw)
To: Liam.Howlett, akpm, akpm, david, jgg, corbet, leon, ljs, mhocko,
rppt, shuah, skhan, surenb, vbabka, skinsburskii, kys, haiyangz,
wei.liu, decui, longli
Cc: linux-doc, linux-kernel, linux-kernel, linux-kselftest, linux-mm,
linux-hyperv
In-Reply-To: <178336023903.504354.7500950448226027718.stgit@skinsburskii>
hmm_range_fault() requires the caller to hold the mmap read lock for the
duration of the call. This is incompatible with mappings whose fault
handler may release the mmap lock, notably userfaultfd-managed regions,
where handle_mm_fault() can return VM_FAULT_RETRY or VM_FAULT_COMPLETED
after dropping the lock. Drivers that need to populate device page tables
for such mappings have no way to do so today.
Add hmm_range_fault_unlocked() for callers that do not need to hold
mmap_lock across any work outside the HMM fault itself. The new helper
takes mmap_read_lock() internally, calls the common HMM fault
implementation, and releases the lock before returning if it is still
held.
When handle_mm_fault() drops mmap_lock, hmm_range_fault_unlocked() returns
-EBUSY. The caller must then restart its HMM walk with a fresh
mmu_interval_read_begin() sequence. If a fatal signal is pending after the
lock has been dropped, -EINTR is returned instead. This gives callers a
simple restart contract while allowing fault handlers that must drop
mmap_lock, including userfaultfd-backed mappings, to make progress.
The common implementation conditionally sets FAULT_FLAG_ALLOW_RETRY and
FAULT_FLAG_KILLABLE only for hmm_range_fault_unlocked(). The existing
hmm_range_fault() path still passes no locked state, does not allow
handle_mm_fault() to drop mmap_lock, and remains a thin wrapper preserving
the existing API contract for current callers.
The previous refactor that moved page fault handling out of the
page-table walk callbacks is what makes this change small. Faults now run
after walk_page_range() has unwound, with only mmap_lock held, so dropping
it does not interact with the walker's pte spinlock or hugetlb_vma_lock.
Hugetlb regions therefore participate in the unlocked path uniformly with
PTE- and PMD-level mappings; no special case is required.
Documentation/mm/hmm.rst is updated with a description of the new API and
the recommended caller pattern.
Signed-off-by: Stanislav Kinsburskii <skinsburskii@gmail.com>
---
Documentation/mm/hmm.rst | 59 +++++++++++++++++++++
include/linux/hmm.h | 1
mm/hmm.c | 129 ++++++++++++++++++++++++++++++++++++----------
3 files changed, 160 insertions(+), 29 deletions(-)
diff --git a/Documentation/mm/hmm.rst b/Documentation/mm/hmm.rst
index 7d61b7a8b65b..844343712abd 100644
--- a/Documentation/mm/hmm.rst
+++ b/Documentation/mm/hmm.rst
@@ -208,6 +208,65 @@ invalidate() callback. That lock must be held before calling
mmu_interval_read_retry() to avoid any race with a concurrent CPU page table
update.
+Dropping the mmap lock during page faults
+=========================================
+
+Some VMAs have fault handlers that need to release the mmap lock while
+servicing a fault (for example, regions managed by ``userfaultfd``).
+``hmm_range_fault()`` cannot be used on such mappings because it must hold the
+mmap lock for the duration of the call. Drivers that need to support them
+should call::
+
+ int hmm_range_fault_unlocked(struct hmm_range *range);
+
+The caller must not hold ``mmap_read_lock`` before the call.
+``hmm_range_fault_unlocked()`` takes the mmap read lock internally and allows
+``handle_mm_fault()`` to drop it during fault handling. If the mmap lock is
+dropped, the function returns ``-EBUSY``. The caller must then restart the
+walk from ``range->start`` with a fresh notifier sequence. ``-EINTR`` is
+returned if a fatal signal is pending during retry handling.
+
+A typical caller looks like this::
+
+ int driver_populate_range_unlocked(...)
+ {
+ struct hmm_range range;
+ ...
+
+ range.notifier = &interval_sub;
+ range.start = ...;
+ range.end = ...;
+ range.hmm_pfns = ...;
+
+ if (!mmget_not_zero(interval_sub.mm))
+ return -EFAULT;
+
+ again:
+ range.notifier_seq = mmu_interval_read_begin(&interval_sub);
+ ret = hmm_range_fault_unlocked(&range);
+ if (ret) {
+ if (ret == -EBUSY)
+ goto again;
+ goto out_put;
+ }
+
+ take_lock(driver->update);
+ if (mmu_interval_read_retry(&interval_sub, range.notifier_seq)) {
+ release_lock(driver->update);
+ goto again;
+ }
+
+ /* Use pfns array content to update device page table,
+ * under the update lock */
+
+ release_lock(driver->update);
+ ret = 0;
+
+ out_put:
+ mmput(interval_sub.mm);
+ return ret;
+ }
+
Leverage default_flags and pfn_flags_mask
=========================================
diff --git a/include/linux/hmm.h b/include/linux/hmm.h
index db75ffc949a7..1b584e5a2900 100644
--- a/include/linux/hmm.h
+++ b/include/linux/hmm.h
@@ -123,6 +123,7 @@ struct hmm_range {
* Please see Documentation/mm/hmm.rst for how to use the range API.
*/
int hmm_range_fault(struct hmm_range *range);
+int hmm_range_fault_unlocked(struct hmm_range *range);
/*
* HMM_RANGE_DEFAULT_TIMEOUT - default timeout (ms) when waiting for a range
diff --git a/mm/hmm.c b/mm/hmm.c
index 2129b1ee4c35..316116994209 100644
--- a/mm/hmm.c
+++ b/mm/hmm.c
@@ -32,6 +32,7 @@
struct hmm_vma_walk {
struct hmm_range *range;
+ int *locked;
unsigned long last;
unsigned long end;
unsigned int required_fault;
@@ -44,6 +45,14 @@ struct hmm_vma_walk {
*/
#define HMM_FAULT_PENDING -EAGAIN
+/*
+ * Internal sentinel returned by hmm_do_fault() when handle_mm_fault()
+ * completes a page fault with the mmap lock dropped. hmm_do_fault() sets
+ * *locked = 0; the outer loop consumes the sentinel and never propagates it
+ * to the caller.
+ */
+#define HMM_FAULT_UNLOCKED -ENOLCK
+
enum {
HMM_NEED_FAULT = 1 << 0,
HMM_NEED_WRITE_FAULT = 1 << 1,
@@ -73,9 +82,9 @@ static int hmm_pfns_fill(unsigned long addr, unsigned long end,
*
* Called by the walk callbacks when they discover that part of the range
* needs a page fault. The callback records what to fault and returns
- * HMM_FAULT_PENDING; the outer loop in hmm_range_fault() drops back out of
- * walk_page_range() and invokes handle_mm_fault() from a context where no
- * page-table or hugetlb_vma_lock is held.
+ * HMM_FAULT_PENDING; the outer loop in hmm_range_fault_locked() drops
+ * back out of walk_page_range() and invokes handle_mm_fault() from a context
+ * where no page-table or hugetlb_vma_lock is held.
*/
static int hmm_record_fault(unsigned long addr, unsigned long end,
unsigned int required_fault,
@@ -624,7 +633,7 @@ static const struct mm_walk_ops hmm_walk_ops = {
/*
* hmm_do_fault - fault in a range recorded by a walk callback
*
- * Called from the outer loop in hmm_range_fault() after a callback
+ * Called from the outer loop in hmm_range_fault_locked() after a callback
* returned HMM_FAULT_PENDING. At this point we hold only mmap_lock;
* the page-table spinlock and any hugetlb_vma_lock acquired by the walk
* framework have already been released by the unwind.
@@ -641,6 +650,9 @@ static int hmm_do_fault(struct mm_struct *mm,
unsigned int fault_flags = FAULT_FLAG_REMOTE;
struct vm_area_struct *vma;
+ if (hmm_vma_walk->locked)
+ fault_flags |= FAULT_FLAG_ALLOW_RETRY | FAULT_FLAG_KILLABLE;
+
vma = vma_lookup(mm, addr);
if (!vma)
return -EFAULT;
@@ -651,37 +663,33 @@ static int hmm_do_fault(struct mm_struct *mm,
fault_flags |= FAULT_FLAG_WRITE;
}
- for (; addr < end; addr += PAGE_SIZE)
- if (handle_mm_fault(vma, addr, fault_flags, NULL) &
- VM_FAULT_ERROR)
- return -EFAULT;
+ for (; addr < end; addr += PAGE_SIZE) {
+ vm_fault_t ret;
+
+ ret = handle_mm_fault(vma, addr, fault_flags, NULL);
+
+ if (ret & (VM_FAULT_COMPLETED | VM_FAULT_RETRY)) {
+ *hmm_vma_walk->locked = 0;
+ return HMM_FAULT_UNLOCKED;
+ }
+
+ if (ret & VM_FAULT_ERROR) {
+ int err = vm_fault_to_errno(ret, 0);
+
+ if (err)
+ return err;
+ BUG();
+ }
+ }
return -EBUSY;
}
-/**
- * hmm_range_fault - try to fault some address in a virtual address range
- * @range: argument structure
- *
- * Returns 0 on success or one of the following error codes:
- *
- * -EINVAL: Invalid arguments or mm or virtual address is in an invalid vma
- * (e.g., device file vma).
- * -ENOMEM: Out of memory.
- * -EPERM: Invalid permission (e.g., asking for write and range is read
- * only).
- * -EBUSY: The range has been invalidated and the caller needs to wait for
- * the invalidation to finish.
- * -EFAULT: A page was requested to be valid and could not be made valid
- * ie it has no backing VMA or it is illegal to access
- *
- * This is similar to get_user_pages(), except that it can read the page tables
- * without mutating them (ie causing faults).
- */
-int hmm_range_fault(struct hmm_range *range)
+static int hmm_range_fault_locked(struct hmm_range *range, int *locked)
{
struct hmm_vma_walk hmm_vma_walk = {
.range = range,
+ .locked = locked,
.last = range->start,
};
struct mm_struct *mm = range->notifier->mm;
@@ -704,8 +712,14 @@ int hmm_range_fault(struct hmm_range *range)
* returns -EBUSY so the loop re-walks and picks up the
* now-present entries.
*/
- if (ret == HMM_FAULT_PENDING)
+ if (ret == HMM_FAULT_PENDING) {
ret = hmm_do_fault(mm, &hmm_vma_walk);
+ if (ret == HMM_FAULT_UNLOCKED) {
+ if (fatal_signal_pending(current))
+ return -EINTR;
+ return -EBUSY;
+ }
+ }
/*
* When -EBUSY is returned the loop restarts with
* hmm_vma_walk.last set to an address that has not been stored
@@ -715,8 +729,65 @@ int hmm_range_fault(struct hmm_range *range)
} while (ret == -EBUSY);
return ret;
}
+
+/**
+ * hmm_range_fault - try to fault some address in a virtual address range
+ * @range: argument structure
+ *
+ * Returns 0 on success or one of the following error codes:
+ *
+ * -EINVAL: Invalid arguments or mm or virtual address is in an invalid vma
+ * (e.g., device file vma).
+ * -ENOMEM: Out of memory.
+ * -EPERM: Invalid permission (e.g., asking for write and range is read
+ * only).
+ * -EBUSY: The range has been invalidated and the caller needs to wait for
+ * the invalidation to finish.
+ * -EFAULT: A page was requested to be valid and could not be made valid
+ * ie it has no backing VMA or it is illegal to access
+ *
+ * This is similar to get_user_pages(), except that it can read the page tables
+ * without mutating them (ie causing faults).
+ *
+ * The mmap lock must be held by the caller and will remain held on return.
+ * For a variant that allows the mmap lock to be dropped during faults (e.g.,
+ * for userfaultfd support), see hmm_range_fault_unlocked().
+ */
+int hmm_range_fault(struct hmm_range *range)
+{
+ return hmm_range_fault_locked(range, NULL);
+}
EXPORT_SYMBOL(hmm_range_fault);
+/**
+ * hmm_range_fault_unlocked - fault in a range, possibly dropping the mmap lock
+ * @range: argument structure
+ *
+ * This is similar to hmm_range_fault(), except the caller must not hold the
+ * mmap lock. The function takes the mmap read lock internally and allows
+ * handle_mm_fault() to drop it during faults. If the lock is dropped, the
+ * function returns -EBUSY and the caller must restart the walk with a fresh
+ * mmu_interval_read_begin() sequence.
+ *
+ * Returns 0 on success or one of the error codes documented for
+ * hmm_range_fault(). -EINTR is returned if a fatal signal is pending during
+ * retry handling.
+ */
+int hmm_range_fault_unlocked(struct hmm_range *range)
+{
+ struct mm_struct *mm = range->notifier->mm;
+ int locked = 1;
+ int ret;
+
+ mmap_read_lock(mm);
+ ret = hmm_range_fault_locked(range, &locked);
+ if (locked)
+ mmap_read_unlock(mm);
+
+ return ret;
+}
+EXPORT_SYMBOL(hmm_range_fault_unlocked);
+
/**
* hmm_dma_map_alloc - Allocate HMM map structure
* @dev: device to allocate structure for
^ permalink raw reply related
* [PATCH v6 1/4] mm/hmm: move page fault handling out of walk callbacks
From: Stanislav Kinsburskii @ 2026-07-06 17:55 UTC (permalink / raw)
To: Liam.Howlett, akpm, akpm, david, jgg, corbet, leon, ljs, mhocko,
rppt, shuah, skhan, surenb, vbabka, skinsburskii, kys, haiyangz,
wei.liu, decui, longli
Cc: linux-doc, linux-kernel, linux-kernel, linux-kselftest, linux-mm,
linux-hyperv
In-Reply-To: <178336023903.504354.7500950448226027718.stgit@skinsburskii>
hmm_range_fault() currently triggers page faults from inside the page-table
walk callbacks: hmm_vma_walk_pmd(), hmm_vma_walk_pud(),
hmm_vma_walk_hugetlb_entry() and the pte-level helper all call
hmm_vma_fault(), which in turn calls handle_mm_fault() while the walker
still holds nested locks. The pte spinlock is dropped explicitly by each
caller, and the hugetlb path manually drops and retakes
hugetlb_vma_lock_read around the fault to dodge a deadlock against the walk
framework's unconditional unlock.
This layering does not extend cleanly to fault handlers that may release
mmap_lock (VM_FAULT_RETRY, VM_FAULT_COMPLETED). If the lock is dropped
while walk_page_range() is mid-traversal, the VMA can be freed before the
walk framework's matching hugetlb_vma_unlock_read(), turning that unlock
into a use-after-free.
Split the responsibilities the way get_user_pages() does. Walk callbacks
become inspect-only: when they detect a range that needs to be faulted in,
they record it in struct hmm_vma_walk and return a private sentinel
(HMM_FAULT_PENDING). The outer loop in hmm_range_fault() then drops out of
walk_page_range(), invokes a new helper hmm_do_fault() that calls
handle_mm_fault() with only mmap_lock held, and restarts the walk so the
now-present entries are collected into hmm_pfns.
No functional change for existing callers. As a side effect the hugetlb
callback no longer needs the hugetlb_vma_{un}lock_read dance, and every
fault-path exit from the callbacks now releases the pte spinlock on a
single, common path. This refactor is also a precursor for adding an
unlockable variant of hmm_range_fault() in a follow-up patch.
Signed-off-by: Stanislav Kinsburskii <skinsburskii@gmail.com>
---
mm/hmm.c | 118 +++++++++++++++++++++++++++++++++++++++-----------------------
1 file changed, 75 insertions(+), 43 deletions(-)
diff --git a/mm/hmm.c b/mm/hmm.c
index 4f3f627d2b47..2129b1ee4c35 100644
--- a/mm/hmm.c
+++ b/mm/hmm.c
@@ -33,8 +33,17 @@
struct hmm_vma_walk {
struct hmm_range *range;
unsigned long last;
+ unsigned long end;
+ unsigned int required_fault;
};
+/*
+ * Internal sentinel returned by walk callbacks when they need a page fault.
+ * The callback stores end/required_fault in hmm_vma_walk; the outer loop
+ * consumes the sentinel and never propagates it to the caller.
+ */
+#define HMM_FAULT_PENDING -EAGAIN
+
enum {
HMM_NEED_FAULT = 1 << 0,
HMM_NEED_WRITE_FAULT = 1 << 1,
@@ -60,37 +69,25 @@ static int hmm_pfns_fill(unsigned long addr, unsigned long end,
}
/*
- * hmm_vma_fault() - fault in a range lacking valid pmd or pte(s)
- * @addr: range virtual start address (inclusive)
- * @end: range virtual end address (exclusive)
- * @required_fault: HMM_NEED_* flags
- * @walk: mm_walk structure
- * Return: -EBUSY after page fault, or page fault error
+ * hmm_record_fault() - record a range that needs to be faulted in
*
- * This function will be called whenever pmd_none() or pte_none() returns true,
- * or whenever there is no page directory covering the virtual address range.
+ * Called by the walk callbacks when they discover that part of the range
+ * needs a page fault. The callback records what to fault and returns
+ * HMM_FAULT_PENDING; the outer loop in hmm_range_fault() drops back out of
+ * walk_page_range() and invokes handle_mm_fault() from a context where no
+ * page-table or hugetlb_vma_lock is held.
*/
-static int hmm_vma_fault(unsigned long addr, unsigned long end,
- unsigned int required_fault, struct mm_walk *walk)
+static int hmm_record_fault(unsigned long addr, unsigned long end,
+ unsigned int required_fault,
+ struct mm_walk *walk)
{
struct hmm_vma_walk *hmm_vma_walk = walk->private;
- struct vm_area_struct *vma = walk->vma;
- unsigned int fault_flags = FAULT_FLAG_REMOTE;
WARN_ON_ONCE(!required_fault);
hmm_vma_walk->last = addr;
-
- if (required_fault & HMM_NEED_WRITE_FAULT) {
- if (!(vma->vm_flags & VM_WRITE))
- return -EPERM;
- fault_flags |= FAULT_FLAG_WRITE;
- }
-
- for (; addr < end; addr += PAGE_SIZE)
- if (handle_mm_fault(vma, addr, fault_flags, NULL) &
- VM_FAULT_ERROR)
- return -EFAULT;
- return -EBUSY;
+ hmm_vma_walk->end = end;
+ hmm_vma_walk->required_fault = required_fault;
+ return HMM_FAULT_PENDING;
}
static unsigned int hmm_pte_need_fault(const struct hmm_vma_walk *hmm_vma_walk,
@@ -174,7 +171,7 @@ static int hmm_vma_walk_hole(unsigned long addr, unsigned long end,
return hmm_pfns_fill(addr, end, range, HMM_PFN_ERROR);
}
if (required_fault)
- return hmm_vma_fault(addr, end, required_fault, walk);
+ return hmm_record_fault(addr, end, required_fault, walk);
return hmm_pfns_fill(addr, end, range, 0);
}
@@ -209,7 +206,7 @@ static int hmm_vma_handle_pmd(struct mm_walk *walk, unsigned long addr,
required_fault =
hmm_range_need_fault(hmm_vma_walk, hmm_pfns, npages, cpu_flags);
if (required_fault)
- return hmm_vma_fault(addr, end, required_fault, walk);
+ return hmm_record_fault(addr, end, required_fault, walk);
pfn = pmd_pfn(pmd) + ((addr & ~PMD_MASK) >> PAGE_SHIFT);
for (i = 0; addr < end; addr += PAGE_SIZE, i++, pfn++) {
@@ -328,7 +325,7 @@ static int hmm_vma_handle_pte(struct mm_walk *walk, unsigned long addr,
fault:
pte_unmap(ptep);
/* Fault any virtual address we were asked to fault */
- return hmm_vma_fault(addr, end, required_fault, walk);
+ return hmm_record_fault(addr, end, required_fault, walk);
}
#ifdef CONFIG_ARCH_SUPPORTS_PMD_SOFTLEAF
@@ -371,7 +368,7 @@ static int hmm_vma_handle_absent_pmd(struct mm_walk *walk, unsigned long start,
npages, 0);
if (required_fault) {
if (softleaf_is_device_private(entry))
- return hmm_vma_fault(addr, end, required_fault, walk);
+ return hmm_record_fault(addr, end, required_fault, walk);
else
return -EFAULT;
}
@@ -517,7 +514,7 @@ static int hmm_vma_walk_pud(pud_t *pudp, unsigned long start, unsigned long end,
npages, cpu_flags);
if (required_fault) {
spin_unlock(ptl);
- return hmm_vma_fault(addr, end, required_fault, walk);
+ return hmm_record_fault(addr, end, required_fault, walk);
}
pfn = pud_pfn(pud) + ((addr & ~PUD_MASK) >> PAGE_SHIFT);
@@ -564,21 +561,8 @@ static int hmm_vma_walk_hugetlb_entry(pte_t *pte, unsigned long hmask,
required_fault =
hmm_pte_need_fault(hmm_vma_walk, pfn_req_flags, cpu_flags);
if (required_fault) {
- int ret;
-
spin_unlock(ptl);
- hugetlb_vma_unlock_read(vma);
- /*
- * Avoid deadlock: drop the vma lock before calling
- * hmm_vma_fault(), which will itself potentially take and
- * drop the vma lock. This is also correct from a
- * protection point of view, because there is no further
- * use here of either pte or ptl after dropping the vma
- * lock.
- */
- ret = hmm_vma_fault(addr, end, required_fault, walk);
- hugetlb_vma_lock_read(vma);
- return ret;
+ return hmm_record_fault(addr, end, required_fault, walk);
}
pfn = pte_pfn(entry) + ((start & ~hmask) >> PAGE_SHIFT);
@@ -637,6 +621,44 @@ static const struct mm_walk_ops hmm_walk_ops = {
.walk_lock = PGWALK_RDLOCK,
};
+/*
+ * hmm_do_fault - fault in a range recorded by a walk callback
+ *
+ * Called from the outer loop in hmm_range_fault() after a callback
+ * returned HMM_FAULT_PENDING. At this point we hold only mmap_lock;
+ * the page-table spinlock and any hugetlb_vma_lock acquired by the walk
+ * framework have already been released by the unwind.
+ *
+ * Returns -EBUSY on success (all pages faulted, caller should re-walk).
+ * Returns a negative errno on failure.
+ */
+static int hmm_do_fault(struct mm_struct *mm,
+ struct hmm_vma_walk *hmm_vma_walk)
+{
+ unsigned long addr = hmm_vma_walk->last;
+ unsigned long end = hmm_vma_walk->end;
+ unsigned int required_fault = hmm_vma_walk->required_fault;
+ unsigned int fault_flags = FAULT_FLAG_REMOTE;
+ struct vm_area_struct *vma;
+
+ vma = vma_lookup(mm, addr);
+ if (!vma)
+ return -EFAULT;
+
+ if (required_fault & HMM_NEED_WRITE_FAULT) {
+ if (!(vma->vm_flags & VM_WRITE))
+ return -EPERM;
+ fault_flags |= FAULT_FLAG_WRITE;
+ }
+
+ for (; addr < end; addr += PAGE_SIZE)
+ if (handle_mm_fault(vma, addr, fault_flags, NULL) &
+ VM_FAULT_ERROR)
+ return -EFAULT;
+
+ return -EBUSY;
+}
+
/**
* hmm_range_fault - try to fault some address in a virtual address range
* @range: argument structure
@@ -674,6 +696,16 @@ int hmm_range_fault(struct hmm_range *range)
return -EBUSY;
ret = walk_page_range(mm, hmm_vma_walk.last, range->end,
&hmm_walk_ops, &hmm_vma_walk);
+ /*
+ * When HMM_FAULT_PENDING is returned a walk callback
+ * recorded a range that needs handle_mm_fault();
+ * hmm_do_fault() runs the fault outside walk_page_range()
+ * (so no page-table or hugetlb_vma_lock is held) and
+ * returns -EBUSY so the loop re-walks and picks up the
+ * now-present entries.
+ */
+ if (ret == HMM_FAULT_PENDING)
+ ret = hmm_do_fault(mm, &hmm_vma_walk);
/*
* When -EBUSY is returned the loop restarts with
* hmm_vma_walk.last set to an address that has not been stored
^ permalink raw reply related
* [PATCH v6 0/4] mm/hmm: Add mmap lock-drop support for userfaultfd-backed mappings
From: Stanislav Kinsburskii @ 2026-07-06 17:54 UTC (permalink / raw)
To: Liam.Howlett, akpm, akpm, david, jgg, corbet, leon, ljs, mhocko,
rppt, shuah, skhan, surenb, vbabka, skinsburskii, kys, haiyangz,
wei.liu, decui, longli
Cc: linux-doc, linux-kernel, linux-kernel, linux-kselftest, linux-mm,
linux-hyperv
This series extends the HMM framework to support userfaultfd-backed memory
by allowing the mmap read lock to be dropped during hmm_range_fault().
Some page fault handlers — most notably userfaultfd — require the mmap lock
to be released so that userspace can resolve the fault. The current HMM
interface never sets FAULT_FLAG_ALLOW_RETRY, making it impossible to fault
in pages from userfaultfd-registered regions.
This series follows the established int *locked pattern from
get_user_pages_remote() in mm/gup.c. A new helper function,
hmm_range_fault_locked(), accepts an int *locked parameter. When the
mmap lock is dropped during fault resolution (VM_FAULT_RETRY or
VM_FAULT_COMPLETED), the function returns 0 with *locked = 0, signalling
the caller to restart its walk. The existing hmm_range_fault() is
refactored into a thin wrapper that passes NULL, preserving current
behavior for all existing callers.
Possible approaches to lift this limitation are documented in
Documentation/mm/hmm.rst.
Changes in v6:
- Reworked the new API from the external int *locked pattern to
hmm_range_fault_unlocked(), which owns mmap_read_lock() internally.
- Changed the dropped-lock contract: hmm_range_fault_unlocked() now returns
-EBUSY when the mmap lock is dropped, and callers restart with a fresh
mmu_interval_read_begin() sequence.
- Kept hmm_range_fault() as the locked variant for existing users, preserving
its caller-held mmap lock contract.
- Added an in-tree user by converting the MSHV region fault path to
hmm_range_fault_unlocked().
- Updated Documentation/mm/hmm.rst and kernel-doc to describe the unlocked
helper and retry pattern.
- Updated commit messages to match the new API and return semantics.
- Kept the userfaultfd HMM selftest using the test_hmm unlocked read ioctl
path.
Changes in v5:
- Rework hmm_range_fault_unlockable() retry handling to retry
VM_FAULT_RETRY internally with FAULT_FLAG_TRIED set, matching the
fixup_user_fault() pattern and avoiding repeated first-retry lock drops.
- Distinguish VM_FAULT_RETRY from VM_FAULT_COMPLETED: retry faults now
reacquire the mmap lock internally, while completed faults return to the
caller with *locked = 0 so the caller can restart with a fresh notifier
sequence.
- Document the two *locked return states, including the -EINTR case when a
fatal signal is pending after the mmap lock has already been dropped.
- Update comments around HMM_FAULT_UNLOCKED and the HMM fault loop to match
the current hmm_range_fault_unlockable() implementation.
Changes in v4:
- Rebased on 7.2-rc1
Changes in v3:
- Return -EFAULT from dmirror_fault_unlockable() when the mirrored mm can
no longer be pinned.
- Add an eventfd stop signal for the userfaultfd handler thread to avoid
waiting for the poll timeout on successful test completion.
Changes in v2:
- Split into a preparatory refactor (new patch 1) that moves
handle_mm_fault() out of the walk callbacks, plus a smaller feature
patch on top. Suggested by David Hildenbrand.
- Hugetlb regions are now supported on the unlockable path; the v1
-EFAULT short-circuit and the hugetlb_vma_lock_read drop/retake
dance are gone.
- Distinct internal sentinels for "needs fault" (HMM_FAULT_PENDING)
and "lock dropped" (HMM_FAULT_UNLOCKED).
- Outer loop now re-walks after a successful internal fault so the
faulted pfns end up in range->hmm_pfns.
- Kernel-doc on hmm_range_fault_unlockable() and the
Documentation/mm/hmm.rst example match the implementation.
- Dropped the mshv driver conversion (v1 patch 2); will post
separately.
- Selftest converted to drive the path through test_hmm with a
userfaultfd handler (new HMM_DMIRROR_READ_UNLOCKABLE ioctl).
---
Stanislav Kinsburskii (4):
mm/hmm: move page fault handling out of walk callbacks
mm/hmm: add hmm_range_fault_unlocked() for mmap lock-drop support
selftests/mm: add userfaultfd test for HMM unlocked path
mshv: Use hmm_range_fault_unlocked() for region faults
Documentation/mm/hmm.rst | 59 ++++++++
drivers/hv/mshv_regions.c | 14 +-
include/linux/hmm.h | 1
lib/test_hmm.c | 114 ++++++++++++++++
lib/test_hmm_uapi.h | 1
mm/hmm.c | 223 +++++++++++++++++++++++---------
tools/testing/selftests/mm/hmm-tests.c | 149 +++++++++++++++++++++
7 files changed, 492 insertions(+), 69 deletions(-)
^ permalink raw reply
* Re: [PATCH v2 3/4] iommu/hyperv: Add para-virtualized IOMMU support for Hyper-V guest
From: Jacob Pan @ 2026-07-06 16:55 UTC (permalink / raw)
To: Yu Zhang
Cc: linux-kernel, linux-hyperv, iommu, linux-pci, linux-arch, wei.liu,
kys, haiyangz, decui, longli, joro, will, robin.murphy, bhelgaas,
kwilczynski, lpieralisi, mani, robh, arnd, jgg, mhklinux,
tgopinath, easwar.hariharan, mrathor, jacob.pan
In-Reply-To: <20260702160518.311234-4-zhangyu1@linux.microsoft.com>
Hi Yu,
On Fri, 3 Jul 2026 00:05:17 +0800
Yu Zhang <zhangyu1@linux.microsoft.com> wrote:
> +static int hv_iommu_attach_dev(struct iommu_domain *domain, struct
> device *dev,
> + struct iommu_domain *old)
> +{
> + u64 status;
> + u32 prefix;
> + unsigned long flags;
> + struct pci_dev *pdev;
> + struct hv_input_attach_device_domain *input;
> + struct hv_iommu_endpoint *vdev = dev_iommu_priv_get(dev);
> + struct hv_iommu_domain *hv_domain =
> to_hv_iommu_domain(domain);
> + int ret;
> +
> + if (vdev->hv_domain == hv_domain)
> + return 0;
> +
Is this needed? seems the core code already skips same domain attach?
i.e.
static int __iommu_group_set_domain_internal(struct iommu_group *group,
struct iommu_domain
*new_domain, unsigned int flags)
{
...
if (group->domain == new_domain)
return 0;
^ permalink raw reply
* Re: [PATCH net-next v11 0/2] net: mana: add ethtool private flag for full-page RX buffers
From: Dipayaan Roy @ 2026-07-05 17:17 UTC (permalink / raw)
To: Maciej Fijalkowski
Cc: kys, haiyangz, wei.liu, decui, andrew+netdev, davem, edumazet,
kuba, pabeni, leon, longli, kotaranov, horms, shradhagupta,
ssengar, ernis, shirazsaleem, linux-hyperv, netdev, linux-kernel,
linux-rdma, stephen, jacob.e.keller, dipayanroy, leitao, kees,
john.fastabend, hawk, bpf, daniel, ast, sdf, yury.norov,
pavan.chebbi
In-Reply-To: <akUn+nMBrjbUts2N@boxer>
On Wed, Jul 01, 2026 at 04:45:14PM +0200, Maciej Fijalkowski wrote:
> On Wed, Jul 01, 2026 at 07:15:44AM -0700, Dipayaan Roy wrote:
> > On some ARM64 platforms with 4K PAGE_SIZE, utilizing page_pool
> > fragments for allocation in the RX refill path (~2kB buffer per fragment)
> > causes 15-20% throughput regression under high connection counts
> > (>16 TCP streams at 180+ Gbps). Using full-page buffers on these
> > platforms shows no regression and restores line-rate performance.
> >
> > This behavior is observed on a single platform; other platforms
> > perform better with page_pool fragments, indicating this is not a
> > page_pool issue but platform-specific.
> >
> > This series adds an ethtool private flag "full-page-rx" to let the
> > user opt in to one RX buffer per page:
> >
> > ethtool --set-priv-flags eth0 full-page-rx on
> >
> > There is no behavioral change by default. The flag can be persisted
> > via udev rule for affected platforms.
>
> Were you able to track down what is the actual bottleneck on the 'broken'
> platform? What is the performance of full-page approach on healthy
> platforms? On changelog below you mention the frag approach 'outperforms'
> the full-page one.
>
Hi Maciej,
The HW team identified a PCIE root port stall occurring due to a PCIe
errata in a HW IP used for this platform. Using full pages increases the
time in packet refill path and indirectly helping to reduce the back pressure
in NIC pipeline caused due to the stall in root port. As per them it
will be fixed in next version of this hw which is not anytime soon.
On various other healthy platforms with 4k base page size we tested, we see
improvements using page fragments than full pages around anywhere between 5 to 15%.
Regards
Dipayaan Roy
> >
> > This series depends on the following fixes now merged in net-next:
> > commit 17bfe0a8c014 ("net: mana: Add NULL guards in teardown path to prevent panic on attach failure")
> > commit 5b05aa36ee24 ("net: mana: Skip redundant detach on already-detached port")
> >
> > Changes in v11:
> > - Rebased on net-next
> > Changes in v10:
> > - Rebased on net-next which now includes the prerequisite fixes.
> > - Recovery logic in mana_set_priv_flags() leverages the idempotent
> > mana_detach() from the merged fixes.
> > Changes in v9:
> > - Added correct tree.
> > Changes in v8:
> > - Fixed queue_reset_work recovery by restoring port_is_up before
> > scheduling reset so the handler can properly re-attach.
> > - Simplified "err && schedule_port_reset" to "schedule_port_reset".
> > Changes in v7:
> > - Rebased onto net-next.
> > - Retained private flag approach after David Wei's testing on
> > Grace (ARM64) confirmed that fragment mode outperforms
> > full-page mode on other platforms, validating this is a
> > single-platform workaround rather than a generic issue.
> > Changes in v6:
> > - Added missed maintainers.
> > Changes in v5:
> > - Split prep refactor into separate patch (patch 1/2)
> > Changes in v4:
> > - Dropping the smbios string parsing and add ethtool priv flag
> > to reconfigure the queues with full page rx buffers.
> > Changes in v3:
> > - changed u8* to char*
> > Changes in v2:
> > - separate reading string index and the string, remove inline.
> >
> > Dipayaan Roy (2):
> > net: mana: refactor mana_get_strings() and mana_get_sset_count() to
> > use switch
> > net: mana: force full-page RX buffers via ethtool private flag
> >
> > drivers/net/ethernet/microsoft/mana/mana_en.c | 22 ++-
> > .../ethernet/microsoft/mana/mana_ethtool.c | 178 +++++++++++++++---
> > include/net/mana/mana.h | 8 +
> > 3 files changed, 177 insertions(+), 31 deletions(-)
> >
> > --
> > 2.43.0
> >
> >
^ permalink raw reply
* Re: [PATCH v2 3/4] iommu/hyperv: Add para-virtualized IOMMU support for Hyper-V guest
From: Jason Gunthorpe @ 2026-07-03 17:32 UTC (permalink / raw)
To: Yu Zhang
Cc: linux-kernel, linux-hyperv, iommu, linux-pci, linux-arch, wei.liu,
kys, haiyangz, decui, longli, joro, will, robin.murphy, bhelgaas,
kwilczynski, lpieralisi, mani, robh, arnd, mhklinux, jacob.pan,
tgopinath, easwar.hariharan, mrathor
In-Reply-To: <20260702160518.311234-4-zhangyu1@linux.microsoft.com>
On Fri, Jul 03, 2026 at 12:05:17AM +0800, Yu Zhang wrote:
> +static bool hv_iommu_capable(struct device *dev, enum iommu_cap cap)
> +{
> + switch (cap) {
> + case IOMMU_CAP_CACHE_COHERENCY:
> + return true;
> + case IOMMU_CAP_DEFERRED_FLUSH:
> + return true;
This CAP isn't necessary anymore
> +static struct iommu_device *hv_iommu_probe_device(struct device *dev)
> +{
> + struct pci_dev *pdev;
> + struct hv_iommu_endpoint *vdev;
> + struct hv_output_get_logical_device_property device_iommu_property = {0};
> +
> + if (!dev_is_pci(dev))
> + return ERR_PTR(-ENODEV);
> +
> + pdev = to_pci_dev(dev);
> +
> + if (hv_iommu_get_logical_device_property(dev,
> + HV_LOGICAL_DEVICE_PROPERTY_PVIOMMU,
> + &device_iommu_property) ||
> + !(device_iommu_property.device_iommu & HV_DEVICE_IOMMU_ENABLED))
> + return ERR_PTR(-ENODEV);
> +
> + vdev = kzalloc_obj(*vdev, GFP_KERNEL);
> + if (!vdev)
> + return ERR_PTR(-ENOMEM);
> +
> + vdev->dev = dev;
> + vdev->hv_iommu = hv_iommu_device;
> + dev_iommu_priv_set(dev, vdev);
> +
> + if (hv_iommu_ats_supported(hv_iommu_device->cap) &&
> + pci_ats_supported(pdev))
> + pci_enable_ats(pdev, __ffs(hv_iommu_device->pgsize_bitmap));
This can probably just be PAGE_SHIFT
Also ATS shouldn't be enabled until a translation is installed,
otherwise the driver cannot participate in the ATS error handling
Nicolin is working on.
> +static void hv_iommu_release_device(struct device *dev)
> +{
> + struct hv_iommu_endpoint *vdev = dev_iommu_priv_get(dev);
> + struct pci_dev *pdev = to_pci_dev(dev);
> +
> + if (pdev->ats_enabled)
> + pci_disable_ats(pdev);
> +
> + dev_iommu_priv_set(dev, NULL);
No necessary, the caller does it
> +static struct iommu_group *hv_iommu_device_group(struct device *dev)
> +{
> + if (dev_is_pci(dev))
> + return pci_device_group(dev);
> +
> + WARN_ON_ONCE(1);
> + return generic_device_group(dev);
I think you can just return failure here instead of WARN_ON ?
> +static int __init hv_initialize_static_domains(void)
> +{
> + int ret;
> + struct hv_iommu_domain *hv_domain;
> +
> + /* Default stage-1 identity domain */
> + hv_domain = &hv_identity_domain;
> +
> + ret = hv_create_device_domain(hv_domain, HV_DEVICE_DOMAIN_TYPE_S1);
> + if (ret)
> + return ret;
> +
> + ret = hv_configure_device_domain(hv_domain, IOMMU_DOMAIN_IDENTITY);
> + if (ret)
> + goto delete_identity_domain;
IMHO I would change this around to have a single function that accepts
a struct hv_input_configure_device_domain as input and does both of
the hypercalls inside. Then here it is easy to directly construct the
hv_input_configure_device_domain for blocking and identity.
I'd be happy if this never touched domain_type, drivers shouldn't be
touching that.
> +static void __init hv_init_iommu_device(struct hv_iommu_dev *hv_iommu,
> + struct hv_output_get_iommu_capabilities *hv_iommu_cap)
> +{
> + ida_init(&hv_iommu->domain_ids);
> +
> + hv_iommu->cap = hv_iommu_cap->iommu_cap;
> + hv_iommu->max_iova_width = hv_iommu_cap->max_iova_width;
> + if (!hv_iommu_5lvl_supported(hv_iommu->cap) &&
> + hv_iommu->max_iova_width > 48) {
> + pr_info("5-level paging not supported, limiting iova width to 48.\n");
> + hv_iommu->max_iova_width = 48;
> + }
> +
> + hv_iommu->geometry = (struct iommu_domain_geometry) {
> + .aperture_start = 0,
> + .aperture_end = (((u64)1) << hv_iommu->max_iova_width) - 1,
> + .force_aperture = true,
> + };
I don't see anything reading this, I don't expect this to be used?
The max_iova_width has to be passed into the iommupt creation, which
it does:
+ cfg.common.hw_max_vasz_lg2 = hv_iommu_device->max_iova_width;
+ cfg.common.hw_max_oasz_lg2 = 52;
+ cfg.top_level = (hv_iommu_device->max_iova_width > 48) ? 4 : 3;
+ ret = pt_iommu_x86_64_init(&hv_domain->pt_iommu_x86_64, &cfg, GFP_KERNEL);
+ if (ret)
So just delete hv->iommu->geometry.
Also, VT-D has weirdness where the HW can require a 4 level table but
only a 3 level worth of IOVA width is being used. This was a
real-world bug we hit when converting to iommupt. This interaction
with the HV doesn't seem able to represent that.
> + /*
> + * The page table code only maps x86 page sizes (4K/2M/1G); require the
> + * hypervisor to advertise a non-empty subset of exactly those.
> + */
> + if (!hv_iommu_cap.pgsize_bitmap ||
> + (hv_iommu_cap.pgsize_bitmap & ~(u64)(SZ_4K | SZ_2M | SZ_1G))) {
> + pr_err("unsupported page sizes: pgsize_bitmap=0x%llx\n",
> + hv_iommu_cap.pgsize_bitmap);
> + return -ENODEV;
> + }
This can just be
if (!(hv_iommu_cap.pgsize_bitmap & PAGE_SHIFT)) {
pr_err("unsupported page sizes: pgsize_bitmap=0x%llx\n",
hv_iommu_cap.pgsize_bitmap);
} return -ENODEV;
Which is all you really need. If the HV doesn't support 1G it is
perfectly fine, the iommupt page bitmap is already masked by this.
> + ret = iommu_device_register(&hv_iommu->iommu, &hv_iommu_ops, NULL);
> + if (ret) {
> + pr_err("iommu_device_register failed: %d\n", ret);
> + goto err_sysfs_remove;
> + }
> +
> + pr_info("successfully initialized\n");
Don't log someting so vauge?
Jason
^ permalink raw reply
* Re: [PATCH v2 4/4] iommu/hyperv: Add page-selective IOTLB flush support
From: Jason Gunthorpe @ 2026-07-03 17:10 UTC (permalink / raw)
To: Yu Zhang
Cc: linux-kernel, linux-hyperv, iommu, linux-pci, linux-arch, wei.liu,
kys, haiyangz, decui, longli, joro, will, robin.murphy, bhelgaas,
kwilczynski, lpieralisi, mani, robh, arnd, mhklinux, jacob.pan,
tgopinath, easwar.hariharan, mrathor
In-Reply-To: <20260702160518.311234-5-zhangyu1@linux.microsoft.com>
On Fri, Jul 03, 2026 at 12:05:18AM +0800, Yu Zhang wrote:
> @@ -401,10 +402,74 @@ static void hv_iommu_flush_iotlb_all(struct iommu_domain *domain)
> hv_flush_device_domain(to_hv_iommu_domain(domain));
> }
>
> +/*
> + * Calculate the minimal power-of-two aligned range that covers [start, end]
> + * (end is inclusive). Returns a single (page_number, page_mask_shift)
> + * descriptor that may over-flush when the range is not naturally aligned.
> + */
> +static void hv_iommu_calc_flush_range(unsigned long start, unsigned long end,
> + union hv_iommu_flush_va *va)
> +{
> + unsigned long start_pfn = HVPFN_DOWN(start);
> + unsigned long last_pfn = HVPFN_UP(end + 1) - 1;
Pedantically end can be ULONG_MAX, you shouldn't be adding to it since
it will overflow.
> + unsigned long mask_shift, aligned_pfn;
> +
> + if (start_pfn == last_pfn) {
> + mask_shift = 0;
> + } else {
> + /*
> + * Find the highest bit position where start_pfn and last_pfn
> + * differ. A range aligned to one above that bit is the
> + * smallest power-of-two region that covers both endpoints.
> + */
> + mask_shift = __fls(start_pfn ^ last_pfn) + 1;
> + }
> +
> + aligned_pfn = ALIGN_DOWN(start_pfn, 1UL << mask_shift);
I think the whole thing is simpler if it stays using bytes until the end:
sz_lg2 = __fls(gather->start ^ gather->end);
if (sz_lg2 < HV_PAGE_SHIFT)
cmd.sz_lg2 = HV_PAGE_SHIFT;
page_number = (gather->start & ~(1UL << sz_lg2)) >> HV_PAGE_SHIFT;
page_mask_shift = sz_lg2 - HV_PAGE_SIFT;
No overflows that way either
Jason
^ permalink raw reply
* [PATCH net-next v5] net: mana: Add Interrupt Moderation support
From: Haiyang Zhang @ 2026-07-02 22:01 UTC (permalink / raw)
To: linux-hyperv, netdev, K. Y. Srinivasan, Haiyang Zhang, Wei Liu,
Dexuan Cui, Long Li, Andrew Lunn, David S. Miller, Eric Dumazet,
Jakub Kicinski, Paolo Abeni, Konstantin Taranov, Simon Horman,
Erni Sri Satya Vennela, Dipayaan Roy, Shradha Gupta, Aditya Garg,
Stanislav Fomichev, Breno Leitao, linux-kernel, linux-rdma
Cc: paulros
From: Haiyang Zhang <haiyangz@microsoft.com>
Add Static and Dynamic Interrupt Moderation (DIM) support for
Rx and Tx.
Update queue creation procedure with new data struct with the related
settings.
Add functions to collect stat for DIM, and workers to update DIM data
and settings.
Update ethtool handler to get/set the moderation settings from a user.
To avoid detach/re-attach ops, ring DIM doorbell to change settings
at run time.
By default, adaptive-rx/tx (DIM) are enabled if supported by HW.
Signed-off-by: Haiyang Zhang <haiyangz@microsoft.com>
Reviewed-by: Simon Horman <horms@kernel.org>
---
v5:
Updated function return type and comments suggested by Paolo Abeni.
v4:
Fixed tx stat, concurrency, and mb issues from Simon's review.
v3:
Updated to avoid detach/re-attach ops as suggested by Paolo.
v2:
Updated with comments from Jedrzej.
---
drivers/net/ethernet/microsoft/Kconfig | 1 +
.../net/ethernet/microsoft/mana/gdma_main.c | 29 +++
drivers/net/ethernet/microsoft/mana/mana_en.c | 175 ++++++++++++++++++
.../ethernet/microsoft/mana/mana_ethtool.c | 167 ++++++++++++++++-
include/net/mana/gdma.h | 24 ++-
include/net/mana/mana.h | 54 ++++++
6 files changed, 441 insertions(+), 9 deletions(-)
diff --git a/drivers/net/ethernet/microsoft/Kconfig b/drivers/net/ethernet/microsoft/Kconfig
index 3f36ee6a8ece..e9be18c92ca5 100644
--- a/drivers/net/ethernet/microsoft/Kconfig
+++ b/drivers/net/ethernet/microsoft/Kconfig
@@ -21,6 +21,7 @@ config MICROSOFT_MANA
depends on X86_64 || (ARM64 && !CPU_BIG_ENDIAN)
depends on PCI_HYPERV
select AUXILIARY_BUS
+ select DIMLIB
select PAGE_POOL
select NET_SHAPER
help
diff --git a/drivers/net/ethernet/microsoft/mana/gdma_main.c b/drivers/net/ethernet/microsoft/mana/gdma_main.c
index e8b7ffb47eb9..aef3b77229c1 100644
--- a/drivers/net/ethernet/microsoft/mana/gdma_main.c
+++ b/drivers/net/ethernet/microsoft/mana/gdma_main.c
@@ -1,6 +1,7 @@
// SPDX-License-Identifier: GPL-2.0 OR BSD-3-Clause
/* Copyright (c) 2021, Microsoft Corporation. */
+#include <linux/bitfield.h>
#include <linux/debugfs.h>
#include <linux/module.h>
#include <linux/pci.h>
@@ -466,6 +467,7 @@ static int mana_gd_disable_queue(struct gdma_queue *queue)
#define DOORBELL_OFFSET_RQ 0x400
#define DOORBELL_OFFSET_CQ 0x800
#define DOORBELL_OFFSET_EQ 0xFF8
+#define DOORBELL_OFFSET_DIM 0x820
static void mana_gd_ring_doorbell(struct gdma_context *gc, u32 db_index,
enum gdma_queue_type q_type, u32 qid,
@@ -506,6 +508,16 @@ static void mana_gd_ring_doorbell(struct gdma_context *gc, u32 db_index,
addr += DOORBELL_OFFSET_SQ;
break;
+ case GDMA_DIM:
+ e.dim.id = qid;
+ e.dim.mod_usec = FIELD_GET(MANA_INTR_MODR_USEC_MAX, tail_ptr);
+ e.dim.mod_usec_vld = !!(tail_ptr & MANA_INTR_MODR_USEC_VLD);
+ e.dim.mod_comps = FIELD_GET(MANA_INTR_MODR_COMP_MASK, tail_ptr);
+ e.dim.mod_comps_vld = num_req;
+
+ addr += DOORBELL_OFFSET_DIM;
+ break;
+
default:
WARN_ON(1);
return;
@@ -540,6 +552,23 @@ void mana_gd_ring_cq(struct gdma_queue *cq, u8 arm_bit)
}
EXPORT_SYMBOL_NS(mana_gd_ring_cq, "NET_MANA");
+void mana_gd_ring_dim(struct gdma_queue *cq, u32 mod_usec, bool mod_usec_vld,
+ u32 mod_comps, bool mod_comps_vld)
+{
+ struct gdma_context *gc = cq->gdma_dev->gdma_context;
+ u32 dim_val;
+
+ /* Convert the DIM values to doorbell parameters */
+ dim_val = FIELD_PREP(MANA_INTR_MODR_USEC_MAX, mod_usec) |
+ FIELD_PREP(MANA_INTR_MODR_COMP_MASK, mod_comps);
+ if (mod_usec_vld)
+ dim_val |= MANA_INTR_MODR_USEC_VLD;
+
+ mana_gd_ring_doorbell(gc, cq->gdma_dev->doorbell, GDMA_DIM, cq->id,
+ dim_val, mod_comps_vld);
+}
+EXPORT_SYMBOL_NS(mana_gd_ring_dim, "NET_MANA");
+
#define MANA_SERVICE_PERIOD 10
static void mana_serv_rescan(struct pci_dev *pdev)
diff --git a/drivers/net/ethernet/microsoft/mana/mana_en.c b/drivers/net/ethernet/microsoft/mana/mana_en.c
index 7438ea6b3f26..5ce0b96c50f6 100644
--- a/drivers/net/ethernet/microsoft/mana/mana_en.c
+++ b/drivers/net/ethernet/microsoft/mana/mana_en.c
@@ -1591,6 +1591,15 @@ int mana_create_wq_obj(struct mana_port_context *apc,
mana_gd_init_req_hdr(&req.hdr, MANA_CREATE_WQ_OBJ,
sizeof(req), sizeof(resp));
+
+ /* Our driver uses different message versions for request and
+ * response in this case.
+ * Our firmware is forward compatible with newer message versions, so
+ * the old firmware still properly handles this message, just the new
+ * feature fields are ignored, and queue creation will be successful.
+ */
+ req.hdr.req.msg_version = GDMA_MESSAGE_V3;
+ req.hdr.resp.msg_version = GDMA_MESSAGE_V2;
req.vport = vport;
req.wq_type = wq_type;
req.wq_gdma_region = wq_spec->gdma_region;
@@ -1599,6 +1608,9 @@ int mana_create_wq_obj(struct mana_port_context *apc,
req.cq_size = cq_spec->queue_size;
req.cq_moderation_ctx_id = cq_spec->modr_ctx_id;
req.cq_parent_qid = cq_spec->attached_eq;
+ req.req_cq_moderation = cq_spec->req_cq_moderation;
+ req.cq_moderation_comp = cq_spec->cq_moderation_comp;
+ req.cq_moderation_usec = cq_spec->cq_moderation_usec;
err = mana_send_request(apc->ac, &req, sizeof(req), &resp,
sizeof(resp));
@@ -1856,6 +1868,7 @@ static void mana_poll_tx_cq(struct mana_cq *cq)
struct gdma_posted_wqe_info *wqe_info;
unsigned int pkt_transmitted = 0;
unsigned int wqe_unit_cnt = 0;
+ unsigned int tx_bytes = 0;
struct mana_txq *txq = cq->txq;
struct mana_port_context *apc;
struct netdev_queue *net_txq;
@@ -1937,6 +1950,8 @@ static void mana_poll_tx_cq(struct mana_cq *cq)
mana_unmap_skb(skb, apc);
+ tx_bytes += skb->len;
+
napi_consume_skb(skb, cq->budget);
pkt_transmitted++;
@@ -1967,6 +1982,10 @@ static void mana_poll_tx_cq(struct mana_cq *cq)
if (atomic_sub_return(pkt_transmitted, &txq->pending_sends) < 0)
WARN_ON_ONCE(1);
+ /* Feed DIM with the completion rate observed here, in NAPI context. */
+ cq->tx_dim_pkts += pkt_transmitted;
+ cq->tx_dim_bytes += tx_bytes;
+
cq->work_done = pkt_transmitted;
}
@@ -2318,6 +2337,117 @@ static void mana_poll_rx_cq(struct mana_cq *cq)
xdp_do_flush();
}
+static void mana_rx_dim_work(struct work_struct *work)
+{
+ struct dim *dim = container_of(work, struct dim, work);
+ struct dim_cq_moder cur_moder;
+ struct mana_cq *cq;
+
+ cur_moder = net_dim_get_rx_moderation(dim->mode, dim->profile_ix);
+ cq = container_of(dim, struct mana_cq, dim);
+
+ cur_moder.usec = min_t(u16, cur_moder.usec, MANA_INTR_MODR_USEC_MAX);
+ cur_moder.pkts = min_t(u16, cur_moder.pkts, MANA_INTR_MODR_COMP_MAX);
+
+ mana_gd_ring_dim(cq->gdma_cq, cur_moder.usec, true,
+ cur_moder.pkts, true);
+
+ dim->state = DIM_START_MEASURE;
+}
+
+static void mana_tx_dim_work(struct work_struct *work)
+{
+ struct dim *dim = container_of(work, struct dim, work);
+ struct dim_cq_moder cur_moder;
+ struct mana_cq *cq;
+
+ cur_moder = net_dim_get_tx_moderation(dim->mode, dim->profile_ix);
+ cq = container_of(dim, struct mana_cq, dim);
+
+ cur_moder.usec = min_t(u16, cur_moder.usec, MANA_INTR_MODR_USEC_MAX);
+ cur_moder.pkts = min_t(u16, cur_moder.pkts, MANA_INTR_MODR_COMP_MAX);
+
+ mana_gd_ring_dim(cq->gdma_cq, cur_moder.usec, true,
+ cur_moder.pkts, true);
+
+ dim->state = DIM_START_MEASURE;
+}
+
+/* The caller must update apc->rx/tx_dim_enabled before disabling and
+ * after enabling. And synchronize_net() before draining the DIM work,
+ * so that NAPI cannot observe a stale flag.
+ */
+void mana_dim_change(struct mana_cq *cq, bool enable)
+{
+ bool is_rx = cq->type == MANA_CQ_TYPE_RX;
+ struct mana_port_context *apc;
+ work_func_t work_func;
+ u32 usec, comp;
+
+ if (is_rx) {
+ apc = netdev_priv(cq->rxq->ndev);
+ usec = apc->intr_modr_rx_usec;
+ comp = apc->intr_modr_rx_comp;
+ work_func = mana_rx_dim_work;
+ } else {
+ apc = netdev_priv(cq->txq->ndev);
+ usec = apc->intr_modr_tx_usec;
+ comp = apc->intr_modr_tx_comp;
+ work_func = mana_tx_dim_work;
+ }
+
+ /* On enable, zero the DIM state so net_dim() starts measuring from
+ * scratch.
+ * On disable, drain any pending DIM work and restore the static
+ * moderation values.
+ */
+ if (enable) {
+ memset(&cq->dim, 0, sizeof(cq->dim));
+ cq->dim.mode = DIM_CQ_PERIOD_MODE_START_FROM_EQE;
+ INIT_WORK(&cq->dim.work, work_func);
+ } else {
+ cancel_work_sync(&cq->dim.work);
+ mana_gd_ring_dim(cq->gdma_cq, usec, true, comp, true);
+ }
+}
+
+static void mana_update_rx_dim(struct mana_cq *cq)
+{
+ struct mana_port_context *apc = netdev_priv(cq->rxq->ndev);
+ struct dim_sample dim_sample = {};
+ struct mana_rxq *rxq = cq->rxq;
+
+ /* Pairs with smp_store_release() in mana_set_coalesce(): observing the
+ * enable flag set guarantees the DIM (re)initialization is visible.
+ */
+ if (!smp_load_acquire(&apc->rx_dim_enabled))
+ return;
+
+ dim_update_sample(READ_ONCE(cq->dim_event_ctr), rxq->stats.packets,
+ rxq->stats.bytes, &dim_sample);
+ net_dim(&cq->dim, &dim_sample);
+}
+
+static void mana_update_tx_dim(struct mana_cq *cq)
+{
+ struct mana_port_context *apc = netdev_priv(cq->txq->ndev);
+ struct dim_sample dim_sample = {};
+
+ /* Pairs with smp_store_release() in mana_set_coalesce(): observing the
+ * enable flag set guarantees the DIM (re)initialization is visible.
+ */
+ if (!smp_load_acquire(&apc->tx_dim_enabled))
+ return;
+
+ /* cq->tx_dim_pkts/bytes are accumulated in mana_poll_tx_cq(), in the
+ * same NAPI context as this read, so they track the hardware
+ * completion rate and need no u64_stats_sync protection.
+ */
+ dim_update_sample(READ_ONCE(cq->dim_event_ctr), cq->tx_dim_pkts,
+ cq->tx_dim_bytes, &dim_sample);
+ net_dim(&cq->dim, &dim_sample);
+}
+
static int mana_cq_handler(void *context, struct gdma_queue *gdma_queue)
{
struct mana_cq *cq = context;
@@ -2336,6 +2466,15 @@ static int mana_cq_handler(void *context, struct gdma_queue *gdma_queue)
if (w < cq->budget) {
mana_gd_ring_cq(gdma_queue, SET_ARM_BIT);
cq->work_done_since_doorbell = 0;
+
+ /* Update DIM before napi_complete_done() to prevent running
+ * net_dim() concurrently.
+ */
+ if (cq->type == MANA_CQ_TYPE_RX)
+ mana_update_rx_dim(cq);
+ else
+ mana_update_tx_dim(cq);
+
napi_complete_done(&cq->napi, w);
} else if (cq->work_done_since_doorbell >=
(cq->gdma_cq->queue_size / COMP_ENTRY_SIZE) * 4) {
@@ -2368,6 +2507,7 @@ static void mana_schedule_napi(void *context, struct gdma_queue *gdma_queue)
{
struct mana_cq *cq = context;
+ WRITE_ONCE(cq->dim_event_ctr, cq->dim_event_ctr + 1);
napi_schedule_irqoff(&cq->napi);
}
@@ -2410,6 +2550,7 @@ static void mana_destroy_txq(struct mana_port_context *apc)
if (apc->tx_qp[i]->txq.napi_initialized) {
napi_synchronize(napi);
napi_disable_locked(napi);
+ cancel_work_sync(&apc->tx_qp[i]->tx_cq.dim.work);
netif_napi_del_locked(napi);
apc->tx_qp[i]->txq.napi_initialized = false;
}
@@ -2543,6 +2684,11 @@ static int mana_create_txq(struct mana_port_context *apc,
cq_spec.modr_ctx_id = 0;
cq_spec.attached_eq = cq->gdma_cq->cq.parent->id;
+ /* DIM setting can be changed at runtime */
+ cq_spec.req_cq_moderation = true;
+ cq_spec.cq_moderation_usec = apc->intr_modr_tx_usec;
+ cq_spec.cq_moderation_comp = apc->intr_modr_tx_comp;
+
err = mana_create_wq_obj(apc, apc->port_handle, GDMA_SQ,
&wq_spec, &cq_spec,
&apc->tx_qp[i]->tx_object);
@@ -2573,6 +2719,13 @@ static int mana_create_txq(struct mana_port_context *apc,
set_bit(NAPI_STATE_NO_BUSY_POLL, &cq->napi.state);
netif_napi_add_locked(net, &cq->napi, mana_poll);
+
+ /* Initialize the DIM work before enabling NAPI, so that a poll
+ * cannot reach net_dim() with an uninitialized cq->dim.work.
+ */
+ INIT_WORK(&cq->dim.work, mana_tx_dim_work);
+ cq->dim.mode = DIM_CQ_PERIOD_MODE_START_FROM_EQE;
+
napi_enable_locked(&cq->napi);
txq->napi_initialized = true;
@@ -2610,6 +2763,7 @@ static void mana_destroy_rxq(struct mana_port_context *apc,
napi_synchronize(napi);
napi_disable_locked(napi);
+ cancel_work_sync(&rxq->rx_cq.dim.work);
netif_napi_del_locked(napi);
}
@@ -2848,6 +3002,11 @@ static struct mana_rxq *mana_create_rxq(struct mana_port_context *apc,
cq_spec.modr_ctx_id = 0;
cq_spec.attached_eq = cq->gdma_cq->cq.parent->id;
+ /* DIM setting can be changed at runtime */
+ cq_spec.req_cq_moderation = true;
+ cq_spec.cq_moderation_usec = apc->intr_modr_rx_usec;
+ cq_spec.cq_moderation_comp = apc->intr_modr_rx_comp;
+
err = mana_create_wq_obj(apc, apc->port_handle, GDMA_RQ,
&wq_spec, &cq_spec, &rxq->rxobj);
if (err)
@@ -2880,6 +3039,12 @@ static struct mana_rxq *mana_create_rxq(struct mana_port_context *apc,
WARN_ON(xdp_rxq_info_reg_mem_model(&rxq->xdp_rxq, MEM_TYPE_PAGE_POOL,
rxq->page_pool));
+ /* Initialize the DIM work before enabling NAPI, so that a poll
+ * cannot reach net_dim() with an uninitialized cq->dim.work.
+ */
+ INIT_WORK(&cq->dim.work, mana_rx_dim_work);
+ cq->dim.mode = DIM_CQ_PERIOD_MODE_START_FROM_EQE;
+
napi_enable_locked(&cq->napi);
mana_gd_ring_cq(cq->gdma_cq, SET_ARM_BIT);
@@ -3546,6 +3711,16 @@ static int mana_probe_port(struct mana_context *ac, int port_idx,
apc->link_cfg_error = 1;
apc->cqe_coalescing_enable = 0;
+ /* Initialize interrupt moderation settings if supported by HW */
+ if (gc->pf_cap_flags1 & GDMA_PF_CAP_FLAG_1_DYN_INTERRUPT_MODERATION) {
+ apc->intr_modr_rx_usec = MANA_INTR_MODR_USEC_DEF;
+ apc->intr_modr_rx_comp = MANA_INTR_MODR_COMP_DEF;
+ apc->intr_modr_tx_usec = MANA_INTR_MODR_USEC_DEF;
+ apc->intr_modr_tx_comp = MANA_INTR_MODR_COMP_DEF;
+ apc->rx_dim_enabled = MANA_ADAPTIVE_RX_DEF;
+ apc->tx_dim_enabled = MANA_ADAPTIVE_TX_DEF;
+ }
+
mutex_init(&apc->vport_mutex);
apc->vport_use_count = 0;
diff --git a/drivers/net/ethernet/microsoft/mana/mana_ethtool.c b/drivers/net/ethernet/microsoft/mana/mana_ethtool.c
index 881df597d7f9..9e31e2595ae3 100644
--- a/drivers/net/ethernet/microsoft/mana/mana_ethtool.c
+++ b/drivers/net/ethernet/microsoft/mana/mana_ethtool.c
@@ -419,6 +419,15 @@ static int mana_get_coalesce(struct net_device *ndev,
!kernel_coal->rx_cqe_nsecs)
kernel_coal->rx_cqe_nsecs = MANA_RX_CQE_NSEC_DEF;
+ ec->rx_coalesce_usecs = apc->intr_modr_rx_usec;
+ ec->rx_max_coalesced_frames = apc->intr_modr_rx_comp;
+
+ ec->tx_coalesce_usecs = apc->intr_modr_tx_usec;
+ ec->tx_max_coalesced_frames = apc->intr_modr_tx_comp;
+
+ ec->use_adaptive_rx_coalesce = apc->rx_dim_enabled;
+ ec->use_adaptive_tx_coalesce = apc->tx_dim_enabled;
+
return 0;
}
@@ -428,9 +437,34 @@ static int mana_set_coalesce(struct net_device *ndev,
struct netlink_ext_ack *extack)
{
struct mana_port_context *apc = netdev_priv(ndev);
- u8 saved_cqe_coalescing_enable;
+ struct {
+ u16 intr_modr_rx_usec;
+ u16 intr_modr_rx_comp;
+ u16 intr_modr_tx_usec;
+ u16 intr_modr_tx_comp;
+ u8 cqe_coalescing_enable;
+ bool rx_dim_enabled;
+ bool tx_dim_enabled;
+ } saved;
+ bool modr_changed = false;
+ bool dim_changed = false;
+ struct gdma_context *gc;
int err;
+ gc = apc->ac->gdma_dev->gdma_context;
+
+ /* Both static and dynamic interrupt moderation (DIM) rely on the
+ * same HW capability advertised by the PF.
+ */
+ if ((ec->use_adaptive_rx_coalesce || ec->use_adaptive_tx_coalesce ||
+ ec->rx_coalesce_usecs || ec->tx_coalesce_usecs ||
+ ec->rx_max_coalesced_frames || ec->tx_max_coalesced_frames) &&
+ !(gc->pf_cap_flags1 & GDMA_PF_CAP_FLAG_1_DYN_INTERRUPT_MODERATION)) {
+ NL_SET_ERR_MSG(extack,
+ "Interrupt Moderation is not supported by HW");
+ return -EOPNOTSUPP;
+ }
+
if (kernel_coal->rx_cqe_frames != 1 &&
kernel_coal->rx_cqe_frames != MANA_RXCOMP_OOB_NUM_PPI) {
NL_SET_ERR_MSG_FMT(extack,
@@ -440,18 +474,129 @@ static int mana_set_coalesce(struct net_device *ndev,
return -EINVAL;
}
- saved_cqe_coalescing_enable = apc->cqe_coalescing_enable;
+ if (ec->rx_coalesce_usecs > MANA_INTR_MODR_USEC_MAX ||
+ ec->tx_coalesce_usecs > MANA_INTR_MODR_USEC_MAX) {
+ NL_SET_ERR_MSG_FMT(extack,
+ "coalesce usecs must be <= %lu",
+ MANA_INTR_MODR_USEC_MAX);
+ return -EINVAL;
+ }
+
+ if (ec->rx_max_coalesced_frames > MANA_INTR_MODR_COMP_MAX ||
+ ec->tx_max_coalesced_frames > MANA_INTR_MODR_COMP_MAX) {
+ NL_SET_ERR_MSG_FMT(extack,
+ "coalesce frames must be <= %lu",
+ MANA_INTR_MODR_COMP_MAX);
+ return -EINVAL;
+ }
+
+ if (ec->rx_coalesce_usecs != apc->intr_modr_rx_usec ||
+ ec->rx_max_coalesced_frames != apc->intr_modr_rx_comp ||
+ ec->tx_coalesce_usecs != apc->intr_modr_tx_usec ||
+ ec->tx_max_coalesced_frames != apc->intr_modr_tx_comp)
+ modr_changed = true;
+
+ saved.intr_modr_rx_usec = apc->intr_modr_rx_usec;
+ saved.intr_modr_rx_comp = apc->intr_modr_rx_comp;
+ saved.intr_modr_tx_usec = apc->intr_modr_tx_usec;
+ saved.intr_modr_tx_comp = apc->intr_modr_tx_comp;
+
+ apc->intr_modr_rx_usec = ec->rx_coalesce_usecs;
+ apc->intr_modr_rx_comp = ec->rx_max_coalesced_frames;
+ apc->intr_modr_tx_usec = ec->tx_coalesce_usecs;
+ apc->intr_modr_tx_comp = ec->tx_max_coalesced_frames;
+
+ if (!!ec->use_adaptive_rx_coalesce != apc->rx_dim_enabled ||
+ !!ec->use_adaptive_tx_coalesce != apc->tx_dim_enabled)
+ dim_changed = true;
+
+ saved.rx_dim_enabled = apc->rx_dim_enabled;
+ saved.tx_dim_enabled = apc->tx_dim_enabled;
+
+ saved.cqe_coalescing_enable = apc->cqe_coalescing_enable;
apc->cqe_coalescing_enable =
kernel_coal->rx_cqe_frames == MANA_RXCOMP_OOB_NUM_PPI;
- if (!apc->port_is_up)
+ if (!apc->port_is_up) {
+ WRITE_ONCE(apc->rx_dim_enabled, !!ec->use_adaptive_rx_coalesce);
+ WRITE_ONCE(apc->tx_dim_enabled, !!ec->use_adaptive_tx_coalesce);
return 0;
+ }
- err = mana_config_rss(apc, TRI_STATE_TRUE, false, false);
- if (err)
- apc->cqe_coalescing_enable = saved_cqe_coalescing_enable;
+ if (apc->cqe_coalescing_enable != saved.cqe_coalescing_enable) {
+ /* CQE coalescing setting is applied via RSS configuration. */
+ err = mana_config_rss(apc, TRI_STATE_TRUE, false, false);
+ if (err) {
+ netdev_err(ndev, "Change CQE coalescing failed: %d\n",
+ err);
+ apc->cqe_coalescing_enable =
+ saved.cqe_coalescing_enable;
+ apc->intr_modr_rx_usec = saved.intr_modr_rx_usec;
+ apc->intr_modr_rx_comp = saved.intr_modr_rx_comp;
+ apc->intr_modr_tx_usec = saved.intr_modr_tx_usec;
+ apc->intr_modr_tx_comp = saved.intr_modr_tx_comp;
+ return err;
+ }
+ }
- return err;
+ if (modr_changed || dim_changed) {
+ bool new_rx_dim = !!ec->use_adaptive_rx_coalesce;
+ bool new_tx_dim = !!ec->use_adaptive_tx_coalesce;
+ bool disable_rx_dim = saved.rx_dim_enabled && !new_rx_dim;
+ bool disable_tx_dim = saved.tx_dim_enabled && !new_tx_dim;
+ bool enable_rx_dim = !saved.rx_dim_enabled && new_rx_dim;
+ bool enable_tx_dim = !saved.tx_dim_enabled && new_tx_dim;
+ int q;
+
+ /* On disable: clear the per-port flag first and
+ * synchronize_net() so any in-flight NAPI poll observes
+ * the new value and will not schedule further DIM work;
+ * then drain pending work and restore the static
+ * moderation values.
+ */
+ if (disable_rx_dim)
+ WRITE_ONCE(apc->rx_dim_enabled, false);
+ if (disable_tx_dim)
+ WRITE_ONCE(apc->tx_dim_enabled, false);
+ if (disable_rx_dim || disable_tx_dim)
+ synchronize_net();
+
+ for (q = 0; q < apc->num_queues; q++) {
+ struct mana_cq *rx_cq = &apc->rxqs[q]->rx_cq;
+ struct mana_cq *tx_cq = &apc->tx_qp[q]->tx_cq;
+
+ if (disable_rx_dim)
+ mana_dim_change(rx_cq, false);
+ else if (enable_rx_dim)
+ mana_dim_change(rx_cq, true);
+ else if (!new_rx_dim && modr_changed)
+ mana_gd_ring_dim(rx_cq->gdma_cq,
+ apc->intr_modr_rx_usec, true,
+ apc->intr_modr_rx_comp, true);
+
+ if (disable_tx_dim)
+ mana_dim_change(tx_cq, false);
+ else if (enable_tx_dim)
+ mana_dim_change(tx_cq, true);
+ else if (!new_tx_dim && modr_changed)
+ mana_gd_ring_dim(tx_cq->gdma_cq,
+ apc->intr_modr_tx_usec, true,
+ apc->intr_modr_tx_comp, true);
+ }
+
+ /* Publish the enable flag with release semantics so a
+ * concurrent NAPI poll that observes it set also sees the DIM
+ * (re)init done by mana_dim_change() above.
+ */
+ if (enable_rx_dim)
+ /* pairs with smp_load_acquire() in mana_update_rx_dim() */
+ smp_store_release(&apc->rx_dim_enabled, true);
+ if (enable_tx_dim)
+ /* pairs with smp_load_acquire() in mana_update_tx_dim() */
+ smp_store_release(&apc->tx_dim_enabled, true);
+ }
+
+ return 0;
}
/* mana_set_channels - change the number of queues on a port
@@ -595,7 +740,13 @@ static int mana_get_link_ksettings(struct net_device *ndev,
}
const struct ethtool_ops mana_ethtool_ops = {
- .supported_coalesce_params = ETHTOOL_COALESCE_RX_CQE_FRAMES,
+ .supported_coalesce_params = ETHTOOL_COALESCE_RX_CQE_FRAMES |
+ ETHTOOL_COALESCE_RX_USECS |
+ ETHTOOL_COALESCE_RX_MAX_FRAMES |
+ ETHTOOL_COALESCE_TX_USECS |
+ ETHTOOL_COALESCE_TX_MAX_FRAMES |
+ ETHTOOL_COALESCE_USE_ADAPTIVE_RX |
+ ETHTOOL_COALESCE_USE_ADAPTIVE_TX,
.op_needs_rtnl = ETHTOOL_OP_NEEDS_RTNL_SCHANNELS |
ETHTOOL_OP_NEEDS_RTNL_SRINGPARAM |
ETHTOOL_OP_NEEDS_RTNL_GLINK,
diff --git a/include/net/mana/gdma.h b/include/net/mana/gdma.h
index 0c395917b214..8529cef0d7c4 100644
--- a/include/net/mana/gdma.h
+++ b/include/net/mana/gdma.h
@@ -47,6 +47,7 @@ enum gdma_queue_type {
GDMA_RQ,
GDMA_CQ,
GDMA_EQ,
+ GDMA_DIM,
};
enum gdma_work_request_flags {
@@ -126,6 +127,17 @@ union gdma_doorbell_entry {
u64 tail_ptr : 31;
u64 arm : 1;
} eq;
+
+ struct {
+ u64 id : 24;
+ u64 reserved : 8;
+ u64 mod_usec : 10;
+ u64 reserve1 : 5;
+ u64 mod_usec_vld : 1;
+ u64 mod_comps : 8;
+ u64 reserve2 : 7;
+ u64 mod_comps_vld: 1;
+ } dim;
}; /* HW DATA */
struct gdma_msg_hdr {
@@ -502,6 +514,9 @@ void mana_gd_ring_cq(struct gdma_queue *cq, u8 arm_bit);
int mana_schedule_serv_work(struct gdma_context *gc, enum gdma_eqe_type type);
+void mana_gd_ring_dim(struct gdma_queue *cq, u32 mod_usec, bool mod_usec_vld,
+ u32 mod_comps, bool mod_comps_vld);
+
struct gdma_wqe {
u32 reserved :24;
u32 last_vbytes :8;
@@ -650,6 +665,9 @@ enum {
/* Driver supports self recovery on Hardware Channel timeouts */
#define GDMA_DRV_CAP_FLAG_1_HWC_TIMEOUT_RECOVERY BIT(25)
+/* Driver supports dynamic interrupt moderation - DIM */
+#define GDMA_DRV_CAP_FLAG_1_DYN_INTERRUPT_MODERATION BIT(28)
+
#define GDMA_DRV_CAP_FLAGS1 \
(GDMA_DRV_CAP_FLAG_1_EQ_SHARING_MULTI_VPORT | \
GDMA_DRV_CAP_FLAG_1_NAPI_WKDONE_FIX | \
@@ -665,7 +683,8 @@ enum {
GDMA_DRV_CAP_FLAG_1_PROBE_RECOVERY | \
GDMA_DRV_CAP_FLAG_1_HANDLE_STALL_SQ_RECOVERY | \
GDMA_DRV_CAP_FLAG_1_HWC_TIMEOUT_RECOVERY | \
- GDMA_DRV_CAP_FLAG_1_EQ_MSI_UNSHARE_MULTI_VPORT)
+ GDMA_DRV_CAP_FLAG_1_EQ_MSI_UNSHARE_MULTI_VPORT | \
+ GDMA_DRV_CAP_FLAG_1_DYN_INTERRUPT_MODERATION)
#define GDMA_DRV_CAP_FLAGS2 0
@@ -701,6 +720,9 @@ struct gdma_verify_ver_req {
u8 os_ver_str4[128];
}; /* HW DATA */
+/* HW supports dynamic interrupt moderation - DIM */
+#define GDMA_PF_CAP_FLAG_1_DYN_INTERRUPT_MODERATION BIT(15)
+
struct gdma_verify_ver_resp {
struct gdma_resp_hdr hdr;
u64 gdma_protocol_ver;
diff --git a/include/net/mana/mana.h b/include/net/mana/mana.h
index 13c87baf018e..48f4445aa87a 100644
--- a/include/net/mana/mana.h
+++ b/include/net/mana/mana.h
@@ -4,6 +4,7 @@
#ifndef _MANA_H
#define _MANA_H
+#include <linux/dim.h>
#include <net/xdp.h>
#include <net/net_shaper.h>
@@ -64,6 +65,19 @@ enum TRI_STATE {
/* Maximum number of packets per coalesced CQE */
#define MANA_RXCOMP_OOB_NUM_PPI 4
+/* Default/max interrupt moderation settings */
+#define MANA_INTR_MODR_USEC_DEF 0
+#define MANA_INTR_MODR_COMP_DEF 0
+
+#define MANA_ADAPTIVE_RX_DEF true
+#define MANA_ADAPTIVE_TX_DEF true
+
+/* DIM doorbell value field layout */
+#define MANA_INTR_MODR_USEC_MAX GENMASK(9, 0)
+#define MANA_INTR_MODR_USEC_VLD BIT(15)
+#define MANA_INTR_MODR_COMP_MAX GENMASK(7, 0)
+#define MANA_INTR_MODR_COMP_MASK GENMASK(23, 16)
+
/* Update this count whenever the respective structures are changed */
#define MANA_STATS_RX_COUNT (6 + MANA_RXCOMP_OOB_NUM_PPI - 1)
#define MANA_STATS_TX_COUNT 11
@@ -297,6 +311,17 @@ struct mana_cq {
int work_done;
int work_done_since_doorbell;
int budget;
+
+ /* DIM - Dynamic Interrupt Moderation */
+ struct dim dim;
+ u16 dim_event_ctr;
+
+ /* Cumulative TX completions fed to DIM. Updated and read only in
+ * NAPI context (mana_poll_tx_cq() / mana_update_tx_dim()), so they
+ * measure the hardware completion rate and need no u64_stats_sync.
+ */
+ u64 tx_dim_pkts;
+ u64 tx_dim_bytes;
};
struct mana_recv_buf_oob {
@@ -573,6 +598,15 @@ struct mana_port_context {
u8 cqe_coalescing_enable;
u32 cqe_coalescing_timeout_ns;
+ /* Interrupt moderation settings */
+ u16 intr_modr_rx_usec;
+ u16 intr_modr_rx_comp;
+ u16 intr_modr_tx_usec;
+ u16 intr_modr_tx_comp;
+
+ bool rx_dim_enabled;
+ bool tx_dim_enabled;
+
struct mana_ethtool_stats eth_stats;
struct mana_ethtool_phy_stats phy_stats;
@@ -598,6 +632,8 @@ int mana_alloc_queues(struct net_device *ndev);
int mana_attach(struct net_device *ndev);
int mana_detach(struct net_device *ndev, bool from_close);
+void mana_dim_change(struct mana_cq *cq, bool enable);
+
int mana_probe(struct gdma_dev *gd, bool resuming);
void mana_remove(struct gdma_dev *gd, bool suspending);
@@ -633,6 +669,9 @@ struct mana_obj_spec {
u32 queue_size;
u32 attached_eq;
u32 modr_ctx_id;
+ u8 req_cq_moderation;
+ u16 cq_moderation_comp;
+ u16 cq_moderation_usec;
};
enum mana_command_code {
@@ -764,6 +803,15 @@ struct mana_create_wqobj_req {
u32 cq_size;
u32 cq_moderation_ctx_id;
u32 cq_parent_qid;
+
+ /* V2 */
+ u8 allow_rqwqe_chain;
+
+ /* V3 */
+ u8 req_cq_moderation;
+ u16 cq_moderation_comp;
+ u16 cq_moderation_usec;
+ u8 reserved2[2];
}; /* HW DATA */
struct mana_create_wqobj_resp {
@@ -771,6 +819,12 @@ struct mana_create_wqobj_resp {
u32 wq_id;
u32 cq_id;
mana_handle_t wq_obj;
+
+ /* V2 */
+ u16 cq_moderation_comp;
+ u16 cq_moderation_usec;
+ u8 cq_moderation_enabled;
+ u8 reserved1[3];
}; /* HW DATA */
/* Destroy WQ Object */
--
2.34.1
^ permalink raw reply related
* RE: [EXTERNAL] Re: [PATCH net-next v4] net: mana: Add Interrupt Moderation support
From: Haiyang Zhang @ 2026-07-02 19:02 UTC (permalink / raw)
To: Paolo Abeni, Haiyang Zhang, linux-hyperv@vger.kernel.org,
netdev@vger.kernel.org, KY Srinivasan, Wei Liu, Dexuan Cui,
Long Li, Andrew Lunn, David S. Miller, Eric Dumazet,
Jakub Kicinski, Konstantin Taranov, Simon Horman,
Erni Sri Satya Vennela, Dipayaan Roy, Aditya Garg, Breno Leitao,
linux-kernel@vger.kernel.org, linux-rdma@vger.kernel.org
Cc: Paul Rosswurm
In-Reply-To: <8906f758-27fe-4ea8-8558-6d15089372d1@redhat.com>
> -----Original Message-----
> From: Paolo Abeni <pabeni@redhat.com>
> Sent: Thursday, July 2, 2026 4:57 AM
> To: Haiyang Zhang <haiyangz@linux.microsoft.com>; linux-
> hyperv@vger.kernel.org; netdev@vger.kernel.org; KY Srinivasan
> <kys@microsoft.com>; Haiyang Zhang <haiyangz@microsoft.com>; Wei Liu
> <wei.liu@kernel.org>; Dexuan Cui <DECUI@microsoft.com>; Long Li
> <longli@microsoft.com>; Andrew Lunn <andrew+netdev@lunn.ch>; David S.
> Miller <davem@davemloft.net>; Eric Dumazet <edumazet@google.com>; Jakub
> Kicinski <kuba@kernel.org>; Konstantin Taranov <kotaranov@microsoft.com>;
> Simon Horman <horms@kernel.org>; Erni Sri Satya Vennela
> <ernis@linux.microsoft.com>; Dipayaan Roy
> <dipayanroy@linux.microsoft.com>; Aditya Garg
> <gargaditya@linux.microsoft.com>; Breno Leitao <leitao@debian.org>; linux-
> kernel@vger.kernel.org; linux-rdma@vger.kernel.org
> Cc: Paul Rosswurm <paulros@microsoft.com>
> Subject: [EXTERNAL] Re: [PATCH net-next v4] net: mana: Add Interrupt
> Moderation support
>
> On 6/29/26 11:36 PM, Haiyang Zhang wrote:
> > diff --git a/drivers/net/ethernet/microsoft/mana/mana_en.c
> b/drivers/net/ethernet/microsoft/mana/mana_en.c
> > index 7438ea6b3f26..9391e9564605 100644
> > --- a/drivers/net/ethernet/microsoft/mana/mana_en.c
> > +++ b/drivers/net/ethernet/microsoft/mana/mana_en.c
> > @@ -1591,6 +1591,9 @@ int mana_create_wq_obj(struct mana_port_context
> *apc,
> >
> > mana_gd_init_req_hdr(&req.hdr, MANA_CREATE_WQ_OBJ,
> > sizeof(req), sizeof(resp));
> > +
> > + req.hdr.req.msg_version = GDMA_MESSAGE_V3;
> > + req.hdr.resp.msg_version = GDMA_MESSAGE_V2;
>
> Double checking the above is intentional; it feels strange to me that
> request and reply use different versions. Possibly a comment for future
> memory would make sense.
Yes, it's intentional. The request and reply versions can be different.
I will add comments.
> > +
> > +/* The caller must update apc->rx/tx_dim_enabled before disabling and
> > + * after enabling. And synchronize_net() before draining the DIM work,
> > + * so that NAPI cannot observe a stale flag.
> > + */
> > +int mana_dim_change(struct mana_cq *cq, bool enable)
>
> This always return 0, and the return value is not checked by the
> callers; return type should likelly changed to void
Will update.
Thanks,
- Haiyang
^ permalink raw reply
* Re: [PATCH net-next v2] net: mana: Add handler for sriov configure
From: sashiko-bot @ 2026-07-02 18:02 UTC (permalink / raw)
To: Haiyang Zhang; +Cc: linux-hyperv
In-Reply-To: <20260701180116.507690-1-haiyangz@linux.microsoft.com>
Thank you for your contribution! Sashiko AI review found 2 potential issue(s) to consider:
- [High] Disabling SR-IOV without checking if Virtual Functions (VFs) are assigned to a guest VM.
- [High] Missing SR-IOV cleanup during PF driver removal and shutdown.
--
commit 3df823cc57367f69930deb6dc48172ce7d77e785
Author: Haiyang Zhang <haiyangz@microsoft.com>
net: mana: Add handler for sriov configure
Add callback function for the pci_driver / sriov_configure.
It asks the NIC to provide certain number of VFs, or disable
VFs if the request is zero.
> diff --git a/drivers/net/ethernet/microsoft/mana/gdma_main.c b/drivers/net/ethernet/microsoft/mana/gdma_main.c
> index e8b7ffb47eb98..f7a79874fd0eb 100644
> --- a/drivers/net/ethernet/microsoft/mana/gdma_main.c
> +++ b/drivers/net/ethernet/microsoft/mana/gdma_main.c
> @@ -2496,6 +2496,20 @@ static void mana_gd_shutdown(struct pci_dev *pdev)
> pci_disable_device(pdev);
> }
[Severity: High]
With the addition of SR-IOV support in this patch, do mana_gd_shutdown() and
mana_gd_remove() need to call pci_disable_sriov()?
If a user enables VFs via sysfs and then unbinds the driver or shuts down the
system, could this omission introduce a regression where orphaned, dead VF PCI
devices are left in the kernel?
Accessing these orphaned VFs might trigger use-after-free crashes or hardware
faults, as the underlying PF hardware and driver context have been destroyed.
> +static int mana_sriov_configure(struct pci_dev *pdev, int numvfs)
> +{
> + int err = 0;
> +
> + dev_info(&pdev->dev, "Requested num VFs: %d\n", numvfs);
> +
> + if (numvfs > 0)
> + err = pci_enable_sriov(pdev, numvfs);
> + else
> + pci_disable_sriov(pdev);
[Severity: High]
Should this path check if the VFs are currently assigned to a guest VM before
disabling SR-IOV?
If a privileged user writes 0 to sriov_numvfs while VFs are actively assigned
to a VM (e.g., via vfio-pci), could forcefully removing the PCI devices cause
a regression resulting in host IOMMU faults and guest kernel panics?
Would it be better to check pci_vfs_assigned(pdev) here, or use the
pci_sriov_configure_simple() helper to safely enforce this?
> +
> + return err ? err : numvfs;
> +}
--
Sashiko AI review · https://sashiko.dev/#/patchset/20260701180116.507690-1-haiyangz@linux.microsoft.com?part=1
^ permalink raw reply
page: next (older) | prev (newer) | latest
- recent:[subjects (threaded)|topics (new)|topics (active)]
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox