From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-alma10-1.taild15c8.ts.net [100.103.45.18]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 2FF0A37F8DF for ; Fri, 10 Jul 2026 22:13:12 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=100.103.45.18 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1783721593; cv=none; b=oQ7+Zcv56SxS/c4KutgpOj55HQWwp8LpbPVpGU08p/4mdto4+cYdtughBNg/70eQS0vm49kTHhFin5QhjlrjH6FfEGYmHF691DHZUi8c99gVTRDxVkMCdow5/uST/fsGE+/U1+N56VZHGcGOzkxCpSyAb/nnhoKXk6EqlL7i4H4= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1783721593; c=relaxed/simple; bh=Yq/XbQki4sZne7MCfBMcnNGN59FiiAy+rGzsaIwyekc=; h=Date:To:From:Subject:Message-Id; b=tvaxthAQZLezz488pm1QTMGk/4CGNkRjPKxFT7suXrUfmYOsn1qHGdCJymVX0T2ucXre9zqufplGeLX5BCp/AXJZRN39zk/DrehwcYn/GLKe9byE3NCaWvwpm66uqNob1GHUVjNNIVczP5VbnFUmn35N+EVicHxuEAQ/b1bn5M8= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=linux-foundation.org header.i=@linux-foundation.org header.b=cZ62iyFm; arc=none smtp.client-ip=100.103.45.18 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=linux-foundation.org header.i=@linux-foundation.org header.b="cZ62iyFm" Received: by smtp.kernel.org (Postfix) with ESMTPSA id CB1101F000E9; Fri, 10 Jul 2026 22:13:11 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux-foundation.org; s=korg; t=1783721591; bh=JDhj9KzlbG401+nA1lVdt1Z5R3RUp9kil2wdZ6vEpQY=; h=Date:To:From:Subject; b=cZ62iyFmb5DUGzsVkLrrPp38IhYxxt4VUccRc1gKQztVm/s47LlSQwusEl8U9YcD3 L4fnUzZAKh/l7LEo6j7uu4sPEz6e680pFGngUJVeYAcInTJMu+gZvLiQcnsrIUXk4P 0wMZW/b1dNTwlxN/DywoPPswNcLCkOIVCgKogoeo= Date: Fri, 10 Jul 2026 15:13:11 -0700 To: mm-commits@vger.kernel.org,skinsburskii@gmail.com,akpm@linux-foundation.org From: Andrew Morton Subject: + mm-hmm-move-page-fault-handling-out-of-walk-callbacks.patch added to mm-new branch Message-Id: <20260710221311.CB1101F000E9@smtp.kernel.org> Precedence: bulk X-Mailing-List: mm-commits@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: The patch titled Subject: mm/hmm: move page fault handling out of walk callbacks has been added to the -mm mm-new branch. Its filename is mm-hmm-move-page-fault-handling-out-of-walk-callbacks.patch This patch will shortly appear at https://git.kernel.org/pub/scm/linux/kernel/git/akpm/25-new.git/tree/patches/mm-hmm-move-page-fault-handling-out-of-walk-callbacks.patch This patch will later appear in the mm-new branch at git://git.kernel.org/pub/scm/linux/kernel/git/akpm/mm Note, mm-new is a provisional staging ground for work-in-progress patches, and acceptance into mm-new is a notification for others take notice and to finish up reviews. Please do not hesitate to respond to review feedback and post updated versions to replace or incrementally fixup patches in mm-new. The mm-new branch of mm.git is not included in linux-next If a few days of testing in mm-new is successful, the patch will me moved into mm.git's mm-unstable branch, which is included in linux-next Before you just go and hit "reply", please: a) Consider who else should be cc'ed b) Prefer to cc a suitable mailing list as well c) Ideally: find the original patch on the mailing list and do a reply-to-all to that, adding suitable additional cc's *** Remember to use Documentation/process/submit-checklist.rst when testing your code *** The -mm tree is included into linux-next via various branches at git://git.kernel.org/pub/scm/linux/kernel/git/akpm/mm and is updated there most days ------------------------------------------------------ From: Stanislav Kinsburskii Subject: mm/hmm: move page fault handling out of walk callbacks Date: Fri, 10 Jul 2026 14:26:27 -0700 Patch series "mm/hmm: Add mmap lock-drop support for userfaultfd-backed mappings", v8. This series extends the HMM framework to support userfaultfd-backed memory by allowing the mmap read lock to be dropped during hmm_range_fault(). Some page fault handlers most notably userfaultfd require the mmap lock to be released so that userspace can resolve the fault. The current HMM interface never sets FAULT_FLAG_ALLOW_RETRY, making it impossible to fault in pages from userfaultfd-registered regions. This series follows the established int *locked pattern from get_user_pages_remote() in mm/gup.c. A new helper function, hmm_range_fault_locked(), accepts an int *locked parameter. When the mmap lock is dropped during fault resolution (VM_FAULT_RETRY or VM_FAULT_COMPLETED), the function returns 0 with *locked = 0, signalling the caller to restart its walk. The existing hmm_range_fault() is refactored into a thin wrapper that passes NULL, preserving current behavior for all existing callers. Possible approaches to lift this limitation are documented in Documentation/mm/hmm.rst. This patch (of 8): hmm_range_fault() currently triggers page faults from inside the page-table walk callbacks: hmm_vma_walk_pmd(), hmm_vma_walk_pud(), hmm_vma_walk_hugetlb_entry() and the pte-level helper all call hmm_vma_fault(), which in turn calls handle_mm_fault() while the walker still holds nested locks. The pte spinlock is dropped explicitly by each caller, and the hugetlb path manually drops and retakes hugetlb_vma_lock_read around the fault to dodge a deadlock against the walk framework's unconditional unlock. This layering does not extend cleanly to fault handlers that may release mmap_lock (VM_FAULT_RETRY, VM_FAULT_COMPLETED). If the lock is dropped while walk_page_range() is mid-traversal, the VMA can be freed before the walk framework's matching hugetlb_vma_unlock_read(), turning that unlock into a use-after-free. Split the responsibilities the way get_user_pages() does. Walk callbacks become inspect-only: when they detect a range that needs to be faulted in, they record it in struct hmm_vma_walk and return a private sentinel (HMM_FAULT_PENDING). The outer loop in hmm_range_fault() then drops out of walk_page_range(), invokes a new helper hmm_do_fault() that calls handle_mm_fault() with only mmap_lock held, and restarts the walk so the now-present entries are collected into hmm_pfns. No functional change for existing callers. As a side effect the hugetlb callback no longer needs the hugetlb_vma_{un}lock_read dance, and every fault-path exit from the callbacks now releases the pte spinlock on a single, common path. This refactor is also a precursor for adding an unlockable variant of hmm_range_fault() in a follow-up patch. Link: https://lore.kernel.org/178371866223.900500.12312667138651735591.stgit@skinsburskii Link: https://lore.kernel.org/178371878774.900500.3975008123082170472.stgit@skinsburskii Signed-off-by: Stanislav Kinsburskii Reviewed-by: Jason Gunthorpe Cc: Danilo Krummrich Cc: Dave Airlie Cc: David Hildenbrand Cc: Dexuan Cui Cc: Haiyang Zhang Cc: Jonathan Corbet Cc: Kees Cook Cc: K. Y. Srinivasan Cc: Leon Romanovsky Cc: Liam R. Howlett Cc: Lizhi Hou Cc: Long Li Cc: Lorenzo Stoakes Cc: Lyude Cc: Maarten Lankhorst Cc: Maxime Ripard Cc: Michal Hocko Cc: Mike Rapoport Cc: Oded Gabbay Cc: Oleg Nesterov Cc: Shuah Khan Cc: Suren Baghdasaryan Cc: Thomas Zimemrmann Cc: Vlastimil Babka Cc: Wei Liu Signed-off-by: Andrew Morton --- mm/hmm.c | 118 +++++++++++++++++++++++++++++++++-------------------- 1 file changed, 75 insertions(+), 43 deletions(-) --- a/mm/hmm.c~mm-hmm-move-page-fault-handling-out-of-walk-callbacks +++ a/mm/hmm.c @@ -33,8 +33,17 @@ struct hmm_vma_walk { struct hmm_range *range; unsigned long last; + unsigned long end; + unsigned int required_fault; }; +/* + * Internal sentinel returned by walk callbacks when they need a page fault. + * The callback stores end/required_fault in hmm_vma_walk; the outer loop + * consumes the sentinel and never propagates it to the caller. + */ +#define HMM_FAULT_PENDING -EAGAIN + enum { HMM_NEED_FAULT = 1 << 0, HMM_NEED_WRITE_FAULT = 1 << 1, @@ -60,37 +69,25 @@ static int hmm_pfns_fill(unsigned long a } /* - * hmm_vma_fault() - fault in a range lacking valid pmd or pte(s) - * @addr: range virtual start address (inclusive) - * @end: range virtual end address (exclusive) - * @required_fault: HMM_NEED_* flags - * @walk: mm_walk structure - * Return: -EBUSY after page fault, or page fault error + * hmm_record_fault() - record a range that needs to be faulted in * - * This function will be called whenever pmd_none() or pte_none() returns true, - * or whenever there is no page directory covering the virtual address range. + * Called by the walk callbacks when they discover that part of the range + * needs a page fault. The callback records what to fault and returns + * HMM_FAULT_PENDING; the outer loop in hmm_range_fault() drops back out of + * walk_page_range() and invokes handle_mm_fault() from a context where no + * page-table or hugetlb_vma_lock is held. */ -static int hmm_vma_fault(unsigned long addr, unsigned long end, - unsigned int required_fault, struct mm_walk *walk) +static int hmm_record_fault(unsigned long addr, unsigned long end, + unsigned int required_fault, + struct mm_walk *walk) { struct hmm_vma_walk *hmm_vma_walk = walk->private; - struct vm_area_struct *vma = walk->vma; - unsigned int fault_flags = FAULT_FLAG_REMOTE; WARN_ON_ONCE(!required_fault); hmm_vma_walk->last = addr; - - if (required_fault & HMM_NEED_WRITE_FAULT) { - if (!(vma->vm_flags & VM_WRITE)) - return -EPERM; - fault_flags |= FAULT_FLAG_WRITE; - } - - for (; addr < end; addr += PAGE_SIZE) - if (handle_mm_fault(vma, addr, fault_flags, NULL) & - VM_FAULT_ERROR) - return -EFAULT; - return -EBUSY; + hmm_vma_walk->end = end; + hmm_vma_walk->required_fault = required_fault; + return HMM_FAULT_PENDING; } static unsigned int hmm_pte_need_fault(const struct hmm_vma_walk *hmm_vma_walk, @@ -174,7 +171,7 @@ static int hmm_vma_walk_hole(unsigned lo return hmm_pfns_fill(addr, end, range, HMM_PFN_ERROR); } if (required_fault) - return hmm_vma_fault(addr, end, required_fault, walk); + return hmm_record_fault(addr, end, required_fault, walk); return hmm_pfns_fill(addr, end, range, 0); } @@ -209,7 +206,7 @@ static int hmm_vma_handle_pmd(struct mm_ required_fault = hmm_range_need_fault(hmm_vma_walk, hmm_pfns, npages, cpu_flags); if (required_fault) - return hmm_vma_fault(addr, end, required_fault, walk); + return hmm_record_fault(addr, end, required_fault, walk); pfn = pmd_pfn(pmd) + ((addr & ~PMD_MASK) >> PAGE_SHIFT); for (i = 0; addr < end; addr += PAGE_SIZE, i++, pfn++) { @@ -328,7 +325,7 @@ out: fault: pte_unmap(ptep); /* Fault any virtual address we were asked to fault */ - return hmm_vma_fault(addr, end, required_fault, walk); + return hmm_record_fault(addr, end, required_fault, walk); } #ifdef CONFIG_ARCH_HAS_PMD_SOFTLEAVES @@ -371,7 +368,7 @@ static int hmm_vma_handle_absent_pmd(str npages, 0); if (required_fault) { if (softleaf_is_device_private(entry)) - return hmm_vma_fault(addr, end, required_fault, walk); + return hmm_record_fault(addr, end, required_fault, walk); else return -EFAULT; } @@ -517,7 +514,7 @@ static int hmm_vma_walk_pud(pud_t *pudp, npages, cpu_flags); if (required_fault) { spin_unlock(ptl); - return hmm_vma_fault(addr, end, required_fault, walk); + return hmm_record_fault(addr, end, required_fault, walk); } pfn = pud_pfn(pud) + ((addr & ~PUD_MASK) >> PAGE_SHIFT); @@ -564,21 +561,8 @@ static int hmm_vma_walk_hugetlb_entry(pt required_fault = hmm_pte_need_fault(hmm_vma_walk, pfn_req_flags, cpu_flags); if (required_fault) { - int ret; - spin_unlock(ptl); - hugetlb_vma_unlock_read(vma); - /* - * Avoid deadlock: drop the vma lock before calling - * hmm_vma_fault(), which will itself potentially take and - * drop the vma lock. This is also correct from a - * protection point of view, because there is no further - * use here of either pte or ptl after dropping the vma - * lock. - */ - ret = hmm_vma_fault(addr, end, required_fault, walk); - hugetlb_vma_lock_read(vma); - return ret; + return hmm_record_fault(addr, end, required_fault, walk); } pfn = pte_pfn(entry) + ((start & ~hmask) >> PAGE_SHIFT); @@ -637,6 +621,44 @@ static const struct mm_walk_ops hmm_walk .walk_lock = PGWALK_RDLOCK, }; +/* + * hmm_do_fault - fault in a range recorded by a walk callback + * + * Called from the outer loop in hmm_range_fault() after a callback + * returned HMM_FAULT_PENDING. At this point we hold only mmap_lock; + * the page-table spinlock and any hugetlb_vma_lock acquired by the walk + * framework have already been released by the unwind. + * + * Returns -EBUSY on success (all pages faulted, caller should re-walk). + * Returns a negative errno on failure. + */ +static int hmm_do_fault(struct mm_struct *mm, + struct hmm_vma_walk *hmm_vma_walk) +{ + unsigned long addr = hmm_vma_walk->last; + unsigned long end = hmm_vma_walk->end; + unsigned int required_fault = hmm_vma_walk->required_fault; + unsigned int fault_flags = FAULT_FLAG_REMOTE; + struct vm_area_struct *vma; + + vma = vma_lookup(mm, addr); + if (!vma) + return -EFAULT; + + if (required_fault & HMM_NEED_WRITE_FAULT) { + if (!(vma->vm_flags & VM_WRITE)) + return -EPERM; + fault_flags |= FAULT_FLAG_WRITE; + } + + for (; addr < end; addr += PAGE_SIZE) + if (handle_mm_fault(vma, addr, fault_flags, NULL) & + VM_FAULT_ERROR) + return -EFAULT; + + return -EBUSY; +} + /** * hmm_range_fault - try to fault some address in a virtual address range * @range: argument structure @@ -675,6 +697,16 @@ int hmm_range_fault(struct hmm_range *ra ret = walk_page_range(mm, hmm_vma_walk.last, range->end, &hmm_walk_ops, &hmm_vma_walk); /* + * When HMM_FAULT_PENDING is returned a walk callback + * recorded a range that needs handle_mm_fault(); + * hmm_do_fault() runs the fault outside walk_page_range() + * (so no page-table or hugetlb_vma_lock is held) and + * returns -EBUSY so the loop re-walks and picks up the + * now-present entries. + */ + if (ret == HMM_FAULT_PENDING) + ret = hmm_do_fault(mm, &hmm_vma_walk); + /* * When -EBUSY is returned the loop restarts with * hmm_vma_walk.last set to an address that has not been stored * in pfns. All entries < last in the pfn array are set to their _ Patches currently in -mm which might be from skinsburskii@gmail.com are lib-test_hmm-use-device-devt-for-coherent-device-range-selection.patch mm-hmm-move-page-fault-handling-out-of-walk-callbacks.patch mm-hmm-add-hmm_range_fault_unlocked_timeout-for-mmap-lock-drop-support.patch selftests-mm-add-hmm-test-for-mmap-lock-dropping-faults.patch mshv-use-hmm_range_fault_unlocked_timeout-for-region-faults.patch drm-nouveau-use-hmm_range_fault_unlocked_timeout-for-svm-faults.patch rdma-umem-use-hmm_range_fault_unlocked_timeout-for-odp-faults.patch accel-amdxdna-use-hmm_range_fault_unlocked_timeout-for-range-population.patch drm-gpusvm-use-hmm_range_fault_unlocked_timeout-for-range-faults.patch