[PATCH] Documentation/userfaultfd: document RWP behavior when the PTE is dropped

[PATCH] Documentation/userfaultfd: document RWP behavior when the PTE is dropped

[Jose A. Perez de Azpillaga]

"Scope of protection" lists only the PTE drops requested by the user and
says the range has to be re-armed afterwards. On shmem the kernel drops
the PTE as well (page reclaim, collapse), and RWP has no marker by design:
the next access is a minor or a missing fault, and the user is expected to
register for those. Document this, together with the modes that sync and
async tracking need.

The VMM example registers guest_mem with RWP and MISSING only, so a vCPU
write to an offset that is not mapped there can be lost during eviction.
Add MINOR to the registration and describe how the handler treats minor
faults.

Link: https://lore.kernel.org/linux-mm/856cdf21-7982-475a-b244-a7ceff4eb6aa@kernel.org/
Link: https://lore.kernel.org/linux-mm/avWnaR7VVEGa7yNp@thinkstation/
Suggested-by: Tal Zussman <tz2294@columbia.edu>
Signed-off-by: Jose A. Perez de Azpillaga <azpijr@gmail.com>
---
this is the first time I'm writing documentation, so I accept any
suggestions to improve it :) and I tried to follow the documentation's
own style.

 Documentation/admin-guide/mm/userfaultfd.rst | 65 ++++++++++++++++----
 1 file changed, 53 insertions(+), 12 deletions(-)

diff --git a/Documentation/admin-guide/mm/userfaultfd.rst b/Documentation/admin-guide/mm/userfaultfd.rst
index 97daea83b7f1..19bdac668f93 100644
--- a/Documentation/admin-guide/mm/userfaultfd.rst
+++ b/Documentation/admin-guide/mm/userfaultfd.rst
@@ -384,17 +384,45 @@ normal missing path (``do_anonymous_page()``, ``do_swap_page()``,
 the population itself, co-register the range with
 ``UFFDIO_REGISTER_MODE_MISSING``.
 
-Protection is preserved across page reclaim: a page swapped out while
-RWP-protected carries the marker on its swap entry, and swap-in restores
-the PROT_NONE state so the first access after swap-in still faults. The
-same applies to pages temporarily replaced by migration entries.
-
-Operations that drop the PTE entirely — ``MADV_DONTNEED`` on anonymous
-memory, hole-punch on shmem, truncation of a file mapping — also drop the
-RWP marker: the next access re-populates the range without protection.
-Unlike WP (which persists via ``PTE_MARKER_UFFD_WP``), there is no
-persistent RWP marker today. The user needs to re-arm the range with
-``UFFDIO_RWPROTECT`` after any operation that explicitly frees PTEs.
+Protection of anonymous memory is preserved across page reclaim: a page
+swapped out while RWP-protected carries the marker on its swap entry, and
+swap-in restores the PROT_NONE state so the first access after swap-in
+still faults. The same applies to pages temporarily replaced by migration
+entries. Shmem is different: reclaiming a shmem page leaves the PTE empty
+rather than installing a swap entry, so the protection is dropped along
+with the PTE.
+
+Operations that drop the PTE entirely also drop the RWP protection. Some
+are requested by the user — ``MADV_DONTNEED``, hole-punch on shmem,
+truncation of a file mapping — and some are initiated by the kernel on
+shmem: page reclaim, and collapse into a huge page by khugepaged or
+``MADV_COLLAPSE``. Unlike WP (which persists via ``PTE_MARKER_UFFD_WP``),
+there is no persistent RWP marker, and this is intentional: once the PTE is
+gone, the next access takes a page fault that userfaultfd can deliver, and
+that fault is the notification. Which fault it is depends on what happened
+to the page:
+
+- The page is still in the page cache or in swap: the access is a minor
+  fault, delivered if the range is co-registered with
+  ``UFFDIO_REGISTER_MODE_MINOR``.
+
+- The page is gone: the access is a missing fault, delivered if the range
+  is co-registered with ``UFFDIO_REGISTER_MODE_MISSING``.
+
+WP cannot use the same scheme — a minor fault would trap reads as well as
+writes — which is why it needs a marker.
+
+Without the matching mode the page is mapped back silently and without
+protection, so what to co-register depends on the tracking mode:
+
+- Sync mode: the access would be lost. Co-register shmem and hugetlbfs
+  ranges with ``UFFDIO_REGISTER_MODE_MINOR`` (and
+  ``UFFDIO_REGISTER_MODE_MISSING`` if the file can have holes), and
+  anonymous ranges, where minor faults are not available, with
+  ``UFFDIO_REGISTER_MODE_MISSING``.
+
+- Async mode: neither is needed. A page faulted back in is mapped without
+  the uffd bit, so ``PAGEMAP_SCAN`` reports it as accessed.
 
 **Fault Handling:**
 
@@ -486,10 +514,16 @@ never traps on its own ::
     ioctl(uffd, UFFDIO_API, &api);
     if (!(api.features & UFFD_FEATURE_RWP))
         /* RWP unavailable on this kernel/arch -- fall back. */
+    /*
+     * MINOR: an offset in the page cache but not mapped in guest_mem
+     * would be mapped back without a fault, and a vCPU write between
+     * the pwrite() and the hole punch in step 4 would be lost.
+     */
     ioctl(uffd, UFFDIO_REGISTER, &(struct uffdio_register){
         .range = { guest_mem, guest_size },
         .mode = UFFDIO_REGISTER_MODE_RWP |
-                UFFDIO_REGISTER_MODE_MISSING,
+                UFFDIO_REGISTER_MODE_MISSING |
+                UFFDIO_REGISTER_MODE_MINOR,
     });
 
     /* Tracking loop */
@@ -550,6 +584,13 @@ progress. After ``fallocate()`` punches the page out and ``UFFDIO_WAKE``
 fires, the vCPU retries the access, faults as ``MISSING``, and the
 handler resolves it with ``UFFDIO_COPY`` from storage.
 
+The same holds for an offset being evicted that is not mapped in
+``guest_mem``, except that the vCPU blocks with a
+``UFFD_PAGEFAULT_FLAG_MINOR`` fault, which the handler leaves pending until
+the page has been punched out. Minor faults on any other offset are
+resolved with ``UFFDIO_CONTINUE``: the page is mapped without RWP
+protection, so the next scan reports it as accessed.
+
 This workflow targets shmem and hugetlbfs (both support a private
 ``io_mem`` mapping over the same fd). Anonymous-memory backings need a
 different inner-loop strategy because the VMM has no way to read the

base-commit: 31a696f9c37defab0f28f774e00ac7f135889ec3
-- 
2.56.0

Re: [PATCH] Documentation/userfaultfd: document RWP behavior when the PTE is dropped

[Kiryl Shutsemau]

On Fri, Nov 13, 2026 at 12:27:42PM +0100, Jose A. Perez de Azpillaga wrote:
> "Scope of protection" lists only the PTE drops requested by the user and
> says the range has to be re-armed afterwards. On shmem the kernel drops
> the PTE as well (page reclaim, collapse), and RWP has no marker by design:
> the next access is a minor or a missing fault, and the user is expected to
> register for those. Document this, together with the modes that sync and
> async tracking need.
> 
> The VMM example registers guest_mem with RWP and MISSING only, so a vCPU
> write to an offset that is not mapped there can be lost during eviction.
> Add MINOR to the registration and describe how the handler treats minor
> faults.
> 
> Link: https://lore.kernel.org/linux-mm/856cdf21-7982-475a-b244-a7ceff4eb6aa@kernel.org/
> Link: https://lore.kernel.org/linux-mm/avWnaR7VVEGa7yNp@thinkstation/
> Suggested-by: Tal Zussman <tz2294@columbia.edu>
> Signed-off-by: Jose A. Perez de Azpillaga <azpijr@gmail.com>
> ---
> this is the first time I'm writing documentation, so I accept any
> suggestions to improve it :) and I tried to follow the documentation's
> own style.
> 
>  Documentation/admin-guide/mm/userfaultfd.rst | 65 ++++++++++++++++----
>  1 file changed, 53 insertions(+), 12 deletions(-)
> 
> diff --git a/Documentation/admin-guide/mm/userfaultfd.rst b/Documentation/admin-guide/mm/userfaultfd.rst
> index 97daea83b7f1..19bdac668f93 100644
> --- a/Documentation/admin-guide/mm/userfaultfd.rst
> +++ b/Documentation/admin-guide/mm/userfaultfd.rst
> @@ -384,17 +384,45 @@ normal missing path (``do_anonymous_page()``, ``do_swap_page()``,
>  the population itself, co-register the range with
>  ``UFFDIO_REGISTER_MODE_MISSING``.
>  
> -Protection is preserved across page reclaim: a page swapped out while
> -RWP-protected carries the marker on its swap entry, and swap-in restores
> -the PROT_NONE state so the first access after swap-in still faults. The
> -same applies to pages temporarily replaced by migration entries.
> -
> -Operations that drop the PTE entirely — ``MADV_DONTNEED`` on anonymous
> -memory, hole-punch on shmem, truncation of a file mapping — also drop the
> -RWP marker: the next access re-populates the range without protection.
> -Unlike WP (which persists via ``PTE_MARKER_UFFD_WP``), there is no
> -persistent RWP marker today. The user needs to re-arm the range with
> -``UFFDIO_RWPROTECT`` after any operation that explicitly frees PTEs.
> +Protection of anonymous memory is preserved across page reclaim: a page
> +swapped out while RWP-protected carries the marker on its swap entry, and
> +swap-in restores the PROT_NONE state so the first access after swap-in
> +still faults. The same applies to pages temporarily replaced by migration
> +entries. Shmem is different: reclaiming a shmem page leaves the PTE empty
> +rather than installing a swap entry, so the protection is dropped along
> +with the PTE.

Migration is not anon-only: migration entries carry the uffd bit for
shmem and hugetlbfs too. Keep that sentence out of the anon part.

And please call it the uffd bit, not the marker. "Carries the marker"
next to "there is no persistent RWP marker" reads as a contradiction.

> +Operations that drop the PTE entirely also drop the RWP protection. Some
> +are requested by the user — ``MADV_DONTNEED``, hole-punch on shmem,
> +truncation of a file mapping — and some are initiated by the kernel on
> +shmem: page reclaim, and collapse into a huge page by khugepaged or
> +``MADV_COLLAPSE``.

I would not list them. The list is not complete: splitting a PMD mapping
of a shmem THP drops it too. The next tool will find the path that is
missing and report it. State the rule instead: on shmem the kernel can
drop the PTE at any time while the page stays in the page cache or swap.
Reclaim and collapse are examples.

> Unlike WP (which persists via ``PTE_MARKER_UFFD_WP``),
> +there is no persistent RWP marker, and this is intentional: once the PTE is
> +gone, the next access takes a page fault that userfaultfd can deliver, and
> +that fault is the notification. Which fault it is depends on what happened
> +to the page:
> +
> +- The page is still in the page cache or in swap: the access is a minor
> +  fault, delivered if the range is co-registered with
> +  ``UFFDIO_REGISTER_MODE_MINOR``.
> +
> +- The page is gone: the access is a missing fault, delivered if the range
> +  is co-registered with ``UFFDIO_REGISTER_MODE_MISSING``.
> +
> +WP cannot use the same scheme — a minor fault would trap reads as well as
> +writes — which is why it needs a marker.
> +
> +Without the matching mode the page is mapped back silently and without
> +protection, so what to co-register depends on the tracking mode:
> +
> +- Sync mode: the access would be lost. Co-register shmem and hugetlbfs
> +  ranges with ``UFFDIO_REGISTER_MODE_MINOR`` (and
> +  ``UFFDIO_REGISTER_MODE_MISSING`` if the file can have holes), and
> +  anonymous ranges, where minor faults are not available, with
> +  ``UFFDIO_REGISTER_MODE_MISSING``.
> +
> +- Async mode: neither is needed. A page faulted back in is mapped without
> +  the uffd bit, so ``PAGEMAP_SCAN`` reports it as accessed.

The registration is fixed, the mode is not. A context that switches to
sync with UFFDIO_SET_MODE, as the example does, needs the sync
registration. That has a cost in async mode: minor faults are always
synchronous, so "async -- no vCPU stalls" in step 1 is no longer true.

Getting it back would need MINOR to follow UFFDIO_SET_MODE: in async
mode, map the page and let the next scan report it as accessed. That
could be a future project if the stalls turn out to matter. For this
patch, just fix the comment.

>  **Fault Handling:**
>  
> @@ -486,10 +514,16 @@ never traps on its own ::
>      ioctl(uffd, UFFDIO_API, &api);
>      if (!(api.features & UFFD_FEATURE_RWP))
>          /* RWP unavailable on this kernel/arch -- fall back. */
> +    /*
> +     * MINOR: an offset in the page cache but not mapped in guest_mem
> +     * would be mapped back without a fault, and a vCPU write between
> +     * the pwrite() and the hole punch in step 4 would be lost.
> +     */

Not yours, but the "if (!(api.features & UFFD_FEATURE_RWP))" has only a
comment as its body, so UFFDIO_REGISTER becomes the body. A second
comment between them makes that harder to see.

Adding {} around the body comment of the branch can help a bit.

>      ioctl(uffd, UFFDIO_REGISTER, &(struct uffdio_register){
>          .range = { guest_mem, guest_size },
>          .mode = UFFDIO_REGISTER_MODE_RWP |
> -                UFFDIO_REGISTER_MODE_MISSING,
> +                UFFDIO_REGISTER_MODE_MISSING |
> +                UFFDIO_REGISTER_MODE_MINOR,
>      });
>  
>      /* Tracking loop */
> @@ -550,6 +584,13 @@ progress. After ``fallocate()`` punches the page out and ``UFFDIO_WAKE``
>  fires, the vCPU retries the access, faults as ``MISSING``, and the
>  handler resolves it with ``UFFDIO_COPY`` from storage.
>  
> +The same holds for an offset being evicted that is not mapped in
> +``guest_mem``, except that the vCPU blocks with a
> +``UFFD_PAGEFAULT_FLAG_MINOR`` fault, which the handler leaves pending until
> +the page has been punched out. Minor faults on any other offset are
> +resolved with ``UFFDIO_CONTINUE``: the page is mapped without RWP
> +protection, so the next scan reports it as accessed.

"Any other offset" is too broad. A cold offset that step 4 has not
reached yet will still be written out and punched. If the handler maps
it with UFFDIO_CONTINUE, a vCPU write between the pwrite() and the punch
is lost. That is the case MINOR was added for. A fault on any offset
this pass will evict has to wait until that offset is punched. The same
goes for RWP faults.


-- 
  Kiryl Shutsemau / Kirill A. Shutemov