[PATCH] Documentation/userfaultfd: document RWP behavior when the PTE is dropped

["Jose A. Perez de Azpillaga" <azpijr@gmail.com>]

"Scope of protection" lists only the PTE drops requested by the user and
says the range has to be re-armed afterwards. On shmem the kernel drops
the PTE as well (page reclaim, collapse), and RWP has no marker by design:
the next access is a minor or a missing fault, and the user is expected to
register for those. Document this, together with the modes that sync and
async tracking need.

The VMM example registers guest_mem with RWP and MISSING only, so a vCPU
write to an offset that is not mapped there can be lost during eviction.
Add MINOR to the registration and describe how the handler treats minor
faults.

Link: https://lore.kernel.org/linux-mm/856cdf21-7982-475a-b244-a7ceff4eb6aa@kernel.org/
Link: https://lore.kernel.org/linux-mm/avWnaR7VVEGa7yNp@thinkstation/
Suggested-by: Tal Zussman <tz2294@columbia.edu>
Signed-off-by: Jose A. Perez de Azpillaga <azpijr@gmail.com>
---
this is the first time I'm writing documentation, so I accept any
suggestions to improve it :) and I tried to follow the documentation's
own style.

 Documentation/admin-guide/mm/userfaultfd.rst | 65 ++++++++++++++++----
 1 file changed, 53 insertions(+), 12 deletions(-)

diff --git a/Documentation/admin-guide/mm/userfaultfd.rst b/Documentation/admin-guide/mm/userfaultfd.rst
index 97daea83b7f1..19bdac668f93 100644
--- a/Documentation/admin-guide/mm/userfaultfd.rst
+++ b/Documentation/admin-guide/mm/userfaultfd.rst
@@ -384,17 +384,45 @@ normal missing path (``do_anonymous_page()``, ``do_swap_page()``,
 the population itself, co-register the range with
 ``UFFDIO_REGISTER_MODE_MISSING``.
 
-Protection is preserved across page reclaim: a page swapped out while
-RWP-protected carries the marker on its swap entry, and swap-in restores
-the PROT_NONE state so the first access after swap-in still faults. The
-same applies to pages temporarily replaced by migration entries.
-
-Operations that drop the PTE entirely — ``MADV_DONTNEED`` on anonymous
-memory, hole-punch on shmem, truncation of a file mapping — also drop the
-RWP marker: the next access re-populates the range without protection.
-Unlike WP (which persists via ``PTE_MARKER_UFFD_WP``), there is no
-persistent RWP marker today. The user needs to re-arm the range with
-``UFFDIO_RWPROTECT`` after any operation that explicitly frees PTEs.
+Protection of anonymous memory is preserved across page reclaim: a page
+swapped out while RWP-protected carries the marker on its swap entry, and
+swap-in restores the PROT_NONE state so the first access after swap-in
+still faults. The same applies to pages temporarily replaced by migration
+entries. Shmem is different: reclaiming a shmem page leaves the PTE empty
+rather than installing a swap entry, so the protection is dropped along
+with the PTE.
+
+Operations that drop the PTE entirely also drop the RWP protection. Some
+are requested by the user — ``MADV_DONTNEED``, hole-punch on shmem,
+truncation of a file mapping — and some are initiated by the kernel on
+shmem: page reclaim, and collapse into a huge page by khugepaged or
+``MADV_COLLAPSE``. Unlike WP (which persists via ``PTE_MARKER_UFFD_WP``),
+there is no persistent RWP marker, and this is intentional: once the PTE is
+gone, the next access takes a page fault that userfaultfd can deliver, and
+that fault is the notification. Which fault it is depends on what happened
+to the page:
+
+- The page is still in the page cache or in swap: the access is a minor
+  fault, delivered if the range is co-registered with
+  ``UFFDIO_REGISTER_MODE_MINOR``.
+
+- The page is gone: the access is a missing fault, delivered if the range
+  is co-registered with ``UFFDIO_REGISTER_MODE_MISSING``.
+
+WP cannot use the same scheme — a minor fault would trap reads as well as
+writes — which is why it needs a marker.
+
+Without the matching mode the page is mapped back silently and without
+protection, so what to co-register depends on the tracking mode:
+
+- Sync mode: the access would be lost. Co-register shmem and hugetlbfs
+  ranges with ``UFFDIO_REGISTER_MODE_MINOR`` (and
+  ``UFFDIO_REGISTER_MODE_MISSING`` if the file can have holes), and
+  anonymous ranges, where minor faults are not available, with
+  ``UFFDIO_REGISTER_MODE_MISSING``.
+
+- Async mode: neither is needed. A page faulted back in is mapped without
+  the uffd bit, so ``PAGEMAP_SCAN`` reports it as accessed.
 
 **Fault Handling:**
 
@@ -486,10 +514,16 @@ never traps on its own ::
     ioctl(uffd, UFFDIO_API, &api);
     if (!(api.features & UFFD_FEATURE_RWP))
         /* RWP unavailable on this kernel/arch -- fall back. */
+    /*
+     * MINOR: an offset in the page cache but not mapped in guest_mem
+     * would be mapped back without a fault, and a vCPU write between
+     * the pwrite() and the hole punch in step 4 would be lost.
+     */
     ioctl(uffd, UFFDIO_REGISTER, &(struct uffdio_register){
         .range = { guest_mem, guest_size },
         .mode = UFFDIO_REGISTER_MODE_RWP |
-                UFFDIO_REGISTER_MODE_MISSING,
+                UFFDIO_REGISTER_MODE_MISSING |
+                UFFDIO_REGISTER_MODE_MINOR,
     });
 
     /* Tracking loop */
@@ -550,6 +584,13 @@ progress. After ``fallocate()`` punches the page out and ``UFFDIO_WAKE``
 fires, the vCPU retries the access, faults as ``MISSING``, and the
 handler resolves it with ``UFFDIO_COPY`` from storage.
 
+The same holds for an offset being evicted that is not mapped in
+``guest_mem``, except that the vCPU blocks with a
+``UFFD_PAGEFAULT_FLAG_MINOR`` fault, which the handler leaves pending until
+the page has been punched out. Minor faults on any other offset are
+resolved with ``UFFDIO_CONTINUE``: the page is mapped without RWP
+protection, so the next scan reports it as accessed.
+
 This workflow targets shmem and hugetlbfs (both support a private
 ``io_mem`` mapping over the same fd). Anonymous-memory backings need a
 different inner-loop strategy because the VMM has no way to read the

base-commit: 31a696f9c37defab0f28f774e00ac7f135889ec3
-- 
2.56.0