[PATCH] userfaultfd: fix mmap_changing checking in mfill_atomic_hugetlb

[PATCH] userfaultfd: fix mmap_changing checking in mfill_atomic_hugetlb

[Lokesh Gidra]

In mfill_atomic_hugetlb(), mmap_changing isn't being checked
again if we drop mmap_lock and reacquire it. When the lock is not held,
mmap_changing could have been incremented. This is also inconsistent
with the behavior in mfill_atomic().

Signed-off-by: Lokesh Gidra <lokeshgidra@google.com>
---
 mm/userfaultfd.c | 15 +++++++++++++--
 1 file changed, 13 insertions(+), 2 deletions(-)

diff --git a/mm/userfaultfd.c b/mm/userfaultfd.c
index 20e3b0d9cf7e..75fcf1f783bc 100644
--- a/mm/userfaultfd.c
+++ b/mm/userfaultfd.c
@@ -357,6 +357,7 @@ static __always_inline ssize_t mfill_atomic_hugetlb(
 					      unsigned long dst_start,
 					      unsigned long src_start,
 					      unsigned long len,
+					      atomic_t *mmap_changing,
 					      uffd_flags_t flags)
 {
 	struct mm_struct *dst_mm = dst_vma->vm_mm;
@@ -472,6 +473,15 @@ static __always_inline ssize_t mfill_atomic_hugetlb(
 				goto out;
 			}
 			mmap_read_lock(dst_mm);
+			/*
+			 * If memory mappings are changing because of non-cooperative
+			 * operation (e.g. mremap) running in parallel, bail out and
+			 * request the user to retry later
+			 */
+			if (mmap_changing && atomic_read(mmap_changing)) {
+				err = -EAGAIN;
+				break;
+			}
 
 			dst_vma = NULL;
 			goto retry;
@@ -506,6 +516,7 @@ extern ssize_t mfill_atomic_hugetlb(struct vm_area_struct *dst_vma,
 				    unsigned long dst_start,
 				    unsigned long src_start,
 				    unsigned long len,
+				    atomic_t *mmap_changing,
 				    uffd_flags_t flags);
 #endif /* CONFIG_HUGETLB_PAGE */
 
@@ -622,8 +633,8 @@ static __always_inline ssize_t mfill_atomic(struct mm_struct *dst_mm,
 	 * If this is a HUGETLB vma, pass off to appropriate routine
 	 */
 	if (is_vm_hugetlb_page(dst_vma))
-		return  mfill_atomic_hugetlb(dst_vma, dst_start,
-					     src_start, len, flags);
+		return  mfill_atomic_hugetlb(dst_vma, dst_start, src_start,
+					     len, mmap_changing, flags);
 
 	if (!vma_is_anonymous(dst_vma) && !vma_is_shmem(dst_vma))
 		goto out_unlock;
-- 
2.43.0.429.g432eaa2c6b-goog

Re: [PATCH] userfaultfd: fix mmap_changing checking in mfill_atomic_hugetlb

[Andrew Morton]

On Wed, 17 Jan 2024 14:37:29 -0800 Lokesh Gidra <lokeshgidra@google.com> wrote:

> In mfill_atomic_hugetlb(), mmap_changing isn't being checked
> again if we drop mmap_lock and reacquire it. When the lock is not held,
> mmap_changing could have been incremented. This is also inconsistent
> with the behavior in mfill_atomic().

Thanks. Could you and reviewers please consider

- what might be the userspace-visible runtime effects?

- Should the fix be backported into earlier kernels?

- A suitable Fixes: target?

Re: [PATCH] userfaultfd: fix mmap_changing checking in mfill_atomic_hugetlb

[Axel Rasmussen]

Apologies, I had forgotten to re-check the "send plaintext" checkbox
in my e-mail client, so the mailing lists rejected my previous mail. I
am duly ashamed. Allow me to try once more. :)

On Thu, Jan 18, 2024 at 1:59 PM Andrew Morton <akpm@linux-foundation.org> wrote:
>
> On Wed, 17 Jan 2024 14:37:29 -0800 Lokesh Gidra <lokeshgidra@google.com> wrote:
>
> > In mfill_atomic_hugetlb(), mmap_changing isn't being checked
> > again if we drop mmap_lock and reacquire it. When the lock is not held,
> > mmap_changing could have been incremented. This is also inconsistent
> > with the behavior in mfill_atomic().

The change looks reasonable to me. I'm not sure I can conclusively say
there isn't some other mechanism specific to hugetlbfs which means
this isn't needed, though.

>
> Thanks. Could you and reviewers please consider
>
> - what might be the userspace-visible runtime effects?
>
> - Should the fix be backported into earlier kernels?
>
> - A suitable Fixes: target?

Hmm, 60d4d2d2b40e4 added __mcopy_atomic_hugetlb without this. But, at
that point in history, none of the other functions had mmap_changing
either.

So, I think the right Fixes: target is df2cc96e77011 ("userfaultfd:
prevent non-cooperative events vs mcopy_atomic races") ? It seems to
have missed the hugetlb path. This was introduced in 4.18.

Based on that commit's message, essentially what can happen if the
race "succeeds" is, memory can be accessed without userfaultfd being
notified of this fact. Depending on what userfaultfd is being used
for, from userspace's perspective this can appear like memory
corruption for example. So, based on that it seems to me reasonable to
backport this to stable kernels (4.19+).

Re: [PATCH] userfaultfd: fix mmap_changing checking in mfill_atomic_hugetlb

[Mike Rapoport]

On Thu, Jan 18, 2024 at 03:17:14PM -0800, Axel Rasmussen wrote:
> 
> On Thu, Jan 18, 2024 at 1:59 PM Andrew Morton <akpm@linux-foundation.org>
> wrote:
> 
>     On Wed, 17 Jan 2024 14:37:29 -0800 Lokesh Gidra <lokeshgidra@google.com>
>     wrote:
> 
>     > In mfill_atomic_hugetlb(), mmap_changing isn't being checked
>     > again if we drop mmap_lock and reacquire it. When the lock is not held,
>     > mmap_changing could have been incremented. This is also inconsistent
>     > with the behavior in mfill_atomic().
> 
> 
> The change looks reasonable to me. I'm not sure I can conclusively say there
> isn't some other mechanism specific to hugetlbfs which means this isn't needed,
> though.
  
There's nothing specific to hugetlb, if a non-cooperative uffdio_copy races
with mremap/fork etc, the vma under it may change
 
>     Thanks. Could you and reviewers please consider
> 
>     - what might be the userspace-visible runtime effects?

For users of non-cooperative uffd with hugetlb, this would fix crashes
caused by races between uffd operations that update memory and the
operations that change the VM layout. Pretty much the same fix as
df2cc96e77011 ("userfaultfd: prevent non-cooperative events vs mcopy_atomic
races") for !hugetlb memory.

I doubt such users exist, though...
 
>     - Should the fix be backported into earlier kernels?
>     - A suitable Fixes: target?
> 
> Hmm, 60d4d2d2b40e4 added __mcopy_atomic_hugetlb without this. But, at that
> point in history, none of the other functions had mmap_changing either.
> 
> So, I think the right Fixes: target is df2cc96e77011 ("userfaultfd: prevent
> non-cooperative events vs mcopy_atomic races") ? It seems to have missed the
> hugetlb path. This was introduced in 4.18.
> 
> Based on that commit's message, essentially what can happen if the race
> "succeeds" is, memory can be accessed without userfaultfd being notified of
> this fact. Depending on what userfaultfd is being used for, from
> userspace's perspective this can appear like memory corruption for example. So,
> based on that it seems to me reasonable to backport this to stable kernels
> (4.19+).

I agree with Axel, 

Fixes: df2cc96e77011 ("userfaultfd: prevent non-cooperative events vs mcopy_atomic races")

seems appropriate.

-- 
Sincerely yours,
Mike.