[PATCH] userfaultfd: fix UFFDIO_MOVE livelock on large folio with extra references

[PATCH] userfaultfd: fix UFFDIO_MOVE livelock on large folio with extra references

[Cen Zhang (Microsoft)]

To move one page out of a large folio, move_pages_ptes() first has to
split the folio.  __folio_split() refuses if the folio has any
reference beyond its page table mappings and the caller's own, and it
reports the refusal as -EAGAIN.  move_pages() treats returned -EAGAIN
as a transient race and keeps retrying the same address, holding the VMA
read locks the whole time.  This is incorrect since in some cases
userspace can hold such a reference indefinitely, such as by
registering the page as an io_uring buffer.

An unprivileged user can trigger this with three threads: first holds
such a reference on a page of a THP, second calls UFFDIO_MOVE on another
page of the same THP and spins in the kernel forever holding the VMA
read locks, and third calls fork() or mprotect(), which takes mmap_lock
for writing and then waits forever for those read locks.  From then on
every mmap_lock user of that mm hangs, including other processes
inspecting it through /proc/<pid>/, until the process is SIGKILLed.
The hung task detector reports the blocked tasks (third one), and with
kernel.hung_task_panic=1 the machine panics:

  INFO: task wedge:95 blocked for more than 122 seconds.
  task:wedge  state:D stack:12528 pid:95 tgid:95 ppid:94
  Call Trace:
   __schedule+0x596/0x1110
   schedule+0x36/0xf0
   __vma_start_exclude_readers+0x100/0x170
   __vma_start_write+0x51/0xd0
   __split_vma+0x208/0x3d0
   vma_modify+0x6a8/0x840
   vma_modify_flags+0xce/0x110
   mprotect_fixup+0xbe/0x480
   do_mprotect_pkey+0x2d9/0x580
   __x64_sys_mprotect+0x1a/0x30

Fix by returning -EBUSY instead of -EAGAIN in move_pages_ptes(), the
caller of split_folio(), when the split fails. Thus, move_pages()
does not retry -EBUSY and the ioctl drops its locks and returns to
userspace instead of spinning.

Fixes: adef440691ba ("userfaultfd: UFFDIO_MOVE uABI")
Reported-by: AutonomousCodeSecurity@microsoft.com
Assisted-by: LLM
Signed-off-by: Cen Zhang (Microsoft) <cenzhang@linux.microsoft.com>
---
 mm/userfaultfd.c | 6 +++++-
 1 file changed, 5 insertions(+), 1 deletion(-)

diff --git a/mm/userfaultfd.c b/mm/userfaultfd.c
index 74f04c323c50..500962be1162 100644
--- a/mm/userfaultfd.c
+++ b/mm/userfaultfd.c
@@ -1663,8 +1663,12 @@ static long move_pages_ptes(struct mm_struct *mm, pmd_t *dst_pmd, pmd_t *src_pmd
 			pte_unmap(dst_pte);
 			src_pte = dst_pte = NULL;
 			ret = split_folio(src_folio);
-			if (ret)
+			if (ret) {
+				/* do not retry, extra refs may be held forever */
+				if (ret == -EAGAIN)
+					ret = -EBUSY;
 				goto out;
+			}
 			/* have to reacquire the folio after it got split */
 			folio_unlock(src_folio);
 			folio_put(src_folio);
-- 
2.55.0

Re: [PATCH] userfaultfd: fix UFFDIO_MOVE livelock on large folio with extra references

[David Hildenbrand (Arm)]

On 10/12/26 22:15, Cen Zhang (Microsoft) wrote:
> To move one page out of a large folio, move_pages_ptes() first has to
> split the folio.  __folio_split() refuses if the folio has any
> reference beyond its page table mappings and the caller's own, and it
> reports the refusal as -EAGAIN.  move_pages() treats returned -EAGAIN
> as a transient race and keeps retrying the same address, holding the VMA
> read locks the whole time.  This is incorrect since in some cases
> userspace can hold such a reference indefinitely, such as by
> registering the page as an io_uring buffer.
> 
> An unprivileged user can trigger this with three threads: first holds
> such a reference on a page of a THP, second calls UFFDIO_MOVE on another
> page of the same THP and spins in the kernel forever holding the VMA
> read locks, and third calls fork() or mprotect(), which takes mmap_lock
> for writing and then waits forever for those read locks.  From then on
> every mmap_lock user of that mm hangs, including other processes
> inspecting it through /proc/<pid>/, until the process is SIGKILLed.
> The hung task detector reports the blocked tasks (third one), and with
> kernel.hung_task_panic=1 the machine panics:
> 
>   INFO: task wedge:95 blocked for more than 122 seconds.
>   task:wedge  state:D stack:12528 pid:95 tgid:95 ppid:94
>   Call Trace:
>    __schedule+0x596/0x1110
>    schedule+0x36/0xf0
>    __vma_start_exclude_readers+0x100/0x170
>    __vma_start_write+0x51/0xd0
>    __split_vma+0x208/0x3d0
>    vma_modify+0x6a8/0x840
>    vma_modify_flags+0xce/0x110
>    mprotect_fixup+0xbe/0x480
>    do_mprotect_pkey+0x2d9/0x580
>    __x64_sys_mprotect+0x1a/0x30
> 
> Fix by returning -EBUSY instead of -EAGAIN in move_pages_ptes(), the
> caller of split_folio(), when the split fails. Thus, move_pages()
> does not retry -EBUSY and the ioctl drops its locks and returns to
> userspace instead of spinning.
> 
> Fixes: adef440691ba ("userfaultfd: UFFDIO_MOVE uABI")
> Reported-by: AutonomousCodeSecurity@microsoft.com
> Assisted-by: LLM
> Signed-off-by: Cen Zhang (Microsoft) <cenzhang@linux.microsoft.com>
> ---
>  mm/userfaultfd.c | 6 +++++-
>  1 file changed, 5 insertions(+), 1 deletion(-)
> 
> diff --git a/mm/userfaultfd.c b/mm/userfaultfd.c
> index 74f04c323c50..500962be1162 100644
> --- a/mm/userfaultfd.c
> +++ b/mm/userfaultfd.c
> @@ -1663,8 +1663,12 @@ static long move_pages_ptes(struct mm_struct *mm, pmd_t *dst_pmd, pmd_t *src_pmd
>  			pte_unmap(dst_pte);
>  			src_pte = dst_pte = NULL;
>  			ret = split_folio(src_folio);
> -			if (ret)
> +			if (ret) {
> +				/* do not retry, extra refs may be held forever */
> +				if (ret == -EAGAIN)
> +					ret = -EBUSY;
>  				goto out;
> +			}
>  			/* have to reacquire the folio after it got split */
>  			folio_unlock(src_folio);
>  			folio_put(src_folio);

Yeah, that makes sense to me.

Acked-by: David Hildenbrand (Arm) <david@kernel.org>

-- 
Cheers,

David