Re: [PATCH] KVM: guest_memfd: Disable VMA merging with VM_DONTEXPAND

[Sean Christopherson <seanjc@google.com>]

On Wed, Feb 04, 2026, Ackerley Tng wrote:
> Ackerley Tng <ackerleytng@google.com> writes:
> 
> > #syz test: git://git.kernel.org/pub/scm/virt/kvm/kvm.git next
> >
> > guest_memfd VMAs don't need to be merged,

Why not?  There are benefits to merging VMAs that have nothing to do with folios.
E.g. map 1GiB of guest_memfd with 512*512 4KiB VMAs, and then it becomes quite
desirable to merge all of those VMAs into one.

Creating _hugepages_ doesn't add value, but that's not the same things as merging
VMAs.

> > especially now, since guest_memfd only supports PAGE_SIZE folios.
> >
> > Set VM_DONTEXPAND on guest_memfd VMAs.
>
> Local tests and syzbot agree that this fixes the issue identified. :)
> 
> I would like to look into madvise(MADV_COLLAPSE) and uprobes triggering
> mapping/folio collapsing before submitting a full patch series.
> 
> David, Michael, Vishal, what do you think of the choice of setting
> VM_DONTEXPAND to disable khugepaged?

I'm not one of the above, but for me it feels very much like treating a symptom
and not fixing the underlying cause.

It seems like what KVM should do is not block one path that triggers hugepage
processing, but instead flat out disallow creating hugepages.  Unfortunately,
AFAICT, there's no existing way to prevent madvise() from clearing VM_NOHUGEPAGE,
so we can't simply force that flag.

I'd prefer not to special case guest_memfd, a la devdax, but I also want to address
this head-on, not by removing a tangentially related trigger.

> + For 4K guest_memfd, there's really nothing to expand
> + For THP and HugeTLB guest_memfd (future), we actually don't want
> expansion of the VMAs.
> 
> IIUC setting VM_DONTEXPAND doesn't affect mremap() as long as the
> remapping does not involve expansion.
> 
> > In addition, this disables khugepaged from operating on guest_memfd folios,
> > which may result in unintended merging of guest_memfd folios.
> >
> > Change-Id: I5867edcb66b075b54b25260afd22a198aee76df1
> > Signed-off-by: Ackerley Tng <ackerleytng@google.com>
> > ---
> >  virt/kvm/guest_memfd.c | 6 ++++++
> >  1 file changed, 6 insertions(+)
> >
> > diff --git a/virt/kvm/guest_memfd.c b/virt/kvm/guest_memfd.c
> > index fdaea3422c30..3d4ac461c28b 100644
> > --- a/virt/kvm/guest_memfd.c
> > +++ b/virt/kvm/guest_memfd.c
> > @@ -480,6 +480,12 @@ static int kvm_gmem_mmap(struct file *file, struct vm_area_struct *vma)
> >  		return -EINVAL;
> >  	}
> >
> > +	/*
> > +	 * Disable VMA merging - guest_memfd VMAs should be
> > +	 * static. This also stops khugepaged from operating on
> > +	 * guest_memfd VMAs and folios.
> > +	 */
> > +	vm_flags_set(vma, VM_DONTEXPAND);
> >  	vma->vm_ops = &kvm_gmem_vm_ops;
> >
> >  	return 0;
> > --
> > 2.53.0.rc2.204.g2597b5adb4-goog