Linux-mm Archive on lore.kernel.org
 help / color / mirror / Atom feed
* [PATCH RFC 0/7] mm: let vm_ops->open() fail for hugetlb
@ 2026-11-18 17:00 Josef Bacik
  2026-11-18 17:00 ` [PATCH RFC 1/7] treewide: make vm_ops->open() return an int Josef Bacik
                   ` (7 more replies)
  0 siblings, 8 replies; 9+ messages in thread
From: Josef Bacik @ 2026-11-18 17:00 UTC (permalink / raw)
  To: Andrew Morton, Lorenzo Stoakes, Liam R. Howlett,
	David Hildenbrand
  Cc: Vlastimil Babka, Jann Horn, Pedro Falcato, Muchun Song,
	Oscar Salvador, linux-mm, Josef Bacik

Hello,

vm_ops->open() can't fail, so anything that allocates in it has to hide the
failure.  hugetlb is the one I care about: if it can't allocate the VMA lock
for a shared mapping it carries on without one, and faults can then race
with truncation.

The callers already have failure paths right around the call, so this series
lets it return an error, and hugetlb then uses that.  The first patch is the
mechanical conversion, which touches a lot of files but only makes every
callback return 0.  Fork needed the call moved ahead of the VMA going into
the tree, the rest is just using the failure paths that are there.

A failed split can be the last step of something that can't be undone, so
I've documented that a callback should only fail if the VMA can't work
otherwise.  For hugetlb this is a small allocation, so in practice it's a task
that is already being killed, and hugetlb splits ahead of a move now so it
doesn't have that problem anyway.

I changed the signature rather than adding a second hook because it seemed
less confusing, even though only hugetlb uses it so far, but I'm happy to go
the other way if you'd prefer the smaller diff.

I tested this with the mm selftests, the VMA tests and a loop that fails each
allocation in turn inside munmap, mprotect, mremap, fork and mmap of hugetlb,
shmem and anon mappings, under KASAN and lockdep.  Everything either worked
or failed with the data intact, and the huge page pool came back to where it
started.  The nommu change is only compile tested.  Thanks,

Josef

---
Josef Bacik (7):
      treewide: make vm_ops->open() return an int
      mm/vma: fail the split if vm_ops->open() fails
      mm/vma: fail copy_vma() if vm_ops->open() fails
      mm/nommu: fail the split if vm_ops->open() fails
      mm/mmap: fail the fork if vm_ops->open() fails
      mm/hugetlb: fail if the VMA lock can't be allocated
      tools/testing/vma: test a failing vm_ops->open()

 Documentation/filesystems/locking.rst             |  2 +-
 Documentation/gpu/drm-mm.rst                      |  2 +-
 arch/x86/kernel/cpu/sgx/encl.c                    |  6 +-
 drivers/accel/amdxdna/amdxdna_gem.c               |  4 +-
 drivers/comedi/comedi_fops.c                      |  4 +-
 drivers/gpu/drm/drm_gem.c                         |  6 +-
 drivers/gpu/drm/drm_gem_shmem_helper.c            |  4 +-
 drivers/gpu/drm/i915/gem/i915_gem_mman.c          |  4 +-
 drivers/gpu/drm/i915/gem/i915_gem_ttm.c           |  4 +-
 drivers/gpu/drm/panthor/panthor_gem.c             |  4 +-
 drivers/gpu/drm/ttm/ttm_bo_vm.c                   |  4 +-
 drivers/hwtracing/intel_th/msu.c                  |  4 +-
 drivers/hwtracing/stm/core.c                      |  4 +-
 drivers/infiniband/core/uverbs_main.c             |  8 ++-
 drivers/infiniband/sw/rdmavt/mmap.c               |  4 +-
 drivers/iommu/iommufd/main.c                      |  4 +-
 drivers/media/common/videobuf2/videobuf2-memops.c |  6 +-
 drivers/misc/genwqe/card_dev.c                    |  4 +-
 drivers/rapidio/devices/rio_mport_cdev.c          |  4 +-
 drivers/scsi/leapraid/leapraid_app.c              |  6 +-
 drivers/staging/media/atomisp/pci/hmm/hmm_bo.c    |  6 +-
 drivers/staging/vme_user/vme_user.c               |  4 +-
 drivers/target/target_core_user.c                 |  4 +-
 drivers/usb/core/devio.c                          |  4 +-
 drivers/usb/mon/mon_bin.c                         |  4 +-
 drivers/video/fbdev/core/fb_defio.c               |  4 +-
 drivers/video/fbdev/omap2/omapfb/omapfb-main.c    |  4 +-
 drivers/video/fbdev/udlfb.c                       |  4 +-
 drivers/xen/gntalloc.c                            |  6 +-
 drivers/xen/gntdev.c                              |  4 +-
 drivers/xen/privcmd-buf.c                         |  6 +-
 fs/afs/file.c                                     |  6 +-
 fs/coda/file.c                                    |  9 ++-
 fs/kernfs/file.c                                  | 11 +--
 include/drm/drm_gem.h                             |  2 +-
 include/drm/ttm/ttm_bo.h                          |  2 +-
 include/linux/mm.h                                |  9 ++-
 ipc/shm.c                                         | 13 ++--
 kernel/bpf/arena.c                                |  4 +-
 kernel/bpf/syscall.c                              |  4 +-
 kernel/events/core.c                              |  4 +-
 kernel/trace/trace.c                              |  4 +-
 mm/hugetlb.c                                      | 40 +++++------
 mm/internal.h                                     | 15 +++++
 mm/mmap.c                                         |  8 ++-
 mm/nommu.c                                        |  6 +-
 mm/vma.c                                          | 15 +++--
 net/packet/af_packet.c                            |  4 +-
 sound/core/pcm_native.c                           |  4 +-
 sound/usb/usx2y/usx2yhwdeppcm.c                   |  3 +-
 tools/testing/vma/include/custom.h                |  3 +-
 tools/testing/vma/include/dup.h                   | 16 ++++-
 tools/testing/vma/tests/merge.c                   | 81 +++++++++++++++++++++++
 53 files changed, 304 insertions(+), 97 deletions(-)
---
base-commit: d0b3eb0ef4ff466d8bc2b8ee77a0ac4386abe75b
change-id: 20261117-b4-vma-open-fail-c3305250ce51



^ permalink raw reply	[flat|nested] 9+ messages in thread

end of thread, other threads:[~2026-11-22 16:05 UTC | newest]

Thread overview: 9+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2026-11-18 17:00 [PATCH RFC 0/7] mm: let vm_ops->open() fail for hugetlb Josef Bacik
2026-11-18 17:00 ` [PATCH RFC 1/7] treewide: make vm_ops->open() return an int Josef Bacik
2026-11-18 17:00 ` [PATCH RFC 2/7] mm/vma: fail the split if vm_ops->open() fails Josef Bacik
2026-11-18 17:00 ` [PATCH RFC 3/7] mm/vma: fail copy_vma() " Josef Bacik
2026-11-18 17:00 ` [PATCH RFC 4/7] mm/nommu: fail the split " Josef Bacik
2026-11-18 17:00 ` [PATCH RFC 5/7] mm/mmap: fail the fork " Josef Bacik
2026-11-18 17:00 ` [PATCH RFC 6/7] mm/hugetlb: fail if the VMA lock can't be allocated Josef Bacik
2026-11-18 17:00 ` [PATCH RFC 7/7] tools/testing/vma: test a failing vm_ops->open() Josef Bacik
2026-11-22 16:05 ` [PATCH RFC 0/7] mm: let vm_ops->open() fail for hugetlb David Hildenbrand (Arm)

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox