[PATCH] hugetlb: block hugetlb file creation if hugetlb is not set up

[Jann Horn <jannh@google.com>]

Many distro kernels enable hugetlb support, but most systems running
those kernels never actually allocate hugepages or enable hugetlb
overcommit.

On such systems, hugetlb is unusable for any legitimate usecase, but it
is still possible to exercise a lot of hugetlb-specific code by creating
MAP_HUGETLB|MAP_NORESERVE VMAs - for example, it is still possible to
create page tables shared across processes.

This is exposed through the mmap() syscall, with no privileges required,
so from a security perspective, this is interesting attack surface.

Lock it down by completely denying creation of hugetlb files if no huge
pages for the hstate could be allocated without administratively
changing huge page limits.

hstate_is_enabled() is written based on documentation in
Documentation/admin-guide/sysctl/vm.rst and
Documentation/admin-guide/mm/hugetlbpage.rst , in particular this:

> nr_overcommit_hugepages
> =======================
>
> Change the maximum size of the hugepage pool. The maximum is
> nr_hugepages + nr_overcommit_hugepages.

and this:

> As long as this condition holds--that is, until
> ``nr_hugepages+nr_overcommit_hugepages`` is increased sufficiently, or
> the surplus huge pages go out of use and are freed-- no more surplus
> huge pages will be allowed to be allocated.

Note that, in the userspace API:

 - `h->nr_overcommit_huge_pages` is called "nr_overcommit_hugepages"
 - `h->max_huge_pages` is called "nr_hugepages"

I am not explicitly marking this for stable backport yet at this point,
but I will want to backport this once it's landed in a point release and
nobody's complained for a while.

Signed-off-by: Jann Horn <jannh@google.com>
---
@akpm: no rush with this one; probably makes sense to wait for an ack
from a hugetlb person before queueing it up, and then send it through
mm-unstable like a feature patch.

@Lorenzo: I'm just CCing you as an FYI in case you're interested, it
doesn't touch any code outside hugetlb
---
 fs/hugetlbfs/inode.c | 19 +++++++++++++++++++
 1 file changed, 19 insertions(+)

diff --git a/fs/hugetlbfs/inode.c b/fs/hugetlbfs/inode.c
index e4de5425838d..fc03dd541b4d 100644
--- a/fs/hugetlbfs/inode.c
+++ b/fs/hugetlbfs/inode.c
@@ -1517,6 +1517,16 @@ static int get_hstate_idx(int page_size_log)
 	return hstate_index(h);
 }
 
+static bool hstate_is_enabled(struct hstate *h)
+{
+	bool is_enabled;
+
+	spin_lock_irq(&hugetlb_lock);
+	is_enabled = h->nr_overcommit_huge_pages || h->max_huge_pages;
+	spin_unlock_irq(&hugetlb_lock);
+	return is_enabled;
+}
+
 /*
  * Note that size should be aligned to proper hugepage size in caller side,
  * otherwise hugetlb_reserve_pages reserves one less hugepages than intended.
@@ -1549,6 +1559,15 @@ struct file *hugetlb_file_setup(const char *name, size_t size,
 		return ERR_PTR(-EPERM);
 	}
 
+	/*
+	 * If no hugetlb pages of this size are supposed to exist, then don't
+	 * even allow creating a hugetlb file (even if the file has size 0 or
+	 * userspace requests MAP_NORESERVE).
+	 * This limits attack surface for systems that don't use hugetlb.
+	 */
+	if (!hstate_is_enabled(HUGETLBFS_SB(mnt->mnt_sb)->hstate))
+		return ERR_PTR(-ENOMEM);
+
 	file = ERR_PTR(-ENOSPC);
 	/* hugetlbfs_vfsmount[] mounts do not use idmapped mounts.  */
 	inode = hugetlbfs_get_inode(mnt->mnt_sb, &nop_mnt_idmap, NULL,

---
base-commit: b1456f6dc167f7f101746e495bede2bac3d0e19f
change-id: 20250524-hugetlb-nerf-cc125f7fc187

-- 
Jann Horn <jannh@google.com>