[PATCH v2 3/3] mm/vmalloc: make vm_struct.nr_pages an unsigned long

[Artem Lytkin <iprintercanon@gmail.com>]

vm_struct::nr_pages is an unsigned int, and the file repeatedly derives a
byte count from it as nr_pages << PAGE_SHIFT. A shift is evaluated in the
type of its promoted left operand, so those expressions are 32-bit
arithmetic and wrap at 4 GiB of bytes, which is only 2^20 pages with
4 KiB pages. Every such site therefore depends on someone remembering a
cast. vmap() has one. The two commits fixed by the preceding two patches
did not.

v1 of this patch was ambiguous about how much of that has been seen in
practice, so to be explicit: none of it comes from a bug report, the
series comes out of reading the code. What is reachable on current
hardware is the 4 GiB wrap. A single vzalloc() of 4 GiB needs a machine
with more than 4 GiB of memory and nothing else, and /proc/kcore then
reads such an area as zeros while reporting a successful read. The
vrealloc() case has the same cause, but no in-tree caller grows an
allocation that far, so it stays latent. The two thresholds above that
are not equally far away. A count above 2^32, where the field itself
truncates, is out of reach: that is 16 TiB, and as Matthew Wilcox notes,
current hardware cannot populate it, topping out near 12 TiB with eight
fully loaded sockets. The 2^31 point, where the plain int page indexes
below stop working, is 8 TiB, which such a machine could in principle
hand to one allocation, since the only size gate on a request is the
totalram_pages() check in __vmalloc_node_range_noprof(). No in-tree caller
comes near it, so that one is unexercised rather than impossible.

Widening the field is still worth doing, because it retires the 4 GiB
class of bug at the type level instead of one call site at a time, and
the casts go with it.

Widen the field and everything that feeds or consumes it:

  - vm_area_alloc_pages() takes and returns a page count that is stored
    into nr_pages, so its parameter, its return type and the nr_allocated
    and nr_remaining accumulators widen with it. The 100 page cap on a
    bulk request stays narrow, only its literal becomes 100UL.

  - nr_small_pages in __vmalloc_area_node() is derived from a 64-bit size
    and compared against nr_pages, so it widens too. Before this the
    store truncated at 16 TiB, which sized area->pages from the wrapped
    count while the mapping still covered the full size. The cast on
    array_size next to it was already redundant and goes away with it.

  - new_nr_pages and old_nr_pages in vrealloc_node_align_noprof() widen,
    which drops four casts, and the casts added by the two preceding
    patches are no longer needed either.

  - vm_area_free_pages() takes a page index range.

Three counters that index area->pages were plain int and would have
overflowed at 2^31 pages: in set_area_direct_map(), in vm_reset_perms()
and in the bulk allocation loop, where the index is seeded from
nr_allocated. They become unsigned long.

Two prints needed fixing. vmalloc_dump_obj() used %u, and
vmalloc_info_show() printed the unsigned field with %d, which would have
shown a negative page count. show_numa_info() used a single unsigned int
as both the page index and the node id, so the page loop gets its own
counter.

The mapping path is deliberately left alone. vmap_pages_pte_range() and
the levels above it carry the page cursor as int *nr, seeded in
vmap_pages_range_noflush_walk(). That is not a cap: past 2^31 base pages
it overflows and indexes area->pages with a negative value. It is also
not made worse here, because an unsigned int nr_pages already spanned
2^31 to 2^32-1, and nothing about a request's size was ever gated on the
width of that cursor. Widening it touches every level of the walker and
is separate work.

Nothing outside mm/vmalloc.c needs a change to build or to behave as
before. Most of those users index the pages array or shift the count by
PAGE_SHIFT, which is unsigned long arithmetic either way. Three hand it
to a narrower parameter: set_memory_rox() in
execmem_cache_populate_alloc() and release_pages() in
free_mod_tags_mem() take an int page count, and clear_pages() in
alloc_thread_stack_node() takes an unsigned int. Two of those already
narrowed an unsigned int, and none of the three can approach 2^31: a
thread stack is THREAD_SIZE / PAGE_SIZE pages, and an execmem block and
the module tag area are both far smaller. kexec_handover stores the count
into a 32-bit field of its own ABI, which caps that interface exactly
where it did before.

On x86-64 sizeof(struct vm_struct) stays 72 bytes with
CONFIG_HAVE_ARCH_HUGE_VMALLOC=n, since the existing padding before
phys_addr absorbs the change, and becomes 80 with it enabled. Both sizes
come from the same kmalloc-96 bucket that __get_vm_area_node() allocates
from, so the footprint per area does not change either way.

Suggested-by: Andrew Morton <akpm@linux-foundation.org>
Assisted-by: Claude:claude-fable-5
Signed-off-by: Artem Lytkin <iprintercanon@gmail.com>
---
 include/linux/vmalloc.h |  2 +-
 mm/vmalloc.c            | 62 ++++++++++++++++++++---------------------
 2 files changed, 31 insertions(+), 33 deletions(-)

diff --git a/include/linux/vmalloc.h b/include/linux/vmalloc.h
index e4d8d0a9f30f9..aed121d729b01 100644
--- a/include/linux/vmalloc.h
+++ b/include/linux/vmalloc.h
@@ -62,7 +62,7 @@ struct vm_struct {
 #ifdef CONFIG_HAVE_ARCH_HUGE_VMALLOC
 	unsigned int		page_order;
 #endif
-	unsigned int		nr_pages;
+	unsigned long		nr_pages;
 	phys_addr_t		phys_addr;
 	const void		*caller;
 	unsigned long		requested_size;
diff --git a/mm/vmalloc.c b/mm/vmalloc.c
index bf9e32cf93fc9..196da8738cf10 100644
--- a/mm/vmalloc.c
+++ b/mm/vmalloc.c
@@ -3404,7 +3404,7 @@ struct vm_struct *remove_vm_area(const void *addr)
 static inline void set_area_direct_map(const struct vm_struct *area,
 				       int (*set_direct_map)(struct page *page))
 {
-	int i;
+	unsigned long i;
 
 	/* HUGE_VMALLOC passes small pages to set_direct_map */
 	for (i = 0; i < area->nr_pages; i++)
@@ -3420,7 +3420,7 @@ static void vm_reset_perms(struct vm_struct *area)
 	unsigned long start = ULONG_MAX, end = 0;
 	unsigned int page_order = vm_area_page_order(area);
 	int flush_dmap = 0;
-	int i;
+	unsigned long i;
 
 	/*
 	 * Find the start and end range of the direct mappings to make sure that
@@ -3493,10 +3493,10 @@ void vfree_atomic(const void *addr)
  * Caller is responsible for unmapping (vunmap_range) and KASAN
  * poisoning before calling this.
  */
-static void vm_area_free_pages(struct vm_struct *vm, unsigned int start_idx,
-			       unsigned int end_idx)
+static void vm_area_free_pages(struct vm_struct *vm, unsigned long start_idx,
+			       unsigned long end_idx)
 {
-	unsigned int i;
+	unsigned long i;
 
 	if (!(vm->flags & VM_MAP_PUT_PAGES)) {
 		for (i = start_idx; i < end_idx; i++)
@@ -3819,12 +3819,12 @@ static inline gfp_t vmalloc_gfp_adjust(gfp_t flags, const bool large)
 	return flags;
 }
 
-static inline unsigned int
+static inline unsigned long
 vm_area_alloc_pages(gfp_t gfp, int nid,
-		unsigned int order, unsigned int nr_pages, struct page **pages)
+		unsigned int order, unsigned long nr_pages, struct page **pages)
 {
-	unsigned int nr_allocated = 0;
-	unsigned int nr_remaining = nr_pages;
+	unsigned long nr_allocated = 0;
+	unsigned long nr_remaining = nr_pages;
 	unsigned int max_attempt_order = MAX_PAGE_ORDER;
 	struct page *page;
 	int i;
@@ -3872,7 +3872,7 @@ vm_area_alloc_pages(gfp_t gfp, int nid,
 	if (!order) {
 		while (nr_allocated < nr_pages) {
 			unsigned int nr, nr_pages_request;
-			int i;
+			unsigned long i;
 
 			/*
 			 * A maximum allowed request is hard-coded and is 100
@@ -3880,7 +3880,7 @@ vm_area_alloc_pages(gfp_t gfp, int nid,
 			 * long preemption off scenario in the bulk-allocator
 			 * so the range is [1:100].
 			 */
-			nr_pages_request = min(100U, nr_pages - nr_allocated);
+			nr_pages_request = min(100UL, nr_pages - nr_allocated);
 
 			/* memory allocation should consider mempolicy, we can't
 			 * wrongly use nearest node when nid == NUMA_NO_NODE,
@@ -4026,12 +4026,12 @@ static void *__vmalloc_area_node(struct vm_struct *area, gfp_t gfp_mask,
 	unsigned long addr = (unsigned long)area->addr;
 	unsigned long size = get_vm_area_size(area);
 	unsigned long array_size;
-	unsigned int nr_small_pages = size >> PAGE_SHIFT;
+	unsigned long nr_small_pages = size >> PAGE_SHIFT;
 	unsigned int page_order;
 	unsigned int flags;
 	int ret;
 
-	array_size = (unsigned long)nr_small_pages * sizeof(struct page *);
+	array_size = nr_small_pages * sizeof(struct page *);
 
 	/* __GFP_NOFAIL and "noblock" flags are mutually exclusive. */
 	if (!gfpflags_allow_blocking(gfp_mask))
@@ -4525,7 +4525,7 @@ void *vrealloc_node_align_noprof(const void *p, size_t size, unsigned long align
 	}
 
 	if (size <= old_size) {
-		unsigned int new_nr_pages = PAGE_ALIGN(size) >> PAGE_SHIFT;
+		unsigned long new_nr_pages = PAGE_ALIGN(size) >> PAGE_SHIFT;
 
 		/* Zero out "freed" memory, potentially for future realloc. */
 		if (want_init_on_free() || want_init_on_alloc(flags))
@@ -4554,7 +4554,7 @@ void *vrealloc_node_align_noprof(const void *p, size_t size, unsigned long align
 		    !(vm->flags & (VM_FLUSH_RESET_PERMS | VM_USERMAP)) &&
 		    gfp_has_io_fs(flags)) {
 			unsigned long addr = (unsigned long)kasan_reset_tag(p);
-			unsigned int old_nr_pages = vm->nr_pages;
+			unsigned long old_nr_pages = vm->nr_pages;
 
 			/*
 			 * Use the node lock to synchronize with concurrent
@@ -4567,16 +4567,13 @@ void *vrealloc_node_align_noprof(const void *p, size_t size, unsigned long align
 			spin_unlock(&vn->busy.lock);
 
 			/* Notify kmemleak of the reduced allocation size before unmapping. */
-			kmemleak_free_part(
-				(void *)addr + ((unsigned long)new_nr_pages
-						<< PAGE_SHIFT),
-				(unsigned long)(old_nr_pages - new_nr_pages)
-					<< PAGE_SHIFT);
+			kmemleak_free_part((void *)addr +
+					   (new_nr_pages << PAGE_SHIFT),
+					   (old_nr_pages - new_nr_pages)
+						<< PAGE_SHIFT);
 
-			vunmap_range(addr + ((unsigned long)new_nr_pages
-					     << PAGE_SHIFT),
-				     addr + ((unsigned long)old_nr_pages
-					     << PAGE_SHIFT));
+			vunmap_range(addr + (new_nr_pages << PAGE_SHIFT),
+				     addr + (old_nr_pages << PAGE_SHIFT));
 
 			vm_area_free_pages(vm, new_nr_pages, old_nr_pages);
 		}
@@ -4588,7 +4585,7 @@ void *vrealloc_node_align_noprof(const void *p, size_t size, unsigned long align
 	/*
 	 * We already have the bytes available in the allocation; use them.
 	 */
-	if (size <= (unsigned long)vm->nr_pages << PAGE_SHIFT) {
+	if (size <= vm->nr_pages << PAGE_SHIFT) {
 		/*
 		 * No need to zero memory here, as unused memory will have
 		 * already been zeroed at initial allocation time or during
@@ -4895,7 +4892,7 @@ long vread_iter(struct iov_iter *iter, const char *addr, size_t count)
 			 * mapping types (vmap, ioremap) don't set nr_pages.
 			 */
 			size = (vm->flags & VM_ALLOC && vm->nr_pages) ?
-				       ((unsigned long)vm->nr_pages << PAGE_SHIFT) :
+				       (vm->nr_pages << PAGE_SHIFT) :
 				       get_vm_area_size(vm);
 		else
 			size = va_size(va);
@@ -5400,7 +5397,7 @@ bool vmalloc_dump_obj(void *object)
 	struct vmap_area *va;
 	struct vmap_node *vn;
 	unsigned long addr;
-	unsigned int nr_pages;
+	unsigned long nr_pages;
 
 	addr = PAGE_ALIGN((unsigned long) object);
 	vn = addr_to_node(addr);
@@ -5420,7 +5417,7 @@ bool vmalloc_dump_obj(void *object)
 	nr_pages = vm->nr_pages;
 	spin_unlock(&vn->busy.lock);
 
-	pr_cont(" %u-page vmalloc region starting at %#lx allocated at %pS\n",
+	pr_cont(" %lu-page vmalloc region starting at %#lx allocated at %pS\n",
 		nr_pages, addr, caller);
 
 	return true;
@@ -5438,16 +5435,17 @@ bool vmalloc_dump_obj(void *object)
 static void show_numa_info(struct seq_file *m, struct vm_struct *v,
 				 unsigned int *counters)
 {
-	unsigned int nr;
 	unsigned int step = 1U << vm_area_page_order(v);
+	unsigned long i;
+	unsigned int nr;
 
 	if (!counters)
 		return;
 
 	memset(counters, 0, nr_node_ids * sizeof(unsigned int));
 
-	for (nr = 0; nr < v->nr_pages; nr += step)
-		counters[page_to_nid(v->pages[nr])] += step;
+	for (i = 0; i < v->nr_pages; i += step)
+		counters[page_to_nid(v->pages[i])] += step;
 	for_each_node_state(nr, N_HIGH_MEMORY)
 		if (counters[nr])
 			seq_printf(m, " N%u=%u", nr, counters[nr]);
@@ -5505,7 +5503,7 @@ static int vmalloc_info_show(struct seq_file *m, void *p)
 				seq_printf(m, " %pS", v->caller);
 
 			if (v->nr_pages)
-				seq_printf(m, " pages=%d", v->nr_pages);
+				seq_printf(m, " pages=%lu", v->nr_pages);
 
 			if (v->phys_addr)
 				seq_printf(m, " phys=%pa", &v->phys_addr);
-- 
2.43.0