[PATCH] bpf: Fix alignment of memory allocator objects on 32-bit archs

[PATCH] bpf: Fix alignment of memory allocator objects on 32-bit archs

[Felix Fietkau]

The BPF memory allocator prefixes each object with a header that holds
struct llist_node while the object sits on a free list, and a pointer
to the owning bpf_mem_cache while it is allocated. The header size was
sizeof(struct llist_node), so on 32-bit architectures the memory handed
out was offset by 4 bytes from the 8-byte aligned kmalloc allocation.

The verifier assumes that allocated objects are 8-byte aligned and
allows BPF_DW atomic instructions on their u64 fields, e.g. on values
of non-preallocated hash maps, objects from bpf_obj_new() and local
storage. On arm32 the JIT does not implement atomics, and the
interpreter executes them via atomic64_*(), which are implemented with
LDREXD/STREXD on ARMv6K and later. Those instructions require 8-byte
aligned addresses; a misaligned address raises an alignment fault and
crashes the kernel.

Pad the header to 8 bytes to preserve the alignment provided by
kmalloc. Store the percpu pointer of percpu objects at the new header
offset instead of assuming the header is pointer-sized, and update the
copy of the header size used by the hash map memory usage accounting.

Fixes: 7c8199e24fa0 ("bpf: Introduce any context BPF specific memory allocator.")
Reported-by: Thibaut VARÈNE <hacks@slashdirt.org>
Reported-by: Stijn Tintel <stijn@linux-ipv6.be>
Signed-off-by: Felix Fietkau <nbd@nbd.name>
---
 kernel/bpf/hashtab.c  | 2 +-
 kernel/bpf/memalloc.c | 8 ++++----
 2 files changed, 5 insertions(+), 5 deletions(-)

diff --git a/kernel/bpf/hashtab.c b/kernel/bpf/hashtab.c
index 3dd9b4924ae4..900c1851d264 100644
--- a/kernel/bpf/hashtab.c
+++ b/kernel/bpf/hashtab.c
@@ -2337,7 +2337,7 @@ static u64 htab_map_mem_usage(const struct bpf_map *map)
 		else if (!lru)
 			usage += sizeof(struct htab_elem *) * num_possible_cpus();
 	} else {
-#define LLIST_NODE_SZ sizeof(struct llist_node)
+#define LLIST_NODE_SZ ALIGN(sizeof(struct llist_node), 8)
 
 		num_entries = htab->use_percpu_counter ?
 					  percpu_counter_sum(&htab->pcount) :
diff --git a/kernel/bpf/memalloc.c b/kernel/bpf/memalloc.c
index e9662db7198f..ca7555b1da34 100644
--- a/kernel/bpf/memalloc.c
+++ b/kernel/bpf/memalloc.c
@@ -33,7 +33,7 @@
  * Every allocated objected is padded with extra 8 bytes that contains
  * struct llist_node.
  */
-#define LLIST_NODE_SZ sizeof(struct llist_node)
+#define LLIST_NODE_SZ ALIGN(sizeof(struct llist_node), 8)
 
 #define BPF_MEM_ALLOC_SIZE_MAX 4096
 
@@ -142,7 +142,7 @@ static struct llist_node notrace *__llist_del_first(struct llist_head *head)
 static void *__alloc(struct bpf_mem_cache *c, int node, gfp_t flags)
 {
 	if (c->percpu_size) {
-		void __percpu **obj = kmalloc_node(c->percpu_size, flags, node);
+		void *obj = kmalloc_node(c->percpu_size, flags, node);
 		void __percpu *pptr = __alloc_percpu_gfp(c->unit_size, 8, flags);
 
 		if (!obj || !pptr) {
@@ -150,7 +150,7 @@ static void *__alloc(struct bpf_mem_cache *c, int node, gfp_t flags)
 			kfree(obj);
 			return NULL;
 		}
-		obj[1] = pptr;
+		*(void __percpu **)(obj + LLIST_NODE_SZ) = pptr;
 		return obj;
 	}
 
@@ -257,7 +257,7 @@ static void alloc_bulk(struct bpf_mem_cache *c, int cnt, int node, bool atomic)
 static void free_one(void *obj, bool percpu)
 {
 	if (percpu)
-		free_percpu(((void __percpu **)obj)[1]);
+		free_percpu(*(void __percpu **)(obj + LLIST_NODE_SZ));
 
 	kfree(obj);
 }
-- 
2.51.0

Re: [PATCH] bpf: Fix alignment of memory allocator objects on 32-bit archs

[sashiko-bot]

Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:

Pre-existing issues:
- [High] On 32-bit architectures, the verifier's hardcoded 8-byte load (BPF_DW) for bpf_this_cpu_ptr()/bpf_per_cpu_ptr() causes a KASAN out-of-bounds read past the 4-byte percpu pointer and passes garbage data on Big Endian systems.
--

--- Patch [1]: [PATCH] bpf: Fix alignment of memory allocator objects on 32-bit archs ---
commit ed5f9ae93bc0fc6463cab51b598b3ca3395812da
Author: Felix Fietkau <nbd@nbd.name>

bpf: Fix alignment of memory allocator objects on 32-bit archs

The BPF memory allocator prefixes each object with a header. On 32-bit
architectures, the memory handed out was offset by 4 bytes. This patch pads
the header to 8 bytes to preserve alignment and updates the percpu pointer
offset accordingly.

> diff --git a/kernel/bpf/memalloc.c b/kernel/bpf/memalloc.c
> index e9662db7198fe..ca7555b1da34f 100644
> --- a/kernel/bpf/memalloc.c
> +++ b/kernel/bpf/memalloc.c
[ ... ]
> @@ -142,7 +142,7 @@ static void *__alloc(struct bpf_mem_cache *c, int node, gfp_t flags)
>  {
>  	if (c->percpu_size) {
> -		void __percpu **obj = kmalloc_node(c->percpu_size, flags, node);
> +		void *obj = kmalloc_node(c->percpu_size, flags, node);
>  		void __percpu *pptr = __alloc_percpu_gfp(c->unit_size, 8, flags);
>  
>  		if (!obj || !pptr) {
> @@ -150,7 +150,7 @@ static void *__alloc(struct bpf_mem_cache *c, int node, gfp_t flags)
>  			kfree(obj);
>  			return NULL;
>  		}
> -		obj[1] = pptr;
> +		*(void __percpu **)(obj + LLIST_NODE_SZ) = pptr;

[Severity: High]
This is a pre-existing issue, but does this 32-bit pointer write lead to an
out-of-bounds read when accessed by the verifier?

On 32-bit systems, the allocation size for percpu objects is typically 12
bytes (LLIST_NODE_SZ + sizeof(void *)). The percpu pointer is stored at offset
8 using a 32-bit store.

However, the verifier unconditionally patches bpf_per_cpu_ptr() and
bpf_this_cpu_ptr() to read this pointer using an 8-byte load:

kernel/bpf/fixups.c:fixup_bpf_calls() {
    ...
    insn_buf[0] = BPF_LDX_MEM(BPF_DW, BPF_REG_1, BPF_REG_1, 0);
    ...
}

Could this 64-bit read access bytes 8-15 of the 12-byte allocation, causing a
KASAN slab-out-of-bounds read?

Furthermore, on 32-bit Big Endian systems, does this 8-byte load place the
4-byte pointer into the upper 32 bits and the out-of-bounds garbage into the
lower 32 bits, leading to memory corruption or a kernel panic when the garbage
is later dereferenced?

>  		return obj;
>  	}

-- 
Sashiko AI review · https://sashiko.dev/#/patchset/20260717183632.99195-1-nbd@nbd.name?part=1