* [PATCH 7.1.y 1/6] bpf: Support for hardening against JIT spraying
2026-07-09 22:22 [PATCH 7.1.y 0/6] cBPF JIT spray hardening Pawan Gupta
@ 2026-07-09 22:23 ` Pawan Gupta
2026-07-09 22:52 ` bot+bpf-ci
2026-07-09 22:23 ` [PATCH 7.1.y 2/6] x86/bugs: Enable IBPB flush on BPF JIT allocation Pawan Gupta
` (5 subsequent siblings)
6 siblings, 1 reply; 9+ messages in thread
From: Pawan Gupta @ 2026-07-09 22:23 UTC (permalink / raw)
To: stable, Greg Kroah-Hartman, Sasha Levin
Cc: bpf, linux-arm-kernel, loongarch, linuxppc-dev, linux-riscv, x86,
Alexei Starovoitov, Daniel Borkmann
commit 96cce16e26dd02a8678f1e87f88a4b5cdb63b995 upstream.
The BPF JIT allocator packs many small programs into larger executable
allocations and reuses space within those allocations as programs are
loaded and freed. When fresh code is written into space that a previous
program occupied, an indirect jump into the new program can reuse a branch
prediction left behind by the old one.
Flush the indirect branch predictors before reusing JIT memory so that
indirect jumps into a newly written program don't reuse predictions from an
old program that occupied the same space.
Introduce bpf_arch_pred_flush_enabled static key and bpf_arch_pred_flush
static call for flushing the branch predictors on JIT memory reuse.
Architectures that need a flush, can update it to a predictor flush
function. By default, its a NOP and does not emit any CALL.
Allocations larger than a pack are not covered by this flush. That is safe
because cBPF programs (the unprivileged attack surface) are bounded well
below a pack size. Issue a warning if this assumption is ever violated
while the flush is active.
Signed-off-by: Pawan Gupta <pawan.kumar.gupta@linux.intel.com>
Acked-by: Daniel Borkmann <daniel@iogearbox.net>
Signed-off-by: Daniel Borkmann <daniel@iogearbox.net>
---
include/linux/filter.h | 10 ++++++++++
kernel/bpf/core.c | 19 +++++++++++++++++++
2 files changed, 29 insertions(+)
diff --git a/include/linux/filter.h b/include/linux/filter.h
index 88a241aac36a..2a7e6cbbbe1d 100644
--- a/include/linux/filter.h
+++ b/include/linux/filter.h
@@ -21,6 +21,7 @@
#include <linux/if_vlan.h>
#include <linux/vmalloc.h>
#include <linux/sockptr.h>
+#include <linux/static_call.h>
#include <linux/u64_stats_sync.h>
#include <net/sch_generic.h>
@@ -1291,6 +1292,15 @@ extern long bpf_jit_limit_max;
typedef void (*bpf_jit_fill_hole_t)(void *area, unsigned int size);
+/*
+ * Flush the indirect branch predictors before reusing JIT memory, so that
+ * indirect jumps into a newly written program don't reuse predictions left
+ * behind by an old program that occupied the same space.
+ */
+void bpf_arch_pred_flush(void);
+DECLARE_STATIC_CALL(bpf_arch_pred_flush, bpf_arch_pred_flush);
+DECLARE_STATIC_KEY_FALSE(bpf_pred_flush_enabled);
+
void bpf_jit_fill_hole_with_zero(void *area, unsigned int size);
struct bpf_binary_header *
diff --git a/kernel/bpf/core.c b/kernel/bpf/core.c
index 6aa2a8b24030..f49b9b23f95e 100644
--- a/kernel/bpf/core.c
+++ b/kernel/bpf/core.c
@@ -20,6 +20,7 @@
#include <uapi/linux/btf.h>
#include <linux/filter.h>
#include <linux/skbuff.h>
+#include <linux/static_call.h>
#include <linux/vmalloc.h>
#include <linux/prandom.h>
#include <linux/bpf.h>
@@ -883,6 +884,15 @@ void bpf_jit_fill_hole_with_zero(void *area, unsigned int size)
memset(area, 0, size);
}
+DEFINE_STATIC_CALL_NULL(bpf_arch_pred_flush, bpf_arch_pred_flush);
+
+/*
+ * Enabled once bpf_arch_pred_flush points at a real flush routine. Lets the
+ * pack allocator test "is a predictor flush wired up at all" with a cheap
+ * static branch instead of repeatedly querying the static call target.
+ */
+DEFINE_STATIC_KEY_FALSE(bpf_pred_flush_enabled);
+
#define BPF_PROG_SIZE_TO_NBITS(size) (round_up(size, BPF_PROG_CHUNK_SIZE) / BPF_PROG_CHUNK_SIZE)
static DEFINE_MUTEX(pack_mutex);
@@ -941,6 +951,14 @@ void *bpf_prog_pack_alloc(u32 size, bpf_jit_fill_hole_t bpf_fill_ill_insns)
mutex_lock(&pack_mutex);
if (size > BPF_PROG_PACK_SIZE) {
+ /*
+ * Allocations larger than a pack get their own pages, and
+ * predictors are not flushed for such allocation. This is only
+ * safe because cBPF programs (the unprivileged attack surface)
+ * are bounded well below a pack size.
+ */
+ if (static_branch_unlikely(&bpf_pred_flush_enabled))
+ pr_warn_once("BPF: Predictors not flushed for allocations greater than BPF_PROG_PACK_SIZE\n");
size = round_up(size, PAGE_SIZE);
ptr = bpf_jit_alloc_exec(size);
if (ptr) {
@@ -971,6 +989,7 @@ void *bpf_prog_pack_alloc(u32 size, bpf_jit_fill_hole_t bpf_fill_ill_insns)
pos = 0;
found_free_area:
+ static_call_cond(bpf_arch_pred_flush)();
bitmap_set(pack->bitmap, pos, nbits);
ptr = (void *)(pack->ptr) + (pos << BPF_PROG_CHUNK_SHIFT);
--
2.43.0
_______________________________________________
linux-riscv mailing list
linux-riscv@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-riscv
^ permalink raw reply related [flat|nested] 9+ messages in thread* Re: [PATCH 7.1.y 1/6] bpf: Support for hardening against JIT spraying
2026-07-09 22:23 ` [PATCH 7.1.y 1/6] bpf: Support for hardening against JIT spraying Pawan Gupta
@ 2026-07-09 22:52 ` bot+bpf-ci
0 siblings, 0 replies; 9+ messages in thread
From: bot+bpf-ci @ 2026-07-09 22:52 UTC (permalink / raw)
To: pawan.kumar.gupta, stable, gregkh, sashal
Cc: bpf, linux-arm-kernel, loongarch, linuxppc-dev, linux-riscv, x86,
ast, daniel, ast, andrii, daniel, martin.lau, eddyz87,
yonghong.song, clm, ihor.solodrai
[-- Attachment #1: Type: text/plain, Size: 1929 bytes --]
> commit 96cce16e26dd02a8678f1e87f88a4b5cdb63b995 upstream.
> The BPF JIT allocator packs many small programs into larger executable
> allocations and reuses space within those allocations as programs are
> loaded and freed. When fresh code is written into space that a previous
> program occupied, an indirect jump into the new program can reuse a branch
> prediction left behind by the old one.
> Flush the indirect branch predictors before reusing JIT memory so that
> indirect jumps into a newly written program don't reuse predictions from an
> old program that occupied the same space.
> Introduce bpf_arch_pred_flush_enabled static key and bpf_arch_pred_flush
> static call for flushing the branch predictors on JIT memory reuse.
> Architectures that need a flush, can update it to a predictor flush
> function. By default, its a NOP and does not emit any CALL.
> Allocations larger than a pack are not covered by this flush. That is safe
> because cBPF programs (the unprivileged attack surface) are bounded well
> below a pack size. Issue a warning if this assumption is ever violated
> while the flush is active.
> Signed-off-by: Pawan Gupta <pawan.kumar.gupta@linux.intel.com>
> Acked-by: Daniel Borkmann <daniel@iogearbox.net>
> Signed-off-by: Daniel Borkmann <daniel@iogearbox.net>
The reuse path this patch hardens (bpf_prog_pack_alloc() finding a free
area within a pack, and the size > BPF_PROG_PACK_SIZE branch) comes from
the pack allocator that first packed many programs into shared executable
allocations and reused freed space within a pack.
Should this carry a Fixes: tag pointing at that commit?
Fixes: 57631054fae6 ("bpf: Introduce bpf_prog_pack allocator")
---
AI reviewed your patch. Please fix the bug or email reply why it's not a bug.
See: https://github.com/kernel-patches/vmtest/blob/master/ci/claude/README.md
CI run summary: https://github.com/kernel-patches/bpf/actions/runs/29054648899
[-- Attachment #2: Type: text/plain, Size: 161 bytes --]
_______________________________________________
linux-riscv mailing list
linux-riscv@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-riscv
^ permalink raw reply [flat|nested] 9+ messages in thread
* [PATCH 7.1.y 2/6] x86/bugs: Enable IBPB flush on BPF JIT allocation
2026-07-09 22:22 [PATCH 7.1.y 0/6] cBPF JIT spray hardening Pawan Gupta
2026-07-09 22:23 ` [PATCH 7.1.y 1/6] bpf: Support for hardening against JIT spraying Pawan Gupta
@ 2026-07-09 22:23 ` Pawan Gupta
2026-07-09 22:23 ` [PATCH 7.1.y 3/6] bpf: Restrict JIT predictor flush to cBPF Pawan Gupta
` (4 subsequent siblings)
6 siblings, 0 replies; 9+ messages in thread
From: Pawan Gupta @ 2026-07-09 22:23 UTC (permalink / raw)
To: stable, Greg Kroah-Hartman, Sasha Levin
Cc: bpf, linux-arm-kernel, loongarch, linuxppc-dev, linux-riscv, x86,
Alexei Starovoitov, Daniel Borkmann, Dave Hansen
commit a3af84b0fa00ead01fcd0e28b5d773ff25990a0d upstream.
Enable hardening against JIT spraying when Spectre-v2 mitigations are in
use. Specifically, issue an IBPB flush on BPF JIT memory reuse. Skip
enabling the IBPB flush if the BPF dispatcher is already using a retpoline
sequence.
This hardening applies only when BPF-JIT is in use. Guard the enabling
under CONFIG_BPF_JIT so that bugs.c still builds with CONFIG_BPF_JIT=n.
Signed-off-by: Pawan Gupta <pawan.kumar.gupta@linux.intel.com>
Acked-by: Daniel Borkmann <daniel@iogearbox.net>
Acked-by: Dave Hansen <dave.hansen@linux.intel.com>
Signed-off-by: Daniel Borkmann <daniel@iogearbox.net>
---
arch/x86/include/asm/nospec-branch.h | 4 +++
arch/x86/kernel/cpu/bugs.c | 50 ++++++++++++++++++++++++++++++++----
2 files changed, 49 insertions(+), 5 deletions(-)
diff --git a/arch/x86/include/asm/nospec-branch.h b/arch/x86/include/asm/nospec-branch.h
index 4f4b5e8a1574..b68892e6d58c 100644
--- a/arch/x86/include/asm/nospec-branch.h
+++ b/arch/x86/include/asm/nospec-branch.h
@@ -388,6 +388,10 @@ extern void srso_alias_return_thunk(void);
extern void entry_untrain_ret(void);
extern void write_ibpb(void);
+#ifdef CONFIG_BPF_JIT
+extern void bpf_arch_ibpb(void);
+#endif
+
#ifdef CONFIG_X86_64
extern void clear_bhb_loop(void);
#endif
diff --git a/arch/x86/kernel/cpu/bugs.c b/arch/x86/kernel/cpu/bugs.c
index 83f51cab0b1e..d9af230c0512 100644
--- a/arch/x86/kernel/cpu/bugs.c
+++ b/arch/x86/kernel/cpu/bugs.c
@@ -16,6 +16,7 @@
#include <linux/sched/smt.h>
#include <linux/pgtable.h>
#include <linux/bpf.h>
+#include <linux/filter.h>
#include <linux/kvm_types.h>
#include <asm/spec-ctrl.h>
@@ -1651,8 +1652,21 @@ static inline const char *spectre_v2_module_string(void)
{
return spectre_v2_bad_module ? " - vulnerable module loaded" : "";
}
+
+/*
+ * The "retpoline sequence" is the "call;mov;ret" sequence that
+ * replaces normal indirect branch instructions. Differentiate
+ * *the* retpoline sequence from the LFENCE-prefixed indirect
+ * branches that simply use the retpoline infrastructure.
+ */
+static inline bool retpoline_seq_enabled(void)
+{
+ return boot_cpu_has(X86_FEATURE_RETPOLINE) && !boot_cpu_has(X86_FEATURE_RETPOLINE_LFENCE);
+}
+
#else
static inline const char *spectre_v2_module_string(void) { return ""; }
+static inline bool retpoline_seq_enabled(void) { return false; }
#endif
#define SPECTRE_V2_LFENCE_MSG "WARNING: LFENCE mitigation is not recommended for this CPU, data leaks possible!\n"
@@ -2095,8 +2109,7 @@ static void __init bhi_apply_mitigation(void)
return;
/* Retpoline mitigates against BHI unless the CPU has RRSBA behavior */
- if (boot_cpu_has(X86_FEATURE_RETPOLINE) &&
- !boot_cpu_has(X86_FEATURE_RETPOLINE_LFENCE)) {
+ if (retpoline_seq_enabled()) {
spec_ctrl_disable_kernel_rrsba();
if (rrsba_disabled)
return;
@@ -2238,6 +2251,27 @@ static void __init spectre_v2_update_mitigation(void)
pr_info("%s\n", spectre_v2_strings[spectre_v2_enabled]);
}
+#ifdef CONFIG_BPF_JIT
+static void __bpf_arch_ibpb(void *unused)
+{
+ write_ibpb();
+}
+
+void bpf_arch_ibpb(void)
+{
+ on_each_cpu(__bpf_arch_ibpb, NULL, 1);
+}
+
+static bool __init cpu_wants_ibpb_bpf(void)
+{
+ /* A genuine retpoline already neutralizes ring0 indirect predictions */
+ if (retpoline_seq_enabled())
+ return false;
+
+ return boot_cpu_has(X86_FEATURE_IBPB);
+}
+#endif
+
static void __init spectre_v2_apply_mitigation(void)
{
if (spectre_v2_enabled == SPECTRE_V2_EIBRS && unprivileged_ebpf_enabled())
@@ -2314,6 +2348,14 @@ static void __init spectre_v2_apply_mitigation(void)
setup_force_cpu_cap(X86_FEATURE_USE_IBRS_FW);
pr_info("Enabling Restricted Speculation for firmware calls\n");
}
+
+#ifdef CONFIG_BPF_JIT
+ if (cpu_wants_ibpb_bpf()) {
+ static_call_update(bpf_arch_pred_flush, bpf_arch_ibpb);
+ static_branch_enable(&bpf_pred_flush_enabled);
+ pr_info("Enabling IBPB for BPF\n");
+ }
+#endif
}
static void update_stibp_msr(void * __unused)
@@ -3490,9 +3532,7 @@ static const char *spectre_bhi_state(void)
return "; BHI: BHI_DIS_S";
else if (boot_cpu_has(X86_FEATURE_CLEAR_BHB_LOOP))
return "; BHI: SW loop, KVM: SW loop";
- else if (boot_cpu_has(X86_FEATURE_RETPOLINE) &&
- !boot_cpu_has(X86_FEATURE_RETPOLINE_LFENCE) &&
- rrsba_disabled)
+ else if (retpoline_seq_enabled() && rrsba_disabled)
return "; BHI: Retpoline";
else if (boot_cpu_has(X86_FEATURE_CLEAR_BHB_VMEXIT))
return "; BHI: Vulnerable, KVM: SW loop";
--
2.43.0
_______________________________________________
linux-riscv mailing list
linux-riscv@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-riscv
^ permalink raw reply related [flat|nested] 9+ messages in thread* [PATCH 7.1.y 3/6] bpf: Restrict JIT predictor flush to cBPF
2026-07-09 22:22 [PATCH 7.1.y 0/6] cBPF JIT spray hardening Pawan Gupta
2026-07-09 22:23 ` [PATCH 7.1.y 1/6] bpf: Support for hardening against JIT spraying Pawan Gupta
2026-07-09 22:23 ` [PATCH 7.1.y 2/6] x86/bugs: Enable IBPB flush on BPF JIT allocation Pawan Gupta
@ 2026-07-09 22:23 ` Pawan Gupta
2026-07-09 22:23 ` [PATCH 7.1.y 4/6] bpf: Skip redundant IBPB in pack allocator Pawan Gupta
` (3 subsequent siblings)
6 siblings, 0 replies; 9+ messages in thread
From: Pawan Gupta @ 2026-07-09 22:23 UTC (permalink / raw)
To: stable, Greg Kroah-Hartman, Sasha Levin
Cc: bpf, linux-arm-kernel, loongarch, linuxppc-dev, linux-riscv, x86,
Alexei Starovoitov, Daniel Borkmann
commit 0bb99f2cfaae6822d734d69722de30af823efdf3 upstream.
Currently predictor flush on memory reuse is done for all BPF JIT
allocations, but only cBPF programs can be loaded by an unprivileged user.
eBPF is privileged by default, and flushing predictors for all CPUs on
every eBPF reuse penalizes the common case for no security benefit.
eBPF allocations can be frequent on busy systems, only flush predictors
for cBPF programs. Trampoline and dispatcher allocations also skip the
flush as they are eBPF-only.
Signed-off-by: Pawan Gupta <pawan.kumar.gupta@linux.intel.com>
Acked-by: Daniel Borkmann <daniel@iogearbox.net>
Signed-off-by: Daniel Borkmann <daniel@iogearbox.net>
---
arch/arm64/net/bpf_jit_comp.c | 4 ++--
arch/loongarch/net/bpf_jit.c | 5 +++--
arch/powerpc/net/bpf_jit_comp.c | 4 ++--
arch/riscv/net/bpf_jit_comp64.c | 2 +-
arch/riscv/net/bpf_jit_core.c | 3 ++-
arch/x86/net/bpf_jit_comp.c | 5 +++--
include/linux/filter.h | 5 +++--
kernel/bpf/core.c | 13 ++++++++-----
kernel/bpf/dispatcher.c | 2 +-
9 files changed, 25 insertions(+), 18 deletions(-)
diff --git a/arch/arm64/net/bpf_jit_comp.c b/arch/arm64/net/bpf_jit_comp.c
index 0816c40fc7af..9491c987e50c 100644
--- a/arch/arm64/net/bpf_jit_comp.c
+++ b/arch/arm64/net/bpf_jit_comp.c
@@ -2094,7 +2094,7 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_verifier_env *env, struct bpf_pr
image_size = extable_offset + extable_size;
ro_header = bpf_jit_binary_pack_alloc(image_size, &ro_image_ptr,
sizeof(u64), &header, &image_ptr,
- jit_fill_hole);
+ jit_fill_hole, was_classic);
if (!ro_header)
goto out_off;
@@ -2782,7 +2782,7 @@ int arch_bpf_trampoline_size(const struct btf_func_model *m, u32 flags,
void *arch_alloc_bpf_trampoline(unsigned int size)
{
- return bpf_prog_pack_alloc(size, jit_fill_hole);
+ return bpf_prog_pack_alloc(size, jit_fill_hole, false);
}
void arch_free_bpf_trampoline(void *image, unsigned int size)
diff --git a/arch/loongarch/net/bpf_jit.c b/arch/loongarch/net/bpf_jit.c
index 24913dc7f4e8..e14a8aa47fc8 100644
--- a/arch/loongarch/net/bpf_jit.c
+++ b/arch/loongarch/net/bpf_jit.c
@@ -1762,7 +1762,7 @@ static int invoke_bpf(struct jit_ctx *ctx, struct bpf_tramp_links *tl,
void *arch_alloc_bpf_trampoline(unsigned int size)
{
- return bpf_prog_pack_alloc(size, jit_fill_hole);
+ return bpf_prog_pack_alloc(size, jit_fill_hole, false);
}
void arch_free_bpf_trampoline(void *image, unsigned int size)
@@ -2228,7 +2228,8 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_verifier_env *env, struct bpf_pr
image_size = prog_size + extable_size;
/* Now we know the size of the structure to make */
ro_header = bpf_jit_binary_pack_alloc(image_size, &ro_image_ptr, sizeof(u32),
- &header, &image_ptr, jit_fill_hole);
+ &header, &image_ptr, jit_fill_hole,
+ bpf_prog_was_classic(prog));
if (!ro_header)
goto out_offset;
diff --git a/arch/powerpc/net/bpf_jit_comp.c b/arch/powerpc/net/bpf_jit_comp.c
index 53ab97ad6074..73ff8a64bb7f 100644
--- a/arch/powerpc/net/bpf_jit_comp.c
+++ b/arch/powerpc/net/bpf_jit_comp.c
@@ -295,7 +295,7 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_verifier_env *env, struct bpf_pr
alloclen = proglen + FUNCTION_DESCR_SIZE + fixup_len + extable_len;
fhdr = bpf_jit_binary_pack_alloc(alloclen, &fimage, 4, &hdr, &image,
- bpf_jit_fill_ill_insns);
+ bpf_jit_fill_ill_insns, bpf_prog_was_classic(fp));
if (!fhdr)
goto out_err;
@@ -583,7 +583,7 @@ bool bpf_jit_inlines_helper_call(s32 imm)
void *arch_alloc_bpf_trampoline(unsigned int size)
{
- return bpf_prog_pack_alloc(size, bpf_jit_fill_ill_insns);
+ return bpf_prog_pack_alloc(size, bpf_jit_fill_ill_insns, false);
}
void arch_free_bpf_trampoline(void *image, unsigned int size)
diff --git a/arch/riscv/net/bpf_jit_comp64.c b/arch/riscv/net/bpf_jit_comp64.c
index 2f1109dbf105..404a98a0fc90 100644
--- a/arch/riscv/net/bpf_jit_comp64.c
+++ b/arch/riscv/net/bpf_jit_comp64.c
@@ -1321,7 +1321,7 @@ int arch_bpf_trampoline_size(const struct btf_func_model *m, u32 flags,
void *arch_alloc_bpf_trampoline(unsigned int size)
{
- return bpf_prog_pack_alloc(size, bpf_fill_ill_insns);
+ return bpf_prog_pack_alloc(size, bpf_fill_ill_insns, false);
}
void arch_free_bpf_trampoline(void *image, unsigned int size)
diff --git a/arch/riscv/net/bpf_jit_core.c b/arch/riscv/net/bpf_jit_core.c
index 4365d07aaf54..ce3bd3762e08 100644
--- a/arch/riscv/net/bpf_jit_core.c
+++ b/arch/riscv/net/bpf_jit_core.c
@@ -109,7 +109,8 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_verifier_env *env, struct bpf_pr
bpf_jit_binary_pack_alloc(prog_size + extable_size,
&jit_data->ro_image, sizeof(u32),
&jit_data->header, &jit_data->image,
- bpf_fill_ill_insns);
+ bpf_fill_ill_insns,
+ bpf_prog_was_classic(prog));
if (!jit_data->ro_header)
goto out_offset;
diff --git a/arch/x86/net/bpf_jit_comp.c b/arch/x86/net/bpf_jit_comp.c
index ea9e707e8abf..ef8b112fd7b6 100644
--- a/arch/x86/net/bpf_jit_comp.c
+++ b/arch/x86/net/bpf_jit_comp.c
@@ -3520,7 +3520,7 @@ static int __arch_prepare_bpf_trampoline(struct bpf_tramp_image *im, void *rw_im
void *arch_alloc_bpf_trampoline(unsigned int size)
{
- return bpf_prog_pack_alloc(size, jit_fill_hole);
+ return bpf_prog_pack_alloc(size, jit_fill_hole, false);
}
void arch_free_bpf_trampoline(void *image, unsigned int size)
@@ -3831,7 +3831,8 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_verifier_env *env, struct bpf_pr
/* allocate module memory for x86 insns and extable */
header = bpf_jit_binary_pack_alloc(roundup(proglen, align) + extable_size,
&image, align, &rw_header, &rw_image,
- jit_fill_hole);
+ jit_fill_hole,
+ bpf_prog_was_classic(prog));
if (!header)
goto out_addrs;
prog->aux->extable = (void *) image + roundup(proglen, align);
diff --git a/include/linux/filter.h b/include/linux/filter.h
index 2a7e6cbbbe1d..ea654453e43a 100644
--- a/include/linux/filter.h
+++ b/include/linux/filter.h
@@ -1315,7 +1315,7 @@ void bpf_jit_free(struct bpf_prog *fp);
struct bpf_binary_header *
bpf_jit_binary_pack_hdr(const struct bpf_prog *fp);
-void *bpf_prog_pack_alloc(u32 size, bpf_jit_fill_hole_t bpf_fill_ill_insns);
+void *bpf_prog_pack_alloc(u32 size, bpf_jit_fill_hole_t bpf_fill_ill_insns, bool was_classic);
void bpf_prog_pack_free(void *ptr, u32 size);
static inline bool bpf_prog_kallsyms_verify_off(const struct bpf_prog *fp)
@@ -1329,7 +1329,8 @@ bpf_jit_binary_pack_alloc(unsigned int proglen, u8 **ro_image,
unsigned int alignment,
struct bpf_binary_header **rw_hdr,
u8 **rw_image,
- bpf_jit_fill_hole_t bpf_fill_ill_insns);
+ bpf_jit_fill_hole_t bpf_fill_ill_insns,
+ bool was_classic);
int bpf_jit_binary_pack_finalize(struct bpf_binary_header *ro_header,
struct bpf_binary_header *rw_header);
void bpf_jit_binary_pack_free(struct bpf_binary_header *ro_header,
diff --git a/kernel/bpf/core.c b/kernel/bpf/core.c
index f49b9b23f95e..1b7e74e63bd4 100644
--- a/kernel/bpf/core.c
+++ b/kernel/bpf/core.c
@@ -942,7 +942,7 @@ static struct bpf_prog_pack *alloc_new_pack(bpf_jit_fill_hole_t bpf_fill_ill_ins
return NULL;
}
-void *bpf_prog_pack_alloc(u32 size, bpf_jit_fill_hole_t bpf_fill_ill_insns)
+void *bpf_prog_pack_alloc(u32 size, bpf_jit_fill_hole_t bpf_fill_ill_insns, bool was_classic)
{
unsigned int nbits = BPF_PROG_SIZE_TO_NBITS(size);
struct bpf_prog_pack *pack;
@@ -957,7 +957,7 @@ void *bpf_prog_pack_alloc(u32 size, bpf_jit_fill_hole_t bpf_fill_ill_insns)
* safe because cBPF programs (the unprivileged attack surface)
* are bounded well below a pack size.
*/
- if (static_branch_unlikely(&bpf_pred_flush_enabled))
+ if (was_classic && static_branch_unlikely(&bpf_pred_flush_enabled))
pr_warn_once("BPF: Predictors not flushed for allocations greater than BPF_PROG_PACK_SIZE\n");
size = round_up(size, PAGE_SIZE);
ptr = bpf_jit_alloc_exec(size);
@@ -989,7 +989,9 @@ void *bpf_prog_pack_alloc(u32 size, bpf_jit_fill_hole_t bpf_fill_ill_insns)
pos = 0;
found_free_area:
- static_call_cond(bpf_arch_pred_flush)();
+ /* Flush only for cBPF as it may contain a crafted gadget */
+ if (static_branch_unlikely(&bpf_pred_flush_enabled) && was_classic)
+ static_call_cond(bpf_arch_pred_flush)();
bitmap_set(pack->bitmap, pos, nbits);
ptr = (void *)(pack->ptr) + (pos << BPF_PROG_CHUNK_SHIFT);
@@ -1149,7 +1151,8 @@ bpf_jit_binary_pack_alloc(unsigned int proglen, u8 **image_ptr,
unsigned int alignment,
struct bpf_binary_header **rw_header,
u8 **rw_image,
- bpf_jit_fill_hole_t bpf_fill_ill_insns)
+ bpf_jit_fill_hole_t bpf_fill_ill_insns,
+ bool was_classic)
{
struct bpf_binary_header *ro_header;
u32 size, hole, start;
@@ -1162,7 +1165,7 @@ bpf_jit_binary_pack_alloc(unsigned int proglen, u8 **image_ptr,
if (bpf_jit_charge_modmem(size))
return NULL;
- ro_header = bpf_prog_pack_alloc(size, bpf_fill_ill_insns);
+ ro_header = bpf_prog_pack_alloc(size, bpf_fill_ill_insns, was_classic);
if (!ro_header) {
bpf_jit_uncharge_modmem(size);
return NULL;
diff --git a/kernel/bpf/dispatcher.c b/kernel/bpf/dispatcher.c
index b77db7413f8c..ea2d60dc1fee 100644
--- a/kernel/bpf/dispatcher.c
+++ b/kernel/bpf/dispatcher.c
@@ -145,7 +145,7 @@ void bpf_dispatcher_change_prog(struct bpf_dispatcher *d, struct bpf_prog *from,
mutex_lock(&d->mutex);
if (!d->image) {
- d->image = bpf_prog_pack_alloc(PAGE_SIZE, bpf_jit_fill_hole_with_zero);
+ d->image = bpf_prog_pack_alloc(PAGE_SIZE, bpf_jit_fill_hole_with_zero, false);
if (!d->image)
goto out;
d->rw_image = bpf_jit_alloc_exec(PAGE_SIZE);
--
2.43.0
_______________________________________________
linux-riscv mailing list
linux-riscv@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-riscv
^ permalink raw reply related [flat|nested] 9+ messages in thread* [PATCH 7.1.y 4/6] bpf: Skip redundant IBPB in pack allocator
2026-07-09 22:22 [PATCH 7.1.y 0/6] cBPF JIT spray hardening Pawan Gupta
` (2 preceding siblings ...)
2026-07-09 22:23 ` [PATCH 7.1.y 3/6] bpf: Restrict JIT predictor flush to cBPF Pawan Gupta
@ 2026-07-09 22:23 ` Pawan Gupta
2026-07-09 22:24 ` [PATCH 7.1.y 5/6] bpf: Prefer packs that won't trigger an IBPB flush on allocation Pawan Gupta
` (2 subsequent siblings)
6 siblings, 0 replies; 9+ messages in thread
From: Pawan Gupta @ 2026-07-09 22:23 UTC (permalink / raw)
To: stable, Greg Kroah-Hartman, Sasha Levin
Cc: bpf, linux-arm-kernel, loongarch, linuxppc-dev, linux-riscv, x86,
Alexei Starovoitov, Daniel Borkmann
commit a23c1c5396a91680703360d1ee28a44657c503c4 upstream.
bpf_prog_pack_alloc() issues IBPB on all CPUs on every cBPF allocation,
even when reusing chunks from an existing pack where no new memory was
touched since the last IBPB.
Since IBPB on all CPUs is heavy, Dave Hansen suggested to track allocation
since last IBPB, and only issue IBPB at reuse for the chunks that have not
seen an IBPB since they were last freed.
Track per-pack whether an IBPB is needed via arch_flush_needed. Set it when
allocating a chunk, reset on IBPB flush. On reuse, conditionally issue the
flush. Since IBPB invalidates all BTB entries, clear the flag on all packs
after flushing.
Signed-off-by: Pawan Gupta <pawan.kumar.gupta@linux.intel.com>
Acked-by: Daniel Borkmann <daniel@iogearbox.net>
Signed-off-by: Daniel Borkmann <daniel@iogearbox.net>
---
kernel/bpf/core.c | 15 ++++++++++++++-
1 file changed, 14 insertions(+), 1 deletion(-)
diff --git a/kernel/bpf/core.c b/kernel/bpf/core.c
index 1b7e74e63bd4..1bf4b8dffe91 100644
--- a/kernel/bpf/core.c
+++ b/kernel/bpf/core.c
@@ -876,6 +876,7 @@ int bpf_jit_add_poke_descriptor(struct bpf_prog *prog,
struct bpf_prog_pack {
struct list_head list;
void *ptr;
+ bool arch_flush_needed;
unsigned long bitmap[];
};
@@ -928,6 +929,8 @@ static struct bpf_prog_pack *alloc_new_pack(bpf_jit_fill_hole_t bpf_fill_ill_ins
bpf_fill_ill_insns(pack->ptr, BPF_PROG_PACK_SIZE);
bitmap_zero(pack->bitmap, BPF_PROG_PACK_SIZE / BPF_PROG_CHUNK_SIZE);
+ if (static_branch_unlikely(&bpf_pred_flush_enabled))
+ pack->arch_flush_needed = true;
set_vm_flush_reset_perms(pack->ptr);
err = set_memory_rox((unsigned long)pack->ptr,
BPF_PROG_PACK_SIZE / PAGE_SIZE);
@@ -990,8 +993,15 @@ void *bpf_prog_pack_alloc(u32 size, bpf_jit_fill_hole_t bpf_fill_ill_insns, bool
found_free_area:
/* Flush only for cBPF as it may contain a crafted gadget */
- if (static_branch_unlikely(&bpf_pred_flush_enabled) && was_classic)
+ if (static_branch_unlikely(&bpf_pred_flush_enabled) &&
+ pack->arch_flush_needed &&
+ was_classic) {
+ struct bpf_prog_pack *p;
+
static_call_cond(bpf_arch_pred_flush)();
+ list_for_each_entry(p, &pack_list, list)
+ p->arch_flush_needed = false;
+ }
bitmap_set(pack->bitmap, pos, nbits);
ptr = (void *)(pack->ptr) + (pos << BPF_PROG_CHUNK_SHIFT);
@@ -1029,6 +1039,9 @@ void bpf_prog_pack_free(void *ptr, u32 size)
"bpf_prog_pack bug: missing bpf_arch_text_invalidate?\n");
bitmap_clear(pack->bitmap, pos, nbits);
+
+ if (static_branch_unlikely(&bpf_pred_flush_enabled))
+ pack->arch_flush_needed = true;
if (bitmap_find_next_zero_area(pack->bitmap, BPF_PROG_CHUNK_COUNT, 0,
BPF_PROG_CHUNK_COUNT, 0) == 0) {
list_del(&pack->list);
--
2.43.0
_______________________________________________
linux-riscv mailing list
linux-riscv@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-riscv
^ permalink raw reply related [flat|nested] 9+ messages in thread* [PATCH 7.1.y 5/6] bpf: Prefer packs that won't trigger an IBPB flush on allocation
2026-07-09 22:22 [PATCH 7.1.y 0/6] cBPF JIT spray hardening Pawan Gupta
` (3 preceding siblings ...)
2026-07-09 22:23 ` [PATCH 7.1.y 4/6] bpf: Skip redundant IBPB in pack allocator Pawan Gupta
@ 2026-07-09 22:24 ` Pawan Gupta
2026-07-09 22:24 ` [PATCH 7.1.y 6/6] bpf: Prefer dirty packs for eBPF allocations Pawan Gupta
2026-07-10 21:03 ` [PATCH 7.1.y 0/6] cBPF JIT spray hardening Sasha Levin
6 siblings, 0 replies; 9+ messages in thread
From: Pawan Gupta @ 2026-07-09 22:24 UTC (permalink / raw)
To: stable, Greg Kroah-Hartman, Sasha Levin
Cc: bpf, linux-arm-kernel, loongarch, linuxppc-dev, linux-riscv, x86,
Alexei Starovoitov, Daniel Borkmann
commit a9b1f19a6a673ba06820898d0f1ad02883ea1639 upstream.
Currently BPF pack allocator picks the chunks from the first available
pack. While this is okay, it naturally leads to more frequent flushes
when there are multiple packs in the system that weren't used since the
last flush.
As an optimization prefer allocating the new programs from packs that
are unused since last flush. When all packs are dirty, allocation forces
a flush and marks all packs clean.
Below are some future optimizations ideas:
1. Currently, the "dirty" tracking is only done at the pack-level.
Flush frequency can further be reduced with chunk-level tracking.
This requires a new bitmap per-pack to track the dirty state.
2. IBPB flush is done on all CPUs, even if only a single CPU ran the
BPF program. On a system with hundreds of CPUs this could be a
major bottleneck forcing hundreds of IPIs to deliver the flush.
The solution is to track the CPUs where a BPF program ran, and
issue IBPB only on those CPUs.
3. Avoid IBPB when flush is already done at other sources (e.g.
context switch).
Signed-off-by: Pawan Gupta <pawan.kumar.gupta@linux.intel.com>
Acked-by: Daniel Borkmann <daniel@iogearbox.net>
Signed-off-by: Daniel Borkmann <daniel@iogearbox.net>
---
kernel/bpf/core.c | 27 ++++++++++++++++++++++++---
1 file changed, 24 insertions(+), 3 deletions(-)
diff --git a/kernel/bpf/core.c b/kernel/bpf/core.c
index 1bf4b8dffe91..bba4acd61d41 100644
--- a/kernel/bpf/core.c
+++ b/kernel/bpf/core.c
@@ -948,8 +948,8 @@ static struct bpf_prog_pack *alloc_new_pack(bpf_jit_fill_hole_t bpf_fill_ill_ins
void *bpf_prog_pack_alloc(u32 size, bpf_jit_fill_hole_t bpf_fill_ill_insns, bool was_classic)
{
unsigned int nbits = BPF_PROG_SIZE_TO_NBITS(size);
- struct bpf_prog_pack *pack;
- unsigned long pos;
+ struct bpf_prog_pack *pack, *fallback_pack = NULL;
+ unsigned long pos, fallback_pos = 0;
void *ptr = NULL;
mutex_lock(&pack_mutex);
@@ -981,8 +981,29 @@ void *bpf_prog_pack_alloc(u32 size, bpf_jit_fill_hole_t bpf_fill_ill_insns, bool
list_for_each_entry(pack, &pack_list, list) {
pos = bitmap_find_next_zero_area(pack->bitmap, BPF_PROG_CHUNK_COUNT, 0,
nbits, 0);
- if (pos < BPF_PROG_CHUNK_COUNT)
+ if (pos >= BPF_PROG_CHUNK_COUNT)
+ continue;
+ /* Flush not enabled, use any pack */
+ if (!static_branch_unlikely(&bpf_pred_flush_enabled))
goto found_free_area;
+ /*
+ * cBPF reuse of a dirty pack triggers a flush, so prefer a
+ * clean pack for cBPF. eBPF never flushes, so pick the first
+ * free pack, dirty or clean.
+ */
+ if (!was_classic || !pack->arch_flush_needed)
+ goto found_free_area;
+ if (!fallback_pack) {
+ fallback_pack = pack;
+ fallback_pos = pos;
+ }
+ }
+
+ /* No preferred pack found */
+ if (fallback_pack) {
+ pack = fallback_pack;
+ pos = fallback_pos;
+ goto found_free_area;
}
pack = alloc_new_pack(bpf_fill_ill_insns);
--
2.43.0
_______________________________________________
linux-riscv mailing list
linux-riscv@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-riscv
^ permalink raw reply related [flat|nested] 9+ messages in thread* [PATCH 7.1.y 6/6] bpf: Prefer dirty packs for eBPF allocations
2026-07-09 22:22 [PATCH 7.1.y 0/6] cBPF JIT spray hardening Pawan Gupta
` (4 preceding siblings ...)
2026-07-09 22:24 ` [PATCH 7.1.y 5/6] bpf: Prefer packs that won't trigger an IBPB flush on allocation Pawan Gupta
@ 2026-07-09 22:24 ` Pawan Gupta
2026-07-10 21:03 ` [PATCH 7.1.y 0/6] cBPF JIT spray hardening Sasha Levin
6 siblings, 0 replies; 9+ messages in thread
From: Pawan Gupta @ 2026-07-09 22:24 UTC (permalink / raw)
To: stable, Greg Kroah-Hartman, Sasha Levin
Cc: bpf, linux-arm-kernel, loongarch, linuxppc-dev, linux-riscv, x86,
Alexei Starovoitov, Daniel Borkmann
commit b72e29e0f7ee329d89f86db8700c8ea99b4a370a upstream.
The pack allocator only flushes predictors when reusing a dirty pack for
cBPF, eBPF allocations never trigger a flush. Currently, eBPF picks the
first free pack, which could be a clean pack. As an optimization, leaving
a clean pack for cBPF can avoid flushes.
Prefer dirty packs for eBPF and keep clean packs free for cBPF. This
mirrors the existing cBPF preference for clean packs: each program kind
prefers the pack that avoids an extra flush, and falls back to the other
kind only when no preferred pack has room. eBPF reuse of a dirty pack is
harmless since eBPF being privileged does not flush.
Signed-off-by: Pawan Gupta <pawan.kumar.gupta@linux.intel.com>
Acked-by: Daniel Borkmann <daniel@iogearbox.net>
Signed-off-by: Daniel Borkmann <daniel@iogearbox.net>
---
kernel/bpf/core.c | 6 +++---
1 file changed, 3 insertions(+), 3 deletions(-)
diff --git a/kernel/bpf/core.c b/kernel/bpf/core.c
index bba4acd61d41..de61e1894452 100644
--- a/kernel/bpf/core.c
+++ b/kernel/bpf/core.c
@@ -988,10 +988,10 @@ void *bpf_prog_pack_alloc(u32 size, bpf_jit_fill_hole_t bpf_fill_ill_insns, bool
goto found_free_area;
/*
* cBPF reuse of a dirty pack triggers a flush, so prefer a
- * clean pack for cBPF. eBPF never flushes, so pick the first
- * free pack, dirty or clean.
+ * clean pack for cBPF. eBPF never flushes, so steer it to a
+ * dirty pack and keep clean packs free for cBPF.
*/
- if (!was_classic || !pack->arch_flush_needed)
+ if (was_classic ^ pack->arch_flush_needed)
goto found_free_area;
if (!fallback_pack) {
fallback_pack = pack;
--
2.43.0
_______________________________________________
linux-riscv mailing list
linux-riscv@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-riscv
^ permalink raw reply related [flat|nested] 9+ messages in thread* Re: [PATCH 7.1.y 0/6] cBPF JIT spray hardening
2026-07-09 22:22 [PATCH 7.1.y 0/6] cBPF JIT spray hardening Pawan Gupta
` (5 preceding siblings ...)
2026-07-09 22:24 ` [PATCH 7.1.y 6/6] bpf: Prefer dirty packs for eBPF allocations Pawan Gupta
@ 2026-07-10 21:03 ` Sasha Levin
6 siblings, 0 replies; 9+ messages in thread
From: Sasha Levin @ 2026-07-10 21:03 UTC (permalink / raw)
To: stable, Greg Kroah-Hartman
Cc: Sasha Levin, bpf, linux-arm-kernel, loongarch, linuxppc-dev,
linux-riscv, x86, Alexei Starovoitov, Daniel Borkmann,
Dave Hansen, Pawan Gupta
On Thu, Jul 09, 2026 at 03:22:54PM -0700, Pawan Gupta wrote:
> These backports harden BPF JIT against spectre-v2 class of attacks. Without
> a predictor flush, execution of new BPF program may use stale prediction
> left behind by the freed one.
>
> To avoid this, issue an IBPB flush on all CPUs on JIT program allocation.
> The flush is conditional to spectre-v2 mitigation applied.
Queued the series for 7.1, thanks.
--
Thanks,
Sasha
_______________________________________________
linux-riscv mailing list
linux-riscv@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-riscv
^ permalink raw reply [flat|nested] 9+ messages in thread