[RFC PATCH 0/8] uprobe/x86: Add support to optimize prologue

[RFC PATCH 0/8] uprobe/x86: Add support to optimize prologue

[Jiri Olsa]

hi,
the subject is bit too optimistic, in nutshell the idea is to allow
optimization on top of emulated instructions and then add support to
emulate more instructions with high presence in function prologues.

This patchset adds support to optimize uprobe on top of instruction
that could be emulated and also adds support to emulate particular
versions of mov and sub instructions to cover some of the user space
functions prologues, like:

  pushq %rbp
  movq  %rsp,%rbp
  subq  $0xb0,%rsp

The idea is to store instructions on underlying 5 bytes and emulate
them during the int3 and uprobe syscall execution:

  - install 'call trampoline' through standard int3 update
  - if int3 is hit before we finish optimizing we emulate
    all underlying instructions
  - when call is installed the uprobe syscall will emulate
    all underlying instructions

There's an additional issue that single instruction replacement does
not have and it's the possibility of the user space code to jump in the
middle of those 5 bytes. I think it's unlikely to happen at the function
prologue, but uprobe could be placed anywhere. I'm not sure how to
mitigate this other than having some enable/disable switch or config
option, which is unfortunate.

The patchset is based on bpf-next/master with [1] changes merged in.

thanks,
jirka


[1] https://lore.kernel.org/lkml/20251117093137.572132-1-jolsa@kernel.org/T/#m95a3208943ec24c5eee17ad6113002fdc6776cf8
---
Jiri Olsa (8):
      uprobe/x86: Introduce struct arch_uprobe_xol object
      uprobe/x86: Use struct arch_uprobe_xol in emulate callback
      uprobe/x86: Add support to emulate mov reg,reg instructions
      uprobe/x86: Add support to emulate sub imm,reg instructions
      uprobe/x86: Add support to optimize on top of emulated instructions
      selftests/bpf: Add test for mov and sub emulation
      selftests/bpf: Add test for uprobe prologue optimization
      selftests/bpf: Add race test for uprobe proglog optimization

 arch/x86/include/asm/uprobes.h                          |  35 +++++++---
 arch/x86/kernel/uprobes.c                               | 336 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++--------------
 include/linux/uprobes.h                                 |   1 +
 kernel/events/uprobes.c                                 |   6 ++
 tools/testing/selftests/bpf/prog_tests/uprobe_syscall.c | 129 ++++++++++++++++++++++++++++++++-----
 5 files changed, 434 insertions(+), 73 deletions(-)

[RFC PATCH 1/8] uprobe/x86: Introduce struct arch_uprobe_xol object

[Jiri Olsa]

Mov xol data into separate arch_uprobe_xol object so we can
hold more of them in following changes.

Signed-off-by: Jiri Olsa <jolsa@kernel.org>
---
 arch/x86/include/asm/uprobes.h | 16 ++++---
 arch/x86/kernel/uprobes.c      | 78 +++++++++++++++++-----------------
 2 files changed, 49 insertions(+), 45 deletions(-)

diff --git a/arch/x86/include/asm/uprobes.h b/arch/x86/include/asm/uprobes.h
index 1ee2e5115955..819e35aa61c4 100644
--- a/arch/x86/include/asm/uprobes.h
+++ b/arch/x86/include/asm/uprobes.h
@@ -27,12 +27,7 @@ enum {
 
 struct uprobe_xol_ops;
 
-struct arch_uprobe {
-	union {
-		u8			insn[MAX_UINSN_BYTES];
-		u8			ixol[MAX_UINSN_BYTES];
-	};
-
+struct arch_uprobe_xol {
 	const struct uprobe_xol_ops	*ops;
 
 	union {
@@ -50,6 +45,15 @@ struct arch_uprobe {
 			u8	ilen;
 		}			push;
 	};
+};
+
+struct arch_uprobe {
+	union {
+		u8			insn[MAX_UINSN_BYTES];
+		u8			ixol[MAX_UINSN_BYTES];
+	};
+
+	struct arch_uprobe_xol		xol;
 
 	unsigned long flags;
 };
diff --git a/arch/x86/kernel/uprobes.c b/arch/x86/kernel/uprobes.c
index 845aeaf36b8d..fb9457b29dbc 100644
--- a/arch/x86/kernel/uprobes.c
+++ b/arch/x86/kernel/uprobes.c
@@ -560,14 +560,14 @@ static void riprel_analyze(struct arch_uprobe *auprobe, struct insn *insn)
 	 */
 	if (reg != 6 && reg2 != 6) {
 		reg2 = 6;
-		auprobe->defparam.fixups |= UPROBE_FIX_RIP_SI;
+		auprobe->xol.defparam.fixups |= UPROBE_FIX_RIP_SI;
 	} else if (reg != 7 && reg2 != 7) {
 		reg2 = 7;
-		auprobe->defparam.fixups |= UPROBE_FIX_RIP_DI;
+		auprobe->xol.defparam.fixups |= UPROBE_FIX_RIP_DI;
 		/* TODO (paranoia): force maskmovq to not use di */
 	} else {
 		reg2 = 3;
-		auprobe->defparam.fixups |= UPROBE_FIX_RIP_BX;
+		auprobe->xol.defparam.fixups |= UPROBE_FIX_RIP_BX;
 	}
 	/*
 	 * Point cursor at the modrm byte.  The next 4 bytes are the
@@ -586,9 +586,9 @@ static void riprel_analyze(struct arch_uprobe *auprobe, struct insn *insn)
 static inline unsigned long *
 scratch_reg(struct arch_uprobe *auprobe, struct pt_regs *regs)
 {
-	if (auprobe->defparam.fixups & UPROBE_FIX_RIP_SI)
+	if (auprobe->xol.defparam.fixups & UPROBE_FIX_RIP_SI)
 		return &regs->si;
-	if (auprobe->defparam.fixups & UPROBE_FIX_RIP_DI)
+	if (auprobe->xol.defparam.fixups & UPROBE_FIX_RIP_DI)
 		return &regs->di;
 	return &regs->bx;
 }
@@ -599,18 +599,18 @@ scratch_reg(struct arch_uprobe *auprobe, struct pt_regs *regs)
  */
 static void riprel_pre_xol(struct arch_uprobe *auprobe, struct pt_regs *regs)
 {
-	if (auprobe->defparam.fixups & UPROBE_FIX_RIP_MASK) {
+	if (auprobe->xol.defparam.fixups & UPROBE_FIX_RIP_MASK) {
 		struct uprobe_task *utask = current->utask;
 		unsigned long *sr = scratch_reg(auprobe, regs);
 
 		utask->autask.saved_scratch_register = *sr;
-		*sr = utask->vaddr + auprobe->defparam.ilen;
+		*sr = utask->vaddr + auprobe->xol.defparam.ilen;
 	}
 }
 
 static void riprel_post_xol(struct arch_uprobe *auprobe, struct pt_regs *regs)
 {
-	if (auprobe->defparam.fixups & UPROBE_FIX_RIP_MASK) {
+	if (auprobe->xol.defparam.fixups & UPROBE_FIX_RIP_MASK) {
 		struct uprobe_task *utask = current->utask;
 		unsigned long *sr = scratch_reg(auprobe, regs);
 
@@ -1265,16 +1265,16 @@ static int default_post_xol_op(struct arch_uprobe *auprobe, struct pt_regs *regs
 	struct uprobe_task *utask = current->utask;
 
 	riprel_post_xol(auprobe, regs);
-	if (auprobe->defparam.fixups & UPROBE_FIX_IP) {
+	if (auprobe->xol.defparam.fixups & UPROBE_FIX_IP) {
 		long correction = utask->vaddr - utask->xol_vaddr;
 		regs->ip += correction;
-	} else if (auprobe->defparam.fixups & UPROBE_FIX_CALL) {
+	} else if (auprobe->xol.defparam.fixups & UPROBE_FIX_CALL) {
 		regs->sp += sizeof_long(regs); /* Pop incorrect return address */
-		if (emulate_push_stack(regs, utask->vaddr + auprobe->defparam.ilen))
+		if (emulate_push_stack(regs, utask->vaddr + auprobe->xol.defparam.ilen))
 			return -ERESTART;
 	}
 	/* popf; tell the caller to not touch TF */
-	if (auprobe->defparam.fixups & UPROBE_FIX_SETF)
+	if (auprobe->xol.defparam.fixups & UPROBE_FIX_SETF)
 		utask->autask.saved_tf = true;
 
 	return 0;
@@ -1293,7 +1293,7 @@ static const struct uprobe_xol_ops default_xol_ops = {
 
 static bool branch_is_call(struct arch_uprobe *auprobe)
 {
-	return auprobe->branch.opc1 == 0xe8;
+	return auprobe->xol.branch.opc1 == 0xe8;
 }
 
 #define CASE_COND					\
@@ -1329,7 +1329,7 @@ static bool check_jmp_cond(struct arch_uprobe *auprobe, struct pt_regs *regs)
 {
 	unsigned long flags = regs->flags;
 
-	switch (auprobe->branch.opc1) {
+	switch (auprobe->xol.branch.opc1) {
 	#define DO(expr)	\
 		return expr;
 	CASE_COND
@@ -1346,8 +1346,8 @@ static bool check_jmp_cond(struct arch_uprobe *auprobe, struct pt_regs *regs)
 
 static bool branch_emulate_op(struct arch_uprobe *auprobe, struct pt_regs *regs)
 {
-	unsigned long new_ip = regs->ip += auprobe->branch.ilen;
-	unsigned long offs = (long)auprobe->branch.offs;
+	unsigned long new_ip = regs->ip += auprobe->xol.branch.ilen;
+	unsigned long offs = (long)auprobe->xol.branch.offs;
 
 	if (branch_is_call(auprobe)) {
 		/*
@@ -1371,11 +1371,11 @@ static bool branch_emulate_op(struct arch_uprobe *auprobe, struct pt_regs *regs)
 
 static bool push_emulate_op(struct arch_uprobe *auprobe, struct pt_regs *regs)
 {
-	unsigned long *src_ptr = (void *)regs + auprobe->push.reg_offset;
+	unsigned long *src_ptr = (void *)regs + auprobe->xol.push.reg_offset;
 
 	if (emulate_push_stack(regs, *src_ptr))
 		return false;
-	regs->ip += auprobe->push.ilen;
+	regs->ip += auprobe->xol.push.ilen;
 	return true;
 }
 
@@ -1469,16 +1469,16 @@ static int branch_setup_xol_ops(struct arch_uprobe *auprobe, struct insn *insn)
 	}
 
 setup:
-	auprobe->branch.opc1 = opc1;
-	auprobe->branch.ilen = insn->length;
-	auprobe->branch.offs = insn->immediate.value;
+	auprobe->xol.branch.opc1 = opc1;
+	auprobe->xol.branch.ilen = insn->length;
+	auprobe->xol.branch.offs = insn->immediate.value;
 
-	auprobe->ops = &branch_xol_ops;
+	auprobe->xol.ops = &branch_xol_ops;
 	return 0;
 }
 
 /* Returns -ENOSYS if push_xol_ops doesn't handle this insn */
-static int push_setup_xol_ops(struct arch_uprobe *auprobe, struct insn *insn)
+static int push_setup_xol_ops(struct arch_uprobe_xol *xol, struct insn *insn)
 {
 	u8 opc1 = OPCODE1(insn), reg_offset = 0;
 
@@ -1552,9 +1552,9 @@ static int push_setup_xol_ops(struct arch_uprobe *auprobe, struct insn *insn)
 		}
 	}
 
-	auprobe->push.reg_offset = reg_offset;
-	auprobe->push.ilen = insn->length;
-	auprobe->ops = &push_xol_ops;
+	xol->push.reg_offset = reg_offset;
+	xol->push.ilen = insn->length;
+	xol->ops = &push_xol_ops;
 	return 0;
 }
 
@@ -1582,7 +1582,7 @@ int arch_uprobe_analyze_insn(struct arch_uprobe *auprobe, struct mm_struct *mm,
 	if (ret != -ENOSYS)
 		return ret;
 
-	ret = push_setup_xol_ops(auprobe, &insn);
+	ret = push_setup_xol_ops(&auprobe->xol, &insn);
 	if (ret != -ENOSYS)
 		return ret;
 
@@ -1592,7 +1592,7 @@ int arch_uprobe_analyze_insn(struct arch_uprobe *auprobe, struct mm_struct *mm,
 	 */
 	switch (OPCODE1(&insn)) {
 	case 0x9d:		/* popf */
-		auprobe->defparam.fixups |= UPROBE_FIX_SETF;
+		auprobe->xol.defparam.fixups |= UPROBE_FIX_SETF;
 		break;
 	case 0xc3:		/* ret or lret -- ip is correct */
 	case 0xcb:
@@ -1618,10 +1618,10 @@ int arch_uprobe_analyze_insn(struct arch_uprobe *auprobe, struct mm_struct *mm,
 		riprel_analyze(auprobe, &insn);
 	}
 
-	auprobe->defparam.ilen = insn.length;
-	auprobe->defparam.fixups |= fix_ip_or_call;
+	auprobe->xol.defparam.ilen = insn.length;
+	auprobe->xol.defparam.fixups |= fix_ip_or_call;
 
-	auprobe->ops = &default_xol_ops;
+	auprobe->xol.ops = &default_xol_ops;
 	return 0;
 }
 
@@ -1634,8 +1634,8 @@ int arch_uprobe_pre_xol(struct arch_uprobe *auprobe, struct pt_regs *regs)
 {
 	struct uprobe_task *utask = current->utask;
 
-	if (auprobe->ops->pre_xol) {
-		int err = auprobe->ops->pre_xol(auprobe, regs);
+	if (auprobe->xol.ops->pre_xol) {
+		int err = auprobe->xol.ops->pre_xol(auprobe, regs);
 		if (err)
 			return err;
 	}
@@ -1686,8 +1686,8 @@ int arch_uprobe_post_xol(struct arch_uprobe *auprobe, struct pt_regs *regs)
 	WARN_ON_ONCE(current->thread.trap_nr != UPROBE_TRAP_NR);
 	current->thread.trap_nr = utask->autask.saved_trap_nr;
 
-	if (auprobe->ops->post_xol) {
-		err = auprobe->ops->post_xol(auprobe, regs);
+	if (auprobe->xol.ops->post_xol) {
+		err = auprobe->xol.ops->post_xol(auprobe, regs);
 		if (err) {
 			/*
 			 * Restore ->ip for restart or post mortem analysis.
@@ -1754,8 +1754,8 @@ void arch_uprobe_abort_xol(struct arch_uprobe *auprobe, struct pt_regs *regs)
 {
 	struct uprobe_task *utask = current->utask;
 
-	if (auprobe->ops->abort)
-		auprobe->ops->abort(auprobe, regs);
+	if (auprobe->xol.ops->abort)
+		auprobe->xol.ops->abort(auprobe, regs);
 
 	current->thread.trap_nr = utask->autask.saved_trap_nr;
 	regs->ip = utask->vaddr;
@@ -1766,8 +1766,8 @@ void arch_uprobe_abort_xol(struct arch_uprobe *auprobe, struct pt_regs *regs)
 
 static bool __skip_sstep(struct arch_uprobe *auprobe, struct pt_regs *regs)
 {
-	if (auprobe->ops->emulate)
-		return auprobe->ops->emulate(auprobe, regs);
+	if (auprobe->xol.ops->emulate)
+		return auprobe->xol.ops->emulate(auprobe, regs);
 	return false;
 }
 
-- 
2.51.1

[RFC PATCH 2/8] uprobe/x86: Use struct arch_uprobe_xol in emulate callback

[Jiri Olsa]

Using struct arch_uprobe_xol also in emulate callback
which will help in following changes.

Signed-off-by: Jiri Olsa <jolsa@kernel.org>
---
 arch/x86/kernel/uprobes.c | 32 +++++++++++++++++---------------
 1 file changed, 17 insertions(+), 15 deletions(-)

diff --git a/arch/x86/kernel/uprobes.c b/arch/x86/kernel/uprobes.c
index fb9457b29dbc..7d7a5e677472 100644
--- a/arch/x86/kernel/uprobes.c
+++ b/arch/x86/kernel/uprobes.c
@@ -1212,7 +1212,7 @@ static bool can_optimize(struct insn *insn, unsigned long vaddr)
 #endif /* CONFIG_X86_64 */
 
 struct uprobe_xol_ops {
-	bool	(*emulate)(struct arch_uprobe *, struct pt_regs *);
+	bool	(*emulate)(struct arch_uprobe*, struct arch_uprobe_xol *, struct pt_regs *);
 	int	(*pre_xol)(struct arch_uprobe *, struct pt_regs *);
 	int	(*post_xol)(struct arch_uprobe *, struct pt_regs *);
 	void	(*abort)(struct arch_uprobe *, struct pt_regs *);
@@ -1291,9 +1291,9 @@ static const struct uprobe_xol_ops default_xol_ops = {
 	.abort	  = default_abort_op,
 };
 
-static bool branch_is_call(struct arch_uprobe *auprobe)
+static bool branch_is_call(struct arch_uprobe_xol *xol)
 {
-	return auprobe->xol.branch.opc1 == 0xe8;
+	return xol->branch.opc1 == 0xe8;
 }
 
 #define CASE_COND					\
@@ -1325,11 +1325,11 @@ static bool is_cond_jmp_opcode(u8 opcode)
 	}
 }
 
-static bool check_jmp_cond(struct arch_uprobe *auprobe, struct pt_regs *regs)
+static bool check_jmp_cond(struct arch_uprobe_xol *xol, struct pt_regs *regs)
 {
 	unsigned long flags = regs->flags;
 
-	switch (auprobe->xol.branch.opc1) {
+	switch (xol->branch.opc1) {
 	#define DO(expr)	\
 		return expr;
 	CASE_COND
@@ -1344,12 +1344,13 @@ static bool check_jmp_cond(struct arch_uprobe *auprobe, struct pt_regs *regs)
 #undef	COND
 #undef	CASE_COND
 
-static bool branch_emulate_op(struct arch_uprobe *auprobe, struct pt_regs *regs)
+static bool branch_emulate_op(struct arch_uprobe *auprobe, struct arch_uprobe_xol *xol,
+			      struct pt_regs *regs)
 {
-	unsigned long new_ip = regs->ip += auprobe->xol.branch.ilen;
-	unsigned long offs = (long)auprobe->xol.branch.offs;
+	unsigned long new_ip = regs->ip += xol->branch.ilen;
+	unsigned long offs = (long)xol->branch.offs;
 
-	if (branch_is_call(auprobe)) {
+	if (branch_is_call(xol)) {
 		/*
 		 * If it fails we execute this (mangled, see the comment in
 		 * branch_clear_offset) insn out-of-line. In the likely case
@@ -1361,7 +1362,7 @@ static bool branch_emulate_op(struct arch_uprobe *auprobe, struct pt_regs *regs)
 		 */
 		if (emulate_push_stack(regs, new_ip))
 			return false;
-	} else if (!check_jmp_cond(auprobe, regs)) {
+	} else if (!check_jmp_cond(xol, regs)) {
 		offs = 0;
 	}
 
@@ -1369,19 +1370,20 @@ static bool branch_emulate_op(struct arch_uprobe *auprobe, struct pt_regs *regs)
 	return true;
 }
 
-static bool push_emulate_op(struct arch_uprobe *auprobe, struct pt_regs *regs)
+static bool push_emulate_op(struct arch_uprobe *auprobe, struct arch_uprobe_xol *xol,
+			    struct pt_regs *regs)
 {
-	unsigned long *src_ptr = (void *)regs + auprobe->xol.push.reg_offset;
+	unsigned long *src_ptr = (void *)regs + xol->push.reg_offset;
 
 	if (emulate_push_stack(regs, *src_ptr))
 		return false;
-	regs->ip += auprobe->xol.push.ilen;
+	regs->ip += xol->push.ilen;
 	return true;
 }
 
 static int branch_post_xol_op(struct arch_uprobe *auprobe, struct pt_regs *regs)
 {
-	BUG_ON(!branch_is_call(auprobe));
+	BUG_ON(!branch_is_call(&auprobe->xol));
 	/*
 	 * We can only get here if branch_emulate_op() failed to push the ret
 	 * address _and_ another thread expanded our stack before the (mangled)
@@ -1767,7 +1769,7 @@ void arch_uprobe_abort_xol(struct arch_uprobe *auprobe, struct pt_regs *regs)
 static bool __skip_sstep(struct arch_uprobe *auprobe, struct pt_regs *regs)
 {
 	if (auprobe->xol.ops->emulate)
-		return auprobe->xol.ops->emulate(auprobe, regs);
+		return auprobe->xol.ops->emulate(auprobe, &auprobe->xol, regs);
 	return false;
 }
 
-- 
2.51.1

[RFC PATCH 3/8] uprobe/x86: Add support to emulate mov reg,reg instructions

[Jiri Olsa]

Adding support to emulate mov reg to reg instructions, because it's
often part of the function prologue.

Signed-off-by: Jiri Olsa <jolsa@kernel.org>
---
 arch/x86/include/asm/uprobes.h |  5 +++
 arch/x86/kernel/uprobes.c      | 61 ++++++++++++++++++++++++++++++++++
 2 files changed, 66 insertions(+)

diff --git a/arch/x86/include/asm/uprobes.h b/arch/x86/include/asm/uprobes.h
index 819e35aa61c4..e6fd87a1cbc3 100644
--- a/arch/x86/include/asm/uprobes.h
+++ b/arch/x86/include/asm/uprobes.h
@@ -44,6 +44,11 @@ struct arch_uprobe_xol {
 			u8	reg_offset;	/* to the start of pt_regs */
 			u8	ilen;
 		}			push;
+		struct {
+			u16	src;	/* to the start of pt_regs */
+			u16	dst;    /* to the start of pt_regs */
+			u8	ilen;
+		}			mov;
 	};
 };
 
diff --git a/arch/x86/kernel/uprobes.c b/arch/x86/kernel/uprobes.c
index 7d7a5e677472..5c44c4b84e99 100644
--- a/arch/x86/kernel/uprobes.c
+++ b/arch/x86/kernel/uprobes.c
@@ -19,6 +19,7 @@
 #include <asm/insn.h>
 #include <asm/mmu_context.h>
 #include <asm/nops.h>
+#include <asm/insn-eval.h>
 
 /* Post-execution fixups. */
 
@@ -1414,6 +1415,19 @@ static void branch_clear_offset(struct arch_uprobe *auprobe, struct insn *insn)
 		0, insn->immediate.nbytes);
 }
 
+static bool mov_emulate_op(struct arch_uprobe *auprobe, struct arch_uprobe_xol *xol,
+			   struct pt_regs *regs)
+{
+	unsigned long *dst, *src;
+
+	dst = (void *) regs + xol->mov.dst;
+	src = (void *) regs + xol->mov.src;
+	*dst = *src;
+
+	regs->ip += xol->mov.ilen;
+	return true;
+}
+
 static const struct uprobe_xol_ops branch_xol_ops = {
 	.emulate  = branch_emulate_op,
 	.post_xol = branch_post_xol_op,
@@ -1423,6 +1437,10 @@ static const struct uprobe_xol_ops push_xol_ops = {
 	.emulate  = push_emulate_op,
 };
 
+static const struct uprobe_xol_ops mov_xol_ops = {
+	.emulate  = mov_emulate_op,
+};
+
 /* Returns -ENOSYS if branch_xol_ops doesn't handle this insn */
 static int branch_setup_xol_ops(struct arch_uprobe *auprobe, struct insn *insn)
 {
@@ -1560,6 +1578,45 @@ static int push_setup_xol_ops(struct arch_uprobe_xol *xol, struct insn *insn)
 	return 0;
 }
 
+#ifdef CONFIG_X86_64
+/* Returns -ENOSYS if mov_xol_ops doesn't handle this insn */
+static int mov_setup_xol_ops(struct arch_uprobe_xol *xol, struct insn *insn)
+{
+	u8 opc1 = OPCODE1(insn);
+	int off_src, off_dst;
+
+	/* validate opcode */
+	if (opc1 != 0x89)
+		return -ENOSYS;
+	if (insn->rex_prefix.nbytes != 1 ||
+	    insn->rex_prefix.bytes[0] != 0x48)
+		return -ENOSYS;
+
+	/* only register operands */
+	if (X86_MODRM_MOD(insn->modrm.value) != 3)
+		return -ENOSYS;
+
+	/* get registers offset */
+	off_src = insn_get_modrm_reg_off(insn);
+	if (off_src < 0)
+		return off_src;
+	off_dst = insn_get_modrm_rm_off(insn);
+	if (off_dst < 0)
+		return off_dst;
+
+	xol->mov.src = off_src;
+	xol->mov.dst = off_dst;
+	xol->mov.ilen = insn->length;
+	xol->ops = &mov_xol_ops;
+	return 0;
+}
+#else
+static int mov_setup_xol_ops(struct arch_uprobe_xol *xol, struct insn *insn)
+{
+	return -ENOSYS;
+}
+#endif
+
 /**
  * arch_uprobe_analyze_insn - instruction analysis including validity and fixups.
  * @auprobe: the probepoint information.
@@ -1588,6 +1645,10 @@ int arch_uprobe_analyze_insn(struct arch_uprobe *auprobe, struct mm_struct *mm,
 	if (ret != -ENOSYS)
 		return ret;
 
+	ret = mov_setup_xol_ops(&auprobe->xol, &insn);
+	if (ret != -ENOSYS)
+		return ret;
+
 	/*
 	 * Figure out which fixups default_post_xol_op() will need to perform,
 	 * and annotate defparam->fixups accordingly.
-- 
2.51.1

[RFC PATCH 4/8] uprobe/x86: Add support to emulate sub imm,reg instructions

[Jiri Olsa]

Adding support to emulate sub reg, imm instructions, because it's
often part of the function prologue.

Signed-off-by: Jiri Olsa <jolsa@kernel.org>
---
 arch/x86/include/asm/uprobes.h |  5 +++
 arch/x86/kernel/uprobes.c      | 73 ++++++++++++++++++++++++++++++++++
 2 files changed, 78 insertions(+)

diff --git a/arch/x86/include/asm/uprobes.h b/arch/x86/include/asm/uprobes.h
index e6fd87a1cbc3..e09aab82b8c1 100644
--- a/arch/x86/include/asm/uprobes.h
+++ b/arch/x86/include/asm/uprobes.h
@@ -49,6 +49,11 @@ struct arch_uprobe_xol {
 			u16	dst;    /* to the start of pt_regs */
 			u8	ilen;
 		}			mov;
+		struct {
+			s32	val;
+			u16	reg;	/* to the start of pt_regs */
+			u8	ilen;
+		}			sub;
 	};
 };
 
diff --git a/arch/x86/kernel/uprobes.c b/arch/x86/kernel/uprobes.c
index 5c44c4b84e99..904c423ea81d 100644
--- a/arch/x86/kernel/uprobes.c
+++ b/arch/x86/kernel/uprobes.c
@@ -1428,6 +1428,40 @@ static bool mov_emulate_op(struct arch_uprobe *auprobe, struct arch_uprobe_xol *
 	return true;
 }
 
+#define EFLAGS_MASK (X86_EFLAGS_OF|X86_EFLAGS_SF|X86_EFLAGS_ZF|X86_EFLAGS_AF|\
+		     X86_EFLAGS_PF|X86_EFLAGS_CF)
+
+static bool sub_emulate_op(struct arch_uprobe *auprobe, struct arch_uprobe_xol *xol,
+			   struct pt_regs *regs)
+{
+	unsigned long dst, flags = regs->flags, val = xol->sub.val;
+	unsigned long *reg = (void *) regs + xol->sub.reg;
+
+	dst = *reg;
+
+	/*
+	 * Emulate sub with 'sub reg,reg' and get result value and
+	 * flags register change. Not sure it's completely equivalent
+	 * to sub reg,imm so perhaps there's better way.
+	 */
+	asm volatile (
+		"pushf \n\t"
+		"push %[flags]; popf \n\t"
+		"subq %[src], %[dst] \n\t"
+		"pushf; popq %[flags] \n\t"
+		"popf \n\t"
+		: [flags] "+D" (flags), [dst] "+r" (dst)
+		: [src] "r" (val)
+	);
+
+	*reg = dst;
+	regs->flags = (regs->flags & ~EFLAGS_MASK) | (flags & EFLAGS_MASK);
+	regs->ip += xol->sub.ilen;
+	return true;
+}
+
+#undef EFLAGS_MASK
+
 static const struct uprobe_xol_ops branch_xol_ops = {
 	.emulate  = branch_emulate_op,
 	.post_xol = branch_post_xol_op,
@@ -1441,6 +1475,10 @@ static const struct uprobe_xol_ops mov_xol_ops = {
 	.emulate  = mov_emulate_op,
 };
 
+static const struct uprobe_xol_ops sub_xol_ops = {
+	.emulate  = sub_emulate_op,
+};
+
 /* Returns -ENOSYS if branch_xol_ops doesn't handle this insn */
 static int branch_setup_xol_ops(struct arch_uprobe *auprobe, struct insn *insn)
 {
@@ -1610,11 +1648,42 @@ static int mov_setup_xol_ops(struct arch_uprobe_xol *xol, struct insn *insn)
 	xol->ops = &mov_xol_ops;
 	return 0;
 }
+
+static int sub_setup_xol_ops(struct arch_uprobe_xol *xol, struct insn *insn)
+{
+	u8 opc1 = OPCODE1(insn);
+	int off;
+
+	if (opc1 != 0x81)
+		return -ENOSYS;
+	if (insn->rex_prefix.nbytes != 1 ||
+	    insn->rex_prefix.bytes[0] != 0x48)
+		return -ENOSYS;
+	if (X86_MODRM_MOD(insn->modrm.value) != 3)
+		return -ENOSYS;
+	if (X86_MODRM_REG(insn->modrm.value) != 5)
+		return -ENOSYS;
+
+	/* get register offset */
+	off = insn_get_modrm_rm_off(insn);
+	if (off < 0)
+		return off;
+
+	xol->sub.reg = off;
+	xol->sub.val = insn->immediate.value;
+	xol->sub.ilen = insn->length;
+	xol->ops = &sub_xol_ops;
+	return 0;
+}
 #else
 static int mov_setup_xol_ops(struct arch_uprobe_xol *xol, struct insn *insn)
 {
 	return -ENOSYS;
 }
+static int sub_setup_xol_ops(struct arch_uprobe_xol *xol, struct insn *insn)
+{
+	return -ENOSYS;
+}
 #endif
 
 /**
@@ -1649,6 +1718,10 @@ int arch_uprobe_analyze_insn(struct arch_uprobe *auprobe, struct mm_struct *mm,
 	if (ret != -ENOSYS)
 		return ret;
 
+	ret = sub_setup_xol_ops(&auprobe->xol, &insn);
+	if (ret != -ENOSYS)
+		return ret;
+
 	/*
 	 * Figure out which fixups default_post_xol_op() will need to perform,
 	 * and annotate defparam->fixups accordingly.
-- 
2.51.1

[RFC PATCH 5/8] uprobe/x86: Add support to optimize on top of emulated instructions

[Jiri Olsa]

Adding support to optimize uprobe on top of instructions that can
be emulated.

The idea is to store instructions on underlying 5 bytes and emulate
them during the int3 and uprobe syscall execution:

  - install 'call trampoline' through standard int3 update
  - if int3 is hit before we finish optimizing we emulate
    all underlying instructions
  - when call is installed the uprobe syscall will emulate
    all underlying instructions

Adding opt_xol_ops that emulate instructions that are replaced
by 5 bytes call instruction used to optimize the uprobe.

Signed-off-by: Jiri Olsa <jolsa@kernel.org>
---
 arch/x86/include/asm/uprobes.h |  13 ++--
 arch/x86/kernel/uprobes.c      | 106 ++++++++++++++++++++++++++++++++-
 include/linux/uprobes.h        |   1 +
 kernel/events/uprobes.c        |   6 ++
 4 files changed, 120 insertions(+), 6 deletions(-)

diff --git a/arch/x86/include/asm/uprobes.h b/arch/x86/include/asm/uprobes.h
index e09aab82b8c1..eaa80dc1c836 100644
--- a/arch/x86/include/asm/uprobes.h
+++ b/arch/x86/include/asm/uprobes.h
@@ -21,8 +21,9 @@ typedef u8 uprobe_opcode_t;
 #define UPROBE_SWBP_INSN_SIZE		   1
 
 enum {
-	ARCH_UPROBE_FLAG_CAN_OPTIMIZE   = 0,
-	ARCH_UPROBE_FLAG_OPTIMIZE_FAIL  = 1,
+	ARCH_UPROBE_FLAG_CAN_OPTIMIZE     = 0,
+	ARCH_UPROBE_FLAG_OPTIMIZE_FAIL    = 1,
+	ARCH_UPROBE_FLAG_OPTIMIZE_EMULATE = 2,
 };
 
 struct uprobe_xol_ops;
@@ -59,11 +60,15 @@ struct arch_uprobe_xol {
 
 struct arch_uprobe {
 	union {
-		u8			insn[MAX_UINSN_BYTES];
+		u8			insn[5*MAX_UINSN_BYTES];
 		u8			ixol[MAX_UINSN_BYTES];
 	};
 
-	struct arch_uprobe_xol		xol;
+	struct arch_uprobe_xol	xol;
+	struct {
+		struct arch_uprobe_xol	xol[5];
+		int			cnt;
+	} opt;
 
 	unsigned long flags;
 };
diff --git a/arch/x86/kernel/uprobes.c b/arch/x86/kernel/uprobes.c
index 904c423ea81d..7f3f537a6425 100644
--- a/arch/x86/kernel/uprobes.c
+++ b/arch/x86/kernel/uprobes.c
@@ -277,13 +277,14 @@ static bool is_prefix_bad(struct insn *insn)
 	return false;
 }
 
-static int uprobe_init_insn(struct arch_uprobe *auprobe, struct insn *insn, bool x86_64)
+static int uprobe_init_insn_offset(struct arch_uprobe *auprobe, unsigned long offset,
+				   struct insn *insn, bool x86_64)
 {
 	enum insn_mode m = x86_64 ? INSN_MODE_64 : INSN_MODE_32;
 	u32 volatile *good_insns;
 	int ret;
 
-	ret = insn_decode(insn, auprobe->insn, sizeof(auprobe->insn), m);
+	ret = insn_decode(insn, auprobe->insn + offset, sizeof(auprobe->insn) - offset, m);
 	if (ret < 0)
 		return -ENOEXEC;
 
@@ -310,6 +311,11 @@ static int uprobe_init_insn(struct arch_uprobe *auprobe, struct insn *insn, bool
 	return -ENOTSUPP;
 }
 
+static int uprobe_init_insn(struct arch_uprobe *auprobe, struct insn *insn, bool x86_64)
+{
+	return uprobe_init_insn_offset(auprobe, 0, insn, x86_64);
+}
+
 #ifdef CONFIG_X86_64
 
 struct uretprobe_syscall_args {
@@ -1462,6 +1468,23 @@ static bool sub_emulate_op(struct arch_uprobe *auprobe, struct arch_uprobe_xol *
 
 #undef EFLAGS_MASK
 
+static bool optimized_emulate(struct arch_uprobe *auprobe, struct arch_uprobe_xol *xol,
+			      struct pt_regs *regs)
+{
+	int i;
+
+	for (i = 0; i < auprobe->opt.cnt; i++) {
+		WARN_ON(!auprobe->opt.xol[i].ops->emulate(auprobe, &auprobe->opt.xol[i], regs));
+	}
+	return true;
+}
+
+void arch_uprobe_optimized_emulate(struct arch_uprobe *auprobe, struct pt_regs *regs)
+{
+	if (test_bit(ARCH_UPROBE_FLAG_OPTIMIZE_EMULATE, &auprobe->flags))
+		optimized_emulate(auprobe, NULL, regs);
+}
+
 static const struct uprobe_xol_ops branch_xol_ops = {
 	.emulate  = branch_emulate_op,
 	.post_xol = branch_post_xol_op,
@@ -1479,6 +1502,10 @@ static const struct uprobe_xol_ops sub_xol_ops = {
 	.emulate  = sub_emulate_op,
 };
 
+static const struct uprobe_xol_ops opt_xol_ops = {
+	.emulate  = optimized_emulate,
+};
+
 /* Returns -ENOSYS if branch_xol_ops doesn't handle this insn */
 static int branch_setup_xol_ops(struct arch_uprobe *auprobe, struct insn *insn)
 {
@@ -1675,6 +1702,73 @@ static int sub_setup_xol_ops(struct arch_uprobe_xol *xol, struct insn *insn)
 	xol->ops = &sub_xol_ops;
 	return 0;
 }
+
+static int opt_setup_xol_insns(struct arch_uprobe *auprobe, struct arch_uprobe_xol *xol,
+			       struct insn *insn)
+{
+	int ret;
+
+	/*
+	 * TODO somehow separate nop emulation out of branch_xol_ops,
+	 * so we could emulate nop instructions in here.
+	 */
+	ret = push_setup_xol_ops(xol, insn);
+	if (ret != -ENOSYS)
+		return ret;
+	ret = mov_setup_xol_ops(xol, insn);
+	if (ret != -ENOSYS)
+		return ret;
+	ret = sub_setup_xol_ops(xol, insn);
+	if (ret != -ENOSYS)
+		return ret;
+
+	return -1;
+}
+
+static int opt_setup_xol_ops(struct arch_uprobe *auprobe, struct insn *insn)
+{
+	unsigned long offset = insn->length;
+	struct insn insnX;
+	int i, ret;
+
+	if (test_bit(ARCH_UPROBE_FLAG_CAN_OPTIMIZE, &auprobe->flags))
+		return -ENOSYS;
+
+	ret = opt_setup_xol_insns(auprobe, &auprobe->opt.xol[0], insn);
+	if (ret)
+		return -ENOSYS;
+
+	auprobe->opt.cnt = 1;
+	if (offset >= 5)
+		goto optimize;
+
+	for (i = 1; i < 5; i++) {
+		ret = uprobe_init_insn_offset(auprobe, offset, &insnX, true);
+		if (ret)
+			break;
+		ret = opt_setup_xol_insns(auprobe, &auprobe->opt.xol[i], &insnX);
+		if (ret)
+			break;
+		offset += insnX.length;
+		auprobe->opt.cnt++;
+		if (offset >= 5)
+			goto optimize;
+	}
+
+	return -ENOSYS;
+
+optimize:
+	set_bit(ARCH_UPROBE_FLAG_CAN_OPTIMIZE, &auprobe->flags);
+	set_bit(ARCH_UPROBE_FLAG_OPTIMIZE_EMULATE, &auprobe->flags);
+	auprobe->xol.ops = &opt_xol_ops;
+
+	/*
+	 * TODO perhaps we could 'emulate' nop, so there would be no need for
+	 * ARCH_UPROBE_FLAG_OPTIMIZE_EMULATE flag, because we would emulate
+	 * allways.
+	 */
+	return 0;
+}
 #else
 static int mov_setup_xol_ops(struct arch_uprobe_xol *xol, struct insn *insn)
 {
@@ -1684,6 +1778,10 @@ static int sub_setup_xol_ops(struct arch_uprobe_xol *xol, struct insn *insn)
 {
 	return -ENOSYS;
 }
+static int opt_setup_xol_ops(struct arch_uprobe *auprobe, struct insn *insn)
+{
+	return -ENOSYS;
+}
 #endif
 
 /**
@@ -1706,6 +1804,10 @@ int arch_uprobe_analyze_insn(struct arch_uprobe *auprobe, struct mm_struct *mm,
 	if (can_optimize(&insn, addr))
 		set_bit(ARCH_UPROBE_FLAG_CAN_OPTIMIZE, &auprobe->flags);
 
+	ret = opt_setup_xol_ops(auprobe, &insn);
+	if (ret != -ENOSYS)
+		return ret;
+
 	ret = branch_setup_xol_ops(auprobe, &insn);
 	if (ret != -ENOSYS)
 		return ret;
diff --git a/include/linux/uprobes.h b/include/linux/uprobes.h
index ee3d36eda45d..4b9f81ad8316 100644
--- a/include/linux/uprobes.h
+++ b/include/linux/uprobes.h
@@ -242,6 +242,7 @@ extern void arch_uprobe_clear_state(struct mm_struct *mm);
 extern void arch_uprobe_init_state(struct mm_struct *mm);
 extern void handle_syscall_uprobe(struct pt_regs *regs, unsigned long bp_vaddr);
 extern void arch_uprobe_optimize(struct arch_uprobe *auprobe, unsigned long vaddr);
+extern void arch_uprobe_optimized_emulate(struct arch_uprobe *auprobe, struct pt_regs *regs);
 #else /* !CONFIG_UPROBES */
 struct uprobes_state {
 };
diff --git a/kernel/events/uprobes.c b/kernel/events/uprobes.c
index f11ceb8be8c4..dd893030e32e 100644
--- a/kernel/events/uprobes.c
+++ b/kernel/events/uprobes.c
@@ -2701,6 +2701,10 @@ void __weak arch_uprobe_optimize(struct arch_uprobe *auprobe, unsigned long vadd
 {
 }
 
+void __weak arch_uprobe_optimized_emulate(struct arch_uprobe *auprobe, struct pt_regs *regs)
+{
+}
+
 /*
  * Run handler and ask thread to singlestep.
  * Ensure all non-fatal signals cannot interrupt thread while it singlesteps.
@@ -2801,6 +2805,8 @@ void handle_syscall_uprobe(struct pt_regs *regs, unsigned long bp_vaddr)
 	if (arch_uprobe_ignore(&uprobe->arch, regs))
 		return;
 	handler_chain(uprobe, regs);
+
+	arch_uprobe_optimized_emulate(&uprobe->arch, regs);
 }
 
 /*
-- 
2.51.1

[RFC PATCH 6/8] selftests/bpf: Add test for mov and sub emulation

[Jiri Olsa]

Adding test code for mov and sub instructions emulation.

TODO add test for sub flags value emulation.

Signed-off-by: Jiri Olsa <jolsa@kernel.org>
---
 .../selftests/bpf/prog_tests/uprobe_syscall.c | 50 +++++++++++++++++++
 1 file changed, 50 insertions(+)

diff --git a/tools/testing/selftests/bpf/prog_tests/uprobe_syscall.c b/tools/testing/selftests/bpf/prog_tests/uprobe_syscall.c
index 955a37751b52..27fa6f309188 100644
--- a/tools/testing/selftests/bpf/prog_tests/uprobe_syscall.c
+++ b/tools/testing/selftests/bpf/prog_tests/uprobe_syscall.c
@@ -765,6 +765,54 @@ static void test_uprobe_error(void)
 	ASSERT_EQ(errno, ENXIO, "errno");
 }
 
+__attribute__((aligned(16)))
+__nocf_check __weak __naked unsigned long emulate_mov_trigger(void)
+{
+	asm volatile (
+		"pushq %rbp\n"
+		"movq  %rsp,%rbp\n"
+		"subq  $0xb0,%rsp\n"
+		"addq  $0xb0,%rsp\n"
+		"pop %rbp\n"
+		"ret\n"
+	);
+}
+
+static void test_emulate(void)
+{
+	struct uprobe_syscall *skel = NULL;
+	unsigned long offset;
+
+	offset = get_uprobe_offset(&emulate_mov_trigger);
+	if (!ASSERT_GE(offset, 0, "get_uprobe_offset"))
+		return;
+
+	skel = uprobe_syscall__open_and_load();
+	if (!ASSERT_OK_PTR(skel, "uprobe_syscall__open_and_load"))
+		goto cleanup;
+
+	/* mov */
+	skel->links.probe = bpf_program__attach_uprobe_opts(skel->progs.probe,
+				0, "/proc/self/exe", offset + 1, NULL);
+	if (!ASSERT_OK_PTR(skel->links.probe, "bpf_program__attach_uprobe_opts"))
+		goto cleanup;
+
+	emulate_mov_trigger();
+
+	bpf_link__destroy(skel->links.probe);
+
+	/* sub */
+	skel->links.probe = bpf_program__attach_uprobe_opts(skel->progs.probe,
+				0, "/proc/self/exe", offset + 4, NULL);
+	if (!ASSERT_OK_PTR(skel->links.probe, "bpf_program__attach_uprobe_opts"))
+		goto cleanup;
+
+	emulate_mov_trigger();
+
+cleanup:
+	uprobe_syscall__destroy(skel);
+}
+
 static void __test_uprobe_syscall(void)
 {
 	if (test__start_subtest("uretprobe_regs_equal"))
@@ -789,6 +837,8 @@ static void __test_uprobe_syscall(void)
 		test_uprobe_regs_equal(false);
 	if (test__start_subtest("regs_change"))
 		test_regs_change();
+	if (test__start_subtest("emulate_mov"))
+		test_emulate();
 }
 #else
 static void __test_uprobe_syscall(void)
-- 
2.51.1

[RFC PATCH 7/8] selftests/bpf: Add test for uprobe prologue optimization

[Jiri Olsa]

Adding test that places uprobe on top of supported prologue
and checks that the uprobe gets properly optimized.

Signed-off-by: Jiri Olsa <jolsa@kernel.org>
---
 .../selftests/bpf/prog_tests/uprobe_syscall.c | 63 ++++++++++++++++---
 1 file changed, 53 insertions(+), 10 deletions(-)

diff --git a/tools/testing/selftests/bpf/prog_tests/uprobe_syscall.c b/tools/testing/selftests/bpf/prog_tests/uprobe_syscall.c
index 27fa6f309188..c6a58afc7ace 100644
--- a/tools/testing/selftests/bpf/prog_tests/uprobe_syscall.c
+++ b/tools/testing/selftests/bpf/prog_tests/uprobe_syscall.c
@@ -428,21 +428,21 @@ static void *check_attach(struct uprobe_syscall_executed *skel, trigger_t trigge
 	return tramp;
 }
 
-static void check_detach(void *addr, void *tramp)
+static void check_detach(void *addr, void *tramp, unsigned char *orig)
 {
 	/* [uprobes_trampoline] stays after detach */
 	ASSERT_OK(find_uprobes_trampoline(tramp), "uprobes_trampoline");
-	ASSERT_OK(memcmp(addr, nop5, 5), "nop5");
+	ASSERT_OK(memcmp(addr, orig, 5), "orig");
 }
 
 static void check(struct uprobe_syscall_executed *skel, struct bpf_link *link,
-		  trigger_t trigger, void *addr, int executed)
+		  trigger_t trigger, void *addr, int executed, unsigned char *orig)
 {
 	void *tramp;
 
 	tramp = check_attach(skel, trigger, addr, executed);
 	bpf_link__destroy(link);
-	check_detach(addr, tramp);
+	check_detach(addr, tramp, orig);
 }
 
 static void test_uprobe_legacy(void)
@@ -470,7 +470,7 @@ static void test_uprobe_legacy(void)
 	if (!ASSERT_OK_PTR(link, "bpf_program__attach_uprobe_opts"))
 		goto cleanup;
 
-	check(skel, link, uprobe_test, uprobe_test, 2);
+	check(skel, link, uprobe_test, uprobe_test, 2, nop5);
 
 	/* uretprobe */
 	skel->bss->executed = 0;
@@ -480,7 +480,7 @@ static void test_uprobe_legacy(void)
 	if (!ASSERT_OK_PTR(link, "bpf_program__attach_uprobe_opts"))
 		goto cleanup;
 
-	check(skel, link, uprobe_test, uprobe_test, 2);
+	check(skel, link, uprobe_test, uprobe_test, 2, nop5);
 
 cleanup:
 	uprobe_syscall_executed__destroy(skel);
@@ -512,7 +512,7 @@ static void test_uprobe_multi(void)
 	if (!ASSERT_OK_PTR(link, "bpf_program__attach_uprobe_multi"))
 		goto cleanup;
 
-	check(skel, link, uprobe_test, uprobe_test, 2);
+	check(skel, link, uprobe_test, uprobe_test, 2, nop5);
 
 	/* uretprobe.multi */
 	skel->bss->executed = 0;
@@ -522,7 +522,7 @@ static void test_uprobe_multi(void)
 	if (!ASSERT_OK_PTR(link, "bpf_program__attach_uprobe_multi"))
 		goto cleanup;
 
-	check(skel, link, uprobe_test, uprobe_test, 2);
+	check(skel, link, uprobe_test, uprobe_test, 2, nop5);
 
 cleanup:
 	uprobe_syscall_executed__destroy(skel);
@@ -555,7 +555,7 @@ static void test_uprobe_session(void)
 	if (!ASSERT_OK_PTR(link, "bpf_program__attach_uprobe_multi"))
 		goto cleanup;
 
-	check(skel, link, uprobe_test, uprobe_test, 4);
+	check(skel, link, uprobe_test, uprobe_test, 4, nop5);
 
 cleanup:
 	uprobe_syscall_executed__destroy(skel);
@@ -584,7 +584,7 @@ static void test_uprobe_usdt(void)
 	if (!ASSERT_OK_PTR(link, "bpf_program__attach_usdt"))
 		goto cleanup;
 
-	check(skel, link, usdt_test, addr, 2);
+	check(skel, link, usdt_test, addr, 2, nop5);
 
 cleanup:
 	uprobe_syscall_executed__destroy(skel);
@@ -813,6 +813,47 @@ static void test_emulate(void)
 	uprobe_syscall__destroy(skel);
 }
 
+__attribute__((aligned(16)))
+__nocf_check __weak __naked void prologue_trigger(void)
+{
+	asm volatile (
+		"pushq %rbp\n"
+		"movq  %rsp,%rbp\n"
+		"subq  $0xb0,%rsp\n"
+		"addq  $0xb0,%rsp\n"
+		"pop %rbp\n"
+		"ret\n"
+	);
+}
+
+static void test_optimize_prologue(void)
+{
+	struct uprobe_syscall_executed *skel = NULL;
+	struct bpf_link *link;
+	unsigned long offset;
+
+	offset = get_uprobe_offset(&prologue_trigger);
+	if (!ASSERT_GE(offset, 0, "get_uprobe_offset"))
+		goto cleanup;
+
+	/* uprobe */
+	skel = uprobe_syscall_executed__open_and_load();
+	if (!ASSERT_OK_PTR(skel, "uprobe_syscall_executed__open_and_load"))
+		return;
+
+	skel->bss->pid = getpid();
+
+	link = bpf_program__attach_uprobe_opts(skel->progs.test_uprobe,
+				0, "/proc/self/exe", offset, NULL);
+	if (!ASSERT_OK_PTR(link, "bpf_program__attach_uprobe_opts"))
+		goto cleanup;
+
+	check(skel, link, prologue_trigger, prologue_trigger, 2, (unsigned char *) prologue_trigger);
+
+cleanup:
+	uprobe_syscall_executed__destroy(skel);
+}
+
 static void __test_uprobe_syscall(void)
 {
 	if (test__start_subtest("uretprobe_regs_equal"))
@@ -839,6 +880,8 @@ static void __test_uprobe_syscall(void)
 		test_regs_change();
 	if (test__start_subtest("emulate_mov"))
 		test_emulate();
+	if (test__start_subtest("optimize_prologue"))
+		test_optimize_prologue();
 }
 #else
 static void __test_uprobe_syscall(void)
-- 
2.51.1

[RFC PATCH 8/8] selftests/bpf: Add race test for uprobe proglog optimization

[Jiri Olsa]

Adding uprobe race test on top of prologue instructions.

Signed-off-by: Jiri Olsa <jolsa@kernel.org>
---
 .../selftests/bpf/prog_tests/uprobe_syscall.c    | 16 ++++++++++------
 1 file changed, 10 insertions(+), 6 deletions(-)

diff --git a/tools/testing/selftests/bpf/prog_tests/uprobe_syscall.c b/tools/testing/selftests/bpf/prog_tests/uprobe_syscall.c
index c6a58afc7ace..8793fbd61ffd 100644
--- a/tools/testing/selftests/bpf/prog_tests/uprobe_syscall.c
+++ b/tools/testing/selftests/bpf/prog_tests/uprobe_syscall.c
@@ -654,10 +654,11 @@ static USDT_DEFINE_SEMA(race);
 
 static void *worker_trigger(void *arg)
 {
+	trigger_t trigger = (trigger_t) arg;
 	unsigned long rounds = 0;
 
 	while (!race_stop) {
-		uprobe_test();
+		trigger();
 		rounds++;
 	}
 
@@ -667,6 +668,7 @@ static void *worker_trigger(void *arg)
 
 static void *worker_attach(void *arg)
 {
+	trigger_t trigger = (trigger_t) arg;
 	LIBBPF_OPTS(bpf_uprobe_opts, opts);
 	struct uprobe_syscall_executed *skel;
 	unsigned long rounds = 0, offset;
@@ -677,7 +679,7 @@ static void *worker_attach(void *arg)
 	unsigned long *ref;
 	int err;
 
-	offset = get_uprobe_offset(&uprobe_test);
+	offset = get_uprobe_offset(trigger);
 	if (!ASSERT_GE(offset, 0, "get_uprobe_offset"))
 		return NULL;
 
@@ -722,7 +724,7 @@ static useconds_t race_msec(void)
 	return 500;
 }
 
-static void test_uprobe_race(void)
+static void test_uprobe_race(trigger_t trigger)
 {
 	int err, i, nr_threads;
 	pthread_t *threads;
@@ -738,7 +740,7 @@ static void test_uprobe_race(void)
 
 	for (i = 0; i < nr_threads; i++) {
 		err = pthread_create(&threads[i], NULL, i % 2 ? worker_trigger : worker_attach,
-				     NULL);
+				     trigger);
 		if (!ASSERT_OK(err, "pthread_create"))
 			goto cleanup;
 	}
@@ -870,8 +872,10 @@ static void __test_uprobe_syscall(void)
 		test_uprobe_session();
 	if (test__start_subtest("uprobe_usdt"))
 		test_uprobe_usdt();
-	if (test__start_subtest("uprobe_race"))
-		test_uprobe_race();
+	if (test__start_subtest("uprobe_race_nop5"))
+		test_uprobe_race(uprobe_test);
+	if (test__start_subtest("uprobe_race_prologue"))
+		test_uprobe_race(prologue_trigger);
 	if (test__start_subtest("uprobe_error"))
 		test_uprobe_error();
 	if (test__start_subtest("uprobe_regs_equal"))
-- 
2.51.1

Re: [RFC PATCH 5/8] uprobe/x86: Add support to optimize on top of emulated instructions

[Oleg Nesterov]

Hi Jiri,

I am trying to understand this series, will try to read it more carefully
later...

(damn why do you always send the patches when I am on PTO? ;)

On 11/17, Jiri Olsa wrote:
>
>  struct arch_uprobe {
>  	union {
> -		u8			insn[MAX_UINSN_BYTES];
> +		u8			insn[5*MAX_UINSN_BYTES];

Hmm. OK, this matches the "for (i = 0; i < 5; i++)" loop in
opt_setup_xol_ops(), but do we really need this change? Please see
the question at the end.

> +static int opt_setup_xol_ops(struct arch_uprobe *auprobe, struct insn *insn)
> +{
> +	unsigned long offset = insn->length;
> +	struct insn insnX;
> +	int i, ret;
> +
> +	if (test_bit(ARCH_UPROBE_FLAG_CAN_OPTIMIZE, &auprobe->flags))
> +		return -ENOSYS;

I think this logic needs some cleanups... If ARCH_UPROBE_FLAG_CAN_OPTIMIZE
is set by the caller, the it doesn't make sense to call xxx_setup_xol_ops(),
right? But lets forget it for now.

> +	ret = opt_setup_xol_insns(auprobe, &auprobe->opt.xol[0], insn);

I think this should go into the main loop, see below

> +	for (i = 1; i < 5; i++) {
> +		ret = uprobe_init_insn_offset(auprobe, offset, &insnX, true);
> +		if (ret)
> +			break;
> +		ret = opt_setup_xol_insns(auprobe, &auprobe->opt.xol[i], &insnX);
> +		if (ret)
> +			break;
> +		offset += insnX.length;
> +		auprobe->opt.cnt++;
> +		if (offset >= 5)
> +			goto optimize;
> +	}
> +
> +	return -ENOSYS;

I don't think -ENOSYS makes sense if opt_setup_xol_insns() succeeds at least once.
IOW, how about

	static int opt_setup_xol_ops(struct arch_uprobe *auprobe, struct insn *insn)
	{
		unsigned long offset = 0;
		struct insn insnX;
		int i, ret;

		if (test_bit(ARCH_UPROBE_FLAG_CAN_OPTIMIZE, &auprobe->flags))
			return -ENOSYS;

		for (i = 0; i < 5; i++) {
			ret = opt_setup_xol_insns(auprobe, &auprobe->opt.xol[i], insn);
			if (ret)
				break;
			offset += insn->length;
			if (offset >= 5)
				break;

			insn = &insnX;
			ret = uprobe_init_insn_offset(auprobe, offset, insn, true);
			if (ret)
				break;
		}

		if (!offset)
			return -ENOSYS;

		if (offset >= 5) {
			auprobe->opt.cnt = i + 1;
			auprobe->xol.ops = &opt_xol_ops;
			set_bit(ARCH_UPROBE_FLAG_CAN_OPTIMIZE, &auprobe->flags);
			set_bit(ARCH_UPROBE_FLAG_OPTIMIZE_EMULATE, &auprobe->flags);
		}

		return 0;
	}

?

This way the caller, arch_uprobe_analyze_insn(), doesn't need to call
push/mov/sub/_setup_xol_ops(), and the code looks a bit simpler to me.

No?

> +      * TODO perhaps we could 'emulate' nop, so there would be no need for
> +      * ARCH_UPROBE_FLAG_OPTIMIZE_EMULATE flag, because we would emulate
> +      * allways.

Agreed... and this connects to "this logic needs some cleanups" above.
I guess we need nop_setup_xol_ops() extracted from branch_setup_xol_ops()
but again, lets forget it for now.

-------------------------------------------------------------------------------
Now the main question. What if we avoid this change

	-             u8                      insn[MAX_UINSN_BYTES];
	+             u8                      insn[5*MAX_UINSN_BYTES];

mentioned above, and change opt_setup_xol_ops() to just do

	-	for (i = 0; i < 5; i++)
	+	for (i = 0;; i++)

?

The main loop stops when offset >= 5 anyway.

And. if auprobe->insn[offset:MAX_UINSN_BYTES] doesn't contain a full/valid
insn at the start, then uprobe_init_insn_offset()->insn_decode() should fail?

Most probably I missed something, but I can't understand this part.

Oleg.

Re: [RFC PATCH 0/8] uprobe/x86: Add support to optimize prologue

[Oleg Nesterov]

On 11/17, Jiri Olsa wrote:
>
> This patchset adds support to optimize uprobe on top of instruction
> that could be emulated and also adds support to emulate particular
> versions of mov and sub instructions to cover some of the user space
> functions prologues, like:
>
>   pushq %rbp
>   movq  %rsp,%rbp
>   subq  $0xb0,%rsp

...

> There's an additional issue that single instruction replacement does
> not have and it's the possibility of the user space code to jump in the
> middle of those 5 bytes. I think it's unlikely to happen at the function
> prologue, but uprobe could be placed anywhere. I'm not sure how to
> mitigate this other than having some enable/disable switch or config
> option, which is unfortunate.

plus this breaks single-stepping... Although perhaps we don't really care.

Oleg.

Re: [RFC PATCH 5/8] uprobe/x86: Add support to optimize on top of emulated instructions

[Jiri Olsa]

On Mon, Nov 24, 2025 at 07:01:14PM +0100, Oleg Nesterov wrote:
> Hi Jiri,
> 
> I am trying to understand this series, will try to read it more carefully
> later...
> 
> (damn why do you always send the patches when I am on PTO? ;)

it's more fun that way ;-) thanks for checking on it

> 
> On 11/17, Jiri Olsa wrote:
> >
> >  struct arch_uprobe {
> >  	union {
> > -		u8			insn[MAX_UINSN_BYTES];
> > +		u8			insn[5*MAX_UINSN_BYTES];
> 
> Hmm. OK, this matches the "for (i = 0; i < 5; i++)" loop in
> opt_setup_xol_ops(), but do we really need this change? Please see
> the question at the end.
> 
> > +static int opt_setup_xol_ops(struct arch_uprobe *auprobe, struct insn *insn)
> > +{
> > +	unsigned long offset = insn->length;
> > +	struct insn insnX;
> > +	int i, ret;
> > +
> > +	if (test_bit(ARCH_UPROBE_FLAG_CAN_OPTIMIZE, &auprobe->flags))
> > +		return -ENOSYS;
> 
> I think this logic needs some cleanups... If ARCH_UPROBE_FLAG_CAN_OPTIMIZE
> is set by the caller, the it doesn't make sense to call xxx_setup_xol_ops(),
> right? But lets forget it for now.
> 
> > +	ret = opt_setup_xol_insns(auprobe, &auprobe->opt.xol[0], insn);
> 
> I think this should go into the main loop, see below
> 
> > +	for (i = 1; i < 5; i++) {
> > +		ret = uprobe_init_insn_offset(auprobe, offset, &insnX, true);
> > +		if (ret)
> > +			break;
> > +		ret = opt_setup_xol_insns(auprobe, &auprobe->opt.xol[i], &insnX);
> > +		if (ret)
> > +			break;
> > +		offset += insnX.length;
> > +		auprobe->opt.cnt++;
> > +		if (offset >= 5)
> > +			goto optimize;
> > +	}
> > +
> > +	return -ENOSYS;
> 
> I don't think -ENOSYS makes sense if opt_setup_xol_insns() succeeds at least once.
> IOW, how about
> 
> 	static int opt_setup_xol_ops(struct arch_uprobe *auprobe, struct insn *insn)
> 	{
> 		unsigned long offset = 0;
> 		struct insn insnX;
> 		int i, ret;
> 
> 		if (test_bit(ARCH_UPROBE_FLAG_CAN_OPTIMIZE, &auprobe->flags))
> 			return -ENOSYS;
> 
> 		for (i = 0; i < 5; i++) {
> 			ret = opt_setup_xol_insns(auprobe, &auprobe->opt.xol[i], insn);
> 			if (ret)
> 				break;
> 			offset += insn->length;
> 			if (offset >= 5)
> 				break;
> 
> 			insn = &insnX;
> 			ret = uprobe_init_insn_offset(auprobe, offset, insn, true);
> 			if (ret)
> 				break;
> 		}
> 
> 		if (!offset)
> 			return -ENOSYS;
> 
> 		if (offset >= 5) {
> 			auprobe->opt.cnt = i + 1;
> 			auprobe->xol.ops = &opt_xol_ops;
> 			set_bit(ARCH_UPROBE_FLAG_CAN_OPTIMIZE, &auprobe->flags);
> 			set_bit(ARCH_UPROBE_FLAG_OPTIMIZE_EMULATE, &auprobe->flags);
> 		}
> 
> 		return 0;
> 	}
> 
> ?
> 
> This way the caller, arch_uprobe_analyze_insn(), doesn't need to call
> push/mov/sub/_setup_xol_ops(), and the code looks a bit simpler to me.

ah nice, will try that

> 
> No?
> 
> > +      * TODO perhaps we could 'emulate' nop, so there would be no need for
> > +      * ARCH_UPROBE_FLAG_OPTIMIZE_EMULATE flag, because we would emulate
> > +      * allways.
> 
> Agreed... and this connects to "this logic needs some cleanups" above.
> I guess we need nop_setup_xol_ops() extracted from branch_setup_xol_ops()
> but again, lets forget it for now.

ok, it will hopefully make the code simpler, will check on that

> 
> -------------------------------------------------------------------------------
> Now the main question. What if we avoid this change
> 
> 	-             u8                      insn[MAX_UINSN_BYTES];
> 	+             u8                      insn[5*MAX_UINSN_BYTES];
> 
> mentioned above, and change opt_setup_xol_ops() to just do
> 
> 	-	for (i = 0; i < 5; i++)
> 	+	for (i = 0;; i++)
> 
> ?
> 
> The main loop stops when offset >= 5 anyway.

> 
> And. if auprobe->insn[offset:MAX_UINSN_BYTES] doesn't contain a full/valid
> insn at the start, then uprobe_init_insn_offset()->insn_decode() should fail?
> 
> Most probably I missed something, but I can't understand this part.

no, I think you're right, I did not realize we fit under MAX_UINSN_BYTES
anyway, call instruction needs only 5 bytes

thanks,
jirka

Re: [RFC PATCH 0/8] uprobe/x86: Add support to optimize prologue

[Jiri Olsa]

On Mon, Nov 17, 2025 at 01:40:49PM +0100, Jiri Olsa wrote:
> hi,
> the subject is bit too optimistic, in nutshell the idea is to allow
> optimization on top of emulated instructions and then add support to
> emulate more instructions with high presence in function prologues.
> 
> This patchset adds support to optimize uprobe on top of instruction
> that could be emulated and also adds support to emulate particular
> versions of mov and sub instructions to cover some of the user space
> functions prologues, like:
> 
>   pushq %rbp
>   movq  %rsp,%rbp
>   subq  $0xb0,%rsp
> 
> The idea is to store instructions on underlying 5 bytes and emulate
> them during the int3 and uprobe syscall execution:
> 
>   - install 'call trampoline' through standard int3 update
>   - if int3 is hit before we finish optimizing we emulate
>     all underlying instructions
>   - when call is installed the uprobe syscall will emulate
>     all underlying instructions

David, sorry I used wrong email.. I think the update here might
be a problem, any chance you could check?

thanks,
jirka


> 
> There's an additional issue that single instruction replacement does
> not have and it's the possibility of the user space code to jump in the
> middle of those 5 bytes. I think it's unlikely to happen at the function
> prologue, but uprobe could be placed anywhere. I'm not sure how to
> mitigate this other than having some enable/disable switch or config
> option, which is unfortunate.
> 
> The patchset is based on bpf-next/master with [1] changes merged in.
> 
> thanks,
> jirka
> 
> 
> [1] https://lore.kernel.org/lkml/20251117093137.572132-1-jolsa@kernel.org/T/#m95a3208943ec24c5eee17ad6113002fdc6776cf8
> ---
> Jiri Olsa (8):
>       uprobe/x86: Introduce struct arch_uprobe_xol object
>       uprobe/x86: Use struct arch_uprobe_xol in emulate callback
>       uprobe/x86: Add support to emulate mov reg,reg instructions
>       uprobe/x86: Add support to emulate sub imm,reg instructions
>       uprobe/x86: Add support to optimize on top of emulated instructions
>       selftests/bpf: Add test for mov and sub emulation
>       selftests/bpf: Add test for uprobe prologue optimization
>       selftests/bpf: Add race test for uprobe proglog optimization
> 
>  arch/x86/include/asm/uprobes.h                          |  35 +++++++---
>  arch/x86/kernel/uprobes.c                               | 336 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++--------------
>  include/linux/uprobes.h                                 |   1 +
>  kernel/events/uprobes.c                                 |   6 ++
>  tools/testing/selftests/bpf/prog_tests/uprobe_syscall.c | 129 ++++++++++++++++++++++++++++++++-----
>  5 files changed, 434 insertions(+), 73 deletions(-)

Re: [RFC PATCH 0/8] uprobe/x86: Add support to optimize prologue

[Masami Hiramatsu]

On Mon, 24 Nov 2025 19:12:42 +0100
Oleg Nesterov <oleg@redhat.com> wrote:

> On 11/17, Jiri Olsa wrote:
> >
> > This patchset adds support to optimize uprobe on top of instruction
> > that could be emulated and also adds support to emulate particular
> > versions of mov and sub instructions to cover some of the user space
> > functions prologues, like:
> >
> >   pushq %rbp
> >   movq  %rsp,%rbp
> >   subq  $0xb0,%rsp
> 
> ...
> 
> > There's an additional issue that single instruction replacement does
> > not have and it's the possibility of the user space code to jump in the
> > middle of those 5 bytes. I think it's unlikely to happen at the function
> > prologue, but uprobe could be placed anywhere. I'm not sure how to
> > mitigate this other than having some enable/disable switch or config
> > option, which is unfortunate.
> 
> plus this breaks single-stepping... Although perhaps we don't really care.

Yeah, and I think we can stop optimization if post_handler is set.

Thanks,

-- 
Masami Hiramatsu (Google) <mhiramat@kernel.org>

Re: [RFC PATCH 0/8] uprobe/x86: Add support to optimize prologue

[Oleg Nesterov]

On 12/08, Masami Hiramatsu wrote:
>
> On Mon, 24 Nov 2025 19:12:42 +0100
> Oleg Nesterov <oleg@redhat.com> wrote:
>
> > On 11/17, Jiri Olsa wrote:
> > >
> > > There's an additional issue that single instruction replacement does
> > > not have and it's the possibility of the user space code to jump in the
> > > middle of those 5 bytes. I think it's unlikely to happen at the function
> > > prologue, but uprobe could be placed anywhere. I'm not sure how to
> > > mitigate this other than having some enable/disable switch or config
> > > option, which is unfortunate.
> >
> > plus this breaks single-stepping... Although perhaps we don't really care.
>
> Yeah, and I think we can stop optimization if post_handler is set.

Hmm, why? This doesn't depend on whether ->ret_handler is set or not...

Oleg.