[PATCH v2 0/4] sparc64: kprobes: Make kprobes work on sun4v

[PATCH v2 0/4] sparc64: kprobes: Make kprobes work on sun4v

[Stian Halseth]

Any kprobe hit on a sun4v machine oopses. The probed instruction is
single-stepped from memory that is not executable there. Patch 1
moves the copy into an instruction slot from get_insn_slot(), as other
architectures do.

Once probes work, three older single-step bugs show up. A taken BPr or
FP branch resumes at the wrong address (patch 2). So does a probe on a
delay slot instruction (patch 3). An annulled branch can skip the
break instruction after the copy (patch 4). Patch 2 overlaps Danish
Khateeb's relbranch_fixup() fix from September. Whichever is applied
first, the other is a small rebase.

Tested on a SPARC T7-1 and T4-1 (sun4v) and a Sun Fire V240 (sun4u):
entry and return probes, probes on all 50 instructions of
__task_pid_nr_ns at once while checking getppid(), ps and /proc, probes
on the SPARC AES routines during AF_ALG encryption (sun4v), and
refusal of ba,a. Without the series the first probe hit oopses on
sun4v. A probe on a taken brnz oopses on sun4u as well.

v2: make flush_insn_slot() a no-op in patch 1 (Masami). The slot is
flushed in arch_prepare_kprobe() when it is written. A killed probe
has a NULL slot pointer. init_aggr_kprobe() flushed through it.
Patch 4 no longer touches the macro. Rebased onto 7.4-rc3. Build
tested on the T7-1.
v1: https://lore.kernel.org/all/20261009180336.1315684-1-stian@itx.no/

Stian Halseth (4):
  sparc64: kprobes: Single-step from an executable instruction slot
  sparc64: kprobes: Fix up register and FP branches after single-step
  sparc64: kprobes: Resume after a delay slot at the saved next PC
  sparc64: kprobes: Handle annulled branches

 arch/sparc/include/asm/kprobes.h | 14 +++----
 arch/sparc/kernel/kprobes.c      | 69 +++++++++++++++++++++++++++-----
 2 files changed, 66 insertions(+), 17 deletions(-)


base-commit: b3dfbde62cf1981c4638939c159dfd087b7a72e9
-- 
2.43.0

[PATCH v2 1/4] sparc64: kprobes: Single-step from an executable instruction slot

[Stian Halseth]

kprobes single-steps the probed instruction from kprobe->ainsn.insn[].
The array is embedded in struct kprobe, in the linear mapping. The
sun4v linear mapping is not executable. Every probe hit oopses:

  sun4v_insn_access_exception: ADDR[fff800005b0cd66c] CTX[0000] TYPE[0006], going.

Allocate the copy with get_insn_slot(), as other architectures do. The
slot comes from executable memory in the module area.

flush_insn_slot() flushed the embedded copy after init_aggr_kprobe()
had copied it into the aggregator. The aggregator now shares the slot
by pointer. arch_prepare_kprobe() flushes the slot when it writes it.
Make the macro a no-op, like x86 and arm64.

Fixes: c4bce90ea2069 ("[SPARC64]: Deal with PTE layout differences in SUN4V.")
Closes: https://github.com/sparclinux/issues/issues/110
Signed-off-by: Stian Halseth <stian@itx.no>
---
 arch/sparc/include/asm/kprobes.h | 12 ++++++------
 arch/sparc/kernel/kprobes.c      | 12 ++++++++++++
 2 files changed, 18 insertions(+), 6 deletions(-)

diff --git a/arch/sparc/include/asm/kprobes.h b/arch/sparc/include/asm/kprobes.h
index aec742cd898f2..d14206c063e25 100644
--- a/arch/sparc/include/asm/kprobes.h
+++ b/arch/sparc/include/asm/kprobes.h
@@ -11,25 +11,25 @@
 #include <linux/types.h>
 #include <linux/percpu.h>
 
+#define __ARCH_WANT_KPROBES_INSN_SLOT
+
 typedef u32 kprobe_opcode_t;
 
 #define MAX_INSN_SIZE 2
 
 #define kretprobe_blacklist_size 0
 
-#define arch_remove_kprobe(p)	do {} while (0)
+struct kprobe;
+void arch_remove_kprobe(struct kprobe *p);
 
-#define flush_insn_slot(p)		\
-do { 	flushi(&(p)->ainsn.insn[0]);	\
-	flushi(&(p)->ainsn.insn[1]);	\
-} while (0)
+#define flush_insn_slot(p)	do { } while (0)
 
 void __kretprobe_trampoline(void);
 
 /* Architecture specific copy of original instruction*/
 struct arch_specific_insn {
 	/* copy of the original instruction */
-	kprobe_opcode_t insn[MAX_INSN_SIZE];
+	kprobe_opcode_t *insn;
 };
 
 struct prev_kprobe {
diff --git a/arch/sparc/kernel/kprobes.c b/arch/sparc/kernel/kprobes.c
index 191bbaca99212..608efda8cb948 100644
--- a/arch/sparc/kernel/kprobes.c
+++ b/arch/sparc/kernel/kprobes.c
@@ -52,6 +52,10 @@ int __kprobes arch_prepare_kprobe(struct kprobe *p)
 	if ((unsigned long) p->addr & 0x3UL)
 		return -EILSEQ;
 
+	p->ainsn.insn = get_insn_slot();
+	if (!p->ainsn.insn)
+		return -ENOMEM;
+
 	p->ainsn.insn[0] = *p->addr;
 	flushi(&p->ainsn.insn[0]);
 
@@ -62,6 +66,14 @@ int __kprobes arch_prepare_kprobe(struct kprobe *p)
 	return 0;
 }
 
+void __kprobes arch_remove_kprobe(struct kprobe *p)
+{
+	if (p->ainsn.insn) {
+		free_insn_slot(p->ainsn.insn, 0);
+		p->ainsn.insn = NULL;
+	}
+}
+
 void __kprobes arch_arm_kprobe(struct kprobe *p)
 {
 	*p->addr = BREAKPOINT_INSTRUCTION;
-- 
2.43.0

[PATCH v2 2/4] sparc64: kprobes: Fix up register and FP branches after single-step

[Stian Halseth]

relbranch_fixup() moves the target of a PC-relative branch from the
single-step slot back to the probed instruction. It only knows call,
Bicc and BPcc. A taken BPr (brz, brnz, ...) or FBPfcc/FBfcc is left
alone. Execution resumes at the target relative to the slot:

  Unable to handle kernel paging request at virtual address 000000000fffff98
  TPC: <0xfffff98>

Treat them like the other PC-relative branches.

Link: https://github.com/sparclinux/issues/issues/110
Signed-off-by: Stian Halseth <stian@itx.no>
---
 arch/sparc/kernel/kprobes.c | 9 ++++++---
 1 file changed, 6 insertions(+), 3 deletions(-)

diff --git a/arch/sparc/kernel/kprobes.c b/arch/sparc/kernel/kprobes.c
index 608efda8cb948..38e6d9b51a207 100644
--- a/arch/sparc/kernel/kprobes.c
+++ b/arch/sparc/kernel/kprobes.c
@@ -219,12 +219,15 @@ static unsigned long __kprobes relbranch_fixup(u32 insn, struct kprobe *p,
 	if (regs->tnpc == regs->tpc + 0x4UL)
 		return real_pc + 0x8UL;
 
-	/* The three cases are call, branch w/prediction,
-	 * and traditional branch.
+	/* The cases are call and the branches BPcc, Bicc, BPr, FBPfcc
+	 * and FBfcc.
 	 */
 	if ((insn & 0xc0000000) == 0x40000000 ||
 	    (insn & 0xc1c00000) == 0x00400000 ||
-	    (insn & 0xc1c00000) == 0x00800000) {
+	    (insn & 0xc1c00000) == 0x00800000 ||
+	    (insn & 0xc1c00000) == 0x00c00000 ||
+	    (insn & 0xc1c00000) == 0x01400000 ||
+	    (insn & 0xc1c00000) == 0x01800000) {
 		unsigned long ainsn_addr;
 
 		ainsn_addr = (unsigned long) &p->ainsn.insn[0];
-- 
2.43.0

[PATCH v2 3/4] sparc64: kprobes: Resume after a delay slot at the saved next PC

[Stian Halseth]

After single-stepping an instruction that does not branch, execution
resumes at the saved next PC. The probed address plus 8 comes next.
For an instruction in the delay slot of a taken branch the saved next
PC is the branch target. Execution then goes back to the probed
address plus 8. It should continue at the target plus 4.

Use the saved next PC plus 4. For any other instruction that is the
probed address plus 8.

Link: https://github.com/sparclinux/issues/issues/110
Signed-off-by: Stian Halseth <stian@itx.no>
---
 arch/sparc/kernel/kprobes.c | 7 ++++---
 1 file changed, 4 insertions(+), 3 deletions(-)

diff --git a/arch/sparc/kernel/kprobes.c b/arch/sparc/kernel/kprobes.c
index 38e6d9b51a207..327d83188a978 100644
--- a/arch/sparc/kernel/kprobes.c
+++ b/arch/sparc/kernel/kprobes.c
@@ -211,13 +211,14 @@ static int __kprobes kprobe_handler(struct pt_regs *regs)
  * 
  */
 static unsigned long __kprobes relbranch_fixup(u32 insn, struct kprobe *p,
-					       struct pt_regs *regs)
+					       struct pt_regs *regs,
+					       struct kprobe_ctlblk *kcb)
 {
 	unsigned long real_pc = (unsigned long) p->addr;
 
 	/* Branch not taken, no mods necessary.  */
 	if (regs->tnpc == regs->tpc + 0x4UL)
-		return real_pc + 0x8UL;
+		return kcb->kprobe_orig_tnpc + 0x4UL;
 
 	/* The cases are call and the branches BPcc, Bicc, BPr, FBPfcc
 	 * and FBfcc.
@@ -294,7 +295,7 @@ static void __kprobes resume_execution(struct kprobe *p,
 {
 	u32 insn = p->ainsn.insn[0];
 
-	regs->tnpc = relbranch_fixup(insn, p, regs);
+	regs->tnpc = relbranch_fixup(insn, p, regs, kcb);
 
 	/* This assignment must occur after relbranch_fixup() */
 	regs->tpc = kcb->kprobe_orig_tnpc;
-- 
2.43.0

[PATCH v2 4/4] sparc64: kprobes: Handle annulled branches

[Stian Halseth]

The single-stepped copy of a branch has a break instruction in its
delay slot. An annulled branch that is not taken skips that slot and
runs past the copy. A taken ba,a or cbcond branches away without
reaching it.

Put a second break instruction after the delay slot. When it is hit,
resume after the original delay slot. Refuse probes on ba,a, fba,a
and cbcond.

Link: https://github.com/sparclinux/issues/issues/110
Signed-off-by: Stian Halseth <stian@itx.no>
---
 arch/sparc/include/asm/kprobes.h |  2 +-
 arch/sparc/kernel/kprobes.c      | 43 ++++++++++++++++++++++++++++----
 2 files changed, 39 insertions(+), 6 deletions(-)

diff --git a/arch/sparc/include/asm/kprobes.h b/arch/sparc/include/asm/kprobes.h
index d14206c063e25..e15fd84c88681 100644
--- a/arch/sparc/include/asm/kprobes.h
+++ b/arch/sparc/include/asm/kprobes.h
@@ -15,7 +15,7 @@
 
 typedef u32 kprobe_opcode_t;
 
-#define MAX_INSN_SIZE 2
+#define MAX_INSN_SIZE 3
 
 #define kretprobe_blacklist_size 0
 
diff --git a/arch/sparc/kernel/kprobes.c b/arch/sparc/kernel/kprobes.c
index 327d83188a978..dc1180d4c2338 100644
--- a/arch/sparc/kernel/kprobes.c
+++ b/arch/sparc/kernel/kprobes.c
@@ -21,7 +21,8 @@
  *
  * In the kprobe->ainsn.insn[] array we store the original
  * instruction at index zero and a break instruction at
- * index one.
+ * index one, and another at index two for a branch that
+ * annuls its delay slot.
  *
  * When we hit a kprobe we:
  * - Run the pre-handler
@@ -47,11 +48,34 @@ DEFINE_PER_CPU(struct kprobe_ctlblk, kprobe_ctlblk);
 
 struct kretprobe_blackpoint kretprobe_blacklist[] = {{NULL, NULL}};
 
+/* cbcond has no delay slot and ba,a annuls it, so when taken the
+ * single-stepped copy branches away without reaching a break instruction.
+ */
+static bool __kprobes kprobe_insn_rejected(u32 insn)
+{
+	if (insn & 0xc0000000)
+		return false;
+
+	switch ((insn >> 22) & 0x7) {
+	case 1:		/* BPcc */
+	case 2:		/* Bicc */
+	case 5:		/* FBPfcc */
+	case 6:		/* FBfcc */
+		return (insn & 0x3e000000) == 0x30000000;
+	case 3:		/* BPr, cbcond */
+		return insn & 0x10000000;
+	}
+	return false;
+}
+
 int __kprobes arch_prepare_kprobe(struct kprobe *p)
 {
 	if ((unsigned long) p->addr & 0x3UL)
 		return -EILSEQ;
 
+	if (kprobe_insn_rejected(*p->addr))
+		return -EINVAL;
+
 	p->ainsn.insn = get_insn_slot();
 	if (!p->ainsn.insn)
 		return -ENOMEM;
@@ -62,6 +86,9 @@ int __kprobes arch_prepare_kprobe(struct kprobe *p)
 	p->ainsn.insn[1] = BREAKPOINT_INSTRUCTION_2;
 	flushi(&p->ainsn.insn[1]);
 
+	p->ainsn.insn[2] = BREAKPOINT_INSTRUCTION_2;
+	flushi(&p->ainsn.insn[2]);
+
 	p->opcode = *p->addr;
 	return 0;
 }
@@ -295,12 +322,18 @@ static void __kprobes resume_execution(struct kprobe *p,
 {
 	u32 insn = p->ainsn.insn[0];
 
-	regs->tnpc = relbranch_fixup(insn, p, regs, kcb);
+	if (regs->tpc == (unsigned long) &p->ainsn.insn[2]) {
+		/* An untaken branch annulled its delay slot. */
+		regs->tpc = kcb->kprobe_orig_tnpc + 0x4UL;
+		regs->tnpc = kcb->kprobe_orig_tnpc + 0x8UL;
+	} else {
+		regs->tnpc = relbranch_fixup(insn, p, regs, kcb);
 
-	/* This assignment must occur after relbranch_fixup() */
-	regs->tpc = kcb->kprobe_orig_tnpc;
+		/* This assignment must occur after relbranch_fixup() */
+		regs->tpc = kcb->kprobe_orig_tnpc;
 
-	retpc_fixup(regs, insn, (unsigned long) p->addr);
+		retpc_fixup(regs, insn, (unsigned long) p->addr);
+	}
 
 	regs->tstate = ((regs->tstate & ~TSTATE_PIL) |
 			kcb->kprobe_orig_tstate_pil);
-- 
2.43.0

Re: [PATCH v2 3/4] sparc64: kprobes: Resume after a delay slot at the saved next PC

[sashiko-bot]

Thank you for your contribution! Sashiko AI review found 1 potential
issue to consider:
- [Low] sparc64: kprobes: missing Fixes trailer for relbranch_fixup()
  delay slot fix
--

commit a946035b56bae1c0df38ae8f1a539756818bbb78
Author: Stian Halseth <stian@itx.no>

> sparc64: kprobes: Resume after a delay slot at the saved next PC
>
> After single-stepping an instruction that does not branch, execution
> resumes at the saved next PC. The probed address plus 8 comes next.
> For an instruction in the delay slot of a taken branch the saved next
> PC is the branch target. Execution then goes back to the probed
> address plus 8. It should continue at the target plus 4.
>
> Use the saved next PC plus 4. For any other instruction that is the
> probed address plus 8.
>
> Link: https://github.com/sparclinux/issues/issues/110
> Signed-off-by: Stian Halseth <stian@itx.no>

[Severity: Low]
This fixes the real_pc + 0x8UL return value in relbranch_fixup().  Should
this include:

  Fixes: f08825896664 ("[SPARC64]: Fix several kprobes bugs.")

-- 
Sashiko AI review · https://sashiko.dev/#/patchset/20261113144057.1697874-1-stian@itx.no?part=3