[PATCH 0/3] sparc64: fix a window fill fixup lockup and two fault bugs

[PATCH 0/3] sparc64: fix a window fill fixup lockup and two fault bugs

[Stian Halseth]

A register window fill that faults is mishandled in three ways on
sparc64.  The first wedges the CPU in kernel mode and takes the machine
with it; the other two corrupt what the kernel reports about the fault,
and were found while chasing the first.

1/3 is the lockup.  user_rtt_fill_fixup_common() re-enters the kernel
without passing through etrap, so %asi is left as the ASI_AIUP that
rtrap set for the fill while the primary context has been restored to
the kernel's.  Every subsequent %asi-based user access then translates
in the kernel context, and a user address below the VA hole can never be
resolved: the fault repeats forever, the task survives SIGKILL, and RCU
eventually reports the CPU stalled.

The reproducer below moves %sp onto a page, flushes the register
windows, mprotects that page PROT_NONE and takes a signal, so the fill
on the signal return path faults.  It needs no unusual configuration and
wedges a stock kernel in a few seconds.  On a fixed kernel it prints
PASS and exits.

Mind where you run it.  On an affected kernel one CPU is pegged in
kernel mode permanently, the task cannot be killed, RCU grace periods
stop completing so fsync() and sync() hang, and the machine needs a hard
reset.

    cc -O0 -o wedge wedge.c && ./wedge

    #include <signal.h>
    #include <stdio.h>
    #include <stdlib.h>
    #include <string.h>
    #include <sys/mman.h>
    #include <unistd.h>

    #define STACK_BIAS 2047
    #define REGION     (1UL << 20)

    static void segv(int sig)
    {
    	static const char m[] = "PASS: SIGSEGV delivered, kernel handled it\n";

    	write(2, m, sizeof(m) - 1);
    	_exit(0);
    }

    /* The normal stack is about to become unusable, so the handler needs its own. */
    static void arm(void)
    {
    	static char sigstk[256 * 1024];
    	stack_t ss = { .ss_sp = sigstk, .ss_size = sizeof sigstk, .ss_flags = 0 };
    	struct sigaction sa;

    	sigaltstack(&ss, NULL);
    	memset(&sa, 0, sizeof sa);
    	sa.sa_handler = segv;
    	sa.sa_flags = SA_ONSTACK | SA_NODEFER;
    	sigaction(SIGSEGV, &sa, NULL);
    	sigaction(SIGBUS, &sa, NULL);
    }

    static void wedge(void)
    {
    	char *region;
    	unsigned long sp;

    	region = mmap(NULL, REGION, PROT_READ | PROT_WRITE,
    		      MAP_PRIVATE | MAP_ANONYMOUS | MAP_NORESERVE, -1, 0);
    	if (region == MAP_FAILED) {
    		perror("mmap");
    		exit(1);
    	}
    	memset(region, 0, REGION);

    	sp = (unsigned long)(region + REGION - 4096) - STACK_BIAS;
    	fprintf(stderr, "%%sp -> %p, flushw, then mprotect PROT_NONE\n",
    		(void *)(sp + STACK_BIAS));
    	fflush(stderr);

    	__asm__ __volatile__("mov %0, %%sp" :: "r"(sp));
    	__asm__ __volatile__("flushw");		/* windows now live in region */
    	mprotect(region, REGION, PROT_NONE);	/* ...and become unreadable */
    	__asm__ __volatile__("nop");		/* the refill on return faults */
    }

    int main(void)
    {
    	arm();
    	fprintf(stderr, "pid %d\n", (int)getpid());
    	fflush(stderr);

    	wedge();

    	fprintf(stderr, "returned normally - no fault triggered\n");
    	return 1;
    }

2/3 and 3/3 came out of that investigation.  do_fault_siginfo() decodes
the instruction at regs->tpc to compute si_addr, which is not the
faulting access when the fault came from a spill or fill; in one capture
it decoded a branch and reported the contents of %g5.  And the huge-page
path in the TSB miss handler tested TL after switching the global
register bank, so it consumed a fault code and a PTE that no longer
existed and killed the task with SIGSEGV at an address that had never
been mapped.  That one needs CONFIG_TRANSPARENT_HUGEPAGE and only
appears under load.

1/3 is reproduced and fixed on an UltraSPARC T4-1 (sun4v, Niagara4) and
on a Sun Fire V240 (sun4u, UltraSPARC-IIIi), which between them cover
both forms of the global register bank switch, SET_GL and the
PSTATE_AG|PSTATE_MG write.  2/3 and 3/3 are tested on the T4-1, where
with all three applied a Go toolchain build now completes five times
running with THP enabled, having previously died within a minute.

Link: https://github.com/sparclinux/issues/issues/87

Stian Halseth (3):
  sparc64: restore %asi in user_rtt_fill_fixup_common
  sparc64: use the fault address for si_addr on window fixup faults
  sparc64: decide the TSB huge-page window fixup before the bank switch

 arch/sparc/kernel/tsb.S       | 24 ++++++++++++++++++++----
 arch/sparc/kernel/urtt_fill.S | 15 +++++++++++++++
 arch/sparc/mm/fault_64.c      |  9 +++++++++
 3 files changed, 44 insertions(+), 4 deletions(-)

-- 
2.51.0

[PATCH 1/3] sparc64: restore %asi in user_rtt_fill_fixup_common

[Stian Halseth]

A window fill that faults re-enters the kernel through
user_rtt_fill_fixup_common(), which does not pass through etrap.  rtrap
has already set %asi to ASI_AIUP for the fill, and etrap is what would
normally re-establish ASI_AIUS from the TSTATE it synthesizes, so the
kernel carries on with %asi = ASI_AIUP while the primary context has
just been restored to the kernel's.

Every %asi-based user access made from there - put_user(), get_user()
and everything built on them - then translates in the kernel context.
User addresses below the VA hole fault forever, because nothing ever
fills a context-zero translation for them, and the CPU is wedged in
kernel mode: the task survives SIGKILL, sits in state R at 100% CPU,
and takes the machine down once RCU stalls.  Addresses above the hole
fail more quietly, silently aliasing the kernel linear mapping.

Restore the invariant before any user access is attempted.

Fixes: 7cafc0b8bf13 ("sparc64: Fix return from trap window fill crashes.")
Reported-by: John Paul Adrian Glaubitz <glaubitz@physik.fu-berlin.de>
Link: https://github.com/sparclinux/issues/issues/87
Signed-off-by: Stian Halseth <stian@itx.no>
---
 arch/sparc/kernel/urtt_fill.S | 15 +++++++++++++++
 1 file changed, 15 insertions(+)

diff --git a/arch/sparc/kernel/urtt_fill.S b/arch/sparc/kernel/urtt_fill.S
index e4cee7be5cd0..5acd27b18b1e 100644
--- a/arch/sparc/kernel/urtt_fill.S
+++ b/arch/sparc/kernel/urtt_fill.S
@@ -1,4 +1,5 @@
 /* SPDX-License-Identifier: GPL-2.0 */
+#include <asm/asi.h>
 #include <asm/thread_info.h>
 #include <asm/trap_block.h>
 #include <asm/spitfire.h>
@@ -32,6 +33,20 @@ user_rtt_fill_fixup_common:
 		sethi	%hi(KERNBASE), %g1
 		flush	%g1
 
+		/* rtrap set %asi to ASI_AIUP for the window fill, and
+		 * we re-enter the kernel here without passing through
+		 * etrap, which would have re-established ASI_AIUS via
+		 * the TSTATE it synthesizes.  The primary context was
+		 * just restored to the kernel's above, so a leftover
+		 * ASI_AIUP makes every %asi-based user access (put_user,
+		 * get_user) translate in the kernel context: user
+		 * addresses below the VA hole then fault forever
+		 * (nothing ever fills a context-zero translation for
+		 * them), and addresses above it silently alias the
+		 * kernel linear mapping.  Restore the kernel invariant.
+		 */
+		wr	%g0, ASI_AIUS, %asi
+
 		mov	%g4, %l4
 		mov	%g5, %l5
 		brnz,pn	%g3, 1f
-- 
2.43.0

[PATCH 2/3] sparc64: use the fault address for si_addr on window fixup faults

[Stian Halseth]

do_fault_siginfo() prefers an address decoded from the instruction at
regs->tpc whenever one could be probed, falling back to the fault time
address only when it could not.

That premise does not hold for a fault taken during a register window
spill or fill.  There the instruction at regs->tpc is the one being
resumed, not the access that faulted, so compute_effective_address()
fabricates an address out of unrelated register contents.  In one
capture it decoded a branch and reported the value it found in %g5 as
si_addr.

Use the fault time address for FAULT_CODE_WINFIXUP.  It may only have
page granularity, but it is the address that actually faulted.

Fixes: 70ffc6ebaead ("sparc64: Fix top-level fault handling bugs.")
Link: https://github.com/sparclinux/issues/issues/87
Signed-off-by: Stian Halseth <stian@itx.no>
---
 arch/sparc/mm/fault_64.c | 9 +++++++++
 1 file changed, 9 insertions(+)

diff --git a/arch/sparc/mm/fault_64.c b/arch/sparc/mm/fault_64.c
index e326caf708c6..c37b24f757f4 100644
--- a/arch/sparc/mm/fault_64.c
+++ b/arch/sparc/mm/fault_64.c
@@ -165,6 +165,15 @@ static void do_fault_siginfo(int code, int sig, struct pt_regs *regs,
 
 	if (fault_code & FAULT_CODE_ITLB) {
 		addr = regs->tpc;
+	} else if (fault_code & FAULT_CODE_WINFIXUP) {
+		/* The fault came from a register window spill or fill, so
+		 * the instruction at regs->tpc is the one being resumed,
+		 * not the access that faulted.  Decoding it would fabricate
+		 * an address out of unrelated register contents; the fault
+		 * time provided address is the real one, even if it only
+		 * has page granularity.
+		 */
+		addr = fault_addr;
 	} else {
 		/* If we were able to probe the faulting instruction, use it
 		 * to compute a precise fault address.  Otherwise use the fault
-- 
2.43.0

[PATCH 3/3] sparc64: decide the TSB huge-page window fixup before the bank switch

[Stian Halseth]

The huge-page path in the TSB miss handler has to route a fault taken
during a register window spill or fill away from hugetlb_setup(), since
a trap stack cannot be built at TL > 1.  That test is made after

	wrpr	%g5, PSTATE_AG | PSTATE_MG, %pstate

(SET_GL(1) on sun4v) has already switched the global register bank.
%g3, holding the fault code, and %g5, holding the PTE, live in the TLB
miss handler's globals and do not survive that switch, so the branch to
winfix_trampoline passes on whatever the new bank happens to contain: a
stale fault code and a garbage fault address.  The task is then killed
with SIGSEGV pointing at an address that was never mapped.

The path predates the fault code being passed at all; it was given one
when winfix_trampoline's %g4 requirement was fixed, and the register it
reads has been the wrong bank's since.

Make the decision before the switch and route to tsb_do_fault, which
re-reads both from the MMU after switching banks, rather than entering
winfix_trampoline with registers that no longer mean anything.

Fixes: 84bd6d8b9c0f ("sparc64: Fix corrupted thread fault code.")
Reported-by: John Paul Adrian Glaubitz <glaubitz@physik.fu-berlin.de>
Link: https://github.com/sparclinux/issues/issues/87
Signed-off-by: Stian Halseth <stian@itx.no>
---
 arch/sparc/kernel/tsb.S | 24 ++++++++++++++++++++----
 1 file changed, 20 insertions(+), 4 deletions(-)

diff --git a/arch/sparc/kernel/tsb.S b/arch/sparc/kernel/tsb.S
index eaed39ce8938..a70d5b124a80 100644
--- a/arch/sparc/kernel/tsb.S
+++ b/arch/sparc/kernel/tsb.S
@@ -144,6 +144,26 @@ tsb_miss_page_table_walk_sun4v_fastpath:
 	bne,pt		%xcc, 60f
 	 nop
 
+	/* If this fault came from a window spill or fill (TL > 1),
+	 * we cannot build a trap stack and call hugetlb_setup() from
+	 * here.  Route it through tsb_do_fault instead, so that
+	 * do_sparc64_fault() runs and its huge-TSB grow check
+	 * allocates the TSB before the access is replayed.
+	 *
+	 * This must be decided before the register bank switch below:
+	 * %g3 (the fault code) and %g5 (now the PTE) live in the TLB
+	 * miss handler's globals and do not survive it.  Consuming
+	 * them after the switch, as this path used to when it went to
+	 * winfix_trampoline directly, records a stale %g3 as the
+	 * fault code and a stale %g5 as the fault address, and the
+	 * task is then killed with a SIGSEGV at a garbage address.
+	 * tsb_do_fault re-fetches both from the MMU after switching.
+	 */
+	rdpr	%tl, %g7
+	cmp	%g7, 1
+	bgu,pn	%xcc, tsb_do_fault
+	 nop
+
 661:	rdpr		%pstate, %g5
 	wrpr		%g5, PSTATE_AG | PSTATE_MG, %pstate
 	.section	.sun4v_2insn_patch, "ax"
@@ -152,10 +172,6 @@ tsb_miss_page_table_walk_sun4v_fastpath:
 	nop
 	.previous
 
-	rdpr	%tl, %g7
-	cmp	%g7, 1
-	bne,pn	%xcc, winfix_trampoline
-	 mov	%g3, %g4
 	ba,pt	%xcc, etrap
 	 rd	%pc, %g7
 	call	hugetlb_setup
-- 
2.43.0