Re: core dump analysis, was Re: stack smashing detected

[Michael Schmitz <schmitzmic@gmail.com>]

Hi Finn,

Am 16.04.2023 um 18:44 schrieb Finn Thain:
>> As I understand this, the call to sys_sigreturn() removes both this code
>> (signal trampoline IIRC) and the signal stack...
>
> I don't see that stuff getting removed when I run dash under gdb under
> QEMU. With breakpoints at the head of onsig() and the tail of __wait3(),
> the memory under USP is the same when examined at either juncture.

OK - with usp restored to what it was before the signal stack was added, 
the stack below original usp can be left as is (will be overwritten 
eventually).

>
> The backtrace confirms that this signal was delivered during execution of
> __wait3(). (Delivery can happen during execution of __libc_fork() but I
> just repeat the test until I get these ducks in a row.)
>
> (gdb) c
> Continuing.
> # x=$(:)
> [Detaching after fork from child process 1055]
>
> Breakpoint 6.1, onsig (signo=17) at trap.c:286
> 286     trap.c: No such file or directory.
> (gdb) bt
> #0  onsig (signo=17) at trap.c:286
> #1  <signal handler called>
> #2  0xc00e81b6 in __GI___wait4_time64 (pid=-1, stat_loc=0xeffff86a, options=2,
>     usage=0x0) at ../sysdeps/unix/sysv/linux/wait4.c:35
> #3  0xc00e8164 in __GI___wait3_time64 (usage=0x0, options=<optimized out>,
>     stat_loc=<optimized out>) at ../sysdeps/unix/sysv/linux/wait3.c:26

Where did that one come from? I don't think we saw __GI___wait3_time64 
called in your disassembly of __wait3 ...

> #4  __wait3 (stat_loc=<optimized out>, options=<optimized out>,
>     usage=<optimized out>) at ../sysdeps/unix/sysv/linux/wait3.c:35
> #5  0xd000c38e in waitproc (status=0xeffff85a, block=1) at jobs.c:1179
> #6  waitone (block=1, job=0xd001f618) at jobs.c:1055
> #7  0xd000c5b8 in dowait (block=1, jp=0xd001f618) at jobs.c:1137
> #8  0xd000ddb0 in waitforjob (jp=0xd001f618) at jobs.c:1014
> #9  0xd000aade in expbackq (flag=68, cmd=0xd001e4c8 <stackbase+36>)
>     at expand.c:520
> #10 argstr (p=<optimized out>, flag=68) at expand.c:335
> #11 0xd000b5ce in expandarg (arg=0xd001e4e8 <stackbase+68>,
>     arglist=0xeffffb08, flag=4) at expand.c:192
> #12 0xd0007e2a in evalcommand (cmd=<optimized out>, flags=<optimized out>)
>     at eval.c:855
> #13 0xd0006ffc in evaltree (n=0xd001e4f8 <stackbase+84>, flags=0) at eval.c:300
> #14 0xd000e3c0 in cmdloop (top=1) at main.c:246
> #15 0xd0005018 in main (argc=<optimized out>, argv=<optimized out>)
>     at main.c:181
>
>
> 0xeffff750:     0xc01a0000                      saved $a5 == libc .got
> 0xeffff74c:     0xc0023e8c                      saved $a3 == &__stack_chk_guard
> 0xeffff748:     0x00000000                      saved $a2
> 0xeffff744:     0x00000001                      saved $d5
> 0xeffff740:     0xeffff86e                      saved $d4
> 0xeffff73c:     0xeffff86a                      saved $d3
> 0xeffff738:     0x00000002                      saved $d2
> 0xeffff734:     0x00000000
> 0xeffff730:     0x00000000
> 0xeffff72c:     0x00000000
> 0xeffff728:     0x00000000
> 0xeffff724:     0x00000000
> 0xeffff720:     0x00000000
> 0xeffff71c:     0x00000000
> 0xeffff718:     0x00000000
> 0xeffff714:     0x00000000
> 0xeffff710:     0x00000000
> 0xeffff70c:     0x00000000
> 0xeffff708:     0x00000000
> 0xeffff704:     0x00000000
> 0xeffff700:     0x00000000
> 0xeffff6fc:     0x00000000
> 0xeffff6f8:     0x00000000
> 0xeffff6f4:     0x00000000
> 0xeffff6f0:     0x00000000
> 0xeffff6ec:     0x00000000
> 0xeffff6e8:     0x00000000
> 0xeffff6e4:     0x00000000
> 0xeffff6e0:     0x00000000
> 0xeffff6dc:     0x00000000
> 0xeffff6d8:     0x00000000
> 0xeffff6d4:     0x00000000
> 0xeffff6d0:     0x00000000
> 0xeffff6cc:     0x00000000
> 0xeffff6c8:     0x00000000
> 0xeffff6c4:     0x00000000
> 0xeffff6c0:     0x00000000
> 0xeffff6bc:     0x00000000
> 0xeffff6b8:     0x00000000
> 0xeffff6b4:     0x00000000
> 0xeffff6b0:     0x00000000
> 0xeffff6ac:     0x00000000
> 0xeffff6a8:     0x00000000
> 0xeffff6a4:     0x00000000
> 0xeffff6a0:     0x00000000
> 0xeffff69c:     0x00000000
> 0xeffff698:     0x00000000
> 0xeffff694:     0x00000000
> 0xeffff690:     0x00000000
> 0xeffff68c:     0x00000000
> 0xeffff688:     0x00000000
> 0xeffff684:     0x00000000
> 0xeffff680:     0x00000000
> 0xeffff67c:     0x00000000
> 0xeffff678:     0x00000000
> 0xeffff674:     0x00000000
> 0xeffff670:     0x00000000
> 0xeffff66c:     0x00000000
> 0xeffff668:     0x00000000
> 0xeffff664:     0x00000000
> 0xeffff660:     0x41000000
> 0xeffff65c:     0x00000000
> 0xeffff658:     0x00000000
> 0xeffff654:     0x00000000
> 0xeffff650:     0x00000000
> 0xeffff64c:     0x80000000
> 0xeffff648:     0x3fff0000
> 0xeffff644:     0x00000000
> 0xeffff640:     0xd0000000
> 0xeffff63c:     0x40020000				<= (sc.formatvec& 0xffff) << 16; fpregs from here on
> 0xeffff638:     0x81b60080      			<= (sc.pc & 0xffff) << 16 | sc.formatvec >> 16
> 0xeffff634:     0x0000c00e				<= sc.sr << 16  sc.pc >> 16
> 0xeffff630:     0xd001e4e3                               saved a1?	<= sc.a1
> 0xeffff62c:     0xc0028780                               saved a0?	<= sc.a0
> 0xeffff628:     0xffffffff                               saved d1?	<= sc.d1
> 0xeffff624:     0x0000041f                               saved d0?	<= sc.d0
> 0xeffff620:     0xeffff738                               saved sp?	<= sc.usp
> 0xeffff61c:     0x00000000      			<= sc.mask
> 0xeffff618:     0x00000000				<= extramask
> 0xeffff614:     0x00000000				<= frame.retcode[1]
> 0xeffff610:     0x70774e40      moveq #119,%d0 ; trap #0
> 0xeffff60c:     0xeffff61c				<= frame->sc
> 0xeffff608:     0x00000080				<= tregs->vector
> 0xeffff604:     0x00000011				<= signal no.
> 0xeffff600:     0xeffff610      return address
>
> The above comes from dash running under gdb under qemu, which does not
> exhibit the failure but is convenient for that kind of experiment.

I would have expected to see a different signal trampoline (for 
sys_rt_sigreturn) ... But anyway:

The saved pc is 0xc00e81b6 which does match the backtrace above. Vector 
offset 80 matches trap 0 which suggests 0xc00e81b6 should be the 
instruction after a trap 0 instruction. d0 is 1055 which is not a signal 
number I recognize.

>
>> Again as far as I understand, the core dump happens on process exit.
>> Stack smashing is detected and process exit is forced only at exit from
>> __wait3() or __wait4_time64(),
>
> I placed an illegal instruction in __wait3. This executes instead of the
> call to __stack_chk_fail because that obliterates stack memory of
> interest.

OK.

>
> Consequently the latest core dump still contains dead stack frames (see
> below) of subroutines that returned before __wait3() dumped core. You can
> see the return address for the branch to __wait4_time64() and below that
> you can see the return address for the branch to __m68k_read_tp().
>
> (gdb) disas __wait4_time64
> Dump of assembler code for function __GI___wait4_time64:
>    0xc00e4174 <+0>:     lea %sp@(-80),%sp
>    0xc00e4178 <+4>:     moveml %d2-%d5/%a2-%a3/%a5,%sp@-
>    0xc00e417c <+8>:     lea %pc@(0xc019c000),%a5
>    0xc00e4184 <+16>:    movel %sp@(116),%d2
>    0xc00e4188 <+20>:    moveal %sp@(124),%a2
>    0xc00e418c <+24>:    moveal %a5@(108),%a3
>    0xc00e4190 <+28>:    movel %a3@,%sp@(104)
>    0xc00e4194 <+32>:    bsrl 0xc0056e2c <__m68k_read_tp@plt>
>
> I gather the signal was delivered before __wait4_time64+38, otherwise the
> return address 0xc00e419a (which appears below) would have been
> overwritten by the signal frame. The signal must have been delivered after
> waitproc() initialized gotsigchld = 0 since gotsigchld is 1 at the time of
> the coredump.
>
> I assume the %a3 corruption happened after __wait4_time64+8 because that's
> when %a3 first appears on the stack. And the corruption must have happened
> before __wait4_time64+238, which is when %a3 was restored.
>
> If it was the signal which somehow corrupted the saved %a3, there's only a
> small window for that. The only syscall in that window is get_thread_area.

I see sys_wait4 called in two places (0xc00e01b4, and then 0xc00e0286 
depending on the return code of the first). The second one again would 
have called  __m68k_read_tp so would have left a return address on the 
stack (0xc00e02d2). Leaves the first.

>
> Here's some stack memory from the core dump.
>
> 0xeffff0dc:     0xd000c38e      return address waitproc+124
> 0xeffff0d8:     0xd001c1ec      frame 0 $fp                   == &suppressint
> 0xeffff0d4:     0x00add14b      canary
> 0xeffff0d0:     0x00000000
> 0xeffff0cc:     0x0000000a
> 0xeffff0c8:     0x00000202
> 0xeffff0c4:     0x00000008
> 0xeffff0c0:     0x00000000
> 0xeffff0bc:     0x00000000
> 0xeffff0b8:     0x00000174
> 0xeffff0b4:     0x00000004
> 0xeffff0b0:     0x00000004
> 0xeffff0ac:     0x00000006
> 0xeffff0a8:     0x000000e0
> 0xeffff0a4:     0x000000e0
> 0xeffff0a0:     0x00171f20
> 0xeffff09c:     0x00171f20
> 0xeffff098:     0x00171f20
> 0xeffff094:     0x00000002
> 0xeffff090:     0x00002000
> 0xeffff08c:     0x00000006
> 0xeffff088:     0x0000e920
> 0xeffff084:     0x00005360
> 0xeffff080:     0x00170700
> 0xeffff07c:     0x00170700
> 0xeffff078:     0x00170700      frame 0 $fp - 96
> 0xeffff074:     0xd001b874                         saved $a5 == dash .got
> 0xeffff070:     0xd001e498                         saved $a3 == &dash_errno
> 0xeffff06c:     0xd001e718      frame 0 $sp        saved $a2 == &gotsigchld
> 0xeffff068:     0x00000000
> 0xeffff064:     0x00000000
> 0xeffff060:     0xeffff11e
> 0xeffff05c:     0xffffffff
> 0xeffff058:     0xc00e4164      return address __wait3+244
> 0xeffff054:     0x00add14b      canary
> 0xeffff050:     0x00000001
> 0xeffff04c:     0x00000004
> 0xeffff048:     0x0000000d
> 0xeffff044:     0x0000000d
> 0xeffff040:     0x0015ef82
> 0xeffff03c:     0x0015ef82
> 0xeffff038:     0x0015ef82
> 0xeffff034:     0x00000003
> 0xeffff030:     0x00000004
> 0xeffff02c:     0x00000004
> 0xeffff028:     0x00000140
> 0xeffff024:     0x00000140
> 0xeffff020:     0x00000034
> 0xeffff01c:     0x00000034
> 0xeffff018:     0x00000034
> 0xeffff014:     0x00000006
> 0xeffff010:     0x003b003a
> 0xeffff00c:     0x000a0028
> 0xeffff008:     0x00340020
> 0xeffff004:     0xc019c000                      saved $a5 == libc .got
> 0xeffff000:     0xeffff068                      saved $a3 (corrupted)
> 0xefffeffc:     0x00000000                      saved $a2
> 0xefffeff8:     0x00000001                      saved $d5
> 0xefffeff4:     0xeffff122                      saved $d4
> 0xefffeff0:     0xeffff11e                      saved $d3
> 0xefffefec:     0x00000000                      saved $d2
> 0xefffefe8:     0xc00e419a      return address __GI___wait4_time64+38
> 0xefffefe4:     0xc0028780
> 0xefffefe0:     0x3c344bfb
> 0xefffefdc:     0x000af353
> 0xefffefd8:     0x3c340170
> 0xefffefd4:     0x00000000
> 0xefffefd0:     0xc00e417c
> 0xefffefcc:     0xc00e417e
> 0xefffefc8:     0xc00e4180
> 0xefffefc4:     0x48e73c34
> 0xefffefc0:     0x00000000
> 0xefffefbc:     0xefffeff8
> 0xefffefb8:     0xefffeffc
> 0xefffefb4:     0x4bfb0170
> 0xefffefb0:     0x0eee0709
> 0xefffefac:     0x00000000
> 0xefffefa8:     0x00000000
> 0xefffefa4:     0x00000000
> 0xefffefa0:     0x00000000
> 0xefffef9c:     0x00000000
> 0xefffef98:     0x00000000
> 0xefffef94:     0x00000000
> 0xefffef90:     0x00000000
> 0xefffef8c:     0x00000000
> 0xefffef88:     0x00000000
> 0xefffef84:     0x00000000
> 0xefffef80:     0x00000000
> 0xefffef7c:     0x00000000
> 0xefffef78:     0x00000000
> 0xefffef74:     0x00000000
> 0xefffef70:     0x00000000
> 0xefffef6c:     0x00000000
> 0xefffef68:     0x00000000
> 0xefffef64:     0x00000000
> 0xefffef60:     0x00000000
> 0xefffef5c:     0x00000000
> 0xefffef58:     0x00000000
> 0xefffef54:     0x00000000
> 0xefffef50:     0x00000000
> 0xefffef4c:     0x00000000
> 0xefffef48:     0x00000000
> 0xefffef44:     0x00000000
> 0xefffef40:     0x00000000
> 0xefffef3c:     0x00000000
> 0xefffef38:     0x00000000
> 0xefffef34:     0x00000000
> 0xefffef30:     0x00000000
> 0xefffef2c:     0x00000000
> 0xefffef28:     0x00000000
> 0xefffef24:     0x00000000
> 0xefffef20:     0x00000000
> 0xefffef1c:     0x00000000
> 0xefffef18:     0x00000000
> 0xefffef14:     0x00000000
> 0xefffef10:     0x7c0effff
> 0xefffef0c:     0xffffffff
> 0xefffef08:     0xaaaaaaaa
> 0xefffef04:     0xaf54eaaa
> 0xefffef00:     0x40040000
> 0xefffeefc:     0x40040000
> 0xefffeef8:     0x2b000000
> 0xefffeef4:     0x00000000
> 0xefffeef0:     0x00000000
> 0xefffeeec:     0x408ece9a
> 0xefffeee8:     0x00000000
> 0xefffeee4:     0xf0ff0000
> 0xefffeee0:     0x0f800000
> 0xefffeedc:     0xf0fff0ff
> 0xefffeed8:     0x1f380000
> 0xefffeed4:     0x00000000
> 0xefffeed0:     0x00000000
> 0xefffeecc:     0x00000000
> 0xefffeec8:     0xffffffff
> 0xefffeec4:     0xffffffff
> 0xefffeec0:     0x7fff0000
> 0xefffeebc:     0xffffffff
> 0xefffeeb8:     0xffffffff
> 0xefffeeb4:     0x7fff0000
>
> The signal frame is not readily apparent (to me).

 From looking at the above stack dump, sc ought to start at 0xefffee90, 
and the trampoline would be three words below that. The last address you 
show corresponds to 0xeffff640 in first dump above, which is at the 
start of the saved fpregs. I'd say we just miss the beginning of the 
signal frame?

(My reasoning is that copy_siginfo_to_user clears the end of the signal 
stack, which is what we can see in both cases.)

Can't explain the 14 words below the saved return address though.

>
> Also, stack memory in the core dump and stack memory as observed upon
> signal handler breakpoint do not agree very well... I'd expect the values
> immediately below the stack pointer to be zeros once the signal frame was
> put there.

Yes, but what you get to see in the core dump is the stack after return 
from __GI___wait4_time64 ... and differences in timing may have resulted 
in different code executed (assuming the gdb trace is from the first 
subshell signaling, and the dump from one of the later ones ...).

>
> I can't explain all of that unless it's discontiguous stack corruption.
>

Me neither, sorry.

Cheers,

	Michael