Re: core dump analysis, was Re: stack smashing detected

Michael Schmitz
2023-04-17T23:33:05Z

Hi Finn,

Am 16.04.2023 um 18:44 schrieb Finn Thain:
>> As I understand this, the call to sys_sigreturn() removes both this code
>> (signal trampoline IIRC) and the signal stack...
>
> I don't see that stuff getting removed when I run dash under gdb under
> QEMU. With breakpoints at the head of onsig() and the tail of __wait3(),
> the memory under USP is the same when examined at either juncture.

OK - with usp restored to what it was before the signal stack was added,
the stack below original usp can be left as is (will be overwritten
eventually).

>
> The backtrace confirms that this signal was delivered during execution of
> __wait3(). (Delivery can happen during execution of __libc_fork() but I
> just repeat the test until I get these ducks in a row.)
>
> (gdb) c
> Continuing.
> # x=$(:)
> [Detaching after fork from child process 1055]
>
> Breakpoint 6.1, onsig (signo=17) at trap.c:286
> 286 trap.c: No such file or directory.
> (gdb) bt
> #0 onsig (signo=17) at trap.c:286
> #1 <signal handler called>
> #2 0xc00e81b6 in __GI___wait4_time64 (pid=-1, stat_loc=0xeffff86a, options=2,
> usage=0x0) at ../sysdeps/unix/sysv/linux/wait4.c:35
> #3 0xc00e8164 in __GI___wait3_time64 (usage=0x0, options=<optimized out>,
> stat_loc=<optimized out>) at ../sysdeps/unix/sysv/linux/wait3.c:26

Where did that one come from? I don't think we saw __GI___wait3_time64
called in your disassembly of __wait3 ...

> #4 __wait3 (stat_loc=<optimized out>, options=<optimized out>,
> usage=<optimized out>) at ../sysdeps/unix/sysv/linux/wait3.c:35
> #5 0xd000c38e in waitproc (status=0xeffff85a, block=1) at jobs.c:1179
> #6 waitone (block=1, job=0xd001f618) at jobs.c:1055
> #7 0xd000c5b8 in dowait (block=1, jp=0xd001f618) at jobs.c:1137
> #8 0xd000ddb0 in waitforjob (jp=0xd001f618) at jobs.c:1014
> #9 0xd000aade in expbackq (flag=68, cmd=0xd001e4c8 <stackbase+36>)
> at expand.c:520
> #10 argstr (p=<optimized out>, flag=68) at expand.c:335
> #11 0xd000b5ce in expandarg (arg=0xd001e4e8 <stackbase+68>,
> arglist=0xeffffb08, flag=4) at expand.c:192
> #12 0xd0007e2a in evalcommand (cmd=<optimized out>, flags=<optimized out>)
> at eval.c:855
> #13 0xd0006ffc in evaltree (n=0xd001e4f8 <stackbase+84>, flags=0) at eval.c:300
> #14 0xd000e3c0 in cmdloop (top=1) at main.c:246
> #15 0xd0005018 in main (argc=<optimized out>, argv=<optimized out>)
> at main.c:181
>
>
> 0xeffff750: 0xc01a0000 saved $a5 == libc .got
> 0xeffff74c: 0xc0023e8c saved $a3 == &__stack_chk_guard
> 0xeffff748: 0x00000000 saved $a2
> 0xeffff744: 0x00000001 saved $d5
> 0xeffff740: 0xeffff86e saved $d4
> 0xeffff73c: 0xeffff86a saved $d3
> 0xeffff738: 0x00000002 saved $d2
> 0xeffff734: 0x00000000
> 0xeffff730: 0x00000000
> 0xeffff72c: 0x00000000
> 0xeffff728: 0x00000000
> 0xeffff724: 0x00000000
> 0xeffff720: 0x00000000
> 0xeffff71c: 0x00000000
> 0xeffff718: 0x00000000
> 0xeffff714: 0x00000000
> 0xeffff710: 0x00000000
> 0xeffff70c: 0x00000000
> 0xeffff708: 0x00000000
> 0xeffff704: 0x00000000
> 0xeffff700: 0x00000000
> 0xeffff6fc: 0x00000000
> 0xeffff6f8: 0x00000000
> 0xeffff6f4: 0x00000000
> 0xeffff6f0: 0x00000000
> 0xeffff6ec: 0x00000000
> 0xeffff6e8: 0x00000000
> 0xeffff6e4: 0x00000000
> 0xeffff6e0: 0x00000000
> 0xeffff6dc: 0x00000000
> 0xeffff6d8: 0x00000000
> 0xeffff6d4: 0x00000000
> 0xeffff6d0: 0x00000000
> 0xeffff6cc: 0x00000000
> 0xeffff6c8: 0x00000000
> 0xeffff6c4: 0x00000000
> 0xeffff6c0: 0x00000000
> 0xeffff6bc: 0x00000000
> 0xeffff6b8: 0x00000000
> 0xeffff6b4: 0x00000000
> 0xeffff6b0: 0x00000000
> 0xeffff6ac: 0x00000000
> 0xeffff6a8: 0x00000000
> 0xeffff6a4: 0x00000000
> 0xeffff6a0: 0x00000000
> 0xeffff69c: 0x00000000
> 0xeffff698: 0x00000000
> 0xeffff694: 0x00000000
> 0xeffff690: 0x00000000
> 0xeffff68c: 0x00000000
> 0xeffff688: 0x00000000
> 0xeffff684: 0x00000000
> 0xeffff680: 0x00000000
> 0xeffff67c: 0x00000000
> 0xeffff678: 0x00000000
> 0xeffff674: 0x00000000
> 0xeffff670: 0x00000000
> 0xeffff66c: 0x00000000
> 0xeffff668: 0x00000000
> 0xeffff664: 0x00000000
> 0xeffff660: 0x41000000
> 0xeffff65c: 0x00000000
> 0xeffff658: 0x00000000
> 0xeffff654: 0x00000000
> 0xeffff650: 0x00000000
> 0xeffff64c: 0x80000000
> 0xeffff648: 0x3fff0000
> 0xeffff644: 0x00000000
> 0xeffff640: 0xd0000000
> 0xeffff63c: 0x40020000 <= (sc.formatvec& 0xffff) << 16; fpregs from here on
> 0xeffff638: 0x81b60080 <= (sc.pc & 0xffff) << 16 | sc.formatvec >> 16
> 0xeffff634: 0x0000c00e <= sc.sr << 16 sc.pc >> 16
> 0xeffff630: 0xd001e4e3 saved a1? <= sc.a1
> 0xeffff62c: 0xc0028780 saved a0? <= sc.a0
> 0xeffff628: 0xffffffff saved d1? <= sc.d1
> 0xeffff624: 0x0000041f saved d0? <= sc.d0
> 0xeffff620: 0xeffff738 saved sp? <= sc.usp
> 0xeffff61c: 0x00000000 <= sc.mask
> 0xeffff618: 0x00000000 <= extramask
> 0xeffff614: 0x00000000 <= frame.retcode[1]
> 0xeffff610: 0x70774e40 moveq #119,%d0 ; trap #0
> 0xeffff60c: 0xeffff61c <= frame->sc
> 0xeffff608: 0x00000080 <= tregs->vector
> 0xeffff604: 0x00000011 <= signal no.
> 0xeffff600: 0xeffff610 return address
>
> The above comes from dash running under gdb under qemu, which does not
> exhibit the failure but is convenient for that kind of experiment.

I would have expected to see a different signal trampoline (for
sys_rt_sigreturn) ... But anyway:

The saved pc is 0xc00e81b6 which does match the backtrace above. Vector
offset 80 matches trap 0 which suggests 0xc00e81b6 should be the
instruction after a trap 0 instruction. d0 is 1055 which is not a signal
number I recognize.

>
>> Again as far as I understand, the core dump happens on process exit.
>> Stack smashing is detected and process exit is forced only at exit from
>> __wait3() or __wait4_time64(),
>
> I placed an illegal instruction in __wait3. This executes instead of the
> call to __stack_chk_fail because that obliterates stack memory of
> interest.

OK.

>
> Consequently the latest core dump still contains dead stack frames (see
> below) of subroutines that returned before __wait3() dumped core. You can
> see the return address for the branch to __wait4_time64() and below that
> you can see the return address for the branch to __m68k_read_tp().
>
> (gdb) disas __wait4_time64
> Dump of assembler code for function __GI___wait4_time64:
> 0xc00e4174 <+0>: lea %sp@(-80),%sp
> 0xc00e4178 <+4>: moveml %d2-%d5/%a2-%a3/%a5,%sp@-
> 0xc00e417c <+8>: lea %pc@(0xc019c000),%a5
> 0xc00e4184 <+16>: movel %sp@(116),%d2
> 0xc00e4188 <+20>: moveal %sp@(124),%a2
> 0xc00e418c <+24>: moveal %a5@(108),%a3
> 0xc00e4190 <+28>: movel %a3@,%sp@(104)
> 0xc00e4194 <+32>: bsrl 0xc0056e2c <__m68k_read_tp@plt>
>
> I gather the signal was delivered before __wait4_time64+38, otherwise the
> return address 0xc00e419a (which appears below) would have been
> overwritten by the signal frame. The signal must have been delivered after
> waitproc() initialized gotsigchld = 0 since gotsigchld is 1 at the time of
> the coredump.
>
> I assume the %a3 corruption happened after __wait4_time64+8 because that's
> when %a3 first appears on the stack. And the corruption must have happened
> before __wait4_time64+238, which is when %a3 was restored.
>
> If it was the signal which somehow corrupted the saved %a3, there's only a
> small window for that. The only syscall in that window is get_thread_area.

I see sys_wait4 called in two places (0xc00e01b4, and then 0xc00e0286
depending on the return code of the first). The second one again would
have called __m68k_read_tp so would have left a return address on the
stack (0xc00e02d2). Leaves the first.

>
> Here's some stack memory from the core dump.
>
> 0xeffff0dc: 0xd000c38e return address waitproc+124
> 0xeffff0d8: 0xd001c1ec frame 0 $fp == &suppressint
> 0xeffff0d4: 0x00add14b canary
> 0xeffff0d0: 0x00000000
> 0xeffff0cc: 0x0000000a
> 0xeffff0c8: 0x00000202
> 0xeffff0c4: 0x00000008
> 0xeffff0c0: 0x00000000
> 0xeffff0bc: 0x00000000
> 0xeffff0b8: 0x00000174
> 0xeffff0b4: 0x00000004
> 0xeffff0b0: 0x00000004
> 0xeffff0ac: 0x00000006
> 0xeffff0a8: 0x000000e0
> 0xeffff0a4: 0x000000e0
> 0xeffff0a0: 0x00171f20
> 0xeffff09c: 0x00171f20
> 0xeffff098: 0x00171f20
> 0xeffff094: 0x00000002
> 0xeffff090: 0x00002000
> 0xeffff08c: 0x00000006
> 0xeffff088: 0x0000e920
> 0xeffff084: 0x00005360
> 0xeffff080: 0x00170700
> 0xeffff07c: 0x00170700
> 0xeffff078: 0x00170700 frame 0 $fp - 96
> 0xeffff074: 0xd001b874 saved $a5 == dash .got
> 0xeffff070: 0xd001e498 saved $a3 == &dash_errno
> 0xeffff06c: 0xd001e718 frame 0 $sp saved $a2 == &gotsigchld
> 0xeffff068: 0x00000000
> 0xeffff064: 0x00000000
> 0xeffff060: 0xeffff11e
> 0xeffff05c: 0xffffffff
> 0xeffff058: 0xc00e4164 return address __wait3+244
> 0xeffff054: 0x00add14b canary
> 0xeffff050: 0x00000001
> 0xeffff04c: 0x00000004
> 0xeffff048: 0x0000000d
> 0xeffff044: 0x0000000d
> 0xeffff040: 0x0015ef82
> 0xeffff03c: 0x0015ef82
> 0xeffff038: 0x0015ef82
> 0xeffff034: 0x00000003
> 0xeffff030: 0x00000004
> 0xeffff02c: 0x00000004
> 0xeffff028: 0x00000140
> 0xeffff024: 0x00000140
> 0xeffff020: 0x00000034
> 0xeffff01c: 0x00000034
> 0xeffff018: 0x00000034
> 0xeffff014: 0x00000006
> 0xeffff010: 0x003b003a
> 0xeffff00c: 0x000a0028
> 0xeffff008: 0x00340020
> 0xeffff004: 0xc019c000 saved $a5 == libc .got
> 0xeffff000: 0xeffff068 saved $a3 (corrupted)
> 0xefffeffc: 0x00000000 saved $a2
> 0xefffeff8: 0x00000001 saved $d5
> 0xefffeff4: 0xeffff122 saved $d4
> 0xefffeff0: 0xeffff11e saved $d3
> 0xefffefec: 0x00000000 saved $d2
> 0xefffefe8: 0xc00e419a return address __GI___wait4_time64+38
> 0xefffefe4: 0xc0028780
> 0xefffefe0: 0x3c344bfb
> 0xefffefdc: 0x000af353
> 0xefffefd8: 0x3c340170
> 0xefffefd4: 0x00000000
> 0xefffefd0: 0xc00e417c
> 0xefffefcc: 0xc00e417e
> 0xefffefc8: 0xc00e4180
> 0xefffefc4: 0x48e73c34
> 0xefffefc0: 0x00000000
> 0xefffefbc: 0xefffeff8
> 0xefffefb8: 0xefffeffc
> 0xefffefb4: 0x4bfb0170
> 0xefffefb0: 0x0eee0709
> 0xefffefac: 0x00000000
> 0xefffefa8: 0x00000000
> 0xefffefa4: 0x00000000
> 0xefffefa0: 0x00000000
> 0xefffef9c: 0x00000000
> 0xefffef98: 0x00000000
> 0xefffef94: 0x00000000
> 0xefffef90: 0x00000000
> 0xefffef8c: 0x00000000
> 0xefffef88: 0x00000000
> 0xefffef84: 0x00000000
> 0xefffef80: 0x00000000
> 0xefffef7c: 0x00000000
> 0xefffef78: 0x00000000
> 0xefffef74: 0x00000000
> 0xefffef70: 0x00000000
> 0xefffef6c: 0x00000000
> 0xefffef68: 0x00000000
> 0xefffef64: 0x00000000
> 0xefffef60: 0x00000000
> 0xefffef5c: 0x00000000
> 0xefffef58: 0x00000000
> 0xefffef54: 0x00000000
> 0xefffef50: 0x00000000
> 0xefffef4c: 0x00000000
> 0xefffef48: 0x00000000
> 0xefffef44: 0x00000000
> 0xefffef40: 0x00000000
> 0xefffef3c: 0x00000000
> 0xefffef38: 0x00000000
> 0xefffef34: 0x00000000
> 0xefffef30: 0x00000000
> 0xefffef2c: 0x00000000
> 0xefffef28: 0x00000000
> 0xefffef24: 0x00000000
> 0xefffef20: 0x00000000
> 0xefffef1c: 0x00000000
> 0xefffef18: 0x00000000
> 0xefffef14: 0x00000000
> 0xefffef10: 0x7c0effff
> 0xefffef0c: 0xffffffff
> 0xefffef08: 0xaaaaaaaa
> 0xefffef04: 0xaf54eaaa
> 0xefffef00: 0x40040000
> 0xefffeefc: 0x40040000
> 0xefffeef8: 0x2b000000
> 0xefffeef4: 0x00000000
> 0xefffeef0: 0x00000000
> 0xefffeeec: 0x408ece9a
> 0xefffeee8: 0x00000000
> 0xefffeee4: 0xf0ff0000
> 0xefffeee0: 0x0f800000
> 0xefffeedc: 0xf0fff0ff
> 0xefffeed8: 0x1f380000
> 0xefffeed4: 0x00000000
> 0xefffeed0: 0x00000000
> 0xefffeecc: 0x00000000
> 0xefffeec8: 0xffffffff
> 0xefffeec4: 0xffffffff
> 0xefffeec0: 0x7fff0000
> 0xefffeebc: 0xffffffff
> 0xefffeeb8: 0xffffffff
> 0xefffeeb4: 0x7fff0000
>
> The signal frame is not readily apparent (to me).

From looking at the above stack dump, sc ought to start at 0xefffee90,
and the trampoline would be three words below that. The last address you
show corresponds to 0xeffff640 in first dump above, which is at the
start of the saved fpregs. I'd say we just miss the beginning of the
signal frame?

(My reasoning is that copy_siginfo_to_user clears the end of the signal
stack, which is what we can see in both cases.)

Can't explain the 14 words below the saved return address though.

>
> Also, stack memory in the core dump and stack memory as observed upon
> signal handler breakpoint do not agree very well... I'd expect the values
> immediately below the stack pointer to be zeros once the signal frame was
> put there.

Yes, but what you get to see in the core dump is the stack after return
from __GI___wait4_time64 ... and differences in timing may have resulted
in different code executed (assuming the gdb trace is from the first
subshell signaling, and the dump from one of the later ones ...).

>
> I can't explain all of that unless it's discontiguous stack corruption.
>

Me neither, sorry.

Cheers,

Michael