Re: Mutual debugging of 2 processes can stuck in unkillable stopped state

[Oleg Nesterov <oleg@redhat.com>]

Hi Igor,

So. As expected, they sleep in EVENT_EXIT _after_ you have already sent SIGKILL.

Oh. I can only repeat that PTRACE_EVENT_EXIT must die ;) Or at least we should
finally define its semantics.

Igor, thanks for your report, but (I think) this has nothing to do with mutual
debugging. I'll return to this problem in a couple of days, I'm a bit busy right
now.

Thanks,

Oleg.

On 03/29, Igor Zhbanov wrote:
> Hi Oleg!
> 
> I've tried both 5.3.18 and 5.10.0. The behavior is the same.
> The important thing is to run "exec strace -p ..." on the second terminal
> to create the loop A->B->A.
> 
> So the last line from the first strace we see is:
> ptrace(PTRACE_SEIZE, 1990, NULL, PTRACE_O_TRACESYSGOOD|PTRACE_O_TRACEEXEC|PTRACE_O_TRACEEXIT
> 
> I.e. it printed the syscall prior to its execution and hanged after the
> execution.
> 
> izh@suse2:~> ps awux|grep strace
> izh       1891  0.0  0.0  24752  3828 pts/1    ts+  19:52   0:00 strace -p 1990
> izh       1990  0.0  0.0  24752  3628 pts/0    t+   19:53   0:00 strace -p 1891
> 
> izh@suse2:~> kill 1990 1891
> izh@suse2:~> kill -9 1990 1891
> 
> izh@suse2:~> sudo cat /proc/1891/stack
> [sudo] password for root:
> [<0>] ptrace_stop+0x14a/0x260
> [<0>] ptrace_do_notify+0x91/0xb0
> [<0>] ptrace_notify+0x4e/0x70
> [<0>] do_exit+0x910/0xb70
> [<0>] do_group_exit+0x3a/0xa0
> [<0>] get_signal+0x124/0x800
> [<0>] arch_do_signal_or_restart+0xa9/0x290
> [<0>] exit_to_user_mode_prepare+0xe7/0x1a0
> [<0>] syscall_exit_to_user_mode+0x18/0x40
> [<0>] entry_SYSCALL_64_after_hwframe+0x44/0xa9
> 
> izh@suse2:~> sudo cat /proc/1990/stack
> [<0>] ptrace_stop+0x14a/0x260
> [<0>] ptrace_do_notify+0x91/0xb0
> [<0>] ptrace_notify+0x4e/0x70
> [<0>] do_exit+0x910/0xb70
> [<0>] do_group_exit+0x3a/0xa0
> [<0>] get_signal+0x124/0x800
> [<0>] arch_do_signal_or_restart+0xa9/0x290
> [<0>] exit_to_user_mode_prepare+0xe7/0x1a0
> [<0>] syscall_exit_to_user_mode+0x18/0x40
> [<0>] entry_SYSCALL_64_after_hwframe+0x44/0xa9
> 
> izh@suse2:~> cat /proc/1891/status
> Name:   strace
> Umask:  0022
> State:  t (tracing stop)
> Tgid:   1891
> Ngid:   0
> Pid:    1891
> PPid:   1890
> TracerPid:      1990
> Uid:    1000    1000    1000    1000
> Gid:    100     100     100     100
> FDSize: 256
> Groups: 100
> NStgid: 1891
> NSpid:  1891
> NSpgid: 1891
> NSsid:  1891
> VmPeak:    24752 kB
> VmSize:    24752 kB
> VmLck:         0 kB
> VmPin:         0 kB
> VmHWM:      3828 kB
> VmRSS:      3828 kB
> RssAnon:             520 kB
> RssFile:            3308 kB
> RssShmem:              0 kB
> VmData:      284 kB
> VmStk:       132 kB
> VmExe:      1108 kB
> VmLib:      2828 kB
> VmPTE:        80 kB
> VmSwap:        0 kB
> HugetlbPages:          0 kB
> CoreDumping:    0
> THP_enabled:    1
> Threads:        1
> SigQ:   4/15639
> SigPnd: 0000000000000000
> ShdPnd: 0000000000014100
> SigBlk: 0000000000002000
> SigIgn: 0000000000300000
> SigCgt: 0000000180007007
> CapInh: 0000000000000000
> CapPrm: 0000000000000000
> CapEff: 0000000000000000
> CapBnd: 000001ffffffffff
> CapAmb: 0000000000000000
> NoNewPrivs:     0
> Seccomp:        0
> Seccomp_filters:        0
> Speculation_Store_Bypass:       vulnerable
> SpeculationIndirectBranch:      always enabled
> Cpus_allowed:   7
> Cpus_allowed_list:      0-2
> Mems_allowed:   00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000001
> Mems_allowed_list:      0
> voluntary_ctxt_switches:        1561
> nonvoluntary_ctxt_switches:     7
> 
> izh@suse2:~> cat /proc/1990/status
> Name:   strace
> Umask:  0022
> State:  t (tracing stop)
> Tgid:   1990
> Ngid:   0
> Pid:    1990
> PPid:   1847
> TracerPid:      1891
> Uid:    1000    1000    1000    1000
> Gid:    100     100     100     100
> FDSize: 256
> Groups: 100
> NStgid: 1990
> NSpid:  1990
> NSpgid: 1990
> NSsid:  1847
> VmPeak:    24752 kB
> VmSize:    24752 kB
> VmLck:         0 kB
> VmPin:         0 kB
> VmHWM:      3628 kB
> VmRSS:      3628 kB
> RssAnon:             520 kB
> RssFile:            3108 kB
> RssShmem:              0 kB
> VmData:      284 kB
> VmStk:       132 kB
> VmExe:      1108 kB
> VmLib:      2828 kB
> VmPTE:        88 kB
> VmSwap:        0 kB
> HugetlbPages:          0 kB
> CoreDumping:    0
> THP_enabled:    1
> Threads:        1
> SigQ:   4/15639
> SigPnd: 0000000000000000
> ShdPnd: 0000000000014100
> SigBlk: 0000000000002000
> SigIgn: 0000000000300000
> SigCgt: 0000000180007007
> CapInh: 0000000000000000
> CapPrm: 0000000000000000
> CapEff: 0000000000000000
> CapBnd: 000001ffffffffff
> CapAmb: 0000000000000000
> NoNewPrivs:     0
> Seccomp:        0
> Seccomp_filters:        0
> Speculation_Store_Bypass:       vulnerable
> SpeculationIndirectBranch:      always enabled
> Cpus_allowed:   7
> Cpus_allowed_list:      0-2
> Mems_allowed:   00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000001
> Mems_allowed_list:      0
> voluntary_ctxt_switches:        180
> nonvoluntary_ctxt_switches:     848
> 
> On 29.03.2021 19:49, Oleg Nesterov wrote:
> >On 03/29, Igor Zhbanov wrote:
> >>
> >>Mutual debugging of 2 processes can stuck in unkillable stopped state
> >
> >can't reproduce and can't understand...
> >
> >>Hi!
> >>
> >>When one process, let's say "A", is tracing the another process "B", and the
> >>process "B" is trying to attach to the process "A", then both of them are
> >>getting stuck in the "t+" state. And they are ignoring all of the signals
> >>including the SIGKILL,
> >
> >Why do you think so? What is your kernel version?
> >
> >"t" means TASK_TRACED, SIGKILL should wake it up and terminate.
> >
> >>so it is not possible to terminate them without
> >>a reboot.
> >>
> >>To reproduce:
> >>1) Run two terminals
> >>2) Attach with "strace -p ..." from the first terminal to the shell (bash) of
> >>    the second terminal.
> >>3) In the second terminal run "exec strace -p ..." to attach to the PID of the
> >>    first strace.
> >>
> >>Then you'll see that the second strace is hanging without any output. And the
> >>first strace will output following and hang too:
> >>ptrace(PTRACE_SEIZE, 11795, NULL,
> >>        PTRACE_O_TRACESYSGOOD|PTRACE_O_TRACEEXEC|PTRACE_O_TRACEEXIT
> >>
> >>(The 11795 is the PID of the first strace itself.)
> >>
> >>And in the process list you will see following:
> >>ps awux | grep strace
> >>user   11776  0.0  0.0  24752  2248 pts/3    t+   13:53   0:00 strace -p 11795
> >>user   11795  0.0  0.0  24752  3888 pts/1    t+   13:54   0:00 strace -p 11776
> >
> >OK, may be they sleep in PTRACE_EVENT_EXIT? After you tried to send SIGKILL?
> >
> >please show us the output from "cat /proc/{11795,11776}/stack". And
> >"cat /proc/{11795,11776}/status" just in case.
>