Re: Mutual debugging of 2 processes can stuck in unkillable stopped state

Igor Zhbanov
2021-03-29T17:11:13Z

Hi Oleg!

I've tried both 5.3.18 and 5.10.0. The behavior is the same.
The important thing is to run "exec strace -p ..." on the second terminal
to create the loop A->B->A.

So the last line from the first strace we see is:
ptrace(PTRACE_SEIZE, 1990, NULL, PTRACE_O_TRACESYSGOOD|PTRACE_O_TRACEEXEC|PTRACE_O_TRACEEXIT

I.e. it printed the syscall prior to its execution and hanged after the
execution.

izh@suse2:~> ps awux|grep strace
izh 1891 0.0 0.0 24752 3828 pts/1 ts+ 19:52 0:00 strace -p 1990
izh 1990 0.0 0.0 24752 3628 pts/0 t+ 19:53 0:00 strace -p 1891

izh@suse2:~> kill 1990 1891
izh@suse2:~> kill -9 1990 1891

izh@suse2:~> sudo cat /proc/1891/stack
[sudo] password for root:
[<0>] ptrace_stop+0x14a/0x260
[<0>] ptrace_do_notify+0x91/0xb0
[<0>] ptrace_notify+0x4e/0x70
[<0>] do_exit+0x910/0xb70
[<0>] do_group_exit+0x3a/0xa0
[<0>] get_signal+0x124/0x800
[<0>] arch_do_signal_or_restart+0xa9/0x290
[<0>] exit_to_user_mode_prepare+0xe7/0x1a0
[<0>] syscall_exit_to_user_mode+0x18/0x40
[<0>] entry_SYSCALL_64_after_hwframe+0x44/0xa9

izh@suse2:~> sudo cat /proc/1990/stack
[<0>] ptrace_stop+0x14a/0x260
[<0>] ptrace_do_notify+0x91/0xb0
[<0>] ptrace_notify+0x4e/0x70
[<0>] do_exit+0x910/0xb70
[<0>] do_group_exit+0x3a/0xa0
[<0>] get_signal+0x124/0x800
[<0>] arch_do_signal_or_restart+0xa9/0x290
[<0>] exit_to_user_mode_prepare+0xe7/0x1a0
[<0>] syscall_exit_to_user_mode+0x18/0x40
[<0>] entry_SYSCALL_64_after_hwframe+0x44/0xa9

izh@suse2:~> cat /proc/1891/status
Name: strace
Umask: 0022
State: t (tracing stop)
Tgid: 1891
Ngid: 0
Pid: 1891
PPid: 1890
TracerPid: 1990
Uid: 1000 1000 1000 1000
Gid: 100 100 100 100
FDSize: 256
Groups: 100
NStgid: 1891
NSpid: 1891
NSpgid: 1891
NSsid: 1891
VmPeak: 24752 kB
VmSize: 24752 kB
VmLck: 0 kB
VmPin: 0 kB
VmHWM: 3828 kB
VmRSS: 3828 kB
RssAnon: 520 kB
RssFile: 3308 kB
RssShmem: 0 kB
VmData: 284 kB
VmStk: 132 kB
VmExe: 1108 kB
VmLib: 2828 kB
VmPTE: 80 kB
VmSwap: 0 kB
HugetlbPages: 0 kB
CoreDumping: 0
THP_enabled: 1
Threads: 1
SigQ: 4/15639
SigPnd: 0000000000000000
ShdPnd: 0000000000014100
SigBlk: 0000000000002000
SigIgn: 0000000000300000
SigCgt: 0000000180007007
CapInh: 0000000000000000
CapPrm: 0000000000000000
CapEff: 0000000000000000
CapBnd: 000001ffffffffff
CapAmb: 0000000000000000
NoNewPrivs: 0
Seccomp: 0
Seccomp_filters: 0
Speculation_Store_Bypass: vulnerable
SpeculationIndirectBranch: always enabled
Cpus_allowed: 7
Cpus_allowed_list: 0-2
Mems_allowed: 00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000001
Mems_allowed_list: 0
voluntary_ctxt_switches: 1561
nonvoluntary_ctxt_switches: 7

izh@suse2:~> cat /proc/1990/status
Name: strace
Umask: 0022
State: t (tracing stop)
Tgid: 1990
Ngid: 0
Pid: 1990
PPid: 1847
TracerPid: 1891
Uid: 1000 1000 1000 1000
Gid: 100 100 100 100
FDSize: 256
Groups: 100
NStgid: 1990
NSpid: 1990
NSpgid: 1990
NSsid: 1847
VmPeak: 24752 kB
VmSize: 24752 kB
VmLck: 0 kB
VmPin: 0 kB
VmHWM: 3628 kB
VmRSS: 3628 kB
RssAnon: 520 kB
RssFile: 3108 kB
RssShmem: 0 kB
VmData: 284 kB
VmStk: 132 kB
VmExe: 1108 kB
VmLib: 2828 kB
VmPTE: 88 kB
VmSwap: 0 kB
HugetlbPages: 0 kB
CoreDumping: 0
THP_enabled: 1
Threads: 1
SigQ: 4/15639
SigPnd: 0000000000000000
ShdPnd: 0000000000014100
SigBlk: 0000000000002000
SigIgn: 0000000000300000
SigCgt: 0000000180007007
CapInh: 0000000000000000
CapPrm: 0000000000000000
CapEff: 0000000000000000
CapBnd: 000001ffffffffff
CapAmb: 0000000000000000
NoNewPrivs: 0
Seccomp: 0
Seccomp_filters: 0
Speculation_Store_Bypass: vulnerable
SpeculationIndirectBranch: always enabled
Cpus_allowed: 7
Cpus_allowed_list: 0-2
Mems_allowed: 00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000001
Mems_allowed_list: 0
voluntary_ctxt_switches: 180
nonvoluntary_ctxt_switches: 848

On 29.03.2021 19:49, Oleg Nesterov wrote:
> On 03/29, Igor Zhbanov wrote:
>>
>> Mutual debugging of 2 processes can stuck in unkillable stopped state
>
> can't reproduce and can't understand...
>
>> Hi!
>>
>> When one process, let's say "A", is tracing the another process "B", and the
>> process "B" is trying to attach to the process "A", then both of them are
>> getting stuck in the "t+" state. And they are ignoring all of the signals
>> including the SIGKILL,
>
> Why do you think so? What is your kernel version?
>
> "t" means TASK_TRACED, SIGKILL should wake it up and terminate.
>
>> so it is not possible to terminate them without
>> a reboot.
>>
>> To reproduce:
>> 1) Run two terminals
>> 2) Attach with "strace -p ..." from the first terminal to the shell (bash) of
>> the second terminal.
>> 3) In the second terminal run "exec strace -p ..." to attach to the PID of the
>> first strace.
>>
>> Then you'll see that the second strace is hanging without any output. And the
>> first strace will output following and hang too:
>> ptrace(PTRACE_SEIZE, 11795, NULL,
>> PTRACE_O_TRACESYSGOOD|PTRACE_O_TRACEEXEC|PTRACE_O_TRACEEXIT
>>
>> (The 11795 is the PID of the first strace itself.)
>>
>> And in the process list you will see following:
>> ps awux | grep strace
>> user 11776 0.0 0.0 24752 2248 pts/3 t+ 13:53 0:00 strace -p 11795
>> user 11795 0.0 0.0 24752 3888 pts/1 t+ 13:54 0:00 strace -p 11776
>
> OK, may be they sleep in PTRACE_EVENT_EXIT? After you tried to send SIGKILL?
>
> please show us the output from "cat /proc/{11795,11776}/stack". And
> "cat /proc/{11795,11776}/status" just in case.