Catching process termination on SIGKILL

Catching process termination on SIGKILL

[hsultan]

Hi,

So I'm curious, auditd catches abnormal process termination (SIGSEGV, 
...) with a 1701 audit message, can catch 'clean' termination by 
monitoring syscall (exit, exitgroup), however I don't see anything to 
catch process termination by a SIGKILL.
if I audit the kill() system call then I see the call to send the 
signal, but I would have expected the system to offer auditing of an 
actual SIGKILL *reception* (because you can pass -1 as target PID to 
sigkill, which kills all processes reachable by the caller and will make 
auditing by syscall very hard to do), am I missing something ? Is there 
a parameter to set somehow that I'm missing ?

Thanks,

Hassan

Re: Catching process termination on SIGKILL

[Steve Grubb]

On Monday, January 26, 2015 03:14:20 PM hsultan@thefroid.net wrote:
> So I'm curious, auditd catches abnormal process termination (SIGSEGV,
> ...) with a 1701 audit message, can catch 'clean' termination by
> monitoring syscall (exit, exitgroup), however I don't see anything to
> catch process termination by a SIGKILL.
> if I audit the kill() system call then I see the call to send the
> signal, but I would have expected the system to offer auditing of an
> actual SIGKILL *reception* (because you can pass -1 as target PID to
> sigkill, which kills all processes reachable by the caller and will make
> auditing by syscall very hard to do), am I missing something ?

I don't think so.

> Is there a parameter to set somehow that I'm missing ?

No. This would probably need some kind of kernel patch to enable. Its never 
really come up that anyone would want to monitor for this. Typically the 
monitoring is on the sending side rather than the receiving side.

We collect anything that leads to a core dump because that is an anomally. No 
one should have segfaulting code on a production system. However, the kernel 
does not allow a SIGKILL to be delivered to processes the user has no rights 
to send it to, so its not really an abnormal event. I could see someone maybe 
wanting to monitor this, but its never been a priority to solve this problem.

-Steve

Re: Catching process termination on SIGKILL

[hsultan]

On 2015-01-26 16:41, Steve Grubb wrote:
> On Monday, January 26, 2015 03:14:20 PM hsultan@thefroid.net wrote:
>> So I'm curious, auditd catches abnormal process termination 
>> (SIGSEGV,
>> ...) with a 1701 audit message, can catch 'clean' termination by
>> monitoring syscall (exit, exitgroup), however I don't see anything 
>> to
>> catch process termination by a SIGKILL.
>> if I audit the kill() system call then I see the call to send the
>> signal, but I would have expected the system to offer auditing of an
>> actual SIGKILL *reception* (because you can pass -1 as target PID to
>> sigkill, which kills all processes reachable by the caller and will 
>> make
>> auditing by syscall very hard to do), am I missing something ?
>
> I don't think so.
>
>> Is there a parameter to set somehow that I'm missing ?
>
> No. This would probably need some kind of kernel patch to enable. Its 
> never
> really come up that anyone would want to monitor for this. Typically 
> the
> monitoring is on the sending side rather than the receiving side.
>
> We collect anything that leads to a core dump because that is an 
> anomally. No
> one should have segfaulting code on a production system. However, the 
> kernel
> does not allow a SIGKILL to be delivered to processes the user has no 
> rights
> to send it to, so its not really an abnormal event. I could see 
> someone maybe
> wanting to monitor this, but its never been a priority to solve this 
> problem.

I see. Auditing SIGKILL reception would allow for easy tracking of 
process activity by following clone/fork/vfork/exit/exit group/abnormal 
termination and then SIGKILL. Without it, it becomes a kludge requiring 
to track kill/tkill/tgkill and trying to find which process will accept 
the SIGKILL sent and which won't, which then requires keeping track of 
process privileges and such.

I'll try to figure out what a patch to audit the KILL reception would 
look like, intent would be to provide the sender's PID + the target PID 
in the audit msg. Should that be a new AUDIT msg type or do you see it 
fit within an existing msg type ?

Thanks,

Hassan

Re: Catching process termination on SIGKILL

[Tetsuo Handa]

Hassan wrote:
> On 2015-01-26 16:41, Steve Grubb wrote:
> > We collect anything that leads to a core dump because that is an 
> > anomally. No
> > one should have segfaulting code on a production system. However, the 
> > kernel
> > does not allow a SIGKILL to be delivered to processes the user has no 
> > rights
> > to send it to, so its not really an abnormal event. I could see 
> > someone maybe
> > wanting to monitor this, but its never been a priority to solve this 
> > problem.

Well, the OOM killer can deliver SIGKILL to processes the user has no rights
to send it to. ;-)

> I see. Auditing SIGKILL reception would allow for easy tracking of 
> process activity by following clone/fork/vfork/exit/exit group/abnormal 
> termination and then SIGKILL. Without it, it becomes a kludge requiring 
> to track kill/tkill/tgkill and trying to find which process will accept 
> the SIGKILL sent and which won't, which then requires keeping track of 
> process privileges and such.

Do you have to implement it using audit subsystem? If you want to track
process activity for temporary (or debug) purpose, SystemTap would do it.

---------- program start ----------
# stap -e '
probe kernel.function("do_exit") {
  if ($code & 0x7F)
    printf("%s %s(%u) exiting with signal %u\n",
           ctime(gettimeofday_s()), execname(), pid(), $code & 0x7F);
}'
---------- program end ----------

---------- output example start ----------
Sat May 3 06:00:39 2014 a.out(2101) exiting with signal 11
Sat May 3 06:00:48 2014 sleep(2102) exiting with signal 2
Sat May 3 06:01:17 2014 sleep(2105) exiting with signal 9
Sat May 3 06:01:21 2014 a.out(2131) exiting with signal 11
---------- output example end ----------

> 
> I'll try to figure out what a patch to audit the KILL reception would 
> look like, intent would be to provide the sender's PID + the target PID 
> in the audit msg. Should that be a new AUDIT msg type or do you see it 
> fit within an existing msg type ?

SystemTap would do it, if you can accept SystemTap.

Re: Catching process termination on SIGKILL

[hsultan]

On 2015-01-27 04:11, Tetsuo Handa wrote:
...
> Do you have to implement it using audit subsystem? If you want to 
> track
> process activity for temporary (or debug) purpose, SystemTap would do 
> it.
>
> ---------- program start ----------
> # stap -e '
> probe kernel.function("do_exit") {
>   if ($code & 0x7F)
>     printf("%s %s(%u) exiting with signal %u\n",
>            ctime(gettimeofday_s()), execname(), pid(), $code & 0x7F);
> }'
> ---------- program end ----------
>
> ---------- output example start ----------
> Sat May 3 06:00:39 2014 a.out(2101) exiting with signal 11
> Sat May 3 06:00:48 2014 sleep(2102) exiting with signal 2
> Sat May 3 06:01:17 2014 sleep(2105) exiting with signal 9
> Sat May 3 06:01:21 2014 a.out(2131) exiting with signal 11
> ---------- output example end ----------
>
>>
>> I'll try to figure out what a patch to audit the KILL reception 
>> would
>> look like, intent would be to provide the sender's PID + the target 
>> PID
>> in the audit msg. Should that be a new AUDIT msg type or do you see 
>> it
>> fit within an existing msg type ?
>
> SystemTap would do it, if you can accept SystemTap.

Sadly I can't use SystemTap as I do not control the systems where my 
code will be running so can't be sure that debug information will be 
available :/

Thanks,

Hassan