Auditd reconfigure using SIGHUP

Auditd reconfigure using SIGHUP

[Bhagwat, Shriniketan Manjunath]

[-- Attachment #1.1: Type: text/plain, Size: 3816 bytes --]

Greetings,

I was trying to reconfigure the Audit using SIGHUP and came across with below behavior of Audit. The audit version I am using is 2.3.6.

Scenario 1: Starting auditd with active audispd plugin.
1. Activate the audispd plugin by setting active = yes in configuration file (example: /etc/audisp/plugins.d/syslog.conf)
2. Start the Auditd. Starting auditd will create the both Auditd and Audispd process.
3. Deactivate the audispd plugin by setting active = no in configuration file.
4. Reconfigure the Auditd by sending SIGHUP to Auditd (>> kill -SIGHUP <auditd_pid>). This reconfiguration stops the audispd process.
5. Activate the audispd plugin by setting active = yes in configuration file.
6. Reconfigure the Auditd by sending SIGHUP to Auditd. This should start the audispd process, however audispd process will not be started in this reconfiguration.

Scenario2: Starting auditd with no active audispd plugin.
1. Start the Auditd. Starting auditd will create the only Auditd process.
2. Activate the audispd plugin by setting active = yes in configuration file.
3. Reconfigure the Auditd by sending SIGHUP to Auditd. This should start the audispd process, however in few cases audispd process will not be started in this reconfiguration.
4. Deactivate the audispd plugin by setting active = no in configuration file.
5. Reconfigure the Auditd by sending SIGHUP to Auditd. This reconfiguration stops the audispd process.
6. Activate the audispd plugin by setting active = yes in configuration file.
7. Reconfigure the Auditd by sending SIGHUP to Auditd. This should start the audispd process, however audispd process will not be started in this reconfiguration.


As per the change log of audit version  2.3.2, below fix was made.

Fix: In auditd, restart dispatcher on SIGHUP if it had previously exited



I have analyzed Auditd code of version 2.3.6 and below is my observation.

When the Auditd is started, it calls init_dispatcher() to start the Audispd. init_dispatcher() starts the Audispd and maintains its pid value in a global variable (auditd-dispatch.c). When the audispd is terminated, SIGCHLD handler of Auditd i.e child_handler() does the waitpid() to remove the child process from zombie state and calls dispatcher_reaped(), where this pid is set to zero. During reconfigure using SIGHUP, reconfigure_dispatcher() checks for this pid value. If pid is valid then, SIGHUP is sent to audispd otherwise init_dispatcher() is called to start the audispd.


Auditd uses event loop ev_signal to track the child process (SIGCHLD). The event loop from libev also has child handler childcb() (in ev.c). This handler childcb() also does the waitpid(). When audispd terminates, libev's child handler is getting called first, waitpid() removes the audispd from zombie state. Then the control is passed to Auditd's child handler child_handler(). Since the audispd has been already removed from process stable, waitpid() call done in child_handler() will return ECHILD. Hence the dispatcher_reaped() is not getting called to make the internally maintained pid variable to zero. This results in subsequent SIGHUP set to Auditd end up in sending SIGHUP to non-existing audispd process.

In case of step 3 of scenario 2: When the auditd is started without any active audispd plugin(step1), Auditd starts the audispd. However since there are no active plugins Audispd will get terminated. In this case shutdown_dispatcher() was called from dispatch_event() where pid value was set to zero. Hence the reconfiguration in step 3 started audispd process in my case.

I have put traces in auditd code to validate the above behavior. This behavior is applicable for Audit version 2.6.4 as well.  Is this behavior a known issue?

Regards,
Ketan














[-- Attachment #1.2: Type: text/html, Size: 8395 bytes --]

[-- Attachment #2: Type: text/plain, Size: 0 bytes --]

Re: Auditd reconfigure using SIGHUP

[Steve Grubb]

On Thursday, January 5, 2017 11:04:55 AM EST Bhagwat, Shriniketan Manjunath 
wrote:
> Greetings,
> 
> I was trying to reconfigure the Audit using SIGHUP and came across with
> below behavior of Audit. The audit version I am using is 2.3.6.

I have been able to reproduce the issue. I am still tracing through the 
problem. As best I can tell, there are at least 3 problems. 1) libev eating 
the signal - this will be fixed by defining EV_CHILD_ENABLE to 0. This will make 
all child interference go away; 2) on startup audispd detects no plugins and 
exits before auditd can register a handler in scenario 2 - this might be fixed 
with a sigaction handler until libev starts processing events; 3) pipe 
descriptors being invalid in audispd sometimes. This one is a mystery.

Still investigating. Thanks for reporting this.

-Steve

> Scenario 1: Starting auditd with active audispd plugin.
> 1. Activate the audispd plugin by setting active = yes in configuration file
> (example: /etc/audisp/plugins.d/syslog.conf) 2. Start the Auditd. Starting
> auditd will create the both Auditd and Audispd process. 3. Deactivate the
> audispd plugin by setting active = no in configuration file. 4. Reconfigure
> the Auditd by sending SIGHUP to Auditd (>> kill -SIGHUP <auditd_pid>). This
> reconfiguration stops the audispd process. 5. Activate the audispd plugin
> by setting active = yes in configuration file. 6. Reconfigure the Auditd by
> sending SIGHUP to Auditd. This should start the audispd process, however
> audispd process will not be started in this reconfiguration.
> 
> Scenario2: Starting auditd with no active audispd plugin.
> 1. Start the Auditd. Starting auditd will create the only Auditd process.
> 2. Activate the audispd plugin by setting active = yes in configuration
> file. 3. Reconfigure the Auditd by sending SIGHUP to Auditd. This should
> start the audispd process, however in few cases audispd process will not be
> started in this reconfiguration. 4. Deactivate the audispd plugin by
> setting active = no in configuration file. 5. Reconfigure the Auditd by
> sending SIGHUP to Auditd. This reconfiguration stops the audispd process.
> 6. Activate the audispd plugin by setting active = yes in configuration
> file. 7. Reconfigure the Auditd by sending SIGHUP to Auditd. This should
> start the audispd process, however audispd process will not be started in
> this reconfiguration.
> 
> 
> As per the change log of audit version  2.3.2, below fix was made.
> 
> Fix: In auditd, restart dispatcher on SIGHUP if it had previously exited
> 
> 
> 
> I have analyzed Auditd code of version 2.3.6 and below is my observation.
> 
> When the Auditd is started, it calls init_dispatcher() to start the Audispd.
> init_dispatcher() starts the Audispd and maintains its pid value in a
> global variable (auditd-dispatch.c). When the audispd is terminated,
> SIGCHLD handler of Auditd i.e child_handler() does the waitpid() to remove
> the child process from zombie state and calls dispatcher_reaped(), where
> this pid is set to zero. During reconfigure using SIGHUP,
> reconfigure_dispatcher() checks for this pid value. If pid is valid then,
> SIGHUP is sent to audispd otherwise init_dispatcher() is called to start
> the audispd.
> 
> 
> Auditd uses event loop ev_signal to track the child process (SIGCHLD). The
> event loop from libev also has child handler childcb() (in ev.c). This
> handler childcb() also does the waitpid(). When audispd terminates, libev's
> child handler is getting called first, waitpid() removes the audispd from
> zombie state. Then the control is passed to Auditd's child handler
> child_handler(). Since the audispd has been already removed from process
> stable, waitpid() call done in child_handler() will return ECHILD. Hence
> the dispatcher_reaped() is not getting called to make the internally
> maintained pid variable to zero. This results in subsequent SIGHUP set to
> Auditd end up in sending SIGHUP to non-existing audispd process.
> 
> In case of step 3 of scenario 2: When the auditd is started without any
> active audispd plugin(step1), Auditd starts the audispd. However since
> there are no active plugins Audispd will get terminated. In this case
> shutdown_dispatcher() was called from dispatch_event() where pid value was
> set to zero. Hence the reconfiguration in step 3 started audispd process in
> my case.
> 
> I have put traces in auditd code to validate the above behavior. This
> behavior is applicable for Audit version 2.6.4 as well.  Is this behavior a
> known issue?
> 
> Regards,
> Ketan

RE: Auditd reconfigure using SIGHUP

[Bhagwat, Shriniketan Manjunath]

Hi Steve,

Thanks for acknowledging the issue. 

In my last email I missed mentioning the fix that I have implemented.

Issue 1) As you said, I have fixed it  by replacing the ev_signal by ev_child as below.

struct ev_child sigchld_watcher;

        ev_child_init (&sigchld_watcher, child_handler, 0, 0);
        ev_child_start (EV_DEFAULT_ &sigchld_watcher);

static void child_handler(EV_P_ ev_child *w, int revents)
{
        int pid;

        if (w->rpid == dispatcher_pid()) {
                       dispatcher_reaped();
        }
}

Issue 2) In auditd.c main(), child_handler is registered not immediately after init_dispatcher() is called. I have modified the audit to register ev_child immediately after init_dispatcher() as below. Or maybe before calling init_dispatcher(). This fixed issue 2 for me. Below extract is from documentation of libev for ev_child: 
" It is permissible to install a child watcher after the child has been forked (which implies it might have already exited), as long as the event loop isn't entered (or is continued from a watcher), i.e., forking and then immediately registering a watcher for the child is fine, but forking and registering a watcher a few event loop iterations later or in the next callback invocation is not." 

        if (init_dispatcher(&config)) {
                if (pidfile)
                        unlink(pidfile);
                tell_parent(FAILURE);
                return 1;
        }
        ev_child_init (&sigchld_watcher, child_handler, 0, 0);
        ev_child_start (EV_DEFAULT_ &sigchld_watcher);

Issue 3) With the above fix for issue 2, I did not see the issue 3 getting occurred for me. This could be because shutdown_dispatcher() is called from dispatcher_reaped() where the status on the pipe is not checked.  

Regards,
Ketan

-----Original Message-----
From: Steve Grubb [mailto:sgrubb@redhat.com] 
Sent: Friday, January 06, 2017 4:12 AM
To: linux-audit@redhat.com
Cc: Bhagwat, Shriniketan Manjunath <shriniketan.bhagwat@hpe.com>
Subject: Re: Auditd reconfigure using SIGHUP

On Thursday, January 5, 2017 11:04:55 AM EST Bhagwat, Shriniketan Manjunath
wrote:
> Greetings,
> 
> I was trying to reconfigure the Audit using SIGHUP and came across 
> with below behavior of Audit. The audit version I am using is 2.3.6.

I have been able to reproduce the issue. I am still tracing through the problem. As best I can tell, there are at least 3 problems. 1) libev eating the signal - this will be fixed by defining EV_CHILD_ENABLE to 0. This will make all child interference go away; 2) on startup audispd detects no plugins and exits before auditd can register a handler in scenario 2 - this might be fixed with a sigaction handler until libev starts processing events; 3) pipe descriptors being invalid in audispd sometimes. This one is a mystery.

Still investigating. Thanks for reporting this.

-Steve

> Scenario 1: Starting auditd with active audispd plugin.
> 1. Activate the audispd plugin by setting active = yes in 
> configuration file
> (example: /etc/audisp/plugins.d/syslog.conf) 2. Start the Auditd. 
> Starting auditd will create the both Auditd and Audispd process. 3. 
> Deactivate the audispd plugin by setting active = no in configuration 
> file. 4. Reconfigure the Auditd by sending SIGHUP to Auditd (>> kill 
> -SIGHUP <auditd_pid>). This reconfiguration stops the audispd process. 
> 5. Activate the audispd plugin by setting active = yes in 
> configuration file. 6. Reconfigure the Auditd by sending SIGHUP to 
> Auditd. This should start the audispd process, however audispd process will not be started in this reconfiguration.
> 
> Scenario2: Starting auditd with no active audispd plugin.
> 1. Start the Auditd. Starting auditd will create the only Auditd process.
> 2. Activate the audispd plugin by setting active = yes in 
> configuration file. 3. Reconfigure the Auditd by sending SIGHUP to 
> Auditd. This should start the audispd process, however in few cases 
> audispd process will not be started in this reconfiguration. 4. 
> Deactivate the audispd plugin by setting active = no in configuration 
> file. 5. Reconfigure the Auditd by sending SIGHUP to Auditd. This reconfiguration stops the audispd process.
> 6. Activate the audispd plugin by setting active = yes in 
> configuration file. 7. Reconfigure the Auditd by sending SIGHUP to 
> Auditd. This should start the audispd process, however audispd process 
> will not be started in this reconfiguration.
> 
> 
> As per the change log of audit version  2.3.2, below fix was made.
> 
> Fix: In auditd, restart dispatcher on SIGHUP if it had previously 
> exited
> 
> 
> 
> I have analyzed Auditd code of version 2.3.6 and below is my observation.
> 
> When the Auditd is started, it calls init_dispatcher() to start the Audispd.
> init_dispatcher() starts the Audispd and maintains its pid value in a 
> global variable (auditd-dispatch.c). When the audispd is terminated, 
> SIGCHLD handler of Auditd i.e child_handler() does the waitpid() to 
> remove the child process from zombie state and calls 
> dispatcher_reaped(), where this pid is set to zero. During reconfigure 
> using SIGHUP,
> reconfigure_dispatcher() checks for this pid value. If pid is valid 
> then, SIGHUP is sent to audispd otherwise init_dispatcher() is called 
> to start the audispd.
> 
> 
> Auditd uses event loop ev_signal to track the child process (SIGCHLD). 
> The event loop from libev also has child handler childcb() (in ev.c). 
> This handler childcb() also does the waitpid(). When audispd 
> terminates, libev's child handler is getting called first, waitpid() 
> removes the audispd from zombie state. Then the control is passed to 
> Auditd's child handler child_handler(). Since the audispd has been 
> already removed from process stable, waitpid() call done in 
> child_handler() will return ECHILD. Hence the dispatcher_reaped() is 
> not getting called to make the internally maintained pid variable to 
> zero. This results in subsequent SIGHUP set to Auditd end up in sending SIGHUP to non-existing audispd process.
> 
> In case of step 3 of scenario 2: When the auditd is started without 
> any active audispd plugin(step1), Auditd starts the audispd. However 
> since there are no active plugins Audispd will get terminated. In this 
> case
> shutdown_dispatcher() was called from dispatch_event() where pid value 
> was set to zero. Hence the reconfiguration in step 3 started audispd 
> process in my case.
> 
> I have put traces in auditd code to validate the above behavior. This 
> behavior is applicable for Audit version 2.6.4 as well.  Is this 
> behavior a known issue?
> 
> Regards,
> Ketan

Re: Auditd reconfigure using SIGHUP

[Steve Grubb]

Hello,

On Friday, January 6, 2017 6:29:59 AM EST Bhagwat, Shriniketan Manjunath 
wrote:
> In my last email I missed mentioning the fix that I have implemented.
> 
> Issue 1) As you said, I have fixed it  by replacing the ev_signal by
> ev_child as below.
> 
> struct ev_child sigchld_watcher;
> 
>         ev_child_init (&sigchld_watcher, child_handler, 0, 0);
>         ev_child_start (EV_DEFAULT_ &sigchld_watcher);
> 
> static void child_handler(EV_P_ ev_child *w, int revents)
> {
>         int pid;
> 
>         if (w->rpid == dispatcher_pid()) {
>                        dispatcher_reaped();
>         }
> }

I tried this as a first step yesterday but what happens is the problem gets 
worse. It thinks the dispatcher is running all the time and never tries to 
restart it.
 
> Issue 2) In auditd.c main(), child_handler is registered not immediately
> after init_dispatcher() is called. I have modified the audit to register
> ev_child immediately after init_dispatcher() as below. Or maybe before
> calling init_dispatcher(). This fixed issue 2 for me. Below extract is from
> documentation of libev for ev_child: " It is permissible to install a child
> watcher after the child has been forked (which implies it might have
> already exited), as long as the event loop isn't entered (or is continued
> from a watcher), i.e., forking and then immediately registering a watcher
> for the child is fine, but forking and registering a watcher a few event
> loop iterations later or in the next callback invocation is not."
> 
>         if (init_dispatcher(&config)) {
>                 if (pidfile)
>                         unlink(pidfile);
>                 tell_parent(FAILURE);
>                 return 1;
>         }
>         ev_child_init (&sigchld_watcher, child_handler, 0, 0);
>         ev_child_start (EV_DEFAULT_ &sigchld_watcher);
> 
> Issue 3) With the above fix for issue 2, I did not see the issue 3 getting
> occurred for me. This could be because shutdown_dispatcher() is called from
> dispatcher_reaped() where the status on the pipe is not checked.

Using the above code I still see the descriptor getting stepped on by 
something. I have added some debug info to audispd in svn which makes the 
problem more clear.

Jan  6 11:43:13 audispd: Failed setting up input(Bad file descriptor, -1), 
exiting

In case anyone else wishes to have a regression test, here's some code:

#!/bin/sh

while [ 1 ]
do
        echo "disabling sedispatch"
        sed -i '/active/s/yes/no/' /etc/audisp/plugins.d/sedispatch.conf
        kill -HUP `pidof auditd`
        sleep 10
        pstree -p `pidof auditd`
        echo "enabling sedispatch"
        sed -i '/active/s/no/yes/' /etc/audisp/plugins.d/sedispatch.conf
        kill -HUP `pidof auditd`
        sleep 10
        pstree -p `pidof auditd`
done

Of course you might want to change the plugin that's being altered to 
something else.

-Steve
 

Re: Auditd reconfigure using SIGHUP

[Steve Grubb]

Hello,

On Friday, January 6, 2017 11:44:21 AM EST Steve Grubb wrote:
> I tried this as a first step yesterday but what happens is the problem gets
> worse. It thinks the dispatcher is running all the time and never tries to
> restart it.

Thanks for reporting this. Its fixed by upstream commits 1458 - 1466. There 
were quite a few issues in this section of the code.

-Steve

RE: Auditd reconfigure using SIGHUP

[Bhagwat, Shriniketan Manjunath]

Hi,

Where can I see the changes? I tried to in the below location, however it's not.

https://fedorahosted.org/audit/browser/branches/2.4/src
https://github.com/linux-audit/audit-userspace/tree/master/src

Regards,
Ketan

-----Original Message-----
From: Steve Grubb [mailto:sgrubb@redhat.com] 
Sent: Monday, January 09, 2017 12:39 AM
To: linux-audit@redhat.com
Cc: Bhagwat, Shriniketan Manjunath <shriniketan.bhagwat@hpe.com>
Subject: Re: Auditd reconfigure using SIGHUP

Hello,

On Friday, January 6, 2017 11:44:21 AM EST Steve Grubb wrote:
> I tried this as a first step yesterday but what happens is the problem 
> gets worse. It thinks the dispatcher is running all the time and never 
> tries to restart it.

Thanks for reporting this. Its fixed by upstream commits 1458 - 1466. There were quite a few issues in this section of the code.

-Steve

Re: Auditd reconfigure using SIGHUP

[Steve Grubb]

On Monday, January 9, 2017 1:30:57 PM EST Bhagwat, Shriniketan Manjunath wrote:
> Hi,
> 
> Where can I see the changes? 

This is the cumulative set of patches:
https://fedorahosted.org/audit/changeset?reponame=&new=1466%40trunk&old=1457%40trunk

> I tried to in the below location, however it's not.
> 
> https://fedorahosted.org/audit/browser/branches/2.4/src

I have not added this to the 2.4 branch.

> https://github.com/linux-audit/audit-userspace/tree/master/src

This may not have sync'ed yet.

-Steve

> -----Original Message-----
> From: Steve Grubb [mailto:sgrubb@redhat.com]
> Sent: Monday, January 09, 2017 12:39 AM
> To: linux-audit@redhat.com
> Cc: Bhagwat, Shriniketan Manjunath <shriniketan.bhagwat@hpe.com>
> Subject: Re: Auditd reconfigure using SIGHUP
> 
> Hello,
> 
> On Friday, January 6, 2017 11:44:21 AM EST Steve Grubb wrote:
> > I tried this as a first step yesterday but what happens is the problem
> > gets worse. It thinks the dispatcher is running all the time and never
> > tries to restart it.
> 
> Thanks for reporting this. Its fixed by upstream commits 1458 - 1466. There
> were quite a few issues in this section of the code.
> 
> -Steve