audit_pid with multiple userspace auditd processes

audit_pid with multiple userspace auditd processes

[Eric Paris]

So I noticed today something strange, but maybe not wrong?

lets say userspace starts 2 copies of auditd.  Then they kill the first
copy.  The kernel at that point thinks there is no userspace auditd
running and will instead send things to dmesg

We could fix it by changing the handling in audit_receive_msg to reject
setting the audit_pid to 0 if the current audit_nlk_pid !=
NETLINK_CB(skb).pid.

It's not a big deal, maybe we just call results of audit with multiple
userspace auditd's running at the same time a undefined and not care.

Anyone think that's worth a patch?

-Eric

Re: audit_pid with multiple userspace auditd processes

[Linda Knippers]

Eric Paris wrote:
> So I noticed today something strange, but maybe not wrong?
> 
> lets say userspace starts 2 copies of auditd.  

Will a second auditd actually start?  Seems like it shouldn't.

> Then they kill the first
> copy.  The kernel at that point thinks there is no userspace auditd
> running and will instead send things to dmesg
> 
> We could fix it by changing the handling in audit_receive_msg to reject
> setting the audit_pid to 0 if the current audit_nlk_pid !=
> NETLINK_CB(skb).pid.
> 
> It's not a big deal, maybe we just call results of audit with multiple
> userspace auditd's running at the same time a undefined and not care.

I think its something to be avoided.  Can the 2nd auditd exit if
there already is one?

-- ljk
> 
> Anyone think that's worth a patch?
> 
> -Eric
> 
> --
> Linux-audit mailing list
> Linux-audit@redhat.com
> https://www.redhat.com/mailman/listinfo/linux-audit

Re: audit_pid with multiple userspace auditd processes

[Eric Paris]

On Wed, 2009-01-07 at 17:04 -0500, Linda Knippers wrote:
> Eric Paris wrote:
> > So I noticed today something strange, but maybe not wrong?
> > 
> > lets say userspace starts 2 copies of auditd.  
> 
> Will a second auditd actually start?  Seems like it shouldn't.

depends how you start it.  if you use init scripts, no it won't start
another one.  If you just start auditd, nothing stops you.....

> 
> > Then they kill the first
> > copy.  The kernel at that point thinks there is no userspace auditd
> > running and will instead send things to dmesg
> > 
> > We could fix it by changing the handling in audit_receive_msg to reject
> > setting the audit_pid to 0 if the current audit_nlk_pid !=
> > NETLINK_CB(skb).pid.
> > 
> > It's not a big deal, maybe we just call results of audit with multiple
> > userspace auditd's running at the same time a undefined and not care.
> 
> I think its something to be avoided.  Can the 2nd auditd exit if
> there already is one?

the kernel keeps track of the last audit set status message it gets.  it
doesn't care who sent it.  even with my "fix" it isn't great since
starting 2 and killing the second one leaves you a userspace auditd but
kernel still thinks there is nothing out there.....

maybe i'll ponder it, but for now i'm not going to do anything about
it...

Re: audit_pid with multiple userspace auditd processes

[Linda Knippers]

Eric Paris wrote:
> On Wed, 2009-01-07 at 17:04 -0500, Linda Knippers wrote:
>> Eric Paris wrote:
>>> So I noticed today something strange, but maybe not wrong?
>>>
>>> lets say userspace starts 2 copies of auditd.  
>> Will a second auditd actually start?  Seems like it shouldn't.
> 
> depends how you start it.  if you use init scripts, no it won't start
> another one.  If you just start auditd, nothing stops you.....

Seems like auditd itself could be a little smarter.

-- ljk

Re: audit_pid with multiple userspace auditd processes

[Steve Grubb]

On Wednesday 07 January 2009 04:36:39 pm Eric Paris wrote:
> lets say userspace starts 2 copies of auditd.

# auditctl -s
AUDIT_STATUS: enabled=1 flag=1 pid=4488 rate_limit=0 backlog_limit=512 lost=1 
backlog=0
# /sbin/auditd 
# auditctl -s
AUDIT_STATUS: enabled=1 flag=1 pid=0 rate_limit=0 backlog_limit=512 lost=1 
backlog=0
# ps -ef | grep auditd
root       580     2  0 08:19 ?        00:00:00 [kauditd]
root      4488     1  0 16:35 ?        00:00:00 auditd
root      5128  3654  0 17:33 pts/1    00:00:00 grep auditd


> Then they kill the first copy.  The kernel at that point thinks there is no
> userspace auditd running and will instead send things to dmesg

Looks to me like the kernel is setting auditd_pid to 0 and the second auditd 
does not start - at least with my current setup.

For some other setups, it probably overwrites the pid with the new one and 
keeps going.


> We could fix it by changing the handling in audit_receive_msg to reject
> setting the audit_pid to 0 if the current audit_nlk_pid !=
> NETLINK_CB(skb).pid.

Well, what if the first crashed and the kernel didn't know it yet? It might be 
better to forcibly break the connection to the original auditd.


> It's not a big deal, maybe we just call results of audit with multiple
> userspace auditd's running at the same time a undefined and not care.

What do you get for auditctl -s before and after starting your second auditd?

-Steve

Re: audit_pid with multiple userspace auditd processes

[Eric Paris]

On Wed, 2009-01-07 at 17:41 -0500, Steve Grubb wrote:
> On Wednesday 07 January 2009 04:36:39 pm Eric Paris wrote:
> > lets say userspace starts 2 copies of auditd.
> 
> # auditctl -s
> AUDIT_STATUS: enabled=1 flag=1 pid=4488 rate_limit=0 backlog_limit=512 lost=1 
> backlog=0
> # /sbin/auditd 
> # auditctl -s
> AUDIT_STATUS: enabled=1 flag=1 pid=0 rate_limit=0 backlog_limit=512 lost=1 
> backlog=0
> # ps -ef | grep auditd
> root       580     2  0 08:19 ?        00:00:00 [kauditd]
> root      4488     1  0 16:35 ?        00:00:00 auditd
> root      5128  3654  0 17:33 pts/1    00:00:00 grep auditd
> 
> 
> > Then they kill the first copy.  The kernel at that point thinks there is no
> > userspace auditd running and will instead send things to dmesg
> 
> Looks to me like the kernel is setting auditd_pid to 0 and the second auditd 
> does not start - at least with my current setup.
> 
> For some other setups, it probably overwrites the pid with the new one and 
> keeps going.
> 
> 
> > We could fix it by changing the handling in audit_receive_msg to reject
> > setting the audit_pid to 0 if the current audit_nlk_pid !=
> > NETLINK_CB(skb).pid.
> 
> Well, what if the first crashed and the kernel didn't know it yet? It might be 
> better to forcibly break the connection to the original auditd.

I'm only talking about allowing userspace to "cleanly" unset it's belief
there is an auditd out there if the message comes from that process.
We'll still handle death by means of the usual netlink socket
failures...

If auditd number 2 is the auditd the kernel knows about why should
auditd number 1 be allowed to "cleanly" say there is no auditd?

> > It's not a big deal, maybe we just call results of audit with multiple
> > userspace auditd's running at the same time a undefined and not care.
> 
> What do you get for auditctl -s before and after starting your second auditd?

[root@unused kernel-2]# auditd
[root@unused kernel-2]# auditctl -s
AUDIT_STATUS: enabled=1 flag=1 pid=752 rate_limit=0 backlog_limit=320
lost=0 backlog=0
[root@unused kernel-2]# ps -ef | grep auditd
root       752     1  0 17:48 ?        00:00:00 auditd
root       962     2  0 14:54 ?        00:00:00 [kauditd]
root      1301  2207  0 17:49 pts/0    00:00:00 grep auditd
[root@unused kernel-2]# auditd
[root@unused kernel-2]# auditctl -s
AUDIT_STATUS: enabled=1 flag=1 pid=1549 rate_limit=0 backlog_limit=320
lost=0 backlog=0
[root@unused kernel-2]# ps -ef | grep auditd
root       752     1  0 17:48 ?        00:00:00 auditd
root       962     2  0 14:54 ?        00:00:00 [kauditd]
root      1549     1  0 17:49 ?        00:00:00 auditd
root      1836  2207  0 17:49 pts/0    00:00:00 grep auditd

Re: audit_pid with multiple userspace auditd processes

[Steve Grubb]

On Wednesday 07 January 2009 05:54:14 pm Eric Paris wrote:
> > Well, what if the first crashed and the kernel didn't know it yet? It
> > might be better to forcibly break the connection to the original auditd.
>
> I'm only talking about allowing userspace to "cleanly" unset it's belief
> there is an auditd out there if the message comes from that process.
> We'll still handle death by means of the usual netlink socket
> failures...
>
> If auditd number 2 is the auditd the kernel knows about why should
> auditd number 1 be allowed to "cleanly" say there is no auditd?

Ok, I see what you mean. We can either leave both running but disallow 
resetting the pid or forcibly disconnect the first in the kernel. Either way 
solves the problem. But doing the second might be cleaner for user space so 
two daemons aren't trying to write to the same file.

-Steve

Re: audit_pid with multiple userspace auditd processes

[Linda Knippers]

Steve Grubb wrote:
> On Wednesday 07 January 2009 05:54:14 pm Eric Paris wrote:
>>> Well, what if the first crashed and the kernel didn't know it yet? It
>>> might be better to forcibly break the connection to the original auditd.
>> I'm only talking about allowing userspace to "cleanly" unset it's belief
>> there is an auditd out there if the message comes from that process.
>> We'll still handle death by means of the usual netlink socket
>> failures...
>>
>> If auditd number 2 is the auditd the kernel knows about why should
>> auditd number 1 be allowed to "cleanly" say there is no auditd?
> 
> Ok, I see what you mean. We can either leave both running but disallow 
> resetting the pid or forcibly disconnect the first in the kernel. Either way 
> solves the problem. But doing the second might be cleaner for user space so 
> two daemons aren't trying to write to the same file.

The first makes more sense to me.  If an auditd is happily running,
starting a second one is an error.  Disconnecting a running auditd
seems problematic.  What happens to audit messages in flight?  Is
there a race where both auditds will be writing to the log?

-- ljk

> 
> -Steve
> 
> --
> Linux-audit mailing list
> Linux-audit@redhat.com
> https://www.redhat.com/mailman/listinfo/linux-audit

Re: audit_pid with multiple userspace auditd processes

[Steve Grubb]

On Wednesday 07 January 2009 06:11:37 pm Linda Knippers wrote:
> The first makes more sense to me.  If an auditd is happily running,
> starting a second one is an error.

Yes, but how can you detect that on an async protocol? The kernel would have 
to look and verify the original pid is alive, then look to see if there is a 
matching netlink socket for that pid. 

At some point in the past, the kernel only knew that an auditd was dead on 
attempting to use the socket. I don't know if its still the same way, but if 
it were, then you don't really know if the audit daemon is alive so you may 
as well trust the second one under the assumption that its a restarted daemon 
to replace the crashed one the kernel didn't know about yet.


> Disconnecting a running auditd seems problematic.  What happens to audit
> messages in flight?

It just won't get anything and will error out next time it tries to read 
events.

> Is there a race where both auditds will be writing to 
> the log? 

Yes, that is why the first needs to go away.

-Steve

Re: audit_pid with multiple userspace auditd processes

[Linda Knippers]

Steve Grubb wrote:
> On Wednesday 07 January 2009 06:11:37 pm Linda Knippers wrote:
>> The first makes more sense to me.  If an auditd is happily running,
>> starting a second one is an error.
> 
> Yes, but how can you detect that on an async protocol? The kernel would have 
> to look and verify the original pid is alive, then look to see if there is a 
> matching netlink socket for that pid. 

Eric said that if you start the auditd using the init script, the right
thing happened (the auditd didn't start?).  Couldn't auditd do the same
check itself, whatever that check is?
> 
> At some point in the past, the kernel only knew that an auditd was dead on 
> attempting to use the socket. I don't know if its still the same way, but if 
> it were, then you don't really know if the audit daemon is alive so you may 
> as well trust the second one under the assumption that its a restarted daemon 
> to replace the crashed one the kernel didn't know about yet.
> 
> 
>> Disconnecting a running auditd seems problematic.  What happens to audit
>> messages in flight?
> 
> It just won't get anything and will error out next time it tries to read 
> events.
> 
>> Is there a race where both auditds will be writing to 
>> the log? 
> 
> Yes, that is why the first needs to go away.

I think the second shouldn't start.

I was actually talking about the case where:
	auditd 1 reads record A
	auditd 2 starts and takes over
	auditd 2 reads record B
	auditd 2 writes record B
	auditd 1 writes record A

If auditd 2 can start while auditd 1 is still alive, that seems
bad.  You either get records out of order or, if you kill auditd 1,
you lose record A.

-- ljk