Auditd hangs hard

Auditd hangs hard

[Matthew Booth]

[-- Attachment #1.1: Type: text/plain, Size: 1091 bytes --]

I have a test server on which, if left to its own devices, the audit
daemon will lock up hard. The most obvious symptoms are:

* An attempt to read /proc/<auditd pid>/ will hang
* auditd cannot be killed with -9
* The logs are full of backlog exceeded messages.

The system in question is doing some fairly severe system call auditing.
The audit daemon is configured not to write to disk at all. Instead it
uses a custom dispatcher which directly wraps the audit messages as a
syslog message and sends it directly via UDP to a central host. The
change which prompted this behaviour seems to be the installation of LSF
analytics, which is quite exceptionally noisy from a syscall POV.

The system is RHEL 4 x86_64 running:
audit-1.0.15-3.EL4
kernel-smp-2.6.9-42.0.8.EL

It's basically RHEL 4 U4 with auditd from U5. Any ideas what might be
causing this, or how to debug?

Thanks,

Matt
-- 
Matthew Booth, RHCA, RHCSS
Red Hat, Global Professional Services

M:       +44 (0)7977 267231
GPG ID:  D33C3490
GPG FPR: 3733 612D 2D05 5458 8A8A 1600 3441 EA19 D33C 3490

[-- Attachment #1.2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

[-- Attachment #2: Type: text/plain, Size: 0 bytes --]

Re: Auditd hangs hard

[Steve Grubb]

On Friday 08 June 2007 10:32:54 Matthew Booth wrote:
> I have a test server on which, if left to its own devices, the audit
> daemon will lock up hard.

Does boosting the priority so auditd runs more often help? I think it defalts 
to 3, you can make it 10 for an experiment.

-Steve

Re: Auditd hangs hard

[Matthew Booth]

[-- Attachment #1.1: Type: text/plain, Size: 1099 bytes --]

On Sat, 2007-06-09 at 07:59 -0400, Steve Grubb wrote:
> On Friday 08 June 2007 10:32:54 Matthew Booth wrote:
> > I have a test server on which, if left to its own devices, the audit
> > daemon will lock up hard.
> 
> Does boosting the priority so auditd runs more often help? I think it defalts 
> to 3, you can make it 10 for an experiment.

Thanks, Steve. This put me on the right track. It turns out that not
only is LSF very noisy, but it also runs itself with niceness -20.
Renicing it to -5 and running auditd at -10 fixes the problem. I've
asked the customer to raise an issue against LSF as this strikes me as a
bug.

It does strike me that audit could cope with overload much better,
though. If it's configured to drop messages rather than kill the system,
it could probably disable auditing entirely when the kernel buffer is
full, and only re-enable it when there's enough space.

Matt
-- 
Matthew Booth, RHCA, RHCSS
Red Hat, Global Professional Services

M:       +44 (0)7977 267231
GPG ID:  D33C3490
GPG FPR: 3733 612D 2D05 5458 8A8A 1600 3441 EA19 D33C 3490

[-- Attachment #1.2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

[-- Attachment #2: Type: text/plain, Size: 0 bytes --]

Re: Auditd hangs hard

[Steve Grubb]

On Tuesday 12 June 2007 04:50:27 Matthew Booth wrote:
> > Does boosting the priority so auditd runs more often help? I think it
> > defalts to 3, you can make it 10 for an experiment.
>
> Thanks, Steve. This put me on the right track. It turns out that not
> only is LSF very noisy, but it also runs itself with niceness -20.
> Renicing it to -5 and running auditd at -10 fixes the problem.

Good.


> It does strike me that audit could cope with overload much better,
> though.

It depends on how you have the configuration set. If you set disp_qos to 
lossy, then it should have discarded packets sent to the dispatcher. The only 
thing that it would be waiting on at that point is disk writing which has 
several tunables, too. If the dispatcher was the limiting factor, you may 
have to make it multi-threaded with one thread assigned to drain the auditd 
interface and write it to a fifo where another thread writes to syslog. This 
would allow the audit system to make better use of its time slice.


> If it's configured to drop messages rather than kill the system, 
> it could probably disable auditing entirely when the kernel buffer is
> full, and only re-enable it when there's enough space.

How big was the kernel buffer when you had problems? (Its adjustable.)

-Steve

Re: Auditd hangs hard

[Matthew Booth]

[-- Attachment #1.1: Type: text/plain, Size: 1324 bytes --]

On Tue, 2007-06-12 at 12:45 -0400, Steve Grubb wrote:
> It depends on how you have the configuration set. If you set disp_qos to 
> lossy, then it should have discarded packets sent to the dispatcher. The only 
> thing that it would be waiting on at that point is disk writing which has 
> several tunables, too. If the dispatcher was the limiting factor, you may 
> have to make it multi-threaded with one thread assigned to drain the auditd 
> interface and write it to a fifo where another thread writes to syslog. This 
> would allow the audit system to make better use of its time slice.

dispatcher qos set to lossy. All writing to disk disabled. Limiting
factor appeared to have been auditd not being scheduled often enough, so
the performance factor appears to be the behaviour of the kernel when
it's buffers are full.

> > If it's configured to drop messages rather than kill the system, 
> > it could probably disable auditing entirely when the kernel buffer is
> > full, and only re-enable it when there's enough space.
> 
> How big was the kernel buffer when you had problems? (Its adjustable.)

32k

Matt
-- 
Matthew Booth, RHCA, RHCSS
Red Hat, Global Professional Services

M:       +44 (0)7977 267231
GPG ID:  D33C3490
GPG FPR: 3733 612D 2D05 5458 8A8A 1600 3441 EA19 D33C 3490

[-- Attachment #1.2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

[-- Attachment #2: Type: text/plain, Size: 0 bytes --]