Re: Odd memory usage in auditd

[Steve Grubb <sgrubb@redhat.com>]

On Thursday, October 07, 2010 05:52:49 am Ross Kirk wrote:
> Has anybody got any advice for the following problem? As I'm seeing some
> very odd behaviour with the auditd daemon in RHEL5.2 where under heavy
> system load the auditd process doesn't free any resources until all memory
> is consumed and the kernel kills the process with an Out Of Memory error.

I seem to recall something about disk flushing causing auditd to look like its 
the culprit. Do you have barriers enabled on ext3? You might also try setting 
the flushing to something else like none and see if that does anything.


> The system I have is a heavily customised RHEL5.2 with some fairly
> stringent auditing rules specified, the config is attached. In addition to
> these rules there will be various SELinux AVCs being raised as well as
> events from my own software so the auditing system is kept quite busy, see
> the attached report.txt for the aureport summary .

I don't see anything terribly unusual. The audit rules didn't make it, but the 
backlog setting is the only thing I would be interested in seeing.



> I can reproduce this behaviour consistently by generating a heavy system
> load (CPU 100% usage) while also generating a significant number of audit
> events. After about 20 minutes the auditd process will have grown from 8Mb
> of memory to around 1Gb;
> 
>   PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
> 
> 3037 root      16  -3 2763m 921m   16 S  3.7 91.2   0:26.49 auditd
> 
> If the system is kept busy eventually auditd will consume all the memory
> available on the system and the process be killed by the kernel with an
> Out Of Memory error.

Try playing with the disk flushing and let us know how that works out. There 
are no known memory leaks in recent version of auditd. I try to keep malloc 
down to a minimum to prevent this and memory fragmentation to creep in.

-Steve