Odd memory usage in auditd

Odd memory usage in auditd

[Ross Kirk]

[-- Attachment #1.1: Type: text/plain, Size: 2788 bytes --]

Hi,

 

Has anybody got any advice for the following problem? As I'm seeing some very odd behaviour with the auditd daemon in RHEL5.2 where under heavy system load the auditd process doesn't free any resources until all memory is consumed and the kernel kills the process with an Out Of Memory error.

 

The system I have is a heavily customised RHEL5.2 with some fairly stringent auditing rules specified, the config is attached. In addition to these rules there will be various SELinux AVCs being raised as well as events from my own software so the auditing system is kept quite busy, see the attached report.txt for the aureport summary .

 

I can reproduce this behaviour consistently by generating a heavy system load (CPU 100% usage) while also generating a significant number of audit events. After about 20 minutes the auditd process will have grown from 8Mb of memory to around 1Gb;

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND    

3037 root      16  -3 2763m 921m   16 S  3.7 91.2   0:26.49 auditd

If the system is kept busy eventually auditd will consume all the memory available on the system and the process be killed by the kernel with an Out Of Memory error.

 

On the face of it this would seem to be a memory leak within auditd however once auditd has started to consume resources if the system load is reduced and therefore the number and frequency of audit events is reduced then auditd will free this memory and return back to the 8Mb memory usage. So it would seem that while auditd is kept busy it keeps hold of any resources it is allocating. Unfortunately I have systems that run under the level of load required to cause this issue fairly frequently.

 

The system is RHEL 5.2 and audit is version audit-1.7.17-3.

 

Regards

Ross Kirk

Software Engineer

N E X O R

connect  transform  protect

Tel:  +44 (0) 115 952 0500

Fax: +44 (0) 115 952 0519

mailto:ross.kirk@nexor.com <mailto:ross.kirk@nexor.com> 

http://www.nexor.com <http://www.nexor.com> 

Nexor is recognised as an Investor in People and is accredited to ISO9001/TickIT and ISO/IEC27001:2005. Further details of Nexor's accreditations can be found on our website.

DISCLAIMER: Privileged or confidential information may be contained in this message or within any files transmitted with it. If you are not the intended recipient, kindly destroy the message and notify the sender by reply email. Opinions, conclusions and other information in this message that do not relate to the official business of Nexor are neither given nor endorsed by it.

Nexor Limited, Bell House, Nottingham Science and Technology Park, University Boulevard, Nottingham, NG7 2RL

A company registered in England, No: 05152465

 


[-- Attachment #1.2: Type: text/html, Size: 7198 bytes --]

[-- Attachment #2: audit.rules --]
[-- Type: application/octet-stream, Size: 2165 bytes --]

    ## Add keys to the audit rules below using the -k option to allow for more
    ## organized and quicker searches with the ausearch tool.  See auditctl(8)
    ## and ausearch(8) for more information.

    # Remove any existing rules
    -D

    # Enable auditing
    -e 1

    # Increase buffer size to handle the increased number of messages.
    -b 16384

    # Failure of auditd causes a kernel panic
    -f 2

    -w /bin/login -p x
    -w /bin/logout -p x

    # DAC permission changes
-a exit,always -S chmod -S chown -S fchmod -S fchown -S lchown -S chown32 -S fchown32 -S lchown32

    # unauthorized file access attempts
-a exit,always -S open -F success=0
-a exit,always -F success=0 -S mknod -S pipe -S mkdir -S creat -S truncate -S ftruncate -S truncate64 -S ftruncate64

    # privileged commands
    -w /usr/sbin/pwck
    -w /bin/chgrp
    -w /usr/bin/newgrp
    -w /usr/sbin/groupadd
    -w /usr/sbin/groupmod
    -w /usr/sbin/groupdel
    -w /usr/sbin/useradd
    -w /usr/sbin/userdel
    -w /usr/sbin/usermod
    -w /usr/bin/chage
    -w /usr/bin/setfacl
    -w /usr/bin/chacl
-a exit,always -S chroot -S mount -S umount2 -S adjtimex -S kill -S umount

    # deleting files
    -a exit,always -S unlink -S rmdir -S rename -S link -S symlink

    # system administration actions
    -w /var/log/audit/audit.log
    -w /var/log/audit/audit[1-4].log
    -w /var/log/messages
    -w /var/log/lastlog
    -w /var/log/faillog
    -w /etc/audit/auditd.conf -p wa
    -w /etc/audit/audit.rules -p wa
    -w /etc/selinux/config -p wa
    -w /etc/passwd -p wa
    -w /etc/shadow -p wa
    -w /etc/group  -p wa
    -w /etc/ld.so.conf -p wa
    -w /etc/ld.so.conf.d -p wa
    -w /etc/ssh/sshd_config
    -w /etc/pam.d
    -w /etc/login.defs
    -w /etc/rc.d/init.d
    -w /etc/inittab -p wa
    -w /var/run/utmp
    -w /var/run/wtmp
-a exit,always -S acct -S reboot -S sched_setparam -S sched_setscheduler -S setdomainname -S setrlimit -S settimeofday -S swapon -S stime

    # security personnel actions
    -a exit,always -S init_module -S delete_module -S setxattr -S lsetxattr -S fsetxattr -S removexattr -S lremovexattr -S fremovexattr
    -w /bin/su


[-- Attachment #3: auditd.conf --]
[-- Type: application/octet-stream, Size: 424 bytes --]

    log_file = /var/log/audit/audit.log
    log_format = RAW
    priority_boost = 3
    flush = INCREMENTAL
    freq = 20
    num_logs = 4
    #dispatcher = /sbin/audispd
    max_log_file = 256
    max_log_file_action = ROTATE
    space_left = 75
    space_left_action = SYSLOG
    action_mail_acct = root
    admin_space_left = 50
    admin_space_left_action = HALT
    disk_full_action = HALT
    disk_error_action = HALT

[-- Attachment #4: report.txt --]
[-- Type: text/plain, Size: 595 bytes --]

Number of changes in configuration: 301
Number of changes to accounts, groups, or roles: 0
Number of logins: 13
Number of failed logins: 14
Number of authentications: 24
Number of failed authentications: 2
Number of users: 2
Number of terminals: 15
Number of host names: 5
Number of executables: 120
Number of files: 348742
Number of AVC's: 23361
Number of MAC events: 1
Number of failed syscalls: 1101653
Number of anomaly events: 0
Number of responses to anomaly events: 0
Number of crypto events: 0
Number of keys: 0
Number of process IDs: 22880
Number of events: 1521282

[-- Attachment #5: Type: text/plain, Size: 0 bytes --]

Re: Odd memory usage in auditd

[Steve Grubb]

On Thursday, October 07, 2010 05:52:49 am Ross Kirk wrote:
> Has anybody got any advice for the following problem? As I'm seeing some
> very odd behaviour with the auditd daemon in RHEL5.2 where under heavy
> system load the auditd process doesn't free any resources until all memory
> is consumed and the kernel kills the process with an Out Of Memory error.

I seem to recall something about disk flushing causing auditd to look like its 
the culprit. Do you have barriers enabled on ext3? You might also try setting 
the flushing to something else like none and see if that does anything.


> The system I have is a heavily customised RHEL5.2 with some fairly
> stringent auditing rules specified, the config is attached. In addition to
> these rules there will be various SELinux AVCs being raised as well as
> events from my own software so the auditing system is kept quite busy, see
> the attached report.txt for the aureport summary .

I don't see anything terribly unusual. The audit rules didn't make it, but the 
backlog setting is the only thing I would be interested in seeing.



> I can reproduce this behaviour consistently by generating a heavy system
> load (CPU 100% usage) while also generating a significant number of audit
> events. After about 20 minutes the auditd process will have grown from 8Mb
> of memory to around 1Gb;
> 
>   PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
> 
> 3037 root      16  -3 2763m 921m   16 S  3.7 91.2   0:26.49 auditd
> 
> If the system is kept busy eventually auditd will consume all the memory
> available on the system and the process be killed by the kernel with an
> Out Of Memory error.

Try playing with the disk flushing and let us know how that works out. There 
are no known memory leaks in recent version of auditd. I try to keep malloc 
down to a minimum to prevent this and memory fragmentation to creep in.

-Steve

RE: Odd memory usage in auditd

[Ross Kirk]

Doh, originally just replied to Steve rather than to the list!

Ross

> -----Original Message-----
> From: linux-audit-bounces@redhat.com [mailto:linux-audit- 
> bounces@redhat.com] On Behalf Of Steve Grubb
> Sent: 07 October 2010 20:51
> To: linux-audit@redhat.com
> Subject: Re: Odd memory usage in auditd
> 
> On Thursday, October 07, 2010 05:52:49 am Ross Kirk wrote:
> > Has anybody got any advice for the following problem? As I'm seeing
> some
> > very odd behaviour with the auditd daemon in RHEL5.2 where under
> heavy
> > system load the auditd process doesn't free any resources until all
> memory
> > is consumed and the kernel kills the process with an Out Of Memory
> error.
> 
> I seem to recall something about disk flushing causing auditd to look 
> like its the culprit. Do you have barriers enabled on ext3? You might 
> also try setting the flushing to something else like none and see if 
> that does anything.

Currently barriers are not enabled as it wasn't something I was aware of. However it does sound like something I may well want to be enabled so I will give the various flush settings a try and see if the barrier option affects anything.
So if auditd is not the culprit what do you suspect the problem is?

> > The system I have is a heavily customised RHEL5.2 with some fairly 
> > stringent auditing rules specified, the config is attached. In
> addition to
> > these rules there will be various SELinux AVCs being raised as well
> as
> > events from my own software so the auditing system is kept quite
> busy, see
> > the attached report.txt for the aureport summary .
> 
> I don't see anything terribly unusual. The audit rules didn't make it, 
> but the backlog setting is the only thing I would be interested in 
> seeing.

-b 16384

> > I can reproduce this behaviour consistently by generating a heavy
> system
> > load (CPU 100% usage) while also generating a significant number of
> audit
> > events. After about 20 minutes the auditd process will have grown
> from 8Mb
> > of memory to around 1Gb;
> >
> >   PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
> >
> > 3037 root      16  -3 2763m 921m   16 S  3.7 91.2   0:26.49 auditd
> >
> > If the system is kept busy eventually auditd will consume all the
> memory
> > available on the system and the process be killed by the kernel with
> an
> > Out Of Memory error.
> 
> Try playing with the disk flushing and let us know how that works out.
> There	
> are no known memory leaks in recent version of auditd. I try to keep 
> malloc down to a minimum to prevent this and memory fragmentation to 
> creep in.

With;
*  flush = none Completely different behaviour from previous, memory usage never changed for my entire test run
*  flush = data No increase in memory usage at all
*  flush = sync No increase in memory usage at all

Changing the ext3 barrier setting didn't make any changes to the above results nor to the behaviour I saw when "incremental" was set.

Well as the other settings are better behaved I can change to using "data" without any problems I believe. Arguably this is probably a better choice for me anyway!

Thanks for your help!

Ross

Re: Odd memory usage in auditd

[Steve Grubb]

On Monday, October 11, 2010 11:50:53 am Ross Kirk wrote:
> > I seem to recall something about disk flushing causing auditd to look
> > like its the culprit. Do you have barriers enabled on ext3? You might also
> > try setting the flushing to something else like none and see if that does
> > anything.
> 
> Currently barriers are not enabled as it wasn't something I was aware of.
> However it does sound like something I may well want to be enabled so I
> will give the various flush settings a try and see if the barrier option
> affects anything. 

Well, I was thinking that if you had it enabled that perhaps things were 
getting backed up.


> So if auditd is not the culprit what do you suspect the
> problem is?

Yes. I think under some situations because the flushing is delayed, the kernel 
memory accounting associates the buffers not yet flushed to disk with the 
process that originates the request. I don't think that is fair, but seems to 
happen.

 
> > Try playing with the disk flushing and let us know how that works out.
> > There are no known memory leaks in recent version of auditd. I try to keep
> > malloc down to a minimum to prevent this and memory fragmentation to creep
> > in.
> 
> With;
> *  flush = none Completely different behaviour from previous, memory usage
> never changed for my entire test run 
> *  flush = data No increase in memory usage at all
> *  flush = sync No increase in memory usage at all
> 
> Changing the ext3 barrier setting didn't make any changes to the above
> results nor to the behaviour I saw when "incremental" was set.
> 
> Well as the other settings are better behaved I can change to using "data"
> without any problems I believe. Arguably this is probably a better choice
> for me anyway!
> 
> Thanks for your help!

Sure.

-Steve