From mboxrd@z Thu Jan 1 00:00:00 1970 From: Steve Grubb Subject: Re: Odd memory usage in auditd Date: Thu, 7 Oct 2010 15:51:23 -0400 Message-ID: <201010071551.23311.sgrubb@redhat.com> References: Mime-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Return-path: Received: from x2.localnet (vpn-9-117.rdu.redhat.com [10.11.9.117]) by int-mx01.intmail.prod.int.phx2.redhat.com (8.13.8/8.13.8) with ESMTP id o97JpViu008854 for ; Thu, 7 Oct 2010 15:51:31 -0400 In-Reply-To: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: linux-audit-bounces@redhat.com Errors-To: linux-audit-bounces@redhat.com To: linux-audit@redhat.com List-Id: linux-audit@redhat.com On Thursday, October 07, 2010 05:52:49 am Ross Kirk wrote: > Has anybody got any advice for the following problem? As I'm seeing some > very odd behaviour with the auditd daemon in RHEL5.2 where under heavy > system load the auditd process doesn't free any resources until all memory > is consumed and the kernel kills the process with an Out Of Memory error. I seem to recall something about disk flushing causing auditd to look like its the culprit. Do you have barriers enabled on ext3? You might also try setting the flushing to something else like none and see if that does anything. > The system I have is a heavily customised RHEL5.2 with some fairly > stringent auditing rules specified, the config is attached. In addition to > these rules there will be various SELinux AVCs being raised as well as > events from my own software so the auditing system is kept quite busy, see > the attached report.txt for the aureport summary . I don't see anything terribly unusual. The audit rules didn't make it, but the backlog setting is the only thing I would be interested in seeing. > I can reproduce this behaviour consistently by generating a heavy system > load (CPU 100% usage) while also generating a significant number of audit > events. After about 20 minutes the auditd process will have grown from 8Mb > of memory to around 1Gb; > > PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND > > 3037 root 16 -3 2763m 921m 16 S 3.7 91.2 0:26.49 auditd > > If the system is kept busy eventually auditd will consume all the memory > available on the system and the process be killed by the kernel with an > Out Of Memory error. Try playing with the disk flushing and let us know how that works out. There are no known memory leaks in recent version of auditd. I try to keep malloc down to a minimum to prevent this and memory fragmentation to creep in. -Steve