From mboxrd@z Thu Jan 1 00:00:00 1970 From: Andy Ruch Subject: Backlog exceeded when using audisp Date: Wed, 13 Aug 2014 14:37:52 -0700 Message-ID: <1407965872.73962.YahooMailNeo@web120705.mail.ne1.yahoo.com> Reply-To: Andy Ruch Mime-Version: 1.0 Content-Type: multipart/mixed; boundary="===============0375962976972210645==" Return-path: Received: from mx1.redhat.com (ext-mx11.extmail.prod.ext.phx2.redhat.com [10.5.110.16]) by int-mx09.intmail.prod.int.phx2.redhat.com (8.14.4/8.14.4) with ESMTP id s7DLbtgL008177 for ; Wed, 13 Aug 2014 17:37:55 -0400 Received: from nm10-vm5.bullet.mail.ne1.yahoo.com (nm10-vm5.bullet.mail.ne1.yahoo.com [98.138.91.232]) by mx1.redhat.com (8.14.4/8.14.4) with ESMTP id s7DLbrUD012208 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO) for ; Wed, 13 Aug 2014 17:37:54 -0400 List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: linux-audit-bounces@redhat.com Errors-To: linux-audit-bounces@redhat.com To: Audit ML List-Id: linux-audit@redhat.com --===============0375962976972210645== Content-Type: multipart/alternative; boundary="1929490892-695622143-1407965872=:73962" --1929490892-695622143-1407965872=:73962 Content-Type: text/plain; charset=us-ascii Hello, I'm trying to send the audit logs on a secure RHEL 6.5 system to rsyslog. Rsyslog will then send them to another system for centralized collection. I can't have audisp send them directly because the connectivity is unreliable and rsyslog provides on disk queues for reliable delivery. I've activated the syslogplugin of audisp to do the transfer. The problem is getting the logs transferred fast enough. The system is configured to panic upon error (-f 2), which it does frequently when I do something like update the SELinux RPM since watching /etc/selinux is required by the STIG. I have the audit buffer size configured to 8192 and the audisp queue set to 120. I'm surprised the 8192 buffer is being overwhelmed. When I look at aureport for just the time frame of the action, I get approximately 350 events. I know that each event may have multiple entries, but it is interesting that the capacity of a buffer over 20 times bigger is being exceeded. Can anyone in a similar situation share any insights? Is there a faster way to transfer the logs rather than the audispsyslogplugin? We use to have rsyslog monitor the audit.log file but ran into some issues when we started dealing with log file rollover. And it just seems cleaner to send the audit logs directly. Thanks, Andrew Ruch --1929490892-695622143-1407965872=:73962 Content-Type: text/html; charset=us-ascii
Hello,

I'm trying to send the audit logs on a secure RHEL 6.5 system to rsyslog. Rsyslog will then send them to another system for centralized collection. I can't have audisp send them directly because the connectivity is unreliable and rsyslog provides on disk queues for reliable delivery. I've activated the syslog plugin of audisp to do the transfer. The problem is getting the logs transferred fast enough. The system is configured to panic upon error (-f 2), which it does frequently when I do something like update the SELinux RPM since watching /etc/selinux is required by the STIG.

I have the audit buffer size configured to 8192 and the audisp queue set to 120. I'm surprised the 8192 buffer is being over whelmed. When I look at aureport for just the time frame of the action, I get approximately 350 events. I know that each event may have multiple entries, but it is interesting that the capacity of a buffer over 20 times bigger is being exceeded.

Can anyone in a similar situation share any insights? Is there a faster way to transfer the logs rather than the audisp syslog plugin? We use to have rsyslog monitor the audit.log file but ran into some issues when we started dealing with log file rollover. And it just seems cleaner to send the audit logs directly.

Thanks,
Andrew Ruch
--1929490892-695622143-1407965872=:73962-- --===============0375962976972210645== Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Disposition: inline --===============0375962976972210645==-- From mboxrd@z Thu Jan 1 00:00:00 1970 From: Steve Grubb Subject: Re: Backlog exceeded when using audisp Date: Thu, 14 Aug 2014 08:50:40 -0400 Message-ID: <1808827.qRzKKWoyqy@x2> References: <1407965872.73962.YahooMailNeo@web120705.mail.ne1.yahoo.com> Mime-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Return-path: In-Reply-To: <1407965872.73962.YahooMailNeo@web120705.mail.ne1.yahoo.com> List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: linux-audit-bounces@redhat.com Errors-To: linux-audit-bounces@redhat.com To: linux-audit@redhat.com, Andy Ruch List-Id: linux-audit@redhat.com On Wednesday, August 13, 2014 02:37:52 PM Andy Ruch wrote: > I'm trying to send the audit logs on a secure RHEL 6.5 system to rsyslog. > Rsyslog will then send them to another system for centralized collection. This is fine. > I can't have audisp send them directly because the connectivity is > unreliable and rsyslog provides on disk queues for reliable delivery. So does auditd. It doesn't lose events unless some limit was exceeded. > I've activated the syslogplugin of audisp to do the transfer. The problem is > getting the logs transferred fast enough. The system is configured to panic > upon error (-f 2), which it does frequently when I do something like update > the SELinux RPM since watching /etc/selinux is required by the STIG. A couple of thoughts here. Perhaps you want to have a policy where when you update selinux policy, you suspend auditing and then update and then resume. Short of that, you'll need to boost the priority and enlarge the queue sizes. The panic will only occur on an in-kernel buffer overflow. User space can't do that. > I have the audit buffer size configured to 8192 and the audisp queue set to > 120. I'm surprised the 8192 buffer is being overwhelmed. When I look at > aureport for just the time frame of the action, I get approximately 350 > events. I know that each event may have multiple entries, but it is > interesting that the capacity of a buffer over 20 times bigger is being > exceeded. Well, if the auditspd buffer is full and you have lossless buffering, the daemon waits until there is room in the queue to continue processing and then the kernel buffer backs up. You have to have settings in user space that allow auditd to keep the kernel queue as empty as possible. > > Can anyone in a similar situation share any insights? Is there a faster way > to transfer the logs rather than the audispsyslogplugin? We use to have > rsyslog monitor the audit.log file but ran into some issues when we started > dealing with log file rollover. And it just seems cleaner to send the audit > logs directly. You can also just load rules via auditctl. The kernel defaults to sending events to syslog if auditd is not running. -Steve From mboxrd@z Thu Jan 1 00:00:00 1970 From: Andy Ruch Subject: Re: Backlog exceeded when using audisp Date: Thu, 14 Aug 2014 13:12:19 -0700 Message-ID: <1408047139.57007.YahooMailNeo@web120702.mail.ne1.yahoo.com> Reply-To: Andy Ruch Mime-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable Return-path: Received: from mx1.redhat.com (ext-mx16.extmail.prod.ext.phx2.redhat.com [10.5.110.21]) by int-mx13.intmail.prod.int.phx2.redhat.com (8.14.4/8.14.4) with ESMTP id s7EKCU1R016241 for ; Thu, 14 Aug 2014 16:12:30 -0400 Received: from nm21-vm3.bullet.mail.ne1.yahoo.com (nm21-vm3.bullet.mail.ne1.yahoo.com [98.138.91.151]) by mx1.redhat.com (8.14.4/8.14.4) with ESMTP id s7EKCNpw021949 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO) for ; Thu, 14 Aug 2014 16:12:24 -0400 List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: linux-audit-bounces@redhat.com Errors-To: linux-audit-bounces@redhat.com To: Audit ML List-Id: linux-audit@redhat.com > On Wednesday, August 13, 2014 02:37:52 PM Andy Ruch wrote: > > I'm trying to send the audit logs on a secure RHEL 6.5 system to rsyslo= g. > > Rsyslog will then send them to another system for centralized collectio= n. = > > This is fine. = > > > I can't have audisp send them directly because the connectivity is > > unreliable and rsyslog provides on disk queues for reliable delivery. = > > So does auditd. It doesn't lose events unless some limit was exceeded. = > > > I've activated the syslogplugin of audisp to do the transfer. The probl= em is > > getting the logs transferred fast enough. The system is configured to p= anic > > upon error (-f 2), which it does frequently when I do something like up= date > > the SELinux RPM since watching /etc/selinux is required by the STIG. = > > A couple of thoughts here. Perhaps you want to have a policy where when y= ou = > update selinux policy, you suspend auditing and then update and then resu= me. = > Short of that, you'll need to boost the priority and enlarge the queue si= zes. = > The panic will only occur on an in-kernel buffer overflow. User space can= 't do = > that. = > > > I have the audit buffer size configured to 8192 and the audisp queue se= t to > > 120. I'm surprised the 8192 buffer is being overwhelmed. When I look at > > aureport for just the time frame of the action, I get approximately 350 > > events. I know that each event may have multiple entries, but it is > > interesting that the capacity of a buffer over 20 times bigger is being > > exceeded. = > > Well, if the auditspd buffer is full and you have lossless buffering, the= daemon = > waits until there is room in the queue to continue processing and then th= e = > kernel buffer backs up. You have to have settings in user space that allo= w = > auditd to keep the kernel queue as empty as possible. = > > > = > > Can anyone in a similar situation share any insights? Is there a faster= way > > to transfer the logs rather than the audispsyslogplugin? We use to have > > rsyslog monitor the audit.log file but ran into some issues when we sta= rted > > dealing with log file rollover. And it just seems cleaner to send the a= udit > > logs directly. = > > You can also just load rules via auditctl. The kernel defaults to sending = > events to syslog if auditd is not running. = > = > -Steve =A0 Upon further testing, I think there might be something else as the root cau= se. For my testing, I'm adding an selinux user (semanage user -a ...). This= will trigger a load_policy command for SELinux. When everything works fine= , auditd processes roughly 2000 events and audisp handles this with no prob= lems. However, sometimes when I run the command, auditd will receive over 1= 5000 events. As far as I can tell, the extra events are all SELinux error e= vents stating "selinux_audit_rule_match: stale rule". What would cause this= and why does it not happen every time? Is this an audit issue or an SELinu= x issue? From mboxrd@z Thu Jan 1 00:00:00 1970 From: Steve Grubb Subject: Re: Backlog exceeded when using audisp Date: Fri, 15 Aug 2014 14:01:22 -0400 Message-ID: <13673462.7keNk3W44N@x2> References: <1408047139.57007.YahooMailNeo@web120702.mail.ne1.yahoo.com> Mime-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Return-path: In-Reply-To: <1408047139.57007.YahooMailNeo@web120702.mail.ne1.yahoo.com> List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: linux-audit-bounces@redhat.com Errors-To: linux-audit-bounces@redhat.com To: linux-audit@redhat.com, Andy Ruch List-Id: linux-audit@redhat.com On Thursday, August 14, 2014 01:12:19 PM Andy Ruch wrote: > Upon further testing, I think there might be something else as the root > cause. For my testing, I'm adding an selinux user (semanage user -a ...). > This will trigger a load_policy command for SELinux. When everything works > fine, auditd processes roughly 2000 events and audisp handles this with no > problems. However, sometimes when I run the command, auditd will receive > over 15000 events. As far as I can tell, the extra events are all SELinux > error events stating "selinux_audit_rule_match: stale rule". What would > cause this Looking at kernel code, it would appear that you have rules that use selinux as part of the matching. When you load new policy, the sids (which is a number from the policy compiler) has likely changed and no longer matches what it thinks it does. The kernel only understands numbers. When you compile human readable selinux rules, it changes the text into numbers. If policy is changed, there is no guarantee that the number for say crond_t is the same as it used to be. So, the kernel detects that the audit rule mapping is using stale (old) sids. > and why does it not happen every time? Is this an audit issue or > an SELinux issue? A little of both. A work around is probably to reload the audit rules immediately so that the sids get resolved against the new policy. The better fix would be to file a support ticket and ask for that code to be improved by doing what current kernels do for that function. They now have a WARN_ONCE message that goes to syslog. That keeps the audit logs cleaner. -Steve