Re: Linux audit performance impact

[Richard Guy Briggs <rgb@redhat.com>]

On 15/02/11, Viswanath, Logeswari P (MCOU OSTL) wrote:
> Another question, why was it decided to have multiple records per audit event?

I seem to recall it was to be able to filter unneeded information to
speed up processing.  It does generate more, but some types of searches
can benefit from avoiding to have to parse records in which it has no
interest.

> For eg:
> 
> type=SYSCALL msg=audit(1420988184.991:65696718): arch=c000003e syscall=2 success=yes exit=3 a0=e9f400 a1=0 a2=0 a3=5 items=1 ppid=2934 pid=2956 auid=0 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=pts3 ses=947 comm="vi" exe="/bin/vi" key=(null)
> type=CWD msg=audit(1420988184.991:65696718):  cwd="/root/ids/bkp"
> type=PATH msg=audit(1420988184.991:65696718): item=0 name="../loader.c" inode=1862956 dev=08:02 mode=0100777 ouid=0 ogid=0 rdev=00:00
> 
> Also, it would be great if one can help me answering my questions in the below mail?
> 
> -----Original Message-----
> From: Viswanath, Logeswari P (MCOU OSTL) 
> Sent: Friday, February 06, 2015 5:23 PM
> To: 'Richard Guy Briggs'
> Cc: 'Satish Chandra Kilaru'; 'Steve Grubb'; 'linux-audit@redhat.com'
> Subject: RE: Linux audit performance impact
> 
> One more question, I have enabled all system calls for auditing and auditd is not running. 
> Will printk result in write system call which in turn be audited?
> If yes, is there any way to ignore auditing for a specific processes
> such as syslogd to avoid auditing these extra write system calls?

Pre-pend a rule to exclude the activity of syslog by PID...

> -----Original Message-----
> From: Viswanath, Logeswari P (MCOU OSTL)
> Sent: Friday, February 06, 2015 12:17 PM
> To: 'Richard Guy Briggs'
> Cc: Satish Chandra Kilaru; Steve Grubb; linux-audit@redhat.com
> Subject: RE: Linux audit performance impact
> 
> Hi all,
> 
> Please find the below the details of the performance test we ran.
> It would be great if we get help to identify the reason behind the degradation and the ways of improving it. 
> 
> Kernel Version:
> root > uname -r
> 3.13.0-36-generic
> 
> OS Version:
> Ubuntu 14.04.1
> 
> No. of CPUs: 
> root > nproc
> 24
> 
> Audit Status:
> root > auditctl -s
> AUDIT_STATUS: enabled=1 flag=1 pid=0 rate_limit=0 backlog_limit=320 lost=57190353 backlog=0
> 
> Rules Configured:
> root > auditctl -l
> LIST_RULES: exit,always arch=3221225534 (0xc000003e) syscall=all
> 
> Attached is the program used to load the system.
> 
> Results:
> 
> Without enabling audit	12.29
> With auditing enabled and no rules configured 12.31
> With auditing enabled, 1 rule configured but auditd not running - kauditd logs audit records to syslog via printk	41.02		
> 
> The degradation is around 200%
> 
> Regards,
> Logeswari.
> 
> -----Original Message-----
> From: Richard Guy Briggs [mailto:rgb@redhat.com]
> Sent: Wednesday, February 04, 2015 9:46 PM
> To: Viswanath, Logeswari P (MCOU OSTL)
> Cc: Satish Chandra Kilaru; Steve Grubb; linux-audit@redhat.com
> Subject: Re: Linux audit performance impact
> 
> On 15/02/04, Viswanath, Logeswari P (MCOU OSTL) wrote:
> > The intent is to calculate the performance impact by the auditing 
> > components such as
> > 
> > 1) impact because of kauditd without auditd - but kauditd writes to syslog, so we are unable to determine the impact just because of kauditd - It is fine even if the audit record is dropped by kauditd. Is there any way to do this?
> 
> Not yet.  That is a mode that has not been useful to anyone yet.  You are welcome to hack a custom kernel to disable klog for doing testing instrumentation.
> 
> > 2) impact because of running auditd - log format NOLOG
> > 3) impact because of running audispd - small plugin is written which will just read the audit records and doesn't processes it.
> > 
> > -----Original Message-----
> > From: Richard Guy Briggs [mailto:rgb@redhat.com]
> > Sent: Tuesday, February 03, 2015 10:33 PM
> > To: Satish Chandra Kilaru
> > Cc: Viswanath, Logeswari P (MCOU OSTL); Steve Grubb; 
> > linux-audit@redhat.com
> > Subject: Re: Linux audit performance impact
> > 
> > On 15/02/03, Satish Chandra Kilaru wrote:
> > > Thanks for The info. But my question was rhetorical... I meant to 
> > > say that it would not be much... She is trying to bombard the system 
> > > with open calls ... So lots and lots of events will be generated and 
> > > kernel has to write down the events some where or discard them...
> > 
> > Exactly.  It is of little practical use.  You have to do I/O at some point, either to the same disk or another, or to a network interface or serial port, otherwise, just chuck it out.  You could do a performance measurement on a short burst, then drain the queue, but what will that actually tell us?
> > 
> > > On Tuesday, February 3, 2015, Richard Guy Briggs <rgb@redhat.com> wrote:
> > > 
> > > > On 15/02/03, Satish Chandra Kilaru wrote:
> > > > > How many events can kernel accumulate without I/o ?
> > > >
> > > > The kernel default is 64 *buffers*, but I think Fedora and RHEL 
> > > > set it to 320.  It is now possible to set it to "0" which means 
> > > > limited only by system resources.  See "man auditctl", "-b"
> > > > option.  An event can be made up of several buffers.
> > > >
> > > > Of course, how long a system lasts before the queue blows up 
> > > > depends on your rule set...
> > > >
> > > > However, at the moment, it will still write out to klog if auditd 
> > > > isn't running.
> > > >
> > > > > On Tuesday, February 3, 2015, Viswanath, Logeswari P (MCOU OSTL) 
> > > > > < logeswari.pv@hp.com <javascript:;>> wrote:
> > > > >
> > > > > > I don't want to disable auditing (i.e. disable audit record
> > > > collection),
> > > > > > but just do not want the records to delivered to user space 
> > > > > > since I
> > > > want to
> > > > > > remove the I/O overhead while running the performance test.
> > > > > > Is there any option for this?
> > > > > >
> > > > > > -----Original Message-----
> > > > > > From: Richard Guy Briggs [mailto:rgb@redhat.com <javascript:;>
> > > > <javascript:;>]
> > > > > > Sent: Thursday, January 29, 2015 10:23 PM
> > > > > > To: Viswanath, Logeswari P (MCOU OSTL)
> > > > > > Cc: Satish Chandra Kilaru; Steve Grubb; linux-audit@redhat.com
> > > > <javascript:;>
> > > > > > <javascript:;>
> > > > > > Subject: Re: Linux audit performance impact
> > > > > >
> > > > > > On 15/01/29, Viswanath, Logeswari P (MCOU OSTL) wrote:
> > > > > > > Please read my question as “Is there any option to configure 
> > > > > > > kaudit not to log audit records to syslog? when auditd not running.”
> > > > > >
> > > > > > Yeah, remove audit=1 from the kernel command line, or set
> > > > > > audit=0 in
> > > > its
> > > > > > place.  This will stop all but AVCs and if auditd has ever run 
> > > > > > since
> > > > boot.
> > > > > > If audit=0 is on the kernel boot line, it will be impossible 
> > > > > > to run
> > > > auditd.
> > > > > >
> > > > > > There is a feature request that is likely coming soon that 
> > > > > > could be
> > > > > > useful:
> > > > > >
> > > > > > https://bugzilla.redhat.com/show_bug.cgi?id=1160046
> > > > > > "If no audit daemon is running, but an audit multicast 
> > > > > > subscriber is around, then the kernel shouldn't forward audit data to kmsg"
> > > > > >
> > > > > > > From: Viswanath, Logeswari P (MCOU OSTL)
> > > > > > > Sent: Thursday, January 29, 2015 11:49 AM
> > > > > > > To: 'Satish Chandra Kilaru'; Steve Grubb
> > > > > > > Cc: linux-audit@redhat.com <javascript:;> <javascript:;>
> > > > > > > Subject: RE: Linux audit performance impact
> > > > > > >
> > > > > > > Is there any option to configure kaudit not to log audit 
> > > > > > > records to
> > > > > > syslog when auditd is running?
> > > > > > > This way we can assess the impact of enabling audit without 
> > > > > > > involving
> > > > > > disk I/o overhead.
> > > > > > >
> > > > > > > From: Satish Chandra Kilaru [mailto:iam.kilaru@gmail.com
> > > > <javascript:;> <javascript:;>]
> > > > > > > Sent: Thursday, January 29, 2015 9:12 AM
> > > > > > > To: Steve Grubb
> > > > > > > Cc: linux-audit@redhat.com <javascript:;> <javascript:;><mailto:
> > > > linux-audit@redhat.com <javascript:;>
> > > > > > <javascript:;>>; Viswanath,
> > > > > > > Logeswari P (MCOU OSTL)
> > > > > > > Subject: Re: Linux audit performance impact
> > > > > > >
> > > > > > > I agree with you... but writing to disk can trigger further 
> > > > > > > events
> > > > > > leading spiralling of events...
> > > > > > > I brought down my server few times with stupid rules...
> > > > > > >
> > > > > > > On Wed, Jan 28, 2015 at 10:39 PM, Steve Grubb 
> > > > > > > <sgrubb@redhat.com
> > > > <javascript:;>
> > > > > > <javascript:;><mailto:sgrubb@redhat.com <javascript:;>
> > > > <javascript:;>>> wrote:
> > > > > > > On Wednesday, January 28, 2015 10:18:47 AM Satish Chandra 
> > > > > > > Kilaru
> > > > wrote:
> > > > > > > > Write your own program to receive audit events directly 
> > > > > > > > without using auditd...
> > > > > > > > That should be faster ....
> > > > > > > > Auditd will log the events to disk causing more I/o than u need...
> > > > > > >
> > > > > > > But even that is configurable in many ways. You can decide 
> > > > > > > if you
> > > > want
> > > > > > > logging to disk or not and what kind of assurance that it 
> > > > > > > made it to disk and the priority of that audit daemon. Then 
> > > > > > > you also have all
> > > > the
> > > > > > > normal tuning knobs for disk throughput that you would use 
> > > > > > > for any disk performance critical system.
> > > > > > >
> > > > > > > -Steve
> > > > > > >
> > > > > > > > On Wednesday, January 28, 2015, Viswanath, Logeswari P 
> > > > > > > > (MCOU
> > > > > > > > OSTL)
> > > > <
> > > > > > > >
> > > > > > > > logeswari.pv@hp.com <javascript:;> <javascript:;><mailto:
> > > > logeswari.pv@hp.com <javascript:;>
> > > > > > <javascript:;>>> wrote:
> > > > > > > > >  Hi Steve,
> > > > > > > > >
> > > > > > > > > I am Logeswari working for HP.
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > We want to know audit performance impact on RHEL and 
> > > > > > > > > Suse linux
> > > > to
> > > > > > > > > help us evaluate linux audit as data source for our host 
> > > > > > > > > based
> > > > IDS.
> > > > > > > > >
> > > > > > > > > When we ran our own performance test with a test audispd 
> > > > > > > > > plugin, we found if a system can perform 200000 
> > > > > > > > > open/close system calls per second without auditing, 
> > > > > > > > > system can perform only 3000 open/close system calls 
> > > > > > > > > auditing is enabled for open/close system call which is 
> > > > > > > > > a HUGE impact on the system performance. It would
> > > > be
> > > > > > > > > great if anyone can help us answering the following questions.
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > 1)      Is this performance impact expected? If yes, what is the
> > > > > > reason
> > > > > > > > > behind it and can we fix it?
> > > > > > > > >
> > > > > > > > > 2)      Have anyone done any benchmarking for performance
> > > > impact? If
> > > > > > yes,
> > > > > > > > > can you please share the numbers and also the 
> > > > > > > > > steps/programs used the run the same.
> > > > > > > > >
> > > > > > > > > 3)      Help us validating the performance test we have done in
> > > > our
> > > > > > test
> > > > > > > > > setup using the steps mentioned along with the results attached.
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > Attached test program (loader.c) to invoke open and 
> > > > > > > > > close system
> > > > > > calls.
> > > > > > > > >
> > > > > > > > > Attached idskerndsp is the audispd plugin program.
> > > > > > > > >
> > > > > > > > > We used time command to determine how much time the 
> > > > > > > > > system took
> > > > to
> > > > > > > > > complete 50000 open/close system calls without (results 
> > > > > > > > > attached
> > > > > > > > > Without-auditing) and with auditing enabled on the 
> > > > > > > > > system (With-auditing-NOLOG-audispd-plugin and
> > > > > > > > > With-auditing-RAW)
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > System details:
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > 1 CPU machine
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > *OS Version*
> > > > > > > > >
> > > > > > > > > RHEL 6.5
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > *Kernel Version*
> > > > > > > > >
> > > > > > > > > uname –r
> > > > > > > > >
> > > > > > > > > 2.6.32-431.el6.x86_64
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > Note: auditd was occupying 35% of CPU and was sleeping 
> > > > > > > > > for most
> > > > of
> > > > > > > > > the time whereas kauditd was occupying 20% of the CPU.
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > Thanks & Regards,
> > > > > > > > >
> > > > > > > > > Logeswari.
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > --
> > > > > > > Please Donate to www.wikipedia.org<http://www.wikipedia.org>
> > > > > >
> > > > > > > --
> > > > > > > Linux-audit mailing list
> > > > > > > Linux-audit@redhat.com <javascript:;> <javascript:;> 
> > > > > > > https://www.redhat.com/mailman/listinfo/linux-audit
> > > > > >
> > > > > >
> > > > > > - RGB
> > > > > >
> > > > > > --
> > > > > > Richard Guy Briggs <rbriggs@redhat.com <javascript:;> 
> > > > > > <javascript:;>> Senior Software Engineer, Kernel Security, 
> > > > > > AMER ENG Base Operating Systems, Red Hat Remote, Ottawa, 
> > > > > > Canada
> > > > > > Voice: +1.647.777.2635, Internal: (81) 32635, Alt: 
> > > > > > +1.613.693.0684x3545
> > > > > >
> > > > >
> > > > >
> > > > > --
> > > > > Please Donate to www.wikipedia.org
> > > >
> > > > - RGB
> > > >
> > > > --
> > > > Richard Guy Briggs <rbriggs@redhat.com <javascript:;>> Senior 
> > > > Software Engineer, Kernel Security, AMER ENG Base Operating 
> > > > Systems, Red Hat Remote, Ottawa, Canada
> > > > Voice: +1.647.777.2635, Internal: (81) 32635, Alt: 
> > > > +1.613.693.0684x3545
> > > >
> > > 
> > > 
> > > --
> > > Please Donate to www.wikipedia.org
> > 
> > - RGB
> > 
> > --
> > Richard Guy Briggs <rbriggs@redhat.com> Senior Software Engineer, 
> > Kernel Security, AMER ENG Base Operating Systems, Red Hat Remote, 
> > Ottawa, Canada
> > Voice: +1.647.777.2635, Internal: (81) 32635, Alt: 
> > +1.613.693.0684x3545
> 
> - RGB
> 
> --
> Richard Guy Briggs <rbriggs@redhat.com>
> Senior Software Engineer, Kernel Security, AMER ENG Base Operating Systems, Red Hat Remote, Ottawa, Canada
> Voice: +1.647.777.2635, Internal: (81) 32635, Alt: +1.613.693.0684x3545

- RGB

--
Richard Guy Briggs <rbriggs@redhat.com>
Senior Software Engineer, Kernel Security, AMER ENG Base Operating Systems, Red Hat
Remote, Ottawa, Canada
Voice: +1.647.777.2635, Internal: (81) 32635, Alt: +1.613.693.0684x3545

--
Linux-audit mailing list
Linux-audit@redhat.com
https://www.redhat.com/mailman/listinfo/linux-audit