Auditd cause high CPU and high Load

Auditd cause high CPU and high Load

[Minh Tien Nguyen]

[-- Attachment #1.1: Type: text/plain, Size: 599 bytes --]

Dear Audit team.

My name is Nguyen Minh Tien. I came from Singapore. I am working as a
developer for Garena LTD. Last week, I met a problem with Audit on our
product servers. The Auditd process had caused of some pick time on our
server. In that times, system CPU cost a lot, around 100%. And the Load
average is over 30. We have tried to find the root cause and have failed.
Could you help us for that case?

The servers, which meet the performance issue, use Redhat 6.8 and their
kernel is 2.6.32.

-- 
Best regards
_________________________
Nguyen Minh Tien - SA Team
email: nguyenmt@garena.com

[-- Attachment #1.2: Type: text/html, Size: 832 bytes --]

[-- Attachment #2: Type: text/plain, Size: 0 bytes --]

Re: Auditd cause high CPU and high Load

[Steve Grubb]

Hello,

On Thursday, December 1, 2016 11:59:47 AM EST Minh Tien Nguyen wrote:
> My name is Nguyen Minh Tien. I came from Singapore. I am working as a
> developer for Garena LTD. Last week, I met a problem with Audit on our
> product servers. The Auditd process had caused of some pick time on our
> server. In that times, system CPU cost a lot, around 100%. And the Load
> average is over 30. We have tried to find the root cause and have failed.
> Could you help us for that case?
> 
> The servers, which meet the performance issue, use Redhat 6.8 and their
> kernel is 2.6.32.

You might want to check the flush setting for /etc/audit/auditd.conf. I would 
recommend using incremental and set the freq to something like 200 or 500. 
Using sync or data will kill performance, but the event is written to disk 
before processing the next event.

-Steve

Re: Auditd cause high CPU and high Load

[Minh Tien Nguyen]

[-- Attachment #1.1: Type: text/plain, Size: 1210 bytes --]

He Steve Grubb.

Nice to heard from you. Thank you so much for your help. We will try to set
that config and see the result.

Thanks & best regards.

On Thu, Dec 1, 2016 at 9:57 PM, Steve Grubb <sgrubb@redhat.com> wrote:

> Hello,
>
> On Thursday, December 1, 2016 11:59:47 AM EST Minh Tien Nguyen wrote:
> > My name is Nguyen Minh Tien. I came from Singapore. I am working as a
> > developer for Garena LTD. Last week, I met a problem with Audit on our
> > product servers. The Auditd process had caused of some pick time on our
> > server. In that times, system CPU cost a lot, around 100%. And the Load
> > average is over 30. We have tried to find the root cause and have failed.
> > Could you help us for that case?
> >
> > The servers, which meet the performance issue, use Redhat 6.8 and their
> > kernel is 2.6.32.
>
> You might want to check the flush setting for /etc/audit/auditd.conf. I
> would
> recommend using incremental and set the freq to something like 200 or 500.
> Using sync or data will kill performance, but the event is written to disk
> before processing the next event.
>
> -Steve
>



-- 
Best regards
_________________________
Nguyen Minh Tien - SA Team
email: nguyenmt@garena.com

[-- Attachment #1.2: Type: text/html, Size: 1926 bytes --]

[-- Attachment #2: Type: text/plain, Size: 0 bytes --]

Re: Auditd cause high CPU and high Load

[Edward Bailey]

[-- Attachment #1.1: Type: text/plain, Size: 1210 bytes --]

We ran into exactly the same issue with the update to 6.8. Using flush and
setting freq to 300 fixed the issue we were experiencing.

Ed

On Thu, Dec 1, 2016 at 9:00 AM Steve Grubb <sgrubb@redhat.com> wrote:

> Hello,
>
> On Thursday, December 1, 2016 11:59:47 AM EST Minh Tien Nguyen wrote:
> > My name is Nguyen Minh Tien. I came from Singapore. I am working as a
> > developer for Garena LTD. Last week, I met a problem with Audit on our
> > product servers. The Auditd process had caused of some pick time on our
> > server. In that times, system CPU cost a lot, around 100%. And the Load
> > average is over 30. We have tried to find the root cause and have failed.
> > Could you help us for that case?
> >
> > The servers, which meet the performance issue, use Redhat 6.8 and their
> > kernel is 2.6.32.
>
> You might want to check the flush setting for /etc/audit/auditd.conf. I
> would
> recommend using incremental and set the freq to something like 200 or 500.
> Using sync or data will kill performance, but the event is written to disk
> before processing the next event.
>
> -Steve
>
> --
> Linux-audit mailing list
> Linux-audit@redhat.com
> https://www.redhat.com/mailman/listinfo/linux-audit
>

[-- Attachment #1.2: Type: text/html, Size: 2212 bytes --]

[-- Attachment #2: Type: text/plain, Size: 0 bytes --]