Re: Handling -ENOBUFS

[Steve Grubb <sgrubb@redhat.com>]

On Wednesday 05 November 2008 11:30:16 Lucas C. Villa Real wrote:
> I'm facing a situation where -ENOBUFS is returned from both
> audit_send() and audit_get_reply(). The system is under high stress,
> with 250k files being created and having creat() and chmod() syscalls
> audited.

Is this what you really wanted to audit? :)


> Looking the code at lib/netlink.c, I saw that audit_send() doesn't
> handle -ENOBUFS. Would it be possible to replace the condition from
> "while (retval < 0 && errno == EINTR)" to "while (retval < 0 && (errno
> == EINTR || errno == ENOBUFS))" to fix the problem when sending
> packets from userspace to kernel?

Have you tried that? Does it fix the problem or just hang the utility?


> My understanding for the problem in audit_get_reply() is that the I/O
> buffers are all full and auditd was just not scheduled at the expected
> rate, causing these buffers to overflow. Does that make sense?

If you go over the backlog limit, you get a syslog message about that unless 
you have it set to ignore. My guess would be that you have a general network 
memory pool depletion and is not related to audit specifically.


> If it does, do you have a suggestion about the best way to approach this
> problem, besides changing auditd's priority?

Increase the backlog and increase auditd's priority. I have not played with 
running auditd with a different scheduler policy than whatever the default 
is. But you may want to see if one of the other scheduler polices treat audit 
better. or maybe you want to tune  /proc/sys/kernel/sched_granularity_ns.


> One interesting thing which I noticed is that 'auditctl -s' doesn't
> report that messages were lost,

They weren't lost by the audit system so it doesn't know they didn't arrive.


> This is happening with an old kernel, 2.6.16.46 + a bunch of patches,
> and audit 1.7.4. I cannot completely upgrade it to a new release, but
> I can certainly backport audit specific bits if you remember having
> fixed something similar since then.

Well, that proc tunable is only available for the CFS scheduler. Not sure what 
you have for older kernels.

-Steve