From mboxrd@z Thu Jan 1 00:00:00 1970 From: Steve Grubb Subject: Re: kauditd hold queue overflow in 4.11 Date: Sat, 09 Sep 2017 20:13:28 -0400 Message-ID: <7894043.DVPTJZrtqs@x2> References: <93a06f2e-b42d-f0ab-e5b9-9ae0f6691f20@debian.org> <1639759.peJ2nIZNxP@x2> Mime-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable Return-path: In-Reply-To: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: linux-audit-bounces@redhat.com Errors-To: linux-audit-bounces@redhat.com To: Laurent Bigonville Cc: linux-audit@redhat.com List-Id: linux-audit@redhat.com On Saturday, September 9, 2017 4:41:19 PM EDT Laurent Bigonville wrote: > Le 09/09/17 =E0 16:22, Steve Grubb a =E9crit : > > On Saturday, September 9, 2017 6:02:02 AM EDT Laurent Bigonville wrote: > >> Le 11/07/17 =E0 00:23, Paul Moore a =E9crit : > >>> On Mon, Jul 10, 2017 at 4:01 PM, Laurent Bigonville > > wrote: > >>>> Le 10/07/17 =E0 18:00, Paul Moore a =E9crit : > >>>>> On Mon, Jul 10, 2017 at 10:59 AM, Laurent Bigonville > >>>>> wrote: > >>>>>> Hi, > >>>>>> = > >>>>>> With 4.11.6 (that has been uploaded in debian unstable) I get a lot > >>>>>> of messages in dmesg like > >>>>>> = > >>>>>> [100052.120468] audit: audit_lost=3D66041 audit_rate_limit=3D0 > >>>>>> audit_backlog_limit=3D8192 > >>>>>> [100052.120470] audit: kauditd hold queue overflow > >>>>>> = > >>>>>> And it also seems that the messages are not stored in auditd logs > >>>>>> anymore. > >>>>>> = > >>>>>> https://git.kernel.org/linus/264d509637d95f9404e52ced5003ad352e0f6= a26 > >>>>>> seems to be included in this release > >>>>>> = > >>>>>> Any idea? > >>>>> = > >>>>> I'm going to assume that your backlog limit is set to a sane value = for > >>>>> your system's configuration, so that leaves me with two commits that > >>>>> may be of interest: > >>>>> = > >>>>> * 1df30f8264b8 ("audit: fix the RCU locking for the auditd_connecti= on > >>>>> structure") > >>>>> = > >>>>> This was a manual backport of a v4.12 patch to v4.11, looking now, I > >>>>> see it should be in +v4.11.5 so that probably isn't your problem ... > >>>>> = > >>>>> * c81be52a3ac0 ("audit: fix a race condition with the auditd tracki= ng > >>>>> code") > >>>>> = > >>>>> This patch is relatively new and was just sent up to Linus during t= he > >>>>> next merge window; it's a race condition fix so reproducing it can = be > >>>>> tricky, although it may be easily reproducible on your system at the > >>>>> moment (luck you!). If you aren't in a position to apply the patch, > >>>>> the workaround is rather simple: restart auditd. > >>>>> = > >>>>> If none of the above works, let me know, but I strongly suspect you= 're > >>>>> tripping over the race condition fixed in that last patch. > >>>> = > >>>> I didn't test the patch yet, but I restarted the auditd daemon 2 tim= es > >>>> and after that the queue has been flushed and I got all the message = since > >>>> this noon in the audit logs. > >>> = > >>> That sounds right; I'm guessing the patch above should be a more > >>> permanent fix. > >> = > >> The patch should be applied in 4.13-rc7 right? > >> = > >> It seems to fix the main issue (all the audit messages being logged in > >> dmesg) but I can still see from time to time the following message: > >> = > >> [ 14.747565] audit: audit_lost=3D59 audit_rate_limit=3D0 > >> audit_backlog_limit=3D64 [ 14.747566] audit: kauditd hold queue over= flow > > = > > That is a very low backlog_limit. Is this during boot? > = > Yes that's during boot (64 is the kernel's default) > = > In my audit.rules I have '-b 8192' OK. I blogged about this issue back in May: http://security-plus-data-science.blogspot.com/2017/05/a-suggested-change-f= or-rhel-7-disa-stig.html TL;DR, you have to set a boot command line option audit_backlog_limit=3D8192. I'm also starting to wonder if we can set the backlog to 6144 now that auditd runs much faster (~90x). -Steve