From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id C80BCC433EF for ; Tue, 25 Jan 2022 20:31:00 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1643142658; h=from:from:sender:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:list-id:list-help: list-unsubscribe:list-subscribe:list-post; bh=xnDzsuxd8VrL/8TisygEF97MpIEbffJh97LFX9PV6yc=; b=XXuDCffIvja3792MQSg/N4/Tny7jmq0EpYBEdc31+1AxF10+f0ODywELz5zO1c0ysRcCI3 EoI7sc9Uv66fxUWCphe1+39uoPgSLH5vySaxPefgpzodQJyXysUa3fnj7hjDfdHsqHnpcH Q2zwg7H1TzToyepshvGYSo1ECotdAC8= Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-54-oLcbAiRePdWT2PII6LN8HA-1; Tue, 25 Jan 2022 15:30:54 -0500 X-MC-Unique: oLcbAiRePdWT2PII6LN8HA-1 Received: from smtp.corp.redhat.com (int-mx03.intmail.prod.int.phx2.redhat.com [10.5.11.13]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id 46AB983DD25; Tue, 25 Jan 2022 20:30:51 +0000 (UTC) Received: from colo-mx.corp.redhat.com (colo-mx01.intmail.prod.int.phx2.redhat.com [10.5.11.20]) by smtp.corp.redhat.com (Postfix) with ESMTPS id 668C04D718; Tue, 25 Jan 2022 20:30:50 +0000 (UTC) Received: from lists01.pubmisc.prod.ext.phx2.redhat.com (lists01.pubmisc.prod.ext.phx2.redhat.com [10.5.19.33]) by colo-mx.corp.redhat.com (Postfix) with ESMTP id 096BA1809CB8; Tue, 25 Jan 2022 20:30:48 +0000 (UTC) Received: from smtp.corp.redhat.com (int-mx05.intmail.prod.int.phx2.redhat.com [10.5.11.15]) by lists01.pubmisc.prod.ext.phx2.redhat.com (8.13.8/8.13.8) with ESMTP id 20PKUj0I023305 for ; Tue, 25 Jan 2022 15:30:45 -0500 Received: by smtp.corp.redhat.com (Postfix) id 12778CB831; Tue, 25 Jan 2022 20:30:45 +0000 (UTC) Received: from x2.localnet (unknown [10.22.9.231]) by smtp.corp.redhat.com (Postfix) with ESMTP id A55DACB83C; Tue, 25 Jan 2022 20:30:35 +0000 (UTC) From: Steve Grubb To: linux-audit@redhat.com, Amjad Gabbar Subject: Re: Maximum Value for q_depth Date: Tue, 25 Jan 2022 15:30:34 -0500 Message-ID: <8063516.NyiUUSuA9g@x2> Organization: Red Hat In-Reply-To: References: <4366969.LvFx2qVVIh@x2> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.15 X-loop: linux-audit@redhat.com X-BeenThere: linux-audit@redhat.com X-Mailman-Version: 2.1.12 Precedence: junk List-Id: Linux Audit Discussion List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: linux-audit-bounces@redhat.com Errors-To: linux-audit-bounces@redhat.com X-Scanned-By: MIMEDefang 2.79 on 10.5.11.13 Authentication-Results: relay.mimecast.com; auth=pass smtp.auth=CUSA124A263 smtp.mailfrom=linux-audit-bounces@redhat.com X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Hello, On Tuesday, January 18, 2022 1:36:40 AM EST Amjad Gabbar wrote: > Reaching out regarding the same issue of syslog containing *"auditd > dispatch error (pipe full) event lost messages". * > > Post excluding the default events(LOGIN, USER_START etc) mentioned in our > previous chat, there has been a significant drop in the log volume and > hence I was expecting these error messages to be resolved. > > But unfortunately, even after increasing the dispatcher queue size(q_depth) > and changing disp_qos to become lossless , I am still seeing mentions of > these pipe full errors in my syslog. > The surprising thing is if I try to take a look at the events/keys causing > this issue, there doesn't seem to be a lot of events for messages to be > dropped. Dispatcher plugins can also cause things to get backed up. You mention that you have the af_unix plugin. Whatever reads from that needs to unload events quickly. It's recommended for the plugin to have 2 threads, one to dequeue and one to process the event. It should have storage to hold events if it's processing gets behind. > Ex- Using the command *"aureport --summary -ts messages start reported in syslog > -te < end time of dropped messages > end reported in syslog > -i (-x/-u/--key)"*, the total events are around > 2000 during this time period. The dispatcher queue size is close to 25,000, > So I am not really sure why the dispatcher is unable to handle these > messages. The queue size is sufficient enough to handle 10x the total > events being seen. Its entirely depending on the plugin to grab it's event. > Some other theoretical questions I had surrounding this are: > > > - The audit daemon picks events from the kernel buffer and sends it to > the dispatcher buffer. Who writes these logs to /var/log/audit.log - is > it the daemon or the dispatcher? The daemon > And also, are the total events reported > in /var/log/audit.log inclusive of the dropped events reported in syslog > or exclusive? It writes to the log and then dispatches the event. > i.e is it possible that all the events have been recorded in > audit.log but syslog has an issue in keeping up with the events as it is > the only plugin that is being used by the dispatcher. You previously mentioned an af_unix plugin. That would be my guess. You can disable the syslog plugin and find out if it's the cause. > - Is there a way to find out what is the total number of events dropped > by the dispatcher? I don't think anything keeps metrics on that. > - In auditd v3+, the daemon itself handles dispatching capabilities. So, > what does q_depth refer to in this scenario? The internal queue between the logging thread and the dispatching thread. > - In the man pages for different distros for disp_qos the following > statement is common - " There is a 128k buffer between the audit daemon > and dispatcher." This buffer is the kernel inter-process pipe size. There are up to 4 buffers in the 2.8 series. The backlog, the inter-process kernel buffer between auditd and audispd, a buffer in audispd, and the inter-process kernel buffer with the plugin. Only 2 of those have config options. > But different distros seem to have different default > values for q_depth ranging from 80 to 1200. How is it possible that > these numbers vary but the size of the buffer remains 128k. It's a different buffer. The 128k refers to the inter-process pipe buffer. -Steve > On Tue, Dec 21, 2021 at 2:39 PM Steve Grubb wrote: > > Hello, > > > > On Tuesday, December 21, 2021 12:55:47 AM EST Amjad Gabbar wrote: > > > Based on our discussion above, I performed some analysis as to why we > > > > were > > > > > seeing so many events. The reason seems to be due to the default rules > > > being triggered every time a cron job runs. We have numerous cron jobs > > > running per minute as a result of which multiple different > > > events(LOGIN, > > > USER_END,CRED_DISP etc) are generated each time a cron job runs. As we > > > do > > > not enable SELinux, disabling these thing use subj_type=crond_t is not > > > a > > > viable option. > > > > > > 1. I have tried the following way to exclude using msg_type and exe > > > together and it seems to work. > > > > > > -a exclude,always -F msgtype=MAC_IPSEC_EVENT -F exe=/usr/sbin/cron > > > -a exclude,always -F msgtype=USER_AUTH -F exe=/usr/sbin/cron > > > -a exclude,always -F msgtype=USER_ACCT -F exe=/usr/sbin/cron > > > -a exclude,always -F msgtype=CRED_REFR -F exe=/usr/sbin/cron > > > -a exclude,always -F msgtype=CRED_DISP -F exe=/usr/sbin/cron > > > -a exclude,always -F msgtype=CRED_ACQ -F exe=/usr/sbin/cron > > > -a exclude,always -F msgtype=USER_START -F exe=/usr/sbin/cron > > > -a exclude,always -F msgtype=USER_END -F exe=/usr/sbin/cron > > > -a exclude,always -F msgtype=SERVICE_START -F exe=/usr/sbin/cron > > > > > > Just want to make sure there is nothing I am missing here and that this > > > only excludes the msg types for the cron executable. > > > > I think so. But it's easy enough to test. Just login and see if you get > > any > > USER_START events from something other than cron. > > > > > 2. Apart from these messages, there is a LOGIN message that gets > > > > generated > > > > > each time a cron runs. Eventhough, the LOGIN message in auditd does not > > > have an exe field, the following statement surprisingly seems to be > > > working. > > > > > > -a exclude,always -F msgtype=LOGIN -F exe=/usr/sbin/cron > > > > > > I can still see LOGIN messages for other users but the cron LOGIN > > > > messages > > > > > seem to be suppressed. Could you provide some detail as to how this is > > > happening and is the expected result. > > > > It doesn't match against the text in the event. It matches against the > > process's attributes. > > > > > 3. Is there a better way to suppress these cron messages that I am not > > > considering apart from the SELinux option mentioned. > > > > I think you found the best way for a non-selinux system. Back when it was > > documented that it could be supressed by selinux type, audit by > > executable > > did not exist. But as you found, that is an effective way to get rid of > > the > > events. > > > > I also think the cronie program might be a little more audit friendly. It > > does not call PAM for the system crontabs run under the root user. PAM is > > run > > only for the local crontab (i.e. the one edited by the crontab command) > > and > > in case of the system crontabs only for jobs that are run under non-root > > user. > > > > -Steve -- Linux-audit mailing list Linux-audit@redhat.com https://listman.redhat.com/mailman/listinfo/linux-audit