Re: [PATCH] audit: improve robustness of the audit queue handling

[cuigaosheng <cuigaosheng1@huawei.com>]

在 2021/12/14 2:31, Paul Moore 写道:
> If the audit daemon were ever to get stuck in a stopped state the
> kernel's kauditd_thread() could get blocked attempting to send audit
> records to the userspace audit daemon.  With the kernel thread
> blocked it is possible that the audit queue could grow unbounded as
> certain audit record generating events must be exempt from the queue
> limits else the system enter a deadlock state.
>
> This patch resolves this problem by lowering the kernel thread's
> socket sending timeout from MAX_SCHEDULE_TIMEOUT to HZ/10 and tweaks
> the kauditd_send_queue() function to better manage the various audit
> queues when connection problems occur between the kernel and the
> audit daemon.  With this patch, the backlog may temporarily grow
> beyond the defined limits when the audit daemon is stopped and the
> system is under heavy audit pressure, but kauditd_thread() will
> continue to make progress and drain the queues as it would for other
> connection problems.  For example, with the audit daemon put into a
> stopped state and the system configured to audit every syscall it
> was still possible to shutdown the system without a kernel panic,
> deadlock, etc.; granted, the system was slow to shutdown but that is
> to be expected given the extreme pressure of recording every syscall.
>
> The timeout value of HZ/10 was chosen primarily through
> experimentation and this developer's "gut feeling".  There is likely
> no one perfect value, but as this scenario is limited in scope (root
> privileges would be needed to send SIGSTOP to the audit daemon), it
> is likely not worth exposing this as a tunable at present.  This can
> always be done at a later date if it proves necessary.
>
> Cc: stable@vger.kernel.org
> Fixes: 5b52330bbfe63 ("audit: fix auditd/kernel connection state tracking")
> Reported-by: Gaosheng Cui <cuigaosheng1@huawei.com>
> Signed-off-by: Paul Moore <paul@paul-moore.com>
> ---
>   kernel/audit.c |   21 ++++++++++-----------
>   1 file changed, 10 insertions(+), 11 deletions(-)
>
> diff --git a/kernel/audit.c b/kernel/audit.c
> index 121d37e700a6..4cebadb5f30d 100644
> --- a/kernel/audit.c
> +++ b/kernel/audit.c
> @@ -718,7 +718,7 @@ static int kauditd_send_queue(struct sock *sk, u32 portid,
>   {
>   	int rc = 0;
>   	struct sk_buff *skb;
> -	static unsigned int failed = 0;
> +	unsigned int failed = 0;
>   
>   	/* NOTE: kauditd_thread takes care of all our locking, we just use
>   	 *       the netlink info passed to us (e.g. sk and portid) */
> @@ -735,32 +735,30 @@ static int kauditd_send_queue(struct sock *sk, u32 portid,
>   			continue;
>   		}
>   
> +retry:
>   		/* grab an extra skb reference in case of error */
>   		skb_get(skb);
>   		rc = netlink_unicast(sk, skb, portid, 0);
>   		if (rc < 0) {
> -			/* fatal failure for our queue flush attempt? */
> +			/* send failed - try a few times unless fatal error */
>   			if (++failed >= retry_limit ||
>   			    rc == -ECONNREFUSED || rc == -EPERM) {
> -				/* yes - error processing for the queue */
>   				sk = NULL;
>   				if (err_hook)
>   					(*err_hook)(skb);
> -				if (!skb_hook)
> -					goto out;
> -				/* keep processing with the skb_hook */
> +				if (rc == -EAGAIN)
> +					rc = 0;
> +				/* continue to drain the queue */
>   				continue;
>   			} else
> -				/* no - requeue to preserve ordering */
> -				skb_queue_head(queue, skb);
> +				goto retry;
>   		} else {
> -			/* it worked - drop the extra reference and continue */
> +			/* skb sent - drop the extra reference and continue */
>   			consume_skb(skb);
>   			failed = 0;
>   		}
>   	}
>   
> -out:
>   	return (rc >= 0 ? 0 : rc);
>   }
>   
> @@ -1609,7 +1607,8 @@ static int __net_init audit_net_init(struct net *net)
>   		audit_panic("cannot initialize netlink socket in namespace");
>   		return -ENOMEM;
>   	}
> -	aunet->sk->sk_sndtimeo = MAX_SCHEDULE_TIMEOUT;
> +	/* limit the timeout in case auditd is blocked/stopped */
> +	aunet->sk->sk_sndtimeo = HZ / 10;
>   
>   	return 0;
>   }
>
> .

Tested-by: Gaosheng Cui<cuigaosheng1@huawei.com>


--
Linux-audit mailing list
Linux-audit@redhat.com
https://listman.redhat.com/mailman/listinfo/linux-audit