From mboxrd@z Thu Jan 1 00:00:00 1970 From: Kangkook Jee Subject: Auditd framework slowdowns (sometimes freezes) the entire system. Date: Thu, 16 Jul 2015 08:38:22 -0400 Message-ID: <09D2CD6A-EA4F-4925-BAB5-44086B86424B@gmail.com> Mime-Version: 1.0 (Mac OS X Mail 8.2 \(2102\)) Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Return-path: Received: from mx1.redhat.com (ext-mx06.extmail.prod.ext.phx2.redhat.com [10.5.110.30]) by int-mx14.intmail.prod.int.phx2.redhat.com (8.14.4/8.14.4) with ESMTP id t6GCcRYY015043 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO) for ; Thu, 16 Jul 2015 08:38:27 -0400 Received: from mail-qk0-f182.google.com (mail-qk0-f182.google.com [209.85.220.182]) by mx1.redhat.com (Postfix) with ESMTPS id 828713897D6 for ; Thu, 16 Jul 2015 12:38:25 +0000 (UTC) Received: by qkfc129 with SMTP id c129so5898190qkf.1 for ; Thu, 16 Jul 2015 05:38:24 -0700 (PDT) Received: from am14-mac3.nec-labs.com ([138.15.165.52]) by smtp.gmail.com with ESMTPSA id 124sm3923695qhx.34.2015.07.16.05.38.22 for (version=TLSv1 cipher=ECDHE-RSA-RC4-SHA bits=128/128); Thu, 16 Jul 2015 05:38:23 -0700 (PDT) List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: linux-audit-bounces@redhat.com Errors-To: linux-audit-bounces@redhat.com To: linux-audit@redhat.com List-Id: linux-audit@redhat.com Hi all, I'm writing a custom user-land auditd client subscribing to kauditd to monitor a number of system calls that we are interested. My auditd client seems to work fine in overall but I found unexpected behavior of auditd framework which slows down (or sometimes freezes) the entire system as the consuming rate of audit client couldn't catch up the speed of audit message generation. Here's the simple code snippet used to reproduce the problem. // // To build. // g++ -o simple_audit -std=c++11 -L/usr/lib/x86_64-linux-gnu/ main.cpp -laudit // #include #include #include #include #include static int32_t fd = -1; static bool au_listen_flag = true; int main(int argc, char* argv[]) { struct audit_reply rep; uint64_t cnt = 0; if (argc != 2) { fprintf(stderr, "Invalid usage: %s \n", argv[0]); exit(1); } uint32_t sleep_time = atoi(argv[1]); fd = audit_open(); if (fd < 0) { // error handling. std::cerr << "Invalid fd returned: " + std::to_string(fd) << std::endl; exit(-1); } int32_t ret = audit_set_pid(fd, getpid(), WAIT_YES); if (ret < 0) { std::cerr << "audit_set_pid failed: " + std::to_string(fd) << std::endl; exit(-1); } while (au_listen_flag) { int32_t rc = audit_get_reply(fd, &rep, GET_REPLY_BLOCKING, 0); if (rc > 0) { cnt++; } usleep(sleep_time); if (cnt % 10000 == 0) { printf ("messages %lu\n", cnt); } } close(fd); } The problem becomes more apparent as we increase the amount of sleep time that is provided as a first command line argument (say a thousand Milli-seconds) and simultaneously run some heavy-load tasks (i.e., kernel build). sudo ./simple_audit 1000 Here's the command line that we used to add system calls to be monitored and enable. # Adding events. /sbin/auditctl -a exit,always -F arch=b64 -S clone -S close -S creat -S dup -S dup2 -S dup3 -S execve -S exit -S exit_group -S fork -S open -S openat -S unlink -S unlinkat -S vfork -S 288 -S accept -S bind -S connect -S listen -S socket -S socketpair # Enabling events. /sbin/auditctl -e1 -b 102400 At the very moment, "auditctl -s" indicating that kernel buffer is filled up but it does not throw away audit messages ('lost' is not increasing ). # auditctl -s AUDIT_STATUS: enabled=1 flag=1 pid=29887 rate_limit=0 backlog_limit=102400 lost=270878600 backlog=102402 # auditctl -s AUDIT_STATUS: enabled=1 flag=1 pid=29887 rate_limit=0 backlog_limit=102400 lost=270878600 backlog=102402 Could anyone guide me how to configure kauditd's buffer setting so that it can dump audit messages when the buffer is filled up and user-land consumer can't catch up the speed of audit message produce? Thanks a lot for your help in advance! Regards, Kangkook