From: Ingo Molnar <mingo@kernel.org>
To: Dmitry Vyukov <dvyukov@google.com>
Cc: Peter Zijlstra <peterz@infradead.org>,
Wang Nan <wangnan0@huawei.com>, Ingo Molnar <mingo@redhat.com>,
LKML <linux-kernel@vger.kernel.org>,
He Kuang <hekuang@huawei.com>,
Alexei Starovoitov <ast@kernel.org>,
Arnaldo Carvalho de Melo <acme@redhat.com>,
Brendan Gregg <brendan.d.gregg@gmail.com>,
Jiri Olsa <jolsa@kernel.org>,
Masami Hiramatsu <masami.hiramatsu.pt@hitachi.com>,
Namhyung Kim <namhyung@kernel.org>, Zefan Li <lizefan@huawei.com>,
pi3orama@163.com
Subject: Re: [RESEND PATCH 0/5] perf core: Support overwrite ring buffer
Date: Tue, 8 Mar 2016 19:22:35 +0100 [thread overview]
Message-ID: <20160308182235.GB4220@gmail.com> (raw)
In-Reply-To: <20160308180238.GA4220@gmail.com>
* Ingo Molnar <mingo@kernel.org> wrote:
> With nproc set to 120 it seems to be chugging along at about 25% system
> utilization:
>
> Tasks: 1271 total, 1 running, 1270 sleeping, 0 stopped, 0 zombie
> %Cpu(s): 4.5 us, 35.3 sy, 0.0 ni, 55.2 id, 0.5 wa, 4.5 hi, 0.0 si, 0.0 st
> KiB Mem : 26401230+total, 25401017+free, 1624640 used, 8377496 buff/cache
> KiB Swap: 0 total, 0 free, 0 used. 26143329+avail Mem
>
> PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
> 87593 mingo 20 0 7850436 162284 11672 S 2941 0.1 33:01.83 syz-fuzzer
> 923 root 20 0 84772 44344 43840 S 22.8 0.0 1:23.86 systemd-journal
> 1369 root 16 -4 114636 3256 2832 S 15.7 0.0 0:29.03 auditd
> 1379 root 12 -8 80236 1764 1432 S 8.3 0.0 0:15.79 audispd
> 878 root 20 0 0 0 0 S 6.9 0.0 0:16.55 jbd2/sda1-8
> 1381 root 16 -4 52216 3232 2892 S 3.8 0.0 0:07.08 sedispatch
>
> Even that one is not ideal - obviously there's way too much systemd-journal
> overhead, but I'm unable to turn the darn thing off ...
The journal.conf man page suggests that putting 'Storage=none' into
/etc/systemd/journal.conf disables journalling - but that's not true.
It's apparently impossible to disable systemd logging via any normal means ...
So the workaround for all that is a brutal:
mv /var/log/journal /var/log/journal.dontuse
after that there does not seem to be systemd logging anymore.
... but there is tons of auditd logging now! ;-)
Fortunately that's easily stopped via:
service stop auditd
with that there's no log IO anymore during fuzzing (yay!).
Except that systemd journald rears its ugly head back:
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
111464 mingo 20 0 7249024 143348 11476 S 2741 0.1 7:41.31 syz-fuzzer
122632 root 20 0 29200 2768 2524 S 9.5 0.0 0:28.74 systemd-journal
111463 root 20 0 165596 5772 3840 R 4.8 0.0 0:00.80 top
112078 mingo 20 0 20928 704 644 S 2.9 0.0 0:00.17 syz-executor
112516 mingo 20 0 20928 708 644 S 2.9 0.0 0:00.16 syz-executor
878 root 20 0 0 0 0 S 1.9 0.0 0:53.83 jbd2/sda1-8
111711 mingo 20 0 20928 704 644 S 1.9 0.0 0:00.16 syz-executor
111901 mingo 20 0 20928 704 644 S 1.9 0.0 0:00.15 syz-executor
8 root 20 0 0 0 0 S 1.0 0.0 1:00.55 rcu_sched
so it's eating about 10% of system overhead despite doing nothing (!).
The workaround for that systemd bug is a brutal:
[root@fomalhaut ~]# mv /usr/lib/systemd/systemd-journald /usr/lib/systemd/systemd-journald.dontuse
[root@fomalhaut ~]#
Thanks,
Ingo
next prev parent reply other threads:[~2016-03-08 18:22 UTC|newest]
Thread overview: 38+ messages / expand[flat|nested] mbox.gz Atom feed top
2016-03-07 3:50 [RESEND PATCH 0/5] perf core: Support overwrite ring buffer Wang Nan
2016-03-07 3:50 ` [RESEND PATCH 1/5] perf core: Introduce new ioctl options to pause and resume " Wang Nan
2016-03-07 3:50 ` [RESEND PATCH 2/5] perf core: Set event's default overflow_handler Wang Nan
2016-03-07 3:50 ` [RESEND PATCH 3/5] perf core: Prepare writing into ring buffer from end Wang Nan
2016-03-07 3:50 ` [RESEND PATCH 4/5] perf core: Add backward attribute to perf event Wang Nan
2016-03-07 3:50 ` [RESEND PATCH 5/5] perf core: Reduce perf event output overhead by new overflow handler Wang Nan
2016-03-08 13:44 ` [RESEND PATCH 0/5] perf core: Support overwrite ring buffer Peter Zijlstra
2016-03-08 13:49 ` Ingo Molnar
2016-03-08 13:57 ` Peter Zijlstra
2016-03-08 15:29 ` Ingo Molnar
2016-03-08 15:35 ` Dmitry Vyukov
2016-03-08 15:54 ` Ingo Molnar
2016-03-08 16:11 ` Dmitry Vyukov
2016-03-08 16:27 ` Ingo Molnar
2016-03-08 16:29 ` Dmitry Vyukov
2016-03-08 16:32 ` Peter Zijlstra
2016-03-08 16:44 ` Ingo Molnar
2016-03-08 16:48 ` Ingo Molnar
2016-03-08 16:59 ` Dmitry Vyukov
2016-03-08 17:24 ` Ingo Molnar
2016-03-08 17:27 ` Dmitry Vyukov
2016-03-08 17:37 ` Ingo Molnar
2016-03-08 17:41 ` Dmitry Vyukov
2016-03-08 17:48 ` Ingo Molnar
2016-03-08 17:56 ` Dmitry Vyukov
2016-03-08 17:56 ` Peter Zijlstra
2016-03-08 17:57 ` Ingo Molnar
2016-03-08 18:02 ` Ingo Molnar
2016-03-08 18:22 ` Ingo Molnar [this message]
2016-03-08 18:31 ` Ingo Molnar
2016-03-08 17:55 ` Peter Zijlstra
2016-03-08 16:30 ` Peter Zijlstra
2016-03-09 10:53 ` Borislav Petkov
2016-03-09 11:19 ` Dmitry Vyukov
2016-03-08 19:56 ` Jiri Olsa
2016-03-08 20:07 ` Peter Zijlstra
2016-03-08 20:44 ` Jiri Olsa
2016-03-08 21:04 ` Peter Zijlstra
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20160308182235.GB4220@gmail.com \
--to=mingo@kernel.org \
--cc=acme@redhat.com \
--cc=ast@kernel.org \
--cc=brendan.d.gregg@gmail.com \
--cc=dvyukov@google.com \
--cc=hekuang@huawei.com \
--cc=jolsa@kernel.org \
--cc=linux-kernel@vger.kernel.org \
--cc=lizefan@huawei.com \
--cc=masami.hiramatsu.pt@hitachi.com \
--cc=mingo@redhat.com \
--cc=namhyung@kernel.org \
--cc=peterz@infradead.org \
--cc=pi3orama@163.com \
--cc=wangnan0@huawei.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.