Re: [RESEND PATCH 0/5] perf core: Support overwrite ring buffer

Ingo Molnar
2016-03-08T18:22:45Z

* Ingo Molnar <mingo@kernel.org> wrote:

> With nproc set to 120 it seems to be chugging along at about 25% system
> utilization:
>
> Tasks: 1271 total, 1 running, 1270 sleeping, 0 stopped, 0 zombie
> %Cpu(s): 4.5 us, 35.3 sy, 0.0 ni, 55.2 id, 0.5 wa, 4.5 hi, 0.0 si, 0.0 st
> KiB Mem : 26401230+total, 25401017+free, 1624640 used, 8377496 buff/cache
> KiB Swap: 0 total, 0 free, 0 used. 26143329+avail Mem
>
> PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
> 87593 mingo 20 0 7850436 162284 11672 S 2941 0.1 33:01.83 syz-fuzzer
> 923 root 20 0 84772 44344 43840 S 22.8 0.0 1:23.86 systemd-journal
> 1369 root 16 -4 114636 3256 2832 S 15.7 0.0 0:29.03 auditd
> 1379 root 12 -8 80236 1764 1432 S 8.3 0.0 0:15.79 audispd
> 878 root 20 0 0 0 0 S 6.9 0.0 0:16.55 jbd2/sda1-8
> 1381 root 16 -4 52216 3232 2892 S 3.8 0.0 0:07.08 sedispatch
>
> Even that one is not ideal - obviously there's way too much systemd-journal
> overhead, but I'm unable to turn the darn thing off ...

The journal.conf man page suggests that putting 'Storage=none' into
/etc/systemd/journal.conf disables journalling - but that's not true.

It's apparently impossible to disable systemd logging via any normal means ...

So the workaround for all that is a brutal:

mv /var/log/journal /var/log/journal.dontuse

after that there does not seem to be systemd logging anymore.

... but there is tons of auditd logging now! ;-)

Fortunately that's easily stopped via:

service stop auditd

with that there's no log IO anymore during fuzzing (yay!).

Except that systemd journald rears its ugly head back:

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
111464 mingo 20 0 7249024 143348 11476 S 2741 0.1 7:41.31 syz-fuzzer
122632 root 20 0 29200 2768 2524 S 9.5 0.0 0:28.74 systemd-journal
111463 root 20 0 165596 5772 3840 R 4.8 0.0 0:00.80 top
112078 mingo 20 0 20928 704 644 S 2.9 0.0 0:00.17 syz-executor
112516 mingo 20 0 20928 708 644 S 2.9 0.0 0:00.16 syz-executor
878 root 20 0 0 0 0 S 1.9 0.0 0:53.83 jbd2/sda1-8
111711 mingo 20 0 20928 704 644 S 1.9 0.0 0:00.16 syz-executor
111901 mingo 20 0 20928 704 644 S 1.9 0.0 0:00.15 syz-executor
8 root 20 0 0 0 0 S 1.0 0.0 1:00.55 rcu_sched

so it's eating about 10% of system overhead despite doing nothing (!).

The workaround for that systemd bug is a brutal:

[root@fomalhaut ~]# mv /usr/lib/systemd/systemd-journald /usr/lib/systemd/systemd-journald.dontuse
[root@fomalhaut ~]#

Thanks,

Ingo