Re: Kernel migration eat CPUs

Alexey Vlasov
2013-09-04T19:00:53Z

On Sun, Aug 25, 2013 at 04:28:37PM +0200, Peter Zijlstra wrote:

> and a function (or function-graph)
> trace for when this happens?

Unfortunately I could not make trace.
First when migration threads start to eat CPUs and when I turn on trace:
# echo function > current_tracer
# echo 1 > tracing_on
then server is down at once. LA comes up to several thousands, it stops
to receive commands. Sometimes I even have to reboot it.

And second when I manage to get trace then it shows little useful data.

https://www.dropbox.com/s/kxgwr9fuckddtgf/trace.txt.gz

Probably this junk appears from grsec patch? And I don't see migration
threads in it.

> Perhaps perf top -p <pick-a-thread> while the problem is happening will
> show something.

Also "perf top" doesn't show anything. It gives same values when
migration threads eat CPUs and when everything is OK on server.

=========================================
PerfTop: 68932 irqs/sec kernel:60.3% exact: 0.0% [4000Hz cycles], (all, 32 CPUs)
---------------------------------------------------------------------------------

64.11% [netconsole] [k] 0xffffffff81000de3
34.19% [unknown] [.] 0x0000000000585c45
0.40% libc-2.15.so [.] 0x0000000000124b10
0.23% perf [.] 0x000000000004df4f
0.14% ld-2.15.so [.] 0x00000000000183e7
0.08% libc-2.15.so [.] _nss_files_parse_grent
0.06% libc-2.15.so [.] _nss_files_parse_pwent
0.04% libc-2.15.so [.] memchr
0.04% php-cgi [.] match
0.03% libcrypt-2.15.so [.] 0x0000000000004eab
0.03% bash [.] 0x000000000001fe5a
0.02% libc-2.15.so [.] _IO_getline_info
0.02% php-cgi [.] php_pcre_exec
0.02% libc-2.15.so [.] _dl_addr
0.02% eaccelerator.so [.] eaccelerator_crc32
0.02% php-cgi [.] zend_hash_quick_find
0.02% libnss_compat-2.15.so [.] 0x00000000000055f1
0.01% php-cgi [.] zendparse
0.01% libc-2.15.so [.] fgets_unlocked
0.01% libnss_compat-2.15.so [.] _nss_compat_initgroups_dyn
=========================================

Now server is more loaded and eating CPUs by migration threads happens more often
and it lasts longer.