Re: Kernel migration eat CPUs

Mike Galbraith
2013-08-29T07:31:35Z

(CCs)

On Thu, 2013-08-22 at 19:00 +0400, Alexey Vlasov wrote:
> Hi,
>
> Beginning from 3.7 including 3.10 kernel, I noticed that from time to
> time all CPUs turn out to be busy 100% and at the moment websites of my
> clients work slowly or don’t work at all.
>
> top screen:
> https://www.dropbox.com/s/psuk2cwn6exinyd/3.9.cpu_migration.jpg
>
> All CPUs are busy with kernel threads migration for about 3-5 minutes.
> Exactly how often it periodically happens I don’t know, but it happens
> once in 1-3 hours.
>
> Approximately 30 my servers work on kernel 2.6.35 and there’s no such
> problem there. We use kernels 3.7 – 3.9 on 5 servers and I see such
> effect everywhere. If you downgrade on 2.6.35, then migration doesn’t
> eat CPUs.
>
> These servers are used for usual LAMP shared hosting.
>
> Can I get migration as if on the kernel 2.6.35 or can this feature be
> disabled?

I've not seen anything like this since the below went into 3.6, nor
heard of same. Before that, migration threads could _appear_ to eat CPU
under certain circumstances. Perhaps perf top -p <pick-a-thread> while
the problem is happening will show something.

commit 8f6189684eb4e85e6c593cd710693f09c944450a
Author: Mike Galbraith <mgalbraith@suse.de>
Date: Sat Aug 4 05:44:14 2012 +0200

sched: Fix migration thread runtime bogosity

Make stop scheduler class do the same accounting as other classes,

Migration threads can be caught in the act while doing exec balancing,
leading to the below due to use of unmaintained ->se.exec_start. The
load that triggered this particular instance was an apparently out of
control heavily threaded application that does system monitoring in
what equated to an exec bomb, with one of the VERY frequently migrated
tasks being ps.

%CPU PID USER CMD
99.3 45 root [migration/10]
97.7 53 root [migration/12]
97.0 57 root [migration/13]
90.1 49 root [migration/11]
89.6 65 root [migration/15]
88.7 17 root [migration/3]
80.4 37 root [migration/8]
78.1 41 root [migration/9]
44.2 13 root [migration/2]