Re: excessive kworker activity when idle. (was Re: vma corruption in today's -git)

[Tejun Heo <tj@kernel.org>]

Hello,

On Wed, Mar 30, 2011 at 08:37:21PM -0700, Linus Torvalds wrote:
> On Wed, Mar 30, 2011 at 8:09 PM, Dave Jones <davej@redhat.com> wrote:
> >
> > But rerunning the same tests on current head (6aba74f2791287ec407e0f92487a725a25908067)
> > I can still reproduce the problem where kworker threads go nutso
> > when the machine should be completely idle.
> >
> > top shows two kworker threads constantly at >80% cpu.
> 
> Ok, I've seen that "tons of cpu by kworker" triggered by a few
> different issues. One was the intel graphics driver doing monitor
> detection constantly, and spending all its time in one of the worker
> threads doing __udelay() for the stupid i2c driver.
> 
> The other case I've seen is a wireless thing that falls back to GPIO,
> and spends a lot of CPU time on that.
> 
> I'm not saying yours is either of those cases, but one of the problems
> with that behavior is that it's actually fairly hard to figure out
> what the hell is happening. You don't see some nice thread description
> in 'top' any more (like you used to when everybody created their own
> threads and didn't do the common worker thread thing), and the best
> approach literally seems to be something like

Yes, not having dedicated workers decrease immediate visibility via
ps/top but I don't think the actual vsibility is worse.  There are two
types for kworker looping - something being scheduled in rapid
succession or a single work consuming lots of CPU cycles.

The first one can be tracked using tracing.

  $ echo workqueue:workqueue_queue_work > /sys/kernel/debug/tracing/set_event
  $ cat /sys/kernel/debug/tracing/trace_pipe > out.txt
  (wait a few secs)
  ^C

If something is busy looping on work queueing, it would be dominating
the output and the offender can be determined with the work item
function.

For the second type, "cat /proc/THE_OFFENDING_KWORKER/stack" is the
easiest.  The work item function will be trivially visible in the
stack trace.

Thanks.

-- 
tejun