From mboxrd@z Thu Jan  1 00:00:00 1970
Return-Path: <linux-kernel-owner@vger.kernel.org>
Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand
        id S1760149AbcIWOKE (ORCPT <rfc822;w@1wt.eu>);
        Fri, 23 Sep 2016 10:10:04 -0400
Received: from mail-wm0-f66.google.com ([74.125.82.66]:33995 "EHLO
        mail-wm0-f66.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org
        with ESMTP id S1759913AbcIWOKC (ORCPT
        <rfc822;linux-kernel@vger.kernel.org>);
        Fri, 23 Sep 2016 10:10:02 -0400
Message-ID: <1474639800.4025.29.camel@gmail.com>
Subject: Re: thousands of kworker processes with 4.7.x and 4.8-rc*
From: Mike Galbraith <umgwanakikbuti@gmail.com>
To: Tomasz Chmielewski <mangoo@wpkg.org>,
        LKML <linux-kernel@vger.kernel.org>
Date: Fri, 23 Sep 2016 16:10:00 +0200
In-Reply-To: <20acd5801291dff03ed56b63919b9249@admin.virtall.com>
References: <35b2acf87bfeaf1d04f2cf3d10b99b6e@admin.virtall.com>
         <20acd5801291dff03ed56b63919b9249@admin.virtall.com>
Content-Type: text/plain; charset="us-ascii"
X-Mailer: Evolution 3.16.5 
Mime-Version: 1.0
Content-Transfer-Encoding: 7bit
Sender: linux-kernel-owner@vger.kernel.org
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

On Fri, 2016-09-23 at 22:23 +0900, Tomasz Chmielewski wrote:
> On 2016-09-19 16:08, Tomasz Chmielewski wrote:
> > On several servers running 4.7.x and 4.8-rc6/7 kernels I'm seeing
> > thousands of kworker processes.
> > # ps auxf|grep -c kworker
> > 2104
> > Load average goes into hundreds on a pretty much idle server (biggest
> > CPU and RAM consumers are probably SSHD with one user logged in and
> > rsyslog writing ~1 line per minute):
> > # uptime
> >  06:58:56 up 26 min,  1 user,  load average: 146.11, 215.46, 105.70
> > # uptime
> >  06:59:48 up 26 min,  1 user,  load average: 305.20, 240.84, 120.25
> > Sometimes seeing lots of them in "D" state:
> > root     19474  0.0  0.0      0     0 ?        D    06:54   0:00  \_
> > [kworker/0:208]
> > root     19475  0.0  0.0      0     0 ?        D    06:54   0:00  \_
> > [kworker/0:209]
> 
> 
> I did some experiments to see when the problem first appeared. Thousands 
> of kworker processes start to show up in 4.7.0-rc5.
> 
> kernel version | kworker count after boot
> -------------------------------------------
> 4.6.3 > 	>         37
> 4.6.4 > 	>         47
> 4.6.5 > 	>         46
> 4.6.6 > 	>         49
> 4.6.7 > 	>         49
> 4.7.0-rc1 > 	> 46
> 4.7.0-rc2 > 	> 49
> 4.7.0-rc3> 	> 45
> 4.7.0-rc4> 	> 47
> 4.7.0-rc5> 	> 1592

Best bet would be to use 'git bisect' to locate the exact commit that
caused this, and post the bisection result along with your config.

AFAIK, nobody else is seeing this, is the kernel virgin source?

	-Mike