From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S965713AbcAUQRE (ORCPT ); Thu, 21 Jan 2016 11:17:04 -0500 Received: from mail-wm0-f68.google.com ([74.125.82.68]:34225 "EHLO mail-wm0-f68.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S965307AbcAUQQ7 (ORCPT ); Thu, 21 Jan 2016 11:16:59 -0500 Date: Thu, 21 Jan 2016 18:16:56 +0200 From: "Kirill A. Shutemov" To: Nalorokk Cc: "Kirill A. Shutemov" , Stefan Strogin , Andrew Morton , Sasha Levin , Mel Gorman , linux-mm@kvack.org, linux-kernel@vger.kernel.org, oleksandr@natalenko.name Subject: Re: [REGRESSION] [BISECTED] kswapd high CPU usage Message-ID: <20160121161656.GA16564@node.shutemov.name> References: MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Disposition: inline Content-Transfer-Encoding: 8bit In-Reply-To: User-Agent: Mutt/1.5.23.1 (2014-03-12) Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On Fri, Jan 22, 2016 at 12:28:10AM +1000, Nalorokk wrote: > It appears that kernels newer than 4.1 have kswapd-related bug resulting in > high CPU usage. CPU 100% usage could last for several minutes or several > days, with CPU being busy entirely with serving kswapd. It happens usually > after server being mostly idle, sometimes after days, sometimes after weeks > of uptime. But the issue appears much sooner if the machine is loaded with > something like building a kernel. > > Here are the graphs of CPU load: first > , > second > . > Perf top output is here as well. > > To find the cause of this problem I've started with the fact that the issue > appeared after 4.1 kernel update. Then I performed longterm test of 3.18, > and discovered that 3.18 is unaffected by this bug. Then I did some tests > of 4.0 to confirm that this version behaves well too. > > Then I performed git bisect from tag v4.0 to v4.1-rc1 and found exact > commits that seem to be reason of high CPU usage. > > The first really "bad" commit is 79553da293d38d63097278de13e28a3b371f43c1. > 2 previous commits cause weird behavior as well resulting in kswapd > consuming more CPU than unaffected kernels, but not that much as the commit > pointed above. I believe those commits are related to the same mm tree > merge. > > I tried to add transparent_hugepage=never to kernel boot parameters, but it > did not change anything. Changing allocator to SLAB from SLUB alters > behavior and makes CPU load lower, but don't solve a problem at all. > > Here is kernel bugzilla > bugreport as well. > > Ideas? ​ Could you try to insert "late_initcall(set_recommended_min_free_kbytes);" back and check if makes any difference. -- Kirill A. Shutemov