From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.3 required=3.0 tests=HEADER_FROM_DIFFERENT_DOMAINS, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_SANE_1 autolearn=no autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 9DA11C4BA0B for ; Wed, 26 Feb 2020 08:04:40 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 3C57721556 for ; Wed, 26 Feb 2020 08:04:39 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 3C57721556 Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=techsingularity.net Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 793A76B0003; Wed, 26 Feb 2020 03:04:39 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 7441B6B0005; Wed, 26 Feb 2020 03:04:39 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 659CA6B0006; Wed, 26 Feb 2020 03:04:39 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0160.hostedemail.com [216.40.44.160]) by kanga.kvack.org (Postfix) with ESMTP id 4B9CC6B0003 for ; Wed, 26 Feb 2020 03:04:39 -0500 (EST) Received: from smtpin07.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay03.hostedemail.com (Postfix) with ESMTP id 2C3938248068 for ; Wed, 26 Feb 2020 08:04:39 +0000 (UTC) X-FDA: 76531541478.07.crowd36_287f4096cdb3c X-HE-Tag: crowd36_287f4096cdb3c X-Filterd-Recvd-Size: 4549 Received: from outbound-smtp18.blacknight.com (outbound-smtp18.blacknight.com [46.22.139.245]) by imf05.hostedemail.com (Postfix) with ESMTP for ; Wed, 26 Feb 2020 08:04:38 +0000 (UTC) Received: from mail.blacknight.com (pemlinmail02.blacknight.ie [81.17.254.11]) by outbound-smtp18.blacknight.com (Postfix) with ESMTPS id 4AC101C449B for ; Wed, 26 Feb 2020 08:04:36 +0000 (GMT) Received: (qmail 14671 invoked from network); 26 Feb 2020 08:04:36 -0000 Received: from unknown (HELO techsingularity.net) (mgorman@techsingularity.net@[84.203.18.57]) by 81.17.254.9 with ESMTPSA (AES256-SHA encrypted, authenticated); 26 Feb 2020 08:04:36 -0000 Date: Wed, 26 Feb 2020 08:04:26 +0000 From: Mel Gorman To: Andrew Morton Cc: Michal Hocko , Vlastimil Babka , Ivan Babrou , Rik van Riel , Linux-MM , Linux Kernel Mailing List Subject: Re: [PATCH 0/3] Limit runaway reclaim due to watermark boosting Message-ID: <20200226080426.GA3818@techsingularity.net> References: <20200225141534.5044-1-mgorman@techsingularity.net> <20200225185130.6a32a8a6920d11b4c098e90e@linux-foundation.org> MIME-Version: 1.0 Content-Type: text/plain; charset=iso-8859-15 Content-Disposition: inline In-Reply-To: <20200225185130.6a32a8a6920d11b4c098e90e@linux-foundation.org> User-Agent: Mutt/1.10.1 (2018-07-13) X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: On Tue, Feb 25, 2020 at 06:51:30PM -0800, Andrew Morton wrote: > On Tue, 25 Feb 2020 14:15:31 +0000 Mel Gorman wrote: > > > Ivan Babrou reported the following > > http://lkml.kernel.org/r/CABWYdi1eOUD1DHORJxTsWPMT3BcZhz++xP1pXhT=x4SgxtgQZA@mail.gmail.com > is helpful. > Noted for future reference. > > Commit 1c30844d2dfe ("mm: reclaim small amounts of memory when > > an external fragmentation event occurs") introduced undesired > > effects in our environment. > > > > * NUMA with 2 x CPU > > * 128GB of RAM > > * THP disabled > > * Upgraded from 4.19 to 5.4 > > > > Before we saw free memory hover at around 1.4GB with no > > spikes. After the upgrade we saw some machines decide that they > > need a lot more than that, with frequent spikes above 10GB, > > often only on a single numa node. > > > > There have been a few reports recently that might be watermark boost > > related. Unfortunately, finding someone that can reproduce the problem > > and test a patch has been problematic. This series intends to limit > > potential damage only. > > It's problematic that we don't understand what's happening. And these > palliatives can only reduce our ability to do that. > Not for certain no, but we do know that there are conditions whereby node 0 can end up reclaiming excessively for extended periods of time. The available evidence does match a pattern whereby a lower zone on node 0 is getting stuck in a boosted state. > Rik seems to have the means to reproduce this (or something similar) > and it seems Ivan can test patches three weeks hence. If Rik can reproduce it great but I have a strong feeling that Ivan may never be able to test this if it requires a production machine which is why I did not wait the three weeks. > So how about a > debug patch which will help figure out what's going on in there? A debug patch would not help much in this case given that we have tracepoints. An ftrace containing mm_page_alloc_extfrag, mm_vmscan_kswapd_wake, mm_vmscan_wakeup_kswapd and mm_vmscan_node_reclaim_begin would be a big help for 30 seconds while the problem is occurring would work. Ideally mm_vmscan_lru_shrink_inactive would also be included to capture the priority but the size of the trace is what's going to be problematic. mm_page_alloc_extfrag would be correlated with the conditions that boost the watermarks and the others would track what kswapd is doing to see if it's persistently reclaiming. If they are, mm_vmscan_lru_shrink_inactive would tell if it's persistently reclaiming at priority DEF_PRIORITY - 2 which would prove the patch would at least mitigate the problem. It would be more preferable to have a description of a testcase that reproduces the problem and I'll capture/analyse the trace myself. It would also be something I could slot into a test grid to catch the problem happening again in the future. -- Mel Gorman SUSE Labs