From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 7C83EC27C53 for ; Fri, 7 Jun 2024 07:12:41 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 0F0346B009E; Fri, 7 Jun 2024 03:12:41 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 0A0946B009F; Fri, 7 Jun 2024 03:12:41 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id EA9D66B00A0; Fri, 7 Jun 2024 03:12:40 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id C65916B009E for ; Fri, 7 Jun 2024 03:12:40 -0400 (EDT) Received: from smtpin29.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay03.hostedemail.com (Postfix) with ESMTP id 3DFCBA1737 for ; Fri, 7 Jun 2024 07:12:40 +0000 (UTC) X-FDA: 82203224880.29.9237E1E Received: from invmail4.hynix.com (exvmail4.hynix.com [166.125.252.92]) by imf02.hostedemail.com (Postfix) with ESMTP id 8DC3C8000C for ; Fri, 7 Jun 2024 07:12:36 +0000 (UTC) Authentication-Results: imf02.hostedemail.com; dkim=none; dmarc=none; spf=pass (imf02.hostedemail.com: domain of byungchul@sk.com designates 166.125.252.92 as permitted sender) smtp.mailfrom=byungchul@sk.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1717744358; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=QN6LTPUoadE0TvvyHljdXRDd3pNfJ7ZDLAoCzii7P2U=; b=h4Hfj0Rg518StjwdQdyxNFz0/XEl5hjMkgJBA+DTIVWK2INB2RayPkBVQHwgcwtRw+7B+m EZJ/SOweLMiqIOAzgCCPgRYaTa2unR1OGns6UIrZwVQ0FrruLpohoJoufEE9UKIYUZEPso 7jJW9zD7P/x96JpaF0uIxfxsHS/S8kA= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1717744358; a=rsa-sha256; cv=none; b=7GAhCQz5/l6cWlPbRC1STIw7uQvJLLBpPCeMT3WJp9rMfTzb301h8Gr3rKX7+r2GfsjjQz 4xSa4V0mWJhMpHNI62MDBYy2Al/0lTGJUDGGEk5rpjxdgyE1BbfTKR1sXpnBDXeGfWbtid a8q6YEX1lXi7kBv+zFNDttS4jzqa3DU= ARC-Authentication-Results: i=1; imf02.hostedemail.com; dkim=none; dmarc=none; spf=pass (imf02.hostedemail.com: domain of byungchul@sk.com designates 166.125.252.92 as permitted sender) smtp.mailfrom=byungchul@sk.com X-AuditID: a67dfc5b-d85ff70000001748-ed-6662b2e1bdcd Date: Fri, 7 Jun 2024 16:12:28 +0900 From: Byungchul Park To: "Huang, Ying" Cc: akpm@linux-foundation.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, kernel_team@skhynix.com, hannes@cmpxchg.org, iamjoonsoo.kim@lge.com, rientjes@google.com Subject: Re: [PATCH v2] mm: let kswapd work again for node that used to be hopeless but may not now Message-ID: <20240607071228.GA76933@system.software.com> References: <20240604072323.10886-1-byungchul@sk.com> <87bk4hcf7h.fsf@yhuang6-desk2.ccr.corp.intel.com> <20240604084533.GA68919@system.software.com> <8734ptccgi.fsf@yhuang6-desk2.ccr.corp.intel.com> <20240605015021.GB75311@system.software.com> <87tti8b10g.fsf@yhuang6-desk2.ccr.corp.intel.com> <20240605021902.GC75311@system.software.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20240605021902.GC75311@system.software.com> User-Agent: Mutt/1.9.4 (2018-02-28) X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFrrHLMWRmVeSWpSXmKPExsXC9ZZnoe6jTUlpBhsmMFrMWb+GzWL1Jl+L ld3NbBaXd81hs7i35j+rRduSjUwWJ2dNZnFg9zj85j2zx4JNpR6L97xk8tj0aRK7R9fbK0we J2b8ZvH4vEkugD2KyyYlNSezLLVI3y6BK2Pn41WMBT8MKlb3rGZsYDyh2sXIySEhYCLx+8wx ti5GDjB7wg5jkDCLgIrEtfNf2EFsNgF1iRs3fjKD2CICGhKfFi4HinNxMAusZZQ4crWFFaRX WCBV4vj2AJAaXgELiTn/l7OA1AgJ3GOS2PtjMiNEQlDi5MwnLCA2s4CWxI1/L5lAepkFpCWW /+MACXMKWEpsP7QfrFxUQFniwLbjTCBzJAT2sEms6zzKAnGzpMTBFTdYJjAKzEIydhaSsbMQ xi5gZF7FKJSZV5abmJljopdRmZdZoZecn7uJERjqy2r/RO9g/HQh+BCjAAejEg/vhU2JaUKs iWXFlbmHGCU4mJVEeP2K49OEeFMSK6tSi/Lji0pzUosPMUpzsCiJ8xp9K08REkhPLEnNTk0t SC2CyTJxcEo1MM6yVZhwm8d8wtyKg3XlC2YwOa16yr7qxUnJeJtday5aL1L21GNbWffmVr5Q z2HF7J/erxim3YifZrfzfsKV2w+z+MuNflwJmB+Y6cfRO/O2s6XNFIv7cw/fzvapt299V3zT QOk2X9u/lavD/009w3H5gPM365Y/hz9kz/DU1mV63VCzQTCgIEyJpTgj0VCLuag4EQDgNlSm cQIAAA== X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFjrKLMWRmVeSWpSXmKPExsXC5WfdrPtwU1Kawd2/ehZz1q9hs1i9yddi ZXczm8XhuSdZLS7vmsNmcW/Nf1aLtiUbmSxOzprM4sDhcfjNe2aPBZtKPRbvecnksenTJHaP rrdXmDxOzPjN4rH4xQcmj8+b5AI4orhsUlJzMstSi/TtErgydj5exVjww6Bidc9qxgbGE6pd jBwcEgImEhN2GHcxcnKwCKhIXDv/hR3EZhNQl7hx4ycziC0ioCHxaeFyoDgXB7PAWkaJI1db WEF6hQVSJY5vDwCp4RWwkJjzfzkLSI2QwD0mib0/JjNCJAQlTs58wgJiMwtoSdz495IJpJdZ QFpi+T8OkDCngKXE9kP7wcpFBZQlDmw7zjSBkXcWku5ZSLpnIXQvYGRexSiSmVeWm5iZY6pX nJ1RmZdZoZecn7uJERi4y2r/TNzB+OWy+yFGAQ5GJR7eGzsT04RYE8uKK3MPMUpwMCuJ8PoV x6cJ8aYkVlalFuXHF5XmpBYfYpTmYFES5/UKT00QEkhPLEnNTk0tSC2CyTJxcEo1MJY/VAyJ spyh/rONuf/y88beih4N44Crtq5dIaaH58gHLU87euSWIOt88TLv1wuub32h8msB4y2Zs9Ur 2a2/zFOvLkq79lHrVzZ7oVtJ8Yo30p9kwp6u21SSVXjP2ffKkumrLTZ8eObv81P797OiLXaM HH2GH4uSF7Iwd+geig048ofVokntphJLcUaioRZzUXEiAFe+ZiZYAgAA X-CFilter-Loop: Reflected X-Stat-Signature: 4rnc9totbis6ctc4gyben9t11ou8dguz X-Rspamd-Queue-Id: 8DC3C8000C X-Rspam-User: X-Rspamd-Server: rspam01 X-HE-Tag: 1717744356-105978 X-HE-Meta: U2FsdGVkX19Wvz9vnvOqNDKJNSQfXMyiM/QFGppbaEKAuaTOGquWm+64j6e3ATy5M1iBYEBd9yeajQvXZJba2lANl7hWrgQkeWz0gHRn/Z2AIzzlSt4qwLZ4oleIIv/W53myEAwhKmjTrpb5CxmFyCE88jc3ZyR18KjvH12YXI8EW9XwTu6Aa5o5hUZdhb4RcyypUFbe4P8PwSm/dL1rqyntZMAABkK5Uhz/ffdCjNCaryu54Ode71Q551js9mtiY5Lu92Kc1aamvJjaAFSzR/Rw2Apv3P3gBAm8b7p0peGv8P1nbViyDR9yrhtfPcYpCh9GDGPU1naKiW+JIE7WKlO5P1sv/auzaKQlWek65kzK5M1bpL4zaFOo2T9nVhM6rjXSngH5woLnneJyPioMSWvF5L9PG+F/AY4/WlPF1HUysbT9/g2UBhnjuoLiisg/z67hN7LP1Z22yj/6ECLgCgNZzN4gssHikA3Gb1fvKeYlnEncEtmiQbGiosSWEMjcNsKe2EogA/QBlvMStmcvJKOM48eP7pLBQuFDxh80QA4tsGq/Sm99gcm+soJLg2m8QI/yMgy+6OGBPmQyMkAUKr9u4S1hQtNbDHVqKe3FNR0LZ4/hhamRp2VkH7uZIK02X3liksGDY5t4rlNIIKDw3YvkDh4JnBg0LSQiY7b/W112F0QN3FKhxdJKiqCYW2rThO60mpgwTOop3IhZgWgx6DKhyAd6WxJi4xR5zEYEkMGHcCbGJX7FQDJ5JulS80yb+q311+WE8Q0pOmx+LcNndDpYe5jcn9lhXfePpxIA7pJMTTWVH0IKM/0cWqAe1MEujRyptq3+ctfZuA7Ou4hEM69gj9HLf2w+aUg15Dn6713MKfbFkUSolw5psbec5yR/JEOtWOJmgmuj8hNvG0ahUab3oyhVgh+xavGM6bfgR/DxYKvzsWjUGSws/UUwLHApCz+Ktq5ihyU+jK6tbMK UF4qbSGk iWzlK7P+jzlTOJHT3/0tWd/AZTm04M8M295cpkV7WaR0OzPzE4lH+T+PrsA== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: On Wed, Jun 05, 2024 at 11:19:02AM +0900, Byungchul Park wrote: > On Wed, Jun 05, 2024 at 10:02:07AM +0800, Huang, Ying wrote: > > Byungchul Park writes: > > > > > On Tue, Jun 04, 2024 at 04:57:17PM +0800, Huang, Ying wrote: > > >> Byungchul Park writes: > > >> > > >> > On Tue, Jun 04, 2024 at 03:57:54PM +0800, Huang, Ying wrote: > > >> >> Byungchul Park writes: > > >> >> > > >> >> > Changes from v1: > > >> >> > 1. Don't allow to resume kswapd if the system is under memory > > >> >> > pressure that might affect direct reclaim by any chance, like > > >> >> > if NR_FREE_PAGES is less than (low wmark + min wmark)/2. > > >> >> > > > >> >> > --->8--- > > >> >> > From 6c73fc16b75907f5da9e6b33aff86bf7d7c9dd64 Mon Sep 17 00:00:00 2001 > > >> >> > From: Byungchul Park > > >> >> > Date: Tue, 4 Jun 2024 15:27:56 +0900 > > >> >> > Subject: [PATCH v2] mm: let kswapd work again for node that used to be hopeless but may not now > > >> >> > > > >> >> > A system should run with kswapd running in background when under memory > > >> >> > pressure, such as when the available memory level is below the low water > > >> >> > mark and there are reclaimable folios. > > >> >> > > > >> >> > However, the current code let the system run with kswapd stopped if > > >> >> > kswapd has been stopped due to more than MAX_RECLAIM_RETRIES failures > > >> >> > until direct reclaim will do for that, even if there are reclaimable > > >> >> > folios that can be reclaimed by kswapd. This case was observed in the > > >> >> > following scenario: > > >> >> > > > >> >> > CONFIG_NUMA_BALANCING enabled > > >> >> > sysctl_numa_balancing_mode set to NUMA_BALANCING_MEMORY_TIERING > > >> >> > numa node0 (500GB local DRAM, 128 CPUs) > > >> >> > numa node1 (100GB CXL memory, no CPUs) > > >> >> > swap off > > >> >> > > > >> >> > 1) Run a workload with big anon pages e.g. mmap(200GB). > > >> >> > 2) Continue adding the same workload to the system. > > >> >> > 3) The anon pages are placed in node0 by promotion/demotion. > > >> >> > 4) kswapd0 stops because of the unreclaimable anon pages in node0. > > >> >> > 5) Kill the memory hoggers to restore the system. > > >> >> > > > >> >> > After restoring the system at 5), the system starts to run without > > >> >> > kswapd. Even worse, tiering mechanism is no longer able to work since > > >> >> > the mechanism relies on kswapd for demotion. > > >> >> > > >> >> We have run into the situation that kswapd is kept in failure state for > > >> >> long in a multiple tiers system. I think that your solution is too > > >> > > > >> > My solution just gives a chance for kswapd to work again even if > > >> > kswapd_failures >= MAX_RECLAIM_RETRIES, if there are potential > > >> > reclaimable folios. That's it. > > >> > > > >> >> limited, because OOM killing may not happen, while the access pattern of > > >> > > > >> > I don't get this. OOM will happen as is, through direct reclaim. > > >> > > >> A system that fails to reclaim via kswapd may succeed to reclaim via > > >> direct reclaim, because more CPUs are used to scanning the page tables. > > > > > > Honestly, I don't think so with this description. > > > > > > The fact that the system hit MAX_RECLAIM_RETRIES means the system is > > > currently hopeless unless reclaiming folios in a stronger way by *direct > > > reclaim*. The solution for this situation should not be about letting > > > more CPUs particiated in reclaiming, again, *at least in this situation*. > > > > > > What you described here is true only in a normal state where the more > > > CPUs work on reclaiming, the more reclaimable folios can be reclaimed. > > > kswapd can be a helper *only* when there are kswapd-reclaimable folios. > > > > Sometimes, we cannot reclaim just because we doesn't scan fast enough so > > the Accessed-bit is set again during scanning. With more CPUs, we can > > scan faster, so make some progress. But, yes, this only cover one > > situation, there are other situations too. > > What I mean is *the issue we try to solve* is not the situation that > can be solved by letting more CPUs participate in reclaiming. Again, in the situation where kswapd has failed more than MAX_RECLAIM_RETRIES, say, holeless, I don't think it makes sense to wake up kswapd every 10 seconds. It'd be more sensible to wake up kwapd only if there are *at least potentially* reclaimable folios. As Ying said, there's no way to precisely track if reclaimable, but it's worth trying when the possibility becomes positive and looks more reasonable. Thoughts? Byungchul > Byungchul > > > -- > > Best Regards, > > Huang, Ying > > > > > Byungchul > > > > > >> In a system with NUMA balancing based page promotion and page demotion > > >> enabled, page promotion will wake up kswapd, but kswapd may fail in some > > >> situations. But page promotion will no trigger direct reclaim or OOM. > > >> > > >> >> the workloads may change. We have a preliminary and simple solution for > > >> >> this as follows, > > >> >> > > >> >> https://git.kernel.org/pub/scm/linux/kernel/git/vishal/tiering.git/commit/?h=tiering-0.8&id=17a24a354e12d4d4675d78481b358f668d5a6866 > > >> > > > >> > Whether tiering is involved or not, the same problem can arise if > > >> > kswapd gets stopped due to kswapd_failures >= MAX_RECLAIM_RETRIES. > > >> > > >> Your description is about tiering too. Can you describe a situation > > >> without tiering? > > >> > > >> -- > > >> Best Regards, > > >> Huang, Ying > > >> > > >> > Byungchul > > >> > > > >> >> where we will try to wake up kswapd to check every 10 seconds if kswapd > > >> >> is in failure state. This is another possible solution. > > >> >> > > >> >> > However, the node0 has pages newly allocated after 5), that might or > > >> >> > might not be reclaimable. Since those are potentially reclaimable, it's > > >> >> > worth hopefully trying reclaim by allowing kswapd to work again. > > >> >> > > > >> >> > > >> >> [snip] > > >> >> > > >> >> -- > > >> >> Best Regards, > > >> >> Huang, Ying