From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id ADC7AC433F5 for ; Fri, 22 Oct 2021 14:47:25 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 4732D60F50 for ; Fri, 22 Oct 2021 14:47:25 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.4.1 mail.kernel.org 4732D60F50 Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=techsingularity.net Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=kvack.org Received: by kanga.kvack.org (Postfix) id A1BD7900003; Fri, 22 Oct 2021 10:47:24 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 9CB5D900002; Fri, 22 Oct 2021 10:47:24 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 8BAE1900003; Fri, 22 Oct 2021 10:47:24 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0160.hostedemail.com [216.40.44.160]) by kanga.kvack.org (Postfix) with ESMTP id 7DE25900002 for ; Fri, 22 Oct 2021 10:47:24 -0400 (EDT) Received: from smtpin38.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay04.hostedemail.com (Postfix) with ESMTP id 416E63947E for ; Fri, 22 Oct 2021 14:47:24 +0000 (UTC) X-FDA: 78724351608.38.92749A0 Received: from outbound-smtp62.blacknight.com (outbound-smtp62.blacknight.com [46.22.136.251]) by imf16.hostedemail.com (Postfix) with ESMTP id 40560F000095 for ; Fri, 22 Oct 2021 14:47:20 +0000 (UTC) Received: from mail.blacknight.com (pemlinmail03.blacknight.ie [81.17.254.16]) by outbound-smtp62.blacknight.com (Postfix) with ESMTPS id 901DAFB3E7 for ; Fri, 22 Oct 2021 15:47:22 +0100 (IST) Received: (qmail 29677 invoked from network); 22 Oct 2021 14:47:22 -0000 Received: from unknown (HELO stampy.112glenside.lan) (mgorman@techsingularity.net@[84.203.17.29]) by 81.17.254.9 with ESMTPA; 22 Oct 2021 14:47:22 -0000 From: Mel Gorman To: Andrew Morton Cc: NeilBrown , Theodore Ts'o , Andreas Dilger , "Darrick J . Wong" , Matthew Wilcox , Michal Hocko , Dave Chinner , Rik van Riel , Vlastimil Babka , Johannes Weiner , Jonathan Corbet , Linux-MM , Linux-fsdevel , LKML , Mel Gorman Subject: [PATCH 2/8] mm/vmscan: Throttle reclaim and compaction when too may pages are isolated Date: Fri, 22 Oct 2021 15:46:45 +0100 Message-Id: <20211022144651.19914-3-mgorman@techsingularity.net> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20211022144651.19914-1-mgorman@techsingularity.net> References: <20211022144651.19914-1-mgorman@techsingularity.net> MIME-Version: 1.0 X-Rspamd-Server: rspam05 X-Rspamd-Queue-Id: 40560F000095 X-Stat-Signature: zohfeijug9se9jtg74uk95tontwthk8r Authentication-Results: imf16.hostedemail.com; dkim=none; spf=pass (imf16.hostedemail.com: domain of mgorman@techsingularity.net designates 46.22.136.251 as permitted sender) smtp.mailfrom=mgorman@techsingularity.net; dmarc=none X-HE-Tag: 1634914040-834216 Content-Transfer-Encoding: quoted-printable X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Page reclaim throttles on congestion if too many parallel reclaim instanc= es have isolated too many pages. This makes no sense, excessive parallelisat= ion has nothing to do with writeback or congestion. This patch creates an additional workqueue to sleep on when too many pages are isolated. The throttled tasks are woken when the number of isolated pages is reduced or a timeout occurs. There may be some false positive wakeups for GFP_NOIO/GFP_NOFS callers but the tasks will throttle again if necessary. [shy828301@gmail.com: Wake up from compaction context] [vbabka@suse.cz: Account number of throttled tasks only for writeback] Signed-off-by: Mel Gorman Acked-by: Vlastimil Babka --- include/linux/mmzone.h | 1 + include/trace/events/vmscan.h | 4 +++- mm/compaction.c | 10 ++++++++-- mm/internal.h | 11 +++++++++++ mm/vmscan.c | 22 ++++++++++++++++------ 5 files changed, 39 insertions(+), 9 deletions(-) diff --git a/include/linux/mmzone.h b/include/linux/mmzone.h index 419304093610..9ccd8d95291b 100644 --- a/include/linux/mmzone.h +++ b/include/linux/mmzone.h @@ -275,6 +275,7 @@ enum lru_list { =20 enum vmscan_throttle_state { VMSCAN_THROTTLE_WRITEBACK, + VMSCAN_THROTTLE_ISOLATED, NR_VMSCAN_THROTTLE, }; =20 diff --git a/include/trace/events/vmscan.h b/include/trace/events/vmscan.= h index c317f9fe0d17..d4905bd9e9c4 100644 --- a/include/trace/events/vmscan.h +++ b/include/trace/events/vmscan.h @@ -28,10 +28,12 @@ ) : "RECLAIM_WB_NONE" =20 #define _VMSCAN_THROTTLE_WRITEBACK (1 << VMSCAN_THROTTLE_WRITEBACK) +#define _VMSCAN_THROTTLE_ISOLATED (1 << VMSCAN_THROTTLE_ISOLATED) =20 #define show_throttle_flags(flags) \ (flags) ? __print_flags(flags, "|", \ - {_VMSCAN_THROTTLE_WRITEBACK, "VMSCAN_THROTTLE_WRITEBACK"} \ + {_VMSCAN_THROTTLE_WRITEBACK, "VMSCAN_THROTTLE_WRITEBACK"}, \ + {_VMSCAN_THROTTLE_ISOLATED, "VMSCAN_THROTTLE_ISOLATED"} \ ) : "VMSCAN_THROTTLE_NONE" =20 =20 diff --git a/mm/compaction.c b/mm/compaction.c index bfc93da1c2c7..7359093d8ac0 100644 --- a/mm/compaction.c +++ b/mm/compaction.c @@ -761,6 +761,8 @@ isolate_freepages_range(struct compact_control *cc, /* Similar to reclaim, but different enough that they don't share logic = */ static bool too_many_isolated(pg_data_t *pgdat) { + bool too_many; + unsigned long active, inactive, isolated; =20 inactive =3D node_page_state(pgdat, NR_INACTIVE_FILE) + @@ -770,7 +772,11 @@ static bool too_many_isolated(pg_data_t *pgdat) isolated =3D node_page_state(pgdat, NR_ISOLATED_FILE) + node_page_state(pgdat, NR_ISOLATED_ANON); =20 - return isolated > (inactive + active) / 2; + too_many =3D isolated > (inactive + active) / 2; + if (!too_many) + wake_throttle_isolated(pgdat); + + return too_many; } =20 /** @@ -822,7 +828,7 @@ isolate_migratepages_block(struct compact_control *cc= , unsigned long low_pfn, if (cc->mode =3D=3D MIGRATE_ASYNC) return -EAGAIN; =20 - congestion_wait(BLK_RW_ASYNC, HZ/10); + reclaim_throttle(pgdat, VMSCAN_THROTTLE_ISOLATED, HZ/10); =20 if (fatal_signal_pending(current)) return -EINTR; diff --git a/mm/internal.h b/mm/internal.h index b495e60c955d..c72d3383ef34 100644 --- a/mm/internal.h +++ b/mm/internal.h @@ -45,6 +45,15 @@ static inline void acct_reclaim_writeback(struct page = *page) __acct_reclaim_writeback(pgdat, page, nr_throttled); } =20 +static inline void wake_throttle_isolated(pg_data_t *pgdat) +{ + wait_queue_head_t *wqh; + + wqh =3D &pgdat->reclaim_wait[VMSCAN_THROTTLE_ISOLATED]; + if (waitqueue_active(wqh)) + wake_up(wqh); +} + vm_fault_t do_swap_page(struct vm_fault *vmf); =20 void free_pgtables(struct mmu_gather *tlb, struct vm_area_struct *start_= vma, @@ -120,6 +129,8 @@ extern unsigned long highest_memmap_pfn; */ extern int isolate_lru_page(struct page *page); extern void putback_lru_page(struct page *page); +extern void reclaim_throttle(pg_data_t *pgdat, enum vmscan_throttle_stat= e reason, + long timeout); =20 /* * in mm/rmap.c: diff --git a/mm/vmscan.c b/mm/vmscan.c index 0c1595065f49..1e54e636b927 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -1006,12 +1006,12 @@ static void handle_write_error(struct address_spa= ce *mapping, unlock_page(page); } =20 -static void -reclaim_throttle(pg_data_t *pgdat, enum vmscan_throttle_state reason, +void reclaim_throttle(pg_data_t *pgdat, enum vmscan_throttle_state reaso= n, long timeout) { wait_queue_head_t *wqh =3D &pgdat->reclaim_wait[reason]; long ret; + bool acct_writeback =3D (reason =3D=3D VMSCAN_THROTTLE_WRITEBACK); DEFINE_WAIT(wait); =20 /* @@ -1023,7 +1023,8 @@ reclaim_throttle(pg_data_t *pgdat, enum vmscan_thro= ttle_state reason, current->flags & (PF_IO_WORKER|PF_KTHREAD)) return; =20 - if (atomic_inc_return(&pgdat->nr_writeback_throttled) =3D=3D 1) { + if (acct_writeback && + atomic_inc_return(&pgdat->nr_writeback_throttled) =3D=3D 1) { WRITE_ONCE(pgdat->nr_reclaim_start, node_page_state(pgdat, NR_THROTTLED_WRITTEN)); } @@ -1031,7 +1032,9 @@ reclaim_throttle(pg_data_t *pgdat, enum vmscan_thro= ttle_state reason, prepare_to_wait(wqh, &wait, TASK_UNINTERRUPTIBLE); ret =3D schedule_timeout(timeout); finish_wait(wqh, &wait); - atomic_dec(&pgdat->nr_writeback_throttled); + + if (acct_writeback) + atomic_dec(&pgdat->nr_writeback_throttled); =20 trace_mm_vmscan_throttled(pgdat->node_id, jiffies_to_usecs(timeout), jiffies_to_usecs(timeout - ret), @@ -2176,6 +2179,7 @@ static int too_many_isolated(struct pglist_data *pg= dat, int file, struct scan_control *sc) { unsigned long inactive, isolated; + bool too_many; =20 if (current_is_kswapd()) return 0; @@ -2199,7 +2203,13 @@ static int too_many_isolated(struct pglist_data *p= gdat, int file, if ((sc->gfp_mask & (__GFP_IO | __GFP_FS)) =3D=3D (__GFP_IO | __GFP_FS)= ) inactive >>=3D 3; =20 - return isolated > inactive; + too_many =3D isolated > inactive; + + /* Wake up tasks throttled due to too_many_isolated. */ + if (!too_many) + wake_throttle_isolated(pgdat); + + return too_many; } =20 /* @@ -2308,8 +2318,8 @@ shrink_inactive_list(unsigned long nr_to_scan, stru= ct lruvec *lruvec, return 0; =20 /* wait a bit for the reclaimer. */ - msleep(100); stalled =3D true; + reclaim_throttle(pgdat, VMSCAN_THROTTLE_ISOLATED, HZ/10); =20 /* We are about to die and free our memory. Return now. */ if (fatal_signal_pending(current)) --=20 2.31.1