From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 525F2C6FD1F for ; Wed, 8 Mar 2023 01:05:49 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229795AbjCHBFr (ORCPT ); Tue, 7 Mar 2023 20:05:47 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:43334 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229812AbjCHBFk (ORCPT ); Tue, 7 Mar 2023 20:05:40 -0500 Received: from ams.source.kernel.org (ams.source.kernel.org [IPv6:2604:1380:4601:e00::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id B2D44A4B2C for ; Tue, 7 Mar 2023 17:05:37 -0800 (PST) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ams.source.kernel.org (Postfix) with ESMTPS id 7332DB81B31 for ; Wed, 8 Mar 2023 01:05:36 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 28204C4339C; Wed, 8 Mar 2023 01:05:35 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=linux-foundation.org; s=korg; t=1678237535; bh=AoR1yB+6kw5MG07W2O7EMPIZnaFUR6nSaGVTiFEFFhs=; h=Date:To:From:Subject:From; b=2tRsJzdOIxjQYHSS3VgUQ7A+N0BDwkDg5qKBnw3GxWpRLcFJgrr8S/7U01UZennR2 TyP5VgdGXqyiwVGTrZV7eech5jJXyQSdPUSFM+lLi7Tk1jcF735lWUY3AQqUQj3qXh EyXvSgAZYDHIpmBFqWu2J7/KlQg7tX6uU3saDz/A= Date: Tue, 07 Mar 2023 17:05:34 -0800 To: mm-commits@vger.kernel.org, ziy@nvidia.com, xhao@linux.alibaba.com, willy@infradead.org, tj@kernel.org, shy828301@gmail.com, shr@devkernel.io, pengfei.xu@intel.com, mike.kravetz@oracle.com, hughd@google.com, hch@lst.de, baolin.wang@linux.alibaba.com, ying.huang@intel.com, akpm@linux-foundation.org From: Andrew Morton Subject: [merged mm-hotfixes-stable] migrate_pages-try-migrate-in-batch-asynchronously-firstly.patch removed from -mm tree Message-Id: <20230308010535.28204C4339C@smtp.kernel.org> Precedence: bulk Reply-To: linux-kernel@vger.kernel.org List-ID: X-Mailing-List: mm-commits@vger.kernel.org The quilt patch titled Subject: migrate_pages: try migrate in batch asynchronously firstly has been removed from the -mm tree. Its filename was migrate_pages-try-migrate-in-batch-asynchronously-firstly.patch This patch was dropped because it was merged into the mm-hotfixes-stable branch of git://git.kernel.org/pub/scm/linux/kernel/git/akpm/mm ------------------------------------------------------ From: Huang Ying Subject: migrate_pages: try migrate in batch asynchronously firstly Date: Fri, 3 Mar 2023 11:01:55 +0800 When we have locked more than one folios, we cannot wait the lock or bit (e.g., page lock, buffer head lock, writeback bit) synchronously. Otherwise deadlock may be triggered. This make it hard to batch the synchronous migration directly. This patch re-enables batching synchronous migration via trying to migrate in batch asynchronously firstly. And any folios that are failed to be migrated asynchronously will be migrated synchronously one by one. Test shows that this can restore the TLB flushing batching performance for synchronous migration effectively. Link: https://lkml.kernel.org/r/20230303030155.160983-4-ying.huang@intel.com Fixes: 5dfab109d519 ("migrate_pages: batch _unmap and _move") Signed-off-by: "Huang, Ying" Tested-by: Hugh Dickins Reviewed-by: Baolin Wang Cc: "Xu, Pengfei" Cc: Christoph Hellwig Cc: Stefan Roesch Cc: Tejun Heo Cc: Xin Hao Cc: Zi Yan Cc: Yang Shi Cc: Matthew Wilcox Cc: Mike Kravetz Signed-off-by: Andrew Morton --- --- a/mm/migrate.c~migrate_pages-try-migrate-in-batch-asynchronously-firstly +++ a/mm/migrate.c @@ -1112,9 +1112,8 @@ static void migrate_folio_done(struct fo /* Obtain the lock on page, remove all ptes. */ static int migrate_folio_unmap(new_page_t get_new_page, free_page_t put_new_page, unsigned long private, struct folio *src, - struct folio **dstp, int force, - enum migrate_mode mode, enum migrate_reason reason, - struct list_head *ret) + struct folio **dstp, enum migrate_mode mode, + enum migrate_reason reason, struct list_head *ret) { struct folio *dst; int rc = -EAGAIN; @@ -1144,7 +1143,7 @@ static int migrate_folio_unmap(new_page_ dst->private = NULL; if (!folio_trylock(src)) { - if (!force || mode == MIGRATE_ASYNC) + if (mode == MIGRATE_ASYNC) goto out; /* @@ -1182,8 +1181,6 @@ static int migrate_folio_unmap(new_page_ rc = -EBUSY; goto out; } - if (!force) - goto out; folio_wait_writeback(src); } @@ -1497,6 +1494,9 @@ static inline int try_split_folio(struct #define NR_MAX_BATCHED_MIGRATION 512 #endif #define NR_MAX_MIGRATE_PAGES_RETRY 10 +#define NR_MAX_MIGRATE_ASYNC_RETRY 3 +#define NR_MAX_MIGRATE_SYNC_RETRY \ + (NR_MAX_MIGRATE_PAGES_RETRY - NR_MAX_MIGRATE_ASYNC_RETRY) struct migrate_pages_stats { int nr_succeeded; /* Normal and large folios migrated successfully, in @@ -1678,8 +1678,7 @@ static int migrate_pages_batch(struct li } rc = migrate_folio_unmap(get_new_page, put_new_page, private, - folio, &dst, pass > 2, mode, - reason, ret_folios); + folio, &dst, mode, reason, ret_folios); /* * The rules are: * Success: folio will be freed @@ -1857,6 +1856,51 @@ out: return rc; } +static int migrate_pages_sync(struct list_head *from, new_page_t get_new_page, + free_page_t put_new_page, unsigned long private, + enum migrate_mode mode, int reason, struct list_head *ret_folios, + struct list_head *split_folios, struct migrate_pages_stats *stats) +{ + int rc, nr_failed = 0; + LIST_HEAD(folios); + struct migrate_pages_stats astats; + + memset(&astats, 0, sizeof(astats)); + /* Try to migrate in batch with MIGRATE_ASYNC mode firstly */ + rc = migrate_pages_batch(from, get_new_page, put_new_page, private, MIGRATE_ASYNC, + reason, &folios, split_folios, &astats, + NR_MAX_MIGRATE_ASYNC_RETRY); + stats->nr_succeeded += astats.nr_succeeded; + stats->nr_thp_succeeded += astats.nr_thp_succeeded; + stats->nr_thp_split += astats.nr_thp_split; + if (rc < 0) { + stats->nr_failed_pages += astats.nr_failed_pages; + stats->nr_thp_failed += astats.nr_thp_failed; + list_splice_tail(&folios, ret_folios); + return rc; + } + stats->nr_thp_failed += astats.nr_thp_split; + nr_failed += astats.nr_thp_split; + /* + * Fall back to migrate all failed folios one by one synchronously. All + * failed folios except split THPs will be retried, so their failure + * isn't counted + */ + list_splice_tail_init(&folios, from); + while (!list_empty(from)) { + list_move(from->next, &folios); + rc = migrate_pages_batch(&folios, get_new_page, put_new_page, + private, mode, reason, ret_folios, + split_folios, stats, NR_MAX_MIGRATE_SYNC_RETRY); + list_splice_tail_init(&folios, ret_folios); + if (rc < 0) + return rc; + nr_failed += rc; + } + + return nr_failed; +} + /* * migrate_pages - migrate the folios specified in a list, to the free folios * supplied as the target for the page migration @@ -1888,7 +1932,7 @@ int migrate_pages(struct list_head *from enum migrate_mode mode, int reason, unsigned int *ret_succeeded) { int rc, rc_gather; - int nr_pages, batch; + int nr_pages; struct folio *folio, *folio2; LIST_HEAD(folios); LIST_HEAD(ret_folios); @@ -1904,10 +1948,6 @@ int migrate_pages(struct list_head *from if (rc_gather < 0) goto out; - if (mode == MIGRATE_ASYNC) - batch = NR_MAX_BATCHED_MIGRATION; - else - batch = 1; again: nr_pages = 0; list_for_each_entry_safe(folio, folio2, from, lru) { @@ -1918,16 +1958,20 @@ again: } nr_pages += folio_nr_pages(folio); - if (nr_pages >= batch) + if (nr_pages >= NR_MAX_BATCHED_MIGRATION) break; } - if (nr_pages >= batch) + if (nr_pages >= NR_MAX_BATCHED_MIGRATION) list_cut_before(&folios, from, &folio2->lru); else list_splice_init(from, &folios); - rc = migrate_pages_batch(&folios, get_new_page, put_new_page, private, - mode, reason, &ret_folios, &split_folios, &stats, - NR_MAX_MIGRATE_PAGES_RETRY); + if (mode == MIGRATE_ASYNC) + rc = migrate_pages_batch(&folios, get_new_page, put_new_page, private, + mode, reason, &ret_folios, &split_folios, &stats, + NR_MAX_MIGRATE_PAGES_RETRY); + else + rc = migrate_pages_sync(&folios, get_new_page, put_new_page, private, + mode, reason, &ret_folios, &split_folios, &stats); list_splice_tail_init(&folios, &ret_folios); if (rc < 0) { rc_gather = rc; _ Patches currently in -mm which might be from ying.huang@intel.com are