From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 15DC1191 for ; Mon, 7 Apr 2025 01:02:31 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1743987752; cv=none; b=YpVAq/san3Ond+gw0ecSjhyNQu9p8zhot+kdO5cPFYT1uAWz68E/T4rViC3kgD/RiRqSOGvl+VzgaUkGoOG8vsafuRK+4tVsUJx2Qev4AqeZk8waiNpZN8TUTEstB/0u2Yd1W4SCPMSVw8F1dbfngf/VI056g/pxpESXmG866jg= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1743987752; c=relaxed/simple; bh=ajb74bhY7wHfBrCdP2KvdQINaPs2tntTnrzpYHvW9Wk=; h=Date:To:From:Subject:Message-Id; b=GcencdSLpQhnofcKANvAgvCJG4WpsZNWlCEy8zkx7MCmY7oD3JQrgsuD21QmKp9K99VwZNDjIOdrJC/IFnXoKhmQaq/zcIR6raDYsB+o5Z5AJs5i1OmgI57K09jcpv40ZGhkzz+Jdd3O8nBVgh11nwoZeSv1eqqR7BqCrSwPGWE= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=linux-foundation.org header.i=@linux-foundation.org header.b=RnEUzWY8; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=linux-foundation.org header.i=@linux-foundation.org header.b="RnEUzWY8" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 41050C4CEE3; Mon, 7 Apr 2025 01:02:31 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=linux-foundation.org; s=korg; t=1743987751; bh=ajb74bhY7wHfBrCdP2KvdQINaPs2tntTnrzpYHvW9Wk=; h=Date:To:From:Subject:From; b=RnEUzWY8m0d+efEqDspxwMvg0G07sqjB8OGaBIXAIxbWZyad0jNScgF7iUST/vy2S L7KhNOPQeTUfDBCVc15aEKUF9SF+7+2EZkYYFhcilSJqo+WuE04sPbOaBM5ICkgVsT hRJvOcgyotBX2P/VheGeurJ6a4SQSojym7boEoZk= Date: Sun, 06 Apr 2025 18:02:30 -0700 To: mm-commits@vger.kernel.org,yuzhao@google.com,shakeel.butt@linux.dev,riel@surriel.com,mgorman@techsingularity.net,liuye@kylinos.cn,hughd@google.com,vny@google.com,akpm@linux-foundation.org From: Andrew Morton Subject: [to-be-updated] mm-vmscan-batch-tlb-flush-during-memory-reclaim.patch removed from -mm tree Message-Id: <20250407010231.41050C4CEE3@smtp.kernel.org> Precedence: bulk X-Mailing-List: mm-commits@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: The quilt patch titled Subject: mm/vmscan: batch TLB flush during memory reclaim has been removed from the -mm tree. Its filename was mm-vmscan-batch-tlb-flush-during-memory-reclaim.patch This patch was dropped because an updated version will be issued ------------------------------------------------------ From: Vinay Banakar Subject: mm/vmscan: batch TLB flush during memory reclaim Date: Fri, 28 Mar 2025 14:20:55 -0400 The current implementation in shrink_folio_list() performs a full TLB flush for every individual folio reclaimed. This causes unnecessary overhead during memory reclaim. The current code: 1. Clears PTEs and unmaps each page individually 2. Performs a full TLB flush on every CPU the mm is running on The new code: 1. Clears PTEs and unmaps each page individually 2. Adds each unmapped page to pageout_folios 3. Flushes the TLB once before procesing pageout_folios This reduces the number of TLB flushes issued by the memory reclaim code by 1/N, where N is the number of mapped folios encountered in the batch processed by shrink_folio_list. [riel@surriel.com: forward port to 6.14, adjust code and naming to match surrounding code] Link: https://lkml.kernel.org/r/20250328142055.313916d1@fangorn Signed-off-by: Vinay Banakar Signed-off-by: Rik van Riel Cc: Hugh Dickins Cc: Liu Ye Cc: Mel Gorman Cc: Shakeel Butt Cc: Yu Zhao Signed-off-by: Andrew Morton --- mm/vmscan.c | 112 ++++++++++++++++++++++++++++++-------------------- 1 file changed, 68 insertions(+), 44 deletions(-) --- a/mm/vmscan.c~mm-vmscan-batch-tlb-flush-during-memory-reclaim +++ a/mm/vmscan.c @@ -1101,6 +1101,7 @@ static unsigned int shrink_folio_list(st struct folio_batch free_folios; LIST_HEAD(ret_folios); LIST_HEAD(demote_folios); + LIST_HEAD(pageout_folios); unsigned int nr_reclaimed = 0, nr_demoted = 0; unsigned int pgactivate = 0; bool do_demote_pass; @@ -1428,51 +1429,10 @@ retry: goto keep_locked; /* - * Folio is dirty. Flush the TLB if a writable entry - * potentially exists to avoid CPU writes after I/O - * starts and then write it out here. + * Add to pageout list for batched TLB flushing and IO submission. */ - try_to_unmap_flush_dirty(); - switch (pageout(folio, mapping, &plug, folio_list)) { - case PAGE_KEEP: - goto keep_locked; - case PAGE_ACTIVATE: - /* - * If shmem folio is split when writeback to swap, - * the tail pages will make their own pass through - * this function and be accounted then. - */ - if (nr_pages > 1 && !folio_test_large(folio)) { - sc->nr_scanned -= (nr_pages - 1); - nr_pages = 1; - } - goto activate_locked; - case PAGE_SUCCESS: - if (nr_pages > 1 && !folio_test_large(folio)) { - sc->nr_scanned -= (nr_pages - 1); - nr_pages = 1; - } - stat->nr_pageout += nr_pages; - - if (folio_test_writeback(folio)) - goto keep; - if (folio_test_dirty(folio)) - goto keep; - - /* - * A synchronous write - probably a ramdisk. Go - * ahead and try to reclaim the folio. - */ - if (!folio_trylock(folio)) - goto keep; - if (folio_test_dirty(folio) || - folio_test_writeback(folio)) - goto keep_locked; - mapping = folio_mapping(folio); - fallthrough; - case PAGE_CLEAN: - ; /* try to free the folio below */ - } + list_add(&folio->lru, &pageout_folios); + continue; } /* @@ -1583,6 +1543,70 @@ keep: } /* 'folio_list' is always empty here */ + if (!list_empty(&pageout_folios)) { + /* + * The loop above unmapped the folios from the page tables. + * One TLB flush takes care of the whole batch. + */ + try_to_unmap_flush_dirty(); + + while (!list_empty(&pageout_folios)) { + struct folio *folio = lru_to_folio(&pageout_folios); + struct address_space *mapping; + list_del(&folio->lru); + + /* Recheck if the page got reactivated */ + if (folio_test_active(folio) || folio_mapped(folio)) + goto skip_pageout_locked; + + mapping = folio_mapping(folio); + switch (pageout(folio, mapping, &plug, &pageout_folios)) { + case PAGE_KEEP: + case PAGE_ACTIVATE: + goto skip_pageout_locked; + case PAGE_SUCCESS: + /* + * If shmem folio is split when writeback to swap, + * the tail pages will make their own pass through + * this loop and be accounted then. + */ + stat->nr_pageout += folio_nr_pages(folio); + + if (folio_test_writeback(folio)) + goto skip_pageout; + if (folio_test_dirty(folio)) + goto skip_pageout; + + /* + * A synchronous write - probably a ramdisk. Go + * ahead and try to reclaim the folio. + */ + if (!folio_trylock(folio)) + goto skip_pageout; + if (folio_test_dirty(folio) || + folio_test_writeback(folio)) + goto skip_pageout_locked; + mapping = folio_mapping(folio); + /* try to free the folio below */ + fallthrough; + case PAGE_CLEAN: + /* try to free the folio */ + if (!mapping || + !remove_mapping(mapping, folio)) + goto skip_pageout_locked; + + nr_reclaimed += folio_nr_pages(folio); + folio_unlock(folio); + continue; + } + +skip_pageout_locked: + folio_unlock(folio); +skip_pageout: + list_add(&folio->lru, &ret_folios); + } + } + /* Migrate folios selected for demotion */ nr_demoted = demote_folio_list(&demote_folios, pgdat); nr_reclaimed += nr_demoted; _ Patches currently in -mm which might be from vny@google.com are