From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 82092C54E58 for ; Tue, 12 Mar 2024 08:45:28 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id EE5048D001F; Tue, 12 Mar 2024 04:45:27 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id E95D08D0017; Tue, 12 Mar 2024 04:45:27 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id D5C6F8D001F; Tue, 12 Mar 2024 04:45:27 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id C5F238D0017 for ; Tue, 12 Mar 2024 04:45:27 -0400 (EDT) Received: from smtpin04.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id 959BF80F9E for ; Tue, 12 Mar 2024 08:45:27 +0000 (UTC) X-FDA: 81887753094.04.8411034 Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by imf20.hostedemail.com (Postfix) with ESMTP id B71F11C0019 for ; Tue, 12 Mar 2024 08:45:25 +0000 (UTC) Authentication-Results: imf20.hostedemail.com; dkim=none; dmarc=pass (policy=none) header.from=arm.com; spf=pass (imf20.hostedemail.com: domain of ryan.roberts@arm.com designates 217.140.110.172 as permitted sender) smtp.mailfrom=ryan.roberts@arm.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1710233126; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=lPebCDu+FT3QhJI3pf2+jX7iu7UIxxScAuiurswtgMY=; b=Dz6VUH/0FMdxHwMLw2rsCuMJNDf2btPp3E8sk/z8JTmlPecH6e+s+8XRVVed2sUuVqN/3m xJwAkeQjlnCjJN0+OqmiTWrQaRUh26GDxGxe/KcJ+12vyIZFC5/Nu3IFeSBUzOXzd8AqZi qJsCfzAbIyQt+STLD7S6quBo29pWavw= ARC-Authentication-Results: i=1; imf20.hostedemail.com; dkim=none; dmarc=pass (policy=none) header.from=arm.com; spf=pass (imf20.hostedemail.com: domain of ryan.roberts@arm.com designates 217.140.110.172 as permitted sender) smtp.mailfrom=ryan.roberts@arm.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1710233126; a=rsa-sha256; cv=none; b=Wmf9fGouUJpBO4/O/i17Sz23QxqeNners8wOkA6h0UgINU4AKas4aXd6nInq8tiRbENP5p XrWpk9qCQGD22aLpAaBRjPm3a32CmUDQUOL86Dhqsl9KMS+GptT5OauSsXBKJhNLlvH+PF okiMzVI2sih+OAMpSxVsLHFILL5eV+Q= Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id ED2B11007; Tue, 12 Mar 2024 01:46:01 -0700 (PDT) Received: from [10.57.68.246] (unknown [10.57.68.246]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id 81D7E3F73F; Tue, 12 Mar 2024 01:45:22 -0700 (PDT) Message-ID: Date: Tue, 12 Mar 2024 08:45:20 +0000 MIME-Version: 1.0 User-Agent: Mozilla Thunderbird Subject: Re: [PATCH v4 0/6] Swap-out mTHP without splitting Content-Language: en-GB To: Andrew Morton , David Hildenbrand , Matthew Wilcox , Huang Ying , Gao Xiang , Yu Zhao , Yang Shi , Michal Hocko , Kefeng Wang , Barry Song <21cnbao@gmail.com>, Chris Li Cc: linux-mm@kvack.org, linux-kernel@vger.kernel.org References: <20240311150058.1122862-1-ryan.roberts@arm.com> From: Ryan Roberts In-Reply-To: <20240311150058.1122862-1-ryan.roberts@arm.com> Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 7bit X-Rspamd-Queue-Id: B71F11C0019 X-Rspam-User: X-Rspamd-Server: rspam05 X-Stat-Signature: 8y7xf9r3dkfbcuriohuphcahi5nhu7co X-HE-Tag: 1710233125-406882 X-HE-Meta: U2FsdGVkX1/cfIrg+zs6pY59qT9MtadXUER87NwIh67NDWpO7Lp/YgmGRQ3q3nzGlRzT4/2tMfjHd3mtD1otdDctwl+64KBxf9PhcWjk5wRAg6XpEI0CLqhaqfLJs0fzHuWWEXhoikfVzkw2iqpY9iI1/hWgsThJR5omgb24+s/lwQ3Z0zuSwvElli5rQfLFELYqy+OmmzsJEZk67VGLv5/Nqd0QbfpkyGtXD9TLJDUORuvpUPSrQ46sWncTc51YoEOEoKClDe0/w1O2bGVNncgSlhLB/XhAS981HBKENhn/naCu06n9mRaA5tazQkKE471FEONUu2Ckd135RLIqXTBdG+8YyI116K3Ik2EerpYHWTT+zq/0cakKKwDgAqGTZwHr13LhchHYoO0Cj09txDMbW/vuLCiUt2drsXYoo1LPyZ5SpoBhAf3ew8d5yjbzsTRMaFO6dCrWGkzuSoJhPqgOfRgOXcSx2QQTWGzB9ILPAZwabqfBlrCm83Ga78jzvnsbLvDce5yFT+mP36OJakd8WXHywl0Tytmvjq2QBEcUQlMz0xcqBFoEqj9nbBAjTKkSYa7xO6NHO+pbsCcFQNLMgSQbEAy3iWamiwWJJrLAfKkgu4rZkLXoSXGjgi888ID3MQswbG3j7PB27nP7aQnGNvzbnKKo+KYSS3NIkPDgrpQ5zdXEjQWXCOag62zt+3yh/Si9crkS50xOOuSn8oxqrUiL4iPf0wGnQJHXHTrdiwPMPUKOhdT6J21CBEcXj+fZarFgkoKsislv8JIJxg7nonkzpPa7+GRwqp6+uXp+rJ9SwekGrQOHtajJMt/YrSu0maZAYJIym5losNxQOpQU6gdb+Ixq X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: On 11/03/2024 15:00, Ryan Roberts wrote: > Hi All, > > This series adds support for swapping out multi-size THP (mTHP) without needing > to first split the large folio via split_huge_page_to_list_to_order(). It > closely follows the approach already used to swap-out PMD-sized THP. > > There are a couple of reasons for swapping out mTHP without splitting: > > - Performance: It is expensive to split a large folio and under extreme memory > pressure some workloads regressed performance when using 64K mTHP vs 4K > small folios because of this extra cost in the swap-out path. This series > not only eliminates the regression but makes it faster to swap out 64K mTHP > vs 4K small folios. > > - Memory fragmentation avoidance: If we can avoid splitting a large folio > memory is less likely to become fragmented, making it easier to re-allocate > a large folio in future. > > - Performance: Enables a separate series [4] to swap-in whole mTHPs, which > means we won't lose the TLB-efficiency benefits of mTHP once the memory has > been through a swap cycle. > > I've done what I thought was the smallest change possible, and as a result, this > approach is only employed when the swap is backed by a non-rotating block device > (just as PMD-sized THP is supported today). Discussion against the RFC concluded > that this is sufficient. > > > Performance Testing > =================== > > I've run some swap performance tests on Ampere Altra VM (arm64) with 8 CPUs. The > VM is set up with a 35G block ram device as the swap device and the test is run > from inside a memcg limited to 40G memory. I've then run `usemem` from > vm-scalability with 70 processes, each allocating and writing 1G of memory. I've > repeated everything 6 times and taken the mean performance improvement relative > to 4K page baseline: > > | alloc size | baseline | + this series | > | | v6.6-rc4+anonfolio | | Oops, just noticed I failed to update these column headers. The baseline is actually mm-unstable (d7182786dd0a) which is based on v6.8-rc5 and already contains "anonfolio" - now called mTHP. > |:-----------|--------------------:|--------------------:| > | 4K Page | 0.0% | 1.4% | > | 64K THP | -14.6% | 44.2% | > | 2M THP | 87.4% | 97.7% | > > So with this change, the 64K swap performance goes from a 15% regression to a > 44% improvement. 4K and 2M swap improves slightly too. > > This test also acts as a good stress test for swap and, more generally mm. A > couple of existing bugs were found as a result [5] [6]. > > > --- > The series applies against mm-unstable (d7182786dd0a). Although I've > additionally been running with a couple of extra fixes to avoid the issues at > [6]. > > > Changes since v3 [3] > ==================== > > - Renamed SWAP_NEXT_NULL -> SWAP_NEXT_INVALID (per Huang, Ying) > - Simplified max offset calculation (per Huang, Ying) > - Reinstated struct percpu_cluster to contain per-cluster, per-order `next` > offset (per Huang, Ying) > - Removed swap_alloc_large() and merged its functionality into > scan_swap_map_slots() (per Huang, Ying) > - Avoid extra cost of folio ref and lock due to removal of CLUSTER_FLAG_HUGE > by freeing swap entries in batches (see patch 2) (per DavidH) > - vmscan splits folio if its partially mapped (per Barry Song, DavidH) > - Avoid splitting in MADV_PAGEOUT path (per Barry Song) > - Dropped "mm: swap: Simplify ssd behavior when scanner steals entry" patch > since it's not actually a problem for THP as I first thought. > > > Changes since v2 [2] > ==================== > > - Reuse scan_swap_map_try_ssd_cluster() between order-0 and order > 0 > allocation. This required some refactoring to make everything work nicely > (new patches 2 and 3). > - Fix bug where nr_swap_pages would say there are pages available but the > scanner would not be able to allocate them because they were reserved for the > per-cpu allocator. We now allow stealing of order-0 entries from the high > order per-cpu clusters (in addition to exisiting stealing from order-0 > per-cpu clusters). > > > Changes since v1 [1] > ==================== > > - patch 1: > - Use cluster_set_count() instead of cluster_set_count_flag() in > swap_alloc_cluster() since we no longer have any flag to set. I was unable > to kill cluster_set_count_flag() as proposed against v1 as other call > sites depend explicitly setting flags to 0. > - patch 2: > - Moved large_next[] array into percpu_cluster to make it per-cpu > (recommended by Huang, Ying). > - large_next[] array is dynamically allocated because PMD_ORDER is not > compile-time constant for powerpc (fixes build error). > > > [1] https://lore.kernel.org/linux-mm/20231010142111.3997780-1-ryan.roberts@arm.com/ > [2] https://lore.kernel.org/linux-mm/20231017161302.2518826-1-ryan.roberts@arm.com/ > [3] https://lore.kernel.org/linux-mm/20231025144546.577640-1-ryan.roberts@arm.com/ > [4] https://lore.kernel.org/linux-mm/20240304081348.197341-1-21cnbao@gmail.com/ > [5] https://lore.kernel.org/linux-mm/20240311084426.447164-1-ying.huang@intel.com/ > [6] https://lore.kernel.org/linux-mm/79dad067-1d26-4867-8eb1-941277b9a77b@arm.com/ > > Thanks, > Ryan > > > Ryan Roberts (6): > mm: swap: Remove CLUSTER_FLAG_HUGE from swap_cluster_info:flags > mm: swap: free_swap_and_cache_nr() as batched free_swap_and_cache() > mm: swap: Simplify struct percpu_cluster > mm: swap: Allow storage of all mTHP orders > mm: vmscan: Avoid split during shrink_folio_list() > mm: madvise: Avoid split during MADV_PAGEOUT and MADV_COLD > > include/linux/pgtable.h | 28 ++++ > include/linux/swap.h | 33 +++-- > mm/huge_memory.c | 3 - > mm/internal.h | 48 +++++++ > mm/madvise.c | 101 ++++++++------ > mm/memory.c | 13 +- > mm/swapfile.c | 298 ++++++++++++++++++++++------------------ > mm/vmscan.c | 9 +- > 8 files changed, 332 insertions(+), 201 deletions(-) > > -- > 2.25.1 >