From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 51B6DC3ABD8 for ; Wed, 14 May 2025 20:18:51 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id A60226B00A9; Wed, 14 May 2025 16:18:50 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 9E8596B00AA; Wed, 14 May 2025 16:18:50 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 813506B00AB; Wed, 14 May 2025 16:18:50 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id 5BF576B00A9 for ; Wed, 14 May 2025 16:18:50 -0400 (EDT) Received: from smtpin02.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay03.hostedemail.com (Postfix) with ESMTP id 3F9A5BEADE for ; Wed, 14 May 2025 20:18:50 +0000 (UTC) X-FDA: 83442626820.02.58C9661 Received: from mail-pj1-f41.google.com (mail-pj1-f41.google.com [209.85.216.41]) by imf30.hostedemail.com (Postfix) with ESMTP id 5F52280003 for ; Wed, 14 May 2025 20:18:48 +0000 (UTC) Authentication-Results: imf30.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=UZseVdDf; spf=pass (imf30.hostedemail.com: domain of ryncsn@gmail.com designates 209.85.216.41 as permitted sender) smtp.mailfrom=ryncsn@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1747253928; a=rsa-sha256; cv=none; b=xwPmKKQmkjTbWUnhUrE7WPpkKUVdf83bOWegz/knD9P4LCN2YSGuWs2MUYqP3XT9Za8Tlq q5YJFhOIZmy8Tca2RZujSEcMTLvKSXM7vM9t9ZCHnzzcEtGSLjMy4pTBHsDgz27lVExKfP VERlvzxvhWVuWkkNINPaWsDDwRCj98w= ARC-Authentication-Results: i=1; imf30.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=UZseVdDf; spf=pass (imf30.hostedemail.com: domain of ryncsn@gmail.com designates 209.85.216.41 as permitted sender) smtp.mailfrom=ryncsn@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1747253928; h=from:from:sender:reply-to:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=4s/QmsKTTQmLGkqDLjmzfFJeHGtuBXIefOE3w5RFPpw=; b=WEMA6+nI3dpR+UOJmlbXbzb4yLVcFd79B9gIVvJLbEp7iNwsMWaUCNlN00URYIUwXwUP1Q liTcgmVnFZG2JIajyKTLyPh/ZfaF2sQaDWKBGLNRNEM9XrObIxyY4vt+aRHGv4bb9wVhMR EqIniR6Zj1pjL4fUWBUjuHIFU+cgTRw= Received: by mail-pj1-f41.google.com with SMTP id 98e67ed59e1d1-30820167b47so259293a91.0 for ; Wed, 14 May 2025 13:18:48 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1747253927; x=1747858727; darn=kvack.org; h=content-transfer-encoding:mime-version:reply-to:references :in-reply-to:message-id:date:subject:cc:to:from:from:to:cc:subject :date:message-id:reply-to; bh=4s/QmsKTTQmLGkqDLjmzfFJeHGtuBXIefOE3w5RFPpw=; b=UZseVdDffkqnyrqnGw+Idjqmn2K8S9ecfVZTAQpKBIC0Vp7n415jwfHx1R1vcuFhhW 2OSWpGXMuGwSGkoKHw4XzryI9C1/8EmO+OnG7/u8PxJ0KXoOwmBoxc18AiFiBJ5iqOc4 /dHsQv34+wSt5uqXFVnSa6Md9IxCQVFc1WarQ6uRAyD7gaWLBkPgd80Ka93FVpcBdMoO qnXiL3iCacBwnhCpT7ykdEIyM2hp6Ov4a0cGINEYlKkjlB+BbPR9ldYmv20q0ZXk3F9+ qdKgZsTRPlxnedso9d5FL4X67qlvnE3PKXYDtz1qkAtGXfdozyMmU3nRswBUETusVWV/ AJYg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1747253927; x=1747858727; h=content-transfer-encoding:mime-version:reply-to:references :in-reply-to:message-id:date:subject:cc:to:from:x-gm-message-state :from:to:cc:subject:date:message-id:reply-to; bh=4s/QmsKTTQmLGkqDLjmzfFJeHGtuBXIefOE3w5RFPpw=; b=jJg+EuSsP5pvz/mm0NFKCAfAnqhl0Hj77e98Vl0m6Hel/M7u0cH4FWEPlPxR3d4zIh u0Z8OUPkSYHDZ3peCBcB+unbyc87pE4Mw+fGLH6ZM7Y6aI5oShxDpodg/WpRX8nlSmKy 1p5O42LlYf0N+Asu6JPWTPttYrcnvv0gMQJRmqpqNRuPv9ZiwAZK/dYaE/LoWudNYWwP U4gwmIj60/YihjQjuDU7jSTRXyD62L3GaxVI2XqRiV2Lvrp+t2Vjf4BYTLfc/ZhaN7h0 Z1VwDizwesbtQOrSLcO3rNwUt3WnAZSaNY1kzu4139bpfH4WGYUNLMzQl39AxelErdU9 WAxg== X-Gm-Message-State: AOJu0YyRHvqJbEZSpgX4Y8GGBQaNP7aPhUscfc5/Dd+vCYZqdGeAjxZO NRG/s0MT51qinVSqaA9OJKf1ZlM9kkOMpohXdHpKRyvH/nA0YSpCcHxkGhP3GG4= X-Gm-Gg: ASbGncuhfnlpYTOjH1QXN9CxSLoohKvgo4oCrkpv64x4NgOCRi9gIjVkkxlh1gGwEw+ 3gLOqH8nXsDq6c9NYMs6x5zd3WDbuLor/eCUgv2Oj8GU59HqVLCavKCmdPvm42j9Avjzj625htL vP2B/ifsRE7j4GJ5+Sao49zIe5CHsyD/7aO1X49lTa3CEOEu10nIIIJeSCjAq/nMLho0XPvOX/h 6FasGTaQ5p1O12vTFpZ5C1N3PKj7+Fci82F91iAUZ9s20/cQ9Onoi/jcxJPnvx+YH6cAZFEDivb xkoJr4adugaiT2Z0vfA45E8FUu4EmzmNOLhgenxHnZ4VMehfZWwBX+bEnCg9KhgDZZHSumWG X-Google-Smtp-Source: AGHT+IG+B1slZQN6/w3ohnuP01AT9ckSlzci2YrvggjzzrH+zdTqUHxw2hQl8e1yDQGhoEt4iqhExw== X-Received: by 2002:a17:90b:3bce:b0:2ff:4a8d:74f9 with SMTP id 98e67ed59e1d1-30e4db4a0b9mr1071062a91.10.1747253926545; Wed, 14 May 2025 13:18:46 -0700 (PDT) Received: from KASONG-MC4.tencent.com ([101.32.222.185]) by smtp.gmail.com with ESMTPSA id 98e67ed59e1d1-30e33401934sm2003692a91.9.2025.05.14.13.18.41 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Wed, 14 May 2025 13:18:45 -0700 (PDT) From: Kairui Song To: linux-mm@kvack.org Cc: Andrew Morton , Matthew Wilcox , Hugh Dickins , Chris Li , David Hildenbrand , Yosry Ahmed , "Huang, Ying" , Nhat Pham , Johannes Weiner , Baolin Wang , Baoquan He , Barry Song , Kalesh Singh , Kemeng Shi , Tim Chen , Ryan Roberts , linux-kernel@vger.kernel.org, Kairui Song Subject: [PATCH 13/28] mm/shmem, swap: avoid redundant Xarray lookup during swapin Date: Thu, 15 May 2025 04:17:13 +0800 Message-ID: <20250514201729.48420-14-ryncsn@gmail.com> X-Mailer: git-send-email 2.49.0 In-Reply-To: <20250514201729.48420-1-ryncsn@gmail.com> References: <20250514201729.48420-1-ryncsn@gmail.com> Reply-To: Kairui Song MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Rspam-User: X-Rspamd-Server: rspam01 X-Rspamd-Queue-Id: 5F52280003 X-Stat-Signature: c76n5p9wueux6pyag5q6zibcxu4kmd6a X-HE-Tag: 1747253928-899697 X-HE-Meta: U2FsdGVkX1/gdnJ2isKsZ4CQvRJoqzaMje4a4K5mcyJymIGRntpEoKwAbqe1tUu7XWZnwXeQJFpeeYO1bEuAocH+PziAYWWi6ivOODSDZ0u9RQHnTT+Pgfs754m6LctbpayjHl22lZ88dHiz398micAQBNZN+2va8EfZyIF4WUxsOYWQP+jhlrtdtyIQAjYu3O/3+kTpvpoT1kh0yyY3UbWQhzUCamYjPxNZGp//0J4CUX/AwarC5OZ4IA8VQWraP9T47ODa+ktsiSKbLM4lr+gHzW4hA+4kOtDzkCDq2IRVQanMgcCo7JdlW3XKAMpIeFg9/uo1zcpOQX0fcmr2d8E9EAYUjLCEtDH0CNsWMmT7Qdy9HkZgRXUs1UnO3KOneR/ch5ek66UO2RvO9WornHA3maG3xZoVaEdvPUNBbWNti1TSHdLUI/EbwcObd34GzRSN04JW5niD7hXqIOWtWRHgr+NXCKfyLHcgct6di/vSeH6oAulq9h/w20EAtdJbXjLaAs4gRxjJb1n+9p2QnpXp7Owzt0+TUZhnKOTenTNR17aj9AtSQd/q0K6FTdTY2jy+GpOi31IWai+nXpGyCA0rFzvWMY/GK+CkGbQXJybzC79Q7cTZiewyhFKtAdS4nI+8H1QigwTgLq4PhlkxK1rkSZYKJIV9LFuIjI3EEfzUYXLXhI/FrD1vERUGUYuQYpLayDWB84dsQkUCIbF97I+VpXPU05rxaLslTEm/n01l7K134k2Z8ZTN5MUywbO9UcPwF/8r7nRB0uAmV7ugnF1Ys0dCw/AUemgtGEGyJUi5ndoGpk44MxVxTgkxoyyzC1Yq6c7OI19XXIOUO+yJPtlRTpEGd9sv/OqlX55WyK7iflVfPaDRaoDvUYSgw6D1oj75YlQLfJ7FZqJHPfaMOu3l1ZBslpeeBbCAtDcWMrQ3GOlSC9+HYAkegS0Lnqn5MqK2+g2b5A08PIxmA+0 C4ftntTB 7b5KZcApTpGnZv+b63zdPkX8Jyjr/TYp/j+cJg/gnNVGh+7A09LViO82/buh7dwZqj81prhvUf+VKAIBMEuPt0WddAsLDxoOpNg4b9Zf0iyu6v3TReO1ckZYbUgbcsn59KSDL/c4eP8Vq++6AiVo3f3kO5Un7pK+7dQTwRz2hOSsDa8nGqIXY3lTbURPg6lVaSZpruc+tfky0m5hfIN1paBlRvYTx86lSul6KQ1dkdpjcQ5KyvdY/z6danP9UFFSKCLufALnKIYG0DeY9l3ABONWVv2yAmv6aC8+XHu3Ka14iiZUxSu+7qjDlRIb5rmJbFL8KeEjdi4niAj76BH3lDvd12szAq9ytMIOj/iWFjIi1HTjYKNfIgQiVKVgqpsmncyfcLFvGVDDfKFepIYolP7BdpT4NRuQvq/XjaKpJuzTjrOmamNxOUwjGdnKcaR9RhxxO03xA/nz0txkyjCQkgni5m/xb+Ct9bqs84Mnz0sCqVglNGOsHATuxiXxE2W9epLtKiGO5wfyv2WFIbNxwtPQceMsFLMPweUsaaiAGUZsDxzuRuWYosMmAtUYHq5qXh9g+JiSMCEHJpm8gnKf0k9MKfQvDXggauvSikSDc0RIP3RA= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: From: Kairui Song Currently shmem calls xa_get_order multiple times to get the swap radix entry order. This can be combined with the swap entry value checking (shmem_confirm_swap) to avoid the duplicated lookup, which should improve the performance. This also provides the helper need for later commits. Signed-off-by: Kairui Song --- mm/shmem.c | 67 ++++++++++++++++++++++++++++++------------------------ 1 file changed, 37 insertions(+), 30 deletions(-) diff --git a/mm/shmem.c b/mm/shmem.c index 0da9e06eaee8..da80a8faa39e 100644 --- a/mm/shmem.c +++ b/mm/shmem.c @@ -509,11 +509,26 @@ static int shmem_replace_entry(struct address_space *mapping, * * Checking folio is not enough: by the time a swapcache folio is locked, it * might be reused, and again be swapcache, using the same swap as before. + * + * Check if the swap entry is still in the shmem mapping and get its order, + * return -1 if it's no longer valid. */ -static bool shmem_confirm_swap(struct address_space *mapping, - pgoff_t index, swp_entry_t swap) +static int shmem_check_swap_entry(struct address_space *mapping, pgoff_t index, + swp_entry_t swap) { - return xa_load(&mapping->i_pages, index) == swp_to_radix_entry(swap); + XA_STATE(xas, &mapping->i_pages, index); + int order = -1; + void *entry; + + rcu_read_lock(); + do { + entry = xas_load(&xas); + if (entry == swp_to_radix_entry(swap)) + order = xas_get_order(&xas); + } while (xas_retry(&xas, entry)); + rcu_read_unlock(); + + return order; } /* @@ -2238,16 +2253,17 @@ static int shmem_swapin_folio(struct inode *inode, pgoff_t index, return -EIO; si = get_swap_device(swap); - if (!si) { - if (!shmem_confirm_swap(mapping, index, swap)) - return -EEXIST; - else - return -EINVAL; + order = shmem_check_swap_entry(mapping, index, swap); + if (order < 0) { + if (si) + put_swap_device(si); + return -EEXIST; } + if (!si) + return -EINVAL; /* Look it up and read it in.. */ folio = swap_cache_get_folio(swap); - order = xa_get_order(&mapping->i_pages, index); if (!folio) { bool fallback_order0 = false; @@ -2303,7 +2319,6 @@ static int shmem_swapin_folio(struct inode *inode, pgoff_t index, */ if (split_order > 0) { pgoff_t offset = index - round_down(index, 1 << split_order); - swap = swp_entry(swp_type(swap), swp_offset(swap) + offset); } @@ -2325,25 +2340,20 @@ static int shmem_swapin_folio(struct inode *inode, pgoff_t index, error = split_order; goto failed; } - - /* - * If the large swap entry has already been split, it is - * necessary to recalculate the new swap entry based on - * the old order alignment. - */ - if (split_order > 0) { - pgoff_t offset = index - round_down(index, 1 << split_order); - - swap = swp_entry(swp_type(swap), swp_offset(swap) + offset); - } } alloced: /* We have to do this with folio locked to prevent races */ folio_lock(folio); - if ((!skip_swapcache && !folio_test_swapcache(folio)) || - folio->swap.val != swap.val || - !shmem_confirm_swap(mapping, index, swap) || - xa_get_order(&mapping->i_pages, index) != folio_order(folio)) { + if (!skip_swapcache && !folio_swap_contains(folio, swap)) { + error = -EEXIST; + goto unlock; + } + + nr_pages = folio_nr_pages(folio); + index = round_down(index, nr_pages); + swap = swp_entry(swp_type(swap), round_down(swp_offset(swap), nr_pages)); + + if (folio_order(folio) != shmem_check_swap_entry(mapping, index, swap)) { error = -EEXIST; goto unlock; } @@ -2354,7 +2364,6 @@ static int shmem_swapin_folio(struct inode *inode, pgoff_t index, goto failed; } folio_wait_writeback(folio); - nr_pages = folio_nr_pages(folio); /* * Some architectures may have to restore extra metadata to the @@ -2368,8 +2377,7 @@ static int shmem_swapin_folio(struct inode *inode, pgoff_t index, goto failed; } - error = shmem_add_to_page_cache(folio, mapping, - round_down(index, nr_pages), + error = shmem_add_to_page_cache(folio, mapping, index, swp_to_radix_entry(swap), gfp); if (error) goto failed; @@ -2392,7 +2400,7 @@ static int shmem_swapin_folio(struct inode *inode, pgoff_t index, *foliop = folio; return 0; failed: - if (!shmem_confirm_swap(mapping, index, swap)) + if (shmem_check_swap_entry(mapping, index, swap) < 0) error = -EEXIST; if (error == -EIO) shmem_set_folio_swapin_error(inode, index, folio, swap, @@ -2405,7 +2413,6 @@ static int shmem_swapin_folio(struct inode *inode, pgoff_t index, folio_put(folio); } put_swap_device(si); - return error; } -- 2.49.0