From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 065C8C369C2 for ; Sun, 20 Apr 2025 05:52:10 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id A200C6B00B7; Sun, 20 Apr 2025 01:52:08 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 9CCEC6B00B8; Sun, 20 Apr 2025 01:52:08 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 895306B00B9; Sun, 20 Apr 2025 01:52:08 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id 67D106B00B7 for ; Sun, 20 Apr 2025 01:52:08 -0400 (EDT) Received: from smtpin04.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id 8B6AF1CE179 for ; Sun, 20 Apr 2025 05:52:09 +0000 (UTC) X-FDA: 83353351578.04.3ADCAC4 Received: from mail-pg1-f174.google.com (mail-pg1-f174.google.com [209.85.215.174]) by imf01.hostedemail.com (Postfix) with ESMTP id AE0D840003 for ; Sun, 20 Apr 2025 05:52:07 +0000 (UTC) Authentication-Results: imf01.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=YVPgd7el; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf01.hostedemail.com: domain of ioworker0@gmail.com designates 209.85.215.174 as permitted sender) smtp.mailfrom=ioworker0@gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1745128327; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:references:dkim-signature; bh=JlR1Cl/7Pf+X4fiHaZz55r1eEkogd8TlZyAeSyaqNBM=; b=UmdzKxn2Rj50alQE/Ly07SPE3XBbyS0rOhzKr7TBLWy22PCdmE+4jK6WlbQMgNecgoHM94 lKai7DMwosq3hmvTaVtI4lOLc5C6i5VIKctRsMH1j4rsRnkPiaYRLPC2SGXLW/WqPKFasd DrrlZIh8b0PCfeOE/YCSw16XK91TOpc= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1745128327; a=rsa-sha256; cv=none; b=Cni+3CaipD+zYVEzYWVKXusU7WkCFOjvv+atiuFeMgD6GLUtEYTcng3QEilGtQq5rCmEAr v1hUy5kWgXGbZfBQHD8xwcHa+CzjFNUx7h6tU+TIVZVcXiTiUe7lcy2ePe8PpkeQbw6+pu fUKsQJ9VsgUwRO+n75ENGT5NW0lgO+I= ARC-Authentication-Results: i=1; imf01.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=YVPgd7el; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf01.hostedemail.com: domain of ioworker0@gmail.com designates 209.85.215.174 as permitted sender) smtp.mailfrom=ioworker0@gmail.com Received: by mail-pg1-f174.google.com with SMTP id 41be03b00d2f7-879d2e419b9so2524596a12.2 for ; Sat, 19 Apr 2025 22:52:07 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1745128326; x=1745733126; darn=kvack.org; h=content-transfer-encoding:mime-version:message-id:date:subject:cc :to:from:from:to:cc:subject:date:message-id:reply-to; bh=JlR1Cl/7Pf+X4fiHaZz55r1eEkogd8TlZyAeSyaqNBM=; b=YVPgd7elHElmOL9lR+iZHR6XQRN1vqB1YT61McBL9edDAmtNYbAigL+o66+9QR4J/K kt1t7YbwYK+4Neslf/LqPUF302woqA+E2M8BkKCWJF4T105NQrsLrpcAT9Ur9XPK/9u4 dZRfbwZhNpFwmFDPpimbF2zA+09wjdi8RqH7rfADPDTVEVEXFQKWMFNTDU2a269vlkaV ni7HjV7RX33S6YhBTUd7hOhGfoCjz3IpF2Pa9AkwS3gEl4V9/P87fuuIA+bKH6EC8jvB /XEMTW68A37dATMABTGI9lbvkLg2gTQEni29/gFwA8ymdaC8ZTB1f+2mbb+fu2zb0oLE MtVw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1745128326; x=1745733126; h=content-transfer-encoding:mime-version:message-id:date:subject:cc :to:from:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=JlR1Cl/7Pf+X4fiHaZz55r1eEkogd8TlZyAeSyaqNBM=; b=DwDi6FhC+a0AVn9Kf428ybzIt+cxc8L5IcZ1f+jR9cJTJMh9w9HP9rFyN7cDyveyjt Bk9alTexTfP9Vt0CTxm3NUperBo8gKxiS1+tFoVNiw68W5V6QHdhUuHYTLdKsNGBpyK7 bN5MCyvNHAaEilSjNzUEU6e/5cO3ER7W0v5QPmhTVnfPjaxv/JbK73PTylzDB9jiJEhd pvEolG21rF9cnILjYTqi+GGbcyWNqGTMw+oWY+s6ltGNnsLmFmjgVIzMTskb3kGxMD/1 JJsgRTbNQ9D6G0OSaIWQPnaqg+txfFdDUqjlxKnfVED5m6J25VNSfXdnUY3gpWaaqtke uLvQ== X-Forwarded-Encrypted: i=1; AJvYcCXEJCJ7Lz30SFsX3ozaLBcUz07kMETL2vBmnfTiqoAf3eCQfb2dbL/pAAUiT2kUQrQHypmfofxfNA==@kvack.org X-Gm-Message-State: AOJu0YzGuJci/TzWPsHPbhCUAPKNTUujEdAOE5ImjpI8YPtFK+BENrmi UL6SDUVdzaaNQJfP8A8YnpO3gh6nh/moWE+TRJb2/M6DpjNSUXdc X-Gm-Gg: ASbGncvCm3UKnVz4geVxKLRt+vySRmcQP/PihE8os8+fXaL8fyneN+uV+NEielWD0ef X30iSuqkY9cCBmaEE8+EwtWT+wYrMQG0CkF7FD+z+lmSUskTwC1TyFx394AdzxR5Hjh2uztzqOC 5DtbhQ2NthVMdyyuulcwM2hvFagmtnpidd7hdn+kk6gzP4+GEn/R0q5nOjHck5rucj9BQbwBcpP VY0CH+ITSzulc5sGVpsz5deOIvEx1tgDXlxZaKFDscZBUDBq4pdKfoqnZObJCGgHJ4kaRNIgyZb PHm4yGYq9goMUfeNN3V/fRGEOW+4JwfjS1o7ISn7mpvl7DoHZg== X-Google-Smtp-Source: AGHT+IF3cll38uRyxsB2kcUfmP59mZlO2KBDMq2EIAY6WmBp89fc1amrCvru7mMoZVUbqHrL11MH6w== X-Received: by 2002:a17:90b:498d:b0:2ff:4e90:3c55 with SMTP id 98e67ed59e1d1-3087bbaeb22mr10719705a91.27.1745128326453; Sat, 19 Apr 2025 22:52:06 -0700 (PDT) Received: from localhost.localdomain ([43.129.202.66]) by smtp.gmail.com with ESMTPSA id 98e67ed59e1d1-3087df4e12asm4062534a91.37.2025.04.19.22.52.03 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Sat, 19 Apr 2025 22:52:06 -0700 (PDT) From: Lance Yang X-Google-Original-From: Lance Yang To: akpm@linux-foundation.org Cc: mingzhe.yang@ly.com, ioworker0@gmail.com, david@redhat.com, willy@infradead.org, ziy@nvidia.com, mhocko@suse.com, vbabka@suse.cz, surenb@google.com, linux-mm@kvack.org, jackmanb@google.com, hannes@cmpxchg.org, linux-kernel@vger.kernel.org, Lance Yang Subject: [PATCH 1/1] mm/rmap: optimize MM-ID mapcount handling with union Date: Sun, 20 Apr 2025 13:51:59 +0800 Message-ID: <20250420055159.55851-1-lance.yang@linux.dev> X-Mailer: git-send-email 2.49.0 MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Rspamd-Server: rspam12 X-Rspamd-Queue-Id: AE0D840003 X-Rspam-User: X-Stat-Signature: ghmybd4q9zswbbsnw6pmh5gyke99r9mr X-HE-Tag: 1745128327-274660 X-HE-Meta: U2FsdGVkX184kdtwJCuTquzC8VEdGBt5bbb7DVJqqRUDlY0CXiYSziJ5rX15T/9gFFBYLCu731VSIjaV5lpq994B6o8ffHqbr6AtVqVZZ5Kht/HLjQx+D67Lxt4tcHIiuHfmgN6dBbGSDqgsQAaTUgk2boECGxVgH7y21aLU9uwopXlccvVnIsp7m3AurPtGcWLZubvknmg4AbxBX/Eu3EGsxkXE/2O84pJ6SgTpGb6WcTuIDSseUd4a3mGvDEE5SUCzxN4V06/GX5QF3ceRs2fKhAugyBApf8ZeZh6VXSCpylODsuoddOe0fjNE1N3CsTTmKmHZhUBLhCuICArQjpL8OqDWi6vJAUDOMSHqaWT3ZlSM4s0LqkqB4jvszfa2P82A1t/K/LGmlKOhSP4W62mScgOAoGUl2PbUw1e+aYqxt5nlFDgxogVlOVppZeERBFkzLZbBpBBtdT5kBxxtZ8FfnIxgz1NwSbuVCZYTcUkeJCaO531Kt+P6juRCx+KNFju80LOI5UZQas2ROfu3zfarAn0FGm2mIMHfhYvu17imuLHlzAO84eFOVmuSrUu89Fb/G4bTB8aBwdTC78SAs3PbTVi01bJAM9f2+DXmrjcd5nJxQPrjYHbPIxSkqSWTt8C+tAnRlgGGbysPTIs/6jEbxWdn6p/K4XibBMDFiVPnsmAkUw2jR9Xz5DADIuCPDKemphVtr6UCF1KHcsFLzRyH14s+JrLsXLiQeXntUiGO7MSTw4sbYi6Rv49OwplWwAgKGkc0e+ZF4jCD5FDjfnO3PZX5I3oDc0aOZJC6v4XlihY4k9pjQf75kFXE7RO/o5LZNBuEf6Pt7e+3lBtljeoDMVvbMcn+bFr5fUdjuUtlFD3Z3MR1NcGOZrMhrejGuZom1WtfPpQNG++34fqmTdjLtr/nuXnSqh0e6WUuCobIhSbUJw7wGj346naRC2aVxTpnX8xBKPIPwqlkGJQ fhjuWuiU VtyFEkywSekhGgMq/+YR/xgLhVYySU7dlUzBoMW8LM+1bck/amabkAQ3+8ntfbiMDoTrbpI2zo5FpgFGBJ7Z9Ke3K9WvL553yD2anb4xLJYikplSFFaJTW7/CyUhzdrpfDAuirQ2+8JnJsK09lsmviKQjVki6rpay3L6DPrJC9DfS2FzokVU++dRp6VZCq3FWm8hVfAudeiBgYTxsc8jJw54bqP6k5fILvTEt6kQ1B3B/Kb78+CRQK9PaWJihNL1QKpDz7CCy3o/GX+IN0gZ2BfGjf9o3Shyo7OBGG6XU3Bl2okpXkZA0pMWvdg== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: From: Lance Yang This patch moves '_mm_id_mapcount' into a union with an unsigned long '_mm_id_mapcounts'. In that way, we can zap both slots to -1 at once via -1UL, which is compatible with both 32/64-bit systems and makes compiler happy without any memory/performance overhead. Also, we remove the two MM_ID_DUMMY checks for each '_mm_id' slot and only validate '_mm_ids' once. Signed-off-by: Lance Yang --- include/linux/mm_types.h | 6 +++++- include/linux/rmap.h | 3 +-- mm/internal.h | 9 +++++++-- mm/page_alloc.c | 11 +++++------ 4 files changed, 18 insertions(+), 11 deletions(-) diff --git a/include/linux/mm_types.h b/include/linux/mm_types.h index 56d07edd01f9..0ac80eaa775e 100644 --- a/include/linux/mm_types.h +++ b/include/linux/mm_types.h @@ -363,6 +363,7 @@ typedef unsigned short mm_id_t; * @_mm_id: Do not use outside of rmap code. * @_mm_ids: Do not use outside of rmap code. * @_mm_id_mapcount: Do not use outside of rmap code. + * @_mm_id_mapcounts: Do not use outside of rmap code. * @_hugetlb_subpool: Do not use directly, use accessor in hugetlb.h. * @_hugetlb_cgroup: Do not use directly, use accessor in hugetlb_cgroup.h. * @_hugetlb_cgroup_rsvd: Do not use directly, use accessor in hugetlb_cgroup.h. @@ -435,7 +436,10 @@ struct folio { atomic_t _entire_mapcount; atomic_t _pincount; #endif /* CONFIG_64BIT */ - mm_id_mapcount_t _mm_id_mapcount[2]; + union { + mm_id_mapcount_t _mm_id_mapcount[2]; + unsigned long _mm_id_mapcounts; + }; union { mm_id_t _mm_id[2]; unsigned long _mm_ids; diff --git a/include/linux/rmap.h b/include/linux/rmap.h index 6b82b618846e..99c0518c1ad8 100644 --- a/include/linux/rmap.h +++ b/include/linux/rmap.h @@ -231,8 +231,7 @@ static __always_inline void folio_set_large_mapcount(struct folio *folio, { __folio_large_mapcount_sanity_checks(folio, mapcount, vma->vm_mm->mm_id); - VM_WARN_ON_ONCE(folio_mm_id(folio, 0) != MM_ID_DUMMY); - VM_WARN_ON_ONCE(folio_mm_id(folio, 1) != MM_ID_DUMMY); + VM_WARN_ON_ONCE(folio->_mm_ids != MM_ID_DUMMY); /* Note: mapcounts start at -1. */ atomic_set(&folio->_large_mapcount, mapcount - 1); diff --git a/mm/internal.h b/mm/internal.h index 838f840ded83..1505174178f4 100644 --- a/mm/internal.h +++ b/mm/internal.h @@ -772,8 +772,13 @@ static inline void prep_compound_head(struct page *page, unsigned int order) atomic_set(&folio->_nr_pages_mapped, 0); if (IS_ENABLED(CONFIG_MM_ID)) { folio->_mm_ids = 0; - folio->_mm_id_mapcount[0] = -1; - folio->_mm_id_mapcount[1] = -1; + /* + * One-shot initialization of both mapcount slots to -1. + * Using 'unsigned long' ensures cross-arch compatibility: + * - 32-bit: Fills both short slots (0xFFFF each) + * - 64-bit: Fills both int slots (0xFFFFFFFF each) + */ + folio->_mm_id_mapcounts = -1UL; } if (IS_ENABLED(CONFIG_64BIT) || order > 1) { atomic_set(&folio->_pincount, 0); diff --git a/mm/page_alloc.c b/mm/page_alloc.c index bbfd8e4ce0df..f5c5829c4bfa 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -976,12 +976,11 @@ static int free_tail_page_prepare(struct page *head_page, struct page *page) goto out; } if (IS_ENABLED(CONFIG_MM_ID)) { - if (unlikely(folio->_mm_id_mapcount[0] != -1)) { - bad_page(page, "nonzero mm mapcount 0"); - goto out; - } - if (unlikely(folio->_mm_id_mapcount[1] != -1)) { - bad_page(page, "nonzero mm mapcount 1"); + if (unlikely(folio->_mm_id_mapcounts != -1UL)) { + if (folio->_mm_id_mapcount[0] != -1) + bad_page(page, "nonzero mm mapcount 0"); + if (folio->_mm_id_mapcount[1] != -1) + bad_page(page, "nonzero mm mapcount 1"); goto out; } } -- 2.49.0