From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id A2E73477E4B for ; Fri, 27 Feb 2026 20:10:15 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1772223015; cv=none; b=J2doDuvhzXRhuaHlHvLU5c4zWjCSYHfWMZDuj4ONDSOMZFOcEeBICWtvynZz7rBSJDRJP4KJR+t5qgwDOiBwJ/w7OLmMdAdjEYy/j0/oBfPSCezMkudyVMdV+oUgUWbqVfT5HuwtvLOyOf4BByErFC5JtI4dm1LHIljBC16Bdx4= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1772223015; c=relaxed/simple; bh=uYOtCYbh4+nkFSZ2M2Bb3QxY1s3O7DU74UtkoVhlKQk=; h=Date:To:From:Subject:Message-Id; b=Mg0u9Lk3ZbK84l8AWKQifGujKs3i/3a48vT++kZPQzLnUNo37i2MqlsUIJBoi7vqOWsa6Lq+OW2HnZ0+ZzuiI9d5rwqXzAZid9k0g98b7Kd4Iv/H2/PQZVfVTBaINRJ2YYWABsY0CA6qCFY0Dj4uss3c5SwLKSWJp7RD3ZuP77E= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=linux-foundation.org header.i=@linux-foundation.org header.b=kKqHGdqA; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=linux-foundation.org header.i=@linux-foundation.org header.b="kKqHGdqA" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 78A94C2BC87; Fri, 27 Feb 2026 20:10:15 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=linux-foundation.org; s=korg; t=1772223015; bh=uYOtCYbh4+nkFSZ2M2Bb3QxY1s3O7DU74UtkoVhlKQk=; h=Date:To:From:Subject:From; b=kKqHGdqACHRZscWgZgbteTIp/7Vg7kTZeFtWRkS9sKNmNS8a3wbu32yhjvZLXr3NM bc8yg9Obd+GRuuMYB6h1qmVVp1+bfYfIF78cd2d8Cy9yBDuB2anj+97uYoEwV3AEvl suuG/9rcak4ePFXmgnb0BYDgWysPFcyl3fhS4IxY= Date: Fri, 27 Feb 2026 12:10:14 -0800 To: mm-commits@vger.kernel.org,ziy@nvidia.com,willy@infradead.org,vbabka@suse.cz,usamaarif642@gmail.com,rppt@kernel.org,roman.gushchin@linux.dev,rientjes@google.com,paul.walmsley@sifive.com,palmer@dabbelt.com,osalvador@suse.de,muchun.song@linux.dev,mhocko@suse.com,lorenzo.stoakes@oracle.com,kernel@xen0n.name,harry.yoo@oracle.com,hannes@cmpxchg.org,fvdl@google.com,david@kernel.org,corbet@lwn.net,cl@gentwo.org,chenhuacai@kernel.org,bhe@redhat.com,aou@eecs.berkeley.edu,alex@ghiti.fr,kas@kernel.org,akpm@linux-foundation.org From: Andrew Morton Subject: + mm-remove-the-branch-from-compound_head.patch added to mm-new branch Message-Id: <20260227201015.78A94C2BC87@smtp.kernel.org> Precedence: bulk X-Mailing-List: mm-commits@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: The patch titled Subject: mm: remove the branch from compound_head() has been added to the -mm mm-new branch. Its filename is mm-remove-the-branch-from-compound_head.patch This patch will shortly appear at https://git.kernel.org/pub/scm/linux/kernel/git/akpm/25-new.git/tree/patches/mm-remove-the-branch-from-compound_head.patch This patch will later appear in the mm-new branch at git://git.kernel.org/pub/scm/linux/kernel/git/akpm/mm Note, mm-new is a provisional staging ground for work-in-progress patches, and acceptance into mm-new is a notification for others take notice and to finish up reviews. Please do not hesitate to respond to review feedback and post updated versions to replace or incrementally fixup patches in mm-new. The mm-new branch of mm.git is not included in linux-next Before you just go and hit "reply", please: a) Consider who else should be cc'ed b) Prefer to cc a suitable mailing list as well c) Ideally: find the original patch on the mailing list and do a reply-to-all to that, adding suitable additional cc's *** Remember to use Documentation/process/submit-checklist.rst when testing your code *** The -mm tree is included into linux-next via various branches at git://git.kernel.org/pub/scm/linux/kernel/git/akpm/mm and is updated there most days ------------------------------------------------------ From: Kiryl Shutsemau Subject: mm: remove the branch from compound_head() Date: Fri, 27 Feb 2026 19:42:54 +0000 The compound_head() function is a hot path. For example, the zap path calls it for every leaf page table entry. Rewrite the helper function in a branchless manner to eliminate the risk of CPU branch misprediction. Link: https://lkml.kernel.org/r/20260227194302.274384-17-kas@kernel.org Signed-off-by: Kiryl Shutsemau Reviewed-by: Muchun Song Reviewed-by: Zi Yan Acked-by: David Hildenbrand (Arm) Reviewed-by: Vlastimil Babka Cc: Albert Ou Cc: Alexandre Ghiti Cc: Baoquan He Cc: Christoph Lameter Cc: David Rientjes Cc: Frank van der Linden Cc: Harry Yoo Cc: Huacai Chen Cc: Johannes Weiner Cc: Jonathan Corbet Cc: Lorenzo Stoakes Cc: Matthew Wilcox (Oracle) Cc: Michal Hocko Cc: Mike Rapoport Cc: Oscar Salvador Cc: Palmer Dabbelt Cc: Paul Walmsley Cc: Roman Gushchin Cc: Usama Arif Cc: WANG Xuerui Signed-off-by: Andrew Morton --- include/linux/page-flags.h | 27 +++++++++++++++++---------- 1 file changed, 17 insertions(+), 10 deletions(-) --- a/include/linux/page-flags.h~mm-remove-the-branch-from-compound_head +++ a/include/linux/page-flags.h @@ -229,25 +229,32 @@ static __always_inline int page_is_fake_ static __always_inline unsigned long _compound_head(const struct page *page) { unsigned long info = READ_ONCE(page->compound_info); + unsigned long mask; - /* Bit 0 encodes PageTail() */ - if (!(info & 1)) - return (unsigned long)page; + if (!compound_info_has_mask()) { + /* Bit 0 encodes PageTail() */ + if (info & 1) + return info - 1; - /* - * If compound_info_has_mask() is false, the rest of compound_info is - * the pointer to the head page. - */ - if (!compound_info_has_mask()) - return info - 1; + return (unsigned long)page; + } /* * If compound_info_has_mask() is true the rest of the info encodes * the mask that converts the address of the tail page to the head page. * * No need to clear bit 0 in the mask as 'page' always has it clear. + * + * Let's do it in a branchless manner. */ - return (unsigned long)page & info; + + /* Non-tail: -1UL, Tail: 0 */ + mask = (info & 1) - 1; + + /* Non-tail: -1UL, Tail: info */ + mask |= info; + + return (unsigned long)page & mask; } #define compound_head(page) ((typeof(page))_compound_head(page)) _ Patches currently in -mm which might be from kas@kernel.org are mm-move-max_folio_order-definition-to-mmzoneh.patch mm-change-the-interface-of-prep_compound_tail.patch mm-rename-the-compound_head-field-in-the-struct-page-to-compound_info.patch mm-move-set-clear_compound_head-next-to-compound_head.patch riscv-mm-align-vmemmap-to-maximal-folio-size.patch loongarch-mm-align-vmemmap-to-maximal-folio-size.patch mm-rework-compound_head-for-power-of-2-sizeofstruct-page.patch mm-sparse-check-memmap-alignment-for-compound_info_has_mask.patch mm-hugetlb-defer-vmemmap-population-for-bootmem-hugepages.patch mm-hugetlb-refactor-code-around-vmemmap_walk.patch x86-vdso-undefine-config_hugetlb_page_optimize_vmemmap-for-vdso32.patch mm-hugetlb-remove-fake-head-pages.patch mm-drop-fake-head-checks.patch hugetlb-remove-vmemmap_synchronize_rcu.patch mm-hugetlb-remove-hugetlb_optimize_vmemmap_key-static-key.patch mm-remove-the-branch-from-compound_head.patch hugetlb-update-vmemmap_deduprst.patch mm-slab-use-compound_head-in-page_slab.patch