From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 885DDC54E64 for ; Mon, 25 Mar 2024 05:24:46 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 1B7466B0082; Mon, 25 Mar 2024 01:24:46 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 167E56B0083; Mon, 25 Mar 2024 01:24:46 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 057426B0085; Mon, 25 Mar 2024 01:24:46 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id E8F8D6B0082 for ; Mon, 25 Mar 2024 01:24:45 -0400 (EDT) Received: from smtpin15.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id BA42DA0CCA for ; Mon, 25 Mar 2024 05:24:45 +0000 (UTC) X-FDA: 81934421730.15.6AD32CC Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) by imf10.hostedemail.com (Postfix) with ESMTP id 49EC9C0003 for ; Mon, 25 Mar 2024 05:24:42 +0000 (UTC) Authentication-Results: imf10.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=exjU2ECM; dmarc=none; spf=none (imf10.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1711344284; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=WkG50pNPcUCBXYHBUiw48oCBdPvRGtbIEGmeVzsnW+I=; b=fEV6O7EevyunQVaFP/TNlxVC4HlY2hUw0IZnNRbd7Ip24sQbdLBe7DHqT4QvnKtslsAUSY vJCMGIgSiFqdwcLzbf++fTgv1BURxclACNxuAI2IOFSX/jvLBqCOUaR2u6Ee/Dr8IpiqF0 aa7jZyZ+3uhsflxDo6FNGo+McO8vrOA= ARC-Authentication-Results: i=1; imf10.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=exjU2ECM; dmarc=none; spf=none (imf10.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1711344284; a=rsa-sha256; cv=none; b=US2Ne1r1yUr7OAli2u+4OmbDhmgHoeXgTLmmmetK/EqaPVzHFMPdyQEQz/7pqA6s19yQhe 6sbQ4P0UERFxOQg8eHz+IhsK3aUPmxoOmGT6+JmoxIBB5eSs/eorKpKAn2iXR/DtOt6PR6 6UAO4jA5MLJ8ngwBv6/3F9kXUCAa3Lw= DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=In-Reply-To:Content-Type:MIME-Version: References:Message-ID:Subject:Cc:To:From:Date:Sender:Reply-To: Content-Transfer-Encoding:Content-ID:Content-Description; bh=WkG50pNPcUCBXYHBUiw48oCBdPvRGtbIEGmeVzsnW+I=; b=exjU2ECMqvCwhEIpfCbFQl7mqU QIYI1V5XtOEjgqkFKArMY61SBe5KyJNsqWRosKKt8h3y+ZxH9p3gb/WxyFakoslFQ/spqtKGYWOnw 7nYh1QiCHphBgQ258s09BI6ZqpvSTJO81Q1BW7SL4BYYOC3reEfBLvgoex7pA9iNYIyFsTTbpf4HG 3anWsTE6uD+EHH1OYIcVKeDp15zTOHrxhRJmb8wTHl1ixdNi4UE3Tn784vxSkvwFxoAqufeRpcDbg 5mpg1UGlIojow7a3CKm0HonG3HfBnG2NZq8TwYQL3icKtEl08w0t5Jqhbhd2RO3Zf5lt6xw9HA15P EYTQAujw==; Received: from willy by casper.infradead.org with local (Exim 4.97.1 #2 (Red Hat Linux)) id 1rocp0-0000000FkWy-0660; Mon, 25 Mar 2024 05:24:34 +0000 Date: Mon, 25 Mar 2024 05:24:33 +0000 From: Matthew Wilcox To: John Hubbard Cc: Andrew Morton , LKML , linux-mm@kvack.org, David Hildenbrand , Mike Rapoport , Theodore Ts'o , Vishal Moola , Peter Collingbourne Subject: Re: [RFC] mm: page-flags.h: remove the bias against tail pages Message-ID: References: <20240325045519.222458-1-jhubbard@nvidia.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20240325045519.222458-1-jhubbard@nvidia.com> X-Rspamd-Server: rspam09 X-Rspamd-Queue-Id: 49EC9C0003 X-Stat-Signature: ss51iadcgbirojjttowen6o5ufnmu8fa X-Rspam-User: X-HE-Tag: 1711344282-561442 X-HE-Meta: U2FsdGVkX1/6ohJcTtNJnQniNJGgArF8l2YiG5zMx0ZGTRFeI+1C0zqqMhPlrizWlR81SWejSjshcofMy3j0Yg14E0oK2AJFehBDVSmU1DphfcXWS7y0iiN/Oy/OXVDqBn/IgTiZ1EqBWPI4yAvlwQ+Xdn7EboxKXc6VsdH/goNQct2HYrArRrsXUsODL7yv6fEJJsSfP3PxFiKT14FLafKmlpgH7+6eQjyockTvsiOxF/cqJUK0h6uR4xNz7OLRz8lC9OVf8dSufF/jC1+j1ssssAbzZmMid1x4esqcri2phcbxpTY9JuoBG2a4y6G5ksdPiaBAfZgvvX447sivw/YpR/ZlpI75qoJCqpKNEeC91Z9ruETnnE3d/HDRaLNykljL6GaxP7uDFd+1azY3VGfjFjJl5yYvkjliOJ4TybDwEeKXrSaLxFwBXTYCEjsavq0xDCU75a9zUsMLW8kq1Q6rgicJsFs5WmsWicaSK5eSFWEXfsYLXEcU9qOv2Y+RHODWt32IuzI6Et/9HusFeUd7zM+4WFI13zWUo09RW4e9prEFW1kCVM9F86eZGF1wl6RiM5S18b1m2HTR/Zx6ZzdDtodnlR3EG8McNvCJTWaq72oxYfQLT8kHsZaQurajOsHmzJODa3LPISp//IUNVRXjm4GYLw9UbYQMvHEnOh/S7vybe/VKQq8luiyYuOMN9wWWdwGPq2u4MvacxxxwccuJSVk1SopKHRvcGvEhprXz98VGmCAFJ6gu/W6c7CyZ445qKiPvFbop+D3aSQ3YuKq1ptyo059jq1/UG+v0n2cZKf0jLTDF7dVlr+qI8mfHh1/8MSB8/D0UvQ2GdzMKaSN9NT4jEd3aTVgP9P2/1IdOZElQ497GOqCGWQZ1rYuzSUFXfoQAQUglYsxS3fTk0NgAASblJKM1WuoRDmkoWse4rucVtx+HjFx3RcE3+WGaz7jgnYxiC1uLBjvTxhZ mi+yoyYj JqAdOdta3cH0vVvdaqwbXHV3Lyx5zCVUmSAyz6054RnE/mHwrlKThONZ+zB5as7NRsdXFZ6p0CqaLjJ9z5W00Bz2CK3WyLD1POB1MRjGHA1vEtNFdMw8/nKKUXDnzxktN9S8r0p2rMYH0yBJTgXA1yMWKxelfYMFMcUCBeeM9rVmkHG17a2mx6bUpZGogPmp2lvei9Ez29bTBAgWh63/Fc2qq+A== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: On Sun, Mar 24, 2024 at 09:55:19PM -0700, John Hubbard wrote: > commit 1d798ca3f1643 ("mm: make compound_head() robust") added > page->compound_head and the associated "unlikely" check for a tail page > in compound_head(): > > if (unlikely(head & 1)) > return (struct page *) (head - 1); > return page; > > That worked nicely in 2015. However, in the 8.5 years since then, things > have changed: folios and huge pages are heavily used, with more uses > coming. See for example the various THP enhancements being proposed. And > hugetlbfs remains alive and well. And large folios are being plumbed > into everything. > > With that in mind, remove the "unlikely" attribute when checking for a > tail page in compound_head(), and let normal CPU branch prediction do > what it may. > Is this reasonable? I haven't gone out and gathered test data, because > the original patch to create this just assumed that compound pages were > uncommon, and so now it's time to stop making that assumption. I think > that's sufficient reasoning here to leave out the compiler hint, right? It's complicated. On the one hand, it's "more likely" because there are more tail pages than there are head pages or order-0 pages. On the other hand, a _lot_ of the time we call compound_head(), it's done with a non-tail page because we tend to pass around head pages (eg, pmd_page() on hugetlbfs, or looking up a folio in the page cache and passing &folio->page to some function that's not yet converted. On the third hand, does the compiler really do much with the annotation? Before your patch: 27d6: a8 01 test $0x1,%al 27d8: 75 02 jne 27dc 27da: eb 59 jmp 2835 27dc: 49 8b 44 24 08 mov 0x8(%r12),%rax 27e1: a8 01 test $0x1,%al 27e3: 75 6f jne 2854 27e5: eb 73 jmp 285a With your patch: 1ee6: a8 01 test $0x1,%al 1ee8: 75 02 jne 1eec 1eea: eb 5f jmp 1f4b 1eec: 49 8b 44 24 08 mov 0x8(%r12),%rax 1ef1: a8 01 test $0x1,%al 1ef3: 75 50 jne 1f45 1ef5: eb 6c jmp 1f63 Looks pretty much the same. bloat-o-meter says: $ ./scripts/bloat-o-meter before.o after.o add/remove: 0/0 grow/shrink: 2/4 up/down: 32/-48 (-16) Function old new delta gather_stats.constprop 730 753 +23 smaps_hugetlb_range 635 644 +9 smaps_page_accumulate 342 338 -4 clear_refs_pte_range 339 328 -11 pagemap_hugetlb_range 422 407 -15 smaps_pte_range 1406 1388 -18 Total: Before=20066, After=20050, chg -0.08% (I was looking at clear_refs_pte_range above). This seems marginal. The benefits of removing a call to compound_head are much less ambiguous: $ ./scripts/bloat-o-meter before.o .build/fs/proc/task_mmu.o add/remove: 0/0 grow/shrink: 0/1 up/down: 0/-101 (-101) Function old new delta clear_refs_pte_range 339 238 -101 Total: Before=20066, After=19965, chg -0.50% I'd describe that as replacing four calls to compound_head() with two: - page = pmd_page(*pmd); + folio = page_folio(pmd_page(*pmd)); /* Clear accessed and referenced bits. */ pmdp_test_and_clear_young(vma, addr, pmd); - test_and_clear_page_young(page); - ClearPageReferenced(page); + folio_test_clear_young(folio); + folio_clear_referenced(folio); ... - page = vm_normal_page(vma, addr, ptent); - if (!page) + folio = vm_normal_folio(vma, addr, ptent); + if (!folio) continue; /* Clear accessed and referenced bits. */ ptep_test_and_clear_young(vma, addr, pte); - test_and_clear_page_young(page); - ClearPageReferenced(page); + folio_test_clear_young(folio); + folio_clear_referenced(folio); I'm not saying this patch is necessarily wrong, I just think it's "not proven".