From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id B535DC7EE31 for ; Fri, 27 Jun 2025 11:55:27 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 8D9536B00B3; Fri, 27 Jun 2025 07:55:24 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 8B34E6B00B4; Fri, 27 Jun 2025 07:55:24 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 6B5F56B00C2; Fri, 27 Jun 2025 07:55:24 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id 5D4B76B00B3 for ; Fri, 27 Jun 2025 07:55:24 -0400 (EDT) Received: from smtpin25.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay03.hostedemail.com (Postfix) with ESMTP id 3254CBADE6 for ; Fri, 27 Jun 2025 11:55:24 +0000 (UTC) X-FDA: 83601025368.25.11065CA Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by imf16.hostedemail.com (Postfix) with ESMTP id 0015E18000B for ; Fri, 27 Jun 2025 11:55:21 +0000 (UTC) Authentication-Results: imf16.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=PGcOTBRS; dmarc=pass (policy=quarantine) header.from=redhat.com; spf=pass (imf16.hostedemail.com: domain of dhildenb@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=dhildenb@redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1751025322; a=rsa-sha256; cv=none; b=A3yiddy/gD9vgg/n426+qRkJTglTYxlL1qj/fELyuqQe3vzZiXqDUVWn05xYIshIhlNY+5 Mvqbo2e9iV+0aQK7pgOkKc9VhCxlp0RPFA14gZ6a5PCvRaqdrFA02i87ON5eSGr1ol+8MB v2lo+giLaAl0OTsw+JLRdKjbmbybgvQ= ARC-Authentication-Results: i=1; imf16.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=PGcOTBRS; dmarc=pass (policy=quarantine) header.from=redhat.com; spf=pass (imf16.hostedemail.com: domain of dhildenb@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=dhildenb@redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1751025322; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=YNZ4VUerWVmEqMtN0xpjRQfwGxXrNwD6y4D9M8fHC9o=; b=2Cwy1/m8PziId4UGkFgj14NnJUEzQaNc7zSHjTTYqI73gxWHZ5bWpVbr52qJBN1Sa1cN7I FTgHEyWpyjsYli4b39oeBBYiH7ZCrad8haIGXERZfnocxk99+MYiMeF6BX4tAQ/Wl4gaWJ gdP1plEg4p8xdEGb9gErSGOXYg05iaM= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1751025321; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=YNZ4VUerWVmEqMtN0xpjRQfwGxXrNwD6y4D9M8fHC9o=; b=PGcOTBRSwXoQNosIyrhnvbuMtYc6qmBFhZIMRRuc8PUWFZRVdZQDCzmnopmQ737G3oIuil 2Z9C9DkrNk9s5usTXnCOGU4mWUlDfNwuywd3VVUJsvh21hIsCDMGgtXGhFcM8Lc3U9PN/X JV2boxn+9ZjXDxdvEQxgF856HGLEBdw= Received: from mail-wr1-f71.google.com (mail-wr1-f71.google.com [209.85.221.71]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-8-sTkPq6HEP8epbBE6ZXVEKg-1; Fri, 27 Jun 2025 07:55:20 -0400 X-MC-Unique: sTkPq6HEP8epbBE6ZXVEKg-1 X-Mimecast-MFC-AGG-ID: sTkPq6HEP8epbBE6ZXVEKg_1751025319 Received: by mail-wr1-f71.google.com with SMTP id ffacd0b85a97d-3a4fac7fa27so707299f8f.0 for ; Fri, 27 Jun 2025 04:55:19 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1751025319; x=1751630119; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=YNZ4VUerWVmEqMtN0xpjRQfwGxXrNwD6y4D9M8fHC9o=; b=d5LDng1M48Zz0chXEknDhh1Wdmc3uLOmQ/Bl2INNiCECBeRrUEvObKq/tTBQCzMwJB +/CxmNbEfP4aAyyBYdDlDKy4vtjLt77FsEZL3vwLgkh2qAHXFmALWgYi/Ehv0Q7DCA0t ZG517PYXmfqyEiCt01FtwjT9/BOvO2A+NIG6Vmmcajt40tKD5/t0PQMIrF8an+UjvKJH FMjWU1wWrxUJSeWvKCpkUgTztrLPHzBqr8x5fJ2tEUYCFqSAfcPzob8imi6ljeTp53En KLrI4asFz3hkMPnzjJcwz89hlD5oryqR5Kd6QmlZkPyw0baxJFkdPFvd65/21QunHjXX 11Uw== X-Gm-Message-State: AOJu0Yw8mKBic6hfRbdSmmyb9KCMUmTfKnVHR+T/HB/5Ie0BNVEVPzmD SX4DCXsleyPbrR4MjJsFMSXURXziT1hIpIHvNF7U/o/acbcNsb7NfWaiVuSM0fecZN1bO0mhq79 eDUj7bLkyo9Hed3qskm5om/bcjK+Jf1S0vBFGydrt/K0yiqaagIQG X-Gm-Gg: ASbGncvgnZaB577Clden+YTRASVbBOhxB+TS6EJw5Z0EyVy2r8RX/wzfKMU4TsBd0ps ZJf4IcE5P3Gem5QLa1Rdbs1h6NkID6vZrBiPmF812rS0KIU4s3xpCFvOvZE7mievgi7wQOrnhcQ Pi2q0GRGinMaitLcQUEQGoGXB4sXfVhaqx1CwvL4lUK865gc+Z7nOHxgzcRr3OEfFuG9N3a0ONW zKwF2hk+LxTa3RqgsJfYej6q6AgsRDyRjA+Yq6EYZbq5h3R41fHlAtFjULyogxHGvGUaf1J8kt9 XOMpWgpFL5FR6wuMm/Jp5hoJRhlE7AW7a4NkYC6kqRqSe9wFkLOgIpDzrIAmmPy4YTNhxF6DFk2 4yp0wgUU= X-Received: by 2002:a05:6000:70a:b0:3a5:39e9:928d with SMTP id ffacd0b85a97d-3a8e986a46dmr2562556f8f.0.1751025318782; Fri, 27 Jun 2025 04:55:18 -0700 (PDT) X-Google-Smtp-Source: AGHT+IGMmga7Hei4JTlN1BHMxoiYcjWrFmvyd2zxxUeKuwdClsZVEASJsZNiVNZKURNd0OK1IzvxGQ== X-Received: by 2002:a05:6000:70a:b0:3a5:39e9:928d with SMTP id ffacd0b85a97d-3a8e986a46dmr2562522f8f.0.1751025318280; Fri, 27 Jun 2025 04:55:18 -0700 (PDT) Received: from localhost (p200300d82f2d5d00f1a32f3065759425.dip0.t-ipconnect.de. [2003:d8:2f2d:5d00:f1a3:2f30:6575:9425]) by smtp.gmail.com with UTF8SMTPSA id ffacd0b85a97d-3a88c7e6f23sm2591779f8f.11.2025.06.27.04.55.17 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Fri, 27 Jun 2025 04:55:17 -0700 (PDT) From: David Hildenbrand To: linux-kernel@vger.kernel.org Cc: linux-mm@kvack.org, David Hildenbrand , Andrew Morton , "Liam R. Howlett" , Lorenzo Stoakes , Vlastimil Babka , Jann Horn , Mike Rapoport , Suren Baghdasaryan , Michal Hocko , Zi Yan , Matthew Brost , Joshua Hahn , Rakie Kim , Byungchul Park , Gregory Price , Ying Huang , Alistair Popple , Pedro Falcato , Rik van Riel , Harry Yoo Subject: [PATCH v1 3/4] mm: split folio_pte_batch() into folio_pte_batch() and folio_pte_batch_ext() Date: Fri, 27 Jun 2025 13:55:09 +0200 Message-ID: <20250627115510.3273675-4-david@redhat.com> X-Mailer: git-send-email 2.49.0 In-Reply-To: <20250627115510.3273675-1-david@redhat.com> References: <20250627115510.3273675-1-david@redhat.com> MIME-Version: 1.0 X-Mimecast-Spam-Score: 0 X-Mimecast-MFC-PROC-ID: Tpup665iCoQjbD8nLbJH6arvRCJCnzgEEBX5BiC-JEw_1751025319 X-Mimecast-Originator: redhat.com Content-Transfer-Encoding: 8bit content-type: text/plain; charset="US-ASCII"; x-default=true X-Rspam-User: X-Stat-Signature: ihgg3hh7n13hzaahy5xpc1ce3c8orcs3 X-Rspamd-Server: rspam02 X-Rspamd-Queue-Id: 0015E18000B X-HE-Tag: 1751025321-206791 X-HE-Meta: U2FsdGVkX1918qhVtSD6urokgewZr+pQPsU6aavP+MZjiRkq5frO/uFOKzU594VDwZ+zHBbIPYvHw+aSFHTpbFrUSjEYANhytctufB8X3EB27PL4NzJGdIrMlRbkRjJE8B89K6AjhIBSRgOVOxx1Le8svFb8ac1xuJp86k9ag+G1bdsHOX2U7UdLY4P48QJ7PyWwiB/SaZFfM6zKsmUKMGt0YeJ+C47QWvnMK9hf0UD49s3daGef6BnKVea0NlKkV1Yff+5DkmhSxvhqXAp18YIpoO8ibzNrJaqk0ex/nKQHNTOrSalwziAUirsY2FpBI2Uq6dsyICGisixUUyunWU1zrLXhhjgPeQIUBvzSuuNnogSs+HoABMnnBpq9aQF1/KPKzlr3Etif9GKojjLLXy8c6i9CCtjO7h4QQ97bnPU/ml+KkPDtefGsM0/s9D1t9Pg6fUBB92HZLloJ7LOxh8vvfYLSdFVYPuP4NB9stYuaJqtSJmgsUtPQcLzxvtP8QDMm8L3fewRV6NfSUrPbWSqosoqCLDcdK6EJriYSGQVncPsD1wNaI110PmMzOs5Hfo5ln15AKn6RfohvZwvhYFXndFQo3ayPz3Kj9fuae/RxAf0Ud5gbsGGIb3aYZowr+eVJIzxTaus4jjZLxyAOjmXd7GzpJFUan/NMzqvf4/Hm/fdD8HlvnNv2+rMbdPME72zlr7g3tvBSZhijgKnvuEx7TuNhclwSQe/LugAFY2EJSDqhvYnBLZ1LLprXgsuzk0uKVitRX+6YzPYD9d80whCBGrGCdmAunrrtC9ajX6Cx004Rnte9dPEShciKnWui7evKKM2le+brXrkKdhjB2OVVBTvP9CM/uKR/A7Oi2jihqq778LyNhR85o7s0rfZ8ZTI9sE3xbaLb5Eo0AHMcYQcsGhqtl7CCqappNcjABAqCPo7DyhjbVQ1FG9Xw3uqpxLjO3vkg4o2/ZsPNdF1 xnzQU3wl us7vgyOXNDn0gyBuxmu7y9ghL70w/ip8K6NASBL5LAOGxMUz3UiT8n0vpm3Iw+N4Rc97AIlxJ3kpjeKsTewo/PJqhCWcgg1277dNhiRjt+ExVeimVZZNCZrb8TUjcRbyQEdC2+6eg821dg0x+8hoWXADYnyg1NqS0b6T9mdCDLNam89VEeZofUu+fbBB74OA7TaU8QemRxsVSqtrGRNjeXdnRowu7kWjaV0hIQD1AL9x0k3EdnAQcN/sD6hn/jIZC2QA0bxKRthkHMbCIAtb5+AxvJJwseMF9RnhvcGNl2se7vjC95Op0a/1UIY2T8NkMqIxGg7ZGUVQMXrXok/HrwM8OkIFoWjiUdLVpQZRdF+Ehk49GHxAAnPd6dM6nyC3FeCL5Pc0ZIsWOrBHKts96i6KUTfuMPMLYIPzON10I7jYtW9uV6sTzX3tGKDkg3zJ2OK7xv0umPfSuSaw5lyrP/L4TXEyb2SkUtCht5KuVHpdL8T3NxHzAVmmlM08BpGUgolhOZi5cLxXcU14mT8LEt6BGwcemevJ7llb7+6Cf1rzIgdrmywgc1SQ4W/PHgf9pmjukz5eRl6+PFvlJ3qQg5yJDO+lgZL5TPnlK415wHGHXfw+QWfWNyI46iDNp4faJG55viIrhNXkiyumoKE69ldU1SqZZWNBpNePC X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Many users (including upcoming ones) don't really need the flags etc, and can live with a function call. So let's provide a basic, non-inlined folio_pte_batch(). In zap_present_ptes(), where we care about performance, the compiler already seem to generate a call to a common inlined folio_pte_batch() variant, shared with fork() code. So calling the new non-inlined variant should not make a difference. While at it, drop the "addr" parameter that is unused. Signed-off-by: David Hildenbrand --- mm/internal.h | 11 ++++++++--- mm/madvise.c | 4 ++-- mm/memory.c | 6 ++---- mm/mempolicy.c | 3 +-- mm/mlock.c | 3 +-- mm/mremap.c | 3 +-- mm/rmap.c | 3 +-- mm/util.c | 29 +++++++++++++++++++++++++++++ 8 files changed, 45 insertions(+), 17 deletions(-) diff --git a/mm/internal.h b/mm/internal.h index ca6590c6d9eab..6000b683f68ee 100644 --- a/mm/internal.h +++ b/mm/internal.h @@ -218,9 +218,8 @@ static inline pte_t __pte_batch_clear_ignored(pte_t pte, fpb_t flags) } /** - * folio_pte_batch - detect a PTE batch for a large folio + * folio_pte_batch_ext - detect a PTE batch for a large folio * @folio: The large folio to detect a PTE batch for. - * @addr: The user virtual address the first page is mapped at. * @ptep: Page table pointer for the first entry. * @pte: Page table entry for the first page. * @max_nr: The maximum number of table entries to consider. @@ -243,9 +242,12 @@ static inline pte_t __pte_batch_clear_ignored(pte_t pte, fpb_t flags) * must be limited by the caller so scanning cannot exceed a single VMA and * a single page table. * + * This function will be inlined to optimize based on the input parameters; + * consider using folio_pte_batch() instead if applicable. + * * Return: the number of table entries in the batch. */ -static inline unsigned int folio_pte_batch(struct folio *folio, unsigned long addr, +static inline unsigned int folio_pte_batch_ext(struct folio *folio, pte_t *ptep, pte_t pte, unsigned int max_nr, fpb_t flags, bool *any_writable, bool *any_young, bool *any_dirty) { @@ -293,6 +295,9 @@ static inline unsigned int folio_pte_batch(struct folio *folio, unsigned long ad return min(nr, max_nr); } +unsigned int folio_pte_batch(struct folio *folio, pte_t *ptep, pte_t pte, + unsigned int max_nr); + /** * pte_move_swp_offset - Move the swap entry offset field of a swap pte * forward or backward by delta diff --git a/mm/madvise.c b/mm/madvise.c index 661bb743d2216..9b9c35a398ed0 100644 --- a/mm/madvise.c +++ b/mm/madvise.c @@ -349,8 +349,8 @@ static inline int madvise_folio_pte_batch(unsigned long addr, unsigned long end, { int max_nr = (end - addr) / PAGE_SIZE; - return folio_pte_batch(folio, addr, ptep, pte, max_nr, 0, NULL, - any_young, any_dirty); + return folio_pte_batch_ext(folio, ptep, pte, max_nr, 0, NULL, + any_young, any_dirty); } static int madvise_cold_or_pageout_pte_range(pmd_t *pmd, diff --git a/mm/memory.c b/mm/memory.c index ab2d6c1425691..43d35d6675f2e 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -995,7 +995,7 @@ copy_present_ptes(struct vm_area_struct *dst_vma, struct vm_area_struct *src_vma if (vma_soft_dirty_enabled(src_vma)) flags |= FPB_HONOR_SOFT_DIRTY; - nr = folio_pte_batch(folio, addr, src_pte, pte, max_nr, flags, + nr = folio_pte_batch_ext(folio, src_pte, pte, max_nr, flags, &any_writable, NULL, NULL); folio_ref_add(folio, nr); if (folio_test_anon(folio)) { @@ -1564,9 +1564,7 @@ static inline int zap_present_ptes(struct mmu_gather *tlb, * by keeping the batching logic separate. */ if (unlikely(folio_test_large(folio) && max_nr != 1)) { - nr = folio_pte_batch(folio, addr, pte, ptent, max_nr, 0, - NULL, NULL, NULL); - + nr = folio_pte_batch(folio, pte, ptent, max_nr); zap_present_folio_ptes(tlb, vma, folio, page, pte, ptent, nr, addr, details, rss, force_flush, force_break, any_skipped); diff --git a/mm/mempolicy.c b/mm/mempolicy.c index 2a25eedc3b1c0..eb83cff7db8c3 100644 --- a/mm/mempolicy.c +++ b/mm/mempolicy.c @@ -711,8 +711,7 @@ static int queue_folios_pte_range(pmd_t *pmd, unsigned long addr, if (!folio || folio_is_zone_device(folio)) continue; if (folio_test_large(folio) && max_nr != 1) - nr = folio_pte_batch(folio, addr, pte, ptent, - max_nr, 0, NULL, NULL, NULL); + nr = folio_pte_batch(folio, pte, ptent, max_nr); /* * vm_normal_folio() filters out zero pages, but there might * still be reserved folios to skip, perhaps in a VDSO. diff --git a/mm/mlock.c b/mm/mlock.c index 2238cdc5eb1c1..a1d93ad33c6db 100644 --- a/mm/mlock.c +++ b/mm/mlock.c @@ -313,8 +313,7 @@ static inline unsigned int folio_mlock_step(struct folio *folio, if (!folio_test_large(folio)) return 1; - return folio_pte_batch(folio, addr, pte, ptent, count, 0, NULL, - NULL, NULL); + return folio_pte_batch(folio, pte, ptent, count); } static inline bool allow_mlock_munlock(struct folio *folio, diff --git a/mm/mremap.c b/mm/mremap.c index d4d3ffc931502..1f5bebbb9c0cb 100644 --- a/mm/mremap.c +++ b/mm/mremap.c @@ -182,8 +182,7 @@ static int mremap_folio_pte_batch(struct vm_area_struct *vma, unsigned long addr if (!folio || !folio_test_large(folio)) return 1; - return folio_pte_batch(folio, addr, ptep, pte, max_nr, 0, NULL, - NULL, NULL); + return folio_pte_batch(folio, ptep, pte, max_nr); } static int move_ptes(struct pagetable_move_control *pmc, diff --git a/mm/rmap.c b/mm/rmap.c index a29d7d29c7283..6658968600b72 100644 --- a/mm/rmap.c +++ b/mm/rmap.c @@ -1859,8 +1859,7 @@ static inline bool can_batch_unmap_folio_ptes(unsigned long addr, if (pte_pfn(pte) != folio_pfn(folio)) return false; - return folio_pte_batch(folio, addr, ptep, pte, max_nr, 0, NULL, - NULL, NULL) == max_nr; + return folio_pte_batch(folio, ptep, pte, max_nr); } /* diff --git a/mm/util.c b/mm/util.c index 0b270c43d7d12..d29dcc135ad28 100644 --- a/mm/util.c +++ b/mm/util.c @@ -1171,3 +1171,32 @@ int compat_vma_mmap_prepare(struct file *file, struct vm_area_struct *vma) return 0; } EXPORT_SYMBOL(compat_vma_mmap_prepare); + +#ifdef CONFIG_MMU +/** + * folio_pte_batch - detect a PTE batch for a large folio + * @folio: The large folio to detect a PTE batch for. + * @ptep: Page table pointer for the first entry. + * @pte: Page table entry for the first page. + * @max_nr: The maximum number of table entries to consider. + * + * This is a simplified variant of folio_pte_batch_ext(). + * + * Detect a PTE batch: consecutive (present) PTEs that map consecutive + * pages of the same large folio in a single VMA and a single page table. + * + * All PTEs inside a PTE batch have the same PTE bits set, excluding the PFN, + * the accessed bit, writable bit, dirt-bit and soft-dirty bit. + * + * ptep must map any page of the folio. max_nr must be at least one and + * must be limited by the caller so scanning cannot exceed a single VMA and + * a single page table. + * + * Return: the number of table entries in the batch. + */ +unsigned int folio_pte_batch(struct folio *folio, pte_t *ptep, pte_t pte, + unsigned int max_nr) +{ + return folio_pte_batch_ext(folio, ptep, pte, max_nr, 0, NULL, NULL, NULL); +} +#endif /* CONFIG_MMU */ -- 2.49.0