From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-alma10-1.taild15c8.ts.net [100.103.45.18]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 8FF292F3621; Thu, 30 Jul 2026 01:03:31 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=100.103.45.18 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1785373413; cv=none; b=Sc3TMnIZ/gMMZzu+5A6qCqHS3tiFl8b4Nu27UDU/p9b6YM5HyfyMfs5KHassggDmo/J6OeCLWvLQmz6Y6e8Lp8dTI70FruRm9rkZrsYJAPUps9tHA/yEnqwzKmWymE/V5cWpG26Vr1t6IyaBinJyNfp155mF0JpAKVn2ZsOGHuE= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1785373413; c=relaxed/simple; bh=z7ATQ6zDzKI/teZ6WhTkmvUTeiWBS92bBzQzxnWF00Y=; h=Date:To:From:Subject:Message-Id; b=RqW8XAOgz4nfXN+W6NE3kWcFkZGnOsznRVuJGGmEaQg1JSDeMVZWwX4sLY20rqdOItn0p7UUwcirTIeH524cBSJRMbSKbpN0bMtAmkm8G/ChRHfGFsg2+yd8mcWULVMOqJXHqxS8z2jDLZSLbgxmVizcSV4GJJJJZiEE2VcrELk= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=linux-foundation.org header.i=@linux-foundation.org header.b=dOXhX4SX; arc=none smtp.client-ip=100.103.45.18 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=linux-foundation.org header.i=@linux-foundation.org header.b="dOXhX4SX" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 569A81F000E9; Thu, 30 Jul 2026 01:03:31 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux-foundation.org; s=korg; t=1785373411; bh=WcmRjgaxljxoQNJPFuVN1plIhrUmfteGL+XoJZ3NJ6U=; h=Date:To:From:Subject; b=dOXhX4SXFzGbktIu0B5OB54ZiTWtXpxpfyfnjGISomLOTHXf1XgdzL3th49k+yjWX SCtWIcFWdIngny+lqwMVvBXrvbxyh4RorQlHlk9ELEc7IdmBlgbBNml+SRtUZe9uRf BQCl0Q2Xrq29OGKxGmmqHOuiLJM5/ACNcLPwnFyg= Date: Wed, 29 Jul 2026 18:03:30 -0700 To: mm-commits@vger.kernel.org,ziy@nvidia.com,xueyuan.chen@vivo.com,uqbarz@gmail.com,stable@vger.kernel.org,shy828301@gmail.com,ryan.roberts@arm.com,p.raghav@samsung.com,npache@redhat.com,liam@infradead.org,lance.yang@linux.dev,kas@kernel.org,hughd@google.com,hare@suse.de,dev.jain@arm.com,david@kernel.org,baolin.wang@linux.alibaba.com,baohua@kernel.org,ljs@kernel.org,akpm@linux-foundation.org From: Andrew Morton Subject: + mm-huge_memory-fix-huge_zero_pfn-race.patch added to mm-new branch Message-Id: <20260730010331.569A81F000E9@smtp.kernel.org> Precedence: bulk X-Mailing-List: mm-commits@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: The patch titled Subject: mm/huge_memory: fix huge_zero_pfn race has been added to the -mm mm-new branch. Its filename is mm-huge_memory-fix-huge_zero_pfn-race.patch This patch will shortly appear at https://git.kernel.org/pub/scm/linux/kernel/git/akpm/25-new.git/tree/patches/mm-huge_memory-fix-huge_zero_pfn-race.patch This patch will later appear in the mm-new branch at git://git.kernel.org/pub/scm/linux/kernel/git/akpm/mm Note, mm-new is a provisional staging ground for work-in-progress patches, and acceptance into mm-new is a notification for others take notice and to finish up reviews. Please do not hesitate to respond to review feedback and post updated versions to replace or incrementally fixup patches in mm-new. The mm-new branch of mm.git is not included in linux-next If a few days of testing in mm-new is successful, the patch will me moved into mm.git's mm-unstable branch, which is included in linux-next Before you just go and hit "reply", please: a) Consider who else should be cc'ed b) Prefer to cc a suitable mailing list as well c) Ideally: find the original patch on the mailing list and do a reply-to-all to that, adding suitable additional cc's *** Remember to use Documentation/process/submit-checklist.rst when testing your code *** The -mm tree is included into linux-next via various branches at git://git.kernel.org/pub/scm/linux/kernel/git/akpm/mm and is updated there most days ------------------------------------------------------ From: "Lorenzo Stoakes (ARM)" Subject: mm/huge_memory: fix huge_zero_pfn race Date: Tue, 28 Jul 2026 13:05:45 +0100 If !CONFIG_PERSISTENT_HUGE_ZERO_FOLIO, the huge_zero_folio is refcounted by huge_zero_refcount and returned by mm_get_huge_zero_folio(). When the caller is done with the huge zero page, its reference count is decremented. Only a shrinker can set the reference count to zero. A race can unfortunately occur between a shrinker decrementing the reference count to zero and a concurrent page fault. This is because shrink_huge_zero_folio_scan() might, if very unlucky, be preempted between setting huge_zero_refcount to zero and writing an invalid value. During this time get_huge_zero_folio() could write to huge_zero_pfn before shrink_huge_zero_folio_scan() resumes. In this event the huge zero folio will be persistently misidentified causing the THP code path to be entered inappropriately for the huge zero folio: CPU 0 CPU 1 =======================================|================================= shrink_huge_zero_folio_scan() | atomic_cmpxchg() sets refcount to 0 | xchg() sets huge_zero_folio to NULL | get_huge_zero_folio() | | atomic_inc_not_zero() -> zero preempted for a long time | Allocate new huge zero folio | | Write valid huge_zero_folio v | Write valid huge_zero_pfn Overwrite huge_zero_pfn with ~0UL <--- Invalid overwrite! This results in is_huge_zero_pfn() and is_huge_zero_pmd() incorrectly returning false for a huge zero page which could result in issues like the huge zero folio being incorrectly split. Note that the issue is with huge_zero_pfn not huge_zero_folio, as get_huge_zero_folio() uses cmpxchg() gated on huge_zero_folio being NULL with a retry loop and shrink_huge_zero_folio_scan() uses xchg() to set huge_zero_folio. Fix the issue by introducing a spinlock, huge_zero_lock, to prevent concurrent write of huge_zero_folio, huge_zero_pfn and huge_zero_refcount. There needs to be significant care taken here to ensure correctness: The fast path in get_huge_zero_folio() uses atomic_inc_not_zero(), which is outside of the critical section when !CONFIG_PERSISTENT_HUGE_ZERO_FOLIO, and means huge zero allocation is gated on zero huge_zero_refcount. The fast path doesn't use huge_zero_lock, so the critical section is irrelevant to it. So invariants are required - huge_zero_refcount MUST: * Only be set in the huge_zero_lock critical section to ensure serialisation of huge_zero_pfn, huge_zero_folio and huge_zero_refcount writes. * Be set non-zero only AFTER huge_zero_[pfn, folio] are set to valid values so installation of the huge zero folio on read page fault ensures concurrent is_huge_zero_*() calls correctly identify the huge zero folio. * Be set zero only BEFORE huge_zero_[pfn, folio] are set to NULL and ~0UL respectively, and atomically. Establish these by: * Only updating huge_zero_refcount in the huge_zero_lock critical section in get_huge_zero_folio() and shrink_huge_zero_folio_scan(). * Using atomic_set_release(&huge_zero_refcount) in get_huge_zero_folio() after huge_zero_[pfn, folio] are set. This is paired with atomic_inc_not_zero() to ensure atomic_inc_not_zero() only observes a non-zero value if huge_zero_[pfn, folio] are set. * Using atomic_cmpxchg() in shrink_huge_zero_folio_scan() to ensure that it is set zero only when equal to 1 and set atomically. * atomic_cmpxchg() being fully ordered ensures this is done prior to huge_zero_[folio, pfn] being set to NULL and ~0UL respectively. Note that only the huge zero shrinker (via shrink_huge_zero_folio_scan()) can actually set huge_zero_refcount to zero, which is the count of mm's which have at least one huge zero folio installed plus one shrinker pin. Additionally convert a BUG_ON() to a VM_WARN_ON_ONCE(). Link: https://lore.kernel.org/20260728-fix-refcounted-huge-zero-v1-2-3f261f5447b4@kernel.org Fixes: 3b77e8c8cde5 ("mm/thp: make is_huge_zero_pmd() safe and quicker") Signed-off-by: Lorenzo Stoakes (ARM) Suggested-by: David Hildenbrand (Arm) Reported-by: Hengbin Zhang Closes: https://lore.kernel.org/linux-mm/20260727154001.4102341-1-uqbarz@gmail.com/ Cc: Baolin Wang Cc: Barry Song Cc: Dev Jain Cc: Hannes Reinecke Cc: Hugh Dickins Cc: Kiryl Shutsemau Cc: Lance Yang Cc: Liam R. Howlett Cc: Nico Pache Cc: Pankaj Raghav Cc: Ryan Roberts Cc: Yang Shi Cc: Zi Yan Cc: Xueyuan Chen Cc: # 6.18.x: dependent on prior commit Signed-off-by: Andrew Morton --- mm/huge_memory.c | 42 ++++++++++++++++++++++++++++-------------- 1 file changed, 28 insertions(+), 14 deletions(-) --- a/mm/huge_memory.c~mm-huge_memory-fix-huge_zero_pfn-race +++ a/mm/huge_memory.c @@ -41,6 +41,7 @@ #include #include #include +#include #include #include "internal.h" @@ -82,6 +83,7 @@ struct folio *huge_zero_folio __read_mos unsigned long huge_zero_pfn __read_mostly = HUGE_ZERO_UNSET_PFN; #ifndef CONFIG_PERSISTENT_HUGE_ZERO_FOLIO static atomic_t huge_zero_refcount; +static DEFINE_SPINLOCK(huge_zero_lock); static struct shrinker *huge_zero_folio_shrinker; #endif @@ -270,7 +272,8 @@ void mm_put_huge_zero_folio(struct mm_st static bool get_huge_zero_folio(void) { struct folio *zero_folio; -retry: + + /* Paired with atomic_set_release(). */ if (likely(atomic_inc_not_zero(&huge_zero_refcount))) return true; @@ -278,17 +281,21 @@ retry: if (unlikely(!zero_folio)) return false; - preempt_disable(); - if (cmpxchg(&huge_zero_folio, NULL, zero_folio)) { - preempt_enable(); + /* Paired with critical section in shrink_huge_zero_folio_scan(). */ + spin_lock(&huge_zero_lock); + if (huge_zero_folio) { + /* Somebody else already installed it. */ + atomic_inc(&huge_zero_refcount); + spin_unlock(&huge_zero_lock); folio_put(zero_folio); - goto retry; + return true; } + WRITE_ONCE(huge_zero_folio, zero_folio); WRITE_ONCE(huge_zero_pfn, folio_pfn(zero_folio)); + /* Paired with atomic_inc_not_zero(). +1 for shrinker pin. */ + atomic_set_release(&huge_zero_refcount, 2); + spin_unlock(&huge_zero_lock); - /* We take additional reference here. It will be put back by shrinker */ - atomic_set(&huge_zero_refcount, 2); - preempt_enable(); count_vm_event(THP_ZERO_PAGE_ALLOC); return true; } @@ -312,15 +319,22 @@ static unsigned long shrink_huge_zero_fo static unsigned long shrink_huge_zero_folio_scan(struct shrinker *shrink, struct shrink_control *sc) { - if (atomic_cmpxchg(&huge_zero_refcount, 1, 0) == 1) { - struct folio *zero_folio = xchg(&huge_zero_folio, NULL); - BUG_ON(zero_folio == NULL); + struct folio *zero_folio; + + /* Paired with critical section in get_huge_zero_folio(). */ + scoped_guard(spinlock, &huge_zero_lock) { + /* Paired with atomic_inc_not_zero() in get_huge_zero_folio(). */ + if (atomic_cmpxchg(&huge_zero_refcount, 1, 0) != 1) + return 0; + + zero_folio = huge_zero_folio; + VM_WARN_ON_ONCE(!huge_zero_folio); + WRITE_ONCE(huge_zero_folio, NULL); WRITE_ONCE(huge_zero_pfn, HUGE_ZERO_UNSET_PFN); - folio_put(zero_folio); - return HPAGE_PMD_NR; } - return 0; + folio_put(zero_folio); + return HPAGE_PMD_NR; } static int __init huge_zero_init(void) _ Patches currently in -mm which might be from ljs@kernel.org are mm-vmalloc-acquire-init_mm-lock-on-huge-vmap-to-avoid-ptdump-uaf.patch mm-ptdump-always-stabilise-against-page-table-freeing-using-init_mm.patch arm64-remove-redundant-concurrent-ptdump-uaf-mitigation.patch x86-mm-pat-acquire-init_mm-write-lock-on-collapse-to-avoid-uaf.patch x86-mm-pat-acquire-init_mm-read-lock-on-attribute-change-to-avoid-uaf.patch x86-mm-pat-allocate-split-page-tables-as-kernel-page-tables.patch mm-move-vma_start_pgoff-into-mmh-and-clean-up.patch mm-add-kdoc-comments-for-vma_start-last_pgoff.patch tools-testing-vma-use-vma_start_pgoff-in-merge-tests.patch mm-introduce-and-use-vma_end_pgoff.patch mm-rmap-update-mm-interval_treec-comments.patch mm-rmap-parameterise-vma_interval_tree_-by-address_space.patch mm-rmap-elide-unnecessary-static-inlines-in-interval_treec.patch mm-rmap-rename-vma_interval_tree_-to-mapping_rmap_tree_.patch mm-rmap-parameterise-anon_vma_interval_tree_-by-anon_vma.patch mm-rmap-rename-anon_vma_interval_tree_-params-and-use-pgoff_t.patch mm-rmap-rename-anon_vma_interval_tree_-to-anon_rmap_tree_.patch maintainers-move-mm-interval_treec-to-rmap-section.patch mm-vma-introduce-and-use-vmg_pages-vmg__pgoff.patch mm-vma-clean-up-anon_vma_compatible.patch mm-vma-refactor-vmg_adjust_set_range-for-clarity.patch mm-vma-minor-cleanup-of-expand_.patch mm-introduce-and-use-linear_page_delta.patch mm-vma-use-vma_start_pgoff-linear_page_index-in-mm-code.patch mm-prefer-vma__pgoff-to-vma-vm_pgoff-in-kernel.patch mm-vma-remove-duplicative-vma_pgoff_offset-helper.patch mm-use-linear_page_-consistently.patch mm-vma-introduce-vma_assert_can_modify.patch mm-vma-add-and-use-vma__pgoff.patch mm-vma-move-__install_special_mapping-to-vmac.patch mm-vma-make-vma_set_range-static-drop-insert_vm_struct-decl.patch mm-vma-update-vma_shrink-to-not-pass-start-pgoff-parameters.patch mm-vma-update-vmg_adjust_set_range-to-offset-pgoff-instead.patch mm-vma-slightly-rework-the-anonymous-check-in-__mmap_new_vma.patch mm-vma-introduce-and-use-vma_set_pgoff.patch mm-vma-correct-incorrect-vmah-inclusion.patch mm-vma-use-guard-clauses-in-can_vma_merge_.patch tools-testing-vma-default-vma-mm-flag-bits-to-64-bit.patch tools-testing-vma-output-compared-expression-on-assert_.patch mm-introduce-vma_flags_can_grow-and-vma_can_grow.patch mm-vma-update-do_mmap-to-use-vma_flags_t.patch mm-convert-__get_unmapped_area-to-use-vma_flags_t.patch mm-update-generic_get_unmapped_area-to-use-vma_flags_t.patch mm-prefer-mm-def_vma_flags-in-mm-logic.patch mm-vma-convert-vm_pgprot_modify-to-use-vma_flags_t-and-rename.patch mm-vma-rename-vma_get_page_prot-to-vma_flags_to_page_prot.patch mm-introduce-vma_get_page_prot-and-use-it.patch mm-vma-update-create_init_stack_vma-to-use-vma_flags_t.patch mm-vma-convert-miscellaneous-uses-of-vma-flags-in-core-mm.patch mm-mlock-convert-mlock-code-to-use-vma_flags_t.patch mm-mprotect-convert-mprotect-code-to-use-vma_flags_t.patch mm-mremap-convert-mremap-code-to-use-vma_flags_t.patch mm-mseal-remove-superfluous-comments-fix-confusion-around-mm.patch mm-mseal-limit-scope-of-mseal-address-zero-to-address-zero.patch mm-mseal-remove-further-superfluous-comments-do_mseal.patch mm-vma-introduce-vma-virtual-page-offset-field-and-add-helpers.patch mm-introduce-linear_virt_page_index.patch mm-abstract-vma_address-and-introduce-vma_anon_address.patch mm-update-print_bad_page_map-to-show-virtual-page-index.patch mm-introduce-and-use-vma_filebacked_address.patch mm-propagate-vma-virtual-page-offset-on-map-remap-split-merge.patch mm-rmap-track-whether-the-page-vma-mapped-walk-is-anonymous.patch mm-introduce-and-use-linear_folio_page_index.patch mm-rmap-use-virt-pgoff-for-map_private-file-backed-anon-folios.patch tools-testing-vma-expand-vma-merge-tests-to-assert-virt-pgoff.patch tools-testing-selftests-mm-test-virtual-page-offset-merge-behaviour.patch mm-vma-only-permit-map_private-dev-zero-to-be-mapped-anonymous.patch mm-vma-make-map_private-mapped-dev-zero-mappings-truly-anonymous.patch tools-testing-vma-add-test-to-assert-map_private-dev-zero-is-anon.patch tools-testing-selftests-mm-add-map_private-dev-zero-merge-tests.patch mm-huge_memory-separate-out-config_persistent_huge_zero_folio-logic.patch mm-huge_memory-fix-huge_zero_pfn-race.patch