From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 8B843332615 for ; Thu, 23 Apr 2026 20:30:57 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1776976257; cv=none; b=eoqt+xaVhUEBgiY/biGPfIS+519CSE/G2r3aOumzVwR71G6TcAmU26p+uEi1DiLI8ngnKp82F52pVy+hZsg2lJRnHmoDEZsuVHZxFTfd5aE1zL8aj/4tOQscjBDoMfKlQ+aVKlDOkBqwBVf4Q/inJ2a7h6OrWTdSkrEEvUWimCM= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1776976257; c=relaxed/simple; bh=iiihwt/nktItFejd3av9GrYSyU7sv84ENyV/+zBLCEU=; h=Date:To:From:Subject:Message-Id; b=ggTgaov5IQjM9G6GfGcdb3/S9sx1q4g0TwZmGlV8O7ev98fwZVZtKmCzEGbooIPrNW2cXfTfVlAr3mny6HGg8KQu3qhM0MBBoHJJNmNh2p265yYxWoVk9Jkz6RJXDOhHAe8i7UfkpV6vdfyw2P3EzYtE1mkeQQWkBJcQiOk7K6E= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=linux-foundation.org header.i=@linux-foundation.org header.b=xXbGJQFx; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=linux-foundation.org header.i=@linux-foundation.org header.b="xXbGJQFx" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 5985CC2BCAF; Thu, 23 Apr 2026 20:30:57 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=linux-foundation.org; s=korg; t=1776976257; bh=iiihwt/nktItFejd3av9GrYSyU7sv84ENyV/+zBLCEU=; h=Date:To:From:Subject:From; b=xXbGJQFx2yUijgraeKZizWyvaQIOlUjaXD2pBZY1cn744H15KW64FIqEoOz/6PdKo xRgy6xULjy/BN9+v8s7ALTXNhkLnjpLqEnex0YMTSmDCtVUYfzCw8f2yoOY4aTbml5 kX4upGKNBgf8xPewcvh0chV1P8FZvkfEasqHMovo= Date: Thu, 23 Apr 2026 13:30:56 -0700 To: mm-commits@vger.kernel.org,npache@redhat.com,akpm@linux-foundation.org From: Andrew Morton Subject: + mm-khugepaged-generalize-collapse_huge_page-for-mthp-collapse.patch added to mm-new branch Message-Id: <20260423203057.5985CC2BCAF@smtp.kernel.org> Precedence: bulk X-Mailing-List: mm-commits@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: The patch titled Subject: mm/khugepaged: generalize collapse_huge_page for mTHP collapse has been added to the -mm mm-new branch. Its filename is mm-khugepaged-generalize-collapse_huge_page-for-mthp-collapse.patch This patch will shortly appear at https://git.kernel.org/pub/scm/linux/kernel/git/akpm/25-new.git/tree/patches/mm-khugepaged-generalize-collapse_huge_page-for-mthp-collapse.patch This patch will later appear in the mm-new branch at git://git.kernel.org/pub/scm/linux/kernel/git/akpm/mm Note, mm-new is a provisional staging ground for work-in-progress patches, and acceptance into mm-new is a notification for others take notice and to finish up reviews. Please do not hesitate to respond to review feedback and post updated versions to replace or incrementally fixup patches in mm-new. The mm-new branch of mm.git is not included in linux-next If a few days of testing in mm-new is successful, the patch will me moved into mm.git's mm-unstable branch, which is included in linux-next Before you just go and hit "reply", please: a) Consider who else should be cc'ed b) Prefer to cc a suitable mailing list as well c) Ideally: find the original patch on the mailing list and do a reply-to-all to that, adding suitable additional cc's *** Remember to use Documentation/process/submit-checklist.rst when testing your code *** The -mm tree is included into linux-next via various branches at git://git.kernel.org/pub/scm/linux/kernel/git/akpm/mm and is updated there most days ------------------------------------------------------ From: Nico Pache Subject: mm/khugepaged: generalize collapse_huge_page for mTHP collapse Date: Sun, 19 Apr 2026 12:57:42 -0600 Pass an order and offset to collapse_huge_page to support collapsing anon memory to arbitrary orders within a PMD. order indicates what mTHP size we are attempting to collapse to, and offset indicates were in the PMD to start the collapse attempt. For non-PMD collapse we must leave the anon VMA write locked until after we collapse the mTHP-- in the PMD case all the pages are isolated, but in the mTHP case this is not true, and we must keep the lock to prevent access/changes to the page tables. This can happen if the rmap walkers hit a pmd_none while the PMD entry is currently unavailable due to being temporarily removed during the collapse phase. Link: https://lore.kernel.org/20260419185750.260784-6-npache@redhat.com Signed-off-by: Nico Pache Cc: Alistair Popple Cc: Andrea Arcangeli Cc: Anshuman Khandual Cc: Bagas Sanjaya Cc: Baolin Wang Cc: Barry Song Cc: Brendan Jackman Cc: Byungchul Park Cc: Catalin Marinas Cc: David Hildenbrand (Arm) Cc: David Rientjes Cc: Dev Jain Cc: Gregory Price Cc: "Huang, Ying" Cc: Hugh Dickins Cc: Jan Kara Cc: Jann Horn Cc: Johannes Weiner Cc: Jonathan Corbet Cc: Joshua Hahn Cc: Kefeng Wang Cc: Lance Yang Cc: Liam Howlett Cc: Lorenzo Stoakes Cc: "Masami Hiramatsu (Google)" Cc: Mathieu Desnoyers Cc: Matthew Brost Cc: Matthew Wilcox (Oracle) Cc: Michal Hocko Cc: Mike Rapoport Cc: Nanyong Sun Cc: Pedro Falcato Cc: Peter Xu Cc: Rafael Aquini Cc: Rakie Kim Cc: Randy Dunlap Cc: Ryan Roberts Cc: Shivank Garg Cc: Steven Rostedt Cc: Suren Baghdasaryan Cc: Takashi Iwai (SUSE) Cc: Thomas Hellström Cc: Usama Arif Cc: Vishal Moola (Oracle) Cc: Vlastimil Babka Cc: Wei Yang Cc: Will Deacon Cc: Yang Shi Cc: Zach O'Keefe Cc: Zi Yan Signed-off-by: Andrew Morton --- mm/khugepaged.c | 103 +++++++++++++++++++++++++--------------------- 1 file changed, 57 insertions(+), 46 deletions(-) --- a/mm/khugepaged.c~mm-khugepaged-generalize-collapse_huge_page-for-mthp-collapse +++ a/mm/khugepaged.c @@ -1198,42 +1198,36 @@ static enum scan_result alloc_charge_fol return SCAN_SUCCEED; } -static enum scan_result collapse_huge_page(struct mm_struct *mm, unsigned long address, - int referenced, int unmapped, struct collapse_control *cc) +static enum scan_result collapse_huge_page(struct mm_struct *mm, unsigned long start_addr, + int referenced, int unmapped, struct collapse_control *cc, + unsigned int order) { LIST_HEAD(compound_pagelist); pmd_t *pmd, _pmd; - pte_t *pte; + pte_t *pte = NULL; pgtable_t pgtable; struct folio *folio; spinlock_t *pmd_ptl, *pte_ptl; enum scan_result result = SCAN_FAIL; struct vm_area_struct *vma; struct mmu_notifier_range range; + bool anon_vma_locked = false; + const unsigned long pmd_addr = start_addr & HPAGE_PMD_MASK; + const unsigned long end_addr = start_addr + (PAGE_SIZE << order); - VM_BUG_ON(address & ~HPAGE_PMD_MASK); - - /* - * Before allocating the hugepage, release the mmap_lock read lock. - * The allocation can take potentially a long time if it involves - * sync compaction, and we do not need to hold the mmap_lock during - * that. We will recheck the vma after taking it again in write mode. - */ - mmap_read_unlock(mm); - - result = alloc_charge_folio(&folio, mm, cc, HPAGE_PMD_ORDER); + result = alloc_charge_folio(&folio, mm, cc, order); if (result != SCAN_SUCCEED) goto out_nolock; mmap_read_lock(mm); - result = hugepage_vma_revalidate(mm, address, true, &vma, cc, - HPAGE_PMD_ORDER); + result = hugepage_vma_revalidate(mm, pmd_addr, /*expect_anon=*/ true, + &vma, cc, order); if (result != SCAN_SUCCEED) { mmap_read_unlock(mm); goto out_nolock; } - result = find_pmd_or_thp_or_none(mm, address, &pmd); + result = find_pmd_or_thp_or_none(mm, pmd_addr, &pmd); if (result != SCAN_SUCCEED) { mmap_read_unlock(mm); goto out_nolock; @@ -1245,8 +1239,8 @@ static enum scan_result collapse_huge_pa * released when it fails. So we jump out_nolock directly in * that case. Continuing to collapse causes inconsistency. */ - result = __collapse_huge_page_swapin(mm, vma, address, pmd, - referenced, HPAGE_PMD_ORDER); + result = __collapse_huge_page_swapin(mm, vma, start_addr, pmd, + referenced, order); if (result != SCAN_SUCCEED) goto out_nolock; } @@ -1261,20 +1255,21 @@ static enum scan_result collapse_huge_pa * mmap_lock. */ mmap_write_lock(mm); - result = hugepage_vma_revalidate(mm, address, true, &vma, cc, - HPAGE_PMD_ORDER); + result = hugepage_vma_revalidate(mm, pmd_addr, /*expect_anon=*/ true, + &vma, cc, order); if (result != SCAN_SUCCEED) goto out_up_write; /* check if the pmd is still valid */ vma_start_write(vma); - result = check_pmd_still_valid(mm, address, pmd); + result = check_pmd_still_valid(mm, pmd_addr, pmd); if (result != SCAN_SUCCEED) goto out_up_write; anon_vma_lock_write(vma->anon_vma); + anon_vma_locked = true; - mmu_notifier_range_init(&range, MMU_NOTIFY_CLEAR, 0, mm, address, - address + HPAGE_PMD_SIZE); + mmu_notifier_range_init(&range, MMU_NOTIFY_CLEAR, 0, mm, start_addr, + end_addr); mmu_notifier_invalidate_range_start(&range); pmd_ptl = pmd_lock(mm, pmd); /* probably unnecessary */ @@ -1286,26 +1281,23 @@ static enum scan_result collapse_huge_pa * Parallel GUP-fast is fine since GUP-fast will back off when * it detects PMD is changed. */ - _pmd = pmdp_collapse_flush(vma, address, pmd); + _pmd = pmdp_collapse_flush(vma, pmd_addr, pmd); spin_unlock(pmd_ptl); mmu_notifier_invalidate_range_end(&range); tlb_remove_table_sync_one(); - pte = pte_offset_map_lock(mm, &_pmd, address, &pte_ptl); + pte = pte_offset_map_lock(mm, &_pmd, start_addr, &pte_ptl); if (pte) { - result = __collapse_huge_page_isolate(vma, address, pte, cc, - HPAGE_PMD_ORDER, - &compound_pagelist); + result = __collapse_huge_page_isolate(vma, start_addr, pte, cc, + order, &compound_pagelist); spin_unlock(pte_ptl); } else { result = SCAN_NO_PTE_TABLE; } if (unlikely(result != SCAN_SUCCEED)) { - if (pte) - pte_unmap(pte); spin_lock(pmd_ptl); - BUG_ON(!pmd_none(*pmd)); + WARN_ON_ONCE(!pmd_none(*pmd)); /* * We can only use set_pmd_at when establishing * hugepmds and never for establishing regular pmds that @@ -1313,21 +1305,24 @@ static enum scan_result collapse_huge_pa */ pmd_populate(mm, pmd, pmd_pgtable(_pmd)); spin_unlock(pmd_ptl); - anon_vma_unlock_write(vma->anon_vma); goto out_up_write; } /* - * All pages are isolated and locked so anon_vma rmap - * can't run anymore. + * For PMD collapse all pages are isolated and locked so anon_vma + * rmap can't run anymore. For mTHP collapse the PMD entry has been + * removed and not all pages are isolated and locked, so we must hold + * the lock to prevent neighboring folios from attempting to access + * this PMD until its reinstalled. */ - anon_vma_unlock_write(vma->anon_vma); + if (is_pmd_order(order)) { + anon_vma_unlock_write(vma->anon_vma); + anon_vma_locked = false; + } result = __collapse_huge_page_copy(pte, folio, pmd, _pmd, - vma, address, pte_ptl, - HPAGE_PMD_ORDER, - &compound_pagelist); - pte_unmap(pte); + vma, start_addr, pte_ptl, + order, &compound_pagelist); if (unlikely(result != SCAN_SUCCEED)) goto out_up_write; @@ -1337,18 +1332,27 @@ static enum scan_result collapse_huge_pa * write. */ __folio_mark_uptodate(folio); - pgtable = pmd_pgtable(_pmd); - spin_lock(pmd_ptl); - BUG_ON(!pmd_none(*pmd)); - pgtable_trans_huge_deposit(mm, pmd, pgtable); - map_anon_folio_pmd_nopf(folio, pmd, vma, address); + WARN_ON_ONCE(!pmd_none(*pmd)); + if (is_pmd_order(order)) { /* PMD collapse */ + pgtable = pmd_pgtable(_pmd); + pgtable_trans_huge_deposit(mm, pmd, pgtable); + map_anon_folio_pmd_nopf(folio, pmd, vma, pmd_addr); + } else { /* mTHP collapse */ + map_anon_folio_pte_nopf(folio, pte, vma, start_addr, /*uffd_wp=*/ false); + smp_wmb(); /* make PTEs visible before PMD. See pmd_install() */ + pmd_populate(mm, pmd, pmd_pgtable(_pmd)); + } spin_unlock(pmd_ptl); folio = NULL; result = SCAN_SUCCEED; out_up_write: + if (anon_vma_locked) + anon_vma_unlock_write(vma->anon_vma); + if (pte) + pte_unmap(pte); mmap_write_unlock(mm); out_nolock: if (folio) @@ -1525,8 +1529,15 @@ static enum scan_result collapse_scan_pm out_unmap: pte_unmap_unlock(pte, ptl); if (result == SCAN_SUCCEED) { + /* + * Before allocating the hugepage, release the mmap_lock read lock. + * The allocation can take potentially a long time if it involves + * sync compaction, and we do not need to hold the mmap_lock during + * that. We will recheck the vma after taking it again in write mode. + */ + mmap_read_unlock(mm); result = collapse_huge_page(mm, start_addr, referenced, - unmapped, cc); + unmapped, cc, HPAGE_PMD_ORDER); /* collapse_huge_page will return with the mmap_lock released */ *lock_dropped = true; } _ Patches currently in -mm which might be from npache@redhat.com are mm-khugepaged-generalize-hugepage_vma_revalidate-for-mthp-support.patch mm-khugepaged-rework-max_ptes_-handling-with-helper-functions.patch mm-khugepaged-generalize-__collapse_huge_page_-for-mthp-support.patch mm-khugepaged-generalize-collapse_huge_page-for-mthp-collapse.patch mm-khugepaged-skip-collapsing-mthp-to-smaller-orders.patch mm-khugepaged-add-per-order-mthp-collapse-failure-statistics.patch mm-khugepaged-improve-tracepoints-for-mthp-orders.patch mm-khugepaged-introduce-collapse_allowable_orders-helper-function.patch mm-khugepaged-introduce-mthp-collapse-support.patch mm-khugepaged-avoid-unnecessary-mthp-collapse-attempts.patch documentation-mm-update-the-admin-guide-for-mthp-collapse.patch