From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 6308D3F9FB for ; Tue, 9 Sep 2025 04:00:31 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1757390431; cv=none; b=mhbWWurOzhpAnKKWluIhya5liVvjjfR/szjgEjegPJilgZrZwDbi4DJ5glW1iqgJVly2wyfxJW3LMij792TkQFJlAmOiAcG76qiXNNe+KUKJ9gFS5CpcOx7kat1gb1AaQZ28bl2Zto+6lCXNWI6AN2qzYbmPqmV6X6L4ezM9sQ0= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1757390431; c=relaxed/simple; bh=DtjZb3w5y3KOucNuciYjClxnwzgWoESKPx0pZflR8eU=; h=Date:To:From:Subject:Message-Id; b=M30gtM0pzlmgGUIdxeciFCpvIwFPoHNnjH8cSl2EHeQ1xfL6X7cyBcBT5nu2B2I7XZTH74lP2HZS8LAHXwyR28DodtpQ2xqrdCfZ1bK+2Ac3dQp2KanguhuRbd6sEI5Zh4ym0FwS+8bcI8hq7QrlIPgOMOCTcumH3edc79KRnBM= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=linux-foundation.org header.i=@linux-foundation.org header.b=eMW88JS8; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=linux-foundation.org header.i=@linux-foundation.org header.b="eMW88JS8" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 313C7C4CEF4; Tue, 9 Sep 2025 04:00:31 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=linux-foundation.org; s=korg; t=1757390431; bh=DtjZb3w5y3KOucNuciYjClxnwzgWoESKPx0pZflR8eU=; h=Date:To:From:Subject:From; b=eMW88JS8OqPwpiLTzP/p1i8Qj6Ap7kng248D5RDyaWI85Ar7fAKQhRbhed6qM23Dt UNkINyqIWLPJHdEGaLk6HsVFENUhat+1WckhVpUTf3bTN8dS/PWUU1J1xZnOgn5UXl el960nuFHFtb13JjnPNEdKW+HslpQVMqfUj9I7q0= Date: Mon, 08 Sep 2025 21:00:30 -0700 To: mm-commits@vger.kernel.org,ziy@nvidia.com,ying.huang@linux.alibaba.com,simona@ffwll.ch,ryan.roberts@arm.com,rcampbell@nvidia.com,rakie.kim@sk.com,osalvador@suse.de,npache@redhat.com,mpenttil@redhat.com,matthew.brost@intel.com,lyude@redhat.com,lorenzo.stoakes@oracle.com,Liam.Howlett@oracle.com,joshua.hahnjy@gmail.com,gourry@gourry.net,francois.dugast@intel.com,dev.jain@arm.com,david@redhat.com,dakr@kernel.org,byungchul@sk.com,baolin.wang@linux.alibaba.com,baohua@kernel.org,apopple@nvidia.com,airlied@gmail.com,balbirs@nvidia.com,akpm@linux-foundation.org From: Andrew Morton Subject: + mm-rmap-extend-rmap-and-migration-support-device-private-entries.patch added to mm-new branch Message-Id: <20250909040031.313C7C4CEF4@smtp.kernel.org> Precedence: bulk X-Mailing-List: mm-commits@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: The patch titled Subject: mm/rmap: extend rmap and migration support device-private entries has been added to the -mm mm-new branch. Its filename is mm-rmap-extend-rmap-and-migration-support-device-private-entries.patch This patch will shortly appear at https://git.kernel.org/pub/scm/linux/kernel/git/akpm/25-new.git/tree/patches/mm-rmap-extend-rmap-and-migration-support-device-private-entries.patch This patch will later appear in the mm-new branch at git://git.kernel.org/pub/scm/linux/kernel/git/akpm/mm Note, mm-new is a provisional staging ground for work-in-progress patches, and acceptance into mm-new is a notification for others take notice and to finish up reviews. Please do not hesitate to respond to review feedback and post updated versions to replace or incrementally fixup patches in mm-new. Before you just go and hit "reply", please: a) Consider who else should be cc'ed b) Prefer to cc a suitable mailing list as well c) Ideally: find the original patch on the mailing list and do a reply-to-all to that, adding suitable additional cc's *** Remember to use Documentation/process/submit-checklist.rst when testing your code *** The -mm tree is included into linux-next via the mm-everything branch at git://git.kernel.org/pub/scm/linux/kernel/git/akpm/mm and is updated there every 2-3 working days ------------------------------------------------------ From: Balbir Singh Subject: mm/rmap: extend rmap and migration support device-private entries Date: Mon, 8 Sep 2025 10:04:36 +1000 Add device-private THP support to reverse mapping infrastructure, enabling proper handling during migration and walk operations. The key changes are: - add_migration_pmd()/remove_migration_pmd(): Handle device-private entries during folio migration and splitting - page_vma_mapped_walk(): Recognize device-private THP entries during VMA traversal operations This change supports folio splitting and migration operations on device-private entries. Link: https://lkml.kernel.org/r/20250908000448.180088-4-balbirs@nvidia.com Signed-off-by: Balbir Singh Cc: David Hildenbrand Cc: Zi Yan Cc: Joshua Hahn Cc: Rakie Kim Cc: Byungchul Park Cc: Gregory Price Cc: Ying Huang Cc: Alistair Popple Cc: Oscar Salvador Cc: Lorenzo Stoakes Cc: Baolin Wang Cc: "Liam R. Howlett" Cc: Nico Pache Cc: Ryan Roberts Cc: Dev Jain Cc: Barry Song Cc: Lyude Paul Cc: Danilo Krummrich Cc: David Airlie Cc: Simona Vetter Cc: Ralph Campbell Cc: Mika Penttilä Cc: Matthew Brost Cc: Francois Dugast Signed-off-by: Andrew Morton --- mm/damon/ops-common.c | 20 +++++++++++++++++--- mm/huge_memory.c | 16 +++++++++++++++- mm/page_idle.c | 5 +++-- mm/page_vma_mapped.c | 12 ++++++++++-- mm/rmap.c | 19 ++++++++++++++++--- 5 files changed, 61 insertions(+), 11 deletions(-) --- a/mm/damon/ops-common.c~mm-rmap-extend-rmap-and-migration-support-device-private-entries +++ a/mm/damon/ops-common.c @@ -75,12 +75,24 @@ void damon_ptep_mkold(pte_t *pte, struct void damon_pmdp_mkold(pmd_t *pmd, struct vm_area_struct *vma, unsigned long addr) { #ifdef CONFIG_TRANSPARENT_HUGEPAGE - struct folio *folio = damon_get_folio(pmd_pfn(pmdp_get(pmd))); + pmd_t pmdval = pmdp_get(pmd); + struct folio *folio; + bool young = false; + unsigned long pfn; + + if (likely(pmd_present(pmdval))) + pfn = pmd_pfn(pmdval); + else + pfn = swp_offset_pfn(pmd_to_swp_entry(pmdval)); + folio = damon_get_folio(pfn); if (!folio) return; - if (pmdp_clear_young_notify(vma, addr, pmd)) + if (likely(pmd_present(pmdval))) + young |= pmdp_clear_young_notify(vma, addr, pmd); + young |= mmu_notifier_clear_young(vma->vm_mm, addr, addr + PAGE_SIZE); + if (young) folio_set_young(folio); folio_set_idle(folio); @@ -203,7 +215,9 @@ static bool damon_folio_young_one(struct mmu_notifier_test_young(vma->vm_mm, addr); } else { #ifdef CONFIG_TRANSPARENT_HUGEPAGE - *accessed = pmd_young(pmdp_get(pvmw.pmd)) || + pmd_t pmd = pmdp_get(pvmw.pmd); + + *accessed = (pmd_present(pmd) && pmd_young(pmd)) || !folio_test_idle(folio) || mmu_notifier_test_young(vma->vm_mm, addr); #else --- a/mm/huge_memory.c~mm-rmap-extend-rmap-and-migration-support-device-private-entries +++ a/mm/huge_memory.c @@ -4646,7 +4646,10 @@ int set_pmd_migration_entry(struct page_ return 0; flush_cache_range(vma, address, address + HPAGE_PMD_SIZE); - pmdval = pmdp_invalidate(vma, address, pvmw->pmd); + if (unlikely(!pmd_present(*pvmw->pmd))) + pmdval = pmdp_huge_get_and_clear(vma->vm_mm, address, pvmw->pmd); + else + pmdval = pmdp_invalidate(vma, address, pvmw->pmd); /* See folio_try_share_anon_rmap_pmd(): invalidate PMD first. */ anon_exclusive = folio_test_anon(folio) && PageAnonExclusive(page); @@ -4696,6 +4699,17 @@ void remove_migration_pmd(struct page_vm entry = pmd_to_swp_entry(*pvmw->pmd); folio_get(folio); pmde = folio_mk_pmd(folio, READ_ONCE(vma->vm_page_prot)); + + if (folio_is_device_private(folio)) { + if (pmd_write(pmde)) + entry = make_writable_device_private_entry( + page_to_pfn(new)); + else + entry = make_readable_device_private_entry( + page_to_pfn(new)); + pmde = swp_entry_to_pmd(entry); + } + if (pmd_swp_soft_dirty(*pvmw->pmd)) pmde = pmd_mksoft_dirty(pmde); if (is_writable_migration_entry(entry)) --- a/mm/page_idle.c~mm-rmap-extend-rmap-and-migration-support-device-private-entries +++ a/mm/page_idle.c @@ -71,8 +71,9 @@ static bool page_idle_clear_pte_refs_one referenced |= ptep_test_and_clear_young(vma, addr, pvmw.pte); referenced |= mmu_notifier_clear_young(vma->vm_mm, addr, addr + PAGE_SIZE); } else if (IS_ENABLED(CONFIG_TRANSPARENT_HUGEPAGE)) { - if (pmdp_clear_young_notify(vma, addr, pvmw.pmd)) - referenced = true; + if (likely(pmd_present(pmdp_get(pvmw.pmd)))) + referenced |= pmdp_clear_young_notify(vma, addr, pvmw.pmd); + referenced |= mmu_notifier_clear_young(vma->vm_mm, addr, addr + PAGE_SIZE); } else { /* unexpected pmd-mapped page? */ WARN_ON_ONCE(1); --- a/mm/page_vma_mapped.c~mm-rmap-extend-rmap-and-migration-support-device-private-entries +++ a/mm/page_vma_mapped.c @@ -250,12 +250,11 @@ restart: pvmw->ptl = pmd_lock(mm, pvmw->pmd); pmde = *pvmw->pmd; if (!pmd_present(pmde)) { - swp_entry_t entry; + swp_entry_t entry = pmd_to_swp_entry(pmde); if (!thp_migration_supported() || !(pvmw->flags & PVMW_MIGRATION)) return not_found(pvmw); - entry = pmd_to_swp_entry(pmde); if (!is_migration_entry(entry) || !check_pmd(swp_offset_pfn(entry), pvmw)) return not_found(pvmw); @@ -277,6 +276,15 @@ restart: * cannot return prematurely, while zap_huge_pmd() has * cleared *pmd but not decremented compound_mapcount(). */ + swp_entry_t entry; + + entry = pmd_to_swp_entry(pmde); + + if (is_device_private_entry(entry)) { + pvmw->ptl = pmd_lock(mm, pvmw->pmd); + return true; + } + if ((pvmw->flags & PVMW_SYNC) && thp_vma_suitable_order(vma, pvmw->address, PMD_ORDER) && --- a/mm/rmap.c~mm-rmap-extend-rmap-and-migration-support-device-private-entries +++ a/mm/rmap.c @@ -1063,8 +1063,10 @@ static int page_vma_mkclean_one(struct p } else { #ifdef CONFIG_TRANSPARENT_HUGEPAGE pmd_t *pmd = pvmw->pmd; - pmd_t entry; + pmd_t entry = pmdp_get(pmd); + if (!pmd_present(entry)) + continue; if (!pmd_dirty(*pmd) && !pmd_write(*pmd)) continue; @@ -2330,6 +2332,11 @@ static bool try_to_migrate_one(struct fo while (page_vma_mapped_walk(&pvmw)) { /* PMD-mapped THP migration entry */ if (!pvmw.pte) { +#ifdef CONFIG_ARCH_ENABLE_THP_MIGRATION + unsigned long pfn; + pmd_t pmdval; +#endif + if (flags & TTU_SPLIT_HUGE_PMD) { split_huge_pmd_locked(vma, pvmw.address, pvmw.pmd, true); @@ -2338,8 +2345,14 @@ static bool try_to_migrate_one(struct fo break; } #ifdef CONFIG_ARCH_ENABLE_THP_MIGRATION - subpage = folio_page(folio, - pmd_pfn(*pvmw.pmd) - folio_pfn(folio)); + pmdval = pmdp_get(pvmw.pmd); + if (likely(pmd_present(pmdval))) + pfn = pmd_pfn(pmdval); + else + pfn = swp_offset_pfn(pmd_to_swp_entry(pmdval)); + + subpage = folio_page(folio, pfn - folio_pfn(folio)); + VM_BUG_ON_FOLIO(folio_test_hugetlb(folio) || !folio_test_pmd_mappable(folio), folio); _ Patches currently in -mm which might be from balbirs@nvidia.com are mm-zone_device-support-large-zone-device-private-folios.patch mm-huge_memory-add-device-private-thp-support-to-pmd-operations.patch mm-rmap-extend-rmap-and-migration-support-device-private-entries.patch mm-huge_memory-implement-device-private-thp-splitting.patch mm-migrate_device-handle-partially-mapped-folios-during-collection.patch mm-migrate_device-implement-thp-migration-of-zone-device-pages.patch mm-memory-fault-add-thp-fault-handling-for-zone-device-private-pages.patch lib-test_hmm-add-zone-device-private-thp-test-infrastructure.patch mm-memremap-add-driver-callback-support-for-folio-splitting.patch mm-migrate_device-add-thp-splitting-during-migration.patch lib-test_hmm-add-large-page-allocation-failure-testing.patch selftests-mm-hmm-tests-new-tests-for-zone-device-thp-migration.patch selftests-mm-hmm-tests-new-throughput-tests-including-thp.patch gpu-drm-nouveau-enable-thp-support-for-gpu-memory-migration.patch