[PATCH v2 0/2] mm/huge_memory: take a reference on a device DAX PUD copied by fork

[PATCH v2 0/2] mm/huge_memory: take a reference on a device DAX PUD copied by fork

[Konstantin Khorenko]

copy_huge_pud() copies a device DAX PUD into the child without the
reference, rmap and counter that vmf_insert_folio_pud() took, so the
child's unmap takes them from the parent. fork copies a device DAX VMA
once it has a guard region, so it takes no more than:

1. map a devdax instance with 1 GiB alignment and fault in a PUD;
2. MADV_GUARD_INSTALL a page elsewhere in the VMA;
3. fork, and let the child exit.

Patch 1 is the fix, patch 2 a selftest for it. I found it while
reworking copy_huge_pmd() for fork after David's review [1]; it is
independent of that series.

v2 is based on mm-hotfixes-unstable, as patch 1 is a fix for stable.

Testing
=======

 * mm-hotfixes-unstable x86_64 KVM guest with CONFIG_DEBUG_VM
 * pmem from memmap=3G!4G
 * devdax made with "ndctl create-namespace -m devdax -a 1G":

  +----------------------+--------------+------------------------------+
  | kernel               | fork_dax_pud | log                          |
  +----------------------+--------------+------------------------------+
  | mm-hotfixes-unstable | not ok       | BUG: Bad rss-counter state   |
  |                      |              | mm:... type:MM_FILEPAGES     |
  |                      |              | val:-262144                  |
  +----------------------+--------------+------------------------------+
  | + this series        | ok           | clean                        |
  +----------------------+--------------+------------------------------+

A variant that writes to the device and forks three times also hit,
on mm-hotfixes-unstable:

  WARNING: ./include/linux/rmap.h:159 at
  __folio_large_mapcount_sanity_checks+0x90/0x150

With the series it is clean, and the guard-regions selftest passes.

The selftest only runs on a device given in FORK_DAX_PUD_DEVICE or
argv[1], as on an unfixed kernel it leaves that device's folio with a
mapcount below zero. On pmem emulated in RAM, with
CONFIG_X86_PMEM_LEGACY and CONFIG_DEV_DAX_PMEM:

  boot with memmap=3G!4G
  ndctl create-namespace -f -e namespace0.0 -m devdax -a 1G
  FORK_DAX_PUD_DEVICE=/dev/dax0.0 ./fork_dax_pud

[1] https://lore.kernel.org/linux-mm/20261014-b4-prep-vstor-149847-copy_huge_pmd-recheck-special-pmd-v3-0-85eaec8417f6@virtuozzo.com/

Assisted-by: LLM
Signed-off-by: Konstantin Khorenko <khorenko@virtuozzo.com>
---
Changes in v2:
- Patch 1: copy_huge_pud() takes only the VMAs, and the TODO moves next
  to the rmap duplication, as in David's fix for the same bug:
  https://lore.kernel.org/all/26862633-c3d1-4b82-8ca2-346cb78570d7@kernel.org/
- Patch 2: read_sysfs() and a new rss_file() in vm_util instead of local
  helpers (Sashiko).
- Assisted-by moved from the patches to this cover letter.
- Rebased onto mm-hotfixes-unstable.
- Link to v1: https://patch.msgid.link/20261021-vstor-150143-copy_huge_pud-folio-refcount-v1-0-8bd88551f9be@virtuozzo.com

---
Konstantin Khorenko (2):
      mm/huge_memory: take a reference on a folio PUD copied by fork
      selftests/mm: test fork of a device DAX PUD

 MAINTAINERS                               |   1 +
 include/linux/huge_mm.h                   |   5 +-
 include/linux/rmap.h                      |  22 ++++++
 mm/huge_memory.c                          |  27 ++++---
 mm/memory.c                               |   4 +-
 tools/testing/selftests/mm/Makefile       |   1 +
 tools/testing/selftests/mm/fork_dax_pud.c | 114 ++++++++++++++++++++++++++++++
 tools/testing/selftests/mm/run_vmtests.sh |   1 +
 tools/testing/selftests/mm/vm_util.c      |  21 ++++++
 tools/testing/selftests/mm/vm_util.h      |   1 +
 10 files changed, 183 insertions(+), 14 deletions(-)
---
base-commit: af05d765a4476d2e48099148c80f9b13f462e309
change-id: 20261021-vstor-150143-copy_huge_pud-folio-refcount-ea269e668270

Best regards,
--  
Konstantin Khorenko,
Virtuozzo Linux Kernel Team

[PATCH v2 1/2] mm/huge_memory: take a reference on a folio PUD copied by fork

[Konstantin Khorenko]

Device DAX maps its memory with vmf_insert_folio_pud(), which takes a
folio reference, adds a PUD rmap and accounts the pages to the mm.
zap_huge_pud() drops all three, but copy_huge_pud() copies the PUD into
the child without taking any of them. Each child's unmap then drops
what the parent took:

- the folio is a reference short, so with two children its refcount
  drops to zero and free_zone_device_folio() runs while the parent
  still maps it;
- the child's file counter goes negative, "Bad rss-counter state" when
  its mm is freed;
- the parent's unmap drives the entire mapcount below zero.

fork copies the page tables of a device DAX mapping that has a guard
region, and commit f807123d578d ("mm: allow guard regions in file-backed
and read-only mappings") allowed MADV_GUARD_INSTALL there.

Take the reference, the rmap and the counter for a PUD that
vm_normal_page_pud() resolves to a page, as copy_present_ptes() does for
a PTE, with a new folio_dup_file_rmap_pud() helper. copy_huge_pud() now
needs both VMAs, so pass only them and take the mms from them.

Fixes: aed877c2b425 ("device/dax: properly refcount device dax pages when mapping")
Suggested-by: David Hildenbrand (Arm) <david@kernel.org>
Cc: stable@vger.kernel.org
Signed-off-by: Konstantin Khorenko <khorenko@virtuozzo.com>
---
 include/linux/huge_mm.h |  5 ++---
 include/linux/rmap.h    | 22 ++++++++++++++++++++++
 mm/huge_memory.c        | 27 +++++++++++++++++++--------
 mm/memory.c             |  4 +---
 4 files changed, 44 insertions(+), 14 deletions(-)

diff --git a/include/linux/huge_mm.h b/include/linux/huge_mm.h
index c745f7ad22987..fac05120ab774 100644
--- a/include/linux/huge_mm.h
+++ b/include/linux/huge_mm.h
@@ -12,9 +12,8 @@ int copy_huge_pmd(struct mm_struct *dst_mm, struct mm_struct *src_mm,
 		  pmd_t *dst_pmd, pmd_t *src_pmd, unsigned long addr,
 		  struct vm_area_struct *dst_vma, struct vm_area_struct *src_vma);
 bool huge_pmd_set_accessed(struct vm_fault *vmf);
-int copy_huge_pud(struct mm_struct *dst_mm, struct mm_struct *src_mm,
-		  pud_t *dst_pud, pud_t *src_pud, unsigned long addr,
-		  struct vm_area_struct *vma);
+int copy_huge_pud(struct vm_area_struct *dst_vma, struct vm_area_struct *src_vma,
+		pud_t *dst_pud, pud_t *src_pud, unsigned long addr);
 
 #ifdef CONFIG_HAVE_ARCH_TRANSPARENT_HUGEPAGE_PUD
 void huge_pud_set_accessed(struct vm_fault *vmf, pud_t orig_pud);
diff --git a/include/linux/rmap.h b/include/linux/rmap.h
index 74cca0e3c7264..52fff8dc74213 100644
--- a/include/linux/rmap.h
+++ b/include/linux/rmap.h
@@ -564,6 +564,28 @@ static inline void folio_dup_file_rmap_pmd(struct folio *folio,
 #endif
 }
 
+/**
+ * folio_dup_file_rmap_pud - duplicate a PUD mapping of a page range of a folio
+ * @folio:	The folio to duplicate the mapping of
+ * @page:	The first page to duplicate the mapping of
+ * @dst_vma:	The destination vm area
+ *
+ * The page range of the folio is defined by [page, page + HPAGE_PUD_NR)
+ *
+ * The caller needs to hold the page table lock.
+ */
+static inline void folio_dup_file_rmap_pud(struct folio *folio,
+		struct page *page, struct vm_area_struct *dst_vma)
+{
+#if defined(CONFIG_TRANSPARENT_HUGEPAGE) && \
+	defined(CONFIG_HAVE_ARCH_TRANSPARENT_HUGEPAGE_PUD)
+	__folio_dup_file_rmap(folio, page, HPAGE_PUD_NR, dst_vma,
+			      PGTABLE_LEVEL_PUD);
+#else
+	WARN_ON_ONCE(true);
+#endif
+}
+
 static __always_inline int __folio_try_dup_anon_rmap(struct folio *folio,
 		struct page *page, int nr_pages, struct vm_area_struct *dst_vma,
 		struct vm_area_struct *src_vma, enum pgtable_level level)
diff --git a/mm/huge_memory.c b/mm/huge_memory.c
index 14599d0fff013..30c21e85d21bf 100644
--- a/mm/huge_memory.c
+++ b/mm/huge_memory.c
@@ -2072,11 +2072,13 @@ void touch_pud(struct vm_area_struct *vma, unsigned long addr,
 		update_mmu_cache_pud(vma, addr, pud);
 }
 
-int copy_huge_pud(struct mm_struct *dst_mm, struct mm_struct *src_mm,
-		  pud_t *dst_pud, pud_t *src_pud, unsigned long addr,
-		  struct vm_area_struct *vma)
+int copy_huge_pud(struct vm_area_struct *dst_vma, struct vm_area_struct *src_vma,
+		pud_t *dst_pud, pud_t *src_pud, unsigned long addr)
 {
+	struct mm_struct *dst_mm = dst_vma->vm_mm;
+	struct mm_struct *src_mm = src_vma->vm_mm;
 	spinlock_t *dst_ptl, *src_ptl;
+	struct page *page;
 	pud_t pud;
 	int ret;
 
@@ -2089,11 +2091,20 @@ int copy_huge_pud(struct mm_struct *dst_mm, struct mm_struct *src_mm,
 	if (unlikely(!pud_trans_huge(pud)))
 		goto out_unlock;
 
-	/*
-	 * TODO: once we support anonymous pages, use
-	 * folio_try_dup_anon_rmap_*() and split if duplicating fails.
-	 */
-	if (vma_is_cow_mapping(vma) && pud_write(pud)) {
+	page = vm_normal_page_pud(src_vma, addr, pud);
+	if (page) {
+		struct folio *folio = page_folio(page);
+
+		folio_get(folio);
+		/*
+		 * TODO: once we support anonymous pages, use
+		 * folio_try_dup_anon_rmap_*() and split if duplicating fails.
+		 */
+		folio_dup_file_rmap_pud(folio, page, dst_vma);
+		add_mm_counter(dst_mm, mm_counter_file(folio), HPAGE_PUD_NR);
+	}
+
+	if (vma_is_cow_mapping(src_vma) && pud_write(pud)) {
 		pudp_set_wrprotect(src_mm, addr, src_pud);
 		pud = pud_wrprotect(pud);
 	}
diff --git a/mm/memory.c b/mm/memory.c
index b65b56fe97148..66fbe8ed5bac0 100644
--- a/mm/memory.c
+++ b/mm/memory.c
@@ -1463,7 +1463,6 @@ copy_pud_range(struct vm_area_struct *dst_vma, struct vm_area_struct *src_vma,
 	       unsigned long end)
 {
 	struct mm_struct *dst_mm = dst_vma->vm_mm;
-	struct mm_struct *src_mm = src_vma->vm_mm;
 	pud_t *src_pud, *dst_pud;
 	unsigned long next;
 
@@ -1477,8 +1476,7 @@ copy_pud_range(struct vm_area_struct *dst_vma, struct vm_area_struct *src_vma,
 			int err;
 
 			VM_BUG_ON_VMA(next-addr != HPAGE_PUD_SIZE, src_vma);
-			err = copy_huge_pud(dst_mm, src_mm,
-					    dst_pud, src_pud, addr, src_vma);
+			err = copy_huge_pud(dst_vma, src_vma, dst_pud, src_pud, addr);
 			if (err == -ENOMEM)
 				return -ENOMEM;
 			if (!err)

-- 
2.43.0

Re: [PATCH v2 1/2] mm/huge_memory: take a reference on a folio PUD copied by fork

[David Hildenbrand (Arm)]

On 11/5/26 18:00, Konstantin Khorenko wrote:
> Device DAX maps its memory with vmf_insert_folio_pud(), which takes a
> folio reference, adds a PUD rmap and accounts the pages to the mm.
> zap_huge_pud() drops all three, but copy_huge_pud() copies the PUD into
> the child without taking any of them. Each child's unmap then drops
> what the parent took:
> 
> - the folio is a reference short, so with two children its refcount
>   drops to zero and free_zone_device_folio() runs while the parent
>   still maps it;
> - the child's file counter goes negative, "Bad rss-counter state" when
>   its mm is freed;
> - the parent's unmap drives the entire mapcount below zero.
> 
> fork copies the page tables of a device DAX mapping that has a guard
> region, and commit f807123d578d ("mm: allow guard regions in file-backed
> and read-only mappings") allowed MADV_GUARD_INSTALL there.
> 
> Take the reference, the rmap and the counter for a PUD that
> vm_normal_page_pud() resolves to a page, as copy_present_ptes() does for
> a PTE, with a new folio_dup_file_rmap_pud() helper. copy_huge_pud() now
> needs both VMAs, so pass only them and take the mms from them.
> 
> Fixes: aed877c2b425 ("device/dax: properly refcount device dax pages when mapping")
> Suggested-by: David Hildenbrand (Arm) <david@kernel.org>
> Cc: stable@vger.kernel.org
> Signed-off-by: Konstantin Khorenko <khorenko@virtuozzo.com>
> ---

Thanks, I'll wait a couple of days and then queue this.

Reviewed-by: David Hildenbrand (Arm) <david@kernel.org>

-- 
Cheers,

David

Re: [PATCH v2 1/2] mm/huge_memory: take a reference on a folio PUD copied by fork

[Zi Yan]

On Thu Nov 5, 2026 at 12:00 PM EST, Konstantin Khorenko wrote:
> Device DAX maps its memory with vmf_insert_folio_pud(), which takes a
> folio reference, adds a PUD rmap and accounts the pages to the mm.
> zap_huge_pud() drops all three, but copy_huge_pud() copies the PUD into
> the child without taking any of them. Each child's unmap then drops
> what the parent took:
>
> - the folio is a reference short, so with two children its refcount
>   drops to zero and free_zone_device_folio() runs while the parent
>   still maps it;
> - the child's file counter goes negative, "Bad rss-counter state" when
>   its mm is freed;
> - the parent's unmap drives the entire mapcount below zero.
>
> fork copies the page tables of a device DAX mapping that has a guard
> region, and commit f807123d578d ("mm: allow guard regions in file-backed
> and read-only mappings") allowed MADV_GUARD_INSTALL there.
>
> Take the reference, the rmap and the counter for a PUD that
> vm_normal_page_pud() resolves to a page, as copy_present_ptes() does for
> a PTE, with a new folio_dup_file_rmap_pud() helper. copy_huge_pud() now
> needs both VMAs, so pass only them and take the mms from them.
>
> Fixes: aed877c2b425 ("device/dax: properly refcount device dax pages when mapping")
> Suggested-by: David Hildenbrand (Arm) <david@kernel.org>
> Cc: stable@vger.kernel.org
> Signed-off-by: Konstantin Khorenko <khorenko@virtuozzo.com>
> ---
>  include/linux/huge_mm.h |  5 ++---
>  include/linux/rmap.h    | 22 ++++++++++++++++++++++
>  mm/huge_memory.c        | 27 +++++++++++++++++++--------
>  mm/memory.c             |  4 +---
>  4 files changed, 44 insertions(+), 14 deletions(-)
>

I assume this supersedes David's version[1], which uses
folio_add_file_rmap_pud().

[1] https://lore.kernel.org/all/20261031-fixes_core-v3-11-d6437171ecc3@kernel.org/

LGTM.

Reviewed-by: Zi Yan <ziy@nvidia.com>


-- 
Best Regards,
Yan, Zi

Re: [PATCH v2 1/2] mm/huge_memory: take a reference on a folio PUD copied by fork

[David Hildenbrand (Arm)]

On 11/5/26 21:19, Zi Yan wrote:
> On Thu Nov 5, 2026 at 12:00 PM EST, Konstantin Khorenko wrote:
>> Device DAX maps its memory with vmf_insert_folio_pud(), which takes a
>> folio reference, adds a PUD rmap and accounts the pages to the mm.
>> zap_huge_pud() drops all three, but copy_huge_pud() copies the PUD into
>> the child without taking any of them. Each child's unmap then drops
>> what the parent took:
>>
>> - the folio is a reference short, so with two children its refcount
>>   drops to zero and free_zone_device_folio() runs while the parent
>>   still maps it;
>> - the child's file counter goes negative, "Bad rss-counter state" when
>>   its mm is freed;
>> - the parent's unmap drives the entire mapcount below zero.
>>
>> fork copies the page tables of a device DAX mapping that has a guard
>> region, and commit f807123d578d ("mm: allow guard regions in file-backed
>> and read-only mappings") allowed MADV_GUARD_INSTALL there.
>>
>> Take the reference, the rmap and the counter for a PUD that
>> vm_normal_page_pud() resolves to a page, as copy_present_ptes() does for
>> a PTE, with a new folio_dup_file_rmap_pud() helper. copy_huge_pud() now
>> needs both VMAs, so pass only them and take the mms from them.
>>
>> Fixes: aed877c2b425 ("device/dax: properly refcount device dax pages when mapping")
>> Suggested-by: David Hildenbrand (Arm) <david@kernel.org>
>> Cc: stable@vger.kernel.org
>> Signed-off-by: Konstantin Khorenko <khorenko@virtuozzo.com>
>> ---
>>  include/linux/huge_mm.h |  5 ++---
>>  include/linux/rmap.h    | 22 ++++++++++++++++++++++
>>  mm/huge_memory.c        | 27 +++++++++++++++++++--------
>>  mm/memory.c             |  4 +---
>>  4 files changed, 44 insertions(+), 14 deletions(-)
>>
> 
> I assume this supersedes David's version[1], which uses
> folio_add_file_rmap_pud().
> 
> [1] https://lore.kernel.org/all/20261031-fixes_core-v3-11-d6437171ecc3@kernel.org/

Ugh, I thought I dropped it to wait for this one here. But turns out the flood
of patches confused me :( (hoping I didn't drop another one by mistake)

It's already part of the fixes PULL request:

https://lore.kernel.org/linux-mm/20261105084704.5703-1-david@kernel.org/

Konstantin, sorry! Can you send a follow up to introduce+use the dup variant?

-- 
Cheers,

David

Re: [PATCH v2 1/2] mm/huge_memory: take a reference on a folio PUD copied by fork

[Konstantin Khorenko]

On 11/5/26 21:31, David Hildenbrand (Arm) wrote:
> On 11/5/26 21:19, Zi Yan wrote:
...
>> I assume this supersedes David's version[1], which uses
>> folio_add_file_rmap_pud().
>>
>> [1] https://lore.kernel.org/all/20261031-fixes_core-v3-11-d6437171ecc3@kernel.org/
> 
> Ugh, I thought I dropped it to wait for this one here. But turns out the flood
> of patches confused me :( (hoping I didn't drop another one by mistake)
> 
> It's already part of the fixes PULL request:
> 
> https://lore.kernel.org/linux-mm/20261105084704.5703-1-david@kernel.org/
> 
> Konstantin, sorry! Can you send a follow up to introduce+use the dup variant?

No problem, will do the follow-up on top of thp-7.5.misc in the mm/core tree.

-- 
Cheers,
Konstantin.