[PATCH] x86/mm: Fix pmd_modify() dropping the dirty bit

[PATCH] x86/mm: Fix pmd_modify() dropping the dirty bit

[Vernon Yang]

From: Vernon Yang <yanglincheng@kylinos.cn>

pmd_modify() masks the old value with (_HPAGE_CHG_MASK & ~_PAGE_DIRTY),
silently discarding the hardware dirty bit.  The subsequent
pmd_mksaveddirty() call is supposed to transfer _PAGE_DIRTY into
_PAGE_SAVED_DIRTY when write-protecting, but the dirty bit was already
stripped from the value, so there is nothing left to transfer.

Contrast with pte_modify(), which keeps _PAGE_DIRTY_BITS in its mask,
and pud_modify(), which keeps _HPAGE_CHG_MASK untouched: pmd_modify()
is the odd one out.  Any pmd_modify() on a writable, dirty PMD loses
the dirty state.

One visible consequence is data loss with MADV_FREE on PMD-mapped THP:

    memset(buf, 0x5A, size);          // PMD-mapped THP, PMD dirty
    madvise(buf, size, MADV_FREE);    // PMD cleaned but left writable,
                                      // folio marked lazyfree
    memset(buf, 0x5A, size);          // hardware sets _PAGE_DIRTY again
    mprotect(buf, size, PROT_READ);   // pmd_modify() drops the dirty bit
    mprotect(buf, size, PROT_READ|PROT_WRITE);
    // ... memory pressure ...

Reclaim (e.g. under memcg pressure) then finds the lazyfree folio with
no dirty bit set anywhere and frees it in
__discard_anon_folio_pmd_locked(), even though the data was rewritten
after MADV_FREE; subsequent reads fault in fresh zero pages.  NUMA
hinting alone can trigger the same loss, as do_huge_pmd_numa_page()
restores the PMD through pmd_modify() as well.

PMD-mapped file THPs are affected too: mprotect()/NUMA hinting dropping
the dirty bit means rewritten data is never written back.

Fix it by keeping _PAGE_DIRTY in the preserved mask, exactly like
pte_modify() and pud_modify() do.  The existing
pmd_mksaveddirty()/pmd_clear_saveddirty() pair then performs the
hardware-dirty <-> saved-dirty transition based on the write bit,
preserving the shadow-stack encoding rules.

Closes: https://lore.kernel.org/r/CAJxLxMUGu1-L+O_nAONOwOXnS=cNbNApCWqdthRjd76LThtSPg@mail.gmail.com/
Fixes: bb3aadf7d446 ("x86/mm: Start actually marking _PAGE_SAVED_DIRTY")
Cc: stable@vger.kernel.org
Signed-off-by: Vernon Yang <yanglincheng@kylinos.cn>
---
 arch/x86/include/asm/pgtable.h | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/arch/x86/include/asm/pgtable.h b/arch/x86/include/asm/pgtable.h
index d5f4917c1edc..d551120a7c88 100644
--- a/arch/x86/include/asm/pgtable.h
+++ b/arch/x86/include/asm/pgtable.h
@@ -806,7 +806,7 @@ static inline pmd_t pmd_modify(pmd_t pmd, pgprot_t newprot)
 	pmdval_t val = pmd_val(pmd), oldval = val;
 	pmd_t pmd_result;
 
-	val &= (_HPAGE_CHG_MASK & ~_PAGE_DIRTY);
+	val &= _HPAGE_CHG_MASK;
 	val |= check_pgprot(newprot) & ~_HPAGE_CHG_MASK;
 	val = flip_protnone_guard(oldval, val, PHYSICAL_PMD_PAGE_MASK);
 
-- 
2.53.0

Re: [PATCH] x86/mm: Fix pmd_modify() dropping the dirty bit

[Andrew Morton]

On Thu,  3 Sep 2026 11:16:08 +0800 Vernon Yang <vernon2gm@gmail.com> wrote:

> From: Vernon Yang <yanglincheng@kylinos.cn>
> 
> pmd_modify() masks the old value with (_HPAGE_CHG_MASK & ~_PAGE_DIRTY),
> silently discarding the hardware dirty bit.  The subsequent
> pmd_mksaveddirty() call is supposed to transfer _PAGE_DIRTY into
> _PAGE_SAVED_DIRTY when write-protecting, but the dirty bit was already
> stripped from the value, so there is nothing left to transfer.
> 
> Contrast with pte_modify(), which keeps _PAGE_DIRTY_BITS in its mask,
> and pud_modify(), which keeps _HPAGE_CHG_MASK untouched: pmd_modify()
> is the odd one out.  Any pmd_modify() on a writable, dirty PMD loses
> the dirty state.
> 
> One visible consequence is data loss with MADV_FREE on PMD-mapped THP:
> 
>     memset(buf, 0x5A, size);          // PMD-mapped THP, PMD dirty
>     madvise(buf, size, MADV_FREE);    // PMD cleaned but left writable,
>                                       // folio marked lazyfree
>     memset(buf, 0x5A, size);          // hardware sets _PAGE_DIRTY again
>     mprotect(buf, size, PROT_READ);   // pmd_modify() drops the dirty bit
>     mprotect(buf, size, PROT_READ|PROT_WRITE);
>     // ... memory pressure ...
> 
> Reclaim (e.g. under memcg pressure) then finds the lazyfree folio with
> no dirty bit set anywhere and frees it in
> __discard_anon_folio_pmd_locked(), even though the data was rewritten
> after MADV_FREE; subsequent reads fault in fresh zero pages.  NUMA
> hinting alone can trigger the same loss, as do_huge_pmd_numa_page()
> restores the PMD through pmd_modify() as well.
> 
> PMD-mapped file THPs are affected too: mprotect()/NUMA hinting dropping
> the dirty bit means rewritten data is never written back.
> 
> Fix it by keeping _PAGE_DIRTY in the preserved mask, exactly like
> pte_modify() and pud_modify() do.  The existing
> pmd_mksaveddirty()/pmd_clear_saveddirty() pair then performs the
> hardware-dirty <-> saved-dirty transition based on the write bit,
> preserving the shadow-stack encoding rules.

Yeah, this is exactly what I came up with, using chatgpt.

> Closes: https://lore.kernel.org/r/CAJxLxMUGu1-L+O_nAONOwOXnS=cNbNApCWqdthRjd76LThtSPg@mail.gmail.com/

We definitely want a Reported-by: Orson here.  He obviously did a ton
of work on this, and it's the least we can do to thank him.  I'll add it.

> Fixes: bb3aadf7d446 ("x86/mm: Start actually marking _PAGE_SAVED_DIRTY")
> Cc: stable@vger.kernel.org
> Signed-off-by: Vernon Yang <yanglincheng@kylinos.cn>

Re: [PATCH] x86/mm: Fix pmd_modify() dropping the dirty bit

[Vernon Yang]

On Wed, Sep 02, 2026 at 09:10:36PM -0700, Andrew Morton wrote:
> On Thu,  3 Sep 2026 11:16:08 +0800 Vernon Yang <vernon2gm@gmail.com> wrote:
>
> > From: Vernon Yang <yanglincheng@kylinos.cn>
> >
> > pmd_modify() masks the old value with (_HPAGE_CHG_MASK & ~_PAGE_DIRTY),
> > silently discarding the hardware dirty bit.  The subsequent
> > pmd_mksaveddirty() call is supposed to transfer _PAGE_DIRTY into
> > _PAGE_SAVED_DIRTY when write-protecting, but the dirty bit was already
> > stripped from the value, so there is nothing left to transfer.
> >
> > Contrast with pte_modify(), which keeps _PAGE_DIRTY_BITS in its mask,
> > and pud_modify(), which keeps _HPAGE_CHG_MASK untouched: pmd_modify()
> > is the odd one out.  Any pmd_modify() on a writable, dirty PMD loses
> > the dirty state.
> >
> > One visible consequence is data loss with MADV_FREE on PMD-mapped THP:
> >
> >     memset(buf, 0x5A, size);          // PMD-mapped THP, PMD dirty
> >     madvise(buf, size, MADV_FREE);    // PMD cleaned but left writable,
> >                                       // folio marked lazyfree
> >     memset(buf, 0x5A, size);          // hardware sets _PAGE_DIRTY again
> >     mprotect(buf, size, PROT_READ);   // pmd_modify() drops the dirty bit
> >     mprotect(buf, size, PROT_READ|PROT_WRITE);
> >     // ... memory pressure ...
> >
> > Reclaim (e.g. under memcg pressure) then finds the lazyfree folio with
> > no dirty bit set anywhere and frees it in
> > __discard_anon_folio_pmd_locked(), even though the data was rewritten
> > after MADV_FREE; subsequent reads fault in fresh zero pages.  NUMA
> > hinting alone can trigger the same loss, as do_huge_pmd_numa_page()
> > restores the PMD through pmd_modify() as well.
> >
> > PMD-mapped file THPs are affected too: mprotect()/NUMA hinting dropping
> > the dirty bit means rewritten data is never written back.
> >
> > Fix it by keeping _PAGE_DIRTY in the preserved mask, exactly like
> > pte_modify() and pud_modify() do.  The existing
> > pmd_mksaveddirty()/pmd_clear_saveddirty() pair then performs the
> > hardware-dirty <-> saved-dirty transition based on the write bit,
> > preserving the shadow-stack encoding rules.
>
> Yeah, this is exactly what I came up with, using chatgpt.
>
> > Closes: https://lore.kernel.org/r/CAJxLxMUGu1-L+O_nAONOwOXnS=cNbNApCWqdthRjd76LThtSPg@mail.gmail.com/
>
> We definitely want a Reported-by: Orson here.  He obviously did a ton
> of work on this, and it's the least we can do to thank him.  I'll add it.

Yeah, I missed adding Reported-by: Orson, Sorry. Thank you for adding
it.

--
Cheers,
Vernon

Re: [PATCH] x86/mm: Fix pmd_modify() dropping the dirty bit

[Orson Peters]

Dear Andrew, Vernon,

Thanks for your quick reply, confirmation and patch.

What I failed to mention in the original report is that this isn't
just a hypothetical issue found by analyzing the code, some of our
users (of Polars, a data analytics library) have hit this in
production. "Live data can change silently" is never a fun issue
report to get.

Transparent huge pages + MADV_FREE + NUMA hinting + cgroup
limits isn't your typical mom-and-pop setup, but it is not an
unreasonable configuration either in high-performance computing. The
only missing ingredient when you use Polars on a multi-NUMA-region
machine running default Ubuntu configuration is the cgroup limits.

Considering that the impact is so severe if the bug does occur (silent
memory corruption), will there be backports of this patch? If I'm not
mistaken the offending line that introduced the bug was first released
in version 6.6.


Best,

Orson

On Thu, 3 Sept 2026 at 06:10, Andrew Morton <akpm@linux-foundation.org> wrote:
>
> On Thu,  3 Sep 2026 11:16:08 +0800 Vernon Yang <vernon2gm@gmail.com> wrote:
>
> > From: Vernon Yang <yanglincheng@kylinos.cn>
> >
> > pmd_modify() masks the old value with (_HPAGE_CHG_MASK & ~_PAGE_DIRTY),
> > silently discarding the hardware dirty bit.  The subsequent
> > pmd_mksaveddirty() call is supposed to transfer _PAGE_DIRTY into
> > _PAGE_SAVED_DIRTY when write-protecting, but the dirty bit was already
> > stripped from the value, so there is nothing left to transfer.
> >
> > Contrast with pte_modify(), which keeps _PAGE_DIRTY_BITS in its mask,
> > and pud_modify(), which keeps _HPAGE_CHG_MASK untouched: pmd_modify()
> > is the odd one out.  Any pmd_modify() on a writable, dirty PMD loses
> > the dirty state.
> >
> > One visible consequence is data loss with MADV_FREE on PMD-mapped THP:
> >
> >     memset(buf, 0x5A, size);          // PMD-mapped THP, PMD dirty
> >     madvise(buf, size, MADV_FREE);    // PMD cleaned but left writable,
> >                                       // folio marked lazyfree
> >     memset(buf, 0x5A, size);          // hardware sets _PAGE_DIRTY again
> >     mprotect(buf, size, PROT_READ);   // pmd_modify() drops the dirty bit
> >     mprotect(buf, size, PROT_READ|PROT_WRITE);
> >     // ... memory pressure ...
> >
> > Reclaim (e.g. under memcg pressure) then finds the lazyfree folio with
> > no dirty bit set anywhere and frees it in
> > __discard_anon_folio_pmd_locked(), even though the data was rewritten
> > after MADV_FREE; subsequent reads fault in fresh zero pages.  NUMA
> > hinting alone can trigger the same loss, as do_huge_pmd_numa_page()
> > restores the PMD through pmd_modify() as well.
> >
> > PMD-mapped file THPs are affected too: mprotect()/NUMA hinting dropping
> > the dirty bit means rewritten data is never written back.
> >
> > Fix it by keeping _PAGE_DIRTY in the preserved mask, exactly like
> > pte_modify() and pud_modify() do.  The existing
> > pmd_mksaveddirty()/pmd_clear_saveddirty() pair then performs the
> > hardware-dirty <-> saved-dirty transition based on the write bit,
> > preserving the shadow-stack encoding rules.
>
> Yeah, this is exactly what I came up with, using chatgpt.
>
> > Closes: https://lore.kernel.org/r/CAJxLxMUGu1-L+O_nAONOwOXnS=cNbNApCWqdthRjd76LThtSPg@mail.gmail.com/
>
> We definitely want a Reported-by: Orson here.  He obviously did a ton
> of work on this, and it's the least we can do to thank him.  I'll add it.
>
> > Fixes: bb3aadf7d446 ("x86/mm: Start actually marking _PAGE_SAVED_DIRTY")
> > Cc: stable@vger.kernel.org
> > Signed-off-by: Vernon Yang <yanglincheng@kylinos.cn>
>

Re: [PATCH] x86/mm: Fix pmd_modify() dropping the dirty bit

[Dave Hansen]

On 9/3/26 00:34, Orson Peters wrote:
> Considering that the impact is so severe if the bug does occur (silent
> memory corruption), will there be backports of this patch? If I'm not
> mistaken the offending line that introduced the bug was first released
> in version 6.6.

Hi Orson,

It's tagged for stable@ so it should hit the stable trees at some point
and make it into the 6.6 series.

Re: [PATCH] x86/mm: Fix pmd_modify() dropping the dirty bit

[Kiryl Shutsemau]

On Thu, Sep 03, 2026 at 11:16:08AM +0800, Vernon Yang wrote:
> From: Vernon Yang <yanglincheng@kylinos.cn>
> 
> pmd_modify() masks the old value with (_HPAGE_CHG_MASK & ~_PAGE_DIRTY),
> silently discarding the hardware dirty bit.  The subsequent
> pmd_mksaveddirty() call is supposed to transfer _PAGE_DIRTY into
> _PAGE_SAVED_DIRTY when write-protecting, but the dirty bit was already
> stripped from the value, so there is nothing left to transfer.
> 
> Contrast with pte_modify(), which keeps _PAGE_DIRTY_BITS in its mask,
> and pud_modify(), which keeps _HPAGE_CHG_MASK untouched: pmd_modify()
> is the odd one out.  Any pmd_modify() on a writable, dirty PMD loses
> the dirty state.
> 
> One visible consequence is data loss with MADV_FREE on PMD-mapped THP:
> 
>     memset(buf, 0x5A, size);          // PMD-mapped THP, PMD dirty
>     madvise(buf, size, MADV_FREE);    // PMD cleaned but left writable,
>                                       // folio marked lazyfree
>     memset(buf, 0x5A, size);          // hardware sets _PAGE_DIRTY again
>     mprotect(buf, size, PROT_READ);   // pmd_modify() drops the dirty bit
>     mprotect(buf, size, PROT_READ|PROT_WRITE);
>     // ... memory pressure ...
> 
> Reclaim (e.g. under memcg pressure) then finds the lazyfree folio with
> no dirty bit set anywhere and frees it in
> __discard_anon_folio_pmd_locked(), even though the data was rewritten
> after MADV_FREE; subsequent reads fault in fresh zero pages.  NUMA
> hinting alone can trigger the same loss, as do_huge_pmd_numa_page()
> restores the PMD through pmd_modify() as well.
> 
> PMD-mapped file THPs are affected too: mprotect()/NUMA hinting dropping
> the dirty bit means rewritten data is never written back.
> 
> Fix it by keeping _PAGE_DIRTY in the preserved mask, exactly like
> pte_modify() and pud_modify() do.  The existing
> pmd_mksaveddirty()/pmd_clear_saveddirty() pair then performs the
> hardware-dirty <-> saved-dirty transition based on the write bit,
> preserving the shadow-stack encoding rules.
> 
> Closes: https://lore.kernel.org/r/CAJxLxMUGu1-L+O_nAONOwOXnS=cNbNApCWqdthRjd76LThtSPg@mail.gmail.com/
> Fixes: bb3aadf7d446 ("x86/mm: Start actually marking _PAGE_SAVED_DIRTY")

Hm. I don't understand why would this commit explicitly exclude
_PAGE_DIRTY from the mask:

-       val &= _HPAGE_CHG_MASK;
+       val &= (_HPAGE_CHG_MASK & ~_PAGE_DIRTY);

Rick, could you comment? It doesn't look like a typo.

-- 
  Kiryl Shutsemau / Kirill A. Shutemov

Re: [PATCH] x86/mm: Fix pmd_modify() dropping the dirty bit

[Dave Hansen]

On 9/3/26 04:54, Kiryl Shutsemau wrote:
>> Closes: https://lore.kernel.org/r/CAJxLxMUGu1-L+O_nAONOwOXnS=cNbNApCWqdthRjd76LThtSPg@mail.gmail.com/
>> Fixes: bb3aadf7d446 ("x86/mm: Start actually marking _PAGE_SAVED_DIRTY")
> Hm. I don't understand why would this commit explicitly exclude
> _PAGE_DIRTY from the mask:
> 
> -       val &= _HPAGE_CHG_MASK;
> +       val &= (_HPAGE_CHG_MASK & ~_PAGE_DIRTY);
> 
> Rick, could you comment? It doesn't look like a typo.

My guess is that it's some remnant from an earlier version of the patch.
The asymmetry with pte_modify() vs. pmd_modify() just can't be explained
any other way. It _might_ have been some attempt to say, "Hey
_PAGE_DIRTY is now a part of the pgprot_t since it's part of the
'permissions' of a shadow stack PTE" that got abandoned.

But I don't see anything wrong with the fix at all.

It does give me pause that this has been losing user data for so long,
but it must just be a weird combination of features that few folks use
together (huge pages + MADV_FREE).

Re: [PATCH] x86/mm: Fix pmd_modify() dropping the dirty bit

[Andrew Morton]

On Thu, 3 Sep 2026 07:18:54 -0700 Dave Hansen <dave.hansen@intel.com> wrote:

> On 9/3/26 04:54, Kiryl Shutsemau wrote:
> >> Closes: https://lore.kernel.org/r/CAJxLxMUGu1-L+O_nAONOwOXnS=cNbNApCWqdthRjd76LThtSPg@mail.gmail.com/
> >> Fixes: bb3aadf7d446 ("x86/mm: Start actually marking _PAGE_SAVED_DIRTY")
> > Hm. I don't understand why would this commit explicitly exclude
> > _PAGE_DIRTY from the mask:
> > 
> > -       val &= _HPAGE_CHG_MASK;
> > +       val &= (_HPAGE_CHG_MASK & ~_PAGE_DIRTY);
> > 
> > Rick, could you comment? It doesn't look like a typo.
> 
> My guess is that it's some remnant from an earlier version of the patch.
> The asymmetry with pte_modify() vs. pmd_modify() just can't be explained
> any other way. It _might_ have been some attempt to say, "Hey
> _PAGE_DIRTY is now a part of the pgprot_t since it's part of the
> 'permissions' of a shadow stack PTE" that got abandoned.
> 
> But I don't see anything wrong with the fix at all.
> 
> It does give me pause that this has been losing user data for so long,
> but it must just be a weird combination of features that few folks use
> together (huge pages + MADV_FREE).
> 

And with heavy reclaim pressure.

Re: [PATCH] x86/mm: Fix pmd_modify() dropping the dirty bit

[Edgecombe, Rick P]

On Thu, 2026-09-03 at 07:18 -0700, Dave Hansen wrote:
> On 9/3/26 04:54, Kiryl Shutsemau wrote:
> > > Closes: https://lore.kernel.org/r/CAJxLxMUGu1-L+O_nAONOwOXnS=cNbNApCWqdthRjd76LThtSPg@mail.gmail.com/
> > > Fixes: bb3aadf7d446 ("x86/mm: Start actually marking _PAGE_SAVED_DIRTY")
> > Hm. I don't understand why would this commit explicitly exclude
> > _PAGE_DIRTY from the mask:
> > 
> > -       val &= _HPAGE_CHG_MASK;
> > +       val &= (_HPAGE_CHG_MASK & ~_PAGE_DIRTY);
> > 
> > Rick, could you comment? It doesn't look like a typo.
> 
> My guess is that it's some remnant from an earlier version of the patch.
> The asymmetry with pte_modify() vs. pmd_modify() just can't be explained
> any other way. It _might_ have been some attempt to say, "Hey
> _PAGE_DIRTY is now a part of the pgprot_t since it's part of the
> 'permissions' of a shadow stack PTE" that got abandoned.
> 
> But I don't see anything wrong with the fix at all.
> 
> It does give me pause that this has been losing user data for so long,
> but it must just be a weird combination of features that few folks use
> together (huge pages + MADV_FREE).

Oof. Looking back through the patch history, the dirty bit used to be handled
separately, such that the stripping was needed. Like this:

 static inline pte_t pte_modify(pte_t pte, pgprot_t newprot)
 {
	pteval_t _page_chg_mask_no_dirty = _PAGE_CHG_MASK & ~_PAGE_DIRTY;
 	pteval_t val = pte_val(pte), oldval = val;
	pte_t pte_result;
 
 	/*
 	 * Chop off the NX bit (if present), and add the NX portion of
 	 * the newprot (if present):
 	 */
	val &= _page_chg_mask_no_dirty;
	val |= check_pgprot(newprot) & ~_page_chg_mask_no_dirty;
 	val = flip_protnone_guard(oldval, val, PTE_PFN_MASK);

	pte_result = __pte(val);

	/*
	 * Dirty bit is not preserved above so it can be done
	 * in a special way for the shadow stack case, where it
	 * may need to set _PAGE_COW. __pte_mkdirty() will do this in
	 * the case of shadow stack.
	 */
	if (pte_dirty(pte))
		pte_result = __pte_mkdirty(pte_result, false);

	return pte_result;
 }
 
 static inline pmd_t pmd_modify(pmd_t pmd, pgprot_t newprot)
 {
	pteval_t _hpage_chg_mask_no_dirty = _HPAGE_CHG_MASK & ~_PAGE_DIRTY;
 	pmdval_t val = pmd_val(pmd), oldval = val;
	pmd_t pmd_result;

	val &= _hpage_chg_mask_no_dirty;
	val |= check_pgprot(newprot) & ~_hpage_chg_mask_no_dirty;
 	val = flip_protnone_guard(oldval, val, PHYSICAL_PMD_PAGE_MASK);


	pmd_result = __pmd(val);

	/*
	 * Dirty bit is not preserved above so it can be done
	 * in a special way for the shadow stack case, where it
	 * may need to set _PAGE_COW. __pmd_mkdirty() will do this in
	 * the case of shadow stack.
	 */
	if (pmd_dirty(pmd))
		pmd_result = __pmd_mkdirty(pmd_result, false);

	return pmd_result;
 }

The dirty bit was removed from the pte, then handled separately to share logic
in the mkdirty helpers. During development it was changed to do the necessary
adjustments depending on the dirty bit in 'val', but the mask adjustment on the
pmd_modify() side didn't get updated. So I don't remember or see any intention
for the difference.

I can't find anything back then that would have prevented it. The bug cause and
fix looks correct to me.

Re: [PATCH] x86/mm: Fix pmd_modify() dropping the dirty bit

[Dave Hansen]

On 9/3/26 10:58, Edgecombe, Rick P wrote:
> I can't find anything back then that would have prevented it. The bug cause and
> fix looks correct to me.

Is that a Reviewed-by? ;)

Re: [PATCH] x86/mm: Fix pmd_modify() dropping the dirty bit

[Edgecombe, Rick P]

On Thu, 2026-09-03 at 11:01 -0700, Dave Hansen wrote:
> On 9/3/26 10:58, Edgecombe, Rick P wrote:
> > I can't find anything back then that would have prevented it. The bug cause and
> > fix looks correct to me.
> 
> Is that a Reviewed-by? ;)

Reviewed-by: Rick Edgecombe <rick.p.edgecombe@intel.com>