[PATCH] Fix incorrect flush address in direct page table reclaim

[PATCH] Fix incorrect flush address in direct page table reclaim

[luto]

From: Andy Lutomirski <luto@kernel.org>

When zap_pte_range reclaims a page table, it does:

    pte_free_tlb(tlb, pmd_pgtable(pmdval), addr);

and this is unconditionally wrong: if this code executes, addr *always*
points one past the end of the range covered by the table.  The addr
parameter is used to flush the TLB (really the paging-structure-cache)
to drop references to the to-be-freed table, and any architecture that
cares about the parameter will flush the wrong address.  (But they'll
still free the correct page).

I think it's worth contemplating why the kernel works at all.

If we hit the offending line of code, we will first clear the PMD entry
(line 1954, zap_empty_pte_table), then we will issue pending flushes if
force_flush is set (tlb_flush_mmu_tlbonly(tlb)), then we will skip the
retry on line 1979 (phew!), and then we will do the offending
pte_free_tlb call.  *Or* we will clear the PMD entry immediately before
pte_free_tlb (line 1983, zap_pte_table_if_empty).

If we have any pending flushes (i.e. we actually zapped any last-level
entries) at the time we clear the PMD entry, then the flush really ought
to flush all references to the table (Linus certainly seems to think it
will on all architectures [0]).

The condition under which we have no accumulated flushes at the time of
the clear is very complex (the whole zap_pte_range function has absurdly
complex control flow).  If we do hit the bad case, then we will end up
clearing the PMD entry after the last time the range is flushed, and any
CPU is free to cache a reference to the (empty) page table.  If this
happens due to an ordinary read or write, it would segfault, so it would
be rare.  But the cache could be speculatively filled as well.  Then
we'll flush the wrong address and then free and possibly reuse the
table.

On x86, even flushing the wrong address works on non-KPTI Intel systems
because INVLPG flushes *all* paging-structure-caches, not just the ones
for the target address.  But INVPCID does not, and flush_tlb_one_user
will use INVPCID if it's available.  And then we're toast.  AMD systems
are more susceptible: we set the EFER.TCE bit, which makes even INVLPG
only flush the target address.

P.S. IMO zap_pte_range is a mess.  The control flow is excessively
complex.  The direct_reclaim variable itself has a confused meaning --
for the first part of the function it means, approximately, "we should
free the table if can_reclaim_pt".  But, later on, it means "we ALREADY
reclaimed the table".  And the goto retry on line 1979 is IMO just
asking for trouble if the condition ever changes such that it might
happen after clearing the PMD.

I think this might fix an issue in ripgrep reported here:
https://github.com/BurntSushi/ripgrep/issues/3494

[0] https://lore.kernel.org/all/CA+55aFzBggoXtNXQeng5d_mRoDnaMBE5Y+URs+PHR67nUpMtaw@mail.gmail.com/T/#u

Fixes: 4c640eb4181c ("mm: move pte table reclaim code to memory.c")
Cc: David Hildenbrand (Red Hat) <david@kernel.org>
Cc: Qi Zheng <zhengqi.arch@bytedance.com>
Cc: Liam Howlett <liam.howlett@oracle.com>
Cc: "Liam R. Howlett" <Liam.Howlett@oracle.com>
Cc: Lorenzo Stoakes <lorenzo.stoakes@oracle.com>
Cc: Michal Hocko <mhocko@suse.com>
Cc: Mike Rapoport <rppt@kernel.org>
Cc: Suren Baghdasaryan <surenb@google.com>
Cc: Vlastimil Babka <vbabka@suse.cz>
Cc: Andrew Morton <akpm@linux-foundation.org>.org>
Cc: stable@vger.kernel.org
Signed-off-by: Andy Lutomirski <luto@kernel.org>
---
 mm/memory.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/mm/memory.c b/mm/memory.c
index 86a973119bd4..13b70861c8a3 100644
--- a/mm/memory.c
+++ b/mm/memory.c
@@ -1981,7 +1981,7 @@ static unsigned long zap_pte_range(struct mmu_gather *tlb,
 
 	if (can_reclaim_pt) {
 		if (direct_reclaim || zap_pte_table_if_empty(mm, pmd, start, &pmdval)) {
-			pte_free_tlb(tlb, pmd_pgtable(pmdval), addr);
+			pte_free_tlb(tlb, pmd_pgtable(pmdval), start);
 			mm_dec_nr_ptes(mm);
 		}
 	}
-- 

Re: [PATCH] Fix incorrect flush address in direct page table reclaim

[Andrew Morton]

On Mon,  3 Aug 2026 17:37:08 -0700 luto@kernel.org wrote:

> From: Andy Lutomirski <luto@kernel.org>
> 
> When zap_pte_range reclaims a page table, it does:
> 
>     pte_free_tlb(tlb, pmd_pgtable(pmdval), addr);
> 
> and this is unconditionally wrong: if this code executes, addr *always*
> points one past the end of the range covered by the table.  The addr
> parameter is used to flush the TLB (really the paging-structure-cache)
> to drop references to the to-be-freed table, and any architecture that
> cares about the parameter will flush the wrong address.  (But they'll
> still free the correct page).
> 
> I think it's worth contemplating why the kernel works at all.
> 
> If we hit the offending line of code, we will first clear the PMD entry
> (line 1954, zap_empty_pte_table), then we will issue pending flushes if
> force_flush is set (tlb_flush_mmu_tlbonly(tlb)), then we will skip the
> retry on line 1979 (phew!), and then we will do the offending
> pte_free_tlb call.  *Or* we will clear the PMD entry immediately before
> pte_free_tlb (line 1983, zap_pte_table_if_empty).
> 
> If we have any pending flushes (i.e. we actually zapped any last-level
> entries) at the time we clear the PMD entry, then the flush really ought
> to flush all references to the table (Linus certainly seems to think it
> will on all architectures [0]).
> 
> The condition under which we have no accumulated flushes at the time of
> the clear is very complex (the whole zap_pte_range function has absurdly
> complex control flow).  If we do hit the bad case, then we will end up
> clearing the PMD entry after the last time the range is flushed, and any
> CPU is free to cache a reference to the (empty) page table.  If this
> happens due to an ordinary read or write, it would segfault, so it would
> be rare.  But the cache could be speculatively filled as well.  Then
> we'll flush the wrong address and then free and possibly reuse the
> table.
> 
> On x86, even flushing the wrong address works on non-KPTI Intel systems
> because INVLPG flushes *all* paging-structure-caches, not just the ones
> for the target address.  But INVPCID does not, and flush_tlb_one_user
> will use INVPCID if it's available.  And then we're toast.  AMD systems
> are more susceptible: we set the EFER.TCE bit, which makes even INVLPG
> only flush the target address.
> 
> P.S. IMO zap_pte_range is a mess.  The control flow is excessively
> complex.  The direct_reclaim variable itself has a confused meaning --
> for the first part of the function it means, approximately, "we should
> free the table if can_reclaim_pt".  But, later on, it means "we ALREADY
> reclaimed the table".  And the goto retry on line 1979 is IMO just
> asking for trouble if the condition ever changes such that it might
> happen after clearing the PMD.
> 
> I think this might fix an issue in ripgrep reported here:
> https://github.com/BurntSushi/ripgrep/issues/3494

Huh, cool.  Very recently I was scratching my head at Daniel's ripgrep
report.  Upon which he obviously did a ton of work (many thanks).

	https://github.com/dfoxfranke/ripgrep-3494-analysis

Daniel also pointed a finger at 4c640eb4181c ("mm: move pte table
reclaim code to memory.c").

The manifestation is that a tiny race window causes the zero page to
magically appear where an anon page was expected.

Anyway, I see Daniel is testing this patch.  Please let us know the
result.  (I think I found his email address).

I can't find an email address for BurntSushi.

Re: [PATCH] Fix incorrect flush address in direct page table reclaim

[Andy Lutomirski]

On Mon, Aug 3, 2026 at 6:30 PM Andrew Morton <akpm@linux-foundation.org> wrote:
>
> On Mon,  3 Aug 2026 17:37:08 -0700 luto@kernel.org wrote:
>
> > From: Andy Lutomirski <luto@kernel.org>
> >
> > When zap_pte_range reclaims a page table, it does:
> >
> >     pte_free_tlb(tlb, pmd_pgtable(pmdval), addr);
> >
> > and this is unconditionally wrong: if this code executes, addr *always*
> > points one past the end of the range covered by the table.  The addr
> > parameter is used to flush the TLB (really the paging-structure-cache)
> > to drop references to the to-be-freed table, and any architecture that
> > cares about the parameter will flush the wrong address.  (But they'll
> > still free the correct page).
> >
> > I think it's worth contemplating why the kernel works at all.
> >
> > If we hit the offending line of code, we will first clear the PMD entry
> > (line 1954, zap_empty_pte_table), then we will issue pending flushes if
> > force_flush is set (tlb_flush_mmu_tlbonly(tlb)), then we will skip the
> > retry on line 1979 (phew!), and then we will do the offending
> > pte_free_tlb call.  *Or* we will clear the PMD entry immediately before
> > pte_free_tlb (line 1983, zap_pte_table_if_empty).
> >
> > If we have any pending flushes (i.e. we actually zapped any last-level
> > entries) at the time we clear the PMD entry, then the flush really ought
> > to flush all references to the table (Linus certainly seems to think it
> > will on all architectures [0]).
> >
> > The condition under which we have no accumulated flushes at the time of
> > the clear is very complex (the whole zap_pte_range function has absurdly
> > complex control flow).  If we do hit the bad case, then we will end up
> > clearing the PMD entry after the last time the range is flushed, and any
> > CPU is free to cache a reference to the (empty) page table.  If this
> > happens due to an ordinary read or write, it would segfault, so it would
> > be rare.  But the cache could be speculatively filled as well.  Then
> > we'll flush the wrong address and then free and possibly reuse the
> > table.
> >
> > On x86, even flushing the wrong address works on non-KPTI Intel systems
> > because INVLPG flushes *all* paging-structure-caches, not just the ones
> > for the target address.  But INVPCID does not, and flush_tlb_one_user
> > will use INVPCID if it's available.  And then we're toast.  AMD systems
> > are more susceptible: we set the EFER.TCE bit, which makes even INVLPG
> > only flush the target address.
> >
> > P.S. IMO zap_pte_range is a mess.  The control flow is excessively
> > complex.  The direct_reclaim variable itself has a confused meaning --
> > for the first part of the function it means, approximately, "we should
> > free the table if can_reclaim_pt".  But, later on, it means "we ALREADY
> > reclaimed the table".  And the goto retry on line 1979 is IMO just
> > asking for trouble if the condition ever changes such that it might
> > happen after clearing the PMD.
> >
> > I think this might fix an issue in ripgrep reported here:
> > https://github.com/BurntSushi/ripgrep/issues/3494
>
> Huh, cool.  Very recently I was scratching my head at Daniel's ripgrep
> report.  Upon which he obviously did a ton of work (many thanks).
>
>         https://github.com/dfoxfranke/ripgrep-3494-analysis

I'm pretty sure that's AI-generated.

>
> Daniel also pointed a finger at 4c640eb4181c ("mm: move pte table
> reclaim code to memory.c").
>
> The manifestation is that a tiny race window causes the zero page to
> magically appear where an anon page was expected.

The analysis says:

pagemap reports the page present, soft-dirty, with PFN 0 — the
kernel's zero page.

ahem.  I would be quite surprised if pfn 0 is the zero page.

If whatever code the AI wrote that generated its trace is actually
correct and there was a PTE entry with pfn 0 and the present bit set,
then maybe we have a genuine race that corrupts a page table.  But the
analysis seems quite sloppy and I'm disinclined to trust it too much.
(A quick experiment suggests that even GPT-5.6-Sol high (which is
supposedly pretty good) will rapidly go off the rails if you ask it a
leading-in-the-wrong-direction question about the kernel.)

In any case, the original reporter was using an AMD system, and the
bug seems extremely sensitive to tiny details, both of which are
consistent with the possibility of a narrow page table caching bug
like this.  I'm frankly a bit amazed that he and his AI were able to
reproduce it as many times as they apparently did.  I wonder if the
specific offending assembly caused the CPU to somewhat reliably
speculatively fill the paging structure cache during the narrow window
after the last correct flush and before the PMD entry was cleared --
if so, this would be consistent with the AI's observation that even
tiny changes to the asm seemed to make the bug go away.

--Andy

Re: [PATCH] Fix incorrect flush address in direct page table reclaim

[Andy Lutomirski]

> On Aug 4, 2026, at 2:34 AM, David Hildenbrand (Arm) <david@kernel.org> wrote:
> 
> On 8/4/26 11:25, Qi Zheng wrote:
>> 
>> 
>>> On 8/4/26 5:05 PM, David Hildenbrand (Arm) wrote:
>>> On 8/4/26 02:37, luto@kernel.org wrote:
>>>> From: Andy Lutomirski <luto@kernel.org>
>>>> 
>>>> When zap_pte_range reclaims a page table, it does:
>>>> 
>>>>      pte_free_tlb(tlb, pmd_pgtable(pmdval), addr);
>>>> 
>>>> and this is unconditionally wrong: if this code executes, addr *always*
>>>> points one past the end of the range covered by the table.  The addr
>>>> parameter is used to flush the TLB (really the paging-structure-cache)
>>>> to drop references to the to-be-freed table, and any architecture that
>>>> cares about the parameter will flush the wrong address.  (But they'll
>>>> still free the correct page).
>>>> 
>>>> I think it's worth contemplating why the kernel works at all.
>>> 
>>> Note that this only triggers when someone does e.g., a MADV_DONTNEED over
>>> a large enough range (covering at least a full PTE table).
>>> 
>>> So this isn't the ordinary munmap()/exit() page table reclaim code.
>>> 
>>> I'm still surprised that it took so long to show up; likely we need more
>>> targeted tests for PT_RECLAIM that
>> 
>> We backported the PT_RECLAIM to our internal tree a while ago (excluding
>> the rework patch being fixed here), and it has been running stably ever
>> since.
> 
> I'm wondering whether we should have a selftest that triggers this exact scenario:
> 
> (a) Populate page table(s)
> (b) MADV_DONTNEED to reclaim the page table(s)
> (c) Allocate plenty of pages for another purpose (draining PCP where the page
>    table likely ended up) and fill them with some data that looks like present page
>    table entries.
> (d) Re-access the reclaimed VA space, expecting re-population of anonymous
>    pages.
> 

If you want a better chance of hitting this bug, you could try:

- First DONTNEED the range in multiple calls so none of them reclaim the table.

- Then PREFETCH the range (or use your favorite speculative read gadget or even do a real read and ignore the signal).  Maybe also do this from multiple CPUs.

- Trigger reclaim

Then do c and d.

The result might be so specialized that it would be of dubious value.

> I can find someone to work on that.
> 
> Any other ideas what we could check? We could certainly also check that the page table
> was actually reclaimed during MADV_DONTNEED.
> 
> --
> Cheers,
> 
> David

Re: [PATCH] Fix incorrect flush address in direct page table reclaim

[Linus Torvalds]

On Tue, 4 Aug 2026 at 02:49, Vlastimil Babka (SUSE) <vbabka@kernel.org> wrote:
>
> +Cc Linus for nerd sniping purposes

I don't know what to say - the patch looks obviously correct to me,
and I certainly can't disagree with the verbiage in the commit message
either.

The fact that the current code looks to be *so* broken makes me a bit
nervous that there's something subtle going on, but I *think* it's all
just a very non-subtle bug and it just happens to work because the
whole zap code typically uses the mmu_gather infrastructure and other
things end up resulting in the tlb being flushed regardless. But
that's just me waving my hands so wildly that I think I had lift-off
for a moment.

                Linus

Re: [PATCH] Fix incorrect flush address in direct page table reclaim

[David Hildenbrand (Arm)]

On 8/4/26 15:43, Andy Lutomirski wrote:
> 
>> On Aug 4, 2026, at 2:34 AM, David Hildenbrand (Arm) <david@kernel.org> wrote:
>>
>> On 8/4/26 11:25, Qi Zheng wrote:
>>>
>>>
>>>
>>> We backported the PT_RECLAIM to our internal tree a while ago (excluding
>>> the rework patch being fixed here), and it has been running stably ever
>>> since.
>>
>> I'm wondering whether we should have a selftest that triggers this exact scenario:
>>
>> (a) Populate page table(s)
>> (b) MADV_DONTNEED to reclaim the page table(s)
>> (c) Allocate plenty of pages for another purpose (draining PCP where the page
>>    table likely ended up) and fill them with some data that looks like present page
>>    table entries.
>> (d) Re-access the reclaimed VA space, expecting re-population of anonymous
>>    pages.
>>
> 
> If you want a better chance of hitting this bug, you could try:
> 
> - First DONTNEED the range in multiple calls so none of them reclaim the table.

Not required. If MADV_DONTNEED spans the full page table we always reclaim if it
is empty after MADV_DONTNEED (there are some oddities around uffd-wp handling,
which wouldn't apply here).

> 
> - Then PREFETCH the range (or use your favorite speculative read gadget or even do a real read and ignore the signal).  Maybe also do this from multiple CPUs.
> 

Not sure if that's required, but if that's required to trigger it that certainly
makes it less elegant.

> - Trigger reclaim

Also not sure if that one would really help.

> 
> Then do c and d.
> 
> The result might be so specialized that it would be of dubious value.
> 

Right, if we cannot easily trigger it then it's way to special.

-- 
Cheers,

David

Re: [PATCH] Fix incorrect flush address in direct page table reclaim

[David Hildenbrand (Arm)]

On 8/4/26 11:34, David Hildenbrand (Arm) wrote:
> On 8/4/26 11:25, Qi Zheng wrote:
>>
>>
>> On 8/4/26 5:05 PM, David Hildenbrand (Arm) wrote:
>>>
>>> Note that this only triggers when someone does e.g., a MADV_DONTNEED over
>>> a large enough range (covering at least a full PTE table).
>>>
>>> So this isn't the ordinary munmap()/exit() page table reclaim code.
>>>
>>> I'm still surprised that it took so long to show up; likely we need more
>>> targeted tests for PT_RECLAIM that
>>
>> We backported the PT_RECLAIM to our internal tree a while ago (excluding
>> the rework patch being fixed here), and it has been running stably ever
>> since.
> 
> I'm wondering whether we should have a selftest that triggers this exact scenario:
> 
> (a) Populate page table(s)
> (b) MADV_DONTNEED to reclaim the page table(s)
> (c) Allocate plenty of pages for another purpose (draining PCP where the page
>     table likely ended up) and fill them with some data that looks like present page
>     table entries.
> (d) Re-access the reclaimed VA space, expecting re-population of anonymous
>     pages.

FWIW, I tried to reproduce above and on x86 it's really not able to trigger
easily (as you say, might require some prefetching tricks).

I was able to trigger "something" on arm64, which is just an endless loop
between HW saying "translation exception" and the MM logic saying "looks good
me, continue".

As soon as the walk cache gets flushed (due to some concurrent activity in the
system), the reproducer continues.

-- 
Cheers,

David

Re: [PATCH] Fix incorrect flush address in direct page table reclaim

[David Hildenbrand (Arm)]

On 8/4/26 17:54, Linus Torvalds wrote:
> On Tue, 4 Aug 2026 at 02:49, Vlastimil Babka (SUSE) <vbabka@kernel.org> wrote:
>>
>> +Cc Linus for nerd sniping purposes
> 
> I don't know what to say - the patch looks obviously correct to me,
> and I certainly can't disagree with the verbiage in the commit message
> either.
Can we get that fix into 7.2?

-- 
Cheers,

David

Re: [PATCH] Fix incorrect flush address in direct page table reclaim

[Linus Torvalds]

On Tue, 4 Aug 2026 at 23:04, David Hildenbrand (Arm) <david@kernel.org> wrote:
>
> Can we get that fix into 7.2?

Applied to my tree - everybody seems to agree it's obviously the correct fix.

           Linus

Re: [PATCH] Fix incorrect flush address in direct page table reclaim

[David Hildenbrand (Arm)]

On 8/5/26 17:02, Linus Torvalds wrote:
> On Tue, 4 Aug 2026 at 23:04, David Hildenbrand (Arm) <david@kernel.org> wrote:
>>
>> Can we get that fix into 7.2?
> 
> Applied to my tree - everybody seems to agree it's obviously the correct fix.

Thanks!

-- 
Cheers,

David