Improve page refcount scalability

Improve page refcount scalability

[Linus Torvalds]

So I've shown this series to people in private and talked about it at
the kernel maintainer summit, so some of you have seen earlier
versions of this already.

This was triggered by several attempts at improving on the scalability
of page ref handling over the last couple of years, particularly for
the common case of normal [p]read() system calls. Willy reported real
users having scalability issues in that area a couple of years ago in
a discussion about buffereed IO scalability limits:

  https://lore.kernel.org/all/Zds8T9O4AYAmdS9d@casper.infradead.org/

and I had a very rough first draft in that discussion:

  https://lore.kernel.org/all/CAHk-=wjOogaW0yLoUqQ0WfQ=etPA4cOFLy56VYCnHVU_DOMLrg@mail.gmail.com/

and it was broken but close enough to working that I kept using it
locally in my own use for a year and a half:

  https://lore.kernel.org/all/CAHk-=wj00-nGmXEkxY=-=Z_qP6kiGUziSFvxHJ9N-cLWry5zpA@mail.gmail.com/

until people pointed out just how broken it was.

Then Kiryl had a slightly fixed-up version of that approach almost
exactly a year ago:

  https://lore.kernel.org/all/20251017141536.577466-1-kirill@shutemov.name/

but that too had problems.

After that, Ilya Gladyshev stepped into the ring with a very different
approach to it, which is now up to version 6:

  https://lore.kernel.org/all/cover.1789239015.git.ilya.gladyshev@linux.dev/

and still has some outstanding known fundamental issues.

So for the kernel maintainer summit this year, where one of the big
topics was going to be LLM use in kernel development, I decided that
hey, I will try to actually *solve* this thing using an LLM to help
with the issues these things have had.

[ I also did another series for a problem with the assoc-array
library. I'll post that separately - that was a lot more pedestrian
"just clean up the code in order to fix a bug" kind of case - not at
all as interesting, but a fallback in case this more interesting
problem were to not pan out. ]

Anyway, the end result is in my kernel tree in the 'rcu-free-v3'
branch in my public git tree. My 'v2' never saw the light of day
because I kept cleaning it up, but an earlier version with some more
background is available under the name 'noref-filemap-read-with-llm'
and has an empty commit with more commentary and background at the
top. I didn't do that cover-letter for this v3 version, and it talks
about some issues that already got resolved (and others that I don't
think are actual real problems, but should be discussed).

The fairly short series of ten patches basically enhances folio lookup
by noticing when folios are hot, and setting a "delay relasing this
folio until after a RCU period", and then using that RCU delay it does
not just the folio lookup itself under RCU (which we've done forever),
but also the actual folio data accesses themselves under RCU. No
refcount taking needed for the fastpath where we can just read the
data from the folio entirely in an RCU-protected region when the user
copy doesn't cause page faults.

Unlike my original approach - because it now actually delays the
freeing of the underlying folio data - there's no need for any double
buffering, and a lot of the original limitations of my original patch
go away too. That improves performance quite a lot. And by "quite a
lot"  I mean "I see a 23x performance increase on my Ryzen
threadripper". Admittedly that is a very artificial case of reading
the same small part of a page over and over in a threaded loop on all
CPUs, so any real life impact will be much much less.

Anyway, I wasn't planning on posting the individual patches, I think
we all use git by now and probably find it easier than fetching
patches from the list. So it's all at

    git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux rcu-free-v3

and the commit messages are a bit more verbose than I would have done
it myself without tool use, but maybe it's better that way.

The showcase number is in the "mm/filemap: read from the page cache
without folio references" commit:

                               reads/s     cycles    instructions
                                          per read     per read
   ...
  64 threads,           before   13.3 M  19857-19939   1094-1101
  64 threads,           after    304 M     735-736        959

ie the number of reads per second had plateaued at 13.3M due to the
cacheline contention on the folio reference count (and the cycles per
read had ballooned up to 20k).

And with that scalability limiter gone, it zooms up to 304M.

And again - that's not a real number in the sense that no real load
will ever see it, but the takeaway is that it basically scales
perfectly with no writes to shared cachelines. And honestly, the
patches don't look bad, and it's possible that other cases might find
use for this RCU-delayed page freeing.

Comments?

                Linus

Re: Improve page refcount scalability

[Matthew Wilcox]

On Tue, Oct 13, 2026 at 11:43:14AM -0700, Linus Torvalds wrote:
> The fairly short series of ten patches basically enhances folio lookup
> by noticing when folios are hot, and setting a "delay relasing this
> folio until after a RCU period", and then using that RCU delay it does
> not just the folio lookup itself under RCU (which we've done forever),
> but also the actual folio data accesses themselves under RCU. No
> refcount taking needed for the fastpath where we can just read the
> data from the folio entirely in an RCU-protected region when the user
> copy doesn't cause page faults.
> 
> Unlike my original approach - because it now actually delays the
> freeing of the underlying folio data - there's no need for any double
> buffering, and a lot of the original limitations of my original patch
> go away too. That improves performance quite a lot. And by "quite a
> lot"  I mean "I see a 23x performance increase on my Ryzen
> threadripper". Admittedly that is a very artificial case of reading
> the same small part of a page over and over in a threaded loop on all
> CPUs, so any real life impact will be much much less.
> 
> Anyway, I wasn't planning on posting the individual patches, I think
> we all use git by now and probably find it easier than fetching
> patches from the list. So it's all at
> 
>     git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux rcu-free-v3
> 
> and the commit messages are a bit more verbose than I would have done
> it myself without tool use, but maybe it's better that way.

Obviously the numbers are compelling.  And thanks to Kiryl for figuring
out a workload that confirmed my fears, then remedying the problem.

I don't like the PG_rcu_free flag.  I hope we're guaranteed that folios
in the process of being freed by RCU will not be mapped into userspace.
That means we can use PGTY_rcu instead, where there is no shortage of
available types.

The workload that Kiryl identified (truncating/holepunching folios
from files) is not the normal way we free folios from the page cache,
so while we need to accommodate it, it's not what we shuld optimise for.
Most page cache folios get freed through reclaim.  I started sketching
out a new state (I was going to call them "ghost folios") when I realised
I was basically reinventing the inactive list.  So maybe we should make
folios on the inactive list immediately freeable as long as one RCU
generation has passed?

This may be where I run into trouble.  I don't know how RCU works
internally.  Is it possible to record some kind of generation number in
the folio such that we can see if it's only recently been added to the
inactive list?  Or do we need three lists --active, purgatory, inactive
where the folio is moved from active to purgatory as it ages, then
from purgatory to inactive after an RCU perid?

This will somewhat complicate filemap_get_entry().  It'll have to be
able to cope with finding folios in all three states, and moving
them from the inactive or purgatory lists back to the active list.
I think it's doable with the right cmpxchg() magic.

What do you think?

Re: Improve page refcount scalability

[Linus Torvalds]

On Tue, 13 Oct 2026 at 14:03, Matthew Wilcox <willy@infradead.org> wrote:
>
> I don't like the PG_rcu_free flag.  I hope we're guaranteed that folios
> in the process of being freed by RCU will not be mapped into userspace.
> That means we can use PGTY_rcu instead, where there is no shortage of
> available types.

I don't see how we could ever map a page into user space *while* it's
in the process of being free'd by RCU.

But no, I don't think we can use the PGTY_xyz flags, because while
they won't be mapped into user space during the actual RCU freeing
period, they *would* be marked for being free'd long before that.
Including, very much, while being mmap'ed into user space.

IOW, the trivial way to do it is to mmap a file, and then doin regular
enough regular 'read()' calls on the pages that they get marked as
hot. End result: folio marked as PG_rcu_free while still being mapped.
The RCU freeing itself obviously won't *happen* until after the page
has been unmapped, but the flag needs to be there before.

> The workload that Kiryl identified (truncating/holepunching folios
> from files) is not the normal way we free folios from the page cache
> so while we need to accommodate it, it's not what we shuld optimise for.

I wouldn't cal;l the current case optimized for that. It was the
simplest test-case for quickly removing a lot of pages from the page
cache, and as such it's the *worst* case in many ways.

> Most page cache folios get freed through reclaim.  I started sketching
> out a new state (I was going to call them "ghost folios") when I realised
> I was basically reinventing the inactive list.  So maybe we should make
> folios on the inactive list immediately freeable as long as one RCU
> generation has passed?

Honestly, I don't think it works. Or rather, you can *force* it to
work, but it won't ever be a good solution. For the simple reason that
you still need to handle this whole "immediate free" case by hole
punching etc.

Hole punching itself may not be the common case - but 'truncate()'
actually *will* be the common case in some situations. It's very easy
to trigger.

So you try to make the different cases (truncate vs reclaim) have
different rules, but I think you will only be making up special cases
and hurting yourself in the process.

End result: I'd put that solidly in the "Doctor, doctor, it hurts when
I punch myself in the face" problem space. The solution is "Don't do
that then", not to try to look for other solutions.

Also, honestly, by now it's been two and a half years of this.

So by now I'd also go: I have a patch series. It's clean. It's ten
patches, each of them cleanly separated, half of them *tiny*, and none
of them more than 150 lines of code. It works.

If you have alternative approaches, I think it's "Show me the
money^Hcode" time, not handwavy "maybe we could do xyz" time.

                Linus

Re: Improve page refcount scalability

[Kiryl Shutsemau]

On Tue, Oct 13, 2026 at 11:43:14AM -0700, Linus Torvalds wrote:
> Comments?

There's a race in filemap_read_rcu_batch(). If the folio is replaced
under us with a new one and then i_size is extended, we can read data
that never got written there:

  CPU0                              CPU1
  filemap_read()
    filemap_read_rcu()
      filemap_read_rcu_batch()
        folio = xas_load()  // A
        test_bit(PG_rcu_free, A)
        xas_reload() == A
                                    fadvise(FADV_DONTNEED)
                                    // A leaves the page cache, its
                                    // free is deferred: A stays valid
                                    write()
                                    // page cache re-populated
                                    // i_size extended
        isize = i_size_read()
        // i_size is extended
        copy_page_to_iter(A)
        // we read from old folio
        // data that user never wrote

We need to re-check that the folio is still in the page cache after we
get the new i_size. See the patch below.

My branch contains a few more changes that address downstream effects of
RCU-freeing the page cache. But we can come back to that later once we get
agreement on the core mechanism.

diff --git a/mm/filemap.c b/mm/filemap.c
index b85de94a1b6e..0d9fcc35dc64 100644
--- a/mm/filemap.c
+++ b/mm/filemap.c
@@ -2956,6 +2956,8 @@ static bool filemap_read_rcu_batch(struct kiocb *iocb, struct iov_iter *iter,
 		isize = i_size_read(inode);
 		if (pos >= isize)
 			break;
+		if (xas_reload(&xas) != folio)
+			break;
 		bytes = min_t(loff_t, isize - pos,
 			      min_t(size_t, iov_iter_count(iter),
 				    PAGE_SIZE - offset));
-- 
  Kiryl Shutsemau / Kirill A. Shutemov

Re: Improve page refcount scalability

[Paul E. McKenney]

On Tue, Oct 13, 2026 at 10:03:12PM +0100, Matthew Wilcox wrote:
> On Tue, Oct 13, 2026 at 11:43:14AM -0700, Linus Torvalds wrote:
> > The fairly short series of ten patches basically enhances folio lookup
> > by noticing when folios are hot, and setting a "delay relasing this
> > folio until after a RCU period", and then using that RCU delay it does
> > not just the folio lookup itself under RCU (which we've done forever),
> > but also the actual folio data accesses themselves under RCU. No
> > refcount taking needed for the fastpath where we can just read the
> > data from the folio entirely in an RCU-protected region when the user
> > copy doesn't cause page faults.
> > 
> > Unlike my original approach - because it now actually delays the
> > freeing of the underlying folio data - there's no need for any double
> > buffering, and a lot of the original limitations of my original patch
> > go away too. That improves performance quite a lot. And by "quite a
> > lot"  I mean "I see a 23x performance increase on my Ryzen
> > threadripper". Admittedly that is a very artificial case of reading
> > the same small part of a page over and over in a threaded loop on all
> > CPUs, so any real life impact will be much much less.
> > 
> > Anyway, I wasn't planning on posting the individual patches, I think
> > we all use git by now and probably find it easier than fetching
> > patches from the list. So it's all at
> > 
> >     git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux rcu-free-v3
> > 
> > and the commit messages are a bit more verbose than I would have done
> > it myself without tool use, but maybe it's better that way.
> 
> Obviously the numbers are compelling.  And thanks to Kiryl for figuring
> out a workload that confirmed my fears, then remedying the problem.
> 
> I don't like the PG_rcu_free flag.  I hope we're guaranteed that folios
> in the process of being freed by RCU will not be mapped into userspace.
> That means we can use PGTY_rcu instead, where there is no shortage of
> available types.
> 
> The workload that Kiryl identified (truncating/holepunching folios
> from files) is not the normal way we free folios from the page cache,
> so while we need to accommodate it, it's not what we shuld optimise for.
> Most page cache folios get freed through reclaim.  I started sketching
> out a new state (I was going to call them "ghost folios") when I realised
> I was basically reinventing the inactive list.  So maybe we should make
> folios on the inactive list immediately freeable as long as one RCU
> generation has passed?
> 
> This may be where I run into trouble.  I don't know how RCU works
> internally.  Is it possible to record some kind of generation number in
> the folio such that we can see if it's only recently been added to the
> inactive list?  Or do we need three lists --active, purgatory, inactive
> where the folio is moved from active to purgatory as it ages, then
> from purgatory to inactive after an RCU perid?

The polled grace-period APIs handle the last sentence for you, but you
knew that already.  I am not sure what "recently added" means in this
context, but I cannot resist suggesting using the jiffies counter.

But tell me more about the meaning of "recently added" and I will try
to be more helpful.  ;-)

							Thanx, Paul

> This will somewhat complicate filemap_get_entry().  It'll have to be
> able to cope with finding folios in all three states, and moving
> them from the inactive or purgatory lists back to the active list.
> I think it's doable with the right cmpxchg() magic.
> 
> What do you think?

Re: Improve page refcount scalability

[Linus Torvalds]

On Tue, 13 Oct 2026 at 15:43, Kiryl Shutsemau <kirill@shutemov.name> wrote:
>
> There's a race in filemap_read_rcu_batch(). If the folio is replaced
> under us with a new one and then i_size is extended, we can read data
> that never got written there:

Hmm. Wouldn't it be better to move the i_size check up, and then have
one single check for that

                if (xas_reload(&xas) != folio)
                        break;

case?

Otherwise we end up with two of them, one inside that "folio != prev" case...

IOW, I wonder if we could just make the whole

                        if (!test_bit(PG_rcu_free, &folio->flags.f))
                                break;
                        smp_rmb();
                        if (xas_reload(&xas) != folio)
                                break;

be unconditional, and move the i_size read up there too (I notice that
the comment about uptodate got in there too - PG_rcu_free indirectly
does imply that, so it's not *wrong*, but it's perhaps a bit
surprising).

That said:

> My branch contains a few more changes that address downstream effects of
> RCU-freeing the page cache. But we can come back to that later once we get
> agreement on the core mechanism.

Would you be willing to take this whole series over? I'm going to be
on vacation starting Friday *and* do the merge window next week, and
realistically won't have time for it. So...

                Linus

Re: Improve page refcount scalability

[Harry Yoo]

On Tue, Oct 13, 2026 at 11:43:14AM -0700, Linus Torvalds wrote:
> Comments?

There should be a limitation on how much memory can be submitted to RCU
by the page cache, otherwise will be a corner case where we have too
much unreclaimable memory waiting for the RCU subsystem to free them.

There's a similar feature, dirty throttling: when a process dirties
memory in page cache much faster than the storage can handle IO,
the kernel throttles it, otherwise it'll allow generating unbounded
amount of unreclaimable memory.

For the same reason, when a process tries to create RCU-delay-freed
pages much faster than RCU can free memory, the kernel needs to limit
that.

-- 
Cheers,e
Harry / Hyeonggon

Re: Improve page refcount scalability

[Linus Torvalds]

On Tue, 13 Oct 2026 at 17:10, Harry Yoo <harry@kernel.org> wrote:
>
> There should be a limitation on how much memory can be submitted to RCU
> by the page cache, otherwise will be a corner case where we have too
> much unreclaimable memory waiting for the RCU subsystem to free them.

There is. In fact, I tried it a few different ways... It's not quite
what you suggest, but try it out.

> For the same reason, when a process tries to create RCU-delay-freed
> pages much faster than RCU can free memory, the kernel needs to limit
> that.

That was actually my initial approach - making the places that set the
PG_rcu_delay bit throttle themselves when there were lots of such
pages in flight.

Except I worried about slowing down simple things like truncating big
files: particularly since I believe firmly that slowing that down is
the *wrong* thing to do from a memory pressure standpoint, because
you'd be slowing down the very thing that is actively freeing pages -
it's just that it then takes a (short) while for those pages to get to
a usable state.

So what I did instead was to actively then expedite RCU when there
were a lot of pages in flight.

That worked quite well, but it hit a problem when you had lots of CPUs
doing it: since an RCU quiet period needs to happen on *each* CPU, you
still end up wanting to at some point wait for things: you need to
synchronize not _one_ thread freeing things, but synchronize across
all the CPU's.

But again: the good part really is that this "lots and lots of pages
in flight" only happens when there are literally lots and lots of
pages that are very much about to be free'd. So you are in a fairly
good position when it happens, you just need to *understand* the
position you are in.

So the other part is to come at the issue from the other end: when you
have trouble allocating pages, but see that there are a ton of pages
in flight due to PG_rcu_free, you basically do the same thing from the
other end, and do synchronize_rcu_expedited().

And in testing, that worked really well under even insane loads where
concurrent hole punching or truncate calls would generate gigabytes of
PG_rcu_free activity each second.

Now, it turns out that Kiryl also suggested another approach to the
pages becoming PG_rcu_free in the first place: instead of being able
to easily trigger gigabytes of such free pages by just truncating a
large file - which marked all those pages PG_rcu_free whether that was
useful or not - the current version actually comes at *that* the other
way too: mark only pages that are hot and are *worth* the PG_rcu_free
treatment (because they get read over and over again) with PG_rcu_free
in the first place, so that you don't generate that flood of
RCU-free'd pages in the first place, and now the PG_rcu_free flood is
in a way throttled by the *use* of said pages.

So there's effectively three different things working together:

 - mark a folio PG_rcu_free only when it's hot and keeps being accessed

   (See commits dabc522cee5b: "mm: have folio_mark_accessed() return
whether the folio is hot" and  bb5908b733a1: "mm/filemap: set
PG_rcu_free on the folios that reads keep coming back to" for details
- both are pretty tiny and not really all that complicated)

 - proactively expedite RCU grace periods when there are a lot of of
pages pending

   (See commit 2ce8df4627fe: "mm: expedite the grace period when a CPU
has many deferred frees")

 - teach the page allocator about the outstanding pages ratehr that failing

   (See commit 82fc13ff3a1e mm: "count the pages waiting for a grace
period and free them before OOM")

and it really does seem to work quite smoothly. But it's not like I
can guarantee it's perfect.

All those commits are fairly trivial. Of the ones listed above, that
"count the pages waiting for a grace period and free them before OOM"
is the biggest by far, but it's not really particularly complex or
subtle.

The actual biggest patches in the series are:

 - 65401f4102b9 ("mm/filemap: read from the page cache without folio
references")

   This really is not very big or complicated, although it *does* have
subtle ordering issues with i_size as pointed out by Kiryl. It's
mostly really just a simplified version of my patch from two and a
half years ago. And it's simplified because now the RCU freeing makes
everything more natural.

 - df0b133495c9 ("mm: free PG_rcu_free folios only after an RCU grace period")

   This is the one that actually runs on the page freeing side and
adds a page that is marked PG_rcu_free to a list and sets up the whole
RCU machinery (ok, there are actually two per-cpu lists: the 'pending'
list and the 'batch' list, and a RCU grace period will free the
current batch and move the pending list to be the next batch - not
rocket science).

and even those "big" patches aren't really all that big. Both are less
than 200 lines of code, and while I wouldn't call either of them
_trivial_, both are very much doing one clear thing and there's
nothing really *strange* about them either.

So it's ten patches, none of them particularly hairy. I really would
encourage you to just take a look and take it for a try.

Now, I *will* say that obviously RCU-freeing pages is an absolutely
conceptually *enormous* change to the VM. But it's in no way at all a
new thing: we've been doing the same thing for kmalloc() for decades.
With the right pattern, you can cause a lot of RCU-delayed kfree
loads. And we've had to tweak things for that.

So do I believe that branch is perfect on the first try? Hell no. I'm
sure there are things to improve there. But there are few syustem
calls more important than 'read()', so I think it's worth doing. We've
been doing the RCU-freeing for almost every other path, and I will
argue very forcefully that the path handling RCU code is an order of
magnitude more complicated than this particular code is.

              Linus

Re: Improve page refcount scalability

[Kiryl Shutsemau]

On Tue, Oct 13, 2026 at 10:03:12PM +0100, Matthew Wilcox wrote:
> On Tue, Oct 13, 2026 at 11:43:14AM -0700, Linus Torvalds wrote:
> > The fairly short series of ten patches basically enhances folio lookup
> > by noticing when folios are hot, and setting a "delay relasing this
> > folio until after a RCU period", and then using that RCU delay it does
> > not just the folio lookup itself under RCU (which we've done forever),
> > but also the actual folio data accesses themselves under RCU. No
> > refcount taking needed for the fastpath where we can just read the
> > data from the folio entirely in an RCU-protected region when the user
> > copy doesn't cause page faults.
> > 
> > Unlike my original approach - because it now actually delays the
> > freeing of the underlying folio data - there's no need for any double
> > buffering, and a lot of the original limitations of my original patch
> > go away too. That improves performance quite a lot. And by "quite a
> > lot"  I mean "I see a 23x performance increase on my Ryzen
> > threadripper". Admittedly that is a very artificial case of reading
> > the same small part of a page over and over in a threaded loop on all
> > CPUs, so any real life impact will be much much less.
> > 
> > Anyway, I wasn't planning on posting the individual patches, I think
> > we all use git by now and probably find it easier than fetching
> > patches from the list. So it's all at
> > 
> >     git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux rcu-free-v3
> > 
> > and the commit messages are a bit more verbose than I would have done
> > it myself without tool use, but maybe it's better that way.
> 
> Obviously the numbers are compelling.  And thanks to Kiryl for figuring
> out a workload that confirmed my fears, then remedying the problem.
> 
> I don't like the PG_rcu_free flag.

I wonder if you would be okay with a 64-bit-only flag? Those are less
scarce. That's the direction I was looking at:

https://git.kernel.org/pub/scm/linux/kernel/git/kas/linux.git/commit/?id=59b23fdc4bb0d008e2a30d24156177eeede8a577

I don't think we care about the scalability of 32-bit machines here.

-- 
  Kiryl Shutsemau / Kirill A. Shutemov

Re: Improve page refcount scalability

[David Hildenbrand (Arm)]

On 10/14/26 12:36, Kiryl Shutsemau wrote:
> On Tue, Oct 13, 2026 at 10:03:12PM +0100, Matthew Wilcox wrote:
>> On Tue, Oct 13, 2026 at 11:43:14AM -0700, Linus Torvalds wrote:
>>> The fairly short series of ten patches basically enhances folio lookup
>>> by noticing when folios are hot, and setting a "delay relasing this
>>> folio until after a RCU period", and then using that RCU delay it does
>>> not just the folio lookup itself under RCU (which we've done forever),
>>> but also the actual folio data accesses themselves under RCU. No
>>> refcount taking needed for the fastpath where we can just read the
>>> data from the folio entirely in an RCU-protected region when the user
>>> copy doesn't cause page faults.
>>>
>>> Unlike my original approach - because it now actually delays the
>>> freeing of the underlying folio data - there's no need for any double
>>> buffering, and a lot of the original limitations of my original patch
>>> go away too. That improves performance quite a lot. And by "quite a
>>> lot"  I mean "I see a 23x performance increase on my Ryzen
>>> threadripper". Admittedly that is a very artificial case of reading
>>> the same small part of a page over and over in a threaded loop on all
>>> CPUs, so any real life impact will be much much less.
>>>
>>> Anyway, I wasn't planning on posting the individual patches, I think
>>> we all use git by now and probably find it easier than fetching
>>> patches from the list. So it's all at
>>>
>>>     git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux rcu-free-v3
>>>
>>> and the commit messages are a bit more verbose than I would have done
>>> it myself without tool use, but maybe it's better that way.
>>
>> Obviously the numbers are compelling.  And thanks to Kiryl for figuring
>> out a workload that confirmed my fears, then remedying the problem.
>>
>> I don't like the PG_rcu_free flag.
> 
> I wonder if you would be okay with a 64-bit-only flag? Those are less
> scarce. That's the direction I was looking at:
> 
> https://git.kernel.org/pub/scm/linux/kernel/git/kas/linux.git/commit/?id=59b23fdc4bb0d008e2a30d24156177eeede8a577
> 
> I don't think we care about the scalability of 32-bit machines here.

I think limiting this functionality to 64bit makes a lot of sense.

-- 
Cheers,

David

Re: Improve page refcount scalability

[Kiryl Shutsemau]

On Tue, Oct 13, 2026 at 04:39:24PM -0700, Linus Torvalds wrote:
> On Tue, 13 Oct 2026 at 15:43, Kiryl Shutsemau <kirill@shutemov.name> wrote:
> >
> > There's a race in filemap_read_rcu_batch(). If the folio is replaced
> > under us with a new one and then i_size is extended, we can read data
> > that never got written there:
> 
> Hmm. Wouldn't it be better to move the i_size check up, and then have
> one single check for that
> 
>                 if (xas_reload(&xas) != folio)
>                         break;
>
> case?

I was not sure it is possible. i_size has to be read after we know the
folio is uptodate, and the uptodate bit only means something after the
reload.

But it is fine: PG_rcu_free is only set after the folio was seen
uptodate, so reading i_size after the flag test and smp_rmb() gives the
same ordering. And i_size_read() is an acquire, so it is ordered before
the reload.

The uptodate check in filemap_read_rcu_can_copy() has to stay, though.
The flag only says the folio was uptodate when it was marked. Some
filesystems clear PG_uptodate on a folio that stays in the page cache,
and that can happen after the folio got the flag.

See the patch below.

> Otherwise we end up with two of them, one inside that "folio != prev" case...
> 
> IOW, I wonder if we could just make the whole
> 
>                         if (!test_bit(PG_rcu_free, &folio->flags.f))
>                                 break;
>                         smp_rmb();
>                         if (xas_reload(&xas) != folio)
>                                 break;
> 
> be unconditional, and move the i_size read up there too (I notice that
> the comment about uptodate got in there too - PG_rcu_free indirectly
> does imply that, so it's not *wrong*, but it's perhaps a bit
> surprising).
> 
> That said:
> 
> > My branch contains a few more changes that address downstream effects of
> > RCU-freeing the page cache. But we can come back to that later once we get
> > agreement on the core mechanism.
> 
> Would you be willing to take this whole series over?

Yes, can do.

diff --git a/mm/filemap.c b/mm/filemap.c
index b85de94a1b6e..1836da9eee5e 100644
--- a/mm/filemap.c
+++ b/mm/filemap.c
@@ -2879,9 +2879,9 @@ static bool filemap_read_rcu_can_copy(struct folio *folio, pgoff_t index,
  * Copy page by page, each page looked up in i_pages.  Without a reference
  * a split can change the folio's size under us, so only the page at the
  * index xas_load() returned the folio for is known to be part of it.
- * The folio's flags are checked once per folio: prev is the folio of the
- * previous page.  *last_index is the page the previous copy ended in, and
- * is set to each page copied, as the slow path sets last_pos.
+ * filemap_read_rcu_can_copy() is called once per folio: prev is the folio
+ * of the previous page.  *last_index is the page the previous copy ended
+ * in, and is set to each page copied, as the slow path sets last_pos.
  *
  * Copy at most FOLIO_BATCH_SIZE pages under one rcu_read_lock().  Return
  * true if that many were copied and the read is not done.  prev is not
@@ -2928,34 +2928,25 @@ static bool filemap_read_rcu_batch(struct kiocb *iocb, struct iov_iter *iter,
 		if (xas_retry(&xas, folio) || !folio || xa_is_value(folio))
 			break;
 
+		/* Not folio_test_rcu_free(): it may not be a folio any more */
+		if (!test_bit(PG_rcu_free, &folio->flags.f))
+			break;
+		/* Read i_size and slot after the flag: it implies uptodate */
+		smp_rmb();
+		isize = i_size_read(inode);
+		if (pos >= isize)
+			break;
+		/* A write fills the slot before it extends i_size */
+		if (xas_reload(&xas) != folio)
+			break;
+		/* Read the rest of the folio only once the slot holds it */
+		smp_rmb();
 		if (folio != prev) {
-			/*
-			 * The memory may be anything by now, a tail page or
-			 * no folio at all, so test the bit and not the folio
-			 * flag.  The loads must happen in this order: the
-			 * flag, then the slot, then everything else.
-			 */
-			if (!test_bit(PG_rcu_free, &folio->flags.f))
-				break;
-			/*
-			 * The flag before the slot: a flag set on a later
-			 * folio in this memory was set after this folio left
-			 * the slot (see above)
-			 */
-			smp_rmb();
-			if (xas_reload(&xas) != folio)
-				break;
-			/* The slot before anything else of the folio */
-			smp_rmb();
 			if (!filemap_read_rcu_can_copy(folio, index, *last_index))
 				break;
 			prev = folio;
 		}
 
-		/* i_size must be checked after we know the folio is uptodate */
-		isize = i_size_read(inode);
-		if (pos >= isize)
-			break;
 		bytes = min_t(loff_t, isize - pos,
 			      min_t(size_t, iov_iter_count(iter),
 				    PAGE_SIZE - offset));
-- 
  Kiryl Shutsemau / Kirill A. Shutemov

Re: Improve page refcount scalability

[Pedro Falcato]

On Tue, Oct 13, 2026 at 06:01:15PM -0700, Linus Torvalds wrote:
> On Tue, 13 Oct 2026 at 17:10, Harry Yoo <harry@kernel.org> wrote:
> >
> > There should be a limitation on how much memory can be submitted to RCU
> > by the page cache, otherwise will be a corner case where we have too
> > much unreclaimable memory waiting for the RCU subsystem to free them.
> 
> There is. In fact, I tried it a few different ways... It's not quite
> what you suggest, but try it out.
> 
> > For the same reason, when a process tries to create RCU-delay-freed
> > pages much faster than RCU can free memory, the kernel needs to limit
> > that.
> 
> That was actually my initial approach - making the places that set the
> PG_rcu_delay bit throttle themselves when there were lots of such
> pages in flight.
> 
> Except I worried about slowing down simple things like truncating big
> files: particularly since I believe firmly that slowing that down is
> the *wrong* thing to do from a memory pressure standpoint, because
> you'd be slowing down the very thing that is actively freeing pages -
> it's just that it then takes a (short) while for those pages to get to
> a usable state.
> 
> So what I did instead was to actively then expedite RCU when there
> were a lot of pages in flight.

Right, this makes sense.

> 
> That worked quite well, but it hit a problem when you had lots of CPUs
> doing it: since an RCU quiet period needs to happen on *each* CPU, you
> still end up wanting to at some point wait for things: you need to
> synchronize not _one_ thread freeing things, but synchronize across
> all the CPU's.
> 
> But again: the good part really is that this "lots and lots of pages
> in flight" only happens when there are literally lots and lots of
> pages that are very much about to be free'd. So you are in a fairly
> good position when it happens, you just need to *understand* the
> position you are in.
> 
> So the other part is to come at the issue from the other end: when you
> have trouble allocating pages, but see that there are a ton of pages
> in flight due to PG_rcu_free, you basically do the same thing from the
> other end, and do synchronize_rcu_expedited().
> 
> And in testing, that worked really well under even insane loads where
> concurrent hole punching or truncate calls would generate gigabytes of
> PG_rcu_free activity each second.
> 
> Now, it turns out that Kiryl also suggested another approach to the
> pages becoming PG_rcu_free in the first place: instead of being able
> to easily trigger gigabytes of such free pages by just truncating a
> large file - which marked all those pages PG_rcu_free whether that was
> useful or not - the current version actually comes at *that* the other
> way too: mark only pages that are hot and are *worth* the PG_rcu_free
> treatment (because they get read over and over again) with PG_rcu_free
> in the first place, so that you don't generate that flood of
> RCU-free'd pages in the first place, and now the PG_rcu_free flood is
> in a way throttled by the *use* of said pages.
> 
> So there's effectively three different things working together:
> 
>  - mark a folio PG_rcu_free only when it's hot and keeps being accessed
> 
>    (See commits dabc522cee5b: "mm: have folio_mark_accessed() return
> whether the folio is hot" and  bb5908b733a1: "mm/filemap: set
> PG_rcu_free on the folios that reads keep coming back to" for details
> - both are pretty tiny and not really all that complicated)

I'm not sure this is a good criteria for checking whether a folio should
be RCU-ified. Namely:
- you don't need that many accesses to get an active, referenced folio. LRU
batching makes it a little harder, but not by much.
- rewinding along a folio doesn't tell you much other than the reader
went back, for some reason. It's also natively racy so funny things like
multiple pread()'s to the same fd can easily confuse this mechanism into
marking this as hot.

How many pages (as a %) are you seeing as marked PG_rcu_free across a boot/
realistic workload?

> 
>  - proactively expedite RCU grace periods when there are a lot of of
> pages pending
> 
>    (See commit 2ce8df4627fe: "mm: expedite the grace period when a CPU
> has many deferred frees")
> 
>  - teach the page allocator about the outstanding pages ratehr that failing
> 
>    (See commit 82fc13ff3a1e mm: "count the pages waiting for a grace
> period and free them before OOM")
> 
> and it really does seem to work quite smoothly. But it's not like I
> can guarantee it's perfect.
> 
> All those commits are fairly trivial. Of the ones listed above, that
> "count the pages waiting for a grace period and free them before OOM"
> is the biggest by far, but it's not really particularly complex or
> subtle.
> 
> The actual biggest patches in the series are:
> 
>  - 65401f4102b9 ("mm/filemap: read from the page cache without folio
> references")
> 
>    This really is not very big or complicated, although it *does* have
> subtle ordering issues with i_size as pointed out by Kiryl. It's
> mostly really just a simplified version of my patch from two and a
> half years ago. And it's simplified because now the RCU freeing makes
> everything more natural.
> 
>  - df0b133495c9 ("mm: free PG_rcu_free folios only after an RCU grace period")
> 
>    This is the one that actually runs on the page freeing side and
> adds a page that is marked PG_rcu_free to a list and sets up the whole
> RCU machinery (ok, there are actually two per-cpu lists: the 'pending'
> list and the 'batch' list, and a RCU grace period will free the
> current batch and move the pending list to be the next batch - not
> rocket science).
> 
> and even those "big" patches aren't really all that big. Both are less
> than 200 lines of code, and while I wouldn't call either of them
> _trivial_, both are very much doing one clear thing and there's
> nothing really *strange* about them either.
> 
> So it's ten patches, none of them particularly hairy. I really would
> encourage you to just take a look and take it for a try.
> 
> Now, I *will* say that obviously RCU-freeing pages is an absolutely
> conceptually *enormous* change to the VM. But it's in no way at all a
> new thing: we've been doing the same thing for kmalloc() for decades.

Well, yes, but generally you have a lot more memory pinned as page cache
than kmalloc. In my current laptop (fairly standard desktop system) that
ratio is about 20x (more page cache than slab).

In any case, I was fairly interested in the problem back then, but I don't
think a solution that may fundamentally alter the VM system is worth it. At
least until we can find an actual real-world use case where performance
substancially changes (i.e not a microbenchmark). Until then, these patches
look to be more trouble than they're worth.

Regarding the RCU'ing, I'm wondering if we could find a similar system where
some type of RCU stability exists, but without delaying the actual frees to
RCU. Like TYPESAFE_BY_RCU, but for folios (where they keep their order and
some other interesting properties like direct map status, (un)acceptability)
such that the old solution would be more palatable. In any case, talk is cheap,
and I definitely have no code :)

(or, and this is another random idea that is currently not feasible due to
$raisins, but the folio refcount could be sharded per-cpu or so; this should
avoid most contention at the expense of the (relatively) rare truncate/reclaim
path)

-- 
Pedro

Re: Improve page refcount scalability

[Matthew Wilcox]

On Wed, Oct 14, 2026 at 12:53:11PM +0100, Kiryl Shutsemau wrote:
> The uptodate check in filemap_read_rcu_can_copy() has to stay, though.
> The flag only says the folio was uptodate when it was marked. Some
> filesystems clear PG_uptodate on a folio that stays in the page cache,
> and that can happen after the folio got the flag.

Those filesystems are buggy.  We have assertions that the uptodate flag
is set on folios mapped into userspace, and filesystems going arund
clearing the uptodate bit will cause problems.  It's something I'm
working on removing, but since it's all connected to writeback errors,
it takes a lot of arguing with each filesystem maintainer.