Re: Improve page refcount scalability

[Matthew Wilcox <willy@infradead.org>]

On Tue, Oct 13, 2026 at 11:43:14AM -0700, Linus Torvalds wrote:
> The fairly short series of ten patches basically enhances folio lookup
> by noticing when folios are hot, and setting a "delay relasing this
> folio until after a RCU period", and then using that RCU delay it does
> not just the folio lookup itself under RCU (which we've done forever),
> but also the actual folio data accesses themselves under RCU. No
> refcount taking needed for the fastpath where we can just read the
> data from the folio entirely in an RCU-protected region when the user
> copy doesn't cause page faults.
> 
> Unlike my original approach - because it now actually delays the
> freeing of the underlying folio data - there's no need for any double
> buffering, and a lot of the original limitations of my original patch
> go away too. That improves performance quite a lot. And by "quite a
> lot"  I mean "I see a 23x performance increase on my Ryzen
> threadripper". Admittedly that is a very artificial case of reading
> the same small part of a page over and over in a threaded loop on all
> CPUs, so any real life impact will be much much less.
> 
> Anyway, I wasn't planning on posting the individual patches, I think
> we all use git by now and probably find it easier than fetching
> patches from the list. So it's all at
> 
>     git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux rcu-free-v3
> 
> and the commit messages are a bit more verbose than I would have done
> it myself without tool use, but maybe it's better that way.

Obviously the numbers are compelling.  And thanks to Kiryl for figuring
out a workload that confirmed my fears, then remedying the problem.

I don't like the PG_rcu_free flag.  I hope we're guaranteed that folios
in the process of being freed by RCU will not be mapped into userspace.
That means we can use PGTY_rcu instead, where there is no shortage of
available types.

The workload that Kiryl identified (truncating/holepunching folios
from files) is not the normal way we free folios from the page cache,
so while we need to accommodate it, it's not what we shuld optimise for.
Most page cache folios get freed through reclaim.  I started sketching
out a new state (I was going to call them "ghost folios") when I realised
I was basically reinventing the inactive list.  So maybe we should make
folios on the inactive list immediately freeable as long as one RCU
generation has passed?

This may be where I run into trouble.  I don't know how RCU works
internally.  Is it possible to record some kind of generation number in
the folio such that we can see if it's only recently been added to the
inactive list?  Or do we need three lists --active, purgatory, inactive
where the folio is moved from active to purgatory as it ages, then
from purgatory to inactive after an RCU perid?

This will somewhat complicate filemap_get_entry().  It'll have to be
able to cope with finding folios in all three states, and moving
them from the inactive or purgatory lists back to the active list.
I think it's doable with the right cmpxchg() magic.

What do you think?