Re: [RFC PATCH 0/2] Defer evicting inodes to a workqueue

[Dave Chinner <david@fromorbit.com>]

[sorry I'm late on this, been working on other things]

On Wed, Sep 24, 2025 at 10:09:55AM +0100, Matthew Wilcox (Oracle) wrote:
> Evicting an inode is a complex process which may require allocating
> memory, running a transaction, etc, etc.  Doing it as part of reclaim
> is a bad idea and leads to hard-to-reproduce bug reports.  This pair of
> patches defers it to a workqueue if we're in reclaim.
> 
> Bugs:
> https://lore.kernel.org/all/CALm_T+3j+dyK02UgPiv9z0f1oj-HM63oxhsB0JF9gVAjeVfm1Q@mail.gmail.com/
> https://lore.kernel.org/all/CALm_T+2cEDUJvjh6Lv+6Mg9QJxGBVAHu-CY+okQgh-emWa7-1A@mail.gmail.com/
> https://lore.kernel.org/all/20250326105914.3803197-1-matt@readmodwrite.com/

So eviction running filesystem shrinkers that then do blocking IO,
folio operations and/or memory allocation and triggering warnings?

Seems like a problem GFP_NOFS was invented to solve, yes?

> I don't know if this is a good idea, to be honest.  We're kind of lying
> to reclaim by pretending that we've freed N inodes when actually we've
> just queued them for eviction.  On the other hand, XFS has been doing
> it for years, so perhaps it's not important.

tl;dr: we aren't lying, but because most people don't understand how
shrinker-based reclaim work is accounted for it can appear that way.

Long story:

memory relcaim (vmscan.c) does not track reclaim progress via the
number of objects the shrinkers report as freed. It tracks reclaim
progress via the amount of memory freed by the shrinker task. This
is what mm_account_reclaimed_pages() does, and it is implicit in
freeing operations as it is called from the slab code when it frees
the backing memory for a slab cache.

For other shrinkers that manually free memory, they have direct
calls to this function to record memory freeing. e.g. in the XFS
buffer cache when the shrinker disposes of a buffer and calls
xfs_buf_free().

This is a completely separate mechanism to the shrinker work
equalisation mechanism that uses object counts to apply the same
pressure to all shrinkable caches in the system. The shrinkers need
to scan a specific number of objects, and return a count based on
the progress they made. This returned count is ignored by the memory
reclaim code, it is only used as measure of progress being made
in certain siutations (e.g. drop_caches) and so is meaningless in
the course of this discussion.

IOWs, all the shrinker has to do is scan that number of objects for
freeing. It does not have to free them, nor does the returned
"objects freed" imply that any memory was actually freed.

Indeed, the VFS does lazy LRU removal, so it's common for the inode
cache shrinker to remove referenced objects from the LRU and not
free them. In that case, the reclaimable object count goes down, but
the freed count does not go up. The cache got "smaller", but we
didn't free anything. This is perfectly fine.

Indeed, in the case of the superblock shrinker, the "freed" count
tracks how many objects had their VFS lifecycle terminated (e.g.
indoes that were evicted), but this does not imply that the objects
were actually freed. They are simply no longer tracked as freeable
cached objects by the VFS caches.

Why is this distinction important? Remember what I said above about
mm_account_reclaimed_pages() accounting actually freed memory? Now
consider that dentries and inodes are RCU freed.

What does that imply about the superblock shrinker actually freeing
memory?

What this means is that the superblock shrinker -never- frees inodes
or dentry objects directly. The memory being freed is -never-
accounted to the shrinker, because the shrinker task has to schedule
before an RCU grace period can expire and run the RCU callbacks to
free the objects.

IOWs, shrinkers are intended to terminate the life cycle of cached
objects, but the life cycle of the allocated object can (and does)
extend beyond the shrinker that terminates the life cycle.

In the case of XFS, the VFS inode is embedded in the XFS inode, and
the VFS lifecycle is a subset of the XFS inode lifecycle. When
the VFS inode is evicted, it's lifecycle is terminated, but it still
a valid, tracked object at the XFS level. Indeed, XFs puts clean
inodes straight back on the internal reclaim list and accounts for
it as reclaimable. So the number of objects the superblock shrinker
is tracking does not actually go down - it is now tracked via the
sb->s_op->nr_cached_objects(sb, sc) callouts in the superblock
shrinker as a reclaimable object.

If the inode needs eviction work, (i.e. inodegc) it doesn't
immediately get re-accounted as a reclaimable object. We do the
work, then once the inode is in a reclaimable state, we put it on
the reclaim queue and account for it at that point.

Put simply: XFS is not "lying" about anything to the
mm/reclaim/shrinker subystems. We are simply taking advantage of the
fact that the superblock shrinker always frees the VFS inodes
asynchronously via RCU. Hence there is always a disconnect between
between VFS object life cycle termination and the object being
freed. XFS inserts it's own "disconnection processing" into that
gap, and actually lets the superblock shrinker track those objects.

> I think the real solution here is to convert the Linux VFS to use the
> same inode lifecycle as IRIX,

This has nothing to do with Irix. I introduced background inode
freeing to XFS in 2011 to avoid the problems of having to wait for
the inode to be fully clean in evict() context. This introduced some
other issues that I solved in ~2020 with non-blocking inode reclaim
processing. And in 2021, we finally moved all the transactional
modifications out of the evict() path with the background inodegc
infrastructure I wrote.

All of this is possible because shrinkers are simply a mechanism for
terminating the life cycle of a given object and there is absolutely
no requirement for life cycle termination to free any memory.
Indeed, they may release references to other resources that pin
memory (e.g. dentries pin inodes) and so memory being freed might
actually be several steps removed...

> but I don't fully understand the downsides
> of that approach.  One major pro of course is that XFS wouldn't have to
> work around the Linux VFS any more.

Not true - this stuff is way, way more complex than you imply.

On of the key factors in Josef's active/passive inode reference
infrastructure is that it could allow us to bring the "unreferenced"
part of the XFS inode lifecycle (i.e. everything that happens after
eviction) up into the VFS via passive VFS inode object references.

And at that point, could bring VFS active reference eviction
processing up into the VFS as an async deferal mechanism. But it
would also require us to bring the XFS inode cache VFS inode
recycling code as we could now get cache hits on a passively
referenced VFS inode which then needs the active VFS inodes state
reinstantiated.

There's other interactions as we'd need to lift as well, because
experience has proven that pushing eviction off to a workqueue will
create performance regressions.  e.g. Try pushing hundreds of
thousands of inodes a second across dozens of CPUs through a single
workqueue (e.g. highly concurrent find across dozens of directories
under memory pressure).

Then we have to throttle the number of inodes you defer for
eviction. Once you have many thousands deferred, it can take a long
time for the workqueue to catch up, especially if each inode has to
free many, many extents. Unbounded queues are bad, especially when
each unit of queued work can largely be unbounded, too. (e.g.
eviction processing after an unlink of a file with tens of millions
of extents).

IOWs, there's a reason the XFS inodegc queuing is complex - it uses
bound depth per-cpu lockless lists and workqueues for the eviction
processing deferral queue. Avoiding performance impacts due to adding
two context switches to every inode eviction is not simple.

Async inode eviction also has interactions with freeze - what
happens when you have dirty inodes or inodes that require dirtying
to evict queued for eviction and then the fs is frozen? We can't
process those evictions whilst the fs is frozen - we can't modify
anything. Hence there is the possibility that every eviction worker
task can get stuck waiting for thaw, and now inode cache reclaim is
effectively stuck until a thaw occurs....

There is also user visible interactions with space usage (e.g. df vs
rm). Half of the unlink work is done on eviction (i.e. when the last
ref is dropped) and so space isn't freed until eviction processing
is complete. Hence if you run a large rm -rf, it queues all the
inode and extent freeing work to the background queues. From the
user perspective, when the rm -rf completes the directory structure
is gone, but the space hasn't been completely freed yet. It may take
minutes for the space to be freed.

So, yeah, async inode eviction is anything but simple, and from
previous attempts to lift this stuff out of XFS to the VFS, I came
to the conclusion the only way to do it sanely was to convert the
VFS inodes to use active/passive reference counting so that the VFS
could guarantee that the inode was truly unreferenced by the VFS
before cache eviction occurs....

Cheers,

Dave.

-- 
Dave Chinner
david@fromorbit.com