Re: [PATCH v6 1/2] mm: Introduce opportunistic_compaction concept to vmscan and shrinkers

[Matthew Brost <matthew.brost@intel.com>]

On Tue, Jun 23, 2026 at 03:01:26PM -0700, Matthew Brost wrote:
> On Tue, Jun 23, 2026 at 03:32:58PM +1000, Dave Chinner wrote:
> > On Mon, Jun 22, 2026 at 05:09:33PM -0700, Matthew Brost wrote:
> > > On Tue, Jun 23, 2026 at 09:10:43AM +1000, Dave Chinner wrote:
> > > > On Tue, Jun 16, 2026 at 08:22:17PM -0700, Matthew Brost wrote:
> > > > > High-order allocations using __GFP_NORETRY or __GFP_RETRY_MAYFAIL
> > > > > are often opportunistic attempts to satisfy fragmentation-sensitive
> > > > > allocations rather than indications of severe memory pressure. In these
> > > > > cases, reclaim may invoke shrinkers that aggressively destroy working
> > > > > sets even though reclaim is unlikely to materially improve the
> > > > > allocation outcome.
> > > > > 
> > > > > Some shrinkers manage expensive backing or migration operations where
> > > > > reclaim can result in substantial working set disruption despite the
> > > > > system having sufficient free memory overall. This is particularly
> > > > > visible in fragmentation-heavy workloads where reclaim repeatedly tears
> > > > > down active state while kswapd attempts to satisfy higher-order
> > > > > allocations.
> > > > > 
> > > > > Introduce an opportunistic_compaction hint in shrink_control that allows
> > > > > kswapd to communicate when reclaim originates from a high-order
> > > > > allocation context that may be fragmentation driven rather than true
> > > > > memory pressure. Shrinkers may use this hint to avoid destructive
> > > > > working set reclaim while still participating normally during order-0
> > > > > or stronger reclaim conditions.
> > > 
> > > Thanks for the input - this is a tough problem.
> > 
> > Yes, that it is.
> > 
> 
> This is part of what makes this fun.
> 
> There’s a lot to go through here—I’m going to start actively working on
> a PoC, hopefully today. I’ll likely have more feedback as this gets
> coded, but let’s continue to stay aligned.
> 
> > > > To be honest, this seems like another "push a hint through to the XE
> > > > shrinker" mechanism under a different name. You seem so focused on
> > > > fixing the XE reproducer that the -systemic problem- that -any-
> > > > high-order folio demand causes is not being acknowleged.
> > > > 
> > > 
> > > I'm not exactly sure I agree here. Communicating via __GFP_NORETRY or
> > > __GFP_RETRY_MAYFAIL with a higher order implies that the caller can
> > > handle higher-order allocation failures, so the shrinker shouldn’t try
> > > too hard to obtain a large page (e.g., evict a working set). I agree
> > > that Xe is currently the only shrinker making use of this, but other
> > > shrinkers could also hook into it. This information simply isn’t
> > > available today.
> > 
> > Right, but "we are doing compaction" isn't information that tells
> > the subsystem shrinker what it needs to do. "memory compaction is
> > occurring" isn't a well defined action like "count reclaimable
> > objects" or "scan N objects and reclaim as many as you can without
> > blocking".
> > 
> > Directed high order object reclaim should be much efficient that
> > trying to use general memory pressure to age out enough objects to
> > reform contiguous pages. We need to help memory compaction, and we
> > can't really do that by layering heuristics over reclaim algorithms
> > designed to maintain working sets efficiently.
> > 
> > > > e.g. we use high-order folios extensively in the page cache these
> > > > days, and there are -many- cases where memory compaction driven by
> > > > high-order demand cause significant performance regressions for page
> > > > cache performance. To date, every single person who has wanted to
> > > > fix the problem they are seeing has effectively attempted to -turn
> > > > off compaction- via GFP flags.
> > > 
> > > So does that mean they clear __GFP_RECLAIM?
> > 
> > Usually __GFP_DIRECT_RECLAIM, as it's the overhead of direct
> > compaction that causes the performance problems.
> > 
> > > That isn't really what in DRM or Xe. In former case we have pools of
> > > lower order pages in TTM not in use that can be shrunk, potentially
> > > freeing multiple lower orders pages so a higher order page formed, and
> > > the later possible BOs (sets of pages) in Xe marked as purgable (not is
> > > in working set) which can also be shrunk. Other DRM drivers have purging
> > > concepts too.
> > > 
> > > I’m not very familiar with what other shrinkers or subsystems want, but
> > > presumably other shrinkers have pools or caches that aren’t currently in
> > > use, where they can say, “OK, I’ll give these pages up for opportunistic
> > > compaction, but I won’t give up my working set.” Of course, as mentioned
> > > above, if someone else explicitly requests large pages by avoiding
> > > __GFP_NORETRY and __GFP_RETRY_MAYFAIL, the shrinker should then give up
> > > its working set.
> > 
> > Most caches are slab-based, so there can be 10s of objects with
> > different life cycles per page. There is no almost possiblity that
> > shrinker reclaim will free pages without substantial
> > amounts of the cache being reclaimed.
> 
> 
> Yes, I agree most shrinkers are slab-based, but there are at least three
> notable exceptions that I’m aware of—potentially more (let me know if
> there are others which are aware of):
> 
> - The TTM (the most common memory allocator for DRM drivers) pool
>   shrinker maintains pools of pages that are free (i.e., have no
>   driver-side references).
> - DRM driver-side shrinkers maintain both non-active pages (marked as
>   purged by user space) and active pages that are either idle or
>   short-term pinned via dma-fences.
> - The deferred_split_shrinker, which holds pages in larger-order folios
>   that have been partially unmapped.
> 
> All of these can free substantial amounts of memory—sometimes very
> quickly, and sometimes with a delay (e.g., when waiting on short-term
> pinned memory via dma-fences).
> 
> The deferred_split_shrinker in also seemingly has an issue where
> it can worsen fragmentation in certain cases. For example, if we need an
> order-9 folio and it shrinks a partially freed folio of the same or
> smaller order, this doesn’t help and may actively make fragmentation
> worse by shattering folios. While we’re here, this likely should be
> addressed—unless I’m missing something.
> 
> > 
> > > > I've even done that myself inside XFS to work around kvmalloc()
> > > > issues with a lack of GFP_NOFAIL support and doing costly high order
> > > > allocations that fail and trigger compaction before falling back to
> > > > vmalloc().  However, these issues have since been fixed in the
> > > > kvmalloc() code, such that it now does the right thing for most
> > > > calling contexts (i.e. tries high-order kmalloc() without triggering
> > > > compaction, then fall back to GFP_NOFAIL vmalloc()). This has made
> > > > kvmalloc() more performant and better behaved for -all users-, not
> > > > just XFS.
> > > > 
> > > > This is not sustainable - we need compaction to be robust and
> > > > performant in the face of high-order folio demands, regardless of
> > > > what subsystem is generating the demand.
> > > > 
> > > > So with that in mind, let me paraphrase the comment in the second
> > > > patch in the Xe shrinker implementation:
> > > > 
> > > > "Shrinker reclaim is based on implementation specific object sizes
> > > > so it is unlikely to ever acheive contiguous page reclaim in a
> > > > manner that will measurably improve compaction rates."
> > > > 
> > > 
> > > This might be slightly misworded—what I really mean is that I don’t want
> > > to give up my working set for higher-order allocations that are allowed
> > > to fail, but I do want to give up my cache.
> > 
> > Right, that's the core of the problem - compaction is the high-order
> > reclaim trigger, the existing shrinker infrastructure reclaim is for
> > the working-set maintenance reclaim algorithm the subsystem uses..
> > 
> 
> Ok. We also need to consider how direct reclaim and compaction interact.
> Shrinkers are currently invoked during direct reclaim, which I believe
> is correct—this helps with compaction and is desirable. More on this
> below.
> 
> > > > You also say:
> > > > 
> > > > > No functional changes are introduced for existing shrinkers.
> > > > 
> > > > Consider how many shrinkable caches the general statement above
> > > > applies to, and then think about the fundamental impedence mismatch
> > > > between the affected shrinkable caches and what this patch actually
> > > > fixes.
> > > > 
> > > 
> > > Yes, as mentioned above, I’m only addressing Xe here, and I agree that
> > > this is likely an issue. Do you know of other shrinkers that have pools
> > > or caches which can be shrunk under the conditions I’m introducing here,
> > > but also have a working set they would prefer not to give up?
> > 
> > The first that comes to mind is the xfs_buf cache. This cache holds
> > cached metadata buffers that have different sizes can each contain
> > up 64kB of contiguous pages. The allocation algorithm uses
> > optimisitic large folios allocation, but if that fails it falls back
> > to vmalloc. The working set is maintained by a prioritised
> > multi-scan LRU so that more frequently accessed metadata is held
> > tighter by the cache than less frequently accessed (e.g. btree roots
> > have higher retention priority than the lowest leaves).
> > 
> > It does not currently track buffer objects by size, by if there was
> > a benefit to doing so then it could be implemented. I'd much prefer
> > to have such tracking separate to the working set maintenance,
> > especially as they will likely need some kind of balancing to
> > prevent high-order buffers in the working set from being thrashed by
> > compaction demand....
> > 
> > I know there are other caches that have variable sized objects, but
> > I'd have to go look at the code to referesh my memory of which ones
> > they are...
> > 
> 
> I'll have to look more here to be able to respond intelligently.
> 
> > > If so, a
> > > link on elixir.bootlin.com would be helpful so I can take a look. I’ll
> > > also try to go through other shrinkers myself.
> > 
> > cscope is your friend.
> > 
> > fs/xfs/xfs_buf.c contains the XFS buffer cache and shrinker
> > infrastructure, but looking at the code without any understanding of
> > the filesystem structures or how it interacts with the other XFS
> > shrinkable caches probably isn't as useful as you might think it
> > will be.
> 
> Agree. Looked at some FS shrinkers and got lost quickly.
> 
> > 
> > > > For example, what happens to slab-based caches if the XE cache is being
> > > > excessively reclaimed under high-order page demand? e.g. the slab-based
> > > > cache may have tens of objects per page and holds a system-level
> > > > performance critical working set of objects. How do these caches
> > > > handle the excessive reclaim demand being generated by compaction
> > > > thrashing?
> > > > 
> > > > Yup, they don't.
> > > > 
> > > 
> > > Agree.
> > > 
> > > > In the case of filesystem caches, the "reclaim and repopulate"
> > > > pattern you describe causing the XE perf problems causes internal
> > > > slab cache fragmentation. Not only does this not improve compaction
> > > > rates, it also results in more memory fragmentation because slab
> > > > pages get pinned by a small number of long lived objects and they
> > > > won't get freed until the cache is largely emptied.  IOWs, things
> > > > get -even worse- from a memory fragmentation POV when compaction
> > > > thrashing causes the working set of a high-object-count-per-page
> > > > slab cache to thrash....
> > > > 
> > > 
> > > Got a link to the code which you are referring to?
> > 
> > Do a lore search for "dentry cache defragmentation". You should be
> > able to find discussions that go back to around 2006 about
> > discussions on identry cache fragmentation and approaches like
> > slab-page based object reclaim to support internal defragmentation.
> > 
> > The fact that we don't have slab cache defragmentation despite many
> > years of people wanting such functionality should tell you how
> > complex the problem is.... :/
> > 
> > > That seems like a problem similar to another issue in DRM/Xe. We found
> > > that the process of shrinking actually drove fragmentation by splitting
> > > folios down to order-0 and then backing pages up one at a time. I have a
> > > separate fix in flight for that.
> > 
> > Possibly, though the life cycle differences I'm talking about can be
> > a few milliseconds (temporary file) vs weeks (long running database
> > instance holding it's table files open the whole time it is
> > running).
> > 
> > > Could the filesystem detect these hints and avoid shrinking in a way
> > > that causes fragmentation?
> > 
> > Not really. The fragmentation problem is caused by physical object
> > placement in the slab pages at allocation time, not the act of
> > reclaiming the object.
> > 
> > i.e. we don't know what the expected cache life time of a dentry or
> > an inode will be when we allocate it, so it just gets allocated in
> > the next free slot in the current partial slab page. When you get a
> > mix of dentries that are pinned by open files in long running
> > applications and dentries for access-once files in the same page, we
> > end up with reclaim freeing all the object slots that contained
> > access-once files. However, the pages are still pinned by the
> > objects for the open files that are in active use.
> > 
> > IOWs, LRU-based reclaim can free >90% of the objects in a cache that
> > held millions of objects with mixed lifetimes and still not free any
> > memory at all. There's nothing reclaim can do about it because the
> > problem is created at allocation time when lifetime is a complete
> > unknown.
> > 
> > > Alternatively, could it perform shrinking in
> > > a way that doesn’t shatter folios, or detect long-lived objects so it
> > > understands that shrinking isn’t going to help reduce fragmentation?
> > 
> > Referenced filesystem objects are not on the LRUs, so the shrinkers
> > aren't even aware of such long lived objects. And, as per the
> > "dentry cache defrag" comment above, we can't ask the slab to
> > reclaim or move objects because we don't track the owners of
> > external references to the objects themselves.
> > 
> > > 
> > > > This isn't isolated to individual subsystem thrashing.  If we run a
> > > > file-based workload that generates high-order folio demand and hence
> > > 
> > > What GFP flags are typical used for file-based workloads?
> > 
> > Mostly GFP_KERNEL, with a mix of GFP_NOFS. non-blocking paths also
> > tend to add GFP_NOWAITS, and memory reclaim sensitive paths often
> > use __GFP_MEMALLOC to prevent reclaim recursion. Some filesystems
> > also make extensive use of GFP_NOFAIL (e.g. XFS).
> > 
> > > > compaction (e.g copy tens of GB of files between two XFS, ext4 or
> > > > btrfs filesystems), that will -also- trash the Xe working set via
> > > > the shrinker being hammered by memory compaction try to free up
> > > > contiguous pages for the page cache.
> > > >
> > > 
> > > I could see this.
> > > 
> > > > Similarly, if we run a Xe workload that generates sustained high
> > > > order folio demand, that will trash the working set in the dentry
> > > > and inode caches and any other shrinkable slab-based cache.
> > > > 
> > > 
> > > I could also see this but DRM / Xe will set __GFP_NORETRY or
> > > __GFP_RETRY_MAYFAIL on higer-orders so those caches should be able to
> > > not trash its working set if looked for this hint. 
> > 
> > This relies on all the allocation code everywhere always doing
> > exactly the right thing so that memory reclaim "behaves". That is
> 
> Yes and no. The original issue was a DRM/TTM self-feedback loop, where
> we need some shrinking/reclaim, but not all. In general, this feedback
> loop also occurs on CPU/GPU client paths, so running XPS alongside heavy
> DRM/TTM with a client part is a very unlikely use case.
> 
> > what I've been saying is not a sustainable approach - all it takes
> > is one allocation or one shrinker not to do the right thing, and
> > we've got another mole to whack.  i.e. memory allocation should do
> > the right/best thing for the system with default parameters.
> > 
> 
> I agree that another allocation could shrink DRM/TTM buffers, but when
> DRM/TTM reallocates, it won’t trigger the same feedback loop since it
> uses the correct GFP flags. While it’s true that two conflicting
> allocations can ping-pong, we are also working on a DRM/TTM reclaim
> backoff mechanism and a defragmenter. However, if a self-feedback loop
> exists, this effort becomes ineffective.
> 
> > > > Hence the abstracted case of the problem we need to solve is this:
> > > > shrinker reclaim is based on x-byte objects is extremely unlikely to
> > > > acheive contiguous page reclaim in a manner that will measurably
> > > > improve compaction rates.
> > > > 
> > > > This is a problem that has to be addresses by the high level
> > > > infrastructure level, not worked around by individual shrinkers.
> > > > 
> > > > IMO, compaction shouldn't trigger shrinkers unless the shrinkers are
> > > > specifically flagged as being able to release contiguous pages of
> > > > memory in short order. I don't think there's very many shrinkable
> > > > caches that even hold a significant quantity of objects larger than
> > > > a single page, so it's clearly questionable as to whether compaction
> > > > based reclaim should run shrinker reclaim to begin with.
> > > > 
> > > 
> > > Yes, sort of do this in Xe by changing '->count_objects' based on the
> > > hint.
> > 
> > I know. That's the problem - it's relying on the infrastructure
> > passing down a specific internal context hint in an existing
> > interface so a specific subsystem can work around a specific
> > problematic behaviour.
> > 
> > Indeed, for compaction we don't actually care about the count, what
> > we largely care about is whether the subsystem has any objects the
> > same size or larger than the current compaction demand. Efficient
> > object reclaim for compaction has a different control variable set
> > (e.g. find objects larger than, objects physically near to, etc),
> > and this can't really be properly fitted into the existing
> > count/scan shrinker reclaim algorithm.
> > 
> > Hence I think it needs new shrinker methods to implement
> > effectively.
> 
> Looping back to direct reclaim above—would these new methods also be
> called during direct reclaim? My thinking is yes.
> 
> > 
> > > > i.e. a subsystem that can track high order folios in a shrinkable
> > > > cache should probably have a "->compaction_scan()" method that is
> > > > run directly from compaction context to try to free high order
> > > 
> > > When you say “compaction context,” which parts of the code are you
> > > referring to? I’d like to explore this option, but I need a bit more
> > > context.
> > 
> > kcompactd does background compaction, similar to how we have kswapd
> > to do background memory reclaim.
> > 
> 
> kcompactd doesn't call shrinkers - this is where we'd call compaction
> shrinkers?
> 
> > Direct compaction (part of direct reclaim) via
> > __alloc_pages_direct_compact() that will be called before direct
> > memory reclaim in the case of a high-order allocation.
> > 
> 
> So maybe this is where compaction shrinkers fits in in direct reclaim?
> 
> > > 
> > > > folios. This provides a direct opt-in mechanism for a subsystem, and
> > > > it allows subsystems that can track low- and high- order objects
> > > > independently to efficiently free objects in a way that will help
> > > > improve compaction rates without impacting the entire working set of
> > > > objects in the cache.
> > > 
> > > 
> > > Does this help if, for example, the cache is holding onto two order-8
> > > folios that could be freed and merged, while the caller really wants an
> > > order-9 folio? This seems like a possible scenario in caches and is
> > > certainly true in TTM pools.
> > 
> > Depends on how the interface is implemented.
> > 
> > IIUC, the direct compaction code will return a right-sized page
> > early if it creates one via compact_zone(). Hence if that path can
> > call into shrinkers to do high-order scanning that results in two
> > mergable order-8 objects being freed and merged into an order-9
> > object that fulfils the compaction requirements, then it will result
> > in compaction succeeding where it currently fails.
> >
> 
> I think this answers my question above - call compaction shrinker here.
> 
> > And I think that kcompactd will run until certain watermarks are
> > met, so again having a high-order shrinker that directly impacts the
> > high order page watermarks would be much more efficient that trying
> > to use general memory pressure to randomly shoot down enough objects
> > to reform contiguous pages.
> > 
> 
> I think this answers my question above - call compaction shrinker here.
> 
> > > > IOWs, this patch to inform kswapd about it's trigger (doesn't it
> > > > already have a "reason" parameter, though?) is likely a necessary
> > > > part of the solution - we don't want kswapd running shrinkers if it
> > > > has been triggered to reclaim pages for compaction. This patch would
> > > > allows kswapd to elide normal shrinker passes when it has been woken
> > > > purely for compaction purposes. Given that the compaction code would
> > > > be running the high-order reclaim capable shrinkers itself, this
> > > > would avoid trashing the working set of most shrinkable caches -by
> > > > default- under high order allocation demand....
> > > >
> > > 
> > > I’m trying to parse this—are you suggesting that, one way or another, we
> > > introduce a heuristic where shrinkers can act on a hint (whether it’s
> > > what I have here or a new ->compaction_scan() vfunc), and then attempt
> > > to fix all shrinkers in this series?
> > 
> > I don't want existing shrinkers to be touched at all.
> > 
> > What I want is for memory reclaim (both direct and kswapd) to elide
> > the shrink_slab() calls into shrinkers when memory reclaim is being
> > driven by high order allocation failure.
> > 

I went and traced the reclaim/compaction interaction in detail before
replying again - I don't think your suggestion works. Let me lay this out
in detail with examples.

The premise of eliding shrink_slab() is stated a few times in your
reply, e.g.,:

'shrinker reclaim is based on x-byte objects is extremely unlikely to
acheive contiguous page reclaim in a manner that will measurably
improve compaction rates.'

This is true for some shrinkers, such as slab-based shrinkers, but it is
clearly not true for others, such as TTM pools or GPU driver shrinkers
that reclaim purgeable or inactive memory outside the working set.
Additionally, XFS appears to retain significant amounts of memory
through folios and kvmalloc allocations.

This is somewhat of a moot point, though, because producing contiguous
memory is not reclaim's responsibility during high-order allocations.
Compaction is responsible for creating contiguous memory. Reclaim's job
is to free enough pages for compaction to run in the first place.

That's not my reading of intent, it's what the code says:

  include/linux/compaction.h:62

    /*
     * Number of free order-0 pages that should be available above given
     * watermark to make sure compaction has reasonable chance of not
     * running out of free pages that it needs to isolate as migration
     * target during its work.
     */
    static inline unsigned long compact_gap(unsigned int order)

  mm/vmscan.c:6064, above should_continue_reclaim()

    /*
     * Reclaim/compaction is used for high-order allocation requests. It
     * reclaims order-0 pages before compacting the zone.
     * should_continue_reclaim() returns true if more pages should be
     * reclaimed such that when the page allocator calls
     * try_to_compact_pages() that it will have enough free pages to
     * succeed.
     */

So the designed pipeline is "reclaim supplies free pages, compaction
assembles them". Scattered order-0 freeing isn't a failure mode of that
pipeline, it's the input to it.

The part that makes this concrete rather than philosophical is that the
free page supply is a hard gate on compaction, not a soft input to it:

  mm/compaction.c:2394, __compaction_suitable()

    watermark += compact_gap(order);
    if (order > PAGE_ALLOC_COSTLY_ORDER)
            watermark += low_wmark_pages(zone) - min_wmark_pages(zone);
    return __zone_watermark_ok(zone, 0, watermark, highest_zoneidx,
                               ALLOC_CMA, free_pages);

  mm/compaction.c:2525, compaction_suit_allocation_order()

    if (!compaction_suitable(zone, order, watermark, highest_zoneidx))
            return COMPACT_SKIPPED;

and compact_zone() bails at line 2563 on anything that isn't
COMPACT_CONTINUE.

Below that threshold compaction does not run at all. Not "runs less
effectively" - COMPACT_SKIPPED, before the migrate and free scanners
are ever started.

Now let's consider two examples:

- A memory-intensive GPU workload has run and completed, leaving pages
  either in the TTM pool or on the driver side outside the working set.
- XFS reclaimable cache is holding onto large amounts of memory.

If reclaim elides shrink_slab() for all high-order allocations, then on
that system:

  - the shrinker-owned memory can't be reclaimed at all
  - LRU reclaim alone cannot lift NR_FREE_PAGES over
    watermark + compact_gap(order),
  - so compaction_suitable() stays false,
  - so compaction never runs,
  - so the high-order allocation fails without a single compaction
    attempt having been made.

Today those allocations succeed, because shrinker-freed pages clear the
gate and compaction then does the assembly. So I think eliding
shrink_slab() for high-order reclaim would regress high-order/THP
allocation success rates.

Hillf made essentially this point upstream in this thread and I don't
think it got picked up:

  "compaction works only when there are enough free regular pages
   available, with nothing to do with shrinker directly"

So my thesis is that shrinkers must run before compaction. We should
give shrinkers a hint indicating whether reclaim is being performed
opportunistically for high-order allocations, in which case it may not
be worth evicting working set data that is expensive to save and
restore or if shrinking actively makes fragmentation worse.

I audited the shrinkers and found roughly ten that could make use of
this hint. Perhaps the most important is the deferred split shrinker,
which actively increases fragmentation when it runs.

By the way, I have the heuristic wrong here, since vma_thp_gfp_mask()
will not trigger the opportunistic hint, even though it should because
it has a fallback path.

So I'd like to post another revision of this series that fixes the
heuristic and also wires the hint up to the deferred split shrinker.
Then individual maintainers can decide whether and how to make use of
the hint in their own shrinkers.

Matt

> 
> I still think compaction shrinkers need a concept of “opportunistic”
> versus “must free memory to compact” to avoid the self-feedback loop I
> described. Without this, the DRM/TTM self-feedback loop would still
> exist.
> 
> > i.e. high-order allocation failure should not generate shrinkable
> > cache memory pressure because shrinkable caches in general cannot
> > return contiguous memory that will allows compaction to make
> 
> Yes, I agree with “shrinkable caches in general,” aside from the
> exceptions I outlined above (though perhaps I’m missing other
> exceptions).
> 
> The risk of what you’re suggesting is that it would suddenly change the
> entire shrinking model across Linux, and we don’t know what unforeseen
> consequences this type of change might have. Nor could a single person
> reasonably test a change of this scope. That’s why an incremental step,
> like this series—which fixes DRM/TTM as opt-in—may make more sense. As
> mentioned above, the deferred_split_shrinker is also a very likely
> candidate for being fixed. That said, I’m going to look into your
> suggestion as well.
> 
> > progress. The existing behaviour has more negative affects on system
> > performance than positive, so we need a fix for "everyone".
> > 
> > I think we should provide a new opt-in ->compaction_scan() method for
> > compaction aware subsystem shrinkers that is run from compact_zone()
> 
> compact_zone() appears to be called from both direct reclaim and
> kcompactd(), so perhaps some of the questions above are already
> answered.
> 
> > context. This allows subsystems that can manage high order objects
> > to optimise the return of high order objects to the free space pool,
> > thereby significantly improving the chance for compaction to
> > succeed without adversely impacting the rest of the shrinkable
> > caches in the system.
> > 
> > Further, we should not kick kswapd because of compaction failures
> > because kcompactd will already be running ->compaction_scan()
> > capable shrinkers from it's callouts to compact_zone() in the
> > background that will do this work as efficiently as possible.
> > 
> > > I’m open to trying to fix other
> > > shrinkers as well. Do you have any particular ones in mind? I count
> > > around 45 shrinkers in Linux, so it’s unlikely I can fix every single
> > > one, though or all shrinkers need to be fixed.
> > 
> > They'd all need to be fixed, which is why I suggested a new method
> > to be added. Avoid calling the existing shrinkers in the adverse
> > situation, call the new one from the right context where it actually
> > benefits compaction and high-order memory allocation.
> > 
> 
> OK, this seems fairly reasonable on its surface despite my concerns
> above, but see my comment above regarding “opportunistic” vs. “must free
> memory to compact.” Without that distinction, I don’t think my original
> problem would be solved.
> 
> Given this scope / risks, can I ask for one of two things:
> 
> - We move ahead with this series as a stopgap fix, or
> - Allow DRM/TTM/Xe to implement shrinker-side heuristics to break the
>   feedback loop
> 
> while I work on this larger refactor. This is the type of work I enjoy
> doing, and I typically have the flexibility to work on what I think is
> most impactful.
> 
> > > On a side note, I just noticed that struct shrinker has count_objects
> > > and scan_objects as individual vfuncs rather than using a const struct
> > > shrinker_ops *ops. Should we change that? The latter seems cleaner and
> > > is typically how things are done in Linux.
> > 
> > We probably should - the current structure is largely historical and
> > there's only ever been two methods. If we are adding another method,
> > then it would probably make sense to add an external ops structure
> > to reduce the memory footprint a little.
> > 
> 
> Let me look into this as well. I don’t know the exact merge flows for
> tree-wide changes, but I can figure it out.
> 
> Matt
> 
> > -Dave.
> > -- 
> > Dave Chinner
> > dgc@kernel.org