Re: [PATCH] bcachefs: Switch to memalloc_flags_do() for vmalloc allocations

[Yafang Shao <laoar.shao@gmail.com>]

On Fri, Aug 30, 2024 at 11:25 PM Vlastimil Babka <vbabka@suse.cz> wrote:
>
> On 8/30/24 11:14, Yafang Shao wrote:
> > On Thu, Aug 29, 2024 at 10:29 PM Dave Chinner <david@fromorbit.com> wrote:
> >
> > Hello Dave,
> >
> > I've noticed that XFS has increasingly replaced kmem_alloc() with
> > __GFP_NOFAIL. For example, in kernel 4.19.y, there are 0 instances of
> > __GFP_NOFAIL under fs/xfs, but in kernel 6.1.y, there are 41
> > occurrences. In kmem_alloc(), there's an explicit
> > memalloc_retry_wait() to throttle the allocator under heavy memory
> > pressure, which aligns with your filesystem design. However, using
> > __GFP_NOFAIL removes this throttling mechanism, potentially causing
> > issues when the system is under heavy memory load. I'm concerned that
> > this shift might not be a beneficial trend.
> >
> > We have been using XFS for our big data servers for years, and it has
> > consistently performed well with older kernels like 4.19.y. However,
> > after upgrading all our servers from 4.19.y to 6.1.y over the past two
> > years, we have frequently encountered livelock issues caused by memory
> > exhaustion. To mitigate this, we've had to limit the RSS of
> > applications, which isn't an ideal solution and represents a worrying
> > trend.
>
> By "livelock issues caused by memory exhaustion" you mean the long-standing
> infamous issue that the system might become thrashing for the remaining
> small amount of page cache, and anonymous memory being swapped out/in,
> instead of issuing OOM, because there's always just enough progress of the
> reclaim to keep going, but the system isn't basically doing anything else?
>

Exactly

> I think that's related to near-exhausted memory by userspace,

If user space is the root cause, the appropriate response should be to
terminate the offending user tasks. However, this doesn't happen at
all.

> so I'm not
> sure why XFS would be to blame here.

Honestly, I'm not sure what to blame, as I don't have a clear
understanding of what's happening during memory allocation. One server
among tens of thousands in production randomly experiences a livelock
within days, making it extremely difficult to pinpoint the root cause.

>
> That said, if memalloc_retry_wait() is indeed a useful mechanism, maybe we
> could perform it inside the page allocator itself for __GFP_NOFAIL?

Perhaps an additional wait or exit mechanism should be implemented
specifically for __GFP_NOFAIL.

--
Regards
Yafang