Re: Caching/buffers become useless after some time

[Michal Hocko <mhocko@suse.com>]

On Tue 21-08-18 02:36:05, Marinko Catovic wrote:
[...]
> > > Well, there are some drivers (mostly out-of-tree) which are high order
> > > hungry. You can try to trace all allocations which with order > 0 and
> > > see who that might be.
> > > # mount -t tracefs none /debug/trace/
> > > # echo stacktrace > /debug/trace/trace_options
> > > # echo "order>0" > /debug/trace/events/kmem/mm_page_alloc/filter
> > > # echo 1 > /debug/trace/events/kmem/mm_page_alloc/enable
> > > # cat /debug/trace/trace_pipe
> > >
> > > And later this to disable tracing.
> > > # echo 0 > /debug/trace/events/kmem/mm_page_alloc/enable
> >
> > I just had a major cache-useless situation, with like 100M/8G usage only
> > and horrible performance. There you go:
> >
> > https://nofile.io/f/mmwVedaTFsd

$ grep mm_page_alloc: trace_pipe | sed 's@.*order=\([0-9]*\) .*gfp_flags=\(.*\)@\1 \2@' | sort | uniq -c
    428 1 __GFP_FS|__GFP_NOWARN|__GFP_NORETRY|__GFP_COMP|__GFP_NOMEMALLOC|__GFP_THISNODE
     10 1 __GFP_HIGH|__GFP_ATOMIC|__GFP_NOWARN|__GFP_COMP|__GFP_THISNODE
      6 1 __GFP_IO|__GFP_FS|__GFP_NOWARN|__GFP_COMP|__GFP_THISNODE
   3061 1 GFP_KERNEL_ACCOUNT|__GFP_ZERO
   8672 1 GFP_NOWAIT|__GFP_IO|__GFP_FS|__GFP_NOWARN|__GFP_NORETRY|__GFP_COMP|__GFP_ACCOUNT
   2547 1 __GFP_NOWARN|__GFP_NORETRY|__GFP_COMP|__GFP_NOMEMALLOC|__GFP_THISNODE
      4 2 __GFP_HIGH|__GFP_ATOMIC|__GFP_NOWARN|__GFP_COMP|__GFP_THISNODE
      5 2 __GFP_IO|__GFP_FS|__GFP_NOWARN|__GFP_COMP|__GFP_THISNODE
  20030 2 GFP_NOWAIT|__GFP_IO|__GFP_FS|__GFP_NOWARN|__GFP_NORETRY|__GFP_COMP|__GFP_ACCOUNT
   1528 3 GFP_ATOMIC|__GFP_NOWARN|__GFP_NORETRY|__GFP_COMP|__GFP_NOMEMALLOC
   2476 3 GFP_NOWAIT|__GFP_IO|__GFP_FS|__GFP_NOWARN|__GFP_NORETRY|__GFP_COMP
   6512 3 GFP_NOWAIT|__GFP_IO|__GFP_FS|__GFP_NOWARN|__GFP_NORETRY|__GFP_COMP|__GFP_ACCOUNT
    277 9 GFP_TRANSHUGE|__GFP_THISNODE

This only covers ~90s of the allocator activity. Most of those requests
are not troggering any reclaim (GFP_NOWAIT/ATOMIC). Vlastimil will
know better but this might mean that we are not envoking kcompactd
enough. But considering that we have suspected that an overly eager
reclaim triggers the page cache reduction I am not really sure I see the
above to match that theory.

Btw. I was probably not specific enough. This data should be collected
_during_ the time when the page cache is disappearing. I suspect you
have started collecting after the fact.

Btw. vast majority of order-3 requests come from the network layer. Are
you using a large MTU (jumbo packets)?

-- 
Michal Hocko
SUSE Labs