Re: userspace pagecache management tool

[Andrew Morton <akpm@linux-foundation.org>]

On Sat, 3 Mar 2007 14:58:48 -0800 "Ray Lee" <madrabbit@gmail.com> wrote:

> On 3/3/07, Andrew Morton <akpm@linux-foundation.org> wrote:
> > It is to address the "waah, backups fill my memory with pagecache" and the
> > "waah, updatedb swapped everything out" and the "waah, copying a DVD
> > gobbled all my memory" problems.
> 
> Is the updatedb problem really due to pagecache?

It's a combination of pagecache, slab cache and of course contention for
the disk.  In my experience the latter preponderates: the disk is sekeing
like mad and I can't get its attention.  Others report lots of swapout,
which will be a combination of slab and pagecache, varying degrees of each.

> > When running
> >
> >         pagecache-management.sh dd if=100-mb-file of=foo
> >         or
> >         pagecache-management.sh cp -a /usr/src/linux-2.6.20 /usr/src/foo
> >
> > the amount of pagecache in the machine is pretty much unaltered.  Maybe a
> > megabyte of additional cache in the second case, because of ext3 indirect
> > blocks.
> 
> ray@phoenix:~/work/home/pagecache-management$ grep ext3_i
> /proc/slabinfo; ./pagecache-management.sh sudo updatedb; grep ext3_i
> /proc/slabinfo
> ext3_inode_cache   21024  23722   1584    2    1 : tunables   24   12
>   0 : slabdata  11861  11861      0
> ext3_inode_cache   41332  41332   1584    2    1 : tunables   24   12
>   0 : slabdata  20666  20666      0
> ray@phoenix:~/work/home/pagecache-management$ echo $(( 1584 * (41332-21024) ))
> 32167872

If 32 MB is the whole lot then by eliminating pagecache, we just solved the
problem.  But perhaps you instantiated a lot more VFS cache and all you're
seeing there is the leftovers.

> Or is there a /proc/sys/vm/* knob that can be tweaked for this
> before/after the updatedb?

/proc/sys/vm/vfs_cache_pressure should help.  I don't recall anyone
reporting its effects with updatedb.

> But yeah, I for one would happily submit patches to upstream authors
> to address this there. There's no reason code should be making the
> kernel guess its intention on these things.

I think so.  We're dealing with super-special cases here and often trying
to fix those in-kernel will degrade other, often more common cases.

<wonders about sys_reclaim_dentry(const char *pathname)>