heavy file i/o on ext3 filesystem leads to huge ext3_inode_cache and dentry_cache that doesn't return to normal for hours

heavy file i/o on ext3 filesystem leads to huge ext3_inode_cache and dentry_cache that doesn't return to normal for hours

[Maarten Maathuis]

I have a kernel specific problem and this seemed like a suitable place to ask.

I would like responces to be CC'ed to me if possible.

I use a 2.6.17-ck1 kernel on an amd64 system. I have observed this
problem on other/older kernels.

Whenever there is serious hard drive activity (such as doing "slocate
-u") ext3_inode_cache and dentry_cache grow to a combined 400-500 MiB.

The amount of objects is more than half a million.

This will slowly decrease to normal, but will take many hours. It does
not result in any OOM, because i have 1 GiB of memory.

As far as i understand hard drive cache should not be in the slab. Are
these just the inode's, because the amount of memory consumption seems
large for that?

I have found a way to clear the memory (and unfortunately most of the cache):

echo 100 > /proc/sys/vm/nr_hugepages
echo 0 > /proc/sys/vm/nr_hugepages

This suggest the kernel can free this memory. It's not the caching
that bothers me, what bothers me is that it seems to reside in the
slab.

I am not a developer, so please keep that in mind when replying.

I hope someone can be of help.

Sincerely,

Maarten Maathuis.

Re: heavy file i/o on ext3 filesystem leads to huge ext3_inode_cache and dentry_cache that doesn't return to normal for hours

[Dave Kleikamp]

On Thu, 2006-08-03 at 18:27 +0200, Maarten Maathuis wrote:
> I have a kernel specific problem and this seemed like a suitable place to ask.
>
> I would like responces to be CC'ed to me if possible.
> 
> I use a 2.6.17-ck1 kernel on an amd64 system. I have observed this
> problem on other/older kernels.
> 
> Whenever there is serious hard drive activity (such as doing "slocate
> -u") ext3_inode_cache and dentry_cache grow to a combined 400-500 MiB.

The behavior of slocate (updatedb) is pretty well-known, but nobody has
come up with a real solution.

> The amount of objects is more than half a million.
> 
> This will slowly decrease to normal, but will take many hours. It does
> not result in any OOM, because i have 1 GiB of memory.
> 
> As far as i understand hard drive cache should not be in the slab. Are
> these just the inode's, because the amount of memory consumption seems
> large for that?

inodes and directory cache entries (dentries).  In general, it's a good
idea to cache inodes and dentries that have recently been read.  slocate
is a special case since it will traverse all of the directories and
never look at them again (until the next time it runs).  The kernel
doesn't have any idea that it may be a good idea to free those objects.

> 
> I have found a way to clear the memory (and unfortunately most of the cache):
> 
> echo 100 > /proc/sys/vm/nr_hugepages
> echo 0 > /proc/sys/vm/nr_hugepages

A better way to clear just the inodes and dentries (that aren't in use)
is:

echo 2 > /proc/sys/vm/drop_caches

This feature is relatively new.

> This suggest the kernel can free this memory. It's not the caching
> that bothers me, what bothers me is that it seems to reside in the
> slab.

I'm not sure why that bothers you.  A more common complaint is that all
the inodes and dentries being cached push out other pages to swap.

Completely free memory doesn't do the system any good.  The kernel
attempts to keep as much as possible in cache in case something is
needed again.  These objects are easily reclaimable, so when more memory
is needed, they can be freed with very little overhead.

> I am not a developer, so please keep that in mind when replying.
> 
> I hope someone can be of help.
> 
> Sincerely,
> 
> Maarten Maathuis.

Shaggy
-- 
David Kleikamp
IBM Linux Technology Center

Re: heavy file i/o on ext3 filesystem leads to huge ext3_inode_cache and dentry_cache that doesn't return to normal for hours

[Jeffrey V. Merkey]

Dave Kleikamp wrote:

>On Thu, 2006-08-03 at 18:27 +0200, Maarten Maathuis wrote:
>  
>
>>I have a kernel specific problem and this seemed like a suitable place to ask.
>>
>>I would like responces to be CC'ed to me if possible.
>>
>>I use a 2.6.17-ck1 kernel on an amd64 system. I have observed this
>>problem on other/older kernels.
>>
>>Whenever there is serious hard drive activity (such as doing "slocate
>>-u") ext3_inode_cache and dentry_cache grow to a combined 400-500 MiB.
>>    
>>
>
>The behavior of slocate (updatedb) is pretty well-known, but nobody has
>come up with a real solution.
>
>  
>
>>The amount of objects is more than half a million.
>>
>>This will slowly decrease to normal, but will take many hours. It does
>>not result in any OOM, because i have 1 GiB of memory.
>>
>>As far as i understand hard drive cache should not be in the slab. Are
>>these just the inode's, because the amount of memory consumption seems
>>large for that?
>>    
>>
>
>inodes and directory cache entries (dentries).  In general, it's a good
>idea to cache inodes and dentries that have recently been read.  slocate
>is a special case since it will traverse all of the directories and
>never look at them again (until the next time it runs).  The kernel
>doesn't have any idea that it may be a good idea to free those objects.
>
>  
>
>>I have found a way to clear the memory (and unfortunately most of the cache):
>>
>>echo 100 > /proc/sys/vm/nr_hugepages
>>echo 0 > /proc/sys/vm/nr_hugepages
>>    
>>
>
>A better way to clear just the inodes and dentries (that aren't in use)
>is:
>
>echo 2 > /proc/sys/vm/drop_caches
>
>This feature is relatively new.
>
>  
>
>>This suggest the kernel can free this memory. It's not the caching
>>that bothers me, what bothers me is that it seems to reside in the
>>slab.
>>    
>>
>
>I'm not sure why that bothers you.  A more common complaint is that all
>the inodes and dentries being cached push out other pages to swap.
>
>Completely free memory doesn't do the system any good.  The kernel
>attempts to keep as much as possible in cache in case something is
>needed again.  These objects are easily reclaimable, so when more memory
>is needed, they can be freed with very little overhead.
>
>  
>
>>I am not a developer, so please keep that in mind when replying.
>>
>>I hope someone can be of help.
>>
>>Sincerely,
>>
>>Maarten Maathuis.
>>    
>>
>
>Shaggy
>  
>
dcache architecture is one of the big bottlenecks. 

Jeff

Re: heavy file i/o on ext3 filesystem leads to huge ext3_inode_cache and dentry_cache that doesn't return to normal for hours

[Maarten Maathuis]

On 8/4/06, Dave Kleikamp <shaggy@austin.ibm.com> wrote:
> On Thu, 2006-08-03 at 18:27 +0200, Maarten Maathuis wrote:
> > I have a kernel specific problem and this seemed like a suitable place to ask.
> >
> > I would like responces to be CC'ed to me if possible.
> >
> > I use a 2.6.17-ck1 kernel on an amd64 system. I have observed this
> > problem on other/older kernels.
> >
> > Whenever there is serious hard drive activity (such as doing "slocate
> > -u") ext3_inode_cache and dentry_cache grow to a combined 400-500 MiB.
>
> The behavior of slocate (updatedb) is pretty well-known, but nobody has
> come up with a real solution.
>
> > The amount of objects is more than half a million.
> >
> > This will slowly decrease to normal, but will take many hours. It does
> > not result in any OOM, because i have 1 GiB of memory.
> >
> > As far as i understand hard drive cache should not be in the slab. Are
> > these just the inode's, because the amount of memory consumption seems
> > large for that?
>
> inodes and directory cache entries (dentries).  In general, it's a good
> idea to cache inodes and dentries that have recently been read.  slocate
> is a special case since it will traverse all of the directories and
> never look at them again (until the next time it runs).  The kernel
> doesn't have any idea that it may be a good idea to free those objects.
>

Wasn't there something like a possible read only flag to prevent that?

> >
> > I have found a way to clear the memory (and unfortunately most of the cache):
> >
> > echo 100 > /proc/sys/vm/nr_hugepages
> > echo 0 > /proc/sys/vm/nr_hugepages
>
> A better way to clear just the inodes and dentries (that aren't in use)
> is:
>
> echo 2 > /proc/sys/vm/drop_caches
>
> This feature is relatively new.
>

Thank you, i tried echo'ing a 1 into that and it had no effect iirc.
Documentation on /proc/sys/vm seems pretty scarce.

> > This suggest the kernel can free this memory. It's not the caching
> > that bothers me, what bothers me is that it seems to reside in the
> > slab.
>
> I'm not sure why that bothers you.  A more common complaint is that all
> the inodes and dentries being cached push out other pages to swap.
>

The reason i asked was because i expected cache related things to be
seperated from "normal" memory usage. Since free already seperates
normal and buffers/cache, it was very strange seeing 400 or 500 MiB
memory usage.

> Completely free memory doesn't do the system any good.  The kernel
> attempts to keep as much as possible in cache in case something is
> needed again.  These objects are easily reclaimable, so when more memory
> is needed, they can be freed with very little overhead.
>
> > I am not a developer, so please keep that in mind when replying.
> >
> > I hope someone can be of help.
> >
> > Sincerely,
> >
> > Maarten Maathuis.
>
> Shaggy
> --
> David Kleikamp
> IBM Linux Technology Center
>
>

Re: heavy file i/o on ext3 filesystem leads to huge ext3_inode_cache and dentry_cache that doesn't return to normal for hours

[Andrew Morton]

On Fri, 4 Aug 2006 09:53:14 +0200
"Maarten Maathuis" <madman2003@gmail.com> wrote:

> On 8/4/06, Dave Kleikamp <shaggy@austin.ibm.com> wrote:
> > On Thu, 2006-08-03 at 18:27 +0200, Maarten Maathuis wrote:
> > > I have a kernel specific problem and this seemed like a suitable place to ask.
> > >
> > > I would like responces to be CC'ed to me if possible.
> > >
> > > I use a 2.6.17-ck1 kernel on an amd64 system. I have observed this
> > > problem on other/older kernels.
> > >
> > > Whenever there is serious hard drive activity (such as doing "slocate
> > > -u") ext3_inode_cache and dentry_cache grow to a combined 400-500 MiB.

Increasing /proc/sys/vm/vfs_cache_pressure should increase the inode/dentry
reclaim rate.

> >
> > echo 2 > /proc/sys/vm/drop_caches
> >
> > This feature is relatively new.
> >
> 
> Thank you, i tried echo'ing a 1 into that and it had no effect iirc.

1 drops pagecache, 2 drops dentries and inodes.  Pagecache pins inodes, so
using 2 will drop less inodes than using 3.

> Documentation on /proc/sys/vm seems pretty scarce.

Documentation/filesystems/proc.txt

Re: heavy file i/o on ext3 filesystem leads to huge ext3_inode_cache and dentry_cache that doesn't return to normal for hours

[Alan Cox]

Ar Gwe, 2006-08-04 am 09:53 +0200, ysgrifennodd Maarten Maathuis:
> > > Whenever there is serious hard drive activity (such as doing "slocate
> > > -u") ext3_inode_cache and dentry_cache grow to a combined 400-500 MiB.
> >
> > The behavior of slocate (updatedb) is pretty well-known, but nobody has
> > come up with a real solution.

Thats not actually true. There is patch to page back in code on a quiet
machine which is in -mm and is designed to pull stuff back after a busy
period. There was also discussion (but not yet code) for splitting the
dcache into two and being much smarter about recycling it.