Re: [PATCH] cvsimport: introduce -L<imit> option to workaround memory leaks

[Linus Torvalds <torvalds@osdl.org>]

On Tue, 23 May 2006, Theodore Tso wrote:
> On Mon, May 22, 2006 at 07:28:37PM -0700, Linus Torvalds wrote:
> > 
> > I actually think that I found a real ext3 performance bug from trying to 
> > determine why git sometimes slows down ridiculously when the tree has been 
> > allowed to go too long without a repack.
> 
> Do you have dir_index (the hashed btree) feature enabled by any chance?

No, and I know I probably should, since it would hopefully help git usage.

But my problem actually happens even with moderately sized directories: 
they were just 40kB or so in size, and the problem isn't high system CPU 
usage, but tons of extra IO. I ran things on a machine with 2GB of RAM, 
and as far as I could tell, the working set _should_ have fit into memory, 
but CPU utilization was consistently in the 1% range.

Now, it's possible that I'm just wrong, and it really didn't fit in 
memory, but I I _suspect_ that the issue is that ext3 directory handling 
still uses the "buffer_head" thing rather than the page cache, and that we 
simply don't LRU the memory appropriately so we don't let the memory 
pressure expand the buffer cache.

Now, using buffer cache in this day and age is insane and horrible 
(there's a reason I suspect the LRU doesn't work that well: the buffer 
heads aren't supposed to be used as a cache, and people are supposed to 
use the page cache for it these days), but Andrew tells me that the whole 
JBD thing basically requires it. Whatever.

Now, repacking obviously hides it entirely (because then the load becomes 
entirely a page-cache load, and the kernel does _that_ beautifully), but 
I'm a bit bummed that I think I hit an ext3 braindamage.

So an unpacked git archive on ext3 (but not ext2, I believe: ext2 should 
use the page cache for directories) ends up being very buffer-cache 
intensive. And the buffer cache is basically deprecated..

		Linus

PS. I'll see if I can figure out the problem, and maybe the good news is 
that I'll be able to just fix a real kernel performance issue. Still, 
there's a _reason_ we tried to get away from the buffer heads as a caching 
entity..