Re: [PATCH] various allocator optimizations

[Hans Reiser <reiser@namesys.com>]

Chris Mason wrote:

>On Thu, 2003-03-13 at 19:15, Hans Reiser wrote:
>
>  
>
>>>Ok, more testing showed that patch wasn't bad, but wasn't great either. 
>>>Under a multiprocess test, the buffers in the path would get moved away
>>>and there wasn't enough data to make a good decision.  So the patch
>>>below changes things around a little, and records the span during
>>>search_by_key instead of trying to compute it after the search is over.
>>>
>>>      
>>>
>>Hunh?  Why don't you maintain a counter in the directory of the number 
>>of nodes in it?  Or are you afraid of causing extra IO?
>>
>>    
>>
>
>That would mean the parent directory counter would have to be updated
>every time we allocated a block in any sub directory.  Plus the counter
>would have to be inherited down the chain in deep directory structures. 
>More importantly, I'd rather not waste space in the stat data to store
>the information when we can get it during a search ;-)
>
The space usage is trivial.

>
>Basing the span on the tree allows us to detect a highly used locality
>regardless of what kind of tree object was using it, and we can do it
>with almost zero overhead.
>
>  
>
>>>In other words, I think this is a really good compromise between the
>>>current defaults and a more optimal case for fragmentation, and I expect
>>>its performance to hold up much better as the filesystem ages.
>>>
>>>      
>>>
>>Let's get lots of different testers.  You may have a nice heuristic here 
>>though....
>>
>>    
>>
>
>If everyone agrees the approach is worth trying, I'll make a patch that
>enables it via a mount option.
>
>  
>
>>How big are your packing localities tending to be?
>>    
>>
>
>Not more than can be pointed to by the leaf level and the level directly
>above it.  I know that's not very specific, but it varies by the
>dataset.  packed tails and long directory names lead to more packing
>localities per MB.
>
Which is why it is the wrong measure, yes?

>
>My 100MB subset of /usr/share/doc has 732 directories in it, and the
>copy phase of stress.sh -n 10 -s for that directory tree will produce
>1400 packing localities, so it depends on size of the tree in general.
>
>-chris
>
>
>
>
>  
>
I am not saying that storing it in the directory is the right thing, 
because of the risk of extra IO, but I think it should be tried.

-- 
Hans