Re: Show of hands, how many set USE_NSEC

[Linus Torvalds <torvalds@linux-foundation.org>]

On Fri, 8 Aug 2008, Shawn O. Pearce wrote:
> 
> Really I'd just like to scrap the entire DIRC file format and do
> it over again.  Having the flat namespace is nuts.  Linus and I
> really disagree here, and since I have never produced code for C
> Git to replace it (and prove why its better) I think he has me in
> his kill file now.  :)

I really disagree, because you have no clue about performance.

The flat file format is absolutely _critical_ for performance.

Try this:

	time cat .git/index > /dev/null
	time git ls-tree -r HEAD > /dev/null

which isn't quite fair (but see how it's unfair both ways later), but 
gives you an idea of the cost of recursively reading lots of small files.

Notice how the latter is about an order-and-a-half slower? 

Then try it with a cold cache. If you thought the "ls-tree" was unfair 
because it used things like zlib and pack-files, realize that the ls-tree 
actually almost certainly has _better_ IO patterns than doing individual 
files. So when you do the cold-cache case, we're actually giving an unfair 
advantage to ls-tree (assuming fully packed repo like I have).

And it still loses in a big big way. If it was actually one file per 
directory, it would be _horrible_. You could kind of approximate the IO 
patterns for that case by doing

	time find . -name don-t-exist > /dev/null

(which obviousyl uses "readdir()" instead of read(), but the IO patterns 
should be similar).

The point is, the index file absolutely *HAS* to be a single file in order 
to perform well in a big project. Otherwise there's no point, and you 
might as well just use a git "tree" object for everything.

Now, if you talk about the _sorting_ order (as opposed to the flatness of 
the file), I could probably agree. The sort order was probably a mistake. 
That said, we're stuck with it.  You can't change it without changing the 
tree object format, so it's not just an "local index file" format issue.

			Linus