Re: [PATCH 00/15] git-note: A mechanisim for providing free-form after-the-fact annotations on commits

["Shawn O. Pearce" <spearce@spearce.org>]

Linus Torvalds <torvalds@linux-foundation.org> wrote:
> 
> On Mon, 28 May 2007, Shawn O. Pearce wrote:
> > 
> > What if we use a "slow" storage by "refs/notes/$objname/$notename",
> 
> This _really_ won't scale. Even if the notes don't exist, just doing the 
> lookup (which will fail for most commits) will be horribly slow, and will 
> populate the dentry cache with negative entries.

Yes.

I think you missed what I was trying to say.  We *definately* do not
want to probe the OS and ask "do you have $objname1? $objname2?"
for exactly the reason you just stated.  (Though you being some
sort of kernel guru means you know a hellva lot more about that
dentry cache thing than I do!)

What I meant was something more like:

	DIR *d = opendir("refs/notes");
	if (d) {
		... notes can exist in both packfiles and "loose" ...
		only_packed_notes = 0;

		while (readdir(d)) {
			... the entry name here is the name of an object ...
			... stuff into a hash table, we can come back later ...
			... for the subdirectory contents if we need it ...
		}
		closedir(d);
		... also examine "packed-refs", in case any are there ...
	} else {
		only_packed_notes = 1;
		... we *only* have notes in packfiles, if we have any at all...
	}

Then looking up a note is just a probe into our in-memory hash
(if only_packed_notes is false) and a probe into the packfile(s) to
find the notes for the object.  Not very expensive if the packfiles
have the reverse obj->tag mappings indexed within them.
 
> To get good filesystem performance, you have to
>  - have reasonable hit-rates (and looking it up for each commit is _not_ 
>    going to do that)
>  - not have millions of objects.

Which is why we:

  a) allow these things to migrate into packed-refs, because
  getting into there is a hellva lot cheaper than getting into
  a packfile;

  b) move them into a packfile when we repack loose objects,
  because then we have really good access.

  c) take them out of packed-refs once they are into a packfile,
  and get them out of the loose refs/notes directory as early as
  possible.

-- 
Shawn.