Re: [PATCH] commit: don't rewrite shared index unnecessarily

[David Turner <dturner@twopensource.com>]

On Mon, 2015-08-31 at 17:02 +0700, Duy Nguyen wrote:
> On Fri, Aug 28, 2015 at 12:07 AM, David Turner <dturner@twopensource.com> wrote:
> > Remove a cache invalidation which would cause the shared index to be
> > rewritten on as-is commits.
> >
> > When the cache-tree has changed, we need to update it.  But we don't
> > necessarily need to update the shared index.  So setting
> > active_cache_changed to SOMETHING_CHANGED is unnecessary.  Instead, we
> > let update_main_cache_tree just update the CACHE_TREE_CHANGED bit.
> >
> > In order to test this, make test-dump-split-index not segfault on
> > missing replace_bitmap/delete_bitmap.  This new codepath is not called
> > now that the test passes, but is necessary to avoid a segfault when the
> > new test is run with the old builtin/commit.c code.
> >
> > Signed-off-by: David Turner <dturner@twopensource.com>
> 
> Ack.
> 
> I made SOMETHING_CHANGED "1" for catching these cases (there were a
> few on-flight topics when this series was being cooked and I was
> afraid I could not cache all active_cache_changed sites).
> 
> > ---
> >
> > I introduced this bug last year while improving the cache-tree code.
> > I guess I probably didn't notice that active_cache_changed wasn't a
> > boolean.
> 
> So.. you did you split-index? Cool. Never heard anyone using it for
> real. It needs the other part to improve reading/refresh side to get
> to full potential though..

I am experimenting with an "unskipped index", to reduce the cost of
index loading and processing in sparse checkouts.  The unskipped index
is an index on the (v3) index -- that is, it lists the byte offsets of
all entries in the sharedindex where CE_SKIP_WORKTREE is 0.  In
addition, there is a list of the offsets of *all* entries, which we can
binary search in.  There's also some other metadata that I'm figuring
out as I go along.  This might allow us to avoid loading the full shared
index in most cases.  Or it might be a total dead end.  Not sure yet.