Re: [PATCH 1/3] Lazily open pack index files on demand

["Shawn O. Pearce" <spearce@spearce.org>]

Junio C Hamano <junkio@cox.net> wrote:
> "Shawn O. Pearce" <spearce@spearce.org> writes:
> 
> >  This conflicts (in a subtle way) with Dana How's
> >  "sha1_file.c:rearrange_packed_git() should consider packs' object
> >  sizes" patch as we now have num_objects = 0 for any indexes we
> >  have not opened.  In the case of Dana's patch this would cause
> >  those packfiles to have very high ranks, possibly sorting much
> >  later than they should have.
> 
> I am keeping that rearrange stuff on hold, partly because I am
> moderately hesitant to do the fp, which feels overkill at that
> low level of code.

Yea, I've actually been having similiar thoughts.
 
> Also, I am hoping that we can discard that the object density
> criteria altogether by making the default repack behaviour
> friendlier to the pathological cases, e.g. by emitting huge
> blobs at the end of the packstream, potentially pushing it out
> to later parts of split packs by themselves and automatically
> marking them with the .keep flag.  Until that kind of
> improvements materialize, people with pathological cases could
> (1) handcraft a pack that contains only megablob, (2) place that
> on central alternate, (3) touch it with artificially old
> timestamp, which hopefully is a good enough workaround.

Right, I was having the same idea.  If we have pack-objects just
shuffle the really big stuff to the end of its object list they
will naturally fall into the end of the packfile, and the split
out packfiles.  Then if we do the mtime flipping you suggested
earlier right before we exit pack-objects the larger blob packs
will automatically sort behind the smaller commit/tree packs.
No fp needed.

My patch was exactly because I did what you say above; I handcrafted
a pack that contains only large-ish blobs, placed them into a
central repo, and connected it by alternates.  Because of the
local flag logic it is automatically behind my commit/tree pack.
But I also rarely (if ever) have to access that megablob packfile.
Yet the .idx was still being opened.  On Cygwin/Windows that penalty
is high enough to have almost doubled the running time of a simple
"git show".

-- 
Shawn.