Re: [RFC] Design for http-pull on repo with packs

[Dan Holmsand <holmsand@gmail.com>]

Junio C Hamano wrote:
> One very minor problem I have with Holmsand approach [*1*] is
> that the original Barkalow puller allowed a really dumb http
> server by not requiring directory index at all.  For somebody
> like me with a cheap ISP account [*2*], it was great that I did
> not have to update 256 index.html files for objects/??/
> directories.  Admittedly, it would be just one directory
> object/pack/, but still...

I totally agree that you shouldn't have to do any special kind of 
prepping to serve a repository thru http. Which was why I thought it was 
a good thing to use the default directory listing of the web-server, 
assuming that this feature would be available on most servers... 
Apparently not yours, though :-(

And Cogito already relies on directory listings (to find tags to download).

But if git-repack-script generates a "pack index file" automagically, 
then of course everything is fine.

> On the other hand, picking an optimum set of packs from
> overlapping set of packs is indeed a very interesting (and hard
> combinatorial) problem to solve.  I am hoping that in practice
> people would not force clients to do it with "interesting" set
> of packs.  I would hope them to have just a full pack and
> incrementals, never having ovelaps, like Linus plans to do on
> his kernel repo.
> 
> On the other hand, for somebody like Jeff Garzik with 50 heads,
> it might make some sense to have a handful different overlapping
> packs, optimized for different sets of people wanting to pull
> some but not all of his heads.

Well, it is an interresting problem... But I don't think that the 
solution is to create more pack files. In fact, you'd want as few pack 
files as possible, for maximum overall efficiency.

I did a little experiment. I cloned Linus' current tree, and git 
repacked everything (that's 63M + 3.3M worth of pack files). Then I got 
something like 25 or so of Jeff's branches. That's 6.9M of object files, 
and 1.4M packed. Total size: 70M for the entire .git/objects/pack directory.

Repacking all of that to a single pack file gives, somewhat 
surprisingly, a pack size of 62M (+ 1.3M index). In other words, the 
cost of getting all those branches, and all of the new stuff from Linus, 
turns out to be *negative* (probably due to some strange deltification 
coincidence).

I think that this shows that (at least in this case), having many 
branches isn't particularly wasteful (1.4M in this case with one 
incremental pack).

And that fewer packs beats many packs quite handily.

The big problem, however, comes when Jeff (or anyone else) decides to 
repack. Then, if you fetch both his repo and Linus', you might end up 
with several really big pack files, that mostly overlap. That could 
easily mean storing most objects many times, if you don't do some smart 
selective un/repacking when fetching.

/dan