Removing redundant packs

Removing redundant packs

[Petr Baudis]

  Hi,

  before exporting whole repo.or.cz over rsync for mirroring, I would
like to clean up the worst repositories (esp. linux-2.6 repositories)
whose objects database looks absolutely horrible now.

  At the same time, I have to be careful not to prune all unreferenced
objects since they can be referenced in forks.

  How would you go about it? My idea was to do a fresh cummulative pack
of all referenced objects with

	git repack -a

and then remove packs that contain _only_ objects that are already in
the fresh pack. But I have trouble with the second step; a very naive
grep loop ran for about a day when I stopped it and it was nowhere near
the half of the packs yet. I didn't figure out how to make use of
git-pack-redundant either.

  Does anyone have any hints?

  Thanks!

-- 
				Petr "Pasky" Baudis
We don't know who it was that discovered water, but we're pretty sure
that it wasn't a fish.		-- Marshall McLuhan

Re: Removing redundant packs

[Junio C Hamano]

Petr Baudis <pasky@ucw.cz> writes:

> At the same time, I have to be careful not to prune all unreferenced
> objects since they can be referenced in forks.

How about...

 (1) In each repository, run "repack -a -d".  That would ensure that
     everybody has the necessary objects that they themselves need.  By
     doing this for a repository that borrows from another makes sure
     pruning the latter would not break the former, so you start from
     leaves and move on to the repositories they borrow from.

 (2) In each repository, run "repack -a -l -d".  By doing this for a
     repository that borrows from another repacks the former without
     objects it borrows from the latter, so the former repository will
     retain what it needs but will borrow what are common from the
     latter, so you start from the repository that does not borrow from
     anybody else, and move on to the repositories that borrow from the
     one you already run this step.

Re: Removing redundant packs

[Jeff King]

On Sat, Dec 08, 2007 at 01:37:44PM -0800, Junio C Hamano wrote:

>  (1) In each repository, run "repack -a -d".  That would ensure that
>      everybody has the necessary objects that they themselves need.  By
>      doing this for a repository that borrows from another makes sure
>      pruning the latter would not break the former, so you start from
>      leaves and move on to the repositories they borrow from.

I'm not sure this works. Try this:

# make parent repo with two commits, fully packed
mkdir parent && cd parent && git init &&
echo content >file && git add file && git commit -m added &&
echo more >>file && git commit -m more -a &&
git repack -a -d

# clone child repo with alternates
git clone -s . ../child

# and now forget about the second commit in the parent
git reset --hard HEAD^
rm -rf .git/logs

# at this point the "parent" repo doesn't know anything about the second
# commit, but contains the objects; the child repo does know about it,
# but relies on the pack in the parent for the objects

# so your advice is to "repack -a -d" first in the child, then in the
# parent.
# child first; output will be: "Nothing new to pack."
cd ../child && git repack -a -d
# and then parent, which will lose the objects for the second commit
cd ../parent && git repack -a -d

# and confirm that child is broken
# output is: "fatal: bad object HEAD"
cd ../child && PAGER=cat git show

You seem to think that repacking without "-l" will pull objects from
alternates, but it doesn't (I have also seen "repack -a -d" recommended
for "how do I break alternates dependencies", but it obviously also
doesn't work for that).

-Peff

Re: Removing redundant packs

[Jeff King]

On Sat, Dec 08, 2007 at 06:44:15PM -0500, Jeff King wrote:

> You seem to think that repacking without "-l" will pull objects from
> alternates, but it doesn't (I have also seen "repack -a -d" recommended
> for "how do I break alternates dependencies", but it obviously also
> doesn't work for that).

And by "you seem to think" I obviously don't mean "stupid n00b, you
don't understand git" but "I'm unclear on what the behavior should be.
You seem to be saying this, which means this is a serious bug."

-Peff