git-repack made my pack 317x larger...

git-repack made my pack 317x larger...

[linux]

I was grabbing a copy of the hwmon-2.6 git tree to play with:

> ~/tmp$ git-clone git://lm-sensors.org/kernel/mhoffman/hwmon-2.6.git
> Initialized empty Git repository in /home/linux/tmp/hwmon-2.6/.git/
> remote: Generating pack...
> remote: Done counting 496311 objects.
> remote: Deltifying 496311 objects.
> remote:  100% (496311/496311) done
> Indexing 496311 objects...
> remote: Total 496311 (delta 400999), reused 496151 (delta 400839)
>  100% (496311/496311) done
> Resolving 400999 deltas...
>  100% (400999/400999) done
> Checking 22409 files out...
>  100% (22409/22409) done
> ~/tmp$ ls -l hwmon-2.6/git/objects/pack
> -r--r--r-- 1 linux users  11912528 Jun 15 03:35 pack-d4ec0478f7530345ff466805ab4fe63efc9251df.idx
> -r--r--r-- 1 linux users 166721991 Jun 15 03:35 pack-d4ec0478f7530345ff466805ab4fe63efc9251df.pack

Then I noticed that it's a Linux kernel... oops!
Sorry for wasting the network bandwidth, but I can save myself
the disk space.

> ~/tmp$ rm -r hwmon-2.6/*
> ~/tmp$ mv hwmon-2.6 hwmon-2.6.old
> ~/tmp$ git clone --reference /usr/src/linux -n hwmon-2.6.old hwmon-2.6
> Initialized empty Git repository in /home/linux/tmp/hwmon-2.6/.git/
> remote: Generating pack...
> remote: Done counting 124 objects.
> remote: Deltifying 124 objects...
> remote:  100% (124/124) done
> Indexing 124 objects...
> remote: Total 124 (delta 83), reused 97 (delta 75)
>  100% (124/124) done
> Resolving 83 deltas...
>  100% (83/83) done
> ~/tmp$ rm -rf hwmon-2.6.olf
> ~/tmp$ cd hwmon-2.6
> ~/tmp/hwmon-2.6$ ls -l .git/objects/pack/
> total 168
> -r--r--r-- 1 linux users   4040 Jun 15 03:44 pack-e4de475aa6c82099b4a4a8f6f410dcc316c7cf19.idx
> -r--r--r-- 1 linux users 161548 Jun 15 03:44 pack-e4de475aa6c82099b4a4a8f6f410dcc316c7cf19.pack

Much better!  But since that's only a few objects, let's repack them even tighter...

> ~/tmp/hwmon-2.6$ git-repack -a -d -f --window=50
> Generating pack...
> Done counting 18090 objects.
> Deltifying 18090 objects...
>  100% (18090/18090) done
> Writing 18090 objects...
>  100% (18090/18090) done
> Total 18090 (delta 2465), reused 7876 (delta 0)
> Pack pack-1af4ab6620f71733f17b16d69a89b6741e2c7fe3 created.
> Removing unused objects 100%...
> Done.
> ~/tmp/hwmon-2.6$ ls -l .git/objects/pack/
> total 50620
> -r--r--r-- 1 linux users   435224 Jun 15 04:00 pack-1af4ab6620f71733f17b16d69a89b6741e2c7fe3.idx
> -r--r--r-- 1 linux users 51333891 Jun 15 04:00 pack-1af4ab6620f71733f17b16d69a89b6741e2c7fe3.pack

Uh... what happened?  It's not a full kernel clone, but it's a lot more
objects than I expected.  Where did all the extra objects come from?

Re: git-repack made my pack 317x larger...

[Nicolas Pitre]

On Fri, 15 Jun 2007, linux@horizon.com wrote:

> > ~/tmp$ git clone --reference /usr/src/linux -n hwmon-2.6.old hwmon-2.6
> > Initialized empty Git repository in /home/linux/tmp/hwmon-2.6/.git/
> > remote: Generating pack...
> > remote: Done counting 124 objects.
> > remote: Deltifying 124 objects...
> > remote:  100% (124/124) done
> > Indexing 124 objects...
> > remote: Total 124 (delta 83), reused 97 (delta 75)
> >  100% (124/124) done
> > Resolving 83 deltas...
> >  100% (83/83) done
> > ~/tmp$ rm -rf hwmon-2.6.olf
> > ~/tmp$ cd hwmon-2.6
> > ~/tmp/hwmon-2.6$ ls -l .git/objects/pack/
> > total 168
> > -r--r--r-- 1 linux users   4040 Jun 15 03:44 pack-e4de475aa6c82099b4a4a8f6f410dcc316c7cf19.idx
> > -r--r--r-- 1 linux users 161548 Jun 15 03:44 pack-e4de475aa6c82099b4a4a8f6f410dcc316c7cf19.pack
> 
> Much better!  But since that's only a few objects, let's repack them even tighter...
> 
> > ~/tmp/hwmon-2.6$ git-repack -a -d -f --window=50
> > Generating pack...
> > Done counting 18090 objects.
> > Deltifying 18090 objects...
> >  100% (18090/18090) done
> > Writing 18090 objects...
> >  100% (18090/18090) done
> > Total 18090 (delta 2465), reused 7876 (delta 0)
> > Pack pack-1af4ab6620f71733f17b16d69a89b6741e2c7fe3 created.
> > Removing unused objects 100%...
> > Done.
> > ~/tmp/hwmon-2.6$ ls -l .git/objects/pack/
> > total 50620
> > -r--r--r-- 1 linux users   435224 Jun 15 04:00 pack-1af4ab6620f71733f17b16d69a89b6741e2c7fe3.idx
> > -r--r--r-- 1 linux users 51333891 Jun 15 04:00 pack-1af4ab6620f71733f17b16d69a89b6741e2c7fe3.pack
> 
> Uh... what happened?  It's not a full kernel clone, but it's a lot more
> objects than I expected.  Where did all the extra objects come from?

Maybe you want to add -l as well to your git-repack invocation.


Nicolas

Re: git-repack made my pack 317x larger...

[linux]

>> Uh... what happened?  It's not a full kernel clone, but it's a lot more
>> objects than I expected.  Where did all the extra objects come from?

> Maybe you want to add -l as well to your git-repack invocation.

Ah.  Thank you.  Indeed, this is another example of git documentation
disease.  git-repack refers to git-pack-objects, which gives a very
technical explanation of what it does, but nowhere is it mentioned that
list of objects suppled to git-pack-object's stdin includes objects
borrowed from alternates.

Of course, reading the description of git-rev-list --objects, you might
get the impression, but it's not exactly hugely obvious.

Given that "git-repack -f" is a not uncommon command, could I suggest
that the default is wrong, and there should be a special flag for
"suck in alternates, so this repository is no longer dependent
on any others".

Mentally, git-repack is a "reduce space consumption" command, not an
increase one.  Having to remember that this repository uses alternates
and add an extra flag to avoid having a space explosion is distinctly
annoying.

(I might complan a little less if git-repack would take -adlf rather
than insisting on -a -d -l -f.  Is that a deliberate choice or has just
nobody stepped up to revamp the option parsing?)

Re: git-repack made my pack 317x larger...

[Benjamin Sergeant]

> (I might complan a little less if git-repack would take -adlf rather
> than insisting on -a -d -l -f.  Is that a deliberate choice or has just
> nobody stepped up to revamp the option parsing?)

getopt is our friend, indeed.

(http://www.hmug.org/man/1/getopt.php)

The following code fragment shows how one might process the arguments for
     a command that can take the options -a and -b, and the option -o, which
     requires an argument.

           args=`getopt abo: $*`
           # you should not use `getopt abo: "$@"` since that would parse
           # the arguments differently from what the set command below does.
           if [ $? != 0 ]
           then
                   echo 'Usage: ...'
                   exit 2
           fi
           set -- $args
           # You cannot use the set command with a backquoted getopt directly,
           # since the exit code from getopt would be shadowed by those of set,
           # which is zero by definition.
           for i
           do
                   case "$i"
                   in
                           -a|-b)
                                   echo flag $i set; sflags="${i#-}$sflags";
                                   shift;;
                           -o)
                                   echo oarg is "'"$2"'"; oarg="$2"; shift;
                                   shift;;
                           --)
                                   shift; break;;
                   esac
           done
           echo single-char flags: "'"$sflags"'"
           echo oarg is "'"$oarg"'"

     This code will accept any of the following as equivalent:

           cmd -aoarg file file
           cmd -a -o arg file file
           cmd -oarg -a file file
           cmd -a -oarg -- file file

Re: git-repack made my pack 317x larger...

[Nicolas Pitre]

On Fri, 15 Jun 2007, linux@horizon.com wrote:

> >> Uh... what happened?  It's not a full kernel clone, but it's a lot more
> >> objects than I expected.  Where did all the extra objects come from?
> 
> > Maybe you want to add -l as well to your git-repack invocation.
> 
> Ah.  Thank you.  Indeed, this is another example of git documentation
> disease.  git-repack refers to git-pack-objects, which gives a very
> technical explanation of what it does, but nowhere is it mentioned that
> list of objects suppled to git-pack-object's stdin includes objects
> borrowed from alternates.

At some point it is necessary for people like you who are not so 
intimate with the packing code, and therefore to whom this doesn't look 
obvious, to raise those issues, and ideally provide patches.

> Given that "git-repack -f" is a not uncommon command, could I suggest
> that the default is wrong, and there should be a special flag for
> "suck in alternates, so this repository is no longer dependent
> on any others".

Well, I tend to disagree here.  I don't think using -f _should_ be that 
common.  It is a really expensive operation and you usualy should have a 
good reason to use it.

> Mentally, git-repack is a "reduce space consumption" command, not an
> increase one.  Having to remember that this repository uses alternates
> and add an extra flag to avoid having a space explosion is distinctly
> annoying.

Why don't you use git-gc then?  Its mental model and actual 
implementation is really about reducing space, maybe even more than 
git-repack is, and it does call git-repack with -l.


Nicolas