git gc / git repack not removing unused objects?

git gc / git repack not removing unused objects?

[Jon Nelson]

[Using git 1.6.4.2]

In one repo I have (136G objects directory, fully packed) I'm having
some trouble.
I've run git-gc --prune=now, git repack -Adf, and so on a half-dozen
times and each time I do so it gets bigger, not smaller.
Setting that aside for the moment, however, I've run into a stranger problem.

So I use "git verify-pack -v > gvp.out" and "sort -k3nr < gvp.out |
head -n 20" to find the top 20 largest blobs.
So I have a blob, b32c3d8e8e24d8d3035cf52f606c2873315fe2b8, and now I
want to know what tree (or trees) it is in, so I try this:


for i in $( git branch -a | sed -e 's/\*//g' | grep -v branch ); do if
git ls-tree -l -r -t $i | grep
b32c3d8e8e24d8d3035cf52f606c2873315fe2b8 > /dev/null; then echo $i;
fi; done

The results: no branch or tree appears to contain that blob.

So I tried a different approach:

for i in $( grep tree gvp.out  | awk '{ print $1 }' ); do if git
ls-tree $i | grep b32c3d8e8e24d8d3035cf52f606c2873315fe2b8 >
/dev/null; then echo $i; fi ; done

This time, I find (at least) one tree
(d813af1537358496ca34958bbff08b87590607bf) with the blob.
But which branches might that tree appear in? None.

For each branch, I ran "git ls-tree -l -r -t" and saved the output in
a file (one per branch).
Then I grepped each file for the tree (
(d813af1537358496ca34958bbff08b87590607bf) - no luck.
I grepped each file for the blob (b32...) - no luck.

The results seem to suggest that I have packed trees which reference
blobs, but that the trees themselves are not referenced in any branch
and therefore I would expect that they would be pruned.


-- 
Jon

Re: git gc / git repack not removing unused objects?

[Nicolas Pitre]

On Fri, 5 Feb 2010, Jon Nelson wrote:

> [Using git 1.6.4.2]
> 
> In one repo I have (136G objects directory, fully packed) I'm having
> some trouble.
> I've run git-gc --prune=now, git repack -Adf, and so on a half-dozen
> times and each time I do so it gets bigger, not smaller.

Please tell us more.

> Setting that aside for the moment, however, I've run into a stranger problem.
> 
> So I use "git verify-pack -v > gvp.out" and "sort -k3nr < gvp.out |
> head -n 20" to find the top 20 largest blobs.
> So I have a blob, b32c3d8e8e24d8d3035cf52f606c2873315fe2b8, and now I
> want to know what tree (or trees) it is in, so I try this:
> 
> 
> for i in $( git branch -a | sed -e 's/\*//g' | grep -v branch ); do if
> git ls-tree -l -r -t $i | grep
> b32c3d8e8e24d8d3035cf52f606c2873315fe2b8 > /dev/null; then echo $i;
> fi; done
> 
> The results: no branch or tree appears to contain that blob.

What you did above is simply to list trees that are reachable from the 
_heads_ of your branches.  If the blob belongs to a commit which isn't 
the latest revision of any of your branches then you won't see it like 
that.

> So I tried a different approach:
> 
> for i in $( grep tree gvp.out  | awk '{ print $1 }' ); do if git
> ls-tree $i | grep b32c3d8e8e24d8d3035cf52f606c2873315fe2b8 >
> /dev/null; then echo $i; fi ; done
> 
> This time, I find (at least) one tree
> (d813af1537358496ca34958bbff08b87590607bf) with the blob.
> But which branches might that tree appear in? None.
> 
> For each branch, I ran "git ls-tree -l -r -t" and saved the output in
> a file (one per branch).
> Then I grepped each file for the tree (
> (d813af1537358496ca34958bbff08b87590607bf) - no luck.
> I grepped each file for the blob (b32...) - no luck.
> 
> The results seem to suggest that I have packed trees which reference
> blobs, but that the trees themselves are not referenced in any branch
> and therefore I would expect that they would be pruned.

NO.  If those trees and blobs are stil there then they do get 
referenced.  But not from the latest commit on any of your branches.  
You need to dig further down in history to find a commit that actually 
references that blob/tree.  One easy method is to do:

	git log --raw --all

and within the pager ('less' by default) simply search for "b32c3d8".


Nicolas

Re: git gc / git repack not removing unused objects?

[Jon Nelson]

On Fri, Feb 5, 2010 at 2:51 PM, Nicolas Pitre <nico@fluxnic.net> wrote:
> On Fri, 5 Feb 2010, Jon Nelson wrote:
>
>> [Using git 1.6.4.2]
>>
>> In one repo I have (136G objects directory, fully packed) I'm having
>> some trouble.
>> I've run git-gc --prune=now, git repack -Adf, and so on a half-dozen
>> times and each time I do so it gets bigger, not smaller.
>
> Please tell us more.

I'll tell you whatever I can -- as soon as I know what it is you want.

>> Setting that aside for the moment, however, I've run into a stranger problem.
>>
>> So I use "git verify-pack -v > gvp.out" and "sort -k3nr < gvp.out |
>> head -n 20" to find the top 20 largest blobs.
>> So I have a blob, b32c3d8e8e24d8d3035cf52f606c2873315fe2b8, and now I
>> want to know what tree (or trees) it is in, so I try this:
>>
>>
>> for i in $( git branch -a | sed -e 's/\*//g' | grep -v branch ); do if
>> git ls-tree -l -r -t $i | grep
>> b32c3d8e8e24d8d3035cf52f606c2873315fe2b8 > /dev/null; then echo $i;
>> fi; done
>>
>> The results: no branch or tree appears to contain that blob.
>
> What you did above is simply to list trees that are reachable from the
> _heads_ of your branches.  If the blob belongs to a commit which isn't
> the latest revision of any of your branches then you won't see it like
> that.
>
>> So I tried a different approach:
>>
>> for i in $( grep tree gvp.out  | awk '{ print $1 }' ); do if git
>> ls-tree $i | grep b32c3d8e8e24d8d3035cf52f606c2873315fe2b8 >
>> /dev/null; then echo $i; fi ; done
>>
>> This time, I find (at least) one tree
>> (d813af1537358496ca34958bbff08b87590607bf) with the blob.
>> But which branches might that tree appear in? None.
>>
>> For each branch, I ran "git ls-tree -l -r -t" and saved the output in
>> a file (one per branch).
>> Then I grepped each file for the tree (
>> (d813af1537358496ca34958bbff08b87590607bf) - no luck.
>> I grepped each file for the blob (b32...) - no luck.
>>
>> The results seem to suggest that I have packed trees which reference
>> blobs, but that the trees themselves are not referenced in any branch
>> and therefore I would expect that they would be pruned.
>
> NO.  If those trees and blobs are stil there then they do get
> referenced.  But not from the latest commit on any of your branches.
> You need to dig further down in history to find a commit that actually
> references that blob/tree.  One easy method is to do:
>
>        git log --raw --all
>
> and within the pager ('less' by default) simply search for "b32c3d8".

OK. I'm piping "git log --raw --all" to a file this very moment. It'll
take a while. However, one thing I did not mention is that there
*should* be a 1:1 correlation between branches and commits. As in,
every time I did a commit, the commit was on a new branch. I'll look
into this, as I've fiddled with the repo a bunch of different ways
lately. I suspect the answer will be found in the logs.

Thanks for the response!

-- 
Jon

Re: git gc / git repack not removing unused objects?

[Nicolas Pitre]

[-- Attachment #1: Type: TEXT/PLAIN, Size: 1822 bytes --]

On Fri, 5 Feb 2010, Jon Nelson wrote:

> On Fri, Feb 5, 2010 at 2:51 PM, Nicolas Pitre <nico@fluxnic.net> wrote:
> > On Fri, 5 Feb 2010, Jon Nelson wrote:
> >
> >> [Using git 1.6.4.2]
> >>
> >> In one repo I have (136G objects directory, fully packed) I'm having
> >> some trouble.
> >> I've run git-gc --prune=now, git repack -Adf, and so on a half-dozen
> >> times and each time I do so it gets bigger, not smaller.
> >
> > Please tell us more.
> 
> I'll tell you whatever I can -- as soon as I know what it is you want.

I want to know what you did and what results you obtained to lead you to 
believe that 'git-gc --prune=now' makes your repository bigger.

> > You need to dig further down in history to find a commit that actually
> > references that blob/tree.  One easy method is to do:
> >
> >        git log --raw --all
> >
> > and within the pager ('less' by default) simply search for "b32c3d8".
> 
> OK. I'm piping "git log --raw --all" to a file this very moment. It'll
> take a while. However, one thing I did not mention is that there
> *should* be a 1:1 correlation between branches and commits. As in,
> every time I did a commit, the commit was on a new branch. 

Ouch.  Why?  I'm curious.

> I'll look into this, as I've fiddled with the repo a bunch of 
> different ways lately. I suspect the answer will be found in the logs.

If the blob you're looking for is not in that log, then it must be 
referenced by one of the reflog entries.  The reflog is a log of all the 
commits your branches, and the HEAD pointer, ever pointed at.  Those 
entries are kept for a certain period (3 months by default).  The HEAD 
reflog will keep a reference of all branches you might have visited, 
even deleted ones.  So another log:

	git log --raw -g HEAD

should ultimately give you the blob reference.


Nicolas

Re: git gc / git repack not removing unused objects?

[Jon Nelson]

On Fri, Feb 5, 2010 at 3:45 PM, Nicolas Pitre <nico@fluxnic.net> wrote:
> On Fri, 5 Feb 2010, Jon Nelson wrote:
>
>> On Fri, Feb 5, 2010 at 2:51 PM, Nicolas Pitre <nico@fluxnic.net> wrote:
>> > On Fri, 5 Feb 2010, Jon Nelson wrote:
>> >
>> >> [Using git 1.6.4.2]
>> >>
>> >> In one repo I have (136G objects directory, fully packed) I'm having
>> >> some trouble.
>> >> I've run git-gc --prune=now, git repack -Adf, and so on a half-dozen
>> >> times and each time I do so it gets bigger, not smaller.
>> >
>> > Please tell us more.
>>
>> I'll tell you whatever I can -- as soon as I know what it is you want.
>
> I want to know what you did and what results you obtained to lead you to
> believe that 'git-gc --prune=now' makes your repository bigger.

Last night, the repo size was 153G after removing some commits and
objects by way of git filter-branch.
I'm using "du -sh" in the .git directory to determine the disk usage.

Before: 136G
git repack -dAl
After: 153G

Then, just to make sure of some things, I changed nothing and simply
re-ran "git repack -dAl".
After: 167G

Config:

[core]
        repositoryformatversion = 0
        filemode = true
        bare = true
        logallrefupdates = true
        compression = 0
        loosecompression = 0
        packedGitLimit = 1500M
        #packedGitWindowSize = 32m
[pack]
        packsizelimit = 256m
        windowMemory = 128m
        deltaCacheSize = 1m
[gc]
    threads = 1
    packrefs = true

[repack]
        usedeltabaseoffset = true

Incorporating the system and global config (git config -l):

pack.packsizelimit=2M
pack.threads=1
gc.auto=0
gc.autopacklimit=0
core.repositoryformatversion=0
core.filemode=true
core.bare=true
core.logallrefupdates=true
core.compression=0
core.loosecompression=0
core.packedgitlimit=1500M
pack.packsizelimit=256m
pack.windowmemory=128m
pack.deltacachesize=1m
gc.threads=1
gc.packrefs=true
repack.usedeltabaseoffset=true

-- 
Jon

Re: git gc / git repack not removing unused objects?

[Nicolas Pitre]

On Sat, 6 Feb 2010, Jon Nelson wrote:

> Last night, the repo size was 153G after removing some commits and
> objects by way of git filter-branch.
> I'm using "du -sh" in the .git directory to determine the disk usage.
> 
> Before: 136G
> git repack -dAl
> After: 153G

Why are you using -A instead of -a ?

> Then, just to make sure of some things, I changed nothing and simply
> re-ran "git repack -dAl".
> After: 167G

Could you run 'git count-objects -v' before and after a repack in such 
cases as well?

> [pack]
>         packsizelimit = 256m

Why are you using this?

> pack.packsizelimit=2M

This is even worse.  150G / 2M per pack = approx 75000 packs.

What you should be aiming for is a _single_ pack for best performances.  
A couple packs is common and doesn't make much of a difference.  A 
hundred packs is getting a bit large.  But 75000 packs is totally 
insane.


Nicolas

Re: git gc / git repack not removing unused objects?

[Jon Nelson]

On Sat, Feb 6, 2010 at 7:16 PM, Nicolas Pitre <nico@fluxnic.net> wrote:
> On Sat, 6 Feb 2010, Jon Nelson wrote:
>
>> Last night, the repo size was 153G after removing some commits and
>> objects by way of git filter-branch.
>> I'm using "du -sh" in the .git directory to determine the disk usage.
>>
>> Before: 136G
>> git repack -dAl
>> After: 153G
>
> Why are you using -A instead of -a ?

As it turns out, I've been using both -a and -A. I suspect -A is a
typo on my part.

>> Then, just to make sure of some things, I changed nothing and simply
>> re-ran "git repack -dAl".
>> After: 167G
>
> Could you run 'git count-objects -v' before and after a repack in such
> cases as well?

Yes.

>> [pack]
>>         packsizelimit = 256m
>
> Why are you using this?

I didn't want my pack files to be too huge. I've bumped that up to 2G.

>> pack.packsizelimit=2M

My ~/.gitconfig normally uses 2M for easy rsyncing. In this repo I
thought the value was overridden by the project's config (which was
specifying 256m and now specifies 2048m).

Suboptimal or not, it still doesn't explain why the repo grows with each repack.

Now running:

 git count-objects -v ; git repack -ad ; git count-objects -v


-- 
Jon

Re: git gc / git repack not removing unused objects?

[Jon Nelson]

On Sun, Feb 7, 2010 at 11:48 AM, Jon Nelson <jnelson@jamponi.net> wrote:
> On Sat, Feb 6, 2010 at 7:16 PM, Nicolas Pitre <nico@fluxnic.net> wrote:
>> On Sat, 6 Feb 2010, Jon Nelson wrote:
>>
>>> Last night, the repo size was 153G after removing some commits and
>>> objects by way of git filter-branch.
>>> I'm using "du -sh" in the .git directory to determine the disk usage.
>>>
>>> Before: 136G
>>> git repack -dAl
>>> After: 153G
>>
>> Why are you using -A instead of -a ?
>
> As it turns out, I've been using both -a and -A. I suspect -A is a
> typo on my part.
>
>>> Then, just to make sure of some things, I changed nothing and simply
>>> re-ran "git repack -dAl".
>>> After: 167G
>>
>> Could you run 'git count-objects -v' before and after a repack in such
>> cases as well?
>
> Yes.
>
>>> [pack]
>>>         packsizelimit = 256m
>>
>> Why are you using this?
>
> I didn't want my pack files to be too huge. I've bumped that up to 2G.
>
>>> pack.packsizelimit=2M
>
> My ~/.gitconfig normally uses 2M for easy rsyncing. In this repo I
> thought the value was overridden by the project's config (which was
> specifying 256m and now specifies 2048m).
>
> Suboptimal or not, it still doesn't explain why the repo grows with each repack.
>
> Now running:
>
>  git count-objects -v ; git repack -ad ; git count-objects -v

The results (two lines of whitespace added for clarity):

turnip:/mnt/backups.git # git count-objects -v ; git repack -ad ; git
count-objects -v
count: 0
size: 0
in-pack: 2408195
packs: 675
size-pack: 174924053
prune-packable: 0
garbage: 0

Counting objects: 266309, done.
Compressing objects: 100% (207820/207820), done.
Writing objects: 100% (266309/266309), done.
Total 266309 (delta 47751), reused 227730 (delta 45117)

count: 0
size: 0
in-pack: 2674504
packs: 686
size-pack: 196517094
prune-packable: 0
garbage: 0
turnip:/mnt/backups.git # du -sh .
188G    .
turnip:/mnt/backups.git # find objects -name '*.pack' | wc -l
686
turnip:/mnt/backups.git #

So we went from 136G to 153G to 167G and now 188G.

The oldest .pack file is Jan 29 at 7.0M
The pack files vary considerably in size from 220B to 2.0G
There are 433 packs that are between 200M and 300M in size.
There are 78 packs that are between 500M and 600M in size.
There are 211 packs that are less than 256M in size.

I ran "git prune -v --prune=now" and it claims to have removed nothing
and the exit status was 0.

I hope this helps!

-- 
Jon

Re: git gc / git repack not removing unused objects?

[Nicolas Pitre]

[-- Attachment #1: Type: TEXT/PLAIN, Size: 2334 bytes --]

On Sun, 7 Feb 2010, Jon Nelson wrote:
> On Sat, Feb 6, 2010 at 7:16 PM, Nicolas Pitre <nico@fluxnic.net> wrote:
> >On Sat, 6 Feb 2010, Jon Nelson wrote:
> >>         packsizelimit = 256m
> >
> > Why are you using this?
> 
> I didn't want my pack files to be too huge. I've bumped that up to 2G.
> 
> >> pack.packsizelimit=2M
> 
> My ~/.gitconfig normally uses 2M for easy rsyncing. In this repo I
> thought the value was overridden by the project's config (which was
> specifying 256m and now specifies 2048m).

Why do you synchronize with rsync?  Why not a simple 'git fetch' which 
is likely to be much faster in any case?

> Suboptimal or not, it still doesn't explain why the repo grows with each repack.

That's strange indeed.  Except that ...

> turnip:/mnt/backups.git # git count-objects -v ; git repack -ad ; git
> count-objects -v
> count: 0
> size: 0
> in-pack: 2408195
> packs: 675
> size-pack: 174924053
> prune-packable: 0
> garbage: 0
> 
> Counting objects: 266309, done.
> Compressing objects: 100% (207820/207820), done.
> Writing objects: 100% (266309/266309), done.
> Total 266309 (delta 47751), reused 227730 (delta 45117)
> 
> count: 0
> size: 0
> in-pack: 2674504
> packs: 686
> size-pack: 196517094
> prune-packable: 0
> garbage: 0

You previously had 2408195 packed objects and after the repack this is 
2674504.  The difference is 266309, which incidentally is the number of 
objects 'git repack' counted.

If that 266309 object count is stable between repack attempts (and it 
should), then something is failing to clean up the old packs.  If I 
remember right, you were playing with modifications to git-repack.sh 
lately?  It is well possible that you broke it.

I tried to simulate your setup but I'm regretting it now.  My test repo 
expanded to 10x its original size with 900 packs.  Git is also painfully 
slow now while I'm trying to repack it properly back into a single pack.  
However I wasn't able to reproduce your repository growth.  Please 
retest with a vanilla Git version.

I think you should really consider using an unlimited pack size.  Not 
only you'll save quite a lot of disk space, but Git will be much faster 
too.  And Git itself is far more efficient than rsync at keeping Git 
repositories in sync, regardless of pack sizes (unless you have too many 
packs of course).


Nicolas

Re: git gc / git repack not removing unused objects?

[Jon Nelson]

On Sun, Feb 7, 2010 at 8:11 PM, Nicolas Pitre <nico@fluxnic.net> wrote:
> On Sun, 7 Feb 2010, Jon Nelson wrote:
>> On Sat, Feb 6, 2010 at 7:16 PM, Nicolas Pitre <nico@fluxnic.net> wrote:
>> >On Sat, 6 Feb 2010, Jon Nelson wrote:
>> >>         packsizelimit = 256m
>> >
>> > Why are you using this?
>>
>> I didn't want my pack files to be too huge. I've bumped that up to 2G.
>>
>> >> pack.packsizelimit=2M
>>
>> My ~/.gitconfig normally uses 2M for easy rsyncing. In this repo I
>> thought the value was overridden by the project's config (which was
>> specifying 256m and now specifies 2048m).
>
> Why do you synchronize with rsync?  Why not a simple 'git fetch' which
> is likely to be much faster in any case?

Often I need to sync the entire config, and am frequently rsyncing
(although not in this case) rsyncing repos made with git-svn. I've had
trouble using git clone on these repos, as I really want the duplicate
repo to use the svn upstream and not the git repo it was cloned from.

..

> You previously had 2408195 packed objects and after the repack this is
> 2674504.  The difference is 266309, which incidentally is the number of
> objects 'git repack' counted.
>
> If that 266309 object count is stable between repack attempts (and it
> should), then something is failing to clean up the old packs.  If I
> remember right, you were playing with modifications to git-repack.sh
> lately?  It is well possible that you broke it.

Indeedly do I did.

The (now expected) results:

turnip:/mnt/backups.git # du -sh ; echo ; git count-objects -v ; echo
; git repack -ad ; echo ; git count-objects -v ; echo ; du -sh
204G    .

count: 0
size: 0
in-pack: 2939744
packs: 694
size-pack: 213284091
prune-packable: 0
garbage: 0

Counting objects: 266309, done.
Compressing objects: 100% (205176/205176), done.
Writing objects: 100% (266309/266309), done.
Total 266309 (delta 47781), reused 230413 (delta 47770)

count: 0
size: 0
in-pack: 266309
packs: 11
size-pack: 21590451
prune-packable: 0
garbage: 0

21G     .
turnip:/mnt/backups.git #

and of course now everything is nice and (reasonably) fast.
Thanks for the help, and pointing out that the problem was .. me.

-- 
Jon