How to delete large files

How to delete large files

[Nils Homer]

I wish to delete some large offending files (10MB to >100MB each) from my
git repository (git://bfast.git.sourceforge.net/gitroot/bfast/bfast).  The
current size of the repo is 656MB.

I created a backup of my repository and then searched for such offending
files using a script found here:
http://stubbisms.wordpress.com/2009/07/10/git-script-to-show-largest-pack-ob
jects-and-trim-your-waist-line/
where I modified the script to output in MB instead of KB.

This gave me a list of files that I wanted to delete:
-e   size  pack                                      SHA
location
113  113   f9f2faab597d4f8ccbfac2864347dbc256353fbf
tests.long/save/save.tar.gz
113  113   926b1ba880a26354c4a6b9391985f57fbc9a1174
tests.long/save/save.tar.gz
113  113   e568480bcb8239e6d1ed8d2da86c309c0d3d101b
tests.long/save/save.tar.gz
113  113   e3c0ee53f20e8ebfb60eaefcd7b405168c26a565
tests.long/save/save.tar.gz
103  103   ee2ee50c5075d05d29764c8d4b9acc2acedda919
tests.long/save/save.tar.gz
35   35    319c75945c27096093dbab5a0bf6a9a08089bc2d
tests.long/data/data.tar.gz
11   11    805193c74ceeffca9da3a2788545e701d77e1caf  tests/save/save.tar.gz
11   11    658e4a78c1028875ab597d6bde5823cd6a1694b9  tests/save/save.tar.gz

So I decided to remove these files using:
git filter-branch --index-filter "git rm -rf --cached --ignore-unmatch
tests.long/save/save.tar.gz tests.long/data/data.tar.gz
tests/save/save.tar.gz" HEAD

I then ran:
rm -rf .git/refs/original
git reflog expire --expire=now ‹all
git gc --prune=now

Still (using du ­h) the repository is 656MB and I can see the above files in
the revision list:
git rev-list --all --objects | grep tests.long/save/save.tar.gz
ee2ee50c5075d05d29764c8d4b9acc2acedda919 tests.long/save/save.tar.gz
e568480bcb8239e6d1ed8d2da86c309c0d3d101b tests.long/save/save.tar.gz
f9f2faab597d4f8ccbfac2864347dbc256353fbf tests.long/save/save.tar.gz
926b1ba880a26354c4a6b9391985f57fbc9a1174 tests.long/save/save.tar.gz
e3c0ee53f20e8ebfb60eaefcd7b405168c26a565 tests.long/save/save.tar.gz

Could this be because of tags that I had previously created?

I am running git version 1.6.3.3.  I appreciate any help in advance,

Nils Homer

Re: How to delete large files

[Matthieu Moy]

Nils Homer <nilshomer@gmail.com> writes:

> Still (using du ­h) the repository is 656MB and I can see the above files in
> the revision list:
> git rev-list --all --objects | grep tests.long/save/save.tar.gz
> ee2ee50c5075d05d29764c8d4b9acc2acedda919 tests.long/save/save.tar.gz
> e568480bcb8239e6d1ed8d2da86c309c0d3d101b tests.long/save/save.tar.gz
> f9f2faab597d4f8ccbfac2864347dbc256353fbf tests.long/save/save.tar.gz
> 926b1ba880a26354c4a6b9391985f57fbc9a1174 tests.long/save/save.tar.gz
> e3c0ee53f20e8ebfb60eaefcd7b405168c26a565 tests.long/save/save.tar.gz
>
> Could this be because of tags that I had previously created?

I guess so. Anything pointing to the old history will prevent Git from
deleting it.

-- 
Matthieu Moy
http://www-verimag.imag.fr/~moy/

Re: How to delete large files

[Mikael Magnusson]

2009/10/2 Nils Homer <nilshomer@gmail.com>:
> I wish to delete some large offending files (10MB to >100MB each) from my
> git repository (git://bfast.git.sourceforge.net/gitroot/bfast/bfast).  The
> current size of the repo is 656MB.
>
> I created a backup of my repository and then searched for such offending
> files using a script found here:
> http://stubbisms.wordpress.com/2009/07/10/git-script-to-show-largest-pack-ob
> jects-and-trim-your-waist-line/
> where I modified the script to output in MB instead of KB.
>
> This gave me a list of files that I wanted to delete:
> -e   size  pack                                      SHA
> location
> 113  113   f9f2faab597d4f8ccbfac2864347dbc256353fbf
> tests.long/save/save.tar.gz
> 113  113   926b1ba880a26354c4a6b9391985f57fbc9a1174
> tests.long/save/save.tar.gz
> 113  113   e568480bcb8239e6d1ed8d2da86c309c0d3d101b
> tests.long/save/save.tar.gz
> 113  113   e3c0ee53f20e8ebfb60eaefcd7b405168c26a565
> tests.long/save/save.tar.gz
> 103  103   ee2ee50c5075d05d29764c8d4b9acc2acedda919
> tests.long/save/save.tar.gz
> 35   35    319c75945c27096093dbab5a0bf6a9a08089bc2d
> tests.long/data/data.tar.gz
> 11   11    805193c74ceeffca9da3a2788545e701d77e1caf  tests/save/save.tar.gz
> 11   11    658e4a78c1028875ab597d6bde5823cd6a1694b9  tests/save/save.tar.gz
>
> So I decided to remove these files using:
> git filter-branch --index-filter "git rm -rf --cached --ignore-unmatch
> tests.long/save/save.tar.gz tests.long/data/data.tar.gz
> tests/save/save.tar.gz" HEAD
>
> I then ran:
> rm -rf .git/refs/original
> git reflog expire --expire=now 卟ll
> git gc --prune=now
>
> Still (using du 虐) the repository is 656MB and I can see the above files in
> the revision list:
> git rev-list --all --objects | grep tests.long/save/save.tar.gz
> ee2ee50c5075d05d29764c8d4b9acc2acedda919 tests.long/save/save.tar.gz
> e568480bcb8239e6d1ed8d2da86c309c0d3d101b tests.long/save/save.tar.gz
> f9f2faab597d4f8ccbfac2864347dbc256353fbf tests.long/save/save.tar.gz
> 926b1ba880a26354c4a6b9391985f57fbc9a1174 tests.long/save/save.tar.gz
> e3c0ee53f20e8ebfb60eaefcd7b405168c26a565 tests.long/save/save.tar.gz
>
> Could this be because of tags that I had previously created?
>
> I am running git version 1.6.3.3.  I appreciate any help in advance,

Well, you just gave "HEAD" to git filter-branch to rewrite, i think
you want --all to rewrite all refs you have.

-- 
Mikael Magnusson

Re: How to delete large files

[Johannes Sixt]

Mikael Magnusson schrieb:
> Well, you just gave "HEAD" to git filter-branch to rewrite, i think
> you want --all to rewrite all refs you have.

... and '--tag-filter cat' to rewrite tags as well.

-- Hannes

Re: How to delete large files

[Nils Homer]

Thank-you for all of your insightful help. Combining all the advice, the
commands that worked are:

git filter-branch --index-filter "git rm -rf --cached --ignore-unmatch
$files" --tag-name-filter cat -- --all

git -rf .git/refs/original

git reflog expire --expire=now

git gc --prune=now

I then cloned the repository to a different location and replaced my
centralized version with the cloned copy.

Thanks,

Nils


On 10/2/09 6:41 AM, "Johannes Sixt" <j.sixt@viscovery.net> wrote:

> Mikael Magnusson schrieb:
>> Well, you just gave "HEAD" to git filter-branch to rewrite, i think
>> you want --all to rewrite all refs you have.
> 
> ... and '--tag-filter cat' to rewrite tags as well.
> 
> -- Hannes