Unable to shrink repository size

Unable to shrink repository size

[Robert Dailey]

I have a git-svn clone that I've been working on which is a full and
complete conversion of our SVN repository at work.

It started out as 1.4GB (git count-objects -v, looking at
'size-pack'). I have run the following script to clean up a directory
in the repo history that I suspect are huge (we had a third party
library checked in that, uncompressed, was about 1.2GB in size):

-------------------------------
files=$@
echo "Removing: $files..."
git filter-branch --index-filter "git rm -rf --cached --ignore-unmatch
$files" -- --all

# remove the temporary history git-filter-branch otherwise leaves
behind for a long time
rm -rf .git/refs/original/ && git reflog expire --expire=now --all &&
git gc --aggressive --prune=now
-------------------------------

Even though I seem to have removed it, the repository size (looking at
'size-pack' again) only went down about 200MB, so it's at 1.2GB now.
There is about 3-5 years of commit history in this repository.

What I'd like to do is somehow hunt down the largest commit (*not*
blob) in the entire history of the repository to hopefully find out
where huge directories have been checked in.

I can't do a search for largest file (which most google results seem
to show to do) since the culprit is really thousands of unnecessary
files checked into a single subdirectory somewhere in history.

Can anyone offer me some advice to help me reduce the size of my repo
further? Thanks.

Re: Unable to shrink repository size

[Elijah Newren]

On Wed, Mar 5, 2014 at 6:55 PM, Robert Dailey <rcdailey.lists@gmail.com> wrote:
> What I'd like to do is somehow hunt down the largest commit (*not*
> blob) in the entire history of the repository to hopefully find out
> where huge directories have been checked in.
>
> I can't do a search for largest file (which most google results seem
> to show to do) since the culprit is really thousands of unnecessary
> files checked into a single subdirectory somewhere in history.
>
> Can anyone offer me some advice to help me reduce the size of my repo
> further? Thanks.

If you are trying to see which commits changed the most files (and/or
lines), you may find
   git log --oneline --shortstat
helpful.  A quick search through that output may help you decide where
to dig further.


Also, I'm sure someone else here probably has a better idea, but
here's a quick hack I came up with to look at "commit sizes":

mkdir tmp && for i in $(git rev-list --all); do git branch -f dummy $i
&& git bundle create tmp/$i.bundle dummy --not dummy^@; done

Follow that up with a quick "ls -alrSh tmp/" and you can see the
"size" of each commit.


Hope that helps,
Elijah

Re: Unable to shrink repository size

[Fredrik Gustafsson]

On Wed, Mar 05, 2014 at 08:55:30PM -0600, Robert Dailey wrote:
> What I'd like to do is somehow hunt down the largest commit (*not*
> blob) in the entire history of the repository to hopefully find out
> where huge directories have been checked in.
> 
> I can't do a search for largest file (which most google results seem
> to show to do) since the culprit is really thousands of unnecessary
> files checked into a single subdirectory somewhere in history.
> 
> Can anyone offer me some advice to help me reduce the size of my repo
> further? Thanks.

I'm not sure if you can do this with git. However since git is a command
line application it's pretty easy to script it with sh. The negative
part beeing the lack of speed, but since this is a one-time thing I
don't think that it matters.

Since you told us that it is a commit with a huge number of files that
you're looking for, I took that approach instead of calculating the size
of each commit, since that would be more expensive to do.

for commit in $(git log --pretty=oneline | cut -d " " -f 1)
do
	nbr=$(git show --pretty="format:" --name-only $commit | wc -l)
	echo "$nbr: $commit"
done | sort | tail -1

This will give you the commit with most files changed. (Although, there
will be a +1 error on the number of files edited).

-- 
Med vänlig hälsning
Fredrik Gustafsson

tel: 0733-608274
e-post: iveqy@iveqy.com

Re: Unable to shrink repository size

[Duy Nguyen]

On Thu, Mar 6, 2014 at 9:55 AM, Robert Dailey <rcdailey.lists@gmail.com> wrote:
> What I'd like to do is somehow hunt down the largest commit (*not*
> blob) in the entire history of the repository to hopefully find out
> where huge directories have been checked in.

Another try

git rev-list --all|while read i;do echo -n "$i "; git ls-tree -lr
$i|awk '{a+=$4;} END {print a;}';done

You may need to feed it to gnuplot or something to see steep slopes.
-- 
Duy

Re: Unable to shrink repository size

[Jeff King]

On Wed, Mar 05, 2014 at 08:55:30PM -0600, Robert Dailey wrote:

> What I'd like to do is somehow hunt down the largest commit (*not*
> blob) in the entire history of the repository to hopefully find out
> where huge directories have been checked in.
> 
> I can't do a search for largest file (which most google results seem
> to show to do) since the culprit is really thousands of unnecessary
> files checked into a single subdirectory somewhere in history.

Other people have offered scripts to look at commit sizes. But it might
also be useful to see sizes by subdirectory. Sort of a "du" across all
of history. Script is below.

Note that this script also uses cat-file's "%(objectsize:disk)". So it
is finding the actual on-disk storage, taking into account delta
storage. You will need git v1.8.5 or later to use this feature.

  git rev-list --objects --all |
  git cat-file --batch-check="%(objectsize:disk) %(rest)" |
  perl -lne '
    my ($size, $path) = split / /, $_, 2;
    next unless defined $path; # commit obj
    do {
      $sizes{$path} += $size;
    } while ($path =~ s{/[^/]+$}{});

    END { print "$sizes{$_} $_" for (keys %sizes) }
  ' |
  sort -rn

-Peff