Re: git-fast-import yields huge packfile

["Ævar Arnfjörð Bjarmason" <avarab@gmail.com>]

On Thu, Mar 21 2019, Johannes Schindelin wrote:

> Hi Richard,
>
> On Sat, 16 Mar 2019, Richard Hipp wrote:
>
>> I'm trying to transform a repository from another VCS into a Git
>> repository using "git fast-import".  It appears to work, but the
>> resulting Git repository is huge relative to the original - 18 times
>> larger. Most of the space seems to be taken up by a single large
>> packfile.  That packfile is about 967 MB which is about 1/4th the
>> total uncompressed size of all 41785 distinct Blobs in the original
>> repository.  The source VCS is able to compress this down to 52 MB by
>> comparison.
>
> I feel your pain, as I had the same problem back in the day. My use case
> was mirroring an upstream Mercurial repository to a Git repository. This
> use case went away, so I do not do that anymore (and there are more, less
> happy reasons why I would no longer work on that git-remote-hg project,
> but that's off topic). As one of the last rem(a)inders, Git for Windows
> carries this patch:
>
> https://github.com/git-for-windows/git/commit/b91911ff8d3e2cf279b4708be89de2e3bc8e9e87
>
> Essentially, it *always* runs `git gc --auto` after running `fast-import`.
>
> Which is a lot more high-level advice than the rather low-level `git
> repack` hint given elsewhere in this thread.
>
> Now, I wonder whether we should integrate this into `fast-import` proper
> (with a knob to turn it off), maybe even offer to run `git gc --auto`
> every <N> imported commits?

My reading of the combination of Linus's & Mike Hommey's E-Mails is that
this just happened to work for you because the blob import order you
used was such that you didn't get any on-the-fly deltas.

But as Linus notes you need to pass "-f" aka. "--no-reuse-delta" down to
pack-objects for this to work in the general case, so a plain "git gc"
in that GFW patch won't do the right thing *unless* you didn't end up
with any deltas at all (or close enough for it not to matter).

So in the general case you need to run "git gc --aggressive" after a
"fast-import". I'll add some docs about this in my re-roll of my
concurrent gc doc series:
https://public-inbox.org/git/20190318161502.7979-1-avarab@gmail.com/

I wonder if we should just leave it at that. The fast-import command is
plumbing, and e.g. someone running N number of those now and doing a
"git gc --aggressive" afterwards would have their use broken by this,
their "gc" would abort if the "--aggressive" we spawned after the 1st
fast-import invocation was still running.

I was thinking of introducing some sub-mode for --aggressive that
doesn't tweak the window size, but just passes down "-f". It would more
generally cover these cases, and eta less CPU than the increased window
size (although "--no-reuse-delta" by itself is very expensive).


>> Maybe I'm doing something wrong with the fast-import stream that is
>> defeating Git's attempts at delta compression....
>>
>> Are there any utility programs available for analyzing packfiles so
>> that I try to figure out where the inefficiencies are cropping up, so
>> that I can try to address them?
>>
>> Anybody have any suggestions on what I should be looking for?
>>
>> If anyone would care to see this oversized packfile and perhaps offer
>> suggestions on how I can make it more space-efficient, it can be
>> cloned from https://github.com/drhsqlite/fossil-mirror.git - at least
>> for now - surely I will delete that repo and regenerate it once I
>> figure out this problem.
>>
>> --
>> D. Richard Hipp
>> drh@sqlite.org
>>