Incremental 'git fetch' downloaded everything again

Incremental 'git fetch' downloaded everything again

[SZEDER Gábor]

I usually try to fetch from https://github.com/git/git daily, and
today morning something unusual happened:

  $ git fetch
  remote: Enumerating objects: 406099, done.
  remote: Counting objects: 100% (1229/1229), done.
  remote: Compressing objects: 100% (1044/1044), done.
  remote: Total 406093 (delta 207), reused 1189 (delta 185), pack-reused 404864 (from 2)
  Receiving objects: 100% (406093/406093), 292.22 MiB | 4.62 MiB/s, done.
  Resolving deltas: 100% (308943/308943), done.
  From https://github.com/git/git
   + ffe2b816f5...106a830b98 jch        -> origin/jch  (forced update)
     e9019fcafe..f85a7e6620  master     -> origin/master
   + c42f45431d...00534a21ce next       -> origin/next  (forced update)
   + f6884212b2...73452939f9 seen       -> origin/seen  (forced update)

Note that it downloaded over 400k objects in an almost 300MB packfile.

Looking at the objects that I already had and the objects in the newly
downloaded packfile:

  $ git rev-list --objects origin/master@{1} origin/next@{1} origin/jch@{1} origin/seen@{1} | cut -d' ' -f1 | sort >existing-objects
  $ git verify-pack -v .git/objects/pack/pack-080fedc9c19f711dd1b22103b382ede8925b90a6.idx | sed -n -E -e 's/^([0-9a-f]{40}) .*/\1/p' | sort >received-objects
  $ wc -l existing-objects received-objects 
    406954 existing-objects
    406093 received-objects
    813047 total
  $ git diff --no-index --stat existing-objects received-objects
   existing-objects => received-objects | 3615 +++++++++++++---------------------
   1 file changed, 1377 insertions(+), 2238 deletions(-)

The vast majority of objects were already available locally.

What's going on?!

This might be a recurring issue: I remember a similar large download
from 2 or 3 weeks ago, but back then I didn't have time to investigate
or to report.
I tried to reproduce this issue by attempting to recreate the state of
my git repository from yesterday in a new repo, but no luck, 'git
fetch' only downloads what's necessary.

I use a Git version based on next, with a bunch of my own patches on
top, but none of them has anything to do with object transfer and I've
been using most of them for years.  I don't have any config set under
'fetch.*' or 'transfer.*'.

Re: Incremental 'git fetch' downloaded everything again

[Junio C Hamano]

SZEDER Gábor <szeder.dev@gmail.com> writes:

> I usually try to fetch from https://github.com/git/git daily, and
> today morning something unusual happened:
>
>   $ git fetch
>   remote: Enumerating objects: 406099, done.
>   remote: Counting objects: 100% (1229/1229), done.
>   remote: Compressing objects: 100% (1044/1044), done.
>   remote: Total 406093 (delta 207), reused 1189 (delta 185), pack-reused 404864 (from 2)
>   Receiving objects: 100% (406093/406093), 292.22 MiB | 4.62 MiB/s, done.
>   Resolving deltas: 100% (308943/308943), done.
>   From https://github.com/git/git
>    + ffe2b816f5...106a830b98 jch        -> origin/jch  (forced update)
>      e9019fcafe..f85a7e6620  master     -> origin/master
>    + c42f45431d...00534a21ce next       -> origin/next  (forced update)
>    + f6884212b2...73452939f9 seen       -> origin/seen  (forced update)

The only difference I can think of is that 'next' has been rewound
recently, but because 'seen' and 'jch' are constantly reound, it
would be very strange that it made such a big difference.

So, sorry, no idea.  If I were bug-hunting this, I would first try
to eliminate whatever GitHub runs on their server end from the
picture.

> Note that it downloaded over 400k objects in an almost 300MB packfile.
>
> Looking at the objects that I already had and the objects in the newly
> downloaded packfile:
>
>   $ git rev-list --objects origin/master@{1} origin/next@{1} origin/jch@{1} origin/seen@{1} | cut -d' ' -f1 | sort >existing-objects
>   $ git verify-pack -v .git/objects/pack/pack-080fedc9c19f711dd1b22103b382ede8925b90a6.idx | sed -n -E -e 's/^([0-9a-f]{40}) .*/\1/p' | sort >received-objects
>   $ wc -l existing-objects received-objects 
>     406954 existing-objects
>     406093 received-objects
>     813047 total
>   $ git diff --no-index --stat existing-objects received-objects
>    existing-objects => received-objects | 3615 +++++++++++++---------------------
>    1 file changed, 1377 insertions(+), 2238 deletions(-)
>
> The vast majority of objects were already available locally.
>
> What's going on?!
>
> This might be a recurring issue: I remember a similar large download
> from 2 or 3 weeks ago, but back then I didn't have time to investigate
> or to report.
> I tried to reproduce this issue by attempting to recreate the state of
> my git repository from yesterday in a new repo, but no luck, 'git
> fetch' only downloads what's necessary.
>
> I use a Git version based on next, with a bunch of my own patches on
> top, but none of them has anything to do with object transfer and I've
> been using most of them for years.  I don't have any config set under
> 'fetch.*' or 'transfer.*'.