Re: [BUG/FEATURE] Git pushing and fetching many more objects than strictly required

[Paul van Loon <nospam@cheerful.com>]

On 2019-11-08 22:21, Jeff King wrote:
> On Fri, Nov 08, 2019 at 09:54:02PM +0100, Paul van Loon wrote:
>
>>>> $ git push -v origin 'refs/replace/*:refs/replace/*'
>>>> Pushing to XXXX
>>>> Enumerating objects: 2681, done.
>>>> Counting objects: 100% (2681/2681), done.
>>>> Delta compression using up to 8 threads
>>>> Compressing objects: 100% (1965/1965), done.
>>>> Writing objects: 100% (2582/2582), 1.96 MiB | 1024 bytes/s, done.
>>>> Total 2582 (delta 95), reused 1446 (delta 58)
>>>> remote: Resolving deltas: 100% (95/95), completed with 33 local objects.
>>>> To XXXX
>>>>  * [new branch]            refs/replace/XXXX -> refs/replace/XXXX
>>>
>>> Could you verify that refs/replace/XXXX (or one of its close ancestors)
>>> was fetched by the "git fetch --all" command? "--all" fetches all
>>> remotes, not all refs.
>>
>> No, it was not fetched. HOWEVER, the ONLY thing the replace commit (1 single object) does is point to an existing parent object. No other new objects are referenced.
>> Those 'ancestor' objects were all fetched.
>
> Was it a parent object at the tip of a ref?

No, it was a newly created replace object (created with git replace --edit)

>
> The push protocol, unlike the fetch protocol, doesn't expend any effort
> to negotiate to find a common base. It just feeds the ref tips of the
> receiver to pack-objects (which then does traverse down to a merge base,
> but it can't always do so if the sender doesn't have all of the
> objects).

So this would be the opportunity for performance improvement I guess.

>
> It's hard to say more without having a reproducible case to look at.
>
> Some possible things to poke at:
>
>   - record the stdin from the local push to the local pack-objects,
>     which shows which objects we're planning to send and which we're
>     claiming the other side has. That would help determine if the push
>     isn't feeding enough information to pack-objects, or if pack-objects
>     isn't trying hard enough to find the minimal set of objects
>
>     There's not really an easy way to do this, but something like strace
>     might help.

That's way above my Git expertise.

>   - try building reachability bitmaps (e.g., "git repack -adb") in the
>     local clone. When those are present, pack-objects will compute the
>     object set more thoroughly (because it can do so efficiently).
>
> I don't _think_ the fact that it's in refs/replace should matter to push
> (in terms of what it feeds to pack-objects). But obviously another thing
> to try is whether pushing to or from a different ref has any impact.

I'll do some additional experiments

> -Peff
>