speeding up git pull from a busy gerrit instance over a slow link?

speeding up git pull from a busy gerrit instance over a slow link?

[Noel Grandin]

Hi

I'm running git version 2.13.1 on Ubuntu 16.04 (x64)

I'm connecting over a very slow (international link) to a very busy gerrit server (gerrit.libreoffice.org) using ssh.
Ping types are on the order of 200ms.

Using GIT_TRACE_PACKET=true, what I am seeing is that the bulk of the time is spent retrieving packets having to do with 
things which I have no interest in, i.e. the refs/changes/* stuff (see below).

Is there any way to deliberately exclude these from the pull process?

My git config looks like:
    remote.origin.url=ssh://logerrit/core
    remote.origin.fetch=+refs/heads/*:refs/remotes/origin/*
    branch.master.remote=origin
    branch.master.merge=refs/heads/master
    branch.sdrshapeid.remote=origin
    branch.sdrshapeid.merge=refs/heads/master

Thanks, Noel Grandin

---------------------
snippet of packet trace
-------------------

14:20:45.705091 pkt-line.c:80           packet:        fetch< c5b026801c729ab37e2af6a610f31ca2e28b51fe 
refs/changes/99/29099/2
14:20:45.705093 pkt-line.c:80           packet:        fetch< 931e2c40aeb4cf4591ae9fcfea1b352b966f0a32 
refs/changes/99/29199/1
14:20:45.705096 pkt-line.c:80           packet:        fetch< 373b44a2fcbe78e8a3ff14cd410826af151a6adf 
refs/changes/99/29199/2
14:20:45.705099 pkt-line.c:80           packet:        fetch< 90e0db5edf83595121818627f590cc6b776f9f45 
refs/changes/99/29299/1
14:20:45.705102 pkt-line.c:80           packet:        fetch< 0f23647484e969fcd3c1b7e7ae72249026b20a80 
refs/changes/99/29299/2
14:20:45.705104 pkt-line.c:80           packet:        fetch< c6b6e54eeb0878a63179d2827d9c1623fe6f54e9 
refs/changes/99/29299/3
14:20:45.705107 pkt-line.c:80           packet:        fetch< 8e466c431e6d61911b2b15895a4b63df12422057 
refs/changes/99/29299/4
14:20:45.705110 pkt-line.c:80           packet:        fetch< 168bb711950f175c0fc4a889b9c8d41cf35bedec 
refs/changes/99/29399/1
14:20:45.705114 pkt-line.c:80           packet:        fetch< 6c76ade9516ada09222a18af69eb381b2e859b24 
refs/changes/99/29399/2
14:20:45.705116 pkt-line.c:80           packet:        fetch< ff88d1ba10b2bfc7af7cbb518d3a3c1122d6dffc 
refs/changes/99/29599/1
14:20:45.705119 pkt-line.c:80           packet:        fetch< 0d93900801224b797741e9a1abf305109fa35665 
refs/changes/99/29599/2
14:20:45.705122 pkt-line.c:80           packet:        fetch< 46f34d1b27f2056bb145cbdb526d4ed48b426f97 
refs/changes/99/29699/1
14:20:45.705124 pkt-line.c:80           packet:        fetch< 90d0a3d08c07d22b9a2b85ee08c72a0c047ed2d5 
refs/changes/99/29699/2
14:20:45.705127 pkt-line.c:80           packet:        fetch< 8821675d8e5a8b9c6f7fa5139a973394bfa67294 
refs/changes/99/29699/3

Disclaimer: http://www.peralex.com/disclaimer.html

Re: speeding up git pull from a busy gerrit instance over a slow link?

[Jonathan Tan]

On Fri, 30 Jun 2017 14:28:15 +0200
Noel Grandin <noel@peralex.com> wrote:

> ---------------------
> snippet of packet trace
> -------------------
> 
> 14:20:45.705091 pkt-line.c:80           packet:        fetch< c5b026801c729ab37e2af6a610f31ca2e28b51fe 
> refs/changes/99/29099/2
> 14:20:45.705093 pkt-line.c:80           packet:        fetch< 931e2c40aeb4cf4591ae9fcfea1b352b966f0a32 
> refs/changes/99/29199/1

Out of curiosity, what is the timestamp difference between the first and
last GIT_TRACE_PACKET log message containing "refs/changes"?

This is an issue for the Android repository too (which also uses
Gerrit). I have some work in progress to avoid the extra refs from being
sent [1], but haven't been working on it recently.

[1] https://public-inbox.org/git/cover.1491851452.git.jonathantanmy@google.com/

Re: speeding up git pull from a busy gerrit instance over a slow link?

[Stefan Beller]

On Fri, Jun 30, 2017 at 5:28 AM, Noel Grandin <noel@peralex.com> wrote:
> Hi
>
> I'm running git version 2.13.1 on Ubuntu 16.04 (x64)
>
> I'm connecting over a very slow (international link) to a very busy gerrit
> server (gerrit.libreoffice.org) using ssh.
> Ping types are on the order of 200ms.
>
> Using GIT_TRACE_PACKET=true, what I am seeing is that the bulk of the time
> is spent retrieving packets having to do with things which I have no
> interest in, i.e. the refs/changes/* stuff (see below).
>
> Is there any way to deliberately exclude these from the pull process?
>
> My git config looks like:
>    remote.origin.url=ssh://logerrit/core
>    remote.origin.fetch=+refs/heads/*:refs/remotes/origin/*
>    branch.master.remote=origin
>    branch.master.merge=refs/heads/master
>    branch.sdrshapeid.remote=origin
>    branch.sdrshapeid.merge=refs/heads/master
>
> Thanks, Noel Grandin

From here I figured, you're talking about gerrit.libreoffice.org/core
My initial clone is taking rather long, it was stuck in the
"Counting objects" phase for quite some time. Maybe ask the
Gerrit administrator if they have pack.useBitmaps enabled?

(This is a tangent, and most likely related to the initial clone only,
you are asking for everyday fetches.)

Re: speeding up git pull from a busy gerrit instance over a slow link?

[Noel Grandin]

On 2017/06/30 11:59 PM, Jonathan Tan wrote:
>
> Out of curiosity, what is the timestamp difference between the first and
> last GIT_TRACE_PACKET log message containing "refs/changes"?
>

Cut down log looks like:

08:37:17.734527 pkt-line.c:80           packet:        fetch< baeb5486c43d39b063371f91cfaae8efc2c8700b 
refs/changes/00/1000/1
... 3 minutes ..
08:40:10.983005 pkt-line.c:80           packet:        fetch< b0dd80238089dfcc0b3bf7bed99564adce649397 
refs/changes/99/6799/2
08:40:10.983054 pkt-line.c:80           packet:        fetch< 5c5dd57b54e3107b3069dc8f82df74df63d13555 
refs/heads/distro/collabora/cp-4.0
..
08:40:11.134355 pkt-line.c:80           packet:        fetch< f8c345808ceafe87be6207e5ae304a9fa6c4cd16 refs/heads/master
08:40:11.134404 pkt-line.c:80           packet:        fetch< 3cacadc5c9e6a0780a4c75cd3614eddc8db8e933 
refs/remotes/origin/HEAD
...
08:40:11.173013 pkt-line.c:80           packet:        fetch< 6c1b76c38f0fc469a2cbf41ffacde4d76df11ead 
refs/remotes/origin/origin/feature/submodules
08:40:11.173025 pkt-line.c:80           packet:        fetch< a7d2fe68fd1db70f16a827f828dc541954f9d0f2 refs/tags/COOL_1.0
...
08:40:11.814181 pkt-line.c:80           packet:        fetch< 9125509a7c6e65336330b8ac42a293aa77b18ee3 
refs/tags/windows_build_successful_2011_11_08^{}
...
08:40:12.724809 pkt-line.c:80           packet:     sideband< 0000


Disclaimer: http://www.peralex.com/disclaimer.html