issue upgrading git from 1.8.2.1 to 2.8.0

issue upgrading git from 1.8.2.1 to 2.8.0

[Gennaro Torre]

Hi,

Last week we upgraded git from 1.8.2.1 to 2.8.0. We saw some very
weird behavior where git clones and pushes were working, but git fetch
was not.

Here is our setup:

We have 10+ nodes caching a few repositories, the repos were
originally cloned via git 1.8.2.1 with `git clone --mirror <repo>`

We have 3000+ nodes that take deploys using the git protocol, via `git
fetch`, the existing repositories were also cloned via git 1.8.2.1
from the caching nodes.

When we upgraded to 2.8.0, all nodes received the upgrade.

We observed that when we tried to deploy code to the nodes (they run
`git fetch` to update the repository) this would fail.

The fix: we deleted the repositories originally cloned via git
1.8.2.1, and did a `git clone <repo>` dropping a fresh repository that
was cloned using git 2.8.0. Everything started working correctly. Our
running theory here is that there was some incompatibility with the
repositories cloned with the old version of git, and trying to run
`git fetch` with the newest version of git.

Thanks,
Gennaro

Re: issue upgrading git from 1.8.2.1 to 2.8.0

[Stefan Beller]

On Fri, Apr 22, 2016 at 3:44 PM, Gennaro Torre <gtorre@tumblr.com> wrote:
>
> Hi,
>
> Last week we upgraded git from 1.8.2.1 to 2.8.0. We saw some very
> weird behavior where git clones and pushes were working, but git fetch
> was not.
>
> Here is our setup:
>
> We have 10+ nodes caching a few repositories, the repos were
> originally cloned via git 1.8.2.1 with `git clone --mirror <repo>`
>
> We have 3000+ nodes that take deploys using the git protocol, via `git
> fetch`, the existing repositories were also cloned via git 1.8.2.1
> from the caching nodes.
>
> When we upgraded to 2.8.0, all nodes received the upgrade.
>
> We observed that when we tried to deploy code to the nodes (they run
> `git fetch` to update the repository) this would fail.
>
> The fix: we deleted the repositories originally cloned via git
> 1.8.2.1, and did a `git clone <repo>` dropping a fresh repository that
> was cloned using git 2.8.0. Everything started working correctly. Our
> running theory here is that there was some incompatibility with the
> repositories cloned with the old version of git, and trying to run
> `git fetch` with the newest version of git.

Do you still have an old repository?
You could compare a new and old repository with e.g. meld
to see if there are differences you don't expect.

Although 1.8..2.8 is a lot, Git ought to stay compatible there.

Thanks,
Stefan

>
>
> Thanks,
> Gennaro
> --
> To unsubscribe from this list: send the line "unsubscribe git" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: issue upgrading git from 1.8.2.1 to 2.8.0

[Junio C Hamano]

Stefan Beller <sbeller@google.com> writes:

> Although 1.8..2.8 is a lot, Git ought to stay compatible there.

In principle that is true, but there were deliberate backward
incompatible changes at around 2.0 boundary, and those who followed
along the upgrade paths were given warnings and advice messages to
guide them to adjust their configurations during the incremental
upgrade path.  It is not all that surprising if somebody who jumped
between the two versions in one hop didn't benefit from any such
guidance, though.

Offhand, perhaps .git/config left by ancient versions and recent
versions may differ in such a way that it affects the particular
workflow deployed there?

Re: issue upgrading git from 1.8.2.1 to 2.8.0

[Jeff King]

On Fri, Apr 22, 2016 at 06:44:02PM -0400, Gennaro Torre wrote:

> When we upgraded to 2.8.0, all nodes received the upgrade.
> 
> We observed that when we tried to deploy code to the nodes (they run
> `git fetch` to update the repository) this would fail.

Did `git fetch` say anything interesting on stderr?

-Peff

Re: issue upgrading git from 1.8.2.1 to 2.8.0

[Jeff King]

On Sat, Apr 23, 2016 at 01:01:06AM -0400, Gennaro Torre wrote:

> @Jeff King nope nothing interesting, it just hangs:
> 
> [gtorre@host1 A:DEVEL tumblr.old]$ git fetch -vv
> Looking up <cache> ... done.
> Connecting to <cache> (port 9418) ... <ip> done.
> Server supports multi_ack_detailed
> Server supports side-band-64k
> Server supports ofs-delta
> Server version is git/2.8.0

Hmm.  If you run with GIT_TRACE_PACKET=1, that may give an indication of
why it is hanging. Or perhaps tracing via gdb to see where we are when
it hangs.

-Peff

Re: issue upgrading git from 1.8.2.1 to 2.8.0

[Jeff King]

On Sat, Apr 23, 2016 at 01:18:50AM -0400, Gennaro Torre wrote:

> This is what I see:
> 
> 01:16:15.789969 pkt-line.c:80           packet:        fetch>
> git-upload-pack /tumblr.git\0host=<cache>\0
> 01:16:15.851231 pkt-line.c:80           packet:        fetch<
> a210029455476c3ae0d34748aeaeccf71864fd81 HEAD\0multi_ack thin-pack
> side-band side-band-64k ofs-delta shallow no-progress include-tag
> multi_ack_detailed symref=HEAD:refs/heads/master agent=git/2.8.0
> 01:16:15.851433 pkt-line.c:80           packet:        fetch<
> 511b1feab8d10b81777a75c02dabc94c7da9e0d4 refs/heads/263-fix
> .....
> 01:17:59.497259 pkt-line.c:80           packet:        fetch<
> cd63ca15d47b469e692faea2e04f32a241a6637a refs/tags/hotposts_v0.1
> 01:17:59.497265 pkt-line.c:80           packet:        fetch<
> 4749ee8a11ca093928c8a43fb0147d22177f1d71 refs/tags/hotposts_v0.1^{}
> 01:17:59.497282 pkt-line.c:80           packet:        fetch< 0000
> Server supports multi_ack_detailed
> Server supports side-band-64k
> Server supports ofs-delta
> Server version is git/2.8.0

So that's weird. The client gets the initial ref advertisement and
then...does nothing? It should drop into find_common() and start sending
want/have lines to the server (or decide it doesn't need any objects and
immediately send a flush packet).

Is the client chewing any CPU, or otherwise doing anything according to
"strace"? Is it possible to connect to it with gdb and get a backtrace?

-Peff