Debugging strange "corrupt pack" errors on SuSE 9

Debugging strange "corrupt pack" errors on SuSE 9

[Martin Langhoff]

A colleage at work is using git to manage code updates to a heavily
firewalled machine at a client site. In a nutshell, we "push" the
interesting code to a repo on usb-stick, and we pull from it on the
client's machine.

We do some coding and testing on that machine, so ocassionally we
bring some patches back.

Now the working repo on the client machine has started to die with
"corrupt pack" errors. I am trying to get my hands on the literal
error messages, and exact software versions installed. Right now all I
know is that it is SuSE 9 x86, git 1.4.x, cogito .17.x . The error
appears on git-diff, git-fsck-objects --full

We did bring a copy of the working copy with the "corrupt pack" to our
office, and here git (v1.5.1 and 1.5.2) thinks it's perfectly well.

So I am a bit puzzled - while we try to get 1.5.x on the client
machine and see what happens, is there anything that could be causing
this? Any additional tests that we should run?

cheers,


martin

Re: Debugging strange "corrupt pack" errors on SuSE 9

[Nicolas Pitre]

On Wed, 20 Jun 2007, Martin Langhoff wrote:

> A colleage at work is using git to manage code updates to a heavily
> firewalled machine at a client site. In a nutshell, we "push" the
> interesting code to a repo on usb-stick, and we pull from it on the
> client's machine.
> 
> We do some coding and testing on that machine, so ocassionally we
> bring some patches back.
> 
> Now the working repo on the client machine has started to die with
> "corrupt pack" errors. I am trying to get my hands on the literal
> error messages, and exact software versions installed. Right now all I
> know is that it is SuSE 9 x86, git 1.4.x, cogito .17.x . The error
> appears on git-diff, git-fsck-objects --full

The full exact error message would be highly useful indeed.

> We did bring a copy of the working copy with the "corrupt pack" to our
> office, and here git (v1.5.1 and 1.5.2) thinks it's perfectly well.
> 
> So I am a bit puzzled - while we try to get 1.5.x on the client
> machine and see what happens, is there anything that could be causing
> this? Any additional tests that we should run?

Maybe the client machine runs git version < 1.4.2.2, in which case it is 
possible that your push created a pack containing delta objects with 
offset to base which git versions prior 1.4.2.2 do not understand.

If this is the problem you are facing (the error message should confirm 
this) then the easiest solution is to upgrade git on the client.

A quick fix for the client is to set repack.usedeltabaseoffset to 
false on the machine where you have git 1.5 installed, then run "git 
repack -a -d", and finally copy the pack over to the client repository.  
But because you push to a local repository (a mounted USB stick is 
considered a local repo) then you don't get to negociate the pack 
capabilities of the final destination, and therefore more "bad" delta 
objects might sneak in again.


Nicolas

Re: Debugging strange "corrupt pack" errors on SuSE 9

[Martin Langhoff]

On 6/20/07, Nicolas Pitre <nico@cam.org> wrote:
> The full exact error message would be highly useful indeed.

Yes. I haven't seen it first hand either, and the machine is closely
guarded :-/ but I think you are on to something: it's a v1.4.0.

# git --version
> git version 1.4.0

# cat /etc/issue
Welcome to SUSE LINUX Enterprise Server 9 (i586) - Kernel \r (\l).
# uname -a
Linux lhostname 2.6.5-7.244-bigsmp-100hz #3 SMP Sun Aug 13 21:50:35 NZST
2006 i686 i686 i386 GNU/Linux

The errors look like

# git-pull /home/jun/moodle-r2.git
ird-mk2:fromusb
fatal: git-unpack-objects exec failed
fatal: git-unpack-objects died with error code 128
Fetch failure: /home/jun/moodle-r2.git

# cg-update fromusb
Using hard links
Fetching head...
Fetching objects...
Getting pack 445d79a6fe09a3d03489449f63faef0d9e9e2668
 which contains 4f4773fb3403f3ec4097ab7c7b1fdec23b9aa924
fatal: corrupted pack file
.git/objects/pack/pack-445d79a6fe09a3d03489449f63faef0d9e9e2668.pack
progress: 2 objects, 0 bytes
cg-fetch: objects fetch failed

> Maybe the client machine runs git version < 1.4.2.2, in which case it is
> possible that your push created a pack containing delta objects with
> offset to base which git versions prior 1.4.2.2 do not understand.

Ouch. We weren't supposed to have non-backwards compatible changes...

> If this is the problem you are facing (the error message should confirm
> this) then the easiest solution is to upgrade git on the client.

Ha ha. Not particularly easy, unfortunately.

> A quick fix for the client is to set repack.usedeltabaseoffset to
> false on the machine where you have git 1.5 installed, then run "git
> repack -a -d", and finally copy the pack over to the client repository.

That'll be a bit easier -- it's a fix we can do on the transfer repo ourselves.

Thanks! I do wonder though -- isn't a backwards-incompatible change
like this worthy of don't we bump core.repositoryformatversion?

> But because you push to a local repository (a mounted USB stick is
> considered a local repo) then you don't get to negociate the pack
> capabilities of the final destination, and therefore more "bad" delta
> objects might sneak in again.

How does that work? So any repo we push _from_ can override (and muck
up) the destination repo, ignoring its config?

That sounds a bit broken - the pack being built for a local
destination should respect the settings of the destination repo.


m

Re: Debugging strange "corrupt pack" errors on SuSE 9

[Martin Langhoff]

On 6/20/07, Martin Langhoff <martin.langhoff@gmail.com> wrote:
> > But because you push to a local repository (a mounted USB stick is
> > considered a local repo) then you don't get to negociate the pack
> > capabilities of the final destination, and therefore more "bad" delta
> > objects might sneak in again.
>
> How does that work? So any repo we push _from_ can override (and muck
> up) the destination repo, ignoring its config?
>
> That sounds a bit broken - the pack being built for a local
> destination should respect the settings of the destination repo.

OTOH, as a workaround, it _should_ work if I force a repack on the
usb-repo after each push, right? It'll wear the USB disk out, waste
human+cpu time and kill some kittens along the way, but it'll do for
the time being.

cheers,


m

Re: Debugging strange "corrupt pack" errors on SuSE 9

[Nicolas Pitre]

On Wed, 20 Jun 2007, Martin Langhoff wrote:

> OTOH, as a workaround, it _should_ work if I force a repack on the
> usb-repo after each push, right? It'll wear the USB disk out, waste
> human+cpu time and kill some kittens along the way, but it'll do for
> the time being.

Right.  And you don't need to use -f with git-repack for it to work.


Nicolas

Re: Debugging strange "corrupt pack" errors on SuSE 9

[Nicolas Pitre]

On Wed, 20 Jun 2007, Martin Langhoff wrote:

> On 6/20/07, Nicolas Pitre <nico@cam.org> wrote:
> > Maybe the client machine runs git version < 1.4.2.2, in which case it is
> > possible that your push created a pack containing delta objects with
> > offset to base which git versions prior 1.4.2.2 do not understand.
> 
> Ouch. We weren't supposed to have non-backwards compatible changes...

Well... to be fair, we should say that your setup is a bit non 
conventional.

> > If this is the problem you are facing (the error message should confirm
> > this) then the easiest solution is to upgrade git on the client.
> 
> Ha ha. Not particularly easy, unfortunately.
> 
> > A quick fix for the client is to set repack.usedeltabaseoffset to
> > false on the machine where you have git 1.5 installed, then run "git
> > repack -a -d", and finally copy the pack over to the client repository.
> 
> That'll be a bit easier -- it's a fix we can do on the transfer repo
> ourselves.
> 
> Thanks! I do wonder though -- isn't a backwards-incompatible change
> like this worthy of don't we bump core.repositoryformatversion?

The repository hasn't really changed, and even such a version bump 
wouldn't help you anyway.

I also note that you're using cogito.  And from what I can deduce, it 
seems that cogito is simply copying the pack(s) over without further 
sanity checks.  This only serves to muddy things even further.

If your client machine is so important then you should consider 
upgrading to a later git version, and stop using cogito.  It will solve 
this pack problem, and you'll get added pack sanity checks as a bonus.


Nicolas

Re: Debugging strange "corrupt pack" errors on SuSE 9

[Martin Langhoff]

On 6/20/07, Nicolas Pitre <nico@cam.org> wrote:
> Well... to be fair, we should say that your setup is a bit non
> conventional.

You mean this doesn't affect things between 1.4.0 and 1.5.0 on either
end of the git protocol?

> I also note that you're using cogito.  And from what I can deduce, it
> seems that cogito is simply copying the pack(s) over without further
> sanity checks.  This only serves to muddy things even further.

Yep - we are moving away from cogito; the machine is important, but
it's very awkward to install/upgrade sw on it.

cheers,


m

Re: Debugging strange "corrupt pack" errors on SuSE 9

[Nicolas Pitre]

On Wed, 20 Jun 2007, Martin Langhoff wrote:

> On 6/20/07, Nicolas Pitre <nico@cam.org> wrote:
> > Well... to be fair, we should say that your setup is a bit non
> > conventional.
> 
> You mean this doesn't affect things between 1.4.0 and 1.5.0 on either
> end of the git protocol?

Exact.


Nicolas

Re: Debugging strange "corrupt pack" errors on SuSE 9

[Nicolas Pitre]

On Wed, 20 Jun 2007, Martin Langhoff wrote:

> On 6/20/07, Nicolas Pitre <nico@cam.org> wrote:
> 
> > But because you push to a local repository (a mounted USB stick is
> > considered a local repo) then you don't get to negociate the pack
> > capabilities of the final destination, and therefore more "bad" delta
> > objects might sneak in again.
> 
> How does that work? So any repo we push _from_ can override (and muck
> up) the destination repo, ignoring its config?
> 
> That sounds a bit broken - the pack being built for a local
> destination should respect the settings of the destination repo.

But it does!  The problem is that _you_ are cheating it by removing the 
USB stick and mounting it somewhere else.

Then you're using cogito which bypasses things.

If you use pure git on the client machine, and actually use git-pull or 
git-fetch _without_ the -l switch then validation of the updated data 
will occur and there would be no way to muck up the destination repo.


Nicolas

Re: Debugging strange "corrupt pack" errors on SuSE 9

[Jakub Narebski]

Nicolas Pitre wrote:

> But because you push to a local repository (a mounted USB stick is 
> considered a local repo) then you don't get to negociate the pack 
> capabilities of the final destination, and therefore more "bad" delta 
> objects might sneak in again.

So if he pushes over ssh (ssh localhost) he would get capabilities
negotiation, at the cost of being slower?

-- 
Jakub Narebski
Warsaw, Poland
ShadeHawk on #git