Remote corruption issue, linked to thin pack code?

Remote corruption issue, linked to thin pack code?

[Thomas Rast]

Hi *,

This one sounds fairly similar to

  http://thread.gmane.org/gmane.comp.version-control.git/123244/

which was never resolved.

Jonas reported repo corruption on IRC along the lines of

  $ make gitpush
  cd dist && git push ssh://<user>@eloxoph.com/repos/mainrepo/bla.git master
   
    -  W e l c o m e   t o   E L O X O P H  -
               friendly landlord

  <user>@eloxoph.com's password: 
  Counting objects: 201, done.
  Delta compression using up to 2 threads.
  Compressing objects: 100% (132/132), done.
  Writing objects: 100% (133/133), 1.01 MiB, done.
  Total 133 (delta 118), reused 0 (delta 0)
  fatal: pack has 114 unresolved deltas
  error: unpack failed: index-pack abnormal exit
  To ssh://<user>@eloxoph.com/repos/mainrepo/bla.git
   ! [remote rejected] master -> master (n/a (unpacker error))
  error: failed to push some refs to 'ssh://<user>@eloxoph.com/repos/mainrepo/bla.git'
  make: *** [gitpush] Error 1

The respective repos show no errors with git-fsck.  Jonas kindly
provides a download link for both:

  http://eloxoph.com/localrepo.zip
  http://eloxoph.com/remoterepo.zip

Versions are 1.7.2.1 locally and 1.5.4.3 remotely.

Beware, there are hooks in both.  Disabling them, however, only
changes the error slightly:

  [...]
  Total 133 (delta 118), reused 0 (delta 0)
  fatal: pack has 114 unresolved deltas
  error: unpack failed: index-pack abnormal exit
  To ssh://mainrepo@eloxoph.com/repos/mainrepo/bla.git
   ! [remote rejected] master -> master (n/a (unpacker error))
  error: failed to push some refs to 'ssh://mainrepo@eloxoph.com/repos/mainrepo/bla.git'
  make: *** [gitpush] Error 1

What's even stranger is that fetching from the repo is also not
possible:

  fatal: git-upload-pack: cannot find object e28ae6b61c384732c506544626c5083557dd2d75:
  fatal: The remote end hung up unexpectedly

despite the object being there.

What's also strange is that while there is a temporary pack inside
objects/, I get

  $ git index-pack --stdin < objects/tmp_pack_oEUkIc
  fatal: pack has 114 unresolved deltas
  $ git index-pack --fix-thin -v --stdin < objects/tmp_pack_oEUkIc
  Receiving objects: 100% (133/133), 1.01 MiB, done.
  Resolving deltas: 100% (118/118), completed with 63 local objects.
  pack    061120577b0a1fec7ba636d6e3162f95f83543aa

So it seems the remote side got a thin pack and can't cope.  But
a4503a1 (Make --no-thin the default in git-push to save server
resources, 2007-09-09), merged way back in 1.5.3.2, claims to enable
--no-thin all the time.  So how did a thin pack get there?

Any ideas?

-- 
Thomas Rast
trast@{inf,student}.ethz.ch

Re: Remote corruption issue, linked to thin pack code?

[Nicolas Pitre]

On Wed, 25 Aug 2010, Thomas Rast wrote:

> Hi *,
> 
> This one sounds fairly similar to
> 
>   http://thread.gmane.org/gmane.comp.version-control.git/123244/
> 
> which was never resolved.

I don't think it actually is the same issue.

> Jonas reported repo corruption on IRC along the lines of
> 
>   $ make gitpush
>   cd dist && git push ssh://<user>@eloxoph.com/repos/mainrepo/bla.git master
>    
>     -  W e l c o m e   t o   E L O X O P H  -
>                friendly landlord
> 
>   <user>@eloxoph.com's password: 
>   Counting objects: 201, done.
>   Delta compression using up to 2 threads.
>   Compressing objects: 100% (132/132), done.
>   Writing objects: 100% (133/133), 1.01 MiB, done.
>   Total 133 (delta 118), reused 0 (delta 0)
>   fatal: pack has 114 unresolved deltas

So... out of 118 deltas, 114 were relying on the remote to already have 
base objects for them (or some of them are deltas against unresolved 
deltas).  In any case, the remote is missing some objects it advertised 
it should have.

> The respective repos show no errors with git-fsck.

What about 'git fsck --full'?  Given git v1.5.4 on the remote, you 
should try --full with 'git fsck' as this wasn't the default back then.

> Jonas kindly
> provides a download link for both:
> 
>   http://eloxoph.com/localrepo.zip
>   http://eloxoph.com/remoterepo.zip

$ wget http://eloxoph.com/remoterepo.zip
--2010-08-25 21:24:52--  http://eloxoph.com/remoterepo.zip
Resolving eloxoph.com... 85.214.104.74
Connecting to eloxoph.com|85.214.104.74|:80... connected.
HTTP request sent, awaiting response... 404 Not Found
2010-08-25 21:24:52 ERROR 404: Not Found.

> What's even stranger is that fetching from the repo is also not
> possible:
> 
>   fatal: git-upload-pack: cannot find object e28ae6b61c384732c506544626c5083557dd2d75:
>   fatal: The remote end hung up unexpectedly
> 
> despite the object being there.

How do you know it is there while git (on the remote) is telling you it 
is not?

What about 'git cat-file -t e28ae6b61c384732c506' ?

> What's also strange is that while there is a temporary pack inside
> objects/, I get
> 
>   $ git index-pack --stdin < objects/tmp_pack_oEUkIc
>   fatal: pack has 114 unresolved deltas
>   $ git index-pack --fix-thin -v --stdin < objects/tmp_pack_oEUkIc
>   Receiving objects: 100% (133/133), 1.01 MiB, done.
>   Resolving deltas: 100% (118/118), completed with 63 local objects.
>   pack    061120577b0a1fec7ba636d6e3162f95f83543aa
> 
> So it seems the remote side got a thin pack and can't cope. 

Hmmm... in this case it must be because git v1.5.4 is buggy.

/me looks

Well... apparently no.  In receive-pack.c:unpack() there is this code:

                keeper[0] = "index-pack";
                keeper[1] = "--stdin";
                keeper[2] = "--fix-thin";
                keeper[3] = hdr_arg;
                keeper[4] = keep_arg;
                keeper[5] = NULL;

so --fix-thin is always unconditionally passed to index-pack.  This is 
also more or less the same code in v1.7.2.

> But a4503a1 (Make --no-thin the default in git-push to save server 
> resources, 2007-09-09), merged way back in 1.5.3.2, claims to enable 
> --no-thin all the time.  So how did a thin pack get there?

Well, this was added back by accident a long time ago, and since then I 
also changed my mind about this, especially since 'git gc --auto' is 
executed on the remote after a push these days, so I never pushed a 
"fix" for it.

Regardless, the receiving end of a push shouldn't care either ways and 
always complete a thin pack.

So there are 2 mysteries here:

1) Why can't the remote repo be fetched? The error from git-upload-pack 
   is straight forward and has nothing to do with any thin packs.

2) Why adding --fix-thin to a manual invocation of index-pack does work 
   while the invocation from receive_pack which should have it doesn't?

Also, what is the OS version and filesystem used on the remote machine?

> Any ideas?

Maybe after I can reproduce this issue locally.


Nicolas

Re: Remote corruption issue, linked to thin pack code?

[Thomas Rast]

Nicolas Pitre wrote:
> > The respective repos show no errors with git-fsck.
> 
> What about 'git fsck --full'?  Given git v1.5.4 on the remote, you 
> should try --full with 'git fsck' as this wasn't the default back then.

Well, I instructed him to try --full, as you can see in the
conversation:

  http://colabti.org/irclogger/irclogger_log/git?date=2010-08-25#l2196

Nothing interesting.

> >   http://eloxoph.com/localrepo.zip
> >   http://eloxoph.com/remoterepo.zip
> 
> 2010-08-25 21:24:52 ERROR 404: Not Found.

Hrm.  Apparently a mis-edit on my part, it should be

  http://eloxoph.com/remoteserverrepo.zip

> How do you know it is there while git (on the remote) is telling you it 
> is not?
> 
> What about 'git cat-file -t e28ae6b61c384732c506' ?

It's fine in my unpacked copy.  Jonas, what were the results on the
server?

> Maybe after I can reproduce this issue locally.

As a side note, I couldn't reproduce locally with the above repos but
then I didn't have the time to downgrade to 1.5.4.

Jonas also has to answer the rest of the questions for obvious
reasons.  (I'm away from the net for a few days anyway...)

-- 
Thomas Rast
trast@{inf,student}.ethz.ch

Re: Remote corruption issue, linked to thin pack code?

[Jonas Thiem]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

Hi,

I'll try to provide the missing information as good as I can:

> What about 'git fsck --full'?  Given git v1.5.4 on the remote, you 
> should try --full with 'git fsck' as this wasn't the default back then.
$ GIT_DIR=./ git fsck --full
dangling commit d298351127861349846fe626a320c66101821d72
dangling commit a71ea8dd4c837c39ef26765574d515253ea2fd5a

> What about 'git cat-file -t e28ae6b61c384732c506' ?
$ GIT_DIR=./ git cat-file -t e28ae6b61c384732c506
commit

> Also, what is the OS version and filesystem used on the remote machine?
OS is Ubuntu 8.04 LTS. It's a virtual server by a hosting company,
therefore I don't know much about the filesystem setup, but maybe that
helps:

$ df -T:
Filesystem    Type   1K-blocks      Used Available Use% Mounted on
/dev/vzfs reiserfs    10485760   4056808   6428952  39% /
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.10 (GNU/Linux)
Comment: Using GnuPG with Fedora - http://enigmail.mozdev.org/

iF4EAREIAAYFAkx2Ph8ACgkQVizsUChSmww9TAEAptTyg+0yqidArbg2wMLmgac+
PrK9T3g4cU20EBOEuXkBALlqLapTQLlyO8UoFi+I3Mq64sDBBdjaEqH8bi6LcTor
=ahRa
-----END PGP SIGNATURE-----