FW: Git pulls failing on ia64 test tree?

FW: Git pulls failing on ia64 test tree?

[Luck, Tony]

I've had two complaints that people can't pull from my tree
on kernel.org.  Here's a snip from the errors.

 > $ git pull 
rsync://rsync.kernel.org/pub/scm/linux/kernel/git/aegl/linux-2.6.git test
 > MOTD:
 > MOTD:   Welcome to the Linux Kernel Archive.
    ... MOTD trimmed ...
 > receiving file list ... done
 >
 > wrote 113 bytes  read 30799 bytes  20608.00 bytes/sec
 > total size is 82275235  speedup is 2661.60
 > * committish: 6c141f61ee4f5fba63d9c54e03cb2faa6877bf1a  refs/heads/test from 
rsync://rsync.kernel.org/pub/scm/linux/kernel/git/aegl/linux-2.6.git
 > error: Could not read 9401c705f2a6a7e5df102f6443dba395c3c5e5a8
 > error: Could not read 357d596bd552ad157a906289ab13ea6ba7e66e3d
 > Updating from aa2dca4563b0629ecd9d9994dfdf39f29ff1b43f to 
6c141f61ee4f5fba63d9c54e03cb2faa6877bf1a.
 > fatal: failed to unpack tree object 6c141f61ee4f5fba63d9c54e03cb2faa6877bf1a
 > $

After the first complaint, I looked around and found out about
"git-update-server-info" (which I'd somehow missed).  So I ran
(on master.kernel.org):

  $ GIT_DIR=/pub/scm/linux/kernel/git/aegl/linux-2.6.git git-update-server-info

and I now have info/refs, info/rev-cache, and objects/info/packs files.

12 hours later, I got another complaint.

I've set up objects/info/alternates to point at linus' tree and deleted
all the packs from my tree.  The web view at kernel.org/git looks ok,
so I think those changes are ok.

So what am I missing?

-Tony

Re: FW: Git pulls failing on ia64 test tree?

[Junio C Hamano]

"Luck, Tony" <tony.luck@intel.com> writes:

> After the first complaint, I looked around and found out about
> "git-update-server-info" (which I'd somehow missed).  So I ran
> (on master.kernel.org):
>
>   $ GIT_DIR=/pub/scm/linux/kernel/git/aegl/linux-2.6.git git-update-server-info
>
> and I now have info/refs, info/rev-cache, and objects/info/packs files.

That is a nice, but rsync bypasses all of that so I suspect
there is something else.  If you did not say:

> 12 hours later, I got another complaint.

I would have suspected that this was when one of the kernel.org
mirror machine was misbehaving and did not update for a day or
so, but I think it is already resolved now so I am puzzled..

> I've set up objects/info/alternates to point at linus' tree and deleted
> all the packs from my tree.  The web view at kernel.org/git looks ok,
> so I think those changes are ok.
>
> So what am I missing?

Let me take a look at your tree on master while I'll try to pull
from your public tree and see if I can reproduce.

Re: FW: Git pulls failing on ia64 test tree?

[Junio C Hamano]

Junio C Hamano <junkio@cox.net> writes:

> "Luck, Tony" <tony.luck@intel.com> writes:
>
>> I've set up objects/info/alternates to point at linus' tree and deleted
>> all the packs from my tree.  The web view at kernel.org/git looks ok,
>> so I think those changes are ok.
>>
>> So what am I missing?
>
> Let me take a look at your tree on master while I'll try to pull
> from your public tree and see if I can reproduce.

Ah, you deleted packs from your repository, which I missed when
I first saw your message.

The objects/info/alternates mechanism is currently only known to
the git that operates on the machine that has the file -- client
side does not make use of that information.  That means only
git-daemon and 'git native protocol over ssh' (not the commit
walker ssh-fetch) can deal with such a repository right now,
because they run git on the side your repository is.  Gitweb
also obviously runs git on the machine a copy of repository is,
so you would not see problems.

The recommendation by Linus to "point him with alternate to
borrow his objects" is a sound one in the long term, but I think
it was a bit premature in practice -- the client side tools are
not ready for that yet.  From the original error report:

 > error: Could not read 9401c705f2a6a7e5df102f6443dba395c3c5e5a8
 > error: Could not read 357d596bd552ad157a906289ab13ea6ba7e66e3d
 > Updating from aa2dca4563b0629ecd9d9994dfdf39f29ff1b43f to 
6c141f61ee4f5fba63d9c54e03cb2faa6877bf1a.
 > fatal: failed to unpack tree object 6c141f61ee4f5fba63d9c54e03cb2faa6877bf1a
 > $

The first two "Could not read" are talking about objects only
found in Linus' repository -- your objects/ hierarchy does not
have them.  Rsync transport slurps everything under _your_
objects/ hierarchy [*1*] and does not know that your objects/
hierarchy is incomplete and needs to borrow from Linus' objects/
directory.  If the downloader has the objects found in Linus'
repository that is fine but otherwise he has problems.

As a workaround, please do this:

    cd /pub/scm/linux/kernel/git/aegl/linux-2.6.git/
    ln ../../torvalds/linux-2.6.git/objects/pack/* objects/pack/.
    mv objects/info/alternates objects/info/alternates+
    GIT_DIR=. git fetch ../../torvalds/linux-2.6.git/
    GIT_DIR=. git fsck-objects
    mv objects/info/alternates+ objects/info/alternates

The 'git fetch' step would complain about a couple of commits
you have but there is no need to worry -- we are deliberately
"corrupting" your repository by temporarily moving alternates
away so that it can fetch missing objects from Linus repository.
The fsck-objects should then report no missing objects.

Sorry for this very unpleasant experience git has caused you
X-<.

[Footnotes]

*1* BTW, I just spotted a bug there -- since we now have
objects/info subdirectory, we should exclude that when copying.

*2* Ideally I would have liked to do git-relink after 'git fetch'
from Linus repo in the above, but it does not seem to want to
relink a naked repository...  I should bug Ryan about it.

Re: FW: Git pulls failing on ia64 test tree?

[Junio C Hamano]

Junio C Hamano <junkio@cox.net> writes:

> As a workaround, please do this:
>
>     cd /pub/scm/linux/kernel/git/aegl/linux-2.6.git/
>     ln ../../torvalds/linux-2.6.git/objects/pack/* objects/pack/.
>     mv objects/info/alternates objects/info/alternates+
>     GIT_DIR=. git fetch ../../torvalds/linux-2.6.git/
>     GIT_DIR=. git fsck-objects
>     mv objects/info/alternates+ objects/info/alternates

Actually, please chuck the last step to re-enable alternates.
Otherwise everytime Linus updates his repository you will be out
of sync and will get hit by the same problem.

> The 'git fetch' step would complain about a couple of commits
> you have but there is no need to worry -- we are deliberately
> "corrupting" your repository by temporarily moving alternates
> away so that it can fetch missing objects from Linus repository.
> The fsck-objects should then report no missing objects.
>
> Sorry for this very unpleasant experience git has caused you
> X-<.