fsck --full is Ok, but clones are not, "missing commits"?!

fsck --full is Ok, but clones are not, "missing commits"?!

[Brian Foster]

   (many many apologies if this turns into a double post,
  there seems to have been problems with the 1st attempt?)

 I've recently inherited a bare git repository,
 which, as far as I can tell (I'm something of
 a newbie with git), seems Ok: `git fsck --full'
 does not report any problems.    however, any
 clones I make from it are not Ok:

	$ git-fsck --full   # clone (same command for bare repo is Ok)
	broken link from  commit dd3f3c0636cfd50719c706b030db5473b0270add
		      to  commit 0fed9c2eb14eee47097e1d870fe8e55a6430edeb
	missing commit fb57c018d15005b60f104e57f198ff34a6035b99
	missing commit f8947cb0b5fe605e6cb5f73c89f262424b64ef3c
	missing commit 0fed9c2eb14eee47097e1d870fe8e55a6430edeb
	missing commit dff364d8da15be0b856a174062fb785acb1c363e
	broken link from  commit 8700854c41a40d333e90104971c3abbbcf082e57
		      to  commit b8b78d1c09e7009f97a1d624f4d771d7e5bd8551
	missing commit ce33a5e8c32816f38862bc41560a1d646d0803a6
	missing commit 91483db9b01b547ae9cc45c8c98b217642acb40a
	missing commit b8b78d1c09e7009f97a1d624f4d771d7e5bd8551
	broken link from  commit c87c46fe892211f8aa4fd363ccff4f667a9aaf7d
		      to  commit ce33a5e8c32816f38862bc41560a1d646d0803a6
	broken link from  commit fb5967688f7b464421cff28f266b64ad2a313a9e
		      to  commit f8947cb0b5fe605e6cb5f73c89f262424b64ef3c
	broken link from  commit e5a60f1636cceac33777bb8098a0b7a4a136a56c
		      to  commit fb57c018d15005b60f104e57f198ff34a6035b99
	broken link from  commit 2dcaaf2decd31ac9a21d616604c0a7c1fa65d5a4
		      to  commit 91483db9b01b547ae9cc45c8c98b217642acb40a
	broken link from  commit 0ff75b3afff6fb306bef221bf1823ccf5ffc568b
		      to  commit dff364d8da15be0b856a174062fb785acb1c363e
	$ 

 the Ok(?) bare repository only has a `master' branch.
 it has numerous tags.  AFAIK, it has been "inactive"
 for the last c.6+ months (or longer?), except (maybe)
 for browsing, probably with gitweb.  there have been,
 as far as I can tell, no commits or other "writes"
 for the last c.6+ months.

 to my extreme disgust, it was never(?!) backed-up,
 except for one not entirely coincidental backup,
 also c.6+ months ago.  ;-(   (this mind-boggling
 silliness is being fixed ASAP!)

 this is happening regardless of whether the clone
 is made on the same machine or a different one,
 regardless of the transport, and for both `--bare'
 and not-bare clones.  nothing fancy is being done
 during the cloning: `git clone [--bare] DIR_or_URL'

 a test (tried by someone else) using that one
 backup apparently produced the same results:  the
 backup seems Ok, but clones made from it are not.

 the missing commits are linux-mips.org/kernel.org
 mergers (mostly).  apologies for being vague here,
 but I don't have my notes with me.  ;-(

 what is going on?  I'm completely baffled!  to-date,
 nothing terribly obvious/similar has shown up in any
 searches I've tried.  (and there's nothing unusual in
 /var/log/* that I can see.)

 I've also no idea how to "fix" the problem.

 explanations, suggestions, questions, &tc are all
 very welcome.  (please keep in mind my current level
 of knowledge/experience about git is very weak. ;-\ )

 this is all(?) using git 1.5.x (where the .x varies
 depending on the machine in question); the distros
 being used include ubuntu 7.10, FC-5, and probably
 others.  (I'll check the filesystem tomorrow, but
 I believe it's sane.)

cheers!
	-blf-
-- 
"How many surrealists does it take to    |  Brian Foster
 change a lightbulb?  Three.  One calms  |  somewhere in south of France
 the warthog, and two fill the bathtub   |     Stop E$$o (ExxonMobil)!
 with brightly-coloured machine tools."  |       http://www.stopesso.com

Re: fsck --full is Ok, but clones are not, "missing commits"?!

[David Kastrup]

Brian Foster <brian.foster@innova-card.com> writes:

>    (many many apologies if this turns into a double post,
>   there seems to have been problems with the 1st attempt?)
>
>  I've recently inherited a bare git repository,
>  which, as far as I can tell (I'm something of
>  a newbie with git), seems Ok: `git fsck --full'
>  does not report any problems.    however, any
>  clones I make from it are not Ok:
>
> 	$ git-fsck --full   # clone (same command for bare repo is Ok)
> 	broken link from  commit dd3f3c0636cfd50719c706b030db5473b0270add
> 		      to  commit 0fed9c2eb14eee47097e1d870fe8e55a6430edeb
> 	missing commit fb57c018d15005b60f104e57f198ff34a6035b99
> 	missing commit f8947cb0b5fe605e6cb5f73c89f262424b64ef3c
> 	missing commit 0fed9c2eb14eee47097e1d870fe8e55a6430edeb
> 	missing commit dff364d8da15be0b856a174062fb785acb1c363e

>From the git-clone manpage


	--shared, -s
	    When the repository to clone is on the local machine,
	    instead of using hard links, automatically setup
	    .git/objects/info/alternates to share the objects with
	    the source repository. The resulting repository starts
	    out without any object of its own. NOTE: this is a
	    possibly dangerous operation; do not use it unless you
	    understand what it does. If you clone your repository
	    using this option, then delete branches in the source
	    repository and then run git-gc(1) using the --prune
	    option in the source repository, it may remove objects
	    which are referenced by the cloned repository.

Did you use something like that?

-- 
David Kastrup

Re: Re: fsck --full is Ok, but clones are not, "missing commits"?!

[Brian Foster]

David Kastrup <dak@gnu.org> sensibly asks:
>  Brian Foster <brian.foster@innova-card.com> writes:
>  >  I've recently inherited a bare git repository,
>  >  which, as far as I can tell (I'm something of
>  >  a newbie with git), seems Ok: `git fsck --full'
>  >  does not report any problems.    however, any
>  >  clones I make from it are not Ok:
>  >
>  >       $ git-fsck --full   # clone (same command for bare repo is Ok)
>  >       broken link from  commit dd3f3c0636cfd50719c706b030db5473b0270add
>  >                     to  commit 0fed9c2eb14eee47097e1d870fe8e55a6430edeb
>  >       missing commit fb57c018d15005b60f104e57f198ff34a6035b99
>  >       missing commit f8947cb0b5fe605e6cb5f73c89f262424b64ef3c
>  >       missing commit 0fed9c2eb14eee47097e1d870fe8e55a6430edeb
>  >       missing commit dff364d8da15be0b856a174062fb785acb1c363e
>
>  From the git-clone manpage
>         --shared, -s  [ ... ]
>
>  Did you use something like that?

 NO, not for any of the clones I've made.  basically,
 I've tried:

   git clone        git://SERVER/PATH   # on a remote machine
   git clone --bare git://SERVER/PATH   # on a remote machine
   git clone        /FULL/PATH          # on the SERVER
   git clone --bare /FULL/PATH          # on the SERVER

 and possibly a few other variants along similar lines.
 nothing fancy.  just the sort of routine simple stuff
 you'd expect to work, and expect(?) a newbie to use.

 none of my clones, nor the repro on SERVER I'm trying
 to clone, have any .../objects/info/alternates

cheers!
	-blf-
-- 
"How many surrealists does it take to    |  Brian Foster
 change a lightbulb?  Three.  One calms  |  somewhere in south of France
 the warthog, and two fill the bathtub   |     Stop E$$o (ExxonMobil)!
 with brightly-coloured machine tools."  |       http://www.stopesso.com

Re: Re: fsck --full is Ok, but clones are not, "missing commits"?!

[Dmitry Potapov]

Hi Brian,

On Wed, Apr 16, 2008 at 11:45:39AM +0200, Brian Foster wrote:
> David Kastrup <dak@gnu.org> sensibly asks:
> >  Brian Foster <brian.foster@innova-card.com> writes:
> >  >  I've recently inherited a bare git repository,
> >  >  which, as far as I can tell (I'm something of
> >  >  a newbie with git), seems Ok: `git fsck --full'
> >  >  does not report any problems.    however, any
> >  >  clones I make from it are not Ok:
> >  >
> >  >       $ git-fsck --full   # clone (same command for bare repo is Ok)
> >  >       broken link from  commit dd3f3c0636cfd50719c706b030db5473b0270add
> >  >                     to  commit 0fed9c2eb14eee47097e1d870fe8e55a6430edeb
> >  >       missing commit fb57c018d15005b60f104e57f198ff34a6035b99
> >  >       missing commit f8947cb0b5fe605e6cb5f73c89f262424b64ef3c
> >  >       missing commit 0fed9c2eb14eee47097e1d870fe8e55a6430edeb
> >  >       missing commit dff364d8da15be0b856a174062fb785acb1c363e

I suspect your original git repository has info/grafts

Dmitry

Re: fsck --full is Ok, but clones are not, "missing commits"?!

[Bryan Donlan]

On Wed, Apr 16, 2008 at 08:37:39AM +0200, Brian Foster wrote:
>    (many many apologies if this turns into a double post,
>   there seems to have been problems with the 1st attempt?)
> 
>  I've recently inherited a bare git repository,
>  which, as far as I can tell (I'm something of
>  a newbie with git), seems Ok: `git fsck --full'
>  does not report any problems.    however, any
>  clones I make from it are not Ok:
 
[snip]
 
>  explanations, suggestions, questions, &tc are all
>  very welcome.  (please keep in mind my current level
>  of knowledge/experience about git is very weak. ;-\ )
> 
>  this is all(?) using git 1.5.x (where the .x varies
>  depending on the machine in question); the distros
>  being used include ubuntu 7.10, FC-5, and probably
>  others.  (I'll check the filesystem tomorrow, but
>  I believe it's sane.)

Is there an info/grafts file? If the repository doesn't have sensitive
information in it, it would probably be helpful to tarball it up and
upload it somewhere, so we can take a look at things directly.

Re: fsck --full is Ok, but clones are not, "missing commits"?!

[Brian Foster]

On Monday 05 May 2008 06:25:46 Bryan Donlan asked:
> On Wed, Apr 16, 2008 at 08:37:39AM +0200, Brian Foster wrote:
> >  I've recently inherited a bare git repository,
> >  which, as far as I can tell (I'm something of
> >  a newbie with git), seems Ok: `git fsck --full'
> >  does not report any problems.    however, any
> >  clones I make from it are not Ok [ ... ]
>
> Is there an info/grafts file?  If the repository doesn't have sensitive
> information in it, it would probably be helpful to tarball it up and
> upload it somewhere, so we can take a look at things directly.

Bryan,

 Yes, the proximate cause was an info/grafts.

 The repository in question is for a port of Linux
 to the MIPS-based "secure" SoC made by the company
 I work for.  The repository was very simple:

    master:  o---o---o---o---o--...--o  HEAD

 where each `o' was a tagged version/release of the
 Linux port.  Since the chip is MIPS-based, some `o'
 were the result of merges with mips-linux baseline.
 Hence, the real history was, broadly:

       master:          o--o--o--o--o--o-----o--...--o
                       /           /  /     /
   mips-linux:  o--o--o----o------o--o--o--o--...--o

 where, of course, `mips-linux' has its own rather
 complicated history (not shown).  As implied by
 the diagrams above, grafts were used to "suppress"
 mips-linux and its history.  (Since the situation
 really was as trivial as drawn above, it was easy
 to recover:  Add the pack representing mips-linux,
 remove grafts, and ensure all the tags exist.)

 What I don't know is the root-cause, that is, WHY
 this was done.  It wasn't a disc-space issue, and
 I've no evidence it was a network-bandwidth issue,
 but there is some anecdotal evidence it was some
 sort of a CPU-cycles issue, albeit just what the
 performance hit was is unknown.

 Another possibility is the repository started life
 as CVS (ugh!) before being migrated to git.  It's
 my (vague) understanding grafts is somehow useful
 as a (temporary?) aid when doing a CVS-->git move;
 and I speculate it was found so useful/simple the
 practice simply continued.

 The developers tended to work from tarballs (not
 clones), so the cloning problem was either unknown,
 not a concern, or mis-understood.

cheers!
	-blf-

-- 
"How many surrealists does it take to   | Brian Foster
 change a lightbulb? Three. One calms   | somewhere in south of France
 the warthog, and two fill the bathtub  |   Stop E$$o (ExxonMobil)!
 with brightly-coloured machine tools." |      http://www.stopesso.com

Re: fsck --full is Ok, but clones are not, "missing commits"?!

[Johannes Sixt]

Brian Foster schrieb:
>  What I don't know is the root-cause, that is, WHY
>  this was done.  It wasn't a disc-space issue, and
>  I've no evidence it was a network-bandwidth issue,
>  but there is some anecdotal evidence it was some
>  sort of a CPU-cycles issue, albeit just what the
>  performance hit was is unknown.

How about this theory:

What happens if you fire up gitk as simple as

   $ gitk

in the history if no grafts are present? Some months ago this took ages to
complete, and even today you get a *huge* list of commits in a *short*
window; hence, the scrollbar thumb is tiny, and if you succeed to get hold
of it without a magnifying glass, it scrolls way more than a page of
commits if you move it by only one pixel.

No wonder that $user wants to have a shorter history. So $user, being
smart, truncates the history at a suitable point with a graft.

-- Hannes

Re: fsck --full is Ok, but clones are not, "missing commits"?!

[Brian Foster]

Johannes Sixt wrote:
> Brian Foster schrieb:
> >  What I don't know is the root-cause, that is, WHY
> >  this was done.  [ ... ]  there is some anecdotal
> >  evidence it was some sort of a CPU-cycles issue,
> >  albeit just what the performance hit was is unknown.
>
> How about this theory:
>
> What happens if you fire up gitk as simple as
>
>    $ gitk
>
> in the history if no grafts are present? Some months ago this took ages to
> complete, and even today you get a *huge* list of commits in a *short*
> window; hence, the scrollbar thumb is tiny, and if you succeed to get hold
> of it without a magnifying glass, it scrolls way more than a page of
> commits if you move it by only one pixel.
>
> No wonder that $user wants to have a shorter history. So $user, being
> smart, truncates the history at a suitable point with a graft.

Hannes,

 Unfortunately, I cannot fire up `gitk' in the exact
 same configuration anymore (that server machine is now
 being used for other purposes, albeit I'm supposed to
 get the hard disc).  The git on the now-vanished server
 was v1.5.3, but that's probably not relevant, since the
 repository must have been created with a much older git
 (it goes back multiple years).

 All the (now-)installed gits I've seen are 1.5.<something>.
 I do not see any noticeable performance issue with 1.5.2.5
 (nor with 1.5.5)?  The scrollbar is, as you say, unusable.

 But how important is `gitk'?  Is it something that'd be
 used frequently enough for the formerly-poor performance
 to be such an issue that creating and maintaining such a
 "truncated" repository is worthwhile?

 It's an interesting and plausible hypothesis, but (in
 the absence of any actual evidence) I'd be more inclined
 to buy it if there was some frequent/critical operation
 where poor performance clearly matters.

cheers!
	-blf-

Re: fsck --full is Ok, but clones are not, "missing commits"?!

[Johannes Sixt]

Brian Foster schrieb:
> Johannes Sixt wrote:
>> What happens if you fire up gitk as simple as
>>
>>    $ gitk
>>
>> in the history if no grafts are present? Some months ago this took ages to
>> complete, and even today you get a *huge* list of commits in a *short*
>> window; hence, the scrollbar thumb is tiny, and if you succeed to get hold
>> of it without a magnifying glass, it scrolls way more than a page of
>> commits if you move it by only one pixel.
>>
>> No wonder that $user wants to have a shorter history. So $user, being
>> smart, truncates the history at a suitable point with a graft.
> 
> Hannes,
> 
>  Unfortunately, I cannot fire up `gitk' in the exact
>  same configuration anymore (that server machine is now
>  being used for other purposes, albeit I'm supposed to
>  get the hard disc).  The git on the now-vanished server
>  was v1.5.3, but that's probably not relevant, since the
>  repository must have been created with a much older git
>  (it goes back multiple years).
> 
>  All the (now-)installed gits I've seen are 1.5.<something>.
>  I do not see any noticeable performance issue with 1.5.2.5
>  (nor with 1.5.5)?  The scrollbar is, as you say, unusable.

For me, the unusable scrollbar alone would be reason enough to truncate
the history. Once it is truncated, performance is no longer an issue
(whether or not it was an issue in the first place).

>  But how important is `gitk'?  Is it something that'd be
>  used frequently enough for the formerly-poor performance
>  to be such an issue that creating and maintaining such a
>  "truncated" repository is worthwhile?

Well, I have gitk running all the time. So, yes, it is "important." But I
 run it basically as 'gitk --all --not origin' and press F5 frequently.
With this set of arguments the scrollbar remains usable, and performance
is not an issue, even on Windows.

-- Hannes