Re: Re: Re: fsck --full is Ok, but clones are not, "missing commits"?!

Re: Re: Re: fsck --full is Ok, but clones are not, "missing commits"?!

[Brian Foster]

Dmitry Potapov <dpotapov@gmail.com> correctly deduced:
>  On Wed, Apr 16, 2008 at 11:45:39AM +0200, Brian Foster wrote:
>  >  Brian Foster <brian.foster@innova-card.com> writes:
>  >  >  I've recently inherited a bare git repository,
>  >  >  which, as far as I can tell (I'm something of a
>  >  >  newbie with git), seems Ok [ but clones are not ].
>
>  I suspect your original git repository has info/grafts

hi Dmitry,

 bingo!  YES, it does:

	$ cat info/grafts
	8700854c41a40d333e90104971c3abbbcf082e57
34757f56bd7cb007bbb48ce064a2cd7e67c32428
	c87c46fe892211f8aa4fd363ccff4f667a9aaf7d
d560b7a27640cdf9ff1e6a30ae161dc6144618bd
	dd3f3c0636cfd50719c706b030db5473b0270add
e6cfa744e0434f9b700cd8a3780dcf9235ed10e3
	e5a60f1636cceac33777bb8098a0b7a4a136a56c
b38536a9a67b8d6b94e51860195529481703286b
	0ff75b3afff6fb306bef221bf1823ccf5ffc568b
01d4f94e5e5e65058234256241c01a6caf1c5feb
	fb5967688f7b464421cff28f266b64ad2a313a9e
00fe1cce8a053ecf9d8ba160991f28574fb11f63
	2dcaaf2decd31ac9a21d616604c0a7c1fa65d5a4
	$

 now, showing that I am indeed essentially a git newbie,
 WTF is a "graft"?  I don't recall running into the term.
 I shall, of course, immediately start trying to find some
 info blah blah ..., but pointers/explanations would help.

 the goal is to put things into a sane state so any new
 clones are healthy.  there's only one(?) existing clone,
 which may or may not be(? become?) an issue.

cheers!
	-blf-
-- 
"How many surrealists does it take to    |  Brian Foster
 change a lightbulb?  Three.  One calms  |  somewhere in south of France
 the warthog, and two fill the bathtub   |     Stop E$$o (ExxonMobil)!
 with brightly-coloured machine tools."  |       http://www.stopesso.com

Re: Re: Re: fsck --full is Ok, but clones are not, "missing commits"?!

[Dmitry Potapov]

On Wed, Apr 16, 2008 at 01:48:13PM +0200, Brian Foster wrote:
> 
>  now, showing that I am indeed essentially a git newbie,
>  WTF is a "graft"?  I don't recall running into the term.

info/grafts::
	This file records fake commit ancestry information, to
	pretend the set of parents a commit has is different
	from how the commit was actually created.  One record
	per line describes a commit and its fake parents by
	listing their 40-byte hexadecimal object names separated
	by a space and terminated by a newline.

Basically, you use it if you want to modify the history in your
repository. It is often used after importing to Git from some
other repository if you want to make some adjustment to history.
So, you edit info/grafts, then use gitk to verify that you got
what you want, and after that you run git-filter-branch to actually
overwrite the history.

info/grafts should never be used during normal work.


>  the goal is to put things into a sane state so any new
>  clones are healthy.  there's only one(?) existing clone,
>  which may or may not be(? become?) an issue.

If you run git-filter-branch then you'll change "fake" history
to the real one, and after that the repository can be cloned.
Please, make sure that you have a backup before any further changes.

Dmitry

Re: Re: Re: fsck --full is Ok, but clones are not, "missing commits"?!

[Petr Baudis]

On Wed, Apr 16, 2008 at 04:14:24PM +0400, Dmitry Potapov wrote:
> info/grafts should never be used during normal work.

I don't really agree with this advice in general. Grafts can be very
useful especially when migrating to Git - a very good strategy might be
just to start from scratch with your current state, then import your
previous history and graft it. The big advantage is that if you find out
later that the import is screwed up, you can fix that and reimport. The
fact that you don't need to haul 10 years of CVS history around unless
you actually know you need it can be perceived as both advantage and
disadvantage.

The _BAD_ grafts are those that replace list of commit's parents instead
of just appending (typically to an empty one). I didn't actually have
any idea people use grafts in such a twisted and perverse way... ;-)

-- 
				Petr "Pasky" Baudis
Whatever you can do, or dream you can, begin it.
Boldness has genius, power, and magic in it.	-- J. W. von Goethe

Re: Re: Re: fsck --full is Ok, but clones are not, "missing commits"?!

[Dmitry Potapov]

On Wed, Apr 16, 2008 at 04:59:00PM +0200, Petr Baudis wrote:
> On Wed, Apr 16, 2008 at 04:14:24PM +0400, Dmitry Potapov wrote:
> > info/grafts should never be used during normal work.
> 
> I don't really agree with this advice in general. Grafts can be very
> useful especially when migrating to Git

I agree that they can be very useful in the process of migrating, but
I would not recommend to leave them _after_ the migration process is
complete.

> The _BAD_ grafts are those that replace list of commit's parents instead
> of just appending (typically to an empty one). I didn't actually have
> any idea people use grafts in such a twisted and perverse way... ;-)

They are used to replace _BAD_ commits imported from CVS. History in
CVS can be very strange and even with best conversation tools, you may
end up with something not exactly what you want. So, you want to replace
some commits with corrected versions of them. So, you add those commits
to Git repo, "fix" history, and then using gitk check that now you have
exactly what you want. If now the history is okay then you can use
git filter-branch to make those changes real.

Dmitry

Re: fsck --full is Ok, but clones are not, "missing commits"?!

[Johannes Sixt]

Brian Foster schrieb:
> Dmitry Potapov <dpotapov@gmail.com> correctly deduced:
>>  I suspect your original git repository has info/grafts
> 
> hi Dmitry,
> 
>  bingo!  YES, it does:
...
>  the goal is to put things into a sane state so any new
>  clones are healthy.  there's only one(?) existing clone,
>  which may or may not be(? become?) an issue.

Just move info/grafts out of the way and you *may* be all set. Don't
delete it - there might be a reason that the file exists.

However!

You might observe one or more of the following:

- Parts of the history are missing. The reason is that the grafts were
used to "splice" histories that are actually independent.

- Previously unkown (to you) parts of the history are now visible. Then
the grafts were used to hide these parts.

- The repository is broken. Then 'git repack -f -d -a' was done while the
grafts were in effect (and obviously the grafts had hidden parts of the
history, which are now no longer available).

Your best bet is to run 'git filter-branch --tag-name-filter cat -- --all'
on *a copy* of your bare repository (with info/grafts in place). (Read the
warning about --tag-name-filter in the man page first.)

-- Hannes

Re: fsck --full is Ok, but clones are not, "missing commits"?!

[Dmitry Potapov]

On Wed, Apr 16, 2008 at 03:22:04PM +0200, Johannes Sixt wrote:
> Brian Foster schrieb:
> > Dmitry Potapov <dpotapov@gmail.com> correctly deduced:
> >>  I suspect your original git repository has info/grafts
> > 
> > hi Dmitry,
> > 
> >  bingo!  YES, it does:
> ...
> >  the goal is to put things into a sane state so any new
> >  clones are healthy.  there's only one(?) existing clone,
> >  which may or may not be(? become?) an issue.
> 
> Just move info/grafts out of the way and you *may* be all set. Don't
> delete it - there might be a reason that the file exists.

I believe that parts of history hidden by info/grafts are removed now,
otherwise git clone would not have problems to clone this repository.
So the best option now is only to turn the "fake" history to the real
one using git filter-branch as you wrote.


> - The repository is broken. Then 'git repack -f -d -a' was done while the
> grafts were in effect (and obviously the grafts had hidden parts of the
> history, which are now no longer available).

Actually, 'git repack -f -d -a' does not remove hidden parts of the
history (because 'git prune-packed' does not remove hidden parts), but
'git prune' does remove hidden parts. I believe 'git prune' relies on
'git-fsck --unreachable', which outputs hidden parts as unreachable.
Perhaps, git-fsck should be corrected, so it will not treat hidden as
unreachable, because anything what is hidden is still reachable for
anyone who is trying to clone the repository.

Dmitry

Re: fsck --full is Ok, but clones are not, "missing commits"?!

[Johannes Sixt]

Dmitry Potapov schrieb:
> On Wed, Apr 16, 2008 at 03:22:04PM +0200, Johannes Sixt wrote:
>> Brian Foster schrieb:
>>> Dmitry Potapov <dpotapov@gmail.com> correctly deduced:
>>>>  I suspect your original git repository has info/grafts
>>> hi Dmitry,
>>>
>>>  bingo!  YES, it does:
>> ...
>>>  the goal is to put things into a sane state so any new
>>>  clones are healthy.  there's only one(?) existing clone,
>>>  which may or may not be(? become?) an issue.
>> Just move info/grafts out of the way and you *may* be all set. Don't
>> delete it - there might be a reason that the file exists.
> 
> I believe that parts of history hidden by info/grafts are removed now,
> otherwise git clone would not have problems to clone this repository.

No. The reason why git clone'd repositories have problems is that
git-upload-pack sends a pack that does not contain the hidden objects; but
since the cloned repository doesn't have the info/grafts, it tries to look
up the hidden, now missing, objects, and fails.

>> - The repository is broken. Then 'git repack -f -d -a' was done while the
>> grafts were in effect (and obviously the grafts had hidden parts of the
>> history, which are now no longer available).
> 
> Actually, 'git repack -f -d -a' does not remove hidden parts of the
> history (because 'git prune-packed' does not remove hidden parts), but
> 'git prune' does remove hidden parts. I believe 'git prune' relies on
> 'git-fsck --unreachable', which outputs hidden parts as unreachable.
> Perhaps, git-fsck should be corrected, so it will not treat hidden as
> unreachable, because anything what is hidden is still reachable for
> anyone who is trying to clone the repository.

Actually, no. The trouble is that *all* tools obey grafts. Hence,
git-repack -f -d -a will remove the hidden objects. But a subsequent fsck
won't notice, because it *also* obeys the grafts. git prune will remove
hidden objects only as long as they are loose; if they are already packed,
then only a repack -f will remove them.

Here's a thread with a proposal by Linus how to improve the situation, but
nobody came along and implemented it:

http://thread.gmane.org/gmane.comp.version-control.git/37744/focus=37908

-- Hannes

Re: fsck --full is Ok, but clones are not, "missing commits"?!

[Dmitry Potapov]

On Wed, Apr 16, 2008 at 04:56:08PM +0200, Johannes Sixt wrote:
> 
> No. The reason why git clone'd repositories have problems is that
> git-upload-pack sends a pack that does not contain the hidden objects; but
> since the cloned repository doesn't have the info/grafts, it tries to look
> up the hidden, now missing, objects, and fails.

In my experiment, there was no problem with hidden commits and objects
as long as they presented in the original repository. Maybe, it is
because I cloned it as locally using:

   git clone PATH-TO-ORIG-REPO PATH-TO-DEST-REPO

Maybe things would be as you say if I used git://

> 
> Actually, no. The trouble is that *all* tools obey grafts. Hence,
> git-repack -f -d -a will remove the hidden objects. But a subsequent fsck
> won't notice, because it *also* obeys the grafts. git prune will remove
> hidden objects only as long as they are loose; if they are already packed,
> then only a repack -f will remove them.

I use git 1.5.5 and I have a script that creates are repo with a hidden
commit and then it cleans the reflog cleaned (as it would be happen after
expiration of gc.reflogExpire). Then I run 'git-repack -f -d -a' and the
hidden commit still presents, but when I run 'git prune' then the hidden
object disappear. You can try it for yourself. Here is the script, I used:

===
#!/bin/sh

set -e

mkdir t5
cd t5
git init

echo 1 > foo
git add foo
git commit -m 'add foo'
R1=$(git rev-parse HEAD)

echo 2 >> foo
git commit -m 'edit foo' -a
R2=$(git rev-parse HEAD)

echo 3 >> foo
git commit -m 'edit foo again' -a
R3=$(git rev-parse HEAD)

set -x
git log
echo "$R3 $R1" > .git/info/grafts
git log
git-fsck --unreachable
: > .git/logs/HEAD
: > .git/logs/refs/heads/master
git-fsck --unreachable
git gc
git-fsck --unreachable


git-repack -f -d -a
git-fsck --unreachable

git prune
git-fsck --unreachable
===

Dmitry

Re: fsck --full is Ok, but clones are not, "missing commits"?!

[Jakub Narebski]

Dmitry Potapov wrote:

> On Wed, Apr 16, 2008 at 04:56:08PM +0200, Johannes Sixt wrote:
>> 
>> No. The reason why git clone'd repositories have problems is that
>> git-upload-pack sends a pack that does not contain the hidden objects;
but
>> since the cloned repository doesn't have the info/grafts, it tries to
look
>> up the hidden, now missing, objects, and fails.
> 
> In my experiment, there was no problem with hidden commits and objects
> as long as they presented in the original repository. Maybe, it is
> because I cloned it as locally using:
> 
>    git clone PATH-TO-ORIG-REPO PATH-TO-DEST-REPO
> 
> Maybe things would be as you say if I used git://

file:// should be enough.

-- 
Jakub Narebski
Warsaw, Poland
ShadeHawk on #git

Re: fsck --full is Ok, but clones are not, "missing commits"?!

[Johannes Sixt]

Dmitry Potapov schrieb:
> On Wed, Apr 16, 2008 at 04:56:08PM +0200, Johannes Sixt wrote:
>> Actually, no. The trouble is that *all* tools obey grafts. Hence,
>> git-repack -f -d -a will remove the hidden objects. But a subsequent fsck
>> won't notice, because it *also* obeys the grafts. git prune will remove
>> hidden objects only as long as they are loose; if they are already packed,
>> then only a repack -f will remove them.
> 
> I use git 1.5.5 and I have a script that creates are repo with a hidden
> commit and then it cleans the reflog cleaned (as it would be happen after
> expiration of gc.reflogExpire). Then I run 'git-repack -f -d -a' and the
> hidden commit still presents, but when I run 'git prune' then the hidden
> object disappear. You can try it for yourself. Here is the script, I used:

And the reason for this is...

> 
> ===
> #!/bin/sh
> 
> set -e
> 
> mkdir t5
> cd t5
> git init
> 
> echo 1 > foo
> git add foo
> git commit -m 'add foo'
> R1=$(git rev-parse HEAD)
> 
> echo 2 >> foo
> git commit -m 'edit foo' -a
> R2=$(git rev-parse HEAD)
> 
> echo 3 >> foo
> git commit -m 'edit foo again' -a
> R3=$(git rev-parse HEAD)
> 
> set -x
> git log
> echo "$R3 $R1" > .git/info/grafts
> git log
> git-fsck --unreachable
> : > .git/logs/HEAD
> : > .git/logs/refs/heads/master
> git-fsck --unreachable
> git gc

... at this point the hidden commit still exists as loose objects (because
the repack inside gc didn't see the commit), and therefore...

> git-fsck --unreachable
> 
> 
> git-repack -f -d -a

... this does not make a difference at all.

> git-fsck --unreachable
> 
> git prune

Here you remove the loose objects (because prune doesn't see the commit,
either), and by doing so you corrupt the repository.

> git-fsck --unreachable
> ===

A repack -a -d makes a difference if info/grafts is installed *after* the
first repack/gc.

-- Hannes

Re: fsck --full is Ok, but clones are not, "missing commits"?!

[Johannes Sixt]

Brian FOSTER schrieb:
> Johannes Sixt kindly replied:
>> [ ... ]
>> For me, the unusable [ `gitk' ] scrollbar alone would be reason enough
>> to truncate the history.
>> [ ... ]  I have gitk running all the time.  So, yes, it is "important."
>> But I run it basically as 'gitk --all --not origin' and press F5 frequently.
>> With this set of arguments the scrollbar remains usable, and performance
>> is not an issue, even on Windows.
> 
> Hannes,
> 
>  May I ask why you are running `gitk' all the time?
>  Is this on some sort of a Linux kernel repository or something else?

Something else (~9000 commits, most of them imported from CVS).

I have gitk running all the time because I want a quick way to see "where
I am". That I get by pressing F5 (instead of typing in a longish command
to fire up gitk) and the mentioned argument list.

Also, quite a lot of the work I do is to polish a topic branch using 'git
rebase -i'. By browsing through the commits, I quickly get an overview
whether the branch is fine or needs more work.

>  Am I correct in interpreting `--all --not origin' as meaning "every
>  branch except those in origin/*" (i.e., all branches except the
>  remote-tracking ones)?

Not quite. It means the same as for 'git log': The commits from all
branches *but not* the commits from branch 'origin' (which is an alias for
origin/master, in my case, that's what is "published"). This is my way to
reduce the number of commits that are displayed and it amounts to ~300
commits.

-- Hannes

Re: fsck --full is Ok, but clones are not, "missing commits"?!

[Brian Foster]

Brandon Casey <casey@nrlssc.navy.mil> justifiably complains:
> p.s. It's more polite to keep the people you are responding
>      to in the cc list.  Take a look at how Hannes did it
>      (the email to which this one is replying).  Usually
>      it's enough to select 'Reply All'.

Brandon et al.,

 oops!  sorry, my bad.  apologies.  I'm well aware of
 this point.  if you look at my _other_ replies, you
 should see that they are Ok.

 the problem is the MTA at work insists on adding HTML
 "if you read this you are a criminal" b*s* attachments
 (at least 3 bads!).  the posts are (rightly) binned by
 majordomo.  so I've been posting from my gmail account
 (mostly), which is not intended for my work.  that's
 my private life, @&*%$(^!   so the posts go to my work
 account, and hence I must manually transfer data such
 as the CC's:

	list/cc ---> work --(fwd/reply, no cc)--> \
	   gmail --(compose reply, add cc)--> list+cc

 this is also why the threading is (probably) broken,
 and may be the reason for the "Re: Re: ..." nonsense
 (albeit I'm not sure just how that happens?), and/or
 sometimes replying to the "wrong" e-address.   ;-(

 I simply forgot.  sorry!  (this reply's from home,
 so I don't have at-hand the data you asked for.
 tomorrow.)  and yes, I know this reply is OT & long.

cheers!
	-blf-
-- 
“How many surrealists does it take to    |  Brian Foster
 change a lightbulb?  Three.  One calms  |  somewhere in south of France
 the warthog, and two fill the bathtub   |     Stop E$$o (ExxonMobil)!
 with brightly-coloured machine tools.”  |       http://www.stopesso.com

Re: fsck --full is Ok, but clones are not, "missing commits"?!

[Brian Foster]

Johannes Sixt <j.sixt@viscovery.net> helpfully observes:
>  Brian Foster schrieb:
>  > Dmitry Potapov <dpotapov@gmail.com> correctly deduced:
>  >>  I suspect your original git repository has info/grafts
>  >  bingo!  YES, it does:
>[ ... ]
>  >  the goal is to put things into a sane state so any new
>  >  clones are healthy.  there's only one(?) existing clone,
>  >  which may or may not be(? become?) an issue.
>
>  Just move info/grafts out of the way and you *may* be all set. Don't
>  delete it - there might be a reason that the file exists.

Hannes & Dmitry,

 thanks for the suggestions.  I've made (and carefully protected)
 multiple backups, and am doing all my experiments on copies, and
 have taken other protective steps, so no worries about accidents.

 moving  info/grafts  out of the way causes `fsck --full' to issue
 the same complaints as it does for clones.  conversely, copying
 info/grafts  to a clone makes `fsck --full' happy.  furthermore,
 close examination of that one pre-existing clone shows it has the
 identical  info/grafts  as the bare repository.  so that seems to
 have been what was previously being done (I'm now attempting to
 confirm this, and also to find out WHY).

 however, no joy at all with the suggestion:

>  Your best bet is to run 'git filter-branch --tag-name-filter cat -- --all'
>  on *a copy* of your bare repository (with info/grafts in place).  [ ... ]

 at toplevel of (a copy of) the bare repository, with  info/grafts
 in-place, and `fsck --full' happy:

	$ git version
	1.5.3
	$ git filter-branch --tag-name-filter cat -- --all   # at bare toplevel
	fatal: Not a git repository: '.git'
	You need to run this command from the toplevel of the working tree.
	$

 at toplevel of a (not-bare) clone, with  info/grafts  in-place,
 and a happy `fsck -full' (same machine so same git version):

	$ git filter-branch --tag-name-filter cat -- --all  # at not-bare toplevel
	Rewrite 7df30811617517bc4d5ec7c190a435667228320c (168/168)
	Ref 'refs/heads/master' was rewritten
	Ref 'refs/remotes/origin/HEAD' was rewritten
	WARNING: Ref 'refs/remotes/origin/master' is unchanged
	Ref 'refs/tags/linux-2.6.15' was rewritten
	error: Ref refs/tags/linux-2.6.15 is at \
		26a33413c95dfda6c70ca4a83da49cddb7b236b9 but expected \
		2dcaaf2decd31ac9a21d616604c0a7c1fa65d5a4
	fatal: refs/tags/linux-2.6.15: cannot lock the ref
	Could not rewrite refs/tags/linux-2.6.15
	$

 in the later case, the  info/grafts  still existed at the end,
 and `fsck --full' was happy.  (I suppose this isn't surprising
 since the filter-branch was not happy.)   moving  info/grafts
 out of the way caused the same complaints from `fsck --full'.

 the missing commits are all(? mostly?) similar to this example:

	$ git remote
	origin
	$ git remote show origin
	* remote origin
	  URL: git://git.kernel.org/pub/scm/linux/kernel/git/ralf/linux.git
        [ ... ]
	$ git show dff364d8da
	commit dff364d8da15be0b856a174062fb785acb1c363e
	Merge: bc59a40... 427abfa...
	Author: Ralf Baechle <ralf@linux-mips.org>
	Date:   Sun Jun 18 03:42:49 2006 +0100
	
	    Merge tag 'v2.6.17' of
git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux-2.6
	
	$

 as such, is there some way of adding them back to the bare
 repository (if that even makes sense?), or whatever?  (i.e.,
 that have not been lost, is it possible to take advantage
 of that fact?)

 also (sorry!), does anyone recognise the development process
 that apparently was used?  (the one pre-existing clone has
 few-to-no clews, since it was used for some fairly trivial
 local development, not for the "merging" (if I can call it
 that) with linux-mips repository.)

cheers!
	-blf-

-- 
"How many surrealists does it take to    |  Brian Foster
 change a lightbulb?  Three.  One calms  |  somewhere in south of France
 the warthog, and two fill the bathtub   |     Stop E$$o (ExxonMobil)!
 with brightly-coloured machine tools."  |       http://www.stopesso.com

Re: fsck --full is Ok, but clones are not, "missing commits"?!

[Johannes Sixt]

Brian Foster schrieb:
>  at toplevel of a (not-bare) clone, with  info/grafts  in-place,
>  and a happy `fsck -full' (same machine so same git version):
> 
> 	$ git filter-branch --tag-name-filter cat -- --all  # at not-bare toplevel
> 	Rewrite 7df30811617517bc4d5ec7c190a435667228320c (168/168)
> 	Ref 'refs/heads/master' was rewritten
> 	Ref 'refs/remotes/origin/HEAD' was rewritten
> 	WARNING: Ref 'refs/remotes/origin/master' is unchanged
> 	Ref 'refs/tags/linux-2.6.15' was rewritten
> 	error: Ref refs/tags/linux-2.6.15 is at \
> 		26a33413c95dfda6c70ca4a83da49cddb7b236b9 but expected \
> 		2dcaaf2decd31ac9a21d616604c0a7c1fa65d5a4
> 	fatal: refs/tags/linux-2.6.15: cannot lock the ref
> 	Could not rewrite refs/tags/linux-2.6.15
> 	$

Actually, I don't know how to overcome this situation; maybe forget about
the tags and remove the '--tag-name-filter cat' part. They wouldn't have
been rewritten correctly anyway (annotated tags loose the message and
become unannotated).

>  as such, is there some way of adding them back to the bare
>  repository (if that even makes sense?), or whatever?  (i.e.,
>  that have not been lost, is it possible to take advantage
>  of that fact?)
> 
>  also (sorry!), does anyone recognise the development process
>  that apparently was used?  (the one pre-existing clone has
>  few-to-no clews, since it was used for some fairly trivial
>  local development, not for the "merging" (if I can call it
>  that) with linux-mips repository.)

In this case you might be able to salvage missing objects by cloning
linux-mips. Just copy the objects/pack/* from that clone into your
objects/pack, remove info/grafts, and maybe things "just work"?

-- Hannes

Re: fsck --full is Ok, but clones are not, "missing commits"?!

[Brandon Casey]

Johannes Sixt wrote:
> Brian Foster schrieb:
>>  at toplevel of a (not-bare) clone, with  info/grafts  in-place,
>>  and a happy `fsck -full' (same machine so same git version):
>>
>> 	$ git filter-branch --tag-name-filter cat -- --all  # at not-bare toplevel
>> 	Rewrite 7df30811617517bc4d5ec7c190a435667228320c (168/168)
>> 	Ref 'refs/heads/master' was rewritten
>> 	Ref 'refs/remotes/origin/HEAD' was rewritten
>> 	WARNING: Ref 'refs/remotes/origin/master' is unchanged
>> 	Ref 'refs/tags/linux-2.6.15' was rewritten
>> 	error: Ref refs/tags/linux-2.6.15 is at \
>> 		26a33413c95dfda6c70ca4a83da49cddb7b236b9 but expected \
>> 		2dcaaf2decd31ac9a21d616604c0a7c1fa65d5a4
>> 	fatal: refs/tags/linux-2.6.15: cannot lock the ref
>> 	Could not rewrite refs/tags/linux-2.6.15
>> 	$
> 
> Actually, I don't know how to overcome this situation; maybe forget about
> the tags and remove the '--tag-name-filter cat' part. They wouldn't have
> been rewritten correctly anyway (annotated tags loose the message and
> become unannotated).


I don't think the tag-name-filter section has been reached yet.

I think the problem is the --all option to rev-parse. This adds everything under
.git/refs to the list of heads to rewrite. Just a guess, but possibly the
linux-2.6.15 is an annotated tag, which is dereferenced to a commit (the oldsha1),
and then when update-ref is called with the newsha1 and the oldsha1, the oldsha1
doesn't match what is in refs/tags/linux-2.6.15 since that points to the sha1 of
the tag object.

Try --branches instead of --all. Or leave it blank if you just want to filter
the current branch.

Also, if you're interested in retaining the annotations from the tags, search the
list for '[PATCH v4] filter-branch.sh: support nearly proper tag name filtering'
or grab the patch from git's pu branch.

-brandon

Re: fsck --full is Ok, but clones are not, "missing commits"?!

[Brian Foster]

Hannes suggested:
> Brian Foster schrieb:
> >[ ... ] is there some way of adding [ the missing commits,
> >  all of which seem to be from linux-mips, ] back to the bare
> >  repository (if that even makes sense?), or whatever?  (i.e.,
> >  [ they ] have not been lost, [ so ] is it possible to take
> >  advantage of that fact?)   [ ... ]
>
> In this case you might be able to salvage missing objects by cloning
> linux-mips. Just copy the objects/pack/* from that clone into your
> objects/pack, remove info/grafts, and maybe things "just work"?

Hannes et al.,

 thanks for the suggestion.  bingo!  that basically worked.
 after copying linux-mips  objects/pack/*  into my goofy
 bare repository,  `fsck --full'  (after moving  grafts
 out of the way) found different issues.  it complained
 about a zillion dangling tags (yawn), plus a handful of
 dangling commits.

 the dangling tags were the linux-mips  refs/tags/*.  copying
 them into my repository's  ref/tags  fixed that.  (there were
 no name collisions with the existing tags, so this was easy.)

 the dangling commits were linux-mips  refs/remotes/origin/*,
 and again could be safely/easily copied into my repository.

 `fsck --full' is now 100% happy.  (yea!)

 the obvious missing thing (which I _think_ is easy to fix?)
 is the remote URL &tc is not in my now-not-so-goofy bare
 repository.  hence `branch -r' shows  origin/*  branches,
 but (I speculate) a `pull' (e.g.) will be confused.

 at this point in time I've not done any extensive testing
 of the seemingly-fixed repository, but things are looking
 much better.

 before trying the copying suggestion, I played some
 more with `filter-branch'.  I had no success at all.
 as one example, with `--branches' instead of `--all'
 (one of Brandon's suggestions) produced:

	$ git filter-branch --tag-name-filter cat -- --branches
	Which ref do you want to rewrite?
	$

 also, Dmitry's suggestion of cloning (with grafts still
 in-place) after `filter-branch ... --all' accomplished
 nothing obvious:  `fsck --full' of the new clone was
 still unhappy (same set of complaints).

 whilst I'm still trying to understand the rationale for
 why things were set-up(? left-in?) this weird state,
 browsing the history gives some vague hints.  however,
 the intended usage model remains opaque.

 and that will soon be my next problem:  what's a better
 (best?) usage model (for this project)?  I need to do
 some more reading here ....  ;-\

cheers!
        -blf-

-- 
"How many surrealists does it take to    |  Brian Foster
 change a lightbulb?  Three.  One calms  |  somewhere in south of France
 the warthog, and two fill the bathtub   |     Stop E$$o (ExxonMobil)!
 with brightly-coloured machine tools."  |       http://www.stopesso.com

Re: fsck --full is Ok, but clones are not, "missing commits"?!

[Brandon Casey]

Brian Foster wrote:

>  before trying the copying suggestion, I played some
>  more with `filter-branch'.  I had no success at all.
>  as one example, with `--branches' instead of `--all'
>  (one of Brandon's suggestions) produced:
> 
> 	$ git filter-branch --tag-name-filter cat -- --branches
> 	Which ref do you want to rewrite?
> 	$

That's strange. Are there any branches in this repository? Yes, looking at
your earlier email:

  at toplevel of a (not-bare) clone, with  info/grafts  in-place,
  and a happy `fsck -full' (same machine so same git version):

	$ git filter-branch --tag-name-filter cat -- --all  # at not-bare toplevel
	Rewrite 7df30811617517bc4d5ec7c190a435667228320c (168/168)
	Ref 'refs/heads/master' was rewritten
	Ref 'refs/remotes/origin/HEAD' was rewritten
  [snip]

Looks like there is a single branch named 'master'.


The command within git-filter-branch.sh that is being called is

   git rev-parse --no-flags --revs-only --symbolic-full-name --default HEAD --branches

In a trim git repository for example this command returns:

   refs/heads/master
   refs/heads/next

Referring to two branches 'master' and 'next'. These are the references
that exist in .git/refs/heads/*.

I'd be interested to see what this command returns in your repo. You can compare
this output to the output you get using --all and also relate that to the
directory structure beneath .git/refs/

-brandon


p.s. It's more polite to keep the people you are responding to in the cc list.
     Take a look at how Hannes did it (the email to which this one is replying).
     Usually it's enough to select 'Reply All'.

Re: fsck --full is Ok, but clones are not, "missing commits"?!

[Brian Foster]

> Brian Foster wrote:
>  > 	$ git filter-branch --tag-name-filter cat -- --branches
>  > 	Which ref do you want to rewrite?
>  > 	$
>
> That's strange. Are there any branches in this repository?  [ ... ]
> Looks like there is a single branch named 'master'.

Brandon et al.,

 correct.  there is just the `master' branch.  it is extremely
 clean and straightforward:   O---O---O---O-- ... --O---O


> The command within git-filter-branch.sh that is being called is
>
>    git rev-parse --no-flags --revs-only --symbolic-full-name --default HEAD --branches
>
> In a trim git repository for example this command returns:
>
>    refs/heads/master
>    refs/heads/next
>
> Referring to two branches 'master' and 'next'. These are the references
> that exist in .git/refs/heads/*.
>
> I'd be interested to see what this command returns in your repo.

 below is a full example session, including, eventually,
 the above command `rev-parse ...' command, and several
 observations.  all the outputs are quite short, so I
 include most in full:

	$ git version
	git version 1.5.3
	$
	$ git clone /pub/scm/linux/kernel/git/linux-2.6-usip.git linux-2.6-usip
	Initialized empty Git repository in
/home/bfoster/usip-linux/linux-2.6-usip/.git/
	165531 blocks
	Checking 21723 files out...
	 100% (21723/21723) done
	$
	$ cd linux-2.6-usip
	$ git fsck --full
	broken link from  commit dd3f3c0636cfd50719c706b030db5473b0270add
	              to  commit 0fed9c2eb14eee47097e1d870fe8e55a6430edeb
	missing commit fb57c018d15005b60f104e57f198ff34a6035b99
	missing commit f8947cb0b5fe605e6cb5f73c89f262424b64ef3c
	missing commit 0fed9c2eb14eee47097e1d870fe8e55a6430edeb
	missing commit dff364d8da15be0b856a174062fb785acb1c363e
	broken link from  commit 8700854c41a40d333e90104971c3abbbcf082e57
	              to  commit b8b78d1c09e7009f97a1d624f4d771d7e5bd8551
	missing commit ce33a5e8c32816f38862bc41560a1d646d0803a6
	missing commit 91483db9b01b547ae9cc45c8c98b217642acb40a
	missing commit b8b78d1c09e7009f97a1d624f4d771d7e5bd8551
	broken link from  commit c87c46fe892211f8aa4fd363ccff4f667a9aaf7d
	              to  commit ce33a5e8c32816f38862bc41560a1d646d0803a6
	broken link from  commit fb5967688f7b464421cff28f266b64ad2a313a9e
	              to  commit f8947cb0b5fe605e6cb5f73c89f262424b64ef3c
	broken link from  commit e5a60f1636cceac33777bb8098a0b7a4a136a56c
	              to  commit fb57c018d15005b60f104e57f198ff34a6035b99
	broken link from  commit 2dcaaf2decd31ac9a21d616604c0a7c1fa65d5a4
	              to  commit 91483db9b01b547ae9cc45c8c98b217642acb40a
	broken link from  commit 0ff75b3afff6fb306bef221bf1823ccf5ffc568b
	              to  commit dff364d8da15be0b856a174062fb785acb1c363e
	$
	$ git filter-branch --tag-name-filter cat -- --branches
	Which ref do you want to rewrite?
	$
	$ ls -laR .git-rewrite
	.git-rewrite:
	total 16
	drwxrwxr-x  3 bfoster bfoster 4096 Apr 18 10:11 .
	drwxrwxr-x 21 bfoster bfoster 4096 Apr 18 10:11 ..
	-rw-rw-r--  1 bfoster bfoster 1864 Apr 18 10:11 backup-refs
	-rw-rw-r--  1 bfoster bfoster    0 Apr 18 10:11 heads
	drwxrwxr-x  2 bfoster bfoster 4096 Apr 18 10:11 t

	.git-rewrite/t:
	total 8
	drwxrwxr-x 2 bfoster bfoster 4096 Apr 18 10:11 .
	drwxrwxr-x 3 bfoster bfoster 4096 Apr 18 10:11 ..
	$

 watching the execution of git-filter-branch with `bash -x',
 the key point seems to be that `check-ref-formats' is being
 called with what seems to be a bogus argument:

	+ git check-ref-format 'refs/heads/master
	refs/remotes/origin/master'

 that prints nothing on stdout and exits with status 1.
 hence  .git-rewrite/heads  is empty, which is the proximate
 cause of the "Which ref ..." message.  (more on this below.)

	$ git filter-branch --tag-name-filter cat -- --branches
	.git-rewrite already exists, please remove it
	$
	$ git rev-parse --no-flags --revs-only \
		--symbolic-full-name --default HEAD --branches
	7df30811617517bc4d5ec7c190a435667228320c
	$


> You can compare this output to the output you get using --all and also
> relate that to the directory structure beneath .git/refs/

	$ git branch -a
	* master
	  origin/HEAD
	  origin/master
	$

 that seemingly bogus `check-ref-format' argument comes
 from the pipeline (where ref=master):

	git for-each-ref --format='%(refname)' | grep /"$ref"
 i.e.:
	git for-each-ref --format='%(refname)' | grep /master

 so...
	$ git for-each-ref --format='%(refname)'
	refs/heads/master
	refs/remotes/origin/HEAD
	refs/remotes/origin/master
	refs/tags/linux-2.6.15
	refs/tags/linux-usip-v2.6.15
	[ ... ]
	$

 note two names match the RE `/master', hence the bogus
 argument.  the reason for that seems to be (this is a
 guess!) that `rev-parse' is *not* being run as:

	git rev-parse --no-flags --revs-only \
		--symbolic-full-name --default HEAD --branches

 as I _think_ you are claiming it should be, but as:

	git rev-parse --revs-only --symbolic --branches

 the former outputs a list of SHA1s (see above).
 the later outputs just the name `master':

	$ git rev-parse --revs-only --symbolic --branches
	master
	$

 just for completeness, below is the output with `--all':

	$ git rev-parse --no-flags --revs-only \
		--symbolic-full-name --default HEAD --all
	7df30811617517bc4d5ec7c190a435667228320c
	7df30811617517bc4d5ec7c190a435667228320c
	7df30811617517bc4d5ec7c190a435667228320c
	26a33413c95dfda6c70ca4a83da49cddb7b236b9
	3e1e17e642cd2fface5691ab136d79f709af9a1c
	8b9e620aa7f6231e30ddf70cf38759aea7c98070
	cd220dbc113774b420c634fe1907860387883aa2
	acb1587f596e4bfe4c748be742b6bf1509fc88ed
	938ed01b502fbf659de24c4ad1da2bb60e0a3347
	d52b1d70e5f4c0509b6220070a1e585ac9f6d727
	5893ac94b533aba4a9864b0a4a3c346830ec004f
	bf0dfbc784e2734c61818aa0683b716cf82f13e2
	824a82029f8ebf2f72846490a9cd455ceaf2f5cc
	b07c23ef78c785168b579242f652b5f49749b02b
	0ae7caf477cd6a8b265497f1fb2e4273378dcfb4
	57545107a114f6f6fb0bcc9a0d655d84dc32113c
	db7e2bc1e16ab3c6611de4b84fda529e59cb378e
	df1ae0303a60ee0268012e926240720ac4d01de9
	b7e5fe765a0afd315781fb536ffcbd32a6666421
	1effee36b5784ebe5e3e7e7d34bfaf6f63be3afa
	b9a029d4c06eb18778f8907b359fe1616d1e2019
	250210f21cadd70ea5cf459b6dca990d1faecbea
	dc8843f2fd0954c941aff8b06205cd15c1f0873f
	36ee52578b10eb83d2f30e324117bfb9457e1e0e
	f8e0127a4cd3c1e42665a8e895e0c1a9dee052ce
	$
	$ cat .git/refs/heads/master
	7df30811617517bc4d5ec7c190a435667228320c
	$

 this smells like a bug in this particular version
 (1.5.3) of `git', or perhaps in the installation?
 like the repository, both are inherited, and I do
 not know the history.  ;-(

cheers!
	-blf-

-- 
"How many surrealists does it take to    |  Brian Foster
 change a lightbulb?  Three.  One calms  |  somewhere in south of France
 the warthog, and two fill the bathtub   |     Stop E$$o (ExxonMobil)!
 with brightly-coloured machine tools."  |       http://www.stopesso.com

Re: fsck --full is Ok, but clones are not, "missing commits"?!

[Johannes Sixt]

Brian Foster schrieb:
>  that seemingly bogus `check-ref-format' argument comes
>  from the pipeline (where ref=master):
> 
> 	git for-each-ref --format='%(refname)' | grep /"$ref"
>  i.e.:
> 	git for-each-ref --format='%(refname)' | grep /master
> 
>  so...
> 	$ git for-each-ref --format='%(refname)'
> 	refs/heads/master
> 	refs/remotes/origin/HEAD
> 	refs/remotes/origin/master
> 	refs/tags/linux-2.6.15
> 	refs/tags/linux-usip-v2.6.15
> 	[ ... ]
> 	$
> 
>  note two names match the RE `/master', hence the bogus
>  argument.  the reason for that seems to be (this is a
>  guess!) that `rev-parse' is *not* being run as:
> 
> 	git rev-parse --no-flags --revs-only \
> 		--symbolic-full-name --default HEAD --branches
> 
>  as I _think_ you are claiming it should be, but as:
> 
> 	git rev-parse --revs-only --symbolic --branches

One of the changes since v.1.5.3 was to get rid of this bogus grep. Can
you install a newer git?

-- Hannes

Re: fsck --full is Ok, but clones are not, "missing commits"?!

[Brian Foster]

Johannes Sixt <j.sixt@viscovery.net> asked:
> One of the changes since v.1.5.3 was to get rid of this bogus grep
> [ in `check-ref-format' ].  Can you install a newer git?

Hannes,

 yes!  about 10 minutes ago I was handed the keys to a
 brand new VM which is to be the company's git server.
 a very important point is it will be backed-up (unlike
 the former repository!).  git on the VM is "my" problem,
 and one of the first things I must do is install it.

 the VM is running "CentOS release 5 (Final)".  all things
 being equal, I'd prefer to install a reliable/stable
 pre-built package (RPM, I assume, since CentOS is based
 on Redhat).  I confess I do not know (have not looked)
 where to find suitable RPMs of git.

 suggestions are most welcome!

cheers!
	-blf-

-- 
"How many surrealists does it take to    |  Brian Foster
 change a lightbulb?  Three.  One calms  |  somewhere in south of France
 the warthog, and two fill the bathtub   |     Stop E$$o (ExxonMobil)!
 with brightly-coloured machine tools."  |       http://www.stopesso.com

Re: fsck --full is Ok, but clones are not, "missing commits"?!

[Dmitry Potapov]

On Wed, Apr 16, 2008 at 05:04:56PM +0200, Brian Foster wrote:
> 
>  in the later case, the  info/grafts  still existed at the end,
>  and `fsck --full' was happy.  (I suppose this isn't surprising
>  since the filter-branch was not happy.)   moving  info/grafts
>  out of the way caused the same complaints from `fsck --full'.

Do not remove info/grafts. Just clone this repository after
filter-branch and then the _newly_ cloned repository should
be fine.

Dmitry

fsck --full is Ok, but clones are not, "missing commits"?!

[Brian Foster]

   (many many apologies if this turns into a double post,
  there seems to have been problems with the 1st attempt?)

 I've recently inherited a bare git repository,
 which, as far as I can tell (I'm something of
 a newbie with git), seems Ok: `git fsck --full'
 does not report any problems.    however, any
 clones I make from it are not Ok:

	$ git-fsck --full   # clone (same command for bare repo is Ok)
	broken link from  commit dd3f3c0636cfd50719c706b030db5473b0270add
		      to  commit 0fed9c2eb14eee47097e1d870fe8e55a6430edeb
	missing commit fb57c018d15005b60f104e57f198ff34a6035b99
	missing commit f8947cb0b5fe605e6cb5f73c89f262424b64ef3c
	missing commit 0fed9c2eb14eee47097e1d870fe8e55a6430edeb
	missing commit dff364d8da15be0b856a174062fb785acb1c363e
	broken link from  commit 8700854c41a40d333e90104971c3abbbcf082e57
		      to  commit b8b78d1c09e7009f97a1d624f4d771d7e5bd8551
	missing commit ce33a5e8c32816f38862bc41560a1d646d0803a6
	missing commit 91483db9b01b547ae9cc45c8c98b217642acb40a
	missing commit b8b78d1c09e7009f97a1d624f4d771d7e5bd8551
	broken link from  commit c87c46fe892211f8aa4fd363ccff4f667a9aaf7d
		      to  commit ce33a5e8c32816f38862bc41560a1d646d0803a6
	broken link from  commit fb5967688f7b464421cff28f266b64ad2a313a9e
		      to  commit f8947cb0b5fe605e6cb5f73c89f262424b64ef3c
	broken link from  commit e5a60f1636cceac33777bb8098a0b7a4a136a56c
		      to  commit fb57c018d15005b60f104e57f198ff34a6035b99
	broken link from  commit 2dcaaf2decd31ac9a21d616604c0a7c1fa65d5a4
		      to  commit 91483db9b01b547ae9cc45c8c98b217642acb40a
	broken link from  commit 0ff75b3afff6fb306bef221bf1823ccf5ffc568b
		      to  commit dff364d8da15be0b856a174062fb785acb1c363e
	$ 

 the Ok(?) bare repository only has a `master' branch.
 it has numerous tags.  AFAIK, it has been "inactive"
 for the last c.6+ months (or longer?), except (maybe)
 for browsing, probably with gitweb.  there have been,
 as far as I can tell, no commits or other "writes"
 for the last c.6+ months.

 to my extreme disgust, it was never(?!) backed-up,
 except for one not entirely coincidental backup,
 also c.6+ months ago.  ;-(   (this mind-boggling
 silliness is being fixed ASAP!)

 this is happening regardless of whether the clone
 is made on the same machine or a different one,
 regardless of the transport, and for both `--bare'
 and not-bare clones.  nothing fancy is being done
 during the cloning: `git clone [--bare] DIR_or_URL'

 a test (tried by someone else) using that one
 backup apparently produced the same results:  the
 backup seems Ok, but clones made from it are not.

 the missing commits are linux-mips.org/kernel.org
 mergers (mostly).  apologies for being vague here,
 but I don't have my notes with me.  ;-(

 what is going on?  I'm completely baffled!  to-date,
 nothing terribly obvious/similar has shown up in any
 searches I've tried.  (and there's nothing unusual in
 /var/log/* that I can see.)

 I've also no idea how to "fix" the problem.

 explanations, suggestions, questions, &tc are all
 very welcome.  (please keep in mind my current level
 of knowledge/experience about git is very weak. ;-\ )

 this is all(?) using git 1.5.x (where the .x varies
 depending on the machine in question); the distros
 being used include ubuntu 7.10, FC-5, and probably
 others.  (I'll check the filesystem tomorrow, but
 I believe it's sane.)

cheers!
	-blf-
-- 
"How many surrealists does it take to    |  Brian Foster
 change a lightbulb?  Three.  One calms  |  somewhere in south of France
 the warthog, and two fill the bathtub   |     Stop E$$o (ExxonMobil)!
 with brightly-coloured machine tools."  |       http://www.stopesso.com

Re: fsck --full is Ok, but clones are not, "missing commits"?!

[David Kastrup]

Brian Foster <brian.foster@innova-card.com> writes:

>    (many many apologies if this turns into a double post,
>   there seems to have been problems with the 1st attempt?)
>
>  I've recently inherited a bare git repository,
>  which, as far as I can tell (I'm something of
>  a newbie with git), seems Ok: `git fsck --full'
>  does not report any problems.    however, any
>  clones I make from it are not Ok:
>
> 	$ git-fsck --full   # clone (same command for bare repo is Ok)
> 	broken link from  commit dd3f3c0636cfd50719c706b030db5473b0270add
> 		      to  commit 0fed9c2eb14eee47097e1d870fe8e55a6430edeb
> 	missing commit fb57c018d15005b60f104e57f198ff34a6035b99
> 	missing commit f8947cb0b5fe605e6cb5f73c89f262424b64ef3c
> 	missing commit 0fed9c2eb14eee47097e1d870fe8e55a6430edeb
> 	missing commit dff364d8da15be0b856a174062fb785acb1c363e

>From the git-clone manpage


	--shared, -s
	    When the repository to clone is on the local machine,
	    instead of using hard links, automatically setup
	    .git/objects/info/alternates to share the objects with
	    the source repository. The resulting repository starts
	    out without any object of its own. NOTE: this is a
	    possibly dangerous operation; do not use it unless you
	    understand what it does. If you clone your repository
	    using this option, then delete branches in the source
	    repository and then run git-gc(1) using the --prune
	    option in the source repository, it may remove objects
	    which are referenced by the cloned repository.

Did you use something like that?

-- 
David Kastrup

Re: fsck --full is Ok, but clones are not, "missing commits"?!

[Bryan Donlan]

On Wed, Apr 16, 2008 at 08:37:39AM +0200, Brian Foster wrote:
>    (many many apologies if this turns into a double post,
>   there seems to have been problems with the 1st attempt?)
> 
>  I've recently inherited a bare git repository,
>  which, as far as I can tell (I'm something of
>  a newbie with git), seems Ok: `git fsck --full'
>  does not report any problems.    however, any
>  clones I make from it are not Ok:
 
[snip]
 
>  explanations, suggestions, questions, &tc are all
>  very welcome.  (please keep in mind my current level
>  of knowledge/experience about git is very weak. ;-\ )
> 
>  this is all(?) using git 1.5.x (where the .x varies
>  depending on the machine in question); the distros
>  being used include ubuntu 7.10, FC-5, and probably
>  others.  (I'll check the filesystem tomorrow, but
>  I believe it's sane.)

Is there an info/grafts file? If the repository doesn't have sensitive
information in it, it would probably be helpful to tarball it up and
upload it somewhere, so we can take a look at things directly.

Re: fsck --full is Ok, but clones are not, "missing commits"?!

[Brian Foster]

On Monday 05 May 2008 06:25:46 Bryan Donlan asked:
> On Wed, Apr 16, 2008 at 08:37:39AM +0200, Brian Foster wrote:
> >  I've recently inherited a bare git repository,
> >  which, as far as I can tell (I'm something of
> >  a newbie with git), seems Ok: `git fsck --full'
> >  does not report any problems.    however, any
> >  clones I make from it are not Ok [ ... ]
>
> Is there an info/grafts file?  If the repository doesn't have sensitive
> information in it, it would probably be helpful to tarball it up and
> upload it somewhere, so we can take a look at things directly.

Bryan,

 Yes, the proximate cause was an info/grafts.

 The repository in question is for a port of Linux
 to the MIPS-based "secure" SoC made by the company
 I work for.  The repository was very simple:

    master:  o---o---o---o---o--...--o  HEAD

 where each `o' was a tagged version/release of the
 Linux port.  Since the chip is MIPS-based, some `o'
 were the result of merges with mips-linux baseline.
 Hence, the real history was, broadly:

       master:          o--o--o--o--o--o-----o--...--o
                       /           /  /     /
   mips-linux:  o--o--o----o------o--o--o--o--...--o

 where, of course, `mips-linux' has its own rather
 complicated history (not shown).  As implied by
 the diagrams above, grafts were used to "suppress"
 mips-linux and its history.  (Since the situation
 really was as trivial as drawn above, it was easy
 to recover:  Add the pack representing mips-linux,
 remove grafts, and ensure all the tags exist.)

 What I don't know is the root-cause, that is, WHY
 this was done.  It wasn't a disc-space issue, and
 I've no evidence it was a network-bandwidth issue,
 but there is some anecdotal evidence it was some
 sort of a CPU-cycles issue, albeit just what the
 performance hit was is unknown.

 Another possibility is the repository started life
 as CVS (ugh!) before being migrated to git.  It's
 my (vague) understanding grafts is somehow useful
 as a (temporary?) aid when doing a CVS-->git move;
 and I speculate it was found so useful/simple the
 practice simply continued.

 The developers tended to work from tarballs (not
 clones), so the cloning problem was either unknown,
 not a concern, or mis-understood.

cheers!
	-blf-

-- 
"How many surrealists does it take to   | Brian Foster
 change a lightbulb? Three. One calms   | somewhere in south of France
 the warthog, and two fill the bathtub  |   Stop E$$o (ExxonMobil)!
 with brightly-coloured machine tools." |      http://www.stopesso.com

Re: fsck --full is Ok, but clones are not, "missing commits"?!

[Johannes Sixt]

Brian Foster schrieb:
>  What I don't know is the root-cause, that is, WHY
>  this was done.  It wasn't a disc-space issue, and
>  I've no evidence it was a network-bandwidth issue,
>  but there is some anecdotal evidence it was some
>  sort of a CPU-cycles issue, albeit just what the
>  performance hit was is unknown.

How about this theory:

What happens if you fire up gitk as simple as

   $ gitk

in the history if no grafts are present? Some months ago this took ages to
complete, and even today you get a *huge* list of commits in a *short*
window; hence, the scrollbar thumb is tiny, and if you succeed to get hold
of it without a magnifying glass, it scrolls way more than a page of
commits if you move it by only one pixel.

No wonder that $user wants to have a shorter history. So $user, being
smart, truncates the history at a suitable point with a graft.

-- Hannes

Re: fsck --full is Ok, but clones are not, "missing commits"?!

[Brian Foster]

Johannes Sixt wrote:
> Brian Foster schrieb:
> >  What I don't know is the root-cause, that is, WHY
> >  this was done.  [ ... ]  there is some anecdotal
> >  evidence it was some sort of a CPU-cycles issue,
> >  albeit just what the performance hit was is unknown.
>
> How about this theory:
>
> What happens if you fire up gitk as simple as
>
>    $ gitk
>
> in the history if no grafts are present? Some months ago this took ages to
> complete, and even today you get a *huge* list of commits in a *short*
> window; hence, the scrollbar thumb is tiny, and if you succeed to get hold
> of it without a magnifying glass, it scrolls way more than a page of
> commits if you move it by only one pixel.
>
> No wonder that $user wants to have a shorter history. So $user, being
> smart, truncates the history at a suitable point with a graft.

Hannes,

 Unfortunately, I cannot fire up `gitk' in the exact
 same configuration anymore (that server machine is now
 being used for other purposes, albeit I'm supposed to
 get the hard disc).  The git on the now-vanished server
 was v1.5.3, but that's probably not relevant, since the
 repository must have been created with a much older git
 (it goes back multiple years).

 All the (now-)installed gits I've seen are 1.5.<something>.
 I do not see any noticeable performance issue with 1.5.2.5
 (nor with 1.5.5)?  The scrollbar is, as you say, unusable.

 But how important is `gitk'?  Is it something that'd be
 used frequently enough for the formerly-poor performance
 to be such an issue that creating and maintaining such a
 "truncated" repository is worthwhile?

 It's an interesting and plausible hypothesis, but (in
 the absence of any actual evidence) I'd be more inclined
 to buy it if there was some frequent/critical operation
 where poor performance clearly matters.

cheers!
	-blf-

Re: fsck --full is Ok, but clones are not, "missing commits"?!

[Johannes Sixt]

Brian Foster schrieb:
> Johannes Sixt wrote:
>> What happens if you fire up gitk as simple as
>>
>>    $ gitk
>>
>> in the history if no grafts are present? Some months ago this took ages to
>> complete, and even today you get a *huge* list of commits in a *short*
>> window; hence, the scrollbar thumb is tiny, and if you succeed to get hold
>> of it without a magnifying glass, it scrolls way more than a page of
>> commits if you move it by only one pixel.
>>
>> No wonder that $user wants to have a shorter history. So $user, being
>> smart, truncates the history at a suitable point with a graft.
> 
> Hannes,
> 
>  Unfortunately, I cannot fire up `gitk' in the exact
>  same configuration anymore (that server machine is now
>  being used for other purposes, albeit I'm supposed to
>  get the hard disc).  The git on the now-vanished server
>  was v1.5.3, but that's probably not relevant, since the
>  repository must have been created with a much older git
>  (it goes back multiple years).
> 
>  All the (now-)installed gits I've seen are 1.5.<something>.
>  I do not see any noticeable performance issue with 1.5.2.5
>  (nor with 1.5.5)?  The scrollbar is, as you say, unusable.

For me, the unusable scrollbar alone would be reason enough to truncate
the history. Once it is truncated, performance is no longer an issue
(whether or not it was an issue in the first place).

>  But how important is `gitk'?  Is it something that'd be
>  used frequently enough for the formerly-poor performance
>  to be such an issue that creating and maintaining such a
>  "truncated" repository is worthwhile?

Well, I have gitk running all the time. So, yes, it is "important." But I
 run it basically as 'gitk --all --not origin' and press F5 frequently.
With this set of arguments the scrollbar remains usable, and performance
is not an issue, even on Windows.

-- Hannes