fetch and bundle don't work in (semi-)broken repo

fetch and bundle don't work in (semi-)broken repo

[Uwe Kleine-König]

Hi,

I have a repo that got broken somehow (don't know the exact details,
probably because it is shared with another repo and I rewrote history).
Now I want to fetch one branch to a different repo (that happens to be
the alternative to the first one, but I think this is unrelated.):

	ukl@hostname:~/path1/linux-2.6$ git fetch ~/path2/linux-2.6 sectionmismatches
	remote: Counting objects: 118, done.
	remote: error: unable to find 40aaeb204dc04d3cf15c060133f65538b43b13b0
	remote: Compressing objects: 100% (83/83), done.
	remote: fatal: unable to read 40aaeb204dc04d3cf15c060133f65538b43b13b0
	error: git upload-pack: git-pack-objects died with error.
	fatal: git upload-pack: aborting due to possible repository corruption on the remote side.
	remote: aborting due to possible repository corruption on the remote side.
	fatal: protocol error: bad pack header

I don't know what 40aaeb204dc04d3cf15c060133f65538b43b13b0 is, but I
think it's not necessary for the sectionmismatches branch:

	ukl@hostname:~/path2/linux-2.6$ git format-patch linus/master..sectionmismatches
	0001-wip-enable-DEBUG_SECTION_MISMATCH.patch
	0002-ARM-sa1111-move-__sa1111_probe-to-.devinit.text.patch
	0003-ARM-omap1-nokia770-mark-some-functions-__init.patch
	0004-ARM-omap-fb-move-omap_init_fb-to-.init.text.patch
	0005-ARM-omap-fb-move-omapfb_reserve_sram-to-.init.text.patch
	0006-ARM-omap-fb-move-get_fbmem_region-to-.init.text.patch
	0007-ARM-omap-move-omap_get_config-et-al.-to-.init.text.patch
	0008-wip-ARM-omap-move-omap_board_config_kernel-to-.init..patch
	0009-ARM-omap-ams-delta-move-config-to-.init.data.patch
	0010-MTD-pxa2xx-move-pxa2xx_flash_probe-to-.devinit.text.patch
	0011-VIDEO-sa1100fb-register-driver-using-platform_driver.patch
	0012-ARM-s3c64xx-don-t-put-smartq_bl_init-in-.init.text.patch
	0013-ARM-s3c64xx-don-t-put-smartq7_leds-in-.init.data.patch
	0014-ARM-s3c64xx-don-t-put-smartq5_leds-in-.init.data.patch
	0015-ARM-nomadik-move-nmk_gpio_probe-to-.devinit.text.patch

and linus/master is contained in ~/path1/linux-2.6, too.

Bundling doesn't work either:

	ukl@hostname:~/path2/linux-2.6$ git bundle create tralala linus/master..sectionmismatches
	Counting objects: 118, done.
	error: unable to find 40aaeb204dc04d3cf15c060133f65538b43b13b0
	Delta compression using up to 8 threads.
	Compressing objects: 100% (83/83), done.
	fatal: unable to read 40aaeb204dc04d3cf15c060133f65538b43b13b0
	error: pack-objects died

git gc fails with a different object:

	ukl@hostname:~/path2/linux-2.6$ git gc
	error: Could not read f6b6cb2336198913371e66664f28c135df01aea5
	fatal: Failed to traverse parents of commit 76d1acb95eef413a2501a63cb7f7f4036b71ed37
	error: failed to run repack

(git gc is fine in ~/path1/linux-2.6)

	ukl@hostname:~$ git version
	git version 1.7.2.3

As a side note I have the general feeling that git recently started to
do more things than necessary (e.g. sometimes `git stash pop` takes >1s
(not mesured) between the status stuff (up to "no changes added to
commit (use ...)") and "Dropped refs/stash@{0} ($commitid)".  I wonder
what git stash is doing during that time.) I don't know if this is the
problem here, though.

Best regards
Uwe

-- 
Pengutronix e.K.                           | Uwe Kleine-König            |
Industrial Linux Solutions                 | http://www.pengutronix.de/  |

Re: fetch and bundle don't work in (semi-)broken repo

[Jonathan Nieder]

Hi,

Uwe Kleine-König wrote:

> 	ukl@hostname:~/path1/linux-2.6$ git fetch ~/path2/linux-2.6 sectionmismatches
> 	remote: Counting objects: 118, done.
> 	remote: error: unable to find 40aaeb204dc04d3cf15c060133f65538b43b13b0
> 	remote: Compressing objects: 100% (83/83), done.
> 	remote: fatal: unable to read 40aaeb204dc04d3cf15c060133f65538b43b13b0
> 	error: git upload-pack: git-pack-objects died with error.
> 	fatal: git upload-pack: aborting due to possible repository corruption on the remote side.

Sounds like alternates or workdir allowed gc to be overzealous, indeed.

Could you:

 1. Make a copy of the corrupted repo, just in case.
 2. Explode all backs with "git unpack-objects"
 3. Identify the missing object, as explained in
    Documentation/howto/recover-corrupted-blob-object.txt?

With that information, it would be easier to examine whether and how
pack-objects could be smarter about fetching the non-missing objects.

> I don't know what 40aaeb204dc04d3cf15c060133f65538b43b13b0 is, but I
> think it's not necessary for the sectionmismatches branch:
> 
> 	ukl@hostname:~/path2/linux-2.6$ git format-patch linus/master..sectionmismatches
> 	0001-wip-enable-DEBUG_SECTION_MISMATCH.patch
[...]
> and linus/master is contained in ~/path1/linux-2.6, too.

Cc-ing Nico, pack-objects wizard.

Thanks for reporting.
Jonathan

Re: fetch and bundle don't work in (semi-)broken repo

[Uwe Kleine-König]

Hi Jonathan,

On Tue, Oct 19, 2010 at 01:39:25PM -0500, Jonathan Nieder wrote:
> Uwe Kleine-König wrote:
> 
> > 	ukl@hostname:~/path1/linux-2.6$ git fetch ~/path2/linux-2.6 sectionmismatches
> > 	remote: Counting objects: 118, done.
> > 	remote: error: unable to find 40aaeb204dc04d3cf15c060133f65538b43b13b0
> > 	remote: Compressing objects: 100% (83/83), done.
> > 	remote: fatal: unable to read 40aaeb204dc04d3cf15c060133f65538b43b13b0
> > 	error: git upload-pack: git-pack-objects died with error.
> > 	fatal: git upload-pack: aborting due to possible repository corruption on the remote side.
> 
> Sounds like alternates or workdir allowed gc to be overzealous, indeed.
> 
> Could you:
> 
>  1. Make a copy of the corrupted repo, just in case.
>  2. Explode all backs with "git unpack-objects"
I did:

	mv .git/objects/pack .git/objects/pack.bak
	rm .git/objects/info/alternates
	for p in .git/objects/pack.bak/*.pack ~/path1/linux-2.6/.git/objects/pack/*.pack; do
		git unpack-objects < $p
	done


>  3. Identify the missing object, as explained in
>     Documentation/howto/recover-corrupted-blob-object.txt?

and I'm running git-fsck --full now over night as it's bedtime here.
 
> With that information, it would be easier to examine whether and how
> pack-objects could be smarter about fetching the non-missing objects.
> 
> > I don't know what 40aaeb204dc04d3cf15c060133f65538b43b13b0 is, but I
> > think it's not necessary for the sectionmismatches branch:
> > 
> > 	ukl@hostname:~/path2/linux-2.6$ git format-patch linus/master..sectionmismatches
> > 	0001-wip-enable-DEBUG_SECTION_MISMATCH.patch
> [...]
> > and linus/master is contained in ~/path1/linux-2.6, too.
> 
> Cc-ing Nico, pack-objects wizard.
> 
> Thanks for reporting.
Thanks for helping

Best regards
Uwe

-- 
Pengutronix e.K.                           | Uwe Kleine-König            |
Industrial Linux Solutions                 | http://www.pengutronix.de/  |

Re: fetch and bundle don't work in (semi-)broken repo

[Nicolas Pitre]

[-- Attachment #1: Type: TEXT/PLAIN, Size: 1832 bytes --]

On Tue, 19 Oct 2010, Uwe Kleine-König wrote:

> Hi Jonathan,
> 
> On Tue, Oct 19, 2010 at 01:39:25PM -0500, Jonathan Nieder wrote:
> > Uwe Kleine-König wrote:
> > 
> > > 	ukl@hostname:~/path1/linux-2.6$ git fetch ~/path2/linux-2.6 sectionmismatches
> > > 	remote: Counting objects: 118, done.
> > > 	remote: error: unable to find 40aaeb204dc04d3cf15c060133f65538b43b13b0
> > > 	remote: Compressing objects: 100% (83/83), done.
> > > 	remote: fatal: unable to read 40aaeb204dc04d3cf15c060133f65538b43b13b0
> > > 	error: git upload-pack: git-pack-objects died with error.
> > > 	fatal: git upload-pack: aborting due to possible repository corruption on the remote side.
> > 
> > Sounds like alternates or workdir allowed gc to be overzealous, indeed.
> > 
> > Could you:
> > 
> >  1. Make a copy of the corrupted repo, just in case.
> >  2. Explode all backs with "git unpack-objects"

That's usually unnecessary.  If the pack itself is corrupted, trying to 
unpack it all won't help.

> I did:
> 
> 	mv .git/objects/pack .git/objects/pack.bak
> 	rm .git/objects/info/alternates
> 	for p in .git/objects/pack.bak/*.pack ~/path1/linux-2.6/.git/objects/pack/*.pack; do
> 		git unpack-objects < $p
> 	done

Ouch!  You will end up with a multi-gigabyte repository, which will be 
abominably slow.

> >  3. Identify the missing object, as explained in
> >     Documentation/howto/recover-corrupted-blob-object.txt?

That's useful when you have only one corrupted object and you want to 
recreate it from raw material.  But ideally you should simply find a 
pack that contains the problematic object in another repository and copy 
it with its index 
file into the broken repository.

> and I'm running git-fsck --full now over night as it's bedtime here.

Given that you exploded your repo into loose objects, it'll take _time_.


Nicolas

Re: fetch and bundle don't work in (semi-)broken repo

[Jonathan Nieder]

Hi,

[out of order for convenience]
Nicolas Pitre wrote:
> On Tue, 19 Oct 2010, Uwe Kleine-König wrote:

>> and I'm running git-fsck --full now over night as it's bedtime here.
>
> Given that you exploded your repo into loose objects, it'll take _time_.

Yep, I gave bad advice. :(  Especially because I forgot that a fsck
would be useful at all.

Better advice would be:

 1. Use "git rev-list --objects" to find out what 40aaeb204dc was.

And if that doesn't work:

 2. Run "git fsck", with packs intact.  This will take a while.  The
    result would include a list of missing objects (like 40aaeb204dc),
    and, most importantly, their type.

Following howto/recover-corrupted-blob-object.txt would be useful for
identifying a corrupt loose object, but iiuc no corrupt objects are
involved here, anyway.

>                                 But ideally you should simply find a 
> pack that contains the problematic object in another repository and copy 
> it with its index 
> file into the broken repository.

I assume the object is gone for good, but if you have it in another
repo that would be interesting, too.

To be clear: I think the important data has been recovered from the
broken repo already in the form of patches (right?) so the question
at hand is whether it would be possible to teach git to do better at
recovering automatically.  Which might depend on the nature of the
missing objects.

Ciao,
Jonathan

Re: fetch and bundle don't work in (semi-)broken repo

[Nicolas Pitre]

On Tue, 19 Oct 2010, Jonathan Nieder wrote:

> Nicolas Pitre wrote:
> 
> >                                 But ideally you should simply find a 
> > pack that contains the problematic object in another repository and 
> > copy it with its index file into the broken repository.
> 
> I assume the object is gone for good, but if you have it in another
> repo that would be interesting, too.
> 
> To be clear: I think the important data has been recovered from the
> broken repo already in the form of patches (right?) so the question
> at hand is whether it would be possible to teach git to do better at
> recovering automatically.  Which might depend on the nature of the
> missing objects.

Sure.  Given that it is possible to create a patch series, that means 
that all the important objects are still available.  Therefore Git 
should be able to produce the pack for the equivalent fetch/bundle as 
well.

So the following patch should help.  I hope that Uwe still has a copy of 
the broken repo to test this patch with.

diff --git a/builtin/pack-objects.c b/builtin/pack-objects.c
index f8eba53..691c2f1 100644
--- a/builtin/pack-objects.c
+++ b/builtin/pack-objects.c
@@ -1299,6 +1299,15 @@ static int try_delta(struct unpacked *trg, struct unpacked *src,
 		src->data = read_sha1_file(src_entry->idx.sha1, &type, &sz);
 		read_unlock();
 		if (!src->data)
+			if (src_entry->preferred_base) {
+				/* 
+				 * Those objects are not included in the
+				 * resulting pack.  Be resilient and ignore
+				 * them, in case the pack could be created
+				 * nevertheless.
+				 */
+				return 0;
+			}
 			die("object %s cannot be read",
 			    sha1_to_hex(src_entry->idx.sha1));
 		if (sz != src_size)


Nicolas

Re: fetch and bundle don't work in (semi-)broken repo

[Uwe Kleine-König]

Hello Nico,

On Tue, Oct 19, 2010 at 11:06:50PM -0400, Nicolas Pitre wrote:
> On Tue, 19 Oct 2010, Jonathan Nieder wrote:
> 
> > Nicolas Pitre wrote:
> > 
> > >                                 But ideally you should simply find a 
> > > pack that contains the problematic object in another repository and 
> > > copy it with its index file into the broken repository.
> > 
> > I assume the object is gone for good, but if you have it in another
> > repo that would be interesting, too.
> > 
> > To be clear: I think the important data has been recovered from the
> > broken repo already in the form of patches (right?) so the question
> > at hand is whether it would be possible to teach git to do better at
> > recovering automatically.  Which might depend on the nature of the
> > missing objects.
> 
> Sure.  Given that it is possible to create a patch series, that means 
> that all the important objects are still available.  Therefore Git 
> should be able to produce the pack for the equivalent fetch/bundle as 
> well.
> 
> So the following patch should help.  I hope that Uwe still has a copy of 
> the broken repo to test this patch with.
> 
> diff --git a/builtin/pack-objects.c b/builtin/pack-objects.c
> index f8eba53..691c2f1 100644
> --- a/builtin/pack-objects.c
> +++ b/builtin/pack-objects.c
> @@ -1299,6 +1299,15 @@ static int try_delta(struct unpacked *trg, struct unpacked *src,
>  		src->data = read_sha1_file(src_entry->idx.sha1, &type, &sz);
>  		read_unlock();
>  		if (!src->data)
> +			if (src_entry->preferred_base) {
> +				/* 
> +				 * Those objects are not included in the
> +				 * resulting pack.  Be resilient and ignore
> +				 * them, in case the pack could be created
> +				 * nevertheless.
> +				 */
> +				return 0;
> +			}
>  			die("object %s cannot be read",
>  			    sha1_to_hex(src_entry->idx.sha1));
>  		if (sz != src_size)
Doesn't help :-(  I added a warning(...) before your return 0, and I
don't see it.  Probably this means this is not the problematic code
path.

The output with your patch applied is:
	user@hostname:~/path/linux-2.6$ ~/gsrc/git/bin-wrappers/git bundle create tra linus/master..sectionmismatches
	Counting objects: 118, done.
	error: unable to find 40aaeb204dc04d3cf15c060133f65538b43b13b0
	Delta compression using up to 8 threads.
	fatal: object 3cf4fa25ab3d078a49e9488effaebf571fa128da cannot be read
	error: pack-objects died

If you want I can provide you the broken repo.

Best regards
Uwe

-- 
Pengutronix e.K.                           | Uwe Kleine-König            |
Industrial Linux Solutions                 | http://www.pengutronix.de/  |

Re: fetch and bundle don't work in (semi-)broken repo

[Nicolas Pitre]

[-- Attachment #1: Type: TEXT/PLAIN, Size: 119 bytes --]

On Wed, 20 Oct 2010, Uwe Kleine-König wrote:

> If you want I can provide you the broken repo.

Yes, please.


Nicolas

format-patch broken [Was: fetch and bundle don't work in (semi-)broken repo]

[Uwe Kleine-König]

[-- Attachment #1: Type: text/plain, Size: 4099 bytes --]

On Wed, Oct 20, 2010 at 01:05:26PM -0400, Nicolas Pitre wrote:
> On Wed, 20 Oct 2010, Uwe Kleine-König wrote:
> 
> > On Wed, Oct 20, 2010 at 09:38:14AM -0400, Nicolas Pitre wrote:
> > > On Wed, 20 Oct 2010, Uwe Kleine-König wrote:
> > > 
> > > > If you want I can provide you the broken repo.
> > > 
> > > Yes, please.
> 
> I'm trying to reproduce the exact error you get. But I'm missing some 
> alternates for objects, and none of the trees I have locally provide the 
> needed objects.
> 
> [nico@xanadu linux-2.6]$ ! git format-patch linus/master..sectionmismatches
> 0001-wip-enable-DEBUG_SECTION_MISMATCH.patch
> 0002-ARM-sa1111-move-__sa1111_probe-to-.devinit.text.patch
> 0003-ARM-omap1-nokia770-mark-some-functions-__init.patch
> fatal: unable to read destination tree (16edb8381f2f2dabec9cc59f4a3d8c9ead899668)
> 
> What do you have in your alternate repo?
Hmm, I just unpacked the archive in a seperate directory, removed
.git/objects/info/alternates and then git format-patch
linus/master..sectionmismatches fails in a different way:

	fatal: Invalid revision range linus/master..sectionmismatches

I guess adding a pristine copy of Linus' tree should do the trick.

[ ... some time later ... creating a fresh clone takes quite some time ... ]

No, that's not enough, I will handpick some objects from the original.

Ah, you only need 16edb8381f2f2dabec9cc59f4a3d8c9ead899668 to make
format-patch work, but still 09b3f464a50111071f7740056b98fa8f36133347 is
missing for this tree.  This doesn't hurt format-patch as it's enough
for it to know that this entry didn't change.  So format-patch needs
less information than bundle/fetch and it's OK that the former succeeds
and the latter fails.

[...]

No, that's not the (only) problem,
40aaeb204dc04d3cf15c060133f65538b43b13b0 is needed, git format-patch is
just ignorant enough and invents something different:

	username@hostname:~/path/linux-2.6$ git rev-list linus/master..sectionmismatches
	eb84720860a90769473b42215a4cb67ee5efe7a7
	2e14a5c831032fa489384763087f4a03d88607cb
	00b18e8058e98927e2e4eae32deae7e58f47467c
	1ad328f663128b5c6e6b4af1ac2da1b443dba530
	2a0e4c23a34c78891db685b2b4851705fd36d656
	089d061c26b00a5b8dbb9e70b81d36a97e1daded
	b7ce4ec88f1bdfbe49fa7ef12df8f985d705605a
	b40acb01793933cd6baaaf826f3fef6dd734f72b
	780e3d47d067b54b17bcac3794d62825e8e60422
	ce06129cf7bbf85afe4fc127afc957d36ba4e9e4
	c2172d687578e7eb037a232802a4a8c6de1b0eea
	0c23684f39714a72f54036ca2be36e8894794b66
	cea2a0668ee1a9dc3617a810954a41c7701a08e9
	2bd6ff604ac3aa4c96636dda1ad80a289205ccba
	7591700d538d08f2e8327bb439b6cb0488e13f3e

	username@hostname:~/path/linux-2.6$ git diff-tree -r 7591700d538d08f2e8327bb439b6cb0488e13f3e
	7591700d538d08f2e8327bb439b6cb0488e13f3e
	:100644 100644 1b4afd2e6ca089de0babdacc5781426ef118da5c 40aaeb204dc04d3cf15c060133f65538b43b13b0 M	lib/Kconfig.debug

	commit 7591700d538d08f2e8327bb439b6cb0488e13f3e
	Author: Uwe Kleine-König <u.kleine-koenig@pengutronix.de>
	Date:   Wed Aug 4 08:52:56 2010 +0200

	    wip: enable DEBUG_SECTION_MISMATCH

	diff --git a/lib/Kconfig.debug b/lib/Kconfig.debug
	index 1b4afd2..40aaeb2 100644
	--- a/lib/Kconfig.debug
	+++ b/lib/Kconfig.debug
	@@ -1,1157 +0,0 @@
	-
	-config PRINTK_TIME
	-	bool "Show timing information on printks"
	-	depends on PRINTK
	-	help
	-	  Selecting this option causes timing information to be
	-	  included in printk output.  This allows you to measure
	-	  the interval between kernel operations, including bootup
	-	  operations.  This is useful for identifying long delays

So format-patch assumes that lib/Kconfig.debug became empty because of
not knowing what 40aaeb204dc04d3cf15c060133f65538b43b13b0 is.

So it's not git-bundle/git-fetch that is broken, but format-patch.

I attached an archive with the two objects
16edb8381f2f2dabec9cc59f4a3d8c9ead899668 and
09b3f464a50111071f7740056b98fa8f36133347 and will try to recreate
40aaeb204dc04d3cf15c060133f65538b43b13b0 now.

Best regards
Uwe

-- 
Pengutronix e.K.                           | Uwe Kleine-König            |
Industrial Linux Solutions                 | http://www.pengutronix.de/  |

[-- Attachment #2: missing-objects.tar.bz2 --]
[-- Type: application/octet-stream, Size: 4865 bytes --]

Re: format-patch broken [Was: fetch and bundle don't work in (semi-)broken repo]

[Uwe Kleine-König]

Hello,

On Thu, Oct 21, 2010 at 09:11:06AM +0200, Uwe Kleine-König wrote:
> On Wed, Oct 20, 2010 at 01:05:26PM -0400, Nicolas Pitre wrote:
> > On Wed, 20 Oct 2010, Uwe Kleine-König wrote:
> > 
> > > On Wed, Oct 20, 2010 at 09:38:14AM -0400, Nicolas Pitre wrote:
> > > > On Wed, 20 Oct 2010, Uwe Kleine-König wrote:
> > > > 
> > > > > If you want I can provide you the broken repo.
> > > > 
> > > > Yes, please.
> > 
> > I'm trying to reproduce the exact error you get. But I'm missing some 
> > alternates for objects, and none of the trees I have locally provide the 
> > needed objects.
> > 
> > [nico@xanadu linux-2.6]$ ! git format-patch linus/master..sectionmismatches
> > 0001-wip-enable-DEBUG_SECTION_MISMATCH.patch
> > 0002-ARM-sa1111-move-__sa1111_probe-to-.devinit.text.patch
> > 0003-ARM-omap1-nokia770-mark-some-functions-__init.patch
> > fatal: unable to read destination tree (16edb8381f2f2dabec9cc59f4a3d8c9ead899668)
> > 
> > What do you have in your alternate repo?
> Hmm, I just unpacked the archive in a seperate directory, removed
> .git/objects/info/alternates and then git format-patch
> linus/master..sectionmismatches fails in a different way:
> 
> 	fatal: Invalid revision range linus/master..sectionmismatches
> 
> I guess adding a pristine copy of Linus' tree should do the trick.
> 
> [ ... some time later ... creating a fresh clone takes quite some time ... ]
> 
> No, that's not enough, I will handpick some objects from the original.
> 
> Ah, you only need 16edb8381f2f2dabec9cc59f4a3d8c9ead899668 to make
> format-patch work, but still 09b3f464a50111071f7740056b98fa8f36133347 is
> missing for this tree.  This doesn't hurt format-patch as it's enough
> for it to know that this entry didn't change.  So format-patch needs
> less information than bundle/fetch and it's OK that the former succeeds
> and the latter fails.
> 
> [...]
> 
> No, that's not the (only) problem,
> 40aaeb204dc04d3cf15c060133f65538b43b13b0 is needed, git format-patch is
> just ignorant enough and invents something different:
> 
> 	username@hostname:~/path/linux-2.6$ git rev-list linus/master..sectionmismatches
> 	eb84720860a90769473b42215a4cb67ee5efe7a7
> 	2e14a5c831032fa489384763087f4a03d88607cb
> 	00b18e8058e98927e2e4eae32deae7e58f47467c
> 	1ad328f663128b5c6e6b4af1ac2da1b443dba530
> 	2a0e4c23a34c78891db685b2b4851705fd36d656
> 	089d061c26b00a5b8dbb9e70b81d36a97e1daded
> 	b7ce4ec88f1bdfbe49fa7ef12df8f985d705605a
> 	b40acb01793933cd6baaaf826f3fef6dd734f72b
> 	780e3d47d067b54b17bcac3794d62825e8e60422
> 	ce06129cf7bbf85afe4fc127afc957d36ba4e9e4
> 	c2172d687578e7eb037a232802a4a8c6de1b0eea
> 	0c23684f39714a72f54036ca2be36e8894794b66
> 	cea2a0668ee1a9dc3617a810954a41c7701a08e9
> 	2bd6ff604ac3aa4c96636dda1ad80a289205ccba
> 	7591700d538d08f2e8327bb439b6cb0488e13f3e
> 
> 	username@hostname:~/path/linux-2.6$ git diff-tree -r 7591700d538d08f2e8327bb439b6cb0488e13f3e
> 	7591700d538d08f2e8327bb439b6cb0488e13f3e
> 	:100644 100644 1b4afd2e6ca089de0babdacc5781426ef118da5c 40aaeb204dc04d3cf15c060133f65538b43b13b0 M	lib/Kconfig.debug
> 
> 	commit 7591700d538d08f2e8327bb439b6cb0488e13f3e
> 	Author: Uwe Kleine-König <u.kleine-koenig@pengutronix.de>
> 	Date:   Wed Aug 4 08:52:56 2010 +0200
> 
> 	    wip: enable DEBUG_SECTION_MISMATCH
> 
> 	diff --git a/lib/Kconfig.debug b/lib/Kconfig.debug
> 	index 1b4afd2..40aaeb2 100644
> 	--- a/lib/Kconfig.debug
> 	+++ b/lib/Kconfig.debug
> 	@@ -1,1157 +0,0 @@
> 	-
> 	-config PRINTK_TIME
> 	-	bool "Show timing information on printks"
> 	-	depends on PRINTK
> 	-	help
> 	-	  Selecting this option causes timing information to be
> 	-	  included in printk output.  This allows you to measure
> 	-	  the interval between kernel operations, including bootup
> 	-	  operations.  This is useful for identifying long delays
> 
> So format-patch assumes that lib/Kconfig.debug became empty because of
> not knowing what 40aaeb204dc04d3cf15c060133f65538b43b13b0 is.
> 
> So it's not git-bundle/git-fetch that is broken, but format-patch.
> 
> I attached an archive with the two objects
> 16edb8381f2f2dabec9cc59f4a3d8c9ead899668 and
> 09b3f464a50111071f7740056b98fa8f36133347 and will try to recreate
> 40aaeb204dc04d3cf15c060133f65538b43b13b0 now.
That was easy:

	git hash-object -w lib/Kconfig.debug

Now git bundle works again.

Nicolas: I forgot to say, that I needed a pristine clone of Linus' repo
as alternate to get it running.

Best regards
Uwe

-- 
Pengutronix e.K.                           | Uwe Kleine-König            |
Industrial Linux Solutions                 | http://www.pengutronix.de/  |

Re: format-patch broken [Was: fetch and bundle don't work in (semi-)broken repo]

[Nicolas Pitre]

[-- Attachment #1: Type: TEXT/PLAIN, Size: 3264 bytes --]

On Thu, 21 Oct 2010, Uwe Kleine-König wrote:

> Ah, you only need 16edb8381f2f2dabec9cc59f4a3d8c9ead899668 to make
> format-patch work, but still 09b3f464a50111071f7740056b98fa8f36133347 is
> missing for this tree.  This doesn't hurt format-patch as it's enough
> for it to know that this entry didn't change.  So format-patch needs
> less information than bundle/fetch and it's OK that the former succeeds
> and the latter fails.

No it is not.  In theory both format-patch and fetch/bundle should 
require the exact same information.

> [...]
> 
> No, that's not the (only) problem,
> 40aaeb204dc04d3cf15c060133f65538b43b13b0 is needed, git format-patch is
> just ignorant enough and invents something different:
> 
> 	username@hostname:~/path/linux-2.6$ git rev-list linus/master..sectionmismatches
> 	eb84720860a90769473b42215a4cb67ee5efe7a7
> 	2e14a5c831032fa489384763087f4a03d88607cb
> 	00b18e8058e98927e2e4eae32deae7e58f47467c
> 	1ad328f663128b5c6e6b4af1ac2da1b443dba530
> 	2a0e4c23a34c78891db685b2b4851705fd36d656
> 	089d061c26b00a5b8dbb9e70b81d36a97e1daded
> 	b7ce4ec88f1bdfbe49fa7ef12df8f985d705605a
> 	b40acb01793933cd6baaaf826f3fef6dd734f72b
> 	780e3d47d067b54b17bcac3794d62825e8e60422
> 	ce06129cf7bbf85afe4fc127afc957d36ba4e9e4
> 	c2172d687578e7eb037a232802a4a8c6de1b0eea
> 	0c23684f39714a72f54036ca2be36e8894794b66
> 	cea2a0668ee1a9dc3617a810954a41c7701a08e9
> 	2bd6ff604ac3aa4c96636dda1ad80a289205ccba
> 	7591700d538d08f2e8327bb439b6cb0488e13f3e
> 
> 	username@hostname:~/path/linux-2.6$ git diff-tree -r 7591700d538d08f2e8327bb439b6cb0488e13f3e
> 	7591700d538d08f2e8327bb439b6cb0488e13f3e
> 	:100644 100644 1b4afd2e6ca089de0babdacc5781426ef118da5c 40aaeb204dc04d3cf15c060133f65538b43b13b0 M	lib/Kconfig.debug
> 
> 	commit 7591700d538d08f2e8327bb439b6cb0488e13f3e
> 	Author: Uwe Kleine-König <u.kleine-koenig@pengutronix.de>
> 	Date:   Wed Aug 4 08:52:56 2010 +0200
> 
> 	    wip: enable DEBUG_SECTION_MISMATCH
> 
> 	diff --git a/lib/Kconfig.debug b/lib/Kconfig.debug
> 	index 1b4afd2..40aaeb2 100644
> 	--- a/lib/Kconfig.debug
> 	+++ b/lib/Kconfig.debug
> 	@@ -1,1157 +0,0 @@
> 	-
> 	-config PRINTK_TIME
> 	-	bool "Show timing information on printks"
> 	-	depends on PRINTK
> 	-	help
> 	-	  Selecting this option causes timing information to be
> 	-	  included in printk output.  This allows you to measure
> 	-	  the interval between kernel operations, including bootup
> 	-	  operations.  This is useful for identifying long delays
> 
> So format-patch assumes that lib/Kconfig.debug became empty because of
> not knowing what 40aaeb204dc04d3cf15c060133f65538b43b13b0 is.
> 
> So it's not git-bundle/git-fetch that is broken, but format-patch.

Or rather the low-level diff code.

diff --git a/diff.c b/diff.c
index 4732b32..b2839f9 100644
--- a/diff.c
+++ b/diff.c
@@ -2386,10 +2386,14 @@ int diff_populate_filespec(struct diff_filespec *s, int size_only)
 	}
 	else {
 		enum object_type type;
-		if (size_only)
+		if (size_only) {
 			type = sha1_object_info(s->sha1, &s->size);
-		else {
+			if (type < 0)
+				die("unable to read %s", sha1_to_hex(s->sha1));
+		} else {
 			s->data = read_sha1_file(s->sha1, &type, &s->size);
+			if (!s->data)
+				die("unable to read %s", sha1_to_hex(s->sha1));
 			s->should_free = 1;
 		}
 	}


Nicolas

Re: fetch and bundle don't work in (semi-)broken repo

[Uwe Kleine-König]

Hello,

On Tue, Oct 19, 2010 at 04:48:23PM -0400, Nicolas Pitre wrote:
> On Tue, 19 Oct 2010, Uwe Kleine-König wrote:
> 
> > Hi Jonathan,
> > 
> > On Tue, Oct 19, 2010 at 01:39:25PM -0500, Jonathan Nieder wrote:
> > > Uwe Kleine-König wrote:
> > > 
> > > > 	ukl@hostname:~/path1/linux-2.6$ git fetch ~/path2/linux-2.6 sectionmismatches
> > > > 	remote: Counting objects: 118, done.
> > > > 	remote: error: unable to find 40aaeb204dc04d3cf15c060133f65538b43b13b0
> > > > 	remote: Compressing objects: 100% (83/83), done.
> > > > 	remote: fatal: unable to read 40aaeb204dc04d3cf15c060133f65538b43b13b0
> > > > 	error: git upload-pack: git-pack-objects died with error.
> > > > 	fatal: git upload-pack: aborting due to possible repository corruption on the remote side.
> > > 
> > > Sounds like alternates or workdir allowed gc to be overzealous, indeed.
> > > 
> > > Could you:
> > > 
> > >  1. Make a copy of the corrupted repo, just in case.
> > >  2. Explode all backs with "git unpack-objects"
> 
> That's usually unnecessary.  If the pack itself is corrupted, trying to 
> unpack it all won't help.
> 
> > I did:
> > 
> > 	mv .git/objects/pack .git/objects/pack.bak
> > 	rm .git/objects/info/alternates
> > 	for p in .git/objects/pack.bak/*.pack ~/path1/linux-2.6/.git/objects/pack/*.pack; do
> > 		git unpack-objects < $p
> > 	done
> 
> Ouch!  You will end up with a multi-gigabyte repository, which will be 
> abominably slow.
> 
> > >  3. Identify the missing object, as explained in
> > >     Documentation/howto/recover-corrupted-blob-object.txt?
> 
> That's useful when you have only one corrupted object and you want to 
> recreate it from raw material.  But ideally you should simply find a 
> pack that contains the problematic object in another repository and copy 
> it with its index 
> file into the broken repository.
> 
> > and I'm running git-fsck --full now over night as it's bedtime here.
> 
> Given that you exploded your repo into loose objects, it'll take _time_.
Well it took 34 minutes, which is OK I guess.

I will study the output a bit now.

For the interested (all lines matching "dangling" removed):

	22:10:57 I: Started git fsck --full
	22:44:14 O: broken link from    tree 519af383e181399db929823299bbd14c04b4229a
	22:44:14 O: to    tree d58c333c44672cb933df5a353dfb63ac571964e8
	22:44:14 O: broken link from  commit e8f7f6a23979c398249a15fb71b3e52dae933fa3
	22:44:14 O: to    tree 7f22979d86cf00c8bd3487feb973353ab5a1beee
	22:44:14 O: broken link from  commit 3164f6598ae44703a89822ced9746c1876ba7fab
	22:44:14 O: to    tree 1017bb1f45b8527ee3c7cfc30288b8098bcf0915
	22:44:14 O: broken link from  commit 124dde2ea387dc9509b0a5574c6f44f7d348a65d
	22:44:14 O: to    tree e4d0ac236995847e4e1d15c6d0afb47787255703
	22:44:14 O: broken link from  commit 60deff2fffd90b217d90284295d5a910f21fe98e
	22:44:14 O: to    tree 18bb32cfd08228820f929d62e63933fe2896b424
	22:44:14 O: broken link from  commit 0b84e651b84dba73772fda15a8a66de8cc274af0
	22:44:14 O: to    tree f8939a09d73b78459381b7991423529592e66324
	22:44:14 O: broken link from  commit e0de1d3c3355f9b1e3474417f05657a1041e7c8a
	22:44:14 O: to    tree 776ad9ac45dab11f2644151a690e1035789a49b6
	22:44:14 O: broken link from  commit 76d1acb95eef413a2501a63cb7f7f4036b71ed37
	22:44:14 O: to  commit f6b6cb2336198913371e66664f28c135df01aea5
	22:44:14 O: broken link from    tree bb473ad85c260b6a1659aa2059cac23b337842e3
	22:44:14 O: to    tree e035bc14698cc3e9abfca1a174feacb25e7e262a
	22:44:14 O: broken link from    tree bb473ad85c260b6a1659aa2059cac23b337842e3
	22:44:14 O: to    tree 8908b2458c1a2c6a6db81e88d96a01aa9a89abe5
	22:44:14 O: broken link from    tree ee35b3a549f45830ed50eb1032836a71ab2b7886
	22:44:14 O: to    tree f2f33722af4b5e32ac17f914cf24cc96c6e80077
	22:44:14 O: broken link from    tree ee35b3a549f45830ed50eb1032836a71ab2b7886
	22:44:14 O: to    tree 70f0188991b8406ec6ec75a504cf50c778fc1001
	22:44:14 O: broken link from    tree 1772732da7d4751d3c0febd7b0ceee61a84702f0
	22:44:14 O: to    tree dc3c1a018fcf2797a860af0cb899fe6be1173da3
	22:44:14 O: broken link from    tree 1772732da7d4751d3c0febd7b0ceee61a84702f0
	22:44:14 O: to    tree 2ade2c0731e57fc6b1baddb6faf683847d706179
	22:44:14 O: broken link from    tree 1772732da7d4751d3c0febd7b0ceee61a84702f0
	22:44:14 O: to    tree 659f4a6280150b7ae7b8f824257ad3cbb7cd1788
	22:44:14 O: broken link from    tree f29915a03ea5ba8739aacfcbb349d40b019481ed
	22:44:14 O: to    tree b52d608d7743b1e096725d4b8d7c7769b19f6674
	22:44:15 O: missing tree 7f22979d86cf00c8bd3487feb973353ab5a1beee
	22:44:15 O: missing tree 70f0188991b8406ec6ec75a504cf50c778fc1001
	22:44:15 O: missing tree dc3c1a018fcf2797a860af0cb899fe6be1173da3
	22:44:15 O: missing tree f8939a09d73b78459381b7991423529592e66324
	22:44:15 O: missing tree e4d0ac236995847e4e1d15c6d0afb47787255703
	22:44:15 O: missing tree 2ade2c0731e57fc6b1baddb6faf683847d706179
	22:44:15 O: missing tree 8908b2458c1a2c6a6db81e88d96a01aa9a89abe5
	22:44:15 O: missing tree 18bb32cfd08228820f929d62e63933fe2896b424
	22:44:15 O: missing tree d58c333c44672cb933df5a353dfb63ac571964e8
	22:44:15 O: missing tree f2f33722af4b5e32ac17f914cf24cc96c6e80077
	22:44:16 O: missing tree 1017bb1f45b8527ee3c7cfc30288b8098bcf0915
	22:44:16 O: missing tree e035bc14698cc3e9abfca1a174feacb25e7e262a
	22:44:16 O: missing tree 659f4a6280150b7ae7b8f824257ad3cbb7cd1788
	22:44:16 O: missing commit f6b6cb2336198913371e66664f28c135df01aea5
	22:44:16 O: missing tree 776ad9ac45dab11f2644151a690e1035789a49b6
	22:44:16 O: missing tree b52d608d7743b1e096725d4b8d7c7769b19f6674
	22:44:16 O: missing blob 40aaeb204dc04d3cf15c060133f65538b43b13b0

Here is the output for the object bundle and fetch wail about, but
nothing about a tree that would need it :-(  Actually it's the only one
of the missing objects that is not described above (and it's the only
blob).

	22:44:17 I: Finished with exitcode 2

Now I'm running

	git for-each-ref | while read sha type name; do
		echo $name;
		git log --raw --full-history $sha | grep 40aaeb204dc04d3cf15c060133f65538b43b13b0;
	done

Best regards
Uwe

-- 
Pengutronix e.K.                           | Uwe Kleine-König            |
Industrial Linux Solutions                 | http://www.pengutronix.de/  |