git fetch --prune fails with "fatal: bad object"

git fetch --prune fails with "fatal: bad object"

[Curley Joe]

git fetch --prune fails with "fatal: bad object" for refs that have an invalid sha1 pointer. It would be nice if "git fetch --prune" could prune these refs, because it is a hassle to do it manually. ("git fetch --prune" only shows the first invalid ref, and you have to run "git fsck" and parse it to find the rest.) If it seems like adding this functionality to --prune is a bad idea, then how about adding an option like "--prune-invalid" ?

Thanks.

Re: git fetch --prune fails with "fatal: bad object"

[Junio C Hamano]

"Curley Joe" <m48cv7wg9w@liamekaens.com> writes:

> git fetch --prune fails with "fatal: bad object" for refs that
> have an invalid sha1 pointer. It would be nice if "git fetch
> --prune" could prune these refs, because it is a hassle to do it
> manually. ("git fetch --prune" only shows the first invalid ref,
> and you have to run "git fsck" and parse it to find the rest.) If
> it seems like adding this functionality to --prune is a bad idea,
> then how about adding an option like "--prune-invalid" ?

A question and a comment.

 - Why did the repository got into this state in the first place?
   It seems that it would be much better solution to prevent refs
   from having garbage values in them or to prevent objects that are
   necessary from going away than any "prune invalid refs" feature.

 - "fetch" still feels a wrong place to have the feature, if it is
   about fixing a local repository corruption.  You should be able
   to recover from such a broken ref even if you are only working
   locally without fetching from anybody.

If you can somehow _enumerate_ such broken refs, you could drive
update-ref to remove them.  Naïvely, an obvious place to add such a
feature might be the "for-each-ref" command that is used to list
refs with various criteria, so it might look like:

   $ git for-each-ref --format='%(refname)' --broken |
     xargs git update-ref -d

or something?

RE: git fetch --prune fails with "fatal: bad object"

[rsbecker]

On Friday, May 31, 2024 7:28 PM, Junio C Hamano wrote:
>"Curley Joe" <m48cv7wg9w@liamekaens.com> writes:
>
>> git fetch --prune fails with "fatal: bad object" for refs that have an
>> invalid sha1 pointer. It would be nice if "git fetch --prune" could
>> prune these refs, because it is a hassle to do it manually. ("git
>> fetch --prune" only shows the first invalid ref, and you have to run
>> "git fsck" and parse it to find the rest.) If it seems like adding
>> this functionality to --prune is a bad idea, then how about adding an
>> option like "--prune-invalid" ?
>
>A question and a comment.
>
> - Why did the repository got into this state in the first place?
>   It seems that it would be much better solution to prevent refs
>   from having garbage values in them or to prevent objects that are
>   necessary from going away than any "prune invalid refs" feature.

I agree. However, there are some configurations where disk write caches are enabled and require a sync or some other flush operation to force a complete write to disk. In such situations, corruptions are always possible despite the best efforts by the application.

> - "fetch" still feels a wrong place to have the feature, if it is
>   about fixing a local repository corruption.  You should be able
>   to recover from such a broken ref even if you are only working
>   locally without fetching from anybody.

I think fsck would be a better place for this.

>If you can somehow _enumerate_ such broken refs, you could drive update-ref to
>remove them.  Naïvely, an obvious place to add such a feature might be the "for-
>each-ref" command that is used to list refs with various criteria, so it might look like:
>
>   $ git for-each-ref --format='%(refname)' --broken |
>     xargs git update-ref -d
>
>or something?

Re: git fetch --prune fails with "fatal: bad object"

[Junio C Hamano]

<rsbecker@nexbridge.com> writes:

>> - Why did the repository got into this state in the first place?
>>   It seems that it would be much better solution to prevent refs
>>   from having garbage values in them or to prevent objects that are
>>   necessary from going away than any "prune invalid refs" feature.
>
> I agree. However, there are some configurations where disk write
> caches are enabled and require a sync or some other flush
> operation to force a complete write to disk. In such situations,
> corruptions are always possible despite the best efforts by the
> application.

The question was posed to see where the current "best efforts" are
still inadequate.

>> - "fetch" still feels a wrong place to have the feature, if it is
>>   about fixing a local repository corruption.  You should be able
>>   to recover from such a broken ref even if you are only working
>>   locally without fetching from anybody.
>
> I think fsck would be a better place for this.
>
>>If you can somehow _enumerate_ such broken refs, you could drive update-ref to
>>remove them.

Interesting.  "git fsck" certainly can be used to help you find out
about them.  In a throw-away repository, after manually crafting
some "broken refs" (because update-ref will refuse to create a ref
pointing at a missing object):

    $ git for-each-ref
    9e830ad6c4f43159cef50cb1c2205f513c79bc8b commit refs/heads/master
    $ echo 9e830ad6c4f43159cef50cb1c2205f513c79bc8a >.git/refs/heads/broken-missing
    $ git rev-parse master: >.git/refs/heads/broken-tree
    $ git rev-parse "master:foo /baz" >.git/refs/heads/broken-blob

running "git fsck" does tell you about them, ...

    $ git fsck
    Checking object directories: 100% (256/256), done.
    error: refs/heads/broken-blob: not a commit
    error: refs/heads/broken-missing: invalid sha1 pointer 9e830ad6c4f43159cef50cb1c2205f513c79bc8a
    error: refs/heads/broken-tree: not a commit

... and using the information, you can

    $ for r in refs/heads/broken-{blob,missing,tree}
      do git update-ref -d "$r"
      done

to unbreak the repository.

Re: git fetch --prune fails with "fatal: bad object"

[Jeff King]

On Sat, Jun 01, 2024 at 08:53:43AM -0700, Junio C Hamano wrote:

> Interesting.  "git fsck" certainly can be used to help you find out
> about them.  In a throw-away repository, after manually crafting
> some "broken refs" (because update-ref will refuse to create a ref
> pointing at a missing object):
> 
>     $ git for-each-ref
>     9e830ad6c4f43159cef50cb1c2205f513c79bc8b commit refs/heads/master
>     $ echo 9e830ad6c4f43159cef50cb1c2205f513c79bc8a >.git/refs/heads/broken-missing
>     $ git rev-parse master: >.git/refs/heads/broken-tree
>     $ git rev-parse "master:foo /baz" >.git/refs/heads/broken-blob
> 
> running "git fsck" does tell you about them, ...
> 
>     $ git fsck
>     Checking object directories: 100% (256/256), done.
>     error: refs/heads/broken-blob: not a commit
>     error: refs/heads/broken-missing: invalid sha1 pointer 9e830ad6c4f43159cef50cb1c2205f513c79bc8a
>     error: refs/heads/broken-tree: not a commit
> 
> ... and using the information, you can
> 
>     $ for r in refs/heads/broken-{blob,missing,tree}
>       do git update-ref -d "$r"
>       done
> 
> to unbreak the repository.

These are good examples. I was going to suggest fsck, as well, just
because I knew it would keep going after seeing bogus results. But more
interesting is that it is finding things in your example that other
programs would _not_ find, because it's being more thorough than just
reading the refs.

Having to manually convert the human-readable fsck output to a cleanup
command is a minor pain. We could provide "git fsck --prune-broken-refs"
or something, but I'm hesitant. Deleting refs in a corrupted repository
is a good way to make recovery even harder, as it opens the door to
removing whatever objects we do still have.

In the case of a refs/remotes entry where you happen to know that you
could re-clone from the other side, it is relatively low stakes. But I
think keeping a human brain in the loop between corruption and deletion
is a good thing. Corruption should not be happening so often that it's a
major pain point.

-Peff

Re: git fetch --prune fails with "fatal: bad object"

[Junio C Hamano]

Jeff King <peff@peff.net> writes:

> These are good examples. I was going to suggest fsck, as well, just
> because I knew it would keep going after seeing bogus results. But more
> interesting is that it is finding things in your example that other
> programs would _not_ find, because it's being more thorough than just
> reading the refs.

True.

I wish for-each-ref and friends had an optional mode that lets them
keep going, but since so many features in them access objects
pointed at by the refs (e.g., "--format='%(objectname:short)'" and
"--no-merged HEAD"), it would be very cumbersome to retrofit such a
mode to the underlying machinery, I suspect.

Thanks.

Re: git fetch --prune fails with "fatal: bad object"

[Jeff King]

On Tue, Jun 04, 2024 at 10:50:30AM -0700, Junio C Hamano wrote:

> Jeff King <peff@peff.net> writes:
> 
> > These are good examples. I was going to suggest fsck, as well, just
> > because I knew it would keep going after seeing bogus results. But more
> > interesting is that it is finding things in your example that other
> > programs would _not_ find, because it's being more thorough than just
> > reading the refs.
> 
> True.
> 
> I wish for-each-ref and friends had an optional mode that lets them
> keep going, but since so many features in them access objects
> pointed at by the refs (e.g., "--format='%(objectname:short)'" and
> "--no-merged HEAD"), it would be very cumbersome to retrofit such a
> mode to the underlying machinery, I suspect.

We try to read as little as possible in for-each-ref for efficiency
purposes. The side effect is that if you don't ask for any details of
the object, we won't try to read it and notice that it's gone. ;)

So you can do:

  git for-each-ref --format='%(objectname) %(refname)'

which will show even broken refs. Of course you still don't know which
ones are broken! For that, I'd rely on something like cat-file, where it
will reliably tell you about missing objects and continue (even if we
just print the object name, it does an existence check). So naively I'd
want something like this to work:

  git for-each-ref --format='%(objectname) %(refname)' |
  git cat-file --batch-check='%(objectname) %(rest)' |
  grep ^missing

But annoyingly, when it encounters a missing object it, cat-file ditches
the whole format string and just prints "$oid missing". You can work
around it with a little shell magic:

  git for-each-ref --format='%(objectname) %(refname)' >both
  cut -d' ' -f1 <both | git cat-file --batch-check='%(objectname)' >exists
  paste -d' ' both exists | cut -d' ' -f2,4

which yields lines like:

  refs/heads/ok
  refs/heads/broken missing

and so on. Turning that into "update-ref --stdin" input is left as an
exercise for the reader.

I suspect it's too late to change the default behavior for a plumbing
tool like cat-file. But we could probably teach it an option like
"--missing=foo" to expand the format but use "foo" for object-related
placeholders that can't be expanded. And then:

  git for-each-ref --format='%(objectname) %(refname)' |
  git cat-file --batch-check='%(objectname) %(rest)' --missing=MISSING |
  grep ^MISSING

would work. I still think it's inferior to fsck, though, as it's a very
shallow check of validity.

-Peff

Re: git fetch --prune fails with "fatal: bad object"

[Fred Long]

On 6/4/2024 3:44 AM, Jeff King peff-at-peff.net |git bugs/Example Allow| 
wrote:
> In the case of a refs/remotes entry where you happen to know that you
> could re-clone from the other side, it is relatively low stakes. But I
> think keeping a human brain in the loop between corruption and deletion
> is a good thing. Corruption should not be happening so often that it's a
> major pain point.
In my case it's not corruption. It's people creating branches, deleting 
them, and then removing the commits. (Maybe our git server has an option 
to automatically prune commits that are not reachable from a branch or 
tag, I don't know.) But this happens very frequently at my work.

Re: git fetch --prune fails with "fatal: bad object"

[Jeff King]

On Tue, Jun 04, 2024 at 01:09:10PM -0700, Fred Long wrote:

> On 6/4/2024 3:44 AM, Jeff King peff-at-peff.net |git bugs/Example Allow|
> wrote:
> > In the case of a refs/remotes entry where you happen to know that you
> > could re-clone from the other side, it is relatively low stakes. But I
> > think keeping a human brain in the loop between corruption and deletion
> > is a good thing. Corruption should not be happening so often that it's a
> > major pain point.
> In my case it's not corruption. It's people creating branches, deleting
> them, and then removing the commits. (Maybe our git server has an option to
> automatically prune commits that are not reachable from a branch or tag, I
> don't know.) But this happens very frequently at my work.

Your local refs should not point to missing objects, though. Each clone
should maintain its own consistency. Are you using "git clone --shared"
or another scheme involving alternates?

-Peff

Re: git fetch --prune fails with "fatal: bad object"

[Fred Long]

On 6/5/2024 1:47 AM, Jeff King wrote:
>
> Your local refs should not point to missing objects, though. Each clone
> should maintain its own consistency. Are you using "git clone --shared"
> or another scheme involving alternates?
>
> -Peff
I use Google's "repo" script to manage Android repositories. The "repo" 
script runs "git repack -a -d", and I think that's what's removing the 
commits locally. Remotely, our bitbucket server removes commits that are 
no longer referenced.

Re: git fetch --prune fails with "fatal: bad object"

[Jeff King]

On Wed, Jun 05, 2024 at 04:43:45PM -0700, Fred Long wrote:

> On 6/5/2024 1:47 AM, Jeff King wrote:
> > 
> > Your local refs should not point to missing objects, though. Each clone
> > should maintain its own consistency. Are you using "git clone --shared"
> > or another scheme involving alternates?
> > 
> I use Google's "repo" script to manage Android repositories. The "repo"
> script runs "git repack -a -d", and I think that's what's removing the
> commits locally. Remotely, our bitbucket server removes commits that are no
> longer referenced.

Using "git repack -a -d" by itself should retain the objects that are
reachable from your local refs/remotes/ refs. But I'd suspect that
"repo" is doing something clever under the hood. I know there's some
support for alternates, and a quick (hah!) "repo init && repo sync" on
the android manifest shows that we end up symlinked objects directories.
So there is ".repo/projects/foo.git" whose objects directory is a
symlink to ".repo/project-objects/foo.git/objects".

I don't know how it is all supposed to work, but naively running "git
repack -a -d" in the "project-objects" version risks corrupting the
"projects" version, because it has no idea what refs are in the latter.

Certainly this warning is ominous:

  https://gerrit.googlesource.com/git-repo/+/refs/tags/v2.45/docs/internal-fs-layout.md#project-objects

-Peff

Re: git fetch --prune fails with "fatal: bad object"

[Fred Long]

On 6/5/2024 6:14 PM, Jeff King wrote:
>
> Certainly this warning is ominous:
>
>    https://gerrit.googlesource.com/git-repo/+/refs/tags/v2.45/docs/internal-fs-layout.md#project-objects
>
> -Peff
Yeah, I saw that, and I think I know what's causing the problem. First I 
create a bare mirror of AOSP with "repo init --mirror" and "repo sync", 
then for all my different source trees I use "repo init --reference" to 
share objects with the mirror using alternates files. Running "repo 
sync" on the bare mirror works fine. The problem occurs when I run "repo 
sync" in one of my source trees. Commits and refs have been deleted from 
the mirror, but the refs are not deleted from the repos that point to 
the mirror. So it all makes sense now. Still, I wish there were a simple 
command I could run to remove the dangling refs.

Re: git fetch --prune fails with "fatal: bad object"

[Jeff King]

On Thu, Jun 06, 2024 at 01:12:31PM -0700, Fred Long wrote:

> Yeah, I saw that, and I think I know what's causing the problem. First I
> create a bare mirror of AOSP with "repo init --mirror" and "repo sync", then
> for all my different source trees I use "repo init --reference" to share
> objects with the mirror using alternates files. Running "repo sync" on the
> bare mirror works fine. The problem occurs when I run "repo sync" in one of
> my source trees. Commits and refs have been deleted from the mirror, but the
> refs are not deleted from the repos that point to the mirror. So it all
> makes sense now. Still, I wish there were a simple command I could run to
> remove the dangling refs.

Yep, that all makes sense.

As I said before, I don't think auto-deleting corrupted refs is great in
general, just because it can make a bad situation worse. But if you want
the foot-gun, here's a more complete script that you could run. It
_just_ looks in refs/remotes/, which is perhaps a bit safer.

  git for-each-ref --format='%(refname)' refs/remotes/ |
  git cat-file --batch-check='%(objectname)' |
  perl -alne 'print "delete $F[0]" if $F[1] eq "missing"' |
  tee /dev/stderr |
  git update-ref --stdin

-Peff

Re: git fetch --prune fails with "fatal: bad object"

[Fred Long]

On 6/8/2024 4:20 AM, Jeff King wrote:
> On Thu, Jun 06, 2024 at 01:12:31PM -0700, Fred Long wrote:
>
>> Yeah, I saw that, and I think I know what's causing the problem. First I
>> create a bare mirror of AOSP with "repo init --mirror" and "repo sync", then
>> for all my different source trees I use "repo init --reference" to share
>> objects with the mirror using alternates files. Running "repo sync" on the
>> bare mirror works fine. The problem occurs when I run "repo sync" in one of
>> my source trees. Commits and refs have been deleted from the mirror, but the
>> refs are not deleted from the repos that point to the mirror. So it all
>> makes sense now. Still, I wish there were a simple command I could run to
>> remove the dangling refs.
> Yep, that all makes sense.
>
> As I said before, I don't think auto-deleting corrupted refs is great in
> general, just because it can make a bad situation worse. But if you want
> the foot-gun, here's a more complete script that you could run. It
> _just_ looks in refs/remotes/, which is perhaps a bit safer.
>
>    git for-each-ref --format='%(refname)' refs/remotes/ |
>    git cat-file --batch-check='%(objectname)' |
>    perl -alne 'print "delete $F[0]" if $F[1] eq "missing"' |
>    tee /dev/stderr |
>    git update-ref --stdin
>
> -Peff
Thanks. Here's what I have been using:

git fsck |& grep "invalid sha1 pointer" | (
     while read err ref rest
     do
         ref=${ref%:}
         echo got $ref, removing .git/$ref
         rm .git/$ref
     done
)

Re: git fetch --prune fails with "fatal: bad object"

[Jeff King]

On Sat, Jun 08, 2024 at 02:02:52PM -0700, Fred Long wrote:

> >    git for-each-ref --format='%(refname)' refs/remotes/ |
> >    git cat-file --batch-check='%(objectname)' |
> >    perl -alne 'print "delete $F[0]" if $F[1] eq "missing"' |
> >    tee /dev/stderr |
> >    git update-ref --stdin
> > 
> Thanks. Here's what I have been using:
> 
> git fsck |& grep "invalid sha1 pointer" | (
>     while read err ref rest
>     do
>         ref=${ref%:}
>         echo got $ref, removing .git/$ref
>         rm .git/$ref
>     done
> )

Parsing fsck's errors should be OK, I think, but your "rm" won't be
reliable. If the ref is packed, then it needs to be removed from
.git/packed-refs, too. Calling "git update-ref -d $ref" should work,
even for a broken ref.

-Peff

Re: git fetch --prune fails with "fatal: bad object"

[Fred Long]

On 6/11/2024 12:31 AM, Jeff King wrote:
> On Sat, Jun 08, 2024 at 02:02:52PM -0700, Fred Long wrote:
>
>>>     git for-each-ref --format='%(refname)' refs/remotes/ |
>>>     git cat-file --batch-check='%(objectname)' |
>>>     perl -alne 'print "delete $F[0]" if $F[1] eq "missing"' |
>>>     tee /dev/stderr |
>>>     git update-ref --stdin
>>>
>> Thanks. Here's what I have been using:
>>
>> git fsck |& grep "invalid sha1 pointer" | (
>>      while read err ref rest
>>      do
>>          ref=${ref%:}
>>          echo got $ref, removing .git/$ref
>>          rm .git/$ref
>>      done
>> )
> Parsing fsck's errors should be OK, I think, but your "rm" won't be
> reliable. If the ref is packed, then it needs to be removed from
> .git/packed-refs, too. Calling "git update-ref -d $ref" should work,
> even for a broken ref.
>
> -Peff
Yes, and your solution is better than a newbie like me could have 
figured out, which is why I have been arguing for a simple documented 
command that people like me can run.