Error:non-monotonic index after failed recursive "sed" command

Error:non-monotonic index after failed recursive "sed" command

[George Karpenkov]

Hi All,

I've managed to corrupt my very valuable repository with a recursive
sed which went wrong.
I wanted to convert all tabs to spaces with the following command:

find ./ -name '*.*' -exec sed -i 's/\t/    /g' {} \;

I think that has changed not only the files in the repo, but the data
files in .git directory itself. As a result, my index became
corrupted, and almost every single command dies:

> git log
error: non-monotonic index
.git/objects/pack/pack-314b1944adebea645526b6724b2044c1313241f5.idx
error: non-monotonic index
.git/objects/pack/pack-75c95b0defe1968b61e4f4e1ab7040d35110bfdc.idx
...
error: non-monotonic index
.git/objects/pack/pack-3da0da48d05140b55f4af1cf87c55a2d7898bdd5.idx
fatal: bad object HEAD

Output for git fsck is even worse:

> git fsck
error: non-monotonic index
.git/objects/pack/pack-434f8445672a92f123accffce651bdb693bd8fcb.idx
...
error: non-monotonic index
.git/objects/pack/pack-0c9d5ae4e2b46dd78dace7533adf6cdfe10326ef.idx
error: non-monotonic index
.git/objects/pack/pack-e8bd5c7f85e96e7e548a62954a8f7c7f223ba9e0.idx
Segmentation fault (core dumped)

Any advice? I've lost about 2 weeks worth of work.
Is there anything better I can try to do other then trying to
reconstruct the data manually from git blobs?

Re: Error:non-monotonic index after failed recursive "sed" command

[Matthieu Moy]

George Karpenkov <george@metaworld.ru> writes:

> Hi All,
>
> I've managed to corrupt my very valuable repository with a recursive
> sed which went wrong.
> I wanted to convert all tabs to spaces with the following command:
>
> find ./ -name '*.*' -exec sed -i 's/\t/    /g' {} \;

Clearly, this is a dangerous command as it impacts .git/. However, Git
partially protects you from this kind of error, since object files and
pack files are read-only by default.

My obvious first advice is: make backups of your corrupted repository.
Yes, I said backup_s_, better safe than sorry.

Then, the errors you get are in *.idx files, which are basically index
for pack files, for quicker access. You can try removing these files,
and then running "git index-pack" on each pack file, like

$ rm .git/objects/pack/pack-*.idx
$ git index-pack .git/objects/pack/pack-4745076928ca4df932a3727b8cc25e83574560bb.pack                        
4745076928ca4df932a3727b8cc25e83574560bb                                                              
$ git index-pack .git/objects/pack/pack-c74a6514f653d0269cdcdf9c1c102d326706bbda.pack
c74a6514f653d0269cdcdf9c1c102d326706bbda

-- 
Matthieu Moy
http://www-verimag.imag.fr/~moy/

Re: Error:non-monotonic index after failed recursive "sed" command

[John Keeping]

On Mon, Jan 14, 2013 at 01:06:16PM +0100, Matthieu Moy wrote:
> George Karpenkov <george@metaworld.ru> writes:
>> I've managed to corrupt my very valuable repository with a recursive
>> sed which went wrong.
>> I wanted to convert all tabs to spaces with the following command:
>>
>> find ./ -name '*.*' -exec sed -i 's/\t/    /g' {} \;
> 
> Clearly, this is a dangerous command as it impacts .git/. However, Git
> partially protects you from this kind of error, since object files and
> pack files are read-only by default.
> 
> My obvious first advice is: make backups of your corrupted repository.
> Yes, I said backup_s_, better safe than sorry.

Having backed up the corrupt state, another option is to just try
running the reverse command:

find .git -name '*.*' -exec sed -i 's/    /\t/g' {} \;

I had a quick grep over some pack indices here and '    ' doesn't occur
in any of mine whereas '\t' is very common so you may find that the only
'    ' sequences are the ones you introduced.


John

Re: Error:non-monotonic index after failed recursive "sed" command

[Johannes Sixt]

Am 1/14/2013 12:40, schrieb George Karpenkov:
> I've managed to corrupt my very valuable repository with a recursive
> sed which went wrong.
> I wanted to convert all tabs to spaces with the following command:
> 
> find ./ -name '*.*' -exec sed -i 's/\t/    /g' {} \;
> 
> I think that has changed not only the files in the repo, but the data
> files in .git directory itself. As a result, my index became
> corrupted, and almost every single command dies:
> 
>> git log
> error: non-monotonic index
> ..git/objects/pack/pack-314b1944adebea645526b6724b2044c1313241f5.idx
> error: non-monotonic index
> ..git/objects/pack/pack-75c95b0defe1968b61e4f4e1ab7040d35110bfdc.idx
> ....
> error: non-monotonic index
> ..git/objects/pack/pack-3da0da48d05140b55f4af1cf87c55a2d7898bdd5.idx
> fatal: bad object HEAD
> 
> Output for git fsck is even worse:
> 
>> git fsck
> error: non-monotonic index
> ..git/objects/pack/pack-434f8445672a92f123accffce651bdb693bd8fcb.idx
> ....
> error: non-monotonic index
> ..git/objects/pack/pack-0c9d5ae4e2b46dd78dace7533adf6cdfe10326ef.idx
> error: non-monotonic index
> ..git/objects/pack/pack-e8bd5c7f85e96e7e548a62954a8f7c7f223ba9e0.idx
> Segmentation fault (core dumped)
> 
> Any advice? I've lost about 2 weeks worth of work.
> Is there anything better I can try to do other then trying to
> reconstruct the data manually from git blobs?

First things first: MAKE A COPY OF THE CURRENT STATE of the repository,
including the worktree, so that you have something to go back to if things
get worse later. (At the very least, we want to debug the segfault that we
see above.)

So far that's all I can say about your case. Everything that follows is
just a shot in the dark, and others may have better ideas. Also, look
here:
https://github.com/gitster/git/blob/master/Documentation/howto/recover-corrupted-blob-object.txt

You can remove the *.idx files, because they do not carry essential
information. Now try git fsck; git repack.

Try the reverse edit:

 find .git -name '*.*' -exec sed -i 's/    /\t/g' {} \;

Remove .git/index; it can be reconstructed (of course, assuming you
started all this with a clean index; if not, you lose the staged changes).

-- Hannes

Re: Error:non-monotonic index after failed recursive "sed" command

[Junio C Hamano]

Johannes Sixt <j.sixt@viscovery.net> writes:

> Am 1/14/2013 12:40, schrieb George Karpenkov:
>> I've managed to corrupt my very valuable repository with a recursive
>> sed which went wrong.
>> I wanted to convert all tabs to spaces with the following command:
>> 
>> find ./ -name '*.*' -exec sed -i 's/\t/    /g' {} \;
>> 
>> I think that has changed not only the files in the repo, but the data
>> files in .git directory itself. As a result, my index became
>> corrupted, and almost every single command dies:
>> 
>>> git log
>> error: non-monotonic index
>> ..git/objects/pack/pack-314b1944adebea645526b6724b2044c1313241f5.idx
>> error: non-monotonic index
>> ..git/objects/pack/pack-75c95b0defe1968b61e4f4e1ab7040d35110bfdc.idx
>> ....
> ...
> Try the reverse edit:
>
>  find .git -name '*.*' -exec sed -i 's/    /\t/g' {} \;
>
> Remove .git/index; it can be reconstructed (of course, assuming you
> started all this with a clean index; if not, you lose the staged changes).

Everybody seems to be getting an impression that .idx is the only
thing that got corrupt.  Where does that come from?

I do not see anything that prevents the original command line from
touching *.pack files.  Loose objects may have been left unmolested
as their names do not match '*.*', though.

Re: Error:non-monotonic index after failed recursive "sed" command

[Matthieu Moy]

Junio C Hamano <gitster@pobox.com> writes:

> Everybody seems to be getting an impression that .idx is the only
> thing that got corrupt.  Where does that come from?

It's the only thing that appear in the error message. This does not
imply that it is the only corrupt thing, but gives a little hope that it
may still be the case.

Actually, I thought the "read-only" protection should have protected
files in object/ directory, but a little testing shows that "sed -i"
gladly accepts to modify read-only files (technically, it does not
modify it, but creates a temporary file with the new content, and then
renames it to the new location). So, the hope that pack files are
uncorrupted is rather thin unfortunately.

-- 
Matthieu Moy
http://www-verimag.imag.fr/~moy/

Re: Error:non-monotonic index after failed recursive "sed" command

[George Karpenkov]

Thanks everyone!

Progress so far:

After executing reverse sed command:
find .git -name '*.*' -exec sed -i 's/    /\t/g' {} \;

And trying to switch the branch I get:

> git checkout X

error: failed to read object 51a980792f26875d00acb79a19f043420f542cfa
at offset 41433013 from
.git/objects/pack/pack-8d629235ee9fec9c6683d42e3edb21a1b0f6e027.pack
fatal: packed object 51a980792f26875d00acb79a19f043420f542cfa (stored
in .git/objects/pack/pack-
8d629235ee9fec9c6683d42e3edb21a1b0f6e027.pack) is corrupt

So the actual .pack file is corrupt, unfortunately.

On Tue, Jan 15, 2013 at 6:13 AM, Matthieu Moy
<Matthieu.Moy@grenoble-inp.fr> wrote:
> Junio C Hamano <gitster@pobox.com> writes:
>
>> Everybody seems to be getting an impression that .idx is the only
>> thing that got corrupt.  Where does that come from?
>
> It's the only thing that appear in the error message. This does not
> imply that it is the only corrupt thing, but gives a little hope that it
> may still be the case.
>
> Actually, I thought the "read-only" protection should have protected
> files in object/ directory, but a little testing shows that "sed -i"
> gladly accepts to modify read-only files (technically, it does not
> modify it, but creates a temporary file with the new content, and then
> renames it to the new location). So, the hope that pack files are
> uncorrupted is rather thin unfortunately.
>
> --
> Matthieu Moy
> http://www-verimag.imag.fr/~moy/

Re: Error:non-monotonic index after failed recursive "sed" command

[Philip Oakley]

From: "George Karpenkov" <george@metaworld.ru>
Sent: Monday, January 14, 2013 10:57 PM
> Thanks everyone!
>
> Progress so far:
>
> After executing reverse sed command:
> find .git -name '*.*' -exec sed -i 's/    /\t/g' {} \;

Have you counted how many substitutions there are in the pack file(s). 
It may be sufficiently small that you can  simply try all the possible 
combinations of fwd and reverse substitutions. Even if it takes a few 
days the computer won't get bored ;-)

>
> And trying to switch the branch I get:
>
>> git checkout X
>
> error: failed to read object 51a980792f26875d00acb79a19f043420f542cfa
> at offset 41433013 from
> .git/objects/pack/pack-8d629235ee9fec9c6683d42e3edb21a1b0f6e027.pack
> fatal: packed object 51a980792f26875d00acb79a19f043420f542cfa (stored
> in .git/objects/pack/pack-
> 8d629235ee9fec9c6683d42e3edb21a1b0f6e027.pack) is corrupt
>
> So the actual .pack file is corrupt, unfortunately.
>
> On Tue, Jan 15, 2013 at 6:13 AM, Matthieu Moy
> <Matthieu.Moy@grenoble-inp.fr> wrote:
>> Junio C Hamano <gitster@pobox.com> writes:
>>
>>> Everybody seems to be getting an impression that .idx is the only
>>> thing that got corrupt.  Where does that come from?
>>
>> It's the only thing that appear in the error message. This does not
>> imply that it is the only corrupt thing, but gives a little hope that 
>> it
>> may still be the case.
>>
>> Actually, I thought the "read-only" protection should have protected
>> files in object/ directory, but a little testing shows that "sed -i"
>> gladly accepts to modify read-only files (technically, it does not
>> modify it, but creates a temporary file with the new content, and 
>> then
>> renames it to the new location). So, the hope that pack files are
>> uncorrupted is rather thin unfortunately.
>>
>> --
>> Matthieu Moy
>> http://www-verimag.imag.fr/~moy/
> --

Re: Error:non-monotonic index after failed recursive "sed" command

[George Karpenkov]

Happy ending!

Turns out i have actually made a backup 3 days ago.

My other work was on a branch + in a stash. Commits done on a branch
were already present in a backup.
I was able to get the stash working by copying corrupted .pack files
from the backup, luckily all the new work wasn't packed yet.

So i've just verified the log messages to see that no new commits were
made, created a patch from the corrupted git repo of the stash,
applied it on the backup, and wo-hooo, everything worked.
And then I've pushed to origin to avoid such silliness in the future.

Thanks and Regards,
George

On Tue, Jan 15, 2013 at 10:45 AM, Philip Oakley <philipoakley@iee.org> wrote:
> From: "George Karpenkov" <george@metaworld.ru>
> Sent: Monday, January 14, 2013 10:57 PM
>
>> Thanks everyone!
>>
>> Progress so far:
>>
>> After executing reverse sed command:
>> find .git -name '*.*' -exec sed -i 's/    /\t/g' {} \;
>
>
> Have you counted how many substitutions there are in the pack file(s). It
> may be sufficiently small that you can  simply try all the possible
> combinations of fwd and reverse substitutions. Even if it takes a few days
> the computer won't get bored ;-)
>
>>
>> And trying to switch the branch I get:
>>
>>> git checkout X
>>
>>
>> error: failed to read object 51a980792f26875d00acb79a19f043420f542cfa
>> at offset 41433013 from
>> .git/objects/pack/pack-8d629235ee9fec9c6683d42e3edb21a1b0f6e027.pack
>> fatal: packed object 51a980792f26875d00acb79a19f043420f542cfa (stored
>> in .git/objects/pack/pack-
>> 8d629235ee9fec9c6683d42e3edb21a1b0f6e027.pack) is corrupt
>>
>> So the actual .pack file is corrupt, unfortunately.
>>
>> On Tue, Jan 15, 2013 at 6:13 AM, Matthieu Moy
>> <Matthieu.Moy@grenoble-inp.fr> wrote:
>>>
>>> Junio C Hamano <gitster@pobox.com> writes:
>>>
>>>> Everybody seems to be getting an impression that .idx is the only
>>>> thing that got corrupt.  Where does that come from?
>>>
>>>
>>> It's the only thing that appear in the error message. This does not
>>> imply that it is the only corrupt thing, but gives a little hope that it
>>> may still be the case.
>>>
>>> Actually, I thought the "read-only" protection should have protected
>>> files in object/ directory, but a little testing shows that "sed -i"
>>> gladly accepts to modify read-only files (technically, it does not
>>> modify it, but creates a temporary file with the new content, and then
>>> renames it to the new location). So, the hope that pack files are
>>> uncorrupted is rather thin unfortunately.
>>>
>>> --
>>> Matthieu Moy
>>> http://www-verimag.imag.fr/~moy/
>>
>> --
>
>