New repo quickly corrupted

New repo quickly corrupted

[Jason Sewall]

I was working on a new repo created from a tar snapshot of an old
project, and after about a half-hour and about 5 commits for hacking,
I had fixed the problem I set out to take care of.

However, I tried to run git gc to clean it up and put it away and I
was met with the following:

[sewall@hobo SWIFT++_1.2 (master)]$ git gc
Counting objects: 227, done.
error: corrupt loose object '680aba836639d14a36b81f9e29da52bd9af69770'
fatal: object 680aba836639d14a36b81f9e29da52bd9af69770 cannot be read
error: failed to run repack

I've followed discussions on the list about corrupted repos and did
[sewall@hobo SWIFT++_1.2 (master)]$ git fsck
error: corrupt loose object '24bbb41919ef906db6f40c3c80f246b7e0cdf9f4'
error: 24bbb41919ef906db6f40c3c80f246b7e0cdf9f4: object corrupt or missing
error: corrupt loose object '680aba836639d14a36b81f9e29da52bd9af69770'
error: 680aba836639d14a36b81f9e29da52bd9af69770: object corrupt or missing
error: 8de8382bdc3cc5f936670df414b33ee63927f3a4: object corrupt or missing
error: corrupt loose object 'd186d23e87ddf344acc56f48e3cf2f61c7a47e16'
error: d186d23e87ddf344acc56f48e3cf2f61c7a47e16: object corrupt or missing
missing blob 680aba836639d14a36b81f9e29da52bd9af69770
missing blob 8de8382bdc3cc5f936670df414b33ee63927f3a4
missing blob d186d23e87ddf344acc56f48e3cf2f61c7a47e16
missing blob 24bbb41919ef906db6f40c3c80f246b7e0cdf9f4

And since these were all blobs, I looked for em with git-log:
[sewall@hobo SWIFT++_1.2 (master)]$ git log --raw --all | grep 24bbb*
[sewall@hobo SWIFT++_1.2 (master)]$ git log --raw --all | grep d186*
[sewall@hobo SWIFT++_1.2 (master)]$ git log --raw --all | grep 8de83*
[sewall@hobo SWIFT++_1.2 (master)]$ git log --raw --all | grep 680ab*

So it would assume that these are dangling objects. Given the simple
history of the repo, the only think I can think of that could have
caused it would be that my first commit was of the original files, and
I immediately remembered that I wanted to run dos2unix on the files so
I did and did a commit -a --amend.

I untarred the original snapshot again and ran git-hash-object over
all the files, and sure enough each of those missing hashes
corresponds to a blob in the original (non-dos2unix'd) file.

I copied the corrupted repo, manually deleted those objects, then
hash-object -w the appropriate originals back in. git-fsck was
mollified.

Finally, I tried repeating the process and the same error occurs.
In summary:
unzip original.zip
git init
git add .
git commit -m "Initial commit"
find . -type f -exec dos2unix {} +
git commit -a --amend
[accept same message]
git fsck
[same error message as above]

I just built my git today, unmodified:
git version 1.5.3.5.721.g039b

What's going on?

Jason

P.S. These files aren't particularly secret, so I'm happy to post them
if need be. Its actually someone else's code (a fellow student who
graduated some time before I arrived in grad school); I was asked to
modernize it enough to compile on on the latest gcc.

Re: New repo quickly corrupted

[Junio C Hamano]

"Jason Sewall" <jasonsewall@gmail.com> writes:

> find . -type f -exec dos2unix {} +
> git commit -a --amend
> [accept same message]
> git fsck
> [same error message as above]
>
> I just built my git today, unmodified:
> git version 1.5.3.5.721.g039b
>
> What's going on?

Carelessness is what is going on.

Notice how nicely your "find . -type f -exec dos2unix {}" goes
down to .git/objects and eats your loose objects

You probably wanted to do

	git ls-files | xargs dos2unix

instead.

Re: New repo quickly corrupted

[Jason Sewall]

On Nov 15, 2007 4:55 PM, Junio C Hamano <gitster@pobox.com> wrote:
> "Jason Sewall" <jasonsewall@gmail.com> writes:
>
> > find . -type f -exec dos2unix {} +
> > git commit -a --amend
> > [accept same message]
> > git fsck
> > [same error message as above]
> >
> > I just built my git today, unmodified:
> > git version 1.5.3.5.721.g039b
> >
> > What's going on?
>
> Carelessness is what is going on.
>
> Notice how nicely your "find . -type f -exec dos2unix {}" goes
> down to .git/objects and eats your loose objects
>
> You probably wanted to do
>
>         git ls-files | xargs dos2unix
>
> instead.
>

Sure enough. I'll chalk that one up to too many mind-numbing meetings today.

Thanks for the wake-up...

Re: New repo quickly corrupted

[Linus Torvalds]

On Thu, 15 Nov 2007, Jason Sewall wrote:
>
> find . -type f -exec dos2unix {} +

Oops.

You just ran "dos2unix" on all the *git* files!

Which certainly explains why all your old objects got corrupted!

> What's going on?

PEBKAC ;)

To avoid this error in the future, may I suggest using

	git ls-files -z | xargs -0 dos2unix

or similar. Please DO NOT write to the .git/objects directory.

		Linus

Re: New repo quickly corrupted

[Jason Sewall]

On Nov 15, 2007 5:01 PM, Linus Torvalds <torvalds@linux-foundation.org> wrote:
>
>
> On Thu, 15 Nov 2007, Jason Sewall wrote:
> >
> > find . -type f -exec dos2unix {} +
>
> Oops.
>
> You just ran "dos2unix" on all the *git* files!
>
> Which certainly explains why all your old objects got corrupted!
>
> > What's going on?
>
> PEBKAC ;)

Yeah, Junio beat you to it. :)

I take comfort in the fact that others have suffered far worse
consequences from find misuse....

Re: New repo quickly corrupted

[Nicolas Pitre]

On Thu, 15 Nov 2007, Linus Torvalds wrote:

> 
> 
> On Thu, 15 Nov 2007, Jason Sewall wrote:
> >
> > find . -type f -exec dos2unix {} +
> 
> Oops.
> 
> You just ran "dos2unix" on all the *git* files!

Does "dos2unix" override file access bits?  Because the object store is 
always made read-only.


Nicolas

Re: New repo quickly corrupted

[Linus Torvalds]

On Thu, 15 Nov 2007, Nicolas Pitre wrote:
> 
> Does "dos2unix" override file access bits?  Because the object store is 
> always made read-only.

Almost all programs like that will entirely ignor the fact that something 
is read-only.

Why? Becuase you end up having to create a new file *anyway*. So nobody 
does modify-in-place, they literally end up doing

 - create temp-file
 - while (data) 
	read old file, write to tempfile
 - rename temp-file over oldfile

and unless you *explicitly* look at the permission bits you'll never even 
notice that the old file was read-only, because none of the steps above 
care at all!

So marking things read-only will give only limited protection. It will 
protect against most editors, and will protect against things that change 
files in-place and literally try to open the original file as read-write, 
but not much else.

			Linus

Re: New repo quickly corrupted

[Christian Couder]

Le jeudi 15 novembre 2007, Linus Torvalds a écrit :
> On Thu, 15 Nov 2007, Nicolas Pitre wrote:
> > Does "dos2unix" override file access bits?  Because the object store is
> > always made read-only.
>
> Almost all programs like that will entirely ignor the fact that something
> is read-only.

What if the .git/objects/ sudirectories were also read-only ?

Christian.

Re: New repo quickly corrupted

[Andreas Ericsson]

Christian Couder wrote:
> Le jeudi 15 novembre 2007, Linus Torvalds a écrit :
>> On Thu, 15 Nov 2007, Nicolas Pitre wrote:
>>> Does "dos2unix" override file access bits?  Because the object store is
>>> always made read-only.
>> Almost all programs like that will entirely ignor the fact that something
>> is read-only.
> 
> What if the .git/objects/ sudirectories were also read-only ?
> 

Then git wouldn't be able to write to it without chmod()'ing it each time.

-- 
Andreas Ericsson                   andreas.ericsson@op5.se
OP5 AB                             www.op5.se
Tel: +46 8-230225                  Fax: +46 8-230231

Re: New repo quickly corrupted

[Christian Couder]

Le vendredi 16 novembre 2007, Andreas Ericsson a écrit :
> Christian Couder wrote:
> > Le jeudi 15 novembre 2007, Linus Torvalds a écrit :
> >> On Thu, 15 Nov 2007, Nicolas Pitre wrote:
> >>> Does "dos2unix" override file access bits?  Because the object store
> >>> is always made read-only.
> >>
> >> Almost all programs like that will entirely ignor the fact that
> >> something is read-only.
> >
> > What if the .git/objects/ sudirectories were also read-only ?
>
> Then git wouldn't be able to write to it without chmod()'ing it each
> time.

Yes, but some (not manly enough) people might want the extra safety even if 
it means a performance penalty.

Christian.

Re: New repo quickly corrupted

[Robin Rosenberg]

lördag 17 november 2007 skrev Christian Couder:
> Le vendredi 16 novembre 2007, Andreas Ericsson a écrit :
> > Christian Couder wrote:
> > > Le jeudi 15 novembre 2007, Linus Torvalds a écrit :
> > >> On Thu, 15 Nov 2007, Nicolas Pitre wrote:
> > >>> Does "dos2unix" override file access bits?  Because the object store
> > >>> is always made read-only.
> > >>
> > >> Almost all programs like that will entirely ignor the fact that
> > >> something is read-only.
> > >
> > > What if the .git/objects/ sudirectories were also read-only ?
> >
> > Then git wouldn't be able to write to it without chmod()'ing it each
> > time.
> 
> Yes, but some (not manly enough) people might want the extra safety even if 
> it means a performance penalty.

Those do manly enough to do run find . -exec  ;)

-- robin