Bug? file at the same time being deleted and not registered

Bug? file at the same time being deleted and not registered

[Tomas Pospisek]

[-- Attachment #1: Type: TEXT/PLAIN, Size: 2680 bytes --]

(I'm not subscribed, so please in case do Cc: me)

Short version:
==============

  $ # we're on a mhddfs backed filesystem here, that doesn't support hardlinks
  $ mkdir bar
  $ cd bar/
  $ git init
  Initialized empty Git repository in /mnt/mhddfs/tmp/bar/.git/

  $ touch a_file
  $ git add a_file
  $ git commit a_file -m bla
  [master (root-commit) ca0ce50] bla
   0 files changed, 0 insertions(+), 0 deletions(-)
   create mode 100644 a_file

  $ git status
  # On branch master
  # Changes to be committed:
  #   (use "git reset HEAD <file>..." to unstage)
  #
  #       deleted:    a_file
  #
  # Untracked files:
  #   (use "git add <file>..." to include in what will be committed)
  #
  #       a_file

So we have a file that was committed, that git thinks it's scheduled to be
deleted and at the same time is untracked. Doesn't make any sense, does 
it?

This happens only on the mhddfs backed filesystem. The same procedure 
works well in /tmp or on plain ext3.

Here's the strace of the commit step (from a identical try):

    http://pastebin.com/htUmWYGh

Jan 'jast' Krüger from the git IRC channel spotted in that strace the fact 
that the 'link' call is failing.

In that case IMHO git should complain and let the user know, that 
something went wrong and it can't proceed meaningfully.


Long version:
=============

  $ mkdir bar
  $ cd bar/
  $ ls -la
  total 8
  drwxr-xr-x 2 foo foo 4096 2010-05-23 22:51 .
  drwxr-xr-x 2 tpo tpo 4096 2010-05-13 19:27 ..

  $ git init
  Initialized empty Git repository in /mnt/mhddfs/tmp/bar/.git/

  $ git status
  # On branch master
  #
  # Initial commit
  #
  nothing to commit (create/copy files and use "git add" to track)

  $ touch a_file
  $ git status
  # On branch master
  #
  # Initial commit
  #
  # Untracked files:
  #   (use "git add <file>..." to include in what will be committed)
  #
  #       a_file
  nothing added to commit but untracked files present (use "git add" to
  track)

  $ git add a_file
  $ git status
  # On branch master
  #
  # Initial commit
  #
  # Changes to be committed:
  #   (use "git rm --cached <file>..." to unstage)
  #
  #       new file:   a_file
  #

  $ git commit a_file -m bla
  [master (root-commit) ca0ce50] bla
   0 files changed, 0 insertions(+), 0 deletions(-)
   create mode 100644 a_file

  $ git status
  # On branch master
  # Changes to be committed:
  #   (use "git reset HEAD <file>..." to unstage)
  #
  #       deleted:    a_file
  #
  # Untracked files:
  #   (use "git add <file>..." to include in what will be committed)
  #
  #       a_file

Re: Bug? file at the same time being deleted and not registered

[Jeff King]

On Sun, May 23, 2010 at 11:09:53PM +0200, Tomas Pospisek wrote:

> So we have a file that was committed, that git thinks it's scheduled to be
> deleted and at the same time is untracked. Doesn't make any sense,
> does it?
> 
> This happens only on the mhddfs backed filesystem. The same procedure
> works well in /tmp or on plain ext3.
> 
> Here's the strace of the commit step (from a identical try):
> 
>    http://pastebin.com/htUmWYGh
> 
> Jan 'jast' Krüger from the git IRC channel spotted in that strace the
> fact that the 'link' call is failing.
> 
> In that case IMHO git should complain and let the user know, that
> something went wrong and it can't proceed meaningfully.

If you mean this (from your strace):

  link(".git/objects/df/tmp_obj_cL0wfQ", ".git/objects/df/2b8fc99e1c1d4dbc0a854d9f72157f1d6ea078") = -1 ENOSYS (Function not implemented)

Note that it is immediately followed by:

  rename(".git/objects/df/tmp_obj_cL0wfQ", ".git/objects/df/2b8fc99e1c1d4dbc0a854d9f72157f1d6ea078") = 0

IOW, git tries the hard-link first, then falls back to the rename. If
the rename also fails, git does complain. So I don't think that is your
problem.

I would guess your real problem is that when we traverse the directory
structure and see "a_file", for some reason git is not connecting that
with the index entry for "a_file". In the past, problems like this are
usually due to filesystems which munge the filename behind git's back.
I.e., if you do "touch foo" you get some file whose name _looks_ like
"foo" but is not bit-for-bit compatible. HFS will do this when
normalizing utf8 characters, but I don't think we've ever seen it on
anything as plain as "a_file".

Can you try running the output of "git diff-files --name-only" and "git
ls-files -o" through xxd or something that would show individual bytes?
My suspicion is that the "a_file" shown in each may not be bit-for-bit
identical.

-Peff

Re: Bug? file at the same time being deleted and not registered

[Tomas Pospisek]

[-- Attachment #1: Type: TEXT/PLAIN, Size: 3251 bytes --]

On Sun, 23 May 2010, Jeff King wrote:

> On Sun, May 23, 2010 at 11:09:53PM +0200, Tomas Pospisek wrote:
>
>> So we have a file that was committed, that git thinks it's scheduled to be
>> deleted and at the same time is untracked. Doesn't make any sense,
>> does it?
>>
>> This happens only on the mhddfs backed filesystem. The same procedure
>> works well in /tmp or on plain ext3.
>>
>> Here's the strace of the commit step (from a identical try):
>>
>>    http://pastebin.com/htUmWYGh
>>
>> Jan 'jast' Krüger from the git IRC channel spotted in that strace the
>> fact that the 'link' call is failing.
>>
>> In that case IMHO git should complain and let the user know, that
>> something went wrong and it can't proceed meaningfully.
>
> If you mean this (from your strace):
>
>  link(".git/objects/df/tmp_obj_cL0wfQ", ".git/objects/df/2b8fc99e1c1d4dbc0a854d9f72157f1d6ea078") = -1 ENOSYS (Function not implemented)
>
> Note that it is immediately followed by:
>
>  rename(".git/objects/df/tmp_obj_cL0wfQ", ".git/objects/df/2b8fc99e1c1d4dbc0a854d9f72157f1d6ea078") = 0
>
> IOW, git tries the hard-link first, then falls back to the rename. If
> the rename also fails, git does complain. So I don't think that is your
> problem.

(Yes, I was refering to those lines)

> I would guess your real problem is that when we traverse the directory
> structure and see "a_file", for some reason git is not connecting that
> with the index entry for "a_file". In the past, problems like this are
> usually due to filesystems which munge the filename behind git's back.
> I.e., if you do "touch foo" you get some file whose name _looks_ like
> "foo" but is not bit-for-bit compatible. HFS will do this when
> normalizing utf8 characters, but I don't think we've ever seen it on
> anything as plain as "a_file".
>
> Can you try running the output of "git diff-files --name-only" and "git
> ls-files -o" through xxd or something that would show individual bytes?
> My suspicion is that the "a_file" shown in each may not be bit-for-bit
> identical.

  $ git diff-files --name-only
  $ # no output at all

  $ git ls-files -o | xxd
  0000000: 615f 6669 6c65 0a                        a_file.
  $ ls|xxd
  0000000: 615f 6669 6c65 0a                        a_file.

For completeness:

  $ git ls-files
  $ git ls-files -c
  $ git ls-files -d
  $ git ls-files -m
  $ git ls-files -s
  $ git ls-files --directory
  $ git ls-files -u
  $ git ls-files -k
  $ git ls-files -t
  $ git ls-files -o -t
  ? a_file
  $ git ls-files --full-name -o
  a_file
  $ git ls-files -s a_file
  $ # nothing


However:

  $ pwd
  /home/tpo/src/projects/abook/bar
  $ ls
  a_file
  $ realpath a_file
  /mnt/mhddfs/home/tpo/src/projects/abook/bar/a_file

Which is because:

  $ ls -l /home/tpo/|grep src
  lrwxrwxrwx  1 tpo tpo     24 2008-08-14 10:29 src -> /mnt/mhddfs/home/tpo/src


The 'name' trail might be the right one though, because mhddfs will 
distribute its files on different backing filesystems (I thin depending on 
where it still has space left). However I 'stat'ed all files below the 
working directory (including a_file and .git*) and they're all on the same 
device.

*t

Re: Bug? file at the same time being deleted and not registered

[Jeff King]

On Mon, May 24, 2010 at 10:02:14AM +0200, Tomas Pospisek wrote:

> >Can you try running the output of "git diff-files --name-only" and "git
> >ls-files -o" through xxd or something that would show individual bytes?
> >My suspicion is that the "a_file" shown in each may not be bit-for-bit
> >identical.
> 
>  $ git diff-files --name-only
>  $ # no output at all

Hrm, I had originally thought that your "git status" output showed
a_file as deleted in the working tree, which is why I suggested
diff-files. But looking at your output again, I see that it is scheduled
for deletion. In other words, the index entry has been removed entirely,
as if "git rm --cached a_file" had been issued.

So I think I was on the wrong track with the filename-munging, then.
Sorry to lead you astray. But the problem is that your index is missing
or bogus, which is even weirder.

Looking at your strace dump, I see it writing out the index (it writes
to index.lock and then renames it into place). Can you take a look at
your .git/index file? It should be 104 bytes and look something like:

  $ xxd .git/index
  0000000: 4449 5243 0000 0002 0000 0001 4bfa 367d  DIRC........K.6}
  0000010: 0000 0000 4bfa 367d 0000 0000 0000 0900  ....K.6}........
  0000020: 0099 801d 0000 81a4 0000 03e8 0000 03e8  ................
  0000030: 0000 0000 e69d e29b b2d1 d643 4b8b 29ae  ...........CK.).
  0000040: 775a d8c2 e48c 5391 0006 615f 6669 6c65  wZ....S...a_file
  0000050: 0000 0000 1f80 d668 c07b 5955 8678 4360  .......h.{YU.xC`
  0000060: 8215 6238 89be 9b4d

It won't match byte-for-byte because there is stat information in there,
but you should see a_file.

If it's empty or 32 bytes, then that explains what status is reporting
(but the question still remains how we got into that state).

-Peff

Re: Bug? file at the same time being deleted and not registered

[Tomas Pospisek]

On Mon, 24 May 2010, Jeff King wrote:

> On Mon, May 24, 2010 at 10:02:14AM +0200, Tomas Pospisek wrote:
>
>>> Can you try running the output of "git diff-files --name-only" and "git
>>> ls-files -o" through xxd or something that would show individual bytes?
>>> My suspicion is that the "a_file" shown in each may not be bit-for-bit
>>> identical.
>>
>>  $ git diff-files --name-only
>>  $ # no output at all
>
> Hrm, I had originally thought that your "git status" output showed
> a_file as deleted in the working tree, which is why I suggested
> diff-files. But looking at your output again, I see that it is scheduled
> for deletion. In other words, the index entry has been removed entirely,
> as if "git rm --cached a_file" had been issued.
>
> So I think I was on the wrong track with the filename-munging, then.
> Sorry to lead you astray. But the problem is that your index is missing
> or bogus, which is even weirder.
>
> Looking at your strace dump, I see it writing out the index (it writes
> to index.lock and then renames it into place). Can you take a look at
> your .git/index file? It should be 104 bytes and look something like:
>
>  $ xxd .git/index
>  0000000: 4449 5243 0000 0002 0000 0001 4bfa 367d  DIRC........K.6}
>  0000010: 0000 0000 4bfa 367d 0000 0000 0000 0900  ....K.6}........
>  0000020: 0099 801d 0000 81a4 0000 03e8 0000 03e8  ................
>  0000030: 0000 0000 e69d e29b b2d1 d643 4b8b 29ae  ...........CK.).
>  0000040: 775a d8c2 e48c 5391 0006 615f 6669 6c65  wZ....S...a_file
>  0000050: 0000 0000 1f80 d668 c07b 5955 8678 4360  .......h.{YU.xC`
>  0000060: 8215 6238 89be 9b4d
>
> It won't match byte-for-byte because there is stat information in there,
> but you should see a_file.
>
> If it's empty or 32 bytes, then that explains what status is reporting
> (but the question still remains how we got into that state).

There's no .git/index file there:

  $ ls -l .git/
  total 36
  drwxr-xr-x 2 tpo tpo 4096 2010-05-23 21:36 branches
  -rw-r--r-- 1 tpo tpo    4 2010-05-23 21:36 COMMIT_EDITMSG
  -rw-r--r-- 1 tpo tpo   73 2010-05-23 21:36 description
  -rw-r--r-- 1 tpo tpo   23 2010-05-23 21:36 HEAD
  drwxr-xr-x 2 tpo tpo 4096 2010-05-23 21:36 hooks
  drwxr-xr-x 2 tpo tpo 4096 2010-05-23 21:36 info
  drwxr-xr-x 3 tpo tpo 4096 2010-05-23 21:36 logs
  drwxr-xr-x 7 tpo tpo 4096 2010-05-24 09:26 objects
  drwxr-xr-x 4 tpo tpo 4096 2010-05-23 21:36 refs

*t

Re: Bug? file at the same time being deleted and not registered

[Jeff King]

On Mon, May 24, 2010 at 10:29:02AM +0200, Tomas Pospisek wrote:

> >If it's empty or 32 bytes, then that explains what status is reporting
> >(but the question still remains how we got into that state).
> 
> There's no .git/index file there:
> 
>  $ ls -l .git/
>  total 36
>  drwxr-xr-x 2 tpo tpo 4096 2010-05-23 21:36 branches
>  -rw-r--r-- 1 tpo tpo    4 2010-05-23 21:36 COMMIT_EDITMSG
>  -rw-r--r-- 1 tpo tpo   73 2010-05-23 21:36 description
>  -rw-r--r-- 1 tpo tpo   23 2010-05-23 21:36 HEAD
>  drwxr-xr-x 2 tpo tpo 4096 2010-05-23 21:36 hooks
>  drwxr-xr-x 2 tpo tpo 4096 2010-05-23 21:36 info
>  drwxr-xr-x 3 tpo tpo 4096 2010-05-23 21:36 logs
>  drwxr-xr-x 7 tpo tpo 4096 2010-05-24 09:26 objects
>  drwxr-xr-x 4 tpo tpo 4096 2010-05-23 21:36 refs

Thanks. That means "git status" is at least reporting the right thing.
Now we just need to figure out why, when the strace of commit shows it
being written and renamed into place, the index file is missing.

I tried setting up a simple mhddfs mount to reproduce your problem, but
everything works fine for me. What version of mhddfs are you using? I'm
using version 0.1.28. I wonder if git is somehow triggering an mhddfs
bug. Looking through the svn logs for mhddfs, between 0.1.27 and 0.1.28,
there is a commit with message "fixed rename bug".

-Peff

Re: Bug? file at the same time being deleted and not registered

[Tomas Pospisek]

On Mon, 24 May 2010, Jeff King wrote:

> On Mon, May 24, 2010 at 10:29:02AM +0200, Tomas Pospisek wrote:
>
>>> If it's empty or 32 bytes, then that explains what status is reporting
>>> (but the question still remains how we got into that state).
>>
>> There's no .git/index file there:
>>
>>  $ ls -l .git/
>>  total 36
>>  drwxr-xr-x 2 tpo tpo 4096 2010-05-23 21:36 branches
>>  -rw-r--r-- 1 tpo tpo    4 2010-05-23 21:36 COMMIT_EDITMSG
>>  -rw-r--r-- 1 tpo tpo   73 2010-05-23 21:36 description
>>  -rw-r--r-- 1 tpo tpo   23 2010-05-23 21:36 HEAD
>>  drwxr-xr-x 2 tpo tpo 4096 2010-05-23 21:36 hooks
>>  drwxr-xr-x 2 tpo tpo 4096 2010-05-23 21:36 info
>>  drwxr-xr-x 3 tpo tpo 4096 2010-05-23 21:36 logs
>>  drwxr-xr-x 7 tpo tpo 4096 2010-05-24 09:26 objects
>>  drwxr-xr-x 4 tpo tpo 4096 2010-05-23 21:36 refs
>
> Thanks. That means "git status" is at least reporting the right thing.
> Now we just need to figure out why, when the strace of commit shows it
> being written and renamed into place, the index file is missing.
>
> I tried setting up a simple mhddfs mount to reproduce your problem, but
> everything works fine for me. What version of mhddfs are you using? I'm
> using version 0.1.28. I wonder if git is somehow triggering an mhddfs
> bug. Looking through the svn logs for mhddfs, between 0.1.27 and 0.1.28,
> there is a commit with message "fixed rename bug".

It's version 0.1.28-1 from Ubuntu.

*t

Re: Bug? file at the same time being deleted and not registered

[Tomas Pospisek]

On Mon, 24 May 2010, Jeff King wrote:

> On Mon, May 24, 2010 at 10:29:02AM +0200, Tomas Pospisek wrote:
>
>>> If it's empty or 32 bytes, then that explains what status is reporting
>>> (but the question still remains how we got into that state).
>>
>> There's no .git/index file there:
>>
>>  $ ls -l .git/
>>  total 36
>>  drwxr-xr-x 2 tpo tpo 4096 2010-05-23 21:36 branches
>>  -rw-r--r-- 1 tpo tpo    4 2010-05-23 21:36 COMMIT_EDITMSG
>>  -rw-r--r-- 1 tpo tpo   73 2010-05-23 21:36 description
>>  -rw-r--r-- 1 tpo tpo   23 2010-05-23 21:36 HEAD
>>  drwxr-xr-x 2 tpo tpo 4096 2010-05-23 21:36 hooks
>>  drwxr-xr-x 2 tpo tpo 4096 2010-05-23 21:36 info
>>  drwxr-xr-x 3 tpo tpo 4096 2010-05-23 21:36 logs
>>  drwxr-xr-x 7 tpo tpo 4096 2010-05-24 09:26 objects
>>  drwxr-xr-x 4 tpo tpo 4096 2010-05-23 21:36 refs
>
> Thanks. That means "git status" is at least reporting the right thing.
> Now we just need to figure out why, when the strace of commit shows it
> being written and renamed into place, the index file is missing.
>
> I tried setting up a simple mhddfs mount to reproduce your problem, but
> everything works fine for me. What version of mhddfs are you using? I'm
> using version 0.1.28. I wonder if git is somehow triggering an mhddfs
> bug. Looking through the svn logs for mhddfs, between 0.1.27 and 0.1.28,
> there is a commit with message "fixed rename bug".

Oh wow (trying to reproduce the bug just from the strace). It's mhddfs. 
Look:

$ touch bla
$ touch bla.lock
$ echo asdf > bla.lock
$ mv bla.lock bla
$ ls
$ # nothing here

That's a quite catastrophic bug in mhddfs :-((( ! Argh.

Moving this over to mhddfs.

Thanks a lot Jeff!
*t

Re: Bug? file at the same time being deleted and not registered

[Tomas Pospisek]

On Mon, 24 May 2010, Tomas Pospisek wrote:

> On Mon, 24 May 2010, Jeff King wrote:
>
>> On Mon, May 24, 2010 at 10:29:02AM +0200, Tomas Pospisek wrote:
>> 
>>>> If it's empty or 32 bytes, then that explains what status is reporting
>>>> (but the question still remains how we got into that state).
>>> 
>>> There's no .git/index file there:
>>>
>>>  $ ls -l .git/
>>>  total 36
>>>  drwxr-xr-x 2 tpo tpo 4096 2010-05-23 21:36 branches
>>>  -rw-r--r-- 1 tpo tpo    4 2010-05-23 21:36 COMMIT_EDITMSG
>>>  -rw-r--r-- 1 tpo tpo   73 2010-05-23 21:36 description
>>>  -rw-r--r-- 1 tpo tpo   23 2010-05-23 21:36 HEAD
>>>  drwxr-xr-x 2 tpo tpo 4096 2010-05-23 21:36 hooks
>>>  drwxr-xr-x 2 tpo tpo 4096 2010-05-23 21:36 info
>>>  drwxr-xr-x 3 tpo tpo 4096 2010-05-23 21:36 logs
>>>  drwxr-xr-x 7 tpo tpo 4096 2010-05-24 09:26 objects
>>>  drwxr-xr-x 4 tpo tpo 4096 2010-05-23 21:36 refs
>> 
>> Thanks. That means "git status" is at least reporting the right thing.
>> Now we just need to figure out why, when the strace of commit shows it
>> being written and renamed into place, the index file is missing.
>> 
>> I tried setting up a simple mhddfs mount to reproduce your problem, but
>> everything works fine for me. What version of mhddfs are you using? I'm
>> using version 0.1.28. I wonder if git is somehow triggering an mhddfs
>> bug. Looking through the svn logs for mhddfs, between 0.1.27 and 0.1.28,
>> there is a commit with message "fixed rename bug".
>
> Oh wow (trying to reproduce the bug just from the strace). It's mhddfs. Look:
>
> $ touch bla
> $ touch bla.lock
> $ echo asdf > bla.lock
> $ mv bla.lock bla
> $ ls
> $ # nothing here
>
> That's a quite catastrophic bug in mhddfs :-((( ! Argh.
>
> Moving this over to mhddfs.

OK, I'm waering sackcloth and ashes. The problem was a config fault of 
mine - one of the backing fs/partitions was included *twice* in the mhdffs 
fs setup in /etc/fstab. And as of now mhdffs doesn't detect a user 
composing the mhddfs filesystem in a braindammaged way.

http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=582888
*t