missing files?

missing files?

[michael]

Hello.

I'm using reiser4 on a Gentoo home system with a 2.6.9 kernel. Until
recently, I had everything except /boot on one partition. My small disk
was getting full, so I purchased another slightly bigger disk last week
and moved /usr onto a new partition. 

All seemed fine, until I noticed that my nightly cron job that runs the
Gentoo emerge program to check for software updates was failing. After
investigating I discovered that a number of directories were missing.
That is, they appeared when using 'ls' with no options, but using 'ls
-l' produced multiple 'No such file or directory' results. 

Now it appears I'm unable to remove those directories. 'rm -f
<filename>' doesn't work, and rmdir reports the directory isn't empty. I
can rename the directory, but that doesn't fix the apparent corruption
in the filesystem. Also, fsck.reiser4 doesn't report any problems.

Can anyone explain why this might have happened and what I might be able
to do to fix it?

thanks!

Mike

Re: missing files?

[Valdis.Kletnieks]

[-- Attachment #1: Type: text/plain, Size: 2160 bytes --]

On Tue, 01 Feb 2005 23:34:13 EST, michael@styer.net said:

> All seemed fine, until I noticed that my nightly cron job that runs the
> Gentoo emerge program to check for software updates was failing. After
> investigating I discovered that a number of directories were missing.
> That is, they appeared when using 'ls' with no options, but using 'ls
> -l' produced multiple 'No such file or directory' results. 
> 
> Now it appears I'm unable to remove those directories. 'rm -f
> <filename>' doesn't work, and rmdir reports the directory isn't empty. I
> can rename the directory, but that doesn't fix the apparent corruption
> in the filesystem. Also, fsck.reiser4 doesn't report any problems.
> 
> Can anyone explain why this might have happened and what I might be able
> to do to fix it?

Sounds like wonky file permissions on the directory - lack of write permission
*on the directory* will cause 'rm' to fail.  Remember that renaming the directory
requires write permission *on it's parent*, not on itself.

(Note the following is on an ext3 filesystem)

[~]2 mkdir /tmp/foo-bar
[~]2 touch /tmp/foo-bar/baz 
[~]2 chmod 555 /tmp/foo-bar/   
[~]2 rm /tmp/foo-bar/baz
rm: cannot remove `/tmp/foo-bar/baz': Permission denied
[~]2 mv /tmp/foo-bar /tmp/foo-bar-quux
[~]2 rm /tmp/foo-bar-quux/baz
rm: cannot remove `/tmp/foo-bar-quux/baz': Permission denied
[~]2 ls /tmp/foo-bar-quux/baz
/tmp/foo-bar-quux/baz
[~]2 ls -l /tmp/foo-bar-quux/baz
-rw-r--r--  1 valdis valdis 0 Feb  2 00:25 /tmp/foo-bar-quux/baz
[~]2 ls -l /tmp/foo-bar-quux
total 1
-rw-r--r--  1 valdis valdis 0 Feb  2 00:25 baz
[~]2 chmod 400 /tmp/foo-bar-quux 
[~]2 ls -l /tmp/foo-bar-quux
total 0
?---------  ? ? ? ?           ? baz
[~]2 chmod 100 /tmp/foo-bar-quux
[~]2 ls -l /tmp/foo-bar-quux
ls: /tmp/foo-bar-quux: Permission denied
[~]2 ls -l /tmp/foo-bar-quux/baz
-rw-r--r--  1 valdis valdis 0 Feb  2 00:25 /tmp/foo-bar-quux/baz
[~]2 ls -ld /tmp/foo-bar-quux/   
d--x------  2 valdis valdis 1024 Feb  2 00:25 /tmp/foo-bar-quux/

Bad umask/chmod? (Note that some shells and scripting languages do
very different things with 'umask 22' and 'umask 022', and 'chmod 600'
and 'chmod 0600'....)

[-- Attachment #2: Type: application/pgp-signature, Size: 226 bytes --]

Re: missing files?

[michael]

On Wed, 02 Feb 2005 00:33:07 -0500, Valdis.Kletnieks@vt.edu said:
>
> > Can anyone explain why this might have happened and what I might be
> > able to do to fix it?
>
> Sounds like wonky file permissions on the directory - lack of write
> permission *on the directory* will cause 'rm' to fail.  Remember that
> renaming the directory requires write permission *on it's parent*, not
> on itself.

No, that's not it, but thanks for the suggestion. I'm doing this as root
and the directory is 755, so I should be able to remove the files no
problem. Also rm -f doesn't complain it can't remove it, just returns
with no output. But afterward ls still reports the directory is there,
and ls -l still can't find it.

Eg.

# ls parent/
target
# ls -l parent/
ls: parent/target: No such file or directory
# rm -f parent/target
# ls parent/
target
# ls -l parent/
ls: parent/target: No such file or directory
# ls -ld parent/
drwxr-xr-x  174 root root 174 Feb  1 23:30 parent/

On Wed, 02 Feb 2005 14:08:40 +0300, "Vladimir Saveliev" <vs@namesys.com>
said:
> Hello
>
> Is there anything about reiser4 in kernel logs?

Yes, in fact; there are lots of messages nearly identical to this:

Feb  2 03:19:31 apollo reiser4[rsync(17957)]: key_warning
(fs/reiser4/plugin/object.c:97)[nikita-717]:
Feb  2 03:19:31 apollo WARNING: Error for inode 483238 (-2)

The only difference between the messages is the process name and pid,
and the inode number (well, and the date and time, obviously).

Does that help?

Mike

Re: missing files?

[Vitaly Fertman]

On Wednesday 02 February 2005 17:18, michael@styer.net wrote:
> On Wed, 02 Feb 2005 00:33:07 -0500, Valdis.Kletnieks@vt.edu said:
> > > Can anyone explain why this might have happened and what I might be
> > > able to do to fix it?
> >
> > Sounds like wonky file permissions on the directory - lack of write
> > permission *on the directory* will cause 'rm' to fail.  Remember that
> > renaming the directory requires write permission *on it's parent*, not
> > on itself.
>
> No, that's not it, but thanks for the suggestion. I'm doing this as root
> and the directory is 755, so I should be able to remove the files no
> problem. Also rm -f doesn't complain it can't remove it, just returns
> with no output. But afterward ls still reports the directory is there,
> and ls -l still can't find it.
>
> Eg.
>
> # ls parent/
> target
> # ls -l parent/
> ls: parent/target: No such file or directory
> # rm -f parent/target
> # ls parent/
> target
> # ls -l parent/
> ls: parent/target: No such file or directory
> # ls -ld parent/
> drwxr-xr-x  174 root root 174 Feb  1 23:30 parent/
>
> On Wed, 02 Feb 2005 14:08:40 +0300, "Vladimir Saveliev" <vs@namesys.com>
>
> said:
> > Hello
> >
> > Is there anything about reiser4 in kernel logs?
>
> Yes, in fact; there are lots of messages nearly identical to this:
>
> Feb  2 03:19:31 apollo reiser4[rsync(17957)]: key_warning
> (fs/reiser4/plugin/object.c:97)[nikita-717]:
> Feb  2 03:19:31 apollo WARNING: Error for inode 483238 (-2)
>
> The only difference between the messages is the process name and pid,
> and the inode number (well, and the date and time, obviously).
>
> Does that help?

would you pack the metadata with 
	debugfs.reiser4 -P <device> | bzip2 -c > <device>.bz2
and let us to download them?

-- 
Thanks,
Vitaly Fertman

Re: missing files?

[Michael Styer]

On Wed, 2 Feb 2005 19:10:15 +0300, "Vitaly Fertman" <vitaly@namesys.com>
said:

> would you pack the metadata with     debugfs.reiser4 -P <device> |
> bzip2 -c > <device>.bz2 and let us to download them?

Is it possible to do this when the device is mounted rw without causing
problems? Or would I have to unmount it and remount it ro?

Michael

Re: missing files?

[Vitaly Fertman]

On Wednesday 02 February 2005 19:25, Michael Styer wrote:
> On Wed, 2 Feb 2005 19:10:15 +0300, "Vitaly Fertman" <vitaly@namesys.com>
>
> said:
> > would you pack the metadata with     debugfs.reiser4 -P <device> |
> > bzip2 -c > <device>.bz2 and let us to download them?
>
> Is it possible to do this when the device is mounted rw without causing
> problems? Or would I have to unmount it and remount it ro?

yes, umount and mount ro.

-- 
Thanks,
Vitaly Fertman

Re: missing files?

[Michael Styer]

On Wed, 2 Feb 2005 20:14:23 +0300, Vitaly Fertman said:
> On Wednesday 02 February 2005 19:25, Michael Styer wrote:
> > On Wed, 2 Feb 2005 19:10:15 +0300, Vitaly Fertman said:
> > > would you pack the metadata with     debugfs.reiser4 -P <device> |
> > > bzip2 -c > <device>.bz2 and let us to download them?
> >
> > Is it possible to do this when the device is mounted rw without
> > causing problems? Or would I have to unmount it and remount it ro?
>
> yes, umount and mount ro.

OK, thanks. I'll send you the link off-list.

Mike

Re: missing files?

[Michael Styer]

On Fri, 4 Feb 2005, Vitaly Fertman said:
>
> so what I do not understand is why reiserfsck does not report any
> problem to you. Which reiserfsprogs do you have? have you 'umount &
> mount ro' or 'remount,ro', btw? reiser4 does not do 'remount,ro'
> properly yet. have you run fsck.reiser4 on umounted fs?

Hmm. I did 'remount,ro', actually; didn't realize that it wouldn't work
with reiser4. I have version 1.0.3 of the Gentoo reiser4progs package;
fsck.reiser4, mkfs.reiser4 and debugfs.reiser4 all report version 1.0.3.
I concluded that fsck.reiser4 wasn't reporting problems just on the
basis of the lack of problems noted in the /var/log/messages output on
boot, but I hadn't run it manually.

OK. So I killed everything that was keeping /usr busy and unmounted it.
I ran debugfs.reiser4 on /dev/hdb1 unmounted. Then I mounted the
partition ( 'mount -t reiser 4 -o ro,noatime /dev/hdb1 /usr' ). Now
debugfs.reiser4 dies with this output:

    Fatal: Can't read master super block. Error: Can't open
    reiser4 on /usr

and fsck.reiser4 says this:

    Fatal: Can't read master superblock. Fatal: Failed to open the
    reiser4 backup. Fatal: Cannot open the FileSysten on (/usr).

    1 fatal corruptions were detected in SuperBlock. Run with --build-sb
    option to fix them.

I have the output of my first debugfs run if that would be helpful. I'll
put it up for download on the same IP with the filename of
mas.devhdb1_unmounted.bz2. I'm going to reboot so that you can get that
file (this is my webserver as well...).

Regarding the output of fsck.reiser4, it suggests to run fsck.reiser4
--build-sb. What's that going to do to my filesystem? Am I likely to
lose the contents (i.e. is it like reformatting) or will it fix the
structure without causing data loss? At the minute it seems I still have
access to the filesystem, so I could tar everything up now before
rebuilding the superblock if doing so would mean losing that data.

Do you have any suggestions or advice as to what I should do from here?

Thanks very much for your help; I appreciate your taking the time to
help me out with this.

Michael

PS: I've included the list address again in case this discussion might
be helpful to anyone else.

Re: missing files?

[Vitaly Fertman]

On Saturday 05 February 2005 19:31, Michael Styer wrote:
> On Fri, 4 Feb 2005, Vitaly Fertman said:
> > so what I do not understand is why reiserfsck does not report any
> > problem to you. Which reiserfsprogs do you have? have you 'umount &
> > mount ro' or 'remount,ro', btw? reiser4 does not do 'remount,ro'
> > properly yet. have you run fsck.reiser4 on umounted fs?
>
> Hmm. I did 'remount,ro', actually; didn't realize that it wouldn't work
> with reiser4. I have version 1.0.3 of the Gentoo reiser4progs package;
> fsck.reiser4, mkfs.reiser4 and debugfs.reiser4 all report version 1.0.3.
> I concluded that fsck.reiser4 wasn't reporting problems just on the
> basis of the lack of problems noted in the /var/log/messages output on
> boot, but I hadn't run it manually.
>
> OK. So I killed everything that was keeping /usr busy and unmounted it.
> I ran debugfs.reiser4 on /dev/hdb1 unmounted. Then I mounted the
> partition ( 'mount -t reiser 4 -o ro,noatime /dev/hdb1 /usr' ). Now
> debugfs.reiser4 dies with this output:
>
>     Fatal: Can't read master super block. Error: Can't open
>     reiser4 on /usr
>
> and fsck.reiser4 says this:
>
>     Fatal: Can't read master superblock. Fatal: Failed to open the
>     reiser4 backup. Fatal: Cannot open the FileSysten on (/usr).
>
>     1 fatal corruptions were detected in SuperBlock. Run with --build-sb
>     option to fix them.

what version of reiser4progs have you created this fs with? 
it looks like the version 1.0.3 cannot open the fs due to some 
changes in the format.

> I have the output of my first debugfs run if that would be helpful. I'll
> put it up for download on the same IP with the filename of
> mas.devhdb1_unmounted.bz2. I'm going to reboot so that you can get that
> file (this is my webserver as well...).

ok

> Regarding the output of fsck.reiser4, it suggests to run fsck.reiser4
> --build-sb. What's that going to do to my filesystem? Am I likely to
> lose the contents (i.e. is it like reformatting) or will it fix the
> structure without causing data loss? At the minute it seems I still have
> access to the filesystem, so I could tar everything up now before
> rebuilding the superblock if doing so would mean losing that data.

backups are never useless.

> Do you have any suggestions or advice as to what I should do from here?

it depends on the reiser4progs version you created the fs with.
if it was 1.0.2, you can just run 'fsck.reiser4 --build-sb <device>', 
of the 1.0.3 version, it will add reiser4 backup info properly.

if nothing has been changed on the fs since mas.devhdb1_unmounted.bz2
was created, you will get 2 fs corruptions that can be fixed with 
'fsck.reiser4 --build-fs <device>' only. about 90 leaves will be cut off
the tree and will be inserted leaf-by-leaf and after that not inserted 
item-by-item.

btw, it is possible to run 
	fsck.reiser4 --build-sb --build-fs <device>
to fix it all together. 

> Thanks very much for your help; I appreciate your taking the time to
> help me out with this.
>
> Michael
>
> PS: I've included the list address again in case this discussion might
> be helpful to anyone else.

-- 
Thanks,
Vitaly Fertman

RE: missing files?

[Michael Styer]

On Monday, 07 February, 2005 10:38 AM Vitaly Fertman wrote:
> On Saturday 05 February 2005 19:31, Michael Styer wrote:
>> OK. So I killed everything that was keeping /usr busy and unmounted it.
>> I ran debugfs.reiser4 on /dev/hdb1 unmounted. Then I mounted the
>> partition ( 'mount -t reiser 4 -o ro,noatime /dev/hdb1 /usr' ). Now
>> debugfs.reiser4 dies with this output:
>>
>>     Fatal: Can't read master super block. Error: Can't open     reiser4
>> on /usr
>>
>> and fsck.reiser4 says this:
>>
>>     Fatal: Can't read master superblock. Fatal: Failed to open the
>>     reiser4 backup. Fatal: Cannot open the FileSysten on (/usr).
>>
>>     1 fatal corruptions were detected in SuperBlock. Run with --build-sb
>>     option to fix them.
>
> what version of reiser4progs have you created this fs with?
> it looks like the version 1.0.3 cannot open the fs due to some
> changes in the format.

I'm not entirely sure when I built this filesystem, but my guess is it was
version 1.0.2.

>> Regarding the output of fsck.reiser4, it suggests to run fsck.reiser4
>> --build-sb. What's that going to do to my filesystem? Am I likely to
>> lose the contents (i.e. is it like reformatting) or will it fix the
>> structure without causing data loss? At the minute it seems I still have
>> access to the filesystem, so I could tar everything up now before
>> rebuilding the superblock if doing so would mean losing that data.
>
> backups are never useless.

Yes, a good point.

>> Do you have any suggestions or advice as to what I should do from here?
>
> it depends on the reiser4progs version you created the fs with.
> if it was 1.0.2, you can just run 'fsck.reiser4 --build-sb <device>',
> of the 1.0.3 version, it will add reiser4 backup info properly.

Do you mean, if I did create the fs with reiser4progs 1.0.2 I can now run
version 1.0.3 of fsck.reiser4 with the --build-sb option and it will do the
right thing?

> if nothing has been changed on the fs since mas.devhdb1_unmounted.bz2
> was created, you will get 2 fs corruptions that can be fixed with
> 'fsck.reiser4 --build-fs <device>' only. about 90 leaves will be cut off
> the tree and will be inserted leaf-by-leaf and after that not inserted
> item-by-item.
>
> btw, it is possible to run
>       fsck.reiser4 --build-sb --build-fs <device>
> to fix it all together.

Just to confirm, I can do this using my current version 1.0.3 of
reiser4progs even if I created the fs with version 1.0.2? And I'm assuming I
have to have the fs mounted ro to do this, correct? Or should I unmount the
fs completely before doing this?

Thanks again.

Michael

Re: missing files?

[Vitaly Fertman]

On Monday 07 February 2005 18:58, Michael Styer wrote:
> On Monday, 07 February, 2005 10:38 AM Vitaly Fertman wrote:
> > On Saturday 05 February 2005 19:31, Michael Styer wrote:
> >> OK. So I killed everything that was keeping /usr busy and unmounted it.
> >> I ran debugfs.reiser4 on /dev/hdb1 unmounted. Then I mounted the
> >> partition ( 'mount -t reiser 4 -o ro,noatime /dev/hdb1 /usr' ). Now
> >> debugfs.reiser4 dies with this output:
> >>
> >>     Fatal: Can't read master super block. Error: Can't open     reiser4
> >> on /usr
> >>
> >> and fsck.reiser4 says this:
> >>
> >>     Fatal: Can't read master superblock. Fatal: Failed to open the
> >>     reiser4 backup. Fatal: Cannot open the FileSysten on (/usr).
> >>
> >>     1 fatal corruptions were detected in SuperBlock. Run with --build-sb
> >>     option to fix them.
> >
> > what version of reiser4progs have you created this fs with?
> > it looks like the version 1.0.3 cannot open the fs due to some
> > changes in the format.
>
> I'm not entirely sure when I built this filesystem, but my guess is it was
> version 1.0.2.
>
> >> Regarding the output of fsck.reiser4, it suggests to run fsck.reiser4
> >> --build-sb. What's that going to do to my filesystem? Am I likely to
> >> lose the contents (i.e. is it like reformatting) or will it fix the
> >> structure without causing data loss? At the minute it seems I still have
> >> access to the filesystem, so I could tar everything up now before
> >> rebuilding the superblock if doing so would mean losing that data.
> >
> > backups are never useless.
>
> Yes, a good point.
>
> >> Do you have any suggestions or advice as to what I should do from here?
> >
> > it depends on the reiser4progs version you created the fs with.
> > if it was 1.0.2, you can just run 'fsck.reiser4 --build-sb <device>',
> > of the 1.0.3 version, it will add reiser4 backup info properly.
>
> Do you mean, if I did create the fs with reiser4progs 1.0.2 I can now run
> version 1.0.3 of fsck.reiser4 with the --build-sb option and it will do the
> right thing?

yes.

> > if nothing has been changed on the fs since mas.devhdb1_unmounted.bz2
> > was created, you will get 2 fs corruptions that can be fixed with
> > 'fsck.reiser4 --build-fs <device>' only. about 90 leaves will be cut off
> > the tree and will be inserted leaf-by-leaf and after that not inserted
> > item-by-item.
> >
> > btw, it is possible to run
> >       fsck.reiser4 --build-sb --build-fs <device>
> > to fix it all together.
>
> Just to confirm, I can do this using my current version 1.0.3 of
> reiser4progs 

yes, you should use the latest reiser4progs, 1.0.3.

> even if I created the fs with version 1.0.2? And I'm assuming
> I have to have the fs mounted ro to do this, correct? Or should I unmount
> the fs completely before doing this?

you may keep it ro mounted if you want to, or unmouted.

-- 
Thanks,
Vitaly Fertman

Re: missing files?

[Vladimir Saveliev]

Hello

On Wed, 2005-02-02 at 07:34, michael@styer.net wrote:
> Hello.
> 
> I'm using reiser4 on a Gentoo home system with a 2.6.9 kernel. Until
> recently, I had everything except /boot on one partition. My small disk
> was getting full, so I purchased another slightly bigger disk last week
> and moved /usr onto a new partition. 
> 
> All seemed fine, until I noticed that my nightly cron job that runs the
> Gentoo emerge program to check for software updates was failing. After
> investigating I discovered that a number of directories were missing.
> That is, they appeared when using 'ls' with no options, but using 'ls
> -l' produced multiple 'No such file or directory' results. 
> 
> Now it appears I'm unable to remove those directories. 'rm -f
> <filename>' doesn't work, and rmdir reports the directory isn't empty. I
> can rename the directory, but that doesn't fix the apparent corruption
> in the filesystem. Also, fsck.reiser4 doesn't report any problems.
> 
> Can anyone explain why this might have happened and what I might be able
> to do to fix it?
> 

Is there anything about reiser4 in kernel logs?

> thanks!
> 
> Mike
> 

Re: missing files?

[Francis Stevens]

michael@styer.net wrote:
> All seemed fine, until I noticed that my nightly cron job that runs the
> Gentoo emerge program to check for software updates was failing. After
> investigating I discovered that a number of directories were missing.
> That is, they appeared when using 'ls' with no options, but using 'ls
> -l' produced multiple 'No such file or directory' results. 
> 
> Now it appears I'm unable to remove those directories. 'rm -f
> <filename>' doesn't work, and rmdir reports the directory isn't empty. I
> can rename the directory, but that doesn't fix the apparent corruption
> in the filesystem. Also, fsck.reiser4 doesn't report any problems.
> 

Not used resier4 myself, but this looks like an effect I had on reiser3 
when the filesystem had gone bad, the only solution I found then was a 
--rebuild-tree with reiserfsck.

FAS