Filesystem/kernel bug?

Filesystem/kernel bug?

[Fao, Sean]

I was attempting to perform an "emerge --sync" as root on my Gentoo 
system to update the portage tree (the actual meaning of this command is 
irrelevant, so don't worry if you're not familiar with it) and had it 
fail while attempting to change directories.  I found this to be quite 
strange, as I was running the synchronization as root.  The directory in 
question was, "/usr/portage/mail-mta/msmtp".  Sure enough, manually 
changing directories failed with a "permission denied".  I then CD'ed to 
"/usr/portage/mail-mta" and executed a "ls -la".  To my surprise, there 
was no directory named "msmtp" in "/usr/portage/mail-mta".  I then tried 
to change to a random directory name, to verify that the error message 
wasn't just generic.  "cd foo" returned "No such file or directory", as 
expected.

Following is basically an outline of the steps I performed, which lead 
to my confusion:

cgx-mail / # cd /usr/portage/mail-mta/

cgx-mail mail-mta # ls -la
ls: msmtp: Permission denied
total 4
drwxr-xr-x   16 root root  416 Dec 27 09:47 .
drwxr-xr-x  135 root root 4008 Dec 27 09:48 ..
drwxr-xr-x    3 root root  336 Dec 15 17:09 courier
drwxr-xr-x    3 root root  200 Nov  1 16:35 esmtp
drwxr-xr-x    3 root root  400 Dec 17 16:06 exim
drwxr-xr-x    3 root root  200 Sep  7 07:10 mini-qmail
drwxr-xr-x    3 root root  400 Dec 22 08:45 nbsmtp
drwxr-xr-x    3 root root  400 Nov 16 04:38 nullmailer
drwxr-xr-x    3 root root  328 Dec 23 01:37 postfix
drwxr-xr-x    3 root root  480 Dec 15 17:37 qmail
drwxr-xr-x    3 root root  352 Jul 20 09:37 qmail-ldap
drwxr-xr-x    3 root root  248 Nov 24 17:07 qmail-mysql
drwxr-xr-x    3 root root  296 Dec 21 09:11 sendmail
drwxr-xr-x    3 root root  240 Dec 25 16:07 ssmtp
drwxr-xr-x    3 root root  280 Dec  6 10:16 xmail

cgx-mail mail-mta # cd msmtp
-bash: cd: msmtp: Permission denied

cgx-mail mail-mta # cd foo
-bash: cd: foo: No such file or directory

cgx-mail mail-mta # uname -r
2.6.9-gentoo-r9

cgx-mail mail-mta # cat /etc/fstab
/dev/sda1               /boot        reiserfs  noatime,notail        1 1
/dev/sda3               /            reiserfs  noatime               0 0
/dev/sda2               none         swap      sw                    0 0
/dev/fd0                /mnt/floppy  auto      noauto                0 0
none                    /proc        proc      defaults              0 0


At this point, I'm not really sure what's going on.

Can anybody help me out?  I've posted any information I thought would be 
relevant.  If you need anything else, feel free to let me know.

Thank you,

-- 
Sean

Re: Filesystem/kernel bug?

[Fao, Sean]

Update:

I found these events --and many similar-- in my log file.

Dec 26 17:55:43 cgx-mail ReiserFS: warning: is_tree_node: node level 0 
does not match to the expected one 1
Dec 26 17:55:43 cgx-mail ReiserFS: sda3: warning: vs-5150: 
search_by_key: invalid format found in block 13706028. Fsck?
Dec 26 17:55:43 cgx-mail ReiserFS: sda3: warning: vs-13070: 
reiserfs_read_locked_inode: i/o failure occurred trying to find stat 
data of [30749 74887 0x0 SD]
Dec 26 17:55:43 cgx-mail ReiserFS: warning: is_tree_node: node level 0 
does not match to the expected one 1
Dec 26 17:55:43 cgx-mail ReiserFS: sda3: warning: vs-5150: 
search_by_key: invalid format found in block 13706028. Fsck?
Dec 26 17:55:43 cgx-mail ReiserFS: sda3: warning: vs-13070: 
reiserfs_read_locked_inode: i/o failure occurred trying to find stat 
data of [30749 74888 0x0 SD]
Dec 26 17:55:43 cgx-mail ReiserFS: warning: is_tree_node: node level 0 
does not match to the expected one 1
Dec 26 17:55:43 cgx-mail ReiserFS: sda3: warning: vs-5150: 
search_by_key: invalid format found in block 13706028. Fsck?
Dec 26 17:55:43 cgx-mail ReiserFS: sda3: warning: vs-13070: 
reiserfs_read_locked_inode: i/o failure occurred trying to find stat 
data of [30749 74888 0x0 SD]

Does this shed any new light?  Does it look like I might have a 
corrupted file system?

Thanks again,

-- 
Sean

Re: Filesystem/kernel bug?

[Vladimir Saveliev]

Hello

On Tue, 2004-12-28 at 16:52, Fao, Sean wrote:
> Update:
> 
> I found these events --and many similar-- in my log file.
> 
> Dec 26 17:55:43 cgx-mail ReiserFS: warning: is_tree_node: node level 0 
> does not match to the expected one 1
> Dec 26 17:55:43 cgx-mail ReiserFS: sda3: warning: vs-5150: 
> search_by_key: invalid format found in block 13706028. Fsck?
> Dec 26 17:55:43 cgx-mail ReiserFS: sda3: warning: vs-13070: 
> reiserfs_read_locked_inode: i/o failure occurred trying to find stat 
> data of [30749 74887 0x0 SD]
> Dec 26 17:55:43 cgx-mail ReiserFS: warning: is_tree_node: node level 0 
> does not match to the expected one 1
> Dec 26 17:55:43 cgx-mail ReiserFS: sda3: warning: vs-5150: 
> search_by_key: invalid format found in block 13706028. Fsck?
> Dec 26 17:55:43 cgx-mail ReiserFS: sda3: warning: vs-13070: 
> reiserfs_read_locked_inode: i/o failure occurred trying to find stat 
> data of [30749 74888 0x0 SD]
> Dec 26 17:55:43 cgx-mail ReiserFS: warning: is_tree_node: node level 0 
> does not match to the expected one 1
> Dec 26 17:55:43 cgx-mail ReiserFS: sda3: warning: vs-5150: 
> search_by_key: invalid format found in block 13706028. Fsck?
> Dec 26 17:55:43 cgx-mail ReiserFS: sda3: warning: vs-13070: 
> reiserfs_read_locked_inode: i/o failure occurred trying to find stat 
> data of [30749 74888 0x0 SD]
> 
> Does this shed any new light?  Does it look like I might have a 
> corrupted file system?
> 

Yes. Please try to reiserfsck your filesystem. Latest reiserfsck can be
obtained from
ftp://ftp.namesys.com/pub/reiserfsprogs/reiserfsprogs-3.6.19.tar.gz.

> Thanks again,

Re: Filesystem/kernel bug?

[Fao, Sean]

Vladimir Saveliev wrote:

>Hello
>
>On Tue, 2004-12-28 at 16:52, Fao, Sean wrote:
>  
>
>>Update:
>>
>>I found these events --and many similar-- in my log file.
>>
>>Dec 26 17:55:43 cgx-mail ReiserFS: warning: is_tree_node: node level 0 
>>does not match to the expected one 1
>>Dec 26 17:55:43 cgx-mail ReiserFS: sda3: warning: vs-5150: 
>>search_by_key: invalid format found in block 13706028. Fsck?
>>Dec 26 17:55:43 cgx-mail ReiserFS: sda3: warning: vs-13070: 
>>reiserfs_read_locked_inode: i/o failure occurred trying to find stat 
>>data of [30749 74887 0x0 SD]
>>Dec 26 17:55:43 cgx-mail ReiserFS: warning: is_tree_node: node level 0 
>>does not match to the expected one 1
>>Dec 26 17:55:43 cgx-mail ReiserFS: sda3: warning: vs-5150: 
>>search_by_key: invalid format found in block 13706028. Fsck?
>>Dec 26 17:55:43 cgx-mail ReiserFS: sda3: warning: vs-13070: 
>>reiserfs_read_locked_inode: i/o failure occurred trying to find stat 
>>data of [30749 74888 0x0 SD]
>>Dec 26 17:55:43 cgx-mail ReiserFS: warning: is_tree_node: node level 0 
>>does not match to the expected one 1
>>Dec 26 17:55:43 cgx-mail ReiserFS: sda3: warning: vs-5150: 
>>search_by_key: invalid format found in block 13706028. Fsck?
>>Dec 26 17:55:43 cgx-mail ReiserFS: sda3: warning: vs-13070: 
>>reiserfs_read_locked_inode: i/o failure occurred trying to find stat 
>>data of [30749 74888 0x0 SD]
>>
>>Does this shed any new light?  Does it look like I might have a 
>>corrupted file system?
>>
>>    
>>
>
>Yes. Please try to reiserfsck your filesystem. Latest reiserfsck can be
>obtained from
>ftp://ftp.namesys.com/pub/reiserfsprogs/reiserfsprogs-3.6.19.tar.gz.
>  
>

Vladimir,

Thank you much for your response.  This is a production email server so 
I had to wait for an appropriate time to shut the system down.

Just a quick summary of what happened.  After rebooting the sever and 
booting from a CD, I ran reiserfsck, which found no corruption in the 
file system.  I then manually checked the directory structure and 
everything looked fine.  I'm not what was wrong, but a reboot apparently 
corrected whatever it was.

I've noticed a couple of patches for reiserfs in the past month or so on 
the LKML.  I'll check after I send this message to see if I can find the 
answer for myself, but, are there any major reiserfs bug fixes related 
to the Linux kernel, which have been corrected since 2.6.9 that I should 
know about?  Also, I'm using a 3ware 8000 series SATA controller, if 
that's at all relevant.

Thanks again for your help,

-- 
Sean

Re: Filesystem/kernel bug?

[Jakob Oestergaard]

On Wed, Dec 29, 2004 at 07:54:20AM -0500, Fao, Sean wrote:
...
> 
> Vladimir,
> 
> Thank you much for your response.  This is a production email server so 
> I had to wait for an appropriate time to shut the system down.
> 
> Just a quick summary of what happened.  After rebooting the sever and 
> booting from a CD, I ran reiserfsck, which found no corruption in the 
> file system.  I then manually checked the directory structure and 
> everything looked fine.  I'm not what was wrong, but a reboot apparently 
> corrected whatever it was.

I've seen this on ext3 and XFS as well.

Both locally (ext3 and XFS) and NFS exported (XFS).

About to try out 2.6.10 with CVS XFS and patches from the list - but
somehow I doubt that the actual problem is in the filesystems.

-- 

 / jakob

Re: Filesystem/kernel bug?

[Fao, Sean]

Jakob Oestergaard wrote:

>I've seen this on ext3 and XFS as well.
>
>Both locally (ext3 and XFS) and NFS exported (XFS).
>
>About to try out 2.6.10 with CVS XFS and patches from the list - but
>somehow I doubt that the actual problem is in the filesystems.
>

Part of my reason for panic was that I recently had my Linux router kick 
the bucket.  I also used reiserfs on this system and *believe* it was 
running a 2.6.8 kernel.

The strange thing about this system was that, at first glance, it 
appeared to have a complete hard disk failure.  However, when I rebooted 
the computer, the system did indeed start to boot.  When it came time to 
mount the root file system, it became apparent that it had become 
corrupt.  I've been running Linux systems for years and this was the 
first system that I ever experienced a corrupted file system on.

As a result of my success with file systems on Linux in the past, I've 
been somewhat lazy when partitioning drives lately.  I've been creating 
100 MB boot, system_ram * 2 swap, and throwing everything else in the 
root.  I might have to go back to my old style to separate /, /home and 
/usr onto separate file systems.

-- 
Sean