Input/Output error

Input/Output error

[Srinivasan T]

[-- Attachment #1.1: Type: text/plain, Size: 3134 bytes --]

Hi,

We are running an C++ application in AWS EC2 instance (CentOS 5.4) 
mounted with an EBS Volume (say /mymountpoint). We do more simultaneous 
writes to the EBS Volume from our application. But at some point we get 
'ERROR: Input/output error'. After this, 'ls -l /mymountpoint' command 
itself fails with the i/o error. The filesystem which we use for the EBS 
Volume is xfs.

I unmounted the drive and done xfs_check and again mounted the drive. 
Now, everything seems to be working fine. But the issue still persists 
everytime when we do simultaneous writes.

I believe the following details will be useful,

[root@domU-12-31-39-07-81-36 StoreGrid]# cat /etc/redhat-release
CentOS release 5.4 (Final)

[root@domU-12-31-39-07-81-36 StoreGrid]# df -lTi
Filesystem    Type    Inodes   IUsed   IFree IUse% Mounted on
/dev/sda1     ext3   1310720  107566 1203154    9% /
/dev/sdb      ext3   19546112      11 19546101    1% /mnt
none         tmpfs    186059       1  186058    1% /dev/shm
/dev/sdh       xfs   1934272  495857 1438415   26% /mymountpoint

[root@domU-12-31-39-07-81-36 StoreGrid]# uname -a
Linux domU-12-31-39-07-81-36 2.6.21.7-2.fc8xen #1 SMP Fri Feb 15 
12:39:36 EST 2008 i686 i686 i386 GNU/Linux

Output of dmesg :

SGI XFS with ACLs, security attributes, large block numbers, no debug 
enabled
SGI XFS Quota Management subsystem
Filesystem "sdh": Disabling barriers, not supported by the underlying device
XFS mounting filesystem sdh
Ending clean XFS mount for filesystem: sdh
Filesystem "sdh": XFS internal error xfs_trans_cancel at line 1138 of 
file fs/xfs/xfs_trans.c.  Caller 0xee201944
  [<ee2032fe>] xfs_trans_cancel+0x59/0xe3 [xfs]
  [<ee201944>] xfs_rename+0x8f8/0x954 [xfs]
  [<ee201944>] xfs_rename+0x8f8/0x954 [xfs]
  [<ee21458c>] xfs_vn_rename+0x30/0x70 [xfs]
  [<c10bb5e3>] selinux_inode_rename+0x11f/0x16d
  [<c1078d88>] vfs_rename+0x2c3/0x441
  [<c107a77f>] sys_renameat+0x15a/0x1b4
  [<c1074b7f>] sys_stat64+0xf/0x23
  [<c1072d3b>] __fput+0x140/0x16a
  [<c10841ee>] mntput_no_expire+0x11/0x6a
  [<c107a800>] sys_rename+0x27/0x2b
  [<c1005688>] syscall_call+0x7/0xb
  =======================
xfs_force_shutdown(sdh,0x8) called from line 1139 of file 
fs/xfs/xfs_trans.c.  Return address = 0xee217778
Filesystem "sdh": Corruption of in-memory data detected.  Shutting down 
filesystem: sdh
Please umount the filesystem, and rectify the problem(s)
I/O error in filesystem ("sdh") meta-data dev sdh block 0x3c0001       
("xfs_trans_read_buf") error 5 buf count 512
I/O error in filesystem ("sdh") meta-data dev sdh block 0x780001       
("xfs_trans_read_buf") error 5 buf count 512
xfs_force_shutdown(sdh,0x1) called from line 423 of file 
fs/xfs/xfs_rw.c.  Return address = 0xee217778
xfs_force_shutdown(sdh,0x1) called from line 423 of file 
fs/xfs/xfs_rw.c.  Return address = 0xee217778
Filesystem "sdh": Disabling barriers, not supported by the underlying device
XFS mounting filesystem sdh
Starting XFS recovery on filesystem: sdh (logdev: internal)
Ending XFS recovery on filesystem: sdh (logdev: internal)

The XFS utilities are in v2.9.4

Any help would be appreciated.

Regards,
Srinivasan

[-- Attachment #1.2: Type: text/html, Size: 4866 bytes --]

[-- Attachment #2: Type: text/plain, Size: 121 bytes --]

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: Input/Output error

[Eric Sandeen]

On 2/22/11 9:44 AM, Srinivasan T wrote:
> Hi,
> 
> We are running an C++ application in AWS EC2 instance (CentOS 5.4)
> mounted with an EBS Volume (say /mymountpoint). We do more
> simultaneous writes to the EBS Volume from our application. But at
> some point we get 'ERROR: Input/output error'. After this, 'ls -l
> /mymountpoint' command itself fails with the i/o error. The
> filesystem which we use for the EBS Volume is xfs.
> 
> I unmounted the drive and done xfs_check and again mounted the drive.

xfs_check is read-only, FWIW, a task best handled these days
by xfs_repair -n.

> Now, everything seems to be working fine. But the issue still
> persists everytime when we do simultaneous writes.> 
> I believe the following details will be useful,
> 
> [root@domU-12-31-39-07-81-36 StoreGrid]# cat /etc/redhat-release
> CentOS release 5.4 (Final)

is quota in use?  Might you be running out of space on the fs?

Sadly I'm not even sure what xfs you might be running; there were centos kmod
rpms of older xfs for a while, and then more recent kernels have xfs.ko
built in, because RHEL5 started with official support for XFS later on.

line 1138 of file fs/xfs/xfs_trans.c does not line up with current RHEL5.

"modinfo xfs" might tell us at least where the xfs module was found,
and from there know where it came from.

But, xfs support on RHEL is best handled by Red Hat, I'm afraid.  And if this
is an old xfs kmod (which *cough* I did for centos years ago) it's not going
to be well supported by anyone.

-if- you have xfs-kmod installed, and a kernel rpm which contains xfs.ko
itself, I'd suggest trying again with the latter.

Barring all that, perhaps this is a known problem more obvious to another
person on this list ...

-Eric

> [root@domU-12-31-39-07-81-36 StoreGrid]# df -lTi
> Filesystem    Type    Inodes   IUsed   IFree IUse% Mounted on
> /dev/sda1     ext3   1310720  107566 1203154    9% /
> /dev/sdb      ext3   19546112      11 19546101    1% /mnt
> none         tmpfs    186059       1  186058    1% /dev/shm
> /dev/sdh       xfs   1934272  495857 1438415   26% /mymountpoint
> 
> [root@domU-12-31-39-07-81-36 StoreGrid]# uname -a
> Linux domU-12-31-39-07-81-36 2.6.21.7-2.fc8xen #1 SMP Fri Feb 15 12:39:36 EST 2008 i686 i686 i386 GNU/Linux
> 
> Output of dmesg :
> 
> SGI XFS with ACLs, security attributes, large block numbers, no debug enabled
> SGI XFS Quota Management subsystem
> Filesystem "sdh": Disabling barriers, not supported by the underlying device
> XFS mounting filesystem sdh
> Ending clean XFS mount for filesystem: sdh
> Filesystem "sdh": XFS internal error xfs_trans_cancel at line 1138 of file fs/xfs/xfs_trans.c.  Caller 0xee201944
>  [<ee2032fe>] xfs_trans_cancel+0x59/0xe3 [xfs]
>  [<ee201944>] xfs_rename+0x8f8/0x954 [xfs]
>  [<ee201944>] xfs_rename+0x8f8/0x954 [xfs]
>  [<ee21458c>] xfs_vn_rename+0x30/0x70 [xfs]
>  [<c10bb5e3>] selinux_inode_rename+0x11f/0x16d
>  [<c1078d88>] vfs_rename+0x2c3/0x441
>  [<c107a77f>] sys_renameat+0x15a/0x1b4
>  [<c1074b7f>] sys_stat64+0xf/0x23
>  [<c1072d3b>] __fput+0x140/0x16a
>  [<c10841ee>] mntput_no_expire+0x11/0x6a
>  [<c107a800>] sys_rename+0x27/0x2b
>  [<c1005688>] syscall_call+0x7/0xb
>  =======================
> xfs_force_shutdown(sdh,0x8) called from line 1139 of file fs/xfs/xfs_trans.c.  Return address = 0xee217778
> Filesystem "sdh": Corruption of in-memory data detected.  Shutting down filesystem: sdh
> Please umount the filesystem, and rectify the problem(s)
> I/O error in filesystem ("sdh") meta-data dev sdh block 0x3c0001       ("xfs_trans_read_buf") error 5 buf count 512
> I/O error in filesystem ("sdh") meta-data dev sdh block 0x780001       ("xfs_trans_read_buf") error 5 buf count 512
> xfs_force_shutdown(sdh,0x1) called from line 423 of file fs/xfs/xfs_rw.c.  Return address = 0xee217778
> xfs_force_shutdown(sdh,0x1) called from line 423 of file fs/xfs/xfs_rw.c.  Return address = 0xee217778
> Filesystem "sdh": Disabling barriers, not supported by the underlying device
> XFS mounting filesystem sdh
> Starting XFS recovery on filesystem: sdh (logdev: internal)
> Ending XFS recovery on filesystem: sdh (logdev: internal)
> 
> The XFS utilities are in v2.9.4
> 
> Any help would be appreciated.
> 
> Regards,
> Srinivasan
> 
> 
> 
> _______________________________________________
> xfs mailing list
> xfs@oss.sgi.com
> http://oss.sgi.com/mailman/listinfo/xfs

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: Input/Output error

[Srinivasan T]

[-- Attachment #1.1: Type: text/plain, Size: 5529 bytes --]

Hi Eric,

There is enough disk space in the fs.

[root@domU-12-31-39-07-81-36 StoreGrid]# df -lh
Filesystem            Size  Used Avail Use% Mounted on
/dev/sda1             9.9G  2.9G  6.6G  31% /
/dev/sdb              147G  188M  140G   1% /mnt
none                  854M     0  854M   0% /dev/shm
/dev/sdh              5.0G   25M  5.0G   1% /mymountpoint

Also, the output of modinfo is as follows,

[root@domU-12-31-39-07-81-36 StoreGrid]# modinfo xfs
filename:       /lib/modules/2.6.21.7-2.fc8xen/kernel/fs/xfs/xfs.ko
license:        GPL
description:    SGI XFS with ACLs, security attributes, large block 
numbers, no debug enabled
author:         Silicon Graphics, Inc.
srcversion:     C7114C18263E3067C64F2BC
depends:
vermagic:       2.6.21-2952.fc8xen SMP mod_unload 686 4KSTACKS

I have installed xfs with yum by using the following command,

yum install xfsprogs

I think I have installed the kernel rpm xfs ko. Can you confirm if this 
is correct.

Regards,
Srinivasan


On 02/23/2011 01:51 AM, Eric Sandeen wrote:
> On 2/22/11 9:44 AM, Srinivasan T wrote:
>> Hi,
>>
>> We are running an C++ application in AWS EC2 instance (CentOS 5.4)
>> mounted with an EBS Volume (say /mymountpoint). We do more
>> simultaneous writes to the EBS Volume from our application. But at
>> some point we get 'ERROR: Input/output error'. After this, 'ls -l
>> /mymountpoint' command itself fails with the i/o error. The
>> filesystem which we use for the EBS Volume is xfs.
>>
>> I unmounted the drive and done xfs_check and again mounted the drive.
> xfs_check is read-only, FWIW, a task best handled these days
> by xfs_repair -n.
>
>> Now, everything seems to be working fine. But the issue still
>> persists everytime when we do simultaneous writes.>
>> I believe the following details will be useful,
>>
>> [root@domU-12-31-39-07-81-36 StoreGrid]# cat /etc/redhat-release
>> CentOS release 5.4 (Final)
> is quota in use?  Might you be running out of space on the fs?
>
> Sadly I'm not even sure what xfs you might be running; there were centos kmod
> rpms of older xfs for a while, and then more recent kernels have xfs.ko
> built in, because RHEL5 started with official support for XFS later on.
>
> line 1138 of file fs/xfs/xfs_trans.c does not line up with current RHEL5.
>
> "modinfo xfs" might tell us at least where the xfs module was found,
> and from there know where it came from.
>
> But, xfs support on RHEL is best handled by Red Hat, I'm afraid.  And if this
> is an old xfs kmod (which *cough* I did for centos years ago) it's not going
> to be well supported by anyone.
>
> -if- you have xfs-kmod installed, and a kernel rpm which contains xfs.ko
> itself, I'd suggest trying again with the latter.
>
> Barring all that, perhaps this is a known problem more obvious to another
> person on this list ...
>
> -Eric
>
>> [root@domU-12-31-39-07-81-36 StoreGrid]# df -lTi
>> Filesystem    Type    Inodes   IUsed   IFree IUse% Mounted on
>> /dev/sda1     ext3   1310720  107566 1203154    9% /
>> /dev/sdb      ext3   19546112      11 19546101    1% /mnt
>> none         tmpfs    186059       1  186058    1% /dev/shm
>> /dev/sdh       xfs   1934272  495857 1438415   26% /mymountpoint
>>
>> [root@domU-12-31-39-07-81-36 StoreGrid]# uname -a
>> Linux domU-12-31-39-07-81-36 2.6.21.7-2.fc8xen #1 SMP Fri Feb 15 12:39:36 EST 2008 i686 i686 i386 GNU/Linux
>>
>> Output of dmesg :
>>
>> SGI XFS with ACLs, security attributes, large block numbers, no debug enabled
>> SGI XFS Quota Management subsystem
>> Filesystem "sdh": Disabling barriers, not supported by the underlying device
>> XFS mounting filesystem sdh
>> Ending clean XFS mount for filesystem: sdh
>> Filesystem "sdh": XFS internal error xfs_trans_cancel at line 1138 of file fs/xfs/xfs_trans.c.  Caller 0xee201944
>>   [<ee2032fe>] xfs_trans_cancel+0x59/0xe3 [xfs]
>>   [<ee201944>] xfs_rename+0x8f8/0x954 [xfs]
>>   [<ee201944>] xfs_rename+0x8f8/0x954 [xfs]
>>   [<ee21458c>] xfs_vn_rename+0x30/0x70 [xfs]
>>   [<c10bb5e3>] selinux_inode_rename+0x11f/0x16d
>>   [<c1078d88>] vfs_rename+0x2c3/0x441
>>   [<c107a77f>] sys_renameat+0x15a/0x1b4
>>   [<c1074b7f>] sys_stat64+0xf/0x23
>>   [<c1072d3b>] __fput+0x140/0x16a
>>   [<c10841ee>] mntput_no_expire+0x11/0x6a
>>   [<c107a800>] sys_rename+0x27/0x2b
>>   [<c1005688>] syscall_call+0x7/0xb
>>   =======================
>> xfs_force_shutdown(sdh,0x8) called from line 1139 of file fs/xfs/xfs_trans.c.  Return address = 0xee217778
>> Filesystem "sdh": Corruption of in-memory data detected.  Shutting down filesystem: sdh
>> Please umount the filesystem, and rectify the problem(s)
>> I/O error in filesystem ("sdh") meta-data dev sdh block 0x3c0001       ("xfs_trans_read_buf") error 5 buf count 512
>> I/O error in filesystem ("sdh") meta-data dev sdh block 0x780001       ("xfs_trans_read_buf") error 5 buf count 512
>> xfs_force_shutdown(sdh,0x1) called from line 423 of file fs/xfs/xfs_rw.c.  Return address = 0xee217778
>> xfs_force_shutdown(sdh,0x1) called from line 423 of file fs/xfs/xfs_rw.c.  Return address = 0xee217778
>> Filesystem "sdh": Disabling barriers, not supported by the underlying device
>> XFS mounting filesystem sdh
>> Starting XFS recovery on filesystem: sdh (logdev: internal)
>> Ending XFS recovery on filesystem: sdh (logdev: internal)
>>
>> The XFS utilities are in v2.9.4
>>
>> Any help would be appreciated.
>>
>> Regards,
>> Srinivasan
>>
>>
>>
>> _______________________________________________
>> xfs mailing list
>> xfs@oss.sgi.com
>> http://oss.sgi.com/mailman/listinfo/xfs


[-- Attachment #1.2: Type: text/html, Size: 7035 bytes --]

[-- Attachment #2: Type: text/plain, Size: 121 bytes --]

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: Input/Output error

[Eric Sandeen]

On 2/23/11 12:33 AM, Srinivasan T wrote:
> Hi Eric,
> 
> There is enough disk space in the fs.
> 
> [root@domU-12-31-39-07-81-36 StoreGrid]# df -lh
> Filesystem            Size  Used Avail Use% Mounted on
> /dev/sda1             9.9G  2.9G  6.6G  31% /
> /dev/sdb              147G  188M  140G   1% /mnt
> none                  854M     0  854M   0% /dev/shm
> /dev/sdh              5.0G   25M  5.0G   1% /mymountpoint
> 
> Also, the output of modinfo is as follows,
> 
> [root@domU-12-31-39-07-81-36 StoreGrid]# modinfo xfs
> filename:       /lib/modules/2.6.21.7-2.fc8xen/kernel/fs/xfs/xfs.ko

so a fedora 8 xen kernel on a centos 5.4 machine... you are pretty far
from the supported realm here, I'm afraid.

Can you check your logs for IO errors as dave suggested; otherwise, try
upstream or at -least- an official RHEL5 install.

-Eric

> license:        GPL
> description:    SGI XFS with ACLs, security attributes, large block numbers, no debug enabled
> author:         Silicon Graphics, Inc.
> srcversion:     C7114C18263E3067C64F2BC
> depends:
> vermagic:       2.6.21-2952.fc8xen SMP mod_unload 686 4KSTACKS
> 
> I have installed xfs with yum by using the following command,
> 
> yum install xfsprogs
> 
> I think I have installed the kernel rpm xfs ko. Can you confirm if this is correct.
> 
> Regards,
> Srinivasan

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: Input/Output error

[Dave Chinner]

On Wed, Feb 23, 2011 at 12:03:12PM +0530, Srinivasan T wrote:
> Hi Eric,
> 
> There is enough disk space in the fs.
> 
> [root@domU-12-31-39-07-81-36 StoreGrid]# df -lh
> Filesystem            Size  Used Avail Use% Mounted on
> /dev/sda1             9.9G  2.9G  6.6G  31% /
> /dev/sdb              147G  188M  140G   1% /mnt
> none                  854M     0  854M   0% /dev/shm
> /dev/sdh              5.0G   25M  5.0G   1% /mymountpoint
> 
> Also, the output of modinfo is as follows,
> 
> [root@domU-12-31-39-07-81-36 StoreGrid]# modinfo xfs
> filename:       /lib/modules/2.6.21.7-2.fc8xen/kernel/fs/xfs/xfs.ko
> license:        GPL
> description:    SGI XFS with ACLs, security attributes, large block
> numbers, no debug enabled
> author:         Silicon Graphics, Inc.
> srcversion:     C7114C18263E3067C64F2BC
> depends:
> vermagic:       2.6.21-2952.fc8xen SMP mod_unload 686 4KSTACKS
                                                        ^^^^^^^^

That's another potential cause of the problem, though blowing a
stack normally results in a crash rather than a shutdown...

Cheers,

Dave.
-- 
Dave Chinner
david@fromorbit.com

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: Input/Output error

[Dave Chinner]

On Tue, Feb 22, 2011 at 09:14:24PM +0530, Srinivasan T wrote:
> Hi,
> 
> We are running an C++ application in AWS EC2 instance (CentOS 5.4)
> mounted with an EBS Volume (say /mymountpoint). We do more
> simultaneous writes to the EBS Volume from our application. But at
> some point we get 'ERROR: Input/output error'. After this, 'ls -l
> /mymountpoint' command itself fails with the i/o error. The
> filesystem which we use for the EBS Volume is xfs.
> 
> I unmounted the drive and done xfs_check and again mounted the
> drive. Now, everything seems to be working fine. But the issue still
> persists everytime when we do simultaneous writes.
> 
> I believe the following details will be useful,
> 
> [root@domU-12-31-39-07-81-36 StoreGrid]# cat /etc/redhat-release
> CentOS release 5.4 (Final)
> 
> [root@domU-12-31-39-07-81-36 StoreGrid]# df -lTi
> Filesystem    Type    Inodes   IUsed   IFree IUse% Mounted on
> /dev/sda1     ext3   1310720  107566 1203154    9% /
> /dev/sdb      ext3   19546112      11 19546101    1% /mnt
> none         tmpfs    186059       1  186058    1% /dev/shm
> /dev/sdh       xfs   1934272  495857 1438415   26% /mymountpoint
> 
> [root@domU-12-31-39-07-81-36 StoreGrid]# uname -a
> Linux domU-12-31-39-07-81-36 2.6.21.7-2.fc8xen #1 SMP Fri Feb 15
> 12:39:36 EST 2008 i686 i686 i386 GNU/Linux
> 
> Output of dmesg :
> 
> SGI XFS with ACLs, security attributes, large block numbers, no
> debug enabled
> SGI XFS Quota Management subsystem
> Filesystem "sdh": Disabling barriers, not supported by the underlying device
> XFS mounting filesystem sdh
> Ending clean XFS mount for filesystem: sdh
> Filesystem "sdh": XFS internal error xfs_trans_cancel at line 1138
> of file fs/xfs/xfs_trans.c.  Caller 0xee201944
>  [<ee2032fe>] xfs_trans_cancel+0x59/0xe3 [xfs]
>  [<ee201944>] xfs_rename+0x8f8/0x954 [xfs]
>  [<ee201944>] xfs_rename+0x8f8/0x954 [xfs]
>  [<ee21458c>] xfs_vn_rename+0x30/0x70 [xfs]
>  [<c10bb5e3>] selinux_inode_rename+0x11f/0x16d
>  [<c1078d88>] vfs_rename+0x2c3/0x441
>  [<c107a77f>] sys_renameat+0x15a/0x1b4
>  [<c1074b7f>] sys_stat64+0xf/0x23
>  [<c1072d3b>] __fput+0x140/0x16a
>  [<c10841ee>] mntput_no_expire+0x11/0x6a
>  [<c107a800>] sys_rename+0x27/0x2b
>  [<c1005688>] syscall_call+0x7/0xb
>  =======================
> xfs_force_shutdown(sdh,0x8) called from line 1139 of file
> fs/xfs/xfs_trans.c.  Return address = 0xee217778
> Filesystem "sdh": Corruption of in-memory data detected.  Shutting
> down filesystem: sdh
> Please umount the filesystem, and rectify the problem(s)
> I/O error in filesystem ("sdh") meta-data dev sdh block 0x3c0001
> ("xfs_trans_read_buf") error 5 buf count 512
> I/O error in filesystem ("sdh") meta-data dev sdh block 0x780001
> ("xfs_trans_read_buf") error 5 buf count 512

Looks like an IO error reading the AGF during a transaction. That
would indicate a problem with your storage, not XFS.

Cheers,

Dave.
-- 
Dave Chinner
david@fromorbit.com

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

input/output error

[Brent Clements]

I am getting the following error when I do this.

time dd if=/dev/zero of=/users/testfile bs=16k count=65536

dd: writing `/users/testf2': Input/output error
41986+0 records in
41985+0 records out

real    0m11.301s
user    0m0.040s
sys     0m1.779s

I'm only getting partial writes of the file. IE..it never completes
writing the file. 

I am using an hp/ux fileserver as my nfs server and it is version 3

The following is the output from /proc/mounts on my linux-2.4.20 ia64 
client.

rtcfiler:/users/users /users nfs
rw,v3,rsize=32768,wsize=32768,soft,udp,lock,addr=rtcfiler 0 0


Does anyone have any idea how I can debug why its doing this? Or has
anyone seen this before?

Thanks,
Brent Clements
Rice University




-------------------------------------------------------
This SF.net email is sponsored by: Etnus, makers of TotalView, The debugger 
for complex code. Debugging C/C++ programs can leave you feeling lost and 
disoriented. TotalView can help you find your way. Available on major UNIX 
and Linux platforms. Try it free. www.etnus.com
_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs

Re: input/output error

[Trond Myklebust]

>>>>> " " == Brent Clements <bclem@rice.edu> write

     > rtcfiler:/users/users /users nfs
     > rw,v3,rsize=32768,wsize=32768,soft,udp,lock,addr=rtcfiler 0 0
                                     ^^^^

Read up on the 'soft' mount option in 'man 5 nfs'.

Cheers,
  Trond


-------------------------------------------------------
This SF.net email is sponsored by: Etnus, makers of TotalView, The debugger 
for complex code. Debugging C/C++ programs can leave you feeling lost and 
disoriented. TotalView can help you find your way. Available on major UNIX 
and Linux platforms. Try it free. www.etnus.com
_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs

Input/output error

[Kenneth Johansson]

I got this on a ppc system running 2-4-19-pre8.

Five files is now in a state where they show up in the file system but
can't be accessed or stat (ls works but ls -l do not).

I see that a few fixes has gone in since this version. Could any of them
fix a problem like this?



-- 
Kenneth Johansson	
Ericsson AB                       Tel: +46 8 404 71 83
Borgafjordsgatan 9                Fax: +46 8 404 72 72
164 80 Stockholm                  kenneth.johansson@etx.ericsson.se

Input/output error

[Marcus Grando]

Hi list,

After reboot in kernel 2.4.15-pre9 this problem occur:

On try start syslog deamon occur this errrors "Input/output error" on many archives /var directory.

I mount the /var at boot linux single, and when try ls on /var/run list all archives .pid, 
before try ls -la on /var/run, not list all archives.
Try remove .pid archives and the error occur again.

/var partition use ext2

Any suggestion?

Tanks in advance,

Regards
Marcus Grando

	

Re: Input/output error

[Rasmus Bøg Hansen]

On Thu, 22 Nov 2001, Marcus Grando wrote:

> On try start syslog deamon occur this errrors "Input/output error" on many archives /var directory.

Try to run fsck on the /var partition. Also you should check the disk 
for bad blocks. What output do you get from the kernel ('dmesg', 
/var/log/messages etc.)?

It could be a bad disk developing bad sectors.

Rasmus

-- 
-- [ Rasmus 'Møffe' Bøg Hansen ] ---------------------------------------
DISCLAIMER: Microsoft, Windows, Windows 98, Bugs, Lacking features, IRQ
conflicts, System crashes, Non-functional multitasking, the Y2K problem
and the Blue Screen of Death are registered trademarks of
Microsoft, Corp., Redmond, USA.
--------------------------------- [ moffe at amagerkollegiet dot dk ] --

Re: Input/output error

[Hartmut Holz]

Rasmus Bøg Hansen wrote:

> On Thu, 22 Nov 2001, Marcus Grando wrote:
> 
> 
>>On try start syslog deamon occur this errrors "Input/output error" on many archives /var directory.
>>
> 
> Try to run fsck on the /var partition. Also you should check the disk 
> for bad blocks. What output do you get from the kernel ('dmesg', 
> /var/log/messages etc.)?
> 
> It could be a bad disk developing bad sectors.
> 
> Rasmus
> 
> 

On my machine (2.4.15 final) it is the same behaviour.  After reboot
the lock files (and only the lock files) are corrupt. With 2.4.14 and 
2.4.13 everything works fine. gcc 2.96, e2fsck 1.25, aic7896/97

e2fsck output:
--------------
Pass 1: Checking inodes, blocks, and sizes
Pass 2: Checking directory structure
Entry 'kudzu' in /lock/subsys (30001) has deleted/unused inode 30005. 
Clear? yes

Entry 'network' in /lock/subsys (30001) has deleted/unused inode 30006. 
  Clear? yes

Entry 'syslog' in /lock/subsys (30001) has deleted/unused inode 30007. 
Clear? yes

Entry 'portmap' in /lock/subsys (30001) has deleted/unused inode 30008. 
  Clear? yes

Entry 'nfslock' in /lock/subsys (30001) has deleted/unused inode 30009. 
  Clear? yes

Entry 'random' in /lock/subsys (30001) has deleted/unused inode 30012. 
Clear? yes

Entry 'netfs' in /lock/subsys (30001) has deleted/unused inode 30013. 
Clear? yes

Entry 'autofs' in /lock/subsys (30001) has deleted/unused inode 30014. 
Clear? yes

Entry 'local' in /lock/subsys (30001) has deleted/unused inode 30029. 
Clear? yes

Entry 'syslogd.pid' in /run (38001) has deleted/unused inode 38009. 
Clear? yes

Entry 'klogd.pid' in /run (38001) has deleted/unused inode 38010. 
Clear? yes

Pass 3: Checking directory connectivity
Pass 4: Checking reference counts
Pass 5: Checking group summary information

/var: ***** FILE SYSTEM WAS MODIFIED *****
/var: 23017/66000 files (5.9% non-contiguous), 115364/263168 blocks

Re: Input/output error

[Andreas Dilger]

On Nov 23, 2001  16:43 +0100, Hartmut Holz wrote:
> On my machine (2.4.15 final) it is the same behaviour.  After reboot
> the lock files (and only the lock files) are corrupt. With 2.4.14 and 
> 2.4.13 everything works fine. gcc 2.96, e2fsck 1.25, aic7896/97
> 
> e2fsck output:
> --------------
> Pass 1: Checking inodes, blocks, and sizes
> Pass 2: Checking directory structure
> Entry 'kudzu' in /lock/subsys (30001) has deleted/unused inode 30005. 
> Entry 'network' in /lock/subsys (30001) has deleted/unused inode 30006. 
> Entry 'syslog' in /lock/subsys (30001) has deleted/unused inode 30007. 
> Entry 'portmap' in /lock/subsys (30001) has deleted/unused inode 30008. 
> Entry 'nfslock' in /lock/subsys (30001) has deleted/unused inode 30009. 
> Entry 'syslogd.pid' in /run (38001) has deleted/unused inode 38009. 
> Entry 'klogd.pid' in /run (38001) has deleted/unused inode 38010. 

I take it that this is after a normal shutdown where you are sure that
the filesystem was unmounted cleanly?  It looks like a case where these
files are deleted, but held open by a process.

Could you please try the following:
- "telinit 1" to change into single user mode
- make sure all of the above processes are stopped (check via ps, and
  "/etc/rc.d/init.d/foo stop" for each one
- "lsof | grep /var" to see if any files are still open on /var
- umount /var
- e2fsck -f /dev/hdX

Cheers, Andreas
--
Andreas Dilger
http://sourceforge.net/projects/ext2resize/
http://www-mddsp.enel.ucalgary.ca/People/adilger/

Re: Input/output error

[Marcus Grando]

	Hello,

	I have this problem too, i execute fsck more 20 times and not 
resolv this problem.

	Execute badblocks(8) and not found bad blocks

	I note before linux-2.4.15-pre8 begin this problems.

	Changelog-pre8:
	 - Andrew Morton: fix ext3/minix/sysv fsync behaviour.

	Maybe? I don´t no.

Regards,

Marcus Grando


On Fri, 23 Nov 2001, Andreas Dilger wrote:

> On Nov 23, 2001  16:43 +0100, Hartmut Holz wrote:
> > On my machine (2.4.15 final) it is the same behaviour.  After reboot
> > the lock files (and only the lock files) are corrupt. With 2.4.14 and 
> > 2.4.13 everything works fine. gcc 2.96, e2fsck 1.25, aic7896/97
> > 
> > e2fsck output:
> > --------------
> > Pass 1: Checking inodes, blocks, and sizes
> > Pass 2: Checking directory structure
> > Entry 'kudzu' in /lock/subsys (30001) has deleted/unused inode 30005. 
> > Entry 'network' in /lock/subsys (30001) has deleted/unused inode 30006. 
> > Entry 'syslog' in /lock/subsys (30001) has deleted/unused inode 30007. 
> > Entry 'portmap' in /lock/subsys (30001) has deleted/unused inode 30008. 
> > Entry 'nfslock' in /lock/subsys (30001) has deleted/unused inode 30009. 
> > Entry 'syslogd.pid' in /run (38001) has deleted/unused inode 38009. 
> > Entry 'klogd.pid' in /run (38001) has deleted/unused inode 38010. 
> 
> I take it that this is after a normal shutdown where you are sure that
> the filesystem was unmounted cleanly?  It looks like a case where these
> files are deleted, but held open by a process.
> 
> Could you please try the following:
> - "telinit 1" to change into single user mode
> - make sure all of the above processes are stopped (check via ps, and
>   "/etc/rc.d/init.d/foo stop" for each one
> - "lsof | grep /var" to see if any files are still open on /var
> - umount /var
> - e2fsck -f /dev/hdX
> 
> Cheers, Andreas
> --
> Andreas Dilger
> http://sourceforge.net/projects/ext2resize/
> http://www-mddsp.enel.ucalgary.ca/People/adilger/
> 
> -
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at  http://www.tux.org/lkml/
> 

-- 
Marcus Grando
Administrador de Rede
UNIVALI

Re: Input/output error

[Hartmut Holz]

Andreas Dilger wrote:


> 
> I take it that this is after a normal shutdown where you are sure that
> the filesystem was unmounted cleanly?  It looks like a case where these
> files are deleted, but held open by a process.
> 


It's a normal shutdown, every filesystem is unmounted cleanly, no
complains from the os.


> Could you please try the following:
> - "telinit 1" to change into single user mode
> - make sure all of the above processes are stopped (check via ps, and
>   "/etc/rc.d/init.d/foo stop" for each one
> - "lsof | grep /var" to see if any files are still open on /var
> - umount /var
> - e2fsck -f /dev/hdX


There are a few processes running: news, something called minilgd.Even 
with these programs running lsof shows no output. If I kill these 
programs and make a normal shutdown, it loocked much better. Only a 
complain about the keyboard lock.

If I acted exactly like your advise, there is no problem with the restart.

The processes running in 2.4.15 single user mode are exactly the same as 
in 2.4.14.


Regards

Hartmut

 

Re: Input/output error

[Tom Eastep]

On Friday 23 November 2001 07:43 am, Hartmut Holz wrote:
> Rasmus Bøg Hansen wrote:
> > On Thu, 22 Nov 2001, Marcus Grando wrote:
> >>On try start syslog deamon occur this errrors "Input/output error" on
> >> many archives /var directory.
> >
> > Try to run fsck on the /var partition. Also you should check the disk
> > for bad blocks. What output do you get from the kernel ('dmesg',
> > /var/log/messages etc.)?
> >
> > It could be a bad disk developing bad sectors.
> >
> > Rasmus
>
> On my machine (2.4.15 final) it is the same behaviour.  After reboot
> the lock files (and only the lock files) are corrupt. With 2.4.14 and
> 2.4.13 everything works fine. gcc 2.96, e2fsck 1.25, aic7896/97

A "me too" here (2.4.15 final) -- ran fsck on the /var partition and saw 
output similar to that below.
>
> e2fsck output:
> --------------
> Pass 1: Checking inodes, blocks, and sizes
> Pass 2: Checking directory structure
> Entry 'kudzu' in /lock/subsys (30001) has deleted/unused inode 30005.
> Clear? yes
>
> Entry 'network' in /lock/subsys (30001) has deleted/unused inode 30006.
>   Clear? yes
>
> Entry 'syslog' in /lock/subsys (30001) has deleted/unused inode 30007.
> Clear? yes
>
> Entry 'portmap' in /lock/subsys (30001) has deleted/unused inode 30008.
>   Clear? yes
>
> Entry 'nfslock' in /lock/subsys (30001) has deleted/unused inode 30009.
>   Clear? yes
>
> Entry 'random' in /lock/subsys (30001) has deleted/unused inode 30012.
> Clear? yes
>
> Entry 'netfs' in /lock/subsys (30001) has deleted/unused inode 30013.
> Clear? yes
>
> Entry 'autofs' in /lock/subsys (30001) has deleted/unused inode 30014.
> Clear? yes
>
> Entry 'local' in /lock/subsys (30001) has deleted/unused inode 30029.
> Clear? yes
>
> Entry 'syslogd.pid' in /run (38001) has deleted/unused inode 38009.
> Clear? yes
>
> Entry 'klogd.pid' in /run (38001) has deleted/unused inode 38010.
> Clear? yes
>
> Pass 3: Checking directory connectivity
> Pass 4: Checking reference counts
> Pass 5: Checking group summary information
>
> /var: ***** FILE SYSTEM WAS MODIFIED *****
> /var: 23017/66000 files (5.9% non-contiguous), 115364/263168 blocks
>

-Tom
-- 
Tom Eastep    \  teastep@shorewall.net
AIM: tmeastep  \  http://www.shorewall.net
ICQ: #60745924  \_________________________

Re: Input/output error

[Horst von Brand]

Tom Eastep <teastep@shorewall.net> said:

[...]

> A "me too" here (2.4.15 final) -- ran fsck on the /var partition and saw 
> output similar to that below.
> >
> > e2fsck output:
> > --------------
> > Pass 1: Checking inodes, blocks, and sizes
> > Pass 2: Checking directory structure
> > Entry 'kudzu' in /lock/subsys (30001) has deleted/unused inode 30005.
> > Clear? yes
> >
> > Entry 'network' in /lock/subsys (30001) has deleted/unused inode 30006.
> >   Clear? yes

[Ad nauseam]

Saw this with 2.4.15pre[89]
-- 
Horst von Brand                             vonbrand@sleipnir.valparaiso.cl
Casilla 9G, Vin~a del Mar, Chile                               +56 32 672616

Re: Input/output error

[Russell King]

On Sun, Nov 25, 2001 at 07:50:06PM -0300, Horst von Brand wrote:
> > > Entry 'network' in /lock/subsys (30001) has deleted/unused inode 30006.
> > >   Clear? yes
> 
> [Ad nauseam]
> 
> Saw this with 2.4.15pre[89]

This is the FS bug that Al Viro has fixed for 2.4.15/2.5.0.

--
Russell King (rmk@arm.linux.org.uk)                The developer of ARM Linux
             http://www.arm.linux.org.uk/personal/aboutme.html