kernel 2.6.27.7 problem

kernel 2.6.27.7 problem

[DS]

Hi,

I have trouble with xfs after upgrade kernel 2.6.24.3 -> 2.6.27.7.

Message on console:
 end_request: I/O error, dev sde, sector 351572544
 I/O error in filesystem ("sde1") meta-data dev sde1 block 0x14f49201
 ("xlog_iodone") error 5 buf count 9728
 xfs_force_shutdown(sde1,0x2) called from line 1056 of file
 fs/xfs/xfs_log.c.  Return address = 0xffffffff80373233
 Filesystem "sde1": Log I/O Error Detected.  Shutting down filesystem:
 sde1
 Please umount the filesystem, and rectify the problem(s)


Remount is possible, but filesystem hangs up again.
After xfs_repair too.
It looks OK now after rebooting to old kernel.

On other similar system (same  disk array, other vol) I use 2.6.27.6 kernel  without problems.


System:
AMD x64, debian stable (etch)
filesytem is on iSCSI  storage

mount options:
rw,noatime,nodiratime,logbsize=256k,logbufs=8,quota

xfs_info:
meta-data=/dev/sde1              isize=256    agcount=16, agsize=5493104
blks
         =                       sectsz=512   attr=0
data     =                       bsize=4096   blocks=87889584,
imaxpct=25
         =                       sunit=16     swidth=112 blks
naming   =version 2              bsize=4096
log      =internal               bsize=4096   blocks=32768, version=2
         =                       sectsz=512   sunit=0 blks, lazy-count=0
realtime =none                   extsz=65536  blocks=0, rtextents=0


Any ideas.
Thanx

Dusan

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: kernel 2.6.27.7 problem

[Eric Sandeen]

DS wrote:
> Hi,
> 
> I have trouble with xfs after upgrade kernel 2.6.24.3 -> 2.6.27.7.
> 
> Message on console:
>  end_request: I/O error, dev sde, sector 351572544
>  I/O error in filesystem ("sde1") meta-data dev sde1 block 0x14f49201
>  ("xlog_iodone") error 5 buf count 9728
>  xfs_force_shutdown(sde1,0x2) called from line 1056 of file
>  fs/xfs/xfs_log.c.  Return address = 0xffffffff80373233
>  Filesystem "sde1": Log I/O Error Detected.  Shutting down filesystem:
>  sde1
>  Please umount the filesystem, and rectify the problem(s)

No messages before that?  It's telling you that you had an IO error
(under the filesystem, potentially a storage proboblem).

-Eric

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: kernel 2.6.27.7 problem

[DS]

No other messages before. No iSCSI/network messages...nothing.
Only this few lines and FS is down. After remount it works ....while and
hangs again.
Few hours it runs back on 2.6.24.3 and everything is OK.

It's production system, but I test it again in late night.
What can I do to identify problem?

Dusan


On Fri, Dec 05, 2008 at 09:27:00PM -0600, Eric Sandeen wrote:
> DS wrote:
> > Hi,
> > 
> > I have trouble with xfs after upgrade kernel 2.6.24.3 -> 2.6.27.7.
> > 
> > Message on console:
> >  end_request: I/O error, dev sde, sector 351572544
> >  I/O error in filesystem ("sde1") meta-data dev sde1 block 0x14f49201
> >  ("xlog_iodone") error 5 buf count 9728
> >  xfs_force_shutdown(sde1,0x2) called from line 1056 of file
> >  fs/xfs/xfs_log.c.  Return address = 0xffffffff80373233
> >  Filesystem "sde1": Log I/O Error Detected.  Shutting down filesystem:
> >  sde1
> >  Please umount the filesystem, and rectify the problem(s)
> 
> No messages before that?  It's telling you that you had an IO error
> (under the filesystem, potentially a storage proboblem).
> 
> -Eric

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: kernel 2.6.27.7 problem

[Justin Piszcz]

On Sat, 6 Dec 2008, DS wrote:

> No other messages before. No iSCSI/network messages...nothing.
> Only this few lines and FS is down. After remount it works ....while and
> hangs again.
> Few hours it runs back on 2.6.24.3 and everything is OK.
>
> It's production system, but I test it again in late night.
> What can I do to identify problem?
>
> Dusan

Run:
smartctl -a /dev/sde

Post output.

Justin.

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: kernel 2.6.27.7 problem

[Justin Piszcz]

On Sat, 6 Dec 2008, Justin Piszcz wrote:

>
>
> On Sat, 6 Dec 2008, DS wrote:
>
>> No other messages before. No iSCSI/network messages...nothing.
>> Only this few lines and FS is down. After remount it works ....while and
>> hangs again.
>> Few hours it runs back on 2.6.24.3 and everything is OK.
>> 
>> It's production system, but I test it again in late night.
>> What can I do to identify problem?
>> 
>> Dusan
>
> Run:
> smartctl -a /dev/sde
>
> Post output.
>
> Justin.
>
>

AMD x64, debian stable (etch)
filesytem is on iSCSI  storage
                 ^^^^^^^^^^^^^^

Nevermind my comment above:

Is there anyway to test the health of the disks on the system?

Justin.

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: kernel 2.6.27.7 problem

[DS]

iSCSI storage is nexsan SATA boy with 2 controlers, firmware was updated
few months ago

other 3 servers has data on this storage without problems (all on XFS)
all is going on same switch, all ports are in one storage VLAN

I don't know where "start debug" and it must be safe for data.

Dusan


On Sat, Dec 06, 2008 at 07:13:22AM -0500, Justin Piszcz wrote:
> 
> 
> On Sat, 6 Dec 2008, Justin Piszcz wrote:
> 
> >
> >
> >On Sat, 6 Dec 2008, DS wrote:
> >
> >>No other messages before. No iSCSI/network messages...nothing.
> >>Only this few lines and FS is down. After remount it works ....while and
> >>hangs again.
> >>Few hours it runs back on 2.6.24.3 and everything is OK.
> >>
> >>It's production system, but I test it again in late night.
> >>What can I do to identify problem?
> >>
> >>Dusan
> >
> >Run:
> >smartctl -a /dev/sde
> >
> >Post output.
> >
> >Justin.
> >
> >
> 
> AMD x64, debian stable (etch)
> filesytem is on iSCSI  storage
>                 ^^^^^^^^^^^^^^
> 
> Nevermind my comment above:
> 
> Is there anyway to test the health of the disks on the system?
> 
> Justin.
> 

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: kernel 2.6.27.7 problem

[Eric Sandeen]

DS wrote:
> No other messages before. No iSCSI/network messages...nothing.
> Only this few lines and FS is down. After remount it works ....while and
> hangs again.
> Few hours it runs back on 2.6.24.3 and everything is OK.
> 
> It's production system, but I test it again in late night.
> What can I do to identify problem?

Complain to the iscsi people perhaps...?

__end_that_request_first() in block/block-core.c() is what is issuing
your first error,

>  end_request: I/O error, dev sde, sector 351572544

and the rest is xfs reacting to that.

Maybe the first thing I'd check is whether that sector nr. is sane for
your device (I'd assume it is, but maybe worth checking).

If that's all you've got I'm not sure offhand what else to debug....

-Eric

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs