Re: need help how to debug xfs crash issue xfs_iunlink_remove: xfs_inotobp() returned error 22

From: Eric Sandeen <sandeen@sandeen.net>
To: 符永涛 <yongtaofu@gmail.com>
Cc: xfs@oss.sgi.com
Subject: Re: need help how to debug xfs crash issue xfs_iunlink_remove: xfs_inotobp() returned error 22
Date: Tue, 09 Apr 2013 10:06:06 -0500	[thread overview]
Message-ID: <51642E5E.3040403@sandeen.net> (raw)
In-Reply-To: <CADFMGuJm5bPPwbbUtYwrCVDL23KExJTw_-VRX2UEEdZjo+i5oA@mail.gmail.com>

On 4/9/13 7:53 AM, 符永涛 wrote:
> Dear xfs experts,
> I really need your help sincerely!!! In our production enviroment we
> run glusterfs over top of xfs on Dell x720D(Raid 6). And the xfs file
> system crash on some of the server frequently about every two weeks.
> Can you help to give me a direction about how to debug this issue and
> how to avoid it? Thank you very very much!

So this happens reliably, but infrequently? (only every 2 weeks or so?)

Can you provoke it any more often?

> uname -a
> Linux cqdx.miaoyan.cluster1.node11.qiyi.domain 2.6.32-279.el6.x86_64
> #1 SMP Wed Jun 13 18:24:36 EDT 2012 x86_64 x86_64 x86_64 GNU/Linux

That's a RHEL6 kernel; I'm assuming that this is a RHEL clone w/o RH support?

I agree with Ben that I'd like to see xfs_repair output.

Since the fs has shut down, you should unmount, remount, and unmount
again to replay the dirty log.  Then do xfs_repair -n, and provide the output
if it discovers any errors.

Thanks,
-Eric

> Every time the crash log is same, as following
> 
> 038 Apr  9 09:41:36 cqdx kernel: XFS (sdb): xfs_iunlink_remove:
> xfs_inotobp() returned error 22.
> 1039 Apr  9 09:41:36 cqdx kernel: XFS (sdb): xfs_inactive: xfs_ifree
> returned error 22
> 1040 Apr  9 09:41:36 cqdx kernel: XFS (sdb):
> xfs_do_force_shutdown(0x1) called from line 1184 of file
> fs/xfs/xfs_vnodeops.c.  Return address = 0xffffffffa02ee20a
> 1041 Apr  9 09:41:36 cqdx kernel: XFS (sdb): I/O Error Detected.
> Shutting down filesystem
> 1042 Apr  9 09:41:36 cqdx kernel: XFS (sdb): Please umount the
> filesystem and rectify the problem(s)
> 1043 Apr  9 09:41:53 cqdx kernel: XFS (sdb): xfs_log_force: error 5 returned.
> 1044 Apr  9 09:42:23 cqdx kernel: XFS (sdb): xfs_log_force: error 5 returned.
> 1045 Apr  9 09:42:53 cqdx kernel: XFS (sdb): xfs_log_force: error 5 returned.
> 1046 Apr  9 09:43:23 cqdx kernel: XFS (sdb): xfs_log_force: error 5 returned.
> 

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs