Re: need help how to debug xfs crash issue xfs_iunlink_remove: xfs_inotobp() returned error 22

From: Ben Myers <bpm@sgi.com>
To: 符永涛 <yongtaofu@gmail.com>
Cc: xfs@oss.sgi.com
Subject: Re: need help how to debug xfs crash issue xfs_iunlink_remove: xfs_inotobp() returned error 22
Date: Tue, 9 Apr 2013 09:52:38 -0500	[thread overview]
Message-ID: <20130409145238.GE22182@sgi.com> (raw)
In-Reply-To: <CADFMGuKqkPbpcU=taqjTR4sA3o=w1LLAnKoEuj=OhJqEbQVijw@mail.gmail.com>

Hey Yongtaofu,

On Tue, Apr 09, 2013 at 09:05:32PM +0800, 符永涛 wrote:
> Also I want to know why all the server, all crash with the same crash stack?
> Thank you, really need your help.

What you've posted so far looks like evidence of a forced shutdown and not a
crash.  Is there a crash in addition to this forced shutdown?  If so, can you
post the stack for that too?

> 
> 2013/4/9, 符永涛 <yongtaofu@gmail.com>:
> > BTW
> > xfs_info /dev/sdb
> > meta-data=/dev/sdb               isize=256    agcount=28, agsize=268435440
> > blks
> >          =                       sectsz=512   attr=2
> > data     =                       bsize=4096   blocks=7324303360, imaxpct=5
> >          =                       sunit=16     swidth=160 blks
> > naming   =version 2              bsize=4096   ascii-ci=0
> > log      =internal               bsize=4096   blocks=521728, version=2
> >          =                       sectsz=512   sunit=16 blks, lazy-count=1
> > realtime =none                   extsz=4096   blocks=0, rtextents=0
> >
> > 2013/4/9, 符永涛 <yongtaofu@gmail.com>:
> >> Dear xfs experts,
> >> I really need your help sincerely!!! In our production enviroment we
> >> run glusterfs over top of xfs on Dell x720D(Raid 6). And the xfs file
> >> system crash on some of the server frequently about every two weeks.
> >> Can you help to give me a direction about how to debug this issue and
> >> how to avoid it? Thank you very very much!
> >>
> >> uname -a
> >> Linux cqdx.miaoyan.cluster1.node11.qiyi.domain 2.6.32-279.el6.x86_64
> >> #1 SMP Wed Jun 13 18:24:36 EDT 2012 x86_64 x86_64 x86_64 GNU/Linux
> >>
> >> Every time the crash log is same, as following

An initial guess is that somehow it is looking up a bad inode number, e.g. it
is beyond the end of the filesystem and xfs_dilocate returns EINVAL.

You could 'xfs_repair -n' to see what it finds (without modifying the
filesystem) as a first step.

> >> 038 Apr  9 09:41:36 cqdx kernel: XFS (sdb): xfs_iunlink_remove:
> >> xfs_inotobp() returned error 22.

Were there any lines of output before this?  In some codebases there are prints
in xfs_inotobp that would help show what happened.

> >> 1039 Apr  9 09:41:36 cqdx kernel: XFS (sdb): xfs_inactive: xfs_ifree
> >> returned error 22
> >> 1040 Apr  9 09:41:36 cqdx kernel: XFS (sdb):
> >> xfs_do_force_shutdown(0x1) called from line 1184 of file
> >> fs/xfs/xfs_vnodeops.c.  Return address = 0xffffffffa02ee20a
> >> 1041 Apr  9 09:41:36 cqdx kernel: XFS (sdb): I/O Error Detected.
> >> Shutting down filesystem
> >> 1042 Apr  9 09:41:36 cqdx kernel: XFS (sdb): Please umount the
> >> filesystem and rectify the problem(s)
> >> 1043 Apr  9 09:41:53 cqdx kernel: XFS (sdb): xfs_log_force: error 5
> >> returned.
> >> 1044 Apr  9 09:42:23 cqdx kernel: XFS (sdb): xfs_log_force: error 5
> >> returned.
> >> 1045 Apr  9 09:42:53 cqdx kernel: XFS (sdb): xfs_log_force: error 5
> >> returned.
> >> 1046 Apr  9 09:43:23 cqdx kernel: XFS (sdb): xfs_log_force: error 5
> >> returned.

The error 5 (EIO) look scary but they are due to the forced shutdown, don't
worry about them.

Thanks,
	Ben

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs