Re: xfs_iunlink_remove: xfs_inotobp() returned error 22 -- debugging

符永涛
2013-04-17T03:48:07Z

[-- Attachment #1.1: Type: text/plain, Size: 7579 bytes --]

Hi Brain,
Can I change as following?
--- a/xfs_inode.c
+++ b/xfs_inode.c
@@ -1773,6 +1773,8 @@ xfs_iunlink_remove(
if (last_ibp != NULL) {
xfs_trans_brelse(tp, last_ibp);
}
+ ASSERT(next_agino != NULLAGINO);
+ ASSERT(next_agino != 0);
next_ino = XFS_AGINO_TO_INO(mp, agno, next_agino);
error = xfs_inotobp(mp, tp, next_ino, &last_dip,
&last_ibp, &last_offset, 0);
@@ -1783,8 +1785,6 @@ xfs_iunlink_remove(
return error;
}
next_agino =
be32_to_cpu(last_dip->di_next_unlinked);
- ASSERT(next_agino != NULLAGINO);
- ASSERT(next_agino != 0);
}
/*
* Now last_ibp points to the buffer previous to us on

Thank you.

2013/4/17 符永涛 <yongtaofu@gmail.com>

> Hi Brain,
> If it is because NULLAGINO is passed in to xfs_inotobp().
> Can I move the following two lines before xfs_inotobp?
>
> For example:
>
> 1767 while (next_agino != agino) {
> 1768 /*
> 1769 * If the last inode wasn't the one pointing
> to
> 1770 * us, then release its buffer since we're not
> 1771 * going to do anything with it.
> 1772 */
> 1773 if (last_ibp != NULL) {
> 1774 xfs_trans_brelse(tp, last_ibp);
> 1775 }
> 1776 next_ino = XFS_AGINO_TO_INO(mp, agno,
> next_agino);
> + ASSERT(next_agino != NULLAGINO);
> + ASSERT(next_agino != 0);
> 1777 error = xfs_inotobp(mp, tp, next_ino,
> &last_dip,
> 1778 &last_ibp, &last_offset,
> 0);
> 1779 if (error) {
> 1780 xfs_warn(mp,
> 1781 "%s: xfs_inotobp() returned
> error %d.",
> 1782 __func__, error);
> 1783 return error;
> 1784 }
> 1785 next_agino =
> be32_to_cpu(last_dip->di_next_unlinked);
> - //ASSERT(next_agino != NULLAGINO);
> - //ASSERT(next_agino != 0);
> 1788 }
> I don't understand xfs well and correct me if I'm totally wrong.
> Thank you very much.
>
>
> 2013/4/17 符永涛 <yongtaofu@gmail.com>
>
>> Hi Brain,
>> I want to ask a question, according to the shutdown trace. The ino in xfs_iunlink_remove
>> is 0x113, why xfs_imap got ino=0xffffffff ?
>>
>> --- xfs_imap -- module("xfs").function("xfs_imap@fs
>> /xfs/xfs_ialloc.c:1257").return -- return=0x16
>> vars: mp=0xffff882017a50800 tp=0xffff881c81797c70 ino=0xffffffff
>>
>> --- xfs_iunlink_remove -- module("xfs").function("xfs_iunlink_remove@fs
>> /xfs/xfs_inode.c:1680").return -- return=0x16
>> vars: tp=0xffff881c81797c70 ip=0xffff881003c13c00 next_ino=? mp=? agi=?
>> dip=? agibp=0xffff880109b47e20 ibp=? agno=? agino=? next_agino=? last_ibp=?
>> last_dip=0xffff882000000000 bucket_index=? offset=?
>> last_offset=0xffffffffffff8810 error=? __func__=[...]
>> ip: i_ino = 0x113, i_flags = 0x0
>>
>> Thank you.
>>
>>
>>
>> 2013/4/17 符永涛 <yongtaofu@gmail.com>
>>
>>> Hi Brain,
>>> Thank you for your update, and I have applied your last kernel patch.
>>> However it is not easy to reproduce especially in out test environment.
>>> Till now is not happens again. I'll update the kernel patch now. BTW is
>>> there any findings in the logs of previous thread?
>>> http://oss.sgi.com/archives/xfs/2013-04/msg00327.html
>>> I guess it tend to happen during glusterfs rebalance because glusterfs
>>> moves a lot of file from one server to another and then unlink it.
>>>
>>> Thank you.
>>>
>>>
>>> 2013/4/17 Brian Foster <bfoster@redhat.com>
>>>
>>>> On 04/16/2013 12:24 PM, Dave Chinner wrote:
>>>> > On Mon, Apr 15, 2013 at 07:14:39PM -0400, Brian Foster wrote:
>>>> >> Hi,
>>>> >>
>>>> >> Thanks for the data in the previous thread:
>>>> >>
>>>> >> http://oss.sgi.com/archives/xfs/2013-04/msg00327.html
>>>> >>
>>>> ...
>>>> >>
>>>> >> echo 1 > /sys/kernel/debug/tracing/events/xfs/xfs_iunlink/enable
>>>> >> echo 1 >
>>>> /sys/kernel/debug/tracing/events/xfs/xfs_iunlink_remove/enable
>>>> >> ... reproduce ...
>>>> >> cat /sys/kernel/debug/tracing/trace > trace.output
>>>> >
>>>> > It's better to use trace-cmd for this. it will result in less
>>>> > dropped events. i.e.:
>>>> >
>>>> > $ trace-cmd record -e xfs_iunlink\*
>>>> > ... reproduce ...
>>>> > ^C
>>>> > $ trace-cmd report > trace.output
>>>> >
>>>> >> --- a/fs/xfs/linux-2.6/xfs_trace.h
>>>> >> +++ b/fs/xfs/linux-2.6/xfs_trace.h
>>>> >> @@ -581,6 +581,8 @@ DEFINE_INODE_EVENT(xfs_file_fsync);
>>>> ...
>>>> >
>>>> > I would suggest that the the tracing shoul dbe at entry of the
>>>> > function, otherwise we won't get a tracepoint for the operation that
>>>> > triggers the shutdown. (That's the reason most tracepoints in XFS
>>>> > are at function entry...)
>>>> >
>>>>
>>>> Good points, thanks Dave. A v2 that pulls up the tracepoints towards
>>>> function entry is appended.
>>>>
>>>> Brian
>>>>
>>>> From 280943e78ebe0b97a774cba51e7815c42f044b55 Mon Sep 17 00:00:00 2001
>>>> From: Brian Foster <bfoster@redhat.com>
>>>> Date: Mon, 15 Apr 2013 18:16:24 -0400
>>>> Subject: [PATCH v2] xfs: add tracepoints for xfs_iunlink and
>>>> xfs_iunlink_remove
>>>>
>>>> ---
>>>> fs/xfs/linux-2.6/xfs_trace.h | 2 ++
>>>> fs/xfs/xfs_inode.c | 4 ++++
>>>> 2 files changed, 6 insertions(+), 0 deletions(-)
>>>>
>>>> diff --git a/fs/xfs/linux-2.6/xfs_trace.h b/fs/xfs/linux-2.6/xfs_trace.h
>>>> index adc6ec4..338a0f9 100644
>>>> --- a/fs/xfs/linux-2.6/xfs_trace.h
>>>> +++ b/fs/xfs/linux-2.6/xfs_trace.h
>>>> @@ -583,6 +583,8 @@ DEFINE_INODE_EVENT(xfs_file_fsync);
>>>> DEFINE_INODE_EVENT(xfs_destroy_inode);
>>>> DEFINE_INODE_EVENT(xfs_dirty_inode);
>>>> DEFINE_INODE_EVENT(xfs_clear_inode);
>>>> +DEFINE_INODE_EVENT(xfs_iunlink);
>>>> +DEFINE_INODE_EVENT(xfs_iunlink_remove);
>>>>
>>>> DEFINE_INODE_EVENT(xfs_dquot_dqalloc);
>>>> DEFINE_INODE_EVENT(xfs_dquot_dqdetach);
>>>> diff --git a/fs/xfs/xfs_inode.c b/fs/xfs/xfs_inode.c
>>>> index 19900f0..d705c77 100644
>>>> --- a/fs/xfs/xfs_inode.c
>>>> +++ b/fs/xfs/xfs_inode.c
>>>> @@ -1615,6 +1615,8 @@ xfs_iunlink(
>>>>
>>>> mp = tp->t_mountp;
>>>>
>>>> + trace_xfs_iunlink(ip);
>>>> +
>>>> /*
>>>> * Get the agi buffer first. It ensures lock ordering
>>>> * on the list.
>>>> @@ -1694,6 +1696,8 @@ xfs_iunlink_remove(
>>>> mp = tp->t_mountp;
>>>> agno = XFS_INO_TO_AGNO(mp, ip->i_ino);
>>>>
>>>> + trace_xfs_iunlink_remove(ip);
>>>> +
>>>> /*
>>>> * Get the agi buffer first. It ensures lock ordering
>>>> * on the list.
>>>> --
>>>> 1.7.7.6
>>>>
>>>>
>>>
>>>
>>> --
>>> 符永涛
>>>
>>
>>
>>
>> --
>> 符永涛
>>
>
>
>
> --
> 符永涛
>

--
符永涛

[-- Attachment #1.2: Type: text/html, Size: 17050 bytes --]