Re: xfs_iunlink_remove: xfs_inotobp() returned error 22 -- debugging

符永涛
2013-04-20T10:11:00Z

[-- Attachment #1.1: Type: text/plain, Size: 3823 bytes --]

Dear Eric,
I have applied your latest patch and collected the following log:

/var/log/message
Apr 20 17:28:23 10 kernel: XFS (sdb): xfs_iunlink_remove: xfs_inotobp()
returned error 22 for inode 0x1b20b ag 0 agino 1b20b
Apr 20 17:28:23 10 kernel:
Apr 20 17:28:23 10 kernel: XFS (sdb): xfs_inactive: xfs_ifree returned
error 22
Apr 20 17:28:23 10 kernel: XFS (sdb): xfs_do_force_shutdown(0x1) called
from line 1184 of file fs/xfs/xfs_vnodeops.c. Return address =
0xffffffffa02d4d0a
Apr 20 17:28:23 10 kernel: XFS (sdb): I/O Error Detected. Shutting down
filesystem
Apr 20 17:28:23 10 kernel: XFS (sdb): Please umount the filesystem and
rectify the problem(s)
Apr 20 17:28:37 10 kernel: XFS (sdb): xfs_log_force: error 5 returned.
Apr 20 17:29:07 10 kernel: XFS (sdb): xfs_log_force: error 5 returned.
Apr 20 17:29:37 10 kernel: XFS (sdb): xfs_log_force: error 5 returned.
Apr 20 17:30:07 10 kernel: XFS (sdb): xfs_log_force: error 5 returned.

debugfs trace:
https://docs.google.com/file/d/0B7n2C4T5tfNCTlZGUVpnZENrZ3M/edit?usp=sharing

Thank you.

2013/4/20 符永涛 <yongtaofu@gmail.com>

> Hi Eric,
> The xfs module is loaded from system kernel, it happens on our production
> server too (I did not touch that till now) and if the xfs module is mess up
> the systemstap may also not working but now it works. As you have
> mentioned, strange thing is xfs shutdown always happens when glusterfs
> rebalance completes.
>
>
> 2013/4/20 Eric Sandeen <sandeen@sandeen.net>
>
>> On 4/19/13 9:03 PM, 符永涛 wrote:
>> > Hi Eric,
>> > I will enable them and run test again. I can only reproduce it with
>> > glusterfs rebalance. Glusterfs uses a mechanism it called syncop to
>> > unlink file. For rebalance it uses
>> > syncop_unlink(glusterfs/libglusterfs/src/syncop.c). In the glusterfs
>> > sync_task framework(glusterfs/libglusterfs/src/syncop.c) it uses
>> > "makecontext/swapcontext"
>> > <
>> http://www.opengroup.org/onlinepubs/009695399/functions/makecontext.html
>> >.
>> > Does it leads to racing unlink from different CPU core?
>>
>> Yep, I understand that it's rebalance. It dies when rebalance finishes
>> because an
>> open but unlinked file trips over the corrupted list from earlier, it
>> seems.
>>
>> I don't know why makecontext would matter...
>>
>> Just to be sure, you are definitely loading the xfs module from the
>> kernel you built, right, and you don't have a "priority" module getting
>> loaded from elsewhere? Seems unlikely, but just to be sure.
>>
>> > Thank you.
>>
>> You could also add this patch to the xfs tracepoints to print more
>> information about the inodes - the mode & flags.
>>
>> -Eric
>>
>>
>> diff --git a/fs/xfs/linux-2.6/xfs_trace.h b/fs/xfs/linux-2.6/xfs_trace.h
>> index e8ce644..c314b87 100644
>> --- a/fs/xfs/linux-2.6/xfs_trace.h
>> +++ b/fs/xfs/linux-2.6/xfs_trace.h
>> @@ -544,14 +544,18 @@ DECLARE_EVENT_CLASS(xfs_inode_class,
>> TP_STRUCT__entry(
>> __field(dev_t, dev)
>> __field(xfs_ino_t, ino)
>> + __field(__u16, mode)
>> + __field(unsigned long, flags)
>> ),
>> TP_fast_assign(
>> __entry->dev = VFS_I(ip)->i_sb->s_dev;
>> __entry->ino = ip->i_ino;
>> + __entry->mode = VFS_I(ip)->i_mode;
>> + __entry->flags = ip->i_flags;
>> ),
>> - TP_printk("dev %d:%d ino 0x%llx",
>> + TP_printk("dev %d:%d ino 0x%llx mode 0%o, flags 0x%lx",
>> MAJOR(__entry->dev), MINOR(__entry->dev),
>> - __entry->ino)
>> + __entry->ino, __entry->mode, __entry->flags)
>> )
>>
>> #define DEFINE_INODE_EVENT(name) \
>>
>>
>>
>>
>
>
> --
> 符永涛
>

--
符永涛

[-- Attachment #1.2: Type: text/html, Size: 5561 bytes --]