3.8-rc5 xfs corruption

3.8-rc5 xfs corruption

[CAI Qian]

Hello,

(Sorry to post to xfs mailing lists but unsure about which one is the
best for this.)

I have seen something like this once during testing on a system with a
EMC VNX FC/multipath back-end. Kernel config file is here,
http://people.redhat.com/qcai/stable/.config

[ 3025.063024] ffff8801a0d50000: 2e 2e 2f 2e 2e 2f 75 73 72 2f 6c 69 62 2f 6d 6f  ../../usr/lib/mo 
[ 3025.071743] XFS (dm-1): Internal error xfs_bmbt_verify at line 747 of file fs/xfs/xfs_bmap_btree.c.  Caller 0xffffffffa01c375e 
[ 3025.071743]  
[ 3025.084589] Pid: 275, comm: xfsaild/dm-1 Tainted: GF            3.8.0-rc5 #1 
[ 3025.091636] Call Trace: 
[ 3025.094176]  [<ffffffffa019910f>] xfs_error_report+0x3f/0x50 [xfs] 
[ 3025.100415]  [<ffffffffa01c375e>] ? xfs_bmbt_write_verify+0xe/0x10 [xfs] 
[ 3025.107163]  [<ffffffffa019917e>] xfs_corruption_error+0x5e/0x90 [xfs] 
[ 3025.113744]  [<ffffffffa01c375e>] ? xfs_bmbt_write_verify+0xe/0x10 [xfs] 
[ 3025.120495]  [<ffffffffa01c3606>] xfs_bmbt_verify+0x76/0x1c0 [xfs] 
[ 3025.126737]  [<ffffffffa01c375e>] ? xfs_bmbt_write_verify+0xe/0x10 [xfs] 
[ 3025.133481]  [<ffffffffa01975e5>] ? xfs_bdstrat_cb+0x65/0xd0 [xfs] 
[ 3025.139714]  [<ffffffffa01c375e>] xfs_bmbt_write_verify+0xe/0x10 [xfs] 
[ 3025.146287]  [<ffffffffa019713e>] _xfs_buf_ioapply+0x5e/0x370 [xfs] 
[ 3025.152565]  [<ffffffff81097c70>] ? try_to_wake_up+0x2d0/0x2d0 
[ 3025.158442]  [<ffffffffa01975e5>] ? xfs_bdstrat_cb+0x65/0xd0 [xfs] 
[ 3025.164669]  [<ffffffffa019752a>] xfs_buf_iorequest+0x4a/0xa0 [xfs] 
[ 3025.170979]  [<ffffffffa01975e5>] xfs_bdstrat_cb+0x65/0xd0 [xfs] 
[ 3025.177032]  [<ffffffffa01977c1>] __xfs_buf_delwri_submit+0x171/0x1e0 [xfs] 
[ 3025.184038]  [<ffffffffa0198110>] ? xfs_buf_delwri_submit_nowait+0x20/0x30 [xfs] 
[ 3025.191491]  [<ffffffffa01f1100>] ? xfs_trans_ail_cursor_first+0x80/0xb0 [xfs] 
[ 3025.198755]  [<ffffffffa0198110>] xfs_buf_delwri_submit_nowait+0x20/0x30 [xfs] 
[ 3025.206030]  [<ffffffffa01f1352>] xfsaild+0x222/0x5e0 [xfs] 
[ 3025.211665]  [<ffffffffa01f1130>] ? xfs_trans_ail_cursor_first+0xb0/0xb0 [xfs] 
[ 3025.218890]  [<ffffffff81085b10>] kthread+0xc0/0xd0 
[ 3025.223784]  [<ffffffff81085a50>] ? kthread_create_on_node+0x120/0x120 
[ 3025.230317]  [<ffffffff816176ec>] ret_from_fork+0x7c/0xb0 
[ 3025.235721]  [<ffffffff81085a50>] ? kthread_create_on_node+0x120/0x120 
[ 3025.242248] XFS (dm-1): Corruption detected. Unmount and run xfs_repair 
[ 3025.248865] XFS (dm-1): xfs_do_force_shutdown(0x8) called from line 1340 of file fs/xfs/xfs_buf.c.  Return address = 0xffffffffa0197411 

Does this ring any bell?

Regards,
CAI Qian

Re: 3.8-rc5 xfs corruption

[Dave Chinner]

On Wed, Jan 30, 2013 at 10:16:47PM -0500, CAI Qian wrote:
> Hello,
> 
> (Sorry to post to xfs mailing lists but unsure about which one is the
> best for this.)

Trimmed to just xfs@oss.sgi.com.

> I have seen something like this once during testing on a system with a
> EMC VNX FC/multipath back-end.

This is a trace from the verifier code that was added in 3.8-rc1 so
I doubt it has anything to do with any problem you've seen in the
past....

Can you tell us what workload you were running and what hardware you
are using as per:

http://xfs.org/index.php/XFS_FAQ#Q:_What_information_should_I_include_when_reporting_a_problem.3F

As it is, if you mounted the filesystem after this problem was
detected, log recovery probably propagated it to disk. I'd suggest
that you run xfs_repair -n on the device and post the output so we
can see if any corruption has actaully made it to disk. If no
corruption made it to disk, it's possible that we've got the
incorrect verifier attached to the buffer.

> [ 3025.063024] ffff8801a0d50000: 2e 2e 2f 2e 2e 2f 75 73 72 2f 6c 69 62 2f 6d 6f  ../../usr/lib/mo 

The start of a block contains a path and the only
type of block that can contain this format of metadata is remote
symlink block. Remote symlink blocks don't have a verifier attached
to them as there is nothing that can currently be used to verify
them as correct.

I can't see exactly how this can occur as stale buffers have the
verifier ops cleared before being returned to the new user, and
newly allocated xfs_bufs are zeroed before being initialised. I
really need to know what you are doing to be able to get to the
bottom of it....

Cheers,

Dave.
-- 
Dave Chinner
david@fromorbit.com

Re: 3.8-rc5 xfs corruption

[CAI Qian]

----- Original Message -----
> From: "Dave Chinner" <david@fromorbit.com>
> To: "CAI Qian" <caiqian@redhat.com>
> Cc: xfs@oss.sgi.com, linux-xfs@vger.kernel.org, "linux-kernel" <linux-kernel@vger.kernel.org>
> Sent: Thursday, January 31, 2013 12:07:48 PM
> Subject: Re: 3.8-rc5 xfs corruption
> 
> On Wed, Jan 30, 2013 at 10:16:47PM -0500, CAI Qian wrote:
> > Hello,
> > 
> > (Sorry to post to xfs mailing lists but unsure about which one is
> > the
> > best for this.)
> 
> Trimmed to just xfs@oss.sgi.com.
Thanks for quick response, Dave.
> 
> > I have seen something like this once during testing on a system
> > with a
> > EMC VNX FC/multipath back-end.
> 
> This is a trace from the verifier code that was added in 3.8-rc1 so
> I doubt it has anything to do with any problem you've seen in the
> past....
> 
> Can you tell us what workload you were running and what hardware you
> are using as per:
> 
> http://xfs.org/index.php/XFS_FAQ#Q:_What_information_should_I_include_when_reporting_a_problem.3F
This was the system,
- AMD Opteron(tm) Processor 4130 (1 socket, 4 cores)
- PowerEdge R415 
- 8G memory
- mptsas local disks

Software version,
- xfsprogs-3.1.10

The workload was running some fs_mark, syscalls tests, some nfs/cifs
connectathon tests, memory, libhugetlbfs tests, and some dynamic debug
(Documentation/dynamic-debug-howto.txt) tests.
> 
> As it is, if you mounted the filesystem after this problem was
> detected, log recovery probably propagated it to disk. I'd suggest
> that you run xfs_repair -n on the device and post the output so we
> can see if any corruption has actaully made it to disk. If no
> corruption made it to disk, it's possible that we've got the
> incorrect verifier attached to the buffer.
The system was taken away from me, so I can only occupy it again later
if needed.

Regards,
CAI Qian
> 
> > [ 3025.063024] ffff8801a0d50000: 2e 2e 2f 2e 2e 2f 75 73 72 2f 6c
> > 69 62 2f 6d 6f  ../../usr/lib/mo
> 
> The start of a block contains a path and the only
> type of block that can contain this format of metadata is remote
> symlink block. Remote symlink blocks don't have a verifier attached
> to them as there is nothing that can currently be used to verify
> them as correct.
> 
> I can't see exactly how this can occur as stale buffers have the
> verifier ops cleared before being returned to the new user, and
> newly allocated xfs_bufs are zeroed before being initialised. I
> really need to know what you are doing to be able to get to the
> bottom of it....
> 
> Cheers,
> 
> Dave.
> --
> Dave Chinner
> david@fromorbit.com
>