* 3.8-rc5 xfs corruption [not found] <614607656.11545086.1359601991846.JavaMail.root@redhat.com> @ 2013-01-31 3:16 ` CAI Qian 0 siblings, 0 replies; 6+ messages in thread From: CAI Qian @ 2013-01-31 3:16 UTC (permalink / raw) To: xfs, linux-xfs; +Cc: linux-kernel Hello, (Sorry to post to xfs mailing lists but unsure about which one is the best for this.) I have seen something like this once during testing on a system with a EMC VNX FC/multipath back-end. Kernel config file is here, http://people.redhat.com/qcai/stable/.config [ 3025.063024] ffff8801a0d50000: 2e 2e 2f 2e 2e 2f 75 73 72 2f 6c 69 62 2f 6d 6f ../../usr/lib/mo [ 3025.071743] XFS (dm-1): Internal error xfs_bmbt_verify at line 747 of file fs/xfs/xfs_bmap_btree.c. Caller 0xffffffffa01c375e [ 3025.071743] [ 3025.084589] Pid: 275, comm: xfsaild/dm-1 Tainted: GF 3.8.0-rc5 #1 [ 3025.091636] Call Trace: [ 3025.094176] [<ffffffffa019910f>] xfs_error_report+0x3f/0x50 [xfs] [ 3025.100415] [<ffffffffa01c375e>] ? xfs_bmbt_write_verify+0xe/0x10 [xfs] [ 3025.107163] [<ffffffffa019917e>] xfs_corruption_error+0x5e/0x90 [xfs] [ 3025.113744] [<ffffffffa01c375e>] ? xfs_bmbt_write_verify+0xe/0x10 [xfs] [ 3025.120495] [<ffffffffa01c3606>] xfs_bmbt_verify+0x76/0x1c0 [xfs] [ 3025.126737] [<ffffffffa01c375e>] ? xfs_bmbt_write_verify+0xe/0x10 [xfs] [ 3025.133481] [<ffffffffa01975e5>] ? xfs_bdstrat_cb+0x65/0xd0 [xfs] [ 3025.139714] [<ffffffffa01c375e>] xfs_bmbt_write_verify+0xe/0x10 [xfs] [ 3025.146287] [<ffffffffa019713e>] _xfs_buf_ioapply+0x5e/0x370 [xfs] [ 3025.152565] [<ffffffff81097c70>] ? try_to_wake_up+0x2d0/0x2d0 [ 3025.158442] [<ffffffffa01975e5>] ? xfs_bdstrat_cb+0x65/0xd0 [xfs] [ 3025.164669] [<ffffffffa019752a>] xfs_buf_iorequest+0x4a/0xa0 [xfs] [ 3025.170979] [<ffffffffa01975e5>] xfs_bdstrat_cb+0x65/0xd0 [xfs] [ 3025.177032] [<ffffffffa01977c1>] __xfs_buf_delwri_submit+0x171/0x1e0 [xfs] [ 3025.184038] [<ffffffffa0198110>] ? xfs_buf_delwri_submit_nowait+0x20/0x30 [xfs] [ 3025.191491] [<ffffffffa01f1100>] ? xfs_trans_ail_cursor_first+0x80/0xb0 [xfs] [ 3025.198755] [<ffffffffa0198110>] xfs_buf_delwri_submit_nowait+0x20/0x30 [xfs] [ 3025.206030] [<ffffffffa01f1352>] xfsaild+0x222/0x5e0 [xfs] [ 3025.211665] [<ffffffffa01f1130>] ? xfs_trans_ail_cursor_first+0xb0/0xb0 [xfs] [ 3025.218890] [<ffffffff81085b10>] kthread+0xc0/0xd0 [ 3025.223784] [<ffffffff81085a50>] ? kthread_create_on_node+0x120/0x120 [ 3025.230317] [<ffffffff816176ec>] ret_from_fork+0x7c/0xb0 [ 3025.235721] [<ffffffff81085a50>] ? kthread_create_on_node+0x120/0x120 [ 3025.242248] XFS (dm-1): Corruption detected. Unmount and run xfs_repair [ 3025.248865] XFS (dm-1): xfs_do_force_shutdown(0x8) called from line 1340 of file fs/xfs/xfs_buf.c. Return address = 0xffffffffa0197411 Does this ring any bell? Regards, CAI Qian _______________________________________________ xfs mailing list xfs@oss.sgi.com http://oss.sgi.com/mailman/listinfo/xfs ^ permalink raw reply [flat|nested] 6+ messages in thread
* 3.8-rc5 xfs corruption @ 2013-01-31 3:16 ` CAI Qian 0 siblings, 0 replies; 6+ messages in thread From: CAI Qian @ 2013-01-31 3:16 UTC (permalink / raw) To: xfs, linux-xfs; +Cc: linux-kernel Hello, (Sorry to post to xfs mailing lists but unsure about which one is the best for this.) I have seen something like this once during testing on a system with a EMC VNX FC/multipath back-end. Kernel config file is here, http://people.redhat.com/qcai/stable/.config [ 3025.063024] ffff8801a0d50000: 2e 2e 2f 2e 2e 2f 75 73 72 2f 6c 69 62 2f 6d 6f ../../usr/lib/mo [ 3025.071743] XFS (dm-1): Internal error xfs_bmbt_verify at line 747 of file fs/xfs/xfs_bmap_btree.c. Caller 0xffffffffa01c375e [ 3025.071743] [ 3025.084589] Pid: 275, comm: xfsaild/dm-1 Tainted: GF 3.8.0-rc5 #1 [ 3025.091636] Call Trace: [ 3025.094176] [<ffffffffa019910f>] xfs_error_report+0x3f/0x50 [xfs] [ 3025.100415] [<ffffffffa01c375e>] ? xfs_bmbt_write_verify+0xe/0x10 [xfs] [ 3025.107163] [<ffffffffa019917e>] xfs_corruption_error+0x5e/0x90 [xfs] [ 3025.113744] [<ffffffffa01c375e>] ? xfs_bmbt_write_verify+0xe/0x10 [xfs] [ 3025.120495] [<ffffffffa01c3606>] xfs_bmbt_verify+0x76/0x1c0 [xfs] [ 3025.126737] [<ffffffffa01c375e>] ? xfs_bmbt_write_verify+0xe/0x10 [xfs] [ 3025.133481] [<ffffffffa01975e5>] ? xfs_bdstrat_cb+0x65/0xd0 [xfs] [ 3025.139714] [<ffffffffa01c375e>] xfs_bmbt_write_verify+0xe/0x10 [xfs] [ 3025.146287] [<ffffffffa019713e>] _xfs_buf_ioapply+0x5e/0x370 [xfs] [ 3025.152565] [<ffffffff81097c70>] ? try_to_wake_up+0x2d0/0x2d0 [ 3025.158442] [<ffffffffa01975e5>] ? xfs_bdstrat_cb+0x65/0xd0 [xfs] [ 3025.164669] [<ffffffffa019752a>] xfs_buf_iorequest+0x4a/0xa0 [xfs] [ 3025.170979] [<ffffffffa01975e5>] xfs_bdstrat_cb+0x65/0xd0 [xfs] [ 3025.177032] [<ffffffffa01977c1>] __xfs_buf_delwri_submit+0x171/0x1e0 [xfs] [ 3025.184038] [<ffffffffa0198110>] ? xfs_buf_delwri_submit_nowait+0x20/0x30 [xfs] [ 3025.191491] [<ffffffffa01f1100>] ? xfs_trans_ail_cursor_first+0x80/0xb0 [xfs] [ 3025.198755] [<ffffffffa0198110>] xfs_buf_delwri_submit_nowait+0x20/0x30 [xfs] [ 3025.206030] [<ffffffffa01f1352>] xfsaild+0x222/0x5e0 [xfs] [ 3025.211665] [<ffffffffa01f1130>] ? xfs_trans_ail_cursor_first+0xb0/0xb0 [xfs] [ 3025.218890] [<ffffffff81085b10>] kthread+0xc0/0xd0 [ 3025.223784] [<ffffffff81085a50>] ? kthread_create_on_node+0x120/0x120 [ 3025.230317] [<ffffffff816176ec>] ret_from_fork+0x7c/0xb0 [ 3025.235721] [<ffffffff81085a50>] ? kthread_create_on_node+0x120/0x120 [ 3025.242248] XFS (dm-1): Corruption detected. Unmount and run xfs_repair [ 3025.248865] XFS (dm-1): xfs_do_force_shutdown(0x8) called from line 1340 of file fs/xfs/xfs_buf.c. Return address = 0xffffffffa0197411 Does this ring any bell? Regards, CAI Qian ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: 3.8-rc5 xfs corruption 2013-01-31 3:16 ` CAI Qian @ 2013-01-31 4:07 ` Dave Chinner -1 siblings, 0 replies; 6+ messages in thread From: Dave Chinner @ 2013-01-31 4:07 UTC (permalink / raw) To: CAI Qian; +Cc: linux-xfs, linux-kernel, xfs On Wed, Jan 30, 2013 at 10:16:47PM -0500, CAI Qian wrote: > Hello, > > (Sorry to post to xfs mailing lists but unsure about which one is the > best for this.) Trimmed to just xfs@oss.sgi.com. > I have seen something like this once during testing on a system with a > EMC VNX FC/multipath back-end. This is a trace from the verifier code that was added in 3.8-rc1 so I doubt it has anything to do with any problem you've seen in the past.... Can you tell us what workload you were running and what hardware you are using as per: http://xfs.org/index.php/XFS_FAQ#Q:_What_information_should_I_include_when_reporting_a_problem.3F As it is, if you mounted the filesystem after this problem was detected, log recovery probably propagated it to disk. I'd suggest that you run xfs_repair -n on the device and post the output so we can see if any corruption has actaully made it to disk. If no corruption made it to disk, it's possible that we've got the incorrect verifier attached to the buffer. > [ 3025.063024] ffff8801a0d50000: 2e 2e 2f 2e 2e 2f 75 73 72 2f 6c 69 62 2f 6d 6f ../../usr/lib/mo The start of a block contains a path and the only type of block that can contain this format of metadata is remote symlink block. Remote symlink blocks don't have a verifier attached to them as there is nothing that can currently be used to verify them as correct. I can't see exactly how this can occur as stale buffers have the verifier ops cleared before being returned to the new user, and newly allocated xfs_bufs are zeroed before being initialised. I really need to know what you are doing to be able to get to the bottom of it.... Cheers, Dave. -- Dave Chinner david@fromorbit.com _______________________________________________ xfs mailing list xfs@oss.sgi.com http://oss.sgi.com/mailman/listinfo/xfs ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: 3.8-rc5 xfs corruption @ 2013-01-31 4:07 ` Dave Chinner 0 siblings, 0 replies; 6+ messages in thread From: Dave Chinner @ 2013-01-31 4:07 UTC (permalink / raw) To: CAI Qian; +Cc: xfs, linux-xfs, linux-kernel On Wed, Jan 30, 2013 at 10:16:47PM -0500, CAI Qian wrote: > Hello, > > (Sorry to post to xfs mailing lists but unsure about which one is the > best for this.) Trimmed to just xfs@oss.sgi.com. > I have seen something like this once during testing on a system with a > EMC VNX FC/multipath back-end. This is a trace from the verifier code that was added in 3.8-rc1 so I doubt it has anything to do with any problem you've seen in the past.... Can you tell us what workload you were running and what hardware you are using as per: http://xfs.org/index.php/XFS_FAQ#Q:_What_information_should_I_include_when_reporting_a_problem.3F As it is, if you mounted the filesystem after this problem was detected, log recovery probably propagated it to disk. I'd suggest that you run xfs_repair -n on the device and post the output so we can see if any corruption has actaully made it to disk. If no corruption made it to disk, it's possible that we've got the incorrect verifier attached to the buffer. > [ 3025.063024] ffff8801a0d50000: 2e 2e 2f 2e 2e 2f 75 73 72 2f 6c 69 62 2f 6d 6f ../../usr/lib/mo The start of a block contains a path and the only type of block that can contain this format of metadata is remote symlink block. Remote symlink blocks don't have a verifier attached to them as there is nothing that can currently be used to verify them as correct. I can't see exactly how this can occur as stale buffers have the verifier ops cleared before being returned to the new user, and newly allocated xfs_bufs are zeroed before being initialised. I really need to know what you are doing to be able to get to the bottom of it.... Cheers, Dave. -- Dave Chinner david@fromorbit.com ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: 3.8-rc5 xfs corruption 2013-01-31 4:07 ` Dave Chinner @ 2013-01-31 8:01 ` CAI Qian -1 siblings, 0 replies; 6+ messages in thread From: CAI Qian @ 2013-01-31 8:01 UTC (permalink / raw) To: Dave Chinner; +Cc: linux-xfs, linux-kernel, xfs ----- Original Message ----- > From: "Dave Chinner" <david@fromorbit.com> > To: "CAI Qian" <caiqian@redhat.com> > Cc: xfs@oss.sgi.com, linux-xfs@vger.kernel.org, "linux-kernel" <linux-kernel@vger.kernel.org> > Sent: Thursday, January 31, 2013 12:07:48 PM > Subject: Re: 3.8-rc5 xfs corruption > > On Wed, Jan 30, 2013 at 10:16:47PM -0500, CAI Qian wrote: > > Hello, > > > > (Sorry to post to xfs mailing lists but unsure about which one is > > the > > best for this.) > > Trimmed to just xfs@oss.sgi.com. Thanks for quick response, Dave. > > > I have seen something like this once during testing on a system > > with a > > EMC VNX FC/multipath back-end. > > This is a trace from the verifier code that was added in 3.8-rc1 so > I doubt it has anything to do with any problem you've seen in the > past.... > > Can you tell us what workload you were running and what hardware you > are using as per: > > http://xfs.org/index.php/XFS_FAQ#Q:_What_information_should_I_include_when_reporting_a_problem.3F This was the system, - AMD Opteron(tm) Processor 4130 (1 socket, 4 cores) - PowerEdge R415 - 8G memory - mptsas local disks Software version, - xfsprogs-3.1.10 The workload was running some fs_mark, syscalls tests, some nfs/cifs connectathon tests, memory, libhugetlbfs tests, and some dynamic debug (Documentation/dynamic-debug-howto.txt) tests. > > As it is, if you mounted the filesystem after this problem was > detected, log recovery probably propagated it to disk. I'd suggest > that you run xfs_repair -n on the device and post the output so we > can see if any corruption has actaully made it to disk. If no > corruption made it to disk, it's possible that we've got the > incorrect verifier attached to the buffer. The system was taken away from me, so I can only occupy it again later if needed. Regards, CAI Qian > > > [ 3025.063024] ffff8801a0d50000: 2e 2e 2f 2e 2e 2f 75 73 72 2f 6c > > 69 62 2f 6d 6f ../../usr/lib/mo > > The start of a block contains a path and the only > type of block that can contain this format of metadata is remote > symlink block. Remote symlink blocks don't have a verifier attached > to them as there is nothing that can currently be used to verify > them as correct. > > I can't see exactly how this can occur as stale buffers have the > verifier ops cleared before being returned to the new user, and > newly allocated xfs_bufs are zeroed before being initialised. I > really need to know what you are doing to be able to get to the > bottom of it.... > > Cheers, > > Dave. > -- > Dave Chinner > david@fromorbit.com > _______________________________________________ xfs mailing list xfs@oss.sgi.com http://oss.sgi.com/mailman/listinfo/xfs ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: 3.8-rc5 xfs corruption @ 2013-01-31 8:01 ` CAI Qian 0 siblings, 0 replies; 6+ messages in thread From: CAI Qian @ 2013-01-31 8:01 UTC (permalink / raw) To: Dave Chinner; +Cc: xfs, linux-xfs, linux-kernel ----- Original Message ----- > From: "Dave Chinner" <david@fromorbit.com> > To: "CAI Qian" <caiqian@redhat.com> > Cc: xfs@oss.sgi.com, linux-xfs@vger.kernel.org, "linux-kernel" <linux-kernel@vger.kernel.org> > Sent: Thursday, January 31, 2013 12:07:48 PM > Subject: Re: 3.8-rc5 xfs corruption > > On Wed, Jan 30, 2013 at 10:16:47PM -0500, CAI Qian wrote: > > Hello, > > > > (Sorry to post to xfs mailing lists but unsure about which one is > > the > > best for this.) > > Trimmed to just xfs@oss.sgi.com. Thanks for quick response, Dave. > > > I have seen something like this once during testing on a system > > with a > > EMC VNX FC/multipath back-end. > > This is a trace from the verifier code that was added in 3.8-rc1 so > I doubt it has anything to do with any problem you've seen in the > past.... > > Can you tell us what workload you were running and what hardware you > are using as per: > > http://xfs.org/index.php/XFS_FAQ#Q:_What_information_should_I_include_when_reporting_a_problem.3F This was the system, - AMD Opteron(tm) Processor 4130 (1 socket, 4 cores) - PowerEdge R415 - 8G memory - mptsas local disks Software version, - xfsprogs-3.1.10 The workload was running some fs_mark, syscalls tests, some nfs/cifs connectathon tests, memory, libhugetlbfs tests, and some dynamic debug (Documentation/dynamic-debug-howto.txt) tests. > > As it is, if you mounted the filesystem after this problem was > detected, log recovery probably propagated it to disk. I'd suggest > that you run xfs_repair -n on the device and post the output so we > can see if any corruption has actaully made it to disk. If no > corruption made it to disk, it's possible that we've got the > incorrect verifier attached to the buffer. The system was taken away from me, so I can only occupy it again later if needed. Regards, CAI Qian > > > [ 3025.063024] ffff8801a0d50000: 2e 2e 2f 2e 2e 2f 75 73 72 2f 6c > > 69 62 2f 6d 6f ../../usr/lib/mo > > The start of a block contains a path and the only > type of block that can contain this format of metadata is remote > symlink block. Remote symlink blocks don't have a verifier attached > to them as there is nothing that can currently be used to verify > them as correct. > > I can't see exactly how this can occur as stale buffers have the > verifier ops cleared before being returned to the new user, and > newly allocated xfs_bufs are zeroed before being initialised. I > really need to know what you are doing to be able to get to the > bottom of it.... > > Cheers, > > Dave. > -- > Dave Chinner > david@fromorbit.com > ^ permalink raw reply [flat|nested] 6+ messages in thread
end of thread, other threads:[~2013-01-31 8:01 UTC | newest]
Thread overview: 6+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
[not found] <614607656.11545086.1359601991846.JavaMail.root@redhat.com>
2013-01-31 3:16 ` 3.8-rc5 xfs corruption CAI Qian
2013-01-31 3:16 ` CAI Qian
2013-01-31 4:07 ` Dave Chinner
2013-01-31 4:07 ` Dave Chinner
2013-01-31 8:01 ` CAI Qian
2013-01-31 8:01 ` CAI Qian
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.