XFS_WANT_CORRUPTED_GOTO report

XFS_WANT_CORRUPTED_GOTO report

[Iustin Pop]

Hi,

I searched the list but didn't find any reports of
XFS_WANT_CORRUPTED_GOTO in xfs_bmap_add_extent_unwritten_real, so here
it goes. My kernel is tainted as I use nvidia's binary driver, so if I'm
told to go away I understand :) Otherwise it's a self compiled amd64
kernel on debian unstable.

The filesystem in question was recently grown, and I did on a file:
xfs_io disk0.img
resvp 0 2G
truncate 8G

(not with G but with the actual numbers). Then I proceeded to write into
this file (it was used as a qemu disk image) and at some point:

XFS internal error XFS_WANT_CORRUPTED_GOTO at line 2058 of file fs/xfs/xfs_bmap_btree.c.  Caller 0xffffffff80318a80
Pid: 281, comm: xfsdatad/1 Tainted: P        2.6.24.3-teal #1

Call Trace:
 [<ffffffff80318a80>] xfs_bmap_add_extent_unwritten_real+0x710/0xce0
 [<ffffffff80323fad>] xfs_bmbt_insert+0x14d/0x150
 [<ffffffff80318a80>] xfs_bmap_add_extent_unwritten_real+0x710/0xce0
 [<ffffffff8031b537>] xfs_bmap_add_extent+0x147/0x440
 [<ffffffff8033a329>] xfs_iext_get_ext+0x49/0x80
 [<ffffffff80324375>] xfs_btree_init_cursor+0x45/0x220
 [<ffffffff8031ef71>] xfs_bmapi+0xc31/0x1360
 [<ffffffff80346258>] xlog_grant_log_space+0x298/0x2e0
 [<ffffffff80350d48>] xfs_trans_reserve+0xa8/0x210
 [<ffffffff803409eb>] xfs_iomap_write_unwritten+0x14b/0x220
 [<ffffffff803405ba>] xfs_iomap+0x25a/0x390
 [<ffffffff805081ee>] thread_return+0x3a/0x56c
 [<ffffffff8035da00>] xfs_end_bio_unwritten+0x0/0x40
 [<ffffffff8035da2f>] xfs_end_bio_unwritten+0x2f/0x40
 [<ffffffff80249a5c>] run_workqueue+0xcc/0x170
 [<ffffffff8024a5f0>] worker_thread+0x0/0x110
 [<ffffffff8024a5f0>] worker_thread+0x0/0x110
 [<ffffffff8024a693>] worker_thread+0xa3/0x110
 [<ffffffff8024e1e0>] autoremove_wake_function+0x0/0x30
 [<ffffffff8024a5f0>] worker_thread+0x0/0x110
 [<ffffffff8024a5f0>] worker_thread+0x0/0x110
 [<ffffffff8024de1b>] kthread+0x4b/0x80
 [<ffffffff8020cac8>] child_rip+0xa/0x12
 [<ffffffff8024ddd0>] kthread+0x0/0x80
 [<ffffffff8020cabe>] child_rip+0x0/0x12

Filesystem "dm-4": XFS internal error xfs_trans_cancel at line 1163 of file fs/xfs/xfs_trans.c.  Caller 0xffffffff80340a9b
Pid: 281, comm: xfsdatad/1 Tainted: P        2.6.24.3-teal #1

Call Trace:
 [<ffffffff80340a9b>] xfs_iomap_write_unwritten+0x1fb/0x220
 [<ffffffff803515d4>] xfs_trans_cancel+0x104/0x130
 [<ffffffff80340a9b>] xfs_iomap_write_unwritten+0x1fb/0x220
 [<ffffffff803405ba>] xfs_iomap+0x25a/0x390
 [<ffffffff805081ee>] thread_return+0x3a/0x56c
 [<ffffffff8035da00>] xfs_end_bio_unwritten+0x0/0x40
 [<ffffffff8035da2f>] xfs_end_bio_unwritten+0x2f/0x40
 [<ffffffff80249a5c>] run_workqueue+0xcc/0x170
 [<ffffffff8024a5f0>] worker_thread+0x0/0x110
 [<ffffffff8024a5f0>] worker_thread+0x0/0x110
 [<ffffffff8024a693>] worker_thread+0xa3/0x110
 [<ffffffff8024e1e0>] autoremove_wake_function+0x0/0x30
 [<ffffffff8024a5f0>] worker_thread+0x0/0x110
 [<ffffffff8024a5f0>] worker_thread+0x0/0x110
 [<ffffffff8024de1b>] kthread+0x4b/0x80
 [<ffffffff8020cac8>] child_rip+0xa/0x12
 [<ffffffff8024ddd0>] kthread+0x0/0x80
 [<ffffffff8020cabe>] child_rip+0x0/0x12

xfs_force_shutdown(dm-4,0x8) called from line 1164 of file fs/xfs/xfs_trans.c.  Return address = 0xffffffff803515ed
Filesystem "dm-4": Corruption of in-memory data detected.  Shutting down filesystem: dm-4
Please umount the filesystem, and rectify the problem(s)


xfs_repair didn't say anything related to corruption, mounting it just
said starting recovery... ending recovery.

After mount, the file in question is heavily fragmented (around 1600
segments). I'm not sure if this file caused the corruption, but I'm
almost certain, as no other traffic should have been at that time.

I also have a metadump (run before recovery) and a full copy of the
filesystem if it's useful.

thanks,
iustin

Re: XFS_WANT_CORRUPTED_GOTO report

[Lachlan McIlroy]

Iustin Pop wrote:
> Hi,
> 
> I searched the list but didn't find any reports of
> XFS_WANT_CORRUPTED_GOTO in xfs_bmap_add_extent_unwritten_real, so here
> it goes. My kernel is tainted as I use nvidia's binary driver, so if I'm
> told to go away I understand :) Otherwise it's a self compiled amd64
> kernel on debian unstable.
> 
> The filesystem in question was recently grown, and I did on a file:
> xfs_io disk0.img
> resvp 0 2G
> truncate 8G
> 
> (not with G but with the actual numbers). Then I proceeded to write into
> this file (it was used as a qemu disk image) and at some point:
> 
> XFS internal error XFS_WANT_CORRUPTED_GOTO at line 2058 of file fs/xfs/xfs_bmap_btree.c.  Caller 0xffffffff80318a80
> Pid: 281, comm: xfsdatad/1 Tainted: P        2.6.24.3-teal #1
> 
> Call Trace:
>  [<ffffffff80318a80>] xfs_bmap_add_extent_unwritten_real+0x710/0xce0
>  [<ffffffff80323fad>] xfs_bmbt_insert+0x14d/0x150
>  [<ffffffff80318a80>] xfs_bmap_add_extent_unwritten_real+0x710/0xce0
>  [<ffffffff8031b537>] xfs_bmap_add_extent+0x147/0x440
>  [<ffffffff8033a329>] xfs_iext_get_ext+0x49/0x80
>  [<ffffffff80324375>] xfs_btree_init_cursor+0x45/0x220
>  [<ffffffff8031ef71>] xfs_bmapi+0xc31/0x1360
>  [<ffffffff80346258>] xlog_grant_log_space+0x298/0x2e0
>  [<ffffffff80350d48>] xfs_trans_reserve+0xa8/0x210
>  [<ffffffff803409eb>] xfs_iomap_write_unwritten+0x14b/0x220
>  [<ffffffff803405ba>] xfs_iomap+0x25a/0x390
>  [<ffffffff805081ee>] thread_return+0x3a/0x56c
>  [<ffffffff8035da00>] xfs_end_bio_unwritten+0x0/0x40
>  [<ffffffff8035da2f>] xfs_end_bio_unwritten+0x2f/0x40
>  [<ffffffff80249a5c>] run_workqueue+0xcc/0x170
>  [<ffffffff8024a5f0>] worker_thread+0x0/0x110
>  [<ffffffff8024a5f0>] worker_thread+0x0/0x110
>  [<ffffffff8024a693>] worker_thread+0xa3/0x110
>  [<ffffffff8024e1e0>] autoremove_wake_function+0x0/0x30
>  [<ffffffff8024a5f0>] worker_thread+0x0/0x110
>  [<ffffffff8024a5f0>] worker_thread+0x0/0x110
>  [<ffffffff8024de1b>] kthread+0x4b/0x80
>  [<ffffffff8020cac8>] child_rip+0xa/0x12
>  [<ffffffff8024ddd0>] kthread+0x0/0x80
>  [<ffffffff8020cabe>] child_rip+0x0/0x12
> 
> Filesystem "dm-4": XFS internal error xfs_trans_cancel at line 1163 of file fs/xfs/xfs_trans.c.  Caller 0xffffffff80340a9b
> Pid: 281, comm: xfsdatad/1 Tainted: P        2.6.24.3-teal #1
> 
> Call Trace:
>  [<ffffffff80340a9b>] xfs_iomap_write_unwritten+0x1fb/0x220
>  [<ffffffff803515d4>] xfs_trans_cancel+0x104/0x130
>  [<ffffffff80340a9b>] xfs_iomap_write_unwritten+0x1fb/0x220
>  [<ffffffff803405ba>] xfs_iomap+0x25a/0x390
>  [<ffffffff805081ee>] thread_return+0x3a/0x56c
>  [<ffffffff8035da00>] xfs_end_bio_unwritten+0x0/0x40
>  [<ffffffff8035da2f>] xfs_end_bio_unwritten+0x2f/0x40
>  [<ffffffff80249a5c>] run_workqueue+0xcc/0x170
>  [<ffffffff8024a5f0>] worker_thread+0x0/0x110
>  [<ffffffff8024a5f0>] worker_thread+0x0/0x110
>  [<ffffffff8024a693>] worker_thread+0xa3/0x110
>  [<ffffffff8024e1e0>] autoremove_wake_function+0x0/0x30
>  [<ffffffff8024a5f0>] worker_thread+0x0/0x110
>  [<ffffffff8024a5f0>] worker_thread+0x0/0x110
>  [<ffffffff8024de1b>] kthread+0x4b/0x80
>  [<ffffffff8020cac8>] child_rip+0xa/0x12
>  [<ffffffff8024ddd0>] kthread+0x0/0x80
>  [<ffffffff8020cabe>] child_rip+0x0/0x12
> 
> xfs_force_shutdown(dm-4,0x8) called from line 1164 of file fs/xfs/xfs_trans.c.  Return address = 0xffffffff803515ed
> Filesystem "dm-4": Corruption of in-memory data detected.  Shutting down filesystem: dm-4
> Please umount the filesystem, and rectify the problem(s)
> 
> 
> xfs_repair didn't say anything related to corruption, mounting it just
> said starting recovery... ending recovery.
That reinforces the message above that the corruption was in-memory and
that the on-disk version is good.

> 
> After mount, the file in question is heavily fragmented (around 1600
> segments). I'm not sure if this file caused the corruption, but I'm
> almost certain, as no other traffic should have been at that time.
The file being written to (that caused the panic) has unwritten extents
and we were trying to convert the extents from unwritten to real after
writing to them.  These XFS_WANT_CORRUPTED_GOTO bugs often occur with
extent tree corruption so this is not surprising.  Could we get output
from xfs_bmap -v on this file?

> 
> I also have a metadump (run before recovery) and a full copy of the
> filesystem if it's useful.
Can we get a copy of that metadump?  I don't hold high hopes for it
though - the filesystem can be inconsistent until the log is replayed
but after the log was replayed the problem was gone.  I don't suppose
you have a copy of the log?

Re: XFS_WANT_CORRUPTED_GOTO report

[Iustin Pop]

On Mon, Mar 03, 2008 at 12:46:38PM +1100, Lachlan McIlroy wrote:
>> After mount, the file in question is heavily fragmented (around 1600
>> segments). I'm not sure if this file caused the corruption, but I'm
>> almost certain, as no other traffic should have been at that time.
> The file being written to (that caused the panic) has unwritten extents
> and we were trying to convert the extents from unwritten to real after
> writing to them.  These XFS_WANT_CORRUPTED_GOTO bugs often occur with
> extent tree corruption so this is not surprising.  Could we get output
> from xfs_bmap -v on this file?
http://www.k1024.org/iusty/plains.bmap

>> I also have a metadump (run before recovery) and a full copy of the
>> filesystem if it's useful.
> Can we get a copy of that metadump?  I don't hold high hopes for it
> though - the filesystem can be inconsistent until the log is replayed
> but after the log was replayed the problem was gone.  I don't suppose
> you have a copy of the log?

The metadump is here http://www.k1024.org/iusty/plains.metadump.bz2
(~80MB), done with default options. Warning, the server is somewhat slow
:)

The copy of the log, done with xfs_logprint -t, is here
http://www.k1024.org/iusty/plains.logprint

The inode of the file in question is 96424401.

Let me know if I can give more information.

thanks,
iustin