XFS umount with IO errors seems to lead to memory corruption

Chris Holcombe
2015-02-09T21:24:18Z

Hi Dave,

http://www.spinics.net/lists/linux-xfs/msg00061.html
Back in Dec 2013 you responded to this message saying that you would
take a look at it. Was a fix for this ever issued? I'm seeing very
similar stacktraces:

Feb 7 00:27:32 node008-cont001 kernel: [83405.490909] INFO: task
umount:29224 blocked for more than 120 seconds.
Feb 7 00:27:32 node008-cont001 kernel: [83405.498645] Tainted:
G W 3.13.0-39-generic #66-Ubuntu
Feb 7 00:27:32 node008-cont001 kernel: [83405.506273] "echo 0 >
/proc/sys/kernel/hung_task_timeout_secs" disables this message.
Feb 7 00:27:32 node008-cont001 kernel: [83405.515244] umount
D ffff880c4fc34480 0 29224 29221 0x00000082
Feb 7 00:27:32 node008-cont001 kernel: [83405.515253]
ffff880201211db0 0000000000000086 ffff880c39cb1800 ffff880201211fd8
Feb 7 00:27:32 node008-cont001 kernel: [83405.515260]
0000000000014480 0000000000014480 ffff880c39cb1800 ffff880c33386480
Feb 7 00:27:32 node008-cont001 kernel: [83405.515267]
ffff880c395e4bc8 ffff880c333864c0 ffff880c333864e8 ffff880c33386490
Feb 7 00:27:32 node008-cont001 kernel: [83405.515274] Call Trace:
Feb 7 00:27:32 node008-cont001 kernel: [83405.515298]
[<ffffffff81723109>] schedule+0x29/0x70
Feb 7 00:27:32 node008-cont001 kernel: [83405.515384]
[<ffffffffa023b0c9>] xfs_ail_push_all_sync+0xa9/0xe0 [xfs]
Feb 7 00:27:32 node008-cont001 kernel: [83405.515396]
[<ffffffff810aafd0>] ? prepare_to_wait_event+0x100/0x100
Feb 7 00:27:32 node008-cont001 kernel: [83405.515438]
[<ffffffffa0236f13>] xfs_log_quiesce+0x33/0x70 [xfs]
Feb 7 00:27:32 node008-cont001 kernel: [83405.515479]
[<ffffffffa0236f62>] xfs_log_unmount+0x12/0x30 [xfs]
Feb 7 00:27:32 node008-cont001 kernel: [83405.515510]
[<ffffffffa01ed846>] xfs_unmountfs+0xc6/0x150 [xfs]
Feb 7 00:27:32 node008-cont001 kernel: [83405.515541]
[<ffffffffa01ef211>] xfs_fs_put_super+0x21/0x60 [xfs]
Feb 7 00:27:32 node008-cont001 kernel: [83405.515550]
[<ffffffff811bf452>] generic_shutdown_super+0x72/0xf0
Feb 7 00:27:32 node008-cont001 kernel: [83405.515557]
[<ffffffff811bf707>] kill_block_super+0x27/0x70
Feb 7 00:27:32 node008-cont001 kernel: [83405.515565]
[<ffffffff811bf9ed>] deactivate_locked_super+0x3d/0x60
Feb 7 00:27:32 node008-cont001 kernel: [83405.515572]
[<ffffffff811bffa6>] deactivate_super+0x46/0x60
Feb 7 00:27:32 node008-cont001 kernel: [83405.515578]
[<ffffffff811dcd96>] mntput_no_expire+0xd6/0x170
Feb 7 00:27:32 node008-cont001 kernel: [83405.515584]
[<ffffffff811de31e>] SyS_umount+0x8e/0x100
Feb 7 00:27:32 node008-cont001 kernel: [83405.515591]
[<ffffffff8172f7ed>] system_call_fastpath+0x1a/0x1f

These type of errors are showing up in the logs:

Feb 7 00:27:34 node008-cont001 kernel: [83407.466853] XFS (dm-8):
metadata I/O error: block 0x0 ("xfs_buf_iodone_callbacks") error 19
numblks 1
...
Feb 7 00:27:39 node008-cont001 kernel: [83412.510982] XFS (dm-8):
metadata I/O error: block 0x0 ("xfs_buf_iodone_callbacks") error 19
numblks 1
Feb 7 00:27:44 node008-cont001 kernel: [83417.555152] XFS (dm-8):
metadata I/O error: block 0x0 ("xfs_buf_iodone_callbacks") error 19
numblks 1
...
Feb 7 00:27:54 node008-cont001 kernel: [83427.643428] XFS (dm-8):
metadata I/O error: block 0x0 ("xfs_buf_iodone_callbacks") error 19
numblks 1
Feb 7 00:27:57 node008-cont001 kernel: [83429.879442] XFS:: 568
callbacks suppressed
Feb 7 00:27:57 node008-cont001 kernel: [83429.879450] XFS (dm-8):
Detected failing async write on buffer block 0x0. Retrying async
write.
Feb 7 00:27:57 node008-cont001 kernel: [83429.879450]
Feb 7 00:27:57 node008-cont001 kernel: [83429.931438] XFS (dm-8):
Detected failing async write on buffer block 0x0. Retrying async
write.
Feb 7 00:27:57 node008-cont001 kernel: [83429.931438]
Feb 7 00:27:57 node008-cont001 kernel: [83429.983444] XFS (dm-8):
Detected failing async write on buffer block 0x0. Retrying async
write.
Feb 7 00:27:57 node008-cont001 kernel: [83429.983444]

Thanks for the help!
Chris