Re: XFS umount with IO errors seems to lead to memory corruption

Dave Chinner
2013-11-21T22:07:24Z

[cc'd the correct xfs list. PLease use xfs@oss.sgi.com in future.]

On Thu, Nov 21, 2013 at 08:04:36PM +0200, Alex Lyakas wrote:
> Greetings,
> I am using stock XFS from kernel 3.8.13, compiled with kmemleak
> enabled. I am testing a particular scenario, in which the underlying
> block device returns IO errors during XFS umount. Almost in all cases
> this results in kernel crashes in various places, and sometimes
> kmemleak complains, and sometimes CPU soft lockup happens. It always
> happens after XFS messages like:

What testing are you doing?

http://xfs.org/index.php/XFS_FAQ#Q:_What_information_should_I_include_when_reporting_a_problem.3F

> kernel: [ 600.190509] XFS (dm-22): metadata I/O error: block
> 0x7600030 ("xlog_iodone") error 125 numblks 64
> kernel: [ 600.192267] XFS (dm-22): xfs_do_force_shutdown(0x2) called
> from line 1115 of file
> /mnt/compile/linux-stable/source/fs/xfs/xfs_log.c. Return address =
> 0xffffffffa05cffa1
> kernel: [ 600.192319] XFS (dm-22): Log I/O Error Detected. Shutting
> down filesystem
> kernel: [ 600.192392] XFS (dm-22): Unable to update superblock
> counters. Freespace may not be correct on next mount.
> kernel: [ 600.192398] XFS (dm-22): xfs_log_force: error 5 returned.
> kernel: [ 600.193687] XFS (º.Z): Please umount the filesystem and
> rectify the problem(s)
>
> you can see here the garbage that XFS prints instead of the block device name.
> In [1] and [2] I am attaching more kernel log from two such crashes.

So, something is corrupting memory and stamping all over the XFS
structures.

What's error 125?

#define ECANCELED 125 /* Operation Canceled */

I can't find an driver that actually returns that error to
filesystems, which....

> kernel: [ 600.227881] Modules linked in: xfs raid1 xfrm_user
> xfrm4_tunnel tunnel4 ipcomp xfrm_ipcomp esp4 ah4 scst_vdisk(O)
> iscsi_scst(O) scst(O) dm_zcache(O) dm_btrfs(O) btrfs(O) libcrc32c
> dm_iostat(O)

.... given you have a bunch of out of tree modules loaded (and some which
are experiemental) suggests that you have a problem with your
storage...

So, something is corrupting memory across a large number of
subsystems, and the trigger is some custom code to run error
injection. Can you reproduce the problem with something like
dm-faulty?

Cheers,

Dave.
--
Dave Chinner
david@fromorbit.com