* Re: list_add corruption after "xfs: mode di_mode to vfs inode" [not found] <56FC9FA6.1080700@stratus.com> @ 2016-04-01 2:44 ` Dave Chinner 2016-04-12 7:54 ` Dave Chinner 0 siblings, 1 reply; 4+ messages in thread From: Dave Chinner @ 2016-04-01 2:44 UTC (permalink / raw) To: Joe Lawrence; +Cc: xfs On Wed, Mar 30, 2016 at 11:55:18PM -0400, Joe Lawrence wrote: > Hi Dave, > > Upon loading 4.6-rc1, I noticed a few linked list corruption messages in > dmesg shortly after boot up. I bisected the kernel, landing on: > > [c19b3b05ae440de50fffe2ac2a9b27392a7448e9] xfs: mode di_mode to vfs inode > > If I revert c19b3b05ae44 from 4.6-rc1, the warnings stop. > > WARNING: CPU: 35 PID: 6715 at lib/list_debug.c:29 __list_add+0x65/0xc0 > list_add corruption. next->prev should be prev (ffff882030928a00), but was ffff88103f00c300. (next=ffff88100fde5ce8). ..... > [<ffffffff812488f0>] ? bdev_test+0x20/0x20 > [<ffffffff813551a5>] __list_add+0x65/0xc0 > [<ffffffff81249bd8>] bd_acquire+0xc8/0xd0 > [<ffffffff8124aa59>] blkdev_open+0x39/0x70 > [<ffffffff8120bc27>] do_dentry_open+0x227/0x320 > [<ffffffff8124aa20>] ? blkdev_get_by_dev+0x50/0x50 > [<ffffffff8120d057>] vfs_open+0x57/0x60 > [<ffffffff8121c9fa>] path_openat+0x1ba/0x1340 > [<ffffffff8121eff1>] do_filp_open+0x91/0x100 > [<ffffffff8122c806>] ? __alloc_fd+0x46/0x180 > [<ffffffff8120d3b4>] do_sys_open+0x124/0x210 > [<ffffffff8120d4be>] SyS_open+0x1e/0x20 > [<ffffffff81003c12>] do_syscall_64+0x62/0x110 > [<ffffffff8169ade1>] entry_SYSCALL64_slow_path+0x25/0x25 .... > According to the bd_acquire+0xc8 offset, we're in bd_acquire() > attempting the list add: .... > 713 bdev = bdget(inode->i_rdev); > 714 if (bdev) { > 715 spin_lock(&bdev_lock); > 716 if (!inode->i_bdev) { > 717 /* > 718 * We take an additional reference to bd_inode, > 719 * and it's released in clear_inode() of inode. > 720 * So, we can access it via ->i_mapping always > 721 * without igrab(). > 722 */ > 723 bdgrab(bdev); > 724 inode->i_bdev = bdev; > 725 inode->i_mapping = bdev->bd_inode->i_mapping; > 726 list_add(&inode->i_devices, &bdev->bd_inodes); So the bdev->bd_inodes list is corrupt, and this call trace is just the messenger. > crash> ps -a | grep mdadm > ... > PID: 6715 TASK: ffff882033ac2d40 CPU: 35 COMMAND: "mdadm" > ARG: /sbin/mdadm --detail --export /var/opt/ft/osm/osm_temporary_md_device_node > ... > > I traced the proprietary-driver-dependent user program to figure out > what it was doing and boiled that down to a repro that hits the same > corruption when running *stock* 4.6-rc1. (Note /tmp is hosted on an > XFS volume): > > -- > > MD=/dev/md1 > LOOP_A=/dev/loop0 > LOOP_B=/dev/loop1 > TMP_A=/tmp/diska > TMP_B=/tmp/diskb > > echo > echo Setting up ... > > dd if=/dev/zero of=$TMP_A bs=1M count=200 > dd if=/dev/zero of=$TMP_B bs=1M count=200 > losetup $LOOP_A $TMP_A > losetup $LOOP_B $TMP_B > > mdadm --create $MD \ > --metadata=1 \ > --level=1 \ > --raid-devices=2 \ > --bitmap=internal \ > $LOOP_A $LOOP_B > > MAJOR=$(stat -c %t $MD) > MINOR=$(stat -c %T $MD) > > echo > echo Testing major: $MAJOR minor: $MINOR ... > > for i in {0..100}; do > mknod --mode=0600 /tmp/tmp_node b $MAJOR $MINOR > mdadm --detail --export /tmp/tmp_node > rm -f /tmp/tmp_node > done > > echo > echo Cleanup ... > > mdadm --stop $MD > losetup -d $LOOP_A $LOOP_B > rm -f $TMP_A $TMP_B > > echo > echo Done. > > -- > > I'm not really sure why the bisect landed on c19b3b05ae44 "xfs: mode > di_mode to vfs inode", but as I mentioned, reverting it made the list > warnings go away. Neither am I at this point as it's the bdev inode (not an xfs inode) that has a corrupted list. I'll have to try to reproduce this. Cheers, Dave. -- Dave Chinner dchinner@redhat.com _______________________________________________ xfs mailing list xfs@oss.sgi.com http://oss.sgi.com/mailman/listinfo/xfs ^ permalink raw reply [flat|nested] 4+ messages in thread
* Re: list_add corruption after "xfs: mode di_mode to vfs inode" 2016-04-01 2:44 ` list_add corruption after "xfs: mode di_mode to vfs inode" Dave Chinner @ 2016-04-12 7:54 ` Dave Chinner 2016-04-13 17:16 ` Joe Lawrence 0 siblings, 1 reply; 4+ messages in thread From: Dave Chinner @ 2016-04-12 7:54 UTC (permalink / raw) To: Dave Chinner; +Cc: Joe Lawrence, xfs On Fri, Apr 01, 2016 at 01:44:13PM +1100, Dave Chinner wrote: > On Wed, Mar 30, 2016 at 11:55:18PM -0400, Joe Lawrence wrote: > > Hi Dave, > > > > Upon loading 4.6-rc1, I noticed a few linked list corruption messages in > > dmesg shortly after boot up. I bisected the kernel, landing on: > > > > [c19b3b05ae440de50fffe2ac2a9b27392a7448e9] xfs: mode di_mode to vfs inode > > > > If I revert c19b3b05ae44 from 4.6-rc1, the warnings stop. > > > > WARNING: CPU: 35 PID: 6715 at lib/list_debug.c:29 __list_add+0x65/0xc0 > > list_add corruption. next->prev should be prev (ffff882030928a00), but was ffff88103f00c300. (next=ffff88100fde5ce8). > ..... > > [<ffffffff812488f0>] ? bdev_test+0x20/0x20 > > [<ffffffff813551a5>] __list_add+0x65/0xc0 > > [<ffffffff81249bd8>] bd_acquire+0xc8/0xd0 > > [<ffffffff8124aa59>] blkdev_open+0x39/0x70 > > [<ffffffff8120bc27>] do_dentry_open+0x227/0x320 > > [<ffffffff8124aa20>] ? blkdev_get_by_dev+0x50/0x50 > > [<ffffffff8120d057>] vfs_open+0x57/0x60 > > [<ffffffff8121c9fa>] path_openat+0x1ba/0x1340 > > [<ffffffff8121eff1>] do_filp_open+0x91/0x100 > > [<ffffffff8122c806>] ? __alloc_fd+0x46/0x180 > > [<ffffffff8120d3b4>] do_sys_open+0x124/0x210 > > [<ffffffff8120d4be>] SyS_open+0x1e/0x20 > > [<ffffffff81003c12>] do_syscall_64+0x62/0x110 > > [<ffffffff8169ade1>] entry_SYSCALL64_slow_path+0x25/0x25 > .... > > According to the bd_acquire+0xc8 offset, we're in bd_acquire() > > attempting the list add: > .... > > 713 bdev = bdget(inode->i_rdev); > > 714 if (bdev) { > > 715 spin_lock(&bdev_lock); > > 716 if (!inode->i_bdev) { > > 717 /* > > 718 * We take an additional reference to bd_inode, > > 719 * and it's released in clear_inode() of inode. > > 720 * So, we can access it via ->i_mapping always > > 721 * without igrab(). > > 722 */ > > 723 bdgrab(bdev); > > 724 inode->i_bdev = bdev; > > 725 inode->i_mapping = bdev->bd_inode->i_mapping; > > 726 list_add(&inode->i_devices, &bdev->bd_inodes); > > So the bdev->bd_inodes list is corrupt, and this call trace is > just the messenger. .... > > I'm not really sure why the bisect landed on c19b3b05ae44 "xfs: mode > > di_mode to vfs inode", but as I mentioned, reverting it made the list > > warnings go away. > > Neither am I at this point as it's the bdev inode (not an xfs > inode) that has a corrupted list. I'll have to try to reproduce this. Patch below should fix the problem. Smoke tested only at this point. Cheers, Dave. -- Dave Chinner david@fromorbit.com xfs: we don't need no steekin ->evict_inode From: Dave Chinner <dchinner@redhat.com> Joe Lawrence reported a list_add corruption with 4.6-rc1 when testing some custom md administration code that made it's own block device nodes for the md array. The simple test loop of: for i in {0..100}; do mknod --mode=0600 $tmp/tmp_node b $MAJOR $MINOR mdadm --detail --export $tmp/tmp_node > /dev/null rm -f $tmp/tmp_node done Would produce this warning in bd_acquire() when mdadm opened the device node: list_add double add: new=ffff88043831c7b8, prev=ffff8804380287d8, next=ffff88043831c7b8. And then produce this from bd_forget from kdevtmpfs evicting a block dev inode: list_del corruption. prev->next should be ffff8800bb83eb10, but was ffff88043831c7b8 This is a regression caused by commit c19b3b05 ("xfs: mode di_mode to vfs inode"). The issue is that xfs_inactive() frees the unlinked inode, and the above commit meant that this freeing zeroed the mode in the struct inode. The problem is that after evict() has called ->evict_inode, it expects the i_mode to be intact so that it can call bd_forget() or cd_forget() to drop the reference to the block device inode attached to the XFS inode. In reality, the only thing we do in xfs_fs_evict_inode() that is not generic is call xfs_inactive(). We can move the xfs_inactive() call to xfs_fs_destroy_inode() without any problems at all, and this will leave the VFS inode intact until it is completely done with it. So, remove xfs_fs_evict_inode(), and do the work it used to do in ->destroy_inode instead. Reported-by: Joe Lawrence <joe.lawrence@stratus.com> Signed-off-by: Dave Chinner <dchinner@redhat.com> --- fs/xfs/xfs_super.c | 28 +++++++--------------------- 1 file changed, 7 insertions(+), 21 deletions(-) diff --git a/fs/xfs/xfs_super.c b/fs/xfs/xfs_super.c index b412bb1..d8424f5 100644 --- a/fs/xfs/xfs_super.c +++ b/fs/xfs/xfs_super.c @@ -928,7 +928,7 @@ xfs_fs_alloc_inode( /* * Now that the generic code is guaranteed not to be accessing - * the linux inode, we can reclaim the inode. + * the linux inode, we can inactivate and reclaim the inode. */ STATIC void xfs_fs_destroy_inode( @@ -938,9 +938,14 @@ xfs_fs_destroy_inode( trace_xfs_destroy_inode(ip); - XFS_STATS_INC(ip->i_mount, vn_reclaim); + ASSERT(!rwsem_is_locked(&ip->i_iolock.mr_lock)); + XFS_STATS_INC(ip->i_mount, vn_rele); + XFS_STATS_INC(ip->i_mount, vn_remove); + + xfs_inactive(ip); ASSERT(XFS_FORCED_SHUTDOWN(ip->i_mount) || ip->i_delayed_blks == 0); + XFS_STATS_INC(ip->i_mount, vn_reclaim); /* * We should never get here with one of the reclaim flags already set. @@ -987,24 +992,6 @@ xfs_fs_inode_init_once( "xfsino", ip->i_ino); } -STATIC void -xfs_fs_evict_inode( - struct inode *inode) -{ - xfs_inode_t *ip = XFS_I(inode); - - ASSERT(!rwsem_is_locked(&ip->i_iolock.mr_lock)); - - trace_xfs_evict_inode(ip); - - truncate_inode_pages_final(&inode->i_data); - clear_inode(inode); - XFS_STATS_INC(ip->i_mount, vn_rele); - XFS_STATS_INC(ip->i_mount, vn_remove); - - xfs_inactive(ip); -} - /* * We do an unlocked check for XFS_IDONTCACHE here because we are already * serialised against cache hits here via the inode->i_lock and igrab() in @@ -1673,7 +1660,6 @@ xfs_fs_free_cached_objects( static const struct super_operations xfs_super_operations = { .alloc_inode = xfs_fs_alloc_inode, .destroy_inode = xfs_fs_destroy_inode, - .evict_inode = xfs_fs_evict_inode, .drop_inode = xfs_fs_drop_inode, .put_super = xfs_fs_put_super, .sync_fs = xfs_fs_sync_fs, _______________________________________________ xfs mailing list xfs@oss.sgi.com http://oss.sgi.com/mailman/listinfo/xfs ^ permalink raw reply related [flat|nested] 4+ messages in thread
* Re: list_add corruption after "xfs: mode di_mode to vfs inode" 2016-04-12 7:54 ` Dave Chinner @ 2016-04-13 17:16 ` Joe Lawrence 2016-04-13 17:46 ` Troy McCorkell 0 siblings, 1 reply; 4+ messages in thread From: Joe Lawrence @ 2016-04-13 17:16 UTC (permalink / raw) To: xfs, dchinner On 04/12/2016 03:54 AM, Dave Chinner wrote: > On Fri, Apr 01, 2016 at 01:44:13PM +1100, Dave Chinner wrote: >> On Wed, Mar 30, 2016 at 11:55:18PM -0400, Joe Lawrence wrote: >>> Hi Dave, >>> >>> Upon loading 4.6-rc1, I noticed a few linked list corruption messages in >>> dmesg shortly after boot up. I bisected the kernel, landing on: >>> >>> [c19b3b05ae440de50fffe2ac2a9b27392a7448e9] xfs: mode di_mode to vfs inode >>> >>> If I revert c19b3b05ae44 from 4.6-rc1, the warnings stop. >>> >>> WARNING: CPU: 35 PID: 6715 at lib/list_debug.c:29 __list_add+0x65/0xc0 >>> list_add corruption. next->prev should be prev (ffff882030928a00), but was ffff88103f00c300. (next=ffff88100fde5ce8). >> ..... >>> [<ffffffff812488f0>] ? bdev_test+0x20/0x20 >>> [<ffffffff813551a5>] __list_add+0x65/0xc0 >>> [<ffffffff81249bd8>] bd_acquire+0xc8/0xd0 >>> [<ffffffff8124aa59>] blkdev_open+0x39/0x70 >>> [<ffffffff8120bc27>] do_dentry_open+0x227/0x320 >>> [<ffffffff8124aa20>] ? blkdev_get_by_dev+0x50/0x50 >>> [<ffffffff8120d057>] vfs_open+0x57/0x60 >>> [<ffffffff8121c9fa>] path_openat+0x1ba/0x1340 >>> [<ffffffff8121eff1>] do_filp_open+0x91/0x100 >>> [<ffffffff8122c806>] ? __alloc_fd+0x46/0x180 >>> [<ffffffff8120d3b4>] do_sys_open+0x124/0x210 >>> [<ffffffff8120d4be>] SyS_open+0x1e/0x20 >>> [<ffffffff81003c12>] do_syscall_64+0x62/0x110 >>> [<ffffffff8169ade1>] entry_SYSCALL64_slow_path+0x25/0x25 >> .... >>> According to the bd_acquire+0xc8 offset, we're in bd_acquire() >>> attempting the list add: >> .... >>> 713 bdev = bdget(inode->i_rdev); >>> 714 if (bdev) { >>> 715 spin_lock(&bdev_lock); >>> 716 if (!inode->i_bdev) { >>> 717 /* >>> 718 * We take an additional reference to bd_inode, >>> 719 * and it's released in clear_inode() of inode. >>> 720 * So, we can access it via ->i_mapping always >>> 721 * without igrab(). >>> 722 */ >>> 723 bdgrab(bdev); >>> 724 inode->i_bdev = bdev; >>> 725 inode->i_mapping = bdev->bd_inode->i_mapping; >>> 726 list_add(&inode->i_devices, &bdev->bd_inodes); >> >> So the bdev->bd_inodes list is corrupt, and this call trace is >> just the messenger. > .... >>> I'm not really sure why the bisect landed on c19b3b05ae44 "xfs: mode >>> di_mode to vfs inode", but as I mentioned, reverting it made the list >>> warnings go away. >> >> Neither am I at this point as it's the bdev inode (not an xfs >> inode) that has a corrupted list. I'll have to try to reproduce this. > > Patch below should fix the problem. Smoke tested only at this point. Thanks Dave, this looks good on both the QEMU and on the originating hardware instances. Let me know if there are any additional tests that I can run, otherwise consider this Tested-by. BTW, cuda.sgi.com is rejecting my mail to the list (even though I subscribed), so apologies for this not making it out to xfs@oss.sgi.com. Thanks again, -- Joe _______________________________________________ xfs mailing list xfs@oss.sgi.com http://oss.sgi.com/mailman/listinfo/xfs ^ permalink raw reply [flat|nested] 4+ messages in thread
* RE: list_add corruption after "xfs: mode di_mode to vfs inode" 2016-04-13 17:16 ` Joe Lawrence @ 2016-04-13 17:46 ` Troy McCorkell 0 siblings, 0 replies; 4+ messages in thread From: Troy McCorkell @ 2016-04-13 17:46 UTC (permalink / raw) To: Joe Lawrence, xfs@oss.sgi.com, dchinner@redhat.com Joe, I'll forward your email to our IT group to update the spam filters. Sorry for the inconvenience. -Troy ________________________________________ From: xfs-bounces@oss.sgi.com [xfs-bounces@oss.sgi.com] on behalf of Joe Lawrence [joe.lawrence@stratus.com] Sent: Wednesday, April 13, 2016 12:16 PM To: xfs@oss.sgi.com; dchinner@redhat.com Subject: Re: list_add corruption after "xfs: mode di_mode to vfs inode" On 04/12/2016 03:54 AM, Dave Chinner wrote: > On Fri, Apr 01, 2016 at 01:44:13PM +1100, Dave Chinner wrote: >> On Wed, Mar 30, 2016 at 11:55:18PM -0400, Joe Lawrence wrote: >>> Hi Dave, >>> >>> Upon loading 4.6-rc1, I noticed a few linked list corruption messages in >>> dmesg shortly after boot up. I bisected the kernel, landing on: >>> >>> [c19b3b05ae440de50fffe2ac2a9b27392a7448e9] xfs: mode di_mode to vfs inode >>> >>> If I revert c19b3b05ae44 from 4.6-rc1, the warnings stop. >>> >>> WARNING: CPU: 35 PID: 6715 at lib/list_debug.c:29 __list_add+0x65/0xc0 >>> list_add corruption. next->prev should be prev (ffff882030928a00), but was ffff88103f00c300. (next=ffff88100fde5ce8). >> ..... >>> [<ffffffff812488f0>] ? bdev_test+0x20/0x20 >>> [<ffffffff813551a5>] __list_add+0x65/0xc0 >>> [<ffffffff81249bd8>] bd_acquire+0xc8/0xd0 >>> [<ffffffff8124aa59>] blkdev_open+0x39/0x70 >>> [<ffffffff8120bc27>] do_dentry_open+0x227/0x320 >>> [<ffffffff8124aa20>] ? blkdev_get_by_dev+0x50/0x50 >>> [<ffffffff8120d057>] vfs_open+0x57/0x60 >>> [<ffffffff8121c9fa>] path_openat+0x1ba/0x1340 >>> [<ffffffff8121eff1>] do_filp_open+0x91/0x100 >>> [<ffffffff8122c806>] ? __alloc_fd+0x46/0x180 >>> [<ffffffff8120d3b4>] do_sys_open+0x124/0x210 >>> [<ffffffff8120d4be>] SyS_open+0x1e/0x20 >>> [<ffffffff81003c12>] do_syscall_64+0x62/0x110 >>> [<ffffffff8169ade1>] entry_SYSCALL64_slow_path+0x25/0x25 >> .... >>> According to the bd_acquire+0xc8 offset, we're in bd_acquire() >>> attempting the list add: >> .... >>> 713 bdev = bdget(inode->i_rdev); >>> 714 if (bdev) { >>> 715 spin_lock(&bdev_lock); >>> 716 if (!inode->i_bdev) { >>> 717 /* >>> 718 * We take an additional reference to bd_inode, >>> 719 * and it's released in clear_inode() of inode. >>> 720 * So, we can access it via ->i_mapping always >>> 721 * without igrab(). >>> 722 */ >>> 723 bdgrab(bdev); >>> 724 inode->i_bdev = bdev; >>> 725 inode->i_mapping = bdev->bd_inode->i_mapping; >>> 726 list_add(&inode->i_devices, &bdev->bd_inodes); >> >> So the bdev->bd_inodes list is corrupt, and this call trace is >> just the messenger. > .... >>> I'm not really sure why the bisect landed on c19b3b05ae44 "xfs: mode >>> di_mode to vfs inode", but as I mentioned, reverting it made the list >>> warnings go away. >> >> Neither am I at this point as it's the bdev inode (not an xfs >> inode) that has a corrupted list. I'll have to try to reproduce this. > > Patch below should fix the problem. Smoke tested only at this point. Thanks Dave, this looks good on both the QEMU and on the originating hardware instances. Let me know if there are any additional tests that I can run, otherwise consider this Tested-by. BTW, cuda.sgi.com is rejecting my mail to the list (even though I subscribed), so apologies for this not making it out to xfs@oss.sgi.com. Thanks again, -- Joe _______________________________________________ xfs mailing list xfs@oss.sgi.com http://oss.sgi.com/mailman/listinfo/xfs _______________________________________________ xfs mailing list xfs@oss.sgi.com http://oss.sgi.com/mailman/listinfo/xfs ^ permalink raw reply [flat|nested] 4+ messages in thread
end of thread, other threads:[~2016-04-13 17:47 UTC | newest]
Thread overview: 4+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
[not found] <56FC9FA6.1080700@stratus.com>
2016-04-01 2:44 ` list_add corruption after "xfs: mode di_mode to vfs inode" Dave Chinner
2016-04-12 7:54 ` Dave Chinner
2016-04-13 17:16 ` Joe Lawrence
2016-04-13 17:46 ` Troy McCorkell
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox