From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mx0a-00082601.pphosted.com ([67.231.145.42]:37654 "EHLO mx0a-00082601.pphosted.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S933292AbcJZVxS (ORCPT ); Wed, 26 Oct 2016 17:53:18 -0400 Subject: Re: bio linked list corruption. To: Linus Torvalds , Dave Jones , Andy Lutomirski , "Andy Lutomirski" , Jens Axboe , Al Viro , Josef Bacik , David Sterba , linux-btrfs , Linux Kernel , Dave Chinner References: <20161021200245.kahjzgqzdfyoe3uz@codemonkey.org.uk> <20161022152033.gkmm3l75kqjzsije@codemonkey.org.uk> <20161024044051.onmh4h6sc2bjxzzc@codemonkey.org.uk> <77d9983d-a00a-1dc1-a9a1-631de1d0c146@fb.com> <20161026002752.qvrm6yxqb54fiqnd@codemonkey.org.uk> <20161026163018.wx57yy554576s6e2@codemonkey.org.uk> <20161026184201.6ofblkd3j5uxystq@codemonkey.org.uk> <488f9edc-6a1c-2c68-0d33-d3aa32ece9a4@fb.com> From: Chris Mason Message-ID: Date: Wed, 26 Oct 2016 17:52:52 -0400 MIME-Version: 1.0 In-Reply-To: <488f9edc-6a1c-2c68-0d33-d3aa32ece9a4@fb.com> Content-Type: text/plain; charset="utf-8" Sender: linux-btrfs-owner@vger.kernel.org List-ID: On 10/26/2016 04:00 PM, Chris Mason wrote: > > > On 10/26/2016 03:06 PM, Linus Torvalds wrote: >> On Wed, Oct 26, 2016 at 11:42 AM, Dave Jones wrote: >>> >>> The stacks show nearly all of them are stuck in sync_inodes_sb >> >> That's just wb_wait_for_completion(), and it means that some IO isn't >> completing. >> >> There's also a lot of processes waiting for inode_lock(), and a few >> waiting for mnt_want_write() >> >> Ignoring those, we have >> >>> [] btrfs_wait_ordered_roots+0x3f/0x200 [btrfs] >>> [] btrfs_sync_fs+0x31/0xc0 [btrfs] >>> [] sync_filesystem+0x6e/0xa0 >>> [] SyS_syncfs+0x3c/0x70 >>> [] do_syscall_64+0x5c/0x170 >>> [] entry_SYSCALL64_slow_path+0x25/0x25 >>> [] 0xffffffffffffffff >> >> Don't know this one. There's a couple of them. Could there be some >> ABBA deadlock on the ordered roots waiting? > > It's always possible, but we haven't changed anything here. > > I've tried a long list of things to reproduce this on my test boxes, > including days of trinity runs and a kernel module to exercise vmalloc, > and thread creation. > > Today I turned off every CONFIG_DEBUG_* except for list debugging, and > ran dbench 2048: > This one is special because CONFIG_VMAP_STACK is not set. Btrfs triggers in < 10 minutes. I've done 30 minutes each with XFS and Ext4 without luck. This is all in a virtual machine that I can copy on to a bunch of hosts. So I'll get some parallel tests going tonight to narrow it down. ------------[ cut here ]------------ WARNING: CPU: 6 PID: 4481 at lib/list_debug.c:33 __list_add+0xbe/0xd0 list_add corruption. prev->next should be next (ffffe8ffffd80b08), but was ffff88012b65fb88. (prev=ffff880128c8d500). Modules linked in: crc32c_intel aesni_intel aes_x86_64 glue_helper lrw gf128mul ablk_helper i2c_piix4 cryptd i2c_core virtio_net serio_raw floppy button pcspkr sch_fq_codel autofs4 virtio_blk CPU: 6 PID: 4481 Comm: dbench Not tainted 4.9.0-rc2-15419-g811d54d #319 Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.9.0-1.fc24 04/01/2014 ffff880104eff868 ffffffff814fde0f ffffffff8151c46e ffff880104eff8c8 ffff880104eff8c8 0000000000000000 ffff880104eff8b8 ffffffff810648cf ffff880128cab2c0 000000213fc57c68 ffff8801384e8928 ffff880128cab180 Call Trace: [] dump_stack+0x53/0x74 [] ? __list_add+0xbe/0xd0 [] __warn+0xff/0x120 [] warn_slowpath_fmt+0x49/0x50 [] __list_add+0xbe/0xd0 [] blk_sq_make_request+0x388/0x580 [] generic_make_request+0x104/0x200 [] submit_bio+0x65/0x130 [] ? __percpu_counter_add+0x96/0xd0 [] btrfs_map_bio+0x23c/0x310 [] btrfs_submit_bio_hook+0xd3/0x190 [] submit_one_bio+0x6d/0xa0 [] flush_epd_write_bio+0x4e/0x70 [] extent_writepages+0x5d/0x70 [] ? btrfs_releasepage+0x50/0x50 [] ? wbc_attach_and_unlock_inode+0x6e/0x170 [] btrfs_writepages+0x27/0x30 [] do_writepages+0x20/0x30 [] __filemap_fdatawrite_range+0xb5/0x100 [] filemap_fdatawrite_range+0x13/0x20 [] btrfs_fdatawrite_range+0x2b/0x70 [] btrfs_sync_file+0x88/0x490 [] ? group_send_sig_info+0x42/0x80 [] ? kill_pid_info+0x5d/0x90 [] ? SYSC_kill+0xba/0x1d0 [] ? __sb_end_write+0x58/0x80 [] vfs_fsync_range+0x4c/0xb0 [] ? syscall_trace_enter+0x201/0x2e0 [] vfs_fsync+0x1c/0x20 [] do_fsync+0x3d/0x70 [] ? syscall_slow_exit_work+0xfb/0x100 [] SyS_fsync+0x10/0x20 [] do_syscall_64+0x55/0xd0 [] ? prepare_exit_to_usermode+0x37/0x40 [] entry_SYSCALL64_slow_path+0x25/0x25 ---[ end trace efe6b17c6dba2a6e ]---