Re: [PATCH] sync: wait_sb_inodes() calls iput() with spinlock held (was Re: [PATCH 0/7] super block scalabilit patches V3)

[Josef Bacik <jbacik@fb.com>]

On 06/21/2015 07:26 PM, Dave Chinner wrote:
> On Tue, Jun 16, 2015 at 07:34:29AM +1000, Dave Chinner wrote:
>> On Thu, Jun 11, 2015 at 03:41:05PM -0400, Josef Bacik wrote:
>>> Here are the cleaned up versions of Dave Chinners super block scalability
>>> patches.  I've been testing them locally for a while and they are pretty solid.
>>> They fix a few big issues, such as the global inode list and soft lockups on
>>> boxes on unmount that have lots of inodes in cache.  Al if you would consider
>>> pulling these in that would be great, you can pull from here
>>>
>>> git://git.kernel.org/pub/scm/linux/kernel/git/josef/btrfs-next.git superblock-scaling
>>
>> Passes all my smoke tests.
>>
>> Tested-by: Dave Chinner <dchinner@redhat.com>
>
> FWIW, I just updated my trees to whatever is in the above branch and
> v4.1-rc8, and now I'm seeing problems with wb.list_lock recursion
> and "sleeping in atomic" scehduling issues. generic/269 produced
> this:
>
>   BUG: spinlock cpu recursion on CPU#1, fsstress/3852
>    lock: 0xffff88042a650c28, .magic: dead4ead, .owner: fsstress/3804, .owner_cpu: 1
>   CPU: 1 PID: 3852 Comm: fsstress Tainted: G        W       4.1.0-rc8-dgc+ #263
>   Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS Bochs 01/01/2011
>    ffff88042a650c28 ffff88039898b8e8 ffffffff81e18ffd ffff88042f250fb0
>    ffff880428f6b8e0 ffff88039898b908 ffffffff81e12f09 ffff88042a650c28
>    ffffffff8221337b ffff88039898b928 ffffffff81e12f34 ffff88042a650c28
>   Call Trace:
>    [<ffffffff81e18ffd>] dump_stack+0x4c/0x6e
>    [<ffffffff81e12f09>] spin_dump+0x90/0x95
>    [<ffffffff81e12f34>] spin_bug+0x26/0x2b
>    [<ffffffff810e762d>] do_raw_spin_lock+0x10d/0x150
>    [<ffffffff81e24975>] _raw_spin_lock+0x15/0x20
>    [<ffffffff811f8ba0>] __mark_inode_dirty+0x2b0/0x450
>    [<ffffffff812003b8>] __set_page_dirty+0x78/0xd0
>    [<ffffffff81200531>] mark_buffer_dirty+0x61/0xf0
>    [<ffffffff81200d91>] __block_commit_write.isra.24+0x81/0xb0
>    [<ffffffff81202406>] block_write_end+0x36/0x70
>    [<ffffffff814fa110>] ? __xfs_get_blocks+0x8a0/0x8a0
>    [<ffffffff81202474>] generic_write_end+0x34/0xb0
>    [<ffffffff8118af3d>] ? wait_for_stable_page+0x1d/0x50
>    [<ffffffff814fa317>] xfs_vm_write_end+0x67/0xc0
>    [<ffffffff811813af>] pagecache_write_end+0x1f/0x30
>    [<ffffffff815060dd>] xfs_iozero+0x10d/0x190
>    [<ffffffff8150666b>] xfs_zero_last_block+0xdb/0x110
>    [<ffffffff815067ba>] xfs_zero_eof+0x11a/0x290
>    [<ffffffff811d69e0>] ? complete_walk+0x60/0x100
>    [<ffffffff811da25f>] ? path_lookupat+0x5f/0x660
>    [<ffffffff81506a6e>] xfs_file_aio_write_checks+0x13e/0x160
>    [<ffffffff81506f15>] xfs_file_buffered_aio_write+0x75/0x250
>    [<ffffffff811ddb0f>] ? user_path_at_empty+0x5f/0xa0
>    [<ffffffff810c601d>] ? __might_sleep+0x4d/0x90
>    [<ffffffff815071f5>] xfs_file_write_iter+0x105/0x120
>    [<ffffffff811cc5ce>] __vfs_write+0xae/0xf0
>    [<ffffffff811ccc01>] vfs_write+0xa1/0x190
>    [<ffffffff811cd999>] SyS_write+0x49/0xb0
>    [<ffffffff811cc781>] ? SyS_lseek+0x91/0xb0
>    [<ffffffff81e24fee>] system_call_fastpath+0x12/0x71
>
> And there are a few tests (including generic/269) producing
> in_atomic/"scheduling while atomic" bugs in the evict() path such as:
>
>   in_atomic(): 1, irqs_disabled(): 0, pid: 3852, name: fsstress
>   CPU: 12 PID: 3852 Comm: fsstress Not tainted 4.1.0-rc8-dgc+ #263
>   Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS Bochs 01/01/2011
>    000000000000015d ffff88039898b6d8 ffffffff81e18ffd 0000000000000000
>    ffff880398865550 ffff88039898b6f8 ffffffff810c5f89 ffff8803f15c45c0
>    ffffffff8227a3bf ffff88039898b728 ffffffff810c601d ffff88039898b758
>   Call Trace:
>    [<ffffffff81e18ffd>] dump_stack+0x4c/0x6e
>    [<ffffffff810c5f89>] ___might_sleep+0xf9/0x140
>    [<ffffffff810c601d>] __might_sleep+0x4d/0x90
>    [<ffffffff81201e8b>] block_invalidatepage+0xab/0x140
>    [<ffffffff814f7579>] xfs_vm_invalidatepage+0x39/0xb0
>    [<ffffffff8118fa77>] truncate_inode_page+0x67/0xa0
>    [<ffffffff8118fc92>] truncate_inode_pages_range+0x1a2/0x6f0
>    [<ffffffff811828d1>] ? find_get_pages_tag+0xf1/0x1b0
>    [<ffffffff8104a663>] ? __switch_to+0x1e3/0x5a0
>    [<ffffffff8118dd05>] ? pagevec_lookup_tag+0x25/0x40
>    [<ffffffff811f620d>] ? __inode_wait_for_writeback+0x6d/0xc0
>    [<ffffffff8119024c>] truncate_inode_pages_final+0x4c/0x60
>    [<ffffffff8151c47f>] xfs_fs_evict_inode+0x4f/0x100
>    [<ffffffff811e8330>] evict+0xc0/0x1a0
>    [<ffffffff811e8d7b>] iput+0x1bb/0x220
>    [<ffffffff811f68b3>] sync_inodes_sb+0x353/0x3d0
>    [<ffffffff8151def8>] xfs_flush_inodes+0x28/0x40
>    [<ffffffff81514648>] xfs_create+0x638/0x770
>    [<ffffffff814e9049>] ? xfs_dir2_sf_lookup+0x199/0x330
>    [<ffffffff81511091>] xfs_generic_create+0xd1/0x300
>    [<ffffffff817a059c>] ? security_inode_permission+0x1c/0x30
>    [<ffffffff815112f6>] xfs_vn_create+0x16/0x20
>    [<ffffffff811d8665>] vfs_create+0xd5/0x140
>    [<ffffffff811dbea3>] do_last+0xff3/0x1200
>    [<ffffffff811d9f36>] ? path_init+0x186/0x450
>    [<ffffffff811dc130>] path_openat+0x80/0x610
>    [<ffffffff81512a24>] ? xfs_iunlock+0xc4/0x210
>    [<ffffffff811ddbfa>] do_filp_open+0x3a/0x90
>    [<ffffffff811dc8bf>] ? getname_flags+0x4f/0x200
>    [<ffffffff81e249ce>] ? _raw_spin_unlock+0xe/0x30
>    [<ffffffff811eab17>] ? __alloc_fd+0xa7/0x130
>    [<ffffffff811cbcf8>] do_sys_open+0x128/0x220
>    [<ffffffff811cbe4e>] SyS_creat+0x1e/0x20
>    [<ffffffff81e24fee>] system_call_fastpath+0x12/0x71
>
> It looks to me like iput() is being called with the wb.list_lock
> held in wait_sb_inodes(), and everything is going downhill from
> there.  Patch below fixes the problem for me.
>

I folded this into "bdi: add a new writeback list for sync" since it was 
there before and to be more bisect friendly.  Let me know if this isn't 
ok with you and I'll undo it.  Thanks,

Josef