From: Chris Mason <clm@fb.com>
To: Miao Xie <miaox@cn.fujitsu.com>, <linux-btrfs@vger.kernel.org>
Subject: Re: [RFC PATCH 5/5] Btrfs: fix broken free space cache after the system crashed
Date: Sun, 18 May 2014 21:33:55 -0400 [thread overview]
Message-ID: <53795F83.20109@fb.com> (raw)
In-Reply-To: <1389787258-10865-5-git-send-email-miaox@cn.fujitsu.com>
On 01/15/2014 07:00 AM, Miao Xie wrote:
> When we mounted the filesystem after the crash, we got the following
> message:
> BTRFS error (device xxx): block group 4315938816 has wrong amount of free space
> BTRFS error (device xxx): failed to load free space cache for block group 4315938816
>
> It is because we didn't update the metadata of the allocated space until
> the file data was written into the disk. During this time, there was no
> information about the allocated spaces in either the extent tree nor the
> free space cache. when we wrote out the free space cache at this time, those
> spaces were lost.
>
> In ordered to fix this problem, I use a state tree for every block group
> to record those allocated spaces. We record the information when they are
> allocated, and clean up the information after the metadata update. Besides
> that, we also introduce a read-write semaphore to avoid the race between
> the allocation and the free space cache write out.
>
> Only data block groups had this problem, so the above change is just
> for data space allocation.
I had this one in the integration branch, but lockdep reported troubles. It
looks like lockdep is correct. find_free_extent is nesting the cache rwsem
inside the groups rwsem, but btrfs_write_out_cache is holding the new cache
rwsem when it calls find_free_extent.
-chris
[ 2857.610731] ======================================================
[ 2857.623158] [ INFO: possible circular locking dependency detected ]
[ 2857.635771] 3.15.0-rc5-mason+ #43 Not tainted
[ 2857.644553] -------------------------------------------------------
[ 2857.657139] btrfs-transacti/19476 is trying to acquire lock:
[ 2857.668518] (&found->groups_sem){++++..}, at: [<ffffffffa059dbc1>] find_free_extent+0x931/0xe20 [btrfs]
[ 2857.687771]
[ 2857.687771] but task is already holding lock:
[ 2857.699566] (&cache->data_rwsem){++++..}, at: [<ffffffffa05f78bf>] btrfs_write_out_cache+0x9f/0x170 [btrfs]
[ 2857.719480]
[ 2857.719480] which lock already depends on the new lock.
[ 2857.719480]
[ 2857.736021]
[ 2857.736021] the existing dependency chain (in reverse order) is:
[ 2857.751120]
-> #1 (&cache->data_rwsem){++++..}:
[ 2857.760823] [<ffffffff810a14fe>] lock_acquire+0x8e/0x110
[ 2857.772772] [<ffffffff81649cf7>] down_read+0x47/0x60
[ 2857.784028] [<ffffffffa059db2c>] find_free_extent+0x89c/0xe20 [btrfs]
[ 2857.798253] [<ffffffffa059e11b>] btrfs_reserve_extent+0x6b/0x140 [btrfs]
[ 2857.813041] [<ffffffffa05b73ec>] cow_file_range+0x13c/0x460 [btrfs]
[ 2857.826892] [<ffffffffa05bc097>] run_delalloc_range+0x347/0x380 [btrfs]
[ 2857.841510] [<ffffffffa05d3f3d>] __extent_writepage+0x70d/0x870 [btrfs]
[ 2857.856129] [<ffffffffa05d456a>] extent_write_cache_pages.clone.6+0x30a/0x410 [btrfs]
[ 2857.873185] [<ffffffffa05d46c2>] extent_writepages+0x52/0x70 [btrfs]
[ 2857.887224] [<ffffffffa05b3d57>] btrfs_writepages+0x27/0x30 [btrfs]
[ 2857.901078] [<ffffffff81142543>] do_writepages+0x23/0x40
[ 2857.913034] [<ffffffff81135b99>] __filemap_fdatawrite_range+0x59/0x60
[ 2857.927240] [<ffffffff81135dec>] filemap_flush+0x1c/0x20
[ 2857.939215] [<ffffffffa05b2502>] btrfs_run_delalloc_work+0x72/0xa0 [btrfs]
[ 2857.954367] [<ffffffffa05e05fe>] normal_work_helper+0x6e/0x2d0 [btrfs]
[ 2857.968749] [<ffffffff8106b9e2>] process_one_work+0x1d2/0x550
[ 2857.981561] [<ffffffff8106cd8f>] worker_thread+0x11f/0x3a0
[ 2857.993856] [<ffffffff8107317e>] kthread+0xde/0x100
[ 2858.004936] [<ffffffff8165436c>] ret_from_fork+0x7c/0xb0
[ 2858.016887]
-> #0 (&found->groups_sem){++++..}:
[ 2858.026590] [<ffffffff810a12de>] __lock_acquire+0x161e/0x17b0
[ 2858.039407] [<ffffffff810a14fe>] lock_acquire+0x8e/0x110
[ 2858.051370] [<ffffffff81649cf7>] down_read+0x47/0x60
[ 2858.062629] [<ffffffffa059dbc1>] find_free_extent+0x931/0xe20 [btrfs]
[ 2858.076841] [<ffffffffa059e11b>] btrfs_reserve_extent+0x6b/0x140 [btrfs]
[ 2858.091629] [<ffffffffa059e307>] btrfs_alloc_free_block+0x117/0x420 [btrfs]
[ 2858.106940] [<ffffffffa0589a5b>] __btrfs_cow_block+0x11b/0x530 [btrfs]
[ 2858.121331] [<ffffffffa058a4a0>] btrfs_cow_block+0x130/0x1e0 [btrfs]
[ 2858.135375] [<ffffffffa058c999>] btrfs_search_slot+0x219/0x9c0 [btrfs]
[ 2858.149760] [<ffffffffa05f7595>] __btrfs_write_out_cache+0x755/0x970 [btrfs]
[ 2858.165245] [<ffffffffa05f7958>] btrfs_write_out_cache+0x138/0x170 [btrfs]
[ 2858.180411] [<ffffffffa059ccb0>] btrfs_write_dirty_block_groups+0x480/0x600 [btrfs]
[ 2858.197107] [<ffffffffa05ae7af>] commit_cowonly_roots+0x19f/0x250 [btrfs]
[ 2858.212084] [<ffffffffa05afbc0>] btrfs_commit_transaction+0x450/0xa60 [btrfs]
[ 2858.227738] [<ffffffffa05aa8a6>] transaction_kthread+0x216/0x290 [btrfs]
[ 2858.242533] [<ffffffff8107317e>] kthread+0xde/0x100
[ 2858.253617] [<ffffffff8165436c>] ret_from_fork+0x7c/0xb0
[ 2858.265569]
[ 2858.265569] other info that might help us debug this:
[ 2858.265569]
[ 2858.281780] Possible unsafe locking scenario:
[ 2858.281780]
[ 2858.293750] CPU0 CPU1
[ 2858.302869] ---- ----
[ 2858.312000] lock(&cache->data_rwsem);
[ 2858.319828] lock(&found->groups_sem);
[ 2858.332661] lock(&cache->data_rwsem);
[ 2858.345508] lock(&found->groups_sem);
[ 2858.353300]
[ 2858.353300] *** DEADLOCK ***
[ 2858.353300]
[ 2858.365337] 4 locks held by btrfs-transacti/19476:
[ 2858.374993] #0: (&fs_info->transaction_kthread_mutex){+.+...}, at: [<ffffffffa05aa740>] transaction_kthread+0xb0/0x290 [btrfs]
[ 2858.398451] #1: (&fs_info->reloc_mutex){+.+...}, at: [<ffffffffa05afaf0>] btrfs_commit_transaction+0x380/0xa60 [btrfs]
[ 2858.420535] #2: (&fs_info->tree_log_mutex){+.+...}, at: [<ffffffffa05afb66>] btrfs_commit_transaction+0x3f6/0xa60 [btrfs]
[ 2858.443135] #3: (&cache->data_rwsem){++++..}, at: [<ffffffffa05f78bf>] btrfs_write_out_cache+0x9f/0x170 [btrfs]
[ 2858.463953]
[ 2858.463953] stack backtrace:
[ 2858.472807] CPU: 25 PID: 19476 Comm: btrfs-transacti Not tainted 3.15.0-rc5-mason+ #43
[ 2858.488772] Hardware name: ZTSYSTEMS Echo Ridge T4 /A9DRPF-10D, BIOS 1.07 05/10/2012
[ 2858.504564] ffffffff820f1b10 ffff8807ff5f94e8 ffffffff8164585c 0000000000000001
[ 2858.519677] ffffffff820e6170 ffff8807ff5f9538 ffffffff8109e322 0000000000000004
[ 2858.534761] ffff8807ff5f9598 ffff8807ff5f9538 ffff8808444fd118 ffff8808444fc890
[ 2858.549850] Call Trace:
[ 2858.554830] [<ffffffff8164585c>] dump_stack+0x51/0x6d
[ 2858.565168] [<ffffffff8109e322>] print_circular_bug+0x212/0x310
[ 2858.577247] [<ffffffff810a12de>] __lock_acquire+0x161e/0x17b0
[ 2858.588979] [<ffffffff810a14fe>] lock_acquire+0x8e/0x110
[ 2858.599845] [<ffffffffa059dbc1>] ? find_free_extent+0x931/0xe20 [btrfs]
[ 2858.613312] [<ffffffff81649cf7>] down_read+0x47/0x60
[ 2858.623501] [<ffffffffa059dbc1>] ? find_free_extent+0x931/0xe20 [btrfs]
[ 2858.636978] [<ffffffffa059dbc1>] find_free_extent+0x931/0xe20 [btrfs]
[ 2858.650092] [<ffffffff8164b60b>] ? _raw_spin_unlock+0x2b/0x40
[ 2858.661842] [<ffffffffa059e11b>] btrfs_reserve_extent+0x6b/0x140 [btrfs]
[ 2858.675483] [<ffffffffa059e307>] btrfs_alloc_free_block+0x117/0x420 [btrfs]
[ 2858.689644] [<ffffffff810a01d0>] ? __lock_acquire+0x510/0x17b0
[ 2858.701564] [<ffffffffa05cc600>] ? find_extent_buffer+0x10/0xf0 [btrfs]
[ 2858.715034] [<ffffffffa0589a5b>] __btrfs_cow_block+0x11b/0x530 [btrfs]
[ 2858.728333] [<ffffffffa058a4a0>] btrfs_cow_block+0x130/0x1e0 [btrfs]
[ 2858.741284] [<ffffffffa058c999>] btrfs_search_slot+0x219/0x9c0 [btrfs]
[ 2858.754597] [<ffffffffa05f7595>] __btrfs_write_out_cache+0x755/0x970 [btrfs]
[ 2858.768946] [<ffffffffa05f78c7>] ? btrfs_write_out_cache+0xa7/0x170 [btrfs]
[ 2858.783108] [<ffffffffa05f7900>] ? btrfs_write_out_cache+0xe0/0x170 [btrfs]
[ 2858.797289] [<ffffffffa05f7958>] btrfs_write_out_cache+0x138/0x170 [btrfs]
[ 2858.811278] [<ffffffff8164b60b>] ? _raw_spin_unlock+0x2b/0x40
[ 2858.823017] [<ffffffffa059ccb0>] btrfs_write_dirty_block_groups+0x480/0x600 [btrfs]
[ 2858.838646] [<ffffffffa05ae7af>] commit_cowonly_roots+0x19f/0x250 [btrfs]
[ 2858.852461] [<ffffffffa05afbc0>] btrfs_commit_transaction+0x450/0xa60 [btrfs]
[ 2858.867043] [<ffffffffa05aa826>] ? transaction_kthread+0x196/0x290 [btrfs]
[ 2858.881032] [<ffffffffa05aa8a6>] transaction_kthread+0x216/0x290 [btrfs]
[ 2858.894679] [<ffffffffa05aa690>] ? close_ctree+0x2d0/0x2d0 [btrfs]
[ 2858.907276] [<ffffffff8107317e>] kthread+0xde/0x100
[ 2858.917280] [<ffffffff810730a0>] ? __init_kthread_worker+0x70/0x70
[ 2858.929873] [<ffffffff8165436c>] ret_from_fork+0x7c/0xb0
[ 2858.940745] [<ffffffff810730a0>] ? __init_kthread_worker+0x70/0x70
[ 2893.109104] EXT4-fs (loop0): mounted filesystem with ordered data mode. Opts: (null)
next prev parent reply other threads:[~2014-05-19 1:32 UTC|newest]
Thread overview: 18+ messages / expand[flat|nested] mbox.gz Atom feed top
2014-01-15 12:00 [PATCH 1/5] Btrfs: change the members' order of btrfs_space_info structure to reduce the cache miss Miao Xie
2014-01-15 12:00 ` [PATCH 2/5] Btrfs: cleanup the redundant code for the block group allocation and init Miao Xie
2014-01-15 12:00 ` [PATCH 3/5] Btrfs: cleanup the code of used_block_group in find_free_extent() Miao Xie
2014-01-15 12:00 ` [PATCH 4/5] Btrfs: fix wrong block group in trace during the free space allocation Miao Xie
2014-01-15 12:00 ` [RFC PATCH 5/5] Btrfs: fix broken free space cache after the system crashed Miao Xie
2014-01-16 5:54 ` Liu Bo
2014-03-01 18:05 ` Alex Lyakas
2014-03-04 6:04 ` Miao Xie
2014-03-08 16:48 ` Alex Lyakas
2014-03-04 15:19 ` Josef Bacik
2014-03-05 7:02 ` Miao Xie
2014-03-05 15:02 ` Josef Bacik
2014-03-26 6:36 ` miaox
2014-04-24 5:31 ` [PATCH V2 1/2] Btrfs: output warning instead of error when loading free space cache failed Miao Xie
2014-04-24 5:31 ` [PATCH V2 2/2] Btrfs: fix broken free space cache after the system crashed Miao Xie
2014-05-19 1:33 ` Chris Mason [this message]
2014-06-10 8:15 ` [RFC PATCH 5/5] " Alin Dobre
2014-01-15 15:56 ` [PATCH 1/5] Btrfs: change the members' order of btrfs_space_info structure to reduce the cache miss David Sterba
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=53795F83.20109@fb.com \
--to=clm@fb.com \
--cc=linux-btrfs@vger.kernel.org \
--cc=miaox@cn.fujitsu.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).