* [PATCH] btrfs: fix deadlock when writing out space cache
@ 2017-11-15 21:20 Josef Bacik
2017-11-15 21:29 ` Omar Sandoval
` (3 more replies)
0 siblings, 4 replies; 8+ messages in thread
From: Josef Bacik @ 2017-11-15 21:20 UTC (permalink / raw)
To: linux-btrfs, kernel-team; +Cc: Josef Bacik
From: Josef Bacik <jbacik@fb.com>
If we fail to prepare our pages for whatever reason (out of memory in
our case) we need to make sure to drop the block_group->data_rwsem,
otherwise hilarity ensues.
Signed-off-by: Josef Bacik <jbacik@fb.com>
---
fs/btrfs/free-space-cache.c | 6 +++++-
1 file changed, 5 insertions(+), 1 deletion(-)
diff --git a/fs/btrfs/free-space-cache.c b/fs/btrfs/free-space-cache.c
index cdc9f4015ec3..a6c643275210 100644
--- a/fs/btrfs/free-space-cache.c
+++ b/fs/btrfs/free-space-cache.c
@@ -1263,8 +1263,12 @@ static int __btrfs_write_out_cache(struct btrfs_root *root, struct inode *inode,
/* Lock all pages first so we can lock the extent safely. */
ret = io_ctl_prepare_pages(io_ctl, inode, 0);
- if (ret)
+ if (ret) {
+ if (block_group &&
+ (block_group->flags & BTRFS_BLOCK_GROUP_DATA))
+ up_write(&block_group->data_rwsem);
goto out;
+ }
lock_extent_bits(&BTRFS_I(inode)->io_tree, 0, i_size_read(inode) - 1,
&cached_state);
--
2.7.5
^ permalink raw reply related [flat|nested] 8+ messages in thread
* Re: [PATCH] btrfs: fix deadlock when writing out space cache
2017-11-15 21:20 [PATCH] btrfs: fix deadlock when writing out space cache Josef Bacik
@ 2017-11-15 21:29 ` Omar Sandoval
2017-11-15 23:46 ` Liu Bo
` (2 subsequent siblings)
3 siblings, 0 replies; 8+ messages in thread
From: Omar Sandoval @ 2017-11-15 21:29 UTC (permalink / raw)
To: Josef Bacik; +Cc: linux-btrfs, kernel-team, Josef Bacik
On Wed, Nov 15, 2017 at 04:20:52PM -0500, Josef Bacik wrote:
> From: Josef Bacik <jbacik@fb.com>
>
> If we fail to prepare our pages for whatever reason (out of memory in
> our case) we need to make sure to drop the block_group->data_rwsem,
> otherwise hilarity ensues.
>
> Signed-off-by: Josef Bacik <jbacik@fb.com>
Reviewed-by: Omar Sandoval <osandov@fb.com>
> ---
> fs/btrfs/free-space-cache.c | 6 +++++-
> 1 file changed, 5 insertions(+), 1 deletion(-)
>
> diff --git a/fs/btrfs/free-space-cache.c b/fs/btrfs/free-space-cache.c
> index cdc9f4015ec3..a6c643275210 100644
> --- a/fs/btrfs/free-space-cache.c
> +++ b/fs/btrfs/free-space-cache.c
> @@ -1263,8 +1263,12 @@ static int __btrfs_write_out_cache(struct btrfs_root *root, struct inode *inode,
>
> /* Lock all pages first so we can lock the extent safely. */
> ret = io_ctl_prepare_pages(io_ctl, inode, 0);
> - if (ret)
> + if (ret) {
> + if (block_group &&
> + (block_group->flags & BTRFS_BLOCK_GROUP_DATA))
> + up_write(&block_group->data_rwsem);
> goto out;
> + }
>
> lock_extent_bits(&BTRFS_I(inode)->io_tree, 0, i_size_read(inode) - 1,
> &cached_state);
> --
> 2.7.5
>
> --
> To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: [PATCH] btrfs: fix deadlock when writing out space cache
2017-11-15 21:20 [PATCH] btrfs: fix deadlock when writing out space cache Josef Bacik
2017-11-15 21:29 ` Omar Sandoval
@ 2017-11-15 23:46 ` Liu Bo
2017-11-16 1:52 ` Chris Mason
2017-11-16 8:09 ` Nikolay Borisov
2017-11-20 17:22 ` David Sterba
3 siblings, 1 reply; 8+ messages in thread
From: Liu Bo @ 2017-11-15 23:46 UTC (permalink / raw)
To: Josef Bacik; +Cc: linux-btrfs, kernel-team, Josef Bacik
On Wed, Nov 15, 2017 at 04:20:52PM -0500, Josef Bacik wrote:
> From: Josef Bacik <jbacik@fb.com>
>
> If we fail to prepare our pages for whatever reason (out of memory in
> our case) we need to make sure to drop the block_group->data_rwsem,
> otherwise hilarity ensues.
>
Reviewed-by: Liu Bo <bo.li.liu@oracle.com>
Thanks,
-liubo
> Signed-off-by: Josef Bacik <jbacik@fb.com>
> ---
> fs/btrfs/free-space-cache.c | 6 +++++-
> 1 file changed, 5 insertions(+), 1 deletion(-)
>
> diff --git a/fs/btrfs/free-space-cache.c b/fs/btrfs/free-space-cache.c
> index cdc9f4015ec3..a6c643275210 100644
> --- a/fs/btrfs/free-space-cache.c
> +++ b/fs/btrfs/free-space-cache.c
> @@ -1263,8 +1263,12 @@ static int __btrfs_write_out_cache(struct btrfs_root *root, struct inode *inode,
>
> /* Lock all pages first so we can lock the extent safely. */
> ret = io_ctl_prepare_pages(io_ctl, inode, 0);
> - if (ret)
> + if (ret) {
> + if (block_group &&
> + (block_group->flags & BTRFS_BLOCK_GROUP_DATA))
> + up_write(&block_group->data_rwsem);
> goto out;
> + }
>
> lock_extent_bits(&BTRFS_I(inode)->io_tree, 0, i_size_read(inode) - 1,
> &cached_state);
> --
> 2.7.5
>
> --
> To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: [PATCH] btrfs: fix deadlock when writing out space cache
2017-11-15 23:46 ` Liu Bo
@ 2017-11-16 1:52 ` Chris Mason
0 siblings, 0 replies; 8+ messages in thread
From: Chris Mason @ 2017-11-16 1:52 UTC (permalink / raw)
To: Josef Bacik; +Cc: linux-btrfs, kernel-team, Josef Bacik
On 11/15/2017 06:46 PM, Liu Bo wrote:
> On Wed, Nov 15, 2017 at 04:20:52PM -0500, Josef Bacik wrote:
>> From: Josef Bacik <jbacik@fb.com>
>>
>> If we fail to prepare our pages for whatever reason (out of memory in
>> our case) we need to make sure to drop the block_group->data_rwsem,
>> otherwise hilarity ensues.
>>
Thanks Josef, I searched all the logs and it looks like we've really
only hit this twice this month. It's surprising we haven't seen this
more given how often we OOM.
-chris
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: [PATCH] btrfs: fix deadlock when writing out space cache
2017-11-15 21:20 [PATCH] btrfs: fix deadlock when writing out space cache Josef Bacik
2017-11-15 21:29 ` Omar Sandoval
2017-11-15 23:46 ` Liu Bo
@ 2017-11-16 8:09 ` Nikolay Borisov
2017-11-16 13:50 ` Chris Mason
2017-11-20 17:22 ` David Sterba
3 siblings, 1 reply; 8+ messages in thread
From: Nikolay Borisov @ 2017-11-16 8:09 UTC (permalink / raw)
To: Josef Bacik, linux-btrfs, kernel-team; +Cc: Josef Bacik
On 15.11.2017 23:20, Josef Bacik wrote:
> From: Josef Bacik <jbacik@fb.com>
>
> If we fail to prepare our pages for whatever reason (out of memory in
> our case) we need to make sure to drop the block_group->data_rwsem,
> otherwise hilarity ensues.
>
> Signed-off-by: Josef Bacik <jbacik@fb.com>
> ---
> fs/btrfs/free-space-cache.c | 6 +++++-
> 1 file changed, 5 insertions(+), 1 deletion(-)
>
> diff --git a/fs/btrfs/free-space-cache.c b/fs/btrfs/free-space-cache.c
> index cdc9f4015ec3..a6c643275210 100644
> --- a/fs/btrfs/free-space-cache.c
> +++ b/fs/btrfs/free-space-cache.c
> @@ -1263,8 +1263,12 @@ static int __btrfs_write_out_cache(struct btrfs_root *root, struct inode *inode,
>
> /* Lock all pages first so we can lock the extent safely. */
> ret = io_ctl_prepare_pages(io_ctl, inode, 0);
> - if (ret)
> + if (ret) {
> + if (block_group &&
> + (block_group->flags & BTRFS_BLOCK_GROUP_DATA))
> + up_write(&block_group->data_rwsem);
> goto out;
> + }
Which function after out: label causes a deadlock - btrfs_update_inode
(unlikely) or invalidate_inode_pages2?
>
> lock_extent_bits(&BTRFS_I(inode)->io_tree, 0, i_size_read(inode) - 1,
> &cached_state);
>
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: [PATCH] btrfs: fix deadlock when writing out space cache
2017-11-16 8:09 ` Nikolay Borisov
@ 2017-11-16 13:50 ` Chris Mason
2017-11-16 13:53 ` Nikolay Borisov
0 siblings, 1 reply; 8+ messages in thread
From: Chris Mason @ 2017-11-16 13:50 UTC (permalink / raw)
To: Nikolay Borisov, Josef Bacik, linux-btrfs, kernel-team; +Cc: Josef Bacik
On 11/16/2017 03:09 AM, Nikolay Borisov wrote:
>
>
> On 15.11.2017 23:20, Josef Bacik wrote:
>> From: Josef Bacik <jbacik@fb.com>
>>
>> If we fail to prepare our pages for whatever reason (out of memory in
>> our case) we need to make sure to drop the block_group->data_rwsem,
>> otherwise hilarity ensues.
>>
>> Signed-off-by: Josef Bacik <jbacik@fb.com>
>> ---
>> fs/btrfs/free-space-cache.c | 6 +++++-
>> 1 file changed, 5 insertions(+), 1 deletion(-)
>>
>> diff --git a/fs/btrfs/free-space-cache.c b/fs/btrfs/free-space-cache.c
>> index cdc9f4015ec3..a6c643275210 100644
>> --- a/fs/btrfs/free-space-cache.c
>> +++ b/fs/btrfs/free-space-cache.c
>> @@ -1263,8 +1263,12 @@ static int __btrfs_write_out_cache(struct btrfs_root *root, struct inode *inode,
>>
>> /* Lock all pages first so we can lock the extent safely. */
>> ret = io_ctl_prepare_pages(io_ctl, inode, 0);
>> - if (ret)
>> + if (ret) {
>> + if (block_group &&
>> + (block_group->flags & BTRFS_BLOCK_GROUP_DATA))
>> + up_write(&block_group->data_rwsem);
>> goto out;
>> + }
>
> Which function after out: label causes a deadlock - btrfs_update_inode
> (unlikely) or invalidate_inode_pages2?
Neither, out: just doesn't drop the data_rwsem mutex, so it leaves the
block group locked.
-chris
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: [PATCH] btrfs: fix deadlock when writing out space cache
2017-11-16 13:50 ` Chris Mason
@ 2017-11-16 13:53 ` Nikolay Borisov
0 siblings, 0 replies; 8+ messages in thread
From: Nikolay Borisov @ 2017-11-16 13:53 UTC (permalink / raw)
To: Chris Mason, Josef Bacik, linux-btrfs, kernel-team; +Cc: Josef Bacik
On 16.11.2017 15:50, Chris Mason wrote:
>
>
> On 11/16/2017 03:09 AM, Nikolay Borisov wrote:
>>
>>
>> On 15.11.2017 23:20, Josef Bacik wrote:
>>> From: Josef Bacik <jbacik@fb.com>
>>>
>>> If we fail to prepare our pages for whatever reason (out of memory in
>>> our case) we need to make sure to drop the block_group->data_rwsem,
>>> otherwise hilarity ensues.
>>>
>>> Signed-off-by: Josef Bacik <jbacik@fb.com>
>>> ---
>>> fs/btrfs/free-space-cache.c | 6 +++++-
>>> 1 file changed, 5 insertions(+), 1 deletion(-)
>>>
>>> diff --git a/fs/btrfs/free-space-cache.c b/fs/btrfs/free-space-cache.c
>>> index cdc9f4015ec3..a6c643275210 100644
>>> --- a/fs/btrfs/free-space-cache.c
>>> +++ b/fs/btrfs/free-space-cache.c
>>> @@ -1263,8 +1263,12 @@ static int __btrfs_write_out_cache(struct
>>> btrfs_root *root, struct inode *inode,
>>> /* Lock all pages first so we can lock the extent safely. */
>>> ret = io_ctl_prepare_pages(io_ctl, inode, 0);
>>> - if (ret)
>>> + if (ret) {
>>> + if (block_group &&
>>> + (block_group->flags & BTRFS_BLOCK_GROUP_DATA))
>>> + up_write(&block_group->data_rwsem);
>>> goto out;
>>> + }
>>
>> Which function after out: label causes a deadlock - btrfs_update_inode
>> (unlikely) or invalidate_inode_pages2?
>
> Neither, out: just doesn't drop the data_rwsem mutex, so it leaves the
> block group locked.
Ah, it has a return ret; and never hits the code under out_nospc, fair
enough.
>
> -chris
>
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: [PATCH] btrfs: fix deadlock when writing out space cache
2017-11-15 21:20 [PATCH] btrfs: fix deadlock when writing out space cache Josef Bacik
` (2 preceding siblings ...)
2017-11-16 8:09 ` Nikolay Borisov
@ 2017-11-20 17:22 ` David Sterba
3 siblings, 0 replies; 8+ messages in thread
From: David Sterba @ 2017-11-20 17:22 UTC (permalink / raw)
To: Josef Bacik; +Cc: linux-btrfs, kernel-team, Josef Bacik
On Wed, Nov 15, 2017 at 04:20:52PM -0500, Josef Bacik wrote:
> From: Josef Bacik <jbacik@fb.com>
>
> If we fail to prepare our pages for whatever reason (out of memory in
> our case) we need to make sure to drop the block_group->data_rwsem,
> otherwise hilarity ensues.
>
> Signed-off-by: Josef Bacik <jbacik@fb.com>
> ---
> fs/btrfs/free-space-cache.c | 6 +++++-
> 1 file changed, 5 insertions(+), 1 deletion(-)
>
> diff --git a/fs/btrfs/free-space-cache.c b/fs/btrfs/free-space-cache.c
> index cdc9f4015ec3..a6c643275210 100644
> --- a/fs/btrfs/free-space-cache.c
> +++ b/fs/btrfs/free-space-cache.c
> @@ -1263,8 +1263,12 @@ static int __btrfs_write_out_cache(struct btrfs_root *root, struct inode *inode,
>
> /* Lock all pages first so we can lock the extent safely. */
> ret = io_ctl_prepare_pages(io_ctl, inode, 0);
> - if (ret)
> + if (ret) {
> + if (block_group &&
> + (block_group->flags & BTRFS_BLOCK_GROUP_DATA))
> + up_write(&block_group->data_rwsem);
> goto out;
The unlocking sequence is in the exit block but does not have a label.
It would be better to reuse the code.
^ permalink raw reply [flat|nested] 8+ messages in thread
end of thread, other threads:[~2017-11-20 17:24 UTC | newest]
Thread overview: 8+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2017-11-15 21:20 [PATCH] btrfs: fix deadlock when writing out space cache Josef Bacik
2017-11-15 21:29 ` Omar Sandoval
2017-11-15 23:46 ` Liu Bo
2017-11-16 1:52 ` Chris Mason
2017-11-16 8:09 ` Nikolay Borisov
2017-11-16 13:50 ` Chris Mason
2017-11-16 13:53 ` Nikolay Borisov
2017-11-20 17:22 ` David Sterba
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).