loop: fix deadlock when sysfs and LOOP_CLR_FD race against each other

loop: fix deadlock when sysfs and LOOP_CLR_FD race against each other

[Kay Sievers]

From: Kay Sievers <kay.sievers@vrfy.org>
Subject: loop: fix deadlock when sysfs and LOOP_CLR_FD race against each other

LOOP_CLR_FD takes lo->lo_ctl_mutex and tries to remove the loop sysfs
files. Sysfs calls show() and waits for lo->lo_ctl_mutex. LOOP_CLR_FD
waits for show() to finish to remove the sysfs file.

  cat /sys/class/block/loop0/loop/backing_file
    mutex_lock_nested+0x176/0x350
    ? loop_attr_do_show_backing_file+0x2f/0xd0 [loop]
    ? loop_attr_do_show_backing_file+0x2f/0xd0 [loop]
    loop_attr_do_show_backing_file+0x2f/0xd0 [loop]
    dev_attr_show+0x1b/0x60
    ? sysfs_read_file+0x86/0x1a0
    ? __get_free_pages+0x12/0x50
    sysfs_read_file+0xaf/0x1a0

  ioctl(LOOP_CLR_FD):
    wait_for_common+0x12c/0x180
    ? try_to_wake_up+0x2a0/0x2a0
    wait_for_completion+0x18/0x20
    sysfs_deactivate+0x178/0x180
    ? sysfs_addrm_finish+0x43/0x70
    ? sysfs_addrm_start+0x1d/0x20
    sysfs_addrm_finish+0x43/0x70
    sysfs_hash_and_remove+0x85/0xa0
    sysfs_remove_group+0x59/0x100
    loop_clr_fd+0x1dc/0x3f0 [loop]
    lo_ioctl+0x223/0x7a0 [loop]

Instead of taking the lo_ctl_mutex from sysfs code, take the inner
lo->lo_lock, to protect the access to the backing_file data.

Thanks to Tejun for help debugging and finding a solution.

Cc: Milan Broz <mbroz@redhat.com>
Cc: Tejun Heo <tj@kernel.org>
Signed-off-by: Kay Sievers <kay.sievers@vrfy.org>
---
 drivers/block/loop.c |    6 ++++--
 1 file changed, 4 insertions(+), 2 deletions(-)

--- a/drivers/block/loop.c
+++ b/drivers/block/loop.c
@@ -743,10 +743,10 @@ static ssize_t loop_attr_backing_file_sh
 	ssize_t ret;
 	char *p = NULL;
 
-	mutex_lock(&lo->lo_ctl_mutex);
+	spin_lock_irq(&lo->lo_lock);
 	if (lo->lo_backing_file)
 		p = d_path(&lo->lo_backing_file->f_path, buf, PAGE_SIZE - 1);
-	mutex_unlock(&lo->lo_ctl_mutex);
+	spin_unlock_irq(&lo->lo_lock);
 
 	if (IS_ERR_OR_NULL(p))
 		ret = PTR_ERR(p);
@@ -1000,7 +1000,9 @@ static int loop_clr_fd(struct loop_devic
 
 	kthread_stop(lo->lo_thread);
 
+	spin_lock_irq(&lo->lo_lock);
 	lo->lo_backing_file = NULL;
+	spin_unlock_irq(&lo->lo_lock);
 
 	loop_release_xfer(lo);
 	lo->transfer = NULL;

Re: loop: fix deadlock when sysfs and LOOP_CLR_FD race against each other

[Jens Axboe]

On 2011-07-30 21:19, Kay Sievers wrote:
> From: Kay Sievers <kay.sievers@vrfy.org>
> Subject: loop: fix deadlock when sysfs and LOOP_CLR_FD race against each other
> 
> LOOP_CLR_FD takes lo->lo_ctl_mutex and tries to remove the loop sysfs
> files. Sysfs calls show() and waits for lo->lo_ctl_mutex. LOOP_CLR_FD
> waits for show() to finish to remove the sysfs file.
> 
>   cat /sys/class/block/loop0/loop/backing_file
>     mutex_lock_nested+0x176/0x350
>     ? loop_attr_do_show_backing_file+0x2f/0xd0 [loop]
>     ? loop_attr_do_show_backing_file+0x2f/0xd0 [loop]
>     loop_attr_do_show_backing_file+0x2f/0xd0 [loop]
>     dev_attr_show+0x1b/0x60
>     ? sysfs_read_file+0x86/0x1a0
>     ? __get_free_pages+0x12/0x50
>     sysfs_read_file+0xaf/0x1a0
> 
>   ioctl(LOOP_CLR_FD):
>     wait_for_common+0x12c/0x180
>     ? try_to_wake_up+0x2a0/0x2a0
>     wait_for_completion+0x18/0x20
>     sysfs_deactivate+0x178/0x180
>     ? sysfs_addrm_finish+0x43/0x70
>     ? sysfs_addrm_start+0x1d/0x20
>     sysfs_addrm_finish+0x43/0x70
>     sysfs_hash_and_remove+0x85/0xa0
>     sysfs_remove_group+0x59/0x100
>     loop_clr_fd+0x1dc/0x3f0 [loop]
>     lo_ioctl+0x223/0x7a0 [loop]
> 
> Instead of taking the lo_ctl_mutex from sysfs code, take the inner
> lo->lo_lock, to protect the access to the backing_file data.
> 
> Thanks to Tejun for help debugging and finding a solution.

Looks good, looks like something that should have a stable tag as well?

-- 
Jens Axboe

Re: loop: fix deadlock when sysfs and LOOP_CLR_FD race against each other

[Kay Sievers]

On Sun, Jul 31, 2011 at 22:20, Jens Axboe <jaxboe@fusionio.com> wrote:
> On 2011-07-30 21:19, Kay Sievers wrote:
>> Instead of taking the lo_ctl_mutex from sysfs code, take the inner
>> lo->lo_lock, to protect the access to the backing_file data.
>>
>> Thanks to Tejun for help debugging and finding a solution.
>
> Looks good, looks like something that should have a stable tag as well?

Right, I think it makes sense to have that in -stable.

It's pretty hard to trigger, I had multiple threads running, crawling
/sys and adding/binding/unbinding/removing 1000s of loop devices, and
it takes several minutes sometimes until its hit. So I only tested it
on top of the 3 loop-control patches, but the issue should exist in
the current code as well.

Thanks,
Kay

Re: loop: fix deadlock when sysfs and LOOP_CLR_FD race against each other

[Jens Axboe]

On 2011-07-31 22:42, Kay Sievers wrote:
> On Sun, Jul 31, 2011 at 22:20, Jens Axboe <jaxboe@fusionio.com> wrote:
>> On 2011-07-30 21:19, Kay Sievers wrote:
>>> Instead of taking the lo_ctl_mutex from sysfs code, take the inner
>>> lo->lo_lock, to protect the access to the backing_file data.
>>>
>>> Thanks to Tejun for help debugging and finding a solution.
>>
>> Looks good, looks like something that should have a stable tag as well?
> 
> Right, I think it makes sense to have that in -stable.
> 
> It's pretty hard to trigger, I had multiple threads running, crawling
> /sys and adding/binding/unbinding/removing 1000s of loop devices, and
> it takes several minutes sometimes until its hit. So I only tested it
> on top of the 3 loop-control patches, but the issue should exist in
> the current code as well.

I applied those for 3.1 as well, but I'm thinking they probably should
have been queued up for 3.2 instead.

-- 
Jens Axboe