Re: INFO: task hung in lo_ioctl

Re: INFO: task hung in lo_ioctl

[Dmitry Vyukov]

On Sun, Dec 10, 2017 at 2:32 PM, syzbot
<bot+abdba5bc6de135d7622f00756da97998425b6de5@syzkaller.appspotmail.com>
wrote:
> Hello,
>
> syzkaller hit the following crash on
> ad4dac17f9d563b9e34aab78a34293b10993e9b5
> git://git.kernel.org/pub/scm/linux/kernel/git/next/linux-next.git/master
> compiler: gcc (GCC) 7.1.1 20170620
> .config is attached
> Raw console output is attached.
>
> Unfortunately, I don't have any reproducer for this bug yet.
>
>
> INFO: task syz-executor3:12157 blocked for more than 120 seconds.
>       Not tainted 4.15.0-rc2-next-20171208+ #63
> "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
> syz-executor3   D25536 12157   3362 0x00000004
> Call Trace:
>  context_switch kernel/sched/core.c:2800 [inline]
>  __schedule+0x8eb/0x2060 kernel/sched/core.c:3376
>  schedule+0xf5/0x430 kernel/sched/core.c:3435
>  schedule_preempt_disabled+0x10/0x20 kernel/sched/core.c:3493
>  __mutex_lock_common kernel/locking/mutex.c:833 [inline]
>  __mutex_lock+0xaad/0x1a80 kernel/locking/mutex.c:893
>  mutex_lock_nested+0x16/0x20 kernel/locking/mutex.c:908
>  lo_ioctl+0x8b/0x1b90 drivers/block/loop.c:1355
>  __blkdev_driver_ioctl block/ioctl.c:303 [inline]
>  blkdev_ioctl+0x1759/0x1e00 block/ioctl.c:601
>  block_ioctl+0xea/0x130 fs/block_dev.c:1860
>  vfs_ioctl fs/ioctl.c:46 [inline]
>  do_vfs_ioctl+0x1b1/0x1530 fs/ioctl.c:686
>  SYSC_ioctl fs/ioctl.c:701 [inline]
>  SyS_ioctl+0x8f/0xc0 fs/ioctl.c:692
>  entry_SYSCALL_64_fastpath+0x1f/0x96
> RIP: 0033:0x452a39
> RSP: 002b:00007fd85b57ec58 EFLAGS: 00000212 ORIG_RAX: 0000000000000010
> RAX: ffffffffffffffda RBX: 0000000000758190 RCX: 0000000000452a39
> RDX: 0000000020e67fd8 RSI: 00000000400454ca RDI: 0000000000000016
> RBP: 000000000000005b R08: 0000000000000000 R09: 0000000000000000
> R10: 0000000000000000 R11: 0000000000000212 R12: 00000000006ee928
> R13: 00000000ffffffff R14: 00007fd85b57f6d4 R15: 0000000000000002
>
> Showing all locks held in the system:
> 2 locks held by khungtaskd/671:
>  #0:  (rcu_read_lock){....}, at: [<0000000053bc3983>]
> check_hung_uninterruptible_tasks kernel/hung_task.c:175 [inline]
>  #0:  (rcu_read_lock){....}, at: [<0000000053bc3983>] watchdog+0x1c5/0xd60
> kernel/hung_task.c:249
>  #1:  (tasklist_lock){.+.+}, at: [<00000000c2c6381b>]
> debug_show_all_locks+0xd3/0x400 kernel/locking/lockdep.c:4554
> 2 locks held by getty/3115:
>  #0:  (&tty->ldisc_sem){++++}, at: [<00000000e7f92d1f>]
> ldsem_down_read+0x37/0x40 drivers/tty/tty_ldsem.c:365
>  #1:  (&ldata->atomic_read_lock){+.+.}, at: [<00000000f15a7c4f>]
> n_tty_read+0x2f2/0x1a10 drivers/tty/n_tty.c:2131
> 2 locks held by getty/3116:
>  #0:  (&tty->ldisc_sem){++++}, at: [<00000000e7f92d1f>]
> ldsem_down_read+0x37/0x40 drivers/tty/tty_ldsem.c:365
>  #1:  (&ldata->atomic_read_lock){+.+.}, at: [<00000000f15a7c4f>]
> n_tty_read+0x2f2/0x1a10 drivers/tty/n_tty.c:2131
> 2 locks held by getty/3117:
>  #0:  (&tty->ldisc_sem){++++}, at: [<00000000e7f92d1f>]
> ldsem_down_read+0x37/0x40 drivers/tty/tty_ldsem.c:365
>  #1:  (&ldata->atomic_read_lock){+.+.}, at: [<00000000f15a7c4f>]
> n_tty_read+0x2f2/0x1a10 drivers/tty/n_tty.c:2131
> 2 locks held by getty/3118:
>  #0:  (&tty->ldisc_sem){++++}, at: [<00000000e7f92d1f>]
> ldsem_down_read+0x37/0x40 drivers/tty/tty_ldsem.c:365
>  #1:  (&ldata->atomic_read_lock){+.+.}, at: [<00000000f15a7c4f>]
> n_tty_read+0x2f2/0x1a10 drivers/tty/n_tty.c:2131
> 2 locks held by getty/3119:
>  #0:  (&tty->ldisc_sem){++++}, at: [<00000000e7f92d1f>]
> ldsem_down_read+0x37/0x40 drivers/tty/tty_ldsem.c:365
>  #1:  (&ldata->atomic_read_lock){+.+.}, at: [<00000000f15a7c4f>]
> n_tty_read+0x2f2/0x1a10 drivers/tty/n_tty.c:2131
> 2 locks held by getty/3120:
>  #0:  (&tty->ldisc_sem){++++}, at: [<00000000e7f92d1f>]
> ldsem_down_read+0x37/0x40 drivers/tty/tty_ldsem.c:365
>  #1:  (&ldata->atomic_read_lock){+.+.}, at: [<00000000f15a7c4f>]
> n_tty_read+0x2f2/0x1a10 drivers/tty/n_tty.c:2131
> 1 lock held by syz-executor3/12157:
>  #0:  (&lo->lo_ctl_mutex/1){+.+.}, at: [<00000000d5130113>]
> lo_ioctl+0x8b/0x1b90 drivers/block/loop.c:1355
> 1 lock held by syz-executor7/12154:
>  #0:  (&lo->lo_ctl_mutex/1){+.+.}, at: [<00000000d5130113>]
> lo_ioctl+0x8b/0x1b90 drivers/block/loop.c:1355
> 1 lock held by syz-executor7/12168:
>  #0:  (&lo->lo_ctl_mutex/1){+.+.}, at: [<00000000d5130113>]
> lo_ioctl+0x8b/0x1b90 drivers/block/loop.c:1355
> 1 lock held by syz-executor5/12216:
>  #0:  (&lo->lo_ctl_mutex/1){+.+.}, at: [<00000000d5130113>]
> lo_ioctl+0x8b/0x1b90 drivers/block/loop.c:1355
> 2 locks held by getty/12231:
>  #0:  (&tty->ldisc_sem){++++}, at: [<00000000e7f92d1f>]
> ldsem_down_read+0x37/0x40 drivers/tty/tty_ldsem.c:365
>  #1:  (&ldata->atomic_read_lock){+.+.}, at: [<00000000f15a7c4f>]
> n_tty_read+0x2f2/0x1a10 drivers/tty/n_tty.c:2131
> 1 lock held by blkid/12232:
>  #0:  (&lo->lo_ctl_mutex/1){+.+.}, at: [<00000000d5130113>]
> lo_ioctl+0x8b/0x1b90 drivers/block/loop.c:1355
>
> =============================================
>
> NMI backtrace for cpu 0
> CPU: 0 PID: 671 Comm: khungtaskd Not tainted 4.15.0-rc2-next-20171208+ #63
> Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS
> Google 01/01/2011
> Call Trace:
>  __dump_stack lib/dump_stack.c:17 [inline]
>  dump_stack+0x194/0x257 lib/dump_stack.c:53
>  nmi_cpu_backtrace+0x1d2/0x210 lib/nmi_backtrace.c:103
>  nmi_trigger_cpumask_backtrace+0x122/0x180 lib/nmi_backtrace.c:62
>  arch_trigger_cpumask_backtrace+0x14/0x20 arch/x86/kernel/apic/hw_nmi.c:38
>  trigger_all_cpu_backtrace include/linux/nmi.h:138 [inline]
>  check_hung_task kernel/hung_task.c:132 [inline]
>  check_hung_uninterruptible_tasks kernel/hung_task.c:190 [inline]
>  watchdog+0x90c/0xd60 kernel/hung_task.c:249
>  kthread+0x37a/0x440 kernel/kthread.c:238
>  ret_from_fork+0x24/0x30 arch/x86/entry/entry_64.S:524
> Sending NMI from CPU 0 to CPUs 1:
> NMI backtrace for cpu 1
> CPU: 1 PID: 0 Comm: swapper/1 Not tainted 4.15.0-rc2-next-20171208+ #63
> Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS
> Google 01/01/2011
> RIP: 0010:unwind_next_frame.part.6+0x5f1/0xb40 include/linux/compiler.h:183
> RSP: 0018:ffff8801db306e28 EFLAGS: 00000806
> RAX: dffffc0000000000 RBX: ffff8801db306f48 RCX: ffff8801db306ef8
> RDX: ffff8801db306f70 RSI: 1ffff1003b660dcf RDI: ffff8801d9f97f48
> RBP: ffff8801db306f20 R08: ffff8801db306f80 R09: ffff8801d9f88300
> R10: 000000000000000b R11: ffffed003b660df3 R12: 1ffff1003b660dcb
> R13: ffff8801db307490 R14: 1ffff1003b660dcf R15: ffff8801db306f98
> FS:  0000000000000000(0000) GS:ffff8801db300000(0000) knlGS:0000000000000000
> CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> CR2: 0000000001e5e000 CR3: 00000001cb1b5000 CR4: 00000000001406e0
> DR0: 0000000020000000 DR1: 0000000020000000 DR2: 0000000000000000
> DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000600
> Call Trace:
>  <IRQ>
>  unwind_next_frame+0x3e/0x50 arch/x86/kernel/unwind_frame.c:287
>  __save_stack_trace+0x6e/0xd0 arch/x86/kernel/stacktrace.c:44
>  save_stack_trace+0x1a/0x20 arch/x86/kernel/stacktrace.c:60
>  add_xhlock kernel/locking/lockdep.c:4880 [inline]
>  check_add_xhlock kernel/locking/lockdep.c:4907 [inline]
>  lock_acquire_crosslock kernel/locking/lockdep.c:4979 [inline]
>  __lock_acquire+0x324e/0x47f0 kernel/locking/lockdep.c:3503
>  lock_acquire+0x1d5/0x580 kernel/locking/lockdep.c:4004
>  __raw_spin_lock_irqsave include/linux/spinlock_api_smp.h:110 [inline]
>  _raw_spin_lock_irqsave+0x96/0xc0 kernel/locking/spinlock.c:159
>  rq_lock_irqsave kernel/sched/sched.h:1750 [inline]
>  update_blocked_averages+0x124/0x1b60 kernel/sched/fair.c:7353
>  rebalance_domains+0x145/0xcc0 kernel/sched/fair.c:9122
>  run_rebalance_domains+0x381/0x780 kernel/sched/fair.c:9383
>  __do_softirq+0x29d/0xbb2 kernel/softirq.c:285
>  invoke_softirq kernel/softirq.c:365 [inline]
>  irq_exit+0x1d3/0x210 kernel/softirq.c:405
>  scheduler_ipi+0x32a/0x830 kernel/sched/core.c:1804
>  smp_reschedule_interrupt+0xe6/0x670 arch/x86/kernel/smp.c:277
>  reschedule_interrupt+0xa9/0xb0 arch/x86/entry/entry_64.S:944
>  </IRQ>
> RIP: 0010:native_safe_halt+0x6/0x10 arch/x86/include/asm/irqflags.h:54
> RSP: 0018:ffff8801d9f97da8 EFLAGS: 00000282 ORIG_RAX: ffffffffffffff02
> RAX: dffffc0000000000 RBX: 1ffff1003b3f2fb8 RCX: 0000000000000000
> RDX: 1ffffffff0c5975c RSI: 0000000000000001 RDI: ffffffff862cbae0
> RBP: ffff8801d9f97da8 R08: 0000000000000000 R09: 0000000000000000
> R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000001
> R13: ffff8801d9f97e60 R14: ffffffff869efaa0 R15: 0000000000000000
>  arch_safe_halt arch/x86/include/asm/paravirt.h:93 [inline]
>  default_idle+0xbf/0x430 arch/x86/kernel/process.c:355
>  arch_cpu_idle+0xa/0x10 arch/x86/kernel/process.c:346
>  default_idle_call+0x36/0x90 kernel/sched/idle.c:98
>  cpuidle_idle_call kernel/sched/idle.c:156 [inline]
>  do_idle+0x24a/0x3b0 kernel/sched/idle.c:246
>  cpu_startup_entry+0x18/0x20 kernel/sched/idle.c:351
>  start_secondary+0x330/0x460 arch/x86/kernel/smpboot.c:277
>  secondary_startup_64+0xa5/0xb0 arch/x86/kernel/head_64.S:237
> Code: 8b 6d 00 4c 89 f6 48 c1 ee 03 80 3c 06 00 0f 85 e9 04 00 00 49 c1 ee
> 03 48 b8 00 00 00 00 00 fc ff df 4c 89 69 80 41 c6 04 06 f8 <e9> 97 fb ff ff
> 80 3d f7 60 6e 05 00 0f 85 fc fc ff ff 49 8d be


This seems to be about loop device, but was mis-attributed to
scheduler. So +loop maintainers.

FTR, mis-attribution in syzbot was fixed in:
https://github.com/google/syzkaller/commit/27f5dfefff3cd00795d9048ecc7bc70965e03ea7

> ---
> This bug is generated by a dumb bot. It may contain errors.
> See https://goo.gl/tpsmEJ for details.
> Direct all questions to syzkaller@googlegroups.com.
> Please credit me with: Reported-by: syzbot <syzkaller@googlegroups.com>
>
> syzbot will keep track of this bug report.
> Once a fix for this bug is merged into any tree, reply to this email with:
> #syz fix: exact-commit-title
> To mark this as a duplicate of another syzbot report, please reply with:
> #syz dup: exact-subject-of-another-report
> If it's a one-off invalid bug report, please reply with:
> #syz invalid
> Note: if the crash happens again, it will cause creation of a new bug
> report.
> Note: all commands must start from beginning of the line in the email body.
>
> --
> You received this message because you are subscribed to the Google Groups
> "syzkaller-bugs" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to syzkaller-bugs+unsubscribe@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/syzkaller-bugs/94eb2c0810d04f5a46055ffc71aa%40google.com.
> For more options, visit https://groups.google.com/d/optout.

Re: INFO: task hung in lo_ioctl

[Tetsuo Handa]

This seems to be an AB-BA deadlock where the lockdep cannot report (due to use of nested lock?).

When PID=6540 was (reported as hung) at mutex_lock_nested(&lo->lo_ctl_mutex, 1) (id=43ca8836),
it was already holding down_write_nested(&s->s_umount, SINGLE_DEPTH_NESTING) (id=566d4c39).
But when PID=6541 was (which would have been reported as hung if sysctl_hung_task_panic
were not set) at down_read(&sb->s_umount) (id=566d4c39), it was already holding
mutex_lock_nested(&lo->lo_ctl_mutex, 1) (id=43ca8836).

----------------------------------------
INFO: task syz-executor0:6540 blocked for more than 120 seconds.
      Not tainted 4.16.0+ #13
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
syz-executor0   D23560  6540   4521 0x80000004
Call Trace:
 context_switch kernel/sched/core.c:2848 [inline]
 __schedule+0x8fb/0x1ef0 kernel/sched/core.c:3490
 schedule+0xf5/0x430 kernel/sched/core.c:3549
 schedule_preempt_disabled+0x10/0x20 kernel/sched/core.c:3607
 __mutex_lock_common kernel/locking/mutex.c:833 [inline]
 __mutex_lock+0xb7f/0x1810 kernel/locking/mutex.c:893
 mutex_lock_nested+0x16/0x20 kernel/locking/mutex.c:908
 lo_ioctl+0x8b/0x1b70 drivers/block/loop.c:1355
 __blkdev_driver_ioctl block/ioctl.c:303 [inline]
 blkdev_ioctl+0x1759/0x1e00 block/ioctl.c:601
 ioctl_by_bdev+0xa5/0x110 fs/block_dev.c:2060
 isofs_get_last_session fs/isofs/inode.c:567 [inline]
 isofs_fill_super+0x2ba9/0x3bc0 fs/isofs/inode.c:660
 mount_bdev+0x2b7/0x370 fs/super.c:1119
 isofs_mount+0x34/0x40 fs/isofs/inode.c:1560
 mount_fs+0x66/0x2d0 fs/super.c:1222

2 locks held by syz-executor0/6540:
 #0: 00000000566d4c39 (&type->s_umount_key#49/1){+.+.}, at: alloc_super fs/super.c:211 [inline]
 #0: 00000000566d4c39 (&type->s_umount_key#49/1){+.+.}, at: sget_userns+0x3b2/0xe60 fs/super.c:502 /* down_write_nested(&s->s_umount, SINGLE_DEPTH_NESTING); */
 #1: 0000000043ca8836 (&lo->lo_ctl_mutex/1){+.+.}, at: lo_ioctl+0x8b/0x1b70 drivers/block/loop.c:1355 /* mutex_lock_nested(&lo->lo_ctl_mutex, 1); */
3 locks held by syz-executor7/6541:
 #0: 0000000043ca8836 (&lo->lo_ctl_mutex/1){+.+.}, at: lo_ioctl+0x8b/0x1b70 drivers/block/loop.c:1355 /* mutex_lock_nested(&lo->lo_ctl_mutex, 1); */
 #1: 000000007bf3d3f9 (&bdev->bd_mutex){+.+.}, at: blkdev_reread_part+0x1e/0x40 block/ioctl.c:192
 #2: 00000000566d4c39 (&type->s_umount_key#50){.+.+}, at: __get_super.part.10+0x1d3/0x280 fs/super.c:663 /* down_read(&sb->s_umount); */
----------------------------------------

sget() is using down_write_nested(&s->s_umount, SINGLE_DEPTH_NESTING)
with a comment block asserting that there is no risk of deadlock

	/*
	 * sget() can have s_umount recursion.
	 *
	 * When it cannot find a suitable sb, it allocates a new
	 * one (this one), and tries again to find a suitable old
	 * one.
	 *
	 * In case that succeeds, it will acquire the s_umount
	 * lock of the old one. Since these are clearly distrinct
	 * locks, and this object isn't exposed yet, there's no
	 * risk of deadlocks.
	 *
	 * Annotate this by putting this lock in a different
	 * subclass.
	 */

but this object (id=566d4c39) is already locked by other thread.
What is happening here?

Re: INFO: task hung in lo_ioctl

[Tetsuo Handa]

On 2018/04/05 0:23, Tetsuo Handa wrote:
> This seems to be an AB-BA deadlock where the lockdep cannot report (due to use of nested lock?).
> What is happening here?
> 

This patch does not directly fix the bug syzbot is reporting, but could be relevant.
Maybe try replacing mutex_lock_killable_nested() with mutex_lock_killable() and
check what the lockdep will say?



>From 0b006d536b2e439f347eb857e482fc304e84fd1d Mon Sep 17 00:00:00 2001
From: Tetsuo Handa <penguin-kernel@I-love.SAKURA.ne.jp>
Date: Fri, 6 Apr 2018 20:52:10 +0900
Subject: [PATCH] block/loop: fix lock imbalance bug at lo_ioctl

Commit 2d1d4c1e591fd40b ("loop: don't call into filesystem while holding
lo_ctl_mutex") introduced mutex lock imbalance bug where syzbot would
trigger. The caller of loop_get_status64() assumes that mutex_unlock() is
already called by loop_get_status() but loop_get_status64() does not
always call loop_get_status().

Commit f028f3b2f987ebc6 ("loop: fix circular locking in loop_clr_fd()")
also dropped the mutex for deadlock avoidance reason. But we should
recheck whether we have to drop the mutex. Dropping the mutex at the
middle of an ioctl() request is not nice, but syzbot is reporting a
deadlock inside loop_reread_partitions() which is called by loop_set_fd()
and loop_change_fd().

Signed-off-by: Tetsuo Handa <penguin-kernel@I-love.SAKURA.ne.jp>
Fixes: 2d1d4c1e591fd40b ("loop: don't call into filesystem while holding lo_ctl_mutex")
Cc: Omar Sandoval <osandov@fb.com>
Cc: Nikanth Karthikesan <knikanth@suse.de>
Cc: Jens Axboe <jens.axboe@oracle.com>
---
 drivers/block/loop.c | 17 ++++++-----------
 1 file changed, 6 insertions(+), 11 deletions(-)

diff --git a/drivers/block/loop.c b/drivers/block/loop.c
index 264abaa..64033e7 100644
--- a/drivers/block/loop.c
+++ b/drivers/block/loop.c
@@ -1362,7 +1362,7 @@ static int lo_ioctl(struct block_device *bdev, fmode_t mode,
 
 	err = mutex_lock_killable_nested(&lo->lo_ctl_mutex, 1);
 	if (err)
-		goto out_unlocked;
+		return err;
 
 	switch (cmd) {
 	case LOOP_SET_FD:
@@ -1372,10 +1372,7 @@ static int lo_ioctl(struct block_device *bdev, fmode_t mode,
 		err = loop_change_fd(lo, bdev, arg);
 		break;
 	case LOOP_CLR_FD:
-		/* loop_clr_fd would have unlocked lo_ctl_mutex on success */
 		err = loop_clr_fd(lo);
-		if (!err)
-			goto out_unlocked;
 		break;
 	case LOOP_SET_STATUS:
 		err = -EPERM;
@@ -1385,8 +1382,7 @@ static int lo_ioctl(struct block_device *bdev, fmode_t mode,
 		break;
 	case LOOP_GET_STATUS:
 		err = loop_get_status_old(lo, (struct loop_info __user *) arg);
-		/* loop_get_status() unlocks lo_ctl_mutex */
-		goto out_unlocked;
+		break;
 	case LOOP_SET_STATUS64:
 		err = -EPERM;
 		if ((mode & FMODE_WRITE) || capable(CAP_SYS_ADMIN))
@@ -1395,8 +1391,7 @@ static int lo_ioctl(struct block_device *bdev, fmode_t mode,
 		break;
 	case LOOP_GET_STATUS64:
 		err = loop_get_status64(lo, (struct loop_info64 __user *) arg);
-		/* loop_get_status() unlocks lo_ctl_mutex */
-		goto out_unlocked;
+		break;
 	case LOOP_SET_CAPACITY:
 		err = -EPERM;
 		if ((mode & FMODE_WRITE) || capable(CAP_SYS_ADMIN))
@@ -1415,9 +1410,9 @@ static int lo_ioctl(struct block_device *bdev, fmode_t mode,
 	default:
 		err = lo->ioctl ? lo->ioctl(lo, cmd, arg) : -EINVAL;
 	}
-	mutex_unlock(&lo->lo_ctl_mutex);
-
-out_unlocked:
+	/* Temporary hack for handling lock imbalance. */
+	if (__mutex_owner(&lo->lo_ctl_mutex) == current)
+		mutex_unlock(&lo->lo_ctl_mutex);
 	return err;
 }
 
-- 
1.8.3.1

Re: INFO: task hung in lo_ioctl

[Peter Zijlstra]

On Fri, Apr 06, 2018 at 09:04:18PM +0900, Tetsuo Handa wrote:
> +	/* Temporary hack for handling lock imbalance. */
> +	if (__mutex_owner(&lo->lo_ctl_mutex) == current)
> +		mutex_unlock(&lo->lo_ctl_mutex);

ARGGH.. you didn't read the comment we put on that?

Re: INFO: task hung in lo_ioctl

[Tetsuo Handa]

Peter Zijlstra wrote:
> On Fri, Apr 06, 2018 at 09:04:18PM +0900, Tetsuo Handa wrote:
> > +	/* Temporary hack for handling lock imbalance. */
> > +	if (__mutex_owner(&lo->lo_ctl_mutex) == current)
> > +		mutex_unlock(&lo->lo_ctl_mutex);
> 
> ARGGH.. you didn't read the comment we put on that?
> 

Commit 5b52330bbfe63b33 ("audit: fix auditd/kernel connection state tracking")
is using __mutex_owner(). ;-)

Of course, regarding loop module, we will be able to add a flag variable
associated with lo->lo_ctl_mutex. But that will be done after we solved
the deadlock problem. I think whether we need to drop lo->lo_ctl_mutex is
not clear. Maybe

-	err = mutex_lock_killable_nested(&lo->lo_ctl_mutex, 1);
+	err = mutex_lock_killable(&lo->lo_ctl_mutex);

on top of this patch and listen to the lockdep?

Commit f028f3b2f987ebc6 ("loop: fix circular locking in loop_clr_fd()") says

  A simple way to silence lockdep could be to mark the lo_ctl_mutex
  in ioctl to be a sub class, but this might mask some other real bugs.

and we are currently hitting a deadlock problem.

Re: INFO: task hung in lo_ioctl

[Peter Zijlstra]

On Fri, Apr 06, 2018 at 10:55:03PM +0900, Tetsuo Handa wrote:
> Peter Zijlstra wrote:
> > On Fri, Apr 06, 2018 at 09:04:18PM +0900, Tetsuo Handa wrote:
> > > +	/* Temporary hack for handling lock imbalance. */
> > > +	if (__mutex_owner(&lo->lo_ctl_mutex) == current)
> > > +		mutex_unlock(&lo->lo_ctl_mutex);
> > 
> > ARGGH.. you didn't read the comment we put on that?
> > 
> 
> Commit 5b52330bbfe63b33 ("audit: fix auditd/kernel connection state tracking")
> is using __mutex_owner(). ;-)

That got removed and the warning added.

Re: INFO: task hung in lo_ioctl

[Omar Sandoval]

On Fri, Apr 06, 2018 at 05:43:43PM +0200, Peter Zijlstra wrote:
> On Fri, Apr 06, 2018 at 10:55:03PM +0900, Tetsuo Handa wrote:
> > Peter Zijlstra wrote:
> > > On Fri, Apr 06, 2018 at 09:04:18PM +0900, Tetsuo Handa wrote:
> > > > +	/* Temporary hack for handling lock imbalance. */
> > > > +	if (__mutex_owner(&lo->lo_ctl_mutex) == current)
> > > > +		mutex_unlock(&lo->lo_ctl_mutex);
> > > 
> > > ARGGH.. you didn't read the comment we put on that?
> > > 
> > 
> > Commit 5b52330bbfe63b33 ("audit: fix auditd/kernel connection state tracking")
> > is using __mutex_owner(). ;-)
> 
> That got removed and the warning added.

Seems easy enough to fix without resorting to __mutex_owner() (untested):


diff --git a/drivers/block/loop.c b/drivers/block/loop.c
index 264abaaff662..cee258d12a1e 100644
--- a/drivers/block/loop.c
+++ b/drivers/block/loop.c
@@ -1300,12 +1300,13 @@ loop_get_status_old(struct loop_device *lo, struct loop_info __user *arg) {
 static int
 loop_get_status64(struct loop_device *lo, struct loop_info64 __user *arg) {
 	struct loop_info64 info64;
-	int err = 0;
+	int err;
 
-	if (!arg)
-		err = -EINVAL;
-	if (!err)
-		err = loop_get_status(lo, &info64);
+	if (!arg) {
+		mutex_unlock(&lo->lo_ctl_mutex);
+		return -EINVAL;
+	}
+	err = loop_get_status(lo, &info64);
 	if (!err && copy_to_user(arg, &info64, sizeof(info64)))
 		err = -EFAULT;
 

I'll test it and send it up when I get into the office.

Re: INFO: task hung in lo_ioctl

[Dmitry Vyukov]

On Fri, Apr 6, 2018 at 5:59 PM Omar Sandoval <osandov@osandov.com> wrote:
>
> On Fri, Apr 06, 2018 at 05:43:43PM +0200, Peter Zijlstra wrote:
> > On Fri, Apr 06, 2018 at 10:55:03PM +0900, Tetsuo Handa wrote:
> > > Peter Zijlstra wrote:
> > > > On Fri, Apr 06, 2018 at 09:04:18PM +0900, Tetsuo Handa wrote:
> > >
 > > +       /* Temporary hack for handling lock imbalance. */
> > > > > +       if (__mutex_owner(&lo->lo_ctl_mutex) == current)
> > > > > +               mutex_unlock(&lo->lo_ctl_mutex);
> > > >
> > > > ARGGH.. you didn't read the comment we put on that?
> > > >
> > >
> > > Commit 5b52330bbfe63b33 ("audit: fix auditd/kernel connection state tracking")
> > > is using __mutex_owner(). ;-)
> >
> > That got removed and the warning added.
>
> Seems easy enough to fix without resorting to __mutex_owner() (untested):
>
>
> diff --git a/drivers/block/loop.c b/drivers/block/loop.c
> index 264abaaff662..cee258d12a1e 100644
> --- a/drivers/block/loop.c
> +++ b/drivers/block/loop.c
> @@ -1300,12 +1300,13 @@ loop_get_status_old(struct loop_device *lo, struct loop_info __user *arg) {
>  static int
>  loop_get_status64(struct loop_device *lo, struct loop_info64 __user *arg) {
>         struct loop_info64 info64;
> -       int err = 0;
> +       int err;
>
> -       if (!arg)
> -               err = -EINVAL;
> -       if (!err)
> -               err = loop_get_status(lo, &info64);
> +       if (!arg) {
> +               mutex_unlock(&lo->lo_ctl_mutex);
> +               return -EINVAL;
> +       }
> +       err = loop_get_status(lo, &info64);
>         if (!err && copy_to_user(arg, &info64, sizeof(info64)))
>                 err = -EFAULT;
>
>
> I'll test it and send it up when I get into the office.


Was this ever submitted? Or some other fix for this?

The bug is still open, but last happened 289 days ago:
https://syzkaller.appspot.com/bug?id=608144371e7fc2cb6285b9ed871fb1eb817a61ce

But it also has 10 duplicates, some of which happened much more recently.
If a fix was submitted, but Reported-by tag wasn't added this open bug
can now mask lots of other new bugs.

Re: INFO: task hung in lo_ioctl

[Tetsuo Handa]

On 2019/01/20 3:56, Dmitry Vyukov wrote:
>> Seems easy enough to fix without resorting to __mutex_owner() (untested):
>>
>>
>> diff --git a/drivers/block/loop.c b/drivers/block/loop.c
>> index 264abaaff662..cee258d12a1e 100644
>> --- a/drivers/block/loop.c
>> +++ b/drivers/block/loop.c
>> @@ -1300,12 +1300,13 @@ loop_get_status_old(struct loop_device *lo, struct loop_info __user *arg) {
>>  static int
>>  loop_get_status64(struct loop_device *lo, struct loop_info64 __user *arg) {
>>         struct loop_info64 info64;
>> -       int err = 0;
>> +       int err;
>>
>> -       if (!arg)
>> -               err = -EINVAL;
>> -       if (!err)
>> -               err = loop_get_status(lo, &info64);
>> +       if (!arg) {
>> +               mutex_unlock(&lo->lo_ctl_mutex);
>> +               return -EINVAL;
>> +       }
>> +       err = loop_get_status(lo, &info64);
>>         if (!err && copy_to_user(arg, &info64, sizeof(info64)))
>>                 err = -EFAULT;
>>
>>
>> I'll test it and send it up when I get into the office.
> 
> 
> Was this ever submitted? Or some other fix for this?
> 
> The bug is still open, but last happened 289 days ago:
> https://syzkaller.appspot.com/bug?id=608144371e7fc2cb6285b9ed871fb1eb817a61ce
> 
> But it also has 10 duplicates, some of which happened much more recently.
> If a fix was submitted, but Reported-by tag wasn't added this open bug
> can now mask lots of other new bugs.
> 

The commit for this specific patch is bdac616db9bbadb9 ("loop: fix LOOP_GET_STATUS
lock imbalance"). But the root cause of these hung tasks would be fixed by a series
containing commit 1dded9acf6dc9a34 ("Avoid circular locking dependency between
loop_ctl_mutex and bd_mutex") or commit 04906b2f542c2362 ("blockdev: Fix livelocks
on loop device"). We were not aware of these bugs when you marked these reports as
duplicates on 2017/12/12. You can undup them and fix them if you want.

Re: INFO: task hung in lo_ioctl

[Dmitry Vyukov]

On Sun, Jan 20, 2019 at 3:36 AM Tetsuo Handa
<penguin-kernel@i-love.sakura.ne.jp> wrote:
>
> On 2019/01/20 3:56, Dmitry Vyukov wrote:
> >> Seems easy enough to fix without resorting to __mutex_owner() (untested):
> >>
> >>
> >> diff --git a/drivers/block/loop.c b/drivers/block/loop.c
> >> index 264abaaff662..cee258d12a1e 100644
> >> --- a/drivers/block/loop.c
> >> +++ b/drivers/block/loop.c
> >> @@ -1300,12 +1300,13 @@ loop_get_status_old(struct loop_device *lo, struct loop_info __user *arg) {
> >>  static int
> >>  loop_get_status64(struct loop_device *lo, struct loop_info64 __user *arg) {
> >>         struct loop_info64 info64;
> >> -       int err = 0;
> >> +       int err;
> >>
> >> -       if (!arg)
> >> -               err = -EINVAL;
> >> -       if (!err)
> >> -               err = loop_get_status(lo, &info64);
> >> +       if (!arg) {
> >> +               mutex_unlock(&lo->lo_ctl_mutex);
> >> +               return -EINVAL;
> >> +       }
> >> +       err = loop_get_status(lo, &info64);
> >>         if (!err && copy_to_user(arg, &info64, sizeof(info64)))
> >>                 err = -EFAULT;
> >>
> >>
> >> I'll test it and send it up when I get into the office.
> >
> >
> > Was this ever submitted? Or some other fix for this?
> >
> > The bug is still open, but last happened 289 days ago:
> > https://syzkaller.appspot.com/bug?id=608144371e7fc2cb6285b9ed871fb1eb817a61ce
> >
> > But it also has 10 duplicates, some of which happened much more recently.
> > If a fix was submitted, but Reported-by tag wasn't added this open bug
> > can now mask lots of other new bugs.
> >
>
> The commit for this specific patch is bdac616db9bbadb9 ("loop: fix LOOP_GET_STATUS
> lock imbalance"). But the root cause of these hung tasks would be fixed by a series
> containing commit 1dded9acf6dc9a34 ("Avoid circular locking dependency between
> loop_ctl_mutex and bd_mutex") or commit 04906b2f542c2362 ("blockdev: Fix livelocks
> on loop device"). We were not aware of these bugs when you marked these reports as
> duplicates on 2017/12/12. You can undup them and fix them if you want.

OK, let's just do then:

#syz fix: blockdev: Fix livelocks on loop device