[syzbot] [block?] [trace?] INFO: task hung in blk_trace_remove (2)

[syzbot] [block?] [trace?] INFO: task hung in blk_trace_remove (2)

[syzbot]

Hello,

syzbot found the following issue on:

HEAD commit:    7a396820222d Merge tag 'v6.8-rc-part2-smb-client' of git:/..
git tree:       upstream
console output: https://syzkaller.appspot.com/x/log.txt?x=10c2406be80000
kernel config:  https://syzkaller.appspot.com/x/.config?x=4059ab9bf06b6ceb
dashboard link: https://syzkaller.appspot.com/bug?extid=2373f6be3e6de4f92562
compiler:       gcc (Debian 12.2.0-14) 12.2.0, GNU ld (GNU Binutils for Debian) 2.40
syz repro:      https://syzkaller.appspot.com/x/repro.syz?x=14669c6fe80000
C reproducer:   https://syzkaller.appspot.com/x/repro.c?x=12d23ae3e80000

Downloadable assets:
disk image: https://storage.googleapis.com/syzbot-assets/6bbe281de19f/disk-7a396820.raw.xz
vmlinux: https://storage.googleapis.com/syzbot-assets/aad457fc635f/vmlinux-7a396820.xz
kernel image: https://storage.googleapis.com/syzbot-assets/b41601ff5beb/bzImage-7a396820.xz

The issue was bisected to:

commit 0d345996e4cb573f8cc81d49b3ee9a7fd2035bef
Author: Pengfei Xu <pengfei.xu@intel.com>
Date:   Mon Jul 31 03:04:18 2023 +0000

    x86/kernel: increase kcov coverage under arch/x86/kernel folder

bisection log:  https://syzkaller.appspot.com/x/bisect.txt?x=1594cacfe80000
final oops:     https://syzkaller.appspot.com/x/report.txt?x=1794cacfe80000
console output: https://syzkaller.appspot.com/x/log.txt?x=1394cacfe80000

IMPORTANT: if you fix the issue, please add the following tag to the commit:
Reported-by: syzbot+2373f6be3e6de4f92562@syzkaller.appspotmail.com
Fixes: 0d345996e4cb ("x86/kernel: increase kcov coverage under arch/x86/kernel folder")

INFO: task syz-executor208:5137 blocked for more than 143 seconds.
      Not tainted 6.7.0-syzkaller-12991-g7a396820222d #0
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
task:syz-executor208 state:D stack:29312 pid:5137  tgid:5134  ppid:5112   flags:0x00004006
Call Trace:
 <TASK>
 context_switch kernel/sched/core.c:5400 [inline]
 __schedule+0xf12/0x5c00 kernel/sched/core.c:6727
 __schedule_loop kernel/sched/core.c:6802 [inline]
 schedule+0xe9/0x270 kernel/sched/core.c:6817
 schedule_preempt_disabled+0x13/0x20 kernel/sched/core.c:6874
 __mutex_lock_common kernel/locking/mutex.c:684 [inline]
 __mutex_lock+0x5b9/0x9d0 kernel/locking/mutex.c:752
 blk_trace_remove+0x1f/0x40 kernel/trace/blktrace.c:406
 sg_ioctl_common drivers/scsi/sg.c:1126 [inline]
 sg_ioctl+0x9ac/0x2760 drivers/scsi/sg.c:1160
 vfs_ioctl fs/ioctl.c:51 [inline]
 __do_sys_ioctl fs/ioctl.c:871 [inline]
 __se_sys_ioctl fs/ioctl.c:857 [inline]
 __x64_sys_ioctl+0x18f/0x210 fs/ioctl.c:857
 do_syscall_x64 arch/x86/entry/common.c:52 [inline]
 do_syscall_64+0xd3/0x250 arch/x86/entry/common.c:83
 entry_SYSCALL_64_after_hwframe+0x63/0x6b
RIP: 0033:0x7f61c0ac50d9
RSP: 002b:00007f61c0a60168 EFLAGS: 00000246 ORIG_RAX: 0000000000000010
RAX: ffffffffffffffda RBX: 00007f61c0b4c3d8 RCX: 00007f61c0ac50d9
RDX: 0000000000000000 RSI: 0000000000001276 RDI: 0000000000000003
RBP: 00007f61c0b4c3d0 R08: 00007ffdb9ff9147 R09: 0000000000000000
R10: 0000000000000000 R11: 0000000000000246 R12: 00007f61c0b4c3dc
R13: 000000000000006e R14: 00007ffdb9ff9060 R15: 00007ffdb9ff9148
 </TASK>

Showing all locks held in the system:
1 lock held by khungtaskd/29:
 #0: ffffffff8d1acba0 (rcu_read_lock){....}-{1:2}, at: rcu_lock_acquire include/linux/rcupdate.h:298 [inline]
 #0: ffffffff8d1acba0 (rcu_read_lock){....}-{1:2}, at: rcu_read_lock include/linux/rcupdate.h:750 [inline]
 #0: ffffffff8d1acba0 (rcu_read_lock){....}-{1:2}, at: debug_show_all_locks+0x75/0x340 kernel/locking/lockdep.c:6614
2 locks held by getty/4817:
 #0: ffff8880297d30a0 (&tty->ldisc_sem){++++}-{0:0}, at: tty_ldisc_ref_wait+0x24/0x80 drivers/tty/tty_ldisc.c:243
 #1: ffffc90002f062f0 (&ldata->atomic_read_lock){+.+.}-{3:3}, at: n_tty_read+0xfc6/0x1490 drivers/tty/n_tty.c:2201
3 locks held by syz-executor208/5136:
1 lock held by syz-executor208/5137:
 #0: ffff88801e9032b0 (&q->debugfs_mutex){+.+.}-{3:3}, at: blk_trace_remove+0x1f/0x40 kernel/trace/blktrace.c:406
1 lock held by syz-executor208/5140:
 #0: ffff88801e9032b0 (&q->debugfs_mutex){+.+.}-{3:3}, at: blk_trace_setup+0x33/0x60 kernel/trace/blktrace.c:648
1 lock held by syz-executor208/5141:
 #0: ffff88801e9032b0 (&q->debugfs_mutex){+.+.}-{3:3}, at: blk_trace_remove+0x1f/0x40 kernel/trace/blktrace.c:406
1 lock held by syz-executor208/5149:
 #0: ffff88801e9032b0 (&q->debugfs_mutex){+.+.}-{3:3}, at: blk_trace_setup+0x33/0x60 kernel/trace/blktrace.c:648
1 lock held by syz-executor208/5152:
 #0: ffff88801e9032b0 (&q->debugfs_mutex){+.+.}-{3:3}, at: blk_trace_remove+0x1f/0x40 kernel/trace/blktrace.c:406
1 lock held by syz-executor208/5151:
 #0: ffff88801e9032b0 (&q->debugfs_mutex){+.+.}-{3:3}, at: blk_trace_setup+0x33/0x60 kernel/trace/blktrace.c:648
1 lock held by syz-executor208/5153:
 #0: ffff88801e9032b0 (&q->debugfs_mutex){+.+.}-{3:3}, at: blk_trace_remove+0x1f/0x40 kernel/trace/blktrace.c:406
1 lock held by syz-executor208/5155:
 #0: ffff88801e9032b0 (&q->debugfs_mutex){+.+.}-{3:3}, at: blk_trace_setup+0x33/0x60 kernel/trace/blktrace.c:648
1 lock held by syz-executor208/5156:
 #0: ffff88801e9032b0 (&q->debugfs_mutex){+.+.}-{3:3}, at: blk_trace_remove+0x1f/0x40 kernel/trace/blktrace.c:406

=============================================

NMI backtrace for cpu 1
CPU: 1 PID: 29 Comm: khungtaskd Not tainted 6.7.0-syzkaller-12991-g7a396820222d #0
Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 11/17/2023
Call Trace:
 <TASK>
 __dump_stack lib/dump_stack.c:88 [inline]
 dump_stack_lvl+0xd9/0x1b0 lib/dump_stack.c:106
 nmi_cpu_backtrace+0x277/0x390 lib/nmi_backtrace.c:113
 nmi_trigger_cpumask_backtrace+0x299/0x300 lib/nmi_backtrace.c:62
 trigger_all_cpu_backtrace include/linux/nmi.h:160 [inline]
 check_hung_uninterruptible_tasks kernel/hung_task.c:222 [inline]
 watchdog+0xf87/0x1210 kernel/hung_task.c:379
 kthread+0x2c6/0x3a0 kernel/kthread.c:388
 ret_from_fork+0x45/0x80 arch/x86/kernel/process.c:147
 ret_from_fork_asm+0x11/0x20 arch/x86/entry/entry_64.S:242
 </TASK>
Sending NMI from CPU 1 to CPUs 0:
NMI backtrace for cpu 0
CPU: 0 PID: 5136 Comm: syz-executor208 Not tainted 6.7.0-syzkaller-12991-g7a396820222d #0
Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 11/17/2023
RIP: 0010:check_kcov_mode kernel/kcov.c:173 [inline]
RIP: 0010:write_comp_data+0x19/0x80 kernel/kcov.c:236
Code: 8b 80 f0 15 00 00 c3 66 2e 0f 1f 84 00 00 00 00 00 49 89 d2 49 89 f8 49 89 f1 65 48 8b 14 25 80 c2 03 00 65 8b 05 bf a6 7b 7e <a9> 00 01 ff 00 74 0f f6 c4 01 74 59 8b 82 fc 15 00 00 85 c0 74 4f
RSP: 0018:ffffc900045e77e0 EFLAGS: 00000202
RAX: 0000000080000001 RBX: 0000000000000002 RCX: ffffffff813a4c8d
RDX: ffff88806d973b80 RSI: 0000000000000003 RDI: 0000000000000001
RBP: ffffffff8ac9ccc0 R08: 0000000000000001 R09: 0000000000000003
R10: 0000000000000002 R11: 0000000000000002 R12: 0000000000000002
R13: 0000000000000001 R14: 0000000000000003 R15: 0000000000000002
FS:  00007f61c0a606c0(0000) GS:ffff8880b9800000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 000055e6254f9440 CR3: 000000006e15a000 CR4: 00000000003506f0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Call Trace:
 <NMI>
 </NMI>
 <TASK>
 __sanitizer_cov_trace_switch+0x54/0x90 kernel/kcov.c:341
 unwind_next_frame+0x77d/0x2390 arch/x86/kernel/unwind_orc.c:581
 __unwind_start+0x5a4/0x880 arch/x86/kernel/unwind_orc.c:760
 unwind_start arch/x86/include/asm/unwind.h:64 [inline]
 arch_stack_walk+0xaf/0x170 arch/x86/kernel/stacktrace.c:24
 stack_trace_save+0x96/0xd0 kernel/stacktrace.c:122
 save_stack+0x160/0x1f0 mm/page_owner.c:129
 __reset_page_owner+0x51/0x2e0 mm/page_owner.c:150
 reset_page_owner include/linux/page_owner.h:24 [inline]
 free_pages_prepare mm/page_alloc.c:1140 [inline]
 free_unref_page_prepare+0x51f/0xb10 mm/page_alloc.c:2346
 free_unref_page+0x33/0x3c0 mm/page_alloc.c:2486
 relay_destroy_buf+0x121/0x3e0 kernel/relay.c:201
 relay_remove_buf kernel/relay.c:221 [inline]
 kref_put include/linux/kref.h:65 [inline]
 relay_close_buf+0x153/0x1b0 kernel/relay.c:430
 relay_close kernel/relay.c:766 [inline]
 relay_close+0x3a8/0x5d0 kernel/relay.c:752
 blk_trace_free+0x37/0x170 kernel/trace/blktrace.c:316
 blk_trace_cleanup kernel/trace/blktrace.c:384 [inline]
 __blk_trace_remove+0x7f/0x130 kernel/trace/blktrace.c:397
 blk_trace_remove+0x27/0x40 kernel/trace/blktrace.c:407
 sg_ioctl_common drivers/scsi/sg.c:1126 [inline]
 sg_ioctl+0x9ac/0x2760 drivers/scsi/sg.c:1160
 vfs_ioctl fs/ioctl.c:51 [inline]
 __do_sys_ioctl fs/ioctl.c:871 [inline]
 __se_sys_ioctl fs/ioctl.c:857 [inline]
 __x64_sys_ioctl+0x18f/0x210 fs/ioctl.c:857
 do_syscall_x64 arch/x86/entry/common.c:52 [inline]
 do_syscall_64+0xd3/0x250 arch/x86/entry/common.c:83
 entry_SYSCALL_64_after_hwframe+0x63/0x6b
RIP: 0033:0x7f61c0ac50d9
Code: 28 00 00 00 75 05 48 83 c4 28 c3 e8 91 1a 00 00 90 48 89 f8 48 89 f7 48 89 d6 48 89 ca 4d 89 c2 4d 89 c8 4c 8b 4c 24 08 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 c7 c1 b0 ff ff ff f7 d8 64 89 01 48
RSP: 002b:00007f61c0a60168 EFLAGS: 00000246 ORIG_RAX: 0000000000000010
RAX: ffffffffffffffda RBX: 00007f61c0b4c3d8 RCX: 00007f61c0ac50d9
RDX: 0000000000000000 RSI: 0000000000001276 RDI: 0000000000000003
RBP: 00007f61c0b4c3d0 R08: 00007ffdb9ff9147 R09: 0000000000000000
R10: 0000000000000000 R11: 0000000000000246 R12: 00007f61c0b4c3dc
R13: 000000000000006e R14: 00007ffdb9ff9060 R15: 00007ffdb9ff9148
 </TASK>
INFO: NMI handler (nmi_cpu_backtrace_handler) took too long to run: 1.919 msecs


---
This report is generated by a bot. It may contain errors.
See https://goo.gl/tpsmEJ for more information about syzbot.
syzbot engineers can be reached at syzkaller@googlegroups.com.

syzbot will keep track of this issue. See:
https://goo.gl/tpsmEJ#status for how to communicate with syzbot.
For information about bisection process see: https://goo.gl/tpsmEJ#bisection

If the report is already addressed, let syzbot know by replying with:
#syz fix: exact-commit-title

If you want syzbot to run the reproducer, reply with:
#syz test: git://repo/address.git branch-or-commit-hash
If you attach or paste a git patch, syzbot will apply it before testing.

If you want to overwrite report's subsystems, reply with:
#syz set subsystems: new-subsystem
(See the list of subsystem names on the web dashboard)

If the report is a duplicate of another one, reply with:
#syz dup: exact-subject-of-another-report

If you want to undo deduplication, reply with:
#syz undup

[PATCH] trace/blktrace: fix task hung in blk_trace_remove

[Edward Adam Davis]

Delete critical sections that are time-consuming and protected by other mutexes
to avoid this issue.

Reported-and-tested-by: syzbot+2373f6be3e6de4f92562@syzkaller.appspotmail.com
Signed-off-by: Edward Adam Davis <eadavis@qq.com>
---
 kernel/trace/blktrace.c | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/kernel/trace/blktrace.c b/kernel/trace/blktrace.c
index d5d94510afd3..4543be718362 100644
--- a/kernel/trace/blktrace.c
+++ b/kernel/trace/blktrace.c
@@ -313,6 +313,7 @@ static void __blk_add_trace(struct blk_trace *bt, sector_t sector, int bytes,
 
 static void blk_trace_free(struct request_queue *q, struct blk_trace *bt)
 {
+	mutex_unlock(&q->debugfs_mutex);
 	relay_close(bt->rchan);
 
 	/*
@@ -325,6 +326,7 @@ static void blk_trace_free(struct request_queue *q, struct blk_trace *bt)
 	} else {
 		debugfs_remove(bt->dir);
 	}
+	mutex_lock(&q->debugfs_mutex);
 	free_percpu(bt->sequence);
 	free_percpu(bt->msg_data);
 	kfree(bt);
-- 
2.43.0

Re: [PATCH] trace/blktrace: fix task hung in blk_trace_remove

[Jens Axboe]

On 1/31/24 6:28 AM, Edward Adam Davis wrote:
> Delete critical sections that are time-consuming and protected by other mutexes
> to avoid this issue.

What is "this issue"?

-- 
Jens Axboe

Re: [syzbot] [block?] [trace?] INFO: task hung in blk_trace_remove (2)

[syzbot]

syzbot suspects this issue was fixed by commit:

commit ae94b263f5f69c180347e795fbefa051b65aacc3
Author: Dmitry Vyukov <dvyukov@google.com>
Date:   Tue Jun 11 07:50:33 2024 +0000

    x86: Ignore stack unwinding in KCOV

bisection log:  https://syzkaller.appspot.com/x/bisect.txt?x=174bc1a7980000
start commit:   7a396820222d Merge tag 'v6.8-rc-part2-smb-client' of git:/..
git tree:       upstream
kernel config:  https://syzkaller.appspot.com/x/.config?x=4059ab9bf06b6ceb
dashboard link: https://syzkaller.appspot.com/bug?extid=2373f6be3e6de4f92562
syz repro:      https://syzkaller.appspot.com/x/repro.syz?x=14669c6fe80000
C reproducer:   https://syzkaller.appspot.com/x/repro.c?x=12d23ae3e80000

If the result looks correct, please mark the issue as fixed by replying with:

#syz fix: x86: Ignore stack unwinding in KCOV

For information about bisection process see: https://goo.gl/tpsmEJ#bisection