* Re: [Kernel Bug] INFO: task hung in show_cons_active
2026-06-09 12:10 ` Greg KH
@ 2026-06-22 13:53 ` Longxing Li
2026-06-22 14:02 ` Greg KH
0 siblings, 1 reply; 4+ messages in thread
From: Longxing Li @ 2026-06-22 13:53 UTC (permalink / raw)
To: Greg KH; +Cc: syzkaller, jirislaby, linux-kernel, linux-serial
Dear Linux kernel developers and maintainers,
Thank you for the feedback. I've added some analysis on this bug, and
also added the original report, which is at the bottom of this email.
Analysis Summary:
The hang involves two tasks blocking on console_sem while holding
console_mutex, and a third task path (unregister_console) that
acquires console_mutex and then blocks on console_sem via
__pr_flush(). The core problem is a lock inversion: __pr_flush()
blocks on console_sem while holding console_list_lock (->
console_mutex).
This can lead to a deadlock when any code path holds console_sem and
subsequently needs console_mutex.
=====================================
Lock Chain Analysis
Three locks are involved in this deadlock scenario:
1. console_mutex (acquired via console_list_lock())
- Protects the console_list linked list
- Used in show_cons_active and unregister_console paths
2. console_sem (binary semaphore)
- Ensures exclusive console output access
- Acquired via console_lock() -> down(&console_sem)
3. tty->termios_rwsem
- Already held by flush_to_ldisc's receive_buf path
=====================================
Task 1 (systemd/1) - currently blocked
show_cons_active() [drivers/tty/tty_io.c:3533]
console_list_lock() [-> mutex_lock(&console_mutex)] [HELD]
...
console_lock() [-> down(&console_sem)] [BLOCKING]
Location: drivers/tty/tty_io.c:3547, 3566
Task 2 (kworker/u4:4/68) - currently blocked
flush_to_ldisc() [drivers/tty/tty_buffer.c:462]
tty->termios_rwsem [HELD]
tty->ldisc_sem [HELD]
con_flush_chars() [drivers/tty/vt/vt.c:3646]
guard(console_lock)() [-> console_lock() -> down(&console_sem)] [BLOCKING]
Location: drivers/tty/vt/vt.c:3653
Critical Path - unregister_console() [trigger point]
register_console() [kernel/printk/printk.c:4060]
console_list_lock()
...
console_sysfs_notify() [called while holding console_list_lock]
unregister_console() [kernel/printk/printk.c:4325]
console_list_lock() [-> mutex_lock(&console_mutex)]
unregister_console_locked() [line 4228]
__pr_flush(console, 1000, true) [kernel/printk/printk.c:4469]
[BLOCKING CALL]
The __pr_flush() function internally (printk.c:4517):
console_lock() [-> down(&console_sem)] [BLOCKS!]
This is the key issue: __pr_flush() BLOCKS on console_sem while called from
within a context that holds console_list_lock (which IS console_mutex).
=====================================
The Problem
unregister_console_locked() calls __pr_flush() while holding console_mutex/
console_list_lock. Inside __pr_flush(), console_lock() will block waiting for
console_sem.
Deadlock condition (ABBA):
A = console_mutex (console_list_lock)
B = console_sem
Path A (show_cons_active): holds A, waits for B
Path B (flush_to_ldisc): holds C, waits for B [C =
termios_rwsem, ldisc_sem]
Path T (some console write): holds B, needs A [if this path exists]
The kernel itself acknowledges this risk in console_unblank()
(printk.c:3458-3461):
/*
* Attempting to trylock the console lock can deadlock
* if another CPU was stopped while modifying the
* semaphore. "Hope and pray" that this is not the
* current situation.
*/
=====================================
__pr_flush() performs a blocking console_lock() (via console_lock() ->
down(&console_sem)) while its caller (unregister_console_locked) holds
console_list_lock (which is console_mutex).
This is a lock ordering violation: if any code path holds console_sem and then
needs console_mutex, we have a deadlock. The syzkaller test conditions create
exactly the extreme timing needed to trigger this scenario.
=====================================
Suggested Fix
1. Move __pr_flush() outside the console_list_lock critical section:
In unregister_console(), perform the flush BEFORE acquiring
console_list_lock(),
with appropriate changes to prevent console re-registration during the wait.
2. Use console_trylock() with retry in __prflush() when called from a context
that already holds console_list_lock (similar to the console_unblank approach
for oops_in_progress).
3. Two-phase __pr_flush():
(a) Release console_list_lock
(b) Wait for console_lock
(c) Re-acquire console_list_lock only when safe
4. Consider whether unregister_console_locked() really needs to call
__pr_flush()
while continuing to hold the list lock - the flush could arguably
be done after
dropping the list lock with careful ordering.
=====================================
Report:
INFO: task systemd:1 blocked for more than 143 seconds.
Not tainted 7.0.6 #1
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
task:systemd state:D stack:21448 pid:1 tgid:1 ppid:0
task_flags:0x400100 flags:0x00080000
Call Trace:
<TASK>
context_switch kernel/sched/core.c:5298 [inline]
__schedule+0x1006/0x5f00 kernel/sched/core.c:6911
__schedule_loop kernel/sched/core.c:6993 [inline]
schedule+0xe7/0x3a0 kernel/sched/core.c:7008
schedule_timeout+0x245/0x280 kernel/time/sleep_timeout.c:75
___down_common kernel/locking/semaphore.c:268 [inline]
__down_common+0x353/0x780 kernel/locking/semaphore.c:293
down+0x74/0xa0 kernel/locking/semaphore.c:100
console_lock+0x5b/0xa0 kernel/printk/printk.c:2896
show_cons_active+0x1f5/0x6e0 drivers/tty/tty_io.c:3566
dev_attr_show+0x56/0xe0 drivers/base/core.c:2421
sysfs_kf_seq_show+0x1f5/0x400 fs/sysfs/file.c:65
seq_read_iter+0x516/0x12e0 fs/seq_file.c:231
kernfs_fop_read_iter+0x46d/0x620 fs/kernfs/file.c:297
new_sync_read fs/read_write.c:493 [inline]
vfs_read+0x865/0xc70 fs/read_write.c:574
ksys_read+0x121/0x240 fs/read_write.c:717
do_syscall_x64 arch/x86/entry/syscall_64.c:63 [inline]
do_syscall_64+0x11b/0xf80 arch/x86/entry/syscall_64.c:94
entry_SYSCALL_64_after_hwframe+0x77/0x7f
RIP: 0033:0x7efece3174ac
RSP: 002b:00007ffe0fbc9010 EFLAGS: 00000246 ORIG_RAX: 0000000000000000
RAX: ffffffffffffffda RBX: 000055850dc62190 RCX: 00007efece3174ac
RDX: 0000000000001000 RSI: 000055850de46180 RDI: 000000000000000d
RBP: 00007efece3f7480 R08: 0000000000000000 R09: 00007efece3fabe0
R10: 0000000000000010 R11: 0000000000000246 R12: 00007efecdec06c8
R13: 00007efece3f6880 R14: 0000000000000d68 R15: 0000000000000d68
</TASK>
INFO: task kworker/u4:4:68 blocked for more than 143 seconds.
Not tainted 7.0.6 #1
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
task:kworker/u4:4 state:D stack:23976 pid:68 tgid:68 ppid:2
task_flags:0x4208060 flags:0x00080000
Workqueue: events_unbound flush_to_ldisc
Call Trace:
<TASK>
context_switch kernel/sched/core.c:5298 [inline]
__schedule+0x1006/0x5f00 kernel/sched/core.c:6911
__schedule_loop kernel/sched/core.c:6993 [inline]
schedule+0xe7/0x3a0 kernel/sched/core.c:7008
schedule_timeout+0x245/0x280 kernel/time/sleep_timeout.c:75
___down_common kernel/locking/semaphore.c:268 [inline]
__down_common+0x353/0x780 kernel/locking/semaphore.c:293
down+0x74/0xa0 kernel/locking/semaphore.c:100
console_lock+0x5b/0xa0 kernel/printk/printk.c:2896
class_console_lock_constructor include/linux/console.h:736 [inline]
con_flush_chars+0x5e/0x80 drivers/tty/vt/vt.c:3653
__receive_buf drivers/tty/n_tty.c:1620 [inline]
n_tty_receive_buf_common+0x535/0x1770 drivers/tty/n_tty.c:1715
tty_ldisc_receive_buf+0xa3/0x1b0 drivers/tty/tty_buffer.c:387
tty_port_default_receive_buf+0x72/0xa0 drivers/tty/tty_port.c:37
receive_buf drivers/tty/tty_buffer.c:445 [inline]
flush_to_ldisc+0x227/0x6c0 drivers/tty/tty_buffer.c:495
process_one_work+0x9de/0x1c60 kernel/workqueue.c:3288
process_scheduled_works kernel/workqueue.c:3371 [inline]
worker_thread+0x693/0xeb0 kernel/workqueue.c:3452
kthread+0x38d/0x4a0 kernel/kthread.c:436
ret_from_fork+0x942/0xe50 arch/x86/kernel/process.c:158
ret_from_fork_asm+0x1a/0x30 arch/x86/entry/entry_64.S:245
</TASK>
Showing all locks held in the system:
4 locks held by systemd/1:
#0: ffff8880207b62f0 (&p->lock){+.+.}-{4:4}, at:
seq_read_iter+0xe7/0x12e0 fs/seq_file.c:183
#1: ffff888052332c88 (&of->mutex){+.+.}-{4:4}, at:
kernfs_seq_start+0x4f/0x2a0 fs/kernfs/file.c:172
#2: ffff88801e8750f8 (kn->active){.+.+}-{0:0}, at:
kernfs_get_active_of fs/kernfs/file.c:80 [inline]
#2: ffff88801e8750f8 (kn->active){.+.+}-{0:0}, at:
kernfs_seq_start+0xbc/0x2a0 fs/kernfs/file.c:173
#3: ffffffff8e5d3548 (console_mutex){+.+.}-{4:4}, at:
show_cons_active+0x80/0x6e0 drivers/tty/tty_io.c:3547
4 locks held by kworker/0:0/9:
1 lock held by khungtaskd/25:
#0: ffffffff8e5e6ce0 (rcu_read_lock){....}-{1:3}, at:
rcu_lock_acquire include/linux/rcupdate.h:312 [inline]
#0: ffffffff8e5e6ce0 (rcu_read_lock){....}-{1:3}, at: rcu_read_lock
include/linux/rcupdate.h:850 [inline]
#0: ffffffff8e5e6ce0 (rcu_read_lock){....}-{1:3}, at:
debug_show_all_locks+0x36/0x1c0 kernel/locking/lockdep.c:6775
5 locks held by kworker/u4:4/68:
#0: ffff88801b894148 ((wq_completion)events_unbound){+.+.}-{0:0}, at:
process_one_work+0x139e/0x1c60 kernel/workqueue.c:3263
#1: ffffc90000bf7d08 ((work_completion)(&buf->work)){+.+.}-{0:0}, at:
process_one_work+0x938/0x1c60 kernel/workqueue.c:3264
#2: ffff8880126cc0b8 (&buf->lock){+.+.}-{4:4}, at:
flush_to_ldisc+0x3a/0x6c0 drivers/tty/tty_buffer.c:467
#3: ffff888000b7f0a0 (&tty->ldisc_sem){++++}-{0:0}, at:
tty_ldisc_ref+0x21/0x90 drivers/tty/tty_ldisc.c:263
#4: ffff888000b7f2e8 (&tty->termios_rwsem){++++}-{4:4}, at:
class_rwsem_read_constructor include/linux/rwsem.h:259 [inline]
#4: ffff888000b7f2e8 (&tty->termios_rwsem){++++}-{4:4}, at:
n_tty_receive_buf_common+0x8c/0x1770 drivers/tty/n_tty.c:1678
1 lock held by in:imklog/9029:
#0: ffff88804fc6b7f8 (&f->f_pos_lock){+.+.}-{4:4}, at:
fdget_pos+0x2a0/0x370 fs/file.c:1261
5 locks held by kworker/0:4/9728:
#0: ffff88801d361d48 ((wq_completion)usb_hub_wq){+.+.}-{0:0}, at:
process_one_work+0x139e/0x1c60 kernel/workqueue.c:3263
#1: ffffc9000d08fd08 ((work_completion)(&hub->events)){+.+.}-{0:0},
at: process_one_work+0x938/0x1c60 kernel/workqueue.c:3264
#2: ffff8880224de1e0 (&dev->mutex){....}-{4:4}, at: device_lock
include/linux/device.h:994 [inline]
#2: ffff8880224de1e0 (&dev->mutex){....}-{4:4}, at:
hub_event+0x1af/0x5020 drivers/usb/core/hub.c:5899
#3: ffff8880527841e0 (&dev->mutex){....}-{4:4}, at: device_lock
include/linux/device.h:994 [inline]
#3: ffff8880527841e0 (&dev->mutex){....}-{4:4}, at:
__device_attach+0x7d/0x4f0 drivers/base/dd.c:1088
#4: ffff88804d6791a8 (&dev->mutex){....}-{4:4}, at: device_lock
include/linux/device.h:994 [inline]
#4: ffff88804d6791a8 (&dev->mutex){....}-{4:4}, at:
__device_attach+0x7d/0x4f0 drivers/base/dd.c:1088
=============================================
NMI backtrace for cpu 0
CPU: 0 UID: 0 PID: 25 Comm: khungtaskd Not tainted 7.0.6 #1 PREEMPT(full)
Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.15.0-1 04/01/2014
Call Trace:
<TASK>
__dump_stack lib/dump_stack.c:94 [inline]
dump_stack_lvl+0x116/0x1b0 lib/dump_stack.c:120
nmi_cpu_backtrace+0x2a0/0x350 lib/nmi_backtrace.c:113
nmi_trigger_cpumask_backtrace+0x29c/0x300 lib/nmi_backtrace.c:62
trigger_all_cpu_backtrace include/linux/nmi.h:161 [inline]
__sys_info lib/sys_info.c:157 [inline]
sys_info+0x133/0x180 lib/sys_info.c:165
check_hung_uninterruptible_tasks kernel/hung_task.c:346 [inline]
watchdog+0xeac/0x11e0 kernel/hung_task.c:515
kthread+0x38d/0x4a0 kernel/kthread.c:436
ret_from_fork+0x942/0xe50 arch/x86/kernel/process.c:158
ret_from_fork_asm+0x1a/0x30 arch/x86/entry/entry_64.S:245
</TASK>
Attachments:
https://drive.google.com/file/d/1Bx2unEf-QntjVi8g6Zw7QNO6OP4cjGO_/view?usp=drive_link
https://drive.google.com/file/d/11hRSaOi-v5Ova5uFzNwCHD2qwF3C-dTV/view?usp=drive_link
https://drive.google.com/file/d/1VsRjuOkgL9Gp3uqRFVnCg9V4KUWLsVYP/view?usp=drive_link
https://drive.google.com/file/d/1P8ALUdJVTdZqwWjlLv11As0b0qJEp1hN/view?usp=drive_link
Best regards,
Longxing Li
Greg KH <gregkh@linuxfoundation.org> 于2026年6月9日周二 20:11写道：
>
> On Tue, Jun 09, 2026 at 07:49:13PM +0800, Longxing Li wrote:
> > Dear Linux kernel developers and maintainers,
> >
> > We would like to report a new kernel bug found by our tool. INFO: task
> > hung in show_cons_active. Details are as follows.
> >
> > Kernel commit: v7.0.6
> > Kernel config: see attachment
> > report: see attachment
> > C repro and Syz repro: see attachment
>
> There are no attachments here, nor any real information.
>
> > We are currently analyzing the root cause and working on a
> > reproducible PoC. We will provide further updates in this thread as
> > soon as we have more information.
>
> Please just send a fix for the issue, that's the best way to get this
> resolved.
>
> thanks,
>
> greg k-h
^ permalink raw reply [flat|nested] 4+ messages in thread