Re: [BUG] ublk: hung tasks in bdev_open during partition scan (v7.4-rc2)

[Ming Lei <tom.leiming@gmail.com>]

On Tue, Nov 10, 2026 at 01:15:43AM +0800, 铮 wrote:
> Hi Ming and Jens,
> 
> I encountered an `INFO: task hung in bdev_open()` while running
> syzkaller on Linux v7.4-rc2.
> 
> Two tasks, a udev worker and a syzkaller executor, were blocked for
> more than 143 seconds in `bdev_open()` (block/bdev.c:990), waiting on
> the same gendisk `open_mutex` while opening block device nodes via
> `openat()`.
> 
> The lock dump also identifies a ublk partition-scan workqueue
> associated with the same mutex. This suggests that the prolonged wait
> may be related to an unfinished ublk partition scan.
> 
> The issue was observed twice during concurrent fuzzing. A subsequent
> state-primed verification run also reproduced the hung-task condition
> on a freshly booted VM.
> 
> All raw logs, crash reports, reproduction programs, and kernel/fuzzing
> configurations are available at:
> 
> https://github.com/Zheng-zheng-codes/bdev-open-hang-7.4-rc2
> 
> All relative paths below refer to files in this repository.
> 
> **Kernel information:**
> 
> - Kernel version: 7.4.0-rc2-kernsight-kasan #1
> - Base: v7.4-rc2 (`7475ea8b5`)
> - Architecture: x86_64
> - Environment: QEMU Standard PC (i440FX + PIIX), KVM, systemd-based guest
> - Configuration: KASAN enabled, `hung_task_panic=0`
> - Fuzzing framework: syzkaller
> - The full kernel configuration and QEMU/fuzzing settings are included
> in the repository.
> 
> **Hung-task report (verification run):**
> 
> ```text
> INFO: task (udev-worker):9713 blocked for more than 143 seconds.
>       Not tainted 7.4.0-rc2-kernsight-kasan #1
> task:(udev-worker) state:D stack:25208 pid:9713 tgid:9713 ppid:4787
> 
> Call Trace:
>  <TASK>
>  __schedule+0x1425/0x6840 kernel/sched/core.c:7389
>  schedule+0xec/0x2d0 kernel/sched/core.c:7481
>  __mutex_lock_common kernel/locking/mutex.c:726 [inline]
>  __mutex_lock+0xe0c/0x1dd0 kernel/locking/mutex.c:821
>  bdev_open+0x3fa/0xe50 block/bdev.c:990
>  blkdev_open+0x355/0x500 block/fops.c:686
>  do_dentry_open+0x693/0x14f0 fs/open.c:1000
>  vfs_open_consume+0xd1/0x460 fs/open.c:1131
>  do_open fs/namei.c:4851 [inline]
>  path_openat+0x1c1b/0x2240 fs/namei.c:5016
>  do_file_open+0x21b/0x470 fs/namei.c:5045
>  do_sys_openat2+0x13e/0x1d0 fs/open.c:1443
>  __x64_sys_openat+0x144/0x200 fs/open.c:1460
>  entry_SYSCALL_64_after_hwframe+0x77/0x7f
>  </TASK>
> 
> INFO: task syz.2.29:12841 blocked for more than 143 seconds.
> task:syz.2.29 state:D stack:27784 pid:12841 tgid:12818 ppid:10317
> 
> Call Trace:
>  <TASK>
>  __mutex_lock
>  bdev_open        block/bdev.c:990
>  blkdev_open      block/fops.c:686
>  do_dentry_open
>  path_openat
>  __x64_sys_openat
>  </TASK>
> ```
> 
> **Lock observations:**
> 
> In the verification report, the lock dump identifies the ublk
> partition-scan worker:
> 
> ```text
> locks held by kworker/2:2/1815:
> 
> #1: ((work_completion)(&ub->partition_scan_work))
>     at process_one_work
> 
> #2: ff1100017b50c350 (&disk->open_mutex)
>     at ublk_partition_scan_work+0xdb/0x170
>     drivers/block/ublk_drv.c:2478
> ```
> 
> The same mutex address appears in the lock records for the two blocked tasks:
> 
> ```text
> locks held by (udev-worker)/9713:
>     ff1100017b50c350 (&disk->open_mutex)
>     at bdev_open+0x3fa/0xe50
> 
> locks held by syz.2.29/12841:
>     ff1100017b50c350 (&disk->open_mutex)
>     at bdev_open+0x3fa/0xe50
> ```
> 
> A separate fuzzing occurrence shows a similar relationship:
> 
> ```text
> kworker/3:2/1260:
>     ublk_partition_scan_work
>     (&disk->open_mutex)
> 
> (udev-worker)/9505:
>     bdev_open
>     (&disk->open_mutex)
> 
> syz.4.2301/42546:
>     bdev_open
>     (&disk->open_mutex)
> ```
> 
> The repeated association between the blocked openers and
> `ublk_partition_scan_work()` suggests that the partition scan may be
> involved in the prolonged wait.
> 
> However, the current logs do not establish why the partition-scan
> worker fails to complete. In particular, I have not determined whether
> it is waiting for outstanding I/O from the userspace ublk backend or
> encountering another condition.
> 
> **Reproduction:**
> 
> The repository contains a minimized target program together with a
> recorded state-reconstruction sequence.
> 
> The minimized `repro/repro.prog` is not sufficient by itself to
> reliably reproduce the hang. The state-primed reproduction uses the
> recorded sequence and the target program in the same VM.
> 
> The relevant files are:
> 
> - `repro/repro.prog`: minimized target program
> - `repro/repro.priming.prog`: recorded state-reconstruction sequence
> - `repro/repro.kernsight.prog`: combined state-primed reproduction input
> - `repro/repro.README.txt`: detailed reproduction instructions
> 
> The state-primed reproduction can be attempted using:
> 
> ```bash
> ./syz-execprog \
>     -executor=./syz-executor \
>     -procs=1 \
>     -repeat=1 \
>     ./repro.kernsight.prog
> ```
> 
> The guest should be booted using the kernel configuration and QEMU
> settings documented in the repository. The same syzkaller environment
> should be used for the reproduction.
> 
> The verification run recorded a hung-task report, and the repository
> contains the corresponding execution log and report.
> 
> One caveat is that the udev worker appears to have entered the blocked
> state before the final minimized target program was executed.
> Therefore, I do not claim that the minimized target program alone
> causes the hang. The preceding state-reconstruction sequence may be
> responsible for establishing the condition.
> 
> **Observed results:**
> 
> - Two occurrences during concurrent syzkaller fuzzing
> - One recorded successful state-primed verification run
> - Both the udev worker and a syzkaller executor were observed waiting
> in `bdev_open()`
> - The lock dumps associate the same `disk->open_mutex` with the ublk
> partition-scan workqueue
> - The VM remained running with `hung_task_panic=0`
> 
> **Repository contents:**
> 
> Reproduction artifacts:
> 
> - `repro/repro.prog`
> - `repro/repro.kernsight.prog`
> - `repro/repro.priming.prog`
> - `repro/repro.README.txt`
> - `repro/repro.report`
> - `repro/repro.log`
> - `repro/repro.state.json`
> - `repro/repro.context.json`
> - `repro/repro.attempts.json`
> - `repro/repro.outcomes.jsonl`
> 
> Original fuzzing evidence:
> 
> - `logs/log0`, `logs/log1`
> - `logs/report0`, `logs/report1`
> - `logs/machineInfo0`, `logs/machineInfo1`
> - `logs/pre_state0`, `logs/pre_state1`
> - `logs/crash-excerpt.txt`
> 
> Environment:
> 
> - `config-7.4.0-rc2-kernsight-kasan`: full kernel configuration
> - `syzkaller.cfg`: syzkaller and QEMU configuration
> 
> **Related upstream work:**
> 
> I noticed the earlier ublk fixes concerning canceled I/O commands and
> partition-scan hangs, including the change titled:
> 
> "ublk: refuse to go live after an io command was canceled"
> 
> This issue was observed on v7.4-rc2, but I have not established
> whether it involves the same underlying condition or a different
> problem.
> 
> Could you please advise whether this corresponds to a known issue, or
> whether additional diagnostics from the partition-scan worker or the
> ublk I/O queues would be useful?


Thanks for the report!

From the ublk program in your log, the same process does the following:

	syz_ublk_setup_queues()			/* FETCH */
	START_DEV
	syz_open_dev$ublk_bdev()		/* open /dev/ublkbN */
	syz_ublk_process_io()			/* handle io */

After START_DEV, ublk scans partitions from one work function, which
grabs disk->open_mutex and reads partition table from the ublk server.
But the server is opening its own disk before handling any io, so it
waits on the same open_mutex in bdev_open(), and the read can't be
served. That is one deadlock in the test program itself.

That is definitely one *wrong* ublk server implementation, ublk server
can't issue IO from its own frontend(`/dev/ublkbN`).


Thanks,
Ming