Re: [bug report] Format FS failed with ublk device

[Ming Lei <ming.lei@redhat.com>]

Hi Guangwu,

Thanks for the report!

On Thu, Apr 25, 2024 at 09:54:04AM +0800, Guangwu Zhang wrote:
> Hi,
> the format FS command will hung up  with ublk device.
> 
> # ublk --version
> ublksrv 1.1-7-gf01c509
> 
> kerne: 6.9.0-rc4.kasan
> 
> 
> nvme0n1                     259:1    0   1.5T  0 disk
> └─nvme0n1p1                 259:2    0     5G  0 part
> # ublk add -t loop -f /dev/nvme0n1p1
> dev id 0: nr_hw_queues 1 queue_depth 128 block size 4096 dev_capacity 10485760
> max rq size 524288 daemon pid 3227 flags 0x42 state LIVE
> ublkc: 245:0 ublkb: 259:3 owner: 0:0
> queue 0: tid 3228 affinity(0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17
> 18 19 20 21 22 23 24 25 26 27 28 29 30 31 )
> target {"backing_file":"/dev/nvme0n1p1","dev_size":5368709120,"direct_io":1,"name":"loop","type":1}
> 
> # mkfs.xfs -f /dev/ublkb0    << can not finish,  pid 3239
> meta-data=/dev/ublkb0            isize=512    agcount=4, agsize=327680 blks
>          =                       sectsz=4096  attr=2, projid32bit=1
>          =                       crc=1        finobt=1, sparse=1, rmapbt=0
>          =                       reflink=1    bigtime=1 inobtcount=1 nrext64=0
> data     =                       bsize=4096   blocks=1310720, imaxpct=25
>          =                       sunit=0      swidth=0 blks
> naming   =version 2              bsize=4096   ascii-ci=0, ftype=1
> log      =internal log           bsize=4096   blocks=16384, version=2
>          =                       sectsz=4096  sunit=1 blks, lazy-count=1
> realtime =none                   extsz=4096   blocks=0, rtextents=0
> 
> # cat /proc/3239/stack
> [<0>] rq_qos_wait+0x12a/0x1f0
> [<0>] wbt_wait+0x11a/0x240
> [<0>] __rq_qos_throttle+0x49/0x90
> [<0>] blk_mq_submit_bio+0x58c/0x19d0
> [<0>] submit_bio_noacct_nocheck+0x40d/0x780
> [<0>] blk_next_bio+0x41/0x50
> [<0>] __blkdev_issue_zero_pages+0x1ba/0x370
> [<0>] blkdev_issue_zeroout+0x1a7/0x390
> [<0>] blkdev_fallocate+0x264/0x3d0
> [<0>] vfs_fallocate+0x2b0/0xad0
> [<0>] __x64_sys_fallocate+0xb4/0x100
> [<0>] do_syscall_64+0x7b/0x160
> [<0>] entry_SYSCALL_64_after_hwframe+0x76/0x7e
> 
> [  862.171377] INFO: task mkfs.xfs:3239 blocked for more than 122 seconds.
> [  862.178073]       Not tainted 6.9.0-rc4.kasan+ #1
> [  862.182820] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs"
> disables this message.

Looks it might be one blk-wbt issue, and ublk-loop doesn't setup
write_zero_max_bytes and it may take a bit long for __blkdev_issue_zero_pages
to complete, but it shouldn't hang.

Can you collect the following bpftrace by starting it before running mkfs?
And I can't reproduce it in my environment.

#!/usr/bin/bpftrace
kretfunc:vfs_fallocate
{
	printf("vfs_fallocate on %s ret %d (%x %lx %u)\n",
		str(args->file->f_path.dentry->d_name.name),
		retval, args->mode, args->offset, args->len);
}


Thanks,
Ming