[Syzkaller & bisect] There is task hung in xlog_grant_head_check in v6.3-rc5

[Pengfei Xu <pengfei.xu@intel.com>]

Hi Dave Chinner and xfs experts,

Greeting!

There is task hung in xlog_grant_head_check in v6.3-rc5 kernel.

Platform: x86 platforms

All detailed info: https://github.com/xupengfe/syzkaller_logs/tree/main/230405_094839_xlog_grant_head_check
Syzkaller reproduced code: https://github.com/xupengfe/syzkaller_logs/blob/main/230405_094839_xlog_grant_head_check/repro.c
Syzkaller analysis repro.report: https://github.com/xupengfe/syzkaller_logs/blob/main/230405_094839_xlog_grant_head_check/repro.report
Syzkaller analysis repro.stats: https://github.com/xupengfe/syzkaller_logs/blob/main/230405_094839_xlog_grant_head_check/repro.stats
Reproduced prog repro.prog: https://github.com/xupengfe/syzkaller_logs/blob/main/230405_094839_xlog_grant_head_check/repro.prog
Kconfig: https://github.com/xupengfe/syzkaller_logs/blob/main/230405_094839_xlog_grant_head_check/kconfig_origin
Bisect info: https://github.com/xupengfe/syzkaller_logs/blob/main/230405_094839_xlog_grant_head_check/bisect_info.log

It could be reproduced in maximum 2100s.
Bisected and found bad commit was:
"
fe08cc5044486096bfb5ce9d3db4e915e53281ea
xfs: open code sb verifier feature checks
"
It's just the suspected commit, because reverted above commit on top of v6.3-rc5
kernel then made kernel failed, could not double confirm for the issue.

"
[   24.818100] memfd_create() without MFD_EXEC nor MFD_NOEXEC_SEAL, pid=339 'systemd'
[   28.230533] loop0: detected capacity change from 0 to 65536
[   28.232522] XFS (loop0): Deprecated V4 format (crc=0) will not be supported after September 2030.
[   28.233447] XFS (loop0): Mounting V10 Filesystem d28317a9-9e04-4f2a-be27-e55b4c413ff6
[   28.234235] XFS (loop0): Log size 66 blocks too small, minimum size is 1968 blocks
[   28.234856] XFS (loop0): Log size out of supported range.
[   28.235289] XFS (loop0): Continuing onwards, but if log hangs are experienced then please report this message in the bug report.
[   28.239290] XFS (loop0): Starting recovery (logdev: internal)
[   28.240979] XFS (loop0): Ending recovery (logdev: internal)
[  300.150944] INFO: task repro:541 blocked for more than 147 seconds.
[  300.151523]       Not tainted 6.3.0-rc5-7e364e56293b+ #1
[  300.152102] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[  300.152716] task:repro           state:D stack:0     pid:541   ppid:540    flags:0x00004004
[  300.153373] Call Trace:
[  300.153580]  <TASK>
[  300.153765]  __schedule+0x40a/0xc30
[  300.154078]  schedule+0x5b/0xe0
[  300.154349]  xlog_grant_head_wait+0x53/0x3a0
[  300.154715]  xlog_grant_head_check+0x1a5/0x1c0
[  300.155113]  xfs_log_reserve+0x145/0x380
[  300.155442]  xfs_trans_reserve+0x226/0x270
[  300.155780]  xfs_trans_alloc+0x147/0x470
[  300.156112]  xfs_qm_qino_alloc+0xcf/0x510
[  300.156441]  ? write_comp_data+0x2f/0x90
[  300.156770]  xfs_qm_init_quotainos+0x30a/0x400
[  300.157139]  xfs_qm_init_quotainfo+0x9d/0x4b0
[  300.157499]  ? write_comp_data+0x2f/0x90
[  300.157827]  xfs_qm_mount_quotas+0x40/0x3c0
[  300.158167]  xfs_mountfs+0xc37/0xce0
[  300.158467]  xfs_fs_fill_super+0x7aa/0xdc0
[  300.158817]  get_tree_bdev+0x24b/0x350
[  300.159126]  ? __pfx_xfs_fs_fill_super+0x10/0x10
[  300.159503]  xfs_fs_get_tree+0x25/0x30
[  300.159815]  vfs_get_tree+0x3b/0x140
[  300.160118]  path_mount+0x769/0x10f0
[  300.160415]  ? write_comp_data+0x2f/0x90
[  300.160743]  do_mount+0xaf/0xd0
[  300.161009]  __x64_sys_mount+0x14b/0x160
[  300.161331]  do_syscall_64+0x3b/0x90
[  300.161632]  entry_SYSCALL_64_after_hwframe+0x72/0xdc
[  300.162041] RIP: 0033:0x7fece24223ae
[  300.162333] RSP: 002b:00007fff584561e8 EFLAGS: 00000206 ORIG_RAX: 00000000000000a5
[  300.162937] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007fece24223ae
[  300.163494] RDX: 000000002000ad00 RSI: 000000002000ad40 RDI: 00007fff58456320
[  300.164051] RBP: 00007fff584563b0 R08: 00007fff58456220 R09: 0000000000000000
[  300.164612] R10: 0000000000000003 R11: 0000000000000206 R12: 0000000000401240
[  300.165168] R13: 00007fff584564f0 R14: 0000000000000000 R15: 0000000000000000
[  300.165732]  </TASK>
[  300.165919] 
[  300.165919] Showing all locks held in the system:
[  300.166402] 1 lock held by rcu_tasks_kthre/11:
[  300.166773]  #0: ffffffff83d63450 (rcu_tasks.tasks_gp_mutex){+.+.}-{3:3}, at: rcu_tasks_one_gp+0x31/0x420
[  300.167530] 1 lock held by rcu_tasks_rude_/12:
[  300.167886]  #0: ffffffff83d631d0 (rcu_tasks_rude.tasks_gp_mutex){+.+.}-{3:3}, at: rcu_tasks_one_gp+0x31/0x420
[  300.168683] 1 lock held by rcu_tasks_trace/13:
[  300.169039]  #0: ffffffff83d62f10 (rcu_tasks_trace.tasks_gp_mutex){+.+.}-{3:3}, at: rcu_tasks_one_gp+0x31/0x420
[  300.169839] 1 lock held by khungtaskd/29:
[  300.170160]  #0: ffffffff83d63e60 (rcu_read_lock){....}-{1:2}, at: debug_show_all_locks+0x1b/0x1e0
[  300.170891] 2 locks held by repro/541:
[  300.171194]  #0: ffff88800de780e0 (&type->s_umount_key#47/1){+.+.}-{3:3}, at: alloc_super+0x12b/0x480
[  300.171926]  #1: ffff88800de78638 (sb_internal#2){.+.+}-{0:0}, at: xfs_qm_qino_alloc+0xcf/0x510
[  300.172634] 
[  300.172769] =============================================
"

I hope the info is helpful.

Thanks!

---

If you don't need the following environment to reproduce the problem or if you
already have one, please ignore the following information.

How to reproduce:
git clone https://gitlab.com/xupengfe/repro_vm_env.git
cd repro_vm_env
tar -xvf repro_vm_env.tar.gz
cd repro_vm_env; ./start3.sh  // it needs qemu-system-x86_64 and I used v7.1.0
   // start3.sh will load bzImage_2241ab53cbb5cdb08a6b2d4688feb13971058f65 v6.2-rc5 kernel
   // You could change the bzImage_xxx as you want
You could use below command to log in, there is no password for root.
ssh -p 10023 root@localhost

After login vm(virtual machine) successfully, you could transfer reproduced
binary to the vm by below way, and reproduce the problem in vm:
gcc -pthread -o repro repro.c
scp -P 10023 repro root@localhost:/root/

Get the bzImage for target kernel:
Please use target kconfig and copy it to kernel_src/.config
make olddefconfig
make -jx bzImage           //x should equal or less than cpu num your pc has

Fill the bzImage file into above start3.sh to load the target kernel in vm.


Tips:
If you already have qemu-system-x86_64, please ignore below info.
If you want to install qemu v7.1.0 version:
git clone https://github.com/qemu/qemu.git
cd qemu
git checkout -f v7.1.0
mkdir build
cd build
yum install -y ninja-build.x86_64
../configure --target-list=x86_64-softmmu --enable-kvm --enable-vnc --enable-gtk --enable-sdl
make
make install

Thanks!
BR.