From: Pengfei Xu <pengfei.xu@intel.com>
To: <dchinner@redhat.com>
Cc: <djwong@kernel.org>, <heng.su@intel.com>,
<linux-xfs@vger.kernel.org>, <linux-fsdevel@vger.kernel.org>,
<lkp@intel.com>
Subject: [Syzkaller & bisect] There is "soft lockup in __cleanup_mnt" in v6.4-rc3 kernel
Date: Thu, 25 May 2023 10:59:37 +0800 [thread overview]
Message-ID: <ZG7PGdRED5A68Jyh@xpf.sh.intel.com> (raw)
Hi Dave,
Greeting!
Platform: Alder lake
There is "soft lockup in __cleanup_mnt" in v6.4-rc3 kernel.
Syzkaller analysis repro.report and bisect detailed info: https://github.com/xupengfe/syzkaller_logs/tree/main/230524_140757___cleanup_mnt
Guest machine info: https://github.com/xupengfe/syzkaller_logs/blob/main/230524_140757___cleanup_mnt/machineInfo0
Reproduced code: https://github.com/xupengfe/syzkaller_logs/blob/main/230524_140757___cleanup_mnt/repro.c
Reproduced syscall: https://github.com/xupengfe/syzkaller_logs/blob/main/230524_140757___cleanup_mnt/repro.prog
Bisect info: https://github.com/xupengfe/syzkaller_logs/blob/main/230524_140757___cleanup_mnt/bisect_info.log
Kconfig origin: https://github.com/xupengfe/syzkaller_logs/blob/main/230524_140757___cleanup_mnt/kconfig_origin
Suspected commit is as follow due to 2 skip commits:
# possible first bad commit: [f8f1ed1ab3babad46b25e2dbe8de43b33fe7aaa6] xfs: return a referenced perag from filestreams allocator
# possible first bad commit: [571e259282a43f58b1f70dcbf2add20d8c83a72b] xfs: pass perag to filestreams tracing (skip)
"
fs/xfs/xfs_filestream.c: In function ‘xfs_filestream_pick_ag’:
fs/xfs/xfs_filestream.c:92:4: error: label ‘next_ag’ used but not defined
goto next_ag;
^~~~
make[3]: *** [scripts/Makefile.build:252: fs/xfs/xfs_filestream.o] Error 1
make[2]: *** [scripts/Makefile.build:504: fs/xfs] Error 2
make[1]: *** [scripts/Makefile.build:504: fs] Error 2
make: *** [Makefile:2021: .] Error 2
"
# possible first bad commit: [eb70aa2d8ed9a6fc3525f305226c550524390cd2] xfs: use for_each_perag_wrap in xfs_filestream_pick_ag (skip)
"
fs/xfs/xfs_filestream.c: In function ‘xfs_filestream_pick_ag’:
fs/xfs/xfs_filestream.c:111:4: error: label ‘next_ag’ used but not defined
goto next_ag;
^~~~
make[3]: *** [scripts/Makefile.build:252: fs/xfs/xfs_filestream.o] Error 1
make[2]: *** [scripts/Makefile.build:504: fs/xfs] Error 2
make[1]: *** [scripts/Makefile.build:504: fs] Error 2
make: *** [Makefile:2021: .] Error 2
"
"
[ 29.223473] XFS (loop0): Unmounting Filesystem d408de26-55fb-48ab-a8ab-aacedb20f9dd
[ 29.223942] XFS (loop0): SB summary counter sanity check failed
[ 29.224173] XFS (loop0): Metadata corruption detected at xfs_sb_write_verify+0x7d/0x180, xfs_sb block 0x0
[ 29.224544] XFS (loop0): Unmount and run xfs_repair
[ 29.224731] XFS (loop0): First 128 bytes of corrupted metadata buffer:
[ 29.224979] 00000000: 58 46 53 42 00 00 04 00 00 00 00 00 00 00 80 00 XFSB............
[ 29.225304] 00000010: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
[ 29.225603] 00000020: d4 08 de 26 55 fb 48 ab a8 ab aa ce db 20 f9 dd ...&U.H...... ..
[ 29.225902] 00000030: 00 00 00 00 00 00 40 08 00 00 00 00 00 00 00 20 ......@........
[ 29.226200] 00000040: 00 00 00 00 00 00 00 21 00 00 00 00 00 00 00 22 .......!......."
[ 29.226503] 00000050: 00 00 00 04 00 00 40 00 00 00 00 02 00 00 00 00 ......@.........
[ 29.226802] 00000060: 00 00 04 98 b4 f5 02 00 02 00 00 02 00 00 00 00 ................
[ 29.227101] 00000070: 00 00 00 00 00 00 00 00 0a 09 09 01 0e 00 00 14 ................
[ 29.228273] XFS (loop0): Corruption of in-memory data (0x8) detected at _xfs_buf_ioapply+0x5d8/0x5f0 (fs/xfs/xfs_buf.c:1552). Shutting down filesystem.
[ 29.228788] XFS (loop0): Please unmount the filesystem and rectify the problem(s)
[ 56.322257] watchdog: BUG: soft lockup - CPU#1 stuck for 26s! [repro:529]
[ 56.322608] Modules linked in:
[ 56.322733] irq event stamp: 22632
[ 56.322866] hardirqs last enabled at (22631): [<ffffffff82fad69e>] irqentry_exit+0x3e/0xa0
[ 56.323185] hardirqs last disabled at (22632): [<ffffffff82fab7b3>] sysvec_apic_timer_interrupt+0x13/0xe0
[ 56.323550] softirqs last enabled at (9060): [<ffffffff82fcf8e9>] __do_softirq+0x2d9/0x3c3
[ 56.323865] softirqs last disabled at (8463): [<ffffffff81126714>] irq_exit_rcu+0xc4/0x100
[ 56.324179] CPU: 1 PID: 529 Comm: repro Not tainted 6.4.0-rc3-44c026a73be8+ #1
[ 56.324455] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS rel-1.16.0-0-gd239552ce722-prebuilt.qemu.org 04/01/2014
[ 56.324877] RIP: 0010:write_comp_data+0x0/0x90
[ 56.325056] Code: 85 d2 74 0b 8b 86 c8 1d 00 00 39 f8 0f 94 c0 5d c3 cc cc cc cc 0f 1f 44 00 00 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 <55> 48 89 e5 41 57 49 89 d7 41 56 49 89 fe bf 03 00 00 00 41 55 49
[ 56.325736] RSP: 0018:ffffc90000f5bc60 EFLAGS: 00000246
[ 56.325936] RAX: 0000000000000001 RBX: 0000000000000001 RCX: ffffffff81a138ea
[ 56.326204] RDX: 0000000000000001 RSI: 0000000000000000 RDI: 0000000000000001
[ 56.326475] RBP: ffffc90000f5bc68 R08: 0000000000000000 R09: 000000000000001c
[ 56.326744] R10: 0000000000000001 R11: ffffffff83d64580 R12: ffffffff81ac0c81
[ 56.327011] R13: 0000000000000000 R14: 0000000000000001 R15: ffff8880134bf900
[ 56.327278] FS: 00007f85f5814740(0000) GS:ffff88807dd00000(0000) knlGS:0000000000000000
[ 56.327580] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 56.327798] CR2: 00007fe75831f000 CR3: 000000000deb4004 CR4: 0000000000770ee0
[ 56.328067] PKRU: 55555554
[ 56.328176] Call Trace:
[ 56.328273] <TASK>
[ 56.328359] ? __sanitizer_cov_trace_const_cmp1+0x1e/0x30
[ 56.328571] xfs_perag_grab_tag+0x27a/0x460
[ 56.328745] xfs_icwalk+0x31/0xf0
[ 56.328884] xfs_reclaim_inodes+0xc6/0x140
[ 56.329051] xfs_unmount_flush_inodes+0x63/0x80
[ 56.329235] xfs_unmountfs+0x69/0x1f0
[ 56.329389] xfs_fs_put_super+0x5a/0x120
[ 56.329548] ? __pfx_xfs_fs_put_super+0x10/0x10
[ 56.329730] generic_shutdown_super+0xac/0x240
[ 56.329909] kill_block_super+0x46/0x90
[ 56.330063] deactivate_locked_super+0x52/0xb0
[ 56.330242] deactivate_super+0xb3/0xd0
[ 56.330400] cleanup_mnt+0x15e/0x1e0
[ 56.330553] __cleanup_mnt+0x1f/0x30
[ 56.330704] task_work_run+0xb6/0x120
[ 56.330853] exit_to_user_mode_prepare+0x200/0x210
[ 56.331045] syscall_exit_to_user_mode+0x2d/0x60
[ 56.331229] do_syscall_64+0x4a/0x90
[ 56.331379] entry_SYSCALL_64_after_hwframe+0x72/0xdc
[ 56.331575] RIP: 0033:0x7f85f59407db
[ 56.331718] Code: 96 0c 00 f7 d8 64 89 01 48 83 c8 ff c3 66 90 f3 0f 1e fa 31 f6 e9 05 00 00 00 0f 1f 44 00 00 f3 0f 1e fa b8 a6 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 95 96 0c 00 f7 d8 64 89 01 48
[ 56.332395] RSP: 002b:00007ffd74badbc8 EFLAGS: 00000202 ORIG_RAX: 00000000000000a6
[ 56.332680] RAX: 0000000000000000 RBX: 0000000000000000 RCX: 00007f85f59407db
[ 56.332945] RDX: 0000000000000000 RSI: 000000000000000a RDI: 00007ffd74badc70
[ 56.333216] RBP: 00007ffd74baecb0 R08: 0000000000fb4333 R09: 0000000000000009
[ 56.333486] R10: 0000000000404071 R11: 0000000000000202 R12: 00000000004012c0
[ 56.333752] R13: 00007ffd74baedf0 R14: 0000000000000000 R15: 0000000000000000
[ 56.334026] </TASK>
[ 56.334116] Kernel panic - not syncing: softlockup: hung tasks
"
I hope this time is accurate and helpful.
Thanks!
---
If you don't need the following environment to reproduce the problem or if you
already have one, please ignore the following information.
How to reproduce:
git clone https://gitlab.com/xupengfe/repro_vm_env.git
cd repro_vm_env
tar -xvf repro_vm_env.tar.gz
cd repro_vm_env; ./start3.sh // it needs qemu-system-x86_64 and I used v7.1.0
// start3.sh will load bzImage_2241ab53cbb5cdb08a6b2d4688feb13971058f65 v6.2-rc5 kernel
// You could change the bzImage_xxx as you want
// Maybe you need to remove line "-drive if=pflash,format=raw,readonly=on,file=./OVMF_CODE.fd \" for different qemu version
You could use below command to log in, there is no password for root.
ssh -p 10023 root@localhost
After login vm(virtual machine) successfully, you could transfer reproduced
binary to the vm by below way, and reproduce the problem in vm:
gcc -pthread -o repro repro.c
scp -P 10023 repro root@localhost:/root/
Get the bzImage for target kernel:
Please use target kconfig and copy it to kernel_src/.config
make olddefconfig
make -jx bzImage //x should equal or less than cpu num your pc has
Fill the bzImage file into above start3.sh to load the target kernel in vm.
Tips:
If you already have qemu-system-x86_64, please ignore below info.
If you want to install qemu v7.1.0 version:
git clone https://github.com/qemu/qemu.git
cd qemu
git checkout -f v7.1.0
mkdir build
cd build
yum install -y ninja-build.x86_64
yum -y install libslirp-devel.x86_64
../configure --target-list=x86_64-softmmu --enable-kvm --enable-vnc --enable-gtk --enable-sdl --enable-usb-redir --enable-slirp
make
make install
Thanks!
BR.
next reply other threads:[~2023-05-25 2:58 UTC|newest]
Thread overview: 11+ messages / expand[flat|nested] mbox.gz Atom feed top
2023-05-25 2:59 Pengfei Xu [this message]
2023-05-25 3:51 ` [Syzkaller & bisect] There is "soft lockup in __cleanup_mnt" in v6.4-rc3 kernel Eric Sandeen
2023-05-25 5:44 ` Pengfei Xu
2023-05-25 6:15 ` Dave Chinner
2023-05-25 17:55 ` Theodore Ts'o
2023-05-26 6:43 ` Pengfei Xu
2023-05-26 17:42 ` Dave Hansen
2023-05-26 20:54 ` Theodore Ts'o
2023-05-26 21:20 ` Dave Hansen
2023-05-26 4:55 ` Pengfei Xu
2023-05-25 14:17 ` Eric Sandeen
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=ZG7PGdRED5A68Jyh@xpf.sh.intel.com \
--to=pengfei.xu@intel.com \
--cc=dchinner@redhat.com \
--cc=djwong@kernel.org \
--cc=heng.su@intel.com \
--cc=linux-fsdevel@vger.kernel.org \
--cc=linux-xfs@vger.kernel.org \
--cc=lkp@intel.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).