* vhost: linux-next: crash at vhost_dev_cleanup()
@ 2025-07-23 15:04 Breno Leitao
2025-07-23 19:09 ` Michael S. Tsirkin
2025-07-24 7:47 ` Michael S. Tsirkin
0 siblings, 2 replies; 12+ messages in thread
From: Breno Leitao @ 2025-07-23 15:04 UTC (permalink / raw)
To: mst, jasowang, eperezma; +Cc: linux-arm-kernel, kvm
Hello,
I've seen a crash in linux-next for a while on my arm64 server, and
I decided to report.
While running stress-ng on linux-next, I see the crash below.
This is happening in a kernel configure with some debug options (KASAN,
LOCKDEP and KMEMLEAK).
Basically running stress-ng in a loop would crash the host in 15-20
minutes:
# while (true); do stress-ng -r 10 -t 10; done
From the early warning "virt_to_phys used for non-linear address",
I suppose corrupted data is at vq->nheads.
Here is the decoded stack against 9798752 ("Add linux-next specific
files for 20250721")
[ 620.685144] [ T250731] VFIO - User Level meta-driver version: 0.3
[ 622.394448] [ T250254] ------------[ cut here ]------------
[ 622.413492] [ T250254] virt_to_phys used for non-linear address: 000000006e69fe64 (0xcfcecdcccbcac9c8)
[ 622.447771] [ T250254] WARNING: arch/arm64/mm/physaddr.c:15 at __virt_to_phys+0x64/0x90, CPU#57: stress-ng-dev/250254
[ 622.487227] [ T250254] Modules linked in: vhost_vsock(E) vfio_iommu_type1(E) vfio(E) unix_diag(E) sch_fq(E) ghes_edac(E) tls(E) tcp_diag(E) inet_diag(E) act_gact(E) cls_bpf(E) nvidia_cspmu(E) ipmi_ssif(E) coresight_trbe(E) arm_cspmu_module(E) arm_smmuv3_pmu(E) ipmi_devintf(E) coresight_stm(E) coresight_funnel(E) coresight_etm4x(E) coresight_tmc(E) stm_core(E) ipmi_msghandler(E) coresight(E) cppc_cpufreq(E) sch_fq_codel(E) drm(E) backlight(E) drm_panel_orientation_quirks(E) sm3_ce(E) sha3_ce(E) spi_tegra210_quad(E) vhost_net(E) tap(E) tun(E) vhost(E) vhost_iotlb(E) mpls_gso(E) mpls_iptunnel(E) mpls_router(E) fou(E) acpi_power_meter(E) loop(E) efivarfs(E) autofs4(E) [last unloaded: test_bpf(E)]
[ 622.734524] [ T250254] Tainted: [W]=WARN, [E]=UNSIGNED_MODULE, [N]=TEST
[ 622.734525] [ T250254] Hardware name: ...
[ 622.734526] [ T250254] pstate: 63401009 (nZCv daif +PAN -UAO +TCO +DIT +SSBS BTYPE=--)
[ 622.734529] [ T250254] pc : __virt_to_phys (/home/user/Devel/linux-next/arch/arm64/mm/physaddr.c:?)
[ 622.734531] [ T250254] lr : __virt_to_phys (/home/user/Devel/linux-next/arch/arm64/mm/physaddr.c:?)
[ 622.734533] [ T250254] sp : ffff800158e8fc60
[ 622.734534] [ T250254] x29: ffff800158e8fc60 x28: ffff0034a7cc7900 x27: 0000000000000000
[ 622.734537] [ T250254] x26: 0000000000000000 x25: ffff0034a7cc7900 x24: 00000000040e001f
[ 622.734539] [ T250254] x23: ffff0010858afb00 x22: cfcecdcccbcac9c8 x21: ffff0033526a01e0
[ 622.734541] [ T250254] x20: 0000000000008000 x19: ffcecdcccbcac9c8 x18: ffff80008149c8e4
[ 622.734543] [ T250254] x17: 0000000000000001 x16: 0000000000000000 x15: 0000000000000003
[ 622.734545] [ T250254] x14: ffff800082962e78 x13: 0000000000000003 x12: ffff003bc6231630
[ 622.734546] [ T250254] x11: 0000000000000000 x10: 0000000000000000 x9 : ed44a220ae716b00
[ 622.734548] [ T250254] x8 : 0001000000000000 x7 : 0720072007200720 x6 : ffff80008018710c
[ 622.734550] [ T250254] x5 : 0000000000000001 x4 : 00000090ecc72ac0 x3 : 0000000000000000
[ 622.734552] [ T250254] x2 : 0000000000000000 x1 : ffff800081a72bc6 x0 : 000000000000004f
[ 622.734554] [ T250254] Call trace:
[ 622.734555] [ T250254] __virt_to_phys (/home/user/Devel/linux-next/arch/arm64/mm/physaddr.c:?) (P)
[ 622.734557] [ T250254] kfree (/home/user/Devel/linux-next/./include/linux/mm.h:1180 /home/user/Devel/linux-next/mm/slub.c:4871)
[ 622.734562] [ T250254] vhost_dev_cleanup (/home/user/Devel/linux-next/drivers/vhost/vhost.c:506 /home/user/Devel/linux-next/drivers/vhost/vhost.c:542 /home/user/Devel/linux-next/drivers/vhost/vhost.c:1214) vhost
[ 622.734571] [ T250254] vhost_vsock_dev_release (/home/user/Devel/linux-next/drivers/vhost/vsock.c:756) vhost_vsock
[ 622.734575] [ T250254] __fput (/home/user/Devel/linux-next/fs/file_table.c:469)
[ 622.734578] [ T250254] fput_close_sync (/home/user/Devel/linux-next/fs/file_table.c:?)
[ 622.734579] [ T250254] __arm64_sys_close (/home/user/Devel/linux-next/fs/open.c:1589 /home/user/Devel/linux-next/fs/open.c:1572 /home/user/Devel/linux-next/fs/open.c:1572)
[ 622.734584] [ T250254] invoke_syscall (/home/user/Devel/linux-next/arch/arm64/kernel/syscall.c:50)
[ 622.734589] [ T250254] el0_svc_common (/home/user/Devel/linux-next/./include/linux/thread_info.h:135 /home/user/Devel/linux-next/arch/arm64/kernel/syscall.c:140)
[ 622.734591] [ T250254] do_el0_svc (/home/user/Devel/linux-next/arch/arm64/kernel/syscall.c:152)
[ 622.734594] [ T250254] el0_svc (/home/user/Devel/linux-next/arch/arm64/kernel/entry-common.c:169 /home/user/Devel/linux-next/arch/arm64/kernel/entry-common.c:182 /home/user/Devel/linux-next/arch/arm64/kernel/entry-common.c:880)
[ 622.734600] [ T250254] el0t_64_sync_handler (/home/user/Devel/linux-next/arch/arm64/kernel/entry-common.c:958)
[ 622.734603] [ T250254] el0t_64_sync (/home/user/Devel/linux-next/arch/arm64/kernel/entry.S:596)
[ 622.734605] [ T250254] irq event stamp: 0
[ 622.734606] [ T250254] hardirqs last enabled at (0): 0x0
[ 622.734610] [ T250254] hardirqs last disabled at (0): copy_process (/home/user/Devel/linux-next/kernel/fork.c:?)
[ 622.734614] [ T250254] softirqs last enabled at (0): copy_process (/home/user/Devel/linux-next/kernel/fork.c:?)
[ 622.734616] [ T250254] softirqs last disabled at (0): 0x0
[ 622.734618] [ T250254] ---[ end trace 0000000000000000 ]---
[ 622.734697] [ T250254] Unable to handle kernel paging request at virtual address 003ff3b33312f288
[ 622.734700] [ T250254] Mem abort info:
[ 622.734701] [ T250254] ESR = 0x0000000096000004
[ 622.734702] [ T250254] EC = 0x25: DABT (current EL), IL = 32 bits
[ 622.734704] [ T250254] SET = 0, FnV = 0
[ 622.734705] [ T250254] EA = 0, S1PTW = 0
[ 622.734706] [ T250254] FSC = 0x04: level 0 translation fault
[ 622.734708] [ T250254] Data abort info:
[ 622.734709] [ T250254] ISV = 0, ISS = 0x00000004, ISS2 = 0x00000000
[ 622.734711] [ T250254] CM = 0, WnR = 0, TnD = 0, TagAccess = 0
[ 622.734712] [ T250254] GCS = 0, Overlay = 0, DirtyBit = 0, Xs = 0
[ 622.734713] [ T250254] [003ff3b33312f288] address between user and kernel address ranges
[ 622.734715] [ T250254] Internal error: Oops: 0000000096000004 [#1] SMP
[ 622.734718] [ T250254] Modules linked in: vhost_vsock(E) vfio_iommu_type1(E) vfio(E) unix_diag(E) sch_fq(E) ghes_edac(E) tls(E) tcp_diag(E) inet_diag(E) act_gact(E) cls_bpf(E) nvidia_cspmu(E) ipmi_ssif(E) coresight_trbe(E) arm_cspmu_module(E) arm_smmuv3_pmu(E) ipmi_devintf(E) coresight_stm(E) coresight_funnel(E) coresight_etm4x(E) coresight_tmc(E) stm_core(E) ipmi_msghandler(E) coresight(E) cppc_cpufreq(E) sch_fq_codel(E) drm(E) backlight(E) drm_panel_orientation_quirks(E) sm3_ce(E) sha3_ce(E) spi_tegra210_quad(E) vhost_net(E) tap(E) tun(E) vhost(E) vhost_iotlb(E) mpls_gso(E) mpls_iptunnel(E) mpls_router(E) fou(E) acpi_power_meter(E) loop(E) efivarfs(E) autofs4(E) [last unloaded: test_bpf(E)]
[ 622.734740] [ T250254] Tainted: [W]=WARN, [E]=UNSIGNED_MODULE, [N]=TEST
[ 622.734740] [ T250254] Hardware name: ...
[ 622.734741] [ T250254] pstate: 63401009 (nZCv daif +PAN -UAO +TCO +DIT +SSBS BTYPE=--)
[ 622.734742] [ T250254] pc : kfree (/home/user/Devel/linux-next/./include/linux/page-flags.h:284 /home/user/Devel/linux-next/./include/linux/mm.h:1182 /home/user/Devel/linux-next/mm/slub.c:4871)
[ 622.734745] [ T250254] lr : kfree (/home/user/Devel/linux-next/./include/linux/mm.h:1180 /home/user/Devel/linux-next/mm/slub.c:4871)
[ 622.734747] [ T250254] sp : ffff800158e8fc80
[ 622.734748] [ T250254] x29: ffff800158e8fc90 x28: ffff0034a7cc7900 x27: 0000000000000000
[ 622.734749] [ T250254] x26: 0000000000000000 x25: ffff0034a7cc7900 x24: 00000000040e001f
[ 622.734751] [ T250254] x23: ffff0010858afb00 x22: cfcecdcccbcac9c8 x21: ffff0033526a01e0
[ 622.734752] [ T250254] x20: 003ff3b33312f280 x19: ffff80000acd1a20 x18: ffff80008149c8e4
[ 622.734754] [ T250254] x17: 0000000000000001 x16: 0000000000000000 x15: 0000000000000003
[ 622.734755] [ T250254] x14: ffff800082962e78 x13: 0000000000000003 x12: ffff003bc6231630
[ 622.734757] [ T250254] x11: 0000000000000000 x10: 0000000000000000 x9 : ffffffdfc0000000
[ 622.734758] [ T250254] x8 : 003ff3d37312f280 x7 : 0720072007200720 x6 : ffff80008018710c
[ 622.734760] [ T250254] x5 : 0000000000000001 x4 : 00000090ecc72ac0 x3 : 0000000000000000
[ 622.734761] [ T250254] x2 : 0000000000000000 x1 : ffff800081a72bc6 x0 : ffcf4dcccbcac9c8
[ 622.734763] [ T250254] Call trace:
[ 622.734763] [ T250254] kfree (/home/user/Devel/linux-next/./include/linux/page-flags.h:284 /home/user/Devel/linux-next/./include/linux/mm.h:1182 /home/user/Devel/linux-next/mm/slub.c:4871) (P)
[ 622.734766] [ T250254] vhost_dev_cleanup (/home/user/Devel/linux-next/drivers/vhost/vhost.c:506 /home/user/Devel/linux-next/drivers/vhost/vhost.c:542 /home/user/Devel/linux-next/drivers/vhost/vhost.c:1214) vhost
[ 622.734769] [ T250254] vhost_vsock_dev_release (/home/user/Devel/linux-next/drivers/vhost/vsock.c:756) vhost_vsock
[ 622.734771] [ T250254] __fput (/home/user/Devel/linux-next/fs/file_table.c:469)
[ 622.734772] [ T250254] fput_close_sync (/home/user/Devel/linux-next/fs/file_table.c:?)
[ 622.734773] [ T250254] __arm64_sys_close (/home/user/Devel/linux-next/fs/open.c:1589 /home/user/Devel/linux-next/fs/open.c:1572 /home/user/Devel/linux-next/fs/open.c:1572)
[ 622.734776] [ T250254] invoke_syscall (/home/user/Devel/linux-next/arch/arm64/kernel/syscall.c:50)
[ 622.734778] [ T250254] el0_svc_common (/home/user/Devel/linux-next/./include/linux/thread_info.h:135 /home/user/Devel/linux-next/arch/arm64/kernel/syscall.c:140)
[ 622.734781] [ T250254] do_el0_svc (/home/user/Devel/linux-next/arch/arm64/kernel/syscall.c:152)
[ 622.734783] [ T250254] el0_svc (/home/user/Devel/linux-next/arch/arm64/kernel/entry-common.c:169 /home/user/Devel/linux-next/arch/arm64/kernel/entry-common.c:182 /home/user/Devel/linux-next/arch/arm64/kernel/entry-common.c:880)
[ 622.734787] [ T250254] el0t_64_sync_handler (/home/user/Devel/linux-next/arch/arm64/kernel/entry-common.c:958)
[ 622.734790] [ T250254] el0t_64_sync (/home/user/Devel/linux-next/arch/arm64/kernel/entry.S:596)
[ 622.734792] [ T250254] Code: f2dffbe9 927abd08 cb141908 8b090114 (f9400688)
All code
========
0:* e9 fb df f2 08 jmp 0x8f2e000 <-- trapping instruction
5: bd 7a 92 08 19 mov $0x1908927a,%ebp
a: 14 cb adc $0xcb,%al
c: 14 01 adc $0x1,%al
e: 09 8b 88 06 40 f9 or %ecx,-0x6bff978(%rbx)
Code starting with the faulting instruction
===========================================
0: 88 06 mov %al,(%rsi)
2: 40 f9 rex stc
[ 622.734795] [ T250254] SMP: stopping secondary CPUs
[ 622.735089] [ T250254] Starting crashdump kernel...
[ 622.735091] [ T250254] Bye!
^ permalink raw reply [flat|nested] 12+ messages in thread* Re: vhost: linux-next: crash at vhost_dev_cleanup() 2025-07-23 15:04 vhost: linux-next: crash at vhost_dev_cleanup() Breno Leitao @ 2025-07-23 19:09 ` Michael S. Tsirkin 2025-07-24 7:47 ` Michael S. Tsirkin 1 sibling, 0 replies; 12+ messages in thread From: Michael S. Tsirkin @ 2025-07-23 19:09 UTC (permalink / raw) To: Breno Leitao; +Cc: jasowang, eperezma, linux-arm-kernel, kvm On Wed, Jul 23, 2025 at 08:04:42AM -0700, Breno Leitao wrote: > Hello, > > I've seen a crash in linux-next for a while on my arm64 server, and > I decided to report. > > While running stress-ng on linux-next, I see the crash below. > > This is happening in a kernel configure with some debug options (KASAN, > LOCKDEP and KMEMLEAK). Thanks for the report! Any chance of a bisect? Much appreciated. > Basically running stress-ng in a loop would crash the host in 15-20 > minutes: > # while (true); do stress-ng -r 10 -t 10; done > > >From the early warning "virt_to_phys used for non-linear address", > I suppose corrupted data is at vq->nheads. > > Here is the decoded stack against 9798752 ("Add linux-next specific > files for 20250721") > > > [ 620.685144] [ T250731] VFIO - User Level meta-driver version: 0.3 > [ 622.394448] [ T250254] ------------[ cut here ]------------ > [ 622.413492] [ T250254] virt_to_phys used for non-linear address: 000000006e69fe64 (0xcfcecdcccbcac9c8) > [ 622.447771] [ T250254] WARNING: arch/arm64/mm/physaddr.c:15 at __virt_to_phys+0x64/0x90, CPU#57: stress-ng-dev/250254 > [ 622.487227] [ T250254] Modules linked in: vhost_vsock(E) vfio_iommu_type1(E) vfio(E) unix_diag(E) sch_fq(E) ghes_edac(E) tls(E) tcp_diag(E) inet_diag(E) act_gact(E) cls_bpf(E) nvidia_cspmu(E) ipmi_ssif(E) coresight_trbe(E) arm_cspmu_module(E) arm_smmuv3_pmu(E) ipmi_devintf(E) coresight_stm(E) coresight_funnel(E) coresight_etm4x(E) coresight_tmc(E) stm_core(E) ipmi_msghandler(E) coresight(E) cppc_cpufreq(E) sch_fq_codel(E) drm(E) backlight(E) drm_panel_orientation_quirks(E) sm3_ce(E) sha3_ce(E) spi_tegra210_quad(E) vhost_net(E) tap(E) tun(E) vhost(E) vhost_iotlb(E) mpls_gso(E) mpls_iptunnel(E) mpls_router(E) fou(E) acpi_power_meter(E) loop(E) efivarfs(E) autofs4(E) [last unloaded: test_bpf(E)] > [ 622.734524] [ T250254] Tainted: [W]=WARN, [E]=UNSIGNED_MODULE, [N]=TEST > [ 622.734525] [ T250254] Hardware name: ... > [ 622.734526] [ T250254] pstate: 63401009 (nZCv daif +PAN -UAO +TCO +DIT +SSBS BTYPE=--) > [ 622.734529] [ T250254] pc : __virt_to_phys (/home/user/Devel/linux-next/arch/arm64/mm/physaddr.c:?) > [ 622.734531] [ T250254] lr : __virt_to_phys (/home/user/Devel/linux-next/arch/arm64/mm/physaddr.c:?) > [ 622.734533] [ T250254] sp : ffff800158e8fc60 > [ 622.734534] [ T250254] x29: ffff800158e8fc60 x28: ffff0034a7cc7900 x27: 0000000000000000 > [ 622.734537] [ T250254] x26: 0000000000000000 x25: ffff0034a7cc7900 x24: 00000000040e001f > [ 622.734539] [ T250254] x23: ffff0010858afb00 x22: cfcecdcccbcac9c8 x21: ffff0033526a01e0 > [ 622.734541] [ T250254] x20: 0000000000008000 x19: ffcecdcccbcac9c8 x18: ffff80008149c8e4 > [ 622.734543] [ T250254] x17: 0000000000000001 x16: 0000000000000000 x15: 0000000000000003 > [ 622.734545] [ T250254] x14: ffff800082962e78 x13: 0000000000000003 x12: ffff003bc6231630 > [ 622.734546] [ T250254] x11: 0000000000000000 x10: 0000000000000000 x9 : ed44a220ae716b00 > [ 622.734548] [ T250254] x8 : 0001000000000000 x7 : 0720072007200720 x6 : ffff80008018710c > [ 622.734550] [ T250254] x5 : 0000000000000001 x4 : 00000090ecc72ac0 x3 : 0000000000000000 > [ 622.734552] [ T250254] x2 : 0000000000000000 x1 : ffff800081a72bc6 x0 : 000000000000004f > [ 622.734554] [ T250254] Call trace: > [ 622.734555] [ T250254] __virt_to_phys (/home/user/Devel/linux-next/arch/arm64/mm/physaddr.c:?) (P) > [ 622.734557] [ T250254] kfree (/home/user/Devel/linux-next/./include/linux/mm.h:1180 /home/user/Devel/linux-next/mm/slub.c:4871) > [ 622.734562] [ T250254] vhost_dev_cleanup (/home/user/Devel/linux-next/drivers/vhost/vhost.c:506 /home/user/Devel/linux-next/drivers/vhost/vhost.c:542 /home/user/Devel/linux-next/drivers/vhost/vhost.c:1214) vhost > [ 622.734571] [ T250254] vhost_vsock_dev_release (/home/user/Devel/linux-next/drivers/vhost/vsock.c:756) vhost_vsock > [ 622.734575] [ T250254] __fput (/home/user/Devel/linux-next/fs/file_table.c:469) > [ 622.734578] [ T250254] fput_close_sync (/home/user/Devel/linux-next/fs/file_table.c:?) > [ 622.734579] [ T250254] __arm64_sys_close (/home/user/Devel/linux-next/fs/open.c:1589 /home/user/Devel/linux-next/fs/open.c:1572 /home/user/Devel/linux-next/fs/open.c:1572) > [ 622.734584] [ T250254] invoke_syscall (/home/user/Devel/linux-next/arch/arm64/kernel/syscall.c:50) > [ 622.734589] [ T250254] el0_svc_common (/home/user/Devel/linux-next/./include/linux/thread_info.h:135 /home/user/Devel/linux-next/arch/arm64/kernel/syscall.c:140) > [ 622.734591] [ T250254] do_el0_svc (/home/user/Devel/linux-next/arch/arm64/kernel/syscall.c:152) > [ 622.734594] [ T250254] el0_svc (/home/user/Devel/linux-next/arch/arm64/kernel/entry-common.c:169 /home/user/Devel/linux-next/arch/arm64/kernel/entry-common.c:182 /home/user/Devel/linux-next/arch/arm64/kernel/entry-common.c:880) > [ 622.734600] [ T250254] el0t_64_sync_handler (/home/user/Devel/linux-next/arch/arm64/kernel/entry-common.c:958) > [ 622.734603] [ T250254] el0t_64_sync (/home/user/Devel/linux-next/arch/arm64/kernel/entry.S:596) > [ 622.734605] [ T250254] irq event stamp: 0 > [ 622.734606] [ T250254] hardirqs last enabled at (0): 0x0 > [ 622.734610] [ T250254] hardirqs last disabled at (0): copy_process (/home/user/Devel/linux-next/kernel/fork.c:?) > [ 622.734614] [ T250254] softirqs last enabled at (0): copy_process (/home/user/Devel/linux-next/kernel/fork.c:?) > [ 622.734616] [ T250254] softirqs last disabled at (0): 0x0 > [ 622.734618] [ T250254] ---[ end trace 0000000000000000 ]--- > [ 622.734697] [ T250254] Unable to handle kernel paging request at virtual address 003ff3b33312f288 > [ 622.734700] [ T250254] Mem abort info: > [ 622.734701] [ T250254] ESR = 0x0000000096000004 > [ 622.734702] [ T250254] EC = 0x25: DABT (current EL), IL = 32 bits > [ 622.734704] [ T250254] SET = 0, FnV = 0 > [ 622.734705] [ T250254] EA = 0, S1PTW = 0 > [ 622.734706] [ T250254] FSC = 0x04: level 0 translation fault > [ 622.734708] [ T250254] Data abort info: > [ 622.734709] [ T250254] ISV = 0, ISS = 0x00000004, ISS2 = 0x00000000 > [ 622.734711] [ T250254] CM = 0, WnR = 0, TnD = 0, TagAccess = 0 > [ 622.734712] [ T250254] GCS = 0, Overlay = 0, DirtyBit = 0, Xs = 0 > [ 622.734713] [ T250254] [003ff3b33312f288] address between user and kernel address ranges > [ 622.734715] [ T250254] Internal error: Oops: 0000000096000004 [#1] SMP > [ 622.734718] [ T250254] Modules linked in: vhost_vsock(E) vfio_iommu_type1(E) vfio(E) unix_diag(E) sch_fq(E) ghes_edac(E) tls(E) tcp_diag(E) inet_diag(E) act_gact(E) cls_bpf(E) nvidia_cspmu(E) ipmi_ssif(E) coresight_trbe(E) arm_cspmu_module(E) arm_smmuv3_pmu(E) ipmi_devintf(E) coresight_stm(E) coresight_funnel(E) coresight_etm4x(E) coresight_tmc(E) stm_core(E) ipmi_msghandler(E) coresight(E) cppc_cpufreq(E) sch_fq_codel(E) drm(E) backlight(E) drm_panel_orientation_quirks(E) sm3_ce(E) sha3_ce(E) spi_tegra210_quad(E) vhost_net(E) tap(E) tun(E) vhost(E) vhost_iotlb(E) mpls_gso(E) mpls_iptunnel(E) mpls_router(E) fou(E) acpi_power_meter(E) loop(E) efivarfs(E) autofs4(E) [last unloaded: test_bpf(E)] > [ 622.734740] [ T250254] Tainted: [W]=WARN, [E]=UNSIGNED_MODULE, [N]=TEST > [ 622.734740] [ T250254] Hardware name: ... > [ 622.734741] [ T250254] pstate: 63401009 (nZCv daif +PAN -UAO +TCO +DIT +SSBS BTYPE=--) > [ 622.734742] [ T250254] pc : kfree (/home/user/Devel/linux-next/./include/linux/page-flags.h:284 /home/user/Devel/linux-next/./include/linux/mm.h:1182 /home/user/Devel/linux-next/mm/slub.c:4871) > [ 622.734745] [ T250254] lr : kfree (/home/user/Devel/linux-next/./include/linux/mm.h:1180 /home/user/Devel/linux-next/mm/slub.c:4871) > [ 622.734747] [ T250254] sp : ffff800158e8fc80 > [ 622.734748] [ T250254] x29: ffff800158e8fc90 x28: ffff0034a7cc7900 x27: 0000000000000000 > [ 622.734749] [ T250254] x26: 0000000000000000 x25: ffff0034a7cc7900 x24: 00000000040e001f > [ 622.734751] [ T250254] x23: ffff0010858afb00 x22: cfcecdcccbcac9c8 x21: ffff0033526a01e0 > [ 622.734752] [ T250254] x20: 003ff3b33312f280 x19: ffff80000acd1a20 x18: ffff80008149c8e4 > [ 622.734754] [ T250254] x17: 0000000000000001 x16: 0000000000000000 x15: 0000000000000003 > [ 622.734755] [ T250254] x14: ffff800082962e78 x13: 0000000000000003 x12: ffff003bc6231630 > [ 622.734757] [ T250254] x11: 0000000000000000 x10: 0000000000000000 x9 : ffffffdfc0000000 > [ 622.734758] [ T250254] x8 : 003ff3d37312f280 x7 : 0720072007200720 x6 : ffff80008018710c > [ 622.734760] [ T250254] x5 : 0000000000000001 x4 : 00000090ecc72ac0 x3 : 0000000000000000 > [ 622.734761] [ T250254] x2 : 0000000000000000 x1 : ffff800081a72bc6 x0 : ffcf4dcccbcac9c8 > [ 622.734763] [ T250254] Call trace: > [ 622.734763] [ T250254] kfree (/home/user/Devel/linux-next/./include/linux/page-flags.h:284 /home/user/Devel/linux-next/./include/linux/mm.h:1182 /home/user/Devel/linux-next/mm/slub.c:4871) (P) > [ 622.734766] [ T250254] vhost_dev_cleanup (/home/user/Devel/linux-next/drivers/vhost/vhost.c:506 /home/user/Devel/linux-next/drivers/vhost/vhost.c:542 /home/user/Devel/linux-next/drivers/vhost/vhost.c:1214) vhost > [ 622.734769] [ T250254] vhost_vsock_dev_release (/home/user/Devel/linux-next/drivers/vhost/vsock.c:756) vhost_vsock > [ 622.734771] [ T250254] __fput (/home/user/Devel/linux-next/fs/file_table.c:469) > [ 622.734772] [ T250254] fput_close_sync (/home/user/Devel/linux-next/fs/file_table.c:?) > [ 622.734773] [ T250254] __arm64_sys_close (/home/user/Devel/linux-next/fs/open.c:1589 /home/user/Devel/linux-next/fs/open.c:1572 /home/user/Devel/linux-next/fs/open.c:1572) > [ 622.734776] [ T250254] invoke_syscall (/home/user/Devel/linux-next/arch/arm64/kernel/syscall.c:50) > [ 622.734778] [ T250254] el0_svc_common (/home/user/Devel/linux-next/./include/linux/thread_info.h:135 /home/user/Devel/linux-next/arch/arm64/kernel/syscall.c:140) > [ 622.734781] [ T250254] do_el0_svc (/home/user/Devel/linux-next/arch/arm64/kernel/syscall.c:152) > [ 622.734783] [ T250254] el0_svc (/home/user/Devel/linux-next/arch/arm64/kernel/entry-common.c:169 /home/user/Devel/linux-next/arch/arm64/kernel/entry-common.c:182 /home/user/Devel/linux-next/arch/arm64/kernel/entry-common.c:880) > [ 622.734787] [ T250254] el0t_64_sync_handler (/home/user/Devel/linux-next/arch/arm64/kernel/entry-common.c:958) > [ 622.734790] [ T250254] el0t_64_sync (/home/user/Devel/linux-next/arch/arm64/kernel/entry.S:596) > [ 622.734792] [ T250254] Code: f2dffbe9 927abd08 cb141908 8b090114 (f9400688) > All code > ======== > 0:* e9 fb df f2 08 jmp 0x8f2e000 <-- trapping instruction > 5: bd 7a 92 08 19 mov $0x1908927a,%ebp > a: 14 cb adc $0xcb,%al > c: 14 01 adc $0x1,%al > e: 09 8b 88 06 40 f9 or %ecx,-0x6bff978(%rbx) > > Code starting with the faulting instruction > =========================================== > 0: 88 06 mov %al,(%rsi) > 2: 40 f9 rex stc > [ 622.734795] [ T250254] SMP: stopping secondary CPUs > [ 622.735089] [ T250254] Starting crashdump kernel... > [ 622.735091] [ T250254] Bye! ^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: vhost: linux-next: crash at vhost_dev_cleanup() 2025-07-23 15:04 vhost: linux-next: crash at vhost_dev_cleanup() Breno Leitao 2025-07-23 19:09 ` Michael S. Tsirkin @ 2025-07-24 7:47 ` Michael S. Tsirkin 2025-07-24 8:14 ` Stefano Garzarella 1 sibling, 1 reply; 12+ messages in thread From: Michael S. Tsirkin @ 2025-07-24 7:47 UTC (permalink / raw) To: Breno Leitao Cc: jasowang, eperezma, linux-arm-kernel, kvm, Stefan Hajnoczi, Stefano Garzarella, netdev On Wed, Jul 23, 2025 at 08:04:42AM -0700, Breno Leitao wrote: > Hello, > > I've seen a crash in linux-next for a while on my arm64 server, and > I decided to report. > > While running stress-ng on linux-next, I see the crash below. > > This is happening in a kernel configure with some debug options (KASAN, > LOCKDEP and KMEMLEAK). > > Basically running stress-ng in a loop would crash the host in 15-20 > minutes: > # while (true); do stress-ng -r 10 -t 10; done > > >From the early warning "virt_to_phys used for non-linear address", > I suppose corrupted data is at vq->nheads. > > Here is the decoded stack against 9798752 ("Add linux-next specific > files for 20250721") > > > [ 620.685144] [ T250731] VFIO - User Level meta-driver version: 0.3 > [ 622.394448] [ T250254] ------------[ cut here ]------------ > [ 622.413492] [ T250254] virt_to_phys used for non-linear address: 000000006e69fe64 (0xcfcecdcccbcac9c8) > [ 622.447771] [ T250254] WARNING: arch/arm64/mm/physaddr.c:15 at __virt_to_phys+0x64/0x90, CPU#57: stress-ng-dev/250254 > [ 622.487227] [ T250254] Modules linked in: vhost_vsock(E) vfio_iommu_type1(E) vfio(E) unix_diag(E) sch_fq(E) ghes_edac(E) tls(E) tcp_diag(E) inet_diag(E) act_gact(E) cls_bpf(E) nvidia_cspmu(E) ipmi_ssif(E) coresight_trbe(E) arm_cspmu_module(E) arm_smmuv3_pmu(E) ipmi_devintf(E) coresight_stm(E) coresight_funnel(E) coresight_etm4x(E) coresight_tmc(E) stm_core(E) ipmi_msghandler(E) coresight(E) cppc_cpufreq(E) sch_fq_codel(E) drm(E) backlight(E) drm_panel_orientation_quirks(E) sm3_ce(E) sha3_ce(E) spi_tegra210_quad(E) vhost_net(E) tap(E) tun(E) vhost(E) vhost_iotlb(E) mpls_gso(E) mpls_iptunnel(E) mpls_router(E) fou(E) acpi_power_meter(E) loop(E) efivarfs(E) autofs4(E) [last unloaded: test_bpf(E)] > [ 622.734524] [ T250254] Tainted: [W]=WARN, [E]=UNSIGNED_MODULE, [N]=TEST > [ 622.734525] [ T250254] Hardware name: ... > [ 622.734526] [ T250254] pstate: 63401009 (nZCv daif +PAN -UAO +TCO +DIT +SSBS BTYPE=--) > [ 622.734529] [ T250254] pc : __virt_to_phys (/home/user/Devel/linux-next/arch/arm64/mm/physaddr.c:?) > [ 622.734531] [ T250254] lr : __virt_to_phys (/home/user/Devel/linux-next/arch/arm64/mm/physaddr.c:?) > [ 622.734533] [ T250254] sp : ffff800158e8fc60 > [ 622.734534] [ T250254] x29: ffff800158e8fc60 x28: ffff0034a7cc7900 x27: 0000000000000000 > [ 622.734537] [ T250254] x26: 0000000000000000 x25: ffff0034a7cc7900 x24: 00000000040e001f > [ 622.734539] [ T250254] x23: ffff0010858afb00 x22: cfcecdcccbcac9c8 x21: ffff0033526a01e0 > [ 622.734541] [ T250254] x20: 0000000000008000 x19: ffcecdcccbcac9c8 x18: ffff80008149c8e4 > [ 622.734543] [ T250254] x17: 0000000000000001 x16: 0000000000000000 x15: 0000000000000003 > [ 622.734545] [ T250254] x14: ffff800082962e78 x13: 0000000000000003 x12: ffff003bc6231630 > [ 622.734546] [ T250254] x11: 0000000000000000 x10: 0000000000000000 x9 : ed44a220ae716b00 > [ 622.734548] [ T250254] x8 : 0001000000000000 x7 : 0720072007200720 x6 : ffff80008018710c > [ 622.734550] [ T250254] x5 : 0000000000000001 x4 : 00000090ecc72ac0 x3 : 0000000000000000 > [ 622.734552] [ T250254] x2 : 0000000000000000 x1 : ffff800081a72bc6 x0 : 000000000000004f > [ 622.734554] [ T250254] Call trace: > [ 622.734555] [ T250254] __virt_to_phys (/home/user/Devel/linux-next/arch/arm64/mm/physaddr.c:?) (P) > [ 622.734557] [ T250254] kfree (/home/user/Devel/linux-next/./include/linux/mm.h:1180 /home/user/Devel/linux-next/mm/slub.c:4871) > [ 622.734562] [ T250254] vhost_dev_cleanup (/home/user/Devel/linux-next/drivers/vhost/vhost.c:506 /home/user/Devel/linux-next/drivers/vhost/vhost.c:542 /home/user/Devel/linux-next/drivers/vhost/vhost.c:1214) vhost > [ 622.734571] [ T250254] vhost_vsock_dev_release (/home/user/Devel/linux-next/drivers/vhost/vsock.c:756) vhost_vsock Cc more vsock maintainers. > [ 622.734575] [ T250254] __fput (/home/user/Devel/linux-next/fs/file_table.c:469) > [ 622.734578] [ T250254] fput_close_sync (/home/user/Devel/linux-next/fs/file_table.c:?) > [ 622.734579] [ T250254] __arm64_sys_close (/home/user/Devel/linux-next/fs/open.c:1589 /home/user/Devel/linux-next/fs/open.c:1572 /home/user/Devel/linux-next/fs/open.c:1572) > [ 622.734584] [ T250254] invoke_syscall (/home/user/Devel/linux-next/arch/arm64/kernel/syscall.c:50) > [ 622.734589] [ T250254] el0_svc_common (/home/user/Devel/linux-next/./include/linux/thread_info.h:135 /home/user/Devel/linux-next/arch/arm64/kernel/syscall.c:140) > [ 622.734591] [ T250254] do_el0_svc (/home/user/Devel/linux-next/arch/arm64/kernel/syscall.c:152) > [ 622.734594] [ T250254] el0_svc (/home/user/Devel/linux-next/arch/arm64/kernel/entry-common.c:169 /home/user/Devel/linux-next/arch/arm64/kernel/entry-common.c:182 /home/user/Devel/linux-next/arch/arm64/kernel/entry-common.c:880) > [ 622.734600] [ T250254] el0t_64_sync_handler (/home/user/Devel/linux-next/arch/arm64/kernel/entry-common.c:958) > [ 622.734603] [ T250254] el0t_64_sync (/home/user/Devel/linux-next/arch/arm64/kernel/entry.S:596) > [ 622.734605] [ T250254] irq event stamp: 0 > [ 622.734606] [ T250254] hardirqs last enabled at (0): 0x0 > [ 622.734610] [ T250254] hardirqs last disabled at (0): copy_process (/home/user/Devel/linux-next/kernel/fork.c:?) > [ 622.734614] [ T250254] softirqs last enabled at (0): copy_process (/home/user/Devel/linux-next/kernel/fork.c:?) > [ 622.734616] [ T250254] softirqs last disabled at (0): 0x0 > [ 622.734618] [ T250254] ---[ end trace 0000000000000000 ]--- > [ 622.734697] [ T250254] Unable to handle kernel paging request at virtual address 003ff3b33312f288 > [ 622.734700] [ T250254] Mem abort info: > [ 622.734701] [ T250254] ESR = 0x0000000096000004 > [ 622.734702] [ T250254] EC = 0x25: DABT (current EL), IL = 32 bits > [ 622.734704] [ T250254] SET = 0, FnV = 0 > [ 622.734705] [ T250254] EA = 0, S1PTW = 0 > [ 622.734706] [ T250254] FSC = 0x04: level 0 translation fault > [ 622.734708] [ T250254] Data abort info: > [ 622.734709] [ T250254] ISV = 0, ISS = 0x00000004, ISS2 = 0x00000000 > [ 622.734711] [ T250254] CM = 0, WnR = 0, TnD = 0, TagAccess = 0 > [ 622.734712] [ T250254] GCS = 0, Overlay = 0, DirtyBit = 0, Xs = 0 > [ 622.734713] [ T250254] [003ff3b33312f288] address between user and kernel address ranges > [ 622.734715] [ T250254] Internal error: Oops: 0000000096000004 [#1] SMP > [ 622.734718] [ T250254] Modules linked in: vhost_vsock(E) vfio_iommu_type1(E) vfio(E) unix_diag(E) sch_fq(E) ghes_edac(E) tls(E) tcp_diag(E) inet_diag(E) act_gact(E) cls_bpf(E) nvidia_cspmu(E) ipmi_ssif(E) coresight_trbe(E) arm_cspmu_module(E) arm_smmuv3_pmu(E) ipmi_devintf(E) coresight_stm(E) coresight_funnel(E) coresight_etm4x(E) coresight_tmc(E) stm_core(E) ipmi_msghandler(E) coresight(E) cppc_cpufreq(E) sch_fq_codel(E) drm(E) backlight(E) drm_panel_orientation_quirks(E) sm3_ce(E) sha3_ce(E) spi_tegra210_quad(E) vhost_net(E) tap(E) tun(E) vhost(E) vhost_iotlb(E) mpls_gso(E) mpls_iptunnel(E) mpls_router(E) fou(E) acpi_power_meter(E) loop(E) efivarfs(E) autofs4(E) [last unloaded: test_bpf(E)] > [ 622.734740] [ T250254] Tainted: [W]=WARN, [E]=UNSIGNED_MODULE, [N]=TEST > [ 622.734740] [ T250254] Hardware name: ... > [ 622.734741] [ T250254] pstate: 63401009 (nZCv daif +PAN -UAO +TCO +DIT +SSBS BTYPE=--) > [ 622.734742] [ T250254] pc : kfree (/home/user/Devel/linux-next/./include/linux/page-flags.h:284 /home/user/Devel/linux-next/./include/linux/mm.h:1182 /home/user/Devel/linux-next/mm/slub.c:4871) > [ 622.734745] [ T250254] lr : kfree (/home/user/Devel/linux-next/./include/linux/mm.h:1180 /home/user/Devel/linux-next/mm/slub.c:4871) > [ 622.734747] [ T250254] sp : ffff800158e8fc80 > [ 622.734748] [ T250254] x29: ffff800158e8fc90 x28: ffff0034a7cc7900 x27: 0000000000000000 > [ 622.734749] [ T250254] x26: 0000000000000000 x25: ffff0034a7cc7900 x24: 00000000040e001f > [ 622.734751] [ T250254] x23: ffff0010858afb00 x22: cfcecdcccbcac9c8 x21: ffff0033526a01e0 > [ 622.734752] [ T250254] x20: 003ff3b33312f280 x19: ffff80000acd1a20 x18: ffff80008149c8e4 > [ 622.734754] [ T250254] x17: 0000000000000001 x16: 0000000000000000 x15: 0000000000000003 > [ 622.734755] [ T250254] x14: ffff800082962e78 x13: 0000000000000003 x12: ffff003bc6231630 > [ 622.734757] [ T250254] x11: 0000000000000000 x10: 0000000000000000 x9 : ffffffdfc0000000 > [ 622.734758] [ T250254] x8 : 003ff3d37312f280 x7 : 0720072007200720 x6 : ffff80008018710c > [ 622.734760] [ T250254] x5 : 0000000000000001 x4 : 00000090ecc72ac0 x3 : 0000000000000000 > [ 622.734761] [ T250254] x2 : 0000000000000000 x1 : ffff800081a72bc6 x0 : ffcf4dcccbcac9c8 > [ 622.734763] [ T250254] Call trace: > [ 622.734763] [ T250254] kfree (/home/user/Devel/linux-next/./include/linux/page-flags.h:284 /home/user/Devel/linux-next/./include/linux/mm.h:1182 /home/user/Devel/linux-next/mm/slub.c:4871) (P) > [ 622.734766] [ T250254] vhost_dev_cleanup (/home/user/Devel/linux-next/drivers/vhost/vhost.c:506 /home/user/Devel/linux-next/drivers/vhost/vhost.c:542 /home/user/Devel/linux-next/drivers/vhost/vhost.c:1214) vhost > [ 622.734769] [ T250254] vhost_vsock_dev_release (/home/user/Devel/linux-next/drivers/vhost/vsock.c:756) vhost_vsock > [ 622.734771] [ T250254] __fput (/home/user/Devel/linux-next/fs/file_table.c:469) > [ 622.734772] [ T250254] fput_close_sync (/home/user/Devel/linux-next/fs/file_table.c:?) > [ 622.734773] [ T250254] __arm64_sys_close (/home/user/Devel/linux-next/fs/open.c:1589 /home/user/Devel/linux-next/fs/open.c:1572 /home/user/Devel/linux-next/fs/open.c:1572) > [ 622.734776] [ T250254] invoke_syscall (/home/user/Devel/linux-next/arch/arm64/kernel/syscall.c:50) > [ 622.734778] [ T250254] el0_svc_common (/home/user/Devel/linux-next/./include/linux/thread_info.h:135 /home/user/Devel/linux-next/arch/arm64/kernel/syscall.c:140) > [ 622.734781] [ T250254] do_el0_svc (/home/user/Devel/linux-next/arch/arm64/kernel/syscall.c:152) > [ 622.734783] [ T250254] el0_svc (/home/user/Devel/linux-next/arch/arm64/kernel/entry-common.c:169 /home/user/Devel/linux-next/arch/arm64/kernel/entry-common.c:182 /home/user/Devel/linux-next/arch/arm64/kernel/entry-common.c:880) > [ 622.734787] [ T250254] el0t_64_sync_handler (/home/user/Devel/linux-next/arch/arm64/kernel/entry-common.c:958) > [ 622.734790] [ T250254] el0t_64_sync (/home/user/Devel/linux-next/arch/arm64/kernel/entry.S:596) > [ 622.734792] [ T250254] Code: f2dffbe9 927abd08 cb141908 8b090114 (f9400688) > All code > ======== > 0:* e9 fb df f2 08 jmp 0x8f2e000 <-- trapping instruction > 5: bd 7a 92 08 19 mov $0x1908927a,%ebp > a: 14 cb adc $0xcb,%al > c: 14 01 adc $0x1,%al > e: 09 8b 88 06 40 f9 or %ecx,-0x6bff978(%rbx) > > Code starting with the faulting instruction > =========================================== > 0: 88 06 mov %al,(%rsi) > 2: 40 f9 rex stc > [ 622.734795] [ T250254] SMP: stopping secondary CPUs > [ 622.735089] [ T250254] Starting crashdump kernel... > [ 622.735091] [ T250254] Bye! ^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: vhost: linux-next: crash at vhost_dev_cleanup() 2025-07-24 7:47 ` Michael S. Tsirkin @ 2025-07-24 8:14 ` Stefano Garzarella 2025-07-24 8:22 ` Michael S. Tsirkin 0 siblings, 1 reply; 12+ messages in thread From: Stefano Garzarella @ 2025-07-24 8:14 UTC (permalink / raw) To: Michael S. Tsirkin, Will Deacon Cc: Breno Leitao, jasowang, eperezma, linux-arm-kernel, kvm, Stefan Hajnoczi, netdev CCing Will On Thu, 24 Jul 2025 at 09:48, Michael S. Tsirkin <mst@redhat.com> wrote: > > On Wed, Jul 23, 2025 at 08:04:42AM -0700, Breno Leitao wrote: > > Hello, > > > > I've seen a crash in linux-next for a while on my arm64 server, and > > I decided to report. > > > > While running stress-ng on linux-next, I see the crash below. > > > > This is happening in a kernel configure with some debug options (KASAN, > > LOCKDEP and KMEMLEAK). > > > > Basically running stress-ng in a loop would crash the host in 15-20 > > minutes: > > # while (true); do stress-ng -r 10 -t 10; done > > > > >From the early warning "virt_to_phys used for non-linear address", mmm, we recently added nonlinear SKBs support in vhost-vsock [1], @Will can this issue be related? I checked next-20250721 tag and I confirm that contains those changes. [1] https://lore.kernel.org/virtualization/20250717090116.11987-1-will@kernel.org/ Thanks, Stefano > > I suppose corrupted data is at vq->nheads. > > > > Here is the decoded stack against 9798752 ("Add linux-next specific > > files for 20250721") > > > > > > [ 620.685144] [ T250731] VFIO - User Level meta-driver version: 0.3 > > [ 622.394448] [ T250254] ------------[ cut here ]------------ > > [ 622.413492] [ T250254] virt_to_phys used for non-linear address: 000000006e69fe64 (0xcfcecdcccbcac9c8) > > [ 622.447771] [ T250254] WARNING: arch/arm64/mm/physaddr.c:15 at __virt_to_phys+0x64/0x90, CPU#57: stress-ng-dev/250254 > > [ 622.487227] [ T250254] Modules linked in: vhost_vsock(E) vfio_iommu_type1(E) vfio(E) unix_diag(E) sch_fq(E) ghes_edac(E) tls(E) tcp_diag(E) inet_diag(E) act_gact(E) cls_bpf(E) nvidia_cspmu(E) ipmi_ssif(E) coresight_trbe(E) arm_cspmu_module(E) arm_smmuv3_pmu(E) ipmi_devintf(E) coresight_stm(E) coresight_funnel(E) coresight_etm4x(E) coresight_tmc(E) stm_core(E) ipmi_msghandler(E) coresight(E) cppc_cpufreq(E) sch_fq_codel(E) drm(E) backlight(E) drm_panel_orientation_quirks(E) sm3_ce(E) sha3_ce(E) spi_tegra210_quad(E) vhost_net(E) tap(E) tun(E) vhost(E) vhost_iotlb(E) mpls_gso(E) mpls_iptunnel(E) mpls_router(E) fou(E) acpi_power_meter(E) loop(E) efivarfs(E) autofs4(E) [last unloaded: test_bpf(E)] > > [ 622.734524] [ T250254] Tainted: [W]=WARN, [E]=UNSIGNED_MODULE, [N]=TEST > > [ 622.734525] [ T250254] Hardware name: ... > > [ 622.734526] [ T250254] pstate: 63401009 (nZCv daif +PAN -UAO +TCO +DIT +SSBS BTYPE=--) > > [ 622.734529] [ T250254] pc : __virt_to_phys (/home/user/Devel/linux-next/arch/arm64/mm/physaddr.c:?) > > [ 622.734531] [ T250254] lr : __virt_to_phys (/home/user/Devel/linux-next/arch/arm64/mm/physaddr.c:?) > > [ 622.734533] [ T250254] sp : ffff800158e8fc60 > > [ 622.734534] [ T250254] x29: ffff800158e8fc60 x28: ffff0034a7cc7900 x27: 0000000000000000 > > [ 622.734537] [ T250254] x26: 0000000000000000 x25: ffff0034a7cc7900 x24: 00000000040e001f > > [ 622.734539] [ T250254] x23: ffff0010858afb00 x22: cfcecdcccbcac9c8 x21: ffff0033526a01e0 > > [ 622.734541] [ T250254] x20: 0000000000008000 x19: ffcecdcccbcac9c8 x18: ffff80008149c8e4 > > [ 622.734543] [ T250254] x17: 0000000000000001 x16: 0000000000000000 x15: 0000000000000003 > > [ 622.734545] [ T250254] x14: ffff800082962e78 x13: 0000000000000003 x12: ffff003bc6231630 > > [ 622.734546] [ T250254] x11: 0000000000000000 x10: 0000000000000000 x9 : ed44a220ae716b00 > > [ 622.734548] [ T250254] x8 : 0001000000000000 x7 : 0720072007200720 x6 : ffff80008018710c > > [ 622.734550] [ T250254] x5 : 0000000000000001 x4 : 00000090ecc72ac0 x3 : 0000000000000000 > > [ 622.734552] [ T250254] x2 : 0000000000000000 x1 : ffff800081a72bc6 x0 : 000000000000004f > > [ 622.734554] [ T250254] Call trace: > > [ 622.734555] [ T250254] __virt_to_phys (/home/user/Devel/linux-next/arch/arm64/mm/physaddr.c:?) (P) > > [ 622.734557] [ T250254] kfree (/home/user/Devel/linux-next/./include/linux/mm.h:1180 /home/user/Devel/linux-next/mm/slub.c:4871) > > [ 622.734562] [ T250254] vhost_dev_cleanup (/home/user/Devel/linux-next/drivers/vhost/vhost.c:506 /home/user/Devel/linux-next/drivers/vhost/vhost.c:542 /home/user/Devel/linux-next/drivers/vhost/vhost.c:1214) vhost > > [ 622.734571] [ T250254] vhost_vsock_dev_release (/home/user/Devel/linux-next/drivers/vhost/vsock.c:756) vhost_vsock > > > Cc more vsock maintainers. > > > > > > [ 622.734575] [ T250254] __fput (/home/user/Devel/linux-next/fs/file_table.c:469) > > [ 622.734578] [ T250254] fput_close_sync (/home/user/Devel/linux-next/fs/file_table.c:?) > > [ 622.734579] [ T250254] __arm64_sys_close (/home/user/Devel/linux-next/fs/open.c:1589 /home/user/Devel/linux-next/fs/open.c:1572 /home/user/Devel/linux-next/fs/open.c:1572) > > [ 622.734584] [ T250254] invoke_syscall (/home/user/Devel/linux-next/arch/arm64/kernel/syscall.c:50) > > [ 622.734589] [ T250254] el0_svc_common (/home/user/Devel/linux-next/./include/linux/thread_info.h:135 /home/user/Devel/linux-next/arch/arm64/kernel/syscall.c:140) > > [ 622.734591] [ T250254] do_el0_svc (/home/user/Devel/linux-next/arch/arm64/kernel/syscall.c:152) > > [ 622.734594] [ T250254] el0_svc (/home/user/Devel/linux-next/arch/arm64/kernel/entry-common.c:169 /home/user/Devel/linux-next/arch/arm64/kernel/entry-common.c:182 /home/user/Devel/linux-next/arch/arm64/kernel/entry-common.c:880) > > [ 622.734600] [ T250254] el0t_64_sync_handler (/home/user/Devel/linux-next/arch/arm64/kernel/entry-common.c:958) > > [ 622.734603] [ T250254] el0t_64_sync (/home/user/Devel/linux-next/arch/arm64/kernel/entry.S:596) > > [ 622.734605] [ T250254] irq event stamp: 0 > > [ 622.734606] [ T250254] hardirqs last enabled at (0): 0x0 > > [ 622.734610] [ T250254] hardirqs last disabled at (0): copy_process (/home/user/Devel/linux-next/kernel/fork.c:?) > > [ 622.734614] [ T250254] softirqs last enabled at (0): copy_process (/home/user/Devel/linux-next/kernel/fork.c:?) > > [ 622.734616] [ T250254] softirqs last disabled at (0): 0x0 > > [ 622.734618] [ T250254] ---[ end trace 0000000000000000 ]--- > > [ 622.734697] [ T250254] Unable to handle kernel paging request at virtual address 003ff3b33312f288 > > [ 622.734700] [ T250254] Mem abort info: > > [ 622.734701] [ T250254] ESR = 0x0000000096000004 > > [ 622.734702] [ T250254] EC = 0x25: DABT (current EL), IL = 32 bits > > [ 622.734704] [ T250254] SET = 0, FnV = 0 > > [ 622.734705] [ T250254] EA = 0, S1PTW = 0 > > [ 622.734706] [ T250254] FSC = 0x04: level 0 translation fault > > [ 622.734708] [ T250254] Data abort info: > > [ 622.734709] [ T250254] ISV = 0, ISS = 0x00000004, ISS2 = 0x00000000 > > [ 622.734711] [ T250254] CM = 0, WnR = 0, TnD = 0, TagAccess = 0 > > [ 622.734712] [ T250254] GCS = 0, Overlay = 0, DirtyBit = 0, Xs = 0 > > [ 622.734713] [ T250254] [003ff3b33312f288] address between user and kernel address ranges > > [ 622.734715] [ T250254] Internal error: Oops: 0000000096000004 [#1] SMP > > [ 622.734718] [ T250254] Modules linked in: vhost_vsock(E) vfio_iommu_type1(E) vfio(E) unix_diag(E) sch_fq(E) ghes_edac(E) tls(E) tcp_diag(E) inet_diag(E) act_gact(E) cls_bpf(E) nvidia_cspmu(E) ipmi_ssif(E) coresight_trbe(E) arm_cspmu_module(E) arm_smmuv3_pmu(E) ipmi_devintf(E) coresight_stm(E) coresight_funnel(E) coresight_etm4x(E) coresight_tmc(E) stm_core(E) ipmi_msghandler(E) coresight(E) cppc_cpufreq(E) sch_fq_codel(E) drm(E) backlight(E) drm_panel_orientation_quirks(E) sm3_ce(E) sha3_ce(E) spi_tegra210_quad(E) vhost_net(E) tap(E) tun(E) vhost(E) vhost_iotlb(E) mpls_gso(E) mpls_iptunnel(E) mpls_router(E) fou(E) acpi_power_meter(E) loop(E) efivarfs(E) autofs4(E) [last unloaded: test_bpf(E)] > > [ 622.734740] [ T250254] Tainted: [W]=WARN, [E]=UNSIGNED_MODULE, [N]=TEST > > [ 622.734740] [ T250254] Hardware name: ... > > [ 622.734741] [ T250254] pstate: 63401009 (nZCv daif +PAN -UAO +TCO +DIT +SSBS BTYPE=--) > > [ 622.734742] [ T250254] pc : kfree (/home/user/Devel/linux-next/./include/linux/page-flags.h:284 /home/user/Devel/linux-next/./include/linux/mm.h:1182 /home/user/Devel/linux-next/mm/slub.c:4871) > > [ 622.734745] [ T250254] lr : kfree (/home/user/Devel/linux-next/./include/linux/mm.h:1180 /home/user/Devel/linux-next/mm/slub.c:4871) > > [ 622.734747] [ T250254] sp : ffff800158e8fc80 > > [ 622.734748] [ T250254] x29: ffff800158e8fc90 x28: ffff0034a7cc7900 x27: 0000000000000000 > > [ 622.734749] [ T250254] x26: 0000000000000000 x25: ffff0034a7cc7900 x24: 00000000040e001f > > [ 622.734751] [ T250254] x23: ffff0010858afb00 x22: cfcecdcccbcac9c8 x21: ffff0033526a01e0 > > [ 622.734752] [ T250254] x20: 003ff3b33312f280 x19: ffff80000acd1a20 x18: ffff80008149c8e4 > > [ 622.734754] [ T250254] x17: 0000000000000001 x16: 0000000000000000 x15: 0000000000000003 > > [ 622.734755] [ T250254] x14: ffff800082962e78 x13: 0000000000000003 x12: ffff003bc6231630 > > [ 622.734757] [ T250254] x11: 0000000000000000 x10: 0000000000000000 x9 : ffffffdfc0000000 > > [ 622.734758] [ T250254] x8 : 003ff3d37312f280 x7 : 0720072007200720 x6 : ffff80008018710c > > [ 622.734760] [ T250254] x5 : 0000000000000001 x4 : 00000090ecc72ac0 x3 : 0000000000000000 > > [ 622.734761] [ T250254] x2 : 0000000000000000 x1 : ffff800081a72bc6 x0 : ffcf4dcccbcac9c8 > > [ 622.734763] [ T250254] Call trace: > > [ 622.734763] [ T250254] kfree (/home/user/Devel/linux-next/./include/linux/page-flags.h:284 /home/user/Devel/linux-next/./include/linux/mm.h:1182 /home/user/Devel/linux-next/mm/slub.c:4871) (P) > > [ 622.734766] [ T250254] vhost_dev_cleanup (/home/user/Devel/linux-next/drivers/vhost/vhost.c:506 /home/user/Devel/linux-next/drivers/vhost/vhost.c:542 /home/user/Devel/linux-next/drivers/vhost/vhost.c:1214) vhost > > [ 622.734769] [ T250254] vhost_vsock_dev_release (/home/user/Devel/linux-next/drivers/vhost/vsock.c:756) vhost_vsock > > [ 622.734771] [ T250254] __fput (/home/user/Devel/linux-next/fs/file_table.c:469) > > [ 622.734772] [ T250254] fput_close_sync (/home/user/Devel/linux-next/fs/file_table.c:?) > > [ 622.734773] [ T250254] __arm64_sys_close (/home/user/Devel/linux-next/fs/open.c:1589 /home/user/Devel/linux-next/fs/open.c:1572 /home/user/Devel/linux-next/fs/open.c:1572) > > [ 622.734776] [ T250254] invoke_syscall (/home/user/Devel/linux-next/arch/arm64/kernel/syscall.c:50) > > [ 622.734778] [ T250254] el0_svc_common (/home/user/Devel/linux-next/./include/linux/thread_info.h:135 /home/user/Devel/linux-next/arch/arm64/kernel/syscall.c:140) > > [ 622.734781] [ T250254] do_el0_svc (/home/user/Devel/linux-next/arch/arm64/kernel/syscall.c:152) > > [ 622.734783] [ T250254] el0_svc (/home/user/Devel/linux-next/arch/arm64/kernel/entry-common.c:169 /home/user/Devel/linux-next/arch/arm64/kernel/entry-common.c:182 /home/user/Devel/linux-next/arch/arm64/kernel/entry-common.c:880) > > [ 622.734787] [ T250254] el0t_64_sync_handler (/home/user/Devel/linux-next/arch/arm64/kernel/entry-common.c:958) > > [ 622.734790] [ T250254] el0t_64_sync (/home/user/Devel/linux-next/arch/arm64/kernel/entry.S:596) > > [ 622.734792] [ T250254] Code: f2dffbe9 927abd08 cb141908 8b090114 (f9400688) > > All code > > ======== > > 0:* e9 fb df f2 08 jmp 0x8f2e000 <-- trapping instruction > > 5: bd 7a 92 08 19 mov $0x1908927a,%ebp > > a: 14 cb adc $0xcb,%al > > c: 14 01 adc $0x1,%al > > e: 09 8b 88 06 40 f9 or %ecx,-0x6bff978(%rbx) > > > > Code starting with the faulting instruction > > =========================================== > > 0: 88 06 mov %al,(%rsi) > > 2: 40 f9 rex stc > > [ 622.734795] [ T250254] SMP: stopping secondary CPUs > > [ 622.735089] [ T250254] Starting crashdump kernel... > > [ 622.735091] [ T250254] Bye! > ^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: vhost: linux-next: crash at vhost_dev_cleanup() 2025-07-24 8:14 ` Stefano Garzarella @ 2025-07-24 8:22 ` Michael S. Tsirkin 2025-07-24 8:44 ` Will Deacon 0 siblings, 1 reply; 12+ messages in thread From: Michael S. Tsirkin @ 2025-07-24 8:22 UTC (permalink / raw) To: Stefano Garzarella Cc: Will Deacon, Breno Leitao, jasowang, eperezma, linux-arm-kernel, kvm, Stefan Hajnoczi, netdev On Thu, Jul 24, 2025 at 10:14:36AM +0200, Stefano Garzarella wrote: > CCing Will > > On Thu, 24 Jul 2025 at 09:48, Michael S. Tsirkin <mst@redhat.com> wrote: > > > > On Wed, Jul 23, 2025 at 08:04:42AM -0700, Breno Leitao wrote: > > > Hello, > > > > > > I've seen a crash in linux-next for a while on my arm64 server, and > > > I decided to report. > > > > > > While running stress-ng on linux-next, I see the crash below. > > > > > > This is happening in a kernel configure with some debug options (KASAN, > > > LOCKDEP and KMEMLEAK). > > > > > > Basically running stress-ng in a loop would crash the host in 15-20 > > > minutes: > > > # while (true); do stress-ng -r 10 -t 10; done > > > > > > >From the early warning "virt_to_phys used for non-linear address", > > mmm, we recently added nonlinear SKBs support in vhost-vsock [1], > @Will can this issue be related? Good point. Breno, if bisecting is too much trouble, would you mind testing the commits c76f3c4364fe523cd2782269eab92529c86217aa and c7991b44d7b44f9270dec63acd0b2965d29aab43 and telling us if this reproduces? > I checked next-20250721 tag and I confirm that contains those changes. > > [1] https://lore.kernel.org/virtualization/20250717090116.11987-1-will@kernel.org/ > > Thanks, > Stefano > > > > I suppose corrupted data is at vq->nheads. > > > > > > Here is the decoded stack against 9798752 ("Add linux-next specific > > > files for 20250721") > > > > > > > > > [ 620.685144] [ T250731] VFIO - User Level meta-driver version: 0.3 > > > [ 622.394448] [ T250254] ------------[ cut here ]------------ > > > [ 622.413492] [ T250254] virt_to_phys used for non-linear address: 000000006e69fe64 (0xcfcecdcccbcac9c8) > > > [ 622.447771] [ T250254] WARNING: arch/arm64/mm/physaddr.c:15 at __virt_to_phys+0x64/0x90, CPU#57: stress-ng-dev/250254 > > > [ 622.487227] [ T250254] Modules linked in: vhost_vsock(E) vfio_iommu_type1(E) vfio(E) unix_diag(E) sch_fq(E) ghes_edac(E) tls(E) tcp_diag(E) inet_diag(E) act_gact(E) cls_bpf(E) nvidia_cspmu(E) ipmi_ssif(E) coresight_trbe(E) arm_cspmu_module(E) arm_smmuv3_pmu(E) ipmi_devintf(E) coresight_stm(E) coresight_funnel(E) coresight_etm4x(E) coresight_tmc(E) stm_core(E) ipmi_msghandler(E) coresight(E) cppc_cpufreq(E) sch_fq_codel(E) drm(E) backlight(E) drm_panel_orientation_quirks(E) sm3_ce(E) sha3_ce(E) spi_tegra210_quad(E) vhost_net(E) tap(E) tun(E) vhost(E) vhost_iotlb(E) mpls_gso(E) mpls_iptunnel(E) mpls_router(E) fou(E) acpi_power_meter(E) loop(E) efivarfs(E) autofs4(E) [last unloaded: test_bpf(E)] > > > [ 622.734524] [ T250254] Tainted: [W]=WARN, [E]=UNSIGNED_MODULE, [N]=TEST > > > [ 622.734525] [ T250254] Hardware name: ... > > > [ 622.734526] [ T250254] pstate: 63401009 (nZCv daif +PAN -UAO +TCO +DIT +SSBS BTYPE=--) > > > [ 622.734529] [ T250254] pc : __virt_to_phys (/home/user/Devel/linux-next/arch/arm64/mm/physaddr.c:?) > > > [ 622.734531] [ T250254] lr : __virt_to_phys (/home/user/Devel/linux-next/arch/arm64/mm/physaddr.c:?) > > > [ 622.734533] [ T250254] sp : ffff800158e8fc60 > > > [ 622.734534] [ T250254] x29: ffff800158e8fc60 x28: ffff0034a7cc7900 x27: 0000000000000000 > > > [ 622.734537] [ T250254] x26: 0000000000000000 x25: ffff0034a7cc7900 x24: 00000000040e001f > > > [ 622.734539] [ T250254] x23: ffff0010858afb00 x22: cfcecdcccbcac9c8 x21: ffff0033526a01e0 > > > [ 622.734541] [ T250254] x20: 0000000000008000 x19: ffcecdcccbcac9c8 x18: ffff80008149c8e4 > > > [ 622.734543] [ T250254] x17: 0000000000000001 x16: 0000000000000000 x15: 0000000000000003 > > > [ 622.734545] [ T250254] x14: ffff800082962e78 x13: 0000000000000003 x12: ffff003bc6231630 > > > [ 622.734546] [ T250254] x11: 0000000000000000 x10: 0000000000000000 x9 : ed44a220ae716b00 > > > [ 622.734548] [ T250254] x8 : 0001000000000000 x7 : 0720072007200720 x6 : ffff80008018710c > > > [ 622.734550] [ T250254] x5 : 0000000000000001 x4 : 00000090ecc72ac0 x3 : 0000000000000000 > > > [ 622.734552] [ T250254] x2 : 0000000000000000 x1 : ffff800081a72bc6 x0 : 000000000000004f > > > [ 622.734554] [ T250254] Call trace: > > > [ 622.734555] [ T250254] __virt_to_phys (/home/user/Devel/linux-next/arch/arm64/mm/physaddr.c:?) (P) > > > [ 622.734557] [ T250254] kfree (/home/user/Devel/linux-next/./include/linux/mm.h:1180 /home/user/Devel/linux-next/mm/slub.c:4871) > > > [ 622.734562] [ T250254] vhost_dev_cleanup (/home/user/Devel/linux-next/drivers/vhost/vhost.c:506 /home/user/Devel/linux-next/drivers/vhost/vhost.c:542 /home/user/Devel/linux-next/drivers/vhost/vhost.c:1214) vhost > > > [ 622.734571] [ T250254] vhost_vsock_dev_release (/home/user/Devel/linux-next/drivers/vhost/vsock.c:756) vhost_vsock > > > > > > Cc more vsock maintainers. > > > > > > > > > > > [ 622.734575] [ T250254] __fput (/home/user/Devel/linux-next/fs/file_table.c:469) > > > [ 622.734578] [ T250254] fput_close_sync (/home/user/Devel/linux-next/fs/file_table.c:?) > > > [ 622.734579] [ T250254] __arm64_sys_close (/home/user/Devel/linux-next/fs/open.c:1589 /home/user/Devel/linux-next/fs/open.c:1572 /home/user/Devel/linux-next/fs/open.c:1572) > > > [ 622.734584] [ T250254] invoke_syscall (/home/user/Devel/linux-next/arch/arm64/kernel/syscall.c:50) > > > [ 622.734589] [ T250254] el0_svc_common (/home/user/Devel/linux-next/./include/linux/thread_info.h:135 /home/user/Devel/linux-next/arch/arm64/kernel/syscall.c:140) > > > [ 622.734591] [ T250254] do_el0_svc (/home/user/Devel/linux-next/arch/arm64/kernel/syscall.c:152) > > > [ 622.734594] [ T250254] el0_svc (/home/user/Devel/linux-next/arch/arm64/kernel/entry-common.c:169 /home/user/Devel/linux-next/arch/arm64/kernel/entry-common.c:182 /home/user/Devel/linux-next/arch/arm64/kernel/entry-common.c:880) > > > [ 622.734600] [ T250254] el0t_64_sync_handler (/home/user/Devel/linux-next/arch/arm64/kernel/entry-common.c:958) > > > [ 622.734603] [ T250254] el0t_64_sync (/home/user/Devel/linux-next/arch/arm64/kernel/entry.S:596) > > > [ 622.734605] [ T250254] irq event stamp: 0 > > > [ 622.734606] [ T250254] hardirqs last enabled at (0): 0x0 > > > [ 622.734610] [ T250254] hardirqs last disabled at (0): copy_process (/home/user/Devel/linux-next/kernel/fork.c:?) > > > [ 622.734614] [ T250254] softirqs last enabled at (0): copy_process (/home/user/Devel/linux-next/kernel/fork.c:?) > > > [ 622.734616] [ T250254] softirqs last disabled at (0): 0x0 > > > [ 622.734618] [ T250254] ---[ end trace 0000000000000000 ]--- > > > [ 622.734697] [ T250254] Unable to handle kernel paging request at virtual address 003ff3b33312f288 > > > [ 622.734700] [ T250254] Mem abort info: > > > [ 622.734701] [ T250254] ESR = 0x0000000096000004 > > > [ 622.734702] [ T250254] EC = 0x25: DABT (current EL), IL = 32 bits > > > [ 622.734704] [ T250254] SET = 0, FnV = 0 > > > [ 622.734705] [ T250254] EA = 0, S1PTW = 0 > > > [ 622.734706] [ T250254] FSC = 0x04: level 0 translation fault > > > [ 622.734708] [ T250254] Data abort info: > > > [ 622.734709] [ T250254] ISV = 0, ISS = 0x00000004, ISS2 = 0x00000000 > > > [ 622.734711] [ T250254] CM = 0, WnR = 0, TnD = 0, TagAccess = 0 > > > [ 622.734712] [ T250254] GCS = 0, Overlay = 0, DirtyBit = 0, Xs = 0 > > > [ 622.734713] [ T250254] [003ff3b33312f288] address between user and kernel address ranges > > > [ 622.734715] [ T250254] Internal error: Oops: 0000000096000004 [#1] SMP > > > [ 622.734718] [ T250254] Modules linked in: vhost_vsock(E) vfio_iommu_type1(E) vfio(E) unix_diag(E) sch_fq(E) ghes_edac(E) tls(E) tcp_diag(E) inet_diag(E) act_gact(E) cls_bpf(E) nvidia_cspmu(E) ipmi_ssif(E) coresight_trbe(E) arm_cspmu_module(E) arm_smmuv3_pmu(E) ipmi_devintf(E) coresight_stm(E) coresight_funnel(E) coresight_etm4x(E) coresight_tmc(E) stm_core(E) ipmi_msghandler(E) coresight(E) cppc_cpufreq(E) sch_fq_codel(E) drm(E) backlight(E) drm_panel_orientation_quirks(E) sm3_ce(E) sha3_ce(E) spi_tegra210_quad(E) vhost_net(E) tap(E) tun(E) vhost(E) vhost_iotlb(E) mpls_gso(E) mpls_iptunnel(E) mpls_router(E) fou(E) acpi_power_meter(E) loop(E) efivarfs(E) autofs4(E) [last unloaded: test_bpf(E)] > > > [ 622.734740] [ T250254] Tainted: [W]=WARN, [E]=UNSIGNED_MODULE, [N]=TEST > > > [ 622.734740] [ T250254] Hardware name: ... > > > [ 622.734741] [ T250254] pstate: 63401009 (nZCv daif +PAN -UAO +TCO +DIT +SSBS BTYPE=--) > > > [ 622.734742] [ T250254] pc : kfree (/home/user/Devel/linux-next/./include/linux/page-flags.h:284 /home/user/Devel/linux-next/./include/linux/mm.h:1182 /home/user/Devel/linux-next/mm/slub.c:4871) > > > [ 622.734745] [ T250254] lr : kfree (/home/user/Devel/linux-next/./include/linux/mm.h:1180 /home/user/Devel/linux-next/mm/slub.c:4871) > > > [ 622.734747] [ T250254] sp : ffff800158e8fc80 > > > [ 622.734748] [ T250254] x29: ffff800158e8fc90 x28: ffff0034a7cc7900 x27: 0000000000000000 > > > [ 622.734749] [ T250254] x26: 0000000000000000 x25: ffff0034a7cc7900 x24: 00000000040e001f > > > [ 622.734751] [ T250254] x23: ffff0010858afb00 x22: cfcecdcccbcac9c8 x21: ffff0033526a01e0 > > > [ 622.734752] [ T250254] x20: 003ff3b33312f280 x19: ffff80000acd1a20 x18: ffff80008149c8e4 > > > [ 622.734754] [ T250254] x17: 0000000000000001 x16: 0000000000000000 x15: 0000000000000003 > > > [ 622.734755] [ T250254] x14: ffff800082962e78 x13: 0000000000000003 x12: ffff003bc6231630 > > > [ 622.734757] [ T250254] x11: 0000000000000000 x10: 0000000000000000 x9 : ffffffdfc0000000 > > > [ 622.734758] [ T250254] x8 : 003ff3d37312f280 x7 : 0720072007200720 x6 : ffff80008018710c > > > [ 622.734760] [ T250254] x5 : 0000000000000001 x4 : 00000090ecc72ac0 x3 : 0000000000000000 > > > [ 622.734761] [ T250254] x2 : 0000000000000000 x1 : ffff800081a72bc6 x0 : ffcf4dcccbcac9c8 > > > [ 622.734763] [ T250254] Call trace: > > > [ 622.734763] [ T250254] kfree (/home/user/Devel/linux-next/./include/linux/page-flags.h:284 /home/user/Devel/linux-next/./include/linux/mm.h:1182 /home/user/Devel/linux-next/mm/slub.c:4871) (P) > > > [ 622.734766] [ T250254] vhost_dev_cleanup (/home/user/Devel/linux-next/drivers/vhost/vhost.c:506 /home/user/Devel/linux-next/drivers/vhost/vhost.c:542 /home/user/Devel/linux-next/drivers/vhost/vhost.c:1214) vhost > > > [ 622.734769] [ T250254] vhost_vsock_dev_release (/home/user/Devel/linux-next/drivers/vhost/vsock.c:756) vhost_vsock > > > [ 622.734771] [ T250254] __fput (/home/user/Devel/linux-next/fs/file_table.c:469) > > > [ 622.734772] [ T250254] fput_close_sync (/home/user/Devel/linux-next/fs/file_table.c:?) > > > [ 622.734773] [ T250254] __arm64_sys_close (/home/user/Devel/linux-next/fs/open.c:1589 /home/user/Devel/linux-next/fs/open.c:1572 /home/user/Devel/linux-next/fs/open.c:1572) > > > [ 622.734776] [ T250254] invoke_syscall (/home/user/Devel/linux-next/arch/arm64/kernel/syscall.c:50) > > > [ 622.734778] [ T250254] el0_svc_common (/home/user/Devel/linux-next/./include/linux/thread_info.h:135 /home/user/Devel/linux-next/arch/arm64/kernel/syscall.c:140) > > > [ 622.734781] [ T250254] do_el0_svc (/home/user/Devel/linux-next/arch/arm64/kernel/syscall.c:152) > > > [ 622.734783] [ T250254] el0_svc (/home/user/Devel/linux-next/arch/arm64/kernel/entry-common.c:169 /home/user/Devel/linux-next/arch/arm64/kernel/entry-common.c:182 /home/user/Devel/linux-next/arch/arm64/kernel/entry-common.c:880) > > > [ 622.734787] [ T250254] el0t_64_sync_handler (/home/user/Devel/linux-next/arch/arm64/kernel/entry-common.c:958) > > > [ 622.734790] [ T250254] el0t_64_sync (/home/user/Devel/linux-next/arch/arm64/kernel/entry.S:596) > > > [ 622.734792] [ T250254] Code: f2dffbe9 927abd08 cb141908 8b090114 (f9400688) > > > All code > > > ======== > > > 0:* e9 fb df f2 08 jmp 0x8f2e000 <-- trapping instruction > > > 5: bd 7a 92 08 19 mov $0x1908927a,%ebp > > > a: 14 cb adc $0xcb,%al > > > c: 14 01 adc $0x1,%al > > > e: 09 8b 88 06 40 f9 or %ecx,-0x6bff978(%rbx) > > > > > > Code starting with the faulting instruction > > > =========================================== > > > 0: 88 06 mov %al,(%rsi) > > > 2: 40 f9 rex stc > > > [ 622.734795] [ T250254] SMP: stopping secondary CPUs > > > [ 622.735089] [ T250254] Starting crashdump kernel... > > > [ 622.735091] [ T250254] Bye! > > ^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: vhost: linux-next: crash at vhost_dev_cleanup() 2025-07-24 8:22 ` Michael S. Tsirkin @ 2025-07-24 8:44 ` Will Deacon 2025-07-24 12:48 ` Breno Leitao 0 siblings, 1 reply; 12+ messages in thread From: Will Deacon @ 2025-07-24 8:44 UTC (permalink / raw) To: Michael S. Tsirkin Cc: Stefano Garzarella, Breno Leitao, jasowang, eperezma, linux-arm-kernel, kvm, Stefan Hajnoczi, netdev On Thu, Jul 24, 2025 at 04:22:15AM -0400, Michael S. Tsirkin wrote: > On Thu, Jul 24, 2025 at 10:14:36AM +0200, Stefano Garzarella wrote: > > CCing Will Thanks. > > On Thu, 24 Jul 2025 at 09:48, Michael S. Tsirkin <mst@redhat.com> wrote: > > > > > > On Wed, Jul 23, 2025 at 08:04:42AM -0700, Breno Leitao wrote: > > > > Hello, > > > > > > > > I've seen a crash in linux-next for a while on my arm64 server, and > > > > I decided to report. > > > > > > > > While running stress-ng on linux-next, I see the crash below. > > > > > > > > This is happening in a kernel configure with some debug options (KASAN, > > > > LOCKDEP and KMEMLEAK). > > > > > > > > Basically running stress-ng in a loop would crash the host in 15-20 > > > > minutes: > > > > # while (true); do stress-ng -r 10 -t 10; done > > > > > > > > >From the early warning "virt_to_phys used for non-linear address", > > > > mmm, we recently added nonlinear SKBs support in vhost-vsock [1], > > @Will can this issue be related? > > Good point. > > Breno, if bisecting is too much trouble, would you mind testing the commits > c76f3c4364fe523cd2782269eab92529c86217aa > and > c7991b44d7b44f9270dec63acd0b2965d29aab43 > and telling us if this reproduces? That's definitely worth doing, but we should be careful not to confuse the "non-linear address" from the warning (which refers to virtual addresses that lie outside of the linear mapping of memory, e.g. in the vmalloc space) and "non-linear SKBs" which refer to SKBs with fragment pages. Breno -- when you say you've been seeing this "for a while", what's the earliest kernel you know you saw it on? > > > > I suppose corrupted data is at vq->nheads. > > > > > > > > Here is the decoded stack against 9798752 ("Add linux-next specific > > > > files for 20250721") > > > > > > > > > > > > [ 620.685144] [ T250731] VFIO - User Level meta-driver version: 0.3 > > > > [ 622.394448] [ T250254] ------------[ cut here ]------------ > > > > [ 622.413492] [ T250254] virt_to_phys used for non-linear address: 000000006e69fe64 (0xcfcecdcccbcac9c8) So here's the bad (non-linear) pointer. Do you know if 0xcfcecdcccbcac9c8 correlates with the packet data that stress-ng is generating? I wonder if we're somehow overflowing vq->iov[]. Will ^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: vhost: linux-next: crash at vhost_dev_cleanup() 2025-07-24 8:44 ` Will Deacon @ 2025-07-24 12:48 ` Breno Leitao 2025-07-24 12:52 ` Stefano Garzarella 0 siblings, 1 reply; 12+ messages in thread From: Breno Leitao @ 2025-07-24 12:48 UTC (permalink / raw) To: Will Deacon Cc: Michael S. Tsirkin, Stefano Garzarella, jasowang, eperezma, linux-arm-kernel, kvm, Stefan Hajnoczi, netdev On Thu, Jul 24, 2025 at 09:44:38AM +0100, Will Deacon wrote: > > > On Thu, 24 Jul 2025 at 09:48, Michael S. Tsirkin <mst@redhat.com> wrote: > > > > > > > > On Wed, Jul 23, 2025 at 08:04:42AM -0700, Breno Leitao wrote: > > > > > Hello, > > > > > > > > > > I've seen a crash in linux-next for a while on my arm64 server, and > > > > > I decided to report. > > > > > > > > > > While running stress-ng on linux-next, I see the crash below. > > > > > > > > > > This is happening in a kernel configure with some debug options (KASAN, > > > > > LOCKDEP and KMEMLEAK). > > > > > > > > > > Basically running stress-ng in a loop would crash the host in 15-20 > > > > > minutes: > > > > > # while (true); do stress-ng -r 10 -t 10; done > > > > > > > > > > >From the early warning "virt_to_phys used for non-linear address", > > > > > > mmm, we recently added nonlinear SKBs support in vhost-vsock [1], > > > @Will can this issue be related? > > > > Good point. > > > > Breno, if bisecting is too much trouble, would you mind testing the commits > > c76f3c4364fe523cd2782269eab92529c86217aa > > and > > c7991b44d7b44f9270dec63acd0b2965d29aab43 > > and telling us if this reproduces? > > That's definitely worth doing, but we should be careful not to confuse > the "non-linear address" from the warning (which refers to virtual > addresses that lie outside of the linear mapping of memory, e.g. in the > vmalloc space) and "non-linear SKBs" which refer to SKBs with fragment > pages. I've tested both commits above, and I see the crash on both commits above, thus, the problem reproduces in both cases. The only difference I noted is the fact that I haven't seen the warning before the crash. Log against c76f3c4364fe ("vhost/vsock: Avoid allocating arbitrarily-sized SKBs") Unable to handle kernel paging request at virtual address 0000001fc0000048 Mem abort info: ESR = 0x0000000096000005 EC = 0x25: DABT (current EL), IL = 32 bits SET = 0, FnV = 0 EA = 0, S1PTW = 0 FSC = 0x05: level 1 translation fault Data abort info: ISV = 0, ISS = 0x00000005, ISS2 = 0x00000000 CM = 0, WnR = 0, TnD = 0, TagAccess = 0 GCS = 0, Overlay = 0, DirtyBit = 0, Xs = 0 user pgtable: 64k pages, 48-bit VAs, pgdp=0000000cdcf2da00 [0000001fc0000048] pgd=0000000000000000, p4d=0000000000000000, pud=0000000000000000 Internal error: Oops: 0000000096000005 [#1] SMP Modules linked in: vfio_iommu_type1 vfio md4 crc32_cryptoapi ghash_generic unix_diag vhost_net tun vhost vhost_iotlb tap mpls_gso mpls_iptunnel mpls_router fou sch_fq ghes_edac tls tcp_diag inet_diag act_gact cls_bpf nvidia_c CPU: 34 UID: 0 PID: 1727297 Comm: stress-ng-dev Kdump: loaded Not tainted 6.16.0-rc6-upstream-00027-gc76f3c4364fe #19 NONE pstate: 23401009 (nzCv daif +PAN -UAO +TCO +DIT +SSBS BTYPE=--) pc : kfree+0x48/0x2a8 lr : vhost_dev_cleanup+0x138/0x2b8 [vhost] sp : ffff80013a0cfcd0 x29: ffff80013a0cfcd0 x28: ffff0008fd0b6240 x27: 0000000000000000 x26: 0000000000000000 x25: 0000000000000000 x24: 0000000000000000 x23: 00000000040e001f x22: ffffffffffffffff x21: ffff00014f1d4ac0 x20: 0000000000000001 x19: ffff00014f1d0000 x18: 0000000000000000 x17: 0000000000000000 x16: 0000000000000000 x15: 0000000000000000 x14: 000000000000001f x13: 000000000000000f x12: 0000000000000001 x11: 0000000000000000 x10: 0000000000000402 x9 : ffffffdfc0000000 x8 : 0000001fc0000040 x7 : 0000000000000000 x6 : 0000000000000000 x5 : ffff000141931840 x4 : 0000000000000000 x3 : 0000000000000008 x2 : ffffffffffffffff x1 : ffffffffffffffff x0 : 0000000000010000 Call trace: kfree+0x48/0x2a8 (P) vhost_dev_cleanup+0x138/0x2b8 [vhost] vhost_net_release+0xa0/0x1a8 [vhost_net] __fput+0xfc/0x2f0 fput_close_sync+0x38/0xc8 __arm64_sys_close+0xb4/0x108 invoke_syscall+0x4c/0xd0 do_el0_svc+0x80/0xb0 el0_svc+0x3c/0xd0 el0t_64_sync_handler+0x70/0x100 el0t_64_sync+0x170/0x178 Code: 8b080008 f2dffbe9 d350fd08 8b081928 (f9400509) Log against c7991b44d7b4 ("vsock/virtio: Allocate nonlinear SKBs for handling large transmit buffers") Unable to handle kernel paging request at virtual address 0010502f8f8f4f08 Mem abort info: ESR = 0x0000000096000004 EC = 0x25: DABT (current EL), IL = 32 bits SET = 0, FnV = 0 EA = 0, S1PTW = 0 FSC = 0x04: level 0 translation fault Data abort info: ISV = 0, ISS = 0x00000004, ISS2 = 0x00000000 CM = 0, WnR = 0, TnD = 0, TagAccess = 0 GCS = 0, Overlay = 0, DirtyBit = 0, Xs = 0 [0010502f8f8f4f08] address between user and kernel address ranges Internal error: Oops: 0000000096000004 [#1] SMP Modules linked in: vhost_vsock vfio_iommu_type1 vfio md4 crc32_cryptoapi ghash_generic vhost_net tun vhost vhost_iotlb tap mpls_gso mpls_iptunnel mpls_router fou sch_fq ghes_edac tls tcp_diag inet_diag act_gact cls_bpf ipmi_s CPU: 47 UID: 0 PID: 1239699 Comm: stress-ng-dev Kdump: loaded Tainted: G W 6.16.0-rc6-upstream-00035-gc7991b44d7b4 #18 NONE Tainted: [W]=WARN pstate: 23401009 (nzCv daif +PAN -UAO +TCO +DIT +SSBS BTYPE=--) pc : kfree+0x48/0x2a8 lr : vhost_dev_cleanup+0x138/0x2b8 [vhost] sp : ffff80016c0cfcd0 x29: ffff80016c0cfcd0 x28: ffff001ad6210d80 x27: 0000000000000000 x26: 0000000000000000 x25: 0000000000000000 x24: 0000000000000000 x23: 00000000040e001f x22: ffffffffffffffff x21: ffff001bb76f00c0 x20: 0000000000000000 x19: ffff001bb76f0000 x18: 0000000000000000 x17: 0000000000000000 x16: 0000000000000000 x15: 0000000000000000 x14: 000000000000001f x13: 000000000000000f x12: 0000000000000001 x11: 0000000000000000 x10: 0000000000000402 x9 : ffffffdfc0000000 x8 : 0010502f8f8f4f00 x7 : 0000000000000000 x6 : 0000000000000000 x5 : ffff00012e7e2128 x4 : 0000000000000000 x3 : 0000000000000008 x2 : ffffffffffffffff x1 : ffffffffffffffff x0 : 41403f3e3d3c3b3a Call trace: kfree+0x48/0x2a8 (P) vhost_dev_cleanup+0x138/0x2b8 [vhost] vhost_net_release+0xa0/0x1a8 [vhost_net] __fput+0xfc/0x2f0 fput_close_sync+0x38/0xc8 __arm64_sys_close+0xb4/0x108 invoke_syscall+0x4c/0xd0 do_el0_svc+0x80/0xb0 el0_svc+0x3c/0xd0 el0t_64_sync_handler+0x70/0x100 el0t_64_sync+0x170/0x178 Code: 8b080008 f2dffbe9 d350fd08 8b081928 (f9400509) > Breno -- when you say you've been seeing this "for a while", what's the > earliest kernel you know you saw it on? Looking at my logs, the older kernel that I saw it was net-next from 20250717, which was around the time I decided to test net-next in preparation for 6.17, so, not very helpful. Sorry. ^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: vhost: linux-next: crash at vhost_dev_cleanup() 2025-07-24 12:48 ` Breno Leitao @ 2025-07-24 12:52 ` Stefano Garzarella 2025-07-24 13:49 ` Breno Leitao 0 siblings, 1 reply; 12+ messages in thread From: Stefano Garzarella @ 2025-07-24 12:52 UTC (permalink / raw) To: Breno Leitao Cc: Will Deacon, Michael S. Tsirkin, jasowang, eperezma, linux-arm-kernel, kvm, Stefan Hajnoczi, netdev On Thu, 24 Jul 2025 at 14:48, Breno Leitao <leitao@debian.org> wrote: > > On Thu, Jul 24, 2025 at 09:44:38AM +0100, Will Deacon wrote: > > > > On Thu, 24 Jul 2025 at 09:48, Michael S. Tsirkin <mst@redhat.com> wrote: > > > > > > > > > > On Wed, Jul 23, 2025 at 08:04:42AM -0700, Breno Leitao wrote: > > > > > > Hello, > > > > > > > > > > > > I've seen a crash in linux-next for a while on my arm64 server, and > > > > > > I decided to report. > > > > > > > > > > > > While running stress-ng on linux-next, I see the crash below. > > > > > > > > > > > > This is happening in a kernel configure with some debug options (KASAN, > > > > > > LOCKDEP and KMEMLEAK). > > > > > > > > > > > > Basically running stress-ng in a loop would crash the host in 15-20 > > > > > > minutes: > > > > > > # while (true); do stress-ng -r 10 -t 10; done > > > > > > > > > > > > >From the early warning "virt_to_phys used for non-linear address", > > > > > > > > mmm, we recently added nonlinear SKBs support in vhost-vsock [1], > > > > @Will can this issue be related? > > > > > > Good point. > > > > > > Breno, if bisecting is too much trouble, would you mind testing the commits > > > c76f3c4364fe523cd2782269eab92529c86217aa > > > and > > > c7991b44d7b44f9270dec63acd0b2965d29aab43 > > > and telling us if this reproduces? > > > > That's definitely worth doing, but we should be careful not to confuse > > the "non-linear address" from the warning (which refers to virtual > > addresses that lie outside of the linear mapping of memory, e.g. in the > > vmalloc space) and "non-linear SKBs" which refer to SKBs with fragment > > pages. > > I've tested both commits above, and I see the crash on both commits > above, thus, the problem reproduces in both cases. The only difference > I noted is the fact that I haven't seen the warning before the crash. > > > Log against c76f3c4364fe ("vhost/vsock: Avoid allocating > arbitrarily-sized SKBs") > > Unable to handle kernel paging request at virtual address 0000001fc0000048 > Mem abort info: > ESR = 0x0000000096000005 > EC = 0x25: DABT (current EL), IL = 32 bits > SET = 0, FnV = 0 > EA = 0, S1PTW = 0 > FSC = 0x05: level 1 translation fault > Data abort info: > ISV = 0, ISS = 0x00000005, ISS2 = 0x00000000 > CM = 0, WnR = 0, TnD = 0, TagAccess = 0 > GCS = 0, Overlay = 0, DirtyBit = 0, Xs = 0 > user pgtable: 64k pages, 48-bit VAs, pgdp=0000000cdcf2da00 > [0000001fc0000048] pgd=0000000000000000, p4d=0000000000000000, pud=0000000000000000 > Internal error: Oops: 0000000096000005 [#1] SMP > Modules linked in: vfio_iommu_type1 vfio md4 crc32_cryptoapi ghash_generic unix_diag vhost_net tun vhost vhost_iotlb tap mpls_gso mpls_iptunnel mpls_router fou sch_fq ghes_edac tls tcp_diag inet_diag act_gact cls_bpf nvidia_c > CPU: 34 UID: 0 PID: 1727297 Comm: stress-ng-dev Kdump: loaded Not tainted 6.16.0-rc6-upstream-00027-gc76f3c4364fe #19 NONE > pstate: 23401009 (nzCv daif +PAN -UAO +TCO +DIT +SSBS BTYPE=--) > pc : kfree+0x48/0x2a8 > lr : vhost_dev_cleanup+0x138/0x2b8 [vhost] > sp : ffff80013a0cfcd0 > x29: ffff80013a0cfcd0 x28: ffff0008fd0b6240 x27: 0000000000000000 > x26: 0000000000000000 x25: 0000000000000000 x24: 0000000000000000 > x23: 00000000040e001f x22: ffffffffffffffff x21: ffff00014f1d4ac0 > x20: 0000000000000001 x19: ffff00014f1d0000 x18: 0000000000000000 > x17: 0000000000000000 x16: 0000000000000000 x15: 0000000000000000 > x14: 000000000000001f x13: 000000000000000f x12: 0000000000000001 > x11: 0000000000000000 x10: 0000000000000402 x9 : ffffffdfc0000000 > x8 : 0000001fc0000040 x7 : 0000000000000000 x6 : 0000000000000000 > x5 : ffff000141931840 x4 : 0000000000000000 x3 : 0000000000000008 > x2 : ffffffffffffffff x1 : ffffffffffffffff x0 : 0000000000010000 > Call trace: > kfree+0x48/0x2a8 (P) > vhost_dev_cleanup+0x138/0x2b8 [vhost] > vhost_net_release+0xa0/0x1a8 [vhost_net] But here is the vhost_net, so I'm confused now. Do you see the same (vhost_net) also on 9798752 ("Add linux-next specific files for 20250721") ? The initial report contained only vhost_vsock traces IIUC, so I'm suspecting something in the vhost core. Thanks, Stefano > __fput+0xfc/0x2f0 > fput_close_sync+0x38/0xc8 > __arm64_sys_close+0xb4/0x108 > invoke_syscall+0x4c/0xd0 > do_el0_svc+0x80/0xb0 > el0_svc+0x3c/0xd0 > el0t_64_sync_handler+0x70/0x100 > el0t_64_sync+0x170/0x178 > Code: 8b080008 f2dffbe9 d350fd08 8b081928 (f9400509) > > Log against c7991b44d7b4 ("vsock/virtio: Allocate nonlinear SKBs for > handling large transmit buffers") > > Unable to handle kernel paging request at virtual address 0010502f8f8f4f08 > Mem abort info: > ESR = 0x0000000096000004 > EC = 0x25: DABT (current EL), IL = 32 bits > SET = 0, FnV = 0 > EA = 0, S1PTW = 0 > FSC = 0x04: level 0 translation fault > Data abort info: > ISV = 0, ISS = 0x00000004, ISS2 = 0x00000000 > CM = 0, WnR = 0, TnD = 0, TagAccess = 0 > GCS = 0, Overlay = 0, DirtyBit = 0, Xs = 0 > [0010502f8f8f4f08] address between user and kernel address ranges > Internal error: Oops: 0000000096000004 [#1] SMP > Modules linked in: vhost_vsock vfio_iommu_type1 vfio md4 crc32_cryptoapi ghash_generic vhost_net tun vhost vhost_iotlb tap mpls_gso mpls_iptunnel mpls_router fou sch_fq ghes_edac tls tcp_diag inet_diag act_gact cls_bpf ipmi_s > CPU: 47 UID: 0 PID: 1239699 Comm: stress-ng-dev Kdump: loaded Tainted: G W 6.16.0-rc6-upstream-00035-gc7991b44d7b4 #18 NONE > Tainted: [W]=WARN > pstate: 23401009 (nzCv daif +PAN -UAO +TCO +DIT +SSBS BTYPE=--) > pc : kfree+0x48/0x2a8 > lr : vhost_dev_cleanup+0x138/0x2b8 [vhost] > sp : ffff80016c0cfcd0 > x29: ffff80016c0cfcd0 x28: ffff001ad6210d80 x27: 0000000000000000 > x26: 0000000000000000 x25: 0000000000000000 x24: 0000000000000000 > x23: 00000000040e001f x22: ffffffffffffffff x21: ffff001bb76f00c0 > x20: 0000000000000000 x19: ffff001bb76f0000 x18: 0000000000000000 > x17: 0000000000000000 x16: 0000000000000000 x15: 0000000000000000 > x14: 000000000000001f x13: 000000000000000f x12: 0000000000000001 > x11: 0000000000000000 x10: 0000000000000402 x9 : ffffffdfc0000000 > x8 : 0010502f8f8f4f00 x7 : 0000000000000000 x6 : 0000000000000000 > x5 : ffff00012e7e2128 x4 : 0000000000000000 x3 : 0000000000000008 > x2 : ffffffffffffffff x1 : ffffffffffffffff x0 : 41403f3e3d3c3b3a > Call trace: > kfree+0x48/0x2a8 (P) > vhost_dev_cleanup+0x138/0x2b8 [vhost] > vhost_net_release+0xa0/0x1a8 [vhost_net] > __fput+0xfc/0x2f0 > fput_close_sync+0x38/0xc8 > __arm64_sys_close+0xb4/0x108 > invoke_syscall+0x4c/0xd0 > do_el0_svc+0x80/0xb0 > el0_svc+0x3c/0xd0 > el0t_64_sync_handler+0x70/0x100 > el0t_64_sync+0x170/0x178 > Code: 8b080008 f2dffbe9 d350fd08 8b081928 (f9400509) > > > > Breno -- when you say you've been seeing this "for a while", what's the > > earliest kernel you know you saw it on? > > Looking at my logs, the older kernel that I saw it was net-next from > 20250717, which was around the time I decided to test net-next in > preparation for 6.17, so, not very helpful. Sorry. > ^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: vhost: linux-next: crash at vhost_dev_cleanup() 2025-07-24 12:52 ` Stefano Garzarella @ 2025-07-24 13:49 ` Breno Leitao 2025-07-29 7:44 ` Jason Wang 0 siblings, 1 reply; 12+ messages in thread From: Breno Leitao @ 2025-07-24 13:49 UTC (permalink / raw) To: Stefano Garzarella Cc: Will Deacon, Michael S. Tsirkin, jasowang, eperezma, linux-arm-kernel, kvm, Stefan Hajnoczi, netdev On Thu, Jul 24, 2025 at 02:52:08PM +0200, Stefano Garzarella wrote: > On Thu, 24 Jul 2025 at 14:48, Breno Leitao <leitao@debian.org> wrote: > > > > On Thu, Jul 24, 2025 at 09:44:38AM +0100, Will Deacon wrote: > > > > > On Thu, 24 Jul 2025 at 09:48, Michael S. Tsirkin <mst@redhat.com> wrote: > > > > > > > > > > > > On Wed, Jul 23, 2025 at 08:04:42AM -0700, Breno Leitao wrote: > > > > > > > Hello, > > > > > > > > > > > > > > I've seen a crash in linux-next for a while on my arm64 server, and > > > > > > > I decided to report. > > > > > > > > > > > > > > While running stress-ng on linux-next, I see the crash below. > > > > > > > > > > > > > > This is happening in a kernel configure with some debug options (KASAN, > > > > > > > LOCKDEP and KMEMLEAK). > > > > > > > > > > > > > > Basically running stress-ng in a loop would crash the host in 15-20 > > > > > > > minutes: > > > > > > > # while (true); do stress-ng -r 10 -t 10; done > > > > > > > > > > > > > > >From the early warning "virt_to_phys used for non-linear address", > > > > > > > > > > mmm, we recently added nonlinear SKBs support in vhost-vsock [1], > > > > > @Will can this issue be related? > > > > > > > > Good point. > > > > > > > > Breno, if bisecting is too much trouble, would you mind testing the commits > > > > c76f3c4364fe523cd2782269eab92529c86217aa > > > > and > > > > c7991b44d7b44f9270dec63acd0b2965d29aab43 > > > > and telling us if this reproduces? > > > > > > That's definitely worth doing, but we should be careful not to confuse > > > the "non-linear address" from the warning (which refers to virtual > > > addresses that lie outside of the linear mapping of memory, e.g. in the > > > vmalloc space) and "non-linear SKBs" which refer to SKBs with fragment > > > pages. > > > > I've tested both commits above, and I see the crash on both commits > > above, thus, the problem reproduces in both cases. The only difference > > I noted is the fact that I haven't seen the warning before the crash. > > > > > > Log against c76f3c4364fe ("vhost/vsock: Avoid allocating > > arbitrarily-sized SKBs") > > > > Unable to handle kernel paging request at virtual address 0000001fc0000048 > > Mem abort info: > > ESR = 0x0000000096000005 > > EC = 0x25: DABT (current EL), IL = 32 bits > > SET = 0, FnV = 0 > > EA = 0, S1PTW = 0 > > FSC = 0x05: level 1 translation fault > > Data abort info: > > ISV = 0, ISS = 0x00000005, ISS2 = 0x00000000 > > CM = 0, WnR = 0, TnD = 0, TagAccess = 0 > > GCS = 0, Overlay = 0, DirtyBit = 0, Xs = 0 > > user pgtable: 64k pages, 48-bit VAs, pgdp=0000000cdcf2da00 > > [0000001fc0000048] pgd=0000000000000000, p4d=0000000000000000, pud=0000000000000000 > > Internal error: Oops: 0000000096000005 [#1] SMP > > Modules linked in: vfio_iommu_type1 vfio md4 crc32_cryptoapi ghash_generic unix_diag vhost_net tun vhost vhost_iotlb tap mpls_gso mpls_iptunnel mpls_router fou sch_fq ghes_edac tls tcp_diag inet_diag act_gact cls_bpf nvidia_c > > CPU: 34 UID: 0 PID: 1727297 Comm: stress-ng-dev Kdump: loaded Not tainted 6.16.0-rc6-upstream-00027-gc76f3c4364fe #19 NONE > > pstate: 23401009 (nzCv daif +PAN -UAO +TCO +DIT +SSBS BTYPE=--) > > pc : kfree+0x48/0x2a8 > > lr : vhost_dev_cleanup+0x138/0x2b8 [vhost] > > sp : ffff80013a0cfcd0 > > x29: ffff80013a0cfcd0 x28: ffff0008fd0b6240 x27: 0000000000000000 > > x26: 0000000000000000 x25: 0000000000000000 x24: 0000000000000000 > > x23: 00000000040e001f x22: ffffffffffffffff x21: ffff00014f1d4ac0 > > x20: 0000000000000001 x19: ffff00014f1d0000 x18: 0000000000000000 > > x17: 0000000000000000 x16: 0000000000000000 x15: 0000000000000000 > > x14: 000000000000001f x13: 000000000000000f x12: 0000000000000001 > > x11: 0000000000000000 x10: 0000000000000402 x9 : ffffffdfc0000000 > > x8 : 0000001fc0000040 x7 : 0000000000000000 x6 : 0000000000000000 > > x5 : ffff000141931840 x4 : 0000000000000000 x3 : 0000000000000008 > > x2 : ffffffffffffffff x1 : ffffffffffffffff x0 : 0000000000010000 > > Call trace: > > kfree+0x48/0x2a8 (P) > > vhost_dev_cleanup+0x138/0x2b8 [vhost] > > vhost_net_release+0xa0/0x1a8 [vhost_net] > > But here is the vhost_net, so I'm confused now. > Do you see the same (vhost_net) also on 9798752 ("Add linux-next > specific files for 20250721") ? I will need to reproduce, but, looking at my logs, I see the following against: c76f3c4364fe ("vhost/vsock: Avoid allocating arbitrarily-sized SKBs"). The logs are a bit intermixed, probably there were multiple CPUs hitting the same code path. virt_to_phys used for non-linear address: 000000001b662678 (0xffe61984a460) WARNING: CPU: 15 PID: 112846 at arch/arm64/mm/physaddr.c:15 __virt_to_phys+0x80/0xa8 Modules linked in: vhost_vsock(E) vhost(E) vhost_iotlb(E) ghes_edac(E) tls(E) act_gact(E) cls_bpf(E) tcp_diag(E) inet_diag(E) ipmi_ssif(E) ipmi_devintf(E) ipmi_msghandler(E) sch_fq_codel(E) drm(E) backlight(E) drm_panel_orientation_quirks(E) acpi_power_meter(E) loop(E) efivarfs(E) autofs4(E) CPU: 15 UID: 0 PID: 112846 Comm: stress-ng-dev Kdump: loaded Tainted: G W E N 6.16.0-rc6-upstream-00027-gc76f3c4364fe #16 PREEMPT(none) Tainted: [W]=WARN, [E]=UNSIGNED_MODULE, [N]=TEST pstate: 63401009 (nZCv daif +PAN -UAO +TCO +DIT +SSBS BTYPE=--) pc : __virt_to_phys+0x80/0xa8 lr : __virt_to_phys+0x7c/0xa8 sp : ffff8001184d7a30 x29: ffff8001184d7a30 x28: 00000000000045d8 x27: 1fffe000e7e88014 x26: 1fffe000e7e888f7 x25: ffff0007e578bf00 x24: 1fffe000e7e88013 x23: 0000000000000000 x22: 0000ffe61984a460 x21: ffff00073f440098 x20: ffffff1000080000 x19: 0000ffe61984a460 x18: 0000000000000002 x17: 6666783028203837 x16: 3632363662313030 x15: 0000000000000001 x14: 1fffe006d52e90f2 x13: 0000000000000000 x12: 0000000000000000 x11: ffff6006d52e90f3 x10: 0000000000000002 x9 : cfc659a21c727d00 x8 : ffff800083c19000 x7 : 0000000000000001 x6 : 0000000000000001 x5 : ffff8001184d7398 x4 : ffff800084866d60 x3 : ffff8000805fdd94 x2 : 0000000000000001 x1 : 0000000000000004 x0 : 000000000000004b Call trace: __virt_to_phys+0x80/0xa8 (P) kfree+0xac/0x4b0 vhost_dev_cleanup+0x484/0x8b0 [vhost] vhost_vsock_dev_release+0x2f4/0x358 [vhost_vsock] __fput+0x2b4/0x608 fput_close_sync+0xe8/0x1e0 __arm64_sys_close+0x84/0xd0 invoke_syscall+0x8c/0x208 do_el0_svc+0x128/0x1a0 el0_svc+0x58/0x160 el0t_64_sync_handler+0x78/0x108 el0t_64_sync+0x198/0x1a0 irq event stamp: 0 hardirqs last enabled at (0): [<0000000000000000>] 0x0 hardirqs last disabled at (0): [<ffff8000801d876c>] copy_process+0xd5c/0x29f8 softirqs last enabled at (0): [<ffff8000801d879c>] copy_process+0xd8c/0x29f8 softirqs last disabled at (0): [<0000000000000000>] 0x0 ---[ end trace 0000000000000000 ]--- Unable to handle kernel paging request at virtual address 0000040053791288 ------------[ cut here ]------------ lr : kfree+0xac/0x4b0 virt_to_phys used for non-linear address: 00000000290839fd (0x2500000000) WARNING: CPU: 41 PID: 112845 at arch/arm64/mm/physaddr.c:15 __virt_to_phys+0x80/0xa8 Modules linked in: vhost_vsock(E) vhost(E) vhost_iotlb(E) ghes_edac(E) tls(E) act_gact(E) cls_bpf(E) tcp_diag(E) inet_diag(E) ipmi_ssif(E) ipmi_devintf(E) ipmi_msghandler(E) sch_fq_codel(E) drm(E) backlight(E) drm_panel_orientation_quirks(E) acpi_power_meter(E) loop(E) efivarfs(E) autofs4(E) CPU: 41 UID: 0 PID: 112845 Comm: stress-ng-dev Kdump: loaded Tainted: G W E N 6.16.0-rc6-upstream-00027-gc76f3c4364fe #16 PREEMPT(none) x23: 0000000000000001 Tainted: [W]=WARN, [E]=UNSIGNED_MODULE, [N]=TEST pstate: 63401009 (nZCv daif +PAN -UAO +TCO +DIT +SSBS BTYPE=--) pc : __virt_to_phys+0x80/0xa8 lr : __virt_to_phys+0x7c/0xa8 sp : ffff8001a8277a30 x29: ffff8001a8277a30 x28: 00000000000045d8 x27: 1fffe000e7e3c014 x26: 1fffe000e7e3c8f7 x25: ffff0007bcff0000 x24: 1fffe000e7e3c013 x23: 0000000000000000 x22: 0000002500000000 x21: ffff00073f1e0098 x20: ffffff1000080000 x19: 0000002500000000 x18: 0000000000000004 x17: 00000000ffffffff x16: 0000000000000001 x15: 0000000000000001 x14: 1fffe006d53920f2 x13: 0000000000000000 x12: 0000000000000000 sp : ffff8001184d7a50 x29: ffff8001184d7a60 x28: 00000000000045d8 x27: 1fffe000e7e88014 x26: 1fffe000e7e888f7 x25: ffff0007e578bf00 x24: 1fffe000e7e88013 x23: 0000000000000000 x22: 0000ffe61984a460 x21: ffff00073f440098 x20: 0000040053791280 x19: ffff80000d0b8bbc x18: 0000000000000002 x17: 6666783028203837 x16: 3632363662313030 x15: 0000000000000001 x22: 0000ffe6199a459d x14: 1fffe006d52e90f2 x21: ffff00073f7f0098 x20: ffffff1000080000 x19: 0000ffe6199a459d x18: 0000000000000004 x17: 54455320203b2d2c x16: 0000000000000011 x15: 0000000000000001 x14: 1fffe006d52bb8f2 x13: 0000000000000000 x12: 0000000000000000 x11: ffff6006d52bb8f3 x10: dfff800000000000 x9 : 77521a2bd3e0be00 x8 : ffff800083c19000 x7 : 0000000000000000 x6 : ffff80008036bc2c x5 : 0000000000000000 x4 : 0000000000000001 x3 : ffff8000805fdd94 x2 : 0000000000000001 x1 : 0000000000000004 x0 : 000000000000004b Call trace: __virt_to_phys+0x80/0xa8 (P) kfree+0xac/0x4b0 vhost_dev_cleanup+0x484/0x8b0 [vhost] vhost_vsock_dev_release+0x2f4/0x358 [vhost_vsock] __fput+0x2b4/0x608 x11: ffff6006d53920f3 fput_close_sync+0xe8/0x1e0 __arm64_sys_close+0x84/0xd0 invoke_syscall+0x8c/0x208 do_el0_svc+0x128/0x1a0 el0_svc+0x58/0x160 el0t_64_sync_handler+0x78/0x108 el0t_64_sync+0x198/0x1a0 irq event stamp: 0 x13: 0000000000000000 hardirqs last enabled at (0): [<0000000000000000>] 0x0 hardirqs last disabled at (0): [<ffff8000801d876c>] copy_process+0xd5c/0x29f8 > The initial report contained only vhost_vsock traces IIUC, so I'm > suspecting something in the vhost core. Right, it seems we are hitting the same code path, on on both vhost_vsock and vhost_net. ^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: vhost: linux-next: crash at vhost_dev_cleanup() 2025-07-24 13:49 ` Breno Leitao @ 2025-07-29 7:44 ` Jason Wang 2025-07-29 9:10 ` Breno Leitao 2025-07-29 9:57 ` Stefano Garzarella 0 siblings, 2 replies; 12+ messages in thread From: Jason Wang @ 2025-07-29 7:44 UTC (permalink / raw) To: Breno Leitao Cc: Stefano Garzarella, Will Deacon, Michael S. Tsirkin, eperezma, linux-arm-kernel, kvm, Stefan Hajnoczi, netdev On Thu, Jul 24, 2025 at 9:50 PM Breno Leitao <leitao@debian.org> wrote: > > On Thu, Jul 24, 2025 at 02:52:08PM +0200, Stefano Garzarella wrote: > > On Thu, 24 Jul 2025 at 14:48, Breno Leitao <leitao@debian.org> wrote: > > > > > > On Thu, Jul 24, 2025 at 09:44:38AM +0100, Will Deacon wrote: > > > > > > On Thu, 24 Jul 2025 at 09:48, Michael S. Tsirkin <mst@redhat.com> wrote: > > > > > > > > > > > > > > On Wed, Jul 23, 2025 at 08:04:42AM -0700, Breno Leitao wrote: > > > > > > > > Hello, > > > > > > > > > > > > > > > > I've seen a crash in linux-next for a while on my arm64 server, and > > > > > > > > I decided to report. > > > > > > > > > > > > > > > > While running stress-ng on linux-next, I see the crash below. > > > > > > > > > > > > > > > > This is happening in a kernel configure with some debug options (KASAN, > > > > > > > > LOCKDEP and KMEMLEAK). > > > > > > > > > > > > > > > > Basically running stress-ng in a loop would crash the host in 15-20 > > > > > > > > minutes: > > > > > > > > # while (true); do stress-ng -r 10 -t 10; done > > > > > > > > > > > > > > > > >From the early warning "virt_to_phys used for non-linear address", > > > > > > > > > > > > mmm, we recently added nonlinear SKBs support in vhost-vsock [1], > > > > > > @Will can this issue be related? > > > > > > > > > > Good point. > > > > > > > > > > Breno, if bisecting is too much trouble, would you mind testing the commits > > > > > c76f3c4364fe523cd2782269eab92529c86217aa > > > > > and > > > > > c7991b44d7b44f9270dec63acd0b2965d29aab43 > > > > > and telling us if this reproduces? > > > > > > > > That's definitely worth doing, but we should be careful not to confuse > > > > the "non-linear address" from the warning (which refers to virtual > > > > addresses that lie outside of the linear mapping of memory, e.g. in the > > > > vmalloc space) and "non-linear SKBs" which refer to SKBs with fragment > > > > pages. > > > > > > I've tested both commits above, and I see the crash on both commits > > > above, thus, the problem reproduces in both cases. The only difference > > > I noted is the fact that I haven't seen the warning before the crash. > > > > > > > > > Log against c76f3c4364fe ("vhost/vsock: Avoid allocating > > > arbitrarily-sized SKBs") > > > > > > Unable to handle kernel paging request at virtual address 0000001fc0000048 > > > Mem abort info: > > > ESR = 0x0000000096000005 > > > EC = 0x25: DABT (current EL), IL = 32 bits > > > SET = 0, FnV = 0 > > > EA = 0, S1PTW = 0 > > > FSC = 0x05: level 1 translation fault > > > Data abort info: > > > ISV = 0, ISS = 0x00000005, ISS2 = 0x00000000 > > > CM = 0, WnR = 0, TnD = 0, TagAccess = 0 > > > GCS = 0, Overlay = 0, DirtyBit = 0, Xs = 0 > > > user pgtable: 64k pages, 48-bit VAs, pgdp=0000000cdcf2da00 > > > [0000001fc0000048] pgd=0000000000000000, p4d=0000000000000000, pud=0000000000000000 > > > Internal error: Oops: 0000000096000005 [#1] SMP > > > Modules linked in: vfio_iommu_type1 vfio md4 crc32_cryptoapi ghash_generic unix_diag vhost_net tun vhost vhost_iotlb tap mpls_gso mpls_iptunnel mpls_router fou sch_fq ghes_edac tls tcp_diag inet_diag act_gact cls_bpf nvidia_c > > > CPU: 34 UID: 0 PID: 1727297 Comm: stress-ng-dev Kdump: loaded Not tainted 6.16.0-rc6-upstream-00027-gc76f3c4364fe #19 NONE > > > pstate: 23401009 (nzCv daif +PAN -UAO +TCO +DIT +SSBS BTYPE=--) > > > pc : kfree+0x48/0x2a8 > > > lr : vhost_dev_cleanup+0x138/0x2b8 [vhost] > > > sp : ffff80013a0cfcd0 > > > x29: ffff80013a0cfcd0 x28: ffff0008fd0b6240 x27: 0000000000000000 > > > x26: 0000000000000000 x25: 0000000000000000 x24: 0000000000000000 > > > x23: 00000000040e001f x22: ffffffffffffffff x21: ffff00014f1d4ac0 > > > x20: 0000000000000001 x19: ffff00014f1d0000 x18: 0000000000000000 > > > x17: 0000000000000000 x16: 0000000000000000 x15: 0000000000000000 > > > x14: 000000000000001f x13: 000000000000000f x12: 0000000000000001 > > > x11: 0000000000000000 x10: 0000000000000402 x9 : ffffffdfc0000000 > > > x8 : 0000001fc0000040 x7 : 0000000000000000 x6 : 0000000000000000 > > > x5 : ffff000141931840 x4 : 0000000000000000 x3 : 0000000000000008 > > > x2 : ffffffffffffffff x1 : ffffffffffffffff x0 : 0000000000010000 > > > Call trace: > > > kfree+0x48/0x2a8 (P) > > > vhost_dev_cleanup+0x138/0x2b8 [vhost] > > > vhost_net_release+0xa0/0x1a8 [vhost_net] > > > > But here is the vhost_net, so I'm confused now. > > Do you see the same (vhost_net) also on 9798752 ("Add linux-next > > specific files for 20250721") ? > > I will need to reproduce, but, looking at my logs, I see the following > against: c76f3c4364fe ("vhost/vsock: Avoid allocating arbitrarily-sized SKBs"). > The logs are a bit intermixed, probably there were multiple CPUs hitting > the same code path. > > virt_to_phys used for non-linear address: 000000001b662678 (0xffe61984a460) > WARNING: CPU: 15 PID: 112846 at arch/arm64/mm/physaddr.c:15 __virt_to_phys+0x80/0xa8 > Modules linked in: vhost_vsock(E) vhost(E) vhost_iotlb(E) ghes_edac(E) tls(E) act_gact(E) cls_bpf(E) tcp_diag(E) inet_diag(E) ipmi_ssif(E) ipmi_devintf(E) ipmi_msghandler(E) sch_fq_codel(E) drm(E) backlight(E) drm_panel_orientation_quirks(E) acpi_power_meter(E) loop(E) efivarfs(E) autofs4(E) > CPU: 15 UID: 0 PID: 112846 Comm: stress-ng-dev Kdump: loaded Tainted: G W E N 6.16.0-rc6-upstream-00027-gc76f3c4364fe #16 PREEMPT(none) > Tainted: [W]=WARN, [E]=UNSIGNED_MODULE, [N]=TEST > pstate: 63401009 (nZCv daif +PAN -UAO +TCO +DIT +SSBS BTYPE=--) > pc : __virt_to_phys+0x80/0xa8 > lr : __virt_to_phys+0x7c/0xa8 > sp : ffff8001184d7a30 > x29: ffff8001184d7a30 x28: 00000000000045d8 x27: 1fffe000e7e88014 > x26: 1fffe000e7e888f7 x25: ffff0007e578bf00 x24: 1fffe000e7e88013 > x23: 0000000000000000 x22: 0000ffe61984a460 x21: ffff00073f440098 > x20: ffffff1000080000 x19: 0000ffe61984a460 x18: 0000000000000002 > x17: 6666783028203837 x16: 3632363662313030 x15: 0000000000000001 > x14: 1fffe006d52e90f2 x13: 0000000000000000 x12: 0000000000000000 > x11: ffff6006d52e90f3 x10: 0000000000000002 x9 : cfc659a21c727d00 > x8 : ffff800083c19000 x7 : 0000000000000001 x6 : 0000000000000001 > x5 : ffff8001184d7398 x4 : ffff800084866d60 x3 : ffff8000805fdd94 > x2 : 0000000000000001 x1 : 0000000000000004 x0 : 000000000000004b > Call trace: > __virt_to_phys+0x80/0xa8 (P) > kfree+0xac/0x4b0 > vhost_dev_cleanup+0x484/0x8b0 [vhost] > vhost_vsock_dev_release+0x2f4/0x358 [vhost_vsock] > __fput+0x2b4/0x608 > fput_close_sync+0xe8/0x1e0 > __arm64_sys_close+0x84/0xd0 > invoke_syscall+0x8c/0x208 > do_el0_svc+0x128/0x1a0 > el0_svc+0x58/0x160 > el0t_64_sync_handler+0x78/0x108 > el0t_64_sync+0x198/0x1a0 > irq event stamp: 0 > hardirqs last enabled at (0): [<0000000000000000>] 0x0 > hardirqs last disabled at (0): [<ffff8000801d876c>] copy_process+0xd5c/0x29f8 > softirqs last enabled at (0): [<ffff8000801d879c>] copy_process+0xd8c/0x29f8 > softirqs last disabled at (0): [<0000000000000000>] 0x0 > ---[ end trace 0000000000000000 ]--- > Unable to handle kernel paging request at virtual address 0000040053791288 > ------------[ cut here ]------------ > lr : kfree+0xac/0x4b0 > virt_to_phys used for non-linear address: 00000000290839fd (0x2500000000) > WARNING: CPU: 41 PID: 112845 at arch/arm64/mm/physaddr.c:15 __virt_to_phys+0x80/0xa8 > Modules linked in: vhost_vsock(E) vhost(E) vhost_iotlb(E) ghes_edac(E) tls(E) act_gact(E) cls_bpf(E) tcp_diag(E) inet_diag(E) ipmi_ssif(E) ipmi_devintf(E) ipmi_msghandler(E) sch_fq_codel(E) drm(E) backlight(E) drm_panel_orientation_quirks(E) acpi_power_meter(E) loop(E) efivarfs(E) autofs4(E) > CPU: 41 UID: 0 PID: 112845 Comm: stress-ng-dev Kdump: loaded Tainted: G W E N 6.16.0-rc6-upstream-00027-gc76f3c4364fe #16 PREEMPT(none) > x23: 0000000000000001 > Tainted: [W]=WARN, [E]=UNSIGNED_MODULE, [N]=TEST > pstate: 63401009 (nZCv daif +PAN -UAO +TCO +DIT +SSBS BTYPE=--) > pc : __virt_to_phys+0x80/0xa8 > lr : __virt_to_phys+0x7c/0xa8 > sp : ffff8001a8277a30 > x29: ffff8001a8277a30 x28: 00000000000045d8 x27: 1fffe000e7e3c014 > x26: 1fffe000e7e3c8f7 x25: ffff0007bcff0000 x24: 1fffe000e7e3c013 > x23: 0000000000000000 x22: 0000002500000000 x21: ffff00073f1e0098 > x20: ffffff1000080000 x19: 0000002500000000 x18: 0000000000000004 > x17: 00000000ffffffff x16: 0000000000000001 x15: 0000000000000001 > x14: 1fffe006d53920f2 x13: 0000000000000000 x12: 0000000000000000 > sp : ffff8001184d7a50 > x29: ffff8001184d7a60 x28: 00000000000045d8 x27: 1fffe000e7e88014 > x26: 1fffe000e7e888f7 x25: ffff0007e578bf00 x24: 1fffe000e7e88013 > x23: 0000000000000000 x22: 0000ffe61984a460 x21: ffff00073f440098 > x20: 0000040053791280 x19: ffff80000d0b8bbc x18: 0000000000000002 > x17: 6666783028203837 x16: 3632363662313030 x15: 0000000000000001 > x22: 0000ffe6199a459d > x14: 1fffe006d52e90f2 > x21: ffff00073f7f0098 > x20: ffffff1000080000 x19: 0000ffe6199a459d x18: 0000000000000004 > x17: 54455320203b2d2c x16: 0000000000000011 x15: 0000000000000001 > x14: 1fffe006d52bb8f2 x13: 0000000000000000 x12: 0000000000000000 > x11: ffff6006d52bb8f3 x10: dfff800000000000 x9 : 77521a2bd3e0be00 > x8 : ffff800083c19000 x7 : 0000000000000000 x6 : ffff80008036bc2c > x5 : 0000000000000000 x4 : 0000000000000001 x3 : ffff8000805fdd94 > x2 : 0000000000000001 x1 : 0000000000000004 x0 : 000000000000004b > Call trace: > __virt_to_phys+0x80/0xa8 (P) > kfree+0xac/0x4b0 > vhost_dev_cleanup+0x484/0x8b0 [vhost] > vhost_vsock_dev_release+0x2f4/0x358 [vhost_vsock] > __fput+0x2b4/0x608 > x11: ffff6006d53920f3 > fput_close_sync+0xe8/0x1e0 > __arm64_sys_close+0x84/0xd0 > invoke_syscall+0x8c/0x208 > do_el0_svc+0x128/0x1a0 > el0_svc+0x58/0x160 > el0t_64_sync_handler+0x78/0x108 > el0t_64_sync+0x198/0x1a0 > irq event stamp: 0 > x13: 0000000000000000 > hardirqs last enabled at (0): [<0000000000000000>] 0x0 > hardirqs last disabled at (0): [<ffff8000801d876c>] copy_process+0xd5c/0x29f8 > > > > The initial report contained only vhost_vsock traces IIUC, so I'm > > suspecting something in the vhost core. > > Right, it seems we are hitting the same code path, on on both > vhost_vsock and vhost_net. > I've posted a fix here: https://lore.kernel.org/virtualization/20250729073916.80647-1-jasowang@redhat.com/T/#u I think it should address this issue. Thanks ^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: vhost: linux-next: crash at vhost_dev_cleanup() 2025-07-29 7:44 ` Jason Wang @ 2025-07-29 9:10 ` Breno Leitao 2025-07-29 9:57 ` Stefano Garzarella 1 sibling, 0 replies; 12+ messages in thread From: Breno Leitao @ 2025-07-29 9:10 UTC (permalink / raw) To: Jason Wang Cc: Stefano Garzarella, Will Deacon, Michael S. Tsirkin, eperezma, linux-arm-kernel, kvm, Stefan Hajnoczi, netdev Hello Jason, On Tue, Jul 29, 2025 at 03:44:49PM +0800, Jason Wang wrote: > On Thu, Jul 24, 2025 at 9:50 PM Breno Leitao <leitao@debian.org> wrote: > > > The initial report contained only vhost_vsock traces IIUC, so I'm > > > suspecting something in the vhost core. > > > > Right, it seems we are hitting the same code path, on on both > > vhost_vsock and vhost_net. > > > > I've posted a fix here: > > https://lore.kernel.org/virtualization/20250729073916.80647-1-jasowang@redhat.com/T/#u > > I think it should address this issue. yes, it does. I've tested the fix on my machine and I was not able to reproduce the error at all. Thanks for the fix, --breno ^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: vhost: linux-next: crash at vhost_dev_cleanup() 2025-07-29 7:44 ` Jason Wang 2025-07-29 9:10 ` Breno Leitao @ 2025-07-29 9:57 ` Stefano Garzarella 1 sibling, 0 replies; 12+ messages in thread From: Stefano Garzarella @ 2025-07-29 9:57 UTC (permalink / raw) To: Jason Wang Cc: Breno Leitao, Will Deacon, Michael S. Tsirkin, eperezma, linux-arm-kernel, kvm, Stefan Hajnoczi, netdev On Tue, Jul 29, 2025 at 03:44:49PM +0800, Jason Wang wrote: >On Thu, Jul 24, 2025 at 9:50 PM Breno Leitao <leitao@debian.org> wrote: >> >> On Thu, Jul 24, 2025 at 02:52:08PM +0200, Stefano Garzarella wrote: >> > On Thu, 24 Jul 2025 at 14:48, Breno Leitao <leitao@debian.org> wrote: >> > > >> > > On Thu, Jul 24, 2025 at 09:44:38AM +0100, Will Deacon wrote: >> > > > > > On Thu, 24 Jul 2025 at 09:48, Michael S. Tsirkin <mst@redhat.com> wrote: >> > > > > > > >> > > > > > > On Wed, Jul 23, 2025 at 08:04:42AM -0700, Breno Leitao wrote: >> > > > > > > > Hello, >> > > > > > > > >> > > > > > > > I've seen a crash in linux-next for a while on my arm64 server, and >> > > > > > > > I decided to report. >> > > > > > > > >> > > > > > > > While running stress-ng on linux-next, I see the crash below. >> > > > > > > > >> > > > > > > > This is happening in a kernel configure with some debug options (KASAN, >> > > > > > > > LOCKDEP and KMEMLEAK). >> > > > > > > > >> > > > > > > > Basically running stress-ng in a loop would crash the host in 15-20 >> > > > > > > > minutes: >> > > > > > > > # while (true); do stress-ng -r 10 -t 10; done >> > > > > > > > >> > > > > > > > >From the early warning "virt_to_phys used for non-linear address", >> > > > > > >> > > > > > mmm, we recently added nonlinear SKBs support in vhost-vsock [1], >> > > > > > @Will can this issue be related? >> > > > > >> > > > > Good point. >> > > > > >> > > > > Breno, if bisecting is too much trouble, would you mind testing the commits >> > > > > c76f3c4364fe523cd2782269eab92529c86217aa >> > > > > and >> > > > > c7991b44d7b44f9270dec63acd0b2965d29aab43 >> > > > > and telling us if this reproduces? >> > > > >> > > > That's definitely worth doing, but we should be careful not to confuse >> > > > the "non-linear address" from the warning (which refers to virtual >> > > > addresses that lie outside of the linear mapping of memory, e.g. in the >> > > > vmalloc space) and "non-linear SKBs" which refer to SKBs with fragment >> > > > pages. >> > > >> > > I've tested both commits above, and I see the crash on both commits >> > > above, thus, the problem reproduces in both cases. The only difference >> > > I noted is the fact that I haven't seen the warning before the crash. >> > > >> > > >> > > Log against c76f3c4364fe ("vhost/vsock: Avoid allocating >> > > arbitrarily-sized SKBs") >> > > >> > > Unable to handle kernel paging request at virtual address 0000001fc0000048 >> > > Mem abort info: >> > > ESR = 0x0000000096000005 >> > > EC = 0x25: DABT (current EL), IL = 32 bits >> > > SET = 0, FnV = 0 >> > > EA = 0, S1PTW = 0 >> > > FSC = 0x05: level 1 translation fault >> > > Data abort info: >> > > ISV = 0, ISS = 0x00000005, ISS2 = 0x00000000 >> > > CM = 0, WnR = 0, TnD = 0, TagAccess = 0 >> > > GCS = 0, Overlay = 0, DirtyBit = 0, Xs = 0 >> > > user pgtable: 64k pages, 48-bit VAs, pgdp=0000000cdcf2da00 >> > > [0000001fc0000048] pgd=0000000000000000, p4d=0000000000000000, pud=0000000000000000 >> > > Internal error: Oops: 0000000096000005 [#1] SMP >> > > Modules linked in: vfio_iommu_type1 vfio md4 crc32_cryptoapi ghash_generic unix_diag vhost_net tun vhost vhost_iotlb tap mpls_gso mpls_iptunnel mpls_router fou sch_fq ghes_edac tls tcp_diag inet_diag act_gact cls_bpf nvidia_c >> > > CPU: 34 UID: 0 PID: 1727297 Comm: stress-ng-dev Kdump: loaded Not tainted 6.16.0-rc6-upstream-00027-gc76f3c4364fe #19 NONE >> > > pstate: 23401009 (nzCv daif +PAN -UAO +TCO +DIT +SSBS BTYPE=--) >> > > pc : kfree+0x48/0x2a8 >> > > lr : vhost_dev_cleanup+0x138/0x2b8 [vhost] >> > > sp : ffff80013a0cfcd0 >> > > x29: ffff80013a0cfcd0 x28: ffff0008fd0b6240 x27: 0000000000000000 >> > > x26: 0000000000000000 x25: 0000000000000000 x24: 0000000000000000 >> > > x23: 00000000040e001f x22: ffffffffffffffff x21: ffff00014f1d4ac0 >> > > x20: 0000000000000001 x19: ffff00014f1d0000 x18: 0000000000000000 >> > > x17: 0000000000000000 x16: 0000000000000000 x15: 0000000000000000 >> > > x14: 000000000000001f x13: 000000000000000f x12: 0000000000000001 >> > > x11: 0000000000000000 x10: 0000000000000402 x9 : ffffffdfc0000000 >> > > x8 : 0000001fc0000040 x7 : 0000000000000000 x6 : 0000000000000000 >> > > x5 : ffff000141931840 x4 : 0000000000000000 x3 : 0000000000000008 >> > > x2 : ffffffffffffffff x1 : ffffffffffffffff x0 : 0000000000010000 >> > > Call trace: >> > > kfree+0x48/0x2a8 (P) >> > > vhost_dev_cleanup+0x138/0x2b8 [vhost] >> > > vhost_net_release+0xa0/0x1a8 [vhost_net] >> > >> > But here is the vhost_net, so I'm confused now. >> > Do you see the same (vhost_net) also on 9798752 ("Add linux-next >> > specific files for 20250721") ? >> >> I will need to reproduce, but, looking at my logs, I see the following >> against: c76f3c4364fe ("vhost/vsock: Avoid allocating arbitrarily-sized SKBs"). >> The logs are a bit intermixed, probably there were multiple CPUs hitting >> the same code path. >> >> virt_to_phys used for non-linear address: 000000001b662678 (0xffe61984a460) >> WARNING: CPU: 15 PID: 112846 at arch/arm64/mm/physaddr.c:15 __virt_to_phys+0x80/0xa8 >> Modules linked in: vhost_vsock(E) vhost(E) vhost_iotlb(E) ghes_edac(E) tls(E) act_gact(E) cls_bpf(E) tcp_diag(E) inet_diag(E) ipmi_ssif(E) ipmi_devintf(E) ipmi_msghandler(E) sch_fq_codel(E) drm(E) backlight(E) drm_panel_orientation_quirks(E) acpi_power_meter(E) loop(E) efivarfs(E) autofs4(E) >> CPU: 15 UID: 0 PID: 112846 Comm: stress-ng-dev Kdump: loaded Tainted: G W E N 6.16.0-rc6-upstream-00027-gc76f3c4364fe #16 PREEMPT(none) >> Tainted: [W]=WARN, [E]=UNSIGNED_MODULE, [N]=TEST >> pstate: 63401009 (nZCv daif +PAN -UAO +TCO +DIT +SSBS BTYPE=--) >> pc : __virt_to_phys+0x80/0xa8 >> lr : __virt_to_phys+0x7c/0xa8 >> sp : ffff8001184d7a30 >> x29: ffff8001184d7a30 x28: 00000000000045d8 x27: 1fffe000e7e88014 >> x26: 1fffe000e7e888f7 x25: ffff0007e578bf00 x24: 1fffe000e7e88013 >> x23: 0000000000000000 x22: 0000ffe61984a460 x21: ffff00073f440098 >> x20: ffffff1000080000 x19: 0000ffe61984a460 x18: 0000000000000002 >> x17: 6666783028203837 x16: 3632363662313030 x15: 0000000000000001 >> x14: 1fffe006d52e90f2 x13: 0000000000000000 x12: 0000000000000000 >> x11: ffff6006d52e90f3 x10: 0000000000000002 x9 : cfc659a21c727d00 >> x8 : ffff800083c19000 x7 : 0000000000000001 x6 : 0000000000000001 >> x5 : ffff8001184d7398 x4 : ffff800084866d60 x3 : ffff8000805fdd94 >> x2 : 0000000000000001 x1 : 0000000000000004 x0 : 000000000000004b >> Call trace: >> __virt_to_phys+0x80/0xa8 (P) >> kfree+0xac/0x4b0 >> vhost_dev_cleanup+0x484/0x8b0 [vhost] >> vhost_vsock_dev_release+0x2f4/0x358 [vhost_vsock] >> __fput+0x2b4/0x608 >> fput_close_sync+0xe8/0x1e0 >> __arm64_sys_close+0x84/0xd0 >> invoke_syscall+0x8c/0x208 >> do_el0_svc+0x128/0x1a0 >> el0_svc+0x58/0x160 >> el0t_64_sync_handler+0x78/0x108 >> el0t_64_sync+0x198/0x1a0 >> irq event stamp: 0 >> hardirqs last enabled at (0): [<0000000000000000>] 0x0 >> hardirqs last disabled at (0): [<ffff8000801d876c>] copy_process+0xd5c/0x29f8 >> softirqs last enabled at (0): [<ffff8000801d879c>] copy_process+0xd8c/0x29f8 >> softirqs last disabled at (0): [<0000000000000000>] 0x0 >> ---[ end trace 0000000000000000 ]--- >> Unable to handle kernel paging request at virtual address 0000040053791288 >> ------------[ cut here ]------------ >> lr : kfree+0xac/0x4b0 >> virt_to_phys used for non-linear address: 00000000290839fd (0x2500000000) >> WARNING: CPU: 41 PID: 112845 at arch/arm64/mm/physaddr.c:15 __virt_to_phys+0x80/0xa8 >> Modules linked in: vhost_vsock(E) vhost(E) vhost_iotlb(E) ghes_edac(E) tls(E) act_gact(E) cls_bpf(E) tcp_diag(E) inet_diag(E) ipmi_ssif(E) ipmi_devintf(E) ipmi_msghandler(E) sch_fq_codel(E) drm(E) backlight(E) drm_panel_orientation_quirks(E) acpi_power_meter(E) loop(E) efivarfs(E) autofs4(E) >> CPU: 41 UID: 0 PID: 112845 Comm: stress-ng-dev Kdump: loaded Tainted: G W E N 6.16.0-rc6-upstream-00027-gc76f3c4364fe #16 PREEMPT(none) >> x23: 0000000000000001 >> Tainted: [W]=WARN, [E]=UNSIGNED_MODULE, [N]=TEST >> pstate: 63401009 (nZCv daif +PAN -UAO +TCO +DIT +SSBS BTYPE=--) >> pc : __virt_to_phys+0x80/0xa8 >> lr : __virt_to_phys+0x7c/0xa8 >> sp : ffff8001a8277a30 >> x29: ffff8001a8277a30 x28: 00000000000045d8 x27: 1fffe000e7e3c014 >> x26: 1fffe000e7e3c8f7 x25: ffff0007bcff0000 x24: 1fffe000e7e3c013 >> x23: 0000000000000000 x22: 0000002500000000 x21: ffff00073f1e0098 >> x20: ffffff1000080000 x19: 0000002500000000 x18: 0000000000000004 >> x17: 00000000ffffffff x16: 0000000000000001 x15: 0000000000000001 >> x14: 1fffe006d53920f2 x13: 0000000000000000 x12: 0000000000000000 >> sp : ffff8001184d7a50 >> x29: ffff8001184d7a60 x28: 00000000000045d8 x27: 1fffe000e7e88014 >> x26: 1fffe000e7e888f7 x25: ffff0007e578bf00 x24: 1fffe000e7e88013 >> x23: 0000000000000000 x22: 0000ffe61984a460 x21: ffff00073f440098 >> x20: 0000040053791280 x19: ffff80000d0b8bbc x18: 0000000000000002 >> x17: 6666783028203837 x16: 3632363662313030 x15: 0000000000000001 >> x22: 0000ffe6199a459d >> x14: 1fffe006d52e90f2 >> x21: ffff00073f7f0098 >> x20: ffffff1000080000 x19: 0000ffe6199a459d x18: 0000000000000004 >> x17: 54455320203b2d2c x16: 0000000000000011 x15: 0000000000000001 >> x14: 1fffe006d52bb8f2 x13: 0000000000000000 x12: 0000000000000000 >> x11: ffff6006d52bb8f3 x10: dfff800000000000 x9 : 77521a2bd3e0be00 >> x8 : ffff800083c19000 x7 : 0000000000000000 x6 : ffff80008036bc2c >> x5 : 0000000000000000 x4 : 0000000000000001 x3 : ffff8000805fdd94 >> x2 : 0000000000000001 x1 : 0000000000000004 x0 : 000000000000004b >> Call trace: >> __virt_to_phys+0x80/0xa8 (P) >> kfree+0xac/0x4b0 >> vhost_dev_cleanup+0x484/0x8b0 [vhost] >> vhost_vsock_dev_release+0x2f4/0x358 [vhost_vsock] >> __fput+0x2b4/0x608 >> x11: ffff6006d53920f3 >> fput_close_sync+0xe8/0x1e0 >> __arm64_sys_close+0x84/0xd0 >> invoke_syscall+0x8c/0x208 >> do_el0_svc+0x128/0x1a0 >> el0_svc+0x58/0x160 >> el0t_64_sync_handler+0x78/0x108 >> el0t_64_sync+0x198/0x1a0 >> irq event stamp: 0 >> x13: 0000000000000000 >> hardirqs last enabled at (0): [<0000000000000000>] 0x0 >> hardirqs last disabled at (0): [<ffff8000801d876c>] copy_process+0xd5c/0x29f8 >> >> >> > The initial report contained only vhost_vsock traces IIUC, so I'm >> > suspecting something in the vhost core. >> >> Right, it seems we are hitting the same code path, on on both >> vhost_vsock and vhost_net. >> > >I've posted a fix here: > >https://lore.kernel.org/virtualization/20250729073916.80647-1-jasowang@redhat.com/T/#u > >I think it should address this issue. Thanks for the fix! Stefano ^ permalink raw reply [flat|nested] 12+ messages in thread
end of thread, other threads:[~2025-07-29 10:01 UTC | newest] Thread overview: 12+ messages (download: mbox.gz follow: Atom feed -- links below jump to the message on this page -- 2025-07-23 15:04 vhost: linux-next: crash at vhost_dev_cleanup() Breno Leitao 2025-07-23 19:09 ` Michael S. Tsirkin 2025-07-24 7:47 ` Michael S. Tsirkin 2025-07-24 8:14 ` Stefano Garzarella 2025-07-24 8:22 ` Michael S. Tsirkin 2025-07-24 8:44 ` Will Deacon 2025-07-24 12:48 ` Breno Leitao 2025-07-24 12:52 ` Stefano Garzarella 2025-07-24 13:49 ` Breno Leitao 2025-07-29 7:44 ` Jason Wang 2025-07-29 9:10 ` Breno Leitao 2025-07-29 9:57 ` Stefano Garzarella
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for NNTP newsgroup(s).