From: "Longpeng (Mike)" <longpeng2@huawei.com>
To: Paolo Bonzini <pbonzini@redhat.com>
Cc: "Alex Williamson" <alex.williamson@redhat.com>,
kvm <kvm@vger.kernel.org>, "Radim Krčmář" <rkrcmar@redhat.com>,
"Huangweidong (C)" <weidong.huang@huawei.com>,
Gonglei <arei.gonglei@huawei.com>,
"wangxin (U)" <wangxinxin.wang@huawei.com>
Subject: Re: [help] host kernel panic in kvm's wakeup_handler()
Date: Tue, 6 Jun 2017 00:21:59 +0800 [thread overview]
Message-ID: <59358527.2090207@huawei.com> (raw)
In-Reply-To: <11b3f9a3-7c89-639e-84e0-219c9e376a66@redhat.com>
Hi Paolo,
We have a reproducer now, it says that the *blocked_vcpu_on_cpu* list is
corruption and double addition.
Do you have any suggestion?
[231298.241923] WARNING: at lib/list_debug.c:36 __list_add+0x8a/0xc0()
[231298.241925] list_add double add: new=ffff881b8bc48050,
prev=ffff881b8bc48050, next=ffff881fffa576f0.
[231298.241926] Modules linked in: guest_kbox_ram(O) igb(OVE) mlx4_ib(OVE)
ib_sa(OVE) ib_mad(OVE) mlx4_en(OVE) mlx4_core(OVE) ib_uverbs(OVE) vhost_scsi(OE)
target_core_pscsi target_core_file target_core_iblock target_core_mod dm_mod
kbox_pci(OVE) ib_core(OVE) ib_addr(OVE) ib_netlink(OVE) compat(OVE) ixgbe(O)
ext3 mbcache jbd signo_catch(O) bum(O) ip_set nfnetlink prio(O) nat(O)
vport_vxlan(O) openvswitch(O) nf_defrag_ipv6 gre libcrc32c kbox(O) pmcint(O)
vxlan ip6_udp_tunnel udp_tunnel sd_mod crc_t10dif crct10dif_generic sg
ipmi_devintf kvm_intel(O) kvm(O) coretemp crct10dif_pclmul crct10dif_common ahci
libahci mpt2sas i2c_i801 i2c_algo_bit libata dca i2c_core raid_class ptp
scsi_transport_sas pps_core ipmi_si ipmi_msghandler nf_conntrack_ipv4
nf_defrag_ipv4 nf_conntrack vhost_net(O) tun(O) vhost(O) macvtap
[231298.241986] macvlan vfio_pci irqbypass vfio_iommu_type1 vfio ip_tables
[last unloaded: guest_kbox_ram]
[231298.241994] CPU: 1 PID: 12431 Comm: CPU 0/KVM Tainted: G W OE
----V------- 3.10.0-327.49.58.52_13.x86_64 #1
[231298.241996] Hardware name: HUAWEI TECHNOLOGIES CO.,LTD. CH80GPUB8/CH80GPUB8,
BIOS GPUBV201 06/18/2015
[231298.241997] ffff881fa372fc60 00000000054b553c ffff881fa372fc18 ffffffff81644aaf
[231298.242002] ffff881fa372fc50 ffffffff8107b1c0 ffff881b8bc48050 ffff881fffa576f0
[231298.242006] ffff881b8bc48050 000000000000a022 000000000000001b ffff881fa372fcb8
[231298.242011] ffffffff8107b25c ffffffff818a9ce8 ffff881b00000030 ffff881fa372fcc8
[231298.242015] ffff881fa372fc88 00000000054b553c 0000000000000001 ffffffff8107b205
[231298.242020] ffffffff81a38960 ffff881b8bc48050 ffff881b8bc48050 ffff881fffa576f0
[231298.242025] ffff881fa372fce0 ffffffff8131a41a ffff881b8bc48000 00000000000176e0
[231298.242029] 0000000000000292 ffff881fa372fdd0 ffffffffa10de6d0 ffff881b8bc48050
[231298.242036] ffff881fa372ffd8 ffff881bb4c70000 ffff88176dfd8048 0000000000000001
[231298.242043] ffff881fa372fe18 ffffffff81656a31 ffffffffa10d9360 ffffffffa10fc140
[231298.242048] ffff881c23580100 0000000000000000 0000000000000000 ffff881b8bc48000
[231298.242052] ffff881fa372fd88 ffffffffa10dad65 0000000000000000 ffffffffa10d9360
[231298.242057] 00000000054b553c ffff881c23580200 0000000000000000 ffff881c23580000
[231298.242061] ffff881b8bc48000 ffff881fa372fdb8 ffff881b8bc48000 ffff881fa372ffd8
[231298.242066] ffff881bb4c70000 ffff88176dfd8048 0000000000000001 ffff881fa372fe18
[231298.242070] ffffffffa05ed1e8 ffffffee7ffbfaff 00000000054b553c ffff881b8bc48000
[231298.242075] ffff883fb857b600 0000000000000000 ffff881ae737dc00 ffff881bb4c70000
[231298.242079] ffff881fa372feb0 ffffffffa05d4b31 0000000000000000 0000000000008000
[231298.242084] ffff881fa372fe70 ffffffff8112f643 000000000000ffff ffff881ae737dc38
[231298.242088] ffffffffa05d4880 0000000000000000 0000000000000000 000000000000ae80
[231298.242093] ffff881ae737dc00 00000000054b553c ffff881ae737dc00 ffff883fd2a0a500
[231298.242097] 0000000000000000 0000000000000000 0000000000000001 ffff881fa372ff28
[231298.242102] ffffffff811fd9d5 000000000000ffff ffff881ae737dc38 0000000000000000
[231298.242106] 0000000000000000 000000000000ae80 0000000000000018 ffff881ae737dc00
[231298.242111] Call Trace:
[231298.242115] [<ffffffff81644aaf>] dump_stack+0x19/0x1b
[231298.242118] [<ffffffff8107b1c0>] warn_slowpath_common+0x70/0xb0
[231298.242122] [<ffffffff8107b25c>] warn_slowpath_fmt+0x5c/0x80
[231298.242126] [<ffffffff8107b205>] ? warn_slowpath_fmt+0x5/0x80
[231298.242130] [<ffffffff8131a41a>] __list_add+0x8a/0xc0
[231298.242136] [<ffffffffa10de6d0>] vmx_pre_block+0xe0/0x220 [kvm_intel]
[231298.242140] [<ffffffff81656a31>] ? ftrace_call+0x5/0x2f
[231298.242145] [<ffffffffa10d9360>] ? vmx_invpcid_supported+0x20/0x20 [kvm_intel]
[231298.242151] [<ffffffffa10dad65>] ? vmx_sync_pir_to_irr+0x5/0x30 [kvm_intel]
[231298.242156] [<ffffffffa10d9360>] ? vmx_invpcid_supported+0x20/0x20 [kvm_intel]
[231298.242167] [<ffffffffa05ed1e8>] kvm_arch_vcpu_ioctl_run+0x178/0x440 [kvm]
[231298.242176] [<ffffffffa05d4b31>] kvm_vcpu_ioctl+0x2b1/0x640 [kvm]
[231298.242180] [<ffffffff8112f643>] ? ftrace_ops_list_func+0x83/0x110
[231298.242189] [<ffffffffa05d4880>] ? vcpu_put+0x30/0x30 [kvm]
[231298.242193] [<ffffffff811fd9d5>] do_vfs_ioctl+0x2e5/0x4c0
[231298.242197] [<ffffffff811fdc51>] SyS_ioctl+0xa1/0xc0
[231298.242201] [<ffffffff81654e09>] system_call_fastpath+0x16/0x1b
[231298.245626] WARNING: at lib/list_debug.c:33 __list_add+0xac/0xc0()
[231298.245628] list_add corruption. prev->next should be next
(ffff881fffa576f0), but was dead000000100100. (prev=ffff881b8bc48050).
[231298.245629] Modules linked in: guest_kbox_ram(O) igb(OVE) mlx4_ib(OVE)
ib_sa(OVE) ib_mad(OVE) mlx4_en(OVE) mlx4_core(OVE) ib_uverbs(OVE) vhost_scsi(OE)
target_core_pscsi target_core_file target_core_iblock target_core_mod dm_mod
kbox_pci(OVE) ib_core(OVE) ib_addr(OVE) ib_netlink(OVE) compat(OVE) ixgbe(O)
ext3 mbcache jbd signo_catch(O) bum(O) ip_set nfnetlink prio(O) nat(O)
vport_vxlan(O) openvswitch(O) nf_defrag_ipv6 gre libcrc32c kbox(O) pmcint(O)
vxlan ip6_udp_tunnel udp_tunnel sd_mod crc_t10dif crct10dif_generic sg
ipmi_devintf kvm_intel(O) kvm(O) coretemp crct10dif_pclmul crct10dif_common ahci
libahci mpt2sas i2c_i801 i2c_algo_bit libata dca i2c_core raid_class ptp
scsi_transport_sas pps_core ipmi_si ipmi_msghandler nf_conntrack_ipv4
nf_defrag_ipv4 nf_conntrack vhost_net(O) tun(O) vhost(O) macvtap
[231298.245711] macvlan vfio_pci irqbypass vfio_iommu_type1 vfio ip_tables
[last unloaded: guest_kbox_ram]
[231298.245725] CPU: 1 PID: 12431 Comm: CPU 0/KVM Tainted: G W OE
----V------- 3.10.0-327.49.58.52_13.x86_64 #1
[231298.245729] Hardware name: HUAWEI TECHNOLOGIES CO.,LTD. CH80GPUB8/CH80GPUB8,
BIOS GPUBV201 06/18/2015
[231298.245732] ffff881fa372fc60 00000000054b553c ffff881fa372fc18 ffffffff81644aaf
[231298.245740] ffff881fa372fc50 ffffffff8107b1c0 ffff881b8bc48050 ffff881fffa576f0
[231298.245748] ffff881b8bc48050 000000000000a022 000000000000001b ffff881fa372fcb8
[231298.245756] ffffffff8107b25c ffffffff818a9c98 ffff881f00000030 ffff881fa372fcc8
[231298.245765] ffff881fa372fc88 00000000054b553c 0000000000000001 ffffffff8107b205
[231298.245773] ffffffff81a38960 ffff881fffa576f0 dead000000100100 ffff881b8bc48050
[231298.245781] ffff881fa372fce0 ffffffff8131a43c ffff881b8bc48000 00000000000176e0
[231298.245791] 0000000000000292 ffff881fa372fdd0 ffffffffa10de6d0 ffff881b8bc48050
[231298.245799] ffff881fa372ffd8 ffff881bb4c70000 ffff88176dfd8048 0000000000000001
[231298.245808] ffff881fa372fe18 ffffffff81656a31 ffffffffa10d9360 ffffffffa10fc140
[231298.245816] ffff881c23580100 0000000000000000 0000000000000000 ffff881b8bc48000
[231298.245826] ffff881fa372fd88 ffffffffa10dad65 0000000000000000 ffffffffa10d9360
[231298.245834] 00000000054b553c ffff881c23580200 0000000000000000 ffff881c23580000
[231298.245842] ffff881b8bc48000 ffff881fa372fdb8 ffff881b8bc48000 ffff881fa372ffd8
[231298.245847] ffff881bb4c70000 ffff88176dfd8048 0000000000000001 ffff881fa372fe18
[231298.245851] ffffffffa05ed1e8 ffffffee7ffbfaff 00000000054b553c ffff881b8bc48000
[231298.245856] ffff883fb857b600 0000000000000000 ffff881ae737dc00 ffff881bb4c70000
[231298.245861] ffff881fa372feb0 ffffffffa05d4b31 0000000000000000 0000000000008000
[231298.245866] ffff881fa372fe70 ffffffff8112f643 000000000000ffff ffff881ae737dc38
[231298.245870] ffffffffa05d4880 0000000000000000 0000000000000000 000000000000ae80
[231298.245875] ffff881ae737dc00 00000000054b553c ffff881ae737dc00 ffff883fd2a0a500
[231298.245879] 0000000000000000 0000000000000000 0000000000000001 ffff881fa372ff28
[231298.245883] ffffffff811fd9d5 000000000000ffff ffff881ae737dc38 0000000000000000
[231298.245888] 0000000000000000 000000000000ae80 0000000000000018 ffff881ae737dc00
[231298.245893] Call Trace:
[231298.245898] [<ffffffff81644aaf>] dump_stack+0x19/0x1b
[231298.245902] [<ffffffff8107b1c0>] warn_slowpath_common+0x70/0xb0
[231298.245906] [<ffffffff8107b25c>] warn_slowpath_fmt+0x5c/0x80
[231298.245910] [<ffffffff8107b205>] ? warn_slowpath_fmt+0x5/0x80
[231298.245913] [<ffffffff8131a43c>] __list_add+0xac/0xc0
[231298.245920] [<ffffffffa10de6d0>] vmx_pre_block+0xe0/0x220 [kvm_intel]
[231298.245924] [<ffffffff81656a31>] ? ftrace_call+0x5/0x2f
[231298.245930] [<ffffffffa10d9360>] ? vmx_invpcid_supported+0x20/0x20 [kvm_intel]
[231298.245936] [<ffffffffa10dad65>] ? vmx_sync_pir_to_irr+0x5/0x30 [kvm_intel]
[231298.245941] [<ffffffffa10d9360>] ? vmx_invpcid_supported+0x20/0x20 [kvm_intel]
[231298.245953] [<ffffffffa05ed1e8>] kvm_arch_vcpu_ioctl_run+0x178/0x440 [kvm]
[231298.245962] [<ffffffffa05d4b31>] kvm_vcpu_ioctl+0x2b1/0x640 [kvm]
[231298.245967] [<ffffffff8112f643>] ? ftrace_ops_list_func+0x83/0x110
[231298.245976] [<ffffffffa05d4880>] ? vcpu_put+0x30/0x30 [kvm]
[231298.245980] [<ffffffff811fd9d5>] do_vfs_ioctl+0x2e5/0x4c0
[231298.245985] [<ffffffff811fdc51>] SyS_ioctl+0xa1/0xc0
[231298.245989] [<ffffffff81654e09>] system_call_fastpath+0x16/0x1b
On 2017/5/26 18:40, Paolo Bonzini wrote:
>
>
> On 24/05/2017 07:04, Longpeng (Mike) wrote:
>>>> it crashed at *1ec1* and %rax get a wrong value(0xdead000000100100) at *1e92*,
>>>> it seems the *blocked_vcpu_on_cpu* list is corrupted, but kvm only access this
>>>> list in pre_block/post_block/wakeup_handler, and these three functions seems good.
>>>>
>>>> kvm version is 4.4-stable.
>>>>
>>>> Do you have any ideas? Any suggestion would be greatly appreciated, thanks!
>>>>
>>> Is this only seen with posted interrupt support enabled? Booting with
>>> intremap=nopost on the kernel commandline would disable it. Thanks,
>>
>> We tested with PI support enabled, but we not sure if it only occurs with PI
>> enabled yet.
>
> This code should not run at all with PI disabled, since the handler is
> only reachable through an IRTE.
>
> As you said, the list manipulation in those function is fairly simple.
> If you have a reproducer, you can try running it with CONFIG_LIST_DEBUG
> and see what you get.
>
> Thanks,
>
> Paolo
>
> .
>
--
Regards,
Longpeng(Mike)
prev parent reply other threads:[~2017-06-05 16:24 UTC|newest]
Thread overview: 6+ messages / expand[flat|nested] mbox.gz Atom feed top
2017-05-24 3:57 [help] host kernel panic in kvm's wakeup_handler() Longpeng (Mike)
2017-05-24 4:34 ` Alex Williamson
2017-05-24 5:04 ` Longpeng (Mike)
2017-05-26 10:40 ` Paolo Bonzini
2017-05-26 10:53 ` Longpeng (Mike)
2017-06-05 16:21 ` Longpeng (Mike) [this message]
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=59358527.2090207@huawei.com \
--to=longpeng2@huawei.com \
--cc=alex.williamson@redhat.com \
--cc=arei.gonglei@huawei.com \
--cc=kvm@vger.kernel.org \
--cc=pbonzini@redhat.com \
--cc=rkrcmar@redhat.com \
--cc=wangxinxin.wang@huawei.com \
--cc=weidong.huang@huawei.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.