3.10.X kernel/jump_label kvm

3.10.X kernel/jump_label kvm

[Stefan Priebe]

Hello,

i got this stack trace multiple times while using a vanilla 3.10.32 
kernel and already sent it to the list in december but got no replies.

[78136.551061] WARNING: at kernel/jump_label.c:80 
__static_key_slow_dec+0xb6/0xc0()
[78136.551062] jump label: negative count!
[78136.551063] Modules linked in: sch_htb act_police cls_u32 sch_ingress 
vhost_net tun macvtap macvlan netconsole ipt_REJECT dlm sctp 
iptable_filter ip_tables x_tables iscsi_tcp libiscsi_tcp libiscsi 
scsi_transport_iscsi nfsd auth_rpcgss oid_registry bonding ext2 8021q 
garp fuse mperf coretemp kvm_intel kvm crc32_pclmul ghash_clmulni_intel 
microcode i2c_i801 button dm_mod raid1 md_mod usbhid usb_storage 
ohci_hcd sg sd_mod ehci_pci ahci ehci_hcd igb libahci i2c_algo_bit isci 
usbcore i2c_core usb_common libsas ptp ixgbe(O) scsi_transport_sas pps_core
[78136.551080] CPU: 21 PID: 47183 Comm: kvm Tainted: G        W  O 
3.10.32+68-ph #1
[78136.551081] Hardware name: Supermicro 
X9DRW-3LN4F+/X9DRW-3TF+/X9DRW-3LN4F+/X9DRW-3TF+, BIOS 3.00 07/05/2013
[78136.551081]  0000000000000009 ffff882f4a669be8 ffffffff81524606 
ffff882f4a669c28
[78136.551085]  ffffffff8104853b 000000004a669c08 ffffffffa045cc40 
00000000000000fa
[78136.551088]  ffffffffa045cc60 ffff882f51460160 ffff882f74ab8110 
ffff882f4a669c88
[78136.551091] Call Trace:
[78136.551093]  [<ffffffff81524606>] dump_stack+0x19/0x1b
[78136.551095]  [<ffffffff8104853b>] warn_slowpath_common+0x6b/0xa0
[78136.551098]  [<ffffffff81048611>] warn_slowpath_fmt+0x41/0x50
[78136.551100]  [<ffffffff810e05f6>] __static_key_slow_dec+0xb6/0xc0
[78136.551102]  [<ffffffff810e0631>] static_key_slow_dec_deferred+0x11/0x20
[78136.551110]  [<ffffffffa043ff60>] kvm_free_lapic+0x90/0xa0 [kvm]
[78136.551116]  [<ffffffffa0429ef3>] kvm_arch_vcpu_uninit+0x23/0x90 [kvm]
[78136.551122]  [<ffffffffa0410a20>] kvm_vcpu_uninit+0x20/0x40 [kvm]
[78136.551125]  [<ffffffffa021fc12>] vmx_free_vcpu+0x52/0x70 [kvm_intel]
[78136.551132]  [<ffffffffa04295ef>] kvm_arch_vcpu_free+0x4f/0x60 [kvm]
[78136.551138]  [<ffffffffa042a112>] kvm_arch_destroy_vm+0xf2/0x1f0 [kvm]
[78136.551141]  [<ffffffff81071048>] ? synchronize_srcu+0x18/0x20
[78136.551143]  [<ffffffff8112677a>] ? mmu_notifier_unregister+0xaa/0xe0
[78136.551149]  [<ffffffffa041380e>] kvm_put_kvm+0x10e/0x1b0 [kvm]
[78136.551155]  [<ffffffffa0413a33>] kvm_vcpu_release+0x13/0x20 [kvm]
[78136.551157]  [<ffffffff811452d1>] __fput+0xe1/0x230
[78136.551160]  [<ffffffff81145429>] ____fput+0x9/0x10
[78136.551162]  [<ffffffff81068de5>] task_work_run+0xb5/0xd0
[78136.551164]  [<ffffffff8104da1c>] do_exit+0x2ac/0xa30
[78136.551166]  [<ffffffff8107a89b>] ? wake_up_state+0xb/0x10
[78136.551169]  [<ffffffff81059fad>] ? signal_wake_up_state+0x1d/0x30
[78136.551171]  [<ffffffff8105b1c3>] ? zap_other_threads+0x83/0xa0
[78136.551173]  [<ffffffff8104e21a>] do_group_exit+0x3a/0xa0
[78136.551175]  [<ffffffff8104e292>] SyS_exit_group+0x12/0x20
[78136.551177]  [<ffffffff81529fd2>] system_call_fastpath+0x16/0x1b
[78136.551178] ---[ end trace b9ebb6de9753ef4c ]---

Thanks!

Stefan

Re: 3.10.X kernel/jump_label kvm

[Andrey Korolyov]

On 02/28/2014 11:47 PM, Stefan Priebe wrote:
> Hello,
> 
> i got this stack trace multiple times while using a vanilla 3.10.32
> kernel and already sent it to the list in december but got no replies.
> 

Hi,

What kind of workload the host system is experiencing at same time? Does
this event correlate with high memory pressure?

> [78136.551061] WARNING: at kernel/jump_label.c:80
> __static_key_slow_dec+0xb6/0xc0()
> [78136.551062] jump label: negative count!
> [78136.551063] Modules linked in: sch_htb act_police cls_u32 sch_ingress
> vhost_net tun macvtap macvlan netconsole ipt_REJECT dlm sctp
> iptable_filter ip_tables x_tables iscsi_tcp libiscsi_tcp libiscsi
> scsi_transport_iscsi nfsd auth_rpcgss oid_registry bonding ext2 8021q
> garp fuse mperf coretemp kvm_intel kvm crc32_pclmul ghash_clmulni_intel
> microcode i2c_i801 button dm_mod raid1 md_mod usbhid usb_storage
> ohci_hcd sg sd_mod ehci_pci ahci ehci_hcd igb libahci i2c_algo_bit isci
> usbcore i2c_core usb_common libsas ptp ixgbe(O) scsi_transport_sas pps_core
> [78136.551080] CPU: 21 PID: 47183 Comm: kvm Tainted: G        W  O
> 3.10.32+68-ph #1
> [78136.551081] Hardware name: Supermicro
> X9DRW-3LN4F+/X9DRW-3TF+/X9DRW-3LN4F+/X9DRW-3TF+, BIOS 3.00 07/05/2013
> [78136.551081]  0000000000000009 ffff882f4a669be8 ffffffff81524606
> ffff882f4a669c28
> [78136.551085]  ffffffff8104853b 000000004a669c08 ffffffffa045cc40
> 00000000000000fa
> [78136.551088]  ffffffffa045cc60 ffff882f51460160 ffff882f74ab8110
> ffff882f4a669c88
> [78136.551091] Call Trace:
> [78136.551093]  [<ffffffff81524606>] dump_stack+0x19/0x1b
> [78136.551095]  [<ffffffff8104853b>] warn_slowpath_common+0x6b/0xa0
> [78136.551098]  [<ffffffff81048611>] warn_slowpath_fmt+0x41/0x50
> [78136.551100]  [<ffffffff810e05f6>] __static_key_slow_dec+0xb6/0xc0
> [78136.551102]  [<ffffffff810e0631>] static_key_slow_dec_deferred+0x11/0x20
> [78136.551110]  [<ffffffffa043ff60>] kvm_free_lapic+0x90/0xa0 [kvm]
> [78136.551116]  [<ffffffffa0429ef3>] kvm_arch_vcpu_uninit+0x23/0x90 [kvm]
> [78136.551122]  [<ffffffffa0410a20>] kvm_vcpu_uninit+0x20/0x40 [kvm]
> [78136.551125]  [<ffffffffa021fc12>] vmx_free_vcpu+0x52/0x70 [kvm_intel]
> [78136.551132]  [<ffffffffa04295ef>] kvm_arch_vcpu_free+0x4f/0x60 [kvm]
> [78136.551138]  [<ffffffffa042a112>] kvm_arch_destroy_vm+0xf2/0x1f0 [kvm]
> [78136.551141]  [<ffffffff81071048>] ? synchronize_srcu+0x18/0x20
> [78136.551143]  [<ffffffff8112677a>] ? mmu_notifier_unregister+0xaa/0xe0
> [78136.551149]  [<ffffffffa041380e>] kvm_put_kvm+0x10e/0x1b0 [kvm]
> [78136.551155]  [<ffffffffa0413a33>] kvm_vcpu_release+0x13/0x20 [kvm]
> [78136.551157]  [<ffffffff811452d1>] __fput+0xe1/0x230
> [78136.551160]  [<ffffffff81145429>] ____fput+0x9/0x10
> [78136.551162]  [<ffffffff81068de5>] task_work_run+0xb5/0xd0
> [78136.551164]  [<ffffffff8104da1c>] do_exit+0x2ac/0xa30
> [78136.551166]  [<ffffffff8107a89b>] ? wake_up_state+0xb/0x10
> [78136.551169]  [<ffffffff81059fad>] ? signal_wake_up_state+0x1d/0x30
> [78136.551171]  [<ffffffff8105b1c3>] ? zap_other_threads+0x83/0xa0
> [78136.551173]  [<ffffffff8104e21a>] do_group_exit+0x3a/0xa0
> [78136.551175]  [<ffffffff8104e292>] SyS_exit_group+0x12/0x20
> [78136.551177]  [<ffffffff81529fd2>] system_call_fastpath+0x16/0x1b
> [78136.551178] ---[ end trace b9ebb6de9753ef4c ]---
> 
> Thanks!
> 
> Stefan
> 
> -- 
> To unsubscribe from this list: send the line "unsubscribe kvm" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: 3.10.X kernel/jump_label kvm

[Stefan Priebe]

Am 01.03.2014 15:27, schrieb Andrey Korolyov:
> On 02/28/2014 11:47 PM, Stefan Priebe wrote:
>> Hello,
>>
>> i got this stack trace multiple times while using a vanilla 3.10.32
>> kernel and already sent it to the list in december but got no replies.
>>
>
> Hi,
>
> What kind of workload the host system is experiencing at same time? Does
> this event correlate with high memory pressure?

Running around 40 virtual machines.

It had 240GB of 378GB free so i think no memory pressure.

Stefan

>> [78136.551061] WARNING: at kernel/jump_label.c:80
>> __static_key_slow_dec+0xb6/0xc0()
>> [78136.551062] jump label: negative count!
>> [78136.551063] Modules linked in: sch_htb act_police cls_u32 sch_ingress
>> vhost_net tun macvtap macvlan netconsole ipt_REJECT dlm sctp
>> iptable_filter ip_tables x_tables iscsi_tcp libiscsi_tcp libiscsi
>> scsi_transport_iscsi nfsd auth_rpcgss oid_registry bonding ext2 8021q
>> garp fuse mperf coretemp kvm_intel kvm crc32_pclmul ghash_clmulni_intel
>> microcode i2c_i801 button dm_mod raid1 md_mod usbhid usb_storage
>> ohci_hcd sg sd_mod ehci_pci ahci ehci_hcd igb libahci i2c_algo_bit isci
>> usbcore i2c_core usb_common libsas ptp ixgbe(O) scsi_transport_sas pps_core
>> [78136.551080] CPU: 21 PID: 47183 Comm: kvm Tainted: G        W  O
>> 3.10.32+68-ph #1
>> [78136.551081] Hardware name: Supermicro
>> X9DRW-3LN4F+/X9DRW-3TF+/X9DRW-3LN4F+/X9DRW-3TF+, BIOS 3.00 07/05/2013
>> [78136.551081]  0000000000000009 ffff882f4a669be8 ffffffff81524606
>> ffff882f4a669c28
>> [78136.551085]  ffffffff8104853b 000000004a669c08 ffffffffa045cc40
>> 00000000000000fa
>> [78136.551088]  ffffffffa045cc60 ffff882f51460160 ffff882f74ab8110
>> ffff882f4a669c88
>> [78136.551091] Call Trace:
>> [78136.551093]  [<ffffffff81524606>] dump_stack+0x19/0x1b
>> [78136.551095]  [<ffffffff8104853b>] warn_slowpath_common+0x6b/0xa0
>> [78136.551098]  [<ffffffff81048611>] warn_slowpath_fmt+0x41/0x50
>> [78136.551100]  [<ffffffff810e05f6>] __static_key_slow_dec+0xb6/0xc0
>> [78136.551102]  [<ffffffff810e0631>] static_key_slow_dec_deferred+0x11/0x20
>> [78136.551110]  [<ffffffffa043ff60>] kvm_free_lapic+0x90/0xa0 [kvm]
>> [78136.551116]  [<ffffffffa0429ef3>] kvm_arch_vcpu_uninit+0x23/0x90 [kvm]
>> [78136.551122]  [<ffffffffa0410a20>] kvm_vcpu_uninit+0x20/0x40 [kvm]
>> [78136.551125]  [<ffffffffa021fc12>] vmx_free_vcpu+0x52/0x70 [kvm_intel]
>> [78136.551132]  [<ffffffffa04295ef>] kvm_arch_vcpu_free+0x4f/0x60 [kvm]
>> [78136.551138]  [<ffffffffa042a112>] kvm_arch_destroy_vm+0xf2/0x1f0 [kvm]
>> [78136.551141]  [<ffffffff81071048>] ? synchronize_srcu+0x18/0x20
>> [78136.551143]  [<ffffffff8112677a>] ? mmu_notifier_unregister+0xaa/0xe0
>> [78136.551149]  [<ffffffffa041380e>] kvm_put_kvm+0x10e/0x1b0 [kvm]
>> [78136.551155]  [<ffffffffa0413a33>] kvm_vcpu_release+0x13/0x20 [kvm]
>> [78136.551157]  [<ffffffff811452d1>] __fput+0xe1/0x230
>> [78136.551160]  [<ffffffff81145429>] ____fput+0x9/0x10
>> [78136.551162]  [<ffffffff81068de5>] task_work_run+0xb5/0xd0
>> [78136.551164]  [<ffffffff8104da1c>] do_exit+0x2ac/0xa30
>> [78136.551166]  [<ffffffff8107a89b>] ? wake_up_state+0xb/0x10
>> [78136.551169]  [<ffffffff81059fad>] ? signal_wake_up_state+0x1d/0x30
>> [78136.551171]  [<ffffffff8105b1c3>] ? zap_other_threads+0x83/0xa0
>> [78136.551173]  [<ffffffff8104e21a>] do_group_exit+0x3a/0xa0
>> [78136.551175]  [<ffffffff8104e292>] SyS_exit_group+0x12/0x20
>> [78136.551177]  [<ffffffff81529fd2>] system_call_fastpath+0x16/0x1b
>> [78136.551178] ---[ end trace b9ebb6de9753ef4c ]---
>>
>> Thanks!
>>
>> Stefan
>>
>> --
>> To unsubscribe from this list: send the line "unsubscribe kvm" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>

Re: 3.10.X kernel/jump_label kvm

[Paolo Bonzini]

Il 28/02/2014 20:47, Stefan Priebe ha scritto:
> Hello,
>
> i got this stack trace multiple times while using a vanilla 3.10.32
> kernel and already sent it to the list in december but got no replies.

Please try the patch of commit 0dce7cd67fd9055c4a2ff278f8af1431e646d346:

diff --git a/arch/x86/kvm/lapic.c b/arch/x86/kvm/lapic.c
index 1673940cf9c3..775702f649ca 100644
--- a/arch/x86/kvm/lapic.c
+++ b/arch/x86/kvm/lapic.c
@@ -1355,7 +1355,7 @@ void kvm_lapic_set_base(struct kvm_vcpu *vcpu, u64 
value)
         vcpu->arch.apic_base = value;

         /* update jump label if enable bit changes */
-       if ((vcpu->arch.apic_base ^ value) & MSR_IA32_APICBASE_ENABLE) {
+       if ((old_value ^ value) & MSR_IA32_APICBASE_ENABLE) {
                 if (value & MSR_IA32_APICBASE_ENABLE)
                         static_key_slow_dec_deferred(&apic_hw_disabled);
                 else

Paolo

Re: 3.10.X kernel/jump_label kvm

[Stefan Priebe]

Am 03.03.2014 17:36, schrieb Paolo Bonzini:
> Il 28/02/2014 20:47, Stefan Priebe ha scritto:
>> Hello,
>>
>> i got this stack trace multiple times while using a vanilla 3.10.32
>> kernel and already sent it to the list in december but got no replies.
>
> Please try the patch of commit 0dce7cd67fd9055c4a2ff278f8af1431e646d346:

Thanks - will try it. So this one should also go into 3.10.X

Stefan

> diff --git a/arch/x86/kvm/lapic.c b/arch/x86/kvm/lapic.c
> index 1673940cf9c3..775702f649ca 100644
> --- a/arch/x86/kvm/lapic.c
> +++ b/arch/x86/kvm/lapic.c
> @@ -1355,7 +1355,7 @@ void kvm_lapic_set_base(struct kvm_vcpu *vcpu, u64
> value)
>          vcpu->arch.apic_base = value;
>
>          /* update jump label if enable bit changes */
> -       if ((vcpu->arch.apic_base ^ value) & MSR_IA32_APICBASE_ENABLE) {
> +       if ((old_value ^ value) & MSR_IA32_APICBASE_ENABLE) {
>                  if (value & MSR_IA32_APICBASE_ENABLE)
>                          static_key_slow_dec_deferred(&apic_hw_disabled);
>                  else
>
> Paolo

Re: 3.10.X kernel/jump_label kvm

[Paolo Bonzini]

Il 03/03/2014 19:17, Stefan Priebe ha scritto:
>
> Am 03.03.2014 17:36, schrieb Paolo Bonzini:
>> Il 28/02/2014 20:47, Stefan Priebe ha scritto:
>>> Hello,
>>>
>>> i got this stack trace multiple times while using a vanilla 3.10.32
>>> kernel and already sent it to the list in december but got no replies.
>>
>> Please try the patch of commit 0dce7cd67fd9055c4a2ff278f8af1431e646d346:
>
> Thanks - will try it. So this one should also go into 3.10.X

Yes, I've sent it to stable@vger.kernel.org already.

Paolo

Re: 3.10.X kernel/jump_label kvm

[Jan Kiszka]

On 2014-03-03 19:17, Stefan Priebe wrote:
> 
> Am 03.03.2014 17:36, schrieb Paolo Bonzini:
>> Il 28/02/2014 20:47, Stefan Priebe ha scritto:
>>> Hello,
>>>
>>> i got this stack trace multiple times while using a vanilla 3.10.32
>>> kernel and already sent it to the list in december but got no replies.
>>
>> Please try the patch of commit 0dce7cd67fd9055c4a2ff278f8af1431e646d346:
> 
> Thanks - will try it. So this one should also go into 3.10.X

It's part of 3.10.29. Only .26..28 were affected by the regression below.

Jan

>> diff --git a/arch/x86/kvm/lapic.c b/arch/x86/kvm/lapic.c
>> index 1673940cf9c3..775702f649ca 100644
>> --- a/arch/x86/kvm/lapic.c
>> +++ b/arch/x86/kvm/lapic.c
>> @@ -1355,7 +1355,7 @@ void kvm_lapic_set_base(struct kvm_vcpu *vcpu, u64
>> value)
>>          vcpu->arch.apic_base = value;
>>
>>          /* update jump label if enable bit changes */
>> -       if ((vcpu->arch.apic_base ^ value) & MSR_IA32_APICBASE_ENABLE) {
>> +       if ((old_value ^ value) & MSR_IA32_APICBASE_ENABLE) {
>>                  if (value & MSR_IA32_APICBASE_ENABLE)
>>                          static_key_slow_dec_deferred(&apic_hw_disabled);
>>                  else
>>
>> Paolo
-- 
Siemens AG, Corporate Technology, CT RTC ITP SES-DE
Corporate Competence Center Embedded Linux

Re: 3.10.X kernel/jump_label kvm

[Stefan Priebe - Profihost AG]

Am 04.03.2014 11:01, schrieb Jan Kiszka:
> On 2014-03-03 19:17, Stefan Priebe wrote:
>>
>> Am 03.03.2014 17:36, schrieb Paolo Bonzini:
>>> Il 28/02/2014 20:47, Stefan Priebe ha scritto:
>>>> Hello,
>>>>
>>>> i got this stack trace multiple times while using a vanilla 3.10.32
>>>> kernel and already sent it to the list in december but got no replies.
>>>
>>> Please try the patch of commit 0dce7cd67fd9055c4a2ff278f8af1431e646d346:
>>
>> Thanks - will try it. So this one should also go into 3.10.X
> 
> It's part of 3.10.29. Only .26..28 were affected by the regression below.
> 
> Jan

HI Jan,

but i'm using 3.10.32. So it is another bug i'm seeing?

Stefan

>>> diff --git a/arch/x86/kvm/lapic.c b/arch/x86/kvm/lapic.c
>>> index 1673940cf9c3..775702f649ca 100644
>>> --- a/arch/x86/kvm/lapic.c
>>> +++ b/arch/x86/kvm/lapic.c
>>> @@ -1355,7 +1355,7 @@ void kvm_lapic_set_base(struct kvm_vcpu *vcpu, u64
>>> value)
>>>          vcpu->arch.apic_base = value;
>>>
>>>          /* update jump label if enable bit changes */
>>> -       if ((vcpu->arch.apic_base ^ value) & MSR_IA32_APICBASE_ENABLE) {
>>> +       if ((old_value ^ value) & MSR_IA32_APICBASE_ENABLE) {
>>>                  if (value & MSR_IA32_APICBASE_ENABLE)
>>>                          static_key_slow_dec_deferred(&apic_hw_disabled);
>>>                  else
>>>
>>> Paolo

Re: 3.10.X kernel/jump_label kvm

[Jan Kiszka]

On 2014-03-04 11:26, Stefan Priebe - Profihost AG wrote:
> Am 04.03.2014 11:01, schrieb Jan Kiszka:
>> On 2014-03-03 19:17, Stefan Priebe wrote:
>>>
>>> Am 03.03.2014 17:36, schrieb Paolo Bonzini:
>>>> Il 28/02/2014 20:47, Stefan Priebe ha scritto:
>>>>> Hello,
>>>>>
>>>>> i got this stack trace multiple times while using a vanilla 3.10.32
>>>>> kernel and already sent it to the list in december but got no replies.
>>>>
>>>> Please try the patch of commit 0dce7cd67fd9055c4a2ff278f8af1431e646d346:
>>>
>>> Thanks - will try it. So this one should also go into 3.10.X
>>
>> It's part of 3.10.29. Only .26..28 were affected by the regression below.
>>
>> Jan
> 
> HI Jan,
> 
> but i'm using 3.10.32. So it is another bug i'm seeing?

Yes, that's the logical consequence.

Jan

-- 
Siemens AG, Corporate Technology, CT RTC ITP SES-DE
Corporate Competence Center Embedded Linux