Re: soft lockup in kvm_flush_remote_tlbs

[Avi Kivity <avi-atKUWr5tajBWk0Htik3J/w@public.gmane.org>]

david ahern wrote:
> As a quick test I added a printk to the loop, right after the while():
>
>      while (atomic_read(&completed) != needed) {
>          printk("kvm_flush_remote_tlbs: completed = %d, needed = %d\n", atomic_read(&completed), needed);
>          cpu_relax();
>          barrier();
>      }
>
>
> This is the output right before a lockup:
>
> Oct 24 16:03:47 bldr-ccm20 kernel: kvm_flush_remote_tlbs: completed = 2, needed = 2
> Oct 24 16:03:47 bldr-ccm20 kernel: kvm_flush_remote_tlbs: completed = 2, needed = 2
> Oct 24 16:03:47 bldr-ccm20 kernel: kvm_flush_remote_tlbs: completed = 1, needed = 2
> Oct 24 16:03:57 bldr-ccm20 last message repeated 105738 times
> Oct 24 16:03:57 bldr-ccm20 kernel: BUG: soft lockup detected on CPU#0!
> Oct 24 16:03:57 bldr-ccm20 kernel:  [<c044a0b7>] softlockup_tick+0x98/0xa6
> Oct 24 16:03:57 bldr-ccm20 kernel:  [<c042cc98>] update_process_times+0x39/0x5c
> Oct 24 16:03:57 bldr-ccm20 kernel:  [<c04176ec>] smp_apic_timer_interrupt+0x5c/0x64
> Oct 24 16:03:57 bldr-ccm20 kernel:  [<c04049bf>] apic_timer_interrupt+0x1f/0x24
> Oct 24 16:03:57 bldr-ccm20 kernel:  [<c0424130>] vprintk+0x288/0x2bc
> Oct 24 16:03:57 bldr-ccm20 kernel:  [<c0459db7>] follow_page+0x168/0x1b6
> Oct 24 16:03:57 bldr-ccm20 kernel:  [<c04d8067>] cfq_slice_async_store+0x5/0x38
> Oct 24 16:03:57 bldr-ccm20 kernel:  [<c0459db7>] follow_page+0x168/0x1b6
> Oct 24 16:03:57 bldr-ccm20 kernel:  [<c0406406>] do_IRQ+0xa5/0xae
> Oct 24 16:03:57 bldr-ccm20 kernel:  [<c040492e>] common_interrupt+0x1a/0x20
> Oct 24 16:03:57 bldr-ccm20 kernel:  [<c042417c>] printk+0x18/0x8e
> Oct 24 16:03:57 bldr-ccm20 kernel:  [<f89a9812>] kvm_flush_remote_tlbs+0xe0/0xf2 [kvm]
> ...
>
>
> I'd like to get a solution for RHEL5, so I am attempting to backport smp_call_function_mask(). I'm open to other suggestions if you think it is corruption or the problem is somewhere else.
>   

No, it looks like the problem is indeed in kvm_flush_remote_tlbs(), and 
not a corruption elsewhere.

Things to check:

- whether cpus_weight(mask) == needed
- whether wrapping the whole thing in preempt_disable()/preempt_enable() 
helps

hey! I see a bug!

>                         continue;
>                 cpu = vcpu->cpu;
>                 if (cpu != -1 && cpu != raw_smp_processor_id())
>                         if (!cpu_isset(cpu, cpus)) {
>                                 cpu_set(cpu, cpus);
>                                 ++needed;
>                         }
>         }
>

vcpu->cpu can change during execution if this snippet due to a vcpu 
being migrated concurrently with this being executed.  Since the 
compiler is free to reload 'cpu' from 'vcpu->cpu', the code can operate 
on corrupted data.

A 'barrier();' after 'cpu = vcpu->cpu;' should fix it, if this is indeed 
the bug.


-- 
error compiling committee.c: too many arguments to function


-------------------------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc.
Still grepping through log files to find problems?  Stop.
Now Search log events and configuration files using AJAX and a browser.
Download your FREE copy of Splunk now >> http://get.splunk.com/