From mboxrd@z Thu Jan 1 00:00:00 1970 From: Avi Kivity Subject: Re: soft lockup in kvm_flush_remote_tlbs Date: Thu, 25 Oct 2007 15:46:11 +0200 Message-ID: <47209E23.8080808@qumranet.com> References: <471FCEA6.6000903@cisco.com> <47203BEB.9070509@qumranet.com> <47209A77.3090503@cisco.com> Mime-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Cc: kvm-devel-5NWGOfrQmneRv+LV9MX5uipxlwaOVQ5f@public.gmane.org To: david ahern Return-path: In-Reply-To: <47209A77.3090503-FYB4Gu1CFyUAvxtiuMwx3w@public.gmane.org> List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: kvm-devel-bounces-5NWGOfrQmneRv+LV9MX5uipxlwaOVQ5f@public.gmane.org Errors-To: kvm-devel-bounces-5NWGOfrQmneRv+LV9MX5uipxlwaOVQ5f@public.gmane.org List-Id: kvm.vger.kernel.org david ahern wrote: > As a quick test I added a printk to the loop, right after the while(): > > while (atomic_read(&completed) != needed) { > printk("kvm_flush_remote_tlbs: completed = %d, needed = %d\n", atomic_read(&completed), needed); > cpu_relax(); > barrier(); > } > > > This is the output right before a lockup: > > Oct 24 16:03:47 bldr-ccm20 kernel: kvm_flush_remote_tlbs: completed = 2, needed = 2 > Oct 24 16:03:47 bldr-ccm20 kernel: kvm_flush_remote_tlbs: completed = 2, needed = 2 > Oct 24 16:03:47 bldr-ccm20 kernel: kvm_flush_remote_tlbs: completed = 1, needed = 2 > Oct 24 16:03:57 bldr-ccm20 last message repeated 105738 times > Oct 24 16:03:57 bldr-ccm20 kernel: BUG: soft lockup detected on CPU#0! > Oct 24 16:03:57 bldr-ccm20 kernel: [] softlockup_tick+0x98/0xa6 > Oct 24 16:03:57 bldr-ccm20 kernel: [] update_process_times+0x39/0x5c > Oct 24 16:03:57 bldr-ccm20 kernel: [] smp_apic_timer_interrupt+0x5c/0x64 > Oct 24 16:03:57 bldr-ccm20 kernel: [] apic_timer_interrupt+0x1f/0x24 > Oct 24 16:03:57 bldr-ccm20 kernel: [] vprintk+0x288/0x2bc > Oct 24 16:03:57 bldr-ccm20 kernel: [] follow_page+0x168/0x1b6 > Oct 24 16:03:57 bldr-ccm20 kernel: [] cfq_slice_async_store+0x5/0x38 > Oct 24 16:03:57 bldr-ccm20 kernel: [] follow_page+0x168/0x1b6 > Oct 24 16:03:57 bldr-ccm20 kernel: [] do_IRQ+0xa5/0xae > Oct 24 16:03:57 bldr-ccm20 kernel: [] common_interrupt+0x1a/0x20 > Oct 24 16:03:57 bldr-ccm20 kernel: [] printk+0x18/0x8e > Oct 24 16:03:57 bldr-ccm20 kernel: [] kvm_flush_remote_tlbs+0xe0/0xf2 [kvm] > ... > > > I'd like to get a solution for RHEL5, so I am attempting to backport smp_call_function_mask(). I'm open to other suggestions if you think it is corruption or the problem is somewhere else. > No, it looks like the problem is indeed in kvm_flush_remote_tlbs(), and not a corruption elsewhere. Things to check: - whether cpus_weight(mask) == needed - whether wrapping the whole thing in preempt_disable()/preempt_enable() helps hey! I see a bug! > continue; > cpu = vcpu->cpu; > if (cpu != -1 && cpu != raw_smp_processor_id()) > if (!cpu_isset(cpu, cpus)) { > cpu_set(cpu, cpus); > ++needed; > } > } > vcpu->cpu can change during execution if this snippet due to a vcpu being migrated concurrently with this being executed. Since the compiler is free to reload 'cpu' from 'vcpu->cpu', the code can operate on corrupted data. A 'barrier();' after 'cpu = vcpu->cpu;' should fix it, if this is indeed the bug. -- error compiling committee.c: too many arguments to function ------------------------------------------------------------------------- This SF.net email is sponsored by: Splunk Inc. Still grepping through log files to find problems? Stop. Now Search log events and configuration files using AJAX and a browser. Download your FREE copy of Splunk now >> http://get.splunk.com/