From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from eggs.gnu.org ([2001:4830:134:3::10]:39119) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1ZCSAa-00057p-46 for qemu-devel@nongnu.org; Tue, 07 Jul 2015 08:40:17 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1ZCSAV-0003BZ-JT for qemu-devel@nongnu.org; Tue, 07 Jul 2015 08:40:16 -0400 Received: from szxga03-in.huawei.com ([119.145.14.66]:11054) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1ZCSAU-000351-R3 for qemu-devel@nongnu.org; Tue, 07 Jul 2015 08:40:11 -0400 Message-ID: <559BC884.3000807@huawei.com> Date: Tue, 7 Jul 2015 20:39:32 +0800 From: zhanghailiang MIME-Version: 1.0 References: <559A342C.6020207@huawei.com> <559A4010.30808@redhat.com> <559A516E.1070000@huawei.com> <20150707132344.04476183@nial.brq.redhat.com> <559BBB67.4000503@huawei.com> <20150707142132.33d7d9d6@nial.brq.redhat.com> In-Reply-To: <20150707142132.33d7d9d6@nial.brq.redhat.com> Content-Type: text/plain; charset="utf-8"; format=flowed Content-Transfer-Encoding: 8bit Subject: Re: [Qemu-devel] [BUG/RFC] Two cpus are not brought up normally in SLES11 sp3 VM after reboot List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , To: Igor Mammedov Cc: Paolo Bonzini , peter.huangpeng@huawei.com, kvm@vger.kernel.org, "qemu-devel@nongnu.org" On 2015/7/7 20:21, Igor Mammedov wrote: > On Tue, 7 Jul 2015 19:43:35 +0800 > zhanghailiang wrote: > >> On 2015/7/7 19:23, Igor Mammedov wrote: >>> On Mon, 6 Jul 2015 17:59:10 +0800 >>> zhanghailiang wrote: >>> >>>> On 2015/7/6 16:45, Paolo Bonzini wrote: >>>>> >>>>> >>>>> On 06/07/2015 09:54, zhanghailiang wrote: >>>>>> >>>>>> From host, we found that QEMU vcpu1 thread and vcpu7 thread were not >>>>>> consuming any cpu (Should be in idle state), >>>>>> All of VCPUs' stacks in host is like bellow: >>>>>> >>>>>> [] kvm_vcpu_block+0x65/0xa0 [kvm] >>>>>> [] __vcpu_run+0xd1/0x260 [kvm] >>>>>> [] kvm_arch_vcpu_ioctl_run+0x68/0x1a0 [kvm] >>>>>> [] kvm_vcpu_ioctl+0x38e/0x580 [kvm] >>>>>> [] do_vfs_ioctl+0x8b/0x3b0 >>>>>> [] sys_ioctl+0xa1/0xb0 >>>>>> [] system_call_fastpath+0x16/0x1b >>>>>> [<00002ab9fe1f99a7>] 0x2ab9fe1f99a7 >>>>>> [] 0xffffffffffffffff >>>>>> >>>>>> We looked into the kernel codes that could leading to the above 'Stuck' >>>>>> warning, >>> in current upstream there isn't any printk(...Stuck...) left since that code path >>> has been reworked. >>> I've often seen this on over-committed host during guest CPUs up/down torture test. >>> Could you update guest kernel to upstream and see if issue reproduces? >>> >> >> Hmm, Unfortunately, it is very hard to reproduce, and we are still trying to reproduce it. >> >> For your test case, is it a kernel bug? >> Or is there any related patch could solve your test problem been merged into >> upstream ? > I don't remember all prerequisite patches but you should be able to find > http://marc.info/?l=linux-kernel&m=140326703108009&w=2 > "x86/smpboot: Initialize secondary CPU only if master CPU will wait for it" > and then look for dependencies. > Er, we have investigated this patch, and it is not related to our problem, :) Thanks. > >> >> Thanks, >> zhanghailiang >> >>>>>> and found that the only possible is the emulation of 'cpuid' instruct in >>>>>> kvm/qemu has something wrong. >>>>>> But since we can’t reproduce this problem, we are not quite sure. >>>>>> Is there any possible that the cupid emulation in kvm/qemu has some bug ? >>>>> >>>>> Can you explain the relationship to the cpuid emulation? What do the >>>>> traces say about vcpus 1 and 7? >>>> >>>> OK, we searched the VM's kernel codes with the 'Stuck' message, and it is located in >>>> do_boot_cpu(). It's in BSP context, the call process is: >>>> BSP executes start_kernel() -> smp_init() -> smp_boot_cpus() -> do_boot_cpu() -> wakeup_secondary_via_INIT() to trigger APs. >>>> It will wait 5s for APs to startup, if some AP not startup normally, it will print 'CPU%d Stuck' or 'CPU%d: Not responding'. >>>> >>>> If it prints 'Stuck', it means the AP has received the SIPI interrupt and begins to execute the code >>>> 'ENTRY(trampoline_data)' (trampoline_64.S) , but be stuck in some places before smp_callin()(smpboot.c). >>>> The follow is the starup process of BSP and AP. >>>> BSP: >>>> start_kernel() >>>> ->smp_init() >>>> ->smp_boot_cpus() >>>> ->do_boot_cpu() >>>> ->start_ip = trampoline_address(); //set the address that AP will go to execute >>>> ->wakeup_secondary_cpu_via_init(); // kick the secondary CPU >>>> ->for (timeout = 0; timeout < 50000; timeout++) >>>> if (cpumask_test_cpu(cpu, cpu_callin_mask)) break;// check if AP startup or not >>>> >>>> APs: >>>> ENTRY(trampoline_data) (trampoline_64.S) >>>> ->ENTRY(secondary_startup_64) (head_64.S) >>>> ->start_secondary() (smpboot.c) >>>> ->cpu_init(); >>>> ->smp_callin(); >>>> ->cpumask_set_cpu(cpuid, cpu_callin_mask); ->Note: if AP comes here, the BSP will not prints the error message. >>>> >>>> From above call process, we can be sure that, the AP has been stuck between trampoline_data and the cpumask_set_cpu() in >>>> smp_callin(), we look through these codes path carefully, and only found a 'hlt' instruct that could block the process. >>>> It is located in trampoline_data(): >>>> >>>> ENTRY(trampoline_data) >>>> ... >>>> >>>> call verify_cpu # Verify the cpu supports long mode >>>> testl %eax, %eax # Check for return code >>>> jnz no_longmode >>>> >>>> ... >>>> >>>> no_longmode: >>>> hlt >>>> jmp no_longmode >>>> >>>> For the process verify_cpu(), >>>> we can only find the 'cpuid' sensitive instruct that could lead VM exit from No-root mode. >>>> This is why we doubt if cpuid emulation is wrong in KVM/QEMU that leading to the fail in verify_cpu. >>>> >>>> From the message in VM, we know vcpu1 and vcpu7 is something wrong. >>>> [ 5.060042] CPU1: Stuck ?? >>>> [ 10.170815] CPU7: Stuck ?? >>>> [ 10.171648] Brought up 6 CPUs >>>> >>>> Besides, the follow is the cpus message got from host. >>>> 80FF72F5-FF6D-E411-A8C8-000000821800:/home/fsp/hrg # virsh qemu-monitor-command instance-0000000 >>>> * CPU #0: pc=0x00007f64160c683d thread_id=68570 >>>> CPU #1: pc=0xffffffff810301f1 (halted) thread_id=68573 >>>> CPU #2: pc=0xffffffff810301e2 (halted) thread_id=68575 >>>> CPU #3: pc=0xffffffff810301e2 (halted) thread_id=68576 >>>> CPU #4: pc=0xffffffff810301e2 (halted) thread_id=68577 >>>> CPU #5: pc=0xffffffff810301e2 (halted) thread_id=68578 >>>> CPU #6: pc=0xffffffff810301e2 (halted) thread_id=68583 >>>> CPU #7: pc=0xffffffff810301f1 (halted) thread_id=68584 >>>> >>>> Oh, i also forgot to mention in the above message that, we have bond each vCPU to different physical CPU in >>>> host. >>>> >>>> Thanks, >>>> zhanghailiang >>>> >>>> >>>> >>>> >>>> -- >>>> To unsubscribe from this list: send the line "unsubscribe kvm" in >>>> the body of a message to majordomo@vger.kernel.org >>>> More majordomo info at http://vger.kernel.org/majordomo-info.html >>> >>> >>> . >>> >> >> >> > > > . >