From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from eggs.gnu.org ([2001:4830:134:3::10]:33108) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1bX6Ld-00018O-5B for qemu-devel@nongnu.org; Tue, 09 Aug 2016 08:41:34 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1bX6LY-0007BQ-SC for qemu-devel@nongnu.org; Tue, 09 Aug 2016 08:41:32 -0400 Received: from mx1.redhat.com ([209.132.183.28]:59256) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1bX6LY-0007AC-Js for qemu-devel@nongnu.org; Tue, 09 Aug 2016 08:41:28 -0400 Date: Tue, 9 Aug 2016 14:41:21 +0200 From: Radim =?utf-8?B?S3LEjW3DocWZ?= Message-ID: <20160809124121.GC2263@potion> References: <1470390377-228219-1-git-send-email-imammedo@redhat.com> <20160808074123.GA32205@g.c> <20160808085714.GG3825@pxdev.xzpeter.org> <20160809043317.GB16425@g.c> <20160809061815.GB3979@pxdev.xzpeter.org> <20160809062429.GC3979@pxdev.xzpeter.org> <20160809070946.GD3979@pxdev.xzpeter.org> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20160809070946.GD3979@pxdev.xzpeter.org> Subject: Re: [Qemu-devel] [PATCH for-2.8 00/18] pc: q35: x2APIC support in kvm_apic mode List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , To: Peter Xu Cc: Jan Kiszka , Igor Mammedov , qemu-devel@nongnu.org, ehabkost@redhat.com, mst@redhat.com, mtosatti@redhat.com, kevin@koconnor.net, pbonzini@redhat.com, lersek@redhat.com, tianyu.lan@intel.com, yong.y.wang@intel.com 2016-08-09 15:09+0800, Peter Xu: > On Tue, Aug 09, 2016 at 08:33:13AM +0200, Jan Kiszka wrote: >> On 2016-08-09 08:24, Peter Xu wrote: >> > On Tue, Aug 09, 2016 at 02:18:15PM +0800, Peter Xu wrote: >> >> On Tue, Aug 09, 2016 at 12:33:17PM +0800, Chao Gao wrote: >> >>> On Mon, Aug 08, 2016 at 04:57:14PM +0800, Peter Xu wrote: >> >>>> On Mon, Aug 08, 2016 at 03:41:23PM +0800, Chao Gao wrote: >> >>>>> HI, everyone. >> >>>>> >> >>>>> We have done some tests after merging this patch set into the lastest qemu >> >>>>> master. In kvm aspect, we use the lastest kvm linux-next branch. Here are >> >>>>> some problems we have met. >> >>>>> >> >>>>> 1. We can't boot up a 288 vcpus linux guest with CLI: >> >>>>> qemu-system-x86_64 -boot c -m 4096 -sdl -monitor pty --enable-kvm \ >> >>>>> -M kernel-irqchip=split -serial stdio -bios bios.bin -smp cpus=288 \ >> >>>>> -hda vdisk.img -device intel-iommu,intremap=on -machine q35. >> >>>>> The problem exists, even after we only assign 32 vcpus to the linux guest. >> >>>>> Maybe the output "do_IRQ: 146.113 No irq handler for vector (irq -1)" is a clue. >> >>>>> The output of qemu and kernel is in attachments. Do you have any idea >> >>>>> about the problem and how to solve it? >> >>>> >> >>>> IIUC, we need to wait for Radim's QEMU patches to finally enable 288 >> >>>> vcpus? >> >>>> >> >>>> Btw, could you please try adding this to the QEMU cmdline when testing >> >>>> with 32 vcpus: >> >>>> >> >>>> -global ioapic.version=0x20 >> >>>> >> >>>> I see that you were running RHEL 7.2 guest with a default e1000. In >> >>>> that case, we may need to boost ioapic version to 0x20. >> >>> >> >>> It doesn't work. My host machine has 16 cpus. When I assign 4 or 8 vcpus to the guest >> >>> or 255 vcpus but set "kernel-irqchip=off", the guest work well. Maybe when irqchip >> >>> is in kernel, intremap can only handle situations that vcpus number is less than >> >>> physical cpus'. Do you think it's right? >> >> >> >> I don't think so. Vcpu number should not be related to host cpu >> >> numbers. >> >> >> >> I think the problem is with x2apic. Currently, x2apic is enabled in >> >> vIOMMU when kernel irqchip is used. This is problematic, since >> >> actually we are throughing away dest_id[31:8] without Radim's patches, >> >> meanwhile I see that by default x2apic is using cluster mode. >> >> >> >> In cluster mode, 8 bits will possibly not suffice (I think the reason >> >> why >17 vcpus will bring trouble is that each cluster has 16 vcpus, >> >> we'll have trouble if we have more than one cluster). >> >> >> >> To temporarily solve your issue, you should not only need "-global >> >> ioapic.version=0x20" in QEMU command line, but also add "x2apic_phys" >> >> to you guest kernel boot parameter, to force guest boot with x2apic >> >> physical mode (not cluster mode). Though this can only work for <255 >> >> vcpus. IMHO we may still need to wait for Radim's patches to test >255 >> >> case. >> > >> > Not sure whether we should temporarily disable EIM by default for now >> > (provide an extra flag to optionally enable it)? Since it might break >> > guests with >17 vcpus. >> > >> > CC Jan as well. >> >> A switch for EIM would be fine for me if it helps. >> >> To my understanding, the issue will be gone with an enhance KVM >> interface that we can then also detect via some cap (to flip the default >> again)? > > Would you help explain how to do it? > > Btw, if we have that switch, the default can go back to EIM mode along > with Radim's future patches. I will post patches today as the feature made it upstream.