From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from eggs.gnu.org ([208.118.235.92]:47891) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1StETo-0000Uf-Tz for qemu-devel@nongnu.org; Mon, 23 Jul 2012 04:59:09 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1StETn-0007uD-KD for qemu-devel@nongnu.org; Mon, 23 Jul 2012 04:59:04 -0400 Received: from mx1.redhat.com ([209.132.183.28]:9226) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1StETn-0007tr-CX for qemu-devel@nongnu.org; Mon, 23 Jul 2012 04:59:03 -0400 Date: Mon, 23 Jul 2012 11:58:59 +0300 From: Gleb Natapov Message-ID: <20120723085859.GU26120@redhat.com> References: <1342562190-25456-1-git-send-email-ehabkost@redhat.com> <20120719095846.GT26120@redhat.com> <20120719142854.GM13029@otherpad.lan.raisama.net> <20120719194635.GO13029@otherpad.lan.raisama.net> MIME-Version: 1.0 Content-Type: text/plain; charset=cp1255 Content-Disposition: inline Content-Transfer-Encoding: quoted-printable In-Reply-To: <20120719194635.GO13029@otherpad.lan.raisama.net> Subject: Re: [Qemu-devel] [RFC seabios PATCH] enumerate APIC IDs directly from CPUs List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , To: Eduardo Habkost Cc: Igor Mammedov , seabios@seabios.org, qemu-devel@nongnu.org On Thu, Jul 19, 2012 at 04:46:35PM -0300, Eduardo Habkost wrote: > On Thu, Jul 19, 2012 at 11:28:54AM -0300, Eduardo Habkost wrote: > > On Thu, Jul 19, 2012 at 12:58:46PM +0300, Gleb Natapov wrote: > > > On Tue, Jul 17, 2012 at 06:56:30PM -0300, Eduardo Habkost wrote: > > > > This patch is an attempt to fix the non-continguous-APIC-ID problem= without the > > > > FW_CFG_LAPIC_INFO approach I have sent proposed last week. > > > >=20 > > > > Basically, this changes Seabios to probe for APIC IDs directly from= the > > > > CPUs on boot, instead of getting it using fw_cfg, store the found A= PIC > > > > IDs on a bitmap, and use that information whe building the MADT, SR= AT, > > > > and SSDT ACPI tables. > > > >=20 > > > > To do this properly, we have to decide the meaning of "CPU IDs" in = the > > > > QEMU<->Seabios interfaces, too. I see two possible approaches: > > > >=20 > > > > 1) Have Seabios and QEMU agree on a a "CPU identifier", that is > > > > independent from the APIC ID. > > > > 2) Always use the Initial APIC ID on all communication between QEMU= and > > > > Seabios. > > > >=20 > > > We need to be prepared to support more than 255 cpus. With > 255 cpus > > > comes x2apic and x2apic has 32bit apic ids. HW does not have to suppo= rt > > > all of the bits though, but potentially all the bitmasks can grow > > > prohibitedly large. > >=20 > > I see only two solutions: > >=20 > > - Specify an interface/convention for QEMU and Seabios agree upon a "CPU > > identifier" <=3D> x2apic <=3D> LAPIC ID mapping for all CPUs. > > - Specify new NUMA-information and CPU hotplug interfaces (or extend the > > existing ones) based on x2apic ID, when Seabios start supporting > > x2apic. > >=20 > > I am not particularly inclined towards any of those two solutions. I > > dislike them equally. :-) >=20 > Oh, it is simpler than I have expected. x2APIC specification: >=20 > "The local APIC ID is initialized by hardware with a 32 bit ID (x2APIC > ID). The lowest 8 bits of the x2APIC ID is the legacy local xAPIC ID, > and is stored in the upper 8 bits of the APIC register for access in > xAPIC mode." >=20 > And the ACPI specification: >=20 > "Logical processors with APIC ID values of 255 and greater are required > to have a Processor Device object and must convey the processor=92s APIC > information to OSPM using the Processor Local X2APIC structure. Logical > processors with APIC ID values less than 255 must use the Processor > Local APIC structure to convey their APIC information to OSPM." >=20 > That means the x2APIC ID and the xAPIC ID are interchangeable, for > values <=3D 255. That means the QEMU<=3D>Seabios communication can be saf= ely > based on "APIC IDs" without any ambiguity. >=20 Yes for <=3D 255 they interchangeable. That's why we can add +x2apic to our cpu models without changes to the BIOS. > The CPU hotplug interface is a bit of a problem because it is based on a > 256-bit bitmap. But on the day it gets extended to support more than 256 > CPUs, it can safely be based on APIC IDs and still keep compatibility > with systems without x2APIC. The bitmap will have to be extended if we will go beyond 256 cpus. Using apic-id to index the bitmap means that the size of the bitmap is a function of max apic-id we want to support, not max number of cpus. >=20 > So, now I am strongly inclined towards the second option from the list > above: just use APIC IDs everywhere to identify CPUs when QEMU and > Seabios communicate with each other, and QEMU can completely ignore the > "Processor ID" used by Seabios. I agree with making "Processor ID" Seabios internal thing. >=20 > >=20 > > Note that I am more worried about the QEMU<->Seabios interfaces. The > > APIC ID bitmap on smp.c, for example, is just an implementation detail: > > if we make Seabios support x2apic, that code can be changed to use a > > different data structure instead. > >=20 > [...] > > > > To try to make things less likely to break on the common, non-hotpl= ug > > > > case, this patch makes the Processor IDs be chosen this way: > > > >=20 > > > > - The CPUs present on boot get contiguous Processor IDs (even if the > > > > APIC IDs are not contiguous); > > > > - The remaining Processor declarations are going to associated to t= he > > > > remaining APIC IDs (immediately after the last present APIC ID), > > > > sequentially. This means that hotplugged CPUs may not get contigu= ous > > > > Processor declarations if their APIC IDs are not contiguous. > > > >=20 > > > I am curious what will happen if cpu will be hot plugged, than hibern= ate > > > and resume is done. After resume hot plugged cpu will have different > > > Processor ID in ACPI. This may or may not be a problem. > >=20 > > True. Keeping those tables stable after hotplug and hibernate may be a > > challenge. > >=20 > > Maybe it would be easier to just leave holes on the MADT and SSDT tables > > (making APIC ID and Processor ID always match), and hope no OS will be > > confused by the holes. >=20 > I am inclined to try this approach first (keep APIC ID =3D=3D ACPI Proces= sor > ID), to keep things simple in Seabios. I am hoping no OS will have > problems with the holes in the list of enabled Processor IDs. >=20 They shouldn't. -- Gleb.