From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from eggs.gnu.org ([2001:4830:134:3::10]:43089) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1Vepw0-0002do-8I for qemu-devel@nongnu.org; Fri, 08 Nov 2013 12:33:32 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1Vepvv-0004iX-4i for qemu-devel@nongnu.org; Fri, 08 Nov 2013 12:33:28 -0500 Received: from mx1.redhat.com ([209.132.183.28]:43745) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1Vepvu-0004iI-SM for qemu-devel@nongnu.org; Fri, 08 Nov 2013 12:33:23 -0500 Date: Fri, 8 Nov 2013 18:33:12 +0100 From: Igor Mammedov Message-ID: <20131108183312.7b5cde34@thinkpad> In-Reply-To: <20131108102212.GA2790@shadowkeep> References: <1383828119-2181-1-git-send-email-vasilis.liaskovitis@profitbricks.com> <20131107130342.GA2212@redhat.com> <20131108102212.GA2790@shadowkeep> Mime-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Subject: Re: [Qemu-devel] [RFC PATCH] i386: Add _PXM method to ACPI CPU objects List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , To: Vasilis Liaskovitis Cc: thilo.fromm@profitbricks.com, kevin@koconnor.net, seabios@seabios.org, qemu-devel@nongnu.org, "Michael S. Tsirkin" On Fri, 8 Nov 2013 12:22:12 +0200 Vasilis Liaskovitis wrote: > Hi, >=20 > On Thu, Nov 07, 2013 at 03:03:42PM +0200, Michael S. Tsirkin wrote: > > On Thu, Nov 07, 2013 at 01:41:59PM +0100, Vasilis Liaskovitis wrote: > > > This patch adds a _PXM method to ACPI CPU objects for the pc machine.= The _PXM > > > value is derived from the passed in guest info, same way as CPU SRAT = entries. > > >=20 > > > The motivation for this patch is a CPU hot-unplug/hot-plug bug observ= ed when > > > using a 3.11 linux guest kernel on a multi-NUMA node qemu/kvm VM. The= linux > > > guest kernel parses the SRAT CPU entries at boot time and stores them= in the > > > array __apicid_to_node. When a CPU is hot-removed, the linux guest ke= rnel > > > resets the removed CPU's __apicid_to_node entry to NO_NUMA_NODE (kern= el commit > > > c4c60524). When the removed cpu is hot-added again, the linux kernel = looks up > > > the hot-added cpu object's _PXM method instead of somehow re-discover= ing the > > > SRAT entry info. With current qemu/seabios, the _PXM method is not fo= und, and > > > the CPU is thus hot-plugged in the default NUMA node 0. (The problem = does not > > > show up on initial hotplug of a cpu; the PXM method is still not foun= d in this > > > case, but the kernel still has the correct proximity value from the C= PU's SRAT > > > entry stored in __apicid_to_node) > > >=20 > > > ACPI spec mentions that the _PXM method is the correct way to determi= ne > > > proximity information at hot-add time. > >=20 > > Where does it say this? > > I found this: > > If the Local APIC ID / Local SAPIC ID / Local x2APIC ID of a dynamically > > added processor is not present in the System Resource Affinity Table > > (SRAT), a _PXM object must exist for the processor=E2=80=99s device or = one of > > its ancestors in the ACPI Namespace. > >=20 > > Does this mean that linux is buggy, and should be fixed up to look up > > the apic ID in SRAT? >=20 > The quote above suggests that if SRAT is absent, _PXM should be present. > Seabios/qemu provide SRAT entries, and no _PXM. The fact that the kernel > resets the parse SRAT info on hot-remove time looks like a kernel problem. >=20 > But As Toshi Kani mentioned in the original thread, here is a quote from = ACPI > 5.0, stating _PXM and only _PXM should be used at hot-plug time: >=20 > =3D=3D=3D > 17.2.1 System Resource Affinity Table Definition >=20 > This optional System Resource Affinity Table (SRAT) provides the boot > time description of the processor and memory ranges belonging to a > system locality. OSPM will consume the SRAT only at boot time. OSPM > should use _PXM for any devices that are hot-added into the system after > boot up. > =3D=3D=3D=3D >=20 > So in this sense, the kernel is correct (kernel only uses _PXM at hot-plu= g time) > , and qemu/Seabios should have _PXM methods for hot operations. in terms of RFC SHOULD doesn't mean MUST, and in my interpretation of above= is that SRAT parsed once but it doesn't mean that OS should forget data from i= t. Anyway we surely can have both in QEMU. >=20 > >=20 > > > So far, qemu/seabios do not provide this > > > method for CPUs. So regardless of kernel behaviour, it is a good idea= to add > > > this _PXM method. Since ACPI table generation has recently been moved= from > > > seabios to qemu, we do this in qemu. > > >=20 > > > Note that the above hot-remove/hot-add scenario has been tested on an= older > > > qemu + non-upstreamed patches for cpu hot-removal support, and not on= qemu > > > master (since cpu-del support is still not on master). The only testi= ng done > > > with qemu/seabios master and this patch, are successful boots of mult= i-node > > > linux and windows8 guests. > > >=20 > > > For the initial discussion on seabios and linux-acpi lists see > > > http://www.spinics.net/lists/linux-acpi/msg47058.html > > >=20 > > > Signed-off-by: Vasilis Liaskovitis > > > Reviewed-by: Thilo Fromm > >=20 > > Even if this is a linux bug, I have no issue with working around > > it in qemu. > >=20 > > But I think proper testing needs to be done with rebased upport for cpu= -del. >=20 > Ok, I can try to rebase cpu-del support for testing. If there are cpu-del= bits > already somewhere (Igor?) and not merged yet, please point me to them. >=20 > >=20 > > > --- > > > hw/i386/acpi-build.c | 2 ++ > > > hw/i386/ssdt-proc.dsl | 2 ++ > > > 2 files changed, 4 insertions(+) > > >=20 > > > diff --git a/hw/i386/acpi-build.c b/hw/i386/acpi-build.c > > > index 6cfa044..9373f5e 100644 > > > --- a/hw/i386/acpi-build.c > > > +++ b/hw/i386/acpi-build.c > > > @@ -603,6 +603,7 @@ static inline char acpi_get_hex(uint32_t val) > > > #define ACPI_PROC_OFFSET_CPUHEX (*ssdt_proc_name - *ssdt_proc_start = + 2) > > > #define ACPI_PROC_OFFSET_CPUID1 (*ssdt_proc_name - *ssdt_proc_start = + 4) > > > #define ACPI_PROC_OFFSET_CPUID2 (*ssdt_proc_id - *ssdt_proc_start) > > > +#define ACPI_PROC_OFFSET_CPUPXM (*ssdt_proc_pxm - *ssdt_proc_start) > > > #define ACPI_PROC_SIZEOF (*ssdt_proc_end - *ssdt_proc_start) > > > #define ACPI_PROC_AML (ssdp_proc_aml + *ssdt_proc_start) > > > =20 > > > @@ -724,6 +725,7 @@ build_ssdt(GArray *table_data, GArray *linker, > > > proc[ACPI_PROC_OFFSET_CPUHEX+1] =3D acpi_get_hex(i); > > > proc[ACPI_PROC_OFFSET_CPUID1] =3D i; > > > proc[ACPI_PROC_OFFSET_CPUID2] =3D i; > > > + proc[ACPI_PROC_OFFSET_CPUPXM] =3D guest_info->node_cpu[i= ]; > > > } > > > =20 > > > /* build this code: > > > diff --git a/hw/i386/ssdt-proc.dsl b/hw/i386/ssdt-proc.dsl > > > index 8229bfd..7eef8b2 100644 > > > --- a/hw/i386/ssdt-proc.dsl > > > +++ b/hw/i386/ssdt-proc.dsl > > > @@ -47,6 +47,8 @@ DefinitionBlock ("ssdt-proc.aml", "SSDT", 0x01, "BX= PC", "BXSSDT", 0x1) > > > * also updating the C code. > > > */ > > > Name(_HID, "ACPI0007") > > > + ACPI_EXTRACT_NAME_BYTE_CONST ssdt_proc_pxm > > > + Name(_PXM, 0xAA) > >=20 > > The ACPI spec says this should be a DWORD value: > >=20 > > Return Value: > > An Integer (DWORD) containing a proximity domain identifier. >=20 > ok, I 'll change this. >=20 > thanks, >=20 > - Vasilis --=20 Regards, Igor