From mboxrd@z Thu Jan 1 00:00:00 1970 Received: by 2002:a5d:4850:0:0:0:0:0 with SMTP id n16csp580009wrs; Mon, 12 Aug 2019 06:47:32 -0700 (PDT) X-Google-Smtp-Source: APXvYqxSB2oFrhgpDqp77D70TzJjQ1g1Qql5ynNBV0awv9KP0I6EiwWsUiKqbxiTOjNUjuAK7ceQ X-Received: by 2002:aa7:da97:: with SMTP id q23mr625186eds.132.1565617652264; Mon, 12 Aug 2019 06:47:32 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1565617652; cv=none; d=google.com; s=arc-20160816; b=Wi2cRoxqFGJwgPZG88Ya6l346yWQ2qJrrwoOAcwQxZGDQlTXRrR57sU38nqOY/4mNk CRyFcpXzr27QqQYnCuUyAD716SuzHZTjOecHLVi3HF8mQdsE8HTbPtDdTq+r5THNiIgg g4MSZ2Auci12WazInUqlNXxfifAGgYj0N/9oPq3g6z6AkCVjyusGTDxUWVcx6WeZSC2D 7uRCNZrBj82kitNG4eWFVDLFXqjw2y6i0ZMv30ZJG469fbOs8b9IZD3Nj3iCegldODfV t9D9lbRMu98Km9YpaYEdnvPXyeAnQpR15fsv0YrkV9YKD0P87hrbyC1BTOLKsDOSRXUz RegQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:cc:list-subscribe:list-help:list-post:list-archive :list-unsubscribe:list-id:precedence:subject :content-transfer-encoding:mime-version:references:in-reply-to :message-id:to:from:date; bh=bb4d5WbdnWONjtol8lkYZz2L/g3Q1JuGIc246+q2Up0=; b=ax97rRQu2xNsYTGnhUkxHk5EnJEj7TiNn1a073tYvuaE8gd2997rRS+5gJc0nITkzl xMQggJCiaXs5FSwd46+8lIc9n9eUTU8bH5nw7eNx2Wf5Flwihd7sci56MeSqr6YQmU3/ rRdTPHblU22qJVgX/HbY9GkabiTJRCBs0YYoQpUFb9LuoMc5lwbAA1UaXOcEr5p4HULq Kw83g05dQEESH/oWCRfc78qx0mMZ8iWu23hreb6SWKX3gnMRsOvqOnqsZ++zUsjBz4Rb MV6sTWhGLI/Qw74tmg5DWlkzMp0YxidKBnEEzVJECixEGIgomRZYKvXHP6BXLoty36vt gMlg== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of qemu-arm-bounces+alex.bennee=linaro.org@nongnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom="qemu-arm-bounces+alex.bennee=linaro.org@nongnu.org"; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=redhat.com Return-Path: Received: from lists.gnu.org (lists.gnu.org. [209.51.188.17]) by mx.google.com with ESMTPS id q19si35995930eju.347.2019.08.12.06.47.32 for (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256); Mon, 12 Aug 2019 06:47:32 -0700 (PDT) Received-SPF: pass (google.com: domain of qemu-arm-bounces+alex.bennee=linaro.org@nongnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; Authentication-Results: mx.google.com; spf=pass (google.com: domain of qemu-arm-bounces+alex.bennee=linaro.org@nongnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom="qemu-arm-bounces+alex.bennee=linaro.org@nongnu.org"; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: from localhost ([::1]:45692 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.86_2) (envelope-from ) id 1hxAfb-0004cq-CY for alex.bennee@linaro.org; Mon, 12 Aug 2019 09:47:31 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:56914) by lists.gnu.org with esmtp (Exim 4.86_2) (envelope-from ) id 1hxAfV-0004ck-29 for qemu-arm@nongnu.org; Mon, 12 Aug 2019 09:47:26 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1hxAfT-0005Kx-Kg for qemu-arm@nongnu.org; Mon, 12 Aug 2019 09:47:25 -0400 Received: from mx1.redhat.com ([209.132.183.28]:35892) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1hxAfT-0005Kg-Bt; Mon, 12 Aug 2019 09:47:23 -0400 Received: from smtp.corp.redhat.com (int-mx06.intmail.prod.int.phx2.redhat.com [10.5.11.16]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id E90EA307F5E4; Mon, 12 Aug 2019 13:47:21 +0000 (UTC) Received: from localhost (unknown [10.43.2.182]) by smtp.corp.redhat.com (Postfix) with ESMTP id 7C7F75C1D4; Mon, 12 Aug 2019 13:47:17 +0000 (UTC) Date: Mon, 12 Aug 2019 15:47:16 +0200 From: Igor Mammedov To: Shameerali Kolothum Thodi Message-ID: <20190812154716.44a6aff8@redhat.com> In-Reply-To: <5FC3163CFD30C246ABAA99954A238FA83F34B396@lhreml524-mbs.china.huawei.com> References: <20190726104519.23812-1-shameerali.kolothum.thodi@huawei.com> <20190726104519.23812-8-shameerali.kolothum.thodi@huawei.com> <20190806152136.3afbfb4b@redhat.com> <5FC3163CFD30C246ABAA99954A238FA83F34B396@lhreml524-mbs.china.huawei.com> MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit X-Scanned-By: MIMEDefang 2.79 on 10.5.11.16 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.5.110.44]); Mon, 12 Aug 2019 13:47:22 +0000 (UTC) X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] X-Received-From: 209.132.183.28 Subject: Re: [Qemu-arm] [Qemu-devel] [PATCH-for-4.2 v8 7/9] hw/arm/virt-acpi-build: Add PC-DIMM in SRAT X-BeenThere: qemu-arm@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: "peter.maydell@linaro.org" , "sameo@linux.intel.com" , "ard.biesheuvel@linaro.org" , "qemu-devel@nongnu.org" , Linuxarm , "shannon.zhaosl@gmail.com" , "qemu-arm@nongnu.org" , "xuwei \(O\)" , "sebastien.boeuf@intel.com" , "lersek@redhat.com" , "eric.auger@redhat.com" Errors-To: qemu-arm-bounces+alex.bennee=linaro.org@nongnu.org Sender: "Qemu-arm" X-TUID: tNDnR7K/+7oZ On Fri, 9 Aug 2019 16:02:39 +0000 Shameerali Kolothum Thodi wrote: > Hi Igor, > > > -----Original Message----- > > From: Qemu-devel > > [mailto:qemu-devel-bounces+shameerali.kolothum.thodi=huawei.com@nongn > > u.org] On Behalf Of Igor Mammedov > > Sent: 06 August 2019 14:22 > > To: Shameerali Kolothum Thodi > > Cc: peter.maydell@linaro.org; sameo@linux.intel.com; > > ard.biesheuvel@linaro.org; shannon.zhaosl@gmail.com; > > qemu-devel@nongnu.org; xuwei (O) ; Linuxarm > > ; eric.auger@redhat.com; qemu-arm@nongnu.org; > > sebastien.boeuf@intel.com; lersek@redhat.com > > Subject: Re: [Qemu-devel] [PATCH-for-4.2 v8 7/9] hw/arm/virt-acpi-build: Add > > PC-DIMM in SRAT > > > > On Fri, 26 Jul 2019 11:45:17 +0100 > > Shameer Kolothum wrote: > > > > > Generate Memory Affinity Structures for PC-DIMM ranges. > > > > > > Signed-off-by: Shameer Kolothum > > > Signed-off-by: Eric Auger > > > Reviewed-by: Igor Mammedov > > > --- > > > hw/arm/virt-acpi-build.c | 9 +++++++++ > > > 1 file changed, 9 insertions(+) > > > > > > diff --git a/hw/arm/virt-acpi-build.c b/hw/arm/virt-acpi-build.c > > > index 018b1e326d..75657caa36 100644 > > > --- a/hw/arm/virt-acpi-build.c > > > +++ b/hw/arm/virt-acpi-build.c > > > @@ -518,6 +518,7 @@ build_srat(GArray *table_data, BIOSLinker *linker, > > VirtMachineState *vms) > > > int i, srat_start; > > > uint64_t mem_base; > > > MachineClass *mc = MACHINE_GET_CLASS(vms); > > > + MachineState *ms = MACHINE(vms); > > > const CPUArchIdList *cpu_list = > > mc->possible_cpu_arch_ids(MACHINE(vms)); > > > > > > srat_start = table_data->len; > > > @@ -543,6 +544,14 @@ build_srat(GArray *table_data, BIOSLinker *linker, > > VirtMachineState *vms) > > > } > > > } > > > > > > + if (ms->device_memory) { > > > + numamem = acpi_data_push(table_data, sizeof *numamem); > > > + build_srat_memory(numamem, ms->device_memory->base, > > > + > > memory_region_size(&ms->device_memory->mr), > > > + nb_numa_nodes - 1, > > > + MEM_AFFINITY_HOTPLUGGABLE | > > MEM_AFFINITY_ENABLED); > > > + } > > > + > > > build_header(linker, table_data, (void *)(table_data->data + > > srat_start), > > > "SRAT", table_data->len - srat_start, 3, NULL, NULL); > > > } > > > > missing entry in > > tests/bios-tables-test-allowed-diff.h > > I can't find any SRAT file in tests/data/acpi/virt. Arm/virt doesn't have much > tests in bios-tables-test.c. So does it make any difference? acpi tests for arm/virt are new and are enabled only since 4.1, now it should be trivial to add extra cases for code you are adding. Since you're touching her SRAT, I'd suggest to enable 'numamem' and 'memhp' tests with this series (for example see: test_acpi_piix4_tcg_numamem/test_acpi_piix4_tcg_memhp). > > PS: > > I don't really know what ARM guest kernel expects but on x86 we had to enable > > numa > > for guest to figure out max_possible_pfn > > (see: in linux.git: 8dd330300197 / ec941c5ffede). > > From whatever I can find, doesn't look like there is any special handling of > max_possible_pfn in ARM64 world. The variable seems to be only updated > in acpi_numa_memory_affinity_init() > > https://elixir.bootlin.com/linux/v5.3-rc3/source/drivers/acpi/numa.c#L298 problem was that drivers (stub dma ops) (guest booted with RAM below 4Gb) were breaking when they received RAM buffers above 4Gb. To fix it we needed to turn on swiotlb if possible max PFN could be above 4Gb. That's where SRAT played its role to let guest know what possible max PFN could be. > Is there any way to test this in Guest to see whether this is actually a problem? from my x86 experience: 1. for linux: * start guest with RAM that not goes over 4Gb PFN mark (for example with -m 1Gb) and native drivers (not virtio ones see linux.git commit message ec941c5ffede4) * hotplug RAM to go over 4Gb boundary * stress test drivers (that should trigger various issues) (on x64 it were ATA and various usb drivers leading to data corruption and not working mouse in guests) 2. for Windows guests memory hotplug doesn't work at all unless NUMA is enabled. Based on above I'd assume, we need to turn on numa for ARM as well if memhp is enabled since SRAT is the only way of describing max possible RAM end to the guest OS. > Thanks, > Shameer > > > It's worth to check if we might need a patch for turning on NUMA > > (how to do it in QEMU see: auto_enable_numa_with_memhp) > From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-6.8 required=3.0 tests=HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_PATCH,MAILING_LIST_MULTI,SIGNED_OFF_BY,SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 7DC39C433FF for ; Mon, 12 Aug 2019 13:47:59 +0000 (UTC) Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 4F7DF20665 for ; Mon, 12 Aug 2019 13:47:59 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 4F7DF20665 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=redhat.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Received: from localhost ([::1]:45696 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.86_2) (envelope-from ) id 1hxAg2-00057u-Ib for qemu-devel@archiver.kernel.org; Mon, 12 Aug 2019 09:47:58 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:56926) by lists.gnu.org with esmtp (Exim 4.86_2) (envelope-from ) id 1hxAfY-0004d2-2x for qemu-devel@nongnu.org; Mon, 12 Aug 2019 09:47:29 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1hxAfW-0005ND-Gz for qemu-devel@nongnu.org; Mon, 12 Aug 2019 09:47:28 -0400 Received: from mx1.redhat.com ([209.132.183.28]:35892) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1hxAfT-0005Kg-Bt; Mon, 12 Aug 2019 09:47:23 -0400 Received: from smtp.corp.redhat.com (int-mx06.intmail.prod.int.phx2.redhat.com [10.5.11.16]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id E90EA307F5E4; Mon, 12 Aug 2019 13:47:21 +0000 (UTC) Received: from localhost (unknown [10.43.2.182]) by smtp.corp.redhat.com (Postfix) with ESMTP id 7C7F75C1D4; Mon, 12 Aug 2019 13:47:17 +0000 (UTC) Date: Mon, 12 Aug 2019 15:47:16 +0200 From: Igor Mammedov To: Shameerali Kolothum Thodi Message-ID: <20190812154716.44a6aff8@redhat.com> In-Reply-To: <5FC3163CFD30C246ABAA99954A238FA83F34B396@lhreml524-mbs.china.huawei.com> References: <20190726104519.23812-1-shameerali.kolothum.thodi@huawei.com> <20190726104519.23812-8-shameerali.kolothum.thodi@huawei.com> <20190806152136.3afbfb4b@redhat.com> <5FC3163CFD30C246ABAA99954A238FA83F34B396@lhreml524-mbs.china.huawei.com> MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit X-Scanned-By: MIMEDefang 2.79 on 10.5.11.16 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.5.110.44]); Mon, 12 Aug 2019 13:47:22 +0000 (UTC) X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] X-Received-From: 209.132.183.28 Subject: Re: [Qemu-devel] [PATCH-for-4.2 v8 7/9] hw/arm/virt-acpi-build: Add PC-DIMM in SRAT X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: "peter.maydell@linaro.org" , "sameo@linux.intel.com" , "ard.biesheuvel@linaro.org" , "qemu-devel@nongnu.org" , Linuxarm , "shannon.zhaosl@gmail.com" , "qemu-arm@nongnu.org" , "xuwei \(O\)" , "sebastien.boeuf@intel.com" , "lersek@redhat.com" , "eric.auger@redhat.com" Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: "Qemu-devel" On Fri, 9 Aug 2019 16:02:39 +0000 Shameerali Kolothum Thodi wrote: > Hi Igor, > > > -----Original Message----- > > From: Qemu-devel > > [mailto:qemu-devel-bounces+shameerali.kolothum.thodi=huawei.com@nongn > > u.org] On Behalf Of Igor Mammedov > > Sent: 06 August 2019 14:22 > > To: Shameerali Kolothum Thodi > > Cc: peter.maydell@linaro.org; sameo@linux.intel.com; > > ard.biesheuvel@linaro.org; shannon.zhaosl@gmail.com; > > qemu-devel@nongnu.org; xuwei (O) ; Linuxarm > > ; eric.auger@redhat.com; qemu-arm@nongnu.org; > > sebastien.boeuf@intel.com; lersek@redhat.com > > Subject: Re: [Qemu-devel] [PATCH-for-4.2 v8 7/9] hw/arm/virt-acpi-build: Add > > PC-DIMM in SRAT > > > > On Fri, 26 Jul 2019 11:45:17 +0100 > > Shameer Kolothum wrote: > > > > > Generate Memory Affinity Structures for PC-DIMM ranges. > > > > > > Signed-off-by: Shameer Kolothum > > > Signed-off-by: Eric Auger > > > Reviewed-by: Igor Mammedov > > > --- > > > hw/arm/virt-acpi-build.c | 9 +++++++++ > > > 1 file changed, 9 insertions(+) > > > > > > diff --git a/hw/arm/virt-acpi-build.c b/hw/arm/virt-acpi-build.c > > > index 018b1e326d..75657caa36 100644 > > > --- a/hw/arm/virt-acpi-build.c > > > +++ b/hw/arm/virt-acpi-build.c > > > @@ -518,6 +518,7 @@ build_srat(GArray *table_data, BIOSLinker *linker, > > VirtMachineState *vms) > > > int i, srat_start; > > > uint64_t mem_base; > > > MachineClass *mc = MACHINE_GET_CLASS(vms); > > > + MachineState *ms = MACHINE(vms); > > > const CPUArchIdList *cpu_list = > > mc->possible_cpu_arch_ids(MACHINE(vms)); > > > > > > srat_start = table_data->len; > > > @@ -543,6 +544,14 @@ build_srat(GArray *table_data, BIOSLinker *linker, > > VirtMachineState *vms) > > > } > > > } > > > > > > + if (ms->device_memory) { > > > + numamem = acpi_data_push(table_data, sizeof *numamem); > > > + build_srat_memory(numamem, ms->device_memory->base, > > > + > > memory_region_size(&ms->device_memory->mr), > > > + nb_numa_nodes - 1, > > > + MEM_AFFINITY_HOTPLUGGABLE | > > MEM_AFFINITY_ENABLED); > > > + } > > > + > > > build_header(linker, table_data, (void *)(table_data->data + > > srat_start), > > > "SRAT", table_data->len - srat_start, 3, NULL, NULL); > > > } > > > > missing entry in > > tests/bios-tables-test-allowed-diff.h > > I can't find any SRAT file in tests/data/acpi/virt. Arm/virt doesn't have much > tests in bios-tables-test.c. So does it make any difference? acpi tests for arm/virt are new and are enabled only since 4.1, now it should be trivial to add extra cases for code you are adding. Since you're touching her SRAT, I'd suggest to enable 'numamem' and 'memhp' tests with this series (for example see: test_acpi_piix4_tcg_numamem/test_acpi_piix4_tcg_memhp). > > PS: > > I don't really know what ARM guest kernel expects but on x86 we had to enable > > numa > > for guest to figure out max_possible_pfn > > (see: in linux.git: 8dd330300197 / ec941c5ffede). > > From whatever I can find, doesn't look like there is any special handling of > max_possible_pfn in ARM64 world. The variable seems to be only updated > in acpi_numa_memory_affinity_init() > > https://elixir.bootlin.com/linux/v5.3-rc3/source/drivers/acpi/numa.c#L298 problem was that drivers (stub dma ops) (guest booted with RAM below 4Gb) were breaking when they received RAM buffers above 4Gb. To fix it we needed to turn on swiotlb if possible max PFN could be above 4Gb. That's where SRAT played its role to let guest know what possible max PFN could be. > Is there any way to test this in Guest to see whether this is actually a problem? from my x86 experience: 1. for linux: * start guest with RAM that not goes over 4Gb PFN mark (for example with -m 1Gb) and native drivers (not virtio ones see linux.git commit message ec941c5ffede4) * hotplug RAM to go over 4Gb boundary * stress test drivers (that should trigger various issues) (on x64 it were ATA and various usb drivers leading to data corruption and not working mouse in guests) 2. for Windows guests memory hotplug doesn't work at all unless NUMA is enabled. Based on above I'd assume, we need to turn on numa for ARM as well if memhp is enabled since SRAT is the only way of describing max possible RAM end to the guest OS. > Thanks, > Shameer > > > It's worth to check if we might need a patch for turning on NUMA > > (how to do it in QEMU see: auto_enable_numa_with_memhp) >