From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-5.5 required=3.0 tests=DKIM_INVALID,DKIM_SIGNED, HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI,MENTIONS_GIT_HOSTING, SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 0DDB8C2BA2B for ; Thu, 9 Apr 2020 14:02:06 +0000 (UTC) Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id CAA1820730 for ; Thu, 9 Apr 2020 14:02:05 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="PX9RgirV" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org CAA1820730 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=redhat.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Received: from localhost ([::1]:50476 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1jMXkq-0007WA-Rv for qemu-devel@archiver.kernel.org; Thu, 09 Apr 2020 10:02:04 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:36939) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1jMXjq-000738-6w for qemu-devel@nongnu.org; Thu, 09 Apr 2020 10:01:06 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1jMXjn-0000z4-Ae for qemu-devel@nongnu.org; Thu, 09 Apr 2020 10:01:01 -0400 Received: from us-smtp-2.mimecast.com ([207.211.31.81]:24315 helo=us-smtp-delivery-1.mimecast.com) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1jMXjn-0000ye-6d for qemu-devel@nongnu.org; Thu, 09 Apr 2020 10:00:59 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1586440858; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=p/yydDbbDBzd89SJj19EC0vdO3nAxFL0aNMhSPc3Z0w=; b=PX9RgirV2o4v50FpzLcS/bAUrkAExAtkj1kz1PedZEgc/ypf9DDHZF4TOFaaSjhIeAgkdj MCGWnKTbd4MWvW6YEVXKwD4Simv3yykZzk60Uwuu3fVR1sbUajJUtFPoNKI00uYal4KHLt E6JM7dQOLmCSnaH75mpZ4RdwxeiAjTk= Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-405-jw3VxnBAN2ODhKsPNTWvQA-1; Thu, 09 Apr 2020 10:00:42 -0400 X-MC-Unique: jw3VxnBAN2ODhKsPNTWvQA-1 Received: from smtp.corp.redhat.com (int-mx08.intmail.prod.int.phx2.redhat.com [10.5.11.23]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id 95298DB60; Thu, 9 Apr 2020 14:00:41 +0000 (UTC) Received: from localhost (unknown [10.40.208.73]) by smtp.corp.redhat.com (Postfix) with ESMTP id 56F9E272C4; Thu, 9 Apr 2020 14:00:39 +0000 (UTC) Date: Thu, 9 Apr 2020 16:00:37 +0200 From: Igor Mammedov To: Philipp Eppelt <1871842@bugs.launchpad.net> Subject: Re: [Bug 1871842] [NEW] AMD CPUID leaf 0x8000'0008 reported number of cores inconsistent with ACPI.MADT Message-ID: <20200409160037.431ec776@redhat.com> In-Reply-To: <158643709116.17430.15995069125716778943.malonedeb@wampee.canonical.com> References: <158643709116.17430.15995069125716778943.malonedeb@wampee.canonical.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.84 on 10.5.11.23 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] [fuzzy] X-Received-From: 207.211.31.81 X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: babu.moger@amd.com, qemu-devel@nongnu.org Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: "Qemu-devel" On Thu, 09 Apr 2020 12:58:11 -0000 Philipp Eppelt <1871842@bugs.launchpad.net> wrote: > Public bug reported: > > Setup: > CPU: AMD EPYC-v2 or host's EPYC cpu > Linux 64-bit fedora host; Kernel version 5.5.15-200.fc31 > qemu version: self build > git-head: f3bac27cc1e303e1860cc55b9b6889ba39dee587 > config: Configured with: '../configure' '--target-list=x86_64-softmmu,mips64el-softmmu,mips64-softmmu,mipsel-softmmu,mips-softmmu,i386-softmmu,aarch64-softmmu,arm-softmmu' '--prefix=/opt/qemu-master' > > Cmdline: > qemu-system-x86_64 -kernel /home/peppelt/code/l4/internal/.build-x86_64/bin/amd64_gen/bootstrap -append "" -initrd "./fiasco/.build-x86_64/fiasco , ... " -serial stdio -nographic -monitor none -nographic -monitor none -cpu EPYC-v2 -m 4G -smp 4 > > Issue: > We are developing an microkernel operating system called L4Re. We recently got an AMD EPYC server for testing and we couldn't execute SMP tests of our system when running Linux + qemu + VM w/ L4Re. > In fact, the kernel did not recognize any APs at all. On AMD CPUs the kernel checks for the number of cores reported in CPUID leaf 0x8000_0008.ECX[NC] or [ApicIdSize]. [0][1] > > The physical machine reports for leaf 0x8000_0008: EAX: 0x3030 EBX: 0x18cf757 ECX: 0x703f EDX: 0x1000 > The lower four bits of ECX are the [NC] field and all set. > > When querying inside qemu with -enable-kvm -cpu host -smp 4 (basically as replacement and addition to the above cmdline) the CPUID leaf shows: EAX: 0x3024, EBX: 0x1001000, ECX: 0x0, EDX: 0x0 > Note, ECX is zero. Indicating that this is no SMP capabale CPU. > > I'm debugging it using my local machine and the QEMU provided EPYC-v2 > CPU model and it is reproducible there as well and reports: EAX: > 0x3028, EBX: 0x0, ECX: 0x0, EDX: 0x0 > > I checked other AMD based CPU models (phenom, opteron_g3/g5) and they behave the same. [2] shows the CPUID 0x8000'0008 handling in the QEMU source. > I believe that behavior here is wrong as ECX[NC] should report the number of cores per processor, as stated in the AMD manual [2] p.584. In my understanding -smp 4 should then lead to ECX[NC] = 0x3. > > The following table shows my findings with the -smp option: > Option | Qemu guest observed ECX value > -smp 4 | 0x0 > -smp 4,cores=4 | 0x3 > -smp 4,cores=2,thread=2 | 0x3 > -smp 4,cores=4,threads=2 | QEMU boot error: topology false. > > Now, I'm asking myself how the terminology of the AMD manual maps to QEMU's -smp option. > Obviously, nr_cores and nr_threads correspond to the cores and threads options on the cmdline and cores * threads <= 4 (in this example), but what corresponds the X in -smp X to? I'd say X corresponds to number of logical CPUs. Depending on presence of other options these are distributed among them in magical manner (see pc_smp_parse() for starters) > Querying 0x8000'0008 on the physical processor results in different > reports than quering QEMU's model as does it with -enable-kvm -cpu host. > > Furthermore, the ACPI.MADT shows 4 local APICs to be present while the > CPU leave reports a single core processor. it matches -smp X as it should be. > > This leads me to the conclusion that CPUID 0x8000'0008.ECX reports the > wrong number. CCed author of recent epyc patches who might know how AMD should work better than me. > > Please let me know, if you need more information from my side. > > > [0] https://github.com/kernkonzept/fiasco/blob/522ccc5f29ab120213cf02d71328e2b879cbbd19/src/kern/ia32/kernel_thread-ia32.cpp#L109 > [1] https://github.com/kernkonzept/fiasco/blob/522ccc5f29ab120213cf02d71328e2b879cbbd19/src/kern/ia32/cpu-ia32.cpp#L1120 > [2] https://github.com/qemu/qemu/blob/f2a8261110c32c4dccd84e774d8dd7a0524e00fb/target/i386/cpu.c#L5835 > [3] https://www.amd.com/system/files/TechDocs/24594.pdf > > ** Affects: qemu > Importance: Undecided > Status: New > From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-5.7 required=3.0 tests=HEADER_FROM_DIFFERENT_DOMAINS, MAILING_LIST_MULTI,MENTIONS_GIT_HOSTING,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 40351C2BA2B for ; Thu, 9 Apr 2020 14:16:34 +0000 (UTC) Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 10E3720771 for ; Thu, 9 Apr 2020 14:16:34 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 10E3720771 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=redhat.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Received: from localhost ([::1]:50654 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1jMXyr-0006Am-9F for qemu-devel@archiver.kernel.org; Thu, 09 Apr 2020 10:16:33 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:38708) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1jMXxz-0005if-NL for qemu-devel@nongnu.org; Thu, 09 Apr 2020 10:15:41 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1jMXxy-0008GA-1t for qemu-devel@nongnu.org; Thu, 09 Apr 2020 10:15:39 -0400 Received: from indium.canonical.com ([91.189.90.7]:58148) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1jMXxx-0008Fi-S4 for qemu-devel@nongnu.org; Thu, 09 Apr 2020 10:15:38 -0400 Received: from loganberry.canonical.com ([91.189.90.37]) by indium.canonical.com with esmtp (Exim 4.86_2 #2 (Debian)) id 1jMXxw-0004H0-CP for ; Thu, 09 Apr 2020 14:15:36 +0000 Received: from loganberry.canonical.com (localhost [127.0.0.1]) by loganberry.canonical.com (Postfix) with ESMTP id 3CD7B2E8108 for ; Thu, 9 Apr 2020 14:15:36 +0000 (UTC) MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Date: Thu, 09 Apr 2020 14:00:37 -0000 From: Igor To: qemu-devel@nongnu.org X-Launchpad-Notification-Type: bug X-Launchpad-Bug: product=qemu; status=New; importance=Undecided; assignee=None; X-Launchpad-Bug-Information-Type: Public X-Launchpad-Bug-Private: no X-Launchpad-Bug-Security-Vulnerability: no X-Launchpad-Bug-Commenters: e-philipp imammedo X-Launchpad-Bug-Reporter: Philipp Eppelt (e-philipp) X-Launchpad-Bug-Modifier: Igor (imammedo) References: <158643709116.17430.15995069125716778943.malonedeb@wampee.canonical.com> Message-Id: <20200409160037.431ec776@redhat.com> Subject: Re: [Bug 1871842] [NEW] AMD CPUID leaf 0x8000'0008 reported number of cores inconsistent with ACPI.MADT X-Launchpad-Message-Rationale: Subscriber (QEMU) @qemu-devel-ml X-Launchpad-Message-For: qemu-devel-ml Precedence: bulk X-Generated-By: Launchpad (canonical.com); Revision="2e26c9bbd21cdca248baaea29aeffb920afcc32a"; Instance="production-secrets-lazr.conf" X-Launchpad-Hash: 40dc5e50896b911b150bed5bad6faba8b0545a9a X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] X-Received-From: 91.189.90.7 X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: Bug 1871842 <1871842@bugs.launchpad.net> Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: "Qemu-devel" Message-ID: <20200409140037.zy1boFJ0dx_xl39ahQhMwKQMQUf8Ib6-RrBPH9x4VVk@z> On Thu, 09 Apr 2020 12:58:11 -0000 Philipp Eppelt <1871842@bugs.launchpad.net> wrote: > Public bug reported: > = > Setup: > CPU: AMD EPYC-v2 or host's EPYC cpu > Linux 64-bit fedora host; Kernel version 5.5.15-200.fc31 > qemu version: self build > git-head: f3bac27cc1e303e1860cc55b9b6889ba39dee587 > config: Configured with: '../configure' '--target-list=3Dx86_64-softmmu,m= ips64el-softmmu,mips64-softmmu,mipsel-softmmu,mips-softmmu,i386-softmmu,aar= ch64-softmmu,arm-softmmu' '--prefix=3D/opt/qemu-master' > = > Cmdline: = > qemu-system-x86_64 -kernel /home/peppelt/code/l4/internal/.build-x86_64/b= in/amd64_gen/bootstrap -append "" -initrd "./fiasco/.build-x86_64/fiasco , = ... " -serial stdio -nographic -monitor none -nographic -monitor none -cpu = EPYC-v2 -m 4G -smp 4 = > = > Issue: > We are developing an microkernel operating system called L4Re. We recentl= y got an AMD EPYC server for testing and we couldn't execute SMP tests of o= ur system when running Linux + qemu + VM w/ L4Re. > In fact, the kernel did not recognize any APs at all. On AMD CPUs the ker= nel checks for the number of cores reported in CPUID leaf 0x8000_0008.ECX[N= C] or [ApicIdSize]. [0][1] > = > The physical machine reports for leaf 0x8000_0008: EAX: 0x3030 EBX: 0x18= cf757 ECX: 0x703f EDX: 0x1000 > The lower four bits of ECX are the [NC] field and all set. > = > When querying inside qemu with -enable-kvm -cpu host -smp 4 (basically as= replacement and addition to the above cmdline) the CPUID leaf shows: EAX: = 0x3024, EBX: 0x1001000, ECX: 0x0, EDX: 0x0 > Note, ECX is zero. Indicating that this is no SMP capabale CPU. > = > I'm debugging it using my local machine and the QEMU provided EPYC-v2 > CPU model and it is reproducible there as well and reports: EAX: > 0x3028, EBX: 0x0, ECX: 0x0, EDX: 0x0 > = > I checked other AMD based CPU models (phenom, opteron_g3/g5) and they beh= ave the same. [2] shows the CPUID 0x8000'0008 handling in the QEMU source. > I believe that behavior here is wrong as ECX[NC] should report the number= of cores per processor, as stated in the AMD manual [2] p.584. In my under= standing -smp 4 should then lead to ECX[NC] =3D 0x3. > = > The following table shows my findings with the -smp option: > Option | Qemu guest observed ECX value > -smp 4 | 0x0 > -smp 4,cores=3D4 | 0x3 > -smp 4,cores=3D2,thread=3D2 | 0x3 > -smp 4,cores=3D4,threads=3D2 | QEMU boot error: topology false. > = > Now, I'm asking myself how the terminology of the AMD manual maps to QEMU= 's -smp option. > Obviously, nr_cores and nr_threads correspond to the cores and threads op= tions on the cmdline and cores * threads <=3D 4 (in this example), but what= corresponds the X in -smp X to? I'd say X corresponds to number of logical CPUs. Depending on presence of other options these are distributed among them in = magical manner (see pc_smp_parse() for starters) > Querying 0x8000'0008 on the physical processor results in different > reports than quering QEMU's model as does it with -enable-kvm -cpu host. > = > Furthermore, the ACPI.MADT shows 4 local APICs to be present while the > CPU leave reports a single core processor. it matches -smp X as it should be. > = > This leads me to the conclusion that CPUID 0x8000'0008.ECX reports the > wrong number. CCed author of recent epyc patches who might know how AMD should work bette= r than me. > = > Please let me know, if you need more information from my side. > = > = > [0] https://github.com/kernkonzept/fiasco/blob/522ccc5f29ab120213cf02d713= 28e2b879cbbd19/src/kern/ia32/kernel_thread-ia32.cpp#L109 > [1] https://github.com/kernkonzept/fiasco/blob/522ccc5f29ab120213cf02d713= 28e2b879cbbd19/src/kern/ia32/cpu-ia32.cpp#L1120 > [2] https://github.com/qemu/qemu/blob/f2a8261110c32c4dccd84e774d8dd7a0524= e00fb/target/i386/cpu.c#L5835 > [3] https://www.amd.com/system/files/TechDocs/24594.pdf > = > ** Affects: qemu > Importance: Undecided > Status: New > -- = You received this bug notification because you are a member of qemu- devel-ml, which is subscribed to QEMU. https://bugs.launchpad.net/bugs/1871842 Title: AMD CPUID leaf 0x8000'0008 reported number of cores inconsistent with ACPI.MADT Status in QEMU: New Bug description: Setup: CPU: AMD EPYC-v2 or host's EPYC cpu Linux 64-bit fedora host; Kernel version 5.5.15-200.fc31 qemu version: self build git-head: f3bac27cc1e303e1860cc55b9b6889ba39dee587 config: Configured with: '../configure' '--target-list=3Dx86_64-softmmu,m= ips64el-softmmu,mips64-softmmu,mipsel-softmmu,mips-softmmu,i386-softmmu,aar= ch64-softmmu,arm-softmmu' '--prefix=3D/opt/qemu-master' Cmdline: = qemu-system-x86_64 -kernel /home/peppelt/code/l4/internal/.build-x86_64/b= in/amd64_gen/bootstrap -append "" -initrd "./fiasco/.build-x86_64/fiasco , = ... " -serial stdio -nographic -monitor none -nographic -monitor none -cpu = EPYC-v2 -m 4G -smp 4 = Issue: We are developing an microkernel operating system called L4Re. We recentl= y got an AMD EPYC server for testing and we couldn't execute SMP tests of o= ur system when running Linux + qemu + VM w/ L4Re. In fact, the kernel did not recognize any APs at all. On AMD CPUs the ker= nel checks for the number of cores reported in CPUID leaf 0x8000_0008.ECX[N= C] or [ApicIdSize]. [0][1] The physical machine reports for leaf 0x8000_0008: EAX: 0x3030 EBX: 0x18= cf757 ECX: 0x703f EDX: 0x1000 The lower four bits of ECX are the [NC] field and all set. When querying inside qemu with -enable-kvm -cpu host -smp 4 (basically as= replacement and addition to the above cmdline) the CPUID leaf shows: EAX: = 0x3024, EBX: 0x1001000, ECX: 0x0, EDX: 0x0 Note, ECX is zero. Indicating that this is no SMP capabale CPU. I'm debugging it using my local machine and the QEMU provided EPYC-v2 CPU model and it is reproducible there as well and reports: EAX: 0x3028, EBX: 0x0, ECX: 0x0, EDX: 0x0 I checked other AMD based CPU models (phenom, opteron_g3/g5) and they beh= ave the same. [2] shows the CPUID 0x8000'0008 handling in the QEMU source. I believe that behavior here is wrong as ECX[NC] should report the number= of cores per processor, as stated in the AMD manual [2] p.584. In my under= standing -smp 4 should then lead to ECX[NC] =3D 0x3. The following table shows my findings with the -smp option: Option | Qemu guest observed ECX value -smp 4 | 0x0 -smp 4,cores=3D4 | 0x3 -smp 4,cores=3D2,thread=3D2 | 0x3 -smp 4,cores=3D4,threads=3D2 | QEMU boot error: topology false. Now, I'm asking myself how the terminology of the AMD manual maps to QEMU= 's -smp option. Obviously, nr_cores and nr_threads correspond to the cores and threads op= tions on the cmdline and cores * threads <=3D 4 (in this example), but what= corresponds the X in -smp X to? Querying 0x8000'0008 on the physical processor results in different reports than quering QEMU's model as does it with -enable-kvm -cpu host. Furthermore, the ACPI.MADT shows 4 local APICs to be present while the CPU leave reports a single core processor. This leads me to the conclusion that CPUID 0x8000'0008.ECX reports the wrong number. = Please let me know, if you need more information from my side. = [0] https://github.com/kernkonzept/fiasco/blob/522ccc5f29ab120213cf02d713= 28e2b879cbbd19/src/kern/ia32/kernel_thread-ia32.cpp#L109 [1] https://github.com/kernkonzept/fiasco/blob/522ccc5f29ab120213cf02d713= 28e2b879cbbd19/src/kern/ia32/cpu-ia32.cpp#L1120 [2] https://github.com/qemu/qemu/blob/f2a8261110c32c4dccd84e774d8dd7a0524= e00fb/target/i386/cpu.c#L5835 [3] https://www.amd.com/system/files/TechDocs/24594.pdf To manage notifications about this bug go to: https://bugs.launchpad.net/qemu/+bug/1871842/+subscriptions