From: David Woodhouse <dwmw2@infradead.org>
To: Sandesh Patel <sandesh.patel@nutanix.com>
Cc: "qemu-devel@nongnu.org" <qemu-devel@nongnu.org>,
Rob Scheepens <rob.scheepens@nutanix.com>,
Prerna Saxena <confluence@nutanix.com>,
Dexuan Cui <decui@microsoft.com>, Alexander Graf <alex@csgraf.de>
Subject: Re: More than 255 vcpus Windows VM setup without viommu ?
Date: Thu, 11 Jul 2024 08:26:30 +0100 [thread overview]
Message-ID: <b37283824ff4b7c6cc3a0c51199e6aa9b4b658a3.camel@infradead.org> (raw)
In-Reply-To: <39b92288418839e9cdba447705a44a8e216dedc3.camel@infradead.org>
[-- Attachment #1: Type: text/plain, Size: 9613 bytes --]
On Mon, 2024-07-08 at 10:13 +0100, David Woodhouse wrote:
> On Wed, 2024-07-03 at 16:01 +0000, Sandesh Patel wrote:
> > >
> > > Interesting. What exactly has Windows *done* in those MSI entries?
> > > That might give a clue about how to support it.
I repeated my experiment with interrupt-only remapping (no DMA
remapping). On two hosts:
vendor_id : AuthenticAMD
cpu family : 25
model : 17
model name : AMD EPYC 9R14 96-Core Processor
vendor_id : GenuineIntel
cpu family : 6
model : 143
model name : Intel(R) Xeon(R) Platinum 8488C
I used identical command lines on both, and on each host I got the same
result with all of '-cpu host', '-cpu EPYC' and -cpu Skylake-Server'.
It's the *host* that makes the difference, not the CPUID presented to
the guest.
On the Intel host it boots:
$ ./qemu-system-x86_64 -accel kvm,kernel-irqchip=split -cdrom ~/Win10_22H2_EnglishInternational_x64v1.iso -m 16G -M q35 -smp 2,cores=12,threads=2,maxcpus=288 -accel kvm,kernel-irqchip=split -device intel-iommu,intremap=on,dma-translation=off -cpu Skylake-Server --trace vtd_ir_\* --trace apic\* --trace kvm_irqchip\*
qemu-system-x86_64: -accel kvm,kernel-irqchip=split: warning: Number of hotpluggable cpus requested (288) exceeds the recommended cpus supported by KVM (192)
kvm_irqchip_add_msi_route dev N/A vector 0 virq 0
kvm_irqchip_add_msi_route dev N/A vector 0 virq 1
kvm_irqchip_add_msi_route dev N/A vector 0 virq 2
kvm_irqchip_add_msi_route dev N/A vector 0 virq 3
kvm_irqchip_add_msi_route dev N/A vector 0 virq 4
kvm_irqchip_add_msi_route dev N/A vector 0 virq 5
kvm_irqchip_add_msi_route dev N/A vector 0 virq 6
kvm_irqchip_add_msi_route dev N/A vector 0 virq 7
kvm_irqchip_add_msi_route dev N/A vector 0 virq 8
kvm_irqchip_add_msi_route dev N/A vector 0 virq 9
kvm_irqchip_add_msi_route dev N/A vector 0 virq 10
kvm_irqchip_add_msi_route dev N/A vector 0 virq 11
kvm_irqchip_add_msi_route dev N/A vector 0 virq 12
kvm_irqchip_add_msi_route dev N/A vector 0 virq 13
kvm_irqchip_add_msi_route dev N/A vector 0 virq 14
kvm_irqchip_add_msi_route dev N/A vector 0 virq 15
kvm_irqchip_add_msi_route dev N/A vector 0 virq 16
kvm_irqchip_add_msi_route dev N/A vector 0 virq 17
kvm_irqchip_add_msi_route dev N/A vector 0 virq 18
kvm_irqchip_add_msi_route dev N/A vector 0 virq 19
kvm_irqchip_add_msi_route dev N/A vector 0 virq 20
kvm_irqchip_add_msi_route dev N/A vector 0 virq 21
kvm_irqchip_add_msi_route dev N/A vector 0 virq 22
kvm_irqchip_add_msi_route dev N/A vector 0 virq 23
kvm_irqchip_commit_routes
qemu-system-x86_64: warning: host doesn't support requested feature: CPUID.07H:EBX.hle [bit 4]
qemu-system-x86_64: warning: host doesn't support requested feature: CPUID.07H:EBX.rtm [bit 11]
qemu-system-x86_64: warning: host doesn't support requested feature: CPUID.07H:EBX.hle [bit 4]
qemu-system-x86_64: warning: host doesn't support requested feature: CPUID.07H:EBX.rtm [bit 11]
kvm_irqchip_commit_routes
kvm_irqchip_commit_routes
kvm_irqchip_commit_routes
kvm_irqchip_commit_routes
kvm_irqchip_commit_routes
kvm_irqchip_commit_routes
kvm_irqchip_commit_routes
kvm_irqchip_commit_routes
kvm_irqchip_commit_routes
kvm_irqchip_commit_routes
kvm_irqchip_commit_routes
kvm_irqchip_commit_routes
kvm_irqchip_commit_routes
kvm_irqchip_commit_routes
kvm_irqchip_commit_routes
kvm_irqchip_commit_routes
kvm_irqchip_commit_routes
kvm_irqchip_commit_routes
kvm_irqchip_commit_routes
kvm_irqchip_commit_routes
kvm_irqchip_commit_routes
kvm_irqchip_commit_routes
kvm_irqchip_commit_routes
kvm_irqchip_commit_routes
kvm_irqchip_commit_routes
kvm_irqchip_commit_routes
kvm_irqchip_commit_routes
kvm_irqchip_commit_routes
kvm_irqchip_commit_routes
kvm_irqchip_commit_routes
kvm_irqchip_commit_routes
kvm_irqchip_commit_routes
kvm_irqchip_commit_routes
kvm_irqchip_commit_routes
kvm_irqchip_commit_routes
kvm_irqchip_commit_routes
kvm_irqchip_commit_routes
kvm_irqchip_commit_routes
kvm_irqchip_commit_routes
kvm_irqchip_commit_routes
kvm_irqchip_commit_routes
kvm_irqchip_commit_routes
kvm_irqchip_commit_routes
kvm_irqchip_commit_routes
kvm_irqchip_commit_routes
kvm_irqchip_commit_routes
kvm_irqchip_commit_routes
kvm_irqchip_commit_routes
kvm_irqchip_commit_routes
kvm_irqchip_commit_routes
kvm_irqchip_commit_routes
kvm_irqchip_commit_routes
vtd_ir_enable enable 1
kvm_irqchip_commit_routes
kvm_irqchip_commit_routes
vtd_ir_remap_msi_req addr 0xfee00010 data 0xd1 sid 0xffff do_fault 0
vtd_ir_irte_get index 0 low 0x0 high 0x100d10005
vtd_ir_remap index 0 trigger 0 vector 209 deliver 0 dest 0x1 mode 1
vtd_ir_remap_type IOAPIC
vtd_ir_remap_msi (addr 0xfee00010, data 0xd1) -> (addr 0xfee01004, data 0x40d1)
kvm_irqchip_update_msi_route Updating MSI route virq=2
kvm_irqchip_commit_routes
vtd_ir_remap_msi_req addr 0xfee00010 data 0xd1 sid 0xffff do_fault 0
vtd_ir_irte_get index 0 low 0x0 high 0x100d10005
vtd_ir_remap index 0 trigger 0 vector 209 deliver 0 dest 0x1 mode 1
vtd_ir_remap_type IOAPIC
vtd_ir_remap_msi (addr 0xfee00010, data 0xd1) -> (addr 0xfee01004, data 0x40d1)
kvm_irqchip_update_msi_route Updating MSI route virq=2
kvm_irqchip_commit_routes
...
On the AMD host it stops at the boot splash screen:
$ ./qemu-system-x86_64 -accel kvm,kernel-irqchip=split -cdrom ~/Win10_22H2_EnglishInternational_x64v1.iso -m 16G -M q35 -smp 2,cores=12,threads=2,maxcpus=288 -accel kvm,kernel-irqchip=split -device intel-iommu,intremap=on,dma-translation=off -cpu Skylake-Server --trace vtd_ir_\* --trace apic\* --trace kvm_irqchip\*
qemu-system-x86_64: -accel kvm,kernel-irqchip=split: warning: Number of hotpluggable cpus requested (288) exceeds the recommended cpus supported by KVM (192)
kvm_irqchip_add_msi_route dev N/A vector 0 virq 0
kvm_irqchip_add_msi_route dev N/A vector 0 virq 1
kvm_irqchip_add_msi_route dev N/A vector 0 virq 2
kvm_irqchip_add_msi_route dev N/A vector 0 virq 3
kvm_irqchip_add_msi_route dev N/A vector 0 virq 4
kvm_irqchip_add_msi_route dev N/A vector 0 virq 5
kvm_irqchip_add_msi_route dev N/A vector 0 virq 6
kvm_irqchip_add_msi_route dev N/A vector 0 virq 7
kvm_irqchip_add_msi_route dev N/A vector 0 virq 8
kvm_irqchip_add_msi_route dev N/A vector 0 virq 9
kvm_irqchip_add_msi_route dev N/A vector 0 virq 10
kvm_irqchip_add_msi_route dev N/A vector 0 virq 11
kvm_irqchip_add_msi_route dev N/A vector 0 virq 12
kvm_irqchip_add_msi_route dev N/A vector 0 virq 13
kvm_irqchip_add_msi_route dev N/A vector 0 virq 14
kvm_irqchip_add_msi_route dev N/A vector 0 virq 15
kvm_irqchip_add_msi_route dev N/A vector 0 virq 16
kvm_irqchip_add_msi_route dev N/A vector 0 virq 17
kvm_irqchip_add_msi_route dev N/A vector 0 virq 18
kvm_irqchip_add_msi_route dev N/A vector 0 virq 19
kvm_irqchip_add_msi_route dev N/A vector 0 virq 20
kvm_irqchip_add_msi_route dev N/A vector 0 virq 21
kvm_irqchip_add_msi_route dev N/A vector 0 virq 22
kvm_irqchip_add_msi_route dev N/A vector 0 virq 23
kvm_irqchip_commit_routes
qemu-system-x86_64: warning: host doesn't support requested feature: CPUID.07H:EBX.hle [bit 4]
qemu-system-x86_64: warning: host doesn't support requested feature: CPUID.07H:EBX.rtm [bit 11]
qemu-system-x86_64: warning: This family of AMD CPU doesn't support hyperthreading(2). Please configure -smp options properly or try enabling topoext feature.
qemu-system-x86_64: warning: host doesn't support requested feature: CPUID.07H:EBX.hle [bit 4]
qemu-system-x86_64: warning: host doesn't support requested feature: CPUID.07H:EBX.rtm [bit 11]
kvm_irqchip_commit_routes
kvm_irqchip_commit_routes
kvm_irqchip_commit_routes
kvm_irqchip_commit_routes
kvm_irqchip_commit_routes
kvm_irqchip_commit_routes
kvm_irqchip_commit_routes
kvm_irqchip_commit_routes
kvm_irqchip_commit_routes
kvm_irqchip_commit_routes
kvm_irqchip_commit_routes
kvm_irqchip_commit_routes
kvm_irqchip_commit_routes
kvm_irqchip_commit_routes
kvm_irqchip_commit_routes
kvm_irqchip_commit_routes
kvm_irqchip_commit_routes
kvm_irqchip_commit_routes
kvm_irqchip_commit_routes
kvm_irqchip_commit_routes
kvm_irqchip_commit_routes
kvm_irqchip_commit_routes
kvm_irqchip_commit_routes
kvm_irqchip_commit_routes
kvm_irqchip_commit_routes
kvm_irqchip_commit_routes
kvm_irqchip_commit_routes
kvm_irqchip_commit_routes
kvm_irqchip_commit_routes
kvm_irqchip_commit_routes
kvm_irqchip_commit_routes
kvm_irqchip_commit_routes
kvm_irqchip_commit_routes
kvm_irqchip_commit_routes
kvm_irqchip_commit_routes
kvm_irqchip_commit_routes
kvm_irqchip_commit_routes
kvm_irqchip_commit_routes
kvm_irqchip_commit_routes
kvm_irqchip_commit_routes
kvm_irqchip_commit_routes
kvm_irqchip_commit_routes
kvm_irqchip_commit_routes
kvm_irqchip_commit_routes
kvm_irqchip_commit_routes
kvm_irqchip_commit_routes
kvm_irqchip_commit_routes
kvm_irqchip_commit_routes
kvm_irqchip_commit_routes
kvm_irqchip_commit_routes
kvm_irqchip_commit_routes
kvm_irqchip_commit_routes
vtd_ir_enable enable 1
kvm_irqchip_commit_routes
kvm_irqchip_commit_routes
vtd_ir_remap_msi_req addr 0xfee00004 data 0x0 sid 0xffff do_fault 0
vtd_ir_remap_msi (addr 0xfee00004, data 0x0) -> (addr 0xfee00004, data 0x0)
kvm_irqchip_update_msi_route Updating MSI route virq=2
kvm_irqchip_commit_routes
^Cqemu: terminating on signal 2
It looks like Windows is putting something bogus into the IOAPIC RTE for irq2?
In the successful boot that was 0xfee00010 / 0xd1; vector 209 on CPU0.
[-- Attachment #2: smime.p7s --]
[-- Type: application/pkcs7-signature, Size: 5965 bytes --]
next prev parent reply other threads:[~2024-07-11 7:27 UTC|newest]
Thread overview: 17+ messages / expand[flat|nested] mbox.gz Atom feed top
2024-07-02 5:17 More than 255 vcpus Windows VM setup without viommu ? Sandesh Patel
2024-07-02 9:04 ` David Woodhouse
2024-07-03 16:01 ` Sandesh Patel
2024-07-08 9:13 ` David Woodhouse
2024-07-11 7:26 ` David Woodhouse [this message]
2024-07-11 11:23 ` David Woodhouse
2024-07-11 11:52 ` Sandesh Patel
2024-07-16 5:13 ` Sandesh Patel
2024-07-24 9:22 ` David Woodhouse
2024-08-01 10:28 ` Sandesh Patel
2024-09-28 14:59 ` David Woodhouse
2024-09-30 15:50 ` David Woodhouse
2024-10-02 11:33 ` Igor Mammedov
2024-10-02 15:30 ` David Woodhouse
2024-10-01 13:33 ` Daniel P. Berrangé
2024-10-01 16:37 ` David Woodhouse
-- strict thread matches above, loose matches on Subject: below --
2024-07-02 7:20 Sandesh Patel
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=b37283824ff4b7c6cc3a0c51199e6aa9b4b658a3.camel@infradead.org \
--to=dwmw2@infradead.org \
--cc=alex@csgraf.de \
--cc=confluence@nutanix.com \
--cc=decui@microsoft.com \
--cc=qemu-devel@nongnu.org \
--cc=rob.scheepens@nutanix.com \
--cc=sandesh.patel@nutanix.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).