Re: More than 255 vcpus Windows VM setup without viommu ?

[David Woodhouse <dwmw2@infradead.org>]

[-- Attachment #1: Type: text/plain, Size: 4240 bytes --]

On Sat, 2024-09-28 at 15:59 +0100, David Woodhouse wrote:
> On Tue, 2024-07-02 at 05:17 +0000, Sandesh Patel wrote:
> > 
> > The error is due to invalid MSIX routing entry passed to KVM.
> > 
> > The VM boots fine if we attach a vIOMMU but adding a vIOMMU can
> > potentially result in IO performance loss in guest.
> > I was interested to know if someone could boot a large Windows VM by
> > some other means like kvm-msi-ext-dest-id.
> 
> I think I may (with Alex Graf's suggestion) have found the Windows bug
> with Intel IOMMU.
> 
> It looks like when interrupt remapping is enabled with an AMD CPU,
> Windows *assumes* it can generate AMD-style MSI messages even if the
> IOMMU is an Intel one. If we put a little hack into the IOMMU interrupt
> remapping to make it interpret an AMD-style message, Windows seems to
> boot at least a little bit further than it did before...

Sadly, Windows has *more* bugs than that.

The previous hack extracted the Interrupt Remapping Table Entry (IRTE)
index from an AMD-style MSI message, and looked it up in the Intel
IOMMU's IR Table.

That works... for the MSIs generated by the I/O APIC.

However... in the Intel IOMMU model, there is a single global IRT, and
each entry specifies which devices are permitted to invoke it. The AMD
model is slightly nicer, in that it allows a per-device IRT.

So for a PCI device, Windows just seems to configure each MSI vector in
order, with IRTE#0, 1, onwards. Because it's a per-device number space,
right? Which means that first MSI vector on a PCI device gets aliased
to IRQ#0 on the I/O APIC.

I dumped the whole IRT, and it isn't just that Windows is using the
wrong index; it hasn't even set up the correct destination in *any* of
the entries. So we can't even do a nasty trick like scanning and
funding the Nth entry which is valid for a particular source-id.

Happily, Windows has *more* bugs than that... if I run with
`-cpu host,+hv-avic' then it puts the high bits of the target APIC ID
into the high bits of the MSI address. This *ought* to mean that MSIs
from device miss the APIC (at 0x00000000FEExxxxx) and scribble over
guest memory at addresses like 0x1FEE00004. But we can add yet
*another* hack to catch that. For now I just hacked it to move the low
7 extra bits in to the "right" place for the 15-bit extension.

--- a/hw/pci/pci.c
+++ b/hw/pci/pci.c
@@ -361,6 +361,14 @@ static void pci_msi_trigger(PCIDevice *dev, MSIMessage msg)
         return;
     }
     attrs.requester_id = pci_requester_id(dev);
+    printf("Send MSI 0x%lx/0x%x from 0x%x\n", msg.address, msg.data, attrs.requester_id);
+    if (msg.address >> 32) {
+        uint64_t ext_id = msg.address >> 32;
+        msg.address &= 0xffffffff;
+        msg.address |= ext_id << 5;
+        printf("Now 0x%lx/0x%x with ext_id %lx\n", msg.address, msg.data, ext_id);
+    }
+        
     address_space_stl_le(&dev->bus_master_as, msg.address, msg.data,
                          attrs, NULL);
 }

We also need to stop forcing Windows to use logical mode, and force it
to use physical mode instead:

--- a/hw/i386/acpi-build.c
+++ b/hw/i386/acpi-build.c
@@ -158,7 +158,7 @@ static void init_common_fadt_data(MachineState *ms, Object *o,
              * used
              */
             ((ms->smp.max_cpus > 8) ?
-                        (1 << ACPI_FADT_F_FORCE_APIC_CLUSTER_MODEL) : 0),
+                        (1 << ACPI_FADT_F_FORCE_APIC_PHYSICAL_DESTINATION_MODE) : 0),
         .int_model = 1 /* Multiple APIC */,
         .rtc_century = RTC_CENTURY,
         .plvl2_lat = 0xfff /* C2 state not supported */,


So now, with *no* IOMMU configured, Windows Server 2022 is booting and
using CPUs > 255:
  Send MSI 0x1fee01000/0x41b0 from 0xfa
  Now 0xfee01020/0x41b0 with ext_id 1

That trick obviously can't work the the I/O APIC, but I haven't managed
to persuade Windows to target I/O APIC interrupts at any CPU other than
#0 yet. I'm trying to make QEMU run with *only* higher APIC IDs, to
test.

It may be that we need to advertise an Intel IOMMU that *only* has the
I/O APIC behind it, and all the actual PCI devices are direct, so we
can abuse that last Windows bug.

[-- Attachment #2: smime.p7s --]
[-- Type: application/pkcs7-signature, Size: 5965 bytes --]