Re: Xen-unstable boot panic due to changeset 26517 AMD, IOMMU: Clean up old entries in remapping tables when creating new one

[Suravee Suthikulanit <suravee.suthikulpanit@amd.com>]

On 2/6/2013 6:52 AM, Jan Beulich wrote:
>>>> On 06.02.13 at 12:24, Sander Eikelenboom <linux@eikelenboom.it> wrote:
>> Hmm with the patch it does boot, but disables the I/O virtualization.
> Good. While, as said before, I still don't understand why it didn't
> crash earlier without that patch, I'm glad it's fixed. Will post the
> patch for inclusion momentarily.
>
>> Output of xl-dmesg attached, do you still need a xen-sums of the situation
>> without the debug patch (where it does crash) ?
> And you can't expect much else with broken ACPI tables:
>
> (XEN) AMD-Vi: IVHD Device Entry: type 0x48 id 0 flags 0xd7
> (XEN) AMD-Vi: IVHD Special: 0000:00:14.0 variety 0x2 handle 0
>
> This is a HPET entry.
>
> (XEN) AMD-Vi: IVHD Device Entry: type 0x48 id 0 flags 0
> (XEN) AMD-Vi: IVHD Special: 0000:00:00.1 variety 0x1 handle 0x7
>
> And this is an entry for IO-APIC #2 (ID 7), whereas FADT says
>
> (XEN) ACPI: IOAPIC (id[0x06] address[0xfec00000] gsi_base[0])
> (XEN) IOAPIC[0]: apic_id 6, version 33, address 0xfec00000, GSI 0-23
> (XEN) ACPI: IOAPIC (id[0x07] address[0xfec20000] gsi_base[24])
> (XEN) IOAPIC[1]: apic_id 7, version 33, address 0xfec20000, GSI 24-55
>
> so the IOMMU table is lacking an entry for the first IO-APIC, and
> without that we can't set up per-device interrupt remapping (in
> which case we choose to disable the IOMMU altogether, albeit it
> had been questioned whether that isn't making a bad situation
> worse in some cases).
>
> If you want the IOMMU back (at the price of re-opening the
> security issue described in XSA-36), you'd have to pass
> "iommu=amd-iommu-perdev-intremap" to the hypervisor.
>
> Jan

Jan,

It seems that all the recent issues with the AMD IOMMU regarding IOAPIC are
mainly caused by mismatch information from IVRS and MADT. Xen sets up "nr_ioapics"
by checking the number of IOAPICs reported in MADT, while the amd/iommu_acpi.c
code uses information from the IVHD entries of the IVRS to initialize IOMMU.

Most of the issues we are seeing are often triggered when platform BIOS decides
to disable one of the two IOAPICs in the RD890s configuration. I am trying to
summarize the cases here:


CASE1: BIOS disable the IOAPIC in the southbridge (SB8X0 chipset)
This is the case we are seeing here with the AMD 890-FX motherboard.
Here, the MADT is reporting 2 IOAPICs as shown by the message:

(XEN) ACPI: IOAPIC (id[0x06] address[0xfec00000] gsi_base[0])
(XEN) IOAPIC[0]: apic_id 6, version 33, address 0xfec00000, GSI 0-23
(XEN) ACPI: IOAPIC (id[0x07] address[0xfec20000] gsi_base[24])
(XEN) IOAPIC[1]: apic_id 7, version 33, address 0xfec20000, GSI 24-55

However, when Xen tries to setup the IOMMU interrupt remapping table using IVHD
entries, there is only one IOAPIC (IOAPIC 1 with apic_id 7).

(XEN) AMD-Vi: IVHD Device Entry: type 0x48 id 0 flags 0
(XEN) AMD-Vi: IVHD Special: 0000:00:00.1 variety 0x1 handle 0x7
(XEN) IVHD Error: no information for IO-APIC 0x6
(XEN) AMD-Vi: Error initialization

In this case, if we should be able to look at the IVHD to correlate IOAPIC ID (0 or 1)
from the "handle" field and map it back to the BDF to setup the remapping table.

CASE2: BIOS disable the IOAPIC in the I/O bridge (RD890s chipset)
This happens in the case when we were testing the per-device interrupt remapping
table patch. (I think this is the issue you might be seeing in one of the Xen test system.)
In this case, the MADT reports 1 IOAPIC while the IVRS contains two IVHD entries with both
entries have the "hahandle" set to "0".  Unfortunately, in this case, there is no obvious
workaround, and the current solution is to disable IOMMU.

I am working with some of the BIOS engineers and vendors to try to issue root-cause
and provide BIOS update.

Jan, Sander:
Could you please provide the system information:
1. Motherboard vendor
2. BIOS version

Thank you,

Suravee


>
>
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xen.org
> http://lists.xen.org/xen-devel
>