From mboxrd@z Thu Jan 1 00:00:00 1970 From: Boris Ostrovsky Subject: Re: Regression with Xen 4.4 when booting with AMD IOMMU on TA890FXE BIOSTAR Date: Thu, 19 Dec 2013 16:17:31 -0500 Message-ID: <52B3626B.7050607@oracle.com> References: <20131219182519.GA6479@pegasus.dumpdata.com> <52B33D32.1080101@citrix.com> <20131219185211.GA8224@andromeda.dapyr.net> Mime-Version: 1.0 Content-Type: text/plain; charset="us-ascii"; Format="flowed" Content-Transfer-Encoding: 7bit Return-path: Received: from mail6.bemta3.messagelabs.com ([195.245.230.39]) by lists.xen.org with esmtp (Exim 4.72) (envelope-from ) id 1VtkyR-0006QF-71 for xen-devel@lists.xenproject.org; Thu, 19 Dec 2013 21:17:39 +0000 In-Reply-To: <20131219185211.GA8224@andromeda.dapyr.net> List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Sender: xen-devel-bounces@lists.xen.org Errors-To: xen-devel-bounces@lists.xen.org To: Konrad Rzeszutek Wilk Cc: Andrew Cooper , suravee.suthikulpanit@amd.com, jbeulich@suse.com, xen-devel@lists.xenproject.org List-Id: xen-devel@lists.xenproject.org On 12/19/2013 01:52 PM, Konrad Rzeszutek Wilk wrote: > On Thu, Dec 19, 2013 at 06:38:42PM +0000, Andrew Cooper wrote: >> On 19/12/2013 18:25, Konrad Rzeszutek Wilk wrote: >>> Hey, >>> >>> With Xen 4.1, 4.2 and 4.3 it boots fine (albeit it complains about the >>> IOMMU a bit -see attached). With Xen 4.4 it blows up: >>> (the xen-syms is at http://darnok.org/xen/xen-syms.gz) >>> >>> This is with commit a0070f7a5ad8652c74c685a0ee5f10215402279d >>> "x86/memshr: fix preemption in relinquish_shared_pages()" though >>> I hadn't run Xen 4.4 on this box before so the issue might have >>> happend much earlier. >> You are getting some console corruption, even with sync_console. >> >> Also, can you try iommu=debug apic_verbosity=debug ? >> >> Could this perhaps be because you have manually set up the interrupt for >> com1 ? > No. > > If I boot with 3726b101a5cfb2614d3fa2d8be29d6e0b4032a6e > ("extract register definitions from ns16550 into a separated header") > it boots fine. If I got one commit further forward: > 2ca9fbd739b8a72b16dd790d0fff7b75f5488fb8 ("AMD IOMMU: allocate IRTE > entries instead of using a static mapping") it blows up. > > I also had booted Xen with just 'console=vga' and saw the same > blow up. > > With iommu=debug and on 2ca9fbd739b8a72b16dd790d0fff7b75f5488fb8 > I see this: > > PXELINUX 3.82 2009-06-09 Copyright (C) 1994-2009 H. Peter Anvin et al > boot: > Loading xen.gz... ok > Loading vmlinuz... ok > Loading initramfs.cpio.gz... ok > __ __ _ _ _ _ _ _ _ > \ \/ /___ _ __ | || | | || | _ _ _ __ ___| |_ __ _| |__ | | ___ > \ // _ \ '_ \ | || |_| || |_ __| | | | '_ \/ __| __/ _` | '_ \| |/ _ \ > / \ __/ | | | |__ _|__ _|__| |_| | | | \__ \ || (_| | |_) | | __/ > /_/\_\___|_| |_| |_|(_) |_| \__,_|_| |_|___/\__\__,_|_.__/|_|\___| > > (XEN) Xen version 4.4-unstable (konrad@(none)) (gcc (GCC) 4.4.4 20100503 (Red Hat 4.4.4-2)) debug=y Thu Dec 19 13:49:43 EST 2013 > (XEN) Latest ChangeSet: Tue Jul 16 11:52:38 2013 +0200 git:2ca9fbd > (XEN) Console output is synchronous. > (XEN) Bootloader: unknown > (XEN) Command line: sync_console iommu=debug,verbose console=vga,com1 com1=115200,8n1,0xD800,0 conring_size=2097152 loglvl=all guest_loglvl=all dom0_mem_max=max:6GB,2G > (XEN) Video information: > (XEN) VGA is text mode 80x25, font 8x16 > (XEN) VBE/DDC methods: V2; EDID transfer time: 1 seconds > (XEN) Disc information: > (XEN) Found 1 MBR signatures > (XEN) Found 1 EDD information structures > (XEN) Xen-e820 RAM map: > (XEN) 0000000000000000 - 000000000009b400 (usable) > (XEN) 000000000009b400 - 00000000000a0000 (reserved) > (XEN) 00000000000e2000 - 0000000000100000 (reserved) > (XEN) 0000000000100000 - 00000000cffb0000 (usable) > (XEN) 00000000cffb0000 - 00000000cffbe000 (ACPI data) > (XEN) 00000000cffbe000 - 00000000cffe0000 (ACPI NVS) > (XEN) 00000000cffe0000 - 00000000d0000000 (reserved) > (XEN) 00000000fff00000 - 0000000100000000 (reserved) > (XEN) 0000000100000000 - 0000000230000000 (usable) > (XEN) ACPI: RSDP 000FB380, 0024 (r2 ACPIAM) > (XEN) ACPI: XSDT CFFB0100, 0064 (r1 071610 XSDT1756 20100716 MSFT 97) > (XEN) ACPI: FACP CFFB0290, 00F4 (r4 071610 FACP1756 20100716 MSFT 97) > (XEN) ACPI: DSDT CFFB0450, 4D0D (r2 89FAD 89FAD716 1 INTL 20051117) > (XEN) ACPI: FACS CFFBE000, 0040 > (XEN) ACPI: APIC CFFB0390, 007C (r2 071610 APIC1756 20100716 MSFT 97) > (XEN) ACPI: MCFG CFFB0410, 003C (r1 071610 OEMMCFG 20100716 MSFT 97) > (XEN) ACPI: OEMB CFFBE040, 0072 (r1 071610 OEMB1756 20100716 MSFT 97) > (XEN) ACPI: SRAT CFFBA450, 0108 (r3 AMD FAM_F_10 2 AMD 1) > (XEN) ACPI: HPET CFFBA560, 0038 (r1 071610 OEMHPET 20100716 MSFT 97) > (XEN) ACPI: IVRS CFFBA5A0, 00B8 (r1 AMD RD890S 202031 AMD 0) > (XEN) ACPI: SSDT CFFBA660, 0DA4 (r1 A M I POWERNOW 1 AMD 1) > (XEN) System RAM: 8191MB (8387884kB) > (XEN) SRAT: PXM 0 -> APIC 0 -> Node 0 > (XEN) SRAT: PXM 0 -> APIC 1 -> Node 0 > (XEN) SRAT: PXM 0 -> APIC 2 -> Node 0 > (XEN) SRAT:Allocated memnodemap from 229329000 - 22932c000 > (XEN) NUMA: Using 8 for the hash shift. > (XEN) Domain heap initialised > (XEN) found SMP MP-table at 000ff780 > (XEN) DMI present. > (XEN) Using APIC driver default > (XEN) EN) ACPI: wakeup_vec[cffbe00c], vec_size[20] > (XEN) ACPI: Local APIC address 0xfee00000 > (XEN) ACPI: LAPIC (acpi_id[0x01] lapic_id[0x00] enabled) > (XEN) Processor #0 0:10 APIC version 16 > (XEN) ACPI: LAPIC (acpi_id[0x02] lapic_id[0x01] enabled) > (XEN) Processor #1 0:10 APIC version 16 > (XEN) ACPI: LAPIC (acpi_id[0x03] lapic_id[0x02] enabled) > (XEN) Processor #2 0:10 APIC version 16 > (XEN) ACPI: LAPIC (acpi_id[0x04] lapic_id[0x03] enabled) > (XEN) Processor #3 0:10 APIC version 16 > (XEN) ACPI: LAPIC (acpi_id[0x05] lapic_id[0x04] enabled) > (XEN) Processor #4 0:10 APIC version 16 > (XEN) ACPI: LAPIC (acpi_id[0x06] lapic_id[0x05] enabled) > (XEN) Processor #5 0:10 APIC version 16 > (XEN) ACPI: IOAPIC (id[0x06] address[0xfec00000] gsi_base[0]) > (XEN) IOAPIC[0]: apic_id 6, version 33, address 0xfec00000, GSI 0-23 > (XEN) ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) > (XEN) ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 low level) > (XEN) ACPI: IRQ0 used by override. > (XEN) ACPI: IRQ2 used by override. > (XEN) ACPI: IRQ9 used by override. > (XEN) Enabling APIC mode: Flat. Using 1 I/O APICs > (XEN) ACPI: HPET id: 0x8300 base: 0xfed00000 > (XEN) ERST table was not found > (XEN) Using ACPI (MADT) for SMP configuration information > (XEN) SMP: Allowing 6 CPUs (0 hotplug CPUs) > (XEN) IRQ limits: 24 GSI, 1144 MSI/MSI-X > (XEN) Using scheduler: SMP Credit Scheduler (credit) > (XEN) Detected 3000.216 MHz processor. > (XEN) Initing memory sharing. > (XEN) AMD Fam10h machine check reporting enabled > (XEN) PI: Not using MCFG for segment 0000 bus 00-ff > (XEN) AMD-Vi: Found MSI capability block at 0x54 > (XEN) AMD-Vi: ACPI Table: > (XEN) AMD-Vi: Signature IVRS > (XEN) AMD-Vi: Length 0xb8 > (XEN) AMD-Vi: Revision 0x1 > (XEN) AMD-Vi: CheckSum 0x4f > (XEN) AMD-Vi: OEM_Id AMD > (XEN) AMD-Vi: OEM_Table_Id RD890S > (XEN) AMD-Vi: OEM_Revision 0x202031 > (XEN) AMD-Vi: Creator_Id AMD > (XEN) AMD-Vi: Creator_Revision 0 > (XEN) AMD-Vi: IVRS Block: type 0x10 flags 0x3e len 0x88 id 0x2 > (XEN) AMD-Vi: IVHD Device Entry: type 0x3 id 0 flags 0 > (XEN) AMD-Vi: Dev_Id Range: 0 -> 0x2 > (XEN) AMD-Vi: IVHD Device Entry: type 0x2 id 0x10 flags 0 > (XEN) AMD-Vi: IVHD Device Entry: type 0x3 id 0x100 flags 0 > (XEN) AMD-Vi: Dev_Id Range: 0x100 -> 0x101 > (XEN) AMD-Vi: IVHD Device Entry: type 0x2 id 0x88 flags 0 > (XEN) AMD-Vi: IVHD Device Entry: type 0x3 id 0x90 flags 0 > (XEN) AMD-Vi: Dev_Id Range: 0x90 -> 0x92 > (XEN) AMD-Vi: IVHD Device Entry: type 0x3 id 0x98 flags 0 > (XEN) AMD-Vi: Dev_Id Range: 0x98 -> 0x9a > (XEN) AMD-Vi: IVHD Device Entry: type 0x2 id 0xa0 flags 0xd7 > (XEN) AMD-Vi: IVHD Device Entry: type 0x2 id 0xa3 flags 0 > (XEN) AMD-Vi: IVHD Device Entry: type 0x2 id 0xa4 flags 0 > (XEN) AMD-Vi: IVHD Device Entry: type 0 id 0 flags 0 > (XEN) AMD-Vi: IVHD Device Entry: type 0x43 id 0x200 flags 0 > (XEN) AMD-Vi: Dev_Id Range: 0x200 -> 0x3ff alias 0xa4 > (XEN) AMD-Vi: IVHD Device Entry: type 0x2 id 0xa5 flags 0 > (XEN) AMD-Vi: IVHD Device Entry: type 0x2 id 0xa8 flags 0 > (XEN) AMD-Vi: IVHD Device Entry: type 0x2 id 0xa9 flags 0 > (XEN) AMD-Vi: IVHD Device Entry: type 0x2 id 0x400 flags 0 > (XEN) AMD-Vi: IVHD Device Entry: type 0x3 id 0xb0 flags 0 > (XEN) AMD-Vi: Dev_Id Range: 0xb0 -> 0xb2 > (XEN) AMD-Vi: IVHD Device Entry: type 0 id 0 flags 0 > (XEN) AMD-Vi: IVHD Device Entry: type 0x48 id 0 flags 0xd7 > (XEN) AMD-Vi: IVHD Special: 0000:00:14.0 variety 0x2 handle 0 > (XEN) AMD-Vi: IVHD Device Entry: type 0x48 id 0 flags 0 > (XEN) AMD-Vi: IVHD Special: 0000:00:00.1 variety 0x1 handle 0x6 > (XEN) AMD-Vi: IOMMU 0 Enabled. > (XEN) AMD-Vi: per-device vector map logic is broken. Using per-device-global maps instead until a fix is found. Looks like IOAPIC definition is missing from IVRS. The one that's listed (0000:00:00.1 variety 0x1 handle 0x6) is usually unused IIRC. I'd expect an entry for the second one (that lives on the southbridge), such as AMD-Vi: IVHD Special: 0000:00:14.0 variety 0x1 handle 0 And page faults that you are seeing may also be related to this since 0xa0 is 00:14.0. BIOS update might help with this problem (looks like Biostar has a newer BIOS for this board) but I think this should be fixed so maybe you want to keep this BIOS until a fix is developed. You could try booting with iommu=no-intremap to work around this in the meantime. -boris > (XEN) I/O virtualisation enabled > (XEN) - Dom0 mode: Relaxed > (XEN) Interrupt remapping enabled > (XEN) ENABLING IO-APIC IRQs > ((XEN) ..MP-BIOS bug: 8254 timer not connected to IO-APIC > (XEN) ...trying to set up timer (IRQ0) through the 8259A ... failed.set up timer as Virtual Wire IRQ... works. > (XEN) Platform timer is 14.318MHz HPET > (XEN) Allocated console ring of 1048576 KiB. > (XEN) HVM: ASIDs enabled. > (XEN) SVM: Supper > (XEN) HVM: SVM enabled > (XEN) HVM: Hardware Assisted Paging (HAP) detected > (XEN) HVM: HAP page sizes: 4kB, 2MB, 1GB > (XEN) microcode: CPU1 collect_cpu_info: patch_id=0x10000bf > (XEN) AMD-Vi: IO_PAGE_FAULT: domain = 0, device id = 0xa0, fault address = 0xfdf8010140, flags = 0x8 > (XEN) AMD-Vi: IO_PAGE_FAULT: domain = 0, device id = 0xa0, fault address = 0xfdf8010140, flags = 0x8 > (XEN) AMD-Vi: IO_PAGE_FAULT: domain = 0, device id = 0xa0, fault address = 0xfdf8010140, flags = 0x8 > (XEN) AMD-Vi: IO_PAGE_FAULT: domain = 0, device id = 0xa0, fault address = 0xfdf8010140, flags = 0x8 > (XEN) AMD-Vi: IO_PAGE_FAULT: domain = 0, device id = 0xa0, fault address = 0xfdf8010140, flags = 0x8 > (XEN) AMD-Vi: IO_PAGE_FAULT: domain = 0, device id = 0xa0, fault address = 0xfdf8010140, flags = 0x8 > (XEN) AMD-Vi: IO_PAGE_FAULT: domain = 0, device id = 0xa0, fault address = 0xfdf8010140, flags = 0x8 > (XEN) AMD-Vi: IO_PAGE_FAULT: domain = 0, device id = 0xa0, fault address = 0xfdf8010140, flags = 0x8 > (XEN) AMD-Vi: IO_PAGE_FAULT: domain = 0, device id = 0xa0, fault address = 0xfdf8010140, flags = 0x8 > (XEN) AMD-Vi: IO_PAGE_FAULT: domain = 0, device id = 0xa0, fault address = 0xfdf8010140, flags = 0x8 > (XEN) microcode: CPU2 collect_cpu_info: patch_id=0x10000bf > (XEN) microcode: CPU3 collect_cpu_info: patch_id=0x10000bf > (XEN) microcode: CPU4 collect_cpu_info: patch_id=0x10000bf > (XEN) Brought up 6 CPUs > (XEN) microcode: CPU5 collect_cpu_info: pat 'get_rte_index(rte) == offset' failed at iommu_intr.c:159 > (XEN) ----[ Xen-4.4-unstable x86_64 debug=y Tainted: C ]---- > (XEN) CPU: 0 > (XEN) RIP: e008:[] amd_iommu_ioapic_update_ire+0x447/0x600 > (XEN) RFLAGS: 0000000000010002 CONTEXT: hypervisor > (XEN) rax: ffff8302292f2004 rbx: 0000000000000015 rcx: 0000000000000000 > (XEN) rdx: 0000000000000000 rsi: 0000000000000000 rdi: ffff82d0802f00c8 > (XEN) rbp: ffff82d0802c7d88 rsp: ffff82d0802c7cb8 r8: 0000000000000020 > (XEN) r9: 0000000000000000 r10: 000000000000000f r11: ffff82d080226f00 > (XEN) r12: 0000000000000000 r13: 0000000000000014 r14: 0000000000000000 > (XEN) r15: ffff8302292cc380 cr0: 000000008005003b cr4: 00000000000006f0 > (XEN) cr3: 00000000cfc73000 cr2: 0000000000000000 > (XEN) ds: 0000 es: 0000 fs: 0000 gs: 0000 ss: 0000 cs: e008 > (XEN) Xen stack trace from rsp=ffff82d0802c7cb8: > (XEN) 0000000000000000 0100000000000086 0000000000000046 0000001480129548 > (XEN) ffff82d080127200 0000000000000000 0000000000000000 0000000000000000 > (XEN) ffff8302292e0070 0100000000000000 0000000100000002 ffff82d0802c7d48 > (XEN) ffff830200000001 ffff830200000001 0000001500000001 0000000000205000 > (XEN) 0100000000010000 ffff82d08016ab76 0000000000010000 0000000000000000 > (XEN) 0000000000000000 ffff830229276b50 ffff82d080263d24 0000000000000002 > (XEN) ffff82cffffff010 0000000000000000 ffff82d0802c7d98 ffff82d080143787 > (XEN) ffff82d0802c7df8 ffff82d0801654e8 0000000000000282 0000000000000292 > (XEN) ffff830229280000 0100000000000282 ffff82d0802c7e08 0000000000000002 > (XEN) ffff830229280000 0000000000000000 ffff82d080260d20 0000000000000000 > (XEN) ffff82d0802c7e38 ffff82d0801656b0 0000000000000006 0000000000000006 > (XEN) ffff83000006efb0 ffff83022931a7f0 ffff83022931a7f0 ffff82d080265080 > (XEN) ffff82d0802c7e48 ffff82d08029d0b7 ffff82d0802c7f08 ffff82d08029c7b7 > (XEN) 0000000000000000 0000000000000000 ffff82d0802406bd 0000000001281000 > (XEN) 0000000000000000 ffff83000006efb0 ffff82d0ffffffff 0000000000000005 > (XEN) ffff83000006eef0 ffff83000006efb0 ffff82d000000005 ffff82d000000006 > (XEN) 0000000800000000 000000010000006e 0000000000000003 00000000000002f8 > (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 > (XEN) 0000000000000000 0000000000000000 0000000000000000 ffff82d0801000b5 > (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 > (XEN) Xen call trace: > (XEN) [] amd_iommu_ioapic_update_ire+0x447/0x600 > (XEN) [] iommu_update_ire_from_apic+0x34/0x36 > (XEN) [] set_ioapic_affinity_irq+0xa5/0x1d6 > (XEN) [] setup_ioapic_dest+0x97/0xca > (XEN) [] smp_cpus_done+0x51/0x61 > (XEN) [] __start_xen+0x29dc/0x2d1a > (XEN) > (XEN) > (XEN) **************************************** > (XEN) Panic on CPU 0: > (XEN) Assertion 'get_rte_index(rte) == offset' failed at iommu_intr.c:159 > (XEN) **************************************** > (XEN) > (XEN) Reboot in five seconds... > > telnet> Connection closed. > [Connecting to system 32 ] > > _______________________________________________ > Xen-devel mailing list > Xen-devel@lists.xen.org > http://lists.xen.org/xen-devel