AMD IOMMU disabled - No Perdev Intremap

AMD IOMMU disabled - No Perdev Intremap

[Ferdinand Nölscher]

Hi!

>From other people posting to this list, I know that there has been a
bug related to the issue described in Xen Security Advisory 36 that
disables iommu for some AMD users like me.

However, even when passing "iommu=no-amd-iommu-perdev-intremap" it
still disables i/o virtualisatoin. Related Xen dmesg output:

(XEN) IVHD Error: Invalid IO-APIC 0
(XEN) AMD-Vi: Error initialization
(XEN) I/O virtualisation disabled

Is there _any_ way I can fix this to make IOMMU work again? Updating
my BIOS had no effect.

I'm using Xen 4.4 unstable from the official git. Any help is greatly
appreciated.

kind regards,
F. Nölscher

Re: AMD IOMMU disabled - No Perdev Intremap

[Sander Eikelenboom]

Sunday, October 6, 2013, 1:36:33 PM, you wrote:

> Hi!

>>From other people posting to this list, I know that there has been a
> bug related to the issue described in Xen Security Advisory 36 that
> disables iommu for some AMD users like me.

What motherboard do you have ?

> However, even when passing "iommu=no-amd-iommu-perdev-intremap" it
> still disables i/o virtualisatoin. Related Xen dmesg output:

> (XEN) IVHD Error: Invalid IO-APIC 0
> (XEN) AMD-Vi: Error initialization
> (XEN) I/O virtualisation disabled

> Is there _any_ way I can fix this to make IOMMU work again? Updating
> my BIOS had no effect.

It depends if it's the north or southbridge ioapic but try using the new xen boot parameter:
(add it to the xen.gz line in grub)

ivrs_ioapic[0]=00:14.0

or

ivrs_ioapic[0]=00:00.1

--
Sander




> I'm using Xen 4.4 unstable from the official git. Any help is greatly
> appreciated.

> kind regards,
> F. Nölscher

Re: AMD IOMMU disabled - No Perdev Intremap

[Ferdinand Nölscher]

Hi Sander,

Thanks for your reply.

On 10/06/2013 02:10 PM, Sander Eikelenboom wrote:
> 
> Sunday, October 6, 2013, 1:36:33 PM, you wrote:
> 
>> Hi!
> 
>>> From other people posting to this list, I know that there has
>>> been a
>> bug related to the issue described in Xen Security Advisory 36
>> that disables iommu for some AMD users like me.
> 
> What motherboard do you have ?

I have an ASUS M5A99X EVO Rev 1.01

> 
>> However, even when passing "iommu=no-amd-iommu-perdev-intremap"
>> it still disables i/o virtualisatoin. Related Xen dmesg output:
> 
>> (XEN) IVHD Error: Invalid IO-APIC 0 (XEN) AMD-Vi: Error
>> initialization (XEN) I/O virtualisation disabled
> 
>> Is there _any_ way I can fix this to make IOMMU work again?
>> Updating my BIOS had no effect.
> 
> It depends if it's the north or southbridge ioapic but try using
> the new xen boot parameter: (add it to the xen.gz line in grub)
> 
> ivrs_ioapic[0]=00:14.0
> 
> or
> 
> ivrs_ioapic[0]=00:00.1

I tried both, it still fails to enable i/o virtualisation.

> 
> -- Sander
> 
> 
> 
> 
>> I'm using Xen 4.4 unstable from the official git. Any help is
>> greatly appreciated.
> 
>> kind regards, F. Nölscher
> 
> 
> 
> 
> _______________________________________________ Xen-devel mailing
> list Xen-devel@lists.xen.org http://lists.xen.org/xen-devel
> 
> 

Re: AMD IOMMU disabled - No Perdev Intremap

[Andrew Cooper]

On 06/10/2013 13:56, Ferdinand Nölscher wrote:
> Hi Sander,
>
> Thanks for your reply.
>
> On 10/06/2013 02:10 PM, Sander Eikelenboom wrote:
>> Sunday, October 6, 2013, 1:36:33 PM, you wrote:
>>
>>> Hi!
>>>> From other people posting to this list, I know that there has
>>>> been a
>>> bug related to the issue described in Xen Security Advisory 36
>>> that disables iommu for some AMD users like me.
>> What motherboard do you have ?
> I have an ASUS M5A99X EVO Rev 1.01
>
>
>> It depends if it's the north or southbridge ioapic but try using
>> the new xen boot parameter: (add it to the xen.gz line in grub)
>>
>> ivrs_ioapic[0]=00:14.0
>>
>> or
>>
>> ivrs_ioapic[0]=00:00.1
> I tried both, it still fails to enable i/o virtualisation.

Can you post a full xl dmesg, booting with iommu=debug,verbose as well?

~Andrew

Re: AMD IOMMU disabled - No Perdev Intremap

[Ferdinand Nölscher]

On 10/06/2013 02:58 PM, Andrew Cooper wrote:
> On 06/10/2013 13:56, Ferdinand Nölscher wrote:
>> Hi Sander,
>>
>> Thanks for your reply.
>>
>> On 10/06/2013 02:10 PM, Sander Eikelenboom wrote:
>>> Sunday, October 6, 2013, 1:36:33 PM, you wrote:
>>>
>>>> Hi!
>>>>> From other people posting to this list, I know that there has
>>>>> been a
>>>> bug related to the issue described in Xen Security Advisory 36
>>>> that disables iommu for some AMD users like me.
>>> What motherboard do you have ?
>> I have an ASUS M5A99X EVO Rev 1.01
>>
>>
>>> It depends if it's the north or southbridge ioapic but try using
>>> the new xen boot parameter: (add it to the xen.gz line in grub)
>>>
>>> ivrs_ioapic[0]=00:14.0
>>>
>>> or
>>>
>>> ivrs_ioapic[0]=00:00.1
>> I tried both, it still fails to enable i/o virtualisation.
> 
> Can you post a full xl dmesg, booting with iommu=debug,verbose as well?
> 
> ~Andrew
> 
> 

Hi Andrew,
This is what I got:

(XEN) Latest ChangeSet: Fri Oct 4 12:58:20 2013 +0200 git:8e0da8c-dirty
(XEN) Bootloader: GRUB 1.99-27+deb7u1
(XEN) Command line: placeholder dom0_mem=3096M iommu=debug,verbose
(XEN) Video information:
(XEN)  VGA is text mode 80x25, font 8x16
(XEN)  VBE/DDC methods: V2; EDID transfer time: 1 seconds
(XEN) Disc information:
(XEN)  Found 4 MBR signatures
(XEN)  Found 4 EDD information structures
(XEN) Xen-e820 RAM map:
(XEN)  0000000000000000 - 000000000009d800 (usable)
(XEN)  000000000009d800 - 00000000000a0000 (reserved)
(XEN)  00000000000e0000 - 0000000000100000 (reserved)
(XEN)  0000000000100000 - 000000009d81c000 (usable)
(XEN)  000000009d81c000 - 000000009d86f000 (ACPI NVS)
(XEN)  000000009d86f000 - 000000009d87a000 (ACPI data)
(XEN)  000000009d87a000 - 000000009d87b000 (ACPI NVS)
(XEN)  000000009d87b000 - 000000009daca000 (reserved)
(XEN)  000000009daca000 - 000000009dadb000 (ACPI NVS)
(XEN)  000000009dadb000 - 000000009daf0000 (reserved)
(XEN)  000000009daf0000 - 000000009daf9000 (usable)
(XEN)  000000009daf9000 - 000000009dafb000 (ACPI NVS)
(XEN)  000000009dafb000 - 000000009db04000 (reserved)

(XEN)  000000009db04000 - 000000009db0a000 (ACPI NVS)

(XEN)  000000009db0a000 - 000000009db6c000 (reserved)

(XEN)  000000009db6c000 - 000000009dd6f000 (ACPI NVS)

(XEN)  000000009dd6f000 - 000000009df00000 (usable)

(XEN)  00000000f8000000 - 00000000fc000000 (reserved)

(XEN)  00000000fec00000 - 00000000fec01000 (reserved)

(XEN)  00000000fec10000 - 00000000fec11000 (reserved)

(XEN)  00000000fec20000 - 00000000fec21000 (reserved)

(XEN)  00000000fed00000 - 00000000fed01000 (reserved)

(XEN)  00000000fed61000 - 00000000fed71000 (reserved)

(XEN)  00000000fed80000 - 00000000fed90000 (reserved)

(XEN)  00000000fef00000 - 0000000100000000 (reserved)

(XEN)  0000000100001000 - 000000025f000000 (usable)

(XEN) ACPI: RSDP 000F0450, 0024 (r2 ALASKA)

(XEN) ACPI: XSDT 9D86F070, 0054 (r1 ALASKA    A M I  1072009 AMI
10013)
(XEN) ACPI: FACP 9D877D18, 00F4 (r4 ALASKA    A M I  1072009 AMI
10013)
(XEN) ACPI Warning (tbfadt-0464): Optional field "Pm2ControlBlock" has
zero address or length: 0000000000000000/1 [20070126]


(XEN) ACPI: DSDT 9D86F158, 8BBC (r2 ALASKA    A M I        0 INTL
20051117)
(XEN) ACPI: FACS 9DB04F80, 0040

(XEN) ACPI: APIC 9D877E10, 009E (r3 ALASKA    A M I  1072009 AMI
10013)
(XEN) ACPI: MCFG 9D877EB0, 003C (r1 ALASKA    A M I  1072009 MSFT
10013)
(XEN) ACPI: HPET 9D877EF0, 0038 (r1 ALASKA    A M I  1072009 AMI
 4)
(XEN) ACPI: SSDT 9D878058, 1714 (r1 AMD    POWERNOW        1 AMD
 1)
(XEN) ACPI: IVRS 9D877F80, 00D8 (r1  AMD     RD890S   202031 AMD
 0)
(XEN) System RAM: 8137MB (8332616kB)

(XEN) No NUMA configuration found

(XEN) Faking a node at 0000000000000000-000000025f000000

(XEN) Domain heap initialised

(XEN) DMI 2.7 present.

(XEN) Using APIC driver default

(XEN) ACPI: PM-Timer IO Port: 0x808

(XEN) ACPI: SLEEP INFO: pm1x_cnt[804,0], pm1x_evt[800,0]

(XEN) ACPI: 32/64X FACS address mismatch in FADT -
9db04f80/0000000000000000, using 32
(XEN) ACPI:             wakeup_vec[9db04f8c], vec_size[20]
(XEN) ACPI: Local APIC address 0xfee00000
(XEN) ACPI: LAPIC (acpi_id[0x01] lapic_id[0x10] enabled)
(XEN) Processor #16 5:1 APIC version 16
(XEN) ACPI: LAPIC (acpi_id[0x02] lapic_id[0x11] enabled)
(XEN) Processor #17 5:1 APIC version 16
(XEN) ACPI: LAPIC (acpi_id[0x03] lapic_id[0x12] enabled)
(XEN) Processor #18 5:1 APIC version 16
(XEN) ACPI: LAPIC (acpi_id[0x04] lapic_id[0x13] enabled)
(XEN) Processor #19 5:1 APIC version 16
(XEN) ACPI: LAPIC (acpi_id[0x05] lapic_id[0x14] enabled)
(XEN) Processor #20 5:1 APIC version 16
(XEN) ACPI: LAPIC (acpi_id[0x06] lapic_id[0x15] enabled)
(XEN) Processor #21 5:1 APIC version 16
(XEN) ACPI: LAPIC (acpi_id[0x07] lapic_id[0x16] enabled)
(XEN) Processor #22 5:1 APIC version 16
(XEN) ACPI: LAPIC (acpi_id[0x08] lapic_id[0x17] enabled)
(XEN) Processor #23 5:1 APIC version 16
(XEN) ACPI: LAPIC_NMI (acpi_id[0xff] high edge lint[0x1])
(XEN) ACPI: IOAPIC (id[0x09] address[0xfec00000] gsi_base[0])
(XEN) IOAPIC[0]: apic_id 9, version 33, address 0xfec00000, GSI 0-23
(XEN) ACPI: IOAPIC (id[0x0a] address[0xfec20000] gsi_base[24])
(XEN) IOAPIC[1]: apic_id 10, version 33, address 0xfec20000, GSI 24-55
(XEN) ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl)
(XEN) ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 low level)
(XEN) ACPI: IRQ0 used by override.
(XEN) ACPI: IRQ2 used by override.
(XEN) ACPI: IRQ9 used by override.
(XEN) Enabling APIC mode:  Flat.  Using 2 I/O APICs
(XEN) ACPI: HPET id: 0x43538210 base: 0xfed00000
(XEN) ERST table was not found
(XEN) Using ACPI (MADT) for SMP configuration information
(XEN) SMP: Allowing 8 CPUs (0 hotplug CPUs)
(XEN) IRQ limits: 56 GSI, 1496 MSI/MSI-X
(XEN) Using scheduler: SMP Credit Scheduler (credit)
(XEN) Detected 3110.525 MHz processor.
(XEN) Initing memory sharing.
(XEN) xstate_init: using cntxt_size: 0x3c0 and states: 0x4000000000000007
(XEN) AMD Fam15h machine check reporting enabled
(XEN) PCI: MCFG configuration 0: base e0000000 segment 0000 buses 00 - ff
(XEN) PCI: Not using MCFG for segment 0000 bus 00-ff
(XEN) AMD-Vi: Found MSI capability block at 0x54
(XEN) AMD-Vi: ACPI Table:
(XEN) AMD-Vi:  Signature IVRS
(XEN) AMD-Vi:  Length 0xd8
(XEN) AMD-Vi:  Revision 0x1
(XEN) AMD-Vi:  CheckSum 0x22
(XEN) AMD-Vi:  OEM_Id AMD
(XEN) AMD-Vi:  OEM_Table_Id RD890S
(XEN) AMD-Vi:  OEM_Revision 0x202031
(XEN) AMD-Vi:  Creator_Id AMD
(XEN) AMD-Vi:  Creator_Revision 0
(XEN) AMD-Vi: IVRS Block: type 0x10 flags 0x3e len 0xa8 id 0x2
(XEN) AMD-Vi: IVHD Device Entry: type 0x3 id 0 flags 0
(XEN) AMD-Vi:  Dev_Id Range: 0 -> 0x2
(XEN) AMD-Vi: IVHD Device Entry: type 0x2 id 0x10 flags 0
(XEN) AMD-Vi: IVHD Device Entry: type 0x3 id 0x100 flags 0
(XEN) AMD-Vi:  Dev_Id Range: 0x100 -> 0x101
(XEN) AMD-Vi: IVHD Device Entry: type 0x2 id 0x20 flags 0
(XEN) AMD-Vi: IVHD Device Entry: type 0x2 id 0x200 flags 0
(XEN) AMD-Vi: IVHD Device Entry: type 0x2 id 0x28 flags 0
(XEN) AMD-Vi: IVHD Device Entry: type 0x2 id 0x300 flags 0
(XEN) AMD-Vi: IVHD Device Entry: type 0x2 id 0x30 flags 0
(XEN) AMD-Vi: IVHD Device Entry: type 0x2 id 0x400 flags 0
(XEN) AMD-Vi: IVHD Device Entry: type 0x2 id 0x38 flags 0
(XEN) AMD-Vi: IVHD Device Entry: type 0x2 id 0x500 flags 0
(XEN) AMD-Vi: IVHD Device Entry: type 0x2 id 0x50 flags 0
(XEN) AMD-Vi: IVHD Device Entry: type 0x2 id 0x600 flags 0
(XEN) AMD-Vi: IVHD Device Entry: type 0x2 id 0x88 flags 0
(XEN) AMD-Vi: IVHD Device Entry: type 0x3 id 0x90 flags 0
(XEN) AMD-Vi:  Dev_Id Range: 0x90 -> 0x92
(XEN) AMD-Vi: IVHD Device Entry: type 0x3 id 0x98 flags 0
(XEN) AMD-Vi:  Dev_Id Range: 0x98 -> 0x9a
(XEN) AMD-Vi: IVHD Device Entry: type 0x2 id 0xa0 flags 0xd7
(XEN) AMD-Vi: IVHD Device Entry: type 0x2 id 0xa2 flags 0
(XEN) AMD-Vi: IVHD Device Entry: type 0x2 id 0xa3 flags 0
(XEN) AMD-Vi: IVHD Device Entry: type 0x2 id 0xa4 flags 0
(XEN) AMD-Vi: IVHD Device Entry: type 0x43 id 0x700 flags 0
(XEN) AMD-Vi:  Dev_Id Range: 0x700 -> 0x7ff alias 0xa4
(XEN) AMD-Vi: IVHD Device Entry: type 0x2 id 0xa5 flags 0
(XEN) AMD-Vi: IVHD Device Entry: type 0x3 id 0xb0 flags 0
(XEN) AMD-Vi:  Dev_Id Range: 0xb0 -> 0xb2
(XEN) AMD-Vi: IVHD Device Entry: type 0x48 id 0 flags 0xd7
(XEN) AMD-Vi: IVHD Special: 0000:00:14.0 variety 0x1 handle 0
(XEN) IVHD Error: Invalid IO-APIC 0
(XEN) AMD-Vi: Error initialization
(XEN) I/O virtualisation disabled
(XEN) ENABLING IO-APIC IRQs
(XEN)  -> Using new ACK method
(XEN) ..TIMER: vector=0xF0 apic1=0 pin1=2 apic2=-1 pin2=-1
(XEN) Platform timer is 14.318MHz HPET
(XEN) Allocated console ring of 64 KiB.
(XEN) HVM: ASIDs enabled.
(XEN) SVM: Supported advanced features:
(XEN)  - Nested Page Tables (NPT)
(XEN)  - Last Branch Record (LBR) Virtualisation
(XEN)  - Next-RIP Saved on #VMEXIT
(XEN)  - VMCB Clean Bits
(XEN)  - DecodeAssists
(XEN)  - Pause-Intercept Filter
(XEN)  - TSC Rate MSR
(XEN) HVM: SVM enabled
(XEN) HVM: Hardware Assisted Paging (HAP) detected
(XEN) HVM: HAP page sizes: 4kB, 2MB, 1GB
(XEN) microcode: CPU1 collect_cpu_info: patch_id=0x6000629
(XEN) microcode: CPU2 collect_cpu_info: patch_id=0x6000629
(XEN) microcode: CPU3 collect_cpu_info: patch_id=0x6000629
(XEN) microcode: CPU4 collect_cpu_info: patch_id=0x6000629
(XEN) microcode: CPU5 collect_cpu_info: patch_id=0x6000629
(XEN) microcode: CPU6 collect_cpu_info: patch_id=0x6000629
(XEN) microcode: CPU7 collect_cpu_info: patch_id=0x6000629
(XEN) Brought up 8 CPUs
(XEN) ACPI sleep modes: S3
(XEN) MCA: Use hw thresholding to adjust polling frequency
(XEN) mcheck_poll: Machine check polling timer started.
(XEN) *** LOADING DOMAIN 0 ***
(XEN) elf_parse_binary: phdr: paddr=0x1000000 memsz=0x585000
(XEN) elf_parse_binary: phdr: paddr=0x1600000 memsz=0x9d0f0
(XEN) elf_parse_binary: phdr: paddr=0x169e000 memsz=0x14bc0
(XEN) elf_parse_binary: phdr: paddr=0x16b3000 memsz=0x5dd000
(XEN) elf_parse_binary: memory: 0x1000000 -> 0x1c90000
(XEN) elf_xen_parse_note: GUEST_OS = "linux"
(XEN) elf_xen_parse_note: GUEST_VERSION = "2.6"
(XEN) elf_xen_parse_note: XEN_VERSION = "xen-3.0"
(XEN) elf_xen_parse_note: VIRT_BASE = 0xffffffff80000000
(XEN) elf_xen_parse_note: ENTRY = 0xffffffff816b31e0
(XEN) elf_xen_parse_note: HYPERCALL_PAGE = 0xffffffff81001000
(XEN) elf_xen_parse_note: FEATURES =
"!writable_page_tables|pae_pgdir_above_4gb"
(XEN) elf_xen_parse_note: PAE_MODE = "yes"
(XEN) elf_xen_parse_note: LOADER = "generic"
(XEN) elf_xen_parse_note: unknown xen elf note (0xd)
(XEN) elf_xen_parse_note: SUSPEND_CANCEL = 0x1
(XEN) elf_xen_parse_note: HV_START_LOW = 0xffff800000000000
(XEN) elf_xen_parse_note: PADDR_OFFSET = 0x0
(XEN) elf_xen_addr_calc_check: addresses:
(XEN)     virt_base        = 0xffffffff80000000
(XEN)     elf_paddr_offset = 0x0
(XEN)     virt_offset      = 0xffffffff80000000
(XEN)     virt_kstart      = 0xffffffff81000000
(XEN)     virt_kend        = 0xffffffff81c90000
(XEN)     virt_entry       = 0xffffffff816b31e0
(XEN)     p2m_base         = 0xffffffffffffffff
(XEN)  Xen  kernel: 64-bit, lsb, compat32
(XEN)  Dom0 kernel: 64-bit, PAE, lsb, paddr 0x1000000 -> 0x1c90000
(XEN) PHYSICAL MEMORY ARRANGEMENT:
(XEN)  Dom0 alloc.:   0000000250000000->0000000254000000 (767107 pages
to be allocated)
(XEN)  Init. ramdisk: 000000025cc83000->000000025f000000
(XEN) VIRTUAL MEMORY ARRANGEMENT:
(XEN)  Loaded kernel: ffffffff81000000->ffffffff81c90000
(XEN)  Init. ramdisk: ffffffff81c90000->ffffffff8400d000
(XEN)  Phys-Mach map: ffffffff8400d000->ffffffff84619000
(XEN)  Start info:    ffffffff84619000->ffffffff846194b4
(XEN)  Page tables:   ffffffff8461a000->ffffffff84641000
(XEN)  Boot stack:    ffffffff84641000->ffffffff84642000
(XEN)  TOTAL:         ffffffff80000000->ffffffff84800000
(XEN)  ENTRY ADDRESS: ffffffff816b31e0
(XEN) Dom0 has maximum 8 VCPUs
(XEN) elf_load_binary: phdr 0 at 0xffffffff81000000 -> 0xffffffff81585000
(XEN) elf_load_binary: phdr 1 at 0xffffffff81600000 -> 0xffffffff8169d0f0
(XEN) elf_load_binary: phdr 2 at 0xffffffff8169e000 -> 0xffffffff816b2bc0
(XEN) elf_load_binary: phdr 3 at 0xffffffff816b3000 -> 0xffffffff81788000
(XEN) Scrubbing Free RAM:
................................................done.
(XEN) Initial low memory virq threshold set at 0x4000 pages.
(XEN) Std. Loglevel: All
(XEN) Guest Loglevel: All
(XEN) Xen is relinquishing VGA console.
(XEN) *** Serial input -> DOM0 (type 'CTRL-a' three times to switch
input to Xen)
(XEN) Freed 268kB init memory.
(XEN) traps.c:2510:d0 Domain attempted WRMSR 00000000c0010201 from
0x0000000000000000 to 0x000000000000ffff.
(XEN) PCI add device 0000:00:00.0
(XEN) PCI add device 0000:00:00.2
(XEN) PCI add device 0000:00:02.0
(XEN) PCI add device 0000:00:04.0
(XEN) PCI add device 0000:00:05.0
(XEN) PCI add device 0000:00:06.0
(XEN) PCI add device 0000:00:07.0
(XEN) PCI add device 0000:00:0a.0
(XEN) PCI add device 0000:00:11.0
(XEN) PCI add device 0000:00:12.0
(XEN) PCI add device 0000:00:12.2
(XEN) PCI add device 0000:00:13.0
(XEN) PCI add device 0000:00:13.2
(XEN) PCI add device 0000:00:14.0
(XEN) PCI add device 0000:00:14.2
(XEN) PCI add device 0000:00:14.3
(XEN) PCI add device 0000:00:14.4
(XEN) PCI add device 0000:00:14.5
(XEN) PCI add device 0000:00:16.0
(XEN) PCI add device 0000:00:16.2
(XEN) PCI add device 0000:00:18.0
(XEN) PCI add device 0000:00:18.1
(XEN) PCI add device 0000:00:18.2
(XEN) PCI add device 0000:00:18.3
(XEN) PCI add device 0000:00:18.4
(XEN) PCI add device 0000:00:18.5
(XEN) PCI add device 0000:01:00.0
(XEN) PCI add device 0000:01:00.1
(XEN) PCI add device 0000:02:00.0
(XEN) PCI add device 0000:03:00.0
(XEN) PCI add device 0000:04:00.0
(XEN) PCI add device 0000:05:00.0
(XEN) PCI add device 0000:06:00.0
(XEN) PCI add device 0000:07:06.0

Re: AMD IOMMU disabled - No Perdev Intremap

[Sander Eikelenboom]

Sunday, October 6, 2013, 3:09:54 PM, you wrote:

> On 10/06/2013 02:58 PM, Andrew Cooper wrote:
>> On 06/10/2013 13:56, Ferdinand Nölscher wrote:
>>> Hi Sander,
>>>
>>> Thanks for your reply.
>>>
>>> On 10/06/2013 02:10 PM, Sander Eikelenboom wrote:
>>>> Sunday, October 6, 2013, 1:36:33 PM, you wrote:
>>>>
>>>>> Hi!
>>>>>> From other people posting to this list, I know that there has
>>>>>> been a
>>>>> bug related to the issue described in Xen Security Advisory 36
>>>>> that disables iommu for some AMD users like me.
>>>> What motherboard do you have ?
>>> I have an ASUS M5A99X EVO Rev 1.01
>>>
>>>
>>>> It depends if it's the north or southbridge ioapic but try using
>>>> the new xen boot parameter: (add it to the xen.gz line in grub)
>>>>
>>>> ivrs_ioapic[0]=00:14.0
>>>>
>>>> or
>>>>
>>>> ivrs_ioapic[0]=00:00.1
>>> I tried both, it still fails to enable i/o virtualisation.
>> 
>> Can you post a full xl dmesg, booting with iommu=debug,verbose as well?
>> 
>> ~Andrew
>> 
>> 

Ah it seems i got the id's mixed up :-)

ivrs_ioapic[9]=00:00.1 ivrs_ioapic[10]=00:14.0

Should do it i guess .. or should the ioapic handle be in hex .. (then the last should be ivrs_ioapic[a]=00:14.0 )

--
Sander

> Hi Andrew,
> This is what I got:

> (XEN) Latest ChangeSet: Fri Oct 4 12:58:20 2013 +0200 git:8e0da8c-dirty
> (XEN) Bootloader: GRUB 1.99-27+deb7u1
> (XEN) Command line: placeholder dom0_mem=3096M iommu=debug,verbose
> (XEN) Video information:
> (XEN)  VGA is text mode 80x25, font 8x16
> (XEN)  VBE/DDC methods: V2; EDID transfer time: 1 seconds
> (XEN) Disc information:
> (XEN)  Found 4 MBR signatures
> (XEN)  Found 4 EDD information structures
> (XEN) Xen-e820 RAM map:
> (XEN)  0000000000000000 - 000000000009d800 (usable)
> (XEN)  000000000009d800 - 00000000000a0000 (reserved)
> (XEN)  00000000000e0000 - 0000000000100000 (reserved)
> (XEN)  0000000000100000 - 000000009d81c000 (usable)
> (XEN)  000000009d81c000 - 000000009d86f000 (ACPI NVS)
> (XEN)  000000009d86f000 - 000000009d87a000 (ACPI data)
> (XEN)  000000009d87a000 - 000000009d87b000 (ACPI NVS)
> (XEN)  000000009d87b000 - 000000009daca000 (reserved)
> (XEN)  000000009daca000 - 000000009dadb000 (ACPI NVS)
> (XEN)  000000009dadb000 - 000000009daf0000 (reserved)
> (XEN)  000000009daf0000 - 000000009daf9000 (usable)
> (XEN)  000000009daf9000 - 000000009dafb000 (ACPI NVS)
> (XEN)  000000009dafb000 - 000000009db04000 (reserved)

> (XEN)  000000009db04000 - 000000009db0a000 (ACPI NVS)

> (XEN)  000000009db0a000 - 000000009db6c000 (reserved)

> (XEN)  000000009db6c000 - 000000009dd6f000 (ACPI NVS)

> (XEN)  000000009dd6f000 - 000000009df00000 (usable)

> (XEN)  00000000f8000000 - 00000000fc000000 (reserved)

> (XEN)  00000000fec00000 - 00000000fec01000 (reserved)

> (XEN)  00000000fec10000 - 00000000fec11000 (reserved)

> (XEN)  00000000fec20000 - 00000000fec21000 (reserved)

> (XEN)  00000000fed00000 - 00000000fed01000 (reserved)

> (XEN)  00000000fed61000 - 00000000fed71000 (reserved)

> (XEN)  00000000fed80000 - 00000000fed90000 (reserved)

> (XEN)  00000000fef00000 - 0000000100000000 (reserved)

> (XEN)  0000000100001000 - 000000025f000000 (usable)

> (XEN) ACPI: RSDP 000F0450, 0024 (r2 ALASKA)

> (XEN) ACPI: XSDT 9D86F070, 0054 (r1 ALASKA    A M I  1072009 AMI
> 10013)
> (XEN) ACPI: FACP 9D877D18, 00F4 (r4 ALASKA    A M I  1072009 AMI
> 10013)
> (XEN) ACPI Warning (tbfadt-0464): Optional field "Pm2ControlBlock" has
> zero address or length: 0000000000000000/1 [20070126]


> (XEN) ACPI: DSDT 9D86F158, 8BBC (r2 ALASKA    A M I        0 INTL
> 20051117)
> (XEN) ACPI: FACS 9DB04F80, 0040

> (XEN) ACPI: APIC 9D877E10, 009E (r3 ALASKA    A M I  1072009 AMI
> 10013)
> (XEN) ACPI: MCFG 9D877EB0, 003C (r1 ALASKA    A M I  1072009 MSFT
> 10013)
> (XEN) ACPI: HPET 9D877EF0, 0038 (r1 ALASKA    A M I  1072009 AMI
>  4)
> (XEN) ACPI: SSDT 9D878058, 1714 (r1 AMD    POWERNOW        1 AMD
>  1)
> (XEN) ACPI: IVRS 9D877F80, 00D8 (r1  AMD     RD890S   202031 AMD
>  0)
> (XEN) System RAM: 8137MB (8332616kB)

> (XEN) No NUMA configuration found

> (XEN) Faking a node at 0000000000000000-000000025f000000

> (XEN) Domain heap initialised

> (XEN) DMI 2.7 present.

> (XEN) Using APIC driver default

> (XEN) ACPI: PM-Timer IO Port: 0x808

> (XEN) ACPI: SLEEP INFO: pm1x_cnt[804,0], pm1x_evt[800,0]

> (XEN) ACPI: 32/64X FACS address mismatch in FADT -
> 9db04f80/0000000000000000, using 32
> (XEN) ACPI:             wakeup_vec[9db04f8c], vec_size[20]
> (XEN) ACPI: Local APIC address 0xfee00000
> (XEN) ACPI: LAPIC (acpi_id[0x01] lapic_id[0x10] enabled)
> (XEN) Processor #16 5:1 APIC version 16
> (XEN) ACPI: LAPIC (acpi_id[0x02] lapic_id[0x11] enabled)
> (XEN) Processor #17 5:1 APIC version 16
> (XEN) ACPI: LAPIC (acpi_id[0x03] lapic_id[0x12] enabled)
> (XEN) Processor #18 5:1 APIC version 16
> (XEN) ACPI: LAPIC (acpi_id[0x04] lapic_id[0x13] enabled)
> (XEN) Processor #19 5:1 APIC version 16
> (XEN) ACPI: LAPIC (acpi_id[0x05] lapic_id[0x14] enabled)
> (XEN) Processor #20 5:1 APIC version 16
> (XEN) ACPI: LAPIC (acpi_id[0x06] lapic_id[0x15] enabled)
> (XEN) Processor #21 5:1 APIC version 16
> (XEN) ACPI: LAPIC (acpi_id[0x07] lapic_id[0x16] enabled)
> (XEN) Processor #22 5:1 APIC version 16
> (XEN) ACPI: LAPIC (acpi_id[0x08] lapic_id[0x17] enabled)
> (XEN) Processor #23 5:1 APIC version 16
> (XEN) ACPI: LAPIC_NMI (acpi_id[0xff] high edge lint[0x1])
> (XEN) ACPI: IOAPIC (id[0x09] address[0xfec00000] gsi_base[0])
> (XEN) IOAPIC[0]: apic_id 9, version 33, address 0xfec00000, GSI 0-23
> (XEN) ACPI: IOAPIC (id[0x0a] address[0xfec20000] gsi_base[24])
> (XEN) IOAPIC[1]: apic_id 10, version 33, address 0xfec20000, GSI 24-55
> (XEN) ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl)
> (XEN) ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 low level)
> (XEN) ACPI: IRQ0 used by override.
> (XEN) ACPI: IRQ2 used by override.
> (XEN) ACPI: IRQ9 used by override.
> (XEN) Enabling APIC mode:  Flat.  Using 2 I/O APICs
> (XEN) ACPI: HPET id: 0x43538210 base: 0xfed00000
> (XEN) ERST table was not found
> (XEN) Using ACPI (MADT) for SMP configuration information
> (XEN) SMP: Allowing 8 CPUs (0 hotplug CPUs)
> (XEN) IRQ limits: 56 GSI, 1496 MSI/MSI-X
> (XEN) Using scheduler: SMP Credit Scheduler (credit)
> (XEN) Detected 3110.525 MHz processor.
> (XEN) Initing memory sharing.
> (XEN) xstate_init: using cntxt_size: 0x3c0 and states: 0x4000000000000007
> (XEN) AMD Fam15h machine check reporting enabled
> (XEN) PCI: MCFG configuration 0: base e0000000 segment 0000 buses 00 - ff
> (XEN) PCI: Not using MCFG for segment 0000 bus 00-ff
> (XEN) AMD-Vi: Found MSI capability block at 0x54
> (XEN) AMD-Vi: ACPI Table:
> (XEN) AMD-Vi:  Signature IVRS
> (XEN) AMD-Vi:  Length 0xd8
> (XEN) AMD-Vi:  Revision 0x1
> (XEN) AMD-Vi:  CheckSum 0x22
> (XEN) AMD-Vi:  OEM_Id AMD
> (XEN) AMD-Vi:  OEM_Table_Id RD890S
> (XEN) AMD-Vi:  OEM_Revision 0x202031
> (XEN) AMD-Vi:  Creator_Id AMD
> (XEN) AMD-Vi:  Creator_Revision 0
> (XEN) AMD-Vi: IVRS Block: type 0x10 flags 0x3e len 0xa8 id 0x2
> (XEN) AMD-Vi: IVHD Device Entry: type 0x3 id 0 flags 0
> (XEN) AMD-Vi:  Dev_Id Range: 0 -> 0x2
> (XEN) AMD-Vi: IVHD Device Entry: type 0x2 id 0x10 flags 0
> (XEN) AMD-Vi: IVHD Device Entry: type 0x3 id 0x100 flags 0
> (XEN) AMD-Vi:  Dev_Id Range: 0x100 -> 0x101
> (XEN) AMD-Vi: IVHD Device Entry: type 0x2 id 0x20 flags 0
> (XEN) AMD-Vi: IVHD Device Entry: type 0x2 id 0x200 flags 0
> (XEN) AMD-Vi: IVHD Device Entry: type 0x2 id 0x28 flags 0
> (XEN) AMD-Vi: IVHD Device Entry: type 0x2 id 0x300 flags 0
> (XEN) AMD-Vi: IVHD Device Entry: type 0x2 id 0x30 flags 0
> (XEN) AMD-Vi: IVHD Device Entry: type 0x2 id 0x400 flags 0
> (XEN) AMD-Vi: IVHD Device Entry: type 0x2 id 0x38 flags 0
> (XEN) AMD-Vi: IVHD Device Entry: type 0x2 id 0x500 flags 0
> (XEN) AMD-Vi: IVHD Device Entry: type 0x2 id 0x50 flags 0
> (XEN) AMD-Vi: IVHD Device Entry: type 0x2 id 0x600 flags 0
> (XEN) AMD-Vi: IVHD Device Entry: type 0x2 id 0x88 flags 0
> (XEN) AMD-Vi: IVHD Device Entry: type 0x3 id 0x90 flags 0
> (XEN) AMD-Vi:  Dev_Id Range: 0x90 -> 0x92
> (XEN) AMD-Vi: IVHD Device Entry: type 0x3 id 0x98 flags 0
> (XEN) AMD-Vi:  Dev_Id Range: 0x98 -> 0x9a
> (XEN) AMD-Vi: IVHD Device Entry: type 0x2 id 0xa0 flags 0xd7
> (XEN) AMD-Vi: IVHD Device Entry: type 0x2 id 0xa2 flags 0
> (XEN) AMD-Vi: IVHD Device Entry: type 0x2 id 0xa3 flags 0
> (XEN) AMD-Vi: IVHD Device Entry: type 0x2 id 0xa4 flags 0
> (XEN) AMD-Vi: IVHD Device Entry: type 0x43 id 0x700 flags 0
> (XEN) AMD-Vi:  Dev_Id Range: 0x700 -> 0x7ff alias 0xa4
> (XEN) AMD-Vi: IVHD Device Entry: type 0x2 id 0xa5 flags 0
> (XEN) AMD-Vi: IVHD Device Entry: type 0x3 id 0xb0 flags 0
> (XEN) AMD-Vi:  Dev_Id Range: 0xb0 -> 0xb2
> (XEN) AMD-Vi: IVHD Device Entry: type 0x48 id 0 flags 0xd7
> (XEN) AMD-Vi: IVHD Special: 0000:00:14.0 variety 0x1 handle 0
> (XEN) IVHD Error: Invalid IO-APIC 0
> (XEN) AMD-Vi: Error initialization
> (XEN) I/O virtualisation disabled
> (XEN) ENABLING IO-APIC IRQs
(XEN)  ->> Using new ACK method
> (XEN) ..TIMER: vector=0xF0 apic1=0 pin1=2 apic2=-1 pin2=-1
> (XEN) Platform timer is 14.318MHz HPET
> (XEN) Allocated console ring of 64 KiB.
> (XEN) HVM: ASIDs enabled.
> (XEN) SVM: Supported advanced features:
> (XEN)  - Nested Page Tables (NPT)
> (XEN)  - Last Branch Record (LBR) Virtualisation
> (XEN)  - Next-RIP Saved on #VMEXIT
> (XEN)  - VMCB Clean Bits
> (XEN)  - DecodeAssists
> (XEN)  - Pause-Intercept Filter
> (XEN)  - TSC Rate MSR
> (XEN) HVM: SVM enabled
> (XEN) HVM: Hardware Assisted Paging (HAP) detected
> (XEN) HVM: HAP page sizes: 4kB, 2MB, 1GB
> (XEN) microcode: CPU1 collect_cpu_info: patch_id=0x6000629
> (XEN) microcode: CPU2 collect_cpu_info: patch_id=0x6000629
> (XEN) microcode: CPU3 collect_cpu_info: patch_id=0x6000629
> (XEN) microcode: CPU4 collect_cpu_info: patch_id=0x6000629
> (XEN) microcode: CPU5 collect_cpu_info: patch_id=0x6000629
> (XEN) microcode: CPU6 collect_cpu_info: patch_id=0x6000629
> (XEN) microcode: CPU7 collect_cpu_info: patch_id=0x6000629
> (XEN) Brought up 8 CPUs
> (XEN) ACPI sleep modes: S3
> (XEN) MCA: Use hw thresholding to adjust polling frequency
> (XEN) mcheck_poll: Machine check polling timer started.
> (XEN) *** LOADING DOMAIN 0 ***
> (XEN) elf_parse_binary: phdr: paddr=0x1000000 memsz=0x585000
> (XEN) elf_parse_binary: phdr: paddr=0x1600000 memsz=0x9d0f0
> (XEN) elf_parse_binary: phdr: paddr=0x169e000 memsz=0x14bc0
> (XEN) elf_parse_binary: phdr: paddr=0x16b3000 memsz=0x5dd000
> (XEN) elf_parse_binary: memory: 0x1000000 -> 0x1c90000
> (XEN) elf_xen_parse_note: GUEST_OS = "linux"
> (XEN) elf_xen_parse_note: GUEST_VERSION = "2.6"
> (XEN) elf_xen_parse_note: XEN_VERSION = "xen-3.0"
> (XEN) elf_xen_parse_note: VIRT_BASE = 0xffffffff80000000
> (XEN) elf_xen_parse_note: ENTRY = 0xffffffff816b31e0
> (XEN) elf_xen_parse_note: HYPERCALL_PAGE = 0xffffffff81001000
> (XEN) elf_xen_parse_note: FEATURES =
> "!writable_page_tables|pae_pgdir_above_4gb"
> (XEN) elf_xen_parse_note: PAE_MODE = "yes"
> (XEN) elf_xen_parse_note: LOADER = "generic"
> (XEN) elf_xen_parse_note: unknown xen elf note (0xd)
> (XEN) elf_xen_parse_note: SUSPEND_CANCEL = 0x1
> (XEN) elf_xen_parse_note: HV_START_LOW = 0xffff800000000000
> (XEN) elf_xen_parse_note: PADDR_OFFSET = 0x0
> (XEN) elf_xen_addr_calc_check: addresses:
> (XEN)     virt_base        = 0xffffffff80000000
> (XEN)     elf_paddr_offset = 0x0
> (XEN)     virt_offset      = 0xffffffff80000000
> (XEN)     virt_kstart      = 0xffffffff81000000
> (XEN)     virt_kend        = 0xffffffff81c90000
> (XEN)     virt_entry       = 0xffffffff816b31e0
> (XEN)     p2m_base         = 0xffffffffffffffff
> (XEN)  Xen  kernel: 64-bit, lsb, compat32
> (XEN)  Dom0 kernel: 64-bit, PAE, lsb, paddr 0x1000000 -> 0x1c90000
> (XEN) PHYSICAL MEMORY ARRANGEMENT:
> (XEN)  Dom0 alloc.:   0000000250000000->0000000254000000 (767107 pages
> to be allocated)
> (XEN)  Init. ramdisk: 000000025cc83000->000000025f000000
> (XEN) VIRTUAL MEMORY ARRANGEMENT:
> (XEN)  Loaded kernel: ffffffff81000000->ffffffff81c90000
> (XEN)  Init. ramdisk: ffffffff81c90000->ffffffff8400d000
> (XEN)  Phys-Mach map: ffffffff8400d000->ffffffff84619000
> (XEN)  Start info:    ffffffff84619000->ffffffff846194b4
> (XEN)  Page tables:   ffffffff8461a000->ffffffff84641000
> (XEN)  Boot stack:    ffffffff84641000->ffffffff84642000
> (XEN)  TOTAL:         ffffffff80000000->ffffffff84800000
> (XEN)  ENTRY ADDRESS: ffffffff816b31e0
> (XEN) Dom0 has maximum 8 VCPUs
> (XEN) elf_load_binary: phdr 0 at 0xffffffff81000000 -> 0xffffffff81585000
> (XEN) elf_load_binary: phdr 1 at 0xffffffff81600000 -> 0xffffffff8169d0f0
> (XEN) elf_load_binary: phdr 2 at 0xffffffff8169e000 -> 0xffffffff816b2bc0
> (XEN) elf_load_binary: phdr 3 at 0xffffffff816b3000 -> 0xffffffff81788000
> (XEN) Scrubbing Free RAM:
> ................................................done.
> (XEN) Initial low memory virq threshold set at 0x4000 pages.
> (XEN) Std. Loglevel: All
> (XEN) Guest Loglevel: All
> (XEN) Xen is relinquishing VGA console.
> (XEN) *** Serial input -> DOM0 (type 'CTRL-a' three times to switch
> input to Xen)
> (XEN) Freed 268kB init memory.
> (XEN) traps.c:2510:d0 Domain attempted WRMSR 00000000c0010201 from
> 0x0000000000000000 to 0x000000000000ffff.
> (XEN) PCI add device 0000:00:00.0
> (XEN) PCI add device 0000:00:00.2
> (XEN) PCI add device 0000:00:02.0
> (XEN) PCI add device 0000:00:04.0
> (XEN) PCI add device 0000:00:05.0
> (XEN) PCI add device 0000:00:06.0
> (XEN) PCI add device 0000:00:07.0
> (XEN) PCI add device 0000:00:0a.0
> (XEN) PCI add device 0000:00:11.0
> (XEN) PCI add device 0000:00:12.0
> (XEN) PCI add device 0000:00:12.2
> (XEN) PCI add device 0000:00:13.0
> (XEN) PCI add device 0000:00:13.2
> (XEN) PCI add device 0000:00:14.0
> (XEN) PCI add device 0000:00:14.2
> (XEN) PCI add device 0000:00:14.3
> (XEN) PCI add device 0000:00:14.4
> (XEN) PCI add device 0000:00:14.5
> (XEN) PCI add device 0000:00:16.0
> (XEN) PCI add device 0000:00:16.2
> (XEN) PCI add device 0000:00:18.0
> (XEN) PCI add device 0000:00:18.1
> (XEN) PCI add device 0000:00:18.2
> (XEN) PCI add device 0000:00:18.3
> (XEN) PCI add device 0000:00:18.4
> (XEN) PCI add device 0000:00:18.5
> (XEN) PCI add device 0000:01:00.0
> (XEN) PCI add device 0000:01:00.1
> (XEN) PCI add device 0000:02:00.0
> (XEN) PCI add device 0000:03:00.0
> (XEN) PCI add device 0000:04:00.0
> (XEN) PCI add device 0000:05:00.0
> (XEN) PCI add device 0000:06:00.0
> (XEN) PCI add device 0000:07:06.0

Re: AMD IOMMU disabled - No Perdev Intremap

[Andrew Cooper]

On 06/10/2013 15:38, Sander Eikelenboom wrote:
> Sunday, October 6, 2013, 3:09:54 PM, you wrote:
>
>> On 10/06/2013 02:58 PM, Andrew Cooper wrote:
>>> On 06/10/2013 13:56, Ferdinand Nölscher wrote:
>>>> Hi Sander,
>>>>
>>>> Thanks for your reply.
>>>>
>>>> On 10/06/2013 02:10 PM, Sander Eikelenboom wrote:
>>>>> Sunday, October 6, 2013, 1:36:33 PM, you wrote:
>>>>>
>>>>>> Hi!
>>>>>>> From other people posting to this list, I know that there has
>>>>>>> been a
>>>>>> bug related to the issue described in Xen Security Advisory 36
>>>>>> that disables iommu for some AMD users like me.
>>>>> What motherboard do you have ?
>>>> I have an ASUS M5A99X EVO Rev 1.01
>>>>
>>>>
>>>>> It depends if it's the north or southbridge ioapic but try using
>>>>> the new xen boot parameter: (add it to the xen.gz line in grub)
>>>>>
>>>>> ivrs_ioapic[0]=00:14.0
>>>>>
>>>>> or
>>>>>
>>>>> ivrs_ioapic[0]=00:00.1
>>>> I tried both, it still fails to enable i/o virtualisation.
>>> Can you post a full xl dmesg, booting with iommu=debug,verbose as well?
>>>
>>> ~Andrew
>>>
>>>
> Ah it seems i got the id's mixed up :-)
>
> ivrs_ioapic[9]=00:00.1 ivrs_ioapic[10]=00:14.0
>
> Should do it i guess .. or should the ioapic handle be in hex .. (then the last should be ivrs_ioapic[a]=00:14.0 )
>
> --
> Sander
>
>

parse_ivrs_ioapic uses simple_strtoul(..., 0), so decimal or 0xa is
fine, but 'a' on its own wont be.

Seeing as I clearly don't understand how this fix is working, do you
mind explaining (as you seem to have worked out the problem) ?

~Andrew

Re: AMD IOMMU disabled - No Perdev Intremap

[Sander Eikelenboom]

Sunday, October 6, 2013, 4:45:18 PM, you wrote:

> On 06/10/2013 15:38, Sander Eikelenboom wrote:
>> Sunday, October 6, 2013, 3:09:54 PM, you wrote:
>>
>>> On 10/06/2013 02:58 PM, Andrew Cooper wrote:
>>>> On 06/10/2013 13:56, Ferdinand Nölscher wrote:
>>>>> Hi Sander,
>>>>>
>>>>> Thanks for your reply.
>>>>>
>>>>> On 10/06/2013 02:10 PM, Sander Eikelenboom wrote:
>>>>>> Sunday, October 6, 2013, 1:36:33 PM, you wrote:
>>>>>>
>>>>>>> Hi!
>>>>>>>> From other people posting to this list, I know that there has
>>>>>>>> been a
>>>>>>> bug related to the issue described in Xen Security Advisory 36
>>>>>>> that disables iommu for some AMD users like me.
>>>>>> What motherboard do you have ?
>>>>> I have an ASUS M5A99X EVO Rev 1.01
>>>>>
>>>>>
>>>>>> It depends if it's the north or southbridge ioapic but try using
>>>>>> the new xen boot parameter: (add it to the xen.gz line in grub)
>>>>>>
>>>>>> ivrs_ioapic[0]=00:14.0
>>>>>>
>>>>>> or
>>>>>>
>>>>>> ivrs_ioapic[0]=00:00.1
>>>>> I tried both, it still fails to enable i/o virtualisation.
>>>> Can you post a full xl dmesg, booting with iommu=debug,verbose as well?
>>>>
>>>> ~Andrew
>>>>
>>>>
>> Ah it seems i got the id's mixed up :-)
>>
>> ivrs_ioapic[9]=00:00.1 ivrs_ioapic[10]=00:14.0
>>
>> Should do it i guess .. or should the ioapic handle be in hex .. (then the last should be ivrs_ioapic[a]=00:14.0 )
>>
>> --
>> Sander
>>
>>

> parse_ivrs_ioapic uses simple_strtoul(..., 0), so decimal or 0xa is
> fine, but 'a' on its own wont be.

> Seeing as I clearly don't understand how this fix is working, do you
> mind explaining (as you seem to have worked out the problem) ?

Bios provided tables are a mess .. regarding to iommu's it seems to be a double mess.
And motherboard manufacturers seem to be quite incapable to provide sane tables for years now, and it doesn't have any kind of priority.
(ok that little rant is out :-p )

But on (current / consumer) AMD systems, the mapping always seems to be the same ioapic northbridge to 00:00.1 and southbridge to 00:14.0.
So if you know the ioapic handle's from the xl dmesg output, you can easily determine the needed override.

It's perhaps a bit unfortunate that the error without iommu=debug,verbose doesn't show the handle but only the id:

(XEN) IVHD Error: Invalid IO-APIC 0
(XEN) AMD-Vi: Error initialization
(XEN) I/O virtualisation disabled

And it should probably be something for the next document day to have some documentation for it on the iommu part of the xen-wiki.

--
Sander

> ~Andrew

Re: AMD IOMMU disabled - No Perdev Intremap

[Ferdinand Nölscher]

On 10/06/2013 04:38 PM, Sander Eikelenboom wrote:
> 
> Sunday, October 6, 2013, 3:09:54 PM, you wrote:
> 
>> On 10/06/2013 02:58 PM, Andrew Cooper wrote:
>>> On 06/10/2013 13:56, Ferdinand Nölscher wrote:
>>>> Hi Sander,
>>>> 
>>>> Thanks for your reply.
>>>> 
>>>> On 10/06/2013 02:10 PM, Sander Eikelenboom wrote:
>>>>> Sunday, October 6, 2013, 1:36:33 PM, you wrote:
>>>>> 
>>>>>> Hi!
>>>>>>> From other people posting to this list, I know that
>>>>>>> there has been a
>>>>>> bug related to the issue described in Xen Security
>>>>>> Advisory 36 that disables iommu for some AMD users like
>>>>>> me.
>>>>> What motherboard do you have ?
>>>> I have an ASUS M5A99X EVO Rev 1.01
>>>> 
>>>> 
>>>>> It depends if it's the north or southbridge ioapic but try
>>>>> using the new xen boot parameter: (add it to the xen.gz
>>>>> line in grub)
>>>>> 
>>>>> ivrs_ioapic[0]=00:14.0
>>>>> 
>>>>> or
>>>>> 
>>>>> ivrs_ioapic[0]=00:00.1
>>>> I tried both, it still fails to enable i/o virtualisation.
>>> 
>>> Can you post a full xl dmesg, booting with iommu=debug,verbose
>>> as well?
>>> 
>>> ~Andrew
>>> 
>>> 
> 
> Ah it seems i got the id's mixed up :-)
> 
> ivrs_ioapic[9]=00:00.1 ivrs_ioapic[10]=00:14.0
> 
> Should do it i guess .. or should the ioapic handle be in hex ..
> (then the last should be ivrs_ioapic[a]=00:14.0 )
> 
> -- Sander
> 

Booting with this parameters I can see "I/O Virtualisation Enabled"
but shortly after that, I get a CPU Panic:

Xen Call Trace:
amd_iommu_ioapic_update_ire ...
iommu_update_ire_from_apic ...
set_ioapic_affinity_irq ...
smp_cpus_done ...
__start_xen ...

Panic on CPU 0:
Assertion "get_rte_index(rte) == offset" failed at iommu_intr.c:188

Re: AMD IOMMU disabled - No Perdev Intremap

[Sander Eikelenboom]

Sunday, October 6, 2013, 7:28:03 PM, you wrote:

> On 10/06/2013 04:38 PM, Sander Eikelenboom wrote:
>> 
>> Sunday, October 6, 2013, 3:09:54 PM, you wrote:
>> 
>>> On 10/06/2013 02:58 PM, Andrew Cooper wrote:
>>>> On 06/10/2013 13:56, Ferdinand Nölscher wrote:
>>>>> Hi Sander,
>>>>> 
>>>>> Thanks for your reply.
>>>>> 
>>>>> On 10/06/2013 02:10 PM, Sander Eikelenboom wrote:
>>>>>> Sunday, October 6, 2013, 1:36:33 PM, you wrote:
>>>>>> 
>>>>>>> Hi!
>>>>>>>> From other people posting to this list, I know that
>>>>>>>> there has been a
>>>>>>> bug related to the issue described in Xen Security
>>>>>>> Advisory 36 that disables iommu for some AMD users like
>>>>>>> me.
>>>>>> What motherboard do you have ?
>>>>> I have an ASUS M5A99X EVO Rev 1.01
>>>>> 
>>>>> 
>>>>>> It depends if it's the north or southbridge ioapic but try
>>>>>> using the new xen boot parameter: (add it to the xen.gz
>>>>>> line in grub)
>>>>>> 
>>>>>> ivrs_ioapic[0]=00:14.0
>>>>>> 
>>>>>> or
>>>>>> 
>>>>>> ivrs_ioapic[0]=00:00.1
>>>>> I tried both, it still fails to enable i/o virtualisation.
>>>> 
>>>> Can you post a full xl dmesg, booting with iommu=debug,verbose
>>>> as well?
>>>> 
>>>> ~Andrew
>>>> 
>>>> 
>> 
>> Ah it seems i got the id's mixed up :-)
>> 
>> ivrs_ioapic[9]=00:00.1 ivrs_ioapic[10]=00:14.0
>> 
>> Should do it i guess .. or should the ioapic handle be in hex ..
>> (then the last should be ivrs_ioapic[a]=00:14.0 )
>> 
>> -- Sander
>> 

> Booting with this parameters I can see "I/O Virtualisation Enabled"
> but shortly after that, I get a CPU Panic:

> Xen Call Trace:
> amd_iommu_ioapic_update_ire ...
> iommu_update_ire_from_apic ...
> set_ioapic_affinity_irq ...
> smp_cpus_done ...
> __start_xen ...

> Panic on CPU 0:
> Assertion "get_rte_index(rte) == offset" failed at iommu_intr.c:188


Hmm, could you check if you can get a recent linux kernel to boot on baremetal (so without xen) with the iommu enabled ?
(that would most certainly require one or both of the ivrs_ioapic overrides as well)

I assumed but don't know for sure, did you previously had successfully used the iommu with the iommu=no-amd-iommu-perdev-intremap boot option ?
And i understand you upgraded your bios, if the previous answer was yes .. did you try to revert to your previous bios version ?

Also added Jan and Suravee to the CC.

--
Sander

BTW: please keep everyone that was in the CC listed in replies.

Re: AMD IOMMU disabled - No Perdev Intremap

[Ferdinand Nölscher]

On 10/06/2013 07:45 PM, Sander Eikelenboom wrote:
> 
> Sunday, October 6, 2013, 7:28:03 PM, you wrote:
> 
>> On 10/06/2013 04:38 PM, Sander Eikelenboom wrote:
>>> 
>>> Sunday, October 6, 2013, 3:09:54 PM, you wrote:
>>> 
>>>> On 10/06/2013 02:58 PM, Andrew Cooper wrote:
>>>>> On 06/10/2013 13:56, Ferdinand Nölscher wrote:
>>>>>> Hi Sander,
>>>>>> 
>>>>>> Thanks for your reply.
>>>>>> 
>>>>>> On 10/06/2013 02:10 PM, Sander Eikelenboom wrote:
>>>>>>> Sunday, October 6, 2013, 1:36:33 PM, you wrote:
>>>>>>> 
>>>>>>>> Hi!
>>>>>>>>> From other people posting to this list, I know
>>>>>>>>> that there has been a
>>>>>>>> bug related to the issue described in Xen Security 
>>>>>>>> Advisory 36 that disables iommu for some AMD users
>>>>>>>> like me.
>>>>>>> What motherboard do you have ?
>>>>>> I have an ASUS M5A99X EVO Rev 1.01
>>>>>> 
>>>>>> 
>>>>>>> It depends if it's the north or southbridge ioapic but
>>>>>>> try using the new xen boot parameter: (add it to the
>>>>>>> xen.gz line in grub)
>>>>>>> 
>>>>>>> ivrs_ioapic[0]=00:14.0
>>>>>>> 
>>>>>>> or
>>>>>>> 
>>>>>>> ivrs_ioapic[0]=00:00.1
>>>>>> I tried both, it still fails to enable i/o
>>>>>> virtualisation.
>>>>> 
>>>>> Can you post a full xl dmesg, booting with
>>>>> iommu=debug,verbose as well?
>>>>> 
>>>>> ~Andrew
>>>>> 
>>>>> 
>>> 
>>> Ah it seems i got the id's mixed up :-)
>>> 
>>> ivrs_ioapic[9]=00:00.1 ivrs_ioapic[10]=00:14.0
>>> 
>>> Should do it i guess .. or should the ioapic handle be in hex
>>> .. (then the last should be ivrs_ioapic[a]=00:14.0 )
>>> 
>>> -- Sander
>>> 
> 
>> Booting with this parameters I can see "I/O Virtualisation
>> Enabled" but shortly after that, I get a CPU Panic:
> 
>> Xen Call Trace: amd_iommu_ioapic_update_ire ... 
>> iommu_update_ire_from_apic ... set_ioapic_affinity_irq ... 
>> smp_cpus_done ... __start_xen ...
> 
>> Panic on CPU 0: Assertion "get_rte_index(rte) == offset" failed
>> at iommu_intr.c:188
> 
> 
> Hmm, could you check if you can get a recent linux kernel to boot
> on baremetal (so without xen) with the iommu enabled ? (that would
> most certainly require one or both of the ivrs_ioapic overrides as
> well)
> 
> I assumed but don't know for sure, did you previously had
> successfully used the iommu with the
> iommu=no-amd-iommu-perdev-intremap boot option ? And i understand
> you upgraded your bios, if the previous answer was yes .. did you
> try to revert to your previous bios version ?
> 
> Also added Jan and Suravee to the CC.
> 
> -- Sander
> 
> BTW: please keep everyone that was in the CC listed in replies.
> 
> 
> _______________________________________________ Xen-devel mailing
> list Xen-devel@lists.xen.org http://lists.xen.org/xen-devel
> 
> 

What "recent" version? Currently, I'm using 3.10-3, is that ok?

I have used iommu successfully before they added the new checking
stuff, so I can't tell for sure.
I just tried the new xen version because it kept panicking after some
changes half a year ago. Now it's working again but without iommu.
That's why I updated my bios, sadly, without any success.

I'll try booting with Andrews patch now.

kind regards

Re: AMD IOMMU disabled - No Perdev Intremap

[Sander Eikelenboom]

Sunday, October 6, 2013, 7:57:42 PM, you wrote:

> On 10/06/2013 07:45 PM, Sander Eikelenboom wrote:
>> 
>> Sunday, October 6, 2013, 7:28:03 PM, you wrote:
>> 
>>> On 10/06/2013 04:38 PM, Sander Eikelenboom wrote:
>>>> 
>>>> Sunday, October 6, 2013, 3:09:54 PM, you wrote:
>>>> 
>>>>> On 10/06/2013 02:58 PM, Andrew Cooper wrote:
>>>>>> On 06/10/2013 13:56, Ferdinand Nölscher wrote:
>>>>>>> Hi Sander,
>>>>>>> 
>>>>>>> Thanks for your reply.
>>>>>>> 
>>>>>>> On 10/06/2013 02:10 PM, Sander Eikelenboom wrote:
>>>>>>>> Sunday, October 6, 2013, 1:36:33 PM, you wrote:
>>>>>>>> 
>>>>>>>>> Hi!
>>>>>>>>>> From other people posting to this list, I know
>>>>>>>>>> that there has been a
>>>>>>>>> bug related to the issue described in Xen Security 
>>>>>>>>> Advisory 36 that disables iommu for some AMD users
>>>>>>>>> like me.
>>>>>>>> What motherboard do you have ?
>>>>>>> I have an ASUS M5A99X EVO Rev 1.01
>>>>>>> 
>>>>>>> 
>>>>>>>> It depends if it's the north or southbridge ioapic but
>>>>>>>> try using the new xen boot parameter: (add it to the
>>>>>>>> xen.gz line in grub)
>>>>>>>> 
>>>>>>>> ivrs_ioapic[0]=00:14.0
>>>>>>>> 
>>>>>>>> or
>>>>>>>> 
>>>>>>>> ivrs_ioapic[0]=00:00.1
>>>>>>> I tried both, it still fails to enable i/o
>>>>>>> virtualisation.
>>>>>> 
>>>>>> Can you post a full xl dmesg, booting with
>>>>>> iommu=debug,verbose as well?
>>>>>> 
>>>>>> ~Andrew
>>>>>> 
>>>>>> 
>>>> 
>>>> Ah it seems i got the id's mixed up :-)
>>>> 
>>>> ivrs_ioapic[9]=00:00.1 ivrs_ioapic[10]=00:14.0
>>>> 
>>>> Should do it i guess .. or should the ioapic handle be in hex
>>>> .. (then the last should be ivrs_ioapic[a]=00:14.0 )
>>>> 
>>>> -- Sander
>>>> 
>> 
>>> Booting with this parameters I can see "I/O Virtualisation
>>> Enabled" but shortly after that, I get a CPU Panic:
>> 
>>> Xen Call Trace: amd_iommu_ioapic_update_ire ... 
>>> iommu_update_ire_from_apic ... set_ioapic_affinity_irq ... 
>>> smp_cpus_done ... __start_xen ...
>> 
>>> Panic on CPU 0: Assertion "get_rte_index(rte) == offset" failed
>>> at iommu_intr.c:188
>> 
>> 
>> Hmm, could you check if you can get a recent linux kernel to boot
>> on baremetal (so without xen) with the iommu enabled ? (that would
>> most certainly require one or both of the ivrs_ioapic overrides as
>> well)
>> 
>> I assumed but don't know for sure, did you previously had
>> successfully used the iommu with the
>> iommu=no-amd-iommu-perdev-intremap boot option ? And i understand
>> you upgraded your bios, if the previous answer was yes .. did you
>> try to revert to your previous bios version ?
>> 
>> Also added Jan and Suravee to the CC.
>> 
>> -- Sander
>> 
>> BTW: please keep everyone that was in the CC listed in replies.
>> 
>> 
>> _______________________________________________ Xen-devel mailing
>> list Xen-devel@lists.xen.org http://lists.xen.org/xen-devel
>> 
>> 

> What "recent" version? Currently, I'm using 3.10-3, is that ok?

I think so, you will need this patchset in the kernel:
http://lkml.indiana.edu/hypermail/linux/kernel/1304.1/01567.html
which implemented the overrides for linux.


> I have used iommu successfully before they added the new checking
> stuff, so I can't tell for sure.
> I just tried the new xen version because it kept panicking after some
> changes half a year ago. Now it's working again but without iommu.
> That's why I updated my bios, sadly, without any success.

I know the pain :-)

> I'll try booting with Andrews patch now.

> kind regards

Re: AMD IOMMU disabled - No Perdev Intremap

[Ferdinand Nölscher]

On 10/06/2013 08:03 PM, Sander Eikelenboom wrote:
> 
> Sunday, October 6, 2013, 7:57:42 PM, you wrote:
> 
>> On 10/06/2013 07:45 PM, Sander Eikelenboom wrote:
>>> 
>>> Sunday, October 6, 2013, 7:28:03 PM, you wrote:
>>> 
>>>> On 10/06/2013 04:38 PM, Sander Eikelenboom wrote:
>>>>> 
>>>>> Sunday, October 6, 2013, 3:09:54 PM, you wrote:
>>>>> 
>>>>>> On 10/06/2013 02:58 PM, Andrew Cooper wrote:
>>>>>>> On 06/10/2013 13:56, Ferdinand Nölscher wrote:
>>>>>>>> Hi Sander,
>>>>>>>> 
>>>>>>>> Thanks for your reply.
>>>>>>>> 
>>>>>>>> On 10/06/2013 02:10 PM, Sander Eikelenboom wrote:
>>>>>>>>> Sunday, October 6, 2013, 1:36:33 PM, you wrote:
>>>>>>>>> 
>>>>>>>>>> Hi!
>>>>>>>>>>> From other people posting to this list, I know 
>>>>>>>>>>> that there has been a
>>>>>>>>>> bug related to the issue described in Xen
>>>>>>>>>> Security Advisory 36 that disables iommu for some
>>>>>>>>>> AMD users like me.
>>>>>>>>> What motherboard do you have ?
>>>>>>>> I have an ASUS M5A99X EVO Rev 1.01
>>>>>>>> 
>>>>>>>> 
>>>>>>>>> It depends if it's the north or southbridge ioapic
>>>>>>>>> but try using the new xen boot parameter: (add it
>>>>>>>>> to the xen.gz line in grub)
>>>>>>>>> 
>>>>>>>>> ivrs_ioapic[0]=00:14.0
>>>>>>>>> 
>>>>>>>>> or
>>>>>>>>> 
>>>>>>>>> ivrs_ioapic[0]=00:00.1
>>>>>>>> I tried both, it still fails to enable i/o 
>>>>>>>> virtualisation.
>>>>>>> 
>>>>>>> Can you post a full xl dmesg, booting with 
>>>>>>> iommu=debug,verbose as well?
>>>>>>> 
>>>>>>> ~Andrew
>>>>>>> 
>>>>>>> 
>>>>> 
>>>>> Ah it seems i got the id's mixed up :-)
>>>>> 
>>>>> ivrs_ioapic[9]=00:00.1 ivrs_ioapic[10]=00:14.0
>>>>> 
>>>>> Should do it i guess .. or should the ioapic handle be in
>>>>> hex .. (then the last should be ivrs_ioapic[a]=00:14.0 )
>>>>> 
>>>>> -- Sander
>>>>> 
>>> 
>>>> Booting with this parameters I can see "I/O Virtualisation 
>>>> Enabled" but shortly after that, I get a CPU Panic:
>>> 
>>>> Xen Call Trace: amd_iommu_ioapic_update_ire ... 
>>>> iommu_update_ire_from_apic ... set_ioapic_affinity_irq ... 
>>>> smp_cpus_done ... __start_xen ...
>>> 
>>>> Panic on CPU 0: Assertion "get_rte_index(rte) == offset"
>>>> failed at iommu_intr.c:188
>>> 
>>> 
>>> Hmm, could you check if you can get a recent linux kernel to
>>> boot on baremetal (so without xen) with the iommu enabled ?
>>> (that would most certainly require one or both of the
>>> ivrs_ioapic overrides as well)
>>> 
>>> I assumed but don't know for sure, did you previously had 
>>> successfully used the iommu with the 
>>> iommu=no-amd-iommu-perdev-intremap boot option ? And i
>>> understand you upgraded your bios, if the previous answer was
>>> yes .. did you try to revert to your previous bios version ?
>>> 
>>> Also added Jan and Suravee to the CC.
>>> 
>>> -- Sander
>>> 
>>> BTW: please keep everyone that was in the CC listed in
>>> replies.
>>> 
>>> 
>>> _______________________________________________ Xen-devel
>>> mailing list Xen-devel@lists.xen.org
>>> http://lists.xen.org/xen-devel
>>> 
>>> 
> 
>> What "recent" version? Currently, I'm using 3.10-3, is that ok?
> 
> I think so, you will need this patchset in the kernel: 
> http://lkml.indiana.edu/hypermail/linux/kernel/1304.1/01567.html 
> which implemented the overrides for linux.
> 
> 
>> I have used iommu successfully before they added the new
>> checking stuff, so I can't tell for sure. I just tried the new
>> xen version because it kept panicking after some changes half a
>> year ago. Now it's working again but without iommu. That's why I
>> updated my bios, sadly, without any success.
> 
> I know the pain :-)
> 
>> I'll try booting with Andrews patch now.
> 
>> kind regards
> 
> 
> 
> 

Today, I tested booting with the latest 3.12 kernel only, without xen,
and it panicked after complaining about faulty remaps - sadly it
didn't write any logs.

I will look into this further the next days and I will post the
information you asked for.

kind regards

Re: AMD IOMMU disabled - No Perdev Intremap

[Sander Eikelenboom]

Tuesday, October 8, 2013, 6:39:13 PM, you wrote:

> On 10/06/2013 08:03 PM, Sander Eikelenboom wrote:
>> 
>> Sunday, October 6, 2013, 7:57:42 PM, you wrote:
>> 
>>> On 10/06/2013 07:45 PM, Sander Eikelenboom wrote:
>>>> 
>>>> Sunday, October 6, 2013, 7:28:03 PM, you wrote:
>>>> 
>>>>> On 10/06/2013 04:38 PM, Sander Eikelenboom wrote:
>>>>>> 
>>>>>> Sunday, October 6, 2013, 3:09:54 PM, you wrote:
>>>>>> 
>>>>>>> On 10/06/2013 02:58 PM, Andrew Cooper wrote:
>>>>>>>> On 06/10/2013 13:56, Ferdinand Nölscher wrote:
>>>>>>>>> Hi Sander,
>>>>>>>>> 
>>>>>>>>> Thanks for your reply.
>>>>>>>>> 
>>>>>>>>> On 10/06/2013 02:10 PM, Sander Eikelenboom wrote:
>>>>>>>>>> Sunday, October 6, 2013, 1:36:33 PM, you wrote:
>>>>>>>>>> 
>>>>>>>>>>> Hi!
>>>>>>>>>>>> From other people posting to this list, I know 
>>>>>>>>>>>> that there has been a
>>>>>>>>>>> bug related to the issue described in Xen
>>>>>>>>>>> Security Advisory 36 that disables iommu for some
>>>>>>>>>>> AMD users like me.
>>>>>>>>>> What motherboard do you have ?
>>>>>>>>> I have an ASUS M5A99X EVO Rev 1.01
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>>> It depends if it's the north or southbridge ioapic
>>>>>>>>>> but try using the new xen boot parameter: (add it
>>>>>>>>>> to the xen.gz line in grub)
>>>>>>>>>> 
>>>>>>>>>> ivrs_ioapic[0]=00:14.0
>>>>>>>>>> 
>>>>>>>>>> or
>>>>>>>>>> 
>>>>>>>>>> ivrs_ioapic[0]=00:00.1
>>>>>>>>> I tried both, it still fails to enable i/o 
>>>>>>>>> virtualisation.
>>>>>>>> 
>>>>>>>> Can you post a full xl dmesg, booting with 
>>>>>>>> iommu=debug,verbose as well?
>>>>>>>> 
>>>>>>>> ~Andrew
>>>>>>>> 
>>>>>>>> 
>>>>>> 
>>>>>> Ah it seems i got the id's mixed up :-)
>>>>>> 
>>>>>> ivrs_ioapic[9]=00:00.1 ivrs_ioapic[10]=00:14.0
>>>>>> 
>>>>>> Should do it i guess .. or should the ioapic handle be in
>>>>>> hex .. (then the last should be ivrs_ioapic[a]=00:14.0 )
>>>>>> 
>>>>>> -- Sander
>>>>>> 
>>>> 
>>>>> Booting with this parameters I can see "I/O Virtualisation 
>>>>> Enabled" but shortly after that, I get a CPU Panic:
>>>> 
>>>>> Xen Call Trace: amd_iommu_ioapic_update_ire ... 
>>>>> iommu_update_ire_from_apic ... set_ioapic_affinity_irq ... 
>>>>> smp_cpus_done ... __start_xen ...
>>>> 
>>>>> Panic on CPU 0: Assertion "get_rte_index(rte) == offset"
>>>>> failed at iommu_intr.c:188
>>>> 
>>>> 
>>>> Hmm, could you check if you can get a recent linux kernel to
>>>> boot on baremetal (so without xen) with the iommu enabled ?
>>>> (that would most certainly require one or both of the
>>>> ivrs_ioapic overrides as well)
>>>> 
>>>> I assumed but don't know for sure, did you previously had 
>>>> successfully used the iommu with the 
>>>> iommu=no-amd-iommu-perdev-intremap boot option ? And i
>>>> understand you upgraded your bios, if the previous answer was
>>>> yes .. did you try to revert to your previous bios version ?
>>>> 
>>>> Also added Jan and Suravee to the CC.
>>>> 
>>>> -- Sander
>>>> 
>>>> BTW: please keep everyone that was in the CC listed in
>>>> replies.
>>>> 
>>>> 
>>>> _______________________________________________ Xen-devel
>>>> mailing list Xen-devel@lists.xen.org
>>>> http://lists.xen.org/xen-devel
>>>> 
>>>> 
>> 
>>> What "recent" version? Currently, I'm using 3.10-3, is that ok?
>> 
>> I think so, you will need this patchset in the kernel: 
>> http://lkml.indiana.edu/hypermail/linux/kernel/1304.1/01567.html 
>> which implemented the overrides for linux.
>> 
>> 
>>> I have used iommu successfully before they added the new
>>> checking stuff, so I can't tell for sure. I just tried the new
>>> xen version because it kept panicking after some changes half a
>>> year ago. Now it's working again but without iommu. That's why I
>>> updated my bios, sadly, without any success.
>> 
>> I know the pain :-)
>> 
>>> I'll try booting with Andrews patch now.
>> 
>>> kind regards
>> 
>> 
>> 
>> 

> Today, I tested booting with the latest 3.12 kernel only, without xen,
> and it panicked after complaining about faulty remaps - sadly it
> didn't write any logs.

> I will look into this further the next days and I will post the
> information you asked for.

One other thing i would also try is to revert to your older bios with which you had
a working iommu in the past on a older xen version (with the no-amd-iommu-perdev-intremap option)

It could very well be you more recent bios is actually worse than your previous was ..

--
Sander

> kind regards

Re: AMD IOMMU disabled - No Perdev Intremap

[Jan Beulich]

>>> On 06.10.13 at 19:28, Ferdinand Nölscher<ferdinand@outlook.com> wrote:
> On 10/06/2013 04:38 PM, Sander Eikelenboom wrote:
>> ivrs_ioapic[9]=00:00.1 ivrs_ioapic[10]=00:14.0
>> 
>> Should do it i guess .. or should the ioapic handle be in hex ..
>> (then the last should be ivrs_ioapic[a]=00:14.0 )
>> 
>> -- Sander
>> 
> 
> Booting with this parameters I can see "I/O Virtualisation Enabled"
> but shortly after that, I get a CPU Panic:
> 
> Xen Call Trace:
> amd_iommu_ioapic_update_ire ...
> iommu_update_ire_from_apic ...
> set_ioapic_affinity_irq ...
> smp_cpus_done ...
> __start_xen ...
> 
> Panic on CPU 0:
> Assertion "get_rte_index(rte) == offset" failed at iommu_intr.c:188

Please provide a _full_ log and, in order to decipher things, a link
to the matching xen-syms or xen.efi (depending on the way you
boot - GrUB2 apparently being used, I'd guess it's the former we
need).

Also - how certain are both of you that the exact values used in
the overrides are valid? After all - if they aren't, we can't make
any guarantees that the system would work _at all_.

Jan

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: AMD IOMMU disabled - No Perdev Intremap

[Andrew Cooper]

On 06/10/2013 14:09, Ferdinand Nölscher wrote:
> On 10/06/2013 02:58 PM, Andrew Cooper wrote:
>> On 06/10/2013 13:56, Ferdinand Nölscher wrote:
>>> Hi Sander,
>>>
>>> Thanks for your reply.
>>>
>>> On 10/06/2013 02:10 PM, Sander Eikelenboom wrote:
>>>> Sunday, October 6, 2013, 1:36:33 PM, you wrote:
>>>>
>>>>> Hi!
>>>>>> From other people posting to this list, I know that there has
>>>>>> been a
>>>>> bug related to the issue described in Xen Security Advisory 36
>>>>> that disables iommu for some AMD users like me.
>>>> What motherboard do you have ?
>>> I have an ASUS M5A99X EVO Rev 1.01
>>>
>>>
>>>> It depends if it's the north or southbridge ioapic but try using
>>>> the new xen boot parameter: (add it to the xen.gz line in grub)
>>>>
>>>> ivrs_ioapic[0]=00:14.0
>>>>
>>>> or
>>>>
>>>> ivrs_ioapic[0]=00:00.1
>>> I tried both, it still fails to enable i/o virtualisation.
>> Can you post a full xl dmesg, booting with iommu=debug,verbose as well?
>>
>> ~Andrew
>>
>>
> Hi Andrew,
> This is what I got:

<snip>

>
> (XEN) AMD-Vi: Found MSI capability block at 0x54
> (XEN) AMD-Vi: ACPI Table:
> (XEN) AMD-Vi:  Signature IVRS
> (XEN) AMD-Vi:  Length 0xd8
> (XEN) AMD-Vi:  Revision 0x1
> (XEN) AMD-Vi:  CheckSum 0x22
> (XEN) AMD-Vi:  OEM_Id AMD
> (XEN) AMD-Vi:  OEM_Table_Id RD890S
> (XEN) AMD-Vi:  OEM_Revision 0x202031
> (XEN) AMD-Vi:  Creator_Id AMD
> (XEN) AMD-Vi:  Creator_Revision 0
> (XEN) AMD-Vi: IVRS Block: type 0x10 flags 0x3e len 0xa8 id 0x2
> (XEN) AMD-Vi: IVHD Device Entry: type 0x3 id 0 flags 0
> (XEN) AMD-Vi:  Dev_Id Range: 0 -> 0x2
> (XEN) AMD-Vi: IVHD Device Entry: type 0x2 id 0x10 flags 0
> (XEN) AMD-Vi: IVHD Device Entry: type 0x3 id 0x100 flags 0
> (XEN) AMD-Vi:  Dev_Id Range: 0x100 -> 0x101
> (XEN) AMD-Vi: IVHD Device Entry: type 0x2 id 0x20 flags 0
> (XEN) AMD-Vi: IVHD Device Entry: type 0x2 id 0x200 flags 0
> (XEN) AMD-Vi: IVHD Device Entry: type 0x2 id 0x28 flags 0
> (XEN) AMD-Vi: IVHD Device Entry: type 0x2 id 0x300 flags 0
> (XEN) AMD-Vi: IVHD Device Entry: type 0x2 id 0x30 flags 0
> (XEN) AMD-Vi: IVHD Device Entry: type 0x2 id 0x400 flags 0
> (XEN) AMD-Vi: IVHD Device Entry: type 0x2 id 0x38 flags 0
> (XEN) AMD-Vi: IVHD Device Entry: type 0x2 id 0x500 flags 0
> (XEN) AMD-Vi: IVHD Device Entry: type 0x2 id 0x50 flags 0
> (XEN) AMD-Vi: IVHD Device Entry: type 0x2 id 0x600 flags 0
> (XEN) AMD-Vi: IVHD Device Entry: type 0x2 id 0x88 flags 0
> (XEN) AMD-Vi: IVHD Device Entry: type 0x3 id 0x90 flags 0
> (XEN) AMD-Vi:  Dev_Id Range: 0x90 -> 0x92
> (XEN) AMD-Vi: IVHD Device Entry: type 0x3 id 0x98 flags 0
> (XEN) AMD-Vi:  Dev_Id Range: 0x98 -> 0x9a
> (XEN) AMD-Vi: IVHD Device Entry: type 0x2 id 0xa0 flags 0xd7
> (XEN) AMD-Vi: IVHD Device Entry: type 0x2 id 0xa2 flags 0
> (XEN) AMD-Vi: IVHD Device Entry: type 0x2 id 0xa3 flags 0
> (XEN) AMD-Vi: IVHD Device Entry: type 0x2 id 0xa4 flags 0
> (XEN) AMD-Vi: IVHD Device Entry: type 0x43 id 0x700 flags 0
> (XEN) AMD-Vi:  Dev_Id Range: 0x700 -> 0x7ff alias 0xa4
> (XEN) AMD-Vi: IVHD Device Entry: type 0x2 id 0xa5 flags 0
> (XEN) AMD-Vi: IVHD Device Entry: type 0x3 id 0xb0 flags 0
> (XEN) AMD-Vi:  Dev_Id Range: 0xb0 -> 0xb2
> (XEN) AMD-Vi: IVHD Device Entry: type 0x48 id 0 flags 0xd7
> (XEN) AMD-Vi: IVHD Special: 0000:00:14.0 variety 0x1 handle 0
> (XEN) IVHD Error: Invalid IO-APIC 0
> (XEN) AMD-Vi: Error initialization
> (XEN) I/O virtualisation disabled
> (XEN) ENABLING IO-APIC IRQs
> (XEN)  -> Using new ACK method

Ok - my gut feeling is that this is a bug with the new command line
override code.  There is certainly not enough debug printing to work out
exactly what is going on, and at a glance of the other information in
the log, the state looks reasonable.  (cc'ing Jan and Suravee as the
authors)

Could you try running with this patch (compile tested only) and seeing
what it says?

~Andrew

8<------
diff --git a/xen/drivers/passthrough/amd/iommu_acpi.c
b/xen/drivers/passthrough/amd/iommu_acpi.c
index fca2037..66755fa 100644
--- a/xen/drivers/passthrough/amd/iommu_acpi.c
+++ b/xen/drivers/passthrough/amd/iommu_acpi.c
@@ -25,6 +25,7 @@
 #include <asm/io_apic.h>
 #include <asm/amd-iommu.h>
 #include <asm/hvm/svm/amd-iommu-proto.h>
+#include <xen/keyhandler.h>

 /* Some helper structures, particularly to deal with ranges. */

@@ -785,6 +786,11 @@ static u16 __init parse_ivhd_device_special(
         {
             printk(XENLOG_ERR "IVHD Error: Invalid IO-APIC %#x\n",
                    special->handle);
+            bitmap_scnprintf(keyhandler_scratch,
sizeof(keyhandler_scratch),
+                             ioapic_cmdline,
+                             BITS_TO_LONGS(ARRAY_SIZE(ioapic_sbdf)) );
+            printk("**DEBUG: apic = %d, nr_ioapics = %d, ioapic_cmdline
= %s\n",
+                   apic, nr_ioapics, keyhandler_scratch);
             return 0;
         }
         break;

Re: AMD IOMMU disabled - No Perdev Intremap

[Ferdinand Nölscher]

On 10/06/2013 04:40 PM, Andrew Cooper wrote:
> On 06/10/2013 14:09, Ferdinand Nölscher wrote:
>> On 10/06/2013 02:58 PM, Andrew Cooper wrote:
>>> On 06/10/2013 13:56, Ferdinand Nölscher wrote:
>>>> Hi Sander,
>>>>
>>>> Thanks for your reply.
>>>>
>>>> On 10/06/2013 02:10 PM, Sander Eikelenboom wrote:
>>>>> Sunday, October 6, 2013, 1:36:33 PM, you wrote:
>>>>>
>>>>>> Hi!
>>>>>>> From other people posting to this list, I know that there has
>>>>>>> been a
>>>>>> bug related to the issue described in Xen Security Advisory 36
>>>>>> that disables iommu for some AMD users like me.
>>>>> What motherboard do you have ?
>>>> I have an ASUS M5A99X EVO Rev 1.01
>>>>
>>>>
>>>>> It depends if it's the north or southbridge ioapic but try using
>>>>> the new xen boot parameter: (add it to the xen.gz line in grub)
>>>>>
>>>>> ivrs_ioapic[0]=00:14.0
>>>>>
>>>>> or
>>>>>
>>>>> ivrs_ioapic[0]=00:00.1
>>>> I tried both, it still fails to enable i/o virtualisation.
>>> Can you post a full xl dmesg, booting with iommu=debug,verbose as well?
>>>
>>> ~Andrew
>>>
>>>
>> Hi Andrew,
>> This is what I got:
> 
> <snip>
> 
>>
>> (XEN) AMD-Vi: Found MSI capability block at 0x54
>> (XEN) AMD-Vi: ACPI Table:
>> (XEN) AMD-Vi:  Signature IVRS
>> (XEN) AMD-Vi:  Length 0xd8
>> (XEN) AMD-Vi:  Revision 0x1
>> (XEN) AMD-Vi:  CheckSum 0x22
>> (XEN) AMD-Vi:  OEM_Id AMD
>> (XEN) AMD-Vi:  OEM_Table_Id RD890S
>> (XEN) AMD-Vi:  OEM_Revision 0x202031
>> (XEN) AMD-Vi:  Creator_Id AMD
>> (XEN) AMD-Vi:  Creator_Revision 0
>> (XEN) AMD-Vi: IVRS Block: type 0x10 flags 0x3e len 0xa8 id 0x2
>> (XEN) AMD-Vi: IVHD Device Entry: type 0x3 id 0 flags 0
>> (XEN) AMD-Vi:  Dev_Id Range: 0 -> 0x2
>> (XEN) AMD-Vi: IVHD Device Entry: type 0x2 id 0x10 flags 0
>> (XEN) AMD-Vi: IVHD Device Entry: type 0x3 id 0x100 flags 0
>> (XEN) AMD-Vi:  Dev_Id Range: 0x100 -> 0x101
>> (XEN) AMD-Vi: IVHD Device Entry: type 0x2 id 0x20 flags 0
>> (XEN) AMD-Vi: IVHD Device Entry: type 0x2 id 0x200 flags 0
>> (XEN) AMD-Vi: IVHD Device Entry: type 0x2 id 0x28 flags 0
>> (XEN) AMD-Vi: IVHD Device Entry: type 0x2 id 0x300 flags 0
>> (XEN) AMD-Vi: IVHD Device Entry: type 0x2 id 0x30 flags 0
>> (XEN) AMD-Vi: IVHD Device Entry: type 0x2 id 0x400 flags 0
>> (XEN) AMD-Vi: IVHD Device Entry: type 0x2 id 0x38 flags 0
>> (XEN) AMD-Vi: IVHD Device Entry: type 0x2 id 0x500 flags 0
>> (XEN) AMD-Vi: IVHD Device Entry: type 0x2 id 0x50 flags 0
>> (XEN) AMD-Vi: IVHD Device Entry: type 0x2 id 0x600 flags 0
>> (XEN) AMD-Vi: IVHD Device Entry: type 0x2 id 0x88 flags 0
>> (XEN) AMD-Vi: IVHD Device Entry: type 0x3 id 0x90 flags 0
>> (XEN) AMD-Vi:  Dev_Id Range: 0x90 -> 0x92
>> (XEN) AMD-Vi: IVHD Device Entry: type 0x3 id 0x98 flags 0
>> (XEN) AMD-Vi:  Dev_Id Range: 0x98 -> 0x9a
>> (XEN) AMD-Vi: IVHD Device Entry: type 0x2 id 0xa0 flags 0xd7
>> (XEN) AMD-Vi: IVHD Device Entry: type 0x2 id 0xa2 flags 0
>> (XEN) AMD-Vi: IVHD Device Entry: type 0x2 id 0xa3 flags 0
>> (XEN) AMD-Vi: IVHD Device Entry: type 0x2 id 0xa4 flags 0
>> (XEN) AMD-Vi: IVHD Device Entry: type 0x43 id 0x700 flags 0
>> (XEN) AMD-Vi:  Dev_Id Range: 0x700 -> 0x7ff alias 0xa4
>> (XEN) AMD-Vi: IVHD Device Entry: type 0x2 id 0xa5 flags 0
>> (XEN) AMD-Vi: IVHD Device Entry: type 0x3 id 0xb0 flags 0
>> (XEN) AMD-Vi:  Dev_Id Range: 0xb0 -> 0xb2
>> (XEN) AMD-Vi: IVHD Device Entry: type 0x48 id 0 flags 0xd7
>> (XEN) AMD-Vi: IVHD Special: 0000:00:14.0 variety 0x1 handle 0
>> (XEN) IVHD Error: Invalid IO-APIC 0
>> (XEN) AMD-Vi: Error initialization
>> (XEN) I/O virtualisation disabled
>> (XEN) ENABLING IO-APIC IRQs
>> (XEN)  -> Using new ACK method
> 
> Ok - my gut feeling is that this is a bug with the new command line
> override code.  There is certainly not enough debug printing to work out
> exactly what is going on, and at a glance of the other information in
> the log, the state looks reasonable.  (cc'ing Jan and Suravee as the
> authors)
> 
> Could you try running with this patch (compile tested only) and seeing
> what it says?
> 
> ~Andrew
> 
> 8<------
> diff --git a/xen/drivers/passthrough/amd/iommu_acpi.c
> b/xen/drivers/passthrough/amd/iommu_acpi.c
> index fca2037..66755fa 100644
> --- a/xen/drivers/passthrough/amd/iommu_acpi.c
> +++ b/xen/drivers/passthrough/amd/iommu_acpi.c
> @@ -25,6 +25,7 @@
>  #include <asm/io_apic.h>
>  #include <asm/amd-iommu.h>
>  #include <asm/hvm/svm/amd-iommu-proto.h>
> +#include <xen/keyhandler.h>
> 
>  /* Some helper structures, particularly to deal with ranges. */
> 
> @@ -785,6 +786,11 @@ static u16 __init parse_ivhd_device_special(
>          {
>              printk(XENLOG_ERR "IVHD Error: Invalid IO-APIC %#x\n",
>                     special->handle);
> +            bitmap_scnprintf(keyhandler_scratch,
> sizeof(keyhandler_scratch),
> +                             ioapic_cmdline,
> +                             BITS_TO_LONGS(ARRAY_SIZE(ioapic_sbdf)) );
> +            printk("**DEBUG: apic = %d, nr_ioapics = %d, ioapic_cmdline
> = %s\n",
> +                   apic, nr_ioapics, keyhandler_scratch);
>              return 0;
>          }
>          break;
> 
> 
> 

I applied your patch, this is the related output:

(XEN) AMD-Vi: Found MSI capability block at 0x54
(XEN) AMD-Vi: ACPI Table:
(XEN) AMD-Vi:  Signature IVRS
(XEN) AMD-Vi:  Length 0xd8
(XEN) AMD-Vi:  Revision 0x1
(XEN) AMD-Vi:  CheckSum 0x22
(XEN) AMD-Vi:  OEM_Id AMD
(XEN) AMD-Vi:  OEM_Table_Id RD890S
(XEN) AMD-Vi:  OEM_Revision 0x202031
(XEN) AMD-Vi:  Creator_Id AMD
(XEN) AMD-Vi:  Creator_Revision 0
(XEN) AMD-Vi: IVRS Block: type 0x10 flags 0x3e len 0xa8 id 0x2
(XEN) AMD-Vi: IVHD Device Entry: type 0x3 id 0 flags 0
(XEN) AMD-Vi:  Dev_Id Range: 0 -> 0x2
(XEN) AMD-Vi: IVHD Device Entry: type 0x2 id 0x10 flags 0
(XEN) AMD-Vi: IVHD Device Entry: type 0x3 id 0x100 flags 0
(XEN) AMD-Vi:  Dev_Id Range: 0x100 -> 0x101
(XEN) AMD-Vi: IVHD Device Entry: type 0x2 id 0x20 flags 0
(XEN) AMD-Vi: IVHD Device Entry: type 0x2 id 0x200 flags 0
(XEN) AMD-Vi: IVHD Device Entry: type 0x2 id 0x28 flags 0
(XEN) AMD-Vi: IVHD Device Entry: type 0x2 id 0x300 flags 0
(XEN) AMD-Vi: IVHD Device Entry: type 0x2 id 0x30 flags 0
(XEN) AMD-Vi: IVHD Device Entry: type 0x2 id 0x400 flags 0
(XEN) AMD-Vi: IVHD Device Entry: type 0x2 id 0x38 flags 0
(XEN) AMD-Vi: IVHD Device Entry: type 0x2 id 0x500 flags 0
(XEN) AMD-Vi: IVHD Device Entry: type 0x2 id 0x50 flags 0
(XEN) AMD-Vi: IVHD Device Entry: type 0x2 id 0x600 flags 0
(XEN) AMD-Vi: IVHD Device Entry: type 0x2 id 0x88 flags 0
(XEN) AMD-Vi: IVHD Device Entry: type 0x3 id 0x90 flags 0
(XEN) AMD-Vi:  Dev_Id Range: 0x90 -> 0x92
(XEN) AMD-Vi: IVHD Device Entry: type 0x3 id 0x98 flags 0
(XEN) AMD-Vi:  Dev_Id Range: 0x98 -> 0x9a
(XEN) AMD-Vi: IVHD Device Entry: type 0x2 id 0xa0 flags 0xd7
(XEN) AMD-Vi: IVHD Device Entry: type 0x2 id 0xa2 flags 0
(XEN) AMD-Vi: IVHD Device Entry: type 0x2 id 0xa3 flags 0
(XEN) AMD-Vi: IVHD Device Entry: type 0x2 id 0xa4 flags 0
(XEN) AMD-Vi: IVHD Device Entry: type 0x43 id 0x700 flags 0
(XEN) AMD-Vi:  Dev_Id Range: 0x700 -> 0x7ff alias 0xa4
(XEN) AMD-Vi: IVHD Device Entry: type 0x2 id 0xa5 flags 0
(XEN) AMD-Vi: IVHD Device Entry: type 0x3 id 0xb0 flags 0
(XEN) AMD-Vi:  Dev_Id Range: 0xb0 -> 0xb2
(XEN) AMD-Vi: IVHD Device Entry: type 0x48 id 0 flags 0xd7
(XEN) AMD-Vi: IVHD Special: 0000:00:14.0 variety 0x1 handle 0
(XEN) IVHD Error: Invalid IO-APIC 0
(XEN) **DEBUG: apic = 2, nr_ioapics = 2, ioapic_cmdline = 0
(XEN) AMD-Vi: Error initialization
(XEN) I/O virtualisation disabled
(XEN) ENABLING IO-APIC IRQs
(XEN)  -> Using new ACK method

kind regards

Re: AMD IOMMU disabled - No Perdev Intremap

[Suravee Suthikulpanit]

[-- Attachment #1.1: Type: text/plain, Size: 8296 bytes --]

On 10/6/2013 1:14 PM, Ferdinand Nölscher wrote:
> On 10/06/2013 04:40 PM, Andrew Cooper wrote:
>> On 06/10/2013 14:09, Ferdinand Nölscher wrote:
>>> On 10/06/2013 02:58 PM, Andrew Cooper wrote:
>>>> On 06/10/2013 13:56, Ferdinand Nölscher wrote:
>>>>> Hi Sander,
>>>>>
>>>>> Thanks for your reply.
>>>>>
>>>>> On 10/06/2013 02:10 PM, Sander Eikelenboom wrote:
>>>>>> Sunday, October 6, 2013, 1:36:33 PM, you wrote:
>>>>>>
>>>>>>> Hi!
>>>>>>>>  From other people posting to this list, I know that there has
>>>>>>>> been a
>>>>>>> bug related to the issue described in Xen Security Advisory 36
>>>>>>> that disables iommu for some AMD users like me.
>>>>>> What motherboard do you have ?
>>>>> I have an ASUS M5A99X EVO Rev 1.01
>>>>>
>>>>>
>>>>>> It depends if it's the north or southbridge ioapic but try using
>>>>>> the new xen boot parameter: (add it to the xen.gz line in grub)
>>>>>>
>>>>>> ivrs_ioapic[0]=00:14.0
>>>>>>
>>>>>> or
>>>>>>
>>>>>> ivrs_ioapic[0]=00:00.1
>>>>> I tried both, it still fails to enable i/o virtualisation.
>>>> Can you post a full xl dmesg, booting with iommu=debug,verbose as well?
>>>>
>>>> ~Andrew
>>>>
>>>>
>>> Hi Andrew,
>>> This is what I got:
>> <snip>
>>
>>> (XEN) AMD-Vi: Found MSI capability block at 0x54
>>> (XEN) AMD-Vi: ACPI Table:
>>> (XEN) AMD-Vi:  Signature IVRS
>>> (XEN) AMD-Vi:  Length 0xd8
>>> (XEN) AMD-Vi:  Revision 0x1
>>> (XEN) AMD-Vi:  CheckSum 0x22
>>> (XEN) AMD-Vi:  OEM_Id AMD
>>> (XEN) AMD-Vi:  OEM_Table_Id RD890S
>>> (XEN) AMD-Vi:  OEM_Revision 0x202031
>>> (XEN) AMD-Vi:  Creator_Id AMD
>>> (XEN) AMD-Vi:  Creator_Revision 0
>>> (XEN) AMD-Vi: IVRS Block: type 0x10 flags 0x3e len 0xa8 id 0x2
>>> (XEN) AMD-Vi: IVHD Device Entry: type 0x3 id 0 flags 0
>>> (XEN) AMD-Vi:  Dev_Id Range: 0 -> 0x2
>>> (XEN) AMD-Vi: IVHD Device Entry: type 0x2 id 0x10 flags 0
>>> (XEN) AMD-Vi: IVHD Device Entry: type 0x3 id 0x100 flags 0
>>> (XEN) AMD-Vi:  Dev_Id Range: 0x100 -> 0x101
>>> (XEN) AMD-Vi: IVHD Device Entry: type 0x2 id 0x20 flags 0
>>> (XEN) AMD-Vi: IVHD Device Entry: type 0x2 id 0x200 flags 0
>>> (XEN) AMD-Vi: IVHD Device Entry: type 0x2 id 0x28 flags 0
>>> (XEN) AMD-Vi: IVHD Device Entry: type 0x2 id 0x300 flags 0
>>> (XEN) AMD-Vi: IVHD Device Entry: type 0x2 id 0x30 flags 0
>>> (XEN) AMD-Vi: IVHD Device Entry: type 0x2 id 0x400 flags 0
>>> (XEN) AMD-Vi: IVHD Device Entry: type 0x2 id 0x38 flags 0
>>> (XEN) AMD-Vi: IVHD Device Entry: type 0x2 id 0x500 flags 0
>>> (XEN) AMD-Vi: IVHD Device Entry: type 0x2 id 0x50 flags 0
>>> (XEN) AMD-Vi: IVHD Device Entry: type 0x2 id 0x600 flags 0
>>> (XEN) AMD-Vi: IVHD Device Entry: type 0x2 id 0x88 flags 0
>>> (XEN) AMD-Vi: IVHD Device Entry: type 0x3 id 0x90 flags 0
>>> (XEN) AMD-Vi:  Dev_Id Range: 0x90 -> 0x92
>>> (XEN) AMD-Vi: IVHD Device Entry: type 0x3 id 0x98 flags 0
>>> (XEN) AMD-Vi:  Dev_Id Range: 0x98 -> 0x9a
>>> (XEN) AMD-Vi: IVHD Device Entry: type 0x2 id 0xa0 flags 0xd7
>>> (XEN) AMD-Vi: IVHD Device Entry: type 0x2 id 0xa2 flags 0
>>> (XEN) AMD-Vi: IVHD Device Entry: type 0x2 id 0xa3 flags 0
>>> (XEN) AMD-Vi: IVHD Device Entry: type 0x2 id 0xa4 flags 0
>>> (XEN) AMD-Vi: IVHD Device Entry: type 0x43 id 0x700 flags 0
>>> (XEN) AMD-Vi:  Dev_Id Range: 0x700 -> 0x7ff alias 0xa4
>>> (XEN) AMD-Vi: IVHD Device Entry: type 0x2 id 0xa5 flags 0
>>> (XEN) AMD-Vi: IVHD Device Entry: type 0x3 id 0xb0 flags 0
>>> (XEN) AMD-Vi:  Dev_Id Range: 0xb0 -> 0xb2
>>> (XEN) AMD-Vi: IVHD Device Entry: type 0x48 id 0 flags 0xd7
>>> (XEN) AMD-Vi: IVHD Special: 0000:00:14.0 variety 0x1 handle 0
>>> (XEN) IVHD Error: Invalid IO-APIC 0
>>> (XEN) AMD-Vi: Error initialization
>>> (XEN) I/O virtualisation disabled
>>> (XEN) ENABLING IO-APIC IRQs
>>> (XEN)  -> Using new ACK method
>> Ok - my gut feeling is that this is a bug with the new command line
>> override code.  There is certainly not enough debug printing to work out
>> exactly what is going on, and at a glance of the other information in
>> the log, the state looks reasonable.  (cc'ing Jan and Suravee as the
>> authors)
>>
>> Could you try running with this patch (compile tested only) and seeing
>> what it says?
>>
>> ~Andrew
>>
>> 8<------
>> diff --git a/xen/drivers/passthrough/amd/iommu_acpi.c
>> b/xen/drivers/passthrough/amd/iommu_acpi.c
>> index fca2037..66755fa 100644
>> --- a/xen/drivers/passthrough/amd/iommu_acpi.c
>> +++ b/xen/drivers/passthrough/amd/iommu_acpi.c
>> @@ -25,6 +25,7 @@
>>   #include <asm/io_apic.h>
>>   #include <asm/amd-iommu.h>
>>   #include <asm/hvm/svm/amd-iommu-proto.h>
>> +#include <xen/keyhandler.h>
>>
>>   /* Some helper structures, particularly to deal with ranges. */
>>
>> @@ -785,6 +786,11 @@ static u16 __init parse_ivhd_device_special(
>>           {
>>               printk(XENLOG_ERR "IVHD Error: Invalid IO-APIC %#x\n",
>>                      special->handle);
>> +            bitmap_scnprintf(keyhandler_scratch,
>> sizeof(keyhandler_scratch),
>> +                             ioapic_cmdline,
>> +                             BITS_TO_LONGS(ARRAY_SIZE(ioapic_sbdf)) );
>> +            printk("**DEBUG: apic = %d, nr_ioapics = %d, ioapic_cmdline
>> = %s\n",
>> +                   apic, nr_ioapics, keyhandler_scratch);
>>               return 0;
>>           }
>>           break;
>>
>>
>>
> I applied your patch, this is the related output:
>
> (XEN) AMD-Vi: Found MSI capability block at 0x54
> (XEN) AMD-Vi: ACPI Table:
> (XEN) AMD-Vi:  Signature IVRS
> (XEN) AMD-Vi:  Length 0xd8
> (XEN) AMD-Vi:  Revision 0x1
> (XEN) AMD-Vi:  CheckSum 0x22
> (XEN) AMD-Vi:  OEM_Id AMD
> (XEN) AMD-Vi:  OEM_Table_Id RD890S
> (XEN) AMD-Vi:  OEM_Revision 0x202031
> (XEN) AMD-Vi:  Creator_Id AMD
> (XEN) AMD-Vi:  Creator_Revision 0
> (XEN) AMD-Vi: IVRS Block: type 0x10 flags 0x3e len 0xa8 id 0x2
> (XEN) AMD-Vi: IVHD Device Entry: type 0x3 id 0 flags 0
> (XEN) AMD-Vi:  Dev_Id Range: 0 -> 0x2
> (XEN) AMD-Vi: IVHD Device Entry: type 0x2 id 0x10 flags 0
> (XEN) AMD-Vi: IVHD Device Entry: type 0x3 id 0x100 flags 0
> (XEN) AMD-Vi:  Dev_Id Range: 0x100 -> 0x101
> (XEN) AMD-Vi: IVHD Device Entry: type 0x2 id 0x20 flags 0
> (XEN) AMD-Vi: IVHD Device Entry: type 0x2 id 0x200 flags 0
> (XEN) AMD-Vi: IVHD Device Entry: type 0x2 id 0x28 flags 0
> (XEN) AMD-Vi: IVHD Device Entry: type 0x2 id 0x300 flags 0
> (XEN) AMD-Vi: IVHD Device Entry: type 0x2 id 0x30 flags 0
> (XEN) AMD-Vi: IVHD Device Entry: type 0x2 id 0x400 flags 0
> (XEN) AMD-Vi: IVHD Device Entry: type 0x2 id 0x38 flags 0
> (XEN) AMD-Vi: IVHD Device Entry: type 0x2 id 0x500 flags 0
> (XEN) AMD-Vi: IVHD Device Entry: type 0x2 id 0x50 flags 0
> (XEN) AMD-Vi: IVHD Device Entry: type 0x2 id 0x600 flags 0
> (XEN) AMD-Vi: IVHD Device Entry: type 0x2 id 0x88 flags 0
> (XEN) AMD-Vi: IVHD Device Entry: type 0x3 id 0x90 flags 0
> (XEN) AMD-Vi:  Dev_Id Range: 0x90 -> 0x92
> (XEN) AMD-Vi: IVHD Device Entry: type 0x3 id 0x98 flags 0
> (XEN) AMD-Vi:  Dev_Id Range: 0x98 -> 0x9a
> (XEN) AMD-Vi: IVHD Device Entry: type 0x2 id 0xa0 flags 0xd7
> (XEN) AMD-Vi: IVHD Device Entry: type 0x2 id 0xa2 flags 0
> (XEN) AMD-Vi: IVHD Device Entry: type 0x2 id 0xa3 flags 0
> (XEN) AMD-Vi: IVHD Device Entry: type 0x2 id 0xa4 flags 0
> (XEN) AMD-Vi: IVHD Device Entry: type 0x43 id 0x700 flags 0
> (XEN) AMD-Vi:  Dev_Id Range: 0x700 -> 0x7ff alias 0xa4
> (XEN) AMD-Vi: IVHD Device Entry: type 0x2 id 0xa5 flags 0
> (XEN) AMD-Vi: IVHD Device Entry: type 0x3 id 0xb0 flags 0
> (XEN) AMD-Vi:  Dev_Id Range: 0xb0 -> 0xb2
> (XEN) AMD-Vi: IVHD Device Entry: type 0x48 id 0 flags 0xd7
> (XEN) AMD-Vi: IVHD Special: 0000:00:14.0 variety 0x1 handle 0
> (XEN) IVHD Error: Invalid IO-APIC 0
> (XEN) **DEBUG: apic = 2, nr_ioapics = 2, ioapic_cmdline = 0
> (XEN) AMD-Vi: Error initialization
> (XEN) I/O virtualisation disabled
> (XEN) ENABLING IO-APIC IRQs
> (XEN)  -> Using new ACK method
>
> kind regards
>
Could you pleasealso provide the output from "acpidump" utility or from
/sys/firmware/acpi/tables/IVRS
/sys/firmware/acpi/tables/APIC
/sys/firmware/acpi/tables/HPET

What optiondid you specify for xen.gz?
Also, are you running from the latest code? Jan and I have been working 
on the patch and just recently add more changes.

Thank you,

Suravee

[-- Attachment #1.2: Type: text/html, Size: 10384 bytes --]

[-- Attachment #2: Type: text/plain, Size: 126 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel