Re: [PATCH 0/4] x86/IOMMU: multi-vector MSI prerequisites

[suravee suthikulpanit <suravee.suthikulpanit@amd.com>]

[-- Attachment #1: Type: text/plain, Size: 3332 bytes --]

On 04/11/2013 02:13 AM, Jan Beulich wrote:
>>>> On 11.04.13 at 03:51, Suravee Suthikulpanit <suravee.suthikulpanit@amd.com> wrote:
>> - From drivers/passthrough/amd/iommu_init.c: enable_iommu(), the
>> enabling code tries to set MSI affinity for IOMMU.
>> - From xen/arch/x86/msi.c: set_msi_affinity(), the code tries to
>> "read_msi_msg()" which eventually calls "amd_iommu_read_msi_from_ire()".
>> - Please note that the "amd_iommu_read_msi_from_ire()" was originally
>> empty prior the patch.
>> - From the patch, the new code try to call "get_intremap_entry()", which
>> is the function that was asserting due to table is NULL.
> But even before the patch update_intremap_entry_from_msi_msg()
> also necessarily called get_intremap_entry() for the IOMMU device
> itself. And the respective write_msi_msg() call follows right after the
> read_msi_msg() one, i.e. there's no room for other setup to be done
> by that point. Plus the assertion was there already.
>
>> Here, the MSI read message was sent to read IOMMU registers.  Since the
>> message is targeting for IOMMU itself (bdf 0x2 in this case), it should
>> not be remapped.  That is why I believe the interrupt remapping table is
>> empty.
> Whether or not the MSI for the IOMMU device itself needs to be
> remapped is an independent question, but judging from the
> current code as well as from the specification not saying otherwise
> it needs to be. Confirmation on this would surely be appreciated
> (because otherwise (a) I'd need to understand how this special
> case works with the code before the patch and (b) new special
> casing may need to be added).
>
> In any case - I think I said this before already - seeing a full log up
> to the crash with "iommu=debug" in place might help. After all the
> ACPI tables ought to mention the IOMMU PCI device in some way,
> and hence there ought to be an IVRS mapping as well as an
> associated intremap table. Or else the only way this can work
> currently would be by way of find_iommu_for_device() returning
> NULL for the IOMMU PCI device (which would mean a similar check
> would need to be added to amd_iommu_read_msi_from_ire(), but
> it would then be bogus for the failure there to cause a log message
> to be issued if iommu_debug - that should be trivial to verify on
> an unpatched or older [say 4.2.x based] hypervisor passing
> "iommu=debug").
>
> Jan
>
I have attached the xl dmesg output from the machine which run the 
non-patched Xen (w/ iommu=debug). It is showing that there is no IVHD 
entry setting up for the bdf 0x2 (the IOMMU).  Also, there is a line 
"AMD-Vi: Fail to find iommu for MSI device id = 0x2) which can be traced 
to xen/drivers/passthrough/amd/iommu_intr.c: 
amd_iommu_msi_msg_update_ire() which is called from xen/arch/x86/msi.c: 
write_msi_msg().

When comparing with other AMD systems in the past (i.e. with SR56XX 
chipset), the IVHD contains IVHD entries such as below:
(XEN) AMD-Vi: IVHD Device Entry: type 0x3 id 0 flags 0
(XEN) AMD-Vi:  Dev_Id Range: 0 -> 0x2

which will generate the mapping entry for bfd 0x0, 0x1, 0x2.

Suravee
PS: I could not get the serial console out from the trouble target 
system (Trinity based), so I could not get the crash message.  But from 
the prior email I sent to first report the issue, you can see that it 
failed during "enable_iommu".


[-- Attachment #2: tn.xldmesg --]
[-- Type: text/plain, Size: 14574 bytes --]

 __  __            _  _    _____                    _        _     _      
 \ \/ /___ _ __   | || |  |___ /    _   _ _ __  ___| |_ __ _| |__ | | ___ 
  \  // _ \ '_ \  | || |_   |_ \ __| | | | '_ \/ __| __/ _` | '_ \| |/ _ \
  /  \  __/ | | | |__   _| ___) |__| |_| | | | \__ \ || (_| | |_) | |  __/
 /_/\_\___|_| |_|    |_|(_)____/    \__,_|_| |_|___/\__\__,_|_.__/|_|\___|
                                                                          
(XEN) Xen version 4.3-unstable (root@) (gcc (Ubuntu/Linaro 4.7.2-2ubuntu1) 4.7.2) debug=y Wed Apr 10 19:41:18 CDT 2013
(XEN) Latest ChangeSet: unavailable
(XEN) Bootloader: GRUB 2.00-7ubuntu11
(XEN) Command line: placeholder dom0_max_vcpus=2 dom0_vcpus_pin dom0_mem=4G,max:4G vpmu iommu=debug
(XEN) Video information:
(XEN)  VGA is text mode 80x25, font 8x16
(XEN)  VBE/DDC methods: V2; EDID transfer time: 1 seconds
(XEN) Disc information:
(XEN)  Found 1 MBR signatures
(XEN)  Found 1 EDD information structures
(XEN) Xen-e820 RAM map:
(XEN)  0000000000000000 - 000000000009c000 (usable)
(XEN)  000000000009c000 - 00000000000a0000 (reserved)
(XEN)  00000000000e0000 - 0000000000100000 (reserved)
(XEN)  0000000000100000 - 00000000cea84000 (usable)
(XEN)  00000000cea84000 - 00000000ced5e000 (reserved)
(XEN)  00000000ced5e000 - 00000000cf0ae000 (ACPI NVS)
(XEN)  00000000cf0ae000 - 00000000cf5e4000 (reserved)
(XEN)  00000000cf5e4000 - 00000000cf5e5000 (usable)
(XEN)  00000000cf5e5000 - 00000000cf7eb000 (ACPI NVS)
(XEN)  00000000cf7eb000 - 00000000cfaa3000 (usable)
(XEN)  00000000cfaa3000 - 00000000cfdf6000 (reserved)
(XEN)  00000000cfdf6000 - 00000000cfe00000 (usable)
(XEN)  00000000fec00000 - 00000000fec01000 (reserved)
(XEN)  00000000fec10000 - 00000000fec11000 (reserved)
(XEN)  00000000fed00000 - 00000000fed01000 (reserved)
(XEN)  00000000fed40000 - 00000000fed45000 (reserved)
(XEN)  00000000fed80000 - 00000000fed90000 (reserved)
(XEN)  00000000ff000000 - 0000000100000000 (reserved)
(XEN)  0000000100001000 - 000000042f000000 (usable)
(XEN) ACPI: RSDP 000F0490, 0024 (r2 ALASKA)
(XEN) ACPI: XSDT CF09B078, 0074 (r1 ALASKA    A M I  1072009 AMI     10013)
(XEN) ACPI: FACP CF0A2A20, 010C (r5 ALASKA    A M I  1072009 AMI     10013)
(XEN) ACPI Warning (tbfadt-0464): Optional field "Pm2ControlBlock" has zero address or length: 0000000000000000/1 [20070126]
(XEN) ACPI: DSDT CF09B188, 7897 (r2 ALASKA    A M I        0 INTL 20051117)
(XEN) ACPI: FACS CF0AC080, 0040
(XEN) ACPI: APIC CF0A2B30, 0072 (r3 ALASKA    A M I  1072009 AMI     10013)
(XEN) ACPI: FPDT CF0A2BA8, 0044 (r1 ALASKA    A M I  1072009 AMI     10013)
(XEN) ACPI: MCFG CF0A2BF0, 003C (r1 ALASKA    A M I  1072009 MSFT    10013)
(XEN) ACPI: HPET CF0A2C30, 0038 (r1 ALASKA    A M I  1072009 AMI         5)
(XEN) ACPI: SPCR CF0A2C68, 0050 (r1  A M I   APTIO4  1072009 AMI.        5)
(XEN) ACPI: SSDT CF0A2CB8, 0D40 (r1 AMD    POWERNOW        1 AMD         1)
(XEN) ACPI: SSDT CF0A39F8, 04B7 (r2    AMD     ALIB        1 MSFT  4000000)
(XEN) ACPI: IVRS CF0A3EB0, 0070 (r2  AMD   AMDIOMMU        1 AMD         0)
(XEN) ACPI: CRAT CF0A3F20, 0398 (r1 AMD    AGESA           1 AMD         1)
(XEN) System RAM: 16348MB (16741256kB)
(XEN) No NUMA configuration found
(XEN) Faking a node at 0000000000000000-000000042f000000
(XEN) Domain heap initialised
(XEN) found SMP MP-table at 000fd7e0
(XEN) DMI 2.7 present.
(XEN) Using APIC driver default
(XEN) ACPI: PM-Timer IO Port: 0x808
(XEN) ACPI: v5 SLEEP INFO: control[0:0], status[0:0]
(XEN) ACPI: SLEEP INFO: pm1x_cnt[804,0], pm1x_evt[800,0]
(XEN) ACPI: 32/64X FACS address mismatch in FADT - cf0ac080/0000000000000000, using 32
(XEN) ACPI:             wakeup_vec[cf0ac08c], vec_size[20]
(XEN) ACPI: Local APIC address 0xfee00000
(XEN) ACPI: LAPIC (acpi_id[0x01] lapic_id[0x10] enabled)
(XEN) Processor #16 5:0 APIC version 16
(XEN) ACPI: LAPIC (acpi_id[0x02] lapic_id[0x11] enabled)
(XEN) Processor #17 5:0 APIC version 16
(XEN) ACPI: LAPIC (acpi_id[0x03] lapic_id[0x12] enabled)
(XEN) Processor #18 5:0 APIC version 16
(XEN) ACPI: LAPIC (acpi_id[0x04] lapic_id[0x13] enabled)
(XEN) Processor #19 5:0 APIC version 16
(XEN) ACPI: LAPIC_NMI (acpi_id[0xff] high edge lint[0x1])
(XEN) ACPI: IOAPIC (id[0x05] address[0xfec00000] gsi_base[0])
(XEN) IOAPIC[0]: apic_id 5, version 33, address 0xfec00000, GSI 0-23
(XEN) ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl)
(XEN) ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 low level)
(XEN) ACPI: IRQ0 used by override.
(XEN) ACPI: IRQ2 used by override.
(XEN) ACPI: IRQ9 used by override.
(XEN) Enabling APIC mode:  Flat.  Using 1 I/O APICs
(XEN) ACPI: HPET id: 0x10228210 base: 0xfed00000
(XEN) ERST table was not found
(XEN) Using ACPI (MADT) for SMP configuration information
(XEN) SMP: Allowing 4 CPUs (0 hotplug CPUs)
(XEN) IRQ limits: 24 GSI, 760 MSI/MSI-X
(XEN) Using scheduler: SMP Credit Scheduler (credit)
(XEN) Detected 3293.935 MHz processor.
(XEN) Initing memory sharing.
(XEN) xstate_init: using cntxt_size: 0x3c0 and states: 0x4000000000000007
(XEN) AMD Fam15h machine check reporting enabled
(XEN) PCI: MCFG configuration 0: base e0000000 segment 0000 buses 00 - ff
(XEN) PCI: Not using MCFG for segment 0000 bus 00-ff
(XEN) AMD-Vi: Found MSI capability block at 0x54
(XEN) AMD-Vi: ACPI Table:
(XEN) AMD-Vi:  Signature IVRS
(XEN) AMD-Vi:  Length 0x70
(XEN) AMD-Vi:  Revision 0x2
(XEN) AMD-Vi:  CheckSum 0xb2
(XEN) AMD-Vi:  OEM_Id AMD  
(XEN) AMD-Vi:  OEM_Table_Id AMDIOMMU
(XEN) AMD-Vi:  OEM_Revision 0x1
(XEN) AMD-Vi:  Creator_Id AMD 
(XEN) AMD-Vi:  Creator_Revision 0
(XEN) AMD-Vi: IVRS Block: type 0x10 flags 0xfe len 0x40 id 0x2
(XEN) AMD-Vi: IVHD Device Entry: type 0x3 id 0x8 flags 0
(XEN) AMD-Vi:  Dev_Id Range: 0x8 -> 0xfffe
(XEN) AMD-Vi: IVHD Device Entry: type 0x43 id 0x200 flags 0
(XEN) AMD-Vi:  Dev_Id Range: 0x200 -> 0x2ff alias 0xa4
(XEN) AMD-Vi: IVHD Device Entry: type 0 id 0 flags 0
(XEN) AMD-Vi: IVHD Device Entry: type 0x48 id 0 flags 0
(XEN) AMD-Vi: IVHD Special: 0000:00:14.0 variety 0x2 handle 0
(XEN) AMD-Vi: IVHD Device Entry: type 0x48 id 0 flags 0xd7
(XEN) AMD-Vi: IVHD Special: 0000:00:14.0 variety 0x1 handle 0x5
(XEN) AMD-Vi: IOMMU Extended Features:
(XEN)  - Prefetch Pages Command
(XEN)  - Peripheral Page Service Request
(XEN)  - Guest Translation
(XEN)  - Invalidate All Command
(XEN) AMD-Vi: Fail to find iommu for MSI device id = 0x2
(XEN) AMD-Vi: PPR Log Enabled.
(XEN) AMD-Vi: Guest Translation Enabled.
(XEN) AMD-Vi: IOMMU 0 Enabled.
(XEN) AMD-Vi: Enabling per-device vector maps
(XEN) I/O virtualisation enabled
(XEN)  - Dom0 mode: Relaxed
(XEN) Interrupt remapping enabled
(XEN) ENABLING IO-APIC IRQs
(XEN)  -> Using new ACK method
(XEN) ..TIMER: vector=0xF0 apic1=0 pin1=2 apic2=-1 pin2=-1
(XEN) Platform timer is 14.318MHz HPET
(XEN) Allocated console ring of 32 KiB.
(XEN) HVM: ASIDs enabled.
(XEN) SVM: Supported advanced features:
(XEN)  - Nested Page Tables (NPT)
(XEN)  - Last Branch Record (LBR) Virtualisation
(XEN)  - Next-RIP Saved on #VMEXIT
(XEN)  - VMCB Clean Bits
(XEN)  - DecodeAssists
(XEN)  - Pause-Intercept Filter
(XEN)  - TSC Rate MSR
(XEN) HVM: SVM enabled
(XEN) HVM: Hardware Assisted Paging (HAP) detected
(XEN) HVM: HAP page sizes: 4kB, 2MB, 1GB
(XEN) microcode: CPU1 collect_cpu_info: patch_id=0x600100e
(XEN) microcode: CPU2 collect_cpu_info: patch_id=0x600100e
(XEN) microcode: CPU3 collect_cpu_info: patch_id=0x600100e
(XEN) Brought up 4 CPUs
(XEN) ACPI sleep modes: S3
(XEN) MCA: Use hw thresholding to adjust polling frequency
(XEN) mcheck_poll: Machine check polling timer started.
(XEN) *** LOADING DOMAIN 0 ***
(XEN) elf_parse_binary: phdr: paddr=0x1000000 memsz=0xae7000
(XEN) elf_parse_binary: phdr: paddr=0x1c00000 memsz=0xdd0e8
(XEN) elf_parse_binary: phdr: paddr=0x1cde000 memsz=0x14680
(XEN) elf_parse_binary: phdr: paddr=0x1cf3000 memsz=0x65c000
(XEN) elf_parse_binary: memory: 0x1000000 -> 0x234f000
(XEN) elf_xen_parse_note: GUEST_OS = "linux"
(XEN) elf_xen_parse_note: GUEST_VERSION = "2.6"
(XEN) elf_xen_parse_note: XEN_VERSION = "xen-3.0"
(XEN) elf_xen_parse_note: VIRT_BASE = 0xffffffff80000000
(XEN) elf_xen_parse_note: ENTRY = 0xffffffff81cf3210
(XEN) elf_xen_parse_note: HYPERCALL_PAGE = 0xffffffff81001000
(XEN) elf_xen_parse_note: FEATURES = "!writable_page_tables|pae_pgdir_above_4gb"
(XEN) elf_xen_parse_note: PAE_MODE = "yes"
(XEN) elf_xen_parse_note: LOADER = "generic"
(XEN) elf_xen_parse_note: unknown xen elf note (0xd)
(XEN) elf_xen_parse_note: SUSPEND_CANCEL = 0x1
(XEN) elf_xen_parse_note: HV_START_LOW = 0xffff800000000000
(XEN) elf_xen_parse_note: PADDR_OFFSET = 0x0
(XEN) elf_xen_addr_calc_check: addresses:
(XEN)     virt_base        = 0xffffffff80000000
(XEN)     elf_paddr_offset = 0x0
(XEN)     virt_offset      = 0xffffffff80000000
(XEN)     virt_kstart      = 0xffffffff81000000
(XEN)     virt_kend        = 0xffffffff8234f000
(XEN)     virt_entry       = 0xffffffff81cf3210
(XEN)     p2m_base         = 0xffffffffffffffff
(XEN)  Xen  kernel: 64-bit, lsb, compat32
(XEN)  Dom0 kernel: 64-bit, PAE, lsb, paddr 0x1000000 -> 0x234f000
(XEN) PHYSICAL MEMORY ARRANGEMENT:
(XEN)  Dom0 alloc.:   0000000238000000->000000023c000000 (1022084 pages to be allocated)
(XEN)  Init. ramdisk: 000000042c884000->000000042efffc00
(XEN) VIRTUAL MEMORY ARRANGEMENT:
(XEN)  Loaded kernel: ffffffff81000000->ffffffff8234f000
(XEN)  Init. ramdisk: ffffffff8234f000->ffffffff84acac00
(XEN)  Phys-Mach map: ffffffff84acb000->ffffffff852cb000
(XEN)  Start info:    ffffffff852cb000->ffffffff852cb4b4
(XEN)  Page tables:   ffffffff852cc000->ffffffff852f9000
(XEN)  Boot stack:    ffffffff852f9000->ffffffff852fa000
(XEN)  TOTAL:         ffffffff80000000->ffffffff85400000
(XEN)  ENTRY ADDRESS: ffffffff81cf3210
(XEN) Dom0 has maximum 2 VCPUs
(XEN) elf_load_binary: phdr 0 at 0xffffffff81000000 -> 0xffffffff81ae7000
(XEN) elf_load_binary: phdr 1 at 0xffffffff81c00000 -> 0xffffffff81cdd0e8
(XEN) elf_load_binary: phdr 2 at 0xffffffff81cde000 -> 0xffffffff81cf2680
(XEN) elf_load_binary: phdr 3 at 0xffffffff81cf3000 -> 0xffffffff81dcd000
(XEN) AMD-Vi: No iommu for device 0000:00:00.0
(XEN) setup 0000:00:00.0 for d0 failed (-19)
(XEN) AMD-Vi: No iommu for device 0000:00:00.2
(XEN) setup 0000:00:00.2 for d0 failed (-19)
(XEN) AMD-Vi: Setup I/O page table: device id = 0x10, root table = 0x23f7a7000, domain = 0, paging mode = 3
(XEN) AMD-Vi: Setup I/O page table: device id = 0x80, root table = 0x23f7a7000, domain = 0, paging mode = 3
(XEN) AMD-Vi: Setup I/O page table: device id = 0x81, root table = 0x23f7a7000, domain = 0, paging mode = 3
(XEN) AMD-Vi: Setup I/O page table: device id = 0x88, root table = 0x23f7a7000, domain = 0, paging mode = 3
(XEN) AMD-Vi: Setup I/O page table: device id = 0x90, root table = 0x23f7a7000, domain = 0, paging mode = 3
(XEN) AMD-Vi: Setup I/O page table: device id = 0x92, root table = 0x23f7a7000, domain = 0, paging mode = 3
(XEN) AMD-Vi: Setup I/O page table: device id = 0x98, root table = 0x23f7a7000, domain = 0, paging mode = 3
(XEN) AMD-Vi: Setup I/O page table: device id = 0x9a, root table = 0x23f7a7000, domain = 0, paging mode = 3
(XEN) AMD-Vi: Setup I/O page table: device id = 0xa0, root table = 0x23f7a7000, domain = 0, paging mode = 3
(XEN) AMD-Vi: Setup I/O page table: device id = 0xa2, root table = 0x23f7a7000, domain = 0, paging mode = 3
(XEN) AMD-Vi: Setup I/O page table: device id = 0xa3, root table = 0x23f7a7000, domain = 0, paging mode = 3
(XEN) AMD-Vi: Setup I/O page table: device id = 0xa4, root table = 0x23f7a7000, domain = 0, paging mode = 3
(XEN) AMD-Vi: Setup I/O page table: device id = 0xa5, root table = 0x23f7a7000, domain = 0, paging mode = 3
(XEN) AMD-Vi: Setup I/O page table: device id = 0xa7, root table = 0x23f7a7000, domain = 0, paging mode = 3
(XEN) AMD-Vi: Setup I/O page table: device id = 0xa8, root table = 0x23f7a7000, domain = 0, paging mode = 3
(XEN) AMD-Vi: Setup I/O page table: device id = 0xa9, root table = 0x23f7a7000, domain = 0, paging mode = 3
(XEN) AMD-Vi: Setup I/O page table: device id = 0xc0, root table = 0x23f7a7000, domain = 0, paging mode = 3
(XEN) AMD-Vi: Setup I/O page table: device id = 0xc1, root table = 0x23f7a7000, domain = 0, paging mode = 3
(XEN) AMD-Vi: Setup I/O page table: device id = 0xc2, root table = 0x23f7a7000, domain = 0, paging mode = 3
(XEN) AMD-Vi: Setup I/O page table: device id = 0xc3, root table = 0x23f7a7000, domain = 0, paging mode = 3
(XEN) AMD-Vi: Setup I/O page table: device id = 0xc4, root table = 0x23f7a7000, domain = 0, paging mode = 3
(XEN) AMD-Vi: Setup I/O page table: device id = 0xc5, root table = 0x23f7a7000, domain = 0, paging mode = 3
(XEN) AMD-Vi: Setup I/O page table: device id = 0x100, root table = 0x23f7a7000, domain = 0, paging mode = 3
(XEN) AMD-Vi: Setup I/O page table: device id = 0x101, root table = 0x23f7a7000, domain = 0, paging mode = 3
(XEN) AMD-Vi: Setup I/O page table: device id = 0x228, root table = 0x23f7a7000, domain = 0, paging mode = 3
(XEN) AMD-Vi: Setup I/O page table: device id = 0x229, root table = 0x23f7a7000, domain = 0, paging mode = 3
(XEN) AMD-Vi: Setup I/O page table: device id = 0x400, root table = 0x23f7a7000, domain = 0, paging mode = 3
(XEN) Scrubbing Free RAM: ....................................................................................................................done.
(XEN) Initial low memory virq threshold set at 0x4000 pages.
(XEN) Std. Loglevel: All
(XEN) Guest Loglevel: All
(XEN) Xen is relinquishing VGA console.
(XEN) *** Serial input -> DOM0 (type 'CTRL-a' three times to switch input to Xen)
(XEN) Freed 248kB init memory.
(XEN) traps.c:2495:d0 Domain attempted WRMSR 00000000c0010201 from 0x0000000000000000 to 0x000000000000abcd.
(XEN) PCI add device 0000:00:00.0
(XEN) PCI add device 0000:00:00.2
(XEN) PCI add device 0000:00:02.0
(XEN) PCI add device 0000:00:10.0
(XEN) PCI add device 0000:00:10.1
(XEN) PCI add device 0000:00:11.0
(XEN) PCI add device 0000:00:12.0
(XEN) PCI add device 0000:00:12.2
(XEN) PCI add device 0000:00:13.0
(XEN) PCI add device 0000:00:13.2
(XEN) PCI add device 0000:00:14.0
(XEN) PCI add device 0000:00:14.2
(XEN) PCI add device 0000:00:14.3
(XEN) PCI add device 0000:00:14.4
(XEN) PCI add device 0000:00:14.5
(XEN) PCI add device 0000:00:14.7
(XEN) PCI add device 0000:00:15.0
(XEN) PCI add device 0000:00:15.1
(XEN) PCI add device 0000:00:18.0
(XEN) PCI add device 0000:00:18.1
(XEN) PCI add device 0000:00:18.2
(XEN) PCI add device 0000:00:18.3
(XEN) PCI add device 0000:00:18.4
(XEN) PCI add device 0000:00:18.5
(XEN) PCI add device 0000:01:00.0
(XEN) PCI add device 0000:01:00.1
(XEN) PCI add device 0000:02:05.0
(XEN) PCI add device 0000:02:05.1
(XEN) PCI add device 0000:04:00.0

[-- Attachment #3: Type: text/plain, Size: 126 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel