Re: Xen 4.3/AMD: setup ssss:bb:dd.f for d0 failed (-ENODEV)

[Suravee Suthikulanit <suravee.suthikulpanit@amd.com>]

On 8/5/2013 7:58 AM, Jan Beulich wrote:
>>>> On 22.07.13 at 14:55, Stefan Bader <stefan.bader@canonical.com> wrote:
>> While testing Xen 4.3 on my AMD testbox I noticed the following output from
>> the
>> hypervisor (this has no visible effect on at least simple operation):
>>
>> (XEN) setup 0000:00:18.0 for d0 failed (-19)
>> (XEN) setup 0000:00:18.1 for d0 failed (-19)
>> (XEN) setup 0000:00:18.2 for d0 failed (-19)
>> (XEN) setup 0000:00:18.3 for d0 failed (-19)
>> (XEN) setup 0000:00:18.4 for d0 failed (-19)
>> (XEN) setup 0000:00:19.0 for d0 failed (-19)
>> (XEN) setup 0000:00:19.1 for d0 failed (-19)
>> (XEN) setup 0000:00:19.2 for d0 failed (-19)
>> (XEN) setup 0000:00:19.3 for d0 failed (-19)
>> (XEN) setup 0000:00:19.4 for d0 failed (-19)
>>
>> The PCI devices related to the output are all PCI host bridges:
>>
>> 00:18.0 Host bridge: Advanced Micro Devices, Inc. [AMD]
>>   Family 10h Processor HyperTransport Configuration
>> 00:18.1 Host bridge: Advanced Micro Devices, Inc. [AMD]
>>   Family 10h Processor Address Map
>> 00:18.2 Host bridge: Advanced Micro Devices, Inc. [AMD]
>>   Family 10h Processor DRAM Controller
>> 00:18.3 Host bridge: Advanced Micro Devices, Inc. [AMD]
>>   Family 10h Processor Miscellaneous Control
>> 00:18.4 Host bridge: Advanced Micro Devices, Inc. [AMD]
>>   Family 10h Processor Link Control
>> 00:19.0 Host bridge: Advanced Micro Devices, Inc. [AMD]
>>   Family 10h Processor HyperTransport Configuration
>> 00:19.1 Host bridge: Advanced Micro Devices, Inc. [AMD]
>>   Family 10h Processor Address Map
>> 00:19.2 Host bridge: Advanced Micro Devices, Inc. [AMD]
>>   Family 10h Processor DRAM Controller
>> 00:19.3 Host bridge: Advanced Micro Devices, Inc. [AMD]
>>   Family 10h Processor Miscellaneous Control
>> 00:19.4 Host bridge: Advanced Micro Devices, Inc. [AMD]
>>   Family 10h Processor Link Control
>>
>> This all seems to be related to PCI passthrough setup started from
>> xen/drivers/passthrough/vtd/iommu.c:intel_iommu_dom0_init() or
>> xen/drivers/passthrough/amd/pci_amd_iommu.c:amd_iommu_dom0_init()
>>
>> While the Intel code skips over bridge type entries, the AMD code has no
>> such
>> exception and will fail the handler with -ENODEV when find_iommu_for_device
>> fails. But I am not sure one can just compare implementations here.
>>
>> Would someone have more insight to decide whether skipping host bridges in
>> amd_iommu_setup_dom0_device would make sense? I do not see any bad effects
>> caused by this but it does not look good and did not happen before.
> Fundamentally this looks like a BIOS bug, and hence doing as you
> suggest unconditionally is not an option imo.

Actually, I don't think this is a BIOS bug.  I have looked through the IVRS table
on the system that I have and it looks correct to me in the sense that it does not list
all these processor host bridge devices for IOMMU.  As Stefan mentioned, the current
dom0 PCI setup code scans through PCI bus does not have filter for the host bridges which
do not need IOMMU. However, these error message should not be harmful.  Also, in debug mode,
it clearly state that "No iommu for device 0000:00:18.0".

> Adding code to skip host bridges (under the assumption that they won't initiate
> transactions that might require translation by the IOMMU) may be
> an option if enabled only via command line option.

Currently, pdev_type() in driver/passthrough/pci.c only match the PCI_CLASS_BRIDGE_PCI (0x0604)
and not the Host/PCI bridge (0x600) which I see what the devices 0:18.x, 0x19.x are reporting in
the PCI class code structure. This might need to change?

Suravee