Re: Xen 4.3/AMD: setup ssss:bb:dd.f for d0 failed (-ENODEV)

[Suravee Suthikulanit <suravee.suthikulpanit@amd.com>]

On 8/6/2013 1:55 AM, Jan Beulich wrote:
>>>> On 06.08.13 at 04:02, Suravee Suthikulanit <suravee.suthikulpanit@amd.com>
> wrote:
>> On 8/5/2013 7:58 AM, Jan Beulich wrote:
>>>>>> On 22.07.13 at 14:55, Stefan Bader <stefan.bader@canonical.com> wrote:
>>>> While testing Xen 4.3 on my AMD testbox I noticed the following output from
>>>> the
>>>> hypervisor (this has no visible effect on at least simple operation):
>>>>
>>>> (XEN) setup 0000:00:18.0 for d0 failed (-19)
>>>> (XEN) setup 0000:00:18.1 for d0 failed (-19)
>>>> (XEN) setup 0000:00:18.2 for d0 failed (-19)
>>>> (XEN) setup 0000:00:18.3 for d0 failed (-19)
>>>> (XEN) setup 0000:00:18.4 for d0 failed (-19)
>>>> (XEN) setup 0000:00:19.0 for d0 failed (-19)
>>>> (XEN) setup 0000:00:19.1 for d0 failed (-19)
>>>> (XEN) setup 0000:00:19.2 for d0 failed (-19)
>>>> (XEN) setup 0000:00:19.3 for d0 failed (-19)
>>>> (XEN) setup 0000:00:19.4 for d0 failed (-19)
>>>>
>>>> The PCI devices related to the output are all PCI host bridges:
>>>>
>>>> 00:18.0 Host bridge: Advanced Micro Devices, Inc. [AMD]
>>>>    Family 10h Processor HyperTransport Configuration
>>>> 00:18.1 Host bridge: Advanced Micro Devices, Inc. [AMD]
>>>>    Family 10h Processor Address Map
>>>> 00:18.2 Host bridge: Advanced Micro Devices, Inc. [AMD]
>>>>    Family 10h Processor DRAM Controller
>>>> 00:18.3 Host bridge: Advanced Micro Devices, Inc. [AMD]
>>>>    Family 10h Processor Miscellaneous Control
>>>> 00:18.4 Host bridge: Advanced Micro Devices, Inc. [AMD]
>>>>    Family 10h Processor Link Control
>>>> 00:19.0 Host bridge: Advanced Micro Devices, Inc. [AMD]
>>>>    Family 10h Processor HyperTransport Configuration
>>>> 00:19.1 Host bridge: Advanced Micro Devices, Inc. [AMD]
>>>>    Family 10h Processor Address Map
>>>> 00:19.2 Host bridge: Advanced Micro Devices, Inc. [AMD]
>>>>    Family 10h Processor DRAM Controller
>>>> 00:19.3 Host bridge: Advanced Micro Devices, Inc. [AMD]
>>>>    Family 10h Processor Miscellaneous Control
>>>> 00:19.4 Host bridge: Advanced Micro Devices, Inc. [AMD]
>>>>    Family 10h Processor Link Control
>>>>
>>>> This all seems to be related to PCI passthrough setup started from
>>>> xen/drivers/passthrough/vtd/iommu.c:intel_iommu_dom0_init() or
>>>> xen/drivers/passthrough/amd/pci_amd_iommu.c:amd_iommu_dom0_init()
>>>>
>>>> While the Intel code skips over bridge type entries, the AMD code has no
>>>> such
>>>> exception and will fail the handler with -ENODEV when find_iommu_for_device
>>>> fails. But I am not sure one can just compare implementations here.
>>>>
>>>> Would someone have more insight to decide whether skipping host bridges in
>>>> amd_iommu_setup_dom0_device would make sense? I do not see any bad effects
>>>> caused by this but it does not look good and did not happen before.
>>> Fundamentally this looks like a BIOS bug, and hence doing as you
>>> suggest unconditionally is not an option imo.
>> Actually, I don't think this is a BIOS bug.  I have looked through the IVRS
>> table
>> on the system that I have and it looks correct to me in the sense that it
>> does not list
>> all these processor host bridge devices for IOMMU.  As Stefan mentioned, the
>> current
>> dom0 PCI setup code scans through PCI bus does not have filter for the host
>> bridges which
>> do not need IOMMU. However, these error message should not be harmful.
>> Also, in debug mode,
>> it clearly state that "No iommu for device 0000:00:18.0".
> So are you saying that you also see those messages in debug mode?

The default (stock unstable) is showing "(XEN) setup 0000:00:18.0 for d0 failed (-19)".
With the "amd-iommu-debug" boot options, it is also showing "(XEN) AMD-Vi: No iommu for device 0000:00:18.0".

> If so, i.e. if they are expected to always appear, we should indeed
> try to avoid issuing them, as they may cause unnecessary worries
> (as is the case for Stefan here).
Agree
>
>>> Adding code to skip host bridges (under the assumption that they won't
>> initiate
>>> transactions that might require translation by the IOMMU) may be
>>> an option if enabled only via command line option.
>> Currently, pdev_type() in driver/passthrough/pci.c only match the
>> PCI_CLASS_BRIDGE_PCI (0x0604)
>> and not the Host/PCI bridge (0x600) which I see what the devices 0:18.x,
>> 0x19.x are reporting in
>> the PCI class code structure. This might need to change?
> We may want to change this, but this needs to be done carefully
> to not cause regressions on the VT-d side (to which host bridges
> currently are the same as ordinary PCI devices/PCIe endpoints).
Which class code are showing on Intel system for host bridges?

Suravee
>
> Jan
>