From mboxrd@z Thu Jan 1 00:00:00 1970 From: Suravee Suthikulanit Subject: Re: Xen 4.3/AMD: setup ssss:bb:dd.f for d0 failed (-ENODEV) Date: Tue, 6 Aug 2013 09:11:07 -0500 Message-ID: <520103FB.6020009@amd.com> References: <51ED2BB3.4050806@canonical.com> <51FFBD7802000078000E947A@nat28.tlf.novell.com> <5200591D.2040102@amd.com> <5200B9FB02000078000E9844@nat28.tlf.novell.com> Mime-Version: 1.0 Content-Type: text/plain; charset="us-ascii"; Format="flowed" Content-Transfer-Encoding: 7bit Return-path: Received: from mail6.bemta3.messagelabs.com ([195.245.230.39]) by lists.xen.org with esmtp (Exim 4.72) (envelope-from ) id 1V6hyn-0003cm-TS for xen-devel@lists.xenproject.org; Tue, 06 Aug 2013 14:11:18 +0000 In-Reply-To: <5200B9FB02000078000E9844@nat28.tlf.novell.com> List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Sender: xen-devel-bounces@lists.xen.org Errors-To: xen-devel-bounces@lists.xen.org To: Jan Beulich Cc: xen-devel , Stefan Bader List-Id: xen-devel@lists.xenproject.org On 8/6/2013 1:55 AM, Jan Beulich wrote: >>>> On 06.08.13 at 04:02, Suravee Suthikulanit > wrote: >> On 8/5/2013 7:58 AM, Jan Beulich wrote: >>>>>> On 22.07.13 at 14:55, Stefan Bader wrote: >>>> While testing Xen 4.3 on my AMD testbox I noticed the following output from >>>> the >>>> hypervisor (this has no visible effect on at least simple operation): >>>> >>>> (XEN) setup 0000:00:18.0 for d0 failed (-19) >>>> (XEN) setup 0000:00:18.1 for d0 failed (-19) >>>> (XEN) setup 0000:00:18.2 for d0 failed (-19) >>>> (XEN) setup 0000:00:18.3 for d0 failed (-19) >>>> (XEN) setup 0000:00:18.4 for d0 failed (-19) >>>> (XEN) setup 0000:00:19.0 for d0 failed (-19) >>>> (XEN) setup 0000:00:19.1 for d0 failed (-19) >>>> (XEN) setup 0000:00:19.2 for d0 failed (-19) >>>> (XEN) setup 0000:00:19.3 for d0 failed (-19) >>>> (XEN) setup 0000:00:19.4 for d0 failed (-19) >>>> >>>> The PCI devices related to the output are all PCI host bridges: >>>> >>>> 00:18.0 Host bridge: Advanced Micro Devices, Inc. [AMD] >>>> Family 10h Processor HyperTransport Configuration >>>> 00:18.1 Host bridge: Advanced Micro Devices, Inc. [AMD] >>>> Family 10h Processor Address Map >>>> 00:18.2 Host bridge: Advanced Micro Devices, Inc. [AMD] >>>> Family 10h Processor DRAM Controller >>>> 00:18.3 Host bridge: Advanced Micro Devices, Inc. [AMD] >>>> Family 10h Processor Miscellaneous Control >>>> 00:18.4 Host bridge: Advanced Micro Devices, Inc. [AMD] >>>> Family 10h Processor Link Control >>>> 00:19.0 Host bridge: Advanced Micro Devices, Inc. [AMD] >>>> Family 10h Processor HyperTransport Configuration >>>> 00:19.1 Host bridge: Advanced Micro Devices, Inc. [AMD] >>>> Family 10h Processor Address Map >>>> 00:19.2 Host bridge: Advanced Micro Devices, Inc. [AMD] >>>> Family 10h Processor DRAM Controller >>>> 00:19.3 Host bridge: Advanced Micro Devices, Inc. [AMD] >>>> Family 10h Processor Miscellaneous Control >>>> 00:19.4 Host bridge: Advanced Micro Devices, Inc. [AMD] >>>> Family 10h Processor Link Control >>>> >>>> This all seems to be related to PCI passthrough setup started from >>>> xen/drivers/passthrough/vtd/iommu.c:intel_iommu_dom0_init() or >>>> xen/drivers/passthrough/amd/pci_amd_iommu.c:amd_iommu_dom0_init() >>>> >>>> While the Intel code skips over bridge type entries, the AMD code has no >>>> such >>>> exception and will fail the handler with -ENODEV when find_iommu_for_device >>>> fails. But I am not sure one can just compare implementations here. >>>> >>>> Would someone have more insight to decide whether skipping host bridges in >>>> amd_iommu_setup_dom0_device would make sense? I do not see any bad effects >>>> caused by this but it does not look good and did not happen before. >>> Fundamentally this looks like a BIOS bug, and hence doing as you >>> suggest unconditionally is not an option imo. >> Actually, I don't think this is a BIOS bug. I have looked through the IVRS >> table >> on the system that I have and it looks correct to me in the sense that it >> does not list >> all these processor host bridge devices for IOMMU. As Stefan mentioned, the >> current >> dom0 PCI setup code scans through PCI bus does not have filter for the host >> bridges which >> do not need IOMMU. However, these error message should not be harmful. >> Also, in debug mode, >> it clearly state that "No iommu for device 0000:00:18.0". > So are you saying that you also see those messages in debug mode? The default (stock unstable) is showing "(XEN) setup 0000:00:18.0 for d0 failed (-19)". With the "amd-iommu-debug" boot options, it is also showing "(XEN) AMD-Vi: No iommu for device 0000:00:18.0". > If so, i.e. if they are expected to always appear, we should indeed > try to avoid issuing them, as they may cause unnecessary worries > (as is the case for Stefan here). Agree > >>> Adding code to skip host bridges (under the assumption that they won't >> initiate >>> transactions that might require translation by the IOMMU) may be >>> an option if enabled only via command line option. >> Currently, pdev_type() in driver/passthrough/pci.c only match the >> PCI_CLASS_BRIDGE_PCI (0x0604) >> and not the Host/PCI bridge (0x600) which I see what the devices 0:18.x, >> 0x19.x are reporting in >> the PCI class code structure. This might need to change? > We may want to change this, but this needs to be done carefully > to not cause regressions on the VT-d side (to which host bridges > currently are the same as ordinary PCI devices/PCIe endpoints). Which class code are showing on Intel system for host bridges? Suravee > > Jan >