* Re: [PATCH v3 00/15] PCI/iommu: Fix DMA alias problems
[not found] <400175834.3289.65.camel@ul30vt.home>
@ 2014-05-16 6:28 ` colin
0 siblings, 0 replies; 5+ messages in thread
From: colin @ 2014-05-16 6:28 UTC (permalink / raw)
To: alex.williamson, iommu, linux-pci; +Cc: acooks, bhelgaas, linux-kernel, linux
Alex Williamson wrote:
> Wow, I didn't think that kind of broken was possible. Maybe instead of
> a bitmap of function aliases we could have a single devfn alias for a
> device. That means we'd only be able to support a single alias for a
> device, but since I don't think we've seen devices that use more than a
> single alias, maybe that's ok.
In my (never finished) patch set for the same problem, the first thing
I did was
diff --git a/include/linux/pci.h b/include/linux/pci.h
index a13d6825..7788870a 100644
--- a/include/linux/pci.h
+++ b/include/linux/pci.h
@@ -251,12 +251,13 @@ struct pci_dev {
struct proc_dir_entry *procent; /* device entry in /proc/bus/pci */
struct pci_slot *slot; /* Physical slot this device is in */
- unsigned int devfn; /* encoded device & function index */
unsigned short vendor;
unsigned short device;
unsigned short subsystem_vendor;
unsigned short subsystem_device;
unsigned int class; /* 3 bytes: (base,sub,prog-if) */
+ u8 devfn; /* encoded device & function index */
+ u8 devfn_quirk; /* zero is non-quirky */
u8 revision; /* PCI revision, low byte of class word */
u8 hdr_type; /* PCI header type (`multi' flag masked out) */
u8 pcie_cap; /* PCIe capability offset */
I encoded "devfn_quirk" as a delta to devfn, so that zero would mean
"no quirk", and no existing intialization would need changing.
^ permalink raw reply related [flat|nested] 5+ messages in thread
* [PATCH v3 00/15] PCI/iommu: Fix DMA alias problems
@ 2014-05-10 15:02 Alex Williamson
2014-05-10 16:51 ` George Spelvin
2014-05-13 22:35 ` eddy0596
0 siblings, 2 replies; 5+ messages in thread
From: Alex Williamson @ 2014-05-10 15:02 UTC (permalink / raw)
To: linux-pci, iommu; +Cc: bhelgaas, acooks, linux-kernel, linux
This version can be found in my git tree:
git://github.com/awilliam/linux-vfio.git dma-alias-v3
Please report any issues.
v3:
- Found several instances where I had PCI_SLOT when I meant
PCI_FUNC. Thanks to Andrew for spotting this. This should
fix the problem he was having with Ricoh quirks. We also
pruned down the func0 quirks to only those that we know are
needed. We can always add them back later.
- Found a case in intel-iommu of using dev_is_pci() where I
really wanted !dev_is_pci(). Fixed.
v2:
- Several new Marvell controllers added to quirks. There's been
a lot of success reported with this series in
https://bugzilla.kernel.org/show_bug.cgi?id=42679
- Add quirk for ASMedia and Tundra PCIe-to-PCI bridges that do
not expose a PCIe capability. These have been shown to use
the standard PCIe-to-PCI bridge requester ID.
- Fix copy/paste duplicate Ricoh quirk ID
- Fixed AMD IOMMU for the "ghost" function case where the DMA
alias is for an absent device. The iommu rlookup table and
data fields need to be initializes.
- Fixed Intel interrupt remapping, I wasn't passing the target
bus number, only the alias bus number.
These patches are split across PCI and IOMMU, but I've front-loaded
all of the PCI infrastructure so that the first 7 patches can be
applied to PCI-core, the IOMMU maintainers can pickup their patches,
then we can finish with dead code removal. Bjorn might also be
willing to carry the IOMMU changes if the maintainers want to ack
them.
Original description:
This series attempts to fix a couple issues we've had outstanding in
the PCI/IOMMU code for a while. The first issue is with devices that
use the wrong requester ID for DMA transactions. We already have a
sort of half-baked attempt to fix this for several Ricoh devices, but
the fix only helps them be useful through IOMMU groups, not the
general DMA case. There are also several Marvell devices which use
use a different wrong requester ID and don't even fit into the DMA
source idea. This series creates a DMA alias iterator that will
step through each possible alias of a device, allowing IOMMUs to
insert mappings for both the device and its aliases.
Hand-in-hand with this is our broken pci_find_upstream_pcie_bridge()
function, which is known to blowup when it finds itself suddenly at
a PCIe device without crossing a PCIe-to-PCI bridge (as identified by
the PCIe capability). It also likes to make the invalid assumption
that a PCIe device never has its requester ID masked by any usptream
bus. We can fix this using the above new DMA alias iterator, since
that's effectively what this function was meant to do.
Finally, with all these helpers, it makes sense to consolidate code
for determining IOMMU groups. The first step in finding the root
of a group is finding the final upstream DMA alias for the device,
then applying additional ACS rules and incorporating device specific
aliases. As this is all common to PCI, create a single implementation
and remove piles of code from the individual IOMMU drivers.
This series allows devices like the Marvell 88SE9123 to finally work
on Linux with either AMD-Vi or VT-d enabled on the box. I've
collected device IDs from various bugs to support as many SKUs of
these devices as possible, but I'm sure there are others that I've
missed.
This should also enable motherboards with an onboard ASmedia
ASM1083/1085 PCIe-to-PCI bridge to work with VT-d enabled. I've
acquired an adapter board with this chip, but it actually exposes
a PCIe capability, unlike most of the onboard controllers. Therefore
I expect this series will fix the WARN_ON currently hit during boot,
but there's a 50/50 chance whether the device behaves like a PCI
bridge or a PCIe bridge with regard to the requester ID that it uses
to take ownership of the transaction. If it turns out to use the
PCIe bridge model, I expect we can quirk it using a dev_flags bit
to identify a PCI bridge that takes ownership as if it was a PCIe
bridge.
Please test and provide feedback. I expect IOMMU group topology
should not change from this series, but if a case is found where it
does, please share. Also, if there are additional quirks we need
to add, please either file new or add to the existing bugs. Thanks,
Alex
---
Alex Williamson (15):
PCI: Add DMA alias iterator
PCI: quirk pci_for_each_dma_alias()
PCI: quirk dma_func_alias for Ricoh devices
PCI: quirk dma_func_alias for Marvell devices
PCI: Quirk pci_for_each_dma_alias() for bridges
PCI: Add quirks for ASMedia and Tundra bridges
PCI: Consolidate isolation domain code
iommu/amd: Use pci_find_dma_isolation_root() for IOMMU groups
iommu/amd: Update to use PCI DMA aliases
iommu/intel: Use pci_find_dma_isolation_root() for IOMMU groups
iommu/intel: Update to use PCI DMA aliases
iommu/fsl: Use pci_find_dma_isolation_root() for IOMMU groups
iommu: Remove pci.h
PCI: Remove pci_find_upstream_pcie_bridge()
PCI: Remove pci_get_dma_source()
drivers/iommu/amd_iommu.c | 195 +++++------------------
drivers/iommu/amd_iommu_types.h | 1
drivers/iommu/fsl_pamu_domain.c | 67 --------
drivers/iommu/intel-iommu.c | 293 +++++++++++++----------------------
drivers/iommu/intel_irq_remapping.c | 55 +++++--
drivers/iommu/pci.h | 29 ---
drivers/pci/quirks.c | 112 ++++++++-----
drivers/pci/search.c | 240 ++++++++++++++++++++++++++---
include/linux/pci.h | 23 +--
9 files changed, 484 insertions(+), 531 deletions(-)
delete mode 100644 drivers/iommu/pci.h
^ permalink raw reply [flat|nested] 5+ messages in thread* Re: [PATCH v3 00/15] PCI/iommu: Fix DMA alias problems
2014-05-10 15:02 Alex Williamson
@ 2014-05-10 16:51 ` George Spelvin
2014-05-13 22:35 ` eddy0596
1 sibling, 0 replies; 5+ messages in thread
From: George Spelvin @ 2014-05-10 16:51 UTC (permalink / raw)
To: alex.williamson, iommu, linux-pci; +Cc: acooks, bhelgaas, linux-kernel, linux
Tested-by: George Spelvin <linux@horizon.com>
^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: [PATCH v3 00/15] PCI/iommu: Fix DMA alias problems
2014-05-10 15:02 Alex Williamson
2014-05-10 16:51 ` George Spelvin
@ 2014-05-13 22:35 ` eddy0596
2014-05-14 17:18 ` Alex Williamson
1 sibling, 1 reply; 5+ messages in thread
From: eddy0596 @ 2014-05-13 22:35 UTC (permalink / raw)
To: linux-kernel
Hello Alex,
Thanks for working on a fix on this long standing issue. I have applied the
amd portion of the IOMMU patches against the 3.14.3 kernel and found the
followings:
1) The computer would not boot up if it's from a cold start. The kernel log
shows that it hangs at the point the kernel attempt to attach the scsi disk
[sdk] that connects to the LSI-SAS2008 controller at pci 04:00.0. I can use
Ctrl+Alt+Del to reboot the computer. So, I guess the kernel didn't "hang"
and I don't see any oops either.
2) After a warm reboot with Ctrl+Alt+Del, the kernel will boot up fine. And,
the Marvell controller behaves properly (More stress test needed) and so as
the two LSI-SAS2008. A warm reboot after a hard reset at BIOS prompt will
also boot up fine.
3) Removing sdk and perform a cold reboot, the kernel stops after attaching
all the ST3000DM001 harddisks that connects to the LSI-SAS2008 at pci
01:00:0. The kernel stops at "ata12: SATA link down (SStatus 0 SControl
300)".
4) Removing sda and sdl that connects to the Marvell 88SE9172 at pci
09:00.0, the kernel stops after attaching the eight ST3000DM001 that
connects to the LSI-SAS2008 at pci 01:00:0.
5) Cold start with a kernel without the IOMMU patches starts up fine except
a number of kernel oops related to the Marvell controller complaining about
invalid PCI access from the AMD IOMMU.
Attached is the kernel boot log that's obtained with all HDDs attached and
successfully boot up after a warm reboot and some information on my setup.
Let me know if you need more information/log to help with debuging.
Best Regards,
Edward Cheung
Motherboard: Gigabyte GA-990FXA-UD5 Revision 1.0. Note that the kernel is
using software IO TLB belief due to broken IVRS table. I am still trying to
find a fix for this. If there's any ivrs kernel boot parameter I should use,
please let me know.
Kernel: 3.14.3 with SCST patches applied.
Base Distro and kernel compile environment: Ubuntu 14.04LTS
Following IOMMU patches applied (Marked with *).
(*)PCI: Add DMA alias iterator
(*)PCI: quirk pci_for_each_dma_alias()
(*)PCI: quirk dma_func_alias for Ricoh devices
(*)PCI: quirk dma_func_alias for Marvell devices
(*)PCI: Quirk pci_for_each_dma_alias() for bridges
PCI: Add quirks for ASMedia and Tundra bridges
(*)PCI: Consolidate isolation domain code
(*)iommu/amd: Use pci_find_dma_isolation_root() for IOMMU groups
(*)iommu/amd: Update to use PCI DMA aliases
iommu/intel: Use pci_find_dma_isolation_root() for IOMMU groups
iommu/intel: Update to use PCI DMA aliases
iommu/fsl: Use pci_find_dma_isolation_root() for IOMMU groups
iommu: Remove pci.h
PCI: Remove pci_find_upstream_pcie_bridge()
PCI: Remove pci_get_dma_source()
root@SAN-1:/mnt# lsscsi
[1:0:0:0] disk ATA ST3000DM001-1CH1 CC24 /dev/sdb
[1:0:1:0] disk ATA ST3000DM001-1CH1 CC24 /dev/sdc
[1:0:2:0] disk ATA ST3000DM001-1CH1 CC24 /dev/sdd
[1:0:3:0] disk ATA ST3000DM001-1CH1 CC24 /dev/sde
[1:0:4:0] disk ATA ST3000DM001-1CH1 CC24 /dev/sdf
[1:0:5:0] disk ATA ST3000DM001-1CH1 CC24 /dev/sdg
[1:0:6:0] disk ATA ST3000DM001-1CH1 CC24 /dev/sdh
[1:0:7:0] disk ATA ST3000DM001-1CH1 CC24 /dev/sdi
[9:0:0:0] disk ATA WDC WD6400AAKS-0 01.0 /dev/sda
[10:0:0:0] disk ATA WDC WD6400AAKS-0 01.0 /dev/sdl
[14:0:0:0] disk Kingston DataTraveler G2 1.00 /dev/sdj
[16:0:0:0] disk ATA ST3750330NS SN04 /dev/sdk
root@SAN-1:/mnt# lspci
00:00.0 Host bridge: Advanced Micro Devices, Inc. [AMD/ATI] RD890 PCI to PCI
bridge (external gfx0 port B) (rev 02)
00:00.2 IOMMU: Advanced Micro Devices, Inc. [AMD/ATI] RD990 I/O Memory
Management Unit (IOMMU)
00:02.0 PCI bridge: Advanced Micro Devices, Inc. [AMD/ATI] RD890 PCI to PCI
bridge (PCI express gpp port B)
00:09.0 PCI bridge: Advanced Micro Devices, Inc. [AMD/ATI] RD890 PCI to PCI
bridge (PCI express gpp port H)
00:0a.0 PCI bridge: Advanced Micro Devices, Inc. [AMD/ATI] RD890 PCI to PCI
bridge (external gfx1 port A)
00:0b.0 PCI bridge: Advanced Micro Devices, Inc. [AMD/ATI] RD890 PCI to PCI
bridge (NB-SB link)
00:0c.0 PCI bridge: Advanced Micro Devices, Inc. [AMD/ATI] RD890S PCI
Express bridge for GPP2 port 1
00:11.0 SATA controller: Advanced Micro Devices, Inc. [AMD/ATI]
SB7x0/SB8x0/SB9x0 SATA Controller [AHCI mode] (rev 40)
00:12.0 USB controller: Advanced Micro Devices, Inc. [AMD/ATI]
SB7x0/SB8x0/SB9x0 USB OHCI0 Controller
00:12.2 USB controller: Advanced Micro Devices, Inc. [AMD/ATI]
SB7x0/SB8x0/SB9x0 USB EHCI Controller
00:13.0 USB controller: Advanced Micro Devices, Inc. [AMD/ATI]
SB7x0/SB8x0/SB9x0 USB OHCI0 Controller
00:13.2 USB controller: Advanced Micro Devices, Inc. [AMD/ATI]
SB7x0/SB8x0/SB9x0 USB EHCI Controller
00:14.0 SMBus: Advanced Micro Devices, Inc. [AMD/ATI] SBx00 SMBus Controller
(rev 42)
00:14.3 ISA bridge: Advanced Micro Devices, Inc. [AMD/ATI] SB7x0/SB8x0/SB9x0
LPC host controller (rev 40)
00:14.4 PCI bridge: Advanced Micro Devices, Inc. [AMD/ATI] SBx00 PCI to PCI
Bridge (rev 40)
00:14.5 USB controller: Advanced Micro Devices, Inc. [AMD/ATI]
SB7x0/SB8x0/SB9x0 USB OHCI2 Controller
00:15.0 PCI bridge: Advanced Micro Devices, Inc. [AMD/ATI] SB700/SB800/SB900
PCI to PCI bridge (PCIE port 0)
00:15.1 PCI bridge: Advanced Micro Devices, Inc. [AMD/ATI] SB700/SB800/SB900
PCI to PCI bridge (PCIE port 1)
00:15.2 PCI bridge: Advanced Micro Devices, Inc. [AMD/ATI] SB900 PCI to PCI
bridge (PCIE port 2)
00:15.3 PCI bridge: Advanced Micro Devices, Inc. [AMD/ATI] SB900 PCI to PCI
bridge (PCIE port 3)
00:16.0 USB controller: Advanced Micro Devices, Inc. [AMD/ATI]
SB7x0/SB8x0/SB9x0 USB OHCI0 Controller
00:16.2 USB controller: Advanced Micro Devices, Inc. [AMD/ATI]
SB7x0/SB8x0/SB9x0 USB EHCI Controller
00:18.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 15h Processor
Function 0
00:18.1 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 15h Processor
Function 1
00:18.2 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 15h Processor
Function 2
00:18.3 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 15h Processor
Function 3
00:18.4 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 15h Processor
Function 4
00:18.5 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 15h Processor
Function 5
01:00.0 Serial Attached SCSI controller: LSI Logic / Symbios Logic SAS2008
PCI-Express Fusion-MPT SAS-2 [Falcon] (rev 03)
02:00.0 USB controller: Etron Technology, Inc. EJ168 USB 3.0 Host Controller
(rev 01)
03:00.0 SATA controller: Marvell Technology Group Ltd. 88SE9172 SATA 6Gb/s
Controller (rev 11)
04:00.0 Serial Attached SCSI controller: LSI Logic / Symbios Logic SAS2008
PCI-Express Fusion-MPT SAS-2 [Falcon] (rev 03)
05:00.0 Fibre Channel: QLogic Corp. ISP2432-based 4Gb Fibre Channel to PCI
Express HBA (rev 02)
05:00.1 Fibre Channel: QLogic Corp. ISP2432-based 4Gb Fibre Channel to PCI
Express HBA (rev 02)
06:06.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI]
Device 515a
07:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd.
RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller (rev 06)
08:00.0 USB controller: Etron Technology, Inc. EJ168 USB 3.0 Host Controller
(rev 01)
09:00.0 SATA controller: Marvell Technology Group Ltd. 88SE9172 SATA 6Gb/s
Controller (rev 11)
0a:00.0 SATA controller: ASMedia Technology Inc. ASM1062 Serial ATA
Controller (rev 01)
--
View this message in context: http://linux-kernel.2935.n7.nabble.com/PATCH-v3-00-15-PCI-iommu-Fix-DMA-alias-problems-tp857369p859747.html
Sent from the Linux Kernel mailing list archive at Nabble.com.
^ permalink raw reply [flat|nested] 5+ messages in thread* Re: [PATCH v3 00/15] PCI/iommu: Fix DMA alias problems
2014-05-13 22:35 ` eddy0596
@ 2014-05-14 17:18 ` Alex Williamson
0 siblings, 0 replies; 5+ messages in thread
From: Alex Williamson @ 2014-05-14 17:18 UTC (permalink / raw)
To: eddy0596; +Cc: linux-kernel, iommu, Bjorn Helgaas, linux-pci
[+cc original lists]
Hi Edward,
On Tue, 2014-05-13 at 15:35 -0700, eddy0596 wrote:
> Hello Alex,
>
> Thanks for working on a fix on this long standing issue. I have applied the
> amd portion of the IOMMU patches against the 3.14.3 kernel and found the
> followings:
> 1) The computer would not boot up if it's from a cold start. The kernel log
> shows that it hangs at the point the kernel attempt to attach the scsi disk
> [sdk] that connects to the LSI-SAS2008 controller at pci 04:00.0. I can use
> Ctrl+Alt+Del to reboot the computer. So, I guess the kernel didn't "hang"
> and I don't see any oops either.
> 2) After a warm reboot with Ctrl+Alt+Del, the kernel will boot up fine. And,
> the Marvell controller behaves properly (More stress test needed) and so as
> the two LSI-SAS2008. A warm reboot after a hard reset at BIOS prompt will
> also boot up fine.
Both of these indicate that the hand-off state of the system is
different between a warm an cold reset. Can you capture the boot
messages (serial console or netconsole) of each case and add the
pci=earlydump option so we can compare the PCI state?
> 3) Removing sdk and perform a cold reboot, the kernel stops after attaching
> all the ST3000DM001 harddisks that connects to the LSI-SAS2008 at pci
> 01:00:0. The kernel stops at "ata12: SATA link down (SStatus 0 SControl
> 300)".
> 4) Removing sda and sdl that connects to the Marvell 88SE9172 at pci
> 09:00.0, the kernel stops after attaching the eight ST3000DM001 that
> connects to the LSI-SAS2008 at pci 01:00:0.
So it's not an issue with those specific disks. Is it possible to
remove or disable the controller in the BIOS to further isolate?
> 5) Cold start with a kernel without the IOMMU patches starts up fine except
> a number of kernel oops related to the Marvell controller complaining about
> invalid PCI access from the AMD IOMMU.
Is this kernel built from the same source tree as below without the
indicated IOMMU patches applied?
> Attached is the kernel boot log that's obtained with all HDDs attached and
> successfully boot up after a warm reboot and some information on my setup.
> Let me know if you need more information/log to help with debuging.
The mailing list doesn't like attachments, but it was included in the
re-send to me where it was inline. An unsuccessful boot log is probably
the most interesting, preferably with the pci=earlydump option (and
continue to use the amd_iommu_dump option as well). Also, what happens
with amd_iommu=off? If we're not getting any IOMMU faults, it seems
like the patches are doing their job and I'm at a bit of a loss to
understand how it would fail only on a cold boot.
It might also be useful to test the branch provided in case there's an
issue with backporting the patches to 3.14.
> Best Regards,
>
> Edward Cheung
>
> Motherboard: Gigabyte GA-990FXA-UD5 Revision 1.0. Note that the kernel is
> using software IO TLB belief due to broken IVRS table. I am still trying to
> find a fix for this.
What brings you to the conclusion that the IVRS table is broken? IIRC,
AMD-Vi initializes the swiotlb to support pasthrough devices that can
only do 32bit DMA... or something like that. So I don't think it's
unusual to see it initialized alongside AMD IOMMU. Thanks,
Alex
^ permalink raw reply [flat|nested] 5+ messages in thread
end of thread, other threads:[~2014-05-16 6:28 UTC | newest]
Thread overview: 5+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
[not found] <400175834.3289.65.camel@ul30vt.home>
2014-05-16 6:28 ` [PATCH v3 00/15] PCI/iommu: Fix DMA alias problems colin
2014-05-10 15:02 Alex Williamson
2014-05-10 16:51 ` George Spelvin
2014-05-13 22:35 ` eddy0596
2014-05-14 17:18 ` Alex Williamson
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox