[PATCH 0/1] alpha: disable DAC for 32-bit PCI on tsunami

[PATCH 0/1] alpha: disable DAC for 32-bit PCI on tsunami

[Magnus Lindholm]

I've spent quite some time trying to make the qla1280 driver work
with 64-bit DMA on Alpha/tsunami systems with more than 2GB RAM,
Many thanks to Martin, James, Maciej, Thomas and Christoph who has take
the time to provide feedback and testing during my attempts.

This is what I've concluded so far:

*The ISP1040B 32-bitcard works with 64-bit DMA_MASK on a 21164 Rawhide
machine, hence the card supports DAC, even though the papers on the chip
don't officially claim support until rev C (just as Thomas Bogendoerfer
pointed out earlier).

*The ISP1080 (64-bit pci slot/card) works with 64-bit DMA_MASK  on a
21224 Tsunami machine, hence the monster window works fine on Tsunami.

*Data gets corrupted on Alpha/tsunami when DAC/Monster window is enabled
and the PCI card is a 32-bit card. The amount of data being corrupted
each run varies a lot, from no data corruption at all to several
kilobytes out of 20mb data. When data is corrupted it is always in
chunks of 64-bytes, which coincides with the cache block size on the
21264 processor. From manual inspection of the data, its clear that
the corrupted data is just memory content from other active processes
running DMA transfers from other drives/controllers.

The approach to fixing the above problems is to limit 32-bit PCI cards
from using DAC/monster window DMA on Tsunami based Alphas. This is
done by setting bus_dma_limit to 32 bits for devices that have no
64-bit memory BARs. There are examples of 64-bit PCI cards that only
have 32-bit memory BARs, like for example the QLogic ISP1080 and
IPS10160 SCSI controllers. These cards will be constrained to not use
DAC even if they are capable of doing so correctly on tsunami based
Alphas. I believe that this does not pose a problem since these cards
are not supported by any SRM firmware versions and hence rarely used
on Alpha.

In practice, I believe that there are very few 32-bit PCI cards that
are likely to be used on Alphas that have drivers which support 64-bit
DMA addressing. The only example I've found so far is the qla1280 driver.
but ISP1040 SCSI controllers are supported by most versions of SRM
firmware and is hence a fairly common controller on Alpha systems.

 arch/alpha/kernel/pci.c       | 25 +++++++++++++++++++++++++
 arch/alpha/kernel/pci_iommu.c | 18 ++++++++++++------
 2 files changed, 37 insertions(+), 6 deletions(-)

-- 
2.49.0

[PATCH 1/1] alpha: disable DAC for 32-bit PCI on tsunami

[Magnus Lindholm]

Disable DAC for 32-bit PCI cards on Tsunami.
This patch will effectively disable DAC for 32-bit PCI cards on
Tsunami based Alphas. This is done by setting bus_dma_limit to 32 bits
for devices that have no 64-bit memory BARs. Relevant functions in
pci_iommu.c are modified to take 'dev->bus_dma_limit' into
account.

Suggested-by: Maciej Rozycki <macro@orcam.me.uk>
Signed-off-by: Magnus Lindholm <linmag7@gmail.com>
---
 arch/alpha/kernel/pci.c       | 25 +++++++++++++++++++++++++
 arch/alpha/kernel/pci_iommu.c | 18 ++++++++++++------
 2 files changed, 37 insertions(+), 6 deletions(-)

diff --git a/arch/alpha/kernel/pci.c b/arch/alpha/kernel/pci.c
index 8e9b4ac86b7e..5b1e4782294b 100644
--- a/arch/alpha/kernel/pci.c
+++ b/arch/alpha/kernel/pci.c
@@ -117,6 +117,31 @@ static void pcibios_fixup_final(struct pci_dev *dev)
 }
 DECLARE_PCI_FIXUP_FINAL(PCI_ANY_ID, PCI_ANY_ID, pcibios_fixup_final);
 
+static void tsunami_dac_quirk(struct pci_dev *pdev)
+{
+	unsigned long flags;
+	int i;
+	bool mem64 = false;
+
+	/* If we're not on a Tsunami based system, do nothing */
+	if (hwrpb->sys_type != 34)
+		return;
+
+	for (i = 0; i < PCI_STD_RESOURCE_END + 1; i++) {
+		flags = pci_resource_flags(pdev, i);
+		if (flags & IORESOURCE_MEM)
+			mem64 |= flags & IORESOURCE_MEM_64;
+	}
+
+	/* Limit DMA to 32 bits effectively disabling DAC */
+	if (!mem64) {
+		pdev->dev.bus_dma_limit = DMA_BIT_MASK(32);
+		dev_dbg(&pdev->dev, "disabling DAC for device");
+	}
+}
+DECLARE_PCI_FIXUP_FINAL(PCI_ANY_ID, PCI_ANY_ID, tsunami_dac_quirk);
+
+
 /* Just declaring that the power-of-ten prefixes are actually the
    power-of-two ones doesn't make it true :) */
 #define KB			1024
diff --git a/arch/alpha/kernel/pci_iommu.c b/arch/alpha/kernel/pci_iommu.c
index dc91de50f906..caf2407f80d3 100644
--- a/arch/alpha/kernel/pci_iommu.c
+++ b/arch/alpha/kernel/pci_iommu.c
@@ -208,7 +208,8 @@ static int pci_dac_dma_supported(struct pci_dev *dev, u64 mask)
 		ok = 0;
 
 	/* The device has to be able to address our DAC bit.  */
-	if ((dac_offset & dev->dma_mask) != dac_offset)
+	if ((dac_offset & min_not_zero(mask,
+		dev->dev.bus_dma_limit)) != dac_offset)
 		ok = 0;
 
 	/* If both conditions above are met, we are fine. */
@@ -228,7 +229,8 @@ pci_map_single_1(struct pci_dev *pdev, void *cpu_addr, size_t size,
 		 int dac_allowed)
 {
 	struct pci_controller *hose = pdev ? pdev->sysdata : pci_isa_hose;
-	dma_addr_t max_dma = pdev ? pdev->dma_mask : ISA_DMA_MASK;
+	dma_addr_t max_dma = pdev ? min_not_zero(pdev->dma_mask,
+		pdev->dev.bus_dma_limit) : ISA_DMA_MASK;
 	struct pci_iommu_arena *arena;
 	long npages, dma_ofs, i;
 	unsigned long paddr;
@@ -332,7 +334,8 @@ static dma_addr_t alpha_pci_map_page(struct device *dev, struct page *page,
 
 	BUG_ON(dir == DMA_NONE);
 
-	dac_allowed = pdev ? pci_dac_dma_supported(pdev, pdev->dma_mask) : 0; 
+	dac_allowed = pdev ? pci_dac_dma_supported(pdev,
+		min_not_zero(pdev->dma_mask, pdev->dev.bus_dma_limit)) : 0;
 	return pci_map_single_1(pdev, (char *)page_address(page) + offset, 
 				size, dac_allowed);
 }
@@ -638,7 +641,8 @@ static int alpha_pci_map_sg(struct device *dev, struct scatterlist *sg,
 
 	BUG_ON(dir == DMA_NONE);
 
-	dac_allowed = dev ? pci_dac_dma_supported(pdev, pdev->dma_mask) : 0;
+	dac_allowed = dev ? pci_dac_dma_supported(pdev,
+		min_not_zero(pdev->dma_mask, pdev->dev.bus_dma_limit)) : 0;
 
 	/* Fast path single entry scatterlists.  */
 	if (nents == 1) {
@@ -660,7 +664,8 @@ static int alpha_pci_map_sg(struct device *dev, struct scatterlist *sg,
 	/* Second, figure out where we're going to map things.  */
 	if (alpha_mv.mv_pci_tbi) {
 		hose = pdev ? pdev->sysdata : pci_isa_hose;
-		max_dma = pdev ? pdev->dma_mask : ISA_DMA_MASK;
+		max_dma = pdev ? min_not_zero(pdev->dma_mask,
+			pdev->dev.bus_dma_limit) : ISA_DMA_MASK;
 		arena = hose->sg_pci;
 		if (!arena || arena->dma_base + arena->size - 1 > max_dma)
 			arena = hose->sg_isa;
@@ -725,7 +730,8 @@ static void alpha_pci_unmap_sg(struct device *dev, struct scatterlist *sg,
 		return;
 
 	hose = pdev ? pdev->sysdata : pci_isa_hose;
-	max_dma = pdev ? pdev->dma_mask : ISA_DMA_MASK;
+	max_dma = pdev ? min_not_zero(pdev->dma_mask,
+		pdev->dev.bus_dma_limit) : ISA_DMA_MASK;
 	arena = hose->sg_pci;
 	if (!arena || arena->dma_base + arena->size - 1 > max_dma)
 		arena = hose->sg_isa;
-- 
2.49.0