Re: pvops: AHCI problems with SB600

[Konrad Rzeszutek Wilk <konrad.wilk@oracle.com>]

On Wed, Sep 23, 2009 at 01:30:44PM -0700, Jeremy Fitzhardinge wrote:
> On 09/23/09 13:09, Jeremy Fitzhardinge wrote:
> > 		if (!no_iommu &&
> > 		    max_pfn > MAX_DMA32_PFN &&
> > 		    !printed_gart_size_msg) {
> > 			printk(KERN_ERR "you are using iommu with agp, but GART size is less than 64M\n");
> > 			printk(KERN_ERR "please increase GART size in your BIOS setup\n");
> > 			printk(KERN_ERR "if BIOS doesn't have that option, contact your HW vendor!\n");
> > 			printed_gart_size_msg = 1;
> > 		}
> >   
> 
> Oh, that's the wrong error message, but the other one has similar
> predicates.  Hm, but it also skips the test if (swiotlb && !valid_agp)...

Right. We don't set the swiotlb. The reason being if you do set it then
the original SWIOTLB kicks in.

The weird part is that the function you copied-n-pasted (gart_iommu_hole_init)
only detectes and allocates a buffer. It does not set the dma_ops at all.
Setting of the dma_ops is done via the gart_iommu_init() call which is done
much later. But with Xen-SWIOTLB already initialized, the gart_iommu_init()
quits right away.

So the kernel sets the dma_ops to the Xen SWIOTLB, and it
allocates an extra 64MB chunk of memory for the GART, which is not
used, and ... somehow all of the ioremap_nocache functions stop working
correctly. Maybe the ioremap_nocache does use some of that memory that
the gart_iommu_hole_init allocated?

With this patch, the GART is forcefully disabled, and the kernel boots fine
(with 6GB, 8GB, etc).

diff --git a/arch/x86/kernel/pci-dma.c b/arch/x86/kernel/pci-dma.c
index 6b76948..1101a9f 100644
--- a/arch/x86/kernel/pci-dma.c
+++ b/arch/x86/kernel/pci-dma.c
@@ -122,6 +122,8 @@ void __init pci_iommu_alloc(void)
 	 * The order of these functions is important for
 	 * fall-back/fail-over reasons
 	 */
+	xen_swiotlb_init();
+
 	gart_iommu_hole_init();
 
 	detect_calgary();
@@ -130,8 +132,6 @@ void __init pci_iommu_alloc(void)
 
 	amd_iommu_detect();
 
-	xen_swiotlb_init();
-
 	pci_swiotlb_init();
 }
 
diff --git a/arch/x86/xen/pci-swiotlb.c b/arch/x86/xen/pci-swiotlb.c
index 5e2c856..00f2260 100644
--- a/arch/x86/xen/pci-swiotlb.c
+++ b/arch/x86/xen/pci-swiotlb.c
@@ -43,6 +43,10 @@
 #include <xen/page.h>
 #include <xen/xen-ops.h>
 
+
+#include <linux/pci.h>
+#include <asm/gart.h>
+
 #define OFFSET(val,align) ((unsigned long)	\
 	                   ( (val) & ( (align) - 1)))
 
@@ -985,5 +989,9 @@ void __init xen_swiotlb_init(void)
 		xen_swiotlb_init_with_default_size(64 * (1<<20));	/* default to 64MB */
 		dma_ops = &xen_swiotlb_dma_ops;
 		iommu_detected = 1;
+#ifdef CONFIG_GART_IOMMU
+		gart_iommu_aperture_disabled = 1;
+#endif
+
 	}
 }