Re: pvops: AHCI problems with SB600

[Jeremy Fitzhardinge <jeremy@goop.org>]

On 09/23/09 14:24, Konrad Rzeszutek Wilk wrote:
> The weird part is that the function you copied-n-pasted (gart_iommu_hole_init)
> only detectes and allocates a buffer. It does not set the dma_ops at all.
> Setting of the dma_ops is done via the gart_iommu_init() call which is done
> much later. But with Xen-SWIOTLB already initialized, the gart_iommu_init()
> quits right away.
>   

Perhaps the fix is to make gart_iommu_hole_init() quit if iommu_detected
too?  Though it is called after xen_swiotlb_init()...

> So the kernel sets the dma_ops to the Xen SWIOTLB, and it
> allocates an extra 64MB chunk of memory for the GART, which is not
> used, and ... somehow all of the ioremap_nocache functions stop working
> correctly. Maybe the ioremap_nocache does use some of that memory that
> the gart_iommu_hole_init allocated?
>   
Can't see how it would affect it.  ioremap allocates a new virtual space
for the mapping and then just plugs in the pfns for the pages you want
to map.  They end up getting _PAGE_IOMAP set in the pte flags, which
causes the xen/mmu.c backend to use the address as-is (ie, as an mfn),
so the mapping will be constructed properly.  Well, that's the theory;
but I'd expect we'd be seeing a lot more havok of ioremap is either
mapping the wrong pages or using the wrong caching.

> With this patch, the GART is forcefully disabled, and the kernel boots fine
> (with 6GB, 8GB, etc).
>   

OK, I'll put it in for now.  Will we have issues with other forms of iommu?

Another thought, could we actually use the gart iommu instead of swiotlb
if it is available?  I think it leads to exactly the same set of issues
as extending normal swiotlb for Xen's use (ie, inserting pfn->mfn
conversion into the correct places, and perhaps allocating the memory
properly).  Worth thinking about; it may shine light on better ways to
fix up swiotlb.

Thanks,
    J