Re: Re: Re: PCI passthrough stopped working, brainache!

[Konrad Rzeszutek Wilk <konrad.wilk@oracle.com>]

On Sat, Oct 15, 2011 at 05:37:20PM +0200, Carsten Schiers wrote:
> Interesting. Konrad, do you remember that one of my DVB cards did 
> consume double CPU time 
> as compared to a) old Xenified 2.6.18 b) Xenified 2.6.34 and c) mem=2G 

Well, your DVB cards started consuming when you added more than 4GB to your
box. I don't remember the 2.6.18 figuring in this picture.

> Dom0? Maybe these
> observations are somehow connected.
> 
> It did work, though... Unfortunately, I sold these cards to have only 
> one, which is 

.. which is?

.. snip..
> A coconut for the gentleman!

Beer!
> 
> Working in dom0 and (with page-alignment of the PCI BARs) in the domU,
> I did check a couple of reboots and a cold start too, just in case!
> 
> So what's to look at for the real cause?

There can be one more test to conclude this and that is to raise right about 4GB
(ought to work) or perhaps to 5GB (it should fail there) under Xen .
Either way it sounds like a DMA issue - either we are:

 1). Not having the right dma_mask set. Check the /sys/bus/pci/devices/<BDF>/dma_mask
     and coherent_dma_mask. Both values should be the same and it ought to be 32
 
 2). The driver is not doing pci_sync_range .. which you should be able
     to easily find out. Try booting the Linux kernel in baremetal, with
     8GB and with "iommu=soft swiotlb=force". That will bounce buffer _everything_
     - which might show the same problem. It also might show problems with some
     other drivers.

 3). I can send you some patches to instrument Xen SWIOTLB to see what DMA pages
     (and where in the driver code is doing it) are using bounce buffer and not
     properly doing 2).