Re: pvops dom0: no sound after boot; possibly causedby swiotlb

[Konrad Rzeszutek Wilk <konrad.wilk@oracle.com>]

On Mon, Feb 01, 2010 at 09:54:02PM +0100, Ronny.Hegewald@online.de wrote:
> >Well, since you are volunteering. The problem sounds like the sound card
> >allocates a buffer from the region above 4GB and tries to DMA to it.
> >Keep in mind that on most machines, when you have 4GB, 768 MB of it are 
> >offset
> >past the 4GB mark. You have these two mega regions: 0-3.3GB, 4GB-4.7GB,
> >the 3.3GB to 4GB is called the PCI hole.
> 
> I added some log-messages in the sound-driver and indeed, it allocates memory from above 4GB.
> 
> On bare metal and with dom0 based on the forward-ported xen-patches from gentoo the dma-memory is allocated in the 4GB range.
> 
> When i checked the dma-memory allocation in the driver i noticed that depending if the sound-card reports that it is 64bit-compatible the driver sets the dma_mask to 64, else to 32 (by calling pci_set_consistent_dma_mask).
> 
> So i changed the driver that it always sets the dma-mask to 32 - and the sound worked without the need to start a domU. 

Good detective job. BTW, what is the name of the driver in the source
code?
> 
> Which left the question why it didnt worked out of the box on pvops-dom0.
> 
> So i followed the code for the dma-allocation from the sound-driver and found the problem in the xen-swiotlb code in arch/x86/xen/pci-swiotlb.
> 
> Bare metal and the forward-ported dom0 both call dma_alloc_coherent_mask if they try to allocate coherent dma-memory - pvops dom0 doesnt. This call sets the DMA-mask to 24 bit when __GFP_DMA is set, else to 32-bit. 

What is your sound-driver detecting the card as? As 64-bit or 32-bit or
none of those?

On pvops we can get away from calling dma_alloc_coherent b/c we have
this piece of logic to determine where the driver can DMA from:

604         if (hwdev != NULL && hwdev->coherent_dma_mask)
605                 mask = hwdev->coherent_dma_mask;
606         else
607                 mask = DMA_BIT_MASK(32);

You mentioned that the sound driver calls the pci_set_consistent_dma_mask. That
function is the one that sets the coherent_dma_mask:

2045 int
2046 pci_set_consistent_dma_mask(struct pci_dev *dev, u64 mask)
2047 {
2048         if (!pci_dma_supported(dev, mask))
2049                 return -EIO;
2050
2051         dev->dev.coherent_dma_mask = mask;

So the 'xen_swiotlb_alloc_coherent' checks if you have the coherent DMA
mask and if not, assumes you have a driver that can only access up to
4GB. While the bare-metal assumes that if the driver doesn't have that
mask , it checks the gfp_t flag and if it has __GFP_DMA make the mask
24-bit, otherwise 32-bit.

The only difference here is that under pvops we behave badly with
devices that have GFP_DMA set and don't have the coherent_dma_mask
(which it does not seem to be the case?).

So is your sound-driver not detecting the card properly and not setting
the coherent_dma_mask and/or dma_mask?

Can you print out both of those entries when the sound driver
calls the 'xen_swiotlb_alloc_coherent' (without setting the flags to 32 forcefully?)