From mboxrd@z Thu Jan 1 00:00:00 1970 From: Konrad Rzeszutek Wilk Subject: Re: pvops dom0: no sound after boot; possibly causedby swiotlb Date: Mon, 1 Feb 2010 16:52:14 -0500 Message-ID: <20100201215214.GD2756@phenom.dumpdata.com> References: <5670117.3805321265057642467.JavaMail.servlet@kundenserver> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Return-path: Content-Disposition: inline In-Reply-To: <5670117.3805321265057642467.JavaMail.servlet@kundenserver> List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Sender: xen-devel-bounces@lists.xensource.com Errors-To: xen-devel-bounces@lists.xensource.com To: Ronny.Hegewald@online.de Cc: xen-devel@lists.xensource.com List-Id: xen-devel@lists.xenproject.org On Mon, Feb 01, 2010 at 09:54:02PM +0100, Ronny.Hegewald@online.de wrote: > >Well, since you are volunteering. The problem sounds like the sound card > >allocates a buffer from the region above 4GB and tries to DMA to it. > >Keep in mind that on most machines, when you have 4GB, 768 MB of it are > >offset > >past the 4GB mark. You have these two mega regions: 0-3.3GB, 4GB-4.7GB, > >the 3.3GB to 4GB is called the PCI hole. > > I added some log-messages in the sound-driver and indeed, it allocates memory from above 4GB. > > On bare metal and with dom0 based on the forward-ported xen-patches from gentoo the dma-memory is allocated in the 4GB range. > > When i checked the dma-memory allocation in the driver i noticed that depending if the sound-card reports that it is 64bit-compatible the driver sets the dma_mask to 64, else to 32 (by calling pci_set_consistent_dma_mask). > > So i changed the driver that it always sets the dma-mask to 32 - and the sound worked without the need to start a domU. Good detective job. BTW, what is the name of the driver in the source code? > > Which left the question why it didnt worked out of the box on pvops-dom0. > > So i followed the code for the dma-allocation from the sound-driver and found the problem in the xen-swiotlb code in arch/x86/xen/pci-swiotlb. > > Bare metal and the forward-ported dom0 both call dma_alloc_coherent_mask if they try to allocate coherent dma-memory - pvops dom0 doesnt. This call sets the DMA-mask to 24 bit when __GFP_DMA is set, else to 32-bit. What is your sound-driver detecting the card as? As 64-bit or 32-bit or none of those? On pvops we can get away from calling dma_alloc_coherent b/c we have this piece of logic to determine where the driver can DMA from: 604 if (hwdev != NULL && hwdev->coherent_dma_mask) 605 mask = hwdev->coherent_dma_mask; 606 else 607 mask = DMA_BIT_MASK(32); You mentioned that the sound driver calls the pci_set_consistent_dma_mask. That function is the one that sets the coherent_dma_mask: 2045 int 2046 pci_set_consistent_dma_mask(struct pci_dev *dev, u64 mask) 2047 { 2048 if (!pci_dma_supported(dev, mask)) 2049 return -EIO; 2050 2051 dev->dev.coherent_dma_mask = mask; So the 'xen_swiotlb_alloc_coherent' checks if you have the coherent DMA mask and if not, assumes you have a driver that can only access up to 4GB. While the bare-metal assumes that if the driver doesn't have that mask , it checks the gfp_t flag and if it has __GFP_DMA make the mask 24-bit, otherwise 32-bit. The only difference here is that under pvops we behave badly with devices that have GFP_DMA set and don't have the coherent_dma_mask (which it does not seem to be the case?). So is your sound-driver not detecting the card properly and not setting the coherent_dma_mask and/or dma_mask? Can you print out both of those entries when the sound driver calls the 'xen_swiotlb_alloc_coherent' (without setting the flags to 32 forcefully?)