pvops dom0: no sound after boot; possibly caused by swiotlb

pvops dom0: no sound after boot; possibly caused by swiotlb

[Ronny.Hegewald]

Software: xen 3.4.1, lastest xen/master (version 2.6.31.6), both 32bit
Hardware: Intel Core2Duo System
          4GB Ram
          Realtek ALC888 soundchip

Initial Symptoms:

When playing audio in dom0 there are just "knock" sounds.

After rmmod the kernel-module for the soundcard (snd-hda-intel) and oss modules (snd-seq-oss and snd-seq-pcm) and inserting them with modprobe again makes the sound work. Doing that only with snd-hda-intel doesnt help.

I compiled the sound-modules directly into the kernel but that didnt changed anything.

This problem doesnt appear with the gentoo-dom0 patches for kernel 2.6.31 so it looks like a pvops dom0 problem.

Strangely that problem doesnt appear on another system with the same xen-version and the exactly same kernel. Main-difference is that the other system is a 2-core AMD-system with 2 GB Ram and a different soundcard. 

But starting the domU with only 2 GB didnt made any difference.


Final findings:

It finally turns out that when the sound-modules are loaded after a pv-domU is started the sound in domU works fine from the beginning. 

As i suspected a problem in the memory-layout that got "fixed" by the start of a PV-domU i started the domU with different memory-sizes and found out that the sound works fine if the domU is started with at least 66 MB. Everything under that and there is no sound (even the knock-sound is not there)

The first thing i found that had could have to do with the 66 MB was the 64 MB swiotlb buffer. To check that this is really the problem i changed the code in arch/x86/xen/pci-swiotlb.c and lowered the allocated buffer to 32MB. After that change the sound worked from the beginning when the domU was started with less then 66 MB.


Further investigations:

>From here i dont know at the moment how to investigate that problem further myself.

Which logs should i post that could help to find the problem?

What further steps could/should i do to investigate that myself?

Re: pvops dom0: no sound after boot; possibly caused by swiotlb

[Keir Fraser]

On 26/01/2010 00:40, "Ronny.Hegewald@online.de" <Ronny.Hegewald@online.de>
wrote:

> When playing audio in dom0 there are just "knock" sounds.

Your bug report seems to talk about dom0 and then switches seamlessly to
talking about sound and swiotlbs in domUs. It's unclear.

 -- Keir

> After rmmod the kernel-module for the soundcard (snd-hda-intel) and oss
> modules (snd-seq-oss and snd-seq-pcm) and inserting them with modprobe again
> makes the sound work. Doing that only with snd-hda-intel doesnt help.
> 
> I compiled the sound-modules directly into the kernel but that didnt changed
> anything.
> 
> This problem doesnt appear with the gentoo-dom0 patches for kernel 2.6.31 so
> it looks like a pvops dom0 problem.
> 
> Strangely that problem doesnt appear on another system with the same
> xen-version and the exactly same kernel. Main-difference is that the other
> system is a 2-core AMD-system with 2 GB Ram and a different soundcard.
> 
> But starting the domU with only 2 GB didnt made any difference.

Re: pvops dom0: no sound after boot; possibly caused by swiotlb

[Konrad Rzeszutek Wilk]

On Tue, Jan 26, 2010 at 01:40:12AM +0100, Ronny.Hegewald@online.de wrote:
> Software: xen 3.4.1, lastest xen/master (version 2.6.31.6), both 32bit
> Hardware: Intel Core2Duo System
>           4GB Ram
>           Realtek ALC888 soundchip

Motherboard?

> 
> Initial Symptoms:
> 
> When playing audio in dom0 there are just "knock" sounds.
> 
> After rmmod the kernel-module for the soundcard (snd-hda-intel) and oss modules (snd-seq-oss and snd-seq-pcm) and inserting them with modprobe again makes the sound work. Doing that only with snd-hda-intel doesnt help.
> 
> I compiled the sound-modules directly into the kernel but that didnt changed anything.
> 
> This problem doesnt appear with the gentoo-dom0 patches for kernel 2.6.31 so it looks like a pvops dom0 problem.
> 
> Strangely that problem doesnt appear on another system with the same xen-version and the exactly same kernel. Main-difference is that the other system is a 2-core AMD-system with 2 GB Ram and a different soundcard. 

If you have 2G in your Intel box does that make the problem go away?
> 
> But starting the domU with only 2 GB didnt made any difference.
> 
> 
> Final findings:
> 
> It finally turns out that when the sound-modules are loaded after a pv-domU is started the sound in domU works fine from the beginning. 
> 
> As i suspected a problem in the memory-layout that got "fixed" by the start of a PV-domU i started the domU with different memory-sizes and found out that the sound works fine if the domU is started with at least 66 MB. Everything under that and there is no sound (even the knock-sound is not there)
> 
> The first thing i found that had could have to do with the 66 MB was the 64 MB swiotlb buffer. To check that this is really the problem i changed the code in arch/x86/xen/pci-swiotlb.c and lowered the allocated buffer to 32MB. After that change the sound worked from the beginning when the domU was started with less then 66 MB.

You could also pass in 'swiotlb=16384' as boot-up parameter- that would change it to 32MB.

> 
> 
> Further investigations:
> 
> >From here i dont know at the moment how to investigate that problem further myself.
> 
> Which logs should i post that could help to find the problem?

I am confused.

What I think you are saying is that, when you perform these steps:

1). Boot up machine, have swiotlb=16384 set (or just hand-coded
swiotlb.c to have a default size of 32MB).

2). Start a DomU with more than 66MB allocated to it.

3). Load the sound modules in Dom0

4). Play music/sound/etc in Dom0, the sound works fine.

What I am curious is your E820 table. That is the first thing Xen
prints. You can get that by doing 'xm dmesg'

Also include your 'dmesg' output for good measure. 

Lastly, try adding this line to your Xen line: dom0_mem=max:2GB and
don't do any of the above mentioned hacks. Just start the sound module
and see if it works.

> 
> What further steps could/should i do to investigate that myself?

Well, since you are volunteering. The problem sounds like the sound card
allocates a buffer from the region above 4GB and tries to DMA to it.
Keep in mind that on most machines, when you have 4GB, 768 MB of it are offset
past the 4GB mark. You have these two mega regions: 0-3.3GB, 4GB-4.7GB,
the 3.3GB to 4GB is called the PCI hole.

Back to the sound card. The sound card can (I think) only DMA up to 4GB,
so when it tries to fetch data from above 4GB it ends up truncating the
physical address down to 32-bit and fetches data from somewhere else. So
you are probably listening to binary code being played :-)

Having a DomU start makes Xen shrink Dom0 by a certain size and the DMA
window for the sound is moved "down" below the 4GB mark (that I think is
the problem you are encountering). This should NOT happen if the sound
card driver is using DMA libraries to allocate that buffer (ie,
dma_alloc_coherent, dma_alloc_free, etc). But it might be using
'vmalloc_32' which on virtualized environments is not guaranteed to give
a swatch of memory below 4GB. Here are the steps to investigate:
look in the sound card drivers and see where and how it allocates the buffer.

Re: pvops dom0: no sound after boot; possibly caused by swiotlb

[Ronny.Hegewald]

>> BTW, what is the name of the driver in the source code? 

Its the driver under sound/pci/hda. In hda_intel.c are all the important calls for DMA and where the coherent_dma_mask is set.

>What is your sound-driver detecting the card as? As 64-bit or 32-bit or
>none of those?

As 64-bit.

>On pvops we can get away from calling dma_alloc_coherent b/c we have
>this piece of logic to determine where the driver can DMA from:
>
>604         if (hwdev != NULL && hwdev->coherent_dma_mask)
>605                 mask = hwdev->coherent_dma_mask;
>606         else
>607                 mask = DMA_BIT_MASK(32);

...

>So the 'xen_swiotlb_alloc_coherent' checks if you have the coherent DMA
>mask and if not, assumes you have a driver that can only access up to
>4GB. While the bare-metal assumes that if the driver doesn't have that
>mask , it checks the gfp_t flag and if it has __GFP_DMA make the mask
>24-bit, otherwise 32-bit.

But thats not quite all whats dma_alloc_coherent does. As it only returns a 32-bit variable all coherent_dma_mask over 32-bit get casted down. This way bare-metal makes sure that the dma-mask is never over 32-bit.

Or are you are saying that when the hardware supports 64 bit and has set the coherent_dma_mask accordingly and dom0 is 32-bit that the allocation of DMA after the 4 GB should work fine? Because thats the assumption i see in the pvops-code.

And from what i understood so far the DMA memory should be allocated preferably in the 24-bit address space or max. 32 bit address space, at least in a 32-bit kernel. 

>The only difference here is that under pvops we behave badly with
>devices that have GFP_DMA set and don't have the coherent_dma_mask
>(which it does not seem to be the case?).

As i understand it the opposite is the case. If coherent_dma_mask is not set xen_swiotlb_alloc_coherent sets it to 32-bit. That should work usually (except the device needs the dma-memory in the 24-bit space). 

The problem-case is that the coherent_dma_mask is set. So pvops-dom0 just uses this value, when bare-metal makes sure that it cant be over 32bit by calling dma_alloc_coherent_mask. 

>So is your sound-driver not detecting the card properly and not setting
>the coherent_dma_mask and/or dma_mask?

>From what i have seen the driver works correct. It checks a register of the soundcard if it supports 64 bit and sets the coherent_dma_mask to 64bit, else to 32bit. 

As my soundcard says that it supports the 64bit the mask is set accordingly.

I added debug-messages in the code to be sure about that, when i researched the issue.

>Can you print out both of those entries when the sound driver
>calls the 'xen_swiotlb_alloc_coherent' (without setting the flags to 32 
>forcefully?)

The value of the coherent_dma_mask in xen_swiotlb_alloc_coherent was 0xFFFFFFFFFFFFFFFF when i didnt set 32bit forcefully. 

Until now i only checked the coherent_dma_mask flag because as i understand it, thats the value that is used when coherent dma memory is requested. And i never saw dma_mask used. 

But i can send the values of dma_mask tomorrow if they are useful in that case.

Re: pvops dom0: no sound after boot; possibly caused by swiotlb

[Konrad Rzeszutek Wilk]

> But thats not quite all whats dma_alloc_coherent does. As it only returns a 32-bit variable all coherent_dma_mask over 32-bit get casted down. This way bare-metal makes sure that the dma-mask is never over 32-bit.

Ooooh. I completly failed to notice that your dom0 was 32-bit.

But having that there would make the mask always be below
4GB, irregardless if the dom0 is 32 or 64-bit. Which is
exactly what it does on bare-metal. Hmmm, ok: 

http://kerneltrap.org/mailarchive/git-commits-head/2008/10/28/3841954

shows what made that happen.

I am bit worried on the casting - it does not seem to have been the
purpose of that change to utilize that, but that is a upstream problem.

I am curious - if you dom0 is 64-bit, does the sound card work?
> 
> Or are you are saying that when the hardware supports 64 bit and has set the coherent_dma_mask accordingly and dom0 is 32-bit that the allocation of DMA after the 4 GB should work fine? Because thats the assumption i see in the pvops-code.

Yes. It should have worked fine.

> 
> And from what i understood so far the DMA memory should be allocated preferably in the 24-bit address space or max. 32 bit address space, at least in a 32-bit kernel. 
> 
> >The only difference here is that under pvops we behave badly with
> >devices that have GFP_DMA set and don't have the coherent_dma_mask
> >(which it does not seem to be the case?).
> 
> As i understand it the opposite is the case. If coherent_dma_mask is not set xen_swiotlb_alloc_coherent sets it to 32-bit. That should work usually (except the device needs the dma-memory in the 24-bit space). 
> 

You are right.

Re: pvops dom0: no sound after boot; possibly caused by swiotlb

[Konrad Rzeszutek Wilk]

On Tue, Feb 02, 2010 at 07:31:16PM -0500, Konrad Rzeszutek Wilk wrote:
> > But thats not quite all whats dma_alloc_coherent does. As it only returns a 32-bit variable all coherent_dma_mask over 32-bit get casted down. This way bare-metal makes sure that the dma-mask is never over 32-bit.
> 
> Ooooh. I completly failed to notice that your dom0 was 32-bit.
> 
> But having that there would make the mask always be below
> 4GB, irregardless if the dom0 is 32 or 64-bit. Which is

<sigh> That is actually incorrect. Looking at the
dma_alloc_coherent_mask I missed the '!' and thought it would set it
irregardless of what the previous value was. That is not the case.

So your fix is good and it should be put in pv-ops. Thanks for tracking
down the problem and fixing it.