Re: pvops dom0: no sound after boot; possibly caused by swiotlb

[Konrad Rzeszutek Wilk <konrad.wilk@oracle.com>]

On Tue, Jan 26, 2010 at 01:40:12AM +0100, Ronny.Hegewald@online.de wrote:
> Software: xen 3.4.1, lastest xen/master (version 2.6.31.6), both 32bit
> Hardware: Intel Core2Duo System
>           4GB Ram
>           Realtek ALC888 soundchip

Motherboard?

> 
> Initial Symptoms:
> 
> When playing audio in dom0 there are just "knock" sounds.
> 
> After rmmod the kernel-module for the soundcard (snd-hda-intel) and oss modules (snd-seq-oss and snd-seq-pcm) and inserting them with modprobe again makes the sound work. Doing that only with snd-hda-intel doesnt help.
> 
> I compiled the sound-modules directly into the kernel but that didnt changed anything.
> 
> This problem doesnt appear with the gentoo-dom0 patches for kernel 2.6.31 so it looks like a pvops dom0 problem.
> 
> Strangely that problem doesnt appear on another system with the same xen-version and the exactly same kernel. Main-difference is that the other system is a 2-core AMD-system with 2 GB Ram and a different soundcard. 

If you have 2G in your Intel box does that make the problem go away?
> 
> But starting the domU with only 2 GB didnt made any difference.
> 
> 
> Final findings:
> 
> It finally turns out that when the sound-modules are loaded after a pv-domU is started the sound in domU works fine from the beginning. 
> 
> As i suspected a problem in the memory-layout that got "fixed" by the start of a PV-domU i started the domU with different memory-sizes and found out that the sound works fine if the domU is started with at least 66 MB. Everything under that and there is no sound (even the knock-sound is not there)
> 
> The first thing i found that had could have to do with the 66 MB was the 64 MB swiotlb buffer. To check that this is really the problem i changed the code in arch/x86/xen/pci-swiotlb.c and lowered the allocated buffer to 32MB. After that change the sound worked from the beginning when the domU was started with less then 66 MB.

You could also pass in 'swiotlb=16384' as boot-up parameter- that would change it to 32MB.

> 
> 
> Further investigations:
> 
> >From here i dont know at the moment how to investigate that problem further myself.
> 
> Which logs should i post that could help to find the problem?

I am confused.

What I think you are saying is that, when you perform these steps:

1). Boot up machine, have swiotlb=16384 set (or just hand-coded
swiotlb.c to have a default size of 32MB).

2). Start a DomU with more than 66MB allocated to it.

3). Load the sound modules in Dom0

4). Play music/sound/etc in Dom0, the sound works fine.

What I am curious is your E820 table. That is the first thing Xen
prints. You can get that by doing 'xm dmesg'

Also include your 'dmesg' output for good measure. 

Lastly, try adding this line to your Xen line: dom0_mem=max:2GB and
don't do any of the above mentioned hacks. Just start the sound module
and see if it works.

> 
> What further steps could/should i do to investigate that myself?

Well, since you are volunteering. The problem sounds like the sound card
allocates a buffer from the region above 4GB and tries to DMA to it.
Keep in mind that on most machines, when you have 4GB, 768 MB of it are offset
past the 4GB mark. You have these two mega regions: 0-3.3GB, 4GB-4.7GB,
the 3.3GB to 4GB is called the PCI hole.

Back to the sound card. The sound card can (I think) only DMA up to 4GB,
so when it tries to fetch data from above 4GB it ends up truncating the
physical address down to 32-bit and fetches data from somewhere else. So
you are probably listening to binary code being played :-)

Having a DomU start makes Xen shrink Dom0 by a certain size and the DMA
window for the sound is moved "down" below the 4GB mark (that I think is
the problem you are encountering). This should NOT happen if the sound
card driver is using DMA libraries to allocate that buffer (ie,
dma_alloc_coherent, dma_alloc_free, etc). But it might be using
'vmalloc_32' which on virtualized environments is not guaranteed to give
a swatch of memory below 4GB. Here are the steps to investigate:
look in the sound card drivers and see where and how it allocates the buffer.