DL385 G2 - PCI-DMA: Random memory would be DMAed

DL385 G2 - PCI-DMA: Random memory would be DMAed

[Zachary Richmond]

Mark suggested that I post this to the xen-devel list (I had orginally posted 
to xen-users):

Original post:

 > I am attempting to run Xen 3.1.1 on a HP DL385 G2 box (opteron based
 > system), using Gentoo.  The machine will boot, and I can login and edit
 > files etc, but as soon as I do anything more intensive such as copying
 > files off a cdrom (making me think it is a cciss driver problem) or running
 > make, I get the following :
 >
 > 	kernel panic not syncing: PCI-DMA: Random memory would be DMAed
 >
 > Searching on this doesn't come up with much except the source code where
 > this error message is written out.  The comment there indicates there is an
 > IOMMU problem, but I am not sure where to go from there. I am pretty sure
 > it is not a hardware issue since, when booting with a non-xen kernel and
 > the same file system, no problem.  Anyone have any ideas or run into this
 > problem before?
 >
 > As a side note, I have two of these boxes which are identical, and I have
 > successfully installed XenExpress on one of them, in the hopes of finding
 > something there.


To answer Mark's question:

 > What kernel are you using?

from gentoo: xen-sources-2.6.20-r6

Regarding the full error message, all I am getting is:

	kernel panic not syncing: PCI-DMA: Random memory would be DMAed

since the machine is rebooting after that.  There maybe somewhere else to get 
more detail, but beyond dmesg or messages I am not sure where else to look.



	Zak

DL385 G2 - PCI-DMA: Random memory would be DMAed

[Zachary Richmond]

Mark suggested that I post this to the xen-devel list (I had orginally posted 
to xen-users):

Original post:

 > I am attempting to run Xen 3.1.1 on a HP DL385 G2 box (opteron based
 > system), using Gentoo.  The machine will boot, and I can login and edit
 > files etc, but as soon as I do anything more intensive such as copying
 > files off a cdrom (making me think it is a cciss driver problem) or running
 > make, I get the following :
 >
 >     kernel panic not syncing: PCI-DMA: Random memory would be DMAed
 >
 > Searching on this doesn't come up with much except the source code where
 > this error message is written out.  The comment there indicates there is an
 > IOMMU problem, but I am not sure where to go from there. I am pretty sure
 > it is not a hardware issue since, when booting with a non-xen kernel and
 > the same file system, no problem.  Anyone have any ideas or run into this
 > problem before?
 >
 > As a side note, I have two of these boxes which are identical, and I have
 > successfully installed XenExpress on one of them, in the hopes of finding
 > something there.


To answer Mark's question:

 > What kernel are you using?

from gentoo: xen-sources-2.6.20-r6

Regarding the full error message, all I am getting is:

     kernel panic not syncing: PCI-DMA: Random memory would be DMAed

since the machine is rebooting after that.  There maybe somewhere else to get 
more detail, but beyond dmesg or messages I am not sure where else to look.



     Zak

RE: DL385 G2 - PCI-DMA: Random memory would be DMAed

[Langsdorf, Mark]

> To answer Mark's question:
> 
>  > What kernel are you using?
> 
> from gentoo: xen-sources-2.6.20-r6
> 
> Regarding the full error message, all I am getting is:
> 
>      kernel panic not syncing: PCI-DMA: Random memory would be DMAed
> 
> since the machine is rebooting after that.  There maybe 
> somewhere else to get more detail, but beyond dmesg or
> messages I am not sure where else to look.

For some reason, your kernel is running with pci-gart.c
enabled, and dom0 is trying to allocate DMA through the
AGP GART in your AMD64 processor's Northbridge.  I 
didn't think Xen supported pci-gart, but maybe I missed
a changeset.

At any rate, passing iommu=soft should disable the pci-gart
and allow you to use bounce buffering.  Hopefully, that
should clear up your problem.

-Mark Langsdorf
Operating System Research Center
AMD

Re: DL385 G2 - PCI-DMA: Random memory would be DMAed

[Mark Williamson]

Thanks for chiming in Mark, I was at a loss to suggest something specific!

> For some reason, your kernel is running with pci-gart.c
> enabled, and dom0 is trying to allocate DMA through the
> AGP GART in your AMD64 processor's Northbridge.  I
> didn't think Xen supported pci-gart, but maybe I missed
> a changeset.

Maybe a Gentoo-specific patch?

> At any rate, passing iommu=soft should disable the pci-gart
> and allow you to use bounce buffering.  Hopefully, that
> should clear up your problem.

If it does fix it, I'd suggest reporting it to the package maintainer.

Cheers,
Mark

-- 
Dave: Just a question. What use is a unicyle with no seat?  And no pedals!
Mark: To answer a question with a question: What use is a skateboard?
Dave: Skateboards have wheels.
Mark: My wheel has a wheel!

RE: DL385 G2 - PCI-DMA: Random memory would be DMAed

[James Harper]

>  > I am attempting to run Xen 3.1.1 on a HP DL385 G2 box (opteron
based
>  > system), using Gentoo.  The machine will boot, and I can login and
edit
>  > files etc, but as soon as I do anything more intensive such as
copying
>  > files off a cdrom (making me think it is a cciss driver problem) or
> running
>  > make, I get the following :
>  >
>  >     kernel panic not syncing: PCI-DMA: Random memory would be DMAed
>  >

FWIW, I am running Xen 3.1.0 with a Debian Etch Dom0 on a DL385G2 in a
production environment without any issues at all...

James

Re: DL385 G2 - PCI-DMA: Random memory would be DMAed

[Keir Fraser]

On 18/10/07 20:05, "Zachary Richmond" <zak@ihsdoor.com> wrote:

> Regarding the full error message, all I am getting is:
> 
> kernel panic not syncing: PCI-DMA: Random memory would be DMAed
> 
> since the machine is rebooting after that.  There maybe somewhere else to get
> more detail, but beyond dmesg or messages I am not sure where else to look.

You should also get a line immediately before that saying:
"PCI-DMA: Out of SW-IOMMU space for xxx bytes at device xxx"

This message makes it clear that your swiotlb is not big enough. The default
size is 2MB if your machine has less than 2GB RAM. Otherwise the default is
64MB. You can increase the swiotlb size by, for example, adding
'swiotlb=128' as a boot parameter to your dom0 kernel (the example here
would set the swiotlb size to be 128MB).

Setting the parameter too small will make your system crash (as you have
seen). Setting it bigger than necessary will waste memory. The optimum
setting will depend on your system.

 -- Keir

Re: DL385 G2 - PCI-DMA: Random memory would be DMAed

[Zachary Richmond]

Keir Fraser wrote:
> On 18/10/07 20:05, "Zachary Richmond" <zak@ihsdoor.com> wrote:
> 
>> Regarding the full error message, all I am getting is:
>>
>> kernel panic not syncing: PCI-DMA: Random memory would be DMAed
>>
>> since the machine is rebooting after that.  There maybe somewhere else to get
>> more detail, but beyond dmesg or messages I am not sure where else to look.
> 
> You should also get a line immediately before that saying:
> "PCI-DMA: Out of SW-IOMMU space for xxx bytes at device xxx"
> 
> This message makes it clear that your swiotlb is not big enough. The default
> size is 2MB if your machine has less than 2GB RAM. Otherwise the default is
> 64MB. You can increase the swiotlb size by, for example, adding
> 'swiotlb=128' as a boot parameter to your dom0 kernel (the example here
> would set the swiotlb size to be 128MB).
> 
> Setting the parameter too small will make your system crash (as you have
> seen). Setting it bigger than necessary will waste memory. The optimum
> setting will depend on your system.
> 
>  -- Keir
> 

Thanks for the tip.  If Mark's suggestion doesn't work I'll have to give this a 
try as well, but I wasn't receiving that error message.  When I looked at the 
source code it seems like I should be getting that error message as well, but 
it is a printk call which I think is sent to dmesg which is overwritten at the 
next boot.  I guess I could boot off a cd or into a different file system to 
preserve dmesg if the problem persists.

	Zak

Re: DL385 G2 - PCI-DMA: Random memory would be DMAed

[Zachary Richmond]

Langsdorf, Mark wrote:
>> To answer Mark's question:
>>
>>  > What kernel are you using?
>>
>> from gentoo: xen-sources-2.6.20-r6
>>
>> Regarding the full error message, all I am getting is:
>>
>>      kernel panic not syncing: PCI-DMA: Random memory would be DMAed
>>
>> since the machine is rebooting after that.  There maybe 
>> somewhere else to get more detail, but beyond dmesg or
>> messages I am not sure where else to look.
> 
> For some reason, your kernel is running with pci-gart.c
> enabled, and dom0 is trying to allocate DMA through the
> AGP GART in your AMD64 processor's Northbridge.  I 
> didn't think Xen supported pci-gart, but maybe I missed
> a changeset.
> 
> At any rate, passing iommu=soft should disable the pci-gart
> and allow you to use bounce buffering.  Hopefully, that
> should clear up your problem.
> 
> -Mark Langsdorf
> Operating System Research Center
> AMD
> 
> 
> 

Unfortunately I am still having the problem.  Having booted off a cd to 
preserve log files a quick grep on dmesg shows:

	PCI-DMA: Using software bounce buffering for IO (SWIOTLB)

So it is enabled, but still the kernel panic.  Any other ideas on where to poke 
around?

	Thanks,

	Zak

Re: DL385 G2 - PCI-DMA: Random memory would be DMAed

[Keir Fraser]

On 19/10/07 20:29, "Zachary Richmond" <admin.ihsdoor@gmail.com> wrote:

> Unfortunately I am still having the problem.  Having booted off a cd to
> preserve log files a quick grep on dmesg shows:
> 
> PCI-DMA: Using software bounce buffering for IO (SWIOTLB)
> 
> So it is enabled, but still the kernel panic.  Any other ideas on where to
> poke 
> around?

Have you increased the swiotlb size yet?

 K.

Re: DL385 G2 - PCI-DMA: Random memory would be DMAed

[Zachary Richmond]

Keir Fraser wrote:
> 
> 
> On 19/10/07 20:29, "Zachary Richmond" <admin.ihsdoor@gmail.com> wrote:
> 
>> Unfortunately I am still having the problem.  Having booted off a cd to
>> preserve log files a quick grep on dmesg shows:
>>
>> PCI-DMA: Using software bounce buffering for IO (SWIOTLB)
>>
>> So it is enabled, but still the kernel panic.  Any other ideas on where to
>> poke 
>> around?
> 
> Have you increased the swiotlb size yet?
> 
>  K.
> 

No, but my system on has 2GB for now.  I am however trying it now.

	Zak

Re: DL385 G2 - PCI-DMA: Random memory would be DMAed

[Keir Fraser]

On 19/10/07 20:56, "Zachary Richmond" <admin.ihsdoor@gmail.com> wrote:

>> Have you increased the swiotlb size yet?
>> 
>>  K.
>> 
> 
> No, but my system on has 2GB for now.  I am however trying it now.

Hence you only have a 2MB swiotlb by default. If your block driver cannot
address 2GB (and some controllers can't) then you'll be hitting the swiotlb
hard. Try swiotlb=16.

 -- Keir

Re: DL385 G2 - PCI-DMA: Random memory would be DMAed

[Zachary Richmond]

Zachary Richmond wrote:
> Keir Fraser wrote:
>>
>>
>> On 19/10/07 20:29, "Zachary Richmond" <admin.ihsdoor@gmail.com> wrote:
>>
>>> Unfortunately I am still having the problem.  Having booted off a cd to
>>> preserve log files a quick grep on dmesg shows:
>>>
>>> PCI-DMA: Using software bounce buffering for IO (SWIOTLB)
>>>
>>> So it is enabled, but still the kernel panic.  Any other ideas on 
>>> where to
>>> poke around?
>>
>> Have you increased the swiotlb size yet?
>>
>>  K.
>>
> 
> No, but my system on has 2GB for now.  I am however trying it now.
> 
>     Zak
> 

I set swiotlb=64, rebooted.  As a quick test, I am now joyfully compiling a 
kernel (after a clean) while running on my dom0 kernel.  I'll have to do a few 
more tests, but since I am running gentoo I will have a bit more compiling to 
do.  My guess is that the default swiotlb value is something else.  Per Mark 
Williamson's suggestion earlier I'll have to let the package maintainer know 
about this, and likely add this to http://gentoo-wiki.com/HOWTO_Xen_and_Gentoo.

 > Try swiotlb=16.

For now I'll probably keep it at 64 for now since I will be adding more memory 
later.  In the meantime I'll have to read up more on the various parameters to 
be able to fine tune the systems.


  Thanks for all the help,

	Zak