Massive problems with 'PCI-DMA: Out of SW-IOMMU space for XXX bytes [..]' with 3Ware controller

Massive problems with 'PCI-DMA: Out of SW-IOMMU space for XXX bytes [..]' with 3Ware controller

[Markus Schuster]

Hi list, 

this days I've tried to update a 32bit XEN 3.0.4 system (kernel 2.6.16) to
XEN 3.1.2 on AMD64 (kernel 2.6.18). The system has 2GB RAM and a 3ware
9500-12 SATA raid controller. 

The old system worked fine for about half a year now. 
The new system boots fine so far but when doing some "bigger" changes to the
filesystem I get a lot of this errors:
> PCI-DMA: Out of SW-IOMMU space for 40960 bytes at device 0000:03:00.0
> 3w-9xxx: scsi0: ERROR: (0x06:0x001C): Failed to map scatter gather list.
(03:00.0 is the 3Ware Controller)

I've increased the 'swiotlb' kernel parameter in steps from 32, 64, 128, 256
up to 512MB. The only effect is, that with 512MB my dom0 uses over 600MB of
RAM (compared to ~50MB with XEN 3.0.4 on 32bit) and more fs-activity can be
done before the error arises. 
I's very easy to reproduce, just do a 'dd if=/dev/zero of=/root/testfile'.
After a short while (some seconds) the error arises. 

The funny thing is: I've tried this dom0 on my workstation before installing
it on a production server. My workstation has an nForce SATA controller and
here I don't have any problems - even without setting the 'swiotlb' kernel
paramter. 

It looks like there are some problems in conjunction with the 3Ware
controller that have not been there on XEN 3.0.4 on 2.6.16 (32bit). 

The complete piece of the logfile can be found here:
http://nopaste.biz/21780
And this is just from writing a 5MB initramfs to an ext2 filesytem (which
had some corruption afterwards). 

Thank's for your help!

Regards, 
Markus Schuster

Re: Massive problems with 'PCI-DMA: Out of SW-IOMMU space for XXX bytes [..]' with 3Ware controller

[Keir Fraser]

What happens if you run the old kernel (32-bit 2.6.16) on new Xen? What
happens if you run a new (2.6.18) 32-bit kernel on new Xen? Do these tests
without inflating the swiotlb size on the command line. Or what about new
2.6.18 kernel on old Xen?

(You can run 32-bit kernel on 64-bit Xen in 3.1.x, but not in 3.0.x, by the
way.)

 -- Keir

On 25/11/07 16:03, "Markus Schuster" <ma.schuster@gmx.de> wrote:

> Hi list, 
> 
> this days I've tried to update a 32bit XEN 3.0.4 system (kernel 2.6.16) to
> XEN 3.1.2 on AMD64 (kernel 2.6.18). The system has 2GB RAM and a 3ware
> 9500-12 SATA raid controller.
> 
> The old system worked fine for about half a year now.
> The new system boots fine so far but when doing some "bigger" changes to the
> filesystem I get a lot of this errors:
>> PCI-DMA: Out of SW-IOMMU space for 40960 bytes at device 0000:03:00.0
>> 3w-9xxx: scsi0: ERROR: (0x06:0x001C): Failed to map scatter gather list.
> (03:00.0 is the 3Ware Controller)
> 
> I've increased the 'swiotlb' kernel parameter in steps from 32, 64, 128, 256
> up to 512MB. The only effect is, that with 512MB my dom0 uses over 600MB of
> RAM (compared to ~50MB with XEN 3.0.4 on 32bit) and more fs-activity can be
> done before the error arises.
> I's very easy to reproduce, just do a 'dd if=/dev/zero of=/root/testfile'.
> After a short while (some seconds) the error arises.
> 
> The funny thing is: I've tried this dom0 on my workstation before installing
> it on a production server. My workstation has an nForce SATA controller and
> here I don't have any problems - even without setting the 'swiotlb' kernel
> paramter. 
> 
> It looks like there are some problems in conjunction with the 3Ware
> controller that have not been there on XEN 3.0.4 on 2.6.16 (32bit).
> 
> The complete piece of the logfile can be found here:
> http://nopaste.biz/21780
> And this is just from writing a 5MB initramfs to an ext2 filesytem (which
> had some corruption afterwards).
> 
> Thank's for your help!
> 
> Regards, 
> Markus Schuster
> 
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel

Re: Massive problems with 'PCI-DMA: Out of SW-IOMMU space for XXX bytes [..]' with 3Ware controller

[Markus Schuster]

Keir Fraser wrote:

Before I start testing, I have some questions:

> What happens if you run the old kernel (32-bit 2.6.16) on new Xen? 

I could simply exchange the Xen hypervisor on my old system, but
1. Will the old xend work with the new hypervisor? I don't want to further
touch the old (stable) system. An exchange of the hypervisor is not the
problem, as it's just one file, but the complete Xen userland...
2. Will a 32bit xend work with a 64bit hypervisor?


> What happens if you run a new (2.6.18) 32-bit kernel on new Xen? 

Ok, I'll have to do some compiling before I can say anything on this (I only
have 64bit packages (Xen and kernel)). 
And I have to install a test-32bit-dom0. 


> Or what about new 2.6.18 kernel on old Xen? 

You mean the 32bit version of 2.6.18? Have to do some compiling...

Regards, 
Markus Schuster

Re: Re: Massive problems with 'PCI-DMA: Out of SW-IOMMU space for XXX bytes [..]' with 3Ware controller

[Keir Fraser]

You have to run old Xen with old tools, and new Xen with new tools. However,
the dom0 kernels should be interchangeable (with the exception that old
64-bit Xen cannot run a 32-bit dom0).

 -- Keir

On 25/11/07 21:08, "Markus Schuster" <ma.schuster@gmx.de> wrote:

> Keir Fraser wrote:
> 
> Before I start testing, I have some questions:
> 
>> What happens if you run the old kernel (32-bit 2.6.16) on new Xen?
> 
> I could simply exchange the Xen hypervisor on my old system, but
> 1. Will the old xend work with the new hypervisor? I don't want to further
> touch the old (stable) system. An exchange of the hypervisor is not the
> problem, as it's just one file, but the complete Xen userland...
> 2. Will a 32bit xend work with a 64bit hypervisor?
> 
> 
>> What happens if you run a new (2.6.18) 32-bit kernel on new Xen?
> 
> Ok, I'll have to do some compiling before I can say anything on this (I only
> have 64bit packages (Xen and kernel)).
> And I have to install a test-32bit-dom0.
> 
> 
>> Or what about new 2.6.18 kernel on old Xen?
> 
> You mean the 32bit version of 2.6.18? Have to do some compiling...
> 
> Regards, 
> Markus Schuster
> 
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel

Re: Re: Massive problems with 'PCI-DMA: Out of SW-IOMMU space for XXX bytes [..]' with 3Ware controller

[Markus Schuster]

Keir Fraser wrote:
> You have to run old Xen with old tools, and new Xen with new tools.
> However, the dom0 kernels should be interchangeable (with the exception
> that old 64-bit Xen cannot run a 32-bit dom0).

OK, then I'll have to install a new 32bit dom0 for testing purposes. 
Compilation of the kernel for 32bit is currently running - Xen is the next
in the queue - but I will need some days to provide reasonable results. 

The good thing is: I'll have Xen 3.1.2 Debian packages for i386
afterwards :)

Regards, 
Markus

Re: Re: Massive problems with 'PCI-DMA: Out of SW-IOMMU space for XXX bytes [..]' with 3Ware controller

[Markus Schuster]

Hi, 

I finally have results for you:

The one you already know:
64bit Xen 3.1.2
64bit 2.6.18
=> FAIL

Here my tested results:
32bit Xen 3.1.2 PAE
32bit 2.6.18 PAE
=> FAIL

64bit Xen 3.1.2
32bit 2.6.18 PAE
=> FAIL

32bit Xen 3.1.2 NON-PAE
32bit 2.6.16 NON-PAE
=> OK (I've written over 15GB to disk without problems)

I had to use a NON-PAE version of Xen for testing my old working 2.6.16
kernel, 'cause it doesn't have PAE enabled (I thought it would have, but
well..). 
But all 3.1.2 Xen binaries have been built from the same source. 

One interesting fact: The server mainboard has some onboard SATA ports. If I
connect a hard disc to this ports, I can write to them without problems.
Just in conjunction with the 3Ware I suffer from this problem. 

The bug seems to be in the kernel part, not the hypervisor itself, as my old
kernel works without problems. 

Regards, 
Markus

Re: Re: Re: Massive problems with 'PCI-DMA: Out of SW-IOMMU space for XXX bytes [..]' with 3Ware controller

[Christopher S. Aker]

Markus Schuster wrote:
> The bug seems to be in the kernel part, not the hypervisor itself, as my old
> kernel works without problems. 

I hit this again over the weekend, causing massive file system damage to 
dom0.

I'd love to get this fixed so we can deploy 3.1.2 -- What can we do to 
help you guys further debug this?  I've got plenty of boxes with this 
hardware setup to test with.

Much appreciated,
-Chris

Re: Massive problems with 'PCI-DMA: Out of SW-IOMMU space for XXX bytes [..]' with 3Ware controller

[Christopher S. Aker]

Markus Schuster wrote:
> PCI-DMA: Out of SW-IOMMU space for 40960 bytes at device 0000:03:00.0
> 3w-9xxx: scsi0: ERROR: (0x06:0x001C): Failed to map scatter gather list.
> (03:00.0 is the 3Ware Controller)

Here's my "me too" data point.

I'm also seeing this on Xen 3.1.2, 64bit Xen, PAE Dom0 (2.6.18).  For 
me, it only happens if Dom0 is low on memory and into swap quite a bit. 
  Compiling a kernel with -j4 or so, with only 256M allocated to Dom0 
will trigger it.  I tried a number of swiotlb settings, none eliminate 
the problem.

-Chris

Re: Massive problems with 'PCI-DMA: Out of SW-IOMMU space for XXX bytes [..]' with 3Ware controller

[Randy McAnally]

Markus Schuster wrote:

> PCI-DMA: Out of SW-IOMMU space for 40960 bytes at device 0000:03:00.0
> 3w-9xxx: scsi0: ERROR: (0x06:0x001C): Failed to map scatter gather list.
> (03:00.0 is the 3Ware Controller)

Me too as well...any progress on this issue?

I notice that IO performance is notably slower than the stock RHEL kernel; 
about 1/2 overall write performance.  I can trigger the bug using DD or by 
copying disk images around.  All versions of Xen 3.1.x and 3.2.x are 
affected.

-- 
Randy McAnally
Fast Serv Networks, LLC
619-819-8252 x201
619-819-9443 fax

Re: Massive problems with 'PCI-DMA: Out of SW-IOMMU space for XXX bytes [..]' with 3Ware controller

[Markus Schuster]

Randy McAnally wrote:
>> PCI-DMA: Out of SW-IOMMU space for 40960 bytes at device 0000:03:00.0
>> 3w-9xxx: scsi0: ERROR: (0x06:0x001C): Failed to map scatter gather list.
>> (03:00.0 is the 3Ware Controller)
> 
> Me too as well...any progress on this issue?

I thought about writing to the mailing-list this days, too. Maybe there has
been some fix in 3.2.0 or 3.1.3?
Keir, do you have any news on this annoying issue?

Regards, Markus

Re: Re: Massive problems with 'PCI-DMA: Out of SW-IOMMU space for XXX bytes [..]' with 3Ware controller

[Markus Schuster]

Keir Fraser wrote:
> No, except to say that it is almost certainly a dom0 kernel issue, not
> core hypervisor. 

I agree on this. 


> So it will not be fixed by just changing hypervisor 
> version if you retain the same dom0 kernel version. 

I agree on this, too. My tests showed that you are right with your
assumption so far.
But i would have used the matching kernel patch, too. 


> What kernel are you using? 

I've tried Debians 2.6.18 patched for XEN 3.1.2 myself.


> Have you tried a vendor Xen kernel for your distro? 

That's the last thing I wanted to try but couldn't find time to reboot the
affected server to test it out. My intention was not to use a distro
kernel, as Debians 2.6.18 is patched for XEN 3.0.3. I could stay with my
current 3.0.4 system and I would be more recent. 

Regards, 
Markus

Re: Re: Re: Massive problems with 'PCI-DMA: Out of SW-IOMMU space for XXX bytes [..]' with 3Ware controller

[Keir Fraser]

On 29/1/08 22:38, "Markus Schuster" <ma.schuster@gmx.de> wrote:

>> What kernel are you using?
> 
> I've tried Debians 2.6.18 patched for XEN 3.1.2 myself.
> 
> 
>> Have you tried a vendor Xen kernel for your distro?
> 
> That's the last thing I wanted to try but couldn't find time to reboot the
> affected server to test it out. My intention was not to use a distro
> kernel, as Debians 2.6.18 is patched for XEN 3.0.3. I could stay with my
> current 3.0.4 system and I would be more recent.

Can you try our more up-to-date Linux kernel at:
 http://xenbits.xensource.com/linux-2.6.18-xen.hg

Yes, you have to pull it from mercurial and configure and build it yourself,
I'm afraid. But it would be useful to know whether this is something we have
fixed (knowingly or not) and simply not yet backported to 3.1 branch.

 -- Keir

Re: Massive problems with 'PCI-DMA: Out of SW-IOMMU space for XXX bytes [..]' with 3Ware controller

[Markus Schuster]

Keir Fraser wrote:
>>> Have you tried a vendor Xen kernel for your distro?
>> 
>> That's the last thing I wanted to try but couldn't find time to reboot
>> the affected server to test it out. My intention was not to use a distro
>> kernel, as Debians 2.6.18 is patched for XEN 3.0.3. I could stay with my
>> current 3.0.4 system and I would be more recent.
> 
> Can you try our more up-to-date Linux kernel at:
>  http://xenbits.xensource.com/linux-2.6.18-xen.hg
> 
> Yes, you have to pull it from mercurial and configure and build it
> yourself, I'm afraid. But it would be useful to know whether this is
> something we have fixed (knowingly or not) and simply not yet backported
> to 3.1 branch.

That's not the problem. I'm able to check out mercurial and compile my own
kernel from the source. Major problem is free time - you will know :)
But I will try if time permits. 

Regards, Markus

Re: Massive problems with 'PCI-DMA: Out of SW-IOMMU space for XXX bytes [..]' with 3Ware controller

[Hugo Rodenburg]

Keir Fraser wrote:
>>> Have you tried a vendor Xen kernel for your distro?
>> 
>> That's the last thing I wanted to try but couldn't find time to
reboot
>> the affected server to test it out. My intention was not to use a
distro
>> kernel, as Debians 2.6.18 is patched for XEN 3.0.3. I could stay with
my
>> current 3.0.4 system and I would be more recent.
> 
> Can you try our more up-to-date Linux kernel at:
>  http://xenbits.xensource.com/linux-2.6.18-xen.hg
> 
> Yes, you have to pull it from mercurial and configure and build it
> yourself, I'm afraid. But it would be useful to know whether this is
> something we have fixed (knowingly or not) and simply not yet
backported
> to 3.1 branch.

Hi Keir,

I Have the same problem as Markus describes
My system:
- dual quad xeon
- 8 GB RAM
- 3ware 9650se with 2 disks in raid 1
- ubuntu 7.10 server
- xen 3.2 and 2.6.18.8 kernel compiled from (xen) source

I tried using the kernel compiled from the mercurial repository. The
build was successful, but the problem stil exists. To support my "new"
controller I had to replace the source files (3w-9xxx.c/h) in the kernel
with the ones downloaded from the 3ware site. I don't think that's
causing the errors, but I just want to mention it as well.
After some tests (running: dd if=/dev/zero of=/tmp/test) the following
messages (still) appear in the syslog:

...
Mar  1 16:47:15 grs-psr35 kernel: [ 1822.161742] PCI-DMA: Out of
SW-IOMMU space for 16384 bytes at device 0000:06:00.0
Mar  1 16:47:15 grs-psr35 kernel: [ 1822.161802] 3w-9xxx: scsi0: ERROR:
(0x06:0x001C): Failed to map scatter gather list.
Mar  1 16:47:15 grs-psr35 kernel: [ 1822.161814] sd 0:0:0:0: SCSI error:
return code = 0x00070000
Mar  1 16:47:15 grs-psr35 kernel: [ 1822.161818] end_request: I/O error,
dev sda, sector 18365314
...

If you need more information, or would like me to test something, I'll
be happy to help.
Please respond to hugo@linuxbak.nl as well, since I'm not a member of
the mailing list.

Thanks for any help.