Massive problems with 'PCI-DMA: Out of SW-IOMMU space for XXX bytes [..]' with 3Ware controller

Massive problems with 'PCI-DMA: Out of SW-IOMMU space for XXX bytes [..]' with 3Ware controller

[Markus Schuster]

Hi list, 

this days I've tried to update a 32bit XEN 3.0.4 system (kernel 2.6.16) to
XEN 3.1.2 on AMD64 (kernel 2.6.18). The system has 2GB RAM and a 3ware
9500-12 SATA raid controller. 

The old system worked fine for about half a year now. 
The new system boots fine so far but when doing some "bigger" changes to the
filesystem I get a lot of this errors:
> PCI-DMA: Out of SW-IOMMU space for 40960 bytes at device 0000:03:00.0
> 3w-9xxx: scsi0: ERROR: (0x06:0x001C): Failed to map scatter gather list.
(03:00.0 is the 3Ware Controller)

I've increased the 'swiotlb' kernel parameter in steps from 32, 64, 128, 256
up to 512MB. The only effect is, that with 512MB my dom0 uses over 600MB of
RAM (compared to ~50MB with XEN 3.0.4 on 32bit) and more fs-activity can be
done before the error arises. 
I's very easy to reproduce, just do a 'dd if=/dev/zero of=/root/testfile'.
After a short while (some seconds) the error arises. 

The funny thing is: I've tried this dom0 on my workstation before installing
it on a production server. My workstation has an nForce SATA controller and
here I don't have any problems - even without setting the 'swiotlb' kernel
paramter. 

It looks like there are some problems in conjunction with the 3Ware
controller that have not been there on XEN 3.0.4 on 2.6.16 (32bit). 

The complete piece of the logfile can be found here:
http://nopaste.biz/21780
And this is just from writing a 5MB initramfs to an ext2 filesytem (which
had some corruption afterwards). 

Thank's for your help!

Regards, 
Markus Schuster

Re: Massive problems with 'PCI-DMA: Out of SW-IOMMU space for XXX bytes [..]' with 3Ware controller

[Keir Fraser]

What happens if you run the old kernel (32-bit 2.6.16) on new Xen? What
happens if you run a new (2.6.18) 32-bit kernel on new Xen? Do these tests
without inflating the swiotlb size on the command line. Or what about new
2.6.18 kernel on old Xen?

(You can run 32-bit kernel on 64-bit Xen in 3.1.x, but not in 3.0.x, by the
way.)

 -- Keir

On 25/11/07 16:03, "Markus Schuster" <ma.schuster@gmx.de> wrote:

> Hi list, 
> 
> this days I've tried to update a 32bit XEN 3.0.4 system (kernel 2.6.16) to
> XEN 3.1.2 on AMD64 (kernel 2.6.18). The system has 2GB RAM and a 3ware
> 9500-12 SATA raid controller.
> 
> The old system worked fine for about half a year now.
> The new system boots fine so far but when doing some "bigger" changes to the
> filesystem I get a lot of this errors:
>> PCI-DMA: Out of SW-IOMMU space for 40960 bytes at device 0000:03:00.0
>> 3w-9xxx: scsi0: ERROR: (0x06:0x001C): Failed to map scatter gather list.
> (03:00.0 is the 3Ware Controller)
> 
> I've increased the 'swiotlb' kernel parameter in steps from 32, 64, 128, 256
> up to 512MB. The only effect is, that with 512MB my dom0 uses over 600MB of
> RAM (compared to ~50MB with XEN 3.0.4 on 32bit) and more fs-activity can be
> done before the error arises.
> I's very easy to reproduce, just do a 'dd if=/dev/zero of=/root/testfile'.
> After a short while (some seconds) the error arises.
> 
> The funny thing is: I've tried this dom0 on my workstation before installing
> it on a production server. My workstation has an nForce SATA controller and
> here I don't have any problems - even without setting the 'swiotlb' kernel
> paramter. 
> 
> It looks like there are some problems in conjunction with the 3Ware
> controller that have not been there on XEN 3.0.4 on 2.6.16 (32bit).
> 
> The complete piece of the logfile can be found here:
> http://nopaste.biz/21780
> And this is just from writing a 5MB initramfs to an ext2 filesytem (which
> had some corruption afterwards).
> 
> Thank's for your help!
> 
> Regards, 
> Markus Schuster
> 
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel

Re: Massive problems with 'PCI-DMA: Out of SW-IOMMU space for XXX bytes [..]' with 3Ware controller

[Markus Schuster]

Keir Fraser wrote:

Before I start testing, I have some questions:

> What happens if you run the old kernel (32-bit 2.6.16) on new Xen? 

I could simply exchange the Xen hypervisor on my old system, but
1. Will the old xend work with the new hypervisor? I don't want to further
touch the old (stable) system. An exchange of the hypervisor is not the
problem, as it's just one file, but the complete Xen userland...
2. Will a 32bit xend work with a 64bit hypervisor?


> What happens if you run a new (2.6.18) 32-bit kernel on new Xen? 

Ok, I'll have to do some compiling before I can say anything on this (I only
have 64bit packages (Xen and kernel)). 
And I have to install a test-32bit-dom0. 


> Or what about new 2.6.18 kernel on old Xen? 

You mean the 32bit version of 2.6.18? Have to do some compiling...

Regards, 
Markus Schuster

Re: Massive problems with 'PCI-DMA: Out of SW-IOMMU space for XXX bytes [..]' with 3Ware controller

[Christopher S. Aker]

Markus Schuster wrote:
> PCI-DMA: Out of SW-IOMMU space for 40960 bytes at device 0000:03:00.0
> 3w-9xxx: scsi0: ERROR: (0x06:0x001C): Failed to map scatter gather list.
> (03:00.0 is the 3Ware Controller)

Here's my "me too" data point.

I'm also seeing this on Xen 3.1.2, 64bit Xen, PAE Dom0 (2.6.18).  For 
me, it only happens if Dom0 is low on memory and into swap quite a bit. 
  Compiling a kernel with -j4 or so, with only 256M allocated to Dom0 
will trigger it.  I tried a number of swiotlb settings, none eliminate 
the problem.

-Chris

Re: Massive problems with 'PCI-DMA: Out of SW-IOMMU space for XXX bytes [..]' with 3Ware controller

[Randy McAnally]

Markus Schuster wrote:

> PCI-DMA: Out of SW-IOMMU space for 40960 bytes at device 0000:03:00.0
> 3w-9xxx: scsi0: ERROR: (0x06:0x001C): Failed to map scatter gather list.
> (03:00.0 is the 3Ware Controller)

Me too as well...any progress on this issue?

I notice that IO performance is notably slower than the stock RHEL kernel; 
about 1/2 overall write performance.  I can trigger the bug using DD or by 
copying disk images around.  All versions of Xen 3.1.x and 3.2.x are 
affected.

-- 
Randy McAnally
Fast Serv Networks, LLC
619-819-8252 x201
619-819-9443 fax

Re: Massive problems with 'PCI-DMA: Out of SW-IOMMU space for XXX bytes [..]' with 3Ware controller

[Markus Schuster]

Randy McAnally wrote:
>> PCI-DMA: Out of SW-IOMMU space for 40960 bytes at device 0000:03:00.0
>> 3w-9xxx: scsi0: ERROR: (0x06:0x001C): Failed to map scatter gather list.
>> (03:00.0 is the 3Ware Controller)
> 
> Me too as well...any progress on this issue?

I thought about writing to the mailing-list this days, too. Maybe there has
been some fix in 3.2.0 or 3.1.3?
Keir, do you have any news on this annoying issue?

Regards, Markus

Re: Re: Massive problems with 'PCI-DMA: Out of SW-IOMMU space for XXX bytes [..]' with 3Ware controller

[Keir Fraser]

On 28/1/08 21:33, "Markus Schuster" <ma.schuster@gmx.de> wrote:

> Randy McAnally wrote:
>>> PCI-DMA: Out of SW-IOMMU space for 40960 bytes at device 0000:03:00.0
>>> 3w-9xxx: scsi0: ERROR: (0x06:0x001C): Failed to map scatter gather list.
>>> (03:00.0 is the 3Ware Controller)
>> 
>> Me too as well...any progress on this issue?
> 
> I thought about writing to the mailing-list this days, too. Maybe there has
> been some fix in 3.2.0 or 3.1.3?
> Keir, do you have any news on this annoying issue?

No, except to say that it is almost certainly a dom0 kernel issue, not core
hypervisor. So it will not be fixed by just changing hypervisor version if
you retain the same dom0 kernel version. What kernel are you using? Have you
tried a vendor Xen kernel for your distro?

I think there have been reports of this problem with our 2.6.18-based
kernel, but noone has provided a patch to fix it as far as I know.

 -- Keir

localtime=1 broken on HVM guests

[Randy McAnally]

I have confirmed that the localtime option in guest config is broken in Xen 
3.1.3-rc2.  No matter what the setting, UTC is always given to the guest.

I have tried putting the HC in UTC and in Localtime, it makes no 
difference.  Aside from setting the system clock to the wrong time (or 
defining an offset in guest config), it is impossible to send localtime to 
the guest.

Is there anyone who has the knowledge to look at this lingering bug and 
finally squash it?

-- 
Randy McAnally
Fast Serv Networks, LLC
619-819-8252 x201
619-819-9443 fax

Re: localtime=1 broken on HVM guests

[Keir Fraser]

With current xen-3.1-testing tip (which is basically not much different from
-rc2) I find the localtime option works fine for me. What I did was change
my dom0 timezone to US Pacific, then create a domU with 'localtime=1' in its
config file, and then create a domU with no localtime config. The former
showed output in 'date' 8 hours behind the 'date' output of the latter. The
guests were created using the usual 'xm create <config file>' method.

Are your dom0 user-space tools definitely up to date?

 -- Keir

On 28/1/08 22:24, "Randy McAnally" <rsm@fast-serv.com> wrote:

> I have confirmed that the localtime option in guest config is broken in Xen
> 3.1.3-rc2.  No matter what the setting, UTC is always given to the guest.
> 
> I have tried putting the HC in UTC and in Localtime, it makes no
> difference.  Aside from setting the system clock to the wrong time (or
> defining an offset in guest config), it is impossible to send localtime to
> the guest.
> 
> Is there anyone who has the knowledge to look at this lingering bug and
> finally squash it?

Re: Re: Massive problems with 'PCI-DMA: Out of SW-IOMMU space for XXX bytes [..]' with 3Ware controller

[Markus Schuster]

Keir Fraser wrote:
> No, except to say that it is almost certainly a dom0 kernel issue, not
> core hypervisor. 

I agree on this. 


> So it will not be fixed by just changing hypervisor 
> version if you retain the same dom0 kernel version. 

I agree on this, too. My tests showed that you are right with your
assumption so far.
But i would have used the matching kernel patch, too. 


> What kernel are you using? 

I've tried Debians 2.6.18 patched for XEN 3.1.2 myself.


> Have you tried a vendor Xen kernel for your distro? 

That's the last thing I wanted to try but couldn't find time to reboot the
affected server to test it out. My intention was not to use a distro
kernel, as Debians 2.6.18 is patched for XEN 3.0.3. I could stay with my
current 3.0.4 system and I would be more recent. 

Regards, 
Markus

Re: Re: Re: Massive problems with 'PCI-DMA: Out of SW-IOMMU space for XXX bytes [..]' with 3Ware controller

[Keir Fraser]

On 29/1/08 22:38, "Markus Schuster" <ma.schuster@gmx.de> wrote:

>> What kernel are you using?
> 
> I've tried Debians 2.6.18 patched for XEN 3.1.2 myself.
> 
> 
>> Have you tried a vendor Xen kernel for your distro?
> 
> That's the last thing I wanted to try but couldn't find time to reboot the
> affected server to test it out. My intention was not to use a distro
> kernel, as Debians 2.6.18 is patched for XEN 3.0.3. I could stay with my
> current 3.0.4 system and I would be more recent.

Can you try our more up-to-date Linux kernel at:
 http://xenbits.xensource.com/linux-2.6.18-xen.hg

Yes, you have to pull it from mercurial and configure and build it yourself,
I'm afraid. But it would be useful to know whether this is something we have
fixed (knowingly or not) and simply not yet backported to 3.1 branch.

 -- Keir

Re: Massive problems with 'PCI-DMA: Out of SW-IOMMU space for XXX bytes [..]' with 3Ware controller

[Markus Schuster]

Keir Fraser wrote:
>>> Have you tried a vendor Xen kernel for your distro?
>> 
>> That's the last thing I wanted to try but couldn't find time to reboot
>> the affected server to test it out. My intention was not to use a distro
>> kernel, as Debians 2.6.18 is patched for XEN 3.0.3. I could stay with my
>> current 3.0.4 system and I would be more recent.
> 
> Can you try our more up-to-date Linux kernel at:
>  http://xenbits.xensource.com/linux-2.6.18-xen.hg
> 
> Yes, you have to pull it from mercurial and configure and build it
> yourself, I'm afraid. But it would be useful to know whether this is
> something we have fixed (knowingly or not) and simply not yet backported
> to 3.1 branch.

That's not the problem. I'm able to check out mercurial and compile my own
kernel from the source. Major problem is free time - you will know :)
But I will try if time permits. 

Regards, Markus

Re: Massive problems with 'PCI-DMA: Out of SW-IOMMU space for XXX bytes [..]' with 3Ware controller

[Hugo Rodenburg]

Keir Fraser wrote:
>>> Have you tried a vendor Xen kernel for your distro?
>> 
>> That's the last thing I wanted to try but couldn't find time to
reboot
>> the affected server to test it out. My intention was not to use a
distro
>> kernel, as Debians 2.6.18 is patched for XEN 3.0.3. I could stay with
my
>> current 3.0.4 system and I would be more recent.
> 
> Can you try our more up-to-date Linux kernel at:
>  http://xenbits.xensource.com/linux-2.6.18-xen.hg
> 
> Yes, you have to pull it from mercurial and configure and build it
> yourself, I'm afraid. But it would be useful to know whether this is
> something we have fixed (knowingly or not) and simply not yet
backported
> to 3.1 branch.

Hi Keir,

I Have the same problem as Markus describes
My system:
- dual quad xeon
- 8 GB RAM
- 3ware 9650se with 2 disks in raid 1
- ubuntu 7.10 server
- xen 3.2 and 2.6.18.8 kernel compiled from (xen) source

I tried using the kernel compiled from the mercurial repository. The
build was successful, but the problem stil exists. To support my "new"
controller I had to replace the source files (3w-9xxx.c/h) in the kernel
with the ones downloaded from the 3ware site. I don't think that's
causing the errors, but I just want to mention it as well.
After some tests (running: dd if=/dev/zero of=/tmp/test) the following
messages (still) appear in the syslog:

...
Mar  1 16:47:15 grs-psr35 kernel: [ 1822.161742] PCI-DMA: Out of
SW-IOMMU space for 16384 bytes at device 0000:06:00.0
Mar  1 16:47:15 grs-psr35 kernel: [ 1822.161802] 3w-9xxx: scsi0: ERROR:
(0x06:0x001C): Failed to map scatter gather list.
Mar  1 16:47:15 grs-psr35 kernel: [ 1822.161814] sd 0:0:0:0: SCSI error:
return code = 0x00070000
Mar  1 16:47:15 grs-psr35 kernel: [ 1822.161818] end_request: I/O error,
dev sda, sector 18365314
...

If you need more information, or would like me to test something, I'll
be happy to help.
Please respond to hugo@linuxbak.nl as well, since I'm not a member of
the mailing list.

Thanks for any help.