Massive problems with 'PCI-DMA: Out of SW-IOMMU space for XXX bytes [..]' with 3Ware controller

[Markus Schuster <ma.schuster@gmx.de>]

Hi list, 

this days I've tried to update a 32bit XEN 3.0.4 system (kernel 2.6.16) to
XEN 3.1.2 on AMD64 (kernel 2.6.18). The system has 2GB RAM and a 3ware
9500-12 SATA raid controller. 

The old system worked fine for about half a year now. 
The new system boots fine so far but when doing some "bigger" changes to the
filesystem I get a lot of this errors:
> PCI-DMA: Out of SW-IOMMU space for 40960 bytes at device 0000:03:00.0
> 3w-9xxx: scsi0: ERROR: (0x06:0x001C): Failed to map scatter gather list.
(03:00.0 is the 3Ware Controller)

I've increased the 'swiotlb' kernel parameter in steps from 32, 64, 128, 256
up to 512MB. The only effect is, that with 512MB my dom0 uses over 600MB of
RAM (compared to ~50MB with XEN 3.0.4 on 32bit) and more fs-activity can be
done before the error arises. 
I's very easy to reproduce, just do a 'dd if=/dev/zero of=/root/testfile'.
After a short while (some seconds) the error arises. 

The funny thing is: I've tried this dom0 on my workstation before installing
it on a production server. My workstation has an nForce SATA controller and
here I don't have any problems - even without setting the 'swiotlb' kernel
paramter. 

It looks like there are some problems in conjunction with the 3Ware
controller that have not been there on XEN 3.0.4 on 2.6.16 (32bit). 

The complete piece of the logfile can be found here:
http://nopaste.biz/21780
And this is just from writing a 5MB initramfs to an ext2 filesytem (which
had some corruption afterwards). 

Thank's for your help!

Regards, 
Markus Schuster