Re: Massive problems with 'PCI-DMA: Out of SW-IOMMU space for XXX bytes [..]' with 3Ware controller

Re: Massive problems with 'PCI-DMA: Out of SW-IOMMU space for XXX bytes [..]' with 3Ware controller

[Randy McAnally]

Markus Schuster wrote:

> PCI-DMA: Out of SW-IOMMU space for 40960 bytes at device 0000:03:00.0
> 3w-9xxx: scsi0: ERROR: (0x06:0x001C): Failed to map scatter gather list.
> (03:00.0 is the 3Ware Controller)

Me too as well...any progress on this issue?

I notice that IO performance is notably slower than the stock RHEL kernel; 
about 1/2 overall write performance.  I can trigger the bug using DD or by 
copying disk images around.  All versions of Xen 3.1.x and 3.2.x are 
affected.

-- 
Randy McAnally
Fast Serv Networks, LLC
619-819-8252 x201
619-819-9443 fax

Re: Massive problems with 'PCI-DMA: Out of SW-IOMMU space for XXX bytes [..]' with 3Ware controller

[Markus Schuster]

Randy McAnally wrote:
>> PCI-DMA: Out of SW-IOMMU space for 40960 bytes at device 0000:03:00.0
>> 3w-9xxx: scsi0: ERROR: (0x06:0x001C): Failed to map scatter gather list.
>> (03:00.0 is the 3Ware Controller)
> 
> Me too as well...any progress on this issue?

I thought about writing to the mailing-list this days, too. Maybe there has
been some fix in 3.2.0 or 3.1.3?
Keir, do you have any news on this annoying issue?

Regards, Markus

Re: Re: Massive problems with 'PCI-DMA: Out of SW-IOMMU space for XXX bytes [..]' with 3Ware controller

[Keir Fraser]

On 28/1/08 21:33, "Markus Schuster" <ma.schuster@gmx.de> wrote:

> Randy McAnally wrote:
>>> PCI-DMA: Out of SW-IOMMU space for 40960 bytes at device 0000:03:00.0
>>> 3w-9xxx: scsi0: ERROR: (0x06:0x001C): Failed to map scatter gather list.
>>> (03:00.0 is the 3Ware Controller)
>> 
>> Me too as well...any progress on this issue?
> 
> I thought about writing to the mailing-list this days, too. Maybe there has
> been some fix in 3.2.0 or 3.1.3?
> Keir, do you have any news on this annoying issue?

No, except to say that it is almost certainly a dom0 kernel issue, not core
hypervisor. So it will not be fixed by just changing hypervisor version if
you retain the same dom0 kernel version. What kernel are you using? Have you
tried a vendor Xen kernel for your distro?

I think there have been reports of this problem with our 2.6.18-based
kernel, but noone has provided a patch to fix it as far as I know.

 -- Keir

Re: Re: Massive problems with 'PCI-DMA: Out of SW-IOMMU space for XXX bytes [..]' with 3Ware controller

[Markus Schuster]

Keir Fraser wrote:
> No, except to say that it is almost certainly a dom0 kernel issue, not
> core hypervisor. 

I agree on this. 


> So it will not be fixed by just changing hypervisor 
> version if you retain the same dom0 kernel version. 

I agree on this, too. My tests showed that you are right with your
assumption so far.
But i would have used the matching kernel patch, too. 


> What kernel are you using? 

I've tried Debians 2.6.18 patched for XEN 3.1.2 myself.


> Have you tried a vendor Xen kernel for your distro? 

That's the last thing I wanted to try but couldn't find time to reboot the
affected server to test it out. My intention was not to use a distro
kernel, as Debians 2.6.18 is patched for XEN 3.0.3. I could stay with my
current 3.0.4 system and I would be more recent. 

Regards, 
Markus

Re: Re: Re: Massive problems with 'PCI-DMA: Out of SW-IOMMU space for XXX bytes [..]' with 3Ware controller

[Keir Fraser]

On 29/1/08 22:38, "Markus Schuster" <ma.schuster@gmx.de> wrote:

>> What kernel are you using?
> 
> I've tried Debians 2.6.18 patched for XEN 3.1.2 myself.
> 
> 
>> Have you tried a vendor Xen kernel for your distro?
> 
> That's the last thing I wanted to try but couldn't find time to reboot the
> affected server to test it out. My intention was not to use a distro
> kernel, as Debians 2.6.18 is patched for XEN 3.0.3. I could stay with my
> current 3.0.4 system and I would be more recent.

Can you try our more up-to-date Linux kernel at:
 http://xenbits.xensource.com/linux-2.6.18-xen.hg

Yes, you have to pull it from mercurial and configure and build it yourself,
I'm afraid. But it would be useful to know whether this is something we have
fixed (knowingly or not) and simply not yet backported to 3.1 branch.

 -- Keir

Re: Massive problems with 'PCI-DMA: Out of SW-IOMMU space for XXX bytes [..]' with 3Ware controller

[Markus Schuster]

Keir Fraser wrote:
>>> Have you tried a vendor Xen kernel for your distro?
>> 
>> That's the last thing I wanted to try but couldn't find time to reboot
>> the affected server to test it out. My intention was not to use a distro
>> kernel, as Debians 2.6.18 is patched for XEN 3.0.3. I could stay with my
>> current 3.0.4 system and I would be more recent.
> 
> Can you try our more up-to-date Linux kernel at:
>  http://xenbits.xensource.com/linux-2.6.18-xen.hg
> 
> Yes, you have to pull it from mercurial and configure and build it
> yourself, I'm afraid. But it would be useful to know whether this is
> something we have fixed (knowingly or not) and simply not yet backported
> to 3.1 branch.

That's not the problem. I'm able to check out mercurial and compile my own
kernel from the source. Major problem is free time - you will know :)
But I will try if time permits. 

Regards, Markus

localtime=1 broken on HVM guests

[Randy McAnally]

I have confirmed that the localtime option in guest config is broken in Xen 
3.1.3-rc2.  No matter what the setting, UTC is always given to the guest.

I have tried putting the HC in UTC and in Localtime, it makes no 
difference.  Aside from setting the system clock to the wrong time (or 
defining an offset in guest config), it is impossible to send localtime to 
the guest.

Is there anyone who has the knowledge to look at this lingering bug and 
finally squash it?

-- 
Randy McAnally
Fast Serv Networks, LLC
619-819-8252 x201
619-819-9443 fax

Re: localtime=1 broken on HVM guests

[Keir Fraser]

With current xen-3.1-testing tip (which is basically not much different from
-rc2) I find the localtime option works fine for me. What I did was change
my dom0 timezone to US Pacific, then create a domU with 'localtime=1' in its
config file, and then create a domU with no localtime config. The former
showed output in 'date' 8 hours behind the 'date' output of the latter. The
guests were created using the usual 'xm create <config file>' method.

Are your dom0 user-space tools definitely up to date?

 -- Keir

On 28/1/08 22:24, "Randy McAnally" <rsm@fast-serv.com> wrote:

> I have confirmed that the localtime option in guest config is broken in Xen
> 3.1.3-rc2.  No matter what the setting, UTC is always given to the guest.
> 
> I have tried putting the HC in UTC and in Localtime, it makes no
> difference.  Aside from setting the system clock to the wrong time (or
> defining an offset in guest config), it is impossible to send localtime to
> the guest.
> 
> Is there anyone who has the knowledge to look at this lingering bug and
> finally squash it?

Re: Massive problems with 'PCI-DMA: Out of SW-IOMMU space for XXX bytes [..]' with 3Ware controller

[Markus Schuster]

Keir Fraser wrote:

Before I start testing, I have some questions:

> What happens if you run the old kernel (32-bit 2.6.16) on new Xen? 

I could simply exchange the Xen hypervisor on my old system, but
1. Will the old xend work with the new hypervisor? I don't want to further
touch the old (stable) system. An exchange of the hypervisor is not the
problem, as it's just one file, but the complete Xen userland...
2. Will a 32bit xend work with a 64bit hypervisor?


> What happens if you run a new (2.6.18) 32-bit kernel on new Xen? 

Ok, I'll have to do some compiling before I can say anything on this (I only
have 64bit packages (Xen and kernel)). 
And I have to install a test-32bit-dom0. 


> Or what about new 2.6.18 kernel on old Xen? 

You mean the 32bit version of 2.6.18? Have to do some compiling...

Regards, 
Markus Schuster

Re: Re: Massive problems with 'PCI-DMA: Out of SW-IOMMU space for XXX bytes [..]' with 3Ware controller

[Keir Fraser]

You have to run old Xen with old tools, and new Xen with new tools. However,
the dom0 kernels should be interchangeable (with the exception that old
64-bit Xen cannot run a 32-bit dom0).

 -- Keir

On 25/11/07 21:08, "Markus Schuster" <ma.schuster@gmx.de> wrote:

> Keir Fraser wrote:
> 
> Before I start testing, I have some questions:
> 
>> What happens if you run the old kernel (32-bit 2.6.16) on new Xen?
> 
> I could simply exchange the Xen hypervisor on my old system, but
> 1. Will the old xend work with the new hypervisor? I don't want to further
> touch the old (stable) system. An exchange of the hypervisor is not the
> problem, as it's just one file, but the complete Xen userland...
> 2. Will a 32bit xend work with a 64bit hypervisor?
> 
> 
>> What happens if you run a new (2.6.18) 32-bit kernel on new Xen?
> 
> Ok, I'll have to do some compiling before I can say anything on this (I only
> have 64bit packages (Xen and kernel)).
> And I have to install a test-32bit-dom0.
> 
> 
>> Or what about new 2.6.18 kernel on old Xen?
> 
> You mean the 32bit version of 2.6.18? Have to do some compiling...
> 
> Regards, 
> Markus Schuster
> 
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel

Re: Re: Massive problems with 'PCI-DMA: Out of SW-IOMMU space for XXX bytes [..]' with 3Ware controller

[Markus Schuster]

Keir Fraser wrote:
> You have to run old Xen with old tools, and new Xen with new tools.
> However, the dom0 kernels should be interchangeable (with the exception
> that old 64-bit Xen cannot run a 32-bit dom0).

OK, then I'll have to install a new 32bit dom0 for testing purposes. 
Compilation of the kernel for 32bit is currently running - Xen is the next
in the queue - but I will need some days to provide reasonable results. 

The good thing is: I'll have Xen 3.1.2 Debian packages for i386
afterwards :)

Regards, 
Markus

Re: Re: Massive problems with 'PCI-DMA: Out of SW-IOMMU space for XXX bytes [..]' with 3Ware controller

[Markus Schuster]

Hi, 

I finally have results for you:

The one you already know:
64bit Xen 3.1.2
64bit 2.6.18
=> FAIL

Here my tested results:
32bit Xen 3.1.2 PAE
32bit 2.6.18 PAE
=> FAIL

64bit Xen 3.1.2
32bit 2.6.18 PAE
=> FAIL

32bit Xen 3.1.2 NON-PAE
32bit 2.6.16 NON-PAE
=> OK (I've written over 15GB to disk without problems)

I had to use a NON-PAE version of Xen for testing my old working 2.6.16
kernel, 'cause it doesn't have PAE enabled (I thought it would have, but
well..). 
But all 3.1.2 Xen binaries have been built from the same source. 

One interesting fact: The server mainboard has some onboard SATA ports. If I
connect a hard disc to this ports, I can write to them without problems.
Just in conjunction with the 3Ware I suffer from this problem. 

The bug seems to be in the kernel part, not the hypervisor itself, as my old
kernel works without problems. 

Regards, 
Markus