Re: Status of "Out of SW-IOMMU space" problem?

Re: Status of "Out of SW-IOMMU space" problem?

[Randy McAnally]

Keir Fraser wrote: 
> Can you please try the repository at 
> http://xenbits.xensource.com/xen-3.1-testing.hg ? 
> 
> The changeset 15705 probably fixes the issue. 
>   

It totally fixes the issue over here.  We were about to give up on Xen as 
all of our servers use 3ware cards...all is good now!

Re: Status of "Out of SW-IOMMU space" problem?

[Markus Schuster]

Randy McAnally wrote:
> Keir Fraser wrote:
>> Can you please try the repository at
>> http://xenbits.xensource.com/xen-3.1-testing.hg ?
>> 
>> The changeset 15705 probably fixes the issue.
>> 
> 
> It totally fixes the issue over here.  We were about to give up on Xen as
> all of our servers use 3ware cards...all is good now!

Randy, you've reminded me to give a status update on this problem back to
the list and to Keir. 
As Keir proposed a few month before, I've tried
http://xenbits.xensource.com/linux-2.6.18-xen.hg (I've used changeset 516).
And my server is working fine since that update for - looking for dom0
uptime - 58 days now. 
After reading your e-mail I know why: cs 15705 in xen-3.1-testing.hg equals
cs 501 in linux-2.6.18-xen.hg. 

Regards, 
Markus

Status of "Out of SW-IOMMU space" problem?

[Christopher S. Aker]

Is the "dma: avoid unnecessarily SWIOTLB bounce buffering" patch likely 
to fix the "Out of SW-IOMMU space" issue with 3ware cards and Xen, or 
any other patches for that matter?

I know some people have had success updating their 3ware card's firmware 
to latest, but that's not easily achievable for our situation.  So, 
we're stuck on 2.6.16.x for dom0 :(

Thanks,
-Chris

Re: Status of "Out of SW-IOMMU space" problem?

[Keir Fraser]

I'd be interested to know if it does. Unfortunately I have my doubts since
the patch fixes a page-boundary straddling issue, and if that were the root
problem it's unlikely to be affected by controller firmware updates.

 -- Keir

On 31/3/08 18:23, "Christopher S. Aker" <caker@theshore.net> wrote:

> Is the "dma: avoid unnecessarily SWIOTLB bounce buffering" patch likely
> to fix the "Out of SW-IOMMU space" issue with 3ware cards and Xen, or
> any other patches for that matter?
> 
> I know some people have had success updating their 3ware card's firmware
> to latest, but that's not easily achievable for our situation.  So,
> we're stuck on 2.6.16.x for dom0 :(
> 
> Thanks,
> -Chris
> 
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel

Re: Status of "Out of SW-IOMMU space" problem?

[Stephan Seitz]

[-- Attachment #1.1.1: Type: text/plain, Size: 1613 bytes --]

No, firmware updates doesn't solve this issue.
At least not for us. I updated different 7000 and 8000 series controller
without any success.
But I'm also interested if there's some patch around for this 'out of sw-iommu'
error ;)

Stephan


Keir Fraser schrieb:
> I'd be interested to know if it does. Unfortunately I have my doubts since
> the patch fixes a page-boundary straddling issue, and if that were the root
> problem it's unlikely to be affected by controller firmware updates.
> 
>  -- Keir
> 
> On 31/3/08 18:23, "Christopher S. Aker" <caker@theshore.net> wrote:
> 
>> Is the "dma: avoid unnecessarily SWIOTLB bounce buffering" patch likely
>> to fix the "Out of SW-IOMMU space" issue with 3ware cards and Xen, or
>> any other patches for that matter?
>>
>> I know some people have had success updating their 3ware card's firmware
>> to latest, but that's not easily achievable for our situation.  So,
>> we're stuck on 2.6.16.x for dom0 :(
>>
>> Thanks,
>> -Chris
>>
>>
>> _______________________________________________
>> Xen-devel mailing list
>> Xen-devel@lists.xensource.com
>> http://lists.xensource.com/xen-devel
> 
> 
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel


-- 
Stephan Seitz
Senior System Administrator

*netz-haut* e.K.
multimediale kommunikation

zweierweg 22
97074 würzburg

fon: +49 931 2876247
fax: +49 931 2876248

web: www.netz-haut.de <http://www.netz-haut.de/>

registriergericht: amtsgericht würzburg, hra 5054

[-- Attachment #1.1.2: s_seitz.vcf --]
[-- Type: text/x-vcard, Size: 335 bytes --]

begin:vcard
fn:Stephan Seitz
n:Seitz;Stephan
org:netz-haut e.K.
adr:;;Zweierweg 22;Wuerzburg;Bayern;97074;Deutschland
email;internet:s.seitz@netz-haut.de
title:Senior System Administrator
tel;work:+49-931-287-6247
tel;fax:+49-931-287-6248
x-mozilla-html:FALSE
url:http://www.schwarz-mode.de/files/
version:2.1
end:vcard


[-- Attachment #1.2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 252 bytes --]

[-- Attachment #2: Type: text/plain, Size: 138 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Re: Status of "Out of SW-IOMMU space" problem?

[Keir Fraser]

Well then, you should try the patch you originally posted about. :-)

 -- Keir

On 1/4/08 22:04, "Stephan Seitz" <s.seitz@netz-haut.de> wrote:

> No, firmware updates doesn't solve this issue.
> At least not for us. I updated different 7000 and 8000 series controller
> without any success.
> But I'm also interested if there's some patch around for this 'out of
> sw-iommu'
> error ;)
> 
> Stephan
> 
> 
> Keir Fraser schrieb:
>> I'd be interested to know if it does. Unfortunately I have my doubts since
>> the patch fixes a page-boundary straddling issue, and if that were the root
>> problem it's unlikely to be affected by controller firmware updates.
>> 
>>  -- Keir
>> 
>> On 31/3/08 18:23, "Christopher S. Aker" <caker@theshore.net> wrote:
>> 
>>> Is the "dma: avoid unnecessarily SWIOTLB bounce buffering" patch likely
>>> to fix the "Out of SW-IOMMU space" issue with 3ware cards and Xen, or
>>> any other patches for that matter?
>>> 
>>> I know some people have had success updating their 3ware card's firmware
>>> to latest, but that's not easily achievable for our situation.  So,
>>> we're stuck on 2.6.16.x for dom0 :(
>>> 
>>> Thanks,
>>> -Chris
>>> 
>>> 
>>> _______________________________________________
>>> Xen-devel mailing list
>>> Xen-devel@lists.xensource.com
>>> http://lists.xensource.com/xen-devel
>> 
>> 
>> 
>> _______________________________________________
>> Xen-devel mailing list
>> Xen-devel@lists.xensource.com
>> http://lists.xensource.com/xen-devel
> 

Re: Status of "Out of SW-IOMMU space" problem?

[Philippe Vanhaesendonck]

Well, I have to add 2 eurocents on this issue ;)

I am struggling with this since months, but I could not do too much, 
having all servers in production. I eventually managed to setup a test 
server, so I hope I can progress on this.

As you say, latest firmware update does not help (to be correct, i have 
the *feeling* it took me more time to reproduce, but it crashed anyway).

So far, I have the following observations:
- Xen 3.1.0 and the associated kernel (2.6.18) works (this is what we 
have on all our production servers)
- Any newer version starting from Xen 3.1.1 (again each time with the 
matching kernel) show the "Out of SW-IOMMU space" problem

In all cases, I am using the same 3ware 3w-9xxx driver from 3ware, as 
the 9650SE is not supported in the stock kernel. So the issue is coming 
from 'elsewhere'.

I had a look in what happened between 3.1.0 and 3.1.1, and we have a 
handful of changes in linux-2.6-xen-sparse/arch/i386/kernel/swiotlb.c (5 
changesets to be precise)

I'll try to spend some time in reverting these changes to see if it helps...

--
Philippe

Stephan Seitz wrote:
> No, firmware updates doesn't solve this issue.
> At least not for us. I updated different 7000 and 8000 series controller
> without any success.
> But I'm also interested if there's some patch around for this 'out of 
> sw-iommu'
> error ;)
>
> Stephan
>
>
> Keir Fraser schrieb:
>> I'd be interested to know if it does. Unfortunately I have my doubts 
>> since
>> the patch fixes a page-boundary straddling issue, and if that were 
>> the root
>> problem it's unlikely to be affected by controller firmware updates.
>>
>>  -- Keir
>>
>> On 31/3/08 18:23, "Christopher S. Aker" <caker@theshore.net> wrote:
>>
>>> Is the "dma: avoid unnecessarily SWIOTLB bounce buffering" patch likely
>>> to fix the "Out of SW-IOMMU space" issue with 3ware cards and Xen, or
>>> any other patches for that matter?
>>>
>>> I know some people have had success updating their 3ware card's 
>>> firmware
>>> to latest, but that's not easily achievable for our situation.  So,
>>> we're stuck on 2.6.16.x for dom0 :(
>>>

Re: Status of "Out of SW-IOMMU space" problem?

[Philippe Vanhaesendonck]

I didn't knew either -- thnx for this.

Unfortunately, it does not solve the SW-IOMMU space issue here -- ran my 
test suite and was able to reproduce.

--
Phil.

Stephan Seitz wrote:
> Hi,
>
> maybe everyone except me knows about this. I found a patch to the 3w-9xxx
> driver adressing this issue in the fedora 2.6.21-xen src.rpm
>
>

Re: Status of "Out of SW-IOMMU space" problem?

[Keir Fraser]

Please try with changeset 15705 from
http://xenbits.xensource.com/xen-3.1-testing.hg

 -- Keir

On 18/4/08 09:57, "Philippe Vanhaesendonck" <xen@e-bulles.be> wrote:

> I didn't knew either -- thnx for this.
> 
> Unfortunately, it does not solve the SW-IOMMU space issue here -- ran my
> test suite and was able to reproduce.
> 
> --
> Phil.
> 
> Stephan Seitz wrote:
>> Hi,
>> 
>> maybe everyone except me knows about this. I found a patch to the 3w-9xxx
>> driver adressing this issue in the fedora 2.6.21-xen src.rpm
>> 
>> 
> 
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel

Re: Status of "Out of SW-IOMMU space" problem?

[Philippe Vanhaesendonck]

Summary from the previous episodes: ;)

A) Xen 3.1.0 runs fine
B) Any newer version generates SW-IOMMU problem when Dom0 is under I/O 
stress.

C) Starting from there, I decided to build a Xen 3.1.3 / Linux 2.6.18.8 
kernel excluding all changes related to swiotlb.c.
That means: backout of changsets 15412, 15409, 15408, 15407 and 15405
This configuration runs fine as well!

Then, I re-applied changeset 15405, and the problem re-appears. So the 
issue is really linked to this changeset.
Looking at it more in detail, there is only one 'intrusive' change there:

swiotlb.c

    575         if (address_needs_mapping(hwdev, dev_addr)) {

becomes

    575         if
    (range_straddles_page_boundary(page_to_pseudophys(sg->page)
    576                           + sg->offset, sg->length)
    577             || address_needs_mapping(hwdev, dev_addr)) {

And indeed, if I remove the test on 'range_straddles_page_boundary', it 
runs fine, if I leave it, I get the SW-IOMMU problem.

To understand how big the impact of the 'range_straddles_page_boundary' 
test is, I added an else/printk and it gives:

    Apr 21 09:07:45 xenon kernel: SWIOTLB -
    range_straddles_page_boundary test bypassed
    Apr 21 09:07:58 xenon last message repeated 3629 times
    ...

so the impact is rather big: about 300 times per second...

This is almost as far as I can go on the topic. My knowledge on what's 
really happening in that part of the code is somewhat limited.
So I appreciate any help on how to progress this further.

(Side note: not sure what's happening here, looks like the mailing lists 
are broken...)

--
Phil.

Re: Status of "Out of SW-IOMMU space" problem?

[Keir Fraser]

On 21/4/08 09:02, "Philippe Vanhaesendonck" <xen@e-bulles.be> wrote:

> C) Starting from there, I decided to build a Xen 3.1.3 / Linux 2.6.18.8
> kernel excluding all changes related to swiotlb.c.
> That means: backout of changsets 15412, 15409, 15408, 15407 and 15405
> This configuration runs fine as well!
> 
> Then, I re-applied changeset 15405, and the problem re-appears. So the
> issue is really linked to this changeset.
> Looking at it more in detail, there is only one 'intrusive' change there:

Can you please try the repository at
http://xenbits.xensource.com/xen-3.1-testing.hg ?

The changeset 15705 probably fixes the issue.

 -- Keir

Re: Status of "Out of SW-IOMMU space" problem?

[Philippe Vanhaesendonck]

Keir Fraser wrote:
> Can you please try the repository at
> http://xenbits.xensource.com/xen-3.1-testing.hg ?
>
> The changeset 15705 probably fixes the issue.
>   
Keir,

I merged the 3.1.4-RC5 in my tree, recompiled the kernel and launched my 
test suite.
Tests are running for more than 18 hours now, without any problems (it 
usually crashes within the first 2 hours).

So I believe this problem is indeed fixed by changeset 15705 which is 
has a 'softer' approach for the range_straddles_page_boundary routine.

Than you very much

--
Philippe