SWIOTLB allocation failed

SWIOTLB allocation failed

[Lukas Hejtmanek]

Hello,

I'm using Xen 3.1 with kernel 2.6.18. If I use swiotlb=force option to boot
domU, I got the following message:
xm create -c skirit60-1
Using config file "/etc/xen/skirit60-1".
Started domain skirit60-1
                         Bootdata ok (command line is root=/dev/sda1 ro
ip=147.251.11.44:1.2.3.4:147.251.11.1:255.255.255.128:skirit60-1:eth0:off
magrathea=eth1:10.0.0.1:10.0.0.101: swiotlb=force)
Linux version 2.6.18-xen31-smp (root@skirit60) (gcc version 4.0.2 20050901
(prerelease) (SUSE Linux)) #19 SMP Tue Jul 10 01:06:20 CEST 2007
BIOS-provided physical RAM map:
 Xen: 0000000000000000 - 000000009cc00000 (usable)
No mptable found.
Built 1 zonelists.  Total pages: 642048
Kernel command line: root=/dev/sda1 ro
ip=147.251.11.44:1.2.3.4:147.251.11.1:255.255.255.128:skirit60-1:eth0:off
magrathea=eth1:10.0.0.1:10.0.0.101: swiotlb=force
Initializing CPU#0
PID hash table entries: 4096 (order: 12, 32768 bytes)
Xen reported: 2992.500 MHz processor.
Console: colour dummy device 80x25
Dentry cache hash table entries: 524288 (order: 10, 4194304 bytes)
Inode-cache hash table entries: 262144 (order: 9, 2097152 bytes)
Kernel panic - not syncing: No suitable physical memory available for SWIOTLB
buffer!
Use dom0_mem Xen boot parameter to reserve
some DMA memory (e.g., dom0_mem=-128M).


is there something I could do to improve domU? Dom0 has only 256MB RAM, system
has 4GB RAM total. (DomU is assigned 2.5GB RAM.)

-- 
Lukáš Hejtmánek

Re: SWIOTLB allocation failed

[pradeep singh rautela]

[-- Attachment #1: Type: text/plain, Size: 1976 bytes --]

On Nov 27, 2007 7:23 PM, Lukas Hejtmanek <xhejtman@ics.muni.cz> wrote:
> Hello,
>
> I'm using Xen 3.1 with kernel 2.6.18. If I use swiotlb=force option to boot

Did you try using swiotlb=XYZ to solve your problem? XYZ is the amount
of the memory for swiotlb.
Did you get the same crash?

Thanks
> domU, I got the following message:
> xm create -c skirit60-1
> Using config file "/etc/xen/skirit60-1".
> Started domain skirit60-1
>                          Bootdata ok (command line is root=/dev/sda1 ro
> ip=147.251.11.44:1.2.3.4:147.251.11.1:255.255.255.128:skirit60-1:eth0:off
> magrathea=eth1:10.0.0.1:10.0.0.101: swiotlb=force)
> Linux version 2.6.18-xen31-smp (root@skirit60) (gcc version 4.0.2 20050901
> (prerelease) (SUSE Linux)) #19 SMP Tue Jul 10 01:06:20 CEST 2007
> BIOS-provided physical RAM map:
>  Xen: 0000000000000000 - 000000009cc00000 (usable)
> No mptable found.
> Built 1 zonelists.  Total pages: 642048
> Kernel command line: root=/dev/sda1 ro
> ip=147.251.11.44:1.2.3.4:147.251.11.1:255.255.255.128:skirit60-1:eth0:off
> magrathea=eth1:10.0.0.1:10.0.0.101: swiotlb=force
> Initializing CPU#0
> PID hash table entries: 4096 (order: 12, 32768 bytes)
> Xen reported: 2992.500 MHz processor.
> Console: colour dummy device 80x25
> Dentry cache hash table entries: 524288 (order: 10, 4194304 bytes)
> Inode-cache hash table entries: 262144 (order: 9, 2097152 bytes)
> Kernel panic - not syncing: No suitable physical memory available for SWIOTLB
> buffer!
> Use dom0_mem Xen boot parameter to reserve
> some DMA memory (e.g., dom0_mem=-128M).
>
>
> is there something I could do to improve domU? Dom0 has only 256MB RAM, system
> has 4GB RAM total. (DomU is assigned 2.5GB RAM.)
>
> --
> Lukáš Hejtmánek
>
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel
>



-- 
--
pradeep singh rautela

"question = ( to ) ? be : ! be;"
                -- Wm. Shakespeare

[-- Attachment #2: Type: text/plain, Size: 138 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Re: SWIOTLB allocation failed

[Lukas Hejtmanek]

On Tue, Nov 27, 2007 at 07:33:40PM +0530, pradeep singh rautela wrote:
> On Nov 27, 2007 7:23 PM, Lukas Hejtmanek <xhejtman@ics.muni.cz> wrote:
> > Hello,
> >
> > I'm using Xen 3.1 with kernel 2.6.18. If I use swiotlb=force option to boot
> 
> Did you try using swiotlb=XYZ to solve your problem? XYZ is the amount
> of the memory for swiotlb.
> Did you get the same crash?

if I use swiotlb=32000000 I got fatal DMA error and recommendation to use
swiotlb=force caused by mthca driver.

Fatal DMA error! Please use 'swiotlb=force'
----------- [cut here ] --------- [please bite here ] ---------


-- 
Lukáš Hejtmánek

Re: SWIOTLB allocation failed

[Keir Fraser]

Have you assigned any physical devices to this domU? If not then it is not
allowed to request contiguous multi-page allocations, and this will cause
swiotlb initialisation to fail (and hence panic the kernel). If you do not
assign any physical devices to the domU then there is no point in it having
a swiotlb anyway.

 -- Keir

On 27/11/07 13:53, "Lukas Hejtmanek" <xhejtman@ics.muni.cz> wrote:

> Hello,
> 
> I'm using Xen 3.1 with kernel 2.6.18. If I use swiotlb=force option to boot
> domU, I got the following message:
> xm create -c skirit60-1
> Using config file "/etc/xen/skirit60-1".
> Started domain skirit60-1
>                          Bootdata ok (command line is root=/dev/sda1 ro
> ip=147.251.11.44:1.2.3.4:147.251.11.1:255.255.255.128:skirit60-1:eth0:off
> magrathea=eth1:10.0.0.1:10.0.0.101: swiotlb=force)
> Linux version 2.6.18-xen31-smp (root@skirit60) (gcc version 4.0.2 20050901
> (prerelease) (SUSE Linux)) #19 SMP Tue Jul 10 01:06:20 CEST 2007
> BIOS-provided physical RAM map:
>  Xen: 0000000000000000 - 000000009cc00000 (usable)
> No mptable found.
> Built 1 zonelists.  Total pages: 642048
> Kernel command line: root=/dev/sda1 ro
> ip=147.251.11.44:1.2.3.4:147.251.11.1:255.255.255.128:skirit60-1:eth0:off
> magrathea=eth1:10.0.0.1:10.0.0.101: swiotlb=force
> Initializing CPU#0
> PID hash table entries: 4096 (order: 12, 32768 bytes)
> Xen reported: 2992.500 MHz processor.
> Console: colour dummy device 80x25
> Dentry cache hash table entries: 524288 (order: 10, 4194304 bytes)
> Inode-cache hash table entries: 262144 (order: 9, 2097152 bytes)
> Kernel panic - not syncing: No suitable physical memory available for SWIOTLB
> buffer!
> Use dom0_mem Xen boot parameter to reserve
> some DMA memory (e.g., dom0_mem=-128M).
> 
> 
> is there something I could do to improve domU? Dom0 has only 256MB RAM, system
> has 4GB RAM total. (DomU is assigned 2.5GB RAM.)

Re: SWIOTLB allocation failed

[Lukas Hejtmanek]

On Wed, Nov 28, 2007 at 03:32:45AM +0000, Keir Fraser wrote:
> Have you assigned any physical devices to this domU? If not then it is not
> allowed to request contiguous multi-page allocations, and this will cause
> swiotlb initialisation to fail (and hence panic the kernel). If you do not
> assign any physical devices to the domU then there is no point in it having
> a swiotlb anyway.

I do have a physical device - Mellanox Infiniband card. 

But it's my fault, DMA fatal error is caused by ipoib (IP over Infiniband)
module that uses incorrectly dma_sync_single on areas not mapped by
dma_map_single.

-- 
Lukáš Hejtmánek

Re: SWIOTLB allocation failed

[Lukas Hejtmanek]

On Tue, Nov 27, 2007 at 07:33:40PM +0530, pradeep singh rautela wrote:
> Did you try using swiotlb=XYZ to solve your problem? XYZ is the amount
> of the memory for swiotlb.
> Did you get the same crash?

I noticed that using swiotlb=30000000 probably does not work at all as I got
the following:
[  189.436069] Kernel command line: root=/dev/sda1 ro
ip=147.251.11.44:1.2.3.4:147.251.11.1:255.255.255.128:skirit60-1:eth0:off
magrathea=eth1:10.0.0.1:10.0.0.101: swiotlb=30000000 xencons=tty
[  189.436416] Initializing CPU#0
[  189.436656] PID hash table entries: 4096 (order: 12, 32768 bytes)
[  189.436695] Xen reported: 2992.498 MHz processor.
[  189.436721] Console: colour dummy device 80x25
[  189.438468] Dentry cache hash table entries: 524288 (order: 10, 4194304
bytes)
[  189.440459] Inode-cache hash table entries: 262144 (order: 9, 2097152
bytes)
[  189.440825] Software IO TLB disabled

-- 
Lukáš Hejtmánek

Re: SWIOTLB allocation failed

[Christopher S. Aker]

Lukas Hejtmanek wrote:
> On Tue, Nov 27, 2007 at 07:33:40PM +0530, pradeep singh rautela wrote:
>> Did you try using swiotlb=XYZ to solve your problem? XYZ is the amount
>> of the memory for swiotlb.
>> Did you get the same crash?
> 
> I noticed that using swiotlb=30000000 probably does not work at all as I got
> the following:

That would attempt to assign 30 million 128k pages, or roughly 3.5 TB of 
memory towards the bounce buffer.  I don't think you want that.

swiotlb defaults to 32 or 64 (I forget) -- try bumping it to 128 or 256.

-Chris

Re: SWIOTLB allocation failed

[Lukas Hejtmanek]

On Tue, Dec 04, 2007 at 09:53:53AM -0500, Christopher S. Aker wrote:
> That would attempt to assign 30 million 128k pages, or roughly 3.5 TB of 
> memory towards the bounce buffer.  I don't think you want that.
>
> swiotlb defaults to 32 or 64 (I forget) -- try bumping it to 128 or 256.

OK, but it still reports disabled:
[  162.852041] Kernel command line: root=/dev/sda1 ro ip=147.251.11.44:1.2.3.4:147.251.11.1:255.255.255.128:skirit60-1:eth0:off magrathea=eth1:10.0.0.1:10.0.0.101: swiotlb=256 xencons=tty
[  162.852392] Initializing CPU#0
[  162.852632] PID hash table entries: 4096 (order: 12, 32768 bytes)
[  162.852672] Xen reported: 2992.498 MHz processor.
[  162.852697] Console: colour dummy device 80x25
[  162.854447] Dentry cache hash table entries: 524288 (order: 10, 4194304 bytes)
[  162.856471] Inode-cache hash table entries: 262144 (order: 9, 2097152 bytes)
[  162.856816] Software IO TLB disabled

-- 
Lukáš Hejtmánek