The "Out of IOMMU space" error and the "Please enable the IOMMU option" option

The "Out of IOMMU space" error and the "Please enable the IOMMU option" option

[Maurice Volaski]

Occasionally,  we get errors like these:

Jun  9 00:56:35 [kernel] [18020.416620] PCI-DMA: Out of IOMMU space 
for 12288 bytes at device 0000:03:01.0

and it looks like it might have something to do with these messages 
(after updating and setting BIOS as below):

Jun 15 16:38:54 [kernel] [    0.000000] Checking aperture...
Jun 15 16:38:54 [kernel] [    0.000000] CPU 0: aperture @ 0 size 256 MB
Jun 15 16:38:54 [kernel] [    0.000000] No AGP bridge found
Jun 15 16:38:54 [kernel] [    0.000000] Your BIOS doesn't leave a 
aperture memory hole
Jun 15 16:38:54 [kernel] [    0.000000] Please enable the IOMMU 
option in the BIOS setup
Jun 15 16:38:54 [kernel] [    0.000000] This costs you 64 MB of RAM
Jun 15 16:38:54 [kernel] [    0.000000] Mapping aperture over 65536 
KB of RAM @ 8000000

The box is a dual Opteron 250 with an Arima HDAMA Rev. D motherboard.

The updated BIOS options are (with * for current setting)
MTRR mapping method [Continuous, Discrete*]
Memhole mapping [Software, Hardware*]
4 GB memory hole adjust [Disabled, Manual, Auto*]
4 GB memory hole size [128 MB]
IOMMU [Disabled, Enabled*]
Size [multiple sizes, 256 MB*]

Initially, I was booting kernel 2.6.16.1 with iommu=merge, but the 
above message still occur when I booted without any iommu kernel 
parameters.

First, am I correct to assume that I'm not getting the 256 MB?

Second, is the the case the BIOS is lying?

Third, what could be a workaround to get enough memory for the PCI 
card that causes the out of space error (3Ware 8506)?
-- 

Maurice Volaski, mvolaski@aecom.yu.edu
Computing Support, Rose F. Kennedy Center
Albert Einstein College of Medicine of Yeshiva University

Re: The "Out of IOMMU space" error and the "Please enable the IOMMU option" option

[Andi Kleen]

On Thursday 15 June 2006 23:17, Maurice Volaski wrote:
> Occasionally,  we get errors like these:
> 
> Jun  9 00:56:35 [kernel] [18020.416620] PCI-DMA: Out of IOMMU space 
> for 12288 bytes at device 0000:03:01.0

 
> and it looks like it might have something to do with these messages 
> (after updating and setting BIOS as below):
> 
> Jun 15 16:38:54 [kernel] [    0.000000] Checking aperture...
> Jun 15 16:38:54 [kernel] [    0.000000] CPU 0: aperture @ 0 size 256 MB
> Jun 15 16:38:54 [kernel] [    0.000000] No AGP bridge found
> Jun 15 16:38:54 [kernel] [    0.000000] Your BIOS doesn't leave a 
> aperture memory hole
> Jun 15 16:38:54 [kernel] [    0.000000] Please enable the IOMMU 
> option in the BIOS setup
> Jun 15 16:38:54 [kernel] [    0.000000] This costs you 64 MB of RAM
> Jun 15 16:38:54 [kernel] [    0.000000] Mapping aperture over 65536 
> KB of RAM @ 8000000
> 
> The box is a dual Opteron 250 with an Arima HDAMA Rev. D motherboard.

Not directly - the kernel will create a fallback aperture if it can't
find one provided by the BIOS. The default is 64MB, but it can be
enlarged with iommu=memaper=4 

There was also a bug that might have caused this over longer uptimes. 
If you still see  it with latest 2.6.16.stable (19 or so). 

If it still happens with those two changes then something is likely
leaking mappings or your have a extreme workload that needs
a lot of mappings.


> First, am I correct to assume that I'm not getting the 256 MB?

Yes you'll get 64MB fallback likely.

> 
> Second, is the the case the BIOS is lying?

Probably yes.
 
-Andi

Re: The "Out of IOMMU space" error and the "Please enable the IOMMU option" option

[Maurice Volaski]

With 2.6.16.20 and iommu=memaper=4,

Jun 19 15:20:31 [kernel] [  103.697664] PCI-DMA: Disabling AGP.
Jun 19 15:20:31 [kernel] [  103.698226] PCI-DMA: using GART IOMMU.
Jun 19 15:20:31 [kernel] [  103.698341] PCI-DMA: Reserving 512MB of 
IOMMU area in the AGP aperture

I'm not sure why it tells me it's disabling AGP, but otherwise it looks good.

>On Thursday 15 June 2006 23:17, Maurice Volaski wrote:
>>  Occasionally,  we get errors like these:
>>
>>  Jun  9 00:56:35 [kernel] [18020.416620] PCI-DMA: Out of IOMMU space
>>  for 12288 bytes at device 0000:03:01.0
>
>
>>  and it looks like it might have something to do with these messages
>>  (after updating and setting BIOS as below):
>>
>>  Jun 15 16:38:54 [kernel] [    0.000000] Checking aperture...
>>  Jun 15 16:38:54 [kernel] [    0.000000] CPU 0: aperture @ 0 size 256 MB
>>  Jun 15 16:38:54 [kernel] [    0.000000] No AGP bridge found
>>  Jun 15 16:38:54 [kernel] [    0.000000] Your BIOS doesn't leave a
>>  aperture memory hole
>>  Jun 15 16:38:54 [kernel] [    0.000000] Please enable the IOMMU
>>  option in the BIOS setup
>>  Jun 15 16:38:54 [kernel] [    0.000000] This costs you 64 MB of RAM
>>  Jun 15 16:38:54 [kernel] [    0.000000] Mapping aperture over 65536
>>  KB of RAM @ 8000000
>>
>>  The box is a dual Opteron 250 with an Arima HDAMA Rev. D motherboard.
>
>Not directly - the kernel will create a fallback aperture if it can't
>find one provided by the BIOS. The default is 64MB, but it can be
>enlarged with iommu=memaper=4
>
>There was also a bug that might have caused this over longer uptimes.
>If you still see  it with latest 2.6.16.stable (19 or so).
>
>If it still happens with those two changes then something is likely
>leaking mappings or your have a extreme workload that needs
>a lot of mappings.
>
>
>>  First, am I correct to assume that I'm not getting the 256 MB?
>
>Yes you'll get 64MB fallback likely.
>
>>
>>  Second, is the the case the BIOS is lying?
>
>Probably yes.
>
>-Andi


-- 

Maurice Volaski, mvolaski@aecom.yu.edu
Computing Support, Rose F. Kennedy Center
Albert Einstein College of Medicine of Yeshiva University

Re: The "Out of IOMMU space" error and the "Please enable the IOMMU option" option

[Andi Kleen]

On Monday 19 June 2006 22:09, Maurice Volaski wrote:
> With 2.6.16.20 and iommu=memaper=4,
>
> Jun 19 15:20:31 [kernel] [  103.697664] PCI-DMA: Disabling AGP.
> Jun 19 15:20:31 [kernel] [  103.698226] PCI-DMA: using GART IOMMU.
> Jun 19 15:20:31 [kernel] [  103.698341] PCI-DMA: Reserving 512MB of
> IOMMU area in the AGP aperture
>
> I'm not sure why it tells me it's disabling AGP, but otherwise it looks
> good.

It does that when it can't find an AGP bridge (= no AGP slot). Many
modern systems only have PCI Express.

-andi