ioremap() fails for >64 MB

ioremap() fails for >64 MB

[Phil Nitschke]

Hi all,

I have 2 GB memory on a 7448 processor, and want to reserve a huge chunk
of it at boot-time, then ioremap() it into the kernel space inside a
device driver.  So far I've succeeded with 64 MB, but can't go any
higher, as mm/vmalloc.c tells me: "allocation failed: out of vmalloc
space - use vmalloc=<size> to increase size."

So I tried adding a vmalloc line to the kernel command line as follows:
Kernel cmd line: root=/dev/nfs rw mem=1920M vmalloc=1024M nfsroot=... 

After booting the processor, here is my memory arrangement:

bash-3.00# cat /proc/meminfo 
MemTotal:      1943232 kB
MemFree:       1910508 kB
...
HighTotal:     1179648 kB
HighFree:      1154608 kB
LowTotal:       763584 kB
LowFree:        755900 kB
...
VmallocTotal:   145024 kB
VmallocUsed:     65944 kB
VmallocChunk:    78972 kB

After inserting my device driver module (which ioremap()s 64 MB),
meminfo is as follows:

bash-3.00# cat /proc/meminfo 
MemTotal:      1943232 kB
MemFree:       1916512 kB
...
HighTotal:     1179648 kB
HighFree:      1160748 kB
LowTotal:       763584 kB
LowFree:        755764 kB
...
VmallocTotal:   145024 kB
VmallocUsed:    133568 kB
VmallocChunk:    10364 kB

So the vmalloc=<size> argument has made no difference.  What do I need
to do to make this work?

TIA,

-- 
Phil

bash-3.00# uname -a
Linux arty9 2.6.16-pmppc744x #211 Fri Aug 18 19:03:36 CST 2006 ppc ppc
ppc GNU/Linux

Re: ioremap() fails for >64 MB

[Alex Zeffertt]

Phil Nitschke wrote:
> Hi all,
> 
> I have 2 GB memory on a 7448 processor, and want to reserve a huge chunk
> of it at boot-time, then ioremap() it into the kernel space inside a
> device driver.  So far I've succeeded with 64 MB, but can't go any
> higher, as mm/vmalloc.c tells me: "allocation failed: out of vmalloc
> space - use vmalloc=<size> to increase size."
> 

I remember reading in Linux Device Drivers that you can use the bigphysarea
patch to allocate large memory, as long as you do it at boot time.  It seems
it's been ported to 2.6 too:

	http://lwn.net/Articles/111132/

Alex

Re: ioremap() fails for >64 MB

[David H. Lynch Jr.]

[-- Attachment #1: Type: text/plain, Size: 1550 bytes --]

    is ioremap() failing or is vmalloc failing ?

    ioremap should just assign a virtual address to a physical address -
does it actually allocate anything ?
    I beleive I am ioremap()ing a greater than 64MB Flash ROM and I do
not think it is failing.

Alex Zeffertt wrote:
> Phil Nitschke wrote:
>   
>> Hi all,
>>
>> I have 2 GB memory on a 7448 processor, and want to reserve a huge chunk
>> of it at boot-time, then ioremap() it into the kernel space inside a
>> device driver.  So far I've succeeded with 64 MB, but can't go any
>> higher, as mm/vmalloc.c tells me: "allocation failed: out of vmalloc
>> space - use vmalloc=<size> to increase size."
>>
>>     
>
> I remember reading in Linux Device Drivers that you can use the bigphysarea
> patch to allocate large memory, as long as you do it at boot time.  It seems
> it's been ported to 2.6 too:
>
> 	http://lwn.net/Articles/111132/
>
> Alex
> _______________________________________________
> Linuxppc-embedded mailing list
> Linuxppc-embedded@ozlabs.org
> https://ozlabs.org/mailman/listinfo/linuxppc-embedded
>   


-- 
Dave Lynch 					  	    DLA Systems
Software Development:  				         Embedded Linux
717.627.3770 	       dhlii@dlasys.net 	  http://www.dlasys.net
fax: 1.253.369.9244 			           Cell: 1.717.587.7774
Over 25 years' experience in platforms, languages, and technologies too numerous to list.

"Any intelligent fool can make things bigger and more complex... It takes a touch of genius - and a lot of courage to move in the opposite direction."
Albert Einstein


[-- Attachment #2: Type: text/html, Size: 2435 bytes --]

Re: ioremap() fails for >64 MB

[Matt Porter]

On Tue, Aug 22, 2006 at 05:05:01PM -0400, David H. Lynch Jr. wrote:
>     is ioremap() failing or is vmalloc failing ?
> 
>     ioremap should just assign a virtual address to a physical address -
> does it actually allocate anything ?
>     I beleive I am ioremap()ing a greater than 64MB Flash ROM and I do
> not think it is failing.

ioremap() allocates virtual address space in order to be able to
do the assignment.  The ability to allocate this vmalloc space
(which is used by ioremap() and vmalloc() calls) varies based
on amount of memory, etc. in a system.  It also depends on how
good of a quality of a board port is done. It's possible to do
some very stupid things that constrict availability of vmalloc
space. So YMMV versus others.

-Matt

Re: ioremap() fails for >64 MB

[Matt Porter]

On Tue, Aug 22, 2006 at 05:11:09PM +0930, Phil Nitschke wrote:
> Hi all,
> 
> I have 2 GB memory on a 7448 processor, and want to reserve a huge chunk
> of it at boot-time, then ioremap() it into the kernel space inside a
> device driver.  So far I've succeeded with 64 MB, but can't go any
> higher, as mm/vmalloc.c tells me: "allocation failed: out of vmalloc
> space - use vmalloc=<size> to increase size."
> 
> So I tried adding a vmalloc line to the kernel command line as follows:
> Kernel cmd line: root=/dev/nfs rw mem=1920M vmalloc=1024M nfsroot=... 
 
Yeah, that suggestion is bogus. That option can't help with getting
more vmalloc space in this case.

> So the vmalloc=<size> argument has made no difference.  What do I need
> to do to make this work?

Go to the "Advanced setup" menu. There's a number of options to provide
fine-grained control of the PPC kernel virtual address space. The key
here is that you have lots of RAM. By default 768MB will be mapped into
kernel lowmem space.  This is mapped from 0xc0000000-0xefffffff. You
probably have highmem on (I hope), so there is a highmem pool at
0xfe000000.  Your vmalloc space should start at 0xf1000000 (16 MB offset
past end of lowmem). Any io_block_map() calls will further constrain your
vmalloc space below the highmem pool. I imagine you have stuff 
permanently mapped that way down through about 0xf6000000 which would
leave just about around 64MB+ for vmalloc space.

The quick test is to modify the "Set maximum low memory" option to
0x20000000 (512MB). This should immediately give you and additional
256MB of vmalloc space as the vmalloc range will now start at
0xe1000000. If that works to allow a 128MB ioremap and you still need
much bigger ioremaps, you can also set "virtual address of kernel base"
down to 0xa0000000 or 0x80000000.

That said, why don't you just use alloc_bootmem() to reserve memory
for your driver at boot time? Then there's no need to bother with
ioremap for your driver then. Just save the pointer to your reserved
area somewhere and then use the space in your driver.  A lot of people
use this approach for unusual cases like this. If you need something
dynamic (i.e. with cmdline control) alloc, then bigphysarea is basically
a wrapper around bootmem allocation.

-Matt

Re: ioremap() fails for >64 MB

[Phil Nitschke]

On Tue, 2006-08-22 at 09:22 -0500, Matt Porter wrote:
> On Tue, Aug 22, 2006 at 05:11:09PM +0930, Phil Nitschke wrote:
> > Hi all,
> > 
> > I have 2 GB memory on a 7448 processor, and want to reserve a huge chunk
> > of it at boot-time, then ioremap() it into the kernel space inside a
> > device driver.  So far I've succeeded with 64 MB, but can't go any
> > higher, as mm/vmalloc.c tells me: "allocation failed: out of vmalloc
> > space - use vmalloc=<size> to increase size."
> > 
> > So I tried adding a vmalloc line to the kernel command line as follows:
> > Kernel cmd line: root=/dev/nfs rw mem=1920M vmalloc=1024M nfsroot=... 
>  
> Yeah, that suggestion is bogus. That option can't help with getting
> more vmalloc space in this case.
> 
> > So the vmalloc=<size> argument has made no difference.  What do I need
> > to do to make this work?
> 
> Go to the "Advanced setup" menu. There's a number of options to provide
> fine-grained control of the PPC kernel virtual address space.

<SNIP>

Thanks Matt (and others) for your suggestions.  Matt has given me the
answers I was looking for.

Since my (2 GB) memory is within the (4 GB) addressable by a 32-bit
processor, why do I need high memory at all?  

Are there performance implications on this platform from having a non
optimal low/high ratio?

> That said, why don't you just use alloc_bootmem() to reserve memory
> for your driver at boot time? 

I avoided this simply because I wanted to load/unload my driver (during
development), and alloc_bootmem() seemed better suited to drivers
compiled into the kernel.  But I'll look again at this idea if further
problems arise with the approach above.

Thanks again,

-- 
Phil

Re: ioremap() fails for >64 MB

[Dan Malek]

On Aug 23, 2006, at 6:00 AM, Phil Nitschke wrote:

> Since my (2 GB) memory is within the (4 GB) addressable by a 32-bit
> processor, why do I need high memory at all?

Because the processor used virtual memory addressing, and
we divide this 4G virtual space up among application, kernel,
IO mapping, etc.  The kernel can't map the whole 4G (or even
the 2G) physical space at one time.


> I avoided this simply because I wanted to load/unload my driver  
> (during
> development),

My personal preference is to debug drivers in a development
mode by building them into the kernel.  I find it easier to debug,
and you are also likely to be rebooting anyway :-)


	-- Dan

RE: ioremap() fails for >64 MB

[alva]

I think 64MB limitation of ioremap() is due to the kernel's page size.
When compiling kernel, it has an option to choose the memory page size
which is default 64MB. To use memory greater than 64MB, there is two
methods. One is to make the kernel's page size larger as Phil Nitschke
said. Another is to modify ioremap() a little bit --- just make it use
another file for "mmap" while larger than 64MB. Since the central
concept of linux is file-based, I think more files are not harmful that
only waste a little bit inode structure. And it is much more feasible
that one can choose to use file in memory or harddisk or mounted device
harddisk/memory ... ...

-----Original Message-----
From: linuxppc-embedded-bounces+vows_siu=yahoo.com.hk@ozlabs.org
[mailto:linuxppc-embedded-bounces+vows_siu=yahoo.com.hk@ozlabs.org] On
Behalf Of Phil Nitschke
Sent: Wednesday, August 23, 2006 6:01 PM
To: Matt Porter
Cc: linuxppc-embedded@ozlabs.org
Subject: Re: ioremap() fails for >64 MB

On Tue, 2006-08-22 at 09:22 -0500, Matt Porter wrote:
> On Tue, Aug 22, 2006 at 05:11:09PM +0930, Phil Nitschke wrote:
> > Hi all,
> > 
> > I have 2 GB memory on a 7448 processor, and want to reserve a huge
chunk
> > of it at boot-time, then ioremap() it into the kernel space inside a
> > device driver.  So far I've succeeded with 64 MB, but can't go any
> > higher, as mm/vmalloc.c tells me: "allocation failed: out of vmalloc
> > space - use vmalloc=<size> to increase size."
> > 
> > So I tried adding a vmalloc line to the kernel command line as
follows:
> > Kernel cmd line: root=/dev/nfs rw mem=1920M vmalloc=1024M
nfsroot=... 
>  
> Yeah, that suggestion is bogus. That option can't help with getting
> more vmalloc space in this case.
> 
> > So the vmalloc=<size> argument has made no difference.  What do I
need
> > to do to make this work?
> 
> Go to the "Advanced setup" menu. There's a number of options to
provide
> fine-grained control of the PPC kernel virtual address space.

<SNIP>

Thanks Matt (and others) for your suggestions.  Matt has given me the
answers I was looking for.

Since my (2 GB) memory is within the (4 GB) addressable by a 32-bit
processor, why do I need high memory at all?  

Are there performance implications on this platform from having a non
optimal low/high ratio?

> That said, why don't you just use alloc_bootmem() to reserve memory
> for your driver at boot time? 

I avoided this simply because I wanted to load/unload my driver (during
development), and alloc_bootmem() seemed better suited to drivers
compiled into the kernel.  But I'll look again at this idea if further
problems arise with the approach above.

Thanks again,

-- 
Phil

_______________________________________________
Linuxppc-embedded mailing list
Linuxppc-embedded@ozlabs.org
https://ozlabs.org/mailman/listinfo/linuxppc-embedded

-- 
No virus found in this incoming message.
Checked by AVG Free Edition.
Version: 7.1.405 / Virus Database: 268.11.5/425 - Release Date:
2006/8/22
 

-- 
No virus found in this outgoing message.
Checked by AVG Free Edition.
Version: 7.1.405 / Virus Database: 268.11.5/426 - Release Date:
2006/8/23
 

Re: ioremap() fails for >64 MB

[Matt Porter]

On Thu, Aug 24, 2006 at 10:54:23AM +0800, alva wrote:
> I think 64MB limitation of ioremap() is due to the kernel's page size.
> When compiling kernel, it has an option to choose the memory page size
> which is default 64MB. To use memory greater than 64MB, there is two
> methods. One is to make the kernel's page size larger as Phil Nitschke
> said. Another is to modify ioremap() a little bit --- just make it use
> another file for "mmap" while larger than 64MB. Since the central
> concept of linux is file-based, I think more files are not harmful that
> only waste a little bit inode structure. And it is much more feasible
> that one can choose to use file in memory or harddisk or mounted device
> harddisk/memory ... ...

This is incorrect. There is no "64MB page size". Unless you are ioremapping
a region overed by a BAT/CAM, the mapping is composed of default 4KB PTEs.
The reason it fails is because of what I explained earlier. Just wanted to
clarify so nobody was confused by these statements. We can map arbitrary
sizes to the limit of what the kernel virtual address space settings
allow. This varies from platform to platform.

-Matt

Re: ioremap() fails for >64 MB

[Matt Porter]

On Wed, Aug 23, 2006 at 07:30:37PM +0930, Phil Nitschke wrote:
> On Tue, 2006-08-22 at 09:22 -0500, Matt Porter wrote:
> > On Tue, Aug 22, 2006 at 05:11:09PM +0930, Phil Nitschke wrote:
> > > Hi all,
> > > 
> > > I have 2 GB memory on a 7448 processor, and want to reserve a huge chunk
> > > of it at boot-time, then ioremap() it into the kernel space inside a
> > > device driver.  So far I've succeeded with 64 MB, but can't go any
> > > higher, as mm/vmalloc.c tells me: "allocation failed: out of vmalloc
> > > space - use vmalloc=<size> to increase size."
> > > 
> > > So I tried adding a vmalloc line to the kernel command line as follows:
> > > Kernel cmd line: root=/dev/nfs rw mem=1920M vmalloc=1024M nfsroot=... 
> >  
> > Yeah, that suggestion is bogus. That option can't help with getting
> > more vmalloc space in this case.
> > 
> > > So the vmalloc=<size> argument has made no difference.  What do I need
> > > to do to make this work?
> > 
> > Go to the "Advanced setup" menu. There's a number of options to provide
> > fine-grained control of the PPC kernel virtual address space.
> 
> <SNIP>
> 
> Thanks Matt (and others) for your suggestions.  Matt has given me the
> answers I was looking for.
> 
> Since my (2 GB) memory is within the (4 GB) addressable by a 32-bit
> processor, why do I need high memory at all?  

Dan answered this one...
 
> Are there performance implications on this platform from having a non
> optimal low/high ratio?
 
It "depends". In PPC, we've always has kernelbase at 0xc0000000, however,
the dirty secret is that we still haven't have a 3:1 user:kernel split
like some other arches.  We've always had TASK_SIZE at 0x80000000 which
really results in a 1:1 split.  So, as long as you don't modify
TASK_SIZE, there's no chance of starving memory hungry user space
applications. There's already this waste space from 0x80000000-0xbfffffff
that was left due to legacy PReP reasons.

Even if you do modify TASK_SIZE down to something like 0x40000000,
for a 1:3 split, many embedded apps are perfectly happy with this
space, YMMV.

It's important that you understand how the userspace works at the
low level, plus VSIDs (on classic PPC), and the general concepts
of the virtual memory organization between kernel and userspace before
you mess with all these advanced options.  They are under advanced
options since they can easily make your system inoperable with the
improper settings.

> > That said, why don't you just use alloc_bootmem() to reserve memory
> > for your driver at boot time? 
> 
> I avoided this simply because I wanted to load/unload my driver (during
> development), and alloc_bootmem() seemed better suited to drivers
> compiled into the kernel.  But I'll look again at this idea if further
> problems arise with the approach above.

I see. It doesn't have any bearing on using a loadable module since
alloc_bootmem() is only called by your board port code. Your loadable
module just uses that reserved area and manages it on its own.

In any case, either approach will work.

-Matt