non-contiguous physical memory

non-contiguous physical memory

[Yaron Presente]

Hi all,
I'm running montavista linux (2.4.18_mvl30-malta-mips_fp_le) on a board 
that has 2 memory banks of physical memory.
1. 32MB from physical address 0x00000000
2. 16MB from physical address 0x20000000

Currently I can only access the first bank (by add_memory_region(0, 32 
<< 20, BOOT_MEM_RAM) in prom_init() ).
I tried the obvious solution of adding another region at 0x20000000 
(add_memory_region(0x20000000, 16 << 20, BOOT_MEM_RAM))
but that didn't seem to work. I've also tried to add a BOOT_MEM_RESERVED 
region in between the regions in order to create a seemingly contiguous 
memory, with no success.
My questions are:
Is it possible to access the second bank as well under MIPS ?
Is there a way to define a "hole" in the physical memory?
Do I have to use CONFIG_DISCONTIGMEM ? is it fully supported ?
Thanks for your help,

-- 
Yaron Presente
MRV International
Direct   : 972-4-9936237
Fax      : 972-4-9890564
Email   : ypresente@mrv.com
www.mrv.com

Re: non-contiguous physical memory

[Ralf Baechle]

On Thu, Jun 24, 2004 at 03:46:43PM +0300, Yaron Presente wrote:

> I'm running montavista linux (2.4.18_mvl30-malta-mips_fp_le) on a board 
> that has 2 memory banks of physical memory.
> 1. 32MB from physical address 0x00000000
> 2. 16MB from physical address 0x20000000
> 
> Currently I can only access the first bank (by add_memory_region(0, 32 
> << 20, BOOT_MEM_RAM) in prom_init() ).
> I tried the obvious solution of adding another region at 0x20000000 
> (add_memory_region(0x20000000, 16 << 20, BOOT_MEM_RAM))
> but that didn't seem to work. I've also tried to add a BOOT_MEM_RESERVED 
> region in between the regions in order to create a seemingly contiguous 
> memory, with no success.
> My questions are:
> Is it possible to access the second bank as well under MIPS ?
> Is there a way to define a "hole" in the physical memory?
> Do I have to use CONFIG_DISCONTIGMEM ? is it fully supported ?
> Thanks for your help,

Your initial approach was nearly right - you can solve the problem of
holes in the memory map as long as they're small enough by only adding
the available regions with add_memory_region().  Typically uses for
this are small holes due to memory in use by firmware, for example.

Now, in your case the whole isn't small.  In fact, with 480MB it's big ;-)
What Linux will try to do is to allocate the mem_map array for the
entire memory range from 0x0 - 0x21000000, that's 528MB.  mem_map contains
one page per 4k page; each entry is 64 bytes in size for 32-bit kernels
so that makes a total size for mem_map[] of 8.25MB of which just 768kB are
actually being used.

Just to make life a little bit more misserable memory 32-bit kernels can
only use memory above the 512MB boundary as highmem.

CONFIG_DISCONTIGMEM can solve this problem - but Linux/MIPS really doesn't
much an attempt to make that easy to use.  Right now only a single MIPS
system is using CONFIG_DISCONTIGMEM and that system is using it in
combination with CONFIG_NUMA which is quite an additional complication.

With CONFIG_DISCONTIGMEM there will be no more mem_map[] array. Instead
there will be one such array for each memory region which means you'll
loose a bit of performance due to additional complexity but you'll save
all the wasted memory.

  Ralf

Re: non-contiguous physical memory

[Yaron Presente]

[-- Attachment #1: Type: text/plain, Size: 2887 bytes --]

Ralf,
Thanks for your help.
We've found out that the board can address more then 32MB in the lower 
bank, so for simplicity reasons,
I think that we just won't use the higher bank.
Yaron

Ralf Baechle wrote:

>On Thu, Jun 24, 2004 at 03:46:43PM +0300, Yaron Presente wrote:
>
>  
>
>>I'm running montavista linux (2.4.18_mvl30-malta-mips_fp_le) on a board 
>>that has 2 memory banks of physical memory.
>>1. 32MB from physical address 0x00000000
>>2. 16MB from physical address 0x20000000
>>
>>Currently I can only access the first bank (by add_memory_region(0, 32 
>><< 20, BOOT_MEM_RAM) in prom_init() ).
>>I tried the obvious solution of adding another region at 0x20000000 
>>(add_memory_region(0x20000000, 16 << 20, BOOT_MEM_RAM))
>>but that didn't seem to work. I've also tried to add a BOOT_MEM_RESERVED 
>>region in between the regions in order to create a seemingly contiguous 
>>memory, with no success.
>>My questions are:
>>Is it possible to access the second bank as well under MIPS ?
>>Is there a way to define a "hole" in the physical memory?
>>Do I have to use CONFIG_DISCONTIGMEM ? is it fully supported ?
>>Thanks for your help,
>>    
>>
>
>Your initial approach was nearly right - you can solve the problem of
>holes in the memory map as long as they're small enough by only adding
>the available regions with add_memory_region().  Typically uses for
>this are small holes due to memory in use by firmware, for example.
>
>Now, in your case the whole isn't small.  In fact, with 480MB it's big ;-)
>What Linux will try to do is to allocate the mem_map array for the
>entire memory range from 0x0 - 0x21000000, that's 528MB.  mem_map contains
>one page per 4k page; each entry is 64 bytes in size for 32-bit kernels
>so that makes a total size for mem_map[] of 8.25MB of which just 768kB are
>actually being used.
>
>Just to make life a little bit more misserable memory 32-bit kernels can
>only use memory above the 512MB boundary as highmem.
>
>CONFIG_DISCONTIGMEM can solve this problem - but Linux/MIPS really doesn't
>much an attempt to make that easy to use.  Right now only a single MIPS
>system is using CONFIG_DISCONTIGMEM and that system is using it in
>combination with CONFIG_NUMA which is quite an additional complication.
>
>With CONFIG_DISCONTIGMEM there will be no more mem_map[] array. Instead
>there will be one such array for each memory region which means you'll
>loose a bit of performance due to additional complexity but you'll save
>all the wasted memory.
>
>  Ralf
> This mail arrived via mail.mrv.com
>
> 
>************************************************************************************
>This footnote confirms that this email message has been scanned by
>PineApp Mail-SeCure for the presence of malicious code, vandals & computer viruses.
>************************************************************************************
>
>  
>




[-- Attachment #2: Type: text/html, Size: 3236 bytes --]

Non-contiguous physical memory

[Aaron Pace]

Hello,

I'm working on a design using a Freescale MPC8572 processor.
We are using 4 gigs of memory, and also need a window of 512 megs for
PCI-E devices.
What I have done is set up the first 2G of memory from 0x0 - 0x7f, the
PCI windows from 0x8 - 0x9f, localbus devices + CCSRBAR from 0xf -
0xffffffff, and the second 2G of ram from 0x1.0000.0000 -
0x1.8000.0000.
I've got this set up in U-boot (although it only uses the low mem),
but Linux will only use the first contiguous physical area (the
message is "Only using first contiguous memory region").
Is it possible to have multiple non-contiguous physical memory chunks
used for memory allocation?
If not, is there a better way to set this up without losing large
chunks of memory?

Thanks,
Aaron

Re: Non-contiguous physical memory

[Kumar Gala]

On Jan 21, 2009, at 9:18 AM, Aaron Pace wrote:

> Hello,
>
> I'm working on a design using a Freescale MPC8572 processor.
> We are using 4 gigs of memory, and also need a window of 512 megs for
> PCI-E devices.
> What I have done is set up the first 2G of memory from 0x0 - 0x7f, the
> PCI windows from 0x8 - 0x9f, localbus devices + CCSRBAR from 0xf -
> 0xffffffff, and the second 2G of ram from 0x1.0000.0000 -
> 0x1.8000.0000.
> I've got this set up in U-boot (although it only uses the low mem),
> but Linux will only use the first contiguous physical area (the
> message is "Only using first contiguous memory region").
> Is it possible to have multiple non-contiguous physical memory chunks
> used for memory allocation?
> If not, is there a better way to set this up without losing large
> chunks of memory?

Its possible in that Linux supports this.  However the PPC32 code does  
not exist and would need to be added to support non-contiguous memory  
ranges.

What exact PCI-E needs do you have?  Is PCI-E performance critical?   
Is (are) your pci device(s) 64-bit address capable?

I ask because depending on the answers doing straight 4G and PCI above  
that range might be sufficient for your needs.

- k

Re: Non-contiguous physical memory

[Aaron Pace]

>
> Its possible in that Linux supports this.  However the PPC32 code does not
> exist and would need to be added to support non-contiguous memory ranges.
>
> What exact PCI-E needs do you have?  Is PCI-E performance critical?  Is
> (are) your pci device(s) 64-bit address capable?
>
> I ask because depending on the answers doing straight 4G and PCI above that
> range might be sufficient for your needs.
>
> - k
>

Hello, thanks for responding.
The first take at this used your idea of mapping the PCI-E ranges
above the 4G boundary.
We have a localbus bridge device on the PCI-E bus that only supports
32-bit MMIO, but I believe we could satisfactorily work around that
problem using the ATMU registers.
However, one question that I had that I wasn't able to verify to my
satisfaction was whether I could ioremap a 36-bit physical address
when building for ppc32, as the PCI-E devices have memory ranges that
need to be accessible from userspace.  If that is possible, then this
may be the way to go, although it would still be nice not to lose the
128 meg for the localbus & CCSR region.

Thanks,
-Aaron

Re: Non-contiguous physical memory

[Kumar Gala]

On Jan 21, 2009, at 12:00 PM, Aaron Pace wrote:

>>
>> Its possible in that Linux supports this.  However the PPC32 code  
>> does not
>> exist and would need to be added to support non-contiguous memory  
>> ranges.
>>
>> What exact PCI-E needs do you have?  Is PCI-E performance  
>> critical?  Is
>> (are) your pci device(s) 64-bit address capable?
>>
>> I ask because depending on the answers doing straight 4G and PCI  
>> above that
>> range might be sufficient for your needs.
>>
>> - k
>>
>
> Hello, thanks for responding.
> The first take at this used your idea of mapping the PCI-E ranges
> above the 4G boundary.
> We have a localbus bridge device on the PCI-E bus that only supports
> 32-bit MMIO, but I believe we could satisfactorily work around that
> problem using the ATMU registers.
> However, one question that I had that I wasn't able to verify to my
> satisfaction was whether I could ioremap a 36-bit physical address
> when building for ppc32, as the PCI-E devices have memory ranges that
> need to be accessible from userspace.  If that is possible, then this
> may be the way to go, although it would still be nice not to lose the
> 128 meg for the localbus & CCSR region.

you can get 36-bit physical addresses but you need to enable "Large  
physical address support".

- k

Re: Non-contiguous physical memory

[Michele Pallaro]

Hello,
	I have a similar problem with my custom board CPU mpc8548 (E500V2) with
4Gbytes of RAM

	  0x0000_0000 --    0x7FFF_FFFF  2Gbytes
	0x1_0000_0000 -- 0x1_7FFF_FFFF   2Gbytes

I Enable the option PHYS_64BIT, and set in dts

memory {
                #address-cells = <2>;
                #size-cells = <1>;   
                device_type = "memory";
                reg = <00000000 00000000 00000000 7FFFFFFF
                       00000001 00000000 00000000 7FFFFFFF>;
        };  

But the kernel can see just 2Gbytes of memory.
How the kernel can see the other 2Gbytes ?

	Michele
	

On Wed, 2009-01-21 at 11:28 -0600, Kumar Gala wrote:
> On Jan 21, 2009, at 9:18 AM, Aaron Pace wrote:
> 
> > Hello,
> >
> > I'm working on a design using a Freescale MPC8572 processor.
> > We are using 4 gigs of memory, and also need a window of 512 megs for
> > PCI-E devices.
> > What I have done is set up the first 2G of memory from 0x0 - 0x7f, the
> > PCI windows from 0x8 - 0x9f, localbus devices + CCSRBAR from 0xf -
> > 0xffffffff, and the second 2G of ram from 0x1.0000.0000 -
> > 0x1.8000.0000.
> > I've got this set up in U-boot (although it only uses the low mem),
> > but Linux will only use the first contiguous physical area (the
> > message is "Only using first contiguous memory region").
> > Is it possible to have multiple non-contiguous physical memory chunks
> > used for memory allocation?
> > If not, is there a better way to set this up without losing large
> > chunks of memory?
> 
> Its possible in that Linux supports this.  However the PPC32 code does  
> not exist and would need to be added to support non-contiguous memory  
> ranges.
> 
> What exact PCI-E needs do you have?  Is PCI-E performance critical?   
> Is (are) your pci device(s) 64-bit address capable?
> 
> I ask because depending on the answers doing straight 4G and PCI above  
> that range might be sufficient for your needs.
> 
> - k
> _______________________________________________
> Linuxppc-dev mailing list
> Linuxppc-dev@ozlabs.org
> https://ozlabs.org/mailman/listinfo/linuxppc-dev

Re: Non-contiguous physical memory

[Kumar Gala]

On Jan 23, 2009, at 11:52 AM, Michele Pallaro wrote:

> Hello,
> 	I have a similar problem with my custom board CPU mpc8548 (E500V2)  
> with
> 4Gbytes of RAM
>
> 	  0x0000_0000 --    0x7FFF_FFFF  2Gbytes
> 	0x1_0000_0000 -- 0x1_7FFF_FFFF   2Gbytes
>
> I Enable the option PHYS_64BIT, and set in dts
>
> memory {
>                #address-cells = <2>;
>                #size-cells = <1>;
>                device_type = "memory";
>                reg = <00000000 00000000 00000000 7FFFFFFF
>                       00000001 00000000 00000000 7FFFFFFF>;
>        };
>
> But the kernel can see just 2Gbytes of memory.
> How the kernel can see the other 2Gbytes ?
>
> 	Michele

This should be:

reg = < 0x00000000 0x00000000 0x00000000 0x80000000
         0x00000001 0x00000000 0x00000000 0x80000000 >;

Its a start, size pair.

- k