Lots of memory on 826x

Lots of memory on 826x

[Rune Torgersen]

Anybody know how to get more than 512MB (1-2GB) of RAM to work with
2.6.11 on a 8265?

Weird stuff and crashes keep happening on kernel start (vmalloc out of
memory, and a pci driver ups) when I tell the kernel I have more than
512MB of ram.
The pci craches happens when I tell the kernel I have 740-760MB or more.

Any idea of where to start to look for this?

Rune Torgersen
System Developer
Innovative Systems LLC
1000 Innovative Drive
Mitchell, SD 57301
Ph: 605-995-6120
www.innovsys.com

Re: Lots of memory on 826x

[Kumar Gala]

I would start looking at what addresses are getting assigned for 
VMALLOC vs any io_block_mapping() calls.  I know I've had problems with 
large amount of memory in systems if the virt addrs allocated in 
io_block_mapping calls conflict with vmalloc addrs.

- kumar

On Mar 8, 2005, at 11:33 AM, Rune Torgersen wrote:

> Anybody know how to get more than 512MB (1-2GB) of RAM to work with
>  2.6.11 on a 8265?
>
> Weird stuff and crashes keep happening on kernel start (vmalloc out of
> memory, and a pci driver ups) when I tell the kernel I have more than
>  512MB of ram.
> The pci craches happens when I tell the kernel I have 740-760MB or 
> more.
>
> Any idea of where to start to look for this?
>
> Rune Torgersen
>  System Developer
>  Innovative Systems LLC
>  1000 Innovative Drive
>  Mitchell, SD 57301
>  Ph: 605-995-6120
>  www.innovsys.com
> _______________________________________________
> Linuxppc-embedded mailing list
>  Linuxppc-embedded@ozlabs.org
> https://ozlabs.org/mailman/listinfo/linuxppc-embedded

RE: Lots of memory on 826x

[Rune Torgersen]

Ok... Got a little closer....

I have PCI (outbound) mapped in the area 0x80000000-0x8fffffff
(prefetch/non-prefetch and IO)
IMMR of cource is at 0xf0000000

So I map the IO as following:
	/* Map IMMR region to a 256MB BAT */
	addr =3D (cpm2_immr !=3D NULL) ? (uint)cpm2_immr : CPM_MAP_ADDR;
	io_block_mapping(addr, addr, 0x10000000, _PAGE_IO);

	io_block_mapping(0x80000000, 0x80000000, 0x10000000, _PAGE_IO);

Now I can almost boot....:
It craches when trying to read from harddisk
(only happens with more than 512MB)

EXT3-fs: mounted filesystem with ordered data mode.
VFS: Mounted root (ext3 filesystem).
Freeing unused kernel memory: 288k init
Machine check in kernel mode.
Caused by (from SRR1=3D49030): Transfer error ack signal
Oops: machine check, sig: 7 [#1]
PREEMPT
NIP: C000E3FC LR: C0037728 SP: EFC11C50 REGS: efc11ba0 TRAP: 0200    Not
tainted
MSR: 00049030 EE: 1 PR: 0 FP: 0 ME: 1 IR/DR: 11
TASK =3D c0be9ae0[1] 'swapper' THREAD: efc10000
Last syscall: 11
GPR00: 00000001 EFC11C50 C0BE9AE0 00000024 FE402FFC 00000000 EFE939FC
81040008
GPR08: 8124000C C02A0000 3FF80581 00000004 00000092 100C1B1C EFE9E98C
EFEA6498
GPR16: 00000007 00000001 00000000 00000000 EFC11D18 EFC11DC8 00000000
EFE9E940
GPR24: 00000000 00000000 00000080 000074A0 C0B06000 FE403000 00000080
EFC11D18
NIP [c000e3fc] __copy_tofrom_user+0xb4/0x234
LR [c0037728] file_read_actor+0x148/0x258
Call trace:
 [c00373e4] do_generic_mapping_read+0x408/0x604
 [c0037a5c] __generic_file_aio_read+0x224/0x254
 [c0037ac8] generic_file_aio_read+0x3c/0x5c
 [c005ef84] do_sync_read+0xa0/0xe4
 [c005f090] vfs_read+0xc8/0x150
 [c006ccbc] kernel_read+0x40/0x58
 [c006d994] prepare_binprm+0xd0/0xec
 [c006e098] do_execve+0x138/0x21c
 [c0006bb8] sys_execve+0x98/0xf8
 [c0003c80] ret_from_syscall+0x0/0x44
 [c00063ac] execve+0x8/0x1c
 [c0003a98] init+0xb0/0xf8
 [c0006384] kernel_thread+0x44/0x60
Kernel panic - not syncing: Attempted to kill init!
 <0>Rebooting in 180 seconds..=20

> -----Original Message-----
> From: Kumar Gala [mailto:kumar.gala@freescale.com]=20
> Sent: Tuesday, March 08, 2005 12:07
> To: Rune Torgersen
> Cc: linuxppc-embedded@ozlabs.org
> Subject: Re: Lots of memory on 826x
>=20
> I would start looking at what addresses are getting assigned for=20
> VMALLOC vs any io_block_mapping() calls.  I know I've had=20
> problems with=20
> large amount of memory in systems if the virt addrs allocated in=20
> io_block_mapping calls conflict with vmalloc addrs.
>=20
> - kumar
>=20
> On Mar 8, 2005, at 11:33 AM, Rune Torgersen wrote:
>=20
> > Anybody know how to get more than 512MB (1-2GB) of RAM to work with
> >  2.6.11 on a 8265?
> >
> > Weird stuff and crashes keep happening on kernel start=20
> (vmalloc out of
> > memory, and a pci driver ups) when I tell the kernel I have=20
> more than
> >  512MB of ram.
> > The pci craches happens when I tell the kernel I have 740-760MB or=20
> > more.
> >
> > Any idea of where to start to look for this?
> >
> > Rune Torgersen
> >  System Developer
> >  Innovative Systems LLC
> >  1000 Innovative Drive
> >  Mitchell, SD 57301
> >  Ph: 605-995-6120
> >  www.innovsys.com
> > _______________________________________________
> > Linuxppc-embedded mailing list
> >  Linuxppc-embedded@ozlabs.org
> > https://ozlabs.org/mailman/listinfo/linuxppc-embedded
>=20
>=20
>=20

Re: Lots of memory on 826x

[Mark Chambers]

>Now I can almost boot....:
>It craches when trying to read from harddisk
>(only happens with more than 512MB)


Don't rule out a hardware problem, namely addressing to SDRAM.  It could be
consistent with what you're seeing - when you start to load a lot of data
and
cross some address you step on older data (or code).

--Mark C.

> > Anybody know how to get more than 512MB (1-2GB) of RAM to work with
> >  2.6.11 on a 8265?
> >
> > Weird stuff and crashes keep happening on kernel start
> (vmalloc out of
> > memory, and a pci driver ups) when I tell the kernel I have
> more than
> >  512MB of ram.
> > The pci craches happens when I tell the kernel I have 740-760MB or
> > more.
> >
> > Any idea of where to start to look for this?
> >

Re: Lots of memory on 826x

[Matt Porter]

On Tue, Mar 08, 2005 at 12:37:33PM -0600, Rune Torgersen wrote:
> Ok... Got a little closer....
> 
> I have PCI (outbound) mapped in the area 0x80000000-0x8fffffff
> (prefetch/non-prefetch and IO)
> IMMR of cource is at 0xf0000000
> 
> So I map the IO as following:
> 	/* Map IMMR region to a 256MB BAT */
> 	addr = (cpm2_immr != NULL) ? (uint)cpm2_immr : CPM_MAP_ADDR;
> 	io_block_mapping(addr, addr, 0x10000000, _PAGE_IO);
> 
> 	io_block_mapping(0x80000000, 0x80000000, 0x10000000, _PAGE_IO);
> 
> Now I can almost boot....:
> It craches when trying to read from harddisk
> (only happens with more than 512MB)

That's expected if you are going to 768MB. You have kernel lowmem
being mapped at 0xc0000000+<size_of_system_memory>. With 768MB that
takes you all the way to where you have the IMMR mapped 1:1 I
guess. This leaves no room for vmalloc space.  With 768MB of ram
the kernel will calculate VMALLOC_START at 0xf1000000 which is a
problem.

A few options are:

* Move the IMMR much higher to allow vmalloc space
* Modify maximum low memory to limit to 512MB using the advanced
  options menu
* Modify KERNELBASE to something like 0xa0000000 using the advanced
  options menu

All of these methods will provide addition vmalloc space.

Don't forget to turn on HIGHMEM to use everything above MAX_LOW_MEM.

-Matt

RE: Lots of memory on 826x

[Rune Torgersen]

What about moving the VMALLOC_START and VMALLOC_END #defines to
something else (like start at 0x80000000 and end at 0xc0000000), or will
that break some assumptions in the kernel about the position of the
vmalloc area?

> -----Original Message-----
> From: Matt Porter [mailto:mporter@kernel.crashing.org]=20
> Sent: Tuesday, March 08, 2005 14:33
> To: Rune Torgersen
> Cc: Kumar Gala; linuxppc-embedded@ozlabs.org
> Subject: Re: Lots of memory on 826x
>=20
> On Tue, Mar 08, 2005 at 12:37:33PM -0600, Rune Torgersen wrote:
> > Ok... Got a little closer....
> >=20
> > I have PCI (outbound) mapped in the area 0x80000000-0x8fffffff
> > (prefetch/non-prefetch and IO)
> > IMMR of cource is at 0xf0000000
> >=20
> > So I map the IO as following:
> > 	/* Map IMMR region to a 256MB BAT */
> > 	addr =3D (cpm2_immr !=3D NULL) ? (uint)cpm2_immr : CPM_MAP_ADDR;
> > 	io_block_mapping(addr, addr, 0x10000000, _PAGE_IO);
> >=20
> > 	io_block_mapping(0x80000000, 0x80000000, 0x10000000, _PAGE_IO);
> >=20
> > Now I can almost boot....:
> > It craches when trying to read from harddisk
> > (only happens with more than 512MB)
>=20
> That's expected if you are going to 768MB. You have kernel lowmem
> being mapped at 0xc0000000+<size_of_system_memory>. With 768MB that
> takes you all the way to where you have the IMMR mapped 1:1 I
> guess. This leaves no room for vmalloc space.  With 768MB of ram
> the kernel will calculate VMALLOC_START at 0xf1000000 which is a
> problem.
>=20
> A few options are:
>=20
> * Move the IMMR much higher to allow vmalloc space
> * Modify maximum low memory to limit to 512MB using the advanced
>   options menu
> * Modify KERNELBASE to something like 0xa0000000 using the advanced
>   options menu
>=20
> All of these methods will provide addition vmalloc space.
>=20
> Don't forget to turn on HIGHMEM to use everything above MAX_LOW_MEM.
>=20
> -Matt
>=20
>=20

Re: Lots of memory on 826x

[Kumar Gala]

Hmm, I think there is an expectation that virt->phys is a simple=20
translation (ie, subtract KERNELBASE) for lowmem.  lowmem most parts=20
tops out at 768M.  So changing VMALLOC_START would break that.

Is there an issue with moving the IMMR virt address up higher?

- kumar

On Mar 8, 2005, at 3:16 PM, Rune Torgersen wrote:

> What about moving the VMALLOC_START and VMALLOC_END #defines to
>  something else (like start at 0x80000000 and end at 0xc0000000), or=20=

> will
>  that break some assumptions in the kernel about the position of the
>  vmalloc area?
>
> > -----Original Message-----
>  > From: Matt Porter [mailto:mporter@kernel.crashing.org]
> > Sent: Tuesday, March 08, 2005 14:33
>  > To: Rune Torgersen
>  > Cc: Kumar Gala; linuxppc-embedded@ozlabs.org
> > Subject: Re: Lots of memory on 826x
>  >
> > On Tue, Mar 08, 2005 at 12:37:33PM -0600, Rune Torgersen wrote:
>  > > Ok... Got a little closer....
>  > >
> > > I have PCI (outbound) mapped in the area 0x80000000-0x8fffffff
>  > > (prefetch/non-prefetch and IO)
>  > > IMMR of cource is at 0xf0000000
>  > >
> > > So I map the IO as following:
>  > > =A0=A0=A0 /* Map IMMR region to a 256MB BAT */
>  > > =A0=A0=A0 addr =3D (cpm2_immr !=3D NULL) ? (uint)cpm2_immr : =
CPM_MAP_ADDR;
> > > =A0=A0=A0 io_block_mapping(addr, addr, 0x10000000, _PAGE_IO);
>  > >
> > > =A0=A0=A0 io_block_mapping(0x80000000, 0x80000000, 0x10000000, =
_PAGE_IO);
>  > >
> > > Now I can almost boot....:
> > > It craches when trying to read from harddisk
>  > > (only happens with more than 512MB)
>  >
> > That's expected if you are going to 768MB. You have kernel lowmem
> > being mapped at 0xc0000000+<size_of_system_memory>. With 768MB that
>  > takes you all the way to where you have the IMMR mapped 1:1 I
>  > guess. This leaves no room for vmalloc space.=A0 With 768MB of ram
>  > the kernel will calculate VMALLOC_START at 0xf1000000 which is a
>  > problem.
>  >
> > A few options are:
>  >
> > * Move the IMMR much higher to allow vmalloc space
>  > * Modify maximum low memory to limit to 512MB using the advanced
>  >=A0=A0 options menu
>  > * Modify KERNELBASE to something like 0xa0000000 using the advanced
>  >=A0=A0 options menu
>  >
> > All of these methods will provide addition vmalloc space.
> >
> > Don't forget to turn on HIGHMEM to use everything above MAX_LOW_MEM.
> >
> > -Matt
>  >
> >

RE: Lots of memory on 826x

[Rune Torgersen]

=20

> -----Original Message-----
> From: Kumar Gala [mailto:kumar.gala@freescale.com]=20
> Sent: Tuesday, March 08, 2005 17:14
> To: Rune Torgersen
> Cc: linuxppc-embedded@ozlabs.org; Matt Porter
> Subject: Re: Lots of memory on 826x
>=20
> Hmm, I think there is an expectation that virt->phys is a simple=20
> translation (ie, subtract KERNELBASE) for lowmem.  lowmem most parts=20
> tops out at 768M.  So changing VMALLOC_START would break that.

Yep noticed..... (tried changing it...)

>=20
> Is there an issue with moving the IMMR virt address up higher?
Not really. Will have to redo a whole lot of address maps for some
external devices, but not too bad...

I did change the kernel start address, and that seems to work better,
except that things is still broken when CONFIG_HIGHMEM is set
When that option is set, it maps the highmem to 0xFE000000, which is in
the IO area mapped for IMMR and other memory mapped devices.
(0xF0000000-0xFFFFFFFF)

This can be changed by CONFIG_HIGHMEM_START. Any good ideas on where to
move it to?

(Maybe change IMMR/IO to 0xF8000000 and up, and set HIGHMEM_START to
0xF0000000?)

Re: Lots of memory on 826x

[Matt Porter]

On Tue, Mar 08, 2005 at 05:20:38PM -0600, Rune Torgersen wrote:
> > -----Original Message-----
> > From: Kumar Gala [mailto:kumar.gala@freescale.com] 
> > Sent: Tuesday, March 08, 2005 17:14
> > To: Rune Torgersen
> > Cc: linuxppc-embedded@ozlabs.org; Matt Porter
> > Subject: Re: Lots of memory on 826x
> > 
> > Hmm, I think there is an expectation that virt->phys is a simple 
> > translation (ie, subtract KERNELBASE) for lowmem.  lowmem most parts 
> > tops out at 768M.  So changing VMALLOC_START would break that.
> 
> Yep noticed..... (tried changing it...)

Yeah, sorry, got busy and couldn't reply to say, "don't do that!". :)
 
> > Is there an issue with moving the IMMR virt address up higher?
> Not really. Will have to redo a whole lot of address maps for some
> external devices, but not too bad...
> 
> I did change the kernel start address, and that seems to work better,
> except that things is still broken when CONFIG_HIGHMEM is set
> When that option is set, it maps the highmem to 0xFE000000, which is in
> the IO area mapped for IMMR and other memory mapped devices.
> (0xF0000000-0xFFFFFFFF)
> 
> This can be changed by CONFIG_HIGHMEM_START. Any good ideas on where to
> move it to?
> 
> (Maybe change IMMR/IO to 0xF8000000 and up, and set HIGHMEM_START to
> 0xF0000000?)

Ahh, but early ioremaps (before mm is inited) grow down from PKMAP_BASE,
so you will be overlapping the top of lowmem with those allocations.
It wants to be above vmalloc space.  Set HIGHMEM_START to 0xF6000000,
IMMR/IO to 0xF8000000 and you then have vmalloc space between
0xF1000000-0xF5FFFFFF.

-Matt

RE: Lots of memory on 826x

[Rune Torgersen]

I can use 1GB of RAM now, if I move kernel base to 0xA0000000 and set=20
lowmem size to 1GB. (No Highmem at all)

> -----Original Message-----
> From: Matt Porter [mailto:mporter@kernel.crashing.org]=20
> Sent: Tuesday, March 08, 2005 17:30
> To: Rune Torgersen
> Cc: Kumar Gala; linuxppc-embedded@ozlabs.org
> Subject: Re: Lots of memory on 826x

> Ahh, but early ioremaps (before mm is inited) grow down from=20
> PKMAP_BASE,
> so you will be overlapping the top of lowmem with those allocations.
> It wants to be above vmalloc space.  Set HIGHMEM_START to 0xF6000000,
> IMMR/IO to 0xF8000000 and you then have vmalloc space between
> 0xF1000000-0xF5FFFFFF.

I'll try this tomorrow and see what happens.=20
(Still need to make sure our board supports 2GB....:)

Re: Lots of memory on 826x

[Matt Porter]

On Tue, Mar 08, 2005 at 05:35:19PM -0600, Rune Torgersen wrote:
> I can use 1GB of RAM now, if I move kernel base to 0xA0000000 and set 
> lowmem size to 1GB. (No Highmem at all)
 
Yup, that's one of many many options. Since TASK_SIZE is only
0x80000000 by default on ppc32, there's no loss anyway.

-Matt

RE: Lots of memory on 826x

[Rune Torgersen]

Yay... It worked....
Had to patch m8260_setup a bit.

In m8260_map_io() I changed
	io_block_mapping(addr, addr, 0x10000000, _PAGE_IO);
into
	io_block_mapping(addr, addr, 0xffffffff - addr + 1, _PAGE_IO);=20

So that I could change the IMMR address in my config file only, and the
mappings were still correct.

I could probably submit a proper patch.

> > -----Original Message-----
> > From: Matt Porter [mailto:mporter@kernel.crashing.org]=20
> > Sent: Tuesday, March 08, 2005 17:30
> > To: Rune Torgersen
> > Cc: Kumar Gala; linuxppc-embedded@ozlabs.org
> > Subject: Re: Lots of memory on 826x
>=20
> > Ahh, but early ioremaps (before mm is inited) grow down from=20
> > PKMAP_BASE,
> > so you will be overlapping the top of lowmem with those allocations.
> > It wants to be above vmalloc space.  Set HIGHMEM_START to=20
> 0xF6000000,
> > IMMR/IO to 0xF8000000 and you then have vmalloc space between
> > 0xF1000000-0xF5FFFFFF.
>=20
> I'll try this tomorrow and see what happens.=20
> (Still need to make sure our board supports 2GB....:)