Discontiguous Memory

Discontiguous Memory

[jbahr]

We have a client building a PPC8548-based product who insists that we
allocate DRAM real address space in two large chunks at 0-2GB and 4-6GB in
the 36-bit address space.  It doesn't look like U-Boot's bd_info structure
allows for that, and it doesn't look like the Linux init routines (which
accesses the passed table) knows how to handle discontiguous memory either
(as opposed to X86 Linux, which can accept an E820 table). 

It looks like Linux cleans out the TLB's pretty quickly, so it wouldn't know
the VA-to-RA mapping.   

I've seen papers on some PPC Linuxes that handle large discontiguous real
DRAM memory spaces, but it doesn't look like the Linux in ELDK does.  Is
that correct? 

This is just the start of a headache between the architects and we poor
implementers.  It's not clear to us that, even with ATMU support, how normal
drivers are going to know how to create real addresses for buffers and such
to use when programming DMA controllers or external PCI devices. 

Are this really a problem or is there kernel provision for the PPC for
discontiguous real memory DRAM space?

Any comments would be VERY appreciated. 

Regards, 
J
-- 
View this message in context: http://www.nabble.com/Discontiguous-Memory-tf2279691.html#a6332580
Sent from the linuxppc-dev forum at Nabble.com.

Re: Discontiguous Memory

[Arnd Bergmann]

Am Friday 15 September 2006 22:22 schrieb jbahr:
> Are this really a problem or is there kernel provision for the PPC for
> discontiguous real memory DRAM space?
>
> Any comments would be VERY appreciated.

You probably want to enable the CONFIG_ARCH_SPARSEMEM_ENABLE
option, which is currently dependant on CONFIG_PPC64.

Look in the early_init_dt_scan_memory function (arch/powerpc/kernel/prom.c)
for how to layout the "memory" nodes in your device tree.

	Arnd <><

RE: Discontiguous Memory

[Jeffery Bahr]

Thanks, Arnd.  As we're running a 32-bit 8548, looks like we might not be
able to enable sparse memory maps.

Good thought, though.

J 

-----Original Message-----
From: Arnd Bergmann [mailto:arnd@arndb.de] 
Sent: Friday, September 15, 2006 5:25 PM
To: linuxppc-dev@ozlabs.org
Cc: jbahr
Subject: Re: Discontiguous Memory

Am Friday 15 September 2006 22:22 schrieb jbahr:
> Are this really a problem or is there kernel provision for the PPC for 
> discontiguous real memory DRAM space?
>
> Any comments would be VERY appreciated.

You probably want to enable the CONFIG_ARCH_SPARSEMEM_ENABLE option, which
is currently dependant on CONFIG_PPC64.

Look in the early_init_dt_scan_memory function (arch/powerpc/kernel/prom.c)
for how to layout the "memory" nodes in your device tree.

	Arnd <><

Re: Discontiguous Memory

[Benjamin Herrenschmidt]

On Fri, 2006-09-15 at 13:22 -0700, jbahr wrote:
> We have a client building a PPC8548-based product who insists that we
> allocate DRAM real address space in two large chunks at 0-2GB and 4-6GB in
> the 36-bit address space.  It doesn't look like U-Boot's bd_info structure
> allows for that, and it doesn't look like the Linux init routines (which
> accesses the passed table) knows how to handle discontiguous memory either
> (as opposed to X86 Linux, which can accept an E820 table). 
> 
> It looks like Linux cleans out the TLB's pretty quickly, so it wouldn't know
> the VA-to-RA mapping.   
> 
> I've seen papers on some PPC Linuxes that handle large discontiguous real
> DRAM memory spaces, but it doesn't look like the Linux in ELDK does.  Is
> that correct? 
> 
> This is just the start of a headache between the architects and we poor
> implementers.  It's not clear to us that, even with ATMU support, how normal
> drivers are going to know how to create real addresses for buffers and such
> to use when programming DMA controllers or external PCI devices. 
> 
> Are this really a problem or is there kernel provision for the PPC for
> discontiguous real memory DRAM space?
> 
> Any comments would be VERY appreciated. 

There is no support for any of this in the current PowerPC 32 bits
kernel. It's possible to add it, though, but it's not alrady there.
There are two main issues:

 - One is to enable support for sparse memory maps and add the necessary
support to the low mm code. Not terribly hard (see how it's done for 64
bits)

 - A more annoying one is support for DMA since a lot of devices cannot
DMA above 32 bits. A lot of 'sane' platforms that provide more memory
than can be DMA'ed in 32 bits also provide an IOMMU that does page
translation of incoming DMAs. This isn't your case however, thus you
would have to implement some of the alternate solutions used on x86_64
(Intel 64 bits platforms also lack an iommu). This essentially consist
of defining a ZONE_DMA32 and possibly also using swiotlb to do bounce
buffering for drivers that don't deal with ZONE_DMA32 yet.

Ben.

Re: Discontiguous Memory

[Arnd Bergmann]

Am Saturday 16 September 2006 02:56 schrieb Benjamin Herrenschmidt:
> =A0- A more annoying one is support for DMA since a lot of devices cannot
> DMA above 32 bits. A lot of 'sane' platforms that provide more memory
> than can be DMA'ed in 32 bits also provide an IOMMU that does page
> translation of incoming DMAs. This isn't your case however, thus you
> would have to implement some of the alternate solutions used on x86_64
> (Intel 64 bits platforms also lack an iommu). This essentially consist
> of defining a ZONE_DMA32 and possibly also using swiotlb to do bounce
> buffering for drivers that don't deal with ZONE_DMA32 yet.

On 32 bit platforms, you normally have ZONE_NORMAL for up to 4G and
ZONE_HIGHMEM beyond, so you don't need an extra ZONE_DMA32.

	Arnd <><

Re: Discontiguous Memory

[Benjamin Herrenschmidt]

On Sat, 2006-09-16 at 11:51 +0200, Arnd Bergmann wrote:
> Am Saturday 16 September 2006 02:56 schrieb Benjamin Herrenschmidt:
> >  - A more annoying one is support for DMA since a lot of devices cannot
> > DMA above 32 bits. A lot of 'sane' platforms that provide more memory
> > than can be DMA'ed in 32 bits also provide an IOMMU that does page
> > translation of incoming DMAs. This isn't your case however, thus you
> > would have to implement some of the alternate solutions used on x86_64
> > (Intel 64 bits platforms also lack an iommu). This essentially consist
> > of defining a ZONE_DMA32 and possibly also using swiotlb to do bounce
> > buffering for drivers that don't deal with ZONE_DMA32 yet.
> 
> On 32 bit platforms, you normally have ZONE_NORMAL for up to 4G and
> ZONE_HIGHMEM beyond, so you don't need an extra ZONE_DMA32.

No, in fact, it's even worse :) Your example works only on platforms
that can do 4G/4G split. On ppc32, currently, we don't (though we could
with some freescale BookE and their new instructions to load/store cross
AS), thus we have a 3G/1G split, which limits ZONE_NORMAL to about
768Mb.

So ZONE_HIGHEMEM is actually most of your memory. But you -can- DMA from
highmem and can't have memory be both HIGHMEM and ZONE_DMA32 (though I
suppose devices requesting ZONE_DMA32 will hit ZONE_NORMAL). It's a
mess...

Ben.