NOMMU - How to reserve 1 MB in top of memory in a clean way

NOMMU - How to reserve 1 MB in top of memory in a clean way

[Bas Vermeulen]

Hey all,

I am currently working on the bfinnommu linux port for the BlackFin 533.
I need to grab the top 1 MB of memory so I can give it out to drivers
that need non-cached memory for DMA operations.

I've tried the following approaches (which each failed, in different
ways):

1. Allocate 1 MB in ZONE_DMA. This doesn't work because ZONE_DMA needs
to be in the bottom of memory (and I couldn't find a way around that),
and needs to be a minimum of 4 MB.
2. Create ZONE_NORMAL with all memory in it, but add a hole of 1 MB.
This crashes in the swapper somewhere.

What I want is a way to cleanly grab all pages in the top megabyte of
memory, so I can give them out in an implementation (to be written) of
dma_alloc_coherent and friends. That top megabyte would be set to
non-cached in the software cache manager.

If anyone can point me in the right direction, that would be great.

Regards,

Bas Vermeulen

Re: NOMMU - How to reserve 1 MB in top of memory in a clean way

[Catalin Marinas]

Bas Vermeulen <bvermeul@blackstar.xs4all.nl> wrote:
> I am currently working on the bfinnommu linux port for the BlackFin 533.
> I need to grab the top 1 MB of memory so I can give it out to drivers
> that need non-cached memory for DMA operations.

I did this long time ago (on a 2.4 kernel), trying to avoid a hardware
problem. I re-ordered the zones so that ZONE_DMA came after
ZONE_NORMAL. Since the DMA memory was quite small (less than 1MB), I
also put a "break" before "case ZONE_DMA" in the
build_zonelists_node() functions to avoid the allocation fallback.

-- 
Catalin

Re: NOMMU - How to reserve 1 MB in top of memory in a clean way

[Richard B. Johnson]

On Wed, 6 Apr 2005, Catalin Marinas wrote:

> Bas Vermeulen <bvermeul@blackstar.xs4all.nl> wrote:
>> I am currently working on the bfinnommu linux port for the BlackFin 533.
>> I need to grab the top 1 MB of memory so I can give it out to drivers
>> that need non-cached memory for DMA operations.
>
> I did this long time ago (on a 2.4 kernel), trying to avoid a hardware
> problem. I re-ordered the zones so that ZONE_DMA came after
> ZONE_NORMAL. Since the DMA memory was quite small (less than 1MB), I
> also put a "break" before "case ZONE_DMA" in the
> build_zonelists_node() functions to avoid the allocation fallback.
>
> -- 
> Catalin
>


1 Megabyte of DMA RAM should be available using conventional
means __get_dma_pages(GFP_KERNEL, 0x100) soon after boot.

Or just use mem= on the boot command line. This will tell
the kernel the extent of memory to use. Any RAM after that
is available. Your driver can access kernel variable, "num_physpages"
to find the last page it is supposed to use. Some kernel versions
actually touch the next page so, to be safe, your code can
use:
    mem = (num_physpages * PAGE_SIZE) + PAGE_SIZE;
... for the first available free RAM.

Note that there may be PCI BARS allocated in this address-space if
you have 4 Gb of RAM. You need to be carefull.


Cheers,
Dick Johnson
Penguin : Linux version 2.6.11 on an i686 machine (5537.79 BogoMips).
  Notice : All mail here is now cached for review by Dictator Bush.
                  98.36% of all statistics are fiction.

Re: NOMMU - How to reserve 1 MB in top of memory in a clean way

[Catalin Marinas]

"Richard B. Johnson" <linux-os@analogic.com> wrote:
> 1 Megabyte of DMA RAM should be available using conventional
> means __get_dma_pages(GFP_KERNEL, 0x100) soon after boot.

The problem is that he needs to get this memory from the last MB only,
__get_dma_pages would return pages from ZONE_DMA but this is usually
at the beginning of RAM.

> Or just use mem= on the boot command line. This will tell
> the kernel the extent of memory to use. Any RAM after that
> is available. Your driver can access kernel variable, "num_physpages"
> to find the last page it is supposed to use.

But this means that you would need to modify all the drivers that need
DMA memory. Modifying the zones is actually transparent for the
drivers.

--
Catalin

Re: NOMMU - How to reserve 1 MB in top of memory in a clean way

[Bas Vermeulen]

On Wed, 2005-04-06 at 16:53, Richard B. Johnson wrote:
> On Wed, 6 Apr 2005, Catalin Marinas wrote:
> 
> > Bas Vermeulen <bvermeul@blackstar.xs4all.nl> wrote:
> >> I am currently working on the bfinnommu linux port for the BlackFin 533.
> >> I need to grab the top 1 MB of memory so I can give it out to drivers
> >> that need non-cached memory for DMA operations.
> >
> > I did this long time ago (on a 2.4 kernel), trying to avoid a hardware
> > problem. I re-ordered the zones so that ZONE_DMA came after
> > ZONE_NORMAL. Since the DMA memory was quite small (less than 1MB), I
> > also put a "break" before "case ZONE_DMA" in the
> > build_zonelists_node() functions to avoid the allocation fallback.
> >
> > -- 
> > Catalin
> >
> 
> 
> 1 Megabyte of DMA RAM should be available using conventional
> means __get_dma_pages(GFP_KERNEL, 0x100) soon after boot.

The 1 Megabyte of DMA RAM needs to be aligned on 1 MB, since I have to
tell the (software) cache manager to make it uncacheable (and that works
in blocks of 1M or 4M).
I probably could allocate 2M, get the alignment correctly, then free the
pages I don't need/want, and feed that range to the cache manager.
I'm not entirely sure I can do that before the call to free_all_bootmem,
since all pages are reserved before that.

Is there a way to do this directly (eg, just nab the pages I want/need,
and tell the zones not to use them?)

> Or just use mem= on the boot command line. This will tell
> the kernel the extent of memory to use. Any RAM after that
> is available. Your driver can access kernel variable, "num_physpages"
> to find the last page it is supposed to use. Some kernel versions
> actually touch the next page so, to be safe, your code can
> use:
>     mem = (num_physpages * PAGE_SIZE) + PAGE_SIZE;
> ... for the first available free RAM.

That's my backup plan, but I'd prefer to do it properly in kernel memory
space (if at all possible).

> Note that there may be PCI BARS allocated in this address-space if
> you have 4 Gb of RAM. You need to be carefull.

BlackFin doesn't have PCI (and no MMU, so a more limited address space
than 4 GB).

Thanks for the insight so far,

Bas Vermeulen

Re: NOMMU - How to reserve 1 MB in top of memory in a clean way

[Bas Vermeulen]

On Wed, 2005-04-06 at 15:26, Catalin Marinas wrote:
> Bas Vermeulen <bvermeul@blackstar.xs4all.nl> wrote:
> > I am currently working on the bfinnommu linux port for the BlackFin 533.
> > I need to grab the top 1 MB of memory so I can give it out to drivers
> > that need non-cached memory for DMA operations.
> 
> I did this long time ago (on a 2.4 kernel), trying to avoid a hardware
> problem. I re-ordered the zones so that ZONE_DMA came after
> ZONE_NORMAL. Since the DMA memory was quite small (less than 1MB), I
> also put a "break" before "case ZONE_DMA" in the
> build_zonelists_node() functions to avoid the allocation fallback.

This will put me in the zone of 'it ain't ever going to be integrated'.
I'd preferrably find a solution without changing the zones. My ideal
solution would be grabbing pages before they are assigned to a zone, or
at least for the zone to recognize them as used.

Bas Vermeulen

Re: NOMMU - How to reserve 1 MB in top of memory in a clean way

[Catalin Marinas]

Bas Vermeulen <bvermeul@blackstar.xs4all.nl> wrote:
> This will put me in the zone of 'it ain't ever going to be integrated'.
> I'd preferrably find a solution without changing the zones. My ideal
> solution would be grabbing pages before they are assigned to a zone, or
> at least for the zone to recognize them as used.

The order of the zones was initially chosen based on x86 and ISA
bus. This is no longer valid for all the platforms (for example, the
1st MB can be SSRAM and not usable for DMA).

This might be possible but I've never tried (and not sure how it would
work with nommu) - define CONFIG_NUMA and use 2 memory banks, one from
0 to max - 1M and the 2nd being 1MB. You can define the zone sizes for
each node when calling free_area_init_node() so that the first node
doesn't have any DMA area and the 2nd one has only DMA.

Maybe other could comment on this, not sure it will work. What core
are you using?

A third option could be to define your own dma_alloc* functions and
not give the top MB to the kernel (mem=...).

-- 
Catalin