how can I cleanly exclude memory from the kernel memory allocator?

how can I cleanly exclude memory from the kernel memory allocator?

[Larry Bassel]

I want to (early in system initialization) exclude some
contiguous physical memory from one or more memory banks
so that it won't be mapped in the normal kernel 1-to-1 mapping
(so that it can be mapped uncached, strongly ordered, etc.
as needed -- I know that it is forbidden to have a cached
and an uncached mapping to the same memory) and so that it
won't be freed into the kernel memory allocator (so that
it won't fragment and can be allocated using genalloc).

I have tried to find a clean way to do this, but none of
the approaches I've considered seem very good:

1. Add a hook to the memory tag parsing routine to (possibly)
change each tag before arm_add_memory() is called,
or alter arm_add_memory() itself.
2. Add a hook at the beginning of paging_init() to inspect
and alter the meminfo memory bank starts and sizes as needed.
3. Wait until after the memory is added to the normal
kernel memory pool and somehow get it back out
(I can't use alloc_bootmem() or related functionality of course
since then the memory will be part of the cached kernel 1-to-1 1M
page mapping, precluding use of ioremap() on it).

A further drawback for #1 is that I'd really like to
see all of the available memory banks before deciding which
memory to steal (we often have multiple memory banks on
our platforms).

Perhaps there is a solution for this problem that already exists
that I'm unaware of (or at least community consensus about how
one should add this functionality).

I believe Russell King has stated that he intends
to carve out memory that can be used for dma_alloc_coherent(),
that won't be mapped otherwise by the kernel.

I'd like to know how he is planning to do this, as I could hopefully
do something similar.

Thanks.

Larry Bassel

-- 
Sent by an employee of the Qualcomm Innovation Center, Inc.
The Qualcomm Innovation Center, Inc. is a member of the Code Aurora Forum.

Re: how can I cleanly exclude memory from the kernel memory allocator?

[Colin Cross]

On Thu, Mar 24, 2011 at 3:18 PM, Larry Bassel <lbassel@codeaurora.org> wrote:
> I want to (early in system initialization) exclude some
> contiguous physical memory from one or more memory banks
> so that it won't be mapped in the normal kernel 1-to-1 mapping
> (so that it can be mapped uncached, strongly ordered, etc.
> as needed -- I know that it is forbidden to have a cached
> and an uncached mapping to the same memory) and so that it
> won't be freed into the kernel memory allocator (so that
> it won't fragment and can be allocated using genalloc).
>
> I have tried to find a clean way to do this, but none of
> the approaches I've considered seem very good:
>
> 1. Add a hook to the memory tag parsing routine to (possibly)
> change each tag before arm_add_memory() is called,
> or alter arm_add_memory() itself.
> 2. Add a hook at the beginning of paging_init() to inspect
> and alter the meminfo memory bank starts and sizes as needed.
> 3. Wait until after the memory is added to the normal
> kernel memory pool and somehow get it back out
> (I can't use alloc_bootmem() or related functionality of course
> since then the memory will be part of the cached kernel 1-to-1 1M
> page mapping, precluding use of ioremap() on it).
>
> A further drawback for #1 is that I'd really like to
> see all of the available memory banks before deciding which
> memory to steal (we often have multiple memory banks on
> our platforms).
>
> Perhaps there is a solution for this problem that already exists
> that I'm unaware of (or at least community consensus about how
> one should add this functionality).
>
> I believe Russell King has stated that he intends
> to carve out memory that can be used for dma_alloc_coherent(),
> that won't be mapped otherwise by the kernel.
>
> I'd like to know how he is planning to do this, as I could hopefully
> do something similar.
>
> Thanks.
>
> Larry Bassel

You want memblock_remove, called from the new .reserve machine entry
function.  See arch/arm/plat-omap/devices.c for an example.

Re: how can I cleanly exclude memory from the kernel memory allocator?

[Larry Bassel]

On 24 Mar 11 16:14, Colin Cross wrote:
> On Thu, Mar 24, 2011 at 3:18 PM, Larry Bassel <lbassel@codeaurora.org> wrote:
> > I want to (early in system initialization) exclude some
> > contiguous physical memory from one or more memory banks
> > so that it won't be mapped in the normal kernel 1-to-1 mapping
> > (so that it can be mapped uncached, strongly ordered, etc.
> > as needed -- I know that it is forbidden to have a cached
> > and an uncached mapping to the same memory) and so that it
> > won't be freed into the kernel memory allocator (so that
> > it won't fragment and can be allocated using genalloc).
> >
> > I have tried to find a clean way to do this, but none of
> > the approaches I've considered seem very good:
> >
> > 1. Add a hook to the memory tag parsing routine to (possibly)
> > change each tag before arm_add_memory() is called,
> > or alter arm_add_memory() itself.
> > 2. Add a hook at the beginning of paging_init() to inspect
> > and alter the meminfo memory bank starts and sizes as needed.
> > 3. Wait until after the memory is added to the normal
> > kernel memory pool and somehow get it back out
> > (I can't use alloc_bootmem() or related functionality of course
> > since then the memory will be part of the cached kernel 1-to-1 1M
> > page mapping, precluding use of ioremap() on it).
> >
> > A further drawback for #1 is that I'd really like to
> > see all of the available memory banks before deciding which
> > memory to steal (we often have multiple memory banks on
> > our platforms).
> >
> > Perhaps there is a solution for this problem that already exists
> > that I'm unaware of (or at least community consensus about how
> > one should add this functionality).
> >
> > I believe Russell King has stated that he intends
> > to carve out memory that can be used for dma_alloc_coherent(),
> > that won't be mapped otherwise by the kernel.
> >
> > I'd like to know how he is planning to do this, as I could hopefully
> > do something similar.
> >
> > Thanks.
> >
> > Larry Bassel
> 
> You want memblock_remove, called from the new .reserve machine entry
> function.  See arch/arm/plat-omap/devices.c for an example.

Yes, this is effectively my option #2 (reserve memory after the tags
are parsed and before paging_init). This looks like a good approach.

Thanks.

Larry

> --
> To unsubscribe from this list: send the line "unsubscribe linux-arm-msm" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

-- 
Sent by an employee of the Qualcomm Innovation Center, Inc.
The Qualcomm Innovation Center, Inc. is a member of the Code Aurora Forum.

Re: how can I cleanly exclude memory from the kernel memory allocator?

[Nicolas Pitre]

On Thu, 24 Mar 2011, Larry Bassel wrote:

> I want to (early in system initialization) exclude some
> contiguous physical memory from one or more memory banks
> so that it won't be mapped in the normal kernel 1-to-1 mapping
> (so that it can be mapped uncached, strongly ordered, etc.
> as needed -- I know that it is forbidden to have a cached
> and an uncached mapping to the same memory) and so that it
> won't be freed into the kernel memory allocator (so that
> it won't fragment and can be allocated using genalloc).
> 
> I have tried to find a clean way to do this, but none of
> the approaches I've considered seem very good:
> 
> 1. Add a hook to the memory tag parsing routine to (possibly)
> change each tag before arm_add_memory() is called,
> or alter arm_add_memory() itself.

There is already a hook for that.  In your machine descriptor you can 
add a .fixup method to do just that.  See tag_fixup_mem32() in 
arch/arm/mach-orion5x/common.c for a usage example.


Nicolas

Re: how can I cleanly exclude memory from the kernel memory allocator?

[Larry Bassel]

On 24 Mar 11 19:27, Nicolas Pitre wrote:
> On Thu, 24 Mar 2011, Larry Bassel wrote:
> 
> > I want to (early in system initialization) exclude some
> > contiguous physical memory from one or more memory banks
> > so that it won't be mapped in the normal kernel 1-to-1 mapping
> > (so that it can be mapped uncached, strongly ordered, etc.
> > as needed -- I know that it is forbidden to have a cached
> > and an uncached mapping to the same memory) and so that it
> > won't be freed into the kernel memory allocator (so that
> > it won't fragment and can be allocated using genalloc).
> > 
> > I have tried to find a clean way to do this, but none of
> > the approaches I've considered seem very good:
> > 
> > 1. Add a hook to the memory tag parsing routine to (possibly)
> > change each tag before arm_add_memory() is called,
> > or alter arm_add_memory() itself.
> 
> There is already a hook for that.  In your machine descriptor you can 
> add a .fixup method to do just that.  See tag_fixup_mem32() in 
> arch/arm/mach-orion5x/common.c for a usage example.

I thought fixup was called before the tags were parsed so
that one could (for instance) set up a memory configuration
which would override the tags (such as what is done in 
arch/arm/mach-msm/board-halibut,c) -- while I could presumably
parse the tags myself there to get the memory map
that the bootloader passed in (and alter it), this seems like
a duplication of effort.

> 
> 
> Nicolas

Larry

> --
> To unsubscribe from this list: send the line "unsubscribe linux-arm-msm" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

-- 
Sent by an employee of the Qualcomm Innovation Center, Inc.
The Qualcomm Innovation Center, Inc. is a member of the Code Aurora Forum.

Re: how can I cleanly exclude memory from the kernel memory allocator?

[Larry Bassel]

On 24 Mar 11 19:27, Nicolas Pitre wrote:
> On Thu, 24 Mar 2011, Larry Bassel wrote:
> 
> > I want to (early in system initialization) exclude some
> > contiguous physical memory from one or more memory banks
> > so that it won't be mapped in the normal kernel 1-to-1 mapping
> > (so that it can be mapped uncached, strongly ordered, etc.
> > as needed -- I know that it is forbidden to have a cached
> > and an uncached mapping to the same memory) and so that it
> > won't be freed into the kernel memory allocator (so that
> > it won't fragment and can be allocated using genalloc).
> > 
> > I have tried to find a clean way to do this, but none of
> > the approaches I've considered seem very good:
> > 
> > 1. Add a hook to the memory tag parsing routine to (possibly)
> > change each tag before arm_add_memory() is called,
> > or alter arm_add_memory() itself.
> 
> There is already a hook for that.  In your machine descriptor you can 
> add a .fixup method to do just that.  See tag_fixup_mem32() in 
> arch/arm/mach-orion5x/common.c for a usage example.

OK, please ignore my previous mail, I see that .fixup can be
used to adjust tags. It appears to me, however, that using a
.reserve function calling memblock_reserve will be an
easier way of doing what I need.

Thanks.

> 
> 
> Nicolas

Larry
> --
> To unsubscribe from this list: send the line "unsubscribe linux-arm-msm" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

-- 
Sent by an employee of the Qualcomm Innovation Center, Inc.
The Qualcomm Innovation Center, Inc. is a member of the Code Aurora Forum.

Re: how can I cleanly exclude memory from the kernel memory allocator?

[Russell King - ARM Linux]

On Fri, Mar 25, 2011 at 10:39:28AM -0700, Larry Bassel wrote:
> OK, please ignore my previous mail, I see that .fixup can be
> used to adjust tags. It appears to me, however, that using a
> .reserve function calling memblock_reserve will be an
> easier way of doing what I need.

Absolutely.

Re: how can I cleanly exclude memory from the kernel memory allocator?

[Larry Bassel]

On 25 Mar 11 20:27, Russell King - ARM Linux wrote:
> On Fri, Mar 25, 2011 at 10:39:28AM -0700, Larry Bassel wrote:
> > OK, please ignore my previous mail, I see that .fixup can be
> > used to adjust tags. It appears to me, however, that using a
> > .reserve function calling memblock_reserve will be an
> > easier way of doing what I need.
> 
> Absolutely.

Hopefully I'm just missing something obvious, but
when I call memblock_reserve (in .38) to reserve a block
of memory away from the general kernel memory pool,
I find that an ioremap of memory within the reserved block
still fails because pte_valid() is apparently true.

Is this intended behavior? If so, what would the
correct way of separating memory from the system be
so that there is no longer a 1-to-1 1M mapping for
this memory and ioremap (creating the *only* mapping
for this memory) is allowed?

Thanks.

Larry

> 
> _______________________________________________
> linux-arm-kernel mailing list
> linux-arm-kernel@lists.infradead.org
> http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

-- 
Sent by an employee of the Qualcomm Innovation Center, Inc.
The Qualcomm Innovation Center, Inc. is a member of the Code Aurora Forum.

Re: how can I cleanly exclude memory from the kernel memory allocator?

[Larry Bassel]

On 19 Apr 11 16:33, Larry Bassel wrote:
> On 25 Mar 11 20:27, Russell King - ARM Linux wrote:
> > On Fri, Mar 25, 2011 at 10:39:28AM -0700, Larry Bassel wrote:
> > > OK, please ignore my previous mail, I see that .fixup can be
> > > used to adjust tags. It appears to me, however, that using a
> > > .reserve function calling memblock_reserve will be an
> > > easier way of doing what I need.
> > 
> > Absolutely.
> 
> Hopefully I'm just missing something obvious, but
> when I call memblock_reserve (in .38) to reserve a block
> of memory away from the general kernel memory pool,
> I find that an ioremap of memory within the reserved block
> still fails because pte_valid() is apparently true.

I meant pfn_valid() not pte_valid() -- sorry.

> 
> Is this intended behavior? If so, what would the
> correct way of separating memory from the system be
> so that there is no longer a 1-to-1 1M mapping for
> this memory and ioremap (creating the *only* mapping
> for this memory) is allowed?
> 
> Thanks.
> 
> Larry
> 
> > 
> > _______________________________________________
> > linux-arm-kernel mailing list
> > linux-arm-kernel@lists.infradead.org
> > http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
> 
> -- 
> Sent by an employee of the Qualcomm Innovation Center, Inc.
> The Qualcomm Innovation Center, Inc. is a member of the Code Aurora Forum.
> 
> _______________________________________________
> linux-arm-kernel mailing list
> linux-arm-kernel@lists.infradead.org
> http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

-- 
Sent by an employee of the Qualcomm Innovation Center, Inc.
The Qualcomm Innovation Center, Inc. is a member of the Code Aurora Forum.

Re: how can I cleanly exclude memory from the kernel memory allocator?

[Colin Cross]

On Tue, Apr 19, 2011 at 4:33 PM, Larry Bassel <lbassel@codeaurora.org> wrote:
> On 25 Mar 11 20:27, Russell King - ARM Linux wrote:
>> On Fri, Mar 25, 2011 at 10:39:28AM -0700, Larry Bassel wrote:
>> > OK, please ignore my previous mail, I see that .fixup can be
>> > used to adjust tags. It appears to me, however, that using a
>> > .reserve function calling memblock_reserve will be an
>> > easier way of doing what I need.
>>
>> Absolutely.
>
> Hopefully I'm just missing something obvious, but
> when I call memblock_reserve (in .38) to reserve a block
> of memory away from the general kernel memory pool,
> I find that an ioremap of memory within the reserved block
> still fails because pte_valid() is apparently true.
>
> Is this intended behavior? If so, what would the
> correct way of separating memory from the system be
> so that there is no longer a 1-to-1 1M mapping for
> this memory and ioremap (creating the *only* mapping
> for this memory) is allowed?

You need to call memblock_remove, not memblock_reserve.

Re: how can I cleanly exclude memory from the kernel memory allocator?

[Larry Bassel]

On 19 Apr 11 23:48, Colin Cross wrote:
> On Tue, Apr 19, 2011 at 4:33 PM, Larry Bassel <lbassel@codeaurora.org> wrote:
> > On 25 Mar 11 20:27, Russell King - ARM Linux wrote:
> >> On Fri, Mar 25, 2011 at 10:39:28AM -0700, Larry Bassel wrote:
> >> > OK, please ignore my previous mail, I see that .fixup can be
> >> > used to adjust tags. It appears to me, however, that using a
> >> > .reserve function calling memblock_reserve will be an
> >> > easier way of doing what I need.
> >>
> >> Absolutely.
> >
> > Hopefully I'm just missing something obvious, but
> > when I call memblock_reserve (in .38) to reserve a block
> > of memory away from the general kernel memory pool,
> > I find that an ioremap of memory within the reserved block
> > still fails because pte_valid() is apparently true.
> >
> > Is this intended behavior? If so, what would the
> > correct way of separating memory from the system be
> > so that there is no longer a 1-to-1 1M mapping for
> > this memory and ioremap (creating the *only* mapping
> > for this memory) is allowed?
> 
> You need to call memblock_remove, not memblock_reserve.

Yes, that works for our platforms which don't use SPARSEMEM. Thanks.

Unfortunately, in the SPARSEMEM case pfn_valid() appears
(include/linux/mmzone.h) to only check if the section the
page frame is in has SECTION_HAS_MEM_MAP set, which memblock_removing
a part of a section doesn't (and presumably shouldn't) unset, so
pfn_valid() is still true for such pages.

Larry

> --
> To unsubscribe from this list: send the line "unsubscribe linux-arm-msm" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

-- 
Sent by an employee of the Qualcomm Innovation Center, Inc.
The Qualcomm Innovation Center, Inc. is a member of the Code Aurora Forum.