- bootmem-avoid-dma32-zone-by-default.patch removed from -mm tree

- bootmem-avoid-dma32-zone-by-default.patch removed from -mm tree

[akpm]

The patch titled
     bootmem: avoid DMA32 zone by default
has been removed from the -mm tree.  Its filename was
     bootmem-avoid-dma32-zone-by-default.patch

This patch was dropped because I'm all confused

The current -mm tree may be found at http://userweb.kernel.org/~akpm/mmotm/

------------------------------------------------------
Subject: bootmem: avoid DMA32 zone by default
From: Johannes Weiner <hannes@cmpxchg.org>

Bootmem already tries normal allocations above the DMA zone to reserve it
for users that can not cope with higher addresses.

The same principle applies to the DMA32 zone, which is currently not
spared from normal allocations.

This can lead to exhaustion of this limited amount of address space
through things that can easily live elsewhere, like the mem_map e.g.

Raise bootmem's default goal beyond DMA32 for architectures with this zone
defined.  For now, these are x86 and mips.

I cc'd stable because this affects already released kernels.  But since
this is the first report of DMA32 memory exhaustion through bootmem that I
hear of, you guys might want to skip this patch due to the fragile nature
of early memory management.

Signed-off-by: Johannes Weiner <hannes@cmpxchg.org>
Reported-by: Jiri Slaby <jslaby@suse.cz>
Cc: Ralf Baechle <ralf@linux-mips.org>
Cc: Ingo Molnar <mingo@elte.hu>
Cc Thomas Gleixner <tglx@linutronix.de>
Cc: "H. Peter Anvin" <hpa@zytor.com>
Cc: Yinghai Lu <yinghai@kernel.org>
Cc: <stable@kernel.org>		[See above note]
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
---

 include/linux/bootmem.h |   20 +++++++++++++-------
 1 file changed, 13 insertions(+), 7 deletions(-)

diff -puN include/linux/bootmem.h~bootmem-avoid-dma32-zone-by-default include/linux/bootmem.h
--- a/include/linux/bootmem.h~bootmem-avoid-dma32-zone-by-default
+++ a/include/linux/bootmem.h
@@ -103,20 +103,26 @@ extern void *__alloc_bootmem_low_node(pg
 				      unsigned long align,
 				      unsigned long goal);
 
+#ifdef MAX_DMA32_PFN
+#define BOOTMEM_DEFAULT_GOAL	(MAX_DMA32_PFN << PAGE_SHIFT)
+#else
+#define BOOTMEM_DEFAULT_GOAL	__pa(MAX_DMA_ADDRESS)
+#endif
+
 #define alloc_bootmem(x) \
-	__alloc_bootmem(x, SMP_CACHE_BYTES, __pa(MAX_DMA_ADDRESS))
+	__alloc_bootmem(x, SMP_CACHE_BYTES, BOOTMEM_DEFAULT_GOAL)
 #define alloc_bootmem_nopanic(x) \
-	__alloc_bootmem_nopanic(x, SMP_CACHE_BYTES, __pa(MAX_DMA_ADDRESS))
+	__alloc_bootmem_nopanic(x, SMP_CACHE_BYTES, BOOTMEM_DEFAULT_GOAL)
 #define alloc_bootmem_pages(x) \
-	__alloc_bootmem(x, PAGE_SIZE, __pa(MAX_DMA_ADDRESS))
+	__alloc_bootmem(x, PAGE_SIZE, BOOTMEM_DEFAULT_GOAL)
 #define alloc_bootmem_pages_nopanic(x) \
-	__alloc_bootmem_nopanic(x, PAGE_SIZE, __pa(MAX_DMA_ADDRESS))
+	__alloc_bootmem_nopanic(x, PAGE_SIZE, BOOTMEM_DEFAULT_GOAL)
 #define alloc_bootmem_node(pgdat, x) \
-	__alloc_bootmem_node(pgdat, x, SMP_CACHE_BYTES, __pa(MAX_DMA_ADDRESS))
+	__alloc_bootmem_node(pgdat, x, SMP_CACHE_BYTES, BOOTMEM_DEFAULT_GOAL)
 #define alloc_bootmem_pages_node(pgdat, x) \
-	__alloc_bootmem_node(pgdat, x, PAGE_SIZE, __pa(MAX_DMA_ADDRESS))
+	__alloc_bootmem_node(pgdat, x, PAGE_SIZE, BOOTMEM_DEFAULT_GOAL)
 #define alloc_bootmem_pages_node_nopanic(pgdat, x) \
-	__alloc_bootmem_node_nopanic(pgdat, x, PAGE_SIZE, __pa(MAX_DMA_ADDRESS))
+	__alloc_bootmem_node_nopanic(pgdat, x, PAGE_SIZE, BOOTMEM_DEFAULT_GOAL)
 
 #define alloc_bootmem_low(x) \
 	__alloc_bootmem_low(x, SMP_CACHE_BYTES, 0)
_

Patches currently in -mm which might be from hannes@cmpxchg.org are

bootmem-avoid-dma32-zone-by-default.patch

further plans on bootmem, was: Re: - bootmem-avoid-dma32-zone-by-default.patch removed from -mm tree

[Johannes Weiner]

On Tue, Mar 09, 2010 at 01:49:02PM -0800, Andrew Morton wrote:
> On Tue, 09 Mar 2010 13:09:55 -0800
> Yinghai Lu <yinghai@kernel.org> wrote:
> 
> > On 03/09/2010 11:40 AM, akpm@linux-foundation.org wrote:
> > > The patch titled
> > >      bootmem: avoid DMA32 zone by default
> > > has been removed from the -mm tree.  Its filename was
> > >      bootmem-avoid-dma32-zone-by-default.patch
> > > 
> > > This patch was dropped because I'm all confused
> > > 
> > 
> > Thanks for that...
> 
> Well.  I did drop it because I'm all confused.  It may come back.
> 
> If Johannes is working in the direction of removing and simplifying
> code then that's a high priority.  So I'm waiting to see where this
> discussion leads (on the mailing list, please!)

I am not working on simplifying in this area at the moment.  I am just
questioning the discrepancy between the motivation of Yinghai's patch
series to skip bootmem on x86 and its actual outcome.

The stated reason for the series was that the amount of memory allocators
involved in bootstrapping mm on x86 'seemed a bit excessive'. [1]

I am perfectly fine with the theory: select one mechanism and see whether
it can be bridged and consequently _removed_.  To shrink the code base,
shrink text size, make the boot process less complex, more robust etc.

What I take away from this patchset, however, is that all it really does
is make the early_res stuff from x86 generic code and add a semantically
different version of the bootmem API on top of it, selectable with a config
option.  The diffstat balance is an increase of around 900 lines of code.

Note that it still uses bootmem to actually bootstrap the page allocator,
that we now have two implementations of the bootmem interface and no real
plan - as far as I am informed - to actually change this.

I also found it weird that it makes x86 skip an allocator level that all
the other architectures are using, and replaces it with 'generic' code that
nobody but x86 is using (sparc, powerpc, sh and microblaze  appear to have
lib/lmb.c at this stage and for this purpose? lmb was also suggested by
benh [4] but I have to admit I do not understand Yinghai's response to it).

When I asked Yinghai for the benefits of this change, he responded with
this [2]:

nobootmem:                                                                                                                                   
   text    data     bss     dec     hex filename                                                                                             
19185736        4148404 12170736        35504876        21dc2ec vmlinux.nobootmem
Memory: 1058662820k/1075838976k available (11388k kernel code, 2106480k absent, 15069676k reserved, 8589k data, 2744k init
[  220.947157] calling  ip_auto_config+0x0/0x24d @ 1


bootmem:                   
   text    data     bss     dec     hex filename
19188441        4153956 12170736        35513133        21de32d vmlinux.bootmem
Memory: 1058662796k/1075838976k available (11388k kernel code, 2106480k absent, 15069700k reserved, 8589k data, 2752k init
[  236.765364] calling  ip_auto_config+0x0/0x24d @ 1

but compare this with the diffstat and memory savings of Joe Perches'
latest dev_<level> macro changes [3].

So the questions I still have is this: can early_res replace bootmem
for other architectures too (and will this be pushed!) or are we stuck
with two implementations of the bootmem API forever?

Or more generically, I can not see that this series is a complete act
of reduction but I also can not see what is supposed to follow up in
order to finish it.  And apparently I am not alone with this.  Can
Yinghai or somebody else of the x86 team shed some light on this?

Thanks.

	Hannes

PS: I think it is common practice on LKML that if you have plans for
a piece of code, you involve all its users and people who last worked
on it in the discussions.  So please check your Cc lists next time
before stuff is about to get merged.  Thanks again.

[1] http://archives.free.net.ph/message/20100209.193211.48e131c7.en.html
[2] http://lkml.org/lkml/2010/3/5/414
[3] http://lkml.org/lkml/2010/3/4/17
[4] http://lkml.org/lkml/2010/2/14/321

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

further plans on bootmem, was: Re: - bootmem-avoid-dma32-zone-by-default.patch removed from -mm tree

[Johannes Weiner]

On Tue, Mar 09, 2010 at 01:49:02PM -0800, Andrew Morton wrote:
> On Tue, 09 Mar 2010 13:09:55 -0800
> Yinghai Lu <yinghai@kernel.org> wrote:
> 
> > On 03/09/2010 11:40 AM, akpm@linux-foundation.org wrote:
> > > The patch titled
> > >      bootmem: avoid DMA32 zone by default
> > > has been removed from the -mm tree.  Its filename was
> > >      bootmem-avoid-dma32-zone-by-default.patch
> > > 
> > > This patch was dropped because I'm all confused
> > > 
> > 
> > Thanks for that...
> 
> Well.  I did drop it because I'm all confused.  It may come back.
> 
> If Johannes is working in the direction of removing and simplifying
> code then that's a high priority.  So I'm waiting to see where this
> discussion leads (on the mailing list, please!)

I am not working on simplifying in this area at the moment.  I am just
questioning the discrepancy between the motivation of Yinghai's patch
series to skip bootmem on x86 and its actual outcome.

The stated reason for the series was that the amount of memory allocators
involved in bootstrapping mm on x86 'seemed a bit excessive'. [1]

I am perfectly fine with the theory: select one mechanism and see whether
it can be bridged and consequently _removed_.  To shrink the code base,
shrink text size, make the boot process less complex, more robust etc.

What I take away from this patchset, however, is that all it really does
is make the early_res stuff from x86 generic code and add a semantically
different version of the bootmem API on top of it, selectable with a config
option.  The diffstat balance is an increase of around 900 lines of code.

Note that it still uses bootmem to actually bootstrap the page allocator,
that we now have two implementations of the bootmem interface and no real
plan - as far as I am informed - to actually change this.

I also found it weird that it makes x86 skip an allocator level that all
the other architectures are using, and replaces it with 'generic' code that
nobody but x86 is using (sparc, powerpc, sh and microblaze  appear to have
lib/lmb.c at this stage and for this purpose? lmb was also suggested by
benh [4] but I have to admit I do not understand Yinghai's response to it).

When I asked Yinghai for the benefits of this change, he responded with
this [2]:

nobootmem:                                                                                                                                   
   text    data     bss     dec     hex filename                                                                                             
19185736        4148404 12170736        35504876        21dc2ec vmlinux.nobootmem
Memory: 1058662820k/1075838976k available (11388k kernel code, 2106480k absent, 15069676k reserved, 8589k data, 2744k init
[  220.947157] calling  ip_auto_config+0x0/0x24d @ 1


bootmem:                   
   text    data     bss     dec     hex filename
19188441        4153956 12170736        35513133        21de32d vmlinux.bootmem
Memory: 1058662796k/1075838976k available (11388k kernel code, 2106480k absent, 15069700k reserved, 8589k data, 2752k init
[  236.765364] calling  ip_auto_config+0x0/0x24d @ 1

but compare this with the diffstat and memory savings of Joe Perches'
latest dev_<level> macro changes [3].

So the questions I still have is this: can early_res replace bootmem
for other architectures too (and will this be pushed!) or are we stuck
with two implementations of the bootmem API forever?

Or more generically, I can not see that this series is a complete act
of reduction but I also can not see what is supposed to follow up in
order to finish it.  And apparently I am not alone with this.  Can
Yinghai or somebody else of the x86 team shed some light on this?

Thanks.

	Hannes

PS: I think it is common practice on LKML that if you have plans for
a piece of code, you involve all its users and people who last worked
on it in the discussions.  So please check your Cc lists next time
before stuff is about to get merged.  Thanks again.

[1] http://archives.free.net.ph/message/20100209.193211.48e131c7.en.html
[2] http://lkml.org/lkml/2010/3/5/414
[3] http://lkml.org/lkml/2010/3/4/17
[4] http://lkml.org/lkml/2010/2/14/321

Re: further plans on bootmem, was: Re: - bootmem-avoid-dma32-zone-by-default.patch removed from -mm tree

[Yinghai Lu]

On 03/09/2010 04:01 PM, Johannes Weiner wrote:
> On Tue, Mar 09, 2010 at 01:49:02PM -0800, Andrew Morton wrote:
>> On Tue, 09 Mar 2010 13:09:55 -0800
>> Yinghai Lu <yinghai@kernel.org> wrote:
>>
>>> On 03/09/2010 11:40 AM, akpm@linux-foundation.org wrote:
>>>> The patch titled
>>>>      bootmem: avoid DMA32 zone by default
>>>> has been removed from the -mm tree.  Its filename was
>>>>      bootmem-avoid-dma32-zone-by-default.patch
>>>>
>>>> This patch was dropped because I'm all confused
>>>>
>>>
>>> Thanks for that...
>>
>> Well.  I did drop it because I'm all confused.  It may come back.
>>
>> If Johannes is working in the direction of removing and simplifying
>> code then that's a high priority.  So I'm waiting to see where this
>> discussion leads (on the mailing list, please!)
> 
> I am not working on simplifying in this area at the moment.  I am just
> questioning the discrepancy between the motivation of Yinghai's patch
> series to skip bootmem on x86 and its actual outcome.
> 
> The stated reason for the series was that the amount of memory allocators
> involved in bootstrapping mm on x86 'seemed a bit excessive'. [1]
> 
> I am perfectly fine with the theory: select one mechanism and see whether
> it can be bridged and consequently _removed_.  To shrink the code base,
> shrink text size, make the boot process less complex, more robust etc.
> 
> What I take away from this patchset, however, is that all it really does
> is make the early_res stuff from x86 generic code and add a semantically
> different version of the bootmem API on top of it, selectable with a config
> option.  The diffstat balance is an increase of around 900 lines of code.
> 
> Note that it still uses bootmem to actually bootstrap the page allocator,
> that we now have two implementations of the bootmem interface and no real
> plan - as far as I am informed - to actually change this.
> 
> I also found it weird that it makes x86 skip an allocator level that all
> the other architectures are using, and replaces it with 'generic' code that
> nobody but x86 is using (sparc, powerpc, sh and microblaze  appear to have
> lib/lmb.c at this stage and for this purpose? lmb was also suggested by
> benh [4] but I have to admit I do not understand Yinghai's response to it).
> 

next steps:
1. create kernel/fw_memmap.c, and move common code from arch/x86/kernel/e820.c to it
2. merge lmb with fw_memmap.c/early_res.c
   so some arch that use lmb will fw_memmap/earl_res.c 

YH

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: further plans on bootmem, was: Re: - bootmem-avoid-dma32-zone-by-default.patch removed from -mm tree

[Yinghai Lu]

On 03/09/2010 04:01 PM, Johannes Weiner wrote:
> On Tue, Mar 09, 2010 at 01:49:02PM -0800, Andrew Morton wrote:
>> On Tue, 09 Mar 2010 13:09:55 -0800
>> Yinghai Lu <yinghai@kernel.org> wrote:
>>
>>> On 03/09/2010 11:40 AM, akpm@linux-foundation.org wrote:
>>>> The patch titled
>>>>      bootmem: avoid DMA32 zone by default
>>>> has been removed from the -mm tree.  Its filename was
>>>>      bootmem-avoid-dma32-zone-by-default.patch
>>>>
>>>> This patch was dropped because I'm all confused
>>>>
>>>
>>> Thanks for that...
>>
>> Well.  I did drop it because I'm all confused.  It may come back.
>>
>> If Johannes is working in the direction of removing and simplifying
>> code then that's a high priority.  So I'm waiting to see where this
>> discussion leads (on the mailing list, please!)
> 
> I am not working on simplifying in this area at the moment.  I am just
> questioning the discrepancy between the motivation of Yinghai's patch
> series to skip bootmem on x86 and its actual outcome.
> 
> The stated reason for the series was that the amount of memory allocators
> involved in bootstrapping mm on x86 'seemed a bit excessive'. [1]
> 
> I am perfectly fine with the theory: select one mechanism and see whether
> it can be bridged and consequently _removed_.  To shrink the code base,
> shrink text size, make the boot process less complex, more robust etc.
> 
> What I take away from this patchset, however, is that all it really does
> is make the early_res stuff from x86 generic code and add a semantically
> different version of the bootmem API on top of it, selectable with a config
> option.  The diffstat balance is an increase of around 900 lines of code.
> 
> Note that it still uses bootmem to actually bootstrap the page allocator,
> that we now have two implementations of the bootmem interface and no real
> plan - as far as I am informed - to actually change this.
> 
> I also found it weird that it makes x86 skip an allocator level that all
> the other architectures are using, and replaces it with 'generic' code that
> nobody but x86 is using (sparc, powerpc, sh and microblaze  appear to have
> lib/lmb.c at this stage and for this purpose? lmb was also suggested by
> benh [4] but I have to admit I do not understand Yinghai's response to it).
> 

next steps:
1. create kernel/fw_memmap.c, and move common code from arch/x86/kernel/e820.c to it
2. merge lmb with fw_memmap.c/early_res.c
   so some arch that use lmb will fw_memmap/earl_res.c 

YH

Re: further plans on bootmem, was: Re: - bootmem-avoid-dma32-zone-by-default.patch removed from -mm tree

[David Miller]

From: Johannes Weiner <hannes@cmpxchg.org>
Date: Wed, 10 Mar 2010 01:01:21 +0100

> I also found it weird that it makes x86 skip an allocator level that all
> the other architectures are using, and replaces it with 'generic' code that
> nobody but x86 is using (sparc, powerpc, sh and microblaze  appear to have
> lib/lmb.c at this stage and for this purpose? lmb was also suggested by
> benh [4] but I have to admit I do not understand Yinghai's response to it).

It kind of irked me that lmb was passed over for whatever vague reason
was given.

It works fine with memory hotplug on powerpc, so a lack of hotplug
support can't be an argument for not using it.

But hey, having yet another early memory allocator instead of making
one of the existing ones do what you want, that's fine right?

Re: further plans on bootmem, was: Re: - bootmem-avoid-dma32-zone-by-default.patch removed from -mm tree

[David Miller]

From: Johannes Weiner <hannes@cmpxchg.org>
Date: Wed, 10 Mar 2010 01:01:21 +0100

> I also found it weird that it makes x86 skip an allocator level that all
> the other architectures are using, and replaces it with 'generic' code that
> nobody but x86 is using (sparc, powerpc, sh and microblaze  appear to have
> lib/lmb.c at this stage and for this purpose? lmb was also suggested by
> benh [4] but I have to admit I do not understand Yinghai's response to it).

It kind of irked me that lmb was passed over for whatever vague reason
was given.

It works fine with memory hotplug on powerpc, so a lack of hotplug
support can't be an argument for not using it.

But hey, having yet another early memory allocator instead of making
one of the existing ones do what you want, that's fine right?

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>