[PATCH 0/4] V4 ia64 SPARSEMEM

[PATCH 0/4] V4 ia64 SPARSEMEM

[Bob Picco]

Tony,

This patchset differs very little from V3. It requires SPARSEMEM EXTREME
patches which are in 2.6.14-rcX.

ChangeLog V2:
	Jesse's review input has been applied to patches 3 and 4.
	These were all cosmetic changes.
ChangeLog V3:
	This patchset has changes for building ia64 with 
	ARCH_SPARSEMEM_EXTREME config options.  The configuration options
	SECTION_BITS and PHYSICAL_MEMORY_BITS have been removed.  I've
	chosen SECTION_SIZE_BITS to be 30. I'm sure this will be debated
	but seems adequate for the near term. 

	This patchset depends on SPARSEMEM EXTREME patch submitted to -mm.
	The -mm patch makes mem_section a one dimensional array of pointers 
	to mem_sections. The two level layout reduces the memory requirements
	of SPARSEMEM for very sparse memory but at the expense of an 
	additional shift and load instruction.

	The SPARSEMEM function memory_present, which marks sections with
	memory, is called from paging_init instead of find_memory. This is
	done because the bootmem allocator in used by SPARSEMEM to allocate
	the SPARSEMEM section arrays. This is the only code change within
	this patchset.
ChanageLog V4:
	SPARSEMEM EXTREME is now the default SPARSEMEM configuration. Thus
	the arch doesn't need to request EXTREME.  A couple of preprocessor
	#elif were cleaned up. There was a merge conflict in discontig.c's
	paging_init function and some minor code rearranging was required.

	Thanks to Kame and Yasunori for review and corrections.

	Memory models SPARSEMEM+EXTREME, DISCONTIG+VIRTUAL_MEM_MAP, 
	FLATMEM and FLATMEM+VIRTUAL_MEM_MAP were boot tested on rx2600
	by me; against -rcX.  Yasunori did the same configuration testing 
	on his Tiger4 box with Node emulation for NUMA; against -rc1.
	

thanks,

bob

RE: [PATCH 0/4] V4 ia64 SPARSEMEM

[Luck, Tony]

>This patchset differs very little from V3. It requires SPARSEMEM EXTREME
>patches which are in 2.6.14-rcX.

Bob,

Can you provide a bit more of a write up on what this change means?
Which config options have gone, what the new ones, and what are the
rules that determine what's legal.  The help options in Kconfig don't
seem very helpful here.

I think it would be a good idea to bundle a set of changes to the
default config files along with this change ... which means we should
have a discussion on what are the sensible defaults out of the new
array of options, and which of the non-default options are still
popular enough that I should make sure to test build them in my
regular scripts.  Right now this whole change is a close to being
a no-op as none of the default configs enables SPARSEMEM.

So can you throw together a straw-man proposal for each of the
default configs (maybe SGI can provide the sn2 version)?  I'm assuming
here that SPARSEMEM is going to be the preferred choice for some
system(s) ... I'm just not sure which.

-Tony

Re: [PATCH 0/4] V4 ia64 SPARSEMEM

[Luck, Tony]

On Mon, Sep 26, 2005 at 01:40:36PM -0700, Luck, Tony wrote:
> I think it would be a good idea to bundle a set of changes to the
> default config files along with this change

Especially as this patch breaks the build for bigsur_defconfig.

I think this is because this config is using:

	CONFIG_DISCONTIGMEM=y
	# CONFIG_NUMA is not set
	# CONFIG_VIRTUAL_MEM_MAP is not set

but it might be more convoluted.  The failure is early, while
building asm-offsets.s

-Tony

Re: [PATCH 0/4] V4 ia64 SPARSEMEM

[Bob Picco]

luck wrote:	[Mon Sep 26 2005, 04:40:36PM EDT]
> >This patchset differs very little from V3. It requires SPARSEMEM EXTREME
> >patches which are in 2.6.14-rcX.
> 
> Bob,
> 
> Can you provide a bit more of a write up on what this change means?
> Which config options have gone, what the new ones, and what are the
> rules that determine what's legal.  The help options in Kconfig don't
> seem very helpful here.
I'm assuming you refer to the config changes in arch/ia64/Kconfig. Well the
new ones are documented in mm/Kconfig.  If that documentation is inadequate,
then I can attempt to supply more.
> 
> I think it would be a good idea to bundle a set of changes to the
> default config files along with this change ... which means we should
> have a discussion on what are the sensible defaults out of the new
> array of options, and which of the non-default options are still
> popular enough that I should make sure to test build them in my
> regular scripts.  Right now this whole change is a close to being
> a no-op as none of the default configs enables SPARSEMEM.
This is true and just about universally true for other arches with SPARSEMEM
enabled. I took a quick look and only i386 selects SPARSEMEM for NUMA.
I couldn't find any default config files with SPARSEMEM.  I think the objective
was for more soak time before defaulting to SPARSEMEM. Also just a few
arches have SPARSEMEM support.  Dave Hansen can possibly fill in my gaps.
> 
> So can you throw together a straw-man proposal for each of the
> default configs (maybe SGI can provide the sn2 version)?  I'm assuming
> here that SPARSEMEM is going to be the preferred choice for some
> system(s) ... I'm just not sure which.
We could be more aggressive than other arches but still follow the cautious
path.  I'm reluctant to suggest this but adding a couple of default 
configuration files with SPARSEMEM might be useful.  Perhaps 
sn2_defconfig_sparsemem and defconfig_sparsemem. 

Certainly a more aggressive strategy would be any config with DISCONTIG,
VIRTUAL_MEM_MAP and  NUMA be change to SPARSEMEM and NUMA.  We've only
have performance numbers done for a large Altix configuration and rx2600.
Tiger, rx2600, Altix and HP Architecture Simulator for NUMA are the only known
machines tested with SPARSEMEM. It's doubtful there would be issues but 
doesn't seem like a good strategy.

Should you think we won't get sufficient test converage without some default
configuration files, then I'd suggest the two mentioned about be introduced.
Long term should SPARSEMEM become the default and DISCONTIG+VIRTUAL_MEM_MAP
be obsoleted then we could remove the config files.
> 
> -Tony

bob

Re: [PATCH 0/4] V4 ia64 SPARSEMEM

[Bob Picco]

luck wrote:	[Mon Sep 26 2005, 06:23:12PM EDT]
> On Mon, Sep 26, 2005 at 01:40:36PM -0700, Luck, Tony wrote:
> > I think it would be a good idea to bundle a set of changes to the
> > default config files along with this change
> 
> Especially as this patch breaks the build for bigsur_defconfig.
> 
> I think this is because this config is using:
> 
> 	CONFIG_DISCONTIGMEM=y
> 	# CONFIG_NUMA is not set
> 	# CONFIG_VIRTUAL_MEM_MAP is not set
> 
> but it might be more convoluted.  The failure is early, while
> building asm-offsets.s
> 
> -Tony
Okay. I obviously should have checked all default configs but didn't. Let me
look into this.  I see partially why this is happening but not the full 
picture yet.  I suspect my conditional change in nodedata.h.

bob

RE: [PATCH 0/4] V4 ia64 SPARSEMEM

[Luck, Tony]

>I'm assuming you refer to the config changes in arch/ia64/Kconfig. Well the
>new ones are documented in mm/Kconfig.  If that documentation is inadequate,
>then I can attempt to supply more.

There aren't any words in there about why I might want to use
sparsemem ... just a standard version of if you don't know what
to pick, go with the old discontig stuff.


>Should you think we won't get sufficient test converage without some default
>configuration files, then I'd suggest the two mentioned about be introduced.

This is definitely the prime issue ... config space on ia64
is already very fragmented, so testting of some combination
of options is rare to non-existant.  If SPARSEMEM isn't turned
on, then it won't be used by anyone that isn't already reading
the lhms list.  Coupled with the lack of sales pitch in the
Kconfig help files, it looks like this isn't going to be used
by anyone!

>Long term should SPARSEMEM become the default and DISCONTIG+VIRTUAL_MEM_MAP
>be obsoleted then we could remove the config files.

If benchmarks show no difference, then I'll consolidate the
configuration options.  I still think that VIRTUAL_MEM_MAP
has a great deal of elegance to it ... auto-sizing to just
about any degree of sparseness, but I think we need to
simplify.

-Tony

Re: [PATCH 0/4] V4 ia64 SPARSEMEM

[David Mosberger-Tang]

On 9/26/05, Luck, Tony <tony.luck@intel.com> wrote:

> >Long term should SPARSEMEM become the default and DISCONTIG+VIRTUAL_MEM_MAP
> >be obsoleted then we could remove the config files.
>
> If benchmarks show no difference, then I'll consolidate the
> configuration options.  I still think that VIRTUAL_MEM_MAP
> has a great deal of elegance to it ... auto-sizing to just
> about any degree of sparseness, but I think we need to
> simplify.

Benchmarks cannot prove the absence of a performance difference in
*general*, they can only do that for specific tests and workloads. 
So, unless SPARSEMEM actually performs *better* on some benchmarks
(and no worse on others), the proper course seems to be to stick with
VIRTUAL_MEM_MAP where SPARSEMEM isn't needed.

I have voice my objection in the past to Bob's suggestion that
SPARSEMEM should replace VIRTUAL_MEM_MAP and since then I haven't seen
any convincing argument why that should be the case, so I do not
understand why Bob keeps bringing that up.

  --david
--
Mosberger Consulting LLC, voice/fax: 510-744-9372,
http://www.mosberger-consulting.com/
35706 Runckel Lane, Fremont, CA 94536

Re: [PATCH 0/4] V4 ia64 SPARSEMEM

[Yasunori Goto]

Hi David-san.

> > >Long term should SPARSEMEM become the default and DISCONTIG+VIRTUAL_MEM_MAP
> > >be obsoleted then we could remove the config files.
> >
> > If benchmarks show no difference, then I'll consolidate the
> > configuration options.  I still think that VIRTUAL_MEM_MAP
> > has a great deal of elegance to it ... auto-sizing to just
> > about any degree of sparseness, but I think we need to
> > simplify.
> 
> Benchmarks cannot prove the absence of a performance difference in
> *general*, they can only do that for specific tests and workloads. 
> So, unless SPARSEMEM actually performs *better* on some benchmarks
> (and no worse on others), the proper course seems to be to stick with
> VIRTUAL_MEM_MAP where SPARSEMEM isn't needed.
> 
> I have voice my objection in the past to Bob's suggestion that
> SPARSEMEM should replace VIRTUAL_MEM_MAP and since then I haven't seen
> any convincing argument why that should be the case, so I do not
> understand why Bob keeps bringing that up.

SPARSEMEM is essential implementation for "memory hotplug" feature now.
It gives basis that system can manage to disable/enable a part of
memory. And it is not only for ia64 but also for ppc64 and x86_64.

Bye.
-- 
Yasunori Goto 

Re: [PATCH 0/4] V4 ia64 SPARSEMEM

[Jack Steiner]

> We could be more aggressive than other arches but still follow the cautious
> path.  I'm reluctant to suggest this but adding a couple of default 
> configuration files with SPARSEMEM might be useful.  Perhaps 
> sn2_defconfig_sparsemem and defconfig_sparsemem. 
> 

Although many developers at SGI use SN-specific configuration files,
customers run SuSE & RH and use kernels built with the GENERIC config 
since the kernel binary runs on all hardware vendors platforms.
It is important to make the default config file run efficiently
on all platforms.

I haven't looked in detail at SPARSEMEM recently - is one config
file for ALL hardware platforms easy to do without significantly 
compromising performance of any individual platform?


-- 
Jack

Re: [PATCH 0/4] V4 ia64 SPARSEMEM

[Bob Picco]

luck wrote:	[Mon Sep 26 2005, 08:03:09PM EDT]
> >I'm assuming you refer to the config changes in arch/ia64/Kconfig. Well the
> >new ones are documented in mm/Kconfig.  If that documentation is inadequate,
> >then I can attempt to supply more.
> 
> There aren't any words in there about why I might want to use
> sparsemem ... just a standard version of if you don't know what
> to pick, go with the old discontig stuff.
I can't say there is disagreement with this point. So you'd obviously like
more help text in mm/Kconfig?
> 
> 
> >Should you think we won't get sufficient test converage without some default
> >configuration files, then I'd suggest the two mentioned about be introduced.
> 
> This is definitely the prime issue ... config space on ia64
> is already very fragmented, so testting of some combination
> of options is rare to non-existant.  If SPARSEMEM isn't turned
> on, then it won't be used by anyone that isn't already reading
> the lhms list.  Coupled with the lack of sales pitch in the
> Kconfig help files, it looks like this isn't going to be used
> by anyone!
Yep. I thought this is what you are after.
> 
> >Long term should SPARSEMEM become the default and DISCONTIG+VIRTUAL_MEM_MAP
> >be obsoleted then we could remove the config files.
> 
> If benchmarks show no difference, then I'll consolidate the
> configuration options.  I still think that VIRTUAL_MEM_MAP
> has a great deal of elegance to it ... auto-sizing to just
> about any degree of sparseness, but I think we need to
> simplify.
Well I'm not attempting to be critical of VIRTUAL_MEM_MAP. Obviously it 
satisfies a solution which was previously unanswered. I'm just hoping to reduce
our configuration options.  We aren't even remotely close to possibly
considering this.
> 
The bigsur problem isn't nodedata.h related. It's in arch/ia64/Kconfig which
figures. I should have this resolved some time today after checking
the default configs.
> -Tony
> -
bob

Re: [PATCH 0/4] V4 ia64 SPARSEMEM

[Bob Picco]

Jack Steiner wrote:	[Tue Sep 27 2005, 08:45:00AM EDT]
> > We could be more aggressive than other arches but still follow the cautious
> > path.  I'm reluctant to suggest this but adding a couple of default 
> > configuration files with SPARSEMEM might be useful.  Perhaps 
> > sn2_defconfig_sparsemem and defconfig_sparsemem. 
> > 
> 
> Although many developers at SGI use SN-specific configuration files,
> customers run SuSE & RH and use kernels built with the GENERIC config 
> since the kernel binary runs on all hardware vendors platforms.
> It is important to make the default config file run efficiently
> on all platforms.
okay.
> 
> I haven't looked in detail at SPARSEMEM recently - is one config
> file for ALL hardware platforms easy to do without significantly 
> compromising performance of any individual platform?
I don't know whether that can be achieved.  We've really done limited 
performance comparisons. It would have to be investigated.
> 
> 
> -- 
> Jack
bob

Re: [PATCH 0/4] V4 ia64 SPARSEMEM

[Bob Picco]

Bob Picco wrote:	[Tue Sep 27 2005, 10:09:20AM EDT]
> luck wrote:	[Mon Sep 26 2005, 08:03:09PM EDT]
> > >I'm assuming you refer to the config changes in arch/ia64/Kconfig. Well the
> > >new ones are documented in mm/Kconfig.  If that documentation is inadequate,
> > >then I can attempt to supply more.
> > 
> > There aren't any words in there about why I might want to use
> > sparsemem ... just a standard version of if you don't know what
> > to pick, go with the old discontig stuff.
> I can't say there is disagreement with this point. So you'd obviously like
> more help text in mm/Kconfig?
> > 
> > 
> > >Should you think we won't get sufficient test converage without some default
> > >configuration files, then I'd suggest the two mentioned about be introduced.
> > 
> > This is definitely the prime issue ... config space on ia64
> > is already very fragmented, so testting of some combination
> > of options is rare to non-existant.  If SPARSEMEM isn't turned
> > on, then it won't be used by anyone that isn't already reading
> > the lhms list.  Coupled with the lack of sales pitch in the
> > Kconfig help files, it looks like this isn't going to be used
> > by anyone!
> Yep. I thought this is what you are after.
> > 
> > >Long term should SPARSEMEM become the default and DISCONTIG+VIRTUAL_MEM_MAP
> > >be obsoleted then we could remove the config files.
> > 
> > If benchmarks show no difference, then I'll consolidate the
> > configuration options.  I still think that VIRTUAL_MEM_MAP
> > has a great deal of elegance to it ... auto-sizing to just
> > about any degree of sparseness, but I think we need to
> > simplify.
> Well I'm not attempting to be critical of VIRTUAL_MEM_MAP. Obviously it 
> satisfies a solution which was previously unanswered. I'm just hoping to reduce
> our configuration options.  We aren't even remotely close to possibly
> considering this.
> > 
> The bigsur problem isn't nodedata.h related. It's in arch/ia64/Kconfig which
> figures. I should have this resolved some time today after checking
> the default configs.
> > -Tony
> > -
> bob
Okay. There was a bug in my Kconfig.  All defconfigs and my own memory model
configs build correctly and produce the correct config options.  Below is
a replacement for patch 1/4 which I've included for your config testing.  I'll
post V5 patchset when we agree on where we want to proceed with config(s).  

I'm thinking all we need is sparsemem_defconfig which is defconfig but with
the SPARSEMEM memory model.  

thanks,

bob

 arch/ia64/Kconfig     |   94 +++++++++++++++++++++++++++++---------------------
 arch/ia64/mm/Makefile |    5 +-
 2 files changed, 58 insertions(+), 41 deletions(-)

Index: linux-2.6.14-rc2/arch/ia64/Kconfig
=================================--- linux-2.6.14-rc2.orig/arch/ia64/Kconfig	2005-09-26 19:16:57.000000000 -0400
+++ linux-2.6.14-rc2/arch/ia64/Kconfig	2005-09-27 14:31:21.000000000 -0400
@@ -63,8 +63,6 @@ config IA64_GENERIC
 	select ACPI
 	select NUMA
 	select ACPI_NUMA
-	select VIRTUAL_MEM_MAP
-	select DISCONTIGMEM
 	help
 	  This selects the system type of your hardware.  A "generic" kernel
 	  will run on any supported IA-64 system.  However, if you configure
@@ -176,40 +174,6 @@ config IA64_L1_CACHE_SHIFT
 	default "6" if ITANIUM
 
 # align cache-sensitive data to 64 bytes
-config NUMA
-	bool "NUMA support"
-	depends on !IA64_HP_SIM
-	default y if IA64_SGI_SN2
-	select ACPI_NUMA
-	help
-	  Say Y to compile the kernel to support NUMA (Non-Uniform Memory
-	  Access).  This option is for configuring high-end multiprocessor
-	  server systems.  If in doubt, say N.
-
-config VIRTUAL_MEM_MAP
-	bool "Virtual mem map"
-	default y if !IA64_HP_SIM
-	help
-	  Say Y to compile the kernel with support for a virtual mem map.
-	  This code also only takes effect if a memory hole of greater than
-	  1 Gb is found during boot.  You must turn this option on if you
-	  require the DISCONTIGMEM option for your machine. If you are
-	  unsure, say Y.
-
-config HOLES_IN_ZONE
-	bool
-	default y if VIRTUAL_MEM_MAP
-
-config ARCH_DISCONTIGMEM_ENABLE
-	bool "Discontiguous memory support"
-	depends on (IA64_DIG || IA64_SGI_SN2 || IA64_GENERIC || IA64_HP_ZX1 || IA64_HP_ZX1_SWIOTLB) && NUMA && VIRTUAL_MEM_MAP
-	default y if (IA64_SGI_SN2 || IA64_GENERIC) && NUMA
-	help
-	  Say Y to support efficient handling of discontiguous physical memory,
-	  for architectures which are either NUMA (Non-Uniform Memory Access)
-	  or have huge holes in the physical address space for other reasons.
-	  See <file:Documentation/vm/numa> for more.
-
 config IA64_CYCLONE
 	bool "Cyclone (EXA) Time Source support"
 	help
@@ -232,8 +196,10 @@ config IA64_SGI_SN_XP
 	  based on a network adapter and DMA messaging.
 
 config FORCE_MAX_ZONEORDER
-	int
-	default "18"
+	int "MAX_ORDER (11 - 17)"  if !HUGETLB_PAGE
+	range 11 17  if !HUGETLB_PAGE
+	default "17" if HUGETLB_PAGE
+	default "11"
 
 config SMP
 	bool "Symmetric multi-processing support"
@@ -298,6 +264,58 @@ config PREEMPT
 
 source "mm/Kconfig"
 
+config ARCH_SELECT_MEMORY_MODEL
+	def_bool y
+
+config ARCH_DISCONTIGMEM_ENABLE
+	def_bool y
+	help
+	  Say Y to support efficient handling of discontiguous physical memory,
+	  for architectures which are either NUMA (Non-Uniform Memory Access)
+	  or have huge holes in the physical address space for other reasons.
+ 	  See <file:Documentation/vm/numa> for more.
+
+config ARCH_FLATMEM_ENABLE
+	def_bool y
+
+config ARCH_SPARSEMEM_ENABLE
+	def_bool y
+	depends on ARCH_DISCONTIGMEM_ENABLE
+
+config ARCH_DISCONTIGMEM_DEFAULT
+	def_bool y if (IA64_SGI_SN2 || IA64_GENERIC || IA64_HP_ZX1 || IA64_HP_ZX1_SWIOTLB)
+	depends on ARCH_DISCONTIGMEM_ENABLE
+
+config NUMA
+	bool "NUMA support"
+	depends on !IA64_HP_SIM && !FLATMEM
+	default y if IA64_SGI_SN2
+	help
+	  Say Y to compile the kernel to support NUMA (Non-Uniform Memory
+	  Access).  This option is for configuring high-end multiprocessor
+	  server systems.  If in doubt, say N.
+
+# VIRTUAL_MEM_MAP and FLAT_NODE_MEM_MAP are functionally equivalent.
+# VIRTUAL_MEM_MAP has been retained for historical reasons.
+config VIRTUAL_MEM_MAP
+	bool "Virtual mem map"
+	depends on !SPARSEMEM
+	default y if !IA64_HP_SIM
+	help
+	  Say Y to compile the kernel with support for a virtual mem map.
+	  This code also only takes effect if a memory hole of greater than
+	  1 Gb is found during boot.  You must turn this option on if you
+	  require the DISCONTIGMEM option for your machine. If you are
+	  unsure, say Y.
+
+config HOLES_IN_ZONE
+	bool
+	default y if VIRTUAL_MEM_MAP
+
+config HAVE_ARCH_EARLY_PFN_TO_NID
+	def_bool y
+	depends on NEED_MULTIPLE_NODES
+
 config IA32_SUPPORT
 	bool "Support for Linux/x86 binaries"
 	help
Index: linux-2.6.14-rc2/arch/ia64/mm/Makefile
=================================--- linux-2.6.14-rc2.orig/arch/ia64/mm/Makefile	2005-09-26 19:16:57.000000000 -0400
+++ linux-2.6.14-rc2/arch/ia64/mm/Makefile	2005-09-26 19:49:27.000000000 -0400
@@ -7,6 +7,5 @@ obj-y := init.o fault.o tlb.o extable.o
 obj-$(CONFIG_HUGETLB_PAGE) += hugetlbpage.o
 obj-$(CONFIG_NUMA)	   += numa.o
 obj-$(CONFIG_DISCONTIGMEM) += discontig.o
-ifndef CONFIG_DISCONTIGMEM
-obj-y += contig.o
-endif
+obj-$(CONFIG_SPARSEMEM)	   += discontig.o
+obj-$(CONFIG_FLATMEM)	   += contig.o

RE: [PATCH 0/4] V4 ia64 SPARSEMEM

[Luck, Tony]

>Okay. There was a bug in my Kconfig.  All defconfigs and my own memory model
>configs build correctly and produce the correct config options.  Below is
>a replacement for patch 1/4 which I've included for your config testing.  I'll
>post V5 patchset when we agree on where we want to proceed with config(s).  

Yup, all my configs now build too (includes some of the distributed ones
with CONFIG_SMP turned off).  I booted three sample kernels (zx1,
generic and generic with s/discontig/sparse/) on rx2620 with nothing
odd happening.

>I'm thinking all we need is sparsemem_defconfig which is defconfig but with
>the SPARSEMEM memory model.

That would do as a starting point ... it gives me an easy hook
into my automatic build script to make sure that SPARSEMEM is being
test compiled.

Do you have, or can you generate, the full list of all the combinations
of memory model related options that Kconfig will accept?

-Tony