2.6.0 test3 does not boot on ia64 NUMA

2.6.0 test3 does not boot on ia64 NUMA

[Xavier Bru]

... but 2.5.72 was OK :-)

Booting on a 4 nodes Itanium machine, with 
	CONFIG_NUMA=y
	CONFIG_ACPI_NUMA=y
	CONFIG_DISCONTIGMEM=y
	# CONFIG_VIRTUAL_MEM_MAP is not set

system hangs after a message:
"On node 1 totalpages: <a very very BIG number that looks garbage>"

Having a look to the code, it seems that problem is due to memory
initialisation changes:
 . unlike 2.5.72, find_memory() is now called before
acpi_numa_init(), and supposes that numnodes = 1. So only 1
bootmem_data_t struct is initialised.
 . acpi_numa_init() then finds in SRAT that 4 nodes exist.
 . when paging_init calls discontig_paging_init only bootmem_data_t
for node 0 is initialized, and garbage is found for nodes 1 to 3.

I tried to put find_memory()  after acpi_numa_init() as in 2.5.72, but 
now we get:
     bootmem alloc of 100 bytes failed! in acpi_table_init()
that now uses the bootmem allocator.

I wonder what should be the right order for initialisation, and if
there are ia64 platforms running 2.6.0 with CONFIG_NUMA. Any help is
apreciated.
Thanks in advance.
Xavier.

-- 

 Sincères salutations.
_____________________________________________________________________
 
Xavier BRU                 BULL ISD/R&D/INTEL office:     FREC B1-422
tel : +33 (0)4 76 29 77 45                    http://www-frec.bull.fr
fax : +33 (0)4 76 29 77 70                 mailto:Xavier.Bru@bull.net
addr: BULL, 1 rue de Provence, BP 208, 38432 Echirolles Cedex, FRANCE
_____________________________________________________________________

Re: 2.6.0 test3 does not boot on ia64 NUMA

[Martin Hicks]

On Tue, Aug 26, 2003 at 05:39:32PM +0200, Xavier Bru wrote:
> 
> I tried to put find_memory()  after acpi_numa_init() as in 2.5.72, but 
> now we get:
>      bootmem alloc of 100 bytes failed! in acpi_table_init()
> that now uses the bootmem allocator.
> 
> I wonder what should be the right order for initialisation, and if
> there are ia64 platforms running 2.6.0 with CONFIG_NUMA. Any help is
> apreciated.

Jesse Barnes has an unreleased patch to make SGI SN2 work, which does
move the find_memory() after acpi_numa_init().  There is another patch,
which I've attached, which removes the bootmem alloc of memory in ACPI.

I haven't played with this recently, so your mileage may vary.  Jesse
should be back from vacation next week, and should release updated SN2
patches then.

mh

-- 
Wild Open Source Inc.                  mort@wildopensource.com


# This is a BitKeeper generated patch for the following project:
# Project Name: Linux kernel tree
# This patch format is intended for GNU patch command version 2.5 or higher.
# This patch includes the following deltas:
#	           ChangeSet	1.1395  -> 1.1396 
#	drivers/acpi/tables.c	1.14    -> 1.15   
#
# The following is the BitKeeper ChangeSet Log
# --------------------------------------------
# 03/07/30	jbarnes@tomahawk.engr.sgi.com	1.1396
# don't use alloc_bootmem in acpi table init
# --------------------------------------------
#
diff -Nru a/drivers/acpi/tables.c b/drivers/acpi/tables.c
--- a/drivers/acpi/tables.c	Wed Jul 30 11:45:27 2003
+++ b/drivers/acpi/tables.c	Wed Jul 30 11:45:27 2003
@@ -69,7 +69,8 @@
 
 static unsigned long		sdt_pa;		/* Physical Address */
 static unsigned long		sdt_count;	/* Table count */
-static struct acpi_table_sdt	*sdt_entry;
+
+static struct acpi_table_sdt	sdt_entry[ACPI_MAX_TABLES];
 
 void
 acpi_table_print (
@@ -413,12 +414,6 @@
 			sdt_count = ACPI_MAX_TABLES;
 		}
 
-		sdt_entry = alloc_bootmem(sdt_count * sizeof(struct acpi_table_sdt));
-		if (!sdt_entry) {
-			printk(KERN_ERR "ACPI: Could not allocate mem for SDT entries!\n");
-			return -ENOMEM;
-		}
-
 		for (i = 0; i < sdt_count; i++)
 			sdt_entry[i].pa = (unsigned long) mapped_xsdt->entry[i];
 	}
@@ -463,12 +458,6 @@
 			printk(KERN_WARNING PREFIX "Truncated %lu RSDT entries\n",
 				(sdt_count - ACPI_MAX_TABLES));
 			sdt_count = ACPI_MAX_TABLES;
-		}
-
-		sdt_entry = alloc_bootmem(sdt_count * sizeof(struct acpi_table_sdt));
-		if (!sdt_entry) {
-			printk(KERN_ERR "ACPI: Could not allocate mem for SDT entries!\n");
-			return -ENOMEM;
 		}
 
 		for (i = 0; i < sdt_count; i++)

Re: 2.6.0 test3 does not boot on ia64 NUMA

[Xavier Bru]

Martin,

Thanks for answering my mail. I applied Jesse's patch ([PATCH] latest
discontig mail dated Fri, 8 Aug 2003) but this does'nt build with:
	CONFIG_NUMA=y
	CONFIG_ACPI_NUMA=y
	CONFIG_DISCONTIGMEM=y
	# CONFIG_VIRTUAL_MEM_MAP is not set
I get:
include/linux/mm.h: In function `lowmem_page_address':
include/linux/mm.h:349: `mem_map' undeclared (first use in this function)

It seems that this only works on Altix, unless I missed something.

BTW, applying your patch for removing alloc_bootmem and moving back
find_memory() to its previous place, system boots OK.
Thanks again for your help.

Xavier
-----------------------------------------------------------------------------
diff --exclude-from /users/xb/proc/diff.exclude -r 0t4/arch/ia64/kernel/setup.c linux-2.6.0-test4/arch/ia64/kernel/setup.c
374a375
> 	find_memory();
388,389d388
< 	find_memory();
< 
diff --exclude-from /users/xb/proc/diff.exclude -r 0t4/drivers/acpi/tables.c linux-2.6.0-test4/drivers/acpi/tables.c
72,73c72
< 
< static struct acpi_table_sdt	sdt_entry[ACPI_MAX_TABLES];
---
> static struct acpi_table_sdt	*sdt_entry;
428a428,433
> 		sdt_entry = alloc_bootmem(sdt_count * sizeof(struct acpi_table_sdt));
> 		if (!sdt_entry) {
> 			printk(KERN_ERR "ACPI: Could not allocate mem for SDT entries!\n");
> 			return -ENOMEM;
> 		}
> 
474a480,485
> 		sdt_entry = alloc_bootmem(sdt_count * sizeof(struct acpi_table_sdt));
> 		if (!sdt_entry) {
> 			printk(KERN_ERR "ACPI: Could not allocate mem for SDT entries!\n");
> 			return -ENOMEM;
> 		}
> 

-- 

 Sincères salutations.
_____________________________________________________________________
 
Xavier BRU                 BULL ISD/R&D/INTEL office:     FREC B1-422
tel : +33 (0)4 76 29 77 45                    http://www-frec.bull.fr
fax : +33 (0)4 76 29 77 70                 mailto:Xavier.Bru@bull.net
addr: BULL, 1 rue de Provence, BP 208, 38432 Echirolles Cedex, FRANCE
_____________________________________________________________________


Martin Hicks writes:
 > 
 > 
 > On Tue, Aug 26, 2003 at 05:39:32PM +0200, Xavier Bru wrote:
 > > 
 > > I tried to put find_memory()  after acpi_numa_init() as in 2.5.72, but 
 > > now we get:
 > >      bootmem alloc of 100 bytes failed! in acpi_table_init()
 > > that now uses the bootmem allocator.
 > > 
 > > I wonder what should be the right order for initialisation, and if
 > > there are ia64 platforms running 2.6.0 with CONFIG_NUMA. Any help is
 > > apreciated.
 > 
 > Jesse Barnes has an unreleased patch to make SGI SN2 work, which does
 > move the find_memory() after acpi_numa_init().  There is another patch,
 > which I've attached, which removes the bootmem alloc of memory in ACPI.
 > 
 > I haven't played with this recently, so your mileage may vary.  Jesse
 > should be back from vacation next week, and should release updated SN2
 > patches then.
 > 
 > mh
 > 
 > -- 
 > Wild Open Source Inc.                  mort@wildopensource.com
 > 
 > 
 > # This is a BitKeeper generated patch for the following project:
 > # Project Name: Linux kernel tree
 > # This patch format is intended for GNU patch command version 2.5 or higher.
 > # This patch includes the following deltas:
 > #	           ChangeSet	1.1395  -> 1.1396 
 > #	drivers/acpi/tables.c	1.14    -> 1.15   
 > #
 > # The following is the BitKeeper ChangeSet Log
 > # --------------------------------------------
 > # 03/07/30	jbarnes@tomahawk.engr.sgi.com	1.1396
 > # don't use alloc_bootmem in acpi table init
 > # --------------------------------------------
 > #
 > diff -Nru a/drivers/acpi/tables.c b/drivers/acpi/tables.c
 > --- a/drivers/acpi/tables.c	Wed Jul 30 11:45:27 2003
 > +++ b/drivers/acpi/tables.c	Wed Jul 30 11:45:27 2003
 > @@ -69,7 +69,8 @@
 >  
 >  static unsigned long		sdt_pa;		/* Physical Address */
 >  static unsigned long		sdt_count;	/* Table count */
 > -static struct acpi_table_sdt	*sdt_entry;
 > +
 > +static struct acpi_table_sdt	sdt_entry[ACPI_MAX_TABLES];
 >  
 >  void
 >  acpi_table_print (
 > @@ -413,12 +414,6 @@
 >  			sdt_count = ACPI_MAX_TABLES;
 >  		}
 >  
 > -		sdt_entry = alloc_bootmem(sdt_count * sizeof(struct acpi_table_sdt));
 > -		if (!sdt_entry) {
 > -			printk(KERN_ERR "ACPI: Could not allocate mem for SDT entries!\n");
 > -			return -ENOMEM;
 > -		}
 > -
 >  		for (i = 0; i < sdt_count; i++)
 >  			sdt_entry[i].pa = (unsigned long) mapped_xsdt->entry[i];
 >  	}
 > @@ -463,12 +458,6 @@
 >  			printk(KERN_WARNING PREFIX "Truncated %lu RSDT entries\n",
 >  				(sdt_count - ACPI_MAX_TABLES));
 >  			sdt_count = ACPI_MAX_TABLES;
 > -		}
 > -
 > -		sdt_entry = alloc_bootmem(sdt_count * sizeof(struct acpi_table_sdt));
 > -		if (!sdt_entry) {
 > -			printk(KERN_ERR "ACPI: Could not allocate mem for SDT entries!\n");
 > -			return -ENOMEM;
 >  		}
 >  
 >  		for (i = 0; i < sdt_count; i++)
 > -
 > To unsubscribe from this list: send the line "unsubscribe linux-ia64" in
 > the body of a message to majordomo@vger.kernel.org
 > More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: 2.6.0 test3 does not boot on ia64 NUMA

[Martin Hicks]

[-- Attachment #1: Type: text/plain, Size: 723 bytes --]



On Thu, Aug 28, 2003 at 06:38:41PM +0200, Xavier Bru wrote:
> 
> Martin,
> 
> Thanks for answering my mail. I applied Jesse's patch ([PATCH] latest
> discontig mail dated Fri, 8 Aug 2003) but this does'nt build with:
> 	CONFIG_NUMA=y
> 	CONFIG_ACPI_NUMA=y
> 	CONFIG_DISCONTIGMEM=y
> 	# CONFIG_VIRTUAL_MEM_MAP is not set
> I get:
> include/linux/mm.h: In function `lowmem_page_address':
> include/linux/mm.h:349: `mem_map' undeclared (first use in this function)
> 
> It seems that this only works on Altix, unless I missed something.

I'm pretty sure that you need CONFIG_VIRTUAL_MEM_MAP=y if you have
CONFIG_DISCONTIGMEM=y

mh

-- 
Wild Open Source Inc.                  mort@wildopensource.com

[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]

Re: 2.6.0 test3 does not boot on ia64 NUMA

[Xavier Bru]

Martin,

Thanks for the information (Until now CONFIG_DISCONTIGMEM was
exclusive with CONFIG_VIRTUAL_MEM_MAP).

System now builds OK with Jesse's patch + the acpi alloc_bootmem patch.

Unfortunately, it dies somewhere after entering paging_init(), and before
printing "On node %d totalpages:":

ifa=0xf000ff54f7c5722b
iip=0xa0000001007780d0
0xa0000001007780d0 <__alloc_bootmem_core+592>:	[MII]       ld8 r17=[r19],8

last traces are:

Mca related initilaization done
kernel unaligned access to 0xf000ff54f000eefb ip=0xa0000001007780d0
kernel unaligned access to 0xf000ff54f000ef03 ip=0xa0000001007780d0
kernel unaligned access to 0xf000ff54f000ef0b ip=0xa0000001007780d0


I will try to look at this a bit more if I find some more time.

Thanks again.

Xavier

-- 

 Sincères salutations.
_____________________________________________________________________
 
Xavier BRU                 BULL ISD/R&D/INTEL office:     FREC B1-422
tel : +33 (0)4 76 29 77 45                    http://www-frec.bull.fr
fax : +33 (0)4 76 29 77 70                 mailto:Xavier.Bru@bull.net
addr: BULL, 1 rue de Provence, BP 208, 38432 Echirolles Cedex, FRANCE
_____________________________________________________________________


Martin Hicks writes:
 > 
 > 
 > On Thu, Aug 28, 2003 at 06:38:41PM +0200, Xavier Bru wrote:
 > > 
 > > Martin,
 > > 
 > > Thanks for answering my mail. I applied Jesse's patch ([PATCH] latest
 > > discontig mail dated Fri, 8 Aug 2003) but this does'nt build with:
 > > 	CONFIG_NUMA=y
 > > 	CONFIG_ACPI_NUMA=y
 > > 	CONFIG_DISCONTIGMEM=y
 > > 	# CONFIG_VIRTUAL_MEM_MAP is not set
 > > I get:
 > > include/linux/mm.h: In function `lowmem_page_address':
 > > include/linux/mm.h:349: `mem_map' undeclared (first use in this function)
 > > 
 > > It seems that this only works on Altix, unless I missed something.
 > 
 > I'm pretty sure that you need CONFIG_VIRTUAL_MEM_MAP=y if you have
 > CONFIG_DISCONTIGMEM=y
 > 
 > mh
 > 
 > -- 
 > Wild Open Source Inc.                  mort@wildopensource.com

Re: 2.6.0 test3 does not boot on ia64 NUMA

[Martin Hicks]

On Fri, Aug 29, 2003 at 06:41:56PM +0200, Xavier Bru wrote:
> 
> printing "On node %d totalpages:":
> 
> ifa=0xf000ff54f7c5722b
> iip=0xa0000001007780d0
> 0xa0000001007780d0 <__alloc_bootmem_core+592>:	[MII]       ld8 r17=[r19],8
> 
> 
> I will try to look at this a bit more if I find some more time.

Do you have this changeset in your tree?  It sounds like a similar
problem to what I was seeing.

ChangeSet@1.1292.1.3, 2003-08-25 12:23:20-07:00, mort@wildopensource.com
  [PATCH] ia64: paddr_to_nid fixup
  
  Here is a small patch for paddr_to_nid().  This fix is already in 2.4
  and is used in the case where a NUMA kernel is running on a machine
  without a SRAT ACPI table.  Without this patch the node info is not
  correctly located.


mh

-- 
Wild Open Source Inc.                  mort@wildopensource.com

Re: 2.6.0 test3 does not boot on ia64 NUMA

[Xavier Bru]

Martin,

I checked that the code for changeset 1.1292.1.3 is already present in 2.6.0 test4. 

# This is a BitKeeper generated patch for the following project:
# Project Name: Linux kernel tree
# This patch format is intended for GNU patch command version 2.5 or higher.
# This patch includes the following deltas:
#	           ChangeSet	1.1292.1.2 -> 1.1292.1.3
#	 arch/ia64/mm/numa.c	1.1     -> 1.2    
#
# The following is the BitKeeper ChangeSet Log
# --------------------------------------------
# 03/08/25	mort@wildopensource.com	1.1292.1.3
# [PATCH] ia64: paddr_to_nid fixup
# 
# Here is a small patch for paddr_to_nid().  This fix is already in 2.4
# and is used in the case where a NUMA kernel is running on a machine
# without a SRAT ACPI table.  Without this patch the node info is not
# correctly located.
# --------------------------------------------
#
diff -Nru a/arch/ia64/mm/numa.c b/arch/ia64/mm/numa.c
--- a/arch/ia64/mm/numa.c	Wed Aug 27 00:40:34 2003
+++ b/arch/ia64/mm/numa.c	Wed Aug 27 00:40:34 2003
@@ -42,5 +42,5 @@
 		    paddr < node_memblk[i].start_paddr + node_memblk[i].size)
 			break;
 
-	return (i < num_memblks) ? node_memblk[i].nid : -1;
+	return (i < num_memblks) ? node_memblk[i].nid : (num_memblks ? -1 : 0);
 }

Thanks for your help.
Xavier 

-- 

 Sincères salutations.
_____________________________________________________________________
 
Xavier BRU                 BULL ISD/R&D/INTEL office:     FREC B1-422
tel : +33 (0)4 76 29 77 45                    http://www-frec.bull.fr
fax : +33 (0)4 76 29 77 70                 mailto:Xavier.Bru@bull.net
addr: BULL, 1 rue de Provence, BP 208, 38432 Echirolles Cedex, FRANCE
_____________________________________________________________________



Martin Hicks writes:
 > 
 > 
 > On Fri, Aug 29, 2003 at 06:41:56PM +0200, Xavier Bru wrote:
 > > 
 > > printing "On node %d totalpages:":
 > > 
 > > ifa=0xf000ff54f7c5722b
 > > iip=0xa0000001007780d0
 > > 0xa0000001007780d0 <__alloc_bootmem_core+592>:	[MII]       ld8 r17=[r19],8
 > > 
 > > 
 > > I will try to look at this a bit more if I find some more time.
 > 
 > Do you have this changeset in your tree?  It sounds like a similar
 > problem to what I was seeing.
 > 
 > ChangeSet@1.1292.1.3, 2003-08-25 12:23:20-07:00, mort@wildopensource.com
 >   [PATCH] ia64: paddr_to_nid fixup
 >   
 >   Here is a small patch for paddr_to_nid().  This fix is already in 2.4
 >   and is used in the case where a NUMA kernel is running on a machine
 >   without a SRAT ACPI table.  Without this patch the node info is not
 >   correctly located.
 > 
 > 
 > mh
 > 
 > -- 
 > Wild Open Source Inc.                  mort@wildopensource.com
 > -
 > To unsubscribe from this list: send the line "unsubscribe linux-ia64" in
 > the body of a message to majordomo@vger.kernel.org
 > More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: 2.6.0 test3 does not boot on ia64 NUMA

[Xavier Bru]

Hello Martin,

I finally found the reason for crashing at init time:
On node 0, our test configuration has:
 2 GB of memory at address 0
 2 GB of memory at address 6 GB (due to PCI hole).

Current code for acpi_numa_memory_affinity_init ignores physical
memory bank if the hole (4GB) is bigger than the bank (2 GB).
As the node_memblk is not present for address 6 GB, paddr_to_nid
returns -1 and alloc_bootmem_pages_node crashes with a Null pointer.

As we now have CONFIG_VIRTUAL_MEM_MAP=y, I suppose we should also use
sparse memory in same node. (Am I right ?)

Now 2.6.0 test4  boots OK in NUMA with:

. Jesse's discontig patch
. Tony's trim patch
. alloc_bootmem patch
. and this small one :-)

diff --exclude-from /users/xb/proc/diff.exclude -Nur linux-2.6.0-test4/arch/ia64/kernel/acpi.c 0t4/arch/ia64/kernel/acpi.c
--- linux-2.6.0-test4/arch/ia64/kernel/acpi.c	2003-08-23 01:55:43.000000000 +0200
+++ 0t4/arch/ia64/kernel/acpi.c	2003-09-02 15:37:17.000000000 +0200
@@ -423,9 +423,8 @@
 
 	if (min_hole_size) {
 		if (min_hole_size > size) {
-			printk(KERN_ERR "Too huge memory hole. Ignoring %ld MBytes at %lx\n",
+			printk(KERN_WARNING "Huge memory hole. Using %ld MBytes at %lx\n",
 			       size/(1024*1024), paddr);
-			return;
 		}
 	}
 
Thanks again for your help.

Xavier
-- 

 Sincères salutations.
_____________________________________________________________________
 
Xavier BRU                 BULL ISD/R&D/INTEL office:     FREC B1-422
tel : +33 (0)4 76 29 77 45                    http://www-frec.bull.fr
fax : +33 (0)4 76 29 77 70                 mailto:Xavier.Bru@bull.net
addr: BULL, 1 rue de Provence, BP 208, 38432 Echirolles Cedex, FRANCE
_____________________________________________________________________

 > Martin Hicks writes:
 >  > 
 >  > 
 >  > On Fri, Aug 29, 2003 at 06:41:56PM +0200, Xavier Bru wrote:
 >  > > 
 >  > > printing "On node %d totalpages:":
 >  > > 
 >  > > ifa=0xf000ff54f7c5722b
 >  > > iip=0xa0000001007780d0
 >  > > 0xa0000001007780d0 <__alloc_bootmem_core+592>:	[MII]       ld8 r17=[r19],8
 >  > > 
 >  > > 
 >  > > I will try to look at this a bit more if I find some more time.
 >  > 
 >  > Do you have this changeset in your tree?  It sounds like a similar
 >  > problem to what I was seeing.
 >  > 
 >  > ChangeSet@1.1292.1.3, 2003-08-25 12:23:20-07:00, mort@wildopensource.com
 >  >   [PATCH] ia64: paddr_to_nid fixup
 >  >   
 >  >   Here is a small patch for paddr_to_nid().  This fix is already in 2.4
 >  >   and is used in the case where a NUMA kernel is running on a machine
 >  >   without a SRAT ACPI table.  Without this patch the node info is not
 >  >   correctly located.
 >  > 
 >  > 
 >  > mh
 >  > 
 >  > -- 
 >  > Wild Open Source Inc.                  mort@wildopensource.com
 >  > -
 >  > To unsubscribe from this list: send the line "unsubscribe linux-ia64" in
 >  > the body of a message to majordomo@vger.kernel.org
 >  > More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: 2.6.0 test3 does not boot on ia64 NUMA

[Jesse Barnes]

Thanks Xavier, I've included this in the latest discontig patch, which
I'll post again next week I think (with the fixes David wanted for
reentrance).

Jesse

On Tue, Sep 02, 2003 at 07:27:53PM +0200, Xavier Bru wrote:
> Hello Martin,
> 
> I finally found the reason for crashing at init time:
> On node 0, our test configuration has:
>  2 GB of memory at address 0
>  2 GB of memory at address 6 GB (due to PCI hole).
> 
> Current code for acpi_numa_memory_affinity_init ignores physical
> memory bank if the hole (4GB) is bigger than the bank (2 GB).
> As the node_memblk is not present for address 6 GB, paddr_to_nid
> returns -1 and alloc_bootmem_pages_node crashes with a Null pointer.
> 
> As we now have CONFIG_VIRTUAL_MEM_MAP=y, I suppose we should also use
> sparse memory in same node. (Am I right ?)
> 
> Now 2.6.0 test4  boots OK in NUMA with:
> 
> . Jesse's discontig patch
> . Tony's trim patch
> . alloc_bootmem patch
> . and this small one :-)
> 
> diff --exclude-from /users/xb/proc/diff.exclude -Nur linux-2.6.0-test4/arch/ia64/kernel/acpi.c 0t4/arch/ia64/kernel/acpi.c
> --- linux-2.6.0-test4/arch/ia64/kernel/acpi.c	2003-08-23 01:55:43.000000000 +0200
> +++ 0t4/arch/ia64/kernel/acpi.c	2003-09-02 15:37:17.000000000 +0200
> @@ -423,9 +423,8 @@
>  
>  	if (min_hole_size) {
>  		if (min_hole_size > size) {
> -			printk(KERN_ERR "Too huge memory hole. Ignoring %ld MBytes at %lx\n",
> +			printk(KERN_WARNING "Huge memory hole. Using %ld MBytes at %lx\n",
>  			       size/(1024*1024), paddr);
> -			return;
>  		}
>  	}
>  
> Thanks again for your help.
> 
> Xavier
> -- 
> 
>  Sinc?res salutations.
> _____________________________________________________________________
>  
> Xavier BRU                 BULL ISD/R&D/INTEL office:     FREC B1-422
> tel : +33 (0)4 76 29 77 45                    http://www-frec.bull.fr
> fax : +33 (0)4 76 29 77 70                 mailto:Xavier.Bru@bull.net
> addr: BULL, 1 rue de Provence, BP 208, 38432 Echirolles Cedex, FRANCE
> _____________________________________________________________________
> 
>  > Martin Hicks writes:
>  >  > 
>  >  > 
>  >  > On Fri, Aug 29, 2003 at 06:41:56PM +0200, Xavier Bru wrote:
>  >  > > 
>  >  > > printing "On node %d totalpages:":
>  >  > > 
>  >  > > ifa=0xf000ff54f7c5722b
>  >  > > iip=0xa0000001007780d0
>  >  > > 0xa0000001007780d0 <__alloc_bootmem_core+592>:	[MII]       ld8 r17=[r19],8
>  >  > > 
>  >  > > 
>  >  > > I will try to look at this a bit more if I find some more time.
>  >  > 
>  >  > Do you have this changeset in your tree?  It sounds like a similar
>  >  > problem to what I was seeing.
>  >  > 
>  >  > ChangeSet@1.1292.1.3, 2003-08-25 12:23:20-07:00, mort@wildopensource.com
>  >  >   [PATCH] ia64: paddr_to_nid fixup
>  >  >   
>  >  >   Here is a small patch for paddr_to_nid().  This fix is already in 2.4
>  >  >   and is used in the case where a NUMA kernel is running on a machine
>  >  >   without a SRAT ACPI table.  Without this patch the node info is not
>  >  >   correctly located.
>  >  > 
>  >  > 
>  >  > mh
>  >  > 
>  >  > -- 
>  >  > Wild Open Source Inc.                  mort@wildopensource.com
>  >  > -
>  >  > To unsubscribe from this list: send the line "unsubscribe linux-ia64" in
>  >  > the body of a message to majordomo@vger.kernel.org
>  >  > More majordomo info at  http://vger.kernel.org/majordomo-info.html
> -
> To unsubscribe from this list: send the line "unsubscribe linux-ia64" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

RE: 2.6.0 test3 does not boot on ia64 NUMA

[Luck, Tony]

> Thanks Xavier, I've included this in the latest discontig patch, which
> I'll post again next week I think (with the fixes David wanted for
> reentrance).
> 
> Jesse
> 
> On Tue, Sep 02, 2003 at 07:27:53PM +0200, Xavier Bru wrote:
> > Hello Martin,
> > 
> > I finally found the reason for crashing at init time:
> > On node 0, our test configuration has:
> >  2 GB of memory at address 0
> >  2 GB of memory at address 6 GB (due to PCI hole).
> > 
> > Current code for acpi_numa_memory_affinity_init ignores physical
> > memory bank if the hole (4GB) is bigger than the bank (2 GB).
> > As the node_memblk is not present for address 6 GB, paddr_to_nid
> > returns -1 and alloc_bootmem_pages_node crashes with a Null pointer.
> > 
> > As we now have CONFIG_VIRTUAL_MEM_MAP=y, I suppose we 
> should also use
> > sparse memory in same node. (Am I right ?)
> > 
> > Now 2.6.0 test4  boots OK in NUMA with:
> > 
> > . Jesse's discontig patch
> > . Tony's trim patch
> > . alloc_bootmem patch
> > . and this small one :-)
> > 
> > diff --exclude-from /users/xb/proc/diff.exclude -Nur 
> linux-2.6.0-test4/arch/ia64/kernel/acpi.c 0t4/arch/ia64/kernel/acpi.c
> > --- linux-2.6.0-test4/arch/ia64/kernel/acpi.c	
> 2003-08-23 01:55:43.000000000 +0200
> > +++ 0t4/arch/ia64/kernel/acpi.c	2003-09-02 
> 15:37:17.000000000 +0200
> > @@ -423,9 +423,8 @@
> >  
> >  	if (min_hole_size) {
> >  		if (min_hole_size > size) {
> > -			printk(KERN_ERR "Too huge memory hole. 
> Ignoring %ld MBytes at %lx\n",
> > +			printk(KERN_WARNING "Huge memory hole. 
> Using %ld MBytes at %lx\n",
> >  			       size/(1024*1024), paddr);
> > -			return;
> >  		}
> >  	}

What are the remaining issues with sparse memory within a node?
CONFIG_VIRTUAL_MEM_MAP should be able to cope with this without
wasting memory on "struct page" for non-existent pages in the holes.

Presumably there are some bootmem bitmap size issues if the gaps
within nodes are too huge.  But a few GB shouldn't be a problem (with
a 16K page size, each GB of memory/hole only takes 8K of bitmap).

Is there anything else that blows up?

If not, then could we just drop the printk altogether?

-Tony

Re: 2.6.0 test3 does not boot on ia64 NUMA

[Jesse Barnes]

On Thu, Sep 04, 2003 at 12:06:46PM -0700, Luck, Tony wrote:
> What are the remaining issues with sparse memory within a node?
> CONFIG_VIRTUAL_MEM_MAP should be able to cope with this without
> wasting memory on "struct page" for non-existent pages in the holes.

Yeah, that's what we're using the latest discontig patches.

> Presumably there are some bootmem bitmap size issues if the gaps
> within nodes are too huge.  But a few GB shouldn't be a problem (with
> a 16K page size, each GB of memory/hole only takes 8K of bitmap).

Right, until we get really huge holes, that'll be ok, and there are
already some optimizations to keep track of the last place a successful
bootmem allocation came from so that you don't keep walking through the
holes, which helped zx1 boots.

> Is there anything else that blows up?

Not that I know of...

> If not, then could we just drop the printk altogether?

Sure, sounds good to me.

Jesse

Re: 2.6.0 test3 does not boot on ia64 NUMA

[Xavier Bru]

Jesse Barnes writes:
 > On Thu, Sep 04, 2003 at 12:06:46PM -0700, Luck, Tony wrote:
 > > What are the remaining issues with sparse memory within a node?
 > > CONFIG_VIRTUAL_MEM_MAP should be able to cope with this without
 > > wasting memory on "struct page" for non-existent pages in the holes.
 > 
 > Yeah, that's what we're using the latest discontig patches.
 > 
 > > Presumably there are some bootmem bitmap size issues if the gaps
 > > within nodes are too huge.  But a few GB shouldn't be a problem (with
 > > a 16K page size, each GB of memory/hole only takes 8K of bitmap).
 > 
 > Right, until we get really huge holes, that'll be ok, and there are
 > already some optimizations to keep track of the last place a successful
 > bootmem allocation came from so that you don't keep walking through the
 > holes, which helped zx1 boots.

This allows supporting our platform without specific modifications.
 
> 
 > > Is there anything else that blows up?
 > 
 > Not that I know of...
 > 
 > > If not, then could we just drop the printk altogether?
 > 
 > Sure, sounds good to me.

About code cleaning, it seems that the IA64_NODESIZE_* 
options are still present in Kconfig but no more used, at least for
DIG platforms.

Thanks again for taking in account our problem.
Xavier

-- 

 Sincères salutations.
_____________________________________________________________________
 
Xavier BRU                 BULL ISD/R&D/INTEL office:     FREC B1-422
tel : +33 (0)4 76 29 77 45                    http://www-frec.bull.fr
fax : +33 (0)4 76 29 77 70                 mailto:Xavier.Bru@bull.net
addr: BULL, 1 rue de Provence, BP 208, 38432 Echirolles Cedex, FRANCE
_____________________________________________________________________

Re: 2.6.0 test3 does not boot on ia64 NUMA

[Jesse Barnes]

On Fri, Sep 05, 2003 at 11:19:13AM +0200, Xavier Bru wrote:
> About code cleaning, it seems that the IA64_NODESIZE_* 
> options are still present in Kconfig but no more used, at least for
> DIG platforms.

I've removed these in the patch as well, I'll post a new one for testing
soon.

Thanks,
Jesse