Re: 2.6.14-rc1-git-now still dying in mm/slab - this time line 1849

[Ravikiran G Thirumalai <kiran@scalex86.org>]

On Tue, Sep 20, 2005 at 03:58:16PM +0200, Petr Vandrovec wrote:
> Andrew Morton wrote:
> >Christoph Lameter <clameter@engr.sgi.com> wrote:
> >
> >...
> >They do.  I don't believe that preemption is the source of this BUG. 
> >(Petr, does CONFIG_PREEMPT=n fix it?)
> 
> No, it does not.  I've even added printks here and there to show node 
> number,
> and everything works as it should.  Maybe there are some problems with
> numa_node_id() and migrating between processors when memory gets released,
> I do not know.
> 
> Only thing I know that if I'll add WARN_ON below to the free_block(), it
> triggers...
> 
> @free_block
>   slabp = GET_PAGE_SLAB(virt_to_page(objp));
>   nodeid = slabp->nodeid;
> +  WARN_ON(nodeid != numa_node_id());             <<<<<
>   l3 = cachep->nodelist[nodeid];
>   list_del(&slabp->list);
>   objnr = (objp - slabp->s_mem) / cachep->objsize;
>   check_spinlock_acquired_node(cachep, nodeid);
>   check_slabp(cachep, slabp);
> 
> ... saying that keventd/0 tries to operate on
> slab belonging to node#1, while having acquired lock for cachep belonging
  ^^^^^^^^^^^^^^^^^^^^^^^^^
> to node #0
  ^^^^^^^^^^

Just might be relevant here, I found a bug with the recent
x86_64 changes to 2.6.14-rc* which causes the node_to_cpumask[] to go bad for
the boot processor.  This happens on both amd and em64t boxes. I guess the
kevent/0 cpus_allowed mask might be changed by the bad node_to_cpumask[]
here?

On a opteron box (courtesy Ananth M)
# cat /sys/devices/system/node/node0/cpumap
00000000

# cat /sys/devices/system/node/node1/cpumap
00000003

On our em64t IBM x460 NUMA,

# cat /sys/devices/system/node/node0/cpumap
0000000e

# cat /sys/devices/system/node/node1/cpumap
000000f1

Here is a fix for that, I have sounded out Andi Kleen on this and waiting
for his comments.  Maybe somebody can test the patch below on amds?

Thanks,
Kiran

---
Patch to fix the BP node_to_cpumask.  2.6.14-rc* broke the boot cpu bit as
the cpu_to_node(0) is now not setup early enough for numa_init_array.
cpu_to_node[] is setup much later at srat_detect_node on acpi srat based
em64t machines.  This seems like a problem on amd machines too,  Tested on
em64t though. /sys/devices/system/node/node0/cpumap shows up sanely after
this patch.

Signed off by: Ravikiran Thirumalai <kiran@scalex86.org>
Signed-off-by: Shai Fultheim <shai@scalex86.org>


Index: linux-2.6.14-rc1/arch/x86_64/mm/numa.c
===================================================================
--- linux-2.6.14-rc1.orig/arch/x86_64/mm/numa.c	2005-09-19 17:58:10.000000000 -0700
+++ linux-2.6.14-rc1/arch/x86_64/mm/numa.c	2005-09-27 01:34:20.000000000 -0700
@@ -178,7 +178,6 @@
 		rr++; 
 	}
 
-	set_bit(0, &node_to_cpumask[cpu_to_node(0)]);
 }
 
 #ifdef CONFIG_NUMA_EMU
@@ -266,9 +265,7 @@
 
 __cpuinit void numa_add_cpu(int cpu)
 {
-	/* BP is initialized elsewhere */
-	if (cpu) 
-		set_bit(cpu, &node_to_cpumask[cpu_to_node(cpu)]);
+	set_bit(cpu, &node_to_cpumask[cpu_to_node(cpu)]);
 } 
 
 unsigned long __init numa_free_all_bootmem(void)