[PATCH] x86: Use online node real index in calulate_tbl_offset()

[PATCH] x86: Use online node real index in calulate_tbl_offset()

[Yinghai Lu]

--- resending, hope in can make into -rc2.

Found one numa system that doesn't have ram installed with first socket
hang during executing init scripts.

bisect to:

|commit 932967202182743c01a2eee4bdfa2c42697bc586
|Author: Shaohua Li <shaohua.li@intel.com>
|Date:   Wed Oct 20 11:07:03 2010 +0800
|
|    x86: Spread tlb flush vector between nodes

It turns out when first socket is not online could have cpus on node1
tlb_offset set to bigger than NUM_INVALIDATE_TLB_VECTORS.

that could affect systems like 4 sockets, but socket 2 doesn't
have installed, sockets 3 will get too big tlb_offset.

Need to use real online node idx.

Signed-off-by: Yinghai Lu <yinghai@kernel.org>
Acked-by: Shaohua Li <shaohua.li@intel.com>

---
 arch/x86/mm/tlb.c |    5 +++--
 1 file changed, 3 insertions(+), 2 deletions(-)

Index: linux-2.6/arch/x86/mm/tlb.c
===================================================================
--- linux-2.6.orig/arch/x86/mm/tlb.c
+++ linux-2.6/arch/x86/mm/tlb.c
@@ -223,7 +223,7 @@ void native_flush_tlb_others(const struc
 
 static void __cpuinit calculate_tlb_offset(void)
 {
-	int cpu, node, nr_node_vecs;
+	int cpu, node, nr_node_vecs, idx = 0;
 	/*
 	 * we are changing tlb_vector_offset for each CPU in runtime, but this
 	 * will not cause inconsistency, as the write is atomic under X86. we
@@ -239,7 +239,7 @@ static void __cpuinit calculate_tlb_offs
 		nr_node_vecs = NUM_INVALIDATE_TLB_VECTORS/nr_online_nodes;
 
 	for_each_online_node(node) {
-		int node_offset = (node % NUM_INVALIDATE_TLB_VECTORS) *
+		int node_offset = (idx % NUM_INVALIDATE_TLB_VECTORS) *
 			nr_node_vecs;
 		int cpu_offset = 0;
 		for_each_cpu(cpu, cpumask_of_node(node)) {
@@ -248,6 +248,7 @@ static void __cpuinit calculate_tlb_offs
 			cpu_offset++;
 			cpu_offset = cpu_offset % nr_node_vecs;
 		}
+		idx++;
 	}
 }
 

[tip:x86/urgent] x86: Use online node real index in calulate_tbl_offset()

[tip-bot for Yinghai Lu]

Commit-ID:  9223081f54e3dc5045fe41a475165d9003c9a779
Gitweb:     http://git.kernel.org/tip/9223081f54e3dc5045fe41a475165d9003c9a779
Author:     Yinghai Lu <yinghai@kernel.org>
AuthorDate: Sat, 13 Nov 2010 10:52:09 -0800
Committer:  Ingo Molnar <mingo@elte.hu>
CommitDate: Thu, 18 Nov 2010 10:10:50 +0100

x86: Use online node real index in calulate_tbl_offset()

Found a NUMA system that doesn't have RAM installed at the first
socket which hangs while executing init scripts.

bisected it to:

 | commit 932967202182743c01a2eee4bdfa2c42697bc586
 | Author: Shaohua Li <shaohua.li@intel.com>
 | Date:   Wed Oct 20 11:07:03 2010 +0800
 |
 |     x86: Spread tlb flush vector between nodes

It turns out when first socket is not online it could have cpus on
node1 tlb_offset set to bigger than NUM_INVALIDATE_TLB_VECTORS.

That could affect systems like 4 sockets, but socket 2 doesn't
have installed, sockets 3 will get too big tlb_offset.

Need to use real online node idx.

Signed-off-by: Yinghai Lu <yinghai@kernel.org>
Acked-by: Shaohua Li <shaohua.li@intel.com>
Cc: Linus Torvalds <torvalds@linux-foundation.org>
LKML-Reference: <4CDEDE59.40603@kernel.org>
Signed-off-by: Ingo Molnar <mingo@elte.hu>
---
 arch/x86/mm/tlb.c |    5 +++--
 1 files changed, 3 insertions(+), 2 deletions(-)

diff --git a/arch/x86/mm/tlb.c b/arch/x86/mm/tlb.c
index 12cdbb1..6acc724 100644
--- a/arch/x86/mm/tlb.c
+++ b/arch/x86/mm/tlb.c
@@ -223,7 +223,7 @@ void native_flush_tlb_others(const struct cpumask *cpumask,
 
 static void __cpuinit calculate_tlb_offset(void)
 {
-	int cpu, node, nr_node_vecs;
+	int cpu, node, nr_node_vecs, idx = 0;
 	/*
 	 * we are changing tlb_vector_offset for each CPU in runtime, but this
 	 * will not cause inconsistency, as the write is atomic under X86. we
@@ -239,7 +239,7 @@ static void __cpuinit calculate_tlb_offset(void)
 		nr_node_vecs = NUM_INVALIDATE_TLB_VECTORS/nr_online_nodes;
 
 	for_each_online_node(node) {
-		int node_offset = (node % NUM_INVALIDATE_TLB_VECTORS) *
+		int node_offset = (idx % NUM_INVALIDATE_TLB_VECTORS) *
 			nr_node_vecs;
 		int cpu_offset = 0;
 		for_each_cpu(cpu, cpumask_of_node(node)) {
@@ -248,6 +248,7 @@ static void __cpuinit calculate_tlb_offset(void)
 			cpu_offset++;
 			cpu_offset = cpu_offset % nr_node_vecs;
 		}
+		idx++;
 	}
 }
 

[PATCH] x86: Use online node real index in calulate_tbl_offset()

[Yinghai Lu]

Found one numa system that doesn't have ram installed in first socket
hang during executing init scripts.

bisect to:

|commit 932967202182743c01a2eee4bdfa2c42697bc586
|Author: Shaohua Li <shaohua.li@intel.com>
|Date:   Wed Oct 20 11:07:03 2010 +0800
|
|    x86: Spread tlb flush vector between nodes

It turns out when first socket is not online could have cpus on node1
tlb_offset set to bigger than NUM_INVALIDATE_TLB_VECTORS.

that could affect systems like 4 sockets, but socket 2 doesn't
have installed, sockets 3 will get too big tlb_offset.

Need to use real online node idx.

Signed-off-by: Yinghai Lu <yinghai@kernel.org>

---
 arch/x86/mm/tlb.c |    5 +++--
 1 file changed, 3 insertions(+), 2 deletions(-)

Index: linux-2.6/arch/x86/mm/tlb.c
===================================================================
--- linux-2.6.orig/arch/x86/mm/tlb.c
+++ linux-2.6/arch/x86/mm/tlb.c
@@ -223,7 +223,7 @@ void native_flush_tlb_others(const struc
 
 static void __cpuinit calculate_tlb_offset(void)
 {
-	int cpu, node, nr_node_vecs;
+	int cpu, node, nr_node_vecs, idx = 0;
 	/*
 	 * we are changing tlb_vector_offset for each CPU in runtime, but this
 	 * will not cause inconsistency, as the write is atomic under X86. we
@@ -239,7 +239,7 @@ static void __cpuinit calculate_tlb_offs
 		nr_node_vecs = NUM_INVALIDATE_TLB_VECTORS/nr_online_nodes;
 
 	for_each_online_node(node) {
-		int node_offset = (node % NUM_INVALIDATE_TLB_VECTORS) *
+		int node_offset = (idx % NUM_INVALIDATE_TLB_VECTORS) *
 			nr_node_vecs;
 		int cpu_offset = 0;
 		for_each_cpu(cpu, cpumask_of_node(node)) {
@@ -248,6 +248,7 @@ static void __cpuinit calculate_tlb_offs
 			cpu_offset++;
 			cpu_offset = cpu_offset % nr_node_vecs;
 		}
+		idx++;
 	}
 }
 

Re: [PATCH] x86: Use online node real index in calulate_tbl_offset()

[H. Peter Anvin]

Shaohua, does this look right to you?

	-hpa


On 10/28/2010 07:18 PM, Yinghai Lu wrote:
> 
> Found one numa system that doesn't have ram installed in first socket
> hang during executing init scripts.
> 
> bisect to:
> 
> |commit 932967202182743c01a2eee4bdfa2c42697bc586
> |Author: Shaohua Li <shaohua.li@intel.com>
> |Date:   Wed Oct 20 11:07:03 2010 +0800
> |
> |    x86: Spread tlb flush vector between nodes
> 
> It turns out when first socket is not online could have cpus on node1
> tlb_offset set to bigger than NUM_INVALIDATE_TLB_VECTORS.
> 
> that could affect systems like 4 sockets, but socket 2 doesn't
> have installed, sockets 3 will get too big tlb_offset.
> 
> Need to use real online node idx.
> 
> Signed-off-by: Yinghai Lu <yinghai@kernel.org>
> 
> ---
>  arch/x86/mm/tlb.c |    5 +++--
>  1 file changed, 3 insertions(+), 2 deletions(-)
> 
> Index: linux-2.6/arch/x86/mm/tlb.c
> ===================================================================
> --- linux-2.6.orig/arch/x86/mm/tlb.c
> +++ linux-2.6/arch/x86/mm/tlb.c
> @@ -223,7 +223,7 @@ void native_flush_tlb_others(const struc
>  
>  static void __cpuinit calculate_tlb_offset(void)
>  {
> -	int cpu, node, nr_node_vecs;
> +	int cpu, node, nr_node_vecs, idx = 0;
>  	/*
>  	 * we are changing tlb_vector_offset for each CPU in runtime, but this
>  	 * will not cause inconsistency, as the write is atomic under X86. we
> @@ -239,7 +239,7 @@ static void __cpuinit calculate_tlb_offs
>  		nr_node_vecs = NUM_INVALIDATE_TLB_VECTORS/nr_online_nodes;
>  
>  	for_each_online_node(node) {
> -		int node_offset = (node % NUM_INVALIDATE_TLB_VECTORS) *
> +		int node_offset = (idx % NUM_INVALIDATE_TLB_VECTORS) *
>  			nr_node_vecs;
>  		int cpu_offset = 0;
>  		for_each_cpu(cpu, cpumask_of_node(node)) {
> @@ -248,6 +248,7 @@ static void __cpuinit calculate_tlb_offs
>  			cpu_offset++;
>  			cpu_offset = cpu_offset % nr_node_vecs;
>  		}
> +		idx++;
>  	}
>  }
>  


-- 
H. Peter Anvin, Intel Open Source Technology Center
I work for Intel.  I don't speak on their behalf.

Re: [PATCH] x86: Use online node real index in calulate_tbl_offset()

[Shaohua Li]

On Fri, 2010-10-29 at 10:18 +0800, Yinghai Lu wrote:
> Found one numa system that doesn't have ram installed in first socket
> hang during executing init scripts.
> 
> bisect to:
> 
> |commit 932967202182743c01a2eee4bdfa2c42697bc586
> |Author: Shaohua Li <shaohua.li@intel.com>
> |Date:   Wed Oct 20 11:07:03 2010 +0800
> |
> |    x86: Spread tlb flush vector between nodes
> 
> It turns out when first socket is not online could have cpus on node1
> tlb_offset set to bigger than NUM_INVALIDATE_TLB_VECTORS.
> 
> that could affect systems like 4 sockets, but socket 2 doesn't
> have installed, sockets 3 will get too big tlb_offset.
> 
> Need to use real online node idx.
> 
Thanks for catching it up.

Acked-by: Shaohua Li <shaohua.li@intel.com>