Re: [RFC] [PATCH] cpu_hotplug on IBM JS20 system

Re: [RFC] [PATCH] cpu_hotplug on IBM JS20 system

[Heiko Carstens]

> Since we are not supported by hardware for cpu hotplug. I have developed
> the patch which will disable cpu hotplug on IBM bladecentre JS20. Please
> let me know your comments on this please.

> +extern  int cpu_hotplug_disabled;
> +extern  struct mutex cpu_add_remove_lock;
[...]
> +	if(rtas_stop_self_args.token == RTAS_UNKNOWN_SERVICE) {
> +		mutex_lock(&cpu_add_remove_lock);
> +		cpu_hotplug_disabled = 1;
> +		mutex_unlock(&cpu_add_remove_lock);
> +	}
> +
>  #endif /* CONFIG_HOTPLUG_CPU */
>  #ifdef CONFIG_RTAS_ERROR_LOGGING
>  	rtas_last_error_token = rtas_token("rtas-last-error");

You should add a function to kernel/cpu.c which you can call in order to
disable cpu hotplug instead of exporting its private data structures.

[RFC] [PATCH] cpu_hotplug on IBM JS20 system

[Srinivasa Ds]

[-- Attachment #1: Type: text/plain, Size: 1438 bytes --]

Hi
when I tried to hot plug a cpu on IBM bladecentre JS20 system,it dropped 
in to xmon. On analyzing the problem,I found out that "self-stop" token  
is not exported
to the OS through rtas(Could be verified by looking in to 
/proc/device-tree/rtas file).

1:mon> e
cpu 0x1: Vector: 700 (Program Check) at [c00000000ff1bab0]
   pc: c00000000001b144: .rtas_stop_self+0x34/0x70
   lr: c0000000000439c0: .pSeries_mach_cpu_die+0x34/0x40
   sp: c00000000ff1bd30
  msr: 8000000000021032
 current = 0xc00000000ff050b0
 paca    = 0xc0000000005ec500
   pid   = 0, comm = swapper
kernel BUG in rtas_stop_self at arch/powerpc/kernel/rtas.c:829!
===========================================
void rtas_stop_self(void)
{
       struct rtas_args *rtas_args = &rtas_stop_self_args;

       local_irq_disable();

       BUG_ON(rtas_args->token == RTAS_UNKNOWN_SERVICE);
===================================================
#ifdef CONFIG_HOTPLUG_CPU
       rtas_stop_self_args.token = rtas_token("stop-self");
#endif /* CONFIG_HOTPLUG_CPU */
#ifdef CONFIG_RTAS_ERROR_LOGGING
       rtas_last_error_token = rtas_token("rtas-last-error");
===================================================

Since we are not supported by hardware for cpu hotplug. I have developed 
the patch which will disable cpu hotplug on IBM bladecentre JS20. Please 
let me know your comments on this please.

Signed-off-by: Srinivasa DS <srinivasa@in.ibm.com>




Thanks
Srinivasa DS



[-- Attachment #2: cpu_hotplug.fix --]
[-- Type: text/plain, Size: 1839 bytes --]

 arch/powerpc/kernel/rtas.c |    9 +++++++++
 kernel/cpu.c               |    4 ++--
 2 files changed, 11 insertions(+), 2 deletions(-)

Index: linux-2.6.19-rc5/arch/powerpc/kernel/rtas.c
===================================================================
--- linux-2.6.19-rc5.orig/arch/powerpc/kernel/rtas.c	2006-11-08 07:54:20.000000000 +0530
+++ linux-2.6.19-rc5/arch/powerpc/kernel/rtas.c	2006-11-13 17:39:10.000000000 +0530
@@ -53,6 +53,9 @@
 
 unsigned long rtas_rmo_buf;
 
+extern  int cpu_hotplug_disabled;
+extern  struct mutex cpu_add_remove_lock;
+
 /*
  * If non-NULL, this gets called when the kernel terminates.
  * This is done like this so rtas_flash can be a module.
@@ -881,6 +884,12 @@
 
 #ifdef CONFIG_HOTPLUG_CPU
 	rtas_stop_self_args.token = rtas_token("stop-self");
+	if(rtas_stop_self_args.token == RTAS_UNKNOWN_SERVICE) {
+		mutex_lock(&cpu_add_remove_lock);
+		cpu_hotplug_disabled = 1;
+		mutex_unlock(&cpu_add_remove_lock);
+	}
+
 #endif /* CONFIG_HOTPLUG_CPU */
 #ifdef CONFIG_RTAS_ERROR_LOGGING
 	rtas_last_error_token = rtas_token("rtas-last-error");
Index: linux-2.6.19-rc5/kernel/cpu.c
===================================================================
--- linux-2.6.19-rc5.orig/kernel/cpu.c	2006-11-08 07:54:20.000000000 +0530
+++ linux-2.6.19-rc5/kernel/cpu.c	2006-11-13 17:36:22.000000000 +0530
@@ -16,7 +16,7 @@
 #include <linux/mutex.h>
 
 /* This protects CPUs going up and down... */
-static DEFINE_MUTEX(cpu_add_remove_lock);
+DEFINE_MUTEX(cpu_add_remove_lock);
 static DEFINE_MUTEX(cpu_bitmask_lock);
 
 static __cpuinitdata RAW_NOTIFIER_HEAD(cpu_chain);
@@ -24,7 +24,7 @@
 /* If set, cpu_up and cpu_down will return -EBUSY and do nothing.
  * Should always be manipulated under cpu_add_remove_lock
  */
-static int cpu_hotplug_disabled;
+int cpu_hotplug_disabled;
 
 #ifdef CONFIG_HOTPLUG_CPU
 

Re: [RFC] [PATCH] cpu_hotplug on IBM JS20 system

[Gautham R Shenoy]

On Mon, Nov 13, 2006 at 02:09:26PM +0100, Heiko Carstens wrote:
> > Since we are not supported by hardware for cpu hotplug. I have developed
> > the patch which will disable cpu hotplug on IBM bladecentre JS20. Please
> > let me know your comments on this please.
> 
> > +extern  int cpu_hotplug_disabled;
> > +extern  struct mutex cpu_add_remove_lock;
> [...]
> > +	if(rtas_stop_self_args.token == RTAS_UNKNOWN_SERVICE) {
> > +		mutex_lock(&cpu_add_remove_lock);
> > +		cpu_hotplug_disabled = 1;
> > +		mutex_unlock(&cpu_add_remove_lock);
> > +	}
> > +
> >  #endif /* CONFIG_HOTPLUG_CPU */
> >  #ifdef CONFIG_RTAS_ERROR_LOGGING
> >  	rtas_last_error_token = rtas_token("rtas-last-error");
> 
> You should add a function to kernel/cpu.c which you can call in order to
> disable cpu hotplug instead of exporting its private data structures.

Yup. Also, considering the fact that enable_nonboot_cpus() can reset
the cpu_hotplug_disabled flag, I would suggest the following:

a) create one additional state for cpu_hotplug_disabled, something
like

#define PERMANENTLY_DISABLED -1 

b) Define a function kernel/cpu.c

void disable_cpu_hotplug_perm()
{
	cpu_hotplug_disabled = PERMANENTLY_DISABLED;
}

and call it in rtas.c

c) Check for status of cpu_hotplug_disabled in functions cpu_up, 
   cpu_down, enable_nonboot_cpus and disable_nonboot_cpus and 
   if cpu_hotplug_disabled == PERMANENTLY_DISABLED, 
   return bypassing further code.

Thoughts?

Regards,
gautham.
-- 
Gautham R Shenoy
Linux Technology Center
IBM India.
"Freedom comes with a price tag of responsibility, which is still a bargain,
because Freedom is priceless!"

Re: [RFC] [PATCH] cpu_hotplug on IBM JS20 system

[Benjamin Herrenschmidt]

On Mon, 2006-11-13 at 18:40 +0530, Srinivasa Ds wrote:
> Hi
> when I tried to hot plug a cpu on IBM bladecentre JS20 system,it dropped 
> in to xmon. On analyzing the problem,I found out that "self-stop" token  
> is not exported
> to the OS through rtas(Could be verified by looking in to 
> /proc/device-tree/rtas file).

Please CC powerpc related patches to linuxppc-dev@ozlabs.org

Re: [RFC] [PATCH] cpu_hotplug on IBM JS20 system

[Srinivasa Ds]

[-- Attachment #1: Type: text/plain, Size: 1406 bytes --]

Hi
when I tried to hot plug a cpu on IBM bladecentre JS20 system,it dropped 
in to xmon. On analyzing the problem,I found out that "self-stop" token  
is not exported
to the OS through rtas(Could be verified by looking in to 
/proc/device-tree/rtas file).

1:mon> e
cpu 0x1: Vector: 700 (Program Check) at [c00000000ff1bab0]
  pc: c00000000001b144: .rtas_stop_self+0x34/0x70
  lr: c0000000000439c0: .pSeries_mach_cpu_die+0x34/0x40
  sp: c00000000ff1bd30
 msr: 8000000000021032
current = 0xc00000000ff050b0
paca    = 0xc0000000005ec500
  pid   = 0, comm = swapper
kernel BUG in rtas_stop_self at arch/powerpc/kernel/rtas.c:829!
===========================================
void rtas_stop_self(void)
{
      struct rtas_args *rtas_args = &rtas_stop_self_args;

      local_irq_disable();

      BUG_ON(rtas_args->token == RTAS_UNKNOWN_SERVICE);
===================================================
#ifdef CONFIG_HOTPLUG_CPU
      rtas_stop_self_args.token = rtas_token("stop-self");
#endif /* CONFIG_HOTPLUG_CPU */
#ifdef CONFIG_RTAS_ERROR_LOGGING
      rtas_last_error_token = rtas_token("rtas-last-error");
===================================================

Since we are not supported by hardware for cpu hotplug. I have developed 
the patch which will disable cpu hotplug on IBM bladecentre JS20.  
Please let me know your comments on this please.

Signed-off-by: Srinivasa DS <srinivasa@in.ibm.com>






[-- Attachment #2: cpu_hotplug.fix --]
[-- Type: text/plain, Size: 3741 bytes --]

 arch/powerpc/kernel/rtas.c |    3 +++
 include/linux/cpu.h        |    4 ++++
 kernel/cpu.c               |   24 ++++++++++++++++++++++--
 3 files changed, 29 insertions(+), 2 deletions(-)

Index: linux-2.6.19-rc5/arch/powerpc/kernel/rtas.c
===================================================================
--- linux-2.6.19-rc5.orig/arch/powerpc/kernel/rtas.c	2006-11-08 07:54:20.000000000 +0530
+++ linux-2.6.19-rc5/arch/powerpc/kernel/rtas.c	2006-11-14 15:59:58.000000000 +0530
@@ -19,6 +19,7 @@
 #include <linux/init.h>
 #include <linux/capability.h>
 #include <linux/delay.h>
+#include <linux/cpu.h>
 
 #include <asm/prom.h>
 #include <asm/rtas.h>
@@ -881,6 +882,8 @@
 
 #ifdef CONFIG_HOTPLUG_CPU
 	rtas_stop_self_args.token = rtas_token("stop-self");
+	if(rtas_stop_self_args.token == RTAS_UNKNOWN_SERVICE)
+		disable_cpu_hotplug_perm();
 #endif /* CONFIG_HOTPLUG_CPU */
 #ifdef CONFIG_RTAS_ERROR_LOGGING
 	rtas_last_error_token = rtas_token("rtas-last-error");
Index: linux-2.6.19-rc5/kernel/cpu.c
===================================================================
--- linux-2.6.19-rc5.orig/kernel/cpu.c	2006-11-08 07:54:20.000000000 +0530
+++ linux-2.6.19-rc5/kernel/cpu.c	2006-11-14 16:06:17.000000000 +0530
@@ -63,6 +63,18 @@
 }
 EXPORT_SYMBOL_GPL(unlock_cpu_hotplug);
 
+void disable_cpu_hotplug_perm(void)
+{
+	mutex_lock(&cpu_add_remove_lock);
+	cpu_hotplug_disabled = PERM_DISABLED_CPU_HOTPLUG;
+	mutex_unlock(&cpu_add_remove_lock);
+}
+
+static int is_cpu_hotplug_perm_disabled()
+{
+	return cpu_hotplug_disabled == PERM_DISABLED_CPU_HOTPLUG;
+}
+
 #endif	/* CONFIG_HOTPLUG_CPU */
 
 /* Need to know about CPUs going up/down? */
@@ -193,7 +205,7 @@
 	int err = 0;
 
 	mutex_lock(&cpu_add_remove_lock);
-	if (cpu_hotplug_disabled)
+	if (cpu_hotplug_disabled || is_cpu_hotplug_perm_disabled())
 		err = -EBUSY;
 	else
 		err = _cpu_down(cpu);
@@ -244,7 +256,7 @@
 	int err = 0;
 
 	mutex_lock(&cpu_add_remove_lock);
-	if (cpu_hotplug_disabled)
+	if (cpu_hotplug_disabled || is_cpu_hotplug_perm_disabled())
 		err = -EBUSY;
 	else
 		err = _cpu_up(cpu);
@@ -261,6 +273,10 @@
 	int cpu, first_cpu, error;
 
 	mutex_lock(&cpu_add_remove_lock);
+	if(is_cpu_hotplug_perm_disabled()) {
+		error = -EBUSY;
+		goto out;
+	}
 	first_cpu = first_cpu(cpu_present_map);
 	if (!cpu_online(first_cpu)) {
 		error = _cpu_up(first_cpu);
@@ -311,6 +327,10 @@
 
 	/* Allow everyone to use the CPU hotplug again */
 	mutex_lock(&cpu_add_remove_lock);
+	if(is_cpu_hotplug_perm_disabled()) {
+		mutex_unlock(&cpu_add_remove_lock);
+		return;
+	}
 	cpu_hotplug_disabled = 0;
 	mutex_unlock(&cpu_add_remove_lock);
 
Index: linux-2.6.19-rc5/include/linux/cpu.h
===================================================================
--- linux-2.6.19-rc5.orig/include/linux/cpu.h	2006-11-14 15:55:26.000000000 +0530
+++ linux-2.6.19-rc5/include/linux/cpu.h	2006-11-14 16:08:36.000000000 +0530
@@ -31,6 +31,8 @@
 	struct sys_device sysdev;
 };
 
+#define  PERM_DISABLED_CPU_HOTPLUG -1
+
 extern int register_cpu(struct cpu *cpu, int num);
 extern struct sys_device *get_cpu_sysdev(unsigned cpu);
 #ifdef CONFIG_HOTPLUG_CPU
@@ -68,6 +70,7 @@
 /* Stop CPUs going up and down. */
 extern void lock_cpu_hotplug(void);
 extern void unlock_cpu_hotplug(void);
+extern void disable_cpu_hotplug_perm(void);
 #define hotcpu_notifier(fn, pri) {				\
 	static struct notifier_block fn##_nb =			\
 		{ .notifier_call = fn, .priority = pri };	\
@@ -80,6 +83,7 @@
 #else
 #define lock_cpu_hotplug()	do { } while (0)
 #define unlock_cpu_hotplug()	do { } while (0)
+#define disable_cpu_hotplug_perm()     do { } while (0)
 #define lock_cpu_hotplug_interruptible() 0
 #define hotcpu_notifier(fn, pri)	do { } while (0)
 #define register_hotcpu_notifier(nb)	do { } while (0)

Re: [RFC] [PATCH] cpu_hotplug on IBM JS20 system

[Srinivasa Ds]

[-- Attachment #1: Type: text/plain, Size: 345 bytes --]

Srinivasa Ds wrote:
>
> Since we are not supported by hardware for cpu hotplug. I have 
> developed the patch which will disable cpu hotplug on IBM bladecentre 
> JS20.  Please let me know your comments on this please.
>
  Iam sorry, Just resending the patch after formatting it again.
 
    Signed-off-by: Srinivasa DS <srinivasa@in.ibm.com>



[-- Attachment #2: cpu_hotplug.fix --]
[-- Type: text/plain, Size: 3874 bytes --]

 arch/powerpc/kernel/rtas.c |    3 +++
 include/linux/cpu.h        |    4 ++++
 kernel/cpu.c               |   24 ++++++++++++++++++++++--
 3 files changed, 29 insertions(+), 2 deletions(-)

Index: linux-2.6.19-rc5/arch/powerpc/kernel/rtas.c
===================================================================
--- linux-2.6.19-rc5.orig/arch/powerpc/kernel/rtas.c
+++ linux-2.6.19-rc5/arch/powerpc/kernel/rtas.c
@@ -19,6 +19,7 @@
 #include <linux/init.h>
 #include <linux/capability.h>
 #include <linux/delay.h>
+#include <linux/cpu.h>
 
 #include <asm/prom.h>
 #include <asm/rtas.h>
@@ -881,6 +882,8 @@ void __init rtas_initialize(void)
 
 #ifdef CONFIG_HOTPLUG_CPU
 	rtas_stop_self_args.token = rtas_token("stop-self");
+	if(rtas_stop_self_args.token == RTAS_UNKNOWN_SERVICE)
+		disable_cpu_hotplug_perm();
 #endif /* CONFIG_HOTPLUG_CPU */
 #ifdef CONFIG_RTAS_ERROR_LOGGING
 	rtas_last_error_token = rtas_token("rtas-last-error");
Index: linux-2.6.19-rc5/kernel/cpu.c
===================================================================
--- linux-2.6.19-rc5.orig/kernel/cpu.c
+++ linux-2.6.19-rc5/kernel/cpu.c
@@ -63,8 +63,20 @@ void unlock_cpu_hotplug(void)
 }
 EXPORT_SYMBOL_GPL(unlock_cpu_hotplug);
 
+void disable_cpu_hotplug_perm(void)
+{
+	mutex_lock(&cpu_add_remove_lock);
+	cpu_hotplug_disabled = PERM_DISABLED_CPU_HOTPLUG;
+	mutex_unlock(&cpu_add_remove_lock);
+}
+
 #endif	/* CONFIG_HOTPLUG_CPU */
 
+static int is_cpu_hotplug_perm_disabled()
+{
+	return cpu_hotplug_disabled == PERM_DISABLED_CPU_HOTPLUG;
+}
+
 /* Need to know about CPUs going up/down? */
 int __cpuinit register_cpu_notifier(struct notifier_block *nb)
 {
@@ -193,7 +205,7 @@ int cpu_down(unsigned int cpu)
 	int err = 0;
 
 	mutex_lock(&cpu_add_remove_lock);
-	if (cpu_hotplug_disabled)
+	if (cpu_hotplug_disabled || is_cpu_hotplug_perm_disabled())
 		err = -EBUSY;
 	else
 		err = _cpu_down(cpu);
@@ -244,7 +256,7 @@ int __devinit cpu_up(unsigned int cpu)
 	int err = 0;
 
 	mutex_lock(&cpu_add_remove_lock);
-	if (cpu_hotplug_disabled)
+	if (cpu_hotplug_disabled || is_cpu_hotplug_perm_disabled())
 		err = -EBUSY;
 	else
 		err = _cpu_up(cpu);
@@ -261,6 +273,10 @@ int disable_nonboot_cpus(void)
 	int cpu, first_cpu, error;
 
 	mutex_lock(&cpu_add_remove_lock);
+	if(is_cpu_hotplug_perm_disabled()) {
+		error = -EBUSY;
+		goto out;
+	}
 	first_cpu = first_cpu(cpu_present_map);
 	if (!cpu_online(first_cpu)) {
 		error = _cpu_up(first_cpu);
@@ -311,6 +327,10 @@ void enable_nonboot_cpus(void)
 
 	/* Allow everyone to use the CPU hotplug again */
 	mutex_lock(&cpu_add_remove_lock);
+	if(is_cpu_hotplug_perm_disabled()) {
+		mutex_unlock(&cpu_add_remove_lock);
+		return;
+	}
 	cpu_hotplug_disabled = 0;
 	mutex_unlock(&cpu_add_remove_lock);
 
Index: linux-2.6.19-rc5/include/linux/cpu.h
===================================================================
--- linux-2.6.19-rc5.orig/include/linux/cpu.h
+++ linux-2.6.19-rc5/include/linux/cpu.h
@@ -31,6 +31,8 @@ struct cpu {
 	struct sys_device sysdev;
 };
 
+#define  PERM_DISABLED_CPU_HOTPLUG -1
+
 extern int register_cpu(struct cpu *cpu, int num);
 extern struct sys_device *get_cpu_sysdev(unsigned cpu);
 #ifdef CONFIG_HOTPLUG_CPU
@@ -68,6 +70,7 @@ extern struct sysdev_class cpu_sysdev_cl
 /* Stop CPUs going up and down. */
 extern void lock_cpu_hotplug(void);
 extern void unlock_cpu_hotplug(void);
+extern void disable_cpu_hotplug_perm(void);
 #define hotcpu_notifier(fn, pri) {				\
 	static struct notifier_block fn##_nb =			\
 		{ .notifier_call = fn, .priority = pri };	\
@@ -80,6 +83,7 @@ int cpu_down(unsigned int cpu);
 #else
 #define lock_cpu_hotplug()	do { } while (0)
 #define unlock_cpu_hotplug()	do { } while (0)
+#define disable_cpu_hotplug_perm()     do { } while (0)
 #define lock_cpu_hotplug_interruptible() 0
 #define hotcpu_notifier(fn, pri)	do { } while (0)
 #define register_hotcpu_notifier(nb)	do { } while (0)

Re: [RFC] [PATCH] cpu_hotplug on IBM JS20 system

[Michael Ellerman]

[-- Attachment #1: Type: text/plain, Size: 487 bytes --]

On Tue, 2006-11-14 at 19:07 +0530, Srinivasa Ds wrote:
> Hi
> when I tried to hot plug a cpu on IBM bladecentre JS20 system,it dropped 
> in to xmon.

How did you try to hot plug a cpu? Through sysfs/hmc/somethingelse ?

cheers

-- 
Michael Ellerman
OzLabs, IBM Australia Development Lab

wwweb: http://michael.ellerman.id.au
phone: +61 2 6212 1183 (tie line 70 21183)

We do not inherit the earth from our ancestors,
we borrow it from our children. - S.M.A.R.T Person

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: [RFC] [PATCH] cpu_hotplug on IBM JS20 system

[Michael Ellerman]

[-- Attachment #1: Type: text/plain, Size: 487 bytes --]

On Tue, 2006-11-14 at 19:07 +0530, Srinivasa Ds wrote:
> Hi
> when I tried to hot plug a cpu on IBM bladecentre JS20 system,it dropped 
> in to xmon.

How did you try to hot plug a cpu? Through sysfs/hmc/somethingelse ?

cheers

-- 
Michael Ellerman
OzLabs, IBM Australia Development Lab

wwweb: http://michael.ellerman.id.au
phone: +61 2 6212 1183 (tie line 70 21183)

We do not inherit the earth from our ancestors,
we borrow it from our children. - S.M.A.R.T Person

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: [RFC] [PATCH] cpu_hotplug on IBM JS20 system

[Srinivasa Ds]

Michael Ellerman wrote:
> On Tue, 2006-11-14 at 19:07 +0530, Srinivasa Ds wrote:
>   
>> Hi
>> when I tried to hot plug a cpu on IBM bladecentre JS20 system,it dropped 
>> in to xmon.
>>     
>
> How did you try to hot plug a cpu? Through sysfs/hmc/somethingelse ?
>   
echo 0 > /sys/devices/system/cpu/cpu1/online
> cheers
>
>   

Re: [RFC] [PATCH] cpu_hotplug on IBM JS20 system

[Srinivasa Ds]

Michael Ellerman wrote:
> On Tue, 2006-11-14 at 19:07 +0530, Srinivasa Ds wrote:
>   
>> Hi
>> when I tried to hot plug a cpu on IBM bladecentre JS20 system,it dropped 
>> in to xmon.
>>     
>
> How did you try to hot plug a cpu? Through sysfs/hmc/somethingelse ?
>   
echo 0 > /sys/devices/system/cpu/cpu1/online
> cheers
>
>