[PATCH]x86-x86_64 flush cache for CPU hotplug

[PATCH]x86-x86_64 flush cache for CPU hotplug

[Shaohua Li]

Hi,
We should flush cache at CPU hotplug. An error has been observed data is
corrupted after CPU hotplug in CPUs with bigger cache.
I guess IA64 requires similar change, Ashok?

Thanks,
Shaohua

Signed-off-by: Shaohua.li<shaohua.li@intel.com>
---

 linux-2.6.12-rc6-mm1-root/arch/i386/kernel/process.c   |    1 +
 linux-2.6.12-rc6-mm1-root/arch/x86_64/kernel/process.c |    1 +
 2 files changed, 2 insertions(+)

diff -puN arch/i386/kernel/process.c~flush_cache_cpuhotplug arch/i386/kernel/process.c
--- linux-2.6.12-rc6-mm1/arch/i386/kernel/process.c~flush_cache_cpuhotplug	2005-06-10 10:56:05.082247160 +0800
+++ linux-2.6.12-rc6-mm1-root/arch/i386/kernel/process.c	2005-06-10 11:05:10.597316264 +0800
@@ -155,6 +155,7 @@ static inline void play_dead(void)
 {
 	/* This must be done before dead CPU ack */
 	cpu_exit_clear();
+	wbinvd();
 	mb();
 	/* Ack it */
 	__get_cpu_var(cpu_state) = CPU_DEAD;
diff -puN arch/x86_64/kernel/process.c~flush_cache_cpuhotplug arch/x86_64/kernel/process.c
--- linux-2.6.12-rc6-mm1/arch/x86_64/kernel/process.c~flush_cache_cpuhotplug	2005-06-10 10:56:18.270242280 +0800
+++ linux-2.6.12-rc6-mm1-root/arch/x86_64/kernel/process.c	2005-06-10 11:05:23.206399392 +0800
@@ -165,6 +165,7 @@ DECLARE_PER_CPU(int, cpu_state);
 static inline void play_dead(void)
 {
 	idle_task_exit();
+	wbinvd();
 	mb();
 	/* Ack it */
 	__get_cpu_var(cpu_state) = CPU_DEAD;
_

Re: [PATCH]x86-x86_64 flush cache for CPU hotplug

[Ashok Raj]

On Fri, Jun 10, 2005 at 11:30:08AM +0800, Shaohua Li wrote:
> Hi,
> We should flush cache at CPU hotplug. An error has been observed data is
> corrupted after CPU hotplug in CPUs with bigger cache.
> I guess IA64 requires similar change, Ashok?

Yes, we need that for IA64 as well. i did put that in my notes tbd, but 
didnt turn around doing it yet. 

Think we also need to define the same for asm-ia64/acpi.h FLUSH_CPU_CACHE
i will try updating both in next.

If you have a targetted test that causes the corruption that would be great
so we can add to our stress test senarios.

Cheers,
ashok
> 
> Thanks,
> Shaohua
> 
> Signed-off-by: Shaohua.li<shaohua.li@intel.com>
> ---
> 
>  linux-2.6.12-rc6-mm1-root/arch/i386/kernel/process.c   |    1 +
>  linux-2.6.12-rc6-mm1-root/arch/x86_64/kernel/process.c |    1 +
>  2 files changed, 2 insertions(+)
> 

Re: [PATCH]x86-x86_64 flush cache for CPU hotplug

[Ashok Raj]

On Fri, Jun 10, 2005 at 11:30:08AM +0800, Shaohua Li wrote:
> Hi,
> We should flush cache at CPU hotplug. An error has been observed data is
> corrupted after CPU hotplug in CPUs with bigger cache.
> I guess IA64 requires similar change, Ashok?
> 

Good catch. Yes, we need it for IA64 as well. Strange i had put it in my
TBD, and out of 2 items, this is one of them. I need to muck something for
mca apart from this pending flush.

I think i will update the ACPI_FLUSH_CPU_CACHE as well when i push this change.

> 
> Signed-off-by: Shaohua.li<shaohua.li@intel.com>
> ---
> 
>  linux-2.6.12-rc6-mm1-root/arch/i386/kernel/process.c   |    1 +
>  linux-2.6.12-rc6-mm1-root/arch/x86_64/kernel/process.c |    1 +
>  2 files changed, 2 insertions(+)

Re: [PATCH]x86-x86_64 flush cache for CPU hotplug

[Andi Kleen]

On Fri, Jun 10, 2005 at 11:30:08AM +0800, Shaohua Li wrote:
> Hi,
> We should flush cache at CPU hotplug. An error has been observed data is
> corrupted after CPU hotplug in CPUs with bigger cache.

Did you see that with actual hardware CPU hotplug? With software
for testing only hotplug it should not make any difference.

/Andi

Re: [PATCH]x86-x86_64 flush cache for CPU hotplug

[Shaohua Li]

On Fri, 2005-06-10 at 19:21 +0200, Andi Kleen wrote:
> On Fri, Jun 10, 2005 at 11:30:08AM +0800, Shaohua Li wrote:
> > Hi,
> > We should flush cache at CPU hotplug. An error has been observed data is
> > corrupted after CPU hotplug in CPUs with bigger cache.
> 
> Did you see that with actual hardware CPU hotplug? With software
> for testing only hotplug it should not make any difference.
Sure, this should only occur on physical CPU hotplug. I haven't hardware
which supports physical CPU hotplug. What I'm testing is supend-to-ram.
In this case, we offline all APs and BIOS possibly will put APs into
some special states (maybe reboot APs). It's very like physical CPU
hotplug.


> If you have a targetted test that causes the corruption that would be great
> so we can add to our stress test senarios.
No, I haven't. IA64 doesn't support suspend/resume, so my test case
can't apply to IA64.

Thanks,
Shaohua

Re: [PATCH]x86-x86_64 flush cache for CPU hotplug

[Zwane Mwaikambo]

Hello Shaohua,

On Fri, 10 Jun 2005, Shaohua Li wrote:

> We should flush cache at CPU hotplug. An error has been observed data is
> corrupted after CPU hotplug in CPUs with bigger cache.
> I guess IA64 requires similar change, Ashok?

Interesting, which processor was this?

> Signed-off-by: Shaohua.li<shaohua.li@intel.com>
> ---
> 
>  linux-2.6.12-rc6-mm1-root/arch/i386/kernel/process.c   |    1 +
>  linux-2.6.12-rc6-mm1-root/arch/x86_64/kernel/process.c |    1 +
>  2 files changed, 2 insertions(+)
> 
> diff -puN arch/i386/kernel/process.c~flush_cache_cpuhotplug arch/i386/kernel/process.c
> --- linux-2.6.12-rc6-mm1/arch/i386/kernel/process.c~flush_cache_cpuhotplug	2005-06-10 10:56:05.082247160 +0800
> +++ linux-2.6.12-rc6-mm1-root/arch/i386/kernel/process.c	2005-06-10 11:05:10.597316264 +0800
> @@ -155,6 +155,7 @@ static inline void play_dead(void)
>  {
>  	/* This must be done before dead CPU ack */
>  	cpu_exit_clear();
> +	wbinvd();
>  	mb();

We shouldn't need that mb() there anymore then, ditto for the other 
location.

Re: [PATCH]x86-x86_64 flush cache for CPU hotplug

[Shaohua Li]

On Fri, 2005-06-10 at 13:11 -0600, Zwane Mwaikambo wrote:
> Hello Shaohua,
> 
> On Fri, 10 Jun 2005, Shaohua Li wrote:
> 
> > We should flush cache at CPU hotplug. An error has been observed data is
> > corrupted after CPU hotplug in CPUs with bigger cache.
> > I guess IA64 requires similar change, Ashok?
> 
> Interesting, which processor was this?
It's a kind of Pentium M. 

> > 
> >  linux-2.6.12-rc6-mm1-root/arch/i386/kernel/process.c   |    1 +
> >  linux-2.6.12-rc6-mm1-root/arch/x86_64/kernel/process.c |    1 +
> >  2 files changed, 2 insertions(+)
> > 
> > diff -puN arch/i386/kernel/process.c~flush_cache_cpuhotplug arch/i386/kernel/process.c
> > --- linux-2.6.12-rc6-mm1/arch/i386/kernel/process.c~flush_cache_cpuhotplug	2005-06-10 10:56:05.082247160 +0800
> > +++ linux-2.6.12-rc6-mm1-root/arch/i386/kernel/process.c	2005-06-10 11:05:10.597316264 +0800
> > @@ -155,6 +155,7 @@ static inline void play_dead(void)
> >  {
> >  	/* This must be done before dead CPU ack */
> >  	cpu_exit_clear();
> > +	wbinvd();
> >  	mb();
> 
> We shouldn't need that mb() there anymore then, ditto for the other 
> location.
Indead. How about just leave it here for clearness?

Thanks,
Shaohua