[PATCH] x86/mce: Distirbute the clear operation of mces_seen to Per-CPU rather than only monarch CPU

[PATCH] x86/mce: Distirbute the clear operation of mces_seen to Per-CPU rather than only monarch CPU

[Chen Yucong]

mces_seen is a Per-CPU variable which should only be accessed by Per-CPU as possible. So the
clear operation of mces_seen should also be lcoal to Per-CPU rather than monarch CPU.

Meanwhile, there is also a potential risk that mces_seen will not be be cleared if a timeout
occors in mce_end for monarch CPU. As a reuslt, the stale value of mces_seen will reappear
on the next mce.

Based on the above reasons, this patch distirbute the clear operation of mces_seen to Per-CPU
rather than only monarch CPU.

Signed-off-by: Chen Yucong <slaoub@gmail.com>
---
 arch/x86/kernel/cpu/mcheck/mce.c |   18 +++++++-----------
 1 file changed, 7 insertions(+), 11 deletions(-)

diff --git a/arch/x86/kernel/cpu/mcheck/mce.c b/arch/x86/kernel/cpu/mcheck/mce.c
index 68317c8..8b8ae98 100644
--- a/arch/x86/kernel/cpu/mcheck/mce.c
+++ b/arch/x86/kernel/cpu/mcheck/mce.c
@@ -785,13 +785,6 @@ static void mce_reign(void)
 	 */
 	if (global_worst <= MCE_KEEP_SEVERITY && mca_cfg.tolerant < 3)
 		mce_panic("Machine check from unknown source", NULL, NULL);
-
-	/*
-	 * Now clear all the mces_seen so that they don't reappear on
-	 * the next mce.
-	 */
-	for_each_possible_cpu(cpu)
-		memset(&per_cpu(mces_seen, cpu), 0, sizeof(struct mce));
 }
 
 static atomic_t global_nwo;
@@ -1137,9 +1130,6 @@ void do_machine_check(struct pt_regs *regs, long error_code)
 		}
 	}
 
-	/* mce_clear_state will clear *final, save locally for use later */
-	m = *final;
-
 	if (!no_way_out)
 		mce_clear_state(toclear);
 
@@ -1161,7 +1151,7 @@ void do_machine_check(struct pt_regs *regs, long error_code)
 			mce_panic("Fatal machine check on current CPU", &m, msg);
 		if (worst == MCE_AR_SEVERITY) {
 			/* schedule action before return to userland */
-			mce_save_info(m.addr, m.mcgstatus & MCG_STATUS_RIPV);
+			mce_save_info(final->addr, final->mcgstatus & MCG_STATUS_RIPV);
 			set_thread_flag(TIF_MCE_NOTIFY);
 		} else if (kill_it) {
 			force_sig(SIGBUS, current);
@@ -1171,6 +1161,12 @@ void do_machine_check(struct pt_regs *regs, long error_code)
 	if (worst > 0)
 		mce_report_event(regs);
 	mce_wrmsrl(MSR_IA32_MCG_STATUS, 0);
+
+	/*
+	 * Now clear the mces_seen of current CPU -*final - so that it does not
+	 * reappear on the next mce.
+	 */
+	memset(final, 0, sizeof(struct mce));
 out:
 	atomic_dec(&mce_entry);
 	sync_core();
-- 
1.7.10.4

Re: [PATCH] x86/mce: Distirbute the clear operation of mces_seen to Per-CPU rather than only monarch CPU

[Borislav Petkov]

On Sun, May 18, 2014 at 11:05:04PM +0800, Chen Yucong wrote:a
> mces_seen is a Per-CPU variable which should only be accessed by
> Per-CPU as possible. So the clear operation of mces_seen should also
> be lcoal to Per-CPU rather than monarch CPU.

No, you need to do the cleaning in mce_reign because the monarch cpu has
to run last after all other cpus have scanned their mce banks.

> Meanwhile, there is also a potential risk that mces_seen will not
> be be cleared if a timeout occors in mce_end for monarch CPU. As a
> reuslt, the stale value of mces_seen will reappear on the next mce.

If that happens, we have a bigger problem.

Is that a real issue you're trying to address?

-- 
Regards/Gruss,
    Boris.

Sent from a fat crate under my desk. Formatting is fine.
--

Re: [PATCH] x86/mce: Distirbute the clear operation of mces_seen to Per-CPU rather than only monarch CPU

[Chen Yucong]

On Sun, 2014-05-18 at 18:35 +0200, Borislav Petkov wrote:
> On Sun, May 18, 2014 at 11:05:04PM +0800, Chen Yucong wrote:a
> > mces_seen is a Per-CPU variable which should only be accessed by
> > Per-CPU as possible. So the clear operation of mces_seen should also
> > be lcoal to Per-CPU rather than monarch CPU.
> 
> No, you need to do the cleaning in mce_reign because the monarch cpu has
> to run last after all other cpus have scanned their mce banks.
> 
But all other CPUs also have to wait monarch CPU to exit from mce_end. 
What's the difference between monarch CPU and Per-CPU for clearing
mces_seen? In practice, there is no difference between them. If we use
monarch CPU to clear mces_seen, then Per-CPU variable can not play out
its advantage.

> > Meanwhile, there is also a potential risk that mces_seen will not
> > be be cleared if a timeout occors in mce_end for monarch CPU. As a
> > reuslt, the stale value of mces_seen will reappear on the next mce.
> 
> If that happens, we have a bigger problem.
Well, in that case, why is there a need for time-out machine in MCE
handler?

Any potential risks --both logical and realistic-- should be avoided as
possible if there are no more questions and performance penalty.

thx!
cyc 
> 
> Is that a real issue you're trying to address?

Re: [PATCH] x86/mce: Distirbute the clear operation of mces_seen to Per-CPU rather than only monarch CPU

[Borislav Petkov]

On Mon, May 19, 2014 at 09:55:40AM +0800, Chen Yucong wrote:
> But all other CPUs also have to wait monarch CPU to exit from mce_end.
> What's the difference between monarch CPU and Per-CPU for clearing
> mces_seen? In practice, there is no difference between them. If we use
> monarch CPU to clear mces_seen, then Per-CPU variable can not play out
> its advantage.

I'll let you stare at mce_reign() a little bit longer... Also, pay
attention to its callsite, that might help.

> Well, in that case, why is there a need for time-out machine in MCE
> handler?

I'll let you figure this out yourself too - it is only fair.

:-)

-- 
Regards/Gruss,
    Boris.

Sent from a fat crate under my desk. Formatting is fine.
--

Re: [PATCH] x86/mce: Distirbute the clear operation of mces_seen to Per-CPU rather than only monarch CPU

[Chen Yucong]

On Mon, 2014-05-19 at 09:26 +0200, Borislav Petkov wrote:
> On Mon, May 19, 2014 at 09:55:40AM +0800, Chen Yucong wrote:
> > But all other CPUs also have to wait monarch CPU to exit from mce_end.
> > What's the difference between monarch CPU and Per-CPU for clearing
> > mces_seen? In practice, there is no difference between them. If we use
> > monarch CPU to clear mces_seen, then Per-CPU variable can not play out
> > its advantage.
> 
> I'll let you stare at mce_reign() a little bit longer... Also, pay
> attention to its callsite, that might help.
> 
We can find the following code segment in mce_end:
-----
...
        if (order == 1) {
                /* CHECKME: Can this race with a parallel hotplug? */
                int cpus = num_online_cpus();

                /*
                 * Monarch: Wait for everyone to go through their
scanning
                 * loops.
                 */
                while (atomic_read(&mce_executing) <= cpus) {
                        if (mce_timed_out(&timeout))
                                goto reset;
                        ndelay(SPINUNIT);
                }

                mce_reign();
                barrier();
                ret = 0;
...
-----
If a timeout occurs in monarch CPU, what will happen for the above code
segment?
The monarch CPU will directly execute -goto reset-, so mce_reign will
not be invoked. That way, the clear operation of mces_seen will be
skipped, and the stale value of mces_seen will reappear on the next mce.

thx!
cyc

Re: [PATCH] x86/mce: Distirbute the clear operation of mces_seen to Per-CPU rather than only monarch CPU

[Borislav Petkov]

On Mon, May 19, 2014 at 03:54:54PM +0800, Chen Yucong wrote:
> If a timeout occurs in monarch CPU, what will happen for the above
> code segment?

I think I answered that question already, didn't I?

-- 
Regards/Gruss,
    Boris.

Sent from a fat crate under my desk. Formatting is fine.
--