Re: [PATCH] x86: serialize LVTT and TSC_DEADLINE write

[Shaohua Li <shli@fb.com>]

On Tue, Sep 08, 2015 at 08:39:37PM -0700, Andi Kleen wrote:
> > Hmm, I didn't mean mfence can't serialize the instructions. For a true
> > IO, a serialization can't guarantee device finishes the IO, we generally
> > read some safe IO registers to wait IO finish. I completely don't know
> > if this case fits here though.
> 
> Sorry for the late answer. We (Intel) analyzed this case in detail and
> can confirm that the following sequence
> 
> 1.   Memory-mapped write to LVT Timer Register, setting bits 18:17 to 10b.
> 23. MFENCE.
> 4.   WRMSR to the IA32_TSC_DEADLINE MSR the desired deadline.
> 
> has the same effect as the loop algorithm described in the SDM on all Intel
> CPUs. So it's fine to use MFENCE here.

Thank you very much, Andi!

Here is the updated patch.


>From b5cdd3c14b0589e5d5a72d607814d9e808d05765 Mon Sep 17 00:00:00 2001
Message-Id: <b5cdd3c14b0589e5d5a72d607814d9e808d05765.1441771848.git.shli@fb.com>
From: Shaohua Li <shli@fb.com>
Date: Thu, 30 Jul 2015 16:24:43 -0700
Subject: [PATCH] x86: serialize LVTT and TSC_DEADLINE write

We saw a strange issue with local APIC timer. Some random CPU doesn't
receive any local APIC timer interrupt, which causes different issues.
The cpu uses TSC-Deadline mode for local APIC timer and APIC is in xAPIC
mode. When this happens, manually writing TSC_DEADLINE MSR can trigger
interrupt again and the system goes normal.

Currently we only see this issue in E5-2660 v2 and E5-2680 v2 CPU.
Compiler version seems mattering too, it's quite easy to reproduce the
issue with v4.7 gcc.

Since the local APIC timer interrupt number is 0, we either lose the
first interrupt or TSC_DEADLINE MSR isn't set correctly. After some
debugging, we believe it's the serialize issue described in Intel SDM.
In xAPIC mode, write to APIC LVTT and write to TSC_DEADLINE isn't
serialized. Debug shows read TSC_DEADLINE MSR followed the very first
MSR write returns 0 in the buggy cpu.

Thanks Andi Kleen confirms a 'mfence' can do the serialization here.
'mfence' instruction should be supported by CPU with TSC_DEADLINE, so we
don't need to check the instruction availability.

Cc: Suresh Siddha <suresh.b.siddha@intel.com>
Cc: Thomas Gleixner <tglx@linutronix.de>
Cc: H. Peter Anvin <hpa@zytor.com>
Cc: Ingo Molnar <mingo@kernel.org>
Cc: Andi Kleen <ak@linux.intel.com>
Cc: stable@vger.kernel.org v3.7+
Signed-off-by: Shaohua Li <shli@fb.com>
---
 arch/x86/kernel/apic/apic.c | 7 +++++++
 1 file changed, 7 insertions(+)

diff --git a/arch/x86/kernel/apic/apic.c b/arch/x86/kernel/apic/apic.c
index 3ca3e46..e3fd4ab 100644
--- a/arch/x86/kernel/apic/apic.c
+++ b/arch/x86/kernel/apic/apic.c
@@ -336,6 +336,13 @@ static void __setup_APIC_LVTT(unsigned int clocks, int oneshot, int irqen)
 	apic_write(APIC_LVTT, lvtt_value);
 
 	if (lvtt_value & APIC_LVT_TIMER_TSCDEADLINE) {
+		/*
+		 * See Intel SDM: TSC-Deadline Mode chapter. In xAPIC mode,
+		 * writing APIC LVTT and TSC_DEADLINE MSR isn't serialized.
+		 * According to Intel, mfence can do the serialization here.
+		 * */
+		asm volatile("mfence" : : : "memory");
+
 		printk_once(KERN_DEBUG "TSC deadline timer enabled\n");
 		return;
 	}
-- 
1.8.1