Re: [PATCH v9 09/17] x86/split_lock: Handle #AC exception for split lock

Xiaoyao Li
2019-10-16T06:58:16Z

On 9/26/2019 2:09 AM, Sean Christopherson wrote:
> On Wed, Jun 26, 2019 at 11:47:40PM +0200, Thomas Gleixner wrote:
>> So only one of the CPUs will win the cmpxchg race, set te variable to 1 and
>> warn, the other and any subsequent AC on any other CPU will not warn
>> either. So you don't need WARN_ONCE() at all. It's redundant and confusing
>> along with the atomic_set().
>>
>> Whithout reading that link [1], what Ingo proposed was surely not the
>> trainwreck which you decided to put into that debugfs thing.
>
> We're trying to sort out the trainwreck, but there's an additional wrinkle
> that I'd like your input on.
>
> We overlooked the fact that MSR_TEST_CTRL is per-core, i.e. shared by
> sibling hyperthreads. This is especially problematic for KVM, as loading
> MSR_TEST_CTRL during VM-Enter could cause spurious #AC faults in the kernel
> and bounce MSR_TEST_CTRL.split_lock.
>
> E.g. if CPU0 and CPU1 are siblings and CPU1 is running a KVM guest with
> MSR_TEST_CTRL.split_lock=1, hitting an #AC on CPU0 in the host kernel will
> lead to suprious #AC faults and constant toggling of of the MSR.
>
> CPU0 CPU1
>
> split_lock=enabled
>
> #AC -> disabled
>
> VM-Enter -> enabled
>
> #AC -> disabled
>
> VM-Enter -> enabled
>
> #AC -> disabled
>
>
>
> My thought to handle this:
>
> - Remove the per-cpu cache.
>
> - Rework the atomic variable to differentiate between "disabled globally"
> and "disabled by kernel (on some CPUs)".
>
> - Modify the #AC handler to test/set the same atomic variable as the
> sysfs knob. This is the "disabled by kernel" flow.
>
> - Modify the debugfs/sysfs knob to only allow disabling split-lock
> detection. This is the "disabled globally" path, i.e. sends IPIs to
> clear MSR_TEST_CTRL.split_lock on all online CPUs.
>
> - Modify the resume/init flow to clear MSR_TEST_CTRL.split_lock if it's
> been disabled on *any* CPU via #AC or via the knob.
>
> - Modify the debugs/sysfs read function to either print the raw atomic
> variable, or differentiate between "enabled", "disabled globally" and
> "disabled by kernel".
>
> - Remove KVM loading of MSR_TEST_CTRL, i.e. KVM *never* writes the CPU's
> actual MSR_TEST_CTRL. KVM still emulates MSR_TEST_CTRL so that the
> guest can do WRMSR and handle its own #AC faults, but KVM doesn't
> change the value in hardware.
>
> * Allowing guest to enable split-lock detection can induce #AC on
> the host after it has been explicitly turned off, e.g. the sibling
> hyperthread hits an #AC in the host kernel, or worse, causes a
> different process in the host to SIGBUS.
>
> * Allowing guest to disable split-lock detection opens up the host
> to DoS attacks.
>
> - KVM advertises split-lock detection to guest/userspace if and only if
> split_lock_detect_disabled is zero.
>
> - Add a pr_warn_once() in KVM that triggers if split locks are disabled
> after support has been advertised to a guest.
>
> Does this sound sane?
>
> The question at the forefront of my mind is: why not have the #AC handler
> send a fire-and-forget IPI to online CPUs to disable split-lock detection
> on all CPUs? Would the IPI be problematic? Globally disabling split-lock
> on any #AC would (marginally) simplify the code and would eliminate the
> oddity of userspace process (and KVM guest) #AC behavior varying based on
> the physical CPU it's running on.
>
>
> Something like:
>
> #define SPLIT_LOCK_DISABLED_IN_KERNEL BIT(0)
> #define SPLIT_LOCK_DISABLED_GLOBALLY BIT(1)
>
> static atomic_t split_lock_detect_disabled = ATOMIT_INIT(0);
>
> void split_lock_detect_ac(void)
> {
> lockdep_assert_irqs_disabled();
>
> /* Disable split lock detection on this CPU to avoid reentrant #AC. */
> wrmsrl(MSR_TEST_CTRL,
> rdmsrl(MSR_TEST_CTRL) & ~MSR_TEST_CTRL_SPLIT_LOCK_DETECT);
>
> /*
> * If split-lock detection has not been disabled, either by the kernel
> * or globally, record that it has been disabled by the kernel and
> * WARN. Guarding WARN with the atomic ensures only the first #AC due
> * to split-lock is logged, e.g. if multiple CPUs encounter #AC or if
> * #AC is retriggered by a perf context NMI that interrupts the
> * original WARN.
> */
> if (atomic_cmpxchg(&split_lock_detect_disabled, 0,
> SPLIT_LOCK_DISABLED_IN_KERNEL) == 0)
> WARN(1, "split lock operation detected\n");
> }
>
> static ssize_t split_lock_detect_wr(struct file *f, const char __user *user_buf,
> size_t count, loff_t *ppos)
> {
> int old;
>
> <parse or ignore input value?>
>
> old = atomic_fetch_or(SPLIT_LOCK_DISABLED_GLOBALLY,
> &split_lock_detect_disabled);
>
> /* Update MSR_TEST_CTRL unless split-lock was already disabled. */
> if (!(old & SPLIT_LOCK_DISABLED_GLOBALLY))
> on_each_cpu(split_lock_update, NULL, 1);
>
> return count;
> }
>

Hi Thomas,

Could you please have a look at Sean's proposal and give your opinion.