Re: [PATCH v8 06/14] task_isolation: provide strict mode configurable signal

[Chris Metcalf <cmetcalf@ezchip.com>]

On 10/21/2015 02:53 PM, Andy Lutomirski wrote:
> On Tue, Oct 20, 2015 at 11:41 PM, Gilad Ben Yossef <giladb@ezchip.com> wrote:
>>
>>> From: Andy Lutomirski [mailto:luto@amacapital.net]
>>> Sent: Wednesday, October 21, 2015 4:43 AM
>>> To: Chris Metcalf
>>> Subject: Re: [PATCH v8 06/14] task_isolation: provide strict mode
>>> configurable signal
>>>
>>> On Tue, Oct 20, 2015 at 6:30 PM, Chris Metcalf <cmetcalf@ezchip.com>
>>> wrote:
>>>> On 10/20/2015 8:56 PM, Steven Rostedt wrote:
>>>>> On Tue, 20 Oct 2015 16:36:04 -0400
>>>>> Chris Metcalf <cmetcalf@ezchip.com> wrote:
>>>>>
>>>>>> Allow userspace to override the default SIGKILL delivered
>>>>>> when a task_isolation process in STRICT mode does a syscall
>>>>>> or otherwise synchronously enters the kernel.
>>>>>>
>> <snip>
>>>> It doesn't map SIGKILL to some other signal unconditionally.  It just allows
>>>> the "hey, you broke the STRICT contract and entered the kernel" signal
>>>> to be something besides the default SIGKILL.
>>>>
>> <snip>
>>> I still dislike this thing.  It seems like a debugging feature being
>>> implemented using signals instead of existing APIs.  I *still* don't
>>> see why perf can't be used to accomplish your goal.
>>>
>> It is not (just) a debugging feature. There are workloads were not performing an action is much preferred to being late.
>>
>> Consider the following artificial but representative scenario: a task running in strict isolation is controlling a radiotherapy alpha emitter.
>> The code runs in a tight event loop, reading an MMIO register with location data, making some calculation and in response writing an
>> MMIO register that triggers the alpha emitter. As a safety measure, each trigger is for a specific very short time frame - the alpha emitter
>> auto stops.
>>
>> The code has a strict assumption that no more than X cycles pass between reading the value and the response and the system is built in
>> such a way that as long as the code has mastery of the CPU the assumption holds true. If something breaks this assumption (unplanned
>> context switch to kernel), what you want to do is just stop place
>> rather than fire the alpha emitter X nanoseconds too late.
>>
>> This feature lets you say: if the "contract" of isolation is broken, notify/kill me at once.
> That's a fair point.  It's risky, though, for quite a few reasons.
>
> 1. If someone builds an alpha emitter like this, they did it wrong.
> The kernel should write a trigger *and* a timestamp to the hardware
> and the hardware should trigger at the specified time if the time is
> in the future and throw an error if it's in the past.  If you need to
> check that you made the deadline, check the actual desired condition
> (did you meat the deadline?) not a proxy (did the signal fire?).

Definitely a better hardware design, but as we all know, hardware
designers too rarely consult the software people who have to
right the actual code to properly drive the hardware :-)

My canonical example is high-performance userspace network
drivers, and though dropping is packet is less likely to kill a
patient, it's still a pretty bad thing if you're trying to design
a robust appliance.  In this case you really want to fix application
bugs that cause the code to enter the kernel when you think
you're in the internal loop running purely in userspace.  Things
like unexpected page faults, and third-party code that almost
never calls the kernel but in some dusty corner it occasionally
does, can screw up your userspace code pretty badly, and
mysteriously.  The "strict" mode support is not a hypothetical
insurance policy but a reaction to lots of Tilera customer support
over the years to folks failing to stay in userspace when they
thought they were doing the right thing.

> 2. This strict mode thing isn't exhaustive.  It's missing, at least,
> coverage for NMI, MCE, and SMI.  Sure, you can think that you've
> disabled all NMI sources, you can try to remember to set the
> appropriate boot flag that panics on MCE (and hope that you don't get
> screwed by broadcast MCE on Intel systems before it got fixed
> (Skylake?  Is the fix even available in a released chip?), and, for
> SMI, good luck...

You are confusing this strict mode support with the debug
support in patch 07/14.

Strict mode is for synchronous application errors.  You might
be right that there are cases that haven't been covered, but
certainly most of them are covered on the three platforms that
are supported in this initial series.  (You pointed me to one
that I would have missed on x86, namely the bounds check
exception from a bad bounds setup.)  I'm pretty confident I
have all of them for tile, since I know that hardware best,
and I think we're in good shape for arm64, though I'm still
coming up to speed on that architecture.

NMIs and machine checks are asynchronous interrupts that
don't have to do with what the application is doing, more or less.
Those should not be delivered to task-isolation cores at all,
so we just generate console spew when you set the
task_isolation_debug boot option.  I honestly don't know enough
about system management interrupts to comment on that,
though again, I would hope one can configure the system to
just not deliver them to nohz_full cores, and I think it would
be reasonable to generate some kernel spew if that happens.

> 3. You haven't dealt with IPIs.  The TLB flush code in particular
> seems like it will break all your assumptions.

Again, not a synchronous application error that we are trying
to catch with this signalling mechanism.

That said it could obviously be a more general application error
(e.g. a process with threads on both nohz_full and housekeeping
cores, where the housekeeping core unmaps some memory and
thus requires a TLB flush IPI).  But this is covered by the
task_isolation_debug patch for kernel/smp.c.

> Maybe it would make sense to whack more of the moles before adding a
> big assertion that there aren't any moles any more.

Maybe, but I've whacked the ones I know how to whack.
If there are ones I've missed I'm happy to add them in a
subsequent version of this series, or in follow-on patches.

-- 
Chris Metcalf, EZChip Semiconductor
http://www.ezchip.com