Re: [RFC PATCH v4 00/19] Core scheduling v4

Thomas Gleixner
2020-03-18T11:53:29Z

Joel,

Joel Fernandes <joel@joelfernandes.org> writes:
> We have only 2 cores (4 HT) on many devices. It is not an option to
> dedicate a core to only running trusted code, that would kill
> performance. Another option is to designate a single HT of a
> particular core to run both untrusted code and an interrupt handler --
> but as Thomas pointed out, this does not work for per-CPU interrupts
> or managed interrupts, and the softirqs that they trigger. But if we
> just consider interrupts for which we can control the affinities (and
> assuming that most interrupts can be controlled like that), then maybe
> it will work? In the ChromeOS model, each untrusted task is in its own
> domain (cookie). So untrusted tasks cannot benefit from parallelism
> (in our case) anyway -- so it seems reasonable to run an affinable
> interrupt and an untrusted task, on a particular designated core.
>
> (Just thinking out loud...). Another option could be a patch that
> Vineeth shared with me (that Peter experimentally wrote) where he
> sends IPI from an interrupt handler to a sibling running untrusted
> guest code which would result in it getting paused. I am hoping
> something like this could work on the host side as well (not just for
> guests). We could also set per-core state from the interrupted HT,
> possibly IPI'ing the untrusted sibling if we have to. If sibling runs
> untrusted code *after* the other's siblings interrupt already started,
> then the schedule() loop on the untrusted sibling would spin knowing
> the other sibling has an interrupt in progress. The softirq is a real
> problem though. Perhaps it can also set similar per-core state.

There is not much difference between bringing the sibling out of guest
mode and bringing it out of host user mode.

Adding state to force spinning until the other side has completed is not
rocket science either.

But the whole concept is prone to starvation issues and full of nasty
corner cases. From experiments I did back in the L1TF days I'm pretty
much convinced that this can't result in a generaly usable solution.

Let me share a few thoughts what might be doable with less horrors, but
be aware that this is mostly a brain dump of half thought out ideas.

1) Managed interrupts on multi queue devices

It should be reasonably simple to force a reduced number of queues
which would in turn allow to shield one ore two CPUs from such
interrupts and queue handling for the price of indirection.

2) Timers and softirqs

If device interrupts are targeted to "safe" CPUs then the amount of
timers and soft interrupt processing will be reduced as well.

That still leaves e.g. network TX side soft interrupts when the task
running on a shielded core does networking. Maybe that's a non issue,
but I'm not familiar enough with the network maze to give an answer.

A possible workaround would be to force softirq processing into
thread context so everything is under scheduler control. How well
that scales is a different story.

That would bring out the timer_list timers and reduce the potential
surface to hrtimer expiry callbacks. Most of them should be fine
(doing wakeups or scheduler housekeeping of some sort). For the
others we might just utilize the mechanism which PREEMPT_RT uses and
force them off into softirq expiry mode.

Thanks,

tglx