Re: [PATCH v5 0/9] "Task_isolation" mode

Yury Norov
2020-12-11T19:46:22Z

On Sun, Dec 06, 2020 at 12:25:45AM +0100, Thomas Gleixner wrote:
> Pavel,
>
> On Sat, Dec 05 2020 at 21:40, Pavel Machek wrote:
> > So... what kind of guarantees does this aim to provide / what tasks it
> > is useful for?
> >
> > For real time response, we have other approaches.
>
> Depends on your requirements. Some problems are actually better solved
> with busy polling. See below.
>
> > If you want to guarantee performnace of the "isolated" task... I don't
> > see how that works. Other tasks on the system still compete for DRAM
> > bandwidth, caches, etc...
>
> Applications which want to run as undisturbed as possible. There is
> quite a range of those:
>
> - Hardware in the loop simulation is today often done with that crude
> approach of "offlining" a CPU and then instead of playing dead
> jumping to a preloaded bare metal executable. That's a horrible hack
> and impossible to debug, but gives them the results they need to
> achieve. These applications are well optimized vs. cache and memory
> foot print, so they don't worry about these things too much and they
> surely don't run on SMI and BIOS value add inflicted machines.
>
> Don't even think about waiting for an interrupt to achieve what
> these folks are doing. So no, there are problems which a general
> purpose realtime OS cannot solve ever.
>
> - HPC computations on large data sets. While the memory foot print is
> large the access patterns are cache optimized.
>
> The problem there is that any unnecessary IPI, tick interrupt or
> whatever nuisance is disturbing the carefully optimized cache usage
> and alone getting rid of the timer interrupt gained them measurable
> performance. Even very low single digit percentage of runtime saving
> is valuable for these folks because the compute time on such beasts
> is expensive.
>
> - Realtime guests in KVM. With posted interrupts and a fully populated
> host side page table there is no point in running host side
> interrupts or IPIs for random accounting or whatever purposes as
> they affect the latency in the guest. With all the side effects
> mitigated and a properly set up guest and host it is possible to get
> to a zero exit situation after the bootup phase which means pretty
> much matching bare metal behaviour.
>
> Yes, you can do that with e.g. Jailhouse as well, but you lose lots
> of the fancy things KVM provides. And people care about these not
> just because they are fancy. They care because their application
> scenario needs them.
>
> There are more reasons why people want to be able to get as much
> isolation from the OS as possible but at the same time have a sane
> execution environment, debugging, performance monitoring and the OS
> provided protection mechanisms instead of horrible hacks.
>
> Isolation makes sense for a range of applications and there is no reason
> why Linux should not support them.

One good client for the task isolation is Open Data Plane. There are
even some code stubs supposed to enable isolation where needed.

> > If you want to guarantee performnace of the "isolated" task... I don't
> > see how that works. Other tasks on the system still compete for DRAM
> > bandwidth, caches, etc...

My experiments say that typical delay caused by dry IPI or syscall is
2000-20000 'ticks'. Typical delay caused by cache miss is 3-30 ticks.

To guarantee cache / memory bandwidth, one can use resctrl. Linux has
implementation of it for x86 only, but arm64 has support for for
resctrl on CPU side.

Thanks,
Yury

> Thanks,
>
> tglx