hyperthreading and RT latency

hyperthreading and RT latency

[Alison Chaiken]

The advice in the RT wiki

    https://rt.wiki.kernel.org/index.php/HOWTO:_Build_an_RT-application#Hyper_threading

about configuring the kernel and building RT applications was written
in 2014, when we were on the 3.x series.   That makes one wonder how
relevant some of it is for the 5.x series, especially since processors
in common use have changed some since then.  Some of the advice,
notably about power management, obviously is timeless.

In particular, Daniel Wagner added:

    https://rt.wiki.kernel.org/index.php/HOWTO:_Build_an_RT-application#Hyper_threading

    "Hyper threading and also out of order execution of CPUs introduces
    'random' latencies. As mentioned in power management, it is
     recommended to disable these feature (if possible) or carefully
     benchmark the performance."

Is the advice still current?   Should we RT-users all still turn
hyperthreading off?   Given the security vulnerabilities associated
with hyperthreading, there are clearly some use cases where doing so
is indicated anyway.

Thanks,
Alison Chaiken
Aurora Technology

Re: hyperthreading and RT latency

[Daniel Wagner]

Hi Alison,

On Thu, Aug 05, 2021 at 01:16:17PM -0700, Alison Chaiken wrote:
> The advice in the RT wiki
> 
>     https://rt.wiki.kernel.org/index.php/HOWTO:_Build_an_RT-application#Hyper_threading
> 
> about configuring the kernel and building RT applications was written
> in 2014, when we were on the 3.x series.   That makes one wonder how
> relevant some of it is for the 5.x series, especially since processors
> in common use have changed some since then.  Some of the advice,
> notably about power management, obviously is timeless.
> 
> In particular, Daniel Wagner added:
> 
>     https://rt.wiki.kernel.org/index.php/HOWTO:_Build_an_RT-application#Hyper_threading
> 
>     "Hyper threading and also out of order execution of CPUs introduces
>     'random' latencies. As mentioned in power management, it is
>      recommended to disable these feature (if possible) or carefully
>      benchmark the performance."
> 
> Is the advice still current?

I don't think the situation has changed. Though, I don't run these test
on modern hardware on regular basis. The last time I played with it on a
bit more modern hardware it was still measurable by cyclictest with
hackbench as workload.

> Should we RT-users all still turn hyperthreading off?

Obviously, it depends on your use case. If your application can tolerate
the added noise by SMT, you don't have to disable it.

> Given the security vulnerabilities associated
> with hyperthreading, there are clearly some use cases where doing so
> is indicated anyway.

Again it depends on your use case.

Daniel

Re: hyperthreading and RT latency

[John Kacur]

On Fri, 6 Aug 2021, Daniel Wagner wrote:

> Hi Alison,
> 
> On Thu, Aug 05, 2021 at 01:16:17PM -0700, Alison Chaiken wrote:
> > The advice in the RT wiki
> > 
> >     https://rt.wiki.kernel.org/index.php/HOWTO:_Build_an_RT-application#Hyper_threading

Note that the above wiki is defunct, the current one to read is:
https://wiki.linuxfoundation.org/realtime/start

> > 
> > about configuring the kernel and building RT applications was written
> > in 2014, when we were on the 3.x series.   That makes one wonder how
> > relevant some of it is for the 5.x series, especially since processors
> > in common use have changed some since then.  Some of the advice,
> > notably about power management, obviously is timeless.
> > 
> > In particular, Daniel Wagner added:
> > 
> >     https://rt.wiki.kernel.org/index.php/HOWTO:_Build_an_RT-application#Hyper_threading
> > 
> >     "Hyper threading and also out of order execution of CPUs introduces
> >     'random' latencies. As mentioned in power management, it is
> >      recommended to disable these feature (if possible) or carefully
> >      benchmark the performance."
> > 
> > Is the advice still current?
> 
> I don't think the situation has changed. Though, I don't run these test
> on modern hardware on regular basis. The last time I played with it on a
> bit more modern hardware it was still measurable by cyclictest with
> hackbench as workload.
> 
> > Should we RT-users all still turn hyperthreading off?
> 
> Obviously, it depends on your use case. If your application can tolerate
> the added noise by SMT, you don't have to disable it.
> 
> > Given the security vulnerabilities associated
> > with hyperthreading, there are clearly some use cases where doing so
> > is indicated anyway.
> 
> Again it depends on your use case.
> 
> Daniel
> 

The safe answer is always to measure it.

John

AW: hyperthreading and RT latency

[Jonathan Schwender]

Hi Alison,

> Is the advice still current?   Should we RT-users all still turn hyperthreading off?

I ran some tests as a part of my master's thesis in the beginning of this year with the 5.10-rt kernel on an Intel Broadwell-EP 2-socket server.
If you are interested, I can dig up the graphs I made, but the jist regarding wake-up latencies measured by _cyclictest_ (24 hours each) is:
1. Task-isolation (placing the RT-task on a dedicated core) + Cache allocation + disabled Hyperthreading yields the best latencies. Something around 4-5us worst-case latencies were possibly with some optimizations.
2. Placing a load (rteval) together with cyclictest increases the latencies, but worst-case latencies of I think 16us are still okay for many applications
3. Isolating a task on a dedicated CPU and placing a load (rteval) on the neighbor CPU sharing the same core yields strictly worse latencies compared to 2). I think it was around 50us worst-case.
4. Isolating a task on a dedicated core (hyperthreading disabled), but enabling hyperthreading for the non-critical cores seems to have a rather small negative impact, as long as CAT is used to reserve cache for the isolated core. I'd have to look up the details though.

I don't think the situation has improved on more modern hardware, since AFAIK the SMT hardware has no knowledge of your tasks priority.

>Thanks,
>Alison Chaiken

Best Regards,

Jonathan Schwender

Re: hyperthreading and RT latency

[Jack Winch]

Hi Alison,

You've already had a number of answers to your query already, but I'll
chime in with my two cents anyhow.

Although I am no longer in a position to provide actual data (having
left my previous employer a few weeks back, where I ran some
experiments for myself), my findings were similar in nature to those
of Jonathan (on a Gen 10 HPE ProLiant server, with a 5.x-rt kernel).

For our application, the 'random' latencies induced by the enabling of
hyper-threading were deemed acceptable and, given our overall
configuration of the target system, allowed for more satisfactory
performance across the range of applications and tasks running on the
system overall.  Hence, we kept it enabled despite the additional
latency noise.  So it all comes down to your use-case, the
configuration and particulars of your target system, and your system
performance requirements.

As John has pointed out already, the best thing to do is to try to
quantify any effects of enabling / disabling hyper-threading on your
system and to evaluate the effects with respect to your system
performance requirements.  This advice applies across the board when
trying to determine what impact certain system configuration changes
have on the real-time performance of your system.  The rt-tests suite
of tools may help you to do so.  If you've not used them already, I
*strongly* recommend you get familiar with these tools and their use,
as they are a crucial component of the RT Linux developer toolkit.
The following Wiki page links to some useful information regarding the
use of these tools, as well as other tools useful for RT Linux
development: https://wiki.linuxfoundation.org/realtime/documentation/howto/tools/start.

Jack

On Tue, Aug 10, 2021 at 7:54 AM Jonathan Schwender
<schwenderjonathan@gmail.com> wrote:
>
> Hi Alison,
>
> > Is the advice still current?   Should we RT-users all still turn hyperthreading off?
>
> I ran some tests as a part of my master's thesis in the beginning of this year with the 5.10-rt kernel on an Intel Broadwell-EP 2-socket server.
> If you are interested, I can dig up the graphs I made, but the jist regarding wake-up latencies measured by _cyclictest_ (24 hours each) is:
> 1. Task-isolation (placing the RT-task on a dedicated core) + Cache allocation + disabled Hyperthreading yields the best latencies. Something around 4-5us worst-case latencies were possibly with some optimizations.
> 2. Placing a load (rteval) together with cyclictest increases the latencies, but worst-case latencies of I think 16us are still okay for many applications
> 3. Isolating a task on a dedicated CPU and placing a load (rteval) on the neighbor CPU sharing the same core yields strictly worse latencies compared to 2). I think it was around 50us worst-case.
> 4. Isolating a task on a dedicated core (hyperthreading disabled), but enabling hyperthreading for the non-critical cores seems to have a rather small negative impact, as long as CAT is used to reserve cache for the isolated core. I'd have to look up the details though.
>
> I don't think the situation has improved on more modern hardware, since AFAIK the SMT hardware has no knowledge of your tasks priority.
>
> >Thanks,
> >Alison Chaiken
>
> Best Regards,
>
> Jonathan Schwender
>

Re: hyperthreading and RT latency

[Jack Winch]

Also, apologies for top posting, folks.  I thought my email client was
configured differently.  Won't happen again.

Jack