Re: 5.15.28-rt35 #2 SMP PREEMPT_RT: Should scheduling latency be as large as 800 usec?

[John Ogness <john.ogness@linutronix.de>]

On 2022-03-23, Gautam Thaker <ghthaker@gmail.com> wrote:
>> > processor       : 31
>> > vendor_id       : GenuineIntel
>> > cpu family      : 6
>> > model           : 63
>> > model name      : Intel(R) Xeon(R) CPU E5-2630 v3 @ 2.40GHz
>> > stepping        : 2
>> > microcode       : 0x46
>> > cpu MHz         : 1200.000
>> > cache size      : 20480 KB
>>
>> Your CPU is running at 1.2GHz although it is capable of 2.4GHz. This
>> looks like you have CPU frequency scaling activated. Investigate:
>>
>> grep -i cpu_freq /boot/config-5.15.28-rt35
>
> node-0> grep -i cpu_freq /boot/config-5.15.28-rt35
> CONFIG_ACPI_CPU_FREQ_PSS=y
> CONFIG_CPU_FREQ=y
> CONFIG_CPU_FREQ_GOV_ATTR_SET=y
> CONFIG_CPU_FREQ_GOV_COMMON=y
> CONFIG_CPU_FREQ_STAT=y
> CONFIG_CPU_FREQ_DEFAULT_GOV_PERFORMANCE=y
> # CONFIG_CPU_FREQ_DEFAULT_GOV_POWERSAVE is not set
> # CONFIG_CPU_FREQ_DEFAULT_GOV_USERSPACE is not set
> # CONFIG_CPU_FREQ_DEFAULT_GOV_SCHEDUTIL is not set
> CONFIG_CPU_FREQ_GOV_PERFORMANCE=y
> CONFIG_CPU_FREQ_GOV_POWERSAVE=y
> CONFIG_CPU_FREQ_GOV_USERSPACE=y
> CONFIG_CPU_FREQ_GOV_ONDEMAND=y
> CONFIG_CPU_FREQ_GOV_CONSERVATIVE=y
> CONFIG_CPU_FREQ_GOV_SCHEDUTIL=y

Your system is configured to support many different governors. For
minimal latency it is important that you are using the performance
governor. The frequency scaling might be the cause of the horrible
hwlatdetect numbers you are seeing.

If you have the cpufrequtils package installed, you can easily switch to
the performance governor with:

cpufreq-set -g performance

With cpufreq-info you can see a nice summary of how your CPUs are
currently set. You should see it running full speed.

You can also configure the performance governor manually using
sysfs. You may want to read through the documentation [0] so that you
understand what you are doing.

[0] https://www.kernel.org/doc/html/latest/admin-guide/pm/cpufreq.html

>> The configuration of your tick may also be interesting:
>>
>> grep -i hz /boot/config-5.15.28-rt35
>>
>
> node-0> grep -i hz /boot/config-5.15.28-rt35
> CONFIG_NO_HZ_COMMON=y
> # CONFIG_HZ_PERIODIC is not set
> CONFIG_NO_HZ_IDLE=y
> # CONFIG_NO_HZ_FULL is not set
> CONFIG_NO_HZ=y
> # CONFIG_HZ_100 is not set
> CONFIG_HZ_250=y
> # CONFIG_HZ_300 is not set
> # CONFIG_HZ_1000 is not set
> CONFIG_HZ=250
> # CONFIG_MACHZ_WDT is not set

The tick configuration won't be responsible for the huge latencies you
are seeing. But when you start getting your latency down, you may want
to consider using CONFIG_HZ_PERIODIC with CONFIG_HZ_100 (or
CONFIG_HZ_1000 if this is a machine with a huge network load). But these
settings can be very dependent on the types of work your system is
doing, so you will need to test the latency affects. Possibly you will
notice no affects.

Adding --secaligned to your cyclictest command can help to force
situations where your latency will be worsened because of the
tick. Optimally you want cyclictest to hit the worst case scenario.

John Ogness