* Re: cpu-freq: running the perf increases the data rate?
2020-08-28 12:34 ` Artem Bityutskiy
@ 2020-08-31 16:22 ` Subhashini Rao Beerisetty
0 siblings, 0 replies; 3+ messages in thread
From: Subhashini Rao Beerisetty @ 2020-08-31 16:22 UTC (permalink / raw)
To: dedekind1; +Cc: linux-pm, kernelnewbies, LKML
[-- Attachment #1: Type: text/plain, Size: 3332 bytes --]
On Fri, Aug 28, 2020 at 6:04 PM Artem Bityutskiy <dedekind1@gmail.com> wrote:
>
> On Thu, 2020-08-27 at 22:25 +0530, Subhashini Rao Beerisetty wrote:
> > I have an application which finds the data rate over the PCIe
> > interface. I’m getting the lesser data rate in one of my Linux X86
> > systems.
>
> Some more description, may be? Do you have a PCIe device reading one
> RAM buffer and then writing to another RAM buffer? Or does it generate
> dome data and writes them to a RAM buffer? Presumably it uses DMA? How
> much is the CPU involved into the process? Are we talking about
> transferring few kilobytes or gigabytes?
Thanks a lot for your help and reply.
Regarding hardware setup, Xilinx PCIe FPGA endpoint is connected to
HOST CPU via PCIe bus.
Xilinx PCIe FPGA endpoint has the DMA_REF block and it provides a
mechanism to DMA transfer data at the maximum rate between host CPU
memory and a FIFO in the DMA-REF block.
The host software sets up some data in it’s memory, it then transfers
the data to the DMA-REF’s FIFO and then reads it back into a different
location in its host memory. This is repeated in a loop. There is a
register in the DMA-REF block that gives an indication of transfer
speed.
>
> > When I change the scaling_governor from "powersave" to "performance"
> > mode for each CPU, then there is slight improvement in the PCIe data
> > rate.
>
> Definitely this makes your CPU(s) run at max speed, but depending on
> platform and settings, this may also affect C-states. Are the CPU(s)
> generally idle while you measure, or busy (involved into the test)? You
> could run 'turbostat' while measuring the bandwidth, to get some CPU
> statistics (e.g., do C-states happen during the PCI test, how busy are
> the CPUs).
>
> > Parallely I started profiling the workload with perf. Whenever I start
> > running the profile command “perf stat -a -d -p <PID>” surprisingly
> > the application resulted in excellent data rate over PCIe, but when I
> > kill the perf command again PCIe data rate drops. I am really confused
> > about this behavior.Any clues from this behaviour?
>
> Well, one possible reason that comes to mind - you get rid of C-states
> when you rung perf, and this increases the PCI bandwidth. You can just
> try disabling C-states (there are sysfs knobs) and check it out.
> Turbostat could be useful to check for this (with and without perf, run
> 'turbostat sleep 10' or something like this (measure for 10 seconds in
> this example), do this while running your PCI test.
Disabling the C-states improved the throughput a lot, thanks a lot for
pointing this out. Could you please give some more explanation on how
disabling C-states improved the throughput?
As you suggested I collected and attached the turbostat log with and
without perf while running the PCIe test.
In my system, only 'performance\powersave' are listed in
scaling_available_governors. Rest other governors
"userspace\ondemand\schedutil" are not listed in available_goverors.
What might be the reason for this?
>
> But I am really just guessing here, I do not know enough about your
> test and the system (e.g., "a Linux x86" system can be so many things,
> like Intel or AMD server or a mobile device)…
It's an Intel Atom processor.
>
>
[-- Attachment #2: trubostat_with_perf.txt --]
[-- Type: text/plain, Size: 2633 bytes --]
turbostat version 17.06.23 - Len Brown <lenb@kernel.org>
CPUID(0): GenuineIntel 11 CPUID levels; family:model:stepping 0x6:37:9 (6:55:9)
CPUID(1): SSE3 MONITOR - EIST TM2 TSC MSR ACPI-TM TM
CPUID(6): APERF, No-TURBO, DTS, No-PTM, No-HWP, No-HWPnotify, No-HWPwindow, No-HWPepp, No-HWPpkg, EPB
cpu2: MSR_IA32_MISC_ENABLE: 0x00850089 (TCC EIST No-MWAIT PREFETCH TURBO)
CPUID(7): No-SGX
SLM BCLK: 83.3 Mhz
cpu2: MSR_CC6_DEMOTION_POLICY_CONFIG: 0x00000000 (DISable-CC6-Demotion)
cpu2: MSR_MC6_DEMOTION_POLICY_CONFIG: 0x00000000 (DISable-MC6-Demotion)
RAPL: 4581 sec. Joule Counter Range, at 30 Watts
cpu2: MSR_PLATFORM_INFO: 0x60000001700
6 * 83.3 = 499.8 MHz max efficiency frequency
23 * 83.3 = 1915.9 MHz base frequency
cpu2: MSR_IA32_POWER_CTL: 0x00000000 (C1E auto-promotion: DISabled)
cpu2: MSR_ATOM_CORE_RATIOS: 0x00170602
2 * 83.3 = 166.6 MHz minimum operating frequency
6 * 83.3 = 499.8 MHz low frequency mode (LFM)
23 * 83.3 = 1915.9 MHz base frequency
cpu2: MSR_ATOM_CORE_TURBO_RATIOS: 0x17171717
23 * 83.3 = 1915.9 MHz max turbo 4 active cores
23 * 83.3 = 1915.9 MHz max turbo 3 active cores
23 * 83.3 = 1915.9 MHz max turbo 2 active cores
23 * 83.3 = 1915.9 MHz max turbo 1 active core
cpu2: MSR_PKG_CST_CONFIG_CONTROL: 0x0017000f (UNlocked: pkg-cstate-limit=15: pc7)
cpu2: POLL: CPUIDLE CORE POLL IDLE
cpu2: C1: MWAIT 0x00
cpu2: C6N: MWAIT 0x58
cpu2: C6S: MWAIT 0x52
cpu2: cpufreq driver: intel_pstate
cpu2: cpufreq governor: performance
cpufreq intel_pstate no_turbo: 1
cpu0: MSR_IA32_ENERGY_PERF_BIAS: 0x00000006 (balanced)
cpu0: MSR_RAPL_POWER_UNIT: 0x00000505 (0.031250 Watts, 0.000032 Joules, 0.000977 sec.)
cpu0: MSR_PKG_POWER_LIMIT: 0x003880fa (UNlocked)
cpu0: PKG Limit #1: ENabled (7.812500 Watts, 262144.000000 sec, clamp DISabled)
cpu0: PKG Limit #2: DISabled (0.000000 Watts, 0.000977* sec, clamp DISabled)
cpu0: MSR_PP0_POWER_LIMIT: 0x00020000 (UNlocked)
cpu0: Cores Limit: DISabled (0.000000 Watts, 0.001953 sec, clamp DISabled)
cpu0: MSR_IA32_TEMPERATURE_TARGET: 0x006e0000 (110 C)
40.004417 sec
Core CPU Avg_MHz Busy% Bzy_MHz TSC_MHz IRQ SMI C1 C6N C6S C1% C6N% C6S% CPU%c1 CPU%c6 Mod%c6 CoreTmp GFX%rc6 Pkg%pc6 PkgWatt CorWatt
- - 748 39.33 1901 1917 458560 0 1566280 1889 7775 43.07 0.29 18.19 42.23 17.85 13.56 29 100.00 11.80 1.14 0.95
0 0 643 33.89 1896 1917 33647 0 405339 492 1950 48.44 0.31 18.28 47.57 17.69 13.32 28 100.00 11.80 1.14 0.95
1 1 1064 55.78 1906 1917 358521 0 449875 25 1910 27.37 0.03 17.83 26.41 17.34 13.32 29
2 2 647 34.14 1895 1917 33821 0 359155 517 1940 48.21 0.29 18.20 47.43 17.86 13.81 28
3 3 638 33.52 1902 1917 32571 0 351911 855 1975 48.27 0.54 18.47 47.51 18.51 13.81 28
[-- Attachment #3: trubostat_without_perf.txt --]
[-- Type: text/plain, Size: 2642 bytes --]
turbostat version 17.06.23 - Len Brown <lenb@kernel.org>
CPUID(0): GenuineIntel 11 CPUID levels; family:model:stepping 0x6:37:9 (6:55:9)
CPUID(1): SSE3 MONITOR - EIST TM2 TSC MSR ACPI-TM TM
CPUID(6): APERF, No-TURBO, DTS, No-PTM, No-HWP, No-HWPnotify, No-HWPwindow, No-HWPepp, No-HWPpkg, EPB
cpu2: MSR_IA32_MISC_ENABLE: 0x00850089 (TCC EIST No-MWAIT PREFETCH TURBO)
CPUID(7): No-SGX
SLM BCLK: 83.3 Mhz
cpu2: MSR_CC6_DEMOTION_POLICY_CONFIG: 0x00000000 (DISable-CC6-Demotion)
cpu2: MSR_MC6_DEMOTION_POLICY_CONFIG: 0x00000000 (DISable-MC6-Demotion)
RAPL: 4581 sec. Joule Counter Range, at 30 Watts
cpu2: MSR_PLATFORM_INFO: 0x60000001700
6 * 83.3 = 499.8 MHz max efficiency frequency
23 * 83.3 = 1915.9 MHz base frequency
cpu2: MSR_IA32_POWER_CTL: 0x00000000 (C1E auto-promotion: DISabled)
cpu2: MSR_ATOM_CORE_RATIOS: 0x00170602
2 * 83.3 = 166.6 MHz minimum operating frequency
6 * 83.3 = 499.8 MHz low frequency mode (LFM)
23 * 83.3 = 1915.9 MHz base frequency
cpu2: MSR_ATOM_CORE_TURBO_RATIOS: 0x17171717
23 * 83.3 = 1915.9 MHz max turbo 4 active cores
23 * 83.3 = 1915.9 MHz max turbo 3 active cores
23 * 83.3 = 1915.9 MHz max turbo 2 active cores
23 * 83.3 = 1915.9 MHz max turbo 1 active core
cpu2: MSR_PKG_CST_CONFIG_CONTROL: 0x0017000f (UNlocked: pkg-cstate-limit=15: pc7)
cpu2: POLL: CPUIDLE CORE POLL IDLE
cpu2: C1: MWAIT 0x00
cpu2: C6N: MWAIT 0x58
cpu2: C6S: MWAIT 0x52
cpu2: cpufreq driver: intel_pstate
cpu2: cpufreq governor: performance
cpufreq intel_pstate no_turbo: 1
cpu0: MSR_IA32_ENERGY_PERF_BIAS: 0x00000006 (balanced)
cpu0: MSR_RAPL_POWER_UNIT: 0x00000505 (0.031250 Watts, 0.000032 Joules, 0.000977 sec.)
cpu0: MSR_PKG_POWER_LIMIT: 0x003880fa (UNlocked)
cpu0: PKG Limit #1: ENabled (7.812500 Watts, 262144.000000 sec, clamp DISabled)
cpu0: PKG Limit #2: DISabled (0.000000 Watts, 0.000977* sec, clamp DISabled)
cpu0: MSR_PP0_POWER_LIMIT: 0x00020000 (UNlocked)
cpu0: Cores Limit: DISabled (0.000000 Watts, 0.001953 sec, clamp DISabled)
cpu0: MSR_IA32_TEMPERATURE_TARGET: 0x006e0000 (110 C)
40.003807 sec
Core CPU Avg_MHz Busy% Bzy_MHz TSC_MHz IRQ SMI C1 C6N C6S C1% C6N% C6S% CPU%c1 CPU%c6 Mod%c6 CoreTmp GFX%rc6 Pkg%pc6 PkgWatt CorWatt
- - 522 27.23 1915 1917 285074 0 784132 86336 168235 31.17 13.41 28.89 30.75 26.97 3.75 30 100.00 3.60 1.12 0.94
0 0 412 21.50 1915 1917 11229 0 155840 13229 69746 24.68 7.95 46.57 24.35 35.28 3.80 29 100.00 3.60 1.12 0.94
1 1 764 39.91 1914 1917 242649 0 269925 25179 32418 26.95 13.44 20.52 26.38 19.26 3.80 29
2 2 476 24.82 1915 1917 16549 0 224377 27839 1269 48.85 21.52 5.44 48.36 19.04 3.70 30
3 3 435 22.70 1915 1917 14647 0 133990 20089 64802 24.19 10.74 43.05 23.90 34.28 3.70 30
^ permalink raw reply [flat|nested] 3+ messages in thread