* runnable tasks never making it to a runqueue where they can run?
@ 2012-03-07 2:19 Roland Dreier
2012-03-07 16:30 ` Peter Zijlstra
2012-03-07 16:31 ` Peter Zijlstra
0 siblings, 2 replies; 7+ messages in thread
From: Roland Dreier @ 2012-03-07 2:19 UTC (permalink / raw)
To: Peter Zijlstra, LKML
[-- Attachment #1: Type: text/plain, Size: 2685 bytes --]
Hi Peter, scheduler developers,
I'm seeing some strange scheduler behavior on a system with a process that has
a mix of SCHED_FIFO and SCHED_OTHER threads bound to various cpu masks.
The short story is that some SCHED_OTHER threads never get runtime, even
though there is one CPU where no SCHED_FIFO threads are allowed to run.
The longer story with details is that I have a system running 2.6.39.4 (I looked
through scheduler changes since then and didn't see anything relevant looking,
but of course I could have missed something -- and unfortunately rerunning
with a newer kernel is not totally trivial for this system).
It's a dual-socket Westmere Xeon system (2 threads per core, 6 cores per
package, 2 packages, so 24 rqs total). The program in question has 20
SCHED_FIFO tasks, 19 of which are bound to a single CPU, and one of
which which is limited to the cpumask 7ff5d7 (CPUs 0-2, 4, 6-8, 10, 12-22).
The SCHED_OTHER threads (of which there are quite a few) are bound
to the cpumask fff5d7 (CPUs 0-2, 4, 6-8, 10, 12-23). The main point here
is that all SCHED_OTHER threads can run on CPU 23, and none of the
SCHED_FIFO threads can run there.
Due to a couple of bugs in the process, the 19 single-CPU realtime threads
went beserk and never went to sleep, so their 19 CPUs are completely
given over to running SCHED_FIFO threads. (I have the default
# grep . /proc/sys/kernel/sched_rt*
/proc/sys/kernel/sched_rt_period_us:1000000
/proc/sys/kernel/sched_rt_runtime_us:950000
but since 5 CPUs never run RT threads, this limit never kicks in, and I
have no group scheduling or anything like that configured)
Another bug led to a SCHED_OTHER thread always being runnable, and this
is where things got weird: that thread seemingly became the only thing that
ran on CPU 23 -- all the other threads get load balanced between runqueues
where they have no chance of ever running because of the RT threads.
For example, I sampled /proc/sched_debug for a while and watched a
particular TID bounce between the rq of cpu 6,7,8,10,18,19,22 but never
onto 23 where it would actually get a chance to run. (Interestingly, those
8 CPUs seem to form a domain)
It seems something about this situation is confusing select_task_rq_fair()
so that CPU 23 always looks really bad, but I can't see what goes wrong.
Is this expected behavior? I can believe that we brought this on ourselves by
misconfiguring things, but in that case it would be good to know what we could
do to avoid this problem.
I'm attaching /proc/sched_debug and /proc/schedstat of the system in this state
in case that helps. Let me know if there's anything else I should gather from
the system.
Thanks!
Roland
[-- Attachment #2: sched_debug --]
[-- Type: application/octet-stream, Size: 53049 bytes --]
Sched Debug Version: v0.10, 2.6.39.4+ #17893
ktime : 74649308.456040
sched_clk : 74536259.822866
cpu_clk : 74536259.822902
jiffies : 4302402226
sched_clock_stable : 1
sysctl_sched
.sysctl_sched_latency : 24.000000
.sysctl_sched_min_granularity : 3.000000
.sysctl_sched_wakeup_granularity : 4.000000
.sysctl_sched_child_runs_first : 0
.sysctl_sched_features : 7279
.sysctl_sched_tunable_scaling : 1 (logaritmic)
cpu#0, 2399.914 MHz
.nr_running : 5
.load : 4096
.nr_switches : 39883336
.nr_load_updates : 7052767
.nr_uninterruptible : 0
.next_balance : 4302.402227
.curr->pid : 13017
.clock : 74536255.672926
.cpu_load[0] : 4096
.cpu_load[1] : 3920
.cpu_load[2] : 3640
.cpu_load[3] : 3277
.cpu_load[4] : 3200
.yld_count : 18073
.sched_switch : 0
.sched_count : 40031217
.sched_goidle : 13926017
.avg_idle : 52704
.ttwu_count : 22747849
.ttwu_local : 15278475
.bkl_count : 0
cfs_rq[0]:
.exec_clock : 2466843.273456
.MIN_vruntime : 2206323.435242
.min_vruntime : 2206335.435242
.max_vruntime : 2206335.435242
.spread : 12.000000
.spread0 : 0.000000
.nr_spread_over : 8
.nr_running : 4
.load : 4096
rt_rq[0]:
.rt_nr_running : 1
.rt_throttled : 0
.rt_time : 313.816652
.rt_runtime : 1000.000000
runnable tasks:
task PID tree-key switches prio exec-runtime sum-exec sum-sleep
----------------------------------------------------------------------------------------------------------
kworker/0:1 383 2206323.435242 90851 120 2206323.435242 630.254188 1022915.774743
cached 3557 2206323.435242 8484 120 2206323.435242 359.617311 892924.619510
foed 12945 2206332.512860 943834 120 2206332.512860 144438.272059 267553.706072
foed 12958 2206335.435242 1270174 120 2206335.435242 400509.484940 1810.851261
R foed 13017 11.987540 1442966 98 11.987540 8719332.628415 0.000000
cpu#1, 2399.914 MHz
.nr_running : 2
.load : 1024
.nr_switches : 39295834
.nr_load_updates : 7250607
.nr_uninterruptible : 0
.next_balance : 4302.402229
.curr->pid : 13018
.clock : 74536255.621246
.cpu_load[0] : 1024
.cpu_load[1] : 704
.cpu_load[2] : 715
.cpu_load[3] : 772
.cpu_load[4] : 803
.yld_count : 4805
.sched_switch : 0
.sched_count : 39343242
.sched_goidle : 13050045
.avg_idle : 11993
.ttwu_count : 23168271
.ttwu_local : 13521893
.bkl_count : 0
cfs_rq[1]:
.exec_clock : 2361704.620750
.MIN_vruntime : 2161323.412213
.min_vruntime : 2161335.412213
.max_vruntime : 2161323.412213
.spread : 0.000000
.spread0 : -45000.023029
.nr_spread_over : 15
.nr_running : 1
.load : 1024
rt_rq[1]:
.rt_nr_running : 1
.rt_throttled : 0
.rt_time : 310.711972
.rt_runtime : 1000.000000
runnable tasks:
task PID tree-key switches prio exec-runtime sum-exec sum-sleep
----------------------------------------------------------------------------------------------------------
kworker/1:0 16865 2161323.412213 205898 120 2161323.412213 1475.571852 2178915.923892
R foed 13018 0.000000 1513251 98 0.000000 8718117.033318 0.000000
cpu#2, 2399.914 MHz
.nr_running : 7
.load : 6144
.nr_switches : 32619375
.nr_load_updates : 6974523
.nr_uninterruptible : 0
.next_balance : 4302.402227
.curr->pid : 13019
.clock : 74536255.622175
.cpu_load[0] : 6144
.cpu_load[1] : 4028
.cpu_load[2] : 3123
.cpu_load[3] : 2918
.cpu_load[4] : 3222
.yld_count : 19932
.sched_switch : 0
.sched_count : 32671459
.sched_goidle : 11605598
.avg_idle : 22751
.ttwu_count : 17877937
.ttwu_local : 9273723
.bkl_count : 0
cfs_rq[2]:
.exec_clock : 3720690.576548
.MIN_vruntime : 3614409.938176
.min_vruntime : 3614421.938176
.max_vruntime : 3614421.938176
.spread : 12.000000
.spread0 : 1408086.502934
.nr_spread_over : 16
.nr_running : 6
.load : 6144
rt_rq[2]:
.rt_nr_running : 1
.rt_throttled : 0
.rt_time : 310.712871
.rt_runtime : 1000.000000
runnable tasks:
task PID tree-key switches prio exec-runtime sum-exec sum-sleep
----------------------------------------------------------------------------------------------------------
kworker/2:2 20173 3614409.938176 208526 120 3614409.938176 916.740861 3377822.296382
cached 3523 3614409.938176 8671 120 3614409.938176 367.915787 895066.048702
platform_framew 12925 3614409.938176 746 120 3614409.938176 34.873797 502978.351192
platform_framew 12970 3614409.938176 261 120 3614409.938176 23.514396 502273.256274
platform_framew 12973 3614409.938176 386 120 3614409.938176 49.856866 4467784.168141
foed 12963 3614421.938176 970070 120 3614421.938176 141110.106450 265295.003109
R foed 13019 11.994146 1513933 98 11.994146 8717946.774317 0.000000
cpu#3, 2399.914 MHz
.nr_running : 1
.load : 1024
.nr_switches : 31701614
.nr_load_updates : 490195
.nr_uninterruptible : 0
.next_balance : 4302.402237
.curr->pid : 13323
.clock : 74536255.759098
.cpu_load[0] : 0
.cpu_load[1] : 0
.cpu_load[2] : 0
.cpu_load[3] : 0
.cpu_load[4] : 0
.yld_count : 5260
.sched_switch : 0
.sched_count : 31774617
.sched_goidle : 15248386
.avg_idle : 876767
.ttwu_count : 16368425
.ttwu_local : 15133414
.bkl_count : 0
cfs_rq[3]:
.exec_clock : 383648.882578
.MIN_vruntime : 0.000001
.min_vruntime : 425927.540867
.max_vruntime : 0.000001
.spread : 0.000000
.spread0 : -1780407.894375
.nr_spread_over : 10
.nr_running : 1
.load : 1024
rt_rq[3]:
.rt_nr_running : 0
.rt_throttled : 0
.rt_time : 0.000000
.rt_runtime : 426.841166
runnable tasks:
task PID tree-key switches prio exec-runtime sum-exec sum-sleep
----------------------------------------------------------------------------------------------------------
R cat 13323 425927.540867 1 120 425927.540867 0.403268 0.017381
cpu#4, 2399.914 MHz
.nr_running : 2
.load : 1024
.nr_switches : 24587614
.nr_load_updates : 3523749
.nr_uninterruptible : 0
.next_balance : 4302.402228
.curr->pid : 13020
.clock : 74536255.573983
.cpu_load[0] : 1024
.cpu_load[1] : 1205
.cpu_load[2] : 2849
.cpu_load[3] : 3662
.cpu_load[4] : 3236
.yld_count : 10029
.sched_switch : 0
.sched_count : 24626870
.sched_goidle : 7628233
.avg_idle : 24270
.ttwu_count : 13803231
.ttwu_local : 5632803
.bkl_count : 0
cfs_rq[4]:
.exec_clock : 4406898.984437
.MIN_vruntime : 4328049.217100
.min_vruntime : 4328061.217100
.max_vruntime : 4328049.217100
.spread : 0.000000
.spread0 : 2121725.781858
.nr_spread_over : 14
.nr_running : 1
.load : 1024
rt_rq[4]:
.rt_nr_running : 1
.rt_throttled : 0
.rt_time : 313.719716
.rt_runtime : 1000.000000
runnable tasks:
task PID tree-key switches prio exec-runtime sum-exec sum-sleep
----------------------------------------------------------------------------------------------------------
kworker/4:1 128 4328049.217100 2764041 120 4328049.217100 17250.589367 65846280.840596
R foed 13020 0.000000 1525867 98 0.000000 8717147.634868 0.000000
cpu#5, 2399.914 MHz
.nr_running : 0
.load : 0
.nr_switches : 17215627
.nr_load_updates : 499205
.nr_uninterruptible : 0
.next_balance : 4302.402250
.curr->pid : 0
.clock : 74536255.463803
.cpu_load[0] : 0
.cpu_load[1] : 0
.cpu_load[2] : 0
.cpu_load[3] : 0
.cpu_load[4] : 0
.yld_count : 23448
.sched_switch : 0
.sched_count : 17309229
.sched_goidle : 8476287
.avg_idle : 1000000
.ttwu_count : 8684208
.ttwu_local : 8325745
.bkl_count : 0
cfs_rq[5]:
.exec_clock : 252310.952089
.MIN_vruntime : 0.000001
.min_vruntime : 310744.360911
.max_vruntime : 0.000001
.spread : 0.000000
.spread0 : -1895591.074331
.nr_spread_over : 15
.nr_running : 0
.load : 0
rt_rq[5]:
.rt_nr_running : 0
.rt_throttled : 0
.rt_time : 0.000000
.rt_runtime : 619.569877
runnable tasks:
task PID tree-key switches prio exec-runtime sum-exec sum-sleep
----------------------------------------------------------------------------------------------------------
cpu#6, 2399.914 MHz
.nr_running : 8
.load : 7168
.nr_switches : 28188442
.nr_load_updates : 6032353
.nr_uninterruptible : 0
.next_balance : 4302.402228
.curr->pid : 13021
.clock : 74536255.647278
.cpu_load[0] : 1024
.cpu_load[1] : 3515
.cpu_load[2] : 5635
.cpu_load[3] : 6845
.cpu_load[4] : 7216
.yld_count : 1340
.sched_switch : 0
.sched_count : 28257963
.sched_goidle : 10266294
.avg_idle : 1000000
.ttwu_count : 14804463
.ttwu_local : 6925645
.bkl_count : 0
cfs_rq[6]:
.exec_clock : 6243978.161786
.MIN_vruntime : 6060781.690398
.min_vruntime : 6060793.690398
.max_vruntime : 6060793.690398
.spread : 12.000000
.spread0 : 3854458.255156
.nr_spread_over : 11
.nr_running : 7
.load : 7168
rt_rq[6]:
.rt_nr_running : 1
.rt_throttled : 0
.rt_time : 303.808177
.rt_runtime : 1000.000000
runnable tasks:
task PID tree-key switches prio exec-runtime sum-exec sum-sleep
----------------------------------------------------------------------------------------------------------
kworker/6:1 131 6060781.690398 4888224 120 6060781.690398 34681.808775 65823263.469743
cached 3540 6060781.690398 8482 120 6060781.690398 362.743190 893058.176086
platform_framew 12921 6060781.690398 735 120 6060781.690398 28.996429 503271.767866
platform_framew 12924 6060781.690398 726 120 6060781.690398 45.632586 498397.079571
platform_framew 12981 6060781.690398 181 120 6060781.690398 43.526042 1420979.434967
foed 12941 6060793.690398 927482 120 6060793.690398 136625.959472 274866.207212
foed 12946 6060793.690398 950397 120 6060793.690398 144306.743281 267202.667578
R foed 13021 0.000000 981995 98 0.000000 8610939.930181 0.000000
cpu#7, 2399.914 MHz
.nr_running : 5
.load : 4096
.nr_switches : 18715763
.nr_load_updates : 2692563
.nr_uninterruptible : 0
.next_balance : 4302.402231
.curr->pid : 13022
.clock : 74536255.611440
.cpu_load[0] : 4096
.cpu_load[1] : 3600
.cpu_load[2] : 3655
.cpu_load[3] : 3925
.cpu_load[4] : 4129
.yld_count : 13959
.sched_switch : 0
.sched_count : 18798936
.sched_goidle : 5637607
.avg_idle : 120628
.ttwu_count : 10115603
.ttwu_local : 3649152
.bkl_count : 0
cfs_rq[7]:
.exec_clock : 6285443.979637
.MIN_vruntime : 6139617.302210
.min_vruntime : 6139629.302210
.max_vruntime : 6139629.302210
.spread : 12.000000
.spread0 : 3933293.866968
.nr_spread_over : 20
.nr_running : 4
.load : 4096
rt_rq[7]:
.rt_nr_running : 1
.rt_throttled : 0
.rt_time : 303.772376
.rt_runtime : 1000.000000
runnable tasks:
task PID tree-key switches prio exec-runtime sum-exec sum-sleep
----------------------------------------------------------------------------------------------------------
kworker/7:2 27765 6139617.302210 1143525 120 6139617.302210 6950.288439 48890195.825934
platform_framew 12975 6139617.302210 580 120 6139617.302210 72.128767 685858.287799
platform_framew 12980 6139617.302210 659 120 6139617.302210 86.420613 3248978.940515
foed 12951 6139629.302210 1001088 120 6139629.302210 421981.020977 2854.121475
R foed 13022 0.000000 979598 98 0.000000 8606865.922826 0.000000
cpu#8, 2399.914 MHz
.nr_running : 13
.load : 12288
.nr_switches : 17001086
.nr_load_updates : 3437193
.nr_uninterruptible : 0
.next_balance : 4302.402231
.curr->pid : 13023
.clock : 74536255.611638
.cpu_load[0] : 12288
.cpu_load[1] : 9499
.cpu_load[2] : 7086
.cpu_load[3] : 6077
.cpu_load[4] : 6013
.yld_count : 1178
.sched_switch : 0
.sched_count : 17065402
.sched_goidle : 5568562
.avg_idle : 87340
.ttwu_count : 9001996
.ttwu_local : 3684011
.bkl_count : 0
cfs_rq[8]:
.exec_clock : 2819981.630138
.MIN_vruntime : 2711016.023437
.min_vruntime : 2711028.023437
.max_vruntime : 2711028.023437
.spread : 12.000000
.spread0 : 504692.588195
.nr_spread_over : 18
.nr_running : 12
.load : 12288
rt_rq[8]:
.rt_nr_running : 1
.rt_throttled : 0
.rt_time : 303.772594
.rt_runtime : 1000.000000
runnable tasks:
task PID tree-key switches prio exec-runtime sum-exec sum-sleep
----------------------------------------------------------------------------------------------------------
kworker/8:0 38 2711016.023437 2161482 120 2711016.023437 17128.438196 65876960.703471
cached 3513 2711016.023437 8554 120 2711016.023437 375.333633 893052.303344
cached 3520 2711016.023437 8391 120 2711016.023437 356.441381 895105.701728
cached 3537 2711016.023437 8415 120 2711016.023437 368.402506 895082.083000
cached 3554 2711016.023437 8404 120 2711016.023437 348.522288 895083.848318
platform_framew 12923 2711016.023437 723 120 2711016.023437 38.378030 503049.470168
platform_framew 12965 2711016.023437 660 120 2711016.023437 91.147462 2639633.085069
platform_framew 12982 2711016.023437 651 120 2711016.023437 83.038954 1116267.626024
foed 12944 2711028.023437 977593 120 2711028.023437 141909.390361 264208.759350
foed 12948 2711028.023437 971768 120 2711028.023437 140196.810138 265478.790073
foed 12949 2711028.023437 957474 120 2711028.023437 137974.082718 268126.425971
foed 12954 2711028.023437 1146580 120 2711028.023437 413458.194091 8954.331871
R foed 13023 0.000000 901252 98 0.000000 8612320.080473 0.000000
cpu#9, 2399.914 MHz
.nr_running : 0
.load : 0
.nr_switches : 9283463
.nr_load_updates : 758619
.nr_uninterruptible : 0
.next_balance : 4302.402250
.curr->pid : 0
.clock : 74536255.435346
.cpu_load[0] : 0
.cpu_load[1] : 0
.cpu_load[2] : 3
.cpu_load[3] : 17
.cpu_load[4] : 22
.yld_count : 1977
.sched_switch : 0
.sched_count : 9306858
.sched_goidle : 4583103
.avg_idle : 1000000
.ttwu_count : 4681256
.ttwu_local : 4445158
.bkl_count : 0
cfs_rq[9]:
.exec_clock : 152571.838856
.MIN_vruntime : 0.000001
.min_vruntime : 151726.127282
.max_vruntime : 0.000001
.spread : 0.000000
.spread0 : -2054609.307960
.nr_spread_over : 22
.nr_running : 0
.load : 0
rt_rq[9]:
.rt_nr_running : 0
.rt_throttled : 0
.rt_time : 0.000000
.rt_runtime : 893.172290
runnable tasks:
task PID tree-key switches prio exec-runtime sum-exec sum-sleep
----------------------------------------------------------------------------------------------------------
cpu#10, 2399.914 MHz
.nr_running : 8
.load : 7168
.nr_switches : 29822463
.nr_load_updates : 7240278
.nr_uninterruptible : 0
.next_balance : 4302.402227
.curr->pid : 13024
.clock : 74536255.657050
.cpu_load[0] : 7168
.cpu_load[1] : 6228
.cpu_load[2] : 6105
.cpu_load[3] : 6466
.cpu_load[4] : 6951
.yld_count : 4093
.sched_switch : 0
.sched_count : 29874767
.sched_goidle : 9176748
.avg_idle : 106752
.ttwu_count : 17988938
.ttwu_local : 12616126
.bkl_count : 0
cfs_rq[10]:
.exec_clock : 5539813.139578
.MIN_vruntime : 5430918.171471
.min_vruntime : 5430930.171471
.max_vruntime : 5430918.171471
.spread : 0.000000
.spread0 : 3224594.736229
.nr_spread_over : 12
.nr_running : 7
.load : 7168
rt_rq[10]:
.rt_nr_running : 1
.rt_throttled : 0
.rt_time : 303.818070
.rt_runtime : 1000.000000
runnable tasks:
task PID tree-key switches prio exec-runtime sum-exec sum-sleep
----------------------------------------------------------------------------------------------------------
kworker/10:1 134 5430918.171471 6290838 120 5430918.171471 25512.500134 65891763.388788
cached 3534 5430918.171471 8473 120 5430918.171471 361.097477 893052.026698
cached 3546 5430918.171471 8659 120 5430918.171471 373.658486 893020.219783
cached 3549 5430918.171471 8332 120 5430918.171471 353.567536 893074.358645
platform_framew 12910 5430918.171471 723 120 5430918.171471 40.951675 503266.019634
platform_framew 12913 5430918.171471 736 120 5430918.171471 42.456908 500705.427582
platform_framew 12915 5430918.171471 745 120 5430918.171471 31.962511 502893.946647
R foed 13024 0.000000 882941 98 0.000000 8612119.833337 0.000000
cpu#11, 2399.914 MHz
.nr_running : 0
.load : 0
.nr_switches : 1452860
.nr_load_updates : 432865
.nr_uninterruptible : 0
.next_balance : 4302.402250
.curr->pid : 0
.clock : 74536251.387981
.cpu_load[0] : 0
.cpu_load[1] : 0
.cpu_load[2] : 2
.cpu_load[3] : 14
.cpu_load[4] : 20
.yld_count : 1329
.sched_switch : 0
.sched_count : 1470959
.sched_goidle : 639998
.avg_idle : 1000000
.ttwu_count : 761118
.ttwu_local : 606034
.bkl_count : 0
cfs_rq[11]:
.exec_clock : 157171.251946
.MIN_vruntime : 0.000001
.min_vruntime : 296225.369175
.max_vruntime : 0.000001
.spread : 0.000000
.spread0 : -1910110.066067
.nr_spread_over : 35
.nr_running : 0
.load : 0
rt_rq[11]:
.rt_nr_running : 0
.rt_throttled : 0
.rt_time : 0.000000
.rt_runtime : 910.416667
runnable tasks:
task PID tree-key switches prio exec-runtime sum-exec sum-sleep
----------------------------------------------------------------------------------------------------------
cpu#12, 2399.914 MHz
.nr_running : 3
.load : 2048
.nr_switches : 32037008
.nr_load_updates : 7177780
.nr_uninterruptible : 0
.next_balance : 4302.402230
.curr->pid : 13025
.clock : 74536255.590797
.cpu_load[0] : 2048
.cpu_load[1] : 1568
.cpu_load[2] : 1991
.cpu_load[3] : 2530
.cpu_load[4] : 2887
.yld_count : 29302
.sched_switch : 0
.sched_count : 32131066
.sched_goidle : 12328808
.avg_idle : 18387
.ttwu_count : 16879327
.ttwu_local : 8541706
.bkl_count : 0
cfs_rq[12]:
.exec_clock : 2587657.205883
.MIN_vruntime : 2482173.114533
.min_vruntime : 2482185.114533
.max_vruntime : 2482173.114533
.spread : 0.000000
.spread0 : 275849.679291
.nr_spread_over : 24
.nr_running : 2
.load : 2048
rt_rq[12]:
.rt_nr_running : 1
.rt_throttled : 0
.rt_time : 303.751706
.rt_runtime : 1000.000000
runnable tasks:
task PID tree-key switches prio exec-runtime sum-exec sum-sleep
----------------------------------------------------------------------------------------------------------
kworker/12:1 3707 2482173.114533 199694 120 2482173.114533 1073.267998 2659353.642532
platform_framew 12912 2482173.114533 742 120 2482173.114533 36.277028 501917.431038
R foed 13025 11.994146 1502243 98 11.994146 8719819.372900 0.000000
cpu#13, 2399.914 MHz
.nr_running : 2
.load : 1024
.nr_switches : 36423359
.nr_load_updates : 7285927
.nr_uninterruptible : 0
.next_balance : 4302.402233
.curr->pid : 13026
.clock : 74536255.587585
.cpu_load[0] : 1024
.cpu_load[1] : 704
.cpu_load[2] : 700
.cpu_load[3] : 739
.cpu_load[4] : 764
.yld_count : 1865
.sched_switch : 0
.sched_count : 36459602
.sched_goidle : 12113278
.avg_idle : 14225
.ttwu_count : 21615621
.ttwu_local : 13576999
.bkl_count : 0
cfs_rq[13]:
.exec_clock : 1447347.913624
.MIN_vruntime : 2904584.457182
.min_vruntime : 2904596.457182
.max_vruntime : 2904584.457182
.spread : 0.000000
.spread0 : 698261.021940
.nr_spread_over : 67
.nr_running : 1
.load : 1024
rt_rq[13]:
.rt_nr_running : 1
.rt_throttled : 0
.rt_time : 303.748532
.rt_runtime : 1000.000000
runnable tasks:
task PID tree-key switches prio exec-runtime sum-exec sum-sleep
----------------------------------------------------------------------------------------------------------
kworker/13:1 31079 2904584.457182 65023 120 2904584.457182 454.957322 1047241.715489
R foed 13026 0.000000 1506668 98 0.000000 8718537.726537 0.000000
cpu#14, 2399.914 MHz
.nr_running : 3
.load : 2048
.nr_switches : 27050045
.nr_load_updates : 5557898
.nr_uninterruptible : 0
.next_balance : 4302.402236
.curr->pid : 13027
.clock : 74536255.546754
.cpu_load[0] : 2048
.cpu_load[1] : 1374
.cpu_load[2] : 1119
.cpu_load[3] : 1239
.cpu_load[4] : 2009
.yld_count : 7320
.sched_switch : 0
.sched_count : 27080872
.sched_goidle : 10069090
.avg_idle : 14722
.ttwu_count : 14440627
.ttwu_local : 6635714
.bkl_count : 0
cfs_rq[14]:
.exec_clock : 1585829.628624
.MIN_vruntime : 1750562.885191
.min_vruntime : 1750574.885191
.max_vruntime : 1750562.885191
.spread : 0.000000
.spread0 : -455760.550051
.nr_spread_over : 18
.nr_running : 2
.load : 2048
rt_rq[14]:
.rt_nr_running : 1
.rt_throttled : 0
.rt_time : 303.707750
.rt_runtime : 1000.000000
runnable tasks:
task PID tree-key switches prio exec-runtime sum-exec sum-sleep
----------------------------------------------------------------------------------------------------------
kworker/14:2 31824 1750562.885191 426718 120 1750562.885191 1869.991120 5839696.428973
platform_framew 12918 1750562.885191 726 120 1750562.885191 35.621516 503770.767671
R foed 13027 11.994146 1509404 98 11.994146 8718228.897751 0.000000
cpu#15, 2399.914 MHz
.nr_running : 2
.load : 1024
.nr_switches : 31841934
.nr_load_updates : 7109047
.nr_uninterruptible : 0
.next_balance : 4302.402245
.curr->pid : 13028
.clock : 74536255.621188
.cpu_load[0] : 1024
.cpu_load[1] : 1024
.cpu_load[2] : 1018
.cpu_load[3] : 979
.cpu_load[4] : 928
.yld_count : 37343
.sched_switch : 0
.sched_count : 31897144
.sched_goidle : 12314506
.avg_idle : 46817
.ttwu_count : 16790804
.ttwu_local : 8357023
.bkl_count : 0
cfs_rq[15]:
.exec_clock : 2208535.410370
.MIN_vruntime : 2313101.076037
.min_vruntime : 2313113.076037
.max_vruntime : 2313101.076037
.spread : 0.000000
.spread0 : 106777.640795
.nr_spread_over : 42
.nr_running : 1
.load : 1024
rt_rq[15]:
.rt_nr_running : 1
.rt_throttled : 0
.rt_time : 303.782333
.rt_runtime : 1000.000000
runnable tasks:
task PID tree-key switches prio exec-runtime sum-exec sum-sleep
----------------------------------------------------------------------------------------------------------
kworker/15:0 27448 2313101.076037 863335 120 2313101.076037 3071.426141 11263146.378757
R foed 13028 0.000000 1690983 98 0.000000 8711031.072813 0.000000
cpu#16, 2399.914 MHz
.nr_running : 8
.load : 7168
.nr_switches : 32587967
.nr_load_updates : 6791612
.nr_uninterruptible : 0
.next_balance : 4302.402230
.curr->pid : 13029
.clock : 74536255.580338
.cpu_load[0] : 7168
.cpu_load[1] : 4726
.cpu_load[2] : 3878
.cpu_load[3] : 3190
.cpu_load[4] : 2437
.yld_count : 2341
.sched_switch : 0
.sched_count : 32607422
.sched_goidle : 11006538
.avg_idle : 52170
.ttwu_count : 18997650
.ttwu_local : 11316632
.bkl_count : 0
cfs_rq[16]:
.exec_clock : 2558114.709416
.MIN_vruntime : 2554884.497493
.min_vruntime : 2554896.497493
.max_vruntime : 2554896.497493
.spread : 12.000000
.spread0 : 348561.062251
.nr_spread_over : 24
.nr_running : 7
.load : 7168
rt_rq[16]:
.rt_nr_running : 1
.rt_throttled : 0
.rt_time : 303.741366
.rt_runtime : 1000.000000
runnable tasks:
task PID tree-key switches prio exec-runtime sum-exec sum-sleep
----------------------------------------------------------------------------------------------------------
kworker/16:1 24214 2554884.497493 501760 120 2554884.497493 2178.084028 9699892.912656
platform_framew 12926 2554884.497493 829 120 2554884.497493 66.153664 503309.154052
platform_framew 12927 2554884.497493 718 120 2554884.497493 33.572440 502690.063147
platform_framew 12968 2554884.497493 274 120 2554884.497493 37.356782 2030388.968040
foed 12942 2554896.497493 979510 120 2554896.497493 409562.095698 17017.286712
foed 12956 2554896.497493 934478 120 2554896.497493 139790.133886 272221.539700
foed 12957 2554896.497493 945745 120 2554896.497493 138788.260367 268979.894954
R foed 13029 0.000000 1522118 98 0.000000 8716381.776204 0.000000
cpu#17, 2399.914 MHz
.nr_running : 2
.load : 1024
.nr_switches : 32672863
.nr_load_updates : 7064177
.nr_uninterruptible : 0
.next_balance : 4302.402236
.curr->pid : 13030
.clock : 74536255.594687
.cpu_load[0] : 1024
.cpu_load[1] : 956
.cpu_load[2] : 874
.cpu_load[3] : 833
.cpu_load[4] : 814
.yld_count : 19673
.sched_switch : 0
.sched_count : 32708912
.sched_goidle : 12592826
.avg_idle : 906791
.ttwu_count : 17259124
.ttwu_local : 8713903
.bkl_count : 0
cfs_rq[17]:
.exec_clock : 1890924.891341
.MIN_vruntime : 1903722.656417
.min_vruntime : 1903734.656417
.max_vruntime : 1903722.656417
.spread : 0.000000
.spread0 : -302600.778825
.nr_spread_over : 31
.nr_running : 1
.load : 1024
rt_rq[17]:
.rt_nr_running : 1
.rt_throttled : 0
.rt_time : 303.755792
.rt_runtime : 1000.000000
runnable tasks:
task PID tree-key switches prio exec-runtime sum-exec sum-sleep
----------------------------------------------------------------------------------------------------------
kworker/17:2 2573 1903722.656417 6296941 120 1903722.656417 29625.610494 64262482.211150
R foed 13030 0.000000 1811700 98 0.000000 8706010.530234 0.000000
cpu#18, 2399.914 MHz
.nr_running : 7
.load : 6144
.nr_switches : 28229801
.nr_load_updates : 6817499
.nr_uninterruptible : 0
.next_balance : 4302.402230
.curr->pid : 13031
.clock : 74536255.553771
.cpu_load[0] : 12288
.cpu_load[1] : 10717
.cpu_load[2] : 9685
.cpu_load[3] : 8322
.cpu_load[4] : 6966
.yld_count : 650
.sched_switch : 0
.sched_count : 28280578
.sched_goidle : 9421464
.avg_idle : 87780
.ttwu_count : 16361621
.ttwu_local : 10815250
.bkl_count : 0
cfs_rq[18]:
.exec_clock : 3026168.773504
.MIN_vruntime : 2920152.345091
.min_vruntime : 2920164.345091
.max_vruntime : 2920164.345091
.spread : 12.000000
.spread0 : 713828.909849
.nr_spread_over : 16
.nr_running : 6
.load : 6144
rt_rq[18]:
.rt_nr_running : 1
.rt_throttled : 0
.rt_time : 303.714651
.rt_runtime : 1000.000000
runnable tasks:
task PID tree-key switches prio exec-runtime sum-exec sum-sleep
----------------------------------------------------------------------------------------------------------
kworker/18:1 214 2920152.345091 5864241 120 2920152.345091 35017.271013 65779986.433638
cached 3509 2920152.345091 8532 120 2920152.345091 365.850863 895106.966978
cached 3543 2920152.345091 8447 120 2920152.345091 350.872831 893070.227600
cached 3563 2920152.345091 8301 120 2920152.345091 347.585095 894973.896491
platform_framew 12960 2920152.345091 336 120 2920152.345091 45.790407 2334998.808147
foed 12955 2920164.345091 932771 120 2920164.345091 140257.352261 270378.647030
R foed 13031 0.000000 1002452 98 0.000000 8609456.162360 0.000000
cpu#19, 2399.914 MHz
.nr_running : 6
.load : 5120
.nr_switches : 23981001
.nr_load_updates : 6773531
.nr_uninterruptible : 0
.next_balance : 4302.402227
.curr->pid : 13032
.clock : 74536255.611306
.cpu_load[0] : 5120
.cpu_load[1] : 5104
.cpu_load[2] : 4916
.cpu_load[3] : 4574
.cpu_load[4] : 4383
.yld_count : 2260
.sched_switch : 0
.sched_count : 24030737
.sched_goidle : 9230103
.avg_idle : 1000000
.ttwu_count : 12375482
.ttwu_local : 7195017
.bkl_count : 0
cfs_rq[19]:
.exec_clock : 2597717.978395
.MIN_vruntime : 3953256.059158
.min_vruntime : 3953268.059158
.max_vruntime : 3953256.059158
.spread : 0.000000
.spread0 : 1746932.623916
.nr_spread_over : 90
.nr_running : 5
.load : 5120
rt_rq[19]:
.rt_nr_running : 1
.rt_throttled : 0
.rt_time : 303.772211
.rt_runtime : 1000.000000
runnable tasks:
task PID tree-key switches prio exec-runtime sum-exec sum-sleep
----------------------------------------------------------------------------------------------------------
kworker/19:2 24820 3953256.059158 568997 120 3953256.059158 2958.921773 11390828.944726
platform_framew 12914 3953256.059158 1529 120 3953256.059158 55.737733 503393.280394
platform_framew 12967 3953256.059158 647 120 3953256.059158 70.369425 2944281.771030
platform_framew 12974 3953256.059158 310 120 3953256.059158 42.439027 3858436.468462
platform_framew 12978 3953256.059158 814 120 3953256.059158 108.423669 1725547.199045
R foed 13032 -4.844494 881583 98 -4.844494 8611945.833131 0.000000
cpu#20, 2399.914 MHz
.nr_running : 7
.load : 6144
.nr_switches : 17951996
.nr_load_updates : 3653460
.nr_uninterruptible : 0
.next_balance : 4302.402236
.curr->pid : 13033
.clock : 74536255.652330
.cpu_load[0] : 6144
.cpu_load[1] : 5376
.cpu_load[2] : 5238
.cpu_load[3] : 5225
.cpu_load[4] : 5229
.yld_count : 572
.sched_switch : 0
.sched_count : 17994231
.sched_goidle : 5646666
.avg_idle : 692587
.ttwu_count : 10275285
.ttwu_local : 5547851
.bkl_count : 0
cfs_rq[20]:
.exec_clock : 2058398.044053
.MIN_vruntime : 2210093.789994
.min_vruntime : 2210105.789994
.max_vruntime : 2210105.789994
.spread : 12.000000
.spread0 : 3770.354752
.nr_spread_over : 73
.nr_running : 6
.load : 6144
rt_rq[20]:
.rt_nr_running : 1
.rt_throttled : 0
.rt_time : 303.813300
.rt_runtime : 1000.000000
runnable tasks:
task PID tree-key switches prio exec-runtime sum-exec sum-sleep
----------------------------------------------------------------------------------------------------------
kworker/20:0 11185 2210093.789994 677081 120 2210093.789994 2797.162747 8624697.066013
cached 3516 2210093.789994 8279 120 2210093.789994 353.477428 894440.182246
platform_framew 12961 2210093.789994 525 120 2210093.789994 71.747226 4772393.558107
platform_framew 12962 2210093.789994 427 120 2210093.789994 51.618128 506310.815525
foed 12952 2210105.789994 961827 120 2210105.789994 147044.084221 261985.999035
foed 12953 2210105.789994 966843 120 2210105.789994 140925.906703 265161.087481
R foed 13033 0.000000 887348 98 0.000000 8610362.083278 0.000000
cpu#21, 2399.914 MHz
.nr_running : 5
.load : 4096
.nr_switches : 17247708
.nr_load_updates : 3408563
.nr_uninterruptible : 0
.next_balance : 4302.402230
.curr->pid : 13034
.clock : 74536255.655857
.cpu_load[0] : 4096
.cpu_load[1] : 4096
.cpu_load[2] : 4085
.cpu_load[3] : 3997
.cpu_load[4] : 3844
.yld_count : 6010
.sched_switch : 0
.sched_count : 17294799
.sched_goidle : 5413323
.avg_idle : 239133
.ttwu_count : 9827028
.ttwu_local : 5137108
.bkl_count : 0
cfs_rq[21]:
.exec_clock : 1225757.623080
.MIN_vruntime : 1239695.782333
.min_vruntime : 1239707.782333
.max_vruntime : 1239707.782333
.spread : 12.000000
.spread0 : -966627.652909
.nr_spread_over : 19
.nr_running : 4
.load : 4096
rt_rq[21]:
.rt_nr_running : 1
.rt_throttled : 0
.rt_time : 303.817219
.rt_runtime : 1000.000000
runnable tasks:
task PID tree-key switches prio exec-runtime sum-exec sum-sleep
----------------------------------------------------------------------------------------------------------
kworker/21:1 215 1239695.782333 2597881 120 1239695.782333 10394.018472 65896257.911119
cached 3551 1239695.782333 8533 120 1239695.782333 356.248467 893016.785453
platform_framew 12922 1239695.782333 852 120 1239695.782333 60.343717 501299.076685
foed 12943 1239707.782333 970578 120 1239707.782333 132388.465003 272947.166391
R foed 13034 0.000000 918974 98 0.000000 8601292.977659 0.000000
cpu#22, 2399.914 MHz
.nr_running : 8
.load : 7168
.nr_switches : 12195006
.nr_load_updates : 1871480
.nr_uninterruptible : 0
.next_balance : 4302.402228
.curr->pid : 13035
.clock : 74536255.643254
.cpu_load[0] : 7168
.cpu_load[1] : 6406
.cpu_load[2] : 6201
.cpu_load[3] : 6097
.cpu_load[4] : 6104
.yld_count : 697
.sched_switch : 0
.sched_count : 12232154
.sched_goidle : 3596591
.avg_idle : 888068
.ttwu_count : 6529390
.ttwu_local : 1984471
.bkl_count : 0
cfs_rq[22]:
.exec_clock : 2753278.823469
.MIN_vruntime : 2729476.857118
.min_vruntime : 2729488.857118
.max_vruntime : 2729488.857118
.spread : 12.000000
.spread0 : 523153.421876
.nr_spread_over : 11
.nr_running : 7
.load : 7168
rt_rq[22]:
.rt_nr_running : 1
.rt_throttled : 0
.rt_time : 303.804289
.rt_runtime : 1000.000000
runnable tasks:
task PID tree-key switches prio exec-runtime sum-exec sum-sleep
----------------------------------------------------------------------------------------------------------
kworker/22:0 94 2729476.857118 836637 120 2729476.857118 5140.679303 65925556.516564
platform_framew 12964 2729476.857118 849 120 2729476.857118 98.925476 4162985.730874
platform_framew 12966 2729476.857118 848 120 2729476.857118 99.878390 811513.828728
platform_framew 12971 2729476.857118 478 120 2729476.857118 60.505962 3553687.292388
platform_framew 12976 2729476.857118 517 120 2729476.857118 79.847259 502790.772162
foed 12947 2729488.857118 1002363 120 2729488.857118 142086.168741 264341.492277
foed 12977 2729488.857118 932379 120 2729488.857118 144302.252104 267422.355049
R foed 13035 0.000000 879697 98 0.000000 8614144.004489 0.000000
cpu#23, 2399.914 MHz
.nr_running : 1
.load : 1024
.nr_switches : 4184269
.nr_load_updates : 1625043
.nr_uninterruptible : 0
.next_balance : 4302.402236
.curr->pid : 12950
.clock : 74536251.388458
.cpu_load[0] : 1024
.cpu_load[1] : 1024
.cpu_load[2] : 1024
.cpu_load[3] : 1024
.cpu_load[4] : 1024
.yld_count : 447
.sched_switch : 0
.sched_count : 4232003
.sched_goidle : 478155
.avg_idle : 875093
.ttwu_count : 2116348
.ttwu_local : 1672186
.bkl_count : 0
cfs_rq[23]:
.exec_clock : 15632911.824241
.MIN_vruntime : 0.000001
.min_vruntime : 15563775.543658
.max_vruntime : 0.000001
.spread : 0.000000
.spread0 : 13357440.108416
.nr_spread_over : 0
.nr_running : 1
.load : 1024
rt_rq[23]:
.rt_nr_running : 0
.rt_throttled : 0
.rt_time : 0.000000
.rt_runtime : 950.000000
runnable tasks:
task PID tree-key switches prio exec-runtime sum-exec sum-sleep
----------------------------------------------------------------------------------------------------------
R foed 12950 15563775.543658 966032 120 15563775.543658 9045998.780625 4950.628670
[-- Attachment #3: schedstat --]
[-- Type: application/octet-stream, Size: 16115 bytes --]
version 15
timestamp 4302402983
cpu0 18073 0 40031369 13926017 22747925 15278476 12479664816092 7706892855290 25958999
domain0 00000000,00001001 5070523 5011821 57596 66627614 1115 496 185 5011636 66178 53013 828 12223602 14033 56 0 53013 7870699 7746854 118690 158180188 5180 1220 2259 7744595 9 1 8 0 0 0 0 0 0 3545209 68463 0
domain1 00000000,0003f03f 1636728 1401162 231231 553727874 4396 2186 43046 1358112 61732 41529 631 122291968 73557 45 7 34725 7865544 7291633 569933 1414110281 4005 1127 439771 6851862 51 1 50 0 0 0 0 0 0 26959831 196130 0
domain2 00000000,00ffffff 633259 624367 8534 10471150 365 63 2427 621914 11836 11733 31 100364 72 0 0 34 7861566 7063317 791995 1210339170 6437 17 173885 6889432 7 0 7 0 0 0 0 0 0 2503067 130890 0
cpu1 4805 0 39343394 13050045 23168347 13521894 12345367206734 8452943693335 26246145
domain0 00000000,00002002 4206833 4154736 51345 56464550 782 475 116 4154620 67796 52375 726 13904035 16437 34 0 52375 7770536 7541807 225470 851652673 3273 1337 76522 7465285 19 0 19 0 0 0 0 0 0 436145 30872 0
domain1 00000000,0003f03f 2714921 2432126 277404 562466921 5496 2866 75610 2356513 65648 44324 629 126773258 77273 50 4 36175 7767277 7276482 484594 1457442477 6250 737 214117 7062365 103 7 96 0 0 0 0 0 0 2694068 203564 0
domain2 00000000,00ffffff 708716 705073 3415 3933376 229 17 570 170131 11508 11491 16 17990 1 0 0 20 7761076 6875785 878902 1282490427 6577 5 171321 6704464 1 0 1 0 0 0 0 0 0 2371768 123288 0
cpu2 19932 0 32671611 11605598 17878013 9273724 13661909845644 7849746488801 21014402
domain0 00000000,00004004 3578552 3545550 32226 37050027 791 481 18 3545532 73372 59691 772 12737295 14750 51 0 59691 7164893 7045182 116923 172251571 2798 1251 3516 7041666 18 3 15 0 0 0 0 0 0 290824 11329 0
domain1 00000000,0003f03f 1325100 1187885 131144 335239472 6183 2286 223870 964008 70510 49586 695 124933428 76118 46 7 35983 7162105 6344407 808934 3611649888 8835 1523 650815 5693592 97 6 91 0 0 0 0 0 0 2217705 160393 0
domain2 00000000,00ffffff 470246 467341 2696 3124899 209 10 152 34416 12563 12546 13 15212 4 0 0 24 7153341 6720882 425748 744632625 6888 8 225409 6495473 0 0 0 0 0 0 0 0 0 2325258 106741 0
cpu3 5260 0 31775546 15248802 16368914 15133782 384639529367 29245804555 16459649
domain0 00000000,00008008 180014 178721 1040 60220638 256 16 21 178701 2097 1963 131 371087 3 0 0 1963 3022552 2677223 344875 1047085062 456 50 1449 2675774 0 0 0 0 0 0 0 0 0 954818 69640 0
domain1 00000000,0003f03f 192163 162137 28857 637389241 1222 337 2824 159211 2398 2371 24 757965 7 0 2 1214 3022098 2360332 659737 3583105500 2052 534 21967 2338365 55 5 50 0 0 0 0 0 0 7643320 32499 0
domain2 00000000,00ffffff 97437 94947 2468 132806435 22 1 112 40085 471 471 0 35308 0 0 0 24 3020069 2509349 510067 3519387874 661 0 115670 2393679 0 0 0 0 0 0 0 0 0 2019138 9254 0
cpu4 10029 0 24627023 7628233 13803307 5632804 14322292786139 7747403467034 16963313
domain0 00000000,00010010 1006283 930105 75246 80654361 952 696 23 930082 82212 67369 894 13577004 15801 40 0 67369 3551325 3425216 123464 142615019 2652 1429 1804 3423412 19 1 18 0 0 0 0 0 0 238084 4608 0
domain1 00000000,0003f03f 1424585 1236450 181948 316535114 6319 2869 23114 1213323 72531 51600 659 125505787 76580 53 15 36492 3548680 3047029 492153 940980180 9574 1659 230347 2816682 118 5 113 0 0 0 0 0 0 2044407 113586 0
domain2 00000000,00ffffff 326600 326305 272 321049 23 0 7 1422 13048 13047 1 787 0 0 0 4 3539182 3060254 471831 752465989 7287 0 169919 2890335 0 0 0 0 0 0 0 0 0 2303090 90993 0
cpu5 23448 0 17310389 8476824 8684826 8326304 253737742959 10642036158 8746007
domain0 00000000,00020020 153919 153266 441 55612985 218 0 16 153250 1241 1115 123 216206 3 0 0 1115 2521893 2296470 225223 961890881 200 11 428 2296042 0 0 0 0 0 0 0 0 0 674910 636 0
domain1 00000000,0003f03f 159644 135128 23304 591894572 1273 434 2358 132702 1770 1748 19 454940 3 0 1 786 2521693 1907152 612287 1233556810 2305 358 18133 1889019 48 4 44 0 0 0 0 0 0 6248899 7509 0
domain2 00000000,00ffffff 80337 80005 327 10949124 5 0 37 3943 425 425 0 0 0 0 0 3 2519439 2205543 313332 3815044845 570 0 78275 2127268 0 0 0 0 0 0 0 0 0 27164 2069 0
cpu6 1340 0 28258335 10266294 14804649 6925732 15679116861160 62801195331258 17926628
domain0 00000000,00040040 1172551 1078867 92846 99877398 845 381 41 1078826 124381 84018 928 115074534 119005 50 0 84018 6266768 6012349 247694 2366913834 6754 1192 40107 5972242 7 1 6 0 0 0 0 0 0 801901 68846 0
domain1 00000000,00fc0fc0 1113699 808676 300708 827051593 4397 1909 40214 768450 117880 47368 1096 1578705564 1102461 103 90 28905 6260043 5859222 392842 677730308 8070 543 127544 5731678 73 7 66 0 0 0 0 0 0 2732236 216106 0
domain2 00000000,00ffffff 716672 702362 13613 19972868 707 103 1106 701220 13986 13886 15 81044 85 0 0 34 6252064 4797715 1444615 708787452 9959 94 636477 4161238 8 0 8 0 0 0 0 0 0 1645490 150159 0
cpu7 13959 0 18799601 5637607 10115937 3649389 15638994974356 65338795535765 13082780
domain0 00000000,00080080 752452 713587 38075 43344112 804 340 53 713534 126296 85953 824 115516972 119501 68 0 85953 2869952 2699490 164896 227743440 5593 1441 2494 2696996 14 2 12 0 0 0 0 0 0 486290 43090 0
domain1 00000000,00fc0fc0 666029 425501 235688 396389254 4925 2021 22714 402771 119030 48247 707 1582947228 1110343 42 17 29397 2864386 2431308 421616 776753602 11567 621 155221 2276087 65 3 62 0 0 0 0 0 0 2457967 280884 0
domain2 00000000,00ffffff 229805 220295 9022 10920378 489 34 359 86248 13635 13614 20 16871 1 0 0 21 2852924 2253181 589484 1614640464 10528 34 135887 2117294 1 0 1 0 0 0 0 0 0 1579592 156235 0
cpu8 1178 0 17065604 5568562 9002097 3684012 12125967090368 65231065797942 11436100
domain0 00000000,00100100 1096336 1072203 22923 27112582 1223 304 59 1072144 103537 63223 650 117845548 122152 42 0 63223 3794731 3563272 227398 916833422 4076 1181 40604 3522668 13 1 12 0 0 0 0 0 0 285076 15274 0
domain1 00000000,00fc0fc0 1066638 771751 289935 686370188 5073 1488 204352 567383 105575 34652 641 1588542382 1121726 48 18 25719 3790670 3302824 474023 785762351 13945 1071 135132 3167692 103 4 99 0 0 0 0 0 0 1817918 171468 0
domain2 00000000,00ffffff 421639 414570 6709 7701299 360 18 319 56272 11526 11513 12 11712 1 0 0 21 3776847 2987365 779193 2195923440 10554 44 302580 2684785 0 0 0 0 0 0 0 0 0 1497204 92290 0
cpu9 1977 0 9307241 4583276 4681464 4445322 319590071615 47470029552 4706417
domain0 00000000,00200200 267782 267158 499 87521762 132 9 54 267104 8544 8469 58 654749 17 0 1 8468 2696161 2444523 251351 1180609586 287 25 1462 2443061 0 0 0 0 0 0 0 0 0 411399 991 0
domain1 00000000,00fc0fc0 319637 289850 28496 1004671415 1308 141 7361 282468 9989 9970 19 355094 0 0 3 1753 2695874 1933465 761264 843827858 1169 283 96851 1836614 18 4 14 0 0 0 0 0 0 3281752 2011 0
domain2 00000000,00ffffff 135453 133322 2079 127344414 52 1 360 40582 1088 1086 2 25687 0 0 0 14 2694729 2138159 554702 3009366077 1879 3 94754 2043405 0 0 0 0 0 0 0 0 0 2230536 5112 0
cpu10 4093 0 29874967 9176748 17989038 12616127 14788993343148 67004748679422 20647315
domain0 00000000,00400400 4557803 4417466 139575 146233030 769 338 37 4417429 120694 80757 570 120366791 124914 24 1 80756 7087539 6943402 140568 203062215 3602 1337 5885 6937517 8 1 7 0 0 0 0 0 0 231926 5061 0
domain1 00000000,00fc0fc0 2782010 2569487 205898 402736448 6731 2070 9417 2560065 116966 45018 755 1591554451 1139899 62 55 25744 7083970 5818677 1250236 3764111507 15202 1015 254930 5563747 84 2 82 0 0 0 0 0 0 2151321 548720 0
domain2 00000000,00ffffff 1073747 1073339 377 418715 31 0 7 2084 13099 13097 2 1587 0 0 0 3 7068913 5502830 1555706 3089667504 10596 0 990577 4512253 0 0 0 0 0 0 0 0 0 1469677 90296 0
cpu11 1329 0 1472100 640357 761742 606645 169198126266 12450224096 819759
domain0 00000000,00800800 118809 116890 1909 19951108 27 23 251 116639 4325 4321 4 20242 0 0 0 4321 587650 583997 3605 25112624 50 6 617 583380 1 0 1 0 0 0 0 0 0 5044 15 0
domain1 00000000,00fc0fc0 123876 105347 17932 1544639574 619 132 1219 104123 4422 4410 11 2138057 1 0 0 416 587602 446448 140138 2475224044 1038 224 2868 443580 20 5 15 0 0 0 0 0 0 71263 742 0
domain2 00000000,00ffffff 63943 63788 152 9155330 3 0 11 2750 250 250 0 0 0 0 0 0 586586 528815 57278 1899075781 504 0 5142 523673 0 0 0 0 0 0 0 0 0 37366 457 0
cpu12 29302 0 32131218 12328808 16879403 8541707 12586204437492 5925702894127 19710204
domain0 00000000,00001001 1510757 1467857 41001 48330275 1921 671 49 1467808 76965 59502 1534 23189883 24505 91 0 59502 7709756 7424120 271799 3361149514 13902 3666 7191 7416929 22 2 20 0 0 0 0 0 0 756485 85950 0
domain1 00000000,0003f03f 4285538 4253914 30697 72805913 940 387 4039 69734 47517 47504 11 12402 2 0 0 80 7695919 6417975 1272429 38131532 5570 152 260923 6157052 7 0 7 0 0 0 0 0 0 2012607 139097 0
domain2 00000000,00ffffff 255826 255777 47 55632 2 0 1 226 10737 10737 0 0 0 0 0 0 7690404 7014329 668955 1110461683 7299 0 289955 6724374 0 0 0 0 0 0 0 0 0 2287197 118603 0
cpu13 1865 0 36459754 12113278 21615697 13577000 11424380646786 6699995855980 24310406
domain0 00000000,00002002 4032146 3953214 77036 88829478 1946 1376 106 3953108 74754 54419 1332 25506046 28040 102 0 54419 7909547 7607949 291815 821049687 9819 4118 42763 7565186 51 3 48 0 0 0 0 0 0 530563 59956 0
domain1 00000000,0003f03f 2332853 2277523 54144 128877036 1195 383 16101 200379 43881 43872 7 8498 2 0 0 69 7899764 7423194 468514 1161459119 8151 204 209469 7213725 3 0 3 0 0 0 0 0 0 2051731 183112 0
domain2 00000000,00ffffff 451568 451525 38 82944 5 0 1 297 10021 10020 1 801 0 0 0 2 7891708 7112159 771926 1228653717 7830 0 183052 6929107 0 0 0 0 0 0 0 0 0 2252237 118016 0
cpu14 7320 0 27081024 10069090 14440703 6635715 11529068227408 6053574584273 16982710
domain0 00000000,00004004 3314684 3163695 149740 158888450 1264 783 121 3163574 71253 52995 1319 24288368 25980 81 3 52992 6054659 5847700 201177 235846397 5800 2157 4435 5843265 16 2 14 0 0 0 0 0 0 307902 25591 0
domain1 00000000,0003f03f 1082518 1009569 71695 139753199 1276 380 19807 111462 46136 46122 11 12065 3 1 0 72 6048877 5344108 695419 1696878038 9456 325 447997 4896111 7 0 7 0 0 0 0 0 0 1945786 144048 0
domain2 00000000,00ffffff 333548 333519 26 31087 3 0 0 149 9946 9946 0 0 0 0 0 0 6039527 5498639 532964 884763600 8163 0 208161 5290478 0 0 0 0 0 0 0 0 0 2220320 100830 0
cpu15 37343 0 31897296 12314506 16790880 8357024 12062378528490 5037920837007 19528481
domain0 00000000,00008008 5334814 5332196 2508 3926657 113 31 12 5332184 24848 24781 50 71713 17 0 0 24781 7760983 7757138 3093 6024652 757 67 35 7757103 3 0 3 0 0 0 0 0 0 32434 2639 0
domain1 00000000,0003f03f 4976905 4969621 7052 14437420 237 55 293 7364 48129 48126 2 2762 1 0 0 5 7760231 4038209 3703307 437339256 18954 86 1083679 2954530 4 0 4 0 0 0 0 0 0 2372791 162396 0
domain2 00000000,00ffffff 130462 130435 25 29338 2 0 0 121 8536 8536 0 0 0 0 0 0 7741516 7102488 630215 2946746570 9041 0 490981 6611507 0 0 0 0 0 0 0 0 0 2446150 105544 0
cpu16 2341 0 32607574 11006538 18997726 11316633 12475960785857 5946647688455 21582184
domain0 00000000,00010010 4051180 3957416 92318 99871643 1473 1117 13 3957403 78055 59083 1101 24595338 27065 73 1 59082 7035351 6781753 247977 1333096199 5638 2661 19639 6762114 29 2 27 0 0 0 0 0 0 248374 13511 0
domain1 00000000,0003f03f 3855891 3800695 53958 86985558 1258 382 6648 72973 48337 48327 8 8433 2 1 1 71 7029730 5157483 1861734 743740151 10613 312 192700 4964783 6 0 6 0 0 0 0 0 0 1889102 158881 0
domain2 00000000,00ffffff 215137 215115 21 24607 1 0 0 41 10669 10667 1 1723 1 0 0 0 7019217 6525177 485497 1317718465 8781 0 339351 6185826 0 0 0 0 0 0 0 0 0 2209240 94922 0
cpu17 19673 0 32709064 12592826 17259200 8713904 11708626914490 4125547259194 20080788
domain0 00000000,00020020 5328481 5327890 512 2335917 79 11 5 5327885 82786 82771 11 17805 4 0 0 82771 8085246 8083020 1605 16363496 622 19 29 8082991 0 0 0 0 0 0 0 0 0 12681 1537 0
domain1 00000000,0003f03f 5117505 5113939 3469 7869707 101 21 357 3785 46354 46354 0 0 0 0 0 0 8084625 3082135 4982643 89355193 20179 31 192498 2889637 2 0 2 0 0 0 0 0 0 2335202 159290 0
domain2 00000000,00ffffff 95215 95193 18 59752 4 0 0 39 13040 13040 0 0 0 0 0 2 8064778 7425395 629213 3178432532 10408 0 478262 6947133 0 0 0 0 0 0 0 0 0 2535390 111784 0
cpu18 650 0 28280788 9421464 16361726 10815251 12437391405513 45132635873844 18809135
domain0 00000000,00040040 4986125 4897144 87589 96552306 1418 614 29 4897115 110521 60439 1785 273484562 269850 77 1 60438 7126722 6856225 255943 352928325 14688 2601 3810 6852415 23 0 23 0 0 0 0 0 0 710306 126322 0
domain1 00000000,00fc0fc0 2824654 2587541 236141 525914661 982 310 20587 372849 48830 48817 11 12175 2 0 3 91 7112168 6343344 758340 1788887951 10637 77 305977 6037367 4 1 3 0 0 0 0 0 0 1660202 203639 0
domain2 00000000,00ffffff 735796 735725 70 77046 1 0 5 139 7348 7348 0 0 0 0 0 1 7101684 5905856 1186160 384665561 9888 0 775053 5130803 0 0 0 0 0 0 0 0 0 1508893 155336 0
cpu19 2260 0 24030937 9230103 12375582 7195018 11933591406078 46426153288653 14751840
domain0 00000000,00080080 4900665 4776819 122347 132709247 1522 1011 88 4776731 107272 57324 1388 275716593 271456 71 0 57324 7229834 6992195 226394 323642073 11333 3239 3480 6988715 22 1 21 0 0 0 0 0 0 574244 80591 0
domain1 00000000,00fc0fc0 4146944 4082298 63438 106529068 1225 325 13471 95991 47783 47769 12 11338 2 0 2 89 7218589 5023746 2181638 721379031 13351 109 1097833 3925913 4 0 4 0 0 0 0 0 0 1817739 251346 0
domain2 00000000,00ffffff 358737 358697 38 41917 2 0 2 101 11698 11698 0 0 0 0 0 0 7205384 5814618 1379920 78287040 11099 0 1027106 4787512 0 0 0 0 0 0 0 0 0 1484077 142860 0
cpu20 572 0 17994439 5646666 10275389 5547852 11344034263968 47096191950476 12308851
domain0 00000000,00100100 2031022 1950676 78927 85297077 1441 448 23 1950653 103014 53351 1122 278297871 276959 73 0 53351 3941636 3742880 190961 281983470 7849 2053 2420 3740460 20 2 18 0 0 0 0 0 0 301703 27913 0
domain1 00000000,00fc0fc0 1477779 1410229 66665 132182355 898 148 48325 101259 45921 45908 11 10988 2 1 1 56 3933841 3215421 703763 1126030537 14814 77 125763 3089658 3 0 3 0 0 0 0 0 0 1639621 176135 0
domain2 00000000,00ffffff 239815 239781 32 32647 2 0 2 64 11701 11701 0 0 0 0 0 0 3919184 3241228 666792 2051671825 11403 0 272954 2968274 0 0 0 0 0 0 0 0 0 1462557 106767 0
cpu21 6010 0 17294995 5413323 9827126 5137109 10451252586389 450715166630 11838144
domain0 00000000,00200200 399785 398504 1134 4480568 148 31 3 398501 72019 71989 23 34633 7 1 0 71989 3791401 3788445 2173 20778820 784 32 39 3788406 2 1 1 0 0 0 0 0 0 13704 990 0
domain1 00000000,00fc0fc0 873709 867797 5729 12227826 188 25 980 8794 48242 48242 0 0 0 0 0 5 3790618 2603301 1164472 2559773724 23716 58 241261 2362040 1 0 1 0 0 0 0 0 0 1698410 105296 0
domain2 00000000,00ffffff 189091 189065 24 28073 2 0 0 56 12770 12769 1 1024 0 0 0 1 3767773 2757294 996875 279195819 13902 0 320918 2436376 0 0 0 0 0 0 0 0 0 1469467 87522 0
cpu22 697 0 12232360 3596591 6529493 1984472 11996999935290 48080541218640 8603366
domain0 00000000,00400400 471298 391691 78352 85173569 1285 572 20 391671 109291 59111 1202 286674185 285192 42 1 59110 2032463 1837127 189026 269295100 6351 2110 3223 1833904 27 0 27 0 0 0 0 0 0 234475 11821 0
domain1 00000000,00fc0fc0 449356 401279 46933 72485236 1162 225 4482 64218 48934 48919 14 15600 1 0 1 90 2026153 1554172 456029 727017660 16114 148 79645 1474527 5 1 4 0 0 0 0 0 0 1505839 130719 0
domain2 00000000,00ffffff 70919 70901 16 22721 2 0 0 26 7525 7525 0 0 0 0 0 0 2010201 1556327 442613 1260804762 11516 0 99703 1456624 0 0 0 0 0 0 0 0 0 1382362 91575 0
cpu23 447 0 4233518 478155 2117106 1672944 15640573217161 168299560669 3712533
domain0 00000000,00800800 60956 60188 700 883689 68 44 0 60188 106518 106481 10 39243 27 0 0 106481 358108 357736 286 486267 87 13 4 357732 1 1 0 0 0 0 0 0 0 5515 9 0
domain1 00000000,00fc0fc0 61223 60302 894 1866580 27 7 31 1318 64212 64212 0 0 0 0 0 3 358022 116166 225622 333965334 17551 21 2958 113208 0 0 0 0 0 0 0 0 0 165546 5012 0
domain2 00000000,00ffffff 2916 2895 18 33398 4 0 0 43 13837 13836 0 512 1 0 0 0 341788 240016 93995 164803699 7989 0 1816 238200 0 0 0 0 0 0 0 0 0 114820 3973 0
^ permalink raw reply [flat|nested] 7+ messages in thread* Re: runnable tasks never making it to a runqueue where they can run?
2012-03-07 2:19 runnable tasks never making it to a runqueue where they can run? Roland Dreier
@ 2012-03-07 16:30 ` Peter Zijlstra
2012-03-07 16:31 ` Peter Zijlstra
1 sibling, 0 replies; 7+ messages in thread
From: Peter Zijlstra @ 2012-03-07 16:30 UTC (permalink / raw)
To: Roland Dreier; +Cc: LKML
On Tue, 2012-03-06 at 18:19 -0800, Roland Dreier wrote:
> Is this expected behavior?
Its not unexpected.. although I had hoped we'd not suck quite as bad.
The 'idea' is that scale_rt_power() will decrease the cpu_power of cpus
that run RT tasks and the rest of the load-balancing will then dtrt.
Clearly its not quite working.
> I can believe that we brought this on ourselves by
> misconfiguring things, but in that case it would be good to know what we could
> do to avoid this problem.
hard-affine everything.. eg. create a system cpuset and only give it
cpu23 (although cpu0 would be more typical) and run only a single task
on every other cpu.
Once you do that, it doesn't really matter if your one task is FIFO or
not, with the exception that system tasks (like ksoftirqd, kworker/n
etc..) can still run when required.
Running FIFO spinners is known to break stuff due to those system tasks
not getting run. Fixing this is somewhat non-trivial :/
Alternatively you can of course play with stuff like commit 4a6184ce
^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: runnable tasks never making it to a runqueue where they can run?
2012-03-07 2:19 runnable tasks never making it to a runqueue where they can run? Roland Dreier
2012-03-07 16:30 ` Peter Zijlstra
@ 2012-03-07 16:31 ` Peter Zijlstra
2012-03-07 18:03 ` Roland Dreier
1 sibling, 1 reply; 7+ messages in thread
From: Peter Zijlstra @ 2012-03-07 16:31 UTC (permalink / raw)
To: Roland Dreier; +Cc: LKML
On Tue, 2012-03-06 at 18:19 -0800, Roland Dreier wrote:
> 6,7,8,10,18,19,22 but never onto 23
> (Interestingly, those 8 CPUs seem to form a domain)
That would be 'interesting'... I have a similar wsm box and you should
have a domain setup like:
0-5,12-17 (node 0)
6-11,18-23 (node 1)
Where the first set (0-5 & 6-11) is smt0 and the second set (12-17 &
18-23) smt1.
Can you boot with sched_debug and send me both /proc/cpuinfo and dmesg?
^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: runnable tasks never making it to a runqueue where they can run?
2012-03-07 16:31 ` Peter Zijlstra
@ 2012-03-07 18:03 ` Roland Dreier
2012-03-08 10:28 ` Peter Zijlstra
0 siblings, 1 reply; 7+ messages in thread
From: Roland Dreier @ 2012-03-07 18:03 UTC (permalink / raw)
To: Peter Zijlstra; +Cc: LKML
On Wed, Mar 7, 2012 at 8:31 AM, Peter Zijlstra <a.p.zijlstra@chello.nl> wrote:
>> 6,7,8,10,18,19,22 but never onto 23
>
>> (Interestingly, those 8 CPUs seem to form a domain)
>
> That would be 'interesting'... I have a similar wsm box and you should
> have a domain setup like:
>
> 0-5,12-17 (node 0)
> 6-11,18-23 (node 1)
>
> Where the first set (0-5 & 6-11) is smt0 and the second set (12-17 &
> 18-23) smt1.
>
> Can you boot with sched_debug and send me both /proc/cpuinfo and dmesg?
Never mind, what I said was partially BS. It is true that
I only see the thread on those runqueues. However you are
right that the domain mask is 0xFC0FC0 (6-11,18-23), but
the thread has an affinity mask of 0xFFF5D7, so the CPUs
in the intersection are 0xFC05C0, ie 9 and 11 aren't in its
affinity mask.
The behavior is still a little strange -- the scheduler is trying
6,7,8,10 and their SMT siblings (I left out 20 by accident in
my list but it is there). This is suggestive:
So could the fact that we don't have CPU 11 in our affinity
mask cause the scheduler not to try the CPU 23 runqueue?
Is it only looking at the first SMT sibling or something?
- R.
^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: runnable tasks never making it to a runqueue where they can run?
2012-03-07 18:03 ` Roland Dreier
@ 2012-03-08 10:28 ` Peter Zijlstra
2012-03-08 11:55 ` Srivatsa Vaddagiri
0 siblings, 1 reply; 7+ messages in thread
From: Peter Zijlstra @ 2012-03-08 10:28 UTC (permalink / raw)
To: Roland Dreier
Cc: LKML, Srivatsa Vaddagiri, Paul Turner, Venkatesh Pallipadi,
Suresh Siddha
On Wed, 2012-03-07 at 10:03 -0800, Roland Dreier wrote:
>
> So could the fact that we don't have CPU 11 in our affinity
> mask cause the scheduler not to try the CPU 23 runqueue?
> Is it only looking at the first SMT sibling or something?
Yes, I think we've conspired against your use-case here.
select_task_rq_fair() will very likely end up in select_idle_sibling()
which will never select the smt1 if smt0 is idle (even if smt0 isn't in
the cpus allowed mask) -- Vatsa is currently reworking that area, this
is something he might want to consider.
The regular load-balancer works by pulling tasks towards itself, this
works on the sched domains, it iterates bottom up through the domains,
but only the first cpu in the specific domain gets to go up.
This means that SMT1 will only ever pull from SMT0, and only SMT0 will
go up to the core level and (possibly) beyond to pull load. Now your
task -- due to its affinity mask -- can never be pulled to SMT0 and
hence will never end up on SMT1.
There's a number of very convoluted hacks in the load-balancer to
kinda-sort-of deal with affinity masks that are miss-aligned wrt the
domain setup, but they're painful and very likely lacking (as
demonstrated here).
For a while I've been toying with the idea of making the least-loaded
cpu in the mask go up in the balance pass -- as opposed to the first cpu
in the mask. This might be the justification, and push, I was needing.
In any case, load-balancing in the presence of affinity masks is an
'interesting' problem :-)
^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: runnable tasks never making it to a runqueue where they can run?
2012-03-08 10:28 ` Peter Zijlstra
@ 2012-03-08 11:55 ` Srivatsa Vaddagiri
2012-03-08 12:00 ` Peter Zijlstra
0 siblings, 1 reply; 7+ messages in thread
From: Srivatsa Vaddagiri @ 2012-03-08 11:55 UTC (permalink / raw)
To: Peter Zijlstra
Cc: Roland Dreier, LKML, Paul Turner, Venkatesh Pallipadi,
Suresh Siddha
* Peter Zijlstra <a.p.zijlstra@chello.nl> [2012-03-08 11:28:00]:
> select_task_rq_fair() will very likely end up in select_idle_sibling()
> which will never select the smt1 if smt0 is idle
s/idle/busy?
> (even if smt0 isn't in
> the cpus allowed mask) -- Vatsa is currently reworking that area, this
> is something he might want to consider.
The patch I sent couple of days back should likely fix this issue:
http://marc.info/?l=linux-kernel&m=133096114129607&w=2
- vatsa
^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: runnable tasks never making it to a runqueue where they can run?
2012-03-08 11:55 ` Srivatsa Vaddagiri
@ 2012-03-08 12:00 ` Peter Zijlstra
0 siblings, 0 replies; 7+ messages in thread
From: Peter Zijlstra @ 2012-03-08 12:00 UTC (permalink / raw)
To: Srivatsa Vaddagiri
Cc: Roland Dreier, LKML, Paul Turner, Venkatesh Pallipadi,
Suresh Siddha
On Thu, 2012-03-08 at 17:25 +0530, Srivatsa Vaddagiri wrote:
> * Peter Zijlstra <a.p.zijlstra@chello.nl> [2012-03-08 11:28:00]:
>
> > select_task_rq_fair() will very likely end up in select_idle_sibling()
> > which will never select the smt1 if smt0 is idle
>
> s/idle/busy?
Uhm, yeah ;-)
> > (even if smt0 isn't in
> > the cpus allowed mask) -- Vatsa is currently reworking that area, this
> > is something he might want to consider.
>
> The patch I sent couple of days back should likely fix this issue:
>
> http://marc.info/?l=linux-kernel&m=133096114129607&w=2
Ah, thanks for reminding me, I need to go stare at that!
^ permalink raw reply [flat|nested] 7+ messages in thread
end of thread, other threads:[~2012-03-08 12:00 UTC | newest]
Thread overview: 7+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2012-03-07 2:19 runnable tasks never making it to a runqueue where they can run? Roland Dreier
2012-03-07 16:30 ` Peter Zijlstra
2012-03-07 16:31 ` Peter Zijlstra
2012-03-07 18:03 ` Roland Dreier
2012-03-08 10:28 ` Peter Zijlstra
2012-03-08 11:55 ` Srivatsa Vaddagiri
2012-03-08 12:00 ` Peter Zijlstra
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox