* 3.19 - "unable" to replace a failed drive - 100% CPU usage in kworker and btrfs-transacti
@ 2015-02-16 2:32 Tomasz Chmielewski
2015-02-16 10:40 ` Liu Bo
0 siblings, 1 reply; 3+ messages in thread
From: Tomasz Chmielewski @ 2015-02-16 2:32 UTC (permalink / raw)
To: linux-btrfs
I had a failed drive in RAID-1, so it was replaced with a good one,
followed by:
btrfs device add /dev/sdb4 /home
btrfs device delete missing /home
4 days later, it got to a state when there is no IO anymore (according
to iostat), "btrfs device delete missing" did not complete:
# uptime
03:29:03 up 4 days, 14:38, 1 user, load average: 2.36, 2.43, 2.54
# btrfs fi show
Label: none uuid: 84d087aa-3a32-46da-844f-a233237cf04f
Total devices 3 FS bytes used 206.53GiB
devid 3 size 1.71TiB used 214.03GiB path /dev/sda4
devid 4 size 1.71TiB used 177.00GiB path /dev/sdb4
*** Some devices missing
Btrfs v3.18.2
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
6269 root 20 0 0 0 0 R 92.5 0.0 2769:33
btrfs-transacti
22247 root 20 0 0 0 0 R 92.5 0.0 42:38.65
kworker/u16:16
# dmesg
[397948.321324] SysRq : Show Blocked State
[397948.321386] task PC stack pid father
[397948.321465] btrfs D ffff8802d67ef948 0 7611 6275
0x00000000
[397948.321521] ffff8802d67ef948 ffff8804d2a9ba40 ffff88081917e040
00000000000113c0
[397948.321621] 0000000000004000 ffff88081be30000 ffff88081917e040
ffff8802d67ef8b8
[397948.321720] ffffffff8105f3c4 ffff88083fa40000 ffff88083fa913c0
ffff88083fa913c0
[397948.321819] Call Trace:
[397948.321872] [<ffffffff8105f3c4>] ? enqueue_task_fair+0x3e5/0x44f
[397948.321927] [<ffffffff81054ab6>] ? resched_curr+0x45/0x55
[397948.321979] [<ffffffff81055122>] ? check_preempt_curr+0x3e/0x6d
[397948.322032] [<ffffffff81055163>] ? ttwu_do_wakeup+0x12/0x7f
[397948.322085] [<ffffffff8105526e>] ?
ttwu_do_activate.constprop.73+0x57/0x5c
[397948.322141] [<ffffffff813be618>] schedule+0x65/0x67
[397948.322193] [<ffffffff813c03b8>] schedule_timeout+0x26/0x18d
[397948.322245] [<ffffffff81057d1e>] ? wake_up_process+0x30/0x34
[397948.322299] [<ffffffff8104b1a3>] ? wake_up_worker+0x1f/0x21
[397948.322352] [<ffffffff8104b3ea>] ? insert_work+0x87/0x94
[397948.322414] [<ffffffffa02e183d>] ? free_block_list+0x1f/0x34
[btrfs]
[397948.322468] [<ffffffff813bef76>] wait_for_common+0x10d/0x13e
[397948.322521] [<ffffffff81057cdf>] ? try_to_wake_up+0x250/0x250
[397948.322574] [<ffffffff813befbf>] wait_for_completion+0x18/0x1a
[397948.322631] [<ffffffffa028dedc>]
btrfs_async_run_delayed_refs+0xc1/0xe4 [btrfs]
[397948.322734] [<ffffffffa02a3189>]
__btrfs_end_transaction+0x315/0x33b [btrfs]
[397948.322836] [<ffffffffa02a31bd>]
btrfs_end_transaction_throttle+0xe/0x10 [btrfs]
[397948.322939] [<ffffffffa02e5cf1>] relocate_block_group+0x2ad/0x4de
[btrfs]
[397948.323039] [<ffffffffa02e607a>]
btrfs_relocate_block_group+0x158/0x278 [btrfs]
[397948.323152] [<ffffffffa02c37e1>]
btrfs_relocate_chunk.isra.69+0x35/0xa5 [btrfs]
[397948.323256] [<ffffffffa02c41da>] btrfs_shrink_device+0x235/0x408
[btrfs]
[397948.323316] [<ffffffffa02c6ba9>] btrfs_rm_device+0x2a9/0x704
[btrfs]
[397948.323371] [<ffffffff810f5441>] ?
__kmalloc_track_caller+0x40/0x178
[397948.323431] [<ffffffffa02cf7a7>] btrfs_ioctl+0xa9c/0x22f5 [btrfs]
[397948.323485] [<ffffffff8110f142>] ? putname+0x23/0x2c
[397948.323537] [<ffffffff8110f6d3>] ? user_path_at_empty+0x60/0x90
[397948.323593] [<ffffffff811783bf>] ? avc_has_perm+0x2e/0xf7
[397948.323654] [<ffffffff8111171b>] do_vfs_ioctl+0x418/0x460
[397948.323706] [<ffffffff81106745>] ? vfs_stat+0x16/0x18
[397948.323758] [<ffffffff811117b1>] SyS_ioctl+0x4e/0x7d
[397948.323809] [<ffffffff81031c5b>] ? do_page_fault+0xc/0x11
[397948.323862] [<ffffffff813c1352>] system_call_fastpath+0x12/0x17
[397948.323917] Sched Debug Version: v0.11, 3.19.0 #1
[397948.323968] ktime :
398158300.945440
[397948.324021] sched_clk :
397948323.916497
[397948.324074] cpu_clk :
397948323.916513
[397948.324127] jiffies : 4334753127
[397948.324180] sched_clock_stable() : 1
[397948.324231]
[397948.324277] sysctl_sched
[397948.324325] .sysctl_sched_latency : 24.000000
[397948.324378] .sysctl_sched_min_granularity : 3.000000
[397948.324430] .sysctl_sched_wakeup_granularity : 4.000000
[397948.324483] .sysctl_sched_child_runs_first : 0
[397948.324534] .sysctl_sched_features : 11899
[397948.324586] .sysctl_sched_tunable_scaling : 1
(logaritmic)
[397948.324639]
[397948.324686] cpu#0, 3411.800 MHz
[397948.324734] .nr_running : 0
[397948.324784] .load : 0
[397948.324834] .nr_switches : 137328510
[397948.324885] .nr_load_updates : 23262071
[397948.324936] .nr_uninterruptible : -738004
[397948.324987] .next_balance : 4334.753128
[397948.325039] .curr->pid : 0
[397948.325089] .clock : 397948323.134939
[397948.325141] .cpu_load[0] : 0
[397948.325191] .cpu_load[1] : 0
[397948.325241] .cpu_load[2] : 0
[397948.325292] .cpu_load[3] : 0
[397948.325343] .cpu_load[4] : 0
[397948.325393] .yld_count : 223168
[397948.325444] .sched_count : 137673310
[397948.325496] .sched_goidle : 50221823
[397948.325547] .avg_idle : 878593
[397948.325599] .max_idle_balance_cost : 500000
[397948.325650] .ttwu_count : 86464371
[397948.325701] .ttwu_local : 52888650
[397948.325752]
[397948.325752] cfs_rq[0]:/autogroup-1114
[397948.325847] .exec_clock : 11.086901
[397948.325899] .MIN_vruntime : 0.000001
[397948.325950] .min_vruntime : 10.038325
[397948.326001] .max_vruntime : 0.000001
[397948.326052] .spread : 0.000000
[397948.326103] .spread0 : -96825607.013236
[397948.326155] .nr_spread_over : 0
[397948.326205] .nr_running : 0
[397948.326256] .load : 0
[397948.326306] .runnable_load_avg : 0
[397948.326357] .blocked_load_avg : 0
[397948.326407] .tg_load_contrib : 0
[397948.326458] .tg_runnable_contrib : 0
[397948.326508] .tg_load_avg : 0
[397948.326559] .tg->runnable_avg : 0
[397948.326609] .tg->cfs_bandwidth.timer_active: 0
[397948.326659] .throttled : 0
[397948.326710] .throttle_count : 0
[397948.326760] .se->exec_start : 397948321.211399
[397948.326812] .se->vruntime : 96825606.516443
[397948.326865] .se->sum_exec_runtime : 11.086901
[397948.326916] .se->statistics.wait_start : 0.000000
[397948.326967] .se->statistics.sleep_start : 0.000000
[397948.327018] .se->statistics.block_start : 0.000000
[397948.327069] .se->statistics.sleep_max : 0.000000
[397948.327121] .se->statistics.block_max : 0.000000
[397948.327172] .se->statistics.exec_max : 2.982069
[397948.327223] .se->statistics.slice_max : 0.000000
[397948.327275] .se->statistics.wait_max : 0.019006
[397948.327326] .se->statistics.wait_sum : 0.019006
[397948.327378] .se->statistics.wait_count : 136
[397948.327429] .se->load.weight : 2
[397948.327479] .se->avg.runnable_avg_sum : 42
[397948.327530] .se->avg.runnable_avg_period : 48335
[397948.327581] .se->avg.load_avg_contrib : 0
[397948.327631] .se->avg.decay_count : 379513094
[397948.327683]
[397948.327683] cfs_rq[0]:/autogroup-6
[397948.327777] .exec_clock : 1407434.382402
[397948.327829] .MIN_vruntime : 0.000001
[397948.327880] .min_vruntime : 1154990.729705
[397948.327932] .max_vruntime : 0.000001
[397948.327984] .spread : 0.000000
[397948.328035] .spread0 : -95670626.321856
[397948.328087] .nr_spread_over : 255
[397948.329388] .nr_running : 0
[397948.329438] .load : 0
[397948.329489] .runnable_load_avg : 0
[397948.329539] .blocked_load_avg : 2
[397948.329590] .tg_load_contrib : 2
[397948.329640] .tg_runnable_contrib : 4
[397948.329690] .tg_load_avg : 12
[397948.329740] .tg->runnable_avg : 4
[397948.329791] .tg->cfs_bandwidth.timer_active: 0
[397948.329841] .throttled : 0
[397948.329892] .throttle_count : 0
[397948.329943] .se->exec_start : 397948323.133861
[397948.329995] .se->vruntime : 96825606.879532
[397948.330047] .se->sum_exec_runtime : 1407466.511198
[397948.330098] .se->statistics.wait_start : 0.000000
[397948.330150] .se->statistics.sleep_start : 0.000000
[397948.330201] .se->statistics.block_start : 0.000000
[397948.330252] .se->statistics.sleep_max : 0.000000
[397948.330303] .se->statistics.block_max : 0.000000
[397948.330354] .se->statistics.exec_max : 10.021049
[397948.330406] .se->statistics.slice_max : 16.519045
[397948.330457] .se->statistics.wait_max : 16.098230
[397948.330509] .se->statistics.wait_sum : 77143.758185
[397948.330561] .se->statistics.wait_count : 12935361
[397948.330613] .se->load.weight : 2
[397948.330663] .se->avg.runnable_avg_sum : 206
[397948.330714] .se->avg.runnable_avg_period : 47204
[397948.330765] .se->avg.load_avg_contrib : 2
[397948.330816] .se->avg.decay_count : 379513096
[397948.330868]
[397948.330868] cfs_rq[0]:/
[397948.330959] .exec_clock : 98523331.737970
[397948.331011] .MIN_vruntime : 0.000001
[397948.331062] .min_vruntime : 96825617.051561
[397948.331115] .max_vruntime : 0.000001
[397948.331166] .spread : 0.000000
[397948.331217] .spread0 : 0.000000
[397948.331269] .nr_spread_over : 270
[397948.331320] .nr_running : 0
[397948.331370] .load : 0
[397948.331421] .runnable_load_avg : 0
[397948.331471] .blocked_load_avg : 2
[397948.331522] .tg_load_contrib : 2
[397948.331572] .tg_runnable_contrib : 6
[397948.331622] .tg_load_avg : 1490
[397948.331673] .tg->runnable_avg : 1068
[397948.331723] .tg->cfs_bandwidth.timer_active: 0
[397948.331774] .throttled : 0
[397948.331824] .throttle_count : 0
[397948.331875] .avg->runnable_avg_sum : 286
[397948.331926] .avg->runnable_avg_period : 46637
[397948.331979]
[397948.331979] rt_rq[0]:/
[397948.332071] .rt_nr_running : 0
[397948.332121] .rt_throttled : 0
[397948.332171] .rt_time : 0.000000
[397948.332222] .rt_runtime : 950.000000
[397948.332274]
[397948.332274] dl_rq[0]:
[397948.332366] .dl_nr_running : 0
[397948.332417]
[397948.332417] runnable tasks:
[397948.332417] task PID tree-key switches prio
exec-runtime sum-exec sum-sleep
[397948.332417]
----------------------------------------------------------------------------------------------------------
[397948.332712] ksoftirqd/0 3 96825605.109947 740015 120
96825605.109947 12247.009129 397859768.730344 /
[397948.332817] kworker/0:0 4 96823284.697816 53 120
96823284.697816 17.309475 397934074.622814 /
[397948.332922] kworker/0:0H 5 1015.287609 5 100
1015.287609 0.041666 1136.081722 /
[397948.333053] rcu_sched 7 96825608.780644 13143448 120
96825608.780644 369650.173462 397567779.683357 /
[397948.333157] rcu_bh 8 117.354314 2 120
117.354314 0.000874 0.000722 /
[397948.333263] migration/0 9 0.000000 40752 0
0.000000 37728.554175 0.000853 /
[397948.333367] watchdog/0 10 -11.786179 99549 0
-11.786179 16602.242686 93.369695 /
[397948.333473] kswapd0 506 96346153.194271 10895 120
96346153.194271 5358.878543 396356848.501371 /
[397948.333577] acpi_thermal_pm 618 478.127841 2 100
478.127841 0.008079 0.004241 /
[397948.333682] ata_sff 832 943.140921 2 100
943.140921 0.005880 0.022278 /
[397948.333786] scsi_eh_0 855 1012.563036 18 120
1012.563036 0.875745 352.505402 /
[397948.333891] kworker/0:1H 879 96825605.051890 19712713 100
96825605.051890 933447.862690 396865888.829352 /
[397948.333995] jbd2/md2-8 935 96824920.898850 673247 120
96824920.898850 25939.369714 397908678.411380 /
[397948.334100] rsyslogd 3062 1154978.849958 554 120
1154978.849958 31.915075 397909811.026141 /autogroup-6
[397948.334206] nginx 3235 4173.366889 645371 120
4173.366889 8854.802084 393651417.627526 /autogroup-10
[397948.334312] mysqld 4080 1154807.414715 5643 120
1154807.414715 799.670146 395527445.704335 /autogroup-6
[397948.334418] mysqld 4094 1154978.441261 796267 120
1154978.441261 7509.987667 397898098.956672 /autogroup-6
[397948.334523] mysqld 4097 1154978.442889 1253821 120
1154978.442889 54810.874030 397848250.280201 /autogroup-6
[397948.334628] mysqld 4098 1154978.441618 1282226 120
1154978.441618 57182.533918 397845845.250436 /autogroup-6
[397948.334734] mysqld 4102 1154978.441697 959988 120
1154978.441697 10108.715585 397895341.475090 /autogroup-6
[397948.334839] mysqld 4103 1154978.440507 1341686 120
1154978.440507 16033.050027 397888981.976267 /autogroup-6
[397948.334945] mysqld 4119 1154990.729705 404501 120
1154990.729705 119263.212981 397785618.876052 /autogroup-6
[397948.335051] mysqld 4150 1039299.577821 7023984 120
1039299.577821 1407433.681869 320824575.671761 /autogroup-6
[397948.335157] mysqld 5811 1154806.870976 1317671 120
1154806.870976 299564.211303 346391209.749613 /autogroup-6
[397948.335264] mysqld 15436 1154807.136306 292526 120
1154807.136306 172005.952402 174398934.514010 /autogroup-6
[397948.335371] getty 4374 0.817181 100 120
0.817181 2.613716 146.560614 /autogroup-23
[397948.335478] btrfs-transacti 6269 96825617.051561 22374147 120
96825617.051561 166025164.637843 164906022.709080 /
[397948.335583] bash 6275 132.300150 276 120
132.300150 79.740920 21744924.508065 /autogroup-184
[397948.335690] kworker/0:1 21018 96823285.353043 1816351 120
96823285.353043 793610.285186 42038707.779192 /
[397948.335795] ssl-params 22730 472.603845 4 120
472.603845 0.877328 0.117947 /autogroup-17
[397948.335902] kworker/u16:0 22761 96825069.356076 67516 120
96825069.356076 540082.349638 226131.772902 /
[397948.336006] sshd 22771 10.038325 150 120
10.038325 13.472329 50175.143940 /autogroup-1114
[397948.336112] kworker/0:2 22822 96825605.578025 507 120
96825605.578025 47.279505 14055.084212 /
[397948.336218]
[397948.336280] cpu#1, 3411.800 MHz
[397948.336329] .nr_running : 1
[397948.336379] .load : 1024
[397948.336430] .nr_switches : 84332707
[397948.336481] .nr_load_updates : 21967275
[397948.336533] .nr_uninterruptible : 483742
[397948.336584] .next_balance : 4334.753129
[397948.336635] .curr->pid : 22247
[397948.336686] .clock : 397948333.217376
[397948.336738] .cpu_load[0] : 1023
[397948.336788] .cpu_load[1] : 1023
[397948.336839] .cpu_load[2] : 1023
[397948.336890] .cpu_load[3] : 1023
[397948.336940] .cpu_load[4] : 1007
[397948.336991] .yld_count : 135412
[397948.337043] .sched_count : 84726950
[397948.337094] .sched_goidle : 29157952
[397948.337145] .avg_idle : 1000000
[397948.337196] .max_idle_balance_cost : 500000
[397948.337247] .ttwu_count : 36438464
[397948.337299] .ttwu_local : 19485954
[397948.337350]
[397948.337350] cfs_rq[1]:/
[397948.337441] .exec_clock : 123202243.887995
[397948.337494] .MIN_vruntime : 0.000001
[397948.337545] .min_vruntime : 119610642.735572
[397948.337597] .max_vruntime : 0.000001
[397948.337649] .spread : 0.000000
[397948.337700] .spread0 : 22785025.684011
[397948.337752] .nr_spread_over : 301
[397948.337802] .nr_running : 1
[397948.337852] .load : 1024
[397948.337903] .runnable_load_avg : 1023
[397948.337954] .blocked_load_avg : 26
[397948.338004] .tg_load_contrib : 1042
[397948.338055] .tg_runnable_contrib : 1010
[397948.338106] .tg_load_avg : 1317
[397948.338156] .tg->runnable_avg : 1293
[397948.338206] .tg->cfs_bandwidth.timer_active: 0
[397948.338256] .throttled : 0
[397948.338307] .throttle_count : 0
[397948.338357] .avg->runnable_avg_sum : 46120
[397948.338408] .avg->runnable_avg_period : 46120
[397948.338459]
[397948.338459] rt_rq[1]:/
[397948.338550] .rt_nr_running : 0
[397948.338600] .rt_throttled : 0
[397948.339900] .rt_time : 0.000000
[397948.339951] .rt_runtime : 950.000000
[397948.340003]
[397948.340003] dl_rq[1]:
[397948.340094] .dl_nr_running : 0
[397948.340144]
[397948.340144] runnable tasks:
[397948.340144] task PID tree-key switches prio
exec-runtime sum-exec sum-sleep
[397948.340144]
----------------------------------------------------------------------------------------------------------
[397948.340438] watchdog/1 11 -2.973570 99543 0
-2.973570 21159.974792 85.639910 /
[397948.340543] migration/1 12 0.000000 37023 0
0.000000 30622.595831 0.000758 /
[397948.340648] ksoftirqd/1 13 119609979.133122 91495 120
119609979.133122 1646.457418 397900354.563118 /
[397948.340752] kworker/1:0H 15 582.372111 6 100
582.372111 0.027131 2733.616385 /
[397948.340858] khungtaskd 204 119577195.763580 3319 120
119577195.763580 182.485563 397829939.496656 /
[397948.340962] kpsmoused 686 25.893051 2 100
25.893051 0.010838 0.003636 /
[397948.341068] kworker/1:2 688 102920251.057920 55 120
102920251.057920 0.504998 355091633.648033 /
[397948.341172] deferwq 731 158.015576 2 100
158.015576 0.009076 0.003883 /
[397948.341277] scsi_tmf_0 856 442.536328 2 100
442.536328 0.008945 0.003987 /
[397948.341382] scsi_eh_1 857 539.461151 16 120
539.461151 0.868588 350.508685 /
[397948.341486] scsi_tmf_1 858 466.554669 2 100
466.554669 0.008786 0.003948 /
[397948.341590] scsi_eh_2 859 514.602593 3 120
514.602593 0.016414 1.372299 /
[397948.341695] scsi_eh_3 861 514.603810 3 120
514.603810 0.016837 1.222996 /
[397948.341799] scsi_tmf_3 862 502.578935 2 100
502.578935 0.008052 0.003896 /
[397948.341904] scsi_tmf_4 864 514.587500 2 100
514.587500 0.008905 0.004404 /
[397948.342009] kworker/1:1H 968 118250346.559508 456123 100
118250346.559508 6526.369618 394080278.360060 /
[397948.342114] cron 3301 777.910417 7708 120
777.910417 405.566912 397904071.914015 /autogroup-12
[397948.342219] sshd 3426 0.515010 28 120
0.515010 3.325645 397861007.941426 /autogroup-15
[397948.342326] mysqld_safe 3464 1418.978306 435 120
1418.978306 26.375085 974.715839 /autogroup-6
[397948.342432] mysqld 4118 690369.357591 400760 120
690369.357591 50887.461054 397853256.747730 /autogroup-6
[397948.342539] getty 4372 0.769188 104 120
0.769188 2.833955 146.554963 /autogroup-21
[397948.342645] btrfs-cleaner 6268 119610036.344925 21815 120
119610036.344925 5411.552915 342317734.332486 /
[397948.342750] btrfs 7611 311971.962451 2055291 120
311971.962451 1498599.957865 319025222.198539 /autogroup-184
[397948.342856] kworker/1:0 21019 119610630.984203 2134678 120
119610630.984203 1030853.178543 41814652.278685 /
[397948.342969] R kworker/u16:16 22247 119610652.481537 293438 120
119610652.481537 2301421.627687 2218308.587596 /
[397948.343099]
[397948.343161] cpu#2, 3411.800 MHz
[397948.343210] .nr_running : 3
[397948.343259] .load : 2048
[397948.343310] .nr_switches : 76867052
[397948.343360] .nr_load_updates : 19847327
[397948.343411] .nr_uninterruptible : 139945
[397948.343462] .next_balance : 4334.753139
[397948.343513] .curr->pid : 22773
[397948.343563] .clock : 397948336.246820
[397948.343615] .cpu_load[0] : 275
[397948.343666] .cpu_load[1] : 212
[397948.343716] .cpu_load[2] : 127
[397948.343767] .cpu_load[3] : 70
[397948.343818] .cpu_load[4] : 50
[397948.343868] .yld_count : 92399
[397948.343919] .sched_count : 77151242
[397948.343970] .sched_goidle : 27487304
[397948.344021] .avg_idle : 1000000
[397948.344072] .max_idle_balance_cost : 500000
[397948.344123] .ttwu_count : 35275362
[397948.344175] .ttwu_local : 17426867
[397948.344227]
[397948.344227] cfs_rq[2]:/autogroup-1115
[397948.344321] .exec_clock : 77.755926
[397948.344372] .MIN_vruntime : 0.000001
[397948.344423] .min_vruntime : 169.689274
[397948.344475] .max_vruntime : 0.000001
[397948.344526] .spread : 0.000000
[397948.344578] .spread0 : -96825447.362287
[397948.344629] .nr_spread_over : 7
[397948.344680] .nr_running : 1
[397948.344730] .load : 1024
[397948.344781] .runnable_load_avg : 271
[397948.344831] .blocked_load_avg : 0
[397948.344882] .tg_load_contrib : 271
[397948.344932] .tg_runnable_contrib : 271
[397948.344983] .tg_load_avg : 271
[397948.345033] .tg->runnable_avg : 271
[397948.345083] .tg->cfs_bandwidth.timer_active: 0
[397948.345133] .throttled : 0
[397948.345183] .throttle_count : 0
[397948.345234] .se->exec_start : 397948336.246820
[397948.345286] .se->vruntime : 100334135.344609
[397948.345338] .se->sum_exec_runtime : 77.755926
[397948.345389] .se->statistics.wait_start : 0.000000
[397948.345440] .se->statistics.sleep_start : 0.000000
[397948.345491] .se->statistics.block_start : 0.000000
[397948.345542] .se->statistics.sleep_max : 0.000000
[397948.345593] .se->statistics.block_max : 0.000000
[397948.345644] .se->statistics.exec_max : 13.202718
[397948.345694] .se->statistics.slice_max : 0.000000
[397948.345745] .se->statistics.wait_max : 0.020544
[397948.345796] .se->statistics.wait_sum : 0.209415
[397948.345847] .se->statistics.wait_count : 128
[397948.345898] .se->load.weight : 1024
[397948.345949] .se->avg.runnable_avg_sum : 12487
[397948.345999] .se->avg.runnable_avg_period : 47064
[397948.346051] .se->avg.load_avg_contrib : 275
[397948.346101] .se->avg.decay_count : 0
[397948.346152]
[397948.346152] cfs_rq[2]:/
[397948.346244] .exec_clock : 102436933.654759
[397948.346296] .MIN_vruntime : 100334120.416350
[397948.346348] .min_vruntime : 100334132.312627
[397948.346401] .max_vruntime : 100334120.416350
[397948.346453] .spread : 0.000000
[397948.346504] .spread0 : 3508515.261066
[397948.346556] .nr_spread_over : 450
[397948.346606] .nr_running : 2
[397948.346657] .load : 2048
[397948.346708] .runnable_load_avg : 275
[397948.346758] .blocked_load_avg : 0
[397948.346809] .tg_load_contrib : 275
[397948.346859] .tg_runnable_contrib : 271
[397948.346910] .tg_load_avg : 1323
[397948.346961] .tg->runnable_avg : 1293
[397948.347011] .tg->cfs_bandwidth.timer_active: 0
[397948.347061] .throttled : 0
[397948.347111] .throttle_count : 0
[397948.347161] .avg->runnable_avg_sum : 12763
[397948.347212] .avg->runnable_avg_period : 48021
[397948.347263]
[397948.347263] rt_rq[2]:/
[397948.347354] .rt_nr_running : 0
[397948.347404] .rt_throttled : 0
[397948.347455] .rt_time : 0.000000
[397948.347506] .rt_runtime : 950.000000
[397948.347557]
[397948.347557] dl_rq[2]:
[397948.347649] .dl_nr_running : 0
[397948.347699]
[397948.347699] runnable tasks:
[397948.347699] task PID tree-key switches prio
exec-runtime sum-exec sum-sleep
[397948.347699]
----------------------------------------------------------------------------------------------------------
[397948.347992] init 1 509.594135 79809 120
509.594135 2619.143889 397945091.854170 /autogroup-2
[397948.348098] watchdog/2 16 -5.976382 99543 0
-5.976382 17476.538367 72.326265 /
[397948.348202] migration/2 17 0.000000 32592 0
0.000000 29450.114444 0.001139 /
[397948.348306] ksoftirqd/2 18 100333573.328598 69839 120
100333573.328598 1929.169352 397913601.175468 /
[397948.348410] kworker/2:0 19 49262954.665776 38 120
49262954.665776 0.122608 231013304.220524 /
[397948.348514] kworker/2:0H 20 3650.646229 6 100
3650.646229 0.037533 34187.208217 /
[397948.348619] scsi_eh_4 863 3072.735732 3 120
3072.735732 0.015446 1.048287 /
[397948.348724] scsi_eh_5 865 3072.733721 3 120
3072.733721 0.012919 0.910030 /
[397948.348828] md1_raid1 910 100169530.338537 1429 120
100169530.338537 66.597411 397503924.718962 /
[397948.348932] kworker/2:1H 1466 100323518.282583 394863 100
100323518.282583 5278.861299 397866015.984528 /
[397948.349036] rsyslogd 3033 428725.147921 28 120
428725.147921 2.679249 322287744.247447 /autogroup-6
[397948.350392] atd 3189 0.622217 112 120
0.622217 2.344474 395791202.210800 /autogroup-9
[397948.350498] acpid 3240 0.478710 41 120
0.478710 1.730662 321163512.935538 /autogroup-11
[397948.350605] dbus-daemon 3404 0.942824 6 120
0.942824 1.484584 55885009.390813 /autogroup-14
[397948.350710] mysqld 4095 459653.767988 848117 120
459653.767988 9524.934466 397895732.816642 /autogroup-6
[397948.350817] mysqld 15425 459502.220847 261927 120
459502.220847 145837.180966 174449080.595390 /autogroup-6
[397948.350923] qmgr 4370 357.389137 2476 120
357.389137 280.199510 397622564.351466 /autogroup-19
[397948.351029] getty 4373 0.640858 102 120
0.640858 2.425952 146.827128 /autogroup-22
[397948.351135] btrfs-worker-hi 6248 1154879.053331 2 100
1154879.053331 0.017567 0.003220 /
[397948.351240] btrfs-delalloc 6249 1154891.063212 2 100
1154891.063212 0.010622 0.003413 /
[397948.351345] btrfs-flush_del 6250 1154903.071666 2 100
1154903.071666 0.009022 0.003320 /
[397948.351449] btrfs-cache 6251 1154915.081079 2 100
1154915.081079 0.015404 0.003203 /
[397948.351554] btrfs-submit 6252 1154915.162054 2 100
1154915.162054 0.096237 0.023172 /
[397948.351659] btrfs-endio-met 6256 1154927.254127 2 100
1154927.254127 0.093435 0.002869 /
[397948.351763] btrfs-rmw 6259 1154939.263666 2 100
1154939.263666 0.010258 0.003176 /
[397948.351868] btrfs-freespace 6261 1154951.272823 2 100
1154951.272823 0.009779 0.003215 /
[397948.351972] btrfs-delayed-m 6262 1154963.281556 2 100
1154963.281556 0.009338 0.003128 /
[397948.352076] btrfs-qgroup-re 6264 1154975.290187 2 100
1154975.290187 0.009265 0.003113 /
[397948.352180] kworker/2:2 16094 100334120.416350 6973086 120
100334120.416350 3053548.894806 163847153.385628 /
[397948.352285] pickup 22744 369.431995 35 120
369.431995 4.254482 1980726.020374 /autogroup-19
[397948.352391] R bash 22773 169.689274 106 120
169.689274 49.563444 46410.281275 /autogroup-1115
[397948.352497]
[397948.352543] cpu#3, 3411.800 MHz
[397948.352591] .nr_running : 0
[397948.352641] .load : 0
[397948.352692] .nr_switches : 78216804
[397948.352743] .nr_load_updates : 20561275
[397948.352794] .nr_uninterruptible : 84689
[397948.352845] .next_balance : 4334.753130
[397948.352896] .curr->pid : 0
[397948.352955] .clock : 397948343.119396
[397948.353033] .cpu_load[0] : 0
[397948.353100] .cpu_load[1] : 0
[397948.353150] .cpu_load[2] : 0
[397948.353200] .cpu_load[3] : 0
[397948.353251] .cpu_load[4] : 0
[397948.353300] .yld_count : 73921
[397948.353351] .sched_count : 78424464
[397948.353402] .sched_goidle : 27817319
[397948.353453] .avg_idle : 1000000
[397948.353504] .max_idle_balance_cost : 500000
[397948.353555] .ttwu_count : 36987465
[397948.353605] .ttwu_local : 18034474
[397948.353656]
[397948.353656] cfs_rq[3]:/autogroup-6
[397948.353751] .exec_clock : 484988.006368
[397948.353802] .MIN_vruntime : 0.000001
[397948.353854] .min_vruntime : 404786.905985
[397948.353905] .max_vruntime : 0.000001
[397948.353956] .spread : 0.000000
[397948.354007] .spread0 : -96420830.145576
[397948.354058] .nr_spread_over : 245
[397948.354109] .nr_running : 0
[397948.354159] .load : 0
[397948.354209] .runnable_load_avg : 0
[397948.354259] .blocked_load_avg : 4
[397948.354310] .tg_load_contrib : 4
[397948.354360] .tg_runnable_contrib : 5
[397948.354409] .tg_load_avg : 4
[397948.354460] .tg->runnable_avg : 7
[397948.354510] .tg->cfs_bandwidth.timer_active: 0
[397948.354560] .throttled : 0
[397948.354610] .throttle_count : 0
[397948.354661] .se->exec_start : 397948323.578248
[397948.354713] .se->vruntime : 102939999.703983
[397948.354766] .se->sum_exec_runtime : 485015.851159
[397948.354817] .se->statistics.wait_start : 0.000000
[397948.354868] .se->statistics.sleep_start : 0.000000
[397948.354919] .se->statistics.block_start : 0.000000
[397948.354970] .se->statistics.sleep_max : 0.000000
[397948.355021] .se->statistics.block_max : 0.000000
[397948.355072] .se->statistics.exec_max : 9.999422
[397948.355123] .se->statistics.slice_max : 9.991785
[397948.355174] .se->statistics.wait_max : 19.987876
[397948.355225] .se->statistics.wait_sum : 7370.667707
[397948.355277] .se->statistics.wait_count : 3844653
[397948.355327] .se->load.weight : 2
[397948.355378] .se->avg.runnable_avg_sum : 266
[397948.355428] .se->avg.runnable_avg_period : 47445
[397948.355479] .se->avg.load_avg_contrib : 5
[397948.355529] .se->avg.decay_count : 379513096
[397948.355581]
[397948.355581] cfs_rq[3]:/
[397948.355673] .exec_clock : 105213301.964322
[397948.355725] .MIN_vruntime : 0.000001
[397948.355776] .min_vruntime : 102940010.804921
[397948.355828] .max_vruntime : 0.000001
[397948.355879] .spread : 0.000000
[397948.355930] .spread0 : 6114393.753360
[397948.355982] .nr_spread_over : 326
[397948.356033] .nr_running : 0
[397948.356083] .load : 0
[397948.356133] .runnable_load_avg : 0
[397948.356183] .blocked_load_avg : 0
[397948.356234] .tg_load_contrib : 0
[397948.356284] .tg_runnable_contrib : 6
[397948.356335] .tg_load_avg : 1319
[397948.356385] .tg->runnable_avg : 1291
[397948.356436] .tg->cfs_bandwidth.timer_active: 0
[397948.356486] .throttled : 0
[397948.356537] .throttle_count : 0
[397948.356587] .avg->runnable_avg_sum : 307
[397948.356637] .avg->runnable_avg_period : 46706
[397948.356688]
[397948.356688] rt_rq[3]:/
[397948.356781] .rt_nr_running : 0
[397948.356831] .rt_throttled : 0
[397948.356882] .rt_time : 0.000000
[397948.356933] .rt_runtime : 950.000000
[397948.356984]
[397948.356984] dl_rq[3]:
[397948.357076] .dl_nr_running : 0
[397948.357127]
[397948.357127] runnable tasks:
[397948.357127] task PID tree-key switches prio
exec-runtime sum-exec sum-sleep
[397948.357127]
----------------------------------------------------------------------------------------------------------
[397948.357422] kthreadd 2 102939126.157704 1315 120
102939126.157704 38.637725 397940297.121677 /
[397948.357527] watchdog/3 21 -5.984614 99544 0
-5.984614 18147.844668 58.998123 /
[397948.357632] migration/3 22 0.000000 32291 0
0.000000 32254.090833 0.000847 /
[397948.357737] ksoftirqd/3 23 102939780.572027 62304 120
102939780.572027 1739.164572 397906895.811056 /
[397948.357841] kworker/3:0 24 33082955.717338 56 120
33082955.717338 0.596786 176596062.108639 /
[397948.357946] kworker/3:0H 25 1600.396479 6 100
1600.396479 0.027776 33504.961049 /
[397948.358051] khelper 46 10.955864 2 100
10.955864 0.005052 0.001801 /
[397948.358156] writeback 205 311.132001 2 100
311.132001 0.003788 0.001292 /
[397948.358261] crypto 208 323.135116 2 100
323.135116 0.003395 0.001548 /
[397948.358365] bioset 209 335.138067 2 100
335.138067 0.003268 0.001389 /
[397948.358470] kblockd 210 347.141115 2 100
347.141115 0.003369 0.001350 /
[397948.358574] md 361 359.144676 2 100
359.144676 0.003908 0.001863 /
[397948.358679] fsnotify_mark 576 1285553.304174 56 120
1285553.304174 1.356477 77382968.604008 /
[397948.358784] bioset 909 1410.076226 2 100
1410.076226 0.008348 0.004053 /
[397948.358888] md2_raid1 920 102939126.258310 5305156 120
102939126.258310 235266.004521 397697550.934379 /
[397948.358994] kworker/3:1H 1233 101531219.068494 383437 100
101531219.068494 4828.918177 393998408.709581 /
[397948.359098] rs:main Q:Reg 3060 404775.410651 3733 120
404775.410651 584.098973 397909253.918002 /autogroup-6
[397948.359205] vnstatd 3041 1938.625022 80791 120
1938.625022 7728.375777 397899296.874075 /autogroup-7
[397948.359311] mdadm 3144 21.070081 419 120
21.070081 64.059078 397466901.661042 /autogroup-8
[397948.359417] nginx 3232 4286.796414 20 120
4286.796414 1.473683 322285293.302498 /autogroup-10
[397948.359524] nginx 3233 5123.741553 498857 120
5123.741553 6408.084381 397902171.919015 /autogroup-10
[397948.359631] nginx 3234 5123.781505 795481 120
5123.781505 10575.489835 397897949.295633 /autogroup-10
[397948.360989] nginx 3236 5123.792442 453567 120
5123.792442 6288.632023 397902307.034538 /autogroup-10
[397948.361094] ntpd 3378 3177.303496 399798 120
3177.303496 11384.417725 397896567.168923 /autogroup-13
[397948.361201] mysqld 4096 404775.038669 1280416 120
404775.038669 56190.329409 397846935.062842 /autogroup-6
[397948.361306] mysqld 4099 404775.037536 1237509 120
404775.037536 53709.818619 397849497.344214 /autogroup-6
[397948.361413] mysqld 4100 404775.128644 1410134 120
404775.128644 17179.021072 397887821.391458 /autogroup-6
[397948.361520] mysqld 4101 404775.136626 1023118 120
404775.136626 11032.276249 397894304.441429 /autogroup-6
[397948.361625] mysqld 4120 404774.916067 79647 120
404774.916067 1272.295740 397899969.656177 /autogroup-6
[397948.361732] mysqld 4121 404774.944173 79565 120
404774.944173 1158.311382 397903499.069766 /autogroup-6
[397948.361838] mysqld 4123 260297.226173 260619 120
260297.226173 48020.432086 156765824.444851 /autogroup-6
[397948.361944] mysqld 4154 404597.326324 2178558 120
404597.326324 373081.589987 395136259.717365 /autogroup-6
[397948.362049] mysqld 4393 404597.164623 1651064 120
404597.164623 338373.325108 395122973.929693 /autogroup-6
[397948.362156] mysqld 4396 404771.119995 1079825 120
404771.119995 305191.529334 397422077.680807 /autogroup-6
[397948.362261] mysqld 15426 404596.822357 784917 120
404596.822357 137130.712105 174453678.251801 /autogroup-6
[397948.362367] mysqld 15435 404596.986208 228061 120
404596.986208 118651.907457 174461622.198681 /autogroup-6
[397948.362473] dovecot 4220 409.745461 1991 120
409.745461 113.183818 397150716.515321 /autogroup-17
[397948.362579] master 4345 415.904754 20039 120
415.904754 991.626364 397859137.298652 /autogroup-19
[397948.362684] getty 4375 0.839336 103 120
0.839336 1.796678 147.205983 /autogroup-24
[397948.362791] tmux 6274 11456.276708 987382 120
11456.276708 43704.262889 342265075.730957 /autogroup-183
[397948.362897] kworker/3:1 13622 102939999.914193 9663630 120
102939999.914193 4235651.348825 217069990.063754 /
[397948.363034] kworker/u16:2 22232 102939126.098258 163122 120
102939126.098258 1130115.311569 3704906.322727 /
[397948.363138] kworker/u17:0 22250 100814026.249739 4 100
100814026.249739 0.109630 5.440267 /
[397948.363243]
[397948.363304] cpu#4, 3411.800 MHz
[397948.363352] .nr_running : 0
[397948.363403] .load : 0
[397948.363452] .nr_switches : 31446353
[397948.363503] .nr_load_updates : 3466700
[397948.363554] .nr_uninterruptible : 12378
[397948.363605] .next_balance : 4334.753132
[397948.363656] .curr->pid : 0
[397948.363706] .clock : 397948363.266319
[397948.363758] .cpu_load[0] : 0
[397948.363809] .cpu_load[1] : 0
[397948.363858] .cpu_load[2] : 0
[397948.363909] .cpu_load[3] : 0
[397948.363958] .cpu_load[4] : 0
[397948.364009] .yld_count : 30496
[397948.364060] .sched_count : 31745499
[397948.364111] .sched_goidle : 13288472
[397948.364162] .avg_idle : 1000000
[397948.364213] .max_idle_balance_cost : 500000
[397948.364264] .ttwu_count : 12079122
[397948.364315] .ttwu_local : 3235209
[397948.364366]
[397948.364366] cfs_rq[4]:/
[397948.364458] .exec_clock : 18189528.762014
[397948.364510] .MIN_vruntime : 0.000001
[397948.364561] .min_vruntime : 22358073.568236
[397948.364613] .max_vruntime : 0.000001
[397948.364664] .spread : 0.000000
[397948.364715] .spread0 : -74467543.483325
[397948.364768] .nr_spread_over : 290
[397948.364818] .nr_running : 0
[397948.364868] .load : 0
[397948.364919] .runnable_load_avg : 0
[397948.364968] .blocked_load_avg : 0
[397948.365018] .tg_load_contrib : 0
[397948.365069] .tg_runnable_contrib : 0
[397948.365119] .tg_load_avg : 1315
[397948.365170] .tg->runnable_avg : 1290
[397948.365220] .tg->cfs_bandwidth.timer_active: 0
[397948.365271] .throttled : 0
[397948.365321] .throttle_count : 0
[397948.365371] .avg->runnable_avg_sum : 0
[397948.365421] .avg->runnable_avg_period : 48326
[397948.365472]
[397948.365472] rt_rq[4]:/
[397948.365564] .rt_nr_running : 0
[397948.365614] .rt_throttled : 0
[397948.365664] .rt_time : 0.000000
[397948.365715] .rt_runtime : 950.000000
[397948.365766]
[397948.365766] dl_rq[4]:
[397948.365858] .dl_nr_running : 0
[397948.365909]
[397948.365909] runnable tasks:
[397948.365909] task PID tree-key switches prio
exec-runtime sum-exec sum-sleep
[397948.365909]
----------------------------------------------------------------------------------------------------------
[397948.366203] watchdog/4 26 -8.974112 99544 0
-8.974112 1925.403645 45.684752 /
[397948.366307] migration/4 27 0.000000 32687 0
0.000000 19510.524185 0.000834 /
[397948.366412] ksoftirqd/4 28 22357044.685798 13474 120
22357044.685798 152.288196 397896961.880208 /
[397948.366516] kworker/4:0 29 1190565.109457 20 120
1190565.109457 0.048792 84471152.489397 /
[397948.366621] kworker/4:0H 30 1300.883814 6 100
1300.883814 0.042505 33602.235307 /
[397948.366726] irq/28-mei_me 673 0.000000 14 49
0.000000 0.320285 0.000000 /
[397948.366831] kworker/4:1H 1287 22130377.005493 40420 100
22130377.005493 571.072101 394074352.751479 /
[397948.366937] mysqld 4148 104218.340695 6 120
104218.340695 0.235452 160281236.866385 /autogroup-6
[397948.367043] anvil 4282 508.617446 540 120
508.617446 25.108427 395113557.223988 /autogroup-17
[397948.367148] getty 4371 3.299146 103 120
3.299146 4.682491 144.819918 /autogroup-20
[397948.367255] kworker/4:2 9937 22358061.586618 2184600 120
22358061.586618 951220.510870 312483748.018300 /
[397948.367360] kworker/u17:6 21267 21365561.424208 10 100
21365561.424208 2.594598 27216695.208805 /
[397948.367465] kworker/u16:1 22824 22358073.568236 4 120
22358073.568236 0.080475 4901.819753 /
[397948.367569]
[397948.367615] cpu#5, 3411.800 MHz
[397948.367664] .nr_running : 0
[397948.367714] .load : 0
[397948.367764] .nr_switches : 29264631
[397948.367816] .nr_load_updates : 3974073
[397948.367867] .nr_uninterruptible : 5752
[397948.367917] .next_balance : 4334.753132
[397948.367969] .curr->pid : 0
[397948.368020] .clock : 397948363.266965
[397948.368072] .cpu_load[0] : 0
[397948.368122] .cpu_load[1] : 0
[397948.368172] .cpu_load[2] : 0
[397948.368222] .cpu_load[3] : 0
[397948.368272] .cpu_load[4] : 0
[397948.368323] .yld_count : 23948
[397948.368373] .sched_count : 29418990
[397948.368424] .sched_goidle : 12116624
[397948.368475] .avg_idle : 1000000
[397948.368526] .max_idle_balance_cost : 500000
[397948.368577] .ttwu_count : 14464828
[397948.368628] .ttwu_local : 3286532
[397948.368679]
[397948.368679] cfs_rq[5]:/
[397948.368771] .exec_clock : 21096887.422727
[397948.368823] .MIN_vruntime : 0.000001
[397948.368873] .min_vruntime : 24070921.092851
[397948.368926] .max_vruntime : 0.000001
[397948.368977] .spread : 0.000000
[397948.369028] .spread0 : -72754695.958710
[397948.369080] .nr_spread_over : 219
[397948.369131] .nr_running : 0
[397948.369181] .load : 0
[397948.369232] .runnable_load_avg : 0
[397948.369282] .blocked_load_avg : 0
[397948.369332] .tg_load_contrib : 0
[397948.369382] .tg_runnable_contrib : 0
[397948.369432] .tg_load_avg : 1315
[397948.369483] .tg->runnable_avg : 1290
[397948.369534] .tg->cfs_bandwidth.timer_active: 0
[397948.369584] .throttled : 0
[397948.369634] .throttle_count : 0
[397948.369685] .avg->runnable_avg_sum : 0
[397948.369735] .avg->runnable_avg_period : 47521
[397948.369786]
[397948.369786] rt_rq[5]:/
[397948.369878] .rt_nr_running : 0
[397948.369928] .rt_throttled : 0
[397948.369979] .rt_time : 0.000000
[397948.370030] .rt_runtime : 950.000000
[397948.370081]
[397948.370081] dl_rq[5]:
[397948.371423] .dl_nr_running : 0
[397948.371474]
[397948.371474] runnable tasks:
[397948.371474] task PID tree-key switches prio
exec-runtime sum-exec sum-sleep
[397948.371474]
----------------------------------------------------------------------------------------------------------
[397948.371769] watchdog/5 31 -8.977217 99543 0
-8.977217 2363.156345 32.364766 /
[397948.371873] migration/5 32 0.000000 25712 0
0.000000 15366.174996 0.000845 /
[397948.371977] ksoftirqd/5 33 24070879.321423 15403 120
24070879.321423 155.107046 397934935.415490 /
[397948.372082] kworker/5:0 34 239.267711 17 120
239.267711 0.034507 632.432629 /
[397948.372186] kworker/5:0H 35 632.111736 6 100
632.111736 0.059549 2815.362585 /
[397948.372291] kworker/5:1 611 24070909.178313 2597656 120
24070909.178313 1203696.900841 396713444.701918 /
[397948.372396] scsi_tmf_2 860 534.239404 2 100
534.239404 0.008896 0.005460 /
[397948.372500] scsi_tmf_5 866 558.255161 2 100
558.255161 0.008179 0.004413 /
[397948.372605] kworker/5:1H 1061 23860072.399440 45682 100
23860072.399440 510.190883 394083026.021665 /
[397948.372711]
[397948.372757] cpu#6, 3411.800 MHz
[397948.372805] .nr_running : 0
[397948.372855] .load : 0
[397948.372905] .nr_switches : 30595250
[397948.372972] .nr_load_updates : 4632598
[397948.373039] .nr_uninterruptible : 6640
[397948.373089] .next_balance : 4334.753132
[397948.373140] .curr->pid : 0
[397948.373191] .clock : 397948363.267655
[397948.373243] .cpu_load[0] : 0
[397948.373292] .cpu_load[1] : 0
[397948.373342] .cpu_load[2] : 0
[397948.373393] .cpu_load[3] : 0
[397948.373443] .cpu_load[4] : 0
[397948.373494] .yld_count : 20767
[397948.373545] .sched_count : 30739543
[397948.373596] .sched_goidle : 12161262
[397948.373647] .avg_idle : 1000000
[397948.373698] .max_idle_balance_cost : 500000
[397948.373749] .ttwu_count : 17363747
[397948.373800] .ttwu_local : 4024881
[397948.373852]
[397948.373852] cfs_rq[6]:/
[397948.373944] .exec_clock : 27066813.057347
[397948.373996] .MIN_vruntime : 0.000001
[397948.374046] .min_vruntime : 29909549.644335
[397948.374099] .max_vruntime : 0.000001
[397948.374150] .spread : 0.000000
[397948.374202] .spread0 : -66916067.407226
[397948.374253] .nr_spread_over : 407
[397948.374304] .nr_running : 0
[397948.374354] .load : 0
[397948.374404] .runnable_load_avg : 0
[397948.374454] .blocked_load_avg : 0
[397948.374504] .tg_load_contrib : 0
[397948.374554] .tg_runnable_contrib : 0
[397948.374605] .tg_load_avg : 1313
[397948.374656] .tg->runnable_avg : 1289
[397948.374706] .tg->cfs_bandwidth.timer_active: 0
[397948.374757] .throttled : 0
[397948.374807] .throttle_count : 0
[397948.374857] .avg->runnable_avg_sum : 0
[397948.374907] .avg->runnable_avg_period : 47423
[397948.374958]
[397948.374958] rt_rq[6]:/
[397948.375050] .rt_nr_running : 0
[397948.375099] .rt_throttled : 0
[397948.375149] .rt_time : 0.000000
[397948.375200] .rt_runtime : 950.000000
[397948.375258]
[397948.375258] dl_rq[6]:
[397948.375359] .dl_nr_running : 0
[397948.375410]
[397948.375410] runnable tasks:
[397948.375410] task PID tree-key switches prio
exec-runtime sum-exec sum-sleep
[397948.375410]
----------------------------------------------------------------------------------------------------------
[397948.375704] watchdog/6 36 -8.979370 99543 0
-8.979370 3873.032642 19.046238 /
[397948.375809] migration/6 37 0.000000 20867 0
0.000000 12947.385988 0.000826 /
[397948.375914] ksoftirqd/6 38 29909537.744505 17979 120
29909537.744505 324.324395 397932684.911680 /
[397948.376018] kworker/6:0 39 2652.785847 15 120
2652.785847 0.034665 619.334482 /
[397948.376123] kworker/6:0H 40 3222.777674 6 100
3222.777674 0.042578 34131.960421 /
[397948.376228] kworker/6:1 613 29909537.712176 3191586 120
29909537.712176 1584243.664849 396333417.817101 /
[397948.376333] bioset 898 2957.541580 2 100
2957.541580 0.008747 0.006389 /
[397948.376438] kworker/6:1H 1463 29688535.473225 50544 100
29688535.473225 612.882638 394061042.264128 /
[397948.376542] kjournald 2216 3410.060184 2 120
3410.060184 0.412389 0.043474 /
[397948.376646] bioset 2222 3422.126220 2 100
3422.126220 0.088585 0.040787 /
[397948.376750] rsyslogd 3061 163480.988716 3499 120
163480.988716 146.745864 397862580.052157 /autogroup-6
[397948.376856] mysqld 4122 163443.475621 644351 120
163443.475621 179019.020540 393898317.914127 /autogroup-6
[397948.376962] udevd 6137 246.142493 30 120
246.142493 6.670412 22056460.101598 /autogroup-4
[397948.377068] udevd 6138 242.586598 15 120
242.586598 3.149366 49194.013301 /autogroup-4
[397948.377174] btrfs-worker 6246 1051403.790298 2 100
1051403.790298 0.011158 0.004647 /
[397948.377278] btrfs-endio-met 6255 1051427.810805 2 100
1051427.810805 0.017787 0.002771 /
[397948.377383] btrfs-endio-rai 6257 1051439.820346 2 100
1051439.820346 0.010462 0.002911 /
[397948.377487] btrfs-endio-rep 6258 1051451.828334 2 100
1051451.828334 0.008652 0.003137 /
[397948.377591] btrfs-endio-wri 6260 1051463.836060 2 100
1051463.836060 0.008410 0.003085 /
[397948.377695] btrfs-readahead 6263 1051475.843882 2 100
1051475.843882 0.008543 0.003153 /
[397948.377799] btrfs-extent-re 6265 1051487.851432 2 100
1051487.851432 0.008154 0.003265 /
[397948.377904]
[397948.377950] cpu#7, 3411.800 MHz
[397948.377998] .nr_running : 0
[397948.378048] .load : 0
[397948.378098] .nr_switches : 28501260
[397948.378150] .nr_load_updates : 3359354
[397948.378200] .nr_uninterruptible : 4860
[397948.378251] .next_balance : 4334.753132
[397948.378302] .curr->pid : 0
[397948.378351] .clock : 397948363.268406
[397948.378403] .cpu_load[0] : 0
[397948.378453] .cpu_load[1] : 0
[397948.378504] .cpu_load[2] : 0
[397948.378554] .cpu_load[3] : 0
[397948.378604] .cpu_load[4] : 0
[397948.378654] .yld_count : 22779
[397948.378705] .sched_count : 28647495
[397948.378757] .sched_goidle : 12446869
[397948.378808] .avg_idle : 1000000
[397948.378859] .max_idle_balance_cost : 500000
[397948.378910] .ttwu_count : 18391339
[397948.378961] .ttwu_local : 2717640
[397948.379012]
[397948.379012] cfs_rq[7]:/
[397948.379112] .exec_clock : 13565979.621209
[397948.379173] .MIN_vruntime : 0.000001
[397948.379225] .min_vruntime : 17262906.632013
[397948.379276] .max_vruntime : 0.000001
[397948.379327] .spread : 0.000000
[397948.379378] .spread0 : -79562710.419548
[397948.379430] .nr_spread_over : 255
[397948.379480] .nr_running : 0
[397948.379530] .load : 0
[397948.379581] .runnable_load_avg : 0
[397948.379631] .blocked_load_avg : 0
[397948.379682] .tg_load_contrib : 0
[397948.379732] .tg_runnable_contrib : 0
[397948.379783] .tg_load_avg : 1313
[397948.379833] .tg->runnable_avg : 1288
[397948.379884] .tg->cfs_bandwidth.timer_active: 0
[397948.379933] .throttled : 0
[397948.379983] .throttle_count : 0
[397948.380034] .avg->runnable_avg_sum : 5
[397948.380084] .avg->runnable_avg_period : 47099
[397948.380135]
[397948.380135] rt_rq[7]:/
[397948.380227] .rt_nr_running : 0
[397948.380277] .rt_throttled : 0
[397948.380327] .rt_time : 0.000000
[397948.380379] .rt_runtime : 950.000000
[397948.380430]
[397948.380430] dl_rq[7]:
[397948.380522] .dl_nr_running : 0
[397948.380572]
[397948.380572] runnable tasks:
[397948.380572] task PID tree-key switches prio
exec-runtime sum-exec sum-sleep
[397948.380572]
----------------------------------------------------------------------------------------------------------
[397948.382132] watchdog/7 41 -8.982956 99544 0
-8.982956 1585.113591 5.725643 /
[397948.382236] migration/7 42 0.000000 21048 0
0.000000 12757.115970 0.000816 /
[397948.382341] ksoftirqd/7 43 17254754.851360 11826 120
17254754.851360 146.221401 397720187.660046 /
[397948.382446] kworker/7:0 44 479.385599 15 120
479.385599 0.031395 606.240559 /
[397948.382550] kworker/7:0H 45 1817.499047 6 100
1817.499047 0.047704 42727.098840 /
[397948.382655] netns 47 10.955152 2 100
10.955152 0.004076 0.001586 /
[397948.382759] perf 50 22.958573 2 100
22.958573 0.003794 0.001394 /
[397948.382863] kworker/7:1 615 17262894.767216 1847690 120
17262894.767216 746881.932822 397173736.970567 /
[397948.382992] md0_raid1 899 17253995.722314 32645 120
17253995.722314 997.731148 397503108.422867 /
[397948.383105] bioset 919 1132.474026 2 100
1132.474026 0.034373 0.004008 /
[397948.383210] ext4-rsv-conver 936 1156.590460 2 100
1156.590460 0.061309 0.026946 /
[397948.383315] udevd 1124 405.566270 290 120
405.566270 53.586152 77350627.244245 /autogroup-4
[397948.383421] ipv6_addrconf 3209 1679.353321 2 100
1679.353321 0.006452 0.002581 /
[397948.383527] kworker/7:1H 4116 16792468.563089 52255 100
16792468.563089 541.499314 393833366.670026 /
[397948.383632] log 4283 574.060901 688 120
574.060901 34.058909 395113548.562211 /autogroup-17
[397948.383739] getty 4376 0.726364 103 120
0.726364 1.882302 146.996217 /autogroup-25
[397948.383845] btrfs-fixup 6253 1049025.643969 2 100
1049025.643969 0.011376 0.003865 /
[397948.383950] btrfs-endio 6254 1049037.659031 2 100
1049037.659031 0.015618 0.001516 /
[397948.384056]
Tomasz Chmielewski
http://www.sslrack.com
^ permalink raw reply [flat|nested] 3+ messages in thread
* Re: 3.19 - "unable" to replace a failed drive - 100% CPU usage in kworker and btrfs-transacti
2015-02-16 2:32 3.19 - "unable" to replace a failed drive - 100% CPU usage in kworker and btrfs-transacti Tomasz Chmielewski
@ 2015-02-16 10:40 ` Liu Bo
2015-02-16 10:51 ` Tomasz Chmielewski
0 siblings, 1 reply; 3+ messages in thread
From: Liu Bo @ 2015-02-16 10:40 UTC (permalink / raw)
To: Tomasz Chmielewski; +Cc: linux-btrfs
On Mon, Feb 16, 2015 at 11:32:56AM +0900, Tomasz Chmielewski wrote:
> I had a failed drive in RAID-1, so it was replaced with a good one,
> followed by:
>
> btrfs device add /dev/sdb4 /home
> btrfs device delete missing /home
>
>
> 4 days later, it got to a state when there is no IO anymore
> (according to iostat), "btrfs device delete missing" did not
> complete:
>
> # uptime
> 03:29:03 up 4 days, 14:38, 1 user, load average: 2.36, 2.43, 2.54
>
> # btrfs fi show
> Label: none uuid: 84d087aa-3a32-46da-844f-a233237cf04f
> Total devices 3 FS bytes used 206.53GiB
> devid 3 size 1.71TiB used 214.03GiB path /dev/sda4
> devid 4 size 1.71TiB used 177.00GiB path /dev/sdb4
> *** Some devices missing
>
> Btrfs v3.18.2
>
> PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
> 6269 root 20 0 0 0 0 R 92.5 0.0 2769:33
> btrfs-transacti
> 22247 root 20 0 0 0 0 R 92.5 0.0 42:38.65
> kworker/u16:16
Can you cat /proc/22247/stack and /proc/6269/stack?
Thanks,
-liubo
>
>
> # dmesg
> [397948.321324] SysRq : Show Blocked State
> [397948.321386] task PC stack pid father
> [397948.321465] btrfs D ffff8802d67ef948 0 7611
> 6275 0x00000000
> [397948.321521] ffff8802d67ef948 ffff8804d2a9ba40 ffff88081917e040
> 00000000000113c0
> [397948.321621] 0000000000004000 ffff88081be30000 ffff88081917e040
> ffff8802d67ef8b8
> [397948.321720] ffffffff8105f3c4 ffff88083fa40000 ffff88083fa913c0
> ffff88083fa913c0
> [397948.321819] Call Trace:
> [397948.321872] [<ffffffff8105f3c4>] ? enqueue_task_fair+0x3e5/0x44f
> [397948.321927] [<ffffffff81054ab6>] ? resched_curr+0x45/0x55
> [397948.321979] [<ffffffff81055122>] ? check_preempt_curr+0x3e/0x6d
> [397948.322032] [<ffffffff81055163>] ? ttwu_do_wakeup+0x12/0x7f
> [397948.322085] [<ffffffff8105526e>] ?
> ttwu_do_activate.constprop.73+0x57/0x5c
> [397948.322141] [<ffffffff813be618>] schedule+0x65/0x67
> [397948.322193] [<ffffffff813c03b8>] schedule_timeout+0x26/0x18d
> [397948.322245] [<ffffffff81057d1e>] ? wake_up_process+0x30/0x34
> [397948.322299] [<ffffffff8104b1a3>] ? wake_up_worker+0x1f/0x21
> [397948.322352] [<ffffffff8104b3ea>] ? insert_work+0x87/0x94
> [397948.322414] [<ffffffffa02e183d>] ? free_block_list+0x1f/0x34
> [btrfs]
> [397948.322468] [<ffffffff813bef76>] wait_for_common+0x10d/0x13e
> [397948.322521] [<ffffffff81057cdf>] ? try_to_wake_up+0x250/0x250
> [397948.322574] [<ffffffff813befbf>] wait_for_completion+0x18/0x1a
> [397948.322631] [<ffffffffa028dedc>]
> btrfs_async_run_delayed_refs+0xc1/0xe4 [btrfs]
> [397948.322734] [<ffffffffa02a3189>]
> __btrfs_end_transaction+0x315/0x33b [btrfs]
> [397948.322836] [<ffffffffa02a31bd>]
> btrfs_end_transaction_throttle+0xe/0x10 [btrfs]
> [397948.322939] [<ffffffffa02e5cf1>]
> relocate_block_group+0x2ad/0x4de [btrfs]
> [397948.323039] [<ffffffffa02e607a>]
> btrfs_relocate_block_group+0x158/0x278 [btrfs]
> [397948.323152] [<ffffffffa02c37e1>]
> btrfs_relocate_chunk.isra.69+0x35/0xa5 [btrfs]
> [397948.323256] [<ffffffffa02c41da>]
> btrfs_shrink_device+0x235/0x408 [btrfs]
> [397948.323316] [<ffffffffa02c6ba9>] btrfs_rm_device+0x2a9/0x704
> [btrfs]
> [397948.323371] [<ffffffff810f5441>] ?
> __kmalloc_track_caller+0x40/0x178
> [397948.323431] [<ffffffffa02cf7a7>] btrfs_ioctl+0xa9c/0x22f5 [btrfs]
> [397948.323485] [<ffffffff8110f142>] ? putname+0x23/0x2c
> [397948.323537] [<ffffffff8110f6d3>] ? user_path_at_empty+0x60/0x90
> [397948.323593] [<ffffffff811783bf>] ? avc_has_perm+0x2e/0xf7
> [397948.323654] [<ffffffff8111171b>] do_vfs_ioctl+0x418/0x460
> [397948.323706] [<ffffffff81106745>] ? vfs_stat+0x16/0x18
> [397948.323758] [<ffffffff811117b1>] SyS_ioctl+0x4e/0x7d
> [397948.323809] [<ffffffff81031c5b>] ? do_page_fault+0xc/0x11
> [397948.323862] [<ffffffff813c1352>] system_call_fastpath+0x12/0x17
> [397948.323917] Sched Debug Version: v0.11, 3.19.0 #1
> [397948.323968] ktime :
> 398158300.945440
> [397948.324021] sched_clk :
> 397948323.916497
> [397948.324074] cpu_clk :
> 397948323.916513
> [397948.324127] jiffies : 4334753127
> [397948.324180] sched_clock_stable() : 1
> [397948.324231]
> [397948.324277] sysctl_sched
> [397948.324325] .sysctl_sched_latency : 24.000000
> [397948.324378] .sysctl_sched_min_granularity : 3.000000
> [397948.324430] .sysctl_sched_wakeup_granularity : 4.000000
> [397948.324483] .sysctl_sched_child_runs_first : 0
> [397948.324534] .sysctl_sched_features : 11899
> [397948.324586] .sysctl_sched_tunable_scaling : 1
> (logaritmic)
> [397948.324639]
> [397948.324686] cpu#0, 3411.800 MHz
> [397948.324734] .nr_running : 0
> [397948.324784] .load : 0
> [397948.324834] .nr_switches : 137328510
> [397948.324885] .nr_load_updates : 23262071
> [397948.324936] .nr_uninterruptible : -738004
> [397948.324987] .next_balance : 4334.753128
> [397948.325039] .curr->pid : 0
> [397948.325089] .clock : 397948323.134939
> [397948.325141] .cpu_load[0] : 0
> [397948.325191] .cpu_load[1] : 0
> [397948.325241] .cpu_load[2] : 0
> [397948.325292] .cpu_load[3] : 0
> [397948.325343] .cpu_load[4] : 0
> [397948.325393] .yld_count : 223168
> [397948.325444] .sched_count : 137673310
> [397948.325496] .sched_goidle : 50221823
> [397948.325547] .avg_idle : 878593
> [397948.325599] .max_idle_balance_cost : 500000
> [397948.325650] .ttwu_count : 86464371
> [397948.325701] .ttwu_local : 52888650
> [397948.325752]
> [397948.325752] cfs_rq[0]:/autogroup-1114
> [397948.325847] .exec_clock : 11.086901
> [397948.325899] .MIN_vruntime : 0.000001
> [397948.325950] .min_vruntime : 10.038325
> [397948.326001] .max_vruntime : 0.000001
> [397948.326052] .spread : 0.000000
> [397948.326103] .spread0 : -96825607.013236
> [397948.326155] .nr_spread_over : 0
> [397948.326205] .nr_running : 0
> [397948.326256] .load : 0
> [397948.326306] .runnable_load_avg : 0
> [397948.326357] .blocked_load_avg : 0
> [397948.326407] .tg_load_contrib : 0
> [397948.326458] .tg_runnable_contrib : 0
> [397948.326508] .tg_load_avg : 0
> [397948.326559] .tg->runnable_avg : 0
> [397948.326609] .tg->cfs_bandwidth.timer_active: 0
> [397948.326659] .throttled : 0
> [397948.326710] .throttle_count : 0
> [397948.326760] .se->exec_start : 397948321.211399
> [397948.326812] .se->vruntime : 96825606.516443
> [397948.326865] .se->sum_exec_runtime : 11.086901
> [397948.326916] .se->statistics.wait_start : 0.000000
> [397948.326967] .se->statistics.sleep_start : 0.000000
> [397948.327018] .se->statistics.block_start : 0.000000
> [397948.327069] .se->statistics.sleep_max : 0.000000
> [397948.327121] .se->statistics.block_max : 0.000000
> [397948.327172] .se->statistics.exec_max : 2.982069
> [397948.327223] .se->statistics.slice_max : 0.000000
> [397948.327275] .se->statistics.wait_max : 0.019006
> [397948.327326] .se->statistics.wait_sum : 0.019006
> [397948.327378] .se->statistics.wait_count : 136
> [397948.327429] .se->load.weight : 2
> [397948.327479] .se->avg.runnable_avg_sum : 42
> [397948.327530] .se->avg.runnable_avg_period : 48335
> [397948.327581] .se->avg.load_avg_contrib : 0
> [397948.327631] .se->avg.decay_count : 379513094
> [397948.327683]
> [397948.327683] cfs_rq[0]:/autogroup-6
> [397948.327777] .exec_clock : 1407434.382402
> [397948.327829] .MIN_vruntime : 0.000001
> [397948.327880] .min_vruntime : 1154990.729705
> [397948.327932] .max_vruntime : 0.000001
> [397948.327984] .spread : 0.000000
> [397948.328035] .spread0 : -95670626.321856
> [397948.328087] .nr_spread_over : 255
> [397948.329388] .nr_running : 0
> [397948.329438] .load : 0
> [397948.329489] .runnable_load_avg : 0
> [397948.329539] .blocked_load_avg : 2
> [397948.329590] .tg_load_contrib : 2
> [397948.329640] .tg_runnable_contrib : 4
> [397948.329690] .tg_load_avg : 12
> [397948.329740] .tg->runnable_avg : 4
> [397948.329791] .tg->cfs_bandwidth.timer_active: 0
> [397948.329841] .throttled : 0
> [397948.329892] .throttle_count : 0
> [397948.329943] .se->exec_start : 397948323.133861
> [397948.329995] .se->vruntime : 96825606.879532
> [397948.330047] .se->sum_exec_runtime : 1407466.511198
> [397948.330098] .se->statistics.wait_start : 0.000000
> [397948.330150] .se->statistics.sleep_start : 0.000000
> [397948.330201] .se->statistics.block_start : 0.000000
> [397948.330252] .se->statistics.sleep_max : 0.000000
> [397948.330303] .se->statistics.block_max : 0.000000
> [397948.330354] .se->statistics.exec_max : 10.021049
> [397948.330406] .se->statistics.slice_max : 16.519045
> [397948.330457] .se->statistics.wait_max : 16.098230
> [397948.330509] .se->statistics.wait_sum : 77143.758185
> [397948.330561] .se->statistics.wait_count : 12935361
> [397948.330613] .se->load.weight : 2
> [397948.330663] .se->avg.runnable_avg_sum : 206
> [397948.330714] .se->avg.runnable_avg_period : 47204
> [397948.330765] .se->avg.load_avg_contrib : 2
> [397948.330816] .se->avg.decay_count : 379513096
> [397948.330868]
> [397948.330868] cfs_rq[0]:/
> [397948.330959] .exec_clock : 98523331.737970
> [397948.331011] .MIN_vruntime : 0.000001
> [397948.331062] .min_vruntime : 96825617.051561
> [397948.331115] .max_vruntime : 0.000001
> [397948.331166] .spread : 0.000000
> [397948.331217] .spread0 : 0.000000
> [397948.331269] .nr_spread_over : 270
> [397948.331320] .nr_running : 0
> [397948.331370] .load : 0
> [397948.331421] .runnable_load_avg : 0
> [397948.331471] .blocked_load_avg : 2
> [397948.331522] .tg_load_contrib : 2
> [397948.331572] .tg_runnable_contrib : 6
> [397948.331622] .tg_load_avg : 1490
> [397948.331673] .tg->runnable_avg : 1068
> [397948.331723] .tg->cfs_bandwidth.timer_active: 0
> [397948.331774] .throttled : 0
> [397948.331824] .throttle_count : 0
> [397948.331875] .avg->runnable_avg_sum : 286
> [397948.331926] .avg->runnable_avg_period : 46637
> [397948.331979]
> [397948.331979] rt_rq[0]:/
> [397948.332071] .rt_nr_running : 0
> [397948.332121] .rt_throttled : 0
> [397948.332171] .rt_time : 0.000000
> [397948.332222] .rt_runtime : 950.000000
> [397948.332274]
> [397948.332274] dl_rq[0]:
> [397948.332366] .dl_nr_running : 0
> [397948.332417]
> [397948.332417] runnable tasks:
> [397948.332417] task PID tree-key switches
> prio exec-runtime sum-exec sum-sleep
> [397948.332417] ----------------------------------------------------------------------------------------------------------
> [397948.332712] ksoftirqd/0 3 96825605.109947 740015
> 120 96825605.109947 12247.009129 397859768.730344 /
> [397948.332817] kworker/0:0 4 96823284.697816 53
> 120 96823284.697816 17.309475 397934074.622814 /
> [397948.332922] kworker/0:0H 5 1015.287609 5
> 100 1015.287609 0.041666 1136.081722 /
> [397948.333053] rcu_sched 7 96825608.780644 13143448
> 120 96825608.780644 369650.173462 397567779.683357 /
> [397948.333157] rcu_bh 8 117.354314 2
> 120 117.354314 0.000874 0.000722 /
> [397948.333263] migration/0 9 0.000000 40752
> 0 0.000000 37728.554175 0.000853 /
> [397948.333367] watchdog/0 10 -11.786179 99549
> 0 -11.786179 16602.242686 93.369695 /
> [397948.333473] kswapd0 506 96346153.194271 10895
> 120 96346153.194271 5358.878543 396356848.501371 /
> [397948.333577] acpi_thermal_pm 618 478.127841 2
> 100 478.127841 0.008079 0.004241 /
> [397948.333682] ata_sff 832 943.140921 2
> 100 943.140921 0.005880 0.022278 /
> [397948.333786] scsi_eh_0 855 1012.563036 18
> 120 1012.563036 0.875745 352.505402 /
> [397948.333891] kworker/0:1H 879 96825605.051890 19712713
> 100 96825605.051890 933447.862690 396865888.829352 /
> [397948.333995] jbd2/md2-8 935 96824920.898850 673247
> 120 96824920.898850 25939.369714 397908678.411380 /
> [397948.334100] rsyslogd 3062 1154978.849958 554
> 120 1154978.849958 31.915075 397909811.026141 /autogroup-6
> [397948.334206] nginx 3235 4173.366889 645371
> 120 4173.366889 8854.802084 393651417.627526 /autogroup-10
> [397948.334312] mysqld 4080 1154807.414715 5643
> 120 1154807.414715 799.670146 395527445.704335 /autogroup-6
> [397948.334418] mysqld 4094 1154978.441261 796267
> 120 1154978.441261 7509.987667 397898098.956672 /autogroup-6
> [397948.334523] mysqld 4097 1154978.442889 1253821
> 120 1154978.442889 54810.874030 397848250.280201 /autogroup-6
> [397948.334628] mysqld 4098 1154978.441618 1282226
> 120 1154978.441618 57182.533918 397845845.250436 /autogroup-6
> [397948.334734] mysqld 4102 1154978.441697 959988
> 120 1154978.441697 10108.715585 397895341.475090 /autogroup-6
> [397948.334839] mysqld 4103 1154978.440507 1341686
> 120 1154978.440507 16033.050027 397888981.976267 /autogroup-6
> [397948.334945] mysqld 4119 1154990.729705 404501
> 120 1154990.729705 119263.212981 397785618.876052 /autogroup-6
> [397948.335051] mysqld 4150 1039299.577821 7023984
> 120 1039299.577821 1407433.681869 320824575.671761 /autogroup-6
> [397948.335157] mysqld 5811 1154806.870976 1317671
> 120 1154806.870976 299564.211303 346391209.749613 /autogroup-6
> [397948.335264] mysqld 15436 1154807.136306 292526
> 120 1154807.136306 172005.952402 174398934.514010 /autogroup-6
> [397948.335371] getty 4374 0.817181 100
> 120 0.817181 2.613716 146.560614 /autogroup-23
> [397948.335478] btrfs-transacti 6269 96825617.051561 22374147
> 120 96825617.051561 166025164.637843 164906022.709080 /
> [397948.335583] bash 6275 132.300150 276
> 120 132.300150 79.740920 21744924.508065
> /autogroup-184
> [397948.335690] kworker/0:1 21018 96823285.353043 1816351
> 120 96823285.353043 793610.285186 42038707.779192 /
> [397948.335795] ssl-params 22730 472.603845 4
> 120 472.603845 0.877328 0.117947 /autogroup-17
> [397948.335902] kworker/u16:0 22761 96825069.356076 67516
> 120 96825069.356076 540082.349638 226131.772902 /
> [397948.336006] sshd 22771 10.038325 150
> 120 10.038325 13.472329 50175.143940
> /autogroup-1114
> [397948.336112] kworker/0:2 22822 96825605.578025 507
> 120 96825605.578025 47.279505 14055.084212 /
> [397948.336218]
> [397948.336280] cpu#1, 3411.800 MHz
> [397948.336329] .nr_running : 1
> [397948.336379] .load : 1024
> [397948.336430] .nr_switches : 84332707
> [397948.336481] .nr_load_updates : 21967275
> [397948.336533] .nr_uninterruptible : 483742
> [397948.336584] .next_balance : 4334.753129
> [397948.336635] .curr->pid : 22247
> [397948.336686] .clock : 397948333.217376
> [397948.336738] .cpu_load[0] : 1023
> [397948.336788] .cpu_load[1] : 1023
> [397948.336839] .cpu_load[2] : 1023
> [397948.336890] .cpu_load[3] : 1023
> [397948.336940] .cpu_load[4] : 1007
> [397948.336991] .yld_count : 135412
> [397948.337043] .sched_count : 84726950
> [397948.337094] .sched_goidle : 29157952
> [397948.337145] .avg_idle : 1000000
> [397948.337196] .max_idle_balance_cost : 500000
> [397948.337247] .ttwu_count : 36438464
> [397948.337299] .ttwu_local : 19485954
> [397948.337350]
> [397948.337350] cfs_rq[1]:/
> [397948.337441] .exec_clock : 123202243.887995
> [397948.337494] .MIN_vruntime : 0.000001
> [397948.337545] .min_vruntime : 119610642.735572
> [397948.337597] .max_vruntime : 0.000001
> [397948.337649] .spread : 0.000000
> [397948.337700] .spread0 : 22785025.684011
> [397948.337752] .nr_spread_over : 301
> [397948.337802] .nr_running : 1
> [397948.337852] .load : 1024
> [397948.337903] .runnable_load_avg : 1023
> [397948.337954] .blocked_load_avg : 26
> [397948.338004] .tg_load_contrib : 1042
> [397948.338055] .tg_runnable_contrib : 1010
> [397948.338106] .tg_load_avg : 1317
> [397948.338156] .tg->runnable_avg : 1293
> [397948.338206] .tg->cfs_bandwidth.timer_active: 0
> [397948.338256] .throttled : 0
> [397948.338307] .throttle_count : 0
> [397948.338357] .avg->runnable_avg_sum : 46120
> [397948.338408] .avg->runnable_avg_period : 46120
> [397948.338459]
> [397948.338459] rt_rq[1]:/
> [397948.338550] .rt_nr_running : 0
> [397948.338600] .rt_throttled : 0
> [397948.339900] .rt_time : 0.000000
> [397948.339951] .rt_runtime : 950.000000
> [397948.340003]
> [397948.340003] dl_rq[1]:
> [397948.340094] .dl_nr_running : 0
> [397948.340144]
> [397948.340144] runnable tasks:
> [397948.340144] task PID tree-key switches
> prio exec-runtime sum-exec sum-sleep
> [397948.340144] ----------------------------------------------------------------------------------------------------------
> [397948.340438] watchdog/1 11 -2.973570 99543
> 0 -2.973570 21159.974792 85.639910 /
> [397948.340543] migration/1 12 0.000000 37023
> 0 0.000000 30622.595831 0.000758 /
> [397948.340648] ksoftirqd/1 13 119609979.133122 91495
> 120 119609979.133122 1646.457418 397900354.563118 /
> [397948.340752] kworker/1:0H 15 582.372111 6
> 100 582.372111 0.027131 2733.616385 /
> [397948.340858] khungtaskd 204 119577195.763580 3319
> 120 119577195.763580 182.485563 397829939.496656 /
> [397948.340962] kpsmoused 686 25.893051 2
> 100 25.893051 0.010838 0.003636 /
> [397948.341068] kworker/1:2 688 102920251.057920 55
> 120 102920251.057920 0.504998 355091633.648033 /
> [397948.341172] deferwq 731 158.015576 2
> 100 158.015576 0.009076 0.003883 /
> [397948.341277] scsi_tmf_0 856 442.536328 2
> 100 442.536328 0.008945 0.003987 /
> [397948.341382] scsi_eh_1 857 539.461151 16
> 120 539.461151 0.868588 350.508685 /
> [397948.341486] scsi_tmf_1 858 466.554669 2
> 100 466.554669 0.008786 0.003948 /
> [397948.341590] scsi_eh_2 859 514.602593 3
> 120 514.602593 0.016414 1.372299 /
> [397948.341695] scsi_eh_3 861 514.603810 3
> 120 514.603810 0.016837 1.222996 /
> [397948.341799] scsi_tmf_3 862 502.578935 2
> 100 502.578935 0.008052 0.003896 /
> [397948.341904] scsi_tmf_4 864 514.587500 2
> 100 514.587500 0.008905 0.004404 /
> [397948.342009] kworker/1:1H 968 118250346.559508 456123
> 100 118250346.559508 6526.369618 394080278.360060 /
> [397948.342114] cron 3301 777.910417 7708
> 120 777.910417 405.566912 397904071.914015 /autogroup-12
> [397948.342219] sshd 3426 0.515010 28
> 120 0.515010 3.325645 397861007.941426 /autogroup-15
> [397948.342326] mysqld_safe 3464 1418.978306 435
> 120 1418.978306 26.375085 974.715839 /autogroup-6
> [397948.342432] mysqld 4118 690369.357591 400760
> 120 690369.357591 50887.461054 397853256.747730 /autogroup-6
> [397948.342539] getty 4372 0.769188 104
> 120 0.769188 2.833955 146.554963 /autogroup-21
> [397948.342645] btrfs-cleaner 6268 119610036.344925 21815
> 120 119610036.344925 5411.552915 342317734.332486 /
> [397948.342750] btrfs 7611 311971.962451 2055291
> 120 311971.962451 1498599.957865 319025222.198539
> /autogroup-184
> [397948.342856] kworker/1:0 21019 119610630.984203 2134678
> 120 119610630.984203 1030853.178543 41814652.278685 /
> [397948.342969] R kworker/u16:16 22247 119610652.481537 293438
> 120 119610652.481537 2301421.627687 2218308.587596 /
> [397948.343099]
> [397948.343161] cpu#2, 3411.800 MHz
> [397948.343210] .nr_running : 3
> [397948.343259] .load : 2048
> [397948.343310] .nr_switches : 76867052
> [397948.343360] .nr_load_updates : 19847327
> [397948.343411] .nr_uninterruptible : 139945
> [397948.343462] .next_balance : 4334.753139
> [397948.343513] .curr->pid : 22773
> [397948.343563] .clock : 397948336.246820
> [397948.343615] .cpu_load[0] : 275
> [397948.343666] .cpu_load[1] : 212
> [397948.343716] .cpu_load[2] : 127
> [397948.343767] .cpu_load[3] : 70
> [397948.343818] .cpu_load[4] : 50
> [397948.343868] .yld_count : 92399
> [397948.343919] .sched_count : 77151242
> [397948.343970] .sched_goidle : 27487304
> [397948.344021] .avg_idle : 1000000
> [397948.344072] .max_idle_balance_cost : 500000
> [397948.344123] .ttwu_count : 35275362
> [397948.344175] .ttwu_local : 17426867
> [397948.344227]
> [397948.344227] cfs_rq[2]:/autogroup-1115
> [397948.344321] .exec_clock : 77.755926
> [397948.344372] .MIN_vruntime : 0.000001
> [397948.344423] .min_vruntime : 169.689274
> [397948.344475] .max_vruntime : 0.000001
> [397948.344526] .spread : 0.000000
> [397948.344578] .spread0 : -96825447.362287
> [397948.344629] .nr_spread_over : 7
> [397948.344680] .nr_running : 1
> [397948.344730] .load : 1024
> [397948.344781] .runnable_load_avg : 271
> [397948.344831] .blocked_load_avg : 0
> [397948.344882] .tg_load_contrib : 271
> [397948.344932] .tg_runnable_contrib : 271
> [397948.344983] .tg_load_avg : 271
> [397948.345033] .tg->runnable_avg : 271
> [397948.345083] .tg->cfs_bandwidth.timer_active: 0
> [397948.345133] .throttled : 0
> [397948.345183] .throttle_count : 0
> [397948.345234] .se->exec_start : 397948336.246820
> [397948.345286] .se->vruntime : 100334135.344609
> [397948.345338] .se->sum_exec_runtime : 77.755926
> [397948.345389] .se->statistics.wait_start : 0.000000
> [397948.345440] .se->statistics.sleep_start : 0.000000
> [397948.345491] .se->statistics.block_start : 0.000000
> [397948.345542] .se->statistics.sleep_max : 0.000000
> [397948.345593] .se->statistics.block_max : 0.000000
> [397948.345644] .se->statistics.exec_max : 13.202718
> [397948.345694] .se->statistics.slice_max : 0.000000
> [397948.345745] .se->statistics.wait_max : 0.020544
> [397948.345796] .se->statistics.wait_sum : 0.209415
> [397948.345847] .se->statistics.wait_count : 128
> [397948.345898] .se->load.weight : 1024
> [397948.345949] .se->avg.runnable_avg_sum : 12487
> [397948.345999] .se->avg.runnable_avg_period : 47064
> [397948.346051] .se->avg.load_avg_contrib : 275
> [397948.346101] .se->avg.decay_count : 0
> [397948.346152]
> [397948.346152] cfs_rq[2]:/
> [397948.346244] .exec_clock : 102436933.654759
> [397948.346296] .MIN_vruntime : 100334120.416350
> [397948.346348] .min_vruntime : 100334132.312627
> [397948.346401] .max_vruntime : 100334120.416350
> [397948.346453] .spread : 0.000000
> [397948.346504] .spread0 : 3508515.261066
> [397948.346556] .nr_spread_over : 450
> [397948.346606] .nr_running : 2
> [397948.346657] .load : 2048
> [397948.346708] .runnable_load_avg : 275
> [397948.346758] .blocked_load_avg : 0
> [397948.346809] .tg_load_contrib : 275
> [397948.346859] .tg_runnable_contrib : 271
> [397948.346910] .tg_load_avg : 1323
> [397948.346961] .tg->runnable_avg : 1293
> [397948.347011] .tg->cfs_bandwidth.timer_active: 0
> [397948.347061] .throttled : 0
> [397948.347111] .throttle_count : 0
> [397948.347161] .avg->runnable_avg_sum : 12763
> [397948.347212] .avg->runnable_avg_period : 48021
> [397948.347263]
> [397948.347263] rt_rq[2]:/
> [397948.347354] .rt_nr_running : 0
> [397948.347404] .rt_throttled : 0
> [397948.347455] .rt_time : 0.000000
> [397948.347506] .rt_runtime : 950.000000
> [397948.347557]
> [397948.347557] dl_rq[2]:
> [397948.347649] .dl_nr_running : 0
> [397948.347699]
> [397948.347699] runnable tasks:
> [397948.347699] task PID tree-key switches
> prio exec-runtime sum-exec sum-sleep
> [397948.347699] ----------------------------------------------------------------------------------------------------------
> [397948.347992] init 1 509.594135 79809
> 120 509.594135 2619.143889 397945091.854170 /autogroup-2
> [397948.348098] watchdog/2 16 -5.976382 99543
> 0 -5.976382 17476.538367 72.326265 /
> [397948.348202] migration/2 17 0.000000 32592
> 0 0.000000 29450.114444 0.001139 /
> [397948.348306] ksoftirqd/2 18 100333573.328598 69839
> 120 100333573.328598 1929.169352 397913601.175468 /
> [397948.348410] kworker/2:0 19 49262954.665776 38
> 120 49262954.665776 0.122608 231013304.220524 /
> [397948.348514] kworker/2:0H 20 3650.646229 6
> 100 3650.646229 0.037533 34187.208217 /
> [397948.348619] scsi_eh_4 863 3072.735732 3
> 120 3072.735732 0.015446 1.048287 /
> [397948.348724] scsi_eh_5 865 3072.733721 3
> 120 3072.733721 0.012919 0.910030 /
> [397948.348828] md1_raid1 910 100169530.338537 1429
> 120 100169530.338537 66.597411 397503924.718962 /
> [397948.348932] kworker/2:1H 1466 100323518.282583 394863
> 100 100323518.282583 5278.861299 397866015.984528 /
> [397948.349036] rsyslogd 3033 428725.147921 28
> 120 428725.147921 2.679249 322287744.247447 /autogroup-6
> [397948.350392] atd 3189 0.622217 112
> 120 0.622217 2.344474 395791202.210800 /autogroup-9
> [397948.350498] acpid 3240 0.478710 41
> 120 0.478710 1.730662 321163512.935538 /autogroup-11
> [397948.350605] dbus-daemon 3404 0.942824 6
> 120 0.942824 1.484584 55885009.390813 /autogroup-14
> [397948.350710] mysqld 4095 459653.767988 848117
> 120 459653.767988 9524.934466 397895732.816642 /autogroup-6
> [397948.350817] mysqld 15425 459502.220847 261927
> 120 459502.220847 145837.180966 174449080.595390 /autogroup-6
> [397948.350923] qmgr 4370 357.389137 2476
> 120 357.389137 280.199510 397622564.351466 /autogroup-19
> [397948.351029] getty 4373 0.640858 102
> 120 0.640858 2.425952 146.827128 /autogroup-22
> [397948.351135] btrfs-worker-hi 6248 1154879.053331 2
> 100 1154879.053331 0.017567 0.003220 /
> [397948.351240] btrfs-delalloc 6249 1154891.063212 2
> 100 1154891.063212 0.010622 0.003413 /
> [397948.351345] btrfs-flush_del 6250 1154903.071666 2
> 100 1154903.071666 0.009022 0.003320 /
> [397948.351449] btrfs-cache 6251 1154915.081079 2
> 100 1154915.081079 0.015404 0.003203 /
> [397948.351554] btrfs-submit 6252 1154915.162054 2
> 100 1154915.162054 0.096237 0.023172 /
> [397948.351659] btrfs-endio-met 6256 1154927.254127 2
> 100 1154927.254127 0.093435 0.002869 /
> [397948.351763] btrfs-rmw 6259 1154939.263666 2
> 100 1154939.263666 0.010258 0.003176 /
> [397948.351868] btrfs-freespace 6261 1154951.272823 2
> 100 1154951.272823 0.009779 0.003215 /
> [397948.351972] btrfs-delayed-m 6262 1154963.281556 2
> 100 1154963.281556 0.009338 0.003128 /
> [397948.352076] btrfs-qgroup-re 6264 1154975.290187 2
> 100 1154975.290187 0.009265 0.003113 /
> [397948.352180] kworker/2:2 16094 100334120.416350 6973086
> 120 100334120.416350 3053548.894806 163847153.385628 /
> [397948.352285] pickup 22744 369.431995 35
> 120 369.431995 4.254482 1980726.020374 /autogroup-19
> [397948.352391] R bash 22773 169.689274 106
> 120 169.689274 49.563444 46410.281275
> /autogroup-1115
> [397948.352497]
> [397948.352543] cpu#3, 3411.800 MHz
> [397948.352591] .nr_running : 0
> [397948.352641] .load : 0
> [397948.352692] .nr_switches : 78216804
> [397948.352743] .nr_load_updates : 20561275
> [397948.352794] .nr_uninterruptible : 84689
> [397948.352845] .next_balance : 4334.753130
> [397948.352896] .curr->pid : 0
> [397948.352955] .clock : 397948343.119396
> [397948.353033] .cpu_load[0] : 0
> [397948.353100] .cpu_load[1] : 0
> [397948.353150] .cpu_load[2] : 0
> [397948.353200] .cpu_load[3] : 0
> [397948.353251] .cpu_load[4] : 0
> [397948.353300] .yld_count : 73921
> [397948.353351] .sched_count : 78424464
> [397948.353402] .sched_goidle : 27817319
> [397948.353453] .avg_idle : 1000000
> [397948.353504] .max_idle_balance_cost : 500000
> [397948.353555] .ttwu_count : 36987465
> [397948.353605] .ttwu_local : 18034474
> [397948.353656]
> [397948.353656] cfs_rq[3]:/autogroup-6
> [397948.353751] .exec_clock : 484988.006368
> [397948.353802] .MIN_vruntime : 0.000001
> [397948.353854] .min_vruntime : 404786.905985
> [397948.353905] .max_vruntime : 0.000001
> [397948.353956] .spread : 0.000000
> [397948.354007] .spread0 : -96420830.145576
> [397948.354058] .nr_spread_over : 245
> [397948.354109] .nr_running : 0
> [397948.354159] .load : 0
> [397948.354209] .runnable_load_avg : 0
> [397948.354259] .blocked_load_avg : 4
> [397948.354310] .tg_load_contrib : 4
> [397948.354360] .tg_runnable_contrib : 5
> [397948.354409] .tg_load_avg : 4
> [397948.354460] .tg->runnable_avg : 7
> [397948.354510] .tg->cfs_bandwidth.timer_active: 0
> [397948.354560] .throttled : 0
> [397948.354610] .throttle_count : 0
> [397948.354661] .se->exec_start : 397948323.578248
> [397948.354713] .se->vruntime : 102939999.703983
> [397948.354766] .se->sum_exec_runtime : 485015.851159
> [397948.354817] .se->statistics.wait_start : 0.000000
> [397948.354868] .se->statistics.sleep_start : 0.000000
> [397948.354919] .se->statistics.block_start : 0.000000
> [397948.354970] .se->statistics.sleep_max : 0.000000
> [397948.355021] .se->statistics.block_max : 0.000000
> [397948.355072] .se->statistics.exec_max : 9.999422
> [397948.355123] .se->statistics.slice_max : 9.991785
> [397948.355174] .se->statistics.wait_max : 19.987876
> [397948.355225] .se->statistics.wait_sum : 7370.667707
> [397948.355277] .se->statistics.wait_count : 3844653
> [397948.355327] .se->load.weight : 2
> [397948.355378] .se->avg.runnable_avg_sum : 266
> [397948.355428] .se->avg.runnable_avg_period : 47445
> [397948.355479] .se->avg.load_avg_contrib : 5
> [397948.355529] .se->avg.decay_count : 379513096
> [397948.355581]
> [397948.355581] cfs_rq[3]:/
> [397948.355673] .exec_clock : 105213301.964322
> [397948.355725] .MIN_vruntime : 0.000001
> [397948.355776] .min_vruntime : 102940010.804921
> [397948.355828] .max_vruntime : 0.000001
> [397948.355879] .spread : 0.000000
> [397948.355930] .spread0 : 6114393.753360
> [397948.355982] .nr_spread_over : 326
> [397948.356033] .nr_running : 0
> [397948.356083] .load : 0
> [397948.356133] .runnable_load_avg : 0
> [397948.356183] .blocked_load_avg : 0
> [397948.356234] .tg_load_contrib : 0
> [397948.356284] .tg_runnable_contrib : 6
> [397948.356335] .tg_load_avg : 1319
> [397948.356385] .tg->runnable_avg : 1291
> [397948.356436] .tg->cfs_bandwidth.timer_active: 0
> [397948.356486] .throttled : 0
> [397948.356537] .throttle_count : 0
> [397948.356587] .avg->runnable_avg_sum : 307
> [397948.356637] .avg->runnable_avg_period : 46706
> [397948.356688]
> [397948.356688] rt_rq[3]:/
> [397948.356781] .rt_nr_running : 0
> [397948.356831] .rt_throttled : 0
> [397948.356882] .rt_time : 0.000000
> [397948.356933] .rt_runtime : 950.000000
> [397948.356984]
> [397948.356984] dl_rq[3]:
> [397948.357076] .dl_nr_running : 0
> [397948.357127]
> [397948.357127] runnable tasks:
> [397948.357127] task PID tree-key switches
> prio exec-runtime sum-exec sum-sleep
> [397948.357127] ----------------------------------------------------------------------------------------------------------
> [397948.357422] kthreadd 2 102939126.157704 1315
> 120 102939126.157704 38.637725 397940297.121677 /
> [397948.357527] watchdog/3 21 -5.984614 99544
> 0 -5.984614 18147.844668 58.998123 /
> [397948.357632] migration/3 22 0.000000 32291
> 0 0.000000 32254.090833 0.000847 /
> [397948.357737] ksoftirqd/3 23 102939780.572027 62304
> 120 102939780.572027 1739.164572 397906895.811056 /
> [397948.357841] kworker/3:0 24 33082955.717338 56
> 120 33082955.717338 0.596786 176596062.108639 /
> [397948.357946] kworker/3:0H 25 1600.396479 6
> 100 1600.396479 0.027776 33504.961049 /
> [397948.358051] khelper 46 10.955864 2
> 100 10.955864 0.005052 0.001801 /
> [397948.358156] writeback 205 311.132001 2
> 100 311.132001 0.003788 0.001292 /
> [397948.358261] crypto 208 323.135116 2
> 100 323.135116 0.003395 0.001548 /
> [397948.358365] bioset 209 335.138067 2
> 100 335.138067 0.003268 0.001389 /
> [397948.358470] kblockd 210 347.141115 2
> 100 347.141115 0.003369 0.001350 /
> [397948.358574] md 361 359.144676 2
> 100 359.144676 0.003908 0.001863 /
> [397948.358679] fsnotify_mark 576 1285553.304174 56
> 120 1285553.304174 1.356477 77382968.604008 /
> [397948.358784] bioset 909 1410.076226 2
> 100 1410.076226 0.008348 0.004053 /
> [397948.358888] md2_raid1 920 102939126.258310 5305156
> 120 102939126.258310 235266.004521 397697550.934379 /
> [397948.358994] kworker/3:1H 1233 101531219.068494 383437
> 100 101531219.068494 4828.918177 393998408.709581 /
> [397948.359098] rs:main Q:Reg 3060 404775.410651 3733
> 120 404775.410651 584.098973 397909253.918002 /autogroup-6
> [397948.359205] vnstatd 3041 1938.625022 80791
> 120 1938.625022 7728.375777 397899296.874075 /autogroup-7
> [397948.359311] mdadm 3144 21.070081 419
> 120 21.070081 64.059078 397466901.661042 /autogroup-8
> [397948.359417] nginx 3232 4286.796414 20
> 120 4286.796414 1.473683 322285293.302498 /autogroup-10
> [397948.359524] nginx 3233 5123.741553 498857
> 120 5123.741553 6408.084381 397902171.919015 /autogroup-10
> [397948.359631] nginx 3234 5123.781505 795481
> 120 5123.781505 10575.489835 397897949.295633 /autogroup-10
> [397948.360989] nginx 3236 5123.792442 453567
> 120 5123.792442 6288.632023 397902307.034538 /autogroup-10
> [397948.361094] ntpd 3378 3177.303496 399798
> 120 3177.303496 11384.417725 397896567.168923 /autogroup-13
> [397948.361201] mysqld 4096 404775.038669 1280416
> 120 404775.038669 56190.329409 397846935.062842 /autogroup-6
> [397948.361306] mysqld 4099 404775.037536 1237509
> 120 404775.037536 53709.818619 397849497.344214 /autogroup-6
> [397948.361413] mysqld 4100 404775.128644 1410134
> 120 404775.128644 17179.021072 397887821.391458 /autogroup-6
> [397948.361520] mysqld 4101 404775.136626 1023118
> 120 404775.136626 11032.276249 397894304.441429 /autogroup-6
> [397948.361625] mysqld 4120 404774.916067 79647
> 120 404774.916067 1272.295740 397899969.656177 /autogroup-6
> [397948.361732] mysqld 4121 404774.944173 79565
> 120 404774.944173 1158.311382 397903499.069766 /autogroup-6
> [397948.361838] mysqld 4123 260297.226173 260619
> 120 260297.226173 48020.432086 156765824.444851 /autogroup-6
> [397948.361944] mysqld 4154 404597.326324 2178558
> 120 404597.326324 373081.589987 395136259.717365 /autogroup-6
> [397948.362049] mysqld 4393 404597.164623 1651064
> 120 404597.164623 338373.325108 395122973.929693 /autogroup-6
> [397948.362156] mysqld 4396 404771.119995 1079825
> 120 404771.119995 305191.529334 397422077.680807 /autogroup-6
> [397948.362261] mysqld 15426 404596.822357 784917
> 120 404596.822357 137130.712105 174453678.251801 /autogroup-6
> [397948.362367] mysqld 15435 404596.986208 228061
> 120 404596.986208 118651.907457 174461622.198681 /autogroup-6
> [397948.362473] dovecot 4220 409.745461 1991
> 120 409.745461 113.183818 397150716.515321 /autogroup-17
> [397948.362579] master 4345 415.904754 20039
> 120 415.904754 991.626364 397859137.298652 /autogroup-19
> [397948.362684] getty 4375 0.839336 103
> 120 0.839336 1.796678 147.205983 /autogroup-24
> [397948.362791] tmux 6274 11456.276708 987382
> 120 11456.276708 43704.262889 342265075.730957
> /autogroup-183
> [397948.362897] kworker/3:1 13622 102939999.914193 9663630
> 120 102939999.914193 4235651.348825 217069990.063754 /
> [397948.363034] kworker/u16:2 22232 102939126.098258 163122
> 120 102939126.098258 1130115.311569 3704906.322727 /
> [397948.363138] kworker/u17:0 22250 100814026.249739 4
> 100 100814026.249739 0.109630 5.440267 /
> [397948.363243]
> [397948.363304] cpu#4, 3411.800 MHz
> [397948.363352] .nr_running : 0
> [397948.363403] .load : 0
> [397948.363452] .nr_switches : 31446353
> [397948.363503] .nr_load_updates : 3466700
> [397948.363554] .nr_uninterruptible : 12378
> [397948.363605] .next_balance : 4334.753132
> [397948.363656] .curr->pid : 0
> [397948.363706] .clock : 397948363.266319
> [397948.363758] .cpu_load[0] : 0
> [397948.363809] .cpu_load[1] : 0
> [397948.363858] .cpu_load[2] : 0
> [397948.363909] .cpu_load[3] : 0
> [397948.363958] .cpu_load[4] : 0
> [397948.364009] .yld_count : 30496
> [397948.364060] .sched_count : 31745499
> [397948.364111] .sched_goidle : 13288472
> [397948.364162] .avg_idle : 1000000
> [397948.364213] .max_idle_balance_cost : 500000
> [397948.364264] .ttwu_count : 12079122
> [397948.364315] .ttwu_local : 3235209
> [397948.364366]
> [397948.364366] cfs_rq[4]:/
> [397948.364458] .exec_clock : 18189528.762014
> [397948.364510] .MIN_vruntime : 0.000001
> [397948.364561] .min_vruntime : 22358073.568236
> [397948.364613] .max_vruntime : 0.000001
> [397948.364664] .spread : 0.000000
> [397948.364715] .spread0 : -74467543.483325
> [397948.364768] .nr_spread_over : 290
> [397948.364818] .nr_running : 0
> [397948.364868] .load : 0
> [397948.364919] .runnable_load_avg : 0
> [397948.364968] .blocked_load_avg : 0
> [397948.365018] .tg_load_contrib : 0
> [397948.365069] .tg_runnable_contrib : 0
> [397948.365119] .tg_load_avg : 1315
> [397948.365170] .tg->runnable_avg : 1290
> [397948.365220] .tg->cfs_bandwidth.timer_active: 0
> [397948.365271] .throttled : 0
> [397948.365321] .throttle_count : 0
> [397948.365371] .avg->runnable_avg_sum : 0
> [397948.365421] .avg->runnable_avg_period : 48326
> [397948.365472]
> [397948.365472] rt_rq[4]:/
> [397948.365564] .rt_nr_running : 0
> [397948.365614] .rt_throttled : 0
> [397948.365664] .rt_time : 0.000000
> [397948.365715] .rt_runtime : 950.000000
> [397948.365766]
> [397948.365766] dl_rq[4]:
> [397948.365858] .dl_nr_running : 0
> [397948.365909]
> [397948.365909] runnable tasks:
> [397948.365909] task PID tree-key switches
> prio exec-runtime sum-exec sum-sleep
> [397948.365909] ----------------------------------------------------------------------------------------------------------
> [397948.366203] watchdog/4 26 -8.974112 99544
> 0 -8.974112 1925.403645 45.684752 /
> [397948.366307] migration/4 27 0.000000 32687
> 0 0.000000 19510.524185 0.000834 /
> [397948.366412] ksoftirqd/4 28 22357044.685798 13474
> 120 22357044.685798 152.288196 397896961.880208 /
> [397948.366516] kworker/4:0 29 1190565.109457 20
> 120 1190565.109457 0.048792 84471152.489397 /
> [397948.366621] kworker/4:0H 30 1300.883814 6
> 100 1300.883814 0.042505 33602.235307 /
> [397948.366726] irq/28-mei_me 673 0.000000 14
> 49 0.000000 0.320285 0.000000 /
> [397948.366831] kworker/4:1H 1287 22130377.005493 40420
> 100 22130377.005493 571.072101 394074352.751479 /
> [397948.366937] mysqld 4148 104218.340695 6
> 120 104218.340695 0.235452 160281236.866385 /autogroup-6
> [397948.367043] anvil 4282 508.617446 540
> 120 508.617446 25.108427 395113557.223988 /autogroup-17
> [397948.367148] getty 4371 3.299146 103
> 120 3.299146 4.682491 144.819918 /autogroup-20
> [397948.367255] kworker/4:2 9937 22358061.586618 2184600
> 120 22358061.586618 951220.510870 312483748.018300 /
> [397948.367360] kworker/u17:6 21267 21365561.424208 10
> 100 21365561.424208 2.594598 27216695.208805 /
> [397948.367465] kworker/u16:1 22824 22358073.568236 4
> 120 22358073.568236 0.080475 4901.819753 /
> [397948.367569]
> [397948.367615] cpu#5, 3411.800 MHz
> [397948.367664] .nr_running : 0
> [397948.367714] .load : 0
> [397948.367764] .nr_switches : 29264631
> [397948.367816] .nr_load_updates : 3974073
> [397948.367867] .nr_uninterruptible : 5752
> [397948.367917] .next_balance : 4334.753132
> [397948.367969] .curr->pid : 0
> [397948.368020] .clock : 397948363.266965
> [397948.368072] .cpu_load[0] : 0
> [397948.368122] .cpu_load[1] : 0
> [397948.368172] .cpu_load[2] : 0
> [397948.368222] .cpu_load[3] : 0
> [397948.368272] .cpu_load[4] : 0
> [397948.368323] .yld_count : 23948
> [397948.368373] .sched_count : 29418990
> [397948.368424] .sched_goidle : 12116624
> [397948.368475] .avg_idle : 1000000
> [397948.368526] .max_idle_balance_cost : 500000
> [397948.368577] .ttwu_count : 14464828
> [397948.368628] .ttwu_local : 3286532
> [397948.368679]
> [397948.368679] cfs_rq[5]:/
> [397948.368771] .exec_clock : 21096887.422727
> [397948.368823] .MIN_vruntime : 0.000001
> [397948.368873] .min_vruntime : 24070921.092851
> [397948.368926] .max_vruntime : 0.000001
> [397948.368977] .spread : 0.000000
> [397948.369028] .spread0 : -72754695.958710
> [397948.369080] .nr_spread_over : 219
> [397948.369131] .nr_running : 0
> [397948.369181] .load : 0
> [397948.369232] .runnable_load_avg : 0
> [397948.369282] .blocked_load_avg : 0
> [397948.369332] .tg_load_contrib : 0
> [397948.369382] .tg_runnable_contrib : 0
> [397948.369432] .tg_load_avg : 1315
> [397948.369483] .tg->runnable_avg : 1290
> [397948.369534] .tg->cfs_bandwidth.timer_active: 0
> [397948.369584] .throttled : 0
> [397948.369634] .throttle_count : 0
> [397948.369685] .avg->runnable_avg_sum : 0
> [397948.369735] .avg->runnable_avg_period : 47521
> [397948.369786]
> [397948.369786] rt_rq[5]:/
> [397948.369878] .rt_nr_running : 0
> [397948.369928] .rt_throttled : 0
> [397948.369979] .rt_time : 0.000000
> [397948.370030] .rt_runtime : 950.000000
> [397948.370081]
> [397948.370081] dl_rq[5]:
> [397948.371423] .dl_nr_running : 0
> [397948.371474]
> [397948.371474] runnable tasks:
> [397948.371474] task PID tree-key switches
> prio exec-runtime sum-exec sum-sleep
> [397948.371474] ----------------------------------------------------------------------------------------------------------
> [397948.371769] watchdog/5 31 -8.977217 99543
> 0 -8.977217 2363.156345 32.364766 /
> [397948.371873] migration/5 32 0.000000 25712
> 0 0.000000 15366.174996 0.000845 /
> [397948.371977] ksoftirqd/5 33 24070879.321423 15403
> 120 24070879.321423 155.107046 397934935.415490 /
> [397948.372082] kworker/5:0 34 239.267711 17
> 120 239.267711 0.034507 632.432629 /
> [397948.372186] kworker/5:0H 35 632.111736 6
> 100 632.111736 0.059549 2815.362585 /
> [397948.372291] kworker/5:1 611 24070909.178313 2597656
> 120 24070909.178313 1203696.900841 396713444.701918 /
> [397948.372396] scsi_tmf_2 860 534.239404 2
> 100 534.239404 0.008896 0.005460 /
> [397948.372500] scsi_tmf_5 866 558.255161 2
> 100 558.255161 0.008179 0.004413 /
> [397948.372605] kworker/5:1H 1061 23860072.399440 45682
> 100 23860072.399440 510.190883 394083026.021665 /
> [397948.372711]
> [397948.372757] cpu#6, 3411.800 MHz
> [397948.372805] .nr_running : 0
> [397948.372855] .load : 0
> [397948.372905] .nr_switches : 30595250
> [397948.372972] .nr_load_updates : 4632598
> [397948.373039] .nr_uninterruptible : 6640
> [397948.373089] .next_balance : 4334.753132
> [397948.373140] .curr->pid : 0
> [397948.373191] .clock : 397948363.267655
> [397948.373243] .cpu_load[0] : 0
> [397948.373292] .cpu_load[1] : 0
> [397948.373342] .cpu_load[2] : 0
> [397948.373393] .cpu_load[3] : 0
> [397948.373443] .cpu_load[4] : 0
> [397948.373494] .yld_count : 20767
> [397948.373545] .sched_count : 30739543
> [397948.373596] .sched_goidle : 12161262
> [397948.373647] .avg_idle : 1000000
> [397948.373698] .max_idle_balance_cost : 500000
> [397948.373749] .ttwu_count : 17363747
> [397948.373800] .ttwu_local : 4024881
> [397948.373852]
> [397948.373852] cfs_rq[6]:/
> [397948.373944] .exec_clock : 27066813.057347
> [397948.373996] .MIN_vruntime : 0.000001
> [397948.374046] .min_vruntime : 29909549.644335
> [397948.374099] .max_vruntime : 0.000001
> [397948.374150] .spread : 0.000000
> [397948.374202] .spread0 : -66916067.407226
> [397948.374253] .nr_spread_over : 407
> [397948.374304] .nr_running : 0
> [397948.374354] .load : 0
> [397948.374404] .runnable_load_avg : 0
> [397948.374454] .blocked_load_avg : 0
> [397948.374504] .tg_load_contrib : 0
> [397948.374554] .tg_runnable_contrib : 0
> [397948.374605] .tg_load_avg : 1313
> [397948.374656] .tg->runnable_avg : 1289
> [397948.374706] .tg->cfs_bandwidth.timer_active: 0
> [397948.374757] .throttled : 0
> [397948.374807] .throttle_count : 0
> [397948.374857] .avg->runnable_avg_sum : 0
> [397948.374907] .avg->runnable_avg_period : 47423
> [397948.374958]
> [397948.374958] rt_rq[6]:/
> [397948.375050] .rt_nr_running : 0
> [397948.375099] .rt_throttled : 0
> [397948.375149] .rt_time : 0.000000
> [397948.375200] .rt_runtime : 950.000000
> [397948.375258]
> [397948.375258] dl_rq[6]:
> [397948.375359] .dl_nr_running : 0
> [397948.375410]
> [397948.375410] runnable tasks:
> [397948.375410] task PID tree-key switches
> prio exec-runtime sum-exec sum-sleep
> [397948.375410] ----------------------------------------------------------------------------------------------------------
> [397948.375704] watchdog/6 36 -8.979370 99543
> 0 -8.979370 3873.032642 19.046238 /
> [397948.375809] migration/6 37 0.000000 20867
> 0 0.000000 12947.385988 0.000826 /
> [397948.375914] ksoftirqd/6 38 29909537.744505 17979
> 120 29909537.744505 324.324395 397932684.911680 /
> [397948.376018] kworker/6:0 39 2652.785847 15
> 120 2652.785847 0.034665 619.334482 /
> [397948.376123] kworker/6:0H 40 3222.777674 6
> 100 3222.777674 0.042578 34131.960421 /
> [397948.376228] kworker/6:1 613 29909537.712176 3191586
> 120 29909537.712176 1584243.664849 396333417.817101 /
> [397948.376333] bioset 898 2957.541580 2
> 100 2957.541580 0.008747 0.006389 /
> [397948.376438] kworker/6:1H 1463 29688535.473225 50544
> 100 29688535.473225 612.882638 394061042.264128 /
> [397948.376542] kjournald 2216 3410.060184 2
> 120 3410.060184 0.412389 0.043474 /
> [397948.376646] bioset 2222 3422.126220 2
> 100 3422.126220 0.088585 0.040787 /
> [397948.376750] rsyslogd 3061 163480.988716 3499
> 120 163480.988716 146.745864 397862580.052157 /autogroup-6
> [397948.376856] mysqld 4122 163443.475621 644351
> 120 163443.475621 179019.020540 393898317.914127 /autogroup-6
> [397948.376962] udevd 6137 246.142493 30
> 120 246.142493 6.670412 22056460.101598 /autogroup-4
> [397948.377068] udevd 6138 242.586598 15
> 120 242.586598 3.149366 49194.013301 /autogroup-4
> [397948.377174] btrfs-worker 6246 1051403.790298 2
> 100 1051403.790298 0.011158 0.004647 /
> [397948.377278] btrfs-endio-met 6255 1051427.810805 2
> 100 1051427.810805 0.017787 0.002771 /
> [397948.377383] btrfs-endio-rai 6257 1051439.820346 2
> 100 1051439.820346 0.010462 0.002911 /
> [397948.377487] btrfs-endio-rep 6258 1051451.828334 2
> 100 1051451.828334 0.008652 0.003137 /
> [397948.377591] btrfs-endio-wri 6260 1051463.836060 2
> 100 1051463.836060 0.008410 0.003085 /
> [397948.377695] btrfs-readahead 6263 1051475.843882 2
> 100 1051475.843882 0.008543 0.003153 /
> [397948.377799] btrfs-extent-re 6265 1051487.851432 2
> 100 1051487.851432 0.008154 0.003265 /
> [397948.377904]
> [397948.377950] cpu#7, 3411.800 MHz
> [397948.377998] .nr_running : 0
> [397948.378048] .load : 0
> [397948.378098] .nr_switches : 28501260
> [397948.378150] .nr_load_updates : 3359354
> [397948.378200] .nr_uninterruptible : 4860
> [397948.378251] .next_balance : 4334.753132
> [397948.378302] .curr->pid : 0
> [397948.378351] .clock : 397948363.268406
> [397948.378403] .cpu_load[0] : 0
> [397948.378453] .cpu_load[1] : 0
> [397948.378504] .cpu_load[2] : 0
> [397948.378554] .cpu_load[3] : 0
> [397948.378604] .cpu_load[4] : 0
> [397948.378654] .yld_count : 22779
> [397948.378705] .sched_count : 28647495
> [397948.378757] .sched_goidle : 12446869
> [397948.378808] .avg_idle : 1000000
> [397948.378859] .max_idle_balance_cost : 500000
> [397948.378910] .ttwu_count : 18391339
> [397948.378961] .ttwu_local : 2717640
> [397948.379012]
> [397948.379012] cfs_rq[7]:/
> [397948.379112] .exec_clock : 13565979.621209
> [397948.379173] .MIN_vruntime : 0.000001
> [397948.379225] .min_vruntime : 17262906.632013
> [397948.379276] .max_vruntime : 0.000001
> [397948.379327] .spread : 0.000000
> [397948.379378] .spread0 : -79562710.419548
> [397948.379430] .nr_spread_over : 255
> [397948.379480] .nr_running : 0
> [397948.379530] .load : 0
> [397948.379581] .runnable_load_avg : 0
> [397948.379631] .blocked_load_avg : 0
> [397948.379682] .tg_load_contrib : 0
> [397948.379732] .tg_runnable_contrib : 0
> [397948.379783] .tg_load_avg : 1313
> [397948.379833] .tg->runnable_avg : 1288
> [397948.379884] .tg->cfs_bandwidth.timer_active: 0
> [397948.379933] .throttled : 0
> [397948.379983] .throttle_count : 0
> [397948.380034] .avg->runnable_avg_sum : 5
> [397948.380084] .avg->runnable_avg_period : 47099
> [397948.380135]
> [397948.380135] rt_rq[7]:/
> [397948.380227] .rt_nr_running : 0
> [397948.380277] .rt_throttled : 0
> [397948.380327] .rt_time : 0.000000
> [397948.380379] .rt_runtime : 950.000000
> [397948.380430]
> [397948.380430] dl_rq[7]:
> [397948.380522] .dl_nr_running : 0
> [397948.380572]
> [397948.380572] runnable tasks:
> [397948.380572] task PID tree-key switches
> prio exec-runtime sum-exec sum-sleep
> [397948.380572] ----------------------------------------------------------------------------------------------------------
> [397948.382132] watchdog/7 41 -8.982956 99544
> 0 -8.982956 1585.113591 5.725643 /
> [397948.382236] migration/7 42 0.000000 21048
> 0 0.000000 12757.115970 0.000816 /
> [397948.382341] ksoftirqd/7 43 17254754.851360 11826
> 120 17254754.851360 146.221401 397720187.660046 /
> [397948.382446] kworker/7:0 44 479.385599 15
> 120 479.385599 0.031395 606.240559 /
> [397948.382550] kworker/7:0H 45 1817.499047 6
> 100 1817.499047 0.047704 42727.098840 /
> [397948.382655] netns 47 10.955152 2
> 100 10.955152 0.004076 0.001586 /
> [397948.382759] perf 50 22.958573 2
> 100 22.958573 0.003794 0.001394 /
> [397948.382863] kworker/7:1 615 17262894.767216 1847690
> 120 17262894.767216 746881.932822 397173736.970567 /
> [397948.382992] md0_raid1 899 17253995.722314 32645
> 120 17253995.722314 997.731148 397503108.422867 /
> [397948.383105] bioset 919 1132.474026 2
> 100 1132.474026 0.034373 0.004008 /
> [397948.383210] ext4-rsv-conver 936 1156.590460 2
> 100 1156.590460 0.061309 0.026946 /
> [397948.383315] udevd 1124 405.566270 290
> 120 405.566270 53.586152 77350627.244245 /autogroup-4
> [397948.383421] ipv6_addrconf 3209 1679.353321 2
> 100 1679.353321 0.006452 0.002581 /
> [397948.383527] kworker/7:1H 4116 16792468.563089 52255
> 100 16792468.563089 541.499314 393833366.670026 /
> [397948.383632] log 4283 574.060901 688
> 120 574.060901 34.058909 395113548.562211 /autogroup-17
> [397948.383739] getty 4376 0.726364 103
> 120 0.726364 1.882302 146.996217 /autogroup-25
> [397948.383845] btrfs-fixup 6253 1049025.643969 2
> 100 1049025.643969 0.011376 0.003865 /
> [397948.383950] btrfs-endio 6254 1049037.659031 2
> 100 1049037.659031 0.015618 0.001516 /
> [397948.384056]
>
>
> Tomasz Chmielewski
> http://www.sslrack.com
>
> --
> To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply [flat|nested] 3+ messages in thread
* Re: 3.19 - "unable" to replace a failed drive - 100% CPU usage in kworker and btrfs-transacti
2015-02-16 10:40 ` Liu Bo
@ 2015-02-16 10:51 ` Tomasz Chmielewski
0 siblings, 0 replies; 3+ messages in thread
From: Tomasz Chmielewski @ 2015-02-16 10:51 UTC (permalink / raw)
To: bo.li.liu; +Cc: linux-btrfs
On 2015-02-16 19:40, Liu Bo wrote:
>> PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
>> 6269 root 20 0 0 0 0 R 92.5 0.0 2769:33
>> btrfs-transacti
>> 22247 root 20 0 0 0 0 R 92.5 0.0 42:38.65
>> kworker/u16:16
>
> Can you cat /proc/22247/stack and /proc/6269/stack?
22247 no longer exists - there is a new kworker now:
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
23570 root 20 0 0 0 0 R 93.8 0.0 108:46.78
kworker/u16:15
6269 root 20 0 0 0 0 R 93.5 0.0 3029:41
btrfs-transacti
# cat /proc/22247/stack
cat: /proc/22247/stack: No such file or directory
# cat /proc/6269/stack
[<ffffffffa02a0b0d>] transaction_kthread+0x197/0x1c2 [btrfs]
[<ffffffff81050067>] kthread+0xcd/0xd5
[<ffffffff813c12ac>] ret_from_fork+0x7c/0xb0
[<ffffffffffffffff>] 0xffffffffffffffff
# cat /proc/23570/stack
[<ffffffffffffffff>] 0xffffffffffffffff
In case a new sysrq-w was needed, too:
[428113.007373] SysRq : Show Blocked State
[428113.007427] task PC stack pid father
[428113.007493] btrfs D ffff8802d67ef948 0 7611 6275
0x00000000
[428113.007549] ffff8802d67ef948 ffff8803fe0409d8 ffff88081917e040
00000000000113c0
[428113.007648] 0000000000004000 ffff88081be31810 ffff88081917e040
ffff8802d67ef8b8
[428113.007748] ffffffff8105f3c4 ffff88083fa80000 ffff88083fa513c0
ffff88083fa513c0
[428113.007847] Call Trace:
[428113.007900] [<ffffffff8105f3c4>] ? enqueue_task_fair+0x3e5/0x44f
[428113.007955] [<ffffffff81054ab6>] ? resched_curr+0x45/0x55
[428113.008008] [<ffffffff81055122>] ? check_preempt_curr+0x3e/0x6d
[428113.008062] [<ffffffff81055163>] ? ttwu_do_wakeup+0x12/0x7f
[428113.008115] [<ffffffff8105526e>] ?
ttwu_do_activate.constprop.73+0x57/0x5c
[428113.008172] [<ffffffff813be618>] schedule+0x65/0x67
[428113.008224] [<ffffffff813c03b8>] schedule_timeout+0x26/0x18d
[428113.008277] [<ffffffff81057d1e>] ? wake_up_process+0x30/0x34
[428113.008331] [<ffffffff8104b1a3>] ? wake_up_worker+0x1f/0x21
[428113.008383] [<ffffffff8104b3ea>] ? insert_work+0x87/0x94
[428113.008447] [<ffffffffa02e183d>] ? free_block_list+0x1f/0x34
[btrfs]
[428113.008501] [<ffffffff813bef76>] wait_for_common+0x10d/0x13e
[428113.008554] [<ffffffff81057cdf>] ? try_to_wake_up+0x250/0x250
[428113.008608] [<ffffffff813befbf>] wait_for_completion+0x18/0x1a
[428113.008666] [<ffffffffa028dedc>]
btrfs_async_run_delayed_refs+0xc1/0xe4 [btrfs]
[428113.008770] [<ffffffffa02a3189>]
__btrfs_end_transaction+0x315/0x33b [btrfs]
[428113.008872] [<ffffffffa02a31bd>]
btrfs_end_transaction_throttle+0xe/0x10 [btrfs]
[428113.008977] [<ffffffffa02e5cf1>] relocate_block_group+0x2ad/0x4de
[btrfs]
[428113.009037] [<ffffffffa02e607a>]
btrfs_relocate_block_group+0x158/0x278 [btrfs]
[428113.009141] [<ffffffffa02c37e1>]
btrfs_relocate_chunk.isra.69+0x35/0xa5 [btrfs]
[428113.009244] [<ffffffffa02c41da>] btrfs_shrink_device+0x235/0x408
[btrfs]
[428113.009304] [<ffffffffa02c6ba9>] btrfs_rm_device+0x2a9/0x704
[btrfs]
[428113.009359] [<ffffffff810f5441>] ?
__kmalloc_track_caller+0x40/0x178
[428113.009419] [<ffffffffa02cf7a7>] btrfs_ioctl+0xa9c/0x22f5 [btrfs]
[428113.009473] [<ffffffff8110f142>] ? putname+0x23/0x2c
[428113.009525] [<ffffffff8110f6d3>] ? user_path_at_empty+0x60/0x90
[428113.009579] [<ffffffff811783bf>] ? avc_has_perm+0x2e/0xf7
[428113.009632] [<ffffffff8111171b>] do_vfs_ioctl+0x418/0x460
[428113.009684] [<ffffffff81106745>] ? vfs_stat+0x16/0x18
[428113.009736] [<ffffffff811117b1>] SyS_ioctl+0x4e/0x7d
[428113.009789] [<ffffffff81031c5b>] ? do_page_fault+0xc/0x11
[428113.009841] [<ffffffff813c1352>] system_call_fastpath+0x12/0x17
[428113.009896] Sched Debug Version: v0.11, 3.19.0 #1
[428113.009947] ktime :
428338904.250198
[428113.010001] sched_clk :
428113009.895839
[428113.010054] cpu_clk :
428113009.895853
[428113.010107] jiffies : 4337771187
[428113.010160] sched_clock_stable() : 1
[428113.010212]
[428113.010257] sysctl_sched
[428113.010305] .sysctl_sched_latency : 24.000000
[428113.010358] .sysctl_sched_min_granularity : 3.000000
[428113.010411] .sysctl_sched_wakeup_granularity : 4.000000
[428113.010463] .sysctl_sched_child_runs_first : 0
[428113.010515] .sysctl_sched_features : 11899
[428113.010568] .sysctl_sched_tunable_scaling : 1
(logaritmic)
[428113.010621]
[428113.010667] cpu#0, 3411.800 MHz
[428113.010715] .nr_running : 0
[428113.010766] .load : 0
[428113.010817] .nr_switches : 141930054
[428113.010869] .nr_load_updates : 24939776
[428113.010920] .nr_uninterruptible : -739558
[428113.010972] .next_balance : 4337.771095
[428113.011023] .curr->pid : 0
[428113.011074] .clock : 428113007.169503
[428113.011126] .cpu_load[0] : 0
[428113.011176] .cpu_load[1] : 0
[428113.011227] .cpu_load[2] : 0
[428113.011277] .cpu_load[3] : 0
[428113.011328] .cpu_load[4] : 24
[428113.011379] .yld_count : 223168
[428113.011430] .sched_count : 142283739
[428113.011482] .sched_goidle : 51488452
[428113.011534] .avg_idle : 1000000
[428113.011584] .max_idle_balance_cost : 500000
[428113.011636] .ttwu_count : 88826891
[428113.011687] .ttwu_local : 54557139
[428113.011739]
[428113.011739] cfs_rq[0]:/autogroup-1201
[428113.011833] .exec_clock : 5.761094
[428113.011884] .MIN_vruntime : 0.000001
[428113.011936] .min_vruntime : 4.697019
[428113.011987] .max_vruntime : 0.000001
[428113.012038] .spread : 0.000000
[428113.012089] .spread0 : -106424728.651767
[428113.012141] .nr_spread_over : 0
[428113.012192] .nr_running : 0
[428113.012242] .load : 0
[428113.012292] .runnable_load_avg : 0
[428113.012343] .blocked_load_avg : 0
[428113.012394] .tg_load_contrib : 0
[428113.012445] .tg_runnable_contrib : 0
[428113.012495] .tg_load_avg : 0
[428113.012546] .tg->runnable_avg : 0
[428113.012596] .tg->cfs_bandwidth.timer_active: 0
[428113.012647] .throttled : 0
[428113.012698] .throttle_count : 0
[428113.012748] .se->exec_start : 428113007.164637
[428113.012801] .se->vruntime : 106424721.380976
[428113.012854] .se->sum_exec_runtime : 5.761094
[428113.012905] .se->statistics.wait_start : 0.000000
[428113.012956] .se->statistics.sleep_start : 0.000000
[428113.013007] .se->statistics.block_start : 0.000000
[428113.013059] .se->statistics.sleep_max : 0.000000
[428113.013110] .se->statistics.block_max : 0.000000
[428113.013161] .se->statistics.exec_max : 0.532615
[428113.013212] .se->statistics.slice_max : 0.000000
[428113.013263] .se->statistics.wait_max : 0.000000
[428113.013314] .se->statistics.wait_sum : 0.000000
[428113.013366] .se->statistics.wait_count : 84
[428113.013416] .se->load.weight : 2
[428113.013466] .se->avg.runnable_avg_sum : 31
[428113.013517] .se->avg.runnable_avg_period : 48192
[428113.013569] .se->avg.load_avg_contrib : 0
[428113.013619] .se->avg.decay_count : 408280380
[428113.013671]
[428113.013671] cfs_rq[0]:/
[428113.013764] .exec_clock : 108726735.130963
[428113.013817] .MIN_vruntime : 0.000001
[428113.013868] .min_vruntime : 106424733.348786
[428113.013920] .max_vruntime : 0.000001
[428113.013971] .spread : 0.000000
[428113.014023] .spread0 : 0.000000
[428113.014074] .nr_spread_over : 291
[428113.014124] .nr_running : 0
[428113.014175] .load : 0
[428113.014225] .runnable_load_avg : 0
[428113.014276] .blocked_load_avg : 0
[428113.014326] .tg_load_contrib : 0
[428113.014377] .tg_runnable_contrib : 0
[428113.014428] .tg_load_avg : 2125
[428113.014479] .tg->runnable_avg : 2029
[428113.015797] .tg->cfs_bandwidth.timer_active: 0
[428113.015847] .throttled : 0
[428113.015897] .throttle_count : 0
[428113.015948] .avg->runnable_avg_sum : 34
[428113.015999] .avg->runnable_avg_period : 48420
[428113.016052]
[428113.016052] rt_rq[0]:/
[428113.016145] .rt_nr_running : 0
[428113.016195] .rt_throttled : 0
[428113.016246] .rt_time : 0.000000
[428113.016297] .rt_runtime : 950.000000
[428113.016349]
[428113.016349] dl_rq[0]:
[428113.016441] .dl_nr_running : 0
[428113.016491]
[428113.016491] runnable tasks:
[428113.016491] task PID tree-key switches prio
exec-runtime sum-exec sum-sleep
[428113.016491]
----------------------------------------------------------------------------------------------------------
[428113.016788] ksoftirqd/0 3 106424458.464162 754529 120
106424458.464162 12412.024323 428020900.243547 /
[428113.016893] kworker/0:0 4 96872056.230102 54 120
96872056.230102 17.312081 398174954.738190 /
[428113.016998] kworker/0:0H 5 1015.287609 5 100
1015.287609 0.041666 1136.081722 /
[428113.017103] rcu_bh 8 117.354314 2 120
117.354314 0.000874 0.000722 /
[428113.017209] migration/0 9 0.000000 42259 0
0.000000 38663.891221 0.000853 /
[428113.017314] watchdog/0 10 -11.786179 107094 0
-11.786179 18301.767349 93.369695 /
[428113.017421] acpi_thermal_pm 618 478.127841 2 100
478.127841 0.008079 0.004241 /
[428113.017527] ata_sff 832 943.140921 2 100
943.140921 0.005880 0.022278 /
[428113.017632] scsi_eh_0 855 1012.563036 18 120
1012.563036 0.875745 352.505402 /
[428113.017738] kworker/0:1H 879 106418747.237080 19756283 100
106418747.237080 936534.886602 427013679.873055 /
[428113.017843] md2_raid1 920 106418747.384047 5312046 120
106418747.384047 235466.210537 427856341.930987 /
[428113.017950] jbd2/md2-8 935 106418747.368760 676645 120
106418747.368760 26050.167995 428067576.860650 /
[428113.018055] rs:main Q:Reg 3060 1156937.947188 3986 120
1156937.947188 610.549422 427956529.353145 /autogroup-6
[428113.018162] rsyslogd 3062 1154980.137945 580 120
1154980.137945 33.203062 397909885.430577 /autogroup-6
[428113.018269] vnstatd 3041 1704.807054 86943 120
1704.807054 8238.508382 428061424.035922 /autogroup-7
[428113.018377] mdadm 3144 27.184928 450 120
27.184928 68.147482 427453633.845909 /autogroup-8
[428113.018483] atd 3189 0.694263 120 120
0.694263 2.561413 424576013.244002 /autogroup-9
[428113.018591] nginx 3232 4317.191565 21 120
4317.191565 1.526614 408639452.655116 /autogroup-10
[428113.018697] nginx 3233 4489.469136 520373 120
4489.469136 6666.843000 427177069.798260 /autogroup-10
[428113.018804] mysqld 4080 1156391.999838 5706 120
1156391.999838 804.810183 419298520.527262 /autogroup-6
[428113.018911] mysqld 4121 1156945.409829 85595 120
1156945.409829 1240.826037 428066936.224975 /autogroup-6
[428113.019017] mysqld 4154 1156391.428236 2178575 120
1156391.428236 373082.741411 418907550.518894 /autogroup-6
[428113.019124] mysqld 4393 1156391.413224 1651105 120
1156391.413224 338375.528086 418894108.235915 /autogroup-6
[428113.019230] mysqld 5811 1156391.734934 1317689 120
1156391.734934 299565.417020 370162247.292961 /autogroup-6
[428113.019337] mysqld 15425 1156391.434917 261945 120
1156391.434917 145838.371509 198220505.720438 /autogroup-6
[428113.019443] mysqld 15436 1156391.985920 292544 120
1156391.985920 172007.149326 198170039.192913 /autogroup-6
[428113.019550] getty 4374 0.817181 100 120
0.817181 2.613716 146.560614 /autogroup-23
[428113.019657] bash 6275 132.300150 276 120
132.300150 79.740920 21744924.508065 /autogroup-184
[428113.019764] kworker/0:2 22822 106424721.355311 1310427 120
106424721.355311 592055.669533 29583164.785882 /
[428113.019869] sshd 23685 4.429642 75 120
4.429642 10.486741 908580.581156 /autogroup-1187
[428113.019976] bash 23687 63.590364 35 120
63.590364 25.369673 13.977239 /autogroup-1188
[428113.020083] sshd 23875 4.697019 109 120
4.697019 10.704193 119911.149251 /autogroup-1201
[428113.020199]
[428113.020246] cpu#1, 3411.800 MHz
[428113.020302] .nr_running : 1
[428113.020354] .load : 1024
[428113.020407] .nr_switches : 88720888
[428113.020461] .nr_load_updates : 23703925
[428113.020515] .nr_uninterruptible : 483951
[428113.020568] .next_balance : 4337.771189
[428113.020621] .curr->pid : 3060
[428113.020675] .clock : 428113020.257470
[428113.020727] .cpu_load[0] : 0
[428113.020786] .cpu_load[1] : 0
[428113.020836] .cpu_load[2] : 0
[428113.020887] .cpu_load[3] : 0
[428113.020937] .cpu_load[4] : 0
[428113.020988] .yld_count : 135412
[428113.021039] .sched_count : 89125261
[428113.021090] .sched_goidle : 30252689
[428113.021141] .avg_idle : 1000000
[428113.021192] .max_idle_balance_cost : 500000
[428113.021244] .ttwu_count : 38652568
[428113.021295] .ttwu_local : 21074490
[428113.021346]
[428113.021346] cfs_rq[1]:/autogroup-183
[428113.021440] .exec_clock : 10198.070166
[428113.021492] .MIN_vruntime : 0.000001
[428113.021543] .min_vruntime : 10197.021590
[428113.021595] .max_vruntime : 0.000001
[428113.021646] .spread : 0.000000
[428113.021697] .spread0 : -106414536.327196
[428113.021749] .nr_spread_over : 0
[428113.021800] .nr_running : 0
[428113.021850] .load : 0
[428113.021900] .runnable_load_avg : 0
[428113.021951] .blocked_load_avg : 0
[428113.022001] .tg_load_contrib : 0
[428113.022052] .tg_runnable_contrib : 0
[428113.022103] .tg_load_avg : 0
[428113.022154] .tg->runnable_avg : 0
[428113.022204] .tg->cfs_bandwidth.timer_active: 0
[428113.022255] .throttled : 0
[428113.022305] .throttle_count : 0
[428113.022356] .se->exec_start : 428113015.541150
[428113.022409] .se->vruntime : 129995516.110494
[428113.022461] .se->sum_exec_runtime : 10198.307701
[428113.022513] .se->statistics.wait_start : 0.000000
[428113.022564] .se->statistics.sleep_start : 0.000000
[428113.022615] .se->statistics.block_start : 0.000000
[428113.022666] .se->statistics.sleep_max : 0.000000
[428113.022717] .se->statistics.block_max : 0.000000
[428113.022769] .se->statistics.exec_max : 2.939902
[428113.022820] .se->statistics.slice_max : 2.075783
[428113.022871] .se->statistics.wait_max : 5.458038
[428113.022922] .se->statistics.wait_sum : 30.977291
[428113.022973] .se->statistics.wait_count : 234554
[428113.023024] .se->load.weight : 2
[428113.023074] .se->avg.runnable_avg_sum : 34
[428113.023125] .se->avg.runnable_avg_period : 47537
[428113.023176] .se->avg.load_avg_contrib : 0
[428113.023226] .se->avg.decay_count : 408280388
[428113.023278]
[428113.023278] cfs_rq[1]:/autogroup-6
[428113.023372] .exec_clock : 808878.880122
[428113.023424] .MIN_vruntime : 0.000001
[428113.023476] .min_vruntime : 692298.789871
[428113.023528] .max_vruntime : 0.000001
[428113.023579] .spread : 0.000000
[428113.023630] .spread0 : -105732434.558915
[428113.023683] .nr_spread_over : 259
[428113.023734] .nr_running : 0
[428113.023784] .load : 0
[428113.023834] .runnable_load_avg : 0
[428113.023885] .blocked_load_avg : 0
[428113.023935] .tg_load_contrib : 0
[428113.023986] .tg_runnable_contrib : 0
[428113.024036] .tg_load_avg : 0
[428113.024087] .tg->runnable_avg : 5
[428113.024138] .tg->cfs_bandwidth.timer_active: 0
[428113.024188] .throttled : 0
[428113.024239] .throttle_count : 0
[428113.024290] .se->exec_start : 428113020.747637
[428113.024342] .se->vruntime : 129995520.664159
[428113.024395] .se->sum_exec_runtime : 808905.454732
[428113.024446] .se->statistics.wait_start : 0.000000
[428113.024497] .se->statistics.sleep_start : 0.000000
[428113.024548] .se->statistics.block_start : 0.000000
[428113.024600] .se->statistics.sleep_max : 0.000000
[428113.024651] .se->statistics.block_max : 0.000000
[428113.024702] .se->statistics.exec_max : 10.030247
[428113.024754] .se->statistics.slice_max : 10.135075
[428113.024805] .se->statistics.wait_max : 14.316925
[428113.024856] .se->statistics.wait_sum : 6528.008676
[428113.024908] .se->statistics.wait_count : 4854101
[428113.024959] .se->load.weight : 2
[428113.025009] .se->avg.runnable_avg_sum : 489
[428113.025059] .se->avg.runnable_avg_period : 47586
[428113.026377] .se->avg.load_avg_contrib : 0
[428113.026428] .se->avg.decay_count : 408280393
[428113.026479]
[428113.026479] cfs_rq[1]:/
[428113.026571] .exec_clock : 134248386.191002
[428113.026624] .MIN_vruntime : 0.000001
[428113.026676] .min_vruntime : 129995526.942391
[428113.026728] .max_vruntime : 0.000001
[428113.026780] .spread : 0.000000
[428113.026831] .spread0 : 23570793.593605
[428113.026883] .nr_spread_over : 311
[428113.026933] .nr_running : 0
[428113.026984] .load : 0
[428113.027035] .runnable_load_avg : 0
[428113.027085] .blocked_load_avg : 0
[428113.027136] .tg_load_contrib : 0
[428113.027186] .tg_runnable_contrib : 10
[428113.027237] .tg_load_avg : 2353
[428113.027288] .tg->runnable_avg : 2284
[428113.027339] .tg->cfs_bandwidth.timer_active: 0
[428113.027389] .throttled : 0
[428113.027439] .throttle_count : 0
[428113.027490] .avg->runnable_avg_sum : 492
[428113.027540] .avg->runnable_avg_period : 47214
[428113.027591]
[428113.027591] rt_rq[1]:/
[428113.027683] .rt_nr_running : 0
[428113.027733] .rt_throttled : 0
[428113.027783] .rt_time : 0.000000
[428113.027834] .rt_runtime : 950.000000
[428113.027886]
[428113.027886] dl_rq[1]:
[428113.027978] .dl_nr_running : 0
[428113.028029]
[428113.028029] runnable tasks:
[428113.028029] task PID tree-key switches prio
exec-runtime sum-exec sum-sleep
[428113.028029]
----------------------------------------------------------------------------------------------------------
[428113.028324] kthreadd 2 129745695.820738 1414 120
129745695.820738 41.527944 427299434.064971 /
[428113.028430] watchdog/1 11 -2.973570 107088 0
-2.973570 23044.650834 85.639910 /
[428113.028535] migration/1 12 0.000000 38706 0
0.000000 32310.384635 0.000758 /
[428113.028641] ksoftirqd/1 13 129993260.237901 96666 120
129993260.237901 1796.063974 428051863.708966 /
[428113.028746] kworker/1:0H 15 582.372111 6 100
582.372111 0.027131 2733.616385 /
[428113.028852] kpsmoused 686 25.893051 2 100
25.893051 0.010838 0.003636 /
[428113.028957] kworker/1:2 688 102920251.057920 55 120
102920251.057920 0.504998 355091633.648033 /
[428113.029062] deferwq 731 158.015576 2 100
158.015576 0.009076 0.003883 /
[428113.029167] scsi_tmf_0 856 442.536328 2 100
442.536328 0.008945 0.003987 /
[428113.029273] scsi_eh_1 857 539.461151 16 120
539.461151 0.868588 350.508685 /
[428113.029378] scsi_tmf_1 858 466.554669 2 100
466.554669 0.008786 0.003948 /
[428113.029484] scsi_eh_2 859 514.602593 3 120
514.602593 0.016414 1.372299 /
[428113.029589] scsi_eh_3 861 514.603810 3 120
514.603810 0.016837 1.222996 /
[428113.029694] scsi_tmf_3 862 502.578935 2 100
502.578935 0.008052 0.003896 /
[428113.029798] scsi_tmf_4 864 514.587500 2 100
514.587500 0.008905 0.004404 /
[428113.029904] kworker/1:1H 968 129831013.397442 456157 100
129831013.397442 6526.830851 427481927.625721 /
[428113.030009] rsyslogd 3033 691065.336881 29 120
691065.336881 2.741841 408640738.003949 /autogroup-6
[428113.030116] rs:main Q:Reg 3060 692287.280038 3987 120
692287.280038 611.039589 428073924.132108 /autogroup-6
[428113.030223] nginx 3234 4197.283423 855782 120
4197.283423 11301.321120 428062021.947465 /autogroup-10
[428113.030330] nginx 3235 4197.323950 684268 120
4197.323950 9289.505173 428064027.946907 /autogroup-10
[428113.030436] cron 3301 927.592784 8289 120
927.592784 433.794782 428068473.223910 /autogroup-12
[428113.030543] ntpd 3378 2401.387432 430091 120
2401.387432 12164.876249 428060869.176748 /autogroup-13
[428113.030650] mysqld_safe 3464 1418.978306 435 120
1418.978306 26.375085 974.715839 /autogroup-6
[428113.030756] mysqld 4094 692286.802491 856622 120
692286.802491 7997.578025 428062557.688191 /autogroup-6
[428113.030863] mysqld 4095 692286.812618 908471 120
692286.812618 10016.493168 428060177.274007 /autogroup-6
[428113.030969] mysqld 4096 692286.794838 1340772 120
692286.794838 56672.194508 428011328.939086 /autogroup-6
[428113.031075] mysqld 4097 692286.802298 1314182 120
692286.802298 55304.687185 428012714.776737 /autogroup-6
[428113.031182] mysqld 4098 692286.798431 1342580 120
692286.798431 57725.305541 428009725.287820 /autogroup-6
[428113.031289] mysqld 4101 692286.801904 1083476 120
692286.801904 11507.690040 428058770.056113 /autogroup-6
[428113.031395] mysqld 4103 692286.796803 1402041 120
692286.796803 16514.909612 428053422.705042 /autogroup-6
[428113.031501] mysqld 4118 692286.906040 430988 120
692286.906040 54290.532636 428014960.090656 /autogroup-6
[428113.031608] mysqld 4119 692298.789871 434800 120
692298.789871 127525.290540 427941309.545286 /autogroup-6
[428113.031715] mysqld 4120 692285.773389 85683 120
692285.773389 1366.692133 428064373.232093 /autogroup-6
[428113.031822] mysqld 4396 692171.636633 1079842 120
692171.636633 305192.243262 426206373.814546 /autogroup-6
[428113.031930] mysqld 15426 691721.033869 784935 120
691721.033869 137131.954258 198224493.143598 /autogroup-6
[428113.032037] getty 4372 0.769188 104 120
0.769188 2.833955 146.554963 /autogroup-21
[428113.032144] btrfs-delalloc 6249 120544673.387058 4 100
120544673.387058 12.566284 345061022.997020 /
[428113.032250] tmux 6274 10197.021590 1077829 120
10197.021590 47181.472538 372426592.567655 /autogroup-183
[428113.032356] kworker/1:0 21019 129995518.151853 3521622 120
129995518.151853 1682209.369392 71323049.000940 /
[428113.032462] sshd 23755 4.481966 346 120
4.481966 33.939190 1200991.721059 /autogroup-1194
[428113.032569] bash 23757 19.596943 121 120
19.596943 36.036523 1106982.698162 /autogroup-1195
[428113.032676] sshd 23823 0.668354 29 120
0.668354 7.093561 202985.617679 /autogroup-1199
[428113.032785]
[428113.032833] cpu#2, 3411.800 MHz
[428113.032891] .nr_running : 1
[428113.032943] .load : 1024
[428113.032995] .nr_switches : 81057822
[428113.033048] .nr_load_updates : 21474857
[428113.033100] .nr_uninterruptible : 140117
[428113.033154] .next_balance : 4337.771194
[428113.033207] .curr->pid : 23570
[428113.033258] .clock : 428113026.128231
[428113.033319] .cpu_load[0] : 1023
[428113.033370] .cpu_load[1] : 1023
[428113.033420] .cpu_load[2] : 1023
[428113.033471] .cpu_load[3] : 1023
[428113.033521] .cpu_load[4] : 1023
[428113.033572] .yld_count : 92399
[428113.033623] .sched_count : 81351574
[428113.033674] .sched_goidle : 28607226
[428113.033725] .avg_idle : 586733
[428113.033776] .max_idle_balance_cost : 500000
[428113.033827] .ttwu_count : 37353396
[428113.033878] .ttwu_local : 18907401
[428113.033929]
[428113.033929] cfs_rq[2]:/
[428113.034021] .exec_clock : 112244043.135162
[428113.034073] .MIN_vruntime : 0.000001
[428113.034125] .min_vruntime : 109548478.645357
[428113.034177] .max_vruntime : 0.000001
[428113.034228] .spread : 0.000000
[428113.034279] .spread0 : 3123745.296571
[428113.034331] .nr_spread_over : 470
[428113.034382] .nr_running : 1
[428113.034432] .load : 1024
[428113.034483] .runnable_load_avg : 1023
[428113.034533] .blocked_load_avg : 44
[428113.034584] .tg_load_contrib : 1056
[428113.034634] .tg_runnable_contrib : 1010
[428113.034685] .tg_load_avg : 2561
[428113.034736] .tg->runnable_avg : 2477
[428113.034787] .tg->cfs_bandwidth.timer_active: 0
[428113.034837] .throttled : 0
[428113.034888] .throttle_count : 0
[428113.034938] .avg->runnable_avg_sum : 47622
[428113.034990] .avg->runnable_avg_period : 47622
[428113.035041]
[428113.035041] rt_rq[2]:/
[428113.035132] .rt_nr_running : 0
[428113.035183] .rt_throttled : 0
[428113.035233] .rt_time : 0.000000
[428113.035284] .rt_runtime : 950.000000
[428113.035336]
[428113.035336] dl_rq[2]:
[428113.035427] .dl_nr_running : 0
[428113.035478]
[428113.035478] runnable tasks:
[428113.035478] task PID tree-key switches prio
exec-runtime sum-exec sum-sleep
[428113.035478]
----------------------------------------------------------------------------------------------------------
[428113.037044] watchdog/2 16 -5.976382 107088 0
-5.976382 19052.269468 72.326265 /
[428113.037149] migration/2 17 0.000000 34097 0
0.000000 32002.834817 0.001139 /
[428113.037254] ksoftirqd/2 18 109545990.177739 74000 120
109545990.177739 2163.416662 428073696.139572 /
[428113.037359] kworker/2:0 19 100410070.169673 40 120
100410070.169673 0.134368 398174981.123393 /
[428113.037465] kworker/2:0H 20 3650.646229 6 100
3650.646229 0.037533 34187.208217 /
[428113.037571] khungtaskd 204 109533300.730792 3571 120
109533300.730792 195.253551 428053978.079872 /
[428113.037676] kswapd0 506 106048732.107333 11160 120
106048732.107333 5508.370790 416651549.349593 /
[428113.037782] scsi_eh_4 863 3072.735732 3 120
3072.735732 0.015446 1.048287 /
[428113.037887] scsi_eh_5 865 3072.733721 3 120
3072.733721 0.012919 0.910030 /
[428113.037993] kworker/2:1H 1466 108820462.412649 394876 100
108820462.412649 5278.985782 425855655.769615 /
[428113.038098] acpid 3240 0.392530 49 120
0.392530 1.816842 407504916.577611 /autogroup-11
[428113.038205] dbus-daemon 3404 0.942824 6 120
0.942824 1.484584 55885009.390813 /autogroup-14
[428113.038313] mysqld 4150 460431.045860 7023989 120
460431.045860 1407433.937451 407178531.151580 /autogroup-6
[428113.038419] master 4345 398.492132 21611 120
398.492132 1062.087947 428043837.677636 /autogroup-19
[428113.038526] getty 4373 0.640858 102 120
0.640858 2.425952 146.827128 /autogroup-22
[428113.038633] btrfs-worker-hi 6248 1154879.053331 2 100
1154879.053331 0.017567 0.003220 /
[428113.038739] btrfs-flush_del 6250 1154903.071666 2 100
1154903.071666 0.009022 0.003320 /
[428113.038844] btrfs-cache 6251 1154915.081079 2 100
1154915.081079 0.015404 0.003203 /
[428113.038949] btrfs-submit 6252 1154915.162054 2 100
1154915.162054 0.096237 0.023172 /
[428113.039054] btrfs-endio-met 6256 1154927.254127 2 100
1154927.254127 0.093435 0.002869 /
[428113.039160] btrfs-rmw 6259 1154939.263666 2 100
1154939.263666 0.010258 0.003176 /
[428113.039265] btrfs-freespace 6261 1154951.272823 2 100
1154951.272823 0.009779 0.003215 /
[428113.039371] btrfs-delayed-m 6262 1154963.281556 2 100
1154963.281556 0.009338 0.003128 /
[428113.039475] btrfs-qgroup-re 6264 1154975.290187 2 100
1154975.290187 0.009265 0.003113 /
[428113.039580] btrfs 7611 332657.739906 2080440 120
332657.739906 1556211.983521 349088605.012752 /autogroup-184
[428113.039687] kworker/2:1 22833 109548476.671264 1253598 120
109548476.671264 577663.225799 29354960.583584 /
[428113.039793] kworker/u17:0 23543 106131220.757333 22 100
106131220.757333 0.419296 905976.409476 /
[428113.039898] R kworker/u16:15 23570 109548488.151519 847746 120
109548488.151519 6605807.116828 4151195.288717 /
[428113.040006]
[428113.040052] cpu#3, 3411.800 MHz
[428113.040108] .nr_running : 2
[428113.040159] .load : 2048
[428113.040211] .nr_switches : 82481964
[428113.040262] .nr_load_updates : 22241954
[428113.040322] .nr_uninterruptible : 84834
[428113.040374] .next_balance : 4337.771192
[428113.040426] .curr->pid : 23877
[428113.040477] .clock : 428113040.005657
[428113.040529] .cpu_load[0] : 501
[428113.040580] .cpu_load[1] : 385
[428113.040630] .cpu_load[2] : 237
[428113.040681] .cpu_load[3] : 132
[428113.040732] .cpu_load[4] : 70
[428113.040783] .yld_count : 73921
[428113.040834] .sched_count : 82696221
[428113.040885] .sched_goidle : 28975510
[428113.040936] .avg_idle : 1000000
[428113.040988] .max_idle_balance_cost : 500000
[428113.041039] .ttwu_count : 39104815
[428113.041090] .ttwu_local : 19561240
[428113.041141]
[428113.041141] cfs_rq[3]:/autogroup-1202
[428113.041235] .exec_clock : 46.587190
[428113.041287] .MIN_vruntime : 0.000001
[428113.041339] .min_vruntime : 117.495213
[428113.041390] .max_vruntime : 0.000001
[428113.041441] .spread : 0.000000
[428113.041493] .spread0 : -106424615.853573
[428113.041545] .nr_spread_over : 1
[428113.041595] .nr_running : 1
[428113.041645] .load : 1024
[428113.041696] .runnable_load_avg : 501
[428113.041747] .blocked_load_avg : 0
[428113.041797] .tg_load_contrib : 501
[428113.041848] .tg_runnable_contrib : 503
[428113.041899] .tg_load_avg : 501
[428113.041950] .tg->runnable_avg : 503
[428113.042001] .tg->cfs_bandwidth.timer_active: 0
[428113.042052] .throttled : 0
[428113.042102] .throttle_count : 0
[428113.042153] .se->exec_start : 428113040.005657
[428113.042206] .se->vruntime : 112122172.726168
[428113.042258] .se->sum_exec_runtime : 46.587190
[428113.042309] .se->statistics.wait_start : 0.000000
[428113.042361] .se->statistics.sleep_start : 0.000000
[428113.042412] .se->statistics.block_start : 0.000000
[428113.042463] .se->statistics.sleep_max : 0.000000
[428113.042514] .se->statistics.block_max : 0.000000
[428113.042565] .se->statistics.exec_max : 12.895535
[428113.042616] .se->statistics.slice_max : 0.000000
[428113.042667] .se->statistics.wait_max : 0.012418
[428113.042719] .se->statistics.wait_sum : 0.012418
[428113.042770] .se->statistics.wait_count : 32
[428113.042820] .se->load.weight : 1024
[428113.042871] .se->avg.runnable_avg_sum : 23121
[428113.042922] .se->avg.runnable_avg_period : 47024
[428113.042972] .se->avg.load_avg_contrib : 501
[428113.043023] .se->avg.decay_count : 0
[428113.043073]
[428113.043073] cfs_rq[3]:/autogroup-6
[428113.043167] .exec_clock : 489341.658980
[428113.043219] .MIN_vruntime : 0.000001
[428113.043271] .min_vruntime : 406937.010943
[428113.043323] .max_vruntime : 0.000001
[428113.043374] .spread : 0.000000
[428113.043425] .spread0 : -106017796.337843
[428113.043477] .nr_spread_over : 245
[428113.043528] .nr_running : 0
[428113.043578] .load : 0
[428113.043628] .runnable_load_avg : 0
[428113.043679] .blocked_load_avg : 0
[428113.043730] .tg_load_contrib : 0
[428113.043781] .tg_runnable_contrib : 0
[428113.043831] .tg_load_avg : 27
[428113.043882] .tg->runnable_avg : 27
[428113.043933] .tg->cfs_bandwidth.timer_active: 0
[428113.043983] .throttled : 0
[428113.044033] .throttle_count : 0
[428113.044083] .se->exec_start : 428112971.994259
[428113.044135] .se->vruntime : 112122139.240601
[428113.044188] .se->sum_exec_runtime : 489369.503771
[428113.044240] .se->statistics.wait_start : 0.000000
[428113.044292] .se->statistics.sleep_start : 0.000000
[428113.044343] .se->statistics.block_start : 0.000000
[428113.044394] .se->statistics.sleep_max : 0.000000
[428113.044445] .se->statistics.block_max : 0.000000
[428113.044496] .se->statistics.exec_max : 9.999422
[428113.044548] .se->statistics.slice_max : 9.991785
[428113.044599] .se->statistics.wait_max : 19.987876
[428113.044650] .se->statistics.wait_sum : 7372.829518
[428113.044702] .se->statistics.wait_count : 4009170
[428113.044753] .se->load.weight : 2
[428113.044803] .se->avg.runnable_avg_sum : 2
[428113.044853] .se->avg.runnable_avg_period : 47424
[428113.044904] .se->avg.load_avg_contrib : 0
[428113.044955] .se->avg.decay_count : 408280346
[428113.045006]
[428113.045006] cfs_rq[3]:/
[428113.045098] .exec_clock : 114980390.622745
[428113.045151] .MIN_vruntime : 112122139.396351
[428113.045203] .min_vruntime : 112122151.162594
[428113.045256] .max_vruntime : 112122139.396351
[428113.045308] .spread : 0.000000
[428113.045359] .spread0 : 5697417.813808
[428113.045411] .nr_spread_over : 342
[428113.045461] .nr_running : 2
[428113.045511] .load : 2048
[428113.045562] .runnable_load_avg : 501
[428113.045620] .blocked_load_avg : 0
[428113.045682] .tg_load_contrib : 501
[428113.045733] .tg_runnable_contrib : 502
[428113.045784] .tg_load_avg : 2611
[428113.045835] .tg->runnable_avg : 2561
[428113.045891] .tg->cfs_bandwidth.timer_active: 0
[428113.045941] .throttled : 0
[428113.045992] .throttle_count : 0
[428113.046042] .avg->runnable_avg_sum : 23288
[428113.046092] .avg->runnable_avg_period : 47497
[428113.046144]
[428113.046144] rt_rq[3]:/
[428113.047487] .rt_nr_running : 0
[428113.047537] .rt_throttled : 0
[428113.047587] .rt_time : 0.000000
[428113.047639] .rt_runtime : 950.000000
[428113.047690]
[428113.047690] dl_rq[3]:
[428113.047782] .dl_nr_running : 0
[428113.047832]
[428113.047832] runnable tasks:
[428113.047832] task PID tree-key switches prio
exec-runtime sum-exec sum-sleep
[428113.047832]
----------------------------------------------------------------------------------------------------------
[428113.048127] init 1 586.822029 85839 120
586.822029 2768.064315 428108381.243581 /autogroup-2
[428113.048233] watchdog/3 21 -5.984614 107089 0
-5.984614 19753.484076 58.998123 /
[428113.048338] migration/3 22 0.000000 33711 0
0.000000 33100.318561 0.000847 /
[428113.048443] ksoftirqd/3 23 112114355.272976 66314 120
112114355.272976 1782.976803 428053105.661252 /
[428113.048548] kworker/3:0 24 33082955.717338 56 120
33082955.717338 0.596786 176596062.108639 /
[428113.048653] kworker/3:0H 25 1600.396479 6 100
1600.396479 0.027776 33504.961049 /
[428113.048758] khelper 46 10.955864 2 100
10.955864 0.005052 0.001801 /
[428113.048864] writeback 205 311.132001 2 100
311.132001 0.003788 0.001292 /
[428113.048969] crypto 208 323.135116 2 100
323.135116 0.003395 0.001548 /
[428113.049074] bioset 209 335.138067 2 100
335.138067 0.003268 0.001389 /
[428113.049179] kblockd 210 347.141115 2 100
347.141115 0.003369 0.001350 /
[428113.049284] md 361 359.144676 2 100
359.144676 0.003908 0.001863 /
[428113.049389] fsnotify_mark 576 1285553.304174 56 120
1285553.304174 1.356477 77382968.604008 /
[428113.049495] bioset 909 1410.076226 2 100
1410.076226 0.008348 0.004053 /
[428113.049600] kworker/3:1H 1233 109291133.379639 383443 100
109291133.379639 4828.980689 418283340.722864 /
[428113.049706] nginx 3236 5458.045701 513869 120
5458.045701 6979.077801 428066409.223752 /autogroup-10
[428113.049814] sshd 3426 0.942592 37 120
0.942592 4.235482 427955415.003546 /autogroup-15
[428113.049920] mysqld 4099 406925.050358 1297863 120
406925.050358 54242.559211 428013387.543354 /autogroup-6
[428113.050027] mysqld 4100 406925.049535 1470492 120
406925.049535 17668.621861 428052266.284702 /autogroup-6
[428113.050135] mysqld 4123 260297.226173 260619 120
260297.226173 48020.432086 156765824.444851 /autogroup-6
[428113.050242] mysqld 15435 406273.301208 228083 120
406273.301208 118653.052456 198232617.393831 /autogroup-6
[428113.050348] getty 4375 0.839336 103 120
0.839336 1.796678 147.205983 /autogroup-24
[428113.050455] btrfs-cleaner 6268 112107699.597918 22244 120
112107699.597918 5420.971640 372440029.550711 /
[428113.050561] kworker/3:1 13622 112122139.396351 10957597 120
112122139.396351 4812976.705719 246653995.669097 /
[428113.050666] ssl-params 23599 422.267712 4 120
422.267712 0.893866 0.114741 /autogroup-17
[428113.050773] kworker/u16:0 23680 112122139.491207 171869 120
112122139.491207 1346238.717160 2149270.672568 /
[428113.050879] bash 23825 114.365698 37 120
114.365698 25.754806 13.396110 /autogroup-1200
[428113.050986] R bash 23877 117.495213 81 120
117.495213 63.565396 116682.313391 /autogroup-1202
[428113.051095]
[428113.051143] cpu#4, 3411.800 MHz
[428113.051200] .nr_running : 1
[428113.051253] .load : 1024
[428113.051305] .nr_switches : 32118320
[428113.051357] .nr_load_updates : 3723003
[428113.051409] .nr_uninterruptible : 12676
[428113.051461] .next_balance : 4337.771202
[428113.051514] .curr->pid : 6269
[428113.051565] .clock : 428113046.666634
[428113.051617] .cpu_load[0] : 1023
[428113.051668] .cpu_load[1] : 1023
[428113.051718] .cpu_load[2] : 1023
[428113.051769] .cpu_load[3] : 1023
[428113.051820] .cpu_load[4] : 1003
[428113.051879] .yld_count : 30496
[428113.051930] .sched_count : 32427582
[428113.051981] .sched_goidle : 13462549
[428113.052032] .avg_idle : 1000000
[428113.052083] .max_idle_balance_cost : 500000
[428113.052134] .ttwu_count : 12413852
[428113.052185] .ttwu_local : 3463562
[428113.052236]
[428113.052236] cfs_rq[4]:/
[428113.052329] .exec_clock : 19891085.291101
[428113.052381] .MIN_vruntime : 0.000001
[428113.052432] .min_vruntime : 23963851.976267
[428113.052484] .max_vruntime : 0.000001
[428113.052536] .spread : 0.000000
[428113.052587] .spread0 : -82460881.372519
[428113.052639] .nr_spread_over : 307
[428113.052689] .nr_running : 1
[428113.052739] .load : 1024
[428113.052790] .runnable_load_avg : 1023
[428113.052840] .blocked_load_avg : 47
[428113.052891] .tg_load_contrib : 1047
[428113.052942] .tg_runnable_contrib : 1015
[428113.052993] .tg_load_avg : 2738
[428113.053044] .tg->runnable_avg : 2685
[428113.053095] .tg->cfs_bandwidth.timer_active: 0
[428113.053146] .throttled : 0
[428113.053196] .throttle_count : 0
[428113.053247] .avg->runnable_avg_sum : 46889
[428113.053298] .avg->runnable_avg_period : 46889
[428113.053350]
[428113.053350] rt_rq[4]:/
[428113.053442] .rt_nr_running : 0
[428113.053492] .rt_throttled : 0
[428113.053542] .rt_time : 0.000000
[428113.053593] .rt_runtime : 950.000000
[428113.053645]
[428113.053645] dl_rq[4]:
[428113.053736] .dl_nr_running : 0
[428113.053787]
[428113.053787] runnable tasks:
[428113.053787] task PID tree-key switches prio
exec-runtime sum-exec sum-sleep
[428113.053787]
----------------------------------------------------------------------------------------------------------
[428113.054082] watchdog/4 26 -8.974112 107089 0
-8.974112 2059.037229 45.684752 /
[428113.054187] migration/4 27 0.000000 33383 0
0.000000 19839.147564 0.000834 /
[428113.054292] ksoftirqd/4 28 23963277.915795 14197 120
23963277.915795 161.575572 428107147.523261 /
[428113.054397] kworker/4:0 29 1190565.109457 20 120
1190565.109457 0.048792 84471152.489397 /
[428113.054502] kworker/4:0H 30 1300.883814 6 100
1300.883814 0.042505 33602.235307 /
[428113.054608] irq/28-mei_me 673 0.000000 14 49
0.000000 0.320285 0.000000 /
[428113.054714] md1_raid1 910 23938903.837682 1461 120
23938903.837682 66.950526 427490660.626113 /
[428113.054821] kworker/4:1H 1287 22856596.596367 40421 100
22856596.596367 571.074116 407462368.250643 /
[428113.054926] mysqld 4148 104218.340695 6 120
104218.340695 0.235452 160281236.866385 /autogroup-6
[428113.055034] getty 4371 3.299146 103 120
3.299146 4.682491 144.819918 /autogroup-20
[428113.055142] Rbtrfs-transacti 6269 23963851.976267 24693859 120
23963851.976267 181831355.421691 178264635.080484 /
[428113.055247] kworker/4:2 9937 23963841.012506 2387667 120
23963841.012506 1048755.187071 342552425.512330 /
[428113.055353] kworker/u16:2 23684 23924997.103420 4 120
23924997.103420 0.030497 2236660.637946 /
[428113.055458] top 23926 67.900517 110 120
67.900517 88.375149 90189.080325 /autogroup-1195
[428113.055566]
[428113.055619] cpu#5, 3411.800 MHz
[428113.055683] .nr_running : 0
[428113.055735] .load : 0
[428113.055794] .nr_switches : 29826080
[428113.055846] .nr_load_updates : 4205435
[428113.055898] .nr_uninterruptible : 5986
[428113.055959] .next_balance : 4337.771193
[428113.056011] .curr->pid : 0
[428113.056061] .clock : 428113055.637128
[428113.056113] .cpu_load[0] : 0
[428113.056164] .cpu_load[1] : 0
[428113.056215] .cpu_load[2] : 0
[428113.056265] .cpu_load[3] : 0
[428113.056316] .cpu_load[4] : 0
[428113.056367] .yld_count : 23948
[428113.056418] .sched_count : 29988603
[428113.056470] .sched_goidle : 12260277
[428113.056521] .avg_idle : 1000000
[428113.056573] .max_idle_balance_cost : 500000
[428113.056623] .ttwu_count : 14760716
[428113.056675] .ttwu_local : 3492006
[428113.056726]
[428113.056726] cfs_rq[5]:/
[428113.058071] .exec_clock : 22532030.048224
[428113.058123] .MIN_vruntime : 0.000001
[428113.058174] .min_vruntime : 25423030.650458
[428113.058226] .max_vruntime : 0.000001
[428113.058277] .spread : 0.000000
[428113.058328] .spread0 : -81001702.698328
[428113.058381] .nr_spread_over : 230
[428113.058431] .nr_running : 0
[428113.058482] .load : 0
[428113.058532] .runnable_load_avg : 0
[428113.058582] .blocked_load_avg : 0
[428113.058632] .tg_load_contrib : 0
[428113.058683] .tg_runnable_contrib : 0
[428113.058734] .tg_load_avg : 2838
[428113.058785] .tg->runnable_avg : 2724
[428113.058836] .tg->cfs_bandwidth.timer_active: 0
[428113.058886] .throttled : 0
[428113.058937] .throttle_count : 0
[428113.058987] .avg->runnable_avg_sum : 29
[428113.059038] .avg->runnable_avg_period : 47630
[428113.059089]
[428113.059089] rt_rq[5]:/
[428113.059181] .rt_nr_running : 0
[428113.059232] .rt_throttled : 0
[428113.059282] .rt_time : 0.000000
[428113.059334] .rt_runtime : 950.000000
[428113.059386]
[428113.059386] dl_rq[5]:
[428113.059477] .dl_nr_running : 0
[428113.059528]
[428113.059528] runnable tasks:
[428113.059528] task PID tree-key switches prio
exec-runtime sum-exec sum-sleep
[428113.059528]
----------------------------------------------------------------------------------------------------------
[428113.059824] rcu_sched 7 25423026.820997 14214418 120
25423026.820997 399659.278793 427702209.389636 /
[428113.059929] watchdog/5 31 -8.977217 107088 0
-8.977217 2499.019149 32.364766 /
[428113.060035] migration/5 32 0.000000 26537 0
0.000000 15739.945360 0.000845 /
[428113.060139] ksoftirqd/5 33 25422651.093839 16195 120
25422651.093839 166.009753 428070175.552405 /
[428113.060245] kworker/5:0 34 239.267711 17 120
239.267711 0.034507 632.432629 /
[428113.060350] kworker/5:0H 35 632.111736 6 100
632.111736 0.059549 2815.362585 /
[428113.060456] kworker/5:1 611 25423018.904992 2773160 120
25423018.904992 1288651.204421 426793243.196345 /
[428113.060561] scsi_tmf_2 860 534.239404 2 100
534.239404 0.008896 0.005460 /
[428113.060667] scsi_tmf_5 866 558.255161 2 100
558.255161 0.008179 0.004413 /
[428113.060772] md0_raid1 899 25396783.503701 32703 120
25396783.503701 998.952947 427489843.030132 /
[428113.060878] kworker/5:1H 1061 24847022.042452 45684 100
24847022.042452 510.479777 416592265.701502 /
[428113.060984] anvil 4282 1335.039244 546 120
1335.039244 25.298050 418304078.786475 /autogroup-17
[428113.061092]
[428113.061138] cpu#6, 3411.800 MHz
[428113.061186] .nr_running : 0
[428113.061236] .load : 0
[428113.061286] .nr_switches : 31375902
[428113.061338] .nr_load_updates : 4945717
[428113.061389] .nr_uninterruptible : 6930
[428113.061440] .next_balance : 4337.771028
[428113.061492] .curr->pid : 0
[428113.061542] .clock : 428112972.478131
[428113.061595] .cpu_load[0] : 0
[428113.061645] .cpu_load[1] : 0
[428113.061695] .cpu_load[2] : 0
[428113.061745] .cpu_load[3] : 0
[428113.061795] .cpu_load[4] : 0
[428113.061846] .yld_count : 20767
[428113.061896] .sched_count : 31528236
[428113.061947] .sched_goidle : 12333544
[428113.061999] .avg_idle : 1000000
[428113.062050] .max_idle_balance_cost : 500000
[428113.062100] .ttwu_count : 17745863
[428113.062152] .ttwu_local : 4311925
[428113.062203]
[428113.062203] cfs_rq[6]:/autogroup-6
[428113.062296] .exec_clock : 173835.323068
[428113.062348] .MIN_vruntime : 0.000001
[428113.062399] .min_vruntime : 163529.123269
[428113.062451] .max_vruntime : 0.000001
[428113.062502] .spread : 0.000000
[428113.062554] .spread0 : -106261204.225517
[428113.062606] .nr_spread_over : 80
[428113.062656] .nr_running : 0
[428113.062706] .load : 0
[428113.062756] .runnable_load_avg : 0
[428113.062807] .blocked_load_avg : 0
[428113.062857] .tg_load_contrib : 0
[428113.062907] .tg_runnable_contrib : 0
[428113.062957] .tg_load_avg : 46
[428113.063008] .tg->runnable_avg : 46
[428113.063058] .tg->cfs_bandwidth.timer_active: 0
[428113.063109] .throttled : 0
[428113.063160] .throttle_count : 0
[428113.063211] .se->exec_start : 428112972.478131
[428113.063263] .se->vruntime : 32014882.056897
[428113.063316] .se->sum_exec_runtime : 173853.720494
[428113.063368] .se->statistics.wait_start : 0.000000
[428113.063419] .se->statistics.sleep_start : 0.000000
[428113.063470] .se->statistics.block_start : 0.000000
[428113.063521] .se->statistics.sleep_max : 0.000000
[428113.063573] .se->statistics.block_max : 0.000000
[428113.063624] .se->statistics.exec_max : 10.006440
[428113.063675] .se->statistics.slice_max : 9.985799
[428113.063726] .se->statistics.wait_max : 14.987432
[428113.063778] .se->statistics.wait_sum : 5142.815457
[428113.063830] .se->statistics.wait_count : 888277
[428113.063881] .se->load.weight : 2
[428113.063932] .se->avg.runnable_avg_sum : 4
[428113.063982] .se->avg.runnable_avg_period : 48033
[428113.064033] .se->avg.load_avg_contrib : 0
[428113.064083] .se->avg.decay_count : 408280347
[428113.064134]
[428113.064134] cfs_rq[6]:/
[428113.064227] .exec_clock : 29301583.069476
[428113.064279] .MIN_vruntime : 0.000001
[428113.064330] .min_vruntime : 32014893.868429
[428113.064382] .max_vruntime : 0.000001
[428113.064433] .spread : 0.000000
[428113.064484] .spread0 : -74409839.480357
[428113.064537] .nr_spread_over : 413
[428113.064587] .nr_running : 0
[428113.064637] .load : 0
[428113.064688] .runnable_load_avg : 0
[428113.064738] .blocked_load_avg : 0
[428113.064789] .tg_load_contrib : 0
[428113.064839] .tg_runnable_contrib : 0
[428113.064890] .tg_load_avg : 2838
[428113.064941] .tg->runnable_avg : 2724
[428113.064992] .tg->cfs_bandwidth.timer_active: 0
[428113.065043] .throttled : 0
[428113.065093] .throttle_count : 0
[428113.065144] .avg->runnable_avg_sum : 4
[428113.065195] .avg->runnable_avg_period : 47831
[428113.065246]
[428113.065246] rt_rq[6]:/
[428113.065338] .rt_nr_running : 0
[428113.065389] .rt_throttled : 0
[428113.065439] .rt_time : 0.000000
[428113.065491] .rt_runtime : 950.000000
[428113.065542]
[428113.065542] dl_rq[6]:
[428113.065644] .dl_nr_running : 0
[428113.065702]
[428113.065702] runnable tasks:
[428113.065702] task PID tree-key switches prio
exec-runtime sum-exec sum-sleep
[428113.065702]
----------------------------------------------------------------------------------------------------------
[428113.065999] watchdog/6 36 -8.979370 107088 0
-8.979370 4171.996810 19.046238 /
[428113.066104] migration/6 37 0.000000 21452 0
0.000000 13182.758087 0.000826 /
[428113.066210] ksoftirqd/6 38 32009668.868629 19084 120
32009668.868629 335.465215 427985619.483782 /
[428113.066315] kworker/6:0 39 2652.785847 15 120
2652.785847 0.034665 619.334482 /
[428113.066420] kworker/6:0H 40 3222.777674 6 100
3222.777674 0.042578 34131.960421 /
[428113.066526] kworker/6:1 613 32014882.656425 3447588 120
32014882.656425 1715810.008839 426365265.308128 /
[428113.066631] bioset 898 2957.541580 2 100
2957.541580 0.008747 0.006389 /
[428113.066737] kworker/6:1H 1463 31224371.088945 50546 100
31224371.088945 612.901074 416298526.150915 /
[428113.066842] kjournald 2216 3410.060184 2 120
3410.060184 0.412389 0.043474 /
[428113.066948] bioset 2222 3422.126220 2 100
3422.126220 0.088585 0.040787 /
[428113.067053] mysqld 4102 163517.397318 1020343 120
163517.397318 10602.850133 428059785.215024 /autogroup-6
[428113.067160] mysqld 4122 163443.475621 644351 120
163443.475621 179019.020540 393898317.914127 /autogroup-6
[428113.068518] dovecot 4220 980.263394 2014 120
980.263394 114.537242 418305299.164771 /autogroup-17
[428113.068624] udevd 6137 246.142493 30 120
246.142493 6.670412 22056460.101598 /autogroup-4
[428113.068731] udevd 6138 242.586598 15 120
242.586598 3.149366 49194.013301 /autogroup-4
[428113.068838] btrfs-worker 6246 1051403.790298 2 100
1051403.790298 0.011158 0.004647 /
[428113.068943] btrfs-endio-met 6255 1051427.810805 2 100
1051427.810805 0.017787 0.002771 /
[428113.069049] btrfs-endio-rai 6257 1051439.820346 2 100
1051439.820346 0.010462 0.002911 /
[428113.069154] btrfs-endio-rep 6258 1051451.828334 2 100
1051451.828334 0.008652 0.003137 /
[428113.069259] btrfs-endio-wri 6260 1051463.836060 2 100
1051463.836060 0.008410 0.003085 /
[428113.069365] btrfs-readahead 6263 1051475.843882 2 100
1051475.843882 0.008543 0.003153 /
[428113.069470] btrfs-extent-re 6265 1051487.851432 2 100
1051487.851432 0.008154 0.003265 /
[428113.069578]
[428113.069632] cpu#7, 3411.800 MHz
[428113.069681] .nr_running : 0
[428113.069734] .load : 0
[428113.069786] .nr_switches : 28961216
[428113.069837] .nr_load_updates : 3537072
[428113.069891] .nr_uninterruptible : 5065
[428113.069943] .next_balance : 4337.771143
[428113.069996] .curr->pid : 0
[428113.070047] .clock : 428112555.897703
[428113.070101] .cpu_load[0] : 0
[428113.070153] .cpu_load[1] : 0
[428113.070211] .cpu_load[2] : 0
[428113.070261] .cpu_load[3] : 0
[428113.070312] .cpu_load[4] : 12
[428113.070363] .yld_count : 22779
[428113.070414] .sched_count : 29115700
[428113.070466] .sched_goidle : 12603487
[428113.070517] .avg_idle : 1000000
[428113.070569] .max_idle_balance_cost : 500000
[428113.070620] .ttwu_count : 18626268
[428113.070671] .ttwu_local : 2870358
[428113.070722]
[428113.070722] cfs_rq[7]:/
[428113.070814] .exec_clock : 14364516.198492
[428113.070867] .MIN_vruntime : 0.000001
[428113.070919] .min_vruntime : 18016573.070883
[428113.070971] .max_vruntime : 0.000001
[428113.071022] .spread : 0.000000
[428113.071073] .spread0 : -88408168.119657
[428113.071126] .nr_spread_over : 261
[428113.071176] .nr_running : 0
[428113.071226] .load : 0
[428113.071277] .runnable_load_avg : 0
[428113.071328] .blocked_load_avg : 0
[428113.071378] .tg_load_contrib : 0
[428113.071429] .tg_runnable_contrib : 0
[428113.071480] .tg_load_avg : 2910
[428113.071530] .tg->runnable_avg : 2849
[428113.071581] .tg->cfs_bandwidth.timer_active: 0
[428113.071632] .throttled : 0
[428113.071682] .throttle_count : 0
[428113.071732] .avg->runnable_avg_sum : 18
[428113.071783] .avg->runnable_avg_period : 47536
[428113.071834]
[428113.071834] rt_rq[7]:/
[428113.071926] .rt_nr_running : 0
[428113.071976] .rt_throttled : 0
[428113.072027] .rt_time : 0.000000
[428113.072078] .rt_runtime : 950.000000
[428113.072130]
[428113.072130] dl_rq[7]:
[428113.072222] .dl_nr_running : 0
[428113.072273]
[428113.072273] runnable tasks:
[428113.072273] task PID tree-key switches prio
exec-runtime sum-exec sum-sleep
[428113.072273]
----------------------------------------------------------------------------------------------------------
[428113.072570] watchdog/7 41 -8.982956 107089 0
-8.982956 1664.683687 5.725643 /
[428113.072674] migration/7 42 0.000000 21715 0
0.000000 13068.322347 0.000816 /
[428113.072780] ksoftirqd/7 43 18013052.310983 12328 120
18013052.310983 152.767844 428092090.014392 /
[428113.072885] kworker/7:0 44 479.385599 15 120
479.385599 0.031395 606.240559 /
[428113.072991] kworker/7:0H 45 1817.499047 6 100
1817.499047 0.047704 42727.098840 /
[428113.073096] netns 47 10.955152 2 100
10.955152 0.004076 0.001586 /
[428113.073201] perf 50 22.958573 2 100
22.958573 0.003794 0.001394 /
[428113.073306] kworker/7:1 615 18016561.430108 1960767 120
18016561.430108 793147.479614 427291244.310703 /
[428113.073413] bioset 919 1132.474026 2 100
1132.474026 0.034373 0.004008 /
[428113.073518] ext4-rsv-conver 936 1156.590460 2 100
1156.590460 0.061309 0.026946 /
[428113.073623] udevd 1124 405.566270 290 120
405.566270 53.586152 77350627.244245 /autogroup-4
[428113.073729] rsyslogd 3061 171793.939660 3747 120
171793.939660 155.674406 427956979.115875 /autogroup-6
[428113.073836] ipv6_addrconf 3209 1679.353321 2 100
1679.353321 0.006452 0.002581 /
[428113.073942] kworker/7:1H 4116 17737413.492285 52256 100
17737413.492285 541.503637 415562752.649104 /
[428113.074048] log 4283 586.242014 697 120
586.242014 34.396858 418304069.988435 /autogroup-17
[428113.074154] qmgr 4370 317.621444 2671 120
317.621444 299.720143 427913275.665071 /autogroup-19
[428113.074262] getty 4376 0.726364 103 120
0.726364 1.882302 146.996217 /autogroup-25
[428113.074369] btrfs-fixup 6253 1049025.643969 2 100
1049025.643969 0.011376 0.003865 /
[428113.074474] btrfs-endio 6254 1049037.659031 2 100
1049037.659031 0.015618 0.001516 /
[428113.074580] kworker/u17:1 23565 17782910.635845 2 100
17782910.635845 0.004742 0.030459 /
[428113.074685] pickup 23745 329.619461 38 120
329.619461 4.298483 2159811.305363 /autogroup-19
[428113.074792]
--
Tomasz Chmielewski
http://www.sslrack.com
^ permalink raw reply [flat|nested] 3+ messages in thread
end of thread, other threads:[~2015-02-16 10:51 UTC | newest]
Thread overview: 3+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2015-02-16 2:32 3.19 - "unable" to replace a failed drive - 100% CPU usage in kworker and btrfs-transacti Tomasz Chmielewski
2015-02-16 10:40 ` Liu Bo
2015-02-16 10:51 ` Tomasz Chmielewski
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox