From: Tomasz Chmielewski <tch@virtall.com>
To: bo.li.liu@oracle.com
Cc: linux-btrfs <linux-btrfs@vger.kernel.org>
Subject: Re: 3.19 - "unable" to replace a failed drive - 100% CPU usage in kworker and btrfs-transacti
Date: Mon, 16 Feb 2015 19:51:08 +0900 [thread overview]
Message-ID: <6830fb54424ede297a6349832bf46d00@admin.virtall.com> (raw)
In-Reply-To: <20150216104026.GC23683@localhost.localdomain>
On 2015-02-16 19:40, Liu Bo wrote:
>> PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
>> 6269 root 20 0 0 0 0 R 92.5 0.0 2769:33
>> btrfs-transacti
>> 22247 root 20 0 0 0 0 R 92.5 0.0 42:38.65
>> kworker/u16:16
>
> Can you cat /proc/22247/stack and /proc/6269/stack?
22247 no longer exists - there is a new kworker now:
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
23570 root 20 0 0 0 0 R 93.8 0.0 108:46.78
kworker/u16:15
6269 root 20 0 0 0 0 R 93.5 0.0 3029:41
btrfs-transacti
# cat /proc/22247/stack
cat: /proc/22247/stack: No such file or directory
# cat /proc/6269/stack
[<ffffffffa02a0b0d>] transaction_kthread+0x197/0x1c2 [btrfs]
[<ffffffff81050067>] kthread+0xcd/0xd5
[<ffffffff813c12ac>] ret_from_fork+0x7c/0xb0
[<ffffffffffffffff>] 0xffffffffffffffff
# cat /proc/23570/stack
[<ffffffffffffffff>] 0xffffffffffffffff
In case a new sysrq-w was needed, too:
[428113.007373] SysRq : Show Blocked State
[428113.007427] task PC stack pid father
[428113.007493] btrfs D ffff8802d67ef948 0 7611 6275
0x00000000
[428113.007549] ffff8802d67ef948 ffff8803fe0409d8 ffff88081917e040
00000000000113c0
[428113.007648] 0000000000004000 ffff88081be31810 ffff88081917e040
ffff8802d67ef8b8
[428113.007748] ffffffff8105f3c4 ffff88083fa80000 ffff88083fa513c0
ffff88083fa513c0
[428113.007847] Call Trace:
[428113.007900] [<ffffffff8105f3c4>] ? enqueue_task_fair+0x3e5/0x44f
[428113.007955] [<ffffffff81054ab6>] ? resched_curr+0x45/0x55
[428113.008008] [<ffffffff81055122>] ? check_preempt_curr+0x3e/0x6d
[428113.008062] [<ffffffff81055163>] ? ttwu_do_wakeup+0x12/0x7f
[428113.008115] [<ffffffff8105526e>] ?
ttwu_do_activate.constprop.73+0x57/0x5c
[428113.008172] [<ffffffff813be618>] schedule+0x65/0x67
[428113.008224] [<ffffffff813c03b8>] schedule_timeout+0x26/0x18d
[428113.008277] [<ffffffff81057d1e>] ? wake_up_process+0x30/0x34
[428113.008331] [<ffffffff8104b1a3>] ? wake_up_worker+0x1f/0x21
[428113.008383] [<ffffffff8104b3ea>] ? insert_work+0x87/0x94
[428113.008447] [<ffffffffa02e183d>] ? free_block_list+0x1f/0x34
[btrfs]
[428113.008501] [<ffffffff813bef76>] wait_for_common+0x10d/0x13e
[428113.008554] [<ffffffff81057cdf>] ? try_to_wake_up+0x250/0x250
[428113.008608] [<ffffffff813befbf>] wait_for_completion+0x18/0x1a
[428113.008666] [<ffffffffa028dedc>]
btrfs_async_run_delayed_refs+0xc1/0xe4 [btrfs]
[428113.008770] [<ffffffffa02a3189>]
__btrfs_end_transaction+0x315/0x33b [btrfs]
[428113.008872] [<ffffffffa02a31bd>]
btrfs_end_transaction_throttle+0xe/0x10 [btrfs]
[428113.008977] [<ffffffffa02e5cf1>] relocate_block_group+0x2ad/0x4de
[btrfs]
[428113.009037] [<ffffffffa02e607a>]
btrfs_relocate_block_group+0x158/0x278 [btrfs]
[428113.009141] [<ffffffffa02c37e1>]
btrfs_relocate_chunk.isra.69+0x35/0xa5 [btrfs]
[428113.009244] [<ffffffffa02c41da>] btrfs_shrink_device+0x235/0x408
[btrfs]
[428113.009304] [<ffffffffa02c6ba9>] btrfs_rm_device+0x2a9/0x704
[btrfs]
[428113.009359] [<ffffffff810f5441>] ?
__kmalloc_track_caller+0x40/0x178
[428113.009419] [<ffffffffa02cf7a7>] btrfs_ioctl+0xa9c/0x22f5 [btrfs]
[428113.009473] [<ffffffff8110f142>] ? putname+0x23/0x2c
[428113.009525] [<ffffffff8110f6d3>] ? user_path_at_empty+0x60/0x90
[428113.009579] [<ffffffff811783bf>] ? avc_has_perm+0x2e/0xf7
[428113.009632] [<ffffffff8111171b>] do_vfs_ioctl+0x418/0x460
[428113.009684] [<ffffffff81106745>] ? vfs_stat+0x16/0x18
[428113.009736] [<ffffffff811117b1>] SyS_ioctl+0x4e/0x7d
[428113.009789] [<ffffffff81031c5b>] ? do_page_fault+0xc/0x11
[428113.009841] [<ffffffff813c1352>] system_call_fastpath+0x12/0x17
[428113.009896] Sched Debug Version: v0.11, 3.19.0 #1
[428113.009947] ktime :
428338904.250198
[428113.010001] sched_clk :
428113009.895839
[428113.010054] cpu_clk :
428113009.895853
[428113.010107] jiffies : 4337771187
[428113.010160] sched_clock_stable() : 1
[428113.010212]
[428113.010257] sysctl_sched
[428113.010305] .sysctl_sched_latency : 24.000000
[428113.010358] .sysctl_sched_min_granularity : 3.000000
[428113.010411] .sysctl_sched_wakeup_granularity : 4.000000
[428113.010463] .sysctl_sched_child_runs_first : 0
[428113.010515] .sysctl_sched_features : 11899
[428113.010568] .sysctl_sched_tunable_scaling : 1
(logaritmic)
[428113.010621]
[428113.010667] cpu#0, 3411.800 MHz
[428113.010715] .nr_running : 0
[428113.010766] .load : 0
[428113.010817] .nr_switches : 141930054
[428113.010869] .nr_load_updates : 24939776
[428113.010920] .nr_uninterruptible : -739558
[428113.010972] .next_balance : 4337.771095
[428113.011023] .curr->pid : 0
[428113.011074] .clock : 428113007.169503
[428113.011126] .cpu_load[0] : 0
[428113.011176] .cpu_load[1] : 0
[428113.011227] .cpu_load[2] : 0
[428113.011277] .cpu_load[3] : 0
[428113.011328] .cpu_load[4] : 24
[428113.011379] .yld_count : 223168
[428113.011430] .sched_count : 142283739
[428113.011482] .sched_goidle : 51488452
[428113.011534] .avg_idle : 1000000
[428113.011584] .max_idle_balance_cost : 500000
[428113.011636] .ttwu_count : 88826891
[428113.011687] .ttwu_local : 54557139
[428113.011739]
[428113.011739] cfs_rq[0]:/autogroup-1201
[428113.011833] .exec_clock : 5.761094
[428113.011884] .MIN_vruntime : 0.000001
[428113.011936] .min_vruntime : 4.697019
[428113.011987] .max_vruntime : 0.000001
[428113.012038] .spread : 0.000000
[428113.012089] .spread0 : -106424728.651767
[428113.012141] .nr_spread_over : 0
[428113.012192] .nr_running : 0
[428113.012242] .load : 0
[428113.012292] .runnable_load_avg : 0
[428113.012343] .blocked_load_avg : 0
[428113.012394] .tg_load_contrib : 0
[428113.012445] .tg_runnable_contrib : 0
[428113.012495] .tg_load_avg : 0
[428113.012546] .tg->runnable_avg : 0
[428113.012596] .tg->cfs_bandwidth.timer_active: 0
[428113.012647] .throttled : 0
[428113.012698] .throttle_count : 0
[428113.012748] .se->exec_start : 428113007.164637
[428113.012801] .se->vruntime : 106424721.380976
[428113.012854] .se->sum_exec_runtime : 5.761094
[428113.012905] .se->statistics.wait_start : 0.000000
[428113.012956] .se->statistics.sleep_start : 0.000000
[428113.013007] .se->statistics.block_start : 0.000000
[428113.013059] .se->statistics.sleep_max : 0.000000
[428113.013110] .se->statistics.block_max : 0.000000
[428113.013161] .se->statistics.exec_max : 0.532615
[428113.013212] .se->statistics.slice_max : 0.000000
[428113.013263] .se->statistics.wait_max : 0.000000
[428113.013314] .se->statistics.wait_sum : 0.000000
[428113.013366] .se->statistics.wait_count : 84
[428113.013416] .se->load.weight : 2
[428113.013466] .se->avg.runnable_avg_sum : 31
[428113.013517] .se->avg.runnable_avg_period : 48192
[428113.013569] .se->avg.load_avg_contrib : 0
[428113.013619] .se->avg.decay_count : 408280380
[428113.013671]
[428113.013671] cfs_rq[0]:/
[428113.013764] .exec_clock : 108726735.130963
[428113.013817] .MIN_vruntime : 0.000001
[428113.013868] .min_vruntime : 106424733.348786
[428113.013920] .max_vruntime : 0.000001
[428113.013971] .spread : 0.000000
[428113.014023] .spread0 : 0.000000
[428113.014074] .nr_spread_over : 291
[428113.014124] .nr_running : 0
[428113.014175] .load : 0
[428113.014225] .runnable_load_avg : 0
[428113.014276] .blocked_load_avg : 0
[428113.014326] .tg_load_contrib : 0
[428113.014377] .tg_runnable_contrib : 0
[428113.014428] .tg_load_avg : 2125
[428113.014479] .tg->runnable_avg : 2029
[428113.015797] .tg->cfs_bandwidth.timer_active: 0
[428113.015847] .throttled : 0
[428113.015897] .throttle_count : 0
[428113.015948] .avg->runnable_avg_sum : 34
[428113.015999] .avg->runnable_avg_period : 48420
[428113.016052]
[428113.016052] rt_rq[0]:/
[428113.016145] .rt_nr_running : 0
[428113.016195] .rt_throttled : 0
[428113.016246] .rt_time : 0.000000
[428113.016297] .rt_runtime : 950.000000
[428113.016349]
[428113.016349] dl_rq[0]:
[428113.016441] .dl_nr_running : 0
[428113.016491]
[428113.016491] runnable tasks:
[428113.016491] task PID tree-key switches prio
exec-runtime sum-exec sum-sleep
[428113.016491]
----------------------------------------------------------------------------------------------------------
[428113.016788] ksoftirqd/0 3 106424458.464162 754529 120
106424458.464162 12412.024323 428020900.243547 /
[428113.016893] kworker/0:0 4 96872056.230102 54 120
96872056.230102 17.312081 398174954.738190 /
[428113.016998] kworker/0:0H 5 1015.287609 5 100
1015.287609 0.041666 1136.081722 /
[428113.017103] rcu_bh 8 117.354314 2 120
117.354314 0.000874 0.000722 /
[428113.017209] migration/0 9 0.000000 42259 0
0.000000 38663.891221 0.000853 /
[428113.017314] watchdog/0 10 -11.786179 107094 0
-11.786179 18301.767349 93.369695 /
[428113.017421] acpi_thermal_pm 618 478.127841 2 100
478.127841 0.008079 0.004241 /
[428113.017527] ata_sff 832 943.140921 2 100
943.140921 0.005880 0.022278 /
[428113.017632] scsi_eh_0 855 1012.563036 18 120
1012.563036 0.875745 352.505402 /
[428113.017738] kworker/0:1H 879 106418747.237080 19756283 100
106418747.237080 936534.886602 427013679.873055 /
[428113.017843] md2_raid1 920 106418747.384047 5312046 120
106418747.384047 235466.210537 427856341.930987 /
[428113.017950] jbd2/md2-8 935 106418747.368760 676645 120
106418747.368760 26050.167995 428067576.860650 /
[428113.018055] rs:main Q:Reg 3060 1156937.947188 3986 120
1156937.947188 610.549422 427956529.353145 /autogroup-6
[428113.018162] rsyslogd 3062 1154980.137945 580 120
1154980.137945 33.203062 397909885.430577 /autogroup-6
[428113.018269] vnstatd 3041 1704.807054 86943 120
1704.807054 8238.508382 428061424.035922 /autogroup-7
[428113.018377] mdadm 3144 27.184928 450 120
27.184928 68.147482 427453633.845909 /autogroup-8
[428113.018483] atd 3189 0.694263 120 120
0.694263 2.561413 424576013.244002 /autogroup-9
[428113.018591] nginx 3232 4317.191565 21 120
4317.191565 1.526614 408639452.655116 /autogroup-10
[428113.018697] nginx 3233 4489.469136 520373 120
4489.469136 6666.843000 427177069.798260 /autogroup-10
[428113.018804] mysqld 4080 1156391.999838 5706 120
1156391.999838 804.810183 419298520.527262 /autogroup-6
[428113.018911] mysqld 4121 1156945.409829 85595 120
1156945.409829 1240.826037 428066936.224975 /autogroup-6
[428113.019017] mysqld 4154 1156391.428236 2178575 120
1156391.428236 373082.741411 418907550.518894 /autogroup-6
[428113.019124] mysqld 4393 1156391.413224 1651105 120
1156391.413224 338375.528086 418894108.235915 /autogroup-6
[428113.019230] mysqld 5811 1156391.734934 1317689 120
1156391.734934 299565.417020 370162247.292961 /autogroup-6
[428113.019337] mysqld 15425 1156391.434917 261945 120
1156391.434917 145838.371509 198220505.720438 /autogroup-6
[428113.019443] mysqld 15436 1156391.985920 292544 120
1156391.985920 172007.149326 198170039.192913 /autogroup-6
[428113.019550] getty 4374 0.817181 100 120
0.817181 2.613716 146.560614 /autogroup-23
[428113.019657] bash 6275 132.300150 276 120
132.300150 79.740920 21744924.508065 /autogroup-184
[428113.019764] kworker/0:2 22822 106424721.355311 1310427 120
106424721.355311 592055.669533 29583164.785882 /
[428113.019869] sshd 23685 4.429642 75 120
4.429642 10.486741 908580.581156 /autogroup-1187
[428113.019976] bash 23687 63.590364 35 120
63.590364 25.369673 13.977239 /autogroup-1188
[428113.020083] sshd 23875 4.697019 109 120
4.697019 10.704193 119911.149251 /autogroup-1201
[428113.020199]
[428113.020246] cpu#1, 3411.800 MHz
[428113.020302] .nr_running : 1
[428113.020354] .load : 1024
[428113.020407] .nr_switches : 88720888
[428113.020461] .nr_load_updates : 23703925
[428113.020515] .nr_uninterruptible : 483951
[428113.020568] .next_balance : 4337.771189
[428113.020621] .curr->pid : 3060
[428113.020675] .clock : 428113020.257470
[428113.020727] .cpu_load[0] : 0
[428113.020786] .cpu_load[1] : 0
[428113.020836] .cpu_load[2] : 0
[428113.020887] .cpu_load[3] : 0
[428113.020937] .cpu_load[4] : 0
[428113.020988] .yld_count : 135412
[428113.021039] .sched_count : 89125261
[428113.021090] .sched_goidle : 30252689
[428113.021141] .avg_idle : 1000000
[428113.021192] .max_idle_balance_cost : 500000
[428113.021244] .ttwu_count : 38652568
[428113.021295] .ttwu_local : 21074490
[428113.021346]
[428113.021346] cfs_rq[1]:/autogroup-183
[428113.021440] .exec_clock : 10198.070166
[428113.021492] .MIN_vruntime : 0.000001
[428113.021543] .min_vruntime : 10197.021590
[428113.021595] .max_vruntime : 0.000001
[428113.021646] .spread : 0.000000
[428113.021697] .spread0 : -106414536.327196
[428113.021749] .nr_spread_over : 0
[428113.021800] .nr_running : 0
[428113.021850] .load : 0
[428113.021900] .runnable_load_avg : 0
[428113.021951] .blocked_load_avg : 0
[428113.022001] .tg_load_contrib : 0
[428113.022052] .tg_runnable_contrib : 0
[428113.022103] .tg_load_avg : 0
[428113.022154] .tg->runnable_avg : 0
[428113.022204] .tg->cfs_bandwidth.timer_active: 0
[428113.022255] .throttled : 0
[428113.022305] .throttle_count : 0
[428113.022356] .se->exec_start : 428113015.541150
[428113.022409] .se->vruntime : 129995516.110494
[428113.022461] .se->sum_exec_runtime : 10198.307701
[428113.022513] .se->statistics.wait_start : 0.000000
[428113.022564] .se->statistics.sleep_start : 0.000000
[428113.022615] .se->statistics.block_start : 0.000000
[428113.022666] .se->statistics.sleep_max : 0.000000
[428113.022717] .se->statistics.block_max : 0.000000
[428113.022769] .se->statistics.exec_max : 2.939902
[428113.022820] .se->statistics.slice_max : 2.075783
[428113.022871] .se->statistics.wait_max : 5.458038
[428113.022922] .se->statistics.wait_sum : 30.977291
[428113.022973] .se->statistics.wait_count : 234554
[428113.023024] .se->load.weight : 2
[428113.023074] .se->avg.runnable_avg_sum : 34
[428113.023125] .se->avg.runnable_avg_period : 47537
[428113.023176] .se->avg.load_avg_contrib : 0
[428113.023226] .se->avg.decay_count : 408280388
[428113.023278]
[428113.023278] cfs_rq[1]:/autogroup-6
[428113.023372] .exec_clock : 808878.880122
[428113.023424] .MIN_vruntime : 0.000001
[428113.023476] .min_vruntime : 692298.789871
[428113.023528] .max_vruntime : 0.000001
[428113.023579] .spread : 0.000000
[428113.023630] .spread0 : -105732434.558915
[428113.023683] .nr_spread_over : 259
[428113.023734] .nr_running : 0
[428113.023784] .load : 0
[428113.023834] .runnable_load_avg : 0
[428113.023885] .blocked_load_avg : 0
[428113.023935] .tg_load_contrib : 0
[428113.023986] .tg_runnable_contrib : 0
[428113.024036] .tg_load_avg : 0
[428113.024087] .tg->runnable_avg : 5
[428113.024138] .tg->cfs_bandwidth.timer_active: 0
[428113.024188] .throttled : 0
[428113.024239] .throttle_count : 0
[428113.024290] .se->exec_start : 428113020.747637
[428113.024342] .se->vruntime : 129995520.664159
[428113.024395] .se->sum_exec_runtime : 808905.454732
[428113.024446] .se->statistics.wait_start : 0.000000
[428113.024497] .se->statistics.sleep_start : 0.000000
[428113.024548] .se->statistics.block_start : 0.000000
[428113.024600] .se->statistics.sleep_max : 0.000000
[428113.024651] .se->statistics.block_max : 0.000000
[428113.024702] .se->statistics.exec_max : 10.030247
[428113.024754] .se->statistics.slice_max : 10.135075
[428113.024805] .se->statistics.wait_max : 14.316925
[428113.024856] .se->statistics.wait_sum : 6528.008676
[428113.024908] .se->statistics.wait_count : 4854101
[428113.024959] .se->load.weight : 2
[428113.025009] .se->avg.runnable_avg_sum : 489
[428113.025059] .se->avg.runnable_avg_period : 47586
[428113.026377] .se->avg.load_avg_contrib : 0
[428113.026428] .se->avg.decay_count : 408280393
[428113.026479]
[428113.026479] cfs_rq[1]:/
[428113.026571] .exec_clock : 134248386.191002
[428113.026624] .MIN_vruntime : 0.000001
[428113.026676] .min_vruntime : 129995526.942391
[428113.026728] .max_vruntime : 0.000001
[428113.026780] .spread : 0.000000
[428113.026831] .spread0 : 23570793.593605
[428113.026883] .nr_spread_over : 311
[428113.026933] .nr_running : 0
[428113.026984] .load : 0
[428113.027035] .runnable_load_avg : 0
[428113.027085] .blocked_load_avg : 0
[428113.027136] .tg_load_contrib : 0
[428113.027186] .tg_runnable_contrib : 10
[428113.027237] .tg_load_avg : 2353
[428113.027288] .tg->runnable_avg : 2284
[428113.027339] .tg->cfs_bandwidth.timer_active: 0
[428113.027389] .throttled : 0
[428113.027439] .throttle_count : 0
[428113.027490] .avg->runnable_avg_sum : 492
[428113.027540] .avg->runnable_avg_period : 47214
[428113.027591]
[428113.027591] rt_rq[1]:/
[428113.027683] .rt_nr_running : 0
[428113.027733] .rt_throttled : 0
[428113.027783] .rt_time : 0.000000
[428113.027834] .rt_runtime : 950.000000
[428113.027886]
[428113.027886] dl_rq[1]:
[428113.027978] .dl_nr_running : 0
[428113.028029]
[428113.028029] runnable tasks:
[428113.028029] task PID tree-key switches prio
exec-runtime sum-exec sum-sleep
[428113.028029]
----------------------------------------------------------------------------------------------------------
[428113.028324] kthreadd 2 129745695.820738 1414 120
129745695.820738 41.527944 427299434.064971 /
[428113.028430] watchdog/1 11 -2.973570 107088 0
-2.973570 23044.650834 85.639910 /
[428113.028535] migration/1 12 0.000000 38706 0
0.000000 32310.384635 0.000758 /
[428113.028641] ksoftirqd/1 13 129993260.237901 96666 120
129993260.237901 1796.063974 428051863.708966 /
[428113.028746] kworker/1:0H 15 582.372111 6 100
582.372111 0.027131 2733.616385 /
[428113.028852] kpsmoused 686 25.893051 2 100
25.893051 0.010838 0.003636 /
[428113.028957] kworker/1:2 688 102920251.057920 55 120
102920251.057920 0.504998 355091633.648033 /
[428113.029062] deferwq 731 158.015576 2 100
158.015576 0.009076 0.003883 /
[428113.029167] scsi_tmf_0 856 442.536328 2 100
442.536328 0.008945 0.003987 /
[428113.029273] scsi_eh_1 857 539.461151 16 120
539.461151 0.868588 350.508685 /
[428113.029378] scsi_tmf_1 858 466.554669 2 100
466.554669 0.008786 0.003948 /
[428113.029484] scsi_eh_2 859 514.602593 3 120
514.602593 0.016414 1.372299 /
[428113.029589] scsi_eh_3 861 514.603810 3 120
514.603810 0.016837 1.222996 /
[428113.029694] scsi_tmf_3 862 502.578935 2 100
502.578935 0.008052 0.003896 /
[428113.029798] scsi_tmf_4 864 514.587500 2 100
514.587500 0.008905 0.004404 /
[428113.029904] kworker/1:1H 968 129831013.397442 456157 100
129831013.397442 6526.830851 427481927.625721 /
[428113.030009] rsyslogd 3033 691065.336881 29 120
691065.336881 2.741841 408640738.003949 /autogroup-6
[428113.030116] rs:main Q:Reg 3060 692287.280038 3987 120
692287.280038 611.039589 428073924.132108 /autogroup-6
[428113.030223] nginx 3234 4197.283423 855782 120
4197.283423 11301.321120 428062021.947465 /autogroup-10
[428113.030330] nginx 3235 4197.323950 684268 120
4197.323950 9289.505173 428064027.946907 /autogroup-10
[428113.030436] cron 3301 927.592784 8289 120
927.592784 433.794782 428068473.223910 /autogroup-12
[428113.030543] ntpd 3378 2401.387432 430091 120
2401.387432 12164.876249 428060869.176748 /autogroup-13
[428113.030650] mysqld_safe 3464 1418.978306 435 120
1418.978306 26.375085 974.715839 /autogroup-6
[428113.030756] mysqld 4094 692286.802491 856622 120
692286.802491 7997.578025 428062557.688191 /autogroup-6
[428113.030863] mysqld 4095 692286.812618 908471 120
692286.812618 10016.493168 428060177.274007 /autogroup-6
[428113.030969] mysqld 4096 692286.794838 1340772 120
692286.794838 56672.194508 428011328.939086 /autogroup-6
[428113.031075] mysqld 4097 692286.802298 1314182 120
692286.802298 55304.687185 428012714.776737 /autogroup-6
[428113.031182] mysqld 4098 692286.798431 1342580 120
692286.798431 57725.305541 428009725.287820 /autogroup-6
[428113.031289] mysqld 4101 692286.801904 1083476 120
692286.801904 11507.690040 428058770.056113 /autogroup-6
[428113.031395] mysqld 4103 692286.796803 1402041 120
692286.796803 16514.909612 428053422.705042 /autogroup-6
[428113.031501] mysqld 4118 692286.906040 430988 120
692286.906040 54290.532636 428014960.090656 /autogroup-6
[428113.031608] mysqld 4119 692298.789871 434800 120
692298.789871 127525.290540 427941309.545286 /autogroup-6
[428113.031715] mysqld 4120 692285.773389 85683 120
692285.773389 1366.692133 428064373.232093 /autogroup-6
[428113.031822] mysqld 4396 692171.636633 1079842 120
692171.636633 305192.243262 426206373.814546 /autogroup-6
[428113.031930] mysqld 15426 691721.033869 784935 120
691721.033869 137131.954258 198224493.143598 /autogroup-6
[428113.032037] getty 4372 0.769188 104 120
0.769188 2.833955 146.554963 /autogroup-21
[428113.032144] btrfs-delalloc 6249 120544673.387058 4 100
120544673.387058 12.566284 345061022.997020 /
[428113.032250] tmux 6274 10197.021590 1077829 120
10197.021590 47181.472538 372426592.567655 /autogroup-183
[428113.032356] kworker/1:0 21019 129995518.151853 3521622 120
129995518.151853 1682209.369392 71323049.000940 /
[428113.032462] sshd 23755 4.481966 346 120
4.481966 33.939190 1200991.721059 /autogroup-1194
[428113.032569] bash 23757 19.596943 121 120
19.596943 36.036523 1106982.698162 /autogroup-1195
[428113.032676] sshd 23823 0.668354 29 120
0.668354 7.093561 202985.617679 /autogroup-1199
[428113.032785]
[428113.032833] cpu#2, 3411.800 MHz
[428113.032891] .nr_running : 1
[428113.032943] .load : 1024
[428113.032995] .nr_switches : 81057822
[428113.033048] .nr_load_updates : 21474857
[428113.033100] .nr_uninterruptible : 140117
[428113.033154] .next_balance : 4337.771194
[428113.033207] .curr->pid : 23570
[428113.033258] .clock : 428113026.128231
[428113.033319] .cpu_load[0] : 1023
[428113.033370] .cpu_load[1] : 1023
[428113.033420] .cpu_load[2] : 1023
[428113.033471] .cpu_load[3] : 1023
[428113.033521] .cpu_load[4] : 1023
[428113.033572] .yld_count : 92399
[428113.033623] .sched_count : 81351574
[428113.033674] .sched_goidle : 28607226
[428113.033725] .avg_idle : 586733
[428113.033776] .max_idle_balance_cost : 500000
[428113.033827] .ttwu_count : 37353396
[428113.033878] .ttwu_local : 18907401
[428113.033929]
[428113.033929] cfs_rq[2]:/
[428113.034021] .exec_clock : 112244043.135162
[428113.034073] .MIN_vruntime : 0.000001
[428113.034125] .min_vruntime : 109548478.645357
[428113.034177] .max_vruntime : 0.000001
[428113.034228] .spread : 0.000000
[428113.034279] .spread0 : 3123745.296571
[428113.034331] .nr_spread_over : 470
[428113.034382] .nr_running : 1
[428113.034432] .load : 1024
[428113.034483] .runnable_load_avg : 1023
[428113.034533] .blocked_load_avg : 44
[428113.034584] .tg_load_contrib : 1056
[428113.034634] .tg_runnable_contrib : 1010
[428113.034685] .tg_load_avg : 2561
[428113.034736] .tg->runnable_avg : 2477
[428113.034787] .tg->cfs_bandwidth.timer_active: 0
[428113.034837] .throttled : 0
[428113.034888] .throttle_count : 0
[428113.034938] .avg->runnable_avg_sum : 47622
[428113.034990] .avg->runnable_avg_period : 47622
[428113.035041]
[428113.035041] rt_rq[2]:/
[428113.035132] .rt_nr_running : 0
[428113.035183] .rt_throttled : 0
[428113.035233] .rt_time : 0.000000
[428113.035284] .rt_runtime : 950.000000
[428113.035336]
[428113.035336] dl_rq[2]:
[428113.035427] .dl_nr_running : 0
[428113.035478]
[428113.035478] runnable tasks:
[428113.035478] task PID tree-key switches prio
exec-runtime sum-exec sum-sleep
[428113.035478]
----------------------------------------------------------------------------------------------------------
[428113.037044] watchdog/2 16 -5.976382 107088 0
-5.976382 19052.269468 72.326265 /
[428113.037149] migration/2 17 0.000000 34097 0
0.000000 32002.834817 0.001139 /
[428113.037254] ksoftirqd/2 18 109545990.177739 74000 120
109545990.177739 2163.416662 428073696.139572 /
[428113.037359] kworker/2:0 19 100410070.169673 40 120
100410070.169673 0.134368 398174981.123393 /
[428113.037465] kworker/2:0H 20 3650.646229 6 100
3650.646229 0.037533 34187.208217 /
[428113.037571] khungtaskd 204 109533300.730792 3571 120
109533300.730792 195.253551 428053978.079872 /
[428113.037676] kswapd0 506 106048732.107333 11160 120
106048732.107333 5508.370790 416651549.349593 /
[428113.037782] scsi_eh_4 863 3072.735732 3 120
3072.735732 0.015446 1.048287 /
[428113.037887] scsi_eh_5 865 3072.733721 3 120
3072.733721 0.012919 0.910030 /
[428113.037993] kworker/2:1H 1466 108820462.412649 394876 100
108820462.412649 5278.985782 425855655.769615 /
[428113.038098] acpid 3240 0.392530 49 120
0.392530 1.816842 407504916.577611 /autogroup-11
[428113.038205] dbus-daemon 3404 0.942824 6 120
0.942824 1.484584 55885009.390813 /autogroup-14
[428113.038313] mysqld 4150 460431.045860 7023989 120
460431.045860 1407433.937451 407178531.151580 /autogroup-6
[428113.038419] master 4345 398.492132 21611 120
398.492132 1062.087947 428043837.677636 /autogroup-19
[428113.038526] getty 4373 0.640858 102 120
0.640858 2.425952 146.827128 /autogroup-22
[428113.038633] btrfs-worker-hi 6248 1154879.053331 2 100
1154879.053331 0.017567 0.003220 /
[428113.038739] btrfs-flush_del 6250 1154903.071666 2 100
1154903.071666 0.009022 0.003320 /
[428113.038844] btrfs-cache 6251 1154915.081079 2 100
1154915.081079 0.015404 0.003203 /
[428113.038949] btrfs-submit 6252 1154915.162054 2 100
1154915.162054 0.096237 0.023172 /
[428113.039054] btrfs-endio-met 6256 1154927.254127 2 100
1154927.254127 0.093435 0.002869 /
[428113.039160] btrfs-rmw 6259 1154939.263666 2 100
1154939.263666 0.010258 0.003176 /
[428113.039265] btrfs-freespace 6261 1154951.272823 2 100
1154951.272823 0.009779 0.003215 /
[428113.039371] btrfs-delayed-m 6262 1154963.281556 2 100
1154963.281556 0.009338 0.003128 /
[428113.039475] btrfs-qgroup-re 6264 1154975.290187 2 100
1154975.290187 0.009265 0.003113 /
[428113.039580] btrfs 7611 332657.739906 2080440 120
332657.739906 1556211.983521 349088605.012752 /autogroup-184
[428113.039687] kworker/2:1 22833 109548476.671264 1253598 120
109548476.671264 577663.225799 29354960.583584 /
[428113.039793] kworker/u17:0 23543 106131220.757333 22 100
106131220.757333 0.419296 905976.409476 /
[428113.039898] R kworker/u16:15 23570 109548488.151519 847746 120
109548488.151519 6605807.116828 4151195.288717 /
[428113.040006]
[428113.040052] cpu#3, 3411.800 MHz
[428113.040108] .nr_running : 2
[428113.040159] .load : 2048
[428113.040211] .nr_switches : 82481964
[428113.040262] .nr_load_updates : 22241954
[428113.040322] .nr_uninterruptible : 84834
[428113.040374] .next_balance : 4337.771192
[428113.040426] .curr->pid : 23877
[428113.040477] .clock : 428113040.005657
[428113.040529] .cpu_load[0] : 501
[428113.040580] .cpu_load[1] : 385
[428113.040630] .cpu_load[2] : 237
[428113.040681] .cpu_load[3] : 132
[428113.040732] .cpu_load[4] : 70
[428113.040783] .yld_count : 73921
[428113.040834] .sched_count : 82696221
[428113.040885] .sched_goidle : 28975510
[428113.040936] .avg_idle : 1000000
[428113.040988] .max_idle_balance_cost : 500000
[428113.041039] .ttwu_count : 39104815
[428113.041090] .ttwu_local : 19561240
[428113.041141]
[428113.041141] cfs_rq[3]:/autogroup-1202
[428113.041235] .exec_clock : 46.587190
[428113.041287] .MIN_vruntime : 0.000001
[428113.041339] .min_vruntime : 117.495213
[428113.041390] .max_vruntime : 0.000001
[428113.041441] .spread : 0.000000
[428113.041493] .spread0 : -106424615.853573
[428113.041545] .nr_spread_over : 1
[428113.041595] .nr_running : 1
[428113.041645] .load : 1024
[428113.041696] .runnable_load_avg : 501
[428113.041747] .blocked_load_avg : 0
[428113.041797] .tg_load_contrib : 501
[428113.041848] .tg_runnable_contrib : 503
[428113.041899] .tg_load_avg : 501
[428113.041950] .tg->runnable_avg : 503
[428113.042001] .tg->cfs_bandwidth.timer_active: 0
[428113.042052] .throttled : 0
[428113.042102] .throttle_count : 0
[428113.042153] .se->exec_start : 428113040.005657
[428113.042206] .se->vruntime : 112122172.726168
[428113.042258] .se->sum_exec_runtime : 46.587190
[428113.042309] .se->statistics.wait_start : 0.000000
[428113.042361] .se->statistics.sleep_start : 0.000000
[428113.042412] .se->statistics.block_start : 0.000000
[428113.042463] .se->statistics.sleep_max : 0.000000
[428113.042514] .se->statistics.block_max : 0.000000
[428113.042565] .se->statistics.exec_max : 12.895535
[428113.042616] .se->statistics.slice_max : 0.000000
[428113.042667] .se->statistics.wait_max : 0.012418
[428113.042719] .se->statistics.wait_sum : 0.012418
[428113.042770] .se->statistics.wait_count : 32
[428113.042820] .se->load.weight : 1024
[428113.042871] .se->avg.runnable_avg_sum : 23121
[428113.042922] .se->avg.runnable_avg_period : 47024
[428113.042972] .se->avg.load_avg_contrib : 501
[428113.043023] .se->avg.decay_count : 0
[428113.043073]
[428113.043073] cfs_rq[3]:/autogroup-6
[428113.043167] .exec_clock : 489341.658980
[428113.043219] .MIN_vruntime : 0.000001
[428113.043271] .min_vruntime : 406937.010943
[428113.043323] .max_vruntime : 0.000001
[428113.043374] .spread : 0.000000
[428113.043425] .spread0 : -106017796.337843
[428113.043477] .nr_spread_over : 245
[428113.043528] .nr_running : 0
[428113.043578] .load : 0
[428113.043628] .runnable_load_avg : 0
[428113.043679] .blocked_load_avg : 0
[428113.043730] .tg_load_contrib : 0
[428113.043781] .tg_runnable_contrib : 0
[428113.043831] .tg_load_avg : 27
[428113.043882] .tg->runnable_avg : 27
[428113.043933] .tg->cfs_bandwidth.timer_active: 0
[428113.043983] .throttled : 0
[428113.044033] .throttle_count : 0
[428113.044083] .se->exec_start : 428112971.994259
[428113.044135] .se->vruntime : 112122139.240601
[428113.044188] .se->sum_exec_runtime : 489369.503771
[428113.044240] .se->statistics.wait_start : 0.000000
[428113.044292] .se->statistics.sleep_start : 0.000000
[428113.044343] .se->statistics.block_start : 0.000000
[428113.044394] .se->statistics.sleep_max : 0.000000
[428113.044445] .se->statistics.block_max : 0.000000
[428113.044496] .se->statistics.exec_max : 9.999422
[428113.044548] .se->statistics.slice_max : 9.991785
[428113.044599] .se->statistics.wait_max : 19.987876
[428113.044650] .se->statistics.wait_sum : 7372.829518
[428113.044702] .se->statistics.wait_count : 4009170
[428113.044753] .se->load.weight : 2
[428113.044803] .se->avg.runnable_avg_sum : 2
[428113.044853] .se->avg.runnable_avg_period : 47424
[428113.044904] .se->avg.load_avg_contrib : 0
[428113.044955] .se->avg.decay_count : 408280346
[428113.045006]
[428113.045006] cfs_rq[3]:/
[428113.045098] .exec_clock : 114980390.622745
[428113.045151] .MIN_vruntime : 112122139.396351
[428113.045203] .min_vruntime : 112122151.162594
[428113.045256] .max_vruntime : 112122139.396351
[428113.045308] .spread : 0.000000
[428113.045359] .spread0 : 5697417.813808
[428113.045411] .nr_spread_over : 342
[428113.045461] .nr_running : 2
[428113.045511] .load : 2048
[428113.045562] .runnable_load_avg : 501
[428113.045620] .blocked_load_avg : 0
[428113.045682] .tg_load_contrib : 501
[428113.045733] .tg_runnable_contrib : 502
[428113.045784] .tg_load_avg : 2611
[428113.045835] .tg->runnable_avg : 2561
[428113.045891] .tg->cfs_bandwidth.timer_active: 0
[428113.045941] .throttled : 0
[428113.045992] .throttle_count : 0
[428113.046042] .avg->runnable_avg_sum : 23288
[428113.046092] .avg->runnable_avg_period : 47497
[428113.046144]
[428113.046144] rt_rq[3]:/
[428113.047487] .rt_nr_running : 0
[428113.047537] .rt_throttled : 0
[428113.047587] .rt_time : 0.000000
[428113.047639] .rt_runtime : 950.000000
[428113.047690]
[428113.047690] dl_rq[3]:
[428113.047782] .dl_nr_running : 0
[428113.047832]
[428113.047832] runnable tasks:
[428113.047832] task PID tree-key switches prio
exec-runtime sum-exec sum-sleep
[428113.047832]
----------------------------------------------------------------------------------------------------------
[428113.048127] init 1 586.822029 85839 120
586.822029 2768.064315 428108381.243581 /autogroup-2
[428113.048233] watchdog/3 21 -5.984614 107089 0
-5.984614 19753.484076 58.998123 /
[428113.048338] migration/3 22 0.000000 33711 0
0.000000 33100.318561 0.000847 /
[428113.048443] ksoftirqd/3 23 112114355.272976 66314 120
112114355.272976 1782.976803 428053105.661252 /
[428113.048548] kworker/3:0 24 33082955.717338 56 120
33082955.717338 0.596786 176596062.108639 /
[428113.048653] kworker/3:0H 25 1600.396479 6 100
1600.396479 0.027776 33504.961049 /
[428113.048758] khelper 46 10.955864 2 100
10.955864 0.005052 0.001801 /
[428113.048864] writeback 205 311.132001 2 100
311.132001 0.003788 0.001292 /
[428113.048969] crypto 208 323.135116 2 100
323.135116 0.003395 0.001548 /
[428113.049074] bioset 209 335.138067 2 100
335.138067 0.003268 0.001389 /
[428113.049179] kblockd 210 347.141115 2 100
347.141115 0.003369 0.001350 /
[428113.049284] md 361 359.144676 2 100
359.144676 0.003908 0.001863 /
[428113.049389] fsnotify_mark 576 1285553.304174 56 120
1285553.304174 1.356477 77382968.604008 /
[428113.049495] bioset 909 1410.076226 2 100
1410.076226 0.008348 0.004053 /
[428113.049600] kworker/3:1H 1233 109291133.379639 383443 100
109291133.379639 4828.980689 418283340.722864 /
[428113.049706] nginx 3236 5458.045701 513869 120
5458.045701 6979.077801 428066409.223752 /autogroup-10
[428113.049814] sshd 3426 0.942592 37 120
0.942592 4.235482 427955415.003546 /autogroup-15
[428113.049920] mysqld 4099 406925.050358 1297863 120
406925.050358 54242.559211 428013387.543354 /autogroup-6
[428113.050027] mysqld 4100 406925.049535 1470492 120
406925.049535 17668.621861 428052266.284702 /autogroup-6
[428113.050135] mysqld 4123 260297.226173 260619 120
260297.226173 48020.432086 156765824.444851 /autogroup-6
[428113.050242] mysqld 15435 406273.301208 228083 120
406273.301208 118653.052456 198232617.393831 /autogroup-6
[428113.050348] getty 4375 0.839336 103 120
0.839336 1.796678 147.205983 /autogroup-24
[428113.050455] btrfs-cleaner 6268 112107699.597918 22244 120
112107699.597918 5420.971640 372440029.550711 /
[428113.050561] kworker/3:1 13622 112122139.396351 10957597 120
112122139.396351 4812976.705719 246653995.669097 /
[428113.050666] ssl-params 23599 422.267712 4 120
422.267712 0.893866 0.114741 /autogroup-17
[428113.050773] kworker/u16:0 23680 112122139.491207 171869 120
112122139.491207 1346238.717160 2149270.672568 /
[428113.050879] bash 23825 114.365698 37 120
114.365698 25.754806 13.396110 /autogroup-1200
[428113.050986] R bash 23877 117.495213 81 120
117.495213 63.565396 116682.313391 /autogroup-1202
[428113.051095]
[428113.051143] cpu#4, 3411.800 MHz
[428113.051200] .nr_running : 1
[428113.051253] .load : 1024
[428113.051305] .nr_switches : 32118320
[428113.051357] .nr_load_updates : 3723003
[428113.051409] .nr_uninterruptible : 12676
[428113.051461] .next_balance : 4337.771202
[428113.051514] .curr->pid : 6269
[428113.051565] .clock : 428113046.666634
[428113.051617] .cpu_load[0] : 1023
[428113.051668] .cpu_load[1] : 1023
[428113.051718] .cpu_load[2] : 1023
[428113.051769] .cpu_load[3] : 1023
[428113.051820] .cpu_load[4] : 1003
[428113.051879] .yld_count : 30496
[428113.051930] .sched_count : 32427582
[428113.051981] .sched_goidle : 13462549
[428113.052032] .avg_idle : 1000000
[428113.052083] .max_idle_balance_cost : 500000
[428113.052134] .ttwu_count : 12413852
[428113.052185] .ttwu_local : 3463562
[428113.052236]
[428113.052236] cfs_rq[4]:/
[428113.052329] .exec_clock : 19891085.291101
[428113.052381] .MIN_vruntime : 0.000001
[428113.052432] .min_vruntime : 23963851.976267
[428113.052484] .max_vruntime : 0.000001
[428113.052536] .spread : 0.000000
[428113.052587] .spread0 : -82460881.372519
[428113.052639] .nr_spread_over : 307
[428113.052689] .nr_running : 1
[428113.052739] .load : 1024
[428113.052790] .runnable_load_avg : 1023
[428113.052840] .blocked_load_avg : 47
[428113.052891] .tg_load_contrib : 1047
[428113.052942] .tg_runnable_contrib : 1015
[428113.052993] .tg_load_avg : 2738
[428113.053044] .tg->runnable_avg : 2685
[428113.053095] .tg->cfs_bandwidth.timer_active: 0
[428113.053146] .throttled : 0
[428113.053196] .throttle_count : 0
[428113.053247] .avg->runnable_avg_sum : 46889
[428113.053298] .avg->runnable_avg_period : 46889
[428113.053350]
[428113.053350] rt_rq[4]:/
[428113.053442] .rt_nr_running : 0
[428113.053492] .rt_throttled : 0
[428113.053542] .rt_time : 0.000000
[428113.053593] .rt_runtime : 950.000000
[428113.053645]
[428113.053645] dl_rq[4]:
[428113.053736] .dl_nr_running : 0
[428113.053787]
[428113.053787] runnable tasks:
[428113.053787] task PID tree-key switches prio
exec-runtime sum-exec sum-sleep
[428113.053787]
----------------------------------------------------------------------------------------------------------
[428113.054082] watchdog/4 26 -8.974112 107089 0
-8.974112 2059.037229 45.684752 /
[428113.054187] migration/4 27 0.000000 33383 0
0.000000 19839.147564 0.000834 /
[428113.054292] ksoftirqd/4 28 23963277.915795 14197 120
23963277.915795 161.575572 428107147.523261 /
[428113.054397] kworker/4:0 29 1190565.109457 20 120
1190565.109457 0.048792 84471152.489397 /
[428113.054502] kworker/4:0H 30 1300.883814 6 100
1300.883814 0.042505 33602.235307 /
[428113.054608] irq/28-mei_me 673 0.000000 14 49
0.000000 0.320285 0.000000 /
[428113.054714] md1_raid1 910 23938903.837682 1461 120
23938903.837682 66.950526 427490660.626113 /
[428113.054821] kworker/4:1H 1287 22856596.596367 40421 100
22856596.596367 571.074116 407462368.250643 /
[428113.054926] mysqld 4148 104218.340695 6 120
104218.340695 0.235452 160281236.866385 /autogroup-6
[428113.055034] getty 4371 3.299146 103 120
3.299146 4.682491 144.819918 /autogroup-20
[428113.055142] Rbtrfs-transacti 6269 23963851.976267 24693859 120
23963851.976267 181831355.421691 178264635.080484 /
[428113.055247] kworker/4:2 9937 23963841.012506 2387667 120
23963841.012506 1048755.187071 342552425.512330 /
[428113.055353] kworker/u16:2 23684 23924997.103420 4 120
23924997.103420 0.030497 2236660.637946 /
[428113.055458] top 23926 67.900517 110 120
67.900517 88.375149 90189.080325 /autogroup-1195
[428113.055566]
[428113.055619] cpu#5, 3411.800 MHz
[428113.055683] .nr_running : 0
[428113.055735] .load : 0
[428113.055794] .nr_switches : 29826080
[428113.055846] .nr_load_updates : 4205435
[428113.055898] .nr_uninterruptible : 5986
[428113.055959] .next_balance : 4337.771193
[428113.056011] .curr->pid : 0
[428113.056061] .clock : 428113055.637128
[428113.056113] .cpu_load[0] : 0
[428113.056164] .cpu_load[1] : 0
[428113.056215] .cpu_load[2] : 0
[428113.056265] .cpu_load[3] : 0
[428113.056316] .cpu_load[4] : 0
[428113.056367] .yld_count : 23948
[428113.056418] .sched_count : 29988603
[428113.056470] .sched_goidle : 12260277
[428113.056521] .avg_idle : 1000000
[428113.056573] .max_idle_balance_cost : 500000
[428113.056623] .ttwu_count : 14760716
[428113.056675] .ttwu_local : 3492006
[428113.056726]
[428113.056726] cfs_rq[5]:/
[428113.058071] .exec_clock : 22532030.048224
[428113.058123] .MIN_vruntime : 0.000001
[428113.058174] .min_vruntime : 25423030.650458
[428113.058226] .max_vruntime : 0.000001
[428113.058277] .spread : 0.000000
[428113.058328] .spread0 : -81001702.698328
[428113.058381] .nr_spread_over : 230
[428113.058431] .nr_running : 0
[428113.058482] .load : 0
[428113.058532] .runnable_load_avg : 0
[428113.058582] .blocked_load_avg : 0
[428113.058632] .tg_load_contrib : 0
[428113.058683] .tg_runnable_contrib : 0
[428113.058734] .tg_load_avg : 2838
[428113.058785] .tg->runnable_avg : 2724
[428113.058836] .tg->cfs_bandwidth.timer_active: 0
[428113.058886] .throttled : 0
[428113.058937] .throttle_count : 0
[428113.058987] .avg->runnable_avg_sum : 29
[428113.059038] .avg->runnable_avg_period : 47630
[428113.059089]
[428113.059089] rt_rq[5]:/
[428113.059181] .rt_nr_running : 0
[428113.059232] .rt_throttled : 0
[428113.059282] .rt_time : 0.000000
[428113.059334] .rt_runtime : 950.000000
[428113.059386]
[428113.059386] dl_rq[5]:
[428113.059477] .dl_nr_running : 0
[428113.059528]
[428113.059528] runnable tasks:
[428113.059528] task PID tree-key switches prio
exec-runtime sum-exec sum-sleep
[428113.059528]
----------------------------------------------------------------------------------------------------------
[428113.059824] rcu_sched 7 25423026.820997 14214418 120
25423026.820997 399659.278793 427702209.389636 /
[428113.059929] watchdog/5 31 -8.977217 107088 0
-8.977217 2499.019149 32.364766 /
[428113.060035] migration/5 32 0.000000 26537 0
0.000000 15739.945360 0.000845 /
[428113.060139] ksoftirqd/5 33 25422651.093839 16195 120
25422651.093839 166.009753 428070175.552405 /
[428113.060245] kworker/5:0 34 239.267711 17 120
239.267711 0.034507 632.432629 /
[428113.060350] kworker/5:0H 35 632.111736 6 100
632.111736 0.059549 2815.362585 /
[428113.060456] kworker/5:1 611 25423018.904992 2773160 120
25423018.904992 1288651.204421 426793243.196345 /
[428113.060561] scsi_tmf_2 860 534.239404 2 100
534.239404 0.008896 0.005460 /
[428113.060667] scsi_tmf_5 866 558.255161 2 100
558.255161 0.008179 0.004413 /
[428113.060772] md0_raid1 899 25396783.503701 32703 120
25396783.503701 998.952947 427489843.030132 /
[428113.060878] kworker/5:1H 1061 24847022.042452 45684 100
24847022.042452 510.479777 416592265.701502 /
[428113.060984] anvil 4282 1335.039244 546 120
1335.039244 25.298050 418304078.786475 /autogroup-17
[428113.061092]
[428113.061138] cpu#6, 3411.800 MHz
[428113.061186] .nr_running : 0
[428113.061236] .load : 0
[428113.061286] .nr_switches : 31375902
[428113.061338] .nr_load_updates : 4945717
[428113.061389] .nr_uninterruptible : 6930
[428113.061440] .next_balance : 4337.771028
[428113.061492] .curr->pid : 0
[428113.061542] .clock : 428112972.478131
[428113.061595] .cpu_load[0] : 0
[428113.061645] .cpu_load[1] : 0
[428113.061695] .cpu_load[2] : 0
[428113.061745] .cpu_load[3] : 0
[428113.061795] .cpu_load[4] : 0
[428113.061846] .yld_count : 20767
[428113.061896] .sched_count : 31528236
[428113.061947] .sched_goidle : 12333544
[428113.061999] .avg_idle : 1000000
[428113.062050] .max_idle_balance_cost : 500000
[428113.062100] .ttwu_count : 17745863
[428113.062152] .ttwu_local : 4311925
[428113.062203]
[428113.062203] cfs_rq[6]:/autogroup-6
[428113.062296] .exec_clock : 173835.323068
[428113.062348] .MIN_vruntime : 0.000001
[428113.062399] .min_vruntime : 163529.123269
[428113.062451] .max_vruntime : 0.000001
[428113.062502] .spread : 0.000000
[428113.062554] .spread0 : -106261204.225517
[428113.062606] .nr_spread_over : 80
[428113.062656] .nr_running : 0
[428113.062706] .load : 0
[428113.062756] .runnable_load_avg : 0
[428113.062807] .blocked_load_avg : 0
[428113.062857] .tg_load_contrib : 0
[428113.062907] .tg_runnable_contrib : 0
[428113.062957] .tg_load_avg : 46
[428113.063008] .tg->runnable_avg : 46
[428113.063058] .tg->cfs_bandwidth.timer_active: 0
[428113.063109] .throttled : 0
[428113.063160] .throttle_count : 0
[428113.063211] .se->exec_start : 428112972.478131
[428113.063263] .se->vruntime : 32014882.056897
[428113.063316] .se->sum_exec_runtime : 173853.720494
[428113.063368] .se->statistics.wait_start : 0.000000
[428113.063419] .se->statistics.sleep_start : 0.000000
[428113.063470] .se->statistics.block_start : 0.000000
[428113.063521] .se->statistics.sleep_max : 0.000000
[428113.063573] .se->statistics.block_max : 0.000000
[428113.063624] .se->statistics.exec_max : 10.006440
[428113.063675] .se->statistics.slice_max : 9.985799
[428113.063726] .se->statistics.wait_max : 14.987432
[428113.063778] .se->statistics.wait_sum : 5142.815457
[428113.063830] .se->statistics.wait_count : 888277
[428113.063881] .se->load.weight : 2
[428113.063932] .se->avg.runnable_avg_sum : 4
[428113.063982] .se->avg.runnable_avg_period : 48033
[428113.064033] .se->avg.load_avg_contrib : 0
[428113.064083] .se->avg.decay_count : 408280347
[428113.064134]
[428113.064134] cfs_rq[6]:/
[428113.064227] .exec_clock : 29301583.069476
[428113.064279] .MIN_vruntime : 0.000001
[428113.064330] .min_vruntime : 32014893.868429
[428113.064382] .max_vruntime : 0.000001
[428113.064433] .spread : 0.000000
[428113.064484] .spread0 : -74409839.480357
[428113.064537] .nr_spread_over : 413
[428113.064587] .nr_running : 0
[428113.064637] .load : 0
[428113.064688] .runnable_load_avg : 0
[428113.064738] .blocked_load_avg : 0
[428113.064789] .tg_load_contrib : 0
[428113.064839] .tg_runnable_contrib : 0
[428113.064890] .tg_load_avg : 2838
[428113.064941] .tg->runnable_avg : 2724
[428113.064992] .tg->cfs_bandwidth.timer_active: 0
[428113.065043] .throttled : 0
[428113.065093] .throttle_count : 0
[428113.065144] .avg->runnable_avg_sum : 4
[428113.065195] .avg->runnable_avg_period : 47831
[428113.065246]
[428113.065246] rt_rq[6]:/
[428113.065338] .rt_nr_running : 0
[428113.065389] .rt_throttled : 0
[428113.065439] .rt_time : 0.000000
[428113.065491] .rt_runtime : 950.000000
[428113.065542]
[428113.065542] dl_rq[6]:
[428113.065644] .dl_nr_running : 0
[428113.065702]
[428113.065702] runnable tasks:
[428113.065702] task PID tree-key switches prio
exec-runtime sum-exec sum-sleep
[428113.065702]
----------------------------------------------------------------------------------------------------------
[428113.065999] watchdog/6 36 -8.979370 107088 0
-8.979370 4171.996810 19.046238 /
[428113.066104] migration/6 37 0.000000 21452 0
0.000000 13182.758087 0.000826 /
[428113.066210] ksoftirqd/6 38 32009668.868629 19084 120
32009668.868629 335.465215 427985619.483782 /
[428113.066315] kworker/6:0 39 2652.785847 15 120
2652.785847 0.034665 619.334482 /
[428113.066420] kworker/6:0H 40 3222.777674 6 100
3222.777674 0.042578 34131.960421 /
[428113.066526] kworker/6:1 613 32014882.656425 3447588 120
32014882.656425 1715810.008839 426365265.308128 /
[428113.066631] bioset 898 2957.541580 2 100
2957.541580 0.008747 0.006389 /
[428113.066737] kworker/6:1H 1463 31224371.088945 50546 100
31224371.088945 612.901074 416298526.150915 /
[428113.066842] kjournald 2216 3410.060184 2 120
3410.060184 0.412389 0.043474 /
[428113.066948] bioset 2222 3422.126220 2 100
3422.126220 0.088585 0.040787 /
[428113.067053] mysqld 4102 163517.397318 1020343 120
163517.397318 10602.850133 428059785.215024 /autogroup-6
[428113.067160] mysqld 4122 163443.475621 644351 120
163443.475621 179019.020540 393898317.914127 /autogroup-6
[428113.068518] dovecot 4220 980.263394 2014 120
980.263394 114.537242 418305299.164771 /autogroup-17
[428113.068624] udevd 6137 246.142493 30 120
246.142493 6.670412 22056460.101598 /autogroup-4
[428113.068731] udevd 6138 242.586598 15 120
242.586598 3.149366 49194.013301 /autogroup-4
[428113.068838] btrfs-worker 6246 1051403.790298 2 100
1051403.790298 0.011158 0.004647 /
[428113.068943] btrfs-endio-met 6255 1051427.810805 2 100
1051427.810805 0.017787 0.002771 /
[428113.069049] btrfs-endio-rai 6257 1051439.820346 2 100
1051439.820346 0.010462 0.002911 /
[428113.069154] btrfs-endio-rep 6258 1051451.828334 2 100
1051451.828334 0.008652 0.003137 /
[428113.069259] btrfs-endio-wri 6260 1051463.836060 2 100
1051463.836060 0.008410 0.003085 /
[428113.069365] btrfs-readahead 6263 1051475.843882 2 100
1051475.843882 0.008543 0.003153 /
[428113.069470] btrfs-extent-re 6265 1051487.851432 2 100
1051487.851432 0.008154 0.003265 /
[428113.069578]
[428113.069632] cpu#7, 3411.800 MHz
[428113.069681] .nr_running : 0
[428113.069734] .load : 0
[428113.069786] .nr_switches : 28961216
[428113.069837] .nr_load_updates : 3537072
[428113.069891] .nr_uninterruptible : 5065
[428113.069943] .next_balance : 4337.771143
[428113.069996] .curr->pid : 0
[428113.070047] .clock : 428112555.897703
[428113.070101] .cpu_load[0] : 0
[428113.070153] .cpu_load[1] : 0
[428113.070211] .cpu_load[2] : 0
[428113.070261] .cpu_load[3] : 0
[428113.070312] .cpu_load[4] : 12
[428113.070363] .yld_count : 22779
[428113.070414] .sched_count : 29115700
[428113.070466] .sched_goidle : 12603487
[428113.070517] .avg_idle : 1000000
[428113.070569] .max_idle_balance_cost : 500000
[428113.070620] .ttwu_count : 18626268
[428113.070671] .ttwu_local : 2870358
[428113.070722]
[428113.070722] cfs_rq[7]:/
[428113.070814] .exec_clock : 14364516.198492
[428113.070867] .MIN_vruntime : 0.000001
[428113.070919] .min_vruntime : 18016573.070883
[428113.070971] .max_vruntime : 0.000001
[428113.071022] .spread : 0.000000
[428113.071073] .spread0 : -88408168.119657
[428113.071126] .nr_spread_over : 261
[428113.071176] .nr_running : 0
[428113.071226] .load : 0
[428113.071277] .runnable_load_avg : 0
[428113.071328] .blocked_load_avg : 0
[428113.071378] .tg_load_contrib : 0
[428113.071429] .tg_runnable_contrib : 0
[428113.071480] .tg_load_avg : 2910
[428113.071530] .tg->runnable_avg : 2849
[428113.071581] .tg->cfs_bandwidth.timer_active: 0
[428113.071632] .throttled : 0
[428113.071682] .throttle_count : 0
[428113.071732] .avg->runnable_avg_sum : 18
[428113.071783] .avg->runnable_avg_period : 47536
[428113.071834]
[428113.071834] rt_rq[7]:/
[428113.071926] .rt_nr_running : 0
[428113.071976] .rt_throttled : 0
[428113.072027] .rt_time : 0.000000
[428113.072078] .rt_runtime : 950.000000
[428113.072130]
[428113.072130] dl_rq[7]:
[428113.072222] .dl_nr_running : 0
[428113.072273]
[428113.072273] runnable tasks:
[428113.072273] task PID tree-key switches prio
exec-runtime sum-exec sum-sleep
[428113.072273]
----------------------------------------------------------------------------------------------------------
[428113.072570] watchdog/7 41 -8.982956 107089 0
-8.982956 1664.683687 5.725643 /
[428113.072674] migration/7 42 0.000000 21715 0
0.000000 13068.322347 0.000816 /
[428113.072780] ksoftirqd/7 43 18013052.310983 12328 120
18013052.310983 152.767844 428092090.014392 /
[428113.072885] kworker/7:0 44 479.385599 15 120
479.385599 0.031395 606.240559 /
[428113.072991] kworker/7:0H 45 1817.499047 6 100
1817.499047 0.047704 42727.098840 /
[428113.073096] netns 47 10.955152 2 100
10.955152 0.004076 0.001586 /
[428113.073201] perf 50 22.958573 2 100
22.958573 0.003794 0.001394 /
[428113.073306] kworker/7:1 615 18016561.430108 1960767 120
18016561.430108 793147.479614 427291244.310703 /
[428113.073413] bioset 919 1132.474026 2 100
1132.474026 0.034373 0.004008 /
[428113.073518] ext4-rsv-conver 936 1156.590460 2 100
1156.590460 0.061309 0.026946 /
[428113.073623] udevd 1124 405.566270 290 120
405.566270 53.586152 77350627.244245 /autogroup-4
[428113.073729] rsyslogd 3061 171793.939660 3747 120
171793.939660 155.674406 427956979.115875 /autogroup-6
[428113.073836] ipv6_addrconf 3209 1679.353321 2 100
1679.353321 0.006452 0.002581 /
[428113.073942] kworker/7:1H 4116 17737413.492285 52256 100
17737413.492285 541.503637 415562752.649104 /
[428113.074048] log 4283 586.242014 697 120
586.242014 34.396858 418304069.988435 /autogroup-17
[428113.074154] qmgr 4370 317.621444 2671 120
317.621444 299.720143 427913275.665071 /autogroup-19
[428113.074262] getty 4376 0.726364 103 120
0.726364 1.882302 146.996217 /autogroup-25
[428113.074369] btrfs-fixup 6253 1049025.643969 2 100
1049025.643969 0.011376 0.003865 /
[428113.074474] btrfs-endio 6254 1049037.659031 2 100
1049037.659031 0.015618 0.001516 /
[428113.074580] kworker/u17:1 23565 17782910.635845 2 100
17782910.635845 0.004742 0.030459 /
[428113.074685] pickup 23745 329.619461 38 120
329.619461 4.298483 2159811.305363 /autogroup-19
[428113.074792]
--
Tomasz Chmielewski
http://www.sslrack.com
prev parent reply other threads:[~2015-02-16 10:51 UTC|newest]
Thread overview: 3+ messages / expand[flat|nested] mbox.gz Atom feed top
2015-02-16 2:32 3.19 - "unable" to replace a failed drive - 100% CPU usage in kworker and btrfs-transacti Tomasz Chmielewski
2015-02-16 10:40 ` Liu Bo
2015-02-16 10:51 ` Tomasz Chmielewski [this message]
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=6830fb54424ede297a6349832bf46d00@admin.virtall.com \
--to=tch@virtall.com \
--cc=bo.li.liu@oracle.com \
--cc=linux-btrfs@vger.kernel.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox