From: Tomasz Chmielewski <tch@virtall.com>
To: bo.li.liu@oracle.com
Cc: linux-btrfs <linux-btrfs@vger.kernel.org>
Subject: Re: 3.19 - "unable" to replace a failed drive - 100% CPU usage in kworker and btrfs-transacti
Date: Mon, 16 Feb 2015 19:51:08 +0900 [thread overview]
Message-ID: <6830fb54424ede297a6349832bf46d00@admin.virtall.com> (raw)
In-Reply-To: <20150216104026.GC23683@localhost.localdomain>
On 2015-02-16 19:40, Liu Bo wrote:
>> PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
>> 6269 root 20 0 0 0 0 R 92.5 0.0 2769:33
>> btrfs-transacti
>> 22247 root 20 0 0 0 0 R 92.5 0.0 42:38.65
>> kworker/u16:16
>
> Can you cat /proc/22247/stack and /proc/6269/stack?
22247 no longer exists - there is a new kworker now:
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
23570 root 20 0 0 0 0 R 93.8 0.0 108:46.78
kworker/u16:15
6269 root 20 0 0 0 0 R 93.5 0.0 3029:41
btrfs-transacti
# cat /proc/22247/stack
cat: /proc/22247/stack: No such file or directory
# cat /proc/6269/stack
[<ffffffffa02a0b0d>] transaction_kthread+0x197/0x1c2 [btrfs]
[<ffffffff81050067>] kthread+0xcd/0xd5
[<ffffffff813c12ac>] ret_from_fork+0x7c/0xb0
[<ffffffffffffffff>] 0xffffffffffffffff
# cat /proc/23570/stack
[<ffffffffffffffff>] 0xffffffffffffffff
In case a new sysrq-w was needed, too:
[428113.007373] SysRq : Show Blocked State
[428113.007427] task PC stack pid father
[428113.007493] btrfs D ffff8802d67ef948 0 7611 6275
0x00000000
[428113.007549] ffff8802d67ef948 ffff8803fe0409d8 ffff88081917e040
00000000000113c0
[428113.007648] 0000000000004000 ffff88081be31810 ffff88081917e040
ffff8802d67ef8b8
[428113.007748] ffffffff8105f3c4 ffff88083fa80000 ffff88083fa513c0
ffff88083fa513c0
[428113.007847] Call Trace:
[428113.007900] [<ffffffff8105f3c4>] ? enqueue_task_fair+0x3e5/0x44f
[428113.007955] [<ffffffff81054ab6>] ? resched_curr+0x45/0x55
[428113.008008] [<ffffffff81055122>] ? check_preempt_curr+0x3e/0x6d
[428113.008062] [<ffffffff81055163>] ? ttwu_do_wakeup+0x12/0x7f
[428113.008115] [<ffffffff8105526e>] ?
ttwu_do_activate.constprop.73+0x57/0x5c
[428113.008172] [<ffffffff813be618>] schedule+0x65/0x67
[428113.008224] [<ffffffff813c03b8>] schedule_timeout+0x26/0x18d
[428113.008277] [<ffffffff81057d1e>] ? wake_up_process+0x30/0x34
[428113.008331] [<ffffffff8104b1a3>] ? wake_up_worker+0x1f/0x21
[428113.008383] [<ffffffff8104b3ea>] ? insert_work+0x87/0x94
[428113.008447] [<ffffffffa02e183d>] ? free_block_list+0x1f/0x34
[btrfs]
[428113.008501] [<ffffffff813bef76>] wait_for_common+0x10d/0x13e
[428113.008554] [<ffffffff81057cdf>] ? try_to_wake_up+0x250/0x250
[428113.008608] [<ffffffff813befbf>] wait_for_completion+0x18/0x1a
[428113.008666] [<ffffffffa028dedc>]
btrfs_async_run_delayed_refs+0xc1/0xe4 [btrfs]
[428113.008770] [<ffffffffa02a3189>]
__btrfs_end_transaction+0x315/0x33b [btrfs]
[428113.008872] [<ffffffffa02a31bd>]
btrfs_end_transaction_throttle+0xe/0x10 [btrfs]
[428113.008977] [<ffffffffa02e5cf1>] relocate_block_group+0x2ad/0x4de
[btrfs]
[428113.009037] [<ffffffffa02e607a>]
btrfs_relocate_block_group+0x158/0x278 [btrfs]
[428113.009141] [<ffffffffa02c37e1>]
btrfs_relocate_chunk.isra.69+0x35/0xa5 [btrfs]
[428113.009244] [<ffffffffa02c41da>] btrfs_shrink_device+0x235/0x408
[btrfs]
[428113.009304] [<ffffffffa02c6ba9>] btrfs_rm_device+0x2a9/0x704
[btrfs]
[428113.009359] [<ffffffff810f5441>] ?
__kmalloc_track_caller+0x40/0x178
[428113.009419] [<ffffffffa02cf7a7>] btrfs_ioctl+0xa9c/0x22f5 [btrfs]
[428113.009473] [<ffffffff8110f142>] ? putname+0x23/0x2c
[428113.009525] [<ffffffff8110f6d3>] ? user_path_at_empty+0x60/0x90
[428113.009579] [<ffffffff811783bf>] ? avc_has_perm+0x2e/0xf7
[428113.009632] [<ffffffff8111171b>] do_vfs_ioctl+0x418/0x460
[428113.009684] [<ffffffff81106745>] ? vfs_stat+0x16/0x18
[428113.009736] [<ffffffff811117b1>] SyS_ioctl+0x4e/0x7d
[428113.009789] [<ffffffff81031c5b>] ? do_page_fault+0xc/0x11
[428113.009841] [<ffffffff813c1352>] system_call_fastpath+0x12/0x17
[428113.009896] Sched Debug Version: v0.11, 3.19.0 #1
[428113.009947] ktime :
428338904.250198
[428113.010001] sched_clk :
428113009.895839
[428113.010054] cpu_clk :
428113009.895853
[428113.010107] jiffies : 4337771187
[428113.010160] sched_clock_stable() : 1
[428113.010212]
[428113.010257] sysctl_sched
[428113.010305] .sysctl_sched_latency : 24.000000
[428113.010358] .sysctl_sched_min_granularity : 3.000000
[428113.010411] .sysctl_sched_wakeup_granularity : 4.000000
[428113.010463] .sysctl_sched_child_runs_first : 0
[428113.010515] .sysctl_sched_features : 11899
[428113.010568] .sysctl_sched_tunable_scaling : 1
(logaritmic)
[428113.010621]
[428113.010667] cpu#0, 3411.800 MHz
[428113.010715] .nr_running : 0
[428113.010766] .load : 0
[428113.010817] .nr_switches : 141930054
[428113.010869] .nr_load_updates : 24939776
[428113.010920] .nr_uninterruptible : -739558
[428113.010972] .next_balance : 4337.771095
[428113.011023] .curr->pid : 0
[428113.011074] .clock : 428113007.169503
[428113.011126] .cpu_load[0] : 0
[428113.011176] .cpu_load[1] : 0
[428113.011227] .cpu_load[2] : 0
[428113.011277] .cpu_load[3] : 0
[428113.011328] .cpu_load[4] : 24
[428113.011379] .yld_count : 223168
[428113.011430] .sched_count : 142283739
[428113.011482] .sched_goidle : 51488452
[428113.011534] .avg_idle : 1000000
[428113.011584] .max_idle_balance_cost : 500000
[428113.011636] .ttwu_count : 88826891
[428113.011687] .ttwu_local : 54557139
[428113.011739]
[428113.011739] cfs_rq[0]:/autogroup-1201
[428113.011833] .exec_clock : 5.761094
[428113.011884] .MIN_vruntime : 0.000001
[428113.011936] .min_vruntime : 4.697019
[428113.011987] .max_vruntime : 0.000001
[428113.012038] .spread : 0.000000
[428113.012089] .spread0 : -106424728.651767
[428113.012141] .nr_spread_over : 0
[428113.012192] .nr_running : 0
[428113.012242] .load : 0
[428113.012292] .runnable_load_avg : 0
[428113.012343] .blocked_load_avg : 0
[428113.012394] .tg_load_contrib : 0
[428113.012445] .tg_runnable_contrib : 0
[428113.012495] .tg_load_avg : 0
[428113.012546] .tg->runnable_avg : 0
[428113.012596] .tg->cfs_bandwidth.timer_active: 0
[428113.012647] .throttled : 0
[428113.012698] .throttle_count : 0
[428113.012748] .se->exec_start : 428113007.164637
[428113.012801] .se->vruntime : 106424721.380976
[428113.012854] .se->sum_exec_runtime : 5.761094
[428113.012905] .se->statistics.wait_start : 0.000000
[428113.012956] .se->statistics.sleep_start : 0.000000
[428113.013007] .se->statistics.block_start : 0.000000
[428113.013059] .se->statistics.sleep_max : 0.000000
[428113.013110] .se->statistics.block_max : 0.000000
[428113.013161] .se->statistics.exec_max : 0.532615
[428113.013212] .se->statistics.slice_max : 0.000000
[428113.013263] .se->statistics.wait_max : 0.000000
[428113.013314] .se->statistics.wait_sum : 0.000000
[428113.013366] .se->statistics.wait_count : 84
[428113.013416] .se->load.weight : 2
[428113.013466] .se->avg.runnable_avg_sum : 31
[428113.013517] .se->avg.runnable_avg_period : 48192
[428113.013569] .se->avg.load_avg_contrib : 0
[428113.013619] .se->avg.decay_count : 408280380
[428113.013671]
[428113.013671] cfs_rq[0]:/
[428113.013764] .exec_clock : 108726735.130963
[428113.013817] .MIN_vruntime : 0.000001
[428113.013868] .min_vruntime : 106424733.348786
[428113.013920] .max_vruntime : 0.000001
[428113.013971] .spread : 0.000000
[428113.014023] .spread0 : 0.000000
[428113.014074] .nr_spread_over : 291
[428113.014124] .nr_running : 0
[428113.014175] .load : 0
[428113.014225] .runnable_load_avg : 0
[428113.014276] .blocked_load_avg : 0
[428113.014326] .tg_load_contrib : 0
[428113.014377] .tg_runnable_contrib : 0
[428113.014428] .tg_load_avg : 2125
[428113.014479] .tg->runnable_avg : 2029
[428113.015797] .tg->cfs_bandwidth.timer_active: 0
[428113.015847] .throttled : 0
[428113.015897] .throttle_count : 0
[428113.015948] .avg->runnable_avg_sum : 34
[428113.015999] .avg->runnable_avg_period : 48420
[428113.016052]
[428113.016052] rt_rq[0]:/
[428113.016145] .rt_nr_running : 0
[428113.016195] .rt_throttled : 0
[428113.016246] .rt_time : 0.000000
[428113.016297] .rt_runtime : 950.000000
[428113.016349]
[428113.016349] dl_rq[0]:
[428113.016441] .dl_nr_running : 0
[428113.016491]
[428113.016491] runnable tasks:
[428113.016491] task PID tree-key switches prio
exec-runtime sum-exec sum-sleep
[428113.016491]
----------------------------------------------------------------------------------------------------------
[428113.016788] ksoftirqd/0 3 106424458.464162 754529 120
106424458.464162 12412.024323 428020900.243547 /
[428113.016893] kworker/0:0 4 96872056.230102 54 120
96872056.230102 17.312081 398174954.738190 /
[428113.016998] kworker/0:0H 5 1015.287609 5 100
1015.287609 0.041666 1136.081722 /
[428113.017103] rcu_bh 8 117.354314 2 120
117.354314 0.000874 0.000722 /
[428113.017209] migration/0 9 0.000000 42259 0
0.000000 38663.891221 0.000853 /
[428113.017314] watchdog/0 10 -11.786179 107094 0
-11.786179 18301.767349 93.369695 /
[428113.017421] acpi_thermal_pm 618 478.127841 2 100
478.127841 0.008079 0.004241 /
[428113.017527] ata_sff 832 943.140921 2 100
943.140921 0.005880 0.022278 /
[428113.017632] scsi_eh_0 855 1012.563036 18 120
1012.563036 0.875745 352.505402 /
[428113.017738] kworker/0:1H 879 106418747.237080 19756283 100
106418747.237080 936534.886602 427013679.873055 /
[428113.017843] md2_raid1 920 106418747.384047 5312046 120
106418747.384047 235466.210537 427856341.930987 /
[428113.017950] jbd2/md2-8 935 106418747.368760 676645 120
106418747.368760 26050.167995 428067576.860650 /
[428113.018055] rs:main Q:Reg 3060 1156937.947188 3986 120
1156937.947188 610.549422 427956529.353145 /autogroup-6
[428113.018162] rsyslogd 3062 1154980.137945 580 120
1154980.137945 33.203062 397909885.430577 /autogroup-6
[428113.018269] vnstatd 3041 1704.807054 86943 120
1704.807054 8238.508382 428061424.035922 /autogroup-7
[428113.018377] mdadm 3144 27.184928 450 120
27.184928 68.147482 427453633.845909 /autogroup-8
[428113.018483] atd 3189 0.694263 120 120
0.694263 2.561413 424576013.244002 /autogroup-9
[428113.018591] nginx 3232 4317.191565 21 120
4317.191565 1.526614 408639452.655116 /autogroup-10
[428113.018697] nginx 3233 4489.469136 520373 120
4489.469136 6666.843000 427177069.798260 /autogroup-10
[428113.018804] mysqld 4080 1156391.999838 5706 120
1156391.999838 804.810183 419298520.527262 /autogroup-6
[428113.018911] mysqld 4121 1156945.409829 85595 120
1156945.409829 1240.826037 428066936.224975 /autogroup-6
[428113.019017] mysqld 4154 1156391.428236 2178575 120
1156391.428236 373082.741411 418907550.518894 /autogroup-6
[428113.019124] mysqld 4393 1156391.413224 1651105 120
1156391.413224 338375.528086 418894108.235915 /autogroup-6
[428113.019230] mysqld 5811 1156391.734934 1317689 120
1156391.734934 299565.417020 370162247.292961 /autogroup-6
[428113.019337] mysqld 15425 1156391.434917 261945 120
1156391.434917 145838.371509 198220505.720438 /autogroup-6
[428113.019443] mysqld 15436 1156391.985920 292544 120
1156391.985920 172007.149326 198170039.192913 /autogroup-6
[428113.019550] getty 4374 0.817181 100 120
0.817181 2.613716 146.560614 /autogroup-23
[428113.019657] bash 6275 132.300150 276 120
132.300150 79.740920 21744924.508065 /autogroup-184
[428113.019764] kworker/0:2 22822 106424721.355311 1310427 120
106424721.355311 592055.669533 29583164.785882 /
[428113.019869] sshd 23685 4.429642 75 120
4.429642 10.486741 908580.581156 /autogroup-1187
[428113.019976] bash 23687 63.590364 35 120
63.590364 25.369673 13.977239 /autogroup-1188
[428113.020083] sshd 23875 4.697019 109 120
4.697019 10.704193 119911.149251 /autogroup-1201
[428113.020199]
[428113.020246] cpu#1, 3411.800 MHz
[428113.020302] .nr_running : 1
[428113.020354] .load : 1024
[428113.020407] .nr_switches : 88720888
[428113.020461] .nr_load_updates : 23703925
[428113.020515] .nr_uninterruptible : 483951
[428113.020568] .next_balance : 4337.771189
[428113.020621] .curr->pid : 3060
[428113.020675] .clock : 428113020.257470
[428113.020727] .cpu_load[0] : 0
[428113.020786] .cpu_load[1] : 0
[428113.020836] .cpu_load[2] : 0
[428113.020887] .cpu_load[3] : 0
[428113.020937] .cpu_load[4] : 0
[428113.020988] .yld_count : 135412
[428113.021039] .sched_count : 89125261
[428113.021090] .sched_goidle : 30252689
[428113.021141] .avg_idle : 1000000
[428113.021192] .max_idle_balance_cost : 500000
[428113.021244] .ttwu_count : 38652568
[428113.021295] .ttwu_local : 21074490
[428113.021346]
[428113.021346] cfs_rq[1]:/autogroup-183
[428113.021440] .exec_clock : 10198.070166
[428113.021492] .MIN_vruntime : 0.000001
[428113.021543] .min_vruntime : 10197.021590
[428113.021595] .max_vruntime : 0.000001
[428113.021646] .spread : 0.000000
[428113.021697] .spread0 : -106414536.327196
[428113.021749] .nr_spread_over : 0
[428113.021800] .nr_running : 0
[428113.021850] .load : 0
[428113.021900] .runnable_load_avg : 0
[428113.021951] .blocked_load_avg : 0
[428113.022001] .tg_load_contrib : 0
[428113.022052] .tg_runnable_contrib : 0
[428113.022103] .tg_load_avg : 0
[428113.022154] .tg->runnable_avg : 0
[428113.022204] .tg->cfs_bandwidth.timer_active: 0
[428113.022255] .throttled : 0
[428113.022305] .throttle_count : 0
[428113.022356] .se->exec_start : 428113015.541150
[428113.022409] .se->vruntime : 129995516.110494
[428113.022461] .se->sum_exec_runtime : 10198.307701
[428113.022513] .se->statistics.wait_start : 0.000000
[428113.022564] .se->statistics.sleep_start : 0.000000
[428113.022615] .se->statistics.block_start : 0.000000
[428113.022666] .se->statistics.sleep_max : 0.000000
[428113.022717] .se->statistics.block_max : 0.000000
[428113.022769] .se->statistics.exec_max : 2.939902
[428113.022820] .se->statistics.slice_max : 2.075783
[428113.022871] .se->statistics.wait_max : 5.458038
[428113.022922] .se->statistics.wait_sum : 30.977291
[428113.022973] .se->statistics.wait_count : 234554
[428113.023024] .se->load.weight : 2
[428113.023074] .se->avg.runnable_avg_sum : 34
[428113.023125] .se->avg.runnable_avg_period : 47537
[428113.023176] .se->avg.load_avg_contrib : 0
[428113.023226] .se->avg.decay_count : 408280388
[428113.023278]
[428113.023278] cfs_rq[1]:/autogroup-6
[428113.023372] .exec_clock : 808878.880122
[428113.023424] .MIN_vruntime : 0.000001
[428113.023476] .min_vruntime : 692298.789871
[428113.023528] .max_vruntime : 0.000001
[428113.023579] .spread : 0.000000
[428113.023630] .spread0 : -105732434.558915
[428113.023683] .nr_spread_over : 259
[428113.023734] .nr_running : 0
[428113.023784] .load : 0
[428113.023834] .runnable_load_avg : 0
[428113.023885] .blocked_load_avg : 0
[428113.023935] .tg_load_contrib : 0
[428113.023986] .tg_runnable_contrib : 0
[428113.024036] .tg_load_avg : 0
[428113.024087] .tg->runnable_avg : 5
[428113.024138] .tg->cfs_bandwidth.timer_active: 0
[428113.024188] .throttled : 0
[428113.024239] .throttle_count : 0
[428113.024290] .se->exec_start : 428113020.747637
[428113.024342] .se->vruntime : 129995520.664159
[428113.024395] .se->sum_exec_runtime : 808905.454732
[428113.024446] .se->statistics.wait_start : 0.000000
[428113.024497] .se->statistics.sleep_start : 0.000000
[428113.024548] .se->statistics.block_start : 0.000000
[428113.024600] .se->statistics.sleep_max : 0.000000
[428113.024651] .se->statistics.block_max : 0.000000
[428113.024702] .se->statistics.exec_max : 10.030247
[428113.024754] .se->statistics.slice_max : 10.135075
[428113.024805] .se->statistics.wait_max : 14.316925
[428113.024856] .se->statistics.wait_sum : 6528.008676
[428113.024908] .se->statistics.wait_count : 4854101
[428113.024959] .se->load.weight : 2
[428113.025009] .se->avg.runnable_avg_sum : 489
[428113.025059] .se->avg.runnable_avg_period : 47586
[428113.026377] .se->avg.load_avg_contrib : 0
[428113.026428] .se->avg.decay_count : 408280393
[428113.026479]
[428113.026479] cfs_rq[1]:/
[428113.026571] .exec_clock : 134248386.191002
[428113.026624] .MIN_vruntime : 0.000001
[428113.026676] .min_vruntime : 129995526.942391
[428113.026728] .max_vruntime : 0.000001
[428113.026780] .spread : 0.000000
[428113.026831] .spread0 : 23570793.593605
[428113.026883] .nr_spread_over : 311
[428113.026933] .nr_running : 0
[428113.026984] .load : 0
[428113.027035] .runnable_load_avg : 0
[428113.027085] .blocked_load_avg : 0
[428113.027136] .tg_load_contrib : 0
[428113.027186] .tg_runnable_contrib : 10
[428113.027237] .tg_load_avg : 2353
[428113.027288] .tg->runnable_avg : 2284
[428113.027339] .tg->cfs_bandwidth.timer_active: 0
[428113.027389] .throttled : 0
[428113.027439] .throttle_count : 0
[428113.027490] .avg->runnable_avg_sum : 492
[428113.027540] .avg->runnable_avg_period : 47214
[428113.027591]
[428113.027591] rt_rq[1]:/
[428113.027683] .rt_nr_running : 0
[428113.027733] .rt_throttled : 0
[428113.027783] .rt_time : 0.000000
[428113.027834] .rt_runtime : 950.000000
[428113.027886]
[428113.027886] dl_rq[1]:
[428113.027978] .dl_nr_running : 0
[428113.028029]
[428113.028029] runnable tasks:
[428113.028029] task PID tree-key switches prio
exec-runtime sum-exec sum-sleep
[428113.028029]
----------------------------------------------------------------------------------------------------------
[428113.028324] kthreadd 2 129745695.820738 1414 120
129745695.820738 41.527944 427299434.064971 /
[428113.028430] watchdog/1 11 -2.973570 107088 0
-2.973570 23044.650834 85.639910 /
[428113.028535] migration/1 12 0.000000 38706 0
0.000000 32310.384635 0.000758 /
[428113.028641] ksoftirqd/1 13 129993260.237901 96666 120
129993260.237901 1796.063974 428051863.708966 /
[428113.028746] kworker/1:0H 15 582.372111 6 100
582.372111 0.027131 2733.616385 /
[428113.028852] kpsmoused 686 25.893051 2 100
25.893051 0.010838 0.003636 /
[428113.028957] kworker/1:2 688 102920251.057920 55 120
102920251.057920 0.504998 355091633.648033 /
[428113.029062] deferwq 731 158.015576 2 100
158.015576 0.009076 0.003883 /
[428113.029167] scsi_tmf_0 856 442.536328 2 100
442.536328 0.008945 0.003987 /
[428113.029273] scsi_eh_1 857 539.461151 16 120
539.461151 0.868588 350.508685 /
[428113.029378] scsi_tmf_1 858 466.554669 2 100
466.554669 0.008786 0.003948 /
[428113.029484] scsi_eh_2 859 514.602593 3 120
514.602593 0.016414 1.372299 /
[428113.029589] scsi_eh_3 861 514.603810 3 120
514.603810 0.016837 1.222996 /
[428113.029694] scsi_tmf_3 862 502.578935 2 100
502.578935 0.008052 0.003896 /
[428113.029798] scsi_tmf_4 864 514.587500 2 100
514.587500 0.008905 0.004404 /
[428113.029904] kworker/1:1H 968 129831013.397442 456157 100
129831013.397442 6526.830851 427481927.625721 /
[428113.030009] rsyslogd 3033 691065.336881 29 120
691065.336881 2.741841 408640738.003949 /autogroup-6
[428113.030116] rs:main Q:Reg 3060 692287.280038 3987 120
692287.280038 611.039589 428073924.132108 /autogroup-6
[428113.030223] nginx 3234 4197.283423 855782 120
4197.283423 11301.321120 428062021.947465 /autogroup-10
[428113.030330] nginx 3235 4197.323950 684268 120
4197.323950 9289.505173 428064027.946907 /autogroup-10
[428113.030436] cron 3301 927.592784 8289 120
927.592784 433.794782 428068473.223910 /autogroup-12
[428113.030543] ntpd 3378 2401.387432 430091 120
2401.387432 12164.876249 428060869.176748 /autogroup-13
[428113.030650] mysqld_safe 3464 1418.978306 435 120
1418.978306 26.375085 974.715839 /autogroup-6
[428113.030756] mysqld 4094 692286.802491 856622 120
692286.802491 7997.578025 428062557.688191 /autogroup-6
[428113.030863] mysqld 4095 692286.812618 908471 120
692286.812618 10016.493168 428060177.274007 /autogroup-6
[428113.030969] mysqld 4096 692286.794838 1340772 120
692286.794838 56672.194508 428011328.939086 /autogroup-6
[428113.031075] mysqld 4097 692286.802298 1314182 120
692286.802298 55304.687185 428012714.776737 /autogroup-6
[428113.031182] mysqld 4098 692286.798431 1342580 120
692286.798431 57725.305541 428009725.287820 /autogroup-6
[428113.031289] mysqld 4101 692286.801904 1083476 120
692286.801904 11507.690040 428058770.056113 /autogroup-6
[428113.031395] mysqld 4103 692286.796803 1402041 120
692286.796803 16514.909612 428053422.705042 /autogroup-6
[428113.031501] mysqld 4118 692286.906040 430988 120
692286.906040 54290.532636 428014960.090656 /autogroup-6
[428113.031608] mysqld 4119 692298.789871 434800 120
692298.789871 127525.290540 427941309.545286 /autogroup-6
[428113.031715] mysqld 4120 692285.773389 85683 120
692285.773389 1366.692133 428064373.232093 /autogroup-6
[428113.031822] mysqld 4396 692171.636633 1079842 120
692171.636633 305192.243262 426206373.814546 /autogroup-6
[428113.031930] mysqld 15426 691721.033869 784935 120
691721.033869 137131.954258 198224493.143598 /autogroup-6
[428113.032037] getty 4372 0.769188 104 120
0.769188 2.833955 146.554963 /autogroup-21
[428113.032144] btrfs-delalloc 6249 120544673.387058 4 100
120544673.387058 12.566284 345061022.997020 /
[428113.032250] tmux 6274 10197.021590 1077829 120
10197.021590 47181.472538 372426592.567655 /autogroup-183
[428113.032356] kworker/1:0 21019 129995518.151853 3521622 120
129995518.151853 1682209.369392 71323049.000940 /
[428113.032462] sshd 23755 4.481966 346 120
4.481966 33.939190 1200991.721059 /autogroup-1194
[428113.032569] bash 23757 19.596943 121 120
19.596943 36.036523 1106982.698162 /autogroup-1195
[428113.032676] sshd 23823 0.668354 29 120
0.668354 7.093561 202985.617679 /autogroup-1199
[428113.032785]
[428113.032833] cpu#2, 3411.800 MHz
[428113.032891] .nr_running : 1
[428113.032943] .load : 1024
[428113.032995] .nr_switches : 81057822
[428113.033048] .nr_load_updates : 21474857
[428113.033100] .nr_uninterruptible : 140117
[428113.033154] .next_balance : 4337.771194
[428113.033207] .curr->pid : 23570
[428113.033258] .clock : 428113026.128231
[428113.033319] .cpu_load[0] : 1023
[428113.033370] .cpu_load[1] : 1023
[428113.033420] .cpu_load[2] : 1023
[428113.033471] .cpu_load[3] : 1023
[428113.033521] .cpu_load[4] : 1023
[428113.033572] .yld_count : 92399
[428113.033623] .sched_count : 81351574
[428113.033674] .sched_goidle : 28607226
[428113.033725] .avg_idle : 586733
[428113.033776] .max_idle_balance_cost : 500000
[428113.033827] .ttwu_count : 37353396
[428113.033878] .ttwu_local : 18907401
[428113.033929]
[428113.033929] cfs_rq[2]:/
[428113.034021] .exec_clock : 112244043.135162
[428113.034073] .MIN_vruntime : 0.000001
[428113.034125] .min_vruntime : 109548478.645357
[428113.034177] .max_vruntime : 0.000001
[428113.034228] .spread : 0.000000
[428113.034279] .spread0 : 3123745.296571
[428113.034331] .nr_spread_over : 470
[428113.034382] .nr_running : 1
[428113.034432] .load : 1024
[428113.034483] .runnable_load_avg : 1023
[428113.034533] .blocked_load_avg : 44
[428113.034584] .tg_load_contrib : 1056
[428113.034634] .tg_runnable_contrib : 1010
[428113.034685] .tg_load_avg : 2561
[428113.034736] .tg->runnable_avg : 2477
[428113.034787] .tg->cfs_bandwidth.timer_active: 0
[428113.034837] .throttled : 0
[428113.034888] .throttle_count : 0
[428113.034938] .avg->runnable_avg_sum : 47622
[428113.034990] .avg->runnable_avg_period : 47622
[428113.035041]
[428113.035041] rt_rq[2]:/
[428113.035132] .rt_nr_running : 0
[428113.035183] .rt_throttled : 0
[428113.035233] .rt_time : 0.000000
[428113.035284] .rt_runtime : 950.000000
[428113.035336]
[428113.035336] dl_rq[2]:
[428113.035427] .dl_nr_running : 0
[428113.035478]
[428113.035478] runnable tasks:
[428113.035478] task PID tree-key switches prio
exec-runtime sum-exec sum-sleep
[428113.035478]
----------------------------------------------------------------------------------------------------------
[428113.037044] watchdog/2 16 -5.976382 107088 0
-5.976382 19052.269468 72.326265 /
[428113.037149] migration/2 17 0.000000 34097 0
0.000000 32002.834817 0.001139 /
[428113.037254] ksoftirqd/2 18 109545990.177739 74000 120
109545990.177739 2163.416662 428073696.139572 /
[428113.037359] kworker/2:0 19 100410070.169673 40 120
100410070.169673 0.134368 398174981.123393 /
[428113.037465] kworker/2:0H 20 3650.646229 6 100
3650.646229 0.037533 34187.208217 /
[428113.037571] khungtaskd 204 109533300.730792 3571 120
109533300.730792 195.253551 428053978.079872 /
[428113.037676] kswapd0 506 106048732.107333 11160 120
106048732.107333 5508.370790 416651549.349593 /
[428113.037782] scsi_eh_4 863 3072.735732 3 120
3072.735732 0.015446 1.048287 /
[428113.037887] scsi_eh_5 865 3072.733721 3 120
3072.733721 0.012919 0.910030 /
[428113.037993] kworker/2:1H 1466 108820462.412649 394876 100
108820462.412649 5278.985782 425855655.769615 /
[428113.038098] acpid 3240 0.392530 49 120
0.392530 1.816842 407504916.577611 /autogroup-11
[428113.038205] dbus-daemon 3404 0.942824 6 120
0.942824 1.484584 55885009.390813 /autogroup-14
[428113.038313] mysqld 4150 460431.045860 7023989 120
460431.045860 1407433.937451 407178531.151580 /autogroup-6
[428113.038419] master 4345 398.492132 21611 120
398.492132 1062.087947 428043837.677636 /autogroup-19
[428113.038526] getty 4373 0.640858 102 120
0.640858 2.425952 146.827128 /autogroup-22
[428113.038633] btrfs-worker-hi 6248 1154879.053331 2 100
1154879.053331 0.017567 0.003220 /
[428113.038739] btrfs-flush_del 6250 1154903.071666 2 100
1154903.071666 0.009022 0.003320 /
[428113.038844] btrfs-cache 6251 1154915.081079 2 100
1154915.081079 0.015404 0.003203 /
[428113.038949] btrfs-submit 6252 1154915.162054 2 100
1154915.162054 0.096237 0.023172 /
[428113.039054] btrfs-endio-met 6256 1154927.254127 2 100
1154927.254127 0.093435 0.002869 /
[428113.039160] btrfs-rmw 6259 1154939.263666 2 100
1154939.263666 0.010258 0.003176 /
[428113.039265] btrfs-freespace 6261 1154951.272823 2 100
1154951.272823 0.009779 0.003215 /
[428113.039371] btrfs-delayed-m 6262 1154963.281556 2 100
1154963.281556 0.009338 0.003128 /
[428113.039475] btrfs-qgroup-re 6264 1154975.290187 2 100
1154975.290187 0.009265 0.003113 /
[428113.039580] btrfs 7611 332657.739906 2080440 120
332657.739906 1556211.983521 349088605.012752 /autogroup-184
[428113.039687] kworker/2:1 22833 109548476.671264 1253598 120
109548476.671264 577663.225799 29354960.583584 /
[428113.039793] kworker/u17:0 23543 106131220.757333 22 100
106131220.757333 0.419296 905976.409476 /
[428113.039898] R kworker/u16:15 23570 109548488.151519 847746 120
109548488.151519 6605807.116828 4151195.288717 /
[428113.040006]
[428113.040052] cpu#3, 3411.800 MHz
[428113.040108] .nr_running : 2
[428113.040159] .load : 2048
[428113.040211] .nr_switches : 82481964
[428113.040262] .nr_load_updates : 22241954
[428113.040322] .nr_uninterruptible : 84834
[428113.040374] .next_balance : 4337.771192
[428113.040426] .curr->pid : 23877
[428113.040477] .clock : 428113040.005657
[428113.040529] .cpu_load[0] : 501
[428113.040580] .cpu_load[1] : 385
[428113.040630] .cpu_load[2] : 237
[428113.040681] .cpu_load[3] : 132
[428113.040732] .cpu_load[4] : 70
[428113.040783] .yld_count : 73921
[428113.040834] .sched_count : 82696221
[428113.040885] .sched_goidle : 28975510
[428113.040936] .avg_idle : 1000000
[428113.040988] .max_idle_balance_cost : 500000
[428113.041039] .ttwu_count : 39104815
[428113.041090] .ttwu_local : 19561240
[428113.041141]
[428113.041141] cfs_rq[3]:/autogroup-1202
[428113.041235] .exec_clock : 46.587190
[428113.041287] .MIN_vruntime : 0.000001
[428113.041339] .min_vruntime : 117.495213
[428113.041390] .max_vruntime : 0.000001
[428113.041441] .spread : 0.000000
[428113.041493] .spread0 : -106424615.853573
[428113.041545] .nr_spread_over : 1
[428113.041595] .nr_running : 1
[428113.041645] .load : 1024
[428113.041696] .runnable_load_avg : 501
[428113.041747] .blocked_load_avg : 0
[428113.041797] .tg_load_contrib : 501
[428113.041848] .tg_runnable_contrib : 503
[428113.041899] .tg_load_avg : 501
[428113.041950] .tg->runnable_avg : 503
[428113.042001] .tg->cfs_bandwidth.timer_active: 0
[428113.042052] .throttled : 0
[428113.042102] .throttle_count : 0
[428113.042153] .se->exec_start : 428113040.005657
[428113.042206] .se->vruntime : 112122172.726168
[428113.042258] .se->sum_exec_runtime : 46.587190
[428113.042309] .se->statistics.wait_start : 0.000000
[428113.042361] .se->statistics.sleep_start : 0.000000
[428113.042412] .se->statistics.block_start : 0.000000
[428113.042463] .se->statistics.sleep_max : 0.000000
[428113.042514] .se->statistics.block_max : 0.000000
[428113.042565] .se->statistics.exec_max : 12.895535
[428113.042616] .se->statistics.slice_max : 0.000000
[428113.042667] .se->statistics.wait_max : 0.012418
[428113.042719] .se->statistics.wait_sum : 0.012418
[428113.042770] .se->statistics.wait_count : 32
[428113.042820] .se->load.weight : 1024
[428113.042871] .se->avg.runnable_avg_sum : 23121
[428113.042922] .se->avg.runnable_avg_period : 47024
[428113.042972] .se->avg.load_avg_contrib : 501
[428113.043023] .se->avg.decay_count : 0
[428113.043073]
[428113.043073] cfs_rq[3]:/autogroup-6
[428113.043167] .exec_clock : 489341.658980
[428113.043219] .MIN_vruntime : 0.000001
[428113.043271] .min_vruntime : 406937.010943
[428113.043323] .max_vruntime : 0.000001
[428113.043374] .spread : 0.000000
[428113.043425] .spread0 : -106017796.337843
[428113.043477] .nr_spread_over : 245
[428113.043528] .nr_running : 0
[428113.043578] .load : 0
[428113.043628] .runnable_load_avg : 0
[428113.043679] .blocked_load_avg : 0
[428113.043730] .tg_load_contrib : 0
[428113.043781] .tg_runnable_contrib : 0
[428113.043831] .tg_load_avg : 27
[428113.043882] .tg->runnable_avg : 27
[428113.043933] .tg->cfs_bandwidth.timer_active: 0
[428113.043983] .throttled : 0
[428113.044033] .throttle_count : 0
[428113.044083] .se->exec_start : 428112971.994259
[428113.044135] .se->vruntime : 112122139.240601
[428113.044188] .se->sum_exec_runtime : 489369.503771
[428113.044240] .se->statistics.wait_start : 0.000000
[428113.044292] .se->statistics.sleep_start : 0.000000
[428113.044343] .se->statistics.block_start : 0.000000
[428113.044394] .se->statistics.sleep_max : 0.000000
[428113.044445] .se->statistics.block_max : 0.000000
[428113.044496] .se->statistics.exec_max : 9.999422
[428113.044548] .se->statistics.slice_max : 9.991785
[428113.044599] .se->statistics.wait_max : 19.987876
[428113.044650] .se->statistics.wait_sum : 7372.829518
[428113.044702] .se->statistics.wait_count : 4009170
[428113.044753] .se->load.weight : 2
[428113.044803] .se->avg.runnable_avg_sum : 2
[428113.044853] .se->avg.runnable_avg_period : 47424
[428113.044904] .se->avg.load_avg_contrib : 0
[428113.044955] .se->avg.decay_count : 408280346
[428113.045006]
[428113.045006] cfs_rq[3]:/
[428113.045098] .exec_clock : 114980390.622745
[428113.045151] .MIN_vruntime : 112122139.396351
[428113.045203] .min_vruntime : 112122151.162594
[428113.045256] .max_vruntime : 112122139.396351
[428113.045308] .spread : 0.000000
[428113.045359] .spread0 : 5697417.813808
[428113.045411] .nr_spread_over : 342
[428113.045461] .nr_running : 2
[428113.045511] .load : 2048
[428113.045562] .runnable_load_avg : 501
[428113.045620] .blocked_load_avg : 0
[428113.045682] .tg_load_contrib : 501
[428113.045733] .tg_runnable_contrib : 502
[428113.045784] .tg_load_avg : 2611
[428113.045835] .tg->runnable_avg : 2561
[428113.045891] .tg->cfs_bandwidth.timer_active: 0
[428113.045941] .throttled : 0
[428113.045992] .throttle_count : 0
[428113.046042] .avg->runnable_avg_sum : 23288
[428113.046092] .avg->runnable_avg_period : 47497
[428113.046144]
[428113.046144] rt_rq[3]:/
[428113.047487] .rt_nr_running : 0
[428113.047537] .rt_throttled : 0
[428113.047587] .rt_time : 0.000000
[428113.047639] .rt_runtime : 950.000000
[428113.047690]
[428113.047690] dl_rq[3]:
[428113.047782] .dl_nr_running : 0
[428113.047832]
[428113.047832] runnable tasks:
[428113.047832] task PID tree-key switches prio
exec-runtime sum-exec sum-sleep
[428113.047832]
----------------------------------------------------------------------------------------------------------
[428113.048127] init 1 586.822029 85839 120
586.822029 2768.064315 428108381.243581 /autogroup-2
[428113.048233] watchdog/3 21 -5.984614 107089 0
-5.984614 19753.484076 58.998123 /
[428113.048338] migration/3 22 0.000000 33711 0
0.000000 33100.318561 0.000847 /
[428113.048443] ksoftirqd/3 23 112114355.272976 66314 120
112114355.272976 1782.976803 428053105.661252 /
[428113.048548] kworker/3:0 24 33082955.717338 56 120
33082955.717338 0.596786 176596062.108639 /
[428113.048653] kworker/3:0H 25 1600.396479 6 100
1600.396479 0.027776 33504.961049 /
[428113.048758] khelper 46 10.955864 2 100
10.955864 0.005052 0.001801 /
[428113.048864] writeback 205 311.132001 2 100
311.132001 0.003788 0.001292 /
[428113.048969] crypto 208 323.135116 2 100
323.135116 0.003395 0.001548 /
[428113.049074] bioset 209 335.138067 2 100
335.138067 0.003268 0.001389 /
[428113.049179] kblockd 210 347.141115 2 100
347.141115 0.003369 0.001350 /
[428113.049284] md 361 359.144676 2 100
359.144676 0.003908 0.001863 /
[428113.049389] fsnotify_mark 576 1285553.304174 56 120
1285553.304174 1.356477 77382968.604008 /
[428113.049495] bioset 909 1410.076226 2 100
1410.076226 0.008348 0.004053 /
[428113.049600] kworker/3:1H 1233 109291133.379639 383443 100
109291133.379639 4828.980689 418283340.722864 /
[428113.049706] nginx 3236 5458.045701 513869 120
5458.045701 6979.077801 428066409.223752 /autogroup-10
[428113.049814] sshd 3426 0.942592 37 120
0.942592 4.235482 427955415.003546 /autogroup-15
[428113.049920] mysqld 4099 406925.050358 1297863 120
406925.050358 54242.559211 428013387.543354 /autogroup-6
[428113.050027] mysqld 4100 406925.049535 1470492 120
406925.049535 17668.621861 428052266.284702 /autogroup-6
[428113.050135] mysqld 4123 260297.226173 260619 120
260297.226173 48020.432086 156765824.444851 /autogroup-6
[428113.050242] mysqld 15435 406273.301208 228083 120
406273.301208 118653.052456 198232617.393831 /autogroup-6
[428113.050348] getty 4375 0.839336 103 120
0.839336 1.796678 147.205983 /autogroup-24
[428113.050455] btrfs-cleaner 6268 112107699.597918 22244 120
112107699.597918 5420.971640 372440029.550711 /
[428113.050561] kworker/3:1 13622 112122139.396351 10957597 120
112122139.396351 4812976.705719 246653995.669097 /
[428113.050666] ssl-params 23599 422.267712 4 120
422.267712 0.893866 0.114741 /autogroup-17
[428113.050773] kworker/u16:0 23680 112122139.491207 171869 120
112122139.491207 1346238.717160 2149270.672568 /
[428113.050879] bash 23825 114.365698 37 120
114.365698 25.754806 13.396110 /autogroup-1200
[428113.050986] R bash 23877 117.495213 81 120
117.495213 63.565396 116682.313391 /autogroup-1202
[428113.051095]
[428113.051143] cpu#4, 3411.800 MHz
[428113.051200] .nr_running : 1
[428113.051253] .load : 1024
[428113.051305] .nr_switches : 32118320
[428113.051357] .nr_load_updates : 3723003
[428113.051409] .nr_uninterruptible : 12676
[428113.051461] .next_balance : 4337.771202
[428113.051514] .curr->pid : 6269
[428113.051565] .clock : 428113046.666634
[428113.051617] .cpu_load[0] : 1023
[428113.051668] .cpu_load[1] : 1023
[428113.051718] .cpu_load[2] : 1023
[428113.051769] .cpu_load[3] : 1023
[428113.051820] .cpu_load[4] : 1003
[428113.051879] .yld_count : 30496
[428113.051930] .sched_count : 32427582
[428113.051981] .sched_goidle : 13462549
[428113.052032] .avg_idle : 1000000
[428113.052083] .max_idle_balance_cost : 500000
[428113.052134] .ttwu_count : 12413852
[428113.052185] .ttwu_local : 3463562
[428113.052236]
[428113.052236] cfs_rq[4]:/
[428113.052329] .exec_clock : 19891085.291101
[428113.052381] .MIN_vruntime : 0.000001
[428113.052432] .min_vruntime : 23963851.976267
[428113.052484] .max_vruntime : 0.000001
[428113.052536] .spread : 0.000000
[428113.052587] .spread0 : -82460881.372519
[428113.052639] .nr_spread_over : 307
[428113.052689] .nr_running : 1
[428113.052739] .load : 1024
[428113.052790] .runnable_load_avg : 1023
[428113.052840] .blocked_load_avg : 47
[428113.052891] .tg_load_contrib : 1047
[428113.052942] .tg_runnable_contrib : 1015
[428113.052993] .tg_load_avg : 2738
[428113.053044] .tg->runnable_avg : 2685
[428113.053095] .tg->cfs_bandwidth.timer_active: 0
[428113.053146] .throttled : 0
[428113.053196] .throttle_count : 0
[428113.053247] .avg->runnable_avg_sum : 46889
[428113.053298] .avg->runnable_avg_period : 46889
[428113.053350]
[428113.053350] rt_rq[4]:/
[428113.053442] .rt_nr_running : 0
[428113.053492] .rt_throttled : 0
[428113.053542] .rt_time : 0.000000
[428113.053593] .rt_runtime : 950.000000
[428113.053645]
[428113.053645] dl_rq[4]:
[428113.053736] .dl_nr_running : 0
[428113.053787]
[428113.053787] runnable tasks:
[428113.053787] task PID tree-key switches prio
exec-runtime sum-exec sum-sleep
[428113.053787]
----------------------------------------------------------------------------------------------------------
[428113.054082] watchdog/4 26 -8.974112 107089 0
-8.974112 2059.037229 45.684752 /
[428113.054187] migration/4 27 0.000000 33383 0
0.000000 19839.147564 0.000834 /
[428113.054292] ksoftirqd/4 28 23963277.915795 14197 120
23963277.915795 161.575572 428107147.523261 /
[428113.054397] kworker/4:0 29 1190565.109457 20 120
1190565.109457 0.048792 84471152.489397 /
[428113.054502] kworker/4:0H 30 1300.883814 6 100
1300.883814 0.042505 33602.235307 /
[428113.054608] irq/28-mei_me 673 0.000000 14 49
0.000000 0.320285 0.000000 /
[428113.054714] md1_raid1 910 23938903.837682 1461 120
23938903.837682 66.950526 427490660.626113 /
[428113.054821] kworker/4:1H 1287 22856596.596367 40421 100
22856596.596367 571.074116 407462368.250643 /
[428113.054926] mysqld 4148 104218.340695 6 120
104218.340695 0.235452 160281236.866385 /autogroup-6
[428113.055034] getty 4371 3.299146 103 120
3.299146 4.682491 144.819918 /autogroup-20
[428113.055142] Rbtrfs-transacti 6269 23963851.976267 24693859 120
23963851.976267 181831355.421691 178264635.080484 /
[428113.055247] kworker/4:2 9937 23963841.012506 2387667 120
23963841.012506 1048755.187071 342552425.512330 /
[428113.055353] kworker/u16:2 23684 23924997.103420 4 120
23924997.103420 0.030497 2236660.637946 /
[428113.055458] top 23926 67.900517 110 120
67.900517 88.375149 90189.080325 /autogroup-1195
[428113.055566]
[428113.055619] cpu#5, 3411.800 MHz
[428113.055683] .nr_running : 0
[428113.055735] .load : 0
[428113.055794] .nr_switches : 29826080
[428113.055846] .nr_load_updates : 4205435
[428113.055898] .nr_uninterruptible : 5986
[428113.055959] .next_balance : 4337.771193
[428113.056011] .curr->pid : 0
[428113.056061] .clock : 428113055.637128
[428113.056113] .cpu_load[0] : 0
[428113.056164] .cpu_load[1] : 0
[428113.056215] .cpu_load[2] : 0
[428113.056265] .cpu_load[3] : 0
[428113.056316] .cpu_load[4] : 0
[428113.056367] .yld_count : 23948
[428113.056418] .sched_count : 29988603
[428113.056470] .sched_goidle : 12260277
[428113.056521] .avg_idle : 1000000
[428113.056573] .max_idle_balance_cost : 500000
[428113.056623] .ttwu_count : 14760716
[428113.056675] .ttwu_local : 3492006
[428113.056726]
[428113.056726] cfs_rq[5]:/
[428113.058071] .exec_clock : 22532030.048224
[428113.058123] .MIN_vruntime : 0.000001
[428113.058174] .min_vruntime : 25423030.650458
[428113.058226] .max_vruntime : 0.000001
[428113.058277] .spread : 0.000000
[428113.058328] .spread0 : -81001702.698328
[428113.058381] .nr_spread_over : 230
[428113.058431] .nr_running : 0
[428113.058482] .load : 0
[428113.058532] .runnable_load_avg : 0
[428113.058582] .blocked_load_avg : 0
[428113.058632] .tg_load_contrib : 0
[428113.058683] .tg_runnable_contrib : 0
[428113.058734] .tg_load_avg : 2838
[428113.058785] .tg->runnable_avg : 2724
[428113.058836] .tg->cfs_bandwidth.timer_active: 0
[428113.058886] .throttled : 0
[428113.058937] .throttle_count : 0
[428113.058987] .avg->runnable_avg_sum : 29
[428113.059038] .avg->runnable_avg_period : 47630
[428113.059089]
[428113.059089] rt_rq[5]:/
[428113.059181] .rt_nr_running : 0
[428113.059232] .rt_throttled : 0
[428113.059282] .rt_time : 0.000000
[428113.059334] .rt_runtime : 950.000000
[428113.059386]
[428113.059386] dl_rq[5]:
[428113.059477] .dl_nr_running : 0
[428113.059528]
[428113.059528] runnable tasks:
[428113.059528] task PID tree-key switches prio
exec-runtime sum-exec sum-sleep
[428113.059528]
----------------------------------------------------------------------------------------------------------
[428113.059824] rcu_sched 7 25423026.820997 14214418 120
25423026.820997 399659.278793 427702209.389636 /
[428113.059929] watchdog/5 31 -8.977217 107088 0
-8.977217 2499.019149 32.364766 /
[428113.060035] migration/5 32 0.000000 26537 0
0.000000 15739.945360 0.000845 /
[428113.060139] ksoftirqd/5 33 25422651.093839 16195 120
25422651.093839 166.009753 428070175.552405 /
[428113.060245] kworker/5:0 34 239.267711 17 120
239.267711 0.034507 632.432629 /
[428113.060350] kworker/5:0H 35 632.111736 6 100
632.111736 0.059549 2815.362585 /
[428113.060456] kworker/5:1 611 25423018.904992 2773160 120
25423018.904992 1288651.204421 426793243.196345 /
[428113.060561] scsi_tmf_2 860 534.239404 2 100
534.239404 0.008896 0.005460 /
[428113.060667] scsi_tmf_5 866 558.255161 2 100
558.255161 0.008179 0.004413 /
[428113.060772] md0_raid1 899 25396783.503701 32703 120
25396783.503701 998.952947 427489843.030132 /
[428113.060878] kworker/5:1H 1061 24847022.042452 45684 100
24847022.042452 510.479777 416592265.701502 /
[428113.060984] anvil 4282 1335.039244 546 120
1335.039244 25.298050 418304078.786475 /autogroup-17
[428113.061092]
[428113.061138] cpu#6, 3411.800 MHz
[428113.061186] .nr_running : 0
[428113.061236] .load : 0
[428113.061286] .nr_switches : 31375902
[428113.061338] .nr_load_updates : 4945717
[428113.061389] .nr_uninterruptible : 6930
[428113.061440] .next_balance : 4337.771028
[428113.061492] .curr->pid : 0
[428113.061542] .clock : 428112972.478131
[428113.061595] .cpu_load[0] : 0
[428113.061645] .cpu_load[1] : 0
[428113.061695] .cpu_load[2] : 0
[428113.061745] .cpu_load[3] : 0
[428113.061795] .cpu_load[4] : 0
[428113.061846] .yld_count : 20767
[428113.061896] .sched_count : 31528236
[428113.061947] .sched_goidle : 12333544
[428113.061999] .avg_idle : 1000000
[428113.062050] .max_idle_balance_cost : 500000
[428113.062100] .ttwu_count : 17745863
[428113.062152] .ttwu_local : 4311925
[428113.062203]
[428113.062203] cfs_rq[6]:/autogroup-6
[428113.062296] .exec_clock : 173835.323068
[428113.062348] .MIN_vruntime : 0.000001
[428113.062399] .min_vruntime : 163529.123269
[428113.062451] .max_vruntime : 0.000001
[428113.062502] .spread : 0.000000
[428113.062554] .spread0 : -106261204.225517
[428113.062606] .nr_spread_over : 80
[428113.062656] .nr_running : 0
[428113.062706] .load : 0
[428113.062756] .runnable_load_avg : 0
[428113.062807] .blocked_load_avg : 0
[428113.062857] .tg_load_contrib : 0
[428113.062907] .tg_runnable_contrib : 0
[428113.062957] .tg_load_avg : 46
[428113.063008] .tg->runnable_avg : 46
[428113.063058] .tg->cfs_bandwidth.timer_active: 0
[428113.063109] .throttled : 0
[428113.063160] .throttle_count : 0
[428113.063211] .se->exec_start : 428112972.478131
[428113.063263] .se->vruntime : 32014882.056897
[428113.063316] .se->sum_exec_runtime : 173853.720494
[428113.063368] .se->statistics.wait_start : 0.000000
[428113.063419] .se->statistics.sleep_start : 0.000000
[428113.063470] .se->statistics.block_start : 0.000000
[428113.063521] .se->statistics.sleep_max : 0.000000
[428113.063573] .se->statistics.block_max : 0.000000
[428113.063624] .se->statistics.exec_max : 10.006440
[428113.063675] .se->statistics.slice_max : 9.985799
[428113.063726] .se->statistics.wait_max : 14.987432
[428113.063778] .se->statistics.wait_sum : 5142.815457
[428113.063830] .se->statistics.wait_count : 888277
[428113.063881] .se->load.weight : 2
[428113.063932] .se->avg.runnable_avg_sum : 4
[428113.063982] .se->avg.runnable_avg_period : 48033
[428113.064033] .se->avg.load_avg_contrib : 0
[428113.064083] .se->avg.decay_count : 408280347
[428113.064134]
[428113.064134] cfs_rq[6]:/
[428113.064227] .exec_clock : 29301583.069476
[428113.064279] .MIN_vruntime : 0.000001
[428113.064330] .min_vruntime : 32014893.868429
[428113.064382] .max_vruntime : 0.000001
[428113.064433] .spread : 0.000000
[428113.064484] .spread0 : -74409839.480357
[428113.064537] .nr_spread_over : 413
[428113.064587] .nr_running : 0
[428113.064637] .load : 0
[428113.064688] .runnable_load_avg : 0
[428113.064738] .blocked_load_avg : 0
[428113.064789] .tg_load_contrib : 0
[428113.064839] .tg_runnable_contrib : 0
[428113.064890] .tg_load_avg : 2838
[428113.064941] .tg->runnable_avg : 2724
[428113.064992] .tg->cfs_bandwidth.timer_active: 0
[428113.065043] .throttled : 0
[428113.065093] .throttle_count : 0
[428113.065144] .avg->runnable_avg_sum : 4
[428113.065195] .avg->runnable_avg_period : 47831
[428113.065246]
[428113.065246] rt_rq[6]:/
[428113.065338] .rt_nr_running : 0
[428113.065389] .rt_throttled : 0
[428113.065439] .rt_time : 0.000000
[428113.065491] .rt_runtime : 950.000000
[428113.065542]
[428113.065542] dl_rq[6]:
[428113.065644] .dl_nr_running : 0
[428113.065702]
[428113.065702] runnable tasks:
[428113.065702] task PID tree-key switches prio
exec-runtime sum-exec sum-sleep
[428113.065702]
----------------------------------------------------------------------------------------------------------
[428113.065999] watchdog/6 36 -8.979370 107088 0
-8.979370 4171.996810 19.046238 /
[428113.066104] migration/6 37 0.000000 21452 0
0.000000 13182.758087 0.000826 /
[428113.066210] ksoftirqd/6 38 32009668.868629 19084 120
32009668.868629 335.465215 427985619.483782 /
[428113.066315] kworker/6:0 39 2652.785847 15 120
2652.785847 0.034665 619.334482 /
[428113.066420] kworker/6:0H 40 3222.777674 6 100
3222.777674 0.042578 34131.960421 /
[428113.066526] kworker/6:1 613 32014882.656425 3447588 120
32014882.656425 1715810.008839 426365265.308128 /
[428113.066631] bioset 898 2957.541580 2 100
2957.541580 0.008747 0.006389 /
[428113.066737] kworker/6:1H 1463 31224371.088945 50546 100
31224371.088945 612.901074 416298526.150915 /
[428113.066842] kjournald 2216 3410.060184 2 120
3410.060184 0.412389 0.043474 /
[428113.066948] bioset 2222 3422.126220 2 100
3422.126220 0.088585 0.040787 /
[428113.067053] mysqld 4102 163517.397318 1020343 120
163517.397318 10602.850133 428059785.215024 /autogroup-6
[428113.067160] mysqld 4122 163443.475621 644351 120
163443.475621 179019.020540 393898317.914127 /autogroup-6
[428113.068518] dovecot 4220 980.263394 2014 120
980.263394 114.537242 418305299.164771 /autogroup-17
[428113.068624] udevd 6137 246.142493 30 120
246.142493 6.670412 22056460.101598 /autogroup-4
[428113.068731] udevd 6138 242.586598 15 120
242.586598 3.149366 49194.013301 /autogroup-4
[428113.068838] btrfs-worker 6246 1051403.790298 2 100
1051403.790298 0.011158 0.004647 /
[428113.068943] btrfs-endio-met 6255 1051427.810805 2 100
1051427.810805 0.017787 0.002771 /
[428113.069049] btrfs-endio-rai 6257 1051439.820346 2 100
1051439.820346 0.010462 0.002911 /
[428113.069154] btrfs-endio-rep 6258 1051451.828334 2 100
1051451.828334 0.008652 0.003137 /
[428113.069259] btrfs-endio-wri 6260 1051463.836060 2 100
1051463.836060 0.008410 0.003085 /
[428113.069365] btrfs-readahead 6263 1051475.843882 2 100
1051475.843882 0.008543 0.003153 /
[428113.069470] btrfs-extent-re 6265 1051487.851432 2 100
1051487.851432 0.008154 0.003265 /
[428113.069578]
[428113.069632] cpu#7, 3411.800 MHz
[428113.069681] .nr_running : 0
[428113.069734] .load : 0
[428113.069786] .nr_switches : 28961216
[428113.069837] .nr_load_updates : 3537072
[428113.069891] .nr_uninterruptible : 5065
[428113.069943] .next_balance : 4337.771143
[428113.069996] .curr->pid : 0
[428113.070047] .clock : 428112555.897703
[428113.070101] .cpu_load[0] : 0
[428113.070153] .cpu_load[1] : 0
[428113.070211] .cpu_load[2] : 0
[428113.070261] .cpu_load[3] : 0
[428113.070312] .cpu_load[4] : 12
[428113.070363] .yld_count : 22779
[428113.070414] .sched_count : 29115700
[428113.070466] .sched_goidle : 12603487
[428113.070517] .avg_idle : 1000000
[428113.070569] .max_idle_balance_cost : 500000
[428113.070620] .ttwu_count : 18626268
[428113.070671] .ttwu_local : 2870358
[428113.070722]
[428113.070722] cfs_rq[7]:/
[428113.070814] .exec_clock : 14364516.198492
[428113.070867] .MIN_vruntime : 0.000001
[428113.070919] .min_vruntime : 18016573.070883
[428113.070971] .max_vruntime : 0.000001
[428113.071022] .spread : 0.000000
[428113.071073] .spread0 : -88408168.119657
[428113.071126] .nr_spread_over : 261
[428113.071176] .nr_running : 0
[428113.071226] .load : 0
[428113.071277] .runnable_load_avg : 0
[428113.071328] .blocked_load_avg : 0
[428113.071378] .tg_load_contrib : 0
[428113.071429] .tg_runnable_contrib : 0
[428113.071480] .tg_load_avg : 2910
[428113.071530] .tg->runnable_avg : 2849
[428113.071581] .tg->cfs_bandwidth.timer_active: 0
[428113.071632] .throttled : 0
[428113.071682] .throttle_count : 0
[428113.071732] .avg->runnable_avg_sum : 18
[428113.071783] .avg->runnable_avg_period : 47536
[428113.071834]
[428113.071834] rt_rq[7]:/
[428113.071926] .rt_nr_running : 0
[428113.071976] .rt_throttled : 0
[428113.072027] .rt_time : 0.000000
[428113.072078] .rt_runtime : 950.000000
[428113.072130]
[428113.072130] dl_rq[7]:
[428113.072222] .dl_nr_running : 0
[428113.072273]
[428113.072273] runnable tasks:
[428113.072273] task PID tree-key switches prio
exec-runtime sum-exec sum-sleep
[428113.072273]
----------------------------------------------------------------------------------------------------------
[428113.072570] watchdog/7 41 -8.982956 107089 0
-8.982956 1664.683687 5.725643 /
[428113.072674] migration/7 42 0.000000 21715 0
0.000000 13068.322347 0.000816 /
[428113.072780] ksoftirqd/7 43 18013052.310983 12328 120
18013052.310983 152.767844 428092090.014392 /
[428113.072885] kworker/7:0 44 479.385599 15 120
479.385599 0.031395 606.240559 /
[428113.072991] kworker/7:0H 45 1817.499047 6 100
1817.499047 0.047704 42727.098840 /
[428113.073096] netns 47 10.955152 2 100
10.955152 0.004076 0.001586 /
[428113.073201] perf 50 22.958573 2 100
22.958573 0.003794 0.001394 /
[428113.073306] kworker/7:1 615 18016561.430108 1960767 120
18016561.430108 793147.479614 427291244.310703 /
[428113.073413] bioset 919 1132.474026 2 100
1132.474026 0.034373 0.004008 /
[428113.073518] ext4-rsv-conver 936 1156.590460 2 100
1156.590460 0.061309 0.026946 /
[428113.073623] udevd 1124 405.566270 290 120
405.566270 53.586152 77350627.244245 /autogroup-4
[428113.073729] rsyslogd 3061 171793.939660 3747 120
171793.939660 155.674406 427956979.115875 /autogroup-6
[428113.073836] ipv6_addrconf 3209 1679.353321 2 100
1679.353321 0.006452 0.002581 /
[428113.073942] kworker/7:1H 4116 17737413.492285 52256 100
17737413.492285 541.503637 415562752.649104 /
[428113.074048] log 4283 586.242014 697 120
586.242014 34.396858 418304069.988435 /autogroup-17
[428113.074154] qmgr 4370 317.621444 2671 120
317.621444 299.720143 427913275.665071 /autogroup-19
[428113.074262] getty 4376 0.726364 103 120
0.726364 1.882302 146.996217 /autogroup-25
[428113.074369] btrfs-fixup 6253 1049025.643969 2 100
1049025.643969 0.011376 0.003865 /
[428113.074474] btrfs-endio 6254 1049037.659031 2 100
1049037.659031 0.015618 0.001516 /
[428113.074580] kworker/u17:1 23565 17782910.635845 2 100
17782910.635845 0.004742 0.030459 /
[428113.074685] pickup 23745 329.619461 38 120
329.619461 4.298483 2159811.305363 /autogroup-19
[428113.074792]
--
Tomasz Chmielewski
http://www.sslrack.com
prev parent reply other threads:[~2015-02-16 10:51 UTC|newest]
Thread overview: 3+ messages / expand[flat|nested] mbox.gz Atom feed top
2015-02-16 2:32 3.19 - "unable" to replace a failed drive - 100% CPU usage in kworker and btrfs-transacti Tomasz Chmielewski
2015-02-16 10:40 ` Liu Bo
2015-02-16 10:51 ` Tomasz Chmielewski [this message]
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=6830fb54424ede297a6349832bf46d00@admin.virtall.com \
--to=tch@virtall.com \
--cc=bo.li.liu@oracle.com \
--cc=linux-btrfs@vger.kernel.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.