From: Liu Bo <bo.li.liu@oracle.com>
To: Tomasz Chmielewski <tch@virtall.com>
Cc: linux-btrfs <linux-btrfs@vger.kernel.org>
Subject: Re: 3.19 - "unable" to replace a failed drive - 100% CPU usage in kworker and btrfs-transacti
Date: Mon, 16 Feb 2015 18:40:28 +0800 [thread overview]
Message-ID: <20150216104026.GC23683@localhost.localdomain> (raw)
In-Reply-To: <de541fe6037c9b5516e398e3a98597f8@admin.virtall.com>
On Mon, Feb 16, 2015 at 11:32:56AM +0900, Tomasz Chmielewski wrote:
> I had a failed drive in RAID-1, so it was replaced with a good one,
> followed by:
>
> btrfs device add /dev/sdb4 /home
> btrfs device delete missing /home
>
>
> 4 days later, it got to a state when there is no IO anymore
> (according to iostat), "btrfs device delete missing" did not
> complete:
>
> # uptime
> 03:29:03 up 4 days, 14:38, 1 user, load average: 2.36, 2.43, 2.54
>
> # btrfs fi show
> Label: none uuid: 84d087aa-3a32-46da-844f-a233237cf04f
> Total devices 3 FS bytes used 206.53GiB
> devid 3 size 1.71TiB used 214.03GiB path /dev/sda4
> devid 4 size 1.71TiB used 177.00GiB path /dev/sdb4
> *** Some devices missing
>
> Btrfs v3.18.2
>
> PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
> 6269 root 20 0 0 0 0 R 92.5 0.0 2769:33
> btrfs-transacti
> 22247 root 20 0 0 0 0 R 92.5 0.0 42:38.65
> kworker/u16:16
Can you cat /proc/22247/stack and /proc/6269/stack?
Thanks,
-liubo
>
>
> # dmesg
> [397948.321324] SysRq : Show Blocked State
> [397948.321386] task PC stack pid father
> [397948.321465] btrfs D ffff8802d67ef948 0 7611
> 6275 0x00000000
> [397948.321521] ffff8802d67ef948 ffff8804d2a9ba40 ffff88081917e040
> 00000000000113c0
> [397948.321621] 0000000000004000 ffff88081be30000 ffff88081917e040
> ffff8802d67ef8b8
> [397948.321720] ffffffff8105f3c4 ffff88083fa40000 ffff88083fa913c0
> ffff88083fa913c0
> [397948.321819] Call Trace:
> [397948.321872] [<ffffffff8105f3c4>] ? enqueue_task_fair+0x3e5/0x44f
> [397948.321927] [<ffffffff81054ab6>] ? resched_curr+0x45/0x55
> [397948.321979] [<ffffffff81055122>] ? check_preempt_curr+0x3e/0x6d
> [397948.322032] [<ffffffff81055163>] ? ttwu_do_wakeup+0x12/0x7f
> [397948.322085] [<ffffffff8105526e>] ?
> ttwu_do_activate.constprop.73+0x57/0x5c
> [397948.322141] [<ffffffff813be618>] schedule+0x65/0x67
> [397948.322193] [<ffffffff813c03b8>] schedule_timeout+0x26/0x18d
> [397948.322245] [<ffffffff81057d1e>] ? wake_up_process+0x30/0x34
> [397948.322299] [<ffffffff8104b1a3>] ? wake_up_worker+0x1f/0x21
> [397948.322352] [<ffffffff8104b3ea>] ? insert_work+0x87/0x94
> [397948.322414] [<ffffffffa02e183d>] ? free_block_list+0x1f/0x34
> [btrfs]
> [397948.322468] [<ffffffff813bef76>] wait_for_common+0x10d/0x13e
> [397948.322521] [<ffffffff81057cdf>] ? try_to_wake_up+0x250/0x250
> [397948.322574] [<ffffffff813befbf>] wait_for_completion+0x18/0x1a
> [397948.322631] [<ffffffffa028dedc>]
> btrfs_async_run_delayed_refs+0xc1/0xe4 [btrfs]
> [397948.322734] [<ffffffffa02a3189>]
> __btrfs_end_transaction+0x315/0x33b [btrfs]
> [397948.322836] [<ffffffffa02a31bd>]
> btrfs_end_transaction_throttle+0xe/0x10 [btrfs]
> [397948.322939] [<ffffffffa02e5cf1>]
> relocate_block_group+0x2ad/0x4de [btrfs]
> [397948.323039] [<ffffffffa02e607a>]
> btrfs_relocate_block_group+0x158/0x278 [btrfs]
> [397948.323152] [<ffffffffa02c37e1>]
> btrfs_relocate_chunk.isra.69+0x35/0xa5 [btrfs]
> [397948.323256] [<ffffffffa02c41da>]
> btrfs_shrink_device+0x235/0x408 [btrfs]
> [397948.323316] [<ffffffffa02c6ba9>] btrfs_rm_device+0x2a9/0x704
> [btrfs]
> [397948.323371] [<ffffffff810f5441>] ?
> __kmalloc_track_caller+0x40/0x178
> [397948.323431] [<ffffffffa02cf7a7>] btrfs_ioctl+0xa9c/0x22f5 [btrfs]
> [397948.323485] [<ffffffff8110f142>] ? putname+0x23/0x2c
> [397948.323537] [<ffffffff8110f6d3>] ? user_path_at_empty+0x60/0x90
> [397948.323593] [<ffffffff811783bf>] ? avc_has_perm+0x2e/0xf7
> [397948.323654] [<ffffffff8111171b>] do_vfs_ioctl+0x418/0x460
> [397948.323706] [<ffffffff81106745>] ? vfs_stat+0x16/0x18
> [397948.323758] [<ffffffff811117b1>] SyS_ioctl+0x4e/0x7d
> [397948.323809] [<ffffffff81031c5b>] ? do_page_fault+0xc/0x11
> [397948.323862] [<ffffffff813c1352>] system_call_fastpath+0x12/0x17
> [397948.323917] Sched Debug Version: v0.11, 3.19.0 #1
> [397948.323968] ktime :
> 398158300.945440
> [397948.324021] sched_clk :
> 397948323.916497
> [397948.324074] cpu_clk :
> 397948323.916513
> [397948.324127] jiffies : 4334753127
> [397948.324180] sched_clock_stable() : 1
> [397948.324231]
> [397948.324277] sysctl_sched
> [397948.324325] .sysctl_sched_latency : 24.000000
> [397948.324378] .sysctl_sched_min_granularity : 3.000000
> [397948.324430] .sysctl_sched_wakeup_granularity : 4.000000
> [397948.324483] .sysctl_sched_child_runs_first : 0
> [397948.324534] .sysctl_sched_features : 11899
> [397948.324586] .sysctl_sched_tunable_scaling : 1
> (logaritmic)
> [397948.324639]
> [397948.324686] cpu#0, 3411.800 MHz
> [397948.324734] .nr_running : 0
> [397948.324784] .load : 0
> [397948.324834] .nr_switches : 137328510
> [397948.324885] .nr_load_updates : 23262071
> [397948.324936] .nr_uninterruptible : -738004
> [397948.324987] .next_balance : 4334.753128
> [397948.325039] .curr->pid : 0
> [397948.325089] .clock : 397948323.134939
> [397948.325141] .cpu_load[0] : 0
> [397948.325191] .cpu_load[1] : 0
> [397948.325241] .cpu_load[2] : 0
> [397948.325292] .cpu_load[3] : 0
> [397948.325343] .cpu_load[4] : 0
> [397948.325393] .yld_count : 223168
> [397948.325444] .sched_count : 137673310
> [397948.325496] .sched_goidle : 50221823
> [397948.325547] .avg_idle : 878593
> [397948.325599] .max_idle_balance_cost : 500000
> [397948.325650] .ttwu_count : 86464371
> [397948.325701] .ttwu_local : 52888650
> [397948.325752]
> [397948.325752] cfs_rq[0]:/autogroup-1114
> [397948.325847] .exec_clock : 11.086901
> [397948.325899] .MIN_vruntime : 0.000001
> [397948.325950] .min_vruntime : 10.038325
> [397948.326001] .max_vruntime : 0.000001
> [397948.326052] .spread : 0.000000
> [397948.326103] .spread0 : -96825607.013236
> [397948.326155] .nr_spread_over : 0
> [397948.326205] .nr_running : 0
> [397948.326256] .load : 0
> [397948.326306] .runnable_load_avg : 0
> [397948.326357] .blocked_load_avg : 0
> [397948.326407] .tg_load_contrib : 0
> [397948.326458] .tg_runnable_contrib : 0
> [397948.326508] .tg_load_avg : 0
> [397948.326559] .tg->runnable_avg : 0
> [397948.326609] .tg->cfs_bandwidth.timer_active: 0
> [397948.326659] .throttled : 0
> [397948.326710] .throttle_count : 0
> [397948.326760] .se->exec_start : 397948321.211399
> [397948.326812] .se->vruntime : 96825606.516443
> [397948.326865] .se->sum_exec_runtime : 11.086901
> [397948.326916] .se->statistics.wait_start : 0.000000
> [397948.326967] .se->statistics.sleep_start : 0.000000
> [397948.327018] .se->statistics.block_start : 0.000000
> [397948.327069] .se->statistics.sleep_max : 0.000000
> [397948.327121] .se->statistics.block_max : 0.000000
> [397948.327172] .se->statistics.exec_max : 2.982069
> [397948.327223] .se->statistics.slice_max : 0.000000
> [397948.327275] .se->statistics.wait_max : 0.019006
> [397948.327326] .se->statistics.wait_sum : 0.019006
> [397948.327378] .se->statistics.wait_count : 136
> [397948.327429] .se->load.weight : 2
> [397948.327479] .se->avg.runnable_avg_sum : 42
> [397948.327530] .se->avg.runnable_avg_period : 48335
> [397948.327581] .se->avg.load_avg_contrib : 0
> [397948.327631] .se->avg.decay_count : 379513094
> [397948.327683]
> [397948.327683] cfs_rq[0]:/autogroup-6
> [397948.327777] .exec_clock : 1407434.382402
> [397948.327829] .MIN_vruntime : 0.000001
> [397948.327880] .min_vruntime : 1154990.729705
> [397948.327932] .max_vruntime : 0.000001
> [397948.327984] .spread : 0.000000
> [397948.328035] .spread0 : -95670626.321856
> [397948.328087] .nr_spread_over : 255
> [397948.329388] .nr_running : 0
> [397948.329438] .load : 0
> [397948.329489] .runnable_load_avg : 0
> [397948.329539] .blocked_load_avg : 2
> [397948.329590] .tg_load_contrib : 2
> [397948.329640] .tg_runnable_contrib : 4
> [397948.329690] .tg_load_avg : 12
> [397948.329740] .tg->runnable_avg : 4
> [397948.329791] .tg->cfs_bandwidth.timer_active: 0
> [397948.329841] .throttled : 0
> [397948.329892] .throttle_count : 0
> [397948.329943] .se->exec_start : 397948323.133861
> [397948.329995] .se->vruntime : 96825606.879532
> [397948.330047] .se->sum_exec_runtime : 1407466.511198
> [397948.330098] .se->statistics.wait_start : 0.000000
> [397948.330150] .se->statistics.sleep_start : 0.000000
> [397948.330201] .se->statistics.block_start : 0.000000
> [397948.330252] .se->statistics.sleep_max : 0.000000
> [397948.330303] .se->statistics.block_max : 0.000000
> [397948.330354] .se->statistics.exec_max : 10.021049
> [397948.330406] .se->statistics.slice_max : 16.519045
> [397948.330457] .se->statistics.wait_max : 16.098230
> [397948.330509] .se->statistics.wait_sum : 77143.758185
> [397948.330561] .se->statistics.wait_count : 12935361
> [397948.330613] .se->load.weight : 2
> [397948.330663] .se->avg.runnable_avg_sum : 206
> [397948.330714] .se->avg.runnable_avg_period : 47204
> [397948.330765] .se->avg.load_avg_contrib : 2
> [397948.330816] .se->avg.decay_count : 379513096
> [397948.330868]
> [397948.330868] cfs_rq[0]:/
> [397948.330959] .exec_clock : 98523331.737970
> [397948.331011] .MIN_vruntime : 0.000001
> [397948.331062] .min_vruntime : 96825617.051561
> [397948.331115] .max_vruntime : 0.000001
> [397948.331166] .spread : 0.000000
> [397948.331217] .spread0 : 0.000000
> [397948.331269] .nr_spread_over : 270
> [397948.331320] .nr_running : 0
> [397948.331370] .load : 0
> [397948.331421] .runnable_load_avg : 0
> [397948.331471] .blocked_load_avg : 2
> [397948.331522] .tg_load_contrib : 2
> [397948.331572] .tg_runnable_contrib : 6
> [397948.331622] .tg_load_avg : 1490
> [397948.331673] .tg->runnable_avg : 1068
> [397948.331723] .tg->cfs_bandwidth.timer_active: 0
> [397948.331774] .throttled : 0
> [397948.331824] .throttle_count : 0
> [397948.331875] .avg->runnable_avg_sum : 286
> [397948.331926] .avg->runnable_avg_period : 46637
> [397948.331979]
> [397948.331979] rt_rq[0]:/
> [397948.332071] .rt_nr_running : 0
> [397948.332121] .rt_throttled : 0
> [397948.332171] .rt_time : 0.000000
> [397948.332222] .rt_runtime : 950.000000
> [397948.332274]
> [397948.332274] dl_rq[0]:
> [397948.332366] .dl_nr_running : 0
> [397948.332417]
> [397948.332417] runnable tasks:
> [397948.332417] task PID tree-key switches
> prio exec-runtime sum-exec sum-sleep
> [397948.332417] ----------------------------------------------------------------------------------------------------------
> [397948.332712] ksoftirqd/0 3 96825605.109947 740015
> 120 96825605.109947 12247.009129 397859768.730344 /
> [397948.332817] kworker/0:0 4 96823284.697816 53
> 120 96823284.697816 17.309475 397934074.622814 /
> [397948.332922] kworker/0:0H 5 1015.287609 5
> 100 1015.287609 0.041666 1136.081722 /
> [397948.333053] rcu_sched 7 96825608.780644 13143448
> 120 96825608.780644 369650.173462 397567779.683357 /
> [397948.333157] rcu_bh 8 117.354314 2
> 120 117.354314 0.000874 0.000722 /
> [397948.333263] migration/0 9 0.000000 40752
> 0 0.000000 37728.554175 0.000853 /
> [397948.333367] watchdog/0 10 -11.786179 99549
> 0 -11.786179 16602.242686 93.369695 /
> [397948.333473] kswapd0 506 96346153.194271 10895
> 120 96346153.194271 5358.878543 396356848.501371 /
> [397948.333577] acpi_thermal_pm 618 478.127841 2
> 100 478.127841 0.008079 0.004241 /
> [397948.333682] ata_sff 832 943.140921 2
> 100 943.140921 0.005880 0.022278 /
> [397948.333786] scsi_eh_0 855 1012.563036 18
> 120 1012.563036 0.875745 352.505402 /
> [397948.333891] kworker/0:1H 879 96825605.051890 19712713
> 100 96825605.051890 933447.862690 396865888.829352 /
> [397948.333995] jbd2/md2-8 935 96824920.898850 673247
> 120 96824920.898850 25939.369714 397908678.411380 /
> [397948.334100] rsyslogd 3062 1154978.849958 554
> 120 1154978.849958 31.915075 397909811.026141 /autogroup-6
> [397948.334206] nginx 3235 4173.366889 645371
> 120 4173.366889 8854.802084 393651417.627526 /autogroup-10
> [397948.334312] mysqld 4080 1154807.414715 5643
> 120 1154807.414715 799.670146 395527445.704335 /autogroup-6
> [397948.334418] mysqld 4094 1154978.441261 796267
> 120 1154978.441261 7509.987667 397898098.956672 /autogroup-6
> [397948.334523] mysqld 4097 1154978.442889 1253821
> 120 1154978.442889 54810.874030 397848250.280201 /autogroup-6
> [397948.334628] mysqld 4098 1154978.441618 1282226
> 120 1154978.441618 57182.533918 397845845.250436 /autogroup-6
> [397948.334734] mysqld 4102 1154978.441697 959988
> 120 1154978.441697 10108.715585 397895341.475090 /autogroup-6
> [397948.334839] mysqld 4103 1154978.440507 1341686
> 120 1154978.440507 16033.050027 397888981.976267 /autogroup-6
> [397948.334945] mysqld 4119 1154990.729705 404501
> 120 1154990.729705 119263.212981 397785618.876052 /autogroup-6
> [397948.335051] mysqld 4150 1039299.577821 7023984
> 120 1039299.577821 1407433.681869 320824575.671761 /autogroup-6
> [397948.335157] mysqld 5811 1154806.870976 1317671
> 120 1154806.870976 299564.211303 346391209.749613 /autogroup-6
> [397948.335264] mysqld 15436 1154807.136306 292526
> 120 1154807.136306 172005.952402 174398934.514010 /autogroup-6
> [397948.335371] getty 4374 0.817181 100
> 120 0.817181 2.613716 146.560614 /autogroup-23
> [397948.335478] btrfs-transacti 6269 96825617.051561 22374147
> 120 96825617.051561 166025164.637843 164906022.709080 /
> [397948.335583] bash 6275 132.300150 276
> 120 132.300150 79.740920 21744924.508065
> /autogroup-184
> [397948.335690] kworker/0:1 21018 96823285.353043 1816351
> 120 96823285.353043 793610.285186 42038707.779192 /
> [397948.335795] ssl-params 22730 472.603845 4
> 120 472.603845 0.877328 0.117947 /autogroup-17
> [397948.335902] kworker/u16:0 22761 96825069.356076 67516
> 120 96825069.356076 540082.349638 226131.772902 /
> [397948.336006] sshd 22771 10.038325 150
> 120 10.038325 13.472329 50175.143940
> /autogroup-1114
> [397948.336112] kworker/0:2 22822 96825605.578025 507
> 120 96825605.578025 47.279505 14055.084212 /
> [397948.336218]
> [397948.336280] cpu#1, 3411.800 MHz
> [397948.336329] .nr_running : 1
> [397948.336379] .load : 1024
> [397948.336430] .nr_switches : 84332707
> [397948.336481] .nr_load_updates : 21967275
> [397948.336533] .nr_uninterruptible : 483742
> [397948.336584] .next_balance : 4334.753129
> [397948.336635] .curr->pid : 22247
> [397948.336686] .clock : 397948333.217376
> [397948.336738] .cpu_load[0] : 1023
> [397948.336788] .cpu_load[1] : 1023
> [397948.336839] .cpu_load[2] : 1023
> [397948.336890] .cpu_load[3] : 1023
> [397948.336940] .cpu_load[4] : 1007
> [397948.336991] .yld_count : 135412
> [397948.337043] .sched_count : 84726950
> [397948.337094] .sched_goidle : 29157952
> [397948.337145] .avg_idle : 1000000
> [397948.337196] .max_idle_balance_cost : 500000
> [397948.337247] .ttwu_count : 36438464
> [397948.337299] .ttwu_local : 19485954
> [397948.337350]
> [397948.337350] cfs_rq[1]:/
> [397948.337441] .exec_clock : 123202243.887995
> [397948.337494] .MIN_vruntime : 0.000001
> [397948.337545] .min_vruntime : 119610642.735572
> [397948.337597] .max_vruntime : 0.000001
> [397948.337649] .spread : 0.000000
> [397948.337700] .spread0 : 22785025.684011
> [397948.337752] .nr_spread_over : 301
> [397948.337802] .nr_running : 1
> [397948.337852] .load : 1024
> [397948.337903] .runnable_load_avg : 1023
> [397948.337954] .blocked_load_avg : 26
> [397948.338004] .tg_load_contrib : 1042
> [397948.338055] .tg_runnable_contrib : 1010
> [397948.338106] .tg_load_avg : 1317
> [397948.338156] .tg->runnable_avg : 1293
> [397948.338206] .tg->cfs_bandwidth.timer_active: 0
> [397948.338256] .throttled : 0
> [397948.338307] .throttle_count : 0
> [397948.338357] .avg->runnable_avg_sum : 46120
> [397948.338408] .avg->runnable_avg_period : 46120
> [397948.338459]
> [397948.338459] rt_rq[1]:/
> [397948.338550] .rt_nr_running : 0
> [397948.338600] .rt_throttled : 0
> [397948.339900] .rt_time : 0.000000
> [397948.339951] .rt_runtime : 950.000000
> [397948.340003]
> [397948.340003] dl_rq[1]:
> [397948.340094] .dl_nr_running : 0
> [397948.340144]
> [397948.340144] runnable tasks:
> [397948.340144] task PID tree-key switches
> prio exec-runtime sum-exec sum-sleep
> [397948.340144] ----------------------------------------------------------------------------------------------------------
> [397948.340438] watchdog/1 11 -2.973570 99543
> 0 -2.973570 21159.974792 85.639910 /
> [397948.340543] migration/1 12 0.000000 37023
> 0 0.000000 30622.595831 0.000758 /
> [397948.340648] ksoftirqd/1 13 119609979.133122 91495
> 120 119609979.133122 1646.457418 397900354.563118 /
> [397948.340752] kworker/1:0H 15 582.372111 6
> 100 582.372111 0.027131 2733.616385 /
> [397948.340858] khungtaskd 204 119577195.763580 3319
> 120 119577195.763580 182.485563 397829939.496656 /
> [397948.340962] kpsmoused 686 25.893051 2
> 100 25.893051 0.010838 0.003636 /
> [397948.341068] kworker/1:2 688 102920251.057920 55
> 120 102920251.057920 0.504998 355091633.648033 /
> [397948.341172] deferwq 731 158.015576 2
> 100 158.015576 0.009076 0.003883 /
> [397948.341277] scsi_tmf_0 856 442.536328 2
> 100 442.536328 0.008945 0.003987 /
> [397948.341382] scsi_eh_1 857 539.461151 16
> 120 539.461151 0.868588 350.508685 /
> [397948.341486] scsi_tmf_1 858 466.554669 2
> 100 466.554669 0.008786 0.003948 /
> [397948.341590] scsi_eh_2 859 514.602593 3
> 120 514.602593 0.016414 1.372299 /
> [397948.341695] scsi_eh_3 861 514.603810 3
> 120 514.603810 0.016837 1.222996 /
> [397948.341799] scsi_tmf_3 862 502.578935 2
> 100 502.578935 0.008052 0.003896 /
> [397948.341904] scsi_tmf_4 864 514.587500 2
> 100 514.587500 0.008905 0.004404 /
> [397948.342009] kworker/1:1H 968 118250346.559508 456123
> 100 118250346.559508 6526.369618 394080278.360060 /
> [397948.342114] cron 3301 777.910417 7708
> 120 777.910417 405.566912 397904071.914015 /autogroup-12
> [397948.342219] sshd 3426 0.515010 28
> 120 0.515010 3.325645 397861007.941426 /autogroup-15
> [397948.342326] mysqld_safe 3464 1418.978306 435
> 120 1418.978306 26.375085 974.715839 /autogroup-6
> [397948.342432] mysqld 4118 690369.357591 400760
> 120 690369.357591 50887.461054 397853256.747730 /autogroup-6
> [397948.342539] getty 4372 0.769188 104
> 120 0.769188 2.833955 146.554963 /autogroup-21
> [397948.342645] btrfs-cleaner 6268 119610036.344925 21815
> 120 119610036.344925 5411.552915 342317734.332486 /
> [397948.342750] btrfs 7611 311971.962451 2055291
> 120 311971.962451 1498599.957865 319025222.198539
> /autogroup-184
> [397948.342856] kworker/1:0 21019 119610630.984203 2134678
> 120 119610630.984203 1030853.178543 41814652.278685 /
> [397948.342969] R kworker/u16:16 22247 119610652.481537 293438
> 120 119610652.481537 2301421.627687 2218308.587596 /
> [397948.343099]
> [397948.343161] cpu#2, 3411.800 MHz
> [397948.343210] .nr_running : 3
> [397948.343259] .load : 2048
> [397948.343310] .nr_switches : 76867052
> [397948.343360] .nr_load_updates : 19847327
> [397948.343411] .nr_uninterruptible : 139945
> [397948.343462] .next_balance : 4334.753139
> [397948.343513] .curr->pid : 22773
> [397948.343563] .clock : 397948336.246820
> [397948.343615] .cpu_load[0] : 275
> [397948.343666] .cpu_load[1] : 212
> [397948.343716] .cpu_load[2] : 127
> [397948.343767] .cpu_load[3] : 70
> [397948.343818] .cpu_load[4] : 50
> [397948.343868] .yld_count : 92399
> [397948.343919] .sched_count : 77151242
> [397948.343970] .sched_goidle : 27487304
> [397948.344021] .avg_idle : 1000000
> [397948.344072] .max_idle_balance_cost : 500000
> [397948.344123] .ttwu_count : 35275362
> [397948.344175] .ttwu_local : 17426867
> [397948.344227]
> [397948.344227] cfs_rq[2]:/autogroup-1115
> [397948.344321] .exec_clock : 77.755926
> [397948.344372] .MIN_vruntime : 0.000001
> [397948.344423] .min_vruntime : 169.689274
> [397948.344475] .max_vruntime : 0.000001
> [397948.344526] .spread : 0.000000
> [397948.344578] .spread0 : -96825447.362287
> [397948.344629] .nr_spread_over : 7
> [397948.344680] .nr_running : 1
> [397948.344730] .load : 1024
> [397948.344781] .runnable_load_avg : 271
> [397948.344831] .blocked_load_avg : 0
> [397948.344882] .tg_load_contrib : 271
> [397948.344932] .tg_runnable_contrib : 271
> [397948.344983] .tg_load_avg : 271
> [397948.345033] .tg->runnable_avg : 271
> [397948.345083] .tg->cfs_bandwidth.timer_active: 0
> [397948.345133] .throttled : 0
> [397948.345183] .throttle_count : 0
> [397948.345234] .se->exec_start : 397948336.246820
> [397948.345286] .se->vruntime : 100334135.344609
> [397948.345338] .se->sum_exec_runtime : 77.755926
> [397948.345389] .se->statistics.wait_start : 0.000000
> [397948.345440] .se->statistics.sleep_start : 0.000000
> [397948.345491] .se->statistics.block_start : 0.000000
> [397948.345542] .se->statistics.sleep_max : 0.000000
> [397948.345593] .se->statistics.block_max : 0.000000
> [397948.345644] .se->statistics.exec_max : 13.202718
> [397948.345694] .se->statistics.slice_max : 0.000000
> [397948.345745] .se->statistics.wait_max : 0.020544
> [397948.345796] .se->statistics.wait_sum : 0.209415
> [397948.345847] .se->statistics.wait_count : 128
> [397948.345898] .se->load.weight : 1024
> [397948.345949] .se->avg.runnable_avg_sum : 12487
> [397948.345999] .se->avg.runnable_avg_period : 47064
> [397948.346051] .se->avg.load_avg_contrib : 275
> [397948.346101] .se->avg.decay_count : 0
> [397948.346152]
> [397948.346152] cfs_rq[2]:/
> [397948.346244] .exec_clock : 102436933.654759
> [397948.346296] .MIN_vruntime : 100334120.416350
> [397948.346348] .min_vruntime : 100334132.312627
> [397948.346401] .max_vruntime : 100334120.416350
> [397948.346453] .spread : 0.000000
> [397948.346504] .spread0 : 3508515.261066
> [397948.346556] .nr_spread_over : 450
> [397948.346606] .nr_running : 2
> [397948.346657] .load : 2048
> [397948.346708] .runnable_load_avg : 275
> [397948.346758] .blocked_load_avg : 0
> [397948.346809] .tg_load_contrib : 275
> [397948.346859] .tg_runnable_contrib : 271
> [397948.346910] .tg_load_avg : 1323
> [397948.346961] .tg->runnable_avg : 1293
> [397948.347011] .tg->cfs_bandwidth.timer_active: 0
> [397948.347061] .throttled : 0
> [397948.347111] .throttle_count : 0
> [397948.347161] .avg->runnable_avg_sum : 12763
> [397948.347212] .avg->runnable_avg_period : 48021
> [397948.347263]
> [397948.347263] rt_rq[2]:/
> [397948.347354] .rt_nr_running : 0
> [397948.347404] .rt_throttled : 0
> [397948.347455] .rt_time : 0.000000
> [397948.347506] .rt_runtime : 950.000000
> [397948.347557]
> [397948.347557] dl_rq[2]:
> [397948.347649] .dl_nr_running : 0
> [397948.347699]
> [397948.347699] runnable tasks:
> [397948.347699] task PID tree-key switches
> prio exec-runtime sum-exec sum-sleep
> [397948.347699] ----------------------------------------------------------------------------------------------------------
> [397948.347992] init 1 509.594135 79809
> 120 509.594135 2619.143889 397945091.854170 /autogroup-2
> [397948.348098] watchdog/2 16 -5.976382 99543
> 0 -5.976382 17476.538367 72.326265 /
> [397948.348202] migration/2 17 0.000000 32592
> 0 0.000000 29450.114444 0.001139 /
> [397948.348306] ksoftirqd/2 18 100333573.328598 69839
> 120 100333573.328598 1929.169352 397913601.175468 /
> [397948.348410] kworker/2:0 19 49262954.665776 38
> 120 49262954.665776 0.122608 231013304.220524 /
> [397948.348514] kworker/2:0H 20 3650.646229 6
> 100 3650.646229 0.037533 34187.208217 /
> [397948.348619] scsi_eh_4 863 3072.735732 3
> 120 3072.735732 0.015446 1.048287 /
> [397948.348724] scsi_eh_5 865 3072.733721 3
> 120 3072.733721 0.012919 0.910030 /
> [397948.348828] md1_raid1 910 100169530.338537 1429
> 120 100169530.338537 66.597411 397503924.718962 /
> [397948.348932] kworker/2:1H 1466 100323518.282583 394863
> 100 100323518.282583 5278.861299 397866015.984528 /
> [397948.349036] rsyslogd 3033 428725.147921 28
> 120 428725.147921 2.679249 322287744.247447 /autogroup-6
> [397948.350392] atd 3189 0.622217 112
> 120 0.622217 2.344474 395791202.210800 /autogroup-9
> [397948.350498] acpid 3240 0.478710 41
> 120 0.478710 1.730662 321163512.935538 /autogroup-11
> [397948.350605] dbus-daemon 3404 0.942824 6
> 120 0.942824 1.484584 55885009.390813 /autogroup-14
> [397948.350710] mysqld 4095 459653.767988 848117
> 120 459653.767988 9524.934466 397895732.816642 /autogroup-6
> [397948.350817] mysqld 15425 459502.220847 261927
> 120 459502.220847 145837.180966 174449080.595390 /autogroup-6
> [397948.350923] qmgr 4370 357.389137 2476
> 120 357.389137 280.199510 397622564.351466 /autogroup-19
> [397948.351029] getty 4373 0.640858 102
> 120 0.640858 2.425952 146.827128 /autogroup-22
> [397948.351135] btrfs-worker-hi 6248 1154879.053331 2
> 100 1154879.053331 0.017567 0.003220 /
> [397948.351240] btrfs-delalloc 6249 1154891.063212 2
> 100 1154891.063212 0.010622 0.003413 /
> [397948.351345] btrfs-flush_del 6250 1154903.071666 2
> 100 1154903.071666 0.009022 0.003320 /
> [397948.351449] btrfs-cache 6251 1154915.081079 2
> 100 1154915.081079 0.015404 0.003203 /
> [397948.351554] btrfs-submit 6252 1154915.162054 2
> 100 1154915.162054 0.096237 0.023172 /
> [397948.351659] btrfs-endio-met 6256 1154927.254127 2
> 100 1154927.254127 0.093435 0.002869 /
> [397948.351763] btrfs-rmw 6259 1154939.263666 2
> 100 1154939.263666 0.010258 0.003176 /
> [397948.351868] btrfs-freespace 6261 1154951.272823 2
> 100 1154951.272823 0.009779 0.003215 /
> [397948.351972] btrfs-delayed-m 6262 1154963.281556 2
> 100 1154963.281556 0.009338 0.003128 /
> [397948.352076] btrfs-qgroup-re 6264 1154975.290187 2
> 100 1154975.290187 0.009265 0.003113 /
> [397948.352180] kworker/2:2 16094 100334120.416350 6973086
> 120 100334120.416350 3053548.894806 163847153.385628 /
> [397948.352285] pickup 22744 369.431995 35
> 120 369.431995 4.254482 1980726.020374 /autogroup-19
> [397948.352391] R bash 22773 169.689274 106
> 120 169.689274 49.563444 46410.281275
> /autogroup-1115
> [397948.352497]
> [397948.352543] cpu#3, 3411.800 MHz
> [397948.352591] .nr_running : 0
> [397948.352641] .load : 0
> [397948.352692] .nr_switches : 78216804
> [397948.352743] .nr_load_updates : 20561275
> [397948.352794] .nr_uninterruptible : 84689
> [397948.352845] .next_balance : 4334.753130
> [397948.352896] .curr->pid : 0
> [397948.352955] .clock : 397948343.119396
> [397948.353033] .cpu_load[0] : 0
> [397948.353100] .cpu_load[1] : 0
> [397948.353150] .cpu_load[2] : 0
> [397948.353200] .cpu_load[3] : 0
> [397948.353251] .cpu_load[4] : 0
> [397948.353300] .yld_count : 73921
> [397948.353351] .sched_count : 78424464
> [397948.353402] .sched_goidle : 27817319
> [397948.353453] .avg_idle : 1000000
> [397948.353504] .max_idle_balance_cost : 500000
> [397948.353555] .ttwu_count : 36987465
> [397948.353605] .ttwu_local : 18034474
> [397948.353656]
> [397948.353656] cfs_rq[3]:/autogroup-6
> [397948.353751] .exec_clock : 484988.006368
> [397948.353802] .MIN_vruntime : 0.000001
> [397948.353854] .min_vruntime : 404786.905985
> [397948.353905] .max_vruntime : 0.000001
> [397948.353956] .spread : 0.000000
> [397948.354007] .spread0 : -96420830.145576
> [397948.354058] .nr_spread_over : 245
> [397948.354109] .nr_running : 0
> [397948.354159] .load : 0
> [397948.354209] .runnable_load_avg : 0
> [397948.354259] .blocked_load_avg : 4
> [397948.354310] .tg_load_contrib : 4
> [397948.354360] .tg_runnable_contrib : 5
> [397948.354409] .tg_load_avg : 4
> [397948.354460] .tg->runnable_avg : 7
> [397948.354510] .tg->cfs_bandwidth.timer_active: 0
> [397948.354560] .throttled : 0
> [397948.354610] .throttle_count : 0
> [397948.354661] .se->exec_start : 397948323.578248
> [397948.354713] .se->vruntime : 102939999.703983
> [397948.354766] .se->sum_exec_runtime : 485015.851159
> [397948.354817] .se->statistics.wait_start : 0.000000
> [397948.354868] .se->statistics.sleep_start : 0.000000
> [397948.354919] .se->statistics.block_start : 0.000000
> [397948.354970] .se->statistics.sleep_max : 0.000000
> [397948.355021] .se->statistics.block_max : 0.000000
> [397948.355072] .se->statistics.exec_max : 9.999422
> [397948.355123] .se->statistics.slice_max : 9.991785
> [397948.355174] .se->statistics.wait_max : 19.987876
> [397948.355225] .se->statistics.wait_sum : 7370.667707
> [397948.355277] .se->statistics.wait_count : 3844653
> [397948.355327] .se->load.weight : 2
> [397948.355378] .se->avg.runnable_avg_sum : 266
> [397948.355428] .se->avg.runnable_avg_period : 47445
> [397948.355479] .se->avg.load_avg_contrib : 5
> [397948.355529] .se->avg.decay_count : 379513096
> [397948.355581]
> [397948.355581] cfs_rq[3]:/
> [397948.355673] .exec_clock : 105213301.964322
> [397948.355725] .MIN_vruntime : 0.000001
> [397948.355776] .min_vruntime : 102940010.804921
> [397948.355828] .max_vruntime : 0.000001
> [397948.355879] .spread : 0.000000
> [397948.355930] .spread0 : 6114393.753360
> [397948.355982] .nr_spread_over : 326
> [397948.356033] .nr_running : 0
> [397948.356083] .load : 0
> [397948.356133] .runnable_load_avg : 0
> [397948.356183] .blocked_load_avg : 0
> [397948.356234] .tg_load_contrib : 0
> [397948.356284] .tg_runnable_contrib : 6
> [397948.356335] .tg_load_avg : 1319
> [397948.356385] .tg->runnable_avg : 1291
> [397948.356436] .tg->cfs_bandwidth.timer_active: 0
> [397948.356486] .throttled : 0
> [397948.356537] .throttle_count : 0
> [397948.356587] .avg->runnable_avg_sum : 307
> [397948.356637] .avg->runnable_avg_period : 46706
> [397948.356688]
> [397948.356688] rt_rq[3]:/
> [397948.356781] .rt_nr_running : 0
> [397948.356831] .rt_throttled : 0
> [397948.356882] .rt_time : 0.000000
> [397948.356933] .rt_runtime : 950.000000
> [397948.356984]
> [397948.356984] dl_rq[3]:
> [397948.357076] .dl_nr_running : 0
> [397948.357127]
> [397948.357127] runnable tasks:
> [397948.357127] task PID tree-key switches
> prio exec-runtime sum-exec sum-sleep
> [397948.357127] ----------------------------------------------------------------------------------------------------------
> [397948.357422] kthreadd 2 102939126.157704 1315
> 120 102939126.157704 38.637725 397940297.121677 /
> [397948.357527] watchdog/3 21 -5.984614 99544
> 0 -5.984614 18147.844668 58.998123 /
> [397948.357632] migration/3 22 0.000000 32291
> 0 0.000000 32254.090833 0.000847 /
> [397948.357737] ksoftirqd/3 23 102939780.572027 62304
> 120 102939780.572027 1739.164572 397906895.811056 /
> [397948.357841] kworker/3:0 24 33082955.717338 56
> 120 33082955.717338 0.596786 176596062.108639 /
> [397948.357946] kworker/3:0H 25 1600.396479 6
> 100 1600.396479 0.027776 33504.961049 /
> [397948.358051] khelper 46 10.955864 2
> 100 10.955864 0.005052 0.001801 /
> [397948.358156] writeback 205 311.132001 2
> 100 311.132001 0.003788 0.001292 /
> [397948.358261] crypto 208 323.135116 2
> 100 323.135116 0.003395 0.001548 /
> [397948.358365] bioset 209 335.138067 2
> 100 335.138067 0.003268 0.001389 /
> [397948.358470] kblockd 210 347.141115 2
> 100 347.141115 0.003369 0.001350 /
> [397948.358574] md 361 359.144676 2
> 100 359.144676 0.003908 0.001863 /
> [397948.358679] fsnotify_mark 576 1285553.304174 56
> 120 1285553.304174 1.356477 77382968.604008 /
> [397948.358784] bioset 909 1410.076226 2
> 100 1410.076226 0.008348 0.004053 /
> [397948.358888] md2_raid1 920 102939126.258310 5305156
> 120 102939126.258310 235266.004521 397697550.934379 /
> [397948.358994] kworker/3:1H 1233 101531219.068494 383437
> 100 101531219.068494 4828.918177 393998408.709581 /
> [397948.359098] rs:main Q:Reg 3060 404775.410651 3733
> 120 404775.410651 584.098973 397909253.918002 /autogroup-6
> [397948.359205] vnstatd 3041 1938.625022 80791
> 120 1938.625022 7728.375777 397899296.874075 /autogroup-7
> [397948.359311] mdadm 3144 21.070081 419
> 120 21.070081 64.059078 397466901.661042 /autogroup-8
> [397948.359417] nginx 3232 4286.796414 20
> 120 4286.796414 1.473683 322285293.302498 /autogroup-10
> [397948.359524] nginx 3233 5123.741553 498857
> 120 5123.741553 6408.084381 397902171.919015 /autogroup-10
> [397948.359631] nginx 3234 5123.781505 795481
> 120 5123.781505 10575.489835 397897949.295633 /autogroup-10
> [397948.360989] nginx 3236 5123.792442 453567
> 120 5123.792442 6288.632023 397902307.034538 /autogroup-10
> [397948.361094] ntpd 3378 3177.303496 399798
> 120 3177.303496 11384.417725 397896567.168923 /autogroup-13
> [397948.361201] mysqld 4096 404775.038669 1280416
> 120 404775.038669 56190.329409 397846935.062842 /autogroup-6
> [397948.361306] mysqld 4099 404775.037536 1237509
> 120 404775.037536 53709.818619 397849497.344214 /autogroup-6
> [397948.361413] mysqld 4100 404775.128644 1410134
> 120 404775.128644 17179.021072 397887821.391458 /autogroup-6
> [397948.361520] mysqld 4101 404775.136626 1023118
> 120 404775.136626 11032.276249 397894304.441429 /autogroup-6
> [397948.361625] mysqld 4120 404774.916067 79647
> 120 404774.916067 1272.295740 397899969.656177 /autogroup-6
> [397948.361732] mysqld 4121 404774.944173 79565
> 120 404774.944173 1158.311382 397903499.069766 /autogroup-6
> [397948.361838] mysqld 4123 260297.226173 260619
> 120 260297.226173 48020.432086 156765824.444851 /autogroup-6
> [397948.361944] mysqld 4154 404597.326324 2178558
> 120 404597.326324 373081.589987 395136259.717365 /autogroup-6
> [397948.362049] mysqld 4393 404597.164623 1651064
> 120 404597.164623 338373.325108 395122973.929693 /autogroup-6
> [397948.362156] mysqld 4396 404771.119995 1079825
> 120 404771.119995 305191.529334 397422077.680807 /autogroup-6
> [397948.362261] mysqld 15426 404596.822357 784917
> 120 404596.822357 137130.712105 174453678.251801 /autogroup-6
> [397948.362367] mysqld 15435 404596.986208 228061
> 120 404596.986208 118651.907457 174461622.198681 /autogroup-6
> [397948.362473] dovecot 4220 409.745461 1991
> 120 409.745461 113.183818 397150716.515321 /autogroup-17
> [397948.362579] master 4345 415.904754 20039
> 120 415.904754 991.626364 397859137.298652 /autogroup-19
> [397948.362684] getty 4375 0.839336 103
> 120 0.839336 1.796678 147.205983 /autogroup-24
> [397948.362791] tmux 6274 11456.276708 987382
> 120 11456.276708 43704.262889 342265075.730957
> /autogroup-183
> [397948.362897] kworker/3:1 13622 102939999.914193 9663630
> 120 102939999.914193 4235651.348825 217069990.063754 /
> [397948.363034] kworker/u16:2 22232 102939126.098258 163122
> 120 102939126.098258 1130115.311569 3704906.322727 /
> [397948.363138] kworker/u17:0 22250 100814026.249739 4
> 100 100814026.249739 0.109630 5.440267 /
> [397948.363243]
> [397948.363304] cpu#4, 3411.800 MHz
> [397948.363352] .nr_running : 0
> [397948.363403] .load : 0
> [397948.363452] .nr_switches : 31446353
> [397948.363503] .nr_load_updates : 3466700
> [397948.363554] .nr_uninterruptible : 12378
> [397948.363605] .next_balance : 4334.753132
> [397948.363656] .curr->pid : 0
> [397948.363706] .clock : 397948363.266319
> [397948.363758] .cpu_load[0] : 0
> [397948.363809] .cpu_load[1] : 0
> [397948.363858] .cpu_load[2] : 0
> [397948.363909] .cpu_load[3] : 0
> [397948.363958] .cpu_load[4] : 0
> [397948.364009] .yld_count : 30496
> [397948.364060] .sched_count : 31745499
> [397948.364111] .sched_goidle : 13288472
> [397948.364162] .avg_idle : 1000000
> [397948.364213] .max_idle_balance_cost : 500000
> [397948.364264] .ttwu_count : 12079122
> [397948.364315] .ttwu_local : 3235209
> [397948.364366]
> [397948.364366] cfs_rq[4]:/
> [397948.364458] .exec_clock : 18189528.762014
> [397948.364510] .MIN_vruntime : 0.000001
> [397948.364561] .min_vruntime : 22358073.568236
> [397948.364613] .max_vruntime : 0.000001
> [397948.364664] .spread : 0.000000
> [397948.364715] .spread0 : -74467543.483325
> [397948.364768] .nr_spread_over : 290
> [397948.364818] .nr_running : 0
> [397948.364868] .load : 0
> [397948.364919] .runnable_load_avg : 0
> [397948.364968] .blocked_load_avg : 0
> [397948.365018] .tg_load_contrib : 0
> [397948.365069] .tg_runnable_contrib : 0
> [397948.365119] .tg_load_avg : 1315
> [397948.365170] .tg->runnable_avg : 1290
> [397948.365220] .tg->cfs_bandwidth.timer_active: 0
> [397948.365271] .throttled : 0
> [397948.365321] .throttle_count : 0
> [397948.365371] .avg->runnable_avg_sum : 0
> [397948.365421] .avg->runnable_avg_period : 48326
> [397948.365472]
> [397948.365472] rt_rq[4]:/
> [397948.365564] .rt_nr_running : 0
> [397948.365614] .rt_throttled : 0
> [397948.365664] .rt_time : 0.000000
> [397948.365715] .rt_runtime : 950.000000
> [397948.365766]
> [397948.365766] dl_rq[4]:
> [397948.365858] .dl_nr_running : 0
> [397948.365909]
> [397948.365909] runnable tasks:
> [397948.365909] task PID tree-key switches
> prio exec-runtime sum-exec sum-sleep
> [397948.365909] ----------------------------------------------------------------------------------------------------------
> [397948.366203] watchdog/4 26 -8.974112 99544
> 0 -8.974112 1925.403645 45.684752 /
> [397948.366307] migration/4 27 0.000000 32687
> 0 0.000000 19510.524185 0.000834 /
> [397948.366412] ksoftirqd/4 28 22357044.685798 13474
> 120 22357044.685798 152.288196 397896961.880208 /
> [397948.366516] kworker/4:0 29 1190565.109457 20
> 120 1190565.109457 0.048792 84471152.489397 /
> [397948.366621] kworker/4:0H 30 1300.883814 6
> 100 1300.883814 0.042505 33602.235307 /
> [397948.366726] irq/28-mei_me 673 0.000000 14
> 49 0.000000 0.320285 0.000000 /
> [397948.366831] kworker/4:1H 1287 22130377.005493 40420
> 100 22130377.005493 571.072101 394074352.751479 /
> [397948.366937] mysqld 4148 104218.340695 6
> 120 104218.340695 0.235452 160281236.866385 /autogroup-6
> [397948.367043] anvil 4282 508.617446 540
> 120 508.617446 25.108427 395113557.223988 /autogroup-17
> [397948.367148] getty 4371 3.299146 103
> 120 3.299146 4.682491 144.819918 /autogroup-20
> [397948.367255] kworker/4:2 9937 22358061.586618 2184600
> 120 22358061.586618 951220.510870 312483748.018300 /
> [397948.367360] kworker/u17:6 21267 21365561.424208 10
> 100 21365561.424208 2.594598 27216695.208805 /
> [397948.367465] kworker/u16:1 22824 22358073.568236 4
> 120 22358073.568236 0.080475 4901.819753 /
> [397948.367569]
> [397948.367615] cpu#5, 3411.800 MHz
> [397948.367664] .nr_running : 0
> [397948.367714] .load : 0
> [397948.367764] .nr_switches : 29264631
> [397948.367816] .nr_load_updates : 3974073
> [397948.367867] .nr_uninterruptible : 5752
> [397948.367917] .next_balance : 4334.753132
> [397948.367969] .curr->pid : 0
> [397948.368020] .clock : 397948363.266965
> [397948.368072] .cpu_load[0] : 0
> [397948.368122] .cpu_load[1] : 0
> [397948.368172] .cpu_load[2] : 0
> [397948.368222] .cpu_load[3] : 0
> [397948.368272] .cpu_load[4] : 0
> [397948.368323] .yld_count : 23948
> [397948.368373] .sched_count : 29418990
> [397948.368424] .sched_goidle : 12116624
> [397948.368475] .avg_idle : 1000000
> [397948.368526] .max_idle_balance_cost : 500000
> [397948.368577] .ttwu_count : 14464828
> [397948.368628] .ttwu_local : 3286532
> [397948.368679]
> [397948.368679] cfs_rq[5]:/
> [397948.368771] .exec_clock : 21096887.422727
> [397948.368823] .MIN_vruntime : 0.000001
> [397948.368873] .min_vruntime : 24070921.092851
> [397948.368926] .max_vruntime : 0.000001
> [397948.368977] .spread : 0.000000
> [397948.369028] .spread0 : -72754695.958710
> [397948.369080] .nr_spread_over : 219
> [397948.369131] .nr_running : 0
> [397948.369181] .load : 0
> [397948.369232] .runnable_load_avg : 0
> [397948.369282] .blocked_load_avg : 0
> [397948.369332] .tg_load_contrib : 0
> [397948.369382] .tg_runnable_contrib : 0
> [397948.369432] .tg_load_avg : 1315
> [397948.369483] .tg->runnable_avg : 1290
> [397948.369534] .tg->cfs_bandwidth.timer_active: 0
> [397948.369584] .throttled : 0
> [397948.369634] .throttle_count : 0
> [397948.369685] .avg->runnable_avg_sum : 0
> [397948.369735] .avg->runnable_avg_period : 47521
> [397948.369786]
> [397948.369786] rt_rq[5]:/
> [397948.369878] .rt_nr_running : 0
> [397948.369928] .rt_throttled : 0
> [397948.369979] .rt_time : 0.000000
> [397948.370030] .rt_runtime : 950.000000
> [397948.370081]
> [397948.370081] dl_rq[5]:
> [397948.371423] .dl_nr_running : 0
> [397948.371474]
> [397948.371474] runnable tasks:
> [397948.371474] task PID tree-key switches
> prio exec-runtime sum-exec sum-sleep
> [397948.371474] ----------------------------------------------------------------------------------------------------------
> [397948.371769] watchdog/5 31 -8.977217 99543
> 0 -8.977217 2363.156345 32.364766 /
> [397948.371873] migration/5 32 0.000000 25712
> 0 0.000000 15366.174996 0.000845 /
> [397948.371977] ksoftirqd/5 33 24070879.321423 15403
> 120 24070879.321423 155.107046 397934935.415490 /
> [397948.372082] kworker/5:0 34 239.267711 17
> 120 239.267711 0.034507 632.432629 /
> [397948.372186] kworker/5:0H 35 632.111736 6
> 100 632.111736 0.059549 2815.362585 /
> [397948.372291] kworker/5:1 611 24070909.178313 2597656
> 120 24070909.178313 1203696.900841 396713444.701918 /
> [397948.372396] scsi_tmf_2 860 534.239404 2
> 100 534.239404 0.008896 0.005460 /
> [397948.372500] scsi_tmf_5 866 558.255161 2
> 100 558.255161 0.008179 0.004413 /
> [397948.372605] kworker/5:1H 1061 23860072.399440 45682
> 100 23860072.399440 510.190883 394083026.021665 /
> [397948.372711]
> [397948.372757] cpu#6, 3411.800 MHz
> [397948.372805] .nr_running : 0
> [397948.372855] .load : 0
> [397948.372905] .nr_switches : 30595250
> [397948.372972] .nr_load_updates : 4632598
> [397948.373039] .nr_uninterruptible : 6640
> [397948.373089] .next_balance : 4334.753132
> [397948.373140] .curr->pid : 0
> [397948.373191] .clock : 397948363.267655
> [397948.373243] .cpu_load[0] : 0
> [397948.373292] .cpu_load[1] : 0
> [397948.373342] .cpu_load[2] : 0
> [397948.373393] .cpu_load[3] : 0
> [397948.373443] .cpu_load[4] : 0
> [397948.373494] .yld_count : 20767
> [397948.373545] .sched_count : 30739543
> [397948.373596] .sched_goidle : 12161262
> [397948.373647] .avg_idle : 1000000
> [397948.373698] .max_idle_balance_cost : 500000
> [397948.373749] .ttwu_count : 17363747
> [397948.373800] .ttwu_local : 4024881
> [397948.373852]
> [397948.373852] cfs_rq[6]:/
> [397948.373944] .exec_clock : 27066813.057347
> [397948.373996] .MIN_vruntime : 0.000001
> [397948.374046] .min_vruntime : 29909549.644335
> [397948.374099] .max_vruntime : 0.000001
> [397948.374150] .spread : 0.000000
> [397948.374202] .spread0 : -66916067.407226
> [397948.374253] .nr_spread_over : 407
> [397948.374304] .nr_running : 0
> [397948.374354] .load : 0
> [397948.374404] .runnable_load_avg : 0
> [397948.374454] .blocked_load_avg : 0
> [397948.374504] .tg_load_contrib : 0
> [397948.374554] .tg_runnable_contrib : 0
> [397948.374605] .tg_load_avg : 1313
> [397948.374656] .tg->runnable_avg : 1289
> [397948.374706] .tg->cfs_bandwidth.timer_active: 0
> [397948.374757] .throttled : 0
> [397948.374807] .throttle_count : 0
> [397948.374857] .avg->runnable_avg_sum : 0
> [397948.374907] .avg->runnable_avg_period : 47423
> [397948.374958]
> [397948.374958] rt_rq[6]:/
> [397948.375050] .rt_nr_running : 0
> [397948.375099] .rt_throttled : 0
> [397948.375149] .rt_time : 0.000000
> [397948.375200] .rt_runtime : 950.000000
> [397948.375258]
> [397948.375258] dl_rq[6]:
> [397948.375359] .dl_nr_running : 0
> [397948.375410]
> [397948.375410] runnable tasks:
> [397948.375410] task PID tree-key switches
> prio exec-runtime sum-exec sum-sleep
> [397948.375410] ----------------------------------------------------------------------------------------------------------
> [397948.375704] watchdog/6 36 -8.979370 99543
> 0 -8.979370 3873.032642 19.046238 /
> [397948.375809] migration/6 37 0.000000 20867
> 0 0.000000 12947.385988 0.000826 /
> [397948.375914] ksoftirqd/6 38 29909537.744505 17979
> 120 29909537.744505 324.324395 397932684.911680 /
> [397948.376018] kworker/6:0 39 2652.785847 15
> 120 2652.785847 0.034665 619.334482 /
> [397948.376123] kworker/6:0H 40 3222.777674 6
> 100 3222.777674 0.042578 34131.960421 /
> [397948.376228] kworker/6:1 613 29909537.712176 3191586
> 120 29909537.712176 1584243.664849 396333417.817101 /
> [397948.376333] bioset 898 2957.541580 2
> 100 2957.541580 0.008747 0.006389 /
> [397948.376438] kworker/6:1H 1463 29688535.473225 50544
> 100 29688535.473225 612.882638 394061042.264128 /
> [397948.376542] kjournald 2216 3410.060184 2
> 120 3410.060184 0.412389 0.043474 /
> [397948.376646] bioset 2222 3422.126220 2
> 100 3422.126220 0.088585 0.040787 /
> [397948.376750] rsyslogd 3061 163480.988716 3499
> 120 163480.988716 146.745864 397862580.052157 /autogroup-6
> [397948.376856] mysqld 4122 163443.475621 644351
> 120 163443.475621 179019.020540 393898317.914127 /autogroup-6
> [397948.376962] udevd 6137 246.142493 30
> 120 246.142493 6.670412 22056460.101598 /autogroup-4
> [397948.377068] udevd 6138 242.586598 15
> 120 242.586598 3.149366 49194.013301 /autogroup-4
> [397948.377174] btrfs-worker 6246 1051403.790298 2
> 100 1051403.790298 0.011158 0.004647 /
> [397948.377278] btrfs-endio-met 6255 1051427.810805 2
> 100 1051427.810805 0.017787 0.002771 /
> [397948.377383] btrfs-endio-rai 6257 1051439.820346 2
> 100 1051439.820346 0.010462 0.002911 /
> [397948.377487] btrfs-endio-rep 6258 1051451.828334 2
> 100 1051451.828334 0.008652 0.003137 /
> [397948.377591] btrfs-endio-wri 6260 1051463.836060 2
> 100 1051463.836060 0.008410 0.003085 /
> [397948.377695] btrfs-readahead 6263 1051475.843882 2
> 100 1051475.843882 0.008543 0.003153 /
> [397948.377799] btrfs-extent-re 6265 1051487.851432 2
> 100 1051487.851432 0.008154 0.003265 /
> [397948.377904]
> [397948.377950] cpu#7, 3411.800 MHz
> [397948.377998] .nr_running : 0
> [397948.378048] .load : 0
> [397948.378098] .nr_switches : 28501260
> [397948.378150] .nr_load_updates : 3359354
> [397948.378200] .nr_uninterruptible : 4860
> [397948.378251] .next_balance : 4334.753132
> [397948.378302] .curr->pid : 0
> [397948.378351] .clock : 397948363.268406
> [397948.378403] .cpu_load[0] : 0
> [397948.378453] .cpu_load[1] : 0
> [397948.378504] .cpu_load[2] : 0
> [397948.378554] .cpu_load[3] : 0
> [397948.378604] .cpu_load[4] : 0
> [397948.378654] .yld_count : 22779
> [397948.378705] .sched_count : 28647495
> [397948.378757] .sched_goidle : 12446869
> [397948.378808] .avg_idle : 1000000
> [397948.378859] .max_idle_balance_cost : 500000
> [397948.378910] .ttwu_count : 18391339
> [397948.378961] .ttwu_local : 2717640
> [397948.379012]
> [397948.379012] cfs_rq[7]:/
> [397948.379112] .exec_clock : 13565979.621209
> [397948.379173] .MIN_vruntime : 0.000001
> [397948.379225] .min_vruntime : 17262906.632013
> [397948.379276] .max_vruntime : 0.000001
> [397948.379327] .spread : 0.000000
> [397948.379378] .spread0 : -79562710.419548
> [397948.379430] .nr_spread_over : 255
> [397948.379480] .nr_running : 0
> [397948.379530] .load : 0
> [397948.379581] .runnable_load_avg : 0
> [397948.379631] .blocked_load_avg : 0
> [397948.379682] .tg_load_contrib : 0
> [397948.379732] .tg_runnable_contrib : 0
> [397948.379783] .tg_load_avg : 1313
> [397948.379833] .tg->runnable_avg : 1288
> [397948.379884] .tg->cfs_bandwidth.timer_active: 0
> [397948.379933] .throttled : 0
> [397948.379983] .throttle_count : 0
> [397948.380034] .avg->runnable_avg_sum : 5
> [397948.380084] .avg->runnable_avg_period : 47099
> [397948.380135]
> [397948.380135] rt_rq[7]:/
> [397948.380227] .rt_nr_running : 0
> [397948.380277] .rt_throttled : 0
> [397948.380327] .rt_time : 0.000000
> [397948.380379] .rt_runtime : 950.000000
> [397948.380430]
> [397948.380430] dl_rq[7]:
> [397948.380522] .dl_nr_running : 0
> [397948.380572]
> [397948.380572] runnable tasks:
> [397948.380572] task PID tree-key switches
> prio exec-runtime sum-exec sum-sleep
> [397948.380572] ----------------------------------------------------------------------------------------------------------
> [397948.382132] watchdog/7 41 -8.982956 99544
> 0 -8.982956 1585.113591 5.725643 /
> [397948.382236] migration/7 42 0.000000 21048
> 0 0.000000 12757.115970 0.000816 /
> [397948.382341] ksoftirqd/7 43 17254754.851360 11826
> 120 17254754.851360 146.221401 397720187.660046 /
> [397948.382446] kworker/7:0 44 479.385599 15
> 120 479.385599 0.031395 606.240559 /
> [397948.382550] kworker/7:0H 45 1817.499047 6
> 100 1817.499047 0.047704 42727.098840 /
> [397948.382655] netns 47 10.955152 2
> 100 10.955152 0.004076 0.001586 /
> [397948.382759] perf 50 22.958573 2
> 100 22.958573 0.003794 0.001394 /
> [397948.382863] kworker/7:1 615 17262894.767216 1847690
> 120 17262894.767216 746881.932822 397173736.970567 /
> [397948.382992] md0_raid1 899 17253995.722314 32645
> 120 17253995.722314 997.731148 397503108.422867 /
> [397948.383105] bioset 919 1132.474026 2
> 100 1132.474026 0.034373 0.004008 /
> [397948.383210] ext4-rsv-conver 936 1156.590460 2
> 100 1156.590460 0.061309 0.026946 /
> [397948.383315] udevd 1124 405.566270 290
> 120 405.566270 53.586152 77350627.244245 /autogroup-4
> [397948.383421] ipv6_addrconf 3209 1679.353321 2
> 100 1679.353321 0.006452 0.002581 /
> [397948.383527] kworker/7:1H 4116 16792468.563089 52255
> 100 16792468.563089 541.499314 393833366.670026 /
> [397948.383632] log 4283 574.060901 688
> 120 574.060901 34.058909 395113548.562211 /autogroup-17
> [397948.383739] getty 4376 0.726364 103
> 120 0.726364 1.882302 146.996217 /autogroup-25
> [397948.383845] btrfs-fixup 6253 1049025.643969 2
> 100 1049025.643969 0.011376 0.003865 /
> [397948.383950] btrfs-endio 6254 1049037.659031 2
> 100 1049037.659031 0.015618 0.001516 /
> [397948.384056]
>
>
> Tomasz Chmielewski
> http://www.sslrack.com
>
> --
> To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
next prev parent reply other threads:[~2015-02-16 10:40 UTC|newest]
Thread overview: 3+ messages / expand[flat|nested] mbox.gz Atom feed top
2015-02-16 2:32 3.19 - "unable" to replace a failed drive - 100% CPU usage in kworker and btrfs-transacti Tomasz Chmielewski
2015-02-16 10:40 ` Liu Bo [this message]
2015-02-16 10:51 ` Tomasz Chmielewski
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20150216104026.GC23683@localhost.localdomain \
--to=bo.li.liu@oracle.com \
--cc=linux-btrfs@vger.kernel.org \
--cc=tch@virtall.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox