All of lore.kernel.org
 help / color / mirror / Atom feed
* 3.19 - "unable" to replace a failed drive - 100% CPU usage in kworker and btrfs-transacti
@ 2015-02-16  2:32 Tomasz Chmielewski
  2015-02-16 10:40 ` Liu Bo
  0 siblings, 1 reply; 3+ messages in thread
From: Tomasz Chmielewski @ 2015-02-16  2:32 UTC (permalink / raw)
  To: linux-btrfs

I had a failed drive in RAID-1, so it was replaced with a good one, 
followed by:

btrfs device add /dev/sdb4 /home
btrfs device delete missing /home


4 days later, it got to a state when there is no IO anymore (according 
to iostat), "btrfs device delete missing" did not complete:

# uptime
  03:29:03 up 4 days, 14:38,  1 user,  load average: 2.36, 2.43, 2.54

# btrfs fi show
Label: none  uuid: 84d087aa-3a32-46da-844f-a233237cf04f
         Total devices 3 FS bytes used 206.53GiB
         devid    3 size 1.71TiB used 214.03GiB path /dev/sda4
         devid    4 size 1.71TiB used 177.00GiB path /dev/sdb4
         *** Some devices missing

Btrfs v3.18.2

   PID USER      PR  NI  VIRT  RES  SHR S  %CPU %MEM    TIME+  COMMAND
  6269 root      20   0     0    0    0 R  92.5  0.0   2769:33 
btrfs-transacti
22247 root      20   0     0    0    0 R  92.5  0.0  42:38.65 
kworker/u16:16


# dmesg
[397948.321324] SysRq : Show Blocked State
[397948.321386]   task                        PC stack   pid father
[397948.321465] btrfs           D ffff8802d67ef948     0  7611   6275 
0x00000000
[397948.321521]  ffff8802d67ef948 ffff8804d2a9ba40 ffff88081917e040 
00000000000113c0
[397948.321621]  0000000000004000 ffff88081be30000 ffff88081917e040 
ffff8802d67ef8b8
[397948.321720]  ffffffff8105f3c4 ffff88083fa40000 ffff88083fa913c0 
ffff88083fa913c0
[397948.321819] Call Trace:
[397948.321872]  [<ffffffff8105f3c4>] ? enqueue_task_fair+0x3e5/0x44f
[397948.321927]  [<ffffffff81054ab6>] ? resched_curr+0x45/0x55
[397948.321979]  [<ffffffff81055122>] ? check_preempt_curr+0x3e/0x6d
[397948.322032]  [<ffffffff81055163>] ? ttwu_do_wakeup+0x12/0x7f
[397948.322085]  [<ffffffff8105526e>] ? 
ttwu_do_activate.constprop.73+0x57/0x5c
[397948.322141]  [<ffffffff813be618>] schedule+0x65/0x67
[397948.322193]  [<ffffffff813c03b8>] schedule_timeout+0x26/0x18d
[397948.322245]  [<ffffffff81057d1e>] ? wake_up_process+0x30/0x34
[397948.322299]  [<ffffffff8104b1a3>] ? wake_up_worker+0x1f/0x21
[397948.322352]  [<ffffffff8104b3ea>] ? insert_work+0x87/0x94
[397948.322414]  [<ffffffffa02e183d>] ? free_block_list+0x1f/0x34 
[btrfs]
[397948.322468]  [<ffffffff813bef76>] wait_for_common+0x10d/0x13e
[397948.322521]  [<ffffffff81057cdf>] ? try_to_wake_up+0x250/0x250
[397948.322574]  [<ffffffff813befbf>] wait_for_completion+0x18/0x1a
[397948.322631]  [<ffffffffa028dedc>] 
btrfs_async_run_delayed_refs+0xc1/0xe4 [btrfs]
[397948.322734]  [<ffffffffa02a3189>] 
__btrfs_end_transaction+0x315/0x33b [btrfs]
[397948.322836]  [<ffffffffa02a31bd>] 
btrfs_end_transaction_throttle+0xe/0x10 [btrfs]
[397948.322939]  [<ffffffffa02e5cf1>] relocate_block_group+0x2ad/0x4de 
[btrfs]
[397948.323039]  [<ffffffffa02e607a>] 
btrfs_relocate_block_group+0x158/0x278 [btrfs]
[397948.323152]  [<ffffffffa02c37e1>] 
btrfs_relocate_chunk.isra.69+0x35/0xa5 [btrfs]
[397948.323256]  [<ffffffffa02c41da>] btrfs_shrink_device+0x235/0x408 
[btrfs]
[397948.323316]  [<ffffffffa02c6ba9>] btrfs_rm_device+0x2a9/0x704 
[btrfs]
[397948.323371]  [<ffffffff810f5441>] ? 
__kmalloc_track_caller+0x40/0x178
[397948.323431]  [<ffffffffa02cf7a7>] btrfs_ioctl+0xa9c/0x22f5 [btrfs]
[397948.323485]  [<ffffffff8110f142>] ? putname+0x23/0x2c
[397948.323537]  [<ffffffff8110f6d3>] ? user_path_at_empty+0x60/0x90
[397948.323593]  [<ffffffff811783bf>] ? avc_has_perm+0x2e/0xf7
[397948.323654]  [<ffffffff8111171b>] do_vfs_ioctl+0x418/0x460
[397948.323706]  [<ffffffff81106745>] ? vfs_stat+0x16/0x18
[397948.323758]  [<ffffffff811117b1>] SyS_ioctl+0x4e/0x7d
[397948.323809]  [<ffffffff81031c5b>] ? do_page_fault+0xc/0x11
[397948.323862]  [<ffffffff813c1352>] system_call_fastpath+0x12/0x17
[397948.323917] Sched Debug Version: v0.11, 3.19.0 #1
[397948.323968] ktime                                   : 
398158300.945440
[397948.324021] sched_clk                               : 
397948323.916497
[397948.324074] cpu_clk                                 : 
397948323.916513
[397948.324127] jiffies                                 : 4334753127
[397948.324180] sched_clock_stable()                    : 1
[397948.324231]
[397948.324277] sysctl_sched
[397948.324325]   .sysctl_sched_latency                    : 24.000000
[397948.324378]   .sysctl_sched_min_granularity            : 3.000000
[397948.324430]   .sysctl_sched_wakeup_granularity         : 4.000000
[397948.324483]   .sysctl_sched_child_runs_first           : 0
[397948.324534]   .sysctl_sched_features                   : 11899
[397948.324586]   .sysctl_sched_tunable_scaling            : 1 
(logaritmic)
[397948.324639]
[397948.324686] cpu#0, 3411.800 MHz
[397948.324734]   .nr_running                    : 0
[397948.324784]   .load                          : 0
[397948.324834]   .nr_switches                   : 137328510
[397948.324885]   .nr_load_updates               : 23262071
[397948.324936]   .nr_uninterruptible            : -738004
[397948.324987]   .next_balance                  : 4334.753128
[397948.325039]   .curr->pid                     : 0
[397948.325089]   .clock                         : 397948323.134939
[397948.325141]   .cpu_load[0]                   : 0
[397948.325191]   .cpu_load[1]                   : 0
[397948.325241]   .cpu_load[2]                   : 0
[397948.325292]   .cpu_load[3]                   : 0
[397948.325343]   .cpu_load[4]                   : 0
[397948.325393]   .yld_count                     : 223168
[397948.325444]   .sched_count                   : 137673310
[397948.325496]   .sched_goidle                  : 50221823
[397948.325547]   .avg_idle                      : 878593
[397948.325599]   .max_idle_balance_cost         : 500000
[397948.325650]   .ttwu_count                    : 86464371
[397948.325701]   .ttwu_local                    : 52888650
[397948.325752]
[397948.325752] cfs_rq[0]:/autogroup-1114
[397948.325847]   .exec_clock                    : 11.086901
[397948.325899]   .MIN_vruntime                  : 0.000001
[397948.325950]   .min_vruntime                  : 10.038325
[397948.326001]   .max_vruntime                  : 0.000001
[397948.326052]   .spread                        : 0.000000
[397948.326103]   .spread0                       : -96825607.013236
[397948.326155]   .nr_spread_over                : 0
[397948.326205]   .nr_running                    : 0
[397948.326256]   .load                          : 0
[397948.326306]   .runnable_load_avg             : 0
[397948.326357]   .blocked_load_avg              : 0
[397948.326407]   .tg_load_contrib               : 0
[397948.326458]   .tg_runnable_contrib           : 0
[397948.326508]   .tg_load_avg                   : 0
[397948.326559]   .tg->runnable_avg              : 0
[397948.326609]   .tg->cfs_bandwidth.timer_active: 0
[397948.326659]   .throttled                     : 0
[397948.326710]   .throttle_count                : 0
[397948.326760]   .se->exec_start                : 397948321.211399
[397948.326812]   .se->vruntime                  : 96825606.516443
[397948.326865]   .se->sum_exec_runtime          : 11.086901
[397948.326916]   .se->statistics.wait_start     : 0.000000
[397948.326967]   .se->statistics.sleep_start    : 0.000000
[397948.327018]   .se->statistics.block_start    : 0.000000
[397948.327069]   .se->statistics.sleep_max      : 0.000000
[397948.327121]   .se->statistics.block_max      : 0.000000
[397948.327172]   .se->statistics.exec_max       : 2.982069
[397948.327223]   .se->statistics.slice_max      : 0.000000
[397948.327275]   .se->statistics.wait_max       : 0.019006
[397948.327326]   .se->statistics.wait_sum       : 0.019006
[397948.327378]   .se->statistics.wait_count     : 136
[397948.327429]   .se->load.weight               : 2
[397948.327479]   .se->avg.runnable_avg_sum      : 42
[397948.327530]   .se->avg.runnable_avg_period   : 48335
[397948.327581]   .se->avg.load_avg_contrib      : 0
[397948.327631]   .se->avg.decay_count           : 379513094
[397948.327683]
[397948.327683] cfs_rq[0]:/autogroup-6
[397948.327777]   .exec_clock                    : 1407434.382402
[397948.327829]   .MIN_vruntime                  : 0.000001
[397948.327880]   .min_vruntime                  : 1154990.729705
[397948.327932]   .max_vruntime                  : 0.000001
[397948.327984]   .spread                        : 0.000000
[397948.328035]   .spread0                       : -95670626.321856
[397948.328087]   .nr_spread_over                : 255
[397948.329388]   .nr_running                    : 0
[397948.329438]   .load                          : 0
[397948.329489]   .runnable_load_avg             : 0
[397948.329539]   .blocked_load_avg              : 2
[397948.329590]   .tg_load_contrib               : 2
[397948.329640]   .tg_runnable_contrib           : 4
[397948.329690]   .tg_load_avg                   : 12
[397948.329740]   .tg->runnable_avg              : 4
[397948.329791]   .tg->cfs_bandwidth.timer_active: 0
[397948.329841]   .throttled                     : 0
[397948.329892]   .throttle_count                : 0
[397948.329943]   .se->exec_start                : 397948323.133861
[397948.329995]   .se->vruntime                  : 96825606.879532
[397948.330047]   .se->sum_exec_runtime          : 1407466.511198
[397948.330098]   .se->statistics.wait_start     : 0.000000
[397948.330150]   .se->statistics.sleep_start    : 0.000000
[397948.330201]   .se->statistics.block_start    : 0.000000
[397948.330252]   .se->statistics.sleep_max      : 0.000000
[397948.330303]   .se->statistics.block_max      : 0.000000
[397948.330354]   .se->statistics.exec_max       : 10.021049
[397948.330406]   .se->statistics.slice_max      : 16.519045
[397948.330457]   .se->statistics.wait_max       : 16.098230
[397948.330509]   .se->statistics.wait_sum       : 77143.758185
[397948.330561]   .se->statistics.wait_count     : 12935361
[397948.330613]   .se->load.weight               : 2
[397948.330663]   .se->avg.runnable_avg_sum      : 206
[397948.330714]   .se->avg.runnable_avg_period   : 47204
[397948.330765]   .se->avg.load_avg_contrib      : 2
[397948.330816]   .se->avg.decay_count           : 379513096
[397948.330868]
[397948.330868] cfs_rq[0]:/
[397948.330959]   .exec_clock                    : 98523331.737970
[397948.331011]   .MIN_vruntime                  : 0.000001
[397948.331062]   .min_vruntime                  : 96825617.051561
[397948.331115]   .max_vruntime                  : 0.000001
[397948.331166]   .spread                        : 0.000000
[397948.331217]   .spread0                       : 0.000000
[397948.331269]   .nr_spread_over                : 270
[397948.331320]   .nr_running                    : 0
[397948.331370]   .load                          : 0
[397948.331421]   .runnable_load_avg             : 0
[397948.331471]   .blocked_load_avg              : 2
[397948.331522]   .tg_load_contrib               : 2
[397948.331572]   .tg_runnable_contrib           : 6
[397948.331622]   .tg_load_avg                   : 1490
[397948.331673]   .tg->runnable_avg              : 1068
[397948.331723]   .tg->cfs_bandwidth.timer_active: 0
[397948.331774]   .throttled                     : 0
[397948.331824]   .throttle_count                : 0
[397948.331875]   .avg->runnable_avg_sum         : 286
[397948.331926]   .avg->runnable_avg_period      : 46637
[397948.331979]
[397948.331979] rt_rq[0]:/
[397948.332071]   .rt_nr_running                 : 0
[397948.332121]   .rt_throttled                  : 0
[397948.332171]   .rt_time                       : 0.000000
[397948.332222]   .rt_runtime                    : 950.000000
[397948.332274]
[397948.332274] dl_rq[0]:
[397948.332366]   .dl_nr_running                 : 0
[397948.332417]
[397948.332417] runnable tasks:
[397948.332417]             task   PID         tree-key  switches  prio  
    exec-runtime         sum-exec        sum-sleep
[397948.332417] 
----------------------------------------------------------------------------------------------------------
[397948.332712]      ksoftirqd/0     3  96825605.109947    740015   120  
96825605.109947     12247.009129 397859768.730344 /
[397948.332817]      kworker/0:0     4  96823284.697816        53   120  
96823284.697816        17.309475 397934074.622814 /
[397948.332922]     kworker/0:0H     5      1015.287609         5   100  
     1015.287609         0.041666      1136.081722 /
[397948.333053]        rcu_sched     7  96825608.780644  13143448   120  
96825608.780644    369650.173462 397567779.683357 /
[397948.333157]           rcu_bh     8       117.354314         2   120  
      117.354314         0.000874         0.000722 /
[397948.333263]      migration/0     9         0.000000     40752     0  
        0.000000     37728.554175         0.000853 /
[397948.333367]       watchdog/0    10       -11.786179     99549     0  
      -11.786179     16602.242686        93.369695 /
[397948.333473]          kswapd0   506  96346153.194271     10895   120  
96346153.194271      5358.878543 396356848.501371 /
[397948.333577]  acpi_thermal_pm   618       478.127841         2   100  
      478.127841         0.008079         0.004241 /
[397948.333682]          ata_sff   832       943.140921         2   100  
      943.140921         0.005880         0.022278 /
[397948.333786]        scsi_eh_0   855      1012.563036        18   120  
     1012.563036         0.875745       352.505402 /
[397948.333891]     kworker/0:1H   879  96825605.051890  19712713   100  
96825605.051890    933447.862690 396865888.829352 /
[397948.333995]       jbd2/md2-8   935  96824920.898850    673247   120  
96824920.898850     25939.369714 397908678.411380 /
[397948.334100]         rsyslogd  3062   1154978.849958       554   120  
  1154978.849958        31.915075 397909811.026141 /autogroup-6
[397948.334206]            nginx  3235      4173.366889    645371   120  
     4173.366889      8854.802084 393651417.627526 /autogroup-10
[397948.334312]           mysqld  4080   1154807.414715      5643   120  
  1154807.414715       799.670146 395527445.704335 /autogroup-6
[397948.334418]           mysqld  4094   1154978.441261    796267   120  
  1154978.441261      7509.987667 397898098.956672 /autogroup-6
[397948.334523]           mysqld  4097   1154978.442889   1253821   120  
  1154978.442889     54810.874030 397848250.280201 /autogroup-6
[397948.334628]           mysqld  4098   1154978.441618   1282226   120  
  1154978.441618     57182.533918 397845845.250436 /autogroup-6
[397948.334734]           mysqld  4102   1154978.441697    959988   120  
  1154978.441697     10108.715585 397895341.475090 /autogroup-6
[397948.334839]           mysqld  4103   1154978.440507   1341686   120  
  1154978.440507     16033.050027 397888981.976267 /autogroup-6
[397948.334945]           mysqld  4119   1154990.729705    404501   120  
  1154990.729705    119263.212981 397785618.876052 /autogroup-6
[397948.335051]           mysqld  4150   1039299.577821   7023984   120  
  1039299.577821   1407433.681869 320824575.671761 /autogroup-6
[397948.335157]           mysqld  5811   1154806.870976   1317671   120  
  1154806.870976    299564.211303 346391209.749613 /autogroup-6
[397948.335264]           mysqld 15436   1154807.136306    292526   120  
  1154807.136306    172005.952402 174398934.514010 /autogroup-6
[397948.335371]            getty  4374         0.817181       100   120  
        0.817181         2.613716       146.560614 /autogroup-23
[397948.335478]  btrfs-transacti  6269  96825617.051561  22374147   120  
96825617.051561 166025164.637843 164906022.709080 /
[397948.335583]             bash  6275       132.300150       276   120  
      132.300150        79.740920  21744924.508065 /autogroup-184
[397948.335690]      kworker/0:1 21018  96823285.353043   1816351   120  
96823285.353043    793610.285186  42038707.779192 /
[397948.335795]       ssl-params 22730       472.603845         4   120  
      472.603845         0.877328         0.117947 /autogroup-17
[397948.335902]    kworker/u16:0 22761  96825069.356076     67516   120  
96825069.356076    540082.349638    226131.772902 /
[397948.336006]             sshd 22771        10.038325       150   120  
       10.038325        13.472329     50175.143940 /autogroup-1114
[397948.336112]      kworker/0:2 22822  96825605.578025       507   120  
96825605.578025        47.279505     14055.084212 /
[397948.336218]
[397948.336280] cpu#1, 3411.800 MHz
[397948.336329]   .nr_running                    : 1
[397948.336379]   .load                          : 1024
[397948.336430]   .nr_switches                   : 84332707
[397948.336481]   .nr_load_updates               : 21967275
[397948.336533]   .nr_uninterruptible            : 483742
[397948.336584]   .next_balance                  : 4334.753129
[397948.336635]   .curr->pid                     : 22247
[397948.336686]   .clock                         : 397948333.217376
[397948.336738]   .cpu_load[0]                   : 1023
[397948.336788]   .cpu_load[1]                   : 1023
[397948.336839]   .cpu_load[2]                   : 1023
[397948.336890]   .cpu_load[3]                   : 1023
[397948.336940]   .cpu_load[4]                   : 1007
[397948.336991]   .yld_count                     : 135412
[397948.337043]   .sched_count                   : 84726950
[397948.337094]   .sched_goidle                  : 29157952
[397948.337145]   .avg_idle                      : 1000000
[397948.337196]   .max_idle_balance_cost         : 500000
[397948.337247]   .ttwu_count                    : 36438464
[397948.337299]   .ttwu_local                    : 19485954
[397948.337350]
[397948.337350] cfs_rq[1]:/
[397948.337441]   .exec_clock                    : 123202243.887995
[397948.337494]   .MIN_vruntime                  : 0.000001
[397948.337545]   .min_vruntime                  : 119610642.735572
[397948.337597]   .max_vruntime                  : 0.000001
[397948.337649]   .spread                        : 0.000000
[397948.337700]   .spread0                       : 22785025.684011
[397948.337752]   .nr_spread_over                : 301
[397948.337802]   .nr_running                    : 1
[397948.337852]   .load                          : 1024
[397948.337903]   .runnable_load_avg             : 1023
[397948.337954]   .blocked_load_avg              : 26
[397948.338004]   .tg_load_contrib               : 1042
[397948.338055]   .tg_runnable_contrib           : 1010
[397948.338106]   .tg_load_avg                   : 1317
[397948.338156]   .tg->runnable_avg              : 1293
[397948.338206]   .tg->cfs_bandwidth.timer_active: 0
[397948.338256]   .throttled                     : 0
[397948.338307]   .throttle_count                : 0
[397948.338357]   .avg->runnable_avg_sum         : 46120
[397948.338408]   .avg->runnable_avg_period      : 46120
[397948.338459]
[397948.338459] rt_rq[1]:/
[397948.338550]   .rt_nr_running                 : 0
[397948.338600]   .rt_throttled                  : 0
[397948.339900]   .rt_time                       : 0.000000
[397948.339951]   .rt_runtime                    : 950.000000
[397948.340003]
[397948.340003] dl_rq[1]:
[397948.340094]   .dl_nr_running                 : 0
[397948.340144]
[397948.340144] runnable tasks:
[397948.340144]             task   PID         tree-key  switches  prio  
    exec-runtime         sum-exec        sum-sleep
[397948.340144] 
----------------------------------------------------------------------------------------------------------
[397948.340438]       watchdog/1    11        -2.973570     99543     0  
       -2.973570     21159.974792        85.639910 /
[397948.340543]      migration/1    12         0.000000     37023     0  
        0.000000     30622.595831         0.000758 /
[397948.340648]      ksoftirqd/1    13 119609979.133122     91495   120 
119609979.133122      1646.457418 397900354.563118 /
[397948.340752]     kworker/1:0H    15       582.372111         6   100  
      582.372111         0.027131      2733.616385 /
[397948.340858]       khungtaskd   204 119577195.763580      3319   120 
119577195.763580       182.485563 397829939.496656 /
[397948.340962]        kpsmoused   686        25.893051         2   100  
       25.893051         0.010838         0.003636 /
[397948.341068]      kworker/1:2   688 102920251.057920        55   120 
102920251.057920         0.504998 355091633.648033 /
[397948.341172]          deferwq   731       158.015576         2   100  
      158.015576         0.009076         0.003883 /
[397948.341277]       scsi_tmf_0   856       442.536328         2   100  
      442.536328         0.008945         0.003987 /
[397948.341382]        scsi_eh_1   857       539.461151        16   120  
      539.461151         0.868588       350.508685 /
[397948.341486]       scsi_tmf_1   858       466.554669         2   100  
      466.554669         0.008786         0.003948 /
[397948.341590]        scsi_eh_2   859       514.602593         3   120  
      514.602593         0.016414         1.372299 /
[397948.341695]        scsi_eh_3   861       514.603810         3   120  
      514.603810         0.016837         1.222996 /
[397948.341799]       scsi_tmf_3   862       502.578935         2   100  
      502.578935         0.008052         0.003896 /
[397948.341904]       scsi_tmf_4   864       514.587500         2   100  
      514.587500         0.008905         0.004404 /
[397948.342009]     kworker/1:1H   968 118250346.559508    456123   100 
118250346.559508      6526.369618 394080278.360060 /
[397948.342114]             cron  3301       777.910417      7708   120  
      777.910417       405.566912 397904071.914015 /autogroup-12
[397948.342219]             sshd  3426         0.515010        28   120  
        0.515010         3.325645 397861007.941426 /autogroup-15
[397948.342326]      mysqld_safe  3464      1418.978306       435   120  
     1418.978306        26.375085       974.715839 /autogroup-6
[397948.342432]           mysqld  4118    690369.357591    400760   120  
   690369.357591     50887.461054 397853256.747730 /autogroup-6
[397948.342539]            getty  4372         0.769188       104   120  
        0.769188         2.833955       146.554963 /autogroup-21
[397948.342645]    btrfs-cleaner  6268 119610036.344925     21815   120 
119610036.344925      5411.552915 342317734.332486 /
[397948.342750]            btrfs  7611    311971.962451   2055291   120  
   311971.962451   1498599.957865 319025222.198539 /autogroup-184
[397948.342856]      kworker/1:0 21019 119610630.984203   2134678   120 
119610630.984203   1030853.178543  41814652.278685 /
[397948.342969] R kworker/u16:16 22247 119610652.481537    293438   120 
119610652.481537   2301421.627687   2218308.587596 /
[397948.343099]
[397948.343161] cpu#2, 3411.800 MHz
[397948.343210]   .nr_running                    : 3
[397948.343259]   .load                          : 2048
[397948.343310]   .nr_switches                   : 76867052
[397948.343360]   .nr_load_updates               : 19847327
[397948.343411]   .nr_uninterruptible            : 139945
[397948.343462]   .next_balance                  : 4334.753139
[397948.343513]   .curr->pid                     : 22773
[397948.343563]   .clock                         : 397948336.246820
[397948.343615]   .cpu_load[0]                   : 275
[397948.343666]   .cpu_load[1]                   : 212
[397948.343716]   .cpu_load[2]                   : 127
[397948.343767]   .cpu_load[3]                   : 70
[397948.343818]   .cpu_load[4]                   : 50
[397948.343868]   .yld_count                     : 92399
[397948.343919]   .sched_count                   : 77151242
[397948.343970]   .sched_goidle                  : 27487304
[397948.344021]   .avg_idle                      : 1000000
[397948.344072]   .max_idle_balance_cost         : 500000
[397948.344123]   .ttwu_count                    : 35275362
[397948.344175]   .ttwu_local                    : 17426867
[397948.344227]
[397948.344227] cfs_rq[2]:/autogroup-1115
[397948.344321]   .exec_clock                    : 77.755926
[397948.344372]   .MIN_vruntime                  : 0.000001
[397948.344423]   .min_vruntime                  : 169.689274
[397948.344475]   .max_vruntime                  : 0.000001
[397948.344526]   .spread                        : 0.000000
[397948.344578]   .spread0                       : -96825447.362287
[397948.344629]   .nr_spread_over                : 7
[397948.344680]   .nr_running                    : 1
[397948.344730]   .load                          : 1024
[397948.344781]   .runnable_load_avg             : 271
[397948.344831]   .blocked_load_avg              : 0
[397948.344882]   .tg_load_contrib               : 271
[397948.344932]   .tg_runnable_contrib           : 271
[397948.344983]   .tg_load_avg                   : 271
[397948.345033]   .tg->runnable_avg              : 271
[397948.345083]   .tg->cfs_bandwidth.timer_active: 0
[397948.345133]   .throttled                     : 0
[397948.345183]   .throttle_count                : 0
[397948.345234]   .se->exec_start                : 397948336.246820
[397948.345286]   .se->vruntime                  : 100334135.344609
[397948.345338]   .se->sum_exec_runtime          : 77.755926
[397948.345389]   .se->statistics.wait_start     : 0.000000
[397948.345440]   .se->statistics.sleep_start    : 0.000000
[397948.345491]   .se->statistics.block_start    : 0.000000
[397948.345542]   .se->statistics.sleep_max      : 0.000000
[397948.345593]   .se->statistics.block_max      : 0.000000
[397948.345644]   .se->statistics.exec_max       : 13.202718
[397948.345694]   .se->statistics.slice_max      : 0.000000
[397948.345745]   .se->statistics.wait_max       : 0.020544
[397948.345796]   .se->statistics.wait_sum       : 0.209415
[397948.345847]   .se->statistics.wait_count     : 128
[397948.345898]   .se->load.weight               : 1024
[397948.345949]   .se->avg.runnable_avg_sum      : 12487
[397948.345999]   .se->avg.runnable_avg_period   : 47064
[397948.346051]   .se->avg.load_avg_contrib      : 275
[397948.346101]   .se->avg.decay_count           : 0
[397948.346152]
[397948.346152] cfs_rq[2]:/
[397948.346244]   .exec_clock                    : 102436933.654759
[397948.346296]   .MIN_vruntime                  : 100334120.416350
[397948.346348]   .min_vruntime                  : 100334132.312627
[397948.346401]   .max_vruntime                  : 100334120.416350
[397948.346453]   .spread                        : 0.000000
[397948.346504]   .spread0                       : 3508515.261066
[397948.346556]   .nr_spread_over                : 450
[397948.346606]   .nr_running                    : 2
[397948.346657]   .load                          : 2048
[397948.346708]   .runnable_load_avg             : 275
[397948.346758]   .blocked_load_avg              : 0
[397948.346809]   .tg_load_contrib               : 275
[397948.346859]   .tg_runnable_contrib           : 271
[397948.346910]   .tg_load_avg                   : 1323
[397948.346961]   .tg->runnable_avg              : 1293
[397948.347011]   .tg->cfs_bandwidth.timer_active: 0
[397948.347061]   .throttled                     : 0
[397948.347111]   .throttle_count                : 0
[397948.347161]   .avg->runnable_avg_sum         : 12763
[397948.347212]   .avg->runnable_avg_period      : 48021
[397948.347263]
[397948.347263] rt_rq[2]:/
[397948.347354]   .rt_nr_running                 : 0
[397948.347404]   .rt_throttled                  : 0
[397948.347455]   .rt_time                       : 0.000000
[397948.347506]   .rt_runtime                    : 950.000000
[397948.347557]
[397948.347557] dl_rq[2]:
[397948.347649]   .dl_nr_running                 : 0
[397948.347699]
[397948.347699] runnable tasks:
[397948.347699]             task   PID         tree-key  switches  prio  
    exec-runtime         sum-exec        sum-sleep
[397948.347699] 
----------------------------------------------------------------------------------------------------------
[397948.347992]             init     1       509.594135     79809   120  
      509.594135      2619.143889 397945091.854170 /autogroup-2
[397948.348098]       watchdog/2    16        -5.976382     99543     0  
       -5.976382     17476.538367        72.326265 /
[397948.348202]      migration/2    17         0.000000     32592     0  
        0.000000     29450.114444         0.001139 /
[397948.348306]      ksoftirqd/2    18 100333573.328598     69839   120 
100333573.328598      1929.169352 397913601.175468 /
[397948.348410]      kworker/2:0    19  49262954.665776        38   120  
49262954.665776         0.122608 231013304.220524 /
[397948.348514]     kworker/2:0H    20      3650.646229         6   100  
     3650.646229         0.037533     34187.208217 /
[397948.348619]        scsi_eh_4   863      3072.735732         3   120  
     3072.735732         0.015446         1.048287 /
[397948.348724]        scsi_eh_5   865      3072.733721         3   120  
     3072.733721         0.012919         0.910030 /
[397948.348828]        md1_raid1   910 100169530.338537      1429   120 
100169530.338537        66.597411 397503924.718962 /
[397948.348932]     kworker/2:1H  1466 100323518.282583    394863   100 
100323518.282583      5278.861299 397866015.984528 /
[397948.349036]         rsyslogd  3033    428725.147921        28   120  
   428725.147921         2.679249 322287744.247447 /autogroup-6
[397948.350392]              atd  3189         0.622217       112   120  
        0.622217         2.344474 395791202.210800 /autogroup-9
[397948.350498]            acpid  3240         0.478710        41   120  
        0.478710         1.730662 321163512.935538 /autogroup-11
[397948.350605]      dbus-daemon  3404         0.942824         6   120  
        0.942824         1.484584  55885009.390813 /autogroup-14
[397948.350710]           mysqld  4095    459653.767988    848117   120  
   459653.767988      9524.934466 397895732.816642 /autogroup-6
[397948.350817]           mysqld 15425    459502.220847    261927   120  
   459502.220847    145837.180966 174449080.595390 /autogroup-6
[397948.350923]             qmgr  4370       357.389137      2476   120  
      357.389137       280.199510 397622564.351466 /autogroup-19
[397948.351029]            getty  4373         0.640858       102   120  
        0.640858         2.425952       146.827128 /autogroup-22
[397948.351135]  btrfs-worker-hi  6248   1154879.053331         2   100  
  1154879.053331         0.017567         0.003220 /
[397948.351240]   btrfs-delalloc  6249   1154891.063212         2   100  
  1154891.063212         0.010622         0.003413 /
[397948.351345]  btrfs-flush_del  6250   1154903.071666         2   100  
  1154903.071666         0.009022         0.003320 /
[397948.351449]      btrfs-cache  6251   1154915.081079         2   100  
  1154915.081079         0.015404         0.003203 /
[397948.351554]     btrfs-submit  6252   1154915.162054         2   100  
  1154915.162054         0.096237         0.023172 /
[397948.351659]  btrfs-endio-met  6256   1154927.254127         2   100  
  1154927.254127         0.093435         0.002869 /
[397948.351763]        btrfs-rmw  6259   1154939.263666         2   100  
  1154939.263666         0.010258         0.003176 /
[397948.351868]  btrfs-freespace  6261   1154951.272823         2   100  
  1154951.272823         0.009779         0.003215 /
[397948.351972]  btrfs-delayed-m  6262   1154963.281556         2   100  
  1154963.281556         0.009338         0.003128 /
[397948.352076]  btrfs-qgroup-re  6264   1154975.290187         2   100  
  1154975.290187         0.009265         0.003113 /
[397948.352180]      kworker/2:2 16094 100334120.416350   6973086   120 
100334120.416350   3053548.894806 163847153.385628 /
[397948.352285]           pickup 22744       369.431995        35   120  
      369.431995         4.254482   1980726.020374 /autogroup-19
[397948.352391] R           bash 22773       169.689274       106   120  
      169.689274        49.563444     46410.281275 /autogroup-1115
[397948.352497]
[397948.352543] cpu#3, 3411.800 MHz
[397948.352591]   .nr_running                    : 0
[397948.352641]   .load                          : 0
[397948.352692]   .nr_switches                   : 78216804
[397948.352743]   .nr_load_updates               : 20561275
[397948.352794]   .nr_uninterruptible            : 84689
[397948.352845]   .next_balance                  : 4334.753130
[397948.352896]   .curr->pid                     : 0
[397948.352955]   .clock                         : 397948343.119396
[397948.353033]   .cpu_load[0]                   : 0
[397948.353100]   .cpu_load[1]                   : 0
[397948.353150]   .cpu_load[2]                   : 0
[397948.353200]   .cpu_load[3]                   : 0
[397948.353251]   .cpu_load[4]                   : 0
[397948.353300]   .yld_count                     : 73921
[397948.353351]   .sched_count                   : 78424464
[397948.353402]   .sched_goidle                  : 27817319
[397948.353453]   .avg_idle                      : 1000000
[397948.353504]   .max_idle_balance_cost         : 500000
[397948.353555]   .ttwu_count                    : 36987465
[397948.353605]   .ttwu_local                    : 18034474
[397948.353656]
[397948.353656] cfs_rq[3]:/autogroup-6
[397948.353751]   .exec_clock                    : 484988.006368
[397948.353802]   .MIN_vruntime                  : 0.000001
[397948.353854]   .min_vruntime                  : 404786.905985
[397948.353905]   .max_vruntime                  : 0.000001
[397948.353956]   .spread                        : 0.000000
[397948.354007]   .spread0                       : -96420830.145576
[397948.354058]   .nr_spread_over                : 245
[397948.354109]   .nr_running                    : 0
[397948.354159]   .load                          : 0
[397948.354209]   .runnable_load_avg             : 0
[397948.354259]   .blocked_load_avg              : 4
[397948.354310]   .tg_load_contrib               : 4
[397948.354360]   .tg_runnable_contrib           : 5
[397948.354409]   .tg_load_avg                   : 4
[397948.354460]   .tg->runnable_avg              : 7
[397948.354510]   .tg->cfs_bandwidth.timer_active: 0
[397948.354560]   .throttled                     : 0
[397948.354610]   .throttle_count                : 0
[397948.354661]   .se->exec_start                : 397948323.578248
[397948.354713]   .se->vruntime                  : 102939999.703983
[397948.354766]   .se->sum_exec_runtime          : 485015.851159
[397948.354817]   .se->statistics.wait_start     : 0.000000
[397948.354868]   .se->statistics.sleep_start    : 0.000000
[397948.354919]   .se->statistics.block_start    : 0.000000
[397948.354970]   .se->statistics.sleep_max      : 0.000000
[397948.355021]   .se->statistics.block_max      : 0.000000
[397948.355072]   .se->statistics.exec_max       : 9.999422
[397948.355123]   .se->statistics.slice_max      : 9.991785
[397948.355174]   .se->statistics.wait_max       : 19.987876
[397948.355225]   .se->statistics.wait_sum       : 7370.667707
[397948.355277]   .se->statistics.wait_count     : 3844653
[397948.355327]   .se->load.weight               : 2
[397948.355378]   .se->avg.runnable_avg_sum      : 266
[397948.355428]   .se->avg.runnable_avg_period   : 47445
[397948.355479]   .se->avg.load_avg_contrib      : 5
[397948.355529]   .se->avg.decay_count           : 379513096
[397948.355581]
[397948.355581] cfs_rq[3]:/
[397948.355673]   .exec_clock                    : 105213301.964322
[397948.355725]   .MIN_vruntime                  : 0.000001
[397948.355776]   .min_vruntime                  : 102940010.804921
[397948.355828]   .max_vruntime                  : 0.000001
[397948.355879]   .spread                        : 0.000000
[397948.355930]   .spread0                       : 6114393.753360
[397948.355982]   .nr_spread_over                : 326
[397948.356033]   .nr_running                    : 0
[397948.356083]   .load                          : 0
[397948.356133]   .runnable_load_avg             : 0
[397948.356183]   .blocked_load_avg              : 0
[397948.356234]   .tg_load_contrib               : 0
[397948.356284]   .tg_runnable_contrib           : 6
[397948.356335]   .tg_load_avg                   : 1319
[397948.356385]   .tg->runnable_avg              : 1291
[397948.356436]   .tg->cfs_bandwidth.timer_active: 0
[397948.356486]   .throttled                     : 0
[397948.356537]   .throttle_count                : 0
[397948.356587]   .avg->runnable_avg_sum         : 307
[397948.356637]   .avg->runnable_avg_period      : 46706
[397948.356688]
[397948.356688] rt_rq[3]:/
[397948.356781]   .rt_nr_running                 : 0
[397948.356831]   .rt_throttled                  : 0
[397948.356882]   .rt_time                       : 0.000000
[397948.356933]   .rt_runtime                    : 950.000000
[397948.356984]
[397948.356984] dl_rq[3]:
[397948.357076]   .dl_nr_running                 : 0
[397948.357127]
[397948.357127] runnable tasks:
[397948.357127]             task   PID         tree-key  switches  prio  
    exec-runtime         sum-exec        sum-sleep
[397948.357127] 
----------------------------------------------------------------------------------------------------------
[397948.357422]         kthreadd     2 102939126.157704      1315   120 
102939126.157704        38.637725 397940297.121677 /
[397948.357527]       watchdog/3    21        -5.984614     99544     0  
       -5.984614     18147.844668        58.998123 /
[397948.357632]      migration/3    22         0.000000     32291     0  
        0.000000     32254.090833         0.000847 /
[397948.357737]      ksoftirqd/3    23 102939780.572027     62304   120 
102939780.572027      1739.164572 397906895.811056 /
[397948.357841]      kworker/3:0    24  33082955.717338        56   120  
33082955.717338         0.596786 176596062.108639 /
[397948.357946]     kworker/3:0H    25      1600.396479         6   100  
     1600.396479         0.027776     33504.961049 /
[397948.358051]          khelper    46        10.955864         2   100  
       10.955864         0.005052         0.001801 /
[397948.358156]        writeback   205       311.132001         2   100  
      311.132001         0.003788         0.001292 /
[397948.358261]           crypto   208       323.135116         2   100  
      323.135116         0.003395         0.001548 /
[397948.358365]           bioset   209       335.138067         2   100  
      335.138067         0.003268         0.001389 /
[397948.358470]          kblockd   210       347.141115         2   100  
      347.141115         0.003369         0.001350 /
[397948.358574]               md   361       359.144676         2   100  
      359.144676         0.003908         0.001863 /
[397948.358679]    fsnotify_mark   576   1285553.304174        56   120  
  1285553.304174         1.356477  77382968.604008 /
[397948.358784]           bioset   909      1410.076226         2   100  
     1410.076226         0.008348         0.004053 /
[397948.358888]        md2_raid1   920 102939126.258310   5305156   120 
102939126.258310    235266.004521 397697550.934379 /
[397948.358994]     kworker/3:1H  1233 101531219.068494    383437   100 
101531219.068494      4828.918177 393998408.709581 /
[397948.359098]    rs:main Q:Reg  3060    404775.410651      3733   120  
   404775.410651       584.098973 397909253.918002 /autogroup-6
[397948.359205]          vnstatd  3041      1938.625022     80791   120  
     1938.625022      7728.375777 397899296.874075 /autogroup-7
[397948.359311]            mdadm  3144        21.070081       419   120  
       21.070081        64.059078 397466901.661042 /autogroup-8
[397948.359417]            nginx  3232      4286.796414        20   120  
     4286.796414         1.473683 322285293.302498 /autogroup-10
[397948.359524]            nginx  3233      5123.741553    498857   120  
     5123.741553      6408.084381 397902171.919015 /autogroup-10
[397948.359631]            nginx  3234      5123.781505    795481   120  
     5123.781505     10575.489835 397897949.295633 /autogroup-10
[397948.360989]            nginx  3236      5123.792442    453567   120  
     5123.792442      6288.632023 397902307.034538 /autogroup-10
[397948.361094]             ntpd  3378      3177.303496    399798   120  
     3177.303496     11384.417725 397896567.168923 /autogroup-13
[397948.361201]           mysqld  4096    404775.038669   1280416   120  
   404775.038669     56190.329409 397846935.062842 /autogroup-6
[397948.361306]           mysqld  4099    404775.037536   1237509   120  
   404775.037536     53709.818619 397849497.344214 /autogroup-6
[397948.361413]           mysqld  4100    404775.128644   1410134   120  
   404775.128644     17179.021072 397887821.391458 /autogroup-6
[397948.361520]           mysqld  4101    404775.136626   1023118   120  
   404775.136626     11032.276249 397894304.441429 /autogroup-6
[397948.361625]           mysqld  4120    404774.916067     79647   120  
   404774.916067      1272.295740 397899969.656177 /autogroup-6
[397948.361732]           mysqld  4121    404774.944173     79565   120  
   404774.944173      1158.311382 397903499.069766 /autogroup-6
[397948.361838]           mysqld  4123    260297.226173    260619   120  
   260297.226173     48020.432086 156765824.444851 /autogroup-6
[397948.361944]           mysqld  4154    404597.326324   2178558   120  
   404597.326324    373081.589987 395136259.717365 /autogroup-6
[397948.362049]           mysqld  4393    404597.164623   1651064   120  
   404597.164623    338373.325108 395122973.929693 /autogroup-6
[397948.362156]           mysqld  4396    404771.119995   1079825   120  
   404771.119995    305191.529334 397422077.680807 /autogroup-6
[397948.362261]           mysqld 15426    404596.822357    784917   120  
   404596.822357    137130.712105 174453678.251801 /autogroup-6
[397948.362367]           mysqld 15435    404596.986208    228061   120  
   404596.986208    118651.907457 174461622.198681 /autogroup-6
[397948.362473]          dovecot  4220       409.745461      1991   120  
      409.745461       113.183818 397150716.515321 /autogroup-17
[397948.362579]           master  4345       415.904754     20039   120  
      415.904754       991.626364 397859137.298652 /autogroup-19
[397948.362684]            getty  4375         0.839336       103   120  
        0.839336         1.796678       147.205983 /autogroup-24
[397948.362791]             tmux  6274     11456.276708    987382   120  
    11456.276708     43704.262889 342265075.730957 /autogroup-183
[397948.362897]      kworker/3:1 13622 102939999.914193   9663630   120 
102939999.914193   4235651.348825 217069990.063754 /
[397948.363034]    kworker/u16:2 22232 102939126.098258    163122   120 
102939126.098258   1130115.311569   3704906.322727 /
[397948.363138]    kworker/u17:0 22250 100814026.249739         4   100 
100814026.249739         0.109630         5.440267 /
[397948.363243]
[397948.363304] cpu#4, 3411.800 MHz
[397948.363352]   .nr_running                    : 0
[397948.363403]   .load                          : 0
[397948.363452]   .nr_switches                   : 31446353
[397948.363503]   .nr_load_updates               : 3466700
[397948.363554]   .nr_uninterruptible            : 12378
[397948.363605]   .next_balance                  : 4334.753132
[397948.363656]   .curr->pid                     : 0
[397948.363706]   .clock                         : 397948363.266319
[397948.363758]   .cpu_load[0]                   : 0
[397948.363809]   .cpu_load[1]                   : 0
[397948.363858]   .cpu_load[2]                   : 0
[397948.363909]   .cpu_load[3]                   : 0
[397948.363958]   .cpu_load[4]                   : 0
[397948.364009]   .yld_count                     : 30496
[397948.364060]   .sched_count                   : 31745499
[397948.364111]   .sched_goidle                  : 13288472
[397948.364162]   .avg_idle                      : 1000000
[397948.364213]   .max_idle_balance_cost         : 500000
[397948.364264]   .ttwu_count                    : 12079122
[397948.364315]   .ttwu_local                    : 3235209
[397948.364366]
[397948.364366] cfs_rq[4]:/
[397948.364458]   .exec_clock                    : 18189528.762014
[397948.364510]   .MIN_vruntime                  : 0.000001
[397948.364561]   .min_vruntime                  : 22358073.568236
[397948.364613]   .max_vruntime                  : 0.000001
[397948.364664]   .spread                        : 0.000000
[397948.364715]   .spread0                       : -74467543.483325
[397948.364768]   .nr_spread_over                : 290
[397948.364818]   .nr_running                    : 0
[397948.364868]   .load                          : 0
[397948.364919]   .runnable_load_avg             : 0
[397948.364968]   .blocked_load_avg              : 0
[397948.365018]   .tg_load_contrib               : 0
[397948.365069]   .tg_runnable_contrib           : 0
[397948.365119]   .tg_load_avg                   : 1315
[397948.365170]   .tg->runnable_avg              : 1290
[397948.365220]   .tg->cfs_bandwidth.timer_active: 0
[397948.365271]   .throttled                     : 0
[397948.365321]   .throttle_count                : 0
[397948.365371]   .avg->runnable_avg_sum         : 0
[397948.365421]   .avg->runnable_avg_period      : 48326
[397948.365472]
[397948.365472] rt_rq[4]:/
[397948.365564]   .rt_nr_running                 : 0
[397948.365614]   .rt_throttled                  : 0
[397948.365664]   .rt_time                       : 0.000000
[397948.365715]   .rt_runtime                    : 950.000000
[397948.365766]
[397948.365766] dl_rq[4]:
[397948.365858]   .dl_nr_running                 : 0
[397948.365909]
[397948.365909] runnable tasks:
[397948.365909]             task   PID         tree-key  switches  prio  
    exec-runtime         sum-exec        sum-sleep
[397948.365909] 
----------------------------------------------------------------------------------------------------------
[397948.366203]       watchdog/4    26        -8.974112     99544     0  
       -8.974112      1925.403645        45.684752 /
[397948.366307]      migration/4    27         0.000000     32687     0  
        0.000000     19510.524185         0.000834 /
[397948.366412]      ksoftirqd/4    28  22357044.685798     13474   120  
22357044.685798       152.288196 397896961.880208 /
[397948.366516]      kworker/4:0    29   1190565.109457        20   120  
  1190565.109457         0.048792  84471152.489397 /
[397948.366621]     kworker/4:0H    30      1300.883814         6   100  
     1300.883814         0.042505     33602.235307 /
[397948.366726]    irq/28-mei_me   673         0.000000        14    49  
        0.000000         0.320285         0.000000 /
[397948.366831]     kworker/4:1H  1287  22130377.005493     40420   100  
22130377.005493       571.072101 394074352.751479 /
[397948.366937]           mysqld  4148    104218.340695         6   120  
   104218.340695         0.235452 160281236.866385 /autogroup-6
[397948.367043]            anvil  4282       508.617446       540   120  
      508.617446        25.108427 395113557.223988 /autogroup-17
[397948.367148]            getty  4371         3.299146       103   120  
        3.299146         4.682491       144.819918 /autogroup-20
[397948.367255]      kworker/4:2  9937  22358061.586618   2184600   120  
22358061.586618    951220.510870 312483748.018300 /
[397948.367360]    kworker/u17:6 21267  21365561.424208        10   100  
21365561.424208         2.594598  27216695.208805 /
[397948.367465]    kworker/u16:1 22824  22358073.568236         4   120  
22358073.568236         0.080475      4901.819753 /
[397948.367569]
[397948.367615] cpu#5, 3411.800 MHz
[397948.367664]   .nr_running                    : 0
[397948.367714]   .load                          : 0
[397948.367764]   .nr_switches                   : 29264631
[397948.367816]   .nr_load_updates               : 3974073
[397948.367867]   .nr_uninterruptible            : 5752
[397948.367917]   .next_balance                  : 4334.753132
[397948.367969]   .curr->pid                     : 0
[397948.368020]   .clock                         : 397948363.266965
[397948.368072]   .cpu_load[0]                   : 0
[397948.368122]   .cpu_load[1]                   : 0
[397948.368172]   .cpu_load[2]                   : 0
[397948.368222]   .cpu_load[3]                   : 0
[397948.368272]   .cpu_load[4]                   : 0
[397948.368323]   .yld_count                     : 23948
[397948.368373]   .sched_count                   : 29418990
[397948.368424]   .sched_goidle                  : 12116624
[397948.368475]   .avg_idle                      : 1000000
[397948.368526]   .max_idle_balance_cost         : 500000
[397948.368577]   .ttwu_count                    : 14464828
[397948.368628]   .ttwu_local                    : 3286532
[397948.368679]
[397948.368679] cfs_rq[5]:/
[397948.368771]   .exec_clock                    : 21096887.422727
[397948.368823]   .MIN_vruntime                  : 0.000001
[397948.368873]   .min_vruntime                  : 24070921.092851
[397948.368926]   .max_vruntime                  : 0.000001
[397948.368977]   .spread                        : 0.000000
[397948.369028]   .spread0                       : -72754695.958710
[397948.369080]   .nr_spread_over                : 219
[397948.369131]   .nr_running                    : 0
[397948.369181]   .load                          : 0
[397948.369232]   .runnable_load_avg             : 0
[397948.369282]   .blocked_load_avg              : 0
[397948.369332]   .tg_load_contrib               : 0
[397948.369382]   .tg_runnable_contrib           : 0
[397948.369432]   .tg_load_avg                   : 1315
[397948.369483]   .tg->runnable_avg              : 1290
[397948.369534]   .tg->cfs_bandwidth.timer_active: 0
[397948.369584]   .throttled                     : 0
[397948.369634]   .throttle_count                : 0
[397948.369685]   .avg->runnable_avg_sum         : 0
[397948.369735]   .avg->runnable_avg_period      : 47521
[397948.369786]
[397948.369786] rt_rq[5]:/
[397948.369878]   .rt_nr_running                 : 0
[397948.369928]   .rt_throttled                  : 0
[397948.369979]   .rt_time                       : 0.000000
[397948.370030]   .rt_runtime                    : 950.000000
[397948.370081]
[397948.370081] dl_rq[5]:
[397948.371423]   .dl_nr_running                 : 0
[397948.371474]
[397948.371474] runnable tasks:
[397948.371474]             task   PID         tree-key  switches  prio  
    exec-runtime         sum-exec        sum-sleep
[397948.371474] 
----------------------------------------------------------------------------------------------------------
[397948.371769]       watchdog/5    31        -8.977217     99543     0  
       -8.977217      2363.156345        32.364766 /
[397948.371873]      migration/5    32         0.000000     25712     0  
        0.000000     15366.174996         0.000845 /
[397948.371977]      ksoftirqd/5    33  24070879.321423     15403   120  
24070879.321423       155.107046 397934935.415490 /
[397948.372082]      kworker/5:0    34       239.267711        17   120  
      239.267711         0.034507       632.432629 /
[397948.372186]     kworker/5:0H    35       632.111736         6   100  
      632.111736         0.059549      2815.362585 /
[397948.372291]      kworker/5:1   611  24070909.178313   2597656   120  
24070909.178313   1203696.900841 396713444.701918 /
[397948.372396]       scsi_tmf_2   860       534.239404         2   100  
      534.239404         0.008896         0.005460 /
[397948.372500]       scsi_tmf_5   866       558.255161         2   100  
      558.255161         0.008179         0.004413 /
[397948.372605]     kworker/5:1H  1061  23860072.399440     45682   100  
23860072.399440       510.190883 394083026.021665 /
[397948.372711]
[397948.372757] cpu#6, 3411.800 MHz
[397948.372805]   .nr_running                    : 0
[397948.372855]   .load                          : 0
[397948.372905]   .nr_switches                   : 30595250
[397948.372972]   .nr_load_updates               : 4632598
[397948.373039]   .nr_uninterruptible            : 6640
[397948.373089]   .next_balance                  : 4334.753132
[397948.373140]   .curr->pid                     : 0
[397948.373191]   .clock                         : 397948363.267655
[397948.373243]   .cpu_load[0]                   : 0
[397948.373292]   .cpu_load[1]                   : 0
[397948.373342]   .cpu_load[2]                   : 0
[397948.373393]   .cpu_load[3]                   : 0
[397948.373443]   .cpu_load[4]                   : 0
[397948.373494]   .yld_count                     : 20767
[397948.373545]   .sched_count                   : 30739543
[397948.373596]   .sched_goidle                  : 12161262
[397948.373647]   .avg_idle                      : 1000000
[397948.373698]   .max_idle_balance_cost         : 500000
[397948.373749]   .ttwu_count                    : 17363747
[397948.373800]   .ttwu_local                    : 4024881
[397948.373852]
[397948.373852] cfs_rq[6]:/
[397948.373944]   .exec_clock                    : 27066813.057347
[397948.373996]   .MIN_vruntime                  : 0.000001
[397948.374046]   .min_vruntime                  : 29909549.644335
[397948.374099]   .max_vruntime                  : 0.000001
[397948.374150]   .spread                        : 0.000000
[397948.374202]   .spread0                       : -66916067.407226
[397948.374253]   .nr_spread_over                : 407
[397948.374304]   .nr_running                    : 0
[397948.374354]   .load                          : 0
[397948.374404]   .runnable_load_avg             : 0
[397948.374454]   .blocked_load_avg              : 0
[397948.374504]   .tg_load_contrib               : 0
[397948.374554]   .tg_runnable_contrib           : 0
[397948.374605]   .tg_load_avg                   : 1313
[397948.374656]   .tg->runnable_avg              : 1289
[397948.374706]   .tg->cfs_bandwidth.timer_active: 0
[397948.374757]   .throttled                     : 0
[397948.374807]   .throttle_count                : 0
[397948.374857]   .avg->runnable_avg_sum         : 0
[397948.374907]   .avg->runnable_avg_period      : 47423
[397948.374958]
[397948.374958] rt_rq[6]:/
[397948.375050]   .rt_nr_running                 : 0
[397948.375099]   .rt_throttled                  : 0
[397948.375149]   .rt_time                       : 0.000000
[397948.375200]   .rt_runtime                    : 950.000000
[397948.375258]
[397948.375258] dl_rq[6]:
[397948.375359]   .dl_nr_running                 : 0
[397948.375410]
[397948.375410] runnable tasks:
[397948.375410]             task   PID         tree-key  switches  prio  
    exec-runtime         sum-exec        sum-sleep
[397948.375410] 
----------------------------------------------------------------------------------------------------------
[397948.375704]       watchdog/6    36        -8.979370     99543     0  
       -8.979370      3873.032642        19.046238 /
[397948.375809]      migration/6    37         0.000000     20867     0  
        0.000000     12947.385988         0.000826 /
[397948.375914]      ksoftirqd/6    38  29909537.744505     17979   120  
29909537.744505       324.324395 397932684.911680 /
[397948.376018]      kworker/6:0    39      2652.785847        15   120  
     2652.785847         0.034665       619.334482 /
[397948.376123]     kworker/6:0H    40      3222.777674         6   100  
     3222.777674         0.042578     34131.960421 /
[397948.376228]      kworker/6:1   613  29909537.712176   3191586   120  
29909537.712176   1584243.664849 396333417.817101 /
[397948.376333]           bioset   898      2957.541580         2   100  
     2957.541580         0.008747         0.006389 /
[397948.376438]     kworker/6:1H  1463  29688535.473225     50544   100  
29688535.473225       612.882638 394061042.264128 /
[397948.376542]        kjournald  2216      3410.060184         2   120  
     3410.060184         0.412389         0.043474 /
[397948.376646]           bioset  2222      3422.126220         2   100  
     3422.126220         0.088585         0.040787 /
[397948.376750]         rsyslogd  3061    163480.988716      3499   120  
   163480.988716       146.745864 397862580.052157 /autogroup-6
[397948.376856]           mysqld  4122    163443.475621    644351   120  
   163443.475621    179019.020540 393898317.914127 /autogroup-6
[397948.376962]            udevd  6137       246.142493        30   120  
      246.142493         6.670412  22056460.101598 /autogroup-4
[397948.377068]            udevd  6138       242.586598        15   120  
      242.586598         3.149366     49194.013301 /autogroup-4
[397948.377174]     btrfs-worker  6246   1051403.790298         2   100  
  1051403.790298         0.011158         0.004647 /
[397948.377278]  btrfs-endio-met  6255   1051427.810805         2   100  
  1051427.810805         0.017787         0.002771 /
[397948.377383]  btrfs-endio-rai  6257   1051439.820346         2   100  
  1051439.820346         0.010462         0.002911 /
[397948.377487]  btrfs-endio-rep  6258   1051451.828334         2   100  
  1051451.828334         0.008652         0.003137 /
[397948.377591]  btrfs-endio-wri  6260   1051463.836060         2   100  
  1051463.836060         0.008410         0.003085 /
[397948.377695]  btrfs-readahead  6263   1051475.843882         2   100  
  1051475.843882         0.008543         0.003153 /
[397948.377799]  btrfs-extent-re  6265   1051487.851432         2   100  
  1051487.851432         0.008154         0.003265 /
[397948.377904]
[397948.377950] cpu#7, 3411.800 MHz
[397948.377998]   .nr_running                    : 0
[397948.378048]   .load                          : 0
[397948.378098]   .nr_switches                   : 28501260
[397948.378150]   .nr_load_updates               : 3359354
[397948.378200]   .nr_uninterruptible            : 4860
[397948.378251]   .next_balance                  : 4334.753132
[397948.378302]   .curr->pid                     : 0
[397948.378351]   .clock                         : 397948363.268406
[397948.378403]   .cpu_load[0]                   : 0
[397948.378453]   .cpu_load[1]                   : 0
[397948.378504]   .cpu_load[2]                   : 0
[397948.378554]   .cpu_load[3]                   : 0
[397948.378604]   .cpu_load[4]                   : 0
[397948.378654]   .yld_count                     : 22779
[397948.378705]   .sched_count                   : 28647495
[397948.378757]   .sched_goidle                  : 12446869
[397948.378808]   .avg_idle                      : 1000000
[397948.378859]   .max_idle_balance_cost         : 500000
[397948.378910]   .ttwu_count                    : 18391339
[397948.378961]   .ttwu_local                    : 2717640
[397948.379012]
[397948.379012] cfs_rq[7]:/
[397948.379112]   .exec_clock                    : 13565979.621209
[397948.379173]   .MIN_vruntime                  : 0.000001
[397948.379225]   .min_vruntime                  : 17262906.632013
[397948.379276]   .max_vruntime                  : 0.000001
[397948.379327]   .spread                        : 0.000000
[397948.379378]   .spread0                       : -79562710.419548
[397948.379430]   .nr_spread_over                : 255
[397948.379480]   .nr_running                    : 0
[397948.379530]   .load                          : 0
[397948.379581]   .runnable_load_avg             : 0
[397948.379631]   .blocked_load_avg              : 0
[397948.379682]   .tg_load_contrib               : 0
[397948.379732]   .tg_runnable_contrib           : 0
[397948.379783]   .tg_load_avg                   : 1313
[397948.379833]   .tg->runnable_avg              : 1288
[397948.379884]   .tg->cfs_bandwidth.timer_active: 0
[397948.379933]   .throttled                     : 0
[397948.379983]   .throttle_count                : 0
[397948.380034]   .avg->runnable_avg_sum         : 5
[397948.380084]   .avg->runnable_avg_period      : 47099
[397948.380135]
[397948.380135] rt_rq[7]:/
[397948.380227]   .rt_nr_running                 : 0
[397948.380277]   .rt_throttled                  : 0
[397948.380327]   .rt_time                       : 0.000000
[397948.380379]   .rt_runtime                    : 950.000000
[397948.380430]
[397948.380430] dl_rq[7]:
[397948.380522]   .dl_nr_running                 : 0
[397948.380572]
[397948.380572] runnable tasks:
[397948.380572]             task   PID         tree-key  switches  prio  
    exec-runtime         sum-exec        sum-sleep
[397948.380572] 
----------------------------------------------------------------------------------------------------------
[397948.382132]       watchdog/7    41        -8.982956     99544     0  
       -8.982956      1585.113591         5.725643 /
[397948.382236]      migration/7    42         0.000000     21048     0  
        0.000000     12757.115970         0.000816 /
[397948.382341]      ksoftirqd/7    43  17254754.851360     11826   120  
17254754.851360       146.221401 397720187.660046 /
[397948.382446]      kworker/7:0    44       479.385599        15   120  
      479.385599         0.031395       606.240559 /
[397948.382550]     kworker/7:0H    45      1817.499047         6   100  
     1817.499047         0.047704     42727.098840 /
[397948.382655]            netns    47        10.955152         2   100  
       10.955152         0.004076         0.001586 /
[397948.382759]             perf    50        22.958573         2   100  
       22.958573         0.003794         0.001394 /
[397948.382863]      kworker/7:1   615  17262894.767216   1847690   120  
17262894.767216    746881.932822 397173736.970567 /
[397948.382992]        md0_raid1   899  17253995.722314     32645   120  
17253995.722314       997.731148 397503108.422867 /
[397948.383105]           bioset   919      1132.474026         2   100  
     1132.474026         0.034373         0.004008 /
[397948.383210]  ext4-rsv-conver   936      1156.590460         2   100  
     1156.590460         0.061309         0.026946 /
[397948.383315]            udevd  1124       405.566270       290   120  
      405.566270        53.586152  77350627.244245 /autogroup-4
[397948.383421]    ipv6_addrconf  3209      1679.353321         2   100  
     1679.353321         0.006452         0.002581 /
[397948.383527]     kworker/7:1H  4116  16792468.563089     52255   100  
16792468.563089       541.499314 393833366.670026 /
[397948.383632]              log  4283       574.060901       688   120  
      574.060901        34.058909 395113548.562211 /autogroup-17
[397948.383739]            getty  4376         0.726364       103   120  
        0.726364         1.882302       146.996217 /autogroup-25
[397948.383845]      btrfs-fixup  6253   1049025.643969         2   100  
  1049025.643969         0.011376         0.003865 /
[397948.383950]      btrfs-endio  6254   1049037.659031         2   100  
  1049037.659031         0.015618         0.001516 /
[397948.384056]


Tomasz Chmielewski
http://www.sslrack.com


^ permalink raw reply	[flat|nested] 3+ messages in thread
end of thread, other threads:[~2015-02-16 10:51 UTC | newest]

Thread overview: 3+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2015-02-16  2:32 3.19 - "unable" to replace a failed drive - 100% CPU usage in kworker and btrfs-transacti Tomasz Chmielewski
2015-02-16 10:40 ` Liu Bo
2015-02-16 10:51   ` Tomasz Chmielewski

This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.