Netdev List
 help / color / mirror / Atom feed
* [loopback] b17c706987f: +175.7% netperf.Throughput_Mbps
From: Fengguang Wu @ 2014-10-01  7:54 UTC (permalink / raw)
  To: Daniel Borkmann; +Cc: Dave Hansen, LKML, lkp, netdev

[-- Attachment #1: Type: text/plain, Size: 116268 bytes --]

Hi Daniel,

FYI, we noticed nice performance improvement in commit

b17c706987fa6f28bdc1771c8266e7a69e22adcb ("loopback: sctp: add NETIF_F_SCTP_CSUM to device features")

test case: lkp-nex04/netperf/300s-200%-10K-SCTP_STREAM_MANY

72f8e06f3ea022d  b17c706987fa6f28bdc1771c8 
---------------  ------------------------- 
     %stddev        %change               %stddev
            \          |                 /
       664 ± 0%    +175.7%       1832 ± 0%  TOTAL netperf.Throughput_Mbps
         0 ± 0%      +Inf%          1 ± 0%  TOTAL sched_debug.cfs_rq[63]:/.nr_running
    947669 ± 2%    +681.8%    7408572 ± 1%  TOTAL sched_debug.cfs_rq[63]:/.min_vruntime
     19701 ± 3%   +2814.0%     574098 ± 1%  TOTAL sched_debug.cpu#63.ttwu_local
     41754 ± 1%     -99.5%        200 ±43%  TOTAL softirqs.HRTIMER
         5 ±20%    +400.0%         29 ± 2%  TOTAL sched_debug.cpu#63.cpu_load[4]
      2.59 ± 1%    -100.0%       0.00 ± 0%  TOTAL perf-profile.cpu-cycles.intel_idle.cpuidle_enter_state.cpuidle_idle_call.arch_cpu_idle.cpu_startup_entry
         0 ± 0%      +Inf%          2 ± 0%  TOTAL sched_debug.cpu#63.nr_running
        72 ±48%     -95.8%          3 ±42%  TOTAL sched_debug.cfs_rq[62]:/.blocked_load_avg
         0 ± 0%      +Inf%          1 ± 0%  TOTAL sched_debug.cfs_rq[62]:/.nr_running
      0.24 ± 7%   +2565.6%       6.50 ± 4%  TOTAL perf-profile.cpu-cycles.sctp_transport_timeout.sctp_transport_reset_timers.sctp_outq_flush.sctp_outq_uncork.sctp_cmd_interpreter
      0.22 ± 3%   +1442.3%       3.42 ± 2%  TOTAL perf-profile.cpu-cycles._raw_spin_lock_irqsave.get_page_from_freelist.__alloc_pages_nodemask.kmalloc_large_node.__kmalloc_node_track_caller
      0.27 ± 6%   +1100.0%       3.29 ± 2%  TOTAL perf-profile.cpu-cycles._raw_spin_lock.free_one_page.__free_pages_ok.__free_pages.__free_memcg_kmem_pages
      0.04 ±10%   +8463.2%       3.25 ± 2%  TOTAL perf-profile.cpu-cycles.lock_timer_base.isra.35.mod_timer.sctp_transport_reset_timers.sctp_outq_flush.sctp_outq_uncork
      0.04 ±13%   +6227.8%       2.28 ± 3%  TOTAL perf-profile.cpu-cycles.memcpy.sctp_packet_transmit_chunk.sctp_outq_flush.sctp_outq_uncork.sctp_cmd_interpreter
      0.00           +Inf%       1.97 ± 8%  TOTAL perf-profile.cpu-cycles._raw_spin_lock_irqsave.mod_timer.sctp_transport_reset_timers.sctp_outq_flush.sctp_outq_uncork
        11 ±44%   +9151.8%       1036 ±31%  TOTAL sched_debug.cfs_rq[62]:/.nr_spread_over
      1.15 ± 7%     -94.4%       0.06 ± 7%  TOTAL perf-profile.cpu-cycles._raw_spin_lock_bh.lock_sock_nested.sctp_recvmsg.sock_common_recvmsg.sock_recvmsg
  33688875 ± 2%     -89.6%    3504734 ±18%  TOTAL cpuidle.C1-NHM.time
    281217 ± 2%     -90.9%      25698 ±41%  TOTAL cpuidle.C1-NHM.usage
  45558795 ± 0%     -99.9%      27898 ±39%  TOTAL cpuidle.C3-NHM.usage
     39.61 ± 0%     -99.7%       0.11 ±30%  TOTAL turbostat.%c1
      5.60 ± 0%     -94.0%       0.34 ± 1%  TOTAL turbostat.%c3
    876992 ± 3%    +736.3%    7333987 ± 1%  TOTAL sched_debug.cfs_rq[62]:/.min_vruntime
     19686 ± 2%   +2810.9%     573039 ± 1%  TOTAL sched_debug.cpu#62.ttwu_local
         5 ±25%    +403.4%         29 ± 3%  TOTAL sched_debug.cpu#62.cpu_load[4]
         0 ± 0%      +Inf%          2 ± 0%  TOTAL sched_debug.cpu#62.nr_running
         0 ± 0%      +Inf%          1 ± 0%  TOTAL sched_debug.cfs_rq[61]:/.nr_running
         8 ±22%  +13481.4%       1168 ±36%  TOTAL sched_debug.cfs_rq[61]:/.nr_spread_over
    899343 ± 6%    +715.7%    7335525 ± 2%  TOTAL sched_debug.cfs_rq[61]:/.min_vruntime
     19673 ± 4%   +2790.5%     568655 ± 2%  TOTAL sched_debug.cpu#61.ttwu_local
    989853 ± 2%     -99.2%       7480 ±48%  TOTAL sched_debug.cpu#61.sched_goidle
         4 ±40%    +581.0%         28 ± 3%  TOTAL sched_debug.cpu#61.cpu_load[4]
         0 ± 0%      +Inf%          2 ± 0%  TOTAL sched_debug.cpu#61.nr_running
         0 ± 0%      +Inf%          1 ± 0%  TOTAL sched_debug.cfs_rq[60]:/.nr_running
    893662 ± 5%    +724.8%    7370847 ± 0%  TOTAL sched_debug.cfs_rq[60]:/.min_vruntime
     19607 ± 3%   +2827.4%     573986 ± 1%  TOTAL sched_debug.cpu#60.ttwu_local
         0 ± 0%      +Inf%          2 ± 0%  TOTAL sched_debug.cpu#60.nr_running
         0 ± 0%      +Inf%          1 ± 0%  TOTAL sched_debug.cfs_rq[59]:/.nr_running
     86308 ±18%   +7142.5%    6250836 ± 8%  TOTAL sched_debug.cfs_rq[59]:/.max_vruntime
    899110 ± 4%    +717.5%    7350607 ± 0%  TOTAL sched_debug.cfs_rq[59]:/.min_vruntime
         0 ± 0%      +Inf%          2 ± 0%  TOTAL sched_debug.cpu#0.nr_running
     86308 ±18%   +7142.5%    6250836 ± 8%  TOTAL sched_debug.cfs_rq[59]:/.MIN_vruntime
     20221 ± 4%   +2739.2%     574120 ± 1%  TOTAL sched_debug.cpu#59.ttwu_local
         0 ± 0%      +Inf%          2 ± 0%  TOTAL sched_debug.cpu#59.nr_running
         0 ± 0%      +Inf%          1 ± 0%  TOTAL sched_debug.cfs_rq[58]:/.nr_running
    370050 ± 3%     -98.8%       4378 ±24%  TOTAL sched_debug.cpu#0.sched_goidle
     32479 ± 4%   +1698.6%     584171 ± 1%  TOTAL sched_debug.cpu#0.ttwu_local
    947194 ± 4%    +676.5%    7354983 ± 1%  TOTAL sched_debug.cfs_rq[58]:/.min_vruntime
        19 ±12%  +26074.0%       5025 ±32%  TOTAL sched_debug.cfs_rq[0]:/.nr_spread_over
         0 ± 0%      +Inf%          1 ± 0%  TOTAL sched_debug.cfs_rq[0]:/.nr_running
     20721 ± 4%   +2663.3%     572601 ± 1%  TOTAL sched_debug.cpu#58.ttwu_local
         0 ± 0%      +Inf%          2 ± 0%  TOTAL sched_debug.cpu#58.nr_running
         0 ± 0%      +Inf%          1 ±33%  TOTAL sched_debug.cfs_rq[57]:/.nr_running
         0 ± 0%      +Inf%          2 ± 0%  TOTAL sched_debug.cpu#1.nr_running
    964313 ± 4%    +662.7%    7355115 ± 0%  TOTAL sched_debug.cfs_rq[57]:/.min_vruntime
     21834 ± 6%   +2531.4%     574540 ± 1%  TOTAL sched_debug.cpu#57.ttwu_local
         0 ± 0%      +Inf%          2 ± 0%  TOTAL sched_debug.cpu#57.nr_running
    354246 ± 1%     -98.7%       4745 ±38%  TOTAL sched_debug.cpu#1.sched_goidle
         0 ± 0%      +Inf%          1 ± 0%  TOTAL sched_debug.cfs_rq[56]:/.nr_running
     28829 ± 5%   +1914.4%     580717 ± 0%  TOTAL sched_debug.cpu#1.ttwu_local
         0 ± 0%      +Inf%          1 ± 0%  TOTAL sched_debug.cfs_rq[1]:/.nr_running
    960131 ± 3%    +667.5%    7368750 ± 1%  TOTAL sched_debug.cfs_rq[56]:/.min_vruntime
     21418 ± 2%   +2584.0%     574857 ± 0%  TOTAL sched_debug.cpu#56.ttwu_local
         0 ± 0%      +Inf%          2 ± 0%  TOTAL sched_debug.cpu#2.nr_running
         4 ±17%    +534.8%         29 ± 3%  TOTAL sched_debug.cpu#56.cpu_load[4]
         5 ±18%    +433.3%         28 ± 2%  TOTAL sched_debug.cpu#56.cpu_load[3]
         0 ± 0%      +Inf%          2 ± 0%  TOTAL sched_debug.cpu#56.nr_running
         0 ± 0%      +Inf%          1 ± 0%  TOTAL sched_debug.cfs_rq[55]:/.nr_running
     71260 ±42%   +8857.8%    6383312 ± 6%  TOTAL sched_debug.cfs_rq[55]:/.max_vruntime
    935340 ± 4%    +686.3%    7354731 ± 1%  TOTAL sched_debug.cfs_rq[55]:/.min_vruntime
    363420 ± 1%     -98.9%       3993 ±40%  TOTAL sched_debug.cpu#2.sched_goidle
     71260 ±42%   +8857.8%    6383312 ± 6%  TOTAL sched_debug.cfs_rq[55]:/.MIN_vruntime
     19268 ± 3%   +2867.0%     571688 ± 2%  TOTAL sched_debug.cpu#55.ttwu_local
     29222 ± 4%   +1889.9%     581489 ± 0%  TOTAL sched_debug.cpu#2.ttwu_local
         5 ±33%    +457.7%         29 ± 2%  TOTAL sched_debug.cpu#55.cpu_load[4]
        21 ±10%   +5539.8%       1218 ±19%  TOTAL sched_debug.cfs_rq[2]:/.nr_spread_over
         0 ± 0%      +Inf%          1 ± 0%  TOTAL sched_debug.cfs_rq[2]:/.nr_running
         0 ± 0%      +Inf%          2 ± 0%  TOTAL sched_debug.cpu#55.nr_running
         0 ± 0%      +Inf%          1 ± 0%  TOTAL sched_debug.cfs_rq[54]:/.nr_running
         0 ± 0%      +Inf%          2 ± 0%  TOTAL sched_debug.cpu#3.nr_running
     45064 ±18%  +14562.1%    6607460 ± 5%  TOTAL sched_debug.cfs_rq[54]:/.max_vruntime
    934539 ± 6%    +685.1%    7337176 ± 1%  TOTAL sched_debug.cfs_rq[54]:/.min_vruntime
     45064 ±18%  +14562.1%    6607460 ± 5%  TOTAL sched_debug.cfs_rq[54]:/.MIN_vruntime
     19262 ± 6%   +2872.1%     572494 ± 1%  TOTAL sched_debug.cpu#54.ttwu_local
         0 ± 0%      +Inf%          2 ± 0%  TOTAL sched_debug.cpu#54.nr_running
    356613 ± 1%     -99.2%       2934 ±29%  TOTAL sched_debug.cpu#3.sched_goidle
         0 ± 0%      +Inf%          1 ± 0%  TOTAL sched_debug.cfs_rq[53]:/.nr_running
     27577 ± 5%   +2003.9%     580222 ± 1%  TOTAL sched_debug.cpu#3.ttwu_local
         0 ± 0%      +Inf%          1 ± 0%  TOTAL sched_debug.cfs_rq[3]:/.nr_running
    933523 ± 4%    +686.2%    7339367 ± 1%  TOTAL sched_debug.cfs_rq[53]:/.min_vruntime
     19832 ± 3%   +2803.2%     575758 ± 1%  TOTAL sched_debug.cpu#53.ttwu_local
         0 ± 0%      +Inf%          2 ± 0%  TOTAL sched_debug.cpu#4.nr_running
         5 ±46%    +410.7%         28 ± 4%  TOTAL sched_debug.cpu#53.cpu_load[3]
         6 ±47%    +354.8%         28 ± 5%  TOTAL sched_debug.cpu#53.cpu_load[0]
         0 ± 0%      +Inf%          2 ± 0%  TOTAL sched_debug.cpu#53.nr_running
         0 ± 0%      +Inf%          1 ± 0%  TOTAL sched_debug.cfs_rq[52]:/.nr_running
    923091 ± 3%    +698.0%    7366148 ± 2%  TOTAL sched_debug.cfs_rq[52]:/.min_vruntime
    348410 ± 2%     -98.8%       4230 ±39%  TOTAL sched_debug.cpu#4.sched_goidle
     19701 ± 3%   +2820.0%     575293 ± 1%  TOTAL sched_debug.cpu#52.ttwu_local
     26645 ± 4%   +2076.8%     580019 ± 1%  TOTAL sched_debug.cpu#4.ttwu_local
    178678 ±25%   +3627.5%    6660167 ± 3%  TOTAL sched_debug.cfs_rq[4]:/.MIN_vruntime
         0 ± 0%      +Inf%          2 ± 0%  TOTAL sched_debug.cpu#52.nr_running
    178678 ±25%   +3627.5%    6660167 ± 3%  TOTAL sched_debug.cfs_rq[4]:/.max_vruntime
         0 ± 0%      +Inf%          1 ± 0%  TOTAL sched_debug.cfs_rq[4]:/.nr_running
         0 ± 0%      +Inf%          1 ± 0%  TOTAL sched_debug.cfs_rq[51]:/.nr_running
         0 ± 0%      +Inf%          2 ± 0%  TOTAL sched_debug.cpu#5.nr_running
    929685 ± 4%    +688.0%    7325719 ± 1%  TOTAL sched_debug.cfs_rq[51]:/.min_vruntime
     19761 ± 4%   +2813.8%     575800 ± 1%  TOTAL sched_debug.cpu#51.ttwu_local
         5 ±37%    +461.5%         29 ± 2%  TOTAL sched_debug.cpu#51.cpu_load[4]
         0 ± 0%      +Inf%          2 ± 0%  TOTAL sched_debug.cpu#51.nr_running
         0 ± 0%      +Inf%          1 ± 0%  TOTAL sched_debug.cfs_rq[50]:/.nr_running
     26189 ± 3%   +2110.9%     579032 ± 0%  TOTAL sched_debug.cpu#5.ttwu_local
    963932 ± 3%    +657.4%    7301196 ± 1%  TOTAL sched_debug.cfs_rq[50]:/.min_vruntime
     23557 ± 2%   +2326.4%     571608 ± 2%  TOTAL sched_debug.cpu#23.ttwu_local
         0 ± 0%      +Inf%          1 ± 0%  TOTAL sched_debug.cfs_rq[5]:/.nr_running
     20098 ± 5%   +2752.9%     573394 ± 1%  TOTAL sched_debug.cpu#50.ttwu_local
         6 ±34%    +380.0%         28 ± 1%  TOTAL sched_debug.cpu#50.cpu_load[4]
         6 ±35%    +343.8%         28 ± 1%  TOTAL sched_debug.cpu#50.cpu_load[3]
         0 ± 0%      +Inf%          2 ± 0%  TOTAL sched_debug.cpu#6.nr_running
         0 ± 0%      +Inf%          2 ± 0%  TOTAL sched_debug.cpu#50.nr_running
         0 ± 0%      +Inf%          1 ± 0%  TOTAL sched_debug.cfs_rq[49]:/.nr_running
        16 ± 8%   +4700.0%        768 ±49%  TOTAL sched_debug.cfs_rq[49]:/.nr_spread_over
     25690 ± 2%   +2152.9%     578774 ± 1%  TOTAL sched_debug.cpu#6.ttwu_local
    988774 ± 1%    +643.1%    7347606 ± 1%  TOTAL sched_debug.cfs_rq[49]:/.min_vruntime
        17 ±16%   +6484.7%       1119 ±36%  TOTAL sched_debug.cfs_rq[6]:/.nr_spread_over
         0 ± 0%      +Inf%          1 ± 0%  TOTAL sched_debug.cfs_rq[6]:/.nr_running
     21655 ± 3%   +2554.9%     574933 ± 1%  TOTAL sched_debug.cpu#49.ttwu_local
    917931 ± 0%     -99.2%       7364 ±47%  TOTAL sched_debug.cpu#49.sched_goidle
         0 ± 0%      +Inf%          2 ± 0%  TOTAL sched_debug.cpu#49.nr_running
         0 ± 0%      +Inf%          2 ± 0%  TOTAL sched_debug.cpu#7.nr_running
         0 ± 0%      +Inf%          1 ± 0%  TOTAL sched_debug.cfs_rq[48]:/.nr_running
    963896 ± 5%    +662.9%    7353616 ± 1%  TOTAL sched_debug.cfs_rq[48]:/.min_vruntime
     20962 ± 5%   +2638.5%     574061 ± 1%  TOTAL sched_debug.cpu#48.ttwu_local
    342563 ± 2%     -99.0%       3297 ±40%  TOTAL sched_debug.cpu#7.sched_goidle
     25834 ± 4%   +2138.5%     578300 ± 0%  TOTAL sched_debug.cpu#7.ttwu_local
         4 ±48%    +500.0%         28 ± 1%  TOTAL sched_debug.cpu#48.cpu_load[3]
         0 ± 0%      +Inf%          1 ± 0%  TOTAL sched_debug.cfs_rq[7]:/.nr_running
         4 ±44%    +513.0%         28 ± 1%  TOTAL sched_debug.cpu#48.cpu_load[0]
         0 ± 0%      +Inf%          2 ± 0%  TOTAL sched_debug.cpu#48.nr_running
         0 ± 0%      +Inf%          2 ± 0%  TOTAL sched_debug.cpu#8.nr_running
         0 ± 0%      +Inf%          1 ± 0%  TOTAL sched_debug.cfs_rq[47]:/.nr_running
   1026500 ± 4%    +606.8%    7255532 ± 1%  TOTAL sched_debug.cfs_rq[47]:/.min_vruntime
     21776 ± 3%   +2530.7%     572873 ± 1%  TOTAL sched_debug.cpu#47.ttwu_local
    940486 ± 1%     -99.3%       6905 ±46%  TOTAL sched_debug.cpu#47.sched_goidle
    375887 ± 2%     -98.8%       4502 ±40%  TOTAL sched_debug.cpu#8.sched_goidle
         0 ± 0%      +Inf%          2 ± 0%  TOTAL sched_debug.cpu#47.nr_running
        46 ±43%     -94.0%          2 ±41%  TOTAL sched_debug.cfs_rq[46]:/.blocked_load_avg
     27667 ± 3%   +1978.0%     574926 ± 1%  TOTAL sched_debug.cpu#8.ttwu_local
         6 ±34%    +370.0%         28 ± 1%  TOTAL sched_debug.cfs_rq[46]:/.runnable_load_avg
         0 ± 0%      +Inf%          1 ± 0%  TOTAL sched_debug.cfs_rq[46]:/.nr_running
         0 ± 0%      +Inf%          1 ± 0%  TOTAL sched_debug.cfs_rq[8]:/.nr_running
    973194 ± 3%    +646.5%    7264433 ± 1%  TOTAL sched_debug.cfs_rq[46]:/.min_vruntime
         0 ± 0%      +Inf%          2 ± 0%  TOTAL sched_debug.cpu#9.nr_running
     22161 ± 2%   +2482.1%     572230 ± 1%  TOTAL sched_debug.cpu#46.ttwu_local
         4 ±15%    +525.0%         30 ± 2%  TOTAL sched_debug.cpu#46.cpu_load[4]
         0 ± 0%      +Inf%          2 ± 0%  TOTAL sched_debug.cpu#46.nr_running
         0 ± 0%      +Inf%          1 ± 0%  TOTAL sched_debug.cfs_rq[45]:/.nr_running
    359971 ± 2%     -99.1%       3085 ±41%  TOTAL sched_debug.cpu#9.sched_goidle
   1018595 ± 4%    +611.1%    7243421 ± 1%  TOTAL sched_debug.cfs_rq[45]:/.min_vruntime
     27131 ± 2%   +2017.8%     574580 ± 1%  TOTAL sched_debug.cpu#9.ttwu_local
    134404 ±31%   +4758.2%    6529609 ± 8%  TOTAL sched_debug.cfs_rq[9]:/.MIN_vruntime
     22294 ± 4%   +2467.1%     572326 ± 1%  TOTAL sched_debug.cpu#45.ttwu_local
    134404 ±31%   +4758.2%    6529609 ± 8%  TOTAL sched_debug.cfs_rq[9]:/.max_vruntime
         0 ± 0%      +Inf%          1 ± 0%  TOTAL sched_debug.cfs_rq[9]:/.nr_running
         5 ±22%    +455.6%         30 ± 2%  TOTAL sched_debug.cpu#45.cpu_load[4]
         0 ± 0%      +Inf%          2 ±18%  TOTAL sched_debug.cpu#45.nr_running
         0 ± 0%      +Inf%          1 ± 0%  TOTAL sched_debug.cfs_rq[44]:/.nr_running
         0 ± 0%      +Inf%          2 ± 0%  TOTAL sched_debug.cpu#10.nr_running
    360383 ± 1%     -98.8%       4421 ±45%  TOTAL sched_debug.cpu#10.sched_goidle
     27338 ± 3%   +2000.0%     574125 ± 1%  TOTAL sched_debug.cpu#10.ttwu_local
   1021430 ± 4%    +609.5%    7247197 ± 1%  TOTAL sched_debug.cfs_rq[44]:/.min_vruntime
    231918 ±24%   +2549.3%    6144271 ± 8%  TOTAL sched_debug.cfs_rq[10]:/.MIN_vruntime
    231918 ±24%   +2549.3%    6144271 ± 8%  TOTAL sched_debug.cfs_rq[10]:/.max_vruntime
         0 ± 0%      +Inf%          1 ± 0%  TOTAL sched_debug.cfs_rq[10]:/.nr_running
     22562 ± 3%   +2442.1%     573549 ± 1%  TOTAL sched_debug.cpu#44.ttwu_local
         5 ±25%    +492.0%         29 ± 1%  TOTAL sched_debug.cpu#44.cpu_load[4]
         0 ± 0%      +Inf%          2 ± 0%  TOTAL sched_debug.cpu#11.nr_running
         0 ± 0%      +Inf%          2 ± 0%  TOTAL sched_debug.cpu#44.nr_running
         0 ± 0%      +Inf%          1 ± 0%  TOTAL sched_debug.cfs_rq[43]:/.nr_running
   1019742 ± 2%    +614.0%    7280699 ± 1%  TOTAL sched_debug.cfs_rq[43]:/.min_vruntime
    366743 ± 3%     -98.8%       4544 ±48%  TOTAL sched_debug.cpu#11.sched_goidle
     22781 ± 3%   +2411.2%     572090 ± 1%  TOTAL sched_debug.cpu#43.ttwu_local
     27283 ± 5%   +1994.9%     571569 ± 1%  TOTAL sched_debug.cpu#11.ttwu_local
         0 ± 0%      +Inf%          2 ± 0%  TOTAL sched_debug.cpu#43.nr_running
         0 ± 0%      +Inf%          1 ± 0%  TOTAL sched_debug.cfs_rq[11]:/.nr_running
         0 ± 0%      +Inf%          1 ± 0%  TOTAL sched_debug.cfs_rq[42]:/.nr_running
         0 ± 0%      +Inf%          2 ± 0%  TOTAL sched_debug.cpu#12.nr_running
   1013625 ± 3%    +610.3%    7199592 ± 1%  TOTAL sched_debug.cfs_rq[42]:/.min_vruntime
     23285 ± 2%   +2357.3%     572199 ± 1%  TOTAL sched_debug.cpu#42.ttwu_local
         0 ± 0%      +Inf%          2 ± 0%  TOTAL sched_debug.cpu#42.nr_running
         0 ± 0%      +Inf%          1 ± 0%  TOTAL sched_debug.cfs_rq[41]:/.nr_running
    363201 ± 5%     -99.2%       3010 ±32%  TOTAL sched_debug.cpu#12.sched_goidle
   1026049 ± 2%    +605.1%    7234511 ± 2%  TOTAL sched_debug.cfs_rq[41]:/.min_vruntime
     26558 ± 1%   +2059.8%     573625 ± 1%  TOTAL sched_debug.cpu#12.ttwu_local
     24712 ± 3%   +2225.8%     574772 ± 1%  TOTAL sched_debug.cpu#41.ttwu_local
         0 ± 0%      +Inf%          1 ± 0%  TOTAL sched_debug.cfs_rq[12]:/.nr_running
    908047 ± 0%     -99.3%       6771 ±46%  TOTAL sched_debug.cpu#41.sched_goidle
         0 ± 0%      +Inf%          2 ± 0%  TOTAL sched_debug.cpu#41.nr_running
         0 ± 0%      +Inf%          1 ± 0%  TOTAL sched_debug.cfs_rq[40]:/.nr_running
         0 ± 0%      +Inf%          2 ± 0%  TOTAL sched_debug.cpu#13.nr_running
   1045087 ± 1%    +592.6%    7238603 ± 1%  TOTAL sched_debug.cfs_rq[40]:/.min_vruntime
     23905 ± 4%   +2300.9%     573931 ± 1%  TOTAL sched_debug.cpu#40.ttwu_local
    362073 ± 4%     -99.0%       3783 ±43%  TOTAL sched_debug.cpu#13.sched_goidle
         6 ±45%    +383.3%         29 ± 0%  TOTAL sched_debug.cpu#40.cpu_load[4]
     27170 ± 6%   +2006.6%     572362 ± 1%  TOTAL sched_debug.cpu#13.ttwu_local
         0 ± 0%      +Inf%          2 ± 0%  TOTAL sched_debug.cpu#40.nr_running
         0 ± 0%      +Inf%          1 ± 0%  TOTAL sched_debug.cfs_rq[13]:/.nr_running
         0 ± 0%      +Inf%          1 ± 0%  TOTAL sched_debug.cfs_rq[39]:/.nr_running
    956859 ± 2%    +664.8%    7318047 ± 1%  TOTAL sched_debug.cfs_rq[39]:/.min_vruntime
         0 ± 0%      +Inf%          2 ± 0%  TOTAL sched_debug.cpu#14.nr_running
     22156 ± 6%   +2515.2%     579425 ± 1%  TOTAL sched_debug.cpu#39.ttwu_local
         5 ±20%    +400.0%         29 ± 2%  TOTAL sched_debug.cpu#39.cpu_load[4]
         0 ± 0%      +Inf%          2 ± 0%  TOTAL sched_debug.cpu#39.nr_running
    342979 ± 2%     -99.1%       3153 ±32%  TOTAL sched_debug.cpu#14.sched_goidle
         0 ± 0%      +Inf%          1 ± 0%  TOTAL sched_debug.cfs_rq[38]:/.nr_running
     25831 ± 2%   +2108.8%     570574 ± 1%  TOTAL sched_debug.cpu#14.ttwu_local
         0 ± 0%      +Inf%          1 ± 0%  TOTAL sched_debug.cfs_rq[14]:/.nr_running
    998185 ± 4%    +630.6%    7292251 ± 1%  TOTAL sched_debug.cfs_rq[38]:/.min_vruntime
     21969 ± 3%   +2536.5%     579224 ± 1%  TOTAL sched_debug.cpu#38.ttwu_local
         0 ± 0%      +Inf%          2 ±18%  TOTAL sched_debug.cpu#15.nr_running
         0 ± 0%      +Inf%          2 ± 0%  TOTAL sched_debug.cpu#38.nr_running
         0 ± 0%      +Inf%          1 ± 0%  TOTAL sched_debug.cfs_rq[37]:/.nr_running
         8 ±22%   +7077.3%        631 ±47%  TOTAL sched_debug.cfs_rq[37]:/.nr_spread_over
    976930 ± 2%    +653.4%    7359797 ± 1%  TOTAL sched_debug.cfs_rq[37]:/.min_vruntime
    365758 ± 2%     -98.9%       3877 ±47%  TOTAL sched_debug.cpu#15.sched_goidle
     22211 ± 5%   +2503.0%     578152 ± 0%  TOTAL sched_debug.cpu#37.ttwu_local
     25988 ± 4%   +2107.9%     573784 ± 1%  TOTAL sched_debug.cpu#15.ttwu_local
         4 ±23%    +559.1%         29 ± 2%  TOTAL sched_debug.cpu#37.cpu_load[4]
         0 ± 0%      +Inf%          1 ± 0%  TOTAL sched_debug.cfs_rq[15]:/.nr_running
         5 ±14%    +442.3%         28 ± 2%  TOTAL sched_debug.cpu#37.cpu_load[3]
         5 ± 9%    +414.8%         27 ± 2%  TOTAL sched_debug.cpu#37.cpu_load[2]
         0 ± 0%      +Inf%          2 ± 0%  TOTAL sched_debug.cpu#37.nr_running
         0 ± 0%      +Inf%          1 ± 0%  TOTAL sched_debug.cfs_rq[36]:/.nr_running
         0 ± 0%      +Inf%          2 ± 0%  TOTAL sched_debug.cpu#16.nr_running
        10 ±17%   +6420.8%        691 ±26%  TOTAL sched_debug.cfs_rq[36]:/.nr_spread_over
    951225 ± 5%    +672.6%    7349084 ± 1%  TOTAL sched_debug.cfs_rq[36]:/.min_vruntime
     22668 ± 5%   +2450.8%     578229 ± 1%  TOTAL sched_debug.cpu#36.ttwu_local
         0 ± 0%      +Inf%          2 ± 0%  TOTAL sched_debug.cpu#36.nr_running
     25231 ± 3%   +2177.7%     574692 ± 1%  TOTAL sched_debug.cpu#16.ttwu_local
         0 ± 0%      +Inf%          1 ± 0%  TOTAL sched_debug.cfs_rq[35]:/.nr_running
         0 ± 0%      +Inf%          1 ± 0%  TOTAL sched_debug.cfs_rq[16]:/.nr_running
    969215 ± 3%    +658.8%    7354084 ± 2%  TOTAL sched_debug.cfs_rq[35]:/.min_vruntime
         0 ± 0%      +Inf%          2 ± 0%  TOTAL sched_debug.cpu#17.nr_running
     22948 ± 6%   +2424.6%     579345 ± 1%  TOTAL sched_debug.cpu#35.ttwu_local
         4 ±44%    +615.0%         28 ± 3%  TOTAL sched_debug.cpu#35.cpu_load[4]
         4 ±48%    +518.2%         27 ± 4%  TOTAL sched_debug.cpu#35.cpu_load[0]
         0 ± 0%      +Inf%          2 ± 0%  TOTAL sched_debug.cpu#35.nr_running
         0 ± 0%      +Inf%          1 ± 0%  TOTAL sched_debug.cfs_rq[34]:/.nr_running
     24619 ± 2%   +2234.0%     574630 ± 1%  TOTAL sched_debug.cpu#17.ttwu_local
    985770 ± 4%    +646.0%    7353877 ± 1%  TOTAL sched_debug.cfs_rq[34]:/.min_vruntime
         0 ± 0%      +Inf%          1 ± 0%  TOTAL sched_debug.cfs_rq[17]:/.nr_running
     23297 ± 6%   +2389.1%     579887 ± 1%  TOTAL sched_debug.cpu#34.ttwu_local
         4 ±42%    +534.8%         29 ± 2%  TOTAL sched_debug.cpu#34.cpu_load[4]
         0 ± 0%      +Inf%          2 ± 0%  TOTAL sched_debug.cpu#18.nr_running
         5 ±41%    +429.6%         28 ± 3%  TOTAL sched_debug.cpu#34.cpu_load[3]
         0 ± 0%      +Inf%          2 ± 0%  TOTAL sched_debug.cpu#34.nr_running
         0 ± 0%      +Inf%          1 ± 0%  TOTAL sched_debug.cfs_rq[33]:/.nr_running
        11 ±39%   +7127.3%        795 ±29%  TOTAL sched_debug.cfs_rq[33]:/.nr_spread_over
    952580 ± 1%    +670.8%    7342099 ± 1%  TOTAL sched_debug.cfs_rq[33]:/.min_vruntime
     24246 ± 4%   +2265.3%     573496 ± 1%  TOTAL sched_debug.cpu#18.ttwu_local
     24645 ± 6%   +2249.7%     579112 ± 1%  TOTAL sched_debug.cpu#33.ttwu_local
    933626 ± 1%     -99.3%       6613 ±47%  TOTAL sched_debug.cpu#33.sched_goidle
         0 ± 0%      +Inf%          1 ± 0%  TOTAL sched_debug.cfs_rq[18]:/.nr_running
         0 ± 0%      +Inf%          2 ± 0%  TOTAL sched_debug.cpu#33.nr_running
         0 ± 0%      +Inf%          1 ± 0%  TOTAL sched_debug.cfs_rq[32]:/.nr_running
         0 ± 0%      +Inf%          2 ± 0%  TOTAL sched_debug.cpu#19.nr_running
    950196 ± 2%    +670.3%    7319391 ± 2%  TOTAL sched_debug.cfs_rq[32]:/.min_vruntime
     25033 ± 5%   +2211.6%     578682 ± 1%  TOTAL sched_debug.cpu#32.ttwu_local
         0 ± 0%      +Inf%          2 ± 0%  TOTAL sched_debug.cpu#32.nr_running
         0 ± 0%      +Inf%          1 ± 0%  TOTAL sched_debug.cfs_rq[31]:/.nr_running
     23931 ± 3%   +2304.2%     575367 ± 1%  TOTAL sched_debug.cpu#19.ttwu_local
     23792 ± 1%   +2309.0%     573173 ± 1%  TOTAL sched_debug.cpu#31.ttwu_local
         0 ± 0%      +Inf%          1 ± 0%  TOTAL sched_debug.cfs_rq[19]:/.nr_running
         0 ± 0%      +Inf%          2 ± 0%  TOTAL sched_debug.cpu#31.nr_running
         0 ± 0%      +Inf%          1 ± 0%  TOTAL sched_debug.cfs_rq[30]:/.nr_running
     45.19 ± 0%    -100.0%       0.00 ± 0%  TOTAL perf-profile.cpu-cycles.__crc32c_le.chksum_update.crypto_shash_update.crc32c.sctp_csum_update
         0 ± 0%      +Inf%          2 ± 0%  TOTAL sched_debug.cpu#20.nr_running
     23990 ± 2%   +2299.0%     575544 ± 1%  TOTAL sched_debug.cpu#30.ttwu_local
         0 ± 0%      +Inf%          2 ± 0%  TOTAL sched_debug.cpu#30.nr_running
         0 ± 0%      +Inf%          1 ± 0%  TOTAL sched_debug.cfs_rq[29]:/.nr_running
    333744 ± 2%     -99.1%       3032 ±30%  TOTAL sched_debug.cpu#20.sched_goidle
     23769 ± 2%   +2321.8%     575648 ± 1%  TOTAL sched_debug.cpu#20.ttwu_local
     23963 ± 2%   +2289.1%     572507 ± 1%  TOTAL sched_debug.cpu#29.ttwu_local
         0 ± 0%      +Inf%          1 ± 0%  TOTAL sched_debug.cfs_rq[20]:/.nr_running
         0 ± 0%      +Inf%          2 ± 0%  TOTAL sched_debug.cpu#29.nr_running
         0 ± 0%      +Inf%          1 ± 0%  TOTAL sched_debug.cfs_rq[28]:/.nr_running
        16 ±16%  +14806.2%       2385 ±48%  TOTAL sched_debug.cfs_rq[28]:/.nr_spread_over
         0 ± 0%      +Inf%          2 ± 0%  TOTAL sched_debug.cpu#21.nr_running
     23955 ± 2%   +2294.9%     573707 ± 1%  TOTAL sched_debug.cpu#28.ttwu_local
         0 ± 0%      +Inf%          2 ± 0%  TOTAL sched_debug.cpu#28.nr_running
         0 ± 0%      +Inf%          1 ± 0%  TOTAL sched_debug.cfs_rq[27]:/.nr_running
    200672 ±41%   +3130.2%    6482112 ±11%  TOTAL sched_debug.cfs_rq[27]:/.max_vruntime
    333344 ± 3%     -99.4%       2160 ±44%  TOTAL sched_debug.cpu#21.sched_goidle
    200672 ±41%   +3130.2%    6482112 ±11%  TOTAL sched_debug.cfs_rq[27]:/.MIN_vruntime
     24388 ± 2%   +2254.3%     574189 ± 1%  TOTAL sched_debug.cpu#27.ttwu_local
     23766 ± 2%   +2316.8%     574378 ± 1%  TOTAL sched_debug.cpu#21.ttwu_local
         0 ± 0%      +Inf%          2 ± 0%  TOTAL sched_debug.cpu#27.nr_running
        16 ±23%   +8230.9%       1349 ±31%  TOTAL sched_debug.cfs_rq[21]:/.nr_spread_over
         0 ± 0%      +Inf%          1 ± 0%  TOTAL sched_debug.cfs_rq[21]:/.nr_running
         0 ± 0%      +Inf%          1 ± 0%  TOTAL sched_debug.cfs_rq[26]:/.nr_running
         0 ± 0%      +Inf%          2 ± 0%  TOTAL sched_debug.cpu#22.nr_running
     24877 ± 2%   +2200.2%     572209 ± 1%  TOTAL sched_debug.cpu#26.ttwu_local
    333909 ± 4%     -98.5%       4876 ±31%  TOTAL sched_debug.cpu#26.sched_goidle
         0 ± 0%      +Inf%          2 ± 0%  TOTAL sched_debug.cpu#26.nr_running
         0 ± 0%      +Inf%          1 ± 0%  TOTAL sched_debug.cfs_rq[25]:/.nr_running
        21 ±22%   +8784.9%       1883 ±35%  TOTAL sched_debug.cfs_rq[25]:/.nr_spread_over
     24697 ± 4%   +2214.3%     571582 ± 1%  TOTAL sched_debug.cpu#25.ttwu_local
     24175 ± 5%   +2283.3%     576162 ± 1%  TOTAL sched_debug.cpu#22.ttwu_local
         0 ± 0%      +Inf%          2 ± 0%  TOTAL sched_debug.cpu#25.nr_running
         0 ± 0%      +Inf%          1 ± 0%  TOTAL sched_debug.cfs_rq[22]:/.nr_running
         0 ± 0%      +Inf%          1 ± 0%  TOTAL sched_debug.cfs_rq[24]:/.nr_running
         0 ± 0%      +Inf%          2 ± 0%  TOTAL sched_debug.cpu#23.nr_running
     25076 ± 2%   +2194.8%     575457 ± 1%  TOTAL sched_debug.cpu#24.ttwu_local
    336969 ± 2%     -99.1%       3193 ±49%  TOTAL sched_debug.cpu#24.sched_goidle
         0 ± 0%      +Inf%          2 ± 0%  TOTAL sched_debug.cpu#24.nr_running
         0 ± 0%      +Inf%          1 ± 0%  TOTAL sched_debug.cfs_rq[23]:/.nr_running
    420529 ±14%     -80.8%      80883 ± 6%  TOTAL sched_debug.cpu#24.avg_idle
        35 ± 1%    +394.3%        174 ± 0%  TOTAL vmstat.procs.r
    320932 ±15%     -77.7%      71461 ± 8%  TOTAL sched_debug.cpu#26.avg_idle
    295750 ±21%     -76.1%      70793 ±17%  TOTAL sched_debug.cpu#2.avg_idle
    623387 ± 0%     -79.5%     127567 ± 1%  TOTAL softirqs.SCHED
         6 ±42%    +320.6%         28 ± 1%  TOTAL sched_debug.cpu#40.cpu_load[3]
         6 ±33%    +376.7%         28 ± 3%  TOTAL sched_debug.cpu#51.cpu_load[3]
         6 ±27%    +358.1%         28 ± 4%  TOTAL sched_debug.cpu#62.cpu_load[3]
         6 ±27%    +358.1%         28 ± 2%  TOTAL sched_debug.cpu#59.cpu_load[4]
         5 ±20%    +403.4%         29 ± 2%  TOTAL sched_debug.cpu#44.cpu_load[3]
         5 ±25%    +437.0%         29 ± 2%  TOTAL sched_debug.cpu#38.cpu_load[4]
         6 ±21%    +361.3%         28 ± 3%  TOTAL sched_debug.cpu#46.cpu_load[1]
         6 ±10%    +380.0%         28 ± 4%  TOTAL sched_debug.cpu#46.cpu_load[2]
         6 ±18%    +364.5%         28 ± 2%  TOTAL sched_debug.cpu#44.cpu_load[2]
         6 ±32%    +364.5%         28 ± 1%  TOTAL sched_debug.cpu#36.cpu_load[4]
         6 ±21%    +358.1%         28 ± 2%  TOTAL sched_debug.cpu#44.cpu_load[1]
         6 ±29%    +354.8%         28 ± 3%  TOTAL sched_debug.cpu#44.cpu_load[0]
         6 ±44%    +361.3%         28 ± 1%  TOTAL sched_debug.cpu#32.cpu_load[4]
         6 ±33%    +373.3%         28 ± 2%  TOTAL sched_debug.cpu#55.cpu_load[3]
         6 ±28%    +323.5%         28 ± 4%  TOTAL sched_debug.cpu#46.cpu_load[0]
         5 ± 6%    +406.9%         29 ± 3%  TOTAL sched_debug.cpu#46.cpu_load[3]
    340647 ±14%     -77.1%      77839 ±14%  TOTAL sched_debug.cpu#25.avg_idle
         6 ±18%    +376.7%         28 ± 2%  TOTAL sched_debug.cpu#38.cpu_load[3]
         4 ±37%    +508.7%         28 ± 2%  TOTAL sched_debug.cpu#35.cpu_load[3]
         6 ±29%    +314.7%         28 ± 4%  TOTAL sched_debug.cpu#56.cpu_load[0]
         6 ±15%    +346.9%         28 ± 2%  TOTAL sched_debug.cpu#63.cpu_load[3]
         7 ±47%    +305.7%         28 ± 1%  TOTAL sched_debug.cpu#32.cpu_load[3]
         6 ±17%    +317.6%         28 ± 3%  TOTAL sched_debug.cpu#52.cpu_load[4]
         6 ±15%    +343.8%         28 ± 3%  TOTAL sched_debug.cpu#56.cpu_load[2]
         6 ±37%    +337.5%         28 ± 3%  TOTAL sched_debug.cpu#55.cpu_load[2]
         6 ±23%    +343.8%         28 ± 1%  TOTAL sched_debug.cpu#39.cpu_load[3]
         6 ±29%    +351.6%         28 ± 5%  TOTAL sched_debug.cpu#34.cpu_load[1]
         6 ±31%    +354.8%         28 ± 4%  TOTAL sched_debug.cpu#34.cpu_load[2]
         6 ±42%    +358.1%         28 ± 5%  TOTAL sched_debug.cpu#53.cpu_load[1]
         6 ±40%    +373.3%         28 ± 3%  TOTAL sched_debug.cpu#53.cpu_load[2]
         6 ±28%    +327.3%         28 ± 1%  TOTAL sched_debug.cpu#36.cpu_load[3]
         6 ±20%    +327.3%         28 ± 2%  TOTAL sched_debug.cpu#56.cpu_load[1]
         7 ±29%    +297.1%         27 ± 1%  TOTAL sched_debug.cpu#50.cpu_load[2]
         7 ±44%    +265.8%         27 ± 1%  TOTAL sched_debug.cpu#32.cpu_load[2]
    317802 ± 9%     -77.7%      70785 ± 9%  TOTAL sched_debug.cpu#4.avg_idle
    346844 ±10%     -77.6%      77616 ± 8%  TOTAL sched_debug.cpu#16.avg_idle
         4 ±40%    +475.0%         27 ± 2%  TOTAL sched_debug.cpu#35.cpu_load[1]
         5 ±35%    +452.0%         27 ± 2%  TOTAL sched_debug.cfs_rq[48]:/.runnable_load_avg
         4 ±40%    +470.8%         27 ± 2%  TOTAL sched_debug.cpu#35.cpu_load[2]
         5 ±18%    +385.7%         27 ± 4%  TOTAL sched_debug.cpu#37.cpu_load[0]
         5 ± 6%    +369.0%         27 ± 2%  TOTAL sched_debug.cpu#37.cpu_load[1]
    366939 ±13%     -78.7%      78115 ± 9%  TOTAL sched_debug.cpu#30.avg_idle
      1023 ±37%    +323.5%       4333 ± 2%  TOTAL sched_debug.cpu#48.curr->pid
    327585 ± 9%     -77.2%      74734 ±11%  TOTAL sched_debug.cpu#17.avg_idle
     12595 ± 0%     -76.7%       2929 ± 0%  TOTAL uptime.idle
    338343 ±14%     -78.4%      73057 ± 5%  TOTAL sched_debug.cpu#31.avg_idle
         6 ±37%    +353.1%         29 ± 4%  TOTAL sched_debug.cpu#58.cpu_load[4]
         7 ±23%    +322.9%         29 ± 3%  TOTAL sched_debug.cpu#47.cpu_load[4]
         7 ±28%    +316.7%         30 ± 2%  TOTAL sched_debug.cpu#42.cpu_load[4]
    316565 ±15%     -74.7%      80204 ±11%  TOTAL sched_debug.cpu#23.avg_idle
    295009 ±22%     -75.0%      73750 ± 9%  TOTAL sched_debug.cpu#22.avg_idle
  19299589 ± 1%    +316.6%   80411460 ± 1%  TOTAL proc-vmstat.pgalloc_dma32
         6 ±26%    +311.8%         28 ± 4%  TOTAL sched_debug.cpu#62.cpu_load[2]
         6 ±21%    +393.3%         29 ± 2%  TOTAL sched_debug.cpu#45.cpu_load[3]
         7 ±18%    +311.4%         28 ± 2%  TOTAL sched_debug.cpu#45.cpu_load[2]
         7 ±10%    +286.8%         29 ± 1%  TOTAL sched_debug.cfs_rq[45]:/.runnable_load_avg
         8 ±27%    +234.9%         28 ± 2%  TOTAL sched_debug.cpu#42.cpu_load[1]
         6 ±19%    +323.5%         28 ± 4%  TOTAL sched_debug.cpu#60.cpu_load[4]
         8 ±26%    +265.0%         29 ± 2%  TOTAL sched_debug.cpu#42.cpu_load[3]
         7 ±13%    +297.2%         28 ± 5%  TOTAL sched_debug.cfs_rq[38]:/.runnable_load_avg
         8 ±27%    +234.9%         28 ± 2%  TOTAL sched_debug.cpu#42.cpu_load[2]
         6 ±40%    +364.5%         28 ± 4%  TOTAL sched_debug.cpu#54.cpu_load[4]
    306463 ±16%     -72.5%      84417 ±11%  TOTAL sched_debug.cpu#7.avg_idle
    294085 ±13%     -74.4%      75243 ±11%  TOTAL sched_debug.cpu#58.avg_idle
    295178 ± 8%     -74.8%      74330 ±13%  TOTAL sched_debug.cpu#19.avg_idle
    310735 ±12%     -74.4%      79628 ±23%  TOTAL sched_debug.cpu#12.avg_idle
         7 ±21%    +275.7%         27 ± 3%  TOTAL sched_debug.cpu#59.cpu_load[3]
         7 ±30%    +277.8%         27 ± 5%  TOTAL sched_debug.cpu#62.cpu_load[1]
         7 ±40%    +300.0%         28 ± 0%  TOTAL sched_debug.cpu#40.cpu_load[2]
         6 ±34%    +314.7%         28 ± 2%  TOTAL sched_debug.cpu#51.cpu_load[2]
         7 ±49%    +280.6%         27 ± 4%  TOTAL sched_debug.cfs_rq[40]:/.runnable_load_avg
         7 ±34%    +297.1%         27 ± 4%  TOTAL sched_debug.cpu#55.cpu_load[1]
         8 ±20%    +245.0%         27 ± 1%  TOTAL sched_debug.cpu#36.cpu_load[1]
         7 ±23%    +286.1%         27 ± 1%  TOTAL sched_debug.cpu#36.cpu_load[2]
         7 ±16%    +288.9%         28 ± 3%  TOTAL sched_debug.cpu#52.cpu_load[3]
         7 ±13%    +278.4%         28 ± 2%  TOTAL sched_debug.cpu#63.cpu_load[2]
         7 ±23%    +288.9%         28 ± 2%  TOTAL sched_debug.cpu#39.cpu_load[2]
         7 ±28%    +271.1%         28 ± 1%  TOTAL sched_debug.cpu#39.cpu_load[1]
         7 ±18%    +288.9%         28 ± 5%  TOTAL sched_debug.cfs_rq[63]:/.runnable_load_avg
         6 ±28%    +302.9%         27 ± 2%  TOTAL sched_debug.cfs_rq[37]:/.runnable_load_avg
         7 ±30%    +255.3%         27 ± 6%  TOTAL sched_debug.cpu#38.cpu_load[0]
         7 ±25%    +283.3%         27 ± 4%  TOTAL sched_debug.cpu#38.cpu_load[1]
         6 ±22%    +324.2%         28 ± 3%  TOTAL sched_debug.cpu#38.cpu_load[2]
         6 ±25%    +308.8%         27 ± 1%  TOTAL sched_debug.cfs_rq[56]:/.runnable_load_avg
         7 ±38%    +267.6%         27 ± 5%  TOTAL sched_debug.cpu#62.cpu_load[0]
    317966 ±13%     -74.2%      82042 ± 9%  TOTAL sched_debug.cpu#27.avg_idle
    296746 ±16%     -74.7%      75078 ± 9%  TOTAL sched_debug.cpu#14.avg_idle
    298670 ±15%     -73.3%      79831 ±22%  TOTAL sched_debug.cpu#6.avg_idle
         6 ±32%    +302.9%         27 ± 5%  TOTAL sched_debug.cpu#34.cpu_load[0]
         7 ±10%    +260.5%         27 ± 3%  TOTAL sched_debug.cpu#52.cpu_load[2]
         6 ±23%    +297.1%         27 ± 0%  TOTAL sched_debug.cfs_rq[61]:/.runnable_load_avg
         8 ±39%    +216.3%         27 ± 1%  TOTAL sched_debug.cpu#32.cpu_load[1]
         7 ±30%    +252.6%         26 ± 2%  TOTAL sched_debug.cpu#50.cpu_load[0]
         6 ±38%    +302.9%         27 ± 1%  TOTAL sched_debug.cpu#40.cpu_load[1]
         7 ±16%    +283.3%         27 ± 2%  TOTAL sched_debug.cpu#63.cpu_load[1]
         7 ±27%    +255.3%         27 ± 2%  TOTAL sched_debug.cpu#50.cpu_load[1]
         6 ±44%    +335.5%         27 ± 3%  TOTAL sched_debug.cfs_rq[35]:/.runnable_load_avg
         6 ±39%    +300.0%         27 ± 2%  TOTAL sched_debug.cpu#40.cpu_load[0]
    294207 ±16%     -73.9%      76789 ±18%  TOTAL sched_debug.cpu#20.avg_idle
      1454 ±47%    +193.6%       4270 ± 1%  TOTAL sched_debug.cpu#32.curr->pid
    274384 ±28%     -64.9%      96227 ±33%  TOTAL sched_debug.cpu#33.avg_idle
    287321 ± 8%     -73.1%      77157 ±10%  TOTAL sched_debug.cpu#3.avg_idle
         7 ±10%    +255.3%         27 ± 4%  TOTAL sched_debug.cpu#52.cpu_load[1]
      1284 ±29%    +228.6%       4222 ± 0%  TOTAL sched_debug.cpu#43.curr->pid
    349008 ±15%     -74.2%      90084 ± 6%  TOTAL sched_debug.cpu#21.avg_idle
         9 ±11%    +230.6%         32 ± 4%  TOTAL sched_debug.cpu#49.cpu_load[4]
         7 ±30%    +281.6%         29 ± 2%  TOTAL sched_debug.cpu#43.cpu_load[2]
         7 ±29%    +297.3%         29 ± 2%  TOTAL sched_debug.cpu#43.cpu_load[3]
         6 ±38%    +323.5%         28 ± 4%  TOTAL sched_debug.cpu#54.cpu_load[3]
         8 ±29%    +223.3%         27 ±10%  TOTAL sched_debug.cfs_rq[58]:/.runnable_load_avg
         7 ±18%    +271.8%         29 ± 2%  TOTAL sched_debug.cpu#47.cpu_load[3]
         7 ±31%    +322.9%         29 ± 2%  TOTAL sched_debug.cpu#43.cpu_load[4]
         7 ±32%    +283.8%         28 ± 4%  TOTAL sched_debug.cpu#58.cpu_load[3]
         9 ±25%    +222.2%         29 ± 3%  TOTAL sched_debug.cpu#42.cpu_load[0]
    291808 ±12%     -72.2%      81207 ±12%  TOTAL sched_debug.cpu#0.avg_idle
     12900 ± 6%    +248.9%      45016 ± 0%  TOTAL sched_debug.cfs_rq[61]:/.avg->runnable_avg_sum
       282 ± 6%    +247.8%        982 ± 0%  TOTAL sched_debug.cfs_rq[61]:/.tg_runnable_contrib
    307360 ±20%     -75.2%      76165 ± 7%  TOTAL sched_debug.cpu#13.avg_idle
      1213 ±36%    +252.3%       4273 ± 0%  TOTAL sched_debug.cpu#35.curr->pid
   2108032 ± 5%     -71.2%     607806 ± 1%  TOTAL sched_debug.cpu#34.sched_count
   2130952 ± 9%     -71.4%     609735 ± 1%  TOTAL sched_debug.cpu#33.sched_count
         7 ±39%    +268.4%         28 ± 3%  TOTAL sched_debug.cpu#54.cpu_load[0]
         7 ±39%    +278.4%         28 ± 3%  TOTAL sched_debug.cfs_rq[53]:/.runnable_load_avg
         8 ±23%    +223.3%         27 ± 2%  TOTAL sched_debug.cpu#36.cpu_load[0]
         7 ±20%    +286.1%         27 ± 4%  TOTAL sched_debug.cpu#60.cpu_load[3]
         7 ±21%    +265.8%         27 ± 4%  TOTAL sched_debug.cfs_rq[50]:/.runnable_load_avg
         7 ±31%    +276.3%         28 ± 2%  TOTAL sched_debug.cpu#45.cpu_load[0]
         8 ±24%    +226.2%         27 ± 2%  TOTAL sched_debug.cfs_rq[55]:/.runnable_load_avg
         6 ±36%    +320.6%         28 ± 4%  TOTAL sched_debug.cfs_rq[44]:/.runnable_load_avg
         7 ±26%    +281.1%         28 ± 2%  TOTAL sched_debug.cpu#51.cpu_load[1]
         7 ±27%    +265.8%         27 ± 4%  TOTAL sched_debug.cpu#51.cpu_load[0]
         8 ±27%    +233.3%         28 ± 2%  TOTAL sched_debug.cpu#39.cpu_load[0]
         7 ±21%    +276.3%         28 ± 2%  TOTAL sched_debug.cpu#45.cpu_load[1]
       293 ± 8%    +234.7%        980 ± 0%  TOTAL sched_debug.cfs_rq[56]:/.tg_runnable_contrib
   2074224 ± 3%     -70.6%     609675 ± 2%  TOTAL sched_debug.cpu#63.sched_count
   2127419 ± 6%     -71.7%     601604 ± 1%  TOTAL sched_debug.cpu#46.sched_count
   2115949 ± 3%     -65.2%     737051 ±35%  TOTAL sched_debug.cpu#36.sched_count
     13414 ± 8%    +235.7%      45027 ± 0%  TOTAL sched_debug.cfs_rq[56]:/.avg->runnable_avg_sum
    267808 ±13%     -72.1%      74782 ±10%  TOTAL sched_debug.cpu#5.avg_idle
   2069493 ± 8%     -68.4%     654987 ±18%  TOTAL sched_debug.cpu#45.sched_count
     14044 ±11%    +220.7%      45039 ± 0%  TOTAL sched_debug.cfs_rq[55]:/.avg->runnable_avg_sum
   2053686 ± 4%     -70.7%     602700 ± 2%  TOTAL sched_debug.cpu#52.sched_count
    289450 ±13%     -68.8%      90401 ±15%  TOTAL sched_debug.cpu#1.avg_idle
       307 ±11%    +219.5%        980 ± 0%  TOTAL sched_debug.cfs_rq[55]:/.tg_runnable_contrib
   2002864 ± 8%     -69.9%     602844 ± 1%  TOTAL sched_debug.cpu#40.sched_count
   2035365 ± 2%     -70.3%     603651 ± 1%  TOTAL sched_debug.cpu#47.sched_count
         9 ±24%    +197.8%         26 ± 1%  TOTAL sched_debug.cfs_rq[59]:/.runnable_load_avg
         8 ±24%    +226.8%         26 ± 6%  TOTAL sched_debug.cpu#60.cpu_load[0]
         8 ±23%    +235.0%         26 ± 4%  TOTAL sched_debug.cpu#59.cpu_load[1]
         7 ±19%    +252.6%         26 ± 4%  TOTAL sched_debug.cpu#60.cpu_load[1]
         8 ±23%    +234.1%         27 ± 3%  TOTAL sched_debug.cpu#59.cpu_load[2]
         8 ±36%    +209.1%         27 ± 2%  TOTAL sched_debug.cfs_rq[32]:/.runnable_load_avg
         7 ±20%    +253.8%         27 ± 2%  TOTAL sched_debug.cpu#63.cpu_load[0]
         8 ±16%    +206.8%         27 ± 4%  TOTAL sched_debug.cfs_rq[52]:/.runnable_load_avg
         7 ±19%    +257.9%         27 ± 4%  TOTAL sched_debug.cpu#60.cpu_load[2]
   2008711 ± 1%     -70.0%     602980 ± 1%  TOTAL sched_debug.cpu#61.nr_switches
   2100707 ± 7%     -71.1%     606142 ± 2%  TOTAL sched_debug.cpu#53.sched_count
   2046800 ± 1%     -65.3%     710307 ±28%  TOTAL sched_debug.cpu#60.sched_count
   2091856 ± 3%     -70.6%     614687 ± 3%  TOTAL sched_debug.cpu#38.sched_count
   2062814 ± 3%     -70.7%     604560 ± 2%  TOTAL sched_debug.cpu#54.sched_count
     13554 ±11%    +232.3%      45048 ± 0%  TOTAL sched_debug.cfs_rq[35]:/.avg->runnable_avg_sum
       296 ±11%    +231.2%        982 ± 0%  TOTAL sched_debug.cfs_rq[35]:/.tg_runnable_contrib
      1474 ±29%    +188.2%       4249 ± 0%  TOTAL sched_debug.cpu#62.curr->pid
   2049101 ± 6%     -70.4%     607303 ± 2%  TOTAL sched_debug.cpu#41.sched_count
   2167538 ± 9%     -71.9%     609753 ± 2%  TOTAL sched_debug.cpu#39.sched_count
   2036680 ± 3%     -70.3%     605379 ± 1%  TOTAL sched_debug.cpu#51.sched_count
   2017924 ± 1%     -70.0%     604898 ± 0%  TOTAL sched_debug.cpu#59.sched_count
     14509 ±13%    +210.5%      45046 ± 0%  TOTAL sched_debug.cfs_rq[63]:/.avg->runnable_avg_sum
   2076058 ± 5%     -69.2%     640162 ±10%  TOTAL sched_debug.cpu#37.sched_count
      1225 ±36%    +253.2%       4329 ± 3%  TOTAL sched_debug.cpu#53.curr->pid
      1148 ±27%    +268.3%       4230 ± 1%  TOTAL sched_debug.cpu#61.curr->pid
     14570 ±13%    +209.0%      45024 ± 0%  TOTAL sched_debug.cfs_rq[34]:/.avg->runnable_avg_sum
   2020274 ± 0%     -69.8%     610215 ± 2%  TOTAL sched_debug.cpu#62.nr_switches
       319 ±13%    +208.0%        982 ± 0%  TOTAL sched_debug.cfs_rq[34]:/.tg_runnable_contrib
       317 ±13%    +208.1%        978 ± 0%  TOTAL sched_debug.cfs_rq[63]:/.tg_runnable_contrib
   2042951 ± 1%     -69.7%     618701 ± 2%  TOTAL sched_debug.cpu#62.sched_count
   2058101 ± 7%     -70.8%     600726 ± 1%  TOTAL sched_debug.cpu#44.sched_count
   1984924 ± 1%     -69.5%     604880 ± 0%  TOTAL sched_debug.cpu#59.nr_switches
   1971315 ± 1%     -69.3%     604542 ± 2%  TOTAL sched_debug.cpu#54.nr_switches
   2014162 ± 6%     -65.2%     700760 ±28%  TOTAL sched_debug.cpu#42.sched_count
   1953776 ± 2%     -68.8%     609656 ± 2%  TOTAL sched_debug.cpu#63.nr_switches
   2055362 ± 7%     -65.1%     717767 ±33%  TOTAL sched_debug.cpu#43.sched_count
      1231 ±29%    +249.4%       4303 ± 2%  TOTAL sched_debug.cpu#50.curr->pid
       313 ±10%    +213.7%        982 ± 0%  TOTAL sched_debug.cfs_rq[62]:/.tg_runnable_contrib
         7 ±13%    +255.3%         27 ± 6%  TOTAL sched_debug.cpu#52.cpu_load[0]
         8 ±31%    +202.3%         26 ± 3%  TOTAL sched_debug.cpu#32.cpu_load[0]
   1991792 ± 1%     -69.3%     611077 ± 1%  TOTAL sched_debug.cpu#60.nr_switches
     14342 ±10%    +214.0%      45031 ± 0%  TOTAL sched_debug.cfs_rq[62]:/.avg->runnable_avg_sum
   1942697 ± 1%     -69.0%     602682 ± 2%  TOTAL sched_debug.cpu#52.nr_switches
   1950644 ± 1%     -69.2%     601592 ± 1%  TOTAL sched_debug.cpu#46.nr_switches
   1992472 ± 1%     -61.6%     765257 ±25%  TOTAL sched_debug.cpu#50.sched_count
         8 ±20%    +241.5%         28 ± 4%  TOTAL sched_debug.cpu#58.cpu_load[1]
         9 ±12%    +215.6%         28 ± 2%  TOTAL sched_debug.cfs_rq[39]:/.runnable_load_avg
         7 ±31%    +283.8%         28 ± 2%  TOTAL sched_debug.cpu#54.cpu_load[2]
         8 ±22%    +238.1%         28 ± 2%  TOTAL sched_debug.cpu#47.cpu_load[2]
         8 ±26%    +250.0%         28 ± 4%  TOTAL sched_debug.cpu#58.cpu_load[2]
     13572 ±12%    +232.0%      45056 ± 0%  TOTAL sched_debug.cfs_rq[37]:/.avg->runnable_avg_sum
   2047201 ± 6%     -67.2%     670954 ±18%  TOTAL sched_debug.cpu#35.sched_count
       297 ±12%    +230.8%        983 ± 0%  TOTAL sched_debug.cfs_rq[37]:/.tg_runnable_contrib
   1955174 ± 2%     -69.1%     603605 ± 1%  TOTAL sched_debug.cpu#55.nr_switches
   1971669 ± 5%     -69.1%     608414 ± 1%  TOTAL sched_debug.cpu#49.sched_count
   1939844 ± 1%     -68.8%     605845 ± 1%  TOTAL sched_debug.cpu#36.nr_switches
   1949899 ± 1%     -68.9%     606124 ± 2%  TOTAL sched_debug.cpu#53.nr_switches
       315 ±12%    +211.7%        984 ± 0%  TOTAL sched_debug.cfs_rq[51]:/.tg_runnable_contrib
        12 ±27%    +166.7%         32 ± 6%  TOTAL sched_debug.cpu#49.cpu_load[2]
        11 ±21%    +184.2%         32 ± 5%  TOTAL sched_debug.cpu#49.cpu_load[3]
     14438 ±12%    +212.0%      45043 ± 0%  TOTAL sched_debug.cfs_rq[51]:/.avg->runnable_avg_sum
   2001341 ± 4%     -66.4%     671981 ±17%  TOTAL sched_debug.cpu#57.sched_count
    303710 ±17%     -70.0%      91263 ±13%  TOTAL sched_debug.cpu#9.avg_idle
   1900887 ± 1%     -68.4%     600434 ± 1%  TOTAL sched_debug.cpu#45.nr_switches
     14272 ±12%    +215.8%      45065 ± 0%  TOTAL sched_debug.cfs_rq[59]:/.avg->runnable_avg_sum
   1937164 ± 0%     -68.5%     609385 ± 1%  TOTAL sched_debug.cpu#37.nr_switches
   1974368 ± 1%     -69.1%     609730 ± 2%  TOTAL sched_debug.cpu#39.nr_switches
   1973848 ± 2%     -62.6%     738151 ±35%  TOTAL sched_debug.cpu#58.sched_count
   1912735 ± 1%     -68.5%     603190 ± 2%  TOTAL sched_debug.cpu#50.nr_switches
   1931643 ± 2%     -68.6%     607250 ± 2%  TOTAL sched_debug.cpu#58.nr_switches
      1377 ±14%    +216.7%       4363 ± 1%  TOTAL sched_debug.cpu#49.curr->pid
       312 ±12%    +214.1%        981 ± 0%  TOTAL sched_debug.cfs_rq[59]:/.tg_runnable_contrib
   1944033 ± 1%     -68.9%     605363 ± 1%  TOTAL sched_debug.cpu#51.nr_switches
   1915699 ± 1%     -68.1%     610629 ± 1%  TOTAL sched_debug.cpu#35.nr_switches
     15047 ±12%    +199.1%      45011 ± 0%  TOTAL sched_debug.cfs_rq[32]:/.avg->runnable_avg_sum
     46418 ± 2%    +213.2%     145374 ± 0%  TOTAL sched_debug.cfs_rq[62]:/.exec_clock
   1924241 ± 2%     -68.3%     609789 ± 2%  TOTAL sched_debug.cpu#38.nr_switches
   1912400 ± 1%     -68.4%     603629 ± 1%  TOTAL sched_debug.cpu#47.nr_switches
       329 ±12%    +197.8%        979 ± 0%  TOTAL sched_debug.cfs_rq[32]:/.tg_runnable_contrib
   1895051 ± 2%     -68.3%     600708 ± 1%  TOTAL sched_debug.cpu#44.nr_switches
     13983 ±14%    +222.1%      45044 ± 0%  TOTAL sched_debug.cfs_rq[44]:/.avg->runnable_avg_sum
   1901704 ± 1%     -67.9%     609715 ± 1%  TOTAL sched_debug.cpu#33.nr_switches
     14514 ± 2%    +214.1%      45583 ± 0%  TOTAL sched_debug.cfs_rq[33]:/.avg->runnable_avg_sum
       306 ±14%    +221.0%        982 ± 0%  TOTAL sched_debug.cfs_rq[44]:/.tg_runnable_contrib
   1926663 ± 0%     -68.2%     613403 ± 2%  TOTAL sched_debug.cpu#48.sched_count
       317 ± 2%    +212.6%        993 ± 0%  TOTAL sched_debug.cfs_rq[33]:/.tg_runnable_contrib
      1203 ±34%    +258.3%       4310 ± 1%  TOTAL sched_debug.cpu#56.curr->pid
     46837 ± 2%    +210.4%     145375 ± 0%  TOTAL sched_debug.cfs_rq[60]:/.exec_clock
   1874381 ± 1%     -67.9%     600905 ± 1%  TOTAL sched_debug.cpu#42.nr_switches
         8 ±16%    +238.1%         28 ± 3%  TOTAL sched_debug.cfs_rq[47]:/.runnable_load_avg
         9 ±12%    +200.0%         27 ± 4%  TOTAL sched_debug.cpu#58.cpu_load[0]
         9 ±24%    +213.3%         28 ± 2%  TOTAL sched_debug.cpu#47.cpu_load[1]
         9 ±28%    +200.0%         28 ± 1%  TOTAL sched_debug.cpu#47.cpu_load[0]
         8 ±34%    +239.0%         27 ± 4%  TOTAL sched_debug.cpu#55.cpu_load[0]
         8 ±15%    +222.7%         28 ± 3%  TOTAL sched_debug.cfs_rq[36]:/.runnable_load_avg
         8 ±23%    +246.3%         28 ± 3%  TOTAL sched_debug.cpu#43.cpu_load[1]
         8 ±22%    +250.0%         28 ± 2%  TOTAL sched_debug.cfs_rq[43]:/.runnable_load_avg
         9 ±37%    +213.3%         28 ± 4%  TOTAL sched_debug.cfs_rq[34]:/.runnable_load_avg
         7 ±32%    +268.4%         28 ± 3%  TOTAL sched_debug.cpu#54.cpu_load[1]
         9 ±37%    +208.9%         27 ± 4%  TOTAL sched_debug.cfs_rq[51]:/.runnable_load_avg
   1916043 ± 2%     -68.3%     607993 ± 2%  TOTAL sched_debug.cpu#57.nr_switches
    265392 ±17%     -64.3%      94795 ±22%  TOTAL sched_debug.cpu#43.avg_idle
   1889707 ± 1%     -68.3%     599744 ± 1%  TOTAL sched_debug.cpu#43.nr_switches
         9 ±29%    +227.1%         31 ± 3%  TOTAL sched_debug.cpu#33.cpu_load[4]
        39 ±46%    +117.4%         84 ±32%  TOTAL sched_debug.cfs_rq[17]:/.tg_load_contrib
   1867119 ± 0%     -67.6%     605812 ± 1%  TOTAL sched_debug.cpu#49.nr_switches
   1889889 ± 1%     -67.8%     607783 ± 1%  TOTAL sched_debug.cpu#34.nr_switches
     47035 ± 4%    +209.1%     145403 ± 0%  TOTAL sched_debug.cfs_rq[61]:/.exec_clock
   1860154 ± 1%     -67.6%     602684 ± 2%  TOTAL sched_debug.cpu#48.nr_switches
       325 ±10%    +202.2%        982 ± 0%  TOTAL sched_debug.cfs_rq[39]:/.tg_runnable_contrib
  66798546 ± 0%    +206.8%  2.049e+08 ± 0%  TOTAL softirqs.NET_RX
     13894 ±13%    +224.1%      45029 ± 0%  TOTAL sched_debug.cfs_rq[48]:/.avg->runnable_avg_sum
     14866 ±10%    +202.7%      44994 ± 0%  TOTAL sched_debug.cfs_rq[39]:/.avg->runnable_avg_sum
     14874 ±11%    +207.3%      45713 ± 0%  TOTAL sched_debug.cfs_rq[57]:/.avg->runnable_avg_sum
   1893238 ± 1%     -67.9%     608368 ± 1%  TOTAL sched_debug.cpu#56.sched_count
       304 ±13%    +222.6%        981 ± 0%  TOTAL sched_debug.cfs_rq[48]:/.tg_runnable_contrib
   1850633 ± 0%     -67.2%     606443 ± 2%  TOTAL sched_debug.cpu#41.nr_switches
    273840 ±22%     -69.2%      84378 ± 8%  TOTAL sched_debug.cpu#60.avg_idle
       325 ±12%    +206.1%        996 ± 0%  TOTAL sched_debug.cfs_rq[57]:/.tg_runnable_contrib
     14957 ±12%    +201.5%      45098 ± 0%  TOTAL sched_debug.cfs_rq[58]:/.avg->runnable_avg_sum
     47628 ± 2%    +205.5%     145491 ± 0%  TOTAL sched_debug.cfs_rq[51]:/.exec_clock
    271141 ±15%     -64.6%      96084 ±33%  TOTAL sched_debug.cpu#47.avg_idle
     47097 ± 2%    +208.9%     145500 ± 0%  TOTAL sched_debug.cfs_rq[59]:/.exec_clock
      1395 ±13%    +210.6%       4334 ± 2%  TOTAL sched_debug.cpu#55.curr->pid
     15193 ±10%    +196.8%      45096 ± 0%  TOTAL sched_debug.cfs_rq[50]:/.avg->runnable_avg_sum
       326 ±12%    +200.6%        981 ± 0%  TOTAL sched_debug.cfs_rq[58]:/.tg_runnable_contrib
       315 ± 8%    +211.4%        982 ± 0%  TOTAL sched_debug.cfs_rq[38]:/.tg_runnable_contrib
     14456 ± 8%    +211.3%      45004 ± 0%  TOTAL sched_debug.cfs_rq[38]:/.avg->runnable_avg_sum
     47896 ± 3%    +203.5%     145372 ± 0%  TOTAL sched_debug.cfs_rq[54]:/.exec_clock
       332 ±10%    +195.7%        984 ± 0%  TOTAL sched_debug.cfs_rq[50]:/.tg_runnable_contrib
   1865184 ± 2%     -67.4%     608353 ± 1%  TOTAL sched_debug.cpu#56.nr_switches
     47722 ± 2%    +204.7%     145411 ± 0%  TOTAL sched_debug.cfs_rq[55]:/.exec_clock
     47334 ± 2%    +207.3%     145475 ± 0%  TOTAL sched_debug.cfs_rq[52]:/.exec_clock
     15320 ± 5%    +193.8%      45011 ± 0%  TOTAL sched_debug.cfs_rq[46]:/.avg->runnable_avg_sum
     48192 ± 1%    +201.6%     145362 ± 0%  TOTAL sched_debug.cfs_rq[63]:/.exec_clock
     48037 ± 1%    +202.7%     145411 ± 0%  TOTAL sched_debug.cfs_rq[32]:/.exec_clock
      1497 ±19%    +193.5%       4395 ± 5%  TOTAL sched_debug.cpu#52.curr->pid
   1846875 ± 0%     -67.1%     607481 ± 1%  TOTAL sched_debug.cpu#32.nr_switches
    287922 ±23%     -70.6%      84722 ±16%  TOTAL sched_debug.cpu#11.avg_idle
     48162 ± 3%    +202.0%     145448 ± 0%  TOTAL sched_debug.cfs_rq[36]:/.exec_clock
       335 ± 5%    +192.7%        982 ± 0%  TOTAL sched_debug.cfs_rq[46]:/.tg_runnable_contrib
     47591 ± 2%    +205.6%     145417 ± 0%  TOTAL sched_debug.cfs_rq[53]:/.exec_clock
    270817 ±22%     -66.3%      91254 ±13%  TOTAL sched_debug.cpu#63.avg_idle
     15169 ±10%    +196.9%      45031 ± 0%  TOTAL sched_debug.cfs_rq[36]:/.avg->runnable_avg_sum
       332 ±10%    +196.3%        984 ± 0%  TOTAL sched_debug.cfs_rq[36]:/.tg_runnable_contrib
         9 ±31%    +195.7%         27 ± 3%  TOTAL sched_debug.cfs_rq[54]:/.runnable_load_avg
      1552 ±27%    +175.8%       4281 ± 2%  TOTAL sched_debug.cpu#60.curr->pid
     49056 ± 1%    +199.2%     146799 ± 0%  TOTAL sched_debug.cfs_rq[33]:/.exec_clock
      1431 ±22%    +201.8%       4321 ± 1%  TOTAL sched_debug.cpu#34.curr->pid
     48712 ± 2%    +198.8%     145545 ± 0%  TOTAL sched_debug.cfs_rq[50]:/.exec_clock
     48311 ± 1%    +201.0%     145402 ± 0%  TOTAL sched_debug.cfs_rq[39]:/.exec_clock
       332 ± 9%    +201.0%        999 ± 0%  TOTAL sched_debug.cfs_rq[49]:/.tg_runnable_contrib
    226926 ±13%     -62.6%      84981 ±24%  TOTAL sched_debug.cpu#38.avg_idle
     15180 ± 9%    +201.5%      45763 ± 0%  TOTAL sched_debug.cfs_rq[49]:/.avg->runnable_avg_sum
     48719 ± 2%    +198.5%     145438 ± 0%  TOTAL sched_debug.cfs_rq[56]:/.exec_clock
   1795031 ± 0%     -66.4%     602823 ± 1%  TOTAL sched_debug.cpu#40.nr_switches
     48496 ± 2%    +199.9%     145443 ± 0%  TOTAL sched_debug.cfs_rq[58]:/.exec_clock
     15260 ±11%    +195.2%      45042 ± 0%  TOTAL sched_debug.cfs_rq[40]:/.avg->runnable_avg_sum
    285472 ±25%     -68.5%      89860 ±28%  TOTAL sched_debug.cpu#15.avg_idle
       334 ±11%    +194.3%        982 ± 0%  TOTAL sched_debug.cfs_rq[40]:/.tg_runnable_contrib
     49933 ± 2%    +194.4%     147012 ± 0%  TOTAL sched_debug.cfs_rq[57]:/.exec_clock
     49101 ± 2%    +196.1%     145384 ± 0%  TOTAL sched_debug.cfs_rq[37]:/.exec_clock
     49003 ± 1%    +196.8%     145444 ± 0%  TOTAL sched_debug.cfs_rq[35]:/.exec_clock
      1390 ±22%    +206.7%       4265 ± 0%  TOTAL sched_debug.cpu#37.curr->pid
     48581 ± 3%    +199.4%     145429 ± 0%  TOTAL sched_debug.cfs_rq[48]:/.exec_clock
      1630 ±25%    +160.5%       4247 ± 2%  TOTAL sched_debug.cpu#47.curr->pid
     49123 ± 2%    +196.0%     145404 ± 0%  TOTAL sched_debug.cfs_rq[46]:/.exec_clock
    266050 ±20%     -59.4%     107922 ±24%  TOTAL sched_debug.cpu#52.avg_idle
     14978 ±10%    +200.9%      45075 ± 0%  TOTAL sched_debug.cfs_rq[45]:/.avg->runnable_avg_sum
      1417 ±21%    +204.3%       4312 ± 2%  TOTAL sched_debug.cpu#39.curr->pid
        12 ±26%    +160.7%         31 ± 9%  TOTAL sched_debug.cpu#49.cpu_load[1]
       327 ±10%    +198.7%        978 ± 0%  TOTAL sched_debug.cfs_rq[45]:/.tg_runnable_contrib
    270121 ±17%     -61.7%     103375 ±20%  TOTAL sched_debug.cpu#32.avg_idle
     50428 ± 2%    +189.1%     145774 ± 0%  TOTAL sched_debug.cfs_rq[42]:/.exec_clock
      1574 ±34%    +166.5%       4194 ± 1%  TOTAL sched_debug.cpu#45.curr->pid
     50663 ± 1%    +189.6%     146717 ± 0%  TOTAL sched_debug.cfs_rq[49]:/.exec_clock
     49273 ± 2%    +195.3%     145509 ± 0%  TOTAL sched_debug.cfs_rq[34]:/.exec_clock
     15310 ±13%    +194.0%      45007 ± 0%  TOTAL sched_debug.cfs_rq[60]:/.avg->runnable_avg_sum
    236078 ±11%     -65.9%      80473 ±14%  TOTAL sched_debug.cpu#41.avg_idle
     49673 ± 2%    +192.7%     145415 ± 0%  TOTAL sched_debug.cfs_rq[38]:/.exec_clock
         8 ±26%    +235.0%         26 ± 5%  TOTAL sched_debug.cpu#59.cpu_load[0]
     50433 ± 2%    +188.4%     145447 ± 0%  TOTAL sched_debug.cfs_rq[45]:/.exec_clock
     15213 ± 5%    +196.2%      45068 ± 0%  TOTAL sched_debug.cfs_rq[52]:/.avg->runnable_avg_sum
       334 ±13%    +193.4%        980 ± 0%  TOTAL sched_debug.cfs_rq[60]:/.tg_runnable_contrib
     50577 ± 1%    +187.9%     145609 ± 0%  TOTAL sched_debug.cfs_rq[43]:/.exec_clock
    249630 ±15%     -65.3%      86741 ± 5%  TOTAL sched_debug.cpu#8.avg_idle
       332 ± 5%    +194.9%        980 ± 0%  TOTAL sched_debug.cfs_rq[52]:/.tg_runnable_contrib
    235932 ± 9%     -62.3%      88954 ±19%  TOTAL sched_debug.cpu#34.avg_idle
     50890 ± 2%    +185.7%     145384 ± 0%  TOTAL sched_debug.cfs_rq[47]:/.exec_clock
     50930 ± 1%    +185.7%     145517 ± 0%  TOTAL sched_debug.cfs_rq[40]:/.exec_clock
     14328 ±16%    +214.3%      45037 ± 0%  TOTAL sched_debug.cfs_rq[53]:/.avg->runnable_avg_sum
     16062 ± 9%    +180.4%      45031 ± 0%  TOTAL sched_debug.cfs_rq[47]:/.avg->runnable_avg_sum
     51840 ± 1%    +183.3%     146858 ± 0%  TOTAL sched_debug.cfs_rq[41]:/.exec_clock
      1499 ±20%    +187.0%       4301 ± 1%  TOTAL sched_debug.cpu#51.curr->pid
     50662 ± 2%    +187.3%     145570 ± 0%  TOTAL sched_debug.cfs_rq[44]:/.exec_clock
    269032 ± 7%     -66.8%      89230 ±17%  TOTAL sched_debug.cpu#59.avg_idle
       313 ±16%    +212.5%        980 ± 0%  TOTAL sched_debug.cfs_rq[53]:/.tg_runnable_contrib
     16303 ±17%    +176.2%      45035 ± 0%  TOTAL sched_debug.cfs_rq[54]:/.avg->runnable_avg_sum
       356 ±17%    +174.9%        979 ± 0%  TOTAL sched_debug.cfs_rq[54]:/.tg_runnable_contrib
       351 ± 9%    +178.0%        977 ± 0%  TOTAL sched_debug.cfs_rq[47]:/.tg_runnable_contrib
     15587 ±11%    +189.2%      45070 ± 0%  TOTAL sched_debug.cfs_rq[43]:/.avg->runnable_avg_sum
       341 ±11%    +187.8%        982 ± 0%  TOTAL sched_debug.cfs_rq[43]:/.tg_runnable_contrib
  99841369 ± 0%    +180.1%  2.796e+08 ± 1%  TOTAL numa-numastat.node2.local_node
  99843858 ± 0%    +180.1%  2.796e+08 ± 1%  TOTAL numa-numastat.node2.numa_hit
        10 ±17%    +196.2%         30 ± 3%  TOTAL sched_debug.cpu#57.cpu_load[4]
        11 ±12%    +189.1%         31 ± 8%  TOTAL sched_debug.cpu#41.cpu_load[4]
      1560 ±11%    +175.1%       4292 ± 1%  TOTAL sched_debug.cpu#59.curr->pid
  49642548 ± 0%    +177.8%  1.379e+08 ± 1%  TOTAL numa-vmstat.node2.numa_local
         8 ±28%    +220.5%         28 ± 4%  TOTAL sched_debug.cpu#43.cpu_load[0]
  49698361 ± 0%    +177.6%   1.38e+08 ± 1%  TOTAL numa-vmstat.node2.numa_hit
    219871 ±18%     -57.9%      92485 ±15%  TOTAL sched_debug.cpu#62.avg_idle
  50123901 ± 0%    +178.0%  1.393e+08 ± 1%  TOTAL numa-vmstat.node0.numa_local
  50127863 ± 0%    +177.9%  1.393e+08 ± 1%  TOTAL numa-vmstat.node0.numa_hit
     16250 ± 5%    +178.5%      45253 ± 0%  TOTAL sched_debug.cfs_rq[42]:/.avg->runnable_avg_sum
  49861680 ± 0%    +176.2%  1.377e+08 ± 1%  TOTAL numa-vmstat.node3.numa_local
       355 ± 5%    +177.0%        985 ± 0%  TOTAL sched_debug.cfs_rq[42]:/.tg_runnable_contrib
  49917445 ± 0%    +176.0%  1.378e+08 ± 1%  TOTAL numa-vmstat.node3.numa_hit
      1564 ±12%    +175.2%       4304 ± 2%  TOTAL sched_debug.cpu#54.curr->pid
  49911338 ± 0%    +175.9%  1.377e+08 ± 1%  TOTAL numa-vmstat.node1.numa_local
    248524 ±15%     -58.0%     104482 ±29%  TOTAL sched_debug.cpu#46.avg_idle
  49966794 ± 0%    +175.7%  1.377e+08 ± 1%  TOTAL numa-vmstat.node1.numa_hit
      1429 ±20%    +198.6%       4269 ± 0%  TOTAL sched_debug.cpu#44.curr->pid
     16484 ± 8%    +175.8%      45465 ± 0%  TOTAL sched_debug.cfs_rq[41]:/.avg->runnable_avg_sum
       361 ± 8%    +174.8%        992 ± 0%  TOTAL sched_debug.cfs_rq[41]:/.tg_runnable_contrib
      1453 ±23%    +197.1%       4318 ± 1%  TOTAL sched_debug.cpu#57.curr->pid
      1424 ±28%    +213.8%       4470 ± 5%  TOTAL sched_debug.cpu#63.curr->pid
      1420 ±23%    +201.2%       4278 ± 3%  TOTAL sched_debug.cpu#58.curr->pid
    211265 ±21%     -57.9%      88934 ±31%  TOTAL sched_debug.cpu#39.avg_idle
        37 ±32%    +129.8%         86 ±10%  TOTAL sched_debug.cfs_rq[3]:/.tg_load_contrib
   2788857 ± 1%    +163.2%    7340919 ± 1%  TOTAL sched_debug.cfs_rq[16]:/.min_vruntime
    224622 ±17%     -59.3%      91521 ±15%  TOTAL sched_debug.cpu#35.avg_idle
   2764205 ± 1%    +161.1%    7216910 ± 1%  TOTAL sched_debug.cfs_rq[8]:/.min_vruntime
   2819278 ± 1%    +156.7%    7238266 ± 1%  TOTAL sched_debug.cfs_rq[10]:/.min_vruntime
   2837162 ± 1%    +159.3%    7357411 ± 1%  TOTAL sched_debug.cfs_rq[24]:/.min_vruntime
   2817770 ± 1%    +160.8%    7349390 ± 1%  TOTAL sched_debug.cfs_rq[17]:/.min_vruntime
        12 ±31%    +160.0%         31 ± 4%  TOTAL sched_debug.cpu#33.cpu_load[3]
   2839736 ± 1%    +154.5%    7225910 ± 1%  TOTAL sched_debug.cfs_rq[13]:/.min_vruntime
        37 ±38%     +86.6%         69 ±45%  TOTAL sched_debug.cfs_rq[27]:/.tg_load_contrib
      1826 ±15%    +133.4%       4262 ± 0%  TOTAL sched_debug.cpu#36.curr->pid
   2880692 ± 3%    +154.0%    7316909 ± 1%  TOTAL sched_debug.cfs_rq[0]:/.min_vruntime
      1543 ±16%    +175.8%       4257 ± 0%  TOTAL sched_debug.cpu#46.curr->pid
   2903723 ± 1%    +153.0%    7346423 ± 1%  TOTAL sched_debug.cfs_rq[20]:/.min_vruntime
   2853451 ± 1%    +156.8%    7328372 ± 1%  TOTAL sched_debug.cfs_rq[18]:/.min_vruntime
   2881811 ± 2%    +154.8%    7342530 ± 0%  TOTAL sched_debug.cfs_rq[25]:/.min_vruntime
      3.22 ± 1%    +149.8%       8.05 ± 3%  TOTAL perf-profile.cpu-cycles.copy_user_generic_string.sctp_user_addto_chunk.sctp_datamsg_from_user.sctp_sendmsg.inet_sendmsg
   2848635 ± 1%    +153.7%    7225776 ± 1%  TOTAL sched_debug.cfs_rq[12]:/.min_vruntime
   2878875 ± 2%    +155.2%    7346322 ± 1%  TOTAL sched_debug.cfs_rq[2]:/.min_vruntime
    222782 ±27%     -61.7%      85268 ± 9%  TOTAL sched_debug.cpu#56.avg_idle
   2831503 ± 1%    +154.5%    7205461 ± 1%  TOTAL sched_debug.cfs_rq[11]:/.min_vruntime
   2835663 ± 1%    +154.6%    7218527 ± 1%  TOTAL sched_debug.cfs_rq[9]:/.min_vruntime
      1704 ±26%    +152.9%       4309 ± 1%  TOTAL sched_debug.cpu#41.curr->pid
        12 ±35%    +130.6%         28 ± 2%  TOTAL sched_debug.cfs_rq[48]:/.load
   2844115 ± 0%    +154.4%    7235577 ± 1%  TOTAL sched_debug.cfs_rq[15]:/.min_vruntime
      1665 ±28%    +159.1%       4315 ± 1%  TOTAL sched_debug.cpu#33.curr->pid
   2892891 ± 1%    +154.2%    7353510 ± 1%  TOTAL sched_debug.cfs_rq[19]:/.min_vruntime
   2901579 ± 2%    +152.7%    7332677 ± 1%  TOTAL sched_debug.cfs_rq[22]:/.min_vruntime
   2896475 ± 2%    +154.3%    7366784 ± 0%  TOTAL sched_debug.cfs_rq[23]:/.min_vruntime
   2911824 ± 1%    +151.7%    7327805 ± 1%  TOTAL sched_debug.cfs_rq[1]:/.min_vruntime
   2966470 ± 1%    +147.9%    7354082 ± 1%  TOTAL sched_debug.cfs_rq[29]:/.min_vruntime
   2884101 ± 0%    +150.3%    7219627 ± 1%  TOTAL sched_debug.cfs_rq[14]:/.min_vruntime
   2925842 ± 1%    +151.9%    7369360 ± 1%  TOTAL sched_debug.cfs_rq[31]:/.min_vruntime
   2902721 ± 1%    +152.7%    7334172 ± 1%  TOTAL sched_debug.cfs_rq[21]:/.min_vruntime
   2924791 ± 2%    +150.8%    7336302 ± 1%  TOTAL sched_debug.cfs_rq[26]:/.min_vruntime
        11 ±26%    +154.2%         30 ± 2%  TOTAL sched_debug.cpu#57.cpu_load[3]
   2910713 ± 1%    +151.6%    7322791 ± 1%  TOTAL sched_debug.cfs_rq[3]:/.min_vruntime
   2952231 ± 2%    +149.6%    7369935 ± 0%  TOTAL sched_debug.cfs_rq[27]:/.min_vruntime
      1327 ±42%    +219.3%       4239 ± 0%  TOTAL sched_debug.cpu#40.curr->pid
   2975600 ± 0%    +146.9%    7348061 ± 1%  TOTAL sched_debug.cfs_rq[28]:/.min_vruntime
   2927020 ± 2%    +150.3%    7326407 ± 1%  TOTAL sched_debug.cfs_rq[5]:/.min_vruntime
   2937147 ± 0%    +148.1%    7287431 ± 1%  TOTAL sched_debug.cfs_rq[7]:/.min_vruntime
        12 ±16%    +151.6%         32 ±12%  TOTAL sched_debug.cpu#49.cpu_load[0]
   2972203 ± 1%    +146.7%    7331240 ± 0%  TOTAL sched_debug.cfs_rq[30]:/.min_vruntime
   2917430 ± 1%    +149.9%    7291729 ± 1%  TOTAL sched_debug.cfs_rq[4]:/.min_vruntime
         9 ±19%    +177.6%         27 ± 4%  TOTAL sched_debug.cfs_rq[60]:/.runnable_load_avg
   2903992 ± 2%    +150.8%    7282050 ± 1%  TOTAL sched_debug.cfs_rq[6]:/.min_vruntime
        37 ±31%    +113.3%         80 ±23%  TOTAL sched_debug.cfs_rq[29]:/.tg_load_contrib
    230273 ±17%     -51.7%     111201 ±32%  TOTAL sched_debug.cpu#50.avg_idle
      0.70 ± 1%    +140.3%       1.68 ± 3%  TOTAL perf-profile.cpu-cycles.get_page_from_freelist.__alloc_pages_nodemask.kmalloc_large_node.__kmalloc_node_track_caller.__kmalloc_reserve
    206691 ±15%     -57.8%      87201 ± 8%  TOTAL sched_debug.cpu#61.avg_idle
    259396 ± 5%    +122.7%     577766 ± 1%  TOTAL sched_debug.cpu#62.ttwu_count
        12 ±34%    +134.9%         29 ± 3%  TOTAL sched_debug.cpu#57.cpu_load[2]
    269666 ±16%     -57.7%     114191 ±17%  TOTAL sched_debug.cpu#36.avg_idle
        14 ±36%    +119.7%         31 ± 5%  TOTAL sched_debug.cpu#33.cpu_load[2]
      1706 ±16%    +149.8%       4262 ± 0%  TOTAL sched_debug.cpu#38.curr->pid
    242840 ± 4%     -54.8%     109685 ±27%  TOTAL sched_debug.cpu#45.avg_idle
    262430 ± 5%    +120.6%     578956 ± 1%  TOTAL sched_debug.cpu#60.ttwu_count
    139692 ± 4%     -54.6%      63480 ± 2%  TOTAL proc-vmstat.numa_hint_faults
        12 ±19%    +148.4%         31 ± 9%  TOTAL sched_debug.cpu#41.cpu_load[3]
    264847 ± 9%    +116.3%     572756 ± 2%  TOTAL sched_debug.cpu#61.ttwu_count
      0.50 ± 7%    +122.0%       1.11 ±12%  TOTAL perf-profile.cpu-cycles.sctp_sendmsg.inet_sendmsg.sock_sendmsg.___sys_sendmsg.__sys_sendmsg
        30 ±12%    +115.3%         64 ±46%  TOTAL sched_debug.cfs_rq[19]:/.tg_load_contrib
        11 ±28%    +139.0%         28 ± 5%  TOTAL sched_debug.cfs_rq[42]:/.runnable_load_avg
    263882 ± 5%    +119.9%     580288 ± 1%  TOTAL sched_debug.cpu#59.ttwu_count
        15 ±41%    +105.3%         30 ± 7%  TOTAL sched_debug.cpu#33.cpu_load[1]
        15 ±41%     +97.5%         31 ±10%  TOTAL sched_debug.cpu#33.cpu_load[0]
    232649 ± 8%     -53.9%     107165 ± 9%  TOTAL sched_debug.cpu#37.avg_idle
      1870 ±19%    +126.4%       4234 ± 0%  TOTAL sched_debug.cpu#42.curr->pid
    276936 ± 2%    +110.2%     582259 ± 1%  TOTAL sched_debug.cpu#32.ttwu_count
        32 ±13%    +116.7%         70 ±31%  TOTAL sched_debug.cfs_rq[24]:/.tg_load_contrib
      3.48 ± 0%    +111.1%       7.35 ± 1%  TOTAL perf-profile.cpu-cycles.copy_user_generic_string.skb_copy_datagram_iovec.sctp_recvmsg.sock_common_recvmsg.sock_recvmsg
    276884 ± 2%    +111.1%     584406 ± 1%  TOTAL sched_debug.cpu#33.ttwu_count
    278248 ± 4%    +107.0%     575968 ± 2%  TOTAL sched_debug.cpu#55.ttwu_count
    276813 ± 5%    +109.4%     579782 ± 1%  TOTAL sched_debug.cpu#51.ttwu_count
    274908 ± 2%    +110.4%     578527 ± 1%  TOTAL sched_debug.cpu#52.ttwu_count
    276883 ± 5%    +108.8%     578248 ± 1%  TOTAL sched_debug.cpu#53.ttwu_count
        13 ±48%    +127.3%         30 ± 8%  TOTAL sched_debug.cpu#57.cpu_load[0]
        12 ±42%    +130.2%         29 ± 3%  TOTAL sched_debug.cpu#57.cpu_load[1]
        12 ±47%    +159.0%         31 ±13%  TOTAL sched_debug.cfs_rq[41]:/.runnable_load_avg
     96427 ± 3%     -51.4%      46839 ± 0%  TOTAL proc-vmstat.numa_hint_faults_local
        14 ±23%    +113.5%         31 ±11%  TOTAL sched_debug.cpu#41.cpu_load[2]
    281119 ± 3%    +105.5%     577638 ± 1%  TOTAL sched_debug.cpu#63.ttwu_count
    279804 ± 7%    +105.6%     575372 ± 1%  TOTAL sched_debug.cpu#54.ttwu_count
    285799 ± 3%    +104.0%     583075 ± 1%  TOTAL sched_debug.cpu#39.ttwu_count
    291642 ±21%     -55.2%     130660 ±18%  TOTAL sched_debug.cpu#40.avg_idle
       140 ±31%     -57.1%         60 ±44%  TOTAL sched_debug.cfs_rq[43]:/.tg_load_contrib
     13879 ± 0%    +104.3%      28362 ± 0%  TOTAL proc-vmstat.pgactivate
    251355 ±27%     -49.9%     126023 ±33%  TOTAL sched_debug.cpu#42.avg_idle
    284829 ± 5%    +104.2%     581719 ± 1%  TOTAL sched_debug.cpu#36.ttwu_count
    287754 ± 5%    +101.5%     579892 ± 1%  TOTAL sched_debug.cpu#57.ttwu_count
    290676 ± 4%     +98.4%     576782 ± 1%  TOTAL sched_debug.cpu#50.ttwu_count
        15 ±34%     +92.4%         30 ±11%  TOTAL sched_debug.cfs_rq[33]:/.runnable_load_avg
        28 ±12%    +100.7%         57 ±30%  TOTAL sched_debug.cfs_rq[31]:/.tg_load_contrib
    287909 ± 4%    +101.5%     580220 ± 0%  TOTAL sched_debug.cpu#56.ttwu_count
    281438 ± 6%    +105.0%     576839 ± 1%  TOTAL sched_debug.cpu#58.ttwu_count
    223246 ±22%     -59.3%      90847 ±16%  TOTAL sched_debug.cpu#57.avg_idle
    292834 ± 2%     +98.9%     582416 ± 1%  TOTAL sched_debug.cpu#35.ttwu_count
    299007 ± 1%     +93.8%     579492 ± 1%  TOTAL sched_debug.cpu#49.ttwu_count
    296051 ± 3%     +96.4%     581581 ± 0%  TOTAL sched_debug.cpu#37.ttwu_count
    155938 ± 3%     -48.9%      79614 ± 2%  TOTAL proc-vmstat.numa_pte_updates
    292818 ± 4%     +99.2%     583153 ± 1%  TOTAL sched_debug.cpu#34.ttwu_count
    291339 ± 3%     +97.8%     576203 ± 1%  TOTAL sched_debug.cpu#46.ttwu_count
   1173228 ±19%     -48.8%     601063 ± 1%  TOTAL sched_debug.cpu#8.sched_count
    291043 ± 6%     +98.2%     576756 ± 1%  TOTAL sched_debug.cpu#48.ttwu_count
     32644 ± 0%     +92.8%      62936 ± 0%  TOTAL sched_debug.cfs_rq[0]:/.tg->runnable_avg
     32648 ± 0%     +92.8%      62937 ± 0%  TOTAL sched_debug.cfs_rq[1]:/.tg->runnable_avg
     32650 ± 0%     +92.8%      62937 ± 0%  TOTAL sched_debug.cfs_rq[2]:/.tg->runnable_avg
     32654 ± 0%     +92.7%      62937 ± 0%  TOTAL sched_debug.cfs_rq[3]:/.tg->runnable_avg
     32665 ± 0%     +92.7%      62937 ± 0%  TOTAL sched_debug.cfs_rq[4]:/.tg->runnable_avg
     32674 ± 0%     +92.6%      62938 ± 0%  TOTAL sched_debug.cfs_rq[5]:/.tg->runnable_avg
     32689 ± 0%     +92.5%      62938 ± 0%  TOTAL sched_debug.cfs_rq[8]:/.tg->runnable_avg
     32678 ± 0%     +92.6%      62937 ± 0%  TOTAL sched_debug.cfs_rq[6]:/.tg->runnable_avg
     32690 ± 0%     +92.5%      62938 ± 0%  TOTAL sched_debug.cfs_rq[10]:/.tg->runnable_avg
     32691 ± 0%     +92.5%      62938 ± 0%  TOTAL sched_debug.cfs_rq[9]:/.tg->runnable_avg
     32686 ± 0%     +92.6%      62937 ± 0%  TOTAL sched_debug.cfs_rq[7]:/.tg->runnable_avg
     32692 ± 0%     +92.5%      62938 ± 0%  TOTAL sched_debug.cfs_rq[11]:/.tg->runnable_avg
     32696 ± 0%     +92.5%      62938 ± 0%  TOTAL sched_debug.cfs_rq[12]:/.tg->runnable_avg
     32701 ± 0%     +92.5%      62938 ± 0%  TOTAL sched_debug.cfs_rq[13]:/.tg->runnable_avg
     32704 ± 0%     +92.4%      62938 ± 0%  TOTAL sched_debug.cfs_rq[14]:/.tg->runnable_avg
     32702 ± 0%     +92.5%      62938 ± 0%  TOTAL sched_debug.cfs_rq[15]:/.tg->runnable_avg
     32717 ± 0%     +92.4%      62937 ± 0%  TOTAL sched_debug.cfs_rq[18]:/.tg->runnable_avg
     32707 ± 0%     +92.4%      62938 ± 0%  TOTAL sched_debug.cfs_rq[16]:/.tg->runnable_avg
     32713 ± 0%     +92.4%      62937 ± 0%  TOTAL sched_debug.cfs_rq[17]:/.tg->runnable_avg
     32722 ± 0%     +92.3%      62938 ± 0%  TOTAL sched_debug.cfs_rq[19]:/.tg->runnable_avg
     32727 ± 0%     +92.3%      62938 ± 0%  TOTAL sched_debug.cfs_rq[20]:/.tg->runnable_avg
     32732 ± 0%     +92.3%      62938 ± 0%  TOTAL sched_debug.cfs_rq[21]:/.tg->runnable_avg
     32740 ± 0%     +92.2%      62938 ± 0%  TOTAL sched_debug.cfs_rq[24]:/.tg->runnable_avg
     32739 ± 0%     +92.2%      62938 ± 0%  TOTAL sched_debug.cfs_rq[23]:/.tg->runnable_avg
     32743 ± 0%     +92.2%      62938 ± 0%  TOTAL sched_debug.cfs_rq[26]:/.tg->runnable_avg
     32739 ± 0%     +92.2%      62938 ± 0%  TOTAL sched_debug.cfs_rq[22]:/.tg->runnable_avg
     32743 ± 0%     +92.2%      62937 ± 0%  TOTAL sched_debug.cfs_rq[27]:/.tg->runnable_avg
     32746 ± 0%     +92.2%      62938 ± 0%  TOTAL sched_debug.cfs_rq[25]:/.tg->runnable_avg
     32751 ± 0%     +92.2%      62938 ± 0%  TOTAL sched_debug.cfs_rq[29]:/.tg->runnable_avg
     32751 ± 0%     +92.2%      62938 ± 0%  TOTAL sched_debug.cfs_rq[30]:/.tg->runnable_avg
     32747 ± 0%     +92.2%      62938 ± 0%  TOTAL sched_debug.cfs_rq[28]:/.tg->runnable_avg
     32752 ± 0%     +92.2%      62939 ± 0%  TOTAL sched_debug.cfs_rq[31]:/.tg->runnable_avg
     32752 ± 0%     +92.2%      62939 ± 0%  TOTAL sched_debug.cfs_rq[32]:/.tg->runnable_avg
     32759 ± 0%     +92.1%      62939 ± 0%  TOTAL sched_debug.cfs_rq[33]:/.tg->runnable_avg
     32770 ± 0%     +92.1%      62939 ± 0%  TOTAL sched_debug.cfs_rq[37]:/.tg->runnable_avg
     32757 ± 0%     +92.1%      62939 ± 0%  TOTAL sched_debug.cfs_rq[34]:/.tg->runnable_avg
     32766 ± 0%     +92.1%      62939 ± 0%  TOTAL sched_debug.cfs_rq[36]:/.tg->runnable_avg
     32765 ± 0%     +92.1%      62939 ± 0%  TOTAL sched_debug.cfs_rq[35]:/.tg->runnable_avg
     32767 ± 0%     +92.1%      62939 ± 0%  TOTAL sched_debug.cfs_rq[38]:/.tg->runnable_avg
     32774 ± 0%     +92.0%      62939 ± 0%  TOTAL sched_debug.cfs_rq[41]:/.tg->runnable_avg
     32775 ± 0%     +92.0%      62939 ± 0%  TOTAL sched_debug.cfs_rq[42]:/.tg->runnable_avg
     32774 ± 0%     +92.0%      62939 ± 0%  TOTAL sched_debug.cfs_rq[39]:/.tg->runnable_avg
     32779 ± 0%     +92.0%      62939 ± 0%  TOTAL sched_debug.cfs_rq[43]:/.tg->runnable_avg
     32773 ± 0%     +92.0%      62939 ± 0%  TOTAL sched_debug.cfs_rq[40]:/.tg->runnable_avg
     32822 ± 0%     +91.8%      62940 ± 0%  TOTAL sched_debug.cfs_rq[63]:/.tg->runnable_avg
     32820 ± 0%     +91.8%      62940 ± 0%  TOTAL sched_debug.cfs_rq[62]:/.tg->runnable_avg
     32793 ± 0%     +91.9%      62939 ± 0%  TOTAL sched_debug.cfs_rq[47]:/.tg->runnable_avg
     32796 ± 0%     +91.9%      62940 ± 0%  TOTAL sched_debug.cfs_rq[48]:/.tg->runnable_avg
     32818 ± 0%     +91.8%      62940 ± 0%  TOTAL sched_debug.cfs_rq[61]:/.tg->runnable_avg
     32821 ± 0%     +91.8%      62940 ± 0%  TOTAL sched_debug.cfs_rq[60]:/.tg->runnable_avg
     32790 ± 0%     +91.9%      62939 ± 0%  TOTAL sched_debug.cfs_rq[44]:/.tg->runnable_avg
     32791 ± 0%     +91.9%      62939 ± 0%  TOTAL sched_debug.cfs_rq[46]:/.tg->runnable_avg
     32819 ± 0%     +91.8%      62941 ± 0%  TOTAL sched_debug.cfs_rq[59]:/.tg->runnable_avg
     32793 ± 0%     +91.9%      62939 ± 0%  TOTAL sched_debug.cfs_rq[45]:/.tg->runnable_avg
     32812 ± 0%     +91.8%      62940 ± 0%  TOTAL sched_debug.cfs_rq[56]:/.tg->runnable_avg
     32817 ± 0%     +91.8%      62940 ± 0%  TOTAL sched_debug.cfs_rq[57]:/.tg->runnable_avg
     32809 ± 0%     +91.8%      62940 ± 0%  TOTAL sched_debug.cfs_rq[52]:/.tg->runnable_avg
     32811 ± 0%     +91.8%      62940 ± 0%  TOTAL sched_debug.cfs_rq[55]:/.tg->runnable_avg
     32801 ± 0%     +91.9%      62940 ± 0%  TOTAL sched_debug.cfs_rq[49]:/.tg->runnable_avg
     32822 ± 0%     +91.8%      62941 ± 0%  TOTAL sched_debug.cfs_rq[58]:/.tg->runnable_avg
     32808 ± 0%     +91.8%      62940 ± 0%  TOTAL sched_debug.cfs_rq[54]:/.tg->runnable_avg
     32812 ± 0%     +91.8%      62940 ± 0%  TOTAL sched_debug.cfs_rq[53]:/.tg->runnable_avg
     32809 ± 0%     +91.8%      62940 ± 0%  TOTAL sched_debug.cfs_rq[51]:/.tg->runnable_avg
     32804 ± 0%     +91.9%      62940 ± 0%  TOTAL sched_debug.cfs_rq[50]:/.tg->runnable_avg
    309179 ± 2%     +87.8%     580793 ± 1%  TOTAL sched_debug.cpu#41.ttwu_count
        81 ±44%     -62.2%         30 ± 4%  TOTAL sched_debug.cfs_rq[62]:/.tg_load_contrib
   5254994 ± 0%     +89.6%    9963624 ± 0%  TOTAL softirqs.TIMER
      3.69 ± 1%     +86.0%       6.87 ± 4%  TOTAL perf-profile.cpu-cycles.memcpy.sctp_outq_flush.sctp_outq_uncork.sctp_cmd_interpreter.sctp_do_sm
    308199 ± 4%     +86.7%     575358 ± 1%  TOTAL sched_debug.cpu#42.ttwu_count
    305544 ± 4%     +90.6%     582390 ± 1%  TOTAL sched_debug.cpu#38.ttwu_count
    309614 ± 1%     +86.5%     577365 ± 1%  TOTAL sched_debug.cpu#43.ttwu_count
        14 ±33%     +98.6%         28 ± 2%  TOTAL sched_debug.cpu#35.load
    309719 ± 5%     +85.8%     575506 ± 1%  TOTAL sched_debug.cpu#45.ttwu_count
   1081161 ± 1%     -46.1%     582374 ± 1%  TOTAL sched_debug.cpu#29.ttwu_count
    316152 ± 3%     +82.7%     577646 ± 1%  TOTAL sched_debug.cpu#40.ttwu_count
        27 ± 8%    +107.9%         57 ±49%  TOTAL sched_debug.cfs_rq[9]:/.tg_load_contrib
   1083438 ± 0%     -45.9%     586638 ± 0%  TOTAL sched_debug.cpu#7.ttwu_count
    315864 ± 5%     +82.6%     576755 ± 1%  TOTAL sched_debug.cpu#47.ttwu_count
        31 ± 7%     +69.2%         52 ±36%  TOTAL sched_debug.cfs_rq[13]:/.tg_load_contrib
   1084154 ± 1%     -45.7%     588437 ± 1%  TOTAL sched_debug.cpu#30.ttwu_count
    311962 ± 5%     +84.9%     576905 ± 1%  TOTAL sched_debug.cpu#44.ttwu_count
   1060523 ± 1%     -45.1%     582533 ± 1%  TOTAL sched_debug.cpu#21.ttwu_count
   1054699 ± 1%     -45.1%     579000 ± 1%  TOTAL sched_debug.cpu#14.ttwu_count
        17 ±22%     +70.8%         30 ± 2%  TOTAL sched_debug.cpu#8.cpu_load[4]
        15 ±26%    +103.9%         31 ±12%  TOTAL sched_debug.cpu#41.cpu_load[1]
        28 ± 4%     +85.4%         53 ±38%  TOTAL sched_debug.cfs_rq[30]:/.tg_load_contrib
   1075268 ± 1%     -45.5%     586405 ± 1%  TOTAL sched_debug.cpu#28.ttwu_count
   1059908 ± 2%     -44.9%     584032 ± 1%  TOTAL sched_debug.cpu#31.ttwu_count
   1068780 ± 1%     -45.5%     582915 ± 1%  TOTAL sched_debug.cpu#27.ttwu_count
   1068310 ± 0%     -44.9%     588354 ± 1%  TOTAL sched_debug.cpu#5.ttwu_count
   1055705 ± 2%     -45.0%     580123 ± 2%  TOTAL sched_debug.cpu#23.ttwu_count
   1054570 ± 1%     -44.7%     583394 ± 1%  TOTAL sched_debug.cpu#20.ttwu_count
    254366 ±43%     -59.3%     103437 ±13%  TOTAL sched_debug.cpu#54.avg_idle
   1063589 ± 2%     -44.7%     587718 ± 1%  TOTAL sched_debug.cpu#6.ttwu_count
   1058139 ± 3%     -44.6%     585929 ± 1%  TOTAL sched_debug.cpu#22.ttwu_count
   1067145 ± 1%     -44.9%     588220 ± 1%  TOTAL sched_debug.cpu#4.ttwu_count
   1057669 ± 0%     -44.3%     588982 ± 1%  TOTAL sched_debug.cpu#1.ttwu_count
   1059310 ± 1%     -44.3%     589806 ± 1%  TOTAL sched_debug.cpu#3.ttwu_count
   1043843 ± 1%     -44.2%     581948 ± 1%  TOTAL sched_debug.cpu#18.ttwu_count
   1031973 ± 1%     -43.9%     579237 ± 1%  TOTAL sched_debug.cpu#11.ttwu_count
        18 ±24%     +70.7%         31 ±13%  TOTAL sched_debug.cpu#41.cpu_load[0]
   1047788 ± 2%     -44.3%     583196 ± 1%  TOTAL sched_debug.cpu#19.ttwu_count
   1044163 ± 2%     -44.2%     583017 ± 1%  TOTAL sched_debug.cpu#26.ttwu_count
   1288766 ±38%     -52.8%     608462 ± 1%  TOTAL sched_debug.cpu#1.sched_count
   1032898 ± 1%     -43.8%     579984 ± 1%  TOTAL sched_debug.cpu#13.ttwu_count
   1049631 ± 1%     -43.7%     590788 ± 0%  TOTAL sched_debug.cpu#2.ttwu_count
   1034514 ± 1%     -43.7%     582699 ± 1%  TOTAL sched_debug.cpu#15.ttwu_count
        16 ±20%     +75.0%         29 ± 4%  TOTAL sched_debug.cpu#9.cpu_load[0]
   1023516 ± 1%     -43.1%     582541 ± 1%  TOTAL sched_debug.cpu#17.ttwu_count
   1039799 ± 3%     -44.3%     579547 ± 2%  TOTAL sched_debug.cpu#25.ttwu_count
   1302779 ±36%     -54.0%     598642 ± 1%  TOTAL sched_debug.cpu#9.sched_count
   1012139 ± 0%     -42.5%     581867 ± 1%  TOTAL sched_debug.cpu#9.ttwu_count
   1032088 ± 2%     -43.7%     581367 ± 1%  TOTAL sched_debug.cpu#12.ttwu_count
   1015497 ± 1%     -42.6%     582939 ± 1%  TOTAL sched_debug.cpu#16.ttwu_count
   1023139 ± 1%     -43.1%     582533 ± 1%  TOTAL sched_debug.cpu#10.ttwu_count
   1039034 ± 1%     -42.7%     595606 ± 1%  TOTAL sched_debug.cpu#0.ttwu_count
   1125131 ±22%     -46.7%     600109 ± 2%  TOTAL sched_debug.cpu#12.sched_count
      9019 ± 6%     -42.0%       5232 ±10%  TOTAL proc-vmstat.pgmigrate_success
      9019 ± 6%     -42.0%       5232 ±10%  TOTAL proc-vmstat.numa_pages_migrated
   1020221 ± 2%     -42.6%     586085 ± 1%  TOTAL sched_debug.cpu#24.ttwu_count
        18 ± 9%     +69.2%         30 ± 2%  TOTAL sched_debug.cpu#9.cpu_load[4]
        19 ±14%     +84.8%         36 ±34%  TOTAL sched_debug.cfs_rq[38]:/.load
        30 ±12%     +56.7%         47 ±18%  TOTAL sched_debug.cfs_rq[25]:/.tg_load_contrib
        17 ±13%     +62.5%         28 ± 3%  TOTAL sched_debug.cpu#1.cpu_load[0]
        17 ±15%     +64.4%         28 ± 1%  TOTAL sched_debug.cpu#22.cpu_load[3]
        19 ±16%     +52.1%         29 ± 2%  TOTAL sched_debug.cpu#25.cpu_load[3]
    991690 ± 0%     -41.2%     582981 ± 1%  TOTAL sched_debug.cpu#8.ttwu_count
        18 ±12%     +60.2%         29 ± 4%  TOTAL sched_debug.cpu#15.cpu_load[4]
        18 ±22%     +66.7%         30 ± 2%  TOTAL sched_debug.cpu#8.cpu_load[3]
        17 ±11%     +72.4%         30 ± 2%  TOTAL sched_debug.cpu#9.cpu_load[3]
        21 ±26%     +70.5%         35 ±38%  TOTAL sched_debug.cpu#61.load
        19 ±25%     +56.8%         29 ± 2%  TOTAL sched_debug.cpu#43.load
        18 ±16%     +52.7%         27 ± 1%  TOTAL sched_debug.cpu#30.cpu_load[2]
        18 ±18%     +56.0%         28 ± 3%  TOTAL sched_debug.cpu#26.cpu_load[4]
        17 ±17%     +59.1%         28 ± 3%  TOTAL sched_debug.cpu#22.cpu_load[1]
        17 ± 4%     +58.0%         27 ± 4%  TOTAL sched_debug.cpu#3.cpu_load[0]
        17 ±16%     +55.1%         27 ± 3%  TOTAL sched_debug.cpu#22.cpu_load[0]
        18 ±19%     +56.7%         28 ± 4%  TOTAL sched_debug.cpu#26.cpu_load[3]
        17 ±17%     +64.0%         28 ± 2%  TOTAL sched_debug.cpu#22.cpu_load[2]
      2664 ±15%     +63.7%       4362 ± 1%  TOTAL sched_debug.cpu#20.curr->pid
        19 ±14%     +56.6%         31 ± 3%  TOTAL sched_debug.cfs_rq[45]:/.load
        71 ±43%     -52.1%         34 ±13%  TOTAL sched_debug.cfs_rq[47]:/.tg_load_contrib
     13306 ± 0%     +61.5%      21495 ± 0%  TOTAL proc-vmstat.nr_shmem
     53249 ± 0%     +61.5%      85996 ± 0%  TOTAL meminfo.Shmem
        18 ± 8%     +58.7%         29 ± 1%  TOTAL sched_debug.cpu#11.cpu_load[3]
        18 ±12%     +57.1%         28 ± 1%  TOTAL sched_debug.cpu#30.cpu_load[3]
        17 ±22%     +65.2%         29 ± 3%  TOTAL sched_debug.cfs_rq[9]:/.runnable_load_avg
        19 ± 6%     +54.7%         29 ± 1%  TOTAL sched_debug.cpu#23.cpu_load[4]
        17 ±15%     +69.8%         29 ± 3%  TOTAL sched_debug.cpu#9.cpu_load[2]
        18 ± 7%     +59.1%         29 ± 2%  TOTAL sched_debug.cpu#11.cpu_load[4]
        17 ±17%     +70.6%         29 ± 4%  TOTAL sched_debug.cpu#9.cpu_load[1]
        18 ± 8%     +55.4%         28 ± 3%  TOTAL sched_debug.cpu#1.cpu_load[1]
        18 ± 7%     +55.3%         29 ± 3%  TOTAL sched_debug.cpu#1.cpu_load[2]
        19 ±14%     +54.7%         29 ± 1%  TOTAL sched_debug.cpu#25.cpu_load[4]
        17 ±14%     +65.5%         28 ± 1%  TOTAL sched_debug.cpu#22.cpu_load[4]
        18 ± 9%     +53.8%         28 ± 4%  TOTAL sched_debug.cfs_rq[13]:/.runnable_load_avg
        19 ±11%     +54.7%         29 ± 5%  TOTAL sched_debug.cpu#15.cpu_load[3]
        18 ±12%     +55.4%         28 ± 1%  TOTAL sched_debug.cfs_rq[23]:/.runnable_load_avg
        18 ± 2%     +57.1%         28 ± 3%  TOTAL sched_debug.cpu#3.cpu_load[3]
        18 ± 2%     +60.4%         29 ± 2%  TOTAL sched_debug.cpu#3.cpu_load[4]
        17 ± 2%     +60.7%         28 ± 3%  TOTAL sched_debug.cpu#3.cpu_load[2]
      2777 ±10%     +60.7%       4463 ± 7%  TOTAL sched_debug.cpu#27.curr->pid
      2744 ±13%     +58.3%       4342 ± 3%  TOTAL sched_debug.cpu#22.curr->pid
    988578 ±10%     -38.9%     604501 ± 1%  TOTAL sched_debug.cpu#24.sched_count
   1083808 ±31%     -42.7%     621044 ± 7%  TOTAL sched_debug.cpu#6.sched_count
        19 ±12%     +53.1%         29 ± 4%  TOTAL sched_debug.cpu#19.cpu_load[3]
        19 ±13%     +53.6%         29 ± 4%  TOTAL sched_debug.cpu#19.cpu_load[4]
    966177 ±24%     -37.8%     600790 ± 2%  TOTAL sched_debug.cpu#27.sched_count
   1146030 ±46%     -46.9%     608122 ± 1%  TOTAL sched_debug.cpu#2.sched_count
        18 ± 3%     +55.6%         28 ± 3%  TOTAL sched_debug.cpu#3.cpu_load[1]
        19 ±19%     +41.4%         28 ± 2%  TOTAL sched_debug.cpu#5.cpu_load[0]
        17 ± 9%     +58.4%         28 ± 5%  TOTAL sched_debug.cfs_rq[3]:/.runnable_load_avg
        18 ±19%     +52.2%         28 ± 5%  TOTAL sched_debug.cpu#26.cpu_load[2]
        18 ±17%     +50.5%         28 ± 3%  TOTAL sched_debug.cpu#30.cpu_load[0]
        18 ±17%     +50.5%         27 ± 2%  TOTAL sched_debug.cpu#30.cpu_load[1]
        19 ±11%     +49.0%         28 ± 4%  TOTAL sched_debug.cpu#15.cpu_load[2]
        18 ±12%     +56.7%         28 ± 1%  TOTAL sched_debug.cpu#13.cpu_load[0]
        20 ±13%     +38.0%         27 ± 3%  TOTAL sched_debug.cfs_rq[31]:/.runnable_load_avg
        20 ±18%     +40.0%         28 ± 2%  TOTAL sched_debug.cpu#5.cpu_load[1]
        18 ±22%     +56.5%         28 ± 5%  TOTAL sched_debug.cpu#8.cpu_load[0]
        19 ±16%     +49.0%         28 ± 2%  TOTAL sched_debug.cpu#25.cpu_load[2]
   1040164 ±27%     -37.2%     653364 ±10%  TOTAL sched_debug.cpu#19.sched_count
        80 ±29%     -45.9%         43 ±26%  TOTAL sched_debug.cfs_rq[37]:/.tg_load_contrib
      2953 ±10%     +47.1%       4345 ± 2%  TOTAL sched_debug.cpu#14.curr->pid
    923395 ± 6%     -24.6%     695896 ±17%  TOTAL sched_debug.cpu#21.sched_count
      2800 ± 5%     +52.5%       4269 ± 1%  TOTAL sched_debug.cpu#11.curr->pid
      2746 ±15%     +57.0%       4312 ± 2%  TOTAL sched_debug.cpu#8.curr->pid
        20 ±12%     +44.1%         29 ± 1%  TOTAL sched_debug.cpu#5.cpu_load[4]
        18 ±13%     +58.1%         29 ± 1%  TOTAL sched_debug.cpu#13.cpu_load[2]
        18 ± 7%     +54.8%         28 ± 2%  TOTAL sched_debug.cpu#11.cpu_load[1]
        18 ±13%     +56.5%         28 ± 1%  TOTAL sched_debug.cpu#13.cpu_load[1]
        18 ±22%     +60.9%         29 ± 5%  TOTAL sched_debug.cpu#8.cpu_load[2]
        19 ± 6%     +47.9%         28 ± 2%  TOTAL sched_debug.cpu#7.cpu_load[2]
        19 ±12%     +51.6%         28 ± 2%  TOTAL sched_debug.cpu#11.cpu_load[0]
        18 ±12%     +58.7%         29 ± 3%  TOTAL sched_debug.cfs_rq[11]:/.runnable_load_avg
        18 ±11%     +62.6%         29 ± 5%  TOTAL sched_debug.cfs_rq[1]:/.runnable_load_avg
        19 ± 5%     +50.5%         28 ± 1%  TOTAL sched_debug.cpu#23.cpu_load[3]
        19 ±21%     +48.0%         29 ± 3%  TOTAL sched_debug.cpu#27.cpu_load[4]
        18 ±12%     +57.0%         29 ± 2%  TOTAL sched_debug.cpu#30.cpu_load[4]
        18 ± 7%     +53.2%         28 ± 2%  TOTAL sched_debug.cpu#11.cpu_load[2]
        19 ± 4%     +58.9%         30 ± 9%  TOTAL sched_debug.cfs_rq[2]:/.runnable_load_avg
   1101084 ±30%     -44.5%     611225 ± 2%  TOTAL sched_debug.cpu#4.sched_count
      2942 ±10%     +44.9%       4263 ± 2%  TOTAL sched_debug.cpu#7.curr->pid
      2905 ± 7%     +53.9%       4471 ± 2%  TOTAL sched_debug.cpu#23.curr->pid
   1035532 ±19%     -41.9%     601372 ± 1%  TOTAL sched_debug.cpu#15.sched_count
       663 ± 7%     +48.7%        987 ± 0%  TOTAL sched_debug.cfs_rq[8]:/.tg_runnable_contrib
     30382 ± 7%     +49.0%      45275 ± 0%  TOTAL sched_debug.cfs_rq[8]:/.avg->runnable_avg_sum
     30719 ± 7%     +46.5%      44998 ± 0%  TOTAL sched_debug.cfs_rq[15]:/.avg->runnable_avg_sum
       671 ± 7%     +45.8%        979 ± 0%  TOTAL sched_debug.cfs_rq[15]:/.tg_runnable_contrib
        20 ± 9%     +46.1%         29 ± 2%  TOTAL sched_debug.cpu#18.cpu_load[4]
        20 ± 6%     +45.6%         30 ± 4%  TOTAL sched_debug.cpu#1.cpu_load[4]
        19 ± 9%     +52.0%         29 ± 2%  TOTAL sched_debug.cpu#14.cpu_load[4]
        19 ±10%     +57.3%         30 ± 2%  TOTAL sched_debug.cpu#13.cpu_load[4]
        19 ±12%     +55.2%         29 ± 2%  TOTAL sched_debug.cpu#13.cpu_load[3]
        20 ±13%     +35.6%         27 ± 1%  TOTAL sched_debug.cfs_rq[22]:/.runnable_load_avg
        19 ±11%     +43.8%         27 ± 2%  TOTAL sched_debug.cfs_rq[5]:/.runnable_load_avg
        19 ±13%     +42.7%         27 ± 3%  TOTAL sched_debug.cpu#31.cpu_load[0]
      1045 ± 1%     +52.1%       1589 ± 7%  TOTAL numa-vmstat.node0.nr_alloc_batch
    737654 ± 1%     +50.3%    1108371 ± 0%  TOTAL softirqs.RCU
      2824 ±15%     +54.7%       4370 ± 1%  TOTAL sched_debug.cpu#31.curr->pid
      2900 ± 3%     +48.4%       4303 ± 1%  TOTAL sched_debug.cpu#17.curr->pid
      1117 ± 2%     +42.8%       1595 ± 6%  TOTAL numa-vmstat.node3.nr_alloc_batch
     31475 ± 6%     +43.6%      45205 ± 0%  TOTAL sched_debug.cfs_rq[26]:/.avg->runnable_avg_sum
      2819 ±13%     +51.7%       4276 ± 2%  TOTAL sched_debug.cpu#26.curr->pid
     31215 ± 4%     +44.3%      45055 ± 0%  TOTAL sched_debug.cfs_rq[30]:/.avg->runnable_avg_sum
     30904 ± 4%     +46.4%      45232 ± 0%  TOTAL sched_debug.cfs_rq[10]:/.avg->runnable_avg_sum
       675 ± 4%     +45.9%        985 ± 0%  TOTAL sched_debug.cfs_rq[10]:/.tg_runnable_contrib
       687 ± 6%     +42.7%        981 ± 0%  TOTAL sched_debug.cfs_rq[26]:/.tg_runnable_contrib
        19 ± 3%     +47.9%         28 ± 3%  TOTAL sched_debug.cpu#4.cpu_load[1]
        21 ±18%     +32.1%         28 ± 2%  TOTAL sched_debug.cpu#31.cpu_load[2]
        19 ±12%     +44.9%         28 ± 5%  TOTAL sched_debug.cpu#15.cpu_load[1]
        18 ±21%     +57.6%         29 ± 6%  TOTAL sched_debug.cpu#8.cpu_load[1]
        19 ±17%     +47.4%         28 ± 3%  TOTAL sched_debug.cpu#25.cpu_load[0]
        19 ± 3%     +49.0%         28 ± 6%  TOTAL sched_debug.cpu#4.cpu_load[0]
        20 ±13%     +42.0%         28 ± 1%  TOTAL sched_debug.cpu#5.cpu_load[3]
        20 ±10%     +41.0%         28 ± 4%  TOTAL sched_debug.cfs_rq[15]:/.runnable_load_avg
        19 ± 4%     +45.9%         28 ± 2%  TOTAL sched_debug.cpu#4.cpu_load[2]
        19 ± 7%     +47.4%         28 ± 2%  TOTAL sched_debug.cpu#23.cpu_load[2]
        18 ± 3%     +47.9%         27 ± 3%  TOTAL sched_debug.cpu#7.cpu_load[0]
        19 ±13%     +46.9%         28 ± 6%  TOTAL sched_debug.cpu#15.cpu_load[0]
        19 ± 8%     +44.8%         27 ± 2%  TOTAL sched_debug.cpu#23.cpu_load[1]
        19 ±16%     +45.4%         28 ± 1%  TOTAL sched_debug.cpu#25.cpu_load[1]
        19 ±11%     +40.8%         27 ± 1%  TOTAL sched_debug.cpu#23.cpu_load[0]
        20 ±14%     +39.6%         28 ± 1%  TOTAL sched_debug.cpu#5.cpu_load[2]
        19 ±14%     +41.4%         28 ± 2%  TOTAL sched_debug.cpu#31.cpu_load[1]
        19 ± 6%     +45.8%         28 ± 2%  TOTAL sched_debug.cpu#7.cpu_load[1]
        20 ±13%     +40.6%         28 ± 3%  TOTAL sched_debug.cfs_rq[25]:/.runnable_load_avg
      2943 ± 8%     +45.8%       4289 ± 0%  TOTAL sched_debug.cpu#1.curr->pid
      2703 ±11%     +59.2%       4302 ± 1%  TOTAL sched_debug.cpu#4.curr->pid
      2989 ±13%     +42.7%       4265 ± 2%  TOTAL sched_debug.cpu#24.curr->pid
      3048 ± 8%     +42.7%       4349 ± 1%  TOTAL sched_debug.cpu#3.curr->pid
     30576 ± 6%     +47.4%      45060 ± 0%  TOTAL sched_debug.cfs_rq[22]:/.avg->runnable_avg_sum
       682 ± 4%     +43.6%        979 ± 0%  TOTAL sched_debug.cfs_rq[30]:/.tg_runnable_contrib
      1100 ± 3%     +49.4%       1643 ± 4%  TOTAL numa-vmstat.node1.nr_alloc_batch
      4389 ± 1%     +46.7%       6439 ± 1%  TOTAL proc-vmstat.nr_alloc_batch
       668 ± 6%     +46.9%        982 ± 0%  TOTAL sched_debug.cfs_rq[22]:/.tg_runnable_contrib
      3115 ± 9%     +38.4%       4310 ± 1%  TOTAL sched_debug.cpu#5.curr->pid
       695 ± 4%     +41.8%        985 ± 0%  TOTAL sched_debug.cfs_rq[18]:/.tg_runnable_contrib
     31812 ± 4%     +42.5%      45324 ± 0%  TOTAL sched_debug.cfs_rq[18]:/.avg->runnable_avg_sum
      1157 ± 4%     +40.2%       1622 ± 7%  TOTAL numa-vmstat.node2.nr_alloc_batch
        19 ± 6%     +51.6%         28 ± 2%  TOTAL sched_debug.cpu#7.cpu_load[3]
        19 ± 6%     +51.5%         29 ± 1%  TOTAL sched_debug.cpu#2.cpu_load[4]
        19 ± 6%     +53.7%         29 ± 2%  TOTAL sched_debug.cpu#7.cpu_load[4]
        19 ±13%     +51.0%         29 ± 3%  TOTAL sched_debug.cpu#19.cpu_load[2]
        19 ±10%     +49.5%         29 ± 4%  TOTAL sched_debug.cpu#2.cpu_load[2]
        19 ±13%     +50.0%         28 ± 4%  TOTAL sched_debug.cpu#19.cpu_load[1]
        19 ± 6%     +49.5%         29 ± 3%  TOTAL sched_debug.cpu#2.cpu_load[3]
        20 ± 8%     +45.5%         29 ± 1%  TOTAL sched_debug.cpu#18.cpu_load[3]
        19 ± 5%     +51.0%         29 ± 2%  TOTAL sched_debug.cpu#20.cpu_load[4]
        20 ± 7%     +41.2%         28 ± 2%  TOTAL sched_debug.cpu#18.cpu_load[2]
        20 ± 7%     +47.0%         29 ± 2%  TOTAL sched_debug.cpu#4.cpu_load[4]
        20 ± 5%     +45.0%         29 ± 2%  TOTAL sched_debug.cpu#4.cpu_load[3]
        19 ±16%     +50.5%         28 ± 1%  TOTAL sched_debug.cpu#27.cpu_load[3]
        19 ±12%     +44.9%         28 ± 4%  TOTAL sched_debug.cpu#16.cpu_load[0]
        19 ± 5%     +48.5%         29 ± 3%  TOTAL sched_debug.cpu#1.cpu_load[3]
        19 ± 6%     +50.0%         29 ± 1%  TOTAL sched_debug.cpu#20.cpu_load[3]
      2924 ±10%     +48.1%       4330 ± 2%  TOTAL sched_debug.cpu#13.curr->pid
      2942 ± 7%     +45.9%       4291 ± 2%  TOTAL sched_debug.cpu#15.curr->pid
      2957 ± 5%     +44.4%       4269 ± 1%  TOTAL sched_debug.cpu#9.curr->pid
      2930 ±12%     +47.6%       4326 ± 3%  TOTAL sched_debug.cpu#16.curr->pid
       684 ± 7%     +44.1%        986 ± 0%  TOTAL sched_debug.cfs_rq[27]:/.tg_runnable_contrib
     32158 ± 4%     +40.1%      45057 ± 0%  TOTAL sched_debug.cfs_rq[24]:/.avg->runnable_avg_sum
      2949 ± 9%     +46.3%       4315 ± 2%  TOTAL sched_debug.cpu#19.curr->pid
        20 ± 9%     +46.5%         29 ± 1%  TOTAL sched_debug.cpu#16.cpu_load[4]
        19 ±14%     +51.0%         29 ± 5%  TOTAL sched_debug.cpu#19.cpu_load[0]
     31356 ± 7%     +43.8%      45098 ± 0%  TOTAL sched_debug.cfs_rq[27]:/.avg->runnable_avg_sum
     33090 ± 7%     +36.2%      45061 ± 0%  TOTAL sched_debug.cfs_rq[29]:/.avg->runnable_avg_sum
       721 ± 7%     +35.7%        979 ± 0%  TOTAL sched_debug.cfs_rq[29]:/.tg_runnable_contrib
     31372 ± 5%     +43.7%      45081 ± 0%  TOTAL sched_debug.cfs_rq[11]:/.avg->runnable_avg_sum
       701 ± 4%     +39.8%        980 ± 0%  TOTAL sched_debug.cfs_rq[24]:/.tg_runnable_contrib
      3005 ± 6%     +44.1%       4331 ± 2%  TOTAL sched_debug.cpu#28.curr->pid
       722 ± 5%     +36.2%        983 ± 0%  TOTAL sched_debug.cfs_rq[28]:/.tg_runnable_contrib
     32215 ± 6%     +40.3%      45183 ± 0%  TOTAL sched_debug.cfs_rq[19]:/.avg->runnable_avg_sum
     32108 ± 3%     +40.5%      45102 ± 0%  TOTAL sched_debug.cfs_rq[13]:/.avg->runnable_avg_sum
       703 ± 6%     +39.9%        983 ± 0%  TOTAL sched_debug.cfs_rq[19]:/.tg_runnable_contrib
     33049 ± 5%     +36.4%      45063 ± 0%  TOTAL sched_debug.cfs_rq[28]:/.avg->runnable_avg_sum
       685 ± 5%     +43.1%        981 ± 0%  TOTAL sched_debug.cfs_rq[11]:/.tg_runnable_contrib
       702 ± 3%     +39.7%        981 ± 0%  TOTAL sched_debug.cfs_rq[13]:/.tg_runnable_contrib
     32365 ± 1%     +39.3%      45086 ± 0%  TOTAL sched_debug.cfs_rq[20]:/.avg->runnable_avg_sum
       706 ± 1%     +38.9%        982 ± 0%  TOTAL sched_debug.cfs_rq[20]:/.tg_runnable_contrib
   1120854 ±41%     -45.0%     616182 ± 4%  TOTAL sched_debug.cpu#5.sched_count
     31452 ± 3%     +44.5%      45437 ± 1%  TOTAL sched_debug.cfs_rq[9]:/.avg->runnable_avg_sum
     32534 ± 5%     +38.4%      45040 ± 0%  TOTAL sched_debug.cfs_rq[31]:/.avg->runnable_avg_sum
       687 ± 3%     +44.1%        990 ± 1%  TOTAL sched_debug.cfs_rq[9]:/.tg_runnable_contrib
       702 ± 7%     +40.4%        985 ± 0%  TOTAL sched_debug.cfs_rq[25]:/.tg_runnable_contrib
       694 ± 5%     +41.6%        983 ± 0%  TOTAL sched_debug.cfs_rq[14]:/.tg_runnable_contrib
        19 ±24%     +50.0%         28 ± 7%  TOTAL sched_debug.cfs_rq[8]:/.runnable_load_avg
        19 ± 9%     +46.9%         28 ± 3%  TOTAL sched_debug.cpu#20.cpu_load[2]
        19 ±12%     +42.4%         28 ± 4%  TOTAL sched_debug.cpu#16.cpu_load[1]
        19 ±10%     +47.5%         29 ± 6%  TOTAL sched_debug.cpu#2.cpu_load[1]
        19 ±14%     +43.9%         28 ± 2%  TOTAL sched_debug.cpu#20.cpu_load[0]
        20 ±13%     +33.7%         27 ± 2%  TOTAL sched_debug.cpu#24.cpu_load[1]
        20 ±13%     +50.0%         30 ±11%  TOTAL sched_debug.cpu#2.cpu_load[0]
        18 ±16%     +48.9%         28 ± 3%  TOTAL sched_debug.cpu#27.cpu_load[2]
        18 ±19%     +46.8%         27 ± 5%  TOTAL sched_debug.cpu#26.cpu_load[1]
        19 ±14%     +43.3%         27 ± 4%  TOTAL sched_debug.cfs_rq[30]:/.runnable_load_avg
        20 ± 6%     +41.6%         28 ± 4%  TOTAL sched_debug.cfs_rq[4]:/.runnable_load_avg
        20 ±11%     +40.0%         28 ± 2%  TOTAL sched_debug.cfs_rq[16]:/.runnable_load_avg
        20 ± 4%     +36.3%         27 ± 2%  TOTAL sched_debug.cfs_rq[20]:/.runnable_load_avg
        19 ±13%     +44.9%         28 ± 1%  TOTAL sched_debug.cpu#16.cpu_load[2]
       710 ± 5%     +38.1%        981 ± 0%  TOTAL sched_debug.cfs_rq[31]:/.tg_runnable_contrib
     32116 ± 7%     +40.5%      45109 ± 0%  TOTAL sched_debug.cfs_rq[25]:/.avg->runnable_avg_sum
     31789 ± 5%     +41.7%      45041 ± 0%  TOTAL sched_debug.cfs_rq[14]:/.avg->runnable_avg_sum
     31727 ± 4%     +42.1%      45080 ± 0%  TOTAL sched_debug.cfs_rq[17]:/.avg->runnable_avg_sum
       696 ± 6%     +41.2%        983 ± 0%  TOTAL sched_debug.cfs_rq[0]:/.tg_runnable_contrib
     31871 ± 6%     +41.4%      45051 ± 0%  TOTAL sched_debug.cfs_rq[0]:/.avg->runnable_avg_sum
     32292 ± 5%     +39.4%      45011 ± 0%  TOTAL sched_debug.cfs_rq[7]:/.avg->runnable_avg_sum
       699 ± 3%     +40.5%        982 ± 0%  TOTAL sched_debug.cfs_rq[16]:/.tg_runnable_contrib
       706 ± 5%     +39.0%        981 ± 0%  TOTAL sched_debug.cfs_rq[7]:/.tg_runnable_contrib
      3025 ±16%     +43.2%       4334 ± 2%  TOTAL sched_debug.cpu#10.curr->pid
     32043 ± 3%     +40.6%      45060 ± 0%  TOTAL sched_debug.cfs_rq[16]:/.avg->runnable_avg_sum
       705 ± 4%     +39.8%        985 ± 0%  TOTAL sched_debug.cfs_rq[3]:/.tg_runnable_contrib
     32260 ± 4%     +39.8%      45100 ± 0%  TOTAL sched_debug.cfs_rq[3]:/.avg->runnable_avg_sum
       694 ± 4%     +41.3%        980 ± 0%  TOTAL sched_debug.cfs_rq[17]:/.tg_runnable_contrib
     32722 ± 3%     +38.1%      45195 ± 0%  TOTAL sched_debug.cfs_rq[2]:/.avg->runnable_avg_sum
      2869 ±11%     +52.0%       4362 ± 3%  TOTAL sched_debug.cpu#30.curr->pid
       698 ± 5%     +40.6%        982 ± 0%  TOTAL sched_debug.cfs_rq[23]:/.tg_runnable_contrib
     32007 ± 5%     +40.7%      45031 ± 0%  TOTAL sched_debug.cfs_rq[23]:/.avg->runnable_avg_sum
        21 ± 7%     +34.3%         29 ± 3%  TOTAL sched_debug.cfs_rq[18]:/.runnable_load_avg
        20 ±11%     +40.8%         29 ± 2%  TOTAL sched_debug.cpu#24.cpu_load[3]
        21 ±10%     +34.6%         28 ± 4%  TOTAL sched_debug.cpu#29.cpu_load[3]
        21 ±10%     +36.8%         29 ± 3%  TOTAL sched_debug.cpu#29.cpu_load[4]
        21 ± 9%     +39.0%         29 ± 1%  TOTAL sched_debug.cpu#24.cpu_load[4]
        21 ±12%     +34.9%         28 ± 5%  TOTAL sched_debug.cpu#29.cpu_load[2]
        20 ±11%     +41.7%         29 ± 2%  TOTAL sched_debug.cpu#21.cpu_load[3]
        20 ±10%     +45.0%         29 ± 2%  TOTAL sched_debug.cpu#14.cpu_load[3]
        20 ±15%     +38.2%         28 ± 3%  TOTAL sched_debug.cpu#14.cpu_load[1]
        20 ±11%     +45.0%         29 ± 2%  TOTAL sched_debug.cpu#16.cpu_load[3]
        23 ±10%     +25.2%         28 ± 2%  TOTAL sched_debug.cpu#55.load
        21 ± 6%     +34.6%         28 ± 2%  TOTAL sched_debug.cpu#28.cpu_load[4]
        20 ± 6%     +38.8%         28 ± 3%  TOTAL sched_debug.cfs_rq[7]:/.runnable_load_avg
        20 ±11%     +44.1%         29 ± 1%  TOTAL sched_debug.cpu#21.cpu_load[4]
      2959 ± 8%     +45.8%       4315 ± 1%  TOTAL sched_debug.cpu#6.curr->pid
     33054 ± 2%     +36.8%      45208 ± 0%  TOTAL sched_debug.cfs_rq[1]:/.avg->runnable_avg_sum
       715 ± 2%     +37.4%        983 ± 0%  TOTAL sched_debug.cfs_rq[2]:/.tg_runnable_contrib
     32267 ± 5%     +39.7%      45074 ± 0%  TOTAL sched_debug.cfs_rq[4]:/.avg->runnable_avg_sum
       705 ± 4%     +38.9%        979 ± 0%  TOTAL sched_debug.cfs_rq[4]:/.tg_runnable_contrib
      2983 ±12%     +47.3%       4394 ± 2%  TOTAL sched_debug.cpu#2.curr->pid
     32679 ± 5%     +37.9%      45073 ± 0%  TOTAL sched_debug.cfs_rq[6]:/.avg->runnable_avg_sum
       722 ± 2%     +35.8%        981 ± 0%  TOTAL sched_debug.cfs_rq[1]:/.tg_runnable_contrib
      2841 ±20%     +54.9%       4399 ± 1%  TOTAL sched_debug.cpu#0.curr->pid
     32684 ± 5%     +37.8%      45024 ± 0%  TOTAL sched_debug.cfs_rq[21]:/.avg->runnable_avg_sum
       714 ± 5%     +37.3%        980 ± 0%  TOTAL sched_debug.cfs_rq[6]:/.tg_runnable_contrib
        22 ±10%     +33.3%         29 ± 1%  TOTAL sched_debug.cpu#12.cpu_load[4]
   1044460 ±29%     -42.8%     597073 ± 1%  TOTAL sched_debug.cpu#31.sched_count
      2962 ±10%     +45.6%       4314 ± 2%  TOTAL sched_debug.cpu#21.curr->pid
    107755 ± 0%     +35.8%     146314 ± 0%  TOTAL sched_debug.cfs_rq[16]:/.exec_clock
      3040 ±14%     +45.8%       4431 ± 3%  TOTAL sched_debug.cpu#18.curr->pid
       713 ± 5%     +37.3%        980 ± 0%  TOTAL sched_debug.cfs_rq[21]:/.tg_runnable_contrib
    107444 ± 0%     +36.2%     146370 ± 0%  TOTAL sched_debug.cfs_rq[8]:/.exec_clock
     33921 ± 7%     +32.9%      45070 ± 0%  TOTAL sched_debug.cfs_rq[5]:/.avg->runnable_avg_sum
        19 ±16%     +41.7%         27 ± 4%  TOTAL sched_debug.cpu#27.cpu_load[0]
        18 ±16%     +46.8%         27 ± 4%  TOTAL sched_debug.cpu#27.cpu_load[1]
        20 ±13%     +35.3%         27 ± 4%  TOTAL sched_debug.cpu#24.cpu_load[0]
    110484 ± 1%     +34.1%     148181 ± 0%  TOTAL sched_debug.cfs_rq[10]:/.exec_clock
    108447 ± 0%     +34.7%     146038 ± 0%  TOTAL sched_debug.cfs_rq[17]:/.exec_clock
       741 ± 7%     +32.3%        980 ± 0%  TOTAL sched_debug.cfs_rq[5]:/.tg_runnable_contrib
      3131 ±10%     +40.9%       4412 ± 5%  TOTAL sched_debug.cpu#29.curr->pid
    108894 ± 1%     +33.9%     145821 ± 0%  TOTAL sched_debug.cfs_rq[24]:/.exec_clock
    108897 ± 0%     +34.3%     146281 ± 0%  TOTAL sched_debug.cfs_rq[9]:/.exec_clock
    110117 ± 2%     +32.5%     145856 ± 0%  TOTAL sched_debug.cfs_rq[25]:/.exec_clock
    109871 ± 1%     +32.8%     145922 ± 0%  TOTAL sched_debug.cfs_rq[15]:/.exec_clock
    110887 ± 1%     +32.8%     147227 ± 0%  TOTAL sched_debug.cfs_rq[12]:/.exec_clock
        20 ±13%     +40.2%         28 ± 4%  TOTAL sched_debug.cpu#0.cpu_load[3]
        21 ± 7%     +30.5%         27 ± 2%  TOTAL sched_debug.cfs_rq[28]:/.runnable_load_avg
        20 ± 7%     +37.9%         28 ± 1%  TOTAL sched_debug.cpu#18.cpu_load[1]
        21 ± 9%     +30.6%         28 ± 2%  TOTAL sched_debug.cfs_rq[29]:/.runnable_load_avg
        19 ±12%     +43.4%         28 ± 1%  TOTAL sched_debug.cpu#20.cpu_load[1]
        20 ± 9%     +38.8%         28 ± 2%  TOTAL sched_debug.cpu#21.cpu_load[2]
        21 ± 5%     +33.0%         28 ± 2%  TOTAL sched_debug.cpu#28.cpu_load[3]
        21 ± 5%     +32.4%         27 ± 2%  TOTAL sched_debug.cpu#28.cpu_load[1]
        20 ± 5%     +33.7%         27 ± 2%  TOTAL sched_debug.cpu#28.cpu_load[2]
        21 ± 9%     +29.0%         27 ± 5%  TOTAL sched_debug.cpu#29.cpu_load[0]
        21 ±13%     +35.2%         28 ± 3%  TOTAL sched_debug.cpu#24.cpu_load[2]
        21 ± 9%     +34.3%         28 ± 1%  TOTAL sched_debug.cpu#18.cpu_load[0]
      3098 ±11%     +39.9%       4335 ± 2%  TOTAL sched_debug.cpu#25.curr->pid
    109876 ± 0%     +33.3%     146491 ± 0%  TOTAL sched_debug.cfs_rq[11]:/.exec_clock
    268016 ± 3%     +32.8%     355948 ± 9%  TOTAL meminfo.Committed_AS
    109686 ± 1%     +33.3%     146181 ± 0%  TOTAL sched_debug.cfs_rq[13]:/.exec_clock
    109930 ± 1%     +33.3%     146529 ± 0%  TOTAL sched_debug.cfs_rq[18]:/.exec_clock
    110814 ± 2%     +31.8%     146073 ± 0%  TOTAL sched_debug.cfs_rq[22]:/.exec_clock
    111231 ± 1%     +31.2%     145955 ± 0%  TOTAL sched_debug.cfs_rq[26]:/.exec_clock
    792742 ± 2%     -24.3%     599925 ± 1%  TOTAL sched_debug.cpu#8.nr_switches
        20 ±13%     +40.6%         28 ± 2%  TOTAL sched_debug.cpu#14.cpu_load[2]
        22 ±11%     +30.4%         29 ± 1%  TOTAL sched_debug.cpu#12.cpu_load[3]
        22 ±10%     +32.7%         29 ± 3%  TOTAL sched_debug.cfs_rq[21]:/.runnable_load_avg
        22 ±11%     +26.5%         28 ± 1%  TOTAL sched_debug.cpu#12.cpu_load[2]
        20 ±11%     +43.1%         29 ± 5%  TOTAL sched_debug.cpu#0.cpu_load[4]
        22 ±11%     +29.1%         28 ± 2%  TOTAL sched_debug.cpu#12.cpu_load[1]
        22 ±21%     +30.0%         28 ± 2%  TOTAL sched_debug.cpu#17.cpu_load[3]
    896719 ±18%     -31.3%     615816 ± 4%  TOTAL sched_debug.cpu#30.sched_count
    110467 ± 1%     +32.0%     145847 ± 0%  TOTAL sched_debug.cfs_rq[19]:/.exec_clock
    110856 ± 1%     +31.7%     145977 ± 0%  TOTAL sched_debug.cfs_rq[20]:/.exec_clock
    112749 ± 1%     +32.1%     148960 ± 0%  TOTAL sched_debug.cfs_rq[2]:/.exec_clock
    110628 ± 1%     +32.0%     146083 ± 0%  TOTAL sched_debug.cfs_rq[23]:/.exec_clock
    116500 ± 1%     +31.0%     152586 ± 0%  TOTAL sched_debug.cfs_rq[0]:/.exec_clock
     33803 ± 6%     +33.2%      45033 ± 0%  TOTAL sched_debug.cfs_rq[12]:/.avg->runnable_avg_sum
       738 ± 6%     +32.9%        981 ± 0%  TOTAL sched_debug.cfs_rq[12]:/.tg_runnable_contrib
    111685 ± 0%     +30.4%     145683 ± 0%  TOTAL sched_debug.cfs_rq[4]:/.exec_clock
    111045 ± 1%     +31.3%     145845 ± 0%  TOTAL sched_debug.cfs_rq[21]:/.exec_clock
    111156 ± 0%     +31.3%     145923 ± 0%  TOTAL sched_debug.cfs_rq[14]:/.exec_clock
    111731 ± 0%     +30.3%     145623 ± 0%  TOTAL sched_debug.cfs_rq[5]:/.exec_clock
    111147 ± 1%     +31.0%     145641 ± 0%  TOTAL sched_debug.cfs_rq[6]:/.exec_clock
      3172 ± 3%     +33.8%       4245 ± 0%  TOTAL sched_debug.cpu#12.curr->pid
    111847 ± 0%     +30.4%     145836 ± 0%  TOTAL sched_debug.cfs_rq[1]:/.exec_clock
        20 ±39%     +43.1%         29 ± 3%  TOTAL sched_debug.cpu#63.load
    111241 ± 0%     +31.0%     145727 ± 0%  TOTAL sched_debug.cfs_rq[3]:/.exec_clock
    111457 ± 1%     +30.7%     145630 ± 0%  TOTAL sched_debug.cfs_rq[31]:/.exec_clock
    112620 ± 1%     +29.4%     145693 ± 0%  TOTAL sched_debug.cfs_rq[27]:/.exec_clock
    112963 ± 1%     +28.9%     145645 ± 0%  TOTAL sched_debug.cfs_rq[29]:/.exec_clock
    112421 ± 0%     +29.6%     145673 ± 0%  TOTAL sched_debug.cfs_rq[7]:/.exec_clock
      7338 ±11%     +23.3%       9050 ± 7%  TOTAL numa-vmstat.node3.nr_anon_pages
     29353 ±11%     +23.3%      36204 ± 7%  TOTAL numa-meminfo.node3.AnonPages
    766344 ± 4%     -22.2%     596356 ± 1%  TOTAL sched_debug.cpu#12.nr_switches
    112871 ± 0%     +29.0%     145613 ± 0%  TOTAL sched_debug.cfs_rq[28]:/.exec_clock
    785724 ± 2%     -22.0%     612543 ± 0%  TOTAL sched_debug.cpu#0.nr_switches
    770632 ± 2%     -22.3%     598517 ± 1%  TOTAL sched_debug.cpu#15.nr_switches
    113328 ± 0%     +28.5%     145614 ± 0%  TOTAL sched_debug.cfs_rq[30]:/.exec_clock
        19 ±16%     +40.2%         27 ± 2%  TOTAL sched_debug.cfs_rq[0]:/.runnable_load_avg
        19 ±23%     +39.2%         27 ± 4%  TOTAL sched_debug.cfs_rq[26]:/.runnable_load_avg
    763775 ± 3%     -21.8%     596955 ± 1%  TOTAL sched_debug.cpu#13.nr_switches
        21 ±10%     +30.6%         28 ± 4%  TOTAL sched_debug.cpu#29.cpu_load[1]
        20 ±10%     +36.5%         28 ± 3%  TOTAL sched_debug.cpu#21.cpu_load[1]
        21 ±18%     +31.1%         27 ± 1%  TOTAL sched_debug.cpu#0.cpu_load[1]
        22 ± 4%     +30.0%         28 ± 4%  TOTAL sched_debug.cpu#52.load
        21 ±11%     +30.8%         28 ± 3%  TOTAL sched_debug.cpu#21.cpu_load[0]
        21 ± 6%     +28.4%         28 ± 3%  TOTAL sched_debug.cpu#28.cpu_load[0]
        21 ±19%     +31.8%         28 ± 2%  TOTAL sched_debug.cpu#17.cpu_load[2]
        19 ±20%     +45.8%         28 ± 3%  TOTAL sched_debug.cpu#26.cpu_load[0]
        22 ±16%     +27.3%         28 ± 5%  TOTAL sched_debug.cfs_rq[27]:/.runnable_load_avg
        21 ±17%     +29.4%         28 ± 2%  TOTAL sched_debug.cpu#17.cpu_load[0]
        21 ±12%     +32.1%         28 ± 2%  TOTAL sched_debug.cfs_rq[24]:/.runnable_load_avg
        20 ±17%     +34.6%         28 ± 2%  TOTAL sched_debug.cpu#0.cpu_load[2]
    773695 ± 2%     -22.5%     599935 ± 1%  TOTAL sched_debug.cpu#11.nr_switches
    760832 ± 2%     -21.3%     598498 ± 1%  TOTAL sched_debug.cpu#9.nr_switches
        23 ±17%     +20.3%         28 ± 5%  TOTAL sched_debug.cfs_rq[52]:/.load
    761399 ± 1%     -21.2%     600107 ± 1%  TOTAL sched_debug.cpu#10.nr_switches
    770683 ± 1%     -21.1%     607908 ± 1%  TOTAL sched_debug.cpu#2.nr_switches
    757803 ± 4%     -20.7%     600923 ± 1%  TOTAL sched_debug.cpu#6.nr_switches
    750665 ± 1%     -19.1%     606952 ± 0%  TOTAL sched_debug.cpu#1.nr_switches
    744077 ± 2%     -19.6%     598358 ± 0%  TOTAL sched_debug.cpu#16.nr_switches
    753828 ± 1%     -19.6%     605817 ± 1%  TOTAL sched_debug.cpu#3.nr_switches
    742810 ± 2%     -18.9%     602251 ± 0%  TOTAL sched_debug.cpu#5.nr_switches
    732991 ± 2%     -18.5%     597448 ± 1%  TOTAL sched_debug.cpu#17.nr_switches
    725349 ± 2%     -17.9%     595187 ± 1%  TOTAL sched_debug.cpu#14.nr_switches
        21 ±18%     +29.6%         28 ± 2%  TOTAL sched_debug.cpu#17.cpu_load[1]
        22 ±14%     +28.2%         28 ± 2%  TOTAL sched_debug.cpu#12.cpu_load[0]
    736992 ± 2%     -17.8%     605506 ± 1%  TOTAL sched_debug.cpu#4.nr_switches
    724402 ± 1%     -16.7%     603369 ± 0%  TOTAL sched_debug.cpu#7.nr_switches
    712130 ± 2%     -15.5%     602079 ± 1%  TOTAL sched_debug.cpu#24.nr_switches
    715409 ± 5%     -16.1%     600473 ± 2%  TOTAL sched_debug.cpu#22.nr_switches
    169672 ± 0%     +19.0%     201867 ± 0%  TOTAL meminfo.Active(anon)
     42418 ± 0%     +18.9%      50446 ± 0%  TOTAL proc-vmstat.nr_active_anon
    710553 ± 3%     -16.0%     597062 ± 1%  TOTAL sched_debug.cpu#31.nr_switches
    723251 ± 2%     -17.1%     599549 ± 1%  TOTAL sched_debug.cpu#18.nr_switches
    706093 ± 4%     -14.9%     601183 ± 1%  TOTAL sched_debug.cpu#26.nr_switches
    704669 ± 2%     -14.9%     599338 ± 1%  TOTAL sched_debug.cpu#20.nr_switches
    703753 ± 3%     -15.3%     596238 ± 1%  TOTAL sched_debug.cpu#21.nr_switches
    702440 ± 5%     -15.1%     596380 ± 1%  TOTAL sched_debug.cpu#25.nr_switches
    702660 ± 4%     -14.7%     599141 ± 1%  TOTAL sched_debug.cpu#19.nr_switches
        22 ±14%     +28.2%         28 ± 4%  TOTAL sched_debug.cfs_rq[12]:/.runnable_load_avg
        22 ±22%     +32.4%         29 ± 1%  TOTAL sched_debug.cpu#34.load
    701182 ± 4%     -14.6%     598606 ± 1%  TOTAL sched_debug.cpu#23.nr_switches
      0.23 ± 5%     -15.8%       0.19 ± 6%  TOTAL turbostat.%pc3
    238557 ± 0%     +13.5%     270767 ± 0%  TOTAL meminfo.Active
    135003 ± 0%     +12.7%     152182 ± 0%  TOTAL sched_debug.cpu#56.nr_load_updates
    135518 ± 1%     +12.2%     152028 ± 0%  TOTAL sched_debug.cpu#63.nr_load_updates
    668473 ± 4%     -10.5%     598372 ± 1%  TOTAL sched_debug.cpu#29.nr_switches
    137963 ± 0%     +11.9%     154381 ± 0%  TOTAL sched_debug.cpu#57.nr_load_updates
    135802 ± 0%     +12.1%     152194 ± 0%  TOTAL sched_debug.cpu#58.nr_load_updates
    673342 ± 3%     -10.3%     604167 ± 2%  TOTAL sched_debug.cpu#30.nr_switches
    136075 ± 0%     +11.8%     152117 ± 0%  TOTAL sched_debug.cpu#60.nr_load_updates
    136265 ± 0%     +11.6%     152124 ± 0%  TOTAL sched_debug.cpu#59.nr_load_updates
    136657 ± 0%     +11.5%     152399 ± 0%  TOTAL sched_debug.cpu#48.nr_load_updates
    672367 ± 4%     -10.6%     600775 ± 2%  TOTAL sched_debug.cpu#27.nr_switches
    136400 ± 0%     +11.6%     152176 ± 0%  TOTAL sched_debug.cpu#61.nr_load_updates
    136146 ± 0%     +11.7%     152103 ± 0%  TOTAL sched_debug.cpu#62.nr_load_updates
      1848 ± 2%     +10.9%       2049 ± 5%  TOTAL numa-meminfo.node0.Mapped
    136683 ± 0%     +11.4%     152199 ± 0%  TOTAL sched_debug.cpu#55.nr_load_updates
    137175 ± 0%     +11.0%     152310 ± 0%  TOTAL sched_debug.cpu#51.nr_load_updates
    139156 ± 0%     +10.8%     154236 ± 0%  TOTAL sched_debug.cpu#49.nr_load_updates
    137076 ± 0%     +11.1%     152253 ± 0%  TOTAL sched_debug.cpu#53.nr_load_updates
    672781 ± 3%     -10.5%     601982 ± 1%  TOTAL sched_debug.cpu#28.nr_switches
    136873 ± 1%     +11.2%     152248 ± 0%  TOTAL sched_debug.cpu#52.nr_load_updates
    137601 ± 0%     +10.8%     152404 ± 0%  TOTAL sched_debug.cpu#50.nr_load_updates
    137409 ± 0%     +10.8%     152228 ± 0%  TOTAL sched_debug.cpu#54.nr_load_updates
    608198 ± 1%   +5895.2%   36462984 ± 0%  TOTAL time.voluntary_context_switches
     62.13 ± 1%    +247.4%     215.86 ± 1%  TOTAL time.user_time
    560843 ± 0%     -53.2%     262235 ± 0%  TOTAL vmstat.system.cs
     54.79 ± 0%     +81.7%      99.56 ± 0%  TOTAL turbostat.%c0
     38022 ± 0%     +76.6%      67144 ± 0%  TOTAL vmstat.system.in
      2726 ± 0%     +68.6%       4596 ± 0%  TOTAL time.percent_of_cpu_this_job_got
      8155 ± 0%     +67.2%      13638 ± 0%  TOTAL time.system_time
    951709 ± 0%     +53.4%    1459554 ± 0%  TOTAL time.involuntary_context_switches
    227030 ± 1%     -16.1%     190393 ± 0%  TOTAL time.minor_page_faults


                                     qperf.sctp.bw

   1.3e+09 ++---------------------------------------------------------------+
   1.2e+09 ++    O                                         O              O |
           O OO O  O       O          O  O  O OO   OO OO O     O    O  O O  |
   1.1e+09 ++        OO OO  O O OO OO   O  O     O            O  OO   O     O
     1e+09 ++                                               O               |
           |                                                                |
     9e+08 ++                                                               |
     8e+08 ++                                                               |
     7e+08 ++                                                               |
           |                                                                |
     6e+08 ++                                                               |
     5e+08 ++                                                               |
           |                                      .**.         *. *.        |
     4e+08 *+**.**.*. *.**.**.*.**.**.*.**.**.**.*    **.*.**.*  *  *.**    |
     3e+08 ++--------*------------------------------------------------------+

	[*] bisect-good sample
	[O] bisect-bad  sample

To reproduce:

	apt-get install ruby ruby-oj
	git clone git://git.kernel.org/pub/scm/linux/kernel/git/wfg/lkp-tests.git
	cd lkp-tests
	bin/setup-local job.yaml # the job file attached in this email
	bin/run-local   job.yaml

Disclaimer:
Results have been estimated based on internal Intel analysis and are provided
for informational purposes only. Any difference in system hardware or software
design or configuration may affect actual performance.

Thanks,
Fengguang

[-- Attachment #2: job.yaml --]
[-- Type: text/plain, Size: 1482 bytes --]

---
testcase: netperf
default_monitors:
  watch-oom: 
  wait: pre-test
  uptime: 
  iostat: 
  vmstat: 
  numa-numastat: 
  numa-vmstat: 
  numa-meminfo: 
  proc-vmstat: 
  proc-stat: 
  meminfo: 
  slabinfo: 
  interrupts: 
  lock_stat: 
  latency_stats: 
  softirqs: 
  bdi_dev_mapping: 
  diskstats: 
  energy: 
  cpuidle: 
  cpufreq: 
  turbostat: 
  sched_debug:
    interval: 10
  pmeter: 
model: Nehalem-EX
memory: 256G
nr_ssd_partitions: 6
ssd_partitions: "/dev/disk/by-id/ata-INTEL_SSD*part1"
swap_partitions: "/dev/disk/by-id/ata-WDC_WD10EARS-00Y5B1_WD-WCAV5F059074-part2"
runtime: 300s
nr_threads:
- 200%
perf-profile:
  freq: 800
netperf:
  send_size: 10K
  test:
  - SCTP_STREAM_MANY
branch: linus/master
commit: 19583ca584d6f574384e17fe7613dfaeadcdc4a6
repeat_to: 3
enqueue_time: 2014-09-25 21:41:16.275537552 +08:00
testbox: lkp-nex04
kconfig: x86_64-rhel
kernel: "/kernel/x86_64-rhel/19583ca584d6f574384e17fe7613dfaeadcdc4a6/vmlinuz-3.16.0"
user: lkp
queue: wfg
result_root: "/result/lkp-nex04/netperf/300s-200%-10K-SCTP_STREAM_MANY/debian-x86_64.cgz/x86_64-rhel/19583ca584d6f574384e17fe7613dfaeadcdc4a6/0"
job_file: "/lkp/scheduled/lkp-nex04/wfg_netperf-300s-200%-10K-SCTP_STREAM_MANY-x86_64-rhel-19583ca584d6f574384e17fe7613dfaeadcdc4a6-2.yaml"
dequeue_time: 2014-09-30 00:21:40.347862294 +08:00
history_time: 300
job_state: finished
loadavg: 139.56 108.61 48.96 2/529 13193
start_time: '1412007779'
end_time: '1412008083'
version: "/lkp/lkp/.src-20140929-152043"

[-- Attachment #3: reproduce --]
[-- Type: text/plain, Size: 6538 bytes --]

netserver
netperf -t SCTP_STREAM_MANY -c -C -l 300 -- -m 10K
netperf -t SCTP_STREAM_MANY -c -C -l 300 -- -m 10K
netperf -t SCTP_STREAM_MANY -c -C -l 300 -- -m 10K
netperf -t SCTP_STREAM_MANY -c -C -l 300 -- -m 10K
netperf -t SCTP_STREAM_MANY -c -C -l 300 -- -m 10K
netperf -t SCTP_STREAM_MANY -c -C -l 300 -- -m 10K
netperf -t SCTP_STREAM_MANY -c -C -l 300 -- -m 10K
netperf -t SCTP_STREAM_MANY -c -C -l 300 -- -m 10K
netperf -t SCTP_STREAM_MANY -c -C -l 300 -- -m 10K
netperf -t SCTP_STREAM_MANY -c -C -l 300 -- -m 10K
netperf -t SCTP_STREAM_MANY -c -C -l 300 -- -m 10K
netperf -t SCTP_STREAM_MANY -c -C -l 300 -- -m 10K
netperf -t SCTP_STREAM_MANY -c -C -l 300 -- -m 10K
netperf -t SCTP_STREAM_MANY -c -C -l 300 -- -m 10K
netperf -t SCTP_STREAM_MANY -c -C -l 300 -- -m 10K
netperf -t SCTP_STREAM_MANY -c -C -l 300 -- -m 10K
netperf -t SCTP_STREAM_MANY -c -C -l 300 -- -m 10K
netperf -t SCTP_STREAM_MANY -c -C -l 300 -- -m 10K
netperf -t SCTP_STREAM_MANY -c -C -l 300 -- -m 10K
netperf -t SCTP_STREAM_MANY -c -C -l 300 -- -m 10K
netperf -t SCTP_STREAM_MANY -c -C -l 300 -- -m 10K
netperf -t SCTP_STREAM_MANY -c -C -l 300 -- -m 10K
netperf -t SCTP_STREAM_MANY -c -C -l 300 -- -m 10K
netperf -t SCTP_STREAM_MANY -c -C -l 300 -- -m 10K
netperf -t SCTP_STREAM_MANY -c -C -l 300 -- -m 10K
netperf -t SCTP_STREAM_MANY -c -C -l 300 -- -m 10K
netperf -t SCTP_STREAM_MANY -c -C -l 300 -- -m 10K
netperf -t SCTP_STREAM_MANY -c -C -l 300 -- -m 10K
netperf -t SCTP_STREAM_MANY -c -C -l 300 -- -m 10K
netperf -t SCTP_STREAM_MANY -c -C -l 300 -- -m 10K
netperf -t SCTP_STREAM_MANY -c -C -l 300 -- -m 10K
netperf -t SCTP_STREAM_MANY -c -C -l 300 -- -m 10K
netperf -t SCTP_STREAM_MANY -c -C -l 300 -- -m 10K
netperf -t SCTP_STREAM_MANY -c -C -l 300 -- -m 10K
netperf -t SCTP_STREAM_MANY -c -C -l 300 -- -m 10K
netperf -t SCTP_STREAM_MANY -c -C -l 300 -- -m 10K
netperf -t SCTP_STREAM_MANY -c -C -l 300 -- -m 10K
netperf -t SCTP_STREAM_MANY -c -C -l 300 -- -m 10K
netperf -t SCTP_STREAM_MANY -c -C -l 300 -- -m 10K
netperf -t SCTP_STREAM_MANY -c -C -l 300 -- -m 10K
netperf -t SCTP_STREAM_MANY -c -C -l 300 -- -m 10K
netperf -t SCTP_STREAM_MANY -c -C -l 300 -- -m 10K
netperf -t SCTP_STREAM_MANY -c -C -l 300 -- -m 10K
netperf -t SCTP_STREAM_MANY -c -C -l 300 -- -m 10K
netperf -t SCTP_STREAM_MANY -c -C -l 300 -- -m 10K
netperf -t SCTP_STREAM_MANY -c -C -l 300 -- -m 10K
netperf -t SCTP_STREAM_MANY -c -C -l 300 -- -m 10K
netperf -t SCTP_STREAM_MANY -c -C -l 300 -- -m 10K
netperf -t SCTP_STREAM_MANY -c -C -l 300 -- -m 10K
netperf -t SCTP_STREAM_MANY -c -C -l 300 -- -m 10K
netperf -t SCTP_STREAM_MANY -c -C -l 300 -- -m 10K
netperf -t SCTP_STREAM_MANY -c -C -l 300 -- -m 10K
netperf -t SCTP_STREAM_MANY -c -C -l 300 -- -m 10K
netperf -t SCTP_STREAM_MANY -c -C -l 300 -- -m 10K
netperf -t SCTP_STREAM_MANY -c -C -l 300 -- -m 10K
netperf -t SCTP_STREAM_MANY -c -C -l 300 -- -m 10K
netperf -t SCTP_STREAM_MANY -c -C -l 300 -- -m 10K
netperf -t SCTP_STREAM_MANY -c -C -l 300 -- -m 10K
netperf -t SCTP_STREAM_MANY -c -C -l 300 -- -m 10K
netperf -t SCTP_STREAM_MANY -c -C -l 300 -- -m 10K
netperf -t SCTP_STREAM_MANY -c -C -l 300 -- -m 10K
netperf -t SCTP_STREAM_MANY -c -C -l 300 -- -m 10K
netperf -t SCTP_STREAM_MANY -c -C -l 300 -- -m 10K
netperf -t SCTP_STREAM_MANY -c -C -l 300 -- -m 10K
netperf -t SCTP_STREAM_MANY -c -C -l 300 -- -m 10K
netperf -t SCTP_STREAM_MANY -c -C -l 300 -- -m 10K
netperf -t SCTP_STREAM_MANY -c -C -l 300 -- -m 10K
netperf -t SCTP_STREAM_MANY -c -C -l 300 -- -m 10K
netperf -t SCTP_STREAM_MANY -c -C -l 300 -- -m 10K
netperf -t SCTP_STREAM_MANY -c -C -l 300 -- -m 10K
netperf -t SCTP_STREAM_MANY -c -C -l 300 -- -m 10K
netperf -t SCTP_STREAM_MANY -c -C -l 300 -- -m 10K
netperf -t SCTP_STREAM_MANY -c -C -l 300 -- -m 10K
netperf -t SCTP_STREAM_MANY -c -C -l 300 -- -m 10K
netperf -t SCTP_STREAM_MANY -c -C -l 300 -- -m 10K
netperf -t SCTP_STREAM_MANY -c -C -l 300 -- -m 10K
netperf -t SCTP_STREAM_MANY -c -C -l 300 -- -m 10K
netperf -t SCTP_STREAM_MANY -c -C -l 300 -- -m 10K
netperf -t SCTP_STREAM_MANY -c -C -l 300 -- -m 10K
netperf -t SCTP_STREAM_MANY -c -C -l 300 -- -m 10K
netperf -t SCTP_STREAM_MANY -c -C -l 300 -- -m 10K
netperf -t SCTP_STREAM_MANY -c -C -l 300 -- -m 10K
netperf -t SCTP_STREAM_MANY -c -C -l 300 -- -m 10K
netperf -t SCTP_STREAM_MANY -c -C -l 300 -- -m 10K
netperf -t SCTP_STREAM_MANY -c -C -l 300 -- -m 10K
netperf -t SCTP_STREAM_MANY -c -C -l 300 -- -m 10K
netperf -t SCTP_STREAM_MANY -c -C -l 300 -- -m 10K
netperf -t SCTP_STREAM_MANY -c -C -l 300 -- -m 10K
netperf -t SCTP_STREAM_MANY -c -C -l 300 -- -m 10K
netperf -t SCTP_STREAM_MANY -c -C -l 300 -- -m 10K
netperf -t SCTP_STREAM_MANY -c -C -l 300 -- -m 10K
netperf -t SCTP_STREAM_MANY -c -C -l 300 -- -m 10K
netperf -t SCTP_STREAM_MANY -c -C -l 300 -- -m 10K
netperf -t SCTP_STREAM_MANY -c -C -l 300 -- -m 10K
netperf -t SCTP_STREAM_MANY -c -C -l 300 -- -m 10K
netperf -t SCTP_STREAM_MANY -c -C -l 300 -- -m 10K
netperf -t SCTP_STREAM_MANY -c -C -l 300 -- -m 10K
netperf -t SCTP_STREAM_MANY -c -C -l 300 -- -m 10K
netperf -t SCTP_STREAM_MANY -c -C -l 300 -- -m 10K
netperf -t SCTP_STREAM_MANY -c -C -l 300 -- -m 10K
netperf -t SCTP_STREAM_MANY -c -C -l 300 -- -m 10K
netperf -t SCTP_STREAM_MANY -c -C -l 300 -- -m 10K
netperf -t SCTP_STREAM_MANY -c -C -l 300 -- -m 10K
netperf -t SCTP_STREAM_MANY -c -C -l 300 -- -m 10K
netperf -t SCTP_STREAM_MANY -c -C -l 300 -- -m 10K
netperf -t SCTP_STREAM_MANY -c -C -l 300 -- -m 10K
netperf -t SCTP_STREAM_MANY -c -C -l 300 -- -m 10K
netperf -t SCTP_STREAM_MANY -c -C -l 300 -- -m 10K
netperf -t SCTP_STREAM_MANY -c -C -l 300 -- -m 10K
netperf -t SCTP_STREAM_MANY -c -C -l 300 -- -m 10K
netperf -t SCTP_STREAM_MANY -c -C -l 300 -- -m 10K
netperf -t SCTP_STREAM_MANY -c -C -l 300 -- -m 10K
netperf -t SCTP_STREAM_MANY -c -C -l 300 -- -m 10K
netperf -t SCTP_STREAM_MANY -c -C -l 300 -- -m 10K
netperf -t SCTP_STREAM_MANY -c -C -l 300 -- -m 10K
netperf -t SCTP_STREAM_MANY -c -C -l 300 -- -m 10K
netperf -t SCTP_STREAM_MANY -c -C -l 300 -- -m 10K
netperf -t SCTP_STREAM_MANY -c -C -l 300 -- -m 10K
netperf -t SCTP_STREAM_MANY -c -C -l 300 -- -m 10K
netperf -t SCTP_STREAM_MANY -c -C -l 300 -- -m 10K
netperf -t SCTP_STREAM_MANY -c -C -l 300 -- -m 10K
netperf -t SCTP_STREAM_MANY -c -C -l 300 -- -m 10K
netperf -t SCTP_STREAM_MANY -c -C -l 300 -- -m 10K
netperf -t SCTP_STREAM_MANY -c -C -l 300 -- -m 10K
netperf -t SCTP_STREAM_MANY -c -C -l 300 -- -m 10K
netperf -t SCTP_STREAM_MANY -c -C -l 300 -- -m 10K
netperf -t SCTP_STREAM_MANY -c -C -l 300 -- -m 10K
netperf -t SCTP_STREAM_MANY -c -C -l 300 -- -m 10K

[-- Attachment #4: Type: text/plain, Size: 85 bytes --]

_______________________________________________
LKP mailing list
LKP@linux.intel.com

^ permalink raw reply

* [datapath minor fixes 0/3] datapath minor fixes
From: Andy Zhou @ 2014-10-01  7:55 UTC (permalink / raw)
  To: davem; +Cc: netdev, Andy Zhou

A few small issues found in preparing for GENEVE support upstreaming

*** BLURB HERE ***

Andy Zhou (3):
  datapath: avoid hard code OVS_VPORT_TYPE_GENEVE
  datapath: avoid double free routing entry in vxlan_port xmit
  datapath: serializing GENEVE options based on flow key's tun_flags

 datapath/flow_netlink.c                           | 2 +-
 datapath/linux/compat/include/linux/openvswitch.h | 2 +-
 datapath/vport-vxlan.c                            | 2 --
 3 files changed, 2 insertions(+), 4 deletions(-)

-- 
1.9.1

^ permalink raw reply

* [datapath minor fixes 1/3] datapath: avoid hard code OVS_VPORT_TYPE_GENEVE
From: Andy Zhou @ 2014-10-01  7:55 UTC (permalink / raw)
  To: davem; +Cc: netdev, Andy Zhou
In-Reply-To: <1412150148-9991-1-git-send-email-azhou@nicira.com>

OVS_VPORT_TYPE_GENEVE is currently hard coded to 6. This is not
necessary since slot 5 has not been taken yet. Drop the hard
coded value to before upstreaming GENEVE support to Linux kernel.

Signed-off-by: Andy Zhou <azhou@nicira.com>
---
 datapath/linux/compat/include/linux/openvswitch.h | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/datapath/linux/compat/include/linux/openvswitch.h b/datapath/linux/compat/include/linux/openvswitch.h
index 306ea86..5d40da5 100644
--- a/datapath/linux/compat/include/linux/openvswitch.h
+++ b/datapath/linux/compat/include/linux/openvswitch.h
@@ -222,7 +222,7 @@ enum ovs_vport_type {
 	OVS_VPORT_TYPE_INTERNAL, /* network device implemented by datapath */
 	OVS_VPORT_TYPE_GRE,	 /* GRE tunnel. */
 	OVS_VPORT_TYPE_VXLAN,    /* VXLAN tunnel */
-	OVS_VPORT_TYPE_GENEVE = 6,  /* Geneve tunnel */
+	OVS_VPORT_TYPE_GENEVE,   /* Geneve tunnel */
 	OVS_VPORT_TYPE_GRE64 = 104, /* GRE tunnel with 64-bit keys */
 	OVS_VPORT_TYPE_LISP = 105,  /* LISP tunnel */
 	__OVS_VPORT_TYPE_MAX
-- 
1.9.1

^ permalink raw reply related

* [datapath minor fixes 2/3] datapath: avoid double free routing entry in vxlan_port xmit
From: Andy Zhou @ 2014-10-01  7:55 UTC (permalink / raw)
  To: davem; +Cc: netdev, Andy Zhou
In-Reply-To: <1412150148-9991-1-git-send-email-azhou@nicira.com>

Route entry will be free on error by vport when freeing skb.
additional error check and free after xmit() will cause double free.

Signed-off-by: Andy Zhou <azhou@nicira.com>
---
 datapath/vport-vxlan.c | 2 --
 1 file changed, 2 deletions(-)

diff --git a/datapath/vport-vxlan.c b/datapath/vport-vxlan.c
index 8689853..44968d0 100644
--- a/datapath/vport-vxlan.c
+++ b/datapath/vport-vxlan.c
@@ -180,8 +180,6 @@ static int vxlan_tnl_send(struct vport *vport, struct sk_buff *skb)
 			     tun_key->ipv4_ttl, df,
 			     src_port, dst_port,
 			     htonl(be64_to_cpu(tun_key->tun_id) << 8));
-	if (err < 0)
-		ip_rt_put(rt);
 error:
 	return err;
 }
-- 
1.9.1

^ permalink raw reply related

* [datapath minor fixes 3/3] datapath: serializing GENEVE options based on flow key's tun_flags
From: Andy Zhou @ 2014-10-01  7:55 UTC (permalink / raw)
  To: davem; +Cc: netdev, Andy Zhou
In-Reply-To: <1412150148-9991-1-git-send-email-azhou@nicira.com>

TUNNEL_OPTIONS_PRESENT should always be checked against the tun_flgs
bits in a flow key, for both flow and mask serialization.

Signed-off-by: Andy Zhou <azhou@nicira.com>
---
 datapath/flow_netlink.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/datapath/flow_netlink.c b/datapath/flow_netlink.c
index a3f34f1..35feedc 100644
--- a/datapath/flow_netlink.c
+++ b/datapath/flow_netlink.c
@@ -1122,7 +1122,7 @@ int ovs_nla_put_flow(const struct sw_flow_key *swkey,
 	if ((swkey->tun_key.ipv4_dst || is_mask)) {
 		const struct geneve_opt *opts = NULL;
 
-		if (output->tun_key.tun_flags & TUNNEL_OPTIONS_PRESENT)
+		if (swkey->tun_key.tun_flags & TUNNEL_OPTIONS_PRESENT)
 			opts = GENEVE_OPTS(output, swkey->tun_opts_len);
 
 		if (ipv4_tun_to_nlattr(skb, &output->tun_key, opts,
-- 
1.9.1

^ permalink raw reply related

* Re: [datapath minor fixes 0/3] datapath minor fixes
From: Andy Zhou @ 2014-10-01  7:59 UTC (permalink / raw)
  To: David Miller; +Cc: netdev@vger.kernel.org, Andy Zhou
In-Reply-To: <1412150148-9991-1-git-send-email-azhou@nicira.com>

Please ignore this series. I meant to send them to dev@openvswitch.com.

On Wed, Oct 1, 2014 at 12:55 AM, Andy Zhou <azhou@nicira.com> wrote:
> A few small issues found in preparing for GENEVE support upstreaming
>
> *** BLURB HERE ***
>
> Andy Zhou (3):
>   datapath: avoid hard code OVS_VPORT_TYPE_GENEVE
>   datapath: avoid double free routing entry in vxlan_port xmit
>   datapath: serializing GENEVE options based on flow key's tun_flags
>
>  datapath/flow_netlink.c                           | 2 +-
>  datapath/linux/compat/include/linux/openvswitch.h | 2 +-
>  datapath/vport-vxlan.c                            | 2 --
>  3 files changed, 2 insertions(+), 4 deletions(-)
>
> --
> 1.9.1
>

^ permalink raw reply

* Re: [PATCH v2 net-next] net: pktgen: packet bursting via skb->xmit_more
From: Jesper Dangaard Brouer @ 2014-10-01  8:16 UTC (permalink / raw)
  To: Alexei Starovoitov; +Cc: David S. Miller, Eric Dumazet, netdev, brouer
In-Reply-To: <1412124801-32096-1-git-send-email-ast@plumgrid.com>

On Tue, 30 Sep 2014 17:53:21 -0700
Alexei Starovoitov <ast@plumgrid.com> wrote:

> This patch demonstrates the effect of delaying update of HW tailptr.
> (based on earlier patch by Jesper)
> 
> burst=1 is the default. It sends one packet with xmit_more=false
> burst=2 sends one packet with xmit_more=true and
>         2nd copy of the same packet with xmit_more=false
> burst=3 sends two copies of the same packet with xmit_more=true and
>         3rd copy with xmit_more=false
> 
> Performance with ixgbe (usec 30):
> burst=1  tx:9.2 Mpps
> burst=2  tx:13.5 Mpps
> burst=3  tx:14.5 Mpps full 10G line rate
> 
> Signed-off-by: Alexei Starovoitov <ast@plumgrid.com>
> ---

Acked-by: Jesper Dangaard Brouer <brouer@redhat.com>

Thanks Alexei

-- 
Best regards,
  Jesper Dangaard Brouer
  MSc.CS, Sr. Network Kernel Developer at Red Hat
  Author of http://www.iptv-analyzer.org
  LinkedIn: http://www.linkedin.com/in/brouer

^ permalink raw reply

* Re: [PATCH] net: xilinx: Remove unnecessary ether_setup after alloc_etherdev
From: Michal Simek @ 2014-10-01  8:25 UTC (permalink / raw)
  To: Michal Simek, linux-kernel
  Cc: Subbaraya Sundeep Bhatta, Anirudha Sarangi, John Linn,
	David S. Miller, Manuel Schölling, Paul Gortmaker,
	Julia Lawall, Joe Perches, Ricardo Ribalda, Eric Dumazet, netdev,
	linux-arm-kernel
In-Reply-To: <d43b465eca3e2a53c930f6e02dc4ac8b3c084693.1412143216.git.michal.simek@xilinx.com>

[-- Attachment #1: Type: text/plain, Size: 1965 bytes --]

On 10/01/2014 08:00 AM, Michal Simek wrote:
> From: Subbaraya Sundeep Bhatta <subbaraya.sundeep.bhatta@xilinx.com>
> 
> Calling ether_setup is redundant since alloc_etherdev calls it.
> 
> Signed-off-by: Subbaraya Sundeep Bhatta <sbhatta@xilinx.com>
> Signed-off-by: Michal Simek <michal.simek@xilinx.com>
> ---
> 
>  drivers/net/ethernet/xilinx/ll_temac_main.c       | 1 -
>  drivers/net/ethernet/xilinx/xilinx_axienet_main.c | 1 -
>  2 files changed, 2 deletions(-)
> 
> diff --git a/drivers/net/ethernet/xilinx/ll_temac_main.c b/drivers/net/ethernet/xilinx/ll_temac_main.c
> index fda5891835d4..629077050fce 100644
> --- a/drivers/net/ethernet/xilinx/ll_temac_main.c
> +++ b/drivers/net/ethernet/xilinx/ll_temac_main.c
> @@ -1012,7 +1012,6 @@ static int temac_of_probe(struct platform_device *op)
>  	if (!ndev)
>  		return -ENOMEM;
> 
> -	ether_setup(ndev);
>  	platform_set_drvdata(op, ndev);
>  	SET_NETDEV_DEV(ndev, &op->dev);
>  	ndev->flags &= ~IFF_MULTICAST;  /* clear multicast */
> diff --git a/drivers/net/ethernet/xilinx/xilinx_axienet_main.c b/drivers/net/ethernet/xilinx/xilinx_axienet_main.c
> index c8fd94133ecd..4ea2d4e6f1d1 100644
> --- a/drivers/net/ethernet/xilinx/xilinx_axienet_main.c
> +++ b/drivers/net/ethernet/xilinx/xilinx_axienet_main.c
> @@ -1485,7 +1485,6 @@ static int axienet_of_probe(struct platform_device *op)
>  	if (!ndev)
>  		return -ENOMEM;
> 
> -	ether_setup(ndev);
>  	platform_set_drvdata(op, ndev);
> 
>  	SET_NETDEV_DEV(ndev, &op->dev);
> --
> 1.8.2.3
> 

Please ignore this one it is already applied for axienet.
Will send v2 just for ll_temac.

Thanks,
Michal

-- 
Michal Simek, Ing. (M.Eng), OpenPGP -> KeyID: FE3D1F91
w: www.monstr.eu p: +42-0-721842854
Maintainer of Linux kernel - Microblaze cpu - http://www.monstr.eu/fdt/
Maintainer of Linux kernel - Xilinx Zynq ARM architecture
Microblaze U-BOOT custodian and responsible for u-boot arm zynq platform



[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

^ permalink raw reply

* Re: [net-next v3 20/29] fm10k: Add support for netdev offloads
From: Or Gerlitz @ 2014-10-01  8:31 UTC (permalink / raw)
  To: Jeff Kirsher, Alexander Duyck
  Cc: David Miller, Linux Netdev List, nhorman, sassmann, jogreene,
	Tom Herbert
In-Reply-To: <1411471017-28213-21-git-send-email-jeffrey.t.kirsher@intel.com>

On Tue, Sep 23, 2014 at 2:16 PM, Jeff Kirsher
<jeffrey.t.kirsher@intel.com> wrote:
> From: Alexander Duyck <alexander.h.duyck@intel.com>
> This patch adds support for basic offloads including TSO, Tx checksum, Rx
> checksum, Rx hash, and the same features applied to VXLAN/NVGRE tunnels.

> --- a/drivers/net/ethernet/intel/fm10k/fm10k_main.c
> +++ b/drivers/net/ethernet/intel/fm10k/fm10k_main.c
[...]
> +#define VXLAN_HLEN (sizeof(struct udphdr) + 8)
> +static struct ethhdr *fm10k_port_is_vxlan(struct sk_buff *skb)
> +{
> +       struct fm10k_intfc *interface = netdev_priv(skb->dev);
> +       struct fm10k_vxlan_port *vxlan_port;
> +
> +       /* we can only offload a vxlan if we recognize it as such */
> +       vxlan_port = list_first_entry_or_null(&interface->vxlan_port,
> +                                             struct fm10k_vxlan_port, list);
> +
> +       if (!vxlan_port)
> +               return NULL;
> +       if (vxlan_port->port != udp_hdr(skb)->dest)
> +               return NULL;
> +
> +       /* return offset of udp_hdr plus 8 bytes for VXLAN header */
> +       return (struct ethhdr *)(skb_transport_header(skb) + VXLAN_HLEN);
> +}
> +
> +#define FM10K_NVGRE_RESERVED0_FLAGS htons(0x9FFF)
> +#define NVGRE_TNI htons(0x2000)
> +struct fm10k_nvgre_hdr {
> +       __be16 flags;
> +       __be16 proto;
> +       __be32 tni;
> +};
> +
> +static struct ethhdr *fm10k_gre_is_nvgre(struct sk_buff *skb)
> +{
> +       struct fm10k_nvgre_hdr *nvgre_hdr;
> +       int hlen = ip_hdrlen(skb);
> +
> +       /* currently only IPv4 is supported due to hlen above */
> +       if (vlan_get_protocol(skb) != htons(ETH_P_IP))
> +               return NULL;
> +
> +       /* our transport header should be NVGRE */
> +       nvgre_hdr = (struct fm10k_nvgre_hdr *)(skb_network_header(skb) + hlen);
> +
> +       /* verify all reserved flags are 0 */
> +       if (nvgre_hdr->flags & FM10K_NVGRE_RESERVED0_FLAGS)
> +               return NULL;
> +
> +       /* verify protocol is transparent Ethernet bridging */
> +       if (nvgre_hdr->proto != htons(ETH_P_TEB))
> +               return NULL;
> +
> +       /* report start of ethernet header */
> +       if (nvgre_hdr->flags & NVGRE_TNI)
> +               return (struct ethhdr *)(nvgre_hdr + 1);
> +
> +       return (struct ethhdr *)(&nvgre_hdr->tni);
> +}

this helper is fully generic, should reside elsewhere in the stack


> +
> +static __be16 fm10k_tx_encap_offload(struct sk_buff *skb)
> +{
> +       struct ethhdr *eth_hdr;
> +       u8 l4_hdr = 0;
> +
> +       switch (vlan_get_protocol(skb)) {
> +       case htons(ETH_P_IP):
> +               l4_hdr = ip_hdr(skb)->protocol;
> +               break;
> +       case htons(ETH_P_IPV6):
> +               l4_hdr = ipv6_hdr(skb)->nexthdr;
> +               break;
> +       default:
> +               return 0;
> +       }
> +
> +       switch (l4_hdr) {
> +       case IPPROTO_UDP:
> +               eth_hdr = fm10k_port_is_vxlan(skb);
> +               break;
> +       case IPPROTO_GRE:
> +               eth_hdr = fm10k_gre_is_nvgre(skb);
> +               break;
> +       default:
> +               return 0;
> +       }
> +
> +       if (!eth_hdr)
> +               return 0;
> +
> +       switch (eth_hdr->h_proto) {
> +       case htons(ETH_P_IP):
> +       case htons(ETH_P_IPV6):
> +               break;
> +       default:
> +               return 0;
> +       }
> +
> +       return eth_hdr->h_proto;
> +}
> +
> +static int fm10k_tso(struct fm10k_ring *tx_ring,
> +                    struct fm10k_tx_buffer *first)
> +{
> +       struct sk_buff *skb = first->skb;
> +       struct fm10k_tx_desc *tx_desc;
> +       unsigned char *th;
> +       u8 hdrlen;
> +
> +       if (skb->ip_summed != CHECKSUM_PARTIAL)
> +               return 0;
> +
> +       if (!skb_is_gso(skb))
> +               return 0;
> +
> +       /* compute header lengths */
> +       if (skb->encapsulation) {
> +               if (!fm10k_tx_encap_offload(skb))
> +                       goto err_vxlan;
> +               th = skb_inner_transport_header(skb);
> +       } else {
> +               th = skb_transport_header(skb);
> +       }
> +
> +       /* compute offset from SOF to transport header and add header len */
> +       hdrlen = (th - skb->data) + (((struct tcphdr *)th)->doff << 2);
> +
> +       first->tx_flags |= FM10K_TX_FLAGS_CSUM;
> +
> +       /* update gso size and bytecount with header size */
> +       first->gso_segs = skb_shinfo(skb)->gso_segs;
> +       first->bytecount += (first->gso_segs - 1) * hdrlen;
> +
> +       /* populate Tx descriptor header size and mss */
> +       tx_desc = FM10K_TX_DESC(tx_ring, tx_ring->next_to_use);
> +       tx_desc->hdrlen = hdrlen;
> +       tx_desc->mss = cpu_to_le16(skb_shinfo(skb)->gso_size);
> +
> +       return 1;
> +err_vxlan:
> +       tx_ring->netdev->features &= ~NETIF_F_GSO_UDP_TUNNEL;
> +       if (!net_ratelimit())
> +               netdev_err(tx_ring->netdev,
> +                          "TSO requested for unsupported tunnel, disabling offload\n");
> +       return -1;
> +}

why? if TSO was requested for some packet the driver can't do, you disable GSO
for udp tunnels for any future packets too? maybe just disable it
permanently  till you
feel safer to run under the current stack?


[...]

> @@ -732,6 +1024,7 @@ netdev_tx_t fm10k_xmit_frame_ring(struct sk_buff *skb,
>                                   struct fm10k_ring *tx_ring)
>  {
>         struct fm10k_tx_buffer *first;
> +       int tso;
>         u32 tx_flags = 0;
>  #if PAGE_SIZE > FM10K_MAX_DATA_PER_TXD
>         unsigned short f;
> @@ -763,11 +1056,23 @@ netdev_tx_t fm10k_xmit_frame_ring(struct sk_buff *skb,
>         /* record initial flags and protocol */
>         first->tx_flags = tx_flags;
>
> +       tso = fm10k_tso(tx_ring, first);
> +       if (tso < 0)
> +               goto out_drop;
> +       else if (!tso)
> +               fm10k_tx_csum(tx_ring, first);
> +
>         fm10k_tx_map(tx_ring, first);
>
>         fm10k_maybe_stop_tx(tx_ring, DESC_NEEDED);
>
>         return NETDEV_TX_OK;
> +
> +out_drop:
> +       dev_kfree_skb_any(first->skb);
> +       first->skb = NULL;
> +
> +       return NETDEV_TX_OK;
>  }

^ permalink raw reply

* [PATCH net-next v6 1/2] bonding: display xmit_hash_policy for non-dynamic-tlb mode
From: Mahesh Bandewar @ 2014-10-01  8:38 UTC (permalink / raw)
  To: Jay Vosburgh, Veaceslav Falico, Andy Gospodarek, David Miller
  Cc: netdev, Mahesh Bandewar, Eric Dumazet, Maciej Zenczykowski

It's a trivial fix to display xmit_hash_policy for this new TLB mode
since it uses transmit-hash-poilicy as part of bonding-master info
(/proc/net/bonding/<bonding-interface).

Signed-off-by: Mahesh Bandewar <maheshb@google.com>
Reviewed-by: Nikolay Aleksandrov <nikolay@redhat.com>
---
v1
 Rebase
v2
 Added bond_mode_uses_xmit_hash() inline function
v3-v6
 Rebase

 drivers/net/bonding/bond_procfs.c | 3 +--
 drivers/net/bonding/bonding.h     | 7 +++++++
 2 files changed, 8 insertions(+), 2 deletions(-)

diff --git a/drivers/net/bonding/bond_procfs.c b/drivers/net/bonding/bond_procfs.c
index bb09d0442aa8..a3948f8d1e53 100644
--- a/drivers/net/bonding/bond_procfs.c
+++ b/drivers/net/bonding/bond_procfs.c
@@ -73,8 +73,7 @@ static void bond_info_show_master(struct seq_file *seq)
 
 	seq_printf(seq, "\n");
 
-	if (BOND_MODE(bond) == BOND_MODE_XOR ||
-		BOND_MODE(bond) == BOND_MODE_8023AD) {
+	if (bond_mode_uses_xmit_hash(bond)) {
 		optval = bond_opt_get_val(BOND_OPT_XMIT_HASH,
 					  bond->params.xmit_policy);
 		seq_printf(seq, "Transmit Hash Policy: %s (%d)\n",
diff --git a/drivers/net/bonding/bonding.h b/drivers/net/bonding/bonding.h
index 57917e63b4e6..5b022da9cad2 100644
--- a/drivers/net/bonding/bonding.h
+++ b/drivers/net/bonding/bonding.h
@@ -274,6 +274,13 @@ static inline bool bond_is_nondyn_tlb(const struct bonding *bond)
 	       (bond->params.tlb_dynamic_lb == 0);
 }
 
+static inline bool bond_mode_uses_xmit_hash(const struct bonding *bond)
+{
+	return (BOND_MODE(bond) == BOND_MODE_8023AD ||
+		BOND_MODE(bond) == BOND_MODE_XOR ||
+		bond_is_nondyn_tlb(bond));
+}
+
 static inline bool bond_mode_uses_arp(int mode)
 {
 	return mode != BOND_MODE_8023AD && mode != BOND_MODE_TLB &&
-- 
2.1.0.rc2.206.gedb03e5

^ permalink raw reply related

* [PATCH net-next v6 2/2] bonding: Simplify the xmit function for modes that use xmit_hash
From: Mahesh Bandewar @ 2014-10-01  8:38 UTC (permalink / raw)
  To: Jay Vosburgh, Veaceslav Falico, Andy Gospodarek, David Miller
  Cc: netdev, Mahesh Bandewar, Eric Dumazet, Maciej Zenczykowski

Earlier change to use usable slave array for TLB mode had an additional
performance advantage. So extending the same logic to all other modes
that use xmit-hash for slave selection (viz 802.3AD, and XOR modes).
Also consolidating this with the earlier TLB change.

The main idea is to build the usable slaves array in the control path
and use that array for slave selection during xmit operation.

Measured performance in a setup with a bond of 4x1G NICs with 200
instances of netperf for the modes involved (3ad, xor, tlb)
cmd: netperf -t TCP_RR -H <TargetHost> -l 60 -s 5

Mode        TPS-Before   TPS-After

802.3ad   : 468,694      493,101
TLB (lb=0): 392,583      392,965
XOR       : 475,696      484,517

Signed-off-by: Mahesh Bandewar <maheshb@google.com>
---
v1:
  (a) If bond_update_slave_arr() fails to allocate memory, it will overwrite
      the slave that need to be removed.
  (b) Freeing of array will assign NULL (to handle bond->down to bond->up
      transition gracefully.
  (c) Change from pr_debug() to pr_err() if bond_update_slave_arr() returns
      failure.
  (d) XOR: bond_update_slave_arr() will consider mii-mon, arp-mon cases and
      will populate the array even if these parameters are not used.
  (e) 3AD: Should handle the ad_agg_selection_logic correctly.
v2:
  (a) Removed rcu_read_{un}lock() calls from array manipulation code.
  (b) Slave link-events now refresh array for all these modes.
  (c) Moved free-array call from bond_close() to bond_uninit().
v3:
  (a) Fixed null pointer dereference.
  (b) Removed bond->lock lockdep dependency.
v4:
  (a) Made to changes to comply with Nikolay's locking changes
  (b) Added a work-queue to refresh slave-array when RTNL is not held
  (c) Array refresh happens ONLY with RTNL now.
  (d) alloc changed from GFP_ATOMIC to GFP_KERNEL
v5:
  (a) Consolidated all delayed slave-array updates at one place in
      3ad_state_machine_handler()
v6:
  (a) Free slave array when there is no active aggregator

 drivers/net/bonding/bond_3ad.c  | 140 +++++++++++------------------
 drivers/net/bonding/bond_alb.c  |  51 ++---------
 drivers/net/bonding/bond_alb.h  |   8 --
 drivers/net/bonding/bond_main.c | 192 +++++++++++++++++++++++++++++++++++++---
 drivers/net/bonding/bonding.h   |  10 +++
 5 files changed, 249 insertions(+), 152 deletions(-)

diff --git a/drivers/net/bonding/bond_3ad.c b/drivers/net/bonding/bond_3ad.c
index 7e9e522fd476..2110215f3528 100644
--- a/drivers/net/bonding/bond_3ad.c
+++ b/drivers/net/bonding/bond_3ad.c
@@ -102,17 +102,20 @@ static const u8 lacpdu_mcast_addr[ETH_ALEN] = MULTICAST_LACPDU_ADDR;
 /* ================= main 802.3ad protocol functions ================== */
 static int ad_lacpdu_send(struct port *port);
 static int ad_marker_send(struct port *port, struct bond_marker *marker);
-static void ad_mux_machine(struct port *port);
+static void ad_mux_machine(struct port *port, bool *update_slave_arr);
 static void ad_rx_machine(struct lacpdu *lacpdu, struct port *port);
 static void ad_tx_machine(struct port *port);
 static void ad_periodic_machine(struct port *port);
-static void ad_port_selection_logic(struct port *port);
-static void ad_agg_selection_logic(struct aggregator *aggregator);
+static void ad_port_selection_logic(struct port *port, bool *update_slave_arr);
+static void ad_agg_selection_logic(struct aggregator *aggregator,
+				   bool *update_slave_arr);
 static void ad_clear_agg(struct aggregator *aggregator);
 static void ad_initialize_agg(struct aggregator *aggregator);
 static void ad_initialize_port(struct port *port, int lacp_fast);
-static void ad_enable_collecting_distributing(struct port *port);
-static void ad_disable_collecting_distributing(struct port *port);
+static void ad_enable_collecting_distributing(struct port *port,
+					      bool *update_slave_arr);
+static void ad_disable_collecting_distributing(struct port *port,
+					       bool *update_slave_arr);
 static void ad_marker_info_received(struct bond_marker *marker_info,
 				    struct port *port);
 static void ad_marker_response_received(struct bond_marker *marker,
@@ -796,8 +799,9 @@ static int ad_marker_send(struct port *port, struct bond_marker *marker)
 /**
  * ad_mux_machine - handle a port's mux state machine
  * @port: the port we're looking at
+ * @update_slave_arr: Does slave array need update?
  */
-static void ad_mux_machine(struct port *port)
+static void ad_mux_machine(struct port *port, bool *update_slave_arr)
 {
 	mux_states_t last_state;
 
@@ -901,7 +905,8 @@ static void ad_mux_machine(struct port *port)
 		switch (port->sm_mux_state) {
 		case AD_MUX_DETACHED:
 			port->actor_oper_port_state &= ~AD_STATE_SYNCHRONIZATION;
-			ad_disable_collecting_distributing(port);
+			ad_disable_collecting_distributing(port,
+							   update_slave_arr);
 			port->actor_oper_port_state &= ~AD_STATE_COLLECTING;
 			port->actor_oper_port_state &= ~AD_STATE_DISTRIBUTING;
 			port->ntt = true;
@@ -913,13 +918,15 @@ static void ad_mux_machine(struct port *port)
 			port->actor_oper_port_state |= AD_STATE_SYNCHRONIZATION;
 			port->actor_oper_port_state &= ~AD_STATE_COLLECTING;
 			port->actor_oper_port_state &= ~AD_STATE_DISTRIBUTING;
-			ad_disable_collecting_distributing(port);
+			ad_disable_collecting_distributing(port,
+							   update_slave_arr);
 			port->ntt = true;
 			break;
 		case AD_MUX_COLLECTING_DISTRIBUTING:
 			port->actor_oper_port_state |= AD_STATE_COLLECTING;
 			port->actor_oper_port_state |= AD_STATE_DISTRIBUTING;
-			ad_enable_collecting_distributing(port);
+			ad_enable_collecting_distributing(port,
+							  update_slave_arr);
 			port->ntt = true;
 			break;
 		default:
@@ -1187,12 +1194,13 @@ static void ad_periodic_machine(struct port *port)
 /**
  * ad_port_selection_logic - select aggregation groups
  * @port: the port we're looking at
+ * @update_slave_arr: Does slave array need update?
  *
  * Select aggregation groups, and assign each port for it's aggregetor. The
  * selection logic is called in the inititalization (after all the handshkes),
  * and after every lacpdu receive (if selected is off).
  */
-static void ad_port_selection_logic(struct port *port)
+static void ad_port_selection_logic(struct port *port, bool *update_slave_arr)
 {
 	struct aggregator *aggregator, *free_aggregator = NULL, *temp_aggregator;
 	struct port *last_port = NULL, *curr_port;
@@ -1347,7 +1355,7 @@ static void ad_port_selection_logic(struct port *port)
 			      __agg_ports_are_ready(port->aggregator));
 
 	aggregator = __get_first_agg(port);
-	ad_agg_selection_logic(aggregator);
+	ad_agg_selection_logic(aggregator, update_slave_arr);
 }
 
 /* Decide if "agg" is a better choice for the new active aggregator that
@@ -1435,6 +1443,7 @@ static int agg_device_up(const struct aggregator *agg)
 /**
  * ad_agg_selection_logic - select an aggregation group for a team
  * @aggregator: the aggregator we're looking at
+ * @update_slave_arr: Does slave array need update?
  *
  * It is assumed that only one aggregator may be selected for a team.
  *
@@ -1457,7 +1466,8 @@ static int agg_device_up(const struct aggregator *agg)
  * __get_active_agg() won't work correctly. This function should be better
  * called with the bond itself, and retrieve the first agg from it.
  */
-static void ad_agg_selection_logic(struct aggregator *agg)
+static void ad_agg_selection_logic(struct aggregator *agg,
+				   bool *update_slave_arr)
 {
 	struct aggregator *best, *active, *origin;
 	struct bonding *bond = agg->slave->bond;
@@ -1550,6 +1560,8 @@ static void ad_agg_selection_logic(struct aggregator *agg)
 				__disable_port(port);
 			}
 		}
+		/* Slave array needs update. */
+		*update_slave_arr = true;
 	}
 
 	/* if the selected aggregator is of join individuals
@@ -1678,24 +1690,30 @@ static void ad_initialize_port(struct port *port, int lacp_fast)
 /**
  * ad_enable_collecting_distributing - enable a port's transmit/receive
  * @port: the port we're looking at
+ * @update_slave_arr: Does slave array need update?
  *
  * Enable @port if it's in an active aggregator
  */
-static void ad_enable_collecting_distributing(struct port *port)
+static void ad_enable_collecting_distributing(struct port *port,
+					      bool *update_slave_arr)
 {
 	if (port->aggregator->is_active) {
 		pr_debug("Enabling port %d(LAG %d)\n",
 			 port->actor_port_number,
 			 port->aggregator->aggregator_identifier);
 		__enable_port(port);
+		/* Slave array needs update */
+		*update_slave_arr = true;
 	}
 }
 
 /**
  * ad_disable_collecting_distributing - disable a port's transmit/receive
  * @port: the port we're looking at
+ * @update_slave_arr: Does slave array need update?
  */
-static void ad_disable_collecting_distributing(struct port *port)
+static void ad_disable_collecting_distributing(struct port *port,
+					       bool *update_slave_arr)
 {
 	if (port->aggregator &&
 	    !MAC_ADDRESS_EQUAL(&(port->aggregator->partner_system),
@@ -1704,6 +1722,8 @@ static void ad_disable_collecting_distributing(struct port *port)
 			 port->actor_port_number,
 			 port->aggregator->aggregator_identifier);
 		__disable_port(port);
+		/* Slave array needs an update */
+		*update_slave_arr = true;
 	}
 }
 
@@ -1868,6 +1888,7 @@ void bond_3ad_unbind_slave(struct slave *slave)
 	struct bonding *bond = slave->bond;
 	struct slave *slave_iter;
 	struct list_head *iter;
+	bool dummy_slave_update; /* Ignore this value as caller updates array */
 
 	/* Sync against bond_3ad_state_machine_handler() */
 	spin_lock_bh(&bond->mode_lock);
@@ -1951,7 +1972,8 @@ void bond_3ad_unbind_slave(struct slave *slave)
 				ad_clear_agg(aggregator);
 
 				if (select_new_active_agg)
-					ad_agg_selection_logic(__get_first_agg(port));
+					ad_agg_selection_logic(__get_first_agg(port),
+							       &dummy_slave_update);
 			} else {
 				netdev_warn(bond->dev, "unbinding aggregator, and could not find a new aggregator for its ports\n");
 			}
@@ -1966,7 +1988,8 @@ void bond_3ad_unbind_slave(struct slave *slave)
 				/* select new active aggregator */
 				temp_aggregator = __get_first_agg(port);
 				if (temp_aggregator)
-					ad_agg_selection_logic(temp_aggregator);
+					ad_agg_selection_logic(temp_aggregator,
+							       &dummy_slave_update);
 			}
 		}
 	}
@@ -1996,7 +2019,8 @@ void bond_3ad_unbind_slave(struct slave *slave)
 					if (select_new_active_agg) {
 						netdev_info(bond->dev, "Removing an active aggregator\n");
 						/* select new active aggregator */
-						ad_agg_selection_logic(__get_first_agg(port));
+						ad_agg_selection_logic(__get_first_agg(port),
+							               &dummy_slave_update);
 					}
 				}
 				break;
@@ -2031,6 +2055,7 @@ void bond_3ad_state_machine_handler(struct work_struct *work)
 	struct slave *slave;
 	struct port *port;
 	bool should_notify_rtnl = BOND_SLAVE_NOTIFY_LATER;
+	bool update_slave_arr = false;
 
 	/* Lock to protect data accessed by all (e.g., port->sm_vars) and
 	 * against running with bond_3ad_unbind_slave. ad_rx_machine may run
@@ -2058,7 +2083,7 @@ void bond_3ad_state_machine_handler(struct work_struct *work)
 			}
 
 			aggregator = __get_first_agg(port);
-			ad_agg_selection_logic(aggregator);
+			ad_agg_selection_logic(aggregator, &update_slave_arr);
 		}
 		bond_3ad_set_carrier(bond);
 	}
@@ -2074,8 +2099,8 @@ void bond_3ad_state_machine_handler(struct work_struct *work)
 
 		ad_rx_machine(NULL, port);
 		ad_periodic_machine(port);
-		ad_port_selection_logic(port);
-		ad_mux_machine(port);
+		ad_port_selection_logic(port, &update_slave_arr);
+		ad_mux_machine(port, &update_slave_arr);
 		ad_tx_machine(port);
 
 		/* turn off the BEGIN bit, since we already handled it */
@@ -2093,6 +2118,9 @@ re_arm:
 	rcu_read_unlock();
 	spin_unlock_bh(&bond->mode_lock);
 
+	if (update_slave_arr)
+		bond_slave_arr_work_rearm(bond, 0);
+
 	if (should_notify_rtnl && rtnl_trylock()) {
 		bond_slave_state_notify(bond);
 		rtnl_unlock();
@@ -2283,6 +2311,11 @@ void bond_3ad_handle_link_change(struct slave *slave, char link)
 	port->sm_vars |= AD_PORT_BEGIN;
 
 	spin_unlock_bh(&slave->bond->mode_lock);
+
+	/* RTNL is held and mode_lock is released so it's safe
+	 * to update slave_array here.
+	 */
+	bond_update_slave_arr(slave->bond, NULL);
 }
 
 /**
@@ -2377,73 +2410,6 @@ int bond_3ad_get_active_agg_info(struct bonding *bond, struct ad_info *ad_info)
 	return ret;
 }
 
-int bond_3ad_xmit_xor(struct sk_buff *skb, struct net_device *dev)
-{
-	struct bonding *bond = netdev_priv(dev);
-	struct slave *slave, *first_ok_slave;
-	struct aggregator *agg;
-	struct ad_info ad_info;
-	struct list_head *iter;
-	int slaves_in_agg;
-	int slave_agg_no;
-	int agg_id;
-
-	if (__bond_3ad_get_active_agg_info(bond, &ad_info)) {
-		netdev_dbg(dev, "__bond_3ad_get_active_agg_info failed\n");
-		goto err_free;
-	}
-
-	slaves_in_agg = ad_info.ports;
-	agg_id = ad_info.aggregator_id;
-
-	if (slaves_in_agg == 0) {
-		netdev_dbg(dev, "active aggregator is empty\n");
-		goto err_free;
-	}
-
-	slave_agg_no = bond_xmit_hash(bond, skb) % slaves_in_agg;
-	first_ok_slave = NULL;
-
-	bond_for_each_slave_rcu(bond, slave, iter) {
-		agg = SLAVE_AD_INFO(slave)->port.aggregator;
-		if (!agg || agg->aggregator_identifier != agg_id)
-			continue;
-
-		if (slave_agg_no >= 0) {
-			if (!first_ok_slave && bond_slave_can_tx(slave))
-				first_ok_slave = slave;
-			slave_agg_no--;
-			continue;
-		}
-
-		if (bond_slave_can_tx(slave)) {
-			bond_dev_queue_xmit(bond, skb, slave->dev);
-			goto out;
-		}
-	}
-
-	if (slave_agg_no >= 0) {
-		netdev_err(dev, "Couldn't find a slave to tx on for aggregator ID %d\n",
-			   agg_id);
-		goto err_free;
-	}
-
-	/* we couldn't find any suitable slave after the agg_no, so use the
-	 * first suitable found, if found.
-	 */
-	if (first_ok_slave)
-		bond_dev_queue_xmit(bond, skb, first_ok_slave->dev);
-	else
-		goto err_free;
-
-out:
-	return NETDEV_TX_OK;
-err_free:
-	/* no suitable interface, frame not sent */
-	dev_kfree_skb_any(skb);
-	goto out;
-}
-
 int bond_3ad_lacpdu_recv(const struct sk_buff *skb, struct bonding *bond,
 			 struct slave *slave)
 {
diff --git a/drivers/net/bonding/bond_alb.c b/drivers/net/bonding/bond_alb.c
index 615f3bebd019..d2eadab787c5 100644
--- a/drivers/net/bonding/bond_alb.c
+++ b/drivers/net/bonding/bond_alb.c
@@ -177,7 +177,6 @@ static int tlb_initialize(struct bonding *bond)
 static void tlb_deinitialize(struct bonding *bond)
 {
 	struct alb_bond_info *bond_info = &(BOND_ALB_INFO(bond));
-	struct tlb_up_slave *arr;
 
 	spin_lock_bh(&bond->mode_lock);
 
@@ -185,10 +184,6 @@ static void tlb_deinitialize(struct bonding *bond)
 	bond_info->tx_hashtbl = NULL;
 
 	spin_unlock_bh(&bond->mode_lock);
-
-	arr = rtnl_dereference(bond_info->slave_arr);
-	if (arr)
-		kfree_rcu(arr, rcu);
 }
 
 static long long compute_gap(struct slave *slave)
@@ -1336,39 +1331,9 @@ out:
 	return NETDEV_TX_OK;
 }
 
-static int bond_tlb_update_slave_arr(struct bonding *bond,
-				     struct slave *skipslave)
-{
-	struct alb_bond_info *bond_info = &(BOND_ALB_INFO(bond));
-	struct slave *tx_slave;
-	struct list_head *iter;
-	struct tlb_up_slave *new_arr, *old_arr;
-
-	new_arr = kzalloc(offsetof(struct tlb_up_slave, arr[bond->slave_cnt]),
-			  GFP_ATOMIC);
-	if (!new_arr)
-		return -ENOMEM;
-
-	bond_for_each_slave(bond, tx_slave, iter) {
-		if (!bond_slave_can_tx(tx_slave))
-			continue;
-		if (skipslave == tx_slave)
-			continue;
-		new_arr->arr[new_arr->count++] = tx_slave;
-	}
-
-	old_arr = rtnl_dereference(bond_info->slave_arr);
-	rcu_assign_pointer(bond_info->slave_arr, new_arr);
-	if (old_arr)
-		kfree_rcu(old_arr, rcu);
-
-	return 0;
-}
-
 int bond_tlb_xmit(struct sk_buff *skb, struct net_device *bond_dev)
 {
 	struct bonding *bond = netdev_priv(bond_dev);
-	struct alb_bond_info *bond_info = &(BOND_ALB_INFO(bond));
 	struct ethhdr *eth_data;
 	struct slave *tx_slave = NULL;
 	u32 hash_index;
@@ -1389,12 +1354,14 @@ int bond_tlb_xmit(struct sk_buff *skb, struct net_device *bond_dev)
 							      hash_index & 0xFF,
 							      skb->len);
 			} else {
-				struct tlb_up_slave *slaves;
+				struct bond_up_slave *slaves;
+				unsigned int count;
 
-				slaves = rcu_dereference(bond_info->slave_arr);
-				if (slaves && slaves->count)
+				slaves = rcu_dereference(bond->slave_arr);
+				count = slaves ? ACCESS_ONCE(slaves->count) : 0;
+				if (likely(count))
 					tx_slave = slaves->arr[hash_index %
-							       slaves->count];
+							       count];
 			}
 			break;
 		}
@@ -1641,10 +1608,6 @@ void bond_alb_deinit_slave(struct bonding *bond, struct slave *slave)
 		rlb_clear_slave(bond, slave);
 	}
 
-	if (bond_is_nondyn_tlb(bond))
-		if (bond_tlb_update_slave_arr(bond, slave))
-			pr_err("Failed to build slave-array for TLB mode.\n");
-
 }
 
 void bond_alb_handle_link_change(struct bonding *bond, struct slave *slave, char link)
@@ -1669,7 +1632,7 @@ void bond_alb_handle_link_change(struct bonding *bond, struct slave *slave, char
 	}
 
 	if (bond_is_nondyn_tlb(bond)) {
-		if (bond_tlb_update_slave_arr(bond, NULL))
+		if (bond_update_slave_arr(bond, NULL))
 			pr_err("Failed to build slave-array for TLB mode.\n");
 	}
 }
diff --git a/drivers/net/bonding/bond_alb.h b/drivers/net/bonding/bond_alb.h
index 3c6a7ff974d7..1ad473b4ade5 100644
--- a/drivers/net/bonding/bond_alb.h
+++ b/drivers/net/bonding/bond_alb.h
@@ -139,19 +139,11 @@ struct tlb_slave_info {
 			 */
 };
 
-struct tlb_up_slave {
-	unsigned int	count;
-	struct rcu_head rcu;
-	struct slave	*arr[0];
-};
-
 struct alb_bond_info {
 	struct tlb_client_info	*tx_hashtbl; /* Dynamically allocated */
 	u32			unbalanced_load;
 	int			tx_rebalance_counter;
 	int			lp_counter;
-	/* -------- non-dynamic tlb mode only ---------*/
-	struct tlb_up_slave __rcu *slave_arr;	  /* Up slaves */
 	/* -------- rlb parameters -------- */
 	int rlb_enabled;
 	struct rlb_client_info	*rx_hashtbl;	/* Receive hash table */
diff --git a/drivers/net/bonding/bond_main.c b/drivers/net/bonding/bond_main.c
index c2adc2755ff6..6f79f495b01a 100644
--- a/drivers/net/bonding/bond_main.c
+++ b/drivers/net/bonding/bond_main.c
@@ -210,6 +210,7 @@ static int bond_init(struct net_device *bond_dev);
 static void bond_uninit(struct net_device *bond_dev);
 static struct rtnl_link_stats64 *bond_get_stats(struct net_device *bond_dev,
 						struct rtnl_link_stats64 *stats);
+static void bond_slave_arr_handler(struct work_struct *work);
 
 /*---------------------------- General routines -----------------------------*/
 
@@ -1551,6 +1552,9 @@ int bond_enslave(struct net_device *bond_dev, struct net_device *slave_dev)
 		unblock_netpoll_tx();
 	}
 
+	if (bond_mode_uses_xmit_hash(bond))
+		bond_update_slave_arr(bond, NULL);
+
 	netdev_info(bond_dev, "Enslaving %s as %s interface with %s link\n",
 		    slave_dev->name,
 		    bond_is_active_slave(new_slave) ? "an active" : "a backup",
@@ -1668,6 +1672,9 @@ static int __bond_release_one(struct net_device *bond_dev,
 	if (BOND_MODE(bond) == BOND_MODE_8023AD)
 		bond_3ad_unbind_slave(slave);
 
+	if (bond_mode_uses_xmit_hash(bond))
+		bond_update_slave_arr(bond, slave);
+
 	netdev_info(bond_dev, "Releasing %s interface %s\n",
 		    bond_is_active_slave(slave) ? "active" : "backup",
 		    slave_dev->name);
@@ -1970,6 +1977,9 @@ static void bond_miimon_commit(struct bonding *bond)
 				bond_alb_handle_link_change(bond, slave,
 							    BOND_LINK_UP);
 
+			if (BOND_MODE(bond) == BOND_MODE_XOR)
+				bond_update_slave_arr(bond, NULL);
+
 			if (!bond->curr_active_slave || slave == primary)
 				goto do_failover;
 
@@ -1997,6 +2007,9 @@ static void bond_miimon_commit(struct bonding *bond)
 				bond_alb_handle_link_change(bond, slave,
 							    BOND_LINK_DOWN);
 
+			if (BOND_MODE(bond) == BOND_MODE_XOR)
+				bond_update_slave_arr(bond, NULL);
+
 			if (slave == rcu_access_pointer(bond->curr_active_slave))
 				goto do_failover;
 
@@ -2453,6 +2466,8 @@ static void bond_loadbalance_arp_mon(struct work_struct *work)
 
 		if (slave_state_changed) {
 			bond_slave_state_change(bond);
+			if (BOND_MODE(bond) == BOND_MODE_XOR)
+				bond_update_slave_arr(bond, NULL);
 		} else if (do_failover) {
 			block_netpoll_tx();
 			bond_select_active_slave(bond);
@@ -2829,8 +2844,20 @@ static int bond_slave_netdev_event(unsigned long event,
 			if (old_duplex != slave->duplex)
 				bond_3ad_adapter_duplex_changed(slave);
 		}
+		/* Refresh slave-array if applicable!
+		 * If the setup does not use miimon or arpmon (mode-specific!),
+		 * then these events will not cause the slave-array to be
+		 * refreshed. This will cause xmit to use a slave that is not
+		 * usable. Avoid such situation by refeshing the array at these
+		 * events. If these (miimon/arpmon) parameters are configured
+		 * then array gets refreshed twice and that should be fine!
+		 */
+		if (bond_mode_uses_xmit_hash(bond))
+			bond_update_slave_arr(bond, NULL);
 		break;
 	case NETDEV_DOWN:
+		if (bond_mode_uses_xmit_hash(bond))
+			bond_update_slave_arr(bond, NULL);
 		break;
 	case NETDEV_CHANGEMTU:
 		/* TODO: Should slaves be allowed to
@@ -3010,6 +3037,7 @@ static void bond_work_init_all(struct bonding *bond)
 	else
 		INIT_DELAYED_WORK(&bond->arp_work, bond_loadbalance_arp_mon);
 	INIT_DELAYED_WORK(&bond->ad_work, bond_3ad_state_machine_handler);
+	INIT_DELAYED_WORK(&bond->slave_arr_work, bond_slave_arr_handler);
 }
 
 static void bond_work_cancel_all(struct bonding *bond)
@@ -3019,6 +3047,7 @@ static void bond_work_cancel_all(struct bonding *bond)
 	cancel_delayed_work_sync(&bond->alb_work);
 	cancel_delayed_work_sync(&bond->ad_work);
 	cancel_delayed_work_sync(&bond->mcast_work);
+	cancel_delayed_work_sync(&bond->slave_arr_work);
 }
 
 static int bond_open(struct net_device *bond_dev)
@@ -3068,6 +3097,9 @@ static int bond_open(struct net_device *bond_dev)
 		bond_3ad_initiate_agg_selection(bond, 1);
 	}
 
+	if (bond_mode_uses_xmit_hash(bond))
+		bond_update_slave_arr(bond, NULL);
+
 	return 0;
 }
 
@@ -3573,20 +3605,148 @@ static int bond_xmit_activebackup(struct sk_buff *skb, struct net_device *bond_d
 	return NETDEV_TX_OK;
 }
 
-/* In bond_xmit_xor() , we determine the output device by using a pre-
- * determined xmit_hash_policy(), If the selected device is not enabled,
- * find the next active slave.
+/* Use this to update slave_array when (a) it's not appropriate to update
+ * slave_array right away (note that update_slave_array() may sleep)
+ * and / or (b) RTNL is not held.
  */
-static int bond_xmit_xor(struct sk_buff *skb, struct net_device *bond_dev)
+void bond_slave_arr_work_rearm(struct bonding *bond, unsigned long delay)
 {
-	struct bonding *bond = netdev_priv(bond_dev);
-	int slave_cnt = ACCESS_ONCE(bond->slave_cnt);
+	queue_delayed_work(bond->wq, &bond->slave_arr_work, delay);
+}
 
-	if (likely(slave_cnt))
-		bond_xmit_slave_id(bond, skb,
-				   bond_xmit_hash(bond, skb) % slave_cnt);
-	else
+/* Slave array work handler. Holds only RTNL */
+static void bond_slave_arr_handler(struct work_struct *work)
+{
+	struct bonding *bond = container_of(work, struct bonding,
+					    slave_arr_work.work);
+	int ret;
+
+	if (!rtnl_trylock())
+		goto err;
+
+	ret = bond_update_slave_arr(bond, NULL);
+	rtnl_unlock();
+	if (ret) {
+		pr_warn_ratelimited("Failed to update slave array from WT\n");
+		goto err;
+	}
+	return;
+
+err:
+	bond_slave_arr_work_rearm(bond, 1);
+}
+
+/* Build the usable slaves array in control path for modes that use xmit-hash
+ * to determine the slave interface -
+ * (a) BOND_MODE_8023AD
+ * (b) BOND_MODE_XOR
+ * (c) BOND_MODE_TLB && tlb_dynamic_lb == 0
+ *
+ * The caller is expected to hold RTNL only and NO other lock!
+ */
+int bond_update_slave_arr(struct bonding *bond, struct slave *skipslave)
+{
+	struct slave *slave;
+	struct list_head *iter;
+	struct bond_up_slave *new_arr, *old_arr;
+	int slaves_in_agg;
+	int agg_id = 0;
+	int ret = 0;
+
+#ifdef CONFIG_LOCKDEP
+	WARN_ON(lockdep_is_held(&bond->mode_lock));
+#endif
+
+	new_arr = kzalloc(offsetof(struct bond_up_slave, arr[bond->slave_cnt]),
+			  GFP_KERNEL);
+	if (!new_arr) {
+		ret = -ENOMEM;
+		pr_err("Failed to build slave-array.\n");
+		goto out;
+	}
+	if (BOND_MODE(bond) == BOND_MODE_8023AD) {
+		struct ad_info ad_info;
+
+		if (bond_3ad_get_active_agg_info(bond, &ad_info)) {
+			pr_debug("bond_3ad_get_active_agg_info failed\n");
+			kfree_rcu(new_arr, rcu);
+			/* No active aggragator means its not safe to use
+			 * the previous array.
+			 */
+			old_arr = rtnl_dereference(bond->slave_arr);
+			if (old_arr) {
+				RCU_INIT_POINTER(bond->slave_arr, NULL);
+				kfree_rcu(old_arr, rcu);
+			}
+			goto out;
+		}
+		slaves_in_agg = ad_info.ports;
+		agg_id = ad_info.aggregator_id;
+	}
+	bond_for_each_slave(bond, slave, iter) {
+		if (BOND_MODE(bond) == BOND_MODE_8023AD) {
+			struct aggregator *agg;
+
+			agg = SLAVE_AD_INFO(slave)->port.aggregator;
+			if (!agg || agg->aggregator_identifier != agg_id)
+				continue;
+		}
+		if (!bond_slave_can_tx(slave))
+			continue;
+		if (skipslave == slave)
+			continue;
+		new_arr->arr[new_arr->count++] = slave;
+	}
+
+	old_arr = rtnl_dereference(bond->slave_arr);
+	rcu_assign_pointer(bond->slave_arr, new_arr);
+	if (old_arr)
+		kfree_rcu(old_arr, rcu);
+out:
+	if (ret != 0 && skipslave) {
+		int idx;
+
+		/* Rare situation where caller has asked to skip a specific
+		 * slave but allocation failed (most likely!). BTW this is
+		 * only possible when the call is initiated from
+		 * __bond_release_one(). In this situation; overwrite the
+		 * skipslave entry in the array with the last entry from the
+		 * array to avoid a situation where the xmit path may choose
+		 * this to-be-skipped slave to send a packet out.
+		 */
+		old_arr = rtnl_dereference(bond->slave_arr);
+		for (idx = 0; idx < old_arr->count; idx++) {
+			if (skipslave == old_arr->arr[idx]) {
+				old_arr->arr[idx] =
+				    old_arr->arr[old_arr->count-1];
+				old_arr->count--;
+				break;
+			}
+		}
+	}
+	return ret;
+}
+
+/* Use this Xmit function for 3AD as well as XOR modes. The current
+ * usable slave array is formed in the control path. The xmit function
+ * just calculates hash and sends the packet out.
+ */
+int bond_3ad_xor_xmit(struct sk_buff *skb, struct net_device *dev)
+{
+	struct bonding *bond = netdev_priv(dev);
+	struct slave *slave;
+	struct bond_up_slave *slaves;
+	unsigned int count;
+
+	slaves = rcu_dereference(bond->slave_arr);
+	count = slaves ? ACCESS_ONCE(slaves->count) : 0;
+	if (likely(count)) {
+		slave = slaves->arr[bond_xmit_hash(bond, skb) % count];
+		bond_dev_queue_xmit(bond, skb, slave->dev);
+	} else {
 		dev_kfree_skb_any(skb);
+		atomic_long_inc(&dev->tx_dropped);
+	}
 
 	return NETDEV_TX_OK;
 }
@@ -3682,12 +3842,11 @@ static netdev_tx_t __bond_start_xmit(struct sk_buff *skb, struct net_device *dev
 		return bond_xmit_roundrobin(skb, dev);
 	case BOND_MODE_ACTIVEBACKUP:
 		return bond_xmit_activebackup(skb, dev);
+	case BOND_MODE_8023AD:
 	case BOND_MODE_XOR:
-		return bond_xmit_xor(skb, dev);
+		return bond_3ad_xor_xmit(skb, dev);
 	case BOND_MODE_BROADCAST:
 		return bond_xmit_broadcast(skb, dev);
-	case BOND_MODE_8023AD:
-		return bond_3ad_xmit_xor(skb, dev);
 	case BOND_MODE_ALB:
 		return bond_alb_xmit(skb, dev);
 	case BOND_MODE_TLB:
@@ -3861,6 +4020,7 @@ static void bond_uninit(struct net_device *bond_dev)
 	struct bonding *bond = netdev_priv(bond_dev);
 	struct list_head *iter;
 	struct slave *slave;
+	struct bond_up_slave *arr;
 
 	bond_netpoll_cleanup(bond_dev);
 
@@ -3869,6 +4029,12 @@ static void bond_uninit(struct net_device *bond_dev)
 		__bond_release_one(bond_dev, slave->dev, true);
 	netdev_info(bond_dev, "Released all slaves\n");
 
+	arr = rtnl_dereference(bond->slave_arr);
+	if (arr) {
+		kfree_rcu(arr, rcu);
+		RCU_INIT_POINTER(bond->slave_arr, NULL);
+	}
+
 	list_del(&bond->bond_list);
 
 	bond_debug_unregister(bond);
diff --git a/drivers/net/bonding/bonding.h b/drivers/net/bonding/bonding.h
index 5b022da9cad2..10920f0686e2 100644
--- a/drivers/net/bonding/bonding.h
+++ b/drivers/net/bonding/bonding.h
@@ -179,6 +179,12 @@ struct slave {
 	struct rtnl_link_stats64 slave_stats;
 };
 
+struct bond_up_slave {
+	unsigned int	count;
+	struct rcu_head rcu;
+	struct slave	*arr[0];
+};
+
 /*
  * Link pseudo-state only used internally by monitors
  */
@@ -193,6 +199,7 @@ struct bonding {
 	struct   slave __rcu *curr_active_slave;
 	struct   slave __rcu *current_arp_slave;
 	struct   slave __rcu *primary_slave;
+	struct   bond_up_slave __rcu *slave_arr; /* Array of usable slaves */
 	bool     force_primary;
 	s32      slave_cnt; /* never change this value outside the attach/detach wrappers */
 	int     (*recv_probe)(const struct sk_buff *, struct bonding *,
@@ -222,6 +229,7 @@ struct bonding {
 	struct   delayed_work alb_work;
 	struct   delayed_work ad_work;
 	struct   delayed_work mcast_work;
+	struct   delayed_work slave_arr_work;
 #ifdef CONFIG_DEBUG_FS
 	/* debugging support via debugfs */
 	struct	 dentry *debug_dir;
@@ -534,6 +542,8 @@ const char *bond_slave_link_status(s8 link);
 struct bond_vlan_tag *bond_verify_device_path(struct net_device *start_dev,
 					      struct net_device *end_dev,
 					      int level);
+int bond_update_slave_arr(struct bonding *bond, struct slave *skipslave);
+void bond_slave_arr_work_rearm(struct bonding *bond, unsigned long delay);
 
 #ifdef CONFIG_PROC_FS
 void bond_create_proc_entry(struct bonding *bond);
-- 
2.1.0.rc2.206.gedb03e5

^ permalink raw reply related

* Re: [PATCH v2 2/3] net: can: c_can: Add syscon/regmap RAMINIT mechanism
From: Roger Quadros @ 2014-10-01  8:45 UTC (permalink / raw)
  To: Marc Kleine-Budde, Wolfram Sang
  Cc: wg, tony, tglx, mugunthanvnm, george.cherian, balbi, nsekhar, nm,
	sergei.shtylyov, linux-omap, linux-can, netdev
In-Reply-To: <542AD483.2020808@pengutronix.de>

On 09/30/2014 07:04 PM, Marc Kleine-Budde wrote:
> On 09/30/2014 05:25 PM, Wolfram Sang wrote:
>>
>>> Yes, but syscon_regmap_lookup_by_phandle() doesn't need any support for
>>> additional parameters. Have a look at:
>>>
>>> drivers/net/ethernet/stmicro/stmmac/dwmac-socfpga.c
>>>
>>> First get the regmap, then the 1st argument is the offset in the regmap,
>>> the 2nd and 3rd could be the bits.
>>
>> So, for one driver the extra arguments are: <reg> <start_bit> <stop_bit>
>> For another driver (the stmmac example): <reg_offset> <reg_shift>
> 
> The DCAN's "reg" is a "reg_offset" as in the stmmc.
> 
> Roger, can we derive both start and done bit from a common reg_shift?

I'm sorry I didn't understand what you meant.

<&syscon_phandl> <reg offset> <start bit> <stop bit> should work well for us.
Even though reg offset is the same for both the DCAN instances.

> 
>> Phew... Then we should really have a "syscon-raminit" property probably,
>> so that at least plain "syscon" has a consistent syntax?
> 
> I think^whope we can have the same syntax as the stmmc :D

I agree too.

> 
>>>> So, I'd rather drop additional arguments.
>>>>
>>>> Why would you like to have it encoded in DT?
>>>
>>> Where put the information then? Into the driver, but where do you get
>>> the reference which instance of the DCAN you are, so that you can look
>>> up the correct bits?
>>
>> Agreed. I thought we had this information in the driver already, but we
>> haven't...
>>
> 
> The current driver relies on the of_alias_get_id(), which isn't
> considered best practice, is it? So I want to avoid this when switching
> to syscon.

cheers,
-roger

^ permalink raw reply

* Re: [PATCH v2 2/3] net: can: c_can: Add syscon/regmap RAMINIT mechanism
From: Marc Kleine-Budde @ 2014-10-01  8:47 UTC (permalink / raw)
  To: Roger Quadros, Wolfram Sang
  Cc: wg, tony, tglx, mugunthanvnm, george.cherian, balbi, nsekhar, nm,
	sergei.shtylyov, linux-omap, linux-can, netdev
In-Reply-To: <542BBF3F.2040803@ti.com>

[-- Attachment #1: Type: text/plain, Size: 1338 bytes --]

On 10/01/2014 10:45 AM, Roger Quadros wrote:
> On 09/30/2014 07:04 PM, Marc Kleine-Budde wrote:
>> On 09/30/2014 05:25 PM, Wolfram Sang wrote:
>>>
>>>> Yes, but syscon_regmap_lookup_by_phandle() doesn't need any support for
>>>> additional parameters. Have a look at:
>>>>
>>>> drivers/net/ethernet/stmicro/stmmac/dwmac-socfpga.c
>>>>
>>>> First get the regmap, then the 1st argument is the offset in the regmap,
>>>> the 2nd and 3rd could be the bits.
>>>
>>> So, for one driver the extra arguments are: <reg> <start_bit> <stop_bit>
>>> For another driver (the stmmac example): <reg_offset> <reg_shift>
>>
>> The DCAN's "reg" is a "reg_offset" as in the stmmc.
>>
>> Roger, can we derive both start and done bit from a common reg_shift?
> 
> I'm sorry I didn't understand what you meant.
> 
> <&syscon_phandl> <reg offset> <start bit> <stop bit> should work well for us.
> Even though reg offset is the same for both the DCAN instances.

What's start bit and stop bit for instance 0 and 1 on that SoC that has
two instances??

Marc

-- 
Pengutronix e.K.                  | Marc Kleine-Budde           |
Industrial Linux Solutions        | Phone: +49-231-2826-924     |
Vertretung West/Dortmund          | Fax:   +49-5121-206917-5555 |
Amtsgericht Hildesheim, HRA 2686  | http://www.pengutronix.de   |


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 181 bytes --]

^ permalink raw reply

* [PATCH v2] net: ll_temac: Remove unnecessary ether_setup after alloc_etherdev
From: Michal Simek @ 2014-10-01  9:01 UTC (permalink / raw)
  To: linux-kernel, monstr
  Cc: Subbaraya Sundeep Bhatta, David S. Miller, Manuel Schölling,
	Paul Gortmaker, Julia Lawall, Joe Perches, Ricardo Ribalda,
	Eric Dumazet, netdev, linux-arm-kernel

[-- Attachment #1: Type: text/plain, Size: 1046 bytes --]

From: Subbaraya Sundeep Bhatta <subbaraya.sundeep.bhatta@xilinx.com>

Calling ether_setup is redundant since alloc_etherdev calls it.

Signed-off-by: Subbaraya Sundeep Bhatta <sbhatta@xilinx.com>
Signed-off-by: Michal Simek <michal.simek@xilinx.com>
---

Changes in v2:
- Remove axienet because it is already applied in separate patch
  https://git.kernel.org/cgit/linux/kernel/git/davem/net-next.git/commit/?id=c706471b2601d1c9058e7b866db77f6eb7dd37af

 drivers/net/ethernet/xilinx/ll_temac_main.c | 1 -
 1 file changed, 1 deletion(-)

diff --git a/drivers/net/ethernet/xilinx/ll_temac_main.c b/drivers/net/ethernet/xilinx/ll_temac_main.c
index fda5891835d4..629077050fce 100644
--- a/drivers/net/ethernet/xilinx/ll_temac_main.c
+++ b/drivers/net/ethernet/xilinx/ll_temac_main.c
@@ -1012,7 +1012,6 @@ static int temac_of_probe(struct platform_device *op)
 	if (!ndev)
 		return -ENOMEM;

-	ether_setup(ndev);
 	platform_set_drvdata(op, ndev);
 	SET_NETDEV_DEV(ndev, &op->dev);
 	ndev->flags &= ~IFF_MULTICAST;  /* clear multicast */
--
1.8.2.3


[-- Attachment #2: Type: application/pgp-signature, Size: 198 bytes --]

^ permalink raw reply related

* Re: [PATCH v2 2/3] net: can: c_can: Add syscon/regmap RAMINIT mechanism
From: Roger Quadros @ 2014-10-01  9:06 UTC (permalink / raw)
  To: Marc Kleine-Budde, Wolfram Sang
  Cc: wg, tony, tglx, mugunthanvnm, george.cherian, balbi, nsekhar, nm,
	sergei.shtylyov, linux-omap, linux-can, netdev
In-Reply-To: <542BBFBE.90406@pengutronix.de>

On 10/01/2014 11:47 AM, Marc Kleine-Budde wrote:
> On 10/01/2014 10:45 AM, Roger Quadros wrote:
>> On 09/30/2014 07:04 PM, Marc Kleine-Budde wrote:
>>> On 09/30/2014 05:25 PM, Wolfram Sang wrote:
>>>>
>>>>> Yes, but syscon_regmap_lookup_by_phandle() doesn't need any support for
>>>>> additional parameters. Have a look at:
>>>>>
>>>>> drivers/net/ethernet/stmicro/stmmac/dwmac-socfpga.c
>>>>>
>>>>> First get the regmap, then the 1st argument is the offset in the regmap,
>>>>> the 2nd and 3rd could be the bits.
>>>>
>>>> So, for one driver the extra arguments are: <reg> <start_bit> <stop_bit>
>>>> For another driver (the stmmac example): <reg_offset> <reg_shift>
>>>
>>> The DCAN's "reg" is a "reg_offset" as in the stmmc.
>>>
>>> Roger, can we derive both start and done bit from a common reg_shift?
>>
>> I'm sorry I didn't understand what you meant.
>>
>> <&syscon_phandl> <reg offset> <start bit> <stop bit> should work well for us.
>> Even though reg offset is the same for both the DCAN instances.
> 
> What's start bit and stop bit for instance 0 and 1 on that SoC that has
> two instances??
> 

we have 3 SoCs at the moment, all have 2 DCAN instances.

AM33xx & AM43xx
instance	start	stop
1		0	8
2		1	9	

DRA7xx
instance	start	stop
1		3	1
2		5	2

cheers,
-roger

^ permalink raw reply

* Re: [PATCH RFC ipsec-next] xfrm: Add sysctl option to enforce inbound policies for transport mode
From: Steffen Klassert @ 2014-10-01  9:58 UTC (permalink / raw)
  To: Tobias Brunner; +Cc: davem, netdev, Herbert Xu
In-Reply-To: <54296113.3080106@strongswan.org>

On Mon, Sep 29, 2014 at 03:39:31PM +0200, Tobias Brunner wrote:
> > 
> > commit 8fe7ee2ba983fd89b2555dce5930ffd0f7f6c361
> > Author: Herbert Xu <herbert@gondor.apana.org.au>
> > Date:   Thu Oct 23 14:57:11 2003 -0700
> > 
> >     [IPSEC]: Strengthen policy checks.
> > 
> > Maybe Herbert remembers why this was done only for tunnel mode.
> 
> Would be great to hear from Herbert about this.
> 
> > If I read section 5.2.1 of RFC 2401 correct, the inbound policy
> > must be enforced regardless of the mode.
> 
> It looks like the wording changed with RFC 4301.  

RFC 4301 did not yet exist when this code change was made,
so I tried to find the reason for that in RFC 2401.

> The SPD and its
> policies are not mentioned explicitly anymore in section 5.2 (like
> they were in step 3 in section 5.2.1 of RFC 2401).  Instead, packets
> must be matched against the "selectors identified by the SAD entry".
> It's not entirely clear to me whether these selectors are part of the
> SPD or properties of the SAD entries themselves, like the single
> selector that can currently be configured on SAs in the kernel.

It reads like the packets must match against the selectors of the
used SA. That's what we do at the beginning of __xfrm_policy_check().

This was already required in RFC 2401. But there was a matching policy
required too, seems this is not necessary anymore with RFC 4301.

Instead they recommend: "Every SPD SHOULD have a nominal, final entry
that catches anything that is otherwise unmatched, and discards it."

> Also, section 4.4.2 explicitly states that manually keyed SAD entries
> do not necessarily need to have a corresponding SPD entry.  Which might
> make sense for simple host-host (transport mode) SAs, but this wouldn't
> be possible anymore when enforcing inbound policies for transport mode.

Well, this states just that such SAD entries can exist. But does not
say that packets transformed by such a SA are allowed to pass without
matching policy.

> 
> Subject: [PATCH] xfrm: Enforce inbound transport mode policies like those in other modes
> 
> Currently inbound policies for transport mode SAs are not enforced.
> If no policy is found or if the templates don't match this is not
> considered an error for transport mode SAs.

If we find a policy, the templates should match. We need to fix this.
But it seems our policy enforcement for tunnel mode is too strict if
no policy is found. So I think we should leave transport mode as
it is here.

^ permalink raw reply

* Re: [PATCH v2 2/3] net: can: c_can: Add syscon/regmap RAMINIT mechanism
From: Marc Kleine-Budde @ 2014-10-01 10:01 UTC (permalink / raw)
  To: Roger Quadros, Wolfram Sang
  Cc: wg, tony, tglx, mugunthanvnm, george.cherian, balbi, nsekhar, nm,
	sergei.shtylyov, linux-omap, linux-can, netdev
In-Reply-To: <542BC42F.2010406@ti.com>

[-- Attachment #1: Type: text/plain, Size: 1852 bytes --]

On 10/01/2014 11:06 AM, Roger Quadros wrote:
> On 10/01/2014 11:47 AM, Marc Kleine-Budde wrote:
>> On 10/01/2014 10:45 AM, Roger Quadros wrote:
>>> On 09/30/2014 07:04 PM, Marc Kleine-Budde wrote:
>>>> On 09/30/2014 05:25 PM, Wolfram Sang wrote:
>>>>>
>>>>>> Yes, but syscon_regmap_lookup_by_phandle() doesn't need any support for
>>>>>> additional parameters. Have a look at:
>>>>>>
>>>>>> drivers/net/ethernet/stmicro/stmmac/dwmac-socfpga.c
>>>>>>
>>>>>> First get the regmap, then the 1st argument is the offset in the regmap,
>>>>>> the 2nd and 3rd could be the bits.
>>>>>
>>>>> So, for one driver the extra arguments are: <reg> <start_bit> <stop_bit>
>>>>> For another driver (the stmmac example): <reg_offset> <reg_shift>
>>>>
>>>> The DCAN's "reg" is a "reg_offset" as in the stmmc.
>>>>
>>>> Roger, can we derive both start and done bit from a common reg_shift?
>>>
>>> I'm sorry I didn't understand what you meant.
>>>
>>> <&syscon_phandl> <reg offset> <start bit> <stop bit> should work well for us.
>>> Even though reg offset is the same for both the DCAN instances.
>>
>> What's start bit and stop bit for instance 0 and 1 on that SoC that has
>> two instances?
>>
> 
> we have 3 SoCs at the moment, all have 2 DCAN instances.
> 
> AM33xx & AM43xx
> instance	start	stop
> 1		0	8
> 2		1	9	

If we use a 0-based numbering for the instances:
instance	start		stop
0		(0 << instance)	(8 << instance)
1		(0 << instance)	(8 << instance)

> DRA7xx
> instance	start	stop
> 1		3	1
> 2		5	2
                ^
5 or 4?

Marc

-- 
Pengutronix e.K.                  | Marc Kleine-Budde           |
Industrial Linux Solutions        | Phone: +49-231-2826-924     |
Vertretung West/Dortmund          | Fax:   +49-5121-206917-5555 |
Amtsgericht Hildesheim, HRA 2686  | http://www.pengutronix.de   |


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 181 bytes --]

^ permalink raw reply

* Re: [PATCH net-next v6 2/2] bonding: Simplify the xmit function for modes that use xmit_hash
From: Nikolay Aleksandrov @ 2014-10-01 10:08 UTC (permalink / raw)
  To: Mahesh Bandewar, Jay Vosburgh, Veaceslav Falico, Andy Gospodarek,
	David Miller
  Cc: netdev, Eric Dumazet, Maciej Zenczykowski
In-Reply-To: <1412152711-12646-1-git-send-email-maheshb@google.com>

On 01/10/14 10:38, Mahesh Bandewar wrote:
> Earlier change to use usable slave array for TLB mode had an additional
> performance advantage. So extending the same logic to all other modes
> that use xmit-hash for slave selection (viz 802.3AD, and XOR modes).
> Also consolidating this with the earlier TLB change.
>
> The main idea is to build the usable slaves array in the control path
> and use that array for slave selection during xmit operation.
>
> Measured performance in a setup with a bond of 4x1G NICs with 200
> instances of netperf for the modes involved (3ad, xor, tlb)
> cmd: netperf -t TCP_RR -H <TargetHost> -l 60 -s 5
>
> Mode        TPS-Before   TPS-After
>
> 802.3ad   : 468,694      493,101
> TLB (lb=0): 392,583      392,965
> XOR       : 475,696      484,517
>
> Signed-off-by: Mahesh Bandewar <maheshb@google.com>
> ---
> v1:
>    (a) If bond_update_slave_arr() fails to allocate memory, it will overwrite
>        the slave that need to be removed.
>    (b) Freeing of array will assign NULL (to handle bond->down to bond->up
>        transition gracefully.
>    (c) Change from pr_debug() to pr_err() if bond_update_slave_arr() returns
>        failure.
>    (d) XOR: bond_update_slave_arr() will consider mii-mon, arp-mon cases and
>        will populate the array even if these parameters are not used.
>    (e) 3AD: Should handle the ad_agg_selection_logic correctly.
> v2:
>    (a) Removed rcu_read_{un}lock() calls from array manipulation code.
>    (b) Slave link-events now refresh array for all these modes.
>    (c) Moved free-array call from bond_close() to bond_uninit().
> v3:
>    (a) Fixed null pointer dereference.
>    (b) Removed bond->lock lockdep dependency.
> v4:
>    (a) Made to changes to comply with Nikolay's locking changes
>    (b) Added a work-queue to refresh slave-array when RTNL is not held
>    (c) Array refresh happens ONLY with RTNL now.
>    (d) alloc changed from GFP_ATOMIC to GFP_KERNEL
> v5:
>    (a) Consolidated all delayed slave-array updates at one place in
>        3ad_state_machine_handler()
> v6:
>    (a) Free slave array when there is no active aggregator
>
>   drivers/net/bonding/bond_3ad.c  | 140 +++++++++++------------------
>   drivers/net/bonding/bond_alb.c  |  51 ++---------
>   drivers/net/bonding/bond_alb.h  |   8 --
>   drivers/net/bonding/bond_main.c | 192 +++++++++++++++++++++++++++++++++++++---
>   drivers/net/bonding/bonding.h   |  10 +++
>   5 files changed, 249 insertions(+), 152 deletions(-)
>
<<<snip>>>
> @@ -3869,6 +4029,12 @@ static void bond_uninit(struct net_device *bond_dev)
>   		__bond_release_one(bond_dev, slave->dev, true);
>   	netdev_info(bond_dev, "Released all slaves\n");
>
> +	arr = rtnl_dereference(bond->slave_arr);
> +	if (arr) {
> +		kfree_rcu(arr, rcu);
> +		RCU_INIT_POINTER(bond->slave_arr, NULL);
> +	}
> +
>   	list_del(&bond->bond_list);
>
>   	bond_debug_unregister(bond);
<<<snip>>>
I'm fine with this version, just one last question about something I just 
noticed in the hunk above:
You first call kfree_rcu() and then RCU_INIT_POINTER(). This feels wrong as the 
currently used slave_arr can get freed before it's set to NULL if we get 
preempted after the kfree_rcu(). Now, I know it's not really a problem because 
at this point the bond device has been closed and shouldn't operate, but just in 
case I think it'd be nice to first NULL it and call kfree_rcu() after that.

Thanks for all your hard work on this.

Signed-off-by: Nikolay Aleksandrov <nikolay@redhat.com>

^ permalink raw reply

* Re: [PATCH v2 2/3] net: can: c_can: Add syscon/regmap RAMINIT mechanism
From: Roger Quadros @ 2014-10-01 10:12 UTC (permalink / raw)
  To: Marc Kleine-Budde, Wolfram Sang
  Cc: wg, tony, tglx, mugunthanvnm, george.cherian, balbi, nsekhar, nm,
	sergei.shtylyov, linux-omap, linux-can, netdev
In-Reply-To: <542BD116.4090809@pengutronix.de>

On 10/01/2014 01:01 PM, Marc Kleine-Budde wrote:
> On 10/01/2014 11:06 AM, Roger Quadros wrote:
>> On 10/01/2014 11:47 AM, Marc Kleine-Budde wrote:
>>> On 10/01/2014 10:45 AM, Roger Quadros wrote:
>>>> On 09/30/2014 07:04 PM, Marc Kleine-Budde wrote:
>>>>> On 09/30/2014 05:25 PM, Wolfram Sang wrote:
>>>>>>
>>>>>>> Yes, but syscon_regmap_lookup_by_phandle() doesn't need any support for
>>>>>>> additional parameters. Have a look at:
>>>>>>>
>>>>>>> drivers/net/ethernet/stmicro/stmmac/dwmac-socfpga.c
>>>>>>>
>>>>>>> First get the regmap, then the 1st argument is the offset in the regmap,
>>>>>>> the 2nd and 3rd could be the bits.
>>>>>>
>>>>>> So, for one driver the extra arguments are: <reg> <start_bit> <stop_bit>
>>>>>> For another driver (the stmmac example): <reg_offset> <reg_shift>
>>>>>
>>>>> The DCAN's "reg" is a "reg_offset" as in the stmmc.
>>>>>
>>>>> Roger, can we derive both start and done bit from a common reg_shift?
>>>>
>>>> I'm sorry I didn't understand what you meant.
>>>>
>>>> <&syscon_phandl> <reg offset> <start bit> <stop bit> should work well for us.
>>>> Even though reg offset is the same for both the DCAN instances.
>>>
>>> What's start bit and stop bit for instance 0 and 1 on that SoC that has
>>> two instances?
>>>
>>
>> we have 3 SoCs at the moment, all have 2 DCAN instances.
>>
>> AM33xx & AM43xx
>> instance	start	stop
>> 1		0	8
>> 2		1	9	
> 
> If we use a 0-based numbering for the instances:
> instance	start		stop
> 0		(0 << instance)	(8 << instance)
> 1		(0 << instance)	(8 << instance)
> 

How does the instance number get set? What happens on boards where the first instance is unused
while the second one is in use?

>> DRA7xx
>> instance	start	stop
>> 1		3	1
>> 2		5	2
>                 ^
> 5 or 4?

Unfortunately it is 5 ;)
We have display IP related bit in between 3 and 5 :P

cheers,
-roger


^ permalink raw reply

* Re: [PATCH v2 2/3] net: can: c_can: Add syscon/regmap RAMINIT mechanism
From: Marc Kleine-Budde @ 2014-10-01 10:26 UTC (permalink / raw)
  To: Roger Quadros, Wolfram Sang
  Cc: wg, tony, tglx, mugunthanvnm, george.cherian, balbi, nsekhar, nm,
	sergei.shtylyov, linux-omap, linux-can, netdev
In-Reply-To: <542BD3A7.5060200@ti.com>

[-- Attachment #1: Type: text/plain, Size: 3071 bytes --]

On 10/01/2014 12:12 PM, Roger Quadros wrote:
> On 10/01/2014 01:01 PM, Marc Kleine-Budde wrote:
>> On 10/01/2014 11:06 AM, Roger Quadros wrote:
>>> On 10/01/2014 11:47 AM, Marc Kleine-Budde wrote:
>>>> On 10/01/2014 10:45 AM, Roger Quadros wrote:
>>>>> On 09/30/2014 07:04 PM, Marc Kleine-Budde wrote:
>>>>>> On 09/30/2014 05:25 PM, Wolfram Sang wrote:
>>>>>>>
>>>>>>>> Yes, but syscon_regmap_lookup_by_phandle() doesn't need any support for
>>>>>>>> additional parameters. Have a look at:
>>>>>>>>
>>>>>>>> drivers/net/ethernet/stmicro/stmmac/dwmac-socfpga.c
>>>>>>>>
>>>>>>>> First get the regmap, then the 1st argument is the offset in the regmap,
>>>>>>>> the 2nd and 3rd could be the bits.
>>>>>>>
>>>>>>> So, for one driver the extra arguments are: <reg> <start_bit> <stop_bit>
>>>>>>> For another driver (the stmmac example): <reg_offset> <reg_shift>
>>>>>>
>>>>>> The DCAN's "reg" is a "reg_offset" as in the stmmc.
>>>>>>
>>>>>> Roger, can we derive both start and done bit from a common reg_shift?
>>>>>
>>>>> I'm sorry I didn't understand what you meant.
>>>>>
>>>>> <&syscon_phandl> <reg offset> <start bit> <stop bit> should work well for us.
>>>>> Even though reg offset is the same for both the DCAN instances.
>>>>
>>>> What's start bit and stop bit for instance 0 and 1 on that SoC that has
>>>> two instances?
>>>>
>>>
>>> we have 3 SoCs at the moment, all have 2 DCAN instances.
>>>
>>> AM33xx & AM43xx
>>> instance	start	stop
>>> 1		0	8
>>> 2		1	9	
>>
>> If we use a 0-based numbering for the instances:
>> instance	start		stop
>> 0		(0 << instance)	(8 << instance)
>> 1		(0 << instance)	(8 << instance)
>>
> 
> How does the instance number get set? What happens on boards where
> the first instance is unused while the second one is in use?

Via a single "instance" parameter of the syscon phandle....

> 
>>> DRA7xx
>>> instance	start	stop
>>> 1		3	1
>>> 2		5	2
>>                 ^
>> 5 or 4?
> 
> Unfortunately it is 5 ;)
> We have display IP related bit in between 3 and 5 :P

What on earth were the HW engineers thinking????????????

...if we just have the instance parameter in the syscon phandle, we have
to put the mapping into the driver, which makes IMHO no sense, because
you have to touch the driver, if there is another SoC with the DCAN core.

AFAICS we have these options:
1) syscon phandle with three parameters:
   reg offset, start bit shift, stop bit shift
   (the name of the syscon phandle is a different topic)
2) a single ti,start-stop-bit option with two parameter
3) two ti,* entries with a single parameter each
   (as in Roger's initial patch)

Wolfram, which solution do you prefer? I'm in favour of 3 (+ plus a
phandle with a reg offset parameter).

puzzled,
Marc

-- 
Pengutronix e.K.                  | Marc Kleine-Budde           |
Industrial Linux Solutions        | Phone: +49-231-2826-924     |
Vertretung West/Dortmund          | Fax:   +49-5121-206917-5555 |
Amtsgericht Hildesheim, HRA 2686  | http://www.pengutronix.de   |


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 181 bytes --]

^ permalink raw reply

* Re: [PATCH 32/34] netfilter: bridge: move br_netfilter out of the core
From: Pablo Neira Ayuso @ 2014-10-01 10:33 UTC (permalink / raw)
  To: Stephen Hemminger; +Cc: netfilter-devel, davem, netdev, fw
In-Reply-To: <20140930085802.15460008@urahara>

On Tue, Sep 30, 2014 at 08:58:02AM -0700, Stephen Hemminger wrote:
> On Mon, 29 Sep 2014 14:39:21 +0200
> Pablo Neira Ayuso <pablo@netfilter.org> wrote:
> 
> > Jesper reported that br_netfilter always registers the hooks since
> > this is part of the bridge core. This harms performance for people that
> > don't need this.
> > 
> > This patch modularizes br_netfilter so it can be rmmod'ed, thus,
> > the hooks can be unregistered. I think the bridge netfilter should have
> > been a separated module since the beginning, Patrick agreed on that.
> > 
> > Note that this is breaking compatibility for users that expect that
> > bridge netfilter is going to be available after explicitly 'modprobe
> > bridge' or via automatic load through brctl.
> > 
> > However, the damage can be easily undone by modprobing br_netfilter.
> > The bridge core also spots a message to provide a clue to people that
> > didn't notice that this has been deprecated.
> > 
> > On top of that, the plan is that nftables will not rely on this software
> > layer, but integrate the connection tracking into the bridge layer to
> > enable stateful filtering and NAT, which is was bridge netfilter users
> > seem to require.
> > 
> > This patch still keeps the fake_dst_ops in the bridge core, since this
> > is required by when the bridge port is initialized. So we can safely
> > modprobe/rmmod br_netfilter anytime.
> > 
> > Signed-off-by: Pablo Neira Ayuso <pablo@netfilter.org>
> > Acked-by: Florian Westphal <fw@strlen.de>
> 
> I think this is a good idea but you can't break users.
> 
> We need to figure out a way to autoload br_netfilter module on first use.

Indeed, we've been discussing several possibilities depending, short
summary:

1) Autoload br_netfilter from the configuration path, as it happens
   with many other netfilter modules. However, {ip,ip6,arp}tables are
   decoupled from br_netfilter, so we don't have an obvious way to know if
   you're using them from the bridge.

2) Florian proposed to detect this from the packet path. The idea is
   to move the bridge-nf-call sysctl entries to the bridge core, then if
   you see the first bridged packet and bridge-nf-call is set to 1
   (from sysctl or sysfs), the br_netfilter is requested via workqueue
   and the packets are dropped until the br_netfilter module is in place.
   Then, wait for some time to disable bridge-nf-call by default.

My consideration to not follow 2) was that users only need to
'modprobe br_netfilter' if they really need this. This breakage is not
forcing users to upgrade userspace binaries. Instead, it is asking for
some explicit consent from the user.

Sorry, I should have Cc you since the beginning in this discussion. It
was too late when I realized. I can live with going 2) if you think
it's the way to go.

Let us know if you have a better idea. Thanks!

^ permalink raw reply

* Re: [PATCH v2 2/3] net: can: c_can: Add syscon/regmap RAMINIT mechanism
From: Wolfram Sang @ 2014-10-01 10:43 UTC (permalink / raw)
  To: Marc Kleine-Budde
  Cc: Roger Quadros, wg, tony, tglx, mugunthanvnm, george.cherian,
	balbi, nsekhar, nm, sergei.shtylyov, linux-omap, linux-can,
	netdev
In-Reply-To: <542BD6F1.2020900@pengutronix.de>

[-- Attachment #1: Type: text/plain, Size: 752 bytes --]


> > Unfortunately it is 5 ;)
> > We have display IP related bit in between 3 and 5 :P
> 
> What on earth were the HW engineers thinking????????????

"Let's test my RNG on the bit-placement of this register" :)

> ...if we just have the instance parameter in the syscon phandle, we have
> to put the mapping into the driver, which makes IMHO no sense, because
> you have to touch the driver, if there is another SoC with the DCAN core.

... which would be my preferred solution. I think new SoCs should have
some kind of:

	compatible = "commodore,c64ultra", "bosch,d_can";

in the DT anyhow to allow for SoC specific quirks/adjustments. And
custom raminit belongs to that IMO (see the ti routine getting more and
more specific).


[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

^ permalink raw reply

* Re: [PATCH v2 2/3] net: can: c_can: Add syscon/regmap RAMINIT mechanism
From: Roger Quadros @ 2014-10-01 10:57 UTC (permalink / raw)
  To: Wolfram Sang, Marc Kleine-Budde
  Cc: wg, tony, tglx, mugunthanvnm, george.cherian, balbi, nsekhar, nm,
	sergei.shtylyov, linux-omap, linux-can, netdev
In-Reply-To: <20141001104317.GB1261@katana>

On 10/01/2014 01:43 PM, Wolfram Sang wrote:
> 
>>> Unfortunately it is 5 ;)
>>> We have display IP related bit in between 3 and 5 :P
>>
>> What on earth were the HW engineers thinking????????????
> 
> "Let's test my RNG on the bit-placement of this register" :)

:D

> 
>> ...if we just have the instance parameter in the syscon phandle, we have
>> to put the mapping into the driver, which makes IMHO no sense, because
>> you have to touch the driver, if there is another SoC with the DCAN core.

My guess is that TI won't come up with a 3rd variant so we won't have to
touch the driver, but you never know for sure.

> 
> ... which would be my preferred solution. I think new SoCs should have
> some kind of:
> 
> 	compatible = "commodore,c64ultra", "bosch,d_can";
> 
> in the DT anyhow to allow for SoC specific quirks/adjustments. And
> custom raminit belongs to that IMO (see the ti routine getting more and
> more specific).
> 

Right. For now we need 2 start/stop definations for the existing TI Socs.

but where to store the raminit start/stop bits? The driver_data currently seems to 
contain the CAN type C_CAN vs D_CAN without containing it in a platform_data structure.

Is it OK to create a new platform_data structure for CAN and put the type and raminit start/stop
bits there?

cheers,
-roger

^ permalink raw reply

* RE: [PATCH v2 3/4] net: stmmac: add support for Intel Quark X1000
From: Kweh, Hock Leong @ 2014-10-01 11:06 UTC (permalink / raw)
  To: 'David Miller'
  Cc: 'peppe.cavallaro@st.com',
	'rayagond@vayavyalabs.com',
	'vbridgers2013@gmail.com',
	'srinivas.kandagatla@st.com', 'wens@csie.org',
	'netdev@vger.kernel.org',
	'linux-kernel@vger.kernel.org', Ong, Boon Leong

> -----Original Message-----
> From: Kweh, Hock Leong
> Sent: Thursday, September 18, 2014 11:18 AM
> > From: David Miller [mailto:davem@davemloft.net]
> > Sent: Wednesday, September 17, 2014 12:56 PM
> > Are you kidding me?  It's a perfect way to identify this device, it
> > properly uses PCI_CLASS_NETWORK_ETHERNET (0x0200) in both cases and
> > this will not match any other function on this PCI device at all.
> >
> > Please do as I suggested and use the PCI class for the differentiation
> > and matching during probing.
> 
> Hi David,
> 
> Thanks for your feedback so far. Appreciate it.
> 
> Sorry to my poorly written description that may have caused some confusion.
> My sincere apology here.
> Unfortunately, I don't really grasp your idea clearly based on your responses
> which I appreciate them a lot.
> Sorry for the long description below but I hope to clearly pen down my
> thinking process so that you can follow my thinking incrementally without
> being confused.
> 
> So, let's roll back a bit so that with my following description, you can help
> correct me if my understanding of using PCI function ID to differentiate PHY
> port that is associated with each Ethernet controller is wrong:
> 
> The high-level idea about the change that I made for STMMAC IP inside
> Quark is as follow:
> 
> (1) Based on Quark-specific PCI ID declared inside stmmac_id_table[],  the
> probe() function is
>        called to continue setting-up STMMAC for Quark.
> 
> @@ -228,11 +303,13 @@ static int stmmac_pci_resume(struct pci_dev *pdev)
> 
>  #define STMMAC_VENDOR_ID 0x700
>  #define STMMAC_DEVICE_ID 0x1108
> +#define STMMAC_QUARK_X1000_ID 0x0937
> 
>  static const struct pci_device_id stmmac_id_table[] = {
>  	{PCI_DEVICE(STMMAC_VENDOR_ID, STMMAC_DEVICE_ID),
> PCI_ANY_ID,
>  			PCI_ANY_ID, CHIP_STMICRO},
>  	{PCI_VDEVICE(STMICRO, PCI_DEVICE_ID_STMICRO_MAC),
> CHIP_STMICRO},
> +	{PCI_VDEVICE(INTEL, STMMAC_QUARK_X1000_ID),
> CHIP_QUARK_X1000},
>  	{}
>  };
> 
> (2) Back-ground on STMMAC hardware configuration on Intel Galileo Gen 1 &
> Gen 2 platforms:
> Intel Quark SoC has 2 MAC controller as described by lspci output below:
> 
> 00:14.6 Class 0200: 8086:0937    ====> 1st MAC controller
> 00:14.7 Class 0200: 8086:0937    ====> 2nd MAC controller
> 
> These Galileo boards use the same Intel Quark SoC and there is only one PHY
> connect to the 1st MAC [00:14.6 Class 0200: 8086:0937] The 2nd MAC [00:14.7
> Class 0200: 8086:0937] is NOT connected to any PHY at all.
> 
> So, it appears to me that the only way that I can differentiate between 1st &
> 2nd MAC are based on PCI function ID, i.e. 14.6 & 14.7. Therefore, within the
> probe() function, for Intel Quark SoC only, the function performs next-level
> discovery of 1st or 2nd MAC controller through quark_run_time_config()
> function.
> For other PCI ID (currently STMICRO_MAC) there is NO next-level discovery
> involved as rt_config is NULL.
> Changes shown below:
> 
> static struct platform_data platform_info[] = { @@ -59,15 +65,76 @@ static
> struct platform_data platform_info[] = {
>  		.phy_reset = NULL,
>  		.phy_mask = 0,
>  		.pbl = 32,
> +		.fixed_burst = 0,
>  		.burst_len = DMA_AXI_BLEN_256,
> +		.rt_config = NULL,  ===================> no 2nd-level
> discovery for other PCI ID
> +	},
> +	[CHIP_QUARK_X1000] = {
> +		.phy_addr = 1,
> +		.interface = PHY_INTERFACE_MODE_RMII,
> +		.clk_csr = 2,
> +		.has_gmac = 1,
> +		.force_sf_dma_mode = 1,
> +		.multicast_filter_bins = HASH_TABLE_SIZE,
> +		.unicast_filter_entries = 1,
> +		.phy_reset = NULL,
> +		.phy_mask = 0,
> +		.pbl = 16,
> +		.fixed_burst = 1,
> +		.burst_len = DMA_AXI_BLEN_256,
> +		.rt_config = &quark_run_time_config,       ==========>
> Quark specific 2nd-level discovery
> +	},
> +};
> 
> (3) Within quark_run_time_config(), due to the only way to differentiate 1st
> or 2nd MAC controller according to difference in function ID explained above,
> the following changes are made:
> 
> +static void quark_run_time_config(int chip_id, struct pci_dev *pdev) {
> +	const char *board_name =
> dmi_get_system_info(DMI_BOARD_NAME);
> +	int i;
> +	int func_num = PCI_FUNC(pdev->devfn);
> +
> +	if (!board_name)
> +		return;
> +
> +	for (i = 0; i < ARRAY_SIZE(quark_x1000_phy_info); i++) {
> +		if ((!strcmp(quark_x1000_phy_info[i].board_name,
> board_name)) &&
> +		    quark_x1000_phy_info[i].pci_func_num == func_num)
> +			platform_info[chip_id].phy_addr =
> +				quark_x1000_phy_info[i].phy_address;
> +	}
> +}
> 
> The reasons for the above proposed condition checks, i.e. "board name" &
> "pci function name" are below:
>   a) As described above, the only difference in both instance of STMMAC IP
> inside Intel Quark SoC is the function ID,
>        so I have proposed to use function ID to be the decision point here to
> differentiate 1st MAC from 2nd MAC.
>   b) Allow future expansion of any other Intel Quark platforms with specific
> need to fix PHY address
>   c) A PHY address set as "-1"  is to mark that the PHY (associated with
> function ID) is not connected to MAC, which
>       is being used here for the 2 Galileo boards -> 2nd MAC port not connected
> with PHY.
> 
> 
> Finally, based on the above description, it appears to me that using PCI
> function ID to decode seems viable for Intel Quark specific hardware
> configuration.
> 
> Appreciate your time and any feedback is very much appreciated.
> 
> Thanks.
> 
> 
> Regards,
> Wilson


Hi Guys,

Just gently ping for the discussion to carry on before forgetting the context.
Anyone have any better idea or comments or concern to this topic?
Hope the above explanation clear out your doubt.


Regards,
Wilson

^ permalink raw reply

* Re: [PATCH v2 2/3] net: can: c_can: Add syscon/regmap RAMINIT mechanism
From: Marc Kleine-Budde @ 2014-10-01 11:06 UTC (permalink / raw)
  To: Roger Quadros, Wolfram Sang
  Cc: wg, tony, tglx, mugunthanvnm, george.cherian, balbi, nsekhar, nm,
	sergei.shtylyov, linux-omap, linux-can, netdev
In-Reply-To: <542BDE32.6030309@ti.com>

[-- Attachment #1: Type: text/plain, Size: 1538 bytes --]

On 10/01/2014 12:57 PM, Roger Quadros wrote:
>>> ...if we just have the instance parameter in the syscon phandle, we have
>>> to put the mapping into the driver, which makes IMHO no sense, because
>>> you have to touch the driver, if there is another SoC with the DCAN core.
> 
> My guess is that TI won't come up with a 3rd variant so we won't have
> to touch the driver, but you never know for sure.

When I comes to 99% compatible hardware.... I've seen some.

>> ... which would be my preferred solution. I think new SoCs should have
>> some kind of:
>>
>> 	compatible = "commodore,c64ultra", "bosch,d_can";
>>
>> in the DT anyhow to allow for SoC specific quirks/adjustments. And
>> custom raminit belongs to that IMO (see the ti routine getting more and
>> more specific).
>>
> 
> Right. For now we need 2 start/stop definations for the existing TI Socs.
> 
> but where to store the raminit start/stop bits? The driver_data currently seems to 
> contain the CAN type C_CAN vs D_CAN without containing it in a platform_data structure.
> 
> Is it OK to create a new platform_data structure for CAN and put the type and raminit start/stop
> bits there?

Yes, have a look how it's handled in the flexcan driver.

regards,
Marc

-- 
Pengutronix e.K.                  | Marc Kleine-Budde           |
Industrial Linux Solutions        | Phone: +49-231-2826-924     |
Vertretung West/Dortmund          | Fax:   +49-5121-206917-5555 |
Amtsgericht Hildesheim, HRA 2686  | http://www.pengutronix.de   |


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 181 bytes --]

^ permalink raw reply


This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox