Hi Peter, On Wed, Aug 28, 2013 at 10:02:40PM +0800, Fengguang Wu wrote: > Hi Peter, > > FYI, we find a little netperf performance drop, together with much > reduced lock contentions. Please let me know if you would like us do > more tests on it. > > 126d9c20bc99ccf1d62581d6e7fd63fd12e222bf is the first bad commit > commit 126d9c20bc99ccf1d62581d6e7fd63fd12e222bf > Author: Peter Zijlstra > Date: Tue Aug 20 15:36:48 2013 +0200 > > another NUMA code dump Now we got more results: there are both ups and downs. 661.10 +2.3% 676.20 TOTAL netperf.Throughput_Mbps 165733.95 -6.7% 154709.48 TOTAL aim7.2000.jobs-per-min 2203.20 -1.7% 2165.50 TOTAL netperf.Throughput_tps 569.20 -68.0% 181.94 TOTAL slabinfo.skbuff_fclone_cache.num_objs 542.84 -66.5% 181.94 TOTAL slabinfo.skbuff_fclone_cache.active_objs 376.92 -69.2% 116.27 TOTAL sched_debug.cpu#2.nr_uninterruptible 441.02 +89.6% 836.36 TOTAL sched_debug.cfs_rq[7]:/.nr_spread_over 62464.94 -36.5% 39676.33 TOTAL interrupts.CPU3.RES 53942.34 -46.8% 28701.16 TOTAL interrupts.CPU1.RES 85.95 +164.3% 227.20 TOTAL sched_debug.cpu#5.nr_uninterruptible 73764.87 -32.5% 49819.94 TOTAL interrupts.CPU2.RES 7698.32 +118.7% 16836.46 TOTAL sched_debug.cfs_rq[7]:/.spread0 89601.90 -31.3% 61544.49 TOTAL interrupts.CPU0.RES 496290.10 -25.8% 368250.18 TOTAL interrupts.RES 11166.54 +82.7% 20397.97 TOTAL sched_debug.cfs_rq[7]:/.min_vruntime 158066.00 -34.2% 103970.26 TOTAL softirqs.CPU0.NET_RX 630988.15 -29.8% 442956.09 TOTAL softirqs.NET_RX 156872.34 -29.2% 110987.49 TOTAL softirqs.CPU2.NET_RX 157622.69 -30.0% 110326.06 TOTAL softirqs.CPU3.NET_RX 7690.77 -15.2% 6523.75 TOTAL sched_debug.cpu#2.ttwu_local 33587.66 -16.5% 28048.84 TOTAL sched_debug.cpu#1.sched_goidle 158427.11 -25.7% 117672.28 TOTAL softirqs.CPU1.NET_RX 8975.45 -21.6% 7038.36 TOTAL sched_debug.cpu#3.ttwu_local 21103.12 -21.6% 16553.45 TOTAL softirqs.CPU3.RCU 21376.43 -20.8% 16934.73 TOTAL softirqs.CPU2.RCU 1663.42 -1.9% 1631.68 TOTAL slabinfo.kmalloc-2048.num_objs 1554.94 +3.8% 1613.65 TOTAL slabinfo.kmalloc-2048.active_objs 3526131.15 +20.5% 4248631.25 TOTAL proc-vmstat.pgalloc_normal 56884.60 +20.3% 68432.74 TOTAL slabinfo.kmalloc-64.num_objs 55951.48 +19.8% 67030.33 TOTAL slabinfo.kmalloc-64.active_objs 16903.28 +18.2% 19976.14 TOTAL softirqs.CPU0.SCHED 24396.71 -16.6% 20337.48 TOTAL interrupts.CPU6.RES 24424.39 -15.4% 20660.17 TOTAL interrupts.CPU5.RES 68269.93 -14.8% 58135.54 TOTAL sched_debug.cpu#6.nr_switches 68301.75 -14.8% 58169.82 TOTAL sched_debug.cpu#6.sched_count 36349.50 -14.3% 31146.73 TOTAL sched_debug.cpu#0.sched_goidle 169088.39 -14.3% 144828.56 TOTAL softirqs.RCU 29181.61 -13.3% 25309.49 TOTAL sched_debug.cpu#5.sched_goidle 29244.19 -14.7% 24941.43 TOTAL sched_debug.cpu#6.sched_goidle 46361.62 +7.7% 49935.54 TOTAL softirqs.SCHED 2254.81 -11.6% 1993.26 TOTAL sched_debug.cfs_rq[2]:/.min_vruntime 84214.17 -12.9% 73322.31 TOTAL sched_debug.cpu#3.nr_switches 21465.75 -15.9% 18061.93 TOTAL softirqs.CPU0.RCU 84259.37 -12.9% 73360.78 TOTAL sched_debug.cpu#3.sched_count 2717.34 +13.4% 3081.63 TOTAL slabinfo.kmalloc-128.num_objs 2717.34 +12.7% 3061.40 TOTAL slabinfo.kmalloc-128.active_objs 44008.82 -13.0% 38280.08 TOTAL sched_debug.cpu#3.ttwu_count 20872.34 -11.8% 18400.51 TOTAL softirqs.CPU4.RCU 1164.51 +13.5% 1321.68 TOTAL buddyinfo.Node.0.zone.Normal.0 1687.56 +12.3% 1895.26 TOTAL slabinfo.filp.active_objs 67219.90 -12.5% 58843.19 TOTAL sched_debug.cpu#5.nr_switches 67253.78 -12.5% 58880.24 TOTAL sched_debug.cpu#5.sched_count 5312453.48 +13.5% 6029326.46 TOTAL proc-vmstat.pgfree 21554.90 -13.8% 18579.77 TOTAL softirqs.CPU6.RCU 36507.99 -14.9% 31078.42 TOTAL sched_debug.cpu#2.sched_goidle 43167.98 -13.2% 37457.64 TOTAL sched_debug.cpu#2.ttwu_count 24019.62 -11.5% 21267.02 TOTAL interrupts.CPU7.RES 82664.49 -15.0% 70263.14 TOTAL sched_debug.cpu#2.sched_count 82620.08 -15.0% 70223.42 TOTAL sched_debug.cpu#2.nr_switches 37659.16 -13.7% 32493.72 TOTAL sched_debug.cpu#6.ttwu_count 20774.12 -10.7% 18544.93 TOTAL softirqs.CPU1.RCU 28506.05 -10.2% 25592.08 TOTAL sched_debug.cpu#7.sched_goidle 35939.20 -11.5% 31822.25 TOTAL sched_debug.cpu#3.sched_goidle 21152.73 -11.4% 18737.90 TOTAL softirqs.CPU5.RCU 861.68 +12.4% 968.22 TOTAL buddyinfo.Node.0.zone.Normal.1 37078.89 -10.4% 33230.12 TOTAL sched_debug.cpu#5.ttwu_count 860788.53 +10.6% 952024.03 TOTAL proc-vmstat.numa_hit 860788.53 +10.6% 952023.96 TOTAL proc-vmstat.numa_local 7802.01 +11.1% 8670.11 TOTAL sched_debug.cpu#5.nr_load_updates 65555.21 -9.3% 59426.89 TOTAL sched_debug.cpu#7.nr_switches 65585.05 -9.3% 59462.10 TOTAL sched_debug.cpu#7.sched_count 8846.95 -14.5% 7563.70 TOTAL sched_debug.cpu#6.ttwu_local 22195.37 -9.0% 20188.74 TOTAL softirqs.CPU3.SCHED 227.63 +8.2% 246.30 TOTAL uptime.idle 23712.01 +9.2% 25893.32 TOTAL interrupts.LOC 4307.40 -99.3% 31.00 TOTAL lock_stat.((&icsk->icsk_retransmit_timer)).acquisitions 7139.90 -96.3% 265.40 TOTAL lock_stat.((&icsk->icsk_delack_timer)).acquisitions 566879.90 -99.9% 657.00 TOTAL lock_stat.&wq->wait.acq-bounces 13827.10 -99.6% 56.80 TOTAL lock_stat.&(&base->lock)->rlock.con-bounces 40546.40 -99.6% 161.40 TOTAL lock_stat.&(&base->lock)->rlock.contentions.lock_timer_base 13827.30 -99.6% 57.00 TOTAL lock_stat.&(&base->lock)->rlock.contentions 74164.23 -99.2% 628.83 TOTAL lock_stat.&(&base->lock)->rlock.waittime-total 4249.11 -100.0% 0.00 TOTAL lock_stat.&(&list->lock)->rlock#2.waittime-total 215936.60 -99.8% 324.60 TOTAL lock_stat.&(&list->lock)->rlock#2.acq-bounces 2975.23 -100.0% 0.00 TOTAL lock_stat.&wq->wait.waittime-total 12327.20 -99.9% 8.30 TOTAL lock_stat.slock-AF_INET.contentions.release_sock 23594.90 -99.9% 13.80 TOTAL lock_stat.slock-AF_INET/1.con-bounces 23594.90 -99.9% 13.80 TOTAL lock_stat.slock-AF_INET/1.contentions 23594.90 -99.9% 13.80 TOTAL lock_stat.slock-AF_INET/1.contentions.tcp_v4_rcv 11909.80 -99.9% 7.40 TOTAL lock_stat.slock-AF_INET/1.contentions.release_sock 11682.50 -99.9% 7.11 TOTAL lock_stat.slock-AF_INET/1.contentions.lock_sock_nested 2891155.00 +793.9% 25844317.50 TOTAL lock_stat.&pipe->mutex/1.acquisitions 584.68 +770.8% 5091.37 TOTAL lock_stat.&pipe->wait.waittime-total 2870693.90 +799.0% 25808748.90 TOTAL lock_stat.&pipe->wait.acquisitions 653854.40 -99.9% 361.90 TOTAL lock_stat.slock-AF_INET/1.acq-bounces 132668.89 -100.0% 58.82 TOTAL lock_stat.slock-AF_INET/1.waittime-total 135481.90 -99.9% 86.50 TOTAL lock_stat.slock-AF_INET.contentions.tcp_v4_rcv 113885.70 -99.9% 75.00 TOTAL lock_stat.slock-AF_INET.contentions.lock_sock_nested 834535.10 -99.9% 695.50 TOTAL lock_stat.slock-AF_INET.acq-bounces 135509.30 -99.9% 86.50 TOTAL lock_stat.slock-AF_INET.con-bounces 115477.01 -96.7% 3808.53 TOTAL lock_stat.((&icsk->icsk_delack_timer)).holdtime-total 135511.30 -99.9% 86.50 TOTAL lock_stat.slock-AF_INET.contentions 916330.86 -93.8% 56991.34 TOTAL lock_stat.&rq->lock.waittime-total 6994048.67 -99.9% 4638.26 TOTAL lock_stat.slock-AF_INET.waittime-total 72918.76 -93.8% 4501.73 TOTAL lock_stat.((&icsk->icsk_retransmit_timer)).holdtime-total 561.79 +366.8% 2622.34 TOTAL lock_stat.&pipe->wait.waittime-max 37855.16 +392.2% 186310.37 TOTAL lock_stat.(&task->u.tk_work).holdtime-max 37856.44 +392.2% 186311.75 TOTAL lock_stat.rpciod-R.holdtime-max 1882726.00 -30.9% 1301055.80 TOTAL lock_stat.&rq->lock.acq-bounces 1192.74 +349.8% 5364.45 TOTAL lock_stat.slock-AF_INET-RPC.waittime-total 1057131.87 -68.5% 332629.78 TOTAL lock_stat.&rq->lock/1.holdtime-total 230850.10 -47.2% 121845.20 TOTAL lock_stat.&rq->lock.contentions.try_to_wake_up 2054554.10 +337.8% 8995698.10 TOTAL lock_stat.&p->pi_lock.acquisitions 958070.10 -86.1% 133493.10 TOTAL lock_stat.&(&base->lock)->rlock.acq-bounces 3001.10 -83.1% 508.26 TOTAL lock_stat.&mm->mmap_sem-R.holdtime-max 41448.45 +277.4% 156411.45 TOTAL lock_stat.zone->wait_table+i.waittime-total 206432.67 +247.1% 716507.42 TOTAL lock_stat.(&task->u.tk_work).holdtime-total 207795.52 +246.0% 718916.47 TOTAL lock_stat.rpciod-R.holdtime-total 42415.30 +13.1% 47964.02 TOTAL lock_stat.&rq->lock/1.waittime-total 1598626.30 +671.8% 12338701.15 TOTAL lock_stat.&pipe->mutex/1.holdtime-total 240154.70 -24.1% 182204.10 TOTAL lock_stat.&rq->lock.contentions 239875.20 -24.1% 182101.50 TOTAL lock_stat.&rq->lock.con-bounces 10334983.54 +260.8% 37290852.98 TOTAL lock_stat.all_cpu_access_lock.holdtime-total 10382840.38 +260.4% 37424524.18 TOTAL lock_stat.trace_event_sem-R.holdtime-total 10682456.18 +257.6% 38195309.13 TOTAL lock_stat.&iter->mutex.holdtime-total 11622.60 +254.2% 41166.90 TOTAL lock_stat.slock-AF_INET-RPC/1.acquisitions 11716.00 +252.2% 41260.90 TOTAL lock_stat._xmit_ETHER#2.acquisitions 23432.00 +252.2% 82521.90 TOTAL lock_stat.&(&list->lock)->rlock.acquisitions 15159239.80 +253.4% 53574107.00 TOTAL lock_stat.buffer->reader_lock_key.acquisitions 13357123.60 +261.4% 48273636.60 TOTAL lock_stat.&rq->lock.acquisitions 3250919.71 +225.8% 10592024.15 TOTAL lock_stat.buffer->reader_lock_key.holdtime-total 15000.00 +219.6% 47943.30 TOTAL lock_stat.slock-AF_INET-RPC.acquisitions 1821865.51 +799.3% 16383840.49 TOTAL lock_stat.&pipe->wait.holdtime-total 6306.35 +216.5% 19962.11 TOTAL lock_stat.&(&list->lock)->rlock.holdtime-total 104745.20 +220.5% 335684.60 TOTAL lock_stat.trace_types_lock.acquisitions 104745.20 +220.5% 335684.00 TOTAL lock_stat.trace_event_sem-R.acquisitions 104745.20 +220.5% 335683.80 TOTAL lock_stat.all_cpu_access_lock.acquisitions 1411.65 +126.4% 3196.43 TOTAL lock_stat.&vfsmount_lock-R.holdtime-max 1669574.80 -60.4% 661408.70 TOTAL lock_stat.&(&base->lock)->rlock.acquisitions 116585.00 +197.0% 346231.10 TOTAL lock_stat.&iter->mutex.acquisitions 39046.00 +188.4% 112601.00 TOTAL lock_stat.trace_types_lock.holdtime-total 2356238.10 -20.5% 1872483.90 TOTAL lock_stat.rcu_read_lock_bh-R.acquisitions 712340.80 -58.5% 295574.20 TOTAL lock_stat.&wq->wait.acquisitions 106482.08 +160.0% 276823.43 TOTAL lock_stat.slock-AF_INET-RPC/1.holdtime-total 161126.72 +183.9% 457435.26 TOTAL lock_stat._xmit_ETHER#2.holdtime-total 134393.55 +183.8% 381435.33 TOTAL lock_stat.sk_lock-AF_INET-RPC.holdtime-total 23948.20 +127.0% 54351.40 TOTAL lock_stat.&sb->s_type->i_lock_key#1.acquisitions 509327.41 -54.9% 229766.53 TOTAL lock_stat.&sig->cred_guard_mutex.holdtime-max 27084.80 +118.9% 59298.00 TOTAL lock_stat.sb_writers#4-R.acquisitions 6221.93 -51.1% 3045.54 TOTAL lock_stat.sk_lock-AF_INET.holdtime-max 4656.87 +181.9% 13126.12 TOTAL lock_stat.&pipe->mutex/1.holdtime-max 10765.33 +123.3% 24037.30 TOTAL lock_stat.(&task->u.tk_work)#2.holdtime-total 43141327.41 -13.1% 37507805.71 TOTAL lock_stat.rcu_read_lock_bh-R.holdtime-total 4100.50 -51.8% 1975.29 TOTAL lock_stat.rcu_read_lock-R.holdtime-max 739556.16 -17.4% 610769.01 TOTAL lock_stat.&mm->mmap_sem-W.holdtime-max 11290.17 +117.2% 24520.74 TOTAL lock_stat.nfsiod-R.holdtime-total 42400.54 -47.1% 22416.67 TOTAL lock_stat.&(&info->si_lock)->rlock.holdtime-total 48336.50 +80.9% 87451.90 TOTAL lock_stat.&(&nf->lru_lock)->rlock.contentions.ip_defrag 1168351.69 +10.8% 1294640.42 TOTAL lock_stat.&pipe->mutex/1.waittime-total 102707.56 -48.3% 53101.39 TOTAL lock_stat.&sb->s_type->i_mutex_key#2.waittime-total 91318694.78 +26.6% 115632734.76 TOTAL lock_stat.&rq->lock.holdtime-total 50189004.65 +18.5% 59471866.33 TOTAL lock_stat.&p->pi_lock.holdtime-total 133163.30 -44.5% 73942.90 TOTAL lock_stat.&anon_vma->rwsem.acq-bounces 66818.00 -45.6% 36333.20 TOTAL lock_stat.&p->pi_lock.acq-bounces 361874.50 +82.2% 659237.20 TOTAL lock_stat.&(&zone->lock)->rlock.contentions.__free_pages_ok 3957.60 -33.3% 2638.84 TOTAL lock_stat.tasklist_lock-W.waittime-total 244764.50 +72.1% 421131.00 TOTAL lock_stat.&(&zone->lock)->rlock.contentions.get_page_from_freelist 6540563.51 -53.2% 3058553.07 TOTAL lock_stat.&(&base->lock)->rlock.holdtime-total 214843.60 +74.5% 374915.00 TOTAL lock_stat.&(&nf->lru_lock)->rlock.con-bounces 214846.50 +74.5% 374916.00 TOTAL lock_stat.&(&nf->lru_lock)->rlock.contentions 641418.67 +75.6% 1126321.06 TOTAL lock_stat.&(&nf->lru_lock)->rlock.waittime-total 192855.90 +76.0% 339378.40 TOTAL lock_stat.&(&nf->lru_lock)->rlock.contentions.inet_frag_find 2322889.38 +111.6% 4915093.78 TOTAL lock_stat.sb_writers#7-R.holdtime-total 63744.63 -42.3% 36809.55 TOTAL lock_stat.&(&n->list_lock)->rlock.holdtime-total 188500.60 +71.4% 323001.70 TOTAL lock_stat.&(&nf->lru_lock)->rlock.contentions.inet_frag_kill 14078.22 +71.4% 24129.54 TOTAL lock_stat.&(&queue->lock)->rlock.holdtime-total 350330.40 +68.4% 589915.00 TOTAL lock_stat.&(&zone->lock)->rlock.contentions 350276.20 +68.2% 589108.30 TOTAL lock_stat.&(&zone->lock)->rlock.con-bounces 2435.44 +68.8% 4109.92 TOTAL lock_stat.&(&hb->chain_lock)->rlock.waittime-total 11039.58 +38.2% 15255.39 TOTAL lock_stat.(&(&wb->dwork)->work).holdtime-total 45302081.74 -8.2% 41581942.88 TOTAL lock_stat.&wq->wait.holdtime-total 182856.27 +59.9% 292446.19 TOTAL lock_stat.zone->wait_table+i.holdtime-total 2182608.45 +60.5% 3502508.79 TOTAL lock_stat.&(&zone->lock)->rlock.waittime-total 21148.30 +57.5% 33298.65 TOTAL lock_stat.&(&xprt->transport_lock)->rlock.holdtime-total 11841.06 +49.7% 17725.30 TOTAL lock_stat.&sb->s_type->i_lock_key#1.holdtime-total 20350.70 +48.1% 30148.50 TOTAL lock_stat.&pipe->mutex/1.acq-bounces 19903.70 +48.3% 29522.80 TOTAL lock_stat.&pipe->wait.acq-bounces 12025.44 +88.2% 22631.20 TOTAL lock_stat.&type->i_mutex_dir_key#2.holdtime-total 12025.44 +88.2% 22631.20 TOTAL lock_stat.&type->i_mutex_dir_key#2.holdtime-min 12025.44 +88.2% 22631.20 TOTAL lock_stat.&type->i_mutex_dir_key#2.holdtime-max 459953.53 +9.5% 503614.41 TOTAL lock_stat.&mapping->i_mmap_mutex.waittime-max 36732.44 +38.4% 50824.74 TOTAL lock_stat.sb_writers#4-R.holdtime-total 439874.42 -5.3% 416534.11 TOTAL lock_stat.clock-AF_INET-R.holdtime-total 277796.51 +44.6% 401667.68 TOTAL lock_stat.&p->pi_lock.waittime-total 2313283.70 -26.8% 1693267.70 TOTAL lock_stat.&(&list->lock)->rlock#3.acquisitions 74032.10 -28.6% 52864.10 TOTAL lock_stat.&rq->lock/1.acquisitions 931324.30 +34.7% 1254788.40 TOTAL lock_stat.&(&zone->lock)->rlock.acq-bounces 14328.30 +25.0% 17906.30 TOTAL lock_stat.hrtimer_bases.lock#7.holdtime-total 37493.70 -26.3% 27624.80 TOTAL lock_stat.&rq->lock/1.acq-bounces 15085692.76 +36.5% 20597805.11 TOTAL lock_stat.rcu_read_lock-R.holdtime-total 31239.40 +36.1% 42514.40 TOTAL lock_stat.zone->wait_table+i.contentions.__wake_up 159447.37 +32.3% 211019.95 TOTAL lock_stat.rcu_node_0.waittime-total 100236.97 -33.9% 66247.10 TOTAL lock_stat.&pipe->mutex/1.waittime-max 1166309.60 -23.3% 894892.70 TOTAL lock_stat.slock-AF_INET/1.acquisitions 4053.60 +29.4% 5245.03 TOTAL lock_stat.sk_lock-AF_INET6.holdtime-total 5036.72 -23.9% 3834.90 TOTAL lock_stat.&(&tsk->delays->lock)->rlock.holdtime-total 10019.40 +45.8% 14613.28 TOTAL lock_stat.slock-AF_INET-RPC.holdtime-total 1417553.50 +23.9% 1755833.90 TOTAL lock_stat.&(&zone->lock)->rlock.acquisitions 6946024.55 -19.3% 5602189.45 TOTAL lock_stat.&(&list->lock)->rlock#3.holdtime-total 5172.44 -19.2% 4177.08 TOTAL lock_stat.&q->lock.holdtime-total 18369.95 +23.1% 22618.18 TOTAL lock_stat.(&(work)->timer).holdtime-total 834643.80 +22.7% 1024368.10 TOTAL lock_stat.&(&nf->lru_lock)->rlock.acq-bounces 25251.78 +22.4% 30899.04 TOTAL lock_stat.&(&pool->lock)->rlock.holdtime-total 3931866.39 +23.0% 4834661.56 TOTAL lock_stat.((&timer)).holdtime-total 400630.70 -19.6% 322076.80 TOTAL lock_stat.&(&dentry->d_lock)->rlock.contentions.path_get 729459.97 -23.7% 556449.31 TOTAL lock_stat.&mapping->i_mmap_mutex.waittime-total 1205124.95 +17.2% 1412040.97 TOTAL lock_stat.&rsp->gp_wq.holdtime-total 778969.60 -18.0% 639110.20 TOTAL lock_stat.&(&dentry->d_lock)->rlock.contentions.dput 203249.44 -17.2% 168359.45 TOTAL lock_stat.&(&dentry->d_lock)->rlock.waittime-total 787586.90 -16.9% 654513.10 TOTAL lock_stat.&(&dentry->d_lock)->rlock.con-bounces 787737.60 -16.9% 654670.40 TOTAL lock_stat.&(&dentry->d_lock)->rlock.contentions 982029.40 +20.0% 1178821.80 TOTAL lock_stat.&f->lock-R.acq-bounces 11377298.70 -15.7% 9591520.20 TOTAL lock_stat.rcu_read_lock-R.acquisitions 21232.88 +19.0% 25274.64 TOTAL lock_stat.&rnp->nocb_gp_wq[1].holdtime-total 22050.24 +18.9% 26217.07 TOTAL lock_stat.&rnp->nocb_gp_wq[0].holdtime-total 175592.50 +16.0% 203669.97 TOTAL lock_stat.clockevents_lock.waittime-total 182770.97 +18.0% 215662.07 TOTAL lock_stat.rcu_sched_state.onoff_mutex.holdtime-total 169134.80 -14.9% 143897.00 TOTAL lock_stat.zone->wait_table+i.acquisitions 11269453.16 +12.6% 12691439.20 TOTAL lock_stat.&(&zone->lock)->rlock.holdtime-total 13525.95 +17.0% 15830.27 TOTAL lock_stat.(&(work)->work).holdtime-total 8975.59 +16.8% 10485.40 TOTAL lock_stat.proc_subdir_lock.holdtime-total 99222.00 +16.0% 115055.75 TOTAL lock_stat.sb_writers#2-R.holdtime-total 124847.90 -13.7% 107753.10 TOTAL lock_stat.zone->wait_table+i.acq-bounces 1518193.46 +10.4% 1676339.59 TOTAL lock_stat.rcu_node_0.holdtime-total 95007.69 +15.2% 109413.23 TOTAL lock_stat.&sig->wait_chldexit.holdtime-total 256302.50 -13.1% 222725.30 TOTAL lock_stat.&mm->mmap_sem-W.acq-bounces 6934.41 +14.6% 7944.90 TOTAL lock_stat.sb_writers#3-R.holdtime-total 15463.56 +13.7% 17577.66 TOTAL lock_stat.entries_lock-R.holdtime-total 263989.70 -13.1% 229316.00 TOTAL lock_stat.&(&mm->page_table_lock)->rlock.acq-bounces 13700.25 +87.2% 25651.38 TOTAL lock_stat.hrtimer_bases.lock#4.holdtime-total 10082.81 +14.1% 11507.23 TOTAL lock_stat.bdev_lock.holdtime-total 258345.90 -12.6% 225740.10 TOTAL lock_stat.&(&newf->file_lock)->rlock.acq-bounces 28512.94 +12.8% 32154.48 TOTAL lock_stat.events-R.holdtime-total 47912.04 -12.3% 41996.01 TOTAL lock_stat.rename_lock.waittime-total 224803.20 -12.6% 196381.70 TOTAL lock_stat.&(&fs->lock)->rlock.acq-bounces 103370.20 -12.7% 90277.20 TOTAL lock_stat.rename_lock.contentions.SyS_getcwd 13222.56 +18.3% 15637.30 TOTAL lock_stat.hrtimer_bases.lock#5.holdtime-total 702867.20 -12.0% 618778.40 TOTAL lock_stat.&mapping->i_mmap_mutex.contentions.vma_link 238640.40 -11.8% 210514.70 TOTAL lock_stat.&(&sighand->siglock)->rlock.acq-bounces 144491.30 +12.8% 163038.30 TOTAL lock_stat.clockevents_lock.con-bounces 289138.00 +12.8% 326170.80 TOTAL lock_stat.clockevents_lock.contentions.clockevents_notify 144569.00 +12.8% 163085.40 TOTAL lock_stat.clockevents_lock.contentions 585553.00 -11.5% 518161.60 TOTAL lock_stat.&mapping->i_mmap_mutex.con-bounces 585553.00 -11.5% 518161.60 TOTAL lock_stat.&mapping->i_mmap_mutex.contentions 174036.80 +11.7% 194446.50 TOTAL lock_stat.((&timer)).acquisitions 198920.50 +12.5% 223756.40 TOTAL lock_stat.rcu_node_0.acquisitions 159105.97 +12.2% 178524.65 TOTAL lock_stat.&(&p->alloc_lock)->rlock.holdtime-total 12962.55 +11.3% 14431.29 TOTAL lock_stat.&child->perf_event_mutex.holdtime-total 467032.10 -10.9% 416276.60 TOTAL lock_stat.&mapping->i_mmap_mutex.contentions.unlink_file_vma 2639031.80 -10.6% 2360489.00 TOTAL lock_stat.sk_lock-AF_INET.acquisitions 23964.02 +9.5% 26236.37 TOTAL lock_stat.&fsnotify_mark_srcu-R.holdtime-total 706694.52 -9.6% 638876.20 TOTAL lock_stat.jiffies_lock.holdtime-total 14111.25 +36.4% 19245.96 TOTAL lock_stat.hrtimer_bases.lock.holdtime-total 47545.73 +11.3% 52915.50 TOTAL lock_stat.pgd_lock.holdtime-total 319799.80 +10.4% 353118.34 TOTAL lock_stat.tasklist_lock-W.holdtime-total 13131948.50 -10.6% 11739786.10 TOTAL lock_stat.&(&dentry->d_lock)->rlock.acq-bounces 5748518.20 -10.3% 5156548.00 TOTAL lock_stat.&mapping->i_mmap_mutex.acq-bounces 10293967.02 -6.7% 9599300.67 TOTAL lock_stat.&mm->mmap_sem-R.holdtime-total 2486744.60 +590.1% 17161466.60 TOTAL perf-stat.context-switches 91101.74 +19.8% 109116.17 TOTAL vmstat.system.cs 15.96 +17.8% 18.79 TOTAL iostat.cpu.user 19.81 +16.5% 23.07 TOTAL vmstat.cpu.us 1818.70 +29.6% 2356.40 TOTAL time.involuntary_context_switches 313376.90 -20.3% 249842.70 TOTAL perf-stat.cpu-migrations 17606.67 -17.4% 14540.72 TOTAL vmstat.system.in 669432.20 -12.7% 584367.90 TOTAL time.voluntary_context_switches 306.12 +0.1% 306.52 TOTAL iostat.cpu.system 72.62 +7.1% 77.79 TOTAL time.elapsed_time 400.28 +0.0% 400.32 TOTAL vmstat.cpu.sy 188.10 -6.8% 175.40 TOTAL time.percent_of_cpu_this_job_got 1.35 -7.9% 1.24 TOTAL iostat.sda.rkB/s 0.35 -7.9% 0.32 TOTAL vmstat.io.bi 178.19 -4.7% 169.90 TOTAL vmstat.cpu.wa 178.08 -4.6% 169.91 TOTAL iostat.cpu.iowait 311.74 +2.6% 319.92 TOTAL boottime.dhcp 269.22 +2.7% 276.37 TOTAL boottime.boot 568.45 +3.1% 586.18 TOTAL boottime.idle 2699.20 +0.6% 2714.10 TOTAL time.minor_page_faults 82.14 +0.3% 82.40 TOTAL iostat.cpu.idle 82.12 +0.3% 82.39 TOTAL vmstat.cpu.id Details attached. Thanks, Fengguang