From mboxrd@z Thu Jan 1 00:00:00 1970 From: =?ISO-8859-2?Q?Pawe=B3_Staszewski?= Subject: Re: weird problem Date: Thu, 25 Jun 2009 18:33:21 +0200 Message-ID: <4A43A6D1.70901@itcare.pl> References: <4A43A098.1@itcare.pl> Mime-Version: 1.0 Content-Type: text/plain; charset=ISO-8859-2; format=flowed Content-Transfer-Encoding: QUOTED-PRINTABLE To: Linux Network Development list Return-path: Received: from smtp.iq.pl ([86.111.241.19]:46136 "EHLO smtp.iq.pl" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1751411AbZFYQdX (ORCPT ); Thu, 25 Jun 2009 12:33:23 -0400 Received: from unknown (HELO [192.168.1.10]) (itcare_pstaszewski@[83.7.206.39]) (envelope-sender ) by smtp.iq.pl with AES256-SHA encrypted SMTP for ; 25 Jun 2009 16:33:22 -0000 In-Reply-To: <4A43A098.1@itcare.pl> Sender: netdev-owner@vger.kernel.org List-ID: oprofile output per cpu CPU: Core 2, speed 3000.08 MHz (estimated) Counted CPU_CLK_UNHALTED events (Clock cycles when not halted) with a=20 unit mask of 0x00 (Unhalted core cycles) count 100000 Samples on CPU 0 Samples on CPU 1 Samples on CPU 2 Samples on CPU 3 Samples on CPU 4 Samples on CPU 5 Samples on CPU 6 Samples on CPU 7 Samples on CPU all samples % samples % samples % samples % = =20 samples % samples % samples % samples % = =20 samples % image name app name symbol name 29520 21.7362 0 0 0 0 0 0 = =20 0 0 0 0 9571 34.0254 0 0 = =20 4701 0.22 55 oprofiled oprofiled (no symbols) 18441 13.5785 0 0 0 0 0 0 = =20 0 0 0 0 0 0 0 0 = =20 2666 0.12 79 vmlinux vmlinux rt_worker_func 18428 13.5690 132089 2.1756 135138 1.4238 21735 42.7703 = =20 775 1.9819 893 5.3180 847 3.0111 1056 3.3130 = =20 37949 1.82 01 vmlinux vmlinux mwait_idle 15797 11.6317 212 0.0035 2879 0.0303 17 0.0335 = =20 2 0.0051 4 0.0238 14 0.0498 185 0.5804 = =20 2895 0.13 89 vmlinux vmlinux free_block 9034 6.6519 1218 0.0201 2727 0.0287 0 0 = =20 0 0 0 0 0 0 0 0 = =20 2004 0.09 61 vmlinux vmlinux dst_destroy 7113 5.2375 632 0.0104 483 0.0051 6615 13.0170 = =20 543 1.3886 461 2.7454 1951 6.9359 1274 3.9970 = =20 4073 0.19 54 vmlinux vmlinux mutex_spin_on_own= er 4382 3.2266 652 0.0107 1880 0.0198 2 0.0039 = =20 0 0 0 0 3 0.0107 7 0.0220 = =20 1050 0.05 04 vmlinux vmlinux __call_rcu 4337 3.1934 418 0.0069 830 0.0087 58 0.1141 = =20 18 0.0460 8 0.0476 20 0.0711 39 0.1224 = =20 917 0.04 40 vmlinux vmlinux =20 __rcu_process_callbacks 3991 2.9387 51 8.4e-04 325 0.0034 136 0.2676 = =20 2 0.0051 1 0.0060 1034 3.6759 205 0.6432 = =20 1110 0.05 32 vmlinux vmlinux =20 copy_user_generic_string 3292 2.4240 275838 4.5432 391639 4.1263 869 1.7100 = =20 431 1.1022 350 2.0843 673 2.3925 1069 3.3538 = =20 89939 4.31 37 vmlinux vmlinux _raw_spin_lock 1909 1.4056 79 0.0013 376 0.0040 0 0 = =20 0 0 0 0 0 0 0 0 = =20 348 0.01 67 vmlinux vmlinux dst_rcu_free 1461 1.0758 0 0 1163 0.0123 0 0 = =20 0 0 0 0 0 0 0 0 = 0 0 vim vim (no symbols) 1122 0.8262 11514 0.1896 6590 0.0694 1857 3.6542 = =20 7638 19.5325 2138 12.7323 1213 4.3123 4025 12.6278 = =20 9676 0.46 41 libc-2.8.so libc-2.8.so (no symbols) 1035 0.7621 9027 0.1487 22216 0.2341 4 0.0079 = =20 0 0 0 0 5 0.0178 15 0.0471 = =20 4310 0.20 67 vmlinux vmlinux kmem_cache_free 1016 0.7481 218 0.0036 465 0.0049 0 0 = =20 0 0 0 0 0 0 0 0 = =20 255 0.01 22 vmlinux vmlinux ipv4_dst_destroy 951 0.7002 4290 0.0707 5514 0.0581 3283 6.4603 = =20 867 2.2172 673 4.0079 1096 3.8963 1316 4.1288 = =20 2593 0.12 44 vmlinux vmlinux tg_shares_up 708 0.5213 104 0.0017 279 0.0029 0 0 = =20 0 0 0 0 0 0 0 0 = =20 159 0.00 76 vmlinux vmlinux call_rcu_bh 581 0.4278 21812 0.3593 38364 0.4042 68 0.1338 = =20 23 0.0588 23 0.1370 65 0.2311 96 0.3012 = =20 8000 0.38 37 vmlinux vmlinux _raw_spin_unlock 563 0.4145 1027 0.0169 2528 0.0266 0 0 = =20 0 0 0 0 0 0 3 0.0094 = =20 545 0.02 61 vmlinux vmlinux local_bh_enable_i= p 509 0.3748 0 0 3 3.2e-05 0 0 = =20 0 0 0 0 4 0.0142 3 0.0094 = =20 76 0.00 36 vmlinux vmlinux can_dirty 430 0.3166 74 0.0012 148 0.0016 25 0.0492 = =20 13 0.0332 12 0.0715 9 0.0320 12 0.0376 = =20 96 0.00 46 vmlinux vmlinux rcu_process_callb= acks 372 0.2739 0 0 0 0 0 0 = =20 0 0 0 0 3 0.0107 2 0.0063 = =20 48 0.00 23 vmlinux vmlinux do_journal_end 347 0.2555 9991 0.1646 16066 0.1693 143 0.2814 = =20 66 0.1688 25 0.1489 46 0.1635 71 0.2228 = =20 3514 0.16 85 vmlinux vmlinux rb_get_reader_pag= e 346 0.2548 952 0.0157 566 0.0060 871 1.7140 = =20 321 0.8209 174 1.0362 324 1.1518 269 0.8439 = =20 655 0.03 14 vmlinux vmlinux find_busiest_grou= p 275 0.2025 1194 0.0197 1228 0.0129 1019 2.0052 = =20 203 0.5191 151 0.8992 290 1.0310 653 2.0487 = =20 799 0.03 83 vmlinux vmlinux find_next_bit 238 0.1752 3053 0.0503 4711 0.0496 116 0.2283 = =20 82 0.2097 54 0.3216 101 0.3591 101 0.3169 = =20 1174 0.05 63 oprofile oprofile (no symbols) 221 0.1627 0 0 0 0 0 0 = =20 0 0 0 0 0 0 0 0 = =20 18 8.6e- 04 vmlinux vmlinux ahci_interrupt 212 0.1561 38335 0.6314 60370 0.6361 329 0.6474 = =20 165 0.4220 136 0.8099 133 0.4728 205 0.6432 = =20 12821 0.61 49 vmlinux vmlinux apic_timer_interr= upt 210 0.1546 117 0.0019 1001 0.0105 2 0.0039 = =20 0 0 0 0 55 0.1955 16 0.0502 = =20 270 0.01 30 vmlinux vmlinux memmove 207 0.1524 0 0 7 7.4e-05 34 0.0669 = =20 0 0 0 0 93 0.3306 62 0.1945 = =20 214 0.01 03 vmlinux vmlinux search_by_key 204 0.1502 26603 0.4382 25086 0.2643 45 0.0886 = =20 18 0.0460 19 0.1131 19 0.0675 25 0.0784 = =20 6580 0.31 56 vmlinux vmlinux __do_softirq 170 0.1252 2618 0.0431 3493 0.0368 69 0.1358 = =20 42 0.1074 51 0.3037 64 0.2275 87 0.2729 = =20 856 0.04 11 vmlinux vmlinux __schedule 151 0.1112 7590 0.1250 8909 0.0939 273 0.5372 = =20 86 0.2199 73 0.4347 99 0.3519 142 0.4455 = =20 2114 0.10 14 vmlinux vmlinux sched_clock_cpu 142 0.1046 200 0.0033 165 0.0017 65 0.1279 = =20 23 0.0588 20 0.1191 15 0.0533 36 0.1129 = =20 94 0.00 45 vmlinux vmlinux raise_softirq 141 0.1038 8048 0.1326 11386 0.1200 2 0.0039 = =20 0 0 0 0 2 0.0071 4 0.0125 = =20 2589 0.12 42 vmlinux vmlinux __phys_addr 141 0.1038 23726 0.3908 24837 0.2617 89 0.1751 = =20 58 0.1483 55 0.3275 66 0.2346 62 0.1945 = =20 5398 0.25 89 vmlinux vmlinux =20 tick_nohz_stop_sched_tick 139 0.1023 43230 0.7120 68500 0.7217 105 0.2066 = =20 66 0.1688 86 0.5121 100 0.3555 100 0.3137 = =20 13725 0.65 83 vmlinux vmlinux getnstimeofday 136 0.1001 0 0 0 0 0 0 = =20 0 0 0 0 0 0 0 0 = =20 36 0.00 17 vmlinux vmlinux remove_journal_ha= sh 133 0.0979 8624 0.1420 9518 0.1003 217 0.4270 = =20 68 0.1739 64 0.3811 79 0.2808 106 0.3326 = =20 2348 0.11 26 vmlinux vmlinux native_sched_cloc= k 129 0.0950 270 0.0044 233 0.0025 355 0.6986 = =20 93 0.2378 60 0.3573 102 0.3626 105 0.3294 = =20 214 0.01 03 vmlinux vmlinux rebalance_domains 117 0.0861 0 0 36 3.8e-04 57 0.1122 = =20 0 0 10 0.0596 66 0.2346 60 0.1882 = =20 220 0.01 06 vmlinux vmlinux find_get_page 112 0.0825 287 0.0047 397 0.0042 121 0.2381 = =20 89 0.2276 63 0.3752 85 0.3022 114 0.3577 = =20 220 0.01 06 vmlinux vmlinux scheduler_tick 107 0.0788 13702 0.2257 16191 0.1706 102 0.2007 = =20 28 0.0716 44 0.2620 41 0.1458 66 0.2071 = =20 3890 0.18 66 vmlinux vmlinux tick_nohz_stop_id= le 105 0.0773 4277 0.0704 7002 0.0738 53 0.1043 = =20 26 0.0665 24 0.1429 24 0.0853 38 0.1192 = =20 1561 0.07 49 vmlinux vmlinux ring_buffer_consu= me 103 0.0758 0 0 2 2.1e-05 5 0.0098 = =20 0 0 0 0 20 0.0711 13 0.0408 = =20 61 0.00 29 vmlinux vmlinux __find_get_block 103 0.0758 49251 0.8112 85999 0.9061 166 0.3267 = =20 105 0.2685 97 0.5777 122 0.4337 138 0.4330 = =20 17108 0.82 06 vmlinux vmlinux read_tsc 98 0.0722 17406 0.2867 40897 0.4309 15 0.0295 = =20 2 0.0051 2 0.0119 62 0.2204 51 0.1600 = =20 7913 0.37 95 vmlinux vmlinux kmem_cache_alloc 98 0.0722 14817 0.2440 21291 0.2243 41 0.0807 = =20 49 0.1253 48 0.2859 62 0.2204 64 0.2008 = =20 4582 0.21 98 vmlinux vmlinux ktime_get_ts 94 0.0692 0 0 1 1.1e-05 2 0.0039 = =20 0 0 0 0 7 0.0249 6 0.0188 = =20 16 7.7e- 04 vmlinux vmlinux __find_get_block_= slow 92 0.0677 1161 0.0191 1411 0.0149 41 0.0807 = =20 39 0.0997 40 0.2382 53 0.1884 49 0.1537 = =20 408 0.01 96 vmlinux vmlinux =20 select_nohz_load_balancer 88 0.0648 11160 0.1838 54990 0.5794 69 0.1358 = =20 43 0.1100 35 0.2084 661 2.3499 55 0.1726 = =20 4513 0.21 65 vmlinux vmlinux =20 get_next_timer_interrupt 75 0.0552 0 0 0 0 0 0 = =20 0 0 0 0 0 0 0 0 = =20 6 2.9e- 04 vmlinux vmlinux make_request 69 0.0508 803 0.0132 1048 0.0110 35 0.0689 = =20 26 0.0665 15 0.0893 24 0.0853 58 0.1820 = =20 334 0.01 --More-- Pawe=B3 Staszewski pisze: > I dont know where to put this message so i start here because server=20 > that i see this funny thing is make only networking > Routing (bgpd) / traffic management (iproute2) > > This starts on kernels >=3D 2.6.29 > > On server i have almost constant traffic that is about 300Mbit/s RX=20 > and 300Mbit/s TX > Server have two interfaces: > 04:00.0 Ethernet controller: Intel Corporation 82573E Gigabit Etherne= t=20 > Controller (Copper) (rev 03) > 05:00.0 Ethernet controller: Intel Corporation 82573L Gigabit Etherne= t=20 > Controller > > iBGP peer < - > [ eth0 - forward + shaping - eth1 ] < - > eBGP > > > And what is weird is CPU Load > it fluctuate from 10% to almost 80% - but traffic is constant : > bwm-ng v0.6 (probing every 10.000s), press 'h' for help > input: /proc/net/dev type: rate > | iface Rx =20 > Tx Total > =20 > =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=20 > > lo: 0.00 b/s 0.00 b/s = =20 > 0.00 b/s > eth0: 200.22 Mb/s 266.74 Mb/s =20 > 466.96 Mb/s > eth1: 264.00 Mb/s 192.59 Mb/s =20 > 456.59 Mb/s > =20 > ---------------------------------------------------------------------= ---------=20 > > total: 464.22 Mb/s 459.34 Mb/s =20 > 923.56 Mb/s > > > bwm-ng v0.6 (probing every 10.000s), press 'h' for help > input: /proc/net/dev type: rate > / iface Rx =20 > Tx Total > =20 > =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=20 > > lo: 0.00 P/s 0.00 P/s = =20 > 0.00 P/s > eth0: 51314.95 P/s 49880.20 P/s =20 > 101195.14 P/s > eth1: 43309.85 P/s 41885.79 P/s =20 > 85195.64 P/s > =20 > ---------------------------------------------------------------------= ---------=20 > > total: 94624.80 P/s 91765.99 P/s =20 > 186390.79 P/s > > > mpstat -P ALL 1 > 18:00:05 CPU %usr %nice %sys %iowait %irq %soft =20 > %steal %guest %idle > 18:00:06 all 0.00 0.00 0.00 0.00 1.33 13.65 =20 > 0.00 0.00 85.02 > 18:00:06 0 0.00 0.00 0.00 0.00 0.00 0.00 =20 > 0.00 0.00 100.00 > 18:00:06 1 0.00 0.00 0.00 0.00 3.77 39.62 =20 > 0.00 0.00 56.60 > 18:00:06 2 0.00 0.00 0.00 0.00 7.92 70.30 =20 > 0.00 0.00 21.78 > 18:00:06 3 0.00 0.00 0.00 0.00 0.00 0.00 =20 > 0.00 0.00 100.00 > 18:00:06 4 0.00 0.00 0.00 0.00 0.00 0.00 =20 > 0.00 0.00 100.00 > 18:00:06 5 0.00 0.00 0.00 0.00 0.00 0.00 =20 > 0.00 0.00 100.00 > 18:00:06 6 0.00 0.00 0.00 0.00 0.00 0.00 =20 > 0.00 0.00 100.00 > 18:00:06 7 0.00 0.00 0.00 0.00 0.00 0.00 =20 > 0.00 0.00 100.00 > > 18:00:06 CPU %usr %nice %sys %iowait %irq %soft =20 > %steal %guest %idle > 18:00:07 all 0.12 0.00 0.12 0.00 1.19 13.35 =20 > 0.00 0.00 85.22 > 18:00:07 0 0.00 0.00 0.00 0.00 0.00 0.00 =20 > 0.00 0.00 100.00 > 18:00:07 1 0.00 0.00 0.00 0.00 2.75 42.20 =20 > 0.00 0.00 55.05 > 18:00:07 2 0.00 0.00 0.99 0.00 5.94 65.35 =20 > 0.00 0.00 27.72 > 18:00:07 3 0.00 0.00 0.00 0.00 0.00 0.00 =20 > 0.00 0.00 100.00 > 18:00:07 4 0.00 0.00 0.00 0.00 0.00 0.00 =20 > 0.00 0.00 100.00 > 18:00:07 5 0.00 0.00 0.00 0.00 0.00 0.00 =20 > 0.00 0.00 100.00 > 18:00:07 6 0.00 0.00 0.99 0.00 0.00 0.00 =20 > 0.00 0.00 99.01 > 18:00:07 7 0.00 0.00 0.00 0.00 0.00 0.00 =20 > 0.00 0.00 100.00 > > 18:00:07 CPU %usr %nice %sys %iowait %irq %soft =20 > %steal %guest %idle > 18:00:08 all 0.00 0.00 0.12 0.00 0.85 13.42 =20 > 0.00 0.00 85.61 > 18:00:08 0 0.00 0.00 0.00 0.00 0.00 0.00 =20 > 0.00 0.00 100.00 > 18:00:08 1 0.00 0.00 0.00 0.00 1.85 38.89 =20 > 0.00 0.00 59.26 > 18:00:08 2 0.00 0.00 0.00 0.00 5.94 68.32 =20 > 0.00 0.00 25.74 > 18:00:08 3 0.00 0.00 0.00 0.00 0.00 0.00 =20 > 0.00 0.00 100.00 > 18:00:08 4 0.00 0.00 0.00 0.00 0.00 0.00 =20 > 0.00 0.00 100.00 > 18:00:08 5 0.00 0.00 0.00 0.00 0.00 0.00 =20 > 0.00 0.00 100.00 > 18:00:08 6 0.00 0.00 0.00 0.00 0.00 0.00 =20 > 0.00 0.00 100.00 > 18:00:08 7 0.00 0.00 0.00 0.00 0.00 0.00 =20 > 0.00 0.00 100.00 > > 18:00:08 CPU %usr %nice %sys %iowait %irq %soft =20 > %steal %guest %idle > 18:00:09 all 0.00 0.00 0.00 0.00 1.33 14.18 =20 > 0.00 0.00 84.48 > 18:00:09 0 0.00 0.00 0.00 0.00 0.00 0.00 =20 > 0.00 0.00 100.00 > 18:00:09 1 0.00 0.00 0.99 0.00 2.97 50.50 =20 > 0.00 0.00 45.54 > 18:00:09 2 0.00 0.00 0.00 0.00 6.93 65.35 =20 > 0.00 0.00 27.72 > 18:00:09 3 0.00 0.00 0.00 0.00 0.00 0.00 =20 > 0.00 0.00 100.00 > 18:00:09 4 0.00 0.00 0.00 0.00 0.00 0.00 =20 > 0.00 0.00 100.00 > 18:00:09 5 0.00 0.00 0.00 0.00 0.00 0.00 =20 > 0.00 0.00 100.00 > 18:00:09 6 0.00 0.00 0.00 0.00 0.00 0.00 =20 > 0.00 0.00 100.00 > 18:00:09 7 0.00 0.00 0.00 0.00 0.00 0.00 =20 > 0.00 0.00 100.00 > > 18:00:09 CPU %usr %nice %sys %iowait %irq %soft =20 > %steal %guest %idle > 18:00:10 all 0.00 0.00 0.24 0.00 1.20 11.81 =20 > 0.00 0.00 86.75 > 18:00:10 0 0.00 0.00 0.00 0.00 0.00 0.00 =20 > 0.00 0.00 100.00 > 18:00:10 1 0.00 0.00 0.95 0.00 4.76 35.24 =20 > 0.00 0.00 59.05 > 18:00:10 2 0.00 0.00 0.00 0.00 4.95 59.41 =20 > 0.00 0.00 35.64 > 18:00:10 3 0.00 0.00 0.00 0.00 0.00 0.00 =20 > 0.00 0.00 100.00 > 18:00:10 4 0.00 0.00 0.00 0.00 0.00 0.00 =20 > 0.00 0.00 100.00 > 18:00:10 5 0.00 0.00 0.00 0.00 0.00 0.00 =20 > 0.00 0.00 100.00 > 18:00:10 6 0.00 0.00 0.95 0.00 0.00 0.00 =20 > 0.00 0.00 99.05 > 18:00:10 7 0.00 0.00 0.00 0.00 0.00 0.00 =20 > 0.00 0.00 100.00 > > 18:00:10 CPU %usr %nice %sys %iowait %irq %soft =20 > %steal %guest %idle > 18:00:11 all 0.00 0.00 0.24 0.00 0.36 5.97 =20 > 0.00 0.00 93.43 > 18:00:11 0 0.00 0.00 0.00 0.00 0.00 0.00 =20 > 0.00 0.00 100.00 > 18:00:11 1 0.00 0.00 0.97 0.00 0.97 17.48 =20 > 0.00 0.00 80.58 > 18:00:11 2 0.00 0.00 0.00 0.00 2.88 31.73 =20 > 0.00 0.00 65.38 > 18:00:11 3 0.00 0.00 0.00 0.00 0.00 0.00 =20 > 0.00 0.00 100.00 > 18:00:11 4 0.00 0.00 0.00 0.00 0.00 0.00 =20 > 0.00 0.00 100.00 > 18:00:11 5 0.00 0.00 0.00 0.00 0.00 0.00 =20 > 0.00 0.00 100.00 > 18:00:11 6 0.00 0.00 0.00 0.00 0.00 0.00 =20 > 0.00 0.00 100.00 > 18:00:11 7 0.00 0.00 0.00 0.00 0.00 0.00 =20 > 0.00 0.00 100.00 > > 18:00:11 CPU %usr %nice %sys %iowait %irq %soft =20 > %steal %guest %idle > 18:00:12 all 0.00 0.00 0.25 0.00 0.37 5.76 =20 > 0.00 0.00 93.63 > 18:00:12 0 0.00 0.00 0.00 1.06 0.00 0.00 =20 > 0.00 0.00 98.94 > 18:00:12 1 0.00 0.00 1.92 0.00 0.00 18.27 =20 > 0.00 0.00 79.81 > 18:00:12 2 0.00 0.00 0.96 0.00 1.92 26.92 =20 > 0.00 0.00 70.19 > 18:00:12 3 0.00 0.00 0.00 0.00 0.00 0.00 =20 > 0.00 0.00 100.00 > 18:00:12 4 0.00 0.00 0.00 0.00 0.00 0.00 =20 > 0.00 0.00 100.00 > 18:00:12 5 0.00 0.00 0.00 0.00 0.00 0.00 =20 > 0.00 0.00 100.00 > 18:00:12 6 0.96 0.00 0.00 0.00 0.00 0.00 =20 > 0.00 0.00 99.04 > 18:00:12 7 0.00 0.00 0.00 0.00 0.00 0.00 =20 > 0.00 0.00 100.00 > > 18:00:12 CPU %usr %nice %sys %iowait %irq %soft =20 > %steal %guest %idle > 18:00:13 all 0.12 0.00 0.25 0.00 0.12 2.34 =20 > 0.00 0.00 97.17 > 18:00:13 0 0.00 0.00 0.00 0.00 0.00 0.00 =20 > 0.00 0.00 100.00 > 18:00:13 1 0.00 0.00 0.00 0.00 0.97 7.77 =20 > 0.00 0.00 91.26 > 18:00:13 2 0.00 0.00 0.98 0.00 0.98 10.78 =20 > 0.00 0.00 87.25 > 18:00:13 3 0.00 0.00 0.00 0.00 0.00 0.00 =20 > 0.00 0.00 100.00 > 18:00:13 4 0.00 0.00 0.00 0.00 0.00 0.00 =20 > 0.00 0.00 100.00 > 18:00:13 5 0.00 0.00 0.00 0.00 0.00 0.00 =20 > 0.00 0.00 100.00 > 18:00:13 6 0.00 0.00 0.00 0.00 0.00 0.00 =20 > 0.00 0.00 100.00 > 18:00:13 7 0.00 0.00 0.00 0.00 0.00 0.00 =20 > 0.00 0.00 100.00 > > 18:00:13 CPU %usr %nice %sys %iowait %irq %soft =20 > %steal %guest %idle > 18:00:14 all 0.00 0.00 0.37 0.00 0.37 2.33 =20 > 0.00 0.00 96.94 > 18:00:14 0 0.00 0.00 0.00 0.00 0.00 0.00 =20 > 0.00 0.00 100.00 > 18:00:14 1 0.00 0.00 1.92 0.00 0.96 7.69 =20 > 0.00 0.00 89.42 > 18:00:14 2 0.00 0.00 0.97 0.00 0.97 10.68 =20 > 0.00 0.00 87.38 > 18:00:14 3 0.00 0.00 0.00 0.00 0.00 0.00 =20 > 0.00 0.00 100.00 > 18:00:14 4 0.00 0.00 0.00 0.00 0.00 0.00 =20 > 0.00 0.00 100.00 > 18:00:14 5 0.00 0.00 0.00 0.00 0.00 0.00 =20 > 0.00 0.00 100.00 > 18:00:14 6 0.00 0.00 0.00 0.00 0.00 0.00 =20 > 0.00 0.00 100.00 > 18:00:14 7 0.00 0.00 0.00 0.00 0.00 0.00 =20 > 0.00 0.00 100.00 > > 18:00:14 CPU %usr %nice %sys %iowait %irq %soft =20 > %steal %guest %idle > 18:00:15 all 0.00 0.00 0.61 0.00 0.25 2.45 =20 > 0.00 0.00 96.69 > 18:00:15 0 0.00 0.00 0.00 0.00 0.00 0.00 =20 > 0.00 0.00 100.00 > 18:00:15 1 0.00 0.00 3.85 0.00 0.96 7.69 =20 > 0.00 0.00 87.50 > 18:00:15 2 0.00 0.00 0.96 0.00 1.92 12.50 =20 > 0.00 0.00 84.62 > 18:00:15 3 0.00 0.00 0.00 0.00 0.00 0.00 =20 > 0.00 0.00 100.00 > 18:00:15 4 0.00 0.00 0.00 0.00 0.00 0.00 =20 > 0.00 0.00 100.00 > 18:00:15 5 0.00 0.00 0.00 0.00 0.00 0.00 =20 > 0.00 0.00 100.00 > 18:00:15 6 0.00 0.00 0.00 0.00 0.00 0.00 =20 > 0.00 0.00 100.00 > 18:00:15 7 0.00 0.00 0.00 0.00 0.00 0.00 =20 > 0.00 0.00 100.00 > > 18:00:15 CPU %usr %nice %sys %iowait %irq %soft =20 > %steal %guest %idle > 18:00:16 all 0.00 0.00 0.37 0.00 0.37 1.83 =20 > 0.00 0.00 97.43 > 18:00:16 0 0.00 0.00 0.00 0.00 0.00 0.00 =20 > 0.00 0.00 100.00 > 18:00:16 1 0.00 0.00 1.96 0.00 0.98 4.90 =20 > 0.00 0.00 92.16 > 18:00:16 2 0.00 0.00 1.00 0.00 1.00 10.00 =20 > 0.00 0.00 88.00 > 18:00:16 3 0.00 0.00 0.00 0.00 0.00 0.00 =20 > 0.00 0.00 100.00 > 18:00:16 4 0.00 0.00 0.00 0.00 0.00 0.00 =20 > 0.00 0.00 100.00 > 18:00:16 5 0.00 0.00 0.00 0.00 0.00 0.00 =20 > 0.00 0.00 100.00 > 18:00:16 6 0.98 0.00 0.00 0.00 0.00 0.00 =20 > 0.00 0.00 99.02 > 18:00:16 7 0.00 0.00 0.00 0.00 0.00 0.00 =20 > 0.00 0.00 100.00 > > 18:00:16 CPU %usr %nice %sys %iowait %irq %soft =20 > %steal %guest %idle > 18:00:17 all 0.00 0.00 0.25 0.00 0.25 2.73 =20 > 0.00 0.00 96.78 > 18:00:17 0 0.00 0.00 0.00 0.00 0.00 0.00 =20 > 0.00 0.00 100.00 > 18:00:17 1 0.00 0.00 1.96 0.00 0.00 8.82 =20 > 0.00 0.00 89.22 > 18:00:17 2 0.00 0.00 0.99 0.00 1.98 11.88 =20 > 0.00 0.00 85.15 > 18:00:17 3 0.00 0.00 0.00 0.00 0.00 0.00 =20 > 0.00 0.00 100.00 > 18:00:17 4 0.00 0.00 0.00 0.00 0.00 0.00 =20 > 0.00 0.00 100.00 > 18:00:17 5 0.00 0.00 0.00 0.00 0.00 0.00 =20 > 0.00 0.00 100.00 > 18:00:17 6 0.00 0.00 0.00 0.00 0.00 0.00 =20 > 0.00 0.00 100.00 > 18:00:17 7 0.00 0.00 0.00 0.00 0.00 0.00 =20 > 0.00 0.00 100.00 > > 18:00:17 CPU %usr %nice %sys %iowait %irq %soft =20 > %steal %guest %idle > 18:00:18 all 0.12 0.00 0.12 0.00 0.95 11.51 =20 > 0.00 0.00 87.31 > 18:00:18 0 0.00 0.00 0.00 0.00 0.00 0.00 =20 > 0.00 0.00 100.00 > 18:00:18 1 0.00 0.00 0.00 0.00 3.03 33.33 =20 > 0.00 0.00 63.64 > 18:00:18 2 0.00 0.00 0.00 0.00 5.00 64.00 =20 > 0.00 0.00 31.00 > 18:00:18 3 0.00 0.00 0.00 0.00 0.00 0.00 =20 > 0.00 0.00 100.00 > 18:00:18 4 0.00 0.00 0.00 0.00 0.00 0.00 =20 > 0.00 0.00 100.00 > 18:00:18 5 0.00 0.00 0.00 0.00 0.00 0.00 =20 > 0.00 0.00 100.00 > 18:00:18 6 0.00 0.00 0.00 0.00 0.00 0.00 =20 > 0.00 0.00 100.00 > 18:00:18 7 0.00 0.00 0.00 0.00 0.00 0.00 =20 > 0.00 0.00 100.00 > > 18:00:18 CPU %usr %nice %sys %iowait %irq %soft =20 > %steal %guest %idle > 18:00:19 all 0.36 0.00 0.12 0.00 1.46 14.48 =20 > 0.00 0.00 83.58 > 18:00:19 0 0.00 0.00 0.00 0.00 0.00 0.00 =20 > 0.00 0.00 100.00 > 18:00:19 1 0.00 0.00 0.00 0.00 4.00 46.00 =20 > 0.00 0.00 50.00 > 18:00:19 2 0.00 0.00 0.99 0.00 7.92 72.28 =20 > 0.00 0.00 18.81 > 18:00:19 3 0.00 0.00 0.00 0.00 0.00 0.00 =20 > 0.00 0.00 100.00 > 18:00:19 4 0.00 0.00 0.00 0.00 0.00 0.00 =20 > 0.00 0.00 100.00 > 18:00:19 5 2.86 0.00 0.00 0.00 0.00 0.00 =20 > 0.00 0.00 97.14 > 18:00:19 6 0.95 0.00 0.00 0.00 0.00 0.00 =20 > 0.00 0.00 99.05 > 18:00:19 7 0.00 0.00 0.00 0.00 0.00 0.00 =20 > 0.00 0.00 100.00 > > 18:00:19 CPU %usr %nice %sys %iowait %irq %soft =20 > %steal %guest %idle > 18:00:20 all 0.00 0.00 0.00 0.00 1.58 14.86 =20 > 0.00 0.00 83.56 > 18:00:20 0 0.00 0.00 0.00 0.00 0.00 0.00 =20 > 0.00 0.00 100.00 > 18:00:20 1 0.00 0.00 0.00 0.00 3.96 49.50 =20 > 0.00 0.00 46.53 > 18:00:20 2 0.00 0.00 0.00 0.00 9.00 73.00 =20 > 0.00 0.00 18.00 > 18:00:20 3 0.00 0.00 0.00 0.00 0.00 0.00 =20 > 0.00 0.00 100.00 > 18:00:20 4 0.00 0.00 0.00 0.00 0.00 0.00 =20 > 0.00 0.00 100.00 > 18:00:20 5 0.00 0.00 0.00 0.00 0.00 0.00 =20 > 0.00 0.00 100.00 > 18:00:20 6 0.00 0.00 0.00 0.00 0.00 0.00 =20 > 0.00 0.00 100.00 > 18:00:20 7 0.00 0.00 0.00 0.00 0.00 0.00 =20 > 0.00 0.00 100.00 > > 18:00:20 CPU %usr %nice %sys %iowait %irq %soft =20 > %steal %guest %idle > 18:00:21 all 0.00 0.00 0.00 0.00 1.71 14.86 =20 > 0.00 0.00 83.43 > 18:00:21 0 0.00 0.00 0.00 0.00 0.00 0.00 =20 > 0.00 0.00 100.00 > 18:00:21 1 0.00 0.00 0.00 0.00 5.05 51.52 =20 > 0.00 0.00 43.43 > 18:00:21 2 0.00 0.00 0.00 0.00 10.00 70.00 =20 > 0.00 0.00 20.00 > 18:00:21 3 0.00 0.00 0.00 0.00 0.00 0.00 =20 > 0.00 0.00 100.00 > 18:00:21 4 0.00 0.00 0.00 0.00 0.00 0.00 =20 > 0.00 0.00 100.00 > 18:00:21 5 0.00 0.00 0.00 0.00 0.00 0.00 =20 > 0.00 0.00 100.00 > 18:00:21 6 0.00 0.00 0.00 0.00 0.00 0.00 =20 > 0.00 0.00 100.00 > 18:00:21 7 0.00 0.00 0.00 0.00 0.00 0.00 =20 > 0.00 0.00 100.00 > > 18:00:21 CPU %usr %nice %sys %iowait %irq %soft =20 > %steal %guest %idle > 18:00:22 all 0.00 0.00 0.24 0.00 1.70 13.97 =20 > 0.00 0.00 84.08 > 18:00:22 0 0.00 0.00 0.00 0.00 0.00 0.00 =20 > 0.00 0.00 100.00 > 18:00:22 1 0.00 0.00 0.00 0.00 4.95 42.57 =20 > 0.00 0.00 52.48 > 18:00:22 2 0.00 0.00 1.01 0.00 8.08 72.73 =20 > 0.00 0.00 18.18 > 18:00:22 3 0.00 0.00 0.00 0.00 0.00 0.00 =20 > 0.00 0.00 100.00 > 18:00:22 4 0.00 0.00 0.00 0.00 0.00 0.00 =20 > 0.00 0.00 100.00 > 18:00:22 5 0.00 0.00 0.00 0.00 0.00 0.00 =20 > 0.00 0.00 100.00 > 18:00:22 6 0.00 0.00 0.95 0.00 0.00 0.00 =20 > 0.00 0.00 99.05 > 18:00:22 7 0.00 0.00 0.00 0.00 0.00 0.00 =20 > 0.00 0.00 100.00 > > 18:00:22 CPU %usr %nice %sys %iowait %irq %soft =20 > %steal %guest %idle > 18:00:23 all 0.12 0.00 0.00 0.00 1.58 14.13 =20 > 0.00 0.00 84.17 > 18:00:23 0 0.00 0.00 0.00 0.00 0.00 0.00 =20 > 0.00 0.00 100.00 > 18:00:23 1 0.00 0.00 0.00 0.00 5.00 45.00 =20 > 0.00 0.00 50.00 > 18:00:23 2 0.99 0.00 0.00 0.00 8.91 70.30 =20 > 0.00 0.00 19.80 > 18:00:23 3 0.00 0.00 0.00 0.00 0.00 0.00 =20 > 0.00 0.00 100.00 > 18:00:23 4 0.00 0.00 0.00 0.00 0.00 0.00 =20 > 0.00 0.00 100.00 > 18:00:23 5 0.00 0.00 0.00 0.00 0.00 0.00 =20 > 0.00 0.00 100.00 > 18:00:23 6 0.00 0.00 0.00 0.00 0.00 0.00 =20 > 0.00 0.00 100.00 > 18:00:23 7 0.00 0.00 0.00 0.00 0.00 0.00 =20 > 0.00 0.00 100.00 > > 18:00:23 CPU %usr %nice %sys %iowait %irq %soft =20 > %steal %guest %idle > 18:00:24 all 0.00 0.00 0.00 0.00 1.34 14.13 =20 > 0.00 0.00 84.53 > 18:00:24 0 0.00 0.00 0.00 0.00 0.00 0.00 =20 > 0.00 0.00 100.00 > 18:00:24 1 0.00 0.00 0.00 0.00 4.04 42.42 =20 > 0.00 0.00 53.54 > 18:00:24 2 0.00 0.00 0.00 0.00 6.93 73.27 =20 > 0.00 0.00 19.80 > 18:00:24 3 0.00 0.00 0.00 0.00 0.00 0.00 =20 > 0.00 0.00 100.00 > 18:00:24 4 0.00 0.00 0.00 0.00 0.00 0.00 =20 > 0.00 0.00 100.00 > 18:00:24 5 0.00 0.00 0.00 0.00 0.00 0.00 =20 > 0.00 0.00 100.00 > 18:00:24 6 0.00 0.00 0.00 0.00 0.00 0.00 =20 > 0.00 0.00 100.00 > 18:00:24 7 0.00 0.00 0.00 0.00 0.00 0.00 =20 > 0.00 0.00 100.00 > > 18:00:24 CPU %usr %nice %sys %iowait %irq %soft =20 > %steal %guest %idle > 18:00:25 all 0.12 0.00 0.12 0.00 1.45 14.67 =20 > 0.00 0.00 83.64 > 18:00:25 0 0.00 0.00 0.00 0.00 0.00 0.00 =20 > 0.00 0.00 100.00 > 18:00:25 1 0.00 0.00 0.99 0.00 3.96 45.54 =20 > 0.00 0.00 49.50 > 18:00:25 2 0.00 0.00 0.00 0.00 8.00 75.00 =20 > 0.00 0.00 17.00 > 18:00:25 3 0.00 0.00 0.00 0.00 0.00 0.00 =20 > 0.00 0.00 100.00 > 18:00:25 4 0.00 0.00 0.00 0.00 0.00 0.00 =20 > 0.00 0.00 100.00 > 18:00:25 5 0.00 0.00 0.00 0.00 0.00 0.00 =20 > 0.00 0.00 100.00 > 18:00:25 6 0.95 0.00 0.00 0.00 0.00 0.00 =20 > 0.00 0.00 99.05 > 18:00:25 7 0.00 0.00 0.00 0.00 0.00 0.00 =20 > 0.00 0.00 100.00 > > 18:00:25 CPU %usr %nice %sys %iowait %irq %soft =20 > %steal %guest %idle > 18:00:26 all 0.00 0.00 0.00 0.00 1.22 14.60 =20 > 0.00 0.00 84.18 > 18:00:26 0 0.00 0.00 0.00 0.00 0.00 0.00 =20 > 0.00 0.00 100.00 > 18:00:26 1 0.99 0.00 0.00 0.00 2.97 50.50 =20 > 0.00 0.00 45.54 > 18:00:26 2 0.00 0.00 0.00 0.00 7.07 68.69 =20 > 0.00 0.00 24.24 > 18:00:26 3 0.00 0.00 0.00 0.00 0.00 0.00 =20 > 0.00 0.00 100.00 > 18:00:26 4 0.00 0.00 0.00 0.00 0.00 0.00 =20 > 0.00 0.00 100.00 > 18:00:26 5 0.00 0.00 0.00 0.00 0.00 0.00 =20 > 0.00 0.00 100.00 > 18:00:26 6 0.00 0.00 0.00 0.00 0.00 0.00 =20 > 0.00 0.00 100.00 > 18:00:26 7 0.00 0.00 0.00 0.00 0.00 0.00 =20 > 0.00 0.00 100.00 > > 18:00:26 CPU %usr %nice %sys %iowait %irq %soft =20 > %steal %guest %idle > 18:00:27 all 0.00 0.00 0.00 0.00 1.59 15.02 =20 > 0.00 0.00 83.39 > 18:00:27 0 0.00 0.00 0.00 0.00 0.00 0.00 =20 > 0.00 0.00 100.00 > 18:00:27 1 0.00 0.00 0.00 0.00 6.00 52.00 =20 > 0.00 0.00 42.00 > 18:00:27 2 0.00 0.00 0.00 0.00 6.93 71.29 =20 > 0.00 0.00 21.78 > 18:00:27 3 0.00 0.00 0.00 0.00 0.00 0.00 =20 > 0.00 0.00 100.00 > 18:00:27 4 0.00 0.00 0.00 0.00 0.00 0.00 =20 > 0.00 0.00 100.00 > 18:00:27 5 0.00 0.00 0.00 0.00 0.00 0.00 =20 > 0.00 0.00 100.00 > 18:00:27 6 0.00 0.00 0.00 0.00 0.00 0.00 =20 > 0.00 0.00 100.00 > 18:00:27 7 0.00 0.00 0.00 0.00 0.00 0.00 =20 > 0.00 0.00 100.00 > > 18:00:27 CPU %usr %nice %sys %iowait %irq %soft =20 > %steal %guest %idle > 18:00:28 all 0.00 0.00 0.00 0.00 1.33 14.56 =20 > 0.00 0.00 84.10 > 18:00:28 0 0.00 0.00 0.00 0.00 0.00 0.99 =20 > 0.00 0.00 99.01 > 18:00:28 1 0.00 0.00 0.00 0.00 4.00 50.00 =20 > 0.00 0.00 46.00 > 18:00:28 2 0.00 0.00 0.00 0.00 6.06 69.70 =20 > 0.00 0.00 24.24 > 18:00:28 3 0.00 0.00 0.00 0.00 0.00 0.00 =20 > 0.00 0.00 100.00 > 18:00:28 4 0.00 0.00 0.00 0.00 0.00 0.00 =20 > 0.00 0.00 100.00 > 18:00:28 5 0.00 0.00 0.00 0.00 0.00 0.00 =20 > 0.00 0.00 100.00 > 18:00:28 6 0.00 0.00 0.00 0.00 0.00 0.00 =20 > 0.00 0.00 100.00 > 18:00:28 7 0.00 0.00 0.00 0.00 0.00 0.00 =20 > 0.00 0.00 100.00 > > at the same time top shows: > PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND > 28 root 15 -5 0 0 0 S 2 0.0 0:16.60 events/1 > 29 root 15 -5 0 0 0 S 2 0.0 0:15.62 events/2 > 10 root 15 -5 0 0 0 S 2 0.0 0:41.01 ksoftirqd= /2 > 7 root 15 -5 0 0 0 S 1 0.0 0:31.17 ksoftirqd= /1 > 30 root 15 -5 0 0 0 S 1 0.0 0:03.00 events/3 > 6545 root 20 0 11112 3244 1780 S 0 0.0 0:08.69 oprofiled > 6845 root 20 0 19084 1384 1008 R 0 0.0 0:00.07 top > 1 root 20 0 3812 612 520 S 0 0.0 0:01.00 init > 2 root 15 -5 0 0 0 S 0 0.0 0:00.00 kthreadd > 3 root RT -5 0 0 0 S 0 0.0 0:00.00 migration= /0 > 4 root 15 -5 0 0 0 S 0 0.0 0:01.04 ksoftirqd= /0 > 5 root RT -5 0 0 0 S 0 0.0 0:00.00 watchdog/= 0 > 6 root RT -5 0 0 0 S 0 0.0 0:00.00 migration= /1 > 8 root RT -5 0 0 0 S 0 0.0 0:00.00 watchdog/= 1 > 9 root RT -5 0 0 0 S 0 0.0 0:00.00 migration= /2 > 11 root RT -5 0 0 0 S 0 0.0 0:00.00 watchdog/= 2 > 12 root RT -5 0 0 0 S 0 0.0 0:00.00 migration= /3 > 13 root 15 -5 0 0 0 S 0 0.0 0:00.00 ksoftirqd= /3 > 14 root RT -5 0 0 0 S 0 0.0 0:00.00 watchdog/= 3 > 15 root RT -5 0 0 0 S 0 0.0 0:00.00 migration= /4 > 16 root 15 -5 0 0 0 S 0 0.0 0:00.00 ksoftirqd= /4 > 17 root RT -5 0 0 0 S 0 0.0 0:00.00 watchdog/= 4 > 18 root RT -5 0 0 0 S 0 0.0 0:00.00 migration= /5 > 19 root 15 -5 0 0 0 S 0 0.0 0:00.00 ksoftirqd= /5 > 20 root RT -5 0 0 0 S 0 0.0 0:00.00 watchdog/= 5 > 21 root RT -5 0 0 0 S 0 0.0 0:00.00 migration= /6 > 22 root 15 -5 0 0 0 S 0 0.0 0:00.00 ksoftirqd= /6 > 23 root RT -5 0 0 0 S 0 0.0 0:00.00 watchdog/= 6 > 24 root RT -5 0 0 0 S 0 0.0 0:00.00 migration= /7 > 25 root 15 -5 0 0 0 S 0 0.0 0:00.00 ksoftirqd= /7 > 26 root RT -5 0 0 0 S 0 0.0 0:00.00 watchdog/= 7 > 27 root 15 -5 0 0 0 S 0 0.0 0:00.00 events/0 > 31 root 15 -5 0 0 0 S 0 0.0 0:00.00 events/4 > 32 root 15 -5 0 0 0 S 0 0.0 0:00.01 events/5 > 33 root 15 -5 0 0 0 S 0 0.0 0:00.02 events/6 > 34 root 15 -5 0 0 0 S 0 0.0 0:00.01 events/7 > 35 root 15 -5 0 0 0 S 0 0.0 0:00.01 khelper > 41 root 15 -5 0 0 0 S 0 0.0 0:00.00 async/mgr > 159 root 15 -5 0 0 0 S 0 0.0 0:00.00 kblockd/0 > 160 root 15 -5 0 0 0 S 0 0.0 0:00.00 kblockd/1 > 161 root 15 -5 0 0 0 S 0 0.0 0:00.00 kblockd/2 > 162 root 15 -5 0 0 0 S 0 0.0 0:00.00 kblockd/3 > 163 root 15 -5 0 0 0 S 0 0.0 0:00.00 kblockd/4 > 164 root 15 -5 0 0 0 S 0 0.0 0:00.00 kblockd/5 > 165 root 15 -5 0 0 0 S 0 0.0 0:00.00 kblockd/6 > 166 root 15 -5 0 0 0 S 0 0.0 0:00.00 kblockd/7 > 167 root 15 -5 0 0 0 S 0 0.0 0:00.00 kacpid > 168 root 15 -5 0 0 0 S 0 0.0 0:00.00 kacpi_not= ify > 246 root 15 -5 0 0 0 S 0 0.0 0:00.00 ata/0 > 247 root 15 -5 0 0 0 S 0 0.0 0:00.00 ata/1 > > and oprofile output: > CPU: Core 2, speed 3000.08 MHz (estimated) > Counted CPU_CLK_UNHALTED events (Clock cycles when not halted) with a= =20 > unit mask of 0x00 (Unhalted core cycles) count 100000 > samples % image name app name =20 > symbol name > 11138304 12.3598 vmlinux vmlinux =20 > rb_next > 7714260 8.5602 vmlinux vmlinux =20 > e1000_clean > 5380466 5.9705 vmlinux vmlinux =20 > e1000_intr_msi > 4881445 5.4168 vmlinux vmlinux =20 > u32_classify > 3977072 4.4132 vmlinux vmlinux =20 > _raw_spin_lock > 3471611 3.8523 vmlinux vmlinux =20 > hfsc_dequeue > 3103493 3.4438 vmlinux vmlinux =20 > hfsc_enqueue > 2636824 2.9260 vmlinux vmlinux =20 > delay_tsc > 2464293 2.7345 vmlinux vmlinux =20 > e1000_xmit_frame > 2035779 2.2590 vmlinux vmlinux =20 > e1000_clean_tx_irq > 1947231 2.1608 vmlinux vmlinux =20 > mwait_idle > 1919540 2.1300 vmlinux vmlinux =20 > ipt_do_table > 1908544 2.1178 vmlinux vmlinux =20 > e1000_clean_rx_irq > 1865996 2.0706 vmlinux vmlinux =20 > update_vf > 1829293 2.0299 vmlinux vmlinux =20 > rb_erase > 1777847 1.9728 vmlinux vmlinux =20 > ip_route_input > 1430773 1.5877 vmlinux vmlinux =20 > rb_insert_color > 1329674 1.4755 vmlinux vmlinux =20 > __alloc_skb > 1091678 1.2114 vmlinux vmlinux =20 > sfq_enqueue > 985105 1.0931 vmlinux vmlinux =20 > rb_first > 944928 1.0486 vmlinux vmlinux =20 > sfq_dequeue > 756452 0.8394 vmlinux vmlinux =20 > read_tsc > 728703 0.8086 vmlinux vmlinux =20 > nf_iterate > 665551 0.7385 vmlinux vmlinux =20 > ip_forward > 662616 0.7353 vmlinux vmlinux =20 > dev_queue_xmit > 648919 0.7201 vmlinux vmlinux =20 > __qdisc_run > 615592 0.6831 vmlinux vmlinux =20 > ip_rcv > 589743 0.6544 vmlinux vmlinux =20 > apic_timer_interrupt > 550581 0.6110 vmlinux vmlinux =20 > irq_entries_start > 536172 0.5950 vmlinux vmlinux =20 > getnstimeofday > 527467 0.5853 vmlinux vmlinux =20 > rb_last > 508305 0.5640 vmlinux vmlinux =20 > skb_release_data > 506763 0.5623 vmlinux vmlinux k= free > 506269 0.5618 vmlinux vmlinux =20 > dst_release > 464412 0.5153 vmlinux vmlinux =20 > __hrtimer_start_range_ns > 456354 0.5064 vmlinux vmlinux =20 > e1000_alloc_rx_buffers > 435339 0.4831 vmlinux vmlinux =20 > dev_hard_start_xmit > 371439 0.4122 vmlinux vmlinux =20 > local_bh_enable > 369266 0.4098 vmlinux vmlinux =20 > __rb_rotate_left > 361627 0.4013 vmlinux vmlinux =20 > net_tx_action > 357797 0.3970 vmlinux vmlinux =20 > rtsc_y2x > 350595 0.3890 vmlinux vmlinux =20 > _raw_spin_unlock > 349359 0.3877 vmlinux vmlinux =20 > kmem_cache_alloc > 336255 0.3731 vmlinux vmlinux =20 > skb_dma_map > 328809 0.3649 vmlinux vmlinux =20 > netif_receive_skb > 315826 0.3505 vmlinux vmlinux =20 > fn_trie_lookup > 313065 0.3474 vmlinux vmlinux =20 > __kmalloc > 307684 0.3414 vmlinux vmlinux =20 > __do_softirq > 298676 0.3314 vmlinux vmlinux =20 > skb_pull > 292352 0.3244 vmlinux vmlinux =20 > __napi_complete > 288878 0.3206 vmlinux vmlinux =20 > get_next_timer_interrupt > 280449 0.3112 vmlinux vmlinux =20 > ip_finish_output > 264482 0.2935 vmlinux vmlinux =20 > consume_skb > 251622 0.2792 vmlinux vmlinux =20 > __remove_hrtimer > 227178 0.2521 vmlinux vmlinux =20 > tc_classify_compat > 226224 0.2510 vmlinux vmlinux =20 > handle_edge_irq > > > > proces that this machine is running is only quagga (zebra + bgpd +=20 > oprofiled) > > > Regards > Pawe=B3 Staszewski > > > --=20 > To unsubscribe from this list: send the line "unsubscribe netdev" in > the body of a message to majordomo@vger.kernel.org > More majordomo info at http://vger.kernel.org/majordomo-info.html > >