* miserable performance of 2.6.21 under network load
@ 2007-07-24 17:25 Aaron Porter
2007-07-24 18:48 ` Andi Kleen
2007-07-24 18:49 ` Chris Snook
0 siblings, 2 replies; 10+ messages in thread
From: Aaron Porter @ 2007-07-24 17:25 UTC (permalink / raw)
To: linux-kernel
I'm in the process up upgrading a pool of apache servers from
2.6.17.8 to 2.6.21.5, and we're seeing a pretty major change in behavior.
Under identical network load, 2.6.21 has a load average more than 3 times
higher, cpu 0 spends well over 90% of its time in interrupts (vs ~30%
under 2.6.17). When we hit 3k apache sessions, ksoftirqd eats 100% of cpu0
and our network traffic drops off rapidly. The end result is that 2.6.17
performs twice as well under this load.
^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: miserable performance of 2.6.21 under network load
2007-07-24 17:25 miserable performance of 2.6.21 under network load Aaron Porter
@ 2007-07-24 18:48 ` Andi Kleen
2007-07-24 18:49 ` Aaron Porter
2007-07-24 18:49 ` Chris Snook
1 sibling, 1 reply; 10+ messages in thread
From: Andi Kleen @ 2007-07-24 18:48 UTC (permalink / raw)
To: Aaron Porter; +Cc: linux-kernel
Aaron Porter <atporter@primate.net> writes:
> I'm in the process up upgrading a pool of apache servers from
> 2.6.17.8 to 2.6.21.5, and we're seeing a pretty major change in behavior.
> Under identical network load, 2.6.21 has a load average more than 3 times
> higher, cpu 0 spends well over 90% of its time in interrupts (vs ~30%
> under 2.6.17). When we hit 3k apache sessions, ksoftirqd eats 100% of cpu0
> and our network traffic drops off rapidly. The end result is that 2.6.17
> performs twice as well under this load.
Can you oprofile it?
-Andi
^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: miserable performance of 2.6.21 under network load
2007-07-24 18:48 ` Andi Kleen
@ 2007-07-24 18:49 ` Aaron Porter
2007-07-24 19:13 ` Jeff Garzik
` (3 more replies)
0 siblings, 4 replies; 10+ messages in thread
From: Aaron Porter @ 2007-07-24 18:49 UTC (permalink / raw)
To: Andi Kleen, linux-kernel
On Tue, Jul 24, 2007 at 08:48:00PM +0200, Andi Kleen wrote:
> Aaron Porter <atporter@primate.net> writes:
>
> > I'm in the process up upgrading a pool of apache servers from
> > 2.6.17.8 to 2.6.21.5, and we're seeing a pretty major change in behavior.
> > Under identical network load, 2.6.21 has a load average more than 3 times
> > higher, cpu 0 spends well over 90% of its time in interrupts (vs ~30%
> > under 2.6.17). When we hit 3k apache sessions, ksoftirqd eats 100% of cpu0
> > and our network traffic drops off rapidly. The end result is that 2.6.17
> > performs twice as well under this load.
>
> Can you oprofile it?
# opreport -l
CPU: AMD64 processors, speed 1994.52 MHz (estimated)
Counted CPU_CLK_UNHALTED events (Cycles outside of halt state) with a unit mask of 0x00 (No unit mask) count 100000
samples % app name symbol name
914379 48.8404 vmlinux-2.6.21.5 check_poison_obj
341920 18.2632 vmlinux-2.6.21.5 poison_obj
37355 1.9953 nf_conntrack (no symbols)
36777 1.9644 vmlinux-2.6.21.5 __copy_to_user_ll
35095 1.8746 vmlinux-2.6.21.5 skb_copy_bits
24256 1.2956 vmlinux-2.6.21.5 __copy_from_user_ll
23988 1.2813 vmlinux-2.6.21.5 cache_alloc_debugcheck_after
16897 0.9025 vmlinux-2.6.21.5 debug_smp_processor_id
14323 0.7650 vmlinux-2.6.21.5 kfree
13143 0.7020 vmlinux-2.6.21.5 tcp_clean_rtx_queue
11742 0.6272 vmlinux-2.6.21.5 sub_preempt_count
11484 0.6134 vmlinux-2.6.21.5 add_preempt_count
9218 0.4924 libc-2.3.2.so (no symbols)
8878 0.4742 vmlinux-2.6.21.5 tcp_write_xmit
8513 0.4547 vmlinux-2.6.21.5 cache_free_debugcheck
7757 0.4143 vmlinux-2.6.21.5 kmem_cache_free
7622 0.4071 vmlinux-2.6.21.5 obj_size
7352 0.3927 oprofiled (no symbols)
7041 0.3761 vmlinux-2.6.21.5 kfree_skb
6955 0.3715 vmlinux-2.6.21.5 tcp_v4_rcv
6893 0.3682 vmlinux-2.6.21.5 tg3_rx
6731 0.3595 iptable_nat (no symbols)
6673 0.3564 vmlinux-2.6.21.5 ip_queue_xmit
6482 0.3462 vmlinux-2.6.21.5 nf_iterate
6345 0.3389 vmlinux-2.6.21.5 kmem_cache_alloc
6280 0.3354 vmlinux-2.6.21.5 skb_clone
5922 0.3163 vmlinux-2.6.21.5 ip_output
5885 0.3143 vmlinux-2.6.21.5 __alloc_skb
5689 0.3039 vmlinux-2.6.21.5 tg3_start_xmit_dma_bug
5617 0.3000 vmlinux-2.6.21.5 check_irq_off
5038 0.2691 vmlinux-2.6.21.5 _raw_spin_unlock
4891 0.2612 nf_nat (no symbols)
4692 0.2506 vmlinux-2.6.21.5 __tcp_select_window
4665 0.2492 vmlinux-2.6.21.5 obj_offset
4556 0.2434 vmlinux-2.6.21.5 __kfree_skb
4425 0.2364 vmlinux-2.6.21.5 schedule
4278 0.2285 vmlinux-2.6.21.5 tcp_sendmsg
4050 0.2163 vmlinux-2.6.21.5 __kmalloc_track_caller
4035 0.2155 vmlinux-2.6.21.5 preempt_schedule
4014 0.2144 nf_conntrack_ipv4 (no symbols)
3564 0.1904 vmlinux-2.6.21.5 copy_skb_header
3547 0.1895 vmlinux-2.6.21.5 skb_release_data
3436 0.1835 vmlinux-2.6.21.5 _raw_spin_trylock
3403 0.1818 vmlinux-2.6.21.5 tcp_rcv_established
3290 0.1757 vmlinux-2.6.21.5 tcp_set_skb_tso_segs
3004 0.1605 vmlinux-2.6.21.5 check_slabp
2947 0.1574 vmlinux-2.6.21.5 tcp_transmit_skb
2919 0.1559 vmlinux-2.6.21.5 nf_hook_slow
2889 0.1543 vmlinux-2.6.21.5 dev_queue_xmit
2803 0.1497 vmlinux-2.6.21.5 __qdisc_run
2645 0.1413 vmlinux-2.6.21.5 fget_light
2633 0.1406 vmlinux-2.6.21.5 cache_alloc_refill
2587 0.1382 vmlinux-2.6.21.5 dbg_redzone1
2477 0.1323 oprofile (no symbols)
2416 0.1290 vmlinux-2.6.21.5 cache_flusharray
2342 0.1251 vmlinux-2.6.21.5 ip_route_input
2263 0.1209 vmlinux-2.6.21.5 skb_make_writable
2156 0.1152 vmlinux-2.6.21.5 tcp_poll
2141 0.1144 vmlinux-2.6.21.5 local_bh_enable_ip
2119 0.1132 vmlinux-2.6.21.5 tcp_ack
2074 0.1108 vmlinux-2.6.21.5 _decode_session4
2029 0.1084 vmlinux-2.6.21.5 find_get_page
1963 0.1049 vmlinux-2.6.21.5 isolate_lru_pages
1943 0.1038 vmlinux-2.6.21.5 mask_and_ack_8259A
1907 0.1019 vmlinux-2.6.21.5 sock_wfree
1837 0.0981 vmlinux-2.6.21.5 ip_rcv
1796 0.0959 vmlinux-2.6.21.5 kfree_skbmem
1765 0.0943 vmlinux-2.6.21.5 check_spinlock_acquired_node
1749 0.0934 vmlinux-2.6.21.5 __switch_to
1720 0.0919 vmlinux-2.6.21.5 tcp_rtt_estimator
1695 0.0905 vmlinux-2.6.21.5 sk_stream_write_space
1682 0.0898 vmlinux-2.6.21.5 csum_partial
1605 0.0857 vmlinux-2.6.21.5 __wake_up_bit
1511 0.0807 vmlinux-2.6.21.5 find_busiest_group
1494 0.0798 vmlinux-2.6.21.5 pfifo_fast_dequeue
1492 0.0797 vmlinux-2.6.21.5 local_bh_enable
1472 0.0786 vmlinux-2.6.21.5 blk_recount_segments
1449 0.0774 vmlinux-2.6.21.5 __mod_zone_page_state
1430 0.0764 vmlinux-2.6.21.5 copy_from_user
1410 0.0753 vmlinux-2.6.21.5 pfifo_fast_enqueue
1354 0.0723 vmlinux-2.6.21.5 do_readv_writev
1350 0.0721 vmlinux-2.6.21.5 netif_receive_skb
1328 0.0709 vmlinux-2.6.21.5 xfrm_decode_session
1317 0.0703 vmlinux-2.6.21.5 do_gettimeofday
1302 0.0695 vmlinux-2.6.21.5 nf_proto_csum_replace4
1273 0.0680 vmlinux-2.6.21.5 transfer_objects
1265 0.0676 vmlinux-2.6.21.5 find_next_bit
1255 0.0670 vmlinux-2.6.21.5 atomic_notifier_call_chain
1224 0.0654 vmlinux-2.6.21.5 dbg_redzone2
1217 0.0650 vmlinux-2.6.21.5 ip_xfrm_me_harder
1181 0.0631 vmlinux-2.6.21.5 tcp_sacktag_write_queue
1150 0.0614 vmlinux-2.6.21.5 bictcp_cong_avoid
1148 0.0613 vmlinux-2.6.21.5 free_pages_bulk
1136 0.0607 vmlinux-2.6.21.5 dbg_userword
1119 0.0598 vmlinux-2.6.21.5 system_call
1118 0.0597 vmlinux-2.6.21.5 cache_grow
1096 0.0585 vmlinux-2.6.21.5 try_to_wake_up
1094 0.0584 vmlinux-2.6.21.5 __mod_timer
1086 0.0580 vmlinux-2.6.21.5 skb_copy
1082 0.0578 vmlinux-2.6.21.5 _raw_read_unlock
1078 0.0576 vmlinux-2.6.21.5 __rmqueue
1040 0.0556 vmlinux-2.6.21.5 ip_local_deliver
1021 0.0545 vmlinux-2.6.21.5 xfrm_lookup
1017 0.0543 vmlinux-2.6.21.5 free_hot_cold_page
1003 0.0536 vmlinux-2.6.21.5 tcp_mtu_probe
953 0.0509 vmlinux-2.6.21.5 default_idle
943 0.0504 vmlinux-2.6.21.5 kunmap_atomic
942 0.0503 vmlinux-2.6.21.5 do_generic_mapping_read
941 0.0503 vmlinux-2.6.21.5 __first_cpu
930 0.0497 vmlinux-2.6.21.5 shrink_page_list
923 0.0493 vmlinux-2.6.21.5 kfree_debugcheck
913 0.0488 vmlinux-2.6.21.5 kmap_atomic
902 0.0482 vmlinux-2.6.21.5 free_block
896 0.0479 vmlinux-2.6.21.5 restore_nocheck
893 0.0477 vmlinux-2.6.21.5 __bio_add_page
875 0.0467 vmlinux-2.6.21.5 tcp_init_tso_segs
867 0.0463 vmlinux-2.6.21.5 tcp_current_mss
859 0.0459 vmlinux-2.6.21.5 radix_tree_lookup
835 0.0446 vmlinux-2.6.21.5 add_to_page_cache
834 0.0445 httpd apr_poll
832 0.0444 vmlinux-2.6.21.5 eth_type_trans
823 0.0440 vmlinux-2.6.21.5 do_mpage_readpage
816 0.0436 vmlinux-2.6.21.5 slab_put_obj
811 0.0433 vmlinux-2.6.21.5 _raw_write_unlock
785 0.0419 sunrpc (no symbols)
715 0.0382 vmlinux-2.6.21.5 _raw_read_trylock
707 0.0378 vmlinux-2.6.21.5 buffered_rmqueue
706 0.0377 vmlinux-2.6.21.5 __d_lookup
694 0.0371 vmlinux-2.6.21.5 do_poll
680 0.0363 vmlinux-2.6.21.5 radix_tree_insert
678 0.0362 httpd apr_socket_sendv
654 0.0349 vmlinux-2.6.21.5 rmqueue_bulk
612 0.0327 vmlinux-2.6.21.5 recalc_task_prio
590 0.0315 vmlinux-2.6.21.5 local_bh_disable
587 0.0314 httpd blowfishEncrypt
583 0.0311 vmlinux-2.6.21.5 remove_mapping
576 0.0308 httpd writev_it_all
573 0.0306 vmlinux-2.6.21.5 do_sys_poll
572 0.0306 vmlinux-2.6.21.5 kmem_flagcheck
567 0.0303 vmlinux-2.6.21.5 internal_add_timer
566 0.0302 vmlinux-2.6.21.5 irq_entries_start
564 0.0301 vmlinux-2.6.21.5 __link_path_walk
561 0.0300 vmlinux-2.6.21.5 mpage_end_io_read
547 0.0292 vmlinux-2.6.21.5 notifier_call_chain
539 0.0288 vmlinux-2.6.21.5 __activate_task
537 0.0287 vmlinux-2.6.21.5 tg3_set_txd
530 0.0283 vmlinux-2.6.21.5 prep_new_page
522 0.0279 vmlinux-2.6.21.5 radix_tree_delete
521 0.0278 vmlinux-2.6.21.5 _write_lock_bh
515 0.0275 vmlinux-2.6.21.5 __do_page_cache_readahead
515 0.0275 vmlinux-2.6.21.5 _read_lock_irqsave
511 0.0273 vmlinux-2.6.21.5 tcp_fastretrans_alert
503 0.0269 vmlinux-2.6.21.5 schedule_timeout
491 0.0262 vmlinux-2.6.21.5 _raw_write_trylock
491 0.0262 vmlinux-2.6.21.5 get_task_mm
485 0.0259 vmlinux-2.6.21.5 effective_prio
485 0.0259 vmlinux-2.6.21.5 xfrm_policy_get_afinfo
483 0.0258 vmlinux-2.6.21.5 blk_rq_map_sg
481 0.0257 vmlinux-2.6.21.5 enable_8259A_irq
478 0.0255 vmlinux-2.6.21.5 twa_interrupt
472 0.0252 vmlinux-2.6.21.5 __tcp_push_pending_frames
472 0.0252 vmlinux-2.6.21.5 move_tasks
464 0.0248 vmlinux-2.6.21.5 find_vma
454 0.0242 vmlinux-2.6.21.5 number
451 0.0241 httpd core_output_filter
450 0.0240 vmlinux-2.6.21.5 file_read_actor
449 0.0240 vmlinux-2.6.21.5 twa_scsiop_execute_scsi
446 0.0238 vmlinux-2.6.21.5 unlock_page
442 0.0236 vmlinux-2.6.21.5 _read_lock_bh
431 0.0230 vmlinux-2.6.21.5 get_page_from_freelist
430 0.0230 vmlinux-2.6.21.5 sched_clock
413 0.0221 vmlinux-2.6.21.5 mempool_free
412 0.0220 vmlinux-2.6.21.5 free_poll_entry
407 0.0217 httpd apr_wait_for_io_or_timeout
406 0.0217 vmlinux-2.6.21.5 dm_any_congested
402 0.0215 vmlinux-2.6.21.5 tcp_v4_do_rcv
394 0.0210 vmlinux-2.6.21.5 tcp_cwnd_validate
390 0.0208 vmlinux-2.6.21.5 dnotify_parent
386 0.0206 vmlinux-2.6.21.5 do_sync_readv_writev
386 0.0206 vmlinux-2.6.21.5 tg3_tx
377 0.0201 vmlinux-2.6.21.5 tcp_select_window
376 0.0201 vmlinux-2.6.21.5 resched_task
375 0.0200 vmlinux-2.6.21.5 reschedule_interrupt
361 0.0193 vmlinux-2.6.21.5 tcp_check_space
355 0.0190 vmlinux-2.6.21.5 lock_timer_base
352 0.0188 vmlinux-2.6.21.5 sk_stream_wait_memory
348 0.0186 vmlinux-2.6.21.5 page_address
339 0.0181 vmlinux-2.6.21.5 generic_make_request
331 0.0177 vmlinux-2.6.21.5 __pagevec_lru_add
330 0.0176 vmlinux-2.6.21.5 dm_table_any_congested
330 0.0176 vmlinux-2.6.21.5 syscall_exit
330 0.0176 vmlinux-2.6.21.5 task_rq_lock
328 0.0175 vmlinux-2.6.21.5 tcp_v4_send_check
323 0.0173 vmlinux-2.6.21.5 cache_init_objs
316 0.0169 vmlinux-2.6.21.5 dm_request
314 0.0168 vmlinux-2.6.21.5 tg3_recycle_rx
313 0.0167 vmlinux-2.6.21.5 dec_pending
313 0.0167 vmlinux-2.6.21.5 tcp_enter_loss
310 0.0166 vmlinux-2.6.21.5 update_send_head
309 0.0165 vmlinux-2.6.21.5 dm_get_table
307 0.0164 vmlinux-2.6.21.5 cond_resched
305 0.0163 vmlinux-2.6.21.5 mod_zone_page_state
300 0.0160 vmlinux-2.6.21.5 disk_round_stats
298 0.0159 vmlinux-2.6.21.5 tcp_event_data_sent
297 0.0159 vmlinux-2.6.21.5 try_to_del_timer_sync
296 0.0158 vmlinux-2.6.21.5 rt_hash_code
292 0.0156 vmlinux-2.6.21.5 tcp_new_space
290 0.0155 vmlinux-2.6.21.5 dummy_file_permission
286 0.0153 vmlinux-2.6.21.5 slab_get_obj
285 0.0152 vmlinux-2.6.21.5 check_spinlock_acquired
272 0.0145 vmlinux-2.6.21.5 zone_watermark_ok
268 0.0143 vmlinux-2.6.21.5 bio_alloc_bioset
267 0.0143 vmlinux-2.6.21.5 __inc_zone_state
262 0.0140 vmlinux-2.6.21.5 vsnprintf
258 0.0138 vmlinux-2.6.21.5 tg3_write32
256 0.0137 vmlinux-2.6.21.5 mempool_alloc
255 0.0136 vmlinux-2.6.21.5 __netdev_alloc_skb
251 0.0134 vmlinux-2.6.21.5 hrtimer_get_softirq_time
250 0.0134 vmlinux-2.6.21.5 tcp_should_expand_sndbuf
249 0.0133 httpd ap_content_length_filter
244 0.0130 vmlinux-2.6.21.5 do_wait
242 0.0129 vmlinux-2.6.21.5 prepare_to_wait
239 0.0128 vmlinux-2.6.21.5 sys_writev
237 0.0127 vmlinux-2.6.21.5 load_balance_newidle
234 0.0125 vmlinux-2.6.21.5 tcp_ack_update_window
232 0.0124 nfs (no symbols)
229 0.0122 vmlinux-2.6.21.5 release_pages
228 0.0122 vmlinux-2.6.21.5 __dec_zone_state
228 0.0122 vmlinux-2.6.21.5 apic_timer_interrupt
228 0.0122 vmlinux-2.6.21.5 xtSearch
226 0.0121 vmlinux-2.6.21.5 mpage_readpages
222 0.0119 vmlinux-2.6.21.5 mod_timer
221 0.0118 vmlinux-2.6.21.5 put_page
218 0.0116 httpd ap_pass_brigade
216 0.0115 vmlinux-2.6.21.5 dequeue_task
207 0.0111 vmlinux-2.6.21.5 tcp_build_and_update_options
206 0.0110 vmlinux-2.6.21.5 __bio_clone
203 0.0108 httpd buffer_output
200 0.0107 bash (no symbols)
200 0.0107 vmlinux-2.6.21.5 sock_poll
198 0.0106 vmlinux-2.6.21.5 __alloc_pages
194 0.0104 vmlinux-2.6.21.5 sd_init_command
194 0.0104 vmlinux-2.6.21.5 tcp_retransmit_skb
193 0.0103 vmlinux-2.6.21.5 memcpy
192 0.0103 vmlinux-2.6.21.5 __split_bio
191 0.0102 httpd swapu32
190 0.0101 vmlinux-2.6.21.5 __wake_up
189 0.0101 vmlinux-2.6.21.5 as_dispatch_request
189 0.0101 vmlinux-2.6.21.5 idle_balance
186 0.0099 vmlinux-2.6.21.5 as_completed_request
186 0.0099 vmlinux-2.6.21.5 tcp_ack_no_tstamp
184 0.0098 vmlinux-2.6.21.5 find_pid
182 0.0097 httpd apr_bucket_destroy_noop
180 0.0096 vmlinux-2.6.21.5 do_sock_write
178 0.0095 vmlinux-2.6.21.5 vfs_writev
177 0.0095 vmlinux-2.6.21.5 mempool_alloc_slab
177 0.0095 vmlinux-2.6.21.5 poll_freewait
176 0.0094 vmlinux-2.6.21.5 sys_times
175 0.0093 vmlinux-2.6.21.5 __sk_dst_check
175 0.0093 vmlinux-2.6.21.5 activate_task
175 0.0093 vmlinux-2.6.21.5 dev_hard_start_xmit
175 0.0093 vmlinux-2.6.21.5 inet_sendmsg
174 0.0093 httpd get_revent
172 0.0092 vmlinux-2.6.21.5 tcp_ack_packets_out
169 0.0090 vmlinux-2.6.21.5 __pagevec_release_nonlru
168 0.0090 vmlinux-2.6.21.5 nf_ip_checksum
167 0.0089 vmlinux-2.6.21.5 tcp_mark_head_lost
166 0.0089 vmlinux-2.6.21.5 scsi_decide_disposition
165 0.0088 vmlinux-2.6.21.5 vfs_read
163 0.0087 vmlinux-2.6.21.5 twa_post_command_packet
162 0.0087 vmlinux-2.6.21.5 kmem_getpages
161 0.0086 vmlinux-2.6.21.5 enqueue_task
161 0.0086 vmlinux-2.6.21.5 sk_reset_timer
159 0.0085 vmlinux-2.6.21.5 kmem_freepages
158 0.0084 httpd apr_brigade_destroy
157 0.0084 vmlinux-2.6.21.5 sys_poll
154 0.0082 vmlinux-2.6.21.5 do_task_stat
153 0.0082 httpd apr_brigade_cleanup
152 0.0081 vmlinux-2.6.21.5 tcp_xmit_retransmit_queue
149 0.0080 vmlinux-2.6.21.5 linear_map
148 0.0079 vmlinux-2.6.21.5 jfs_get_block
148 0.0079 vmlinux-2.6.21.5 max_io_len
147 0.0079 httpd handleRequest
147 0.0079 vmlinux-2.6.21.5 _atomic_dec_and_lock
147 0.0079 vmlinux-2.6.21.5 tcp_data_queue
146 0.0078 vmlinux-2.6.21.5 __make_request
146 0.0078 vmlinux-2.6.21.5 clone_endio
145 0.0077 httpd perform_idle_server_maintenance
145 0.0077 vmlinux-2.6.21.5 alloc_slabmgmt
145 0.0077 vmlinux-2.6.21.5 tg3_poll
144 0.0077 vmlinux-2.6.21.5 tcp_time_to_recover
144 0.0077 vmlinux-2.6.21.5 twa_map_scsi_sg_data
143 0.0076 vmlinux-2.6.21.5 release_sock
142 0.0076 httpd apr_pool_clear
142 0.0076 vmlinux-2.6.21.5 dm_table_find_target
142 0.0076 vmlinux-2.6.21.5 scsi_request_fn
139 0.0074 vmlinux-2.6.21.5 __wake_up_common
139 0.0074 vmlinux-2.6.21.5 jiffies_read
139 0.0074 vmlinux-2.6.21.5 tcp_try_to_open
138 0.0074 vmlinux-2.6.21.5 down_read
133 0.0071 vmlinux-2.6.21.5 bictcp_acked
131 0.0070 vmlinux-2.6.21.5 twa_scsiop_execute_scsi_complete
130 0.0069 vmlinux-2.6.21.5 ip_append_data
130 0.0069 vmlinux-2.6.21.5 scsi_dispatch_cmd
130 0.0069 vmlinux-2.6.21.5 sys_read
130 0.0069 vmlinux-2.6.21.5 tcp_rcv_state_process
129 0.0069 vmlinux-2.6.21.5 __remove_from_page_cache
129 0.0069 vmlinux-2.6.21.5 run_timer_softirq
128 0.0068 vmlinux-2.6.21.5 generic_file_aio_read
126 0.0067 vmlinux-2.6.21.5 as_update_iohist
123 0.0066 vmlinux-2.6.21.5 rw_copy_check_uvector
122 0.0065 vmlinux-2.6.21.5 page_waitqueue
122 0.0065 vmlinux-2.6.21.5 tcp_snd_test
120 0.0064 vmlinux-2.6.21.5 dm_table_put
119 0.0064 vmlinux-2.6.21.5 poll_initwait
118 0.0063 httpd apr_brigade_write
117 0.0062 vmlinux-2.6.21.5 generic_unplug_device
114 0.0061 vmlinux-2.6.21.5 __end_that_request_first
113 0.0060 httpd apr_vformatter
113 0.0060 vmlinux-2.6.21.5 lock_sock_nested
112 0.0060 httpd __divdi3
112 0.0060 httpd apr_pool_cleanup_kill
112 0.0060 vmlinux-2.6.21.5 softlockup_tick
112 0.0060 vmlinux-2.6.21.5 twa_check_bits
111 0.0059 vmlinux-2.6.21.5 cpu_idle
110 0.0059 vmlinux-2.6.21.5 as_queue_empty
109 0.0058 vmlinux-2.6.21.5 collect_sigign_sigcatch
108 0.0058 vmlinux-2.6.21.5 dm_unplug_all
108 0.0058 vmlinux-2.6.21.5 inotify_inode_queue_event
104 0.0056 vmlinux-2.6.21.5 try_atomic_semop
103 0.0055 vmlinux-2.6.21.5 hrtimer_run_queues
102 0.0054 vmlinux-2.6.21.5 radix_tree_preload
102 0.0054 vmlinux-2.6.21.5 tcp_fast_parse_options
100 0.0053 vmlinux-2.6.21.5 xtLookup
99 0.0053 vmlinux-2.6.21.5 groups_search
98 0.0052 vmlinux-2.6.21.5 dput
97 0.0052 vmlinux-2.6.21.5 __net_timestamp
97 0.0052 vmlinux-2.6.21.5 do_lookup
97 0.0052 vmlinux-2.6.21.5 profile_tick
97 0.0052 vmlinux-2.6.21.5 slab_destroy_objs
95 0.0051 vmlinux-2.6.21.5 generic_permission
94 0.0050 vmlinux-2.6.21.5 end_io_acct
94 0.0050 vmlinux-2.6.21.5 net_rx_action
94 0.0050 vmlinux-2.6.21.5 run_posix_cpu_timers
94 0.0050 vmlinux-2.6.21.5 touch_atime
93 0.0050 vmlinux-2.6.21.5 task_running_tick
92 0.0049 vmlinux-2.6.21.5 cubic_root
91 0.0049 vmlinux-2.6.21.5 remove_wait_queue
91 0.0049 vmlinux-2.6.21.5 tick_handle_periodic
90 0.0048 vmlinux-2.6.21.5 tcp_recvmsg
89 0.0048 ld-2.3.2.so (no symbols)
89 0.0048 vmlinux-2.6.21.5 bio_add_page
89 0.0048 vmlinux-2.6.21.5 elv_next_request
89 0.0048 vmlinux-2.6.21.5 idle_cpu
89 0.0048 vmlinux-2.6.21.5 scheduler_tick
89 0.0048 vmlinux-2.6.21.5 send_IPI_mask_bitmask
89 0.0048 vmlinux-2.6.21.5 stripe_map
89 0.0048 vmlinux-2.6.21.5 up_read
88 0.0047 vmlinux-2.6.21.5 add_timer_randomness
88 0.0047 vmlinux-2.6.21.5 strncpy_from_user
87 0.0046 vmlinux-2.6.21.5 udp_sendmsg
86 0.0046 vmlinux-2.6.21.5 deactivate_task
86 0.0046 vmlinux-2.6.21.5 finish_wait
85 0.0045 httpd get_event
85 0.0045 vmlinux-2.6.21.5 rw_verify_area
85 0.0045 vmlinux-2.6.21.5 tcp_parse_options
84 0.0045 vmlinux-2.6.21.5 __mutex_lock_slowpath
83 0.0044 vmlinux-2.6.21.5 account_system_time
83 0.0044 vmlinux-2.6.21.5 get_unused_fd
83 0.0044 vmlinux-2.6.21.5 tcp_urg
81 0.0043 httpd blowfishSetup
81 0.0043 vmlinux-2.6.21.5 __clone_and_map
81 0.0043 vmlinux-2.6.21.5 sock_aio_write
80 0.0043 vmlinux-2.6.21.5 dm_bio_destructor
80 0.0043 vmlinux-2.6.21.5 do_select
80 0.0043 vmlinux-2.6.21.5 fput
79 0.0042 httpd apr_bucket_free
79 0.0042 vmlinux-2.6.21.5 __do_softirq
79 0.0042 vmlinux-2.6.21.5 inotify_dentry_parent_queue_event
78 0.0042 httpd ap_directory_walk
78 0.0042 vmlinux-2.6.21.5 bio_endio
78 0.0042 vmlinux-2.6.21.5 permission
78 0.0042 vmlinux-2.6.21.5 twa_get_request_id
76 0.0041 vmlinux-2.6.21.5 as_close_req
76 0.0041 vmlinux-2.6.21.5 do_sync_read
75 0.0040 vmlinux-2.6.21.5 can_migrate_task
75 0.0040 vmlinux-2.6.21.5 inet_sk_rebuild_header
75 0.0040 vmlinux-2.6.21.5 sys_close
75 0.0040 vmlinux-2.6.21.5 tcp_cong_avoid
73 0.0039 vmlinux-2.6.21.5 ret_from_intr
72 0.0038 httpd status_handler
72 0.0038 vmlinux-2.6.21.5 __next_cpu
72 0.0038 vmlinux-2.6.21.5 dm_table_unplug_all
72 0.0038 vmlinux-2.6.21.5 put_io_context
71 0.0038 vmlinux-2.6.21.5 __block_prepare_write
70 0.0037 vmlinux-2.6.21.5 as_update_seekdist
70 0.0037 vmlinux-2.6.21.5 mark_page_accessed
70 0.0037 vmlinux-2.6.21.5 scsi_io_completion
69 0.0037 vmlinux-2.6.21.5 clone_bio
69 0.0037 vmlinux-2.6.21.5 scsi_softirq_done
69 0.0037 vmlinux-2.6.21.5 shrink_active_list
68 0.0036 httpd apr_palloc
68 0.0036 vmlinux-2.6.21.5 rcu_pending
68 0.0036 vmlinux-2.6.21.5 tcp_write_wakeup
67 0.0036 libproc.so.3.2.1 (no symbols)
67 0.0036 vmlinux-2.6.21.5 default_wake_function
67 0.0036 vmlinux-2.6.21.5 tcp_write_timer
66 0.0035 vmlinux-2.6.21.5 del_timer
65 0.0035 vmlinux-2.6.21.5 __get_metapage
65 0.0035 vmlinux-2.6.21.5 do_path_lookup
64 0.0034 httpd ap_rwrite
64 0.0034 vmlinux-2.6.21.5 datagram_poll
64 0.0034 vmlinux-2.6.21.5 elv_insert
64 0.0034 vmlinux-2.6.21.5 scsi_done
64 0.0034 vmlinux-2.6.21.5 worker_thread
63 0.0034 vmlinux-2.6.21.5 ext3_do_update_inode
62 0.0033 vmlinux-2.6.21.5 get_request
62 0.0033 vmlinux-2.6.21.5 update_process_times
61 0.0033 vmlinux-2.6.21.5 _write_lock_irqsave
61 0.0033 vmlinux-2.6.21.5 as_get_io_context
61 0.0033 vmlinux-2.6.21.5 blk_complete_request
61 0.0033 vmlinux-2.6.21.5 tick_periodic
60 0.0032 httpd ap_merge_per_dir_configs
60 0.0032 vmlinux-2.6.21.5 __generic_file_aio_write_nolock
60 0.0032 vmlinux-2.6.21.5 do_get_write_access
60 0.0032 vmlinux-2.6.21.5 sd_rw_intr
60 0.0032 vmlinux-2.6.21.5 udp_queue_rcv_skb
59 0.0032 httpd match
59 0.0032 vmlinux-2.6.21.5 as_can_break_anticipation
59 0.0032 vmlinux-2.6.21.5 slab_map_pages
59 0.0032 vmlinux-2.6.21.5 twa_scsi_queue
58 0.0031 vmlinux-2.6.21.5 __rcu_pending
58 0.0031 vmlinux-2.6.21.5 add_wait_queue
58 0.0031 vmlinux-2.6.21.5 queue_work
58 0.0031 vmlinux-2.6.21.5 slab_destroy
58 0.0031 vmlinux-2.6.21.5 tcp_retransmit_timer
57 0.0030 httpd blowfishDecrypt
57 0.0030 vmlinux-2.6.21.5 as_choose_req
57 0.0030 vmlinux-2.6.21.5 kref_get
57 0.0030 vmlinux-2.6.21.5 rcu_check_callbacks
57 0.0030 vmlinux-2.6.21.5 scsi_prep_fn
56 0.0030 httpd run_cleanups
55 0.0029 vmlinux-2.6.21.5 __udp4_lib_lookup
55 0.0029 vmlinux-2.6.21.5 debug_mutex_lock_common
54 0.0029 vmlinux-2.6.21.5 __mark_inode_dirty
54 0.0029 vmlinux-2.6.21.5 __sk_stream_mem_reclaim
54 0.0029 vmlinux-2.6.21.5 rb_erase
53 0.0028 httpd merge_core_dir_configs
53 0.0028 vmlinux-2.6.21.5 blockable_page_cache_readahead
53 0.0028 vmlinux-2.6.21.5 generic_file_buffered_write
53 0.0028 vmlinux-2.6.21.5 ip_push_pending_frames
53 0.0028 vmlinux-2.6.21.5 journal_add_journal_head
52 0.0028 httpd pcre_exec
52 0.0028 vmlinux-2.6.21.5 as_may_queue
52 0.0028 vmlinux-2.6.21.5 dm_table_get_devices
52 0.0028 vmlinux-2.6.21.5 page_referenced
52 0.0028 vmlinux-2.6.21.5 twa_free_request_id
51 0.0027 vmlinux-2.6.21.5 call_rcu
51 0.0027 vmlinux-2.6.21.5 fn_hash_lookup
50 0.0027 httpd apr_table_get
50 0.0027 vmlinux-2.6.21.5 __dentry_open
50 0.0027 vmlinux-2.6.21.5 as_put_io_context
50 0.0027 vmlinux-2.6.21.5 do_wp_page
50 0.0027 vmlinux-2.6.21.5 sk_stream_mem_schedule
50 0.0027 vmlinux-2.6.21.5 update_load
49 0.0026 vmlinux-2.6.21.5 kref_put
49 0.0026 vmlinux-2.6.21.5 scsi_device_unbusy
48 0.0026 vmlinux-2.6.21.5 __ip_route_output_key
48 0.0026 vmlinux-2.6.21.5 note_interrupt
48 0.0026 vmlinux-2.6.21.5 scsi_add_timer
48 0.0026 vmlinux-2.6.21.5 start_this_handle
48 0.0026 vmlinux-2.6.21.5 tcp_cwnd_down
48 0.0026 vmlinux-2.6.21.5 tcp_may_send_now
48 0.0026 vmlinux-2.6.21.5 udp_poll
47 0.0025 vmlinux-2.6.21.5 __inc_zone_page_state
47 0.0025 vmlinux-2.6.21.5 __udp4_lib_rcv
47 0.0025 vmlinux-2.6.21.5 as_add_request
47 0.0025 vmlinux-2.6.21.5 blk_remove_plug
47 0.0025 vmlinux-2.6.21.5 dtSearch
47 0.0025 vmlinux-2.6.21.5 kobject_get
47 0.0025 vmlinux-2.6.21.5 page_cache_readahead
47 0.0025 vmlinux-2.6.21.5 smp_apic_timer_interrupt
47 0.0025 vmlinux-2.6.21.5 submit_bio
46 0.0025 vmlinux-2.6.21.5 __fput
46 0.0025 vmlinux-2.6.21.5 as_remove_queued_request
46 0.0025 vmlinux-2.6.21.5 blk_run_queue
46 0.0025 vmlinux-2.6.21.5 debug_mutex_unlock
46 0.0025 vmlinux-2.6.21.5 init_timer
46 0.0025 vmlinux-2.6.21.5 may_open
46 0.0025 vmlinux-2.6.21.5 rb_next
46 0.0025 vmlinux-2.6.21.5 tasklet_action
45 0.0024 vmlinux-2.6.21.5 clear_bdi_congested
45 0.0024 vmlinux-2.6.21.5 journal_stop
45 0.0024 vmlinux-2.6.21.5 rb_insert_color
45 0.0024 vmlinux-2.6.21.5 run_local_timers
44 0.0024 vmlinux-2.6.21.5 as_fifo_expired
44 0.0024 vmlinux-2.6.21.5 dm_disk
44 0.0024 vmlinux-2.6.21.5 handle_level_irq
44 0.0024 vmlinux-2.6.21.5 tcp_write_timeout
43 0.0023 vmlinux-2.6.21.5 __queue_work
43 0.0023 vmlinux-2.6.21.5 blk_do_ordered
43 0.0023 vmlinux-2.6.21.5 exec_permission_lite
43 0.0023 vmlinux-2.6.21.5 journal_put_journal_head
43 0.0023 vmlinux-2.6.21.5 ordered_bio_endio
43 0.0023 vmlinux-2.6.21.5 rq_init
42 0.0022 httpd ap_update_child_status_from_indexes
42 0.0022 vmlinux-2.6.21.5 bio_fs_destructor
42 0.0022 vmlinux-2.6.21.5 get_empty_filp
42 0.0022 vmlinux-2.6.21.5 page_fault
42 0.0022 vmlinux-2.6.21.5 scsi_finish_command
42 0.0022 vmlinux-2.6.21.5 scsi_init_io
42 0.0022 vmlinux-2.6.21.5 static_prio_timeslice
42 0.0022 vmlinux-2.6.21.5 sys_socketcall
41 0.0022 httpd apr_bucket_transient_create
41 0.0022 httpd apr_sendv
41 0.0022 libcrypto.so.0.9.7 (no symbols)
41 0.0022 vmlinux-2.6.21.5 __pollwait
41 0.0022 vmlinux-2.6.21.5 __rcu_process_callbacks
41 0.0022 vmlinux-2.6.21.5 link_path_walk
41 0.0022 vmlinux-2.6.21.5 run_workqueue
41 0.0022 vmlinux-2.6.21.5 start_io_acct
40 0.0021 vmlinux-2.6.21.5 bio_get_nr_vecs
40 0.0021 vmlinux-2.6.21.5 blk_done_softirq
40 0.0021 vmlinux-2.6.21.5 eligible_child
40 0.0021 vmlinux-2.6.21.5 scsi_get_cmd_from_req
40 0.0021 vmlinux-2.6.21.5 zap_pte_range
39 0.0021 vmlinux-2.6.21.5 activate_page
39 0.0021 vmlinux-2.6.21.5 ext3_get_inode_block
39 0.0021 vmlinux-2.6.21.5 radix_tree_node_alloc
39 0.0021 vmlinux-2.6.21.5 raise_softirq
39 0.0021 vmlinux-2.6.21.5 scsi_end_request
38 0.0020 httpd listener_thread
38 0.0020 vmlinux-2.6.21.5 elv_dispatch_sort
38 0.0020 vmlinux-2.6.21.5 lookup_mnt
38 0.0020 vmlinux-2.6.21.5 update_wall_time
37 0.0020 vmlinux-2.6.21.5 __follow_mount
37 0.0020 vmlinux-2.6.21.5 __scsi_done
37 0.0020 vmlinux-2.6.21.5 as_update_rq
37 0.0020 vmlinux-2.6.21.5 del_timer_sync
37 0.0020 vmlinux-2.6.21.5 end_that_request_last
37 0.0020 vmlinux-2.6.21.5 ext3_permission
37 0.0020 vmlinux-2.6.21.5 journal_dirty_metadata
37 0.0020 vmlinux-2.6.21.5 resume_kernel
36 0.0019 vmlinux-2.6.21.5 do_no_page
36 0.0019 vmlinux-2.6.21.5 handle_IRQ_event
36 0.0019 vmlinux-2.6.21.5 jfs_readpages
36 0.0019 vmlinux-2.6.21.5 lookup_bh_lru
36 0.0019 vmlinux-2.6.21.5 mntput_no_expire
36 0.0019 vmlinux-2.6.21.5 mpage_alloc
36 0.0019 vmlinux-2.6.21.5 mutex_remove_waiter
36 0.0019 vmlinux-2.6.21.5 netif_rx
36 0.0019 vmlinux-2.6.21.5 percpu_counter_mod
36 0.0019 vmlinux-2.6.21.5 release_metapage
36 0.0019 vmlinux-2.6.21.5 scsi_get_command
36 0.0019 vmlinux-2.6.21.5 scsi_setup_fs_cmnd
36 0.0019 vmlinux-2.6.21.5 sigprocmask
35 0.0019 cronolog main
35 0.0019 libdns.so.11.1.3 (no symbols)
35 0.0019 vmlinux-2.6.21.5 core_sys_select
35 0.0019 vmlinux-2.6.21.5 shrink_inactive_list
35 0.0019 vmlinux-2.6.21.5 tcp_update_scoreboard
34 0.0018 httpd apr_bucket_alloc
34 0.0018 vmlinux-2.6.21.5 __free_pages
34 0.0018 vmlinux-2.6.21.5 __mutex_unlock_slowpath
34 0.0018 vmlinux-2.6.21.5 __pagevec_free
34 0.0018 vmlinux-2.6.21.5 in_group_p
34 0.0018 vmlinux-2.6.21.5 pid_revalidate
34 0.0018 vmlinux-2.6.21.5 radix_tree_node_ctor
34 0.0018 vmlinux-2.6.21.5 udp_push_pending_frames
33 0.0018 httpd apr_time_now
33 0.0018 ip_tables (no symbols)
33 0.0018 vmlinux-2.6.21.5 elv_rb_add
33 0.0018 vmlinux-2.6.21.5 scsi_init_cmd_errh
33 0.0018 vmlinux-2.6.21.5 scsi_log_completion
33 0.0018 vmlinux-2.6.21.5 sys_gettimeofday
32 0.0017 vmlinux-2.6.21.5 __generic_unplug_device
32 0.0017 vmlinux-2.6.21.5 as_move_to_dispatch
32 0.0017 vmlinux-2.6.21.5 dnotify_flush
32 0.0017 vmlinux-2.6.21.5 elv_queue_empty
32 0.0017 vmlinux-2.6.21.5 elv_rqhash_add
32 0.0017 vmlinux-2.6.21.5 irq_enter
31 0.0017 httpd ap_filter_flush
31 0.0017 vmlinux-2.6.21.5 as_update_thinktime
31 0.0017 vmlinux-2.6.21.5 ip_route_output_slow
31 0.0017 vmlinux-2.6.21.5 load_balance
31 0.0017 vmlinux-2.6.21.5 open_namei
30 0.0016 httpd add_any_filter_handle
30 0.0016 httpd apr_cpystrn
30 0.0016 httpd apr_table_setn
30 0.0016 vmlinux-2.6.21.5 init_request_from_bio
30 0.0016 vmlinux-2.6.21.5 memmove
30 0.0016 vmlinux-2.6.21.5 read_cache_page
30 0.0016 vmlinux-2.6.21.5 skb_copy_datagram_iovec
29 0.0015 httpd simple_bucket_read
29 0.0015 top (no symbols)
29 0.0015 vmlinux-2.6.21.5 __ext3_get_inode_loc
29 0.0015 vmlinux-2.6.21.5 do_sys_open
29 0.0015 vmlinux-2.6.21.5 find_busiest_queue
29 0.0015 vmlinux-2.6.21.5 pipe_read
29 0.0015 vmlinux-2.6.21.5 run_rebalance_domains
29 0.0015 vmlinux-2.6.21.5 scsi_log_send
28 0.0015 httpd ap_escape_logitem
28 0.0015 httpd apr_socket_opt_set
28 0.0015 httpd proc_mutex_sysv_acquire
28 0.0015 vmlinux-2.6.21.5 as_antic_stop
28 0.0015 vmlinux-2.6.21.5 as_find_next_rq
28 0.0015 vmlinux-2.6.21.5 current_io_context
28 0.0015 vmlinux-2.6.21.5 dm_table_get
28 0.0015 vmlinux-2.6.21.5 drain_array
28 0.0015 vmlinux-2.6.21.5 elv_completed_request
28 0.0015 vmlinux-2.6.21.5 futex_wake
28 0.0015 vmlinux-2.6.21.5 local_apic_timer_interrupt
28 0.0015 vmlinux-2.6.21.5 process_backlog
28 0.0015 vmlinux-2.6.21.5 sys_semtimedop
27 0.0014 httpd .plt
27 0.0014 httpd ap_getword_white
27 0.0014 vmlinux-2.6.21.5 __blk_put_request
27 0.0014 vmlinux-2.6.21.5 __lookup_mnt
27 0.0014 vmlinux-2.6.21.5 bictcp_recalc_ssthresh
27 0.0014 vmlinux-2.6.21.5 blk_queue_bounce
27 0.0014 vmlinux-2.6.21.5 ciCompare
27 0.0014 vmlinux-2.6.21.5 copy_from_dinode
27 0.0014 vmlinux-2.6.21.5 get_futex_key
27 0.0014 vmlinux-2.6.21.5 smp_reschedule_interrupt
26 0.0014 amd nfs_program_2
26 0.0014 httpd allocator_alloc
26 0.0014 httpd ap_parse_htaccess
26 0.0014 vmlinux-2.6.21.5 __handle_mm_fault
26 0.0014 vmlinux-2.6.21.5 __tasklet_schedule
26 0.0014 vmlinux-2.6.21.5 do_tcp_sendpages
26 0.0014 vmlinux-2.6.21.5 ext3_journal_start_sb
26 0.0014 vmlinux-2.6.21.5 find_lock_page
26 0.0014 vmlinux-2.6.21.5 journal_start
26 0.0014 vmlinux-2.6.21.5 pipe_write
26 0.0014 vmlinux-2.6.21.5 rcu_check_quiescent_state
26 0.0014 vmlinux-2.6.21.5 sockfd_lookup_light
26 0.0014 vmlinux-2.6.21.5 sys_open
26 0.0014 vmlinux-2.6.21.5 tcp_enter_cwr
25 0.0013 libncurses.so.5.4 (no symbols)
25 0.0013 vmlinux-2.6.21.5 __copy_user_intel_nocache
25 0.0013 vmlinux-2.6.21.5 __dec_zone_page_state
25 0.0013 vmlinux-2.6.21.5 d_alloc
25 0.0013 vmlinux-2.6.21.5 diRead
25 0.0013 vmlinux-2.6.21.5 get_device
25 0.0013 vmlinux-2.6.21.5 mmput
25 0.0013 vmlinux-2.6.21.5 recalc_sigpending_tsk
24 0.0013 httpd ap_read_request
24 0.0013 httpd apply_rewrite_rule
24 0.0013 httpd core_input_filter
24 0.0013 httpd proxy_detect
24 0.0013 vmlinux-2.6.21.5 autoremove_wake_function
24 0.0013 vmlinux-2.6.21.5 blk_alloc_request
24 0.0013 vmlinux-2.6.21.5 clocksource_adjust
24 0.0013 vmlinux-2.6.21.5 copy_to_user
24 0.0013 vmlinux-2.6.21.5 elv_rqhash_del
24 0.0013 vmlinux-2.6.21.5 fget
24 0.0013 vmlinux-2.6.21.5 find_next_zero_bit
24 0.0013 vmlinux-2.6.21.5 irq_exit
24 0.0013 vmlinux-2.6.21.5 path_release
24 0.0013 vmlinux-2.6.21.5 shrink_slab
23 0.0012 vmlinux-2.6.21.5 __scsi_get_command
23 0.0012 vmlinux-2.6.21.5 add_disk_randomness
23 0.0012 vmlinux-2.6.21.5 as_antic_expired
23 0.0012 vmlinux-2.6.21.5 bictcp_state
23 0.0012 vmlinux-2.6.21.5 debug_mutex_free_waiter
23 0.0012 vmlinux-2.6.21.5 do_IRQ
23 0.0012 vmlinux-2.6.21.5 do_softirq
23 0.0012 vmlinux-2.6.21.5 double_lock_balance
23 0.0012 vmlinux-2.6.21.5 fib_get_table
23 0.0012 vmlinux-2.6.21.5 ip_route_input_slow
23 0.0012 vmlinux-2.6.21.5 jiffies_to_clock_t
23 0.0012 vmlinux-2.6.21.5 udp_recvmsg
22 0.0012 httpd ap_location_walk
22 0.0012 httpd ap_process_request_internal
22 0.0012 libpthread-0.60.so read
22 0.0012 vmlinux-2.6.21.5 __freed_request
22 0.0012 vmlinux-2.6.21.5 as_activate_request
22 0.0012 vmlinux-2.6.21.5 bio_alloc
22 0.0012 vmlinux-2.6.21.5 bio_phys_segments
22 0.0012 vmlinux-2.6.21.5 ext3_prepare_write
22 0.0012 vmlinux-2.6.21.5 freed_request
22 0.0012 vmlinux-2.6.21.5 inet_csk_search_req
22 0.0012 vmlinux-2.6.21.5 inet_lookup_listener_slow
22 0.0012 vmlinux-2.6.21.5 ip_route_output_flow
22 0.0012 vmlinux-2.6.21.5 poll_get_entry
22 0.0012 vmlinux-2.6.21.5 setfl
22 0.0012 vmlinux-2.6.21.5 sys_ipc
22 0.0012 vmlinux-2.6.21.5 tcp_slow_start
21 0.0011 httpd apr_filepath_merge
21 0.0011 httpd apr_pool_create_ex
21 0.0011 httpd hook_uri2file
21 0.0011 vmlinux-2.6.21.5 __release_sock
21 0.0011 vmlinux-2.6.21.5 cache_reap
21 0.0011 vmlinux-2.6.21.5 current_fs_time
21 0.0011 vmlinux-2.6.21.5 do_mmap_pgoff
21 0.0011 vmlinux-2.6.21.5 ip_build_and_send_pkt
21 0.0011 vmlinux-2.6.21.5 mempool_free_slab
21 0.0011 vmlinux-2.6.21.5 nameidata_to_filp
21 0.0011 vmlinux-2.6.21.5 sock_alloc_send_pskb
21 0.0011 vmlinux-2.6.21.5 tcp_cleanup_rbuf
21 0.0011 vmlinux-2.6.21.5 tcp_make_synack
21 0.0011 vmlinux-2.6.21.5 wake_up_bit
20 0.0011 amd run_rpc
20 0.0011 httpd ap_rgetline_core
20 0.0011 libpthread-0.60.so __pthread_disable_asynccancel
20 0.0011 vmlinux-2.6.21.5 __elv_add_request
20 0.0011 vmlinux-2.6.21.5 __scsi_put_command
20 0.0011 vmlinux-2.6.21.5 do_filp_open
20 0.0011 vmlinux-2.6.21.5 filp_close
20 0.0011 vmlinux-2.6.21.5 free_pages
20 0.0011 vmlinux-2.6.21.5 scsi_put_command
20 0.0011 vmlinux-2.6.21.5 scsi_release_buffers
20 0.0011 vmlinux-2.6.21.5 tcp_close
20 0.0011 vmlinux-2.6.21.5 xfrm_policy_put_afinfo
19 0.0010 httpd apr_socket_recv
19 0.0010 libpthread-0.60.so __open_nocancel
19 0.0010 libpthread-0.60.so accept
19 0.0010 libpthread-0.60.so pthread_mutex_unlock
19 0.0010 vmlinux-2.6.21.5 __kmalloc
19 0.0010 vmlinux-2.6.21.5 d_instantiate
19 0.0010 vmlinux-2.6.21.5 do_futex
19 0.0010 vmlinux-2.6.21.5 dtCompare
19 0.0010 vmlinux-2.6.21.5 dummy_inode_permission
19 0.0010 vmlinux-2.6.21.5 fib_semantic_match
19 0.0010 vmlinux-2.6.21.5 jfs_lookup
19 0.0010 vmlinux-2.6.21.5 page_add_file_rmap
19 0.0010 vmlinux-2.6.21.5 rcu_process_callbacks
19 0.0010 vmlinux-2.6.21.5 scsi_run_queue
19 0.0010 vmlinux-2.6.21.5 sock_common_recvmsg
18 9.6e-04 amd do_select
18 9.6e-04 httpd allocator_free
18 9.6e-04 httpd ap_file_walk
18 9.6e-04 httpd ap_recent_rfc822_date
18 9.6e-04 httpd apr_file_write
18 9.6e-04 httpd regexec
18 9.6e-04 httpd worker_thread
18 9.6e-04 vmlinux-2.6.21.5 __ext3_journal_get_write_access
18 9.6e-04 vmlinux-2.6.21.5 __netif_rx_schedule
18 9.6e-04 vmlinux-2.6.21.5 bio_free
18 9.6e-04 vmlinux-2.6.21.5 bio_put
18 9.6e-04 vmlinux-2.6.21.5 bit_waitqueue
18 9.6e-04 vmlinux-2.6.21.5 copy_io_context
18 9.6e-04 vmlinux-2.6.21.5 debug_mutex_add_waiter
18 9.6e-04 vmlinux-2.6.21.5 elv_rb_del
18 9.6e-04 vmlinux-2.6.21.5 get_pid_task
18 9.6e-04 vmlinux-2.6.21.5 get_request_wait
18 9.6e-04 vmlinux-2.6.21.5 ip_generic_getfrag
18 9.6e-04 vmlinux-2.6.21.5 journal_dirty_data
18 9.6e-04 vmlinux-2.6.21.5 raise_softirq_irqoff
18 9.6e-04 vmlinux-2.6.21.5 rcu_do_batch
18 9.6e-04 vmlinux-2.6.21.5 read_pages
18 9.6e-04 vmlinux-2.6.21.5 sk_clone
18 9.6e-04 vmlinux-2.6.21.5 sys_futex
18 9.6e-04 vmlinux-2.6.21.5 sys_select
18 9.6e-04 vmlinux-2.6.21.5 sys_write
18 9.6e-04 vmlinux-2.6.21.5 tcp_ack_saw_tstamp
17 9.1e-04 httpd ap_discard_request_body
17 9.1e-04 httpd ap_get_mime_headers_core
17 9.1e-04 httpd apr_pool_cleanup_register
17 9.1e-04 httpd free_proc_chain
17 9.1e-04 httpd heap_bucket_destroy
17 9.1e-04 vmlinux-2.6.21.5 __path_lookup_intent_open
17 9.1e-04 vmlinux-2.6.21.5 alloc_sock_iocb
17 9.1e-04 vmlinux-2.6.21.5 common_interrupt
17 9.1e-04 vmlinux-2.6.21.5 double_rq_lock
17 9.1e-04 vmlinux-2.6.21.5 ext3_file_write
17 9.1e-04 vmlinux-2.6.21.5 ext3_ordered_commit_write
17 9.1e-04 vmlinux-2.6.21.5 futex_wait
17 9.1e-04 vmlinux-2.6.21.5 iput
17 9.1e-04 vmlinux-2.6.21.5 locks_remove_flock
17 9.1e-04 vmlinux-2.6.21.5 module_put
17 9.1e-04 vmlinux-2.6.21.5 profile_pc
17 9.1e-04 vmlinux-2.6.21.5 rb_prev
17 9.1e-04 vmlinux-2.6.21.5 recalc_sigpending
17 9.1e-04 vmlinux-2.6.21.5 restore_all
17 9.1e-04 vmlinux-2.6.21.5 sys_accept
17 9.1e-04 vmlinux-2.6.21.5 tcp_unhash
16 8.5e-04 httpd ap_byterange_filter
16 8.5e-04 httpd ap_process_http_connection
16 8.5e-04 httpd apr_array_push_noclear
16 8.5e-04 httpd apr_brigade_writev
16 8.5e-04 httpd cached_explode
16 8.5e-04 httpd check_dir_access
16 8.5e-04 httpd conv_10
16 8.5e-04 httpd find_ct
16 8.5e-04 libpthread-0.60.so __i686.get_pc_thunk.bx
16 8.5e-04 vmlinux-2.6.21.5 __inet_lookup_listener
16 8.5e-04 vmlinux-2.6.21.5 __journal_file_buffer
16 8.5e-04 vmlinux-2.6.21.5 _spin_lock_irqsave
16 8.5e-04 vmlinux-2.6.21.5 clocksource_get_next
16 8.5e-04 vmlinux-2.6.21.5 copy_pte_range
16 8.5e-04 vmlinux-2.6.21.5 dentry_iput
16 8.5e-04 vmlinux-2.6.21.5 down_write
16 8.5e-04 vmlinux-2.6.21.5 filemap_nopage
16 8.5e-04 vmlinux-2.6.21.5 find_extend_vma
16 8.5e-04 vmlinux-2.6.21.5 generic_file_aio_write
16 8.5e-04 vmlinux-2.6.21.5 generic_pipe_buf_pin
16 8.5e-04 vmlinux-2.6.21.5 getname
16 8.5e-04 vmlinux-2.6.21.5 jfs_permission
16 8.5e-04 vmlinux-2.6.21.5 journal_get_write_access
16 8.5e-04 vmlinux-2.6.21.5 kthread_should_stop
16 8.5e-04 vmlinux-2.6.21.5 loopback_xmit
16 8.5e-04 vmlinux-2.6.21.5 msecs_to_jiffies
16 8.5e-04 vmlinux-2.6.21.5 proc_fill_cache
16 8.5e-04 vmlinux-2.6.21.5 rt_intern_hash
16 8.5e-04 vmlinux-2.6.21.5 sock_def_wakeup
16 8.5e-04 vmlinux-2.6.21.5 sock_sendmsg
16 8.5e-04 vmlinux-2.6.21.5 tcp_push_one
16 8.5e-04 vmlinux-2.6.21.5 timespec_trunc
16 8.5e-04 vmlinux-2.6.21.5 wake_bit_function
15 8.0e-04 httpd ap_add_input_filter_handle
15 8.0e-04 httpd ap_get_remote_host
15 8.0e-04 httpd ap_getword_conf
15 8.0e-04 httpd ap_lingering_close
15 8.0e-04 httpd ap_unescape_url
15 8.0e-04 httpd apr_base64_decode_binary
15 8.0e-04 httpd apr_file_open
15 8.0e-04 httpd apr_snprintf
15 8.0e-04 httpd apr_table_set
15 8.0e-04 httpd net_time_filter
15 8.0e-04 httpd process_socket
15 8.0e-04 httpd ssl_hook_pre_connection
15 8.0e-04 vmlinux-2.6.21.5 __rb_erase_color
15 8.0e-04 vmlinux-2.6.21.5 __spin_lock_init
15 8.0e-04 vmlinux-2.6.21.5 do_page_fault
15 8.0e-04 vmlinux-2.6.21.5 elv_merge
15 8.0e-04 vmlinux-2.6.21.5 ext3_writepage_trans_blocks
15 8.0e-04 vmlinux-2.6.21.5 hweight32
15 8.0e-04 vmlinux-2.6.21.5 inet_csk_accept
15 8.0e-04 vmlinux-2.6.21.5 inode_init_once
15 8.0e-04 vmlinux-2.6.21.5 jfs_check_acl
15 8.0e-04 vmlinux-2.6.21.5 kobject_put
15 8.0e-04 vmlinux-2.6.21.5 locks_remove_posix
15 8.0e-04 vmlinux-2.6.21.5 make_ahead_window
15 8.0e-04 vmlinux-2.6.21.5 put_pid
15 8.0e-04 vmlinux-2.6.21.5 sock_def_write_space
15 8.0e-04 vmlinux-2.6.21.5 sock_from_file
15 8.0e-04 vmlinux-2.6.21.5 sys_recvmsg
15 8.0e-04 vmlinux-2.6.21.5 tcp_check_req
15 8.0e-04 vmlinux-2.6.21.5 vfs_write
15 8.0e-04 vmlinux-2.6.21.5 vma_adjust
14 7.5e-04 httpd ap_http_header_filter
14 7.5e-04 httpd ap_queue_pop
14 7.5e-04 httpd apr_brigade_create
14 7.5e-04 httpd apr_bucket_flush_create
14 7.5e-04 httpd apr_off_t_toa
14 7.5e-04 httpd apr_table_make
14 7.5e-04 libnetsnmp.so.5.1.2 (no symbols)
14 7.5e-04 vmlinux-2.6.21.5 __copy_from_user_ll_nozero
14 7.5e-04 vmlinux-2.6.21.5 __user_walk_fd
14 7.5e-04 vmlinux-2.6.21.5 __wait_on_bit_lock
14 7.5e-04 vmlinux-2.6.21.5 bio_init
14 7.5e-04 vmlinux-2.6.21.5 clear_inode
14 7.5e-04 vmlinux-2.6.21.5 delayed_work_timer_fn
14 7.5e-04 vmlinux-2.6.21.5 dupfd
14 7.5e-04 vmlinux-2.6.21.5 generic_forget_inode
14 7.5e-04 vmlinux-2.6.21.5 get_wchan
14 7.5e-04 vmlinux-2.6.21.5 jfs_release
14 7.5e-04 vmlinux-2.6.21.5 kmap_high
14 7.5e-04 vmlinux-2.6.21.5 metapage_readpage
14 7.5e-04 vmlinux-2.6.21.5 sock_queue_rcv_skb
14 7.5e-04 vmlinux-2.6.21.5 tcp_v4_conn_request
14 7.5e-04 vmlinux-2.6.21.5 vfs_getattr
14 7.5e-04 vmlinux-2.6.21.5 walk_page_buffers
13 6.9e-04 httpd add_any_filter
13 6.9e-04 httpd ap_run_handler
13 6.9e-04 httpd ap_update_child_status
13 6.9e-04 httpd basic_http_header_check
13 6.9e-04 httpd get_offset
13 6.9e-04 httpd make_array_core
13 6.9e-04 libpthread-0.60.so pthread_cond_wait@@GLIBC_2.3.2
13 6.9e-04 libpthread-0.60.so pthread_mutex_lock
13 6.9e-04 squid (no symbols)
13 6.9e-04 vmlinux-2.6.21.5 __lock_page
13 6.9e-04 vmlinux-2.6.21.5 __netif_schedule
13 6.9e-04 vmlinux-2.6.21.5 _local_bh_enable
13 6.9e-04 vmlinux-2.6.21.5 ea_get
13 6.9e-04 vmlinux-2.6.21.5 elv_iosched_allow_merge
13 6.9e-04 vmlinux-2.6.21.5 elv_rqhash_find
13 6.9e-04 vmlinux-2.6.21.5 expand_files
13 6.9e-04 vmlinux-2.6.21.5 ext3_follow_link
13 6.9e-04 vmlinux-2.6.21.5 file_move
13 6.9e-04 vmlinux-2.6.21.5 inet_sock_destruct
13 6.9e-04 vmlinux-2.6.21.5 init_waitqueue_head
13 6.9e-04 vmlinux-2.6.21.5 journal_cancel_revoke
13 6.9e-04 vmlinux-2.6.21.5 kmap
13 6.9e-04 vmlinux-2.6.21.5 net_tx_action
13 6.9e-04 vmlinux-2.6.21.5 page_remove_rmap
13 6.9e-04 vmlinux-2.6.21.5 pid_task
13 6.9e-04 vmlinux-2.6.21.5 pipe_wait
13 6.9e-04 vmlinux-2.6.21.5 tcp_close_state
13 6.9e-04 vmlinux-2.6.21.5 tcp_send_probe0
13 6.9e-04 vmlinux-2.6.21.5 tcp_time_wait
13 6.9e-04 vmlinux-2.6.21.5 tcp_v4_destroy_sock
13 6.9e-04 vmlinux-2.6.21.5 tcp_v4_hnd_req
13 6.9e-04 vmlinux-2.6.21.5 unmap_vmas
12 6.4e-04 amd fh_to_mp3
12 6.4e-04 httpd ap_http_filter
12 6.4e-04 httpd apply_rewrite_list
12 6.4e-04 httpd apr_pool_destroy
12 6.4e-04 httpd apr_pstrcat
12 6.4e-04 httpd apr_stat
12 6.4e-04 httpd apr_thread_mutex_lock
12 6.4e-04 httpd heap_bucket_read
12 6.4e-04 httpd match_headers
12 6.4e-04 libpthread-0.60.so __errno_location
12 6.4e-04 libpthread-0.60.so __i686.get_pc_thunk.cx
12 6.4e-04 libpthread-0.60.so _pthread_cleanup_push_defer
12 6.4e-04 libpthread-0.60.so fcntl
12 6.4e-04 vmlinux-2.6.21.5 __find_get_block
12 6.4e-04 vmlinux-2.6.21.5 __get_user_4
12 6.4e-04 vmlinux-2.6.21.5 __getblk
12 6.4e-04 vmlinux-2.6.21.5 __journal_temp_unlink_buffer
12 6.4e-04 vmlinux-2.6.21.5 balance_dirty_pages_ratelimited_nr
12 6.4e-04 vmlinux-2.6.21.5 blk_plug_device
12 6.4e-04 vmlinux-2.6.21.5 block_sync_page
12 6.4e-04 vmlinux-2.6.21.5 call_rcu_bh
12 6.4e-04 vmlinux-2.6.21.5 cond_resched_softirq
12 6.4e-04 vmlinux-2.6.21.5 cp_new_stat64
12 6.4e-04 vmlinux-2.6.21.5 d_splice_alias
12 6.4e-04 vmlinux-2.6.21.5 do_sync_write
12 6.4e-04 vmlinux-2.6.21.5 fib_validate_source
12 6.4e-04 vmlinux-2.6.21.5 generic_commit_write
12 6.4e-04 vmlinux-2.6.21.5 generic_fillattr
12 6.4e-04 vmlinux-2.6.21.5 get_new_inode_fast
12 6.4e-04 vmlinux-2.6.21.5 get_nr_files
12 6.4e-04 vmlinux-2.6.21.5 in_sched_functions
12 6.4e-04 vmlinux-2.6.21.5 inet_csk_destroy_sock
12 6.4e-04 vmlinux-2.6.21.5 inet_csk_route_req
12 6.4e-04 vmlinux-2.6.21.5 inet_release
12 6.4e-04 vmlinux-2.6.21.5 iov_fault_in_pages_write
12 6.4e-04 vmlinux-2.6.21.5 kmem_cache_zalloc
12 6.4e-04 vmlinux-2.6.21.5 scsi_alloc_sgtable
12 6.4e-04 vmlinux-2.6.21.5 should_remove_suid
12 6.4e-04 vmlinux-2.6.21.5 skb_dequeue
12 6.4e-04 vmlinux-2.6.21.5 sock_alloc_send_skb
12 6.4e-04 vmlinux-2.6.21.5 sock_def_readable
12 6.4e-04 vmlinux-2.6.21.5 sync_page
12 6.4e-04 vmlinux-2.6.21.5 sys_time
12 6.4e-04 vmlinux-2.6.21.5 tcp_create_openreq_child
12 6.4e-04 vmlinux-2.6.21.5 tcp_rcv_space_adjust
12 6.4e-04 vmlinux-2.6.21.5 tcp_try_undo_dsack
12 6.4e-04 vmlinux-2.6.21.5 vfs_stat_fd
11 5.9e-04 amd nfsproc_getattr_2_svc
11 5.9e-04 httpd ap_headers_insert_output_filter
11 5.9e-04 httpd ap_invoke_handler
11 5.9e-04 httpd ap_make_full_path
11 5.9e-04 httpd ap_pcfg_openfile
11 5.9e-04 httpd ap_process_request
11 5.9e-04 httpd apr_brigade_split_line
11 5.9e-04 httpd apr_bucket_shared_split
11 5.9e-04 httpd apr_socket_accept
11 5.9e-04 httpd config_log_transaction
11 5.9e-04 httpd core_create_conn
11 5.9e-04 httpd multi_log_transaction
11 5.9e-04 httpd prep_walk_cache
11 5.9e-04 httpd table_mergesort
11 5.9e-04 httpd translate_alias_redir
11 5.9e-04 libpthread-0.60.so __pthread_enable_asynccancel
11 5.9e-04 vmlinux-2.6.21.5 __jfs_getxattr
11 5.9e-04 vmlinux-2.6.21.5 __log_space_left
11 5.9e-04 vmlinux-2.6.21.5 __map_bio
11 5.9e-04 vmlinux-2.6.21.5 __put_unused_fd
11 5.9e-04 vmlinux-2.6.21.5 as_can_anticipate
11 5.9e-04 vmlinux-2.6.21.5 ext3_getblk
11 5.9e-04 vmlinux-2.6.21.5 ext3_new_blocks
11 5.9e-04 vmlinux-2.6.21.5 file_kill
11 5.9e-04 vmlinux-2.6.21.5 file_update_time
11 5.9e-04 vmlinux-2.6.21.5 free_hot_page
11 5.9e-04 vmlinux-2.6.21.5 ipc_lock
11 5.9e-04 vmlinux-2.6.21.5 memcpy_toiovec
11 5.9e-04 vmlinux-2.6.21.5 metapage_read_end_io
11 5.9e-04 vmlinux-2.6.21.5 put_device
11 5.9e-04 vmlinux-2.6.21.5 task_dumpable
11 5.9e-04 vmlinux-2.6.21.5 tcp_clear_retrans
11 5.9e-04 vmlinux-2.6.21.5 udp_rcv
11 5.9e-04 vmlinux-2.6.21.5 vfs_permission
10 5.3e-04 commoncap (no symbols)
10 5.3e-04 httpd ap_get_brigade
10 5.3e-04 httpd ap_make_method_list
10 5.3e-04 httpd ap_no2slash
10 5.3e-04 httpd ap_queue_push
10 5.3e-04 httpd ap_run_fixups
10 5.3e-04 httpd ap_run_process_connection
10 5.3e-04 httpd ap_run_quick_handler
10 5.3e-04 httpd ap_save_brigade
10 5.3e-04 httpd apr_pstrdup
10 5.3e-04 httpd apr_pvsprintf
10 5.3e-04 httpd apr_socket_sendfile
10 5.3e-04 httpd apr_socket_timeout_set
10 5.3e-04 httpd apr_table_copy
10 5.3e-04 httpd apr_uri_parse
10 5.3e-04 httpd find_allowdeny
10 5.3e-04 httpd find_entry
10 5.3e-04 httpd fix_hostname
10 5.3e-04 httpd process_item
10 5.3e-04 httpd ssl_hook_Access
10 5.3e-04 httpd terminate_header
10 5.3e-04 httpd translate_userdir
10 5.3e-04 vmlinux-2.6.21.5 __ext3_journal_dirty_metadata
10 5.3e-04 vmlinux-2.6.21.5 __mutex_init
10 5.3e-04 vmlinux-2.6.21.5 arch_get_unmapped_area_topdown
10 5.3e-04 vmlinux-2.6.21.5 balance_pgdat
10 5.3e-04 vmlinux-2.6.21.5 bget_one
10 5.3e-04 vmlinux-2.6.21.5 blk_unplug_timeout
10 5.3e-04 vmlinux-2.6.21.5 change_clocksource
10 5.3e-04 vmlinux-2.6.21.5 do_sock_read
10 5.3e-04 vmlinux-2.6.21.5 fd_install
10 5.3e-04 vmlinux-2.6.21.5 find_get_pages
10 5.3e-04 vmlinux-2.6.21.5 find_vma_prepare
10 5.3e-04 vmlinux-2.6.21.5 io_schedule
10 5.3e-04 vmlinux-2.6.21.5 path_lookup_open
10 5.3e-04 vmlinux-2.6.21.5 pipe_iov_copy_from_user
10 5.3e-04 vmlinux-2.6.21.5 prune_dcache
10 5.3e-04 vmlinux-2.6.21.5 put_cmsg
10 5.3e-04 vmlinux-2.6.21.5 put_filp
10 5.3e-04 vmlinux-2.6.21.5 real_lookup
10 5.3e-04 vmlinux-2.6.21.5 sock_alloc_inode
10 5.3e-04 vmlinux-2.6.21.5 tcp_cwnd_restart
10 5.3e-04 vmlinux-2.6.21.5 tcp_send_fin
10 5.3e-04 vmlinux-2.6.21.5 unmap_page_range
9 4.8e-04 amd clocktime
9 4.8e-04 amd xdr_fattr
9 4.8e-04 httpd ap_default_log_writer
9 4.8e-04 httpd ap_get_request_note
9 4.8e-04 httpd ap_invoke_filter_init
9 4.8e-04 httpd ap_satisfies
9 4.8e-04 httpd apr_base64_decode_len
9 4.8e-04 httpd apr_bucket_shared_destroy
9 4.8e-04 httpd apr_filepath_root
9 4.8e-04 httpd apr_os_sock_get
9 4.8e-04 httpd do_expand
9 4.8e-04 httpd fixup_dir
9 4.8e-04 httpd handle_audio_request
9 4.8e-04 httpd lookup_builtin_method
9 4.8e-04 httpd merge_env_dir_configs
9 4.8e-04 httpd sendfile_it_all
9 4.8e-04 httpd substring_conf
9 4.8e-04 libpthread-0.60.so __read_nocancel
9 4.8e-04 libpthread-0.60.so open
9 4.8e-04 vmlinux-2.6.21.5 __try_to_free_cp_buf
9 4.8e-04 vmlinux-2.6.21.5 alloc_inode
9 4.8e-04 vmlinux-2.6.21.5 as_add_rq_rb
9 4.8e-04 vmlinux-2.6.21.5 blk_backing_dev_unplug
9 4.8e-04 vmlinux-2.6.21.5 current_tick_length
9 4.8e-04 vmlinux-2.6.21.5 debug_mutex_init
9 4.8e-04 vmlinux-2.6.21.5 destroy_inode
9 4.8e-04 vmlinux-2.6.21.5 do_getname
9 4.8e-04 vmlinux-2.6.21.5 file_ra_state_init
9 4.8e-04 vmlinux-2.6.21.5 fn_hash_select_default
9 4.8e-04 vmlinux-2.6.21.5 get_dirty_limits
9 4.8e-04 vmlinux-2.6.21.5 get_vmalloc_info
9 4.8e-04 vmlinux-2.6.21.5 hash_futex
9 4.8e-04 vmlinux-2.6.21.5 inet_shutdown
9 4.8e-04 vmlinux-2.6.21.5 invalidate_inode_buffers
9 4.8e-04 vmlinux-2.6.21.5 jfs_read_inode
9 4.8e-04 vmlinux-2.6.21.5 journal_blocks_per_page
9 4.8e-04 vmlinux-2.6.21.5 mpage_bio_submit
9 4.8e-04 vmlinux-2.6.21.5 new_inode
9 4.8e-04 vmlinux-2.6.21.5 proc_pid_readdir
9 4.8e-04 vmlinux-2.6.21.5 queue_delayed_work
9 4.8e-04 vmlinux-2.6.21.5 radix_tree_tag_set
9 4.8e-04 vmlinux-2.6.21.5 sock_recvmsg
9 4.8e-04 vmlinux-2.6.21.5 split_vma
9 4.8e-04 vmlinux-2.6.21.5 sys_dup
9 4.8e-04 vmlinux-2.6.21.5 tcp_grow_window
9 4.8e-04 vmlinux-2.6.21.5 tcp_sendpage
9 4.8e-04 vmlinux-2.6.21.5 tcp_v4_syn_recv_sock
9 4.8e-04 vmlinux-2.6.21.5 udp4_hwcsum_outgoing
9 4.8e-04 vmlinux-2.6.21.5 unmap_region
9 4.8e-04 xt_tcpudp (no symbols)
8 4.3e-04 amd autofs_handle_fdset
8 4.3e-04 amd get_next_exported_ap
8 4.3e-04 amd xdr_nfs_fh
8 4.3e-04 apache (no symbols)
8 4.3e-04 httpd __moddi3
8 4.3e-04 httpd ap_run_default_port
8 4.3e-04 httpd ap_set_keepalive
8 4.3e-04 httpd apr_array_make
8 4.3e-04 httpd apr_psprintf
8 4.3e-04 httpd blockDecryptCBC
8 4.3e-04 httpd core_create_req
8 4.3e-04 httpd find_default_server
8 4.3e-04 httpd hook_fixup
8 4.3e-04 httpd log_request_time
8 4.3e-04 httpd proc_mutex_sysv_release
8 4.3e-04 httpd proxy_handler
8 4.3e-04 httpd proxy_trans
8 4.3e-04 httpd removeNewlines
8 4.3e-04 httpd ssl_hook_ReadReq
8 4.3e-04 httpd unixd_accept
8 4.3e-04 libnetsnmpagent.so.5.1.2 (no symbols)
8 4.3e-04 libpthread-0.60.so close
8 4.3e-04 syslog-ng (no symbols)
8 4.3e-04 vmlinux-2.6.21.5 bictcp_init
8 4.3e-04 vmlinux-2.6.21.5 blk_unplug_work
8 4.3e-04 vmlinux-2.6.21.5 d_lookup
8 4.3e-04 vmlinux-2.6.21.5 do_tcp_setsockopt
8 4.3e-04 vmlinux-2.6.21.5 ext3_dirty_inode
8 4.3e-04 vmlinux-2.6.21.5 ext3_mark_inode_dirty
8 4.3e-04 vmlinux-2.6.21.5 ext3_release_file
8 4.3e-04 vmlinux-2.6.21.5 ext3_try_to_allocate_with_rsv
8 4.3e-04 vmlinux-2.6.21.5 in_lock_functions
8 4.3e-04 vmlinux-2.6.21.5 init_once
8 4.3e-04 vmlinux-2.6.21.5 ip_mc_drop_socket
8 4.3e-04 vmlinux-2.6.21.5 jfs_open
8 4.3e-04 vmlinux-2.6.21.5 jfs_set_inode_flags
8 4.3e-04 vmlinux-2.6.21.5 journal_commit_transaction
8 4.3e-04 vmlinux-2.6.21.5 neigh_hh_init
8 4.3e-04 vmlinux-2.6.21.5 neigh_lookup
8 4.3e-04 vmlinux-2.6.21.5 new_handle
8 4.3e-04 vmlinux-2.6.21.5 next_tgid
8 4.3e-04 vmlinux-2.6.21.5 no_singlestep
8 4.3e-04 vmlinux-2.6.21.5 rb_first
8 4.3e-04 vmlinux-2.6.21.5 scsi_next_command
8 4.3e-04 vmlinux-2.6.21.5 secure_tcp_sequence_number
8 4.3e-04 vmlinux-2.6.21.5 shrink_dcache_memory
8 4.3e-04 vmlinux-2.6.21.5 skb_recv_datagram
8 4.3e-04 vmlinux-2.6.21.5 smp_send_reschedule
8 4.3e-04 vmlinux-2.6.21.5 sock_fasync
8 4.3e-04 vmlinux-2.6.21.5 sync_sb_inodes
8 4.3e-04 vmlinux-2.6.21.5 sys_brk
8 4.3e-04 vmlinux-2.6.21.5 sys_munmap
8 4.3e-04 vmlinux-2.6.21.5 sys_stat64
8 4.3e-04 vmlinux-2.6.21.5 tcp_add_reno_sack
8 4.3e-04 vmlinux-2.6.21.5 tcp_check_reno_reordering
8 4.3e-04 vmlinux-2.6.21.5 tcp_check_sack_reneging
8 4.3e-04 vmlinux-2.6.21.5 tcp_event_data_recv
8 4.3e-04 vmlinux-2.6.21.5 tcp_probe_timer
8 4.3e-04 vmlinux-2.6.21.5 tcp_try_undo_loss
8 4.3e-04 vmlinux-2.6.21.5 tcp_try_undo_partial
8 4.3e-04 vmlinux-2.6.21.5 tcp_try_undo_recovery
8 4.3e-04 vmlinux-2.6.21.5 unlock_buffer
8 4.3e-04 vmlinux-2.6.21.5 vm_acct_memory
8 4.3e-04 vmlinux-2.6.21.5 wake_up_inode
7 3.7e-04 amd autofs_add_fdset
7 3.7e-04 amd checkup
7 3.7e-04 amd new_ttl
7 3.7e-04 httpd ap_core_translate
7 3.7e-04 httpd ap_index_of_response
7 3.7e-04 httpd ap_run_log_transaction
7 3.7e-04 httpd ap_some_auth_required
7 3.7e-04 httpd ap_update_vhost_from_headers
7 3.7e-04 httpd apr_array_append
7 3.7e-04 httpd apr_table_compress
7 3.7e-04 httpd apr_table_elts
7 3.7e-04 httpd apr_table_mergen
7 3.7e-04 httpd apr_time_ansi_put
7 3.7e-04 httpd apr_unix_mode2perms
7 3.7e-04 httpd check_pipeline_flush
7 3.7e-04 httpd display_info
7 3.7e-04 httpd fill_out_finfo
7 3.7e-04 httpd fixup_redir
7 3.7e-04 httpd include_fixup
7 3.7e-04 httpd read_request_line
7 3.7e-04 httpd try_alias_list
7 3.7e-04 libpthread-0.60.so __lll_mutex_unlock_wake
7 3.7e-04 libpthread-0.60.so write
7 3.7e-04 vmlinux-2.6.21.5 __copy_from_user_ll_nocache_nozero
7 3.7e-04 vmlinux-2.6.21.5 __inet_twsk_hashdance
7 3.7e-04 vmlinux-2.6.21.5 block_prepare_write
7 3.7e-04 vmlinux-2.6.21.5 do_fcntl
7 3.7e-04 vmlinux-2.6.21.5 do_munmap
7 3.7e-04 vmlinux-2.6.21.5 dummy_inode_alloc_security
7 3.7e-04 vmlinux-2.6.21.5 elv_put_request
7 3.7e-04 vmlinux-2.6.21.5 elv_rb_find
7 3.7e-04 vmlinux-2.6.21.5 ext3_reserve_inode_write
7 3.7e-04 vmlinux-2.6.21.5 find_ge_pid
7 3.7e-04 vmlinux-2.6.21.5 find_get_pages_tag
7 3.7e-04 vmlinux-2.6.21.5 find_inode_fast
7 3.7e-04 vmlinux-2.6.21.5 find_undo
7 3.7e-04 vmlinux-2.6.21.5 flush_tlb_page
7 3.7e-04 vmlinux-2.6.21.5 generic_drop_inode
7 3.7e-04 vmlinux-2.6.21.5 get_UCSname
7 3.7e-04 vmlinux-2.6.21.5 half_md4_transform
7 3.7e-04 vmlinux-2.6.21.5 inet_twsk_schedule
7 3.7e-04 vmlinux-2.6.21.5 iov_fault_in_pages_read
7 3.7e-04 vmlinux-2.6.21.5 is_bad_inode
7 3.7e-04 vmlinux-2.6.21.5 jfs_alloc_inode
7 3.7e-04 vmlinux-2.6.21.5 mb_cache_shrink_fn
7 3.7e-04 vmlinux-2.6.21.5 move_addr_to_kernel
7 3.7e-04 vmlinux-2.6.21.5 proc_info_read
7 3.7e-04 vmlinux-2.6.21.5 pull_task
7 3.7e-04 vmlinux-2.6.21.5 shrink_zone
7 3.7e-04 vmlinux-2.6.21.5 skb_queue_tail
7 3.7e-04 vmlinux-2.6.21.5 sock_rfree
7 3.7e-04 vmlinux-2.6.21.5 tcp_cleanup_congestion_control
7 3.7e-04 vmlinux-2.6.21.5 tcp_init_congestion_control
7 3.7e-04 vmlinux-2.6.21.5 tcp_measure_rcv_mss
7 3.7e-04 vmlinux-2.6.21.5 update_queue
7 3.7e-04 vmlinux-2.6.21.5 wake_up_process
6 3.2e-04 httpd analyze_ct
6 3.2e-04 httpd ap_bucket_eoc_create
6 3.2e-04 httpd ap_explode_recent_localtime
6 3.2e-04 httpd ap_find_token
6 3.2e-04 httpd ap_get_server_name
6 3.2e-04 httpd ap_get_server_version
6 3.2e-04 httpd ap_getparents
6 3.2e-04 httpd ap_queue_info_set_idle
6 3.2e-04 httpd ap_queue_info_wait_for_idler
6 3.2e-04 httpd ap_rationalize_mtime
6 3.2e-04 httpd ap_run_create_request
6 3.2e-04 httpd ap_run_type_checker
6 3.2e-04 httpd ap_send_http_trace
6 3.2e-04 httpd apply_rewrite_cond
6 3.2e-04 httpd apr_bucket_alloc_create
6 3.2e-04 httpd apr_bucket_eos_create
6 3.2e-04 httpd apr_poll_revents_get
6 3.2e-04 httpd apr_proc_mutex_unlock
6 3.2e-04 httpd apr_table_vdo
6 3.2e-04 httpd constant_item
6 3.2e-04 httpd default_handler
6 3.2e-04 httpd expires_insert_filter
6 3.2e-04 httpd extractParam
6 3.2e-04 httpd fixup_vary
6 3.2e-04 httpd imap_handler
6 3.2e-04 httpd inet_ntop4
6 3.2e-04 httpd logDRMAccess
6 3.2e-04 httpd proxy_fixup
6 3.2e-04 libpthread-0.60.so __pthread_mutex_unlock_usercnt
6 3.2e-04 libpthread-0.60.so pthread_cond_signal@@GLIBC_2.3.2
6 3.2e-04 named (no symbols)
6 3.2e-04 vmlinux-2.6.21.5 __init_rwsem
6 3.2e-04 vmlinux-2.6.21.5 __vm_enough_memory
6 3.2e-04 vmlinux-2.6.21.5 cascade
6 3.2e-04 vmlinux-2.6.21.5 clear_page_dirty_for_io
6 3.2e-04 vmlinux-2.6.21.5 double_rq_unlock
6 3.2e-04 vmlinux-2.6.21.5 down_read_trylock
6 3.2e-04 vmlinux-2.6.21.5 dummy_inode_getattr
6 3.2e-04 vmlinux-2.6.21.5 elv_set_request
6 3.2e-04 vmlinux-2.6.21.5 ext3_get_blocks_handle
6 3.2e-04 vmlinux-2.6.21.5 ext3_get_branch
6 3.2e-04 vmlinux-2.6.21.5 ext3_get_inode_loc
6 3.2e-04 vmlinux-2.6.21.5 fib_rule_match
6 3.2e-04 vmlinux-2.6.21.5 file_free_rcu
6 3.2e-04 vmlinux-2.6.21.5 filldir64
6 3.2e-04 vmlinux-2.6.21.5 free_pgtables
6 3.2e-04 vmlinux-2.6.21.5 generic_file_open
6 3.2e-04 vmlinux-2.6.21.5 generic_writepages
6 3.2e-04 vmlinux-2.6.21.5 get_unmapped_area
6 3.2e-04 vmlinux-2.6.21.5 inet_accept
6 3.2e-04 vmlinux-2.6.21.5 inet_csk_clone
6 3.2e-04 vmlinux-2.6.21.5 journal_submit_data_buffers
6 3.2e-04 vmlinux-2.6.21.5 kblockd_schedule_work
6 3.2e-04 vmlinux-2.6.21.5 kernel_sendmsg
6 3.2e-04 vmlinux-2.6.21.5 lru_cache_add_active
6 3.2e-04 vmlinux-2.6.21.5 max_select_fd
6 3.2e-04 vmlinux-2.6.21.5 need_resched
6 3.2e-04 vmlinux-2.6.21.5 opost_block
6 3.2e-04 vmlinux-2.6.21.5 out_of_line_wait_on_bit
6 3.2e-04 vmlinux-2.6.21.5 page_add_new_anon_rmap
6 3.2e-04 vmlinux-2.6.21.5 prepare_to_wait_exclusive
6 3.2e-04 vmlinux-2.6.21.5 proc_pid_fill_cache
6 3.2e-04 vmlinux-2.6.21.5 schedule_delayed_work
6 3.2e-04 vmlinux-2.6.21.5 sk_alloc
6 3.2e-04 vmlinux-2.6.21.5 sk_free
6 3.2e-04 vmlinux-2.6.21.5 sk_stop_timer
6 3.2e-04 vmlinux-2.6.21.5 sock_common_setsockopt
6 3.2e-04 vmlinux-2.6.21.5 sock_release
6 3.2e-04 vmlinux-2.6.21.5 sys_mmap2
6 3.2e-04 vmlinux-2.6.21.5 sys_sendmsg
6 3.2e-04 vmlinux-2.6.21.5 sys_setsockopt
6 3.2e-04 vmlinux-2.6.21.5 tcp_fin
6 3.2e-04 vmlinux-2.6.21.5 tcp_init_cwnd
6 3.2e-04 vmlinux-2.6.21.5 tcp_setsockopt
6 3.2e-04 vmlinux-2.6.21.5 tcp_shutdown
6 3.2e-04 vmlinux-2.6.21.5 tcp_update_metrics
6 3.2e-04 vmlinux-2.6.21.5 tcp_v4_send_synack
6 3.2e-04 vmlinux-2.6.21.5 unlink_file_vma
6 3.2e-04 vmlinux-2.6.21.5 unqueue_me
6 3.2e-04 vmlinux-2.6.21.5 verify_iovec
6 3.2e-04 vmlinux-2.6.21.5 vm_normal_page
5 2.7e-04 amd amu_svc_getcaller
5 2.7e-04 httpd action_handler
5 2.7e-04 httpd ap_create_sb_handle
5 2.7e-04 httpd ap_regexec
5 2.7e-04 httpd ap_remove_output_filter
5 2.7e-04 httpd ap_rputc
5 2.7e-04 httpd ap_run_access_checker
5 2.7e-04 httpd ap_run_insert_filter
5 2.7e-04 httpd apr_allocator_free
5 2.7e-04 httpd apr_brigade_length
5 2.7e-04 httpd apr_bucket_eos_make
5 2.7e-04 httpd apr_bucket_heap_make
5 2.7e-04 httpd apr_bucket_simple_split
5 2.7e-04 httpd apr_pool_cleanup_run
5 2.7e-04 httpd apr_shutdown
5 2.7e-04 httpd apr_sockaddr_ip_get
5 2.7e-04 httpd apr_sockaddr_vars_set
5 2.7e-04 httpd apr_thread_cond_wait
5 2.7e-04 httpd apr_thread_mutex_unlock
5 2.7e-04 httpd conv_10_quad
5 2.7e-04 httpd file_bucket_destroy
5 2.7e-04 httpd find_poll_sock
5 2.7e-04 httpd maint_handler
5 2.7e-04 httpd rewritelog
5 2.7e-04 httpd socket_bucket_read
5 2.7e-04 httpd ssl_hook_default_port
5 2.7e-04 httpd table_reindex
5 2.7e-04 libpthread-0.60.so _pthread_cleanup_pop_restore
5 2.7e-04 ntpd (no symbols)
5 2.7e-04 vmlinux-2.6.21.5 __block_commit_write
5 2.7e-04 vmlinux-2.6.21.5 __block_write_full_page
5 2.7e-04 vmlinux-2.6.21.5 __d_rehash
5 2.7e-04 vmlinux-2.6.21.5 _spin_lock_bh
5 2.7e-04 vmlinux-2.6.21.5 block_write_full_page
5 2.7e-04 vmlinux-2.6.21.5 check_userspace
5 2.7e-04 vmlinux-2.6.21.5 create_empty_buffers
5 2.7e-04 vmlinux-2.6.21.5 d_rehash
5 2.7e-04 vmlinux-2.6.21.5 end_bio_bh_io_sync
5 2.7e-04 vmlinux-2.6.21.5 ext3_get_block
5 2.7e-04 vmlinux-2.6.21.5 fib_rules_lookup
5 2.7e-04 vmlinux-2.6.21.5 get_task_comm
5 2.7e-04 vmlinux-2.6.21.5 inet_csk_reqsk_queue_prune
5 2.7e-04 vmlinux-2.6.21.5 inet_select_addr
5 2.7e-04 vmlinux-2.6.21.5 inet_twsk_alloc
5 2.7e-04 vmlinux-2.6.21.5 inotify_d_instantiate
5 2.7e-04 vmlinux-2.6.21.5 ip_cmsg_recv_pktinfo
5 2.7e-04 vmlinux-2.6.21.5 ip_options_echo
5 2.7e-04 vmlinux-2.6.21.5 jfs_strtoUCS
5 2.7e-04 vmlinux-2.6.21.5 journal_get_undo_access
5 2.7e-04 vmlinux-2.6.21.5 journal_grab_journal_head
5 2.7e-04 vmlinux-2.6.21.5 kill_fasync
5 2.7e-04 vmlinux-2.6.21.5 locate_fd
5 2.7e-04 vmlinux-2.6.21.5 lock_task_sighand
5 2.7e-04 vmlinux-2.6.21.5 lru_cache_add
5 2.7e-04 vmlinux-2.6.21.5 metapage_get_blocks
5 2.7e-04 vmlinux-2.6.21.5 move_addr_to_user
5 2.7e-04 vmlinux-2.6.21.5 mpage_readpage
5 2.7e-04 vmlinux-2.6.21.5 neigh_resolve_output
5 2.7e-04 vmlinux-2.6.21.5 proc_pid_statm
5 2.7e-04 vmlinux-2.6.21.5 proc_readdir
5 2.7e-04 vmlinux-2.6.21.5 sk_stream_kill_queues
5 2.7e-04 vmlinux-2.6.21.5 sk_stream_wait_close
5 2.7e-04 vmlinux-2.6.21.5 sys_rt_sigprocmask
5 2.7e-04 vmlinux-2.6.21.5 tcp_mtup_init
5 2.7e-04 vmlinux-2.6.21.5 tcp_send_ack
4 2.1e-04 amd softclock
4 2.1e-04 amd xdr_nfsstat
4 2.1e-04 httpd ap_add_output_filter_handle
4 2.1e-04 httpd ap_add_output_filters_by_type
4 2.1e-04 httpd ap_get_scoreboard_worker
4 2.1e-04 httpd ap_meets_conditions
4 2.1e-04 httpd ap_method_number_of
4 2.1e-04 httpd ap_old_write_filter
4 2.1e-04 httpd ap_run_create_connection
4 2.1e-04 httpd ap_run_map_to_storage
4 2.1e-04 httpd ap_set_content_length
4 2.1e-04 httpd ap_str_tolower
4 2.1e-04 httpd apr_allocator_alloc
4 2.1e-04 httpd apr_array_copy_hdr
4 2.1e-04 httpd apr_inet_ntop
4 2.1e-04 httpd apr_itoa
4 2.1e-04 httpd apr_proc_mutex_lock
4 2.1e-04 httpd apr_rfc822_date
4 2.1e-04 httpd apr_time_exp_gmt
4 2.1e-04 httpd apr_unix_file_cleanup
4 2.1e-04 httpd apr_unix_perms2mode
4 2.1e-04 httpd brigade_cleanup
4 2.1e-04 httpd core_map_to_storage
4 2.1e-04 httpd decrypt
4 2.1e-04 httpd filetype_from_mode
4 2.1e-04 httpd find_closing_bracket
4 2.1e-04 httpd getClientIP
4 2.1e-04 httpd handle_autoindex
4 2.1e-04 httpd hook_mimetype
4 2.1e-04 httpd processV4Query
4 2.1e-04 libpthread-0.60.so __pthread_mutex_cond_lock
4 2.1e-04 libpthread-0.60.so __write_nocancel
4 2.1e-04 mawk (no symbols)
4 2.1e-04 sshd (no symbols)
4 2.1e-04 vmlinux-2.6.21.5 __brelse
4 2.1e-04 vmlinux-2.6.21.5 __ext3_journal_stop
4 2.1e-04 vmlinux-2.6.21.5 __find_get_block_slow
4 2.1e-04 vmlinux-2.6.21.5 __inet_twsk_kill
4 2.1e-04 vmlinux-2.6.21.5 __lookup_hash
4 2.1e-04 vmlinux-2.6.21.5 __lru_add_drain
4 2.1e-04 vmlinux-2.6.21.5 __rwlock_init
4 2.1e-04 vmlinux-2.6.21.5 __tcp_ack_snd_check
4 2.1e-04 vmlinux-2.6.21.5 anon_vma_unlink
4 2.1e-04 vmlinux-2.6.21.5 can_vma_merge_before
4 2.1e-04 vmlinux-2.6.21.5 copy_process
4 2.1e-04 vmlinux-2.6.21.5 detach_vmas_to_be_unmapped
4 2.1e-04 vmlinux-2.6.21.5 diIAGRead
4 2.1e-04 vmlinux-2.6.21.5 dispose_list
4 2.1e-04 vmlinux-2.6.21.5 do_anonymous_page
4 2.1e-04 vmlinux-2.6.21.5 do_sendfile
4 2.1e-04 vmlinux-2.6.21.5 do_timer
4 2.1e-04 vmlinux-2.6.21.5 drive_stat_acct
4 2.1e-04 vmlinux-2.6.21.5 drop_buffers
4 2.1e-04 vmlinux-2.6.21.5 ea_release
4 2.1e-04 vmlinux-2.6.21.5 elv_dequeue_request
4 2.1e-04 vmlinux-2.6.21.5 ext3_bmap
4 2.1e-04 vmlinux-2.6.21.5 ext3_get_group_desc
4 2.1e-04 vmlinux-2.6.21.5 flush_tlb_mm
4 2.1e-04 vmlinux-2.6.21.5 inet_csk_reqsk_queue_hash_add
4 2.1e-04 vmlinux-2.6.21.5 inet_getname
4 2.1e-04 vmlinux-2.6.21.5 ip_cmsg_send
4 2.1e-04 vmlinux-2.6.21.5 ipcperms
4 2.1e-04 vmlinux-2.6.21.5 jfs_readpage
4 2.1e-04 vmlinux-2.6.21.5 journal_clean_one_cp_list
4 2.1e-04 vmlinux-2.6.21.5 last_read_complete
4 2.1e-04 vmlinux-2.6.21.5 load_elf_binary
4 2.1e-04 vmlinux-2.6.21.5 lookup_hash
4 2.1e-04 vmlinux-2.6.21.5 lru_add_drain
4 2.1e-04 vmlinux-2.6.21.5 mutex_unlock
4 2.1e-04 vmlinux-2.6.21.5 n_tty_receive_buf
4 2.1e-04 vmlinux-2.6.21.5 page_put_link
4 2.1e-04 vmlinux-2.6.21.5 pipe_iov_copy_to_user
4 2.1e-04 vmlinux-2.6.21.5 prune_icache
4 2.1e-04 vmlinux-2.6.21.5 put_pages_list
4 2.1e-04 vmlinux-2.6.21.5 radix_tree_tag_clear
4 2.1e-04 vmlinux-2.6.21.5 read_chan
4 2.1e-04 vmlinux-2.6.21.5 remove_vma_list
4 2.1e-04 vmlinux-2.6.21.5 ret_from_exception
4 2.1e-04 vmlinux-2.6.21.5 sk_stream_rfree
4 2.1e-04 vmlinux-2.6.21.5 sock_aio_read
4 2.1e-04 vmlinux-2.6.21.5 sock_destroy_inode
4 2.1e-04 vmlinux-2.6.21.5 sock_wmalloc
4 2.1e-04 vmlinux-2.6.21.5 sys_fcntl64
4 2.1e-04 vmlinux-2.6.21.5 sys_sendfile
4 2.1e-04 vmlinux-2.6.21.5 syscall_call
4 2.1e-04 vmlinux-2.6.21.5 tcp_ack_probe
4 2.1e-04 vmlinux-2.6.21.5 tcp_child_process
4 2.1e-04 vmlinux-2.6.21.5 tcp_fixup_sndbuf
4 2.1e-04 vmlinux-2.6.21.5 tcp_init_buffer_space
4 2.1e-04 vmlinux-2.6.21.5 tcp_mtu_to_mss
4 2.1e-04 vmlinux-2.6.21.5 tcp_select_initial_window
4 2.1e-04 vmlinux-2.6.21.5 tcp_syn_build_options
4 2.1e-04 vmlinux-2.6.21.5 tcp_v4_save_options
4 2.1e-04 vmlinux-2.6.21.5 tg3_alloc_rx_skb
4 2.1e-04 vmlinux-2.6.21.5 unmap_mapping_range
4 2.1e-04 vmlinux-2.6.21.5 vfs_fstat
4 2.1e-04 vmlinux-2.6.21.5 vfs_stat
4 2.1e-04 vmlinux-2.6.21.5 vm_stat_account
3 1.6e-04 amd .plt
3 1.6e-04 amd xdr_attrstat
3 1.6e-04 amd xmalloc
3 1.6e-04 httpd alloc_cleanup
3 1.6e-04 httpd ap_add_output_filter
3 1.6e-04 httpd ap_explode_recent_gmt
3 1.6e-04 httpd ap_getword
3 1.6e-04 httpd ap_log_cerror
3 1.6e-04 httpd ap_make_dirstr_parent
3 1.6e-04 httpd ap_parse_uri
3 1.6e-04 httpd ap_process_connection
3 1.6e-04 httpd ap_run_post_read_request
3 1.6e-04 httpd ap_set_content_type
3 1.6e-04 httpd apr_accept
3 1.6e-04 httpd apr_allocator_mutex_get
3 1.6e-04 httpd apr_allocator_owner_get
3 1.6e-04 httpd apr_bucket_file_create
3 1.6e-04 httpd apr_bucket_heap_create
3 1.6e-04 httpd apr_bucket_shared_make
3 1.6e-04 httpd apr_parse_addr_port
3 1.6e-04 httpd apr_socket_addr_get
3 1.6e-04 httpd apr_table_do
3 1.6e-04 httpd apr_table_unset
3 1.6e-04 httpd apr_thread_cond_signal
3 1.6e-04 httpd apr_time_exp_lt
3 1.6e-04 httpd checkAccess
3 1.6e-04 httpd core_insert_filter
3 1.6e-04 httpd core_opts_merge
3 1.6e-04 httpd core_pre_connection
3 1.6e-04 httpd create_empty_config
3 1.6e-04 httpd decryptBlowfish
3 1.6e-04 httpd etag_ulong_to_hex
3 1.6e-04 httpd explode_time
3 1.6e-04 httpd fix_encoding
3 1.6e-04 httpd fixup_env_module
3 1.6e-04 httpd handle_multi
3 1.6e-04 httpd http_create_request
3 1.6e-04 httpd log_cookie
3 1.6e-04 httpd log_error_core
3 1.6e-04 httpd log_status
3 1.6e-04 httpd lookup_variable
3 1.6e-04 httpd remove_any_filter
3 1.6e-04 httpd set_socket_vars
3 1.6e-04 httpd sononblock
3 1.6e-04 httpd x2c
3 1.6e-04 libpthread-0.60.so __close_nocancel
3 1.6e-04 libresolv-2.3.2.so (no symbols)
3 1.6e-04 snmpd (no symbols)
3 1.6e-04 vmlinux-2.6.21.5 __anon_vma_link
3 1.6e-04 vmlinux-2.6.21.5 __anon_vma_merge
3 1.6e-04 vmlinux-2.6.21.5 __free_pages_ok
3 1.6e-04 vmlinux-2.6.21.5 __inet_put_port
3 1.6e-04 vmlinux-2.6.21.5 __lookup
3 1.6e-04 vmlinux-2.6.21.5 __pagevec_lru_add_active
3 1.6e-04 vmlinux-2.6.21.5 __rb_rotate_right
3 1.6e-04 vmlinux-2.6.21.5 __sync_single_inode
3 1.6e-04 vmlinux-2.6.21.5 __wait_on_bit
3 1.6e-04 vmlinux-2.6.21.5 add_to_page_cache_lru
3 1.6e-04 vmlinux-2.6.21.5 arch_unmap_area_topdown
3 1.6e-04 vmlinux-2.6.21.5 arp_hash
3 1.6e-04 vmlinux-2.6.21.5 arp_process
3 1.6e-04 vmlinux-2.6.21.5 as_antic_waitnext
3 1.6e-04 vmlinux-2.6.21.5 as_merge
3 1.6e-04 vmlinux-2.6.21.5 bh_lru_install
3 1.6e-04 vmlinux-2.6.21.5 block_read_full_page
3 1.6e-04 vmlinux-2.6.21.5 can_vma_merge_after
3 1.6e-04 vmlinux-2.6.21.5 copy_page_range
3 1.6e-04 vmlinux-2.6.21.5 copy_strings
3 1.6e-04 vmlinux-2.6.21.5 csum_partial_copy_generic
3 1.6e-04 vmlinux-2.6.21.5 d_free
3 1.6e-04 vmlinux-2.6.21.5 dummy_ipc_permission
3 1.6e-04 vmlinux-2.6.21.5 end_buffer_async_write
3 1.6e-04 vmlinux-2.6.21.5 ext3_blks_to_allocate
3 1.6e-04 vmlinux-2.6.21.5 ext3_find_entry
3 1.6e-04 vmlinux-2.6.21.5 ext3_journal_dirty_data
3 1.6e-04 vmlinux-2.6.21.5 ext3_mkdir
3 1.6e-04 vmlinux-2.6.21.5 ext3_ordered_writepage
3 1.6e-04 vmlinux-2.6.21.5 ext3_setattr
3 1.6e-04 vmlinux-2.6.21.5 ext3_test_allocatable
3 1.6e-04 vmlinux-2.6.21.5 ext3_try_to_allocate
3 1.6e-04 vmlinux-2.6.21.5 file_send_actor
3 1.6e-04 vmlinux-2.6.21.5 find_vma_prev
3 1.6e-04 vmlinux-2.6.21.5 free_one_page
3 1.6e-04 vmlinux-2.6.21.5 free_pgd_range
3 1.6e-04 vmlinux-2.6.21.5 generic_file_sendfile
3 1.6e-04 vmlinux-2.6.21.5 get_io_context
3 1.6e-04 vmlinux-2.6.21.5 get_write_access
3 1.6e-04 vmlinux-2.6.21.5 hash
3 1.6e-04 vmlinux-2.6.21.5 hrtimer_try_to_cancel
3 1.6e-04 vmlinux-2.6.21.5 inet_bind_bucket_destroy
3 1.6e-04 vmlinux-2.6.21.5 inet_csk_clear_xmit_timers
3 1.6e-04 vmlinux-2.6.21.5 inet_put_port
3 1.6e-04 vmlinux-2.6.21.5 init_once
3 1.6e-04 vmlinux-2.6.21.5 inode_has_buffers
3 1.6e-04 vmlinux-2.6.21.5 ipv4_negative_advice
3 1.6e-04 vmlinux-2.6.21.5 jfs_destroy_inode
3 1.6e-04 vmlinux-2.6.21.5 jfs_get_acl
3 1.6e-04 vmlinux-2.6.21.5 jiffies_64_to_clock_t
3 1.6e-04 vmlinux-2.6.21.5 journal_write_metadata_buffer
3 1.6e-04 vmlinux-2.6.21.5 kunmap
3 1.6e-04 vmlinux-2.6.21.5 load_elf_interp
3 1.6e-04 vmlinux-2.6.21.5 memcpy_fromiovecend
3 1.6e-04 vmlinux-2.6.21.5 nr_blockdev_pages
3 1.6e-04 vmlinux-2.6.21.5 pipe_poll
3 1.6e-04 vmlinux-2.6.21.5 preempt_schedule_irq
3 1.6e-04 vmlinux-2.6.21.5 proc_root_readdir
3 1.6e-04 vmlinux-2.6.21.5 proc_tgid_stat
3 1.6e-04 vmlinux-2.6.21.5 rcu_start_batch
3 1.6e-04 vmlinux-2.6.21.5 refresh_cpu_vm_stats
3 1.6e-04 vmlinux-2.6.21.5 remove_suid
3 1.6e-04 vmlinux-2.6.21.5 requeue_task
3 1.6e-04 vmlinux-2.6.21.5 scsi_delete_timer
3 1.6e-04 vmlinux-2.6.21.5 sock_alloc
3 1.6e-04 vmlinux-2.6.21.5 sprintf
3 1.6e-04 vmlinux-2.6.21.5 sys_fstat64
3 1.6e-04 vmlinux-2.6.21.5 sys_shutdown
3 1.6e-04 vmlinux-2.6.21.5 task_vsize
3 1.6e-04 vmlinux-2.6.21.5 tcp_init_metrics
3 1.6e-04 vmlinux-2.6.21.5 tcp_initialize_rcv_mss
3 1.6e-04 vmlinux-2.6.21.5 tcp_mss_to_mtu
3 1.6e-04 vmlinux-2.6.21.5 tcp_sync_mss
3 1.6e-04 vmlinux-2.6.21.5 tcp_timewait_state_process
3 1.6e-04 vmlinux-2.6.21.5 test_set_page_writeback
3 1.6e-04 vmlinux-2.6.21.5 tg3_write_mem
3 1.6e-04 vmlinux-2.6.21.5 up_write
3 1.6e-04 vmlinux-2.6.21.5 vma_merge
2 1.1e-04 amd path_to_exported_ap
2 1.1e-04 amd xdr_nfstime
2 1.1e-04 grep (no symbols)
2 1.1e-04 httpd alloc_socket
2 1.1e-04 httpd ap_finalize_request_protocol
2 1.1e-04 httpd ap_flush_conn
2 1.1e-04 httpd ap_make_content_type
2 1.1e-04 httpd ap_make_etag
2 1.1e-04 httpd ap_run_header_parser
2 1.1e-04 httpd ap_run_translate_name
2 1.1e-04 httpd ap_set_byterange
2 1.1e-04 httpd ap_set_etag
2 1.1e-04 httpd ap_set_last_modified
2 1.1e-04 httpd ap_time_process_request
2 1.1e-04 httpd ap_update_mtime
2 1.1e-04 httpd ap_update_vhost_given_ip
2 1.1e-04 httpd apr_array_push
2 1.1e-04 httpd apr_bucket_flush_make
2 1.1e-04 httpd apr_bucket_transient_make
2 1.1e-04 httpd apr_hash_get
2 1.1e-04 httpd apr_is_empty_table
2 1.1e-04 httpd apr_pool_allocator_get
2 1.1e-04 httpd apr_pool_tag
2 1.1e-04 httpd apr_pstrmemdup
2 1.1e-04 httpd apr_recv
2 1.1e-04 httpd apr_sendfile
2 1.1e-04 httpd apr_socket_close
2 1.1e-04 httpd apr_time_exp_tz
2 1.1e-04 httpd calcTimePlayed
2 1.1e-04 httpd cgid_handler
2 1.1e-04 httpd clf_log_bytes_sent
2 1.1e-04 httpd do_nothing
2 1.1e-04 httpd end_output_stream
2 1.1e-04 httpd form_header_field
2 1.1e-04 httpd handler_redirect
2 1.1e-04 httpd http_method
2 1.1e-04 httpd http_port
2 1.1e-04 httpd loadBlowfishKey
2 1.1e-04 httpd logMessage
2 1.1e-04 httpd log_remote_user
2 1.1e-04 httpd proxy_map_location
2 1.1e-04 httpd send_all_header_fields
2 1.1e-04 httpd socket_cleanup
2 1.1e-04 httpd ssl_hook_Fixup
2 1.1e-04 httpd toLong
2 1.1e-04 httpd validate_status_line
2 1.1e-04 libpthread-0.60.so __i686.get_pc_thunk.dx
2 1.1e-04 nrpe wait_for_connections
2 1.1e-04 python2.4 PyEval_EvalFrame
2 1.1e-04 vmlinux-2.6.21.5 __bread
2 1.1e-04 vmlinux-2.6.21.5 __get_free_pages
2 1.1e-04 vmlinux-2.6.21.5 __insert_vm_struct
2 1.1e-04 vmlinux-2.6.21.5 __journal_refile_buffer
2 1.1e-04 vmlinux-2.6.21.5 __journal_remove_journal_head
2 1.1e-04 vmlinux-2.6.21.5 __journal_unfile_buffer
2 1.1e-04 vmlinux-2.6.21.5 __page_set_anon_rmap
2 1.1e-04 vmlinux-2.6.21.5 __set_page_dirty_buffers
2 1.1e-04 vmlinux-2.6.21.5 __tcp_grow_window
2 1.1e-04 vmlinux-2.6.21.5 account_user_time
2 1.1e-04 vmlinux-2.6.21.5 alarm_setitimer
2 1.1e-04 vmlinux-2.6.21.5 alloc_buffer_head
2 1.1e-04 vmlinux-2.6.21.5 bio_hw_segments
2 1.1e-04 vmlinux-2.6.21.5 blocking_notifier_call_chain
2 1.1e-04 vmlinux-2.6.21.5 check_irq_on
2 1.1e-04 vmlinux-2.6.21.5 convert_fxsr_to_user
2 1.1e-04 vmlinux-2.6.21.5 cookie_v4_check
2 1.1e-04 vmlinux-2.6.21.5 cpu_quiet
2 1.1e-04 vmlinux-2.6.21.5 dec_zone_page_state
2 1.1e-04 vmlinux-2.6.21.5 do_irq_balance
2 1.1e-04 vmlinux-2.6.21.5 drop_key_refs
2 1.1e-04 vmlinux-2.6.21.5 dst_alloc
2 1.1e-04 vmlinux-2.6.21.5 dummy_file_alloc_security
2 1.1e-04 vmlinux-2.6.21.5 dummy_task_wait
2 1.1e-04 vmlinux-2.6.21.5 dup_fd
2 1.1e-04 vmlinux-2.6.21.5 dup_mm
2 1.1e-04 vmlinux-2.6.21.5 elv_activate_rq
2 1.1e-04 vmlinux-2.6.21.5 ext3_block_to_path
2 1.1e-04 vmlinux-2.6.21.5 ext3_bread
2 1.1e-04 vmlinux-2.6.21.5 ext3_free_blocks_sb
2 1.1e-04 vmlinux-2.6.21.5 ext3_has_free_blocks
2 1.1e-04 vmlinux-2.6.21.5 ext3_new_inode
2 1.1e-04 vmlinux-2.6.21.5 ext3_read_inode
2 1.1e-04 vmlinux-2.6.21.5 ext3_truncate
2 1.1e-04 vmlinux-2.6.21.5 fcntl_setlk
2 1.1e-04 vmlinux-2.6.21.5 fib4_rule_match
2 1.1e-04 vmlinux-2.6.21.5 fib_lookup
2 1.1e-04 vmlinux-2.6.21.5 fib_rules_tclass
2 1.1e-04 vmlinux-2.6.21.5 find_group_orlov
2 1.1e-04 vmlinux-2.6.21.5 find_mergeable_anon_vma
2 1.1e-04 vmlinux-2.6.21.5 find_revoke_record
2 1.1e-04 vmlinux-2.6.21.5 flush_old_files
2 1.1e-04 vmlinux-2.6.21.5 free_uid
2 1.1e-04 vmlinux-2.6.21.5 get_jiffies_64
2 1.1e-04 vmlinux-2.6.21.5 grow_dev_page
2 1.1e-04 vmlinux-2.6.21.5 ifind_fast
2 1.1e-04 vmlinux-2.6.21.5 iget_locked
2 1.1e-04 vmlinux-2.6.21.5 inet_twdr_do_twkill_work
2 1.1e-04 vmlinux-2.6.21.5 inode_add_bytes
2 1.1e-04 vmlinux-2.6.21.5 inode_sub_bytes
2 1.1e-04 vmlinux-2.6.21.5 invalidate_mapping_pages
2 1.1e-04 vmlinux-2.6.21.5 inverted_lock
2 1.1e-04 vmlinux-2.6.21.5 ioprio_best
2 1.1e-04 vmlinux-2.6.21.5 ip_dev_find
2 1.1e-04 vmlinux-2.6.21.5 ip_reply_glue_bits
2 1.1e-04 vmlinux-2.6.21.5 ipc_checkid
2 1.1e-04 vmlinux-2.6.21.5 journal_get_descriptor_buffer
2 1.1e-04 vmlinux-2.6.21.5 journal_try_to_free_buffers
2 1.1e-04 vmlinux-2.6.21.5 ksoftirqd
2 1.1e-04 vmlinux-2.6.21.5 kswapd
2 1.1e-04 vmlinux-2.6.21.5 kunmap_high
2 1.1e-04 vmlinux-2.6.21.5 ll_back_merge_fn
2 1.1e-04 vmlinux-2.6.21.5 lock_hrtimer_base
2 1.1e-04 vmlinux-2.6.21.5 mapping_tagged
2 1.1e-04 vmlinux-2.6.21.5 mark_buffer_dirty
2 1.1e-04 vmlinux-2.6.21.5 may_expand_vm
2 1.1e-04 vmlinux-2.6.21.5 migration_thread
2 1.1e-04 vmlinux-2.6.21.5 mutex_lock
2 1.1e-04 vmlinux-2.6.21.5 neigh_periodic_timer
2 1.1e-04 vmlinux-2.6.21.5 page_slot
2 1.1e-04 vmlinux-2.6.21.5 path_lookup_create
2 1.1e-04 vmlinux-2.6.21.5 prio_tree_expand
2 1.1e-04 vmlinux-2.6.21.5 radix_tree_node_rcu_free
2 1.1e-04 vmlinux-2.6.21.5 recalc_bh_state
2 1.1e-04 vmlinux-2.6.21.5 rt_check_expire
2 1.1e-04 vmlinux-2.6.21.5 rt_may_expire
2 1.1e-04 vmlinux-2.6.21.5 scsi_free_sgtable
2 1.1e-04 vmlinux-2.6.21.5 second_overflow
2 1.1e-04 vmlinux-2.6.21.5 shrink_icache_memory
2 1.1e-04 vmlinux-2.6.21.5 simple_read_from_buffer
2 1.1e-04 vmlinux-2.6.21.5 sock_alloc_fd
2 1.1e-04 vmlinux-2.6.21.5 sock_sendpage
2 1.1e-04 vmlinux-2.6.21.5 sockfs_delete_dentry
2 1.1e-04 vmlinux-2.6.21.5 strnlen_user
2 1.1e-04 vmlinux-2.6.21.5 submit_bh
2 1.1e-04 vmlinux-2.6.21.5 sys_faccessat
2 1.1e-04 vmlinux-2.6.21.5 sys_lstat64
2 1.1e-04 vmlinux-2.6.21.5 sys_sigreturn
2 1.1e-04 vmlinux-2.6.21.5 sys_waitpid
2 1.1e-04 vmlinux-2.6.21.5 task_prio
2 1.1e-04 vmlinux-2.6.21.5 tcp_fixup_rcvbuf
2 1.1e-04 vmlinux-2.6.21.5 tcp_keepalive_timer
2 1.1e-04 vmlinux-2.6.21.5 tcp_remove_reno_sacks
2 1.1e-04 vmlinux-2.6.21.5 tcp_reset
2 1.1e-04 vmlinux-2.6.21.5 tcp_v4_timewait_ack
2 1.1e-04 vmlinux-2.6.21.5 truncate_inode_pages_range
2 1.1e-04 vmlinux-2.6.21.5 try_to_release_page
2 1.1e-04 vmlinux-2.6.21.5 twa_unmap_scsi_data
2 1.1e-04 vmlinux-2.6.21.5 vfs_lstat_fd
2 1.1e-04 vmlinux-2.6.21.5 vfs_readdir
2 1.1e-04 vmlinux-2.6.21.5 wake_futex
2 1.1e-04 vmlinux-2.6.21.5 wakeup_kswapd
2 1.1e-04 vmlinux-2.6.21.5 xfsbufd_wakeup
1 5.3e-05 amd get_exported_ap
1 5.3e-05 amd get_first_exported_ap
1 5.3e-05 cat (no symbols)
1 5.3e-05 httpd ap_allow_options
1 5.3e-05 httpd ap_count_dirs
1 5.3e-05 httpd ap_create_conn_config
1 5.3e-05 httpd ap_create_request_config
1 5.3e-05 httpd ap_get_limit_req_body
1 5.3e-05 httpd ap_get_server_port
1 5.3e-05 httpd ap_increment_counts
1 5.3e-05 httpd ap_run_http_method
1 5.3e-05 httpd ap_run_insert_error_filter
1 5.3e-05 httpd ap_run_pre_connection
1 5.3e-05 httpd apr_brigade_split
1 5.3e-05 httpd apr_bucket_alloc_create_ex
1 5.3e-05 httpd apr_bucket_file_make
1 5.3e-05 httpd apr_bucket_mmap_make
1 5.3e-05 httpd apr_bucket_simple_copy
1 5.3e-05 httpd apr_bucket_socket_create
1 5.3e-05 httpd apr_file_flags_get
1 5.3e-05 httpd apr_is_empty_array
1 5.3e-05 httpd apr_pstrndup
1 5.3e-05 httpd apr_table_addn
1 5.3e-05 httpd apr_table_overlay
1 5.3e-05 httpd apr_vsnprintf
1 5.3e-05 httpd basic_http_header
1 5.3e-05 httpd blowfishFeedback
1 5.3e-05 httpd core_override_type
1 5.3e-05 httpd handle_map_file
1 5.3e-05 httpd log_connection_status
1 5.3e-05 httpd log_remote_host
1 5.3e-05 httpd log_request_line
1 5.3e-05 httpd lookup_header
1 5.3e-05 httpd stuffbuffer
1 5.3e-05 httpd trimString
1 5.3e-05 libnetsnmphelpers.so.5.1.2 (no symbols)
1 5.3e-05 libnetsnmpmibs.so.5.1.2 (no symbols)
1 5.3e-05 libpthread-0.60.so __libc_sigaction
1 5.3e-05 libpthread-0.60.so __waitpid_nocancel
1 5.3e-05 libpthread-0.60.so pthread_rwlock_rdlock
1 5.3e-05 libpthread-0.60.so waitpid
1 5.3e-05 python2.4 Exception__init__
1 5.3e-05 python2.4 PyDict_GetItem
1 5.3e-05 python2.4 PyEval_EvalCodeEx
1 5.3e-05 python2.4 PyFunction_New
1 5.3e-05 python2.4 PyObject_GetAttr
1 5.3e-05 python2.4 PyObject_Malloc
1 5.3e-05 python2.4 PyObject_Size
1 5.3e-05 python2.4 PyString_FromFormatV
1 5.3e-05 python2.4 load_args
1 5.3e-05 python2.4 posix_listdir
1 5.3e-05 python2.4 sre_charset
1 5.3e-05 python2.4 string_startswith
1 5.3e-05 python2.4 vgetargs1
1 5.3e-05 python2.4 vgetargskeywords
1 5.3e-05 vmlinux-2.6.21.5 __add_entropy_words
1 5.3e-05 vmlinux-2.6.21.5 __clear_page_buffers
1 5.3e-05 vmlinux-2.6.21.5 __d_free
1 5.3e-05 vmlinux-2.6.21.5 __ext3_journal_get_create_access
1 5.3e-05 vmlinux-2.6.21.5 __fib_res_prefsrc
1 5.3e-05 vmlinux-2.6.21.5 __group_send_sig_info
1 5.3e-05 vmlinux-2.6.21.5 __insert_inode_hash
1 5.3e-05 vmlinux-2.6.21.5 __jbd_kmalloc
1 5.3e-05 vmlinux-2.6.21.5 __journal_drop_transaction
1 5.3e-05 vmlinux-2.6.21.5 __journal_insert_checkpoint
1 5.3e-05 vmlinux-2.6.21.5 __journal_remove_checkpoint
1 5.3e-05 vmlinux-2.6.21.5 __lookup_tag
1 5.3e-05 vmlinux-2.6.21.5 __posix_lock_file_conf
1 5.3e-05 vmlinux-2.6.21.5 __put_user_4
1 5.3e-05 vmlinux-2.6.21.5 __random32
1 5.3e-05 vmlinux-2.6.21.5 __rb_rotate_left
1 5.3e-05 vmlinux-2.6.21.5 __set_page_dirty_nobuffers
1 5.3e-05 vmlinux-2.6.21.5 __sigqueue_alloc
1 5.3e-05 vmlinux-2.6.21.5 __sigqueue_free
1 5.3e-05 vmlinux-2.6.21.5 __vma_link_rb
1 5.3e-05 vmlinux-2.6.21.5 __writeback_single_inode
1 5.3e-05 vmlinux-2.6.21.5 _d_rehash
1 5.3e-05 vmlinux-2.6.21.5 _tw32_flush
1 5.3e-05 vmlinux-2.6.21.5 alloc_new_reservation
1 5.3e-05 vmlinux-2.6.21.5 anon_pipe_buf_release
1 5.3e-05 vmlinux-2.6.21.5 anon_vma_prepare
1 5.3e-05 vmlinux-2.6.21.5 arp_bind_neighbour
1 5.3e-05 vmlinux-2.6.21.5 arp_rcv
1 5.3e-05 vmlinux-2.6.21.5 as_antic_timeout
1 5.3e-05 vmlinux-2.6.21.5 as_work_handler
1 5.3e-05 vmlinux-2.6.21.5 bitmap_search_next_usable_block
1 5.3e-05 vmlinux-2.6.21.5 blk_start_queueing
1 5.3e-05 vmlinux-2.6.21.5 can_share_swap_page
1 5.3e-05 vmlinux-2.6.21.5 check_itimerval
1 5.3e-05 vmlinux-2.6.21.5 copy_thread_group_keys
1 5.3e-05 vmlinux-2.6.21.5 de_thread
1 5.3e-05 vmlinux-2.6.21.5 debug_mutex_set_owner
1 5.3e-05 vmlinux-2.6.21.5 do_exit
1 5.3e-05 vmlinux-2.6.21.5 do_mremap
1 5.3e-05 vmlinux-2.6.21.5 do_notify_resume
1 5.3e-05 vmlinux-2.6.21.5 do_setitimer
1 5.3e-05 vmlinux-2.6.21.5 do_signal
1 5.3e-05 vmlinux-2.6.21.5 do_sys_ftruncate
1 5.3e-05 vmlinux-2.6.21.5 do_syslog
1 5.3e-05 vmlinux-2.6.21.5 do_truncate
1 5.3e-05 vmlinux-2.6.21.5 do_writepages
1 5.3e-05 vmlinux-2.6.21.5 dst_run_gc
1 5.3e-05 vmlinux-2.6.21.5 dummy_file_fcntl
1 5.3e-05 vmlinux-2.6.21.5 dummy_file_lock
1 5.3e-05 vmlinux-2.6.21.5 dummy_file_mmap
1 5.3e-05 vmlinux-2.6.21.5 dummy_inode_follow_link
1 5.3e-05 vmlinux-2.6.21.5 dummy_sem_semop
1 5.3e-05 vmlinux-2.6.21.5 dummy_task_to_inode
1 5.3e-05 vmlinux-2.6.21.5 dup_task_struct
1 5.3e-05 vmlinux-2.6.21.5 elv_may_queue
1 5.3e-05 vmlinux-2.6.21.5 elv_merged_request
1 5.3e-05 vmlinux-2.6.21.5 end_buffer_write_sync
1 5.3e-05 vmlinux-2.6.21.5 eth_header
1 5.3e-05 vmlinux-2.6.21.5 exec_mmap
1 5.3e-05 vmlinux-2.6.21.5 exit_mm
1 5.3e-05 vmlinux-2.6.21.5 ext3_alloc_branch
1 5.3e-05 vmlinux-2.6.21.5 ext3_check_dir_entry
1 5.3e-05 vmlinux-2.6.21.5 ext3_clear_inode
1 5.3e-05 vmlinux-2.6.21.5 ext3_releasepage
1 5.3e-05 vmlinux-2.6.21.5 ext3_rsv_window_add
1 5.3e-05 vmlinux-2.6.21.5 ext3_splice_branch
1 5.3e-05 vmlinux-2.6.21.5 ext3_update_dx_flag
1 5.3e-05 vmlinux-2.6.21.5 fasync_helper
1 5.3e-05 vmlinux-2.6.21.5 fib_select_default
1 5.3e-05 vmlinux-2.6.21.5 find_dcookie
1 5.3e-05 vmlinux-2.6.21.5 find_next_usable_block
1 5.3e-05 vmlinux-2.6.21.5 find_or_create_page
1 5.3e-05 vmlinux-2.6.21.5 flock_to_posix_lock
1 5.3e-05 vmlinux-2.6.21.5 forget_original_parent
1 5.3e-05 vmlinux-2.6.21.5 free_pages_and_swap_cache
1 5.3e-05 vmlinux-2.6.21.5 generic_pipe_buf_map
1 5.3e-05 vmlinux-2.6.21.5 get_init_ra_size
1 5.3e-05 vmlinux-2.6.21.5 get_random_int
1 5.3e-05 vmlinux-2.6.21.5 get_signal_to_deliver
1 5.3e-05 vmlinux-2.6.21.5 getnstimeofday
1 5.3e-05 vmlinux-2.6.21.5 handle_signal
1 5.3e-05 vmlinux-2.6.21.5 hrtimer_init
1 5.3e-05 vmlinux-2.6.21.5 hrtimer_start
1 5.3e-05 vmlinux-2.6.21.5 inet_csk_get_port
1 5.3e-05 vmlinux-2.6.21.5 inet_csk_init_xmit_timers
1 5.3e-05 vmlinux-2.6.21.5 inet_csk_reset_keepalive_timer
1 5.3e-05 vmlinux-2.6.21.5 inet_sendpage
1 5.3e-05 vmlinux-2.6.21.5 inet_twdr_hangman
1 5.3e-05 vmlinux-2.6.21.5 init_buffer_head
1 5.3e-05 vmlinux-2.6.21.5 inode_setattr
1 5.3e-05 vmlinux-2.6.21.5 invalidate_complete_page
1 5.3e-05 vmlinux-2.6.21.5 ip_cmsg_recv
1 5.3e-05 vmlinux-2.6.21.5 ipc_unlock
1 5.3e-05 vmlinux-2.6.21.5 ipv4_dst_check
1 5.3e-05 vmlinux-2.6.21.5 itimer_get_remtime
1 5.3e-05 vmlinux-2.6.21.5 jiffies_to_timespec
1 5.3e-05 vmlinux-2.6.21.5 job_control
1 5.3e-05 vmlinux-2.6.21.5 journal_alloc_journal_head
1 5.3e-05 vmlinux-2.6.21.5 journal_remove_journal_head
1 5.3e-05 vmlinux-2.6.21.5 journal_unfile_buffer
1 5.3e-05 vmlinux-2.6.21.5 journal_write_revoke_records
1 5.3e-05 vmlinux-2.6.21.5 kernel_read
1 5.3e-05 vmlinux-2.6.21.5 ktime_get
1 5.3e-05 vmlinux-2.6.21.5 locks_init_lock
1 5.3e-05 vmlinux-2.6.21.5 math_state_restore
1 5.3e-05 vmlinux-2.6.21.5 mutex_trylock
1 5.3e-05 vmlinux-2.6.21.5 n_tty_chars_in_buffer
1 5.3e-05 vmlinux-2.6.21.5 neigh_update
1 5.3e-05 vmlinux-2.6.21.5 next_pidmap
1 5.3e-05 vmlinux-2.6.21.5 nsec_to_clock_t
1 5.3e-05 vmlinux-2.6.21.5 old_mmap
1 5.3e-05 vmlinux-2.6.21.5 page_mkclean_file
1 5.3e-05 vmlinux-2.6.21.5 pmd_ctor
1 5.3e-05 vmlinux-2.6.21.5 posix_lock_file
1 5.3e-05 vmlinux-2.6.21.5 prio_tree_insert
1 5.3e-05 vmlinux-2.6.21.5 proc_alloc_inode
1 5.3e-05 vmlinux-2.6.21.5 proc_flush_task
1 5.3e-05 vmlinux-2.6.21.5 proc_sys_lookup_table_one
1 5.3e-05 vmlinux-2.6.21.5 process_timeout
1 5.3e-05 vmlinux-2.6.21.5 profile_munmap
1 5.3e-05 vmlinux-2.6.21.5 pty_unthrottle
1 5.3e-05 vmlinux-2.6.21.5 pty_write_room
1 5.3e-05 vmlinux-2.6.21.5 put_unused_fd
1 5.3e-05 vmlinux-2.6.21.5 radix_tree_gang_lookup
1 5.3e-05 vmlinux-2.6.21.5 release_task
1 5.3e-05 vmlinux-2.6.21.5 release_vm86_irqs
1 5.3e-05 vmlinux-2.6.21.5 resume_userspace
1 5.3e-05 vmlinux-2.6.21.5 rt_set_nexthop
1 5.3e-05 vmlinux-2.6.21.5 rtc_cmos_read
1 5.3e-05 vmlinux-2.6.21.5 save_i387
1 5.3e-05 vmlinux-2.6.21.5 sched_migrate_task
1 5.3e-05 vmlinux-2.6.21.5 search_reserve_window
1 5.3e-05 vmlinux-2.6.21.5 secure_ip_id
1 5.3e-05 vmlinux-2.6.21.5 select_parent
1 5.3e-05 vmlinux-2.6.21.5 send_group_sig_info
1 5.3e-05 vmlinux-2.6.21.5 seq_read
1 5.3e-05 vmlinux-2.6.21.5 set_page_dirty
1 5.3e-05 vmlinux-2.6.21.5 setup_frame
1 5.3e-05 vmlinux-2.6.21.5 sha_init
1 5.3e-05 vmlinux-2.6.21.5 shrink_dqcache_memory
1 5.3e-05 vmlinux-2.6.21.5 signal_wake_up
1 5.3e-05 vmlinux-2.6.21.5 skb_free_datagram
1 5.3e-05 vmlinux-2.6.21.5 skb_split
1 5.3e-05 vmlinux-2.6.21.5 sock_attach_fd
1 5.3e-05 vmlinux-2.6.21.5 sock_disable_timestamp
1 5.3e-05 vmlinux-2.6.21.5 sync_supers
1 5.3e-05 vmlinux-2.6.21.5 sys_alarm
1 5.3e-05 vmlinux-2.6.21.5 sys_execve
1 5.3e-05 vmlinux-2.6.21.5 sys_exit_group
1 5.3e-05 vmlinux-2.6.21.5 sys_getpid
1 5.3e-05 vmlinux-2.6.21.5 sys_llseek
1 5.3e-05 vmlinux-2.6.21.5 sys_lseek
1 5.3e-05 vmlinux-2.6.21.5 sys_mkdirat
1 5.3e-05 vmlinux-2.6.21.5 sys_pipe
1 5.3e-05 vmlinux-2.6.21.5 sys_wait4
1 5.3e-05 vmlinux-2.6.21.5 tcp_incr_quickack
1 5.3e-05 vmlinux-2.6.21.5 tcp_init_xmit_timers
1 5.3e-05 vmlinux-2.6.21.5 tcp_update_reordering
1 5.3e-05 vmlinux-2.6.21.5 tcp_v4_send_ack
1 5.3e-05 vmlinux-2.6.21.5 tcp_v4_send_reset
1 5.3e-05 vmlinux-2.6.21.5 tcp_xmit_probe_skb
1 5.3e-05 vmlinux-2.6.21.5 test_clear_page_writeback
1 5.3e-05 vmlinux-2.6.21.5 tg3_timer
1 5.3e-05 vmlinux-2.6.21.5 throttle_vm_writeout
1 5.3e-05 vmlinux-2.6.21.5 try_to_free_buffers
1 5.3e-05 vmlinux-2.6.21.5 tty_default_put_char
1 5.3e-05 vmlinux-2.6.21.5 tty_hung_up_p
1 5.3e-05 vmlinux-2.6.21.5 tty_ldisc_ref_wait
1 5.3e-05 vmlinux-2.6.21.5 tty_ldisc_try
1 5.3e-05 vmlinux-2.6.21.5 tty_paranoia_check
1 5.3e-05 vmlinux-2.6.21.5 tty_poll
1 5.3e-05 vmlinux-2.6.21.5 tty_wakeup
1 5.3e-05 vmlinux-2.6.21.5 unlock_new_inode
1 5.3e-05 vmlinux-2.6.21.5 unmap_underlying_metadata
1 5.3e-05 vmlinux-2.6.21.5 unshare_files
1 5.3e-05 vmlinux-2.6.21.5 uptime_read_proc
1 5.3e-05 vmlinux-2.6.21.5 verify_chain
1 5.3e-05 vmlinux-2.6.21.5 vfs_llseek
1 5.3e-05 vmlinux-2.6.21.5 vfs_lstat
1 5.3e-05 vmlinux-2.6.21.5 vma_prio_tree_insert
1 5.3e-05 vmlinux-2.6.21.5 vma_prio_tree_remove
1 5.3e-05 vmlinux-2.6.21.5 vmtruncate
1 5.3e-05 vmlinux-2.6.21.5 wake_up_new_task
1 5.3e-05 vmlinux-2.6.21.5 writeback_inodes
^ permalink raw reply [flat|nested] 10+ messages in thread* Re: miserable performance of 2.6.21 under network load
2007-07-24 18:49 ` Aaron Porter
@ 2007-07-24 19:13 ` Jeff Garzik
2007-07-24 19:13 ` Andi Kleen
` (2 subsequent siblings)
3 siblings, 0 replies; 10+ messages in thread
From: Jeff Garzik @ 2007-07-24 19:13 UTC (permalink / raw)
To: Aaron Porter; +Cc: Andi Kleen, linux-kernel
Turn off slab debugging and preempt, and see if that helps.
Jeff
^ permalink raw reply [flat|nested] 10+ messages in thread* Re: miserable performance of 2.6.21 under network load
2007-07-24 18:49 ` Aaron Porter
2007-07-24 19:13 ` Jeff Garzik
@ 2007-07-24 19:13 ` Andi Kleen
2007-07-24 20:41 ` Aaron Porter
2007-07-24 19:22 ` Chris Snook
2007-07-24 19:45 ` David Miller
3 siblings, 1 reply; 10+ messages in thread
From: Andi Kleen @ 2007-07-24 19:13 UTC (permalink / raw)
To: Aaron Porter; +Cc: Andi Kleen, linux-kernel
> # opreport -l
> CPU: AMD64 processors, speed 1994.52 MHz (estimated)
> Counted CPU_CLK_UNHALTED events (Cycles outside of halt state) with a unit mask of 0x00 (No unit mask) count 100000
> samples % app name symbol name
> 914379 48.8404 vmlinux-2.6.21.5 check_poison_obj
> 341920 18.2632 vmlinux-2.6.21.5 poison_obj
You have slab debugging enabled. That makes everything slow.
It costs you ~66% of your CPU time. Disable it.
-Andi
^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: miserable performance of 2.6.21 under network load
2007-07-24 18:49 ` Aaron Porter
2007-07-24 19:13 ` Jeff Garzik
2007-07-24 19:13 ` Andi Kleen
@ 2007-07-24 19:22 ` Chris Snook
2007-07-24 19:45 ` David Miller
3 siblings, 0 replies; 10+ messages in thread
From: Chris Snook @ 2007-07-24 19:22 UTC (permalink / raw)
To: Aaron Porter; +Cc: Andi Kleen, linux-kernel
Aaron Porter wrote:
> On Tue, Jul 24, 2007 at 08:48:00PM +0200, Andi Kleen wrote:
>> Aaron Porter <atporter@primate.net> writes:
>>
>>> I'm in the process up upgrading a pool of apache servers from
>>> 2.6.17.8 to 2.6.21.5, and we're seeing a pretty major change in behavior.
>>> Under identical network load, 2.6.21 has a load average more than 3 times
>>> higher, cpu 0 spends well over 90% of its time in interrupts (vs ~30%
>>> under 2.6.17). When we hit 3k apache sessions, ksoftirqd eats 100% of cpu0
>>> and our network traffic drops off rapidly. The end result is that 2.6.17
>>> performs twice as well under this load.
>> Can you oprofile it?
>
> # opreport -l
> CPU: AMD64 processors, speed 1994.52 MHz (estimated)
> Counted CPU_CLK_UNHALTED events (Cycles outside of halt state) with a unit mask of 0x00 (No unit mask) count 100000
> samples % app name symbol name
> 914379 48.8404 vmlinux-2.6.21.5 check_poison_obj
> 341920 18.2632 vmlinux-2.6.21.5 poison_obj
I bet you have CONFIG_DEBUG_SLAB turned off in your 2.6.17 kernel, and turned on
in your 2.6.21 kernel.
-- Chris
^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: miserable performance of 2.6.21 under network load
2007-07-24 18:49 ` Aaron Porter
` (2 preceding siblings ...)
2007-07-24 19:22 ` Chris Snook
@ 2007-07-24 19:45 ` David Miller
3 siblings, 0 replies; 10+ messages in thread
From: David Miller @ 2007-07-24 19:45 UTC (permalink / raw)
To: atporter; +Cc: andi, linux-kernel
From: Aaron Porter <atporter@primate.net>
Date: Tue, 24 Jul 2007 11:49:09 -0700
> samples % app name symbol name
> 914379 48.8404 vmlinux-2.6.21.5 check_poison_obj
> 341920 18.2632 vmlinux-2.6.21.5 poison_obj
> 37355 1.9953 nf_conntrack (no symbols)
You have SLAB debugging enabled, turn it off if you want
decent performance.
^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: miserable performance of 2.6.21 under network load
2007-07-24 17:25 miserable performance of 2.6.21 under network load Aaron Porter
2007-07-24 18:48 ` Andi Kleen
@ 2007-07-24 18:49 ` Chris Snook
2007-07-24 18:53 ` Aaron Porter
1 sibling, 1 reply; 10+ messages in thread
From: Chris Snook @ 2007-07-24 18:49 UTC (permalink / raw)
To: Aaron Porter; +Cc: linux-kernel
Aaron Porter wrote:
> I'm in the process up upgrading a pool of apache servers from
> 2.6.17.8 to 2.6.21.5, and we're seeing a pretty major change in behavior.
> Under identical network load, 2.6.21 has a load average more than 3 times
> higher, cpu 0 spends well over 90% of its time in interrupts (vs ~30%
> under 2.6.17). When we hit 3k apache sessions, ksoftirqd eats 100% of cpu0
> and our network traffic drops off rapidly. The end result is that 2.6.17
> performs twice as well under this load.
Is it always CPU 0, or does it move? Are you running irqbalance? If you're
running irqbalance, you can run a script that alternates between 'cat
/proc/interrupts' and 'mpstat -P ALL 5 10' and watch the offending interrupt
jump around between processors. It's not as informative as oprofile, as Andi
suggested, but it's really easy to set up.
-- Chris
^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: miserable performance of 2.6.21 under network load
2007-07-24 18:49 ` Chris Snook
@ 2007-07-24 18:53 ` Aaron Porter
0 siblings, 0 replies; 10+ messages in thread
From: Aaron Porter @ 2007-07-24 18:53 UTC (permalink / raw)
To: Chris Snook, linux-kernel
On Tue, Jul 24, 2007 at 02:49:47PM -0400, Chris Snook wrote:
> Is it always CPU 0, or does it move? Are you running irqbalance? If
> you're running irqbalance, you can run a script that alternates between
> 'cat /proc/interrupts' and 'mpstat -P ALL 5 10' and watch the offending
> interrupt jump around between processors. It's not as informative as
> oprofile, as Andi suggested, but it's really easy to set up.
Always cpu0
# cat /proc/interrupts
CPU0 CPU1 CPU2 CPU3
0: 166 0 0 0 XT-PIC-XT timer
1: 8 0 0 0 XT-PIC-XT i8042
2: 0 0 0 0 XT-PIC-XT cascade
3: 11 0 0 0 XT-PIC-XT serial
4: 830 0 0 0 XT-PIC-XT serial
5: 370787109 0 0 0 XT-PIC-XT eth0
6: 3 0 0 0 XT-PIC-XT floppy
8: 4 0 0 0 XT-PIC-XT rtc
9: 0 0 0 0 XT-PIC-XT ohci_hcd:usb1, ohci_hcd:usb2
10: 80937699 0 0 0 XT-PIC-XT 3w-9xxx, 3w-9xxx
12: 5 0 0 0 XT-PIC-XT i8042
14: 50931 0 0 0 XT-PIC-XT ide0
NMI: 1110304 275156 276732 233082
LOC: 278410000 278410022 278410013 278410014
ERR: 55003177
MIS: 0
# mpstat -P ALL 5 1
Linux 2.6.21.5 (*.com) 07/24/07
11:52:30 CPU %user %nice %system %iowait %irq %soft %idle intr/s
11:52:35 all 0.40 0.00 16.12 13.56 0.25 25.13 44.54 299.80
11:52:35 0 0.00 0.00 0.00 0.00 1.00 99.00 0.00 299.80
11:52:35 1 0.80 0.00 23.65 23.05 0.00 0.40 52.10 0.00
11:52:35 2 0.40 0.00 21.64 14.23 0.00 0.40 63.53 0.00
11:52:35 3 0.60 0.00 19.44 17.03 0.00 0.60 62.73 0.00
Average: CPU %user %nice %system %iowait %irq %soft %idle intr/s
Average: all 0.40 0.00 16.12 13.56 0.25 25.13 44.54 299.80
Average: 0 0.00 0.00 0.00 0.00 1.00 99.00 0.00 299.80
Average: 1 0.80 0.00 23.65 23.05 0.00 0.40 52.10 0.00
Average: 2 0.40 0.00 21.64 14.23 0.00 0.40 63.53 0.00
Average: 3 0.60 0.00 19.44 17.03 0.00 0.60 62.73 0.00
#
^ permalink raw reply [flat|nested] 10+ messages in thread
end of thread, other threads:[~2007-07-24 20:41 UTC | newest]
Thread overview: 10+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2007-07-24 17:25 miserable performance of 2.6.21 under network load Aaron Porter
2007-07-24 18:48 ` Andi Kleen
2007-07-24 18:49 ` Aaron Porter
2007-07-24 19:13 ` Jeff Garzik
2007-07-24 19:13 ` Andi Kleen
2007-07-24 20:41 ` Aaron Porter
2007-07-24 19:22 ` Chris Snook
2007-07-24 19:45 ` David Miller
2007-07-24 18:49 ` Chris Snook
2007-07-24 18:53 ` Aaron Porter
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox