* [RFC] Use RCU for tcp_ehash lookup
@ 2004-08-31 12:59 Srivatsa Vaddagiri
2004-08-31 13:04 ` Srivatsa Vaddagiri
` (3 more replies)
0 siblings, 4 replies; 11+ messages in thread
From: Srivatsa Vaddagiri @ 2004-08-31 12:59 UTC (permalink / raw)
To: davem; +Cc: netdev, linux-kernel, Dipankar, paulmck
Hello,
I have been looking at making tcp_ehash table lookup lock-free using
RCU technique. I found _significant_ reduction in profile count (~56% savings)
for __tcp_v4_lookup_established if it is made lock-free. Hence I am presenting
the patch below for discussion/consideration.
Some notes on the patch:
- Although readprofile shows improvement in tick count for
__tcp_v4_lookup_established, I haven't come across any benchmarks that is
benefited noticeably by the lock-free lookup. I have tried httperf, netperf
and simple file transfer tests so far.
This could possibly be because the hash table size on the machines I was
testing was high (tcp_ehash_size = 128K), leading to low contention rate on
the hash bucket locks. Also because of the fact that lookup could happen in
parallel to socket input packet processing.
I would be interested to know if anyone has seen high-rate of lock contention
for hash bucket lock. Such workloads would benefit from the lock-free lookup.
In the absence of any workload which resulted in lock contention, I resorted
to disabling NAPI and irq balance (noirqbalance) to study the effect of cache
bouncing on the lookup routine. The result was that CPU usage of the stack
was halved in lock-free case, which IMHO, is a strong enough reason for us
to consider this seriously.
- I presume that one of the reasons for keeping the hash table so big is to
keep lock contention low (& to reduce the size of hash chains). If the lookup
is made lock-free, then could the size of the hash table be reduced (without
adversely impacting performance)?
- Biggest problem I had converting over to RCU was the refcount race between
sock_put and sock_hold. sock_put might see the refcount go to zero and decide
to free the object, while on some other CPU, sock_get's are pending against
the same object. The patch handles the race by deciding to free the object
only from the RCU callback.
- Generic code like sock_put/__sk_add_node/sk_for_each, which are common to
other protocols, have been modified to be RCU aware. I need to abstract this
out so that other protocols not using RCU aren't affected by my changes.
- Socket table lookups that happens thr', say /proc/net/tcp or tcpdiag_dump, is
not lock-free yet. This is because of movement of socket performed in
__tcp_tw_hashdance, between established half to time-wait half.
There is a window during this movement, when the same socket is present
on both time-wait half as well as established half. I felt that it is not
good to have /proc/net/tcp report two instances of the same socket. Hence
I resorted to have /proc/net/tcp and tcpdiag_dump doing the lookup using
a spinlock.
Note that __tcp_v4_lookup_established should not be affected by the above
movement because I found it scans the established half first and _then_ the
time wait half. So even if the same socket is present in both established half
and time wait half, __tcp_v4_lookup_established will lookup only one of them
(& not both).
Details of the test conducted are as below:
-------- 10/100MBps Ethernet Hub --------
| llm09 | <--------------------------- | llm10 |
-------- --------
10.0.1.3 10.0.1.1
llm09 - 8way 1GHz Intel P3 box with 1GB RAM
llm10 - 4way 1.2GHz Power4 box (p630) with 4GB RAM
Both these machines were connected to a hub with IP addresses
10.0.1.3 (llm09) and 10.0.1.1 (llm10).
File of size 526MB was downloaded on llm09 from llm10. Snapshot of profile
information was taken over this file transfer period. This test was repeated
10 times using a script as below:
i=0
while [ $i -lt 10 ]
do
cat /proc/interrupts | grep eth >> logs/intlog
readprofile -r
(time -p scp -q root@10.0.1.1:~vatsa/delme/x . 2> /dev/null) 2>>timelog
cp /proc/profile logs/profile-$i
readprofile -p logs/profile-$i -m <System.map> >> rplog
cat /proc/interrupts | grep eth >> logs/intlog
rm -f x
i=`expr $i + 1`
done
I have attached rplog for both lock-free as well as read-lock cases.
Actual patch, against 2.6.8.1, follows. Flames wellcome :)
---
linux-2.6.8.1-vatsa/include/net/sock.h | 23 ++++++++++--
linux-2.6.8.1-vatsa/include/net/tcp.h | 23 +++++++++---
linux-2.6.8.1-vatsa/net/core/sock.c | 11 +++++
linux-2.6.8.1-vatsa/net/ipv4/tcp.c | 2 -
linux-2.6.8.1-vatsa/net/ipv4/tcp_diag.c | 11 +++--
linux-2.6.8.1-vatsa/net/ipv4/tcp_ipv4.c | 50 ++++++++++++++++-----------
linux-2.6.8.1-vatsa/net/ipv4/tcp_minisocks.c | 47 ++++++++++++++++++++-----
linux-2.6.8.1-vatsa/net/ipv6/tcp_ipv6.c | 22 +++++++----
8 files changed, 136 insertions(+), 53 deletions(-)
diff -puN include/net/sock.h~tcp_ehash include/net/sock.h
--- linux-2.6.8.1/include/net/sock.h~tcp_ehash 2004-08-25 18:06:42.000000000 +0530
+++ linux-2.6.8.1-vatsa/include/net/sock.h 2004-08-30 12:23:17.000000000 +0530
@@ -50,6 +50,7 @@
#include <linux/security.h>
#include <linux/filter.h>
+#include <linux/rcupdate.h>
#include <asm/atomic.h>
#include <net/dst.h>
@@ -178,6 +179,7 @@ struct sock_common {
* @sk_error_report - callback to indicate errors (e.g. %MSG_ERRQUEUE)
* @sk_backlog_rcv - callback to process the backlog
* @sk_destruct - called at sock freeing time, i.e. when all refcnt == 0
+ * @sk_rcu - RCU callback structure
*/
struct sock {
/*
@@ -266,6 +268,7 @@ struct sock {
int (*sk_backlog_rcv)(struct sock *sk,
struct sk_buff *skb);
void (*sk_destruct)(struct sock *sk);
+ struct rcu_head sk_rcu;
};
/*
@@ -350,7 +353,7 @@ static __inline__ int sk_del_node_init(s
static __inline__ void __sk_add_node(struct sock *sk, struct hlist_head *list)
{
- hlist_add_head(&sk->sk_node, list);
+ hlist_add_head_rcu(&sk->sk_node, list);
}
static __inline__ void sk_add_node(struct sock *sk, struct hlist_head *list)
@@ -371,7 +374,7 @@ static __inline__ void sk_add_bind_node(
}
#define sk_for_each(__sk, node, list) \
- hlist_for_each_entry(__sk, node, list, sk_node)
+ hlist_for_each_entry_rcu(__sk, node, list, sk_node)
#define sk_for_each_from(__sk, node) \
if (__sk && ({ node = &(__sk)->sk_node; 1; })) \
hlist_for_each_entry_from(__sk, node, sk_node)
@@ -703,6 +706,7 @@ extern void FASTCALL(release_sock(struct
extern struct sock * sk_alloc(int family, int priority, int zero_it,
kmem_cache_t *slab);
extern void sk_free(struct sock *sk);
+extern void sk_free_rcu(struct rcu_head *head);
extern struct sk_buff *sock_wmalloc(struct sock *sk,
unsigned long size, int force,
@@ -888,8 +892,19 @@ static inline void sk_filter_charge(stru
/* Ungrab socket and destroy it, if it was the last reference. */
static inline void sock_put(struct sock *sk)
{
- if (atomic_dec_and_test(&sk->sk_refcnt))
- sk_free(sk);
+sp_loop:
+ if (atomic_dec_and_test(&sk->sk_refcnt)) {
+ /* Restore ref count and schedule callback.
+ * If we don't restore ref count, then the callback can be
+ * scheduled by more than one CPU.
+ */
+ atomic_inc(&sk->sk_refcnt);
+
+ if (atomic_read(&sk->sk_refcnt) == 1)
+ call_rcu(&sk->sk_rcu, sk_free_rcu);
+ else
+ goto sp_loop;
+ }
}
/* Detach socket from process context.
diff -puN include/net/tcp.h~tcp_ehash include/net/tcp.h
--- linux-2.6.8.1/include/net/tcp.h~tcp_ehash 2004-08-25 18:06:42.000000000 +0530
+++ linux-2.6.8.1-vatsa/include/net/tcp.h 2004-08-26 16:57:50.000000000 +0530
@@ -30,6 +30,7 @@
#include <linux/slab.h>
#include <linux/cache.h>
#include <linux/percpu.h>
+#include <linux/rcupdate.h>
#include <net/checksum.h>
#include <net/sock.h>
#include <net/snmp.h>
@@ -44,7 +45,7 @@
* for the rest. I'll experiment with dynamic table growth later.
*/
struct tcp_ehash_bucket {
- rwlock_t lock;
+ spinlock_t lock;
struct hlist_head chain;
} __attribute__((__aligned__(8)));
@@ -222,12 +223,13 @@ struct tcp_tw_bucket {
struct in6_addr tw_v6_rcv_saddr;
int tw_v6_ipv6only;
#endif
+ struct rcu_head tw_rcu;
};
static __inline__ void tw_add_node(struct tcp_tw_bucket *tw,
struct hlist_head *list)
{
- hlist_add_head(&tw->tw_node, list);
+ hlist_add_head_rcu(&tw->tw_node, list);
}
static __inline__ void tw_add_bind_node(struct tcp_tw_bucket *tw,
@@ -305,14 +307,23 @@ static inline int tcp_v6_ipv6only(const
#endif
extern kmem_cache_t *tcp_timewait_cachep;
+extern void tcp_tw_free(struct rcu_head *head);
static inline void tcp_tw_put(struct tcp_tw_bucket *tw)
{
+tput_loop:
if (atomic_dec_and_test(&tw->tw_refcnt)) {
-#ifdef INET_REFCNT_DEBUG
- printk(KERN_DEBUG "tw_bucket %p released\n", tw);
-#endif
- kmem_cache_free(tcp_timewait_cachep, tw);
+ /* Restore ref count and schedule callback.
+ * If we don't restore ref count, then the callback can be
+ * scheduled by more than one CPU.
+ */
+
+ atomic_inc(&tw->tw_refcnt);
+
+ if (atomic_read(&tw->tw_refcnt) == 1)
+ call_rcu(&tw->tw_rcu, tcp_tw_free);
+ else
+ goto tput_loop;
}
}
diff -puN net/core/sock.c~tcp_ehash net/core/sock.c
--- linux-2.6.8.1/net/core/sock.c~tcp_ehash 2004-08-25 18:06:42.000000000 +0530
+++ linux-2.6.8.1-vatsa/net/core/sock.c 2004-08-26 16:53:14.000000000 +0530
@@ -657,6 +657,16 @@ void sk_free(struct sock *sk)
module_put(owner);
}
+/* RCU callback to free a socket */
+
+void sk_free_rcu(struct rcu_head *head)
+{
+ struct sock *sk = container_of(head, struct sock, sk_rcu);
+
+ if (atomic_dec_and_test(&sk->sk_refcnt))
+ sk_free(sk);
+}
+
void __init sk_init(void)
{
sk_cachep = kmem_cache_create("sock", sizeof(struct sock), 0,
@@ -1347,6 +1357,7 @@ EXPORT_SYMBOL(__lock_sock);
EXPORT_SYMBOL(__release_sock);
EXPORT_SYMBOL(sk_alloc);
EXPORT_SYMBOL(sk_free);
+EXPORT_SYMBOL(sk_free_rcu);
EXPORT_SYMBOL(sk_send_sigurg);
EXPORT_SYMBOL(sock_alloc_send_pskb);
EXPORT_SYMBOL(sock_alloc_send_skb);
diff -puN net/ipv4/tcp_ipv4.c~tcp_ehash net/ipv4/tcp_ipv4.c
--- linux-2.6.8.1/net/ipv4/tcp_ipv4.c~tcp_ehash 2004-08-25 18:06:42.000000000 +0530
+++ linux-2.6.8.1-vatsa/net/ipv4/tcp_ipv4.c 2004-08-25 18:07:27.000000000 +0530
@@ -351,7 +351,8 @@ void tcp_listen_wlock(void)
static __inline__ void __tcp_v4_hash(struct sock *sk, const int listen_possible)
{
struct hlist_head *list;
- rwlock_t *lock;
+ rwlock_t *lock = NULL;
+ spinlock_t *slock = NULL;
BUG_TRAP(sk_unhashed(sk));
if (listen_possible && sk->sk_state == TCP_LISTEN) {
@@ -360,14 +361,16 @@ static __inline__ void __tcp_v4_hash(str
tcp_listen_wlock();
} else {
list = &tcp_ehash[(sk->sk_hashent = tcp_sk_hashfn(sk))].chain;
- lock = &tcp_ehash[sk->sk_hashent].lock;
- write_lock(lock);
+ slock = &tcp_ehash[sk->sk_hashent].lock;
+ spin_lock(slock);
}
__sk_add_node(sk, list);
sock_prot_inc_use(sk->sk_prot);
- write_unlock(lock);
- if (listen_possible && sk->sk_state == TCP_LISTEN)
+ if (listen_possible && sk->sk_state == TCP_LISTEN) {
+ write_unlock(lock);
wake_up(&tcp_lhash_wait);
+ } else
+ spin_unlock(slock);
}
static void tcp_v4_hash(struct sock *sk)
@@ -381,7 +384,8 @@ static void tcp_v4_hash(struct sock *sk)
void tcp_unhash(struct sock *sk)
{
- rwlock_t *lock;
+ rwlock_t *lock = NULL;
+ spinlock_t *slock = NULL;
if (sk_unhashed(sk))
goto ende;
@@ -392,17 +396,20 @@ void tcp_unhash(struct sock *sk)
lock = &tcp_lhash_lock;
} else {
struct tcp_ehash_bucket *head = &tcp_ehash[sk->sk_hashent];
- lock = &head->lock;
- write_lock_bh(&head->lock);
+ slock = &head->lock;
+ spin_lock_bh(&head->lock);
}
if (__sk_del_node_init(sk))
sock_prot_dec_use(sk->sk_prot);
- write_unlock_bh(lock);
+ if (sk->sk_state != TCP_LISTEN)
+ spin_unlock_bh(slock);
+ else {
+ write_unlock_bh(lock);
ende:
- if (sk->sk_state == TCP_LISTEN)
wake_up(&tcp_lhash_wait);
+ }
}
/* Don't inline this cruft. Here are some nice properties to
@@ -494,7 +501,7 @@ static inline struct sock *__tcp_v4_look
*/
int hash = tcp_hashfn(daddr, hnum, saddr, sport);
head = &tcp_ehash[hash];
- read_lock(&head->lock);
+ rcu_read_lock();
sk_for_each(sk, node, &head->chain) {
if (TCP_IPV4_MATCH(sk, acookie, saddr, daddr, ports, dif))
goto hit; /* You sunk my battleship! */
@@ -507,7 +514,7 @@ static inline struct sock *__tcp_v4_look
}
sk = NULL;
out:
- read_unlock(&head->lock);
+ rcu_read_unlock();
return sk;
hit:
sock_hold(sk);
@@ -559,7 +566,7 @@ static int __tcp_v4_check_established(st
struct hlist_node *node;
struct tcp_tw_bucket *tw;
- write_lock(&head->lock);
+ spin_lock(&head->lock);
/* Check TIME-WAIT sockets first. */
sk_for_each(sk2, node, &(head + tcp_ehash_size)->chain) {
@@ -614,7 +621,7 @@ unique:
BUG_TRAP(sk_unhashed(sk));
__sk_add_node(sk, &head->chain);
sock_prot_inc_use(sk->sk_prot);
- write_unlock(&head->lock);
+ spin_unlock(&head->lock);
if (twp) {
*twp = tw;
@@ -630,7 +637,7 @@ unique:
return 0;
not_unique:
- write_unlock(&head->lock);
+ spin_unlock(&head->lock);
return -EADDRNOTAVAIL;
}
@@ -2228,7 +2235,10 @@ static void *established_get_first(struc
struct hlist_node *node;
struct tcp_tw_bucket *tw;
- read_lock(&tcp_ehash[st->bucket].lock);
+ /* Take the spinlock. Otherwise a dancing socket
+ * (__tcp_tw_hashdance) may be reported twice!
+ */
+ spin_lock(&tcp_ehash[st->bucket].lock);
sk_for_each(sk, node, &tcp_ehash[st->bucket].chain) {
if (sk->sk_family != st->family) {
continue;
@@ -2245,7 +2255,7 @@ static void *established_get_first(struc
rc = tw;
goto out;
}
- read_unlock(&tcp_ehash[st->bucket].lock);
+ spin_unlock(&tcp_ehash[st->bucket].lock);
st->state = TCP_SEQ_STATE_ESTABLISHED;
}
out:
@@ -2272,10 +2282,10 @@ get_tw:
cur = tw;
goto out;
}
- read_unlock(&tcp_ehash[st->bucket].lock);
+ spin_unlock(&tcp_ehash[st->bucket].lock);
st->state = TCP_SEQ_STATE_ESTABLISHED;
if (++st->bucket < tcp_ehash_size) {
- read_lock(&tcp_ehash[st->bucket].lock);
+ spin_lock(&tcp_ehash[st->bucket].lock);
sk = sk_head(&tcp_ehash[st->bucket].chain);
} else {
cur = NULL;
@@ -2385,7 +2395,7 @@ static void tcp_seq_stop(struct seq_file
case TCP_SEQ_STATE_TIME_WAIT:
case TCP_SEQ_STATE_ESTABLISHED:
if (v)
- read_unlock(&tcp_ehash[st->bucket].lock);
+ spin_unlock(&tcp_ehash[st->bucket].lock);
local_bh_enable();
break;
}
diff -puN net/ipv4/tcp.c~tcp_ehash net/ipv4/tcp.c
--- linux-2.6.8.1/net/ipv4/tcp.c~tcp_ehash 2004-08-25 18:06:42.000000000 +0530
+++ linux-2.6.8.1-vatsa/net/ipv4/tcp.c 2004-08-25 18:07:27.000000000 +0530
@@ -2258,7 +2258,7 @@ void __init tcp_init(void)
if (!tcp_ehash)
panic("Failed to allocate TCP established hash table\n");
for (i = 0; i < (tcp_ehash_size << 1); i++) {
- tcp_ehash[i].lock = RW_LOCK_UNLOCKED;
+ tcp_ehash[i].lock = SPIN_LOCK_UNLOCKED;
INIT_HLIST_HEAD(&tcp_ehash[i].chain);
}
diff -puN net/ipv4/tcp_diag.c~tcp_ehash net/ipv4/tcp_diag.c
--- linux-2.6.8.1/net/ipv4/tcp_diag.c~tcp_ehash 2004-08-25 18:06:42.000000000 +0530
+++ linux-2.6.8.1-vatsa/net/ipv4/tcp_diag.c 2004-08-25 18:07:27.000000000 +0530
@@ -522,7 +522,10 @@ skip_listen_ht:
if (i > s_i)
s_num = 0;
- read_lock_bh(&head->lock);
+ /* Take the spinlock. Otherwise a dancing socket
+ * (__tcp_tw_hashdance) may be reported twice!
+ */
+ spin_lock_bh(&head->lock);
num = 0;
sk_for_each(sk, node, &head->chain) {
@@ -542,7 +545,7 @@ skip_listen_ht:
if (tcpdiag_fill(skb, sk, r->tcpdiag_ext,
NETLINK_CB(cb->skb).pid,
cb->nlh->nlmsg_seq) <= 0) {
- read_unlock_bh(&head->lock);
+ spin_unlock_bh(&head->lock);
goto done;
}
++num;
@@ -568,13 +571,13 @@ skip_listen_ht:
if (tcpdiag_fill(skb, sk, r->tcpdiag_ext,
NETLINK_CB(cb->skb).pid,
cb->nlh->nlmsg_seq) <= 0) {
- read_unlock_bh(&head->lock);
+ spin_unlock_bh(&head->lock);
goto done;
}
++num;
}
}
- read_unlock_bh(&head->lock);
+ spin_unlock_bh(&head->lock);
}
done:
diff -puN net/ipv4/tcp_minisocks.c~tcp_ehash net/ipv4/tcp_minisocks.c
--- linux-2.6.8.1/net/ipv4/tcp_minisocks.c~tcp_ehash 2004-08-25 18:06:42.000000000 +0530
+++ linux-2.6.8.1-vatsa/net/ipv4/tcp_minisocks.c 2004-08-26 16:58:08.000000000 +0530
@@ -64,14 +64,14 @@ static void tcp_timewait_kill(struct tcp
/* Unlink from established hashes. */
ehead = &tcp_ehash[tw->tw_hashent];
- write_lock(&ehead->lock);
+ spin_lock(&ehead->lock);
if (hlist_unhashed(&tw->tw_node)) {
- write_unlock(&ehead->lock);
+ spin_unlock(&ehead->lock);
return;
}
__hlist_del(&tw->tw_node);
sk_node_init(&tw->tw_node);
- write_unlock(&ehead->lock);
+ spin_unlock(&ehead->lock);
/* Disassociate with bind bucket. */
bhead = &tcp_bhash[tcp_bhashfn(tw->tw_num)];
@@ -308,17 +308,28 @@ static void __tcp_tw_hashdance(struct so
tw_add_bind_node(tw, &tw->tw_tb->owners);
spin_unlock(&bhead->lock);
- write_lock(&ehead->lock);
+ spin_lock(&ehead->lock);
- /* Step 2: Remove SK from established hash. */
- if (__sk_del_node_init(sk))
- sock_prot_dec_use(sk->sk_prot);
+ /*
+ * We have to be carefull here since there could be racing
+ * (lock-free) lookups happening on other CPUs. If we remove SK first
+ * and then add TW, then there is a tiny window where this socket is
+ * in neither the established half nor in the TIMEWAIT half of the ehash
+ * table. Lookups occuring in that window can drop packets!
+ * Hence we first add TW and then remove SK, with a barrier in between.
+ */
- /* Step 3: Hash TW into TIMEWAIT half of established hash table. */
+ /* Step 2: Hash TW into TIMEWAIT half of established hash table. */
tw_add_node(tw, &(ehead + tcp_ehash_size)->chain);
atomic_inc(&tw->tw_refcnt);
- write_unlock(&ehead->lock);
+ smp_wmb();
+
+ /* Step 3: Remove SK from established hash. */
+ if (__sk_del_node_init(sk))
+ sock_prot_dec_use(sk->sk_prot);
+
+ spin_unlock(&ehead->lock);
}
/*
@@ -1069,11 +1080,29 @@ int tcp_child_process(struct sock *paren
return ret;
}
+/* RCU callback to free a timewait bucket */
+
+void tcp_tw_free(struct rcu_head *head)
+{
+ struct tcp_tw_bucket *tw =
+ container_of(head, struct tcp_tw_bucket, tw_rcu);
+
+ if (atomic_dec_and_test(&tw->tw_refcnt)) {
+#ifdef INET_REFCNT_DEBUG
+ printk(KERN_DEBUG "tw_bucket %p released\n", tw);
+#endif
+ kmem_cache_free(tcp_timewait_cachep, tw);
+ }
+}
+
+
+
EXPORT_SYMBOL(tcp_check_req);
EXPORT_SYMBOL(tcp_child_process);
EXPORT_SYMBOL(tcp_create_openreq_child);
EXPORT_SYMBOL(tcp_timewait_state_process);
EXPORT_SYMBOL(tcp_tw_deschedule);
+EXPORT_SYMBOL(tcp_tw_free);
#ifdef CONFIG_SYSCTL
EXPORT_SYMBOL(sysctl_tcp_tw_recycle);
diff -puN net/ipv6/tcp_ipv6.c~tcp_ehash net/ipv6/tcp_ipv6.c
--- linux-2.6.8.1/net/ipv6/tcp_ipv6.c~tcp_ehash 2004-08-25 18:06:42.000000000 +0530
+++ linux-2.6.8.1-vatsa/net/ipv6/tcp_ipv6.c 2004-08-25 18:07:27.000000000 +0530
@@ -210,7 +210,8 @@ fail:
static __inline__ void __tcp_v6_hash(struct sock *sk)
{
struct hlist_head *list;
- rwlock_t *lock;
+ rwlock_t *lock = NULL;
+ spinlock_t *slock = NULL;
BUG_TRAP(sk_unhashed(sk));
@@ -221,13 +222,16 @@ static __inline__ void __tcp_v6_hash(str
} else {
sk->sk_hashent = tcp_v6_sk_hashfn(sk);
list = &tcp_ehash[sk->sk_hashent].chain;
- lock = &tcp_ehash[sk->sk_hashent].lock;
- write_lock(lock);
+ slock = &tcp_ehash[sk->sk_hashent].lock;
+ spin_lock(slock);
}
__sk_add_node(sk, list);
sock_prot_inc_use(sk->sk_prot);
- write_unlock(lock);
+ if (sk->sk_state == TCP_LISTEN)
+ write_unlock(lock);
+ else
+ spin_unlock(slock);
}
@@ -307,7 +311,7 @@ static inline struct sock *__tcp_v6_look
*/
hash = tcp_v6_hashfn(daddr, hnum, saddr, sport);
head = &tcp_ehash[hash];
- read_lock(&head->lock);
+ rcu_read_lock();
sk_for_each(sk, node, &head->chain) {
/* For IPV6 do the cheaper port and family tests first. */
if(TCP_IPV6_MATCH(sk, saddr, daddr, ports, dif))
@@ -326,12 +330,12 @@ static inline struct sock *__tcp_v6_look
goto hit;
}
}
- read_unlock(&head->lock);
+ rcu_read_unlock();
return NULL;
hit:
sock_hold(sk);
- read_unlock(&head->lock);
+ rcu_read_unlock();
return sk;
}
@@ -452,7 +456,7 @@ static int tcp_v6_check_established(stru
struct hlist_node *node;
struct tcp_tw_bucket *tw;
- write_lock_bh(&head->lock);
+ spin_lock_bh(&head->lock);
/* Check TIME-WAIT sockets first. */
sk_for_each(sk2, node, &(head + tcp_ehash_size)->chain) {
@@ -491,7 +495,7 @@ unique:
__sk_add_node(sk, &head->chain);
sk->sk_hashent = hash;
sock_prot_inc_use(sk->sk_prot);
- write_unlock_bh(&head->lock);
+ spin_unlock_bh(&head->lock);
if (tw) {
/* Silly. Should hash-dance instead... */
_
--
Thanks and Regards,
Srivatsa Vaddagiri,
Linux Technology Center,
IBM Software Labs,
Bangalore, INDIA - 560017
^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: [RFC] Use RCU for tcp_ehash lookup
2004-08-31 12:59 [RFC] Use RCU for tcp_ehash lookup Srivatsa Vaddagiri
@ 2004-08-31 13:04 ` Srivatsa Vaddagiri
2004-08-31 13:12 ` Srivatsa Vaddagiri
` (2 subsequent siblings)
3 siblings, 0 replies; 11+ messages in thread
From: Srivatsa Vaddagiri @ 2004-08-31 13:04 UTC (permalink / raw)
To: davem; +Cc: netdev, linux-kernel, Dipankar, paulmck
On Tue, Aug 31, 2004 at 06:29:41PM +0530, Srivatsa Vaddagiri wrote:
> I found _significant_ reduction in profile count (~56% savings)
> for __tcp_v4_lookup_established if it is made lock-free.
I had made __tcp_v4_lookup_established to be a function call (rather
than an inline function) to gather readprofile statistics for the same.
--
Thanks and Regards,
Srivatsa Vaddagiri,
Linux Technology Center,
IBM Software Labs,
Bangalore, INDIA - 560017
^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: [RFC] Use RCU for tcp_ehash lookup
2004-08-31 12:59 [RFC] Use RCU for tcp_ehash lookup Srivatsa Vaddagiri
2004-08-31 13:04 ` Srivatsa Vaddagiri
@ 2004-08-31 13:12 ` Srivatsa Vaddagiri
2004-08-31 13:54 ` Andi Kleen
2004-09-02 5:41 ` David S. Miller
3 siblings, 0 replies; 11+ messages in thread
From: Srivatsa Vaddagiri @ 2004-08-31 13:12 UTC (permalink / raw)
To: davem; +Cc: netdev, linux-kernel, Dipankar, paulmck
[-- Attachment #1: Type: text/plain, Size: 313 bytes --]
On Tue, Aug 31, 2004 at 06:29:41PM +0530, Srivatsa Vaddagiri wrote:
> I have attached rplog for both lock-free as well as read-lock cases.
Sorry, forgot to attach them last time. Here they are.
--
Thanks and Regards,
Srivatsa Vaddagiri,
Linux Technology Center,
IBM Software Labs,
Bangalore, INDIA - 560017
[-- Attachment #2: rplog-norcu --]
[-- Type: text/plain, Size: 239239 bytes --]
25 default_idle 0.5556
428835 poll_idle 7393.7069
41 cpu_idle 0.6406
7 need_resched 0.1400
130 system_call 2.9545
2 syscall_call 0.1818
7 syscall_exit 0.6364
2 handle_IRQ_event 0.0200
1 sys_ipc 0.0016
1 show_cpuinfo 0.0015
51 sched_clock 0.3750
2 flush_tlb_page 0.0113
4 pte_alloc_one 0.0519
21 do_page_fault 0.0150
1723 finish_task_switch 11.5638
1 sched_migrate_task 0.0050
1057 __wake_up 8.9576
207 __might_sleep 0.9810
78 add_wait_queue 0.6290
77 remove_wait_queue 0.6016
41 prepare_to_wait 0.2789
51 finish_wait 0.3228
1 mm_init 0.0040
1 copy_mm 0.0009
1 unshare_files 0.0102
1 copy_process 0.0003
7 .text.lock.fork 0.0354
1 sys_gettimeofday 0.0079
125 current_kernel_time 1.8116
5035 __do_softirq 29.1040
167 local_bh_enable 1.1597
807 __mod_timer 1.5609
89 mod_timer 0.8396
330 del_timer 2.1429
2 .text.lock.timer 0.0217
1 flush_signal_handlers 0.0149
2 in_group_p 0.0167
59 add_to_page_cache 0.2408
12 page_waitqueue 0.2500
41 wake_up_page 0.5325
16 unlock_page 0.4324
13 end_page_writeback 0.1757
50 find_get_page 0.4464
30 find_lock_page 0.1034
44 find_get_pages_tag 0.2767
5 filemap_nopage 0.0053
8 remove_suid 0.0537
141 generic_file_aio_write_nolock 0.0487
20 generic_file_aio_write 0.1266
51 mempool_alloc 0.1339
18 mempool_free 0.0857
1 mempool_alloc_slab 0.0476
2 mempool_free_slab 0.0952
15 bad_range 0.1500
19 prep_new_page 0.2346
40 free_hot_cold_page 0.1342
1 free_hot_page 0.1429
142 buffered_rmqueue 0.2669
85 __alloc_pages 0.0957
8 __get_free_pages 0.1356
20 __free_pages 0.2564
5 free_pages 0.1111
1 __read_page_state 0.0179
9 balance_dirty_pages_ratelimited 0.0448
78 __set_page_dirty_nobuffers 0.2074
3 set_page_dirty 0.0357
15 clear_page_dirty_for_io 0.1316
36 test_clear_page_writeback 0.1412
44 test_set_page_writeback 0.1467
35 .text.lock.page_writeback 0.2734
1 prio_tree_insert 0.0026
1 prio_tree_remove 0.0044
1 vma_prio_tree_add 0.0060
2 do_page_cache_readahead 0.0043
1 kmem_getpages 0.0048
9 cache_init_objs 0.0865
1 kmem_flagcheck 0.0233
1 cache_grow 0.0022
139 kmem_cache_alloc 1.1301
2 kmem_ptr_validate 0.0263
77 __kmalloc 0.5000
183 kmem_cache_free 2.1034
236 kfree 2.1651
32 mark_page_accessed 0.6154
36 release_pages 0.0796
33 __pagevec_lru_add 0.1041
2 __pagevec_lru_add_active 0.0060
19 percpu_counter_mod 0.1086
3 blk_queue_bounce 0.0390
1 clear_page_tables 0.0032
1 pte_alloc_map 0.0040
11 copy_page_range 0.0142
21 zap_pte_range 0.0303
13 do_wp_page 0.0153
12 do_anonymous_page 0.0267
286 do_no_page 0.3210
1 remove_vm_struct 0.0062
1 find_vma_prepare 0.0099
1 vma_adjust 0.0010
1 do_mmap_pgoff 0.0005
2 find_vma 0.0247
1 find_vma_prev 0.0123
1 do_munmap 0.0022
1 exit_mmap 0.0022
2 anon_vma_unlink 0.0128
4 page_add_anon_rmap 0.0122
1 page_add_file_rmap 0.0046
7 page_remove_rmap 0.0293
2 put_unused_fd 0.0161
1 fd_install 0.0068
10 do_sync_read 0.0578
85 vfs_read 0.2881
31 do_sync_write 0.1792
67 vfs_write 0.2271
32 sys_read 0.2500
28 sys_write 0.2188
54 fput 2.5714
1 __fput 0.0047
85 fget 0.6641
43 fget_light 0.2575
1 file_move 0.0071
4 bh_waitq_head 0.1818
58 wake_up_buffer 0.6988
1 sync_buffer 0.0149
38 unlock_buffer 1.5833
16 __set_page_buffers 0.6400
77 end_buffer_write_sync 0.6364
23 __find_get_block_slow 0.0531
31 end_buffer_async_write 0.1003
1 buffer_insert_list 0.0067
23 create_buffers 0.1411
1 init_page_buffers 0.0089
8 mark_buffer_dirty 0.2000
48 __brelse 0.6154
1 bh_lru_install 0.0044
152 __find_get_block 0.6179
14 __getblk 0.1386
11 __bread 0.1667
10 set_bh_page 0.1852
20 create_empty_buffers 0.0990
5 unmap_underlying_metadata 0.0442
37 __block_write_full_page 0.0385
56 __block_prepare_write 0.0564
25 __block_commit_write 0.1678
12 block_prepare_write 0.1558
11 generic_commit_write 0.0743
10 block_write_full_page 0.0375
1 generic_block_bmap 0.0119
18 end_bio_bh_io_sync 0.2903
38 submit_bh 0.1166
48 ll_rw_block 0.3529
4 recalc_bh_state 0.0223
16 alloc_buffer_head 0.1524
33 init_buffer_head 0.6226
1 init_buffer 0.0526
3 .text.lock.buffer 0.0161
3 bio_destructor 0.0319
15 bio_alloc 0.0352
50 bio_put 0.9259
3 bio_endio 0.0236
1 sync_supers 0.0036
16 inode_add_bytes 0.1127
66 pipe_wait 0.3687
253 pipe_readv 0.3579
10 pipe_read 0.1695
134 pipe_writev 0.1616
4 pipe_write 0.0678
172 pipe_poll 1.3871
1 getname 0.0041
2 do_lookup 0.0124
3 link_path_walk 0.0008
1 path_lookup 0.0021
1 may_open 0.0019
1 do_fcntl 0.0021
1 sys_fcntl64 0.0067
10 kill_fasync 0.0862
4 poll_initwait 0.1600
31 poll_freewait 0.4844
33 __pollwait 0.1675
20 max_select_fd 0.0889
167 do_select 0.2136
4 select_bits_alloc 0.1250
2 select_bits_free 0.4000
135 sys_select 0.1047
1 d_instantiate 0.0066
3 __d_lookup 0.0072
165 inode_times_differ 2.4265
51 update_atime 0.2297
45 inode_update_time 0.2103
2 wake_up_inode 0.0278
223 dnotify_parent 0.8814
1 seq_printf 0.0116
84 __mark_inode_dirty 0.1736
1 writeback_acquire 0.0455
1 writeback_release 0.0208
8 mpage_writepages 0.0095
1 load_elf_interp 0.0016
1 get_vmalloc_info 0.0052
6 show_stat 0.0026
9 write_profile 0.0577
27 ext3_get_group_desc 0.1436
10 read_block_bitmap 0.0658
3 find_next_usable_block 0.0035
52 ext3_try_to_allocate 0.0584
8 ext3_has_free_blocks 0.0650
56 ext3_new_block 0.0431
24 ext3_file_write 0.1127
6 ext3_alloc_block 0.1017
19 ext3_block_to_path 0.0597
42 ext3_get_branch 0.1780
27 ext3_find_goal 0.1971
6 ext3_alloc_branch 0.0087
38 ext3_splice_branch 0.0802
83 ext3_get_block_handle 0.0934
11 ext3_get_block 0.0632
76 walk_page_buffers 0.6786
25 ext3_prepare_write 0.0786
6 ext3_journal_dirty_data 0.0645
98 ext3_ordered_commit_write 0.4279
11 bget_one 1.0000
4 journal_dirty_data_fn 0.2105
10 ext3_ordered_writepage 0.0217
62 ext3_get_inode_block 0.2375
31 ext3_get_inode_loc 0.0487
82 ext3_do_update_inode 0.0888
33 ext3_writepage_trans_blocks 0.2324
33 ext3_mark_iloc_dirty 0.6000
15 ext3_reserve_inode_write 0.0711
18 ext3_mark_inode_dirty 0.2222
27 ext3_dirty_inode 0.2000
1 ext3_find_entry 0.0009
52 ext3_journal_start 0.5843
50 __ext3_journal_stop 0.6250
116 start_this_handle 0.0905
12 new_handle 0.1500
22 journal_start 0.1048
166 do_get_write_access 0.0929
23 journal_get_write_access 0.2840
32 journal_get_undo_access 0.0771
65 journal_dirty_data 0.0889
137 journal_dirty_metadata 0.2495
86 journal_stop 0.1057
97 __journal_unfile_buffer 0.1876
42 __journal_file_buffer 0.0599
16 inverted_lock 0.1667
195 journal_commit_transaction 0.0319
1 __journal_clean_checkpoint_list 0.0057
52 journal_cancel_revoke 0.1499
1 journal_switch_revoke_table 0.0109
7 __log_space_left 0.0680
1 __log_start_commit 0.0159
11 journal_blocks_per_page 0.3793
8 journal_alloc_journal_head 0.0690
6 journal_free_journal_head 0.2143
274 journal_add_journal_head 0.7486
49 __journal_remove_journal_head 0.1231
54 journal_remove_journal_head 0.4355
81 journal_put_journal_head 0.3632
1 copy_semundo 0.0042
1 __bitmap_weight 0.0035
7 radix_tree_preload 0.0395
1 rb_insert_color 0.0044
1 __rb_erase_color 0.0024
8 number 0.0102
6 vsnprintf 0.0049
2 find_next_zero_bit 0.0114
30 csum_partial 0.1042
1 atomic_dec_and_lock 0.0086
19 memcpy 0.3333
6176 __copy_user_intel 35.9070
4424 __copy_user_zeroing_intel 25.7209
90 __copy_to_user_ll 0.7500
2340 __copy_from_user_ll 19.5000
24 copy_to_user 0.2449
31 copy_from_user 0.2460
1 add_disk_randomness 0.0192
3 blk_run_queue 0.0242
5 get_request 0.0067
106 __make_request 0.0763
8 generic_make_request 0.0163
5 submit_bio 0.0173
50 __end_that_request_first 0.0951
2 put_io_context 0.0227
5 .text.lock.ll_rw_blk 0.0368
1 as_set_request 0.0076
299 e1000_xmit_frame 0.1405
5 scsi_put_command 0.0221
15 scsi_dispatch_cmd 0.0278
5 scsi_add_timer 0.0463
3 scsi_decide_disposition 0.0100
3 scsi_device_unbusy 0.0163
1 scsi_run_queue 0.0036
21 scsi_end_request 0.0795
3 scsi_io_completion 0.0029
109 scsi_request_fn 0.1005
12 .text.lock.scsi_lib 0.0561
34 ahc_linux_isr 0.0489
3 sd_rw_intr 0.0045
38 sock_aio_read 0.1293
12 sock_aio_write 0.0408
22 sock_poll 0.4490
1 sys_socketcall 0.0017
192 sock_wfree 2.5263
12 __release_sock 0.0779
385 sock_def_readable 2.1389
10 sk_reset_timer 0.2128
90 lock_sock 0.7258
46 release_sock 0.3026
29 sock_common_recvmsg 0.2479
188 .text.lock.sock 1.5161
213 alloc_skb 0.9509
396 skb_release_data 2.2759
12 kfree_skbmem 0.2727
652 __kfree_skb 2.8596
84 skb_clone 0.2188
25 copy_skb_header 0.1008
21 skb_copy 0.0946
279 skb_copy_bits 0.5304
4 skb_checksum 0.0065
17 memcpy_toiovec 0.1789
1 memcpy_tokerneliovec 0.0133
134 skb_copy_datagram_iovec 0.2472
43 sk_stream_rfree 1.5926
1 sk_stream_error 0.0108
10 skb_checksum_help 0.0309
87 dev_queue_xmit 0.1093
37 net_tx_action 0.0961
77 netif_receive_skb 0.1730
128 process_backlog 0.4689
43 net_rx_action 0.1660
12 neigh_resolve_output 0.0228
190 nf_iterate 1.1515
93 nf_hook_slow 0.3252
2 eth_header 0.0064
173 qdisc_restart 0.2932
152 pfifo_fast_enqueue 1.0483
115 pfifo_fast_dequeue 1.4198
29 rt_hash_code 0.2117
260 ip_route_input 0.4779
30 ip_local_deliver 0.0593
150 ip_rcv 0.1174
69 ip_local_deliver_finish 0.1769
33 ip_rcv_finish 0.0509
17 ip_finish_output 0.0320
27 ip_output 0.2250
355 ip_queue_xmit 0.2354
8 dst_output 0.1951
219 ip_finish_output2 0.4944
138 tcp_poll 0.3594
177 tcp_sendmsg 0.0389
44 cleanup_rbuf 0.1429
414 tcp_recvmsg 0.2062
1 tcp_incr_quickack 0.0130
33 __tcp_grow_window 0.2519
48 tcp_rcv_rtt_update 0.5275
36 tcp_rcv_space_adjust 0.1104
258 tcp_event_data_recv 0.4314
39 tcp_rtt_estimator 0.1161
1 tcp_try_to_open 0.0020
7 tcp_ack_saw_tstamp 0.0833
63 tcp_clean_rtx_queue 0.0662
27 tcp_ack 0.0184
1 tcp_ofo_queue 0.0016
6 __tcp_data_snd_check 0.0254
56 __tcp_ack_snd_check 0.3478
1 tcp_check_urg 0.0036
788 tcp_rcv_established 0.3411
240 tcp_transmit_skb 0.1378
51 tcp_write_xmit 0.0703
47 __tcp_select_window 0.1582
23 tcp_send_delayed_ack 0.1041
82 tcp_send_ack 0.4205
507 __tcp_v4_lookup_established 1.9205
34 tcp_v4_send_check 0.1344
1 tcp_v4_send_reset 0.0028
52 tcp_v4_do_rcv 0.1635
456 tcp_v4_rcv 0.2316
1 inet_sock_destruct 0.0018
11 inet_sendmsg 0.1236
19 ip_confirm 0.3958
34 ip_conntrack_defrag 0.4857
21 ip_refrag 0.1469
31 ip_conntrack_local 0.2123
25 __ip_ct_find_proto 0.3846
214 ip_ct_find_proto 2.0000
62 hash_conntrack 0.4026
42 get_tuple 0.4516
2 invert_tuple 0.0426
82 __ip_conntrack_find 0.3744
159 ip_conntrack_find_get 1.3140
12 ip_conntrack_get 0.3333
129 ip_conntrack_in 0.2098
285 ip_ct_refresh 1.5000
2 ip_ct_gather_frags 0.0104
30 tcp_pkt_to_tuple 0.3529
12 get_conntrack_index 0.1967
453 tcp_packet 0.9577
875 ipt_do_table 0.9030
8 ipt_hook 0.1356
18 ipt_local_out_hook 0.1538
37 ipt_route_hook 0.6271
19 ipt_local_hook 0.0815
55 ip_nat_fn 0.0914
17 ip_nat_out 0.0988
201 do_bindings 0.2466
1 unix_create1 0.0023
5 __sched_text_start 0.0172
1 __down_failed 0.0833
3 __write_lock_failed 0.0938
1 __read_lock_failed 0.0500
249 schedule 0.1197
2 __preempt_spin_lock 0.0220
382 .text.lock.sched 1.9490
5 schedule_timeout 0.0262
473008 total 0.1538
28 default_idle 0.6222
443439 poll_idle 7645.5000
69 cpu_idle 1.0781
1 __up 0.0333
9 need_resched 0.1800
133 system_call 3.0227
3 syscall_call 0.2727
9 syscall_exit 0.8182
2 handle_IRQ_event 0.0200
2 old_mmap 0.0060
1 sys_ipc 0.0016
2 show_cpuinfo 0.0030
1 get_offset_tsc 0.0435
56 sched_clock 0.4118
1 flush_tlb_page 0.0056
5 pte_alloc_one 0.0649
2 pgd_ctor 0.0119
19 do_page_fault 0.0136
1583 finish_task_switch 10.6242
1 sched_migrate_task 0.0050
1049 __wake_up 8.8898
198 __might_sleep 0.9384
81 add_wait_queue 0.6532
55 remove_wait_queue 0.4297
35 prepare_to_wait 0.2381
47 finish_wait 0.2975
1 dup_task_struct 0.0047
1 mm_init 0.0040
1 copy_mm 0.0009
1 copy_process 0.0003
6 .text.lock.fork 0.0303
1 exit_notify 0.0004
1 sys_wait4 0.0016
592 current_kernel_time 8.5797
5361 __do_softirq 30.9884
198 local_bh_enable 1.3750
674 __mod_timer 1.3037
86 mod_timer 0.8113
321 del_timer 2.0844
1 del_timer_sync 0.0055
3 .text.lock.timer 0.0326
1 sigprocmask 0.0041
1 groups_search 0.0094
1 rcu_do_batch 0.0323
55 add_to_page_cache 0.2245
14 page_waitqueue 0.2917
41 wake_up_page 0.5325
11 unlock_page 0.2973
6 end_page_writeback 0.0811
45 find_get_page 0.4018
29 find_lock_page 0.1000
1 find_get_pages 0.0078
46 find_get_pages_tag 0.2893
1 do_generic_mapping_read 0.0008
1 file_read_actor 0.0041
1 __generic_file_aio_read 0.0018
5 filemap_nopage 0.0053
8 remove_suid 0.0537
131 generic_file_aio_write_nolock 0.0452
21 generic_file_aio_write 0.1329
34 mempool_alloc 0.0892
24 mempool_free 0.1143
1 mempool_free_slab 0.0476
20 bad_range 0.2000
18 prep_new_page 0.2222
37 free_hot_cold_page 0.1242
167 buffered_rmqueue 0.3139
89 __alloc_pages 0.1002
9 __get_free_pages 0.1525
20 __free_pages 0.2564
2 free_pages 0.0444
1 __read_page_state 0.0179
10 balance_dirty_pages_ratelimited 0.0498
63 __set_page_dirty_nobuffers 0.1676
3 set_page_dirty 0.0357
8 clear_page_dirty_for_io 0.0702
40 test_clear_page_writeback 0.1569
52 test_set_page_writeback 0.1733
1 .text.lock.page_writeback 0.0078
1 pdflush_operation 0.0042
1 file_ra_state_init 0.0270
1 do_page_cache_readahead 0.0022
1 page_cache_readahead 0.0018
1 kmem_getpages 0.0048
1 alloc_slabmgmt 0.0104
5 cache_init_objs 0.0481
1 set_slab_attr 0.0172
4 cache_grow 0.0088
174 kmem_cache_alloc 1.4146
4 kmem_ptr_validate 0.0526
75 __kmalloc 0.4870
196 kmem_cache_free 2.2529
223 kfree 2.0459
21 mark_page_accessed 0.4038
1 lru_cache_add_active 0.0105
32 release_pages 0.0708
29 __pagevec_lru_add 0.0915
3 __pagevec_lru_add_active 0.0089
8 percpu_counter_mod 0.0457
1 clear_page_tables 0.0032
2 pte_alloc_map 0.0081
12 copy_page_range 0.0155
24 zap_pte_range 0.0346
22 do_wp_page 0.0259
37 do_anonymous_page 0.0824
278 do_no_page 0.3120
5 handle_mm_fault 0.0114
3 remove_vm_struct 0.0185
1 __vma_link 0.0087
1 vma_merge 0.0020
2 do_mmap_pgoff 0.0011
2 get_unmapped_area 0.0067
2 find_vma 0.0247
2 do_munmap 0.0044
1 sys_munmap 0.0082
1 exit_mmap 0.0022
1 anon_vma_prepare 0.0028
1 page_add_anon_rmap 0.0031
2 page_add_file_rmap 0.0093
9 page_remove_rmap 0.0377
1 free_pages_and_swap_cache 0.0076
1 filp_open 0.0095
2 dentry_open 0.0051
13 do_sync_read 0.0751
52 vfs_read 0.1763
23 do_sync_write 0.1329
52 vfs_write 0.1763
34 sys_read 0.2656
12 sys_write 0.0938
35 fput 1.6667
80 fget 0.6250
35 fget_light 0.2096
1 file_move 0.0071
7 bh_waitq_head 0.3182
69 wake_up_buffer 0.8313
32 unlock_buffer 1.3333
1 __wait_on_buffer 0.0061
10 __set_page_buffers 0.4000
1 __clear_page_buffers 0.0141
95 end_buffer_write_sync 0.7851
17 __find_get_block_slow 0.0393
1 buffer_insert_list 0.0067
15 create_buffers 0.0920
13 mark_buffer_dirty 0.3250
52 __brelse 0.6667
1 bh_lru_install 0.0044
145 __find_get_block 0.5894
20 __getblk 0.1980
16 __bread 0.2424
7 set_bh_page 0.1296
18 create_empty_buffers 0.0891
5 unmap_underlying_metadata 0.0442
46 __block_write_full_page 0.0479
63 __block_prepare_write 0.0634
28 __block_commit_write 0.1879
6 block_prepare_write 0.0779
23 generic_commit_write 0.1554
22 block_write_full_page 0.0824
27 end_bio_bh_io_sync 0.4355
32 submit_bh 0.0982
41 ll_rw_block 0.3015
2 recalc_bh_state 0.0112
8 alloc_buffer_head 0.0762
31 init_buffer_head 0.5849
9 bio_destructor 0.0957
23 bio_alloc 0.0540
48 bio_put 0.8889
3 bio_endio 0.0236
1 vfs_getattr 0.0065
1 cp_new_stat64 0.0037
8 inode_add_bytes 0.0563
1 copy_strings 0.0018
1 compute_creds 0.0053
84 pipe_wait 0.4693
212 pipe_readv 0.2999
9 pipe_read 0.1525
117 pipe_writev 0.1411
6 pipe_write 0.1017
146 pipe_poll 1.1774
1 vfs_permission 0.0032
2 do_lookup 0.0124
5 link_path_walk 0.0013
3 path_lookup 0.0064
1 __lookup_hash 0.0047
2 open_namei 0.0011
13 kill_fasync 0.1121
28 poll_freewait 0.4375
27 __pollwait 0.1371
23 max_select_fd 0.1022
143 do_select 0.1829
1 select_bits_alloc 0.0312
2 select_bits_free 0.4000
106 sys_select 0.0822
1 dput 0.0016
5 __d_lookup 0.0120
1 iput 0.0081
130 inode_times_differ 1.9118
34 update_atime 0.1532
52 inode_update_time 0.2430
1 wake_up_inode 0.0139
203 dnotify_parent 0.8024
1 lookup_mnt 0.0060
2 seq_printf 0.0233
103 __mark_inode_dirty 0.2128
3 writeback_inodes 0.0075
1 writeback_acquire 0.0455
9 mpage_writepages 0.0107
1 proc_lookup 0.0030
2 show_stat 0.0009
14 write_profile 0.0897
23 ext3_get_group_desc 0.1223
8 read_block_bitmap 0.0526
3 find_next_usable_block 0.0035
50 ext3_try_to_allocate 0.0562
9 ext3_has_free_blocks 0.0732
59 ext3_new_block 0.0454
20 ext3_file_write 0.0939
4 ext3_alloc_block 0.0678
21 ext3_block_to_path 0.0660
38 ext3_get_branch 0.1610
25 ext3_find_goal 0.1825
15 ext3_alloc_branch 0.0217
40 ext3_splice_branch 0.0844
69 ext3_get_block_handle 0.0776
11 ext3_get_block 0.0632
76 walk_page_buffers 0.6786
30 ext3_prepare_write 0.0943
11 ext3_journal_dirty_data 0.1183
112 ext3_ordered_commit_write 0.4891
13 bget_one 1.1818
7 journal_dirty_data_fn 0.3684
20 ext3_ordered_writepage 0.0434
82 ext3_get_inode_block 0.3142
37 ext3_get_inode_loc 0.0582
112 ext3_do_update_inode 0.1213
46 ext3_writepage_trans_blocks 0.3239
28 ext3_mark_iloc_dirty 0.5091
34 ext3_reserve_inode_write 0.1611
14 ext3_mark_inode_dirty 0.1728
32 ext3_dirty_inode 0.2370
19 ext3_journal_start 0.2135
18 __ext3_journal_stop 0.2250
73 start_this_handle 0.0569
12 new_handle 0.1500
18 journal_start 0.0857
166 do_get_write_access 0.0929
20 journal_get_write_access 0.2469
27 journal_get_undo_access 0.0651
74 journal_dirty_data 0.1012
134 journal_dirty_metadata 0.2441
91 journal_stop 0.1118
101 __journal_unfile_buffer 0.1954
53 __journal_file_buffer 0.0756
1 journal_end_buffer_io_sync 0.0323
35 inverted_lock 0.3646
253 journal_commit_transaction 0.0414
2 journal_brelse_array 0.0455
51 journal_cancel_revoke 0.1470
1 journal_write_revoke_records 0.0039
13 __log_space_left 0.1262
1 journal_bmap 0.0059
5 journal_blocks_per_page 0.1724
5 journal_alloc_journal_head 0.0431
2 journal_free_journal_head 0.0714
271 journal_add_journal_head 0.7404
57 __journal_remove_journal_head 0.1432
69 journal_remove_journal_head 0.5565
96 journal_put_journal_head 0.4305
1 ipc_lock 0.0062
1 kobject_put 0.0400
13 radix_tree_preload 0.0734
1 number 0.0013
9 vsnprintf 0.0073
1 find_next_zero_bit 0.0057
35 csum_partial 0.1215
2 atomic_dec_and_lock 0.0172
13 memcpy 0.2281
1 strncpy_from_user 0.0101
5950 __copy_user_intel 34.5930
4272 __copy_user_zeroing_intel 24.8372
97 __copy_to_user_ll 0.8083
2251 __copy_from_user_ll 18.7583
25 copy_to_user 0.2551
28 copy_from_user 0.2222
1 add_timer_randomness 0.0034
1 add_disk_randomness 0.0192
1 generic_unplug_device 0.0100
4 blk_run_queue 0.0323
5 get_request 0.0067
101 __make_request 0.0727
19 generic_make_request 0.0386
10 submit_bio 0.0346
66 __end_that_request_first 0.1255
4 put_io_context 0.0455
1 get_io_context 0.0080
14 .text.lock.ll_rw_blk 0.1029
5 as_set_request 0.0382
326 e1000_xmit_frame 0.1532
14 scsi_put_command 0.0619
37 scsi_dispatch_cmd 0.0686
6 scsi_finish_command 0.0335
5 scsi_add_timer 0.0463
1 scsi_decide_disposition 0.0033
3 scsi_init_cmd_errh 0.0156
15 scsi_device_unbusy 0.0815
6 scsi_run_queue 0.0218
31 scsi_end_request 0.1174
4 scsi_io_completion 0.0039
172 scsi_request_fn 0.1585
11 .text.lock.scsi_lib 0.0514
67 ahc_linux_isr 0.0964
3 sd_rw_intr 0.0045
1 sock_alloc 0.0079
26 sock_aio_read 0.0884
22 sock_aio_write 0.0748
16 sock_poll 0.3265
235 sock_wfree 3.0921
16 __release_sock 0.1039
342 sock_def_readable 1.9000
16 sk_reset_timer 0.3404
80 lock_sock 0.6452
45 release_sock 0.2961
35 sock_common_recvmsg 0.2991
179 .text.lock.sock 1.4435
246 alloc_skb 1.0982
384 skb_release_data 2.2069
17 kfree_skbmem 0.3864
627 __kfree_skb 2.7500
86 skb_clone 0.2240
27 copy_skb_header 0.1089
26 skb_copy 0.1171
269 skb_copy_bits 0.5114
2 skb_checksum 0.0032
23 memcpy_toiovec 0.2421
3 memcpy_tokerneliovec 0.0400
120 skb_copy_datagram_iovec 0.2214
27 sk_stream_rfree 1.0000
1 sk_stream_error 0.0108
13 skb_checksum_help 0.0401
66 dev_queue_xmit 0.0829
24 net_tx_action 0.0623
59 netif_receive_skb 0.1326
120 process_backlog 0.4396
66 net_rx_action 0.2548
188 nf_iterate 1.1394
91 nf_hook_slow 0.3182
151 qdisc_restart 0.2559
141 pfifo_fast_enqueue 0.9724
94 pfifo_fast_dequeue 1.1605
30 rt_hash_code 0.2190
253 ip_route_input 0.4651
37 ip_local_deliver 0.0731
171 ip_rcv 0.1338
56 ip_local_deliver_finish 0.1436
39 ip_rcv_finish 0.0602
21 ip_finish_output 0.0395
18 ip_output 0.1500
361 ip_queue_xmit 0.2394
14 dst_output 0.3415
235 ip_finish_output2 0.5305
126 tcp_poll 0.3281
169 tcp_sendmsg 0.0372
43 cleanup_rbuf 0.1396
422 tcp_recvmsg 0.2102
28 __tcp_grow_window 0.2137
50 tcp_rcv_rtt_update 0.5495
64 tcp_rcv_space_adjust 0.1963
239 tcp_event_data_recv 0.3997
23 tcp_rtt_estimator 0.0685
8 tcp_ack_saw_tstamp 0.0952
57 tcp_clean_rtx_queue 0.0599
36 tcp_ack 0.0246
3 tcp_data_queue 0.0010
1 tcp_cwnd_application_limited 0.0067
7 __tcp_data_snd_check 0.0297
68 __tcp_ack_snd_check 0.4224
1 tcp_urg 0.0041
797 tcp_rcv_established 0.3450
264 tcp_transmit_skb 0.1515
49 tcp_write_xmit 0.0676
23 __tcp_select_window 0.0774
29 tcp_send_delayed_ack 0.1312
87 tcp_send_ack 0.4462
514 __tcp_v4_lookup_established 1.9470
26 tcp_v4_send_check 0.1028
64 tcp_v4_do_rcv 0.2013
403 tcp_v4_rcv 0.2047
6 inet_sendmsg 0.0674
35 ip_confirm 0.7292
28 ip_conntrack_defrag 0.4000
16 ip_refrag 0.1119
17 ip_conntrack_local 0.1164
21 __ip_ct_find_proto 0.3231
241 ip_ct_find_proto 2.2523
61 hash_conntrack 0.3961
39 get_tuple 0.4194
2 invert_tuple 0.0426
76 __ip_conntrack_find 0.3470
129 ip_conntrack_find_get 1.0661
9 ip_conntrack_get 0.2500
1 init_conntrack 0.0010
128 ip_conntrack_in 0.2081
307 ip_ct_refresh 1.6158
1 ip_ct_gather_frags 0.0052
20 tcp_pkt_to_tuple 0.2353
8 get_conntrack_index 0.1311
461 tcp_packet 0.9746
964 ipt_do_table 0.9948
18 ipt_hook 0.3051
15 ipt_local_out_hook 0.1282
34 ipt_route_hook 0.5763
26 ipt_local_hook 0.1116
73 ip_nat_fn 0.1213
20 ip_nat_out 0.1163
213 do_bindings 0.2613
1 unix_release_sock 0.0010
12 __sched_text_start 0.0414
2 __read_lock_failed 0.1000
241 schedule 0.1158
2 __preempt_spin_lock 0.0220
370 .text.lock.sched 1.8878
2 schedule_timeout 0.0105
487741 total 0.1585
23 default_idle 0.5111
416561 poll_idle 7182.0862
35 cpu_idle 0.5469
1 sys_set_thread_area 0.0020
6 need_resched 0.1200
119 system_call 2.7045
5 syscall_call 0.4545
11 syscall_exit 1.0000
3 handle_IRQ_event 0.0300
3 show_cpuinfo 0.0044
29 sched_clock 0.2132
1 flush_tlb_page 0.0056
3 pte_alloc_one 0.0390
1 pgd_ctor 0.0060
15 do_page_fault 0.0107
1691 finish_task_switch 11.3490
1 nr_context_switches 0.0070
1006 __wake_up 8.5254
204 __might_sleep 0.9668
88 add_wait_queue 0.7097
54 remove_wait_queue 0.4219
26 prepare_to_wait 0.1769
37 finish_wait 0.2342
1 mm_alloc 0.0154
1 copy_mm 0.0009
1 copy_files 0.0010
1 copy_process 0.0003
8 .text.lock.fork 0.0404
2 release_task 0.0037
1 sys_waitpid 0.0233
102 current_kernel_time 1.4783
5158 __do_softirq 29.8150
167 local_bh_enable 1.1597
596 __mod_timer 1.1528
106 mod_timer 1.0000
307 del_timer 1.9935
2 .text.lock.timer 0.0217
1 get_signal_to_deliver 0.0010
1 do_sigaction 0.0016
1 wait_on_page_writeback_range 0.0031
52 add_to_page_cache 0.2122
13 page_waitqueue 0.2708
35 wake_up_page 0.4545
20 unlock_page 0.5405
19 end_page_writeback 0.2568
54 find_get_page 0.4821
33 find_lock_page 0.1138
45 find_get_pages_tag 0.2830
1 do_generic_mapping_read 0.0008
2 filemap_nopage 0.0021
7 remove_suid 0.0470
139 generic_file_aio_write_nolock 0.0480
16 generic_file_aio_write 0.1013
48 mempool_alloc 0.1260
20 mempool_free 0.0952
2 mempool_alloc_slab 0.0952
2 mempool_free_slab 0.0952
27 bad_range 0.2700
12 prep_new_page 0.1481
45 free_hot_cold_page 0.1510
152 buffered_rmqueue 0.2857
84 __alloc_pages 0.0946
6 __get_free_pages 0.1017
1 __pagevec_free 0.0303
14 __free_pages 0.1795
1 free_pages 0.0222
9 balance_dirty_pages_ratelimited 0.0448
1 do_writepages 0.0149
54 __set_page_dirty_nobuffers 0.1436
4 set_page_dirty 0.0476
12 clear_page_dirty_for_io 0.1053
51 test_clear_page_writeback 0.2000
64 test_set_page_writeback 0.2133
21 .text.lock.page_writeback 0.1641
1 pdflush_operation 0.0042
1 prio_tree_insert 0.0026
1 vma_prio_tree_remove 0.0039
2 do_page_cache_readahead 0.0043
13 cache_init_objs 0.1250
2 kmem_flagcheck 0.0465
5 cache_grow 0.0110
161 kmem_cache_alloc 1.3089
2 kmem_ptr_validate 0.0263
74 __kmalloc 0.4805
197 kmem_cache_free 2.2644
191 kfree 1.7523
28 mark_page_accessed 0.5385
1 lru_cache_add_active 0.0105
30 release_pages 0.0664
33 __pagevec_lru_add 0.1041
3 __pagevec_lru_add_active 0.0089
1 pagevec_lookup_tag 0.0156
12 percpu_counter_mod 0.0686
2 blk_queue_bounce 0.0260
4 clear_page_tables 0.0130
2 pte_alloc_map 0.0081
12 copy_page_range 0.0155
24 zap_pte_range 0.0346
2 unmap_vmas 0.0035
19 do_wp_page 0.0224
27 do_anonymous_page 0.0601
388 do_no_page 0.4355
6 handle_mm_fault 0.0137
1 __remove_shared_vm_struct 0.0101
1 remove_vm_struct 0.0062
1 __insert_vm_struct 0.0076
1 vma_adjust 0.0010
4 do_mmap_pgoff 0.0022
5 find_vma 0.0617
1 do_munmap 0.0022
1 change_protection 0.0022
1 anon_vma_prepare 0.0028
1 anon_vma_link 0.0079
1 page_add_anon_rmap 0.0031
1 page_add_file_rmap 0.0046
11 page_remove_rmap 0.0460
2 free_pages_and_swap_cache 0.0153
1 dentry_open 0.0025
1 get_unused_fd 0.0039
1 sys_open 0.0060
9 do_sync_read 0.0520
67 vfs_read 0.2271
21 do_sync_write 0.1214
56 vfs_write 0.1898
24 sys_read 0.1875
24 sys_write 0.1875
1 get_empty_filp 0.0043
51 fput 2.4286
83 fget 0.6484
44 fget_light 0.2635
2 file_move 0.0142
5 bh_waitq_head 0.2273
76 wake_up_buffer 0.9157
39 unlock_buffer 1.6250
17 __set_page_buffers 0.6800
70 end_buffer_write_sync 0.5785
16 __find_get_block_slow 0.0370
22 end_buffer_async_write 0.0712
1 buffer_insert_list 0.0067
24 create_buffers 0.1472
1 grow_dev_page 0.0025
7 mark_buffer_dirty 0.1750
43 __brelse 0.5513
2 bh_lru_install 0.0088
157 __find_get_block 0.6382
15 __getblk 0.1485
10 __bread 0.1515
10 set_bh_page 0.1852
1 try_to_release_page 0.0091
25 create_empty_buffers 0.1238
9 unmap_underlying_metadata 0.0796
46 __block_write_full_page 0.0479
53 __block_prepare_write 0.0534
14 __block_commit_write 0.0940
9 block_prepare_write 0.1169
9 generic_commit_write 0.0608
23 block_write_full_page 0.0861
31 end_bio_bh_io_sync 0.5000
41 submit_bh 0.1258
22 ll_rw_block 0.1618
3 recalc_bh_state 0.0168
15 alloc_buffer_head 0.1429
30 init_buffer_head 0.5660
5 .text.lock.buffer 0.0269
7 bio_destructor 0.0745
21 bio_alloc 0.0493
59 bio_put 1.0926
4 bio_endio 0.0315
1 chrdev_open 0.0017
2 cp_new_stat64 0.0074
1 sys_fstat64 0.0175
16 inode_add_bytes 0.1127
1 copy_strings 0.0018
63 pipe_wait 0.3520
211 pipe_readv 0.2984
17 pipe_read 0.2881
177 pipe_writev 0.2135
4 pipe_write 0.0678
1 bad_pipe_r 0.1667
113 pipe_poll 0.9113
1 vfs_permission 0.0032
1 deny_write_access 0.0072
1 follow_mount 0.0068
2 do_lookup 0.0124
7 link_path_walk 0.0018
2 path_lookup 0.0042
2 may_open 0.0039
1 open_namei 0.0006
19 kill_fasync 0.1638
3 poll_initwait 0.1200
31 poll_freewait 0.4844
28 __pollwait 0.1421
20 max_select_fd 0.0889
150 do_select 0.1918
6 select_bits_alloc 0.1875
1 select_bits_free 0.2000
119 sys_select 0.0922
1 locks_remove_posix 0.0030
3 dput 0.0049
7 __d_lookup 0.0168
108 inode_times_differ 1.5882
28 update_atime 0.1261
37 inode_update_time 0.1729
1 wake_up_inode 0.0139
193 dnotify_parent 0.7628
1 seq_read 0.0014
1 seq_printf 0.0116
49 __mark_inode_dirty 0.1012
1 sync_sb_inodes 0.0013
1 writeback_inodes 0.0025
11 mpage_writepages 0.0131
1 load_elf_interp 0.0016
1 proc_alloc_inode 0.0085
2 proc_lookup 0.0060
4 show_stat 0.0018
10 write_profile 0.0641
27 ext3_get_group_desc 0.1436
11 read_block_bitmap 0.0724
1 find_next_usable_block 0.0012
75 ext3_try_to_allocate 0.0843
7 ext3_has_free_blocks 0.0569
64 ext3_new_block 0.0493
25 ext3_file_write 0.1174
6 ext3_alloc_block 0.1017
20 ext3_block_to_path 0.0629
24 ext3_get_branch 0.1017
16 ext3_find_goal 0.1168
18 ext3_alloc_branch 0.0260
38 ext3_splice_branch 0.0802
82 ext3_get_block_handle 0.0922
17 ext3_get_block 0.0977
68 walk_page_buffers 0.6071
22 ext3_prepare_write 0.0692
3 ext3_journal_dirty_data 0.0323
99 ext3_ordered_commit_write 0.4323
14 bget_one 1.2727
3 bput_one 0.2727
2 journal_dirty_data_fn 0.1053
9 ext3_ordered_writepage 0.0195
47 ext3_get_inode_block 0.1801
37 ext3_get_inode_loc 0.0582
93 ext3_do_update_inode 0.1008
26 ext3_writepage_trans_blocks 0.1831
33 ext3_mark_iloc_dirty 0.6000
27 ext3_reserve_inode_write 0.1280
20 ext3_mark_inode_dirty 0.2469
18 ext3_dirty_inode 0.1333
21 ext3_journal_start 0.2360
19 __ext3_journal_stop 0.2375
58 start_this_handle 0.0452
22 new_handle 0.2750
33 journal_start 0.1571
175 do_get_write_access 0.0980
24 journal_get_write_access 0.2963
31 journal_get_undo_access 0.0747
68 journal_dirty_data 0.0930
131 journal_dirty_metadata 0.2386
89 journal_stop 0.1093
93 __journal_unfile_buffer 0.1799
44 __journal_file_buffer 0.0628
1 journal_file_buffer 0.0046
26 inverted_lock 0.2708
237 journal_commit_transaction 0.0388
2 journal_brelse_array 0.0455
1 __journal_clean_checkpoint_list 0.0057
60 journal_cancel_revoke 0.1729
1 journal_write_revoke_records 0.0039
8 __log_space_left 0.0777
9 journal_blocks_per_page 0.3103
6 journal_alloc_journal_head 0.0517
4 journal_free_journal_head 0.1429
268 journal_add_journal_head 0.7322
55 __journal_remove_journal_head 0.1382
58 journal_remove_journal_head 0.4677
104 journal_put_journal_head 0.4664
1 ipc_lock 0.0062
11 radix_tree_preload 0.0621
1 rb_insert_color 0.0044
4 vsnprintf 0.0032
1 find_next_zero_bit 0.0057
25 csum_partial 0.0868
2 atomic_dec_and_lock 0.0172
19 memcpy 0.3333
3 strncpy_from_user 0.0303
6130 __copy_user_intel 35.6395
4451 __copy_user_zeroing_intel 25.8779
86 __copy_to_user_ll 0.7167
2375 __copy_from_user_ll 19.7917
38 copy_to_user 0.3878
17 copy_from_user 0.1349
2 add_disk_randomness 0.0385
1 put_device 0.1000
1 elv_set_request 0.0172
2 blk_run_queue 0.0161
2 get_request 0.0027
115 __make_request 0.0827
3 generic_make_request 0.0061
11 submit_bio 0.0381
52 __end_that_request_first 0.0989
1 put_io_context 0.0114
10 .text.lock.ll_rw_blk 0.0735
1 as_set_request 0.0076
327 e1000_xmit_frame 0.1537
8 scsi_put_command 0.0354
16 scsi_dispatch_cmd 0.0297
2 scsi_softirq 0.0096
2 scsi_finish_command 0.0112
2 scsi_add_timer 0.0185
2 scsi_init_cmd_errh 0.0104
5 scsi_device_unbusy 0.0272
2 scsi_run_queue 0.0073
17 scsi_end_request 0.0644
2 scsi_free_sgtable 0.0444
1 scsi_io_completion 0.0010
122 scsi_request_fn 0.1124
12 .text.lock.scsi_lib 0.0561
37 ahc_linux_isr 0.0532
1 .text.lock.aic7xxx_osm 0.0014
3 sd_rw_intr 0.0045
18 sock_aio_read 0.0612
9 sock_aio_write 0.0306
17 sock_poll 0.3469
1 __sock_create 0.0027
180 sock_wfree 2.3684
19 __release_sock 0.1234
297 sock_def_readable 1.6500
8 sk_reset_timer 0.1702
73 lock_sock 0.5887
48 release_sock 0.3158
41 sock_common_recvmsg 0.3504
163 .text.lock.sock 1.3145
239 alloc_skb 1.0670
397 skb_release_data 2.2816
16 kfree_skbmem 0.3636
637 __kfree_skb 2.7939
66 skb_clone 0.1719
34 copy_skb_header 0.1371
21 skb_copy 0.0946
310 skb_copy_bits 0.5894
1 skb_checksum 0.0016
15 memcpy_toiovec 0.1579
1 memcpy_tokerneliovec 0.0133
122 skb_copy_datagram_iovec 0.2251
37 sk_stream_rfree 1.3704
4 sk_stream_error 0.0430
33 skb_checksum_help 0.1019
88 dev_queue_xmit 0.1106
27 net_tx_action 0.0701
62 netif_receive_skb 0.1393
101 process_backlog 0.3700
58 net_rx_action 0.2239
194 nf_iterate 1.1758
89 nf_hook_slow 0.3112
141 qdisc_restart 0.2390
144 pfifo_fast_enqueue 0.9931
106 pfifo_fast_dequeue 1.3086
19 rt_hash_code 0.1387
261 ip_route_input 0.4798
45 ip_local_deliver 0.0889
145 ip_rcv 0.1135
75 ip_local_deliver_finish 0.1923
42 ip_rcv_finish 0.0648
22 ip_finish_output 0.0414
19 ip_output 0.1583
373 ip_queue_xmit 0.2473
9 dst_output 0.2195
234 ip_finish_output2 0.5282
116 tcp_poll 0.3021
173 tcp_sendmsg 0.0381
48 cleanup_rbuf 0.1558
414 tcp_recvmsg 0.2062
162 __tcp_grow_window 1.2366
109 tcp_rcv_rtt_update 1.1978
99 tcp_rcv_space_adjust 0.3037
403 tcp_event_data_recv 0.6739
29 tcp_rtt_estimator 0.0863
1 tcp_try_to_open 0.0020
10 tcp_ack_saw_tstamp 0.1190
72 tcp_clean_rtx_queue 0.0757
39 tcp_ack 0.0266
1 tcp_ofo_queue 0.0016
2 tcp_data_queue 0.0007
11 __tcp_data_snd_check 0.0466
49 __tcp_ack_snd_check 0.3043
1 tcp_urg 0.0041
824 tcp_rcv_established 0.3567
257 tcp_transmit_skb 0.1475
51 tcp_write_xmit 0.0703
31 __tcp_select_window 0.1044
20 tcp_send_delayed_ack 0.0905
64 tcp_send_ack 0.3282
458 __tcp_v4_lookup_established 1.7348
22 tcp_v4_send_check 0.0870
50 tcp_v4_do_rcv 0.1572
430 tcp_v4_rcv 0.2184
3 inet_sendmsg 0.0337
22 ip_confirm 0.4583
31 ip_conntrack_defrag 0.4429
13 ip_refrag 0.0909
16 ip_conntrack_local 0.1096
18 __ip_ct_find_proto 0.2769
224 ip_ct_find_proto 2.0935
71 hash_conntrack 0.4610
30 get_tuple 0.3226
2 invert_tuple 0.0426
82 __ip_conntrack_find 0.3744
163 ip_conntrack_find_get 1.3471
18 ip_conntrack_get 0.5000
112 ip_conntrack_in 0.1821
296 ip_ct_refresh 1.5579
4 ip_ct_gather_frags 0.0207
31 tcp_pkt_to_tuple 0.3647
9 get_conntrack_index 0.1475
460 tcp_packet 0.9725
908 ipt_do_table 0.9370
13 ipt_hook 0.2203
17 ipt_local_out_hook 0.1453
30 ipt_route_hook 0.5085
29 ipt_local_hook 0.1245
63 ip_nat_fn 0.1047
22 ip_nat_out 0.1279
186 do_bindings 0.2282
1 unix_create1 0.0023
4 __sched_text_start 0.0138
1 __down_failed 0.0833
1 __write_lock_failed 0.0312
2 __read_lock_failed 0.1000
236 schedule 0.1134
3 __preempt_spin_lock 0.0330
387 .text.lock.sched 1.9745
1 schedule_timeout 0.0052
460551 total 0.1497
21 default_idle 0.4667
415641 poll_idle 7166.2241
43 cpu_idle 0.6719
4 need_resched 0.0800
137 system_call 3.1136
4 syscall_call 0.3636
6 syscall_exit 0.5455
1 error_code 0.0179
5 handle_IRQ_event 0.0500
1 sys_mmap2 0.0047
1 sys_ipc 0.0016
1 show_cpuinfo 0.0015
34 sched_clock 0.2500
1 flush_tlb_others 0.0033
1 flush_tlb_page 0.0056
3 pte_alloc_one 0.0390
25 do_page_fault 0.0179
1 try_to_wake_up 0.0015
1701 finish_task_switch 11.4161
1 nr_context_switches 0.0070
1032 __wake_up 8.7458
227 __might_sleep 1.0758
104 add_wait_queue 0.8387
63 remove_wait_queue 0.4922
36 prepare_to_wait 0.2449
42 finish_wait 0.2658
2 copy_mm 0.0017
1 copy_files 0.0010
1 copy_process 0.0003
1 do_fork 0.0022
4 .text.lock.fork 0.0202
1 release_task 0.0018
1 sys_wait4 0.0016
95 current_kernel_time 1.3768
5236 __do_softirq 30.2659
159 local_bh_enable 1.1042
593 __mod_timer 1.1470
106 mod_timer 1.0000
325 del_timer 2.1104
1 del_timer_sync 0.0055
48 add_to_page_cache 0.1959
7 page_waitqueue 0.1458
49 wake_up_page 0.6364
20 unlock_page 0.5405
14 end_page_writeback 0.1892
43 find_get_page 0.3839
37 find_lock_page 0.1276
49 find_get_pages_tag 0.3082
2 do_generic_mapping_read 0.0016
9 filemap_nopage 0.0095
10 remove_suid 0.0671
116 generic_file_aio_write_nolock 0.0400
21 generic_file_aio_write 0.1329
51 mempool_alloc 0.1339
12 mempool_free 0.0571
2 mempool_alloc_slab 0.0952
2 mempool_free_slab 0.0952
23 bad_range 0.2300
8 prep_new_page 0.0988
42 free_hot_cold_page 0.1409
1 free_hot_page 0.1429
190 buffered_rmqueue 0.3571
66 __alloc_pages 0.0743
7 __get_free_pages 0.1186
1 get_zeroed_page 0.0098
15 __free_pages 0.1923
2 free_pages 0.0444
1 nr_free_pages 0.0159
1 get_writeback_state 0.0128
1 balance_dirty_pages 0.0032
7 balance_dirty_pages_ratelimited 0.0348
63 __set_page_dirty_nobuffers 0.1676
4 set_page_dirty 0.0476
18 clear_page_dirty_for_io 0.1579
45 test_clear_page_writeback 0.1765
61 test_set_page_writeback 0.2033
14 .text.lock.page_writeback 0.1094
1 __pdflush 0.0018
1 prio_tree_replace 0.0101
2 prio_tree_insert 0.0052
1 prio_tree_remove 0.0044
1 vma_prio_tree_remove 0.0039
1 page_cache_readahead 0.0018
2 kmem_getpages 0.0096
2 alloc_slabmgmt 0.0208
6 cache_init_objs 0.0577
1 kmem_flagcheck 0.0233
6 cache_grow 0.0132
160 kmem_cache_alloc 1.3008
4 kmem_ptr_validate 0.0526
80 __kmalloc 0.5195
193 kmem_cache_free 2.2184
211 kfree 1.9358
31 mark_page_accessed 0.5962
1 lru_cache_add_active 0.0105
32 release_pages 0.0708
29 __pagevec_lru_add 0.0915
2 __pagevec_lru_add_active 0.0060
1 pagevec_lookup_tag 0.0156
14 percpu_counter_mod 0.0800
1 blk_queue_bounce 0.0130
1 clear_page_tables 0.0032
7 copy_page_range 0.0090
21 zap_pte_range 0.0303
2 unmap_vmas 0.0035
16 do_wp_page 0.0189
15 do_anonymous_page 0.0334
282 do_no_page 0.3165
5 handle_mm_fault 0.0114
1 vma_link 0.0030
1 __insert_vm_struct 0.0076
1 vma_merge 0.0020
2 find_vma 0.0247
1 unmap_region 0.0035
1 change_protection 0.0022
2 anon_vma_unlink 0.0128
3 page_add_anon_rmap 0.0092
1 page_add_file_rmap 0.0046
12 page_remove_rmap 0.0502
1 dentry_open 0.0025
1 get_unused_fd 0.0039
2 filp_close 0.0149
10 do_sync_read 0.0578
74 vfs_read 0.2508
16 do_sync_write 0.0925
58 vfs_write 0.1966
27 sys_read 0.2109
30 sys_write 0.2344
30 fput 1.4286
3 __fput 0.0142
100 fget 0.7812
44 fget_light 0.2635
4 bh_waitq_head 0.1818
70 wake_up_buffer 0.8434
34 unlock_buffer 1.4167
17 __set_page_buffers 0.6800
86 end_buffer_write_sync 0.7107
23 __find_get_block_slow 0.0531
17 end_buffer_async_write 0.0550
1 buffer_insert_list 0.0067
17 create_buffers 0.1043
1 grow_dev_page 0.0025
6 mark_buffer_dirty 0.1500
36 __brelse 0.4615
1 bh_lru_install 0.0044
150 __find_get_block 0.6098
13 __getblk 0.1287
14 __bread 0.2121
4 set_bh_page 0.0741
1 try_to_release_page 0.0091
26 create_empty_buffers 0.1287
4 unmap_underlying_metadata 0.0354
46 __block_write_full_page 0.0479
63 __block_prepare_write 0.0634
26 __block_commit_write 0.1745
9 block_prepare_write 0.1169
5 generic_commit_write 0.0338
13 block_write_full_page 0.0487
24 end_bio_bh_io_sync 0.3871
36 submit_bh 0.1104
28 ll_rw_block 0.2059
7 recalc_bh_state 0.0391
10 alloc_buffer_head 0.0952
26 init_buffer_head 0.4906
4 .text.lock.buffer 0.0215
11 bio_destructor 0.1170
16 bio_alloc 0.0376
50 bio_put 0.9259
7 bio_endio 0.0551
1 cp_new_stat64 0.0037
13 inode_add_bytes 0.0915
1 copy_strings 0.0018
1 do_execve 0.0015
46 pipe_wait 0.2570
232 pipe_readv 0.3281
5 pipe_read 0.0847
147 pipe_writev 0.1773
4 pipe_write 0.0678
127 pipe_poll 1.0242
1 getname 0.0041
1 vfs_permission 0.0032
1 follow_mount 0.0068
1 do_lookup 0.0062
6 link_path_walk 0.0015
1 open_namei 0.0006
6 kill_fasync 0.0517
1 poll_initwait 0.0400
32 poll_freewait 0.5000
36 __pollwait 0.1827
19 max_select_fd 0.0844
169 do_select 0.2161
10 select_bits_alloc 0.3125
3 select_bits_free 0.6000
113 sys_select 0.0876
1 dput 0.0016
1 d_lookup 0.0125
8 __d_lookup 0.0192
1 d_rehash 0.0046
131 inode_times_differ 1.9265
16 update_atime 0.0721
37 inode_update_time 0.1729
199 dnotify_parent 0.7866
60 __mark_inode_dirty 0.1240
14 mpage_writepages 0.0167
1 proc_lookup 0.0030
1 meminfo_read_proc 0.0020
9 show_stat 0.0040
8 write_profile 0.0513
30 ext3_get_group_desc 0.1596
14 read_block_bitmap 0.0921
2 find_next_usable_block 0.0024
63 ext3_try_to_allocate 0.0708
5 ext3_has_free_blocks 0.0407
55 ext3_new_block 0.0423
1 ext3_open_file 0.0222
21 ext3_file_write 0.0986
4 ext3_alloc_block 0.0678
24 ext3_block_to_path 0.0755
30 ext3_get_branch 0.1271
24 ext3_find_goal 0.1752
15 ext3_alloc_branch 0.0217
35 ext3_splice_branch 0.0738
63 ext3_get_block_handle 0.0709
11 ext3_get_block 0.0632
67 walk_page_buffers 0.5982
1 do_journal_get_write_access 0.0083
15 ext3_prepare_write 0.0472
4 ext3_journal_dirty_data 0.0430
94 ext3_ordered_commit_write 0.4105
15 bget_one 1.3636
1 bput_one 0.0909
7 journal_dirty_data_fn 0.3684
16 ext3_ordered_writepage 0.0347
62 ext3_get_inode_block 0.2375
27 ext3_get_inode_loc 0.0425
89 ext3_do_update_inode 0.0964
36 ext3_writepage_trans_blocks 0.2535
30 ext3_mark_iloc_dirty 0.5455
21 ext3_reserve_inode_write 0.0995
16 ext3_mark_inode_dirty 0.1975
20 ext3_dirty_inode 0.1481
22 ext3_journal_start 0.2472
7 __ext3_journal_stop 0.0875
64 start_this_handle 0.0499
16 new_handle 0.2000
19 journal_start 0.0905
187 do_get_write_access 0.1047
21 journal_get_write_access 0.2593
21 journal_get_undo_access 0.0506
70 journal_dirty_data 0.0958
129 journal_dirty_metadata 0.2350
116 journal_stop 0.1425
97 __journal_unfile_buffer 0.1876
34 __journal_file_buffer 0.0485
22 inverted_lock 0.2292
241 journal_commit_transaction 0.0395
1 journal_brelse_array 0.0227
67 journal_cancel_revoke 0.1931
1 journal_write_revoke_records 0.0039
6 __log_space_left 0.0583
10 journal_blocks_per_page 0.3448
5 journal_alloc_journal_head 0.0431
1 journal_free_journal_head 0.0357
243 journal_add_journal_head 0.6639
57 __journal_remove_journal_head 0.1432
48 journal_remove_journal_head 0.3871
83 journal_put_journal_head 0.3722
1 copy_semundo 0.0042
9 radix_tree_preload 0.0508
1 __rb_rotate_left 0.0179
1 skip_atoi 0.0154
5 number 0.0064
5 vsnprintf 0.0041
1 find_next_zero_bit 0.0057
33 csum_partial 0.1146
28 memcpy 0.4912
2 strncpy_from_user 0.0202
5942 __copy_user_intel 34.5465
4433 __copy_user_zeroing_intel 25.7733
106 __copy_to_user_ll 0.8833
2211 __copy_from_user_ll 18.4250
19 copy_to_user 0.1939
31 copy_from_user 0.2460
2 add_disk_randomness 0.0385
2 blk_run_queue 0.0161
3 get_request 0.0040
1 attempt_merge 0.0030
94 __make_request 0.0676
11 generic_make_request 0.0224
9 submit_bio 0.0311
59 __end_that_request_first 0.1122
2 put_io_context 0.0227
1 get_io_context 0.0080
10 .text.lock.ll_rw_blk 0.0735
2 as_set_request 0.0153
300 e1000_xmit_frame 0.1410
7 scsi_put_command 0.0310
36 scsi_dispatch_cmd 0.0668
2 scsi_finish_command 0.0112
2 scsi_add_timer 0.0185
2 scsi_decide_disposition 0.0067
8 scsi_init_cmd_errh 0.0417
10 scsi_device_unbusy 0.0543
2 scsi_run_queue 0.0073
37 scsi_end_request 0.1402
2 scsi_free_sgtable 0.0444
3 scsi_io_completion 0.0029
163 scsi_request_fn 0.1502
16 .text.lock.scsi_lib 0.0748
52 ahc_linux_isr 0.0748
4 sd_rw_intr 0.0060
24 sock_aio_read 0.0816
10 sock_aio_write 0.0340
14 sock_poll 0.2857
192 sock_wfree 2.5263
6 __release_sock 0.0390
309 sock_def_readable 1.7167
4 sk_reset_timer 0.0851
75 lock_sock 0.6048
48 release_sock 0.3158
38 sock_common_recvmsg 0.3248
175 .text.lock.sock 1.4113
225 alloc_skb 1.0045
367 skb_release_data 2.1092
13 kfree_skbmem 0.2955
620 __kfree_skb 2.7193
80 skb_clone 0.2083
34 copy_skb_header 0.1371
16 skb_copy 0.0721
284 skb_copy_bits 0.5399
10 skb_checksum 0.0162
23 memcpy_toiovec 0.2421
2 memcpy_tokerneliovec 0.0267
125 skb_copy_datagram_iovec 0.2306
31 sk_stream_rfree 1.1481
2 sk_stream_error 0.0215
19 skb_checksum_help 0.0586
99 dev_queue_xmit 0.1244
25 net_tx_action 0.0649
63 netif_receive_skb 0.1416
109 process_backlog 0.3993
52 net_rx_action 0.2008
204 nf_iterate 1.2364
110 nf_hook_slow 0.3846
157 qdisc_restart 0.2661
123 pfifo_fast_enqueue 0.8483
106 pfifo_fast_dequeue 1.3086
38 rt_hash_code 0.2774
1 rt_cache_get_first 0.0119
262 ip_route_input 0.4816
55 ip_local_deliver 0.1087
144 ip_rcv 0.1127
65 ip_local_deliver_finish 0.1667
46 ip_rcv_finish 0.0710
24 ip_finish_output 0.0451
16 ip_output 0.1333
358 ip_queue_xmit 0.2374
5 dst_output 0.1220
262 ip_finish_output2 0.5914
128 tcp_poll 0.3333
173 tcp_sendmsg 0.0381
54 cleanup_rbuf 0.1753
462 tcp_recvmsg 0.2301
30 __tcp_grow_window 0.2290
61 tcp_rcv_rtt_update 0.6703
30 tcp_rcv_space_adjust 0.0920
219 tcp_event_data_recv 0.3662
30 tcp_rtt_estimator 0.0893
11 tcp_ack_saw_tstamp 0.1310
58 tcp_clean_rtx_queue 0.0610
41 tcp_ack 0.0280
2 tcp_data_queue 0.0007
8 __tcp_data_snd_check 0.0339
63 __tcp_ack_snd_check 0.3913
788 tcp_rcv_established 0.3411
263 tcp_transmit_skb 0.1510
54 tcp_write_xmit 0.0745
32 __tcp_select_window 0.1077
19 tcp_send_delayed_ack 0.0860
76 tcp_send_ack 0.3897
493 __tcp_v4_lookup_established 1.8674
29 tcp_v4_send_check 0.1146
49 tcp_v4_do_rcv 0.1541
407 tcp_v4_rcv 0.2067
15 inet_sendmsg 0.1685
16 ip_confirm 0.3333
25 ip_conntrack_defrag 0.3571
19 ip_refrag 0.1329
16 ip_conntrack_local 0.1096
19 __ip_ct_find_proto 0.2923
230 ip_ct_find_proto 2.1495
67 hash_conntrack 0.4351
37 get_tuple 0.3978
70 __ip_conntrack_find 0.3196
157 ip_conntrack_find_get 1.2975
6 ip_conntrack_get 0.1667
133 ip_conntrack_in 0.2163
293 ip_ct_refresh 1.5421
26 tcp_pkt_to_tuple 0.3059
15 get_conntrack_index 0.2459
485 tcp_packet 1.0254
899 ipt_do_table 0.9278
22 ipt_hook 0.3729
5 ipt_local_out_hook 0.0427
30 ipt_route_hook 0.5085
26 ipt_local_hook 0.1116
42 ip_nat_fn 0.0698
19 ip_nat_out 0.1105
208 do_bindings 0.2552
1 unix_mkname 0.0066
10 __sched_text_start 0.0345
2 __write_lock_failed 0.0625
3 __read_lock_failed 0.1500
239 schedule 0.1148
3 __preempt_spin_lock 0.0330
381 .text.lock.sched 1.9439
2 schedule_timeout 0.0105
458994 total 0.1492
15 default_idle 0.3333
418731 poll_idle 7219.5000
60 cpu_idle 0.9375
10 need_resched 0.2000
140 system_call 3.1818
5 syscall_call 0.4545
19 syscall_exit 1.7273
4 handle_IRQ_event 0.0400
1 show_cpuinfo 0.0015
38 sched_clock 0.2794
1 flush_tlb_page 0.0056
4 pte_alloc_one 0.0519
12 do_page_fault 0.0086
1703 finish_task_switch 11.4295
987 __wake_up 8.3644
2 migration_thread 0.0044
215 __might_sleep 1.0190
84 add_wait_queue 0.6774
48 remove_wait_queue 0.3750
21 prepare_to_wait 0.1429
36 finish_wait 0.2278
5 .text.lock.fork 0.0253
1 profile_exec_unmap 0.0104
121 current_kernel_time 1.7536
5225 __do_softirq 30.2023
151 local_bh_enable 1.0486
640 __mod_timer 1.2379
110 mod_timer 1.0377
339 del_timer 2.2013
1 del_timer_sync 0.0055
2 .text.lock.timer 0.0217
1 sigprocmask 0.0041
1 do_sigaction 0.0016
1 do_futex 0.0064
49 add_to_page_cache 0.2000
12 page_waitqueue 0.2500
44 wake_up_page 0.5714
20 unlock_page 0.5405
4 end_page_writeback 0.0541
40 find_get_page 0.3571
26 find_lock_page 0.0897
46 find_get_pages_tag 0.2893
1 do_generic_mapping_read 0.0008
1 __generic_file_aio_read 0.0018
1 remove_suid 0.0067
122 generic_file_aio_write_nolock 0.0421
20 generic_file_aio_write 0.1266
45 mempool_alloc 0.1181
13 mempool_free 0.0619
1 mempool_alloc_slab 0.0476
1 mempool_free_slab 0.0476
24 bad_range 0.2400
16 prep_new_page 0.1975
62 free_hot_cold_page 0.2081
151 buffered_rmqueue 0.2838
86 __alloc_pages 0.0968
10 __get_free_pages 0.1695
20 __free_pages 0.2564
3 free_pages 0.0667
1 balance_dirty_pages 0.0032
5 balance_dirty_pages_ratelimited 0.0249
56 __set_page_dirty_nobuffers 0.1489
2 set_page_dirty 0.0238
6 clear_page_dirty_for_io 0.0526
43 test_clear_page_writeback 0.1686
48 test_set_page_writeback 0.1600
1 vma_prio_tree_insert 0.0130
1 vma_prio_tree_remove 0.0039
1 do_page_cache_readahead 0.0022
1 kmem_getpages 0.0048
13 cache_init_objs 0.1250
1 cache_grow 0.0022
166 kmem_cache_alloc 1.3496
3 kmem_ptr_validate 0.0395
78 __kmalloc 0.5065
193 kmem_cache_free 2.2184
232 kfree 2.1284
33 mark_page_accessed 0.6346
1 lru_cache_add_active 0.0105
22 release_pages 0.0487
29 __pagevec_lru_add 0.0915
2 __pagevec_lru_add_active 0.0060
16 percpu_counter_mod 0.0914
1 blk_queue_bounce 0.0130
3 clear_page_tables 0.0097
1 pte_alloc_map 0.0040
14 copy_page_range 0.0181
14 zap_pte_range 0.0202
2 zap_pmd_range 0.0174
1 unmap_vmas 0.0017
19 do_wp_page 0.0224
21 do_anonymous_page 0.0468
277 do_no_page 0.3109
2 handle_mm_fault 0.0046
1 __remove_shared_vm_struct 0.0101
1 remove_vm_struct 0.0062
1 find_vma_prepare 0.0099
1 __vma_link 0.0087
1 vma_link 0.0030
2 find_mergeable_anon_vma 0.0085
3 do_mmap_pgoff 0.0016
1 get_unmapped_area 0.0034
3 find_vma 0.0370
1 free_pgtables 0.0067
1 split_vma 0.0035
1 change_protection 0.0022
1 __anon_vma_link 0.0189
1 anon_vma_unlink 0.0064
6 page_add_anon_rmap 0.0183
14 page_remove_rmap 0.0586
2 free_pages_and_swap_cache 0.0153
1 sys_access 0.0030
1 dentry_open 0.0025
1 sys_close 0.0049
10 do_sync_read 0.0578
58 vfs_read 0.1966
23 do_sync_write 0.1329
56 vfs_write 0.1898
21 sys_read 0.1641
26 sys_write 0.2031
42 fput 2.0000
99 fget 0.7734
36 fget_light 0.2156
1 file_move 0.0071
1 file_kill 0.0081
7 bh_waitq_head 0.3182
71 wake_up_buffer 0.8554
37 unlock_buffer 1.5417
14 __set_page_buffers 0.5600
86 end_buffer_write_sync 0.7107
17 __find_get_block_slow 0.0393
1 end_buffer_async_write 0.0032
30 create_buffers 0.1840
5 mark_buffer_dirty 0.1250
47 __brelse 0.6026
168 __find_get_block 0.6829
18 __getblk 0.1782
18 __bread 0.2727
8 set_bh_page 0.1481
1 try_to_release_page 0.0091
17 create_empty_buffers 0.0842
3 unmap_underlying_metadata 0.0265
46 __block_write_full_page 0.0479
47 __block_prepare_write 0.0473
20 __block_commit_write 0.1342
11 block_prepare_write 0.1429
14 generic_commit_write 0.0946
1 end_buffer_read_nobh 0.0323
14 block_write_full_page 0.0524
25 end_bio_bh_io_sync 0.4032
27 submit_bh 0.0828
43 ll_rw_block 0.3162
4 recalc_bh_state 0.0223
12 alloc_buffer_head 0.1143
25 init_buffer_head 0.4717
4 bio_destructor 0.0426
19 bio_alloc 0.0446
56 bio_put 1.0370
7 bio_endio 0.0551
1 sys_fstat64 0.0175
16 inode_add_bytes 0.1127
1 copy_strings 0.0018
68 pipe_wait 0.3799
220 pipe_readv 0.3112
9 pipe_read 0.1525
127 pipe_writev 0.1532
6 pipe_write 0.1017
141 pipe_poll 1.1371
1 real_lookup 0.0038
1 follow_mount 0.0068
1 do_lookup 0.0062
4 link_path_walk 0.0010
1 may_open 0.0019
19 kill_fasync 0.1638
3 poll_initwait 0.1200
27 poll_freewait 0.4219
49 __pollwait 0.2487
23 max_select_fd 0.1022
162 do_select 0.2072
7 select_bits_alloc 0.2188
2 select_bits_free 0.4000
124 sys_select 0.0961
1 __posix_lock_file 0.0006
1 fcntl_setlk 0.0015
1 dput 0.0016
6 __d_lookup 0.0144
1 alloc_inode 0.0030
118 inode_times_differ 1.7353
26 update_atime 0.1171
45 inode_update_time 0.2103
179 dnotify_parent 0.7075
1 seq_printf 0.0116
64 __mark_inode_dirty 0.1322
1 writeback_inodes 0.0025
14 mpage_writepages 0.0167
1 load_elf_binary 0.0003
1 proc_read_inode 0.0167
1 get_vmalloc_info 0.0052
11 show_stat 0.0048
11 write_profile 0.0705
27 ext3_get_group_desc 0.1436
7 read_block_bitmap 0.0461
2 find_next_usable_block 0.0024
66 ext3_try_to_allocate 0.0742
7 ext3_has_free_blocks 0.0569
57 ext3_new_block 0.0439
1 ext3_release_file 0.0098
16 ext3_file_write 0.0751
5 ext3_alloc_block 0.0847
21 ext3_block_to_path 0.0660
24 ext3_get_branch 0.1017
22 ext3_find_goal 0.1606
14 ext3_alloc_branch 0.0202
38 ext3_splice_branch 0.0802
66 ext3_get_block_handle 0.0742
10 ext3_get_block 0.0575
61 walk_page_buffers 0.5446
1 do_journal_get_write_access 0.0083
31 ext3_prepare_write 0.0975
3 ext3_journal_dirty_data 0.0323
84 ext3_ordered_commit_write 0.3668
11 bget_one 1.0000
1 bput_one 0.0909
3 journal_dirty_data_fn 0.1579
10 ext3_ordered_writepage 0.0217
72 ext3_get_inode_block 0.2759
24 ext3_get_inode_loc 0.0377
102 ext3_do_update_inode 0.1105
29 ext3_writepage_trans_blocks 0.2042
34 ext3_mark_iloc_dirty 0.6182
30 ext3_reserve_inode_write 0.1422
13 ext3_mark_inode_dirty 0.1605
26 ext3_dirty_inode 0.1926
20 ext3_journal_start 0.2247
14 __ext3_journal_stop 0.1750
1 ext3_follow_link 0.0357
76 start_this_handle 0.0593
20 new_handle 0.2500
25 journal_start 0.1190
199 do_get_write_access 0.1114
21 journal_get_write_access 0.2593
25 journal_get_undo_access 0.0602
75 journal_dirty_data 0.1026
101 journal_dirty_metadata 0.1840
93 journal_stop 0.1143
115 __journal_unfile_buffer 0.2224
55 __journal_file_buffer 0.0785
21 inverted_lock 0.2188
271 journal_commit_transaction 0.0444
2 journal_brelse_array 0.0455
1 __journal_clean_checkpoint_list 0.0057
1 __journal_drop_transaction 0.0011
1 find_revoke_record 0.0045
60 journal_cancel_revoke 0.1729
1 journal_write_metadata_buffer 0.0011
9 __log_space_left 0.0874
8 journal_blocks_per_page 0.2759
1 __jbd_kmalloc 0.0227
9 journal_alloc_journal_head 0.0776
3 journal_free_journal_head 0.1071
264 journal_add_journal_head 0.7213
54 __journal_remove_journal_head 0.1357
75 journal_remove_journal_head 0.6048
113 journal_put_journal_head 0.5067
2 ipc_lock 0.0124
9 radix_tree_preload 0.0508
2 number 0.0026
7 vsnprintf 0.0057
32 csum_partial 0.1111
1 atomic_dec_and_lock 0.0086
1 bad_get_user 0.1111
9 memcpy 0.1579
1 strncpy_from_user 0.0101
6117 __copy_user_intel 35.5640
4311 __copy_user_zeroing_intel 25.0640
90 __copy_to_user_ll 0.7500
2228 __copy_from_user_ll 18.5667
26 copy_to_user 0.2653
22 copy_from_user 0.1746
2 add_disk_randomness 0.0385
1 put_device 0.1000
1 elv_set_request 0.0172
1 generic_unplug_device 0.0100
6 blk_run_queue 0.0484
3 get_request 0.0040
95 __make_request 0.0683
12 generic_make_request 0.0244
8 submit_bio 0.0277
67 __end_that_request_first 0.1274
3 put_io_context 0.0341
2 get_io_context 0.0160
13 .text.lock.ll_rw_blk 0.0956
303 e1000_xmit_frame 0.1424
6 scsi_put_command 0.0265
13 scsi_dispatch_cmd 0.0241
2 scsi_softirq 0.0096
2 scsi_finish_command 0.0112
3 scsi_add_timer 0.0278
2 scsi_decide_disposition 0.0067
2 scsi_init_cmd_errh 0.0104
7 scsi_device_unbusy 0.0380
3 scsi_run_queue 0.0109
4 scsi_next_command 0.0976
20 scsi_end_request 0.0758
1 scsi_free_sgtable 0.0222
3 scsi_io_completion 0.0029
123 scsi_request_fn 0.1134
15 .text.lock.scsi_lib 0.0701
34 ahc_linux_isr 0.0489
2 sd_rw_intr 0.0030
2 vgacon_save_screen 0.0200
21 sock_aio_read 0.0714
18 sock_aio_write 0.0612
23 sock_poll 0.4694
221 sock_wfree 2.9079
14 __release_sock 0.0909
310 sock_def_readable 1.7222
14 sk_reset_timer 0.2979
73 lock_sock 0.5887
51 release_sock 0.3355
41 sock_common_recvmsg 0.3504
177 .text.lock.sock 1.4274
241 alloc_skb 1.0759
401 skb_release_data 2.3046
21 kfree_skbmem 0.4773
677 __kfree_skb 2.9693
68 skb_clone 0.1771
36 copy_skb_header 0.1452
26 skb_copy 0.1171
270 skb_copy_bits 0.5133
2 skb_checksum 0.0032
18 memcpy_toiovec 0.1895
136 skb_copy_datagram_iovec 0.2509
36 sk_stream_rfree 1.3333
4 sk_stream_error 0.0430
19 skb_checksum_help 0.0586
102 dev_queue_xmit 0.1281
26 net_tx_action 0.0675
73 netif_receive_skb 0.1640
123 process_backlog 0.4505
56 net_rx_action 0.2162
185 nf_iterate 1.1212
76 nf_hook_slow 0.2657
149 qdisc_restart 0.2525
174 pfifo_fast_enqueue 1.2000
122 pfifo_fast_dequeue 1.5062
29 rt_hash_code 0.2117
261 ip_route_input 0.4798
31 ip_local_deliver 0.0613
171 ip_rcv 0.1338
56 ip_local_deliver_finish 0.1436
48 ip_rcv_finish 0.0741
26 ip_finish_output 0.0489
29 ip_output 0.2417
399 ip_queue_xmit 0.2646
6 dst_output 0.1463
208 ip_finish_output2 0.4695
131 tcp_poll 0.3411
176 tcp_sendmsg 0.0387
59 cleanup_rbuf 0.1916
405 tcp_recvmsg 0.2017
43 __tcp_grow_window 0.3282
54 tcp_rcv_rtt_update 0.5934
46 tcp_rcv_space_adjust 0.1411
242 tcp_event_data_recv 0.4047
31 tcp_rtt_estimator 0.0923
8 tcp_ack_saw_tstamp 0.0952
61 tcp_clean_rtx_queue 0.0641
38 tcp_ack 0.0260
2 tcp_data_queue 0.0007
10 __tcp_data_snd_check 0.0424
64 __tcp_ack_snd_check 0.3975
1 tcp_urg 0.0041
780 tcp_rcv_established 0.3377
259 tcp_transmit_skb 0.1487
36 tcp_write_xmit 0.0497
38 __tcp_select_window 0.1279
21 tcp_send_delayed_ack 0.0950
68 tcp_send_ack 0.3487
468 __tcp_v4_lookup_established 1.7727
25 tcp_v4_send_check 0.0988
58 tcp_v4_do_rcv 0.1824
397 tcp_v4_rcv 0.2016
10 inet_sendmsg 0.1124
22 ip_confirm 0.4583
20 ip_conntrack_defrag 0.2857
25 ip_refrag 0.1748
16 ip_conntrack_local 0.1096
18 __ip_ct_find_proto 0.2769
194 ip_ct_find_proto 1.8131
54 hash_conntrack 0.3506
41 get_tuple 0.4409
2 invert_tuple 0.0426
74 __ip_conntrack_find 0.3379
155 ip_conntrack_find_get 1.2810
4 ip_conntrack_get 0.1111
125 ip_conntrack_in 0.2033
292 ip_ct_refresh 1.5368
3 ip_ct_gather_frags 0.0155
23 tcp_pkt_to_tuple 0.2706
10 get_conntrack_index 0.1639
437 tcp_packet 0.9239
893 ipt_do_table 0.9216
11 ipt_hook 0.1864
18 ipt_local_out_hook 0.1538
35 ipt_route_hook 0.5932
22 ipt_local_hook 0.0944
47 ip_nat_fn 0.0781
24 ip_nat_out 0.1395
1 ip_nat_setup_info 0.0011
215 do_bindings 0.2638
1 unix_sock_destructor 0.0041
3 __sched_text_start 0.0103
3 __write_lock_failed 0.0938
3 __read_lock_failed 0.1500
228 schedule 0.1096
1 __preempt_spin_lock 0.0110
425 .text.lock.sched 2.1684
3 schedule_timeout 0.0157
462215 total 0.1502
22 default_idle 0.4889
417273 poll_idle 7194.3621
53 cpu_idle 0.8281
7 need_resched 0.1400
132 system_call 3.0000
6 syscall_call 0.5455
8 syscall_exit 0.7273
3 handle_IRQ_event 0.0300
36 sched_clock 0.2647
7 pte_alloc_one 0.0909
3 pgd_ctor 0.0179
11 do_page_fault 0.0079
1 sched_fork 0.0060
1 wake_up_forked_process 0.0028
1644 finish_task_switch 11.0336
940 __wake_up 7.9661
239 __might_sleep 1.1327
90 add_wait_queue 0.7258
58 remove_wait_queue 0.4531
44 prepare_to_wait 0.2993
31 finish_wait 0.1962
1 dup_task_struct 0.0047
1 mm_init 0.0040
1 copy_mm 0.0009
2 copy_process 0.0007
9 .text.lock.fork 0.0455
1 profile_exit_task 0.0104
2 exit_notify 0.0009
116 current_kernel_time 1.6812
5143 __do_softirq 29.7283
149 local_bh_enable 1.0347
671 __mod_timer 1.2979
108 mod_timer 1.0189
357 del_timer 2.3182
1 do_sigaction 0.0016
1 sys_rt_sigaction 0.0052
1 groups_search 0.0094
1 in_group_p 0.0083
56 add_to_page_cache 0.2286
16 page_waitqueue 0.3333
52 wake_up_page 0.6753
10 unlock_page 0.2703
16 end_page_writeback 0.2162
60 find_get_page 0.5357
27 find_lock_page 0.0931
49 find_get_pages_tag 0.3082
5 filemap_nopage 0.0053
10 remove_suid 0.0671
175 generic_file_aio_write_nolock 0.0604
18 generic_file_aio_write 0.1139
41 mempool_alloc 0.1076
21 mempool_free 0.1000
2 mempool_alloc_slab 0.0952
2 mempool_free_slab 0.0952
20 bad_range 0.2000
18 prep_new_page 0.2222
53 free_hot_cold_page 0.1779
1 free_hot_page 0.1429
181 buffered_rmqueue 0.3402
83 __alloc_pages 0.0935
3 __get_free_pages 0.0508
17 __free_pages 0.2179
4 free_pages 0.0889
7 balance_dirty_pages_ratelimited 0.0348
54 __set_page_dirty_nobuffers 0.1436
3 set_page_dirty 0.0357
15 clear_page_dirty_for_io 0.1316
37 test_clear_page_writeback 0.1451
58 test_set_page_writeback 0.1933
9 .text.lock.page_writeback 0.0703
1 pdflush_operation 0.0042
1 file_ra_state_init 0.0270
2 do_page_cache_readahead 0.0043
1 page_cache_readahead 0.0018
1 kmem_getpages 0.0048
8 cache_init_objs 0.0769
2 kmem_flagcheck 0.0465
1 cache_grow 0.0022
187 kmem_cache_alloc 1.5203
3 kmem_ptr_validate 0.0395
71 __kmalloc 0.4610
220 kmem_cache_free 2.5287
215 kfree 1.9725
37 mark_page_accessed 0.7115
2 lru_cache_add_active 0.0211
1 lru_add_drain 0.0089
33 release_pages 0.0730
1 __pagevec_release 0.0185
35 __pagevec_lru_add 0.1104
5 __pagevec_lru_add_active 0.0149
9 percpu_counter_mod 0.0514
1 blk_queue_bounce 0.0130
2 clear_page_tables 0.0065
15 copy_page_range 0.0194
24 zap_pte_range 0.0346
1 zap_pmd_range 0.0087
2 unmap_vmas 0.0035
15 do_wp_page 0.0177
27 do_anonymous_page 0.0601
355 do_no_page 0.3984
2 handle_mm_fault 0.0046
1 find_vma_prepare 0.0099
1 find_vma 0.0123
2 unmap_region 0.0070
1 detach_vmas_to_be_unmapped 0.0101
1 split_vma 0.0035
1 do_munmap 0.0022
1 sys_mprotect 0.0018
3 anon_vma_prepare 0.0085
1 anon_vma_link 0.0079
2 page_add_anon_rmap 0.0061
3 page_add_file_rmap 0.0139
15 page_remove_rmap 0.0628
1 free_pages_and_swap_cache 0.0076
1 get_unused_fd 0.0039
10 do_sync_read 0.0578
77 vfs_read 0.2610
27 do_sync_write 0.1561
50 vfs_write 0.1695
21 sys_read 0.1641
30 sys_write 0.2344
1 get_empty_filp 0.0043
26 fput 1.2381
99 fget 0.7734
35 fget_light 0.2096
1 file_move 0.0071
2 bh_waitq_head 0.0909
80 wake_up_buffer 0.9639
31 unlock_buffer 1.2917
10 __set_page_buffers 0.4000
66 end_buffer_write_sync 0.5455
22 __find_get_block_slow 0.0508
25 end_buffer_async_write 0.0809
2 buffer_insert_list 0.0134
16 create_buffers 0.0982
11 mark_buffer_dirty 0.2750
60 __brelse 0.7692
1 bh_lru_install 0.0044
205 __find_get_block 0.8333
22 __getblk 0.2178
10 __bread 0.1515
5 set_bh_page 0.0926
27 create_empty_buffers 0.1337
4 unmap_underlying_metadata 0.0354
46 __block_write_full_page 0.0479
59 __block_prepare_write 0.0594
20 __block_commit_write 0.1342
12 block_prepare_write 0.1558
5 generic_commit_write 0.0338
19 block_write_full_page 0.0712
21 end_bio_bh_io_sync 0.3387
40 submit_bh 0.1227
35 ll_rw_block 0.2574
3 recalc_bh_state 0.0168
13 alloc_buffer_head 0.1238
18 init_buffer_head 0.3396
8 .text.lock.buffer 0.0430
10 bio_destructor 0.1064
16 bio_alloc 0.0376
51 bio_put 0.9444
4 bio_endio 0.0315
1 nr_blockdev_pages 0.0075
1 generic_fillattr 0.0061
54 inode_add_bytes 0.3803
1 setup_arg_pages 0.0021
1 flush_old_exec 0.0004
1 do_execve 0.0015
42 pipe_wait 0.2346
226 pipe_readv 0.3197
7 pipe_read 0.1186
143 pipe_writev 0.1725
10 pipe_write 0.1695
1 bad_pipe_r 0.1667
127 pipe_poll 1.0242
1 vfs_permission 0.0032
1 path_release 0.0143
1 follow_mount 0.0068
2 do_lookup 0.0124
2 link_path_walk 0.0005
2 path_lookup 0.0042
2 may_open 0.0039
2 open_namei 0.0011
10 kill_fasync 0.0862
3 poll_initwait 0.1200
46 poll_freewait 0.7188
37 __pollwait 0.1878
13 max_select_fd 0.0578
140 do_select 0.1790
7 select_bits_alloc 0.2188
2 select_bits_free 0.4000
104 sys_select 0.0806
1 locks_remove_posix 0.0030
2 dput 0.0032
3 __d_lookup 0.0072
1 iput 0.0081
131 inode_times_differ 1.9265
34 update_atime 0.1532
40 inode_update_time 0.1869
1 wake_up_inode 0.0139
199 dnotify_parent 0.7866
1 lookup_mnt 0.0060
1 seq_printf 0.0116
154 __mark_inode_dirty 0.3182
1 sync_sb_inodes 0.0013
2 writeback_inodes 0.0050
13 mpage_writepages 0.0155
1 load_elf_interp 0.0016
6 show_stat 0.0026
7 write_profile 0.0449
35 ext3_get_group_desc 0.1862
6 read_block_bitmap 0.0395
4 find_next_usable_block 0.0047
59 ext3_try_to_allocate 0.0663
4 ext3_has_free_blocks 0.0325
82 ext3_new_block 0.0631
17 ext3_file_write 0.0798
6 ext3_alloc_block 0.1017
98 ext3_block_to_path 0.3082
40 ext3_get_branch 0.1695
21 ext3_find_goal 0.1533
25 ext3_alloc_branch 0.0361
40 ext3_splice_branch 0.0844
78 ext3_get_block_handle 0.0877
11 ext3_get_block 0.0632
74 walk_page_buffers 0.6607
20 ext3_prepare_write 0.0629
7 ext3_journal_dirty_data 0.0753
106 ext3_ordered_commit_write 0.4629
13 bget_one 1.1818
2 bput_one 0.1818
3 journal_dirty_data_fn 0.1579
19 ext3_ordered_writepage 0.0412
57 ext3_get_inode_block 0.2184
40 ext3_get_inode_loc 0.0629
98 ext3_do_update_inode 0.1062
41 ext3_writepage_trans_blocks 0.2887
25 ext3_mark_iloc_dirty 0.4545
26 ext3_reserve_inode_write 0.1232
18 ext3_mark_inode_dirty 0.2222
44 ext3_dirty_inode 0.3259
19 ext3_journal_start 0.2135
27 __ext3_journal_stop 0.3375
117 start_this_handle 0.0913
23 new_handle 0.2875
27 journal_start 0.1286
158 do_get_write_access 0.0885
26 journal_get_write_access 0.3210
29 journal_get_undo_access 0.0699
64 journal_dirty_data 0.0876
120 journal_dirty_metadata 0.2186
748 journal_stop 0.9189
95 __journal_unfile_buffer 0.1838
36 __journal_file_buffer 0.0514
17 inverted_lock 0.1771
252 journal_commit_transaction 0.0413
2 journal_brelse_array 0.0455
1 __journal_clean_checkpoint_list 0.0057
1 __journal_insert_checkpoint 0.0048
69 journal_cancel_revoke 0.1988
11 __log_space_left 0.1068
2 __log_start_commit 0.0317
1 journal_get_descriptor_buffer 0.0053
8 journal_blocks_per_page 0.2759
4 journal_alloc_journal_head 0.0345
1 journal_free_journal_head 0.0357
275 journal_add_journal_head 0.7514
60 __journal_remove_journal_head 0.1508
66 journal_remove_journal_head 0.5323
94 journal_put_journal_head 0.4215
1 semctl_main 0.0010
1 do_shmat 0.0013
1 cap_bprm_apply_creds 0.0027
2 kobject_put 0.0800
16 radix_tree_preload 0.0904
1 rb_erase 0.0044
6 number 0.0077
9 vsnprintf 0.0073
1 find_next_bit 0.0066
27 csum_partial 0.0938
3 atomic_dec_and_lock 0.0259
18 memcpy 0.3158
1 strncpy_from_user 0.0101
1 strnlen_user 0.0104
6212 __copy_user_intel 36.1163
4486 __copy_user_zeroing_intel 26.0814
92 __copy_to_user_ll 0.7667
2302 __copy_from_user_ll 19.1833
37 copy_to_user 0.3776
37 copy_from_user 0.2937
1 add_timer_randomness 0.0034
4 add_disk_randomness 0.0769
2 blk_run_queue 0.0161
3 get_request 0.0040
124 __make_request 0.0892
10 generic_make_request 0.0203
8 submit_bio 0.0277
48 __end_that_request_first 0.0913
3 put_io_context 0.0341
2 get_io_context 0.0160
12 .text.lock.ll_rw_blk 0.0882
2 as_set_request 0.0153
312 e1000_xmit_frame 0.1466
5 scsi_put_command 0.0221
29 scsi_dispatch_cmd 0.0538
1 scsi_softirq 0.0048
6 scsi_finish_command 0.0335
2 scsi_add_timer 0.0185
8 scsi_init_cmd_errh 0.0417
11 scsi_device_unbusy 0.0598
4 scsi_run_queue 0.0145
32 scsi_end_request 0.1212
1 scsi_free_sgtable 0.0222
11 scsi_io_completion 0.0108
139 scsi_request_fn 0.1281
17 .text.lock.scsi_lib 0.0794
68 ahc_linux_isr 0.0978
1 sock_map_fd 0.0031
1 sockfd_lookup 0.0084
17 sock_aio_read 0.0578
16 sock_aio_write 0.0544
21 sock_poll 0.4286
196 sock_wfree 2.5789
15 __release_sock 0.0974
294 sock_def_readable 1.6333
13 sk_reset_timer 0.2766
90 lock_sock 0.7258
53 release_sock 0.3487
35 sock_common_recvmsg 0.2991
189 .text.lock.sock 1.5242
240 alloc_skb 1.0714
394 skb_release_data 2.2644
10 kfree_skbmem 0.2273
632 __kfree_skb 2.7719
91 skb_clone 0.2370
31 copy_skb_header 0.1250
18 skb_copy 0.0811
294 skb_copy_bits 0.5589
5 skb_checksum 0.0081
1 skb_dequeue 0.0085
23 memcpy_toiovec 0.2421
1 memcpy_tokerneliovec 0.0133
117 skb_copy_datagram_iovec 0.2159
39 sk_stream_rfree 1.4444
3 sk_stream_error 0.0323
1 sk_stream_mem_schedule 0.0029
22 skb_checksum_help 0.0679
95 dev_queue_xmit 0.1193
24 net_tx_action 0.0623
104 netif_receive_skb 0.2337
118 process_backlog 0.4322
61 net_rx_action 0.2355
202 nf_iterate 1.2242
75 nf_hook_slow 0.2622
167 qdisc_restart 0.2831
150 pfifo_fast_enqueue 1.0345
104 pfifo_fast_dequeue 1.2840
31 rt_hash_code 0.2263
266 ip_route_input 0.4890
39 ip_local_deliver 0.0771
181 ip_rcv 0.1416
70 ip_local_deliver_finish 0.1795
36 ip_rcv_finish 0.0556
21 ip_finish_output 0.0395
17 ip_output 0.1417
352 ip_queue_xmit 0.2334
10 dst_output 0.2439
257 ip_finish_output2 0.5801
127 tcp_poll 0.3307
185 tcp_sendmsg 0.0407
60 cleanup_rbuf 0.1948
428 tcp_recvmsg 0.2131
40 __tcp_grow_window 0.3053
61 tcp_rcv_rtt_update 0.6703
46 tcp_rcv_space_adjust 0.1411
238 tcp_event_data_recv 0.3980
26 tcp_rtt_estimator 0.0774
12 tcp_ack_saw_tstamp 0.1429
92 tcp_clean_rtx_queue 0.0967
27 tcp_ack 0.0184
9 __tcp_data_snd_check 0.0381
54 __tcp_ack_snd_check 0.3354
760 tcp_rcv_established 0.3290
292 tcp_transmit_skb 0.1676
47 tcp_write_xmit 0.0648
43 __tcp_select_window 0.1448
23 tcp_send_delayed_ack 0.1041
87 tcp_send_ack 0.4462
487 __tcp_v4_lookup_established 1.8447
33 tcp_v4_send_check 0.1304
54 tcp_v4_do_rcv 0.1698
395 tcp_v4_rcv 0.2006
7 inet_sendmsg 0.0787
15 ip_confirm 0.3125
30 ip_conntrack_defrag 0.4286
16 ip_refrag 0.1119
23 ip_conntrack_local 0.1575
21 __ip_ct_find_proto 0.3231
221 ip_ct_find_proto 2.0654
74 hash_conntrack 0.4805
40 get_tuple 0.4301
2 invert_tuple 0.0426
84 __ip_conntrack_find 0.3836
143 ip_conntrack_find_get 1.1818
14 ip_conntrack_get 0.3889
117 ip_conntrack_in 0.1902
269 ip_ct_refresh 1.4158
30 tcp_pkt_to_tuple 0.3529
10 get_conntrack_index 0.1639
457 tcp_packet 0.9662
889 ipt_do_table 0.9174
10 ipt_hook 0.1695
13 ipt_local_out_hook 0.1111
35 ipt_route_hook 0.5932
29 ipt_local_hook 0.1245
67 ip_nat_fn 0.1113
23 ip_nat_out 0.1337
1 find_nat_proto 0.0149
195 do_bindings 0.2393
1 unix_release_sock 0.0010
13 __sched_text_start 0.0448
2 __down_failed 0.1667
2 __write_lock_failed 0.0625
3 __read_lock_failed 0.1500
213 schedule 0.1024
1 io_schedule 0.0208
4 __preempt_spin_lock 0.0440
438 .text.lock.sched 2.2347
9 schedule_timeout 0.0471
462536 total 0.1504
22 default_idle 0.4889
415897 poll_idle 7170.6379
42 cpu_idle 0.6562
4 need_resched 0.0800
132 system_call 3.0000
6 syscall_call 0.5455
8 syscall_exit 0.7273
4 handle_IRQ_event 0.0400
1 sys_mmap2 0.0047
2 show_cpuinfo 0.0030
43 sched_clock 0.3162
1 flush_tlb_mm 0.0059
2 flush_tlb_page 0.0113
2 pte_alloc_one 0.0260
1 pgd_ctor 0.0060
13 do_page_fault 0.0093
1705 finish_task_switch 11.4430
1016 __wake_up 8.6102
216 __might_sleep 1.0237
71 add_wait_queue 0.5726
42 remove_wait_queue 0.3281
32 prepare_to_wait 0.2177
52 finish_wait 0.3291
2 copy_mm 0.0017
1 copy_files 0.0010
1 copy_process 0.0003
5 .text.lock.fork 0.0253
1 release_task 0.0018
1 do_exit 0.0009
131 current_kernel_time 1.8986
5134 __do_softirq 29.6763
152 local_bh_enable 1.0556
719 __mod_timer 1.3907
94 mod_timer 0.8868
355 del_timer 2.3052
1 del_timer_sync 0.0055
5 .text.lock.timer 0.0543
1 sys_rt_sigaction 0.0052
1 groups_search 0.0094
54 add_to_page_cache 0.2204
10 page_waitqueue 0.2083
46 wake_up_page 0.5974
16 unlock_page 0.4324
22 end_page_writeback 0.2973
54 find_get_page 0.4821
43 find_lock_page 0.1483
38 find_get_pages_tag 0.2390
2 do_generic_mapping_read 0.0016
1 file_read_actor 0.0041
2 filemap_nopage 0.0021
1 generic_file_mmap 0.0145
5 remove_suid 0.0336
126 generic_file_aio_write_nolock 0.0435
12 generic_file_aio_write 0.0759
44 mempool_alloc 0.1155
22 mempool_free 0.1048
1 mempool_alloc_slab 0.0476
1 mempool_free_slab 0.0476
14 bad_range 0.1400
18 prep_new_page 0.2222
55 free_hot_cold_page 0.1846
159 buffered_rmqueue 0.2989
88 __alloc_pages 0.0991
12 __get_free_pages 0.2034
21 __free_pages 0.2692
3 free_pages 0.0667
9 balance_dirty_pages_ratelimited 0.0448
58 __set_page_dirty_nobuffers 0.1543
1 set_page_dirty 0.0119
11 clear_page_dirty_for_io 0.0965
43 test_clear_page_writeback 0.1686
53 test_set_page_writeback 0.1767
18 .text.lock.page_writeback 0.1406
1 prio_tree_remove 0.0044
1 vma_prio_tree_add 0.0060
2 do_page_cache_readahead 0.0043
1 page_cache_readahead 0.0018
1 alloc_slabmgmt 0.0104
10 cache_init_objs 0.0962
1 kmem_flagcheck 0.0233
2 cache_grow 0.0044
159 kmem_cache_alloc 1.2927
1 kmem_ptr_validate 0.0132
88 __kmalloc 0.5714
187 kmem_cache_free 2.1494
233 kfree 2.1376
33 mark_page_accessed 0.6346
1 lru_cache_add_active 0.0105
31 release_pages 0.0686
22 __pagevec_lru_add 0.0694
2 __pagevec_lru_add_active 0.0060
1 pagevec_lookup_tag 0.0156
14 percpu_counter_mod 0.0800
3 clear_page_tables 0.0097
1 pte_alloc_map 0.0040
9 copy_page_range 0.0116
22 zap_pte_range 0.0317
1 unmap_vmas 0.0017
17 do_wp_page 0.0200
16 do_anonymous_page 0.0356
281 do_no_page 0.3154
7 handle_mm_fault 0.0159
2 remove_vm_struct 0.0123
1 find_vma_prepare 0.0099
1 vma_link 0.0030
1 vma_merge 0.0020
2 find_mergeable_anon_vma 0.0085
3 do_mmap_pgoff 0.0016
1 find_vma_prev 0.0123
1 do_munmap 0.0022
1 exit_mmap 0.0022
1 anon_vma_prepare 0.0028
2 anon_vma_unlink 0.0128
3 page_add_anon_rmap 0.0092
1 page_add_file_rmap 0.0046
5 page_remove_rmap 0.0209
1 get_unused_fd 0.0039
1 filp_close 0.0075
14 do_sync_read 0.0809
50 vfs_read 0.1695
22 do_sync_write 0.1272
46 vfs_write 0.1559
25 sys_read 0.1953
30 sys_write 0.2344
48 fput 2.2857
84 fget 0.6562
34 fget_light 0.2036
6 bh_waitq_head 0.2727
74 wake_up_buffer 0.8916
34 unlock_buffer 1.4167
9 __set_page_buffers 0.3600
56 end_buffer_write_sync 0.4628
21 __find_get_block_slow 0.0485
26 end_buffer_async_write 0.0841
2 buffer_insert_list 0.0134
23 create_buffers 0.1411
1 grow_dev_page 0.0025
7 mark_buffer_dirty 0.1750
45 __brelse 0.5769
1 bh_lru_install 0.0044
148 __find_get_block 0.6016
17 __getblk 0.1683
13 __bread 0.1970
4 set_bh_page 0.0741
25 create_empty_buffers 0.1238
5 unmap_underlying_metadata 0.0442
52 __block_write_full_page 0.0542
53 __block_prepare_write 0.0534
24 __block_commit_write 0.1611
8 block_prepare_write 0.1039
8 generic_commit_write 0.0541
17 block_write_full_page 0.0637
22 end_bio_bh_io_sync 0.3548
41 submit_bh 0.1258
22 ll_rw_block 0.1618
6 recalc_bh_state 0.0335
7 alloc_buffer_head 0.0667
20 init_buffer_head 0.3774
5 .text.lock.buffer 0.0269
8 bio_destructor 0.0851
12 bio_alloc 0.0282
41 bio_put 0.7593
2 bio_endio 0.0157
1 generic_fillattr 0.0061
14 inode_add_bytes 0.0986
1 copy_strings 0.0018
47 pipe_wait 0.2626
221 pipe_readv 0.3126
12 pipe_read 0.2034
111 pipe_writev 0.1339
5 pipe_write 0.0847
111 pipe_poll 0.8952
1 getname 0.0041
1 deny_write_access 0.0072
1 follow_mount 0.0068
3 do_lookup 0.0186
5 link_path_walk 0.0013
1 path_lookup 0.0021
19 kill_fasync 0.1638
1 poll_initwait 0.0400
30 poll_freewait 0.4688
30 __pollwait 0.1523
18 max_select_fd 0.0800
186 do_select 0.2379
5 select_bits_alloc 0.1562
1 select_bits_free 0.2000
127 sys_select 0.0984
5 __d_lookup 0.0120
124 inode_times_differ 1.8235
28 update_atime 0.1261
38 inode_update_time 0.1776
1 wake_up_inode 0.0139
1 setattr_mask 0.0096
199 dnotify_parent 0.7866
1 lookup_mnt 0.0060
82 __mark_inode_dirty 0.1694
1 sync_sb_inodes 0.0013
1 writeback_inodes 0.0025
13 mpage_writepages 0.0155
1 proc_pid_unhash 0.0063
1 proc_lookup 0.0030
3 show_stat 0.0013
10 write_profile 0.0641
16 ext3_get_group_desc 0.0851
7 read_block_bitmap 0.0461
2 find_next_usable_block 0.0024
65 ext3_try_to_allocate 0.0730
7 ext3_has_free_blocks 0.0569
71 ext3_new_block 0.0547
23 ext3_file_write 0.1080
1 ext3_alloc_block 0.0169
14 ext3_block_to_path 0.0440
36 ext3_get_branch 0.1525
20 ext3_find_goal 0.1460
15 ext3_alloc_branch 0.0217
36 ext3_splice_branch 0.0759
72 ext3_get_block_handle 0.0810
7 ext3_get_block 0.0402
68 walk_page_buffers 0.6071
1 do_journal_get_write_access 0.0083
20 ext3_prepare_write 0.0629
2 ext3_journal_dirty_data 0.0215
85 ext3_ordered_commit_write 0.3712
9 bget_one 0.8182
2 bput_one 0.1818
9 journal_dirty_data_fn 0.4737
13 ext3_ordered_writepage 0.0282
53 ext3_get_inode_block 0.2031
44 ext3_get_inode_loc 0.0692
84 ext3_do_update_inode 0.0910
33 ext3_writepage_trans_blocks 0.2324
42 ext3_mark_iloc_dirty 0.7636
23 ext3_reserve_inode_write 0.1090
12 ext3_mark_inode_dirty 0.1481
21 ext3_dirty_inode 0.1556
20 ext3_journal_start 0.2247
10 __ext3_journal_stop 0.1250
77 start_this_handle 0.0601
16 new_handle 0.2000
36 journal_start 0.1714
148 do_get_write_access 0.0829
18 journal_get_write_access 0.2222
1 journal_get_create_access 0.0015
22 journal_get_undo_access 0.0530
87 journal_dirty_data 0.1190
112 journal_dirty_metadata 0.2040
114 journal_stop 0.1400
93 __journal_unfile_buffer 0.1799
41 __journal_file_buffer 0.0585
1 journal_file_buffer 0.0046
20 inverted_lock 0.2083
230 journal_commit_transaction 0.0377
1 journal_brelse_array 0.0227
2 __journal_clean_checkpoint_list 0.0115
56 journal_cancel_revoke 0.1614
1 journal_write_revoke_records 0.0039
2 journal_write_metadata_buffer 0.0022
9 __log_space_left 0.0874
1 journal_get_descriptor_buffer 0.0053
8 journal_blocks_per_page 0.2759
5 journal_alloc_journal_head 0.0431
1 journal_free_journal_head 0.0357
282 journal_add_journal_head 0.7705
53 __journal_remove_journal_head 0.1332
64 journal_remove_journal_head 0.5161
96 journal_put_journal_head 0.4305
1 ipc_findkey 0.0204
1 cap_vm_enough_memory 0.0038
1 kobject_put 0.0400
8 radix_tree_preload 0.0452
2 number 0.0026
12 vsnprintf 0.0097
22 csum_partial 0.0764
1 atomic_dec_and_lock 0.0086
12 memcpy 0.2105
1 strnlen_user 0.0104
6132 __copy_user_intel 35.6512
4273 __copy_user_zeroing_intel 24.8430
83 __copy_to_user_ll 0.6917
2169 __copy_from_user_ll 18.0750
32 copy_to_user 0.3265
27 copy_from_user 0.2143
2 add_disk_randomness 0.0385
1 blk_run_queue 0.0081
2 get_request 0.0027
106 __make_request 0.0763
7 generic_make_request 0.0142
9 submit_bio 0.0311
69 __end_that_request_first 0.1312
3 put_io_context 0.0341
1 get_io_context 0.0080
5 .text.lock.ll_rw_blk 0.0368
294 e1000_xmit_frame 0.1382
3 scsi_put_command 0.0133
19 scsi_dispatch_cmd 0.0353
3 scsi_finish_command 0.0168
2 scsi_add_timer 0.0185
4 scsi_init_cmd_errh 0.0208
8 scsi_device_unbusy 0.0435
3 scsi_run_queue 0.0109
30 scsi_end_request 0.1136
3 scsi_io_completion 0.0029
128 scsi_request_fn 0.1180
14 .text.lock.scsi_lib 0.0654
32 ahc_linux_isr 0.0460
25 sock_aio_read 0.0850
4 sock_aio_write 0.0136
30 sock_poll 0.6122
205 sock_wfree 2.6974
14 __release_sock 0.0909
331 sock_def_readable 1.8389
11 sk_reset_timer 0.2340
73 lock_sock 0.5887
38 release_sock 0.2500
37 sock_common_recvmsg 0.3162
180 .text.lock.sock 1.4516
231 alloc_skb 1.0312
394 skb_release_data 2.2644
13 kfree_skbmem 0.2955
627 __kfree_skb 2.7500
85 skb_clone 0.2214
34 copy_skb_header 0.1371
19 skb_copy 0.0856
276 skb_copy_bits 0.5247
3 skb_checksum 0.0049
18 memcpy_toiovec 0.1895
2 memcpy_tokerneliovec 0.0267
126 skb_copy_datagram_iovec 0.2325
36 sk_stream_rfree 1.3333
1 sk_stream_error 0.0108
17 skb_checksum_help 0.0525
109 dev_queue_xmit 0.1369
27 net_tx_action 0.0701
83 netif_receive_skb 0.1865
100 process_backlog 0.3663
53 net_rx_action 0.2046
191 nf_iterate 1.1576
96 nf_hook_slow 0.3357
161 qdisc_restart 0.2729
132 pfifo_fast_enqueue 0.9103
103 pfifo_fast_dequeue 1.2716
20 rt_hash_code 0.1460
253 ip_route_input 0.4651
40 ip_local_deliver 0.0791
152 ip_rcv 0.1189
61 ip_local_deliver_finish 0.1564
53 ip_rcv_finish 0.0818
14 ip_finish_output 0.0263
21 ip_output 0.1750
388 ip_queue_xmit 0.2573
6 dst_output 0.1463
190 ip_finish_output2 0.4289
134 tcp_poll 0.3490
166 tcp_sendmsg 0.0365
45 cleanup_rbuf 0.1461
420 tcp_recvmsg 0.2092
30 __tcp_grow_window 0.2290
50 tcp_rcv_rtt_update 0.5495
30 tcp_rcv_space_adjust 0.0920
223 tcp_event_data_recv 0.3729
25 tcp_rtt_estimator 0.0744
8 tcp_ack_saw_tstamp 0.0952
58 tcp_clean_rtx_queue 0.0610
42 tcp_ack 0.0287
8 __tcp_data_snd_check 0.0339
62 __tcp_ack_snd_check 0.3851
783 tcp_rcv_established 0.3390
250 tcp_transmit_skb 0.1435
45 tcp_write_xmit 0.0621
52 __tcp_select_window 0.1751
17 tcp_send_delayed_ack 0.0769
69 tcp_send_ack 0.3538
456 __tcp_v4_lookup_established 1.7273
32 tcp_v4_send_check 0.1265
48 tcp_v4_do_rcv 0.1509
412 tcp_v4_rcv 0.2092
8 inet_sendmsg 0.0899
27 ip_confirm 0.5625
37 ip_conntrack_defrag 0.5286
12 ip_refrag 0.0839
22 ip_conntrack_local 0.1507
20 __ip_ct_find_proto 0.3077
263 ip_ct_find_proto 2.4579
64 hash_conntrack 0.4156
27 get_tuple 0.2903
4 invert_tuple 0.0851
76 __ip_conntrack_find 0.3470
157 ip_conntrack_find_get 1.2975
5 ip_conntrack_get 0.1389
129 ip_conntrack_in 0.2098
271 ip_ct_refresh 1.4263
6 ip_ct_gather_frags 0.0311
38 tcp_pkt_to_tuple 0.4471
12 get_conntrack_index 0.1967
446 tcp_packet 0.9429
870 ipt_do_table 0.8978
22 ipt_hook 0.3729
21 ipt_local_out_hook 0.1795
52 ipt_route_hook 0.8814
23 ipt_local_hook 0.0987
58 ip_nat_fn 0.0963
24 ip_nat_out 0.1395
199 do_bindings 0.2442
1 unix_stream_connect 0.0008
3 __sched_text_start 0.0103
1 __down_failed 0.0833
2 __write_lock_failed 0.0625
5 __read_lock_failed 0.2500
240 schedule 0.1153
1 preempt_schedule 0.0145
5 __preempt_spin_lock 0.0549
405 .text.lock.sched 2.0663
6 schedule_timeout 0.0314
459010 total 0.1492
28 default_idle 0.6222
414380 poll_idle 7144.4828
43 cpu_idle 0.6719
1 restore_sigcontext 0.0031
7 need_resched 0.1400
132 system_call 3.0000
5 syscall_call 0.4545
8 syscall_exit 0.7273
4 handle_IRQ_event 0.0400
1 show_cpuinfo 0.0015
49 sched_clock 0.3603
2 pte_alloc_one 0.0260
21 do_page_fault 0.0150
1751 finish_task_switch 11.7517
1008 __wake_up 8.5424
237 __might_sleep 1.1232
78 add_wait_queue 0.6290
36 remove_wait_queue 0.2812
36 prepare_to_wait 0.2449
35 finish_wait 0.2215
1 mm_init 0.0040
1 copy_process 0.0003
1 do_fork 0.0022
5 .text.lock.fork 0.0253
3 exit_notify 0.0013
1 sys_wait4 0.0016
145 current_kernel_time 2.1014
5115 __do_softirq 29.5665
156 local_bh_enable 1.0833
812 __mod_timer 1.5706
87 mod_timer 0.8208
321 del_timer 2.0844
1 run_timer_softirq 0.0022
2 .text.lock.timer 0.0217
1 flush_signal_handlers 0.0149
3 in_group_p 0.0250
58 add_to_page_cache 0.2367
18 page_waitqueue 0.3750
49 wake_up_page 0.6364
14 unlock_page 0.3784
8 end_page_writeback 0.1081
33 find_get_page 0.2946
23 find_lock_page 0.0793
49 find_get_pages_tag 0.3082
1 do_generic_mapping_read 0.0008
2 file_read_actor 0.0083
1 generic_file_aio_read 0.0086
4 filemap_nopage 0.0042
5 remove_suid 0.0336
145 generic_file_aio_write_nolock 0.0501
19 generic_file_aio_write 0.1203
37 mempool_alloc 0.0971
19 mempool_free 0.0905
3 mempool_alloc_slab 0.1429
32 bad_range 0.3200
21 prep_new_page 0.2593
51 free_hot_cold_page 0.1711
165 buffered_rmqueue 0.3102
79 __alloc_pages 0.0890
7 __get_free_pages 0.1186
13 __free_pages 0.1667
2 free_pages 0.0444
7 balance_dirty_pages_ratelimited 0.0348
74 __set_page_dirty_nobuffers 0.1968
3 set_page_dirty 0.0357
14 clear_page_dirty_for_io 0.1228
32 test_clear_page_writeback 0.1255
51 test_set_page_writeback 0.1700
9 .text.lock.page_writeback 0.0703
1 file_ra_state_init 0.0270
1 do_page_cache_readahead 0.0022
1 page_cache_readahead 0.0018
3 kmem_getpages 0.0144
6 cache_init_objs 0.0577
1 kmem_flagcheck 0.0233
3 cache_grow 0.0066
145 kmem_cache_alloc 1.1789
3 kmem_ptr_validate 0.0395
88 __kmalloc 0.5714
218 kmem_cache_free 2.5057
205 kfree 1.8807
36 mark_page_accessed 0.6923
1 lru_cache_add_active 0.0105
1 lru_add_drain 0.0089
32 release_pages 0.0708
1 __pagevec_release 0.0185
29 __pagevec_lru_add 0.0915
4 __pagevec_lru_add_active 0.0119
1 vm_acct_memory 0.0122
15 percpu_counter_mod 0.0857
3 clear_page_tables 0.0097
1 pte_alloc_map 0.0040
12 copy_page_range 0.0155
24 zap_pte_range 0.0346
1 unmap_page_range 0.0088
9 do_wp_page 0.0106
35 do_anonymous_page 0.0780
325 do_no_page 0.3648
1 handle_mm_fault 0.0023
1 remove_vm_struct 0.0062
1 find_vma_prepare 0.0099
1 vma_adjust 0.0010
1 can_vma_merge_after 0.0090
2 vma_merge 0.0041
2 do_mmap_pgoff 0.0011
4 find_vma 0.0494
1 find_vma_prev 0.0123
2 expand_stack 0.0069
1 free_pgtables 0.0067
1 do_munmap 0.0022
1 sys_munmap 0.0082
1 anon_vma_link 0.0079
6 page_add_anon_rmap 0.0183
3 page_add_file_rmap 0.0139
9 page_remove_rmap 0.0377
1 free_pages_and_swap_cache 0.0076
2 dentry_open 0.0051
1 get_unused_fd 0.0039
1 sys_close 0.0049
1 generic_file_llseek 0.0046
11 do_sync_read 0.0636
59 vfs_read 0.2000
23 do_sync_write 0.1329
55 vfs_write 0.1864
24 sys_read 0.1875
27 sys_write 0.2109
1 get_empty_filp 0.0043
72 fput 3.4286
104 fget 0.8125
37 fget_light 0.2216
3 bh_waitq_head 0.1364
70 wake_up_buffer 0.8434
32 unlock_buffer 1.3333
9 __set_page_buffers 0.3600
74 end_buffer_write_sync 0.6116
14 __find_get_block_slow 0.0323
12 end_buffer_async_write 0.0388
1 buffer_insert_list 0.0067
30 create_buffers 0.1840
1 init_page_buffers 0.0089
12 mark_buffer_dirty 0.3000
45 __brelse 0.5769
162 __find_get_block 0.6585
17 __getblk 0.1683
9 __bread 0.1364
5 set_bh_page 0.0926
2 try_to_release_page 0.0182
22 create_empty_buffers 0.1089
55 __block_write_full_page 0.0573
61 __block_prepare_write 0.0614
18 __block_commit_write 0.1208
5 block_prepare_write 0.0649
14 generic_commit_write 0.0946
26 block_write_full_page 0.0974
21 end_bio_bh_io_sync 0.3387
25 submit_bh 0.0767
30 ll_rw_block 0.2206
4 recalc_bh_state 0.0223
15 alloc_buffer_head 0.1429
10 init_buffer_head 0.1887
1 .text.lock.buffer 0.0054
5 bio_destructor 0.0532
14 bio_alloc 0.0329
50 bio_put 0.9259
3 bio_endio 0.0236
1 cp_new_stat64 0.0037
1 sys_fstat64 0.0175
15 inode_add_bytes 0.1056
2 flush_old_exec 0.0008
74 pipe_wait 0.4134
222 pipe_readv 0.3140
9 pipe_read 0.1525
128 pipe_writev 0.1544
10 pipe_write 0.1695
120 pipe_poll 0.9677
1 pipe_release 0.0043
1 get_pipe_inode 0.0052
1 vfs_permission 0.0032
2 deny_write_access 0.0145
1 follow_mount 0.0068
2 do_lookup 0.0124
7 link_path_walk 0.0018
1 __user_walk 0.0081
1 may_open 0.0019
1 open_namei 0.0006
1 do_fcntl 0.0021
12 kill_fasync 0.1034
5 poll_initwait 0.2000
31 poll_freewait 0.4844
32 __pollwait 0.1624
19 max_select_fd 0.0844
149 do_select 0.1905
7 select_bits_alloc 0.2188
1 select_bits_free 0.2000
109 sys_select 0.0845
1 __posix_lock_file 0.0006
2 dput 0.0032
1 d_alloc 0.0022
5 __d_lookup 0.0120
102 inode_times_differ 1.5000
33 update_atime 0.1486
56 inode_update_time 0.2617
1 i_waitq_head 0.0455
1 is_bad_inode 0.0455
241 dnotify_parent 0.9526
93 __mark_inode_dirty 0.1921
1 __sync_single_inode 0.0018
12 mpage_writepages 0.0143
1 proc_lookup 0.0030
1 meminfo_read_proc 0.0020
4 show_stat 0.0018
11 write_profile 0.0705
32 ext3_get_group_desc 0.1702
3 read_block_bitmap 0.0197
1 find_next_usable_block 0.0012
64 ext3_try_to_allocate 0.0719
5 ext3_has_free_blocks 0.0407
68 ext3_new_block 0.0523
25 ext3_file_write 0.1174
5 ext3_alloc_block 0.0847
16 ext3_block_to_path 0.0503
39 ext3_get_branch 0.1653
9 ext3_find_goal 0.0657
14 ext3_alloc_branch 0.0202
44 ext3_splice_branch 0.0928
62 ext3_get_block_handle 0.0697
7 ext3_get_block 0.0402
61 walk_page_buffers 0.5446
2 do_journal_get_write_access 0.0165
32 ext3_prepare_write 0.1006
6 ext3_journal_dirty_data 0.0645
78 ext3_ordered_commit_write 0.3406
16 bget_one 1.4545
1 bput_one 0.0909
2 journal_dirty_data_fn 0.1053
8 ext3_ordered_writepage 0.0174
48 ext3_get_inode_block 0.1839
35 ext3_get_inode_loc 0.0550
95 ext3_do_update_inode 0.1029
29 ext3_writepage_trans_blocks 0.2042
36 ext3_mark_iloc_dirty 0.6545
18 ext3_reserve_inode_write 0.0853
15 ext3_mark_inode_dirty 0.1852
25 ext3_dirty_inode 0.1852
20 ext3_journal_start 0.2247
14 __ext3_journal_stop 0.1750
50 start_this_handle 0.0390
20 new_handle 0.2500
35 journal_start 0.1667
177 do_get_write_access 0.0991
22 journal_get_write_access 0.2716
24 journal_get_undo_access 0.0578
53 journal_dirty_data 0.0725
109 journal_dirty_metadata 0.1985
100 journal_stop 0.1229
100 __journal_unfile_buffer 0.1934
37 __journal_file_buffer 0.0528
28 inverted_lock 0.2917
272 journal_commit_transaction 0.0445
1 __journal_remove_checkpoint 0.0069
73 journal_cancel_revoke 0.2104
12 __log_space_left 0.1165
12 journal_blocks_per_page 0.4138
6 journal_alloc_journal_head 0.0517
2 journal_free_journal_head 0.0714
260 journal_add_journal_head 0.7104
39 __journal_remove_journal_head 0.0980
55 journal_remove_journal_head 0.4435
75 journal_put_journal_head 0.3363
7 radix_tree_preload 0.0395
1 rb_erase 0.0044
2 number 0.0026
10 vsnprintf 0.0081
1 find_next_zero_bit 0.0057
29 csum_partial 0.1007
16 memcpy 0.2807
6043 __copy_user_intel 35.1337
4277 __copy_user_zeroing_intel 24.8663
91 __copy_to_user_ll 0.7583
2215 __copy_from_user_ll 18.4583
26 copy_to_user 0.2653
27 copy_from_user 0.2143
2 add_disk_randomness 0.0385
2 blk_run_queue 0.0161
1 get_request 0.0013
120 __make_request 0.0863
13 generic_make_request 0.0264
11 submit_bio 0.0381
53 __end_that_request_first 0.1008
1 get_io_context 0.0080
9 .text.lock.ll_rw_blk 0.0662
2 as_set_request 0.0153
303 e1000_xmit_frame 0.1424
5 scsi_put_command 0.0221
19 scsi_dispatch_cmd 0.0353
2 scsi_finish_command 0.0112
1 scsi_add_timer 0.0093
1 scsi_decide_disposition 0.0033
2 scsi_init_cmd_errh 0.0104
10 scsi_device_unbusy 0.0543
4 scsi_run_queue 0.0145
22 scsi_end_request 0.0833
4 scsi_io_completion 0.0039
121 scsi_request_fn 0.1115
20 .text.lock.scsi_lib 0.0935
37 ahc_linux_isr 0.0532
1 sd_rw_intr 0.0015
1 move_addr_to_kernel 0.0139
28 sock_aio_read 0.0952
7 sock_aio_write 0.0238
23 sock_poll 0.4694
182 sock_wfree 2.3947
1 sock_wmalloc 0.0106
18 __release_sock 0.1169
397 sock_def_readable 2.2056
11 sk_reset_timer 0.2340
95 lock_sock 0.7661
51 release_sock 0.3355
31 sock_common_recvmsg 0.2650
191 .text.lock.sock 1.5403
247 alloc_skb 1.1027
407 skb_release_data 2.3391
13 kfree_skbmem 0.2955
658 __kfree_skb 2.8860
74 skb_clone 0.1927
35 copy_skb_header 0.1411
23 skb_copy 0.1036
285 skb_copy_bits 0.5418
5 skb_checksum 0.0081
1 verify_iovec 0.0068
17 memcpy_toiovec 0.1789
4 memcpy_tokerneliovec 0.0533
131 skb_copy_datagram_iovec 0.2417
28 sk_stream_rfree 1.0370
3 sk_stream_error 0.0323
10 skb_checksum_help 0.0309
101 dev_queue_xmit 0.1269
18 net_tx_action 0.0468
73 netif_receive_skb 0.1640
112 process_backlog 0.4103
56 net_rx_action 0.2162
197 nf_iterate 1.1939
112 nf_hook_slow 0.3916
162 qdisc_restart 0.2746
133 pfifo_fast_enqueue 0.9172
101 pfifo_fast_dequeue 1.2469
22 rt_hash_code 0.1606
298 ip_route_input 0.5478
39 ip_local_deliver 0.0771
153 ip_rcv 0.1197
71 ip_local_deliver_finish 0.1821
51 ip_rcv_finish 0.0787
25 ip_finish_output 0.0470
13 ip_output 0.1083
349 ip_queue_xmit 0.2314
10 dst_output 0.2439
225 ip_finish_output2 0.5079
126 tcp_poll 0.3281
163 tcp_sendmsg 0.0359
55 cleanup_rbuf 0.1786
431 tcp_recvmsg 0.2146
1 tcp_enter_quickack_mode 0.0263
29 __tcp_grow_window 0.2214
55 tcp_rcv_rtt_update 0.6044
46 tcp_rcv_space_adjust 0.1411
230 tcp_event_data_recv 0.3846
31 tcp_rtt_estimator 0.0923
14 tcp_ack_saw_tstamp 0.1667
59 tcp_clean_rtx_queue 0.0620
33 tcp_ack 0.0225
9 __tcp_data_snd_check 0.0381
64 __tcp_ack_snd_check 0.3975
803 tcp_rcv_established 0.3476
278 tcp_transmit_skb 0.1596
46 tcp_write_xmit 0.0634
31 __tcp_select_window 0.1044
17 tcp_send_delayed_ack 0.0769
80 tcp_send_ack 0.4103
494 __tcp_v4_lookup_established 1.8712
33 tcp_v4_send_check 0.1304
42 tcp_v4_do_rcv 0.1321
424 tcp_v4_rcv 0.2153
9 inet_sendmsg 0.1011
21 ip_confirm 0.4375
39 ip_conntrack_defrag 0.5571
15 ip_refrag 0.1049
13 ip_conntrack_local 0.0890
25 __ip_ct_find_proto 0.3846
234 ip_ct_find_proto 2.1869
65 hash_conntrack 0.4221
26 get_tuple 0.2796
2 invert_tuple 0.0426
66 __ip_conntrack_find 0.3014
159 ip_conntrack_find_get 1.3140
12 ip_conntrack_get 0.3333
114 ip_conntrack_in 0.1854
269 ip_ct_refresh 1.4158
39 tcp_pkt_to_tuple 0.4588
8 get_conntrack_index 0.1311
421 tcp_packet 0.8901
940 ipt_do_table 0.9701
11 ipt_hook 0.1864
13 ipt_local_out_hook 0.1111
42 ipt_route_hook 0.7119
28 ipt_local_hook 0.1202
51 ip_nat_fn 0.0847
24 ip_nat_out 0.1395
214 do_bindings 0.2626
1 unix_release_sock 0.0010
1 __sched_text_start 0.0034
1 __up_wakeup 0.0833
4 __write_lock_failed 0.1250
2 __read_lock_failed 0.1000
223 schedule 0.1072
1 __preempt_spin_lock 0.0110
432 .text.lock.sched 2.2041
4 schedule_timeout 0.0209
458065 total 0.1489
29 default_idle 0.6444
418979 poll_idle 7223.7759
68 cpu_idle 1.0625
4 need_resched 0.0800
120 system_call 2.7273
6 syscall_call 0.5455
5 syscall_exit 0.4545
11 handle_IRQ_event 0.1100
1 old_mmap 0.0030
1 show_cpuinfo 0.0015
58 sched_clock 0.4265
2 delay_tsc 0.0952
6 pte_alloc_one 0.0779
1 pgd_ctor 0.0060
17 do_page_fault 0.0122
1689 finish_task_switch 11.3356
1001 __wake_up 8.4831
203 __might_sleep 0.9621
87 add_wait_queue 0.7016
49 remove_wait_queue 0.3828
32 prepare_to_wait 0.2177
46 finish_wait 0.2911
1 copy_mm 0.0009
1 copy_process 0.0003
5 .text.lock.fork 0.0253
1 release_task 0.0018
1 sys_wait4 0.0016
597 current_kernel_time 8.6522
5519 __do_softirq 31.9017
174 local_bh_enable 1.2083
688 __mod_timer 1.3308
71 mod_timer 0.6698
335 del_timer 2.1753
1 sys_getpid 0.0625
1 .text.lock.timer 0.0109
1 do_sigaction 0.0016
3 in_group_p 0.0250
70 add_to_page_cache 0.2857
13 page_waitqueue 0.2708
48 wake_up_page 0.6234
24 unlock_page 0.6486
12 end_page_writeback 0.1622
55 find_get_page 0.4911
34 find_lock_page 0.1172
34 find_get_pages_tag 0.2138
2 do_generic_mapping_read 0.0016
1 __generic_file_aio_read 0.0018
4 filemap_nopage 0.0042
4 remove_suid 0.0268
118 generic_file_aio_write_nolock 0.0407
24 generic_file_aio_write 0.1519
43 mempool_alloc 0.1129
18 mempool_free 0.0857
1 mempool_alloc_slab 0.0476
4 mempool_free_slab 0.1905
20 bad_range 0.2000
13 prep_new_page 0.1605
1 __rmqueue 0.0037
44 free_hot_cold_page 0.1477
177 buffered_rmqueue 0.3327
74 __alloc_pages 0.0833
2 __get_free_pages 0.0339
14 __free_pages 0.1795
3 free_pages 0.0667
2 __read_page_state 0.0357
1 get_dirty_limits 0.0049
10 balance_dirty_pages_ratelimited 0.0498
59 __set_page_dirty_nobuffers 0.1569
2 set_page_dirty 0.0238
9 clear_page_dirty_for_io 0.0789
58 test_clear_page_writeback 0.2275
54 test_set_page_writeback 0.1800
18 .text.lock.page_writeback 0.1406
1 file_ra_state_init 0.0270
1 kmem_getpages 0.0048
2 alloc_slabmgmt 0.0208
3 cache_init_objs 0.0288
1 kmem_flagcheck 0.0233
1 set_slab_attr 0.0172
3 cache_grow 0.0066
162 kmem_cache_alloc 1.3171
3 kmem_ptr_validate 0.0395
81 __kmalloc 0.5260
186 kmem_cache_free 2.1379
206 kfree 1.8899
27 mark_page_accessed 0.5192
1 lru_cache_add 0.0105
1 lru_cache_add_active 0.0105
1 lru_add_drain 0.0089
34 release_pages 0.0752
29 __pagevec_lru_add 0.0915
2 __pagevec_lru_add_active 0.0060
1 vm_acct_memory 0.0122
16 percpu_counter_mod 0.0914
2 blk_queue_bounce 0.0260
2 clear_page_tables 0.0065
3 pte_alloc_map 0.0121
7 copy_page_range 0.0090
18 zap_pte_range 0.0259
1 unmap_vmas 0.0017
17 do_wp_page 0.0200
16 do_anonymous_page 0.0356
295 do_no_page 0.3311
3 handle_mm_fault 0.0068
1 remove_vm_struct 0.0062
1 find_vma_prepare 0.0099
2 vma_link 0.0061
1 vma_adjust 0.0010
1 can_vma_merge_before 0.0098
1 can_vma_merge_after 0.0090
1 do_mmap_pgoff 0.0005
2 find_vma 0.0247
1 expand_stack 0.0034
1 unmap_region 0.0035
3 split_vma 0.0105
2 page_add_anon_rmap 0.0061
1 page_add_file_rmap 0.0046
10 page_remove_rmap 0.0418
11 do_sync_read 0.0636
105 vfs_read 0.3559
15 do_sync_write 0.0867
57 vfs_write 0.1932
24 sys_read 0.1875
24 sys_write 0.1875
1 get_empty_filp 0.0043
62 fput 2.9524
1 __fput 0.0047
104 fget 0.8125
30 fget_light 0.1796
1 file_move 0.0071
5 bh_waitq_head 0.2273
70 wake_up_buffer 0.8434
35 unlock_buffer 1.4583
16 __set_page_buffers 0.6400
65 end_buffer_write_sync 0.5372
18 __find_get_block_slow 0.0416
13 end_buffer_async_write 0.0421
28 create_buffers 0.1718
11 mark_buffer_dirty 0.2750
52 __brelse 0.6667
146 __find_get_block 0.5935
20 __getblk 0.1980
8 __bread 0.1212
7 set_bh_page 0.1296
1 try_to_release_page 0.0091
26 create_empty_buffers 0.1287
1 unmap_underlying_metadata 0.0088
43 __block_write_full_page 0.0448
49 __block_prepare_write 0.0493
24 __block_commit_write 0.1611
15 block_prepare_write 0.1948
4 generic_commit_write 0.0270
10 block_write_full_page 0.0375
17 end_bio_bh_io_sync 0.2742
38 submit_bh 0.1166
32 ll_rw_block 0.2353
2 recalc_bh_state 0.0112
12 alloc_buffer_head 0.1143
17 init_buffer_head 0.3208
5 .text.lock.buffer 0.0269
6 bio_destructor 0.0638
18 bio_alloc 0.0423
55 bio_put 1.0185
5 bio_endio 0.0394
1 cp_new_stat64 0.0037
9 inode_add_bytes 0.0634
54 pipe_wait 0.3017
273 pipe_readv 0.3861
7 pipe_read 0.1186
141 pipe_writev 0.1701
7 pipe_write 0.1186
153 pipe_poll 1.2339
1 deny_write_access 0.0072
1 follow_mount 0.0068
4 link_path_walk 0.0010
3 path_lookup 0.0064
1 may_open 0.0019
1 open_namei 0.0006
19 kill_fasync 0.1638
2 poll_initwait 0.0800
29 poll_freewait 0.4531
40 __pollwait 0.2030
25 max_select_fd 0.1111
156 do_select 0.1995
7 select_bits_alloc 0.2188
119 sys_select 0.0922
1 __posix_lock_file 0.0006
1 __d_lookup 0.0024
1 alloc_inode 0.0030
147 inode_times_differ 2.1618
50 update_atime 0.2252
61 inode_update_time 0.2850
240 dnotify_parent 0.9486
1 seq_printf 0.0116
102 __mark_inode_dirty 0.2107
1 writeback_inodes 0.0025
8 mpage_writepages 0.0095
1 load_elf_binary 0.0003
4 show_stat 0.0018
11 write_profile 0.0705
22 ext3_get_group_desc 0.1170
5 read_block_bitmap 0.0329
2 find_next_usable_block 0.0024
44 ext3_try_to_allocate 0.0494
5 ext3_has_free_blocks 0.0407
61 ext3_new_block 0.0470
1 ext3_release_file 0.0098
24 ext3_file_write 0.1127
6 ext3_alloc_block 0.1017
28 ext3_block_to_path 0.0881
24 ext3_get_branch 0.1017
22 ext3_find_goal 0.1606
12 ext3_alloc_branch 0.0173
33 ext3_splice_branch 0.0696
76 ext3_get_block_handle 0.0855
16 ext3_get_block 0.0920
68 walk_page_buffers 0.6071
24 ext3_prepare_write 0.0755
5 ext3_journal_dirty_data 0.0538
61 ext3_ordered_commit_write 0.2664
11 bget_one 1.0000
8 journal_dirty_data_fn 0.4211
6 ext3_ordered_writepage 0.0130
44 ext3_get_inode_block 0.1686
24 ext3_get_inode_loc 0.0377
86 ext3_do_update_inode 0.0932
29 ext3_writepage_trans_blocks 0.2042
37 ext3_mark_iloc_dirty 0.6727
22 ext3_reserve_inode_write 0.1043
14 ext3_mark_inode_dirty 0.1728
27 ext3_dirty_inode 0.2000
43 ext3_journal_start 0.4831
42 __ext3_journal_stop 0.5250
136 start_this_handle 0.1061
8 new_handle 0.1000
46 journal_start 0.2190
169 do_get_write_access 0.0946
24 journal_get_write_access 0.2963
26 journal_get_undo_access 0.0627
76 journal_dirty_data 0.1040
140 journal_dirty_metadata 0.2550
104 journal_stop 0.1278
87 __journal_unfile_buffer 0.1683
41 __journal_file_buffer 0.0585
20 inverted_lock 0.2083
213 journal_commit_transaction 0.0349
2 journal_brelse_array 0.0455
74 journal_cancel_revoke 0.2133
1 journal_write_revoke_records 0.0039
1 kjournald 0.0014
9 __log_space_left 0.0874
5 journal_blocks_per_page 0.1724
8 journal_alloc_journal_head 0.0690
1 journal_free_journal_head 0.0357
243 journal_add_journal_head 0.6639
66 __journal_remove_journal_head 0.1658
64 journal_remove_journal_head 0.5161
98 journal_put_journal_head 0.4395
1 sys_shmdt 0.0028
1 kobject_put 0.0400
10 radix_tree_preload 0.0565
4 number 0.0051
6 vsnprintf 0.0049
29 csum_partial 0.1007
1 atomic_dec_and_lock 0.0086
9 memcpy 0.1579
2 strncpy_from_user 0.0202
5885 __copy_user_intel 34.2151
4237 __copy_user_zeroing_intel 24.6337
92 __copy_to_user_ll 0.7667
2218 __copy_from_user_ll 18.4833
23 copy_to_user 0.2347
36 copy_from_user 0.2857
1 add_entropy_words 0.0023
1 add_disk_randomness 0.0192
1 put_device 0.1000
2 get_request 0.0027
99 __make_request 0.0712
14 generic_make_request 0.0285
8 submit_bio 0.0277
52 __end_that_request_first 0.0989
8 .text.lock.ll_rw_blk 0.0588
338 e1000_xmit_frame 0.1588
4 scsi_put_command 0.0177
14 scsi_dispatch_cmd 0.0260
1 scsi_softirq 0.0048
2 scsi_finish_command 0.0112
2 scsi_decide_disposition 0.0067
3 scsi_init_cmd_errh 0.0156
10 scsi_device_unbusy 0.0543
4 scsi_run_queue 0.0145
22 scsi_end_request 0.0833
1 scsi_free_sgtable 0.0222
3 scsi_io_completion 0.0029
114 scsi_request_fn 0.1051
9 .text.lock.scsi_lib 0.0421
32 ahc_linux_isr 0.0460
1 sd_rw_intr 0.0015
25 sock_aio_read 0.0850
15 sock_aio_write 0.0510
27 sock_poll 0.5510
1 sock_fasync 0.0021
212 sock_wfree 2.7895
12 __release_sock 0.0779
293 sock_def_readable 1.6278
9 sk_reset_timer 0.1915
69 lock_sock 0.5565
49 release_sock 0.3224
33 sock_common_recvmsg 0.2821
190 .text.lock.sock 1.5323
241 alloc_skb 1.0759
406 skb_release_data 2.3333
19 kfree_skbmem 0.4318
651 __kfree_skb 2.8553
88 skb_clone 0.2292
23 copy_skb_header 0.0927
26 skb_copy 0.1171
299 skb_copy_bits 0.5684
4 skb_checksum 0.0065
18 memcpy_toiovec 0.1895
130 skb_copy_datagram_iovec 0.2399
31 sk_stream_rfree 1.1481
4 sk_stream_error 0.0430
11 skb_checksum_help 0.0340
88 dev_queue_xmit 0.1106
27 net_tx_action 0.0701
72 netif_receive_skb 0.1618
112 process_backlog 0.4103
68 net_rx_action 0.2625
201 nf_iterate 1.2182
93 nf_hook_slow 0.3252
153 qdisc_restart 0.2593
142 pfifo_fast_enqueue 0.9793
106 pfifo_fast_dequeue 1.3086
24 rt_hash_code 0.1752
261 ip_route_input 0.4798
26 ip_local_deliver 0.0514
185 ip_rcv 0.1448
63 ip_local_deliver_finish 0.1615
55 ip_rcv_finish 0.0849
17 ip_finish_output 0.0320
26 ip_output 0.2167
376 ip_queue_xmit 0.2493
6 dst_output 0.1463
209 ip_finish_output2 0.4718
114 tcp_poll 0.2969
189 tcp_sendmsg 0.0416
47 cleanup_rbuf 0.1526
449 tcp_recvmsg 0.2236
46 __tcp_grow_window 0.3511
59 tcp_rcv_rtt_update 0.6484
66 tcp_rcv_space_adjust 0.2025
237 tcp_event_data_recv 0.3963
35 tcp_rtt_estimator 0.1042
8 tcp_ack_saw_tstamp 0.0952
56 tcp_clean_rtx_queue 0.0589
33 tcp_ack 0.0225
2 tcp_data_queue 0.0007
11 __tcp_data_snd_check 0.0466
63 __tcp_ack_snd_check 0.3913
805 tcp_rcv_established 0.3485
254 tcp_transmit_skb 0.1458
47 tcp_write_xmit 0.0648
48 __tcp_select_window 0.1616
14 tcp_send_delayed_ack 0.0633
89 tcp_send_ack 0.4564
479 __tcp_v4_lookup_established 1.8144
28 tcp_v4_send_check 0.1107
51 tcp_v4_do_rcv 0.1604
425 tcp_v4_rcv 0.2158
10 inet_sendmsg 0.1124
23 ip_confirm 0.4792
31 ip_conntrack_defrag 0.4429
12 ip_refrag 0.0839
15 ip_conntrack_local 0.1027
22 __ip_ct_find_proto 0.3385
215 ip_ct_find_proto 2.0093
76 hash_conntrack 0.4935
49 get_tuple 0.5269
2 invert_tuple 0.0426
68 __ip_conntrack_find 0.3105
129 ip_conntrack_find_get 1.0661
10 ip_conntrack_get 0.2778
133 ip_conntrack_in 0.2163
313 ip_ct_refresh 1.6474
1 ip_ct_gather_frags 0.0052
38 tcp_pkt_to_tuple 0.4471
11 get_conntrack_index 0.1803
488 tcp_packet 1.0317
913 ipt_do_table 0.9422
10 ipt_hook 0.1695
12 ipt_local_out_hook 0.1026
31 ipt_route_hook 0.5254
29 ipt_local_hook 0.1245
70 ip_nat_fn 0.1163
23 ip_nat_out 0.1337
179 do_bindings 0.2196
1 unix_release_sock 0.0010
5 __sched_text_start 0.0172
1 __down_failed 0.0833
5 __write_lock_failed 0.1562
2 __read_lock_failed 0.1000
246 schedule 0.1182
7 __preempt_spin_lock 0.0769
430 .text.lock.sched 2.1939
8 schedule_timeout 0.0419
463392 total 0.1506
24 default_idle 0.5333
417483 poll_idle 7197.9828
48 cpu_idle 0.7500
1 flush_thread 0.0083
1 sys_execve 0.0064
126 system_call 2.8636
7 syscall_call 0.6364
15 syscall_exit 1.3636
1 handle_IRQ_event 0.0100
3 show_cpuinfo 0.0044
33 sched_clock 0.2426
2 pte_alloc_one 0.0260
1 pgd_ctor 0.0060
17 do_page_fault 0.0122
1660 finish_task_switch 11.1409
964 __wake_up 8.1695
1 migration_thread 0.0022
202 __might_sleep 0.9573
1 task_curr 0.0044
97 add_wait_queue 0.7823
53 remove_wait_queue 0.4141
33 prepare_to_wait 0.2245
35 finish_wait 0.2215
1 dup_task_struct 0.0047
2 copy_mm 0.0017
1 copy_files 0.0010
1 copy_process 0.0003
5 .text.lock.fork 0.0253
2 release_task 0.0037
1 do_setitimer 0.0021
90 current_kernel_time 1.3043
5208 __do_softirq 30.1040
152 local_bh_enable 1.0556
611 __mod_timer 1.1818
81 mod_timer 0.7642
323 del_timer 2.0974
1 run_timer_softirq 0.0022
1 sys_geteuid 0.0625
1 .text.lock.timer 0.0109
1 do_sigaction 0.0016
1 groups_search 0.0094
1 in_group_p 0.0083
50 add_to_page_cache 0.2041
12 page_waitqueue 0.2500
50 wake_up_page 0.6494
10 unlock_page 0.2703
16 end_page_writeback 0.2162
52 find_get_page 0.4643
36 find_lock_page 0.1241
48 find_get_pages_tag 0.3019
1 do_generic_mapping_read 0.0008
5 filemap_nopage 0.0053
7 remove_suid 0.0470
126 generic_file_aio_write_nolock 0.0435
25 generic_file_aio_write 0.1582
39 mempool_alloc 0.1024
21 mempool_free 0.1000
1 mempool_free_slab 0.0476
30 bad_range 0.3000
17 prep_new_page 0.2099
56 free_hot_cold_page 0.1879
157 buffered_rmqueue 0.2951
82 __alloc_pages 0.0923
5 __get_free_pages 0.0847
14 __free_pages 0.1795
6 free_pages 0.1333
1 __read_page_state 0.0179
9 balance_dirty_pages_ratelimited 0.0448
1 background_writeout 0.0058
1 do_writepages 0.0149
61 __set_page_dirty_nobuffers 0.1622
8 clear_page_dirty_for_io 0.0702
49 test_clear_page_writeback 0.1922
53 test_set_page_writeback 0.1767
9 .text.lock.page_writeback 0.0703
1 vma_prio_tree_add 0.0060
2 do_page_cache_readahead 0.0043
4 kmem_getpages 0.0191
5 cache_init_objs 0.0481
1 kmem_flagcheck 0.0233
1 set_slab_attr 0.0172
3 cache_grow 0.0066
161 kmem_cache_alloc 1.3089
75 __kmalloc 0.4870
171 kmem_cache_free 1.9655
217 kfree 1.9908
28 mark_page_accessed 0.5385
1 lru_cache_add_active 0.0105
1 lru_add_drain 0.0089
37 release_pages 0.0819
2 __pagevec_release 0.0370
32 __pagevec_lru_add 0.1009
5 __pagevec_lru_add_active 0.0149
14 percpu_counter_mod 0.0800
2 blk_queue_bounce 0.0260
4 clear_page_tables 0.0130
1 pte_alloc_map 0.0040
11 copy_page_range 0.0142
27 zap_pte_range 0.0389
1 unmap_page_range 0.0088
22 do_wp_page 0.0259
19 do_anonymous_page 0.0423
299 do_no_page 0.3356
8 handle_mm_fault 0.0182
1 remove_vm_struct 0.0062
1 find_vma_prepare 0.0099
2 vma_link 0.0061
1 do_mmap_pgoff 0.0005
2 find_vma 0.0247
2 do_munmap 0.0044
1 mprotect_fixup 0.0027
2 anon_vma_unlink 0.0128
2 page_add_anon_rmap 0.0061
4 page_add_file_rmap 0.0185
7 page_remove_rmap 0.0293
1 free_pages_and_swap_cache 0.0076
1 dentry_open 0.0025
1 put_unused_fd 0.0081
10 do_sync_read 0.0578
72 vfs_read 0.2441
29 do_sync_write 0.1676
66 vfs_write 0.2237
37 sys_read 0.2891
36 sys_write 0.2812
47 fput 2.2381
116 fget 0.9062
42 fget_light 0.2515
3 bh_waitq_head 0.1364
48 wake_up_buffer 0.5783
44 unlock_buffer 1.8333
9 __set_page_buffers 0.3600
70 end_buffer_write_sync 0.5785
13 __find_get_block_slow 0.0300
23 end_buffer_async_write 0.0744
2 buffer_insert_list 0.0134
24 create_buffers 0.1472
2 grow_dev_page 0.0049
6 mark_buffer_dirty 0.1500
41 __brelse 0.5256
2 bh_lru_install 0.0088
171 __find_get_block 0.6951
29 __getblk 0.2871
11 __bread 0.1667
3 set_bh_page 0.0556
25 create_empty_buffers 0.1238
6 unmap_underlying_metadata 0.0531
58 __block_write_full_page 0.0604
51 __block_prepare_write 0.0514
29 __block_commit_write 0.1946
12 block_prepare_write 0.1558
9 generic_commit_write 0.0608
19 block_write_full_page 0.0712
27 end_bio_bh_io_sync 0.4355
42 submit_bh 0.1288
27 ll_rw_block 0.1985
4 recalc_bh_state 0.0223
6 alloc_buffer_head 0.0571
17 init_buffer_head 0.3208
10 .text.lock.buffer 0.0538
8 bio_destructor 0.0851
16 bio_alloc 0.0376
65 bio_put 1.2037
4 bio_endio 0.0315
1 sys_fstat64 0.0175
9 inode_add_bytes 0.0634
1 copy_strings 0.0018
1 kernel_read 0.0105
1 do_execve 0.0015
58 pipe_wait 0.3240
216 pipe_readv 0.3055
7 pipe_read 0.1186
157 pipe_writev 0.1894
4 pipe_write 0.0678
162 pipe_poll 1.3065
1 .text.lock.pipe 0.0050
1 getname 0.0041
2 deny_write_access 0.0145
3 do_lookup 0.0186
2 link_path_walk 0.0005
1 __user_walk 0.0081
16 kill_fasync 0.1379
1 poll_initwait 0.0400
27 poll_freewait 0.4219
33 __pollwait 0.1675
20 max_select_fd 0.0889
167 do_select 0.2136
8 select_bits_alloc 0.2500
1 select_bits_free 0.2000
104 sys_select 0.0806
1 locks_remove_flock 0.0039
9 __d_lookup 0.0216
112 inode_times_differ 1.6471
25 update_atime 0.1126
48 inode_update_time 0.2243
192 dnotify_parent 0.7589
54 __mark_inode_dirty 0.1116
1 sync_sb_inodes 0.0013
1 writeback_inodes 0.0025
1 writeback_acquire 0.0455
12 mpage_writepages 0.0143
1 proc_pid_unhash 0.0063
7 show_stat 0.0031
6 write_profile 0.0385
30 ext3_get_group_desc 0.1596
8 read_block_bitmap 0.0526
1 find_next_usable_block 0.0012
67 ext3_try_to_allocate 0.0753
10 ext3_has_free_blocks 0.0813
56 ext3_new_block 0.0431
1 ext3_release_file 0.0098
1 ext3_open_file 0.0222
23 ext3_file_write 0.1080
3 ext3_alloc_block 0.0508
23 ext3_block_to_path 0.0723
33 ext3_get_branch 0.1398
21 ext3_find_goal 0.1533
15 ext3_alloc_branch 0.0217
39 ext3_splice_branch 0.0823
69 ext3_get_block_handle 0.0776
13 ext3_get_block 0.0747
83 walk_page_buffers 0.7411
24 ext3_prepare_write 0.0755
4 ext3_journal_dirty_data 0.0430
130 ext3_ordered_commit_write 0.5677
10 bget_one 0.9091
4 journal_dirty_data_fn 0.2105
12 ext3_ordered_writepage 0.0260
84 ext3_get_inode_block 0.3218
28 ext3_get_inode_loc 0.0440
102 ext3_do_update_inode 0.1105
27 ext3_writepage_trans_blocks 0.1901
30 ext3_mark_iloc_dirty 0.5455
26 ext3_reserve_inode_write 0.1232
17 ext3_mark_inode_dirty 0.2099
21 ext3_dirty_inode 0.1556
25 ext3_journal_start 0.2809
7 __ext3_journal_stop 0.0875
65 start_this_handle 0.0507
21 new_handle 0.2625
34 journal_start 0.1619
161 do_get_write_access 0.0901
26 journal_get_write_access 0.3210
1 journal_get_create_access 0.0015
26 journal_get_undo_access 0.0627
47 journal_dirty_data 0.0643
123 journal_dirty_metadata 0.2240
112 journal_stop 0.1376
105 __journal_unfile_buffer 0.2031
37 __journal_file_buffer 0.0528
1 __journal_refile_buffer 0.0029
30 inverted_lock 0.3125
249 journal_commit_transaction 0.0408
2 journal_brelse_array 0.0455
1 __journal_clean_checkpoint_list 0.0057
57 journal_cancel_revoke 0.1643
11 __log_space_left 0.1068
2 journal_get_descriptor_buffer 0.0106
3 journal_blocks_per_page 0.1034
12 journal_alloc_journal_head 0.1034
279 journal_add_journal_head 0.7623
50 __journal_remove_journal_head 0.1256
71 journal_remove_journal_head 0.5726
102 journal_put_journal_head 0.4574
1 sys_semtimedop 0.0008
1 kobject_put 0.0400
7 radix_tree_preload 0.0395
1 __rb_erase_color 0.0024
1 number 0.0013
6 vsnprintf 0.0049
1 find_next_bit 0.0066
25 csum_partial 0.0868
2 atomic_dec_and_lock 0.0172
1 bad_get_user 0.1111
19 memcpy 0.3333
2 strncpy_from_user 0.0202
1 clear_user 0.0096
6073 __copy_user_intel 35.3081
4281 __copy_user_zeroing_intel 24.8895
108 __copy_to_user_ll 0.9000
2419 __copy_from_user_ll 20.1583
34 copy_to_user 0.3469
29 copy_from_user 0.2302
2 add_disk_randomness 0.0385
9 blk_run_queue 0.0726
5 get_request 0.0067
112 __make_request 0.0806
11 generic_make_request 0.0224
7 submit_bio 0.0242
70 __end_that_request_first 0.1331
6 put_io_context 0.0682
11 .text.lock.ll_rw_blk 0.0809
2 as_set_request 0.0153
306 e1000_xmit_frame 0.1438
5 scsi_put_command 0.0221
31 scsi_dispatch_cmd 0.0575
1 scsi_softirq 0.0048
2 scsi_finish_command 0.0112
1 scsi_add_timer 0.0093
3 scsi_decide_disposition 0.0100
2 scsi_init_cmd_errh 0.0104
15 scsi_device_unbusy 0.0815
5 scsi_run_queue 0.0182
29 scsi_end_request 0.1098
1 scsi_free_sgtable 0.0222
6 scsi_io_completion 0.0059
168 scsi_request_fn 0.1548
10 .text.lock.scsi_lib 0.0467
60 ahc_linux_isr 0.0863
1 sd_rw_intr 0.0015
11 sock_aio_read 0.0374
17 sock_aio_write 0.0578
33 sock_poll 0.6735
187 sock_wfree 2.4605
10 __release_sock 0.0649
300 sock_def_readable 1.6667
17 sk_reset_timer 0.3617
75 lock_sock 0.6048
40 release_sock 0.2632
25 sock_common_recvmsg 0.2137
186 .text.lock.sock 1.5000
241 alloc_skb 1.0759
369 skb_release_data 2.1207
15 kfree_skbmem 0.3409
682 __kfree_skb 2.9912
64 skb_clone 0.1667
31 copy_skb_header 0.1250
18 skb_copy 0.0811
296 skb_copy_bits 0.5627
6 skb_checksum 0.0097
26 memcpy_toiovec 0.2737
152 skb_copy_datagram_iovec 0.2804
32 sk_stream_rfree 1.1852
10 sk_stream_error 0.1075
1 sk_stream_mem_schedule 0.0029
17 skb_checksum_help 0.0525
97 dev_queue_xmit 0.1219
23 net_tx_action 0.0597
59 netif_receive_skb 0.1326
98 process_backlog 0.3590
58 net_rx_action 0.2239
187 nf_iterate 1.1333
96 nf_hook_slow 0.3357
162 qdisc_restart 0.2746
130 pfifo_fast_enqueue 0.8966
111 pfifo_fast_dequeue 1.3704
27 rt_hash_code 0.1971
257 ip_route_input 0.4724
43 ip_local_deliver 0.0850
196 ip_rcv 0.1534
64 ip_local_deliver_finish 0.1641
38 ip_rcv_finish 0.0586
29 ip_finish_output 0.0545
27 ip_output 0.2250
353 ip_queue_xmit 0.2341
12 dst_output 0.2927
220 ip_finish_output2 0.4966
121 tcp_poll 0.3151
177 tcp_sendmsg 0.0389
47 cleanup_rbuf 0.1526
411 tcp_recvmsg 0.2047
34 __tcp_grow_window 0.2595
54 tcp_rcv_rtt_update 0.5934
39 tcp_rcv_space_adjust 0.1196
208 tcp_event_data_recv 0.3478
24 tcp_rtt_estimator 0.0714
1 tcp_time_to_recover 0.0017
11 tcp_ack_saw_tstamp 0.1310
67 tcp_clean_rtx_queue 0.0705
37 tcp_ack 0.0253
1 tcp_sack_new_ofo_skb 0.0035
3 tcp_data_queue 0.0010
11 __tcp_data_snd_check 0.0466
58 __tcp_ack_snd_check 0.3602
1 tcp_check_urg 0.0036
772 tcp_rcv_established 0.3342
249 tcp_transmit_skb 0.1429
40 tcp_write_xmit 0.0552
42 __tcp_select_window 0.1414
21 tcp_send_delayed_ack 0.0950
62 tcp_send_ack 0.3179
483 __tcp_v4_lookup_established 1.8295
33 tcp_v4_send_check 0.1304
40 tcp_v4_do_rcv 0.1258
439 tcp_v4_rcv 0.2230
6 inet_sendmsg 0.0674
23 ip_confirm 0.4792
24 ip_conntrack_defrag 0.3429
18 ip_refrag 0.1259
11 ip_conntrack_local 0.0753
25 __ip_ct_find_proto 0.3846
278 ip_ct_find_proto 2.5981
66 hash_conntrack 0.4286
37 get_tuple 0.3978
4 invert_tuple 0.0851
67 __ip_conntrack_find 0.3059
158 ip_conntrack_find_get 1.3058
9 ip_conntrack_get 0.2500
114 ip_conntrack_in 0.1854
267 ip_ct_refresh 1.4053
32 tcp_pkt_to_tuple 0.3765
10 get_conntrack_index 0.1639
434 tcp_packet 0.9175
908 ipt_do_table 0.9370
10 ipt_hook 0.1695
13 ipt_local_out_hook 0.1111
25 ipt_route_hook 0.4237
21 ipt_local_hook 0.0901
62 ip_nat_fn 0.1030
16 ip_nat_out 0.0930
199 do_bindings 0.2442
8 __sched_text_start 0.0276
1 __down_failed 0.0833
2 __write_lock_failed 0.0625
2 __read_lock_failed 0.1000
231 schedule 0.1110
2 io_schedule 0.0417
1 io_schedule_timeout 0.0196
1 __preempt_spin_lock 0.0110
458 .text.lock.sched 2.3367
9 schedule_timeout 0.0471
461152 total 0.1499
[-- Attachment #3: rplog-rcu --]
[-- Type: text/plain, Size: 241348 bytes --]
18 default_idle 0.4000
433553 poll_idle 7475.0517
41 cpu_idle 0.6406
1 sys_execve 0.0064
10 need_resched 0.2000
114 system_call 2.5909
2 syscall_call 0.1818
15 syscall_exit 1.3636
8 handle_IRQ_event 0.0800
1 old_mmap 0.0030
3 sys_ipc 0.0048
1 save_i387 0.0037
1 show_cpuinfo 0.0015
36 sched_clock 0.2647
2 delay_tsc 0.0952
1 flush_tlb_others 0.0033
3 flush_tlb_mm 0.0176
4 flush_tlb_page 0.0226
9 pte_alloc_one 0.1169
37 do_page_fault 0.0265
2 try_to_wake_up 0.0030
1 wake_up_forked_process 0.0028
1740 finish_task_switch 11.6779
1 schedule_tail 0.0098
2 nr_iowait 0.0174
1007 __wake_up 8.5339
1 migration_thread 0.0022
210 __might_sleep 0.9953
1 __put_task_struct 0.0038
85 add_wait_queue 0.6855
47 remove_wait_queue 0.3672
30 prepare_to_wait 0.2041
46 finish_wait 0.2911
1 dup_task_struct 0.0047
1 mm_init 0.0040
15 copy_mm 0.0129
3 copy_files 0.0031
5 copy_process 0.0017
8 .text.lock.fork 0.0404
3 release_task 0.0055
6 exit_notify 0.0026
1 sys_wait4 0.0016
99 current_kernel_time 1.4348
5108 __do_softirq 29.5260
159 local_bh_enable 1.1042
653 __mod_timer 1.2631
65 mod_timer 0.6132
326 del_timer 2.1169
1 del_timer_sync 0.0055
3 .text.lock.timer 0.0326
1 get_signal_to_deliver 0.0010
3 sigprocmask 0.0124
1 do_sigaction 0.0016
1 set_user 0.0075
3 groups_search 0.0283
1 sys_setgroups 0.0062
4 in_group_p 0.0333
1 worker_thread 0.0013
1 alloc_pidmap 0.0026
68 add_to_page_cache 0.2776
17 page_waitqueue 0.3542
62 wake_up_page 0.8052
22 unlock_page 0.5946
8 end_page_writeback 0.1081
77 find_get_page 0.6875
47 find_lock_page 0.1621
41 find_get_pages_tag 0.2579
1 do_generic_mapping_read 0.0008
1 __generic_file_aio_read 0.0018
21 filemap_nopage 0.0223
1 generic_file_mmap 0.0145
7 remove_suid 0.0470
120 generic_file_aio_write_nolock 0.0414
15 generic_file_aio_write 0.0949
50 mempool_alloc 0.1312
13 mempool_free 0.0619
3 mempool_alloc_slab 0.1429
1 mempool_free_slab 0.0476
20 bad_range 0.2000
18 prep_new_page 0.2222
50 free_hot_cold_page 0.1678
1 free_hot_page 0.1429
180 buffered_rmqueue 0.3383
71 __alloc_pages 0.0800
4 __get_free_pages 0.0678
10 __free_pages 0.1282
3 free_pages 0.0667
1 get_dirty_limits 0.0049
13 balance_dirty_pages_ratelimited 0.0647
56 __set_page_dirty_nobuffers 0.1489
7 set_page_dirty 0.0833
9 clear_page_dirty_for_io 0.0789
43 test_clear_page_writeback 0.1686
51 test_set_page_writeback 0.1700
20 .text.lock.page_writeback 0.1562
1 prio_tree_expand 0.0048
1 prio_tree_replace 0.0101
1 prio_tree_insert 0.0026
1 do_page_cache_readahead 0.0022
1 page_cache_readahead 0.0018
1 max_sane_readahead 0.0156
3 kmem_getpages 0.0144
6 cache_init_objs 0.0577
4 kmem_flagcheck 0.0930
8 cache_grow 0.0176
158 kmem_cache_alloc 1.2846
2 kmem_ptr_validate 0.0263
73 __kmalloc 0.4740
187 kmem_cache_free 2.1494
232 kfree 2.1284
4 activate_page 0.0177
30 mark_page_accessed 0.5769
1 lru_cache_add_active 0.0105
47 release_pages 0.1040
1 __pagevec_release 0.0185
39 __pagevec_lru_add 0.1230
13 __pagevec_lru_add_active 0.0387
1 vm_acct_memory 0.0122
14 percpu_counter_mod 0.0800
1 blk_queue_bounce 0.0130
11 clear_page_tables 0.0357
7 pte_alloc_map 0.0282
102 copy_page_range 0.1316
81 zap_pte_range 0.1167
1 unmap_page_range 0.0088
3 unmap_vmas 0.0052
116 do_wp_page 0.1368
82 do_anonymous_page 0.1826
809 do_no_page 0.9080
19 handle_mm_fault 0.0433
5 remove_vm_struct 0.0309
1 find_vma_prepare 0.0099
2 vma_link 0.0061
2 vma_adjust 0.0020
1 can_vma_merge_after 0.0090
1 find_mergeable_anon_vma 0.0043
4 do_mmap_pgoff 0.0022
9 find_vma 0.1111
1 free_pgtables 0.0067
3 split_vma 0.0105
1 do_munmap 0.0022
1 sys_munmap 0.0082
2 anon_vma_prepare 0.0057
4 anon_vma_link 0.0315
4 anon_vma_unlink 0.0256
4 page_add_anon_rmap 0.0122
6 page_add_file_rmap 0.0278
36 page_remove_rmap 0.1506
1 shmem_swp_entry 0.0047
30 shmem_getpage 0.0119
5 free_pages_and_swap_cache 0.0382
1 sys_access 0.0030
2 dentry_open 0.0051
2 get_unused_fd 0.0078
2 filp_close 0.0149
2 sys_close 0.0098
1 generic_file_llseek 0.0046
11 do_sync_read 0.0636
57 vfs_read 0.1932
24 do_sync_write 0.1387
50 vfs_write 0.1695
27 sys_read 0.2109
30 sys_write 0.2344
45 fput 2.1429
2 __fput 0.0094
108 fget 0.8438
35 fget_light 0.2096
2 file_kill 0.0163
2 bh_waitq_head 0.0909
63 wake_up_buffer 0.7590
32 unlock_buffer 1.3333
1 __wait_on_buffer 0.0061
8 __set_page_buffers 0.3200
69 end_buffer_write_sync 0.5702
17 __find_get_block_slow 0.0393
16 end_buffer_async_write 0.0518
7 create_buffers 0.0429
1 init_page_buffers 0.0089
5 mark_buffer_dirty 0.1250
40 __brelse 0.5128
1 bh_lru_install 0.0044
147 __find_get_block 0.5976
13 __getblk 0.1287
10 __bread 0.1515
6 set_bh_page 0.1111
22 create_empty_buffers 0.1089
3 unmap_underlying_metadata 0.0265
43 __block_write_full_page 0.0448
44 __block_prepare_write 0.0443
20 __block_commit_write 0.1342
7 block_prepare_write 0.0909
9 generic_commit_write 0.0608
18 block_write_full_page 0.0674
23 end_bio_bh_io_sync 0.3710
34 submit_bh 0.1043
28 ll_rw_block 0.2059
4 recalc_bh_state 0.0223
21 alloc_buffer_head 0.2000
28 init_buffer_head 0.5283
2 .text.lock.buffer 0.0108
5 bio_destructor 0.0532
21 bio_alloc 0.0493
53 bio_put 0.9815
2 __bio_add_page 0.0059
7 bio_endio 0.0551
1 generic_fillattr 0.0061
1 cp_new_stat64 0.0037
15 inode_add_bytes 0.1056
1 copy_strings 0.0018
1 setup_arg_pages 0.0021
48 pipe_wait 0.2682
235 pipe_readv 0.3324
7 pipe_read 0.1186
154 pipe_writev 0.1858
9 pipe_write 0.1525
123 pipe_poll 0.9919
1 getname 0.0041
1 deny_write_access 0.0072
1 path_release 0.0143
2 do_lookup 0.0124
7 link_path_walk 0.0018
1 path_lookup 0.0021
1 __user_walk 0.0081
4 may_open 0.0078
2 open_namei 0.0011
1 sys_dup2 0.0030
10 kill_fasync 0.0862
3 poll_initwait 0.1200
37 poll_freewait 0.5781
31 __pollwait 0.1574
18 max_select_fd 0.0800
136 do_select 0.1739
5 select_bits_alloc 0.1562
1 select_bits_free 0.2000
98 sys_select 0.0760
1 flock64_to_posix_lock 0.0027
1 locks_remove_posix 0.0030
3 dput 0.0049
12 __d_lookup 0.0288
1 inode_init_once 0.0032
1 iget_locked 0.0039
130 inode_times_differ 1.9118
29 update_atime 0.1306
32 inode_update_time 0.1495
1 wake_up_inode 0.0139
200 dnotify_parent 0.7905
1 lookup_mnt 0.0060
1 seq_read 0.0014
1 seq_printf 0.0116
64 __mark_inode_dirty 0.1322
1 sync_sb_inodes 0.0013
1 writeback_release 0.0208
4 mpage_end_io_read 0.0336
3 do_mpage_readpage 0.0025
13 mpage_writepages 0.0155
1 set_brk 0.0104
1 load_elf_binary 0.0003
1 proc_lookup 0.0030
10 show_stat 0.0044
7 write_profile 0.0449
27 ext3_get_group_desc 0.1436
7 read_block_bitmap 0.0461
3 find_next_usable_block 0.0035
61 ext3_try_to_allocate 0.0685
11 ext3_has_free_blocks 0.0894
50 ext3_new_block 0.0385
1 ext3_check_dir_entry 0.0043
26 ext3_file_write 0.1221
2 ext3_alloc_block 0.0339
19 ext3_block_to_path 0.0597
42 ext3_get_branch 0.1780
20 ext3_find_goal 0.1460
12 ext3_alloc_branch 0.0173
35 ext3_splice_branch 0.0738
72 ext3_get_block_handle 0.0810
13 ext3_get_block 0.0747
66 walk_page_buffers 0.5893
3 do_journal_get_write_access 0.0248
24 ext3_prepare_write 0.0755
4 ext3_journal_dirty_data 0.0430
108 ext3_ordered_commit_write 0.4716
7 bget_one 0.6364
2 bput_one 0.1818
2 journal_dirty_data_fn 0.1053
8 ext3_ordered_writepage 0.0174
69 ext3_get_inode_block 0.2644
25 ext3_get_inode_loc 0.0393
1 ext3_read_inode 0.0014
77 ext3_do_update_inode 0.0834
25 ext3_writepage_trans_blocks 0.1761
35 ext3_mark_iloc_dirty 0.6364
23 ext3_reserve_inode_write 0.1090
14 ext3_mark_inode_dirty 0.1728
19 ext3_dirty_inode 0.1407
2 ext3_find_entry 0.0018
1 ext3_lookup 0.0051
31 ext3_journal_start 0.3483
17 __ext3_journal_stop 0.2125
1 ext3_follow_link 0.0357
69 start_this_handle 0.0538
15 new_handle 0.1875
24 journal_start 0.1143
173 do_get_write_access 0.0969
17 journal_get_write_access 0.2099
27 journal_get_undo_access 0.0651
79 journal_dirty_data 0.1081
99 journal_dirty_metadata 0.1803
87 journal_stop 0.1069
82 __journal_unfile_buffer 0.1586
39 __journal_file_buffer 0.0556
1 journal_end_buffer_io_sync 0.0323
22 inverted_lock 0.2292
255 journal_commit_transaction 0.0418
3 journal_brelse_array 0.0682
65 journal_cancel_revoke 0.1873
1 journal_write_revoke_records 0.0039
1 journal_write_metadata_buffer 0.0011
9 __log_space_left 0.0874
3 journal_blocks_per_page 0.1034
1 __jbd_kmalloc 0.0227
5 journal_alloc_journal_head 0.0431
4 journal_free_journal_head 0.1429
294 journal_add_journal_head 0.8033
50 __journal_remove_journal_head 0.1256
65 journal_remove_journal_head 0.5242
120 journal_put_journal_head 0.5381
1 try_atomic_semop 0.0034
1 __bitmap_weight 0.0035
1 kobject_put 0.0400
13 radix_tree_preload 0.0734
1 rb_erase 0.0044
1 rwsem_wake 0.0031
8 number 0.0102
12 vsnprintf 0.0097
29 csum_partial 0.1007
2 atomic_dec_and_lock 0.0172
14 memcpy 0.2456
1 strncpy_from_user 0.0101
2 strnlen_user 0.0208
6256 __copy_user_intel 36.3721
4257 __copy_user_zeroing_intel 24.7500
103 __copy_to_user_ll 0.8583
2222 __copy_from_user_ll 18.5167
27 copy_to_user 0.2755
30 copy_from_user 0.2381
2 add_disk_randomness 0.0385
1 put_device 0.1000
1 generic_unplug_device 0.0100
1 get_request 0.0013
108 __make_request 0.0777
18 generic_make_request 0.0366
12 submit_bio 0.0415
78 __end_that_request_first 0.1483
1 put_io_context 0.0114
11 .text.lock.ll_rw_blk 0.0809
297 e1000_xmit_frame 0.1396
2 e1000_update_stats 0.0013
5 scsi_put_command 0.0221
20 scsi_dispatch_cmd 0.0371
4 scsi_finish_command 0.0223
1 scsi_add_timer 0.0093
2 scsi_decide_disposition 0.0067
3 scsi_init_cmd_errh 0.0156
8 scsi_device_unbusy 0.0435
2 scsi_run_queue 0.0073
21 scsi_end_request 0.0795
1 scsi_io_completion 0.0010
125 scsi_request_fn 0.1152
13 .text.lock.scsi_lib 0.0607
48 ahc_linux_isr 0.0691
2 sd_rw_intr 0.0030
21 sock_aio_read 0.0714
11 sock_aio_write 0.0374
23 sock_poll 0.4694
1 sk_alloc 0.0056
196 sock_wfree 2.2791
20 __release_sock 0.1299
343 sock_def_readable 1.9056
10 sk_reset_timer 0.2128
78 lock_sock 0.6290
45 release_sock 0.2961
34 sock_common_recvmsg 0.2906
177 .text.lock.sock 1.4508
250 alloc_skb 1.1161
435 skb_release_data 2.5000
13 kfree_skbmem 0.2955
649 __kfree_skb 2.8465
89 skb_clone 0.2318
41 copy_skb_header 0.1653
17 skb_copy 0.0766
276 skb_copy_bits 0.5247
9 skb_checksum 0.0146
17 memcpy_toiovec 0.1789
1 memcpy_tokerneliovec 0.0133
123 skb_copy_datagram_iovec 0.2269
39 sk_stream_rfree 1.4444
8 sk_stream_error 0.0860
11 skb_checksum_help 0.0340
69 dev_queue_xmit 0.0867
24 net_tx_action 0.0623
50 netif_receive_skb 0.1124
124 process_backlog 0.4542
45 net_rx_action 0.1737
20 neigh_resolve_output 0.0380
202 nf_iterate 1.2242
119 nf_hook_slow 0.4161
6 eth_header 0.0192
154 qdisc_restart 0.2610
153 pfifo_fast_enqueue 1.0552
109 pfifo_fast_dequeue 1.3457
31 rt_hash_code 0.2263
308 ip_route_input 0.5662
28 ip_local_deliver 0.0553
149 ip_rcv 0.1166
74 ip_local_deliver_finish 0.1897
42 ip_rcv_finish 0.0648
20 ip_finish_output 0.0376
19 ip_output 0.1583
278 ip_queue_xmit 0.1844
7 dst_output 0.1707
218 ip_finish_output2 0.4921
113 tcp_poll 0.2943
158 tcp_sendmsg 0.0348
62 cleanup_rbuf 0.2013
406 tcp_recvmsg 0.2022
25 __tcp_grow_window 0.1908
70 tcp_rcv_rtt_update 0.7692
33 tcp_rcv_space_adjust 0.1012
199 tcp_event_data_recv 0.3328
39 tcp_rtt_estimator 0.1161
5 tcp_ack_saw_tstamp 0.0595
63 tcp_clean_rtx_queue 0.0662
49 tcp_ack 0.0335
1 tcp_data_queue 0.0003
5 __tcp_data_snd_check 0.0212
47 __tcp_ack_snd_check 0.2919
681 tcp_rcv_established 0.2948
314 tcp_transmit_skb 0.1803
42 tcp_write_xmit 0.0579
48 __tcp_select_window 0.1616
23 tcp_send_delayed_ack 0.1041
72 tcp_send_ack 0.3692
218 __tcp_v4_lookup_established 0.8755
28 tcp_v4_send_check 0.1107
34 tcp_v4_do_rcv 0.1069
437 tcp_v4_rcv 0.2166
4 inet_sendmsg 0.0449
20 ip_confirm 0.4167
24 ip_conntrack_defrag 0.3429
22 ip_refrag 0.1538
33 ip_conntrack_local 0.2260
19 __ip_ct_find_proto 0.2923
276 ip_ct_find_proto 2.5794
54 hash_conntrack 0.3506
43 get_tuple 0.4624
1 invert_tuple 0.0213
60 __ip_conntrack_find 0.2740
151 ip_conntrack_find_get 1.2479
13 ip_conntrack_get 0.3611
138 ip_conntrack_in 0.2244
311 ip_ct_refresh 1.6368
19 tcp_pkt_to_tuple 0.2235
13 get_conntrack_index 0.2131
464 tcp_packet 0.9810
958 ipt_do_table 0.9886
10 ipt_hook 0.1695
14 ipt_local_out_hook 0.1197
25 ipt_route_hook 0.4237
33 ipt_local_hook 0.1416
67 ip_nat_fn 0.1113
21 ip_nat_out 0.1221
179 do_bindings 0.2196
1 unix_create1 0.0023
3 __sched_text_start 0.0103
2 __write_lock_failed 0.0625
1 __read_lock_failed 0.0500
202 schedule 0.0971
3 __preempt_spin_lock 0.0330
421 .text.lock.sched 2.1480
477836 total 0.1553
24 default_idle 0.5333
455407 poll_idle 7851.8448
38 cpu_idle 0.5938
9 need_resched 0.1800
135 system_call 3.0682
6 syscall_call 0.5455
9 syscall_exit 0.8182
5 handle_IRQ_event 0.0500
1 show_cpuinfo 0.0015
44 sched_clock 0.3235
1 flush_tlb_mm 0.0059
1 flush_tlb_page 0.0056
4 pte_alloc_one 0.0519
21 do_page_fault 0.0150
1689 finish_task_switch 11.3356
1043 __wake_up 8.8390
208 __might_sleep 0.9858
75 add_wait_queue 0.6048
58 remove_wait_queue 0.4531
35 prepare_to_wait 0.2381
44 finish_wait 0.2785
2 copy_mm 0.0017
3 .text.lock.fork 0.0152
119 current_kernel_time 1.7246
5118 __do_softirq 29.5838
151 local_bh_enable 1.0486
622 __mod_timer 1.2031
84 mod_timer 0.7925
276 del_timer 1.7922
1 .text.lock.timer 0.0109
2 flush_signal_handlers 0.0299
1 get_signal_to_deliver 0.0010
2 in_group_p 0.0167
1 hash_futex 0.0071
47 add_to_page_cache 0.1918
15 page_waitqueue 0.3125
40 wake_up_page 0.5195
14 unlock_page 0.3784
20 end_page_writeback 0.2703
40 find_get_page 0.3571
45 find_lock_page 0.1552
47 find_get_pages_tag 0.2956
2 do_generic_mapping_read 0.0016
5 filemap_nopage 0.0053
7 remove_suid 0.0470
138 generic_file_aio_write_nolock 0.0476
19 generic_file_aio_write 0.1203
36 mempool_alloc 0.0945
14 mempool_free 0.0667
3 mempool_alloc_slab 0.1429
23 bad_range 0.2300
19 prep_new_page 0.2346
45 free_hot_cold_page 0.1510
163 buffered_rmqueue 0.3064
68 __alloc_pages 0.0766
3 __get_free_pages 0.0508
1 get_zeroed_page 0.0098
1 __pagevec_free 0.0303
13 __free_pages 0.1667
5 free_pages 0.1111
6 balance_dirty_pages_ratelimited 0.0299
1 do_writepages 0.0149
60 __set_page_dirty_nobuffers 0.1596
1 set_page_dirty 0.0119
26 clear_page_dirty_for_io 0.2281
54 test_clear_page_writeback 0.2118
52 test_set_page_writeback 0.1733
13 .text.lock.page_writeback 0.1016
2 prio_tree_insert 0.0052
1 file_ra_state_init 0.0270
2 do_page_cache_readahead 0.0043
1 page_cache_readahead 0.0018
1 kmem_getpages 0.0048
2 alloc_slabmgmt 0.0208
11 cache_init_objs 0.1058
4 cache_grow 0.0088
136 kmem_cache_alloc 1.1057
3 kmem_ptr_validate 0.0395
66 __kmalloc 0.4286
210 kmem_cache_free 2.4138
184 kfree 1.6881
33 mark_page_accessed 0.6346
27 release_pages 0.0597
1 __pagevec_release 0.0185
20 __pagevec_lru_add 0.0631
3 __pagevec_lru_add_active 0.0089
20 percpu_counter_mod 0.1143
3 blk_queue_bounce 0.0390
1 clear_page_tables 0.0032
1 pte_alloc_map 0.0040
7 copy_page_range 0.0090
29 zap_pte_range 0.0418
2 unmap_vmas 0.0035
26 do_wp_page 0.0307
16 do_anonymous_page 0.0356
341 do_no_page 0.3827
7 handle_mm_fault 0.0159
1 find_vma_prepare 0.0099
1 can_vma_merge_after 0.0090
3 do_mmap_pgoff 0.0016
1 get_unmapped_area 0.0034
1 find_vma 0.0123
1 find_vma_prev 0.0123
1 free_pgtables 0.0067
1 detach_vmas_to_be_unmapped 0.0101
1 anon_vma_unlink 0.0064
2 page_add_file_rmap 0.0093
13 page_remove_rmap 0.0544
1 dentry_open 0.0025
2 get_unused_fd 0.0078
11 do_sync_read 0.0636
54 vfs_read 0.1831
18 do_sync_write 0.1040
40 vfs_write 0.1356
29 sys_read 0.2266
24 sys_write 0.1875
2 get_empty_filp 0.0085
44 fput 2.0952
1 __fput 0.0047
104 fget 0.8125
57 fget_light 0.3413
1 put_filp 0.0175
2 file_move 0.0142
4 bh_waitq_head 0.1818
84 wake_up_buffer 1.0120
43 unlock_buffer 1.7917
8 __set_page_buffers 0.3200
96 end_buffer_write_sync 0.7934
15 __find_get_block_slow 0.0346
23 end_buffer_async_write 0.0744
1 buffer_insert_list 0.0067
22 create_buffers 0.1350
1 grow_dev_page 0.0025
3 mark_buffer_dirty 0.0750
49 __brelse 0.6282
156 __find_get_block 0.6341
21 __getblk 0.2079
17 __bread 0.2576
5 set_bh_page 0.0926
20 create_empty_buffers 0.0990
3 unmap_underlying_metadata 0.0265
44 __block_write_full_page 0.0458
49 __block_prepare_write 0.0493
17 __block_commit_write 0.1141
9 block_prepare_write 0.1169
6 generic_commit_write 0.0405
25 block_write_full_page 0.0936
47 end_bio_bh_io_sync 0.7581
46 submit_bh 0.1411
28 ll_rw_block 0.2059
6 recalc_bh_state 0.0335
10 alloc_buffer_head 0.0952
24 init_buffer_head 0.4528
10 .text.lock.buffer 0.0538
4 bio_destructor 0.0426
32 bio_alloc 0.0751
66 bio_put 1.2222
6 bio_endio 0.0472
1 sys_readlink 0.0065
13 inode_add_bytes 0.0915
1 copy_strings 0.0018
52 pipe_wait 0.2905
233 pipe_readv 0.3296
9 pipe_read 0.1525
137 pipe_writev 0.1653
7 pipe_write 0.1186
1 bad_pipe_r 0.1667
131 pipe_poll 1.0565
1 pipe_release 0.0043
2 vfs_permission 0.0064
2 do_lookup 0.0124
4 link_path_walk 0.0010
3 path_lookup 0.0064
15 kill_fasync 0.1293
1 sys_ioctl 0.0015
3 poll_initwait 0.1200
40 poll_freewait 0.6250
32 __pollwait 0.1624
19 max_select_fd 0.0844
177 do_select 0.2263
2 select_bits_alloc 0.0625
2 select_bits_free 0.4000
100 sys_select 0.0775
8 __d_lookup 0.0192
103 inode_times_differ 1.5147
31 update_atime 0.1396
36 inode_update_time 0.1682
183 dnotify_parent 0.7233
1 lookup_mnt 0.0060
77 __mark_inode_dirty 0.1591
1 writeback_inodes 0.0025
11 mpage_writepages 0.0131
3 load_elf_binary 0.0009
7 show_stat 0.0031
9 write_profile 0.0577
19 ext3_get_group_desc 0.1011
7 read_block_bitmap 0.0461
4 find_next_usable_block 0.0047
54 ext3_try_to_allocate 0.0607
9 ext3_has_free_blocks 0.0732
68 ext3_new_block 0.0523
19 ext3_file_write 0.0892
5 ext3_alloc_block 0.0847
21 ext3_block_to_path 0.0660
31 ext3_get_branch 0.1314
17 ext3_find_goal 0.1241
13 ext3_alloc_branch 0.0188
44 ext3_splice_branch 0.0928
60 ext3_get_block_handle 0.0675
9 ext3_get_block 0.0517
71 walk_page_buffers 0.6339
42 ext3_prepare_write 0.1321
4 ext3_journal_dirty_data 0.0430
94 ext3_ordered_commit_write 0.4105
10 bget_one 0.9091
2 bput_one 0.1818
6 journal_dirty_data_fn 0.3158
9 ext3_ordered_writepage 0.0195
47 ext3_get_inode_block 0.1801
35 ext3_get_inode_loc 0.0550
86 ext3_do_update_inode 0.0932
28 ext3_writepage_trans_blocks 0.1972
31 ext3_mark_iloc_dirty 0.5636
28 ext3_reserve_inode_write 0.1327
13 ext3_mark_inode_dirty 0.1605
24 ext3_dirty_inode 0.1778
18 ext3_journal_start 0.2022
20 __ext3_journal_stop 0.2500
58 start_this_handle 0.0452
19 new_handle 0.2375
27 journal_start 0.1286
163 do_get_write_access 0.0913
19 journal_get_write_access 0.2346
24 journal_get_undo_access 0.0578
59 journal_dirty_data 0.0807
116 journal_dirty_metadata 0.2113
126 journal_stop 0.1548
94 __journal_unfile_buffer 0.1818
51 __journal_file_buffer 0.0728
1 journal_file_buffer 0.0046
1 journal_end_buffer_io_sync 0.0323
23 inverted_lock 0.2396
282 journal_commit_transaction 0.0462
1 journal_brelse_array 0.0227
57 journal_cancel_revoke 0.1643
1 journal_write_revoke_records 0.0039
7 __log_space_left 0.0680
7 journal_blocks_per_page 0.2414
1 __jbd_kmalloc 0.0227
7 journal_alloc_journal_head 0.0603
3 journal_free_journal_head 0.1071
258 journal_add_journal_head 0.7049
61 __journal_remove_journal_head 0.1533
64 journal_remove_journal_head 0.5161
93 journal_put_journal_head 0.4170
1 lookup_undo 0.0154
1 sys_shmget 0.0031
2 kobject_put 0.0800
9 radix_tree_preload 0.0508
5 number 0.0064
6 vsnprintf 0.0049
1 find_next_zero_bit 0.0057
38 csum_partial 0.1319
17 memcpy 0.2982
1 strnlen_user 0.0104
5953 __copy_user_intel 34.6105
4385 __copy_user_zeroing_intel 25.4942
108 __copy_to_user_ll 0.9000
2323 __copy_from_user_ll 19.3583
24 copy_to_user 0.2449
29 copy_from_user 0.2302
4 blk_run_queue 0.0323
12 get_request 0.0161
103 __make_request 0.0741
7 generic_make_request 0.0142
14 submit_bio 0.0484
38 __end_that_request_first 0.0722
5 put_io_context 0.0568
12 .text.lock.ll_rw_blk 0.0882
3 as_set_request 0.0229
311 e1000_xmit_frame 0.1461
4 scsi_put_command 0.0177
26 scsi_dispatch_cmd 0.0482
5 scsi_finish_command 0.0279
2 scsi_add_timer 0.0185
2 scsi_decide_disposition 0.0067
6 scsi_init_cmd_errh 0.0312
10 scsi_device_unbusy 0.0543
5 scsi_run_queue 0.0182
1 scsi_next_command 0.0244
40 scsi_end_request 0.1515
2 scsi_free_sgtable 0.0444
6 scsi_io_completion 0.0059
148 scsi_request_fn 0.1364
17 .text.lock.scsi_lib 0.0794
67 ahc_linux_isr 0.0964
4 sd_rw_intr 0.0060
25 sock_aio_read 0.0850
18 sock_aio_write 0.0612
27 sock_poll 0.5510
195 sock_wfree 2.2674
12 __release_sock 0.0779
387 sock_def_readable 2.1500
7 sk_reset_timer 0.1489
76 lock_sock 0.6129
41 release_sock 0.2697
32 sock_common_recvmsg 0.2735
197 .text.lock.sock 1.6148
236 alloc_skb 1.0536
413 skb_release_data 2.3736
25 kfree_skbmem 0.5682
664 __kfree_skb 2.9123
69 skb_clone 0.1797
32 copy_skb_header 0.1290
21 skb_copy 0.0946
278 skb_copy_bits 0.5285
2 skb_checksum 0.0032
20 memcpy_toiovec 0.2105
5 memcpy_tokerneliovec 0.0667
139 skb_copy_datagram_iovec 0.2565
25 sk_stream_rfree 0.9259
6 sk_stream_error 0.0645
11 skb_checksum_help 0.0340
54 dev_queue_xmit 0.0678
21 net_tx_action 0.0545
74 netif_receive_skb 0.1663
104 process_backlog 0.3810
43 net_rx_action 0.1660
201 nf_iterate 1.2182
88 nf_hook_slow 0.3077
157 qdisc_restart 0.2661
142 pfifo_fast_enqueue 0.9793
104 pfifo_fast_dequeue 1.2840
34 rt_hash_code 0.2482
3 rt_cache_get_first 0.0357
289 ip_route_input 0.5312
29 ip_local_deliver 0.0573
139 ip_rcv 0.1088
60 ip_local_deliver_finish 0.1538
53 ip_rcv_finish 0.0818
22 ip_finish_output 0.0414
21 ip_output 0.1750
276 ip_queue_xmit 0.1830
6 dst_output 0.1463
217 ip_finish_output2 0.4898
134 tcp_poll 0.3490
166 tcp_sendmsg 0.0365
46 cleanup_rbuf 0.1494
388 tcp_recvmsg 0.1932
32 __tcp_grow_window 0.2443
78 tcp_rcv_rtt_update 0.8571
50 tcp_rcv_space_adjust 0.1534
198 tcp_event_data_recv 0.3311
33 tcp_rtt_estimator 0.0982
3 tcp_ack_saw_tstamp 0.0357
75 tcp_clean_rtx_queue 0.0789
1 tcp_ack_update_window 0.0039
44 tcp_ack 0.0301
1 tcp_data_queue 0.0003
6 __tcp_data_snd_check 0.0254
52 __tcp_ack_snd_check 0.3230
1 tcp_check_urg 0.0036
749 tcp_rcv_established 0.3242
296 tcp_transmit_skb 0.1699
44 tcp_write_xmit 0.0607
46 __tcp_select_window 0.1549
16 tcp_send_delayed_ack 0.0724
56 tcp_send_ack 0.2872
212 __tcp_v4_lookup_established 0.8514
31 tcp_v4_send_check 0.1225
55 tcp_v4_do_rcv 0.1730
448 tcp_v4_rcv 0.2220
5 inet_sendmsg 0.0562
20 ip_confirm 0.4167
36 ip_conntrack_defrag 0.5143
23 ip_refrag 0.1608
14 ip_conntrack_local 0.0959
19 __ip_ct_find_proto 0.2923
280 ip_ct_find_proto 2.6168
74 hash_conntrack 0.4805
53 get_tuple 0.5699
5 invert_tuple 0.1064
65 __ip_conntrack_find 0.2968
151 ip_conntrack_find_get 1.2479
8 ip_conntrack_get 0.2222
145 ip_conntrack_in 0.2358
261 ip_ct_refresh 1.3737
7 ip_ct_gather_frags 0.0302
31 tcp_pkt_to_tuple 0.3647
8 get_conntrack_index 0.1311
498 tcp_packet 1.0529
870 ipt_do_table 0.8978
14 ipt_hook 0.2373
20 ipt_local_out_hook 0.1709
25 ipt_route_hook 0.4237
25 ipt_local_hook 0.1073
48 ip_nat_fn 0.0797
16 ip_nat_out 0.0930
194 do_bindings 0.2380
1 unix_create1 0.0023
9 __sched_text_start 0.0310
1 __write_lock_failed 0.0312
224 schedule 0.1076
3 __preempt_spin_lock 0.0330
439 .text.lock.sched 2.2398
3 schedule_timeout 0.0157
498603 total 0.1620
19 default_idle 0.4222
416167 poll_idle 7175.2931
40 cpu_idle 0.6250
8 need_resched 0.1600
123 system_call 2.7955
5 syscall_call 0.4545
7 syscall_exit 0.6364
1 error_code 0.0179
3 handle_IRQ_event 0.0300
1 sys_mmap2 0.0047
2 show_cpuinfo 0.0030
31 sched_clock 0.2279
1 delay_tsc 0.0476
1 flush_tlb_mm 0.0059
3 pte_alloc_one 0.0390
1 pgd_ctor 0.0060
17 do_page_fault 0.0122
1740 finish_task_switch 11.6779
1 wake_up_forked_thread 0.0017
1070 __wake_up 9.0678
223 __might_sleep 1.0569
87 add_wait_queue 0.7016
47 remove_wait_queue 0.3672
30 prepare_to_wait 0.2041
32 finish_wait 0.2025
2 dup_task_struct 0.0095
1 mmput 0.0067
2 copy_process 0.0007
6 .text.lock.fork 0.0303
1 profile_exec_unmap 0.0104
2 exit_notify 0.0009
117 current_kernel_time 1.6957
5156 __do_softirq 29.8035
149 local_bh_enable 1.0347
663 __mod_timer 1.2824
64 mod_timer 0.6038
294 del_timer 1.9091
1 del_timer_sync 0.0055
2 .text.lock.timer 0.0217
2 flush_signal_handlers 0.0299
1 sigprocmask 0.0041
1 do_sigaction 0.0016
1 in_group_p 0.0083
1 filemap_fdatawait 0.0286
50 add_to_page_cache 0.2041
19 page_waitqueue 0.3958
48 wake_up_page 0.6234
20 unlock_page 0.5405
13 end_page_writeback 0.1757
55 find_get_page 0.4911
38 find_lock_page 0.1310
1 find_or_create_page 0.0054
55 find_get_pages_tag 0.3459
2 do_generic_mapping_read 0.0016
2 filemap_nopage 0.0021
11 remove_suid 0.0738
137 generic_file_aio_write_nolock 0.0473
16 generic_file_aio_write 0.1013
45 mempool_alloc 0.1181
12 mempool_free 0.0571
2 mempool_alloc_slab 0.0952
3 mempool_free_slab 0.1429
17 bad_range 0.1700
16 prep_new_page 0.1975
46 free_hot_cold_page 0.1544
155 buffered_rmqueue 0.2914
56 __alloc_pages 0.0631
4 __get_free_pages 0.0678
1 __pagevec_free 0.0303
16 __free_pages 0.2051
2 free_pages 0.0444
10 balance_dirty_pages_ratelimited 0.0498
1 do_writepages 0.0149
65 __set_page_dirty_nobuffers 0.1729
3 set_page_dirty 0.0357
13 clear_page_dirty_for_io 0.1140
49 test_clear_page_writeback 0.1922
54 test_set_page_writeback 0.1800
24 .text.lock.page_writeback 0.1875
1 __pdflush 0.0018
1 prio_tree_remove 0.0044
1 do_page_cache_readahead 0.0022
1 page_cache_readahead 0.0018
2 kmem_getpages 0.0096
7 cache_init_objs 0.0673
2 cache_grow 0.0044
173 kmem_cache_alloc 1.4065
1 kmem_ptr_validate 0.0132
79 __kmalloc 0.5130
181 kmem_cache_free 2.0805
195 kfree 1.7890
30 mark_page_accessed 0.5769
1 lru_cache_add_active 0.0105
38 release_pages 0.0841
27 __pagevec_lru_add 0.0852
3 __pagevec_lru_add_active 0.0089
1 vm_acct_memory 0.0122
9 percpu_counter_mod 0.0514
1 blk_queue_bounce 0.0130
5 clear_page_tables 0.0162
4 pte_alloc_map 0.0161
7 copy_page_range 0.0090
15 zap_pte_range 0.0216
1 unmap_vmas 0.0017
20 do_wp_page 0.0236
16 do_anonymous_page 0.0356
270 do_no_page 0.3030
9 handle_mm_fault 0.0205
1 find_vma_prepare 0.0099
1 can_vma_merge_after 0.0090
3 do_mmap_pgoff 0.0016
7 find_vma 0.0864
3 split_vma 0.0105
2 do_munmap 0.0044
1 change_protection 0.0022
1 mprotect_fixup 0.0027
1 anon_vma_prepare 0.0028
1 anon_vma_link 0.0079
3 page_add_file_rmap 0.0139
12 page_remove_rmap 0.0502
1 get_unused_fd 0.0039
1 fd_install 0.0068
1 sys_close 0.0049
12 do_sync_read 0.0694
67 vfs_read 0.2271
18 do_sync_write 0.1040
47 vfs_write 0.1593
30 sys_read 0.2344
25 sys_write 0.1953
60 fput 2.8571
90 fget 0.7031
41 fget_light 0.2455
1 file_move 0.0071
3 bh_waitq_head 0.1364
66 wake_up_buffer 0.7952
37 unlock_buffer 1.5417
6 __set_page_buffers 0.2400
57 end_buffer_write_sync 0.4711
21 __find_get_block_slow 0.0485
35 end_buffer_async_write 0.1133
1 buffer_insert_list 0.0067
27 create_buffers 0.1656
5 mark_buffer_dirty 0.1250
36 __brelse 0.4615
2 bh_lru_install 0.0088
157 __find_get_block 0.6382
22 __getblk 0.2178
8 __bread 0.1212
7 set_bh_page 0.1296
18 create_empty_buffers 0.0891
5 unmap_underlying_metadata 0.0442
42 __block_write_full_page 0.0437
54 __block_prepare_write 0.0544
22 __block_commit_write 0.1477
9 block_prepare_write 0.1169
8 generic_commit_write 0.0541
12 block_write_full_page 0.0449
25 end_bio_bh_io_sync 0.4032
34 submit_bh 0.1043
36 ll_rw_block 0.2647
7 recalc_bh_state 0.0391
14 alloc_buffer_head 0.1333
24 init_buffer_head 0.4528
5 .text.lock.buffer 0.0269
7 bio_destructor 0.0745
18 bio_alloc 0.0423
56 bio_put 1.0370
4 bio_endio 0.0315
2 vfs_getattr 0.0130
1 vfs_fstat 0.0125
1 cp_new_stat64 0.0037
11 inode_add_bytes 0.0775
53 pipe_wait 0.2961
254 pipe_readv 0.3593
12 pipe_read 0.2034
162 pipe_writev 0.1954
9 pipe_write 0.1525
134 pipe_poll 1.0806
2 getname 0.0082
1 permission 0.0133
1 real_lookup 0.0038
1 do_lookup 0.0062
2 link_path_walk 0.0005
1 path_lookup 0.0021
12 kill_fasync 0.1034
4 poll_initwait 0.1600
35 poll_freewait 0.5469
37 __pollwait 0.1878
22 max_select_fd 0.0978
164 do_select 0.2097
2 select_bits_alloc 0.0625
1 select_bits_free 0.2000
102 sys_select 0.0791
1 dput 0.0016
1 __d_lookup 0.0024
1 bmap 0.0105
118 inode_times_differ 1.7353
21 update_atime 0.0946
50 inode_update_time 0.2336
189 dnotify_parent 0.7470
85 __mark_inode_dirty 0.1756
2 sync_sb_inodes 0.0026
10 mpage_writepages 0.0119
1 proc_lookup 0.0030
2 show_stat 0.0009
7 write_profile 0.0449
21 ext3_get_group_desc 0.1117
5 read_block_bitmap 0.0329
2 find_next_usable_block 0.0024
57 ext3_try_to_allocate 0.0640
15 ext3_has_free_blocks 0.1220
56 ext3_new_block 0.0431
19 ext3_file_write 0.0892
12 ext3_alloc_block 0.2034
14 ext3_block_to_path 0.0440
40 ext3_get_branch 0.1695
15 ext3_find_goal 0.1095
17 ext3_alloc_branch 0.0246
44 ext3_splice_branch 0.0928
56 ext3_get_block_handle 0.0630
8 ext3_get_block 0.0460
71 walk_page_buffers 0.6339
30 ext3_prepare_write 0.0943
7 ext3_journal_dirty_data 0.0753
103 ext3_ordered_commit_write 0.4498
8 bget_one 0.7273
1 bput_one 0.0909
11 journal_dirty_data_fn 0.5789
5 ext3_ordered_writepage 0.0108
41 ext3_get_inode_block 0.1571
37 ext3_get_inode_loc 0.0582
82 ext3_do_update_inode 0.0888
24 ext3_writepage_trans_blocks 0.1690
39 ext3_mark_iloc_dirty 0.7091
21 ext3_reserve_inode_write 0.0995
13 ext3_mark_inode_dirty 0.1605
28 ext3_dirty_inode 0.2074
20 ext3_journal_start 0.2247
15 __ext3_journal_stop 0.1875
68 start_this_handle 0.0530
19 new_handle 0.2375
24 journal_start 0.1143
185 do_get_write_access 0.1036
26 journal_get_write_access 0.3210
12 journal_get_undo_access 0.0289
63 journal_dirty_data 0.0862
139 journal_dirty_metadata 0.2532
102 journal_stop 0.1253
89 __journal_unfile_buffer 0.1721
42 __journal_file_buffer 0.0599
24 inverted_lock 0.2500
195 journal_commit_transaction 0.0319
1 journal_brelse_array 0.0227
78 journal_cancel_revoke 0.2248
1 journal_write_revoke_records 0.0039
1 write_one_revoke_record 0.0050
4 __log_space_left 0.0388
15 journal_blocks_per_page 0.5172
5 journal_alloc_journal_head 0.0431
4 journal_free_journal_head 0.1429
260 journal_add_journal_head 0.7104
61 __journal_remove_journal_head 0.1533
52 journal_remove_journal_head 0.4194
106 journal_put_journal_head 0.4753
1 kobject_put 0.0400
10 radix_tree_preload 0.0565
3 number 0.0038
4 vsnprintf 0.0032
1 find_next_zero_bit 0.0057
27 csum_partial 0.0938
1 atomic_dec_and_lock 0.0086
17 memcpy 0.2982
1 strncpy_from_user 0.0101
6206 __copy_user_intel 36.0814
4378 __copy_user_zeroing_intel 25.4535
103 __copy_to_user_ll 0.8583
2404 __copy_from_user_ll 20.0333
32 copy_to_user 0.3265
30 copy_from_user 0.2381
1 add_disk_randomness 0.0192
3 blk_run_queue 0.0242
1 get_request 0.0013
86 __make_request 0.0619
16 generic_make_request 0.0325
15 submit_bio 0.0519
71 __end_that_request_first 0.1350
16 .text.lock.ll_rw_blk 0.1176
2 as_set_request 0.0153
332 e1000_xmit_frame 0.1560
5 scsi_put_command 0.0221
18 scsi_dispatch_cmd 0.0334
2 scsi_finish_command 0.0112
2 scsi_add_timer 0.0185
1 scsi_decide_disposition 0.0033
3 scsi_init_cmd_errh 0.0156
5 scsi_device_unbusy 0.0272
3 scsi_run_queue 0.0109
1 scsi_next_command 0.0244
19 scsi_end_request 0.0720
1 scsi_free_sgtable 0.0222
3 scsi_io_completion 0.0029
109 scsi_request_fn 0.1005
14 .text.lock.scsi_lib 0.0654
34 ahc_linux_isr 0.0489
21 sock_aio_read 0.0714
15 sock_aio_write 0.0510
13 sock_poll 0.2653
1 sk_alloc 0.0056
183 sock_wfree 2.1279
14 __release_sock 0.0909
349 sock_def_readable 1.9389
10 sk_reset_timer 0.2128
68 lock_sock 0.5484
32 release_sock 0.2105
38 sock_common_recvmsg 0.3248
201 .text.lock.sock 1.6475
261 alloc_skb 1.1652
390 skb_release_data 2.2414
12 kfree_skbmem 0.2727
637 __kfree_skb 2.7939
81 skb_clone 0.2109
42 copy_skb_header 0.1694
22 skb_copy 0.0991
268 skb_copy_bits 0.5095
5 skb_checksum 0.0081
21 memcpy_toiovec 0.2211
4 memcpy_tokerneliovec 0.0533
123 skb_copy_datagram_iovec 0.2269
35 sk_stream_rfree 1.2963
2 sk_stream_error 0.0215
12 skb_checksum_help 0.0370
53 dev_queue_xmit 0.0666
18 net_tx_action 0.0468
90 netif_receive_skb 0.2022
107 process_backlog 0.3919
48 net_rx_action 0.1853
215 nf_iterate 1.3030
100 nf_hook_slow 0.3497
163 qdisc_restart 0.2763
148 pfifo_fast_enqueue 1.0207
117 pfifo_fast_dequeue 1.4444
30 rt_hash_code 0.2190
2 rt_cache_get_first 0.0238
260 ip_route_input 0.4779
38 ip_local_deliver 0.0751
133 ip_rcv 0.1041
54 ip_local_deliver_finish 0.1385
41 ip_rcv_finish 0.0633
12 ip_finish_output 0.0226
28 ip_output 0.2333
309 ip_queue_xmit 0.2049
2 dst_output 0.0488
222 ip_finish_output2 0.5011
125 tcp_poll 0.3255
177 tcp_sendmsg 0.0389
56 cleanup_rbuf 0.1818
427 tcp_recvmsg 0.2126
24 __tcp_grow_window 0.1832
74 tcp_rcv_rtt_update 0.8132
54 tcp_rcv_space_adjust 0.1656
176 tcp_event_data_recv 0.2943
24 tcp_rtt_estimator 0.0714
3 tcp_ack_saw_tstamp 0.0357
74 tcp_clean_rtx_queue 0.0778
43 tcp_ack 0.0294
2 tcp_data_queue 0.0007
1 tcp_cwnd_application_limited 0.0067
6 __tcp_data_snd_check 0.0254
44 __tcp_ack_snd_check 0.2733
1 tcp_check_urg 0.0036
1 tcp_urg 0.0041
708 tcp_rcv_established 0.3065
309 tcp_transmit_skb 0.1774
51 tcp_write_xmit 0.0703
51 __tcp_select_window 0.1717
16 tcp_send_delayed_ack 0.0724
74 tcp_send_ack 0.3795
215 __tcp_v4_lookup_established 0.8635
15 tcp_v4_send_check 0.0593
48 tcp_v4_do_rcv 0.1509
442 tcp_v4_rcv 0.2190
9 inet_sendmsg 0.1011
19 ip_confirm 0.3958
22 ip_conntrack_defrag 0.3143
19 ip_refrag 0.1329
17 ip_conntrack_local 0.1164
20 __ip_ct_find_proto 0.3077
293 ip_ct_find_proto 2.7383
58 hash_conntrack 0.3766
44 get_tuple 0.4731
2 invert_tuple 0.0426
69 __ip_conntrack_find 0.3151
162 ip_conntrack_find_get 1.3388
3 ip_conntrack_get 0.0833
125 ip_conntrack_in 0.2033
296 ip_ct_refresh 1.5579
7 ip_ct_gather_frags 0.0302
35 tcp_pkt_to_tuple 0.4118
7 get_conntrack_index 0.1148
452 tcp_packet 0.9556
906 ipt_do_table 0.9350
17 ipt_hook 0.2881
18 ipt_local_out_hook 0.1538
33 ipt_route_hook 0.5593
23 ipt_local_hook 0.0987
64 ip_nat_fn 0.1063
30 ip_nat_out 0.1744
202 do_bindings 0.2479
3 unix_stream_connect 0.0021
5 __sched_text_start 0.0172
1 __write_lock_failed 0.0312
2 __read_lock_failed 0.1000
219 schedule 0.1052
1 wait_for_completion 0.0039
2 __preempt_spin_lock 0.0220
364 .text.lock.sched 1.8571
4 schedule_timeout 0.0209
459460 total 0.1493
21 default_idle 0.4667
418118 poll_idle 7208.9310
40 cpu_idle 0.6250
1 __up 0.0333
6 need_resched 0.1200
116 system_call 2.6364
4 syscall_call 0.3636
7 syscall_exit 0.6364
1 error_code 0.0179
4 handle_IRQ_event 0.0400
2 old_mmap 0.0060
1 show_cpuinfo 0.0015
34 sched_clock 0.2500
1 delay_tsc 0.0476
1 flush_tlb_page 0.0056
4 pte_alloc_one 0.0519
25 do_page_fault 0.0179
1694 finish_task_switch 11.3691
1 nr_running 0.0091
1 nr_iowait 0.0087
1059 __wake_up 8.9746
234 __might_sleep 1.1090
84 add_wait_queue 0.6774
54 remove_wait_queue 0.4219
35 prepare_to_wait 0.2381
50 finish_wait 0.3165
2 dup_task_struct 0.0095
3 copy_mm 0.0026
1 count_open_files 0.0233
6 .text.lock.fork 0.0303
1 release_task 0.0018
2 sys_wait4 0.0031
113 current_kernel_time 1.6377
5065 __do_softirq 29.2775
188 local_bh_enable 1.3056
875 __mod_timer 1.6925
53 mod_timer 0.5000
331 del_timer 2.1494
1 del_timer_sync 0.0055
1 .text.lock.timer 0.0109
1 do_sigaction 0.0016
1 groups_search 0.0094
58 add_to_page_cache 0.2367
13 page_waitqueue 0.2708
50 wake_up_page 0.6494
15 unlock_page 0.4054
14 end_page_writeback 0.1892
47 find_get_page 0.4196
31 find_lock_page 0.1069
58 find_get_pages_tag 0.3648
2 do_generic_mapping_read 0.0016
8 filemap_nopage 0.0085
9 remove_suid 0.0604
112 generic_file_aio_write_nolock 0.0387
13 generic_file_aio_write 0.0823
32 mempool_alloc 0.0840
20 mempool_free 0.0952
2 mempool_alloc_slab 0.0952
1 mempool_free_slab 0.0476
12 bad_range 0.1200
16 prep_new_page 0.1975
34 free_hot_cold_page 0.1141
1 free_hot_page 0.1429
170 buffered_rmqueue 0.3195
79 __alloc_pages 0.0890
2 __get_free_pages 0.0339
1 get_zeroed_page 0.0098
12 __free_pages 0.1538
2 free_pages 0.0444
1 get_dirty_limits 0.0049
11 balance_dirty_pages_ratelimited 0.0547
80 __set_page_dirty_nobuffers 0.2128
3 set_page_dirty 0.0357
8 clear_page_dirty_for_io 0.0702
52 test_clear_page_writeback 0.2039
53 test_set_page_writeback 0.1767
14 .text.lock.page_writeback 0.1094
1 vma_prio_tree_add 0.0060
1 do_page_cache_readahead 0.0022
1 page_cache_readahead 0.0018
3 kmem_getpages 0.0144
1 alloc_slabmgmt 0.0104
7 cache_init_objs 0.0673
1 kmem_flagcheck 0.0233
1 cache_grow 0.0022
159 kmem_cache_alloc 1.2927
1 kmem_ptr_validate 0.0132
83 __kmalloc 0.5390
189 kmem_cache_free 2.1724
184 kfree 1.6881
41 mark_page_accessed 0.7885
1 lru_cache_add_active 0.0105
37 release_pages 0.0819
40 __pagevec_lru_add 0.1262
2 vm_acct_memory 0.0244
19 percpu_counter_mod 0.1086
4 blk_queue_bounce 0.0519
2 clear_page_tables 0.0065
1 pte_alloc_map 0.0040
11 copy_page_range 0.0142
25 zap_pte_range 0.0360
16 do_wp_page 0.0189
14 do_anonymous_page 0.0312
320 do_no_page 0.3591
2 remove_vm_struct 0.0123
3 find_vma_prepare 0.0297
1 vma_adjust 0.0010
1 can_vma_merge_after 0.0090
3 do_mmap_pgoff 0.0016
2 find_vma 0.0247
1 detach_vmas_to_be_unmapped 0.0101
1 split_vma 0.0035
1 change_protection 0.0022
3 anon_vma_prepare 0.0085
4 page_add_anon_rmap 0.0122
2 page_add_file_rmap 0.0093
15 page_remove_rmap 0.0628
1 can_share_swap_page 0.0094
1 sys_access 0.0030
1 filp_open 0.0095
1 dentry_open 0.0025
1 fd_install 0.0068
5 do_sync_read 0.0289
49 vfs_read 0.1661
24 do_sync_write 0.1387
60 vfs_write 0.2034
36 sys_read 0.2812
36 sys_write 0.2812
43 fput 2.0476
96 fget 0.7500
36 fget_light 0.2156
1 file_kill 0.0081
4 bh_waitq_head 0.1818
67 wake_up_buffer 0.8072
39 unlock_buffer 1.6250
11 __set_page_buffers 0.4400
1 __clear_page_buffers 0.0141
76 end_buffer_write_sync 0.6281
10 __find_get_block_slow 0.0231
25 end_buffer_async_write 0.0809
30 create_buffers 0.1840
2 __getblk_slow 0.0056
6 mark_buffer_dirty 0.1500
41 __brelse 0.5256
143 __find_get_block 0.5813
21 __getblk 0.2079
10 __bread 0.1515
6 set_bh_page 0.1111
32 create_empty_buffers 0.1584
3 unmap_underlying_metadata 0.0265
56 __block_write_full_page 0.0583
71 __block_prepare_write 0.0715
17 __block_commit_write 0.1141
12 block_prepare_write 0.1558
8 generic_commit_write 0.0541
26 block_write_full_page 0.0974
31 end_bio_bh_io_sync 0.5000
43 submit_bh 0.1319
27 ll_rw_block 0.1985
8 recalc_bh_state 0.0447
8 alloc_buffer_head 0.0762
21 init_buffer_head 0.3962
7 bio_destructor 0.0745
29 bio_alloc 0.0681
52 bio_put 0.9630
8 bio_endio 0.0630
1 generic_fillattr 0.0061
1 cp_new_stat64 0.0037
19 inode_add_bytes 0.1338
1 setup_arg_pages 0.0021
1 kernel_read 0.0105
1 prepare_binprm 0.0040
1 do_execve 0.0015
49 pipe_wait 0.2737
255 pipe_readv 0.3607
4 pipe_read 0.0678
145 pipe_writev 0.1749
5 pipe_write 0.0847
120 pipe_poll 0.9677
2 do_lookup 0.0124
2 link_path_walk 0.0005
1 may_open 0.0019
1 open_namei 0.0006
15 kill_fasync 0.1293
2 poll_initwait 0.0800
39 poll_freewait 0.6094
21 __pollwait 0.1066
17 max_select_fd 0.0756
162 do_select 0.2072
1 select_bits_free 0.2000
94 sys_select 0.0729
1 flock64_to_posix_lock 0.0027
1 dput 0.0016
6 __d_lookup 0.0144
1 generic_forget_inode 0.0027
128 inode_times_differ 1.8824
31 update_atime 0.1396
39 inode_update_time 0.1822
1 dnotify_flush 0.0047
203 dnotify_parent 0.8024
1 lookup_mnt 0.0060
3 seq_printf 0.0349
84 __mark_inode_dirty 0.1736
1 __sync_single_inode 0.0018
1 writeback_release 0.0208
13 mpage_writepages 0.0155
1 proc_alloc_inode 0.0085
1 proc_get_inode 0.0036
10 show_stat 0.0044
8 write_profile 0.0513
19 ext3_get_group_desc 0.1011
9 read_block_bitmap 0.0592
3 find_next_usable_block 0.0035
66 ext3_try_to_allocate 0.0742
5 ext3_has_free_blocks 0.0407
72 ext3_new_block 0.0554
25 ext3_file_write 0.1174
8 ext3_alloc_block 0.1356
21 ext3_block_to_path 0.0660
25 ext3_get_branch 0.1059
17 ext3_find_goal 0.1241
15 ext3_alloc_branch 0.0217
45 ext3_splice_branch 0.0949
60 ext3_get_block_handle 0.0675
9 ext3_get_block 0.0517
73 walk_page_buffers 0.6518
22 ext3_prepare_write 0.0692
9 ext3_journal_dirty_data 0.0968
87 ext3_ordered_commit_write 0.3799
16 bget_one 1.4545
1 bput_one 0.0909
3 journal_dirty_data_fn 0.1579
13 ext3_ordered_writepage 0.0282
73 ext3_get_inode_block 0.2797
31 ext3_get_inode_loc 0.0487
75 ext3_do_update_inode 0.0813
31 ext3_writepage_trans_blocks 0.2183
33 ext3_mark_iloc_dirty 0.6000
26 ext3_reserve_inode_write 0.1232
12 ext3_mark_inode_dirty 0.1481
14 ext3_dirty_inode 0.1037
13 ext3_journal_start 0.1461
10 __ext3_journal_stop 0.1250
65 start_this_handle 0.0507
19 new_handle 0.2375
26 journal_start 0.1238
172 do_get_write_access 0.0963
28 journal_get_write_access 0.3457
1 journal_get_create_access 0.0015
30 journal_get_undo_access 0.0723
67 journal_dirty_data 0.0917
133 journal_dirty_metadata 0.2423
86 journal_stop 0.1057
115 __journal_unfile_buffer 0.2224
49 __journal_file_buffer 0.0699
1 journal_file_buffer 0.0046
18 inverted_lock 0.1875
231 journal_commit_transaction 0.0378
2 journal_brelse_array 0.0455
2 __journal_clean_checkpoint_list 0.0115
63 journal_cancel_revoke 0.1816
1 journal_write_revoke_records 0.0039
7 __log_space_left 0.0680
1 journal_get_descriptor_buffer 0.0053
10 journal_blocks_per_page 0.3448
1 __jbd_kmalloc 0.0227
8 journal_alloc_journal_head 0.0690
2 journal_free_journal_head 0.0714
257 journal_add_journal_head 0.7022
56 __journal_remove_journal_head 0.1407
55 journal_remove_journal_head 0.4435
103 journal_put_journal_head 0.4619
1 lookup_undo 0.0154
11 radix_tree_preload 0.0621
1 __rb_rotate_left 0.0179
1 __rb_erase_color 0.0024
5 number 0.0064
10 vsnprintf 0.0081
1 find_next_bit 0.0066
2 find_next_zero_bit 0.0114
27 csum_partial 0.0938
2 atomic_dec_and_lock 0.0172
13 memcpy 0.2281
1 strncpy_from_user 0.0101
6152 __copy_user_intel 35.7674
4411 __copy_user_zeroing_intel 25.6453
114 __copy_to_user_ll 0.9500
2323 __copy_from_user_ll 19.3583
30 copy_to_user 0.3061
36 copy_from_user 0.2857
3 add_disk_randomness 0.0577
1 generic_unplug_device 0.0100
2 blk_run_queue 0.0161
3 get_request 0.0040
110 __make_request 0.0791
14 generic_make_request 0.0285
8 submit_bio 0.0277
49 __end_that_request_first 0.0932
6 put_io_context 0.0682
22 .text.lock.ll_rw_blk 0.1618
2 as_set_request 0.0153
310 e1000_xmit_frame 0.1457
4 scsi_put_command 0.0177
28 scsi_dispatch_cmd 0.0519
4 scsi_softirq 0.0191
4 scsi_finish_command 0.0223
2 scsi_add_timer 0.0185
2 scsi_decide_disposition 0.0067
1 scsi_init_cmd_errh 0.0052
7 scsi_device_unbusy 0.0380
6 scsi_run_queue 0.0218
27 scsi_end_request 0.1023
2 scsi_free_sgtable 0.0444
5 scsi_io_completion 0.0049
163 scsi_request_fn 0.1502
14 .text.lock.scsi_lib 0.0654
49 ahc_linux_isr 0.0705
1 sd_rw_intr 0.0015
1 sock_release 0.0070
16 sock_aio_read 0.0544
14 sock_aio_write 0.0476
24 sock_poll 0.4898
167 sock_wfree 1.9419
15 __release_sock 0.0974
390 sock_def_readable 2.1667
10 sk_reset_timer 0.2128
77 lock_sock 0.6210
29 release_sock 0.1908
48 sock_common_recvmsg 0.4103
193 .text.lock.sock 1.5820
240 alloc_skb 1.0714
399 skb_release_data 2.2931
13 kfree_skbmem 0.2955
639 __kfree_skb 2.8026
85 skb_clone 0.2214
34 copy_skb_header 0.1371
17 skb_copy 0.0766
260 skb_copy_bits 0.4943
5 skb_checksum 0.0081
18 memcpy_toiovec 0.1895
112 skb_copy_datagram_iovec 0.2066
42 sk_stream_rfree 1.5556
7 sk_stream_error 0.0753
10 skb_checksum_help 0.0309
74 dev_queue_xmit 0.0930
21 net_tx_action 0.0545
60 netif_receive_skb 0.1348
115 process_backlog 0.4212
66 net_rx_action 0.2548
187 nf_iterate 1.1333
85 nf_hook_slow 0.2972
165 qdisc_restart 0.2797
133 pfifo_fast_enqueue 0.9172
103 pfifo_fast_dequeue 1.2716
24 rt_hash_code 0.1752
1 rt_cache_get_first 0.0119
266 ip_route_input 0.4890
33 ip_local_deliver 0.0652
141 ip_rcv 0.1103
73 ip_local_deliver_finish 0.1872
62 ip_rcv_finish 0.0957
15 ip_finish_output 0.0282
16 ip_output 0.1333
257 ip_queue_xmit 0.1704
8 dst_output 0.1951
226 ip_finish_output2 0.5102
137 tcp_poll 0.3568
173 tcp_sendmsg 0.0381
59 cleanup_rbuf 0.1916
426 tcp_recvmsg 0.2122
25 __tcp_grow_window 0.1908
82 tcp_rcv_rtt_update 0.9011
39 tcp_rcv_space_adjust 0.1196
170 tcp_event_data_recv 0.2843
39 tcp_rtt_estimator 0.1161
5 tcp_ack_saw_tstamp 0.0595
57 tcp_clean_rtx_queue 0.0599
1 tcp_ack_update_window 0.0039
38 tcp_ack 0.0260
1 tcp_data_queue 0.0003
10 __tcp_data_snd_check 0.0424
59 __tcp_ack_snd_check 0.3665
1 tcp_check_urg 0.0036
702 tcp_rcv_established 0.3039
338 tcp_transmit_skb 0.1940
44 tcp_write_xmit 0.0607
73 __tcp_select_window 0.2458
18 tcp_send_delayed_ack 0.0814
62 tcp_send_ack 0.3179
201 __tcp_v4_lookup_established 0.8072
23 tcp_v4_send_check 0.0909
41 tcp_v4_do_rcv 0.1289
425 tcp_v4_rcv 0.2106
11 inet_sendmsg 0.1236
1 inet_shutdown 0.0042
17 ip_confirm 0.3542
25 ip_conntrack_defrag 0.3571
25 ip_refrag 0.1748
25 ip_conntrack_local 0.1712
16 __ip_ct_find_proto 0.2462
273 ip_ct_find_proto 2.5514
65 hash_conntrack 0.4221
48 get_tuple 0.5161
3 invert_tuple 0.0638
79 __ip_conntrack_find 0.3607
135 ip_conntrack_find_get 1.1157
15 ip_conntrack_get 0.4167
137 ip_conntrack_in 0.2228
282 ip_ct_refresh 1.4842
3 ip_ct_gather_frags 0.0129
17 tcp_pkt_to_tuple 0.2000
12 get_conntrack_index 0.1967
436 tcp_packet 0.9218
882 ipt_do_table 0.9102
18 ipt_hook 0.3051
13 ipt_local_out_hook 0.1111
26 ipt_route_hook 0.4407
22 ipt_local_hook 0.0944
65 ip_nat_fn 0.1080
25 ip_nat_out 0.1453
201 do_bindings 0.2466
1 __sched_text_start 0.0034
2 __down_failed 0.1667
3 __write_lock_failed 0.0938
1 __read_lock_failed 0.0500
206 schedule 0.0990
7 __preempt_spin_lock 0.0769
384 .text.lock.sched 1.9592
7 schedule_timeout 0.0366
461591 total 0.1500
22 default_idle 0.4889
415993 poll_idle 7172.2931
60 cpu_idle 0.9375
2 __up 0.0667
7 need_resched 0.1400
126 system_call 2.8636
3 syscall_call 0.2727
7 syscall_exit 0.6364
1 error_code 0.0179
5 handle_IRQ_event 0.0500
80 sched_clock 0.5882
1 delay_tsc 0.0476
1 flush_tlb_others 0.0033
1 flush_tlb_page 0.0056
4 pte_alloc_one 0.0519
1 pgd_ctor 0.0060
12 do_page_fault 0.0086
1708 finish_task_switch 11.4631
1033 __wake_up 8.7542
182 __might_sleep 0.8626
1 task_curr 0.0044
75 add_wait_queue 0.6048
45 remove_wait_queue 0.3516
36 prepare_to_wait 0.2449
42 finish_wait 0.2658
1 dup_task_struct 0.0047
5 copy_mm 0.0043
1 copy_files 0.0010
1 copy_process 0.0003
7 .text.lock.fork 0.0354
1 profile_exec_unmap 0.0104
2 exit_notify 0.0009
1 do_setitimer 0.0021
428 current_kernel_time 6.2029
5423 __do_softirq 31.3468
177 local_bh_enable 1.2292
878 __mod_timer 1.6983
48 mod_timer 0.4528
343 del_timer 2.2273
1 run_timer_softirq 0.0022
2 .text.lock.timer 0.0217
1 flush_signal_handlers 0.0149
1 do_sigaction 0.0016
54 add_to_page_cache 0.2204
20 page_waitqueue 0.4167
48 wake_up_page 0.6234
13 unlock_page 0.3514
21 end_page_writeback 0.2838
55 find_get_page 0.4911
32 find_lock_page 0.1103
59 find_get_pages_tag 0.3711
4 filemap_nopage 0.0042
11 remove_suid 0.0738
117 generic_file_aio_write_nolock 0.0404
21 generic_file_aio_write 0.1329
45 mempool_alloc 0.1181
18 mempool_free 0.0857
2 mempool_alloc_slab 0.0952
3 mempool_free_slab 0.1429
25 bad_range 0.2500
10 prep_new_page 0.1235
43 free_hot_cold_page 0.1443
155 buffered_rmqueue 0.2914
84 __alloc_pages 0.0946
7 __get_free_pages 0.1186
8 __free_pages 0.1026
2 free_pages 0.0444
1 get_dirty_limits 0.0049
10 balance_dirty_pages_ratelimited 0.0498
58 __set_page_dirty_nobuffers 0.1543
14 clear_page_dirty_for_io 0.1228
57 test_clear_page_writeback 0.2235
50 test_set_page_writeback 0.1667
12 .text.lock.page_writeback 0.0938
1 prio_tree_expand 0.0048
1 prio_tree_replace 0.0101
1 vma_prio_tree_remove 0.0039
1 do_page_cache_readahead 0.0022
1 alloc_slabmgmt 0.0104
6 cache_init_objs 0.0577
1 set_slab_attr 0.0172
2 cache_grow 0.0044
151 kmem_cache_alloc 1.2276
71 __kmalloc 0.4610
197 kmem_cache_free 2.2644
211 kfree 1.9358
1 activate_page 0.0044
37 mark_page_accessed 0.7115
1 lru_cache_add_active 0.0105
29 release_pages 0.0642
26 __pagevec_lru_add 0.0820
3 __pagevec_lru_add_active 0.0089
1 vm_acct_memory 0.0122
14 percpu_counter_mod 0.0800
3 clear_page_tables 0.0097
3 pte_alloc_map 0.0121
15 copy_page_range 0.0194
22 zap_pte_range 0.0317
2 unmap_vmas 0.0035
20 do_wp_page 0.0236
31 do_anonymous_page 0.0690
304 do_no_page 0.3412
10 handle_mm_fault 0.0228
2 remove_vm_struct 0.0123
1 can_vma_merge_after 0.0090
1 vma_merge 0.0020
7 find_vma 0.0864
2 find_vma_prev 0.0247
1 split_vma 0.0035
1 anon_vma_unlink 0.0064
4 page_add_file_rmap 0.0185
9 page_remove_rmap 0.0377
2 fd_install 0.0136
1 sys_open 0.0060
1 sys_close 0.0049
15 do_sync_read 0.0867
57 vfs_read 0.1932
25 do_sync_write 0.1445
59 vfs_write 0.2000
19 sys_read 0.1484
29 sys_write 0.2266
3 get_empty_filp 0.0128
48 fput 2.2857
1 __fput 0.0047
83 fget 0.6484
56 fget_light 0.3353
6 bh_waitq_head 0.2727
82 wake_up_buffer 0.9880
1 sync_buffer 0.0149
37 unlock_buffer 1.5417
15 __set_page_buffers 0.6000
45 end_buffer_write_sync 0.3719
19 __find_get_block_slow 0.0439
49 end_buffer_async_write 0.1586
7 buffer_insert_list 0.0470
40 create_buffers 0.2454
5 mark_buffer_dirty 0.1250
52 __brelse 0.6667
1 bh_lru_install 0.0044
152 __find_get_block 0.6179
18 __getblk 0.1782
23 __bread 0.3485
7 set_bh_page 0.1296
1 try_to_release_page 0.0091
26 create_empty_buffers 0.1287
4 unmap_underlying_metadata 0.0354
46 __block_write_full_page 0.0479
66 __block_prepare_write 0.0665
20 __block_commit_write 0.1342
13 block_prepare_write 0.1688
9 generic_commit_write 0.0608
25 block_write_full_page 0.0936
21 end_bio_bh_io_sync 0.3387
38 submit_bh 0.1166
24 ll_rw_block 0.1765
9 recalc_bh_state 0.0503
14 alloc_buffer_head 0.1333
18 init_buffer_head 0.3396
9 .text.lock.buffer 0.0484
4 bio_destructor 0.0426
29 bio_alloc 0.0681
64 bio_put 1.1852
1 bio_endio 0.0079
1 cp_new_stat64 0.0037
12 inode_add_bytes 0.0845
1 prepare_binprm 0.0040
60 pipe_wait 0.3352
241 pipe_readv 0.3409
9 pipe_read 0.1525
151 pipe_writev 0.1821
13 pipe_write 0.2203
118 pipe_poll 0.9516
1 pipe_release 0.0043
2 vfs_permission 0.0064
4 link_path_walk 0.0010
2 path_lookup 0.0042
1 open_namei 0.0006
1 vfs_mkdir 0.0052
13 kill_fasync 0.1121
5 poll_initwait 0.2000
27 poll_freewait 0.4219
38 __pollwait 0.1929
21 max_select_fd 0.0933
164 do_select 0.2097
5 select_bits_alloc 0.1562
3 select_bits_free 0.6000
99 sys_select 0.0767
1 fcntl_setlk 0.0015
1 fcntl_setlk64 0.0015
5 dput 0.0081
4 __d_lookup 0.0096
125 inode_times_differ 1.8382
30 update_atime 0.1351
40 inode_update_time 0.1869
183 dnotify_parent 0.7233
1 seq_read 0.0014
148 __mark_inode_dirty 0.3058
1 __writeback_single_inode 0.0045
3 sync_sb_inodes 0.0039
18 mpage_writepages 0.0215
1 eventpoll_init_file 0.0303
1 get_vmalloc_info 0.0052
5 show_stat 0.0022
11 write_profile 0.0705
15 ext3_get_group_desc 0.0798
8 read_block_bitmap 0.0526
6 find_next_usable_block 0.0071
64 ext3_try_to_allocate 0.0719
5 ext3_has_free_blocks 0.0407
66 ext3_new_block 0.0508
21 ext3_file_write 0.0986
11 ext3_alloc_block 0.1864
24 ext3_block_to_path 0.0755
39 ext3_get_branch 0.1653
23 ext3_find_goal 0.1679
19 ext3_alloc_branch 0.0275
44 ext3_splice_branch 0.0928
61 ext3_get_block_handle 0.0686
16 ext3_get_block 0.0920
73 walk_page_buffers 0.6518
22 ext3_prepare_write 0.0692
5 ext3_journal_dirty_data 0.0538
76 ext3_ordered_commit_write 0.3319
19 bget_one 1.7273
2 bput_one 0.1818
4 journal_dirty_data_fn 0.2105
10 ext3_ordered_writepage 0.0217
55 ext3_get_inode_block 0.2107
27 ext3_get_inode_loc 0.0425
83 ext3_do_update_inode 0.0899
36 ext3_writepage_trans_blocks 0.2535
36 ext3_mark_iloc_dirty 0.6545
23 ext3_reserve_inode_write 0.1090
16 ext3_mark_inode_dirty 0.1975
29 ext3_dirty_inode 0.2148
18 ext3_journal_start 0.2022
13 __ext3_journal_stop 0.1625
60 start_this_handle 0.0468
17 new_handle 0.2125
38 journal_start 0.1810
218 do_get_write_access 0.1221
21 journal_get_write_access 0.2593
21 journal_get_undo_access 0.0506
72 journal_dirty_data 0.0985
139 journal_dirty_metadata 0.2532
100 journal_stop 0.1229
102 __journal_unfile_buffer 0.1973
2 journal_unfile_buffer 0.0102
38 __journal_file_buffer 0.0542
16 inverted_lock 0.1667
198 journal_commit_transaction 0.0324
1 journal_brelse_array 0.0227
1 find_revoke_record 0.0045
64 journal_cancel_revoke 0.1844
12 __log_space_left 0.1165
1 __log_start_commit 0.0159
8 journal_blocks_per_page 0.2759
12 journal_alloc_journal_head 0.1034
1 journal_free_journal_head 0.0357
262 journal_add_journal_head 0.7158
41 __journal_remove_journal_head 0.1030
43 journal_remove_journal_head 0.3468
116 journal_put_journal_head 0.5202
1 copy_semundo 0.0042
1 sys_shmdt 0.0028
2 cap_vm_enough_memory 0.0077
1 __bitmap_weight 0.0035
12 radix_tree_preload 0.0678
1 rb_erase 0.0044
11 number 0.0140
5 vsnprintf 0.0041
34 csum_partial 0.1181
1 atomic_dec_and_lock 0.0086
10 memcpy 0.1754
1 strncpy_from_user 0.0101
3 strnlen_user 0.0312
6005 __copy_user_intel 34.9128
4369 __copy_user_zeroing_intel 25.4012
107 __copy_to_user_ll 0.8917
2241 __copy_from_user_ll 18.6750
28 copy_to_user 0.2857
13 copy_from_user 0.1032
3 add_disk_randomness 0.0577
2 blk_run_queue 0.0161
1 get_request 0.0013
120 __make_request 0.0863
13 generic_make_request 0.0264
14 submit_bio 0.0484
65 __end_that_request_first 0.1236
4 put_io_context 0.0455
2 get_io_context 0.0160
11 .text.lock.ll_rw_blk 0.0809
1 as_set_request 0.0076
332 e1000_xmit_frame 0.1560
2 scsi_put_command 0.0088
25 scsi_dispatch_cmd 0.0464
3 scsi_softirq 0.0144
1 scsi_finish_command 0.0056
2 scsi_decide_disposition 0.0067
3 scsi_init_cmd_errh 0.0156
3 scsi_device_unbusy 0.0163
4 scsi_run_queue 0.0145
2 scsi_next_command 0.0488
29 scsi_end_request 0.1098
2 scsi_free_sgtable 0.0444
7 scsi_io_completion 0.0069
112 scsi_request_fn 0.1032
12 .text.lock.scsi_lib 0.0561
45 ahc_linux_isr 0.0647
1 sd_rw_intr 0.0015
32 sock_aio_read 0.1088
7 sock_aio_write 0.0238
18 sock_poll 0.3673
1 __sock_create 0.0027
216 sock_wfree 2.5116
17 __release_sock 0.1104
383 sock_def_readable 2.1278
11 sk_reset_timer 0.2340
83 lock_sock 0.6694
35 release_sock 0.2303
37 sock_common_recvmsg 0.3162
206 .text.lock.sock 1.6885
237 alloc_skb 1.0580
416 skb_release_data 2.3908
17 kfree_skbmem 0.3864
669 __kfree_skb 2.9342
77 skb_clone 0.2005
38 copy_skb_header 0.1532
24 skb_copy 0.1081
290 skb_copy_bits 0.5513
5 skb_checksum 0.0081
20 memcpy_toiovec 0.2105
1 memcpy_tokerneliovec 0.0133
129 skb_copy_datagram_iovec 0.2380
23 sk_stream_rfree 0.8519
3 sk_stream_error 0.0323
1 sk_stream_mem_schedule 0.0029
16 skb_checksum_help 0.0494
71 dev_queue_xmit 0.0892
17 net_tx_action 0.0442
69 netif_receive_skb 0.1551
106 process_backlog 0.3883
60 net_rx_action 0.2317
208 nf_iterate 1.2606
70 nf_hook_slow 0.2448
162 qdisc_restart 0.2746
132 pfifo_fast_enqueue 0.9103
88 pfifo_fast_dequeue 1.0864
33 rt_hash_code 0.2409
2 rt_cache_get_first 0.0238
271 ip_route_input 0.4982
47 ip_local_deliver 0.0929
140 ip_rcv 0.1095
62 ip_local_deliver_finish 0.1590
47 ip_rcv_finish 0.0725
11 ip_finish_output 0.0207
18 ip_output 0.1500
304 ip_queue_xmit 0.2016
8 dst_output 0.1951
235 ip_finish_output2 0.5305
104 tcp_poll 0.2708
145 tcp_sendmsg 0.0319
55 cleanup_rbuf 0.1786
398 tcp_recvmsg 0.1982
27 __tcp_grow_window 0.2061
95 tcp_rcv_rtt_update 1.0440
73 tcp_rcv_space_adjust 0.2239
208 tcp_event_data_recv 0.3478
29 tcp_rtt_estimator 0.0863
4 tcp_ack_saw_tstamp 0.0476
66 tcp_clean_rtx_queue 0.0694
1 tcp_ack_update_window 0.0039
42 tcp_ack 0.0287
1 tcp_data_queue 0.0003
6 __tcp_data_snd_check 0.0254
51 __tcp_ack_snd_check 0.3168
2 tcp_check_urg 0.0071
702 tcp_rcv_established 0.3039
331 tcp_transmit_skb 0.1900
40 tcp_write_xmit 0.0552
25 __tcp_select_window 0.0842
16 tcp_send_delayed_ack 0.0724
76 tcp_send_ack 0.3897
191 __tcp_v4_lookup_established 0.7671
41 tcp_v4_send_check 0.1621
44 tcp_v4_do_rcv 0.1384
475 tcp_v4_rcv 0.2354
10 inet_sendmsg 0.1124
18 ip_confirm 0.3750
24 ip_conntrack_defrag 0.3429
27 ip_refrag 0.1888
15 ip_conntrack_local 0.1027
14 __ip_ct_find_proto 0.2154
300 ip_ct_find_proto 2.8037
67 hash_conntrack 0.4351
33 get_tuple 0.3548
3 invert_tuple 0.0638
59 __ip_conntrack_find 0.2694
150 ip_conntrack_find_get 1.2397
4 ip_conntrack_get 0.1111
138 ip_conntrack_in 0.2244
281 ip_ct_refresh 1.4789
1 ip_ct_gather_frags 0.0043
32 tcp_pkt_to_tuple 0.3765
3 get_conntrack_index 0.0492
480 tcp_packet 1.0148
882 ipt_do_table 0.9102
14 ipt_hook 0.2373
17 ipt_local_out_hook 0.1453
30 ipt_route_hook 0.5085
30 ipt_local_hook 0.1288
50 ip_nat_fn 0.0831
17 ip_nat_out 0.0988
225 do_bindings 0.2761
14 __sched_text_start 0.0483
2 __up_wakeup 0.1667
2 __write_lock_failed 0.0625
2 __read_lock_failed 0.1000
204 schedule 0.0980
1 wait_for_completion 0.0039
2 io_schedule_timeout 0.0392
1 __preempt_spin_lock 0.0110
358 .text.lock.sched 1.8265
5 schedule_timeout 0.0262
460167 total 0.1495
13 default_idle 0.2889
418136 poll_idle 7209.2414
41 cpu_idle 0.6406
1 exit_thread 0.0076
5 need_resched 0.1000
142 system_call 3.2273
5 syscall_call 0.4545
7 syscall_exit 0.6364
6 handle_IRQ_event 0.0600
2 show_cpuinfo 0.0030
33 sched_clock 0.2426
7 pte_alloc_one 0.0909
1 pgd_ctor 0.0060
14 do_page_fault 0.0100
1636 finish_task_switch 10.9799
1018 __wake_up 8.6271
228 __might_sleep 1.0806
102 add_wait_queue 0.8226
40 remove_wait_queue 0.3125
31 prepare_to_wait 0.2109
1 prepare_to_wait_exclusive 0.0068
54 finish_wait 0.3418
1 dup_task_struct 0.0047
2 copy_mm 0.0017
1 copy_process 0.0003
7 .text.lock.fork 0.0354
1 release_task 0.0018
122 current_kernel_time 1.7681
5163 __do_softirq 29.8439
184 local_bh_enable 1.2778
610 __mod_timer 1.1799
56 mod_timer 0.5283
298 del_timer 1.9351
1 del_timer_sync 0.0055
2 .text.lock.timer 0.0217
1 flush_signal_handlers 0.0149
3 in_group_p 0.0250
1 call_rcu 0.0127
62 add_to_page_cache 0.2531
6 page_waitqueue 0.1250
45 wake_up_page 0.5844
25 unlock_page 0.6757
26 end_page_writeback 0.3514
54 find_get_page 0.4821
42 find_lock_page 0.1448
51 find_get_pages_tag 0.3208
4 do_generic_mapping_read 0.0031
3 filemap_nopage 0.0032
5 remove_suid 0.0336
127 generic_file_aio_write_nolock 0.0438
13 generic_file_aio_write 0.0823
41 mempool_alloc 0.1076
20 mempool_free 0.0952
1 mempool_alloc_slab 0.0476
4 mempool_free_slab 0.1905
16 bad_range 0.1600
21 prep_new_page 0.2593
42 free_hot_cold_page 0.1409
1 free_hot_page 0.1429
179 buffered_rmqueue 0.3365
66 __alloc_pages 0.0743
5 __get_free_pages 0.0847
8 __free_pages 0.1026
2 free_pages 0.0444
2 __read_page_state 0.0357
6 balance_dirty_pages_ratelimited 0.0299
94 __set_page_dirty_nobuffers 0.2500
1 set_page_dirty 0.0119
14 clear_page_dirty_for_io 0.1228
51 test_clear_page_writeback 0.2000
60 test_set_page_writeback 0.2000
16 .text.lock.page_writeback 0.1250
1 pdflush_operation 0.0042
1 vma_prio_tree_add 0.0060
1 vma_prio_tree_remove 0.0039
1 page_cache_readahead 0.0018
2 kmem_getpages 0.0096
1 alloc_slabmgmt 0.0104
2 cache_init_objs 0.0192
2 kmem_flagcheck 0.0465
3 cache_grow 0.0066
166 kmem_cache_alloc 1.3496
4 kmem_ptr_validate 0.0526
78 __kmalloc 0.5065
219 kmem_cache_free 2.5172
228 kfree 2.0917
39 mark_page_accessed 0.7500
3 lru_cache_add_active 0.0316
41 release_pages 0.0907
23 __pagevec_lru_add 0.0726
3 __pagevec_lru_add_active 0.0089
1 vm_acct_memory 0.0122
13 percpu_counter_mod 0.0743
2 blk_queue_bounce 0.0260
4 clear_page_tables 0.0130
5 pte_alloc_map 0.0202
7 copy_page_range 0.0090
18 zap_pte_range 0.0259
1 zap_pmd_range 0.0087
1 unmap_page_range 0.0088
1 unmap_vmas 0.0017
15 do_wp_page 0.0177
33 do_anonymous_page 0.0735
320 do_no_page 0.3591
2 handle_mm_fault 0.0046
1 find_vma_prepare 0.0099
1 vma_link 0.0030
4 do_mmap_pgoff 0.0022
11 find_vma 0.1358
1 find_vma_prev 0.0123
1 expand_stack 0.0034
1 detach_vmas_to_be_unmapped 0.0101
1 change_protection 0.0022
1 anon_vma_unlink 0.0064
4 page_add_anon_rmap 0.0122
2 page_add_file_rmap 0.0093
15 page_remove_rmap 0.0628
3 free_pages_and_swap_cache 0.0229
1 dentry_open 0.0025
1 get_unused_fd 0.0039
10 do_sync_read 0.0578
50 vfs_read 0.1695
31 do_sync_write 0.1792
37 vfs_write 0.1254
31 sys_read 0.2422
33 sys_write 0.2578
59 fput 2.8095
98 fget 0.7656
50 fget_light 0.2994
5 bh_waitq_head 0.2273
78 wake_up_buffer 0.9398
40 unlock_buffer 1.6667
13 __set_page_buffers 0.5200
71 end_buffer_write_sync 0.5868
16 __find_get_block_slow 0.0370
32 end_buffer_async_write 0.1036
1 buffer_insert_list 0.0067
28 create_buffers 0.1718
6 mark_buffer_dirty 0.1500
44 __brelse 0.5641
1 bh_lru_install 0.0044
184 __find_get_block 0.7480
22 __getblk 0.2178
11 __bread 0.1667
5 set_bh_page 0.0926
25 create_empty_buffers 0.1238
6 unmap_underlying_metadata 0.0531
48 __block_write_full_page 0.0500
64 __block_prepare_write 0.0645
25 __block_commit_write 0.1678
14 block_prepare_write 0.1818
6 generic_commit_write 0.0405
15 block_write_full_page 0.0562
23 end_bio_bh_io_sync 0.3710
38 submit_bh 0.1166
31 ll_rw_block 0.2279
5 recalc_bh_state 0.0279
5 alloc_buffer_head 0.0476
24 init_buffer_head 0.4528
8 .text.lock.buffer 0.0430
5 bio_destructor 0.0532
25 bio_alloc 0.0587
53 bio_put 0.9815
4 bio_endio 0.0315
1 cp_new_stat64 0.0037
12 inode_add_bytes 0.0845
1 copy_strings 0.0018
50 pipe_wait 0.2793
257 pipe_readv 0.3635
13 pipe_read 0.2203
132 pipe_writev 0.1592
5 pipe_write 0.0847
119 pipe_poll 0.9597
6 link_path_walk 0.0015
1 path_lookup 0.0021
14 kill_fasync 0.1207
3 poll_initwait 0.1200
31 poll_freewait 0.4844
31 __pollwait 0.1574
16 max_select_fd 0.0711
170 do_select 0.2174
5 select_bits_alloc 0.1562
91 sys_select 0.0705
1 fcntl_setlk64 0.0015
5 __d_lookup 0.0120
128 inode_times_differ 1.8824
27 update_atime 0.1216
35 inode_update_time 0.1636
225 dnotify_parent 0.8893
44 __mark_inode_dirty 0.0909
1 __writeback_single_inode 0.0045
2 sync_sb_inodes 0.0026
1 writeback_acquire 0.0455
15 mpage_writepages 0.0179
1 proc_lookup 0.0030
1 get_vmalloc_info 0.0052
13 show_stat 0.0057
10 write_profile 0.0641
27 ext3_get_group_desc 0.1436
6 read_block_bitmap 0.0395
4 find_next_usable_block 0.0047
70 ext3_try_to_allocate 0.0787
6 ext3_has_free_blocks 0.0488
58 ext3_new_block 0.0446
29 ext3_file_write 0.1362
6 ext3_alloc_block 0.1017
19 ext3_block_to_path 0.0597
34 ext3_get_branch 0.1441
22 ext3_find_goal 0.1606
24 ext3_alloc_branch 0.0347
41 ext3_splice_branch 0.0865
70 ext3_get_block_handle 0.0787
18 ext3_get_block 0.1034
66 walk_page_buffers 0.5893
30 ext3_prepare_write 0.0943
5 ext3_journal_dirty_data 0.0538
88 ext3_ordered_commit_write 0.3843
15 bget_one 1.3636
1 bput_one 0.0909
4 journal_dirty_data_fn 0.2105
12 ext3_ordered_writepage 0.0260
55 ext3_get_inode_block 0.2107
37 ext3_get_inode_loc 0.0582
77 ext3_do_update_inode 0.0834
28 ext3_writepage_trans_blocks 0.1972
34 ext3_mark_iloc_dirty 0.6182
21 ext3_reserve_inode_write 0.0995
24 ext3_mark_inode_dirty 0.2963
18 ext3_dirty_inode 0.1333
20 ext3_journal_start 0.2247
18 __ext3_journal_stop 0.2250
77 start_this_handle 0.0601
15 new_handle 0.1875
38 journal_start 0.1810
179 do_get_write_access 0.1002
21 journal_get_write_access 0.2593
1 journal_get_create_access 0.0015
27 journal_get_undo_access 0.0651
57 journal_dirty_data 0.0780
141 journal_dirty_metadata 0.2568
101 journal_stop 0.1241
118 __journal_unfile_buffer 0.2282
32 __journal_file_buffer 0.0456
1 __journal_refile_buffer 0.0029
25 inverted_lock 0.2604
241 journal_commit_transaction 0.0395
1 journal_brelse_array 0.0227
60 journal_cancel_revoke 0.1729
3 __log_space_left 0.0291
8 journal_blocks_per_page 0.2759
14 journal_alloc_journal_head 0.1207
2 journal_free_journal_head 0.0714
270 journal_add_journal_head 0.7377
38 __journal_remove_journal_head 0.0955
59 journal_remove_journal_head 0.4758
110 journal_put_journal_head 0.4933
1 semctl_main 0.0010
1 __bitmap_weight 0.0035
6 radix_tree_preload 0.0339
4 number 0.0051
5 vsnprintf 0.0041
2 find_next_zero_bit 0.0114
31 csum_partial 0.1076
2 atomic_dec_and_lock 0.0172
22 memcpy 0.3860
6003 __copy_user_intel 34.9012
4315 __copy_user_zeroing_intel 25.0872
101 __copy_to_user_ll 0.8417
2347 __copy_from_user_ll 19.5583
30 copy_to_user 0.3061
28 copy_from_user 0.2222
3 add_disk_randomness 0.0577
1 blk_run_queue 0.0081
119 __make_request 0.0856
18 generic_make_request 0.0366
12 submit_bio 0.0415
67 __end_that_request_first 0.1274
1 put_io_context 0.0114
15 .text.lock.ll_rw_blk 0.1103
2 as_set_request 0.0153
303 e1000_xmit_frame 0.1424
9 scsi_put_command 0.0398
39 scsi_dispatch_cmd 0.0724
1 scsi_softirq 0.0048
2 scsi_finish_command 0.0112
7 scsi_decide_disposition 0.0234
5 scsi_init_cmd_errh 0.0260
3 scsi_device_unbusy 0.0163
3 scsi_run_queue 0.0109
30 scsi_end_request 0.1136
1 scsi_free_sgtable 0.0222
3 scsi_io_completion 0.0029
154 scsi_request_fn 0.1419
22 .text.lock.scsi_lib 0.1028
65 ahc_linux_isr 0.0935
27 sock_aio_read 0.0918
21 sock_aio_write 0.0714
30 sock_poll 0.6122
181 sock_wfree 2.1047
18 __release_sock 0.1169
357 sock_def_readable 1.9833
11 sk_reset_timer 0.2340
73 lock_sock 0.5887
54 release_sock 0.3553
40 sock_common_recvmsg 0.3419
158 .text.lock.sock 1.2951
232 alloc_skb 1.0357
368 skb_release_data 2.1149
15 kfree_skbmem 0.3409
640 __kfree_skb 2.8070
81 skb_clone 0.2109
31 copy_skb_header 0.1250
23 skb_copy 0.1036
282 skb_copy_bits 0.5361
10 skb_checksum 0.0162
15 memcpy_toiovec 0.1579
2 memcpy_tokerneliovec 0.0267
131 skb_copy_datagram_iovec 0.2417
32 sk_stream_rfree 1.1852
2 sk_stream_error 0.0215
18 skb_checksum_help 0.0556
61 dev_queue_xmit 0.0766
15 net_tx_action 0.0390
68 netif_receive_skb 0.1528
118 process_backlog 0.4322
52 net_rx_action 0.2008
210 nf_iterate 1.2727
90 nf_hook_slow 0.3147
167 qdisc_restart 0.2831
124 pfifo_fast_enqueue 0.8552
113 pfifo_fast_dequeue 1.3951
36 rt_hash_code 0.2628
275 ip_route_input 0.5055
31 ip_local_deliver 0.0613
156 ip_rcv 0.1221
60 ip_local_deliver_finish 0.1538
46 ip_rcv_finish 0.0710
10 ip_finish_output 0.0188
31 ip_output 0.2583
288 ip_queue_xmit 0.1910
12 dst_output 0.2927
216 ip_finish_output2 0.4876
124 tcp_poll 0.3229
187 tcp_sendmsg 0.0411
44 cleanup_rbuf 0.1429
427 tcp_recvmsg 0.2126
27 __tcp_grow_window 0.2061
73 tcp_rcv_rtt_update 0.8022
42 tcp_rcv_space_adjust 0.1288
215 tcp_event_data_recv 0.3595
25 tcp_rtt_estimator 0.0744
1 tcp_fastretrans_alert 0.0005
7 tcp_ack_saw_tstamp 0.0833
83 tcp_clean_rtx_queue 0.0873
1 tcp_ack_update_window 0.0039
36 tcp_ack 0.0246
2 tcp_ofo_queue 0.0033
1 tcp_data_queue 0.0003
7 __tcp_data_snd_check 0.0297
38 __tcp_ack_snd_check 0.2360
1 tcp_check_urg 0.0036
705 tcp_rcv_established 0.3052
310 tcp_transmit_skb 0.1780
54 tcp_write_xmit 0.0745
58 __tcp_select_window 0.1953
27 tcp_send_delayed_ack 0.1222
68 tcp_send_ack 0.3487
217 __tcp_v4_lookup_established 0.8715
17 tcp_v4_send_check 0.0672
55 tcp_v4_do_rcv 0.1730
432 tcp_v4_rcv 0.2141
5 inet_sendmsg 0.0562
21 ip_confirm 0.4375
22 ip_conntrack_defrag 0.3143
23 ip_refrag 0.1608
21 ip_conntrack_local 0.1438
22 __ip_ct_find_proto 0.3385
314 ip_ct_find_proto 2.9346
65 hash_conntrack 0.4221
33 get_tuple 0.3548
3 invert_tuple 0.0638
50 __ip_conntrack_find 0.2283
149 ip_conntrack_find_get 1.2314
7 ip_conntrack_get 0.1944
110 ip_conntrack_in 0.1789
321 ip_ct_refresh 1.6895
3 ip_ct_gather_frags 0.0129
18 tcp_pkt_to_tuple 0.2118
14 get_conntrack_index 0.2295
442 tcp_packet 0.9345
882 ipt_do_table 0.9102
9 ipt_hook 0.1525
17 ipt_local_out_hook 0.1453
31 ipt_route_hook 0.5254
29 ipt_local_hook 0.1245
45 ip_nat_fn 0.0748
35 ip_nat_out 0.2035
171 do_bindings 0.2098
1 unix_create1 0.0023
1 unix_accept 0.0024
7 __sched_text_start 0.0241
1 __down_failed 0.0833
1 __write_lock_failed 0.0312
4 __read_lock_failed 0.2000
215 schedule 0.1033
1 io_schedule_timeout 0.0196
4 __preempt_spin_lock 0.0440
476 .text.lock.sched 2.4286
4 schedule_timeout 0.0209
461526 total 0.1500
19 default_idle 0.4222
415965 poll_idle 7171.8103
41 cpu_idle 0.6406
2 need_resched 0.0400
117 system_call 2.6591
4 syscall_call 0.3636
21 syscall_exit 1.9091
3 handle_IRQ_event 0.0300
1 old_mmap 0.0030
2 show_cpuinfo 0.0030
27 sched_clock 0.1985
1 flush_tlb_page 0.0056
2 pte_alloc_one 0.0260
1 pgd_ctor 0.0060
16 do_page_fault 0.0115
1631 finish_task_switch 10.9463
1 sched_balance_exec 0.0058
1027 __wake_up 8.7034
219 __might_sleep 1.0379
80 add_wait_queue 0.6452
53 remove_wait_queue 0.4141
30 prepare_to_wait 0.2041
1 prepare_to_wait_exclusive 0.0068
45 finish_wait 0.2848
1 copy_mm 0.0009
1 copy_files 0.0010
1 unshare_files 0.0102
7 .text.lock.fork 0.0354
1 do_exit 0.0009
1 do_setitimer 0.0021
79 current_kernel_time 1.1449
2 get_jiffies_64 0.0345
5187 __do_softirq 29.9827
169 local_bh_enable 1.1736
645 __mod_timer 1.2476
55 mod_timer 0.5189
284 del_timer 1.8442
1 del_timer_sync 0.0055
2 .text.lock.timer 0.0217
3 get_signal_to_deliver 0.0029
2 groups_search 0.0189
46 add_to_page_cache 0.1878
15 page_waitqueue 0.3125
60 wake_up_page 0.7792
12 unlock_page 0.3243
16 end_page_writeback 0.2162
43 find_get_page 0.3839
35 find_lock_page 0.1207
62 find_get_pages_tag 0.3899
2 do_generic_mapping_read 0.0016
2 file_read_actor 0.0083
1 __generic_file_aio_read 0.0018
1 filemap_nopage 0.0011
4 remove_suid 0.0268
132 generic_file_aio_write_nolock 0.0456
18 generic_file_aio_write 0.1139
32 mempool_alloc 0.0840
20 mempool_free 0.0952
5 mempool_alloc_slab 0.2381
21 bad_range 0.2100
19 prep_new_page 0.2346
43 free_hot_cold_page 0.1443
3 free_hot_page 0.4286
160 buffered_rmqueue 0.3008
68 __alloc_pages 0.0766
8 __get_free_pages 0.1356
1 get_zeroed_page 0.0098
20 __free_pages 0.2564
3 free_pages 0.0667
1 get_zone_counts 0.0090
1 get_writeback_state 0.0128
1 get_dirty_limits 0.0049
14 balance_dirty_pages_ratelimited 0.0697
66 __set_page_dirty_nobuffers 0.1755
19 clear_page_dirty_for_io 0.1667
58 test_clear_page_writeback 0.2275
47 test_set_page_writeback 0.1567
17 .text.lock.page_writeback 0.1328
1 __pdflush 0.0018
1 do_page_cache_readahead 0.0022
1 kmem_getpages 0.0048
2 alloc_slabmgmt 0.0208
6 cache_init_objs 0.0577
1 kmem_flagcheck 0.0233
4 cache_grow 0.0088
162 kmem_cache_alloc 1.3171
1 kmem_ptr_validate 0.0132
88 __kmalloc 0.5714
203 kmem_cache_free 2.3333
186 kfree 1.7064
37 mark_page_accessed 0.7115
38 release_pages 0.0841
1 __pagevec_release 0.0185
35 __pagevec_lru_add 0.1104
1 __pagevec_lru_add_active 0.0030
1 vm_acct_memory 0.0122
10 percpu_counter_mod 0.0571
1 blk_queue_bounce 0.0130
1 clear_page_tables 0.0032
10 copy_page_range 0.0129
21 zap_pte_range 0.0303
1 zap_pmd_range 0.0087
11 do_wp_page 0.0130
17 do_anonymous_page 0.0379
303 do_no_page 0.3401
5 handle_mm_fault 0.0114
1 vma_link 0.0030
3 vma_adjust 0.0030
1 can_vma_merge_before 0.0098
1 can_vma_merge_after 0.0090
3 do_mmap_pgoff 0.0016
1 find_vma 0.0123
1 expand_stack 0.0034
1 split_vma 0.0035
1 sys_munmap 0.0082
1 exit_mmap 0.0022
1 anon_vma_link 0.0079
1 page_add_anon_rmap 0.0031
4 page_add_file_rmap 0.0185
8 page_remove_rmap 0.0335
7 do_sync_read 0.0405
59 vfs_read 0.2000
23 do_sync_write 0.1329
56 vfs_write 0.1898
20 sys_read 0.1562
21 sys_write 0.1641
38 fput 1.8095
1 __fput 0.0047
102 fget 0.7969
50 fget_light 0.2994
1 file_move 0.0071
8 bh_waitq_head 0.3636
60 wake_up_buffer 0.7229
27 unlock_buffer 1.1250
13 __set_page_buffers 0.5200
65 end_buffer_write_sync 0.5372
15 __find_get_block_slow 0.0346
38 end_buffer_async_write 0.1230
22 create_buffers 0.1350
7 mark_buffer_dirty 0.1750
43 __brelse 0.5513
1 bh_lru_install 0.0044
140 __find_get_block 0.5691
21 __getblk 0.2079
13 __bread 0.1970
11 set_bh_page 0.2037
21 create_empty_buffers 0.1040
3 unmap_underlying_metadata 0.0265
50 __block_write_full_page 0.0521
56 __block_prepare_write 0.0564
19 __block_commit_write 0.1275
11 block_prepare_write 0.1429
12 generic_commit_write 0.0811
21 block_write_full_page 0.0787
21 end_bio_bh_io_sync 0.3387
34 submit_bh 0.1043
28 ll_rw_block 0.2059
5 recalc_bh_state 0.0279
10 alloc_buffer_head 0.0952
20 init_buffer_head 0.3774
5 .text.lock.buffer 0.0269
5 bio_destructor 0.0532
25 bio_alloc 0.0587
53 bio_put 0.9815
5 bio_endio 0.0394
1 __put_super 0.0233
16 inode_add_bytes 0.1127
1 setup_arg_pages 0.0021
57 pipe_wait 0.3184
227 pipe_readv 0.3211
6 pipe_read 0.1017
145 pipe_writev 0.1749
9 pipe_write 0.1525
130 pipe_poll 1.0484
1 getname 0.0041
1 do_lookup 0.0062
5 link_path_walk 0.0013
1 may_open 0.0019
1 open_namei 0.0006
10 kill_fasync 0.0862
3 poll_initwait 0.1200
33 poll_freewait 0.5156
31 __pollwait 0.1574
4 max_select_fd 0.0178
142 do_select 0.1816
5 select_bits_alloc 0.1562
2 select_bits_free 0.4000
114 sys_select 0.0884
1 locks_remove_flock 0.0039
1 dput 0.0016
7 __d_lookup 0.0168
1 d_rehash 0.0046
132 inode_times_differ 1.9412
38 update_atime 0.1712
28 inode_update_time 0.1308
191 dnotify_parent 0.7549
58 __mark_inode_dirty 0.1198
2 __sync_single_inode 0.0036
1 sync_sb_inodes 0.0013
13 mpage_writepages 0.0155
1 proc_lookup 0.0030
1 get_vmalloc_info 0.0052
6 show_stat 0.0026
6 write_profile 0.0385
26 ext3_get_group_desc 0.1383
4 read_block_bitmap 0.0263
3 find_next_usable_block 0.0035
50 ext3_try_to_allocate 0.0562
18 ext3_has_free_blocks 0.1463
48 ext3_new_block 0.0370
14 ext3_file_write 0.0657
6 ext3_alloc_block 0.1017
25 ext3_block_to_path 0.0786
45 ext3_get_branch 0.1907
1 ext3_find_near 0.0078
22 ext3_find_goal 0.1606
14 ext3_alloc_branch 0.0202
30 ext3_splice_branch 0.0633
78 ext3_get_block_handle 0.0877
12 ext3_get_block 0.0690
60 walk_page_buffers 0.5357
34 ext3_prepare_write 0.1069
6 ext3_journal_dirty_data 0.0645
94 ext3_ordered_commit_write 0.4105
1 ext3_bmap 0.0065
14 bget_one 1.2727
2 bput_one 0.1818
5 journal_dirty_data_fn 0.2632
18 ext3_ordered_writepage 0.0390
62 ext3_get_inode_block 0.2375
33 ext3_get_inode_loc 0.0519
100 ext3_do_update_inode 0.1083
33 ext3_writepage_trans_blocks 0.2324
26 ext3_mark_iloc_dirty 0.4727
19 ext3_reserve_inode_write 0.0900
18 ext3_mark_inode_dirty 0.2222
25 ext3_dirty_inode 0.1852
31 ext3_journal_start 0.3483
18 __ext3_journal_stop 0.2250
56 start_this_handle 0.0437
20 new_handle 0.2500
18 journal_start 0.0857
202 do_get_write_access 0.1131
22 journal_get_write_access 0.2716
1 journal_get_create_access 0.0015
30 journal_get_undo_access 0.0723
58 journal_dirty_data 0.0793
129 journal_dirty_metadata 0.2350
99 journal_stop 0.1216
96 __journal_unfile_buffer 0.1857
34 __journal_file_buffer 0.0485
1 release_buffer_page 0.0063
27 inverted_lock 0.2812
236 journal_commit_transaction 0.0387
1 journal_brelse_array 0.0227
1 __try_to_free_cp_buf 0.0054
51 journal_cancel_revoke 0.1470
5 __log_space_left 0.0485
1 journal_get_descriptor_buffer 0.0053
7 journal_blocks_per_page 0.2414
7 journal_alloc_journal_head 0.0603
3 journal_free_journal_head 0.1071
270 journal_add_journal_head 0.7377
45 __journal_remove_journal_head 0.1131
41 journal_remove_journal_head 0.3306
97 journal_put_journal_head 0.4350
9 radix_tree_preload 0.0508
1 number 0.0013
5 vsnprintf 0.0041
1 find_next_zero_bit 0.0057
24 csum_partial 0.0833
3 atomic_dec_and_lock 0.0259
15 memcpy 0.2632
1 strncpy_from_user 0.0101
6193 __copy_user_intel 36.0058
4430 __copy_user_zeroing_intel 25.7558
99 __copy_to_user_ll 0.8250
2306 __copy_from_user_ll 19.2167
31 copy_to_user 0.3163
31 copy_from_user 0.2460
2 add_disk_randomness 0.0385
1 SHATransform 0.0033
1 put_device 0.1000
2 blk_run_queue 0.0161
3 get_request 0.0040
116 __make_request 0.0835
14 generic_make_request 0.0285
8 submit_bio 0.0277
66 __end_that_request_first 0.1255
2 put_io_context 0.0227
1 get_io_context 0.0080
10 .text.lock.ll_rw_blk 0.0735
1 as_set_request 0.0076
332 e1000_xmit_frame 0.1560
4 scsi_put_command 0.0177
23 scsi_dispatch_cmd 0.0427
2 scsi_softirq 0.0096
2 scsi_finish_command 0.0112
1 scsi_add_timer 0.0093
1 scsi_decide_disposition 0.0033
1 scsi_init_cmd_errh 0.0052
6 scsi_device_unbusy 0.0326
3 scsi_run_queue 0.0109
2 scsi_next_command 0.0488
20 scsi_end_request 0.0758
1 scsi_free_sgtable 0.0222
2 scsi_io_completion 0.0020
120 scsi_request_fn 0.1106
10 .text.lock.scsi_lib 0.0467
49 ahc_linux_isr 0.0705
2 sd_rw_intr 0.0030
23 sock_aio_read 0.0782
14 sock_aio_write 0.0476
22 sock_poll 0.4490
192 sock_wfree 2.2326
13 __release_sock 0.0844
357 sock_def_readable 1.9833
12 sk_reset_timer 0.2553
75 lock_sock 0.6048
33 release_sock 0.2171
41 sock_common_recvmsg 0.3504
187 .text.lock.sock 1.5328
237 alloc_skb 1.0580
375 skb_release_data 2.1552
11 kfree_skbmem 0.2500
608 __kfree_skb 2.6667
83 skb_clone 0.2161
31 copy_skb_header 0.1250
23 skb_copy 0.1036
297 skb_copy_bits 0.5646
10 skb_checksum 0.0162
14 memcpy_toiovec 0.1474
125 skb_copy_datagram_iovec 0.2306
34 sk_stream_rfree 1.2593
6 sk_stream_error 0.0645
15 skb_checksum_help 0.0463
56 dev_queue_xmit 0.0704
28 net_tx_action 0.0727
64 netif_receive_skb 0.1438
107 process_backlog 0.3919
60 net_rx_action 0.2317
193 nf_iterate 1.1697
96 nf_hook_slow 0.3357
156 qdisc_restart 0.2644
147 pfifo_fast_enqueue 1.0138
108 pfifo_fast_dequeue 1.3333
34 rt_hash_code 0.2482
2 rt_cache_get_first 0.0238
303 ip_route_input 0.5570
39 ip_local_deliver 0.0771
122 ip_rcv 0.0955
63 ip_local_deliver_finish 0.1615
46 ip_rcv_finish 0.0710
8 ip_finish_output 0.0150
19 ip_output 0.1583
317 ip_queue_xmit 0.2102
10 dst_output 0.2439
231 ip_finish_output2 0.5214
135 tcp_poll 0.3516
169 tcp_sendmsg 0.0372
45 cleanup_rbuf 0.1461
410 tcp_recvmsg 0.2042
31 __tcp_grow_window 0.2366
55 tcp_rcv_rtt_update 0.6044
32 tcp_rcv_space_adjust 0.0982
177 tcp_event_data_recv 0.2960
35 tcp_rtt_estimator 0.1042
1 tcp_fastretrans_alert 0.0005
7 tcp_ack_saw_tstamp 0.0833
57 tcp_clean_rtx_queue 0.0599
1 tcp_ack_update_window 0.0039
51 tcp_ack 0.0348
11 __tcp_data_snd_check 0.0466
45 __tcp_ack_snd_check 0.2795
726 tcp_rcv_established 0.3143
328 tcp_transmit_skb 0.1883
41 tcp_write_xmit 0.0566
70 __tcp_select_window 0.2357
26 tcp_send_delayed_ack 0.1176
66 tcp_send_ack 0.3385
236 __tcp_v4_lookup_established 0.9478
23 tcp_v4_send_check 0.0909
49 tcp_v4_do_rcv 0.1541
390 tcp_v4_rcv 0.1933
8 inet_sendmsg 0.0899
26 ip_confirm 0.5417
23 ip_conntrack_defrag 0.3286
20 ip_refrag 0.1399
18 ip_conntrack_local 0.1233
22 __ip_ct_find_proto 0.3385
260 ip_ct_find_proto 2.4299
55 hash_conntrack 0.3571
45 get_tuple 0.4839
1 invert_tuple 0.0213
55 __ip_conntrack_find 0.2511
152 ip_conntrack_find_get 1.2562
8 ip_conntrack_get 0.2222
125 ip_conntrack_in 0.2033
265 ip_ct_refresh 1.3947
3 ip_ct_gather_frags 0.0129
19 tcp_pkt_to_tuple 0.2235
11 get_conntrack_index 0.1803
456 tcp_packet 0.9641
862 ipt_do_table 0.8896
15 ipt_hook 0.2542
14 ipt_local_out_hook 0.1197
40 ipt_route_hook 0.6780
23 ipt_local_hook 0.0987
50 ip_nat_fn 0.0831
20 ip_nat_out 0.1163
191 do_bindings 0.2344
7 __sched_text_start 0.0241
1 __write_lock_failed 0.0312
4 __read_lock_failed 0.2000
214 schedule 0.1028
1 io_schedule_timeout 0.0196
5 __preempt_spin_lock 0.0549
460 .text.lock.sched 2.3469
4 schedule_timeout 0.0209
458958 total 0.1491
19 default_idle 0.4222
416458 poll_idle 7180.3103
32 cpu_idle 0.5000
5 need_resched 0.1000
144 system_call 3.2727
7 syscall_call 0.6364
10 syscall_exit 0.9091
4 handle_IRQ_event 0.0400
1 old_mmap 0.0030
1 show_cpuinfo 0.0015
22 sched_clock 0.1618
2 delay_tsc 0.0952
5 pte_alloc_one 0.0649
20 do_page_fault 0.0143
1683 finish_task_switch 11.2953
1022 __wake_up 8.6610
1 migration_thread 0.0022
221 __might_sleep 1.0474
1 task_curr 0.0044
87 add_wait_queue 0.7016
52 remove_wait_queue 0.4062
29 prepare_to_wait 0.1973
45 finish_wait 0.2848
2 copy_mm 0.0017
1 copy_process 0.0003
4 .text.lock.fork 0.0202
1 release_task 0.0018
1 exit_notify 0.0004
1 eligible_child 0.0051
117 current_kernel_time 1.6957
5062 __do_softirq 29.2601
177 local_bh_enable 1.2292
690 __mod_timer 1.3346
64 mod_timer 0.6038
303 del_timer 1.9675
1 del_timer_sync 0.0055
2 .text.lock.timer 0.0217
1 flush_signal_handlers 0.0149
1 do_sigaction 0.0016
2 in_group_p 0.0167
1 sys_futex 0.0037
45 add_to_page_cache 0.1837
12 page_waitqueue 0.2500
49 wake_up_page 0.6364
9 unlock_page 0.2432
14 end_page_writeback 0.1892
47 find_get_page 0.4196
29 find_lock_page 0.1000
64 find_get_pages_tag 0.4025
3 filemap_nopage 0.0032
6 remove_suid 0.0403
126 generic_file_aio_write_nolock 0.0435
16 generic_file_aio_write 0.1013
29 mempool_alloc 0.0761
21 mempool_free 0.1000
3 mempool_alloc_slab 0.1429
2 mempool_free_slab 0.0952
22 bad_range 0.2200
12 prep_new_page 0.1481
68 free_hot_cold_page 0.2282
1 free_hot_page 0.1429
163 buffered_rmqueue 0.3064
78 __alloc_pages 0.0878
2 __get_free_pages 0.0339
1 __pagevec_free 0.0303
21 __free_pages 0.2692
3 free_pages 0.0667
1 __read_page_state 0.0179
7 balance_dirty_pages_ratelimited 0.0348
73 __set_page_dirty_nobuffers 0.1941
3 set_page_dirty 0.0357
14 clear_page_dirty_for_io 0.1228
42 test_clear_page_writeback 0.1647
62 test_set_page_writeback 0.2067
5 .text.lock.page_writeback 0.0391
1 pdflush_operation 0.0042
1 vma_prio_tree_remove 0.0039
3 do_page_cache_readahead 0.0065
1 kmem_getpages 0.0048
2 alloc_slabmgmt 0.0208
9 cache_init_objs 0.0865
1 kmem_flagcheck 0.0233
1 set_slab_attr 0.0172
3 cache_grow 0.0066
164 kmem_cache_alloc 1.3333
3 kmem_ptr_validate 0.0395
79 __kmalloc 0.5130
202 kmem_cache_free 2.3218
192 kfree 1.7615
27 mark_page_accessed 0.5192
1 lru_cache_add 0.0105
1 lru_cache_add_active 0.0105
35 release_pages 0.0774
29 __pagevec_lru_add 0.0915
4 __pagevec_lru_add_active 0.0119
17 percpu_counter_mod 0.0971
2 clear_page_tables 0.0065
3 pte_alloc_map 0.0121
8 copy_page_range 0.0103
15 zap_pte_range 0.0216
1 unmap_page_range 0.0088
1 unmap_vmas 0.0017
24 do_wp_page 0.0283
19 do_anonymous_page 0.0423
329 do_no_page 0.3692
3 handle_mm_fault 0.0068
2 remove_vm_struct 0.0123
1 find_vma_prepare 0.0099
1 vma_adjust 0.0010
1 can_vma_merge_after 0.0090
1 vma_merge 0.0020
4 do_mmap_pgoff 0.0022
1 find_vma 0.0123
1 free_pgtables 0.0067
1 unmap_vma_list 0.0250
2 split_vma 0.0070
2 anon_vma_prepare 0.0057
1 anon_vma_link 0.0079
2 page_add_anon_rmap 0.0061
12 page_remove_rmap 0.0502
1 free_pages_and_swap_cache 0.0076
1 filp_open 0.0095
2 get_unused_fd 0.0078
1 fd_install 0.0068
10 do_sync_read 0.0578
64 vfs_read 0.2169
19 do_sync_write 0.1098
62 vfs_write 0.2102
28 sys_read 0.2188
26 sys_write 0.2031
40 fput 1.9048
1 __fput 0.0047
95 fget 0.7422
48 fget_light 0.2874
6 bh_waitq_head 0.2727
83 wake_up_buffer 1.0000
24 unlock_buffer 1.0000
12 __set_page_buffers 0.4800
80 end_buffer_write_sync 0.6612
12 __find_get_block_slow 0.0277
26 end_buffer_async_write 0.0841
28 create_buffers 0.1718
15 mark_buffer_dirty 0.3750
38 __brelse 0.4872
144 __find_get_block 0.5854
18 __getblk 0.1782
7 __bread 0.1061
7 set_bh_page 0.1296
25 create_empty_buffers 0.1238
5 unmap_underlying_metadata 0.0442
54 __block_write_full_page 0.0563
61 __block_prepare_write 0.0614
21 __block_commit_write 0.1409
4 block_prepare_write 0.0519
7 generic_commit_write 0.0473
20 block_write_full_page 0.0749
33 end_bio_bh_io_sync 0.5323
42 submit_bh 0.1288
32 ll_rw_block 0.2353
5 recalc_bh_state 0.0279
12 alloc_buffer_head 0.1143
15 init_buffer_head 0.2830
2 .text.lock.buffer 0.0108
3 bio_destructor 0.0319
19 bio_alloc 0.0446
54 bio_put 1.0000
7 bio_endio 0.0551
1 nr_blockdev_pages 0.0075
1 cdev_put 0.0435
16 inode_add_bytes 0.1127
1 copy_strings 0.0018
1 install_arg_page 0.0034
49 pipe_wait 0.2737
243 pipe_readv 0.3437
12 pipe_read 0.2034
139 pipe_writev 0.1677
7 pipe_write 0.1186
1 bad_pipe_r 0.1667
134 pipe_poll 1.0806
1 vfs_permission 0.0032
3 do_lookup 0.0186
3 link_path_walk 0.0008
2 path_lookup 0.0042
1 may_open 0.0019
7 kill_fasync 0.0603
3 poll_initwait 0.1200
30 poll_freewait 0.4688
29 __pollwait 0.1472
12 max_select_fd 0.0533
170 do_select 0.2174
3 select_bits_alloc 0.0938
2 select_bits_free 0.4000
101 sys_select 0.0783
1 dput 0.0016
1 __d_lookup 0.0024
1 iget_locked 0.0039
126 inode_times_differ 1.8529
19 update_atime 0.0856
36 inode_update_time 0.1682
223 dnotify_parent 0.8814
1 seq_printf 0.0116
73 __mark_inode_dirty 0.1508
1 __sync_single_inode 0.0018
1 __writeback_single_inode 0.0045
1 writeback_release 0.0208
14 mpage_writepages 0.0167
1 de_put 0.0037
5 show_stat 0.0022
12 write_profile 0.0769
22 ext3_get_group_desc 0.1170
10 read_block_bitmap 0.0658
4 find_next_usable_block 0.0047
48 ext3_try_to_allocate 0.0539
9 ext3_has_free_blocks 0.0732
67 ext3_new_block 0.0516
16 ext3_file_write 0.0751
6 ext3_alloc_block 0.1017
21 ext3_block_to_path 0.0660
31 ext3_get_branch 0.1314
24 ext3_find_goal 0.1752
12 ext3_alloc_branch 0.0173
41 ext3_splice_branch 0.0865
67 ext3_get_block_handle 0.0754
14 ext3_get_block 0.0805
80 walk_page_buffers 0.7143
28 ext3_prepare_write 0.0881
6 ext3_journal_dirty_data 0.0645
90 ext3_ordered_commit_write 0.3930
19 bget_one 1.7273
4 bput_one 0.3636
5 journal_dirty_data_fn 0.2632
14 ext3_ordered_writepage 0.0304
61 ext3_get_inode_block 0.2337
39 ext3_get_inode_loc 0.0613
70 ext3_do_update_inode 0.0758
30 ext3_writepage_trans_blocks 0.2113
35 ext3_mark_iloc_dirty 0.6364
24 ext3_reserve_inode_write 0.1137
13 ext3_mark_inode_dirty 0.1605
19 ext3_dirty_inode 0.1407
19 ext3_journal_start 0.2135
18 __ext3_journal_stop 0.2250
85 start_this_handle 0.0663
17 new_handle 0.2125
26 journal_start 0.1238
181 do_get_write_access 0.1013
23 journal_get_write_access 0.2840
1 journal_get_create_access 0.0015
22 journal_get_undo_access 0.0530
53 journal_dirty_data 0.0725
122 journal_dirty_metadata 0.2222
97 journal_stop 0.1192
109 __journal_unfile_buffer 0.2108
51 __journal_file_buffer 0.0728
31 inverted_lock 0.3229
268 journal_commit_transaction 0.0439
2 __journal_clean_checkpoint_list 0.0115
67 journal_cancel_revoke 0.1931
7 __log_space_left 0.0680
9 journal_blocks_per_page 0.3103
1 __jbd_kmalloc 0.0227
7 journal_alloc_journal_head 0.0603
4 journal_free_journal_head 0.1429
263 journal_add_journal_head 0.7186
50 __journal_remove_journal_head 0.1256
61 journal_remove_journal_head 0.4919
109 journal_put_journal_head 0.4888
1 sys_semtimedop 0.0008
1 sys_shmget 0.0031
2 __bitmap_weight 0.0071
4 radix_tree_preload 0.0226
9 number 0.0115
6 vsnprintf 0.0049
1 sprintf 0.0286
37 csum_partial 0.1285
2 atomic_dec_and_lock 0.0172
11 memcpy 0.1930
1 strncpy_from_user 0.0101
6219 __copy_user_intel 36.1570
4370 __copy_user_zeroing_intel 25.4070
84 __copy_to_user_ll 0.7000
2333 __copy_from_user_ll 19.4417
29 copy_to_user 0.2959
34 copy_from_user 0.2698
1 add_disk_randomness 0.0192
2 blk_run_queue 0.0161
4 get_request 0.0054
108 __make_request 0.0777
10 generic_make_request 0.0203
12 submit_bio 0.0415
59 __end_that_request_first 0.1122
9 .text.lock.ll_rw_blk 0.0662
1 as_set_request 0.0076
317 e1000_xmit_frame 0.1490
2 scsi_put_command 0.0088
11 scsi_dispatch_cmd 0.0204
2 scsi_finish_command 0.0112
1 scsi_add_timer 0.0093
1 scsi_decide_disposition 0.0033
1 scsi_init_cmd_errh 0.0052
1 scsi_device_unbusy 0.0054
1 scsi_run_queue 0.0036
24 scsi_end_request 0.0909
3 scsi_io_completion 0.0029
119 scsi_request_fn 0.1097
6 .text.lock.scsi_lib 0.0280
35 ahc_linux_isr 0.0504
3 sd_rw_intr 0.0045
25 sock_aio_read 0.0850
19 sock_aio_write 0.0646
21 sock_poll 0.4286
213 sock_wfree 2.4767
18 __release_sock 0.1169
335 sock_def_readable 1.8611
10 sk_reset_timer 0.2128
72 lock_sock 0.5806
49 release_sock 0.3224
37 sock_common_recvmsg 0.3162
182 .text.lock.sock 1.4918
241 alloc_skb 1.0759
409 skb_release_data 2.3506
15 kfree_skbmem 0.3409
646 __kfree_skb 2.8333
58 skb_clone 0.1510
29 copy_skb_header 0.1169
20 skb_copy 0.0901
313 skb_copy_bits 0.5951
12 skb_checksum 0.0194
15 memcpy_toiovec 0.1579
1 memcpy_tokerneliovec 0.0133
128 skb_copy_datagram_iovec 0.2362
47 sk_stream_rfree 1.7407
7 sk_stream_error 0.0753
1 sk_stream_mem_schedule 0.0029
11 skb_checksum_help 0.0340
64 dev_queue_xmit 0.0804
26 net_tx_action 0.0675
82 netif_receive_skb 0.1843
127 process_backlog 0.4652
55 net_rx_action 0.2124
187 nf_iterate 1.1333
88 nf_hook_slow 0.3077
152 qdisc_restart 0.2576
151 pfifo_fast_enqueue 1.0414
88 pfifo_fast_dequeue 1.0864
33 rt_hash_code 0.2409
256 ip_route_input 0.4706
39 ip_local_deliver 0.0771
150 ip_rcv 0.1174
64 ip_local_deliver_finish 0.1641
35 ip_rcv_finish 0.0540
15 ip_finish_output 0.0282
21 ip_output 0.1750
305 ip_queue_xmit 0.2023
8 dst_output 0.1951
226 ip_finish_output2 0.5102
121 tcp_poll 0.3151
180 tcp_sendmsg 0.0396
39 cleanup_rbuf 0.1266
448 tcp_recvmsg 0.2231
24 __tcp_grow_window 0.1832
73 tcp_rcv_rtt_update 0.8022
37 tcp_rcv_space_adjust 0.1135
187 tcp_event_data_recv 0.3127
37 tcp_rtt_estimator 0.1101
3 tcp_ack_saw_tstamp 0.0357
70 tcp_clean_rtx_queue 0.0736
26 tcp_ack 0.0178
2 tcp_data_queue 0.0007
12 __tcp_data_snd_check 0.0508
45 __tcp_ack_snd_check 0.2795
727 tcp_rcv_established 0.3147
356 tcp_transmit_skb 0.2044
52 tcp_write_xmit 0.0717
35 __tcp_select_window 0.1178
14 tcp_send_delayed_ack 0.0633
56 tcp_send_ack 0.2872
205 __tcp_v4_lookup_established 0.8233
25 tcp_v4_send_check 0.0988
53 tcp_v4_do_rcv 0.1667
430 tcp_v4_rcv 0.2131
5 inet_sendmsg 0.0562
22 ip_confirm 0.4583
17 ip_conntrack_defrag 0.2429
19 ip_refrag 0.1329
11 ip_conntrack_local 0.0753
23 __ip_ct_find_proto 0.3538
284 ip_ct_find_proto 2.6542
68 hash_conntrack 0.4416
44 get_tuple 0.4731
4 invert_tuple 0.0851
64 __ip_conntrack_find 0.2922
154 ip_conntrack_find_get 1.2727
5 ip_conntrack_get 0.1389
156 ip_conntrack_in 0.2537
266 ip_ct_refresh 1.4000
2 ip_ct_gather_frags 0.0086
24 tcp_pkt_to_tuple 0.2824
6 get_conntrack_index 0.0984
474 tcp_packet 1.0021
897 ipt_do_table 0.9257
7 ipt_hook 0.1186
11 ipt_local_out_hook 0.0940
39 ipt_route_hook 0.6610
27 ipt_local_hook 0.1159
56 ip_nat_fn 0.0930
29 ip_nat_out 0.1686
215 do_bindings 0.2638
2 __sched_text_start 0.0069
1 __write_lock_failed 0.0312
1 __read_lock_failed 0.0500
207 schedule 0.0995
1 io_schedule_timeout 0.0196
4 __preempt_spin_lock 0.0440
495 .text.lock.sched 2.5255
5 schedule_timeout 0.0262
459892 total 0.1494
21 default_idle 0.4667
416413 poll_idle 7179.5345
43 cpu_idle 0.6719
1 setup_sigcontext 0.0034
7 need_resched 0.1400
135 system_call 3.0682
8 syscall_call 0.7273
12 syscall_exit 1.0909
1 error_code 0.0179
5 handle_IRQ_event 0.0500
1 init_new_context 0.0039
25 sched_clock 0.1838
1 delay_tsc 0.0476
4 pte_alloc_one 0.0519
1 pgd_ctor 0.0060
1 pgd_free 0.0357
12 do_page_fault 0.0086
1 wake_up_forked_process 0.0028
1695 finish_task_switch 11.3758
1010 __wake_up 8.5593
204 __might_sleep 0.9668
82 add_wait_queue 0.6613
49 remove_wait_queue 0.3828
24 prepare_to_wait 0.1633
50 finish_wait 0.3165
1 __mmdrop 0.0143
1 count_open_files 0.0233
7 .text.lock.fork 0.0354
1 do_exit 0.0009
122 current_kernel_time 1.7681
5090 __do_softirq 29.4220
164 local_bh_enable 1.1389
615 __mod_timer 1.1896
71 mod_timer 0.6698
304 del_timer 1.9740
1 sigprocmask 0.0041
1 notifier_call_chain 0.0161
1 set_user 0.0075
51 add_to_page_cache 0.2082
18 page_waitqueue 0.3750
43 wake_up_page 0.5584
17 unlock_page 0.4595
15 end_page_writeback 0.2027
62 find_get_page 0.5536
37 find_lock_page 0.1276
66 find_get_pages_tag 0.4151
3 filemap_nopage 0.0032
8 remove_suid 0.0537
125 generic_file_aio_write_nolock 0.0431
24 generic_file_aio_write 0.1519
30 mempool_alloc 0.0787
12 mempool_free 0.0571
14 bad_range 0.1400
19 prep_new_page 0.2346
45 free_hot_cold_page 0.1510
145 buffered_rmqueue 0.2726
75 __alloc_pages 0.0845
5 __get_free_pages 0.0847
8 __free_pages 0.1026
8 balance_dirty_pages_ratelimited 0.0398
1 do_writepages 0.0149
65 __set_page_dirty_nobuffers 0.1729
3 set_page_dirty 0.0357
20 clear_page_dirty_for_io 0.1754
49 test_clear_page_writeback 0.1922
63 test_set_page_writeback 0.2100
29 .text.lock.page_writeback 0.2266
2 pdflush_operation 0.0085
1 prio_tree_insert 0.0026
2 do_page_cache_readahead 0.0043
1 kmem_getpages 0.0048
8 cache_init_objs 0.0769
5 cache_grow 0.0110
156 kmem_cache_alloc 1.2683
82 __kmalloc 0.5325
194 kmem_cache_free 2.2299
201 kfree 1.8440
30 mark_page_accessed 0.5769
1 lru_add_drain 0.0089
27 release_pages 0.0597
27 __pagevec_lru_add 0.0852
1 __pagevec_lru_add_active 0.0030
9 percpu_counter_mod 0.0514
3 blk_queue_bounce 0.0390
9 copy_page_range 0.0116
19 zap_pte_range 0.0274
1 zap_pmd_range 0.0087
1 unmap_vmas 0.0017
15 do_wp_page 0.0177
9 do_anonymous_page 0.0200
293 do_no_page 0.3288
6 handle_mm_fault 0.0137
1 remove_vm_struct 0.0062
1 find_vma_prepare 0.0099
1 vma_link 0.0030
1 vma_adjust 0.0010
1 can_vma_merge_after 0.0090
1 vma_merge 0.0020
1 do_mmap_pgoff 0.0005
2 find_vma 0.0247
1 find_vma_prev 0.0123
1 anon_vma_unlink 0.0064
3 page_add_anon_rmap 0.0092
1 page_add_file_rmap 0.0046
14 page_remove_rmap 0.0586
2 free_pages_and_swap_cache 0.0153
2 dentry_open 0.0051
1 put_unused_fd 0.0081
8 do_sync_read 0.0462
56 vfs_read 0.1898
23 do_sync_write 0.1329
47 vfs_write 0.1593
32 sys_read 0.2500
28 sys_write 0.2188
1 get_empty_filp 0.0043
53 fput 2.5238
78 fget 0.6094
44 fget_light 0.2635
5 bh_waitq_head 0.2273
66 wake_up_buffer 0.7952
27 unlock_buffer 1.1250
14 __set_page_buffers 0.5600
71 end_buffer_write_sync 0.5868
14 __find_get_block_slow 0.0323
38 end_buffer_async_write 0.1230
2 buffer_insert_list 0.0134
23 create_buffers 0.1411
1 init_page_buffers 0.0089
1 __getblk_slow 0.0028
12 mark_buffer_dirty 0.3000
50 __brelse 0.6410
167 __find_get_block 0.6789
25 __getblk 0.2475
9 __bread 0.1364
2 set_bh_page 0.0370
2 try_to_release_page 0.0182
26 create_empty_buffers 0.1287
2 unmap_underlying_metadata 0.0177
44 __block_write_full_page 0.0458
55 __block_prepare_write 0.0554
23 __block_commit_write 0.1544
12 block_prepare_write 0.1558
5 generic_commit_write 0.0338
16 block_write_full_page 0.0599
16 end_bio_bh_io_sync 0.2581
42 submit_bh 0.1288
20 ll_rw_block 0.1471
4 recalc_bh_state 0.0223
16 alloc_buffer_head 0.1524
24 init_buffer_head 0.4528
14 .text.lock.buffer 0.0753
7 bio_destructor 0.0745
22 bio_alloc 0.0516
68 bio_put 1.2593
5 bio_endio 0.0394
14 inode_add_bytes 0.0986
1 flush_old_exec 0.0004
52 pipe_wait 0.2905
258 pipe_readv 0.3649
8 pipe_read 0.1356
152 pipe_writev 0.1834
5 pipe_write 0.0847
131 pipe_poll 1.0565
1 pipe_write_fasync 0.0075
2 vfs_permission 0.0064
1 follow_mount 0.0068
1 do_lookup 0.0062
4 link_path_walk 0.0010
1 path_lookup 0.0021
1 may_open 0.0019
17 kill_fasync 0.1466
5 poll_initwait 0.2000
33 poll_freewait 0.5156
31 __pollwait 0.1574
16 max_select_fd 0.0711
155 do_select 0.1982
2 select_bits_alloc 0.0625
2 select_bits_free 0.4000
86 sys_select 0.0667
1 dput 0.0016
10 __d_lookup 0.0240
137 inode_times_differ 2.0147
29 update_atime 0.1306
51 inode_update_time 0.2383
1 wake_up_inode 0.0139
1 dnotify_flush 0.0047
227 dnotify_parent 0.8972
2 seq_read 0.0029
1 seq_printf 0.0116
82 __mark_inode_dirty 0.1694
14 mpage_writepages 0.0167
1 load_elf_binary 0.0003
1 proc_lookup 0.0030
1 get_vmalloc_info 0.0052
3 show_stat 0.0013
10 write_profile 0.0641
26 ext3_get_group_desc 0.1383
3 read_block_bitmap 0.0197
4 find_next_usable_block 0.0047
63 ext3_try_to_allocate 0.0708
9 ext3_has_free_blocks 0.0732
66 ext3_new_block 0.0508
21 ext3_file_write 0.0986
5 ext3_alloc_block 0.0847
27 ext3_block_to_path 0.0849
39 ext3_get_branch 0.1653
16 ext3_find_goal 0.1168
13 ext3_alloc_branch 0.0188
53 ext3_splice_branch 0.1118
72 ext3_get_block_handle 0.0810
9 ext3_get_block 0.0517
75 walk_page_buffers 0.6696
22 ext3_prepare_write 0.0692
4 ext3_journal_dirty_data 0.0430
93 ext3_ordered_commit_write 0.4061
11 bget_one 1.0000
3 bput_one 0.2727
8 journal_dirty_data_fn 0.4211
14 ext3_ordered_writepage 0.0304
66 ext3_get_inode_block 0.2529
29 ext3_get_inode_loc 0.0456
80 ext3_do_update_inode 0.0867
23 ext3_writepage_trans_blocks 0.1620
30 ext3_mark_iloc_dirty 0.5455
19 ext3_reserve_inode_write 0.0900
13 ext3_mark_inode_dirty 0.1605
22 ext3_dirty_inode 0.1630
19 ext3_journal_start 0.2135
14 __ext3_journal_stop 0.1750
77 start_this_handle 0.0601
21 new_handle 0.2625
35 journal_start 0.1667
204 do_get_write_access 0.1142
28 journal_get_write_access 0.3457
29 journal_get_undo_access 0.0699
69 journal_dirty_data 0.0944
157 journal_dirty_metadata 0.2860
103 journal_stop 0.1265
110 __journal_unfile_buffer 0.2128
53 __journal_file_buffer 0.0756
28 inverted_lock 0.2917
225 journal_commit_transaction 0.0368
4 journal_brelse_array 0.0909
1 __journal_clean_checkpoint_list 0.0057
69 journal_cancel_revoke 0.1988
1 journal_write_metadata_buffer 0.0011
5 __log_space_left 0.0485
1 __log_start_commit 0.0159
6 journal_blocks_per_page 0.2069
9 journal_alloc_journal_head 0.0776
1 journal_free_journal_head 0.0357
264 journal_add_journal_head 0.7213
59 __journal_remove_journal_head 0.1482
52 journal_remove_journal_head 0.4194
104 journal_put_journal_head 0.4664
1 ipc_lock 0.0062
1 shm_open 0.0108
3 kobject_put 0.1200
16 radix_tree_preload 0.0904
6 number 0.0077
7 vsnprintf 0.0057
35 csum_partial 0.1215
22 memcpy 0.3860
1 strncpy_from_user 0.0101
6141 __copy_user_intel 35.7035
4321 __copy_user_zeroing_intel 25.1221
99 __copy_to_user_ll 0.8250
2350 __copy_from_user_ll 19.5833
33 copy_to_user 0.3367
36 copy_from_user 0.2857
1 add_timer_randomness 0.0034
1 add_disk_randomness 0.0192
4 blk_run_queue 0.0323
3 get_request 0.0040
111 __make_request 0.0799
12 generic_make_request 0.0244
13 submit_bio 0.0450
52 __end_that_request_first 0.0989
1 put_io_context 0.0114
1 get_io_context 0.0080
10 .text.lock.ll_rw_blk 0.0735
4 as_set_request 0.0305
337 e1000_xmit_frame 0.1584
7 scsi_put_command 0.0310
21 scsi_dispatch_cmd 0.0390
3 scsi_softirq 0.0144
2 scsi_add_timer 0.0185
1 scsi_decide_disposition 0.0033
3 scsi_init_cmd_errh 0.0156
1 scsi_setup_cmd_retry 0.0093
16 scsi_device_unbusy 0.0870
1 scsi_next_command 0.0244
25 scsi_end_request 0.0947
3 scsi_free_sgtable 0.0667
6 scsi_io_completion 0.0059
162 scsi_request_fn 0.1493
16 .text.lock.scsi_lib 0.0748
54 ahc_linux_isr 0.0777
1 .text.lock.aic7xxx_osm 0.0014
3 sd_rw_intr 0.0045
2 vgacon_save_screen 0.0200
31 sock_aio_read 0.1054
13 sock_aio_write 0.0442
21 sock_poll 0.4286
196 sock_wfree 2.2791
1 sock_wmalloc 0.0106
17 __release_sock 0.1104
368 sock_def_readable 2.0444
10 sk_reset_timer 0.2128
91 lock_sock 0.7339
56 release_sock 0.3684
33 sock_common_recvmsg 0.2821
205 .text.lock.sock 1.6803
232 alloc_skb 1.0357
390 skb_release_data 2.2414
16 kfree_skbmem 0.3636
640 __kfree_skb 2.8070
82 skb_clone 0.2135
25 copy_skb_header 0.1008
19 skb_copy 0.0856
251 skb_copy_bits 0.4772
4 skb_checksum 0.0065
13 memcpy_toiovec 0.1368
123 skb_copy_datagram_iovec 0.2269
38 sk_stream_rfree 1.4074
3 sk_stream_error 0.0323
25 skb_checksum_help 0.0772
70 dev_queue_xmit 0.0879
27 net_tx_action 0.0701
84 netif_receive_skb 0.1888
111 process_backlog 0.4066
66 net_rx_action 0.2548
194 nf_iterate 1.1758
85 nf_hook_slow 0.2972
155 qdisc_restart 0.2627
145 pfifo_fast_enqueue 1.0000
101 pfifo_fast_dequeue 1.2469
26 rt_hash_code 0.1898
278 ip_route_input 0.5110
24 ip_local_deliver 0.0474
142 ip_rcv 0.1111
59 ip_local_deliver_finish 0.1513
43 ip_rcv_finish 0.0664
11 ip_finish_output 0.0207
10 ip_output 0.0833
288 ip_queue_xmit 0.1910
4 dst_output 0.0976
198 ip_finish_output2 0.4470
116 tcp_poll 0.3021
164 tcp_sendmsg 0.0361
41 cleanup_rbuf 0.1331
436 tcp_recvmsg 0.2171
31 __tcp_grow_window 0.2366
64 tcp_rcv_rtt_update 0.7033
47 tcp_rcv_space_adjust 0.1442
175 tcp_event_data_recv 0.2926
22 tcp_rtt_estimator 0.0655
1 tcp_ack_saw_tstamp 0.0119
62 tcp_clean_rtx_queue 0.0652
34 tcp_ack 0.0232
1 tcp_data_queue 0.0003
11 __tcp_data_snd_check 0.0466
41 __tcp_ack_snd_check 0.2547
717 tcp_rcv_established 0.3104
346 tcp_transmit_skb 0.1986
49 tcp_write_xmit 0.0676
31 __tcp_select_window 0.1044
16 tcp_send_delayed_ack 0.0724
77 tcp_send_ack 0.3949
218 __tcp_v4_lookup_established 0.8755
19 tcp_v4_send_check 0.0751
43 tcp_v4_do_rcv 0.1352
452 tcp_v4_rcv 0.2240
5 inet_sendmsg 0.0562
21 ip_confirm 0.4375
25 ip_conntrack_defrag 0.3571
22 ip_refrag 0.1538
16 ip_conntrack_local 0.1096
12 __ip_ct_find_proto 0.1846
250 ip_ct_find_proto 2.3364
78 hash_conntrack 0.5065
31 get_tuple 0.3333
1 invert_tuple 0.0213
59 __ip_conntrack_find 0.2694
134 ip_conntrack_find_get 1.1074
10 ip_conntrack_get 0.2778
138 ip_conntrack_in 0.2244
302 ip_ct_refresh 1.5895
4 ip_ct_gather_frags 0.0172
25 tcp_pkt_to_tuple 0.2941
13 get_conntrack_index 0.2131
441 tcp_packet 0.9323
950 ipt_do_table 0.9804
19 ipt_hook 0.3220
27 ipt_local_out_hook 0.2308
29 ipt_route_hook 0.4915
21 ipt_local_hook 0.0901
67 ip_nat_fn 0.1113
29 ip_nat_out 0.1686
182 do_bindings 0.2233
1 unix_create1 0.0023
2 __sched_text_start 0.0069
5 __write_lock_failed 0.1562
1 __read_lock_failed 0.0500
210 schedule 0.1009
1 io_schedule 0.0208
1 __preempt_spin_lock 0.0110
484 .text.lock.sched 2.4694
4 schedule_timeout 0.0209
459746 total 0.1494
17 default_idle 0.3778
429479 poll_idle 7404.8103
39 cpu_idle 0.6094
4 need_resched 0.0800
122 system_call 2.7727
4 syscall_call 0.3636
10 syscall_exit 0.9091
3 handle_IRQ_event 0.0300
3 old_mmap 0.0090
1 show_cpuinfo 0.0015
1 c_start 0.0278
36 sched_clock 0.2647
3 delay_tsc 0.1429
1 pte_alloc_one 0.0130
32 do_page_fault 0.0229
1 try_to_wake_up 0.0015
1687 finish_task_switch 11.3221
1016 __wake_up 8.6102
189 __might_sleep 0.8957
91 add_wait_queue 0.7339
52 remove_wait_queue 0.4062
40 prepare_to_wait 0.2721
24 finish_wait 0.1519
1 copy_mm 0.0009
1 copy_process 0.0003
4 .text.lock.fork 0.0202
1 profile_exit_mmap 0.0104
1 release_task 0.0018
1 sys_wait4 0.0016
1 sys_time 0.0083
126 current_kernel_time 1.8261
5066 __do_softirq 29.2832
168 local_bh_enable 1.1667
722 __mod_timer 1.3965
80 mod_timer 0.7547
299 del_timer 1.9416
1 del_timer_sync 0.0055
1 do_sigaction 0.0016
1 audit_alloc 0.0049
57 add_to_page_cache 0.2327
17 page_waitqueue 0.3542
31 wake_up_page 0.4026
21 unlock_page 0.5676
20 end_page_writeback 0.2703
55 find_get_page 0.4911
37 find_lock_page 0.1276
43 find_get_pages_tag 0.2704
2 do_generic_mapping_read 0.0016
5 filemap_nopage 0.0053
11 remove_suid 0.0738
111 generic_file_aio_write_nolock 0.0383
22 generic_file_aio_write 0.1392
39 mempool_alloc 0.1024
14 mempool_free 0.0667
1 mempool_alloc_slab 0.0476
24 bad_range 0.2400
21 prep_new_page 0.2593
57 free_hot_cold_page 0.1913
149 buffered_rmqueue 0.2801
61 __alloc_pages 0.0687
9 __get_free_pages 0.1525
21 __free_pages 0.2692
1 __read_page_state 0.0179
11 balance_dirty_pages_ratelimited 0.0547
62 __set_page_dirty_nobuffers 0.1649
16 clear_page_dirty_for_io 0.1404
52 test_clear_page_writeback 0.2039
45 test_set_page_writeback 0.1500
38 .text.lock.page_writeback 0.2969
1 do_page_cache_readahead 0.0022
1 kmem_getpages 0.0048
1 alloc_slabmgmt 0.0104
6 cache_init_objs 0.0577
1 kmem_flagcheck 0.0233
2 cache_grow 0.0044
153 kmem_cache_alloc 1.2439
1 kmem_ptr_validate 0.0132
84 __kmalloc 0.5455
194 kmem_cache_free 2.2299
222 kfree 2.0367
32 mark_page_accessed 0.6154
2 lru_cache_add_active 0.0211
31 release_pages 0.0686
35 __pagevec_lru_add 0.1104
1 __pagevec_lru_add_active 0.0030
16 percpu_counter_mod 0.0914
2 clear_page_tables 0.0065
1 pte_alloc_map 0.0040
12 copy_page_range 0.0155
23 zap_pte_range 0.0331
1 zap_pmd_range 0.0087
1 unmap_page_range 0.0088
19 do_wp_page 0.0224
41 do_anonymous_page 0.0913
311 do_no_page 0.3490
1 remove_vm_struct 0.0062
2 find_vma_prepare 0.0198
1 vma_link 0.0030
1 __insert_vm_struct 0.0076
3 do_mmap_pgoff 0.0016
2 find_vma 0.0247
3 find_vma_prev 0.0370
1 do_munmap 0.0022
1 change_protection 0.0022
2 anon_vma_unlink 0.0128
3 page_add_anon_rmap 0.0092
3 page_add_file_rmap 0.0139
12 page_remove_rmap 0.0502
1 free_pages_and_swap_cache 0.0076
1 sys_access 0.0030
1 dentry_open 0.0025
2 get_unused_fd 0.0078
1 fd_install 0.0068
1 sys_open 0.0060
8 do_sync_read 0.0462
69 vfs_read 0.2339
20 do_sync_write 0.1156
53 vfs_write 0.1797
30 sys_read 0.2344
19 sys_write 0.1484
51 fput 2.4286
73 fget 0.5703
51 fget_light 0.3054
3 bh_waitq_head 0.1364
71 wake_up_buffer 0.8554
37 unlock_buffer 1.5417
10 __set_page_buffers 0.4000
66 end_buffer_write_sync 0.5455
20 __find_get_block_slow 0.0462
34 end_buffer_async_write 0.1100
1 buffer_insert_list 0.0067
40 create_buffers 0.2454
8 mark_buffer_dirty 0.2000
45 __brelse 0.5769
144 __find_get_block 0.5854
18 __getblk 0.1782
13 __bread 0.1970
4 set_bh_page 0.0741
19 create_empty_buffers 0.0941
4 unmap_underlying_metadata 0.0354
37 __block_write_full_page 0.0385
50 __block_prepare_write 0.0504
22 __block_commit_write 0.1477
11 block_prepare_write 0.1429
11 generic_commit_write 0.0743
10 block_write_full_page 0.0375
26 end_bio_bh_io_sync 0.4194
42 submit_bh 0.1288
28 ll_rw_block 0.2059
1 sync_dirty_buffer 0.0055
3 recalc_bh_state 0.0168
12 alloc_buffer_head 0.1143
11 init_buffer_head 0.2075
1 init_buffer 0.0526
3 .text.lock.buffer 0.0161
6 bio_destructor 0.0638
23 bio_alloc 0.0540
50 bio_put 0.9259
1 bio_endio 0.0079
2 generic_fillattr 0.0122
1 sys_stat64 0.0175
15 inode_add_bytes 0.1056
54 pipe_wait 0.3017
219 pipe_readv 0.3098
7 pipe_read 0.1186
127 pipe_writev 0.1532
7 pipe_write 0.1186
107 pipe_poll 0.8629
7 link_path_walk 0.0018
2 path_lookup 0.0042
1 open_namei 0.0006
13 kill_fasync 0.1121
2 poll_initwait 0.0800
27 poll_freewait 0.4219
31 __pollwait 0.1574
19 max_select_fd 0.0844
135 do_select 0.1726
3 select_bits_alloc 0.0938
2 select_bits_free 0.4000
96 sys_select 0.0744
1 flock_to_posix_lock 0.0030
1 fcntl_setlk 0.0015
1 dput 0.0016
1 d_alloc 0.0022
1 d_instantiate 0.0066
9 __d_lookup 0.0216
2 d_rehash 0.0092
1 alloc_inode 0.0030
1 find_inode_fast 0.0123
148 inode_times_differ 2.1765
29 update_atime 0.1306
47 inode_update_time 0.2196
216 dnotify_parent 0.8538
1 lookup_mnt 0.0060
1 seq_printf 0.0116
75 __mark_inode_dirty 0.1550
13 mpage_writepages 0.0155
3 proc_lookup 0.0090
5 show_stat 0.0022
7 write_profile 0.0449
23 ext3_get_group_desc 0.1223
4 read_block_bitmap 0.0263
4 find_next_usable_block 0.0047
58 ext3_try_to_allocate 0.0652
7 ext3_has_free_blocks 0.0569
49 ext3_new_block 0.0377
1 ext3_open_file 0.0222
21 ext3_file_write 0.0986
10 ext3_alloc_block 0.1695
35 ext3_block_to_path 0.1101
34 ext3_get_branch 0.1441
17 ext3_find_goal 0.1241
19 ext3_alloc_branch 0.0275
40 ext3_splice_branch 0.0844
62 ext3_get_block_handle 0.0697
6 ext3_get_block 0.0345
73 walk_page_buffers 0.6518
2 do_journal_get_write_access 0.0165
30 ext3_prepare_write 0.0943
8 ext3_journal_dirty_data 0.0860
94 ext3_ordered_commit_write 0.4105
5 bget_one 0.4545
2 bput_one 0.1818
1 journal_dirty_data_fn 0.0526
6 ext3_ordered_writepage 0.0130
58 ext3_get_inode_block 0.2222
39 ext3_get_inode_loc 0.0613
73 ext3_do_update_inode 0.0791
34 ext3_writepage_trans_blocks 0.2394
39 ext3_mark_iloc_dirty 0.7091
20 ext3_reserve_inode_write 0.0948
18 ext3_mark_inode_dirty 0.2222
15 ext3_dirty_inode 0.1111
23 ext3_journal_start 0.2584
15 __ext3_journal_stop 0.1875
72 start_this_handle 0.0562
28 new_handle 0.3500
34 journal_start 0.1619
175 do_get_write_access 0.0980
21 journal_get_write_access 0.2593
20 journal_get_undo_access 0.0482
79 journal_dirty_data 0.1081
121 journal_dirty_metadata 0.2204
102 journal_stop 0.1253
77 __journal_unfile_buffer 0.1489
41 __journal_file_buffer 0.0585
1 journal_file_buffer 0.0046
24 inverted_lock 0.2500
222 journal_commit_transaction 0.0364
2 journal_brelse_array 0.0455
1 __try_to_free_cp_buf 0.0054
1 __journal_clean_checkpoint_list 0.0057
70 journal_cancel_revoke 0.2017
1 journal_switch_revoke_table 0.0109
8 __log_space_left 0.0777
3 journal_blocks_per_page 0.1034
5 journal_alloc_journal_head 0.0431
3 journal_free_journal_head 0.1071
258 journal_add_journal_head 0.7049
45 __journal_remove_journal_head 0.1131
56 journal_remove_journal_head 0.4516
93 journal_put_journal_head 0.4170
1 ipc_lock 0.0062
1 copy_semundo 0.0042
2 cap_vm_enough_memory 0.0077
15 radix_tree_preload 0.0847
3 number 0.0038
2 vsnprintf 0.0016
1 find_next_zero_bit 0.0057
22 csum_partial 0.0764
2 atomic_dec_and_lock 0.0172
11 memcpy 0.1930
1 strnlen_user 0.0104
6276 __copy_user_intel 36.4884
4430 __copy_user_zeroing_intel 25.7558
92 __copy_to_user_ll 0.7667
2390 __copy_from_user_ll 19.9167
19 copy_to_user 0.1939
38 copy_from_user 0.3016
1 elv_set_request 0.0172
2 blk_run_queue 0.0161
2 get_request 0.0027
89 __make_request 0.0640
17 generic_make_request 0.0346
8 submit_bio 0.0277
53 __end_that_request_first 0.1008
2 get_io_context 0.0160
4 .text.lock.ll_rw_blk 0.0294
1 as_set_request 0.0076
313 e1000_xmit_frame 0.1471
6 scsi_put_command 0.0265
24 scsi_dispatch_cmd 0.0445
1 scsi_softirq 0.0048
1 scsi_add_timer 0.0093
1 scsi_decide_disposition 0.0033
2 scsi_init_cmd_errh 0.0104
3 scsi_device_unbusy 0.0163
3 scsi_run_queue 0.0109
2 scsi_next_command 0.0488
17 scsi_end_request 0.0644
4 scsi_io_completion 0.0039
137 scsi_request_fn 0.1263
14 .text.lock.scsi_lib 0.0654
44 ahc_linux_isr 0.0633
2 sd_rw_intr 0.0030
1 move_addr_to_kernel 0.0139
27 sock_aio_read 0.0918
12 sock_aio_write 0.0408
27 sock_poll 0.5510
187 sock_wfree 2.1744
8 __release_sock 0.0519
368 sock_def_readable 2.0444
8 sk_reset_timer 0.1702
68 lock_sock 0.5484
32 release_sock 0.2105
30 sock_common_recvmsg 0.2564
183 .text.lock.sock 1.5000
246 alloc_skb 1.0982
398 skb_release_data 2.2874
19 kfree_skbmem 0.4318
642 __kfree_skb 2.8158
68 skb_clone 0.1771
30 copy_skb_header 0.1210
21 skb_copy 0.0946
273 skb_copy_bits 0.5190
5 skb_checksum 0.0081
21 memcpy_toiovec 0.2211
2 memcpy_tokerneliovec 0.0267
129 skb_copy_datagram_iovec 0.2380
22 sk_stream_rfree 0.8148
8 sk_stream_error 0.0860
8 skb_checksum_help 0.0247
49 dev_queue_xmit 0.0616
15 net_tx_action 0.0390
72 netif_receive_skb 0.1618
116 process_backlog 0.4249
58 net_rx_action 0.2239
189 nf_iterate 1.1455
100 nf_hook_slow 0.3497
147 qdisc_restart 0.2492
121 pfifo_fast_enqueue 0.8345
114 pfifo_fast_dequeue 1.4074
35 rt_hash_code 0.2555
295 ip_route_input 0.5423
36 ip_local_deliver 0.0711
150 ip_rcv 0.1174
71 ip_local_deliver_finish 0.1821
49 ip_rcv_finish 0.0756
13 ip_finish_output 0.0244
24 ip_output 0.2000
290 ip_queue_xmit 0.1923
6 dst_output 0.1463
209 ip_finish_output2 0.4718
117 tcp_poll 0.3047
187 tcp_sendmsg 0.0411
52 cleanup_rbuf 0.1688
440 tcp_recvmsg 0.2191
28 __tcp_grow_window 0.2137
75 tcp_rcv_rtt_update 0.8242
55 tcp_rcv_space_adjust 0.1687
172 tcp_event_data_recv 0.2876
39 tcp_rtt_estimator 0.1161
4 tcp_ack_saw_tstamp 0.0476
55 tcp_clean_rtx_queue 0.0578
41 tcp_ack 0.0280
1 tcp_ofo_queue 0.0016
3 tcp_data_queue 0.0010
8 __tcp_data_snd_check 0.0339
50 __tcp_ack_snd_check 0.3106
739 tcp_rcv_established 0.3199
339 tcp_transmit_skb 0.1946
32 tcp_write_xmit 0.0441
99 __tcp_select_window 0.3333
17 tcp_send_delayed_ack 0.0769
67 tcp_send_ack 0.3436
212 __tcp_v4_lookup_established 0.8514
23 tcp_v4_send_check 0.0909
44 tcp_v4_do_rcv 0.1384
409 tcp_v4_rcv 0.2027
7 inet_sendmsg 0.0787
20 ip_confirm 0.4167
26 ip_conntrack_defrag 0.3714
22 ip_refrag 0.1538
17 ip_conntrack_local 0.1164
20 __ip_ct_find_proto 0.3077
246 ip_ct_find_proto 2.2991
66 hash_conntrack 0.4286
39 get_tuple 0.4194
1 invert_tuple 0.0213
71 __ip_conntrack_find 0.3242
141 ip_conntrack_find_get 1.1653
4 ip_conntrack_get 0.1111
148 ip_conntrack_in 0.2407
299 ip_ct_refresh 1.5737
4 ip_ct_gather_frags 0.0172
20 tcp_pkt_to_tuple 0.2353
11 get_conntrack_index 0.1803
461 tcp_packet 0.9746
894 ipt_do_table 0.9226
18 ipt_hook 0.3051
20 ipt_local_out_hook 0.1709
30 ipt_route_hook 0.5085
27 ipt_local_hook 0.1159
58 ip_nat_fn 0.0963
35 ip_nat_out 0.2035
179 do_bindings 0.2196
1 unix_release_sock 0.0010
5 __sched_text_start 0.0172
1 __up_wakeup 0.0833
1 __write_lock_failed 0.0312
2 __read_lock_failed 0.1000
217 schedule 0.1043
1 wait_for_completion 0.0039
2 __preempt_spin_lock 0.0220
410 .text.lock.sched 2.0918
3 schedule_timeout 0.0157
472721 total 0.1536
^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: [RFC] Use RCU for tcp_ehash lookup
2004-08-31 12:59 [RFC] Use RCU for tcp_ehash lookup Srivatsa Vaddagiri
2004-08-31 13:04 ` Srivatsa Vaddagiri
2004-08-31 13:12 ` Srivatsa Vaddagiri
@ 2004-08-31 13:54 ` Andi Kleen
2004-09-01 11:36 ` Srivatsa Vaddagiri
2004-09-02 5:43 ` David S. Miller
2004-09-02 5:41 ` David S. Miller
3 siblings, 2 replies; 11+ messages in thread
From: Andi Kleen @ 2004-08-31 13:54 UTC (permalink / raw)
To: Srivatsa Vaddagiri; +Cc: davem, netdev, linux-kernel, Dipankar, paulmck
On Tue, Aug 31, 2004 at 06:29:41PM +0530, Srivatsa Vaddagiri wrote:
>
> I would be interested to know if anyone has seen high-rate of lock contention
> for hash bucket lock. Such workloads would benefit from the lock-free lookup.
I would suspect something that does IO from multiple threads over
a single connection. However there is also the socket lock, which
may prevent too much parallelism.
>
> In the absence of any workload which resulted in lock contention, I resorted
> to disabling NAPI and irq balance (noirqbalance) to study the effect of cache
> bouncing on the lookup routine. The result was that CPU usage of the stack
> was halved in lock-free case, which IMHO, is a strong enough reason for us
> to consider this seriously.
Yes, sounds very nice.
I bet also when you just do rdtsc timing for the TCP receive
path the cycle numbers will be way down (excluding the copy).
And it should also fix the performance problems with
cat /proc/net/tcp on ppc64/ia64 for large hash tables because the rw locks
are gone.
>
> - I presume that one of the reasons for keeping the hash table so big is to
> keep lock contention low (& to reduce the size of hash chains). If the lookup
> is made lock-free, then could the size of the hash table be reduced (without
> adversely impacting performance)?
Definitely worth trying IMHO. The current hash tables are far
too big. I would do that as followon patches though.
I haven't studied it in detail (yet), just two minor style
comments:
> - sk_free(sk);
> +sp_loop:
> + if (atomic_dec_and_test(&sk->sk_refcnt)) {
> + /* Restore ref count and schedule callback.
> + * If we don't restore ref count, then the callback can be
> + * scheduled by more than one CPU.
> + */
> + atomic_inc(&sk->sk_refcnt);
> +
> + if (atomic_read(&sk->sk_refcnt) == 1)
> + call_rcu(&sk->sk_rcu, sk_free_rcu);
> + else
> + goto sp_loop;
> + }
Can you rewrite that without goto?
> +tput_loop:
> if (atomic_dec_and_test(&tw->tw_refcnt)) {
> -#ifdef INET_REFCNT_DEBUG
> - printk(KERN_DEBUG "tw_bucket %p released\n", tw);
> -#endif
> - kmem_cache_free(tcp_timewait_cachep, tw);
> + /* Restore ref count and schedule callback.
> + * If we don't restore ref count, then the callback can be
> + * scheduled by more than one CPU.
> + */
> +
> + atomic_inc(&tw->tw_refcnt);
> +
> + if (atomic_read(&tw->tw_refcnt) == 1)
> + call_rcu(&tw->tw_rcu, tcp_tw_free);
> + else
> + goto tput_loop;
And that too.
-Andi
^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: [RFC] Use RCU for tcp_ehash lookup
2004-08-31 13:54 ` Andi Kleen
@ 2004-09-01 11:36 ` Srivatsa Vaddagiri
2004-09-02 5:45 ` David S. Miller
2004-09-02 21:19 ` Andi Kleen
2004-09-02 5:43 ` David S. Miller
1 sibling, 2 replies; 11+ messages in thread
From: Srivatsa Vaddagiri @ 2004-09-01 11:36 UTC (permalink / raw)
To: Andi Kleen; +Cc: davem, netdev, linux-kernel, Dipankar, paulmck
On Tue, Aug 31, 2004 at 03:54:20PM +0200, Andi Kleen wrote:
> I bet also when you just do rdtsc timing for the TCP receive
> path the cycle numbers will be way down (excluding the copy).
I got cycle numbers for the lookup routine (with CONFIG_PREEMPT turned off).
They were taken on a 900MHz 8way Intel P3 SMP box. The results are as below:
-------------------------------------------------------------------------------
| 2.6.8.1 | 2.6.8.1 + my patch
-------------------------------------------------------------------------------
Average cycles | |
spent in | |
__tcp_v4_lookup_established | 2970.65 | 668.227
| (~3.3 micro-seconds) | (~0.74 microseconds)
-------------------------------------------------------------------------------
This repesents improvement by a factor of 77.5%!
>
> And it should also fix the performance problems with
> cat /proc/net/tcp on ppc64/ia64 for large hash tables because the rw locks
> are gone.
But spinlocks are in! Would that still improve the performance compared to rw
locks? (See me earlier note where I have explained that lookup done for
/proc/net/tcp is _not_ lock-free yet).
> I haven't studied it in detail (yet), just two minor style
> comments:
[snip]
> Can you rewrite that without goto?
[snip]
> And that too.
I have avoided the goto's in the updated patch below.
Thanks!!
---
linux-2.6.8.1-vatsa/include/net/sock.h | 22 +++++++++--
linux-2.6.8.1-vatsa/include/net/tcp.h | 24 +++++++++---
linux-2.6.8.1-vatsa/net/core/sock.c | 11 +++++
linux-2.6.8.1-vatsa/net/ipv4/tcp.c | 2 -
linux-2.6.8.1-vatsa/net/ipv4/tcp_diag.c | 11 +++--
linux-2.6.8.1-vatsa/net/ipv4/tcp_ipv4.c | 50 ++++++++++++++++-----------
linux-2.6.8.1-vatsa/net/ipv4/tcp_minisocks.c | 47 ++++++++++++++++++++-----
linux-2.6.8.1-vatsa/net/ipv6/tcp_ipv6.c | 22 +++++++----
8 files changed, 135 insertions(+), 54 deletions(-)
diff -puN include/net/sock.h~tcp_ehash include/net/sock.h
--- linux-2.6.8.1/include/net/sock.h~tcp_ehash 2004-08-25 18:06:42.000000000 +0530
+++ linux-2.6.8.1-vatsa/include/net/sock.h 2004-09-01 10:09:43.000000000 +0530
@@ -50,6 +50,7 @@
#include <linux/security.h>
#include <linux/filter.h>
+#include <linux/rcupdate.h>
#include <asm/atomic.h>
#include <net/dst.h>
@@ -178,6 +179,7 @@ struct sock_common {
* @sk_error_report - callback to indicate errors (e.g. %MSG_ERRQUEUE)
* @sk_backlog_rcv - callback to process the backlog
* @sk_destruct - called at sock freeing time, i.e. when all refcnt == 0
+ * @sk_rcu - RCU callback structure
*/
struct sock {
/*
@@ -266,6 +268,7 @@ struct sock {
int (*sk_backlog_rcv)(struct sock *sk,
struct sk_buff *skb);
void (*sk_destruct)(struct sock *sk);
+ struct rcu_head sk_rcu;
};
/*
@@ -350,7 +353,7 @@ static __inline__ int sk_del_node_init(s
static __inline__ void __sk_add_node(struct sock *sk, struct hlist_head *list)
{
- hlist_add_head(&sk->sk_node, list);
+ hlist_add_head_rcu(&sk->sk_node, list);
}
static __inline__ void sk_add_node(struct sock *sk, struct hlist_head *list)
@@ -371,7 +374,7 @@ static __inline__ void sk_add_bind_node(
}
#define sk_for_each(__sk, node, list) \
- hlist_for_each_entry(__sk, node, list, sk_node)
+ hlist_for_each_entry_rcu(__sk, node, list, sk_node)
#define sk_for_each_from(__sk, node) \
if (__sk && ({ node = &(__sk)->sk_node; 1; })) \
hlist_for_each_entry_from(__sk, node, sk_node)
@@ -703,6 +706,7 @@ extern void FASTCALL(release_sock(struct
extern struct sock * sk_alloc(int family, int priority, int zero_it,
kmem_cache_t *slab);
extern void sk_free(struct sock *sk);
+extern void sk_free_rcu(struct rcu_head *head);
extern struct sk_buff *sock_wmalloc(struct sock *sk,
unsigned long size, int force,
@@ -888,8 +892,18 @@ static inline void sk_filter_charge(stru
/* Ungrab socket and destroy it, if it was the last reference. */
static inline void sock_put(struct sock *sk)
{
- if (atomic_dec_and_test(&sk->sk_refcnt))
- sk_free(sk);
+ while (atomic_dec_and_test(&sk->sk_refcnt)) {
+ /* Restore ref count and schedule callback.
+ * If we don't restore ref count, then the callback can be
+ * scheduled by more than one CPU.
+ */
+ atomic_inc(&sk->sk_refcnt);
+
+ if (atomic_read(&sk->sk_refcnt) == 1) {
+ call_rcu(&sk->sk_rcu, sk_free_rcu);
+ break;
+ }
+ }
}
/* Detach socket from process context.
diff -puN include/net/tcp.h~tcp_ehash include/net/tcp.h
--- linux-2.6.8.1/include/net/tcp.h~tcp_ehash 2004-08-25 18:06:42.000000000 +0530
+++ linux-2.6.8.1-vatsa/include/net/tcp.h 2004-09-01 10:13:40.000000000 +0530
@@ -30,6 +30,7 @@
#include <linux/slab.h>
#include <linux/cache.h>
#include <linux/percpu.h>
+#include <linux/rcupdate.h>
#include <net/checksum.h>
#include <net/sock.h>
#include <net/snmp.h>
@@ -44,7 +45,7 @@
* for the rest. I'll experiment with dynamic table growth later.
*/
struct tcp_ehash_bucket {
- rwlock_t lock;
+ spinlock_t lock;
struct hlist_head chain;
} __attribute__((__aligned__(8)));
@@ -222,12 +223,13 @@ struct tcp_tw_bucket {
struct in6_addr tw_v6_rcv_saddr;
int tw_v6_ipv6only;
#endif
+ struct rcu_head tw_rcu;
};
static __inline__ void tw_add_node(struct tcp_tw_bucket *tw,
struct hlist_head *list)
{
- hlist_add_head(&tw->tw_node, list);
+ hlist_add_head_rcu(&tw->tw_node, list);
}
static __inline__ void tw_add_bind_node(struct tcp_tw_bucket *tw,
@@ -305,14 +307,22 @@ static inline int tcp_v6_ipv6only(const
#endif
extern kmem_cache_t *tcp_timewait_cachep;
+extern void tcp_tw_free(struct rcu_head *head);
static inline void tcp_tw_put(struct tcp_tw_bucket *tw)
{
- if (atomic_dec_and_test(&tw->tw_refcnt)) {
-#ifdef INET_REFCNT_DEBUG
- printk(KERN_DEBUG "tw_bucket %p released\n", tw);
-#endif
- kmem_cache_free(tcp_timewait_cachep, tw);
+ while (atomic_dec_and_test(&tw->tw_refcnt)) {
+ /* Restore ref count and schedule callback.
+ * If we don't restore ref count, then the callback can be
+ * scheduled by more than one CPU.
+ */
+
+ atomic_inc(&tw->tw_refcnt);
+
+ if (atomic_read(&tw->tw_refcnt) == 1) {
+ call_rcu(&tw->tw_rcu, tcp_tw_free);
+ break;
+ }
}
}
diff -puN net/core/sock.c~tcp_ehash net/core/sock.c
--- linux-2.6.8.1/net/core/sock.c~tcp_ehash 2004-08-25 18:06:42.000000000 +0530
+++ linux-2.6.8.1-vatsa/net/core/sock.c 2004-08-26 16:53:14.000000000 +0530
@@ -657,6 +657,16 @@ void sk_free(struct sock *sk)
module_put(owner);
}
+/* RCU callback to free a socket */
+
+void sk_free_rcu(struct rcu_head *head)
+{
+ struct sock *sk = container_of(head, struct sock, sk_rcu);
+
+ if (atomic_dec_and_test(&sk->sk_refcnt))
+ sk_free(sk);
+}
+
void __init sk_init(void)
{
sk_cachep = kmem_cache_create("sock", sizeof(struct sock), 0,
@@ -1347,6 +1357,7 @@ EXPORT_SYMBOL(__lock_sock);
EXPORT_SYMBOL(__release_sock);
EXPORT_SYMBOL(sk_alloc);
EXPORT_SYMBOL(sk_free);
+EXPORT_SYMBOL(sk_free_rcu);
EXPORT_SYMBOL(sk_send_sigurg);
EXPORT_SYMBOL(sock_alloc_send_pskb);
EXPORT_SYMBOL(sock_alloc_send_skb);
diff -puN net/ipv4/tcp_ipv4.c~tcp_ehash net/ipv4/tcp_ipv4.c
--- linux-2.6.8.1/net/ipv4/tcp_ipv4.c~tcp_ehash 2004-08-25 18:06:42.000000000 +0530
+++ linux-2.6.8.1-vatsa/net/ipv4/tcp_ipv4.c 2004-08-25 18:07:27.000000000 +0530
@@ -351,7 +351,8 @@ void tcp_listen_wlock(void)
static __inline__ void __tcp_v4_hash(struct sock *sk, const int listen_possible)
{
struct hlist_head *list;
- rwlock_t *lock;
+ rwlock_t *lock = NULL;
+ spinlock_t *slock = NULL;
BUG_TRAP(sk_unhashed(sk));
if (listen_possible && sk->sk_state == TCP_LISTEN) {
@@ -360,14 +361,16 @@ static __inline__ void __tcp_v4_hash(str
tcp_listen_wlock();
} else {
list = &tcp_ehash[(sk->sk_hashent = tcp_sk_hashfn(sk))].chain;
- lock = &tcp_ehash[sk->sk_hashent].lock;
- write_lock(lock);
+ slock = &tcp_ehash[sk->sk_hashent].lock;
+ spin_lock(slock);
}
__sk_add_node(sk, list);
sock_prot_inc_use(sk->sk_prot);
- write_unlock(lock);
- if (listen_possible && sk->sk_state == TCP_LISTEN)
+ if (listen_possible && sk->sk_state == TCP_LISTEN) {
+ write_unlock(lock);
wake_up(&tcp_lhash_wait);
+ } else
+ spin_unlock(slock);
}
static void tcp_v4_hash(struct sock *sk)
@@ -381,7 +384,8 @@ static void tcp_v4_hash(struct sock *sk)
void tcp_unhash(struct sock *sk)
{
- rwlock_t *lock;
+ rwlock_t *lock = NULL;
+ spinlock_t *slock = NULL;
if (sk_unhashed(sk))
goto ende;
@@ -392,17 +396,20 @@ void tcp_unhash(struct sock *sk)
lock = &tcp_lhash_lock;
} else {
struct tcp_ehash_bucket *head = &tcp_ehash[sk->sk_hashent];
- lock = &head->lock;
- write_lock_bh(&head->lock);
+ slock = &head->lock;
+ spin_lock_bh(&head->lock);
}
if (__sk_del_node_init(sk))
sock_prot_dec_use(sk->sk_prot);
- write_unlock_bh(lock);
+ if (sk->sk_state != TCP_LISTEN)
+ spin_unlock_bh(slock);
+ else {
+ write_unlock_bh(lock);
ende:
- if (sk->sk_state == TCP_LISTEN)
wake_up(&tcp_lhash_wait);
+ }
}
/* Don't inline this cruft. Here are some nice properties to
@@ -494,7 +501,7 @@ static inline struct sock *__tcp_v4_look
*/
int hash = tcp_hashfn(daddr, hnum, saddr, sport);
head = &tcp_ehash[hash];
- read_lock(&head->lock);
+ rcu_read_lock();
sk_for_each(sk, node, &head->chain) {
if (TCP_IPV4_MATCH(sk, acookie, saddr, daddr, ports, dif))
goto hit; /* You sunk my battleship! */
@@ -507,7 +514,7 @@ static inline struct sock *__tcp_v4_look
}
sk = NULL;
out:
- read_unlock(&head->lock);
+ rcu_read_unlock();
return sk;
hit:
sock_hold(sk);
@@ -559,7 +566,7 @@ static int __tcp_v4_check_established(st
struct hlist_node *node;
struct tcp_tw_bucket *tw;
- write_lock(&head->lock);
+ spin_lock(&head->lock);
/* Check TIME-WAIT sockets first. */
sk_for_each(sk2, node, &(head + tcp_ehash_size)->chain) {
@@ -614,7 +621,7 @@ unique:
BUG_TRAP(sk_unhashed(sk));
__sk_add_node(sk, &head->chain);
sock_prot_inc_use(sk->sk_prot);
- write_unlock(&head->lock);
+ spin_unlock(&head->lock);
if (twp) {
*twp = tw;
@@ -630,7 +637,7 @@ unique:
return 0;
not_unique:
- write_unlock(&head->lock);
+ spin_unlock(&head->lock);
return -EADDRNOTAVAIL;
}
@@ -2228,7 +2235,10 @@ static void *established_get_first(struc
struct hlist_node *node;
struct tcp_tw_bucket *tw;
- read_lock(&tcp_ehash[st->bucket].lock);
+ /* Take the spinlock. Otherwise a dancing socket
+ * (__tcp_tw_hashdance) may be reported twice!
+ */
+ spin_lock(&tcp_ehash[st->bucket].lock);
sk_for_each(sk, node, &tcp_ehash[st->bucket].chain) {
if (sk->sk_family != st->family) {
continue;
@@ -2245,7 +2255,7 @@ static void *established_get_first(struc
rc = tw;
goto out;
}
- read_unlock(&tcp_ehash[st->bucket].lock);
+ spin_unlock(&tcp_ehash[st->bucket].lock);
st->state = TCP_SEQ_STATE_ESTABLISHED;
}
out:
@@ -2272,10 +2282,10 @@ get_tw:
cur = tw;
goto out;
}
- read_unlock(&tcp_ehash[st->bucket].lock);
+ spin_unlock(&tcp_ehash[st->bucket].lock);
st->state = TCP_SEQ_STATE_ESTABLISHED;
if (++st->bucket < tcp_ehash_size) {
- read_lock(&tcp_ehash[st->bucket].lock);
+ spin_lock(&tcp_ehash[st->bucket].lock);
sk = sk_head(&tcp_ehash[st->bucket].chain);
} else {
cur = NULL;
@@ -2385,7 +2395,7 @@ static void tcp_seq_stop(struct seq_file
case TCP_SEQ_STATE_TIME_WAIT:
case TCP_SEQ_STATE_ESTABLISHED:
if (v)
- read_unlock(&tcp_ehash[st->bucket].lock);
+ spin_unlock(&tcp_ehash[st->bucket].lock);
local_bh_enable();
break;
}
diff -puN net/ipv4/tcp.c~tcp_ehash net/ipv4/tcp.c
--- linux-2.6.8.1/net/ipv4/tcp.c~tcp_ehash 2004-08-25 18:06:42.000000000 +0530
+++ linux-2.6.8.1-vatsa/net/ipv4/tcp.c 2004-08-25 18:07:27.000000000 +0530
@@ -2258,7 +2258,7 @@ void __init tcp_init(void)
if (!tcp_ehash)
panic("Failed to allocate TCP established hash table\n");
for (i = 0; i < (tcp_ehash_size << 1); i++) {
- tcp_ehash[i].lock = RW_LOCK_UNLOCKED;
+ tcp_ehash[i].lock = SPIN_LOCK_UNLOCKED;
INIT_HLIST_HEAD(&tcp_ehash[i].chain);
}
diff -puN net/ipv4/tcp_diag.c~tcp_ehash net/ipv4/tcp_diag.c
--- linux-2.6.8.1/net/ipv4/tcp_diag.c~tcp_ehash 2004-08-25 18:06:42.000000000 +0530
+++ linux-2.6.8.1-vatsa/net/ipv4/tcp_diag.c 2004-08-25 18:07:27.000000000 +0530
@@ -522,7 +522,10 @@ skip_listen_ht:
if (i > s_i)
s_num = 0;
- read_lock_bh(&head->lock);
+ /* Take the spinlock. Otherwise a dancing socket
+ * (__tcp_tw_hashdance) may be reported twice!
+ */
+ spin_lock_bh(&head->lock);
num = 0;
sk_for_each(sk, node, &head->chain) {
@@ -542,7 +545,7 @@ skip_listen_ht:
if (tcpdiag_fill(skb, sk, r->tcpdiag_ext,
NETLINK_CB(cb->skb).pid,
cb->nlh->nlmsg_seq) <= 0) {
- read_unlock_bh(&head->lock);
+ spin_unlock_bh(&head->lock);
goto done;
}
++num;
@@ -568,13 +571,13 @@ skip_listen_ht:
if (tcpdiag_fill(skb, sk, r->tcpdiag_ext,
NETLINK_CB(cb->skb).pid,
cb->nlh->nlmsg_seq) <= 0) {
- read_unlock_bh(&head->lock);
+ spin_unlock_bh(&head->lock);
goto done;
}
++num;
}
}
- read_unlock_bh(&head->lock);
+ spin_unlock_bh(&head->lock);
}
done:
diff -puN net/ipv4/tcp_minisocks.c~tcp_ehash net/ipv4/tcp_minisocks.c
--- linux-2.6.8.1/net/ipv4/tcp_minisocks.c~tcp_ehash 2004-08-25 18:06:42.000000000 +0530
+++ linux-2.6.8.1-vatsa/net/ipv4/tcp_minisocks.c 2004-08-26 16:58:08.000000000 +0530
@@ -64,14 +64,14 @@ static void tcp_timewait_kill(struct tcp
/* Unlink from established hashes. */
ehead = &tcp_ehash[tw->tw_hashent];
- write_lock(&ehead->lock);
+ spin_lock(&ehead->lock);
if (hlist_unhashed(&tw->tw_node)) {
- write_unlock(&ehead->lock);
+ spin_unlock(&ehead->lock);
return;
}
__hlist_del(&tw->tw_node);
sk_node_init(&tw->tw_node);
- write_unlock(&ehead->lock);
+ spin_unlock(&ehead->lock);
/* Disassociate with bind bucket. */
bhead = &tcp_bhash[tcp_bhashfn(tw->tw_num)];
@@ -308,17 +308,28 @@ static void __tcp_tw_hashdance(struct so
tw_add_bind_node(tw, &tw->tw_tb->owners);
spin_unlock(&bhead->lock);
- write_lock(&ehead->lock);
+ spin_lock(&ehead->lock);
- /* Step 2: Remove SK from established hash. */
- if (__sk_del_node_init(sk))
- sock_prot_dec_use(sk->sk_prot);
+ /*
+ * We have to be carefull here since there could be racing
+ * (lock-free) lookups happening on other CPUs. If we remove SK first
+ * and then add TW, then there is a tiny window where this socket is
+ * in neither the established half nor in the TIMEWAIT half of the ehash
+ * table. Lookups occuring in that window can drop packets!
+ * Hence we first add TW and then remove SK, with a barrier in between.
+ */
- /* Step 3: Hash TW into TIMEWAIT half of established hash table. */
+ /* Step 2: Hash TW into TIMEWAIT half of established hash table. */
tw_add_node(tw, &(ehead + tcp_ehash_size)->chain);
atomic_inc(&tw->tw_refcnt);
- write_unlock(&ehead->lock);
+ smp_wmb();
+
+ /* Step 3: Remove SK from established hash. */
+ if (__sk_del_node_init(sk))
+ sock_prot_dec_use(sk->sk_prot);
+
+ spin_unlock(&ehead->lock);
}
/*
@@ -1069,11 +1080,29 @@ int tcp_child_process(struct sock *paren
return ret;
}
+/* RCU callback to free a timewait bucket */
+
+void tcp_tw_free(struct rcu_head *head)
+{
+ struct tcp_tw_bucket *tw =
+ container_of(head, struct tcp_tw_bucket, tw_rcu);
+
+ if (atomic_dec_and_test(&tw->tw_refcnt)) {
+#ifdef INET_REFCNT_DEBUG
+ printk(KERN_DEBUG "tw_bucket %p released\n", tw);
+#endif
+ kmem_cache_free(tcp_timewait_cachep, tw);
+ }
+}
+
+
+
EXPORT_SYMBOL(tcp_check_req);
EXPORT_SYMBOL(tcp_child_process);
EXPORT_SYMBOL(tcp_create_openreq_child);
EXPORT_SYMBOL(tcp_timewait_state_process);
EXPORT_SYMBOL(tcp_tw_deschedule);
+EXPORT_SYMBOL(tcp_tw_free);
#ifdef CONFIG_SYSCTL
EXPORT_SYMBOL(sysctl_tcp_tw_recycle);
diff -puN net/ipv6/tcp_ipv6.c~tcp_ehash net/ipv6/tcp_ipv6.c
--- linux-2.6.8.1/net/ipv6/tcp_ipv6.c~tcp_ehash 2004-08-25 18:06:42.000000000 +0530
+++ linux-2.6.8.1-vatsa/net/ipv6/tcp_ipv6.c 2004-08-25 18:07:27.000000000 +0530
@@ -210,7 +210,8 @@ fail:
static __inline__ void __tcp_v6_hash(struct sock *sk)
{
struct hlist_head *list;
- rwlock_t *lock;
+ rwlock_t *lock = NULL;
+ spinlock_t *slock = NULL;
BUG_TRAP(sk_unhashed(sk));
@@ -221,13 +222,16 @@ static __inline__ void __tcp_v6_hash(str
} else {
sk->sk_hashent = tcp_v6_sk_hashfn(sk);
list = &tcp_ehash[sk->sk_hashent].chain;
- lock = &tcp_ehash[sk->sk_hashent].lock;
- write_lock(lock);
+ slock = &tcp_ehash[sk->sk_hashent].lock;
+ spin_lock(slock);
}
__sk_add_node(sk, list);
sock_prot_inc_use(sk->sk_prot);
- write_unlock(lock);
+ if (sk->sk_state == TCP_LISTEN)
+ write_unlock(lock);
+ else
+ spin_unlock(slock);
}
@@ -307,7 +311,7 @@ static inline struct sock *__tcp_v6_look
*/
hash = tcp_v6_hashfn(daddr, hnum, saddr, sport);
head = &tcp_ehash[hash];
- read_lock(&head->lock);
+ rcu_read_lock();
sk_for_each(sk, node, &head->chain) {
/* For IPV6 do the cheaper port and family tests first. */
if(TCP_IPV6_MATCH(sk, saddr, daddr, ports, dif))
@@ -326,12 +330,12 @@ static inline struct sock *__tcp_v6_look
goto hit;
}
}
- read_unlock(&head->lock);
+ rcu_read_unlock();
return NULL;
hit:
sock_hold(sk);
- read_unlock(&head->lock);
+ rcu_read_unlock();
return sk;
}
@@ -452,7 +456,7 @@ static int tcp_v6_check_established(stru
struct hlist_node *node;
struct tcp_tw_bucket *tw;
- write_lock_bh(&head->lock);
+ spin_lock_bh(&head->lock);
/* Check TIME-WAIT sockets first. */
sk_for_each(sk2, node, &(head + tcp_ehash_size)->chain) {
@@ -491,7 +495,7 @@ unique:
__sk_add_node(sk, &head->chain);
sk->sk_hashent = hash;
sock_prot_inc_use(sk->sk_prot);
- write_unlock_bh(&head->lock);
+ spin_unlock_bh(&head->lock);
if (tw) {
/* Silly. Should hash-dance instead... */
_
--
Thanks and Regards,
Srivatsa Vaddagiri,
Linux Technology Center,
IBM Software Labs,
Bangalore, INDIA - 560017
^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: [RFC] Use RCU for tcp_ehash lookup
2004-08-31 12:59 [RFC] Use RCU for tcp_ehash lookup Srivatsa Vaddagiri
` (2 preceding siblings ...)
2004-08-31 13:54 ` Andi Kleen
@ 2004-09-02 5:41 ` David S. Miller
2004-09-02 14:04 ` Srivatsa Vaddagiri
2004-09-02 16:31 ` Paul E. McKenney
3 siblings, 2 replies; 11+ messages in thread
From: David S. Miller @ 2004-09-02 5:41 UTC (permalink / raw)
To: vatsa; +Cc: netdev, linux-kernel, dipankar, paulmck
On Tue, 31 Aug 2004 18:29:41 +0530
Srivatsa Vaddagiri <vatsa@in.ibm.com> wrote:
> Some notes on the patch:
>
> - Although readprofile shows improvement in tick count for
> __tcp_v4_lookup_established, I haven't come across any benchmarks that is
> benefited noticeably by the lock-free lookup. I have tried httperf, netperf
> and simple file transfer tests so far.
>
> This could possibly be because the hash table size on the machines I was
> testing was high (tcp_ehash_size = 128K), leading to low contention rate on
> the hash bucket locks. Also because of the fact that lookup could happen in
> parallel to socket input packet processing.
>
> I would be interested to know if anyone has seen high-rate of lock contention
> for hash bucket lock. Such workloads would benefit from the lock-free lookup.
The reason you don't see any improvement is that the ehash table is
pretty write heavy.
I'm not totally against your patch, I just don't think that the TCP established
hash table qualifies as "read heavy" as per what RCU is truly effective for.
> - I presume that one of the reasons for keeping the hash table so big is to
> keep lock contention low (& to reduce the size of hash chains). If the lookup
> is made lock-free, then could the size of the hash table be reduced (without
> adversely impacting performance)?
It's large so that the hash itself is effective, not for locking reasons.
> - Biggest problem I had converting over to RCU was the refcount race between
> sock_put and sock_hold. sock_put might see the refcount go to zero and decide
> to free the object, while on some other CPU, sock_get's are pending against
> the same object. The patch handles the race by deciding to free the object
> only from the RCU callback.
That's exactly what I was concerned about when I saw that you had attempted
this change. It is incredibly important for state changes and updates to
be seen as atomic by the packet input processing engine. It would be illegal
for a cpu running TCP input to see a socket in two tables at the same time
(for example, in the main established area and in the second half for TIME_WAIT
buckets).
If the visibility of the socket is wrong, sockets could be erroneously
be reset during the transition from established to TIME_WAIT state.
Beware!
> - Socket table lookups that happens thr', say /proc/net/tcp or tcpdiag_dump, is
> not lock-free yet. This is because of movement of socket performed in
> __tcp_tw_hashdance, between established half to time-wait half.
> There is a window during this movement, when the same socket is present
> on both time-wait half as well as established half. I felt that it is not
> good to have /proc/net/tcp report two instances of the same socket. Hence
> I resorted to have /proc/net/tcp and tcpdiag_dump doing the lookup using
> a spinlock.
/proc/net/tcp should simply not be used by people, we
have the netlink interface to get socket listings which
actually scales.
Leaving /proc/net/tcp readable on servers with real users is
a DoS waiting to happen.
> Note that __tcp_v4_lookup_established should not be affected by the above
> movement because I found it scans the established half first and _then_ the
> time wait half. So even if the same socket is present in both established half
> and time wait half, __tcp_v4_lookup_established will lookup only one of them
> (& not both).
I hope this is true.
^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: [RFC] Use RCU for tcp_ehash lookup
2004-08-31 13:54 ` Andi Kleen
2004-09-01 11:36 ` Srivatsa Vaddagiri
@ 2004-09-02 5:43 ` David S. Miller
1 sibling, 0 replies; 11+ messages in thread
From: David S. Miller @ 2004-09-02 5:43 UTC (permalink / raw)
To: Andi Kleen; +Cc: vatsa, davem, netdev, linux-kernel, dipankar, paulmck
On Tue, 31 Aug 2004 15:54:20 +0200
Andi Kleen <ak@suse.de> wrote:
> And it should also fix the performance problems with
> cat /proc/net/tcp on ppc64/ia64 for large hash tables because the rw locks
> are gone.
Time to convert netstat et al. over the netlink too.
> > - I presume that one of the reasons for keeping the hash table so big is to
> > keep lock contention low (& to reduce the size of hash chains). If the lookup
> > is made lock-free, then could the size of the hash table be reduced (without
> > adversely impacting performance)?
>
> Definitely worth trying IMHO. The current hash tables are far
> too big. I would do that as followon patches though.
The hashes are big to make the hash effective, not to help the locking
contention.
^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: [RFC] Use RCU for tcp_ehash lookup
2004-09-01 11:36 ` Srivatsa Vaddagiri
@ 2004-09-02 5:45 ` David S. Miller
2004-09-02 21:19 ` Andi Kleen
1 sibling, 0 replies; 11+ messages in thread
From: David S. Miller @ 2004-09-02 5:45 UTC (permalink / raw)
To: vatsa; +Cc: ak, davem, netdev, linux-kernel, dipankar, paulmck
On Wed, 1 Sep 2004 17:06:41 +0530
Srivatsa Vaddagiri <vatsa@in.ibm.com> wrote:
> On Tue, Aug 31, 2004 at 03:54:20PM +0200, Andi Kleen wrote:
> > I bet also when you just do rdtsc timing for the TCP receive
> > path the cycle numbers will be way down (excluding the copy).
>
> I got cycle numbers for the lookup routine (with CONFIG_PREEMPT turned off).
> They were taken on a 900MHz 8way Intel P3 SMP box. The results are as below:
>
>
> -------------------------------------------------------------------------------
> | 2.6.8.1 | 2.6.8.1 + my patch
> -------------------------------------------------------------------------------
> Average cycles | |
> spent in | |
> __tcp_v4_lookup_established | 2970.65 | 668.227
> | (~3.3 micro-seconds) | (~0.74 microseconds)
> -------------------------------------------------------------------------------
>
> This repesents improvement by a factor of 77.5%!
And yet none of your benchmarks show noticable
improvements, which means that this micro-measurement
is totally unimportant in the grand scheme of things
as far as we know.
I'm not adding in a patch that merely provides some
micro-measurement improvement that someone can do a
shamans dance over. :) If we're going to add this
new level of complexity to the TCP code we need to
see some real usage performance improvement, not just
something that shows up when we put a microscope on
a single function.
^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: [RFC] Use RCU for tcp_ehash lookup
2004-09-02 5:41 ` David S. Miller
@ 2004-09-02 14:04 ` Srivatsa Vaddagiri
2004-09-02 16:31 ` Paul E. McKenney
1 sibling, 0 replies; 11+ messages in thread
From: Srivatsa Vaddagiri @ 2004-09-02 14:04 UTC (permalink / raw)
To: David S. Miller; +Cc: netdev, linux-kernel, dipankar, paulmck
On Wed, Sep 01, 2004 at 10:41:08PM -0700, David S. Miller wrote:
> The reason you don't see any improvement is that the ehash table is
> pretty write heavy.
In my simple one-file-transfer-test-at-a-time, it should have been read-mostly.
Probably the fact lookups are not serialized wrt input pakcet processing
may have shadowed the benefits of lock-free lookup. However perhaps
if I have multiple file transfer sessions in progress (one per cpu maybe),
then the benefit of reduced time spent in looking up a socket, could be passed
on to threads doing network input.
> I'm not totally against your patch, I just don't think that the TCP established
> hash table qualifies as "read heavy" as per what RCU is truly effective for.
IMHO the benefits of lock-free will be seen only in such scenarios, i.e where
read_lock ended up having to spin-wait on a update to finish. In the lock-free
case, there is no such wait.
> That's exactly what I was concerned about when I saw that you had attempted
> this change. It is incredibly important for state changes and updates to
> be seen as atomic by the packet input processing engine. It would be illegal
> for a cpu running TCP input to see a socket in two tables at the same time
> (for example, in the main established area and in the second half for TIME_WAIT
> buckets).
>
> If the visibility of the socket is wrong, sockets could be erroneously
> be reset during the transition from established to TIME_WAIT state.
> Beware!
This is precisely the reason why I changed the order of movement in
__tcp_tw_hashdance. Earlier, it was removing the socket from the
established half and _then_ adding it to time-wait half. This would
have lead to a window where the socket is neither in established-half
not in the time-wait half. A packet arriving in this window (& doing
lock-free lookup) would have been dropped.
Hence I reversed the order of movement to add in time-wait first
before removing from established half.
> > Note that __tcp_v4_lookup_established should not be affected by the above
> > movement because I found it scans the established half first and _then_ the
> > time wait half. So even if the same socket is present in both established half
> > and time wait half, __tcp_v4_lookup_established will lookup only one of them
> > (& not both).
>
> I hope this is true.
AFAICS it is true! If __tcp_v4_lookup_established finds it in the established
half, it does no further lookup in the time-wait half.
--
Thanks and Regards,
Srivatsa Vaddagiri,
Linux Technology Center,
IBM Software Labs,
Bangalore, INDIA - 560017
^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: [RFC] Use RCU for tcp_ehash lookup
2004-09-02 5:41 ` David S. Miller
2004-09-02 14:04 ` Srivatsa Vaddagiri
@ 2004-09-02 16:31 ` Paul E. McKenney
1 sibling, 0 replies; 11+ messages in thread
From: Paul E. McKenney @ 2004-09-02 16:31 UTC (permalink / raw)
To: David S. Miller; +Cc: vatsa, netdev, linux-kernel, dipankar
On Wed, Sep 01, 2004 at 10:41:08PM -0700, David S. Miller wrote:
> On Tue, 31 Aug 2004 18:29:41 +0530
> Srivatsa Vaddagiri <vatsa@in.ibm.com> wrote:
> > - Biggest problem I had converting over to RCU was the refcount race between
> > sock_put and sock_hold. sock_put might see the refcount go to zero and decide
> > to free the object, while on some other CPU, sock_get's are pending against
> > the same object. The patch handles the race by deciding to free the object
> > only from the RCU callback.
>
> That's exactly what I was concerned about when I saw that you had attempted
> this change. It is incredibly important for state changes and updates to
> be seen as atomic by the packet input processing engine. It would be illegal
> for a cpu running TCP input to see a socket in two tables at the same time
> (for example, in the main established area and in the second half for TIME_WAIT
> buckets).
>
> If the visibility of the socket is wrong, sockets could be erroneously
> be reset during the transition from established to TIME_WAIT state.
> Beware!
If the usages is too write-intensive, then RCU will certainly be less
likely to work well. But there is nothing quite like actually trying
it to see how it works. ;-)
That aside, it -is- possible to make such state changes appear atomic,
even when moving elements from one list to another. One way of doing
this is to atomically replace the element with a "tombstone" element.
Normal pointer writes suffice. The "tombstone" is set up so that searches
for the outgoing element will stall (e.g., spin or sleep, depending
on the environment). The element is moved to its destination list.
At this point, searches for the element in the old list will still
stall, while searches for the element in the new list will succeed.
The tombstone is now marked so that CPUs stall on it now resume, but
indicating failure to find the element in the old list.
Of course, this approach makes writes more expensive than they otherwise
would be, so, again, RCU is best for read-intensive uses. ;-)
The fact that this data structure is not very read-intensive is due
to the fact that short-lived TCP connections are quite common, right?
Or am I missing the finer points of this data structure's workings?
Thanx, Paul
^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: [RFC] Use RCU for tcp_ehash lookup
2004-09-01 11:36 ` Srivatsa Vaddagiri
2004-09-02 5:45 ` David S. Miller
@ 2004-09-02 21:19 ` Andi Kleen
1 sibling, 0 replies; 11+ messages in thread
From: Andi Kleen @ 2004-09-02 21:19 UTC (permalink / raw)
To: Srivatsa Vaddagiri
Cc: Andi Kleen, davem, netdev, linux-kernel, Dipankar, paulmck
On Wed, Sep 01, 2004 at 05:06:41PM +0530, Srivatsa Vaddagiri wrote:
> | 2.6.8.1 | 2.6.8.1 + my patch
> -------------------------------------------------------------------------------
> Average cycles | |
> spent in | |
> __tcp_v4_lookup_established | 2970.65 | 668.227
> | (~3.3 micro-seconds) | (~0.74 microseconds)
> -------------------------------------------------------------------------------
>
> This repesents improvement by a factor of 77.5%!
Nice.
>
>
> >
> > And it should also fix the performance problems with
> > cat /proc/net/tcp on ppc64/ia64 for large hash tables because the rw locks
> > are gone.
>
> But spinlocks are in! Would that still improve the performance compared to rw
> locks? (See me earlier note where I have explained that lookup done for
> /proc/net/tcp is _not_ lock-free yet).
Yes, spinlocks are much faster than rwlocks.
-Andi
^ permalink raw reply [flat|nested] 11+ messages in thread
end of thread, other threads:[~2004-09-02 21:19 UTC | newest]
Thread overview: 11+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2004-08-31 12:59 [RFC] Use RCU for tcp_ehash lookup Srivatsa Vaddagiri
2004-08-31 13:04 ` Srivatsa Vaddagiri
2004-08-31 13:12 ` Srivatsa Vaddagiri
2004-08-31 13:54 ` Andi Kleen
2004-09-01 11:36 ` Srivatsa Vaddagiri
2004-09-02 5:45 ` David S. Miller
2004-09-02 21:19 ` Andi Kleen
2004-09-02 5:43 ` David S. Miller
2004-09-02 5:41 ` David S. Miller
2004-09-02 14:04 ` Srivatsa Vaddagiri
2004-09-02 16:31 ` Paul E. McKenney
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).