[RFC][PATCH 0/2]Extend ext3 filesystem limit from 8TB to 16TB

[RFC][PATCH 0/2]Extend ext3 filesystem limit from 8TB to 16TB

[Mingming Cao]

There are places in ext3 code to use "int" to represent block numbers in
kernel(not on-disk). This seems the "only" reason that why we can only
have 8TB ext3 rather than 16TB.  Most times it just a bug with no
particular reason why not use unsigned 32 bit value, so the fix is easy.

However, it is not so straightforward fix for the ext3 block allocation
code, as ext3_new_block() returns a block number, and "-1" to indicating
block allocation failure. Ext3 block reservation code, called by
ext3_new_block(), thus also use "int" for block numbers in some places.

The following patches fixed both the ext3 block allocation code, as well
as the simple ones.

This work is inspired by Takashi's extend ext2/3 file/filesystem
limitation work, but rather, it focus on ext3 filesystem limit only, and
fixed the block allocation/reservation code to support in-kernel 2**32
block number. Also thanks to Laurent for his review.

Have verified these two patches on a 64 bit machine with 10TB ext3
filesystem, fsx runs fine for a few hours. Also testes on 32 bit machine
with <8TB ext3.

Please review this patches and I appreciate comments.

The things need to be done to complete this work is the issue with
current percpu counter, which could not handle u32 type count well. 

[RFC][PATCH 1/2]ext3 block allocation/reservation fixes to support 2**32 block numbers

[Mingming Cao]

This patch fixed the issue around the ext3 block allocation code to
treat block numbers to int type, as some places relies on block number
to be "-1" to indicate allocation failures.

The block allocation failure returned from ext3_new_blocks() is being
captured by the error anyway, so there is no need to keep the returned
block number as "int" type from ext3_new_blocks(). We could still keep
the returned allocated block as "int" from ext3_try_to_allocate_with_rsv
(), as it's a block group relative value so a 4 bytes is enough for now.

patch against 2.6.16-mm2.

Signed-Off-By: Mingming Cao <cmm@us.ibm.com>

---

 linux-2.6.16-ming/fs/ext3/balloc.c        |   67 +++++++++++++++---------------
 linux-2.6.16-ming/fs/ext3/xattr.c         |    6 +-
 linux-2.6.16-ming/include/linux/ext3_fs.h |    4 -
 3 files changed, 40 insertions(+), 37 deletions(-)

diff -puN fs/ext3/balloc.c~ext3_rsv_int-fix fs/ext3/balloc.c
--- linux-2.6.16/fs/ext3/balloc.c~ext3_rsv_int-fix	2006-03-29 15:49:41.199815437 -0800
+++ linux-2.6.16-ming/fs/ext3/balloc.c	2006-03-29 15:49:41.211814047 -0800
@@ -223,7 +223,7 @@ void ext3_rsv_window_add(struct super_bl
 {
 	struct rb_root *root = &EXT3_SB(sb)->s_rsv_window_root;
 	struct rb_node *node = &rsv->rsv_node;
-	unsigned int start = rsv->rsv_start;
+	unsigned long start = rsv->rsv_start;
 
 	struct rb_node ** p = &root->rb_node;
 	struct rb_node * parent = NULL;
@@ -656,7 +656,8 @@ ext3_try_to_allocate(struct super_block 
 			struct buffer_head *bitmap_bh, int goal,
 			unsigned long *count, struct ext3_reserve_window *my_rsv)
 {
-	int group_first_block, start, end;
+	unsigned long group_first_block;
+	int start, end;
 	unsigned long num = 0;
 
 	/* we do allocation within the reservation window if we have a window */
@@ -766,12 +767,13 @@ fail_access:
 static int find_next_reservable_window(
 				struct ext3_reserve_window_node *search_head,
 				struct ext3_reserve_window_node *my_rsv,
-				struct super_block * sb, int start_block,
-				int last_block)
+				struct super_block * sb,
+				unsigned long start_block,
+				unsigned long last_block)
 {
 	struct rb_node *next;
 	struct ext3_reserve_window_node *rsv, *prev;
-	int cur;
+	unsigned long cur;
 	int size = my_rsv->rsv_goal_size;
 
 	/* TODO: make the start of the reservation window byte-aligned */
@@ -889,8 +891,8 @@ static int alloc_new_reservation(struct 
 		unsigned int group, struct buffer_head *bitmap_bh)
 {
 	struct ext3_reserve_window_node *search_head;
-	int group_first_block, group_end_block, start_block;
-	int first_free_block;
+	unsigned long group_first_block, group_end_block, start_block;
+	unsigned long first_free_block;
 	struct rb_root *fs_rsv_root = &EXT3_SB(sb)->s_rsv_window_root;
 	unsigned long size;
 	int ret;
@@ -1200,16 +1202,17 @@ int ext3_should_retry_alloc(struct super
  * bitmap, and then for any free bit if that fails.
  * This function also updates quota and i_blocks field.
  */
-int ext3_new_blocks(handle_t *handle, struct inode *inode,
+unsigned long ext3_new_blocks(handle_t *handle, struct inode *inode,
 			unsigned long goal, unsigned long *count, int *errp)
 {
 	struct buffer_head *bitmap_bh = NULL;
 	struct buffer_head *gdp_bh;
 	int group_no;
 	int goal_group;
-	int ret_block;
+	int group_target_blk;
+	int group_allocated_blk;
+	unsigned long ret_block;
 	int bgi;			/* blockgroup iteration index */
-	int target_block;
 	int fatal = 0, err;
 	int performed_allocation = 0;
 	int free_blocks;
@@ -1285,16 +1288,17 @@ retry:
 		my_rsv = NULL;
 
 	if (free_blocks > 0) {
-		ret_block = ((goal - le32_to_cpu(es->s_first_data_block)) %
+		group_target_blk = ((goal - le32_to_cpu(es->s_first_data_block)) %
 				EXT3_BLOCKS_PER_GROUP(sb));
 		bitmap_bh = read_block_bitmap(sb, group_no);
 		if (!bitmap_bh)
 			goto io_error;
-		ret_block = ext3_try_to_allocate_with_rsv(sb, handle, group_no,
-					bitmap_bh, ret_block, my_rsv, &num, &fatal);
+		group_allocated_blk = ext3_try_to_allocate_with_rsv(sb, handle,
+					 group_no, bitmap_bh,
+					group_target_blk, my_rsv, &num, &fatal);
 		if (fatal)
 			goto out;
-		if (ret_block >= 0)
+		if (group_allocated_blk >= 0)
 			goto allocated;
 	}
 
@@ -1327,11 +1331,12 @@ retry:
 		bitmap_bh = read_block_bitmap(sb, group_no);
 		if (!bitmap_bh)
 			goto io_error;
-		ret_block = ext3_try_to_allocate_with_rsv(sb, handle, group_no,
+		group_allocated_blk = ext3_try_to_allocate_with_rsv(sb, handle,
+					group_no,
 					bitmap_bh, -1, my_rsv, &num, &fatal);
 		if (fatal)
 			goto out;
-		if (ret_block >= 0) 
+		if (group_allocated_blk >= 0)
 			goto allocated;
 	}
 	/*
@@ -1360,18 +1365,19 @@ allocated:
 	if (fatal)
 		goto out;
 
-	target_block = ret_block + group_no * EXT3_BLOCKS_PER_GROUP(sb)
+	ret_block = group_allocated_blk + group_no *
+				EXT3_BLOCKS_PER_GROUP(sb)
 				+ le32_to_cpu(es->s_first_data_block);
 
-	if (in_range(le32_to_cpu(gdp->bg_block_bitmap), target_block, num) ||
-	    in_range(le32_to_cpu(gdp->bg_inode_bitmap), target_block, num) ||
-	    in_range(target_block, le32_to_cpu(gdp->bg_inode_table),
+	if (in_range(le32_to_cpu(gdp->bg_block_bitmap), ret_block, num) ||
+	    in_range(le32_to_cpu(gdp->bg_inode_bitmap), ret_block, num) ||
+	    in_range(ret_block, le32_to_cpu(gdp->bg_inode_table),
 		      EXT3_SB(sb)->s_itb_per_group) ||
-	    in_range(target_block + num - 1, le32_to_cpu(gdp->bg_inode_table),
+	    in_range(ret_block + num - 1, le32_to_cpu(gdp->bg_inode_table),
 		      EXT3_SB(sb)->s_itb_per_group))
 		ext3_error(sb, "ext3_new_block",
 			    "Allocating block in system zone - "
-			    "blocks from %u, length %lu", target_block, num);
+			    "blocks from %lu, length %lu", ret_block, num);
 
 	performed_allocation = 1;
 
@@ -1380,7 +1386,7 @@ allocated:
 		struct buffer_head *debug_bh;
 
 		/* Record bitmap buffer state in the newly allocated block */
-		debug_bh = sb_find_get_block(sb, target_block);
+		debug_bh = sb_find_get_block(sb, ret_block);
 		if (debug_bh) {
 			BUFFER_TRACE(debug_bh, "state when allocated");
 			BUFFER_TRACE2(debug_bh, bitmap_bh, "bitmap state");
@@ -1393,24 +1399,21 @@ allocated:
 		int i;
 
 		for (i = 0; i < num; i++) {
-			if (ext3_test_bit(ret_block,
+			if (ext3_test_bit(group_allocated_blk,
 					bh2jh(bitmap_bh)->b_committed_data)) {
 				printk("%s: block was unexpectedly set in "
 					"b_committed_data\n", __FUNCTION__);
 			}
 		}
 	}
-	ext3_debug("found bit %d\n", ret_block);
+	ext3_debug("found bit %d\n", group_allocated_blk);
 	spin_unlock(sb_bgl_lock(sbi, group_no));
 	jbd_unlock_bh_state(bitmap_bh);
 #endif
 
-	/* ret_block was blockgroup-relative.  Now it becomes fs-relative */
-	ret_block = target_block;
-
 	if (ret_block + num - 1 >= le32_to_cpu(es->s_blocks_count)) {
 		ext3_error(sb, "ext3_new_block",
-			    "block(%d) >= blocks count(%d) - "
+			    "block(%lu) >= blocks count(%d) - "
 			    "block_group = %d, es == %p ", ret_block,
 			le32_to_cpu(es->s_blocks_count), group_no, es);
 		goto out;
@@ -1421,8 +1424,8 @@ allocated:
 	 * list of some description.  We don't know in advance whether
 	 * the caller wants to use it as metadata or data.
 	 */
-	ext3_debug("allocating block %d. Goal hits %d of %d.\n",
-			ret_block, goal_hits, goal_attempts);
+	ext3_debug("allocating block %lu. Goal hits %d of %d.\n",
+		ret_block, goal_hits, goal_attempts);
 
 	spin_lock(sb_bgl_lock(sbi, group_no));
 	gdp->bg_free_blocks_count =
@@ -1461,7 +1464,7 @@ out:
 	return 0;
 }
 
-int ext3_new_block(handle_t *handle, struct inode *inode,
+unsigned long ext3_new_block(handle_t *handle, struct inode *inode,
 			unsigned long goal, int *errp)
 {
 	unsigned long count = 1;
diff -puN fs/ext3/xattr.c~ext3_rsv_int-fix fs/ext3/xattr.c
--- linux-2.6.16/fs/ext3/xattr.c~ext3_rsv_int-fix	2006-03-29 15:49:41.202815089 -0800
+++ linux-2.6.16-ming/fs/ext3/xattr.c	2006-03-29 15:49:41.213813815 -0800
@@ -792,14 +792,14 @@ inserted:
 			get_bh(new_bh);
 		} else {
 			/* We need to allocate a new block */
-			int goal = le32_to_cpu(
+			unsigned long goal = le32_to_cpu(
 					EXT3_SB(sb)->s_es->s_first_data_block) +
 				EXT3_I(inode)->i_block_group *
 				EXT3_BLOCKS_PER_GROUP(sb);
-			int block = ext3_new_block(handle, inode, goal, &error);
+			unsigned long block = ext3_new_block(handle, inode, goal, &error);
 			if (error)
 				goto cleanup;
-			ea_idebug(inode, "creating block %d", block);
+			ea_idebug(inode, "creating block %lu", block);
 
 			new_bh = sb_getblk(sb, block);
 			if (!new_bh) {
diff -puN include/linux/ext3_fs.h~ext3_rsv_int-fix include/linux/ext3_fs.h
--- linux-2.6.16/include/linux/ext3_fs.h~ext3_rsv_int-fix	2006-03-29 15:49:41.205814742 -0800
+++ linux-2.6.16-ming/include/linux/ext3_fs.h	2006-03-29 15:49:41.214813699 -0800
@@ -732,8 +732,8 @@ struct dir_private_info {
 /* balloc.c */
 extern int ext3_bg_has_super(struct super_block *sb, int group);
 extern unsigned long ext3_bg_num_gdb(struct super_block *sb, int group);
-extern int ext3_new_block (handle_t *, struct inode *, unsigned long, int *);
-extern int ext3_new_blocks (handle_t *, struct inode *, unsigned long,
+extern unsigned long ext3_new_block (handle_t *, struct inode *, unsigned long, int *);
+extern unsigned long ext3_new_blocks (handle_t *, struct inode *, unsigned long,
 			unsigned long *, int *);
 extern void ext3_free_blocks (handle_t *, struct inode *, unsigned long,
 			      unsigned long);

_

[RFC][PATCH 2/2]Other ext3 in-kernel block number type fix to support 2**32 block numbers

[Mingming Cao]

This trivial patch fixed other places in ext3 code(non block allocation
part) to replace "int" type filesystem block number with "unsigned
long".


Signed-Off-By: Mingming Cao <cmm@us.ibm.com>

---

 linux-2.6.16-ming/fs/ext3/balloc.c        |    4 ++--
 linux-2.6.16-ming/fs/ext3/ialloc.c        |    2 +-
 linux-2.6.16-ming/fs/ext3/inode.c         |    2 +-
 linux-2.6.16-ming/fs/ext3/resize.c        |    4 ++--
 linux-2.6.16-ming/fs/ext3/xattr.c         |   16 ++++++++--------
 linux-2.6.16-ming/include/linux/ext3_fs.h |    2 +-
 6 files changed, 15 insertions(+), 15 deletions(-)

diff -puN fs/ext3/balloc.c~ext3_32bit_kernel_fix fs/ext3/balloc.c
--- linux-2.6.16/fs/ext3/balloc.c~ext3_32bit_kernel_fix	2006-03-24 21:32:32.000000000 -0800
+++ linux-2.6.16-ming/fs/ext3/balloc.c	2006-03-27 15:47:17.344404203 -0800
@@ -496,7 +496,7 @@ void ext3_free_blocks(handle_t *handle, 
 			unsigned long block, unsigned long count)
 {
 	struct super_block * sb;
-	int dquot_freed_blocks;
+	unsigned long dquot_freed_blocks;
 
 	sb = inode->i_sb;
 	if (!sb) {
@@ -1166,7 +1166,7 @@ out:
 
 static int ext3_has_free_blocks(struct ext3_sb_info *sbi)
 {
-	int free_blocks, root_blocks;
+	unsigned long free_blocks, root_blocks;
 
 	free_blocks = percpu_counter_read_positive(&sbi->s_freeblocks_counter);
 	root_blocks = le32_to_cpu(sbi->s_es->s_r_blocks_count);
diff -puN fs/ext3/ialloc.c~ext3_32bit_kernel_fix fs/ext3/ialloc.c
--- linux-2.6.16/fs/ext3/ialloc.c~ext3_32bit_kernel_fix	2006-03-24 21:32:32.000000000 -0800
+++ linux-2.6.16-ming/fs/ext3/ialloc.c	2006-03-24 21:32:32.000000000 -0800
@@ -262,7 +262,7 @@ static int find_group_orlov(struct super
 	int ngroups = sbi->s_groups_count;
 	int inodes_per_group = EXT3_INODES_PER_GROUP(sb);
 	int freei, avefreei;
-	int freeb, avefreeb;
+	unsigned long freeb, avefreeb;
 	int blocks_per_dir, ndirs;
 	int max_debt, max_dirs, min_blocks, min_inodes;
 	int group = -1, i;
diff -puN fs/ext3/inode.c~ext3_32bit_kernel_fix fs/ext3/inode.c
--- linux-2.6.16/fs/ext3/inode.c~ext3_32bit_kernel_fix	2006-03-24 21:32:32.000000000 -0800
+++ linux-2.6.16-ming/fs/ext3/inode.c	2006-03-24 21:32:32.000000000 -0800
@@ -62,7 +62,7 @@ static int ext3_inode_is_fast_symlink(st
  * still needs to be revoked.
  */
 int ext3_forget(handle_t *handle, int is_metadata, struct inode *inode,
-			struct buffer_head *bh, int blocknr)
+			struct buffer_head *bh, unsigned long blocknr)
 {
 	int err;
 
diff -puN include/linux/ext3_fs.h~ext3_32bit_kernel_fix include/linux/ext3_fs.h
--- linux-2.6.16/include/linux/ext3_fs.h~ext3_32bit_kernel_fix	2006-03-24 21:32:32.000000000 -0800
+++ linux-2.6.16-ming/include/linux/ext3_fs.h	2006-03-24 21:32:32.000000000 -0800
@@ -775,7 +775,7 @@ extern unsigned long ext3_count_free (st
 
 
 /* inode.c */
-int ext3_forget(handle_t *, int, struct inode *, struct buffer_head *, int);
+int ext3_forget(handle_t *, int, struct inode *, struct buffer_head *, unsigned long);
 struct buffer_head * ext3_getblk (handle_t *, struct inode *, long, int, int *);
 struct buffer_head * ext3_bread (handle_t *, struct inode *, int, int, int *);
 int ext3_get_blocks_handle(handle_t *handle, struct inode *inode,
diff -puN fs/ext3/resize.c~ext3_32bit_kernel_fix fs/ext3/resize.c
--- linux-2.6.16/fs/ext3/resize.c~ext3_32bit_kernel_fix	2006-03-24 21:32:32.000000000 -0800
+++ linux-2.6.16-ming/fs/ext3/resize.c	2006-03-24 21:32:32.000000000 -0800
@@ -990,10 +990,10 @@ int ext3_group_extend(struct super_block
 	ext3_journal_dirty_metadata(handle, EXT3_SB(sb)->s_sbh);
 	sb->s_dirt = 1;
 	unlock_super(sb);
-	ext3_debug("freeing blocks %ld through %ld\n", o_blocks_count,
+	ext3_debug("freeing blocks %lu through %lu\n", o_blocks_count,
 		   o_blocks_count + add);
 	ext3_free_blocks_sb(handle, sb, o_blocks_count, add, &freed_blocks);
-	ext3_debug("freed blocks %ld through %ld\n", o_blocks_count,
+	ext3_debug("freed blocks %lu through %lu\n", o_blocks_count,
 		   o_blocks_count + add);
 	if ((err = ext3_journal_stop(handle)))
 		goto exit_put;
diff -puN fs/ext3/xattr.c~ext3_32bit_kernel_fix fs/ext3/xattr.c
--- linux-2.6.16/fs/ext3/xattr.c~ext3_32bit_kernel_fix	2006-03-24 21:32:32.000000000 -0800
+++ linux-2.6.16-ming/fs/ext3/xattr.c	2006-03-24 21:32:32.000000000 -0800
@@ -225,7 +225,7 @@ ext3_xattr_block_get(struct inode *inode
 	error = -ENODATA;
 	if (!EXT3_I(inode)->i_file_acl)
 		goto cleanup;
-	ea_idebug(inode, "reading block %d", EXT3_I(inode)->i_file_acl);
+	ea_idebug(inode, "reading block %u", EXT3_I(inode)->i_file_acl);
 	bh = sb_bread(inode->i_sb, EXT3_I(inode)->i_file_acl);
 	if (!bh)
 		goto cleanup;
@@ -233,7 +233,7 @@ ext3_xattr_block_get(struct inode *inode
 		atomic_read(&(bh->b_count)), le32_to_cpu(BHDR(bh)->h_refcount));
 	if (ext3_xattr_check_block(bh)) {
 bad_block:	ext3_error(inode->i_sb, __FUNCTION__,
-			   "inode %ld: bad block %d", inode->i_ino,
+			   "inode %ld: bad block %u", inode->i_ino,
 			   EXT3_I(inode)->i_file_acl);
 		error = -EIO;
 		goto cleanup;
@@ -366,7 +366,7 @@ ext3_xattr_block_list(struct inode *inod
 	error = 0;
 	if (!EXT3_I(inode)->i_file_acl)
 		goto cleanup;
-	ea_idebug(inode, "reading block %d", EXT3_I(inode)->i_file_acl);
+	ea_idebug(inode, "reading block %u", EXT3_I(inode)->i_file_acl);
 	bh = sb_bread(inode->i_sb, EXT3_I(inode)->i_file_acl);
 	error = -EIO;
 	if (!bh)
@@ -375,7 +375,7 @@ ext3_xattr_block_list(struct inode *inod
 		atomic_read(&(bh->b_count)), le32_to_cpu(BHDR(bh)->h_refcount));
 	if (ext3_xattr_check_block(bh)) {
 		ext3_error(inode->i_sb, __FUNCTION__,
-			   "inode %ld: bad block %d", inode->i_ino,
+			   "inode %ld: bad block %u", inode->i_ino,
 			   EXT3_I(inode)->i_file_acl);
 		error = -EIO;
 		goto cleanup;
@@ -647,7 +647,7 @@ ext3_xattr_block_find(struct inode *inod
 			le32_to_cpu(BHDR(bs->bh)->h_refcount));
 		if (ext3_xattr_check_block(bs->bh)) {
 			ext3_error(sb, __FUNCTION__,
-				"inode %ld: bad block %d", inode->i_ino,
+				"inode %ld: bad block %u", inode->i_ino,
 				EXT3_I(inode)->i_file_acl);
 			error = -EIO;
 			goto cleanup;
@@ -847,7 +847,7 @@ cleanup_dquot:
 
 bad_block:
 	ext3_error(inode->i_sb, __FUNCTION__,
-		   "inode %ld: bad block %d", inode->i_ino,
+		   "inode %ld: bad block %u", inode->i_ino,
 		   EXT3_I(inode)->i_file_acl);
 	goto cleanup;
 
@@ -1076,14 +1076,14 @@ ext3_xattr_delete_inode(handle_t *handle
 	bh = sb_bread(inode->i_sb, EXT3_I(inode)->i_file_acl);
 	if (!bh) {
 		ext3_error(inode->i_sb, __FUNCTION__,
-			"inode %ld: block %d read error", inode->i_ino,
+			"inode %ld: block %u read error", inode->i_ino,
 			EXT3_I(inode)->i_file_acl);
 		goto cleanup;
 	}
 	if (BHDR(bh)->h_magic != cpu_to_le32(EXT3_XATTR_MAGIC) ||
 	    BHDR(bh)->h_blocks != cpu_to_le32(1)) {
 		ext3_error(inode->i_sb, __FUNCTION__,
-			"inode %ld: bad block %d", inode->i_ino,
+			"inode %ld: bad block %u", inode->i_ino,
 			EXT3_I(inode)->i_file_acl);
 		goto cleanup;
 	}

_

Re: [RFC][PATCH 0/2]Extend ext3 filesystem limit from 8TB to 16TB

[Andrew Morton]

Mingming Cao <cmm@us.ibm.com> wrote:
>
> The things need to be done to complete this work is the issue with
>  current percpu counter, which could not handle u32 type count well. 

I'm surprised there's much of a problem here.  It is a 32-bit value, so it
should mainly be a matter of treating the return value from
percpu_counter_read() as unsigned long.

However a stickier problem is when dealing with a filesystem which has,
say, 0xffff_ff00 blocks.  Because percpu counters are approximate, and a
counter which really has a value of 0xffff_feee might return 0x00000123. 
What do we do then?

Of course the simple option is to nuke the percpu counters in ext3 and use
atomic_long_t (which is signed, so appropriate treat-it-as-unsigned code
would be needed).  I doubt if the percpu counters in ext3 are gaining us
much.


-------------------------------------------------------
This SF.Net email is sponsored by xPML, a groundbreaking scripting language
that extends applications into web and mobile media. Attend the live webcast
and join the prime developer group breaking into this new coding territory!
http://sel.as-us.falkag.net/sel?cmd=lnk&kid=110944&bid=241720&dat=121642

Re: [RFC][PATCH 0/2]Extend ext3 filesystem limit from 8TB to 16TB

[Andreas Dilger]

On Mar 29, 2006  17:38 -0800, Mingming Cao wrote:
> There are places in ext3 code to use "int" to represent block numbers in
> kernel(not on-disk). This seems the "only" reason that why we can only
> have 8TB ext3 rather than 16TB.  Most times it just a bug with no
> particular reason why not use unsigned 32 bit value, so the fix is easy.
> 
> However, it is not so straightforward fix for the ext3 block allocation
> code, as ext3_new_block() returns a block number, and "-1" to indicating
> block allocation failure. Ext3 block reservation code, called by
> ext3_new_block(), thus also use "int" for block numbers in some places.

What might make the code a lot clearer, easier to audit, and easier to
fix in the future is to declare new types for fs block offsets and group
block offsets.  Something like "ext3_fsblk" and "ext3_grblk".  That way,
we can declare ext3_fsblk as "unsigned long" and "ext3_grblk" as "unsigned
int", and we could optionally change ext3_fsblk to be "unsigned long long"
later to support 64-bit filesystems without having to re-patch all of the
code.

It would be more clear what type of block offset a function is handling
(fs-wide or group-relative).  If we wanted to be able to overload the
block number with an error code we could use ERR_PTR and PTR_ERR like
macros, and just restrict the filesystem to 2^32 - 1024 blocks until we
extend it to 64 bits.

Cheers, Andreas
--
Andreas Dilger
Principal Software Engineer
Cluster File Systems, Inc.

Re: [RFC][PATCH 0/2]Extend ext3 filesystem limit from 8TB to 16TB

[Andreas Dilger]

On Mar 29, 2006  17:38 -0800, Mingming Cao wrote:
> Have verified these two patches on a 64 bit machine with 10TB ext3
> filesystem, fsx runs fine for a few hours. Also testes on 32 bit machine
> with <8TB ext3.

Have you done tests _near_ 8TB with a 32-bit machine, even without these
patches?  In particular, filling up the filesystem to be close to full
so that we really depend on the > 2TB code to work properly?  Also, in
theory with these patches even a 32-bit machine could run > 8TB, right?

There have been sporadic reports of failure for large ext3 filesystems,
and some of them say that 32-bit systems fail and 64-bit systems work.
There is a kernel bugzilla bug open for this, but it was never really
identified what the source of the problem was.

Cheers, Andreas
--
Andreas Dilger
Principal Software Engineer
Cluster File Systems, Inc.

Re: [RFC][PATCH 0/2]Extend ext3 filesystem limit from 8TB to 16TB

[Mingming Cao]

On Thu, 2006-03-30 at 10:36 -0700, Andreas Dilger wrote:
> On Mar 29, 2006  17:38 -0800, Mingming Cao wrote:
> > There are places in ext3 code to use "int" to represent block numbers in
> > kernel(not on-disk). This seems the "only" reason that why we can only
> > have 8TB ext3 rather than 16TB.  Most times it just a bug with no
> > particular reason why not use unsigned 32 bit value, so the fix is easy.
> > 
> > However, it is not so straightforward fix for the ext3 block allocation
> > code, as ext3_new_block() returns a block number, and "-1" to indicating
> > block allocation failure. Ext3 block reservation code, called by
> > ext3_new_block(), thus also use "int" for block numbers in some places.
> 
Hi Andreas,

> What might make the code a lot clearer, easier to audit, and easier to
> fix in the future is to declare new types for fs block offsets and group
> block offsets.  Something like "ext3_fsblk" and "ext3_grblk".  That way,
> we can declare ext3_fsblk as "unsigned long" and "ext3_grblk" as "unsigned
> int", 

Yep, that makes sense. If we do this, the patch needs more audit, as the
existing code uses "unsigned long" for block numbers in many many
places.

Also I think it might make sense to define "ext3_fileblk" for logical
block type, as right now many functions called "block" in many places
for file logical block, and it takes some to determine whether it's a
file logical block or physical block.

> and we could optionally change ext3_fsblk to be "unsigned long long"
> later to support 64-bit filesystems without having to re-patch all of the
> code.
> 
I do have an untested patch which tries to change all fs-wide block
numbers from "unsigned long" to "sector_t" type as Laurent suggested. He
did this in his 64 bit ext3 block number support patch. I wasn't sure if
we should do this for current 32 bit ext3 or wait until other 64 bit
patches.

Yeah, with the suggestion you made above, this change should be easy to
support 64bit filesystem without go through all the code again.

> It would be more clear what type of block offset a function is handling
> (fs-wide or group-relative). 

Okey, I will add more comments in the function.

>  If we wanted to be able to overload the
> block number with an error code we could use ERR_PTR and PTR_ERR like
> macros, and just restrict the filesystem to 2^32 - 1024 blocks until we
> extend it to 64 bits.
> 
> Cheers, Andreas
> --
> Andreas Dilger
> Principal Software Engineer
> Cluster File Systems, Inc.
> 
> -
> To unsubscribe from this list: send the line "unsubscribe linux-fsdevel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [RFC][PATCH 0/2]Extend ext3 filesystem limit from 8TB to 16TB

[Mingming Cao]

On Thu, 2006-03-30 at 10:40 -0700, Andreas Dilger wrote:
> On Mar 29, 2006  17:38 -0800, Mingming Cao wrote:
> > Have verified these two patches on a 64 bit machine with 10TB ext3
> > filesystem, fsx runs fine for a few hours. Also testes on 32 bit machine
> > with <8TB ext3.
> 
> Have you done tests _near_ 8TB with a 32-bit machine, even without these
> patches?
No I haven't. The >8TB right now is attached to a 64 bit machine, but we
should able to move it to a 32 bit machine.

>   In particular, filling up the filesystem to be close to full
> so that we really depend on the > 2TB code to work properly?

I made a kernel patch to allow a file to specify which block group it
wants it's blocks to allocate from(using ioctl to set the goal
allocation block group). I set the goal block group falls to somewhere
>8TB, and did dd tests on that file. Verified this with debugfs, the
allocated block numbers are beyond 2**31.

Also before run fsx tests, created many directories (32768 at most:) and
verified one directory's inode is located in block group >8TB space. So
when we do fsx test on files under that directory, we are
creating/testing files >8TB.

BTW, do you think this ioctl is useful in general for other users? I
attached the patch here.

I also plan to hack the code of inode allocation to force all files's
inode is put in the block group >8TB, so that we could do a full
filesystem tests there.


>   Also, in
> theory with these patches even a 32-bit machine could run > 8TB, right?
> 
> There have been sporadic reports of failure for large ext3 filesystems,
> and some of them say that 32-bit systems fail and 64-bit systems work.
> There is a kernel bugzilla bug open for this, but it was never really
> identified what the source of the problem was.
> 

Sure, I will verify that on my 32 bit machine with >8TB.

> Cheers, Andreas
> --
> Andreas Dilger
> Principal Software Engineer
> Cluster File Systems, Inc.
> 

Thanks,

Mingming

Re: [RFC][PATCH 0/2]Extend ext3 filesystem limit from 8TB to 16TB

[Mingming Cao]

On Thu, 2006-03-30 at 11:16 -0800, Mingming Cao wrote:
> On Thu, 2006-03-30 at 10:40 -0700, Andreas Dilger wrote:
> > On Mar 29, 2006  17:38 -0800, Mingming Cao wrote:
> > > Have verified these two patches on a 64 bit machine with 10TB ext3
> > > filesystem, fsx runs fine for a few hours. Also testes on 32 bit machine
> > > with <8TB ext3.
> > 
> > Have you done tests _near_ 8TB with a 32-bit machine, even without these
> > patches?
> No I haven't. The >8TB right now is attached to a 64 bit machine, but we
> should able to move it to a 32 bit machine.
> 
> >   In particular, filling up the filesystem to be close to full
> > so that we really depend on the > 2TB code to work properly?
> 
> I made a kernel patch to allow a file to specify which block group it
> wants it's blocks to allocate from(using ioctl to set the goal
> allocation block group). I set the goal block group falls to somewhere
> >8TB, and did dd tests on that file. Verified this with debugfs, the
> allocated block numbers are beyond 2**31.
> 
> Also before run fsx tests, created many directories (32768 at most:) and
> verified one directory's inode is located in block group >8TB space. So
> when we do fsx test on files under that directory, we are
> creating/testing files >8TB.
> 
> BTW, do you think this ioctl is useful in general for other users? I
> attached the patch here.
> 
---

 linux-2.6.16-ming/fs/ext3/balloc.c          |   24 ++++++++++++++---------
 linux-2.6.16-ming/fs/ext3/ioctl.c           |   29 ++++++++++++++++++++++++++++
 linux-2.6.16-ming/include/linux/ext3_fs.h   |    1 
 linux-2.6.16-ming/include/linux/ext3_fs_i.h |    1 
 4 files changed, 46 insertions(+), 9 deletions(-)

diff -puN fs/ext3/ioctl.c~ext3_set_alloc_blk_group_hack fs/ext3/ioctl.c
--- linux-2.6.16/fs/ext3/ioctl.c~ext3_set_alloc_blk_group_hack	2006-03-28 15:19:58.000000000 -0800
+++ linux-2.6.16-ming/fs/ext3/ioctl.c	2006-03-28 15:54:14.000000000 -0800
@@ -22,6 +22,7 @@ int ext3_ioctl (struct inode * inode, st
 	struct ext3_inode_info *ei = EXT3_I(inode);
 	unsigned int flags;
 	unsigned short rsv_window_size;
+	unsigned int blk_group;
 
 	ext3_debug ("cmd = %u, arg = %lu\n", cmd, arg);
 
@@ -193,6 +194,34 @@ flags_err:
 		mutex_unlock(&ei->truncate_mutex);
 		return 0;
 	}
+	case EXT3_IOC_SETALLOCBLKGRP: {
+
+		if (!test_opt(inode->i_sb, RESERVATION) ||!S_ISREG(inode->i_mode))
+			return -ENOTTY;
+
+		if (IS_RDONLY(inode))
+			return -EROFS;
+
+		if ((current->fsuid != inode->i_uid) && !capable(CAP_FOWNER))
+			return -EACCES;
+
+		if (get_user(blk_group, (int __user *)arg))
+			return -EFAULT;
+
+		/*
+		 * need to allocate reservation structure for this inode
+		 * before set the window size
+		 */
+		mutex_lock(&ei->truncate_mutex);
+		if (!ei->i_block_alloc_info)
+			ext3_init_block_alloc_info(inode);
+
+		if (ei->i_block_alloc_info){
+			ei->i_block_alloc_info->goal_block_group = blk_group;
+		}
+		mutex_unlock(&ei->truncate_mutex);
+		return 0;
+	}
 	case EXT3_IOC_GROUP_EXTEND: {
 		unsigned long n_blocks_count;
 		struct super_block *sb = inode->i_sb;
diff -puN include/linux/ext3_fs.h~ext3_set_alloc_blk_group_hack include/linux/ext3_fs.h
--- linux-2.6.16/include/linux/ext3_fs.h~ext3_set_alloc_blk_group_hack	2006-03-28 15:42:51.000000000 -0800
+++ linux-2.6.16-ming/include/linux/ext3_fs.h	2006-03-28 15:51:48.000000000 -0800
@@ -238,6 +238,7 @@ struct ext3_new_group_data {
 #endif
 #define EXT3_IOC_GETRSVSZ		_IOR('f', 5, long)
 #define EXT3_IOC_SETRSVSZ		_IOW('f', 6, long)
+#define EXT3_IOC_SETALLOCBLKGRP		_IOW('f', 9, long)
 
 /*
  *  Mount options
diff -puN include/linux/ext3_fs_i.h~ext3_set_alloc_blk_group_hack include/linux/ext3_fs_i.h
--- linux-2.6.16/include/linux/ext3_fs_i.h~ext3_set_alloc_blk_group_hack	2006-03-28 15:43:59.000000000 -0800
+++ linux-2.6.16-ming/include/linux/ext3_fs_i.h	2006-03-28 15:47:54.000000000 -0800
@@ -51,6 +51,7 @@ struct ext3_block_alloc_info {
 	 * allocation when we detect linearly ascending requests.
 	 */
 	__u32                   last_alloc_physical_block;
+	__u32			goal_block_group;
 };
 
 #define rsv_start rsv_window._rsv_start
diff -puN fs/ext3/balloc.c~ext3_set_alloc_blk_group_hack fs/ext3/balloc.c
--- linux-2.6.16/fs/ext3/balloc.c~ext3_set_alloc_blk_group_hack	2006-03-28 15:45:30.000000000 -0800
+++ linux-2.6.16-ming/fs/ext3/balloc.c	2006-03-28 16:03:55.000000000 -0800
@@ -285,6 +285,7 @@ void ext3_init_block_alloc_info(struct i
 		rsv->rsv_alloc_hit = 0;
 		block_i->last_alloc_logical_block = 0;
 		block_i->last_alloc_physical_block = 0;
+		block_i->goal_block_group = 0;
 	}
 	ei->i_block_alloc_info = block_i;
 }
@@ -1263,15 +1264,20 @@ unsigned long ext3_new_blocks(handle_t *
 		*errp = -ENOSPC;
 		goto out;
 	}
-
-	/*
-	 * First, test whether the goal block is free.
-	 */
-	if (goal < le32_to_cpu(es->s_first_data_block) ||
-	    goal >= le32_to_cpu(es->s_blocks_count))
-		goal = le32_to_cpu(es->s_first_data_block);
-	group_no = (goal - le32_to_cpu(es->s_first_data_block)) /
-			EXT3_BLOCKS_PER_GROUP(sb);
+	if (block_i->goal_block_group) {
+		group_no = block_i->goal_block_group;
+		goal = le32_to_cpu(EXT3_SB(sb)->s_es->s_first_data_block) +                                group_no * EXT3_BLOCKS_PER_GROUP(sb);
+		block_i->goal_block_group = 0;
+	} else {
+		/*
+		 * First, test whether the goal block is free.
+		 */
+		if (goal < le32_to_cpu(es->s_first_data_block) ||
+		    goal >= le32_to_cpu(es->s_blocks_count))
+			goal = le32_to_cpu(es->s_first_data_block);
+		group_no = (goal - le32_to_cpu(es->s_first_data_block)) /
+				EXT3_BLOCKS_PER_GROUP(sb);
+	}
 	gdp = ext3_get_group_desc(sb, group_no, &gdp_bh);
 	if (!gdp)
 		goto io_error;

_

Re: [RFC][PATCH 0/2]Extend ext3 filesystem limit from 8TB to 16TB

[Andreas Dilger]

On Mar 30, 2006  11:22 -0800, Mingming Cao wrote:
> I made a kernel patch to allow a file to specify which block group it
> wants it's blocks to allocate from(using ioctl to set the goal
> allocation block group). I set the goal block group falls to somewhere
> >8TB, and did dd tests on that file. Verified this with debugfs, the
> allocated block numbers are beyond 2**31.
> 
> Also before run fsx tests, created many directories (32768 at most:) and
> verified one directory's inode is located in block group >8TB space. So
> when we do fsx test on files under that directory, we are
> creating/testing files >8TB.

While useful, I don't think it is critical.  As you mention, it is possible
to do this by creating a lot of directories, though it might be tedious
(need over 16k directories for a 2TB filesystem, 64k for an 8TB fs).

Also, since this increases the allocation for each inode's reservation from
16 bytes to 20 (really 32 because it is in a slab), it might have a small
performance hit.

If it was available under some sort of compile-time configuration option
it might make sense for developers.

Cheers, Andreas
--
Andreas Dilger
Principal Software Engineer
Cluster File Systems, Inc.

Re: [RFC][PATCH 0/2]Extend ext3 filesystem limit from 8TB to 16TB

[Andi Kleen]

Mingming Cao <cmm@us.ibm.com> writes:

> On Thu, 2006-03-30 at 10:40 -0700, Andreas Dilger wrote:
> > On Mar 29, 2006  17:38 -0800, Mingming Cao wrote:
> > > Have verified these two patches on a 64 bit machine with 10TB ext3
> > > filesystem, fsx runs fine for a few hours. Also testes on 32 bit machine
> > > with <8TB ext3.
> > 
> > Have you done tests _near_ 8TB with a 32-bit machine, even without these
> > patches?
> No I haven't. The >8TB right now is attached to a 64 bit machine, but we
> should able to move it to a 32 bit machine.

If you use XFS or JFS as backing fs you can use a holey loop device
to simulate it.  When I tried this last time JFS worked better for me.
XFS doesn't seem to like that many extents as will be created by 
mkfs.ext2.

-Andi

Re: [RFC][PATCH 0/2]Extend ext3 filesystem limit from 8TB to 16TB

[Mingming Cao]

On Wed, 2006-03-29 at 17:54 -0800, Andrew Morton wrote:
> Mingming Cao <cmm@us.ibm.com> wrote:
> >
> > The things need to be done to complete this work is the issue with
> >  current percpu counter, which could not handle u32 type count well. 
> 
> I'm surprised there's much of a problem here.  It is a 32-bit value, so it
> should mainly be a matter of treating the return value from
> percpu_counter_read() as unsigned long.
> 
> However a stickier problem is when dealing with a filesystem which has,
> say, 0xffff_ff00 blocks.  Because percpu counters are approximate, and a
> counter which really has a value of 0xffff_feee might return 0x00000123. 
> What do we do then?
> 

Hmm... I think we had this issue already even with today's 2**31 ext3.
Since ext2/3 always use percpu_counter_read_positive() to get the total
number of free blocks, so if the real free blocks is 0x0fff_feee, and
the approximate value from the percpu counter is 0xf000_0123, the
percpu_counter_read_positive() will return back 0x0000123.

> Of course the simple option is to nuke the percpu counters in ext3 and use
> atomic_long_t (which is signed, so appropriate treat-it-as-unsigned code
> would be needed).  I doubt if the percpu counters in ext3 are gaining us
> much.

Sounds like the simple solution so far.

Mingming

Re: [RFC][PATCH 0/2]Extend ext3 filesystem limit from 8TB to 16TB

[Nathan Scott]

Hi Andi,

On Fri, Mar 31, 2006 at 03:33:24PM +0200, Andi Kleen wrote:
> Mingming Cao <cmm@us.ibm.com> writes:
> > > Have you done tests _near_ 8TB with a 32-bit machine, even without these
> > > patches?
> > No I haven't. The >8TB right now is attached to a 64 bit machine, but we
> > should able to move it to a 32 bit machine.
> 
> If you use XFS or JFS as backing fs you can use a holey loop device
> to simulate it.  When I tried this last time JFS worked better for me.
> XFS doesn't seem to like that many extents as will be created by 
> mkfs.ext2.

Mainline has this issue resolved now (very recently, post-.16).

This (loopback on a local file) technique will get you up to 16TB
for 32 bit platforms, where you hit the unsigned long page->index
limit (but sounds like thats fine for the testing you're doing).

A related technique we've used in the past in testing XFS on large
devices (we've successfully tested in petabyte ranges using this,
on 64 bit systems of course) is to write a tool that modifies the
values in the ondisk data structures managing the "lower" areas of
the device to say "all the space here is used", which then forces
new allocations to be done in the "higher" parts of the device
address space.  Testing then follows this recipe: mkfs-on-loop,
then run the tool, then mount, then run the usual test suites ...
perhaps thats useful here too (I dunno if the ext2/3 format lends
itself to that or not).

cheers.

-- 
Nathan

Re: [RFC][PATCH 0/2]Extend ext3 filesystem limit from 8TB to 16TB

[Mingming Cao]

On Fri, 2006-03-31 at 14:42 -0800, Mingming Cao wrote: 
> On Wed, 2006-03-29 at 17:54 -0800, Andrew Morton wrote:
> > Mingming Cao <cmm@us.ibm.com> wrote:
> > >
> > > The things need to be done to complete this work is the issue with
> > >  current percpu counter, which could not handle u32 type count well. 
> > 
> > I'm surprised there's much of a problem here.  It is a 32-bit value, so it
> > should mainly be a matter of treating the return value from
> > percpu_counter_read() as unsigned long.
> > 
> > However a stickier problem is when dealing with a filesystem which has,
> > say, 0xffff_ff00 blocks.  Because percpu counters are approximate, and a
> > counter which really has a value of 0xffff_feee might return 0x00000123. 
> > What do we do then?
> > 
> 
> Hmm... I think we had this issue already even with today's 2**31 ext3.
> Since ext2/3 always use percpu_counter_read_positive() to get the total
> number of free blocks, so if the real free blocks is 0x0fff_feee, and
> the approximate value from the percpu counter is 0xf000_0123, the
> percpu_counter_read_positive() will return back 0x0000123.
> 

In fact, even worse, percpu_counter_read_positive() always return 1 if
the value is negative (>2**31). So this is not suitable for ext3's
2**32 block numbers. I think we should use percpu_counter_read() and
cast it to unsigned long for ext3's free blocks (and probably for free
inodes also).

Think over again, I think we could fix the possible overflow issue
(caused by approximate value) Andrew was concerned about: Before update
the global counter, check to see if we are trying to increase the global
counter but get a smaller value, or we are trying to decrease the global
counter but instead get a larger value. If any of them is true, we
should not update the global counter at that moment. This check only
happens when try to update the global counter from an local counter, and
probably not needed for those who don't care about unsigned long
counters. This way we shall not get ridiculous values from the counter. 

Comments?

Mingming

Re: [Ext2-devel] Re: [RFC][PATCH 0/2]Extend ext3 filesystem limit from 8TB to 16TB

[Andrew Morton]

Laurent Vivier <Laurent.Vivier@bull.net> wrote:
>
> Does the attached patch look like the thing you though about ?

I guess so.  But it'll need a lot of performance testing on big SMP
to work out what the impact is.

Re: [Ext2-devel] Re: [RFC][PATCH 0/2]Extend ext3 filesystem limit from 8TB to 16TB

[Laurent Vivier]

[-- Attachment #1: Type: text/plain, Size: 1155 bytes --]

Le jeu 30/03/2006 à 03:54, Andrew Morton a écrit :
> Mingming Cao <cmm@us.ibm.com> wrote:
> >
> > The things need to be done to complete this work is the issue with
> >  current percpu counter, which could not handle u32 type count well. 
> 
> I'm surprised there's much of a problem here.  It is a 32-bit value, so it
> should mainly be a matter of treating the return value from
> percpu_counter_read() as unsigned long.
> 
> However a stickier problem is when dealing with a filesystem which has,
> say, 0xffff_ff00 blocks.  Because percpu counters are approximate, and a
> counter which really has a value of 0xffff_feee might return 0x00000123. 
> What do we do then?
> 
> Of course the simple option is to nuke the percpu counters in ext3 and use
> atomic_long_t (which is signed, so appropriate treat-it-as-unsigned code
> would be needed).  I doubt if the percpu counters in ext3 are gaining us
> much.

I tried to make something in this way.
Does the attached patch look like the thing you though about ?

Regards,
Laurent
-- 
Laurent Vivier
Bull, Architect of an Open World (TM)
http://www.bullopensource.org/ext4

[-- Attachment #2: cpu_counter --]
[-- Type: text/x-patch, Size: 6792 bytes --]

Index: linux-2.6.16-lv/fs/ext3/balloc.c
===================================================================
--- linux-2.6.16-lv.orig/fs/ext3/balloc.c	2006-04-07 16:27:11.000000000 +0200
+++ linux-2.6.16-lv/fs/ext3/balloc.c	2006-04-07 17:05:28.000000000 +0200
@@ -471,7 +471,7 @@ do_more:
 		cpu_to_le16(le16_to_cpu(desc->bg_free_blocks_count) +
 			group_freed);
 	spin_unlock(sb_bgl_lock(sbi, block_group));
-	percpu_counter_mod(&sbi->s_freeblocks_counter, count);
+	atomic_long_set(&sbi->s_freeblocks_counter, count);
 
 	/* We dirtied the bitmap block */
 	BUFFER_TRACE(bitmap_bh, "dirtied bitmap block");
@@ -1129,7 +1129,7 @@ static int ext3_has_free_blocks(struct e
 {
 	sector_t free_blocks, root_blocks;
 
-	free_blocks = percpu_counter_read_positive(&sbi->s_freeblocks_counter);
+	free_blocks = (sector_t)atomic_long_read(&sbi->s_freeblocks_counter);
 	root_blocks = le32_to_cpu(sbi->s_es->s_r_blocks_count);
 	if (free_blocks < root_blocks + 1 && !capable(CAP_SYS_RESOURCE) &&
 		sbi->s_resuid != current->fsuid &&
@@ -1381,7 +1381,7 @@ allocated:
 	gdp->bg_free_blocks_count =
 			cpu_to_le16(le16_to_cpu(gdp->bg_free_blocks_count) - 1);
 	spin_unlock(sb_bgl_lock(sbi, group_no));
-	percpu_counter_mod(&sbi->s_freeblocks_counter, -1);
+	atomic_long_dec(&sbi->s_freeblocks_counter);
 
 	BUFFER_TRACE(gdp_bh, "journal_dirty_metadata for group descriptor");
 	err = ext3_journal_dirty_metadata(handle, gdp_bh);
Index: linux-2.6.16-lv/include/linux/ext3_fs_sb.h
===================================================================
--- linux-2.6.16-lv.orig/include/linux/ext3_fs_sb.h	2006-04-07 16:27:11.000000000 +0200
+++ linux-2.6.16-lv/include/linux/ext3_fs_sb.h	2006-04-07 17:01:23.000000000 +0200
@@ -20,7 +20,6 @@
 #include <linux/timer.h>
 #include <linux/wait.h>
 #include <linux/blockgroup_lock.h>
-#include <linux/percpu_counter.h>
 #endif
 #include <linux/rbtree.h>
 
@@ -54,9 +53,9 @@ struct ext3_sb_info {
 	u32 s_next_generation;
 	u32 s_hash_seed[4];
 	int s_def_hash_version;
-	struct percpu_counter s_freeblocks_counter;
-	struct percpu_counter s_freeinodes_counter;
-	struct percpu_counter s_dirs_counter;
+	atomic_long_t s_freeblocks_counter;
+	atomic_long_t s_freeinodes_counter;
+	atomic_long_t s_dirs_counter;
 	struct blockgroup_lock s_blockgroup_lock;
 
 	/* root of the per fs reservation window tree */
Index: linux-2.6.16-lv/fs/ext3/super.c
===================================================================
--- linux-2.6.16-lv.orig/fs/ext3/super.c	2006-04-07 16:27:11.000000000 +0200
+++ linux-2.6.16-lv/fs/ext3/super.c	2006-04-07 17:14:22.000000000 +0200
@@ -404,9 +404,6 @@ static void ext3_put_super (struct super
 	for (i = 0; i < sbi->s_gdb_count; i++)
 		brelse(sbi->s_group_desc[i]);
 	kfree(sbi->s_group_desc);
-	percpu_counter_destroy(&sbi->s_freeblocks_counter);
-	percpu_counter_destroy(&sbi->s_freeinodes_counter);
-	percpu_counter_destroy(&sbi->s_dirs_counter);
 	brelse(sbi->s_sbh);
 #ifdef CONFIG_QUOTA
 	for (i = 0; i < MAXQUOTAS; i++)
@@ -1580,9 +1577,9 @@ static int ext3_fill_super (struct super
 		goto failed_mount;
 	}
 
-	percpu_counter_init(&sbi->s_freeblocks_counter);
-	percpu_counter_init(&sbi->s_freeinodes_counter);
-	percpu_counter_init(&sbi->s_dirs_counter);
+	atomic_long_set(&sbi->s_freeblocks_counter, 0);
+	atomic_long_set(&sbi->s_freeinodes_counter, 0);
+	atomic_long_set(&sbi->s_dirs_counter, 0);
 	bgl_lock_init(&sbi->s_blockgroup_lock);
 
 	for (i = 0; i < db_count; i++) {
@@ -1730,11 +1727,11 @@ static int ext3_fill_super (struct super
 		test_opt(sb,DATA_FLAGS) == EXT3_MOUNT_ORDERED_DATA ? "ordered":
 		"writeback");
 
-	percpu_counter_mod(&sbi->s_freeblocks_counter,
+	atomic_long_set(&sbi->s_freeblocks_counter,
 		ext3_count_free_blocks(sb));
-	percpu_counter_mod(&sbi->s_freeinodes_counter,
+	atomic_long_set(&sbi->s_freeinodes_counter,
 		ext3_count_free_inodes(sb));
-	percpu_counter_mod(&sbi->s_dirs_counter,
+	atomic_long_set(&sbi->s_dirs_counter,
 		ext3_count_dirs(sb));
 
 	lock_kernel();
Index: linux-2.6.16-lv/fs/ext3/resize.c
===================================================================
--- linux-2.6.16-lv.orig/fs/ext3/resize.c	2006-04-07 16:27:11.000000000 +0200
+++ linux-2.6.16-lv/fs/ext3/resize.c	2006-04-07 17:12:13.000000000 +0200
@@ -871,9 +871,9 @@ int ext3_group_add(struct super_block *s
 		input->reserved_blocks);
 
 	/* Update the free space counts */
-	percpu_counter_mod(&sbi->s_freeblocks_counter,
+	atomic_long_set(&sbi->s_freeblocks_counter,
 			   input->free_blocks_count);
-	percpu_counter_mod(&sbi->s_freeinodes_counter,
+	atomic_long_set(&sbi->s_freeinodes_counter,
 			   EXT3_INODES_PER_GROUP(sb));
 
 	ext3_journal_dirty_metadata(handle, sbi->s_sbh);
Index: linux-2.6.16-lv/fs/ext3/ialloc.c
===================================================================
--- linux-2.6.16-lv.orig/fs/ext3/ialloc.c	2006-04-07 16:27:11.000000000 +0200
+++ linux-2.6.16-lv/fs/ext3/ialloc.c	2006-04-07 17:09:54.000000000 +0200
@@ -170,9 +170,9 @@ void ext3_free_inode (handle_t *handle, 
 				gdp->bg_used_dirs_count = cpu_to_le16(
 				  le16_to_cpu(gdp->bg_used_dirs_count) - 1);
 			spin_unlock(sb_bgl_lock(sbi, block_group));
-			percpu_counter_inc(&sbi->s_freeinodes_counter);
+			atomic_long_inc(&sbi->s_freeinodes_counter);
 			if (is_directory)
-				percpu_counter_dec(&sbi->s_dirs_counter);
+				atomic_long_dec(&sbi->s_dirs_counter);
 
 		}
 		BUFFER_TRACE(bh2, "call ext3_journal_dirty_metadata");
@@ -207,7 +207,7 @@ static int find_group_dir(struct super_b
 	struct buffer_head *bh;
 	int group, best_group = -1;
 
-	freei = percpu_counter_read_positive(&EXT3_SB(sb)->s_freeinodes_counter);
+	freei = (sector_t)atomic_long_read(&EXT3_SB(sb)->s_freeinodes_counter);
 	avefreei = freei / ngroups;
 
 	for (group = 0; group < ngroups; group++) {
@@ -269,11 +269,11 @@ static int find_group_orlov(struct super
 	struct ext3_group_desc *desc;
 	struct buffer_head *bh;
 
-	freei = percpu_counter_read_positive(&sbi->s_freeinodes_counter);
+	freei = (sector_t)atomic_long_read(&sbi->s_freeinodes_counter);
 	avefreei = freei / ngroups;
-	freeb = percpu_counter_read_positive(&sbi->s_freeblocks_counter);
+	freeb = (sector_t)atomic_long_read(&sbi->s_freeblocks_counter);
 	avefreeb = freeb / ngroups;
-	ndirs = percpu_counter_read_positive(&sbi->s_dirs_counter);
+	ndirs = (sector_t)atomic_long_read(&sbi->s_dirs_counter);
 
 	if ((parent == sb->s_root->d_inode) ||
 	    (EXT3_I(parent)->i_flags & EXT3_TOPDIR_FL)) {
@@ -539,9 +539,9 @@ got:
 	err = ext3_journal_dirty_metadata(handle, bh2);
 	if (err) goto fail;
 
-	percpu_counter_dec(&sbi->s_freeinodes_counter);
+	atomic_long_dec(&sbi->s_freeinodes_counter);
 	if (S_ISDIR(mode))
-		percpu_counter_inc(&sbi->s_dirs_counter);
+		atomic_long_inc(&sbi->s_dirs_counter);
 	sb->s_dirt = 1;
 
 	inode->i_uid = current->fsuid;

Re: [Ext2-devel] Re: [RFC][PATCH 0/2]Extend ext3 filesystem limit from 8TB to 16TB

[Mingming Cao]

On Mon, 2006-04-10 at 11:11 +0200, Laurent Vivier wrote:
> Le jeu 30/03/2006 à 03:54, Andrew Morton a écrit :
> > Mingming Cao <cmm@us.ibm.com> wrote:
> > >
> > > The things need to be done to complete this work is the issue with
> > >  current percpu counter, which could not handle u32 type count well. 
> > 
> > I'm surprised there's much of a problem here.  It is a 32-bit value, so it
> > should mainly be a matter of treating the return value from
> > percpu_counter_read() as unsigned long.
> > 
> > However a stickier problem is when dealing with a filesystem which has,
> > say, 0xffff_ff00 blocks.  Because percpu counters are approximate, and a
> > counter which really has a value of 0xffff_feee might return 0x00000123. 
> > What do we do then?
> > 
> > Of course the simple option is to nuke the percpu counters in ext3 and use
> > atomic_long_t (which is signed, so appropriate treat-it-as-unsigned code
> > would be needed).  I doubt if the percpu counters in ext3 are gaining us
> > much.
> 
> I tried to make something in this way.
> Does the attached patch look like the thing you though about ?
> 

I tried the other way -- I am trying to keep the percpu counter in use
in ext2/3 as much as possible.  I proposed a fix for percpu counter to
deal with the possible "overflow" (i.e, a counter really has a value of
0xfff_feee and after updating one local counter it truens 0x00000123).
Will send the proposed patch out for review and comments soon.

Mingming

-
To unsubscribe from this list: send the line "unsubscribe linux-fsdevel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [Ext2-devel] Re: [RFC][PATCH 0/2]Extend ext3 filesystem limit from 8TB to 16TB

[Mingming Cao]

On Mon, 2006-04-10 at 09:57 -0700, Mingming Cao wrote:
> On Mon, 2006-04-10 at 11:11 +0200, Laurent Vivier wrote:
> > Le jeu 30/03/2006 à 03:54, Andrew Morton a écrit :
> > > Mingming Cao <cmm@us.ibm.com> wrote:
> > > >
> > > > The things need to be done to complete this work is the issue with
> > > >  current percpu counter, which could not handle u32 type count well. 
> > > 
> > > I'm surprised there's much of a problem here.  It is a 32-bit value, so it
> > > should mainly be a matter of treating the return value from
> > > percpu_counter_read() as unsigned long.
> > > 
> > > However a stickier problem is when dealing with a filesystem which has,
> > > say, 0xffff_ff00 blocks.  Because percpu counters are approximate, and a
> > > counter which really has a value of 0xffff_feee might return 0x00000123. 
> > > What do we do then?
> > > 
> > > Of course the simple option is to nuke the percpu counters in ext3 and use
> > > atomic_long_t (which is signed, so appropriate treat-it-as-unsigned code
> > > would be needed).  I doubt if the percpu counters in ext3 are gaining us
> > > much.
> > 
> > I tried to make something in this way.
> > Does the attached patch look like the thing you though about ?
> > 
> 
Hi Laurent,

Just looked at your patch, shouldn't we use atomic_long_add() instead of
atomic_long_set() to replace percpu_counter_mod()?

> I tried the other way -- I am trying to keep the percpu counter in use
> in ext2/3 as much as possible.  I proposed a fix for percpu counter to
> deal with the possible "overflow" (i.e, a counter really has a value of
> 0xfff_feee and after updating one local counter it truens 0x00000123).
> Will send the proposed patch out for review and comments soon.
> 

Anyway, I am not against the atomic way. Just thought there must be
reasons where we use percpu counters -- the cache pollution on smp
machine is certainly a concern if we use atomic instead, so I  tried to
fix percpu counter first.

I think my fix for percpu counter should work, and the changes doesn't
affect other users of current percpu counters(vfs and network).  Kiran,
Andrew, please review it (posted in another seperate thread). If not,
then I guess we have to use atomic counter -- this is performance vs
capacity kind of trade off.

But both methods don't support 64 bit ext3 block number on 32 bit
machine...I am not happy with this but can't think of a way to fix this
without taking a global lock:(


Mingming
> -------------------------------------------------------
> This SF.Net email is sponsored by xPML, a groundbreaking scripting language
> that extends applications into web and mobile media. Attend the live webcast
> and join the prime developer group breaking into this new coding territory!
> http://sel.as-us.falkag.net/sel?cmd=lnk&kid\x110944&bid$1720&dat\x121642
> _______________________________________________
> Ext2-devel mailing list
> Ext2-devel@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/ext2-devel

-
To unsubscribe from this list: send the line "unsubscribe linux-fsdevel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Re: [RFC][PATCH 0/2]Extend ext3 filesystem limit from 8TB to 16TB

[Laurent Vivier]

[-- Attachment #1: Type: text/plain, Size: 2038 bytes --]

Le lun 10/04/2006 à 21:06, Mingming Cao a écrit :
> On Mon, 2006-04-10 at 09:57 -0700, Mingming Cao wrote:
> > On Mon, 2006-04-10 at 11:11 +0200, Laurent Vivier wrote:
[...]
> Hi Laurent,
> 
> Just looked at your patch, shouldn't we use atomic_long_add() instead of
> atomic_long_set() to replace percpu_counter_mod()?

Yes, thank you.

> > I tried the other way -- I am trying to keep the percpu counter in use
> > in ext2/3 as much as possible.  I proposed a fix for percpu counter to
> > deal with the possible "overflow" (i.e, a counter really has a value of
> > 0xfff_feee and after updating one local counter it truens 0x00000123).
> > Will send the proposed patch out for review and comments soon.
> > 
> 
> Anyway, I am not against the atomic way. Just thought there must be
> reasons where we use percpu counters -- the cache pollution on smp
> machine is certainly a concern if we use atomic instead, so I  tried to
> fix percpu counter first.
> 
> I think my fix for percpu counter should work, and the changes doesn't
> affect other users of current percpu counters(vfs and network).  Kiran,
> Andrew, please review it (posted in another seperate thread). If not,
> then I guess we have to use atomic counter -- this is performance vs
> capacity kind of trade off.

I made some tests with iozone on 2 CPU hyperthreaded computer (= 4 CPUs,
Bull Express 5800 120 Lh), and it seems atomic_t is faster than
"percpu_counter". I'll try to make some tests on IBM x440 (8 CPUs, 16 if
hyperthreaded) with iozone and sysbench.
Moreover, I think percpu_counter uses a lot of memory...

> But both methods don't support 64 bit ext3 block number on 32 bit
> machine...I am not happy with this but can't think of a way to fix this
> without taking a global lock:(

Anyway, wa can't have a 64bit addressing space on a 32bit machine, so I
think, for the moment, it's not a problem.

Regards,
Laurent
-- 
Laurent Vivier
Bull, Architect of an Open World (TM)
http://www.bullopensource.org/ext4

[-- Attachment #2: Ceci est une partie de message numériquement signée. --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: [Ext2-devel] Re: [RFC][PATCH 0/2]Extend ext3 filesystem limit from 8TB to 16TB

[Laurent Vivier]

[-- Attachment #1: Type: text/plain, Size: 816 bytes --]

Le lun 10/04/2006 à 10:24, Andrew Morton a écrit :
> Laurent Vivier <Laurent.Vivier@bull.net> wrote:
> >
> > Does the attached patch look like the thing you though about ?
> 
> I guess so.  But it'll need a lot of performance testing on big SMP
> to work out what the impact is.

I made some tests with dbench:

IBM x440: 8 CPUs hyperthreaded = 16 CPUs (Xeon at 1.4 Ghz)

with percpu_counter:

        Throughput 188.365 MB/sec 16 procs
        Throughput 226.164 MB/sec 32 procs
        Throughput 142.913 MB/sec 64 procs

with atomic_long_t:

        Throughput 194.385 MB/sec 16 procs
        Throughput 237.273 MB/sec 32 procs
        Throughput 160.751 MB/sec 64 procs

Regards,
Laurent
-- 
Laurent Vivier
Bull, Architect of an Open World (TM)
http://www.bullopensource.org/ext4

[-- Attachment #2: Ceci est une partie de message numériquement signée. --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: [Ext2-devel] Re: [RFC][PATCH 0/2]Extend ext3 filesystem limit from 8TB to 16TB

[Ravikiran G Thirumalai]

Hi Laurent,

On Tue, Apr 11, 2006 at 09:07:39AM +0200, Laurent Vivier wrote:
> ...  
> I made some tests with iozone on 2 CPU hyperthreaded computer (= 4 CPUs,
> Bull Express 5800 120 Lh), and it seems atomic_t is faster than
> "percpu_counter". I'll try to make some tests on IBM x440 (8 CPUs, 16 if
> hyperthreaded) with iozone and sysbench.
> Moreover, I think percpu_counter uses a lot of memory...

Was this just one iozone thread doing io?  What was the performance
difference?  Please let me know what kind of test you are doing, and I can 
run the same on an IBM x460 with 16 cores here.

Thanks,
Kiran

Re: [Ext2-devel] Re: [RFC][PATCH 0/2]Extend ext3 filesystem limit from 8TB to 16TB

[Ravikiran G Thirumalai]

On Thu, Apr 13, 2006 at 05:26:39PM +0200, Laurent Vivier wrote:
> Le lun 10/04/2006 à 10:24, Andrew Morton a écrit :
> > Laurent Vivier <Laurent.Vivier@bull.net> wrote:
> > >
> > > Does the attached patch look like the thing you though about ?
> > 
> > I guess so.  But it'll need a lot of performance testing on big SMP
> > to work out what the impact is.
> 
> I made some tests with dbench:
> 
> IBM x440: 8 CPUs hyperthreaded = 16 CPUs (Xeon at 1.4 Ghz)
> 

I ran the same tests on a 16 core EM64T box very similar to the one you ran
dbench on :). Dbench results on ext3 varies quite a bit.  I couldn't get 
to a statistically significant conclusion  For eg,

With atomic counters, 32 clients, 3 runs
Throughput 187.712 MB/sec 32 procs
Throughput 197.059 MB/sec 32 procs
Throughput 203.522 MB/sec 32 procs

Without atomic counters (per-cpu counters), 32 clients, 3 runs
Throughput 228.805 MB/sec 32 procs
Throughput 155.831 MB/sec 32 procs
Throughput 134.777 MB/sec 32 procs

The oprofile profiles for the atomic counter case looks like this:

CPU: P4 / Xeon with 2 hyper-threads, speed 3002.77 MHz (estimated)
Counted GLOBAL_POWER_EVENTS events (time during which processor is not
stopped) with a unit mask of 0x01 (mandatory) count 100000
samples  %        app name                 symbol name
180505286 57.7844  vmlinux-t                poll_idle
51944524 16.6288  vmlinux-t                ext3_test_allocatable
43648955 13.9731  vmlinux-t                bitmap_search_next_usable_block
2892251   0.9259  vmlinux-t                copy_user_generic
2099969   0.6723  vmlinux-t                do_get_write_access
1459523   0.4672  vmlinux-t                journal_dirty_metadata
1393413   0.4461  vmlinux-t                journal_stop

So the atomic counters in question are not even hotspots on this workload,
so IMHO, dbench cannot be used to come to any conclusion regarding per-cpu
counters vs atomics here.

Thanks,
Kiran

-
To unsubscribe from this list: send the line "unsubscribe linux-fsdevel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [Ext2-devel] Re: [RFC][PATCH 0/2]Extend ext3 filesystem limit from 8TB to 16TB

[Arjan van de Ven]

On Mon, 2006-04-17 at 14:07 -0700, Ravikiran G Thirumalai wrote:
> 
> 
> I ran the same tests on a 16 core EM64T box very similar to the one
> you ran
> dbench on :). Dbench results on ext3 varies quite a bit.  I couldn't
> get 
> to a statistically significant conclusion  For eg,


dbench is not a good performance benchmark. At all. Don't use it for
that ;)

Re: [Ext2-devel] Re: [RFC][PATCH 0/2]Extend ext3 filesystem limit from 8TB to 16TB

[Ravikiran G Thirumalai]

On Mon, Apr 17, 2006 at 11:09:36PM +0200, Arjan van de Ven wrote:
> On Mon, 2006-04-17 at 14:07 -0700, Ravikiran G Thirumalai wrote:
> > 
> > 
> > I ran the same tests on a 16 core EM64T box very similar to the one
> > you ran
> > dbench on :). Dbench results on ext3 varies quite a bit.  I couldn't
> > get 
> > to a statistically significant conclusion  For eg,
> 
> 
> dbench is not a good performance benchmark. At all. Don't use it for
> that ;)

Agreed. (I did not mean to use it in the first place :).  I was just trying 
to verify the benchmark results posted earlier)

Thanks,
Kiran

Re: Re: [RFC][PATCH 0/2]Extend ext3 filesystem limit from 8TB to 16TB

[Laurent Vivier]

[-- Attachment #1: Type: text/plain, Size: 934 bytes --]

Le lun 17/04/2006 à 23:32, Ravikiran G Thirumalai a écrit :
> On Mon, Apr 17, 2006 at 11:09:36PM +0200, Arjan van de Ven wrote:
> > On Mon, 2006-04-17 at 14:07 -0700, Ravikiran G Thirumalai wrote:
> > > 
> > > 
> > > I ran the same tests on a 16 core EM64T box very similar to the one
> > > you ran
> > > dbench on :). Dbench results on ext3 varies quite a bit.  I couldn't
> > > get 
> > > to a statistically significant conclusion  For eg,
> > 
> > 
> > dbench is not a good performance benchmark. At all. Don't use it for
> > that ;)
> 
> Agreed. (I did not mean to use it in the first place :).  I was just trying 
> to verify the benchmark results posted earlier)
> 
> Thanks,
> Kiran

What is the good performance benchmark to know if we should use atomic_t
instead of percpu_counter ?

Regards,
Laurent
-- 
Laurent Vivier
Bull, Architect of an Open World (TM)
http://www.bullopensource.org/ext4

[-- Attachment #2: Ceci est une partie de message numériquement signée. --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: [Ext2-devel] Re: [RFC][PATCH 0/2]Extend ext3 filesystem limit from 8TB to 16TB

[Arjan van de Ven]

On Tue, 2006-04-18 at 09:14 +0200, Laurent Vivier wrote:
> Le lun 17/04/2006 à 23:32, Ravikiran G Thirumalai a écrit :
> > On Mon, Apr 17, 2006 at 11:09:36PM +0200, Arjan van de Ven wrote:
> > > On Mon, 2006-04-17 at 14:07 -0700, Ravikiran G Thirumalai wrote:
> > > > 
> > > > 
> > > > I ran the same tests on a 16 core EM64T box very similar to the one
> > > > you ran
> > > > dbench on :). Dbench results on ext3 varies quite a bit.  I couldn't
> > > > get 
> > > > to a statistically significant conclusion  For eg,
> > > 
> > > 
> > > dbench is not a good performance benchmark. At all. Don't use it for
> > > that ;)
> > 
> > Agreed. (I did not mean to use it in the first place :).  I was just trying 
> > to verify the benchmark results posted earlier)
> > 
> > Thanks,
> > Kiran
> 
> What is the good performance benchmark to know if we should use atomic_t
> instead of percpu_counter ?

you probably want something like postal/postmark instead or so (although
that's not ideal either), at least that's reproducable

-
To unsubscribe from this list: send the line "unsubscribe linux-fsdevel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [Ext2-devel] Re: [RFC][PATCH 0/2]Extend ext3 filesystem limit from 8TB to 16TB

[Laurent Vivier]

[-- Attachment #1: Type: text/plain, Size: 2751 bytes --]

Le mar 18/04/2006 à 09:30, Arjan van de Ven a écrit :
> On Tue, 2006-04-18 at 09:14 +0200, Laurent Vivier wrote:
> > Le lun 17/04/2006 à 23:32, Ravikiran G Thirumalai a écrit :
> > > On Mon, Apr 17, 2006 at 11:09:36PM +0200, Arjan van de Ven wrote:
> > > > On Mon, 2006-04-17 at 14:07 -0700, Ravikiran G Thirumalai wrote:
> > > > > 
> > > > > 
> > > > > I ran the same tests on a 16 core EM64T box very similar to the one
> > > > > you ran
> > > > > dbench on :). Dbench results on ext3 varies quite a bit.  I couldn't
> > > > > get 
> > > > > to a statistically significant conclusion  For eg,
> > > > 
> > > > 
> > > > dbench is not a good performance benchmark. At all. Don't use it for
> > > > that ;)
> > > 
> > > Agreed. (I did not mean to use it in the first place :).  I was just trying 
> > > to verify the benchmark results posted earlier)
> > > 
> > > Thanks,
> > > Kiran
> > 
> > What is the good performance benchmark to know if we should use atomic_t
> > instead of percpu_counter ?
> 
> you probably want something like postal/postmark instead or so (although
> that's not ideal either), at least that's reproducable

I made tests on same system (x440) with postmark-1.51 :

pm> set numbers 100000
pm> set transactions 250000
pm> run

With atomic_t:

Time:
        3761 seconds total
        2414 seconds of transactions (103 per second)

Files:
        225064 created (59 per second)
                Creation alone: 100000 files (87 per second)
                Mixed with transactions: 125064 files (51 per second)
        124961 read (51 per second)
        124895 appended (51 per second)
        225064 deleted (59 per second)
                Deletion alone: 100128 files (503 per second)
                Mixed with transactions: 124936 files (51 per second)

Data:
        731.14 megabytes read (199.07 kilobytes per second)
        1359.02 megabytes written (370.02 kilobytes per second)

With percpu_counter:

Time:
        3787 seconds total
        2422 seconds of transactions (103 per second)

Files:
        225064 created (59 per second)
                Creation alone: 100000 files (85 per second)
                Mixed with transactions: 125064 files (51 per second)
        124961 read (51 per second)
        124895 appended (51 per second)
        225064 deleted (59 per second)
                Deletion alone: 100128 files (503 per second)
                Mixed with transactions: 124936 files (51 per second)

Data:
        731.14 megabytes read (197.70 kilobytes per second)
        1359.02 megabytes written (367.48 kilobytes per second)

-- 
Laurent Vivier
Bull, Architect of an Open World (TM)
http://www.bullopensource.org/ext4

[-- Attachment #2: Ceci est une partie de message numériquement signée. --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: Re: [RFC][PATCH 0/2]Extend ext3 filesystem limit from 8TB to 16TB

[Laurent Vivier]

[-- Attachment #1: Type: text/plain, Size: 4300 bytes --]

Le mar 18/04/2006 à 09:30, Arjan van de Ven a écrit :
> On Tue, 2006-04-18 at 09:14 +0200, Laurent Vivier wrote:
> > Le lun 17/04/2006 à 23:32, Ravikiran G Thirumalai a écrit :
> > > On Mon, Apr 17, 2006 at 11:09:36PM +0200, Arjan van de Ven wrote:
> > > > On Mon, 2006-04-17 at 14:07 -0700, Ravikiran G Thirumalai wrote:
> > > > > 
> > > > > 
> > > > > I ran the same tests on a 16 core EM64T box very similar to the one
> > > > > you ran
> > > > > dbench on :). Dbench results on ext3 varies quite a bit.  I couldn't
> > > > > get 
> > > > > to a statistically significant conclusion  For eg,
> > > > 
> > > > 
> > > > dbench is not a good performance benchmark. At all. Don't use it for
> > > > that ;)
> > > 
> > > Agreed. (I did not mean to use it in the first place :).  I was just trying 
> > > to verify the benchmark results posted earlier)
> > > 
> > > Thanks,
> > > Kiran
> > 
> > What is the good performance benchmark to know if we should use atomic_t
> > instead of percpu_counter ?
> 
> you probably want something like postal/postmark instead or so (although
> that's not ideal either), at least that's reproducable

I made some tests with kernbench too:

***** With percpu_counter:

16 cpus found
Cleaning source tree...
Caching kernel source in ram...
No old config found, using defconfig
Making mrproper
Making defconfig...
Kernel 2.6.16
Performing 5 runs of
make -j 8
make -j 64
make -j

All data logged to kernbench.log
Warmup run...
Half load -j 8 run number 1...
Half load -j 8 run number 2...
Half load -j 8 run number 3...
Half load -j 8 run number 4...
Half load -j 8 run number 5...
Average Half load -j 8 Run (std deviation):
Elapsed Time 120.68 (0.425558)
User Time 583.488 (0.54099)
System Time 84.716 (0.345948)
Percent CPU 553 (2)
Context Switches 13146.4 (66.3272)
Sleeps 26998.2 (297.078)

Optimal load -j 64 run number 1...
Optimal load -j 64 run number 2...
Optimal load -j 64 run number 3...
Optimal load -j 64 run number 4...
Optimal load -j 64 run number 5...
Average Optimal load -j 64 Run (std deviation):
Elapsed Time 86.496 (0.335827)
User Time 809.699 (238.449)
System Time 103.137 (19.423)
Percent CPU 945.3 (413.544)
Context Switches 32549.5 (20471.2)
Sleeps 34308 (7795.17)

Maximal load -j run number 1...
Maximal load -j run number 2...
Maximal load -j run number 3...
Maximal load -j run number 4...
Maximal load -j run number 5...
Average Maximal load -j Run (std deviation):
Elapsed Time 86.47 (0.321636)
User Time 883.568 (219.647)
System Time 108.728 (17.597)
Percent CPU 1073.8 (381.226)
Context Switches 31920.4 (16443.3)
Sleeps 30472.5 (8402.01)

***** With atomic_long_t

16 cpus found
Cleaning source tree...
Caching kernel source in ram...
No old config found, using defconfig
Making mrproper
Making defconfig...
Kernel 2.6.16
Performing 5 runs of
make -j 8
make -j 64
make -j

All data logged to kernbench.log
Warmup run...
Half load -j 8 run number 1...
Half load -j 8 run number 2...
Half load -j 8 run number 3...
Half load -j 8 run number 4...
Half load -j 8 run number 5...
Average Half load -j 8 Run (std deviation):
Elapsed Time 120.468 (0.724134)
User Time 581.226 (0.497624)
System Time 84.358 (0.45417)
Percent CPU 551.8 (3.19374)
Context Switches 13085.6 (108.579)
Sleeps 26965.8 (189.384)

Optimal load -j 64 run number 1...
Optimal load -j 64 run number 2...
Optimal load -j 64 run number 3...
Optimal load -j 64 run number 4...
Optimal load -j 64 run number 5...
Average Optimal load -j 64 Run (std deviation):
Elapsed Time 86.25 (0.263439)
User Time 805.828 (236.752)
System Time 102.262 (18.8792)
Percent CPU 942.7 (412.059)
Context Switches 32339.7 (20299.6)
Sleeps 34301.9 (7741.15)

Maximal load -j run number 1...
Maximal load -j run number 2...
Maximal load -j run number 3...
Maximal load -j run number 4...
Maximal load -j run number 5...
Average Maximal load -j Run (std deviation):
Elapsed Time 85.868 (0.757905)
User Time 879.129 (218.053)
System Time 107.847 (17.2136)
Percent CPU 1072.73 (381.349)
Context Switches 31854.5 (16297.1)
Sleeps 30436.2 (8399.98)


Laurent
-- 
Laurent Vivier
Bull, Architect of an Open World (TM)
http://www.bullopensource.org/ext4

[-- Attachment #2: Ceci est une partie de message numériquement signée. --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: [Ext2-devel] Re: [RFC][PATCH 0/2]Extend ext3 filesystem limit from 8TB to 16TB

[Ravikiran G Thirumalai]

On Tue, Apr 18, 2006 at 12:57:00PM +0200, Laurent Vivier wrote:
> 
> I made tests on same system (x440) with postmark-1.51 :
> 
> pm> set numbers 100000
> pm> set transactions 250000
> pm> run
> 
> With atomic_t:
> 
> Time:
>         3761 seconds total
>         2414 seconds of transactions (103 per second)
> 
> Files:
>         225064 created (59 per second)
>                 Creation alone: 100000 files (87 per second)
>                 Mixed with transactions: 125064 files (51 per second)
>         124961 read (51 per second)
>         124895 appended (51 per second)
>         225064 deleted (59 per second)
>                 Deletion alone: 100128 files (503 per second)
>                 Mixed with transactions: 124936 files (51 per second)
> 
> Data:
>         731.14 megabytes read (199.07 kilobytes per second)
>         1359.02 megabytes written (370.02 kilobytes per second)
> 
> With percpu_counter:
> 
> Time:
>         3787 seconds total
>         2422 seconds of transactions (103 per second)
> 
> Files:
>         225064 created (59 per second)
>                 Creation alone: 100000 files (85 per second)
>                 Mixed with transactions: 125064 files (51 per second)
>         124961 read (51 per second)
>         124895 appended (51 per second)
>         225064 deleted (59 per second)
>                 Deletion alone: 100128 files (503 per second)
>                 Mixed with transactions: 124936 files (51 per second)
> 
> Data:
>         731.14 megabytes read (197.70 kilobytes per second)
>         1359.02 megabytes written (367.48 kilobytes per second)

Can we get oprofile output for these tests please?  It will give us a clue as
to how much of hot spots the ext3 atomic counters are with this benchmark.
Also, it will be nice to have results for 3-5 iterations of the test
to make sure we are looking at statistically significant numbers.

Thanks,
Kiran

Re: [Ext2-devel] Re: [RFC][PATCH 0/2]Extend ext3 filesystem limit from 8TB to 16TB

[Mingming Cao]

On Tue, 2006-04-18 at 09:30 +0200, Arjan van de Ven wrote:
> On Tue, 2006-04-18 at 09:14 +0200, Laurent Vivier wrote:
> > Le lun 17/04/2006 à 23:32, Ravikiran G Thirumalai a écrit :
> > > On Mon, Apr 17, 2006 at 11:09:36PM +0200, Arjan van de Ven wrote:
> > > > On Mon, 2006-04-17 at 14:07 -0700, Ravikiran G Thirumalai wrote:
> > > > > 
> > > > > 
> > > > > I ran the same tests on a 16 core EM64T box very similar to the one
> > > > > you ran
> > > > > dbench on :). Dbench results on ext3 varies quite a bit.  I couldn't
> > > > > get 
> > > > > to a statistically significant conclusion  For eg,
> > > > 
> > > > 
> > > > dbench is not a good performance benchmark. At all. Don't use it for
> > > > that ;)
> > > 
> > > Agreed. (I did not mean to use it in the first place :).  I was just trying 
> > > to verify the benchmark results posted earlier)
> > > 
> > > Thanks,
> > > Kiran
> > 
> > What is the good performance benchmark to know if we should use atomic_t
> > instead of percpu_counter ?
> 
> you probably want something like postal/postmark instead or so (although
> that's not ideal either), at least that's reproducable
> 
postmark is a single threaded benchmark.

The ext3 filesystem free blocks counter is mostly being updated at block
allocation and free code. So, a test with many many threads doing block
allocation/deallocation simultaneously will stress the free blocks
counter accounting better than a single threaded fs benchmark. After
all, the main reason we choose to use percpu counter for the free blocks
counter at the first place, I believe, was to support parallel block
allocation. 

I would suggest run tiobench with many threads (>256), or even better,
run tiobench with many dd tests at the background.


Mingming

-
To unsubscribe from this list: send the line "unsubscribe linux-fsdevel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [Ext2-devel] Re: [RFC][PATCH 0/2]Extend ext3 filesystem limit from 8TB to 16TB

[Laurent Vivier]

[-- Attachment #1.1: Type: text/plain, Size: 2305 bytes --]

Le mar 18/04/2006 à 23:01, Mingming Cao a écrit :
> On Tue, 2006-04-18 at 09:30 +0200, Arjan van de Ven wrote:
> > On Tue, 2006-04-18 at 09:14 +0200, Laurent Vivier wrote:
> > > Le lun 17/04/2006 à 23:32, Ravikiran G Thirumalai a écrit :
> > > > On Mon, Apr 17, 2006 at 11:09:36PM +0200, Arjan van de Ven wrote:
> > > > > On Mon, 2006-04-17 at 14:07 -0700, Ravikiran G Thirumalai wrote:
> > > > > > 
> > > > > > 
> > > > > > I ran the same tests on a 16 core EM64T box very similar to the one
> > > > > > you ran
> > > > > > dbench on :). Dbench results on ext3 varies quite a bit.  I couldn't
> > > > > > get 
> > > > > > to a statistically significant conclusion  For eg,
> > > > > 
> > > > > 
> > > > > dbench is not a good performance benchmark. At all. Don't use it for
> > > > > that ;)
> > > > 
> > > > Agreed. (I did not mean to use it in the first place :).  I was just trying 
> > > > to verify the benchmark results posted earlier)
> > > > 
> > > > Thanks,
> > > > Kiran
> > > 
> > > What is the good performance benchmark to know if we should use atomic_t
> > > instead of percpu_counter ?
> > 
> > you probably want something like postal/postmark instead or so (although
> > that's not ideal either), at least that's reproducable
> > 
> postmark is a single threaded benchmark.
> 
> The ext3 filesystem free blocks counter is mostly being updated at block
> allocation and free code. So, a test with many many threads doing block
> allocation/deallocation simultaneously will stress the free blocks
> counter accounting better than a single threaded fs benchmark. After
> all, the main reason we choose to use percpu counter for the free blocks
> counter at the first place, I believe, was to support parallel block
> allocation. 
> 
> I would suggest run tiobench with many threads (>256), or even better,
> run tiobench with many dd tests at the background.

You can find attached my results with tiobench (256 threads, always on
x440 with 8 CPUs hyperthreaded = 16).

But, as the results are very different, I think we can't really
conclude... in fact, I think atomic_t or percpu_counter have no impact
on the results.

Regards,
Laurent
-- 
Laurent Vivier
Bull, Architect of an Open World (TM)
http://www.bullopensource.org/ext4

[-- Attachment #1.2: tiobench.txt --]
[-- Type: text/plain, Size: 9165 bytes --]

tiobench.pl --size 16 --numruns 10 --threads 256

Unit information
================
File size = megabytes
Blk Size  = bytes
Rate      = megabytes per second
CPU%      = percentage of CPU used during the test
Latency   = milliseconds
Lat%      = percent of requests that took longer than X seconds
CPU Eff   = Rate divided by CPU% - throughput per cpu load

Sequential Reads
atomic_long_t                  16   4096  256  394.38 1540.%     2.822     5960.36   0.00000  0.00000    26
atomic_long_t                  16   4096  256  384.70 1547.%     2.087     6187.27   0.00000  0.00000    25
atomic_long_t                  16   4096  256  395.69 1548.%     2.001     5988.64   0.00000  0.00000    26
atomic_long_t                  16   4096  256  396.10 1544.%     1.987     5836.74   0.00000  0.00000    26
atomic_long_t                  16   4096  256  377.66 1551.%     2.435     6038.05   0.00000  0.00000    24
atomic_long_t                  16   4096  256  378.35 1550.%     2.145     6369.26   0.00000  0.00000    24
atomic_long_t                  16   4096  256  375.38 1545.%     2.236     6318.84   0.00000  0.00000    24
atomic_long_t                  16   4096  256  399.97 1546.%     2.021     5911.10   0.00000  0.00000    26
atomic_long_t                  16   4096  256  386.59 1542.%     2.045     5968.13   0.00000  0.00000    25
atomic_long_t                  16   4096  256  401.45 1547.%     2.000     5918.17   0.00000  0.00000    26
percpu_counter                 16   4096  256  396.22 1539.%     1.965     5984.66   0.00000  0.00000    26
percpu_counter                 16   4096  256  403.50 1547.%     2.373     5503.98   0.00000  0.00000    26
percpu_counter                 16   4096  256  388.54 1547.%     2.047     6100.05   0.00000  0.00000    25
percpu_counter                 16   4096  256  397.43 1540.%     2.096     6010.45   0.00000  0.00000    26
percpu_counter                 16   4096  256  398.81 1543.%     2.134     5677.78   0.00000  0.00000    26
percpu_counter                 16   4096  256  399.85 1548.%     1.980     5805.21   0.00000  0.00000    26
percpu_counter                 16   4096  256  394.70 1551.%     2.021     5960.13   0.00000  0.00000    25
percpu_counter                 16   4096  256  396.64 1543.%     2.132     5901.40   0.00000  0.00000    26
percpu_counter                 16   4096  256  390.70 1550.%     1.906     5972.05   0.00000  0.00000    25
percpu_counter                 16   4096  256  401.98 1547.%     2.049     5839.44   0.00000  0.00000    26


Random Reads
atomic_long_t                  16   4096  256  100.79 1230.%     0.351        3.82   0.00000  0.00000     8
atomic_long_t                  16   4096  256  112.61 1342.%     0.367       13.42   0.00000  0.00000     8
atomic_long_t                  16   4096  256  111.16 1354.%     0.503      450.22   0.00000  0.00000     8
atomic_long_t                  16   4096  256  112.79 1333.%     0.366        4.73   0.00000  0.00000     8
atomic_long_t                  16   4096  256  114.61 1322.%     0.375       12.74   0.00000  0.00000     9
atomic_long_t                  16   4096  256  111.18 1350.%     0.368       19.89   0.00000  0.00000     8
atomic_long_t                  16   4096  256  112.14 1344.%     0.395      143.76   0.00000  0.00000     8
atomic_long_t                  16   4096  256  112.41 1346.%     0.374       31.40   0.00000  0.00000     8
atomic_long_t                  16   4096  256  112.89 1353.%     0.372       12.64   0.00000  0.00000     8
atomic_long_t                  16   4096  256  112.40 1354.%     0.365       10.88   0.00000  0.00000     8
percpu_counter                 16   4096  256  112.68 1341.%     0.439      263.93   0.00000  0.00000     8
percpu_counter                 16   4096  256  109.77 1356.%     0.372       35.58   0.00000  0.00000     8
percpu_counter                 16   4096  256  114.18 1339.%     0.371       29.47   0.00000  0.00000     9
percpu_counter                 16   4096  256  112.61 1339.%     0.430      241.78   0.00000  0.00000     8
percpu_counter                 16   4096  256  111.09 1343.%     0.372       46.84   0.00000  0.00000     8
percpu_counter                 16   4096  256  111.98 1355.%     0.358       16.41   0.00000  0.00000     8
percpu_counter                 16   4096  256  112.55 1348.%     0.393      121.30   0.00000  0.00000     8
percpu_counter                 16   4096  256  114.84 1324.%     0.398      112.03   0.00000  0.00000     9
percpu_counter                 16   4096  256  111.45 1353.%     0.368       22.09   0.00000  0.00000     8
percpu_counter                 16   4096  256  112.21 1352.%     0.405      150.62   0.00000  0.00000     8

Sequential Writes
atomic_long_t                  16   4096  256   28.72 350.0%   103.526     3087.83   0.00000  0.00000     8
atomic_long_t                  16   4096  256   28.47 350.1%   107.563     4127.54   0.00000  0.00000     8
atomic_long_t                  16   4096  256   28.10 346.6%   108.709     2767.21   0.00000  0.00000     8
atomic_long_t                  16   4096  256   28.24 345.1%   106.619     3025.58   0.00000  0.00000     8
atomic_long_t                  16   4096  256   28.35 358.4%   110.779     3844.84   0.00000  0.00000     8
atomic_long_t                  16   4096  256   28.14 349.9%   109.956     3580.34   0.00000  0.00000     8
atomic_long_t                  16   4096  256   28.23 349.6%   110.011     2770.21   0.00000  0.00000     8
atomic_long_t                  16   4096  256   28.46 355.4%   108.701     2694.87   0.00000  0.00000     8
atomic_long_t                  16   4096  256   28.14 346.3%   108.114     3461.73   0.00000  0.00000     8
atomic_long_t                  16   4096  256   28.49 348.8%   109.625     3160.93   0.00000  0.00000     8
percpu_counter                 16   4096  256   27.92 344.3%   109.420     3497.14   0.00000  0.00000     8
percpu_counter                 16   4096  256   28.23 343.2%   110.146     3279.80   0.00000  0.00000     8
percpu_counter                 16   4096  256   28.35 345.6%   110.027     3527.68   0.00000  0.00000     8
percpu_counter                 16   4096  256   28.17 348.9%   107.143     3735.24   0.00000  0.00000     8
percpu_counter                 16   4096  256   28.07 333.8%   107.581     2442.35   0.00000  0.00000     8
percpu_counter                 16   4096  256   28.36 343.2%   106.625     2740.56   0.00000  0.00000     8
percpu_counter                 16   4096  256   28.33 343.6%   107.201     3029.49   0.00000  0.00000     8
percpu_counter                 16   4096  256   28.28 339.9%   107.849     3100.73   0.00000  0.00000     8
percpu_counter                 16   4096  256   28.27 344.8%   108.008     3753.06   0.00000  0.00000     8
percpu_counter                 16   4096  256   28.54 354.1%   108.254     2831.67   0.00000  0.00000     8

Random Writes
atomic_long_t                  16   4096  256    2.47 64.55%     5.690     1295.08   0.00000  0.00000     4
atomic_long_t                  16   4096  256    2.45 63.52%     6.021     1386.22   0.00000  0.00000     4
atomic_long_t                  16   4096  256    2.47 63.87%     5.621      912.91   0.00000  0.00000     4
atomic_long_t                  16   4096  256    2.45 64.34%     6.361     1444.26   0.00000  0.00000     4
atomic_long_t                  16   4096  256    2.47 64.04%     5.793     1307.02   0.00000  0.00000     4
atomic_long_t                  16   4096  256    2.47 64.14%     5.979     1690.00   0.00000  0.00000     4
atomic_long_t                  16   4096  256    2.49 64.63%     5.993     1820.59   0.00000  0.00000     4
atomic_long_t                  16   4096  256    2.48 64.98%     6.400     1829.93   0.00000  0.00000     4
atomic_long_t                  16   4096  256    2.44 64.05%     5.763     1631.14   0.00000  0.00000     4
atomic_long_t                  16   4096  256    2.53 66.19%     5.728      919.34   0.00000  0.00000     4
percpu_counter                 16   4096  256    2.43 63.69%     5.927      851.10   0.00000  0.00000     4
percpu_counter                 16   4096  256    2.42 62.73%     5.997     1371.13   0.00000  0.00000     4
percpu_counter                 16   4096  256    2.46 64.17%     6.223     1808.24   0.00000  0.00000     4
percpu_counter                 16   4096  256    2.46 64.04%     5.897     1410.98   0.00000  0.00000     4
percpu_counter                 16   4096  256    2.47 63.98%     5.829     1197.30   0.00000  0.00000     4
percpu_counter                 16   4096  256    2.48 64.15%     5.660     1079.85   0.00000  0.00000     4
percpu_counter                 16   4096  256    2.47 63.81%     6.041     1401.02   0.00000  0.00000     4
percpu_counter                 16   4096  256    2.48 64.88%     6.078     1218.58   0.00000  0.00000     4
percpu_counter                 16   4096  256    2.49 64.90%     6.148     1468.47   0.00000  0.00000     4
percpu_counter                 16   4096  256    2.47 65.64%     5.724     1148.28   0.00000  0.00000     4


[-- Attachment #2: Ceci est une partie de message numériquement signée. --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: Re: [RFC][PATCH 0/2]Extend ext3 filesystem limit from 8TB to 16TB

[Laurent Vivier]

[-- Attachment #1.1: Type: text/plain, Size: 3730 bytes --]

Le mar 18/04/2006 à 23:01, Mingming Cao a écrit :
> On Tue, 2006-04-18 at 09:30 +0200, Arjan van de Ven wrote:
> > On Tue, 2006-04-18 at 09:14 +0200, Laurent Vivier wrote:
> > > Le lun 17/04/2006 à 23:32, Ravikiran G Thirumalai a écrit :
> > > > On Mon, Apr 17, 2006 at 11:09:36PM +0200, Arjan van de Ven wrote:
> > > > > On Mon, 2006-04-17 at 14:07 -0700, Ravikiran G Thirumalai wrote:
> > > > > > 
> > > > > > 
> > > > > > I ran the same tests on a 16 core EM64T box very similar to the one
> > > > > > you ran
> > > > > > dbench on :). Dbench results on ext3 varies quite a bit.  I couldn't
> > > > > > get 
> > > > > > to a statistically significant conclusion  For eg,
> > > > > 
> > > > > 
> > > > > dbench is not a good performance benchmark. At all. Don't use it for
> > > > > that ;)
> > > > 
> > > > Agreed. (I did not mean to use it in the first place :).  I was just trying 
> > > > to verify the benchmark results posted earlier)
> > > > 
> > > > Thanks,
> > > > Kiran
> > > 
> > > What is the good performance benchmark to know if we should use atomic_t
> > > instead of percpu_counter ?
> > 
> > you probably want something like postal/postmark instead or so (although
> > that's not ideal either), at least that's reproducable
> > 
> postmark is a single threaded benchmark.
> 
> The ext3 filesystem free blocks counter is mostly being updated at block
> allocation and free code. So, a test with many many threads doing block
> allocation/deallocation simultaneously will stress the free blocks
> counter accounting better than a single threaded fs benchmark. After
> all, the main reason we choose to use percpu counter for the free blocks
> counter at the first place, I believe, was to support parallel block
> allocation. 
> 
> I would suggest run tiobench with many threads (>256), or even better,
> run tiobench with many dd tests at the background.

I made some measurements with tiobench and oprofile too:

** percpu_counter**

TIOBENCH:

Sequential Reads                 16   4096  256  527.08 1505.%     1.398     4487.81   0.00000  0.00000    35
Random Reads                     16   4096  256  128.02 1315.%     0.493      118.07   0.00000  0.00000    10
Sequential Writes                16   4096  256   28.46 367.9%   107.293     2999.53   0.00000  0.00000     8
Random Writes                    16   4096  256    2.49 58.87%     5.315     1421.55   0.00000  0.00000     4

OPROFILE:

4075      0.0017  vmlinux                  vmlinux                  percpu_counter_mod

[percpu_counter_inc() calls percpu_counter_mod()]
[percpu_counter_read_positive() is inlined]

** atomic_long_t **

TIOBENCH:

Sequential Reads               16   4096  256  519.81 1496.%     1.702     4239.57   0.00000  0.00000    35
Random Reads                   16   4096  256  123.99 1315.%     0.493      112.48   0.00000  0.00000     9
Sequential Writes              16   4096  256   28.51 360.8%   107.353     2971.17   0.00000  0.00000     8
Random Writes                  16   4096  256    2.49 58.90%     5.259     1243.42   0.00000  0.00000     4

OPROFILE:

352      1.5e-04  vmlinux                  vmlinux                  percpu_counter_mod

[we have some time in "percpu_counter_mod" because my root filesystem
uses unmodified ext2]

The functions added by my patch are following and as they are atomic
(one machine instruction) they are not measurable and don't appears in
oprofile.

atomic_long_add
atomic_long_read
atomic_long_set
atomic_long_inc

Any comments ?
(oprofile.out are in attachment)

Regards,
Laurent
-- 
Laurent Vivier
Bull, Architect of an Open World (TM)
http://www.bullopensource.org/ext4

[-- Attachment #1.2: oprofile-percpu_counter.txt --]
[-- Type: text/plain, Size: 133059 bytes --]

CPU: P4 / Xeon with 2 hyper-threads, speed 1397.42 MHz (estimated)
Counted GLOBAL_POWER_EVENTS events (time during which processor is not stopped) with a unit mask of 0x01 (mandatory) count 100000
samples  %        image name               app name                 symbol name
143450190 59.0871  libc-2.3.3.so            libc-2.3.3.so            __sleep
32300788 13.3047  tiotest                  tiotest                  start_proc
27613186 11.3739  vmlinux                  vmlinux                  poll_idle
24500184 10.0916  libc-2.3.3.so            libc-2.3.3.so            __i686.get_pc_thunk.bx
5182611   2.1347  tiotest                  tiotest                  anonymous symbol from section .plt
1337784   0.5510  oprofiled                oprofiled                (no symbols)
1024000   0.4218  vmlinux                  vmlinux                  _spin_lock_irqsave
940408    0.3874  vmlinux                  vmlinux                  _spin_lock
495195    0.2040  vmlinux                  vmlinux                  __blk_queue_bounce
406947    0.1676  vmlinux                  vmlinux                  __copy_to_user_ll
303681    0.1251  vmlinux                  vmlinux                  journal_dirty_metadata
299682    0.1234  vmlinux                  vmlinux                  do_get_write_access
268284    0.1105  vmlinux                  vmlinux                  __copy_from_user_ll
227950    0.0939  oprofile.ko              oprofile                 add_event_entry
227285    0.0936  oprofile.ko              oprofile                 lookup_dcookie
208188    0.0858  oprofile.ko              oprofile                 add_us_sample
182142    0.0750  vmlinux                  vmlinux                  get_offset_cyclone
181681    0.0748  vmlinux                  vmlinux                  find_vma
147865    0.0609  oprofile.ko              oprofile                 increment_tail
133056    0.0548  vmlinux                  vmlinux                  ext3_test_allocatable
125025    0.0515  oprofile.ko              oprofile                 sync_buffer
117348    0.0483  vmlinux                  vmlinux                  journal_add_journal_head
111204    0.0458  vmlinux                  vmlinux                  __wake_up_common
107606    0.0443  vmlinux                  vmlinux                  journal_get_undo_access
88727     0.0365  vmlinux                  vmlinux                  ext3_try_to_allocate
85912     0.0354  vmlinux                  vmlinux                  apic_timer_interrupt
73744     0.0304  oprofile.ko              oprofile                 add_sample_entry
68236     0.0281  vmlinux                  vmlinux                  lookup_bh_lru
61009     0.0251  vmlinux                  vmlinux                  system_call
57137     0.0235  vmlinux                  vmlinux                  journal_put_journal_head
56795     0.0234  oprofile.ko              oprofile                 add_sample
54246     0.0223  vmlinux                  vmlinux                  mark_offset_cyclone
52039     0.0214  vmlinux                  vmlinux                  schedule
43929     0.0181  vmlinux                  vmlinux                  mutex_debug_check_no_locks_freed
43351     0.0179  vmlinux                  vmlinux                  find_busiest_group
42619     0.0176  vmlinux                  vmlinux                  activate_page
41962     0.0173  vmlinux                  vmlinux                  wake_bit_function
38579     0.0159  vmlinux                  vmlinux                  do_gettimeofday
38404     0.0158  libc-2.3.3.so            libc-2.3.3.so            __GI___gettimeofday_internal
35807     0.0147  vmlinux                  vmlinux                  journal_dirty_data
35312     0.0145  vmlinux                  vmlinux                  journal_cancel_revoke
30979     0.0128  vmlinux                  vmlinux                  kmem_cache_free
29941     0.0123  vmlinux                  vmlinux                  journal_stop
27583     0.0114  vmlinux                  vmlinux                  __brelse
26958     0.0111  vmlinux                  vmlinux                  try_to_wake_up
26898     0.0111  vmlinux                  vmlinux                  dependent_sleeper
26427     0.0109  vmlinux                  vmlinux                  rebalance_tick
24755     0.0102  vmlinux                  vmlinux                  timer_interrupt
24664     0.0102  vmlinux                  vmlinux                  __wait_on_bit_lock
23547     0.0097  vmlinux                  vmlinux                  start_this_handle
22242     0.0092  vmlinux                  vmlinux                  mark_page_accessed
22134     0.0091  vmlinux                  vmlinux                  __switch_to
21906     0.0090  vmlinux                  vmlinux                  ext3_journal_start_sb
20392     0.0084  vmlinux                  vmlinux                  __wake_up_bit
20089     0.0083  vmlinux                  vmlinux                  ext3_get_inode_block
18450     0.0076  vmlinux                  vmlinux                  kmem_cache_alloc
18378     0.0076  vmlinux                  vmlinux                  kmap_atomic
18241     0.0075  vmlinux                  vmlinux                  _read_lock_irqsave
18143     0.0075  vmlinux                  vmlinux                  kunmap
17751     0.0073  vmlinux                  vmlinux                  do_flush_tlb_all
17377     0.0072  vmlinux                  vmlinux                  smp_invalidate_interrupt
17220     0.0071  vmlinux                  vmlinux                  __journal_file_buffer
16337     0.0067  vmlinux                  vmlinux                  generic_file_buffered_write
16290     0.0067  vmlinux                  vmlinux                  ext3_new_block
15741     0.0065  vmlinux                  vmlinux                  ext3_do_update_inode
14937     0.0062  vmlinux                  vmlinux                  unlock_buffer
14071     0.0058  oprofile.ko              oprofile                 add_cookie_switch
13901     0.0057  vmlinux                  vmlinux                  sys_gettimeofday
13895     0.0057  vmlinux                  vmlinux                  find_get_page
13718     0.0057  vmlinux                  vmlinux                  find_next_bit
13653     0.0056  vmlinux                  vmlinux                  sched_clock
13155     0.0054  libpthread-0.10.so       libpthread-0.10.so       __libc_write
12899     0.0053  vmlinux                  vmlinux                  prepare_to_wait_exclusive
12820     0.0053  vmlinux                  vmlinux                  __do_softirq
12714     0.0052  vmlinux                  vmlinux                  bit_waitqueue
12612     0.0052  vmlinux                  vmlinux                  scheduler_tick
12391     0.0051  libpthread-0.10.so       libpthread-0.10.so       __libc_read
12361     0.0051  vmlinux                  vmlinux                  sync_buffer
12356     0.0051  vmlinux                  vmlinux                  _write_lock_irqsave
12348     0.0051  vmlinux                  vmlinux                  __block_prepare_write
11853     0.0049  vmlinux                  vmlinux                  ext3_get_block_handle
11829     0.0049  vmlinux                  vmlinux                  page_address
11828     0.0049  vmlinux                  vmlinux                  __mark_inode_dirty
11808     0.0049  vmlinux                  vmlinux                  radix_tree_lookup
11247     0.0046  vmlinux                  vmlinux                  buffered_rmqueue
11235     0.0046  vmlinux                  vmlinux                  mod_page_state_offset
11131     0.0046  vmlinux                  vmlinux                  end_buffer_async_write
10935     0.0045  vmlinux                  vmlinux                  release_pages
10823     0.0045  vmlinux                  vmlinux                  run_timer_softirq
10633     0.0044  vmlinux                  vmlinux                  put_page
10472     0.0043  vmlinux                  vmlinux                  debug_mutex_free_waiter
10425     0.0043  vmlinux                  vmlinux                  ext3_mark_iloc_dirty
10327     0.0043  vmlinux                  vmlinux                  smp_apic_timer_interrupt
10151     0.0042  vmlinux                  vmlinux                  __mutex_unlock_slowpath
10104     0.0042  vmlinux                  vmlinux                  blk_rq_map_sg
9987      0.0041  vmlinux                  vmlinux                  fget_light
9812      0.0040  vmlinux                  vmlinux                  ext3_ordered_commit_write
9770      0.0040  vmlinux                  vmlinux                  __mutex_lock_slowpath
9681      0.0040  vmlinux                  vmlinux                  restore_nocheck
9656      0.0040  tiotest                  tiotest                  do_write_test
9474      0.0039  vmlinux                  vmlinux                  _spin_unlock
9388      0.0039  vmlinux                  vmlinux                  current_fs_time
9294      0.0038  vmlinux                  vmlinux                  __journal_temp_unlink_buffer
9102      0.0037  vmlinux                  vmlinux                  debug_mutex_set_owner
9066      0.0037  vmlinux                  vmlinux                  __rcu_pending
9046      0.0037  vmlinux                  vmlinux                  __generic_file_aio_write_nolock
9042      0.0037  vmlinux                  vmlinux                  __ext3_get_inode_loc
8977      0.0037  vmlinux                  vmlinux                  _spin_unlock_irqrestore
8922      0.0037  vmlinux                  vmlinux                  update_process_times
8826      0.0036  vmlinux                  vmlinux                  __make_request
8734      0.0036  vmlinux                  vmlinux                  fput
8728      0.0036  vmlinux                  vmlinux                  mempool_free
8691      0.0036  vmlinux                  vmlinux                  walk_page_buffers
8441      0.0035  vmlinux                  vmlinux                  _read_unlock_irq
8361      0.0034  vmlinux                  vmlinux                  hrtimer_run_queues
8281      0.0034  vmlinux                  vmlinux                  radix_tree_tag_set
8227      0.0034  vmlinux                  vmlinux                  account_user_time
8219      0.0034  vmlinux                  vmlinux                  run_posix_cpu_timers
8209      0.0034  vmlinux                  vmlinux                  __log_space_left
8000      0.0033  vmlinux                  vmlinux                  free_block
7988      0.0033  vmlinux                  vmlinux                  journal_start
7878      0.0032  vmlinux                  vmlinux                  __set_page_dirty_nobuffers
7801      0.0032  vmlinux                  vmlinux                  set_page_address
7726      0.0032  vmlinux                  vmlinux                  __find_get_block_slow
7689      0.0032  vmlinux                  vmlinux                  raise_softirq
7598      0.0031  vmlinux                  vmlinux                  ktime_get_ts
7539      0.0031  vmlinux                  vmlinux                  as_find_arq_hash
7514      0.0031  vmlinux                  vmlinux                  irq_exit
7449      0.0031  vmlinux                  vmlinux                  resched_task
7393      0.0030  vmlinux                  vmlinux                  _spin_unlock_irq
7350      0.0030  vmlinux                  vmlinux                  do_softirq
7331      0.0030  vmlinux                  vmlinux                  submit_bh
7296      0.0030  vmlinux                  vmlinux                  rcu_pending
7176      0.0030  vmlinux                  vmlinux                  rb_next
6936      0.0029  vmlinux                  vmlinux                  ext3_prepare_write
6735      0.0028  vmlinux                  vmlinux                  __ext3_journal_stop
6664      0.0027  vmlinux                  vmlinux                  do_generic_mapping_read
6656      0.0027  vmlinux                  vmlinux                  __rmqueue
6653      0.0027  vmlinux                  vmlinux                  _write_unlock_irqrestore
6518      0.0027  vmlinux                  vmlinux                  init_buffer_head
6453      0.0027  vmlinux                  vmlinux                  kmap_high
6378      0.0026  vmlinux                  vmlinux                  __lock_buffer
6376      0.0026  vmlinux                  vmlinux                  bio_put
6253      0.0026  vmlinux                  vmlinux                  ll_back_merge_fn
6243      0.0026  vmlinux                  vmlinux                  __find_get_block
6177      0.0025  vmlinux                  vmlinux                  enqueue_task
6066      0.0025  vmlinux                  vmlinux                  device_not_available
5999      0.0025  vmlinux                  vmlinux                  blk_recount_segments
5969      0.0025  vmlinux                  vmlinux                  bounce_end_io
5926      0.0024  vmlinux                  vmlinux                  dnotify_parent
5762      0.0024  vmlinux                  vmlinux                  irq_entries_start
5712      0.0024  vmlinux                  vmlinux                  free_pages_bulk
5707      0.0024  vmlinux                  vmlinux                  elv_merge
5646      0.0023  vmlinux                  vmlinux                  generic_file_aio_write
5562      0.0023  vmlinux                  vmlinux                  wake_idle
5519      0.0023  vmlinux                  vmlinux                  ktime_get_real
5508      0.0023  vmlinux                  vmlinux                  ext3_reserve_inode_write
5404      0.0022  vmlinux                  vmlinux                  idle_cpu
5125      0.0021  vmlinux                  vmlinux                  _write_unlock_irq
5081      0.0021  vmlinux                  vmlinux                  generic_make_request
5040      0.0021  vmlinux                  vmlinux                  do_sync_write
5027      0.0021  vmlinux                  vmlinux                  kunmap_high
4996      0.0021  vmlinux                  vmlinux                  profile_tick
4984      0.0021  vmlinux                  vmlinux                  journal_commit_transaction
4973      0.0020  vmlinux                  vmlinux                  vfs_write
4939      0.0020  vmlinux                  vmlinux                  load_balance
4932      0.0020  vmlinux                  vmlinux                  block_prepare_write
4842      0.0020  tiotest                  tiotest                  do_read_test
4737      0.0020  vmlinux                  vmlinux                  reap_alien
4723      0.0019  vmlinux                  vmlinux                  __block_commit_write
4720      0.0019  vmlinux                  vmlinux                  mempool_alloc
4607      0.0019  vmlinux                  vmlinux                  invalidate_interrupt
4562      0.0019  vmlinux                  vmlinux                  io_schedule
4541      0.0019  vmlinux                  vmlinux                  __block_write_full_page
4466      0.0018  vmlinux                  vmlinux                  ext3_try_to_allocate_with_rsv
4183      0.0017  vmlinux                  vmlinux                  free_hot_cold_page
4128      0.0017  vmlinux                  vmlinux                  flush_tlb_others
4121      0.0017  vmlinux                  vmlinux                  unlock_page
4075      0.0017  vmlinux                  vmlinux                  percpu_counter_mod
4062      0.0017  vmlinux                  vmlinux                  ahc_linux_run_command
4059      0.0017  vmlinux                  vmlinux                  zap_pte_range
4048      0.0017  vmlinux                  vmlinux                  touch_atime
4002      0.0016  vmlinux                  vmlinux                  math_state_restore
3855      0.0016  vmlinux                  vmlinux                  slab_put_obj
3806      0.0016  vmlinux                  vmlinux                  get_page_from_freelist
3801      0.0016  vmlinux                  vmlinux                  ktime_get
3780      0.0016  vmlinux                  vmlinux                  wake_up_bit
3771      0.0016  vmlinux                  vmlinux                  drive_stat_acct
3764      0.0016  vmlinux                  vmlinux                  ext3_splice_branch
3763      0.0015  vmlinux                  vmlinux                  add_to_page_cache
3745      0.0015  vmlinux                  vmlinux                  create_empty_buffers
3717      0.0015  vmlinux                  vmlinux                  clear_page_dirty_for_io
3622      0.0015  vmlinux                  vmlinux                  cache_alloc_refill
3598      0.0015  vmlinux                  vmlinux                  getnstimeofday
3545      0.0015  vmlinux                  vmlinux                  bio_alloc_bioset
3463      0.0014  vmlinux                  vmlinux                  journal_unmap_buffer
3451      0.0014  vmlinux                  vmlinux                  journal_invalidatepage
3432      0.0014  vmlinux                  vmlinux                  kfree
3342      0.0014  vmlinux                  vmlinux                  __pagevec_lru_add
3339      0.0014  vmlinux                  vmlinux                  mpage_writepages
3322      0.0014  vmlinux                  vmlinux                  ret_from_intr
3276      0.0013  vmlinux                  vmlinux                  ext3_ordered_writepage
3269      0.0013  vmlinux                  vmlinux                  get_task_mm
3256      0.0013  vmlinux                  vmlinux                  file_read_actor
3256      0.0013  vmlinux                  vmlinux                  wake_sleeping_dependent
3225      0.0013  vmlinux                  vmlinux                  as_add_arq_hash
3223      0.0013  vmlinux                  vmlinux                  __end_that_request_first
3162      0.0013  vmlinux                  vmlinux                  radix_tree_tag_clear
3144      0.0013  vmlinux                  vmlinux                  file_update_time
3141      0.0013  vmlinux                  vmlinux                  ext3_get_group_desc
3100      0.0013  vmlinux                  vmlinux                  ext3_block_to_path
3097      0.0013  vmlinux                  vmlinux                  ext3_writepage_trans_blocks
3063      0.0013  vmlinux                  vmlinux                  account_system_time
3050      0.0013  vmlinux                  vmlinux                  __journal_remove_journal_head
3033      0.0012  vmlinux                  vmlinux                  __getblk
3032      0.0012  vmlinux                  vmlinux                  alloc_pages_current
3022      0.0012  vmlinux                  vmlinux                  ahc_linux_isr
2995      0.0012  vmlinux                  vmlinux                  bh_lru_install
2988      0.0012  vmlinux                  vmlinux                  ext3_file_write
2963      0.0012  vmlinux                  vmlinux                  journal_remove_journal_head
2962      0.0012  vmlinux                  vmlinux                  recalc_task_prio
2956      0.0012  vmlinux                  vmlinux                  drain_array_locked
2920      0.0012  vmlinux                  vmlinux                  kunmap_atomic
2907      0.0012  vmlinux                  vmlinux                  activate_task
2831      0.0012  vmlinux                  vmlinux                  debug_mutex_unlock
2828      0.0012  vmlinux                  vmlinux                  copy_to_user
2804      0.0012  libpthread-0.10.so       libpthread-0.10.so       __pthread_disable_asynccancel
2796      0.0012  vmlinux                  vmlinux                  ext3_get_branch
2774      0.0011  vmlinux                  vmlinux                  alloc_page_buffers
2772      0.0011  oprofile.ko              oprofile                 mark_done
2716      0.0011  vmlinux                  vmlinux                  ext3_dirty_inode
2697      0.0011  vmlinux                  vmlinux                  journal_blocks_per_page
2677      0.0011  vmlinux                  vmlinux                  find_get_pages_tag
2578      0.0011  vmlinux                  vmlinux                  ext3_alloc_branch
2568      0.0011  vmlinux                  vmlinux                  mempool_alloc_slab
2566      0.0011  vmlinux                  vmlinux                  __alloc_pages
2533      0.0010  vmlinux                  vmlinux                  blk_remove_plug
2528      0.0010  vmlinux                  vmlinux                  vfs_read
2491      0.0010  vmlinux                  vmlinux                  radix_tree_insert
2490      0.0010  vmlinux                  vmlinux                  dequeue_task
2471      0.0010  vmlinux                  vmlinux                  __free_pages
2446      0.0010  vmlinux                  vmlinux                  out_of_line_wait_on_bit_lock
2436      0.0010  vmlinux                  vmlinux                  preempt_schedule
2429      0.0010  vmlinux                  vmlinux                  smp_call_function
2390     9.8e-04  vmlinux                  vmlinux                  elv_queue_empty
2336     9.6e-04  vmlinux                  vmlinux                  journal_get_write_access
2257     9.3e-04  vmlinux                  vmlinux                  __generic_file_aio_read
2250     9.3e-04  vmlinux                  vmlinux                  page_waitqueue
2198     9.1e-04  oprofile.ko              oprofile                 release_mm
2194     9.0e-04  vmlinux                  vmlinux                  slab_get_obj
2149     8.9e-04  vmlinux                  vmlinux                  alloc_buffer_head
2097     8.6e-04  vmlinux                  vmlinux                  mark_buffer_dirty
2084     8.6e-04  vmlinux                  vmlinux                  read_block_bitmap
2083     8.6e-04  vmlinux                  vmlinux                  restore_all
2079     8.6e-04  oprofile.ko              oprofile                 get_slots
2075     8.5e-04  vmlinux                  vmlinux                  ext3_free_blocks_sb
2054     8.5e-04  vmlinux                  vmlinux                  truncate_inode_pages_range
2043     8.4e-04  oprofile.ko              oprofile                 add_user_ctx_switch
2032     8.4e-04  vmlinux                  vmlinux                  ext3_mark_inode_dirty
2030     8.4e-04  vmlinux                  vmlinux                  zonelist_policy
2014     8.3e-04  vmlinux                  vmlinux                  cpu_idle
1985     8.2e-04  vmlinux                  vmlinux                  test_set_page_writeback
1977     8.1e-04  vmlinux                  vmlinux                  send_IPI_mask_sequence
1927     7.9e-04  vmlinux                  vmlinux                  find_lock_page
1917     7.9e-04  vmlinux                  vmlinux                  rw_verify_area
1916     7.9e-04  vmlinux                  vmlinux                  remove_from_page_cache
1876     7.7e-04  vmlinux                  vmlinux                  __mod_page_state_offset
1873     7.7e-04  vmlinux                  vmlinux                  generic_commit_write
1856     7.6e-04  vmlinux                  vmlinux                  test_clear_page_writeback
1844     7.6e-04  vmlinux                  vmlinux                  zone_watermark_ok
1835     7.6e-04  vmlinux                  vmlinux                  tasklet_action
1819     7.5e-04  vmlinux                  vmlinux                  do_sync_read
1814     7.5e-04  vmlinux                  vmlinux                  cache_reap
1792     7.4e-04  vmlinux                  vmlinux                  blk_backing_dev_unplug
1779     7.3e-04  vmlinux                  vmlinux                  new_handle
1760     7.2e-04  vmlinux                  vmlinux                  test_clear_page_dirty
1759     7.2e-04  vmlinux                  vmlinux                  reschedule_interrupt
1755     7.2e-04  vmlinux                  vmlinux                  inotify_dentry_parent_queue_event
1737     7.2e-04  vmlinux                  vmlinux                  __lookup_tag
1734     7.1e-04  vmlinux                  vmlinux                  set_normalized_timespec
1720     7.1e-04  vmlinux                  vmlinux                  __wake_up
1715     7.1e-04  vmlinux                  vmlinux                  sys_write
1713     7.1e-04  vmlinux                  vmlinux                  finish_wait
1711     7.0e-04  oprofile.ko              oprofile                 take_tasks_mm
1690     7.0e-04  vmlinux                  vmlinux                  generic_file_llseek
1651     6.8e-04  vmlinux                  vmlinux                  end_page_writeback
1645     6.8e-04  vmlinux                  vmlinux                  smp_call_function_interrupt
1611     6.6e-04  vmlinux                  vmlinux                  sys_read
1550     6.4e-04  libpthread-0.10.so       libpthread-0.10.so       __libc_lseek
1550     6.4e-04  vmlinux                  vmlinux                  __bread
1545     6.4e-04  vmlinux                  vmlinux                  check_deadlock
1542     6.4e-04  vmlinux                  vmlinux                  cond_resched
1513     6.2e-04  vmlinux                  vmlinux                  mmput
1503     6.2e-04  vmlinux                  vmlinux                  block_write_full_page
1502     6.2e-04  vmlinux                  vmlinux                  set_bh_page
1501     6.2e-04  vmlinux                  vmlinux                  balance_dirty_pages_ratelimited
1492     6.1e-04  vmlinux                  vmlinux                  bio_init
1477     6.1e-04  vmlinux                  vmlinux                  inverted_lock
1474     6.1e-04  vmlinux                  vmlinux                  attempt_merge
1473     6.1e-04  vmlinux                  vmlinux                  profile_hit
1472     6.1e-04  vmlinux                  vmlinux                  ext3_get_inode_loc
1462     6.0e-04  oprofile.ko              oprofile                 get_exec_dcookie
1443     5.9e-04  libpthread-0.10.so       libpthread-0.10.so       __pthread_enable_asynccancel
1382     5.7e-04  vmlinux                  vmlinux                  syscall_exit
1378     5.7e-04  vmlinux                  vmlinux                  rt_run_flush
1374     5.7e-04  vmlinux                  vmlinux                  radix_tree_preload
1372     5.7e-04  vmlinux                  vmlinux                  zone_statistics
1369     5.6e-04  vmlinux                  vmlinux                  radix_tree_delete
1367     5.6e-04  libpthread-0.10.so       libpthread-0.10.so       __i686.get_pc_thunk.cx
1346     5.5e-04  vmlinux                  vmlinux                  __d_lookup
1346     5.5e-04  vmlinux                  vmlinux                  bget_one
1300     5.4e-04  vmlinux                  vmlinux                  deactivate_task
1270     5.2e-04  vmlinux                  vmlinux                  elv_rq_merge_ok
1244     5.1e-04  vmlinux                  vmlinux                  inode_add_bytes
1243     5.1e-04  vmlinux                  vmlinux                  syscall_call
1229     5.1e-04  vmlinux                  vmlinux                  debug_mutex_init_waiter
1222     5.0e-04  vmlinux                  vmlinux                  _spin_lock_bh
1209     5.0e-04  vmlinux                  vmlinux                  memcpy
1202     5.0e-04  vmlinux                  vmlinux                  ext3_get_block
1191     4.9e-04  vmlinux                  vmlinux                  page_pool_alloc
1176     4.8e-04  vmlinux                  vmlinux                  effective_prio
1173     4.8e-04  vmlinux                  vmlinux                  resume_userspace
1172     4.8e-04  vmlinux                  vmlinux                  get_request
1171     4.8e-04  vmlinux                  vmlinux                  do_wait
1164     4.8e-04  vmlinux                  vmlinux                  flush_tlb_mm
1163     4.8e-04  vmlinux                  vmlinux                  rt_check_expire
1156     4.8e-04  vmlinux                  vmlinux                  __do_IRQ
1155     4.8e-04  vmlinux                  vmlinux                  ext3_find_goal
1148     4.7e-04  vmlinux                  vmlinux                  idle_balance
1142     4.7e-04  vmlinux                  vmlinux                  bput_one
1139     4.7e-04  vmlinux                  vmlinux                  as_merge
1138     4.7e-04  vmlinux                  vmlinux                  debug_mutex_add_waiter
1137     4.7e-04  vmlinux                  vmlinux                  find_next_zero_bit
1093     4.5e-04  vmlinux                  vmlinux                  as_merged_request
1072     4.4e-04  vmlinux                  vmlinux                  find_get_pages
1066     4.4e-04  libperl.so               libperl.so               (no symbols)
1064     4.4e-04  vmlinux                  vmlinux                  mark_buffer_async_write
1050     4.3e-04  vmlinux                  vmlinux                  page_slot
1037     4.3e-04  vmlinux                  vmlinux                  _read_lock_irq
1024     4.2e-04  vmlinux                  vmlinux                  prep_new_page
1021     4.2e-04  vmlinux                  vmlinux                  recalc_bh_state
1017     4.2e-04  vmlinux                  vmlinux                  __down
1009     4.2e-04  tiotest                  tiotest                  do_random_write_test
1005     4.1e-04  vmlinux                  vmlinux                  cache_init_objs
991      4.1e-04  vmlinux                  vmlinux                  delayed_work_timer_fn
985      4.1e-04  vmlinux                  vmlinux                  _atomic_dec_and_lock
969      4.0e-04  libpthread-0.10.so       libpthread-0.10.so       pthread_reap_children
957      3.9e-04  vmlinux                  vmlinux                  i8042_interrupt
952      3.9e-04  vmlinux                  vmlinux                  generic_file_aio_read
950      3.9e-04  vmlinux                  vmlinux                  ext3_find_entry
950      3.9e-04  vmlinux                  vmlinux                  run_workqueue
947      3.9e-04  vmlinux                  vmlinux                  submit_bio
931      3.8e-04  vmlinux                  vmlinux                  page_cache_readahead
930      3.8e-04  vmlinux                  vmlinux                  internal_add_timer
925      3.8e-04  vmlinux                  vmlinux                  task_timeslice
923      3.8e-04  vmlinux                  vmlinux                  ext3_free_data
908      3.7e-04  oprofile.ko              oprofile                 add_kernel_ctx_switch
901      3.7e-04  vmlinux                  vmlinux                  __link_path_walk
898      3.7e-04  vmlinux                  vmlinux                  truncate_complete_page
890      3.7e-04  vmlinux                  vmlinux                  _read_lock
883      3.6e-04  vmlinux                  vmlinux                  generic_unplug_device
882      3.6e-04  vmlinux                  vmlinux                  search_reserve_window
877      3.6e-04  vmlinux                  vmlinux                  as_find_arq_rb
873      3.6e-04  vmlinux                  vmlinux                  page_fault
871      3.6e-04  vmlinux                  vmlinux                  __bitmap_weight
867      3.6e-04  vmlinux                  vmlinux                  ext3_invalidatepage
867      3.6e-04  vmlinux                  vmlinux                  inotify_inode_queue_event
861      3.5e-04  vmlinux                  vmlinux                  autoremove_wake_function
850      3.5e-04  vmlinux                  vmlinux                  _spin_lock_irq
850      3.5e-04  vmlinux                  vmlinux                  goal_in_my_reservation
848      3.5e-04  vmlinux                  vmlinux                  bio_free
824      3.4e-04  vmlinux                  vmlinux                  handle_IRQ_event
814      3.4e-04  libpthread-0.10.so       libpthread-0.10.so       __pthread_manager
795      3.3e-04  vmlinux                  vmlinux                  lock_timer_base
794      3.3e-04  vmlinux                  vmlinux                  __rb_erase_color
792      3.3e-04  vmlinux                  vmlinux                  journal_alloc_journal_head
786      3.2e-04  vmlinux                  vmlinux                  do_page_fault
772      3.2e-04  vmlinux                  vmlinux                  e1000_update_stats
763      3.1e-04  vmlinux                  vmlinux                  find_next_reservable_window
751      3.1e-04  vmlinux                  vmlinux                  rb_erase
748      3.1e-04  vmlinux                  vmlinux                  as_choose_req
743      3.1e-04  vmlinux                  vmlinux                  rb_insert_color
736      3.0e-04  vmlinux                  vmlinux                  blk_queue_bounce
734      3.0e-04  vmlinux                  vmlinux                  bio_endio
731      3.0e-04  vmlinux                  vmlinux                  remove_suid
723      3.0e-04  vmlinux                  vmlinux                  ext3_journal_dirty_data
715      2.9e-04  vmlinux                  vmlinux                  unmap_underlying_metadata
712      2.9e-04  vmlinux                  vmlinux                  find_idlest_group
706      2.9e-04  vmlinux                  vmlinux                  timespec_trunc
703      2.9e-04  vmlinux                  vmlinux                  worker_thread
702      2.9e-04  tiotest                  tiotest                  do_random_read_test
690      2.8e-04  vmlinux                  vmlinux                  elv_latter_request
689      2.8e-04  vmlinux                  vmlinux                  as_move_to_dispatch
685      2.8e-04  vmlinux                  vmlinux                  mutex_remove_waiter
684      2.8e-04  vmlinux                  vmlinux                  __rcu_process_callbacks
668      2.8e-04  vmlinux                  vmlinux                  copy_process
664      2.7e-04  vmlinux                  vmlinux                  load_balance_newidle
654      2.7e-04  vmlinux                  vmlinux                  bitmap_search_next_usable_block
651      2.7e-04  vmlinux                  vmlinux                  __mod_timer
645      2.7e-04  vmlinux                  vmlinux                  eligible_child
638      2.6e-04  oprofile.ko              oprofile                 wq_sync_buffer
635      2.6e-04  vmlinux                  vmlinux                  add_dirent_to_buf
634      2.6e-04  vmlinux                  vmlinux                  call_function_interrupt
630      2.6e-04  vmlinux                  vmlinux                  rcu_check_quiescent_state
629      2.6e-04  vmlinux                  vmlinux                  rmqueue_bulk
619      2.5e-04  vmlinux                  vmlinux                  move_tasks
611      2.5e-04  vmlinux                  vmlinux                  scsi_request_fn
606      2.5e-04  vmlinux                  vmlinux                  kmap
604      2.5e-04  vmlinux                  vmlinux                  common_interrupt
602      2.5e-04  vmlinux                  vmlinux                  flush_all_zero_pkmaps
583      2.4e-04  vmlinux                  vmlinux                  ext3_rsv_window_add
564      2.3e-04  vmlinux                  vmlinux                  queue_delayed_work
561      2.3e-04  vmlinux                  vmlinux                  rcu_process_callbacks
553      2.3e-04  vmlinux                  vmlinux                  ll_rw_block
548      2.3e-04  vmlinux                  vmlinux                  __generic_unplug_device
548      2.3e-04  vmlinux                  vmlinux                  mempool_free_slab
545      2.2e-04  libc-2.3.3.so            libc-2.3.3.so            __clone
543      2.2e-04  vmlinux                  vmlinux                  do_mmap_pgoff
540      2.2e-04  vmlinux                  vmlinux                  rcu_check_callbacks
537      2.2e-04  vmlinux                  vmlinux                  ack_edge_ioapic_irq
535      2.2e-04  vmlinux                  vmlinux                  cpu_quiet
532      2.2e-04  vmlinux                  vmlinux                  find_pid
529      2.2e-04  vmlinux                  vmlinux                  mutex_debug_check_no_locks_held
527      2.2e-04  vmlinux                  vmlinux                  ahc_run_qoutfifo
521      2.1e-04  vmlinux                  vmlinux                  local_bh_enable
514      2.1e-04  vmlinux                  vmlinux                  bounce_end_io_write
509      2.1e-04  vmlinux                  vmlinux                  run_local_timers
508      2.1e-04  vmlinux                  vmlinux                  end_bio_bh_io_sync
504      2.1e-04  vmlinux                  vmlinux                  try_to_free_buffers
502      2.1e-04  libc-2.3.3.so            libc-2.3.3.so            _int_malloc
501      2.1e-04  vmlinux                  vmlinux                  get_empty_filp
496      2.0e-04  vmlinux                  vmlinux                  __clear_page_buffers
496      2.0e-04  vmlinux                  vmlinux                  elv_merged_request
494      2.0e-04  vmlinux                  vmlinux                  bio_hw_segments
494      2.0e-04  vmlinux                  vmlinux                  do_wp_page
490      2.0e-04  vmlinux                  vmlinux                  ahc_done
488      2.0e-04  vmlinux                  vmlinux                  init_request_from_bio
478      2.0e-04  vmlinux                  vmlinux                  _write_lock_irq
474      2.0e-04  vmlinux                  vmlinux                  as_latter_request
472      1.9e-04  vmlinux                  vmlinux                  do_timer
469      1.9e-04  vmlinux                  vmlinux                  kref_get
462      1.9e-04  vmlinux                  vmlinux                  drop_buffers
455      1.9e-04  vmlinux                  vmlinux                  __handle_mm_fault
451      1.9e-04  vmlinux                  vmlinux                  __lookup
450      1.9e-04  vmlinux                  vmlinux                  do_path_lookup
447      1.8e-04  vmlinux                  vmlinux                  cache_grow
445      1.8e-04  vmlinux                  vmlinux                  as_update_iohist
443      1.8e-04  vmlinux                  vmlinux                  bio_alloc
442      1.8e-04  vmlinux                  vmlinux                  get_unused_fd
440      1.8e-04  vmlinux                  vmlinux                  kref_put
436      1.8e-04  vmlinux                  vmlinux                  as_queue_empty
434      1.8e-04  vmlinux                  vmlinux                  __remove_from_page_cache
431      1.8e-04  libpthread-0.10.so       libpthread-0.10.so       pthread_start_thread
431      1.8e-04  vmlinux                  vmlinux                  as_remove_queued_request
431      1.8e-04  vmlinux                  vmlinux                  pull_task
428      1.8e-04  vmlinux                  vmlinux                  __tasklet_schedule
428      1.8e-04  vmlinux                  vmlinux                  ext3_has_free_blocks
422      1.7e-04  vmlinux                  vmlinux                  k_getrusage
420      1.7e-04  libc-2.3.3.so            libc-2.3.3.so            __sigprocmask
416      1.7e-04  vmlinux                  vmlinux                  delay_cyclone
414      1.7e-04  vmlinux                  vmlinux                  as_dispatch_request
408      1.7e-04  vmlinux                  vmlinux                  sys_lseek
397      1.6e-04  vmlinux                  vmlinux                  note_interrupt
396      1.6e-04  vmlinux                  vmlinux                  as_add_request
388      1.6e-04  vmlinux                  vmlinux                  strncpy_from_user
385      1.6e-04  vmlinux                  vmlinux                  put_io_context
383      1.6e-04  vmlinux                  vmlinux                  scsi_get_command
380      1.6e-04  vmlinux                  vmlinux                  __as_add_arq_rb
377      1.6e-04  vmlinux                  vmlinux                  as_can_break_anticipation
377      1.6e-04  vmlinux                  vmlinux                  scsi_prep_fn
370      1.5e-04  vmlinux                  vmlinux                  kmem_getpages
368      1.5e-04  vmlinux                  vmlinux                  sd_init_command
361      1.5e-04  vmlinux                  vmlinux                  open_namei
357      1.5e-04  vmlinux                  vmlinux                  ext3_clear_blocks
357      1.5e-04  vmlinux                  vmlinux                  journal_dirty_data_fn
353      1.5e-04  vmlinux                  vmlinux                  blk_queue_start_tag
353      1.5e-04  vmlinux                  vmlinux                  do_exit
351      1.4e-04  vmlinux                  vmlinux                  prepare_to_wait
348      1.4e-04  libc-2.3.3.so            libc-2.3.3.so            __GI___getrusage
348      1.4e-04  libpthread-0.10.so       libpthread-0.10.so       __pthread_create_2_1
347      1.4e-04  libc-2.3.3.so            libc-2.3.3.so            _int_free
344      1.4e-04  oprofile.ko              oprofile                 cpu_buffer_reset
341      1.4e-04  vmlinux                  vmlinux                  __sigqueue_alloc
341      1.4e-04  vmlinux                  vmlinux                  recalc_sigpending_tsk
340      1.4e-04  vmlinux                  vmlinux                  remove_wait_queue
338      1.4e-04  vmlinux                  vmlinux                  do_irq_balance
337      1.4e-04  libpthread-0.10.so       libpthread-0.10.so       __i686.get_pc_thunk.bx
336      1.4e-04  ld-2.3.3.so              ld-2.3.3.so              memset
335      1.4e-04  vmlinux                  vmlinux                  do_anonymous_page
333      1.4e-04  libpthread-0.10.so       libpthread-0.10.so       __pthread_lock
328      1.4e-04  vmlinux                  vmlinux                  find_next_usable_block
324      1.3e-04  vmlinux                  vmlinux                  dput
323      1.3e-04  vmlinux                  vmlinux                  copy_io_context
322      1.3e-04  vmlinux                  vmlinux                  __do_page_cache_readahead
319      1.3e-04  vmlinux                  vmlinux                  end_that_request_last
317      1.3e-04  vmlinux                  vmlinux                  bio_phys_segments
311      1.3e-04  vmlinux                  vmlinux                  as_update_seekdist
311      1.3e-04  vmlinux                  vmlinux                  elv_insert
306      1.3e-04  vmlinux                  vmlinux                  default_wake_function
306      1.3e-04  vmlinux                  vmlinux                  may_open
305      1.3e-04  vmlinux                  vmlinux                  bio_fs_destructor
305      1.3e-04  vmlinux                  vmlinux                  scsi_end_request
304      1.3e-04  vmlinux                  vmlinux                  del_timer
302      1.2e-04  vmlinux                  vmlinux                  find_vma_prev
300      1.2e-04  vmlinux                  vmlinux                  do_invalidatepage
298      1.2e-04  vmlinux                  vmlinux                  scsi_dispatch_cmd
297      1.2e-04  vmlinux                  vmlinux                  ext3_forget
295      1.2e-04  libc-2.3.3.so            libc-2.3.3.so            __mmap
295      1.2e-04  vmlinux                  vmlinux                  as_completed_request
295      1.2e-04  vmlinux                  vmlinux                  elv_next_request
279      1.1e-04  vmlinux                  vmlinux                  init_fpu
279      1.1e-04  vmlinux                  vmlinux                  mutex_unlock
278      1.1e-04  vmlinux                  vmlinux                  kmem_freepages
275      1.1e-04  bash                     bash                     (no symbols)
274      1.1e-04  libpthread-0.10.so       libpthread-0.10.so       __pthread_do_exit
268      1.1e-04  vmlinux                  vmlinux                  __follow_mount
264      1.1e-04  vmlinux                  vmlinux                  work_resched
262      1.1e-04  libpthread-0.10.so       libpthread-0.10.so       __pthread_destroy_specifics
261      1.1e-04  vmlinux                  vmlinux                  alloc_new_reservation
261      1.1e-04  vmlinux                  vmlinux                  free_buffer_head
261      1.1e-04  vmlinux                  vmlinux                  wait_task_zombie
255      1.1e-04  vmlinux                  vmlinux                  blk_complete_request
250      1.0e-04  vmlinux                  vmlinux                  as_may_queue
250      1.0e-04  vmlinux                  vmlinux                  as_put_io_context
249      1.0e-04  vmlinux                  vmlinux                  cache_flusharray
247      1.0e-04  vmlinux                  vmlinux                  exec_permission_lite
245      1.0e-04  vmlinux                  vmlinux                  as_set_request
244      1.0e-04  vmlinux                  vmlinux                  __queue_work
244      1.0e-04  vmlinux                  vmlinux                  ext3_check_dir_entry
241      9.9e-05  oprofile.ko              oprofile                 add_cpu_switch
240      9.9e-05  vmlinux                  vmlinux                  blk_queue_end_tag
240      9.9e-05  vmlinux                  vmlinux                  update_wall_time_one_tick
239      9.8e-05  vmlinux                  vmlinux                  ahc_linux_queue
236      9.7e-05  vmlinux                  vmlinux                  page_pool_free
232      9.6e-05  vmlinux                  vmlinux                  ext3_release_file
232      9.6e-05  vmlinux                  vmlinux                  remove_vma_list
231      9.5e-05  vmlinux                  vmlinux                  elv_completed_request
229      9.4e-05  vmlinux                  vmlinux                  scsi_init_cmd_errh
227      9.4e-05  vmlinux                  vmlinux                  get_io_context
226      9.3e-05  vmlinux                  vmlinux                  generic_file_open
224      9.2e-05  vmlinux                  vmlinux                  _read_unlock
222      9.1e-05  libpthread-0.10.so       libpthread-0.10.so       __libc_open
218      9.0e-05  libc-2.3.3.so            libc-2.3.3.so            __GI___poll
216      8.9e-05  vmlinux                  vmlinux                  end_level_ioapic_irq
213      8.8e-05  vmlinux                  vmlinux                  __dentry_open
213      8.8e-05  vmlinux                  vmlinux                  _spin_unlock_bh
210      8.6e-05  vmlinux                  vmlinux                  elv_dispatch_sort
209      8.6e-05  vmlinux                  vmlinux                  can_migrate_task
208      8.6e-05  libpthread-0.10.so       libpthread-0.10.so       __pthread_alt_unlock
208      8.6e-05  vmlinux                  vmlinux                  scsi_init_io
206      8.5e-05  ld-2.3.3.so              ld-2.3.3.so              do_lookup_x
203      8.4e-05  oprofile.ko              oprofile                 event_buffer_read
202      8.3e-05  vmlinux                  vmlinux                  raise_softirq_irqoff
202      8.3e-05  vmlinux                  vmlinux                  resume_kernel
201      8.3e-05  vmlinux                  vmlinux                  __pagevec_lru_add_active
201      8.3e-05  vmlinux                  vmlinux                  task_curr
200      8.2e-05  vmlinux                  vmlinux                  kthread_should_stop
199      8.2e-05  vmlinux                  vmlinux                  disk_round_stats
196      8.1e-05  vmlinux                  vmlinux                  __rb_rotate_left
196      8.1e-05  vmlinux                  vmlinux                  notifier_call_chain
195      8.0e-05  libpthread-0.10.so       libpthread-0.10.so       pthread_join
194      8.0e-05  vmlinux                  vmlinux                  error_code
193      7.9e-05  libc-2.3.3.so            libc-2.3.3.so            __GI_read
191      7.9e-05  libc-2.3.3.so            libc-2.3.3.so            __GI_getpid
190      7.8e-05  vmlinux                  vmlinux                  sys_close
188      7.7e-05  libc-2.3.3.so            libc-2.3.3.so            __GI___libc_malloc
186      7.7e-05  vmlinux                  vmlinux                  rb_prev
185      7.6e-05  vmlinux                  vmlinux                  generic_permission
184      7.6e-05  vmlinux                  vmlinux                  ext3_orphan_add
183      7.5e-05  vmlinux                  vmlinux                  scsi_device_unbusy
182      7.5e-05  vmlinux                  vmlinux                  __scsi_done
182      7.5e-05  vmlinux                  vmlinux                  read_page_state_offset
179      7.4e-05  libc-2.3.3.so            libc-2.3.3.so            __GI___munmap
179      7.4e-05  vmlinux                  vmlinux                  mutex_lock
178      7.3e-05  vmlinux                  vmlinux                  scsi_run_queue
177      7.3e-05  vmlinux                  vmlinux                  find_busiest_queue
176      7.2e-05  libc-2.3.3.so            libc-2.3.3.so            rand_r
176      7.2e-05  vmlinux                  vmlinux                  exit_mm
176      7.2e-05  vmlinux                  vmlinux                  fget
175      7.2e-05  vmlinux                  vmlinux                  collect_signal
175      7.2e-05  vmlinux                  vmlinux                  file_move
173      7.1e-05  vmlinux                  vmlinux                  add_disk_randomness
171      7.0e-05  vmlinux                  vmlinux                  ahc_linux_queue_cmd_complete
171      7.0e-05  vmlinux                  vmlinux                  as_activate_request
171      7.0e-05  vmlinux                  vmlinux                  schedule_timeout
170      7.0e-05  vmlinux                  vmlinux                  __freed_request
170      7.0e-05  vmlinux                  vmlinux                  do_IRQ
170      7.0e-05  vmlinux                  vmlinux                  sys_mmap2
169      7.0e-05  vmlinux                  vmlinux                  permission
169      7.0e-05  vmlinux                  vmlinux                  sched_balance_self
167      6.9e-05  vmlinux                  vmlinux                  __copy_user_intel
167      6.9e-05  vmlinux                  vmlinux                  __drain_alien_cache
167      6.9e-05  vmlinux                  vmlinux                  get_request_wait
166      6.8e-05  libpthread-0.10.so       libpthread-0.10.so       __pthread_unlock
165      6.8e-05  libpthread-0.10.so       libpthread-0.10.so       __libc_close
165      6.8e-05  vmlinux                  vmlinux                  blk_do_ordered
165      6.8e-05  vmlinux                  vmlinux                  profile_task_exit
163      6.7e-05  libc-2.3.3.so            libc-2.3.3.so            __GI___res_state
163      6.7e-05  vmlinux                  vmlinux                  release_task
162      6.7e-05  vmlinux                  vmlinux                  ext3_new_inode
162      6.7e-05  vmlinux                  vmlinux                  schedule_delayed_work
162      6.7e-05  vmlinux                  vmlinux                  scsi_io_completion
161      6.6e-05  vmlinux                  vmlinux                  __fput
161      6.6e-05  vmlinux                  vmlinux                  find_vma_prepare
161      6.6e-05  vmlinux                  vmlinux                  init_timer
161      6.6e-05  vmlinux                  vmlinux                  unmap_page_range
160      6.6e-05  vmlinux                  vmlinux                  exit_notify
159      6.5e-05  vmlinux                  vmlinux                  dup_task_struct
159      6.5e-05  vmlinux                  vmlinux                  radix_tree_gang_lookup_tag
158      6.5e-05  vmlinux                  vmlinux                  ext3_truncate
157      6.5e-05  vmlinux                  vmlinux                  as_update_arq
155      6.4e-05  vmlinux                  vmlinux                  put_files_struct
154      6.3e-05  vmlinux                  vmlinux                  filemap_fdatawait
153      6.3e-05  vmlinux                  vmlinux                  split_vma
152      6.3e-05  vmlinux                  vmlinux                  wake_up_new_task
151      6.2e-05  vmlinux                  vmlinux                  radix_tree_node_ctor
150      6.2e-05  vmlinux                  vmlinux                  blk_done_softirq
148      6.1e-05  vmlinux                  vmlinux                  __sigqueue_free
147      6.1e-05  libc-2.3.3.so            libc-2.3.3.so            _IO_default_xsputn_internal
145      6.0e-05  vmlinux                  vmlinux                  __journal_clean_checkpoint_list
145      6.0e-05  vmlinux                  vmlinux                  cascade
145      6.0e-05  vmlinux                  vmlinux                  journal_get_descriptor_buffer
145      6.0e-05  vmlinux                  vmlinux                  mutex_trylock
144      5.9e-05  vmlinux                  vmlinux                  unmap_vmas
143      5.9e-05  vmlinux                  vmlinux                  current_io_context
142      5.8e-05  libpthread-0.10.so       libpthread-0.10.so       __libc_fsync
141      5.8e-05  libc-2.3.3.so            libc-2.3.3.so            __calloc
139      5.7e-05  vmlinux                  vmlinux                  e1000_clean_rx_irq
139      5.7e-05  vmlinux                  vmlinux                  page_remove_rmap
138      5.7e-05  vmlinux                  vmlinux                  __detach_pid
138      5.7e-05  vmlinux                  vmlinux                  __pagevec_free
138      5.7e-05  vmlinux                  vmlinux                  blockable_page_cache_readahead
137      5.6e-05  libc-2.3.3.so            libc-2.3.3.so            __GI___sched_setscheduler
137      5.6e-05  libc-2.3.3.so            libc-2.3.3.so            __kill
137      5.6e-05  oprofile.ko              oprofile                 munmap_notify
136      5.6e-05  vmlinux                  vmlinux                  link_path_walk
135      5.6e-05  vmlinux                  vmlinux                  wake_up_process
134      5.5e-05  vmlinux                  vmlinux                  del_timer_sync
133      5.5e-05  libpthread-0.10.so       libpthread-0.10.so       __pthread_internal_tsd_get
133      5.5e-05  vmlinux                  vmlinux                  scsi_put_command
132      5.4e-05  vmlinux                  vmlinux                  ext3_discard_reservation
128      5.3e-05  vmlinux                  vmlinux                  __lru_add_drain
128      5.3e-05  vmlinux                  vmlinux                  do_sys_poll
128      5.3e-05  vmlinux                  vmlinux                  handle_stop_signal
127      5.2e-05  vmlinux                  vmlinux                  do_no_page
126      5.2e-05  libc-2.3.3.so            libc-2.3.3.so            __uselocale
126      5.2e-05  vmlinux                  vmlinux                  anon_vma_prepare
126      5.2e-05  vmlinux                  vmlinux                  do_group_exit
126      5.2e-05  vmlinux                  vmlinux                  smp_reschedule_interrupt
125      5.1e-05  vmlinux                  vmlinux                  drain_node_pages
125      5.1e-05  vmlinux                  vmlinux                  vm_normal_page
124      5.1e-05  libc-2.3.3.so            libc-2.3.3.so            __cfree
124      5.1e-05  vmlinux                  vmlinux                  add_timer_randomness
123      5.1e-05  libc-2.3.3.so            libc-2.3.3.so            __GI___mbrtowc
123      5.1e-05  vmlinux                  vmlinux                  blk_run_queue
123      5.1e-05  vmlinux                  vmlinux                  clear_queue_congested
122      5.0e-05  vmlinux                  vmlinux                  __exit_signal
122      5.0e-05  vmlinux                  vmlinux                  attach_pid
121      5.0e-05  libc-2.3.3.so            libc-2.3.3.so            __GI_open
121      5.0e-05  vmlinux                  vmlinux                  copy_from_user
121      5.0e-05  vmlinux                  vmlinux                  update_wall_time
120      4.9e-05  vmlinux                  vmlinux                  scsi_next_command
120      4.9e-05  vmlinux                  vmlinux                  vm_acct_memory
119      4.9e-05  vmlinux                  vmlinux                  __group_complete_signal
119      4.9e-05  vmlinux                  vmlinux                  as_put_request
119      4.9e-05  vmlinux                  vmlinux                  sys_munmap
118      4.9e-05  libc-2.3.3.so            libc-2.3.3.so            _IO_new_fclose
118      4.9e-05  vmlinux                  vmlinux                  __pte_alloc
118      4.9e-05  vmlinux                  vmlinux                  hrtimer_init
118      4.9e-05  vmlinux                  vmlinux                  try_to_del_timer_sync
117      4.8e-05  libc-2.3.3.so            libc-2.3.3.so            __gconv_transform_utf8_internal
117      4.8e-05  libpthread-0.10.so       libpthread-0.10.so       pthread_handle_sigrestart
117      4.8e-05  oprofile.ko              oprofile                 .text.lock.buffer_sync
117      4.8e-05  vmlinux                  vmlinux                  rcu_do_batch
116      4.8e-05  libpthread-0.10.so       libpthread-0.10.so       __GI___pthread_mutex_trylock
116      4.8e-05  vmlinux                  vmlinux                  call_rcu
116      4.8e-05  vmlinux                  vmlinux                  sd_rw_intr
115      4.7e-05  vmlinux                  vmlinux                  __lookup_mnt
115      4.7e-05  vmlinux                  vmlinux                  file_ra_state_init
115      4.7e-05  vmlinux                  vmlinux                  memmove
115      4.7e-05  vmlinux                  vmlinux                  smp_send_timer_broadcast_ipi
113      4.7e-05  libc-2.3.3.so            libc-2.3.3.so            malloc_consolidate
113      4.7e-05  vmlinux                  vmlinux                  dnotify_flush
113      4.7e-05  vmlinux                  vmlinux                  send_signal
113      4.7e-05  vmlinux                  vmlinux                  work_pending
112      4.6e-05  vmlinux                  vmlinux                  pipe_writev
111      4.6e-05  vmlinux                  vmlinux                  __elv_add_request
111      4.6e-05  vmlinux                  vmlinux                  sys_rt_sigprocmask
111      4.6e-05  vmlinux                  vmlinux                  unmap_region
111      4.6e-05  vmlinux                  vmlinux                  vm_stat_account
110      4.5e-05  libpthread-0.10.so       libpthread-0.10.so       pthread_handle_sigcancel
110      4.5e-05  vmlinux                  vmlinux                  cleanup_timers
109      4.5e-05  vmlinux                  vmlinux                  add_wait_queue
108      4.4e-05  vmlinux                  vmlinux                  scsi_alloc_sgtable
107      4.4e-05  vmlinux                  vmlinux                  alloc_slabmgmt
107      4.4e-05  vmlinux                  vmlinux                  fd_install
106      4.4e-05  vmlinux                  vmlinux                  as_fifo_expired
106      4.4e-05  vmlinux                  vmlinux                  get_signal_to_deliver
105      4.3e-05  vmlinux                  vmlinux                  __rb_rotate_right
105      4.3e-05  vmlinux                  vmlinux                  radix_tree_gang_lookup
104      4.3e-05  vmlinux                  vmlinux                  vma_link
103      4.2e-05  vmlinux                  vmlinux                  elv_set_request
103      4.2e-05  vmlinux                  vmlinux                  free_pgd_range
103      4.2e-05  vmlinux                  vmlinux                  vfs_llseek
102      4.2e-05  vmlinux                  vmlinux                  __down_failed
102      4.2e-05  vmlinux                  vmlinux                  __pagevec_release
102      4.2e-05  vmlinux                  vmlinux                  vfs_permission
101      4.2e-05  vmlinux                  vmlinux                  __blk_put_request
101      4.2e-05  vmlinux                  vmlinux                  set_queue_congested
100      4.1e-05  vmlinux                  vmlinux                  __up_wakeup
99       4.1e-05  vmlinux                  vmlinux                  __kmalloc
99       4.1e-05  vmlinux                  vmlinux                  do_poll
99       4.1e-05  vmlinux                  vmlinux                  lookup_mnt
99       4.1e-05  vmlinux                  vmlinux                  lru_cache_add_active
98       4.0e-05  vmlinux                  vmlinux                  __journal_unfile_buffer
98       4.0e-05  vmlinux                  vmlinux                  scsi_done
97       4.0e-05  vmlinux                  vmlinux                  scsi_add_timer
96       4.0e-05  libc-2.3.3.so            libc-2.3.3.so            res_thread_freeres
96       4.0e-05  libpthread-0.10.so       libpthread-0.10.so       __pthread_alt_lock
96       4.0e-05  vmlinux                  vmlinux                  as_find_next_arq
96       4.0e-05  vmlinux                  vmlinux                  free_pgtables
96       4.0e-05  vmlinux                  vmlinux                  kobject_get
95       3.9e-05  vmlinux                  vmlinux                  ext3_setattr
95       3.9e-05  vmlinux                  vmlinux                  rcu_start_batch
95       3.9e-05  vmlinux                  vmlinux                  schedule_tail
95       3.9e-05  vmlinux                  vmlinux                  scsi_decide_disposition
94       3.9e-05  libc-2.3.3.so            libc-2.3.3.so            __GI_write
94       3.9e-05  vmlinux                  vmlinux                  inode_init_once
94       3.9e-05  vmlinux                  vmlinux                  notify_change
93       3.8e-05  tiotest                  tiotest                  wait_for_threads
93       3.8e-05  vmlinux                  vmlinux                  __path_lookup_intent_open
93       3.8e-05  vmlinux                  vmlinux                  elv_may_queue
93       3.8e-05  vmlinux                  vmlinux                  requeue_task
92       3.8e-05  vmlinux                  vmlinux                  __scsi_get_command
92       3.8e-05  vmlinux                  vmlinux                  __up
92       3.8e-05  vmlinux                  vmlinux                  do_sys_open
92       3.8e-05  vmlinux                  vmlinux                  ext3_sync_file
92       3.8e-05  vmlinux                  vmlinux                  setup_frame
91       3.7e-05  libc-2.3.3.so            libc-2.3.3.so            _IO_un_link_internal
91       3.7e-05  vmlinux                  vmlinux                  remove_vma
91       3.7e-05  vmlinux                  vmlinux                  wake_up_inode
91       3.7e-05  vmlinux                  vmlinux                  writeback_inodes
90       3.7e-05  libc-2.3.3.so            libc-2.3.3.so            _IO_link_in_internal
90       3.7e-05  vmlinux                  vmlinux                  pipe_readv
89       3.7e-05  libpthread-0.10.so       libpthread-0.10.so       __pthread_sigsuspend
89       3.7e-05  vmlinux                  vmlinux                  change_protection
89       3.7e-05  vmlinux                  vmlinux                  inode_setattr
89       3.7e-05  vmlinux                  vmlinux                  journal_write_metadata_buffer
88       3.6e-05  libc-2.3.3.so            libc-2.3.3.so            _IO_fwrite_internal
88       3.6e-05  libc-2.3.3.so            libc-2.3.3.so            __mprotect
88       3.6e-05  vmlinux                  vmlinux                  __put_task_struct_cb
88       3.6e-05  vmlinux                  vmlinux                  e1000_intr
88       3.6e-05  vmlinux                  vmlinux                  vma_adjust
87       3.6e-05  libc-2.3.3.so            libc-2.3.3.so            __GI_memcpy
87       3.6e-05  vmlinux                  vmlinux                  as_close_req
87       3.6e-05  vmlinux                  vmlinux                  lru_add_drain
87       3.6e-05  vmlinux                  vmlinux                  scsi_softirq_done
86       3.5e-05  vmlinux                  vmlinux                  d_alloc
85       3.5e-05  libc-2.3.3.so            libc-2.3.3.so            _IO_file_close_it_internal
85       3.5e-05  vmlinux                  vmlinux                  freed_request
84       3.5e-05  libc-2.3.3.so            libc-2.3.3.so            __flockfile
84       3.5e-05  vmlinux                  vmlinux                  flush_tlb_page
83       3.4e-05  vmlinux                  vmlinux                  ext3_free_blocks
83       3.4e-05  vmlinux                  vmlinux                  inode_sub_bytes
83       3.4e-05  vmlinux                  vmlinux                  sys_mprotect
82       3.4e-05  vmlinux                  vmlinux                  __anon_vma_link
82       3.4e-05  vmlinux                  vmlinux                  __cache_alloc_node
82       3.4e-05  vmlinux                  vmlinux                  __lookup_hash
82       3.4e-05  vmlinux                  vmlinux                  getname
82       3.4e-05  vmlinux                  vmlinux                  group_send_sig_info
81       3.3e-05  vmlinux                  vmlinux                  anon_vma_unlink
81       3.3e-05  vmlinux                  vmlinux                  do_notify_parent
81       3.3e-05  vmlinux                  vmlinux                  sys_open
80       3.3e-05  libc-2.3.3.so            libc-2.3.3.so            _IO_file_overflow_internal
80       3.3e-05  libc-2.3.3.so            libc-2.3.3.so            ___fxstat64
80       3.3e-05  vmlinux                  vmlinux                  scsi_finish_command
79       3.3e-05  libc-2.3.3.so            libc-2.3.3.so            __GI___lseek
79       3.3e-05  libc-2.3.3.so            libc-2.3.3.so            __libc_thread_freeres
79       3.3e-05  vmlinux                  vmlinux                  exit_io_context
78       3.2e-05  ld-2.3.3.so              ld-2.3.3.so              strcmp
78       3.2e-05  vmlinux                  vmlinux                  __sync_single_inode
78       3.2e-05  vmlinux                  vmlinux                  alloc_page_vma
78       3.2e-05  vmlinux                  vmlinux                  ext3_permission
78       3.2e-05  vmlinux                  vmlinux                  profile_munmap
78       3.2e-05  vmlinux                  vmlinux                  sys_getrusage
77       3.2e-05  libc-2.3.3.so            libc-2.3.3.so            __getppid
77       3.2e-05  vmlinux                  vmlinux                  end_buffer_write_sync
77       3.2e-05  vmlinux                  vmlinux                  slab_destroy
77       3.2e-05  vmlinux                  vmlinux                  sys_set_thread_area
77       3.2e-05  vmlinux                  vmlinux                  vma_merge
76       3.1e-05  vmlinux                  vmlinux                  hrtimer_try_to_cancel
75       3.1e-05  vmlinux                  vmlinux                  ext3_orphan_del
74       3.0e-05  ld-2.3.3.so              ld-2.3.3.so              _dl_relocate_object
74       3.0e-05  vmlinux                  vmlinux                  do_lookup
74       3.0e-05  vmlinux                  vmlinux                  ext3_alloc_block
74       3.0e-05  vmlinux                  vmlinux                  kobject_put
73       3.0e-05  libc-2.3.3.so            libc-2.3.3.so            _IO_file_close_internal
73       3.0e-05  vmlinux                  vmlinux                  do_filp_open
72       3.0e-05  vmlinux                  vmlinux                  pagevec_lookup_tag
71       2.9e-05  vmlinux                  vmlinux                  as_update_thinktime
71       2.9e-05  vmlinux                  vmlinux                  journal_get_create_access
71       2.9e-05  vmlinux                  vmlinux                  scsi_free_sgtable
70       2.9e-05  vmlinux                  vmlinux                  copy_pte_range
70       2.9e-05  vmlinux                  vmlinux                  wait_on_page_writeback_range
69       2.8e-05  vmlinux                  vmlinux                  alloc_inode
69       2.8e-05  vmlinux                  vmlinux                  do_fsync
69       2.8e-05  vmlinux                  vmlinux                  journal_file_buffer
69       2.8e-05  vmlinux                  vmlinux                  log_wait_commit
69       2.8e-05  vmlinux                  vmlinux                  pipe_poll
69       2.8e-05  vmlinux                  vmlinux                  rsv_window_remove
69       2.8e-05  vmlinux                  vmlinux                  rwsem_wake
69       2.8e-05  vmlinux                  vmlinux                  set_slab_attr
68       2.8e-05  libc-2.3.3.so            libc-2.3.3.so            __GI_strstr
68       2.8e-05  tiotest                  tiotest                  do_test
68       2.8e-05  vmlinux                  vmlinux                  __filemap_fdatawrite_range
68       2.8e-05  vmlinux                  vmlinux                  as_antic_stop
68       2.8e-05  vmlinux                  vmlinux                  cp_new_stat64
68       2.8e-05  vmlinux                  vmlinux                  in_group_p
68       2.8e-05  vmlinux                  vmlinux                  make_ahead_window
68       2.8e-05  vmlinux                  vmlinux                  need_resched
67       2.8e-05  vmlinux                  vmlinux                  elv_put_request
67       2.8e-05  vmlinux                  vmlinux                  locks_remove_posix
67       2.8e-05  vmlinux                  vmlinux                  may_expand_vm
67       2.8e-05  vmlinux                  vmlinux                  mntput_no_expire
66       2.7e-05  libc-2.3.3.so            libc-2.3.3.so            _IO_new_file_xsputn
66       2.7e-05  libc-2.3.3.so            libc-2.3.3.so            _int_memalign
66       2.7e-05  vmlinux                  vmlinux                  filemap_nopage
66       2.7e-05  vmlinux                  vmlinux                  sched_exit
65       2.7e-05  vmlinux                  vmlinux                  find_mergeable_anon_vma
64       2.6e-05  init                     init                     (no symbols)
64       2.6e-05  libc-2.3.3.so            libc-2.3.3.so            _IO_new_file_fopen
64       2.6e-05  vmlinux                  vmlinux                  detach_vmas_to_be_unmapped
63       2.6e-05  tiotest                  tiotest                  timer_usertime
63       2.6e-05  vmlinux                  vmlinux                  __pollwait
63       2.6e-05  vmlinux                  vmlinux                  __vm_enough_memory
62       2.6e-05  vmlinux                  vmlinux                  __log_start_commit
62       2.6e-05  vmlinux                  vmlinux                  filp_close
62       2.6e-05  vmlinux                  vmlinux                  radix_tree_node_alloc
61       2.5e-05  libc-2.3.3.so            libc-2.3.3.so            __i686.get_pc_thunk.cx
61       2.5e-05  vmlinux                  vmlinux                  next_thread
61       2.5e-05  vmlinux                  vmlinux                  smp_send_reschedule
60       2.5e-05  ld-2.3.3.so              ld-2.3.3.so              __GI__dl_allocate_tls
60       2.5e-05  vmlinux                  vmlinux                  ext3_getblk
59       2.4e-05  libc-2.3.3.so            libc-2.3.3.so            _IO_new_do_write
59       2.4e-05  vmlinux                  vmlinux                  __mutex_init
59       2.4e-05  vmlinux                  vmlinux                  alloc_pidmap
59       2.4e-05  vmlinux                  vmlinux                  groups_search
58       2.4e-05  vmlinux                  vmlinux                  __group_send_sig_info
58       2.4e-05  vmlinux                  vmlinux                  as_get_io_context
57       2.3e-05  vmlinux                  vmlinux                  handle_signal
57       2.3e-05  vmlinux                  vmlinux                  ioc_set_batching
56       2.3e-05  ld-2.3.3.so              ld-2.3.3.so              _dl_lookup_symbol_x
56       2.3e-05  libpthread-0.10.so       libpthread-0.10.so       __GI___pthread_mutex_unlock
56       2.3e-05  vmlinux                  vmlinux                  end_edge_ioapic_irq
56       2.3e-05  vmlinux                  vmlinux                  forget_original_parent
56       2.3e-05  vmlinux                  vmlinux                  unmap_mapping_range
55       2.3e-05  vmlinux                  vmlinux                  copy_thread
55       2.3e-05  vmlinux                  vmlinux                  get_device
55       2.3e-05  vmlinux                  vmlinux                  get_write_access
54       2.2e-05  libc-2.3.3.so            libc-2.3.3.so            __rpc_thread_destroy
54       2.2e-05  vmlinux                  vmlinux                  mod_timer
54       2.2e-05  vmlinux                  vmlinux                  second_overflow
54       2.2e-05  vmlinux                  vmlinux                  setup_sigcontext
53       2.2e-05  libc-2.3.3.so            libc-2.3.3.so            _IO_no_init
53       2.2e-05  vmlinux                  vmlinux                  __put_unused_fd
53       2.2e-05  vmlinux                  vmlinux                  do_signal
53       2.2e-05  vmlinux                  vmlinux                  sys_rt_sigsuspend
52       2.1e-05  libc-2.3.3.so            libc-2.3.3.so            __memalign
52       2.1e-05  libpthread-0.10.so       libpthread-0.10.so       __pthread_wait_for_restart_signal
52       2.1e-05  vmlinux                  vmlinux                  double_rq_lock
52       2.1e-05  vmlinux                  vmlinux                  e1000_alloc_rx_buffers
52       2.1e-05  vmlinux                  vmlinux                  nameidata_to_filp
52       2.1e-05  vmlinux                  vmlinux                  restore_sigcontext
52       2.1e-05  vmlinux                  vmlinux                  sync_sb_inodes
51       2.1e-05  libpthread-0.10.so       libpthread-0.10.so       anonymous symbol from section .plt
51       2.1e-05  vmlinux                  vmlinux                  journal_grab_journal_head
51       2.1e-05  vmlinux                  vmlinux                  mprotect_fixup
51       2.1e-05  vmlinux                  vmlinux                  proc_pid_unhash
51       2.1e-05  vmlinux                  vmlinux                  sig_ignored
51       2.1e-05  vmlinux                  vmlinux                  signal_wake_up
50       2.1e-05  oprofile.ko              oprofile                 task_exit_notify
50       2.1e-05  vmlinux                  vmlinux                  free_task
50       2.1e-05  vmlinux                  vmlinux                  mask_and_ack_level_ioapic_irq
50       2.1e-05  vmlinux                  vmlinux                  sys_getppid
49       2.0e-05  libc-2.3.3.so            libc-2.3.3.so            _IO_new_file_write
49       2.0e-05  vmlinux                  vmlinux                  find_group_other
49       2.0e-05  vmlinux                  vmlinux                  sys_exit_group
48       2.0e-05  ld-2.3.3.so              ld-2.3.3.so              fixup
48       2.0e-05  vmlinux                  vmlinux                  as_can_anticipate
48       2.0e-05  vmlinux                  vmlinux                  eventpoll_init_file
48       2.0e-05  vmlinux                  vmlinux                  free_pte_range
48       2.0e-05  vmlinux                  vmlinux                  get_vma_policy
48       2.0e-05  vmlinux                  vmlinux                  next_reap_node
48       2.0e-05  vmlinux                  vmlinux                  sys_sigreturn
47       1.9e-05  libc-2.3.3.so            libc-2.3.3.so            __funlockfile
46       1.9e-05  vmlinux                  vmlinux                  __page_set_anon_rmap
46       1.9e-05  vmlinux                  vmlinux                  journal_brelse_array
45       1.9e-05  vmlinux                  vmlinux                  do_munmap
45       1.9e-05  vmlinux                  vmlinux                  poll_freewait
45       1.9e-05  vmlinux                  vmlinux                  radix_tree_extend
44       1.8e-05  vmlinux                  vmlinux                  elv_dequeue_request
44       1.8e-05  vmlinux                  vmlinux                  get_unmapped_area
43       1.8e-05  libc-2.3.3.so            libc-2.3.3.so            __GI___getpagesize
43       1.8e-05  vmlinux                  vmlinux                  adjtime_adjustment
43       1.8e-05  vmlinux                  vmlinux                  free_pages
43       1.8e-05  vmlinux                  vmlinux                  journal_write_revoke_records
43       1.8e-05  vmlinux                  vmlinux                  truncate_inode_pages
42       1.7e-05  vmlinux                  vmlinux                  __add_entropy_words
42       1.7e-05  vmlinux                  vmlinux                  __writeback_single_inode
42       1.7e-05  vmlinux                  vmlinux                  sched_setscheduler
41       1.7e-05  libc-2.3.3.so            libc-2.3.3.so            _IO_file_doallocate_internal
41       1.7e-05  libc-2.3.3.so            libc-2.3.3.so            __libc_enable_asynccancel
41       1.7e-05  vmlinux                  vmlinux                  copy_mm
41       1.7e-05  vmlinux                  vmlinux                  ext3_find_near
41       1.7e-05  vmlinux                  vmlinux                  free_pages_and_swap_cache
41       1.7e-05  vmlinux                  vmlinux                  getrusage
41       1.7e-05  vmlinux                  vmlinux                  i8042_timer_func
41       1.7e-05  vmlinux                  vmlinux                  ip_route_input_slow
41       1.7e-05  vmlinux                  vmlinux                  scsi_delete_timer
40       1.6e-05  ld-2.3.3.so              ld-2.3.3.so              __GI__dl_deallocate_tls
40       1.6e-05  libc-2.3.3.so            libc-2.3.3.so            __fopen_internal
40       1.6e-05  vmlinux                  vmlinux                  arch_get_unmapped_area_topdown
40       1.6e-05  vmlinux                  vmlinux                  dup_mm
40       1.6e-05  vmlinux                  vmlinux                  unlink_file_vma
39       1.6e-05  vmlinux                  vmlinux                  __dequeue_signal
39       1.6e-05  vmlinux                  vmlinux                  __try_to_free_cp_buf
39       1.6e-05  vmlinux                  vmlinux                  cap_vm_enough_memory
39       1.6e-05  vmlinux                  vmlinux                  interleave_nodes
39       1.6e-05  vmlinux                  vmlinux                  sys_fstat64
38       1.6e-05  libpthread-0.10.so       libpthread-0.10.so       pthread_free
38       1.6e-05  vmlinux                  vmlinux                  free_uid
38       1.6e-05  vmlinux                  vmlinux                  generic_fillattr
38       1.6e-05  vmlinux                  vmlinux                  locks_remove_flock
38       1.6e-05  vmlinux                  vmlinux                  put_device
38       1.6e-05  vmlinux                  vmlinux                  vmtruncate
37       1.5e-05  libc-2.3.3.so            libc-2.3.3.so            __GI__IO_file_open
37       1.5e-05  vmlinux                  vmlinux                  exit_itimers
37       1.5e-05  vmlinux                  vmlinux                  get_nr_files
37       1.5e-05  vmlinux                  vmlinux                  kmem_flagcheck
37       1.5e-05  vmlinux                  vmlinux                  netif_rx
37       1.5e-05  vmlinux                  vmlinux                  process_timeout
36       1.5e-05  libc-2.3.3.so            libc-2.3.3.so            _IO_default_finish_internal
36       1.5e-05  vmlinux                  vmlinux                  do_mpage_readpage
35       1.4e-05  libc-2.3.3.so            libc-2.3.3.so            _IO_vfprintf_internal
35       1.4e-05  vmlinux                  vmlinux                  __vma_link
35       1.4e-05  vmlinux                  vmlinux                  debug_mutex_init
35       1.4e-05  vmlinux                  vmlinux                  ext3_add_entry
35       1.4e-05  vmlinux                  vmlinux                  sys_getpid
34       1.4e-05  libc-2.3.3.so            libc-2.3.3.so            _IO_setb_internal
34       1.4e-05  vmlinux                  vmlinux                  __exit_sighand
34       1.4e-05  vmlinux                  vmlinux                  __unhash_process
34       1.4e-05  vmlinux                  vmlinux                  do_fork
34       1.4e-05  vmlinux                  vmlinux                  lock_hrtimer_base
33       1.4e-05  libc-2.3.3.so            libc-2.3.3.so            __GI_memset
33       1.4e-05  tiotest                  tiotest                  cleanup_test
33       1.4e-05  vmlinux                  vmlinux                  __insert_inode_hash
33       1.4e-05  vmlinux                  vmlinux                  file_kill
33       1.4e-05  vmlinux                  vmlinux                  mm_release
33       1.4e-05  vmlinux                  vmlinux                  next_signal
33       1.4e-05  vmlinux                  vmlinux                  pagevec_lookup
33       1.4e-05  vmlinux                  vmlinux                  rwsem_down_write_failed
33       1.4e-05  vmlinux                  vmlinux                  sched_fork
33       1.4e-05  vmlinux                  vmlinux                  vfs_getattr
32       1.3e-05  vmlinux                  vmlinux                  __vma_link_rb
32       1.3e-05  vmlinux                  vmlinux                  __wake_up_locked
32       1.3e-05  vmlinux                  vmlinux                  as_find_first_arq
32       1.3e-05  vmlinux                  vmlinux                  do_pollfd
32       1.3e-05  vmlinux                  vmlinux                  inode_change_ok
32       1.3e-05  vmlinux                  vmlinux                  may_delete
31       1.3e-05  libc-2.3.3.so            libc-2.3.3.so            __GI_strchr
31       1.3e-05  tiotest                  tiotest                  timer_stop
31       1.3e-05  vmlinux                  vmlinux                  free_pidmap
31       1.3e-05  vmlinux                  vmlinux                  journal_free_journal_head
31       1.3e-05  vmlinux                  vmlinux                  journal_switch_revoke_table
30       1.2e-05  vmlinux                  vmlinux                  dequeue_signal
30       1.2e-05  vmlinux                  vmlinux                  detach_pid
30       1.2e-05  vmlinux                  vmlinux                  e1000_check_for_link
30       1.2e-05  vmlinux                  vmlinux                  fn_hash_lookup
29       1.2e-05  vmlinux                  vmlinux                  netif_receive_skb
29       1.2e-05  vmlinux                  vmlinux                  rwsem_down_read_failed
28       1.2e-05  libc-2.3.3.so            libc-2.3.3.so            _IO_doallocbuf_internal
28       1.2e-05  libc-2.3.3.so            libc-2.3.3.so            __GI__exit
28       1.2e-05  vmlinux                  vmlinux                  capable
28       1.2e-05  vmlinux                  vmlinux                  debug_mutex_wake_waiter
28       1.2e-05  vmlinux                  vmlinux                  do_writepages
28       1.2e-05  vmlinux                  vmlinux                  nr_running
28       1.2e-05  vmlinux                  vmlinux                  path_lookup_open
28       1.2e-05  vmlinux                  vmlinux                  schedule_timeout_uninterruptible
27       1.1e-05  libpthread-0.10.so       libpthread-0.10.so       __errno_location
27       1.1e-05  vmlinux                  vmlinux                  __free_pages_ok
27       1.1e-05  vmlinux                  vmlinux                  do_sched_setscheduler
27       1.1e-05  vmlinux                  vmlinux                  exit_sem
27       1.1e-05  vmlinux                  vmlinux                  fib_semantic_match
27       1.1e-05  vmlinux                  vmlinux                  find_idlest_cpu
27       1.1e-05  vmlinux                  vmlinux                  new_inode
27       1.1e-05  vmlinux                  vmlinux                  page_add_file_rmap
27       1.1e-05  vmlinux                  vmlinux                  read_inode_bitmap
27       1.1e-05  vmlinux                  vmlinux                  start_transaction
26       1.1e-05  libc-2.3.3.so            libc-2.3.3.so            _IO_old_init
26       1.1e-05  libpthread-0.10.so       libpthread-0.10.so       pthread_getspecific
26       1.1e-05  vmlinux                  vmlinux                  arp_process
26       1.1e-05  vmlinux                  vmlinux                  do_notify_resume
26       1.1e-05  vmlinux                  vmlinux                  journal_forget
26       1.1e-05  vmlinux                  vmlinux                  lock_kernel
26       1.1e-05  vmlinux                  vmlinux                  sys_poll
25       1.0e-05  libc-2.3.3.so            libc-2.3.3.so            _IO_file_stat_internal
25       1.0e-05  libpthread-0.10.so       libpthread-0.10.so       __pthread_perform_cleanup
25       1.0e-05  tiotest                  tiotest                  initialize_test
25       1.0e-05  tiotest                  tiotest                  timer_start
25       1.0e-05  vmlinux                  vmlinux                  copy_fdtable
25       1.0e-05  vmlinux                  vmlinux                  default_llseek
25       1.0e-05  vmlinux                  vmlinux                  ext3_free_inode
25       1.0e-05  vmlinux                  vmlinux                  pipe_iov_copy_from_user
25       1.0e-05  vmlinux                  vmlinux                  queue_work
25       1.0e-05  vmlinux                  vmlinux                  sys_kill
24       9.9e-06  libc-2.3.3.so            libc-2.3.3.so            __GI__dl_mcount_wrapper_check
24       9.9e-06  vmlinux                  vmlinux                  blk_phys_contig_segment
24       9.9e-06  vmlinux                  vmlinux                  copy_files
24       9.9e-06  vmlinux                  vmlinux                  ext3_create
24       9.9e-06  vmlinux                  vmlinux                  kill_proc_info
24       9.9e-06  vmlinux                  vmlinux                  recalc_sigpending
24       9.9e-06  vmlinux                  vmlinux                  sigprocmask
24       9.9e-06  vmlinux                  vmlinux                  write_inode
23       9.5e-06  libc-2.3.3.so            libc-2.3.3.so            _IO_new_fopen
23       9.5e-06  libc-2.3.3.so            libc-2.3.3.so            _IO_unsave_markers_internal
23       9.5e-06  libc-2.3.3.so            libc-2.3.3.so            __GI_strcmp
23       9.5e-06  vmlinux                  vmlinux                  __alloc_skb
23       9.5e-06  vmlinux                  vmlinux                  cached_lookup
23       9.5e-06  vmlinux                  vmlinux                  do_getname
23       9.5e-06  vmlinux                  vmlinux                  free_hot_page
23       9.5e-06  vmlinux                  vmlinux                  hrtimer_cancel
23       9.5e-06  vmlinux                  vmlinux                  ip_route_input
23       9.5e-06  vmlinux                  vmlinux                  preempt_schedule_irq
23       9.5e-06  vmlinux                  vmlinux                  sys_fsync
22       9.1e-06  vmlinux                  vmlinux                  anon_pipe_buf_map
22       9.1e-06  vmlinux                  vmlinux                  blocks_for_truncate
22       9.1e-06  vmlinux                  vmlinux                  do_truncate
22       9.1e-06  vmlinux                  vmlinux                  ext3_delete_entry
22       9.1e-06  vmlinux                  vmlinux                  lookup_hash
22       9.1e-06  vmlinux                  vmlinux                  slab_node
21       8.6e-06  libc-2.3.3.so            libc-2.3.3.so            __GI___unlink
21       8.6e-06  libpthread-0.10.so       libpthread-0.10.so       __pthread_restart_new
21       8.6e-06  oprofile.ko              oprofile                 process_task_mortuary
21       8.6e-06  vmlinux                  vmlinux                  __copy_user_zeroing_intel
21       8.6e-06  vmlinux                  vmlinux                  __wake_up_sync
21       8.6e-06  vmlinux                  vmlinux                  anon_pipe_buf_unmap
21       8.6e-06  vmlinux                  vmlinux                  copy_namespace
21       8.6e-06  vmlinux                  vmlinux                  sync_supers
21       8.6e-06  vmlinux                  vmlinux                  sys_sched_setscheduler
20       8.2e-06  ld-2.3.3.so              ld-2.3.3.so              anonymous symbol from section .plt
20       8.2e-06  vmlinux                  vmlinux                  can_vma_merge_before
20       8.2e-06  vmlinux                  vmlinux                  e1000_clean_tx_irq
20       8.2e-06  vmlinux                  vmlinux                  e1000_read_phy_reg_ex
20       8.2e-06  vmlinux                  vmlinux                  kill_fasync
20       8.2e-06  vmlinux                  vmlinux                  save_i387
19       7.8e-06  gawk                     gawk                     (no symbols)
19       7.8e-06  libpthread-0.10.so       libpthread-0.10.so       __pthread_internal_tsd_set
19       7.8e-06  vmlinux                  vmlinux                  _spin_trylock
19       7.8e-06  vmlinux                  vmlinux                  blk_plug_device
19       7.8e-06  vmlinux                  vmlinux                  check_kill_permission
19       7.8e-06  vmlinux                  vmlinux                  expand_files
19       7.8e-06  vmlinux                  vmlinux                  file_free_rcu
19       7.8e-06  vmlinux                  vmlinux                  find_revoke_record
19       7.8e-06  vmlinux                  vmlinux                  find_task_by_pid_type
19       7.8e-06  vmlinux                  vmlinux                  memset
18       7.4e-06  libc-2.3.3.so            libc-2.3.3.so            _IO_new_file_init
18       7.4e-06  vmlinux                  vmlinux                  alloc_as_io_context
18       7.4e-06  vmlinux                  vmlinux                  arp_rcv
18       7.4e-06  vmlinux                  vmlinux                  copy_semundo
18       7.4e-06  vmlinux                  vmlinux                  do_unlinkat
18       7.4e-06  vmlinux                  vmlinux                  dup_fd
18       7.4e-06  vmlinux                  vmlinux                  exit_as_io_context
18       7.4e-06  vmlinux                  vmlinux                  journal_end_buffer_io_sync
18       7.4e-06  vmlinux                  vmlinux                  slab_destroy_objs
17       7.0e-06  libc-2.3.3.so            libc-2.3.3.so            _IO_file_finish_internal
17       7.0e-06  libc-2.3.3.so            libc-2.3.3.so            anonymous symbol from section .plt
17       7.0e-06  libcrypto.so.0.9.7a      libcrypto.so.0.9.7a      (no symbols)
17       7.0e-06  libpthread-0.10.so       libpthread-0.10.so       __GI___pthread_mutex_lock
17       7.0e-06  tiotest                  tiotest                  print_results
17       7.0e-06  vmlinux                  vmlinux                  __get_free_pages
17       7.0e-06  vmlinux                  vmlinux                  ext3_init_acl
17       7.0e-06  vmlinux                  vmlinux                  ext3_unlink
17       7.0e-06  vmlinux                  vmlinux                  mapping_tagged
17       7.0e-06  vmlinux                  vmlinux                  neigh_lookup
17       7.0e-06  vmlinux                  vmlinux                  pmd_ctor
17       7.0e-06  vmlinux                  vmlinux                  strnlen_user
16       6.6e-06  libc-2.3.3.so            libc-2.3.3.so            __GI___printf_fp
16       6.6e-06  vmlinux                  vmlinux                  __insert_vm_struct
16       6.6e-06  vmlinux                  vmlinux                  _read_unlock_irqrestore
16       6.6e-06  vmlinux                  vmlinux                  as_antic_timeout
16       6.6e-06  vmlinux                  vmlinux                  d_lookup
16       6.6e-06  vmlinux                  vmlinux                  inotify_inode_is_dead
16       6.6e-06  vmlinux                  vmlinux                  process_backlog
15       6.2e-06  vmlinux                  vmlinux                  as_work_handler
15       6.2e-06  vmlinux                  vmlinux                  cap_capable
15       6.2e-06  vmlinux                  vmlinux                  double_rq_unlock
15       6.2e-06  vmlinux                  vmlinux                  exit_thread
15       6.2e-06  vmlinux                  vmlinux                  pipe_iov_copy_to_user
15       6.2e-06  vmlinux                  vmlinux                  release_vm86_irqs
15       6.2e-06  vmlinux                  vmlinux                  rt_hash_code
15       6.2e-06  vmlinux                  vmlinux                  work_notifysig
14       5.8e-06  libc-2.3.3.so            libc-2.3.3.so            __find_specmb
14       5.8e-06  vmlinux                  vmlinux                  d_instantiate
14       5.8e-06  vmlinux                  vmlinux                  ext3_set_inode_flags
14       5.8e-06  vmlinux                  vmlinux                  ret_from_exception
14       5.8e-06  vmlinux                  vmlinux                  sys_clone
14       5.8e-06  vmlinux                  vmlinux                  sys_waitpid
14       5.8e-06  vmlinux                  vmlinux                  try_to_extend_transaction
13       5.4e-06  op_help                  op_help                  (no symbols)
13       5.4e-06  vmlinux                  vmlinux                  as_add_arq_rb
13       5.4e-06  vmlinux                  vmlinux                  eth_type_trans
13       5.4e-06  vmlinux                  vmlinux                  ext3_free_branches
13       5.4e-06  vmlinux                  vmlinux                  journal_release_buffer
13       5.4e-06  vmlinux                  vmlinux                  page_add_new_anon_rmap
12       4.9e-06  ld-2.3.3.so              ld-2.3.3.so              __i686.get_pc_thunk.bx
12       4.9e-06  ld-2.3.3.so              ld-2.3.3.so              _dl_runtime_resolve
12       4.9e-06  libc-2.3.3.so            libc-2.3.3.so            _IO_vsprintf_internal
12       4.9e-06  libc-2.3.3.so            libc-2.3.3.so            __GI_sigdelset
12       4.9e-06  libc-2.3.3.so            libc-2.3.3.so            __GI_strcpy
12       4.9e-06  vmlinux                  vmlinux                  count_open_files
12       4.9e-06  vmlinux                  vmlinux                  do_select
12       4.9e-06  vmlinux                  vmlinux                  ext3_clear_inode
12       4.9e-06  vmlinux                  vmlinux                  ext3_lookup
12       4.9e-06  vmlinux                  vmlinux                  ext3_write_inode
12       4.9e-06  vmlinux                  vmlinux                  grow_dev_page
12       4.9e-06  vmlinux                  vmlinux                  sha_transform
11       4.5e-06  ld-2.3.3.so              ld-2.3.3.so              _dl_map_object_from_fd
11       4.5e-06  libc-2.3.3.so            libc-2.3.3.so            __GI_memmove
11       4.5e-06  libpthread-0.10.so       libpthread-0.10.so       __pthread_manager_adjust_prio
11       4.5e-06  vmlinux                  vmlinux                  __getblk_slow
11       4.5e-06  vmlinux                  vmlinux                  __setscheduler
11       4.5e-06  vmlinux                  vmlinux                  e1000_watchdog_task
11       4.5e-06  vmlinux                  vmlinux                  journal_unfile_buffer
11       4.5e-06  vmlinux                  vmlinux                  kill_something_info
11       4.5e-06  vmlinux                  vmlinux                  migration_thread
11       4.5e-06  vmlinux                  vmlinux                  prepare_to_copy
11       4.5e-06  vmlinux                  vmlinux                  sys_wait4
11       4.5e-06  vmlinux                  vmlinux                  vfs_fstat
10       4.1e-06  ld-2.3.3.so              ld-2.3.3.so              _dl_load_cache_lookup
10       4.1e-06  ld-2.3.3.so              ld-2.3.3.so              _dl_name_match_p
10       4.1e-06  libc-2.3.3.so            libc-2.3.3.so            __libc_disable_asynccancel
10       4.1e-06  libpthread-0.10.so       libpthread-0.10.so       __pthread_manager_sighandler
10       4.1e-06  vmlinux                  vmlinux                  arch_unmap_area_topdown
10       4.1e-06  vmlinux                  vmlinux                  as_merged_requests
10       4.1e-06  vmlinux                  vmlinux                  ext3_add_nondir
10       4.1e-06  vmlinux                  vmlinux                  ext3_alloc_inode
10       4.1e-06  vmlinux                  vmlinux                  ext3_delete_inode
10       4.1e-06  vmlinux                  vmlinux                  insert_revoke_hash
10       4.1e-06  vmlinux                  vmlinux                  sync_inode
9        3.7e-06  ld-2.3.3.so              ld-2.3.3.so              dl_main
9        3.7e-06  libc-2.3.3.so            libc-2.3.3.so            __GI___sigaction
9        3.7e-06  oprofile.ko              oprofile                 task_free_notify
9        3.7e-06  sshd                     sshd                     (no symbols)
9        3.7e-06  vmlinux                  vmlinux                  clear_inode
9        3.7e-06  vmlinux                  vmlinux                  core_sys_select
9        3.7e-06  vmlinux                  vmlinux                  filemap_fdatawrite
9        3.7e-06  vmlinux                  vmlinux                  flush_sigqueue
9        3.7e-06  vmlinux                  vmlinux                  is_bad_inode
9        3.7e-06  vmlinux                  vmlinux                  kick_process
9        3.7e-06  vmlinux                  vmlinux                  prio_tree_insert
9        3.7e-06  vmlinux                  vmlinux                  sys_unlink
8        3.3e-06  ld-2.3.3.so              ld-2.3.3.so              _dl_cache_libcmp
8        3.3e-06  libc-2.3.3.so            libc-2.3.3.so            __GI_strlen
8        3.3e-06  libc-2.3.3.so            libc-2.3.3.so            __realloc
8        3.3e-06  libc-2.3.3.so            libc-2.3.3.so            getc
8        3.3e-06  oprofile.ko              oprofile                 .text.lock.event_buffer
8        3.3e-06  tiotest                  tiotest                  get_random_seed
8        3.3e-06  vmlinux                  vmlinux                  __bread_slow
8        3.3e-06  vmlinux                  vmlinux                  arp_hash
8        3.3e-06  vmlinux                  vmlinux                  can_vma_merge_after
8        3.3e-06  vmlinux                  vmlinux                  end_that_request_chunk
8        3.3e-06  vmlinux                  vmlinux                  journal_check_used_features
8        3.3e-06  vmlinux                  vmlinux                  journal_revoke
8        3.3e-06  vmlinux                  vmlinux                  kblockd_schedule_work
8        3.3e-06  vmlinux                  vmlinux                  path_release
8        3.3e-06  vmlinux                  vmlinux                  radix_tree_tagged
8        3.3e-06  vmlinux                  vmlinux                  release_buffer_page
8        3.3e-06  vmlinux                  vmlinux                  unlock_kernel
7        2.9e-06  ld-2.3.3.so              ld-2.3.3.so              match_symbol
7        2.9e-06  libc-2.3.3.so            libc-2.3.3.so            __fopen_maybe_mmap
7        2.9e-06  libc-2.3.3.so            libc-2.3.3.so            _int_realloc
7        2.9e-06  libc-2.3.3.so            libc-2.3.3.so            _itoa_word
7        2.9e-06  libc-2.3.3.so            libc-2.3.3.so            _nl_intern_locale_data
7        2.9e-06  vmlinux                  vmlinux                  __put_user_4
7        2.9e-06  vmlinux                  vmlinux                  anon_vma_link
7        2.9e-06  vmlinux                  vmlinux                  bmap
7        2.9e-06  vmlinux                  vmlinux                  d_rehash
7        2.9e-06  vmlinux                  vmlinux                  ext3_force_commit
7        2.9e-06  vmlinux                  vmlinux                  get_writeback_state
7        2.9e-06  vmlinux                  vmlinux                  init_page_buffers
7        2.9e-06  vmlinux                  vmlinux                  journal_force_commit
7        2.9e-06  vmlinux                  vmlinux                  pipe_read
7        2.9e-06  vmlinux                  vmlinux                  pte_alloc_one
7        2.9e-06  vmlinux                  vmlinux                  vfs_unlink
6        2.5e-06  grep                     grep                     (no symbols)
6        2.5e-06  libc-2.3.3.so            libc-2.3.3.so            __GI_getenv
6        2.5e-06  libc-2.3.3.so            libc-2.3.3.so            _xstat
6        2.5e-06  vmlinux                  vmlinux                  .text.lock.sys_i386
6        2.5e-06  vmlinux                  vmlinux                  __journal_insert_checkpoint
6        2.5e-06  vmlinux                  vmlinux                  __put_super_and_need_restart
6        2.5e-06  vmlinux                  vmlinux                  add_interrupt_randomness
6        2.5e-06  vmlinux                  vmlinux                  as_antic_waitreq
6        2.5e-06  vmlinux                  vmlinux                  free_one_page
6        2.5e-06  vmlinux                  vmlinux                  get_dirty_limits
6        2.5e-06  vmlinux                  vmlinux                  invalidate_inode_buffers
6        2.5e-06  vmlinux                  vmlinux                  n_tty_receive_buf
6        2.5e-06  vmlinux                  vmlinux                  old_mmap
6        2.5e-06  vmlinux                  vmlinux                  posix_cpu_timers_exit_group
6        2.5e-06  vmlinux                  vmlinux                  sys_stat64
6        2.5e-06  vmlinux                  vmlinux                  vfs_create
6        2.5e-06  vmlinux                  vmlinux                  vsnprintf
6        2.5e-06  vmlinux                  vmlinux                  wake_up_state
6        2.5e-06  vmlinux                  vmlinux                  writeback_in_progress
5        2.1e-06  ld-2.3.3.so              ld-2.3.3.so              _dl_map_object
5        2.1e-06  ld-2.3.3.so              ld-2.3.3.so              _dl_map_object_deps
5        2.1e-06  ld-2.3.3.so              ld-2.3.3.so              _dl_sysdep_start
5        2.1e-06  libc-2.3.3.so            libc-2.3.3.so            __GI___libc_sigaction
5        2.1e-06  libc-2.3.3.so            libc-2.3.3.so            __ctype_get_mb_cur_max
5        2.1e-06  libpthread-0.10.so       libpthread-0.10.so       __pthread_sigaction
5        2.1e-06  tiotest                  tiotest                  timer_systime
5        2.1e-06  vmlinux                  vmlinux                  __dispose_buffer
5        2.1e-06  vmlinux                  vmlinux                  __set_page_dirty_buffers
5        2.1e-06  vmlinux                  vmlinux                  __wait_on_bit
5        2.1e-06  vmlinux                  vmlinux                  balanced_irq
5        2.1e-06  vmlinux                  vmlinux                  blk_congestion_wait
5        2.1e-06  vmlinux                  vmlinux                  copy_strings
5        2.1e-06  vmlinux                  vmlinux                  double_lock_balance
5        2.1e-06  vmlinux                  vmlinux                  ext3_bmap
5        2.1e-06  vmlinux                  vmlinux                  ext3_set_aops
5        2.1e-06  vmlinux                  vmlinux                  find_or_create_page
5        2.1e-06  vmlinux                  vmlinux                  flush_signal_handlers
5        2.1e-06  vmlinux                  vmlinux                  get_init_ra_size
5        2.1e-06  vmlinux                  vmlinux                  grow_buffers
5        2.1e-06  vmlinux                  vmlinux                  half_md4_transform
5        2.1e-06  vmlinux                  vmlinux                  inode_has_buffers
5        2.1e-06  vmlinux                  vmlinux                  journal_bmap
5        2.1e-06  vmlinux                  vmlinux                  journal_next_log_block
5        2.1e-06  vmlinux                  vmlinux                  net_rx_action
5        2.1e-06  vmlinux                  vmlinux                  number
5        2.1e-06  vmlinux                  vmlinux                  posix_cpu_timers_exit
5        2.1e-06  vmlinux                  vmlinux                  ret_from_fork
4        1.6e-06  ld-2.3.3.so              ld-2.3.3.so              __open
4        1.6e-06  ld-2.3.3.so              ld-2.3.3.so              _dl_check_map_versions
4        1.6e-06  ld-2.3.3.so              ld-2.3.3.so              _dl_important_hwcaps
4        1.6e-06  libc-2.3.3.so            libc-2.3.3.so            _IO_vfscanf_internal
4        1.6e-06  libc-2.3.3.so            libc-2.3.3.so            __GI___errno_location
4        1.6e-06  libc-2.3.3.so            libc-2.3.3.so            __GI_close
4        1.6e-06  libc-2.3.3.so            libc-2.3.3.so            __GI_select
4        1.6e-06  libc-2.3.3.so            libc-2.3.3.so            _getopt_internal_r
4        1.6e-06  libpthread-0.10.so       libpthread-0.10.so       __pthread_attr_destroy
4        1.6e-06  syslogd                  syslogd                  (no symbols)
4        1.6e-06  vmlinux                  vmlinux                  __const_udelay
4        1.6e-06  vmlinux                  vmlinux                  __get_user_4
4        1.6e-06  vmlinux                  vmlinux                  __journal_remove_checkpoint
4        1.6e-06  vmlinux                  vmlinux                  _read_unlock_bh
4        1.6e-06  vmlinux                  vmlinux                  anon_pipe_buf_release
4        1.6e-06  vmlinux                  vmlinux                  as_antic_waitnext
4        1.6e-06  vmlinux                  vmlinux                  blk_hw_contig_segment
4        1.6e-06  vmlinux                  vmlinux                  e1000_read_phy_reg
4        1.6e-06  vmlinux                  vmlinux                  elv_merge_requests
4        1.6e-06  vmlinux                  vmlinux                  expand_fdtable
4        1.6e-06  vmlinux                  vmlinux                  ext3_bread
4        1.6e-06  vmlinux                  vmlinux                  ext3_init_block_alloc_info
4        1.6e-06  vmlinux                  vmlinux                  flush_tlb_all
4        1.6e-06  vmlinux                  vmlinux                  generic_block_bmap
4        1.6e-06  vmlinux                  vmlinux                  journal_write_commit_record
4        1.6e-06  vmlinux                  vmlinux                  kjournald
4        1.6e-06  vmlinux                  vmlinux                  prio_tree_remove
4        1.6e-06  vmlinux                  vmlinux                  release_thread
4        1.6e-06  vmlinux                  vmlinux                  tcp_sendmsg
4        1.6e-06  vmlinux                  vmlinux                  vma_prio_tree_remove
3        1.2e-06  ld-2.3.3.so              ld-2.3.3.so              _dl_start
3        1.2e-06  ld-2.3.3.so              ld-2.3.3.so              index
3        1.2e-06  ld-2.3.3.so              ld-2.3.3.so              malloc
3        1.2e-06  libc-2.3.3.so            libc-2.3.3.so            __GI___default_morecore
3        1.2e-06  libc-2.3.3.so            libc-2.3.3.so            __GI_mempcpy
3        1.2e-06  libc-2.3.3.so            libc-2.3.3.so            __GI_sigemptyset
3        1.2e-06  libc-2.3.3.so            libc-2.3.3.so            __GI_strcat
3        1.2e-06  libc-2.3.3.so            libc-2.3.3.so            __mkdir
3        1.2e-06  libc-2.3.3.so            libc-2.3.3.so            __xstat32_conv
3        1.2e-06  libc-2.3.3.so            libc-2.3.3.so            _nl_load_locale_from_archive
3        1.2e-06  ls                       ls                       (no symbols)
3        1.2e-06  vmlinux                  vmlinux                  __kfree_skb
3        1.2e-06  vmlinux                  vmlinux                  __user_walk_fd
3        1.2e-06  vmlinux                  vmlinux                  _read_lock_bh
3        1.2e-06  vmlinux                  vmlinux                  active_load_balance
3        1.2e-06  vmlinux                  vmlinux                  add_to_page_cache_lru
3        1.2e-06  vmlinux                  vmlinux                  credit_entropy_store
3        1.2e-06  vmlinux                  vmlinux                  d_path
3        1.2e-06  vmlinux                  vmlinux                  d_splice_alias
3        1.2e-06  vmlinux                  vmlinux                  dentry_iput
3        1.2e-06  vmlinux                  vmlinux                  dev_watchdog
3        1.2e-06  vmlinux                  vmlinux                  exit_mmap
3        1.2e-06  vmlinux                  vmlinux                  ext3_xattr_delete_inode
3        1.2e-06  vmlinux                  vmlinux                  fasync_helper
3        1.2e-06  vmlinux                  vmlinux                  generic_delete_inode
3        1.2e-06  vmlinux                  vmlinux                  io_schedule_timeout
3        1.2e-06  vmlinux                  vmlinux                  ip_local_deliver
3        1.2e-06  vmlinux                  vmlinux                  journal_set_features
3        1.2e-06  vmlinux                  vmlinux                  load_elf_binary
3        1.2e-06  vmlinux                  vmlinux                  lru_cache_add
3        1.2e-06  vmlinux                  vmlinux                  neigh_periodic_timer
3        1.2e-06  vmlinux                  vmlinux                  nr_uninterruptible
3        1.2e-06  vmlinux                  vmlinux                  opost_block
3        1.2e-06  vmlinux                  vmlinux                  path_lookup_create
3        1.2e-06  vmlinux                  vmlinux                  pipe_write
3        1.2e-06  vmlinux                  vmlinux                  poll_initwait
3        1.2e-06  vmlinux                  vmlinux                  prepare_binprm
3        1.2e-06  vmlinux                  vmlinux                  proc_lookup
3        1.2e-06  vmlinux                  vmlinux                  profile_handoff_task
3        1.2e-06  vmlinux                  vmlinux                  schedule_work
3        1.2e-06  vmlinux                  vmlinux                  skb_release_data
3        1.2e-06  vmlinux                  vmlinux                  sys_rt_sigaction
3        1.2e-06  vmlinux                  vmlinux                  sys_select
3        1.2e-06  vmlinux                  vmlinux                  syscall_exit_work
3        1.2e-06  vmlinux                  vmlinux                  write_one_revoke_record
2        8.2e-07  ld-2.3.3.so              ld-2.3.3.so              ___xstat64
2        8.2e-07  ld-2.3.3.so              ld-2.3.3.so              __libc_close
2        8.2e-07  ld-2.3.3.so              ld-2.3.3.so              __libc_memalign
2        8.2e-07  ld-2.3.3.so              ld-2.3.3.so              _dl_fini
2        8.2e-07  ld-2.3.3.so              ld-2.3.3.so              _dl_next_ld_env_entry
2        8.2e-07  ld-2.3.3.so              ld-2.3.3.so              _dl_protect_relro
2        8.2e-07  ld-2.3.3.so              ld-2.3.3.so              strlen
2        8.2e-07  libc-2.3.3.so            libc-2.3.3.so            _IO_flush_all_lockp
2        8.2e-07  libc-2.3.3.so            libc-2.3.3.so            _IO_sputbackc_internal
2        8.2e-07  libc-2.3.3.so            libc-2.3.3.so            _IO_vsscanf
2        8.2e-07  libc-2.3.3.so            libc-2.3.3.so            __GI___fork
2        8.2e-07  libc-2.3.3.so            libc-2.3.3.so            __GI___strtol_internal
2        8.2e-07  libc-2.3.3.so            libc-2.3.3.so            __GI_stpcpy
2        8.2e-07  libc-2.3.3.so            libc-2.3.3.so            __GI_time
2        8.2e-07  libc-2.3.3.so            libc-2.3.3.so            __ctype_b_loc
2        8.2e-07  libc-2.3.3.so            libc-2.3.3.so            __glob64
2        8.2e-07  libc-2.3.3.so            libc-2.3.3.so            __libc_nanosleep
2        8.2e-07  libc-2.3.3.so            libc-2.3.3.so            __mpn_divrem
2        8.2e-07  libc-2.3.3.so            libc-2.3.3.so            __mpn_mul_1
2        8.2e-07  libc-2.3.3.so            libc-2.3.3.so            __new_exitfn
2        8.2e-07  libc-2.3.3.so            libc-2.3.3.so            __tcgetattr
2        8.2e-07  libc-2.3.3.so            libc-2.3.3.so            _getopt_internal
2        8.2e-07  libc-2.3.3.so            libc-2.3.3.so            malloc_hook_ini
2        8.2e-07  libc-2.3.3.so            libc-2.3.3.so            msort_with_tmp
2        8.2e-07  libc-2.3.3.so            libc-2.3.3.so            printf
2        8.2e-07  libpopt.so.0.0.0         libpopt.so.0.0.0         (no symbols)
2        8.2e-07  libpthread-0.10.so       libpthread-0.10.so       __pthread_attr_init_2_1
2        8.2e-07  libtermcap.so.2.0.8      libtermcap.so.2.0.8      (no symbols)
2        8.2e-07  vmlinux                  vmlinux                  .text.lock.exit
2        8.2e-07  vmlinux                  vmlinux                  __journal_refile_buffer
2        8.2e-07  vmlinux                  vmlinux                  __migrate_task
2        8.2e-07  vmlinux                  vmlinux                  __mmdrop
2        8.2e-07  vmlinux                  vmlinux                  __remove_shared_vm_struct
2        8.2e-07  vmlinux                  vmlinux                  alloc_page_interleave
2        8.2e-07  vmlinux                  vmlinux                  arch_pick_mmap_layout
2        8.2e-07  vmlinux                  vmlinux                  as_antic_expired
2        8.2e-07  vmlinux                  vmlinux                  balance_dirty_pages
2        8.2e-07  vmlinux                  vmlinux                  block_read_full_page
2        8.2e-07  vmlinux                  vmlinux                  cond_resched_lock
2        8.2e-07  vmlinux                  vmlinux                  copy_page_range
2        8.2e-07  vmlinux                  vmlinux                  count
2        8.2e-07  vmlinux                  vmlinux                  destroy_inode
2        8.2e-07  vmlinux                  vmlinux                  do_brk
2        8.2e-07  vmlinux                  vmlinux                  do_setitimer
2        8.2e-07  vmlinux                  vmlinux                  do_sigaction
2        8.2e-07  vmlinux                  vmlinux                  e1000_update_adaptive
2        8.2e-07  vmlinux                  vmlinux                  e1000_watchdog
2        8.2e-07  vmlinux                  vmlinux                  exec_mmap
2        8.2e-07  vmlinux                  vmlinux                  flush_old_exec
2        8.2e-07  vmlinux                  vmlinux                  get_transaction
2        8.2e-07  vmlinux                  vmlinux                  hrtimer_nanosleep
2        8.2e-07  vmlinux                  vmlinux                  init_new_context
2        8.2e-07  vmlinux                  vmlinux                  init_once
2        8.2e-07  vmlinux                  vmlinux                  ip_output
2        8.2e-07  vmlinux                  vmlinux                  ip_rcv
2        8.2e-07  vmlinux                  vmlinux                  jbd_unexpected_dirty_buffer
2        8.2e-07  vmlinux                  vmlinux                  ll_merge_requests_fn
2        8.2e-07  vmlinux                  vmlinux                  max_select_fd
2        8.2e-07  vmlinux                  vmlinux                  out_of_line_wait_on_bit
2        8.2e-07  vmlinux                  vmlinux                  prio_tree_replace
2        8.2e-07  vmlinux                  vmlinux                  read_chan
2        8.2e-07  vmlinux                  vmlinux                  search_binary_handler
2        8.2e-07  vmlinux                  vmlinux                  search_exception_tables
2        8.2e-07  vmlinux                  vmlinux                  sock_poll
2        8.2e-07  vmlinux                  vmlinux                  swap_io_context
2        8.2e-07  vmlinux                  vmlinux                  sys_alarm
2        8.2e-07  vmlinux                  vmlinux                  sys_dup2
2        8.2e-07  vmlinux                  vmlinux                  tcp_write_xmit
2        8.2e-07  vmlinux                  vmlinux                  tty_default_put_char
2        8.2e-07  vmlinux                  vmlinux                  tty_ldisc_ref_wait
2        8.2e-07  vmlinux                  vmlinux                  update_write_batch
2        8.2e-07  vmlinux                  vmlinux                  wb_kupdate
2        8.2e-07  vmlinux                  vmlinux                  wb_timer_fn
1        4.1e-07  binfmt_misc.ko           binfmt_misc              load_misc_binary
1        4.1e-07  cat                      cat                      (no symbols)
1        4.1e-07  cut                      cut                      (no symbols)
1        4.1e-07  expr                     expr                     (no symbols)
1        4.1e-07  ld-2.3.3.so              ld-2.3.3.so              __brk
1        4.1e-07  ld-2.3.3.so              ld-2.3.3.so              __mmap
1        4.1e-07  ld-2.3.3.so              ld-2.3.3.so              _dl_check_all_versions
1        4.1e-07  ld-2.3.3.so              ld-2.3.3.so              _dl_init_paths
1        4.1e-07  ld-2.3.3.so              ld-2.3.3.so              _dl_initial_error_catch_tsd
1        4.1e-07  ld-2.3.3.so              ld-2.3.3.so              _dl_new_object
1        4.1e-07  ld-2.3.3.so              ld-2.3.3.so              _dl_setup_hash
1        4.1e-07  ld-2.3.3.so              ld-2.3.3.so              access
1        4.1e-07  ld-2.3.3.so              ld-2.3.3.so              memcpy
1        4.1e-07  ld-2.3.3.so              ld-2.3.3.so              open_path
1        4.1e-07  libc-2.3.3.so            libc-2.3.3.so            _IO_ferror
1        4.1e-07  libc-2.3.3.so            libc-2.3.3.so            _IO_file_sync_internal
1        4.1e-07  libc-2.3.3.so            libc-2.3.3.so            _IO_getline_info_internal
1        4.1e-07  libc-2.3.3.so            libc-2.3.3.so            _IO_str_init_static_internal
1        4.1e-07  libc-2.3.3.so            libc-2.3.3.so            _IO_vsnprintf
1        4.1e-07  libc-2.3.3.so            libc-2.3.3.so            __GI__IO_list_resetlock
1        4.1e-07  libc-2.3.3.so            libc-2.3.3.so            __GI___fxstat
1        4.1e-07  libc-2.3.3.so            libc-2.3.3.so            __GI___isnan
1        4.1e-07  libc-2.3.3.so            libc-2.3.3.so            __GI___nss_rpc_lookup
1        4.1e-07  libc-2.3.3.so            libc-2.3.3.so            __GI___overflow
1        4.1e-07  libc-2.3.3.so            libc-2.3.3.so            __GI___strcasecmp
1        4.1e-07  libc-2.3.3.so            libc-2.3.3.so            __GI___strcoll_l
1        4.1e-07  libc-2.3.3.so            libc-2.3.3.so            __GI___strtoll_internal
1        4.1e-07  libc-2.3.3.so            libc-2.3.3.so            __GI___sysconf
1        4.1e-07  libc-2.3.3.so            libc-2.3.3.so            __GI___uflow
1        4.1e-07  libc-2.3.3.so            libc-2.3.3.so            __GI_exit
1        4.1e-07  libc-2.3.3.so            libc-2.3.3.so            __GI_inet_pton
1        4.1e-07  libc-2.3.3.so            libc-2.3.3.so            __GI_memchr
1        4.1e-07  libc-2.3.3.so            libc-2.3.3.so            __GI_setlocale
1        4.1e-07  libc-2.3.3.so            libc-2.3.3.so            __GI_sigaddset
1        4.1e-07  libc-2.3.3.so            libc-2.3.3.so            __GI_sprintf
1        4.1e-07  libc-2.3.3.so            libc-2.3.3.so            __GI_strncmp
1        4.1e-07  libc-2.3.3.so            libc-2.3.3.so            __GI_strrchr
1        4.1e-07  libc-2.3.3.so            libc-2.3.3.so            __GI_uname
1        4.1e-07  libc-2.3.3.so            libc-2.3.3.so            ____strtol_l_internal
1        4.1e-07  libc-2.3.3.so            libc-2.3.3.so            ____strtold_l_internal
1        4.1e-07  libc-2.3.3.so            libc-2.3.3.so            ____strtoul_l_internal
1        4.1e-07  libc-2.3.3.so            libc-2.3.3.so            ___xstat64
1        4.1e-07  libc-2.3.3.so            libc-2.3.3.so            __argp_parse
1        4.1e-07  libc-2.3.3.so            libc-2.3.3.so            __atomic_readv_replacement
1        4.1e-07  libc-2.3.3.so            libc-2.3.3.so            __btowc
1        4.1e-07  libc-2.3.3.so            libc-2.3.3.so            __cxa_finalize
1        4.1e-07  libc-2.3.3.so            libc-2.3.3.so            __fgetspent_r
1        4.1e-07  libc-2.3.3.so            libc-2.3.3.so            __ftruncate
1        4.1e-07  libc-2.3.3.so            libc-2.3.3.so            __gconv_load_cache
1        4.1e-07  libc-2.3.3.so            libc-2.3.3.so            __gconv_lookup_cache
1        4.1e-07  libc-2.3.3.so            libc-2.3.3.so            __gen_tempname
1        4.1e-07  libc-2.3.3.so            libc-2.3.3.so            __getdents
1        4.1e-07  libc-2.3.3.so            libc-2.3.3.so            __isatty
1        4.1e-07  libc-2.3.3.so            libc-2.3.3.so            __libc_malloc_pthread_startup
1        4.1e-07  libc-2.3.3.so            libc-2.3.3.so            __mmap64
1        4.1e-07  libc-2.3.3.so            libc-2.3.3.so            __mpn_extract_double
1        4.1e-07  libc-2.3.3.so            libc-2.3.3.so            __mpn_lshift
1        4.1e-07  libc-2.3.3.so            libc-2.3.3.so            __mpn_rshift
1        4.1e-07  libc-2.3.3.so            libc-2.3.3.so            __on_exit
1        4.1e-07  libc-2.3.3.so            libc-2.3.3.so            __snprintf
1        4.1e-07  libc-2.3.3.so            libc-2.3.3.so            __strchrnul
1        4.1e-07  libc-2.3.3.so            libc-2.3.3.so            __wcsmbs_load_conv
1        4.1e-07  libc-2.3.3.so            libc-2.3.3.so            _des_crypt
1        4.1e-07  libc-2.3.3.so            libc-2.3.3.so            _nl_find_msg
1        4.1e-07  libc-2.3.3.so            libc-2.3.3.so            _nl_load_domain
1        4.1e-07  libc-2.3.3.so            libc-2.3.3.so            _nl_make_l10nflist
1        4.1e-07  libc-2.3.3.so            libc-2.3.3.so            _nl_normalize_codeset
1        4.1e-07  libc-2.3.3.so            libc-2.3.3.so            argp_default_parser
1        4.1e-07  libc-2.3.3.so            libc-2.3.3.so            clearerr
1        4.1e-07  libc-2.3.3.so            libc-2.3.3.so            fscanf
1        4.1e-07  libc-2.3.3.so            libc-2.3.3.so            getopt_long
1        4.1e-07  libc-2.3.3.so            libc-2.3.3.so            key_get_conv
1        4.1e-07  libc-2.3.3.so            libc-2.3.3.so            new_composite_name
1        4.1e-07  libc-2.3.3.so            libc-2.3.3.so            phys_pages_info
1        4.1e-07  libc-2.3.3.so            libc-2.3.3.so            ptmalloc_init
1        4.1e-07  libcrypt-2.3.3.so        libcrypt-2.3.3.so        __do_global_ctors_aux
1        4.1e-07  libdl-2.3.3.so           libdl-2.3.3.so           frame_dummy
1        4.1e-07  libpthread-0.10.so       libpthread-0.10.so       __GI___pthread_mutex_init
1        4.1e-07  libpthread-0.10.so       libpthread-0.10.so       __fresetlockfiles
1        4.1e-07  libpthread-0.10.so       libpthread-0.10.so       __pthread_attr_setscope
1        4.1e-07  libpthread-0.10.so       libpthread-0.10.so       __pthread_init_max_stacksize
1        4.1e-07  libpthread-0.10.so       libpthread-0.10.so       _fini
1        4.1e-07  libpthread-0.10.so       libpthread-0.10.so       call_initialize_minimal
1        4.1e-07  libpthread-0.10.so       libpthread-0.10.so       fork
1        4.1e-07  libselinux.so.1          libselinux.so.1          (no symbols)
1        4.1e-07  mkdir                    mkdir                    (no symbols)
1        4.1e-07  tiotest                  tiotest                  do_tests
1        4.1e-07  tiotest                  tiotest                  main
1        4.1e-07  tiotest                  tiotest                  timer_realtime
1        4.1e-07  vmlinux                  vmlinux                  .text.lock.mprotect
1        4.1e-07  vmlinux                  vmlinux                  __journal_drop_transaction
1        4.1e-07  vmlinux                  vmlinux                  __modify_IO_APIC_irq
1        4.1e-07  vmlinux                  vmlinux                  __tcp_push_pending_frames
1        4.1e-07  vmlinux                  vmlinux                  __wait_on_buffer
1        4.1e-07  vmlinux                  vmlinux                  _write_lock
1        4.1e-07  vmlinux                  vmlinux                  _write_unlock
1        4.1e-07  vmlinux                  vmlinux                  alloc_fdset
1        4.1e-07  vmlinux                  vmlinux                  blk_unplug_timeout
1        4.1e-07  vmlinux                  vmlinux                  blkdev_writepage
1        4.1e-07  vmlinux                  vmlinux                  cap_bprm_apply_creds
1        4.1e-07  vmlinux                  vmlinux                  change_pte_range
1        4.1e-07  vmlinux                  vmlinux                  chrdev_open
1        4.1e-07  vmlinux                  vmlinux                  clear_user
1        4.1e-07  vmlinux                  vmlinux                  copy_strings_kernel
1        4.1e-07  vmlinux                  vmlinux                  copy_vma
1        4.1e-07  vmlinux                  vmlinux                  create_elf_tables
1        4.1e-07  vmlinux                  vmlinux                  current_kernel_time
1        4.1e-07  vmlinux                  vmlinux                  datagram_poll
1        4.1e-07  vmlinux                  vmlinux                  dcache_readdir
1        4.1e-07  vmlinux                  vmlinux                  deny_write_access
1        4.1e-07  vmlinux                  vmlinux                  dev_queue_xmit
1        4.1e-07  vmlinux                  vmlinux                  do_sys_ftruncate
1        4.1e-07  vmlinux                  vmlinux                  do_sysctl_strategy
1        4.1e-07  vmlinux                  vmlinux                  dst_alloc
1        4.1e-07  vmlinux                  vmlinux                  elf_map
1        4.1e-07  vmlinux                  vmlinux                  ext3_mkdir
1        4.1e-07  vmlinux                  vmlinux                  ext3_show_options
1        4.1e-07  vmlinux                  vmlinux                  fn_hash_select_default
1        4.1e-07  vmlinux                  vmlinux                  free_fdtable
1        4.1e-07  vmlinux                  vmlinux                  generic_writepages
1        4.1e-07  vmlinux                  vmlinux                  get_index
1        4.1e-07  vmlinux                  vmlinux                  get_vmalloc_info
1        4.1e-07  vmlinux                  vmlinux                  hrtimer_start
1        4.1e-07  vmlinux                  vmlinux                  iput
1        4.1e-07  vmlinux                  vmlinux                  ll_front_merge_fn
1        4.1e-07  vmlinux                  vmlinux                  load_elf_interp
1        4.1e-07  vmlinux                  vmlinux                  move_addr_to_kernel
1        4.1e-07  vmlinux                  vmlinux                  move_ptes
1        4.1e-07  vmlinux                  vmlinux                  mpage_readpage
1        4.1e-07  vmlinux                  vmlinux                  mpage_readpages
1        4.1e-07  vmlinux                  vmlinux                  msync_pte_range
1        4.1e-07  vmlinux                  vmlinux                  normal_poll
1        4.1e-07  vmlinux                  vmlinux                  nr_free_pages
1        4.1e-07  vmlinux                  vmlinux                  pdflush_operation
1        4.1e-07  vmlinux                  vmlinux                  pgd_alloc
1        4.1e-07  vmlinux                  vmlinux                  pipe_read_release
1        4.1e-07  vmlinux                  vmlinux                  prio_tree_expand
1        4.1e-07  vmlinux                  vmlinux                  proc_pident_lookup
1        4.1e-07  vmlinux                  vmlinux                  pty_chars_in_buffer
1        4.1e-07  vmlinux                  vmlinux                  pty_unthrottle
1        4.1e-07  vmlinux                  vmlinux                  rm_from_queue_full
1        4.1e-07  vmlinux                  vmlinux                  sched_migrate_task
1        4.1e-07  vmlinux                  vmlinux                  set_close_on_exec
1        4.1e-07  vmlinux                  vmlinux                  set_page_dirty
1        4.1e-07  vmlinux                  vmlinux                  set_task_comm
1        4.1e-07  vmlinux                  vmlinux                  sock_create
1        4.1e-07  vmlinux                  vmlinux                  sync_dirty_buffer
1        4.1e-07  vmlinux                  vmlinux                  sys_brk
1        4.1e-07  vmlinux                  vmlinux                  sys_faccessat
1        4.1e-07  vmlinux                  vmlinux                  sys_fork
1        4.1e-07  vmlinux                  vmlinux                  sys_ftruncate
1        4.1e-07  vmlinux                  vmlinux                  sys_mkdirat
1        4.1e-07  vmlinux                  vmlinux                  sys_newuname
1        4.1e-07  vmlinux                  vmlinux                  tcp_init_tso_segs
1        4.1e-07  vmlinux                  vmlinux                  tcp_poll
1        4.1e-07  vmlinux                  vmlinux                  tty_ldisc_try
1        4.1e-07  vmlinux                  vmlinux                  tty_write
1        4.1e-07  vmlinux                  vmlinux                  udp_rcv
1        4.1e-07  vmlinux                  vmlinux                  unshare_files
1        4.1e-07  vmlinux                  vmlinux                  vfs_stat_fd
1        4.1e-07  vmlinux                  vmlinux                  vma_prio_tree_add
1        4.1e-07  vmlinux                  vmlinux                  write_chan
1        4.1e-07  vmlinux                  vmlinux                  writeback_acquire
1        4.1e-07  vmlinux                  vmlinux                  writeback_release

[-- Attachment #1.3: oprofile-atomic_long_t.txt --]
[-- Type: text/plain, Size: 134775 bytes --]

CPU: P4 / Xeon with 2 hyper-threads, speed 1397.42 MHz (estimated)
Counted GLOBAL_POWER_EVENTS events (time during which processor is not stopped) with a unit mask of 0x01 (mandatory) count 100000
samples  %        image name               app name                 symbol name
132356470 58.2796  libc-2.3.3.so            libc-2.3.3.so            __sleep
30066775 13.2391  tiotest                  tiotest                  start_proc
27792324 12.2376  vmlinux                  vmlinux                  poll_idle
22607815  9.9547  libc-2.3.3.so            libc-2.3.3.so            __i686.get_pc_thunk.bx
4920224   2.1665  tiotest                  tiotest                  anonymous symbol from section .plt
1217087   0.5359  oprofiled                oprofiled                (no symbols)
1009231   0.4444  vmlinux                  vmlinux                  _spin_lock_irqsave
928364    0.4088  vmlinux                  vmlinux                  _spin_lock
487840    0.2148  vmlinux                  vmlinux                  __blk_queue_bounce
387521    0.1706  vmlinux                  vmlinux                  __copy_to_user_ll
305171    0.1344  vmlinux                  vmlinux                  journal_dirty_metadata
299325    0.1318  vmlinux                  vmlinux                  do_get_write_access
268132    0.1181  vmlinux                  vmlinux                  __copy_from_user_ll
212565    0.0936  oprofile.ko              oprofile                 add_event_entry
207397    0.0913  oprofile.ko              oprofile                 lookup_dcookie
185834    0.0818  oprofile.ko              oprofile                 add_us_sample
175608    0.0773  vmlinux                  vmlinux                  get_offset_cyclone
172217    0.0758  vmlinux                  vmlinux                  find_vma
136905    0.0603  oprofile.ko              oprofile                 increment_tail
123675    0.0545  vmlinux                  vmlinux                  ext3_test_allocatable
118707    0.0523  vmlinux                  vmlinux                  journal_add_journal_head
114924    0.0506  vmlinux                  vmlinux                  __wake_up_common
113496    0.0500  oprofile.ko              oprofile                 sync_buffer
105685    0.0465  vmlinux                  vmlinux                  journal_get_undo_access
84030     0.0370  vmlinux                  vmlinux                  apic_timer_interrupt
74588     0.0328  vmlinux                  vmlinux                  ext3_try_to_allocate
69494     0.0306  oprofile.ko              oprofile                 add_sample_entry
59526     0.0262  vmlinux                  vmlinux                  lookup_bh_lru
59370     0.0261  vmlinux                  vmlinux                  system_call
57322     0.0252  vmlinux                  vmlinux                  journal_put_journal_head
52471     0.0231  oprofile.ko              oprofile                 add_sample
50821     0.0224  vmlinux                  vmlinux                  schedule
48125     0.0212  vmlinux                  vmlinux                  mark_offset_cyclone
42950     0.0189  vmlinux                  vmlinux                  find_busiest_group
42177     0.0186  vmlinux                  vmlinux                  activate_page
41652     0.0183  vmlinux                  vmlinux                  wake_bit_function
38262     0.0168  vmlinux                  vmlinux                  do_gettimeofday
37686     0.0166  libc-2.3.3.so            libc-2.3.3.so            __GI___gettimeofday_internal
37295     0.0164  vmlinux                  vmlinux                  mutex_debug_check_no_locks_freed
34647     0.0153  vmlinux                  vmlinux                  journal_cancel_revoke
33348     0.0147  vmlinux                  vmlinux                  journal_dirty_data
31677     0.0139  vmlinux                  vmlinux                  kmem_cache_free
29328     0.0129  vmlinux                  vmlinux                  journal_stop
26972     0.0119  vmlinux                  vmlinux                  try_to_wake_up
26919     0.0119  vmlinux                  vmlinux                  ext3_new_block
26001     0.0114  vmlinux                  vmlinux                  __brelse
25971     0.0114  vmlinux                  vmlinux                  rebalance_tick
25846     0.0114  vmlinux                  vmlinux                  dependent_sleeper
25425     0.0112  vmlinux                  vmlinux                  ext3_get_inode_block
24442     0.0108  vmlinux                  vmlinux                  __wait_on_bit_lock
22483     0.0099  vmlinux                  vmlinux                  __switch_to
21980     0.0097  vmlinux                  vmlinux                  timer_interrupt
21852     0.0096  vmlinux                  vmlinux                  mark_page_accessed
21350     0.0094  vmlinux                  vmlinux                  ext3_journal_start_sb
21298     0.0094  vmlinux                  vmlinux                  start_this_handle
21058     0.0093  vmlinux                  vmlinux                  __wake_up_bit
19070     0.0084  vmlinux                  vmlinux                  kmem_cache_alloc
18418     0.0081  vmlinux                  vmlinux                  kmap_atomic
18025     0.0079  vmlinux                  vmlinux                  _read_lock_irqsave
17979     0.0079  vmlinux                  vmlinux                  kunmap
17393     0.0077  vmlinux                  vmlinux                  do_flush_tlb_all
17346     0.0076  vmlinux                  vmlinux                  sched_clock
16793     0.0074  vmlinux                  vmlinux                  generic_file_buffered_write
16654     0.0073  vmlinux                  vmlinux                  smp_invalidate_interrupt
16156     0.0071  vmlinux                  vmlinux                  __journal_file_buffer
16122     0.0071  vmlinux                  vmlinux                  ext3_do_update_inode
15164     0.0067  vmlinux                  vmlinux                  unlock_buffer
13724     0.0060  vmlinux                  vmlinux                  find_get_page
13475     0.0059  vmlinux                  vmlinux                  sys_gettimeofday
13410     0.0059  libpthread-0.10.so       libpthread-0.10.so       __libc_write
13269     0.0058  oprofile.ko              oprofile                 add_cookie_switch
13176     0.0058  vmlinux                  vmlinux                  find_next_bit
13102     0.0058  vmlinux                  vmlinux                  prepare_to_wait_exclusive
12761     0.0056  vmlinux                  vmlinux                  __mark_inode_dirty
12688     0.0056  vmlinux                  vmlinux                  scheduler_tick
12457     0.0055  vmlinux                  vmlinux                  __block_prepare_write
12205     0.0054  vmlinux                  vmlinux                  _write_lock_irqsave
12045     0.0053  vmlinux                  vmlinux                  run_timer_softirq
11944     0.0053  vmlinux                  vmlinux                  __do_softirq
11771     0.0052  libpthread-0.10.so       libpthread-0.10.so       __libc_read
11686     0.0051  vmlinux                  vmlinux                  ext3_get_block_handle
11656     0.0051  vmlinux                  vmlinux                  release_pages
11440     0.0050  vmlinux                  vmlinux                  page_address
11292     0.0050  vmlinux                  vmlinux                  end_buffer_async_write
11272     0.0050  vmlinux                  vmlinux                  buffered_rmqueue
11111     0.0049  vmlinux                  vmlinux                  mod_page_state_offset
11052     0.0049  vmlinux                  vmlinux                  bit_waitqueue
10803     0.0048  vmlinux                  vmlinux                  radix_tree_lookup
10574     0.0047  vmlinux                  vmlinux                  put_page
10355     0.0046  vmlinux                  vmlinux                  debug_mutex_free_waiter
10300     0.0045  vmlinux                  vmlinux                  ext3_mark_iloc_dirty
9970      0.0044  vmlinux                  vmlinux                  blk_rq_map_sg
9936      0.0044  vmlinux                  vmlinux                  __mutex_unlock_slowpath
9864      0.0043  vmlinux                  vmlinux                  sync_buffer
9604      0.0042  vmlinux                  vmlinux                  __mutex_lock_slowpath
9323      0.0041  vmlinux                  vmlinux                  smp_apic_timer_interrupt
9288      0.0041  vmlinux                  vmlinux                  __generic_file_aio_write_nolock
9193      0.0040  vmlinux                  vmlinux                  __make_request
9082      0.0040  vmlinux                  vmlinux                  ext3_ordered_commit_write
9081      0.0040  vmlinux                  vmlinux                  fget_light
8992      0.0040  vmlinux                  vmlinux                  restore_nocheck
8991      0.0040  vmlinux                  vmlinux                  _spin_unlock_irqrestore
8919      0.0039  vmlinux                  vmlinux                  __journal_temp_unlink_buffer
8899      0.0039  tiotest                  tiotest                  do_write_test
8732      0.0038  vmlinux                  vmlinux                  debug_mutex_set_owner
8708      0.0038  vmlinux                  vmlinux                  walk_page_buffers
8704      0.0038  vmlinux                  vmlinux                  mempool_free
8700      0.0038  vmlinux                  vmlinux                  fput
8588      0.0038  vmlinux                  vmlinux                  _read_unlock_irq
8556      0.0038  vmlinux                  vmlinux                  current_fs_time
8492      0.0037  vmlinux                  vmlinux                  run_posix_cpu_timers
8327      0.0037  vmlinux                  vmlinux                  update_process_times
7917      0.0035  vmlinux                  vmlinux                  hrtimer_run_queues
7795      0.0034  vmlinux                  vmlinux                  ext3_prepare_write
7775      0.0034  vmlinux                  vmlinux                  __log_space_left
7758      0.0034  vmlinux                  vmlinux                  _spin_unlock
7746      0.0034  vmlinux                  vmlinux                  __find_get_block_slow
7716      0.0034  vmlinux                  vmlinux                  resched_task
7711      0.0034  vmlinux                  vmlinux                  free_block
7599      0.0033  vmlinux                  vmlinux                  radix_tree_tag_set
7580      0.0033  vmlinux                  vmlinux                  __rcu_pending
7376      0.0032  vmlinux                  vmlinux                  set_page_address
7272      0.0032  vmlinux                  vmlinux                  _spin_unlock_irq
7139      0.0031  vmlinux                  vmlinux                  journal_start
7136      0.0031  vmlinux                  vmlinux                  raise_softirq
7026      0.0031  vmlinux                  vmlinux                  account_user_time
7014      0.0031  vmlinux                  vmlinux                  blk_recount_segments
6963      0.0031  vmlinux                  vmlinux                  __ext3_get_inode_loc
6736      0.0030  vmlinux                  vmlinux                  rb_next
6705      0.0030  vmlinux                  vmlinux                  _write_unlock_irqrestore
6672      0.0029  vmlinux                  vmlinux                  submit_bh
6566      0.0029  vmlinux                  vmlinux                  as_find_arq_hash
6545      0.0029  vmlinux                  vmlinux                  __rmqueue
6480      0.0029  vmlinux                  vmlinux                  do_generic_mapping_read
6433      0.0028  vmlinux                  vmlinux                  __lock_buffer
6383      0.0028  vmlinux                  vmlinux                  ktime_get_ts
6335      0.0028  vmlinux                  vmlinux                  ll_back_merge_fn
6305      0.0028  vmlinux                  vmlinux                  __ext3_journal_stop
6255      0.0028  vmlinux                  vmlinux                  init_buffer_head
6205      0.0027  vmlinux                  vmlinux                  enqueue_task
6099      0.0027  vmlinux                  vmlinux                  elv_merge
6091      0.0027  vmlinux                  vmlinux                  device_not_available
6040      0.0027  vmlinux                  vmlinux                  generic_file_aio_write
6036      0.0027  vmlinux                  vmlinux                  rcu_pending
6021      0.0027  vmlinux                  vmlinux                  do_softirq
5950      0.0026  vmlinux                  vmlinux                  bounce_end_io
5903      0.0026  vmlinux                  vmlinux                  kmap_high
5872      0.0026  vmlinux                  vmlinux                  idle_cpu
5811      0.0026  vmlinux                  vmlinux                  bio_put
5706      0.0025  vmlinux                  vmlinux                  profile_tick
5700      0.0025  vmlinux                  vmlinux                  free_pages_bulk
5662      0.0025  vmlinux                  vmlinux                  wake_idle
5658      0.0025  vmlinux                  vmlinux                  irq_exit
5413      0.0024  vmlinux                  vmlinux                  journal_commit_transaction
5366      0.0024  vmlinux                  vmlinux                  _write_unlock_irq
5228      0.0023  vmlinux                  vmlinux                  irq_entries_start
5201      0.0023  vmlinux                  vmlinux                  ext3_reserve_inode_write
5146      0.0023  vmlinux                  vmlinux                  generic_make_request
5145      0.0023  vmlinux                  vmlinux                  kunmap_high
5103      0.0022  vmlinux                  vmlinux                  block_prepare_write
5064      0.0022  vmlinux                  vmlinux                  do_sync_write
5061      0.0022  vmlinux                  vmlinux                  ext3_block_to_path
5001      0.0022  vmlinux                  vmlinux                  ktime_get_real
4995      0.0022  vmlinux                  vmlinux                  __find_get_block
4973      0.0022  vmlinux                  vmlinux                  reap_alien
4925      0.0022  vmlinux                  vmlinux                  __block_commit_write
4898      0.0022  vmlinux                  vmlinux                  load_balance
4742      0.0021  vmlinux                  vmlinux                  ext3_try_to_allocate_with_rsv
4703      0.0021  vmlinux                  vmlinux                  io_schedule
4690      0.0021  vmlinux                  vmlinux                  vfs_write
4566      0.0020  vmlinux                  vmlinux                  invalidate_interrupt
4549      0.0020  vmlinux                  vmlinux                  zap_pte_range
4530      0.0020  tiotest                  tiotest                  do_read_test
4407      0.0019  vmlinux                  vmlinux                  mempool_alloc
4195      0.0018  vmlinux                  vmlinux                  __block_write_full_page
4093      0.0018  vmlinux                  vmlinux                  drive_stat_acct
4048      0.0018  vmlinux                  vmlinux                  math_state_restore
4044      0.0018  vmlinux                  vmlinux                  flush_tlb_others
4026      0.0018  vmlinux                  vmlinux                  unlock_page
4021      0.0018  vmlinux                  vmlinux                  dnotify_parent
4020      0.0018  vmlinux                  vmlinux                  free_hot_cold_page
4017      0.0018  vmlinux                  vmlinux                  add_to_page_cache
3945      0.0017  vmlinux                  vmlinux                  ahc_linux_run_command
3900      0.0017  vmlinux                  vmlinux                  slab_put_obj
3876      0.0017  vmlinux                  vmlinux                  kfree
3860      0.0017  vmlinux                  vmlinux                  journal_invalidatepage
3803      0.0017  vmlinux                  vmlinux                  touch_atime
3760      0.0017  vmlinux                  vmlinux                  create_empty_buffers
3734      0.0016  vmlinux                  vmlinux                  ext3_splice_branch
3717      0.0016  vmlinux                  vmlinux                  __set_page_dirty_nobuffers
3716      0.0016  vmlinux                  vmlinux                  get_page_from_freelist
3676      0.0016  vmlinux                  vmlinux                  journal_unmap_buffer
3555      0.0016  vmlinux                  vmlinux                  ext3_ordered_writepage
3523      0.0016  vmlinux                  vmlinux                  bio_alloc_bioset
3506      0.0015  vmlinux                  vmlinux                  cache_alloc_refill
3483      0.0015  vmlinux                  vmlinux                  wake_up_bit
3424      0.0015  vmlinux                  vmlinux                  ktime_get
3359      0.0015  vmlinux                  vmlinux                  wake_sleeping_dependent
3310      0.0015  vmlinux                  vmlinux                  getnstimeofday
3234      0.0014  vmlinux                  vmlinux                  bh_lru_install
3218      0.0014  vmlinux                  vmlinux                  set_normalized_timespec
3203      0.0014  vmlinux                  vmlinux                  ext3_get_group_desc
3186      0.0014  vmlinux                  vmlinux                  mpage_writepages
3149      0.0014  vmlinux                  vmlinux                  file_update_time
3145      0.0014  vmlinux                  vmlinux                  ext3_dirty_inode
3145      0.0014  vmlinux                  vmlinux                  file_read_actor
3140      0.0014  vmlinux                  vmlinux                  activate_task
3131      0.0014  vmlinux                  vmlinux                  ext3_file_write
3120      0.0014  vmlinux                  vmlinux                  get_task_mm
3037      0.0013  vmlinux                  vmlinux                  __bread
3025      0.0013  vmlinux                  vmlinux                  ahc_linux_isr
3021      0.0013  vmlinux                  vmlinux                  ext3_writepage_trans_blocks
3018      0.0013  vmlinux                  vmlinux                  kunmap_atomic
2993      0.0013  vmlinux                  vmlinux                  __pagevec_lru_add
2981      0.0013  vmlinux                  vmlinux                  __end_that_request_first
2951      0.0013  vmlinux                  vmlinux                  __journal_remove_journal_head
2950      0.0013  vmlinux                  vmlinux                  radix_tree_tag_clear
2925      0.0013  vmlinux                  vmlinux                  ret_from_intr
2891      0.0013  vmlinux                  vmlinux                  copy_to_user
2885      0.0013  vmlinux                  vmlinux                  as_add_arq_hash
2828      0.0012  vmlinux                  vmlinux                  recalc_task_prio
2809      0.0012  vmlinux                  vmlinux                  alloc_pages_current
2809      0.0012  vmlinux                  vmlinux                  drain_array_locked
2808      0.0012  vmlinux                  vmlinux                  journal_remove_journal_head
2739      0.0012  vmlinux                  vmlinux                  radix_tree_insert
2667      0.0012  vmlinux                  vmlinux                  __getblk
2613      0.0012  vmlinux                  vmlinux                  account_system_time
2573      0.0011  libpthread-0.10.so       libpthread-0.10.so       __pthread_disable_asynccancel
2540      0.0011  vmlinux                  vmlinux                  ext3_alloc_branch
2522      0.0011  vmlinux                  vmlinux                  alloc_page_buffers
2520      0.0011  vmlinux                  vmlinux                  dequeue_task
2496      0.0011  oprofile.ko              oprofile                 mark_done
2474      0.0011  vmlinux                  vmlinux                  find_get_pages_tag
2464      0.0011  vmlinux                  vmlinux                  ext3_get_branch
2460      0.0011  vmlinux                  vmlinux                  __alloc_pages
2451      0.0011  vmlinux                  vmlinux                  debug_mutex_unlock
2450      0.0011  vmlinux                  vmlinux                  out_of_line_wait_on_bit_lock
2448      0.0011  vmlinux                  vmlinux                  vfs_read
2396      0.0011  vmlinux                  vmlinux                  smp_call_function
2353      0.0010  vmlinux                  vmlinux                  page_waitqueue
2338      0.0010  vmlinux                  vmlinux                  journal_get_write_access
2330      0.0010  vmlinux                  vmlinux                  journal_blocks_per_page
2297      0.0010  vmlinux                  vmlinux                  elv_queue_empty
2268     1.0e-03  vmlinux                  vmlinux                  blk_remove_plug
2238     9.9e-04  vmlinux                  vmlinux                  __free_pages
2198     9.7e-04  vmlinux                  vmlinux                  ext3_mark_inode_dirty
2198     9.7e-04  vmlinux                  vmlinux                  mempool_alloc_slab
2195     9.7e-04  vmlinux                  vmlinux                  ext3_free_blocks_sb
2185     9.6e-04  vmlinux                  vmlinux                  alloc_buffer_head
2154     9.5e-04  oprofile.ko              oprofile                 release_mm
2153     9.5e-04  vmlinux                  vmlinux                  cpu_idle
2146     9.4e-04  vmlinux                  vmlinux                  inotify_dentry_parent_queue_event
2081     9.2e-04  vmlinux                  vmlinux                  slab_get_obj
2079     9.2e-04  vmlinux                  vmlinux                  __generic_file_aio_read
2072     9.1e-04  vmlinux                  vmlinux                  send_IPI_mask_sequence
2057     9.1e-04  vmlinux                  vmlinux                  preempt_schedule
2054     9.0e-04  oprofile.ko              oprofile                 add_user_ctx_switch
1971     8.7e-04  vmlinux                  vmlinux                  __mod_page_state_offset
1955     8.6e-04  vmlinux                  vmlinux                  mark_buffer_dirty
1925     8.5e-04  vmlinux                  vmlinux                  test_clear_page_writeback
1919     8.4e-04  vmlinux                  vmlinux                  test_set_page_writeback
1899     8.4e-04  vmlinux                  vmlinux                  generic_commit_write
1898     8.4e-04  vmlinux                  vmlinux                  truncate_inode_pages_range
1898     8.4e-04  vmlinux                  vmlinux                  zone_watermark_ok
1898     8.4e-04  vmlinux                  vmlinux                  zonelist_policy
1882     8.3e-04  vmlinux                  vmlinux                  restore_all
1876     8.3e-04  vmlinux                  vmlinux                  rw_verify_area
1868     8.2e-04  oprofile.ko              oprofile                 get_slots
1852     8.2e-04  vmlinux                  vmlinux                  blk_backing_dev_unplug
1842     8.1e-04  vmlinux                  vmlinux                  set_bh_page
1818     8.0e-04  vmlinux                  vmlinux                  __wake_up
1807     8.0e-04  vmlinux                  vmlinux                  test_clear_page_dirty
1804     7.9e-04  vmlinux                  vmlinux                  reschedule_interrupt
1797     7.9e-04  vmlinux                  vmlinux                  do_sync_read
1792     7.9e-04  vmlinux                  vmlinux                  clear_page_dirty_for_io
1779     7.8e-04  vmlinux                  vmlinux                  finish_wait
1767     7.8e-04  vmlinux                  vmlinux                  cache_reap
1747     7.7e-04  vmlinux                  vmlinux                  profile_hit
1744     7.7e-04  vmlinux                  vmlinux                  sys_write
1743     7.7e-04  vmlinux                  vmlinux                  __lookup_tag
1718     7.6e-04  oprofile.ko              oprofile                 take_tasks_mm
1700     7.5e-04  vmlinux                  vmlinux                  smp_call_function_interrupt
1696     7.5e-04  vmlinux                  vmlinux                  generic_file_llseek
1696     7.5e-04  vmlinux                  vmlinux                  remove_from_page_cache
1683     7.4e-04  vmlinux                  vmlinux                  end_page_writeback
1677     7.4e-04  libpthread-0.10.so       libpthread-0.10.so       __pthread_enable_asynccancel
1663     7.3e-04  vmlinux                  vmlinux                  new_handle
1654     7.3e-04  vmlinux                  vmlinux                  read_block_bitmap
1589     7.0e-04  vmlinux                  vmlinux                  find_lock_page
1571     6.9e-04  vmlinux                  vmlinux                  tasklet_action
1570     6.9e-04  libpthread-0.10.so       libpthread-0.10.so       __libc_lseek
1528     6.7e-04  vmlinux                  vmlinux                  sys_read
1508     6.6e-04  vmlinux                  vmlinux                  cond_resched
1498     6.6e-04  vmlinux                  vmlinux                  inverted_lock
1488     6.6e-04  vmlinux                  vmlinux                  ext3_get_inode_loc
1476     6.5e-04  vmlinux                  vmlinux                  __d_lookup
1476     6.5e-04  vmlinux                  vmlinux                  effective_prio
1463     6.4e-04  vmlinux                  vmlinux                  check_deadlock
1451     6.4e-04  vmlinux                  vmlinux                  bio_init
1449     6.4e-04  vmlinux                  vmlinux                  deactivate_task
1439     6.3e-04  vmlinux                  vmlinux                  attempt_merge
1438     6.3e-04  vmlinux                  vmlinux                  mmput
1413     6.2e-04  oprofile.ko              oprofile                 get_exec_dcookie
1410     6.2e-04  vmlinux                  vmlinux                  ext3_find_goal
1380     6.1e-04  vmlinux                  vmlinux                  block_write_full_page
1360     6.0e-04  vmlinux                  vmlinux                  debug_mutex_init_waiter
1343     5.9e-04  vmlinux                  vmlinux                  syscall_exit
1337     5.9e-04  vmlinux                  vmlinux                  zone_statistics
1298     5.7e-04  vmlinux                  vmlinux                  elv_rq_merge_ok
1254     5.5e-04  tiotest                  tiotest                  do_random_write_test
1250     5.5e-04  libpthread-0.10.so       libpthread-0.10.so       __i686.get_pc_thunk.cx
1220     5.4e-04  vmlinux                  vmlinux                  radix_tree_preload
1216     5.4e-04  vmlinux                  vmlinux                  bget_one
1212     5.3e-04  vmlinux                  vmlinux                  debug_mutex_add_waiter
1207     5.3e-04  vmlinux                  vmlinux                  idle_balance
1192     5.2e-04  vmlinux                  vmlinux                  ext3_has_free_blocks
1185     5.2e-04  vmlinux                  vmlinux                  radix_tree_delete
1179     5.2e-04  vmlinux                  vmlinux                  syscall_call
1170     5.2e-04  vmlinux                  vmlinux                  flush_tlb_mm
1166     5.1e-04  vmlinux                  vmlinux                  memcpy
1158     5.1e-04  vmlinux                  vmlinux                  do_wait
1157     5.1e-04  vmlinux                  vmlinux                  as_merged_request
1152     5.1e-04  vmlinux                  vmlinux                  as_merge
1142     5.0e-04  vmlinux                  vmlinux                  page_pool_alloc
1134     5.0e-04  vmlinux                  vmlinux                  __do_IRQ
1114     4.9e-04  vmlinux                  vmlinux                  task_timeslice
1113     4.9e-04  vmlinux                  vmlinux                  prep_new_page
1099     4.8e-04  vmlinux                  vmlinux                  balance_dirty_pages_ratelimited
1091     4.8e-04  vmlinux                  vmlinux                  get_request
1073     4.7e-04  vmlinux                  vmlinux                  page_slot
1050     4.6e-04  vmlinux                  vmlinux                  bput_one
1033     4.5e-04  libperl.so               libperl.so               (no symbols)
1033     4.5e-04  libpthread-0.10.so       libpthread-0.10.so       pthread_reap_children
1017     4.5e-04  vmlinux                  vmlinux                  find_next_zero_bit
1001     4.4e-04  vmlinux                  vmlinux                  ext3_get_block
1000     4.4e-04  vmlinux                  vmlinux                  ext3_journal_dirty_data
1000     4.4e-04  vmlinux                  vmlinux                  truncate_complete_page
998      4.4e-04  vmlinux                  vmlinux                  resume_userspace
984      4.3e-04  vmlinux                  vmlinux                  _atomic_dec_and_lock
984      4.3e-04  vmlinux                  vmlinux                  _read_lock
981      4.3e-04  vmlinux                  vmlinux                  __down
965      4.2e-04  tiotest                  tiotest                  do_random_read_test
964      4.2e-04  vmlinux                  vmlinux                  recalc_bh_state
963      4.2e-04  vmlinux                  vmlinux                  cache_init_objs
961      4.2e-04  vmlinux                  vmlinux                  __bitmap_weight
954      4.2e-04  vmlinux                  vmlinux                  ext3_find_entry
953      4.2e-04  vmlinux                  vmlinux                  generic_file_aio_read
953      4.2e-04  vmlinux                  vmlinux                  mark_buffer_async_write
946      4.2e-04  vmlinux                  vmlinux                  ext3_free_data
924      4.1e-04  vmlinux                  vmlinux                  rt_check_expire
918      4.0e-04  vmlinux                  vmlinux                  find_get_pages
910      4.0e-04  vmlinux                  vmlinux                  as_find_arq_rb
890      3.9e-04  vmlinux                  vmlinux                  inode_add_bytes
890      3.9e-04  vmlinux                  vmlinux                  page_cache_readahead
881      3.9e-04  vmlinux                  vmlinux                  page_fault
879      3.9e-04  vmlinux                  vmlinux                  _spin_lock_irq
877      3.9e-04  vmlinux                  vmlinux                  i8042_interrupt
877      3.9e-04  vmlinux                  vmlinux                  run_workqueue
871      3.8e-04  vmlinux                  vmlinux                  lock_timer_base
865      3.8e-04  vmlinux                  vmlinux                  ext3_invalidatepage
857      3.8e-04  vmlinux                  vmlinux                  generic_unplug_device
855      3.8e-04  vmlinux                  vmlinux                  search_reserve_window
850      3.7e-04  oprofile.ko              oprofile                 add_kernel_ctx_switch
803      3.5e-04  vmlinux                  vmlinux                  __link_path_walk
803      3.5e-04  vmlinux                  vmlinux                  __rb_erase_color
800      3.5e-04  vmlinux                  vmlinux                  cpu_quiet
795      3.5e-04  libpthread-0.10.so       libpthread-0.10.so       __pthread_manager
788      3.5e-04  vmlinux                  vmlinux                  submit_bio
784      3.5e-04  vmlinux                  vmlinux                  internal_add_timer
779      3.4e-04  vmlinux                  vmlinux                  blk_queue_bounce
778      3.4e-04  vmlinux                  vmlinux                  inotify_inode_queue_event
769      3.4e-04  vmlinux                  vmlinux                  journal_alloc_journal_head
768      3.4e-04  vmlinux                  vmlinux                  goal_in_my_reservation
762      3.4e-04  vmlinux                  vmlinux                  delay_cyclone
754      3.3e-04  vmlinux                  vmlinux                  bio_free
747      3.3e-04  vmlinux                  vmlinux                  rb_erase
746      3.3e-04  vmlinux                  vmlinux                  autoremove_wake_function
743      3.3e-04  vmlinux                  vmlinux                  do_page_fault
736      3.2e-04  vmlinux                  vmlinux                  delayed_work_timer_fn
736      3.2e-04  vmlinux                  vmlinux                  handle_IRQ_event
735      3.2e-04  vmlinux                  vmlinux                  find_next_reservable_window
731      3.2e-04  vmlinux                  vmlinux                  load_balance_newidle
730      3.2e-04  vmlinux                  vmlinux                  __rcu_process_callbacks
721      3.2e-04  vmlinux                  vmlinux                  rcu_check_quiescent_state
711      3.1e-04  vmlinux                  vmlinux                  e1000_update_stats
705      3.1e-04  vmlinux                  vmlinux                  bio_endio
704      3.1e-04  vmlinux                  vmlinux                  remove_suid
703      3.1e-04  vmlinux                  vmlinux                  as_choose_req
695      3.1e-04  vmlinux                  vmlinux                  mutex_remove_waiter
693      3.1e-04  vmlinux                  vmlinux                  eligible_child
666      2.9e-04  vmlinux                  vmlinux                  rmqueue_bulk
664      2.9e-04  vmlinux                  vmlinux                  worker_thread
663      2.9e-04  vmlinux                  vmlinux                  scsi_request_fn
659      2.9e-04  vmlinux                  vmlinux                  unmap_underlying_metadata
647      2.8e-04  vmlinux                  vmlinux                  find_idlest_group
643      2.8e-04  vmlinux                  vmlinux                  timespec_trunc
642      2.8e-04  vmlinux                  vmlinux                  flush_all_zero_pkmaps
637      2.8e-04  vmlinux                  vmlinux                  rb_insert_color
630      2.8e-04  vmlinux                  vmlinux                  common_interrupt
630      2.8e-04  vmlinux                  vmlinux                  mutex_debug_check_no_locks_held
627      2.8e-04  vmlinux                  vmlinux                  rcu_process_callbacks
621      2.7e-04  vmlinux                  vmlinux                  call_function_interrupt
615      2.7e-04  vmlinux                  vmlinux                  as_move_to_dispatch
610      2.7e-04  vmlinux                  vmlinux                  add_dirent_to_buf
610      2.7e-04  vmlinux                  vmlinux                  move_tasks
606      2.7e-04  vmlinux                  vmlinux                  elv_latter_request
593      2.6e-04  vmlinux                  vmlinux                  kmap
592      2.6e-04  vmlinux                  vmlinux                  rcu_check_callbacks
589      2.6e-04  vmlinux                  vmlinux                  copy_process
582      2.6e-04  vmlinux                  vmlinux                  __mod_timer
563      2.5e-04  oprofile.ko              oprofile                 wq_sync_buffer
561      2.5e-04  vmlinux                  vmlinux                  _write_lock_irq
555      2.4e-04  vmlinux                  vmlinux                  find_pid
547      2.4e-04  vmlinux                  vmlinux                  __clear_page_buffers
545      2.4e-04  vmlinux                  vmlinux                  do_wp_page
542      2.4e-04  vmlinux                  vmlinux                  elv_merged_request
541      2.4e-04  libc-2.3.3.so            libc-2.3.3.so            __clone
541      2.4e-04  vmlinux                  vmlinux                  bounce_end_io_write
535      2.4e-04  vmlinux                  vmlinux                  ext3_rsv_window_add
530      2.3e-04  vmlinux                  vmlinux                  bitmap_search_next_usable_block
528      2.3e-04  vmlinux                  vmlinux                  as_dispatch_request
524      2.3e-04  vmlinux                  vmlinux                  mempool_free_slab
517      2.3e-04  vmlinux                  vmlinux                  queue_delayed_work
508      2.2e-04  vmlinux                  vmlinux                  run_local_timers
500      2.2e-04  vmlinux                  vmlinux                  ack_edge_ioapic_irq
499      2.2e-04  vmlinux                  vmlinux                  ahc_done
496      2.2e-04  vmlinux                  vmlinux                  init_request_from_bio
492      2.2e-04  vmlinux                  vmlinux                  do_mmap_pgoff
490      2.2e-04  vmlinux                  vmlinux                  __generic_unplug_device
489      2.2e-04  libc-2.3.3.so            libc-2.3.3.so            _int_malloc
487      2.1e-04  vmlinux                  vmlinux                  __remove_from_page_cache
476      2.1e-04  vmlinux                  vmlinux                  ahc_run_qoutfifo
476      2.1e-04  vmlinux                  vmlinux                  ll_rw_block
474      2.1e-04  vmlinux                  vmlinux                  end_bio_bh_io_sync
465      2.0e-04  vmlinux                  vmlinux                  __handle_mm_fault
465      2.0e-04  vmlinux                  vmlinux                  __lookup
463      2.0e-04  vmlinux                  vmlinux                  as_update_iohist
456      2.0e-04  vmlinux                  vmlinux                  sd_init_command
455      2.0e-04  vmlinux                  vmlinux                  do_path_lookup
448      2.0e-04  vmlinux                  vmlinux                  as_remove_queued_request
447      2.0e-04  vmlinux                  vmlinux                  drop_buffers
440      1.9e-04  vmlinux                  vmlinux                  as_latter_request
434      1.9e-04  libpthread-0.10.so       libpthread-0.10.so       pthread_start_thread
433      1.9e-04  vmlinux                  vmlinux                  try_to_free_buffers
432      1.9e-04  vmlinux                  vmlinux                  get_unused_fd
431      1.9e-04  vmlinux                  vmlinux                  do_timer
430      1.9e-04  vmlinux                  vmlinux                  get_empty_filp
421      1.9e-04  vmlinux                  vmlinux                  bio_alloc
418      1.8e-04  vmlinux                  vmlinux                  note_interrupt
418      1.8e-04  vmlinux                  vmlinux                  sys_lseek
416      1.8e-04  vmlinux                  vmlinux                  __as_add_arq_rb
412      1.8e-04  vmlinux                  vmlinux                  del_timer
404      1.8e-04  vmlinux                  vmlinux                  kref_put
399      1.8e-04  vmlinux                  vmlinux                  put_io_context
397      1.7e-04  vmlinux                  vmlinux                  k_getrusage
396      1.7e-04  libpthread-0.10.so       libpthread-0.10.so       __i686.get_pc_thunk.bx
393      1.7e-04  vmlinux                  vmlinux                  cache_grow
392      1.7e-04  vmlinux                  vmlinux                  kref_get
391      1.7e-04  vmlinux                  vmlinux                  strncpy_from_user
389      1.7e-04  vmlinux                  vmlinux                  pull_task
387      1.7e-04  vmlinux                  vmlinux                  recalc_sigpending_tsk
385      1.7e-04  vmlinux                  vmlinux                  __tasklet_schedule
378      1.7e-04  vmlinux                  vmlinux                  scsi_get_command
377      1.7e-04  libc-2.3.3.so            libc-2.3.3.so            __sigprocmask
371      1.6e-04  vmlinux                  vmlinux                  scsi_prep_fn
370      1.6e-04  vmlinux                  vmlinux                  kmem_getpages
369      1.6e-04  libpthread-0.10.so       libpthread-0.10.so       __pthread_create_2_1
366      1.6e-04  vmlinux                  vmlinux                  ext3_forget
363      1.6e-04  vmlinux                  vmlinux                  journal_dirty_data_fn
359      1.6e-04  libpthread-0.10.so       libpthread-0.10.so       __pthread_lock
352      1.5e-04  vmlinux                  vmlinux                  percpu_counter_mod
351      1.5e-04  vmlinux                  vmlinux                  blk_queue_end_tag
350      1.5e-04  vmlinux                  vmlinux                  as_add_request
350      1.5e-04  vmlinux                  vmlinux                  blk_queue_start_tag
350      1.5e-04  vmlinux                  vmlinux                  elv_next_request
350      1.5e-04  vmlinux                  vmlinux                  free_buffer_head
343      1.5e-04  vmlinux                  vmlinux                  bio_fs_destructor
341      1.5e-04  vmlinux                  vmlinux                  open_namei
340      1.5e-04  vmlinux                  vmlinux                  do_exit
337      1.5e-04  vmlinux                  vmlinux                  do_anonymous_page
329      1.4e-04  libc-2.3.3.so            libc-2.3.3.so            __GI___getrusage
328      1.4e-04  oprofile.ko              oprofile                 cpu_buffer_reset
327      1.4e-04  vmlinux                  vmlinux                  __sigqueue_alloc
326      1.4e-04  vmlinux                  vmlinux                  bio_hw_segments
325      1.4e-04  vmlinux                  vmlinux                  __do_page_cache_readahead
324      1.4e-04  vmlinux                  vmlinux                  as_queue_empty
322      1.4e-04  vmlinux                  vmlinux                  wait_task_zombie
321      1.4e-04  vmlinux                  vmlinux                  ext3_clear_blocks
313      1.4e-04  bash                     bash                     (no symbols)
308      1.4e-04  vmlinux                  vmlinux                  as_update_seekdist
308      1.4e-04  vmlinux                  vmlinux                  prepare_to_wait
307      1.4e-04  vmlinux                  vmlinux                  do_irq_balance
305      1.3e-04  vmlinux                  vmlinux                  raise_softirq_irqoff
298      1.3e-04  ld-2.3.3.so              ld-2.3.3.so              memset
296      1.3e-04  vmlinux                  vmlinux                  as_can_break_anticipation
291      1.3e-04  vmlinux                  vmlinux                  end_that_request_last
290      1.3e-04  vmlinux                  vmlinux                  exec_permission_lite
287      1.3e-04  vmlinux                  vmlinux                  bio_phys_segments
287      1.3e-04  vmlinux                  vmlinux                  copy_io_context
286      1.3e-04  vmlinux                  vmlinux                  _read_unlock
286      1.3e-04  vmlinux                  vmlinux                  kmem_freepages
286      1.3e-04  vmlinux                  vmlinux                  may_open
283      1.2e-04  vmlinux                  vmlinux                  as_completed_request
279      1.2e-04  libc-2.3.3.so            libc-2.3.3.so            _int_free
278      1.2e-04  libpthread-0.10.so       libpthread-0.10.so       __pthread_do_exit
277      1.2e-04  vmlinux                  vmlinux                  scsi_dispatch_cmd
273      1.2e-04  vmlinux                  vmlinux                  dput
271      1.2e-04  vmlinux                  vmlinux                  alloc_new_reservation
271      1.2e-04  vmlinux                  vmlinux                  mutex_unlock
270      1.2e-04  vmlinux                  vmlinux                  scsi_end_request
267      1.2e-04  libpthread-0.10.so       libpthread-0.10.so       __pthread_destroy_specifics
258      1.1e-04  libc-2.3.3.so            libc-2.3.3.so            __mmap
254      1.1e-04  vmlinux                  vmlinux                  scsi_init_io
253      1.1e-04  vmlinux                  vmlinux                  as_put_io_context
253      1.1e-04  vmlinux                  vmlinux                  page_pool_free
253      1.1e-04  vmlinux                  vmlinux                  scsi_init_cmd_errh
250      1.1e-04  vmlinux                  vmlinux                  __follow_mount
250      1.1e-04  vmlinux                  vmlinux                  default_wake_function
250      1.1e-04  vmlinux                  vmlinux                  ext3_check_dir_entry
242      1.1e-04  vmlinux                  vmlinux                  work_resched
240      1.1e-04  vmlinux                  vmlinux                  blk_complete_request
239      1.1e-04  vmlinux                  vmlinux                  find_vma_prev
238      1.0e-04  vmlinux                  vmlinux                  _read_lock_irq
238      1.0e-04  vmlinux                  vmlinux                  as_may_queue
238      1.0e-04  vmlinux                  vmlinux                  init_fpu
235      1.0e-04  libpthread-0.10.so       libpthread-0.10.so       __pthread_alt_unlock
232      1.0e-04  vmlinux                  vmlinux                  end_level_ioapic_irq
228      1.0e-04  vmlinux                  vmlinux                  get_io_context
227      1.0e-04  vmlinux                  vmlinux                  elv_insert
226      1.0e-04  vmlinux                  vmlinux                  __rb_rotate_left
226      1.0e-04  vmlinux                  vmlinux                  generic_permission
226      1.0e-04  vmlinux                  vmlinux                  sys_close
225      9.9e-05  vmlinux                  vmlinux                  __pagevec_lru_add_active
224      9.9e-05  vmlinux                  vmlinux                  remove_wait_queue
221      9.7e-05  vmlinux                  vmlinux                  ahc_linux_queue
221      9.7e-05  vmlinux                  vmlinux                  scsi_device_unbusy
220      9.7e-05  libc-2.3.3.so            libc-2.3.3.so            __GI___poll
219      9.6e-05  oprofile.ko              oprofile                 add_cpu_switch
218      9.6e-05  vmlinux                  vmlinux                  as_set_request
215      9.5e-05  ld-2.3.3.so              ld-2.3.3.so              do_lookup_x
215      9.5e-05  vmlinux                  vmlinux                  do_invalidatepage
211      9.3e-05  vmlinux                  vmlinux                  cache_flusharray
210      9.2e-05  libpthread-0.10.so       libpthread-0.10.so       __libc_open
207      9.1e-05  vmlinux                  vmlinux                  elv_completed_request
206      9.1e-05  vmlinux                  vmlinux                  find_busiest_queue
205      9.0e-05  oprofile.ko              oprofile                 event_buffer_read
204      9.0e-05  vmlinux                  vmlinux                  ext3_orphan_add
202      8.9e-05  vmlinux                  vmlinux                  ext3_release_file
201      8.9e-05  vmlinux                  vmlinux                  do_IRQ
201      8.9e-05  vmlinux                  vmlinux                  exit_mm
198      8.7e-05  vmlinux                  vmlinux                  error_code
198      8.7e-05  vmlinux                  vmlinux                  kthread_should_stop
198      8.7e-05  vmlinux                  vmlinux                  radix_tree_gang_lookup_tag
197      8.7e-05  vmlinux                  vmlinux                  update_wall_time_one_tick
196      8.6e-05  vmlinux                  vmlinux                  can_migrate_task
195      8.6e-05  vmlinux                  vmlinux                  __scsi_done
195      8.6e-05  vmlinux                  vmlinux                  get_request_wait
195      8.6e-05  vmlinux                  vmlinux                  remove_vma_list
194      8.5e-05  vmlinux                  vmlinux                  __dentry_open
192      8.5e-05  vmlinux                  vmlinux                  __fput
192      8.5e-05  vmlinux                  vmlinux                  task_curr
191      8.4e-05  vmlinux                  vmlinux                  elv_dispatch_sort
191      8.4e-05  vmlinux                  vmlinux                  scsi_io_completion
189      8.3e-05  vmlinux                  vmlinux                  ext3_new_inode
187      8.2e-05  vmlinux                  vmlinux                  put_files_struct
185      8.1e-05  vmlinux                  vmlinux                  resume_kernel
184      8.1e-05  libc-2.3.3.so            libc-2.3.3.so            rand_r
179      7.9e-05  libc-2.3.3.so            libc-2.3.3.so            __GI___libc_malloc
178      7.8e-05  vmlinux                  vmlinux                  dup_task_struct
176      7.7e-05  libc-2.3.3.so            libc-2.3.3.so            __GI___munmap
176      7.7e-05  vmlinux                  vmlinux                  collect_signal
175      7.7e-05  vmlinux                  vmlinux                  fget
174      7.7e-05  vmlinux                  vmlinux                  __drain_alien_cache
173      7.6e-05  vmlinux                  vmlinux                  __freed_request
173      7.6e-05  vmlinux                  vmlinux                  disk_round_stats
171      7.5e-05  vmlinux                  vmlinux                  filemap_fdatawait
171      7.5e-05  vmlinux                  vmlinux                  profile_task_exit
170      7.5e-05  vmlinux                  vmlinux                  permission
169      7.4e-05  vmlinux                  vmlinux                  sys_mmap2
168      7.4e-05  vmlinux                  vmlinux                  release_task
167      7.4e-05  vmlinux                  vmlinux                  __elv_add_request
164      7.2e-05  libpthread-0.10.so       libpthread-0.10.so       __pthread_unlock
164      7.2e-05  vmlinux                  vmlinux                  blk_do_ordered
163      7.2e-05  libc-2.3.3.so            libc-2.3.3.so            __GI_getpid
163      7.2e-05  libpthread-0.10.so       libpthread-0.10.so       __libc_fsync
163      7.2e-05  vmlinux                  vmlinux                  __queue_work
163      7.2e-05  vmlinux                  vmlinux                  ext3_truncate
162      7.1e-05  libc-2.3.3.so            libc-2.3.3.so            __GI_read
162      7.1e-05  vmlinux                  vmlinux                  read_page_state_offset
162      7.1e-05  vmlinux                  vmlinux                  sched_balance_self
161      7.1e-05  vmlinux                  vmlinux                  generic_file_open
160      7.0e-05  vmlinux                  vmlinux                  scsi_run_queue
159      7.0e-05  vmlinux                  vmlinux                  unmap_page_range
158      7.0e-05  vmlinux                  vmlinux                  init_timer
158      7.0e-05  vmlinux                  vmlinux                  schedule_timeout
156      6.9e-05  libpthread-0.10.so       libpthread-0.10.so       pthread_join
156      6.9e-05  vmlinux                  vmlinux                  find_vma_prepare
156      6.9e-05  vmlinux                  vmlinux                  rb_prev
156      6.9e-05  vmlinux                  vmlinux                  split_vma
156      6.9e-05  vmlinux                  vmlinux                  wake_up_new_task
154      6.8e-05  vmlinux                  vmlinux                  mutex_trylock
152      6.7e-05  vmlinux                  vmlinux                  mutex_lock
150      6.6e-05  libc-2.3.3.so            libc-2.3.3.so            __gconv_transform_utf8_internal
149      6.6e-05  libpthread-0.10.so       libpthread-0.10.so       __GI___pthread_mutex_trylock
148      6.5e-05  libc-2.3.3.so            libc-2.3.3.so            __GI___res_state
148      6.5e-05  vmlinux                  vmlinux                  journal_get_descriptor_buffer
147      6.5e-05  vmlinux                  vmlinux                  schedule_delayed_work
146      6.4e-05  vmlinux                  vmlinux                  __detach_pid
146      6.4e-05  vmlinux                  vmlinux                  e1000_clean_rx_irq
145      6.4e-05  libc-2.3.3.so            libc-2.3.3.so            __GI___sched_setscheduler
145      6.4e-05  vmlinux                  vmlinux                  __pagevec_free
145      6.4e-05  vmlinux                  vmlinux                  as_activate_request
145      6.4e-05  vmlinux                  vmlinux                  notifier_call_chain
144      6.3e-05  vmlinux                  vmlinux                  elv_set_request
143      6.3e-05  libc-2.3.3.so            libc-2.3.3.so            _IO_default_xsputn_internal
143      6.3e-05  vmlinux                  vmlinux                  exit_notify
142      6.3e-05  vmlinux                  vmlinux                  __sigqueue_free
141      6.2e-05  vmlinux                  vmlinux                  clear_queue_congested
140      6.2e-05  vmlinux                  vmlinux                  del_timer_sync
140      6.2e-05  vmlinux                  vmlinux                  unmap_vmas
139      6.1e-05  vmlinux                  vmlinux                  current_io_context
137      6.0e-05  vmlinux                  vmlinux                  page_remove_rmap
134      5.9e-05  libpthread-0.10.so       libpthread-0.10.so       __libc_close
134      5.9e-05  vmlinux                  vmlinux                  blockable_page_cache_readahead
134      5.9e-05  vmlinux                  vmlinux                  do_sys_poll
133      5.9e-05  vmlinux                  vmlinux                  do_no_page
133      5.9e-05  vmlinux                  vmlinux                  ext3_discard_reservation
133      5.9e-05  vmlinux                  vmlinux                  send_signal
132      5.8e-05  libc-2.3.3.so            libc-2.3.3.so            __cfree
132      5.8e-05  vmlinux                  vmlinux                  add_timer_randomness
132      5.8e-05  vmlinux                  vmlinux                  blk_done_softirq
131      5.8e-05  vmlinux                  vmlinux                  radix_tree_gang_lookup
131      5.8e-05  vmlinux                  vmlinux                  vfs_llseek
130      5.7e-05  vmlinux                  vmlinux                  __lru_add_drain
128      5.6e-05  libpthread-0.10.so       libpthread-0.10.so       __pthread_internal_tsd_get
128      5.6e-05  vmlinux                  vmlinux                  scsi_alloc_sgtable
128      5.6e-05  vmlinux                  vmlinux                  vma_adjust
127      5.6e-05  vmlinux                  vmlinux                  vm_normal_page
126      5.5e-05  vmlinux                  vmlinux                  __exit_signal
126      5.5e-05  vmlinux                  vmlinux                  rcu_do_batch
126      5.5e-05  vmlinux                  vmlinux                  scsi_next_command
125      5.5e-05  vmlinux                  vmlinux                  radix_tree_node_ctor
124      5.5e-05  vmlinux                  vmlinux                  __blk_put_request
122      5.4e-05  vmlinux                  vmlinux                  as_put_request
121      5.3e-05  libc-2.3.3.so            libc-2.3.3.so            __kill
121      5.3e-05  vmlinux                  vmlinux                  drain_node_pages
121      5.3e-05  vmlinux                  vmlinux                  get_signal_to_deliver
121      5.3e-05  vmlinux                  vmlinux                  scsi_put_command
119      5.2e-05  libc-2.3.3.so            libc-2.3.3.so            __calloc
119      5.2e-05  vmlinux                  vmlinux                  __copy_user_intel
119      5.2e-05  vmlinux                  vmlinux                  try_to_del_timer_sync
118      5.2e-05  libpthread-0.10.so       libpthread-0.10.so       pthread_handle_sigcancel
118      5.2e-05  vmlinux                  vmlinux                  cleanup_timers
118      5.2e-05  vmlinux                  vmlinux                  link_path_walk
118      5.2e-05  vmlinux                  vmlinux                  pipe_writev
117      5.2e-05  vmlinux                  vmlinux                  bounce_copy_vec
117      5.2e-05  vmlinux                  vmlinux                  kobject_get
117      5.2e-05  vmlinux                  vmlinux                  scsi_decide_disposition
116      5.1e-05  vmlinux                  vmlinux                  elv_may_queue
115      5.1e-05  vmlinux                  vmlinux                  as_update_arq
115      5.1e-05  vmlinux                  vmlinux                  blk_run_queue
114      5.0e-05  vmlinux                  vmlinux                  smp_send_timer_broadcast_ipi
113      5.0e-05  vmlinux                  vmlinux                  __down_failed
112      4.9e-05  vmlinux                  vmlinux                  cascade
112      4.9e-05  vmlinux                  vmlinux                  vm_acct_memory
111      4.9e-05  libpthread-0.10.so       libpthread-0.10.so       pthread_handle_sigrestart
111      4.9e-05  vmlinux                  vmlinux                  handle_stop_signal
111      4.9e-05  vmlinux                  vmlinux                  wake_up_process
110      4.8e-05  oprofile.ko              oprofile                 munmap_notify
110      4.8e-05  vmlinux                  vmlinux                  __group_complete_signal
110      4.8e-05  vmlinux                  vmlinux                  add_disk_randomness
110      4.8e-05  vmlinux                  vmlinux                  as_update_thinktime
109      4.8e-05  oprofile.ko              oprofile                 .text.lock.buffer_sync
109      4.8e-05  vmlinux                  vmlinux                  dnotify_flush
109      4.8e-05  vmlinux                  vmlinux                  do_group_exit
109      4.8e-05  vmlinux                  vmlinux                  hrtimer_init
109      4.8e-05  vmlinux                  vmlinux                  rcu_start_batch
108      4.8e-05  vmlinux                  vmlinux                  call_rcu
108      4.8e-05  vmlinux                  vmlinux                  set_queue_congested
107      4.7e-05  libc-2.3.3.so            libc-2.3.3.so            __GI_open
107      4.7e-05  libc-2.3.3.so            libc-2.3.3.so            malloc_consolidate
107      4.7e-05  vmlinux                  vmlinux                  __pte_alloc
107      4.7e-05  vmlinux                  vmlinux                  scsi_softirq_done
106      4.7e-05  vmlinux                  vmlinux                  __pagevec_release
106      4.7e-05  vmlinux                  vmlinux                  schedule_tail
106      4.7e-05  vmlinux                  vmlinux                  unmap_region
105      4.6e-05  vmlinux                  vmlinux                  __lookup_mnt
105      4.6e-05  vmlinux                  vmlinux                  attach_pid
105      4.6e-05  vmlinux                  vmlinux                  exit_io_context
105      4.6e-05  vmlinux                  vmlinux                  memmove
104      4.6e-05  vmlinux                  vmlinux                  update_wall_time
103      4.5e-05  vmlinux                  vmlinux                  inode_init_once
102      4.5e-05  vmlinux                  vmlinux                  freed_request
102      4.5e-05  vmlinux                  vmlinux                  smp_send_reschedule
101      4.4e-05  libc-2.3.3.so            libc-2.3.3.so            __GI___mbrtowc
101      4.4e-05  vmlinux                  vmlinux                  __journal_unfile_buffer
101      4.4e-05  vmlinux                  vmlinux                  lookup_mnt
100      4.4e-05  vmlinux                  vmlinux                  file_move
99       4.4e-05  vmlinux                  vmlinux                  __journal_clean_checkpoint_list
99       4.4e-05  vmlinux                  vmlinux                  ahc_linux_queue_cmd_complete
99       4.4e-05  vmlinux                  vmlinux                  d_alloc
99       4.4e-05  vmlinux                  vmlinux                  fd_install
99       4.4e-05  vmlinux                  vmlinux                  journal_write_metadata_buffer
98       4.3e-05  vmlinux                  vmlinux                  anon_vma_prepare
98       4.3e-05  vmlinux                  vmlinux                  do_poll
98       4.3e-05  vmlinux                  vmlinux                  log_wait_commit
98       4.3e-05  vmlinux                  vmlinux                  vm_stat_account
96       4.2e-05  vmlinux                  vmlinux                  as_find_next_arq
96       4.2e-05  vmlinux                  vmlinux                  lru_cache_add_active
96       4.2e-05  vmlinux                  vmlinux                  sd_rw_intr
95       4.2e-05  vmlinux                  vmlinux                  profile_munmap
94       4.1e-05  libc-2.3.3.so            libc-2.3.3.so            __getppid
94       4.1e-05  libc-2.3.3.so            libc-2.3.3.so            __mprotect
94       4.1e-05  vmlinux                  vmlinux                  find_next_usable_block
94       4.1e-05  vmlinux                  vmlinux                  notify_change
94       4.1e-05  vmlinux                  vmlinux                  smp_reschedule_interrupt
94       4.1e-05  vmlinux                  vmlinux                  sys_rt_sigprocmask
94       4.1e-05  vmlinux                  vmlinux                  writeback_inodes
93       4.1e-05  vmlinux                  vmlinux                  free_pgd_range
93       4.1e-05  vmlinux                  vmlinux                  pipe_readv
92       4.1e-05  vmlinux                  vmlinux                  __kmalloc
92       4.1e-05  vmlinux                  vmlinux                  add_wait_queue
92       4.1e-05  vmlinux                  vmlinux                  lru_add_drain
92       4.1e-05  vmlinux                  vmlinux                  scsi_finish_command
92       4.1e-05  vmlinux                  vmlinux                  wake_up_inode
91       4.0e-05  ld-2.3.3.so              ld-2.3.3.so              strcmp
90       4.0e-05  libc-2.3.3.so            libc-2.3.3.so            __uselocale
90       4.0e-05  tiotest                  tiotest                  wait_for_threads
90       4.0e-05  vmlinux                  vmlinux                  copy_from_user
89       3.9e-05  vmlinux                  vmlinux                  __path_lookup_intent_open
89       3.9e-05  vmlinux                  vmlinux                  sched_exit
89       3.9e-05  vmlinux                  vmlinux                  scsi_free_sgtable
89       3.9e-05  vmlinux                  vmlinux                  sys_getrusage
88       3.9e-05  libpthread-0.10.so       libpthread-0.10.so       __pthread_sigsuspend
88       3.9e-05  vmlinux                  vmlinux                  __sync_single_inode
88       3.9e-05  vmlinux                  vmlinux                  alloc_slabmgmt
88       3.9e-05  vmlinux                  vmlinux                  sys_munmap
87       3.8e-05  libc-2.3.3.so            libc-2.3.3.so            _IO_new_fclose
87       3.8e-05  libc-2.3.3.so            libc-2.3.3.so            __GI_write
86       3.8e-05  vmlinux                  vmlinux                  as_can_anticipate
86       3.8e-05  vmlinux                  vmlinux                  do_notify_parent
86       3.8e-05  vmlinux                  vmlinux                  flush_tlb_page
86       3.8e-05  vmlinux                  vmlinux                  wait_on_page_writeback_range
85       3.7e-05  libpthread-0.10.so       libpthread-0.10.so       __pthread_alt_lock
85       3.7e-05  vmlinux                  vmlinux                  __rb_rotate_right
84       3.7e-05  vmlinux                  vmlinux                  __up_wakeup
83       3.7e-05  vmlinux                  vmlinux                  __put_task_struct_cb
83       3.7e-05  vmlinux                  vmlinux                  anon_vma_unlink
83       3.7e-05  vmlinux                  vmlinux                  as_fifo_expired
82       3.6e-05  vmlinux                  vmlinux                  e1000_intr
82       3.6e-05  vmlinux                  vmlinux                  requeue_task
82       3.6e-05  vmlinux                  vmlinux                  vfs_permission
81       3.6e-05  vmlinux                  vmlinux                  __up
81       3.6e-05  vmlinux                  vmlinux                  remove_vma
81       3.6e-05  vmlinux                  vmlinux                  setup_frame
81       3.6e-05  vmlinux                  vmlinux                  slab_destroy
81       3.6e-05  vmlinux                  vmlinux                  sys_open
80       3.5e-05  vmlinux                  vmlinux                  ext3_setattr
79       3.5e-05  libc-2.3.3.so            libc-2.3.3.so            ___fxstat64
79       3.5e-05  vmlinux                  vmlinux                  __anon_vma_link
79       3.5e-05  vmlinux                  vmlinux                  file_ra_state_init
79       3.5e-05  vmlinux                  vmlinux                  mntput_no_expire
78       3.4e-05  vmlinux                  vmlinux                  do_fsync
78       3.4e-05  vmlinux                  vmlinux                  get_device
78       3.4e-05  vmlinux                  vmlinux                  hrtimer_try_to_cancel
78       3.4e-05  vmlinux                  vmlinux                  work_pending
77       3.4e-05  libc-2.3.3.so            libc-2.3.3.so            _IO_file_overflow_internal
77       3.4e-05  vmlinux                  vmlinux                  ext3_orphan_del
77       3.4e-05  vmlinux                  vmlinux                  vma_link
76       3.3e-05  libc-2.3.3.so            libc-2.3.3.so            _IO_fwrite_internal
75       3.3e-05  vmlinux                  vmlinux                  __scsi_get_command
75       3.3e-05  vmlinux                  vmlinux                  change_protection
75       3.3e-05  vmlinux                  vmlinux                  sys_mprotect
74       3.3e-05  libc-2.3.3.so            libc-2.3.3.so            _int_memalign
74       3.3e-05  vmlinux                  vmlinux                  alloc_page_vma
74       3.3e-05  vmlinux                  vmlinux                  getname
73       3.2e-05  tiotest                  tiotest                  do_test
73       3.2e-05  vmlinux                  vmlinux                  __group_send_sig_info
73       3.2e-05  vmlinux                  vmlinux                  __vm_enough_memory
73       3.2e-05  vmlinux                  vmlinux                  pagevec_lookup_tag
73       3.2e-05  vmlinux                  vmlinux                  rsv_window_remove
72       3.2e-05  vmlinux                  vmlinux                  ext3_free_blocks
72       3.2e-05  vmlinux                  vmlinux                  journal_get_create_access
72       3.2e-05  vmlinux                  vmlinux                  pipe_poll
71       3.1e-05  vmlinux                  vmlinux                  alloc_inode
71       3.1e-05  vmlinux                  vmlinux                  ext3_alloc_block
71       3.1e-05  vmlinux                  vmlinux                  locks_remove_posix
69       3.0e-05  ld-2.3.3.so              ld-2.3.3.so              _dl_lookup_symbol_x
69       3.0e-05  vmlinux                  vmlinux                  do_lookup
69       3.0e-05  vmlinux                  vmlinux                  group_send_sig_info
68       3.0e-05  libc-2.3.3.so            libc-2.3.3.so            __GI_strstr
68       3.0e-05  vmlinux                  vmlinux                  inode_setattr
68       3.0e-05  vmlinux                  vmlinux                  vma_merge
67       3.0e-05  ld-2.3.3.so              ld-2.3.3.so              __GI__dl_allocate_tls
67       3.0e-05  vmlinux                  vmlinux                  alloc_pidmap
67       3.0e-05  vmlinux                  vmlinux                  ext3_permission
67       3.0e-05  vmlinux                  vmlinux                  ext3_sync_file
66       2.9e-05  libc-2.3.3.so            libc-2.3.3.so            __GI_memcpy
66       2.9e-05  vmlinux                  vmlinux                  __add_entropy_words
66       2.9e-05  vmlinux                  vmlinux                  __cache_alloc_node
66       2.9e-05  vmlinux                  vmlinux                  filemap_nopage
65       2.9e-05  vmlinux                  vmlinux                  free_pgtables
65       2.9e-05  vmlinux                  vmlinux                  in_group_p
65       2.9e-05  vmlinux                  vmlinux                  ioc_set_batching
65       2.9e-05  vmlinux                  vmlinux                  set_slab_attr
64       2.8e-05  vmlinux                  vmlinux                  rwsem_wake
63       2.8e-05  ld-2.3.3.so              ld-2.3.3.so              _dl_relocate_object
63       2.8e-05  libc-2.3.3.so            libc-2.3.3.so            _IO_file_close_internal
63       2.8e-05  libc-2.3.3.so            libc-2.3.3.so            __i686.get_pc_thunk.cx
63       2.8e-05  libc-2.3.3.so            libc-2.3.3.so            __libc_thread_freeres
63       2.8e-05  vmlinux                  vmlinux                  __lookup_hash
63       2.8e-05  vmlinux                  vmlinux                  __pollwait
63       2.8e-05  vmlinux                  vmlinux                  do_filp_open
62       2.7e-05  vmlinux                  vmlinux                  __put_unused_fd
62       2.7e-05  vmlinux                  vmlinux                  as_close_req
62       2.7e-05  vmlinux                  vmlinux                  free_pte_range
62       2.7e-05  vmlinux                  vmlinux                  may_expand_vm
62       2.7e-05  vmlinux                  vmlinux                  sig_ignored
61       2.7e-05  libc-2.3.3.so            libc-2.3.3.so            _IO_new_file_fopen
60       2.6e-05  libc-2.3.3.so            libc-2.3.3.so            _IO_new_file_xsputn
60       2.6e-05  vmlinux                  vmlinux                  do_signal
60       2.6e-05  vmlinux                  vmlinux                  kobject_put
59       2.6e-05  libc-2.3.3.so            libc-2.3.3.so            res_thread_freeres
59       2.6e-05  vmlinux                  vmlinux                  as_get_io_context
59       2.6e-05  vmlinux                  vmlinux                  copy_pte_range
59       2.6e-05  vmlinux                  vmlinux                  need_resched
58       2.6e-05  libc-2.3.3.so            libc-2.3.3.so            __GI___lseek
58       2.6e-05  vmlinux                  vmlinux                  do_sys_open
58       2.6e-05  vmlinux                  vmlinux                  find_group_other
58       2.6e-05  vmlinux                  vmlinux                  get_write_access
58       2.6e-05  vmlinux                  vmlinux                  journal_free_journal_head
58       2.6e-05  vmlinux                  vmlinux                  proc_pid_unhash
58       2.6e-05  vmlinux                  vmlinux                  signal_wake_up
57       2.5e-05  vmlinux                  vmlinux                  arch_get_unmapped_area_topdown
57       2.5e-05  vmlinux                  vmlinux                  journal_grab_journal_head
57       2.5e-05  vmlinux                  vmlinux                  scsi_add_timer
56       2.5e-05  libpthread-0.10.so       libpthread-0.10.so       anonymous symbol from section .plt
56       2.5e-05  vmlinux                  vmlinux                  blk_plug_device
56       2.5e-05  vmlinux                  vmlinux                  detach_vmas_to_be_unmapped
56       2.5e-05  vmlinux                  vmlinux                  handle_signal
56       2.5e-05  vmlinux                  vmlinux                  sys_set_thread_area
54       2.4e-05  libc-2.3.3.so            libc-2.3.3.so            _IO_new_do_write
54       2.4e-05  vmlinux                  vmlinux                  journal_file_buffer
54       2.4e-05  vmlinux                  vmlinux                  sync_sb_inodes
53       2.3e-05  libc-2.3.3.so            libc-2.3.3.so            _IO_file_close_it_internal
53       2.3e-05  libc-2.3.3.so            libc-2.3.3.so            __rpc_thread_destroy
53       2.3e-05  vmlinux                  vmlinux                  __writeback_single_inode
53       2.3e-05  vmlinux                  vmlinux                  end_edge_ioapic_irq
53       2.3e-05  vmlinux                  vmlinux                  inode_sub_bytes
52       2.3e-05  init                     init                     (no symbols)
52       2.3e-05  libc-2.3.3.so            libc-2.3.3.so            _IO_link_in_internal
52       2.3e-05  libc-2.3.3.so            libc-2.3.3.so            __memalign
52       2.3e-05  vmlinux                  vmlinux                  e1000_alloc_rx_buffers
52       2.3e-05  vmlinux                  vmlinux                  restore_sigcontext
51       2.2e-05  libpthread-0.10.so       libpthread-0.10.so       __pthread_wait_for_restart_signal
51       2.2e-05  vmlinux                  vmlinux                  filp_close
51       2.2e-05  vmlinux                  vmlinux                  free_task
51       2.2e-05  vmlinux                  vmlinux                  next_thread
51       2.2e-05  vmlinux                  vmlinux                  rwsem_down_read_failed
50       2.2e-05  libc-2.3.3.so            libc-2.3.3.so            __GI__IO_file_open
50       2.2e-05  libc-2.3.3.so            libc-2.3.3.so            __flockfile
50       2.2e-05  libpthread-0.10.so       libpthread-0.10.so       __GI___pthread_mutex_unlock
50       2.2e-05  tiotest                  tiotest                  timer_usertime
50       2.2e-05  vmlinux                  vmlinux                  __mutex_init
50       2.2e-05  vmlinux                  vmlinux                  adjtime_adjustment
50       2.2e-05  vmlinux                  vmlinux                  copy_thread
50       2.2e-05  vmlinux                  vmlinux                  forget_original_parent
50       2.2e-05  vmlinux                  vmlinux                  nameidata_to_filp
49       2.2e-05  oprofile.ko              oprofile                 task_exit_notify
49       2.2e-05  vmlinux                  vmlinux                  double_rq_lock
49       2.2e-05  vmlinux                  vmlinux                  get_vma_policy
48       2.1e-05  libc-2.3.3.so            libc-2.3.3.so            _IO_un_link_internal
48       2.1e-05  vmlinux                  vmlinux                  __filemap_fdatawrite_range
48       2.1e-05  vmlinux                  vmlinux                  radix_tree_extend
48       2.1e-05  vmlinux                  vmlinux                  scsi_delete_timer
48       2.1e-05  vmlinux                  vmlinux                  scsi_done
48       2.1e-05  vmlinux                  vmlinux                  sys_sigreturn
47       2.1e-05  vmlinux                  vmlinux                  cp_new_stat64
47       2.1e-05  vmlinux                  vmlinux                  do_munmap
47       2.1e-05  vmlinux                  vmlinux                  get_nr_files
47       2.1e-05  vmlinux                  vmlinux                  sys_fstat64
46       2.0e-05  vmlinux                  vmlinux                  process_timeout
46       2.0e-05  vmlinux                  vmlinux                  sys_rt_sigsuspend
46       2.0e-05  vmlinux                  vmlinux                  unlink_file_vma
45       2.0e-05  vmlinux                  vmlinux                  __vma_link
45       2.0e-05  vmlinux                  vmlinux                  detach_pid
45       2.0e-05  vmlinux                  vmlinux                  locks_remove_flock
45       2.0e-05  vmlinux                  vmlinux                  make_ahead_window
45       2.0e-05  vmlinux                  vmlinux                  sigprocmask
45       2.0e-05  vmlinux                  vmlinux                  unmap_mapping_range
45       2.0e-05  vmlinux                  vmlinux                  vmtruncate
44       1.9e-05  vmlinux                  vmlinux                  eventpoll_init_file
44       1.9e-05  vmlinux                  vmlinux                  sys_getppid
43       1.9e-05  libpthread-0.10.so       libpthread-0.10.so       pthread_free
43       1.9e-05  vmlinux                  vmlinux                  find_mergeable_anon_vma
43       1.9e-05  vmlinux                  vmlinux                  mask_and_ack_level_ioapic_irq
43       1.9e-05  vmlinux                  vmlinux                  setup_sigcontext
42       1.8e-05  vmlinux                  vmlinux                  debug_mutex_init
42       1.8e-05  vmlinux                  vmlinux                  mprotect_fixup
42       1.8e-05  vmlinux                  vmlinux                  pmd_ctor
41       1.8e-05  libc-2.3.3.so            libc-2.3.3.so            __funlockfile
41       1.8e-05  vmlinux                  vmlinux                  free_pidmap
41       1.8e-05  vmlinux                  vmlinux                  interleave_nodes
41       1.8e-05  vmlinux                  vmlinux                  next_reap_node
40       1.8e-05  libc-2.3.3.so            libc-2.3.3.so            __GI_memset
40       1.8e-05  vmlinux                  vmlinux                  cap_vm_enough_memory
40       1.8e-05  vmlinux                  vmlinux                  do_truncate
40       1.8e-05  vmlinux                  vmlinux                  elv_dequeue_request
40       1.8e-05  vmlinux                  vmlinux                  ext3_free_inode
40       1.8e-05  vmlinux                  vmlinux                  journal_forget
39       1.7e-05  vmlinux                  vmlinux                  __dequeue_signal
39       1.7e-05  vmlinux                  vmlinux                  groups_search
39       1.7e-05  vmlinux                  vmlinux                  journal_brelse_array
39       1.7e-05  vmlinux                  vmlinux                  new_inode
39       1.7e-05  vmlinux                  vmlinux                  second_overflow
39       1.7e-05  vmlinux                  vmlinux                  sys_poll
38       1.7e-05  ld-2.3.3.so              ld-2.3.3.so              fixup
38       1.7e-05  vmlinux                  vmlinux                  free_uid
38       1.7e-05  vmlinux                  vmlinux                  ip_route_input_slow
38       1.7e-05  vmlinux                  vmlinux                  put_device
38       1.7e-05  vmlinux                  vmlinux                  sched_setscheduler
38       1.7e-05  vmlinux                  vmlinux                  vfs_getattr
37       1.6e-05  tiotest                  tiotest                  timer_start
37       1.6e-05  vmlinux                  vmlinux                  __try_to_free_cp_buf
37       1.6e-05  vmlinux                  vmlinux                  as_antic_stop
37       1.6e-05  vmlinux                  vmlinux                  i8042_timer_func
37       1.6e-05  vmlinux                  vmlinux                  pagevec_lookup
36       1.6e-05  ld-2.3.3.so              ld-2.3.3.so              __GI__dl_deallocate_tls
36       1.6e-05  libc-2.3.3.so            libc-2.3.3.so            _IO_file_doallocate_internal
36       1.6e-05  libc-2.3.3.so            libc-2.3.3.so            __GI__exit
36       1.6e-05  tiotest                  tiotest                  cleanup_test
36       1.6e-05  vmlinux                  vmlinux                  preempt_schedule_irq
36       1.6e-05  vmlinux                  vmlinux                  queue_work
36       1.6e-05  vmlinux                  vmlinux                  sched_fork
36       1.6e-05  vmlinux                  vmlinux                  sys_exit_group
35       1.5e-05  vmlinux                  vmlinux                  exit_sem
35       1.5e-05  vmlinux                  vmlinux                  free_pages_and_swap_cache
35       1.5e-05  vmlinux                  vmlinux                  page_add_file_rmap
34       1.5e-05  libc-2.3.3.so            libc-2.3.3.so            _IO_new_file_write
34       1.5e-05  tiotest                  tiotest                  timer_stop
34       1.5e-05  vmlinux                  vmlinux                  __page_set_anon_rmap
34       1.5e-05  vmlinux                  vmlinux                  do_notify_resume
34       1.5e-05  vmlinux                  vmlinux                  ext3_getblk
34       1.5e-05  vmlinux                  vmlinux                  mod_timer
34       1.5e-05  vmlinux                  vmlinux                  poll_freewait
34       1.5e-05  vmlinux                  vmlinux                  sys_getpid
33       1.5e-05  libc-2.3.3.so            libc-2.3.3.so            __fopen_internal
33       1.5e-05  libc-2.3.3.so            libc-2.3.3.so            __libc_enable_asynccancel
33       1.5e-05  libpthread-0.10.so       libpthread-0.10.so       __GI___pthread_mutex_lock
33       1.5e-05  vmlinux                  vmlinux                  arp_process
33       1.5e-05  vmlinux                  vmlinux                  copy_mm
33       1.5e-05  vmlinux                  vmlinux                  e1000_clean_tx_irq
33       1.5e-05  vmlinux                  vmlinux                  exit_itimers
33       1.5e-05  vmlinux                  vmlinux                  rwsem_down_write_failed
32       1.4e-05  libc-2.3.3.so            libc-2.3.3.so            _IO_vfprintf_internal
32       1.4e-05  vmlinux                  vmlinux                  __unhash_process
32       1.4e-05  vmlinux                  vmlinux                  copy_fdtable
32       1.4e-05  vmlinux                  vmlinux                  do_fork
32       1.4e-05  vmlinux                  vmlinux                  ext3_find_near
32       1.4e-05  vmlinux                  vmlinux                  journal_switch_revoke_table
32       1.4e-05  vmlinux                  vmlinux                  vgacon_save_screen
31       1.4e-05  vmlinux                  vmlinux                  __wake_up_locked
31       1.4e-05  vmlinux                  vmlinux                  can_vma_merge_before
31       1.4e-05  vmlinux                  vmlinux                  do_sched_setscheduler
31       1.4e-05  vmlinux                  vmlinux                  ext3_add_entry
31       1.4e-05  vmlinux                  vmlinux                  find_idlest_cpu
31       1.4e-05  vmlinux                  vmlinux                  fn_hash_lookup
30       1.3e-05  libc-2.3.3.so            libc-2.3.3.so            _IO_setb_internal
30       1.3e-05  vmlinux                  vmlinux                  __insert_inode_hash
30       1.3e-05  vmlinux                  vmlinux                  _spin_trylock
30       1.3e-05  vmlinux                  vmlinux                  do_pollfd
30       1.3e-05  vmlinux                  vmlinux                  e1000_watchdog_task
30       1.3e-05  vmlinux                  vmlinux                  ret_from_exception
29       1.3e-05  libc-2.3.3.so            libc-2.3.3.so            __GI___getpagesize
29       1.3e-05  libc-2.3.3.so            libc-2.3.3.so            anonymous symbol from section .plt
29       1.3e-05  tiotest                  tiotest                  print_results
29       1.3e-05  vmlinux                  vmlinux                  generic_fillattr
29       1.3e-05  vmlinux                  vmlinux                  getrusage
29       1.3e-05  vmlinux                  vmlinux                  hrtimer_cancel
29       1.3e-05  vmlinux                  vmlinux                  inode_change_ok
29       1.3e-05  vmlinux                  vmlinux                  mm_release
29       1.3e-05  vmlinux                  vmlinux                  sys_fsync
28       1.2e-05  libc-2.3.3.so            libc-2.3.3.so            _IO_default_finish_internal
28       1.2e-05  vmlinux                  vmlinux                  __insert_vm_struct
28       1.2e-05  vmlinux                  vmlinux                  dequeue_signal
28       1.2e-05  vmlinux                  vmlinux                  e1000_check_for_link
28       1.2e-05  vmlinux                  vmlinux                  journal_write_revoke_records
28       1.2e-05  vmlinux                  vmlinux                  kmem_flagcheck
28       1.2e-05  vmlinux                  vmlinux                  lock_hrtimer_base
27       1.2e-05  libc-2.3.3.so            libc-2.3.3.so            _IO_file_stat_internal
27       1.2e-05  libc-2.3.3.so            libc-2.3.3.so            __GI_strchr
27       1.2e-05  libpthread-0.10.so       libpthread-0.10.so       __errno_location
27       1.2e-05  vmlinux                  vmlinux                  __copy_user_zeroing_intel
27       1.2e-05  vmlinux                  vmlinux                  __exit_sighand
27       1.2e-05  vmlinux                  vmlinux                  do_mpage_readpage
27       1.2e-05  vmlinux                  vmlinux                  get_unmapped_area
27       1.2e-05  vmlinux                  vmlinux                  neigh_lookup
27       1.2e-05  vmlinux                  vmlinux                  pipe_iov_copy_from_user
27       1.2e-05  vmlinux                  vmlinux                  schedule_timeout_uninterruptible
27       1.2e-05  vmlinux                  vmlinux                  start_transaction
27       1.2e-05  vmlinux                  vmlinux                  sys_sched_setscheduler
26       1.1e-05  libc-2.3.3.so            libc-2.3.3.so            _IO_no_init
26       1.1e-05  vmlinux                  vmlinux                  next_signal
25       1.1e-05  ld-2.3.3.so              ld-2.3.3.so              anonymous symbol from section .plt
25       1.1e-05  vmlinux                  vmlinux                  __log_start_commit
25       1.1e-05  vmlinux                  vmlinux                  __vma_link_rb
25       1.1e-05  vmlinux                  vmlinux                  __wake_up_sync
25       1.1e-05  vmlinux                  vmlinux                  copy_files
25       1.1e-05  vmlinux                  vmlinux                  end_buffer_write_sync
25       1.1e-05  vmlinux                  vmlinux                  ext3_unlink
25       1.1e-05  vmlinux                  vmlinux                  free_pages
25       1.1e-05  vmlinux                  vmlinux                  netif_rx
25       1.1e-05  vmlinux                  vmlinux                  nr_running
25       1.1e-05  vmlinux                  vmlinux                  path_lookup_open
24       1.1e-05  libc-2.3.3.so            libc-2.3.3.so            _IO_old_init
24       1.1e-05  libc-2.3.3.so            libc-2.3.3.so            __GI_strcmp
24       1.1e-05  vmlinux                  vmlinux                  __free_pages_ok
24       1.1e-05  vmlinux                  vmlinux                  dup_mm
24       1.1e-05  vmlinux                  vmlinux                  ext3_free_branches
24       1.1e-05  vmlinux                  vmlinux                  pipe_iov_copy_to_user
23       1.0e-05  libc-2.3.3.so            libc-2.3.3.so            __GI___unlink
23       1.0e-05  oprofile.ko              oprofile                 process_task_mortuary
23       1.0e-05  tiotest                  tiotest                  initialize_test
23       1.0e-05  vmlinux                  vmlinux                  copy_semundo
23       1.0e-05  vmlinux                  vmlinux                  e1000_read_phy_reg_ex
23       1.0e-05  vmlinux                  vmlinux                  file_kill
23       1.0e-05  vmlinux                  vmlinux                  lookup_hash
23       1.0e-05  vmlinux                  vmlinux                  slab_destroy_objs
22       9.7e-06  libpthread-0.10.so       libpthread-0.10.so       __pthread_perform_cleanup
22       9.7e-06  libpthread-0.10.so       libpthread-0.10.so       pthread_getspecific
22       9.7e-06  vmlinux                  vmlinux                  anon_pipe_buf_unmap
22       9.7e-06  vmlinux                  vmlinux                  blk_phys_contig_segment
22       9.7e-06  vmlinux                  vmlinux                  ext3_create
22       9.7e-06  vmlinux                  vmlinux                  release_vm86_irqs
22       9.7e-06  vmlinux                  vmlinux                  sync_supers
21       9.2e-06  ld-2.3.3.so              ld-2.3.3.so              __i686.get_pc_thunk.bx
21       9.2e-06  libc-2.3.3.so            libc-2.3.3.so            _IO_doallocbuf_internal
21       9.2e-06  libc-2.3.3.so            libc-2.3.3.so            _IO_new_fopen
21       9.2e-06  vmlinux                  vmlinux                  double_rq_unlock
21       9.2e-06  vmlinux                  vmlinux                  ext3_set_inode_flags
21       9.2e-06  vmlinux                  vmlinux                  file_free_rcu
21       9.2e-06  vmlinux                  vmlinux                  journal_unfile_buffer
21       9.2e-06  vmlinux                  vmlinux                  kill_proc_info
21       9.2e-06  vmlinux                  vmlinux                  radix_tree_node_alloc
21       9.2e-06  vmlinux                  vmlinux                  save_i387
21       9.2e-06  vmlinux                  vmlinux                  sys_kill
20       8.8e-06  libc-2.3.3.so            libc-2.3.3.so            __find_specmb
20       8.8e-06  vmlinux                  vmlinux                  __alloc_skb
20       8.8e-06  vmlinux                  vmlinux                  as_work_handler
20       8.8e-06  vmlinux                  vmlinux                  do_writepages
20       8.8e-06  vmlinux                  vmlinux                  expand_files
20       8.8e-06  vmlinux                  vmlinux                  ext3_delete_inode
20       8.8e-06  vmlinux                  vmlinux                  find_task_by_pid_type
20       8.8e-06  vmlinux                  vmlinux                  free_hot_page
20       8.8e-06  vmlinux                  vmlinux                  page_add_new_anon_rmap
20       8.8e-06  vmlinux                  vmlinux                  truncate_inode_pages
19       8.4e-06  libc-2.3.3.so            libc-2.3.3.so            __GI_sigdelset
19       8.4e-06  libpthread-0.10.so       libpthread-0.10.so       __pthread_internal_tsd_set
19       8.4e-06  vmlinux                  vmlinux                  check_kill_permission
19       8.4e-06  vmlinux                  vmlinux                  elv_put_request
19       8.4e-06  vmlinux                  vmlinux                  ext3_delete_entry
19       8.4e-06  vmlinux                  vmlinux                  may_delete
19       8.4e-06  vmlinux                  vmlinux                  slab_node
19       8.4e-06  vmlinux                  vmlinux                  strnlen_user
18       7.9e-06  ld-2.3.3.so              ld-2.3.3.so              _dl_name_match_p
18       7.9e-06  libc-2.3.3.so            libc-2.3.3.so            __GI___printf_fp
18       7.9e-06  libc-2.3.3.so            libc-2.3.3.so            __GI_strcpy
18       7.9e-06  libpthread-0.10.so       libpthread-0.10.so       __pthread_manager_adjust_prio
18       7.9e-06  vmlinux                  vmlinux                  cached_lookup
18       7.9e-06  vmlinux                  vmlinux                  capable
18       7.9e-06  vmlinux                  vmlinux                  ip_route_input
18       7.9e-06  vmlinux                  vmlinux                  sys_clone
17       7.5e-06  libcrypto.so.0.9.7a      libcrypto.so.0.9.7a      (no symbols)
17       7.5e-06  sshd                     sshd                     (no symbols)
17       7.5e-06  vmlinux                  vmlinux                  __get_free_pages
17       7.5e-06  vmlinux                  vmlinux                  count_open_files
17       7.5e-06  vmlinux                  vmlinux                  exit_as_io_context
17       7.5e-06  vmlinux                  vmlinux                  journal_end_buffer_io_sync
17       7.5e-06  vmlinux                  vmlinux                  netif_receive_skb
16       7.0e-06  libc-2.3.3.so            libc-2.3.3.so            __GI__dl_mcount_wrapper_check
16       7.0e-06  vmlinux                  vmlinux                  arp_rcv
16       7.0e-06  vmlinux                  vmlinux                  blocks_for_truncate
16       7.0e-06  vmlinux                  vmlinux                  do_getname
16       7.0e-06  vmlinux                  vmlinux                  do_select
16       7.0e-06  vmlinux                  vmlinux                  do_unlinkat
16       7.0e-06  vmlinux                  vmlinux                  ext3_write_inode
16       7.0e-06  vmlinux                  vmlinux                  lock_kernel
15       6.6e-06  libc-2.3.3.so            libc-2.3.3.so            _IO_new_file_init
15       6.6e-06  vmlinux                  vmlinux                  cap_capable
15       6.6e-06  vmlinux                  vmlinux                  fib_semantic_match
15       6.6e-06  vmlinux                  vmlinux                  find_revoke_record
15       6.6e-06  vmlinux                  vmlinux                  grow_dev_page
15       6.6e-06  vmlinux                  vmlinux                  vfs_create
15       6.6e-06  vmlinux                  vmlinux                  write_inode
14       6.2e-06  libc-2.3.3.so            libc-2.3.3.so            __GI_memmove
14       6.2e-06  oprofile.ko              oprofile                 .text.lock.event_buffer
14       6.2e-06  vmlinux                  vmlinux                  __setscheduler
14       6.2e-06  vmlinux                  vmlinux                  arch_unmap_area_topdown
14       6.2e-06  vmlinux                  vmlinux                  as_find_first_arq
14       6.2e-06  vmlinux                  vmlinux                  bmap
14       6.2e-06  vmlinux                  vmlinux                  eth_type_trans
14       6.2e-06  vmlinux                  vmlinux                  kill_fasync
14       6.2e-06  vmlinux                  vmlinux                  read_inode_bitmap
13       5.7e-06  oprofile.ko              oprofile                 task_free_notify
13       5.7e-06  vmlinux                  vmlinux                  anon_vma_link
13       5.7e-06  vmlinux                  vmlinux                  as_add_arq_rb
13       5.7e-06  vmlinux                  vmlinux                  copy_namespace
13       5.7e-06  vmlinux                  vmlinux                  filemap_fdatawrite
13       5.7e-06  vmlinux                  vmlinux                  inotify_inode_is_dead
13       5.7e-06  vmlinux                  vmlinux                  process_backlog
12       5.3e-06  ld-2.3.3.so              ld-2.3.3.so              dl_main
12       5.3e-06  libc-2.3.3.so            libc-2.3.3.so            _IO_unsave_markers_internal
12       5.3e-06  libc-2.3.3.so            libc-2.3.3.so            __GI_strlen
12       5.3e-06  libc-2.3.3.so            libc-2.3.3.so            getc
12       5.3e-06  libpthread-0.10.so       libpthread-0.10.so       __pthread_restart_new
12       5.3e-06  vmlinux                  vmlinux                  _read_unlock_irqrestore
12       5.3e-06  vmlinux                  vmlinux                  ext3_alloc_inode
12       5.3e-06  vmlinux                  vmlinux                  get_init_ra_size
12       5.3e-06  vmlinux                  vmlinux                  posix_cpu_timers_exit
11       4.8e-06  vmlinux                  vmlinux                  debug_mutex_wake_waiter
11       4.8e-06  vmlinux                  vmlinux                  dup_fd
11       4.8e-06  vmlinux                  vmlinux                  ext3_lookup
11       4.8e-06  vmlinux                  vmlinux                  is_bad_inode
11       4.8e-06  vmlinux                  vmlinux                  ll_merge_requests_fn
11       4.8e-06  vmlinux                  vmlinux                  recalc_sigpending
11       4.8e-06  vmlinux                  vmlinux                  try_to_extend_transaction
11       4.8e-06  vmlinux                  vmlinux                  work_notifysig
10       4.4e-06  libpthread-0.10.so       libpthread-0.10.so       __pthread_manager_sighandler
10       4.4e-06  vmlinux                  vmlinux                  __dispose_buffer
10       4.4e-06  vmlinux                  vmlinux                  anon_pipe_buf_map
10       4.4e-06  vmlinux                  vmlinux                  init_page_buffers
10       4.4e-06  vmlinux                  vmlinux                  journal_check_used_features
10       4.4e-06  vmlinux                  vmlinux                  memset
10       4.4e-06  vmlinux                  vmlinux                  path_lookup_create
10       4.4e-06  vmlinux                  vmlinux                  pipe_write
10       4.4e-06  vmlinux                  vmlinux                  rt_hash_code
10       4.4e-06  vmlinux                  vmlinux                  sys_waitpid
10       4.4e-06  vmlinux                  vmlinux                  wb_kupdate
9        4.0e-06  gawk                     gawk                     (no symbols)
9        4.0e-06  ld-2.3.3.so              ld-2.3.3.so              _dl_runtime_resolve
9        4.0e-06  libc-2.3.3.so            libc-2.3.3.so            __GI___sigaction
9        4.0e-06  libc-2.3.3.so            libc-2.3.3.so            __GI_select
9        4.0e-06  libc-2.3.3.so            libc-2.3.3.so            __fopen_maybe_mmap
9        4.0e-06  vmlinux                  vmlinux                  __getblk_slow
9        4.0e-06  vmlinux                  vmlinux                  __wait_on_bit
9        4.0e-06  vmlinux                  vmlinux                  as_merged_requests
9        4.0e-06  vmlinux                  vmlinux                  clear_inode
9        4.0e-06  vmlinux                  vmlinux                  default_llseek
9        4.0e-06  vmlinux                  vmlinux                  ext3_init_block_alloc_info
9        4.0e-06  vmlinux                  vmlinux                  iput
9        4.0e-06  vmlinux                  vmlinux                  journal_bmap
9        4.0e-06  vmlinux                  vmlinux                  journal_set_features
9        4.0e-06  vmlinux                  vmlinux                  mapping_tagged
9        4.0e-06  vmlinux                  vmlinux                  vfs_fstat
9        4.0e-06  vmlinux                  vmlinux                  wake_up_state
8        3.5e-06  ld-2.3.3.so              ld-2.3.3.so              _dl_load_cache_lookup
8        3.5e-06  ld-2.3.3.so              ld-2.3.3.so              _dl_map_object
8        3.5e-06  libc-2.3.3.so            libc-2.3.3.so            _IO_file_finish_internal
8        3.5e-06  libc-2.3.3.so            libc-2.3.3.so            _IO_vfscanf_internal
8        3.5e-06  libc-2.3.3.so            libc-2.3.3.so            __realloc
8        3.5e-06  libc-2.3.3.so            libc-2.3.3.so            _itoa_word
8        3.5e-06  op_help                  op_help                  (no symbols)
8        3.5e-06  vmlinux                  vmlinux                  __put_user_4
8        3.5e-06  vmlinux                  vmlinux                  balance_dirty_pages
8        3.5e-06  vmlinux                  vmlinux                  d_instantiate
8        3.5e-06  vmlinux                  vmlinux                  double_lock_balance
8        3.5e-06  vmlinux                  vmlinux                  ext3_clear_inode
8        3.5e-06  vmlinux                  vmlinux                  ext3_set_aops
8        3.5e-06  vmlinux                  vmlinux                  journal_revoke
8        3.5e-06  vmlinux                  vmlinux                  kick_process
7        3.1e-06  ld-2.3.3.so              ld-2.3.3.so              __mmap
7        3.1e-06  ld-2.3.3.so              ld-2.3.3.so              __open
7        3.1e-06  libc-2.3.3.so            libc-2.3.3.so            _IO_vsprintf_internal
7        3.1e-06  libc-2.3.3.so            libc-2.3.3.so            __GI___errno_location
7        3.1e-06  libc-2.3.3.so            libc-2.3.3.so            __libc_disable_asynccancel
7        3.1e-06  tiotest                  tiotest                  timer_systime
7        3.1e-06  vmlinux                  vmlinux                  arp_hash
7        3.1e-06  vmlinux                  vmlinux                  credit_entropy_store
7        3.1e-06  vmlinux                  vmlinux                  dentry_iput
7        3.1e-06  vmlinux                  vmlinux                  ext3_force_commit
7        3.1e-06  vmlinux                  vmlinux                  insert_revoke_hash
7        3.1e-06  vmlinux                  vmlinux                  kill_something_info
7        3.1e-06  vmlinux                  vmlinux                  pipe_read
7        3.1e-06  vmlinux                  vmlinux                  sha_transform
7        3.1e-06  vmlinux                  vmlinux                  skb_release_data
7        3.1e-06  vmlinux                  vmlinux                  sync_inode
6        2.6e-06  grep                     grep                     (no symbols)
6        2.6e-06  libc-2.3.3.so            libc-2.3.3.so            _int_realloc
6        2.6e-06  tiotest                  tiotest                  get_random_seed
6        2.6e-06  vmlinux                  vmlinux                  .text.lock.sys_i386
6        2.6e-06  vmlinux                  vmlinux                  __iget
6        2.6e-06  vmlinux                  vmlinux                  alloc_as_io_context
6        2.6e-06  vmlinux                  vmlinux                  core_sys_select
6        2.6e-06  vmlinux                  vmlinux                  d_rehash
6        2.6e-06  vmlinux                  vmlinux                  exit_thread
6        2.6e-06  vmlinux                  vmlinux                  ext3_init_acl
6        2.6e-06  vmlinux                  vmlinux                  kjournald
6        2.6e-06  vmlinux                  vmlinux                  poll_initwait
6        2.6e-06  vmlinux                  vmlinux                  radix_tree_tagged
6        2.6e-06  vmlinux                  vmlinux                  search_extable
6        2.6e-06  vmlinux                  vmlinux                  sys_unlink
5        2.2e-06  ld-2.3.3.so              ld-2.3.3.so              _dl_cache_libcmp
5        2.2e-06  ld-2.3.3.so              ld-2.3.3.so              _dl_map_object_deps
5        2.2e-06  libc-2.3.3.so            libc-2.3.3.so            __GI_getenv
5        2.2e-06  libc-2.3.3.so            libc-2.3.3.so            __GI_time
5        2.2e-06  libc-2.3.3.so            libc-2.3.3.so            __ctype_get_mb_cur_max
5        2.2e-06  libc-2.3.3.so            libc-2.3.3.so            __mpn_divrem
5        2.2e-06  libc-2.3.3.so            libc-2.3.3.so            __mpn_mul_1
5        2.2e-06  libc-2.3.3.so            libc-2.3.3.so            _nl_intern_locale_data
5        2.2e-06  syslogd                  syslogd                  (no symbols)
5        2.2e-06  vmlinux                  vmlinux                  __bforget
5        2.2e-06  vmlinux                  vmlinux                  __journal_remove_checkpoint
5        2.2e-06  vmlinux                  vmlinux                  as_antic_timeout
5        2.2e-06  vmlinux                  vmlinux                  blk_hw_contig_segment
5        2.2e-06  vmlinux                  vmlinux                  d_lookup
5        2.2e-06  vmlinux                  vmlinux                  elv_merge_requests
5        2.2e-06  vmlinux                  vmlinux                  end_that_request_chunk
5        2.2e-06  vmlinux                  vmlinux                  exec_mmap
5        2.2e-06  vmlinux                  vmlinux                  flush_tlb_all
5        2.2e-06  vmlinux                  vmlinux                  grow_buffers
5        2.2e-06  vmlinux                  vmlinux                  init_once
5        2.2e-06  vmlinux                  vmlinux                  journal_force_commit
5        2.2e-06  vmlinux                  vmlinux                  lru_cache_add
5        2.2e-06  vmlinux                  vmlinux                  path_release
5        2.2e-06  vmlinux                  vmlinux                  posix_cpu_timers_exit_group
5        2.2e-06  vmlinux                  vmlinux                  proc_pid_flush
5        2.2e-06  vmlinux                  vmlinux                  pte_alloc_one
5        2.2e-06  vmlinux                  vmlinux                  release_thread
5        2.2e-06  vmlinux                  vmlinux                  swap_io_context
5        2.2e-06  vmlinux                  vmlinux                  sys_wait4
5        2.2e-06  vmlinux                  vmlinux                  syscall_exit_work
5        2.2e-06  vmlinux                  vmlinux                  vfs_unlink
4        1.8e-06  ld-2.3.3.so              ld-2.3.3.so              _dl_map_object_from_fd
4        1.8e-06  ld-2.3.3.so              ld-2.3.3.so              _dl_sysdep_start
4        1.8e-06  libc-2.3.3.so            libc-2.3.3.so            __GI_pipe
4        1.8e-06  libc-2.3.3.so            libc-2.3.3.so            __brk
4        1.8e-06  libc-2.3.3.so            libc-2.3.3.so            __cxa_finalize
4        1.8e-06  libc-2.3.3.so            libc-2.3.3.so            __libc_nanosleep
4        1.8e-06  libc-2.3.3.so            libc-2.3.3.so            _nl_load_locale_from_archive
4        1.8e-06  libc-2.3.3.so            libc-2.3.3.so            _xstat
4        1.8e-06  libc-2.3.3.so            libc-2.3.3.so            msort_with_tmp
4        1.8e-06  vmlinux                  vmlinux                  __get_user_4
4        1.8e-06  vmlinux                  vmlinux                  _read_lock_bh
4        1.8e-06  vmlinux                  vmlinux                  active_load_balance
4        1.8e-06  vmlinux                  vmlinux                  as_antic_waitnext
4        1.8e-06  vmlinux                  vmlinux                  copy_strings
4        1.8e-06  vmlinux                  vmlinux                  d_delete
4        1.8e-06  vmlinux                  vmlinux                  deny_write_access
4        1.8e-06  vmlinux                  vmlinux                  do_sigaction
4        1.8e-06  vmlinux                  vmlinux                  exit_mmap
4        1.8e-06  vmlinux                  vmlinux                  ext3_add_nondir
4        1.8e-06  vmlinux                  vmlinux                  ext3_bread
4        1.8e-06  vmlinux                  vmlinux                  ext3_update_dx_flag
4        1.8e-06  vmlinux                  vmlinux                  net_rx_action
4        1.8e-06  vmlinux                  vmlinux                  ret_from_fork
4        1.8e-06  vmlinux                  vmlinux                  sys_rt_sigaction
4        1.8e-06  vmlinux                  vmlinux                  writeback_in_progress
3        1.3e-06  ld-2.3.3.so              ld-2.3.3.so              ___fxstat64
3        1.3e-06  ld-2.3.3.so              ld-2.3.3.so              _dl_important_hwcaps
3        1.3e-06  ld-2.3.3.so              ld-2.3.3.so              index
3        1.3e-06  libc-2.3.3.so            libc-2.3.3.so            __GI___fxstat
3        1.3e-06  libc-2.3.3.so            libc-2.3.3.so            __GI___libc_sigaction
3        1.3e-06  libc-2.3.3.so            libc-2.3.3.so            __GI___uflow
3        1.3e-06  libc-2.3.3.so            libc-2.3.3.so            __GI_mempcpy
3        1.3e-06  libc-2.3.3.so            libc-2.3.3.so            __GI_setlocale
3        1.3e-06  libc-2.3.3.so            libc-2.3.3.so            ptmalloc_init
3        1.3e-06  libpthread-0.10.so       libpthread-0.10.so       __pthread_attr_init_2_1
3        1.3e-06  libpthread-0.10.so       libpthread-0.10.so       __pthread_initialize_minimal
3        1.3e-06  libpthread-0.10.so       libpthread-0.10.so       __pthread_sigaction
3        1.3e-06  vmlinux                  vmlinux                  .text.lock.exit
3        1.3e-06  vmlinux                  vmlinux                  .text.lock.fault
3        1.3e-06  vmlinux                  vmlinux                  __bread_slow
3        1.3e-06  vmlinux                  vmlinux                  __kfree_skb
3        1.3e-06  vmlinux                  vmlinux                  __put_super_and_need_restart
3        1.3e-06  vmlinux                  vmlinux                  __remove_shared_vm_struct
3        1.3e-06  vmlinux                  vmlinux                  _read_unlock_bh
3        1.3e-06  vmlinux                  vmlinux                  _write_lock
3        1.3e-06  vmlinux                  vmlinux                  anon_pipe_buf_release
3        1.3e-06  vmlinux                  vmlinux                  balanced_irq
3        1.3e-06  vmlinux                  vmlinux                  block_read_full_page
3        1.3e-06  vmlinux                  vmlinux                  d_splice_alias
3        1.3e-06  vmlinux                  vmlinux                  destroy_inode
3        1.3e-06  vmlinux                  vmlinux                  e1000_update_adaptive
3        1.3e-06  vmlinux                  vmlinux                  e1000_xmit_frame
3        1.3e-06  vmlinux                  vmlinux                  ext3_read_inode
3        1.3e-06  vmlinux                  vmlinux                  find_or_create_page
3        1.3e-06  vmlinux                  vmlinux                  get_dirty_limits
3        1.3e-06  vmlinux                  vmlinux                  invalidate_inode_buffers
3        1.3e-06  vmlinux                  vmlinux                  io_schedule_timeout
3        1.3e-06  vmlinux                  vmlinux                  ip_rcv
3        1.3e-06  vmlinux                  vmlinux                  journal_release_buffer
3        1.3e-06  vmlinux                  vmlinux                  ll_front_merge_fn
3        1.3e-06  vmlinux                  vmlinux                  load_elf_binary
3        1.3e-06  vmlinux                  vmlinux                  migration_thread
3        1.3e-06  vmlinux                  vmlinux                  neigh_periodic_timer
3        1.3e-06  vmlinux                  vmlinux                  number
3        1.3e-06  vmlinux                  vmlinux                  prepare_to_copy
3        1.3e-06  vmlinux                  vmlinux                  profile_handoff_task
3        1.3e-06  vmlinux                  vmlinux                  release_buffer_page
3        1.3e-06  vmlinux                  vmlinux                  setup_arg_pages
3        1.3e-06  vmlinux                  vmlinux                  sys_select
3        1.3e-06  vmlinux                  vmlinux                  sys_stat64
3        1.3e-06  vmlinux                  vmlinux                  unlock_kernel
3        1.3e-06  vmlinux                  vmlinux                  vga_pal_blank
3        1.3e-06  vmlinux                  vmlinux                  writeback_acquire
2        8.8e-07  binfmt_misc.ko           binfmt_misc              load_misc_binary
2        8.8e-07  ld-2.3.3.so              ld-2.3.3.so              ___xstat64
2        8.8e-07  ld-2.3.3.so              ld-2.3.3.so              __brk
2        8.8e-07  ld-2.3.3.so              ld-2.3.3.so              __libc_close
2        8.8e-07  ld-2.3.3.so              ld-2.3.3.so              _dl_catch_error
2        8.8e-07  ld-2.3.3.so              ld-2.3.3.so              _dl_fini
2        8.8e-07  ld-2.3.3.so              ld-2.3.3.so              _dl_sysdep_read_whole_file
2        8.8e-07  ld-2.3.3.so              ld-2.3.3.so              memcpy
2        8.8e-07  ld-2.3.3.so              ld-2.3.3.so              mempcpy
2        8.8e-07  libc-2.3.3.so            libc-2.3.3.so            _IO_check_libio
2        8.8e-07  libc-2.3.3.so            libc-2.3.3.so            _IO_default_setbuf
2        8.8e-07  libc-2.3.3.so            libc-2.3.3.so            _IO_str_init_static_internal
2        8.8e-07  libc-2.3.3.so            libc-2.3.3.so            __GI___register_atfork
2        8.8e-07  libc-2.3.3.so            libc-2.3.3.so            __GI___strtol_internal
2        8.8e-07  libc-2.3.3.so            libc-2.3.3.so            __GI___sysconf
2        8.8e-07  libc-2.3.3.so            libc-2.3.3.so            __GI_alarm
2        8.8e-07  libc-2.3.3.so            libc-2.3.3.so            __GI_close
2        8.8e-07  libc-2.3.3.so            libc-2.3.3.so            __GI_sprintf
2        8.8e-07  libc-2.3.3.so            libc-2.3.3.so            __mkdir
2        8.8e-07  libc-2.3.3.so            libc-2.3.3.so            __xstat32_conv
2        8.8e-07  libc-2.3.3.so            libc-2.3.3.so            _nl_postload_ctype
2        8.8e-07  libc-2.3.3.so            libc-2.3.3.so            analyze_tree
2        8.8e-07  libc-2.3.3.so            libc-2.3.3.so            init
2        8.8e-07  libc-2.3.3.so            libc-2.3.3.so            new_composite_name
2        8.8e-07  libc-2.3.3.so            libc-2.3.3.so            re_dfa_add_tree_node
2        8.8e-07  libpthread-0.10.so       libpthread-0.10.so       __pthread_attr_destroy
2        8.8e-07  vmlinux                  vmlinux                  .text.lock.mmap
2        8.8e-07  vmlinux                  vmlinux                  __d_rehash
2        8.8e-07  vmlinux                  vmlinux                  __journal_drop_transaction
2        8.8e-07  vmlinux                  vmlinux                  __journal_insert_checkpoint
2        8.8e-07  vmlinux                  vmlinux                  __journal_refile_buffer
2        8.8e-07  vmlinux                  vmlinux                  __migrate_task
2        8.8e-07  vmlinux                  vmlinux                  __page_cache_release
2        8.8e-07  vmlinux                  vmlinux                  __pdflush
2        8.8e-07  vmlinux                  vmlinux                  add_interrupt_randomness
2        8.8e-07  vmlinux                  vmlinux                  add_to_page_cache_lru
2        8.8e-07  vmlinux                  vmlinux                  as_antic_waitreq
2        8.8e-07  vmlinux                  vmlinux                  blk_congestion_wait
2        8.8e-07  vmlinux                  vmlinux                  can_share_swap_page
2        8.8e-07  vmlinux                  vmlinux                  cond_resched_lock
2        8.8e-07  vmlinux                  vmlinux                  copy_page_range
2        8.8e-07  vmlinux                  vmlinux                  copy_vma
2        8.8e-07  vmlinux                  vmlinux                  create_elf_tables
2        8.8e-07  vmlinux                  vmlinux                  dev_watchdog
2        8.8e-07  vmlinux                  vmlinux                  do_fcntl
2        8.8e-07  vmlinux                  vmlinux                  elf_map
2        8.8e-07  vmlinux                  vmlinux                  enqueue_hrtimer
2        8.8e-07  vmlinux                  vmlinux                  fasync_helper
2        8.8e-07  vmlinux                  vmlinux                  find_group_orlov
2        8.8e-07  vmlinux                  vmlinux                  flush_signal_handlers
2        8.8e-07  vmlinux                  vmlinux                  flush_sigqueue
2        8.8e-07  vmlinux                  vmlinux                  free_fdtable
2        8.8e-07  vmlinux                  vmlinux                  free_fdtable_rcu
2        8.8e-07  vmlinux                  vmlinux                  generic_delete_inode
2        8.8e-07  vmlinux                  vmlinux                  get_pipe_inode
2        8.8e-07  vmlinux                  vmlinux                  get_transaction
2        8.8e-07  vmlinux                  vmlinux                  get_writeback_state
2        8.8e-07  vmlinux                  vmlinux                  half_md4_transform
2        8.8e-07  vmlinux                  vmlinux                  hrtimer_start
2        8.8e-07  vmlinux                  vmlinux                  journal_next_log_block
2        8.8e-07  vmlinux                  vmlinux                  journal_write_commit_record
2        8.8e-07  vmlinux                  vmlinux                  kblockd_schedule_work
2        8.8e-07  vmlinux                  vmlinux                  move_ptes
2        8.8e-07  vmlinux                  vmlinux                  mpage_end_io_read
2        8.8e-07  vmlinux                  vmlinux                  nr_uninterruptible
2        8.8e-07  vmlinux                  vmlinux                  pgd_alloc
2        8.8e-07  vmlinux                  vmlinux                  pgd_free
2        8.8e-07  vmlinux                  vmlinux                  prio_tree_insert
2        8.8e-07  vmlinux                  vmlinux                  prio_tree_remove
2        8.8e-07  vmlinux                  vmlinux                  rt_may_expire
2        8.8e-07  vmlinux                  vmlinux                  schedule_hrtimer
2        8.8e-07  vmlinux                  vmlinux                  search_binary_handler
2        8.8e-07  vmlinux                  vmlinux                  search_exception_tables
2        8.8e-07  vmlinux                  vmlinux                  sock_poll
2        8.8e-07  vmlinux                  vmlinux                  sync_dirty_buffer
2        8.8e-07  vmlinux                  vmlinux                  sys_brk
2        8.8e-07  vmlinux                  vmlinux                  sys_fork
2        8.8e-07  vmlinux                  vmlinux                  sys_time
2        8.8e-07  vmlinux                  vmlinux                  tcp_poll
2        8.8e-07  vmlinux                  vmlinux                  unshare_files
2        8.8e-07  vmlinux                  vmlinux                  vma_prio_tree_remove
2        8.8e-07  vmlinux                  vmlinux                  vsnprintf
2        8.8e-07  vmlinux                  vmlinux                  write_chan
1        4.4e-07  cat                      cat                      (no symbols)
1        4.4e-07  ld-2.3.3.so              ld-2.3.3.so              __GI__dl_tls_setup
1        4.4e-07  ld-2.3.3.so              ld-2.3.3.so              __read
1        4.4e-07  ld-2.3.3.so              ld-2.3.3.so              _dl_check_map_versions
1        4.4e-07  ld-2.3.3.so              ld-2.3.3.so              _dl_init_paths
1        4.4e-07  ld-2.3.3.so              ld-2.3.3.so              _dl_new_object
1        4.4e-07  ld-2.3.3.so              ld-2.3.3.so              _dl_protect_relro
1        4.4e-07  ld-2.3.3.so              ld-2.3.3.so              _dl_setup_hash
1        4.4e-07  ld-2.3.3.so              ld-2.3.3.so              _dl_start
1        4.4e-07  ld-2.3.3.so              ld-2.3.3.so              access
1        4.4e-07  ld-2.3.3.so              ld-2.3.3.so              match_symbol
1        4.4e-07  ld-2.3.3.so              ld-2.3.3.so              open_verify
1        4.4e-07  ld-2.3.3.so              ld-2.3.3.so              rtld_lock_default_lock_recursive
1        4.4e-07  libc-2.3.3.so            libc-2.3.3.so            _IO_file_sync_internal
1        4.4e-07  libc-2.3.3.so            libc-2.3.3.so            _IO_flush_all_lockp
1        4.4e-07  libc-2.3.3.so            libc-2.3.3.so            _IO_sputbackc_internal
1        4.4e-07  libc-2.3.3.so            libc-2.3.3.so            _IO_vsnprintf
1        4.4e-07  libc-2.3.3.so            libc-2.3.3.so            _IO_vsscanf
1        4.4e-07  libc-2.3.3.so            libc-2.3.3.so            __GI__IO_iter_next
1        4.4e-07  libc-2.3.3.so            libc-2.3.3.so            __GI__IO_list_lock
1        4.4e-07  libc-2.3.3.so            libc-2.3.3.so            __GI__IO_list_unlock
1        4.4e-07  libc-2.3.3.so            libc-2.3.3.so            __GI___fork
1        4.4e-07  libc-2.3.3.so            libc-2.3.3.so            __GI___rawmemchr
1        4.4e-07  libc-2.3.3.so            libc-2.3.3.so            __GI___strxfrm_l
1        4.4e-07  libc-2.3.3.so            libc-2.3.3.so            __GI__dl_close
1        4.4e-07  libc-2.3.3.so            libc-2.3.3.so            __GI_fgets_unlocked
1        4.4e-07  libc-2.3.3.so            libc-2.3.3.so            __GI_fileno
1        4.4e-07  libc-2.3.3.so            libc-2.3.3.so            __GI_sigemptyset
1        4.4e-07  libc-2.3.3.so            libc-2.3.3.so            __GI_sigfillset
1        4.4e-07  libc-2.3.3.so            libc-2.3.3.so            __GI_stpcpy
1        4.4e-07  libc-2.3.3.so            libc-2.3.3.so            __GI_strncmp
1        4.4e-07  libc-2.3.3.so            libc-2.3.3.so            __bsd_signal
1        4.4e-07  libc-2.3.3.so            libc-2.3.3.so            __collidx_table_lookup
1        4.4e-07  libc-2.3.3.so            libc-2.3.3.so            __cxa_atexit_internal
1        4.4e-07  libc-2.3.3.so            libc-2.3.3.so            __gconv_load_cache
1        4.4e-07  libc-2.3.3.so            libc-2.3.3.so            __gconv_lookup_cache
1        4.4e-07  libc-2.3.3.so            libc-2.3.3.so            __hash_string
1        4.4e-07  libc-2.3.3.so            libc-2.3.3.so            __libc_fini
1        4.4e-07  libc-2.3.3.so            libc-2.3.3.so            __mmap64
1        4.4e-07  libc-2.3.3.so            libc-2.3.3.so            __mpn_lshift
1        4.4e-07  libc-2.3.3.so            libc-2.3.3.so            __mpn_mul
1        4.4e-07  libc-2.3.3.so            libc-2.3.3.so            __new_exitfn
1        4.4e-07  libc-2.3.3.so            libc-2.3.3.so            __new_getrlimit
1        4.4e-07  libc-2.3.3.so            libc-2.3.3.so            __random_r
1        4.4e-07  libc-2.3.3.so            libc-2.3.3.so            __sigsetjmp
1        4.4e-07  libc-2.3.3.so            libc-2.3.3.so            __snprintf
1        4.4e-07  libc-2.3.3.so            libc-2.3.3.so            __strchrnul
1        4.4e-07  libc-2.3.3.so            libc-2.3.3.so            __tcgetattr
1        4.4e-07  libc-2.3.3.so            libc-2.3.3.so            __unregister_atfork
1        4.4e-07  libc-2.3.3.so            libc-2.3.3.so            _getopt_internal
1        4.4e-07  libc-2.3.3.so            libc-2.3.3.so            _getopt_internal_r
1        4.4e-07  libc-2.3.3.so            libc-2.3.3.so            _nl_explode_name
1        4.4e-07  libc-2.3.3.so            libc-2.3.3.so            _nl_find_locale
1        4.4e-07  libc-2.3.3.so            libc-2.3.3.so            dl_open_worker
1        4.4e-07  libc-2.3.3.so            libc-2.3.3.so            ftok
1        4.4e-07  libc-2.3.3.so            libc-2.3.3.so            glob_in_dir
1        4.4e-07  libc-2.3.3.so            libc-2.3.3.so            mblen
1        4.4e-07  libc-2.3.3.so            libc-2.3.3.so            nss_parse_service_list
1        4.4e-07  libc-2.3.3.so            libc-2.3.3.so            parse_backtick
1        4.4e-07  libc-2.3.3.so            libc-2.3.3.so            peek_token
1        4.4e-07  libc-2.3.3.so            libc-2.3.3.so            printf
1        4.4e-07  libc-2.3.3.so            libc-2.3.3.so            re_compile_internal
1        4.4e-07  libc-2.3.3.so            libc-2.3.3.so            realloc_hook_ini
1        4.4e-07  libc-2.3.3.so            libc-2.3.3.so            rendezvous_request
1        4.4e-07  libc-2.3.3.so            libc-2.3.3.so            set_binding_values
1        4.4e-07  libc-2.3.3.so            libc-2.3.3.so            strsignal
1        4.4e-07  libc-2.3.3.so            libc-2.3.3.so            svcudp_recv
1        4.4e-07  libc-2.3.3.so            libc-2.3.3.so            xdr_int8_t
1        4.4e-07  libdl-2.3.3.so           libdl-2.3.3.so           __do_global_dtors_aux
1        4.4e-07  libpthread-0.10.so       libpthread-0.10.so       __fresetlockfiles
1        4.4e-07  libpthread-0.10.so       libpthread-0.10.so       __libc_open64
1        4.4e-07  libpthread-0.10.so       libpthread-0.10.so       __llseek
1        4.4e-07  libpthread-0.10.so       libpthread-0.10.so       __pthread_fork
1        4.4e-07  libpthread-0.10.so       libpthread-0.10.so       __pthread_initialize_manager
1        4.4e-07  libpthread-0.10.so       libpthread-0.10.so       __pthread_rwlock_wrlock
1        4.4e-07  libpthread-0.10.so       libpthread-0.10.so       pthread_onexit_process
1        4.4e-07  libselinux.so.1          libselinux.so.1          (no symbols)
1        4.4e-07  ls                       ls                       (no symbols)
1        4.4e-07  oprofile.ko              oprofile                 oprofilefs_str_to_user
1        4.4e-07  tiotest                  tiotest                  main
1        4.4e-07  vmlinux                  vmlinux                  __bio_add_page
1        4.4e-07  vmlinux                  vmlinux                  __const_udelay
1        4.4e-07  vmlinux                  vmlinux                  __d_path
1        4.4e-07  vmlinux                  vmlinux                  __get_zone_counts
1        4.4e-07  vmlinux                  vmlinux                  __jbd_kmalloc
1        4.4e-07  vmlinux                  vmlinux                  __mask_IO_APIC_irq
1        4.4e-07  vmlinux                  vmlinux                  __remove_hrtimer
1        4.4e-07  vmlinux                  vmlinux                  __user_walk_fd
1        4.4e-07  vmlinux                  vmlinux                  alloc_dcookie
1        4.4e-07  vmlinux                  vmlinux                  alloc_fd_array
1        4.4e-07  vmlinux                  vmlinux                  alloc_fdtable
1        4.4e-07  vmlinux                  vmlinux                  alloc_files
1        4.4e-07  vmlinux                  vmlinux                  alloc_page_interleave
1        4.4e-07  vmlinux                  vmlinux                  can_vma_merge_after
1        4.4e-07  vmlinux                  vmlinux                  cap_bprm_apply_creds
1        4.4e-07  vmlinux                  vmlinux                  cap_bprm_set_security
1        4.4e-07  vmlinux                  vmlinux                  clear_user
1        4.4e-07  vmlinux                  vmlinux                  convert_fxsr_to_user
1        4.4e-07  vmlinux                  vmlinux                  count
1        4.4e-07  vmlinux                  vmlinux                  count_active_tasks
1        4.4e-07  vmlinux                  vmlinux                  current_kernel_time
1        4.4e-07  vmlinux                  vmlinux                  d_free
1        4.4e-07  vmlinux                  vmlinux                  dcache_readdir
1        4.4e-07  vmlinux                  vmlinux                  dev_queue_xmit
1        4.4e-07  vmlinux                  vmlinux                  do_execve
1        4.4e-07  vmlinux                  vmlinux                  do_page_cache_readahead
1        4.4e-07  vmlinux                  vmlinux                  do_setitimer
1        4.4e-07  vmlinux                  vmlinux                  do_sys_ftruncate
1        4.4e-07  vmlinux                  vmlinux                  e1000_read_phy_reg
1        4.4e-07  vmlinux                  vmlinux                  e1000_watchdog
1        4.4e-07  vmlinux                  vmlinux                  expand_fdtable
1        4.4e-07  vmlinux                  vmlinux                  ext3_bmap
1        4.4e-07  vmlinux                  vmlinux                  ext3_destroy_inode
1        4.4e-07  vmlinux                  vmlinux                  ext3_ioctl
1        4.4e-07  vmlinux                  vmlinux                  ext3_write_super
1        4.4e-07  vmlinux                  vmlinux                  ext3_xattr_delete_inode
1        4.4e-07  vmlinux                  vmlinux                  extract_buf
1        4.4e-07  vmlinux                  vmlinux                  extract_entropy
1        4.4e-07  vmlinux                  vmlinux                  flush_thread
1        4.4e-07  vmlinux                  vmlinux                  free_as_io_context
1        4.4e-07  vmlinux                  vmlinux                  free_one_page
1        4.4e-07  vmlinux                  vmlinux                  generic_block_bmap
1        4.4e-07  vmlinux                  vmlinux                  generic_drop_inode
1        4.4e-07  vmlinux                  vmlinux                  generic_forget_inode
1        4.4e-07  vmlinux                  vmlinux                  get_dcookie
1        4.4e-07  vmlinux                  vmlinux                  get_index
1        4.4e-07  vmlinux                  vmlinux                  get_locked_pte
1        4.4e-07  vmlinux                  vmlinux                  get_vmalloc_info
1        4.4e-07  vmlinux                  vmlinux                  hrtimer_nanosleep
1        4.4e-07  vmlinux                  vmlinux                  icmp_send
1        4.4e-07  vmlinux                  vmlinux                  init_new_context
1        4.4e-07  vmlinux                  vmlinux                  inode_has_buffers
1        4.4e-07  vmlinux                  vmlinux                  ip_dev_find
1        4.4e-07  vmlinux                  vmlinux                  ip_output
1        4.4e-07  vmlinux                  vmlinux                  jbd_unexpected_dirty_buffer
1        4.4e-07  vmlinux                  vmlinux                  kfree_skbmem
1        4.4e-07  vmlinux                  vmlinux                  ksoftirqd
1        4.4e-07  vmlinux                  vmlinux                  local_bh_enable
1        4.4e-07  vmlinux                  vmlinux                  mask_IO_APIC_irq
1        4.4e-07  vmlinux                  vmlinux                  mm_alloc
1        4.4e-07  vmlinux                  vmlinux                  msync_pte_range
1        4.4e-07  vmlinux                  vmlinux                  neigh_timer_handler
1        4.4e-07  vmlinux                  vmlinux                  normal_poll
1        4.4e-07  vmlinux                  vmlinux                  old_mmap
1        4.4e-07  vmlinux                  vmlinux                  open_exec
1        4.4e-07  vmlinux                  vmlinux                  opost_block
1        4.4e-07  vmlinux                  vmlinux                  out_of_line_wait_on_bit
1        4.4e-07  vmlinux                  vmlinux                  parse_table
1        4.4e-07  vmlinux                  vmlinux                  peer_check_expire
1        4.4e-07  vmlinux                  vmlinux                  pfifo_fast_enqueue
1        4.4e-07  vmlinux                  vmlinux                  prepare_binprm
1        4.4e-07  vmlinux                  vmlinux                  prio_tree_replace
1        4.4e-07  vmlinux                  vmlinux                  proc_exe_link
1        4.4e-07  vmlinux                  vmlinux                  proc_lookup
1        4.4e-07  vmlinux                  vmlinux                  read_chan
1        4.4e-07  vmlinux                  vmlinux                  read_pages
1        4.4e-07  vmlinux                  vmlinux                  rm_from_queue_full
1        4.4e-07  vmlinux                  vmlinux                  schedule_timeout_interruptible
1        4.4e-07  vmlinux                  vmlinux                  select_parent
1        4.4e-07  vmlinux                  vmlinux                  seq_escape
1        4.4e-07  vmlinux                  vmlinux                  seq_open
1        4.4e-07  vmlinux                  vmlinux                  seq_puts
1        4.4e-07  vmlinux                  vmlinux                  set_ioapic_affinity_irq
1        4.4e-07  vmlinux                  vmlinux                  set_page_dirty
1        4.4e-07  vmlinux                  vmlinux                  show_vfsmnt
1        4.4e-07  vmlinux                  vmlinux                  simple_read_from_buffer
1        4.4e-07  vmlinux                  vmlinux                  sync_page
1        4.4e-07  vmlinux                  vmlinux                  sys_access
1        4.4e-07  vmlinux                  vmlinux                  sys_fcntl64
1        4.4e-07  vmlinux                  vmlinux                  sys_getgroups
1        4.4e-07  vmlinux                  vmlinux                  sys_mkdir
1        4.4e-07  vmlinux                  vmlinux                  sys_mkdirat
1        4.4e-07  vmlinux                  vmlinux                  sys_newuname
1        4.4e-07  vmlinux                  vmlinux                  tcp_rcv_established
1        4.4e-07  vmlinux                  vmlinux                  tcp_select_window
1        4.4e-07  vmlinux                  vmlinux                  tcp_sendmsg
1        4.4e-07  vmlinux                  vmlinux                  tcp_v4_do_rcv
1        4.4e-07  vmlinux                  vmlinux                  tcp_v4_send_check
1        4.4e-07  vmlinux                  vmlinux                  tty_ioctl
1        4.4e-07  vmlinux                  vmlinux                  tty_ldisc_try
1        4.4e-07  vmlinux                  vmlinux                  use_table
1        4.4e-07  vmlinux                  vmlinux                  vfs_stat_fd
1        4.4e-07  vmlinux                  vmlinux                  vma_prio_tree_insert
1        4.4e-07  vmlinux                  vmlinux                  wait_for_completion
1        4.4e-07  vmlinux                  vmlinux                  write_one_revoke_record
1        4.4e-07  vmlinux                  vmlinux                  writeback_release

[-- Attachment #2: Ceci est une partie de message numériquement signée. --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: [Ext2-devel] Re: [RFC][PATCH 0/2]Extend ext3 filesystem limit from 8TB to 16TB

[Laurent Vivier]

[-- Attachment #1: Type: text/plain, Size: 1905 bytes --]

Le jeu 20/04/2006 à 16:39, Laurent Vivier a écrit : 
> The functions added by my patch are following and as they are atomic
> (one machine instruction) they are not measurable and don't appears in
> oprofile.
> 
> atomic_long_add
> atomic_long_read
> atomic_long_set
> atomic_long_inc

I think, as these commands are atomic/inlined we should measure the time
of the functions modified by the patches.

The functions modified by the patch are:

ext3_free_blocks_sb
ext3_has_free_blocks
ext3_new_block
ext3_put_super
ext3_fill_super
ext3_fill_super
ext3_free_inode
find_group_dir
find_group_orlov
ext3_new_inode
ext3_group_add

If we make a "grep" on tiobench oprofile.out, we have:

atomic_t:

26919     0.0119  vmlinux                  vmlinux                  ext3_new_block
2195     9.7e-04  vmlinux                  vmlinux                  ext3_free_blocks_sb
1192     5.2e-04  vmlinux                  vmlinux                  ext3_has_free_blocks
189      8.3e-05  vmlinux                  vmlinux                  ext3_new_inode
40       1.8e-05  vmlinux                  vmlinux                  ext3_free_inode
2        8.8e-07  vmlinux                  vmlinux                  find_group_orlov

percpu_counter:

16290     0.0067  vmlinux                  vmlinux                  ext3_new_block
2075     8.5e-04  vmlinux                  vmlinux                  ext3_free_blocks_sb
428      1.8e-04  vmlinux                  vmlinux                  ext3_has_free_blocks
162      6.7e-05  vmlinux                  vmlinux                  ext3_new_inode
25       1.0e-05  vmlinux                  vmlinux                  ext3_free_inode

As we can using atomic_long_t is slower than percpu_counter so ...
forget my patch.

Regards,
Laurent 

-- 
Laurent Vivier
Bull, Architect of an Open World (TM)
http://www.bullopensource.org/ext4

[-- Attachment #2: Ceci est une partie de message numériquement signée. --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

[PATCH 0/2]Define ext3 in-kernel filesystem block types and extend ext3 filesystem limit from 8TB to 16TB

[Mingming Cao]

Some of the in-kernel ext3 block variable type are treated as signed 4 bytes
int type, thus limited ext3 filesystem to 8TB (4kblock size based). While
trying to fix them, it seems quite confusing in the ext3 code where some
blocks are filesystem-wide blocks, some are group relative offsets that need
to be signed value (as -1 has special meaning). So it seem saner to define two
types of physical blocks: one is filesystem wide blocks, another is
group-relative blocks.  The following patches clarify these two types of
blocks in the ext3 code, and fix the type bugs which limit current 32 bit ext3
filesystem limit to 8TB.

With this series of patches and the percpu counter data type changes in the mm
tree, we are able to extend exts filesystem limit to 16TB.

This work is also a pre-request for the recent >32 bit ext3 work, and makes
the kernel to able to address 48 bit ext3 block a lot easier: Simply
redefine ext3_fsblk_t from unsigned long to sector_t and redefine the format
string for ext3 filesystem block corresponding.

Two RFC with a series patches have been posted to ext2-devel list and have
been reviewed and discussed:
http://marc.theaimsgroup.com/?l=ext2-devel&m=114722190816690&w=2

http://marc.theaimsgroup.com/?l=ext2-devel&m=114784919525942&w=2

The following patches are updated and intergreated patches from two RFC posted
previous:

[Patch 1]ext3_fsblk_t, ext3_grpblk_t and type fixes.
	defines ext3_fsblk_t and ext3_grpblk_t, and the printk format string
	for filesystem wide blocks.
	
	This patch classifies all block group relative blocks, and
	ext3_fsblk_t blocks occurs in the same function where used to
	be confusing before. Also include kernel bug fixes for filesystem
	wide in-kernel block variables. There are some fileystem wide
	blocks are treated as int/unsigned int type in the kernel currently,
	especially in ext3 block allocation and reservation code. 
	This patch fixed those bugs by converting those variables to
	ext3_fsblk_t(unsigned long) type.

[Patch 2] Convert the ext3 in-kernel filesystem blocks to ext3_fsblk_t.
	Convert the rest of all unsigned long type in-kernel filesystem
	blocks to ext3_fsblk_t, and replace the printk format string
	respondingly.

Patches are tested on both 32 bit machine and 64 bit machine, <8TB ext3 and
>8TB ext3 filesystem(with the latest to be released e2fsprogs-1.39). Tests
 includes overnight fsx, tiobench, dbench and fsstress.

Patches are appliable
to 2.6.17-rc4-mm3, also applied to 2.6.17-rc4 kernel(need to apply percpu
counter changes to support >31 bit ext3 free blocks counters. 2.6.17-rc4
version of percpu cpu counter data type change patch could be found at:

http://ext2.sourceforge.net/48bitext3/patches/patches-2.6.17-rc4-05242006/percpu_counter_longlong.patch


Signed-Off-By: Mingming Cao <cmm@us.ibm.com>

Re: [PATCH 0/2]Define ext3 in-kernel filesystem block types and extend ext3 filesystem limit from 8TB to 16TB

[Andrew Morton]

Mingming Cao <cmm@us.ibm.com> wrote:
>
> Some of the in-kernel ext3 block variable type are treated as signed 4 bytes
>  int type, thus limited ext3 filesystem to 8TB (4kblock size based). While
>  trying to fix them, it seems quite confusing in the ext3 code where some
>  blocks are filesystem-wide blocks, some are group relative offsets that need
>  to be signed value (as -1 has special meaning). So it seem saner to define two
>  types of physical blocks: one is filesystem wide blocks, another is
>  group-relative blocks.  The following patches clarify these two types of
>  blocks in the ext3 code, and fix the type bugs which limit current 32 bit ext3
>  filesystem limit to 8TB.
> 
>  With this series of patches and the percpu counter data type changes in the mm
>  tree, we are able to extend exts filesystem limit to 16TB.

Did you look at the `gcc -W' output before and after these patches are
applied?  That would have found the bug which the previous version
of these patches introduced.

We already get a pile of `warning: comparison between signed and unsigned'
warnings which should be checked, too..

Re: [PATCH 0/2]Define ext3 in-kernel filesystem block types and extend ext3 filesystem limit from 8TB to 16TB

[Mingming Cao]

On Fri, 2006-05-26 at 11:08 -0700, Andrew Morton wrote:
> Mingming Cao <cmm@us.ibm.com> wrote:
> >
> > Some of the in-kernel ext3 block variable type are treated as signed 4 bytes
> >  int type, thus limited ext3 filesystem to 8TB (4kblock size based). While
> >  trying to fix them, it seems quite confusing in the ext3 code where some
> >  blocks are filesystem-wide blocks, some are group relative offsets that need
> >  to be signed value (as -1 has special meaning). So it seem saner to define two
> >  types of physical blocks: one is filesystem wide blocks, another is
> >  group-relative blocks.  The following patches clarify these two types of
> >  blocks in the ext3 code, and fix the type bugs which limit current 32 bit ext3
> >  filesystem limit to 8TB.
> > 
> >  With this series of patches and the percpu counter data type changes in the mm
> >  tree, we are able to extend exts filesystem limit to 16TB.
> 
> Did you look at the `gcc -W' output before and after these patches are
> applied?  That would have found the bug which the previous version
> of these patches introduced.
> 
Sorry for the delay, was out for the past holiday.

Yes, I did used gcc -Wall -Wextra.  Pretty careful about it this time. 

> We already get a pile of `warning: comparison between signed and unsigned'
> warnings which should be checked, too..
> 

Yes, indeed.

Mingming