* RE: ext3 performance bottleneck as the number of spindles gets la rge
@ 2002-06-20 15:26 Griffiths, Richard A
2002-06-20 20:18 ` ext3 performance bottleneck as the number of spindles gets large Andrew Morton
0 siblings, 1 reply; 16+ messages in thread
From: Griffiths, Richard A @ 2002-06-20 15:26 UTC (permalink / raw)
To: 'Jens Axboe', Andrew Morton
Cc: mgross, Linux Kernel Mailing List, lse-tech, Griffiths, Richard A
We ran without highmem enabled so the Kernel only saw 1GB of memory.
Richard
-----Original Message-----
From: Jens Axboe [mailto:axboe@suse.de]
Sent: Wednesday, June 19, 2002 11:05 PM
To: Andrew Morton
Cc: mgross@unix-os.sc.intel.com; Linux Kernel Mailing List;
lse-tech@lists.sourceforge.net; richard.a.griffiths@intel.com
Subject: Re: ext3 performance bottleneck as the number of spindles gets
large
On Wed, Jun 19 2002, Andrew Morton wrote:
> mgross wrote:
> >
> > We've been doing some throughput comparisons and benchmarks of block I/O
> > throughput for 8KB writes as the number of SCSI addapters and drives per
> > adapter is increased.
> >
> > The Linux platform is a dual processor 1.2GHz PIII, 2Gig or RAM, 2U box.
> > Similar results have been seen with both 2.4.16 and 2.4.18 base kernel,
as
> > well as one of those patched up O(1) 2.4.18 kernels out there.
>
> umm. Are you not using block-highmem? That is a must-have.
>
>
http://www.kernel.org/pub/linux/kernel/people/andrea/kernels/v2.4/2.4.19pre9
aa2/00_block-highmem-all-18b-12.gz
please use
http://www.kernel.org/pub/linux/kernel/people/axboe/patches/v2.4/2.4.19-pre1
0/block-highmem-all-19.bz2
--
Jens Axboe
^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: ext3 performance bottleneck as the number of spindles gets large
2002-06-20 15:26 ext3 performance bottleneck as the number of spindles gets la rge Griffiths, Richard A
@ 2002-06-20 20:18 ` Andrew Morton
2002-06-20 18:08 ` mgross
0 siblings, 1 reply; 16+ messages in thread
From: Andrew Morton @ 2002-06-20 20:18 UTC (permalink / raw)
To: Griffiths, Richard A
Cc: 'Jens Axboe', mgross, Linux Kernel Mailing List, lse-tech
"Griffiths, Richard A" wrote:
>
> We ran without highmem enabled so the Kernel only saw 1GB of memory.
>
Yup. I take it back - high ext3 lock contention happens on 2.5
as well, which has block-highmem. With heavy write traffic onto
six disks, two controllers, six filesystems, four CPUs the machine
spends about 40% of the time spinning on locks in fs/ext3/inode.c
You're un dual CPU, so the contention is less.
Not very nice. But given that the longest spin time was some
tens of milliseconds, with the average much lower, it shouldn't
affect overall I/O throughput.
Possibly something else is happening. Have you tested ext2?
^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: ext3 performance bottleneck as the number of spindles gets large
2002-06-20 20:18 ` ext3 performance bottleneck as the number of spindles gets large Andrew Morton
@ 2002-06-20 18:08 ` mgross
2002-06-20 21:25 ` Andrew Morton
0 siblings, 1 reply; 16+ messages in thread
From: mgross @ 2002-06-20 18:08 UTC (permalink / raw)
To: Andrew Morton, Griffiths, Richard A
Cc: 'Jens Axboe', Linux Kernel Mailing List, lse-tech
On Thursday 20 June 2002 04:18 pm, Andrew Morton wrote:
> Yup. I take it back - high ext3 lock contention happens on 2.5
> as well, which has block-highmem. With heavy write traffic onto
> six disks, two controllers, six filesystems, four CPUs the machine
> spends about 40% of the time spinning on locks in fs/ext3/inode.c
> You're un dual CPU, so the contention is less.
>
> Not very nice. But given that the longest spin time was some
> tens of milliseconds, with the average much lower, it shouldn't
> affect overall I/O throughput.
How could losing 40% of your CPU's to spin locks NOT spank your throughtput?
Can you copy your lockmeter data from its kernel_flag section? Id like to
see it.
>
> Possibly something else is happening. Have you tested ext2?
No. We're attempting to see if we can scale to large numbers of spindles
with EXT3 at the moment. Perhaps we can effect positive changes to ext3
before giving up on it and moving to another Journaled FS.
--mgross
^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: ext3 performance bottleneck as the number of spindles gets large
2002-06-20 18:08 ` mgross
@ 2002-06-20 21:25 ` Andrew Morton
0 siblings, 0 replies; 16+ messages in thread
From: Andrew Morton @ 2002-06-20 21:25 UTC (permalink / raw)
To: mgross
Cc: Griffiths, Richard A, 'Jens Axboe',
Linux Kernel Mailing List, lse-tech
mgross wrote:
>
> On Thursday 20 June 2002 04:18 pm, Andrew Morton wrote:
> > Yup. I take it back - high ext3 lock contention happens on 2.5
> > as well, which has block-highmem. With heavy write traffic onto
> > six disks, two controllers, six filesystems, four CPUs the machine
> > spends about 40% of the time spinning on locks in fs/ext3/inode.c
> > You're un dual CPU, so the contention is less.
> >
> > Not very nice. But given that the longest spin time was some
> > tens of milliseconds, with the average much lower, it shouldn't
> > affect overall I/O throughput.
>
> How could losing 40% of your CPU's to spin locks NOT spank your throughtput?
The limiting factor is usually disk bandwidth, seek latency, rotational
latency. That's why I want to know your bandwidth.
> Can you copy your lockmeter data from its kernel_flag section? Id like to
> see it.
I don't find lockmeter very useful. Here's oprofile output for 2.5.23:
c013ec08 873 1.07487 rmqueue
c018a8e4 950 1.16968 do_get_write_access
c013b00c 969 1.19307 kmem_cache_alloc_batch
c018165c 1120 1.37899 ext3_writepage
c0193120 1457 1.79392 journal_add_journal_head
c0180e30 1458 1.79515 ext3_prepare_write
c0136948 6546 8.05969 generic_file_write
c01838ac 42608 52.4606 .text.lock.inode
So I lost two CPUs on the BKL in fs/ext3/inode.c. The remaining
two should be enough to saturate all but the most heroic disk
subsystems.
A couple of possibilities come to mind:
1: Processes which should be submitting I/O against disk "A" are
instead spending tons of time asleep in the page allocator waiting
for I/O to complete against disk "B".
2: ext3 is just too slow for the rate of data which you're trying to
push at it. This exhibits as lock contention, but the root cause
is the cost of things like ext3_mark_inode_dirty(). And *that*
is something we can fix - can shave 75% off the cost of that.
Need more data...
> >
> > Possibly something else is happening. Have you tested ext2?
>
> No. We're attempting to see if we can scale to large numbers of spindles
> with EXT3 at the moment. Perhaps we can effect positive changes to ext3
> before giving up on it and moving to another Journaled FS.
Have you tried *any* other fs?
-
^ permalink raw reply [flat|nested] 16+ messages in thread
* RE: ext3 performance bottleneck as the number of spindles gets la rge
@ 2002-06-20 21:50 Griffiths, Richard A
2002-06-21 7:58 ` ext3 performance bottleneck as the number of spindles gets large Andrew Morton
2002-06-23 4:02 ` Christopher E. Brown
0 siblings, 2 replies; 16+ messages in thread
From: Griffiths, Richard A @ 2002-06-20 21:50 UTC (permalink / raw)
To: 'Andrew Morton', mgross
Cc: Griffiths, Richard A, 'Jens Axboe',
Linux Kernel Mailing List, lse-tech
I should have mentioned the throughput we saw on 4 adapters 6 drives was
126KB/s. The max theoretical bus bandwith is 640MB/s.
-----Original Message-----
From: Andrew Morton [mailto:akpm@zip.com.au]
Sent: Thursday, June 20, 2002 2:26 PM
To: mgross@unix-os.sc.intel.com
Cc: Griffiths, Richard A; 'Jens Axboe'; Linux Kernel Mailing List;
lse-tech@lists.sourceforge.net
Subject: Re: ext3 performance bottleneck as the number of spindles gets
large
mgross wrote:
>
> On Thursday 20 June 2002 04:18 pm, Andrew Morton wrote:
> > Yup. I take it back - high ext3 lock contention happens on 2.5
> > as well, which has block-highmem. With heavy write traffic onto
> > six disks, two controllers, six filesystems, four CPUs the machine
> > spends about 40% of the time spinning on locks in fs/ext3/inode.c
> > You're un dual CPU, so the contention is less.
> >
> > Not very nice. But given that the longest spin time was some
> > tens of milliseconds, with the average much lower, it shouldn't
> > affect overall I/O throughput.
>
> How could losing 40% of your CPU's to spin locks NOT spank your
throughtput?
The limiting factor is usually disk bandwidth, seek latency, rotational
latency. That's why I want to know your bandwidth.
> Can you copy your lockmeter data from its kernel_flag section? Id like to
> see it.
I don't find lockmeter very useful. Here's oprofile output for 2.5.23:
c013ec08 873 1.07487 rmqueue
c018a8e4 950 1.16968 do_get_write_access
c013b00c 969 1.19307 kmem_cache_alloc_batch
c018165c 1120 1.37899 ext3_writepage
c0193120 1457 1.79392 journal_add_journal_head
c0180e30 1458 1.79515 ext3_prepare_write
c0136948 6546 8.05969 generic_file_write
c01838ac 42608 52.4606 .text.lock.inode
So I lost two CPUs on the BKL in fs/ext3/inode.c. The remaining
two should be enough to saturate all but the most heroic disk
subsystems.
A couple of possibilities come to mind:
1: Processes which should be submitting I/O against disk "A" are
instead spending tons of time asleep in the page allocator waiting
for I/O to complete against disk "B".
2: ext3 is just too slow for the rate of data which you're trying to
push at it. This exhibits as lock contention, but the root cause
is the cost of things like ext3_mark_inode_dirty(). And *that*
is something we can fix - can shave 75% off the cost of that.
Need more data...
> >
> > Possibly something else is happening. Have you tested ext2?
>
> No. We're attempting to see if we can scale to large numbers of spindles
> with EXT3 at the moment. Perhaps we can effect positive changes to ext3
> before giving up on it and moving to another Journaled FS.
Have you tried *any* other fs?
-
^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: ext3 performance bottleneck as the number of spindles gets large
2002-06-20 21:50 ext3 performance bottleneck as the number of spindles gets la rge Griffiths, Richard A
@ 2002-06-21 7:58 ` Andrew Morton
2002-06-21 18:46 ` mgross
2002-06-23 4:02 ` Christopher E. Brown
1 sibling, 1 reply; 16+ messages in thread
From: Andrew Morton @ 2002-06-21 7:58 UTC (permalink / raw)
To: Griffiths, Richard A
Cc: mgross, 'Jens Axboe', Linux Kernel Mailing List, lse-tech
"Griffiths, Richard A" wrote:
>
> I should have mentioned the throughput we saw on 4 adapters 6 drives was
> 126KB/s. The max theoretical bus bandwith is 640MB/s.
I hope that was 128MB/s?
Please try the below patch (againt 2.4.19-pre10). It halves the lock
contention, and it does that by making the fs twice as efficient, so
that's a bonus.
I wouldn't be surprised if it made no difference. I'm not seeing
much difference between ext2 and ext3 here.
If you have time, please test ext2 and/or reiserfs and/or ext3
in writeback mode.
And please tell us some more details regarding the performance bottleneck.
I assume that you mean that the IO rate per disk slows as more
disks are added to an adapter? Or does the total throughput through
the adapter fall as more disks are added?
Thanks.
--- 2.4.19-pre10/fs/ext3/inode.c~ext3-speedup-1 Fri Jun 21 00:28:59 2002
+++ 2.4.19-pre10-akpm/fs/ext3/inode.c Fri Jun 21 00:28:59 2002
@@ -1016,21 +1016,20 @@ static int ext3_prepare_write(struct fil
int ret, needed_blocks = ext3_writepage_trans_blocks(inode);
handle_t *handle;
- lock_kernel();
handle = ext3_journal_start(inode, needed_blocks);
if (IS_ERR(handle)) {
ret = PTR_ERR(handle);
goto out;
}
- unlock_kernel();
ret = block_prepare_write(page, from, to, ext3_get_block);
- lock_kernel();
if (ret != 0)
goto prepare_write_failed;
if (ext3_should_journal_data(inode)) {
+ lock_kernel();
ret = walk_page_buffers(handle, page->buffers,
from, to, NULL, do_journal_get_write_access);
+ unlock_kernel();
if (ret) {
/*
* We're going to fail this prepare_write(),
@@ -1043,10 +1042,12 @@ static int ext3_prepare_write(struct fil
}
}
prepare_write_failed:
- if (ret)
+ if (ret) {
+ lock_kernel();
ext3_journal_stop(handle, inode);
+ unlock_kernel();
+ }
out:
- unlock_kernel();
return ret;
}
@@ -1094,7 +1095,6 @@ static int ext3_commit_write(struct file
struct inode *inode = page->mapping->host;
int ret = 0, ret2;
- lock_kernel();
if (ext3_should_journal_data(inode)) {
/*
* Here we duplicate the generic_commit_write() functionality
@@ -1102,22 +1102,43 @@ static int ext3_commit_write(struct file
int partial = 0;
loff_t pos = ((loff_t)page->index << PAGE_CACHE_SHIFT) + to;
+ lock_kernel();
ret = walk_page_buffers(handle, page->buffers,
from, to, &partial, commit_write_fn);
+ unlock_kernel();
if (!partial)
SetPageUptodate(page);
kunmap(page);
if (pos > inode->i_size)
inode->i_size = pos;
EXT3_I(inode)->i_state |= EXT3_STATE_JDATA;
+ if (inode->i_size > inode->u.ext3_i.i_disksize) {
+ inode->u.ext3_i.i_disksize = inode->i_size;
+ lock_kernel();
+ ret2 = ext3_mark_inode_dirty(handle, inode);
+ unlock_kernel();
+ if (!ret)
+ ret = ret2;
+ }
} else {
if (ext3_should_order_data(inode)) {
+ lock_kernel();
ret = walk_page_buffers(handle, page->buffers,
from, to, NULL, journal_dirty_sync_data);
+ unlock_kernel();
}
/* Be careful here if generic_commit_write becomes a
* required invocation after block_prepare_write. */
if (ret == 0) {
+ /*
+ * generic_commit_write() will run mark_inode_dirty()
+ * if i_size changes. So let's piggyback the
+ * i_disksize mark_inode_dirty into that.
+ */
+ loff_t new_i_size =
+ ((loff_t)page->index << PAGE_CACHE_SHIFT) + to;
+ if (new_i_size > EXT3_I(inode)->i_disksize)
+ EXT3_I(inode)->i_disksize = new_i_size;
ret = generic_commit_write(file, page, from, to);
} else {
/*
@@ -1129,12 +1150,7 @@ static int ext3_commit_write(struct file
kunmap(page);
}
}
- if (inode->i_size > inode->u.ext3_i.i_disksize) {
- inode->u.ext3_i.i_disksize = inode->i_size;
- ret2 = ext3_mark_inode_dirty(handle, inode);
- if (!ret)
- ret = ret2;
- }
+ lock_kernel();
ret2 = ext3_journal_stop(handle, inode);
unlock_kernel();
if (!ret)
@@ -2165,9 +2181,11 @@ bad_inode:
/*
* Post the struct inode info into an on-disk inode location in the
* buffer-cache. This gobbles the caller's reference to the
- * buffer_head in the inode location struct.
+ * buffer_head in the inode location struct.
+ *
+ * On entry, the caller *must* have journal write access to the inode's
+ * backing block, at iloc->bh.
*/
-
static int ext3_do_update_inode(handle_t *handle,
struct inode *inode,
struct ext3_iloc *iloc)
@@ -2176,12 +2194,6 @@ static int ext3_do_update_inode(handle_t
struct buffer_head *bh = iloc->bh;
int err = 0, rc, block;
- if (handle) {
- BUFFER_TRACE(bh, "get_write_access");
- err = ext3_journal_get_write_access(handle, bh);
- if (err)
- goto out_brelse;
- }
raw_inode->i_mode = cpu_to_le16(inode->i_mode);
if(!(test_opt(inode->i_sb, NO_UID32))) {
raw_inode->i_uid_low = cpu_to_le16(low_16_bits(inode->i_uid));
--- 2.4.19-pre10/mm/filemap.c~ext3-speedup-1 Fri Jun 21 00:28:59 2002
+++ 2.4.19-pre10-akpm/mm/filemap.c Fri Jun 21 00:28:59 2002
@@ -2924,6 +2924,7 @@ generic_file_write(struct file *file,con
long status = 0;
int err;
unsigned bytes;
+ time_t time_now;
if ((ssize_t) count < 0)
return -EINVAL;
@@ -3026,8 +3027,12 @@ generic_file_write(struct file *file,con
goto out;
remove_suid(inode);
- inode->i_ctime = inode->i_mtime = CURRENT_TIME;
- mark_inode_dirty_sync(inode);
+ time_now = CURRENT_TIME;
+ if (inode->i_ctime != time_now || inode->i_mtime != time_now) {
+ inode->i_ctime = time_now;
+ inode->i_mtime = time_now;
+ mark_inode_dirty_sync(inode);
+ }
if (file->f_flags & O_DIRECT)
goto o_direct;
--- 2.4.19-pre10/fs/jbd/transaction.c~ext3-speedup-1 Fri Jun 21 00:28:59 2002
+++ 2.4.19-pre10-akpm/fs/jbd/transaction.c Fri Jun 21 00:28:59 2002
@@ -237,7 +237,9 @@ handle_t *journal_start(journal_t *journ
handle->h_ref = 1;
current->journal_info = handle;
+ lock_kernel();
err = start_this_handle(journal, handle);
+ unlock_kernel();
if (err < 0) {
kfree(handle);
current->journal_info = NULL;
@@ -1388,8 +1390,10 @@ int journal_stop(handle_t *handle)
transaction->t_outstanding_credits -= handle->h_buffer_credits;
transaction->t_updates--;
if (!transaction->t_updates) {
- wake_up(&journal->j_wait_updates);
- if (journal->j_barrier_count)
+ if (waitqueue_active(&journal->j_wait_updates))
+ wake_up(&journal->j_wait_updates);
+ if (journal->j_barrier_count &&
+ waitqueue_active(&journal->j_wait_transaction_locked))
wake_up(&journal->j_wait_transaction_locked);
}
-
^ permalink raw reply [flat|nested] 16+ messages in thread* Re: ext3 performance bottleneck as the number of spindles gets large
2002-06-21 7:58 ` ext3 performance bottleneck as the number of spindles gets large Andrew Morton
@ 2002-06-21 18:46 ` mgross
2002-06-21 19:26 ` Chris Mason
2002-06-21 19:56 ` Andrew Morton
0 siblings, 2 replies; 16+ messages in thread
From: mgross @ 2002-06-21 18:46 UTC (permalink / raw)
To: Andrew Morton
Cc: Griffiths, Richard A, 'Jens Axboe',
Linux Kernel Mailing List, lse-tech
Andrew Morton wrote:
>"Griffiths, Richard A" wrote:
>
>>I should have mentioned the throughput we saw on 4 adapters 6 drives was
>>126KB/s. The max theoretical bus bandwith is 640MB/s.
>>
>
>I hope that was 128MB/s?
>
Yes that was MB/s, the data was taken in KB a set of 3 zeros where missing.
>
>
>Please try the below patch (againt 2.4.19-pre10). It halves the lock
>contention, and it does that by making the fs twice as efficient, so
>that's a bonus.
>
We'll give it a try. I'm on travel right now so it may be a few days if
Richard doesn't get to before I get back.
>
>
>I wouldn't be surprised if it made no difference. I'm not seeing
>much difference between ext2 and ext3 here.
>
>If you have time, please test ext2 and/or reiserfs and/or ext3
>in writeback mode.
>
Soon after we finish beating the ext3 file system up I'll take a swing
at some other file systems.
>
>And please tell us some more details regarding the performance bottleneck.
>I assume that you mean that the IO rate per disk slows as more
>disks are added to an adapter? Or does the total throughput through
>the adapter fall as more disks are added?
>
No, the IO block write throughput for the system goes down as drives are
added under this work load. We measure the system throughput not the
per drive throughput, but one could infer the per drive throughput by
dividing.
Running bonnie++ on with 300MB files doing 8Kb sequential writes we get
the following system wide throughput as a function of the number of
drives attached and by number of addapters.
One addapter
1 drive per addapter 127,702KB/Sec
2 drives per addapter 93,283 KB/Sec
6 drives per addapter 85,626 KB/Sec
2 addapters
1 drive per addapter 92,095 KB/Sec
2 drives per addapter 110,956 KB/Sec
6 drives per addapter 106,883 KB/Sec
4 addapters
1 drive per addapter 121,125 KB/Sec
2 drives per addapter 117,575 KB/Sec
6 drives per addapter 116,570 KB/Sec
Not too pritty.
--mgross
^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: ext3 performance bottleneck as the number of spindles gets large
2002-06-21 18:46 ` mgross
@ 2002-06-21 19:26 ` Chris Mason
2002-06-21 19:56 ` Andrew Morton
1 sibling, 0 replies; 16+ messages in thread
From: Chris Mason @ 2002-06-21 19:26 UTC (permalink / raw)
To: mgross
Cc: Andrew Morton, Griffiths, Richard A, 'Jens Axboe',
Linux Kernel Mailing List, lse-tech
On Fri, 2002-06-21 at 14:46, mgross wrote:
> Andrew Morton wrote:
> >
> >Please try the below patch (againt 2.4.19-pre10). It halves the lock
> >contention, and it does that by making the fs twice as efficient, so
> >that's a bonus.
> >
> We'll give it a try. I'm on travel right now so it may be a few days if
> Richard doesn't get to before I get back.
You might want to try this too, Andrew fixed UPDATE_ATIME() to only call
the dirty_inode method once per second, but generic_file_write should do
the same. It reduces BKL contention by reducing calls to ext3 and
reiserfs dirty_inode calls, which are much more expensive than simply
marking the inode dirty.
-chris
--- linux/mm/filemap.c Mon, 28 Jan 2002 09:51:50 -0500
+++ linux/mm/filemap.c Sun, 12 May 2002 16:16:59 -0400
@@ -2826,6 +2826,14 @@
}
}
+static void update_inode_times(struct inode *inode)
+{
+ time_t now = CURRENT_TIME;
+ if (inode->i_ctime != now || inode->i_mtime != now) {
+ inode->i_ctime = inode->i_mtime = now;
+ mark_inode_dirty_sync(inode);
+ }
+}
/*
* Write to a file through the page cache.
*
@@ -2955,8 +2963,7 @@
goto out;
remove_suid(inode);
- inode->i_ctime = inode->i_mtime = CURRENT_TIME;
- mark_inode_dirty_sync(inode);
+ update_inode_times(inode);
if (file->f_flags & O_DIRECT)
goto o_direct;
^ permalink raw reply [flat|nested] 16+ messages in thread* Re: ext3 performance bottleneck as the number of spindles gets large
2002-06-21 18:46 ` mgross
2002-06-21 19:26 ` Chris Mason
@ 2002-06-21 19:56 ` Andrew Morton
1 sibling, 0 replies; 16+ messages in thread
From: Andrew Morton @ 2002-06-21 19:56 UTC (permalink / raw)
To: mgross
Cc: Griffiths, Richard A, 'Jens Axboe',
Linux Kernel Mailing List, lse-tech
mgross wrote:
>
> ...
> >And please tell us some more details regarding the performance bottleneck.
> >I assume that you mean that the IO rate per disk slows as more
> >disks are added to an adapter? Or does the total throughput through
> >the adapter fall as more disks are added?
> >
> No, the IO block write throughput for the system goes down as drives are
> added under this work load. We measure the system throughput not the
> per drive throughput, but one could infer the per drive throughput by
> dividing.
>
> Running bonnie++ on with 300MB files doing 8Kb sequential writes we get
> the following system wide throughput as a function of the number of
> drives attached and by number of addapters.
>
> One addapter
> 1 drive per addapter 127,702KB/Sec
> 2 drives per addapter 93,283 KB/Sec
> 6 drives per addapter 85,626 KB/Sec
127 megabytes/sec to a single disk? Either that's a very
fast disk, or you're using very small bytes :)
> 2 addapters
> 1 drive per addapter 92,095 KB/Sec
> 2 drives per addapter 110,956 KB/Sec
> 6 drives per addapter 106,883 KB/Sec
>
> 4 addapters
> 1 drive per addapter 121,125 KB/Sec
> 2 drives per addapter 117,575 KB/Sec
> 6 drives per addapter 116,570 KB/Sec
>
Possibly what is happening here is that a significant amount
of dirty data is being left in memory and is escaping the
measurement period. When you run the test against more disks,
the *total* amount of dirty memory is increased, so the kernel
is forced to perform more writeback within the measurement period.
So with two filesystems, you're actually performing more I/O.
You need to either ensure that all I/O is occurring *within the
measurement interval*, or make the test write so much data (wrt
main memory size) that any leftover unwritten stuff is insignificant.
bonnie++ is too complex for this work. Suggest you use
http://www.zip.com.au/~akpm/linux/write-and-fsync.c
which will just write and fsync a file. Time how long that
takes. Or you could experiment with bonnie++'s fsync option.
My suggestion is to work with this workload:
for i in /mnt/1 /mnt/2 /mnt/3 /mnt/4 ...
do
write-and-fsync $i/foo 4000 &
done
which will write a 4 gig file to each disk. This will defeat
any caching effects and is just a way simpler workload, which
will allow you to test one thing in isolation.
So anyway. All this possibly explains the "negative scalability"
in the single-adapter case. For four adapters with one disk on
each, 120 megs/sec seems reasonable, assuming the sustained
write bandwidth of a single disk is 30 megs/sec.
For four adapters, six disks on each you should be doing better.
Something does appear to be wrong there.
-
^ permalink raw reply [flat|nested] 16+ messages in thread
* RE: ext3 performance bottleneck as the number of spindles gets large
2002-06-20 21:50 ext3 performance bottleneck as the number of spindles gets la rge Griffiths, Richard A
2002-06-21 7:58 ` ext3 performance bottleneck as the number of spindles gets large Andrew Morton
@ 2002-06-23 4:02 ` Christopher E. Brown
2002-06-23 4:33 ` Andreas Dilger
1 sibling, 1 reply; 16+ messages in thread
From: Christopher E. Brown @ 2002-06-23 4:02 UTC (permalink / raw)
To: Griffiths, Richard A
Cc: 'Andrew Morton', mgross, 'Jens Axboe',
Linux Kernel Mailing List, lse-tech
On Thu, 20 Jun 2002, Griffiths, Richard A wrote:
> I should have mentioned the throughput we saw on 4 adapters 6 drives was
> 126KB/s. The max theoretical bus bandwith is 640MB/s.
This is *NOT* correct. Assuming a 64bit 66Mhz PCI bus your MAX is
503MB/sec minus PCI overhead...
This of course assumes nothing else is using the PCI bus.
120 something MB/sec sounds a hell of a lot like topping out a 32bit
33Mhz PCI bus, but IIRC the earlier posting listed 39160 cards, PCI
64bit w/ backward compat to 32bit.
You do have *ALL* of these cards plugged into a full PCI 64bit/66Mhz
slot right? Not plugging them into a 32bit/33Mhz slot?
32bit/33Mhz (32 * 33,000,000) / (1024 * 1024 * 8) = 125.89 MByte/sec
64bit/33Mhz (64 * 33,000,000) / (1024 * 1024 * 8) = 251.77 MByte/sec
64bit/66Mhz (64 * 66,000,000) / (1024 * 1024 * 8) = 503.54 MByte/sec
NOTE: PCI transfer rates are often listed as
32bit/33Mhz, 132 MByte/sec
64bit/33Mhz, 264 MByte/sec
64bit/66Mhz, 528 MByte/sec
This is somewhat true, but only if we start with Mbit rates as used in
transmission rates (1,000,000 bits/sec) and work from there, instead
of 2^20 (1,048,576). I will not argue about PCI 32bit/33Mhz being
1056Mbit, if talking about line rate, but when we are talking about
storage media and transfers to/from as measured by files remember to
convert.
--
I route, therefore you are.
^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: ext3 performance bottleneck as the number of spindles gets large
2002-06-23 4:02 ` Christopher E. Brown
@ 2002-06-23 4:33 ` Andreas Dilger
2002-06-23 6:00 ` Christopher E. Brown
0 siblings, 1 reply; 16+ messages in thread
From: Andreas Dilger @ 2002-06-23 4:33 UTC (permalink / raw)
To: Christopher E. Brown
Cc: Griffiths, Richard A, 'Andrew Morton', mgross,
'Jens Axboe', Linux Kernel Mailing List, lse-tech
On Jun 22, 2002 22:02 -0600, Christopher E. Brown wrote:
> On Thu, 20 Jun 2002, Griffiths, Richard A wrote:
>
> > I should have mentioned the throughput we saw on 4 adapters 6 drives was
> > 126KB/s. The max theoretical bus bandwith is 640MB/s.
>
> This is *NOT* correct. Assuming a 64bit 66Mhz PCI bus your MAX is
> 503MB/sec minus PCI overhead...
Assuming you only have a single PCI bus...
Cheers, Andreas
--
Andreas Dilger
http://www-mddsp.enel.ucalgary.ca/People/adilger/
http://sourceforge.net/projects/ext2resize/
^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: ext3 performance bottleneck as the number of spindles gets large
2002-06-23 4:33 ` Andreas Dilger
@ 2002-06-23 6:00 ` Christopher E. Brown
0 siblings, 0 replies; 16+ messages in thread
From: Christopher E. Brown @ 2002-06-23 6:00 UTC (permalink / raw)
To: Andreas Dilger
Cc: Griffiths, Richard A, 'Andrew Morton', mgross,
'Jens Axboe', Linux Kernel Mailing List, lse-tech
On Sat, 22 Jun 2002, Andreas Dilger wrote:
> On Jun 22, 2002 22:02 -0600, Christopher E. Brown wrote:
> > On Thu, 20 Jun 2002, Griffiths, Richard A wrote:
> >
> > > I should have mentioned the throughput we saw on 4 adapters 6 drives was
> > > 126KB/s. The max theoretical bus bandwith is 640MB/s.
> >
> > This is *NOT* correct. Assuming a 64bit 66Mhz PCI bus your MAX is
> > 503MB/sec minus PCI overhead...
>
> Assuming you only have a single PCI bus...
Yes, we could (for example) assume a DP264 board, it features 2/4/8
way memory interleave, dual 21264 CPUs, and 2 separate PCI 64bit 66Mhz
buses.
However, multiple busses are *rare* on x86. There are alot of chained
busses via PCI to PCI bridge, but few systems with 2 or more PCI
busses of any type with parallel access to the CPU.
--
I route, therefore you are.
^ permalink raw reply [flat|nested] 16+ messages in thread
* ext3 performance bottleneck as the number of spindles gets large
@ 2002-06-19 21:29 mgross
2002-06-20 0:54 ` Andrew Morton
2002-06-20 1:55 ` Andrew Morton
0 siblings, 2 replies; 16+ messages in thread
From: mgross @ 2002-06-19 21:29 UTC (permalink / raw)
To: Linux Kernel Mailing List, lse-tech; +Cc: richard.a.griffiths
[-- Attachment #1: Type: text/plain, Size: 2309 bytes --]
We've been doing some throughput comparisons and benchmarks of block I/O
throughput for 8KB writes as the number of SCSI addapters and drives per
adapter is increased.
The Linux platform is a dual processor 1.2GHz PIII, 2Gig or RAM, 2U box.
Similar results have been seen with both 2.4.16 and 2.4.18 base kernel, as
well as one of those patched up O(1) 2.4.18 kernels out there.
The benchmark is Bonnie++.
What seems to be happening is the throughput for 8Kb sequential Write's with
300MB files goes down with the number of spindles. We have negative scale WRT
spindles per SCSI adapter, and very poor scaling per SCSI adapter.
(The other 2 processor + OS platform sees its throughput go up with adapters and
spindles. )
Running this benchmark with lockmeter ends up pointing a big finger at BKL
contention in: ext3_commit_write, ext3_dirty_inode, ext3_get_block_handle
and, ext3_prepare_write (twice!). Attached is the output from the worst
case, 4 SCSI adapters with 6 drives per adapter.
Has anyone done any work looking into the I/O scaling of Linux / ext3 per
spindle or per adapter? We would like to compare notes.
I've only just started to look at the ext3 code but it seems to me that replacing the
BKL with a per - ext3 file system lock could remove some of the contention thats
getting measured. What data are the BKL protecting in these ext3 functions? Could a
lock per FS approach work?
Thoughts?
Comments?
Ideas?
--mgross
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
SPINLOCKS HOLD WAIT
UTIL CON MEAN( MAX ) MEAN( MAX )(% CPU) TOTAL NOWAIT SPIN RJECT NAME
3.7% 0.7us( 44ms) 7.8us( 44ms)(22.9%) 49644038 96.3% 3.7% 0.00% *TOTAL*
26.6% 71.2% 13us( 44ms) 8.0us(8076us)( 5.8%) 632107 28.8% 71.2% 0% ext3_commit_write+0x38
4.4% 30.3% 4.3us( 360us) 13us(7511us)( 2.1%) 316124 69.7% 30.3% 0% ext3_dirty_inode+0x2c
28.1% 7.9% 14us(1660us) 9.7us(6842us)(0.78%) 632239 92.1% 7.9% 0% ext3_get_block_handle+0x8c
1.2% 27.2% 0.6us( 240us) 11us(6604us)( 3.0%) 632107 72.8% 27.2% 0% ext3_prepare_write+0x34
0.26% 88.1% 0.1us( 74us) 9.6us(7026us)( 8.6%) 632107 11.9% 88.1% 0% ext3_prepare_write+0xe0
[-- Attachment #2: lm_4x6_300MBw --]
[-- Type: text/plain, Size: 40755 bytes --]
Lockmeter statistics are now RESET
Lockmeter statistics are now ON
___________________________________________________________________________________________
System: Linux TSRLT2 2.4.18 #2 SMP Mon Jun 17 08:28:25 PDT 2002 i686
Total counts
All (2) CPUs
Start time: Mon Jun 17 11:12:36 2002
End time: Mon Jun 17 11:13:07 2002
Delta Time: 30.99 sec.
Hash table slots in use: 314.
Global read lock slots in use: 884.
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
SPINLOCKS HOLD WAIT
UTIL CON MEAN( MAX ) MEAN( MAX )(% CPU) TOTAL NOWAIT SPIN RJECT NAME
3.7% 0.7us( 44ms) 7.8us( 44ms)(22.9%) 49644038 96.3% 3.7% 0.00% *TOTAL*
0.00% 0% 1.6us( 3.4us) 0us 3 100% 0% 0% [0xdff2bf90]
0.00% 0% 3.4us( 3.4us) 0us 1 100% 0% 0% complete+0x1c
0.00% 0% 1.3us( 1.3us) 0us 1 100% 0% 0% wait_for_completion+0x18
0.00% 0% 0.1us( 0.1us) 0us 1 100% 0% 0% wait_for_completion+0x98
0.00% 0% 0.1us( 0.1us) 0us 1 100% 0% 0% [0xdff56694]
0.00% 0% 0.1us( 0.1us) 0us 1 100% 0% 0% exec_mmap+0x8c
0.00% 0% 1.2us( 2.0us) 0us 7 100% 0% 0% [0xf410c22c]
0.00% 0% 0.9us( 1.5us) 0us 5 100% 0% 0% unmap_fixup+0x8c
0.00% 0% 1.9us( 2.0us) 0us 2 100% 0% 0% unmap_fixup+0x134
0.00% 0% 0.1us( 0.1us) 0us 1 100% 0% 0% [0xf683bf0c]
0.00% 0% 0.1us( 0.1us) 0us 1 100% 0% 0% neigh_destroy+0x108
0.00% 0% 2.5us( 3.9us) 0us 15 100% 0% 0% [0xf6e170d0]
0.00% 0% 2.5us( 3.9us) 0us 15 100% 0% 0% dev_watchdog+0x14
0.00% 0% 0.4us( 0.7us) 0us 2 100% 0% 0% [0xf703f504]
0.00% 0% 0.1us( 0.1us) 0us 1 100% 0% 0% skb_recv_datagram+0x90
0.00% 0% 0.7us( 0.7us) 0us 1 100% 0% 0% unix_dgram_sendmsg+0x35c
0.37% 0.31% 2.5us( 12us) 3.9us( 10us)(0.00%) 45356 99.7% 0.31% 0% allocator_request_lock
0.07% 0.49% 1.0us( 7.8us) 3.9us( 10us)(0.00%) 22678 99.5% 0.49% 0% scsi_free+0x1c
0.30% 0.14% 4.1us( 12us) 3.7us( 7.0us)(0.00%) 22678 99.9% 0.14% 0% scsi_malloc+0x48
0.00% 0% 0.1us( 0.6us) 0us 36 100% 0% 0% arbitration_lock
0.00% 0% 0.1us( 0.6us) 0us 30 100% 0% 0% deny_write_access+0xc
0.00% 0% 0.2us( 0.3us) 0us 6 100% 0% 0% get_write_access+0xc
0.00% 0% 0.7us( 2.8us) 0us 1011 100% 0% 0% bdev_lock
0.00% 0% 0.7us( 2.8us) 0us 1011 100% 0% 0% bdget+0x34
0.00% 0% 4.1us( 5.8us) 0us 18 100% 0% 0% call_lock
0.00% 0% 4.1us( 5.8us) 0us 18 100% 0% 0% smp_call_function+0x58
0.00% 0% 0.2us( 0.9us) 0us 6 100% 0% 0% cdev_lock
0.00% 0% 0.2us( 0.9us) 0us 6 100% 0% 0% cdput+0x28
2.9% 0.59% 0.7us( 6.5us) 1.0us( 3.6us)(0.01%) 1256678 99.4% 0.59% 0% contig_page_data+0xa8
0.72% 1.1% 0.4us( 5.8us) 1.0us( 3.4us)(0.01%) 628341 98.9% 1.1% 0% __free_pages_ok+0xc8
2.2% 0.06% 1.1us( 6.5us) 0.9us( 3.6us)(0.00%) 628337 100% 0.06% 0% rmqueue+0x28
0.14% 0.02% 0.1us( 84us) 4.4us( 47us)(0.00%) 325106 100% 0.02% 0% dcache_lock
0.00% 0% 0.1us( 0.6us) 0us 20 100% 0% 0% d_alloc+0x128
0.00% 0% 0.1us( 0.1us) 0us 2 100% 0% 0% d_delete+0x10
0.00% 0% 0.1us( 0.3us) 0us 23 100% 0% 0% d_instantiate+0x1c
0.01% 0.02% 0.4us( 24us) 0.9us( 0.9us)(0.00%) 4375 100% 0.02% 0% d_lookup+0x5c
0.00% 0% 0.2us( 1.2us) 0us 20 100% 0% 0% d_rehash+0x40
0.00% 0% 1.2us( 1.2us) 0us 1 100% 0% 0% do_readv_writev+0x28c
0.00% 0.09% 0.1us( 3.1us) 1.2us( 1.2us)(0.00%) 1069 100% 0.09% 0% dput+0x30
0.00% 0% 2.0us( 3.4us) 0us 10 100% 0% 0% link_path_walk+0x2a8
0.00% 0% 0.2us( 0.7us) 0us 4 100% 0% 0% notify_change+0xec
0.00% 0% 1.1us( 1.9us) 0us 4 100% 0% 0% prune_dcache+0x14
0.00% 0.87% 0.5us( 59us) 0.9us( 1.5us)(0.00%) 3231 99.1% 0.87% 0% prune_dcache+0x138
0.00% 0% 2.3us( 2.3us) 0us 1 100% 0% 0% sys_getcwd+0xc8
0.00% 0.30% 0.2us( 1.4us) 0.6us( 0.6us)(0.00%) 334 99.7% 0.30% 0% sys_read+0xac
0.13% 0.01% 0.1us( 84us) 8.0us( 47us)(0.00%) 316012 100% 0.01% 0% sys_write+0xac
0.20% 0.10% 1.3us( 41us) 1.7us( 6.1us)(0.00%) 47856 99.9% 0.10% 0% device_request_lock
0.01% 0.15% 0.1us( 1.6us) 2.0us( 6.1us)(0.00%) 23928 99.8% 0.15% 0% __scsi_release_command+0x14
0.19% 0.05% 2.4us( 41us) 1.0us( 1.7us)(0.00%) 23928 100% 0.05% 0% scsi_allocate_device+0x30
0.00% 0% 0.2us( 2.5us) 0us 642 100% 0% 0% files_lock
0.00% 0% 0.1us( 1.1us) 0us 200 100% 0% 0% file_move+0x18
0.00% 0% 0.1us( 0.8us) 0us 201 100% 0% 0% fput+0x80
0.00% 0% 0.3us( 2.5us) 0us 203 100% 0% 0% get_empty_filp+0xc
0.00% 0% 0.7us( 1.4us) 0us 37 100% 0% 0% get_empty_filp+0xdc
0.00% 0% 0.1us( 0.1us) 0us 1 100% 0% 0% put_filp+0x18
2.9% 8.9% 32us(1727us) 0us 27945 91.1% 0% 8.9% global_bh_lock
2.9% 8.9% 32us(1727us) 0us 27945 91.1% 0% 8.9% bh_action+0x18
0.05% 0% 5.2us( 7.8us) 0us 3099 100% 0% 0% i8253_lock
0.05% 0% 5.2us( 7.8us) 0us 3099 100% 0% 0% timer_interrupt+0x2c
0.01% 0% 1.2us( 2.5us) 0us 3099 100% 0% 0% i8259A_lock
0.01% 0% 1.2us( 2.5us) 0us 3099 100% 0% 0% timer_interrupt+0x90
0.00% 0% 0.4us( 0.4us) 0us 1 100% 0% 0% inet_peer_unused_lock
0.00% 0% 0.4us( 0.4us) 0us 1 100% 0% 0% cleanup_once+0x24
0.00% 0% 0.1us( 1.2us) 0us 34 100% 0% 0% init_mm+0x2c
0.00% 0% 0.9us( 1.2us) 0us 2 100% 0% 0% __vmalloc+0x70
0.00% 0% 0.1us( 0.3us) 0us 32 100% 0% 0% __vmalloc+0x120
0.00% 0% 0.5us( 295us) 0us 2768 100% 0% 0% inode_lock
0.00% 0% 0.4us( 1.9us) 0us 309 100% 0% 0% __mark_inode_dirty+0x48
0.00% 0% 0.8us( 1.0us) 0us 4 100% 0% 0% get_empty_inode+0x24
0.00% 0% 0.7us( 1.3us) 0us 12 100% 0% 0% get_new_inode+0x34
0.00% 0% 1.3us( 2.0us) 0us 12 100% 0% 0% iget4+0x3c
0.00% 0% 0.5us( 0.5us) 0us 2 100% 0% 0% insert_inode_hash+0x44
0.00% 0% 0.1us( 2.1us) 0us 2138 100% 0% 0% iput+0x68
0.00% 0% 182us( 295us) 0us 4 100% 0% 0% prune_icache+0x1c
0.00% 0% 5.8us( 8.0us) 0us 6 100% 0% 0% sync_unlocked_inodes+0x10
0.00% 0% 1.1us( 42us) 0us 281 100% 0% 0% sync_unlocked_inodes+0x10c
2.0% 2.6% 0.8us( 103us) 2.6us( 29us)(0.08%) 788392 97.4% 2.6% 0% io_request_lock
0.00% 6.5% 0.6us( 2.0us) 4.6us( 13us)(0.00%) 262 93.5% 6.5% 0% __get_request_wait+0x90
1.1% 2.0% 0.5us( 46us) 3.0us( 29us)(0.06%) 666006 98.0% 2.0% 0% __make_request+0xc0
0.19% 4.4% 2.4us( 36us) 2.8us( 22us)(0.00%) 23953 95.6% 4.4% 0% ahc_linux_isr+0x2ec
0.03% 9.6% 4.6us( 46us) 1.6us( 15us)(0.00%) 2319 90.4% 9.6% 0% generic_unplug_device+0x10
0.31% 8.9% 4.0us( 103us) 1.5us( 27us)(0.01%) 24068 91.1% 8.9% 0% scsi_dispatch_cmd+0x11c
0.02% 3.9% 0.3us( 1.9us) 2.3us( 20us)(0.00%) 23928 96.1% 3.9% 0% scsi_finish_command+0x18
0.17% 3.0% 2.2us( 36us) 2.2us( 15us)(0.00%) 23928 97.0% 3.0% 0% scsi_queue_next_request+0x18
0.16% 7.0% 2.0us( 37us) 1.4us( 11us)(0.00%) 23928 93.0% 7.0% 0% scsi_request_fn+0x31c
0.51% 0% 0.1us( 147us) 0us 1300436 100% 0% 0% jh_splice_lock
0.27% 0% 0.1us( 59us) 0us 665500 100% 0% 0% __journal_remove_journal_head+0xe8
0.24% 0% 0.1us( 147us) 0us 634936 100% 0% 0% journal_add_journal_head+0xd0
12.8% 0.49% 0.1us( 15ms) 1.3us( 15ms)(0.33%) 33219940 99.5% 0.49% 0% journal_datalist_lock
0.00% 0% 2.3us( 2.3us) 0us 1 100% 0% 0% dispose_buffer+0x18
0.00% 2.7% 1.1us( 7.8us) 0.7us( 1.0us)(0.00%) 828 97.3% 2.7% 0% do_get_write_access+0x9c
1.6% 0.25% 0.1us( 163us) 1.0us( 84us)(0.03%) 8220760 99.8% 0.25% 0% do_get_write_access+0x204
2.0% 0.40% 0.1us( 208us) 0.9us( 182us)(0.05%) 8855491 99.6% 0.40% 0% journal_add_journal_head+0x10
0.62% 0.49% 0.3us( 158us) 0.8us( 6.0us)(0.00%) 634936 99.5% 0.49% 0% journal_add_journal_head+0x88
0.01% 0% 25us(1934us) 0us 179 100% 0% 0% journal_commit_transaction+0x1bc
3.0% 0.53% 2460us( 15ms) 1.0us( 1.2us)(0.00%) 377 99.5% 0.53% 0% journal_commit_transaction+0x258
0.08% 0% 23us( 369us) 0us 1111 100% 0% 0% journal_commit_transaction+0x3c0
0.01% 0.73% 1.4us( 50us) 1.0us( 1.4us)(0.00%) 1786 99.3% 0.73% 0% journal_commit_transaction+0xd5c
0.00% 1.1% 0.2us( 0.8us) 1.1us( 1.3us)(0.00%) 179 98.9% 1.1% 0% journal_commit_transaction+0xee4
0.45% 0.44% 0.2us( 111us) 1.0us( 2.8us)(0.00%) 632107 99.6% 0.44% 0% journal_dirty_data+0x54
2.7% 0.28% 0.2us( 578us) 0.9us( 50us)(0.02%) 5376063 99.7% 0.28% 0% journal_dirty_metadata+0x54
0.01% 0.70% 0.4us( 19us) 1.0us( 2.9us)(0.00%) 5537 99.3% 0.70% 0% journal_file_buffer+0x18
0.00% 0% 2.2us( 20us) 0us 619 100% 0% 0% journal_get_create_access+0x130
0.60% 9.4% 0.3us( 106us) 1.8us( 15ms)(0.18%) 632510 90.6% 9.4% 0% journal_try_to_free_buffers+0x4c
0.00% 0.36% 0.2us( 2.4us) 0.8us( 1.1us)(0.00%) 1965 99.6% 0.36% 0% journal_unfile_buffer+0xc
1.7% 0.28% 0.1us( 320us) 1.0us( 263us)(0.04%) 8855491 99.7% 0.28% 0% journal_unlock_journal_head+0xc
0.01% 0% 21us( 41us) 0us 176 100% 0% 0% kbd_controller_lock
0.01% 0% 21us( 41us) 0us 176 100% 0% 0% keyboard_interrupt+0x14
64.2% 46.6% 7.0us( 44ms) 10us( 13ms)(21.4%) 2845886 53.4% 46.6% 0% kernel_flag
0.00% 0% 1.3us( 1.3us) 0us 1 100% 0% 0% chrdev_open+0x4c
0.00% 57.1% 0.4us( 0.5us) 8.6us( 20us)(0.00%) 7 42.9% 57.1% 0% de_put+0x28
0.00% 20.0% 89us( 142us) 4.2us( 4.2us)(0.00%) 5 80.0% 20.0% 0% do_exit+0xd8
26.6% 71.2% 13us( 44ms) 8.0us(8076us)( 5.8%) 632107 28.8% 71.2% 0% ext3_commit_write+0x38
0.00% 0% 43us( 43us) 0us 1 100% 0% 0% ext3_delete_inode+0x48
4.4% 30.3% 4.3us( 360us) 13us(7511us)( 2.1%) 316124 69.7% 30.3% 0% ext3_dirty_inode+0x2c
0.00% 0% 2.2us( 2.2us) 0us 1 100% 0% 0% ext3_force_commit+0x38
28.1% 7.9% 14us(1660us) 9.7us(6842us)(0.78%) 632239 92.1% 7.9% 0% ext3_get_block_handle+0x8c
1.2% 27.2% 0.6us( 240us) 11us(6604us)( 3.0%) 632107 72.8% 27.2% 0% ext3_prepare_write+0x34
0.26% 88.1% 0.1us( 74us) 9.6us(7026us)( 8.6%) 632107 11.9% 88.1% 0% ext3_prepare_write+0xe0
0.00% 0% 5.1us( 5.1us) 0us 1 100% 0% 0% get_chrfops+0x88
0.00% 100% 0.8us( 0.8us) 33us( 33us)(0.00%) 1 0% 100% 0% locks_remove_posix+0x3c
0.00% 0% 137us( 221us) 0us 2 100% 0% 0% lookup_hash+0x7c
0.00% 25.0% 17us( 49us) 27us( 27us)(0.00%) 4 75.0% 25.0% 0% notify_change+0x50
0.00% 50.0% 40us( 121us) 5.6us( 9.4us)(0.00%) 16 50.0% 50.0% 0% real_lookup+0x64
3.7% 58.4% 1007us( 15ms) 1062us( 13ms)( 1.1%) 1136 41.6% 58.4% 0% schedule+0x508
0.00% 83.3% 181us( 284us) 17us( 57us)(0.00%) 6 16.7% 83.3% 0% sync_old_buffers+0x1c
0.00% 0% 3.2us( 4.2us) 0us 3 100% 0% 0% sys_ioctl+0x4c
0.00% 50.0% 1.3us( 2.1us) 33us( 105us)(0.00%) 8 50.0% 50.0% 0% sys_llseek+0x88
0.00% 0% 0.8us( 0.8us) 0us 1 100% 0% 0% sys_lseek+0x70
0.00% 50.0% 7.5us( 9.1us) 13us( 13us)(0.00%) 2 50.0% 50.0% 0% sys_sysctl+0x70
0.00% 0% 115us( 172us) 0us 2 100% 0% 0% vfs_create+0x84
0.00% 0% 13us( 13us) 0us 1 100% 0% 0% vfs_link+0xa4
0.00% 0% 12us( 24us) 0us 2 100% 0% 0% vfs_readdir+0x68
0.00% 0% 15us( 16us) 0us 2 100% 0% 0% vfs_unlink+0x108
0.00% 0% 0.6us( 0.9us) 0us 5 100% 0% 0% lastpid_lock
0.00% 0% 0.6us( 0.9us) 0us 5 100% 0% 0% get_pid+0x20
0.00% 0% 0.5us( 1.3us) 0us 176 100% 0% 0% logbuf_lock
0.00% 0% 0.5us( 1.3us) 0us 176 100% 0% 0% release_console_sem+0x1c
7.5% 0.74% 0.9us( 44ms) 14us( 44ms)(0.46%) 2683880 99.3% 0.74% 0% lru_list_lock
1.2% 1.5% 7.1us( 499us) 2.5us( 134us)(0.00%) 50765 98.5% 1.5% 0% balance_dirty+0x18
0.06% 16.8% 24us( 121us) 12us( 91us)(0.00%) 792 83.2% 16.8% 0% bdflush+0x98
0.54% 0% 28ms( 44ms) 0us 6 100% 0% 0% bdflush+0xb8
0.14% 0.42% 0.1us( 85us) 2.0us( 19us)(0.01%) 632101 99.6% 0.42% 0% buffer_insert_inode_data_queue+0x10
0.00% 100% 0.4us( 0.4us) 11us( 11us)(0.00%) 1 0% 100% 0% fsync_inode_buffers+0x28
0.00% 0% 0.7us( 0.7us) 0us 1 100% 0% 0% fsync_inode_data_buffers+0x28
0.00% 0% 0.6us( 0.6us) 0us 1 100% 0% 0% fsync_inode_data_buffers+0xb4
0.00% 0% 0.1us( 0.1us) 0us 1 100% 0% 0% fsync_inode_data_buffers+0x128
0.00% 3.8% 0.1us( 0.7us) 1.4us( 3.5us)(0.00%) 1760 96.2% 3.8% 0% inode_has_buffers+0x10
0.00% 1.1% 0.1us( 0.7us) 1.1us( 1.9us)(0.00%) 877 98.9% 1.1% 0% invalidate_inode_buffers+0x10
0.19% 0% 9733us( 17ms) 0us 6 100% 0% 0% kupdate+0x98
0.00% 0% 1.0us( 1.0us) 0us 1 100% 0% 0% osync_inode_buffers+0x14
0.00% 0% 0.8us( 0.8us) 0us 1 100% 0% 0% osync_inode_data_buffers+0x14
1.2% 0.29% 0.3us( 192us) 45us( 44ms)(0.29%) 1363983 99.7% 0.29% 0% refile_buffer+0xc
0.00% 0% 0.5us( 0.8us) 0us 6 100% 0% 0% sync_old_buffers+0x64
4.2% 1.9% 2.1us( 384us) 7.7us( 29ms)(0.15%) 633578 98.1% 1.9% 0% try_to_free_buffers+0x1c
0.00% 0% 0.5us( 1.9us) 0us 55 100% 0% 0% mmlist_lock
0.00% 0% 0.1us( 0.1us) 0us 4 100% 0% 0% copy_mm+0x120
0.00% 0% 0.5us( 0.5us) 0us 1 100% 0% 0% exec_mmap+0x50
0.00% 0% 0.4us( 0.6us) 0us 5 100% 0% 0% mmput+0x28
0.00% 0% 0.6us( 1.9us) 0us 45 100% 0% 0% swap_out+0x50
0.00% 0% 0.3us( 1.7us) 0us 223 100% 0% 0% page_uptodate_lock.0
0.00% 0% 0.3us( 1.7us) 0us 223 100% 0% 0% end_buffer_io_async+0x38
5.7% 0.86% 0.9us( 278us) 1.6us( 240us)(0.04%) 1901672 99.1% 0.86% 0% pagecache_lock
0.00% 2.5% 0.5us( 2.4us) 1.2us( 2.4us)(0.00%) 1028 97.5% 2.5% 0% __find_get_page+0x18
2.6% 0.35% 1.3us( 278us) 4.7us( 240us)(0.02%) 632107 99.7% 0.35% 0% __find_lock_page+0xc
1.6% 1.1% 0.8us( 196us) 1.0us( 158us)(0.01%) 632348 98.9% 1.1% 0% add_to_page_cache_unique+0x18
0.00% 0.29% 0.6us( 3.5us) 1.4us( 1.4us)(0.00%) 349 99.7% 0.29% 0% do_generic_file_read+0x1a4
0.00% 0% 1.1us( 1.5us) 0us 2 100% 0% 0% do_generic_file_read+0x370
0.00% 0% 0.1us( 0.9us) 0us 282 100% 0% 0% filemap_fdatasync+0x20
0.00% 0.35% 0.1us( 1.1us) 2.0us( 2.0us)(0.00%) 282 99.6% 0.35% 0% filemap_fdatawait+0x14
0.00% 0.69% 0.8us( 54us) 1.0us( 2.0us)(0.00%) 1011 99.3% 0.69% 0% find_or_create_page+0x38
0.00% 0.49% 0.7us( 2.4us) 1.0us( 1.4us)(0.00%) 1011 99.5% 0.49% 0% find_or_create_page+0x78
0.00% 0.68% 0.1us( 1.2us) 0.8us( 0.8us)(0.00%) 146 99.3% 0.68% 0% page_cache_read+0x48
0.00% 0% 0.3us( 0.3us) 0us 1 100% 0% 0% remove_inode_page+0x18
0.00% 4.5% 0.1us( 0.9us) 1.4us( 1.5us)(0.00%) 111 95.5% 4.5% 0% set_page_dirty+0x24
1.5% 1.1% 0.7us( 163us) 1.1us( 69us)(0.01%) 632992 98.9% 1.1% 0% shrink_cache+0x2c0
0.00% 0% 0.9us( 0.9us) 0us 1 100% 0% 0% truncate_inode_pages+0x38
0.00% 0% 0.2us( 0.2us) 0us 1 100% 0% 0% truncate_list_pages+0x158
6.3% 20.8% 1.5us( 956us) 1.2us( 953us)(0.54%) 1310161 79.2% 20.8% 0% pagemap_lru_lock
0.00% 2.3% 0.7us( 71us) 1.0us( 2.2us)(0.00%) 1688 97.7% 2.3% 0% activate_page+0xc
0.69% 2.6% 0.3us( 166us) 1.5us( 254us)(0.04%) 634142 97.4% 2.6% 0% lru_cache_add+0x1c
0.00% 0.70% 0.5us( 135us) 1.0us( 1.2us)(0.00%) 714 99.3% 0.70% 0% lru_cache_del+0xc
0.03% 12.7% 0.4us( 30us) 1.7us( 79us)(0.01%) 19785 87.3% 12.7% 0% refill_inactive+0x10
0.11% 11.6% 1.8us( 71us) 2.3us( 122us)(0.01%) 19785 88.4% 11.6% 0% shrink_cache+0x50
0.00% 1.6% 1.0us( 26us) 1.5us( 1.9us)(0.00%) 385 98.4% 1.6% 0% shrink_cache+0x194
0.00% 0% 1.2us( 13us) 0us 85 100% 0% 0% shrink_cache+0x21c
5.5% 39.8% 2.7us( 956us) 1.2us( 953us)(0.48%) 632981 60.2% 39.8% 0% shrink_cache+0x290
0.00% 18.0% 1.0us( 14us) 1.1us( 4.0us)(0.00%) 596 82.0% 18.0% 0% shrink_cache+0x2b0
0.45% 1.8% 3.8us( 21us) 3.3us( 11us)(0.00%) 36592 98.2% 1.8% 0% runqueue_lock
0.09% 4.6% 3.2us( 11us) 3.5us( 8.7us)(0.00%) 9263 95.4% 4.6% 0% __wake_up+0x5c
0.00% 0% 0.7us( 0.7us) 0us 1 100% 0% 0% complete+0x6c
0.00% 0% 0.3us( 0.7us) 0us 3 100% 0% 0% deliver_signal+0x48
0.00% 0% 3.0us( 9.8us) 0us 20 100% 0% 0% process_timeout+0x14
0.00% 0% 3.6us( 3.6us) 0us 1 100% 0% 0% schedule_tail+0x58
0.31% 0.35% 5.4us( 21us) 4.7us( 11us)(0.00%) 18215 99.7% 0.35% 0% schedule+0xa0
0.00% 0% 1.6us( 4.6us) 0us 43 100% 0% 0% schedule+0x264
0.04% 2.2% 1.5us( 8.3us) 2.1us( 7.1us)(0.00%) 8325 97.8% 2.2% 0% schedule+0x4c8
0.00% 0% 0.8us( 8.4us) 0us 721 100% 0% 0% wake_up_process+0x14
0.00% 0% 0.4us( 5.9us) 0us 496 100% 0% 0% sb_lock
0.00% 0% 0.1us( 0.1us) 0us 168 100% 0% 0% drop_super+0x24
0.00% 0% 0.5us( 4.4us) 0us 174 100% 0% 0% sync_supers+0x6c
0.00% 0% 4.0us( 5.9us) 0us 6 100% 0% 0% sync_unlocked_inodes+0x18
0.00% 0% 0.4us( 1.3us) 0us 148 100% 0% 0% sync_unlocked_inodes+0x18c
0.03% 0.06% 0.1us( 1.9us) 1.2us( 2.1us)(0.00%) 68015 100% 0.06% 0% scsi_bhqueue_lock
0.01% 0.07% 0.1us( 1.2us) 1.3us( 2.1us)(0.00%) 43947 100% 0.07% 0% scsi_bottom_half_handler+0x1c
0.02% 0.04% 0.2us( 1.9us) 0.9us( 1.5us)(0.00%) 24068 100% 0.04% 0% scsi_done+0x3c
0.00% 0% 0.4us( 2.1us) 0us 2366 100% 0% 0% semaphore_lock
0.00% 0% 0.4us( 1.9us) 0us 1483 100% 0% 0% __down+0x44
0.00% 0% 0.5us( 2.1us) 0us 715 100% 0% 0% __down+0x78
0.00% 0% 0.3us( 0.3us) 0us 168 100% 0% 0% __down_trylock+0x10
0.00% 0% 0.1us( 6.3us) 0us 416 100% 0% 0% swap_info+0x8
0.00% 0% 0.2us( 6.3us) 0us 111 100% 0% 0% get_swap_page+0x74
0.00% 0% 0.1us( 1.3us) 0us 123 100% 0% 0% swap_duplicate+0x54
0.00% 0% 0.1us( 0.8us) 0us 182 100% 0% 0% swap_info_get+0xb4
0.00% 0% 0.4us( 9.0us) 0us 294 100% 0% 0% swaplock
0.00% 0% 0.4us( 9.0us) 0us 111 100% 0% 0% get_swap_page+0x20
0.00% 0% 1.7us( 1.7us) 0us 1 100% 0% 0% si_swapinfo+0x18
0.00% 0% 0.4us( 2.3us) 0us 182 100% 0% 0% swap_info_get+0x88
0.04% 0.07% 0.2us( 15us) 1.5us( 7.8us)(0.00%) 52142 100% 0.07% 0% timerlist_lock
0.01% 0.07% 0.2us( 2.9us) 1.3us( 3.1us)(0.00%) 24279 100% 0.07% 0% add_timer+0x10
0.01% 0.07% 0.1us( 1.4us) 1.6us( 7.8us)(0.00%) 24423 100% 0.07% 0% del_timer+0x14
0.00% 0% 0.3us( 0.7us) 0us 27 100% 0% 0% del_timer_sync+0x1c
0.00% 0% 0.8us( 1.7us) 0us 197 100% 0% 0% mod_timer+0x18
0.02% 0% 1.5us( 15us) 0us 3099 100% 0% 0% timer_bh+0xcc
0.00% 0% 0.2us( 0.9us) 0us 117 100% 0% 0% timer_bh+0x254
0.00% 0.02% 0.1us( 0.9us) 1.4us( 1.4us)(0.00%) 5566 100% 0.02% 0% tqueue_lock
0.00% 0% 0.1us( 0.8us) 0us 2259 100% 0% 0% __run_task_queue+0x14
0.00% 0% 0.1us( 0.8us) 0us 1067 100% 0% 0% batch_entropy_store+0x7c
0.00% 0.05% 0.1us( 0.6us) 1.4us( 1.4us)(0.00%) 2064 100% 0.05% 0% generic_plug_device+0x34
0.00% 0% 0.2us( 0.9us) 0us 176 100% 0% 0% schedule_task+0x28
0.00% 0% 0.9us( 1.6us) 0us 3 100% 0% 0% uidhash_lock
0.00% 0% 1.6us( 1.6us) 0us 1 100% 0% 0% alloc_uid+0x10
0.00% 0% 0.1us( 0.1us) 0us 1 100% 0% 0% alloc_uid+0x94
0.00% 0% 1.0us( 1.0us) 0us 1 100% 0% 0% free_uid+0x28
1.5% 0.12% 0.4us( 192us) 2.0us( 147us)(0.00%) 1269889 99.9% 0.12% 0% unused_list_lock
0.37% 0.19% 0.2us( 192us) 2.0us( 142us)(0.00%) 635121 99.8% 0.19% 0% get_unused_buffer_head+0x8
0.00% 0.22% 0.1us( 1.3us) 1.0us( 2.4us)(0.00%) 1786 99.8% 0.22% 0% put_unused_buffer_head+0xc
1.2% 0.05% 0.6us( 157us) 2.0us( 147us)(0.00%) 632982 100% 0.05% 0% try_to_free_buffers+0x54
0.00% 0% 0.3us( 1.1us) 0us 8 100% 0% 0% __kmem_cache_shrink+0x18
0.00% 0% 0.1us( 0.3us) 0us 142 100% 0% 0% __kmem_cache_shrink+0x48
0.71% 0.01% 0.1us( 34us) 1.8us( 8.9us)(0.00%) 2045139 100% 0.01% 0% __wake_up+0x24
0.00% 0% 0.1us( 0.6us) 0us 8216 100% 0% 0% add_wait_queue+0x10
0.00% 6.8% 0.1us( 1.3us) 2.9us( 8.4us)(0.00%) 1490 93.2% 6.8% 0% add_wait_queue_exclusive+0x10
0.50% 0.08% 4.3us( 124us) 2.8us( 10us)(0.00%) 35856 100% 0.08% 0% ahc_linux_isr+0x24
0.28% 0.21% 3.6us( 34us) 6.4us( 93us)(0.00%) 24068 99.8% 0.21% 0% ahc_linux_queue+0x34
0.00% 0% 0.7us( 1.6us) 0us 24 100% 0% 0% change_protection+0x34
0.00% 0% 11us( 15us) 0us 9 100% 0% 0% clear_page_tables+0x1c
0.00% 0% 0.1us( 0.4us) 0us 63 100% 0% 0% copy_mm+0x1e8
0.00% 0% 2.8us( 48us) 0us 84 100% 0% 0% copy_mm+0x230
0.00% 0% 3.9us( 48us) 0us 84 100% 0% 0% copy_page_range+0x100
0.02% 0% 0.3us( 2.2us) 0us 27234 100% 0% 0% do_IRQ+0x40
0.02% 0% 0.3us( 2.4us) 0us 27234 100% 0% 0% do_IRQ+0xc0
0.00% 0% 0.9us( 5.2us) 0us 617 100% 0% 0% do_anonymous_page+0x5c
0.00% 0% 0.4us( 0.9us) 0us 5 100% 0% 0% do_brk+0x1d4
0.00% 0% 0.1us( 0.1us) 0us 5 100% 0% 0% do_exit+0x124
0.00% 0% 0.1us( 0.3us) 0us 5 100% 0% 0% do_exit+0x84
0.00% 0% 0.1us( 0.1us) 0us 5 100% 0% 0% do_exit+0xf4
0.00% 0% 0.6us( 2.7us) 0us 86 100% 0% 0% do_mmap_pgoff+0x40c
0.00% 0% 0.5us( 3.8us) 0us 168 100% 0% 0% do_mmap_pgoff+0x418
0.00% 0% 0.1us( 0.8us) 0us 38 100% 0% 0% do_munmap+0x1b8
0.00% 0% 0.6us( 5.4us) 0us 114 100% 0% 0% do_munmap+0xe0
0.00% 0% 0.1us( 1.3us) 0us 1025 100% 0% 0% do_no_page+0xdc
0.00% 0% 0.3us( 1.0us) 0us 14 100% 0% 0% do_page_fault+0xe0
0.00% 0% 0.4us( 4.9us) 0us 114 100% 0% 0% do_sigaction+0x58
0.00% 0% 0.6us( 1.5us) 0us 9 100% 0% 0% do_sigaction+0xd8
0.00% 0% 2.6us( 6.2us) 0us 7 100% 0% 0% do_signal+0x54
0.00% 0% 1.1us( 4.2us) 0us 205 100% 0% 0% do_wp_page+0x118
0.00% 0% 0.1us( 0.1us) 0us 9 100% 0% 0% exit_mmap+0x18
0.00% 0% 0.1us( 0.7us) 0us 143 100% 0% 0% exit_mmap+0x88
0.00% 0% 1.7us( 2.8us) 0us 5 100% 0% 0% exit_sighand+0x18
0.13% 0.02% 7.9us( 30us) 1.2us( 1.2us)(0.00%) 5232 100% 0.02% 0% free_block+0x1c
0.00% 0% 0.3us( 17us) 0us 2025 100% 0% 0% handle_mm_fault+0x34
0.00% 0% 0.4us( 0.7us) 0us 5 100% 0% 0% handle_signal+0xb0
0.00% 0% 1.7us( 3.3us) 0us 5 100% 0% 0% insert_vm_struct+0x60
0.00% 0% 0.1us( 0.3us) 0us 182 100% 0% 0% interruptible_sleep_on+0x28
0.00% 0% 0.1us( 0.2us) 0us 182 100% 0% 0% interruptible_sleep_on+0x54
0.19% 0.02% 2.9us( 51us) 2.1us( 2.4us)(0.00%) 19844 100% 0.02% 0% kmem_cache_alloc_batch+0x18
0.00% 0% 0.1us( 1.3us) 0us 6765 100% 0% 0% kmem_cache_grow+0x1d4
0.01% 0.01% 0.3us( 1.5us) 1.5us( 1.5us)(0.00%) 6765 100% 0.01% 0% kmem_cache_grow+0x80
0.00% 0% 0.1us( 0.6us) 0us 160 100% 0% 0% kmem_cache_reap+0x25c
0.00% 0.07% 0.1us( 0.6us) 10us( 15us)(0.00%) 7063 100% 0.07% 0% kmem_cache_reap+0x2c4
0.72% 0.00% 1.2us( 327us) 3.0us( 6.9us)(0.00%) 194356 100% 0.00% 0% kmem_cache_reap+0xa4
0.00% 0% 1.0us( 2.0us) 0us 24 100% 0% 0% mprotect_fixup+0x2a8
0.00% 0% 0.7us( 1.5us) 0us 24 100% 0% 0% mprotect_fixup+0x2b4
0.00% 0% 5.0us( 41us) 0us 27 100% 0% 0% pte_alloc+0x88
0.00% 0% 0.3us( 0.8us) 0us 5 100% 0% 0% put_dirty_page+0x3c
0.00% 0% 0.1us( 0.4us) 0us 5 100% 0% 0% release_task+0x3c
0.00% 2.9% 0.1us( 1.2us) 3.5us( 9.2us)(0.00%) 9706 97.1% 2.9% 0% remove_wait_queue+0x10
0.07% 0% 1.1us( 263us) 0us 18189 100% 0% 0% schedule+0x478
0.00% 0% 1.7us( 7.4us) 0us 5 100% 0% 0% schedule_tail+0x20
0.00% 0% 13us( 31us) 0us 6 100% 0% 0% send_sig_info+0x4c
0.00% 0% 0.1us( 0.4us) 0us 40 100% 0% 0% sleep_on+0x28
0.00% 0% 0.1us( 0.8us) 0us 40 100% 0% 0% sleep_on+0x54
0.01% 0% 57us( 546us) 0us 45 100% 0% 0% swap_out+0xc8
0.00% 0% 0.1us( 0.1us) 0us 9 100% 0% 0% sys_rt_sigprocmask+0x18c
0.00% 0% 0.3us( 1.8us) 0us 69 100% 0% 0% sys_rt_sigprocmask+0x98
0.00% 0% 0.4us( 0.7us) 0us 5 100% 0% 0% sys_sigreturn+0x84
0.00% 0% 1.1us( 1.7us) 0us 8 100% 0% 0% unmap_fixup+0xa8
0.00% 0% 0.6us( 1.0us) 0us 8 100% 0% 0% unmap_fixup+0xb8
0.00% 0% 0.1us( 4.1us) 0us 249 100% 0% 0% vma_merge+0x54
0.01% 0% 5.9us( 398us) 0us 300 100% 0% 0% zap_page_range+0x48
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
RWLOCK READS HOLD MAX RDR BUSY PERIOD WAIT
UTIL CON MEAN RDRS MEAN( MAX ) MEAN( MAX )( %CPU) TOTAL NOWAIT SPIN NAME
0.68% 2.1us( 229us)(0.11%) 5063033 99.3% 0.68% *TOTAL*
0.00% 0% 3.9us 1 3.9us( 19us) 0us 3 100% 0% [0xe6f1b044]
0% 0us 3 100% 0% copy_files+0x158
0.00% 0% 0.2us 1 0.2us( 5.8us) 0us 1 100% 0% [0xf417a044]
0% 0us 1 100% 0% do_fcntl+0x104
0.00% 0% 1.2us 1 1.2us( 1.2us) 0us 1 100% 0% [0xf703f224]
0% 0us 1 100% 0% unix_write_space+0x14
0.00% 0% 1.3us 1 1.3us( 1.3us) 0us 1 100% 0% [0xf703f448]
0% 0us 1 100% 0% unix_dgram_sendmsg+0x80
0.00% 0% 10.1us 1 10us( 10us) 0us 1 100% 0% [0xf703f564]
0% 0us 1 100% 0% sock_def_readable+0x14
0.00% 0% 4.5us 1 4.5us( 4.5us) 0us 1 100% 0% [0xf703f788]
0% 0us 1 100% 0% unix_dgram_sendmsg+0x21c
0.00% 0% 0.2us 1 0.2us( 3.3us) 0us 1 100% 0% [0xf7658d24]
0% 0us 1 100% 0% sys_getcwd+0x38
0.00% 0% 0.7us 1 0.7us( 0.9us) 0us 2 100% 0% arp_tbl+0xc4
0% 0us 2 100% 0% neigh_lookup+0x40
0.00% 0% 0.9us 1 0.9us( 1.3us) 0us 5 100% 0% binfmt_lock
0% 0us 5 100% 0% search_binary_handler+0x38
0.00% 0% 1.3us 1 1.3us( 1.3us) 0us 1 100% 0% chrdevs_lock
0% 0us 1 100% 0% get_chrfops+0x28
0.00% 0% 3.4us 1 3.4us( 4.2us) 0us 4 100% 0% fib_hash_lock
0% 0us 4 100% 0% fn_hash_lookup+0x10
2.9% 0.72% 0.2us 2 0.2us( 288us) 2.1us( 229us)(0.11%) 4745142 99.3% 0.72% hash_table_lock
0.72% 2.1us( 229us)(0.11%) 4745142 99.3% 0.72% get_hash_table+0x60
0.00% 0% 0.8us 1 0.8us( 1.4us) 0us 4 100% 0% inetdev_lock
0% 0us 2 100% 0% arp_rcv+0x28
0% 0us 2 100% 0% ip_route_input_slow+0x18
0.01% 0% 28.0us 2 28us( 80us) 0us 69 100% 0% tasklist_lock
0% 0us 6 100% 0% count_active_tasks+0xc
0% 0us 5 100% 0% exit_notify+0x18
0% 0us 43 100% 0% schedule+0x218
0% 0us 1 100% 0% sys_setsid+0x10
0% 0us 14 100% 0% sys_wait4+0x8c
0.00% 0% 2.1us 1 2.1us( 3.2us) 0us 3 100% 0% udp_hash_lock
0% 0us 3 100% 0% udp_v4_mcast_deliver+0x10
0.00% 0% 0.6us 1 0.6us( 1.4us) 0us 36 100% 0% xtime_lock
0% 0us 36 100% 0% do_gettimeofday+0x14
0% 0us 5 100% 0% copy_files+0x100
0% 0us 5 100% 0% do_fork+0x35c
0% 0us 14 100% 0% do_select+0x24
0% 0us 316664 100% 0% fget+0x1c
0% 0us 5 100% 0% ip_route_input+0x88
0% 0us 5 100% 0% net_rx_action+0x48
0% 0us 4 100% 0% path_init+0x114
0% 0us 1056 100% 0% path_init+0x30
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
RWLOCK WRITES HOLD WAIT (ALL) WAIT (WW)
UTIL CON MEAN( MAX ) MEAN( MAX )( %CPU) MEAN( MAX ) TOTAL NOWAIT SPIN( WW ) NAME
4.9% 1.4us( 217us) 1.2us( 218us)(0.06%) 0.3us( 34us) 643483 95.1% 3.5%( 1.4%) *TOTAL*
0.00% 0% 0.6us( 1.3us) 0us 0us 6 100% 0%( 0%) [0xe6f1b204]
0.00% 0% 0.5us( 1.3us) 0us 0us 3 100% 0%( 0%) copy_files+0x12c
0.00% 0% 0.6us( 0.7us) 0us 0us 3 100% 0%( 0%) expand_fd_array+0x88
0.00% 0% 4.4us( 21us) 0us 0us 5 100% 0%( 0%) [0xf417a044]
0.00% 0% 0.1us( 0.1us) 0us 0us 2 100% 0%( 0%) do_fcntl+0x140
0.00% 0% 7.3us( 21us) 0us 0us 3 100% 0%( 0%) sys_dup2+0x2c
0.00% 0% 0.1us( 0.1us) 0us 0us 4 100% 0%( 0%) [0xf50ce044]
0.00% 0% 0.1us( 0.1us) 0us 0us 2 100% 0%( 0%) do_pipe+0x174
0.00% 0% 0.1us( 0.1us) 0us 0us 2 100% 0%( 0%) do_pipe+0x1a4
0.00% 0% 0.7us( 0.7us) 0us 0us 1 100% 0%( 0%) [0xf7658d24]
0.00% 0% 0.7us( 0.7us) 0us 0us 1 100% 0%( 0%) sys_chdir+0x9c
0.00% 0% 0.1us( 0.1us) 0us 0us 1 100% 0%( 0%) [0xf7df24f4]
0.00% 0% 0.1us( 0.1us) 0us 0us 1 100% 0%( 0%) neigh_destroy+0x8c
0.00% 0% 104us( 104us) 0us 0us 1 100% 0%( 0%) arp_tbl+0xc4
0.00% 0% 104us( 104us) 0us 0us 1 100% 0%( 0%) neigh_periodic_timer__thr+0x20
0.00% 0% 0.5us( 0.6us) 0us 0us 2 100% 0%( 0%) dn_lock
0.00% 0% 0.5us( 0.6us) 0us 0us 2 100% 0%( 0%) fcntl_dirnotify+0x94
2.9% 5.0% 1.4us( 217us) 1.2us( 218us)(0.06%) 0.3us( 34us) 634589 95.0% 3.6%( 1.4%) hash_table_lock
0.00% 0.59% 1.3us( 30us) 17us( 34us)(0.00%) 8.8us( 34us) 1011 99.4% 0.10%(0.49%) hash_page_buffers+0x48
2.9% 5.0% 1.4us( 217us) 1.2us( 218us)(0.06%) 0.2us( 4.9us) 633578 95.0% 3.6%( 1.4%) try_to_free_buffers+0x28
0.00% 0% 7.6us( 38us) 0us 0us 15 100% 0%( 0%) tasklist_lock
0.00% 0% 2.2us( 2.7us) 0us 0us 5 100% 0%( 0%) do_fork+0x530
0.00% 0% 20us( 38us) 0us 0us 5 100% 0%( 0%) exit_notify+0x1b0
0.00% 0% 0.6us( 1.0us) 0us 0us 5 100% 0%( 0%) release_task+0x7c
0.00% 0% 20us( 40us) 0us 0us 4 100% 0%( 0%) vmlist_lock
0.00% 0% 4.4us( 5.2us) 0us 0us 2 100% 0%( 0%) get_vm_area+0x3c
0.00% 0% 35us( 40us) 0us 0us 2 100% 0%( 0%) vfree+0x58
0.12% 0% 5.9us( 19us) 0us 0us 6198 100% 0%( 0%) xtime_lock
0.02% 0% 1.7us( 6.7us) 0us 0us 3099 100% 0%( 0%) timer_bh+0xc
0.10% 0% 10us( 19us) 0us 0us 3099 100% 0%( 0%) timer_interrupt+0x10
0.00% 0% 0.9us( 1.2us) 0us 0us 5 100% 0%( 0%) flush_old_exec+0x22c
0.00% 0% 0.2us( 3.3us) 0us 0us 813 100% 0%( 0%) get_unused_fd+0x24
0.00% 0% 0.1us( 0.1us) 0us 0us 5 100% 0%( 0%) load_elf_binary+0x190
0.00% 0% 0.4us( 0.5us) 0us 0us 4 100% 0%( 0%) neigh_periodic_timer__thr+0xa8
0.00% 0% 0.1us( 4.1us) 0us 0us 820 100% 0%( 0%) rt_check_expire__thr+0x64
0.00% 0% 0.2us( 2.6us) 0us 0us 206 100% 0%( 0%) sys_close+0x1c
0.00% 0% 0.1us( 1.7us) 0us 0us 188 100% 0%( 0%) sys_open+0x60
0.00% 0% 0.1us( 0.8us) 0us 0us 616 100% 0%( 0%) sys_open+0xa8
_________________________________________________________________________________________________________________________
Number of read locks found=16
Lockmeter statistics are now OFF
^ permalink raw reply [flat|nested] 16+ messages in thread* Re: ext3 performance bottleneck as the number of spindles gets large
2002-06-19 21:29 mgross
@ 2002-06-20 0:54 ` Andrew Morton
2002-06-20 9:54 ` Stephen C. Tweedie
2002-06-20 1:55 ` Andrew Morton
1 sibling, 1 reply; 16+ messages in thread
From: Andrew Morton @ 2002-06-20 0:54 UTC (permalink / raw)
To: mgross; +Cc: Linux Kernel Mailing List, lse-tech, richard.a.griffiths
mgross wrote:
>
> ...
> Has anyone done any work looking into the I/O scaling of Linux / ext3 per
> spindle or per adapter? We would like to compare notes.
No. ext3 scalability is very poor, I'm afraid. The fs really wasn't
up and running until kernel 2.4.5 and we just didn't have time to
address that issue.
> I've only just started to look at the ext3 code but it seems to me that replacing the
> BKL with a per - ext3 file system lock could remove some of the contention thats
> getting measured. What data are the BKL protecting in these ext3 functions? Could a
> lock per FS approach work?
The vague plan there is to replace lock_kernel with lock_journal
where appropriate. But ext3 scalability work of this nature
will be targetted at the 2.5 kernel, most probably.
I'll take a look, see if there's any low-hanging fruit in there,
but I doubt that the results will be fantastic.
-
^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: ext3 performance bottleneck as the number of spindles gets large
2002-06-20 0:54 ` Andrew Morton
@ 2002-06-20 9:54 ` Stephen C. Tweedie
0 siblings, 0 replies; 16+ messages in thread
From: Stephen C. Tweedie @ 2002-06-20 9:54 UTC (permalink / raw)
To: Andrew Morton
Cc: mgross, Linux Kernel Mailing List, lse-tech, richard.a.griffiths,
ext2-devel
Hi,
On Wed, Jun 19, 2002 at 05:54:46PM -0700, Andrew Morton wrote:
> The vague plan there is to replace lock_kernel with lock_journal
> where appropriate. But ext3 scalability work of this nature
> will be targetted at the 2.5 kernel, most probably.
I think we can do better than that, with care. lock_journal could
easily become a read/write lock to protect the transaction state
machine, as there's really only one place --- the commit thread ---
where we end up changing the state of a transaction itself (eg. from
running to committing). For short-lived buffer transformations, we
already have the datalist spinlock.
There are a few intermediate types of operation, such as the
do_get_write_access. That's a buffer operation, but it relies on us
being able to allocate memory for the old version of the buffer if we
happen to be committing the bh to disk already. All of those cases
are already prepared to accept BKL being dropped during the memory
allocation, so there's no problem with doing the same for a short-term
buffer spinlock; and if the journal_lock is only taken shared in such
places, then there's no urgent need to drop that over the malloc.
Even the commit thread can probably avoid taking the journal lock in
many cases --- it would need it exclusively while changing a
transaction's global state, but while it's just manipulating blocks on
the committing transaction it can probably get away with much less
locking.
Cheers,
Stephen
^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: ext3 performance bottleneck as the number of spindles gets large
2002-06-19 21:29 mgross
2002-06-20 0:54 ` Andrew Morton
@ 2002-06-20 1:55 ` Andrew Morton
2002-06-20 6:05 ` Jens Axboe
1 sibling, 1 reply; 16+ messages in thread
From: Andrew Morton @ 2002-06-20 1:55 UTC (permalink / raw)
To: mgross; +Cc: Linux Kernel Mailing List, lse-tech, richard.a.griffiths
mgross wrote:
>
> We've been doing some throughput comparisons and benchmarks of block I/O
> throughput for 8KB writes as the number of SCSI addapters and drives per
> adapter is increased.
>
> The Linux platform is a dual processor 1.2GHz PIII, 2Gig or RAM, 2U box.
> Similar results have been seen with both 2.4.16 and 2.4.18 base kernel, as
> well as one of those patched up O(1) 2.4.18 kernels out there.
umm. Are you not using block-highmem? That is a must-have.
http://www.kernel.org/pub/linux/kernel/people/andrea/kernels/v2.4/2.4.19pre9aa2/00_block-highmem-all-18b-12.gz
-
^ permalink raw reply [flat|nested] 16+ messages in thread
end of thread, other threads:[~2002-06-23 6:08 UTC | newest]
Thread overview: 16+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2002-06-20 15:26 ext3 performance bottleneck as the number of spindles gets la rge Griffiths, Richard A
2002-06-20 20:18 ` ext3 performance bottleneck as the number of spindles gets large Andrew Morton
2002-06-20 18:08 ` mgross
2002-06-20 21:25 ` Andrew Morton
-- strict thread matches above, loose matches on Subject: below --
2002-06-20 21:50 ext3 performance bottleneck as the number of spindles gets la rge Griffiths, Richard A
2002-06-21 7:58 ` ext3 performance bottleneck as the number of spindles gets large Andrew Morton
2002-06-21 18:46 ` mgross
2002-06-21 19:26 ` Chris Mason
2002-06-21 19:56 ` Andrew Morton
2002-06-23 4:02 ` Christopher E. Brown
2002-06-23 4:33 ` Andreas Dilger
2002-06-23 6:00 ` Christopher E. Brown
2002-06-19 21:29 mgross
2002-06-20 0:54 ` Andrew Morton
2002-06-20 9:54 ` Stephen C. Tweedie
2002-06-20 1:55 ` Andrew Morton
2002-06-20 6:05 ` Jens Axboe
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox