New performance results

New performance results

[Steven Pratt]

I am continuing to do runs to provide more data on the random write 
issues with btrfs. I have just posted 2 sets of runs here:
http://btrfs.boxacle.net/repository/raid/longrun/

these are on a pull of the btrfs-unstable experimental branch from 4/3.

These are 100 minute runs of the 128 thread random write workload on the 
raid system (1 for btrfs and 1 for ext3).  Included in these runs are 
graphs of all the iostat, sar and mpstat data (see analysis directories).

A couple of interesting things. First, we see the choppiness of the IO 
in btrfs compared to ext3.
http://btrfs.boxacle.net/repository/raid/longrun/btrfs-longrun/btrfs1.ffsb.random_writes__threads_0128.09-04-06_10.25.03/analysis/iostat-processed.001/chart.html 

http://btrfs.boxacle.net/repository/raid/longrun/ext3-longrun/btrfs1.ffsb.random_writes__threads_0128.09-04-06_13.44.49/analysis/iostat-processed.001/chart.html 


In particular look at graphs 7 and 11 which show write iops and 
throughput.  Ext3 is nice and smooth, while btrfs has a repeating 
pattern of dips and spikes, with IO going to 0 on  a regular basis.

Another interesting observation is what looks a lot like a memory leak.  
Looking at chart 6 Memory at :
http://btrfs.boxacle.net/repository/raid/longrun/btrfs-longrun/btrfs1.ffsb.random_writes__threads_0128.09-04-06_10.25.03/analysis/sar-processed.001/chart.html 

we see that the amount of page cache drops slowly throughout the entire 
run.  Starting up around 3.5GB and dropping to about 2.3GB by the end of 
the run.  The memory seems to have moved to the slab which grew to 
1.5GB.  Doing a repeat of the run while watching slabtop, we see that 
size-2048 is responsible for the majority of the slab usage (over 1GB).

We do not see this behavior under ext3.


Steve

Re: New performance results

[Josef Bacik]

On Mon, Apr 06, 2009 at 05:01:11PM -0500, Steven Pratt wrote:
> I am continuing to do runs to provide more data on the random write  
> issues with btrfs. I have just posted 2 sets of runs here:
> http://btrfs.boxacle.net/repository/raid/longrun/
>
> these are on a pull of the btrfs-unstable experimental branch from 4/3.
>
> These are 100 minute runs of the 128 thread random write workload on the  
> raid system (1 for btrfs and 1 for ext3).  Included in these runs are  
> graphs of all the iostat, sar and mpstat data (see analysis directories).
>
> A couple of interesting things. First, we see the choppiness of the IO  
> in btrfs compared to ext3.
 
> http://btrfs.boxacle.net/repository/raid/longrun/btrfs-longrun/btrfs1.ffsb.random_writes__threads_0128.09-04-06_10.25.03/analysis/iostat-processed.001/chart.html 
> 
>
 
> http://btrfs.boxacle.net/repository/raid/longrun/ext3-longrun/btrfs1.ffsb.random_writes__threads_0128.09-04-06_13.44.49/analysis/iostat-processed.001/chart.html 
> 
>
>
> In particular look at graphs 7 and 11 which show write iops and  
> throughput.  Ext3 is nice and smooth, while btrfs has a repeating  
> pattern of dips and spikes, with IO going to 0 on  a regular basis.
>
> Another interesting observation is what looks a lot like a memory leak.   
> Looking at chart 6 Memory at :
 
> http://btrfs.boxacle.net/repository/raid/longrun/btrfs-longrun/btrfs1.ffsb.random_writes__threads_0128.09-04-06_10.25.03/analysis/sar-processed.001/chart.html 
> 
>
> we see that the amount of page cache drops slowly throughout the entire  
> run.  Starting up around 3.5GB and dropping to about 2.3GB by the end of  
> the run.  The memory seems to have moved to the slab which grew to  
> 1.5GB.  Doing a repeat of the run while watching slabtop, we see that  
> size-2048 is responsible for the majority of the slab usage (over 1GB).
>
> We do not see this behavior under ext3.

Yeah, which tree are you running against?  I had this problem a few weeks ago,
but I don't remember if it was fixed or anything usefull like that, it may be
worth it to try running against the newest stuff.  Thanks,

Josef

Re: New performance results

[Josef Bacik]

On Mon, Apr 06, 2009 at 06:19:58PM -0400, Josef Bacik wrote:
> On Mon, Apr 06, 2009 at 05:01:11PM -0500, Steven Pratt wrote:
> > I am continuing to do runs to provide more data on the random write  
> > issues with btrfs. I have just posted 2 sets of runs here:
> > http://btrfs.boxacle.net/repository/raid/longrun/
> >
> > these are on a pull of the btrfs-unstable experimental branch from 4/3.

Hey there it is, sorry, I will look at this and see if I can remember what the
problem is.  Thanks,

Josef

Re: New performance results

[Chris Mason]

On Mon, 2009-04-06 at 17:01 -0500, Steven Pratt wrote:
> I am continuing to do runs to provide more data on the random write 
> issues with btrfs. I have just posted 2 sets of runs here:
> http://btrfs.boxacle.net/repository/raid/longrun/
> 
> these are on a pull of the btrfs-unstable experimental branch from 4/3.
> 
> These are 100 minute runs of the 128 thread random write workload on the 
> raid system (1 for btrfs and 1 for ext3).  Included in these runs are 
> graphs of all the iostat, sar and mpstat data (see analysis directories).
> 
> A couple of interesting things. First, we see the choppiness of the IO 
> in btrfs compared to ext3.
> http://btrfs.boxacle.net/repository/raid/longrun/btrfs-longrun/btrfs1.ffsb.random_writes__threads_0128.09-04-06_10.25.03/analysis/iostat-processed.001/chart.html 
> 
> http://btrfs.boxacle.net/repository/raid/longrun/ext3-longrun/btrfs1.ffsb.random_writes__threads_0128.09-04-06_13.44.49/analysis/iostat-processed.001/chart.html 
> 
> 
> In particular look at graphs 7 and 11 which show write iops and 
> throughput.  Ext3 is nice and smooth, while btrfs has a repeating 
> pattern of dips and spikes, with IO going to 0 on  a regular basis.
> 

The dips and spikes may be from the allocator.  Basically what happens
is after each commit we end up with a bunch of small blocks available
for filling again.  Could you please try with -o ssd?


> Another interesting observation is what looks a lot like a memory leak.  
> Looking at chart 6 Memory at :
> http://btrfs.boxacle.net/repository/raid/longrun/btrfs-longrun/btrfs1.ffsb.random_writes__threads_0128.09-04-06_10.25.03/analysis/sar-processed.001/chart.html 
> 
> we see that the amount of page cache drops slowly throughout the entire 
> run.  Starting up around 3.5GB and dropping to about 2.3GB by the end of 
> the run.  The memory seems to have moved to the slab which grew to 
> 1.5GB.  Doing a repeat of the run while watching slabtop, we see that 
> size-2048 is responsible for the majority of the slab usage (over 1GB).
> 

size-2048?  That's probably the csums.  I'll give it a shot when I get
back next week.

-chris

Re: New performance results

[Steven Pratt]

Chris Mason wrote:
> On Mon, 2009-04-06 at 17:01 -0500, Steven Pratt wrote:
>   
>> I am continuing to do runs to provide more data on the random write 
>> issues with btrfs. I have just posted 2 sets of runs here:
>> http://btrfs.boxacle.net/repository/raid/longrun/
>>
>> these are on a pull of the btrfs-unstable experimental branch from 4/3.
>>
>> These are 100 minute runs of the 128 thread random write workload on the 
>> raid system (1 for btrfs and 1 for ext3).  Included in these runs are 
>> graphs of all the iostat, sar and mpstat data (see analysis directories).
>>
>> A couple of interesting things. First, we see the choppiness of the IO 
>> in btrfs compared to ext3.
>> http://btrfs.boxacle.net/repository/raid/longrun/btrfs-longrun/btrfs1.ffsb.random_writes__threads_0128.09-04-06_10.25.03/analysis/iostat-processed.001/chart.html 
>>
>> http://btrfs.boxacle.net/repository/raid/longrun/ext3-longrun/btrfs1.ffsb.random_writes__threads_0128.09-04-06_13.44.49/analysis/iostat-processed.001/chart.html 
>>
>>
>> In particular look at graphs 7 and 11 which show write iops and 
>> throughput.  Ext3 is nice and smooth, while btrfs has a repeating 
>> pattern of dips and spikes, with IO going to 0 on  a regular basis.
>>
>>     
>
> The dips and spikes may be from the allocator.  Basically what happens
> is after each commit we end up with a bunch of small blocks available
> for filling again.  Could you please try with -o ssd?
>
>   
Will give it a shot.
>   
>> Another interesting observation is what looks a lot like a memory leak.  
>> Looking at chart 6 Memory at :
>> http://btrfs.boxacle.net/repository/raid/longrun/btrfs-longrun/btrfs1.ffsb.random_writes__threads_0128.09-04-06_10.25.03/analysis/sar-processed.001/chart.html 
>>
>> we see that the amount of page cache drops slowly throughout the entire 
>> run.  Starting up around 3.5GB and dropping to about 2.3GB by the end of 
>> the run.  The memory seems to have moved to the slab which grew to 
>> 1.5GB.  Doing a repeat of the run while watching slabtop, we see that 
>> size-2048 is responsible for the majority of the slab usage (over 1GB).
>>
>>     
>
> size-2048?  That's probably the csums.  I'll give it a shot when I get
> back next week

Ok.

One other thing I noticed that is really bad.  For ext3, we see 
115MB/sec both from the benchmark reporting and from iostat write 
throughput.  However, for btrfs, we see a benchmark throughput of 
2.5MB/sec while iostat shows a whopping 35MB/sec of writes.  That to me 
implies that btrfs is doing an additional 32-33MB/sec of metadata or 
journal writes.  More than 10x the amount of actual data being written.  
Can that be right?

Steve
> .
>
> -chris
>
>   

Re: New performance results

[Steven Pratt]

Steven Pratt wrote:
> Chris Mason wrote:
>> On Mon, 2009-04-06 at 17:01 -0500, Steven Pratt wrote:
>>  
>>> I am continuing to do runs to provide more data on the random write 
>>> issues with btrfs. I have just posted 2 sets of runs here:
>>> http://btrfs.boxacle.net/repository/raid/longrun/
>>>
>>> these are on a pull of the btrfs-unstable experimental branch from 4/3.
>>>
>>> These are 100 minute runs of the 128 thread random write workload on 
>>> the raid system (1 for btrfs and 1 for ext3).  Included in these 
>>> runs are graphs of all the iostat, sar and mpstat data (see analysis 
>>> directories).
>>>
>>> A couple of interesting things. First, we see the choppiness of the 
>>> IO in btrfs compared to ext3.
>>> http://btrfs.boxacle.net/repository/raid/longrun/btrfs-longrun/btrfs1.ffsb.random_writes__threads_0128.09-04-06_10.25.03/analysis/iostat-processed.001/chart.html 
>>>
>>> http://btrfs.boxacle.net/repository/raid/longrun/ext3-longrun/btrfs1.ffsb.random_writes__threads_0128.09-04-06_13.44.49/analysis/iostat-processed.001/chart.html 
>>>
>>>
>>> In particular look at graphs 7 and 11 which show write iops and 
>>> throughput.  Ext3 is nice and smooth, while btrfs has a repeating 
>>> pattern of dips and spikes, with IO going to 0 on  a regular basis.
>>>
>>>     
>>
>> The dips and spikes may be from the allocator.  Basically what happens
>> is after each commit we end up with a bunch of small blocks available
>> for filling again.  Could you please try with -o ssd?
>>
>>   
> Will give it a shot.

Results with -o ssd were not much different. 2.85MB/sec vs 2.5MB/sec. 
Also, the spiky behavior still exists.  All 3 runs at:
http://btrfs.boxacle.net/repository/raid/longrun/

Also, finally have the blktrace runs you wanted.  A 128thread odirect 
random write workload is tarred up at:
http://btrfs.boxacle.net/repository/raid/blktrcrun.tar.bz

blktrace is inteh analysis/blktrace.001 dir.

Steve

>>  
>>> Another interesting observation is what looks a lot like a memory 
>>> leak.  Looking at chart 6 Memory at :
>>> http://btrfs.boxacle.net/repository/raid/longrun/btrfs-longrun/btrfs1.ffsb.random_writes__threads_0128.09-04-06_10.25.03/analysis/sar-processed.001/chart.html 
>>>
>>> we see that the amount of page cache drops slowly throughout the 
>>> entire run.  Starting up around 3.5GB and dropping to about 2.3GB by 
>>> the end of the run.  The memory seems to have moved to the slab 
>>> which grew to 1.5GB.  Doing a repeat of the run while watching 
>>> slabtop, we see that size-2048 is responsible for the majority of 
>>> the slab usage (over 1GB).
>>>
>>>     
>>
>> size-2048?  That's probably the csums.  I'll give it a shot when I get
>> back next week
>
> Ok.
>
> One other thing I noticed that is really bad.  For ext3, we see 
> 115MB/sec both from the benchmark reporting and from iostat write 
> throughput.  However, for btrfs, we see a benchmark throughput of 
> 2.5MB/sec while iostat shows a whopping 35MB/sec of writes.  That to 
> me implies that btrfs is doing an additional 32-33MB/sec of metadata 
> or journal writes.  More than 10x the amount of actual data being 
> written.  Can that be right?
>
> Steve
>> .
>>
>> -chris
>>
>>   
>
> -- 
> To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

New performance results

[Steven Pratt]

Well, it's been a while but been very busy on other things. Thanks to 
Keith Mannthey for helping get some of these runs done.

New raid performance on 2.6.34-rc3:

http://btrfs.boxacle.net/repository/raid/2010-04-14_2004/2.6.34-rc3/2.6.34-rc3.html

Also, the long running history graphs are here:

http://btrfs.boxacle.net/repository/raid/history/History.html


Steve