File System Performance results

File System Performance results

[Steven Pratt]

We have set up a new page which is intended mainly for tracking the 
performance of BTRFS, but in doing so we are testing other filesystems 
as well (ext3, ext4, xfs and jfs).  Thought some people here might find 
the results useful.


The main page is here:

http://btrfs.boxacle.net/

Information about the machine configuration, tests run, how to reproduce 
the run and link to graphs of all the results are provided off of this 
page.  When looking at any individual test, links are provided to the 
detail output from the tests including iostat, mpstat, oprofile data and 
more.

Feedback and request for additional tests are always welcomed.

Steve

Re: File System Performance results

[Andreas Dilger]

On Oct 22, 2008  15:06 -0500, Steven Pratt wrote:
> We have set up a new page which is intended mainly for tracking the  
> performance of BTRFS, but in doing so we are testing other filesystems  
> as well (ext3, ext4, xfs and jfs).  Thought some people here might find  
> the results useful.
>
>
> The main page is here:
>
> http://btrfs.boxacle.net/
>
> Information about the machine configuration, tests run, how to reproduce  
> the run and link to graphs of all the results are provided off of this  
> page.  When looking at any individual test, links are provided to the  
> detail output from the tests including iostat, mpstat, oprofile data and  
> more.

Steve,
thanks for posting the numbers.  They are definitely interesting.  On
the surface, ext4 is doing quite well overall (yay!), but the important
point to realize is that btrfs is also providing a lot of extra function
under the covers so it isn't necessarily a clear-cut answer on which one
to pick.

The extra CPU cost of btrfs will become increasingly irrelevant in the
future I think.

Cheers, Andreas
--
Andreas Dilger
Sr. Staff Engineer, Lustre Group
Sun Microsystems of Canada, Inc.

Re: File System Performance results

[Steven Pratt]

Andreas Dilger wrote:
> On Oct 22, 2008  15:06 -0500, Steven Pratt wrote:
>   
>> We have set up a new page which is intended mainly for tracking the  
>> performance of BTRFS, but in doing so we are testing other filesystems  
>> as well (ext3, ext4, xfs and jfs).  Thought some people here might find  
>> the results useful.
>>
>>
>> The main page is here:
>>
>> http://btrfs.boxacle.net/
>>
>> Information about the machine configuration, tests run, how to reproduce  
>> the run and link to graphs of all the results are provided off of this  
>> page.  When looking at any individual test, links are provided to the  
>> detail output from the tests including iostat, mpstat, oprofile data and  
>> more.
>>     
>
> Steve,
> thanks for posting the numbers.  They are definitely interesting.  On
> the surface, ext4 is doing quite well overall (yay!),
Yes, that was good news.  Along these lines if there is anything else we 
can do to help out ext4, just let us know.

>  but the important
> point to realize is that btrfs is also providing a lot of extra function
> under the covers so it isn't necessarily a clear-cut answer on which one
> to pick.
>
> The extra CPU cost of btrfs will become increasingly irrelevant in the
> future I think.
>   
While I agree that CPU usage is becoming less and less of an issue, I 
think that at this point in the development cycle of btrfs, we still 
need to take a hard look at any areas where cpu usage is excessive, and 
see if we can keep that to a minimum.  This is the main reason we did 
runs without checksumming, so we could see a better apple to apple 
comparison, not because it is not a useful feature.  It will be very 
interesting to see how much HW checksumming changes this with Nehelam.

Steve

> Cheers, Andreas
> --
> Andreas Dilger
> Sr. Staff Engineer, Lustre Group
> Sun Microsystems of Canada, Inc.
>   

Re: File System Performance results

[Chris Mason]

On Mon, 2008-10-27 at 09:28 -0500, Steven Pratt wrote:
> Andreas Dilger wrote:
> > On Oct 22, 2008  15:06 -0500, Steven Pratt wrote:
> >   
> >> We have set up a new page which is intended mainly for tracking the  
> >> performance of BTRFS, but in doing so we are testing other filesystems  
> >> as well (ext3, ext4, xfs and jfs).  Thought some people here might find  
> >> the results useful.
> >>
> >>
> >> The main page is here:
> >>
> >> http://btrfs.boxacle.net/

I meant to ask if this is a permanent site for the results?  It might
make sense to add a more generic fsperf.boxacle.net page.

> >>
> >> Information about the machine configuration, tests run, how to reproduce  
> >> the run and link to graphs of all the results are provided off of this  
> >> page.  When looking at any individual test, links are provided to the  
> >> detail output from the tests including iostat, mpstat, oprofile data and  
> >> more.
> >>     
> >
> > Steve,
> > thanks for posting the numbers.  They are definitely interesting.  On
> > the surface, ext4 is doing quite well overall (yay!),
> Yes, that was good news.  Along these lines if there is anything else we 
> can do to help out ext4, just let us know.
> 
> >  but the important
> > point to realize is that btrfs is also providing a lot of extra function
> > under the covers so it isn't necessarily a clear-cut answer on which one
> > to pick.
> >
> > The extra CPU cost of btrfs will become increasingly irrelevant in the
> > future I think.
> >   
> While I agree that CPU usage is becoming less and less of an issue, I 
> think that at this point in the development cycle of btrfs, we still 
> need to take a hard look at any areas where cpu usage is excessive, and 
> see if we can keep that to a minimum.

Very true, especially performance results with checksumming off (like
the nodatacow results for random writes).

>   This is the main reason we did 
> runs without checksumming, so we could see a better apple to apple 
> comparison, not because it is not a useful feature.  It will be very 
> interesting to see how much HW checksumming changes this with Nehelam.

See the btrfs crc header file for the #define to enable the hw assist
mode if you're got the hardware that can do it.  In my runs here, it
makes the checksumming free aside from the time spent storing the extra
metadata.

-chris

Re: File System Performance results

[Steven Pratt]

Chris Mason wrote:
> On Mon, 2008-10-27 at 09:28 -0500, Steven Pratt wrote:
>   
>> Andreas Dilger wrote:
>>     
>>> On Oct 22, 2008  15:06 -0500, Steven Pratt wrote:
>>>   
>>>       
>>>> We have set up a new page which is intended mainly for tracking the  
>>>> performance of BTRFS, but in doing so we are testing other filesystems  
>>>> as well (ext3, ext4, xfs and jfs).  Thought some people here might find  
>>>> the results useful.
>>>>
>>>>
>>>> The main page is here:
>>>>
>>>> http://btrfs.boxacle.net/
>>>>         
>
> I meant to ask if this is a permanent site for the results?  It might
> make sense to add a more generic fsperf.boxacle.net page.
>   
We hope the site is permanent, as long as traffic and volume does not 
get too high. I'll see what I can do about the new page.

>   
>>>> Information about the machine configuration, tests run, how to reproduce  
>>>> the run and link to graphs of all the results are provided off of this  
>>>> page.  When looking at any individual test, links are provided to the  
>>>> detail output from the tests including iostat, mpstat, oprofile data and  
>>>> more.
>>>>     
>>>>         
>>> Steve,
>>> thanks for posting the numbers.  They are definitely interesting.  On
>>> the surface, ext4 is doing quite well overall (yay!),
>>>       
>> Yes, that was good news.  Along these lines if there is anything else we 
>> can do to help out ext4, just let us know.
>>
>>     
>>>  but the important
>>> point to realize is that btrfs is also providing a lot of extra function
>>> under the covers so it isn't necessarily a clear-cut answer on which one
>>> to pick.
>>>
>>> The extra CPU cost of btrfs will become increasingly irrelevant in the
>>> future I think.
>>>   
>>>       
>> While I agree that CPU usage is becoming less and less of an issue, I 
>> think that at this point in the development cycle of btrfs, we still 
>> need to take a hard look at any areas where cpu usage is excessive, and 
>> see if we can keep that to a minimum.
>>     
>
> Very true, especially performance results with checksumming off (like
> the nodatacow results for random writes).
>
>   
>>   This is the main reason we did 
>> runs without checksumming, so we could see a better apple to apple 
>> comparison, not because it is not a useful feature.  It will be very 
>> interesting to see how much HW checksumming changes this with Nehelam.
>>     
>
> See the btrfs crc header file for the #define to enable the hw assist
> mode if you're got the hardware that can do it.  In my runs here, it
> makes the checksumming free aside from the time spent storing the extra
> metadata.
>
>   
Great, but you are making me jealous. I've been waiting for my Nehelam 
box for months!

Steve
> -chris
>
>   

Re: File System Performance results

[Theodore Tso]

On Mon, Oct 27, 2008 at 09:28:55AM -0500, Steven Pratt wrote:
>> thanks for posting the numbers.  They are definitely interesting.  On
>> the surface, ext4 is doing quite well overall (yay!),
>
> Yes, that was good news.  Along these lines if there is anything else we  
> can do to help out ext4, just let us know.

Indeed, thanks Steven, for doing this work; it's much appreciated!

Couple of quick questions/suggestions

*) What version of e2fsprogs did you use for your ext4 tests, and what
 if any options did you give to mke2fs when creating the filesystem?

*) For all of the filesystems except for btrfs, where you mentioned
 different mount options in use, is it fair to assume that no mount
 options were specified so the filesystem defaults were used?

*) With the full knowledge that ext4 will probably not do terribly
 well with this suggested change, it would be interesting if you did a
 variant of the mail server benchmark where the benchmark code
 performed an fsync() before closing the new file, and an fsync on the
 containing directory after deleting a file, to simulate what real
 MTA's tend to do in order to assure correct SMTP semantics.  (Some
 won't do the fsync on delete, on the assumption that sending an
 e-mail twice is harmless, where as losing an e-mail is completely
 unacceptable.  In a world where 97% of the mail messages transiting
 the backbone is spam, some might disagree with those sentiments, but
 the SMTP protocol requires, and most MTA's do perform, an fsync() an
 incoming mail message before they acknowledge that they mail message
 has been accepted and they are now taking responsibility for sending
 the mail message on to its final destination. :-)

In a 2-3 days after I get the ext4 patch queue back into shape after
2.6.28-rc2, it would interesting to get a benchmark run of 2.6.28-rc2
plus the ext4 patch queue with and without the akpm_lock_hack mount
option.  I'm not sure how much time/effort it takes to do a complete
set of benchmark runs for a different kernel version and/or mount
option, but there are definitely a number of experiments where it
would be very useful to crank through your benchmark systems, and as
long as it doesn't burden your primary benchmarking objectings overly
much, it would be great to see what those experiments run on your
benchmarking setup would turn up.

Thanks, regards,

						- Ted

Re: File System Performance results

[Steven Pratt]

Theodore Tso wrote:
> On Mon, Oct 27, 2008 at 09:28:55AM -0500, Steven Pratt wrote:
>   
>>> thanks for posting the numbers.  They are definitely interesting.  On
>>> the surface, ext4 is doing quite well overall (yay!),
>>>       
>> Yes, that was good news.  Along these lines if there is anything else we  
>> can do to help out ext4, just let us know.
>>     
>
> Indeed, thanks Steven, for doing this work; it's much appreciated!
>
> Couple of quick questions/suggestions
>
> *) What version of e2fsprogs did you use for your ext4 tests, 
[root@btrfs1 ~]# mkfs.ext4dev -V
mke2fs 1.41.2 (02-Oct-2008)
        Using EXT2FS Library version 1.41.2
[root@btrfs1 
~]#                                                                                         

> and what
>  if any options did you give to mke2fs when creating the filesystem?
>   
Basically none.

'mkfs.ext4dev -F /dev/ffsbdev1'




> *) For all of the filesystems except for btrfs, where you mentioned
>  different mount options in use, is it fair to assume that no mount
>  options were specified so the filesystem defaults were used?
>   
Right, except where noted on btrfs, no mount options were used on any 
filesystem.

> *) With the full knowledge that ext4 will probably not do terribly
>  well with this suggested change, it would be interesting if you did a
>  variant of the mail server benchmark where the benchmark code
>  performed an fsync() before closing the new file, and an fsync on the
>  containing directory after deleting a file, to simulate what real
>  MTA's tend to do in order to assure correct SMTP semantics.  (Some
>  won't do the fsync on delete, on the assumption that sending an
>  e-mail twice is harmless, where as losing an e-mail is completely
>  unacceptable.  In a world where 97% of the mail messages transiting
>  the backbone is spam, some might disagree with those sentiments, but
>  the SMTP protocol requires, and most MTA's do perform, an fsync() an
>  incoming mail message before they acknowledge that they mail message
>  has been accepted and they are now taking responsibility for sending
>  the mail message on to its final destination. :-)
>   
Yes, Chris Mason had already requested this.  Should have it this week.

> In a 2-3 days after I get the ext4 patch queue back into shape after
> 2.6.28-rc2, it would interesting to get a benchmark run of 2.6.28-rc2
> plus the ext4 patch queue with and without the akpm_lock_hack mount
> option.  I'm not sure how much time/effort it takes to do a complete
> set of benchmark runs for a different kernel version and/or mount
> option, but there are definitely a number of experiments where it
> would be very useful to crank through your benchmark systems, and as
> long as it doesn't burden your primary benchmarking objectings overly
> much, it would be great to see what those experiments run on your
> benchmarking setup would turn up.
>   
It is not much effort at all.  Runs take about 3 hours per filesystem 
(for currently defined set of benchmarks) so we can complete full set in 
about 15hours.  It is all automated, so assuming no breakage, about 30 
minutes to set up for a new set of runs, and 30 minutes after all 
complete to verify and graph it all.

If there are specific runs you would like to see, just let me know 
specifics.

Steve
> Thanks, regards,
>
> 						- Ted
>   

Re: File System Performance results

[Chris Mason]

On Wed, 2008-10-22 at 15:06 -0500, Steven Pratt wrote:
> We have set up a new page which is intended mainly for tracking the 
> performance of BTRFS, but in doing so we are testing other filesystems 
> as well (ext3, ext4, xfs and jfs).  Thought some people here might find 
> the results useful.

I think I understand the bad read performance in btrfs.  I was forcing a
tiny max readahead size.

The current git tree has fixes for it, along with a ton of new code.

-chris

Re: File System Performance results

[Steven Pratt]

Chris Mason wrote:
> On Wed, 2008-10-22 at 15:06 -0500, Steven Pratt wrote:
>   
>> We have set up a new page which is intended mainly for tracking the 
>> performance of BTRFS, but in doing so we are testing other filesystems 
>> as well (ext3, ext4, xfs and jfs).  Thought some people here might find 
>> the results useful.
>>     
>
> I think I understand the bad read performance in btrfs.  I was forcing a
> tiny max readahead size.
>
> The current git tree has fixes for it, along with a ton of new code.
>   
Ok, we'll kick off some new runs.

Also, just need to push out the data fro odirect random write and mail 
server with fsync. 

Steve

> -chris
>
>   

Re: File System Performance results

[Steven Pratt]

Steven Pratt wrote:
> Chris Mason wrote:
>> On Wed, 2008-10-22 at 15:06 -0500, Steven Pratt wrote:
>>  
>>> We have set up a new page which is intended mainly for tracking the 
>>> performance of BTRFS, but in doing so we are testing other 
>>> filesystems as well (ext3, ext4, xfs and jfs).  Thought some people 
>>> here might find the results useful.
>>>     
>>
>> I think I understand the bad read performance in btrfs.  I was forcing a
>> tiny max readahead size.
>>
>> The current git tree has fixes for it, along with a ton of new code.
>>   
> Ok, we'll kick off some new runs.
>
> Also, just need to push out the data fro odirect random write and mail 
> server with fsync.
> Steve
We have completed testing on the random write tests with odirect.  
Results can be found at:
Single disk:
http://btrfs.boxacle.net/repository/single-disk/Oct28-odirect-random-writes/Oct28-odirect-random-writes.html
Raid:
http://btrfs.boxacle.net/repository/raid/Oct28-odirect-random-writes/Oct28-odirect-random-writes.html

Also, the changes to the mail server emulation to use fsync on the 
creates is also complete:
Single disk:
http://btrfs.boxacle.net/repository/single-disk/Oct28-mail-server-fsync-creates/Oct28-mail-server-fsync-creates.html
Raid:
http://btrfs.boxacle.net/repository/raid/Oct28-mail-server-fsync-creates/mail-server-fsync-creates.html

Steve

>
>> -chris
>>
>>   
>
> -- 
> To unsubscribe from this list: send the line "unsubscribe 
> linux-fsdevel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html