bonnie++ benchmarks for ext2,ext3,ext4,jfs,reiserfs,xfs,zfs on software raid 5

bonnie++ benchmarks for ext2,ext3,ext4,jfs,reiserfs,xfs,zfs on software raid 5

[Justin Piszcz]

CONFIG:

Software RAID 5 (400GB x 6): Default mkfs parameters for all filesystems.
Kernel was 2.6.21 or 2.6.22, did these awhile ago.
Hardware was SATA with PCI-e only, nothing on the PCI bus.

ZFS was userspace+fuse of course.
Reiser was V3.
EXT4 was created using the recommended options on its project page.

RAW:

ext2,7760M,56728,96.3333,180505,51,85484,17.3333,50946.7,80.3333,235541,21.3333,373.667,0,16:100000:16/64,2354,27,0,0,8455.67,14.6667,2211.67,26.3333,0,0,9724,22.3333
ext3,7760M,52702.7,94.3333,165005,60,82294.7,20.6667,52664,83.6667,258788,33.3333,335.8,0,16:100000:16/64,858.333,10.6667,10250.3,28.6667,4084,15,897,12.6667,4024.33,12.3333,2754,11.3333
ext4,7760M,53129.7,95,164515,59.3333,101678,31.6667,62194.3,98.6667,266716,22.3333,405.767,0,16:100000:16/64,1963.67,23.6667,0,0,20859,73.6667,1731,21.3333,9022,23.6667,16410,65.6667
jfs,7760M,54606,92,191997,52,112764,33.6667,63585.3,99,274921,22.3333,383.8,0,16:100000:16/64,344,1,0,0,539.667,0,297.667,1,0,0,340,0
reiserfs,7760M,51056.7,96,180607,67,106907,38.3333,61231.3,97.6667,275339,29.3333,441.167,0,16:100000:16/64,2516,60.6667,19174.3,60.6667,8194.33,54.3333,2011,42.6667,6963.67,19.6667,9168.33,68.6667
xfs,7760M,52985.7,93,158342,45,79682,14,60547.3,98,239101,20.3333,359.667,0,16:100000:16/64,415,4,0,0,1774.67,10.6667,454,4.66667,14526.3,40,1572,12.6667
zfs,7760M,25601,43.3333,32198.7,4,13266.3,2,44145.3,68.6667,129278,9,245.167,0,16:100000:16/64,218.333,2,2698.33,11.6667,7434.67,14.3333,244,2,2191.33,11.6667,5613.33,13.3333

HTML

http://home.comcast.net/~jpiszcz/benchmark/allfs.html

THOUGHTS

Overall JFS seems the fastest but reviewing the mailing list for JFS it 
seems like there a lot of problems, especially when people who use JFS > 1 
year, their speed goes to 5 MiB/s over time and the defragfs tool has been 
removed(?) from the source/Makefile and on Google it says not to use it 
due to corruption.

Justin.

Re: bonnie++ benchmarks for ext2,ext3,ext4,jfs,reiserfs,xfs,zfs on software raid 5

[Al Boldi]

Justin Piszcz wrote:
> CONFIG:
>
> Software RAID 5 (400GB x 6): Default mkfs parameters for all filesystems.
> Kernel was 2.6.21 or 2.6.22, did these awhile ago.
> Hardware was SATA with PCI-e only, nothing on the PCI bus.
>
> ZFS was userspace+fuse of course.

Wow! Userspace and still that efficient.

> Reiser was V3.
> EXT4 was created using the recommended options on its project page.
>
> RAW:
>
> ext2,7760M,56728,96.3333,180505,51,85484,17.3333,50946.7,80.3333,235541,21
>.3333,373.667,0,16:100000:16/64,2354,27,0,0,8455.67,14.6667,2211.67,26.3333
>,0,0,9724,22.3333
> ext3,7760M,52702.7,94.3333,165005,60,82294.7,20.6667,52664,83.6667,258788,
>33.3333,335.8,0,16:100000:16/64,858.333,10.6667,10250.3,28.6667,4084,15,897
>,12.6667,4024.33,12.3333,2754,11.3333
> ext4,7760M,53129.7,95,164515,59.3333,101678,31.6667,62194.3,98.6667,266716
>,22.3333,405.767,0,16:100000:16/64,1963.67,23.6667,0,0,20859,73.6667,1731,2
>1.3333,9022,23.6667,16410,65.6667
> jfs,7760M,54606,92,191997,52,112764,33.6667,63585.3,99,274921,22.3333,383.
>8,0,16:100000:16/64,344,1,0,0,539.667,0,297.667,1,0,0,340,0
> reiserfs,7760M,51056.7,96,180607,67,106907,38.3333,61231.3,97.6667,275339,
>29.3333,441.167,0,16:100000:16/64,2516,60.6667,19174.3,60.6667,8194.33,54.3
>333,2011,42.6667,6963.67,19.6667,9168.33,68.6667
> xfs,7760M,52985.7,93,158342,45,79682,14,60547.3,98,239101,20.3333,359.667,
>0,16:100000:16/64,415,4,0,0,1774.67,10.6667,454,4.66667,14526.3,40,1572,12.
>6667

> zfs,7760M,

Dissecting some of these numbers:

  speed	%cpu  
> 25601,43.3333,
> 32198.7,4,
> 13266.3, 2,
> 44145.3,68.6667,
> 129278,9,
> 245.167,0,

> 16:100000:16/64,

  speed	%cpu  
> 218.333,2,
> 2698.33,11.6667,
> 7434.67,14.3333,
> 244,2,
> 2191.33,11.6667,
> 5613.33,13.3333

Extrapolating these %cpu number makes ZFS the fastest.

Are you sure these numbers are correct?


Thanks!

--
Al

Re: bonnie++ benchmarks for ext2,ext3,ext4,jfs,reiserfs,xfs,zfs on software raid 5

[Miklos Szeredi]

> Extrapolating these %cpu number makes ZFS the fastest.
> 
> Are you sure these numbers are correct?

Note, that %cpu numbers for fuse filesystems are inherently skewed,
because the CPU usage of the filesystem process itself is not taken
into account.

So the numbers are not all that good, but according to the zfs-fuse
author it hasn't been optimized yet, so they may improve.

Miklos

Re: bonnie++ benchmarks for ext2,ext3,ext4,jfs,reiserfs,xfs,zfs on software raid 5

[Dave Kleikamp]

On Mon, 2007-07-30 at 10:29 -0400, Justin Piszcz wrote:

> Overall JFS seems the fastest but reviewing the mailing list for JFS it 
> seems like there a lot of problems, especially when people who use JFS > 1 
> year, their speed goes to 5 MiB/s over time and the defragfs tool has been 
> removed(?) from the source/Makefile and on Google it says not to use it 
> due to corruption.

The defragfs tool was an unported holdover from OS/2, which is why it
was removed.  There never was a working Linux version.  I have some
ideas to improve jfs allocation to avoid fragmentation problems, but jfs
isn't my full-time job anymore, so I can't promise anything.  I'm not
sure about the corruption claims.  I'd like to hear some specifics on
that.

Anyway, for enterprise use, I couldn't recommend jfs, since there is no
full-time maintainer.

Thanks,
Shaggy
-- 
David Kleikamp
IBM Linux Technology Center

Re: bonnie++ benchmarks for ext2,ext3,ext4,jfs,reiserfs,xfs,zfs on software raid 5

[Justin Piszcz]

On Mon, 30 Jul 2007, Miklos Szeredi wrote:

>> Extrapolating these %cpu number makes ZFS the fastest.
>>
>> Are you sure these numbers are correct?
>
> Note, that %cpu numbers for fuse filesystems are inherently skewed,
> because the CPU usage of the filesystem process itself is not taken
> into account.
>
> So the numbers are not all that good, but according to the zfs-fuse
> author it hasn't been optimized yet, so they may improve.
>
> Miklos
> -
> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>

This was performed on an E6300, 1 core was ZFS/FUSE (or quite a bit of it 
anyway)

Re: bonnie++ benchmarks for ext2,ext3,ext4,jfs,reiserfs,xfs,zfs on software raid 5

[Theodore Tso]

On Mon, Jul 30, 2007 at 09:39:39PM +0200, Miklos Szeredi wrote:
> > Extrapolating these %cpu number makes ZFS the fastest.
> > 
> > Are you sure these numbers are correct?
> 
> Note, that %cpu numbers for fuse filesystems are inherently skewed,
> because the CPU usage of the filesystem process itself is not taken
> into account.
> 
> So the numbers are not all that good, but according to the zfs-fuse
> author it hasn't been optimized yet, so they may improve.

Also, something which is data i/o intensive is going to be the best
case for a FUSE filesystem.  If you try something which is much more
metadata intensive (i.e., lots of file creates and deletes, chmods,
etc.) like say with a Postmark benchmark, you would almost certainly
get very different results.  That's not to say that bonnie++
benchmarks aren't useful, but when doing comparisons between
filesystems, it's a good idea to use a wide variety of benchmarks to
avoid getting potentially misleading results.

					- Ted