XFS drops create/delete files to 6.6% of EXT3 (software raid) and to 0.6% of EXT3 (3ware hardware raid)

XFS drops create/delete files to 6.6% of EXT3 (software raid) and to 0.6% of EXT3 (3ware hardware raid)

[Michael Darling]

I am preparing a new server, and benchmarking EXT3 against XFS, both using
software RAID and hardware RAID using a 3ware 9650SE-4LPML.

Using bonnie++ as a benchmark, I am seeing significant performance boosts in
my block sequential reads and writes moving from EXT3 to XFS.  I am aware
that XFS won't create and delete files as quickly as EXT3, however I am
seeing drops from 29455/second to 1957/second using software RAID, and from
32524/second to 189/second using hardware RAID.  I'm not sure if when using
software raid, if creating and deleting files should drop to 6.6% of EXT3.
But, what I'm pretty sure of, is when using hardware raid, that creating and
deleting files shouldn't drop to 0.6% of EXT3.

When using the 3ware card, mkfs.xfs defaulted to "data sunit=0 swidth=0
blks", which made me think that might be the problem, so I tried "-d
sunit=128,swidth=384" with no effect.  My 3ware card is using 64k stripes,
so I calculated 64k stripes / 512 bytes = sunit 128, and multiplied by 3
drives (3 usable in a 4 drive raid 5) to get swidth 384.  This is my first
time calculating these values, so I'm not sure if I did this right.
Unfortunately, these settings had no performance change.

If anyone could give me some pointers, I would much appreciate it!

BONNIE++ - SOFTWARE RAID 5 - 4 DRIVES - SETRA 16384 - EXT3
-----------------------------------------------------------------------------------------------------

"/sbin/mdadm --create /dev/md0 --verbose --level=raid5 --raid-devices=4
/dev/sd{b,c,d,e}"
"/sbin/mke2fs -j /dev/md0"
"mount /dev/md0 /newraid"
"chmod 777 /newraid"
"bonnie++ -d /newraid" (as non-root)

 Version  1.03       ------Sequential Output------ --Sequential Input-
--Random-
                    -Per Chr- --Block-- -Rewrite- -Per Chr- --Block--
--Seeks--
Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec
%CP
servo.runtimecc. 8G 54852  96 116904  37 52572  15 63851  96 189073  25
352.8   1
                    ------Sequential Create------ --------Random
Create--------
                    -Create-- --Read--- -Delete-- -Create-- --Read---
-Delete--
              files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec
%CP
                 16 29455  89 +++++ +++ +++++ +++ +++++ +++ +++++ +++ +++++
+++

BONNIE++ - SOFTWARE RAID 5 - 4 DRIVES - SETRA 16384 - XFS
---------------------------------------------------------------------------------------------------
"/sbin/mdadm --create /dev/md0 --verbose --level=raid5 --raid-devices=4
/dev/sd{b,c,d,e}"
"/sbin/mkfs.xfs -i size=512 /dev/md0"
"mount /dev/md0 /newraid"
"chmod 777 /newraid"
"bonnie++ -d /newraid" (as non-root)

Version  1.03       ------Sequential Output------ --Sequential Input-
--Random-
                    -Per Chr- --Block-- -Rewrite- -Per Chr- --Block--
--Seeks--
Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec
%CP
servo.runtimecc. 8G 61281  97 146610  32 48509  17 57977  89 180168  26
479.4   0
                    ------Sequential Create------ --------Random
Create--------
                    -Create-- --Read--- -Delete-- -Create-- --Read---
-Delete--
              files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec
%CP
                 16  1597  11 +++++ +++  1348   7  1398  10 +++++ +++
954   5

BONNIE++ - 9650SE HARDWARE RAID 5 - 4 DRIVES - SETRA 16384 - EXT3
------------------------------------------------------------------------------------------------------------------

(Using default 9650SE values, except turning write cache on and setting
StorSave to balance)
(I tried max_sectors_kb=64 and nr_requests=512 alone and together, and
performance went down)
"/sbin/mke2fs -j /dev/sdb1"
"mount /dev/sdb1 /newraid"
"chmod 777 /newraid"
"bonnie++ -d /newraid" (as non-root)

Version  1.03       ------Sequential Output------ --Sequential Input-
--Random-
                    -Per Chr- --Block-- -Rewrite- -Per Chr- --Block--
--Seeks--
Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec
%CP
servo.runtimecc. 8G 53483  93 108314  31 57674  12 62134  91 227582  19
329.9   0
                    ------Sequential Create------ --------Random
Create--------
                    -Create-- --Read--- -Delete-- -Create-- --Read---
-Delete--
              files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec
%CP
                 16 32524  95 +++++ +++ +++++ +++ +++++ +++ +++++ +++ +++++
+++

BONNIE++ - 9650SE HARDWARE RAID 5 - 4 DRIVES - SETRA 16384 - XFS
-----------------------------------------------------------------------------------------------------------------

(Using default 9650SE values, except turning write cache on and setting
StorSave to balance)
(I tried max_sectors_kb=64 and nr_requests=512 alone and together, and
performance went down)
"/sbin/mkfs.xfs -i size=512 /dev/sdb1"
meta-data=/dev/sdb1              isize=512    agcount=32, agsize=11443802
blks
         =                       sectsz=512   attr=0
data     =                       bsize=4096   blocks=366201664, imaxpct=25
         =                       sunit=0      swidth=0 blks, unwritten=1
naming   =version 2              bsize=4096
log      =internal log           bsize=4096   blocks=32768, version=1
         =                       sectsz=512   sunit=0 blks, lazy-count=0
realtime =none                   extsz=4096   blocks=0, rtextents=0

sunit=swidth=0 worried me, so I tried "/sbin/mkfs.xfs -i size=512 -d
sunit=128,swidth=384 /dev/sdb1" and got within 1-2% of the results below.
"mount /dev/sdb1 /newraid"
"chmod 777 /newraid"
"bonnie++ -d /newraid" (as non-root)

Version  1.03       ------Sequential Output------ --Sequential Input-
--Random-
                    -Per Chr- --Block-- -Rewrite- -Per Chr- --Block--
--Seeks--
Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec
%CP
servo.runtimecc. 8G 62518  99 211622  33 78473  12 68182  99 214218  15
484.6   0
                    ------Sequential Create------ --------Random
Create--------
                    -Create-- --Read--- -Delete-- -Create-- --Read---
-Delete--
              files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec
%CP
                 16   189   1 +++++ +++   172   0   186   1 +++++ +++
122   0


[[HTML alternate version deleted]]

Re: XFS drops create/delete files to 6.6% of EXT3 (software raid) and to 0.6% of EXT3 (3ware hardware raid)

[Eric Sandeen]

Michael Darling wrote:
> I am preparing a new server, and benchmarking EXT3 against XFS, both using
> software RAID and hardware RAID using a 3ware 9650SE-4LPML.
> 
> Using bonnie++ as a benchmark, I am seeing significant performance boosts in
> my block sequential reads and writes moving from EXT3 to XFS.  I am aware
> that XFS won't create and delete files as quickly as EXT3, however I am
> seeing drops from 29455/second to 1957/second using software RAID, and from
> 32524/second to 189/second using hardware RAID.  I'm not sure if when using
> software raid, if creating and deleting files should drop to 6.6% of EXT3.
> But, what I'm pretty sure of, is when using hardware raid, that creating and
> deleting files shouldn't drop to 0.6% of EXT3.

So I played with this a little on 2.6.25, on plain partitions.

I saw similar numbers; for example, sequential creates:

ext3:		23698/s
xfs:		  319/s
xfs,nobarrier:	 4478/s


then on a whim I tried an external log on a devicemapper zero target :)

xfs,fakelog:	15156/s

Others can probably wax eloquent as to why xfs is slower on this test,
but it's not unique to your setup, at least.

-Eric

Re: XFS drops create/delete files to 6.6% of EXT3 (software raid) and to 0.6% of EXT3 (3ware hardware raid)

[David Chinner]

On Wed, Apr 23, 2008 at 11:01:51PM -0500, Eric Sandeen wrote:
> Michael Darling wrote:
> > I am preparing a new server, and benchmarking EXT3 against XFS, both using
> > software RAID and hardware RAID using a 3ware 9650SE-4LPML.
> > 
> > Using bonnie++ as a benchmark, I am seeing significant performance boosts in
> > my block sequential reads and writes moving from EXT3 to XFS.  I am aware
> > that XFS won't create and delete files as quickly as EXT3, however I am
> > seeing drops from 29455/second to 1957/second using software RAID, and from
> > 32524/second to 189/second using hardware RAID.  I'm not sure if when using
> > software raid, if creating and deleting files should drop to 6.6% of EXT3.
> > But, what I'm pretty sure of, is when using hardware raid, that creating and
> > deleting files shouldn't drop to 0.6% of EXT3.
> 
> So I played with this a little on 2.6.25, on plain partitions.
> 
> I saw similar numbers; for example, sequential creates:
> 
> ext3:		23698/s
> xfs:		  319/s
> xfs,nobarrier:	 4478/s

And with logbsize=256k?

Cheers,

Dave.
-- 
Dave Chinner
Principal Engineer
SGI Australian Software Group

Re: XFS drops create/delete files to 6.6% of EXT3 (software raid) and to 0.6% of EXT3 (3ware hardware raid)

[Michael Darling]

Eric, for the numbers you provided, are you using a single drive, software
raid, or hardware raid?  If a hardware raid, is it a 3ware card?

I hadn't seen the nobarrier mount option before.  Using that changes
sequential creates from about 190/second to about 2500/second, and changes
sequential deletes from about 170/second to about 3700/second.

I don't yet have a BBU for the 3ware card, but would certainly get one if we
go with the 3ware card before we start putting real data on the raid.  Am I
right that with a BBU unit and a battery backup for the server as a whole,
that nobarrier would be safe to use?

Not using nobarrier, but using logbsize 256k changes sequential creates from
about 190/second to about 270/second.  So, it's an improvement, but no where
near where a software raid performs (1600/second) or where the hardware raid
performs with nobarrier.


On Thu, Apr 24, 2008 at 1:33 AM, David Chinner <dgc@sgi.com> wrote:

> On Wed, Apr 23, 2008 at 11:01:51PM -0500, Eric Sandeen wrote:
> > Michael Darling wrote:
> > > I am preparing a new server, and benchmarking EXT3 against XFS, both
> using
> > > software RAID and hardware RAID using a 3ware 9650SE-4LPML.
> > >
> > > Using bonnie++ as a benchmark, I am seeing significant performance
> boosts in
> > > my block sequential reads and writes moving from EXT3 to XFS.  I am
> aware
> > > that XFS won't create and delete files as quickly as EXT3, however I
> am
> > > seeing drops from 29455/second to 1957/second using software RAID, and
> from
> > > 32524/second to 189/second using hardware RAID.  I'm not sure if when
> using
> > > software raid, if creating and deleting files should drop to 6.6% of
> EXT3.
> > > But, what I'm pretty sure of, is when using hardware raid, that
> creating and
> > > deleting files shouldn't drop to 0.6% of EXT3.
> >
> > So I played with this a little on 2.6.25, on plain partitions.
> >
> > I saw similar numbers; for example, sequential creates:
> >
> > ext3:         23698/s
> > xfs:            319/s
> > xfs,nobarrier:         4478/s
>
> And with logbsize=256k?
>
> Cheers,
>
> Dave.
> --
> Dave Chinner
> Principal Engineer
> SGI Australian Software Group
>


[[HTML alternate version deleted]]

Re: XFS drops create/delete files to 6.6% of EXT3 (software raid) and to 0.6% of EXT3 (3ware hardware raid)

[David Chinner]

On Thu, Apr 24, 2008 at 02:23:45AM -0400, Michael Darling wrote:
> Eric, for the numbers you provided, are you using a single drive, software
> raid, or hardware raid?  If a hardware raid, is it a 3ware card?
> 
> I hadn't seen the nobarrier mount option before.  Using that changes
> sequential creates from about 190/second to about 2500/second, and changes
> sequential deletes from about 170/second to about 3700/second.
> 
> I don't yet have a BBU for the 3ware card, but would certainly get one if we
> go with the 3ware card before we start putting real data on the raid.  Am I
> right that with a BBU unit and a battery backup for the server as a whole,
> that nobarrier would be safe to use?

Are you using write caching on the RAID card? If yes, then you need a BBU
for the card if you want to enable the write cache safely.

> Not using nobarrier, but using logbsize 256k changes sequential creates from
> about 190/second to about 270/second.  So, it's an improvement, but no where
> near where a software raid performs (1600/second) or where the hardware raid
> performs with nobarrier.

Sounds like the hardware RAID is doing a full cache flush on barrier I/Os
which will cause lots and lots RMW cycles in the back end. i.e. slow.

Software raid ignores barriers, so is unsafe with any write cache turned
on (and that goes for any filesystem, not just XFS).

Cheers,

Dave.
-- 
Dave Chinner
Principal Engineer
SGI Australian Software Group

Re: XFS drops create/delete files to 6.6% of EXT3 (software raid) and to 0.6% of EXT3 (3ware hardware raid)

[Michael Darling]

On Thu, Apr 24, 2008 at 2:31 AM, David Chinner <dgc@sgi.com> wrote:
> Are you using write caching on the RAID card? If yes, then you need a BBU
> for the card if you want to enable the write cache safely.

I am using write caching.  Without write caching, the 3ware
9650SE-4LMPL crawls along at 5MB/sec writing.  3ware tech support said
awful write performance without write caching to a RAID 5 was typical,
that write caching really needs to be on for a RAID 5.

Is nobarrier safe to use, as long as before putting real data on the
RAID I obtain a BBU for the 3ware card, in addition to the existing
battery backup for the server as a whole?


On Thu, Apr 24, 2008 at 10:20 AM, Justin Piszcz <jpiszcz@lucidpixels.com> wrote:
> Which program is this that you are using to measure this, bonnie++, or?

I'm also using bonnie++, v1.03 from http://www.coker.com.au/bonnie++/

Re: XFS drops create/delete files to 6.6% of EXT3 (software raid) and to 0.6% of EXT3 (3ware hardware raid)

[Eric Sandeen]

Michael Darling wrote:
> Eric, for the numbers you provided, are you using a single drive,
> software raid, or hardware raid?  If a hardware raid, is it a 3ware card?

it's a simple partition on a sata drive.

> I hadn't seen the nobarrier mount option before.  Using that changes
> sequential creates from about 190/second to about 2500/second, and
> changes sequential deletes from about 170/second to about 3700/second.

I wasn't recommending, necessarily, that you use it, but was just
looking for bottlenecks...

> I don't yet have a BBU for the 3ware card, but would certainly get one
> if we go with the 3ware card before we start putting real data on the
> raid.  Am I right that with a BBU unit and a battery backup for the
> server as a whole, that nobarrier would be safe to use?
>  
> Not using nobarrier, but using logbsize 256k changes sequential creates
> from about 190/second to about 270/second.  So, it's an improvement, but
> no where near where a software raid performs (1600/second) or where the
> hardware raid performs with nobarrier.

Default mkfs & mount w/ 256k logbufs:

xfs,256k logbuf:	1353/s

> On Thu, Apr 24, 2008 at 1:33 AM, David Chinner <dgc@sgi.com
> <mailto:dgc@sgi.com>> wrote:
>     > So I played with this a little on 2.6.25, on plain partitions.
>     >
>     > I saw similar numbers; for example, sequential creates:
>     >
>     > ext3:         23698/s
>     > xfs:            319/s
>     > xfs,nobarrier:         4478/s
> 
>     And with logbsize=256k?
> 
>     Cheers,
> 
>     Dave.

Re: XFS drops create/delete files to 6.6% of EXT3 (software raid) and to 0.6% of EXT3 (3ware hardware raid)

[Justin Piszcz]

On Thu, 24 Apr 2008, Eric Sandeen wrote:

> Michael Darling wrote:
>> Eric, for the numbers you provided, are you using a single drive,
>> software raid, or hardware raid?  If a hardware raid, is it a 3ware card?
>
> it's a simple partition on a sata drive.
>
>> I hadn't seen the nobarrier mount option before.  Using that changes
>> sequential creates from about 190/second to about 2500/second, and
>> changes sequential deletes from about 170/second to about 3700/second.
>
> I wasn't recommending, necessarily, that you use it, but was just
> looking for bottlenecks...
>
>> I don't yet have a BBU for the 3ware card, but would certainly get one
>> if we go with the 3ware card before we start putting real data on the
>> raid.  Am I right that with a BBU unit and a battery backup for the
>> server as a whole, that nobarrier would be safe to use?
>>
>> Not using nobarrier, but using logbsize 256k changes sequential creates
>> from about 190/second to about 270/second.  So, it's an improvement, but
>> no where near where a software raid performs (1600/second) or where the
>> hardware raid performs with nobarrier.
>
> Default mkfs & mount w/ 256k logbufs:
>
> xfs,256k logbuf:	1353/s
>
>> On Thu, Apr 24, 2008 at 1:33 AM, David Chinner <dgc@sgi.com
>> <mailto:dgc@sgi.com>> wrote:
>>    > So I played with this a little on 2.6.25, on plain partitions.
>>    >
>>    > I saw similar numbers; for example, sequential creates:
>>    >
>>    > ext3:         23698/s
>>    > xfs:            319/s
>>    > xfs,nobarrier:         4478/s
>>
>>     And with logbsize=256k?
>>
>>     Cheers,
>>
>>     Dave.
>
>

Which program is this that you are using to measure this, bonnie++, or?

Re: XFS drops create/delete files to 6.6% of EXT3 (software raid) and to 0.6% of EXT3 (3ware hardware raid)

[Eric Sandeen]

Justin Piszcz wrote:

> Which program is this that you are using to measure this, bonnie++, or?

yes, bonnie++

-Eric

Re: XFS drops create/delete files to 6.6% of EXT3 (software raid) and to 0.6% of EXT3 (3ware hardware raid)

[Eric Sandeen]

Michael Darling wrote:
> I am preparing a new server, and benchmarking EXT3 against XFS, both using
> software RAID and hardware RAID using a 3ware 9650SE-4LPML.
> 
> Using bonnie++ as a benchmark, I am seeing significant performance boosts in
> my block sequential reads and writes moving from EXT3 to XFS.  I am aware
> that XFS won't create and delete files as quickly as EXT3, however I am
> seeing drops from 29455/second to 1957/second using software RAID, and from
> 32524/second to 189/second using hardware RAID.  I'm not sure if when using
> software raid, if creating and deleting files should drop to 6.6% of EXT3.
> But, what I'm pretty sure of, is when using hardware raid, that creating and
> deleting files shouldn't drop to 0.6% of EXT3.

Chris Mason reminded me that bonnie++ may not be creating enough files
to actually cause much if any IO on ext3; you might play with the -n
option to get it to the point where it's actually doing a significant
number of creates (I don't know what the default is offhand... checks
source... it's 16k maybe?) but more sustained creation might be interesting.

-Eric

Re: XFS drops create/delete files to 6.6% of EXT3 (software raid) and to 0.6% of EXT3 (3ware hardware raid)

[Michael Darling]

On Thu, Apr 24, 2008 at 1:18 PM, Eric Sandeen <sandeen@sandeen.net> wrote:
> Chris Mason reminded me that bonnie++ may not be creating enough files
> to actually cause much if any IO on ext3; you might play with the -n
> option to get it to the point where it's actually doing a significant
> number of creates (I don't know what the default is offhand... checks
> source... it's 16k maybe?) but more sustained creation might be
interesting.
>
> -Eric
>

That's definately a good point worth testing, the default 16k option
completes extremely quickly.  Increasing the number of files affected the
ext3 benchmarks by only a little bit below 256k.  I think the 128k benchmark
can be treated as representative especially since it puts the ext3
filesystem through a load for 23.67 seconds.  I think the 256k benchmark is
hitting a problem in the ext3 filesystem, because CPU usage is bottoming out
and correspondingly so the transactions per second is bottoming out as well.

Regardless, no matter which benchmark below is representative of the ext3
filesystem, it's still substantially faster than xfs on my 3ware
9650SE-4LPML (at least with barriers on.)

 _______------Sequential Create------ --------Random Create--------
_______-Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
___16 31771_ 95 +++++ +++ +++++ +++ +++++ +++ +++++ +++ +++++ +++ (2.09
seconds)
___64 31544_ 97 +++++ +++ 52547_ 99 17424_ 53 +++++ +++ 53954 100 (8.76
seconds)
__128 30287_ 97 +++++ +++ 48493_ 95_ 9987_ 32 +++++ +++ 49988_ 99 (23.67
seconds)
__256 14709_ 49 323792 99_ 4211__ 8 14866_ 49 416920 99_ 1580__ 3 (4 minutes
27.46 seconds)


[[HTML alternate version deleted]]

Re: XFS drops create/delete files to 6.6% of EXT3 (software raid) and to 0.6% of EXT3 (3ware hardware raid)

[Emmanuel Florac]

Le Wed, 23 Apr 2008 21:48:53 -0400 vous écriviez:

> If anyone could give me some pointers, I would much appreciate it!

You may greatly enhance performance on hardware RAID by using the
"nobarrier" mount option. However in this case you must have proper
power and/or optional 3Ware BBU (Battery backup unit). The BBU costs
around 60 euros so I'd take it anyway ;)
XFS is very (and sanely) touchy about the atomicity of logging
operations. Each delete action implies several small seeks and writes
( at the very least one to the data volume and one to the log) and that
hinders performance. "nobarrier" removes the enforcement to sync'ed
disk writes to the log, however this is dangerous if you're having
power supply problems.
Another option wich gave very satisfactory results to me is to use an
ultra-fast small SSD drive to store the filesystem log. However a single
professional SSD is as expensive as a 3Ware 9650 + several high
capacity SATA drives, but the combination is mostly comparable in
performance to a SAS array of lesser capacity and much higher power
consumption.

-- 
--------------------------------------------------
Emmanuel Florac               www.intellique.com   
--------------------------------------------------