[linux-lvm] Why the dramatic increase in filesystem performance when usingxfs????

[linux-lvm] Why the dramatic increase in filesystem performance when usingxfs????

[Gary Mansell]

Hi,

I have always run ext3 filesystem with journalling on Redhat AS as it is
the only supported filesystem.

One of my colleagues runs xfs, though, and on comparable hardware
configs he gets twice the performance compared to my ext3 tests.

The test that I perform is to create a file at least twice the size of
the RAM installed in the system to avoid the possibility of cacheing,
measuring the time to write and read the file back gives me the
performance figure that I am after. I realise that this is a very simple
test of large sequential IO but it is good enough for my needs.

ie

Write test:

# time dd if=/dev/zero of=./testfile bs=16384 count=250000 ; time sync

Read test:

# time dd if=./testfile of=/dev/null bs=16384


As the xfs performance comes back about twice the performance of ext3
for this test I am of the opinion that xfs must be cheating somehow. It
has always been my opinion that the IO bottleneck is the hardware and
not the filesystem hence changing the filesystem but using the same
hardware should not make a huge difference to performance (you still
have to get the same amount of data out to disk at the end of the day)

I am struggling to comprehend how xfs can cheat, though, as it can't
cache such a huge file as there is not enough memory. Is it perhaps
cheating because the file is comprised entirely of zero's?

Can someone please enlighten me

Thanks in advance

Gary Mansell
-- 

This e-mail and any files transmitted with it are confidential and intended
solely for the use of the individual or entity to whom they are addressed.
If you have received this e-mail in error please notify the sender immediately
and delete this e-mail from your system.  Please note that any views or opinions
presented in this e-mail are solely those of the author and do not necessarily
represent those of Ricardo (save for reports and other documentation formally
approved and signed for release to the intended recipient).  Only Directors
or Duly Authorised Officers are authorised to enter into legally binding
obligations on behalf of Ricardo unless the obligation is contained within
a Ricardo Purchase Order.

Ricardo may monitor outgoing and incoming e-mails and other telecommunications
on its e-mail and telecommunications systems.  By replying to this e-mail you
give consent to such monitoring.  The recipient should check this e-mail and
any attachments for the presence of viruses.  Ricardo accepts no liability for
any damage caused by any virus transmitted by this e-mail.  "Ricardo" means
Ricardo plc and its subsidiary companies.

Ricardo plc is a public limited company registered in England with registered
number 00222915.
The registered office of Ricardo plc is Bridge Works, Shoreham-by Sea,
West Sussex, BN43 5FG.
 

Re: [linux-lvm] Why the dramatic increase in filesystem performance when usingxfs????

[Greg Freemyer]

On Fri, 21 Jan 2005 10:51:09 +0000, Gary Mansell wrote:
> Hi,
> 
> I have always run ext3 filesystem with journalling on Redhat AS as it is
> the only supported filesystem.
> 
> One of my colleagues runs xfs, though, and on comparable hardware
> configs he gets twice the performance compared to my ext3 tests.
> 
> The test that I perform is to create a file at least twice the size of
> the RAM installed in the system to avoid the possibility of cacheing,
> measuring the time to write and read the file back gives me the
> performance figure that I am after. I realise that this is a very simple
> test of large sequential IO but it is good enough for my needs.
> 
> ie
> 
> Write test:
> 
> # time dd if=/dev/zero of=./testfile bs=16384 count=250000 ; time sync
> 
> Read test:
> 
> # time dd if=./testfile of=/dev/null bs=16384
> 
> As the xfs performance comes back about twice the performance of ext3
> for this test I am of the opinion that xfs must be cheating somehow. It
> has always been my opinion that the IO bottleneck is the hardware and
> not the filesystem hence changing the filesystem but using the same
> hardware should not make a huge difference to performance (you still
> have to get the same amount of data out to disk at the end of the day)
> 
> I am struggling to comprehend how xfs can cheat, though, as it can't
> cache such a huge file as there is not enough memory. Is it perhaps
> cheating because the file is comprised entirely of zero's?
> 
> Can someone please enlighten me
> 
> Thanks in advance
> 
> Gary Mansell
> --
> 
> This e-mail and any files transmitted with it are confidential and intended
> solely for the use of the individual or entity to whom they are addressed.
> If you have received this e-mail in error please notify the sender immediately
> and delete this e-mail from your system.  Please note that any views or opinions
> presented in this e-mail are solely those of the author and do not necessarily
> represent those of Ricardo (save for reports and other documentation formally
> approved and signed for release to the intended recipient).  Only Directors
> or Duly Authorised Officers are authorised to enter into legally binding
> obligations on behalf of Ricardo unless the obligation is contained within
> a Ricardo Purchase Order.
> 
> Ricardo may monitor outgoing and incoming e-mails and other telecommunications
> on its e-mail and telecommunications systems.  By replying to this e-mail you
> give consent to such monitoring.  The recipient should check this e-mail and
> any attachments for the presence of viruses.  Ricardo accepts no liability for
> any damage caused by any virus transmitted by this e-mail.  "Ricardo" means
> Ricardo plc and its subsidiary companies.
> 
> Ricardo plc is a public limited company registered in England with registered
> number 00222915.
> The registered office of Ricardo plc is Bridge Works, Shoreham-by Sea,
> West Sussex, BN43 5FG.
> 
Nobody is "cheating".

Different filesystems are tuned for different capabilities.

XFS is highly tuned to handle large files because SGI had a major
interest in video preparation and distribution capability.  This
tuning involves eliminating excess disk seeks to handle meta-data
while the actual data is being read/written.  (i.e. Modifying inodes,
etc. in the middle of a large write operation is very time consuming
because of the disk head seeks involved.)

Thus, your test is for exactly the enviroment XFS was designed to
handle efficiently.

OTOH, if you want to run a mail server or a Internet News Feed, you
need to handle lots of small files.  ReiserFS was tuned for this
environment a couple years ago, so it would be the way to go.

EXT3, is a good overall filesystem and I have heard people say it
survives unexpected power-outages the best.

You need to figure out what your needs are from a filesystem, then
find comparative studies of the various ones and see what fits your
needs.

Specifically, I would not recommend XFS for the core filesystems.  i.e
/, /var, /usr, etc.  A lot of debug tools still don't support XFS and
XFS brings no value to the party.

I personnally use ext3 for all my core filesystems, and XFS for my
data drives.  (I work with very large files, typically split down to
iso size.  I also keep a copy on tape, so filesystem reliability is
less important than speed to me.)

Greg
-- 
Greg Freemyer

Re: [linux-lvm] Why the dramatic increase in filesystem performance when usingxfs????

[David S.]

> 
> As the xfs performance comes back about twice the performance of ext3
> for this test I am of the opinion that xfs must be cheating somehow. It
> has always been my opinion that the IO bottleneck is the hardware and
> not the filesystem hence changing the filesystem but using the same
> hardware should not make a huge difference to performance (you still
> have to get the same amount of data out to disk at the end of the day)
> 
> I am struggling to comprehend how xfs can cheat, though, as it can't
> cache such a huge file as there is not enough memory. Is it perhaps
> cheating because the file is comprised entirely of zero's?
> 
> Can someone please enlighten me

Try mounting your ext3 file system with the "data=writeback" option.
See mount(8).

David S.

> 
> Thanks in advance
> 
> Gary Mansell
> -- 
> 
> This e-mail and any files transmitted with it are confidential and intended
> solely for the use of the individual or entity to whom they are addressed.
> If you have received this e-mail in error please notify the sender immediately
> and delete this e-mail from your system.  Please note that any views or opinions
> presented in this e-mail are solely those of the author and do not necessarily
> represent those of Ricardo (save for reports and other documentation formally
> approved and signed for release to the intended recipient).  Only Directors
> or Duly Authorised Officers are authorised to enter into legally binding
> obligations on behalf of Ricardo unless the obligation is contained within
> a Ricardo Purchase Order.
> 
> Ricardo may monitor outgoing and incoming e-mails and other telecommunications
> on its e-mail and telecommunications systems.  By replying to this e-mail you
> give consent to such monitoring.  The recipient should check this e-mail and
> any attachments for the presence of viruses.  Ricardo accepts no liability for
> any damage caused by any virus transmitted by this e-mail.  "Ricardo" means
> Ricardo plc and its subsidiary companies.
> 
> Ricardo plc is a public limited company registered in England with registered
> number 00222915.
> The registered office of Ricardo plc is Bridge Works, Shoreham-by Sea,
> West Sussex, BN43 5FG.
>  
> 
> _______________________________________________
> linux-lvm mailing list
> linux-lvm@redhat.com
> https://www.redhat.com/mailman/listinfo/linux-lvm
> read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/

Re: [linux-lvm] Why the dramatic increase in filesystem performance when usingxfs????

[Markus Baertschi]

If you browse through the extensive filesystem benchmarks at

http://fsbench.netnation.com/

You'll find results quite similar to yours. XFS is performing well for 
large sequential operations while ext3 without the writeback option can 
be half the speed.

I usually use ext3 for the OS volumes only, except with SuSE where 
reiser3 is the default. For other filesystems I prefer jfs for no 
exceptionally good reason but that I came to like it on AIX.

Markus


Gary Mansell wrote:
> Hi,
> 
> I have always run ext3 filesystem with journalling on Redhat AS as it is
> the only supported filesystem.
> 
> One of my colleagues runs xfs, though, and on comparable hardware
> configs he gets twice the performance compared to my ext3 tests.
> 
> The test that I perform is to create a file at least twice the size of
> the RAM installed in the system to avoid the possibility of cacheing,
> measuring the time to write and read the file back gives me the
> performance figure that I am after. I realise that this is a very simple
> test of large sequential IO but it is good enough for my needs.
> 
> ie
> 
> Write test:
> 
> # time dd if=/dev/zero of=./testfile bs=16384 count=250000 ; time sync
> 
> Read test:
> 
> # time dd if=./testfile of=/dev/null bs=16384
> 
> 
> As the xfs performance comes back about twice the performance of ext3
> for this test I am of the opinion that xfs must be cheating somehow. It
> has always been my opinion that the IO bottleneck is the hardware and
> not the filesystem hence changing the filesystem but using the same
> hardware should not make a huge difference to performance (you still
> have to get the same amount of data out to disk at the end of the day)
> 
> I am struggling to comprehend how xfs can cheat, though, as it can't
> cache such a huge file as there is not enough memory. Is it perhaps
> cheating because the file is comprised entirely of zero's?
> 
> Can someone please enlighten me
> 
> Thanks in advance
> 
> Gary Mansell


-- 
   Markus Baertschi             Phone: ++41 (21) 807 1677
   Bas du Ross� 14b             Fax  : ++41 (21) 807 1678
   CH-1163, Etoy                Email: markus@markus.org
   Switzerland                  Homepage: www.markus.org

Re: [linux-lvm] Why the dramatic increase in filesystem performance when usingxfs????

[Nathan Scott]

On Fri, Jan 21, 2005 at 11:24:29AM -0500, Greg Freemyer wrote:
> Specifically, I would not recommend XFS for the core filesystems.  i.e
> /, /var, /usr, etc.  A lot of debug tools still don't support XFS and
> XFS brings no value to the party.

Which debug tools are you referring to there?  I've found XFS
to be an excellent "core" filesystem, but I may be biased. :)

cheers.

-- 
Nathan

Re: [linux-lvm] Why the dramatic increase in filesystem performance when usingxfs????

[Greg Freemyer]

On Mon, 24 Jan 2005 11:32:05 +1100, Nathan Scott <nathans@sgi.com> wrote:
> On Fri, Jan 21, 2005 at 11:24:29AM -0500, Greg Freemyer wrote:
> > Specifically, I would not recommend XFS for the core filesystems.  i.e
> > /, /var, /usr, etc.  A lot of debug tools still don't support XFS and
> > XFS brings no value to the party.
> 
> Which debug tools are you referring to there?  I've found XFS
> to be an excellent "core" filesystem, but I may be biased. :)
> 
> cheers.
> 
> --
> Nathan
> 
Nathan,

You biased, never.  

One issue I recently saw (last summer) was SUSE 9.1 QC testing failed
to test for XFS capability.

That resulted in a full distro release with no XFS support.  Work
arounds were immediately provided, but none of the CDs that came with
the distro allowed you to boot from the CD and access a XFS
filesystem.

For the test system I was upgrading it was not too bad because XFS was
purely on my data partitions.  (ie. I upgraded the OS and core
filesystems, used YOU to get the online updates, then accessed my
fliesystems.)

If I had my core filesystems under XFS, I'm not sure what the fix
would have been.  Further booting SUSE 9.1 into rescue mode created an
environment that could not mount any of the XFS partitions.

Also, last springs release of Knoppix may have XFS support (I don't
know), but it is so unreliable, I still use the 2-year old version. 
No XFS support there.

Another problem I've recently read about is that XFS on / and software
RAID5 are incompatible.  (I have not tried this.)

Basically, I just don't think the various distros get enough testing
of XFS to use it for core filesystems and if you need to use
distros/repair CDs from 12+ months ago you also have problems with
XFS.

Now a question for you:

I have not read a status update of XFS/snapshots/2.6.x.   Can you tell
me if this is expected to work or not?  (Obviously, I don't care about
'/').

This issue and my percieved lack of confidence in its suppport is one
of the prime reasons my production fileservers are still running 2.4
kernels.

If you think it is working, I'm ready to load SUSE 9.2 onto one of my
test platforms and test the above.  It is easy enough to compile a
2.6.10 vanilla kernel if that is likely to be better than the 2.6.8+
kernel that SUSE distributes.

Thanks
Greg
-- 
Greg Freemyer

Re: [linux-lvm] Why the dramatic increase in filesystem performance when usingxfs????

[Nathan Scott]

On Mon, Jan 24, 2005 at 12:53:01PM -0500, Greg Freemyer wrote:
> 
> You biased, never.  

;-)

> One issue I recently saw (last summer) was SUSE 9.1 QC testing failed
> to test for XFS capability.

Ugh, don't remind me.  :|

> If I had my core filesystems under XFS, I'm not sure what the fix
> would have been.  Further booting SUSE 9.1 into rescue mode created an
> environment that could not mount any of the XFS partitions.

There'll be a supported patch by nowsure.  It was not an XFS fix,
so lets just leave it there.  You should expect that distribution
to get alot better wrt XFS in upcoming releases, as they actively
work with us on supporting XFS nowadays.

> Also, last springs release of Knoppix may have XFS support (I don't
> know), but it is so unreliable, I still use the 2-year old version. 
> No XFS support there.

Hmm (looks aside into desk draw) - I have Knoppix 1.0 and it's a
wonderful thing with complete XFS support.  Pretty sure thats at
least a couple of years old, so not sure what you've got there?
Knoppix has supported XFS for as long as I can remember.

> Another problem I've recently read about is that XFS on / and software
> RAID5 are incompatible.  (I have not tried this.)

That is not correct.

> Basically, I just don't think the various distros get enough testing
> of XFS to use it for core filesystems and if you need to use
> distros/repair CDs from 12+ months ago you also have problems with
> XFS.

Some are better than others.  :)

FWIW, all systems that SGI ship to customers and support in the
field use XFS as the primary filesystem.

> Now a question for you:
> 
> I have not read a status update of XFS/snapshots/2.6.x.   Can you tell
> me if this is expected to work or not?  (Obviously, I don't care about
> '/').

Hmm, off the top of my head, not sure - does that test you wrote
function on device mapper?; I should find time to look into that.
I remember Christoph doing some work to tie XFS into the 2.6 API
once that was merged, but I haven't needed to look at it since.
If there are problems, send details to us on the linux-xfs list.

cheers.

-- 
Nathan

[linux-lvm] XFS and snapshots [WAS: Re: Why the dramatic increase in filesystem performance when usingxfs????]

[Greg Freemyer]

On Tue, 25 Jan 2005 08:55:26 +1100, Nathan Scott wrote:
> On Mon, Jan 24, 2005 at 12:53:01PM -0500, Greg Freemyer wrote:
> 
> > Also, last springs release of Knoppix may have XFS support (I don't
> > know), but it is so unreliable, I still use the 2-year old version.
> > No XFS support there.
> 
> Hmm (looks aside into desk draw) - I have Knoppix 1.0 and it's a
> wonderful thing with complete XFS support.  Pretty sure thats at
> least a couple of years old, so not sure what you've got there?
> Knoppix has supported XFS for as long as I can remember.
> 
Mine works too.  :)

I guess I need to verify before I post.

> > Another problem I've recently read about is that XFS on / and software
> > RAID5 are incompatible.  (I have not tried this.)
> 
> That is not correct.

Good to hear.
 
> > Now a question for you:
> >
> > I have not read a status update of XFS/snapshots/2.6.x.   Can you tell
> > me if this is expected to work or not?  (Obviously, I don't care about
> > '/').
> 
> Hmm, off the top of my head, not sure - does that test you wrote
> function on device mapper?; I should find time to look into that.
> I remember Christoph doing some work to tie XFS into the 2.6 API
> once that was merged, but I haven't needed to look at it since.
> If there are problems, send details to us on the linux-xfs list.
> 

The test script uses LVM 1 userland tools.  I have not yet done any
LVM2 userland testing.

I'm not sure if there are any differences between LVM1 and LVM2 CLI commands.

Does anyone on the list know?

The script is available at 

http://oss.sgi.com/cgi-bin/cvsweb.cgi/xfs-cmds/xfstests/068?rev=1.7&content-type=text/x-cvsweb-markup

and the lvm sections are:

#Verify we have the lvm user tools
[ -x /sbin/lvcreate ] || _notrun "lvcreate utility is not installed in /sbin"
[ -x /sbin/lvremove ] || _notrun "lvremove utility is not installed in /sbin"

# If the above fails for LVM 2, the below may be useful notes
#    try "lvm vgdisplay --version" and look for a > 1.9x version number, eg,
# 
#     LVM version:     1.95.10-cvs (2002-05-31)
#     Library version: 0.96.03-ioctl-cvs (2002-06-27)
#     Driver version:  1.0.3

LVM=false

# Check if LVM 1 is in the kernel
if grep lvm /proc/devices > /dev/null; then LVM=true; fi

# Check if LVM 2 is in the kernel
if grep device-mapper /proc/devices > /dev/null; then LVM=true; fi

<snip>
#TODO' are commented out.  I run them by hand, but for an automated
script they can be uncommented.
# Create a PV set from the scratch partition
#TODO # (I don't know if this is needed.and it is dangerous because it
intentionally deletes the partition table!!!
#TODO # dd if=/dev/zero of="$SCRATCH_LVM_DEV" bs=512 count=1
#TODO # pvcreate "$SCRATCH_LVM_DEV"

# Create a VG from the PV
#TODO # vgcreate "$VG" "$SCRATCH_LVM_DEV"

# Create a LV in the VG to snapshot
#TODO # lvcreate -L 2G -n scratch "$VG"

<snip>
	xfs_freeze -f "$SCRATCH_MNT"
	[ $? != 0 ] && echo xfs_freeze -f "$SCRATCH_MNT" failed
	(
		lvcreate --snapshot --size 1G --name scratch_snap "$VG/scratch"\
			> /dev/null 2>&1

<snip>
	lvremove -f "$VG/scratch_snap" > /dev/null 2>&1
	[ $? -eq 0 ] || echo lvremove for "$VG/scratch_snap" failed

> cheers.
> 
> --
> Nathan
> 
Greg
-- 
Greg Freemyer

Re: [linux-lvm] XFS and snapshots [WAS: Re: Why the dramatic increase in filesystem performance when usingxfs????]

[Kristina Clair]

> <snip>
>         xfs_freeze -f "$SCRATCH_MNT"
>         [ $? != 0 ] && echo xfs_freeze -f "$SCRATCH_MNT" failed
>         (
>                 lvcreate --snapshot --size 1G --name scratch_snap "$VG/scratch"\
>                         > /dev/null 2>&1
> 

I'm confused.  I've gotten the impression from other posts to this
list that one should not use xfs_freeze before creating snapshots. 
The reason being that xfs_freeze suspends the filesystem, and then
lvcreate will also attempt to suspend the filesystem...

???

Kristina

Re: [linux-lvm] XFS and snapshots [WAS: Re: Why the dramatic increase in filesystem performance when usingxfs????]

[Greg Freemyer]

On Tue, 25 Jan 2005 12:36:45 -0500, Kristina Clair <kclair@gmail.com> wrote:
> > <snip>
> >         xfs_freeze -f "$SCRATCH_MNT"
> >         [ $? != 0 ] && echo xfs_freeze -f "$SCRATCH_MNT" failed
> >         (
> >                 lvcreate --snapshot --size 1G --name scratch_snap "$VG/scratch"\
> >                         > /dev/null 2>&1
> >
> 
> I'm confused.  I've gotten the impression from other posts to this
> list that one should not use xfs_freeze before creating snapshots.
> The reason being that xfs_freeze suspends the filesystem, and then
> lvcreate will also attempt to suspend the filesystem...
> 
> ???
> 
> Kristina
> 
Kristina,

I was not clear. 

I posted about a working snapshot test script for LVM1 / 2.4 kernel / XFS.

It has not yet been tested with the 2.6 kernel at all, and I know
nothing about LVM2.

I was looking for feedback about obvious changes to the syntax between
LVM1 and LVM2.  Do they even use the same command names?

Your statement about xfs_freeze being a problem may be true and if so
would be the first known (to me) incompatibility in the script.

Nathan, is there a bugzilla about xfs_freeze causing problems witht he
2.6 kernel?

Greg
-- 
Greg Freemyer