observed significant performance improvement using "delaylog" in a real-world application

observed significant performance improvement using "delaylog" in a real-world application

[Peter Niemayer]

Hi all,

we use XFS for a very I/O-intensive, in-house developed real-time 
database application, and whenever we see new or significantly changed 
file-systems becoming available, we run a benchmark using this 
application on a conserved, fixed real-world data set.

I'm pleased to state that using the experimental "delaylog" mount option 
(in vanilla linux-2.6.35) we measured a 17% performance increase
for our benchmark scenario. (Other mount-options in use both before
and after the "delaylog" option: noatime,nodiratime,nobarrier)

That's a lot given that XFS was the fastest performing file-system
for this application already.

It's also a promising result regarding stability, as several other
tests (using e.g. reiser4 or ceph) in the past led to crashes in the
same benchmark scenario.

So thanks to all contributing developers for this significant optimization!

Regards,

Peter Niemayer


_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: observed significant performance improvement using "delaylog" in a real-world application

[Dave Chinner]

On Tue, Aug 10, 2010 at 06:01:33PM +0200, Peter Niemayer wrote:
> Hi all,
> 
> we use XFS for a very I/O-intensive, in-house developed real-time
> database application, and whenever we see new or significantly
> changed file-systems becoming available, we run a benchmark using
> this application on a conserved, fixed real-world data set.
> 
> I'm pleased to state that using the experimental "delaylog" mount
> option (in vanilla linux-2.6.35) we measured a 17% performance
> increase
> for our benchmark scenario. (Other mount-options in use both before
> and after the "delaylog" option: noatime,nodiratime,nobarrier)

That's great to hear. One thing that you might want to try to
further improve performance is the logbsize=262144 option as well.
That will help flush log IO faster by doing less IOs.

Also, if your workload is doing lots of fsync calls, then the
optimisations I posted a few days ago should also help improve
delaylog throughput.

> That's a lot given that XFS was the fastest performing file-system
> for this application already.
> 
> It's also a promising result regarding stability, as several other
> tests (using e.g. reiser4 or ceph) in the past led to crashes in the
> same benchmark scenario.

That's definitely encouraging. ;)

> So thanks to all contributing developers for this significant optimization!

And thanks for the feedback.

Cheers,

Dave.
-- 
Dave Chinner
david@fromorbit.com

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: observed significant performance improvement using "delaylog" in a real-world application

[Michael Monnerie]

[-- Attachment #1.1: Type: Text/Plain, Size: 679 bytes --]

On Dienstag, 10. August 2010 Peter Niemayer wrote:
> 17% performance increase for our benchmark scenario
 
Can you say what kind of data accesses you have? Database, fileserver, 
virtualization platform with VM Images on XFS... there are so many. Just 
to get an idea in which area delaylog helps that much.

-- 
mit freundlichen Grüssen,
Michael Monnerie, Ing. BSc

it-management Internet Services
http://proteger.at [gesprochen: Prot-e-schee]
Tel: 0660 / 415 65 31

****** Aktuelles Radiointerview! ******
http://www.it-podcast.at/aktuelle-sendung.html

// Wir haben im Moment zwei Häuser zu verkaufen:
// http://zmi.at/langegg/
// http://zmi.at/haus2009/

[-- Attachment #1.2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

[-- Attachment #2: Type: text/plain, Size: 121 bytes --]

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: observed significant performance improvement using "delaylog" in a real-world application

[Peter Niemayer]

On 08/11/2010 10:03 AM, Michael Monnerie wrote:
> On Dienstag, 10. August 2010 Peter Niemayer wrote:
>> 17% performance increase for our benchmark scenario
>
> Can you say what kind of data accesses you have?

The application basically writes a large data set to ~ 100,000 files,
which consists of many small messages. Each message consists of a 
primary key (an integer in the range of 0 to approx. 1,000,000) and
an (almost random) number of data bytes (length in the range from 10 to
~1000 byte).

For each message, the application opens the file that is determined by
the primary key with O_APPEND, write()s the data bytes to the file.
Then it closes the file. (There are usually a few to many messages per
primary key / file).
The application runs 4 threads in parallel to spread the above action
over 4 CPU cores, each thread processes a quarter of the primary keys
(primary_key & 0x03).

This description is, of course, somewhat simplified, but it should
get you an idea on the kind of I/O-operations.

Regards,

Peter Niemayer


_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: observed significant performance improvement using "delaylog" in a real-world application

[Michael Monnerie]

[-- Attachment #1.1: Type: Text/Plain, Size: 647 bytes --]

On Mittwoch, 11. August 2010 Peter Niemayer wrote:
> This description is, of course, somewhat simplified, but it should
> get you an idea on the kind of I/O-operations.
 
Thank you. Are those files located within one dir or do you use a hash 
structure like squid cache does?

-- 
mit freundlichen Grüssen,
Michael Monnerie, Ing. BSc

it-management Internet Services
http://proteger.at [gesprochen: Prot-e-schee]
Tel: 0660 / 415 65 31

****** Aktuelles Radiointerview! ******
http://www.it-podcast.at/aktuelle-sendung.html

// Wir haben im Moment zwei Häuser zu verkaufen:
// http://zmi.at/langegg/
// http://zmi.at/haus2009/

[-- Attachment #1.2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

[-- Attachment #2: Type: text/plain, Size: 121 bytes --]

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: observed significant performance improvement using "delaylog" in a real-world application

[Peter Niemayer]

On 08/11/2010 02:28 PM, Michael Monnerie wrote:
> Thank you. Are those files located within one dir or do you use a hash
> structure like squid cache does?

There's only a shallow hierarchy (for functional, not for distribution
reasons), so the relevant directories have thousands of files in them.

I think after the "ext2"-age no serious file system ever had
a real problem dealing with lots of files in one directory -
or do you have contradicting information?

Regards,

Peter Niemayer

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: observed significant performance improvement using "delaylog" in a real-world application

[Peter Niemayer]

On 08/11/2010 02:08 AM, Dave Chinner wrote:
> One thing that you might want to try to
> further improve performance is the logbsize=262144 option as well.
> That will help flush log IO faster by doing less IOs.

Tried that - but this did not have a positive effect.
(In fact a slightly negative one, but that could be within
the measurement accuracy).

> Also, if your workload is doing lots of fsync calls, then the
> optimisations I posted a few days ago should also help improve
> delaylog throughput.

We don't to any fsync's.

Regards,

Peter Niemayer


_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: observed significant performance improvement using "delaylog" in a real-world application

[Michael Monnerie]

[-- Attachment #1.1: Type: Text/Plain, Size: 1040 bytes --]

On Mittwoch, 11. August 2010 Peter Niemayer wrote:
> I think after the "ext2"-age no serious file system ever had
> a real problem dealing with lots of files in one directory -
> or do you have contradicting information?
 
IIRC 2010 there was someone on this list reporting problems when a 
certain amount of files were created in a dir. He had to do with 
audio/video recording, where delays are important (not to have). At one 
point file creating time increased a lot, but I don't remember if it was 
a filesystem problem or something else. But as it was the only report I 
saw, I'd say you are right and a big number of files per dir doesn't 
matter anymore.

-- 
mit freundlichen Grüssen,
Michael Monnerie, Ing. BSc

it-management Internet Services
http://proteger.at [gesprochen: Prot-e-schee]
Tel: 0660 / 415 65 31

****** Aktuelles Radiointerview! ******
http://www.it-podcast.at/aktuelle-sendung.html

// Wir haben im Moment zwei Häuser zu verkaufen:
// http://zmi.at/langegg/
// http://zmi.at/haus2009/

[-- Attachment #1.2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

[-- Attachment #2: Type: text/plain, Size: 121 bytes --]

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: observed significant performance improvement using "delaylog" in a real-world application

[Dave Chinner]

On Wed, Aug 11, 2010 at 07:01:31PM +0200, Peter Niemayer wrote:
> On 08/11/2010 02:28 PM, Michael Monnerie wrote:
> >Thank you. Are those files located within one dir or do you use a hash
> >structure like squid cache does?
> 
> There's only a shallow hierarchy (for functional, not for distribution
> reasons), so the relevant directories have thousands of files in them.
> 
> I think after the "ext2"-age no serious file system ever had
> a real problem dealing with lots of files in one directory -
> or do you have contradicting information?

Define "lots of files". :)

>From my numbers, ext3/4 still fall way behind XFS and btrfs when it
comes to handling directories with tens of thousands of entries or
larger. Especially on cold-cache random lookups.

XFS also has quite sophisticated internal directory readahead, so
under the cold cache directory performance of XFS is far better than
ext3/4 can acheive, even for relatively small directories. IIRC this
difference in directory lookup performance was one of the prime
reasons kernel.org switched from ext3 to XFS a couple of years
back...

Cheers,

Dave.
-- 
Dave Chinner
david@fromorbit.com

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs