More testing: 4x parallel 2G writes, sequential reads

More testing: 4x parallel 2G writes, sequential reads

[Eric Sandeen]

I tried ext4 vs. xfs doing 4 parallel 2G IO writes in 1M units to 4
different subdirectories of the root of the filesystem:

http://people.redhat.com/esandeen/seekwatcher/ext4_4_threads.png
http://people.redhat.com/esandeen/seekwatcher/xfs_4_threads.png
http://people.redhat.com/esandeen/seekwatcher/ext4_xfs_4_threads.png

and then read them back sequentially:

http://people.redhat.com/esandeen/seekwatcher/ext4_4_threads_read.png
http://people.redhat.com/esandeen/seekwatcher/xfs_4_threads_read.png
http://people.redhat.com/esandeen/seekwatcher/ext4_xfs_4_read_threads.png

At the end of the write, ext4 had on the order of 400 extents/file, xfs
had on the order of 30 extents/file.  It's clear especially from the
read graph that ext4 is interleaving the 4 files, in about 5M chunks on
average.  Throughput seems comparable between ext4 & xfs nonetheless.

Again this was on a decent HW raid so seek penalties are probably not
too bad.

-Eric

Re: More testing: 4x parallel 2G writes, sequential reads

[Andreas Dilger]

On Nov 07, 2007  16:42 -0600, Eric Sandeen wrote:
> I tried ext4 vs. xfs doing 4 parallel 2G IO writes in 1M units to 4
> different subdirectories of the root of the filesystem:
> 
> http://people.redhat.com/esandeen/seekwatcher/ext4_4_threads.png
> http://people.redhat.com/esandeen/seekwatcher/xfs_4_threads.png
> http://people.redhat.com/esandeen/seekwatcher/ext4_xfs_4_threads.png
> 
> and then read them back sequentially:
> 
> http://people.redhat.com/esandeen/seekwatcher/ext4_4_threads_read.png
> http://people.redhat.com/esandeen/seekwatcher/xfs_4_threads_read.png
> http://people.redhat.com/esandeen/seekwatcher/ext4_xfs_4_read_threads.png
> 
> At the end of the write, ext4 had on the order of 400 extents/file, xfs
> had on the order of 30 extents/file.  It's clear especially from the
> read graph that ext4 is interleaving the 4 files, in about 5M chunks on
> average.  Throughput seems comparable between ext4 & xfs nonetheless.

The question is what the "best" result is for this kind of workload?
In HPC applications the common case is that you will also have the data
files read back in parallel instead of serially.

The test shows ext4 finishing marginally faster in the write case, and
marginally slower in the read case.  What happens if you have 4 parallel
readers?

Cheers, Andreas
--
Andreas Dilger
Sr. Software Engineer, Lustre Group
Sun Microsystems of Canada, Inc.

Re: More testing: 4x parallel 2G writes, sequential reads

[Eric Sandeen]

Andreas Dilger wrote:

> The question is what the "best" result is for this kind of workload?
> In HPC applications the common case is that you will also have the data
> files read back in parallel instead of serially.

Agreed, I'm not trying to argue what's better or worse, I'm just seeing
what it's doing.

The main reason I did sequential reads back is that it more clearly
shows the file layout for each file on the graph.  :)  I'm just getting
a handle on how the allocations are going for various types of writes.

> The test shows ext4 finishing marginally faster in the write case, and
> marginally slower in the read case.  What happens if you have 4 parallel
> readers?

I'll test that a bit later (have to run now); I expect parallel readers
may go faster, since the blocks are interleaved, and it might be able to
suck them up pretty much in order across all 4 files.

I'd also like to test some of this under a single head, rather than on
HW raid...

-Eric

Re: More testing: 4x parallel 2G writes, sequential reads

[Eric Sandeen]

Andreas Dilger wrote:

> The test shows ext4 finishing marginally faster in the write case, and
> marginally slower in the read case.  What happens if you have 4 parallel
> readers?

http://people.redhat.com/esandeen/seekwatcher/ext4_4_thread_par_read.png
http://people.redhat.com/esandeen/seekwatcher/xfs_4_thread_par_read.png
http://people.redhat.com/esandeen/seekwatcher/ext4_xfs_4_thread_par_read.png

-Eric

Re: More testing: 4x parallel 2G writes, sequential reads

[Alex Tomas]

Hi,

could you try to larger preallocation? like 512/1024/2048 blocks, please?

thanks, Alex

Eric Sandeen wrote:
> I tried ext4 vs. xfs doing 4 parallel 2G IO writes in 1M units to 4
> different subdirectories of the root of the filesystem:
> 
> http://people.redhat.com/esandeen/seekwatcher/ext4_4_threads.png
> http://people.redhat.com/esandeen/seekwatcher/xfs_4_threads.png
> http://people.redhat.com/esandeen/seekwatcher/ext4_xfs_4_threads.png
> 
> and then read them back sequentially:
> 
> http://people.redhat.com/esandeen/seekwatcher/ext4_4_threads_read.png
> http://people.redhat.com/esandeen/seekwatcher/xfs_4_threads_read.png
> http://people.redhat.com/esandeen/seekwatcher/ext4_xfs_4_read_threads.png
> 
> At the end of the write, ext4 had on the order of 400 extents/file, xfs
> had on the order of 30 extents/file.  It's clear especially from the
> read graph that ext4 is interleaving the 4 files, in about 5M chunks on
> average.  Throughput seems comparable between ext4 & xfs nonetheless.
> 
> Again this was on a decent HW raid so seek penalties are probably not
> too bad.
> 
> -Eric
> -
> To unsubscribe from this list: send the line "unsubscribe linux-ext4" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 

Re: More testing: 4x parallel 2G writes, sequential reads

[Shapor Naghibzadeh]

On Wed, Nov 07, 2007 at 04:42:59PM -0600, Eric Sandeen wrote:
> Again this was on a decent HW raid so seek penalties are probably not
> too bad.

You may want to verify that by doing a benchmark on the raw device.  I
recently did some benchmarks doing random I/O on a Dell 2850 w/ a PERC
(megaraid) RAID5 w/ 128MB onboard writeback cache and 6x 15krpm drives
and noticed appoximately one order of magnitude throughput drop on
small (stripe-sized) random reads versus linear.  It maxed out at ~100
random read IOPs or "seeks/sec" (suprisingly low).

Out of curiousity, how are you counting the seeks?

Shapor

Re: More testing: 4x parallel 2G writes, sequential reads

[Eric Sandeen]

Shapor Naghibzadeh wrote:
> On Wed, Nov 07, 2007 at 04:42:59PM -0600, Eric Sandeen wrote:
>> Again this was on a decent HW raid so seek penalties are probably not
>> too bad.
> 
> You may want to verify that by doing a benchmark on the raw device.  I
> recently did some benchmarks doing random I/O on a Dell 2850 w/ a PERC
> (megaraid) RAID5 w/ 128MB onboard writeback cache and 6x 15krpm drives
> and noticed appoximately one order of magnitude throughput drop on
> small (stripe-sized) random reads versus linear.  It maxed out at ~100
> random read IOPs or "seeks/sec" (suprisingly low).
> 
> Out of curiousity, how are you counting the seeks?

Chris Mason's seekwatcher (google can find it for you) is doing the
graphing, it uses blocktrace for the raw data.

-Eric

> Shapor