EXT4 RAID read performance

EXT4 RAID read performance

[wbrana]

Hello,

I'm considering replacing Reiser3 filesystem with some newer one.
I ran compilebench benchmark. Read performance is lower with ext4.
Is it expected or is it possible to fix it?

Ext4:
read tree total runs 11 avg 20.47 MB/s (user 0.53s sys 0.74s)
read compiled tree total runs 4 avg 32.94 MB/s (user 0.61s sys 1.17s)

Reiser3:
read tree total runs 11 avg 26.33 MB/s (user 0.54s sys 0.81s)
read compiled tree total runs 4 avg 41.82 MB/s (user 0.62s sys 1.36s)

RAID details:

md8 : active raid10 sda7[0] sdd7[3] sdc7[2] sdb7[1]
      62925824 blocks 256K chunks 2 far-copies [4/4] [UUUU]

Ext4:
mkfs.ext4 -E stride=64,stripe-width=128 /dev/md8
mount -t ext4 -o noatime,auto_da_alloc,commit=600 /dev/md8 /mnt/md8

Reiser3:
mount -t reiserfs /dev/md8 /mnt/md8
mount -t reiserfs -o noatime,notail /dev/md8 /dev/md8

Ext4 results:
intial create total runs 10 avg 172.76 MB/s (user 0.43s sys 0.60s)
create total runs 14 avg 36.49 MB/s (user 0.42s sys 0.59s)
patch total runs 15 avg 15.16 MB/s (user 0.24s sys 0.49s)
compile total runs 14 avg 64.07 MB/s (user 0.10s sys 0.59s)
clean total runs 10 avg 393.43 MB/s (user 0.02s sys 0.06s)
read tree total runs 11 avg 20.47 MB/s (user 0.53s sys 0.74s)
read compiled tree total runs 4 avg 32.94 MB/s (user 0.61s sys 1.17s)
delete tree total runs 10 avg 2.51 seconds (user 0.24s sys 0.42s)
delete compiled tree total runs 4 avg 2.63 seconds (user 0.28s sys 0.50s)
stat tree total runs 11 avg 1.99 seconds (user 0.23s sys 0.18s)
stat compiled tree total runs 7 avg 2.11 seconds (user 0.27s sys 0.21s)

Reiser3 results:
intial create total runs 10 avg 82.74 MB/s (user 0.45s sys 1.13s)
create total runs 14 avg 28.54 MB/s (user 0.45s sys 1.19s)
patch total runs 15 avg 10.91 MB/s (user 0.24s sys 0.86s)
compile total runs 14 avg 47.49 MB/s (user 0.10s sys 1.27s)
clean total runs 10 avg 270.21 MB/s (user 0.02s sys 0.15s)
read tree total runs 11 avg 26.33 MB/s (user 0.54s sys 0.81s)
read compiled tree total runs 4 avg 41.82 MB/s (user 0.62s sys 1.36s)
delete tree total runs 10 avg 3.38 seconds (user 0.24s sys 0.72s)
delete compiled tree total runs 4 avg 4.14 seconds (user 0.27s sys 0.88s)
stat tree total runs 11 avg 2.09 seconds (user 0.22s sys 0.18s)
stat compiled tree total runs 7 avg 2.27 seconds (user 0.25s sys 0.21s)

Re: EXT4 RAID read performance

[wbrana]

creating tar of compiled kernel takes much longer with ext4

time tar c linux-2.6.31 | cat >/dev/null

reiser3:
real    0m11.425s
user    0m0.229s
sys     0m1.586s

ext4:
real    0m23.942s
user    0m0.183s
sys     0m1.440s

partition was remounted before test

>
> RAID details:
>
> md8 : active raid10 sda7[0] sdd7[3] sdc7[2] sdb7[1]
>      62925824 blocks 256K chunks 2 far-copies [4/4] [UUUU]
>
> Ext4:
> mkfs.ext4 -E stride=64,stripe-width=128 /dev/md8
> mount -t ext4 -o noatime,auto_da_alloc,commit=600 /dev/md8 /mnt/md8
>
> Reiser3:
> mkfs.reiserfs /dev/md8
> mount -t reiserfs -o noatime,notail /dev/md8 /mnt/md8
>

Re: EXT4 RAID read performance

[Theodore Tso]

On Sat, Sep 19, 2009 at 08:12:31PM +0200, wbrana@gmail.com wrote:
> 
> I'm considering replacing Reiser3 filesystem with some newer one.
> I ran compilebench benchmark. Read performance is lower with ext4.
> Is it expected or is it possible to fix it?

You didn't say which version of the kernel you are using, which could
be important when asking these sorts of questions about potential
performance problems.

However, in this case, I suspect the issue is the nature of how
compilebench is structured.  Compilebench does the following which
makes it work particularly well for filesystems like reiserfs and
btrfs, and not so much for ext3 and ext4.  Quoting from the
compilebench web page:

   compilebench starts by putting these lists of file names into an
   order native to the filesystem it is working on. The files are
   created in sorted order based on the filename, and then readdir is
   used to find the order the filesystem uses for storing the
   names. After this initial phase, the filesystem native order is
   used for creates, patches and compile. Deleting, reading and
   stating the trees are done in readdir order.

The key here is that it reads the tree in readdir order.  Normally,
when you compile a kernel, the order in which you read and write files
is controlled by the Makefile; you don't get to read and write the
files in the order which just happens to be the most convenient for
the file system's b-tree hash algorithm.

Now, there are some workloads which compilebench might accurately
model --- for example, tar'ing up a directory.  However, despite the
name of the benchmark, it doesn't accurately model a kernel compile.

If you only care about compilebench numbers, you can try creating the
file system with the dir_index feature disabled.  This is the feature
that speeds up random access to directories; unfortunately, it means
that when you read files in readdir order, it causes extra random
reads to th einode table.  However, if your real-life workload is one
where file reads are always magically in readdir order, dir_index adds
overhead without adding any benefit.   

The bottom line is that I'm not terribly worried about trying to
improve ext4's performance on compilebench, since I don't believe it's
a benchmark that models realistic real-life workloads.

Regards,

						- Ted

Re: EXT4 RAID read performance

[wbrana]

On Sun, Sep 20, 2009 at 1:18 AM, Theodore Tso <tytso@mit.edu> wrote:
> You didn't say which version of the kernel you are using, which could
> be important when asking these sorts of questions about potential
> performance problems.
>
2.6.31

>
> Now, there are some workloads which compilebench might accurately
> model --- for example, tar'ing up a directory.  However, despite the
> name of the benchmark, it doesn't accurately model a kernel compile.
>

You are right. I created simple benchmark which does real compilation of kernel
with following results:

reiser3
16925 msec: tar xf /usr/src/linux.tar
119173 msec: make -j4 all

ext4
14296 msec: tar xf /usr/src/linux.tar
116692 msec: make -j4 all

btrfs
13493 msec: tar xf /usr/src/linux.tar
116015 msec: make -j4 all

MD RAID10-f2 with 256 kB chunk size was used in all cases.

Source code:
http://www.sendspace.com/file/fmikv7
It depends on Qt Core library. There are hard-coded paths.
It can delete important data if it is started without customization !!!

Re: EXT4 RAID read performance

[wbrana]

On Sun, Sep 20, 2009 at 3:47 AM, Daniel J Blueman
<daniel.blueman@gmail.com> wrote:
> On Sep 19, 7:20 pm, wbr...@gmail.com wrote:
>>
>> Ext4:
>> read tree total runs 11 avg 20.47 MB/s (user 0.53s sys 0.74s)
>> read compiled tree total runs 4 avg 32.94 MB/s (user 0.61s sys 1.17s)
>>
>> Reiser3:
>> read tree total runs 11 avg 26.33 MB/s (user 0.54s sys 0.81s)
>> read compiled tree total runs 4 avg 41.82 MB/s (user 0.62s sys 1.36s)
>
> It would be interesting to also compare against BTRFS if on 2.6.30 or
> newer, if you can.
>

BTRFS 2.6.31

mkfs.btrfs -d raid10 -m raid10 /dev/sda7 /dev/sdb7 /dev/sdc7 /dev/sdd7
mount -t btrfs -o noatime /dev/sda7 /mnt/md8

intial create total runs 10 avg 158.85 MB/s (user 0.45s sys 0.93s)
create total runs 14 avg 32.67 MB/s (user 0.44s sys 0.90s)
patch total runs 15 avg 8.91 MB/s (user 0.22s sys 0.84s)
compile total runs 14 avg 61.02 MB/s (user 0.09s sys 0.50s)
clean total runs 10 avg 245.12 MB/s (user 0.02s sys 0.18s)

read tree total runs 11 avg 14.03 MB/s (user 0.48s sys 0.87s)
read compiled tree total runs 4 avg 29.14 MB/s (user 0.54s sys 1.37s)

delete tree total runs 10 avg 9.77 seconds (user 0.28s sys 1.37s)
delete compiled tree total runs 4 avg 11.91 seconds (user 0.31s sys 1.60s)
stat tree total runs 11 avg 4.36 seconds (user 0.25s sys 0.33s)
stat compiled tree total runs 7 avg 5.29 seconds (user 0.29s sys 0.37s)