Random write result differences between RAID device and XFS

[Christian Affolter <c.affolter@purplehaze.ch>]

Hi everyone,

I'm trying to understand the differences of some bandwidth and IOPs test
results I see while running a random-write full-stripe-width aligned fio
test (using libaio with direct IO) on a hardware RAID 6 raw device
versus on the same device with the XFS file system on top of it.

On the raw device I get:
write: io=24828MB, bw=423132KB/s, iops=137, runt= 60085msec

With XFS on top of it:
write: io=14658MB, bw=249407KB/s, iops=81, runt= 60182msec


The hardware RAID 6 volume consists out of 5 HDDs (3 data disks), the
stripe unit size is 1 MiB and full stripe width is 3 MiB.

XFS was initialized and mounted with the following commands:

mkfs.xfs -d su=1024k,sw=3 -L LV-TEST-02 /dev/sdd
mount -o inode64,noatime -L LV-TEST-02 /mnt/lv-test-02

mkfs.xfs version 3.2.2

xfs_info /mnt/lv-test-02
meta-data=/dev/sdd               isize=256    agcount=16, agsize=819200 blks
         =                       sectsz=4096  attr=2, projid32bit=1
         =                       crc=0        finobt=0
data     =                       bsize=4096   blocks=13106688, imaxpct=25
         =                       sunit=256    swidth=768 blks
naming   =version 2              bsize=4096   ascii-ci=0 ftype=0
log      =internal               bsize=4096   blocks=6399, version=2
         =                       sectsz=4096  sunit=1 blks, lazy-count=1
realtime =none                   extsz=4096   blocks=0, rtextents=0



The RAID controller does not export optimal_io_size.


Controller details:
Product Name = AVAGO 3108 MegaRAID
FW Package Build = 24.9.0-0022
BIOS Version = 6.25.03.0_4.17.08.00_0x060E0300
FW Version = 4.290.00-4536
Driver Name = megaraid_sas
Driver Version = 06.808.16.00-rc1


Virtual drive:

--------------------------------------------------------------------
DG/VD TYPE  State Access Consist Cache sCC      Size Name
--------------------------------------------------------------------
1/2   RAID6 Optl  RW     Yes      RWBD  -   49.998 GB vd-hdd-test-01
--------------------------------------------------------------------

R=Read Ahead
WB=WriteBack (with a battery backup)
D=Direct IO

The physical disk cache is disabled.


Disks:
5x HGST HUH728080AL5200 (Firmware Revision = A515)


Kernel:
4.4.0-2.el7.elrepo.x86_64


fio command and output for RAID raw device:

fio --filename=/dev/sdd \
    --direct=1 \
    --rw=randwrite \
    --ioengine=libaio \
    --iodepth=16 \
    --numjobs=1 \
    --runtime=60 \
    --exec_prerun="/opt/MegaRAID/storcli/storcli64 /c0 flushcache" \
    --name=direct-raid-hdd-random-write-full-stripe-aligned-3072k \
    --bs=3072k

direct-raid-hdd-random-write-full-stripe-aligned-3072k: (g=0):
rw=randwrite, bs=3M-3M/3M-3M/3M-3M, ioengine=libaio, iodepth=16
fio-2.2.8
Starting 1 process
direct-raid-hdd-random-write-full-stripe-aligned-3072k : Saving output
of prerun in
direct-raid-hdd-random-write-full-stripe-aligned-3072k.prerun.txt
Jobs: 1 (f=1): [w(1)] [100.0% done] [0KB/375.0MB/0KB /s] [0/125/0 iops]
[eta 00m:00s]
direct-raid-hdd-random-write-full-stripe-aligned-3072k: (groupid=0,
jobs=1): err= 0: pid=1847: Fri Jan 29 11:47:17 2016
  write: io=24828MB, bw=423132KB/s, iops=137, runt= 60085msec
    slat (usec): min=250, max=91308, avg=7250.27, stdev=11767.73
    clat (msec): min=8, max=223, avg=108.89, stdev=32.22
     lat (msec): min=8, max=224, avg=116.14, stdev=32.19
    clat percentiles (msec):
     |  1.00th=[    9],  5.00th=[   43], 10.00th=[   78], 20.00th=[   91],
     | 30.00th=[   99], 40.00th=[  106], 50.00th=[  113], 60.00th=[  119],
     | 70.00th=[  126], 80.00th=[  133], 90.00th=[  145], 95.00th=[  153],
     | 99.00th=[  169], 99.50th=[  176], 99.90th=[  198], 99.95th=[  202],
     | 99.99th=[  225]
    bw (KB  /s): min=348681, max=2599384, per=100.00%, avg=423757.22,
stdev=204979.51
    lat (msec) : 10=4.22%, 20=0.76%, 50=0.19%, 100=25.86%, 250=68.97%
  cpu          : usr=2.49%, sys=3.57%, ctx=2959, majf=0, minf=1642
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=99.8%, 32=0.0%,
>=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%,
>=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%,
>=64=0.0%
     issued    : total=r=0/w=8276/d=0, short=r=0/w=0/d=0, drop=r=0/w=0/d=0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
  WRITE: io=24828MB, aggrb=423131KB/s, minb=423131KB/s, maxb=423131KB/s,
mint=60085msec, maxt=60085msec

Disk stats (read/write):
  sdd: ios=59/98996, merge=0/0, ticks=150/8434406, in_queue=8443626,
util=100.00%



fio command and output for XFS:

fio --directory=/mnt/lv-test-02 \
    --filename=test.fio  \
    --size=30g \
    --direct=1 \
    --rw=randwrite \
    --ioengine=libaio \
    --iodepth=16 \
    --numjobs=1 \
    --runtime=60 \
    --exec_prerun="/opt/MegaRAID/storcli/storcli64 /c0 flushcache" \
    --name=xfs-ĥdd-random-write-full-stripe-aligned-3072k \
    --bs=3072k


xfs-ĥdd-random-write-full-stripe-aligned-3072k: (g=0): rw=randwrite,
bs=3M-3M/3M-3M/3M-3M, ioengine=libaio, iodepth=16
fio-2.2.8
Starting 1 process
xfs-ĥdd-random-write-full-stripe-aligned-3072k: Laying out IO file(s) (1
file(s) / 30720MB)
xfs-ĥdd-random-write-full-stripe-aligned-3072k : Saving output of prerun
in xfs-ĥdd-random-write-full-stripe-aligned-3072k.prerun.txt
Jobs: 1 (f=1): [w(1)] [100.0% done] [0KB/186.0MB/0KB /s] [0/62/0 iops]
[eta 00m:00s]
xfs-ĥdd-random-write-full-stripe-aligned-3072k: (groupid=0, jobs=1):
err= 0: pid=1899: Fri Jan 29 11:50:21 2016
  write: io=14658MB, bw=249407KB/s, iops=81, runt= 60182msec
    slat (usec): min=231, max=133647, avg=12279.35, stdev=20234.23
    clat (msec): min=4, max=1987, avg=184.75, stdev=81.74
     lat (msec): min=5, max=1987, avg=197.03, stdev=83.84
    clat percentiles (msec):
     |  1.00th=[    7],  5.00th=[    8], 10.00th=[  110], 20.00th=[  143],
     | 30.00th=[  161], 40.00th=[  174], 50.00th=[  188], 60.00th=[  202],
     | 70.00th=[  217], 80.00th=[  237], 90.00th=[  269], 95.00th=[  293],
     | 99.00th=[  363], 99.50th=[  416], 99.90th=[  742], 99.95th=[  922],
     | 99.99th=[ 1991]
    bw (KB  /s): min=130620, max=2460047, per=100.00%, avg=250120.43,
stdev=212307.87
    lat (msec) : 10=8.04%, 100=0.88%, 250=76.44%, 500=14.31%, 750=0.25%
    lat (msec) : 1000=0.04%, 2000=0.04%
  cpu          : usr=1.10%, sys=2.30%, ctx=1891, majf=0, minf=1096
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.2%, 16=99.7%, 32=0.0%,
>=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%,
>=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%,
>=64=0.0%
     issued    : total=r=0/w=4886/d=0, short=r=0/w=0/d=0, drop=r=0/w=0/d=0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
  WRITE: io=14658MB, aggrb=249406KB/s, minb=249406KB/s, maxb=249406KB/s,
mint=60182msec, maxt=60182msec

Disk stats (read/write):
  sdd: ios=0/58627, merge=0/12, ticks=0/8552722, in_queue=8559550,
util=99.84%


Many thanks in advance
Chris

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs