storage with "very high Average Read/Write Request Time"

storage with "very high Average Read/Write Request Time"

[Peter Grandi]

I am copying here (without endorsement or comment) some extracts by
different people from a thread in the XFS mailing list that seems mostly
about RAID tuning:

http://article.gmane.org/gmane.comp.file-systems.xfs.general/65029
>>> Hi, I am using xfs on a raid 5 (~100TB) and put log on external ssd
>>> device, the mount information is: /dev/sdc on
>>> /data/fhgfs/fhgfs_storage type xfs
>>> (rw,relatime,attr2,delaylog,logdev=/dev/sdb1,sunit=512,swidth=15872,noquota).
>>> when doing only reading / only writing , the speed is very
>>> fast(~1.5G), but when do both the speed is very slow (100M), and
>>> high r_await(160) and w_await(200000).
>>> 1. how can I reduce average request time?
>>> 2. can I use ssd as write/read cache for xfs?

http://article.gmane.org/gmane.comp.file-systems.xfs.general/65034
>> There is a ratio of 31 (thirty one) between 'swidth' and 'sunit' and
>> assuming that this reflects the geometry of the RAID5 set and given
>> commonly available disk sizes it can be guessed that with amazing
>> "bravery" someone has configured a RAID5 out of 32 (thirty two) high
>> capacity/low IOPS 3TB drives, or something similar. It is even
>> "braver" than that: if the device name "/data/fhgfs/fhgfs_storage" is
>> dedscriptive, this "brave" RAID5 set is supposed to hold the object
>> storage layer of a BeeFS highly parallel filesystem, and therefore
>> will likely have mostly-random accesses. This issue should be moved
>> to the 'linux-raid' mailing list as from the reported information it
>> has nothing to do with XFS.

http://article.gmane.org/gmane.comp.file-systems.xfs.general/65036
> You apparently have 31 effective SATA 7.2k RPM spindles with 256 KiB
> chunk, 7.75 MiB stripe width, in RAID5.  That should yield 3-4.6 GiB/s
> of streaming throughput assuming no cable, expander, nor HBA
> limitations.  You're achieving only 1/3rd to 1/2 of this. Which
> hardware RAID controller is this?  What are the specs?  Cache RAM,
> host and back end cable count and type?
> When you say read or write is fast individually, but read+write is
> slow, what types of files are you reading and writing, and how many in
> parallel?  This combined pattern is likely the cause of the slowdown
> due to excessive seeking in the drives. As others mentioned this isn't
> an XFS problem.
> The problem is that your RAID geometry doesn't match your workload.
> Your very wide parity stripe is apparently causing excessive seeking
> with your read+write workload due to read-modify-write operations.  To
> mitigate this, and to increase resiliency, you should switch to RAID6
> with a smaller chunk.  If you need maximum capacity make a single
> RAID6 array with 16 KiB chunk size.  This will yield a 496 KiB stripe
> width, increasing the odds that all writes are a full stripe, and
> hopefully eliminating much of the RMW problem.
> A better option might be making three 10 drive RAID6 arrays (two
> spares) with 32 KiB chunk, 256 KiB stripe width, and concatenating
> the 3 arrays with mdadm --linear.  You'd have 24 spindles of capacity
> and throughput instead of 31, but no more RMW operations, or at least
> very few.  You'd format the linear md device with
>   mkfs.xfs -d su=32k,sw=8 /dev/mdX
> As long as your file accesses are spread fairly evenly across at
> least 3 directories you should achieve excellent parallel throughput,
> though single file streaming throughput will peak at 800-1200 MiB/s,
> that of 8 drives.  With a little understanding of how this setup
> works, you can write two streaming files and read a third without any
> of the 3 competing with one another for disk seeks/bandwidth--which
> is your current problem.  Or you could do one read and one write to
> each of 3 directories, and no pair of two would interfere with the
> other pairs.  Scale up from here.
> Basically what we're doing is isolating each RAID LUN into a set of
> directories.  When you write to one of those directories the file
> goes into only one of the 3 RAID arrays.  Doing this isolates RMWs
> for a given write to only a subset of your disks, and minimizes the
> amount of seeks generated by parallel accesses.

Re: storage with "very high Average Read/Write Request Time"

[Peter Grandi]

>>> Hi, I am using xfs on a raid 5 (~100TB) and put log on external ssd
>>> device, the mount information is: /dev/sdc [ ... ] when doing only
>>> reading / only writing , the speed is very fast(~1.5G), but when do
>>> both the speed is very slow (100M), and high r_await(160) and
>>> w_await(200000).

>> There is a ratio of 31 (thirty one) between 'swidth' and 'sunit' and
>> assuming that this reflects the geometry of the RAID5 set and given
>> commonly available disk sizes it can be guessed that with amazing
>> "bravery" someone has configured a RAID5 out of 32 (thirty two) high
>> capacity/low IOPS 3TB drives, or something similar. [ ... ]

>> You apparently have 31 effective SATA 7.2k RPM spindles with 256 KiB
>> chunk, 7.75 MiB stripe width, in RAID5.

That's highly likely, could be nearline SAS drives, but not a big
difference. But given the context SATA is likely as it is cheaper.
Conceivably the storage could also be a chunk of a RAID SAN over a
single FC 16Gb/s link or FCoE over dual bonded roundrobin 10Gb/s links,
but this seems to me to fit less well with the other scant clues
available (e.g. "on a raid 5 (~100TB)" and "log on external ssd
device").

>> That should yield 3-4.6 GiB/s of streaming throughput assuming no
>> cable, expander, nor HBA limitations. You're achieving only 1/3rd to
>> 1/2 of this.

That 1/3 to 1/2 may be not too bad, also considering that the RAID set
is so wide and there will be quite a bit of variance of rotational
positions across it, and that perhaps the hw has channels with a 2GB/s
max chokepoint.

Anyhow these are are some mostly reasonable questions on the details:

>> Which hardware RAID controller is this? What are the specs? Cache
>> RAM, host and back end cable count and type?

I'll do some plausible (hopefully) additional speculation...

The mention of '/dev/sdc' suggests that this is a hw RAID HA (aka HBA),
and the context gives me vibes that:

* The 3TB disks are likely 3.5in, so given typical enclosure geometries
  the likely count of 32 suggests that the disks are in two classic
  16-slot enclosures.

* That the RAID set seems to have 32 drives and there is the ceiling of
  1.5GB/s suggests that there is a single not very recent hw RAID HA and
  the two enclosures are daisy-chained.

* Because oldish RAID HAs typically max at 2GiB/s overall as they have
  8x lane PCIe 1.x host bus connectors; or they are recent PCIe 2.0 ones
  but plugged into oldish PCIe 1.x host bus slots (or perhaps even
  plugged into slots with less than 8x lanes).

There is an enormous difference in read vs. write in «r_await(160) and
w_await(200000)» and the latter is apocalyptic at 200 seconds ('man
iostat' confirms that 'r_await' and 'w_await' are in ms). I have seen
that kind of horror before and it suggests that this is one of several
common types of hw RAID HA which a massively misdesigned buggy IO
scheduler and cache manager in the firmware.

>>> 1. how can I reduce average request time?

Some better alternative geometries have been already suggested, some of
which I don't like...

But overall if the speculation above applies, the current setup seems to
be "audaciously" aimed at the lowest possible upfront price, "targeting"
a workload made almost entirely of reads of data archived elsewhere, or
single-stream writes or reads, and that needs changing.

To change that I think that the two main suggestions are:

1. Stop using the hw RAID mode and use instead Linux MD raid.
3. Significantly boost IOPS when doing concurrent read-writes as per
   typical HPC distributed filesystem object stores. If the aim was
   indeed "lowest possible upfront price" that's not to easy.

As to the latter point it depends a bit whether the object store is
meant to hold transient or permanent data, and the nature of cluster
jobs expected, but here are some general thoughts:

*  Ideally reach the same capacity with 1TB drives, as they have a much
   better IOPS-per-GB ratio than 3TB drives. I consider for various
   reasons current 2TB drives as already having a too low IOPS-per-GB
   ratio for most cases of "live" data, for example.

*  Change the geometry of the RAID sets, for example multiple sets with
   each set being:
   - RAID10 with 16 members, or RAID10 with 8 members.
   - RAID5 with 3 members, with hot spares.
   - RAID5 with 5 members, with hot spares.
   - With some reluctance, RAID6 with 6-8-10 members.

*  Some apposite tuning of the MD RAID parameters depending on the RAID
   set geometry and expected workload, the usual on stripe cache sizes,
   elevators, read-ahead, dirty page lifetime and ERC/TLER timeouts.

The reason behind the multiple sets is to reduce correlation, by
reducing RMW, and for example also permitting parallel 'fsck' and
probably also better backups impact.

Having a single very large storage pool per server is particularly
"insipid" if this is meant to be an object store for HPC parallel
filesystems like BeeGFS, because usually they are configured to slice
each files into 1MB segments and distributed these segments across many
available object stores.

>>> 2. can I use ssd as write/read cache for xfs?

I think that if the workload is that of a typical HPC distributed
filesystem object store that's not going to help much as object stores
cannot cache that much because of overall randomish access patterns.
Caches don't "always" increase IOPS across the board. :-)
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html