* RAID5 Performance
@ 2016-07-27 2:24 Adam Goryachev
2016-07-27 3:15 ` Brad Campbell
` (2 more replies)
0 siblings, 3 replies; 18+ messages in thread
From: Adam Goryachev @ 2016-07-27 2:24 UTC (permalink / raw)
To: linux-raid@vger.kernel.org
Hi all,
I know, age old question, but I have the chance to change things up a
bit, and I wanted to collect some thoughts/ideas.
Currently I am using 8 x 480GB Intel SSD in a RAID5, then LVM on top,
DRBD on top, and finally iSCSI on top (and then used as VM raw disks for
mostly windows VM's).
My current array looks like this:
/dev/md1:
Version : 1.2
Creation Time : Wed Aug 22 00:47:03 2012
Raid Level : raid5
Array Size : 3281935552 (3129.90 GiB 3360.70 GB)
Used Dev Size : 468847936 (447.13 GiB 480.10 GB)
Raid Devices : 8
Total Devices : 8
Persistence : Superblock is persistent
Update Time : Wed Jul 27 11:32:00 2016
State : active
Active Devices : 8
Working Devices : 8
Failed Devices : 0
Spare Devices : 0
Layout : left-symmetric
Chunk Size : 64K
Name : san1:1 (local to host san1)
UUID : 707957c0:b7195438:06da5bc4:485d301c
Events : 2185221
Number Major Minor RaidDevice State
7 8 65 0 active sync /dev/sde1
13 8 1 1 active sync /dev/sda1
8 8 81 2 active sync /dev/sdf1
5 8 113 3 active sync /dev/sdh1
9 8 97 4 active sync /dev/sdg1
12 8 17 5 active sync /dev/sdb1
10 8 49 6 active sync /dev/sdd1
11 8 33 7 active sync /dev/sdc1
I've configured the following non-standard options:
echo 4096 > /sys/block/md1/md/stripe_cache_size
The following apply to all SSD's installed:
echo noop > $disk/queue/scheduler
echo 128 > ${disk}/queue/nr_requests
What I can measure (at peak periods) with iostat:
Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz
avgqu-sz await r_await w_await svctm %util
sdi 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 0.00 0.00 0.00 0.00 0.00
sda 78.00 59.00 79.00 86.00 0.74 0.52
15.55 0.02 0.15 0.20 0.09 0.15 2.40
sdg 35.00 48.00 68.00 79.00 0.52 0.44
13.39 0.02 0.14 0.24 0.05 0.11 1.60
sdf 46.00 65.00 86.00 98.00 0.76 0.58
14.96 0.03 0.17 0.09 0.24 0.09 1.60
sdh 97.00 45.00 70.00 141.00 0.66 0.68
12.96 0.08 0.36 0.29 0.40 0.34 7.20
sde 101.00 75.00 87.00 94.00 0.79 0.61
15.76 0.08 0.42 0.32 0.51 0.29 5.20
sdb 85.00 54.00 94.00 102.00 0.84 0.56
14.62 0.01 0.04 0.09 0.00 0.04 0.80
sdc 85.00 74.00 98.00 106.00 0.79 0.66
14.53 0.01 0.06 0.04 0.08 0.04 0.80
sdd 230.00 199.00 266.00 353.00 2.19 2.11
14.24 0.18 0.28 0.23 0.32 0.16 9.60
drbd0 0.00 0.00 0.00 2.00 0.00 0.00
4.50 0.08 38.00 0.00 38.00 20.00 4.00
drbd12 0.00 0.00 1.00 1.00 0.00 0.00
7.50 0.03 14.00 4.00 24.00 14.00 2.80
drbd1 0.00 0.00 0.00 2.00 0.00 0.03
32.00 0.09 44.00 0.00 44.00 22.00 4.40
drbd9 0.00 0.00 2.00 0.00 0.01 0.00
8.00 0.00 0.00 0.00 0.00 0.00 0.00
drbd2 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 0.00 0.00 0.00 0.00 0.00
drbd11 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 0.00 0.00 0.00 0.00 0.00
drbd3 0.00 0.00 4.00 197.00 0.02 1.01
10.47 7.92 41.03 0.00 41.87 4.98 100.00
drbd4 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 0.00 0.00 0.00 0.00 0.00
drbd17 0.00 0.00 1.00 0.00 0.00 0.00
8.00 0.00 0.00 0.00 0.00 0.00 0.00
drbd5 0.00 0.00 0.00 7.00 0.00 0.03
8.00 0.22 30.29 0.00 30.29 28.57 20.00
drbd19 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 0.00 0.00 0.00 0.00 0.00
drbd6 0.00 0.00 2.00 0.00 0.01 0.00
8.00 0.00 0.00 0.00 0.00 0.00 0.00
drbd7 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 0.00 0.00 0.00 0.00 0.00
drbd8 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 0.00 0.00 0.00 0.00 0.00
drbd13 0.00 0.00 90.00 44.00 1.74 0.38
32.35 1.72 13.46 0.40 40.18 4.27 57.20
drbd15 0.00 0.00 2.00 33.00 0.02 0.29
17.86 1.40 40.91 0.00 43.39 28.34 99.20
drbd18 0.00 0.00 1.00 3.00 0.00 0.03
16.00 0.08 21.00 0.00 28.00 21.00 8.40
drbd14 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 0.00 0.00 0.00 0.00 0.00
drbd10 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 0.00 0.00 0.00 0.00 0.00
As you can see, the DRBD devices are busy, and slowing down the VM's,
looking at the drives on the second server we can see why:
Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz
avgqu-sz await r_await w_await svctm %util
sdf 67.00 76.00 64.00 113.00 0.52 0.62
13.17 0.26 1.47 0.06 2.27 1.45 25.60
sdg 39.00 61.00 50.00 114.00 0.35 0.56
11.38 0.45 2.76 0.08 3.93 2.71 44.40
sdd 49.00 67.00 50.00 109.00 0.39 0.57
12.40 0.75 4.73 0.00 6.90 4.70 74.80
sdh 55.00 54.00 52.00 104.00 0.42 0.51
12.12 0.81 5.21 0.23 7.69 5.13 80.00
sde 67.00 67.00 75.00 129.00 0.56 0.65
12.13 0.94 4.59 0.69 6.85 4.24 86.40
sda 64.00 76.00 58.00 109.00 0.48 0.61
13.29 0.84 5.03 0.21 7.60 4.89 81.60
sdb 35.00 72.00 57.00 104.00 0.36 0.57
11.84 0.69 4.27 0.14 6.54 4.22 68.00
sdc 118.00 144.00 228.00 269.00 1.39 1.50
11.92 1.21 2.43 1.88 2.90 1.50 74.40
md1 0.00 0.00 0.00 260.00 0.00 1.70
13.38 0.00 0.00 0.00 0.00 0.00 0.00
I've confirmed that the problem is that we have mixed two models of SSD
(520 series and 530 series), and that the 530 series drives perform
significantly worse (under load) in comparison. Above, the two 520
series are sdf and sdg while the other drives are 530 series. So, we
will be replacing all of the drives across both systems with 545s series
1000GB SSD's (which I've confirmed will operate same or better than the
520 series, sdc on the first machine above is one of these already).
Over the years, I've learned a lot about RAID and optimisation,
originally I configured things to optimise for super fast streaming
reads and streaming writes, but in practice, the actual work-load is
small random read/write, with the writes causing the biggest load.
Looking at this:
http://serverfault.com/questions/384273/optimizing-raid-5-for-backuppc-use-small-random-reads
>
> *
>
> Enhance the queue depth. Standard kernel queue depth is OK for old
> single drives with small caches, but not for modern drives or RAID
> arrays:
>
> echo 512 > /sys/block/sda/queue/nr_requests
>
So my question is should I increase the configured nr_requests above the
current 128?
If the chunk size is 64k, and there are 8 drives in total, then the
stripe size is currently 64k*7 = 448k, is this too big? My reading of
the mdadm man page suggests the minimum chunk size is 4k ("In any case
it must be a multiple of 4KB"). If I set the chunk size to 4k, then the
stripe size becomes 28k, which means for a random 4k write, we only need
to write 28k instead of 448k ?
The drives report a sector size of 512k, which I guess means the
smallest meaningful write that the drive can do is 512k, so should I
increase the chunk size to 512k to match? Or does that make it even worse?
Finally, the drive reports Host_Writes_32MiB in SMART, does that mean
that the drive needs to replace a entire 32MB chunk in order to
overwrite a sector? I'm guessing a chunk size of 32M is just crazy though...
Is there a better way to actually measure the different sizes and
quantity of read/writes being issued, so that I can make a more accurate
decision on chunk size/stripe size/etc... iostat seems to show an
average numbers, but not the number of 1k read/write, 4k read/write, 16k
read/write etc...
My suspicion is that the actual load is made up of rather small random
read/write, because that is the scenario that produced the worst
performance results when I was initially setting this up, and seems to
be what we are getting in practice.
The last option is, what if I moved to RAID10? Would that provide a
significant performance boost (completely removes the need to worry
about chunk/stripe size because we always just write the exact data we
want, no need to read/compute/write)?
OR, is that read/compute overhead negligible since I'm using SSD and
read performance is so quick?
For completeness, PV information:
PV Name /dev/md1
VG Name vg0
PV Size 3.06 TiB / not usable 2.94 MiB
Allocatable yes
PE Size 4.00 MiB
Total PE 801253
Free PE 33281
Allocated PE 767972
PV UUID c0PIEb-tUka-zBk3-lcGM-H89s-ayde-hcMUBZ
Any advice or assistance would be greatly appreciated.
Regards,
Adam
--
Adam Goryachev Website Managers www.websitemanagers.com.au
^ permalink raw reply [flat|nested] 18+ messages in thread
* Re: RAID5 Performance
2016-07-27 2:24 RAID5 Performance Adam Goryachev
@ 2016-07-27 3:15 ` Brad Campbell
2016-07-27 5:36 ` Doug Dumitru
2016-07-27 14:26 ` Peter Grandi
2 siblings, 0 replies; 18+ messages in thread
From: Brad Campbell @ 2016-07-27 3:15 UTC (permalink / raw)
To: Adam Goryachev, linux-raid@vger.kernel.org
On 27/07/16 10:24, Adam Goryachev wrote:
> Hi all,
>
> I know, age old question, but I have the chance to change things up a
> bit, and I wanted to collect some thoughts/ideas.
>
> Currently I am using 8 x 480GB Intel SSD in a RAID5, then LVM on top,
> DRBD on top, and finally iSCSI on top (and then used as VM raw disks for
> mostly windows VM's).
Wow. More layers than a wedding cake.
>
> My suspicion is that the actual load is made up of rather small random
> read/write, because that is the scenario that produced the worst
> performance results when I was initially setting this up, and seems to
> be what we are getting in practice.
>
> The last option is, what if I moved to RAID10? Would that provide a
> significant performance boost (completely removes the need to worry
> about chunk/stripe size because we always just write the exact data we
> want, no need to read/compute/write)?
> OR, is that read/compute overhead negligible since I'm using SSD and
> read performance is so quick?
I'll only comment from personal experience with anecdotal evidence.
I have a RAID10 comprised of 6 256GB SSD (3 Intel & 3 Samsung) used as
the backing for multiple VMs (raw files on Ext4).
Initially I played with a number of RAID types when setting up the array
(back in 2012) and found RAID10 offered the best compromise for my use
case. This was based on CPU usage (raid 5 & 6 parity calculations on
*every write*), the need for RMW cycles for small writes and trying to
balance block sizes. None of these things are an issue with RAID10 and
in general I found a measurable reduction in overhead and consequential
performance boost in the VM's.
This is a single machine with a relatively underpowered AMD FX-8350 CPU.
I found that with multiple VM's hitting the disks I was losing enough
CPU time to RAID overhead that it made things noticeably less responsive.
My only issue is half the disks return a deterministic value after
discard and the other half don't, so any raid check operations return a
gazillion mismatches. Not an operational issue, but one worth mentioning
if you were to use different model drives.
Regards,
Brad
^ permalink raw reply [flat|nested] 18+ messages in thread
* Re: RAID5 Performance
2016-07-27 2:24 RAID5 Performance Adam Goryachev
2016-07-27 3:15 ` Brad Campbell
@ 2016-07-27 5:36 ` Doug Dumitru
2016-07-27 23:26 ` Adam Goryachev
[not found] ` <7af0cc98-e395-9446-05eb-a6c0ca20f187@websitemanagers.com.au>
2016-07-27 14:26 ` Peter Grandi
2 siblings, 2 replies; 18+ messages in thread
From: Doug Dumitru @ 2016-07-27 5:36 UTC (permalink / raw)
To: Adam Goryachev; +Cc: linux-raid@vger.kernel.org
On Tue, Jul 26, 2016 at 7:24 PM, Adam Goryachev
<mailinglists@websitemanagers.com.au> wrote:
> Hi all,
>
> I know, age old question, but I have the chance to change things up a bit,
> and I wanted to collect some thoughts/ideas.
>
> Currently I am using 8 x 480GB Intel SSD in a RAID5, then LVM on top, DRBD
> on top, and finally iSCSI on top (and then used as VM raw disks for mostly
> windows VM's).
>
> My current array looks like this:
>
> /dev/md1:
> Version : 1.2
> Creation Time : Wed Aug 22 00:47:03 2012
> Raid Level : raid5
> Array Size : 3281935552 (3129.90 GiB 3360.70 GB)
> Used Dev Size : 468847936 (447.13 GiB 480.10 GB)
> Raid Devices : 8
> Total Devices : 8
> Persistence : Superblock is persistent
>
> Update Time : Wed Jul 27 11:32:00 2016
> State : active
> Active Devices : 8
> Working Devices : 8
> Failed Devices : 0
> Spare Devices : 0
>
> Layout : left-symmetric
> Chunk Size : 64K
>
> Name : san1:1 (local to host san1)
> UUID : 707957c0:b7195438:06da5bc4:485d301c
> Events : 2185221
>
> Number Major Minor RaidDevice State
> 7 8 65 0 active sync /dev/sde1
> 13 8 1 1 active sync /dev/sda1
> 8 8 81 2 active sync /dev/sdf1
> 5 8 113 3 active sync /dev/sdh1
> 9 8 97 4 active sync /dev/sdg1
> 12 8 17 5 active sync /dev/sdb1
> 10 8 49 6 active sync /dev/sdd1
> 11 8 33 7 active sync /dev/sdc1
>
> I've configured the following non-standard options:
>
> echo 4096 > /sys/block/md1/md/stripe_cache_size
>
> The following apply to all SSD's installed:
> echo noop > $disk/queue/scheduler
> echo 128 > ${disk}/queue/nr_requests
>
> What I can measure (at peak periods) with iostat:
> Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz
> avgqu-sz await r_await w_await svctm %util
> sdi 0.00 0.00 0.00 0.00 0.00 0.00 0.00
> 0.00 0.00 0.00 0.00 0.00 0.00
> sda 78.00 59.00 79.00 86.00 0.74 0.52 15.55
> 0.02 0.15 0.20 0.09 0.15 2.40
> sdg 35.00 48.00 68.00 79.00 0.52 0.44 13.39
> 0.02 0.14 0.24 0.05 0.11 1.60
> sdf 46.00 65.00 86.00 98.00 0.76 0.58 14.96
> 0.03 0.17 0.09 0.24 0.09 1.60
> sdh 97.00 45.00 70.00 141.00 0.66 0.68 12.96
> 0.08 0.36 0.29 0.40 0.34 7.20
> sde 101.00 75.00 87.00 94.00 0.79 0.61 15.76
> 0.08 0.42 0.32 0.51 0.29 5.20
> sdb 85.00 54.00 94.00 102.00 0.84 0.56 14.62
> 0.01 0.04 0.09 0.00 0.04 0.80
> sdc 85.00 74.00 98.00 106.00 0.79 0.66 14.53
> 0.01 0.06 0.04 0.08 0.04 0.80
> sdd 230.00 199.00 266.00 353.00 2.19 2.11 14.24
> 0.18 0.28 0.23 0.32 0.16 9.60
> drbd0 0.00 0.00 0.00 2.00 0.00 0.00 4.50
> 0.08 38.00 0.00 38.00 20.00 4.00
> drbd12 0.00 0.00 1.00 1.00 0.00 0.00 7.50
> 0.03 14.00 4.00 24.00 14.00 2.80
> drbd1 0.00 0.00 0.00 2.00 0.00 0.03 32.00
> 0.09 44.00 0.00 44.00 22.00 4.40
> drbd9 0.00 0.00 2.00 0.00 0.01 0.00 8.00
> 0.00 0.00 0.00 0.00 0.00 0.00
> drbd2 0.00 0.00 0.00 0.00 0.00 0.00 0.00
> 0.00 0.00 0.00 0.00 0.00 0.00
> drbd11 0.00 0.00 0.00 0.00 0.00 0.00 0.00
> 0.00 0.00 0.00 0.00 0.00 0.00
> drbd3 0.00 0.00 4.00 197.00 0.02 1.01 10.47
> 7.92 41.03 0.00 41.87 4.98 100.00
> drbd4 0.00 0.00 0.00 0.00 0.00 0.00 0.00
> 0.00 0.00 0.00 0.00 0.00 0.00
> drbd17 0.00 0.00 1.00 0.00 0.00 0.00 8.00
> 0.00 0.00 0.00 0.00 0.00 0.00
> drbd5 0.00 0.00 0.00 7.00 0.00 0.03 8.00
> 0.22 30.29 0.00 30.29 28.57 20.00
> drbd19 0.00 0.00 0.00 0.00 0.00 0.00 0.00
> 0.00 0.00 0.00 0.00 0.00 0.00
> drbd6 0.00 0.00 2.00 0.00 0.01 0.00 8.00
> 0.00 0.00 0.00 0.00 0.00 0.00
> drbd7 0.00 0.00 0.00 0.00 0.00 0.00 0.00
> 0.00 0.00 0.00 0.00 0.00 0.00
> drbd8 0.00 0.00 0.00 0.00 0.00 0.00 0.00
> 0.00 0.00 0.00 0.00 0.00 0.00
> drbd13 0.00 0.00 90.00 44.00 1.74 0.38 32.35
> 1.72 13.46 0.40 40.18 4.27 57.20
> drbd15 0.00 0.00 2.00 33.00 0.02 0.29 17.86
> 1.40 40.91 0.00 43.39 28.34 99.20
> drbd18 0.00 0.00 1.00 3.00 0.00 0.03 16.00
> 0.08 21.00 0.00 28.00 21.00 8.40
> drbd14 0.00 0.00 0.00 0.00 0.00 0.00 0.00
> 0.00 0.00 0.00 0.00 0.00 0.00
> drbd10 0.00 0.00 0.00 0.00 0.00 0.00 0.00
> 0.00 0.00 0.00 0.00 0.00 0.00
>
> As you can see, the DRBD devices are busy, and slowing down the VM's,
> looking at the drives on the second server we can see why:
> Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz
> avgqu-sz await r_await w_await svctm %util
> sdf 67.00 76.00 64.00 113.00 0.52 0.62 13.17
> 0.26 1.47 0.06 2.27 1.45 25.60
> sdg 39.00 61.00 50.00 114.00 0.35 0.56 11.38
> 0.45 2.76 0.08 3.93 2.71 44.40
> sdd 49.00 67.00 50.00 109.00 0.39 0.57 12.40
> 0.75 4.73 0.00 6.90 4.70 74.80
> sdh 55.00 54.00 52.00 104.00 0.42 0.51 12.12
> 0.81 5.21 0.23 7.69 5.13 80.00
> sde 67.00 67.00 75.00 129.00 0.56 0.65 12.13
> 0.94 4.59 0.69 6.85 4.24 86.40
> sda 64.00 76.00 58.00 109.00 0.48 0.61 13.29
> 0.84 5.03 0.21 7.60 4.89 81.60
> sdb 35.00 72.00 57.00 104.00 0.36 0.57 11.84
> 0.69 4.27 0.14 6.54 4.22 68.00
> sdc 118.00 144.00 228.00 269.00 1.39 1.50 11.92
> 1.21 2.43 1.88 2.90 1.50 74.40
> md1 0.00 0.00 0.00 260.00 0.00 1.70 13.38
> 0.00 0.00 0.00 0.00 0.00 0.00
>
> I've confirmed that the problem is that we have mixed two models of SSD (520
> series and 530 series), and that the 530 series drives perform significantly
> worse (under load) in comparison. Above, the two 520 series are sdf and sdg
> while the other drives are 530 series. So, we will be replacing all of the
> drives across both systems with 545s series 1000GB SSD's (which I've
> confirmed will operate same or better than the 520 series, sdc on the first
> machine above is one of these already).
>
> Over the years, I've learned a lot about RAID and optimisation, originally I
> configured things to optimise for super fast streaming reads and streaming
> writes, but in practice, the actual work-load is small random read/write,
> with the writes causing the biggest load.
>
> Looking at this:
> http://serverfault.com/questions/384273/optimizing-raid-5-for-backuppc-use-small-random-reads
>>
>>
>> *
>>
>> Enhance the queue depth. Standard kernel queue depth is OK for old
>> single drives with small caches, but not for modern drives or RAID
>> arrays:
>>
>> echo 512 > /sys/block/sda/queue/nr_requests
>>
> So my question is should I increase the configured nr_requests above the
> current 128?
With your workload, it probably won't matter too much. Really high
queue depths are great on paper, but hard to actually see.
>
> If the chunk size is 64k, and there are 8 drives in total, then the stripe
> size is currently 64k*7 = 448k, is this too big? My reading of the mdadm man
> page suggests the minimum chunk size is 4k ("In any case it must be a
> multiple of 4KB"). If I set the chunk size to 4k, then the stripe size
> becomes 28k, which means for a random 4k write, we only need to write 28k
> instead of 448k ?
This is not how a random write works. If you are running raid-5
before the 4.4 kernel, you get the "old" read/modify/write algorithm.
If you write 4K, the system will read 4K from (n-2) drives, add in
your 4K to compute parity, and write 2 drives. This is n-2 reads + 2
writes. With the "new" logic in 4.4, you read the old contents of the
4K plus parity, and re-write the 4k plus parity, so there are 2 reads
and 2 writes. With big arrays, the "new" logic can help quite a bit,
but the chatter rate is still high. Note that the new logic is only
raid-5. raid-6 cannot use the new logic and has to read the stripe
from every drive.
The stripe size impacts when the system does can avoid doing a
read/modify/write. If you write a full stripe [ 64K * (n-1) ], and
the write is exactly on a stripe boundary, and you get lucky and the
background thread does not wake up at just the wrong time, you will do
the write with zero reads. I personally run with very small chunks,
but I have code that always writes perfect stripe writes and stock
file systems don't act that way.
DRBD can saturate GigE without any problem with random 4K writes. I
have a pair of systems here that pushes 110 MB/sec at 4K or 28,000
IOPS. The target arrays needs to keep up, but that is another story.
My testing with DRBD is that it starts to peter out at 10Gig, so if
you want more bandwidth you need some other approach. Some vendors
use SRP over Infiniband with software raid-1 as a mirror. iSCSI with
iSER should give you similar results with RDMA capable ethernet.
Linbit (the people who write DRBD) have a non GPL extension to DRBD
that uses RDMA so you can get more bandwidth that way as well.
> The drives report a sector size of 512k, which I guess means the smallest
> meaningful write that the drive can do is 512k, so should I increase the
> chunk size to 512k to match? Or does that make it even worse?
> Finally, the drive reports Host_Writes_32MiB in SMART, does that mean that
> the drive needs to replace a entire 32MB chunk in order to overwrite a
> sector? I'm guessing a chunk size of 32M is just crazy though...
This is probably not true. If the drive really had to update 512K at
a time, then 4K writes would be 128x wear amplification. SSDs can be
bad, but usually not that bad.
>
> Is there a better way to actually measure the different sizes and quantity
> of read/writes being issued, so that I can make a more accurate decision on
> chunk size/stripe size/etc... iostat seems to show an average numbers, but
> not the number of 1k read/write, 4k read/write, 16k read/write etc...
The problem is that the FTL of the SSDs are a black box and as the
array gets bigger, the slowest drive dictates the array performance.
This is why the "big vendors" all map SSDs in the host and avoid or
minimize writing randomly. I know of one vendor install that has 4000
VDI seats (using ESXI as compute hosts) from a single HA pair of 24
SSD shelves. The connection to ESXI is FC and the hosts are HA with
an IB/SRP raid-1 link between them. Unfortunately, you need 500K+
random write IOPS to pull this off, which I think is impossible with
stock parity raid, and very hard with raid-10.
>
> My suspicion is that the actual load is made up of rather small random
> read/write, because that is the scenario that produced the worst performance
> results when I was initially setting this up, and seems to be what we are
> getting in practice.
>
> The last option is, what if I moved to RAID10? Would that provide a
> significant performance boost (completely removes the need to worry about
> chunk/stripe size because we always just write the exact data we want, no
> need to read/compute/write)?
RAID-10 will be faster, but you pay for this with capacity. It is
also a double-edged sword as SSDs themselves run faster if you leave
more free space on them, so RAID-10 absolutely might not be a lot
faster than RAID-5 with some space left over. Also remember that free
space on the SSDs only counts if it is actually unallocated. So you
need to trim the SSDs or start with a secure erased drive and then
never use the full capacity. It is best to leave an empty partition
that is untouched.
> OR, is that read/compute overhead negligible since I'm using SSD and read
> performance is so quick?
The reads, especially with the pre 4.4 code or with raid-6 definitely
take their toll. Most SSDs are also not quite symmetrical in terms of
performance. If your SSD does 50K read IOPS and 50K write IOPS, it
will probably not do 25K reads and 25K writes concurrently, but
instead stop somewhere around 18K. But your mileage may vary. If you
have 8 drives that do 20 read/write symmetric, with new raid-5, each
4K write is 2 reads and 2 writes. 8 drives will give you 8*20K = 160K
reads and writes or 320K total OPS. Each 4K write takes 4 OPS, so
your data rate ends up maxing out at 80K IOPS. With the old raid-5
logic, you end up with 6 reads plus two writes per "OP", so you tend
to max out around 320K/(6+2) = 40K IOPS. With more than 8 drives,
these computations tend to fall apart, so 24 SSD arrays are not 3x
faster than 8 SSD arrays, at least with stock code.
You also need to consider what raid does to the SSD FTL. As you
chatter a drive, its wear goes up and its performance goes down.
Different SSD models can vary wildly, but again the rule of thumb is
keep as much free space as possible on the drives. raid-5 or
mirroring is also 2:1 write amplification (ie, you are writing two
drives) and raid-6 is 3:1, on top of whatever the FTL write
amplification is at the time.
>
> For completeness, PV information:
> PV Name /dev/md1
> VG Name vg0
> PV Size 3.06 TiB / not usable 2.94 MiB
> Allocatable yes
> PE Size 4.00 MiB
> Total PE 801253
> Free PE 33281
> Allocated PE 767972
> PV UUID c0PIEb-tUka-zBk3-lcGM-H89s-ayde-hcMUBZ
>
> Any advice or assistance would be greatly appreciated.
>
> Regards,
> Adam
> --
> Adam Goryachev Website Managers www.websitemanagers.com.au
> --
> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
--
Doug Dumitru
WileFire Storage. http://www.wildfire-storage.com
^ permalink raw reply [flat|nested] 18+ messages in thread
* Re: RAID5 Performance
2016-07-27 2:24 RAID5 Performance Adam Goryachev
2016-07-27 3:15 ` Brad Campbell
2016-07-27 5:36 ` Doug Dumitru
@ 2016-07-27 14:26 ` Peter Grandi
2016-07-27 17:38 ` Doug Dumitru
2 siblings, 1 reply; 18+ messages in thread
From: Peter Grandi @ 2016-07-27 14:26 UTC (permalink / raw)
To: Linux RAID
First a terminology point: you are reporting a *speed* problem,
not a performance problem. My impression that you are getting
pretty good performance, given your hardware, configuration and
workload. The difference between "speed" and "performance" is
that the first is a simple rate of stuff done per unit of time,
the second is an envelope embodying several tradeoffs, among
them with cost. In your question you describe a low rate of
stuff done per unit of time. :-)
> Currently I am using 8 x 480GB Intel SSD in a RAID5, then LVM
> on top, DRBD on top, and finally iSCSI on top (and then used
> as VM raw disks for mostly windows VM's).
A very brave configuration, a shining example of the "syntactic"
mindset, according to which any arbitrary combination of
legitimate features must be fine :-).
First server DRBD primary disks:
> Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
> sdi 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
> sda 78.00 59.00 79.00 86.00 0.74 0.52 15.55 0.02 0.15 0.20 0.09 0.15 2.40
> sdg 35.00 48.00 68.00 79.00 0.52 0.44 13.39 0.02 0.14 0.24 0.05 0.11 1.60
> sdf 46.00 65.00 86.00 98.00 0.76 0.58 14.96 0.03 0.17 0.09 0.24 0.09 1.60
> sdh 97.00 45.00 70.00 141.00 0.66 0.68 12.96 0.08 0.36 0.29 0.40 0.34 7.20
> sde 101.00 75.00 87.00 94.00 0.79 0.61 15.76 0.08 0.42 0.32 0.51 0.29 5.20
> sdb 85.00 54.00 94.00 102.00 0.84 0.56 14.62 0.01 0.04 0.09 0.00 0.04 0.80
> sdc 85.00 74.00 98.00 106.00 0.79 0.66 14.53 0.01 0.06 0.04 0.08 0.04 0.80
> sdd 230.00 199.00 266.00 353.00 2.19 2.11 14.24 0.18 0.28 0.23 0.32 0.16 9.60
Second server DRBD secondary disks:
> Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
> sdf 67.00 76.00 64.00 113.00 0.52 0.62 13.17 0.26 1.47 0.06 2.27 1.45 25.60
> sdg 39.00 61.00 50.00 114.00 0.35 0.56 11.38 0.45 2.76 0.08 3.93 2.71 44.40
> sdd 49.00 67.00 50.00 109.00 0.39 0.57 12.40 0.75 4.73 0.00 6.90 4.70 74.80
> sdh 55.00 54.00 52.00 104.00 0.42 0.51 12.12 0.81 5.21 0.23 7.69 5.13 80.00
> sde 67.00 67.00 75.00 129.00 0.56 0.65 12.13 0.94 4.59 0.69 6.85 4.24 86.40
> sda 64.00 76.00 58.00 109.00 0.48 0.61 13.29 0.84 5.03 0.21 7.60 4.89 81.60
> sdb 35.00 72.00 57.00 104.00 0.36 0.57 11.84 0.69 4.27 0.14 6.54 4.22 68.00
> sdc 118.00 144.00 228.00 269.00 1.39 1.50 11.92 1.21 2.43 1.88 2.90 1.50 74.40
> md1 0.00 0.00 0.00 260.00 0.00 1.70 13.38 0.00 0.00 0.00 0.00 0.00 0.00
> I've confirmed that the problem is that we have mixed two
> models of SSD (520 series and 530 series), and that the 530
> series drives perform significantly worse (under load) in
> comparison.
The queue sizes and waiting time on the second server are very
low (on a somewhat similar system using 4TB disks I see waiting
times in the 1-5 seconds range, not milliseconds).
The impression I get is that there is some issue with DRBD
latency, because the second server's storage seems to me very
underutilized. This latency may be related to the flash SSDs
that you are using, because by default DRBD uses the "C"
synchronization protocol. Probably if you switched to the "B" or
even "A" protocols speed could improve, maybe a lot, even if
performance arguably would be the same or much worse.
Thus the most likely issue here is the 'fsync' problem: for
"consumerish" SSDs barrier-writes are synchronous, because they
don't have a battery/capacitor-backed cache, and rather slow for
small writes, because of the large size of erase blocks, which
can be mitigated with higher over-provisioning. These have much
of the story:
https://www.sebastien-han.fr/blog/2014/10/10/ceph-how-to-test-if-your-ssd-is-suitable-as-a-journal-device/
https://www.redhat.com/en/resources/ceph-pcie-ssd-performance-part-1
http://www.spinics.net/lists/ceph-users/msg25928.html
The 520s seem not too bad, but still a long way from the disks
with battery/capacity-backed cache.
> the actual work-load is small random read/write, with the
> writes causing the biggest load.
Here most of the wise comments from the reply from D Dimitru
apply, to summarize:
* Small writes are a challenging workload for DRBD, regardless
of other issues.
* Small writes are a very challenging workload for flash SSDs
without battery/capacitor-backed caches.
* Parity RAID is a bad idea in general, in particular for
workloads with many small writes, for they amplify writes via
RMW.
etc. etc. :-)
^ permalink raw reply [flat|nested] 18+ messages in thread
* Re: RAID5 Performance
2016-07-27 14:26 ` Peter Grandi
@ 2016-07-27 17:38 ` Doug Dumitru
2016-07-28 12:19 ` Peter Grandi
0 siblings, 1 reply; 18+ messages in thread
From: Doug Dumitru @ 2016-07-27 17:38 UTC (permalink / raw)
To: Peter Grandi; +Cc: Linux RAID
On Wed, Jul 27, 2016 at 7:26 AM, Peter Grandi <pg@lxra2.for.sabi.co.uk> wrote:
> First a terminology point: you are reporting a *speed* problem,
> not a performance problem. My impression that you are getting
> pretty good performance, given your hardware, configuration and
> workload. The difference between "speed" and "performance" is
> that the first is a simple rate of stuff done per unit of time,
> the second is an envelope embodying several tradeoffs, among
> them with cost. In your question you describe a low rate of
> stuff done per unit of time. :-)
>
>> Currently I am using 8 x 480GB Intel SSD in a RAID5, then LVM
>> on top, DRBD on top, and finally iSCSI on top (and then used
>> as VM raw disks for mostly windows VM's).
>
> A very brave configuration, a shining example of the "syntactic"
> mindset, according to which any arbitrary combination of
> legitimate features must be fine :-).
While you may say that this configuration is very "brave", it is
actually quite common for VDI "appliance" deployments. If you look at
cluster solutions like Ganeti, it is exactly this stack less the iSCSI
(Ganeti runs storage and VM compute on the same nodes). It is also a
"LOT" faster and a lot lower wear than using a file system like ZFS to
create ZVOLs and export those. The other option is to run a
file-system and export files as block devices. This just replaces LVM
with EXT4/XFS, and for static "blobs", LVM is a bunch faster and
safer.
>
> First server DRBD primary disks:
>
>> Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
>> sdi 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
>> sda 78.00 59.00 79.00 86.00 0.74 0.52 15.55 0.02 0.15 0.20 0.09 0.15 2.40
>> sdg 35.00 48.00 68.00 79.00 0.52 0.44 13.39 0.02 0.14 0.24 0.05 0.11 1.60
>> sdf 46.00 65.00 86.00 98.00 0.76 0.58 14.96 0.03 0.17 0.09 0.24 0.09 1.60
>> sdh 97.00 45.00 70.00 141.00 0.66 0.68 12.96 0.08 0.36 0.29 0.40 0.34 7.20
>> sde 101.00 75.00 87.00 94.00 0.79 0.61 15.76 0.08 0.42 0.32 0.51 0.29 5.20
>> sdb 85.00 54.00 94.00 102.00 0.84 0.56 14.62 0.01 0.04 0.09 0.00 0.04 0.80
>> sdc 85.00 74.00 98.00 106.00 0.79 0.66 14.53 0.01 0.06 0.04 0.08 0.04 0.80
>> sdd 230.00 199.00 266.00 353.00 2.19 2.11 14.24 0.18 0.28 0.23 0.32 0.16 9.60
>
> Second server DRBD secondary disks:
>
>> Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
>> sdf 67.00 76.00 64.00 113.00 0.52 0.62 13.17 0.26 1.47 0.06 2.27 1.45 25.60
>> sdg 39.00 61.00 50.00 114.00 0.35 0.56 11.38 0.45 2.76 0.08 3.93 2.71 44.40
>> sdd 49.00 67.00 50.00 109.00 0.39 0.57 12.40 0.75 4.73 0.00 6.90 4.70 74.80
>> sdh 55.00 54.00 52.00 104.00 0.42 0.51 12.12 0.81 5.21 0.23 7.69 5.13 80.00
>> sde 67.00 67.00 75.00 129.00 0.56 0.65 12.13 0.94 4.59 0.69 6.85 4.24 86.40
>> sda 64.00 76.00 58.00 109.00 0.48 0.61 13.29 0.84 5.03 0.21 7.60 4.89 81.60
>> sdb 35.00 72.00 57.00 104.00 0.36 0.57 11.84 0.69 4.27 0.14 6.54 4.22 68.00
>> sdc 118.00 144.00 228.00 269.00 1.39 1.50 11.92 1.21 2.43 1.88 2.90 1.50 74.40
>> md1 0.00 0.00 0.00 260.00 0.00 1.70 13.38 0.00 0.00 0.00 0.00 0.00 0.00
>
>> I've confirmed that the problem is that we have mixed two
>> models of SSD (520 series and 530 series), and that the 530
>> series drives perform significantly worse (under load) in
>> comparison.
>
> The queue sizes and waiting time on the second server are very
> low (on a somewhat similar system using 4TB disks I see waiting
> times in the 1-5 seconds range, not milliseconds).
The expectation, in terms of performance for VDI is quite high.
vmWare like to say you can get away with 8-12 IOPS per virtual. Most
people think you only get good performance with 100 IOPS per virtual.
The "bad" time for VDI is what is called a "boot storm". Boot, or
reboot, all of the windows clients at the same time and see how long
they take to settle. The IO workload for this is 80%+ 4K random
writes. At 100 IOPS, it takes windows about 2 minutes to boot, so if
you need to support 500 VDI seats from a storage node, that nodes
needs to sustain 500 x 100 = 50,000 IOPS of 4K random writes. This is
so far past what hard disks can do as to be silly. Even with SSDs,
you need reasonably large arrays running RAID-10 to sustain this. If
you want to support 5000 VDI seats like this, then stock Linux just
can't get there, but it can be done.
>
> The impression I get is that there is some issue with DRBD
> latency, because the second server's storage seems to me very
> underutilized. This latency may be related to the flash SSDs
> that you are using, because by default DRBD uses the "C"
> synchronization protocol. Probably if you switched to the "B" or
> even "A" protocols speed could improve, maybe a lot, even if
> performance arguably would be the same or much worse.
>
> Thus the most likely issue here is the 'fsync' problem: for
> "consumerish" SSDs barrier-writes are synchronous, because they
> don't have a battery/capacitor-backed cache, and rather slow for
> small writes, because of the large size of erase blocks, which
> can be mitigated with higher over-provisioning. These have much
> of the story:
On many consumer SSDs, barrier writes are only barriers, and are not
syncs at all. You are guaranteed serialization but not actual
storage. Then again, in a server setup, especially with redundant
power supplies, power loss to the SSDs is rare. You are more
protecting against system hangs and other inter-connectivity issues.
The real system solution is to have some quantity of non volatile DRAM
that you can stage writes (either a PCI-e card like a FlashTec or one
or more nvDIMMs). This is how the "major vendors" deal with sync
writes.
> https://www.sebastien-han.fr/blog/2014/10/10/ceph-how-to-test-if-your-ssd-is-suitable-as-a-journal-device/
> https://www.redhat.com/en/resources/ceph-pcie-ssd-performance-part-1
> http://www.spinics.net/lists/ceph-users/msg25928.html
>
> The 520s seem not too bad, but still a long way from the disks
> with battery/capacity-backed cache.
>
>> the actual work-load is small random read/write, with the
>> writes causing the biggest load.
>
> Here most of the wise comments from the reply from D Dimitru
> apply, to summarize:
>
> * Small writes are a challenging workload for DRBD, regardless
> of other issues.
My comment about DRBD was not that small writes are harder, but that
if your target can keep up with them at low queue depths, then DRBD
can saturate GigE at 4K q=1 on a single thread. So DRBD is not really
the issue, but the latency/IOPS behaviour of the target.
> * Small writes are a very challenging workload for flash SSDs
> without battery/capacitor-backed caches.
Even with battery backup, small writes create garbage collection, so
while batteries may give you some short term bursts, longer term, you
still have to do the writes. A main benefit of batter backup in the
SSDs is that the meta data (mapping information) does not need to be
flushed with the actual data real-time, which makes the FTL algorithms
easier to implement.
> * Parity RAID is a bad idea in general, in particular for
> workloads with many small writes, for they amplify writes via
> RMW.
>
> etc. etc. :-)
> --
> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
--
Doug Dumitru
WildFire Storage
http://www.wildfire-storage.com
^ permalink raw reply [flat|nested] 18+ messages in thread
* Re: RAID5 Performance
2016-07-27 5:36 ` Doug Dumitru
@ 2016-07-27 23:26 ` Adam Goryachev
[not found] ` <7af0cc98-e395-9446-05eb-a6c0ca20f187@websitemanagers.com.au>
1 sibling, 0 replies; 18+ messages in thread
From: Adam Goryachev @ 2016-07-27 23:26 UTC (permalink / raw)
To: doug; +Cc: linux-raid@vger.kernel.org
On 27/07/2016 15:36, Doug Dumitru wrote:
> On Tue, Jul 26, 2016 at 7:24 PM, Adam Goryachev
> <mailinglists@websitemanagers.com.au> wrote:
>> Hi all,
>>
>> I know, age old question, but I have the chance to change things up a bit,
>> and I wanted to collect some thoughts/ideas.
>>
>> Currently I am using 8 x 480GB Intel SSD in a RAID5, then LVM on top, DRBD
>> on top, and finally iSCSI on top (and then used as VM raw disks for mostly
>> windows VM's).
>>
>> My current array looks like this:
>>
>> /dev/md1:
>> Version : 1.2
>> Creation Time : Wed Aug 22 00:47:03 2012
>> Raid Level : raid5
>> Array Size : 3281935552 (3129.90 GiB 3360.70 GB)
>> Used Dev Size : 468847936 (447.13 GiB 480.10 GB)
>> Raid Devices : 8
>> Total Devices : 8
>> Persistence : Superblock is persistent
>>
>> Update Time : Wed Jul 27 11:32:00 2016
>> State : active
>> Active Devices : 8
>> Working Devices : 8
>> Failed Devices : 0
>> Spare Devices : 0
>>
>> Layout : left-symmetric
>> Chunk Size : 64K
>>
>> Name : san1:1 (local to host san1)
>> UUID : 707957c0:b7195438:06da5bc4:485d301c
>> Events : 2185221
>>
>> Number Major Minor RaidDevice State
>> 7 8 65 0 active sync /dev/sde1
>> 13 8 1 1 active sync /dev/sda1
>> 8 8 81 2 active sync /dev/sdf1
>> 5 8 113 3 active sync /dev/sdh1
>> 9 8 97 4 active sync /dev/sdg1
>> 12 8 17 5 active sync /dev/sdb1
>> 10 8 49 6 active sync /dev/sdd1
>> 11 8 33 7 active sync /dev/sdc1
>>
>> I've configured the following non-standard options:
>>
>> echo 4096 > /sys/block/md1/md/stripe_cache_size
>>
>> The following apply to all SSD's installed:
>> echo noop > $disk/queue/scheduler
>> echo 128 > ${disk}/queue/nr_requests
>>
>> What I can measure (at peak periods) with iostat:
>> Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz
>> avgqu-sz await r_await w_await svctm %util
>> sdi 0.00 0.00 0.00 0.00 0.00 0.00 0.00
>> 0.00 0.00 0.00 0.00 0.00 0.00
>> sda 78.00 59.00 79.00 86.00 0.74 0.52 15.55
>> 0.02 0.15 0.20 0.09 0.15 2.40
>> sdg 35.00 48.00 68.00 79.00 0.52 0.44 13.39
>> 0.02 0.14 0.24 0.05 0.11 1.60
>> sdf 46.00 65.00 86.00 98.00 0.76 0.58 14.96
>> 0.03 0.17 0.09 0.24 0.09 1.60
>> sdh 97.00 45.00 70.00 141.00 0.66 0.68 12.96
>> 0.08 0.36 0.29 0.40 0.34 7.20
>> sde 101.00 75.00 87.00 94.00 0.79 0.61 15.76
>> 0.08 0.42 0.32 0.51 0.29 5.20
>> sdb 85.00 54.00 94.00 102.00 0.84 0.56 14.62
>> 0.01 0.04 0.09 0.00 0.04 0.80
>> sdc 85.00 74.00 98.00 106.00 0.79 0.66 14.53
>> 0.01 0.06 0.04 0.08 0.04 0.80
>> sdd 230.00 199.00 266.00 353.00 2.19 2.11 14.24
>> 0.18 0.28 0.23 0.32 0.16 9.60
>> drbd0 0.00 0.00 0.00 2.00 0.00 0.00 4.50
>> 0.08 38.00 0.00 38.00 20.00 4.00
>> drbd12 0.00 0.00 1.00 1.00 0.00 0.00 7.50
>> 0.03 14.00 4.00 24.00 14.00 2.80
>> drbd1 0.00 0.00 0.00 2.00 0.00 0.03 32.00
>> 0.09 44.00 0.00 44.00 22.00 4.40
>> drbd9 0.00 0.00 2.00 0.00 0.01 0.00 8.00
>> 0.00 0.00 0.00 0.00 0.00 0.00
>> drbd2 0.00 0.00 0.00 0.00 0.00 0.00 0.00
>> 0.00 0.00 0.00 0.00 0.00 0.00
>> drbd11 0.00 0.00 0.00 0.00 0.00 0.00 0.00
>> 0.00 0.00 0.00 0.00 0.00 0.00
>> drbd3 0.00 0.00 4.00 197.00 0.02 1.01 10.47
>> 7.92 41.03 0.00 41.87 4.98 100.00
>> drbd4 0.00 0.00 0.00 0.00 0.00 0.00 0.00
>> 0.00 0.00 0.00 0.00 0.00 0.00
>> drbd17 0.00 0.00 1.00 0.00 0.00 0.00 8.00
>> 0.00 0.00 0.00 0.00 0.00 0.00
>> drbd5 0.00 0.00 0.00 7.00 0.00 0.03 8.00
>> 0.22 30.29 0.00 30.29 28.57 20.00
>> drbd19 0.00 0.00 0.00 0.00 0.00 0.00 0.00
>> 0.00 0.00 0.00 0.00 0.00 0.00
>> drbd6 0.00 0.00 2.00 0.00 0.01 0.00 8.00
>> 0.00 0.00 0.00 0.00 0.00 0.00
>> drbd7 0.00 0.00 0.00 0.00 0.00 0.00 0.00
>> 0.00 0.00 0.00 0.00 0.00 0.00
>> drbd8 0.00 0.00 0.00 0.00 0.00 0.00 0.00
>> 0.00 0.00 0.00 0.00 0.00 0.00
>> drbd13 0.00 0.00 90.00 44.00 1.74 0.38 32.35
>> 1.72 13.46 0.40 40.18 4.27 57.20
>> drbd15 0.00 0.00 2.00 33.00 0.02 0.29 17.86
>> 1.40 40.91 0.00 43.39 28.34 99.20
>> drbd18 0.00 0.00 1.00 3.00 0.00 0.03 16.00
>> 0.08 21.00 0.00 28.00 21.00 8.40
>> drbd14 0.00 0.00 0.00 0.00 0.00 0.00 0.00
>> 0.00 0.00 0.00 0.00 0.00 0.00
>> drbd10 0.00 0.00 0.00 0.00 0.00 0.00 0.00
>> 0.00 0.00 0.00 0.00 0.00 0.00
>>
>> As you can see, the DRBD devices are busy, and slowing down the VM's,
>> looking at the drives on the second server we can see why:
>> Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz
>> avgqu-sz await r_await w_await svctm %util
>> sdf 67.00 76.00 64.00 113.00 0.52 0.62 13.17
>> 0.26 1.47 0.06 2.27 1.45 25.60
>> sdg 39.00 61.00 50.00 114.00 0.35 0.56 11.38
>> 0.45 2.76 0.08 3.93 2.71 44.40
>> sdd 49.00 67.00 50.00 109.00 0.39 0.57 12.40
>> 0.75 4.73 0.00 6.90 4.70 74.80
>> sdh 55.00 54.00 52.00 104.00 0.42 0.51 12.12
>> 0.81 5.21 0.23 7.69 5.13 80.00
>> sde 67.00 67.00 75.00 129.00 0.56 0.65 12.13
>> 0.94 4.59 0.69 6.85 4.24 86.40
>> sda 64.00 76.00 58.00 109.00 0.48 0.61 13.29
>> 0.84 5.03 0.21 7.60 4.89 81.60
>> sdb 35.00 72.00 57.00 104.00 0.36 0.57 11.84
>> 0.69 4.27 0.14 6.54 4.22 68.00
>> sdc 118.00 144.00 228.00 269.00 1.39 1.50 11.92
>> 1.21 2.43 1.88 2.90 1.50 74.40
>> md1 0.00 0.00 0.00 260.00 0.00 1.70 13.38
>> 0.00 0.00 0.00 0.00 0.00 0.00
>>
>> I've confirmed that the problem is that we have mixed two models of SSD (520
>> series and 530 series), and that the 530 series drives perform significantly
>> worse (under load) in comparison. Above, the two 520 series are sdf and sdg
>> while the other drives are 530 series. So, we will be replacing all of the
>> drives across both systems with 545s series 1000GB SSD's (which I've
>> confirmed will operate same or better than the 520 series, sdc on the first
>> machine above is one of these already).
>>
>> Over the years, I've learned a lot about RAID and optimisation, originally I
>> configured things to optimise for super fast streaming reads and streaming
>> writes, but in practice, the actual work-load is small random read/write,
>> with the writes causing the biggest load.
>>
>> Looking at this:
>> http://serverfault.com/questions/384273/optimizing-raid-5-for-backuppc-use-small-random-reads
>>> *
>>>
>>> Enhance the queue depth. Standard kernel queue depth is OK for old
>>> single drives with small caches, but not for modern drives or RAID
>>> arrays:
>>>
>>> echo 512 > /sys/block/sda/queue/nr_requests
>>>
>> So my question is should I increase the configured nr_requests above the
>> current 128?
> With your workload, it probably won't matter too much. Really high
> queue depths are great on paper, but hard to actually see.
Is there some way to see if this would help or not?
Would it hurt to increase this (even if it doesn't help)?
>
>> If the chunk size is 64k, and there are 8 drives in total, then the stripe
>> size is currently 64k*7 = 448k, is this too big? My reading of the mdadm man
>> page suggests the minimum chunk size is 4k ("In any case it must be a
>> multiple of 4KB"). If I set the chunk size to 4k, then the stripe size
>> becomes 28k, which means for a random 4k write, we only need to write 28k
>> instead of 448k ?
> This is not how a random write works. If you are running raid-5
> before the 4.4 kernel, you get the "old" read/modify/write algorithm.
> If you write 4K, the system will read 4K from (n-2) drives, add in
> your 4K to compute parity, and write 2 drives. This is n-2 reads + 2
> writes. With the "new" logic in 4.4, you read the old contents of the
> 4K plus parity, and re-write the 4k plus parity, so there are 2 reads
> and 2 writes. With big arrays, the "new" logic can help quite a bit,
> but the chatter rate is still high. Note that the new logic is only
> raid-5. raid-6 cannot use the new logic and has to read the stripe
> from every drive.
Hmmm, so an upgrade to kernel 4.6.3 (debian backports version) should
provide a significant performance boost even if nothing else changes.
> The stripe size impacts when the system does can avoid doing a
> read/modify/write. If you write a full stripe [ 64K * (n-1) ], and
> the write is exactly on a stripe boundary, and you get lucky and the
> background thread does not wake up at just the wrong time, you will do
> the write with zero reads. I personally run with very small chunks,
> but I have code that always writes perfect stripe writes and stock
> file systems don't act that way.
So reducing the chunk size will have minimal impact... but reducing it
should still provide some performance boost. Since I'm recreating the
array anyway, what size makes the most sense? 16k or go straight to the
minimum of 4k? Would a smaller chunk size increase the IOPS because we
need to make more (smaller) requests for the same data, potentially from
more drives?
ie, currently, a single read request for 4k will be done by reading one
chunk (64k) from one of the 8 drives (1 IOPS)
currently, a single write request for 4k will be done by reading one
chunk (64k) from 6 drives, and then writing one chunk (64k) to two
drives (8 IOPS)
However, a read (or write) 48k request would be identical to the above,
while a smaller chunk size (4k) would mean:
read request - reading 2 x 4k chunks from 5 disks and 1 x 4k chunk from
2 disks (7 IOPS)
write request - write 8 x 4k (full stripe) (assuming it is stripe
aligned somewhere, but it might not be)
- read 2 x 4k chunks (the only 2 data chunks that
won't be written) + write 6 x 4k chunks
Total of 16 IOPS in the best case, worst case is two partial stripe
writes + 1 full stripe write in the middle: 8 reads + 16 writes or 24 IOPS.
Either the above is wrong, or I've just convinced myself that reducing
the chunk size is not a good idea...
> DRBD can saturate GigE without any problem with random 4K writes. I
> have a pair of systems here that pushes 110 MB/sec at 4K or 28,000
> IOPS. The target arrays needs to keep up, but that is another story.
> My testing with DRBD is that it starts to peter out at 10Gig, so if
> you want more bandwidth you need some other approach. Some vendors
> use SRP over Infiniband with software raid-1 as a mirror. iSCSI with
> iSER should give you similar results with RDMA capable ethernet.
> Linbit (the people who write DRBD) have a non GPL extension to DRBD
> that uses RDMA so you can get more bandwidth that way as well.
I have 10G ethernet for the crossover between the two servers, and
another 10G ethernet to connect off to the "clients". Bandwidth
utilisation on either of these is rather low (I think it maxed out at
around 15 to 20%) definitely not anywhere near 100%. My thought here was
on the latency of the connection, but I really didn't have any ideas on
how to measure that, and how to test if it would really help. Also
equipment seems a little less common, and complex...
>> The drives report a sector size of 512k, which I guess means the smallest
>> meaningful write that the drive can do is 512k, so should I increase the
>> chunk size to 512k to match? Or does that make it even worse?
>> Finally, the drive reports Host_Writes_32MiB in SMART, does that mean that
>> the drive needs to replace a entire 32MB chunk in order to overwrite a
>> sector? I'm guessing a chunk size of 32M is just crazy though...
> This is probably not true. If the drive really had to update 512K at
> a time, then 4K writes would be 128x wear amplification. SSDs can be
> bad, but usually not that bad.
>
>> Is there a better way to actually measure the different sizes and quantity
>> of read/writes being issued, so that I can make a more accurate decision on
>> chunk size/stripe size/etc... iostat seems to show an average numbers, but
>> not the number of 1k read/write, 4k read/write, 16k read/write etc...
> The problem is that the FTL of the SSDs are a black box and as the
> array gets bigger, the slowest drive dictates the array performance.
> This is why the "big vendors" all map SSDs in the host and avoid or
> minimize writing randomly. I know of one vendor install that has 4000
> VDI seats (using ESXI as compute hosts) from a single HA pair of 24
> SSD shelves. The connection to ESXI is FC and the hosts are HA with
> an IB/SRP raid-1 link between them. Unfortunately, you need 500K+
> random write IOPS to pull this off, which I think is impossible with
> stock parity raid, and very hard with raid-10.
My environment is rather small in comparison, it is only around 20 VM's
supporting around 80 users. 5 of the VM's are RDP servers.
>
>> My suspicion is that the actual load is made up of rather small random
>> read/write, because that is the scenario that produced the worst performance
>> results when I was initially setting this up, and seems to be what we are
>> getting in practice.
>>
>> The last option is, what if I moved to RAID10? Would that provide a
>> significant performance boost (completely removes the need to worry about
>> chunk/stripe size because we always just write the exact data we want, no
>> need to read/compute/write)?
> RAID-10 will be faster, but you pay for this with capacity. It is
> also a double-edged sword as SSDs themselves run faster if you leave
> more free space on them, so RAID-10 absolutely might not be a lot
> faster than RAID-5 with some space left over. Also remember that free
> space on the SSDs only counts if it is actually unallocated. So you
> need to trim the SSDs or start with a secure erased drive and then
> never use the full capacity. It is best to leave an empty partition
> that is untouched.
Good point, when I initially provisioned the drives, I only used the
first 400GB, and left 80GB on each drive unpartitioned. As we ran out of
space, I was forced to allocate all of it. The place is to only end up
with 960GB of each 1000GB drive in use, so I could again leave a small
chunk of un-allocated space.
>> OR, is that read/compute overhead negligible since I'm using SSD and read
>> performance is so quick?
> The reads, especially with the pre 4.4 code or with raid-6 definitely
> take their toll. Most SSDs are also not quite symmetrical in terms of
> performance. If your SSD does 50K read IOPS and 50K write IOPS, it
> will probably not do 25K reads and 25K writes concurrently, but
> instead stop somewhere around 18K. But your mileage may vary. If you
> have 8 drives that do 20 read/write symmetric, with new raid-5, each
> 4K write is 2 reads and 2 writes. 8 drives will give you 8*20K = 160K
> reads and writes or 320K total OPS. Each 4K write takes 4 OPS, so
> your data rate ends up maxing out at 80K IOPS. With the old raid-5
> logic, you end up with 6 reads plus two writes per "OP", so you tend
> to max out around 320K/(6+2) = 40K IOPS. With more than 8 drives,
> these computations tend to fall apart, so 24 SSD arrays are not 3x
> faster than 8 SSD arrays, at least with stock code.
What if I moved to RAID50 and split my 8 disks into 2 x 4 disk RAID5 and
then combined to RAID0 (or linear)? I'd end up with 6TB of usable space
(8 x 1TB - 2 parity) though I'm guessing it is better to upgrade to
kernel 4.4 instead which would basically do the same thing?
> You also need to consider what raid does to the SSD FTL. As you
> chatter a drive, its wear goes up and its performance goes down.
> Different SSD models can vary wildly, but again the rule of thumb is
> keep as much free space as possible on the drives. raid-5 or
> mirroring is also 2:1 write amplification (ie, you are writing two
> drives) and raid-6 is 3:1, on top of whatever the FTL write
> amplification is at the time.
Overall drive wear is doing pretty well, it is sitting at around 5% to
8% per year.
Tell me I'm crazy, but one option that I considered is using different
RAID levels. Right now I have RAID51 in that I have RAID5 on each
machine and DRBD (RAID1) between them.
What if I used RAID01 with DRBD between the machines doing the RAID1. In
this way, each machine has RAID0 (across 8 drives), which should provide
maximum performance and storage capacity and DRBD doing RAID1 between
the two machines. It feels rather risky, but perhaps it isn't a terrible
idea?
Slightly better would be RAID10 with DRBD between each pair of drives,
and then RAID0 across the DRBD device. It adds another layer of RAID,
and more complexity, but better security than RAID01...
Regards,
Adam
^ permalink raw reply [flat|nested] 18+ messages in thread
* Re: RAID5 Performance
[not found] ` <7af0cc98-e395-9446-05eb-a6c0ca20f187@websitemanagers.com.au>
@ 2016-07-28 0:11 ` Doug Dumitru
2016-07-28 13:08 ` Anthony Youngman
2016-07-28 14:10 ` Adam Goryachev
0 siblings, 2 replies; 18+ messages in thread
From: Doug Dumitru @ 2016-07-28 0:11 UTC (permalink / raw)
To: Adam Goryachev; +Cc: linux-raid@vger.kernel.org
On Wed, Jul 27, 2016 at 4:25 PM, Adam Goryachev
<mailinglists@websitemanagers.com.au> wrote:
>
>
> On 27/07/2016 15:36, Doug Dumitru wrote:
>
> On Tue, Jul 26, 2016 at 7:24 PM, Adam Goryachev
> <mailinglists@websitemanagers.com.au> wrote:
>
> Hi all,
>
> I know, age old question, but I have the chance to change things up a bit,
> and I wanted to collect some thoughts/ideas.
>
> Currently I am using 8 x 480GB Intel SSD in a RAID5, then LVM on top, DRBD
> on top, and finally iSCSI on top (and then used as VM raw disks for mostly
> windows VM's).
>
> My current array looks like this:
>
> /dev/md1:
> Version : 1.2
> Creation Time : Wed Aug 22 00:47:03 2012
> Raid Level : raid5
> Array Size : 3281935552 (3129.90 GiB 3360.70 GB)
> Used Dev Size : 468847936 (447.13 GiB 480.10 GB)
> Raid Devices : 8
> Total Devices : 8
> Persistence : Superblock is persistent
>
> Update Time : Wed Jul 27 11:32:00 2016
> State : active
> Active Devices : 8
> Working Devices : 8
> Failed Devices : 0
> Spare Devices : 0
>
> Layout : left-symmetric
> Chunk Size : 64K
>
> Name : san1:1 (local to host san1)
> UUID : 707957c0:b7195438:06da5bc4:485d301c
> Events : 2185221
>
> Number Major Minor RaidDevice State
> 7 8 65 0 active sync /dev/sde1
> 13 8 1 1 active sync /dev/sda1
> 8 8 81 2 active sync /dev/sdf1
> 5 8 113 3 active sync /dev/sdh1
> 9 8 97 4 active sync /dev/sdg1
> 12 8 17 5 active sync /dev/sdb1
> 10 8 49 6 active sync /dev/sdd1
> 11 8 33 7 active sync /dev/sdc1
>
> I've configured the following non-standard options:
>
> echo 4096 > /sys/block/md1/md/stripe_cache_size
>
> The following apply to all SSD's installed:
> echo noop > $disk/queue/scheduler
> echo 128 > ${disk}/queue/nr_requests
>
> What I can measure (at peak periods) with iostat:
> Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz
> avgqu-sz await r_await w_await svctm %util
> sdi 0.00 0.00 0.00 0.00 0.00 0.00 0.00
> 0.00 0.00 0.00 0.00 0.00 0.00
> sda 78.00 59.00 79.00 86.00 0.74 0.52 15.55
> 0.02 0.15 0.20 0.09 0.15 2.40
> sdg 35.00 48.00 68.00 79.00 0.52 0.44 13.39
> 0.02 0.14 0.24 0.05 0.11 1.60
> sdf 46.00 65.00 86.00 98.00 0.76 0.58 14.96
> 0.03 0.17 0.09 0.24 0.09 1.60
> sdh 97.00 45.00 70.00 141.00 0.66 0.68 12.96
> 0.08 0.36 0.29 0.40 0.34 7.20
> sde 101.00 75.00 87.00 94.00 0.79 0.61 15.76
> 0.08 0.42 0.32 0.51 0.29 5.20
> sdb 85.00 54.00 94.00 102.00 0.84 0.56 14.62
> 0.01 0.04 0.09 0.00 0.04 0.80
> sdc 85.00 74.00 98.00 106.00 0.79 0.66 14.53
> 0.01 0.06 0.04 0.08 0.04 0.80
> sdd 230.00 199.00 266.00 353.00 2.19 2.11 14.24
> 0.18 0.28 0.23 0.32 0.16 9.60
> drbd0 0.00 0.00 0.00 2.00 0.00 0.00 4.50
> 0.08 38.00 0.00 38.00 20.00 4.00
> drbd12 0.00 0.00 1.00 1.00 0.00 0.00 7.50
> 0.03 14.00 4.00 24.00 14.00 2.80
> drbd1 0.00 0.00 0.00 2.00 0.00 0.03 32.00
> 0.09 44.00 0.00 44.00 22.00 4.40
> drbd9 0.00 0.00 2.00 0.00 0.01 0.00 8.00
> 0.00 0.00 0.00 0.00 0.00 0.00
> drbd2 0.00 0.00 0.00 0.00 0.00 0.00 0.00
> 0.00 0.00 0.00 0.00 0.00 0.00
> drbd11 0.00 0.00 0.00 0.00 0.00 0.00 0.00
> 0.00 0.00 0.00 0.00 0.00 0.00
> drbd3 0.00 0.00 4.00 197.00 0.02 1.01 10.47
> 7.92 41.03 0.00 41.87 4.98 100.00
> drbd4 0.00 0.00 0.00 0.00 0.00 0.00 0.00
> 0.00 0.00 0.00 0.00 0.00 0.00
> drbd17 0.00 0.00 1.00 0.00 0.00 0.00 8.00
> 0.00 0.00 0.00 0.00 0.00 0.00
> drbd5 0.00 0.00 0.00 7.00 0.00 0.03 8.00
> 0.22 30.29 0.00 30.29 28.57 20.00
> drbd19 0.00 0.00 0.00 0.00 0.00 0.00 0.00
> 0.00 0.00 0.00 0.00 0.00 0.00
> drbd6 0.00 0.00 2.00 0.00 0.01 0.00 8.00
> 0.00 0.00 0.00 0.00 0.00 0.00
> drbd7 0.00 0.00 0.00 0.00 0.00 0.00 0.00
> 0.00 0.00 0.00 0.00 0.00 0.00
> drbd8 0.00 0.00 0.00 0.00 0.00 0.00 0.00
> 0.00 0.00 0.00 0.00 0.00 0.00
> drbd13 0.00 0.00 90.00 44.00 1.74 0.38 32.35
> 1.72 13.46 0.40 40.18 4.27 57.20
> drbd15 0.00 0.00 2.00 33.00 0.02 0.29 17.86
> 1.40 40.91 0.00 43.39 28.34 99.20
> drbd18 0.00 0.00 1.00 3.00 0.00 0.03 16.00
> 0.08 21.00 0.00 28.00 21.00 8.40
> drbd14 0.00 0.00 0.00 0.00 0.00 0.00 0.00
> 0.00 0.00 0.00 0.00 0.00 0.00
> drbd10 0.00 0.00 0.00 0.00 0.00 0.00 0.00
> 0.00 0.00 0.00 0.00 0.00 0.00
>
> As you can see, the DRBD devices are busy, and slowing down the VM's,
> looking at the drives on the second server we can see why:
> Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz
> avgqu-sz await r_await w_await svctm %util
> sdf 67.00 76.00 64.00 113.00 0.52 0.62 13.17
> 0.26 1.47 0.06 2.27 1.45 25.60
> sdg 39.00 61.00 50.00 114.00 0.35 0.56 11.38
> 0.45 2.76 0.08 3.93 2.71 44.40
> sdd 49.00 67.00 50.00 109.00 0.39 0.57 12.40
> 0.75 4.73 0.00 6.90 4.70 74.80
> sdh 55.00 54.00 52.00 104.00 0.42 0.51 12.12
> 0.81 5.21 0.23 7.69 5.13 80.00
> sde 67.00 67.00 75.00 129.00 0.56 0.65 12.13
> 0.94 4.59 0.69 6.85 4.24 86.40
> sda 64.00 76.00 58.00 109.00 0.48 0.61 13.29
> 0.84 5.03 0.21 7.60 4.89 81.60
> sdb 35.00 72.00 57.00 104.00 0.36 0.57 11.84
> 0.69 4.27 0.14 6.54 4.22 68.00
> sdc 118.00 144.00 228.00 269.00 1.39 1.50 11.92
> 1.21 2.43 1.88 2.90 1.50 74.40
> md1 0.00 0.00 0.00 260.00 0.00 1.70 13.38
> 0.00 0.00 0.00 0.00 0.00 0.00
>
> I've confirmed that the problem is that we have mixed two models of SSD (520
> series and 530 series), and that the 530 series drives perform significantly
> worse (under load) in comparison. Above, the two 520 series are sdf and sdg
> while the other drives are 530 series. So, we will be replacing all of the
> drives across both systems with 545s series 1000GB SSD's (which I've
> confirmed will operate same or better than the 520 series, sdc on the first
> machine above is one of these already).
>
> Over the years, I've learned a lot about RAID and optimisation, originally I
> configured things to optimise for super fast streaming reads and streaming
> writes, but in practice, the actual work-load is small random read/write,
> with the writes causing the biggest load.
>
> Looking at this:
> http://serverfault.com/questions/384273/optimizing-raid-5-for-backuppc-use-small-random-reads
>
> *
>
> Enhance the queue depth. Standard kernel queue depth is OK for old
> single drives with small caches, but not for modern drives or RAID
> arrays:
>
> echo 512 > /sys/block/sda/queue/nr_requests
>
> So my question is should I increase the configured nr_requests above the
> current 128?
>
> With your workload, it probably won't matter too much. Really high
> queue depths are great on paper, but hard to actually see.
>
>
> Is there some way to see if this would help or not?
> Would it hurt to increase this (even if it doesn't help)?
>
>
> If the chunk size is 64k, and there are 8 drives in total, then the stripe
> size is currently 64k*7 = 448k, is this too big? My reading of the mdadm man
> page suggests the minimum chunk size is 4k ("In any case it must be a
> multiple of 4KB"). If I set the chunk size to 4k, then the stripe size
> becomes 28k, which means for a random 4k write, we only need to write 28k
> instead of 448k ?
>
> This is not how a random write works. If you are running raid-5
> before the 4.4 kernel, you get the "old" read/modify/write algorithm.
> If you write 4K, the system will read 4K from (n-2) drives, add in
> your 4K to compute parity, and write 2 drives. This is n-2 reads + 2
> writes. With the "new" logic in 4.4, you read the old contents of the
> 4K plus parity, and re-write the 4k plus parity, so there are 2 reads
> and 2 writes. With big arrays, the "new" logic can help quite a bit,
> but the chatter rate is still high. Note that the new logic is only
> raid-5. raid-6 cannot use the new logic and has to read the stripe
> from every drive.
>
> Hmmm, so an upgrade to kernel 4.6.3 (debian backports version) should
> provide a significant performance boost even if nothing else changes.
This should help your raid-5 array, at least noticeably, provided the
new kernel actually has the
Facebook Read/Modify/Write new logic included. Based on the version
it should. You can very this by doing random writes and looking at
iostat. If you see 2 reads and 2 writes for every inbound write, you
have the new code. If you see 6 reads and 2 writes for every inbound
write, you have the old code.
While this sounds huge, the change will be moderated by the behaviour
of SSDs. Random writes are much more expensive than read and the new
logic only lowers the number of reads. ... and raid-6 is not impacted
at all.
> The stripe size impacts when the system does can avoid doing a
> read/modify/write. If you write a full stripe [ 64K * (n-1) ], and
> the write is exactly on a stripe boundary, and you get lucky and the
> background thread does not wake up at just the wrong time, you will do
> the write with zero reads. I personally run with very small chunks,
> but I have code that always writes perfect stripe writes and stock
> file systems don't act that way.
>
> So reducing the chunk size will have minimal impact... but reducing it
> should still provide some performance boost. Since I'm recreating the array
> anyway, what size makes the most sense? 16k or go straight to the minimum of
> 4k? Would a smaller chunk size increase the IOPS because we need to make
> more (smaller) requests for the same data, potentially from more drives?
>
> ie, currently, a single read request for 4k will be done by reading one
> chunk (64k) from one of the 8 drives (1 IOPS)
> currently, a single write request for 4k will be done by reading one chunk
> (64k) from 6 drives, and then writing one chunk (64k) to two drives (8 IOPS)
> However, a read (or write) 48k request would be identical to the above,
> while a smaller chunk size (4k) would mean:
> read request - reading 2 x 4k chunks from 5 disks and 1 x 4k chunk from 2
> disks (7 IOPS)
> write request - write 8 x 4k (full stripe) (assuming it is stripe aligned
> somewhere, but it might not be)
> - read 2 x 4k chunks (the only 2 data chunks that
> won't be written) + write 6 x 4k chunks
> Total of 16 IOPS in the best case, worst case is two partial stripe writes +
> 1 full stripe write in the middle: 8 reads + 16 writes or 24 IOPS.
You are confused about what chunk size is. It is not the IO size
limit. It is just a layout calculation. If your chunk is 64K, then
64K is written to one disk before the array moves on to the next disk.
If you read 4K, then only 4K is read. You never need to read (or
write) and entire chunk.
Lower chunk sizes are useful if your application does enough long
writes to reach full stripes. At 64K x 7 drives, this is 448KB. If
you are writing multi-megabytes, then 64K chunks is a good idea. If
you are writing 128KB, you might want to go down to 16KB chunks. The
problem with little chunks is if you read 64K from and array with 16KB
chunks, you will cut your IO request into four parts. This is
sometimes faster and sometimes slower. For hard disks, bigger chunks
seems to be the way to go. For SSDs, smaller. I think 16K is
probably the lowest reasonable limit unless you have tested your
workload extensively, and over a long period of time, and have looked
at drive wear issues.;
> Either the above is wrong, or I've just convinced myself that reducing the
> chunk size is not a good idea...
>
> DRBD can saturate GigE without any problem with random 4K writes. I
> have a pair of systems here that pushes 110 MB/sec at 4K or 28,000
> IOPS. The target arrays needs to keep up, but that is another story.
> My testing with DRBD is that it starts to peter out at 10Gig, so if
> you want more bandwidth you need some other approach. Some vendors
> use SRP over Infiniband with software raid-1 as a mirror. iSCSI with
> iSER should give you similar results with RDMA capable ethernet.
> Linbit (the people who write DRBD) have a non GPL extension to DRBD
> that uses RDMA so you can get more bandwidth that way as well.
>
> I have 10G ethernet for the crossover between the two servers, and another
> 10G ethernet to connect off to the "clients". Bandwidth utilisation on
> either of these is rather low (I think it maxed out at around 15 to 20%)
> definitely not anywhere near 100%. My thought here was on the latency of the
> connection, but I really didn't have any ideas on how to measure that, and
> how to test if it would really help. Also equipment seems a little less
> common, and complex...
I know that DRBD will not hit 40G. I have actually not done that much
testing at 10G.
> The drives report a sector size of 512k, which I guess means the smallest
> meaningful write that the drive can do is 512k, so should I increase the
> chunk size to 512k to match? Or does that make it even worse?
> Finally, the drive reports Host_Writes_32MiB in SMART, does that mean that
> the drive needs to replace a entire 32MB chunk in order to overwrite a
> sector? I'm guessing a chunk size of 32M is just crazy though...
>
> This is probably not true. If the drive really had to update 512K at
> a time, then 4K writes would be 128x wear amplification. SSDs can be
> bad, but usually not that bad.
>
> Is there a better way to actually measure the different sizes and quantity
> of read/writes being issued, so that I can make a more accurate decision on
> chunk size/stripe size/etc... iostat seems to show an average numbers, but
> not the number of 1k read/write, 4k read/write, 16k read/write etc...
>
> The problem is that the FTL of the SSDs are a black box and as the
> array gets bigger, the slowest drive dictates the array performance.
> This is why the "big vendors" all map SSDs in the host and avoid or
> minimize writing randomly. I know of one vendor install that has 4000
> VDI seats (using ESXI as compute hosts) from a single HA pair of 24
> SSD shelves. The connection to ESXI is FC and the hosts are HA with
> an IB/SRP raid-1 link between them. Unfortunately, you need 500K+
> random write IOPS to pull this off, which I think is impossible with
> stock parity raid, and very hard with raid-10.
>
>
> My environment is rather small in comparison, it is only around 20 VM's
> supporting around 80 users. 5 of the VM's are RDP servers.
>
>
> My suspicion is that the actual load is made up of rather small random
> read/write, because that is the scenario that produced the worst performance
> results when I was initially setting this up, and seems to be what we are
> getting in practice.
>
> The last option is, what if I moved to RAID10? Would that provide a
> significant performance boost (completely removes the need to worry about
> chunk/stripe size because we always just write the exact data we want, no
> need to read/compute/write)?
>
> RAID-10 will be faster, but you pay for this with capacity. It is
> also a double-edged sword as SSDs themselves run faster if you leave
> more free space on them, so RAID-10 absolutely might not be a lot
> faster than RAID-5 with some space left over. Also remember that free
> space on the SSDs only counts if it is actually unallocated. So you
> need to trim the SSDs or start with a secure erased drive and then
> never use the full capacity. It is best to leave an empty partition
> that is untouched.
>
> Good point, when I initially provisioned the drives, I only used the first
> 400GB, and left 80GB on each drive unpartitioned. As we ran out of space, I
> was forced to allocate all of it. The place is to only end up with 960GB of
> each 1000GB drive in use, so I could again leave a small chunk of
> un-allocated space.
>
> OR, is that read/compute overhead negligible since I'm using SSD and read
> performance is so quick?
>
> The reads, especially with the pre 4.4 code or with raid-6 definitely
> take their toll. Most SSDs are also not quite symmetrical in terms of
> performance. If your SSD does 50K read IOPS and 50K write IOPS, it
> will probably not do 25K reads and 25K writes concurrently, but
> instead stop somewhere around 18K. But your mileage may vary. If you
> have 8 drives that do 20 read/write symmetric, with new raid-5, each
> 4K write is 2 reads and 2 writes. 8 drives will give you 8*20K = 160K
> reads and writes or 320K total OPS. Each 4K write takes 4 OPS, so
> your data rate ends up maxing out at 80K IOPS. With the old raid-5
> logic, you end up with 6 reads plus two writes per "OP", so you tend
> to max out around 320K/(6+2) = 40K IOPS. With more than 8 drives,
> these computations tend to fall apart, so 24 SSD arrays are not 3x
> faster than 8 SSD arrays, at least with stock code.
>
> What if I moved to RAID50 and split my 8 disks into 2 x 4 disk RAID5 and
> then combined to RAID0 (or linear)? I'd end up with 6TB of usable space (8 x
> 1TB - 2 parity) though I'm guessing it is better to upgrade to kernel 4.4
> instead which would basically do the same thing?
>
> You also need to consider what raid does to the SSD FTL. As you
> chatter a drive, its wear goes up and its performance goes down.
> Different SSD models can vary wildly, but again the rule of thumb is
> keep as much free space as possible on the drives. raid-5 or
> mirroring is also 2:1 write amplification (ie, you are writing two
> drives) and raid-6 is 3:1, on top of whatever the FTL write
> amplification is at the time.
>
> Overall drive wear is doing pretty well, it is sitting at around 5% to 8%
> per year.
>
> Tell me I'm crazy, but one option that I considered is using different RAID
> levels. Right now I have RAID51 in that I have RAID5 on each machine and
> DRBD (RAID1) between them.
> What if I used RAID01 with DRBD between the machines doing the RAID1. In
> this way, each machine has RAID0 (across 8 drives), which should provide
> maximum performance and storage capacity and DRBD doing RAID1 between the
> two machines. It feels rather risky, but perhaps it isn't a terrible idea?
> Slightly better would be RAID10 with DRBD between each pair of drives, and
> then RAID0 across the DRBD device. It adds another layer of RAID, and more
> complexity, but better security than RAID01...
Your 5 to 7% wear per year is pretty safe. I have a pair of systems
with proprietary code that is saturating dual 10GigE ports looking at
wearout at 100+ years. Then again, the plastic cases of the drives
will be dust by then.
I don't know about you, but I do have SSDs, even from major vendors,
that fail. They usually "just fall off the bus" with no warning. So
I dislike skipping redundancy. RAID turned an emergency into a
mundane task. It is really a cost issue. If you can afford RAID-10
and extra space, that will work best. I don't think RAID-50 with this
few drives makes much sense.
Doug
>
> Regards,
> Adam
>
>
--
Doug Dumitru
EasyCo LLC
^ permalink raw reply [flat|nested] 18+ messages in thread
* Re: RAID5 Performance
2016-07-27 17:38 ` Doug Dumitru
@ 2016-07-28 12:19 ` Peter Grandi
2016-07-28 13:28 ` Peter Grandi
2016-07-28 13:50 ` Adam Goryachev
0 siblings, 2 replies; 18+ messages in thread
From: Peter Grandi @ 2016-07-28 12:19 UTC (permalink / raw)
To: Linux RAID
[ ... ]
>> A very brave configuration, a shining example of the
>> "syntactic" mindset, according to which any arbitrary
>> combination of legitimate features must be fine :-).
> While you may say that this configuration is very "brave", it
> is actually quite common for VDI "appliance" deployments. [
> ... ]
There are a lot of very "brave" sysadms out there, and often I
have to clean up after them :-).
But then I am one of those boring people who think that «VDI
"appliance" deployments» are usually a phenomenally bad idea, as
it requires a storage layer that has to cover all possible IO
workloads optimally, as indeed in:
> [ ... ] The expectation, in terms of performance for VDI is
> quite high. vmWare like to say you can get away with 8-12
> IOPS per virtual. Most people think you only get good
> performance with 100 IOPS per virtual. [ ... ]
Those 100 random IOPS per VM are a bit "random", but roughly
translate to one "disk arm" per VM, which is not necessarily
enough: http://www.sabi.co.uk/blog/15-one.html#150305
[ ... ]
>> The queue sizes and waiting time on the second server are
>> very low (on a somewhat similar system using 4TB disks I see
>> waiting times in the 1-5 seconds range, not milliseconds).
> The expectation, in terms of performance for VDI is quite high.
> [ ... ]
Sure, but the point here as to the speed issue is not that the
SSDs are overwhelmed with IO, as the traffic on them is low and
has relatively low latency, it is that very few IOPS are getting
retired.
>> Thus the most likely issue here is the 'fsync' problem: for
>> "consumerish" SSDs barrier-writes are synchronous, because
>> they don't have a battery/capacitor-backed cache, and rather
>> slow for small writes, because of the large size of erase
>> blocks, which can be mitigated with higher over-provisioning.
> On many consumer SSDs, barrier writes are only barriers, and
> are not syncs at all. You are guaranteed serialization but not
> actual storage.
Probably in this case that is irrelevant, because the numbers
coming out from both the OP's experience and the tests in the
links I mentioned show that small sync writes seem synchronous
indeed for the 520/530, resulting in small write rates of around
1-5 MB/s, which matches the reported stats.
> Then again, in a server setup, especially with redundant power
> supplies, power loss to the SSDs is rare. You are more
> protecting against system hangs and other inter-connectivity
> issues.
That is also likely irrelevant here. The firmware in the flash
SSD does not know about the system setup, and the DRBD is
probably configured to request synchronous writes on the
secondary with protocol "C".
BTW I don't know whether the process(es) writing to the DRBD
primary also request synchronous writes, but that's hopefully the
case too, if the VD layer has been configured properly.
> The real system solution is to have some quantity of non
> volatile DRAM that you can stage writes (either a PCI-e card
> like a FlashTec or one or more nvDIMMs).
If this were the case then the VD layer and the DRBD layer could
be told not to use sync writes, but the numbers reported seem to
indicate that sync writes are happening.
> This is how the "major vendors" deal with sync writes.
At the system level, but at the device level the "major vendors"
put a large capacitor in "enterprise" SSDs for two reasons, one
of them to allow the persistence of the RAM write buffer, to
minimize write amplification and erase latency (the other is not
relevant here).
[ ... ]
>> * Small writes are a very challenging workload for flash SSDs
>> without battery/capacitor-backed caches.
> Even with battery backup, small writes create garbage
> collection, so while batteries may give you some short term
> bursts,
That problem is mitigated with bigger overprovisioning in
"enterprise" class flash SSDs. It can also be done in those of
the "consumerish" class by partitioning them appropriately, or
with 'hdparm -N'; but that does not seem to be the case here,
becase the reported stats show a small number of IOPS with lowish
queues sizes and not that huge latencies.
> longer term, you still have to do the writes.
Unfortunately flash SSDs don't merely have to "do the writes",
as things are quite different: as I mentioned above the issue is
the large erase blocks (and the several milliseconds it takes to
erase one).
In the absence of power backing for the write cache, every sync
write, for example a 4KiB one, is (usually) stored immediately to
a flash chip, which means (usually) a lot of write amplification
because of RMW on the 8MiB (or larger) erase block plus the large
latency (often near 10 milliseconds) of the erase operation
before erase block programming.
That largely explains why in the tests I have mentioned small
sync write IOPS for many "consumerish" flash SSDs top at around
100, instead of the usual > 10,000 for small non-sync writes.
Some flash SSDs use an additional SLC buffer with smaller erase
blocks and lower latency to reduce the problem with flushing sync
writes directly to MLC etc, and that may explain why the 520s are
better than the 530s (if the 520s have an SLC buffer, but IIRC
intel started using an SLC buffer with the 540 series).
Flash SSDs have only been popular for around 5 years, so it is
understandable that some important aspects of their performance
envelope (like what may happen on sync writes) is not well known
yet.
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply [flat|nested] 18+ messages in thread
* Re: RAID5 Performance
2016-07-28 0:11 ` Doug Dumitru
@ 2016-07-28 13:08 ` Anthony Youngman
2016-07-28 14:10 ` Adam Goryachev
1 sibling, 0 replies; 18+ messages in thread
From: Anthony Youngman @ 2016-07-28 13:08 UTC (permalink / raw)
To: doug, Adam Goryachev; +Cc: linux-raid@vger.kernel.org
On 28/07/16 01:11, Doug Dumitru wrote:
> I don't know about you, but I do have SSDs, even from major vendors,
> that fail. They usually "just fall off the bus" with no warning. So
> I dislike skipping redundancy. RAID turned an emergency into a
> mundane task. It is really a cost issue. If you can afford RAID-10
> and extra space, that will work best. I don't think RAID-50 with this
> few drives makes much sense.
I came across an article about testing SSDs to destruction. First the
good news - they tended to last much longer than expected. And the bad
news? They typically contain a self-destruct switch. Once they start
failing, a power-cycle will (intentionally) kill them dead. ESPECIALLY
if they're from a major vendor.
So if you don't notice they're dying, or (as in the case of the tester)
you have power problems that tip them over the edge, your data WILL be
gone without warning.
Cheers,
Wol
^ permalink raw reply [flat|nested] 18+ messages in thread
* Re: RAID5 Performance
2016-07-28 12:19 ` Peter Grandi
@ 2016-07-28 13:28 ` Peter Grandi
2016-07-28 13:57 ` Adam Goryachev
2016-07-28 13:50 ` Adam Goryachev
1 sibling, 1 reply; 18+ messages in thread
From: Peter Grandi @ 2016-07-28 13:28 UTC (permalink / raw)
To: Linux RAID
[ ... ]
> That largely explains why in the tests I have mentioned small
> sync write IOPS for many "consumerish" flash SSDs top at around
> 100, instead of the usual > 10,000 for small non-sync writes.
[ ... ]
To summarize the preceding long discussion:
* The stats reported show a low level of IOPS being carried out.
* The critical part of the workload seems to be synchronous
small writes.
* Probably then the primary issue is the use of flash SSDs that
have a limited number of IOPS for small synchronous writes.
* A secondary issue is that RAID5 results in RMW for small
writes.
There are two possible options:
* Replace the flash SSDs with those that are known to deliver
high (at least > 10,000 single threaded) small synchronous
write IOPS.
* Relax the requirement for synchronous writes on *both* the
primary and secondary DRBD servers, if feeling lucky.
The third option, which is to change the workload so that it
does not emit small synchronous writes to the storage layer,
seems not practical in the context.
Ideally the system would also be switched from RAID5 to RAID10
to avoid the large penalty on small writes at the RAID level
too.
That may be considered expensive, but as I wrote:
> [ ... ] it requires a storage layer that has to cover all
> possible IO workloads optimally, [ ... ]
^ permalink raw reply [flat|nested] 18+ messages in thread
* Re: RAID5 Performance
2016-07-28 12:19 ` Peter Grandi
2016-07-28 13:28 ` Peter Grandi
@ 2016-07-28 13:50 ` Adam Goryachev
1 sibling, 0 replies; 18+ messages in thread
From: Adam Goryachev @ 2016-07-28 13:50 UTC (permalink / raw)
To: Peter Grandi, Linux RAID
On 28/07/2016 22:19, Peter Grandi wrote:
> [ ... ]
>
>>> A very brave configuration, a shining example of the
>>> "syntactic" mindset, according to which any arbitrary
>>> combination of legitimate features must be fine :-).
>> While you may say that this configuration is very "brave", it
>> is actually quite common for VDI "appliance" deployments. [
>> ... ]
> There are a lot of very "brave" sysadms out there, and often I
> have to clean up after them :-).
>
> But then I am one of those boring people who think that «VDI
> "appliance" deployments» are usually a phenomenally bad idea, as
> it requires a storage layer that has to cover all possible IO
> workloads optimally, as indeed in:
Could I ask what you would "clean up" in the above system? What layers
would you remove/simplify?
At it's simplest, this system should be able to export a block of disk
space which the remote machine can present as a block device to the VM
(using Xen). Keep in mind there are actually multiple of these remote
machines. The method I've chosen is re-written here:
8 x 480GB Intel SSD (mix of 520 and 530 models)
Linux MD RAID5
LVM2
DRBD (takes an LV from each san and joins it together)
iSCSI (exports the block device to the 10Gbps network)
iSCSI (imported to the remove machine 2 x 1Gbps network)
multipathd (join the two iSCSI connections together)
Xen
Also, the two san machines have a second 10Gbps connection directly
between them.
Originally I had DRBD below LVM, but the folks at linbit switched those
two around to improve DRBD performance (multiple smaller DRBD devices is
better than one large, this might have changed since that happened
around 3 years ago).
> > [ ... ] The expectation, in terms of performance for VDI is
> > quite high. vmWare like to say you can get away with 8-12
> > IOPS per virtual. Most people think you only get good
> > performance with 100 IOPS per virtual. [ ... ]
>
> Those 100 random IOPS per VM are a bit "random", but roughly
> translate to one "disk arm" per VM, which is not necessarily
> enough: http://www.sabi.co.uk/blog/15-one.html#150305
>
> [ ... ]
>
>>> The queue sizes and waiting time on the second server are
>>> very low (on a somewhat similar system using 4TB disks I see
>>> waiting times in the 1-5 seconds range, not milliseconds).
>> The expectation, in terms of performance for VDI is quite high.
>> [ ... ]
> Sure, but the point here as to the speed issue is not that the
> SSDs are overwhelmed with IO, as the traffic on them is low and
> has relatively low latency, it is that very few IOPS are getting
> retired.
>
Previously when I was doing lots of tests on the system, I found I could
get great IO using larger block sizes, up to 2.5GB/s read and 1.5GB/s write.
Eventually, I found that using more real-world sized blocks, eg, 1k to
4k, I got abysmal transfer rates.
>>> Thus the most likely issue here is the 'fsync' problem: for
>>> "consumerish" SSDs barrier-writes are synchronous, because
>>> they don't have a battery/capacitor-backed cache, and rather
>>> slow for small writes, because of the large size of erase
>>> blocks, which can be mitigated with higher over-provisioning.
>> On many consumer SSDs, barrier writes are only barriers, and
>> are not syncs at all. You are guaranteed serialization but not
>> actual storage.
> Probably in this case that is irrelevant, because the numbers
> coming out from both the OP's experience and the tests in the
> links I mentioned show that small sync writes seem synchronous
> indeed for the 520/530, resulting in small write rates of around
> 1-5 MB/s, which matches the reported stats.
>
>> Then again, in a server setup, especially with redundant power
>> supplies, power loss to the SSDs is rare. You are more
>> protecting against system hangs and other inter-connectivity
>> issues.
> That is also likely irrelevant here. The firmware in the flash
> SSD does not know about the system setup, and the DRBD is
> probably configured to request synchronous writes on the
> secondary with protocol "C".
Yep, using protocol C at the moment.
> BTW I don't know whether the process(es) writing to the DRBD
> primary also request synchronous writes, but that's hopefully the
> case too, if the VD layer has been configured properly.
>
>> The real system solution is to have some quantity of non
>> volatile DRAM that you can stage writes (either a PCI-e card
>> like a FlashTec or one or more nvDIMMs).
> If this were the case then the VD layer and the DRBD layer could
> be told not to use sync writes, but the numbers reported seem to
> indicate that sync writes are happening.
>
>> This is how the "major vendors" deal with sync writes.
> At the system level, but at the device level the "major vendors"
> put a large capacitor in "enterprise" SSDs for two reasons, one
> of them to allow the persistence of the RAM write buffer, to
> minimize write amplification and erase latency (the other is not
> relevant here).
>
> [ ... ]
>
>>> * Small writes are a very challenging workload for flash SSDs
>>> without battery/capacitor-backed caches.
>> Even with battery backup, small writes create garbage
>> collection, so while batteries may give you some short term
>> bursts,
> That problem is mitigated with bigger overprovisioning in
> "enterprise" class flash SSDs. It can also be done in those of
> the "consumerish" class by partitioning them appropriately, or
> with 'hdparm -N'; but that does not seem to be the case here,
> becase the reported stats show a small number of IOPS with lowish
> queues sizes and not that huge latencies.
Can you advise what numbers I should look for, or worry about, which
would indicate that the problem is (or isn't) a erase cycle delay problem?
>> longer term, you still have to do the writes.
> Unfortunately flash SSDs don't merely have to "do the writes",
> as things are quite different: as I mentioned above the issue is
> the large erase blocks (and the several milliseconds it takes to
> erase one).
>
> In the absence of power backing for the write cache, every sync
> write, for example a 4KiB one, is (usually) stored immediately to
> a flash chip, which means (usually) a lot of write amplification
> because of RMW on the 8MiB (or larger) erase block plus the large
> latency (often near 10 milliseconds) of the erase operation
> before erase block programming.
>
> That largely explains why in the tests I have mentioned small
> sync write IOPS for many "consumerish" flash SSDs top at around
> 100, instead of the usual > 10,000 for small non-sync writes.
>
> Some flash SSDs use an additional SLC buffer with smaller erase
> blocks and lower latency to reduce the problem with flushing sync
> writes directly to MLC etc, and that may explain why the 520s are
> better than the 530s (if the 520s have an SLC buffer, but IIRC
> intel started using an SLC buffer with the 540 series).
I can see the 545s series performs similar (or better, can't tell yet)
than the 520 series, but certainly it is better than the 530.
>
> Flash SSDs have only been popular for around 5 years, so it is
> understandable that some important aspects of their performance
> envelope (like what may happen on sync writes) is not well known
> yet.
>
Thank you, I appreciate all the responses.
So far, I've decided to make the following two changes:
1) Replace all 16 existing SSD's with the 1000GB 545s model, this will
double the capacity, and remove any of the 530 model drives. My concern
was (and is) that making actual use of this double capacity with the
same performance per drive will in effect halve the performance. I will
be able to leave 40G per drive un-partitioned, or partitioned and left
blank whichever is better.... Not sure if 40G per drive is enough to
help with the write/erase problem, but I guess it should be better than
nothing.
I think this change will produce a 40% improvement (potentially, given
the 520 drives are at 40% while the 530 is at 100%)
2) Upgrade to Linux kernel 4.6 from debian backports.
I think this change will give approx 30% improvement, because it will
reduce reads by 4 for each write, but a read is quicker than a write, so
I'm hoping for 30% overall. It sounds like the performance should be
somewhere between current and RAID10.
With the above two improvements, I'm hoping it will be enough to solve
the problem.
At this stage, if it is not enough to solve the problem, my fall-back
option is to convert to RAID10 but it's something I'd prefer to avoid
based on cost, storage capacity, and the fact it is difficult to expand
the existing system past 8 drives (hence capacity)...
I'm not convinced that changing chunk size will offer any benefit
(positive or negative), so will likely leave that as it is (64k chunk size).
Regards,
Adam
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply [flat|nested] 18+ messages in thread
* Re: RAID5 Performance
2016-07-28 13:28 ` Peter Grandi
@ 2016-07-28 13:57 ` Adam Goryachev
2016-07-28 17:20 ` Peter Grandi
0 siblings, 1 reply; 18+ messages in thread
From: Adam Goryachev @ 2016-07-28 13:57 UTC (permalink / raw)
To: Peter Grandi, Linux RAID
On 28/07/2016 23:28, Peter Grandi wrote:
> [ ... ]
>> That largely explains why in the tests I have mentioned small
>> sync write IOPS for many "consumerish" flash SSDs top at around
>> 100, instead of the usual > 10,000 for small non-sync writes.
> [ ... ]
>
> To summarize the preceding long discussion:
>
> * The stats reported show a low level of IOPS being carried out.
>
> * The critical part of the workload seems to be synchronous
> small writes.
>
> * Probably then the primary issue is the use of flash SSDs that
> have a limited number of IOPS for small synchronous writes.
>
> * A secondary issue is that RAID5 results in RMW for small
> writes.
>
> There are two possible options:
>
> * Replace the flash SSDs with those that are known to deliver
> high (at least > 10,000 single threaded) small synchronous
> write IOPS.
Is there a "known" SSD that you would suggest? My problem is that Intel
spec sheets seem to suggest that there is little performance difference
across the range of SSD's, so it's really not clear which SSD model I
should buy. Obviously it's not something I can afford to buy one of each
and test them either.
> * Relax the requirement for synchronous writes on *both* the
> primary and secondary DRBD servers, if feeling lucky.
I have the following entries for DRBD which were suggested by linbit
(which previously lifted performance from abysmal to more than
sufficient around 2+ years ago). I guess we are demanding more from the
system now, and we have added the 530 model drives later...
disk-barrier no;
disk-flushes no;
md-flushes no;
I've not configured anything special for LVM/MD/iSCSI/xen in relation to
cache/buffer/etc, and windows has disabled the write back buffer option
(within the VM).
> The third option, which is to change the workload so that it
> does not emit small synchronous writes to the storage layer,
> seems not practical in the context.
>
> Ideally the system would also be switched from RAID5 to RAID10
> to avoid the large penalty on small writes at the RAID level
> too.
>
> That may be considered expensive, but as I wrote:
>
>> [ ... ] it requires a storage layer that has to cover all
>> possible IO workloads optimally, [ ... ]
That will be the final optimisation/fallback option if still needed. I'd
prefer to avoid that as it limits the capacity of the system, and will
obviously cost more.
Do you have any other suggestions or ideas that might assist?
Thanks,
Adam
^ permalink raw reply [flat|nested] 18+ messages in thread
* Re: RAID5 Performance
2016-07-28 0:11 ` Doug Dumitru
2016-07-28 13:08 ` Anthony Youngman
@ 2016-07-28 14:10 ` Adam Goryachev
2016-07-28 17:45 ` Peter Grandi
1 sibling, 1 reply; 18+ messages in thread
From: Adam Goryachev @ 2016-07-28 14:10 UTC (permalink / raw)
To: doug; +Cc: linux-raid@vger.kernel.org
On 28/07/2016 10:11, Doug Dumitru wrote:
> On Wed, Jul 27, 2016 at 4:25 PM, Adam Goryachev
> <mailinglists@websitemanagers.com.au> wrote:
>>
>> On 27/07/2016 15:36, Doug Dumitru wrote:
>>
>> On Tue, Jul 26, 2016 at 7:24 PM, Adam Goryachev
>> <mailinglists@websitemanagers.com.au> wrote:
>>
>> Hi all,
>>
>> I know, age old question, but I have the chance to change things up a bit,
>> and I wanted to collect some thoughts/ideas.
>>
>> Currently I am using 8 x 480GB Intel SSD in a RAID5, then LVM on top, DRBD
>> on top, and finally iSCSI on top (and then used as VM raw disks for mostly
>> windows VM's).
>>
>>
>> This should help your raid-5 array, at least noticeably, provided the
>> new kernel actually has the
>> Facebook Read/Modify/Write new logic included. Based on the version
>> it should. You can very this by doing random writes and looking at
>> iostat. If you see 2 reads and 2 writes for every inbound write, you
>> have the new code. If you see 6 reads and 2 writes for every inbound
>> write, you have the old code.
>>
>> While this sounds huge, the change will be moderated by the behaviour
>> of SSDs. Random writes are much more expensive than read and the new
>> logic only lowers the number of reads. ... and raid-6 is not impacted
>> at all.
Will definitely try this, it seems a pretty simple low-cost, and pretty
low risk option.
>> The stripe size impacts when the system does can avoid doing a
>> read/modify/write. If you write a full stripe [ 64K * (n-1) ], and
>> the write is exactly on a stripe boundary, and you get lucky and the
>> background thread does not wake up at just the wrong time, you will do
>> the write with zero reads. I personally run with very small chunks,
>> but I have code that always writes perfect stripe writes and stock
>> file systems don't act that way.
>>
>> So reducing the chunk size will have minimal impact... but reducing it
>> should still provide some performance boost. Since I'm recreating the array
>> anyway, what size makes the most sense? 16k or go straight to the minimum of
>> 4k? Would a smaller chunk size increase the IOPS because we need to make
>> more (smaller) requests for the same data, potentially from more drives?
>>
>> ie, currently, a single read request for 4k will be done by reading one
>> chunk (64k) from one of the 8 drives (1 IOPS)
>> currently, a single write request for 4k will be done by reading one chunk
>> (64k) from 6 drives, and then writing one chunk (64k) to two drives (8 IOPS)
>> However, a read (or write) 48k request would be identical to the above,
>> while a smaller chunk size (4k) would mean:
>> read request - reading 2 x 4k chunks from 5 disks and 1 x 4k chunk from 2
>> disks (7 IOPS)
>> write request - write 8 x 4k (full stripe) (assuming it is stripe aligned
>> somewhere, but it might not be)
>> - read 2 x 4k chunks (the only 2 data chunks that
>> won't be written) + write 6 x 4k chunks
>> Total of 16 IOPS in the best case, worst case is two partial stripe writes +
>> 1 full stripe write in the middle: 8 reads + 16 writes or 24 IOPS.
> You are confused about what chunk size is. It is not the IO size
> limit. It is just a layout calculation. If your chunk is 64K, then
> 64K is written to one disk before the array moves on to the next disk.
> If you read 4K, then only 4K is read. You never need to read (or
> write) and entire chunk.
>
> Lower chunk sizes are useful if your application does enough long
> writes to reach full stripes. At 64K x 7 drives, this is 448KB. If
> you are writing multi-megabytes, then 64K chunks is a good idea. If
> you are writing 128KB, you might want to go down to 16KB chunks. The
> problem with little chunks is if you read 64K from and array with 16KB
> chunks, you will cut your IO request into four parts. This is
> sometimes faster and sometimes slower. For hard disks, bigger chunks
> seems to be the way to go. For SSDs, smaller. I think 16K is
> probably the lowest reasonable limit unless you have tested your
> workload extensively, and over a long period of time, and have looked
> at drive wear issues.;
I'm not sure here.... since my issue seems to be IOPS, wouldn't
splitting a single IOP (ie, in your example the 64k read) into 4 IOPS (4
x 16k reads), which would seem to exacerbate the issue (not enough IOPS
available).
In which case, it could be beneficial to move to larger chunk sizes, so
even a 128k request can be kept as a single IOP instead of split into 2
? though there must also be an upper limit on the benefits here too?
At the moment, I'm thinking I will just leave the chunk size the same....
>> Either the above is wrong, or I've just convinced myself that reducing the
>> chunk size is not a good idea...
>>
>> DRBD can saturate GigE without any problem with random 4K writes. I
>> have a pair of systems here that pushes 110 MB/sec at 4K or 28,000
>> IOPS. The target arrays needs to keep up, but that is another story.
>> My testing with DRBD is that it starts to peter out at 10Gig, so if
>> you want more bandwidth you need some other approach. Some vendors
>> use SRP over Infiniband with software raid-1 as a mirror. iSCSI with
>> iSER should give you similar results with RDMA capable ethernet.
>> Linbit (the people who write DRBD) have a non GPL extension to DRBD
>> that uses RDMA so you can get more bandwidth that way as well.
>>
>> I have 10G ethernet for the crossover between the two servers, and another
>> 10G ethernet to connect off to the "clients". Bandwidth utilisation on
>> either of these is rather low (I think it maxed out at around 15 to 20%)
>> definitely not anywhere near 100%. My thought here was on the latency of the
>> connection, but I really didn't have any ideas on how to measure that, and
>> how to test if it would really help. Also equipment seems a little less
>> common, and complex...
> I know that DRBD will not hit 40G. I have actually not done that much
> testing at 10G.
My concern is that even if I solve *this* bottleneck (ie, the 530 model
SSD being too busy), that there will be another bottleneck afterwards
(well, of course there will be, there is always one piece that is
limiting performance). How will I know what/where it is (assuming it
isn't the SSD/raid itself....).
>> The drives report a sector size of 512k, which I guess means the smallest
>> meaningful write that the drive can do is 512k, so should I increase the
>> chunk size to 512k to match? Or does that make it even worse?
>> Finally, the drive reports Host_Writes_32MiB in SMART, does that mean that
>> the drive needs to replace a entire 32MB chunk in order to overwrite a
>> sector? I'm guessing a chunk size of 32M is just crazy though...
>>
>> This is probably not true. If the drive really had to update 512K at
>> a time, then 4K writes would be 128x wear amplification. SSDs can be
>> bad, but usually not that bad.
>>
>> Is there a better way to actually measure the different sizes and quantity
>> of read/writes being issued, so that I can make a more accurate decision on
>> chunk size/stripe size/etc... iostat seems to show an average numbers, but
>> not the number of 1k read/write, 4k read/write, 16k read/write etc...
>>
>> The problem is that the FTL of the SSDs are a black box and as the
>> array gets bigger, the slowest drive dictates the array performance.
>> This is why the "big vendors" all map SSDs in the host and avoid or
>> minimize writing randomly. I know of one vendor install that has 4000
>> VDI seats (using ESXI as compute hosts) from a single HA pair of 24
>> SSD shelves. The connection to ESXI is FC and the hosts are HA with
>> an IB/SRP raid-1 link between them. Unfortunately, you need 500K+
>> random write IOPS to pull this off, which I think is impossible with
>> stock parity raid, and very hard with raid-10.
>>
>>
>> My environment is rather small in comparison, it is only around 20 VM's
>> supporting around 80 users. 5 of the VM's are RDP servers.
>>
>>
>> My suspicion is that the actual load is made up of rather small random
>> read/write, because that is the scenario that produced the worst performance
>> results when I was initially setting this up, and seems to be what we are
>> getting in practice.
>>
>> The last option is, what if I moved to RAID10? Would that provide a
>> significant performance boost (completely removes the need to worry about
>> chunk/stripe size because we always just write the exact data we want, no
>> need to read/compute/write)?
>>
>> RAID-10 will be faster, but you pay for this with capacity. It is
>> also a double-edged sword as SSDs themselves run faster if you leave
>> more free space on them, so RAID-10 absolutely might not be a lot
>> faster than RAID-5 with some space left over. Also remember that free
>> space on the SSDs only counts if it is actually unallocated. So you
>> need to trim the SSDs or start with a secure erased drive and then
>> never use the full capacity. It is best to leave an empty partition
>> that is untouched.
>>
>> Good point, when I initially provisioned the drives, I only used the first
>> 400GB, and left 80GB on each drive unpartitioned. As we ran out of space, I
>> was forced to allocate all of it. The place is to only end up with 960GB of
>> each 1000GB drive in use, so I could again leave a small chunk of
>> un-allocated space.
>>
>> OR, is that read/compute overhead negligible since I'm using SSD and read
>> performance is so quick?
>>
>> The reads, especially with the pre 4.4 code or with raid-6 definitely
>> take their toll. Most SSDs are also not quite symmetrical in terms of
>> performance. If your SSD does 50K read IOPS and 50K write IOPS, it
>> will probably not do 25K reads and 25K writes concurrently, but
>> instead stop somewhere around 18K. But your mileage may vary. If you
>> have 8 drives that do 20 read/write symmetric, with new raid-5, each
>> 4K write is 2 reads and 2 writes. 8 drives will give you 8*20K = 160K
>> reads and writes or 320K total OPS. Each 4K write takes 4 OPS, so
>> your data rate ends up maxing out at 80K IOPS. With the old raid-5
>> logic, you end up with 6 reads plus two writes per "OP", so you tend
>> to max out around 320K/(6+2) = 40K IOPS. With more than 8 drives,
>> these computations tend to fall apart, so 24 SSD arrays are not 3x
>> faster than 8 SSD arrays, at least with stock code.
>>
>> What if I moved to RAID50 and split my 8 disks into 2 x 4 disk RAID5 and
>> then combined to RAID0 (or linear)? I'd end up with 6TB of usable space (8 x
>> 1TB - 2 parity) though I'm guessing it is better to upgrade to kernel 4.4
>> instead which would basically do the same thing?
>>
>> You also need to consider what raid does to the SSD FTL. As you
>> chatter a drive, its wear goes up and its performance goes down.
>> Different SSD models can vary wildly, but again the rule of thumb is
>> keep as much free space as possible on the drives. raid-5 or
>> mirroring is also 2:1 write amplification (ie, you are writing two
>> drives) and raid-6 is 3:1, on top of whatever the FTL write
>> amplification is at the time.
>>
>> Overall drive wear is doing pretty well, it is sitting at around 5% to 8%
>> per year.
>>
>> Tell me I'm crazy, but one option that I considered is using different RAID
>> levels. Right now I have RAID51 in that I have RAID5 on each machine and
>> DRBD (RAID1) between them.
>> What if I used RAID01 with DRBD between the machines doing the RAID1. In
>> this way, each machine has RAID0 (across 8 drives), which should provide
>> maximum performance and storage capacity and DRBD doing RAID1 between the
>> two machines. It feels rather risky, but perhaps it isn't a terrible idea?
>> Slightly better would be RAID10 with DRBD between each pair of drives, and
>> then RAID0 across the DRBD device. It adds another layer of RAID, and more
>> complexity, but better security than RAID01...
> Your 5 to 7% wear per year is pretty safe. I have a pair of systems
> with proprietary code that is saturating dual 10GigE ports looking at
> wearout at 100+ years. Then again, the plastic cases of the drives
> will be dust by then.
Yep, I expect that we will outgrow the capacity of the drives before
they "wear out". I do monitor the drive reported wear values, and alert
on those (each time it drops 10% I get alerts, until I reset the alert
level) so that I won't be suddenly surprised when they hit 10% or
whatever....
> I don't know about you, but I do have SSDs, even from major vendors,
> that fail. They usually "just fall off the bus" with no warning. So
> I dislike skipping redundancy. RAID turned an emergency into a
> mundane task. It is really a cost issue. If you can afford RAID-10
> and extra space, that will work best. I don't think RAID-50 with this
> few drives makes much sense.
>
I'm not sure, but I think I've had one of the 480GB drives fail, and 3
of the smaller 60GB and 80GB drives fail. So far, only the 480G failure
was "catastrophic", the others were still operating . All were replaced
by Intel.
The last one was just reported some large numbers in SMART, so I
questioned Intel, and their advice was replace under warranty, which I did.
I've had many more spinning disks fail over the years though, so I'm
pretty sure SSD's are more reliable, but certainly they do still fail,
and that's one of the reasons for RAID (and backups of course).
Regards,
Adam
Regards,
Adam
^ permalink raw reply [flat|nested] 18+ messages in thread
* Re: RAID5 Performance
2016-07-28 13:57 ` Adam Goryachev
@ 2016-07-28 17:20 ` Peter Grandi
2016-07-28 18:45 ` Doug Dumitru
2016-08-02 7:09 ` Adam Goryachev
0 siblings, 2 replies; 18+ messages in thread
From: Peter Grandi @ 2016-07-28 17:20 UTC (permalink / raw)
To: Linux RAID
[ ... ]
>> * Replace the flash SSDs with those that are known to deliver
>> high (at least > 10,000 single threaded) small synchronous
>> write IOPS.
> Is there a "known" SSD that you would suggest? My problem is
> that Intel spec sheets seem to suggest that there is little
> performance difference across the range of SSD's, so it's
> really not clear which SSD model I should buy.
The links I wrote earlier have lists:
>>> https://www.sebastien-han.fr/blog/2014/10/10/ceph-how-to-test-if-your-ssd-is-suitable-as-a-journal-device/
>>> http://www.spinics.net/lists/ceph-users/msg25928.html
>>> https://www.redhat.com/en/resources/ceph-pcie-ssd-performance-part-1
As one of those pages says the Samsung SM863 looks attractive,
but for historical reasons so far I have only seen Intel DCs in
similar use. There discussions of other models in various posts
related to Ceph journal SSD usage.
> Obviously it's not something I can afford to buy one of each
> and test them either.
In addition to the lists above I have justed tested my three
home flash SSDs:
* Micron M4 256GB:
# dd bs=4k count=100000 oflag=direct,dsync if=/dev/zero of=/var/tmp/TEST
100000+0 records in
100000+0 records out
409600000 bytes (410 MB) copied, 1200.3 s, 341 kB/s
* Samsung 850 Pro 256GB:
# dd bs=4k count=100000 oflag=direct,dsync if=/dev/zero of=/var/tmp/TEST
100000+0 records in
100000+0 records out
409600000 bytes (410 MB) copied, 1732.93 s, 236 kB/s
* Hynix SK SH910 256GB:
# dd bs=4k count=100000 oflag=direct,dsync if=/dev/zero of=/var/tmp/TEST
100000+0 records in
100000+0 records out
409600000 bytes (410 MB) copied, 644.742 s, 635 kB/s
So I would not recommend any of them for "small sync writes"
workloads :-), but they are quite good otherwise. I do notice
they are slow on small sync writes when downloading mail, as
each message is duly 'fsync'ed.
BTW as bonus material, I have done on the SH910 an abbreviated
test with block sizes between 4KiB and 1024KiB:
# for N in 4k 16k 64k 128k 256k 512k 1024k; do echo -n "$N: "; dd bs=$N count=1000 oflag=dsync if=/dev/zero of=/var/tmp/TEST |& grep copied; done
4k: 4096000 bytes (4.1 MB) copied, 6.23481 s, 657 kB/s
16k: 16384000 bytes (16 MB) copied, 6.29379 s, 2.6 MB/s
64k: 65536000 bytes (66 MB) copied, 6.09223 s, 10.8 MB/s
128k: 131072000 bytes (131 MB) copied, 6.5487 s, 20.0 MB/s
256k: 262144000 bytes (262 MB) copied, 6.93361 s, 37.8 MB/s
512k: 524288000 bytes (524 MB) copied, 7.73957 s, 67.7 MB/s
1024k: 1048576000 bytes (1.0 GB) copied, 12.8671 s, 81.5 MB/s
Note how the time to write 1000 blocks is essentially the same
betweeen 4KiB and 128KiB, which is quite amusing. Probably the
flash-page size is around 256KiB.
For additional bonus value the same on a "fastish" consumer 2TB
disk, a Seagate ST2000DM001:
# for N in 4k 16k 64k 128k 256k 512k 1024k; do echo -n "$N: "; dd bs=$N count=1000 oflag=dsync if=/dev/zero of=/fs/sdb6/tmp/TEST |& grep copied; done
4k: 4096000 bytes (4.1 MB) copied, 44.9177 s, 91.2 kB/s
16k: 16384000 bytes (16 MB) copied, 38.131 s, 430 kB/s
64k: 65536000 bytes (66 MB) copied, 35.8263 s, 1.8 MB/s
128k: 131072000 bytes (131 MB) copied, 35.8188 s, 3.7 MB/s
256k: 262144000 bytes (262 MB) copied, 36.6838 s, 7.1 MB/s
512k: 524288000 bytes (524 MB) copied, 37.0612 s, 14.1 MB/s
1024k: 1048576000 bytes (1.0 GB) copied, 42.0844 s, 24.9 MB/s
>> * Relax the requirement for synchronous writes on *both* the
>> primary and secondary DRBD servers, if feeling lucky.
> I have the following entries for DRBD which were suggested by
> linbit (which previously lifted performance from abysmal to
> more than sufficient around 2+ years ago). [ ... ]
That's an inappropriate use of "performance" here:
> disk-barrier no;
> disk-flushes no;
> md-flushes no;
That "feeling lucky" list seems to me to have made performance
lower (in the sense that the performance of writing to
'/dev/null' is zero, even if the speed is really good :->).
With those settings the data sync policy is "disk-drain", which
also involves some waiting, but somewhat dangerous, except "In
case your backing storage device has battery-backed write cache"
(and "device" here means system and host adapter and disk); it
is not clear to me for metadata what "md-flushes no" gives.
BTW if you have battery-backed everything on the secondary side
you could use protocol "B".
However given those it looks likely that the bottleneck is also
on the primary DRBD side.
> Do you have any other suggestions or ideas that might assist?
* Smaller RAID5 stripes, as in 4+1 or 2+1, are cheaper in space
than RAID10 and enormously raise the chances that a full
stripe-write can happen (it still has the write-hole problem
of parity RAID).
* Make sure the DRBD journal is also on a separate device that
allows fast small sync writes.
Also, I have appended a sample DRBD configuration I have used:
----------------------------------------------------------------
resource r0
{
device /dev/drbd_r0 minor 0;
# A: "local disk and local TCP send buffer"
# B: "local disk and remote buffer cache"
# C: "both local and remote disk"
protocol C;
net
{
# As mentioned on IRC by a DRBD guy, this is not really a
# secret, but more a "unique id" that ensures that replicas
# of different resources don't get accidentally connected.
# Still to be ABR-ized.
shared-secret "xxxxxxxxxxxx";
cram-hmac-alg sha1;
ping-timeout 50;
after-sb-0pri discard-zero-changes;
after-sb-1pri discard-secondary;
after-sb-2pri disconnect;
# http://article.gmane.org/gmane.linux.network.drbd/18348
# http://www.drbd.org/users-guide-8.3/s-throughput-tuning.html
# https://alteeve.ca/w/AN!Cluster_Tutorial_2_-_Performance_Tuning
# http://fghaas.wordpress.com/2007/06/22/performance-tuning-drbd-setups/
sndbuf-size 0;
rcvbuf-size 0;
max-buffers 16384;
unplug-watermark 16384;
max-epoch-size 16384;
}
syncer
{
csums-alg sha1;
# At 45MB/s takes 6 hour per 1TB.
rate 95M;
use-rle;
}
startup
{
wfc-timeout 15;
degr-wfc-timeout 15;
outdated-wfc-timeout 15;
# Cannot be an address, must be output of 'hostname'.
become-primary-on host-1;
}
on host-1
{
address 192.168.1.11:7788;
disk /dev/md2;
flexible-meta-disk /dev/local0/r0_md;
}
on host-2
{
address 192.168.1.12:7788;
disk /dev/md2;
flexible-meta-disk /dev/local0/r0_md;
}
}
^ permalink raw reply [flat|nested] 18+ messages in thread
* Re: RAID5 Performance
2016-07-28 14:10 ` Adam Goryachev
@ 2016-07-28 17:45 ` Peter Grandi
0 siblings, 0 replies; 18+ messages in thread
From: Peter Grandi @ 2016-07-28 17:45 UTC (permalink / raw)
To: Linux RAID
[ ... ]
> My concern is that even if I solve *this* bottleneck (ie, the
> 530 model SSD being too busy), that there will be another
> bottleneck afterwards
There is always another bottleneck ;-).
[ ... ]
> I'm not sure, but I think I've had one of the 480GB drives
> fail, and 3 of the smaller 60GB and 80GB drives fail. So far,
> only the 480G failure was "catastrophic", the others were
> still operating . All were replaced by Intel.
[ ... ]
When flash SSD drives run over their "expected" write amount
they behave differently:
http://www.sabi.co.uk/blog/15-one.html#150406b
Intel flash SSDs apparently do the following:
* They switch immediately to read-only.
* On the next power up they refuse even to *read*.
BTW, as to the Samsung SM863 there is a relatively recent "group
test" here:
http://www.storagereview.com/samsung_sm863_ssd_review
The average and max latency graphs are interesting, especially
those under "Preconditioning Curve".
^ permalink raw reply [flat|nested] 18+ messages in thread
* Re: RAID5 Performance
2016-07-28 17:20 ` Peter Grandi
@ 2016-07-28 18:45 ` Doug Dumitru
2016-08-02 7:09 ` Adam Goryachev
1 sibling, 0 replies; 18+ messages in thread
From: Doug Dumitru @ 2016-07-28 18:45 UTC (permalink / raw)
To: Peter Grandi; +Cc: Linux RAID
... boy this thread is getting long.
A couple of points.
* I am one of a reasonably small group of people that have actually
written an FTL and have it in production use. FTLs in SSDs are some
of the most closely guarded "implementations" I have ever seen. I am
not sure if my FTL matches others, as I have not seen the others, but
the patent office thinks my version is unique enough (not that it
really matters).
* Many SSDs, even consumer models, and even models without battery
backup, usually enforce correct serialization. If you have a single
drive in a laptop, this is what is important. If you have an array in
a server, and if the power to the SSDs are protected, then this also
protects your data. You need to separate the failures you are trying
to protect. SuperCaps on an SSD that is behind redundant power
supplies on redundant UPSs is perhaps not the best place to spend your
money. Likewise, if you have an HA link, the write is not ACKed until
the other node gets the data at least into the memory buffer. Are you
trying to engineer against multiple failures at multiple sites. You
need to decide on the level of redundancy of redundancy of redundancy.
* Assumptions that FTLs wear sync writes at the ratio of the write
block to the erase block sizes are usually wrong. Older "dumb" flash
like CF and SD cards sometimes work like this, but even there they
have gotten better. An easier assumption is that "normal" SSDs will
have write amplification at the inverse of free space percentage. So
your consumer drive with 8% free has 1/.08=12.5:1 write amplification.
This is why data center drives have more free space. "Better" FTLs,
when working with 100% random workloads can lower this to just over
50% of this value. My FTL sees 5.45:1 write amp on a 100% random
workload steady state at 10% free.
* Real workloads are sometimes that same as random workloads and
sometimes very different. Some FTLs can exploit the patterns in a
real file system workload and some cannot. For example, my FTL sees
1.3:1 write amp with the JEDEC 128GB client trace at 10% free versus
the typical 9:1 for most consumer SSDs on the same trace with about
the same free space.
* Additional games are possible if you start to reach "into" the
blocks. Compression that only saves you 10% might not seem like much,
but if it moves the free space from 8% to 18%, it matters a lot. The
above examples were without compression. Compression can also make
file system write overhead "go away". It is not uncommon for a
journal and directory entry write to compress 80+% even though the
data is binary. This makes old hard drive optimizations like
"-noatime" unnecessary.
* You can do a lot more with an FTL if you move it "in front" of raid.
This basically eliminates the raid read/modify/write operation and
overhead entirely. It does introduce a new "write hole" aspect to the
array, but this can be plugged with nvRAM hardware.
Happy Hunting.
Doug Dumitru
^ permalink raw reply [flat|nested] 18+ messages in thread
* Re: RAID5 Performance
2016-07-28 17:20 ` Peter Grandi
2016-07-28 18:45 ` Doug Dumitru
@ 2016-08-02 7:09 ` Adam Goryachev
2016-08-03 21:23 ` Peter Grandi
1 sibling, 1 reply; 18+ messages in thread
From: Adam Goryachev @ 2016-08-02 7:09 UTC (permalink / raw)
To: Peter Grandi, Linux RAID
On 29/07/16 03:20, Peter Grandi wrote:
> [ ... ]
>
>>> * Replace the flash SSDs with those that are known to deliver
>>> high (at least > 10,000 single threaded) small synchronous
>>> write IOPS.
>> Is there a "known" SSD that you would suggest? My problem is
>> that Intel spec sheets seem to suggest that there is little
>> performance difference across the range of SSD's, so it's
>> really not clear which SSD model I should buy.
> The links I wrote earlier have lists:
Thanks for reminding me of that. I see that the list reflects my
experience (if we assume the 530 model is equivalent to the 535 model on
the list, and my 520 480GB is equivalent to the 520 on the list).
However, I can't get the budget for those really awesome drives up the
top of the list, that would require around $20k... or more.
For now, I've got 16 x 545s TB drives, and have replaced the first half
(ie, all drives in one server). Now I can see that the drives themselves
don't seem to be the bottleneck (the drives don't run at 100% util,
while the DRBD device does run at 100%).
I've written a small script to keep track of the number of seconds each
drive util value fits into each bracket (increments of 10%). Let me know
if you would like a copy (it's just a perl script which reads iostat
output, I'm sure it could be written much nicer).
So far, this is what I get on the secondary (with the new 8 x 845s 1TB
drives):
Drive 10 20 30 40 50 60 70 80
90 100
md1 19265 0 0 0 0 0 0 0
0 0
sda 17029 1579 404 137 49 45 13 4 4 1
sdb 16983 1453 477 179 77 63 22 6 3 2
sdc 16867 1579 492 182 76 40 17 8 1 3
sdd 17043 1499 445 154 59 40 14 6 3 2
sde 17064 1506 415 152 68 32 15 4 6 3
sdf 17138 1467 396 152 46 37 11 10 4 4
sdg 17118 1493 401 139 56 31 14 7 2 4
sdh 16997 1577 407 138 62 45 11 10 7 6
sdi 19236 12 4 4 2 0 2
3 0 1
Hopefully that will line up right !
So, out of the last 19265 seconds, each of the underlying drives was at
100% for only a couple of seconds (sdi is the OS drive). ie, the last
column shows the number of seconds the drive was at 90 to 100% util as
reported by iostat. The 10 column shows number of seconds between 0 and
10%, etc...
Looking at the primary, with all 520 series drives (except sda which is
a 545s series) and the DRBD drives I see this:
Drive 10 20 30 40 50 60 70 80
90 100
drbd0 19971 108 54 36 13 2 0 2 1
1
drbd1 19842 165 77 48 34 4 6 5 3
3
drbd10 19766 279 62 35 23 7 6 4
2 1
drbd11 20081 37 32 21 12 1 3 1
0 0
drbd12 20041 79 38 19 9 1 0 0
1 0
drbd13 16195 2335 758 338 220 131 77 39 32 58
drbd14 19765 230 90 49 30 9 4 6 2
1
drbd15 3473 6323 4136 2250 1390 913 614 443 418 220
drbd17 20175 9 1 0 3 0 0
0 0 0
drbd18 19878 170 65 29 23 10 4 0 6 1
drbd19 19255 368 138 86 87 100 39 35 44 35
drbd2 20188 0 0 0 0 0 0
0 0 0
drbd3 17457 1276 610 316 175 140 66 43 33 56
drbd4 20154 17 6 6 5 0 0
0 0 0
drbd5 19859 141 59 38 26 10 4 5 3 42
drbd6 20112 39 20 9 3 1 1 1
1 0
drbd7 20188 0 0 0 0 0 0
0 0 0
drbd8 19894 136 78 44 22 5 3 2 0
2
drbd9 19476 289 211 123 41 21 9 6
3 7
md1 20188 0 0 0 0 0 0
0 0 0
sda 16948 1696 439 286 213 206 316 81 3 0
sdb 16059 2177 844 402 290 352 50 13 1 0
sdc 16141 2132 852 388 312 328 30 5 0 0
sdd 15914 2182 956 395 300 362 72 6 1 0
sde 16099 2137 801 393 256 366 124 10 1 1
sdf 16000 2169 898 408 322 340 39 9 3
0
sdg 15929 2265 822 418 259 290 195 8 2 0
sdh 16107 2129 822 419 324 337 41 9 0 0
sdi 20155 3 3 7 14 6 0
0 0 0
So on the primary, I see even less of a bottleneck on the underlying
drives, which doesn't make a lot of sense to me. The secondary has less
read load (since all reads are handled by the primary), and should only
need to deal with raid rmw. Also, I'm not sure, but I think the
secondary does less meta data updates for DRBD. So I can only presume
the new drives are much better than the 530 series, but still not as
good as the 520 series. I'll need to run some tests before I put the
drives live next time.
However, the point of note is that DRBD devices are showing high util
levels much more frequently than the underlying devices, so I can only
assume that the current limitation is caused by DRBD rather than the
drives. Though probably solving the DRBD issue will then go back to the
drives being the limit, with not a lot of difference. See below for my
(your) ideas on improving both of those things.....
>>>> https://www.sebastien-han.fr/blog/2014/10/10/ceph-how-to-test-if-your-ssd-is-suitable-as-a-journal-device/
>>>> http://www.spinics.net/lists/ceph-users/msg25928.html
>>>> https://www.redhat.com/en/resources/ceph-pcie-ssd-performance-part-1
> As one of those pages says the Samsung SM863 looks attractive,
> but for historical reasons so far I have only seen Intel DCs in
> similar use. There discussions of other models in various posts
> related to Ceph journal SSD usage.
>
>> Obviously it's not something I can afford to buy one of each
>> and test them either.
> In addition to the lists above I have justed tested my three
> home flash SSDs:
>
> * Micron M4 256GB:
> # dd bs=4k count=100000 oflag=direct,dsync if=/dev/zero of=/var/tmp/TEST
> 100000+0 records in
> 100000+0 records out
> 409600000 bytes (410 MB) copied, 1200.3 s, 341 kB/s
> * Samsung 850 Pro 256GB:
> # dd bs=4k count=100000 oflag=direct,dsync if=/dev/zero of=/var/tmp/TEST
> 100000+0 records in
> 100000+0 records out
> 409600000 bytes (410 MB) copied, 1732.93 s, 236 kB/s
> * Hynix SK SH910 256GB:
> # dd bs=4k count=100000 oflag=direct,dsync if=/dev/zero of=/var/tmp/TEST
> 100000+0 records in
> 100000+0 records out
> 409600000 bytes (410 MB) copied, 644.742 s, 635 kB/s
>
> So I would not recommend any of them for "small sync writes"
> workloads :-), but they are quite good otherwise. I do notice
> they are slow on small sync writes when downloading mail, as
> each message is duly 'fsync'ed.
>
> BTW as bonus material, I have done on the SH910 an abbreviated
> test with block sizes between 4KiB and 1024KiB:
>
> # for N in 4k 16k 64k 128k 256k 512k 1024k; do echo -n "$N: "; dd bs=$N count=1000 oflag=dsync if=/dev/zero of=/var/tmp/TEST |& grep copied; done
> 4k: 4096000 bytes (4.1 MB) copied, 6.23481 s, 657 kB/s
> 16k: 16384000 bytes (16 MB) copied, 6.29379 s, 2.6 MB/s
> 64k: 65536000 bytes (66 MB) copied, 6.09223 s, 10.8 MB/s
> 128k: 131072000 bytes (131 MB) copied, 6.5487 s, 20.0 MB/s
> 256k: 262144000 bytes (262 MB) copied, 6.93361 s, 37.8 MB/s
> 512k: 524288000 bytes (524 MB) copied, 7.73957 s, 67.7 MB/s
> 1024k: 1048576000 bytes (1.0 GB) copied, 12.8671 s, 81.5 MB/s
>
> Note how the time to write 1000 blocks is essentially the same
> betweeen 4KiB and 128KiB, which is quite amusing. Probably the
> flash-page size is around 256KiB.
>
> For additional bonus value the same on a "fastish" consumer 2TB
> disk, a Seagate ST2000DM001:
>
> # for N in 4k 16k 64k 128k 256k 512k 1024k; do echo -n "$N: "; dd bs=$N count=1000 oflag=dsync if=/dev/zero of=/fs/sdb6/tmp/TEST |& grep copied; done
> 4k: 4096000 bytes (4.1 MB) copied, 44.9177 s, 91.2 kB/s
> 16k: 16384000 bytes (16 MB) copied, 38.131 s, 430 kB/s
> 64k: 65536000 bytes (66 MB) copied, 35.8263 s, 1.8 MB/s
> 128k: 131072000 bytes (131 MB) copied, 35.8188 s, 3.7 MB/s
> 256k: 262144000 bytes (262 MB) copied, 36.6838 s, 7.1 MB/s
> 512k: 524288000 bytes (524 MB) copied, 37.0612 s, 14.1 MB/s
> 1024k: 1048576000 bytes (1.0 GB) copied, 42.0844 s, 24.9 MB/s
>
Yep, definitely won't be going backwards to spinning disks :)
>>> * Relax the requirement for synchronous writes on *both* the
>>> primary and secondary DRBD servers, if feeling lucky.
>> I have the following entries for DRBD which were suggested by
>> linbit (which previously lifted performance from abysmal to
>> more than sufficient around 2+ years ago). [ ... ]
> That's an inappropriate use of "performance" here:
>
>> disk-barrier no;
>> disk-flushes no;
>> md-flushes no;
> That "feeling lucky" list seems to me to have made performance
> lower (in the sense that the performance of writing to
> '/dev/null' is zero, even if the speed is really good :->).
>
> With those settings the data sync policy is "disk-drain", which
> also involves some waiting, but somewhat dangerous, except "In
> case your backing storage device has battery-backed write cache"
> (and "device" here means system and host adapter and disk); it
> is not clear to me for metadata what "md-flushes no" gives.
>
> BTW if you have battery-backed everything on the secondary side
> you could use protocol "B".
From my understanding, the times these settings can cause a problem:
1) When both servers hard power off - possibly all the latest data is
not written to disk that the VM's expect. If this is the case, all the
VM's were also hard powered off, and so the VM has no idea about what it
expects to be written/not. The end user may need to redo some work/etc,
but that is acceptable. Worst case scenario, a DB file is corrupted and
needs to be restored from the previous night backup, and users must redo
all work, which is also "acceptable" (from a risk point of view).
2) One server hard power off, perhaps power supply failure/etc - When it
powers on again, it should re-sync with the DRBD primary, and
potentially we do a DRBD verify to confirm everything is good. As long
as there is no failure on the primary, then everything is good. Worst
case, catastrophic failure of the primary before the verify is complete,
or before the secondary comes on-line again, and basically we treat it
as above.
We can't deal with every possible scenario, as the cost is prohibitive,
we can only deal with the more common scenarios, and those that are
cheaper to deal with. eg, all equipment is protected by UPS, using
redundancy RAID instead of linear/striping, and using DRBD for
replication. Most likely failures are disk, power supply, or network
cables (ie, unplugged by accident/etc), and this setup protects well for
all three of those.
> However given those it looks likely that the bottleneck is also
> on the primary DRBD side.
>
>> Do you have any other suggestions or ideas that might assist?
> * Smaller RAID5 stripes, as in 4+1 or 2+1, are cheaper in space
> than RAID10 and enormously raise the chances that a full
> stripe-write can happen (it still has the write-hole problem
> of parity RAID).
I was planning to upgrade to the 4.4.x kernel, which would kind of solve
this, since it will only read from 2 drives anyway, but it turns out
that is more difficult than I expected. (iscsitarget kernel module
doesn't compile cleanly with the new kernel, and it doesn't seem to be
well supported into such recent kernel versions. I'll probably wait
until debian testing becomes stable, or at least a lot closer, before
going down that path).
I could potentially move to 2 x RAID5 with 3+1 and then linear or stripe
those, which means I only lose one extra disk of capacity.... Will need
to think about that further...
> * Make sure the DRBD journal is also on a separate device that
> allows fast small sync writes.
I think this would be the next option to investigate. Currently the DRBD
journal is on the same devices.
Reading from:
http://www.drbd.org/en/doc/users-guide-84/ch-internals#s-internal-meta-data
>
> *Advantage. *For some write operations, using external meta data
> produces a somewhat improved latency behavior.
>
Do you have any more knowledge on the expected performance advantage?
ie, would half the writes move from the data drive to the meta data drive?
I'm thinking it might be plausible to purchase 2 x Intel P3700 400GB and
put one in each DRBD server for the meta data updates. Although if this
isn't going to make much difference (eg, only 20%) then it is less
likely to be worthwhile...
Can anyone suggest what kind of performance improvement might I see by
doing this?
The alternative (for double the cost + a bit more) would be to migrate
from RAID5 to RAID10, is that likely to produce a better/worse result?
2 x P3700 400GB is probably around $2500, while 12 x 545s 1000GB is
around $4800, but would need to add another SATA controller card, which
probably means changing motherboard/CPU/etc as well, so that becomes a
lot more....
> Also, I have appended a sample DRBD configuration I have used:
>
> ----------------------------------------------------------------
>
> # http://article.gmane.org/gmane.linux.network.drbd/18348
> # http://www.drbd.org/users-guide-8.3/s-throughput-tuning.html
> # https://alteeve.ca/w/AN!Cluster_Tutorial_2_-_Performance_Tuning
> # http://fghaas.wordpress.com/2007/06/22/performance-tuning-drbd-setups/
> sndbuf-size 0;
> rcvbuf-size 0;
> max-buffers 16384;
> unplug-watermark 16384;
> max-epoch-size 16384;
I have similar values, but will need to investigate the above options
further. rcvbuf-size doesn't seem to be well documented, at least in the
DRBD 8.4 manual, but will research these some more. Then will also need
to check how to modify the values without causing a system meltdown....
Thanks again for your advice/information, it is very helpful.
Regards,
Adam
--
Adam Goryachev Website Managers www.websitemanagers.com.au
^ permalink raw reply [flat|nested] 18+ messages in thread
* Re: RAID5 Performance
2016-08-02 7:09 ` Adam Goryachev
@ 2016-08-03 21:23 ` Peter Grandi
0 siblings, 0 replies; 18+ messages in thread
From: Peter Grandi @ 2016-08-03 21:23 UTC (permalink / raw)
To: Linux RAID
[ ... ]
> However, I can't get the budget for those really awesome
> drives up the top of the list, that would require around
> $20k... or more.
> For now, I've got 16 x 545s TB drives, and have replaced the
> first half (ie, all drives in one server). Now I can see that
> the drives themselves don't seem to be the bottleneck (the
> drives don't run at 100% util, while the DRBD device does run
> at 100%).
The "%util" number is not that easy to interpret, especially for
flash SSD and in some situations which probably include this
one:
https://brooker.co.za/blog/2014/07/04/iostat-pct.html
> Hopefully that will line up right !
It is hard to read, and I don't understand what the numbers are,
but it does not matter a lot.
> So I can only presume the new drives are much better than the
> 530 series, but still not as good as the 520 series.
The 540s have an SLC write buffer as I mentioned previously,
which should help.
> However, the point of note is that DRBD devices are showing
> high util levels much more frequently than the underlying
> devices, so I can only assume that the current limitation is
> caused by DRBD rather than the drives.
I like guessing, but this assumptions seems to me a bit
excessive.
> From my understanding, the times these settings can cause a
> problem: [ ... ]
If you don't have reliable sync barriers at all levels (not just
DRBD) *any* crash (e.g. bug crash, mistake-crash, memory-full
crash, not just power crash) is going to cause massive trouble,
especially in a mostly-write workload where what is being
written is cache spill. Some interesting pages:
http://blog.2ndquadrant.com/intel_ssd_now_off_the_sherr_sh/
http://wiki.postgresql.org/wiki/Reliable_Writes
http://archive.is/WTeAE
https://news.ycombinator.com/item?id=6973179
http://lkcl.net/reports/ssd_analysis.html
>>> Do you have any other suggestions or ideas that might
>>> assist?
Another one that would likely give a bit of relief as you can't
budget for write-optimized "enterprise" flash SSDs is a SATA/SAS
host adapter with a very large battery-backed RAM buffer. As the
tests that I previously mentioned show, longer writes result in
much improved write rates on "consumer" flash SSD devices, and
hopefully the large buffer results in:
#1 When the large write buffer flushes, *hopefully* much longer
writes to the flash SSD will happen on average.
#2 Thanks to the battery backing, writes are reporte completed
to the OS when they reach the host adapter's buffer, rather
than the flash SSD layer.
If #1 does not happen #2 won't help much if writes are at the
flash SSD saturation level, only if they are bursty and on
average below it.
>> * Smaller RAID5 stripes, as in 4+1 or 2+1, are cheaper in
>> space than RAID10 and enormously raise the chances that a
>> full stripe-write can happen (it still has the write-hole
>> problem of parity RAID).
> I was planning to upgrade to the 4.4.x kernel, which would
> kind of solve this, [ ... ]
The write hole workaround in MD RAID relies on a mostly-write
journal device like for DRBD.
>> * Make sure the DRBD journal is also on a separate device
>> that allows fast small sync writes.
> I think this would be the next option to investigate.
> Currently the DRBD journal is on the same devices.
That means that every sync'ed write becomes two writes to the
same device.
> 2 x P3700 400GB is probably around $2500,
The Samsung SM863 I ahve already mentioned are write-optimized
too and much cheaper, at around $300-350 for the 480GB model.
> while 12 x 545s 1000GB is around $4800, [ ... ]
Many people try to use "consumer" drives to build manager wowing
systems that have huge capacity and low cost, but vendors are not
stupid, and make sure that premium priced "enteprise" drives have
some critical advantage for at least some important workloads
(usually write heavy, guessing that "enterprise" workloads that
can command premium prices are transactional); sometimes like for
SSDs the advantages are based on real stuff, capacitors and
overprovisioning, which do cost money, sometimes artificial like
disabling SCT/ERC control.
^ permalink raw reply [flat|nested] 18+ messages in thread
end of thread, other threads:[~2016-08-03 21:23 UTC | newest]
Thread overview: 18+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2016-07-27 2:24 RAID5 Performance Adam Goryachev
2016-07-27 3:15 ` Brad Campbell
2016-07-27 5:36 ` Doug Dumitru
2016-07-27 23:26 ` Adam Goryachev
[not found] ` <7af0cc98-e395-9446-05eb-a6c0ca20f187@websitemanagers.com.au>
2016-07-28 0:11 ` Doug Dumitru
2016-07-28 13:08 ` Anthony Youngman
2016-07-28 14:10 ` Adam Goryachev
2016-07-28 17:45 ` Peter Grandi
2016-07-27 14:26 ` Peter Grandi
2016-07-27 17:38 ` Doug Dumitru
2016-07-28 12:19 ` Peter Grandi
2016-07-28 13:28 ` Peter Grandi
2016-07-28 13:57 ` Adam Goryachev
2016-07-28 17:20 ` Peter Grandi
2016-07-28 18:45 ` Doug Dumitru
2016-08-02 7:09 ` Adam Goryachev
2016-08-03 21:23 ` Peter Grandi
2016-07-28 13:50 ` Adam Goryachev
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox