RAID5 Performance

RAID5 Performance

[Adam Goryachev]

Hi all,

I know, age old question, but I have the chance to change things up a 
bit, and I wanted to collect some thoughts/ideas.

Currently I am using 8 x 480GB Intel SSD in a RAID5, then LVM on top, 
DRBD on top, and finally iSCSI on top (and then used as VM raw disks for 
mostly windows VM's).

My current array looks like this:

/dev/md1:
         Version : 1.2
   Creation Time : Wed Aug 22 00:47:03 2012
      Raid Level : raid5
      Array Size : 3281935552 (3129.90 GiB 3360.70 GB)
   Used Dev Size : 468847936 (447.13 GiB 480.10 GB)
    Raid Devices : 8
   Total Devices : 8
     Persistence : Superblock is persistent

     Update Time : Wed Jul 27 11:32:00 2016
           State : active
  Active Devices : 8
Working Devices : 8
  Failed Devices : 0
   Spare Devices : 0

          Layout : left-symmetric
      Chunk Size : 64K

            Name : san1:1  (local to host san1)
            UUID : 707957c0:b7195438:06da5bc4:485d301c
          Events : 2185221

     Number   Major   Minor   RaidDevice State
        7       8       65        0      active sync   /dev/sde1
       13       8        1        1      active sync   /dev/sda1
        8       8       81        2      active sync   /dev/sdf1
        5       8      113        3      active sync   /dev/sdh1
        9       8       97        4      active sync   /dev/sdg1
       12       8       17        5      active sync   /dev/sdb1
       10       8       49        6      active sync   /dev/sdd1
       11       8       33        7      active sync   /dev/sdc1

I've configured the following non-standard options:

echo 4096 > /sys/block/md1/md/stripe_cache_size

The following apply to all SSD's installed:
echo noop > $disk/queue/scheduler
echo 128 > ${disk}/queue/nr_requests

What I can measure (at peak periods) with iostat:
Device:         rrqm/s   wrqm/s     r/s     w/s    rMB/s wMB/s avgrq-sz 
avgqu-sz   await r_await w_await  svctm  %util
sdi               0.00     0.00    0.00    0.00     0.00 0.00     
0.00     0.00    0.00    0.00    0.00   0.00   0.00
sda              78.00    59.00   79.00   86.00     0.74 0.52    
15.55     0.02    0.15    0.20    0.09   0.15   2.40
sdg              35.00    48.00   68.00   79.00     0.52 0.44    
13.39     0.02    0.14    0.24    0.05   0.11   1.60
sdf              46.00    65.00   86.00   98.00     0.76 0.58    
14.96     0.03    0.17    0.09    0.24   0.09   1.60
sdh              97.00    45.00   70.00  141.00     0.66 0.68    
12.96     0.08    0.36    0.29    0.40   0.34   7.20
sde             101.00    75.00   87.00   94.00     0.79 0.61    
15.76     0.08    0.42    0.32    0.51   0.29   5.20
sdb              85.00    54.00   94.00  102.00     0.84 0.56    
14.62     0.01    0.04    0.09    0.00   0.04   0.80
sdc              85.00    74.00   98.00  106.00     0.79 0.66    
14.53     0.01    0.06    0.04    0.08   0.04   0.80
sdd             230.00   199.00  266.00  353.00     2.19 2.11    
14.24     0.18    0.28    0.23    0.32   0.16   9.60
drbd0             0.00     0.00    0.00    2.00     0.00 0.00     
4.50     0.08   38.00    0.00   38.00  20.00   4.00
drbd12            0.00     0.00    1.00    1.00     0.00 0.00     
7.50     0.03   14.00    4.00   24.00  14.00   2.80
drbd1             0.00     0.00    0.00    2.00     0.00 0.03    
32.00     0.09   44.00    0.00   44.00  22.00   4.40
drbd9             0.00     0.00    2.00    0.00     0.01 0.00     
8.00     0.00    0.00    0.00    0.00   0.00   0.00
drbd2             0.00     0.00    0.00    0.00     0.00 0.00     
0.00     0.00    0.00    0.00    0.00   0.00   0.00
drbd11            0.00     0.00    0.00    0.00     0.00 0.00     
0.00     0.00    0.00    0.00    0.00   0.00   0.00
drbd3             0.00     0.00    4.00  197.00     0.02 1.01    
10.47     7.92   41.03    0.00   41.87   4.98 100.00
drbd4             0.00     0.00    0.00    0.00     0.00 0.00     
0.00     0.00    0.00    0.00    0.00   0.00   0.00
drbd17            0.00     0.00    1.00    0.00     0.00 0.00     
8.00     0.00    0.00    0.00    0.00   0.00   0.00
drbd5             0.00     0.00    0.00    7.00     0.00 0.03     
8.00     0.22   30.29    0.00   30.29  28.57  20.00
drbd19            0.00     0.00    0.00    0.00     0.00 0.00     
0.00     0.00    0.00    0.00    0.00   0.00   0.00
drbd6             0.00     0.00    2.00    0.00     0.01 0.00     
8.00     0.00    0.00    0.00    0.00   0.00   0.00
drbd7             0.00     0.00    0.00    0.00     0.00 0.00     
0.00     0.00    0.00    0.00    0.00   0.00   0.00
drbd8             0.00     0.00    0.00    0.00     0.00 0.00     
0.00     0.00    0.00    0.00    0.00   0.00   0.00
drbd13            0.00     0.00   90.00   44.00     1.74 0.38    
32.35     1.72   13.46    0.40   40.18   4.27  57.20
drbd15            0.00     0.00    2.00   33.00     0.02 0.29    
17.86     1.40   40.91    0.00   43.39  28.34  99.20
drbd18            0.00     0.00    1.00    3.00     0.00 0.03    
16.00     0.08   21.00    0.00   28.00  21.00   8.40
drbd14            0.00     0.00    0.00    0.00     0.00 0.00     
0.00     0.00    0.00    0.00    0.00   0.00   0.00
drbd10            0.00     0.00    0.00    0.00     0.00 0.00     
0.00     0.00    0.00    0.00    0.00   0.00   0.00

As you can see, the DRBD devices are busy, and slowing down the VM's, 
looking at the drives on the second server we can see why:
Device:         rrqm/s   wrqm/s     r/s     w/s    rMB/s wMB/s avgrq-sz 
avgqu-sz   await r_await w_await  svctm  %util
sdf              67.00    76.00   64.00  113.00     0.52 0.62    
13.17     0.26    1.47    0.06    2.27   1.45  25.60
sdg              39.00    61.00   50.00  114.00     0.35 0.56    
11.38     0.45    2.76    0.08    3.93   2.71  44.40
sdd              49.00    67.00   50.00  109.00     0.39 0.57    
12.40     0.75    4.73    0.00    6.90   4.70  74.80
sdh              55.00    54.00   52.00  104.00     0.42 0.51    
12.12     0.81    5.21    0.23    7.69   5.13  80.00
sde              67.00    67.00   75.00  129.00     0.56 0.65    
12.13     0.94    4.59    0.69    6.85   4.24  86.40
sda              64.00    76.00   58.00  109.00     0.48 0.61    
13.29     0.84    5.03    0.21    7.60   4.89  81.60
sdb              35.00    72.00   57.00  104.00     0.36 0.57    
11.84     0.69    4.27    0.14    6.54   4.22  68.00
sdc             118.00   144.00  228.00  269.00     1.39 1.50    
11.92     1.21    2.43    1.88    2.90   1.50  74.40
md1               0.00     0.00    0.00  260.00     0.00 1.70    
13.38     0.00    0.00    0.00    0.00   0.00   0.00

I've confirmed that the problem is that we have mixed two models of SSD 
(520 series and 530 series), and that the 530 series drives perform 
significantly worse (under load) in comparison. Above, the two 520 
series are sdf and sdg while the other drives are 530 series. So, we 
will be replacing all of the drives across both systems with 545s series 
1000GB SSD's (which I've confirmed will operate same or better than the 
520 series, sdc on the first machine above is one of these already).

Over the years, I've learned a lot about RAID and optimisation, 
originally I configured things to optimise for super fast streaming 
reads and streaming writes, but in practice, the actual work-load is 
small random read/write, with the writes causing the biggest load.

Looking at this:
http://serverfault.com/questions/384273/optimizing-raid-5-for-backuppc-use-small-random-reads
>
>  *
>
>     Enhance the queue depth. Standard kernel queue depth is OK for old
>     single drives with small caches, but not for modern drives or RAID
>     arrays:
>
>     echo 512 > /sys/block/sda/queue/nr_requests
>
So my question is should I increase the configured nr_requests above the 
current 128?

If the chunk size is 64k, and there are 8 drives in total, then the 
stripe size is currently 64k*7 = 448k, is this too big? My reading of 
the mdadm man page suggests the minimum chunk size is 4k ("In any case 
it must be a multiple of 4KB"). If I set the chunk size to 4k, then the 
stripe size becomes 28k, which means for a random 4k write, we only need 
to write 28k instead of 448k ?
The drives report a sector size of 512k, which I guess means the 
smallest meaningful write that the drive can do is 512k, so should I 
increase the chunk size to 512k to match? Or does that make it even worse?
Finally, the drive reports Host_Writes_32MiB in SMART, does that mean 
that the drive needs to replace a entire 32MB chunk in order to 
overwrite a sector? I'm guessing a chunk size of 32M is just crazy though...

Is there a better way to actually measure the different sizes and 
quantity of read/writes being issued, so that I can make a more accurate 
decision on chunk size/stripe size/etc... iostat seems to show an 
average numbers, but not the number of 1k read/write, 4k read/write, 16k 
read/write etc...

My suspicion is that the actual load is made up of rather small random 
read/write, because that is the scenario that produced the worst 
performance results when I was initially setting this up, and seems to 
be what we are getting in practice.

The last option is, what if I moved to RAID10? Would that provide a 
significant performance boost (completely removes the need to worry 
about chunk/stripe size because we always just write the exact data we 
want, no need to read/compute/write)?
OR, is that read/compute overhead negligible since I'm using SSD and 
read performance is so quick?

For completeness, PV information:
   PV Name               /dev/md1
   VG Name               vg0
   PV Size               3.06 TiB / not usable 2.94 MiB
   Allocatable           yes
   PE Size               4.00 MiB
   Total PE              801253
   Free PE               33281
   Allocated PE          767972
   PV UUID c0PIEb-tUka-zBk3-lcGM-H89s-ayde-hcMUBZ

Any advice or assistance would be greatly appreciated.

Regards,
Adam
-- 
Adam Goryachev Website Managers www.websitemanagers.com.au

Re: RAID5 Performance

[Brad Campbell]

On 27/07/16 10:24, Adam Goryachev wrote:
> Hi all,
>
> I know, age old question, but I have the chance to change things up a
> bit, and I wanted to collect some thoughts/ideas.
>
> Currently I am using 8 x 480GB Intel SSD in a RAID5, then LVM on top,
> DRBD on top, and finally iSCSI on top (and then used as VM raw disks for
> mostly windows VM's).


Wow. More layers than a wedding cake.

>
> My suspicion is that the actual load is made up of rather small random
> read/write, because that is the scenario that produced the worst
> performance results when I was initially setting this up, and seems to
> be what we are getting in practice.
>
> The last option is, what if I moved to RAID10? Would that provide a
> significant performance boost (completely removes the need to worry
> about chunk/stripe size because we always just write the exact data we
> want, no need to read/compute/write)?
> OR, is that read/compute overhead negligible since I'm using SSD and
> read performance is so quick?

I'll only comment from personal experience with anecdotal evidence.

I have a RAID10 comprised of 6 256GB SSD (3 Intel & 3 Samsung) used as 
the backing for multiple VMs (raw files on Ext4).

Initially I played with a number of RAID types when setting up the array 
(back in 2012) and found RAID10 offered the best compromise for my use 
case. This was based on CPU usage (raid 5 & 6 parity calculations on 
*every write*), the need for RMW cycles for small writes and trying to 
balance block sizes. None of these things are an issue with RAID10 and 
in general I found a measurable reduction in overhead and consequential 
performance boost in the VM's.

This is a single machine with a relatively underpowered AMD FX-8350 CPU. 
I found that with multiple VM's hitting the disks I was losing enough 
CPU time to RAID overhead that it made things noticeably less responsive.

My only issue is half the disks return a deterministic value after 
discard and the other half don't, so any raid check operations return a 
gazillion mismatches. Not an operational issue, but one worth mentioning 
if you were to use different model drives.

Regards,
Brad

Re: RAID5 Performance

[Doug Dumitru]

On Tue, Jul 26, 2016 at 7:24 PM, Adam Goryachev
<mailinglists@websitemanagers.com.au> wrote:
> Hi all,
>
> I know, age old question, but I have the chance to change things up a bit,
> and I wanted to collect some thoughts/ideas.
>
> Currently I am using 8 x 480GB Intel SSD in a RAID5, then LVM on top, DRBD
> on top, and finally iSCSI on top (and then used as VM raw disks for mostly
> windows VM's).
>
> My current array looks like this:
>
> /dev/md1:
>         Version : 1.2
>   Creation Time : Wed Aug 22 00:47:03 2012
>      Raid Level : raid5
>      Array Size : 3281935552 (3129.90 GiB 3360.70 GB)
>   Used Dev Size : 468847936 (447.13 GiB 480.10 GB)
>    Raid Devices : 8
>   Total Devices : 8
>     Persistence : Superblock is persistent
>
>     Update Time : Wed Jul 27 11:32:00 2016
>           State : active
>  Active Devices : 8
> Working Devices : 8
>  Failed Devices : 0
>   Spare Devices : 0
>
>          Layout : left-symmetric
>      Chunk Size : 64K
>
>            Name : san1:1  (local to host san1)
>            UUID : 707957c0:b7195438:06da5bc4:485d301c
>          Events : 2185221
>
>     Number   Major   Minor   RaidDevice State
>        7       8       65        0      active sync   /dev/sde1
>       13       8        1        1      active sync   /dev/sda1
>        8       8       81        2      active sync   /dev/sdf1
>        5       8      113        3      active sync   /dev/sdh1
>        9       8       97        4      active sync   /dev/sdg1
>       12       8       17        5      active sync   /dev/sdb1
>       10       8       49        6      active sync   /dev/sdd1
>       11       8       33        7      active sync   /dev/sdc1
>
> I've configured the following non-standard options:
>
> echo 4096 > /sys/block/md1/md/stripe_cache_size
>
> The following apply to all SSD's installed:
> echo noop > $disk/queue/scheduler
> echo 128 > ${disk}/queue/nr_requests
>
> What I can measure (at peak periods) with iostat:
> Device:         rrqm/s   wrqm/s     r/s     w/s    rMB/s wMB/s avgrq-sz
> avgqu-sz   await r_await w_await  svctm  %util
> sdi               0.00     0.00    0.00    0.00     0.00 0.00     0.00
> 0.00    0.00    0.00    0.00   0.00   0.00
> sda              78.00    59.00   79.00   86.00     0.74 0.52    15.55
> 0.02    0.15    0.20    0.09   0.15   2.40
> sdg              35.00    48.00   68.00   79.00     0.52 0.44    13.39
> 0.02    0.14    0.24    0.05   0.11   1.60
> sdf              46.00    65.00   86.00   98.00     0.76 0.58    14.96
> 0.03    0.17    0.09    0.24   0.09   1.60
> sdh              97.00    45.00   70.00  141.00     0.66 0.68    12.96
> 0.08    0.36    0.29    0.40   0.34   7.20
> sde             101.00    75.00   87.00   94.00     0.79 0.61    15.76
> 0.08    0.42    0.32    0.51   0.29   5.20
> sdb              85.00    54.00   94.00  102.00     0.84 0.56    14.62
> 0.01    0.04    0.09    0.00   0.04   0.80
> sdc              85.00    74.00   98.00  106.00     0.79 0.66    14.53
> 0.01    0.06    0.04    0.08   0.04   0.80
> sdd             230.00   199.00  266.00  353.00     2.19 2.11    14.24
> 0.18    0.28    0.23    0.32   0.16   9.60
> drbd0             0.00     0.00    0.00    2.00     0.00 0.00     4.50
> 0.08   38.00    0.00   38.00  20.00   4.00
> drbd12            0.00     0.00    1.00    1.00     0.00 0.00     7.50
> 0.03   14.00    4.00   24.00  14.00   2.80
> drbd1             0.00     0.00    0.00    2.00     0.00 0.03    32.00
> 0.09   44.00    0.00   44.00  22.00   4.40
> drbd9             0.00     0.00    2.00    0.00     0.01 0.00     8.00
> 0.00    0.00    0.00    0.00   0.00   0.00
> drbd2             0.00     0.00    0.00    0.00     0.00 0.00     0.00
> 0.00    0.00    0.00    0.00   0.00   0.00
> drbd11            0.00     0.00    0.00    0.00     0.00 0.00     0.00
> 0.00    0.00    0.00    0.00   0.00   0.00
> drbd3             0.00     0.00    4.00  197.00     0.02 1.01    10.47
> 7.92   41.03    0.00   41.87   4.98 100.00
> drbd4             0.00     0.00    0.00    0.00     0.00 0.00     0.00
> 0.00    0.00    0.00    0.00   0.00   0.00
> drbd17            0.00     0.00    1.00    0.00     0.00 0.00     8.00
> 0.00    0.00    0.00    0.00   0.00   0.00
> drbd5             0.00     0.00    0.00    7.00     0.00 0.03     8.00
> 0.22   30.29    0.00   30.29  28.57  20.00
> drbd19            0.00     0.00    0.00    0.00     0.00 0.00     0.00
> 0.00    0.00    0.00    0.00   0.00   0.00
> drbd6             0.00     0.00    2.00    0.00     0.01 0.00     8.00
> 0.00    0.00    0.00    0.00   0.00   0.00
> drbd7             0.00     0.00    0.00    0.00     0.00 0.00     0.00
> 0.00    0.00    0.00    0.00   0.00   0.00
> drbd8             0.00     0.00    0.00    0.00     0.00 0.00     0.00
> 0.00    0.00    0.00    0.00   0.00   0.00
> drbd13            0.00     0.00   90.00   44.00     1.74 0.38    32.35
> 1.72   13.46    0.40   40.18   4.27  57.20
> drbd15            0.00     0.00    2.00   33.00     0.02 0.29    17.86
> 1.40   40.91    0.00   43.39  28.34  99.20
> drbd18            0.00     0.00    1.00    3.00     0.00 0.03    16.00
> 0.08   21.00    0.00   28.00  21.00   8.40
> drbd14            0.00     0.00    0.00    0.00     0.00 0.00     0.00
> 0.00    0.00    0.00    0.00   0.00   0.00
> drbd10            0.00     0.00    0.00    0.00     0.00 0.00     0.00
> 0.00    0.00    0.00    0.00   0.00   0.00
>
> As you can see, the DRBD devices are busy, and slowing down the VM's,
> looking at the drives on the second server we can see why:
> Device:         rrqm/s   wrqm/s     r/s     w/s    rMB/s wMB/s avgrq-sz
> avgqu-sz   await r_await w_await  svctm  %util
> sdf              67.00    76.00   64.00  113.00     0.52 0.62    13.17
> 0.26    1.47    0.06    2.27   1.45  25.60
> sdg              39.00    61.00   50.00  114.00     0.35 0.56    11.38
> 0.45    2.76    0.08    3.93   2.71  44.40
> sdd              49.00    67.00   50.00  109.00     0.39 0.57    12.40
> 0.75    4.73    0.00    6.90   4.70  74.80
> sdh              55.00    54.00   52.00  104.00     0.42 0.51    12.12
> 0.81    5.21    0.23    7.69   5.13  80.00
> sde              67.00    67.00   75.00  129.00     0.56 0.65    12.13
> 0.94    4.59    0.69    6.85   4.24  86.40
> sda              64.00    76.00   58.00  109.00     0.48 0.61    13.29
> 0.84    5.03    0.21    7.60   4.89  81.60
> sdb              35.00    72.00   57.00  104.00     0.36 0.57    11.84
> 0.69    4.27    0.14    6.54   4.22  68.00
> sdc             118.00   144.00  228.00  269.00     1.39 1.50    11.92
> 1.21    2.43    1.88    2.90   1.50  74.40
> md1               0.00     0.00    0.00  260.00     0.00 1.70    13.38
> 0.00    0.00    0.00    0.00   0.00   0.00
>
> I've confirmed that the problem is that we have mixed two models of SSD (520
> series and 530 series), and that the 530 series drives perform significantly
> worse (under load) in comparison. Above, the two 520 series are sdf and sdg
> while the other drives are 530 series. So, we will be replacing all of the
> drives across both systems with 545s series 1000GB SSD's (which I've
> confirmed will operate same or better than the 520 series, sdc on the first
> machine above is one of these already).
>
> Over the years, I've learned a lot about RAID and optimisation, originally I
> configured things to optimise for super fast streaming reads and streaming
> writes, but in practice, the actual work-load is small random read/write,
> with the writes causing the biggest load.
>
> Looking at this:
> http://serverfault.com/questions/384273/optimizing-raid-5-for-backuppc-use-small-random-reads
>>
>>
>>  *
>>
>>     Enhance the queue depth. Standard kernel queue depth is OK for old
>>     single drives with small caches, but not for modern drives or RAID
>>     arrays:
>>
>>     echo 512 > /sys/block/sda/queue/nr_requests
>>
> So my question is should I increase the configured nr_requests above the
> current 128?

With your workload, it probably won't matter too much.  Really high
queue depths are great on paper, but hard to actually see.

>
> If the chunk size is 64k, and there are 8 drives in total, then the stripe
> size is currently 64k*7 = 448k, is this too big? My reading of the mdadm man
> page suggests the minimum chunk size is 4k ("In any case it must be a
> multiple of 4KB"). If I set the chunk size to 4k, then the stripe size
> becomes 28k, which means for a random 4k write, we only need to write 28k
> instead of 448k ?

This is not how a random write works.  If you are running raid-5
before the 4.4 kernel, you get the "old" read/modify/write algorithm.
If you write 4K, the system will read 4K from (n-2) drives, add in
your 4K to compute parity, and write 2 drives.  This is n-2 reads + 2
writes.  With the "new" logic in 4.4, you read the old contents of the
4K plus parity, and re-write the 4k plus parity, so there are 2 reads
and 2 writes.  With big arrays, the "new" logic can help quite a bit,
but the chatter rate is still high.  Note that the new logic is only
raid-5.  raid-6 cannot use the new logic and has to read the stripe
from every drive.

The stripe size impacts when the system does can avoid doing a
read/modify/write.  If you write a full stripe [ 64K * (n-1) ], and
the write is exactly on a stripe boundary, and you get lucky and the
background thread does not wake up at just the wrong time, you will do
the write with zero reads.  I personally run with very small chunks,
but I have code that always writes perfect stripe writes and stock
file systems don't act that way.

DRBD can saturate GigE without any problem with random 4K writes.  I
have a pair of systems here that pushes 110 MB/sec at 4K or 28,000
IOPS.  The target arrays needs to keep up, but that is another story.
My testing with DRBD is that it starts to peter out at 10Gig, so if
you want more bandwidth you need some other approach.  Some vendors
use SRP over Infiniband with software raid-1 as a mirror.  iSCSI with
iSER should give you similar results with RDMA capable ethernet.
Linbit (the people who write DRBD) have a non GPL extension to DRBD
that uses RDMA so you can get more bandwidth that way as well.

> The drives report a sector size of 512k, which I guess means the smallest
> meaningful write that the drive can do is 512k, so should I increase the
> chunk size to 512k to match? Or does that make it even worse?
> Finally, the drive reports Host_Writes_32MiB in SMART, does that mean that
> the drive needs to replace a entire 32MB chunk in order to overwrite a
> sector? I'm guessing a chunk size of 32M is just crazy though...

This is probably not true.  If the drive really had to update 512K at
a time, then 4K writes would be 128x wear amplification.  SSDs can be
bad, but usually not that bad.

>
> Is there a better way to actually measure the different sizes and quantity
> of read/writes being issued, so that I can make a more accurate decision on
> chunk size/stripe size/etc... iostat seems to show an average numbers, but
> not the number of 1k read/write, 4k read/write, 16k read/write etc...

The problem is that the FTL of the SSDs are a black box and as the
array gets bigger, the slowest drive dictates the array performance.
This is why the "big vendors" all map SSDs in the host and avoid or
minimize writing randomly.  I know of one vendor install that has 4000
VDI seats (using ESXI as compute hosts) from a single HA pair of 24
SSD shelves.  The connection to ESXI is FC and the hosts are HA with
an IB/SRP raid-1 link between them.  Unfortunately, you need 500K+
random write IOPS to pull this off, which I think is impossible with
stock parity raid, and very hard with raid-10.

>
> My suspicion is that the actual load is made up of rather small random
> read/write, because that is the scenario that produced the worst performance
> results when I was initially setting this up, and seems to be what we are
> getting in practice.
>
> The last option is, what if I moved to RAID10? Would that provide a
> significant performance boost (completely removes the need to worry about
> chunk/stripe size because we always just write the exact data we want, no
> need to read/compute/write)?

RAID-10 will be faster, but you pay for this with capacity.  It is
also a double-edged sword as SSDs themselves run faster if you leave
more free space on them, so RAID-10 absolutely might not be a lot
faster than RAID-5 with some space left over.  Also remember that free
space on the SSDs only counts if it is actually unallocated.  So you
need to trim the SSDs or start with a secure erased drive and then
never use the full capacity.  It is best to leave an empty partition
that is untouched.

> OR, is that read/compute overhead negligible since I'm using SSD and read
> performance is so quick?

The reads, especially with the pre 4.4 code or with raid-6 definitely
take their toll.  Most SSDs are also not quite symmetrical in terms of
performance.  If your SSD does 50K read IOPS and 50K write IOPS, it
will probably not do 25K reads and 25K writes concurrently, but
instead stop somewhere around 18K.  But your mileage may vary.  If you
have 8 drives that do 20 read/write symmetric, with new raid-5, each
4K write is 2 reads and 2 writes.  8 drives will give you 8*20K = 160K
reads and writes or 320K total OPS.  Each 4K write takes 4 OPS, so
your data rate ends up maxing out at 80K IOPS.  With the old raid-5
logic, you end up with 6 reads plus two writes per "OP", so you tend
to max out around 320K/(6+2) = 40K IOPS.  With more than 8 drives,
these computations tend to fall apart, so 24 SSD arrays are not 3x
faster than 8 SSD arrays, at least with stock code.

You also need to consider what raid does to the SSD FTL.  As you
chatter a drive, its wear goes up and its performance goes down.
Different SSD models can vary wildly, but again the rule of thumb is
keep as much free space as possible on the drives.  raid-5 or
mirroring is also 2:1 write amplification (ie, you are writing two
drives) and raid-6 is 3:1, on top of whatever the FTL write
amplification is at the time.

>
> For completeness, PV information:
>   PV Name               /dev/md1
>   VG Name               vg0
>   PV Size               3.06 TiB / not usable 2.94 MiB
>   Allocatable           yes
>   PE Size               4.00 MiB
>   Total PE              801253
>   Free PE               33281
>   Allocated PE          767972
>   PV UUID c0PIEb-tUka-zBk3-lcGM-H89s-ayde-hcMUBZ
>
> Any advice or assistance would be greatly appreciated.
>
> Regards,
> Adam
> --
> Adam Goryachev Website Managers www.websitemanagers.com.au
> --
> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html



-- 
Doug Dumitru
WileFire Storage.  http://www.wildfire-storage.com

Re: RAID5 Performance

[Adam Goryachev]

On 27/07/2016 15:36, Doug Dumitru wrote:
> On Tue, Jul 26, 2016 at 7:24 PM, Adam Goryachev
> <mailinglists@websitemanagers.com.au>  wrote:
>> Hi all,
>>
>> I know, age old question, but I have the chance to change things up a bit,
>> and I wanted to collect some thoughts/ideas.
>>
>> Currently I am using 8 x 480GB Intel SSD in a RAID5, then LVM on top, DRBD
>> on top, and finally iSCSI on top (and then used as VM raw disks for mostly
>> windows VM's).
>>
>> My current array looks like this:
>>
>> /dev/md1:
>>          Version : 1.2
>>    Creation Time : Wed Aug 22 00:47:03 2012
>>       Raid Level : raid5
>>       Array Size : 3281935552 (3129.90 GiB 3360.70 GB)
>>    Used Dev Size : 468847936 (447.13 GiB 480.10 GB)
>>     Raid Devices : 8
>>    Total Devices : 8
>>      Persistence : Superblock is persistent
>>
>>      Update Time : Wed Jul 27 11:32:00 2016
>>            State : active
>>   Active Devices : 8
>> Working Devices : 8
>>   Failed Devices : 0
>>    Spare Devices : 0
>>
>>           Layout : left-symmetric
>>       Chunk Size : 64K
>>
>>             Name : san1:1  (local to host san1)
>>             UUID : 707957c0:b7195438:06da5bc4:485d301c
>>           Events : 2185221
>>
>>      Number   Major   Minor   RaidDevice State
>>         7       8       65        0      active sync   /dev/sde1
>>        13       8        1        1      active sync   /dev/sda1
>>         8       8       81        2      active sync   /dev/sdf1
>>         5       8      113        3      active sync   /dev/sdh1
>>         9       8       97        4      active sync   /dev/sdg1
>>        12       8       17        5      active sync   /dev/sdb1
>>        10       8       49        6      active sync   /dev/sdd1
>>        11       8       33        7      active sync   /dev/sdc1
>>
>> I've configured the following non-standard options:
>>
>> echo 4096 > /sys/block/md1/md/stripe_cache_size
>>
>> The following apply to all SSD's installed:
>> echo noop > $disk/queue/scheduler
>> echo 128 > ${disk}/queue/nr_requests
>>
>> What I can measure (at peak periods) with iostat:
>> Device:         rrqm/s   wrqm/s     r/s     w/s    rMB/s wMB/s avgrq-sz
>> avgqu-sz   await r_await w_await  svctm  %util
>> sdi               0.00     0.00    0.00    0.00     0.00 0.00     0.00
>> 0.00    0.00    0.00    0.00   0.00   0.00
>> sda              78.00    59.00   79.00   86.00     0.74 0.52    15.55
>> 0.02    0.15    0.20    0.09   0.15   2.40
>> sdg              35.00    48.00   68.00   79.00     0.52 0.44    13.39
>> 0.02    0.14    0.24    0.05   0.11   1.60
>> sdf              46.00    65.00   86.00   98.00     0.76 0.58    14.96
>> 0.03    0.17    0.09    0.24   0.09   1.60
>> sdh              97.00    45.00   70.00  141.00     0.66 0.68    12.96
>> 0.08    0.36    0.29    0.40   0.34   7.20
>> sde             101.00    75.00   87.00   94.00     0.79 0.61    15.76
>> 0.08    0.42    0.32    0.51   0.29   5.20
>> sdb              85.00    54.00   94.00  102.00     0.84 0.56    14.62
>> 0.01    0.04    0.09    0.00   0.04   0.80
>> sdc              85.00    74.00   98.00  106.00     0.79 0.66    14.53
>> 0.01    0.06    0.04    0.08   0.04   0.80
>> sdd             230.00   199.00  266.00  353.00     2.19 2.11    14.24
>> 0.18    0.28    0.23    0.32   0.16   9.60
>> drbd0             0.00     0.00    0.00    2.00     0.00 0.00     4.50
>> 0.08   38.00    0.00   38.00  20.00   4.00
>> drbd12            0.00     0.00    1.00    1.00     0.00 0.00     7.50
>> 0.03   14.00    4.00   24.00  14.00   2.80
>> drbd1             0.00     0.00    0.00    2.00     0.00 0.03    32.00
>> 0.09   44.00    0.00   44.00  22.00   4.40
>> drbd9             0.00     0.00    2.00    0.00     0.01 0.00     8.00
>> 0.00    0.00    0.00    0.00   0.00   0.00
>> drbd2             0.00     0.00    0.00    0.00     0.00 0.00     0.00
>> 0.00    0.00    0.00    0.00   0.00   0.00
>> drbd11            0.00     0.00    0.00    0.00     0.00 0.00     0.00
>> 0.00    0.00    0.00    0.00   0.00   0.00
>> drbd3             0.00     0.00    4.00  197.00     0.02 1.01    10.47
>> 7.92   41.03    0.00   41.87   4.98 100.00
>> drbd4             0.00     0.00    0.00    0.00     0.00 0.00     0.00
>> 0.00    0.00    0.00    0.00   0.00   0.00
>> drbd17            0.00     0.00    1.00    0.00     0.00 0.00     8.00
>> 0.00    0.00    0.00    0.00   0.00   0.00
>> drbd5             0.00     0.00    0.00    7.00     0.00 0.03     8.00
>> 0.22   30.29    0.00   30.29  28.57  20.00
>> drbd19            0.00     0.00    0.00    0.00     0.00 0.00     0.00
>> 0.00    0.00    0.00    0.00   0.00   0.00
>> drbd6             0.00     0.00    2.00    0.00     0.01 0.00     8.00
>> 0.00    0.00    0.00    0.00   0.00   0.00
>> drbd7             0.00     0.00    0.00    0.00     0.00 0.00     0.00
>> 0.00    0.00    0.00    0.00   0.00   0.00
>> drbd8             0.00     0.00    0.00    0.00     0.00 0.00     0.00
>> 0.00    0.00    0.00    0.00   0.00   0.00
>> drbd13            0.00     0.00   90.00   44.00     1.74 0.38    32.35
>> 1.72   13.46    0.40   40.18   4.27  57.20
>> drbd15            0.00     0.00    2.00   33.00     0.02 0.29    17.86
>> 1.40   40.91    0.00   43.39  28.34  99.20
>> drbd18            0.00     0.00    1.00    3.00     0.00 0.03    16.00
>> 0.08   21.00    0.00   28.00  21.00   8.40
>> drbd14            0.00     0.00    0.00    0.00     0.00 0.00     0.00
>> 0.00    0.00    0.00    0.00   0.00   0.00
>> drbd10            0.00     0.00    0.00    0.00     0.00 0.00     0.00
>> 0.00    0.00    0.00    0.00   0.00   0.00
>>
>> As you can see, the DRBD devices are busy, and slowing down the VM's,
>> looking at the drives on the second server we can see why:
>> Device:         rrqm/s   wrqm/s     r/s     w/s    rMB/s wMB/s avgrq-sz
>> avgqu-sz   await r_await w_await  svctm  %util
>> sdf              67.00    76.00   64.00  113.00     0.52 0.62    13.17
>> 0.26    1.47    0.06    2.27   1.45  25.60
>> sdg              39.00    61.00   50.00  114.00     0.35 0.56    11.38
>> 0.45    2.76    0.08    3.93   2.71  44.40
>> sdd              49.00    67.00   50.00  109.00     0.39 0.57    12.40
>> 0.75    4.73    0.00    6.90   4.70  74.80
>> sdh              55.00    54.00   52.00  104.00     0.42 0.51    12.12
>> 0.81    5.21    0.23    7.69   5.13  80.00
>> sde              67.00    67.00   75.00  129.00     0.56 0.65    12.13
>> 0.94    4.59    0.69    6.85   4.24  86.40
>> sda              64.00    76.00   58.00  109.00     0.48 0.61    13.29
>> 0.84    5.03    0.21    7.60   4.89  81.60
>> sdb              35.00    72.00   57.00  104.00     0.36 0.57    11.84
>> 0.69    4.27    0.14    6.54   4.22  68.00
>> sdc             118.00   144.00  228.00  269.00     1.39 1.50    11.92
>> 1.21    2.43    1.88    2.90   1.50  74.40
>> md1               0.00     0.00    0.00  260.00     0.00 1.70    13.38
>> 0.00    0.00    0.00    0.00   0.00   0.00
>>
>> I've confirmed that the problem is that we have mixed two models of SSD (520
>> series and 530 series), and that the 530 series drives perform significantly
>> worse (under load) in comparison. Above, the two 520 series are sdf and sdg
>> while the other drives are 530 series. So, we will be replacing all of the
>> drives across both systems with 545s series 1000GB SSD's (which I've
>> confirmed will operate same or better than the 520 series, sdc on the first
>> machine above is one of these already).
>>
>> Over the years, I've learned a lot about RAID and optimisation, originally I
>> configured things to optimise for super fast streaming reads and streaming
>> writes, but in practice, the actual work-load is small random read/write,
>> with the writes causing the biggest load.
>>
>> Looking at this:
>> http://serverfault.com/questions/384273/optimizing-raid-5-for-backuppc-use-small-random-reads
>>>   *
>>>
>>>      Enhance the queue depth. Standard kernel queue depth is OK for old
>>>      single drives with small caches, but not for modern drives or RAID
>>>      arrays:
>>>
>>>      echo 512 > /sys/block/sda/queue/nr_requests
>>>
>> So my question is should I increase the configured nr_requests above the
>> current 128?
> With your workload, it probably won't matter too much.  Really high
> queue depths are great on paper, but hard to actually see.

Is there some way to see if this would help or not?
Would it hurt to increase this (even if it doesn't help)?

>
>> If the chunk size is 64k, and there are 8 drives in total, then the stripe
>> size is currently 64k*7 = 448k, is this too big? My reading of the mdadm man
>> page suggests the minimum chunk size is 4k ("In any case it must be a
>> multiple of 4KB"). If I set the chunk size to 4k, then the stripe size
>> becomes 28k, which means for a random 4k write, we only need to write 28k
>> instead of 448k ?
> This is not how a random write works.  If you are running raid-5
> before the 4.4 kernel, you get the "old" read/modify/write algorithm.
> If you write 4K, the system will read 4K from (n-2) drives, add in
> your 4K to compute parity, and write 2 drives.  This is n-2 reads + 2
> writes.  With the "new" logic in 4.4, you read the old contents of the
> 4K plus parity, and re-write the 4k plus parity, so there are 2 reads
> and 2 writes.  With big arrays, the "new" logic can help quite a bit,
> but the chatter rate is still high.  Note that the new logic is only
> raid-5.  raid-6 cannot use the new logic and has to read the stripe
> from every drive.
Hmmm, so an upgrade to kernel 4.6.3 (debian backports version) should 
provide a significant performance boost even if nothing else changes.
> The stripe size impacts when the system does can avoid doing a
> read/modify/write.  If you write a full stripe [ 64K * (n-1) ], and
> the write is exactly on a stripe boundary, and you get lucky and the
> background thread does not wake up at just the wrong time, you will do
> the write with zero reads.  I personally run with very small chunks,
> but I have code that always writes perfect stripe writes and stock
> file systems don't act that way.
So reducing the chunk size will have minimal impact... but reducing it 
should still provide some performance boost. Since I'm recreating the 
array anyway, what size makes the most sense? 16k or go straight to the 
minimum of 4k? Would a smaller chunk size increase the IOPS because we 
need to make more (smaller) requests for the same data, potentially from 
more drives?

ie, currently, a single read request for 4k will be done by reading one 
chunk (64k) from one of the 8 drives (1 IOPS)
currently, a single write request for 4k will be done by reading one 
chunk (64k) from 6 drives, and then writing one chunk (64k) to two 
drives (8 IOPS)
However, a read (or write) 48k request would be identical to the above, 
while a smaller chunk size (4k) would mean:
read request - reading 2 x 4k chunks from 5 disks and 1 x 4k chunk from 
2 disks (7 IOPS)
write request - write 8 x 4k (full stripe) (assuming it is stripe 
aligned somewhere, but it might not be)
                       - read 2 x 4k chunks (the only 2 data chunks that 
won't be written) + write 6 x 4k chunks
Total of 16 IOPS in the best case, worst case is two partial stripe 
writes + 1 full stripe write in the middle: 8 reads + 16 writes or 24 IOPS.

Either the above is wrong, or I've just convinced myself that reducing 
the chunk size is not a good idea...

> DRBD can saturate GigE without any problem with random 4K writes.  I
> have a pair of systems here that pushes 110 MB/sec at 4K or 28,000
> IOPS.  The target arrays needs to keep up, but that is another story.
> My testing with DRBD is that it starts to peter out at 10Gig, so if
> you want more bandwidth you need some other approach.  Some vendors
> use SRP over Infiniband with software raid-1 as a mirror.  iSCSI with
> iSER should give you similar results with RDMA capable ethernet.
> Linbit (the people who write DRBD) have a non GPL extension to DRBD
> that uses RDMA so you can get more bandwidth that way as well.
I have 10G ethernet for the crossover between the two servers, and 
another 10G ethernet to connect off to the "clients". Bandwidth 
utilisation on either of these is rather low (I think it maxed out at 
around 15 to 20%) definitely not anywhere near 100%. My thought here was 
on the latency of the connection, but I really didn't have any ideas on 
how to measure that, and how to test if it would really help. Also 
equipment seems a little less common, and complex...
>> The drives report a sector size of 512k, which I guess means the smallest
>> meaningful write that the drive can do is 512k, so should I increase the
>> chunk size to 512k to match? Or does that make it even worse?
>> Finally, the drive reports Host_Writes_32MiB in SMART, does that mean that
>> the drive needs to replace a entire 32MB chunk in order to overwrite a
>> sector? I'm guessing a chunk size of 32M is just crazy though...
> This is probably not true.  If the drive really had to update 512K at
> a time, then 4K writes would be 128x wear amplification.  SSDs can be
> bad, but usually not that bad.
>
>> Is there a better way to actually measure the different sizes and quantity
>> of read/writes being issued, so that I can make a more accurate decision on
>> chunk size/stripe size/etc... iostat seems to show an average numbers, but
>> not the number of 1k read/write, 4k read/write, 16k read/write etc...
> The problem is that the FTL of the SSDs are a black box and as the
> array gets bigger, the slowest drive dictates the array performance.
> This is why the "big vendors" all map SSDs in the host and avoid or
> minimize writing randomly.  I know of one vendor install that has 4000
> VDI seats (using ESXI as compute hosts) from a single HA pair of 24
> SSD shelves.  The connection to ESXI is FC and the hosts are HA with
> an IB/SRP raid-1 link between them.  Unfortunately, you need 500K+
> random write IOPS to pull this off, which I think is impossible with
> stock parity raid, and very hard with raid-10.

My environment is rather small in comparison, it is only around 20 VM's 
supporting around 80 users. 5 of the VM's are RDP servers.

>
>> My suspicion is that the actual load is made up of rather small random
>> read/write, because that is the scenario that produced the worst performance
>> results when I was initially setting this up, and seems to be what we are
>> getting in practice.
>>
>> The last option is, what if I moved to RAID10? Would that provide a
>> significant performance boost (completely removes the need to worry about
>> chunk/stripe size because we always just write the exact data we want, no
>> need to read/compute/write)?
> RAID-10 will be faster, but you pay for this with capacity.  It is
> also a double-edged sword as SSDs themselves run faster if you leave
> more free space on them, so RAID-10 absolutely might not be a lot
> faster than RAID-5 with some space left over.  Also remember that free
> space on the SSDs only counts if it is actually unallocated.  So you
> need to trim the SSDs or start with a secure erased drive and then
> never use the full capacity.  It is best to leave an empty partition
> that is untouched.
Good point, when I initially provisioned the drives, I only used the 
first 400GB, and left 80GB on each drive unpartitioned. As we ran out of 
space, I was forced to allocate all of it. The place is to only end up 
with 960GB of each 1000GB drive in use, so I could again leave a small 
chunk of un-allocated space.

>> OR, is that read/compute overhead negligible since I'm using SSD and read
>> performance is so quick?
> The reads, especially with the pre 4.4 code or with raid-6 definitely
> take their toll.  Most SSDs are also not quite symmetrical in terms of
> performance.  If your SSD does 50K read IOPS and 50K write IOPS, it
> will probably not do 25K reads and 25K writes concurrently, but
> instead stop somewhere around 18K.  But your mileage may vary.  If you
> have 8 drives that do 20 read/write symmetric, with new raid-5, each
> 4K write is 2 reads and 2 writes.  8 drives will give you 8*20K = 160K
> reads and writes or 320K total OPS.  Each 4K write takes 4 OPS, so
> your data rate ends up maxing out at 80K IOPS.  With the old raid-5
> logic, you end up with 6 reads plus two writes per "OP", so you tend
> to max out around 320K/(6+2) = 40K IOPS.  With more than 8 drives,
> these computations tend to fall apart, so 24 SSD arrays are not 3x
> faster than 8 SSD arrays, at least with stock code.
What if I moved to RAID50 and split my 8 disks into 2 x 4 disk RAID5 and 
then combined to RAID0 (or linear)? I'd end up with 6TB of usable space 
(8 x 1TB - 2 parity) though I'm guessing it is better to upgrade to 
kernel 4.4 instead which would basically do the same thing?
> You also need to consider what raid does to the SSD FTL.  As you
> chatter a drive, its wear goes up and its performance goes down.
> Different SSD models can vary wildly, but again the rule of thumb is
> keep as much free space as possible on the drives.  raid-5 or
> mirroring is also 2:1 write amplification (ie, you are writing two
> drives) and raid-6 is 3:1, on top of whatever the FTL write
> amplification is at the time.
Overall drive wear is doing pretty well, it is sitting at around 5% to 
8% per year.

Tell me I'm crazy, but one option that I considered is using different 
RAID levels. Right now I have RAID51 in that I have RAID5 on each 
machine and DRBD (RAID1) between them.
What if I used RAID01 with DRBD between the machines doing the RAID1. In 
this way, each machine has RAID0 (across 8 drives), which should provide 
maximum performance and storage capacity and DRBD doing RAID1 between 
the two machines. It feels rather risky, but perhaps it isn't a terrible 
idea?
Slightly better would be RAID10 with DRBD between each pair of drives, 
and then RAID0 across the DRBD device. It adds another layer of RAID, 
and more complexity, but better security than RAID01...

Regards,
Adam

Re: RAID5 Performance

[Doug Dumitru]

On Wed, Jul 27, 2016 at 4:25 PM, Adam Goryachev
<mailinglists@websitemanagers.com.au> wrote:
>
>
> On 27/07/2016 15:36, Doug Dumitru wrote:
>
> On Tue, Jul 26, 2016 at 7:24 PM, Adam Goryachev
> <mailinglists@websitemanagers.com.au> wrote:
>
> Hi all,
>
> I know, age old question, but I have the chance to change things up a bit,
> and I wanted to collect some thoughts/ideas.
>
> Currently I am using 8 x 480GB Intel SSD in a RAID5, then LVM on top, DRBD
> on top, and finally iSCSI on top (and then used as VM raw disks for mostly
> windows VM's).
>
> My current array looks like this:
>
> /dev/md1:
>         Version : 1.2
>   Creation Time : Wed Aug 22 00:47:03 2012
>      Raid Level : raid5
>      Array Size : 3281935552 (3129.90 GiB 3360.70 GB)
>   Used Dev Size : 468847936 (447.13 GiB 480.10 GB)
>    Raid Devices : 8
>   Total Devices : 8
>     Persistence : Superblock is persistent
>
>     Update Time : Wed Jul 27 11:32:00 2016
>           State : active
>  Active Devices : 8
> Working Devices : 8
>  Failed Devices : 0
>   Spare Devices : 0
>
>          Layout : left-symmetric
>      Chunk Size : 64K
>
>            Name : san1:1  (local to host san1)
>            UUID : 707957c0:b7195438:06da5bc4:485d301c
>          Events : 2185221
>
>     Number   Major   Minor   RaidDevice State
>        7       8       65        0      active sync   /dev/sde1
>       13       8        1        1      active sync   /dev/sda1
>        8       8       81        2      active sync   /dev/sdf1
>        5       8      113        3      active sync   /dev/sdh1
>        9       8       97        4      active sync   /dev/sdg1
>       12       8       17        5      active sync   /dev/sdb1
>       10       8       49        6      active sync   /dev/sdd1
>       11       8       33        7      active sync   /dev/sdc1
>
> I've configured the following non-standard options:
>
> echo 4096 > /sys/block/md1/md/stripe_cache_size
>
> The following apply to all SSD's installed:
> echo noop > $disk/queue/scheduler
> echo 128 > ${disk}/queue/nr_requests
>
> What I can measure (at peak periods) with iostat:
> Device:         rrqm/s   wrqm/s     r/s     w/s    rMB/s wMB/s avgrq-sz
> avgqu-sz   await r_await w_await  svctm  %util
> sdi               0.00     0.00    0.00    0.00     0.00 0.00     0.00
> 0.00    0.00    0.00    0.00   0.00   0.00
> sda              78.00    59.00   79.00   86.00     0.74 0.52    15.55
> 0.02    0.15    0.20    0.09   0.15   2.40
> sdg              35.00    48.00   68.00   79.00     0.52 0.44    13.39
> 0.02    0.14    0.24    0.05   0.11   1.60
> sdf              46.00    65.00   86.00   98.00     0.76 0.58    14.96
> 0.03    0.17    0.09    0.24   0.09   1.60
> sdh              97.00    45.00   70.00  141.00     0.66 0.68    12.96
> 0.08    0.36    0.29    0.40   0.34   7.20
> sde             101.00    75.00   87.00   94.00     0.79 0.61    15.76
> 0.08    0.42    0.32    0.51   0.29   5.20
> sdb              85.00    54.00   94.00  102.00     0.84 0.56    14.62
> 0.01    0.04    0.09    0.00   0.04   0.80
> sdc              85.00    74.00   98.00  106.00     0.79 0.66    14.53
> 0.01    0.06    0.04    0.08   0.04   0.80
> sdd             230.00   199.00  266.00  353.00     2.19 2.11    14.24
> 0.18    0.28    0.23    0.32   0.16   9.60
> drbd0             0.00     0.00    0.00    2.00     0.00 0.00     4.50
> 0.08   38.00    0.00   38.00  20.00   4.00
> drbd12            0.00     0.00    1.00    1.00     0.00 0.00     7.50
> 0.03   14.00    4.00   24.00  14.00   2.80
> drbd1             0.00     0.00    0.00    2.00     0.00 0.03    32.00
> 0.09   44.00    0.00   44.00  22.00   4.40
> drbd9             0.00     0.00    2.00    0.00     0.01 0.00     8.00
> 0.00    0.00    0.00    0.00   0.00   0.00
> drbd2             0.00     0.00    0.00    0.00     0.00 0.00     0.00
> 0.00    0.00    0.00    0.00   0.00   0.00
> drbd11            0.00     0.00    0.00    0.00     0.00 0.00     0.00
> 0.00    0.00    0.00    0.00   0.00   0.00
> drbd3             0.00     0.00    4.00  197.00     0.02 1.01    10.47
> 7.92   41.03    0.00   41.87   4.98 100.00
> drbd4             0.00     0.00    0.00    0.00     0.00 0.00     0.00
> 0.00    0.00    0.00    0.00   0.00   0.00
> drbd17            0.00     0.00    1.00    0.00     0.00 0.00     8.00
> 0.00    0.00    0.00    0.00   0.00   0.00
> drbd5             0.00     0.00    0.00    7.00     0.00 0.03     8.00
> 0.22   30.29    0.00   30.29  28.57  20.00
> drbd19            0.00     0.00    0.00    0.00     0.00 0.00     0.00
> 0.00    0.00    0.00    0.00   0.00   0.00
> drbd6             0.00     0.00    2.00    0.00     0.01 0.00     8.00
> 0.00    0.00    0.00    0.00   0.00   0.00
> drbd7             0.00     0.00    0.00    0.00     0.00 0.00     0.00
> 0.00    0.00    0.00    0.00   0.00   0.00
> drbd8             0.00     0.00    0.00    0.00     0.00 0.00     0.00
> 0.00    0.00    0.00    0.00   0.00   0.00
> drbd13            0.00     0.00   90.00   44.00     1.74 0.38    32.35
> 1.72   13.46    0.40   40.18   4.27  57.20
> drbd15            0.00     0.00    2.00   33.00     0.02 0.29    17.86
> 1.40   40.91    0.00   43.39  28.34  99.20
> drbd18            0.00     0.00    1.00    3.00     0.00 0.03    16.00
> 0.08   21.00    0.00   28.00  21.00   8.40
> drbd14            0.00     0.00    0.00    0.00     0.00 0.00     0.00
> 0.00    0.00    0.00    0.00   0.00   0.00
> drbd10            0.00     0.00    0.00    0.00     0.00 0.00     0.00
> 0.00    0.00    0.00    0.00   0.00   0.00
>
> As you can see, the DRBD devices are busy, and slowing down the VM's,
> looking at the drives on the second server we can see why:
> Device:         rrqm/s   wrqm/s     r/s     w/s    rMB/s wMB/s avgrq-sz
> avgqu-sz   await r_await w_await  svctm  %util
> sdf              67.00    76.00   64.00  113.00     0.52 0.62    13.17
> 0.26    1.47    0.06    2.27   1.45  25.60
> sdg              39.00    61.00   50.00  114.00     0.35 0.56    11.38
> 0.45    2.76    0.08    3.93   2.71  44.40
> sdd              49.00    67.00   50.00  109.00     0.39 0.57    12.40
> 0.75    4.73    0.00    6.90   4.70  74.80
> sdh              55.00    54.00   52.00  104.00     0.42 0.51    12.12
> 0.81    5.21    0.23    7.69   5.13  80.00
> sde              67.00    67.00   75.00  129.00     0.56 0.65    12.13
> 0.94    4.59    0.69    6.85   4.24  86.40
> sda              64.00    76.00   58.00  109.00     0.48 0.61    13.29
> 0.84    5.03    0.21    7.60   4.89  81.60
> sdb              35.00    72.00   57.00  104.00     0.36 0.57    11.84
> 0.69    4.27    0.14    6.54   4.22  68.00
> sdc             118.00   144.00  228.00  269.00     1.39 1.50    11.92
> 1.21    2.43    1.88    2.90   1.50  74.40
> md1               0.00     0.00    0.00  260.00     0.00 1.70    13.38
> 0.00    0.00    0.00    0.00   0.00   0.00
>
> I've confirmed that the problem is that we have mixed two models of SSD (520
> series and 530 series), and that the 530 series drives perform significantly
> worse (under load) in comparison. Above, the two 520 series are sdf and sdg
> while the other drives are 530 series. So, we will be replacing all of the
> drives across both systems with 545s series 1000GB SSD's (which I've
> confirmed will operate same or better than the 520 series, sdc on the first
> machine above is one of these already).
>
> Over the years, I've learned a lot about RAID and optimisation, originally I
> configured things to optimise for super fast streaming reads and streaming
> writes, but in practice, the actual work-load is small random read/write,
> with the writes causing the biggest load.
>
> Looking at this:
> http://serverfault.com/questions/384273/optimizing-raid-5-for-backuppc-use-small-random-reads
>
>  *
>
>     Enhance the queue depth. Standard kernel queue depth is OK for old
>     single drives with small caches, but not for modern drives or RAID
>     arrays:
>
>     echo 512 > /sys/block/sda/queue/nr_requests
>
> So my question is should I increase the configured nr_requests above the
> current 128?
>
> With your workload, it probably won't matter too much.  Really high
> queue depths are great on paper, but hard to actually see.
>
>
> Is there some way to see if this would help or not?
> Would it hurt to increase this (even if it doesn't help)?
>
>
> If the chunk size is 64k, and there are 8 drives in total, then the stripe
> size is currently 64k*7 = 448k, is this too big? My reading of the mdadm man
> page suggests the minimum chunk size is 4k ("In any case it must be a
> multiple of 4KB"). If I set the chunk size to 4k, then the stripe size
> becomes 28k, which means for a random 4k write, we only need to write 28k
> instead of 448k ?
>
> This is not how a random write works.  If you are running raid-5
> before the 4.4 kernel, you get the "old" read/modify/write algorithm.
> If you write 4K, the system will read 4K from (n-2) drives, add in
> your 4K to compute parity, and write 2 drives.  This is n-2 reads + 2
> writes.  With the "new" logic in 4.4, you read the old contents of the
> 4K plus parity, and re-write the 4k plus parity, so there are 2 reads
> and 2 writes.  With big arrays, the "new" logic can help quite a bit,
> but the chatter rate is still high.  Note that the new logic is only
> raid-5.  raid-6 cannot use the new logic and has to read the stripe
> from every drive.
>
> Hmmm, so an upgrade to kernel 4.6.3 (debian backports version) should
> provide a significant performance boost even if nothing else changes.

This should help your raid-5 array, at least noticeably, provided the
new kernel actually has the
Facebook Read/Modify/Write new logic included.  Based on the version
it should.  You can very this by doing random writes and looking at
iostat.  If you see 2 reads and 2 writes for every inbound write, you
have the new code.  If you see 6 reads and 2 writes for every inbound
write, you have the old code.

While this sounds huge, the change will be moderated by the behaviour
of SSDs.  Random writes are much more expensive than read and the new
logic only lowers the number of reads.  ... and raid-6 is not impacted
at all.

> The stripe size impacts when the system does can avoid doing a
> read/modify/write.  If you write a full stripe [ 64K * (n-1) ], and
> the write is exactly on a stripe boundary, and you get lucky and the
> background thread does not wake up at just the wrong time, you will do
> the write with zero reads.  I personally run with very small chunks,
> but I have code that always writes perfect stripe writes and stock
> file systems don't act that way.
>
> So reducing the chunk size will have minimal impact... but reducing it
> should still provide some performance boost. Since I'm recreating the array
> anyway, what size makes the most sense? 16k or go straight to the minimum of
> 4k? Would a smaller chunk size increase the IOPS because we need to make
> more (smaller) requests for the same data, potentially from more drives?
>
> ie, currently, a single read request for 4k will be done by reading one
> chunk (64k) from one of the 8 drives (1 IOPS)
> currently, a single write request for 4k will be done by reading one chunk
> (64k) from 6 drives, and then writing one chunk (64k) to two drives (8 IOPS)
> However, a read (or write) 48k request would be identical to the above,
> while a smaller chunk size (4k) would mean:
> read request - reading 2 x 4k chunks from 5 disks and 1 x 4k chunk from 2
> disks (7 IOPS)
> write request - write 8 x 4k (full stripe) (assuming it is stripe aligned
> somewhere, but it might not be)
>                       - read 2 x 4k chunks (the only 2 data chunks that
> won't be written) + write 6 x 4k chunks
> Total of 16 IOPS in the best case, worst case is two partial stripe writes +
> 1 full stripe write in the middle: 8 reads + 16 writes or 24 IOPS.

You are confused about what chunk size is.  It is not the IO size
limit.  It is just a layout calculation.  If your chunk is 64K, then
64K is written to one disk before the array moves on to the next disk.
If you read 4K, then only 4K is read.  You never need to read (or
write) and entire chunk.

Lower chunk sizes are useful if your application does enough long
writes to reach full stripes.  At 64K x 7 drives, this is 448KB.  If
you are writing multi-megabytes, then 64K chunks is a good idea.  If
you are writing 128KB, you might want to go down to 16KB chunks.  The
problem with little chunks is if you read 64K from and array with 16KB
chunks, you will cut your IO request into four parts.  This is
sometimes faster and sometimes slower.  For hard disks, bigger chunks
seems to be the way to go.  For SSDs, smaller.  I think 16K is
probably the lowest reasonable limit unless you have tested your
workload extensively, and over a long period of time, and have looked
at drive wear issues.;

> Either the above is wrong, or I've just convinced myself that reducing the
> chunk size is not a good idea...
>
> DRBD can saturate GigE without any problem with random 4K writes.  I
> have a pair of systems here that pushes 110 MB/sec at 4K or 28,000
> IOPS.  The target arrays needs to keep up, but that is another story.
> My testing with DRBD is that it starts to peter out at 10Gig, so if
> you want more bandwidth you need some other approach.  Some vendors
> use SRP over Infiniband with software raid-1 as a mirror.  iSCSI with
> iSER should give you similar results with RDMA capable ethernet.
> Linbit (the people who write DRBD) have a non GPL extension to DRBD
> that uses RDMA so you can get more bandwidth that way as well.
>
> I have 10G ethernet for the crossover between the two servers, and another
> 10G ethernet to connect off to the "clients". Bandwidth utilisation on
> either of these is rather low (I think it maxed out at around 15 to 20%)
> definitely not anywhere near 100%. My thought here was on the latency of the
> connection, but I really didn't have any ideas on how to measure that, and
> how to test if it would really help. Also equipment seems a little less
> common, and complex...

I know that DRBD will not hit 40G.  I have actually not done that much
testing at 10G.

> The drives report a sector size of 512k, which I guess means the smallest
> meaningful write that the drive can do is 512k, so should I increase the
> chunk size to 512k to match? Or does that make it even worse?
> Finally, the drive reports Host_Writes_32MiB in SMART, does that mean that
> the drive needs to replace a entire 32MB chunk in order to overwrite a
> sector? I'm guessing a chunk size of 32M is just crazy though...
>
> This is probably not true.  If the drive really had to update 512K at
> a time, then 4K writes would be 128x wear amplification.  SSDs can be
> bad, but usually not that bad.
>
> Is there a better way to actually measure the different sizes and quantity
> of read/writes being issued, so that I can make a more accurate decision on
> chunk size/stripe size/etc... iostat seems to show an average numbers, but
> not the number of 1k read/write, 4k read/write, 16k read/write etc...
>
> The problem is that the FTL of the SSDs are a black box and as the
> array gets bigger, the slowest drive dictates the array performance.
> This is why the "big vendors" all map SSDs in the host and avoid or
> minimize writing randomly.  I know of one vendor install that has 4000
> VDI seats (using ESXI as compute hosts) from a single HA pair of 24
> SSD shelves.  The connection to ESXI is FC and the hosts are HA with
> an IB/SRP raid-1 link between them.  Unfortunately, you need 500K+
> random write IOPS to pull this off, which I think is impossible with
> stock parity raid, and very hard with raid-10.
>
>
> My environment is rather small in comparison, it is only around 20 VM's
> supporting around 80 users. 5 of the VM's are RDP servers.
>
>
> My suspicion is that the actual load is made up of rather small random
> read/write, because that is the scenario that produced the worst performance
> results when I was initially setting this up, and seems to be what we are
> getting in practice.
>
> The last option is, what if I moved to RAID10? Would that provide a
> significant performance boost (completely removes the need to worry about
> chunk/stripe size because we always just write the exact data we want, no
> need to read/compute/write)?
>
> RAID-10 will be faster, but you pay for this with capacity.  It is
> also a double-edged sword as SSDs themselves run faster if you leave
> more free space on them, so RAID-10 absolutely might not be a lot
> faster than RAID-5 with some space left over.  Also remember that free
> space on the SSDs only counts if it is actually unallocated.  So you
> need to trim the SSDs or start with a secure erased drive and then
> never use the full capacity.  It is best to leave an empty partition
> that is untouched.
>
> Good point, when I initially provisioned the drives, I only used the first
> 400GB, and left 80GB on each drive unpartitioned. As we ran out of space, I
> was forced to allocate all of it. The place is to only end up with 960GB of
> each 1000GB drive in use, so I could again leave a small chunk of
> un-allocated space.
>
> OR, is that read/compute overhead negligible since I'm using SSD and read
> performance is so quick?
>
> The reads, especially with the pre 4.4 code or with raid-6 definitely
> take their toll.  Most SSDs are also not quite symmetrical in terms of
> performance.  If your SSD does 50K read IOPS and 50K write IOPS, it
> will probably not do 25K reads and 25K writes concurrently, but
> instead stop somewhere around 18K.  But your mileage may vary.  If you
> have 8 drives that do 20 read/write symmetric, with new raid-5, each
> 4K write is 2 reads and 2 writes.  8 drives will give you 8*20K = 160K
> reads and writes or 320K total OPS.  Each 4K write takes 4 OPS, so
> your data rate ends up maxing out at 80K IOPS.  With the old raid-5
> logic, you end up with 6 reads plus two writes per "OP", so you tend
> to max out around 320K/(6+2) = 40K IOPS.  With more than 8 drives,
> these computations tend to fall apart, so 24 SSD arrays are not 3x
> faster than 8 SSD arrays, at least with stock code.
>
> What if I moved to RAID50 and split my 8 disks into 2 x 4 disk RAID5 and
> then combined to RAID0 (or linear)? I'd end up with 6TB of usable space (8 x
> 1TB - 2 parity) though I'm guessing it is better to upgrade to kernel 4.4
> instead which would basically do the same thing?
>
> You also need to consider what raid does to the SSD FTL.  As you
> chatter a drive, its wear goes up and its performance goes down.
> Different SSD models can vary wildly, but again the rule of thumb is
> keep as much free space as possible on the drives.  raid-5 or
> mirroring is also 2:1 write amplification (ie, you are writing two
> drives) and raid-6 is 3:1, on top of whatever the FTL write
> amplification is at the time.
>
> Overall drive wear is doing pretty well, it is sitting at around 5% to 8%
> per year.
>
> Tell me I'm crazy, but one option that I considered is using different RAID
> levels. Right now I have RAID51 in that I have RAID5 on each machine and
> DRBD (RAID1) between them.
> What if I used RAID01 with DRBD between the machines doing the RAID1. In
> this way, each machine has RAID0 (across 8 drives), which should provide
> maximum performance and storage capacity and DRBD doing RAID1 between the
> two machines. It feels rather risky, but perhaps it isn't a terrible idea?
> Slightly better would be RAID10 with DRBD between each pair of drives, and
> then RAID0 across the DRBD device. It adds another layer of RAID, and more
> complexity, but better security than RAID01...

Your 5 to 7% wear per year is pretty safe.  I have a pair of systems
with proprietary code that is saturating dual 10GigE ports looking at
wearout at 100+ years.  Then again, the plastic cases of the drives
will be dust by then.

I don't know about you, but I do have SSDs, even from major vendors,
that fail.  They usually "just fall off the bus" with no warning.  So
I dislike skipping redundancy.  RAID turned an emergency into a
mundane task.  It is really a cost issue.  If you can afford RAID-10
and extra space, that will work best.  I don't think RAID-50 with this
few drives makes much sense.

Doug

>
> Regards,
> Adam
>
>



-- 
Doug Dumitru
EasyCo LLC

Re: RAID5 Performance

[Anthony Youngman]

On 28/07/16 01:11, Doug Dumitru wrote:
> I don't know about you, but I do have SSDs, even from major vendors,
> that fail.  They usually "just fall off the bus" with no warning.  So
> I dislike skipping redundancy.  RAID turned an emergency into a
> mundane task.  It is really a cost issue.  If you can afford RAID-10
> and extra space, that will work best.  I don't think RAID-50 with this
> few drives makes much sense.
I came across an article about testing SSDs to destruction. First the 
good news - they tended to last much longer than expected. And the bad 
news? They typically contain a self-destruct switch. Once they start 
failing, a power-cycle will (intentionally) kill them dead. ESPECIALLY 
if they're from a major vendor.

So if you don't notice they're dying, or (as in the case of the tester) 
you have power problems that tip them over the edge, your data WILL be 
gone without warning.

Cheers,
Wol

Re: RAID5 Performance

[Adam Goryachev]

On 28/07/2016 10:11, Doug Dumitru wrote:
> On Wed, Jul 27, 2016 at 4:25 PM, Adam Goryachev
> <mailinglists@websitemanagers.com.au> wrote:
>>
>> On 27/07/2016 15:36, Doug Dumitru wrote:
>>
>> On Tue, Jul 26, 2016 at 7:24 PM, Adam Goryachev
>> <mailinglists@websitemanagers.com.au> wrote:
>>
>> Hi all,
>>
>> I know, age old question, but I have the chance to change things up a bit,
>> and I wanted to collect some thoughts/ideas.
>>
>> Currently I am using 8 x 480GB Intel SSD in a RAID5, then LVM on top, DRBD
>> on top, and finally iSCSI on top (and then used as VM raw disks for mostly
>> windows VM's).
>>
>>
>> This should help your raid-5 array, at least noticeably, provided the
>> new kernel actually has the
>> Facebook Read/Modify/Write new logic included.  Based on the version
>> it should.  You can very this by doing random writes and looking at
>> iostat.  If you see 2 reads and 2 writes for every inbound write, you
>> have the new code.  If you see 6 reads and 2 writes for every inbound
>> write, you have the old code.
>>
>> While this sounds huge, the change will be moderated by the behaviour
>> of SSDs.  Random writes are much more expensive than read and the new
>> logic only lowers the number of reads.  ... and raid-6 is not impacted
>> at all.
Will definitely try this, it seems a pretty simple low-cost, and pretty 
low risk option.
>> The stripe size impacts when the system does can avoid doing a
>> read/modify/write.  If you write a full stripe [ 64K * (n-1) ], and
>> the write is exactly on a stripe boundary, and you get lucky and the
>> background thread does not wake up at just the wrong time, you will do
>> the write with zero reads.  I personally run with very small chunks,
>> but I have code that always writes perfect stripe writes and stock
>> file systems don't act that way.
>>
>> So reducing the chunk size will have minimal impact... but reducing it
>> should still provide some performance boost. Since I'm recreating the array
>> anyway, what size makes the most sense? 16k or go straight to the minimum of
>> 4k? Would a smaller chunk size increase the IOPS because we need to make
>> more (smaller) requests for the same data, potentially from more drives?
>>
>> ie, currently, a single read request for 4k will be done by reading one
>> chunk (64k) from one of the 8 drives (1 IOPS)
>> currently, a single write request for 4k will be done by reading one chunk
>> (64k) from 6 drives, and then writing one chunk (64k) to two drives (8 IOPS)
>> However, a read (or write) 48k request would be identical to the above,
>> while a smaller chunk size (4k) would mean:
>> read request - reading 2 x 4k chunks from 5 disks and 1 x 4k chunk from 2
>> disks (7 IOPS)
>> write request - write 8 x 4k (full stripe) (assuming it is stripe aligned
>> somewhere, but it might not be)
>>                        - read 2 x 4k chunks (the only 2 data chunks that
>> won't be written) + write 6 x 4k chunks
>> Total of 16 IOPS in the best case, worst case is two partial stripe writes +
>> 1 full stripe write in the middle: 8 reads + 16 writes or 24 IOPS.
> You are confused about what chunk size is.  It is not the IO size
> limit.  It is just a layout calculation.  If your chunk is 64K, then
> 64K is written to one disk before the array moves on to the next disk.
> If you read 4K, then only 4K is read.  You never need to read (or
> write) and entire chunk.
>
> Lower chunk sizes are useful if your application does enough long
> writes to reach full stripes.  At 64K x 7 drives, this is 448KB.  If
> you are writing multi-megabytes, then 64K chunks is a good idea.  If
> you are writing 128KB, you might want to go down to 16KB chunks.  The
> problem with little chunks is if you read 64K from and array with 16KB
> chunks, you will cut your IO request into four parts.  This is
> sometimes faster and sometimes slower.  For hard disks, bigger chunks
> seems to be the way to go.  For SSDs, smaller.  I think 16K is
> probably the lowest reasonable limit unless you have tested your
> workload extensively, and over a long period of time, and have looked
> at drive wear issues.;
I'm not sure here.... since my issue seems to be IOPS, wouldn't 
splitting a single IOP (ie, in your example the 64k read) into 4 IOPS (4 
x 16k reads), which would seem to exacerbate the issue (not enough IOPS 
available).

In which case, it could be beneficial to move to larger chunk sizes, so 
even a 128k request can be kept as a single IOP instead of split into 2 
? though there must also be an upper limit on the benefits here too?

At the moment, I'm thinking I will just leave the chunk size the same....
>> Either the above is wrong, or I've just convinced myself that reducing the
>> chunk size is not a good idea...
>>
>> DRBD can saturate GigE without any problem with random 4K writes.  I
>> have a pair of systems here that pushes 110 MB/sec at 4K or 28,000
>> IOPS.  The target arrays needs to keep up, but that is another story.
>> My testing with DRBD is that it starts to peter out at 10Gig, so if
>> you want more bandwidth you need some other approach.  Some vendors
>> use SRP over Infiniband with software raid-1 as a mirror.  iSCSI with
>> iSER should give you similar results with RDMA capable ethernet.
>> Linbit (the people who write DRBD) have a non GPL extension to DRBD
>> that uses RDMA so you can get more bandwidth that way as well.
>>
>> I have 10G ethernet for the crossover between the two servers, and another
>> 10G ethernet to connect off to the "clients". Bandwidth utilisation on
>> either of these is rather low (I think it maxed out at around 15 to 20%)
>> definitely not anywhere near 100%. My thought here was on the latency of the
>> connection, but I really didn't have any ideas on how to measure that, and
>> how to test if it would really help. Also equipment seems a little less
>> common, and complex...
> I know that DRBD will not hit 40G.  I have actually not done that much
> testing at 10G.

My concern is that even if I solve *this* bottleneck (ie, the 530 model 
SSD being too busy), that there will be another bottleneck afterwards 
(well, of course there will be, there is always one piece that is 
limiting performance). How will I know what/where it is (assuming it 
isn't the SSD/raid itself....).
>> The drives report a sector size of 512k, which I guess means the smallest
>> meaningful write that the drive can do is 512k, so should I increase the
>> chunk size to 512k to match? Or does that make it even worse?
>> Finally, the drive reports Host_Writes_32MiB in SMART, does that mean that
>> the drive needs to replace a entire 32MB chunk in order to overwrite a
>> sector? I'm guessing a chunk size of 32M is just crazy though...
>>
>> This is probably not true.  If the drive really had to update 512K at
>> a time, then 4K writes would be 128x wear amplification.  SSDs can be
>> bad, but usually not that bad.
>>
>> Is there a better way to actually measure the different sizes and quantity
>> of read/writes being issued, so that I can make a more accurate decision on
>> chunk size/stripe size/etc... iostat seems to show an average numbers, but
>> not the number of 1k read/write, 4k read/write, 16k read/write etc...
>>
>> The problem is that the FTL of the SSDs are a black box and as the
>> array gets bigger, the slowest drive dictates the array performance.
>> This is why the "big vendors" all map SSDs in the host and avoid or
>> minimize writing randomly.  I know of one vendor install that has 4000
>> VDI seats (using ESXI as compute hosts) from a single HA pair of 24
>> SSD shelves.  The connection to ESXI is FC and the hosts are HA with
>> an IB/SRP raid-1 link between them.  Unfortunately, you need 500K+
>> random write IOPS to pull this off, which I think is impossible with
>> stock parity raid, and very hard with raid-10.
>>
>>
>> My environment is rather small in comparison, it is only around 20 VM's
>> supporting around 80 users. 5 of the VM's are RDP servers.
>>
>>
>> My suspicion is that the actual load is made up of rather small random
>> read/write, because that is the scenario that produced the worst performance
>> results when I was initially setting this up, and seems to be what we are
>> getting in practice.
>>
>> The last option is, what if I moved to RAID10? Would that provide a
>> significant performance boost (completely removes the need to worry about
>> chunk/stripe size because we always just write the exact data we want, no
>> need to read/compute/write)?
>>
>> RAID-10 will be faster, but you pay for this with capacity.  It is
>> also a double-edged sword as SSDs themselves run faster if you leave
>> more free space on them, so RAID-10 absolutely might not be a lot
>> faster than RAID-5 with some space left over.  Also remember that free
>> space on the SSDs only counts if it is actually unallocated.  So you
>> need to trim the SSDs or start with a secure erased drive and then
>> never use the full capacity.  It is best to leave an empty partition
>> that is untouched.
>>
>> Good point, when I initially provisioned the drives, I only used the first
>> 400GB, and left 80GB on each drive unpartitioned. As we ran out of space, I
>> was forced to allocate all of it. The place is to only end up with 960GB of
>> each 1000GB drive in use, so I could again leave a small chunk of
>> un-allocated space.
>>
>> OR, is that read/compute overhead negligible since I'm using SSD and read
>> performance is so quick?
>>
>> The reads, especially with the pre 4.4 code or with raid-6 definitely
>> take their toll.  Most SSDs are also not quite symmetrical in terms of
>> performance.  If your SSD does 50K read IOPS and 50K write IOPS, it
>> will probably not do 25K reads and 25K writes concurrently, but
>> instead stop somewhere around 18K.  But your mileage may vary.  If you
>> have 8 drives that do 20 read/write symmetric, with new raid-5, each
>> 4K write is 2 reads and 2 writes.  8 drives will give you 8*20K = 160K
>> reads and writes or 320K total OPS.  Each 4K write takes 4 OPS, so
>> your data rate ends up maxing out at 80K IOPS.  With the old raid-5
>> logic, you end up with 6 reads plus two writes per "OP", so you tend
>> to max out around 320K/(6+2) = 40K IOPS.  With more than 8 drives,
>> these computations tend to fall apart, so 24 SSD arrays are not 3x
>> faster than 8 SSD arrays, at least with stock code.
>>
>> What if I moved to RAID50 and split my 8 disks into 2 x 4 disk RAID5 and
>> then combined to RAID0 (or linear)? I'd end up with 6TB of usable space (8 x
>> 1TB - 2 parity) though I'm guessing it is better to upgrade to kernel 4.4
>> instead which would basically do the same thing?
>>
>> You also need to consider what raid does to the SSD FTL.  As you
>> chatter a drive, its wear goes up and its performance goes down.
>> Different SSD models can vary wildly, but again the rule of thumb is
>> keep as much free space as possible on the drives.  raid-5 or
>> mirroring is also 2:1 write amplification (ie, you are writing two
>> drives) and raid-6 is 3:1, on top of whatever the FTL write
>> amplification is at the time.
>>
>> Overall drive wear is doing pretty well, it is sitting at around 5% to 8%
>> per year.
>>
>> Tell me I'm crazy, but one option that I considered is using different RAID
>> levels. Right now I have RAID51 in that I have RAID5 on each machine and
>> DRBD (RAID1) between them.
>> What if I used RAID01 with DRBD between the machines doing the RAID1. In
>> this way, each machine has RAID0 (across 8 drives), which should provide
>> maximum performance and storage capacity and DRBD doing RAID1 between the
>> two machines. It feels rather risky, but perhaps it isn't a terrible idea?
>> Slightly better would be RAID10 with DRBD between each pair of drives, and
>> then RAID0 across the DRBD device. It adds another layer of RAID, and more
>> complexity, but better security than RAID01...
> Your 5 to 7% wear per year is pretty safe.  I have a pair of systems
> with proprietary code that is saturating dual 10GigE ports looking at
> wearout at 100+ years.  Then again, the plastic cases of the drives
> will be dust by then.
Yep, I expect that we will outgrow the capacity of the drives before 
they "wear out". I do monitor the drive reported wear values, and alert 
on those (each time it drops 10% I get alerts, until I reset the alert 
level) so that I won't be suddenly surprised when they hit 10% or 
whatever....
> I don't know about you, but I do have SSDs, even from major vendors,
> that fail.  They usually "just fall off the bus" with no warning.  So
> I dislike skipping redundancy.  RAID turned an emergency into a
> mundane task.  It is really a cost issue.  If you can afford RAID-10
> and extra space, that will work best.  I don't think RAID-50 with this
> few drives makes much sense.
>
I'm not sure, but I think I've had one of the 480GB drives fail, and 3 
of the smaller 60GB and 80GB drives fail. So far, only the 480G failure 
was "catastrophic", the others were still operating . All were replaced 
by Intel.

The last one was just reported some large numbers in SMART, so I 
questioned Intel, and their advice was replace under warranty, which I did.
I've had many more spinning disks fail over the years though, so I'm 
pretty sure SSD's are more reliable, but certainly they do still fail, 
and that's one of the reasons for RAID (and backups of course).

Regards,
Adam

Regards,
Adam

Re: RAID5 Performance

[Peter Grandi]

[ ... ]
> My concern is that even if I solve *this* bottleneck (ie, the
> 530 model SSD being too busy), that there will be another
> bottleneck afterwards

There is always another bottleneck ;-).

[ ... ]
> I'm not sure, but I think I've had one of the 480GB drives
> fail, and 3 of the smaller 60GB and 80GB drives fail. So far,
> only the 480G failure was "catastrophic", the others were
> still operating . All were replaced by Intel.
[ ... ]

When flash SSD drives run over their "expected" write amount
they behave differently:

  http://www.sabi.co.uk/blog/15-one.html#150406b

Intel flash SSDs apparently do the following:

* They switch immediately to read-only.
* On the next power up they refuse even to *read*.

BTW, as to the Samsung SM863 there is a relatively recent "group
test" here:

  http://www.storagereview.com/samsung_sm863_ssd_review

The average and max latency graphs are interesting, especially
those under "Preconditioning Curve".

Re: RAID5 Performance

[Peter Grandi]

First a terminology point: you are reporting a *speed* problem,
not a performance problem. My impression that you are getting
pretty good performance, given your hardware, configuration and
workload. The difference between "speed" and "performance" is
that the first is a simple rate of stuff done per unit of time,
the second is an envelope embodying several tradeoffs, among
them with cost. In your question you describe a low rate of
stuff done per unit of time. :-)

> Currently I am using 8 x 480GB Intel SSD in a RAID5, then LVM
> on top, DRBD on top, and finally iSCSI on top (and then used
> as VM raw disks for mostly windows VM's).

A very brave configuration, a shining example of the "syntactic"
mindset, according to which any arbitrary combination of
legitimate features must be fine :-).

First server DRBD primary disks:

> Device:         rrqm/s   wrqm/s     r/s     w/s    rMB/s wMB/s avgrq-sz avgqu-sz   await r_await w_await  svctm  %util
> sdi               0.00     0.00    0.00    0.00     0.00 0.00     0.00     0.00    0.00    0.00    0.00   0.00   0.00
> sda              78.00    59.00   79.00   86.00     0.74 0.52     15.55     0.02    0.15    0.20    0.09   0.15   2.40
> sdg              35.00    48.00   68.00   79.00     0.52 0.44     13.39     0.02    0.14    0.24    0.05   0.11   1.60
> sdf              46.00    65.00   86.00   98.00     0.76 0.58     14.96     0.03    0.17    0.09    0.24   0.09   1.60
> sdh              97.00    45.00   70.00  141.00     0.66 0.68     12.96     0.08    0.36    0.29    0.40   0.34   7.20
> sde             101.00    75.00   87.00   94.00     0.79 0.61     15.76     0.08    0.42    0.32    0.51   0.29   5.20
> sdb              85.00    54.00   94.00  102.00     0.84 0.56     14.62     0.01    0.04    0.09    0.00   0.04   0.80
> sdc              85.00    74.00   98.00  106.00     0.79 0.66     14.53     0.01    0.06    0.04    0.08   0.04   0.80
> sdd             230.00   199.00  266.00  353.00     2.19 2.11     14.24     0.18    0.28    0.23    0.32   0.16   9.60

Second server DRBD secondary disks:

> Device:         rrqm/s   wrqm/s     r/s     w/s    rMB/s wMB/s avgrq-sz avgqu-sz   await r_await w_await  svctm  %util
> sdf              67.00    76.00   64.00  113.00     0.52 0.62    13.17     0.26    1.47    0.06    2.27   1.45  25.60
> sdg              39.00    61.00   50.00  114.00     0.35 0.56    11.38     0.45    2.76    0.08    3.93   2.71  44.40
> sdd              49.00    67.00   50.00  109.00     0.39 0.57    12.40     0.75    4.73    0.00    6.90   4.70  74.80
> sdh              55.00    54.00   52.00  104.00     0.42 0.51    12.12     0.81    5.21    0.23    7.69   5.13  80.00
> sde              67.00    67.00   75.00  129.00     0.56 0.65    12.13     0.94    4.59    0.69    6.85   4.24  86.40
> sda              64.00    76.00   58.00  109.00     0.48 0.61    13.29     0.84    5.03    0.21    7.60   4.89  81.60
> sdb              35.00    72.00   57.00  104.00     0.36 0.57    11.84     0.69    4.27    0.14    6.54   4.22  68.00
> sdc             118.00   144.00  228.00  269.00     1.39 1.50    11.92     1.21    2.43    1.88    2.90   1.50  74.40
> md1               0.00     0.00    0.00  260.00     0.00 1.70    13.38     0.00    0.00    0.00    0.00   0.00   0.00

> I've confirmed that the problem is that we have mixed two
> models of SSD (520 series and 530 series), and that the 530
> series drives perform significantly worse (under load) in
> comparison.

The queue sizes and waiting time on the second server are very
low (on a somewhat similar system using 4TB disks I see waiting
times in the 1-5 seconds range, not milliseconds).

The impression I get is that there is some issue with DRBD
latency, because the second server's storage seems to me very
underutilized. This latency may be related to the flash SSDs
that you are using, because by default DRBD uses the "C"
synchronization protocol. Probably if you switched to the "B" or
even "A" protocols speed could improve, maybe a lot, even if
performance arguably would be the same or much worse.

Thus the most likely issue here is the 'fsync' problem: for
"consumerish" SSDs barrier-writes are synchronous, because they
don't have a battery/capacitor-backed cache, and rather slow for
small writes, because of the large size of erase blocks, which
can be mitigated with higher over-provisioning. These have much
of the story:

  https://www.sebastien-han.fr/blog/2014/10/10/ceph-how-to-test-if-your-ssd-is-suitable-as-a-journal-device/
  https://www.redhat.com/en/resources/ceph-pcie-ssd-performance-part-1
  http://www.spinics.net/lists/ceph-users/msg25928.html

The 520s seem not too bad, but still a long way from the disks
with battery/capacity-backed cache.

> the actual work-load is small random read/write, with the
> writes causing the biggest load.

Here most of the wise comments from the reply from D Dimitru
apply, to summarize:

* Small writes are a challenging workload for DRBD, regardless
  of other issues.

* Small writes are a very challenging workload for flash SSDs
  without battery/capacitor-backed caches.

* Parity RAID is a bad idea in general, in particular for
  workloads with many small writes, for they amplify writes via
  RMW.

etc. etc. :-)

Re: RAID5 Performance

[Doug Dumitru]

On Wed, Jul 27, 2016 at 7:26 AM, Peter Grandi <pg@lxra2.for.sabi.co.uk> wrote:
> First a terminology point: you are reporting a *speed* problem,
> not a performance problem. My impression that you are getting
> pretty good performance, given your hardware, configuration and
> workload. The difference between "speed" and "performance" is
> that the first is a simple rate of stuff done per unit of time,
> the second is an envelope embodying several tradeoffs, among
> them with cost. In your question you describe a low rate of
> stuff done per unit of time. :-)
>
>> Currently I am using 8 x 480GB Intel SSD in a RAID5, then LVM
>> on top, DRBD on top, and finally iSCSI on top (and then used
>> as VM raw disks for mostly windows VM's).
>
> A very brave configuration, a shining example of the "syntactic"
> mindset, according to which any arbitrary combination of
> legitimate features must be fine :-).

While you may say that this configuration is very "brave", it is
actually quite common for VDI "appliance" deployments.  If you look at
cluster solutions like Ganeti, it is exactly this stack less the iSCSI
(Ganeti runs storage and VM compute on the same nodes).  It is also a
"LOT" faster and a lot lower wear than using a file system like ZFS to
create ZVOLs and export those.  The other option is to run a
file-system and export files as block devices.  This just replaces LVM
with EXT4/XFS, and for static "blobs", LVM is a bunch faster and
safer.

>
> First server DRBD primary disks:
>
>> Device:         rrqm/s   wrqm/s     r/s     w/s    rMB/s wMB/s avgrq-sz avgqu-sz   await r_await w_await  svctm  %util
>> sdi               0.00     0.00    0.00    0.00     0.00 0.00     0.00     0.00    0.00    0.00    0.00   0.00   0.00
>> sda              78.00    59.00   79.00   86.00     0.74 0.52     15.55     0.02    0.15    0.20    0.09   0.15   2.40
>> sdg              35.00    48.00   68.00   79.00     0.52 0.44     13.39     0.02    0.14    0.24    0.05   0.11   1.60
>> sdf              46.00    65.00   86.00   98.00     0.76 0.58     14.96     0.03    0.17    0.09    0.24   0.09   1.60
>> sdh              97.00    45.00   70.00  141.00     0.66 0.68     12.96     0.08    0.36    0.29    0.40   0.34   7.20
>> sde             101.00    75.00   87.00   94.00     0.79 0.61     15.76     0.08    0.42    0.32    0.51   0.29   5.20
>> sdb              85.00    54.00   94.00  102.00     0.84 0.56     14.62     0.01    0.04    0.09    0.00   0.04   0.80
>> sdc              85.00    74.00   98.00  106.00     0.79 0.66     14.53     0.01    0.06    0.04    0.08   0.04   0.80
>> sdd             230.00   199.00  266.00  353.00     2.19 2.11     14.24     0.18    0.28    0.23    0.32   0.16   9.60
>
> Second server DRBD secondary disks:
>
>> Device:         rrqm/s   wrqm/s     r/s     w/s    rMB/s wMB/s avgrq-sz avgqu-sz   await r_await w_await  svctm  %util
>> sdf              67.00    76.00   64.00  113.00     0.52 0.62    13.17     0.26    1.47    0.06    2.27   1.45  25.60
>> sdg              39.00    61.00   50.00  114.00     0.35 0.56    11.38     0.45    2.76    0.08    3.93   2.71  44.40
>> sdd              49.00    67.00   50.00  109.00     0.39 0.57    12.40     0.75    4.73    0.00    6.90   4.70  74.80
>> sdh              55.00    54.00   52.00  104.00     0.42 0.51    12.12     0.81    5.21    0.23    7.69   5.13  80.00
>> sde              67.00    67.00   75.00  129.00     0.56 0.65    12.13     0.94    4.59    0.69    6.85   4.24  86.40
>> sda              64.00    76.00   58.00  109.00     0.48 0.61    13.29     0.84    5.03    0.21    7.60   4.89  81.60
>> sdb              35.00    72.00   57.00  104.00     0.36 0.57    11.84     0.69    4.27    0.14    6.54   4.22  68.00
>> sdc             118.00   144.00  228.00  269.00     1.39 1.50    11.92     1.21    2.43    1.88    2.90   1.50  74.40
>> md1               0.00     0.00    0.00  260.00     0.00 1.70    13.38     0.00    0.00    0.00    0.00   0.00   0.00
>
>> I've confirmed that the problem is that we have mixed two
>> models of SSD (520 series and 530 series), and that the 530
>> series drives perform significantly worse (under load) in
>> comparison.
>
> The queue sizes and waiting time on the second server are very
> low (on a somewhat similar system using 4TB disks I see waiting
> times in the 1-5 seconds range, not milliseconds).

The expectation, in terms of performance for VDI is quite high.
vmWare like to say you can get away with 8-12 IOPS per virtual.  Most
people think you only get good performance with 100 IOPS per virtual.
The "bad" time for VDI is what is called a "boot storm".  Boot, or
reboot, all of the windows clients at the same time and see how long
they take to settle.  The IO workload for this is 80%+ 4K random
writes.  At 100 IOPS, it takes windows about 2 minutes to boot, so if
you need to support 500 VDI seats from a storage node, that nodes
needs to sustain 500 x 100 = 50,000 IOPS of 4K random writes.  This is
so far past what hard disks can do as to be silly.  Even with SSDs,
you need reasonably large arrays running RAID-10 to sustain this.  If
you want to support 5000 VDI seats like this, then stock Linux just
can't get there, but it can be done.
>
> The impression I get is that there is some issue with DRBD
> latency, because the second server's storage seems to me very
> underutilized. This latency may be related to the flash SSDs
> that you are using, because by default DRBD uses the "C"
> synchronization protocol. Probably if you switched to the "B" or
> even "A" protocols speed could improve, maybe a lot, even if
> performance arguably would be the same or much worse.
>
> Thus the most likely issue here is the 'fsync' problem: for
> "consumerish" SSDs barrier-writes are synchronous, because they
> don't have a battery/capacitor-backed cache, and rather slow for
> small writes, because of the large size of erase blocks, which
> can be mitigated with higher over-provisioning. These have much
> of the story:

On many consumer SSDs, barrier writes are only barriers, and are not
syncs at all.  You are guaranteed serialization but not actual
storage.  Then again, in a server setup, especially with redundant
power supplies, power loss to the SSDs is rare.  You are more
protecting against system hangs and other inter-connectivity issues.
The real system solution is to have some quantity of non volatile DRAM
that you can stage writes (either a PCI-e card like a FlashTec or one
or more nvDIMMs).  This is how the "major vendors" deal with sync
writes.

>   https://www.sebastien-han.fr/blog/2014/10/10/ceph-how-to-test-if-your-ssd-is-suitable-as-a-journal-device/
>   https://www.redhat.com/en/resources/ceph-pcie-ssd-performance-part-1
>   http://www.spinics.net/lists/ceph-users/msg25928.html
>
> The 520s seem not too bad, but still a long way from the disks
> with battery/capacity-backed cache.
>
>> the actual work-load is small random read/write, with the
>> writes causing the biggest load.
>
> Here most of the wise comments from the reply from D Dimitru
> apply, to summarize:
>
> * Small writes are a challenging workload for DRBD, regardless
>   of other issues.

My comment about DRBD was not that small writes are harder, but that
if your target can keep up with them at low queue depths, then DRBD
can saturate GigE at 4K q=1 on a single thread.  So DRBD is not really
the issue, but the latency/IOPS behaviour of the target.

> * Small writes are a very challenging workload for flash SSDs
>   without battery/capacitor-backed caches.

Even with battery backup, small writes create garbage collection, so
while batteries may give you some short term bursts, longer term, you
still have to do the writes.  A main benefit of batter backup in the
SSDs is that the meta data (mapping information) does not need to be
flushed with the actual data real-time, which makes the FTL algorithms
easier to implement.

> * Parity RAID is a bad idea in general, in particular for
>   workloads with many small writes, for they amplify writes via
>   RMW.
>
> etc. etc. :-)
> --
> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html



-- 
Doug Dumitru
WildFire Storage
http://www.wildfire-storage.com

Re: RAID5 Performance

[Peter Grandi]

[ ... ]

>> A very brave configuration, a shining example of the
>> "syntactic" mindset, according to which any arbitrary
>> combination of legitimate features must be fine :-).

> While you may say that this configuration is very "brave", it
> is actually quite common for VDI "appliance" deployments. [
> ... ]

There are a lot of very "brave" sysadms out there, and often I
have to clean up after them :-).

But then I am one of those boring people who think that «VDI
"appliance" deployments» are usually a phenomenally bad idea, as
it requires a storage layer that has to cover all possible IO
workloads optimally, as indeed in:

  > [ ... ] The expectation, in terms of performance for VDI is
  > quite high.  vmWare like to say you can get away with 8-12
  > IOPS per virtual. Most people think you only get good
  > performance with 100 IOPS per virtual. [ ... ]

Those 100 random IOPS per VM are a bit "random", but roughly
translate to one "disk arm" per VM, which is not necessarily
enough: http://www.sabi.co.uk/blog/15-one.html#150305

[ ... ]

>> The queue sizes and waiting time on the second server are
>> very low (on a somewhat similar system using 4TB disks I see
>> waiting times in the 1-5 seconds range, not milliseconds).

> The expectation, in terms of performance for VDI is quite high.
> [ ... ]

Sure, but the point here as to the speed issue is not that the
SSDs are overwhelmed with IO, as the traffic on them is low and
has relatively low latency, it is that very few IOPS are getting
retired.

>> Thus the most likely issue here is the 'fsync' problem: for
>> "consumerish" SSDs barrier-writes are synchronous, because
>> they don't have a battery/capacitor-backed cache, and rather
>> slow for small writes, because of the large size of erase
>> blocks, which can be mitigated with higher over-provisioning.

> On many consumer SSDs, barrier writes are only barriers, and
> are not syncs at all. You are guaranteed serialization but not
> actual storage.

Probably in this case that is irrelevant, because the numbers
coming out from both the OP's experience and the tests in the
links I mentioned show that small sync writes seem synchronous
indeed for the 520/530, resulting in small write rates of around
1-5 MB/s, which matches the reported stats.

> Then again, in a server setup, especially with redundant power
> supplies, power loss to the SSDs is rare. You are more
> protecting against system hangs and other inter-connectivity
> issues.

That is also likely irrelevant here. The firmware in the flash
SSD does not know about the system setup, and the DRBD is
probably configured to request synchronous writes on the
secondary with protocol "C".

BTW I don't know whether the process(es) writing to the DRBD
primary also request synchronous writes, but that's hopefully the
case too, if the VD layer has been configured properly.

> The real system solution is to have some quantity of non
> volatile DRAM that you can stage writes (either a PCI-e card
> like a FlashTec or one or more nvDIMMs).

If this were the case then the VD layer and the DRBD layer could
be told not to use sync writes, but the numbers reported seem to
indicate that sync writes are happening.

> This is how the "major vendors" deal with sync writes.

At the system level, but at the device level the "major vendors"
put a large capacitor in "enterprise" SSDs for two reasons, one
of them to allow the persistence of the RAM write buffer, to
minimize write amplification and erase latency (the other is not
relevant here).

[ ... ]

>> * Small writes are a very challenging workload for flash SSDs
>>   without battery/capacitor-backed caches.

> Even with battery backup, small writes create garbage
> collection, so while batteries may give you some short term
> bursts,

That problem is mitigated with bigger overprovisioning in
"enterprise" class flash SSDs. It can also be done in those of
the "consumerish" class by partitioning them appropriately, or
with 'hdparm -N'; but that does not seem to be the case here,
becase the reported stats show a small number of IOPS with lowish
queues sizes and not that huge latencies.

> longer term, you still have to do the writes.

Unfortunately flash SSDs don't merely have to "do the writes",
as things are quite different: as I mentioned above the issue is
the large erase blocks (and the several milliseconds it takes to
erase one).

In the absence of power backing for the write cache, every sync
write, for example a 4KiB one, is (usually) stored immediately to
a flash chip, which means (usually) a lot of write amplification
because of RMW on the 8MiB (or larger) erase block plus the large
latency (often near 10 milliseconds) of the erase operation
before erase block programming.

That largely explains why in the tests I have mentioned small
sync write IOPS for many "consumerish" flash SSDs top at around
100, instead of the usual > 10,000 for small non-sync writes.

Some flash SSDs use an additional SLC buffer with smaller erase
blocks and lower latency to reduce the problem with flushing sync
writes directly to MLC etc, and that may explain why the 520s are
better than the 530s (if the 520s have an SLC buffer, but IIRC
intel started using an SLC buffer with the 540 series).

Flash SSDs have only been popular for around 5 years, so it is
understandable that some important aspects of their performance
envelope (like what may happen on sync writes) is not well known
yet.
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: RAID5 Performance

[Peter Grandi]

[ ... ]
> That largely explains why in the tests I have mentioned small
> sync write IOPS for many "consumerish" flash SSDs top at around
> 100, instead of the usual > 10,000 for small non-sync writes.
[ ... ]

To summarize the preceding long discussion:

* The stats reported show a low level of IOPS being carried out.

* The critical part of the workload seems to be synchronous
  small writes.

* Probably then the primary issue is the use of flash SSDs that
  have a limited number of IOPS for small synchronous writes.

* A secondary issue is that RAID5 results in RMW for small
  writes.

There are two possible options:

* Replace the flash SSDs with those that are known to deliver
  high (at least > 10,000 single threaded) small synchronous
  write IOPS.

* Relax the requirement for synchronous writes on *both* the
  primary and secondary DRBD servers, if feeling lucky.

The third option, which is to change the workload so that it
does not emit small synchronous writes to the storage layer,
seems not practical in the context.

Ideally the system would also be switched from RAID5 to RAID10
to avoid the large penalty on small writes at the RAID level
too.

That may be considered expensive, but as I wrote:

> [ ... ] it requires a storage layer that has to cover all
> possible IO workloads optimally, [ ... ]

Re: RAID5 Performance

[Adam Goryachev]

On 28/07/2016 23:28, Peter Grandi wrote:
> [ ... ]
>> That largely explains why in the tests I have mentioned small
>> sync write IOPS for many "consumerish" flash SSDs top at around
>> 100, instead of the usual > 10,000 for small non-sync writes.
> [ ... ]
>
> To summarize the preceding long discussion:
>
> * The stats reported show a low level of IOPS being carried out.
>
> * The critical part of the workload seems to be synchronous
>    small writes.
>
> * Probably then the primary issue is the use of flash SSDs that
>    have a limited number of IOPS for small synchronous writes.
>
> * A secondary issue is that RAID5 results in RMW for small
>    writes.
>
> There are two possible options:
>
> * Replace the flash SSDs with those that are known to deliver
>    high (at least > 10,000 single threaded) small synchronous
>    write IOPS.
Is there a "known" SSD that you would suggest? My problem is that Intel 
spec sheets seem to suggest that there is little performance difference 
across the range of SSD's, so it's really not clear which SSD model I 
should buy. Obviously it's not something I can afford to buy one of each 
and test them either.
> * Relax the requirement for synchronous writes on *both* the
>    primary and secondary DRBD servers, if feeling lucky.
I have the following entries for DRBD which were suggested by linbit 
(which previously lifted performance from abysmal to more than 
sufficient around 2+ years ago). I guess we are demanding more from the 
system now, and we have added the 530 model drives later...

         disk-barrier no;
         disk-flushes no;
         md-flushes no;

I've not configured anything special for LVM/MD/iSCSI/xen in relation to 
cache/buffer/etc, and windows has disabled the write back buffer option 
(within the VM).
> The third option, which is to change the workload so that it
> does not emit small synchronous writes to the storage layer,
> seems not practical in the context.
>
> Ideally the system would also be switched from RAID5 to RAID10
> to avoid the large penalty on small writes at the RAID level
> too.
>
> That may be considered expensive, but as I wrote:
>
>> [ ... ] it requires a storage layer that has to cover all
>> possible IO workloads optimally, [ ... ]
That will be the final optimisation/fallback option if still needed. I'd 
prefer to avoid that as it limits the capacity of the system, and will 
obviously cost more.

Do you have any other suggestions or ideas that might assist?

Thanks,
Adam

Re: RAID5 Performance

[Peter Grandi]

[ ... ]

>> * Replace the flash SSDs with those that are known to deliver
>>   high (at least > 10,000 single threaded) small synchronous
>>   write IOPS.

> Is there a "known" SSD that you would suggest? My problem is
> that Intel spec sheets seem to suggest that there is little
> performance difference across the range of SSD's, so it's
> really not clear which SSD model I should buy.

The links I wrote earlier have lists:

>>>   https://www.sebastien-han.fr/blog/2014/10/10/ceph-how-to-test-if-your-ssd-is-suitable-as-a-journal-device/
>>>   http://www.spinics.net/lists/ceph-users/msg25928.html
>>>   https://www.redhat.com/en/resources/ceph-pcie-ssd-performance-part-1

As one of those pages says the Samsung SM863 looks attractive,
but for historical reasons so far I have only seen Intel DCs in
similar use. There discussions of other models in various posts
related to Ceph journal SSD usage.

> Obviously it's not something I can afford to buy one of each
> and test them either.

In addition to the lists above I have justed tested my three
home flash SSDs:

* Micron M4 256GB:
    #  dd bs=4k count=100000 oflag=direct,dsync if=/dev/zero of=/var/tmp/TEST
    100000+0 records in
    100000+0 records out
    409600000 bytes (410 MB) copied, 1200.3 s, 341 kB/s
* Samsung 850 Pro 256GB:
    #  dd bs=4k count=100000 oflag=direct,dsync if=/dev/zero of=/var/tmp/TEST
    100000+0 records in
    100000+0 records out
    409600000 bytes (410 MB) copied, 1732.93 s, 236 kB/s
* Hynix SK SH910 256GB:
    #  dd bs=4k count=100000 oflag=direct,dsync if=/dev/zero of=/var/tmp/TEST
    100000+0 records in
    100000+0 records out
    409600000 bytes (410 MB) copied, 644.742 s, 635 kB/s

So I would not recommend any of them for "small sync writes"
workloads :-), but they are quite good otherwise. I do notice
they are slow on small sync writes when downloading mail, as
each message is duly 'fsync'ed.

BTW as bonus material, I have done on the SH910 an abbreviated
test with block sizes between 4KiB and 1024KiB:

  #  for N in 4k 16k 64k 128k 256k 512k 1024k; do echo -n "$N: "; dd bs=$N count=1000 oflag=dsync if=/dev/zero of=/var/tmp/TEST |& grep copied; done
  4k: 4096000 bytes (4.1 MB) copied, 6.23481 s, 657 kB/s
  16k: 16384000 bytes (16 MB) copied, 6.29379 s, 2.6 MB/s
  64k: 65536000 bytes (66 MB) copied, 6.09223 s, 10.8 MB/s
  128k: 131072000 bytes (131 MB) copied, 6.5487 s, 20.0 MB/s
  256k: 262144000 bytes (262 MB) copied, 6.93361 s, 37.8 MB/s
  512k: 524288000 bytes (524 MB) copied, 7.73957 s, 67.7 MB/s
  1024k: 1048576000 bytes (1.0 GB) copied, 12.8671 s, 81.5 MB/s

Note how the time to write 1000 blocks is essentially the same
betweeen 4KiB and 128KiB, which is quite amusing. Probably the
flash-page size is around 256KiB.

For additional bonus value the same on a "fastish" consumer 2TB
disk, a Seagate ST2000DM001:

  #  for N in 4k 16k 64k 128k 256k 512k 1024k; do echo -n "$N: "; dd bs=$N count=1000 oflag=dsync if=/dev/zero of=/fs/sdb6/tmp/TEST |& grep copied; done
  4k: 4096000 bytes (4.1 MB) copied, 44.9177 s, 91.2 kB/s
  16k: 16384000 bytes (16 MB) copied, 38.131 s, 430 kB/s
  64k: 65536000 bytes (66 MB) copied, 35.8263 s, 1.8 MB/s
  128k: 131072000 bytes (131 MB) copied, 35.8188 s, 3.7 MB/s
  256k: 262144000 bytes (262 MB) copied, 36.6838 s, 7.1 MB/s
  512k: 524288000 bytes (524 MB) copied, 37.0612 s, 14.1 MB/s
  1024k: 1048576000 bytes (1.0 GB) copied, 42.0844 s, 24.9 MB/s

>> * Relax the requirement for synchronous writes on *both* the
>>   primary and secondary DRBD servers, if feeling lucky.

> I have the following entries for DRBD which were suggested by
> linbit (which previously lifted performance from abysmal to
> more than sufficient around 2+ years ago). [ ... ]

That's an inappropriate use of "performance" here:

>          disk-barrier no;
>          disk-flushes no;
>          md-flushes no;

That "feeling lucky" list seems to me to have made performance
lower (in the sense that the performance of writing to
'/dev/null' is zero, even if the speed is really good :->).

With those settings the data sync policy is "disk-drain", which
also involves some waiting, but somewhat dangerous, except "In
case your backing storage device has battery-backed write cache"
(and "device" here means system and host adapter and disk); it
is not clear to me for metadata what "md-flushes no" gives.

BTW if you have battery-backed everything on the secondary side
you could use protocol "B".

However given those it looks likely that the bottleneck is also
on the primary DRBD side.

> Do you have any other suggestions or ideas that might assist?

* Smaller RAID5 stripes, as in 4+1 or 2+1, are cheaper in space
  than RAID10 and enormously raise the chances that a full
  stripe-write can happen (it still has the write-hole problem
  of parity RAID).

* Make sure the DRBD journal is also on a separate device that
  allows fast small sync writes.

Also, I have appended a sample DRBD configuration I have used:

----------------------------------------------------------------

resource r0
{
  device		  /dev/drbd_r0 minor 0;
  # A: "local disk and local TCP send buffer"
  # B: "local disk and remote buffer cache"
  # C: "both local and remote disk"
  protocol		  C;

  net
  {
    # As mentioned on IRC by a DRBD guy, this is not really a
    # secret, but more a "unique id" that ensures that replicas
    # of different resources don't get accidentally connected.
    # Still to be ABR-ized.
    shared-secret	    "xxxxxxxxxxxx";

    cram-hmac-alg	    sha1;
    ping-timeout	    50;

    after-sb-0pri	    discard-zero-changes;
    after-sb-1pri	    discard-secondary;
    after-sb-2pri	    disconnect;

    # http://article.gmane.org/gmane.linux.network.drbd/18348
    # http://www.drbd.org/users-guide-8.3/s-throughput-tuning.html
    # https://alteeve.ca/w/AN!Cluster_Tutorial_2_-_Performance_Tuning
    # http://fghaas.wordpress.com/2007/06/22/performance-tuning-drbd-setups/
    sndbuf-size		    0;
    rcvbuf-size		    0;
    max-buffers		    16384;
    unplug-watermark	    16384;
    max-epoch-size	    16384;
  }

  syncer
  {
    csums-alg		    sha1;
    # At 45MB/s takes 6 hour per 1TB.
    rate		    95M;

    use-rle;
  }

  startup
  {
    wfc-timeout		    15;
    degr-wfc-timeout	    15;
    outdated-wfc-timeout    15;

    # Cannot be an address, must be output of 'hostname'.
    become-primary-on	    host-1;
  }

  on host-1
  {
    address		    192.168.1.11:7788;
    disk		    /dev/md2;
    flexible-meta-disk	    /dev/local0/r0_md;
  }

  on host-2
  {
    address		    192.168.1.12:7788;
    disk		    /dev/md2;
    flexible-meta-disk	    /dev/local0/r0_md;
  }
}

Re: RAID5 Performance

[Doug Dumitru]

... boy this thread is getting long.

A couple of points.

* I am one of a reasonably small group of people that have actually
written an FTL and have it in production use.  FTLs in SSDs are some
of the most closely guarded "implementations" I have ever seen.  I am
not sure if my FTL matches others, as I have not seen the others, but
the patent office thinks my version is unique enough (not that it
really matters).

* Many SSDs, even consumer models, and even models without battery
backup, usually enforce correct serialization.  If you have a single
drive in a laptop, this is what is important.  If you have an array in
a server, and if the power to the SSDs are protected, then this also
protects your data.  You need to separate the failures you are trying
to protect.  SuperCaps on an SSD that is behind redundant power
supplies on redundant UPSs is perhaps not the best place to spend your
money.  Likewise, if you have an HA link, the write is not ACKed until
the other node gets the data at least into the memory buffer.  Are you
trying to engineer against multiple failures at multiple sites.  You
need to decide on the level of redundancy of redundancy of redundancy.

* Assumptions that FTLs wear sync writes at the ratio of the write
block to the erase block sizes are usually wrong.  Older "dumb" flash
like CF and SD cards sometimes work like this, but even there they
have gotten better.  An easier assumption is that "normal" SSDs will
have write amplification at the inverse of free space percentage.  So
your consumer drive with 8% free has 1/.08=12.5:1 write amplification.
This is why data center drives have more free space.  "Better" FTLs,
when working with 100% random workloads can lower this to just over
50% of this value.  My FTL sees 5.45:1 write amp on a 100% random
workload steady state at 10% free.

* Real workloads are sometimes that same as random workloads and
sometimes very different.  Some FTLs can exploit the patterns in a
real file system workload and some cannot.  For example, my FTL sees
1.3:1 write amp with the JEDEC 128GB client trace at 10% free versus
the typical 9:1 for most consumer SSDs on the same trace with about
the same free space.

* Additional games are possible if you start to reach "into" the
blocks.  Compression that only saves you 10% might not seem like much,
but if it moves the free space from 8% to 18%, it matters a lot.  The
above examples were without compression.  Compression can also make
file system write overhead "go away".  It is not uncommon for a
journal and directory entry write to compress 80+% even though the
data is binary.  This makes old hard drive optimizations like
"-noatime" unnecessary.

* You can do a lot more with an FTL if you move it "in front" of raid.
This basically eliminates the raid read/modify/write operation and
overhead entirely.  It does introduce a new "write hole" aspect to the
array, but this can be plugged with nvRAM hardware.

Happy Hunting.

Doug Dumitru

Re: RAID5 Performance

[Adam Goryachev]

On 29/07/16 03:20, Peter Grandi wrote:
> [ ... ]
>
>>> * Replace the flash SSDs with those that are known to deliver
>>>    high (at least > 10,000 single threaded) small synchronous
>>>    write IOPS.
>> Is there a "known" SSD that you would suggest? My problem is
>> that Intel spec sheets seem to suggest that there is little
>> performance difference across the range of SSD's, so it's
>> really not clear which SSD model I should buy.
> The links I wrote earlier have lists:
Thanks for reminding me of that. I see that the list reflects my 
experience (if we assume the 530 model is equivalent to the 535 model on 
the list, and my 520 480GB is equivalent to the 520 on the list).

However, I can't get the budget for those really awesome drives up the 
top of the list, that would require around $20k... or more.

For now, I've got 16 x 545s TB drives, and have replaced the first half 
(ie, all drives in one server). Now I can see that the drives themselves 
don't seem to be the bottleneck (the drives don't run at 100% util, 
while the DRBD device does run at 100%).

I've written a small script to keep track of the number of seconds each 
drive util value fits into each bracket (increments of 10%). Let me know 
if you would like a copy (it's just a perl script which reads iostat 
output, I'm sure it could be written much nicer).
So far, this is what I get on the secondary (with the new 8 x 845s 1TB 
drives):
Drive          10        20       30       40    50    60    70 80    
90   100
md1        19265         0        0        0      0      0 0      0      
0    0
sda         17029    1579    404    137    49    45    13 4      4    1
sdb         16983    1453    477    179    77    63    22      6 3     2
sdc         16867    1579    492    182    76    40    17      8 1     3
sdd         17043    1499    445    154    59    40    14      6 3     2
sde         17064    1506    415    152    68    32    15      4 6     3
sdf          17138    1467    396    152    46    37    11    10 4     4
sdg         17118    1493    401    139    56    31    14      7 2     4
sdh         16997    1577    407    138    62    45    11    10 7     6
sdi          19236        12        4        4      2      0      2      
3     0     1

Hopefully that will line up right !
So, out of the last 19265 seconds, each of the underlying drives was at 
100% for only a couple of seconds (sdi is the OS drive). ie, the last 
column shows the number of seconds the drive was at 90 to 100% util as 
reported by iostat. The 10 column shows number of seconds between 0 and 
10%, etc...

Looking at the primary, with all 520 series drives (except sda which is 
a 545s series) and the DRBD drives I see this:

Drive           10         20        30     40     50    60    70 80    
90    100
drbd0        19971    108       54     36     13      2      0 2      1 
      1
drbd1        19842    165       77     48     34      4      6 5      3  
     3
drbd10      19766    279       62     35     23      7      6 4      
2      1
drbd11      20081    37         32     21     12      1      3 1      
0      0
drbd12      20041    79         38     19       9      1      0 0      
1      0
drbd13      16195    2335   758   338   220  131    77    39 32    58
drbd14      19765    230       90     49     30      9      4 6      2   
    1
drbd15        3473    6323 4136 2250 1390  913  614  443  418  220
drbd17      20175    9             1       0       3      0      0      
0      0      0
drbd18      19878    170       65     29     23    10      4 0      6      1
drbd19      19255    368     138     86     87  100    39    35 44    35
drbd2        20188    0             0       0       0      0      0     
0      0      0
drbd3        17457    1276   610   316   175  140    66    43 33    56
drbd4        20154    17          6        6       5      0      0     
0      0      0
drbd5        19859    141      59      38     26    10      4 5      3    42
drbd6        20112    39        20        9       3      1      1   1    
   1      0
drbd7        20188    0            0        0       0      0      0     
0      0      0
drbd8        19894    136      78      44     22      5      3 2      0  
     2
drbd9        19476    289    211    123     41    21      9      6     
3      7
md1          20188    0            0        0       0      0      0     
0      0      0
sda           16948    1696   439    286   213  206  316    81 3      0
sdb           16059    2177   844    402   290  352    50    13 1      0
sdc           16141    2132   852    388   312  328    30      5    0      0
sdd           15914    2182   956    395   300  362    72      6    1      0
sde           16099    2137   801    393   256  366  124    10 1      1
sdf            16000    2169   898    408   322  340    39      9    3   
    0
sdg           15929    2265   822    418   259  290  195      8 2      0
sdh           16107    2129   822    419   324  337    41      9    0      0
sdi            20155    3             3        7     14      6 0      
0      0      0

So on the primary, I see even less of a bottleneck on the underlying 
drives, which doesn't make a lot of sense to me. The secondary has less 
read load (since all reads are handled by the primary), and should only 
need to deal with raid rmw. Also, I'm not sure, but I think the 
secondary does less meta data updates for DRBD. So I can only presume 
the new drives are much better than the 530 series, but still not as 
good as the 520 series. I'll need to run some tests before I put the 
drives live next time.

However, the point of note is that DRBD devices are showing high util 
levels much more frequently than the underlying devices, so I can only 
assume that the current limitation is caused by DRBD rather than the 
drives. Though probably solving the DRBD issue will then go back to the 
drives being the limit, with not a lot of difference. See below for my 
(your) ideas on improving both of those things.....

>>>>    https://www.sebastien-han.fr/blog/2014/10/10/ceph-how-to-test-if-your-ssd-is-suitable-as-a-journal-device/
>>>>    http://www.spinics.net/lists/ceph-users/msg25928.html
>>>>    https://www.redhat.com/en/resources/ceph-pcie-ssd-performance-part-1
> As one of those pages says the Samsung SM863 looks attractive,
> but for historical reasons so far I have only seen Intel DCs in
> similar use. There discussions of other models in various posts
> related to Ceph journal SSD usage.
>
>> Obviously it's not something I can afford to buy one of each
>> and test them either.
> In addition to the lists above I have justed tested my three
> home flash SSDs:
>
> * Micron M4 256GB:
>      #  dd bs=4k count=100000 oflag=direct,dsync if=/dev/zero of=/var/tmp/TEST
>      100000+0 records in
>      100000+0 records out
>      409600000 bytes (410 MB) copied, 1200.3 s, 341 kB/s
> * Samsung 850 Pro 256GB:
>      #  dd bs=4k count=100000 oflag=direct,dsync if=/dev/zero of=/var/tmp/TEST
>      100000+0 records in
>      100000+0 records out
>      409600000 bytes (410 MB) copied, 1732.93 s, 236 kB/s
> * Hynix SK SH910 256GB:
>      #  dd bs=4k count=100000 oflag=direct,dsync if=/dev/zero of=/var/tmp/TEST
>      100000+0 records in
>      100000+0 records out
>      409600000 bytes (410 MB) copied, 644.742 s, 635 kB/s
>
> So I would not recommend any of them for "small sync writes"
> workloads :-), but they are quite good otherwise. I do notice
> they are slow on small sync writes when downloading mail, as
> each message is duly 'fsync'ed.
>
> BTW as bonus material, I have done on the SH910 an abbreviated
> test with block sizes between 4KiB and 1024KiB:
>
>    #  for N in 4k 16k 64k 128k 256k 512k 1024k; do echo -n "$N: "; dd bs=$N count=1000 oflag=dsync if=/dev/zero of=/var/tmp/TEST |& grep copied; done
>    4k: 4096000 bytes (4.1 MB) copied, 6.23481 s, 657 kB/s
>    16k: 16384000 bytes (16 MB) copied, 6.29379 s, 2.6 MB/s
>    64k: 65536000 bytes (66 MB) copied, 6.09223 s, 10.8 MB/s
>    128k: 131072000 bytes (131 MB) copied, 6.5487 s, 20.0 MB/s
>    256k: 262144000 bytes (262 MB) copied, 6.93361 s, 37.8 MB/s
>    512k: 524288000 bytes (524 MB) copied, 7.73957 s, 67.7 MB/s
>    1024k: 1048576000 bytes (1.0 GB) copied, 12.8671 s, 81.5 MB/s
>
> Note how the time to write 1000 blocks is essentially the same
> betweeen 4KiB and 128KiB, which is quite amusing. Probably the
> flash-page size is around 256KiB.
>
> For additional bonus value the same on a "fastish" consumer 2TB
> disk, a Seagate ST2000DM001:
>
>    #  for N in 4k 16k 64k 128k 256k 512k 1024k; do echo -n "$N: "; dd bs=$N count=1000 oflag=dsync if=/dev/zero of=/fs/sdb6/tmp/TEST |& grep copied; done
>    4k: 4096000 bytes (4.1 MB) copied, 44.9177 s, 91.2 kB/s
>    16k: 16384000 bytes (16 MB) copied, 38.131 s, 430 kB/s
>    64k: 65536000 bytes (66 MB) copied, 35.8263 s, 1.8 MB/s
>    128k: 131072000 bytes (131 MB) copied, 35.8188 s, 3.7 MB/s
>    256k: 262144000 bytes (262 MB) copied, 36.6838 s, 7.1 MB/s
>    512k: 524288000 bytes (524 MB) copied, 37.0612 s, 14.1 MB/s
>    1024k: 1048576000 bytes (1.0 GB) copied, 42.0844 s, 24.9 MB/s
>

Yep, definitely won't be going backwards to spinning disks :)

>>> * Relax the requirement for synchronous writes on *both* the
>>>    primary and secondary DRBD servers, if feeling lucky.
>> I have the following entries for DRBD which were suggested by
>> linbit (which previously lifted performance from abysmal to
>> more than sufficient around 2+ years ago). [ ... ]
> That's an inappropriate use of "performance" here:
>
>>           disk-barrier no;
>>           disk-flushes no;
>>           md-flushes no;
> That "feeling lucky" list seems to me to have made performance
> lower (in the sense that the performance of writing to
> '/dev/null' is zero, even if the speed is really good :->).
>
> With those settings the data sync policy is "disk-drain", which
> also involves some waiting, but somewhat dangerous, except "In
> case your backing storage device has battery-backed write cache"
> (and "device" here means system and host adapter and disk); it
> is not clear to me for metadata what "md-flushes no" gives.
>
> BTW if you have battery-backed everything on the secondary side
> you could use protocol "B".
 From my understanding, the times these settings can cause a problem:
1) When both servers hard power off - possibly all the latest data is 
not written to disk that the VM's expect. If this is the case, all the 
VM's were also hard powered off, and so the VM has no idea about what it 
expects to be written/not. The end user may need to redo some work/etc, 
but that is acceptable. Worst case scenario, a DB file is corrupted and 
needs to be restored from the previous night backup, and users must redo 
all work, which is also "acceptable" (from a risk point of view).
2) One server hard power off, perhaps power supply failure/etc - When it 
powers on again, it should re-sync with the DRBD primary, and 
potentially we do a DRBD verify to confirm everything is good. As long 
as there is no failure on the primary, then everything is good. Worst 
case, catastrophic failure of the primary before the verify is complete, 
or before the secondary comes on-line again, and basically we treat it 
as above.

We can't deal with every possible scenario, as the cost is prohibitive, 
we can only deal with the more common scenarios, and those that are 
cheaper to deal with. eg, all equipment is protected by UPS, using 
redundancy RAID instead of linear/striping, and using DRBD for 
replication. Most likely failures are disk, power supply, or network 
cables (ie, unplugged by accident/etc), and this setup protects well for 
all three of those.
> However given those it looks likely that the bottleneck is also
> on the primary DRBD side.
>
>> Do you have any other suggestions or ideas that might assist?
> * Smaller RAID5 stripes, as in 4+1 or 2+1, are cheaper in space
>    than RAID10 and enormously raise the chances that a full
>    stripe-write can happen (it still has the write-hole problem
>    of parity RAID).
I was planning to upgrade to the 4.4.x kernel, which would kind of solve 
this, since it will only read from 2 drives anyway, but it turns out 
that is more difficult than I expected. (iscsitarget kernel module 
doesn't compile cleanly with the new kernel, and it doesn't seem to be 
well supported into such recent kernel versions. I'll probably wait 
until debian testing becomes stable, or at least a lot closer, before 
going down that path).

I could potentially move to 2 x RAID5 with 3+1 and then linear or stripe 
those, which means I only lose one extra disk of capacity.... Will need 
to think about that further...

> * Make sure the DRBD journal is also on a separate device that
>    allows fast small sync writes.

I think this would be the next option to investigate. Currently the DRBD 
journal is on the same devices.
Reading from: 
http://www.drbd.org/en/doc/users-guide-84/ch-internals#s-internal-meta-data
>
> *Advantage. *For some write operations, using external meta data 
> produces a somewhat improved latency behavior.
>
Do you have any more knowledge on the expected performance advantage? 
ie, would half the writes move from the data drive to the meta data drive?
I'm thinking it might be plausible to purchase 2 x Intel P3700 400GB and 
put one in each DRBD server for the meta data updates. Although if this 
isn't going to make much difference (eg, only 20%) then it is less 
likely to be worthwhile...
Can anyone suggest what kind of performance improvement might I see by 
doing this?
The alternative (for double the cost + a bit more) would be to migrate 
from RAID5 to RAID10, is that likely to produce a better/worse result?

2 x P3700 400GB is probably around $2500, while 12 x 545s 1000GB is 
around $4800, but would need to add another SATA controller card, which 
probably means changing motherboard/CPU/etc as well, so that becomes a 
lot more....

> Also, I have appended a sample DRBD configuration I have used:
>
> ----------------------------------------------------------------
>
>      # http://article.gmane.org/gmane.linux.network.drbd/18348
>      # http://www.drbd.org/users-guide-8.3/s-throughput-tuning.html
>      # https://alteeve.ca/w/AN!Cluster_Tutorial_2_-_Performance_Tuning
>      # http://fghaas.wordpress.com/2007/06/22/performance-tuning-drbd-setups/
>      sndbuf-size		    0;
>      rcvbuf-size		    0;
>      max-buffers		    16384;
>      unplug-watermark	    16384;
>      max-epoch-size	    16384;
I have similar values, but will need to investigate the above options 
further. rcvbuf-size doesn't seem to be well documented, at least in the 
DRBD 8.4 manual, but will research these some more. Then will also need 
to check how to modify the values without causing a system meltdown....

Thanks again for your advice/information, it is very helpful.

Regards,
Adam



-- 
Adam Goryachev Website Managers www.websitemanagers.com.au

Re: RAID5 Performance

[Peter Grandi]

[ ... ]

> However, I can't get the budget for those really awesome
> drives up the top of the list, that would require around
> $20k... or more.

> For now, I've got 16 x 545s TB drives, and have replaced the
> first half (ie, all drives in one server). Now I can see that
> the drives themselves don't seem to be the bottleneck (the
> drives don't run at 100% util, while the DRBD device does run
> at 100%).

The "%util" number is not that easy to interpret, especially for
flash SSD and in some situations which probably include this
one:

  https://brooker.co.za/blog/2014/07/04/iostat-pct.html

> Hopefully that will line up right !

It is hard to read, and I don't understand what the numbers are,
but it does not matter a lot.

> So I can only presume the new drives are much better than the
> 530 series, but still not as good as the 520 series.

The 540s have an SLC write buffer as I mentioned previously,
which should help.

> However, the point of note is that DRBD devices are showing
> high util levels much more frequently than the underlying
> devices, so I can only assume that the current limitation is
> caused by DRBD rather than the drives.

I like guessing, but this assumptions seems to me a bit
excessive.

> From my understanding, the times these settings can cause a
> problem: [ ... ]

If you don't have reliable sync barriers at all levels (not just
DRBD) *any* crash (e.g. bug crash, mistake-crash, memory-full
crash, not just power crash) is going to cause massive trouble,
especially in a mostly-write workload where what is being
written is cache spill. Some interesting pages:

http://blog.2ndquadrant.com/intel_ssd_now_off_the_sherr_sh/
http://wiki.postgresql.org/wiki/Reliable_Writes
http://archive.is/WTeAE
https://news.ycombinator.com/item?id=6973179
http://lkcl.net/reports/ssd_analysis.html

>>> Do you have any other suggestions or ideas that might
>>> assist?

Another one that would likely give a bit of relief as you can't
budget for write-optimized "enterprise" flash SSDs is a SATA/SAS
host adapter with a very large battery-backed RAM buffer. As the
tests that I previously mentioned show, longer writes result in
much improved write rates on "consumer" flash SSD devices, and
hopefully the large buffer results in:

 #1 When the large write buffer flushes, *hopefully* much longer
    writes to the flash SSD will happen on average.

 #2 Thanks to the battery backing, writes are reporte completed
    to the OS when they reach the host adapter's buffer, rather
    than the flash SSD layer.

If #1 does not happen #2 won't help much if writes are at the
flash SSD saturation level, only if they are bursty and on
average below it.

>> * Smaller RAID5 stripes, as in 4+1 or 2+1, are cheaper in
>>   space than RAID10 and enormously raise the chances that a
>>   full stripe-write can happen (it still has the write-hole
>>   problem of parity RAID).

> I was planning to upgrade to the 4.4.x kernel, which would
> kind of solve this, [ ... ]

The write hole workaround in MD RAID relies on a mostly-write
journal device like for DRBD.

>> * Make sure the DRBD journal is also on a separate device
>>   that allows fast small sync writes.

> I think this would be the next option to investigate.
> Currently the DRBD journal is on the same devices.

That means that every sync'ed write becomes two writes to the
same device.

> 2 x P3700 400GB is probably around $2500,

The Samsung SM863 I ahve already mentioned are write-optimized
too and much cheaper, at around $300-350 for the 480GB model.

> while 12 x 545s 1000GB is around $4800, [ ... ]

Many people try to use "consumer" drives to build manager wowing
systems that have huge capacity and low cost, but vendors are not
stupid, and make sure that premium priced "enteprise" drives have
some critical advantage for at least some important workloads
(usually write heavy, guessing that "enterprise" workloads that
can command premium prices are transactional); sometimes like for
SSDs the advantages are based on real stuff, capacitors and
overprovisioning, which do cost money, sometimes artificial like
disabling SCT/ERC control.

Re: RAID5 Performance

[Adam Goryachev]

On 28/07/2016 22:19, Peter Grandi wrote:
> [ ... ]
>
>>> A very brave configuration, a shining example of the
>>> "syntactic" mindset, according to which any arbitrary
>>> combination of legitimate features must be fine :-).
>> While you may say that this configuration is very "brave", it
>> is actually quite common for VDI "appliance" deployments. [
>> ... ]
> There are a lot of very "brave" sysadms out there, and often I
> have to clean up after them :-).
>
> But then I am one of those boring people who think that «VDI
> "appliance" deployments» are usually a phenomenally bad idea, as
> it requires a storage layer that has to cover all possible IO
> workloads optimally, as indeed in:
Could I ask what you would "clean up" in the above system? What layers 
would you remove/simplify?
At it's simplest, this system should be able to export a block of disk 
space which the remote machine can present as a block device to the VM 
(using Xen). Keep in mind there are actually multiple of these remote 
machines. The method I've chosen is re-written here:
8 x 480GB Intel SSD (mix of 520 and 530 models)
Linux MD RAID5
LVM2
DRBD (takes an LV from each san and joins it together)
iSCSI (exports the block device to the 10Gbps network)
iSCSI (imported to the remove machine 2 x 1Gbps network)
multipathd (join the two iSCSI connections together)
Xen

Also, the two san machines have a second 10Gbps connection directly 
between them.

Originally I had DRBD below LVM, but the folks at linbit switched those 
two around to improve DRBD performance (multiple smaller DRBD devices is 
better than one large, this might have changed since that happened 
around 3 years ago).
>    > [ ... ] The expectation, in terms of performance for VDI is
>    > quite high.  vmWare like to say you can get away with 8-12
>    > IOPS per virtual. Most people think you only get good
>    > performance with 100 IOPS per virtual. [ ... ]
>
> Those 100 random IOPS per VM are a bit "random", but roughly
> translate to one "disk arm" per VM, which is not necessarily
> enough: http://www.sabi.co.uk/blog/15-one.html#150305
>
> [ ... ]
>
>>> The queue sizes and waiting time on the second server are
>>> very low (on a somewhat similar system using 4TB disks I see
>>> waiting times in the 1-5 seconds range, not milliseconds).
>> The expectation, in terms of performance for VDI is quite high.
>> [ ... ]
> Sure, but the point here as to the speed issue is not that the
> SSDs are overwhelmed with IO, as the traffic on them is low and
> has relatively low latency, it is that very few IOPS are getting
> retired.
>
Previously when I was doing lots of tests on the system, I found I could 
get great IO using larger block sizes, up to 2.5GB/s read and 1.5GB/s write.
Eventually, I found that using more real-world sized blocks, eg, 1k to 
4k, I got abysmal transfer rates.

>>> Thus the most likely issue here is the 'fsync' problem: for
>>> "consumerish" SSDs barrier-writes are synchronous, because
>>> they don't have a battery/capacitor-backed cache, and rather
>>> slow for small writes, because of the large size of erase
>>> blocks, which can be mitigated with higher over-provisioning.
>> On many consumer SSDs, barrier writes are only barriers, and
>> are not syncs at all. You are guaranteed serialization but not
>> actual storage.
> Probably in this case that is irrelevant, because the numbers
> coming out from both the OP's experience and the tests in the
> links I mentioned show that small sync writes seem synchronous
> indeed for the 520/530, resulting in small write rates of around
> 1-5 MB/s, which matches the reported stats.
>
>> Then again, in a server setup, especially with redundant power
>> supplies, power loss to the SSDs is rare. You are more
>> protecting against system hangs and other inter-connectivity
>> issues.
> That is also likely irrelevant here. The firmware in the flash
> SSD does not know about the system setup, and the DRBD is
> probably configured to request synchronous writes on the
> secondary with protocol "C".
Yep, using protocol C at the moment.
> BTW I don't know whether the process(es) writing to the DRBD
> primary also request synchronous writes, but that's hopefully the
> case too, if the VD layer has been configured properly.
>
>> The real system solution is to have some quantity of non
>> volatile DRAM that you can stage writes (either a PCI-e card
>> like a FlashTec or one or more nvDIMMs).
> If this were the case then the VD layer and the DRBD layer could
> be told not to use sync writes, but the numbers reported seem to
> indicate that sync writes are happening.
>
>> This is how the "major vendors" deal with sync writes.
> At the system level, but at the device level the "major vendors"
> put a large capacitor in "enterprise" SSDs for two reasons, one
> of them to allow the persistence of the RAM write buffer, to
> minimize write amplification and erase latency (the other is not
> relevant here).
>
> [ ... ]
>
>>> * Small writes are a very challenging workload for flash SSDs
>>>    without battery/capacitor-backed caches.
>> Even with battery backup, small writes create garbage
>> collection, so while batteries may give you some short term
>> bursts,
> That problem is mitigated with bigger overprovisioning in
> "enterprise" class flash SSDs. It can also be done in those of
> the "consumerish" class by partitioning them appropriately, or
> with 'hdparm -N'; but that does not seem to be the case here,
> becase the reported stats show a small number of IOPS with lowish
> queues sizes and not that huge latencies.
Can you advise what numbers I should look for, or worry about, which 
would indicate that the problem is (or isn't) a erase cycle delay problem?
>> longer term, you still have to do the writes.
> Unfortunately flash SSDs don't merely have to "do the writes",
> as things are quite different: as I mentioned above the issue is
> the large erase blocks (and the several milliseconds it takes to
> erase one).
>
> In the absence of power backing for the write cache, every sync
> write, for example a 4KiB one, is (usually) stored immediately to
> a flash chip, which means (usually) a lot of write amplification
> because of RMW on the 8MiB (or larger) erase block plus the large
> latency (often near 10 milliseconds) of the erase operation
> before erase block programming.
>
> That largely explains why in the tests I have mentioned small
> sync write IOPS for many "consumerish" flash SSDs top at around
> 100, instead of the usual > 10,000 for small non-sync writes.
>
> Some flash SSDs use an additional SLC buffer with smaller erase
> blocks and lower latency to reduce the problem with flushing sync
> writes directly to MLC etc, and that may explain why the 520s are
> better than the 530s (if the 520s have an SLC buffer, but IIRC
> intel started using an SLC buffer with the 540 series).
I can see the 545s series performs similar (or better, can't tell yet) 
than the 520 series, but certainly it is better than the 530.
>
> Flash SSDs have only been popular for around 5 years, so it is
> understandable that some important aspects of their performance
> envelope (like what may happen on sync writes) is not well known
> yet.
>
Thank you, I appreciate all the responses.

So far, I've decided to make the following two changes:
1) Replace all 16 existing SSD's with the 1000GB 545s model, this will 
double the capacity, and remove any of the 530 model drives. My concern 
was (and is) that making actual use of this double capacity with the 
same performance per drive will in effect halve the performance. I will 
be able to leave 40G per drive un-partitioned, or partitioned and left 
blank whichever is better.... Not sure if 40G per drive is enough to 
help with the write/erase problem, but I guess it should be better than 
nothing.
I think this change will produce a 40% improvement (potentially, given 
the 520 drives are at 40% while the 530 is at 100%)

2) Upgrade to Linux kernel 4.6 from debian backports.
I think this change will give approx 30% improvement, because it will 
reduce reads by 4 for each write, but a read is quicker than a write, so 
I'm hoping for 30% overall. It sounds like the performance should be 
somewhere between current and RAID10.

With the above two improvements, I'm hoping it will be enough to solve 
the problem.

At this stage, if it is not enough to solve the problem, my fall-back 
option is to convert to RAID10 but it's something I'd prefer to avoid 
based on cost, storage capacity, and the fact it is difficult to expand 
the existing system past 8 drives (hence capacity)...

I'm not convinced that changing chunk size will offer any benefit 
(positive or negative), so will likely leave that as it is (64k chunk size).

Regards,
Adam
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html