Re: performance of raid5 on fast devices

[Roman Mamedov <rm@romanrm.net>]

On Mon, 16 Jan 2017 21:35:21 -0500
Jake Yao <jgyao1@gmail.com> wrote:

> I have a raid5 array on 4 NVMe drives, and the performance on the
> array is only marginally better than a single drive. Unlike a similar
> raid5 array on 4 SAS SSD or HDD,  the performance on array is 3x
> better than a single drive, which is expected.
> 
> It looks like when the single kernel thread associated with the raid
> device running at 100%, the array performance hit its peak. This can
> happen easily for fast devices like NVMe.
> 
> This can reproduced by creating a raid5 with 4 ramdisks as well, and
> comparing performance on the array and one ramdisk. Sometimes the
> performance on the array is worse than a single ramdisk.
> 
> The kernel version is 4.9.0-rc3 and mdadm is release 3.4, no write
> journal is configured.
> 
> Is this a known issue?

How do you measure the performance?

Sure it may be CPU-bound in the end, but also why not try the usual
optimization tricks, such as:

  * increase your stripe_cache_size, it's not uncommon that this can speed up
    linear writes by as much as several times;

  * if you meant reads, you could look into read-ahead settings for the array;

  * and in both cases, try experimenting with different stripe sizes (if you
    were using 512K, try with 64K stripes).

-- 
With respect,
Roman