RAID-10 near vs. RAID-1

RAID-10 near vs. RAID-1

[Piergiorgio Sartor]

Hi all,

I'm setting up a system with 2 SSD M.2 (NVME).

I was wondering if would it be better, performace
wise, to have a RAID-10 near layout or a RAID-1.

Looking around I found only one benchmark:

https://strugglers.net/~andy/blog/2019/06/02/exploring-different-linux-raid-10-layouts-with-unbalanced-devices/

Which uses mixed SSD, NVME and SATA.

Does anybody have any suggestions, links, or
ideas on the topic?

BTW, practically speaking, what's the difference,
between the two RAIDs?

Thanks!

bye,

-- 

piergiorgio

Re: RAID-10 near vs. RAID-1

[Reindl Harald]

Am 11.06.24 um 20:31 schrieb Piergiorgio Sartor:
> I'm setting up a system with 2 SSD M.2 (NVME).
> 
> I was wondering if would it be better, performace
> wise, to have a RAID-10 near layout or a RAID-1.
> 
> Looking around I found only one benchmark:
> 
> https://strugglers.net/~andy/blog/2019/06/02/exploring-different-linux-raid-10-layouts-with-unbalanced-devices/
> 
> Which uses mixed SSD, NVME and SATA.
> 
> Does anybody have any suggestions, links, or
> ideas on the topic?
> 
> BTW, practically speaking, what's the difference,
> between the two RAIDs?

i wouldn't even consider a RAID10 with two disks, especially with SSD 
and practically you end with a unsupported RAID1 because there are no 
stripes with 2 disks

Re: RAID-10 near vs. RAID-1

[Paul E Luse]

On Wed, 12 Jun 2024 01:04:18 +0200
Reindl Harald <h.reindl@thelounge.net> wrote:

> 
> 
> Am 11.06.24 um 20:31 schrieb Piergiorgio Sartor:
> > I'm setting up a system with 2 SSD M.2 (NVME).
> > 
> > I was wondering if would it be better, performace
> > wise, to have a RAID-10 near layout or a RAID-1.
> > 
> > Looking around I found only one benchmark:
> > 
> > https://strugglers.net/~andy/blog/2019/06/02/exploring-different-linux-raid-10-layouts-with-unbalanced-devices/
> > 
> > Which uses mixed SSD, NVME and SATA.
> > 
> > Does anybody have any suggestions, links, or
> > ideas on the topic?
> > 
> > BTW, practically speaking, what's the difference,
> > between the two RAIDs?
> 
> i wouldn't even consider a RAID10 with two disks, especially with SSD 
> and practically you end with a unsupported RAID1 because there are no 
> stripes with 2 disks
> 
> 
I don't disagree but I would recommend you try each variation and
measure the performance for yourself.  It's a great learning experience
if you haven't done it before and there's nothing like trusting your
own data over on your own system/config something that someone else has
done when there are so many factors that can affect performance.

-Paul

Re: RAID-10 near vs. RAID-1

[Reindl Harald]

Am 12.06.24 um 02:14 schrieb Paul E Luse:
> On Wed, 12 Jun 2024 01:04:18 +0200
> Reindl Harald <h.reindl@thelounge.net> wrote:
> 
>>
>>
>> Am 11.06.24 um 20:31 schrieb Piergiorgio Sartor:
>>> I'm setting up a system with 2 SSD M.2 (NVME).
>>>
>>> I was wondering if would it be better, performace
>>> wise, to have a RAID-10 near layout or a RAID-1.
>>>
>>> Looking around I found only one benchmark:
>>>
>>> https://strugglers.net/~andy/blog/2019/06/02/exploring-different-linux-raid-10-layouts-with-unbalanced-devices/
>>>
>>> Which uses mixed SSD, NVME and SATA.
>>>
>>> Does anybody have any suggestions, links, or
>>> ideas on the topic?
>>>
>>> BTW, practically speaking, what's the difference,
>>> between the two RAIDs?
>>
>> i wouldn't even consider a RAID10 with two disks, especially with SSD
>> and practically you end with a unsupported RAID1 because there are no
>> stripes with 2 disks
>>
>>
> I don't disagree but I would recommend you try each variation and
> measure the performance for yourself.  It's a great learning experience
> if you haven't done it before and there's nothing like trusting your
> own data over on your own system/config something that someone else has
> done when there are so many factors that can affect performance.

the problem with benchmarks is that they often don't reflect mixed, 
real-world performance - for virtual machine workload the "far" layout 
wins in case of HDD

with NVME RAID i pretend it don't matter enough to even waste your time

Re: RAID-10 near vs. RAID-1

[Reindl Harald]

Am 12.06.24 um 09:06 schrieb Reindl Harald:
> 
> 
> Am 12.06.24 um 02:14 schrieb Paul E Luse:
>> On Wed, 12 Jun 2024 01:04:18 +0200
>> Reindl Harald <h.reindl@thelounge.net> wrote:
>>
>>>
>>>
>>> Am 11.06.24 um 20:31 schrieb Piergiorgio Sartor:
>>>> I'm setting up a system with 2 SSD M.2 (NVME).
>>>>
>>>> I was wondering if would it be better, performace
>>>> wise, to have a RAID-10 near layout or a RAID-1.
>>>>
>>>> Looking around I found only one benchmark:
>>>>
>>>> https://strugglers.net/~andy/blog/2019/06/02/exploring-different-linux-raid-10-layouts-with-unbalanced-devices/
>>>>
>>>> Which uses mixed SSD, NVME and SATA.
>>>>
>>>> Does anybody have any suggestions, links, or
>>>> ideas on the topic?
>>>>
>>>> BTW, practically speaking, what's the difference,
>>>> between the two RAIDs?
>>>
>>> i wouldn't even consider a RAID10 with two disks, especially with SSD
>>> and practically you end with a unsupported RAID1 because there are no
>>> stripes with 2 disks
>>>
>>>
>> I don't disagree but I would recommend you try each variation and
>> measure the performance for yourself.  It's a great learning experience
>> if you haven't done it before and there's nothing like trusting your
>> own data over on your own system/config something that someone else has
>> done when there are so many factors that can affect performance.
> 
> the problem with benchmarks is that they often don't reflect mixed, 
> real-world performance - for virtual machine workload the "far" layout 
> wins in case of HDD
> 
> with NVME RAID i pretend it don't matter enough to even waste your time

and with only TWO disks thins might look compleltly different

a 4 disk RAID10 in case of large reads can use all 4 drives at the same 
time while with only two disks there isn't much to gain when you end 
with a practical RAID1 but the complexer scheme

the whole point of RAID10 is that you can double size *and* performance 
because of a least 4 disks and striping

if performance *really* matters buy 4 NVME drives with half the size or 
just keep thins simple

Re: RAID-10 near vs. RAID-1

[Piergiorgio Sartor]

On Tue, Jun 11, 2024 at 05:14:33PM -0700, Paul E Luse wrote:
> On Wed, 12 Jun 2024 01:04:18 +0200
> Reindl Harald <h.reindl@thelounge.net> wrote:
> 
> > 
> > 
> > Am 11.06.24 um 20:31 schrieb Piergiorgio Sartor:
> > > I'm setting up a system with 2 SSD M.2 (NVME).
> > > 
> > > I was wondering if would it be better, performace
> > > wise, to have a RAID-10 near layout or a RAID-1.
> > > 
> > > Looking around I found only one benchmark:
> > > 
> > > https://strugglers.net/~andy/blog/2019/06/02/exploring-different-linux-raid-10-layouts-with-unbalanced-devices/
> > > 
> > > Which uses mixed SSD, NVME and SATA.
> > > 
> > > Does anybody have any suggestions, links, or
> > > ideas on the topic?
> > > 
> > > BTW, practically speaking, what's the difference,
> > > between the two RAIDs?
> > 
> > i wouldn't even consider a RAID10 with two disks, especially with SSD 
> > and practically you end with a unsupported RAID1 because there are no 
> > stripes with 2 disks
> > 
> > 
> I don't disagree but I would recommend you try each variation and
> measure the performance for yourself.  It's a great learning experience
> if you haven't done it before and there's nothing like trusting your
> own data over on your own system/config something that someone else has
> done when there are so many factors that can affect performance.
> 
> -Paul
> 

Hi, thanks to you as well.

About benchmarking, you're absolutely right, I did
in the past for other RAIDs (-5, -6 and -10).

I was just hoping to skip it, this time... :-)

I do not have really a performance problem to solve.
I'm just curious, given that the two RAIDs seem
identical, if someone already tested the two.

Thanks,

bye,

-- 

piergiorgio

Re: RAID-10 near vs. RAID-1

[Piergiorgio Sartor]

On Wed, Jun 12, 2024 at 01:04:18AM +0200, Reindl Harald wrote:
> 
> 
> Am 11.06.24 um 20:31 schrieb Piergiorgio Sartor:
> > I'm setting up a system with 2 SSD M.2 (NVME).
> > 
> > I was wondering if would it be better, performace
> > wise, to have a RAID-10 near layout or a RAID-1.
> > 
> > Looking around I found only one benchmark:
> > 
> > https://strugglers.net/~andy/blog/2019/06/02/exploring-different-linux-raid-10-layouts-with-unbalanced-devices/
> > 
> > Which uses mixed SSD, NVME and SATA.
> > 
> > Does anybody have any suggestions, links, or
> > ideas on the topic?
> > 
> > BTW, practically speaking, what's the difference,
> > between the two RAIDs?
> 
> i wouldn't even consider a RAID10 with two disks, especially with SSD and
> practically you end with a unsupported RAID1 because there are no stripes
> with 2 disks
> 

Hi, thanks for the answer.

I'm a bit confused here. What do you mean
with "unsupported RAID1"?

As far as I know, but please correct me if
I'm wrong, a Linux md RAID-10 *near* layout,
with 2 devices, has identical data distribution
as a RAID-1 with 2 devices.
Meaning the 2 devices are a mirror.

The difference, if I understood it correctly,
is that the RAID-10 has chunks, and hence stripes,
while the RAID-1 does not have stripes.
Furthermore, the read operation on RAID-10 are
interleaved, delivering (for SSDs) double
sequential read speed (for 2 devices), while
the RAID-1 can handle two independent (one per
device) read stream, each with single device
reading speed.

Of course, depending on the requirements,
assuming what I wrote is correct, performances
might be different.

I was just wondering if anybody has some hints,
some experience, some references, or, as you
suggested, not to care at all.

Thanks again,

bye,

-- 

piergiorgio

Re: RAID-10 near vs. RAID-1

[Reindl Harald]

Am 12.06.24 um 19:22 schrieb Piergiorgio Sartor:
> On Wed, Jun 12, 2024 at 01:04:18AM +0200, Reindl Harald wrote:
>>
>>
>> Am 11.06.24 um 20:31 schrieb Piergiorgio Sartor:
>>> I'm setting up a system with 2 SSD M.2 (NVME).
>>>
>>> I was wondering if would it be better, performace
>>> wise, to have a RAID-10 near layout or a RAID-1.
>>>
>>> Looking around I found only one benchmark:
>>>
>>> https://strugglers.net/~andy/blog/2019/06/02/exploring-different-linux-raid-10-layouts-with-unbalanced-devices/
>>>
>>> Which uses mixed SSD, NVME and SATA.
>>>
>>> Does anybody have any suggestions, links, or
>>> ideas on the topic?
>>>
>>> BTW, practically speaking, what's the difference,
>>> between the two RAIDs?
>>
>> i wouldn't even consider a RAID10 with two disks, especially with SSD and
>> practically you end with a unsupported RAID1 because there are no stripes
>> with 2 disks
>>
> I'm a bit confused here. What do you mean
> with "unsupported RAID1"?

well, the normal operation mode of a RAID10 isn't only 2 drives

> As far as I know, but please correct me if
> I'm wrong, a Linux md RAID-10 *near* layout,
> with 2 devices, has identical data distribution
> as a RAID-1 with 2 devices.
> Meaning the 2 devices are a mirror.
> 
> The difference, if I understood it correctly,
> is that the RAID-10 has chunks, and hence stripes,
> while the RAID-1 does not have stripes.

how do you imagine stripes with only two drives?

stripes are "half of the file on disk 1, the other half on disk 2"
that's not possible with only 2 drives

> Furthermore, the read operation on RAID-10 are
> interleaved, delivering (for SSDs) double
> sequential read speed (for 2 devices), while
> the RAID-1 can handle two independent (one per
> device) read stream, each with single device
> reading speed.

but how should this work with only 2 drives?

Re: RAID-10 near vs. RAID-1

[Dragan Milivojević]

> stripes are "half of the file on disk 1, the other half on disk 2"
> that's not possible with only 2 drives
>

https://en.wikipedia.org/wiki/Non-standard_RAID_levels#Linux_MD_RAID_10

Re: RAID-10 near vs. RAID-1

[Reindl Harald]

Am 13.06.24 um 01:46 schrieb Dragan Milivojević:
>> stripes are "half of the file on disk 1, the other half on disk 2"
>> that's not possible with only 2 drives
> 
> https://en.wikipedia.org/wiki/Non-standard_RAID_levels#Linux_MD_RAID_10


"The two-drive example is equivalent to RAID 1"

what else - when you have only two drives
dunno what magic you expect performance wise

the only advantage is that you later can add 2 drives and make a real 
RAID10 out of it

Re: RAID-10 near vs. RAID-1

[Robin Hill]

On Thu Jun 13, 2024 at 07:38:34AM +0200, Reindl Harald wrote:

> 
> 
> Am 13.06.24 um 01:46 schrieb Dragan Milivojević:
> >> stripes are "half of the file on disk 1, the other half on disk 2"
> >> that's not possible with only 2 drives
> > 
> > https://en.wikipedia.org/wiki/Non-standard_RAID_levels#Linux_MD_RAID_10
> 
Far mode would give you striping, but not near mode.

> 
> "The two-drive example is equivalent to RAID 1"
> 
> what else - when you have only two drives
> dunno what magic you expect performance wise
> 
> the only advantage is that you later can add 2 drives and make a real 
> RAID10 out of it
> 

You'll have to rebuild though, as RAID10 does not support changing the
number of devices in an array.

Cheers,
    Robin
-- 
     ___        
    ( ' }     |       Robin Hill        <robin@robinhill.me.uk> |
   / / )      | Little Jim says ....                            |
  // !!       |      "He fallen in de water !!"                 |

Re: RAID-10 near vs. RAID-1

[Dragan Milivojević]

> the whole discussion is nonsense
>
> you won't find any difference between a !! NVME RAID !! with TWO disks
> which is worth even to open a discussion
>

Previous response got blocked, maybe because it was just a link. Let's
see if this works.
Summary:

| fio iodepth=256, numjobs=4                       |  IOPS |     BW
| lat (usec) avg |
|--------------------------------------------------|:-----:|:---------:|:--------------:|
| Sequential 4k read, single disk                  |  828k | 3233MiB/s
|           1236 |
| Sequential 4k read, 4 disk RAID0, 64k chunk      |  666k | 2602MiB/s
|           1536 |
| Sequential 512k read, single disk                | 13.6k | 6798MiB/s
|          75300 |
| Sequential 512k read, 4 disk RAID0, 64k chunk    | 47.1k |   23GiB/s
|          21745 |
| Sequential 4k read, 2 disk RAID10F2, 64k chunk   |  523k | 2044MiB/s
|           1956 |
| Sequential 512k read, 2 disk RAID10F2, 64k chunk | 27.2k | 13.3GiB/s
|          37675 |


full test log: https://pastebin.com/raw/eq2CbjY7

Re: RAID-10 near vs. RAID-1

[Reindl Harald]

Am 13.06.24 um 21:54 schrieb Dragan Milivojević:
>> the whole discussion is nonsense
>>
>> you won't find any difference between a !! NVME RAID !! with TWO disks
>> which is worth even to open a discussion
>>
> 
> Previous response got blocked, maybe because it was just a link. Let's
> see if this works.
> Summary:
> 
> | fio iodepth=256, numjobs=4                       |  IOPS |     BW
> | lat (usec) avg |
> |--------------------------------------------------|:-----:|:---------:|:--------------:|
> | Sequential 4k read, single disk                  |  828k | 3233MiB/s
> |           1236 |
> | Sequential 4k read, 4 disk RAID0, 64k chunk      |  666k | 2602MiB/s
> |           1536 |
> | Sequential 512k read, single disk                | 13.6k | 6798MiB/s
> |          75300 |
> | Sequential 512k read, 4 disk RAID0, 64k chunk    | 47.1k |   23GiB/s
> |          21745 |
> | Sequential 4k read, 2 disk RAID10F2, 64k chunk   |  523k | 2044MiB/s
> |           1956 |
> | Sequential 512k read, 2 disk RAID10F2, 64k chunk | 27.2k | 13.3GiB/s
> |          37675 |
> 
> 
> full test log: https://pastebin.com/raw/eq2CbjY7
not very appealing

Sequential 4k read, single disk                  |  828k | 3233MiB/s
Sequential 4k read, 2 disk RAID10F2, 64k chunk   |  523k | 2044MiB/s

RAID0 is off-topic when it comes to RAID1/RAID10 with two disk and not a 
RAID at all

Re: RAID-10 near vs. RAID-1

[Reindl Harald]

Am 13.06.24 um 22:27 schrieb Dragan Milivojević:
>>> full test log: https://pastebin.com/raw/eq2CbjY7
>> not very appealing
>>
>> Sequential 4k read, single disk                  |  828k | 3233MiB/s
>> Sequential 4k read, 2 disk RAID10F2, 64k chunk   |  523k | 2044MiB/s
>>
>> RAID0 is off-topic when it comes to RAID1/RAID10 with two disk and not a
>> RAID at all
> 
> There is a reason why I included the RAID0 results. It seems that you desire
> to argue for the sake of argument, so I will stop responding to your messages.

i am thrilled to hear the reason why you compare RADI0 with RAID10 but 
no RAID1 when the whole question is "do i gain anything with a cripppled 
RAID10 with two disks versus a ordinary RAID1"

i could understand the reason to *additionally* add RAID0 while 
everybody knows it's faster but the opposite of redundancy

coming up with "RAID10 versus RAID0" is idiotic

Re: RAID-10 near vs. RAID-1

[Phillip Susi]

Piergiorgio Sartor <piergiorgio.sartor@nexgo.de> writes:

> As far as I know, but please correct me if
> I'm wrong, a Linux md RAID-10 *near* layout,
> with 2 devices, has identical data distribution
> as a RAID-1 with 2 devices.
> Meaning the 2 devices are a mirror.

That's correct.

> The difference, if I understood it correctly,
> is that the RAID-10 has chunks, and hence stripes,
> while the RAID-1 does not have stripes.
> Furthermore, the read operation on RAID-10 are
> interleaved, delivering (for SSDs) double
> sequential read speed (for 2 devices), while
> the RAID-1 can handle two independent (one per
> device) read stream, each with single device
> reading speed.

No, since the layout is exactly the same as raid1, large sequential
reads can not be sent to both drives at the same time.

Now a two disk raid-10 in the offset or far layout however, does
interleave the data across the drives so they can both be read at the
same time to increase throughput.