RAID1 vs RAID10 and best way to set up 6 disks

RAID1 vs RAID10 and best way to set up 6 disks

[Mitchell Fossen]

Hello,

I have 6 WD Red Pro drives, each 6TB in space. My question is, what is
the best way to set these up? 

The system drive (and root) are on a 500GB SSD, so these drives will
only be used for /home and file storage.

Is there any caveats between RAID1 on all 6 vs RAID10?

Thanks for the help,

Mitch

Re: RAID1 vs RAID10 and best way to set up 6 disks

[Christoph Anton Mitterer]

[-- Attachment #1: Type: text/plain, Size: 323 bytes --]

On Fri, 2016-06-03 at 13:10 -0500, Mitchell Fossen wrote:
> Is there any caveats between RAID1 on all 6 vs RAID10?

Just to be safe: RAID1 in btrfs means not what RAID1 means in any other
terminology about RAID.

The former has only two duplicates, the later means full mirroring of
all devices.


Cheers,
Chris.

[-- Attachment #2: smime.p7s --]
[-- Type: application/x-pkcs7-signature, Size: 5930 bytes --]

Re: RAID1 vs RAID10 and best way to set up 6 disks

[Mitchell Fossen]

Thanks for pointing that out, so if I'm thinking correctly, with RAID1
it's just that there is a copy of the data somewhere on some other
drive.

With RAID10, there's still only 1 other copy, but the entire "original"
disk is mirrored to another one, right?

On Fri, 2016-06-03 at 20:13 +0200, Christoph Anton Mitterer wrote:
> On Fri, 2016-06-03 at 13:10 -0500, Mitchell Fossen wrote:
> > 
> > Is there any caveats between RAID1 on all 6 vs RAID10?
> Just to be safe: RAID1 in btrfs means not what RAID1 means in any
> other
> terminology about RAID.
> 
> The former has only two duplicates, the later means full mirroring of
> all devices.
> 
> 
> Cheers,
> Chris.

Re: RAID1 vs RAID10 and best way to set up 6 disks

[Christoph Anton Mitterer]

[-- Attachment #1: Type: text/plain, Size: 903 bytes --]

On Fri, 2016-06-03 at 13:42 -0500, Mitchell Fossen wrote:
> Thanks for pointing that out, so if I'm thinking correctly, with
> RAID1
> it's just that there is a copy of the data somewhere on some other
> drive.
> 
> With RAID10, there's still only 1 other copy, but the entire
> "original"
> disk is mirrored to another one, right?

To be honest, I couldn't tell you for sure :-/ ... IMHO the btrfs
documentation has some "issues".

mkfs.btrfs(8) says: 2 copies for RAID10, so I'd assume it's just the
striped version of what btrfs - for whichever questionable reason -
calls "RAID1".

Especially, when you have an odd number devices (or devices with
different sizes), its not clear to me, personally, at all how far that
redundancy actually goes respectively what btrfs actually does... could
be that you have your 2 copies, but maybe on the same device then?


Cheers,
Chris.

[-- Attachment #2: smime.p7s --]
[-- Type: application/x-pkcs7-signature, Size: 5930 bytes --]

Re: RAID1 vs RAID10 and best way to set up 6 disks

[Brendan Hide]

On 06/03/16 20:59, Christoph Anton Mitterer wrote:
> On Fri, 2016-06-03 at 13:42 -0500, Mitchell Fossen wrote:
>> Thanks for pointing that out, so if I'm thinking correctly, with
>> RAID1
>> it's just that there is a copy of the data somewhere on some other
>> drive.
>>
>> With RAID10, there's still only 1 other copy, but the entire
>> "original"
>> disk is mirrored to another one, right?
As Justin mentioned, btrfs doesn't raid whole disks/devices. Instead, it 
works with chunks.

>
> To be honest, I couldn't tell you for sure :-/ ... IMHO the btrfs
> documentation has some "issues".
>
> mkfs.btrfs(8) says: 2 copies for RAID10, so I'd assume it's just the
> striped version of what btrfs - for whichever questionable reason -
> calls "RAID1".

The "questionable reason" is simply the fact that it is, now as well as 
at the time the features were added, the closest existing terminology 
that best describes what it does. Even now, it would be difficult on the 
spot adequately to explain what it means for redundancy without also 
mentioning "RAID".

Btrfs does not raid disks/devices. It works with chunks that are 
allocated to devices when the previous chunk/chunk-set is full.

We're all very aware of the inherent problem of language - and have 
discussed various ways to address it. You will find that some on the 
list (but not everyone) are very careful to never call it "RAID" - but 
instead raid (very small difference, I know). Hugo Mills previously made 
headway in getting discussion and consensus of proper nomenclature. *

> Especially, when you have an odd number devices (or devices with
> different sizes), its not clear to me, personally, at all how far that
> redundancy actually goes respectively what btrfs actually does... could
> be that you have your 2 copies, but maybe on the same device then?

No, btrfs' raid1 naively guarantees that the two copies will *never* be 
on the same device. raid10 does the same thing - but in stripes on as 
many devices as possible.

The reason I say "naively" is that there is little to stop you from 
creating a 2-device "raid1" using two partitions on the same physical 
device. This is especially difficult to detect if you add abstraction 
layers (lvm, dm-crypt, etc). This same problem does apply to mdadm however.

Though it won't necessarily answer all questions about allocation, I 
strongly suggest checking out Hugo's btrfs calculator **

I hope this is helpful.

* http://comments.gmane.org/gmane.comp.file-systems.btrfs/34717 / 
https://www.spinics.net/lists/linux-btrfs/msg33742.html
* http://comments.gmane.org/gmane.comp.file-systems.btrfs/34792
** http://carfax.org.uk/btrfs-usage/

>
>
> Cheers,
> Chris.
>

-- 
__________
Brendan Hide
http://swiftspirit.co.za/
http://www.webafrica.co.za/?AFF1E97

Re: RAID1 vs RAID10 and best way to set up 6 disks

[Christoph Anton Mitterer]

[-- Attachment #1: Type: text/plain, Size: 3880 bytes --]

On Sun, 2016-06-05 at 02:41 +0200, Brendan Hide wrote:
> The "questionable reason" is simply the fact that it is, now as well
> as 
> at the time the features were added, the closest existing
> terminology 
> that best describes what it does. Even now, it would be difficult on
> the 
> spot adequately to explain what it means for redundancy without also 
> mentioning "RAID".
Well the RAID1 was IMHO still bad choice as it's pretty ambiguous.

A better choice would have been something simple like rep2
(rep=replicas), mirror2, or dup with either adding some additional
string that it's guaranteed to be on different devices here, or one
that it's not guaranteed on what's currently "DUP".

But DUP(licate) seems anyway a little bit "restricted". It's not so
unlikely that some people want a level that has always exactly three
copies, or one with for.
So the repN / replicaN seems good to me.

Since the standard behaviour should be to enforce replicas being on
different devices I'd have said, one could have made analogous levels
named e.g. "same-device-repN" (or something like that just better),
with same-device-rep2, being what our current DUP is


> Btrfs does not raid disks/devices. It works with chunks that are 
> allocated to devices when the previous chunk/chunk-set is full.
Sure, but effectively this is quite close.
And whether it works on whole device level or chunk level doesn't
change that it's pretty important to be able to have the guarantee that
the different replicas are actually on different devices.

> 
> We're all very aware of the inherent problem of language - and have 
> discussed various ways to address it. You will find that some on the 
> list (but not everyone) are very careful to never call it "RAID" -
> but 
> instead raid (very small difference, I know).

Really very very small... to non-existent. ;)


>  Hugo Mills previously made 
> headway in getting discussion and consensus of proper nomenclature. *

Well I'd say, for btrfs: do away with the term "RAID" at all, use e.g.:

linear = just a bunch of devices put together, no striping
         basically what MD's linear is
mirror (or perhaps something like clones) = each device in the fs
                                            contains a copy of
                                            everything (i.e. classic
                                            RAID1)
striped = basically what RAID0 is
replicaN = N replicas of each chunk on distinct devices
<whatever>-replicaN = N replicas of each chunk NOT necessarily on
                      distinct devices
parityN = n parity chunks i.e. parity1 ~= RAID5, parity2 ~= RAID6
or perhaps better: striped-parityN or striped+parityN ??

And just mention in the manpage, which of these names comes closest to
what people understand by RAID level i.


> 
> The reason I say "naively" is that there is little to stop you from 
> creating a 2-device "raid1" using two partitions on the same
> physical 
> device. This is especially difficult to detect if you add
> abstraction 
> layers (lvm, dm-crypt, etc). This same problem does apply to mdadm
> however.
Sure... I think software should try to prevent people from doing stupid
things, but not by all means ;-)
If one makes n partitions on the same device an puts a RAID on that,
one probably doesn't deserve it any better ;-)

I'd guess it's probably doable to detect such stupidness for e.g.
partitions and dm-crypt (because these are linearly on one device)...
but for lvm/MD it really depends on the actual block allocation/layout,
whether it's safe or not.
Maybe the tools could detect *if* lvm/MD is in between and just give a
general warning what that could mean.


Best wishes,
Chris.

[-- Attachment #2: smime.p7s --]
[-- Type: application/x-pkcs7-signature, Size: 5930 bytes --]

Re: RAID1 vs RAID10 and best way to set up 6 disks

[Chris Murphy]

On Sat, Jun 4, 2016 at 7:10 PM, Christoph Anton Mitterer
<calestyo@scientia.net> wrote:

> Well the RAID1 was IMHO still bad choice as it's pretty ambiguous.

That's ridiculous. It isn't incorrect to refer to only 2 copies as
raid1. You have to explicitly ask both mdadm and lvcreate for the
number of copies you want, it doesn't automatically happen. The man
page for mkfs.btrfs is very clear you only get two copies.

What's ambiguous is raid10 expectations with multiple device failures.


> Well I'd say, for btrfs: do away with the term "RAID" at all, use e.g.:
>
> linear = just a bunch of devices put together, no striping
>          basically what MD's linear is

Except this isn't really how Btrfs single works. The difference
between mdadm linear and Btrfs single is more different in behavior
than the difference between mdadm raid1 and btrfs raid1. So you're
proposing tolerating a bigger difference, while criticizing a smaller
one. *shrug*



> mirror (or perhaps something like clones) = each device in the fs
>                                             contains a copy of
>                                             everything (i.e. classic
>                                             RAID1)


If a metaphor is going to be used for a technical thing, it would be
mirrors or mirroring. Mirror would mean exactly two (the original and
the mirror). See lvcreate --mirrors. Also, the lvm mirror segment type
is legacy, having been replaced with raid1 (man lvcreate uses the term
raid1, not RAID1 or RAID-1). So I'm not a big fan of this term.


> striped = basically what RAID0 is

lvcreate uses only striped, not raid0. mdadm uses only RAID0, not
striped. Since striping is also employed with RAIDs 4, 5, 6, 7, it
seems ambiguous even though without further qualification whether
parity exists, it's considered to mean non-parity striping. The
ambiguity is probably less of a problem than the contradiction that is
RAID0.



> replicaN = N replicas of each chunk on distinct devices
> <whatever>-replicaN = N replicas of each chunk NOT necessarily on
>                       distinct devices

This is kinda interesting. At least it's a new term so all the new
rules can be stuffed into that new term and helps distinguish it from
other implementations, not entirely different with how ZFS does this
with their raidz.



> parityN = n parity chunks i.e. parity1 ~= RAID5, parity2 ~= RAID6
> or perhaps better: striped-parityN or striped+parityN ??

It's not easy, is it?


>
> And just mention in the manpage, which of these names comes closest to
> what people understand by RAID level i.

It already does this. What version of btrfs-progs are you basing your
criticism on that there's some inconsistency, deficiency, or ambiguity
when it comes to these raid levels? The one that's unequivocally
problematic alone without reading the man page is raid10. The historic
understanding is that it's a stripe of mirrors, and this suggests you
can lose a mirror of each stripe i.e. multiple disks and not lose
data, which is not true for Btrfs raid10. But the man page makes that
clear, you have 2 copies for redundancy, that's it.





>
>
>>
>> The reason I say "naively" is that there is little to stop you from
>> creating a 2-device "raid1" using two partitions on the same
>> physical
>> device. This is especially difficult to detect if you add
>> abstraction
>> layers (lvm, dm-crypt, etc). This same problem does apply to mdadm
>> however.
> Sure... I think software should try to prevent people from doing stupid
> things, but not by all means ;-)
> If one makes n partitions on the same device an puts a RAID on that,
> one probably doesn't deserve it any better ;-)
>
> I'd guess it's probably doable to detect such stupidness for e.g.
> partitions and dm-crypt (because these are linearly on one device)...
> but for lvm/MD it really depends on the actual block allocation/layout,
> whether it's safe or not.
> Maybe the tools could detect *if* lvm/MD is in between and just give a
> general warning what that could mean.

On the CLI? Not worth it. If the user is that ignorant, too bad, use a
GUI program to help build the storage stack from scratch. I'm really
not sympathetic if a user creates a raid1 from two partitions of the
same block device anymore than if it's ultimately the same physical
device managed by a device mapper variant.

Anyway, I think there's a whole separate github discussion on Btrfs
UI/Ux that presumably also includes terminology concerns like this.


-- 
Chris Murphy

Re: RAID1 vs RAID10 and best way to set up 6 disks

[Christoph Anton Mitterer]

[-- Attachment #1: Type: text/plain, Size: 5033 bytes --]

On Sun, 2016-06-05 at 09:36 -0600, Chris Murphy wrote:
> That's ridiculous. It isn't incorrect to refer to only 2 copies as
> raid1.
No, if there are only two devices then not.
But obviously we're talking about how btrfs does RAID1, in which even
with n>2 devices there are only 2 copies - that's incorrect.


>  You have to explicitly ask both mdadm
Aha, and which option would that be?

>  and lvcreate for the
> number of copies you want, it doesn't automatically happen.
I've said that before, but at least it allows you to use the full
number of disks, so we're again back to that it's closer to the
original and common meaning of RAID1 than what btrfs does.


>  The man
> page for mkfs.btrfs is very clear you only get two copies.

I haven't denied that... but one shouldn't use terms that are commonly
understood in a different mannor and require people to read all the
small printed.
One could also have changed it's RAID0 with RAID1, and I guess people
wouldn't be too delighted if the excuse was "well it's in the manpage".


> 
> > Well I'd say, for btrfs: do away with the term "RAID" at all, use
> > e.g.:
> > 
> > linear = just a bunch of devices put together, no striping
> >          basically what MD's linear is
> Except this isn't really how Btrfs single works. The difference
> between mdadm linear and Btrfs single is more different in behavior
> than the difference between mdadm raid1 and btrfs raid1. So you're
> proposing tolerating a bigger difference, while criticizing a smaller
> one. *shrug*

What's the big difference? Would you care to explain? But I'm happy
with "single" either, it just doesn't really tell that there is no
striping, I mean "single" points more towards "we have no resilience
but only 1 copy", whether this is striped or not.



> If a metaphor is going to be used for a technical thing, it would be
> mirrors or mirroring. Mirror would mean exactly two (the original and
> the mirror). See lvcreate --mirrors. Also, the lvm mirror segment
> type
> is legacy, having been replaced with raid1 (man lvcreate uses the
> term
> raid1, not RAID1 or RAID-1). So I'm not a big fan of this term.

Admittedly, I didn't like the "mirror(s)" either... I was just trying
to show that different names could be used that are already a bit
better.


> > striped = basically what RAID0 is
> 
> lvcreate uses only striped, not raid0. mdadm uses only RAID0, not
> striped. Since striping is also employed with RAIDs 4, 5, 6, 7, it
> seems ambiguous even though without further qualification whether
> parity exists, it's considered to mean non-parity striping. The
> ambiguity is probably less of a problem than the contradiction that
> is
> RAID0.

Mhh,.. well or one makes schema names that contain all possible
properties of a "RAID", something like:
replicasN-parityN-[not]striped

SINGLE would be something like "replicas1-parity0-notstriped".
RAID5 would be something like "replicas0-parity1-striped".


> > And just mention in the manpage, which of these names comes closest
> > to
> > what people understand by RAID level i.
> 
> It already does this. What version of btrfs-progs are you basing your
> criticism on that there's some inconsistency, deficiency, or
> ambiguity
> when it comes to these raid levels?

Well first, the terminology thing is the least serious issue from my
original list ;-) ... TBH I don't know why such a large discussion came
out of that point.

Even though I'm not reading along all mails here, we have probably at
least every month someone who wasn't aware that RAID1 is not what he
assumes it to be.
And I don't think these people can be blamed for not RTFM, because IMHO
this is a term commonly understood as mirror all available devices.
That's how the original paper describes it, it's how Wikipedia
describes it and all other sources I've ever read to the topic.


>  The one that's unequivocally
> problematic alone without reading the man page is raid10. The
> historic
> understanding is that it's a stripe of mirrors, and this suggests you
> can lose a mirror of each stripe i.e. multiple disks and not lose
> data, which is not true for Btrfs raid10. But the man page makes that
> clear, you have 2 copies for redundancy, that's it.
Yes, same basic problem.


> On the CLI? Not worth it. If the user is that ignorant, too bad, use
> a
> GUI program to help build the storage stack from scratch. I'm really
> not sympathetic if a user creates a raid1 from two partitions of the
> same block device anymore than if it's ultimately the same physical
> device managed by a device mapper variant.

Well one I have no strong opinion on that... if testing for it (or at
least simple cases) would be easy, why not.
Not every situation may be as easily visible as creating a RAID1 on
/dev/sda1 and /dev/sda2.
One may use LABELs, or UUIDs and accidentally catch the wrong, and in
such cases a check may help.


Cheers,
Chris.

[-- Attachment #2: smime.p7s --]
[-- Type: application/x-pkcs7-signature, Size: 5930 bytes --]

Re: RAID1 vs RAID10 and best way to set up 6 disks

[Chris Murphy]

On Sun, Jun 5, 2016 at 2:31 PM, Christoph Anton Mitterer
<calestyo@scientia.net> wrote:
> On Sun, 2016-06-05 at 09:36 -0600, Chris Murphy wrote:
>> That's ridiculous. It isn't incorrect to refer to only 2 copies as
>> raid1.
> No, if there are only two devices then not.
> But obviously we're talking about how btrfs does RAID1, in which even
> with n>2 devices there are only 2 copies - that's incorrect.

OK and I think the assertion is asinine. You reject the only neutral
party's definition and distinction of RAID-1 types, and then claim on
the basis of opinion that Btrf's raid1 is not merely different from
traditional/classic/common understandings of RAID-1, but that they're
incorrect to have called it raid1. It's just nonsense. I find your
argument uncompellling.


>>  You have to explicitly ask both mdadm
> Aha, and which option would that be?

For mdadm it's implied as a combination of -n -x and number of
devices. For lvcreate it's explicit with -m. This is in the man page,
so I don't understand why you're asking.


>
>>  and lvcreate for the
>> number of copies you want, it doesn't automatically happen.
> I've said that before, but at least it allows you to use the full
> number of disks, so we're again back to that it's closer to the
> original and common meaning of RAID1 than what btrfs does.

The original and common meaning defined by whom, where? You're welcome
to go take it up with Wikipedia but they're using SNIA definitions for
the standard RAID levels.




>
>
>>  The man
>> page for mkfs.btrfs is very clear you only get two copies.
>
> I haven't denied that... but one shouldn't use terms that are commonly
> understood in a different mannor and require people to read all the
> small printed.

And I disagree because what you required is more reading by the user
to understand entirely new nomenclature.



> One could also have changed it's RAID0 with RAID1, and I guess people
> wouldn't be too delighted if the excuse was "well it's in the manpage".

Except nothing that crazy has been done so I fail to see the point.



>
>
>>
>> > Well I'd say, for btrfs: do away with the term "RAID" at all, use
>> > e.g.:
>> >
>> > linear = just a bunch of devices put together, no striping
>> >          basically what MD's linear is
>> Except this isn't really how Btrfs single works. The difference
>> between mdadm linear and Btrfs single is more different in behavior
>> than the difference between mdadm raid1 and btrfs raid1. So you're
>> proposing tolerating a bigger difference, while criticizing a smaller
>> one. *shrug*
>
> What's the big difference? Would you care to explain?

It's not linear. The archives detail how block groups are allocated to
devices. There are rules, linearity isn't one of them.



> But I'm happy
> with "single" either, it just doesn't really tell that there is no
> striping, I mean "single" points more towards "we have no resilience
> but only 1 copy", whether this is striped or not.
>
>
>
>> If a metaphor is going to be used for a technical thing, it would be
>> mirrors or mirroring. Mirror would mean exactly two (the original and
>> the mirror). See lvcreate --mirrors. Also, the lvm mirror segment
>> type
>> is legacy, having been replaced with raid1 (man lvcreate uses the
>> term
>> raid1, not RAID1 or RAID-1). So I'm not a big fan of this term.
>
> Admittedly, I didn't like the "mirror(s)" either... I was just trying
> to show that different names could be used that are already a bit
> better.
>
>
>> > striped = basically what RAID0 is
>>
>> lvcreate uses only striped, not raid0. mdadm uses only RAID0, not
>> striped. Since striping is also employed with RAIDs 4, 5, 6, 7, it
>> seems ambiguous even though without further qualification whether
>> parity exists, it's considered to mean non-parity striping. The
>> ambiguity is probably less of a problem than the contradiction that
>> is
>> RAID0.
>
> Mhh,.. well or one makes schema names that contain all possible
> properties of a "RAID", something like:
> replicasN-parityN-[not]striped



SNIA has created such a schema.




>
> SINGLE would be something like "replicas1-parity0-notstriped".
> RAID5 would be something like "replicas0-parity1-striped".
>
>
>> > And just mention in the manpage, which of these names comes closest
>> > to
>> > what people understand by RAID level i.
>>
>> It already does this. What version of btrfs-progs are you basing your
>> criticism on that there's some inconsistency, deficiency, or
>> ambiguity
>> when it comes to these raid levels?
>
> Well first, the terminology thing is the least serious issue from my
> original list ;-) ... TBH I don't know why such a large discussion came
> out of that point.
>
> Even though I'm not reading along all mails here, we have probably at
> least every month someone who wasn't aware that RAID1 is not what he
> assumes it to be.
> And I don't think these people can be blamed for not RTFM, because IMHO
> this is a term commonly understood as mirror all available devices.


You have the wrong understanding. The most common is exactly two disk RAID-1.


> That's how the original paper describes it, it's how Wikipedia
> describes it and all other sources I've ever read to the topic.

Wikipedia:

"RAID 1 consists of an exact copy (or mirror) of a set of data on two
or more disks; a classic RAID 1 mirrored pair contains two disks. "

An exact copy = one copy. It does not say or imply n-way.




-- 
Chris Murphy

Re: RAID1 vs RAID10 and best way to set up 6 disks

[Austin S. Hemmelgarn]

On 2016-06-05 16:31, Christoph Anton Mitterer wrote:
> On Sun, 2016-06-05 at 09:36 -0600, Chris Murphy wrote:
>> That's ridiculous. It isn't incorrect to refer to only 2 copies as
>> raid1.
> No, if there are only two devices then not.
> But obviously we're talking about how btrfs does RAID1, in which even
> with n>2 devices there are only 2 copies - that's incorrect.
Go read the original standards that defined the term RAID (assuming you 
can find a openly accessible copy), it defines RAID1 as a mirrored 
_pair_ of disks.  This is how every hardware RAID controller I've ever 
seen implements RAID1, and in fact how most software RAID 
implementations (including the fake raid in some motherboards) 
implements it with the sole exception of Linux's MD-RAID and it's direct 
derivatives (which includes LVM/DM based RAID, as well as BSD's GEOM 
framework).
>
>
>>  You have to explicitly ask both mdadm
> Aha, and which option would that be?
Specifying more than two disks.  The request is more correctly an 
implicit one, but the fact that it's implied by a now largely obsolete 
piece of software does not mean that BTRFS should have the same 
implications.
>
>>  and lvcreate for the
>> number of copies you want, it doesn't automatically happen.
> I've said that before, but at least it allows you to use the full
> number of disks, so we're again back to that it's closer to the
> original and common meaning of RAID1 than what btrfs does.
/me inserts reflink to the first part of my reply.
>
>
>>  The man
>> page for mkfs.btrfs is very clear you only get two copies.
>
> I haven't denied that... but one shouldn't use terms that are commonly
> understood in a different mannor and require people to read all the
> small printed.
> One could also have changed it's RAID0 with RAID1, and I guess people
> wouldn't be too delighted if the excuse was "well it's in the manpage".
You can leave the hyperbolic theoreticals out of this, they really do 
detract from your argument.
>
>
>>
>>> Well I'd say, for btrfs: do away with the term "RAID" at all, use
>>> e.g.:
>>>
>>> linear = just a bunch of devices put together, no striping
>>>          basically what MD's linear is
>> Except this isn't really how Btrfs single works. The difference
>> between mdadm linear and Btrfs single is more different in behavior
>> than the difference between mdadm raid1 and btrfs raid1. So you're
>> proposing tolerating a bigger difference, while criticizing a smaller
>> one. *shrug*
>
> What's the big difference? Would you care to explain? But I'm happy
> with "single" either, it just doesn't really tell that there is no
> striping, I mean "single" points more towards "we have no resilience
> but only 1 copy", whether this is striped or not.
On this point I actually do kind of agree with you, but Chris is also 
correct here, BTRFS single mode is just as different from MD linear mode 
as BTRFS raid1 is from MD RAID1, if not more so.
>
>
>
>> If a metaphor is going to be used for a technical thing, it would be
>> mirrors or mirroring. Mirror would mean exactly two (the original and
>> the mirror). See lvcreate --mirrors. Also, the lvm mirror segment
>> type
>> is legacy, having been replaced with raid1 (man lvcreate uses the
>> term
>> raid1, not RAID1 or RAID-1). So I'm not a big fan of this term.
>
> Admittedly, I didn't like the "mirror(s)" either... I was just trying
> to show that different names could be used that are already a bit
> better.
>
>
>>> striped = basically what RAID0 is
>>
>> lvcreate uses only striped, not raid0. mdadm uses only RAID0, not
>> striped. Since striping is also employed with RAIDs 4, 5, 6, 7, it
>> seems ambiguous even though without further qualification whether
>> parity exists, it's considered to mean non-parity striping. The
>> ambiguity is probably less of a problem than the contradiction that
>> is
>> RAID0.
>
> Mhh,.. well or one makes schema names that contain all possible
> properties of a "RAID", something like:
> replicasN-parityN-[not]striped
>
> SINGLE would be something like "replicas1-parity0-notstriped".
> RAID5 would be something like "replicas0-parity1-striped".
It's worth pointing out that both programmers and sysadmins are still 
lazy typists, so it would more likely end up being:
rep1-par0-strip0
rep0-par1-stripN (with N being the number of desired stripes).

Having a number to indicate the striping is actually useful (there are 
legitimate cases for not striping across everything we can, and we need 
some way to represent stripes that weren't allocated at full width for 
some reason).

Such a scheme was actually proposed back when the higher order parity 
patches were being discussed.  Like those patches, it was decided to 
wait until we had basic feature completeness before trying to tackle that.
>
>
>>> And just mention in the manpage, which of these names comes closest
>>> to
>>> what people understand by RAID level i.
>>
>> It already does this. What version of btrfs-progs are you basing your
>> criticism on that there's some inconsistency, deficiency, or
>> ambiguity
>> when it comes to these raid levels?
>
> Well first, the terminology thing is the least serious issue from my
> original list ;-) ... TBH I don't know why such a large discussion came
> out of that point.
>
> Even though I'm not reading along all mails here, we have probably at
> least every month someone who wasn't aware that RAID1 is not what he
> assumes it to be.
And it's also almost always someone who didn't properly read the 
documentation, which means they would have gotten bitten by something 
else eventually as well.  This expectation that anyone should be able to 
pick up any piece of software and immediately use it without reading 
documentation needs to stop.  Catering to such people is not a 
sustainable design choice for a piece of software as complex as a 
filesystem.
> And I don't think these people can be blamed for not RTFM, because IMHO
> this is a term commonly understood as mirror all available devices.
> That's how the original paper describes it, it's how Wikipedia
> describes it and all other sources I've ever read to the topic.
The first sentence of 
https://en.wikipedia.org/wiki/Standard_RAID_levels#RAID_1 as of right 
now is exactly:
RAID 1 consists of an exact copy (or mirror) of a set of data on two or 
more disks; a classic RAID 1 mirrored pair contains two disks.

The 'classic RAID 1 mirrored pair' it refers to describes every RAID-1 
implementation I or anyone I've asked (including people of the pre-UNIX 
vintage who still deal with computers) has ever seen with the sole 
exception of the software implementations in Linux and BSD.
>
>>  The one that's unequivocally
>> problematic alone without reading the man page is raid10. The
>> historic
>> understanding is that it's a stripe of mirrors, and this suggests you
>> can lose a mirror of each stripe i.e. multiple disks and not lose
>> data, which is not true for Btrfs raid10. But the man page makes that
>> clear, you have 2 copies for redundancy, that's it.
> Yes, same basic problem.
>
>
>> On the CLI? Not worth it. If the user is that ignorant, too bad, use
>> a
>> GUI program to help build the storage stack from scratch. I'm really
>> not sympathetic if a user creates a raid1 from two partitions of the
>> same block device anymore than if it's ultimately the same physical
>> device managed by a device mapper variant.
>
> Well one I have no strong opinion on that... if testing for it (or at
> least simple cases) would be easy, why not.
> Not every situation may be as easily visible as creating a RAID1 on
> /dev/sda1 and /dev/sda2.
> One may use LABELs, or UUIDs and accidentally catch the wrong, and in
> such cases a check may help.
To paraphrase something from the Debian IRC:
'Our job is to make sure that if the user points a gun at his foot and 
pulls the trigger, the bullet gets to the intended location'

There are limits to what we can protect against, and we shouldn't be 
preventing people from doing things that while they seem unreasonable at 
face value, are perfectly legitimate uses of the software.  BTRFS raid1 
mode on a single device falls solidly into this category for a couple of 
reasons:
1. Until recently, this was the only way without mixed mode to get data 
chunk redundancy on a single device.  In many distros, this is still the 
only way to do so.
2. Because of how it's implemented, this actually doesn't get insanely 
horrible performance like trying to do the same thing with MD or LVM 
based RAID would.
3. This allows for forcing specific organization of data on the media, 
which can combined with item 1, be particularly useful (for example, 
some SSD's have SLC flash backing the area up through where the MFT 
would be on NTFS, and MLC for the rest, or some 'hybrid' disks actually 
just have flash for the first part of the logical device they expose, 
and traditional magnetic storage for the rest).

Re: RAID1 vs RAID10 and best way to set up 6 disks

[Justin Brown]

> Mitchell wrote:
> With RAID10, there's still only 1 other copy, but the entire "original"
disk is mirrored to another one, right?

No, full disks are never mirrored in any configuration.

Here's how I understand Btrfs' non-parity redundancy profiles:

single: only a single instance of a file exists across the file system
dup: two instances of a file exist across the file system, and they
may reside on the same physical disk (4.5.1+ required to use dup
profile on multi-disk file system)
raid1: same as dup but the instances are guaranteed to be on different disks
raid0: single but  can be striped between multiple disks
raid10: data is guaranteed to exist on two separate devices but if n>2
the data is load balanced between disks*

Even though my explanation is imperfect, I hopes that illustrates that
Btrfs RAID is different than traditional RAID. Btrfs provides the same
physical redundancy as RAID, but the implementation mechanisms are
quite a bit different. This has wonderful consequences for
flexibility, and it's what allowed me to run a 5x2TB RAID10 array for
nearly two years and essentially allow complete allocation. The
downside is that since allocations aren't enforced from start (eg. MD
requiring certain number of disks and identical sizes), it's possible
to get weird allocations over time, but the resolution is simple: run
a balance from time to time.

> Christoph wrote:
> Especially, when you have an odd number devices (or devices with
different sizes), its not clear to me, personally, at all how far that
redundancy actually goes respectively what btrfs actually does... could
be that you have your 2 copies, but maybe on the same device then?

RAID1 (and transitively RAID10) guarantees two copies on different
disks, always. Only dup allows the copies to reside on the same disk.
This is guaranteed is preserved, even when n=2k+1 and mixed-capacity
disks. If disks run out of available chunks to satisfy the redundancy
profile, the result is ENOSPC and requires the administrator to
balance the file system before new allocations can succeed. The
question essentially is asking if Btrfs will spontaneously degrade
into "dup" if chunks cannot be allocated on some devices. That will
never happen.


On Fri, Jun 3, 2016 at 1:42 PM, Mitchell Fossen <msfossen@gmail.com> wrote:
> Thanks for pointing that out, so if I'm thinking correctly, with RAID1
> it's just that there is a copy of the data somewhere on some other
> drive.
>
> With RAID10, there's still only 1 other copy, but the entire "original"
> disk is mirrored to another one, right?
>
> On Fri, 2016-06-03 at 20:13 +0200, Christoph Anton Mitterer wrote:
>> On Fri, 2016-06-03 at 13:10 -0500, Mitchell Fossen wrote:
>> >
>> > Is there any caveats between RAID1 on all 6 vs RAID10?
>> Just to be safe: RAID1 in btrfs means not what RAID1 means in any
>> other
>> terminology about RAID.
>>
>> The former has only two duplicates, the later means full mirroring of
>> all devices.
>>
>>
>> Cheers,
>> Chris.
> --
> To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html