pro/cons of raid1 with mdadm/lvm2

pro/cons of raid1 with mdadm/lvm2

[Gour]

Hello,

recently I was migrating from Debian to openSUSE and wanted to make it
smooth by dismantling my 2x1TB raid-1 array, install SUSE on the 1st
disk, cp /home data, check everything is OK and then add 2nd disk
containing Debian into raid-1 array.

However, in orde to accomplish it I learnt that one cannot simply
degrade mount and use such array like with mdadm, but I had to convert
system with -dconvert=single -mconvert=single which takes some time.

That's why I'm considering to put all my partitions (swap, root with
several subvolumes, home) under LVM2 volumes and then create raid-1
array with mdadm since that would enable to me more easy and quickly
temporarily dismantle raid-1 array, do some data manipulation from one
disk to another (sometimes I use 2nd disk as temporatily storage when
restoring some archived data from tapes etc.) and then resilver raid-1
array.

However, I wonder if there are some 'cons' in having raid-1 partition
under mdadm and not using native mirroring capabilities of btrfs fs?

Let me add that I also want to take advantage of using SUSE's snapshots
features, but I hope that's not the obstacle for the above-mentioned
layout - I'd still use btrfs' snapshot facility.


Sincerely,
Gour

-- 
Whenever and wherever there is a decline in religious practice, 
O descendant of Bharata, and a predominant rise of irreligion — 
at that time I descend Myself.

Re: pro/cons of raid1 with mdadm/lvm2

[Russell Coker]

When the 2 disks have different data mdadm has no way of knowing which one is correct and has a 50% chance of overwriting good data. But BTRFS does checksums on all reads and solves the problem of corrupt data - as long as you don't have 2 corrupt sectors in matching blocks.
-- 
Sent from my Samsung Galaxy Note 3 with K-9 Mail.

Re: pro/cons of raid1 with mdadm/lvm2

[Chris Murphy]

On Sun, Nov 30, 2014 at 3:06 PM, Russell Coker <russell@coker.com.au> wrote:
> When the 2 disks have different data mdadm has no way of knowing which one is correct and has a 50% chance of overwriting good data. But BTRFS does checksums on all reads and solves the problem of corrupt data - as long as you don't have 2 corrupt sectors in matching blocks.

Yeah. I'm not sure though if openSUSE 13.2 prevents users from
creating btrfs raid1 volumes entirely, or if it's just an install time
limitation.

I know that Fedora's installer won't allow the user to create Btrfs on
LVM, and it probably doesn't allow it on md raid either.

-- 
Chris Murphy

Re: pro/cons of raid1 with mdadm/lvm2

[Roman Mamedov]

On Sun, 30 Nov 2014 12:11:47 +0100
Gour <gour@atmarama.net> wrote:

> However, I wonder if there are some 'cons' in having raid-1 partition
> under mdadm and not using native mirroring capabilities of btrfs fs?

Pros:

  * mdadm RAID has much better read balancing;
    Btrfs reads are satisfied from what's in effect a random drive (PID-based
    balancing of threads to drives), mdadm reads from the less-loaded drive.
    Also mdadm has a way to specify some RAID1 array members as to be never
    used for reads if at all possible ("write-mostly"), which helps in RAID1 of
    HDD and SSD.

  * mdadm RAID has much better write submission;
    In my experience [1] Btrfs RAID1 on heavy write operations first writes to
    one drive, then to another. The whole process takes up to 2x longer than
    with a single drive. On the other hand mdadm writes to both drives
    simultaneously.

[1] https://www.mail-archive.com/linux-btrfs@vger.kernel.org/msg34103.html

Con:

  * You only get the ability to recover from a checksum failure with Btrfs
    RAID1, not with mdadm RAID1 (see Russell's reply).

-- 
With respect,
Roman

Re: pro/cons of raid1 with mdadm/lvm2

[Russell Coker]

On Mon, 1 Dec 2014, Chris Murphy <lists@colorremedies.com> wrote:
> On Sun, Nov 30, 2014 at 3:06 PM, Russell Coker <russell@coker.com.au> wrote:
> > When the 2 disks have different data mdadm has no way of knowing which
> > one is correct and has a 50% chance of overwriting good data. But BTRFS
> > does checksums on all reads and solves the problem of corrupt data - as
> > long as you don't have 2 corrupt sectors in matching blocks.
> 
> Yeah. I'm not sure though if openSUSE 13.2 prevents users from
> creating btrfs raid1 volumes entirely, or if it's just an install time
> limitation.

With BTRFS you can make it RAID-1 afterwards.  The possibility of data loss 
during system install usually isn't something you are concerned about so this 
shouldn't be a problem.

> I know that Fedora's installer won't allow the user to create Btrfs on
> LVM, and it probably doesn't allow it on md raid either.

For LVM that's reasonable, for MD-RAID that would be a bug IMHO.

On Mon, 1 Dec 2014, Roman Mamedov <rm@romanrm.net> wrote:
>   * mdadm RAID has much better read balancing;
>     Btrfs reads are satisfied from what's in effect a random drive
> (PID-based balancing of threads to drives), mdadm reads from the
> less-loaded drive. Also mdadm has a way to specify some RAID1 array
> members as to be never used for reads if at all possible ("write-mostly"),
> which helps in RAID1 of HDD and SSD.

True.  But that's just a lack of performance tuning in the current code, it 
will be fixed at some future time.

-- 
My Main Blog         http://etbe.coker.com.au/
My Documents Blog    http://doc.coker.com.au/

Re: pro/cons of raid1 with mdadm/lvm2

[Gour]

On Mon, 01 Dec 2014 09:06:19 +1100
Russell Coker <russell@coker.com.au> wrote:

> When the 2 disks have different data mdadm has no way of knowing
> which one is correct and has a 50% chance of overwriting good data.
> But BTRFS does checksums on all reads and solves the problem of
> corrupt data - as long as you don't have 2 corrupt sectors in
> matching blocks.

Hmm, this is very interesting and valuable info. Thank you.


Sincerely,
Gour

-- 
Many, many births both you and I have passed. I can remember 
all of them, but you cannot, O subduer of the enemy!

-- 
One is understood to be in full knowledge whose every endeavor 
is devoid of desire for sense gratification. He is said by sages 
to be a worker for whom the reactions of work have been burned 
up by the fire of perfect knowledge.

http://www.atmarama.net | Hlapicina (Croatia) | GPG: 52B5C810

Re: pro/cons of raid1 with mdadm/lvm2

[Gour]

On Sun, 30 Nov 2014 18:00:37 -0700
Chris Murphy <lists@colorremedies.com> wrote:

> Yeah. I'm not sure though if openSUSE 13.2 prevents users from
> creating btrfs raid1 volumes entirely, or if it's just an install time
> limitation.

I was able to create btrfs raid1 volumes under lvm, but installer failed
at installing Grub2.

Don't know if it is installer limitation and decided to use btrfs'
native raid1, at least for root. Will see whether to use it for /home as
well or use XFS as suggested by SUSE.


Sincerely,
Gour

-- 
In the material world, one who is unaffected by whatever good 
or evil he may obtain, neither praising it nor despising it, 
is firmly fixed in perfect knowledge.

-- 
The humble sages, by virtue of true knowledge, see with equal 
vision a learned and gentle brāhmana, a cow, an elephant, a dog 
and a dog-eater.

http://www.atmarama.net | Hlapicina (Croatia) | GPG: 52B5C810

Re: pro/cons of raid1 with mdadm/lvm2

[Gour]

On Mon, 1 Dec 2014 10:42:36 +0500
Roman Mamedov <rm@romanrm.net> wrote:

> Pros:

[...]

> Con:
> 
>   * You only get the ability to recover from a checksum failure with
> Btrfs RAID1, not with mdadm RAID1 (see Russell's reply).

For the reasons you mentioned I'll keep my root under btrfs' native raid
and do some more thinking in regard to /home and possibility to use XFS
for it.


Sincerely,
Gour

-- 
Abandoning all attachment to the results of his activities, 
ever satisfied and independent, he performs no fruitive action, 
although engaged in all kinds of undertakings.

Re: pro/cons of raid1 with mdadm/lvm2

[Robert White]

On 12/01/2014 01:26 AM, Gour wrote:
> On Mon, 01 Dec 2014 09:06:19 +1100
> Russell Coker <russell@coker.com.au> wrote:
>
>> When the 2 disks have different data mdadm has no way of knowing
>> which one is correct and has a 50% chance of overwriting good data.
>> But BTRFS does checksums on all reads and solves the problem of
>> corrupt data - as long as you don't have 2 corrupt sectors in
>> matching blocks.
>
> Hmm, this is very interesting and valuable info. Thank you.
>

Don't get too excited. If the data disagrees in either system because of 
partial starts (e.g. one mount /dev/sda1 is hot, the next mount 
/dev/sdb1 is hot) leaving both disks with equally valid but disagreeing 
data, "magical guessing" _will_ ensue.

In the BTRFS case the system will guess based on generation numbers.

In the MDADM case the system will guess based on the write intent bitmaps.

In both cases, you are _way_ better off invalidating any missing array 
elements if you find yourself reaching a write-available event with said 
missing elements.

BTRFS checksums really help when a write goes to both devices (e.g. 
/dev/sda1 and /dev/sdb1) and the disk subsystem silently fails one of 
the necessary writes, creating a disagreement between the checksum and 
the data block on the drive in question.

It's an outstanding feature, but it doesn't protect you from weak 
maintenance after a partial start.