raid1 on uneven-sized disks

raid1 on uneven-sized disks

[Jim MacBaine]

Hi,

How does btrfs handle raid1 on a bunch of uneven sized disks? Can I
just keep adding arbitrarily sized disks to an existing raid1 and
expect the file system to continue to keep two copies of everything,
so I could survive the loss of any single disk without data loss? Does
btrfs work this way?

For backing up several machines at home, I use an old Debian machine
which is equipped with some spare disks I could get my hands on.
Currently that would be 2 x 2TB, 1 x 1.5TB , 3 x 1TB disks.

Traditionally I'm using rsync to create hardlinked backups on ext3/4
on md-raid1. This setup has been working reliably for many years now,
including the survival of two disk failures. But it is quite
cumbersome to reshape the structure of the raids when I get a new
disk, an old one fails, or space requirements change. A single btrfs
with snapshots would be much easier to handle. I'm not aiming for
absolute reliability here -- all the really important stuff is backed
up in a third place as well. But I would rather not like my file
system to be the weakest link in the backup chain.

Regards,
Jim

Re: raid1 on uneven-sized disks

[Hugo Mills]

[-- Attachment #1: Type: text/plain, Size: 1495 bytes --]

On Sun, Aug 09, 2015 at 02:29:53PM +0200, Jim MacBaine wrote:
> Hi,
> 
> How does btrfs handle raid1 on a bunch of uneven sized disks? Can I
> just keep adding arbitrarily sized disks to an existing raid1 and
> expect the file system to continue to keep two copies of everything,
> so I could survive the loss of any single disk without data loss? Does
> btrfs work this way?

   Yes, exactly.

   You may find that http://carfax.org.uk/btrfs-usage/ is helpful.

   Hugo.

> For backing up several machines at home, I use an old Debian machine
> which is equipped with some spare disks I could get my hands on.
> Currently that would be 2 x 2TB, 1 x 1.5TB , 3 x 1TB disks.
> 
> Traditionally I'm using rsync to create hardlinked backups on ext3/4
> on md-raid1. This setup has been working reliably for many years now,
> including the survival of two disk failures. But it is quite
> cumbersome to reshape the structure of the raids when I get a new
> disk, an old one fails, or space requirements change. A single btrfs
> with snapshots would be much easier to handle. I'm not aiming for
> absolute reliability here -- all the really important stuff is backed
> up in a third place as well. But I would rather not like my file
> system to be the weakest link in the backup chain.
> 
> Regards,
> Jim

-- 
Hugo Mills             | Guards! Help! We're being rescued!
hugo@... carfax.org.uk |
http://carfax.org.uk/  |
PGP: E2AB1DE4          |                       The Stainless Steel Rat Forever

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 836 bytes --]

Re: raid1 on uneven-sized disks

[Rich Freeman]

On Sun, Aug 9, 2015 at 8:47 AM, Hugo Mills <hugo@carfax.org.uk> wrote:
> On Sun, Aug 09, 2015 at 02:29:53PM +0200, Jim MacBaine wrote:
>> Hi,
>>
>> How does btrfs handle raid1 on a bunch of uneven sized disks? Can I
>> just keep adding arbitrarily sized disks to an existing raid1 and
>> expect the file system to continue to keep two copies of everything,
>> so I could survive the loss of any single disk without data loss? Does
>> btrfs work this way?
>
>    Yes, exactly.
>
>    You may find that http://carfax.org.uk/btrfs-usage/ is helpful.
>

The key is that btrfs manages "raid" at the chunk level, not the
device level.  When btrfs needs more disk space it allocates a new
chunk from unallocated space on a device.  If it is in raid1 mode it
will allocate a pair of chunks from two different drives, storing the
same data in each.  The allocation algorithm is reasonably smart so if
you have 2x1TB drives and 1x3TB drive you'll end up with about 2TB of
data stored and not 1TB on each of the two 1TB drives and an empty
unusable 3TB drive.

This is also why you can switch between raid modes "on the fly" -
switching modes only affects newly-allocated chunks, and the old ones
operate in whatever mode they were previously in.  A balance operation
rewrites the existing data to new chunks which would force everything
to use the new mode.

This also lets you do things like add a disk to a raid5.  If you have
5 disks and add one more, existing chunks will be striped across 5
drives, and new chunks will be striped across 6, unless you balance
them.

That may be a bit oversimplified, and obviously others on the list
know all the details...

--
Rich

Re: raid1 on uneven-sized disks

[Duncan]

Rich Freeman posted on Sun, 09 Aug 2015 22:25:35 -0400 as excerpted:

> The key is that btrfs manages "raid" at the chunk level, not the
> device level.  When btrfs needs more disk space it allocates a new
> chunk from unallocated space on a device.  If it is in raid1 mode it
> will allocate a pair of chunks from two different drives, storing the
> same data in each.  The allocation algorithm is reasonably smart so if
> you have 2x1TB drives and 1x3TB drive you'll end up with about 2TB of
> data stored and not 1TB on each of the two 1TB drives and an empty
> unusable 3TB drive.
> 
> This is also why you can switch between raid modes "on the fly"[. ...]

Very good explanation, Rich. =:^)

> This also lets you do things like add a disk to a raid5.  If you have
> 5 disks and add one more, existing chunks will be striped across 5
> drives, and new chunks will be striped across 6, unless you balance
> them.

I'm actually posting to add a caveat to this.

While btrfs itself is "getting stable", that is, good enough for daily 
work, but not as mature and well tested as some of the other filesystems 
so do keep things you don't want to lose backed up, btrfs raid56 mode 
(that's raid5 and raid6, the same module, raid56, deals with both) was 
only code-complete with kernel/progs 3.19, and is still quite immature 
and not well tested compared to the rest of btrfs.  With 4.2 it should be 
out of its most critical phase, but I've been warning people to give it a 
year (after 3.19) before considering its stability to be close to the 
rest of btrfs.  Since there's normally ~5 kernel cycles per year, that 
would be 4.4.  Until then, if you're willing to be a guinea pig on it, 
reporting any bugs and working with the devs to test fixes, THANKS, it's 
people like you who help make it dependable for others.  Otherwise, stick 
with the more stable raid1 or raid10 modes until 4.4 or so, and at that 
point if you're still interested in raid56 mode, check the list to see 
how it's doing, and assuming nothing exceptional is going on, /then/ do 
the raid56 thing.

Of course, since the OP question was about raid1, this is probably just 
extra info for him, and mostly to cover anyone else that /might/ be 
looking at raid56 mode at this time, and happens onto this thread.


-- 
Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman

Re: raid1 on uneven-sized disks

[Duncan]

Jim MacBaine posted on Sun, 09 Aug 2015 14:29:53 +0200 as excerpted:

> Traditionally I'm using rsync to create hardlinked backups on ext3/4 on
> md-raid1. This setup has been working reliably for many years now,
> including the survival of two disk failures. But it is quite cumbersome
> to reshape the structure of the raids when I get a new disk, an old one
> fails, or space requirements change. A single btrfs with snapshots would
> be much easier to handle. I'm not aiming for absolute reliability here
> -- all the really important stuff is backed up in a third place as well.
> But I would rather not like my file system to be the weakest link in the
> backup chain.

Others answered your raid1 question well, but left this, so let me tackle 
it.

My standard answer to btrfs stability questions is that btrfs is "not yet 
fully stable and mature", it's stabilizing, and definitely more stable 
that it has been, but it has only been a few kernel cycles since some 
really big bugs that hit a lot of people, and in fact, user-space side, 
there was a critical bug in mkfs.btrfs in btrfs-progs v4.1.1 (with v4.1.2 
being current), so they do still happen from time to time, tho that one 
only affected people happening to do a mkfs.btrfs with the affected 
version, which was only out a couple weeks or so.

IOW, the standard sysadmin rule that if it's not backed up, by 
definition, you don't care about it, still applies double to btrfs -- 
really, keep your backups, or your actions demonstrate your lack of care 
and you may see the results thereof.

That said, btrfs is working well in daily use for many people, and some 
distros are shipping it as the default for at least some partitions 
(often root, where the snapshot feature is used to manage update 
rollbacks where necessary), so it's stable /enough/, as long as you do 
have those backups and are prepared to use them, should it be necessary.

We do recommend that you stay relatively current on both kernel and 
userspace, however.  So a current 4.1 series kernel and btrfs-progs 4.1.2 
are excellent, but consider another filesystem if you're the type who was 
still on a 2.x kernel thru 3.12 or so. =:^)  

And... there's a couple btrfs features to avoid at this time.  As 
mentioned elsewhere, raid56 mode is still quite new and not yet mature, 
yet, so avoid it unless you intend to take on the risk of leading-edge 
testing, and the btrfs quota code continues to be a source of issues, so 
I recommend not using quotas on btrfs.  If you need quotas, you want 
another filesystem anyway, because quotas simply aren't reliable on btrfs 
yet.

-- 
Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman

Re: raid1 on uneven-sized disks

[Jim MacBaine]

On Mon, Aug 10, 2015 at 6:13 AM, Duncan <1i5t5.duncan@cox.net> wrote:

> We do recommend that you stay relatively current on both kernel and
> userspace, however.  So a current 4.1 series kernel and btrfs-progs 4.1.2
> are excellent, but consider another filesystem if you're the type who was
> still on a 2.x kernel thru 3.12 or so. =:^)

Thank you for the good reminder.

I'm using btrfs for a few years on less important systems. And it has
been quite some time since I last had to throw away and restore an
unrecoverable btrfs filesystem. And I am comfortable with building a
custom kernel for my machines from kernel.org sources.

But that backup system I was talking about is neither a playground
nor the right place for experiments. It is deliberately a minimal system
running Debian Stable (which is currently at a 3.16 kernel) without
any modifications.

Sacrificing ease of use, comfort or even availability is not a big deal.
But sacrificing the durability of the data is not an option. The single
purpose of this system is to keep the data reliably.

So I think I'll start slow with easy-to-recover data. And give it some
more time before I start storing the important stuff on btrfs.

Cheers,
Jim