Drives failures in irregular RAID1-Pool

Drives failures in irregular RAID1-Pool

[Stefan Malte Schumacher]

Hello,

I recently read something about raidz and truenas, which led to me
realizing that despite using it for years as my main file storage I
couldn't answer the same question regarding btrfs. Here it comes:

I have a pool of harddisks of different sizes using RAID1 for Data and
Metadata. Can the largest drive fail without causing any data loss? I
always assumed that the data would be distributed in a way that would
prevent data loss regardless of the drive size, but now I realize I
have never experienced this before and should prepare for this
scenario.

Total devices 6 FS bytes used 27.72TiB
devid    7 size 9.10TiB used 6.89TiB path /dev/sdb
devid    8 size 16.37TiB used 14.15TiB path /dev/sdf
devid    9 size 9.10TiB used 6.90TiB path /dev/sda
devid   10 size 12.73TiB used 10.53TiB path /dev/sdd
devid   11 size 12.73TiB used 10.54TiB path /dev/sde
devid   12 size 9.10TiB used 6.90TiB path /dev/sdc

Yours sincerely
Stefan Schumacher

Re: Drives failures in irregular RAID1-Pool

[Andrei Borzenkov]

On 28.07.2023 16:59, Stefan Malte Schumacher wrote:
> Hello,
> 
> I recently read something about raidz and truenas, which led to me
> realizing that despite using it for years as my main file storage I
> couldn't answer the same question regarding btrfs. Here it comes:
> 
> I have a pool of harddisks of different sizes using RAID1 for Data and
> Metadata. Can the largest drive fail without causing any data loss? I
> always assumed that the data would be distributed in a way that would
> prevent data loss regardless of the drive size, but now I realize I
> have never experienced this before and should prepare for this
> scenario.
> 

RAID1 should store each data copy on a different drive, which means all 
data on a failed disk must have another copy on some other disk.

> Total devices 6 FS bytes used 27.72TiB
> devid    7 size 9.10TiB used 6.89TiB path /dev/sdb
> devid    8 size 16.37TiB used 14.15TiB path /dev/sdf
> devid    9 size 9.10TiB used 6.90TiB path /dev/sda
> devid   10 size 12.73TiB used 10.53TiB path /dev/sdd
> devid   11 size 12.73TiB used 10.54TiB path /dev/sde
> devid   12 size 9.10TiB used 6.90TiB path /dev/sdc
> 
> Yours sincerely
> Stefan Schumacher

Re: Drives failures in irregular RAID1-Pool

[Stefan Malte Schumacher]

Thanks for the quick reply. Is there any way for me to validate if the
filesystem has redundant copies of all my files on different drives?
I read that it was suggested to do a full rebalance when adding a
drive to a RAID5 array. Should I do the same when adding a new disk to
my array?

Yours sincerely
Stefan

Am Fr., 28. Juli 2023 um 17:22 Uhr schrieb Andrei Borzenkov
<arvidjaar@gmail.com>:
>
> On 28.07.2023 16:59, Stefan Malte Schumacher wrote:
> > Hello,
> >
> > I recently read something about raidz and truenas, which led to me
> > realizing that despite using it for years as my main file storage I
> > couldn't answer the same question regarding btrfs. Here it comes:
> >
> > I have a pool of harddisks of different sizes using RAID1 for Data and
> > Metadata. Can the largest drive fail without causing any data loss? I
> > always assumed that the data would be distributed in a way that would
> > prevent data loss regardless of the drive size, but now I realize I
> > have never experienced this before and should prepare for this
> > scenario.
> >
>
> RAID1 should store each data copy on a different drive, which means all
> data on a failed disk must have another copy on some other disk.
>
> > Total devices 6 FS bytes used 27.72TiB
> > devid    7 size 9.10TiB used 6.89TiB path /dev/sdb
> > devid    8 size 16.37TiB used 14.15TiB path /dev/sdf
> > devid    9 size 9.10TiB used 6.90TiB path /dev/sda
> > devid   10 size 12.73TiB used 10.53TiB path /dev/sdd
> > devid   11 size 12.73TiB used 10.54TiB path /dev/sde
> > devid   12 size 9.10TiB used 6.90TiB path /dev/sdc
> >
> > Yours sincerely
> > Stefan Schumacher
>

Re: Drives failures in irregular RAID1-Pool

[joshua]

You should not need to.

But you can verify all your data is stored as raid1 by running ‘btrfs fi usage/mountpoint’

It will show what your storage is used for (data, metadata, etc) as well as what raid profiles are in use for different data.

—Joshua Villwock

> On Jul 28, 2023, at 8:26 AM, Stefan Malte Schumacher <s.schumacher@netcologne.de> wrote:
> 
> Thanks for the quick reply. Is there any way for me to validate if the
> filesystem has redundant copies of all my files on different drives?
> I read that it was suggested to do a full rebalance when adding a
> drive to a RAID5 array. Should I do the same when adding a new disk to
> my array?
> 
> Yours sincerely
> Stefan
> 
>> Am Fr., 28. Juli 2023 um 17:22 Uhr schrieb Andrei Borzenkov
>> <arvidjaar@gmail.com>:
>> 
>>> On 28.07.2023 16:59, Stefan Malte Schumacher wrote:
>>> Hello,
>>> 
>>> I recently read something about raidz and truenas, which led to me
>>> realizing that despite using it for years as my main file storage I
>>> couldn't answer the same question regarding btrfs. Here it comes:
>>> 
>>> I have a pool of harddisks of different sizes using RAID1 for Data and
>>> Metadata. Can the largest drive fail without causing any data loss? I
>>> always assumed that the data would be distributed in a way that would
>>> prevent data loss regardless of the drive size, but now I realize I
>>> have never experienced this before and should prepare for this
>>> scenario.
>>> 
>> 
>> RAID1 should store each data copy on a different drive, which means all
>> data on a failed disk must have another copy on some other disk.
>> 
>>> Total devices 6 FS bytes used 27.72TiB
>>> devid    7 size 9.10TiB used 6.89TiB path /dev/sdb
>>> devid    8 size 16.37TiB used 14.15TiB path /dev/sdf
>>> devid    9 size 9.10TiB used 6.90TiB path /dev/sda
>>> devid   10 size 12.73TiB used 10.53TiB path /dev/sdd
>>> devid   11 size 12.73TiB used 10.54TiB path /dev/sde
>>> devid   12 size 9.10TiB used 6.90TiB path /dev/sdc
>>> 
>>> Yours sincerely
>>> Stefan Schumacher
>> 

Re: Drives failures in irregular RAID1-Pool

[Andrei Borzenkov]

On 28.07.2023 18:26, Stefan Malte Schumacher wrote:
> Thanks for the quick reply. Is there any way for me to validate if the
> filesystem has redundant copies of all my files on different drives?

I can only think of "btrfs scrub".

> I read that it was suggested to do a full rebalance when adding a
> drive to a RAID5 array. Should I do the same when adding a new disk to
> my array?
> 

I do not know where you read it and I cannot comment on something I have 
not seen. But for RAID1 I do not see any reason to do it.

> Yours sincerely
> Stefan
> 
> Am Fr., 28. Juli 2023 um 17:22 Uhr schrieb Andrei Borzenkov
> <arvidjaar@gmail.com>:
>>
>> On 28.07.2023 16:59, Stefan Malte Schumacher wrote:
>>> Hello,
>>>
>>> I recently read something about raidz and truenas, which led to me
>>> realizing that despite using it for years as my main file storage I
>>> couldn't answer the same question regarding btrfs. Here it comes:
>>>
>>> I have a pool of harddisks of different sizes using RAID1 for Data and
>>> Metadata. Can the largest drive fail without causing any data loss? I
>>> always assumed that the data would be distributed in a way that would
>>> prevent data loss regardless of the drive size, but now I realize I
>>> have never experienced this before and should prepare for this
>>> scenario.
>>>
>>
>> RAID1 should store each data copy on a different drive, which means all
>> data on a failed disk must have another copy on some other disk.
>>
>>> Total devices 6 FS bytes used 27.72TiB
>>> devid    7 size 9.10TiB used 6.89TiB path /dev/sdb
>>> devid    8 size 16.37TiB used 14.15TiB path /dev/sdf
>>> devid    9 size 9.10TiB used 6.90TiB path /dev/sda
>>> devid   10 size 12.73TiB used 10.53TiB path /dev/sdd
>>> devid   11 size 12.73TiB used 10.54TiB path /dev/sde
>>> devid   12 size 9.10TiB used 6.90TiB path /dev/sdc
>>>
>>> Yours sincerely
>>> Stefan Schumacher
>>

Re: Drives failures in irregular RAID1-Pool

[Forza]

---- From: Andrei Borzenkov <arvidjaar@gmail.com> -- Sent: 2023-07-28 - 19:00 ----

> On 28.07.2023 18:26, Stefan Malte Schumacher wrote:
>> Thanks for the quick reply. Is there any way for me to validate if the
>> filesystem has redundant copies of all my files on different drives?
> 
> I can only think of "btrfs scrub".
> 
>> I read that it was suggested to do a full rebalance when adding a
>> drive to a RAID5 array. Should I do the same when adding a new disk to
>> my array?
>> 
> 
> I do not know where you read it and I cannot comment on something I have 
> not seen. But for RAID1 I do not see any reason to do it.

It may be needed if the layout becomes too unbalanced, else you may end up with no free space ENOSPC errors. 

There is also the size restrictions to think of when mixing different sizes. Use the btrfs size calculator to see if you end up with some unusable space with the sizes you have. 

https://carfax.org.uk/btrfs-usage/?c=2&slo=1&shi=1&p=0&dg=1&d=910&d=1273&d=1273&d=910&d=1637&d=910


> 
>> Yours sincerely
>> Stefan
>> 
>> Am Fr., 28. Juli 2023 um 17:22 Uhr schrieb Andrei Borzenkov
>> <arvidjaar@gmail.com>:
>>>
>>> On 28.07.2023 16:59, Stefan Malte Schumacher wrote:
>>>> Hello,
>>>>
>>>> I recently read something about raidz and truenas, which led to me
>>>> realizing that despite using it for years as my main file storage I
>>>> couldn't answer the same question regarding btrfs. Here it comes:
>>>>
>>>> I have a pool of harddisks of different sizes using RAID1 for Data and
>>>> Metadata. Can the largest drive fail without causing any data loss? I
>>>> always assumed that the data would be distributed in a way that would
>>>> prevent data loss regardless of the drive size, but now I realize I
>>>> have never experienced this before and should prepare for this
>>>> scenario.
>>>>
>>>
>>> RAID1 should store each data copy on a different drive, which means all
>>> data on a failed disk must have another copy on some other disk.
>>>
>>>> Total devices 6 FS bytes used 27.72TiB
>>>> devid    7 size 9.10TiB used 6.89TiB path /dev/sdb
>>>> devid    8 size 16.37TiB used 14.15TiB path /dev/sdf
>>>> devid    9 size 9.10TiB used 6.90TiB path /dev/sda
>>>> devid   10 size 12.73TiB used 10.53TiB path /dev/sdd
>>>> devid   11 size 12.73TiB used 10.54TiB path /dev/sde
>>>> devid   12 size 9.10TiB used 6.90TiB path /dev/sdc
>>>>
>>>> Yours sincerely
>>>> Stefan Schumacher
>>>
>