Fixing a degraded RAID

Fixing a degraded RAID

[Michael Welsh Duggan]

I had a disk in my RAID0 die:

  root@maru2:~# /usr/local/src/btrfs-progs/btrfs fi show 8c530f6f-7592-4d57-854d-1fae33ae7cb6
  Label: none  uuid: 8c530f6f-7592-4d57-854d-1fae33ae7cb6
          Total devices 3 FS bytes used 329.66GiB
          devid    1 size 1.79TiB used 357.03GiB path /dev/sdd1
          devid    3 size 931.51GiB used 178.00GiB path /dev/sdf
          *** Some devices missing
  
  Btrfs v3.12-43-gc2081e2-dirty

So, I try to mount it with -o degraded.  That fails, implying that a
degraded mount needs to be read-only.  So I successfully mount it read
only.  Then I want to add a new disk to replace the missing one:

  root@maru2:~# /usr/local/src/btrfs-progs/btrfs dev add /dev/sdg /mnt/
  ERROR: error adding the device '/dev/sdg' - Read-only file system

So I can't add a disk if it's mounted read-only, and can't mount it
read-write?  What am I missing?  What's my way around this?

-- 
Michael Welsh Duggan
(md5i@md5i.com)

Re: Fixing a degraded RAID

[Chris Murphy]

On Apr 18, 2014, at 8:36 PM, Michael Welsh Duggan <mwd@md5i.com> wrote:

> I had a disk in my RAID0 die:
> 
>  root@maru2:~# /usr/local/src/btrfs-progs/btrfs fi show 8c530f6f-7592-4d57-854d-1fae33ae7cb6
>  Label: none  uuid: 8c530f6f-7592-4d57-854d-1fae33ae7cb6
>          Total devices 3 FS bytes used 329.66GiB
>          devid    1 size 1.79TiB used 357.03GiB path /dev/sdd1
>          devid    3 size 931.51GiB used 178.00GiB path /dev/sdf
>          *** Some devices missing
> 
>  Btrfs v3.12-43-gc2081e2-dirty
> 
> So, I try to mount it with -o degraded.  That fails, implying that a
> degraded mount needs to be read-only.  So I successfully mount it read
> only.  Then I want to add a new disk to replace the missing one:
> 
>  root@maru2:~# /usr/local/src/btrfs-progs/btrfs dev add /dev/sdg /mnt/
>  ERROR: error adding the device '/dev/sdg' - Read-only file system
> 
> So I can't add a disk if it's mounted read-only, and can't mount it
> read-write?  What am I missing?  What's my way around this?

Not really because it's raid0. The minimum number of disks for a 3 disk raid0 is 3 disks. Upon losing one, the data portion of the file system is toast. I'm guessing it lets you mount it ro,degraded because the metadata and hence the file system itself, is raid1 (the default for raid0 data volumes).

The file system could determine what files aren't damaged, i.e they exist entirely on one of the remaining two drives, with no dependency on the dead one. But I don't think this code exists yet. You could possibly rsync the drive to another file system, and bad files damaged by the missing disk will be reported in dmesg while the good ones would still copy over. I haven't tried this.

Chris Murphy

Re: Fixing a degraded RAID

[Michael Welsh Duggan]

Chris Murphy <lists@colorremedies.com> writes:

> On Apr 18, 2014, at 8:36 PM, Michael Welsh Duggan <mwd@md5i.com> wrote:
>
>> I had a disk in my RAID0 die:
>> 
>>  root@maru2:~# /usr/local/src/btrfs-progs/btrfs fi show 8c530f6f-7592-4d57-854d-1fae33ae7cb6
>>  Label: none  uuid: 8c530f6f-7592-4d57-854d-1fae33ae7cb6
>>          Total devices 3 FS bytes used 329.66GiB
>>          devid    1 size 1.79TiB used 357.03GiB path /dev/sdd1
>>          devid    3 size 931.51GiB used 178.00GiB path /dev/sdf
>>          *** Some devices missing
>> 
>>  Btrfs v3.12-43-gc2081e2-dirty
>> 
>> So, I try to mount it with -o degraded.  That fails, implying that a
>> degraded mount needs to be read-only.  So I successfully mount it read
>> only.  Then I want to add a new disk to replace the missing one:
>> 
>>  root@maru2:~# /usr/local/src/btrfs-progs/btrfs dev add /dev/sdg /mnt/
>>  ERROR: error adding the device '/dev/sdg' - Read-only file system
>> 
>> So I can't add a disk if it's mounted read-only, and can't mount it
>> read-write?  What am I missing?  What's my way around this?
>
> Not really because it's raid0. The minimum number of disks for a 3
> disk raid0 is 3 disks. Upon losing one, the data portion of the file
> system is toast. I'm guessing it lets you mount it ro,degraded because
> the metadata and hence the file system itself, is raid1 (the default
> for raid0 data volumes).
>
> The file system could determine what files aren't damaged, i.e they
> exist entirely on one of the remaining two drives, with no dependency
> on the dead one. But I don't think this code exists yet. You could
> possibly rsync the drive to another file system, and bad files damaged
> by the missing disk will be reported in dmesg while the good ones
> would still copy over. I haven't tried this.

My mistake.  I meant RAID1.

-- 
Michael Welsh Duggan
(md5i@md5i.com)

Re: Fixing a degraded RAID

[Michael Welsh Duggan]

Michael Welsh Duggan <mwd@md5i.com> writes:

> I had a disk in my RAID0 die:

My mistake.  This was a stupid typo.  The drives were in a RAID 1
configuration.  Both data and metadata are DUP'd.

root@maru2:~# /usr/local/src/btrfs-progs/btrfs fi df /mnt/
Data, RAID1: total=353.00GiB, used=328.24GiB
System, single: total=32.00MiB, used=56.00KiB
Metadata, RAID1: total=4.00GiB, used=1.43GiB

>   root@maru2:~# /usr/local/src/btrfs-progs/btrfs fi show 8c530f6f-7592-4d57-854d-1fae33ae7cb6
>   Label: none  uuid: 8c530f6f-7592-4d57-854d-1fae33ae7cb6
>           Total devices 3 FS bytes used 329.66GiB
>           devid    1 size 1.79TiB used 357.03GiB path /dev/sdd1
>           devid    3 size 931.51GiB used 178.00GiB path /dev/sdf
>           *** Some devices missing
>   
>   Btrfs v3.12-43-gc2081e2-dirty
>
> So, I try to mount it with -o degraded.  That fails, implying that a
> degraded mount needs to be read-only.  So I successfully mount it read
> only.  Then I want to add a new disk to replace the missing one:
>
>   root@maru2:~# /usr/local/src/btrfs-progs/btrfs dev add /dev/sdg /mnt/
>   ERROR: error adding the device '/dev/sdg' - Read-only file system
>
> So I can't add a disk if it's mounted read-only, and can't mount it
> read-write?  What am I missing?  What's my way around this?

-- 
Michael Welsh Duggan
(md5i@md5i.com)

Re: Fixing a degraded RAID

[Chris Murphy]

On Apr 18, 2014, at 9:23 PM, Michael Welsh Duggan <mwd@md5i.com> wrote:

> My mistake.  This was a stupid typo.  The drives were in a RAID 1
> configuration.  Both data and metadata are DUP'd.
> 
> root@maru2:~# /usr/local/src/btrfs-progs/btrfs fi df /mnt/
> Data, RAID1: total=353.00GiB, used=328.24GiB
> System, single: total=32.00MiB, used=56.00KiB
> Metadata, RAID1: total=4.00GiB, used=1.43GiB

Somehow system is set to single instead of raid1. I haven't seen system and metadata be set to different profiles. Only 56KB of system data is being used, but I don't know how that's distributed across three drives when the profile is single.

Chris Murphy

Re: Fixing a degraded RAID

[Xavier Bassery]

On Fri, 18 Apr 2014 23:23:56 -0400
Michael Welsh Duggan <mwd@md5i.com> wrote:

> Michael Welsh Duggan <mwd@md5i.com> writes:
> 
> > I had a disk in my RAID0 die:
> 
> My mistake.  This was a stupid typo.  The drives were in a RAID 1
> configuration.  Both data and metadata are DUP'd.
> 
> root@maru2:~# /usr/local/src/btrfs-progs/btrfs fi df /mnt/
> Data, RAID1: total=353.00GiB, used=328.24GiB
> System, single: total=32.00MiB, used=56.00KiB
> Metadata, RAID1: total=4.00GiB, used=1.43GiB
> 
> >   root@maru2:~# /usr/local/src/btrfs-progs/btrfs fi show
> > 8c530f6f-7592-4d57-854d-1fae33ae7cb6 Label: none  uuid:
> > 8c530f6f-7592-4d57-854d-1fae33ae7cb6 Total devices 3 FS bytes used
> > 329.66GiB devid    1 size 1.79TiB used 357.03GiB path /dev/sdd1
> >           devid    3 size 931.51GiB used 178.00GiB path /dev/sdf
> >           *** Some devices missing
> >   
> >   Btrfs v3.12-43-gc2081e2-dirty
> >
> > So, I try to mount it with -o degraded.  That fails, implying that a
> > degraded mount needs to be read-only.  So I successfully mount it
> > read only.  Then I want to add a new disk to replace the missing
> > one:
> >
> >   root@maru2:~# /usr/local/src/btrfs-progs/btrfs dev
> > add /dev/sdg /mnt/ ERROR: error adding the device '/dev/sdg' -
> > Read-only file system
> >
> > So I can't add a disk if it's mounted read-only, and can't mount it
> > read-write?  What am I missing?  What's my way around this?
> 

As Chris Murphy noted, your "System" chunk still reported as "single" is
likely the issue here.
This reminds me of a bug i stumbled upon once:
https://bugzilla.kernel.org/show_bug.cgi?id=60594

Have you converted your fs from SINGLE to RAID-1 profile by running a
balance?

This conversion bug should no longer occur with the patch
"Btrfs: stop refusing the relocation of chunk 0" which has been merged
since 3.12.
But chances are that you've converted your fs with an older kernel
without that patch.
While Ilya Dryomov investigated the issue on my system, we managed to
work around this by using a patched module that ignored the safety
measure refusing to mount rw in this case.

I still have the patch, but i think the safest way and only other
option would be to copy the data from your ro fs and recreate it from
scratch.
Doing so with recent btrfs progs (v3.12 or newer) you'll benefit from
the new default metadata blocksize of 16KB that should increase
performance.

Xavier Bassery

Re: Fixing a degraded RAID

[Michael Welsh Duggan]

Xavier Bassery <xavier@bartica.org> writes:

> On Fri, 18 Apr 2014 23:23:56 -0400
> Michael Welsh Duggan <mwd@md5i.com> wrote:
>
>> Michael Welsh Duggan <mwd@md5i.com> writes:
>> 
>> root@maru2:~# /usr/local/src/btrfs-progs/btrfs fi df /mnt/
>> Data, RAID1: total=353.00GiB, used=328.24GiB
>> System, single: total=32.00MiB, used=56.00KiB
>> Metadata, RAID1: total=4.00GiB, used=1.43GiB
>
> As Chris Murphy noted, your "System" chunk still reported as "single"
> is likely the issue here.  This reminds me of a bug i stumbled upon
> once: https://bugzilla.kernel.org/show_bug.cgi?id=60594
>
> Have you converted your fs from SINGLE to RAID-1 profile by running a
> balance?

Yes, I did.  And I think I was still on 3.10 or 11 when this happened.

> This conversion bug should no longer occur with the patch "Btrfs: stop
> refusing the relocation of chunk 0" which has been merged since 3.12.
> But chances are that you've converted your fs with an older kernel
> without that patch.  While Ilya Dryomov investigated the issue on my
> system, we managed to work around this by using a patched module that
> ignored the safety measure refusing to mount rw in this case.
>
> I still have the patch, but i think the safest way and only other
> option would be to copy the data from your ro fs and recreate it from
> scratch.  Doing so with recent btrfs progs (v3.12 or newer) you'll
> benefit from the new default metadata blocksize of 16KB that should
> increase performance.

Right.  *sigh*  A pain, but needs must, and all that.  Thanks for your
help.

-- 
Michael Welsh Duggan
(md5i@md5i.com)