Raid1 won't mount degraded

Raid1 won't mount degraded

[Subscription Account]

Hi,

I had to remove one disk from raid 1 and I rebooted the system and was
able to mount in degraded mode. Then I powered off the system added a
new disk and when I am trying to mount the btrfs filesystem in
degraded mode it will no longer mount it read-write. I can mount
read-only though.

I tried running btrfs-zero-log as well and that gices a core dump.

My system details as follows:

rajeev@NAS:~$ uname -a
Linux NAS 4.4.0-59-generic #80-Ubuntu SMP Fri Jan 6 17:47:47 UTC 2017
x86_64 x86_64 x86_64 GNU/Linux

root@NAS:~# btrfs --version
Btrfs v3.12

root@NAS:~# btrfs fi show
Label: none  uuid: f89c12fb-2304-4dce-bed5-fda008b86bde
        Total devices 2 FS bytes used 2.49TiB
        devid    1 size 2.73TiB used 2.52TiB path /dev/sdb1
        *** Some devices missing

Btrfs v3.12

root@NAS:~# mount -o degraded /dev/sdb1 /local/nas
mount: wrong fs type, bad option, bad superblock on /dev/sdb1,
       missing codepage or helper program, or other error

       In some cases useful info is found in syslog - try
       dmesg | tail or so.

root@NAS:~# dmesg | tail
[ 2506.816795] BTRFS: missing devices(1) exceeds the limit(0),
writeable mount is not allowed
[ 2506.878031] BTRFS: open_ctree failed
[ 4590.339548] BTRFS info (device sdb1): allowing degraded mounts
[ 4590.505739] BTRFS info (device sdb1): bdev /dev/sdb1 errs: wr 0, rd
90, flush 0, corrupt 0, gen 0
[ 4609.481310] BTRFS: missing devices(1) exceeds the limit(0),
writeable mount is not allowed
[ 4609.536509] BTRFS: open_ctree failed
[ 4852.498305] BTRFS info (device sdb1): allowing degraded mounts
[ 4853.294311] BTRFS info (device sdb1): bdev /dev/sdb1 errs: wr 0, rd
90, flush 0, corrupt 0, gen 0
[ 4873.026855] BTRFS: missing devices(1) exceeds the limit(0),
writeable mount is not allowed
[ 4873.083840] BTRFS: open_ctree failed

root@NAS:~# mount -o degraded,ro /dev/sdb1 /local/nas

root@NAS:~# btrfs fi df /local/nas
Data, RAID1: total=2.51TiB, used=2.49TiB
Data, single: total=1.01GiB, used=88.27MiB
System, RAID1: total=8.00MiB, used=352.00KiB
System, single: total=4.00MiB, used=12.00KiB
Metadata, RAID1: total=6.00GiB, used=5.01GiB
Metadata, single: total=8.00MiB, used=3.31MiB
unknown, single: total=512.00MiB, used=0.00

In the read-only mode I am not able to add a new device or replace :(.
Please help.

Thanks,

--
raj

Re: Raid1 won't mount degraded

[Subscription Account]

When I try to run any of the commands like init-csum-tree etc the
btrfs program core dumps. Here is an example

root@NAS:/root# btrfsck --init-csum-tree /dev/sdb1
Creating a new CRC tree
warning, device 2 is missing
parent transid verify failed on 8388608 wanted 136966 found 136967
parent transid verify failed on 8388608 wanted 136966 found 136967
Ignoring transid failure
Checking filesystem on /dev/sdb1
UUID: f89c12fb-2304-4dce-bed5-fda008b86bde
Reinitialize checksum tree
extent-tree.c:2703: btrfs_reserve_extent: BUG_ON `ret` triggered, value -28
btrfs check[0x44e371]
btrfs check(btrfs_reserve_extent+0xbc6)[0x4540e6]
btrfs check(btrfs_alloc_free_block+0x5a)[0x45415c]
btrfs check[0x41d788]
btrfs check(cmd_check+0x1270)[0x430633]
btrfs check(main+0x155)[0x40a813]
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf0)[0x7f4a89555830]
btrfs check(_start+0x29)[0x40a409]
Aborted (core dumped)




On Sun, Jan 29, 2017 at 8:12 AM, Subscription Account
<4subsacct+btrfs@gmail.com> wrote:
> Hi,
>
> I had to remove one disk from raid 1 and I rebooted the system and was
> able to mount in degraded mode. Then I powered off the system added a
> new disk and when I am trying to mount the btrfs filesystem in
> degraded mode it will no longer mount it read-write. I can mount
> read-only though.
>
> I tried running btrfs-zero-log as well and that gices a core dump.
>
> My system details as follows:
>
> rajeev@NAS:~$ uname -a
> Linux NAS 4.4.0-59-generic #80-Ubuntu SMP Fri Jan 6 17:47:47 UTC 2017
> x86_64 x86_64 x86_64 GNU/Linux
>
> root@NAS:~# btrfs --version
> Btrfs v3.12
>
> root@NAS:~# btrfs fi show
> Label: none  uuid: f89c12fb-2304-4dce-bed5-fda008b86bde
>         Total devices 2 FS bytes used 2.49TiB
>         devid    1 size 2.73TiB used 2.52TiB path /dev/sdb1
>         *** Some devices missing
>
> Btrfs v3.12
>
> root@NAS:~# mount -o degraded /dev/sdb1 /local/nas
> mount: wrong fs type, bad option, bad superblock on /dev/sdb1,
>        missing codepage or helper program, or other error
>
>        In some cases useful info is found in syslog - try
>        dmesg | tail or so.
>
> root@NAS:~# dmesg | tail
> [ 2506.816795] BTRFS: missing devices(1) exceeds the limit(0),
> writeable mount is not allowed
> [ 2506.878031] BTRFS: open_ctree failed
> [ 4590.339548] BTRFS info (device sdb1): allowing degraded mounts
> [ 4590.505739] BTRFS info (device sdb1): bdev /dev/sdb1 errs: wr 0, rd
> 90, flush 0, corrupt 0, gen 0
> [ 4609.481310] BTRFS: missing devices(1) exceeds the limit(0),
> writeable mount is not allowed
> [ 4609.536509] BTRFS: open_ctree failed
> [ 4852.498305] BTRFS info (device sdb1): allowing degraded mounts
> [ 4853.294311] BTRFS info (device sdb1): bdev /dev/sdb1 errs: wr 0, rd
> 90, flush 0, corrupt 0, gen 0
> [ 4873.026855] BTRFS: missing devices(1) exceeds the limit(0),
> writeable mount is not allowed
> [ 4873.083840] BTRFS: open_ctree failed
>
> root@NAS:~# mount -o degraded,ro /dev/sdb1 /local/nas
>
> root@NAS:~# btrfs fi df /local/nas
> Data, RAID1: total=2.51TiB, used=2.49TiB
> Data, single: total=1.01GiB, used=88.27MiB
> System, RAID1: total=8.00MiB, used=352.00KiB
> System, single: total=4.00MiB, used=12.00KiB
> Metadata, RAID1: total=6.00GiB, used=5.01GiB
> Metadata, single: total=8.00MiB, used=3.31MiB
> unknown, single: total=512.00MiB, used=0.00
>
> In the read-only mode I am not able to add a new device or replace :(.
> Please help.
>
> Thanks,
>
> --
> raj

Re: Raid1 won't mount degraded

[Lakshmipathi.G]

>powered off the system added a
>new disk

>root@NAS:~# btrfs fi show
>Label: none  uuid: f89c12fb-2304-4dce-bed5-fda008b86bde
>       Total devices 2 FS bytes used 2.49TiB
>      devid    1 size 2.73TiB used 2.52TiB path /dev/sdb1
>     *** Some devices missing

Are you sure 'device add' was successful? Whats the new  device name?
'btrfs fi show' shows only device (sdb1).



----
Cheers,
Lakshmipathi.G

Re: Raid1 won't mount degraded

[Subscription Account]

On Sun, Jan 29, 2017 at 10:57 AM, Lakshmipathi.G
<lakshmipathi.g@gmail.com> wrote:
>>powered off the system added a
>>new disk
>
>>root@NAS:~# btrfs fi show
>>Label: none  uuid: f89c12fb-2304-4dce-bed5-fda008b86bde
>>       Total devices 2 FS bytes used 2.49TiB
>>      devid    1 size 2.73TiB used 2.52TiB path /dev/sdb1
>>     *** Some devices missing
>
> Are you sure 'device add' was successful? Whats the new  device name?
> 'btrfs fi show' shows only device (sdb1).
>

I haven't been able to get to the device add stage since I cannot
mount degraded :(.

>
>
> ----
> Cheers,
> Lakshmipathi.G

Re: Raid1 won't mount degraded

[Adam Borowski]

[-- Attachment #1: Type: text/plain, Size: 903 bytes --]

On Sun, Jan 29, 2017 at 08:12:56AM -0500, Subscription Account wrote:
> I had to remove one disk from raid 1 and I rebooted the system and was
> able to mount in degraded mode. Then I powered off the system added a
> new disk and when I am trying to mount the btrfs filesystem in
> degraded mode it will no longer mount it read-write. I can mount
> read-only though.
> 
> [ 2506.816795] BTRFS: missing devices(1) exceeds the limit(0),
> writeable mount is not allowed
> 
> In the read-only mode I am not able to add a new device or replace :(.
> Please help.

A known problem; you can mount rw degraded only once, if you don't fix the
degradation somehow (by adding a device or converting down), you can't mount
rw again.

If you know how to build a kernel, here's a crude patch.


Meow
-- 
Autotools hint: to do a zx-spectrum build on a pdp11 host, type:
  ./configure --host=zx-spectrum --build=pdp11

[-- Attachment #2: 0001-NOT-FOR-MERGING-btrfs-make-too-many-missing-devices-.patch --]
[-- Type: text/x-diff, Size: 1559 bytes --]

>From 1367d3da6b0189797f6090b11d8716a1cc136593 Mon Sep 17 00:00:00 2001
From: Adam Borowski <kilobyte@angband.pl>
Date: Mon, 23 Jan 2017 19:03:20 +0100
Subject: [PATCH] [NOT-FOR-MERGING] btrfs: make "too many missing devices"
 check non-fatal

It breaks degraded mounts of multi-device filesystems that have any single
blocks, which are naturally created if it has been mounted degraded before.
Obviously, any further device loss will result in data loss, but the user
has already specified -odegraded so that's understood.

For a real fix, we'd want to check whether any of single blocks are missing,
as that would allow telling apart broken JBOD filesystems from bona-fide
degraded RAIDs.

(This patch is for the benefit of folks who'd have to recreate a filesystem
just because it got degraded.)
---
 fs/btrfs/disk-io.c | 3 +--
 1 file changed, 1 insertion(+), 2 deletions(-)

diff --git a/fs/btrfs/disk-io.c b/fs/btrfs/disk-io.c
index 18004169552c..1b25b9e24662 100644
--- a/fs/btrfs/disk-io.c
+++ b/fs/btrfs/disk-io.c
@@ -3060,10 +3060,9 @@ int open_ctree(struct super_block *sb,
 	     fs_info->num_tolerated_disk_barrier_failures &&
 	    !(sb->s_flags & MS_RDONLY)) {
 		btrfs_warn(fs_info,
-"missing devices (%llu) exceeds the limit (%d), writeable mount is not allowed",
+"missing devices (%llu) exceeds the limit (%d), add more or risk data loss",
 			fs_info->fs_devices->missing_devices,
 			fs_info->num_tolerated_disk_barrier_failures);
-		goto fail_sysfs;
 	}
 
 	fs_info->cleaner_kthread = kthread_run(cleaner_kthread, tree_root,
-- 
2.11.0

Re: Raid1 won't mount degraded

[Subscription Account]

On Sun, Jan 29, 2017 at 3:16 PM, Adam Borowski <kilobyte@angband.pl> wrote:
> On Sun, Jan 29, 2017 at 08:12:56AM -0500, Subscription Account wrote:
>> I had to remove one disk from raid 1 and I rebooted the system and was
>> able to mount in degraded mode. Then I powered off the system added a
>> new disk and when I am trying to mount the btrfs filesystem in
>> degraded mode it will no longer mount it read-write. I can mount
>> read-only though.
>>
>> [ 2506.816795] BTRFS: missing devices(1) exceeds the limit(0),
>> writeable mount is not allowed
>>
>> In the read-only mode I am not able to add a new device or replace :(.
>> Please help.
>
> A known problem; you can mount rw degraded only once, if you don't fix the
> degradation somehow (by adding a device or converting down), you can't mount
> rw again.

Uh oh! I wish I know that I only had one shot :(. Just to be clear,
once the rw mount is done first time, and a replace operation or a add
followed by delete missing is executed, irrespective of raid rebuild
status, if the machine reboots, it will still be mounted without
issues and rebuild/balance will continue? Because my concern is that
there could still be blocks with single copy if the balance has not
completed?


>
> If you know how to build a kernel, here's a crude patch.
>
>

I am feeling a little lucky because I still have the other disk and I
am assuming if I remove my current disk and put the other disk in, I
would be able to mount it once again?

Also, since I have couple times already tried btrfs repair etc, can I
trust the current disk anyways?

if I get into issue again, I would definitely use the patch to
recompile the kernel and thanks a lot for the same.

Thanks again,

--
Raj

> Meow
> --
> Autotools hint: to do a zx-spectrum build on a pdp11 host, type:
>   ./configure --host=zx-spectrum --build=pdp11

Re: Raid1 won't mount degraded

[Adam Borowski]

On Sun, Jan 29, 2017 at 08:26:54PM -0500, Subscription Account wrote:
> On Sun, Jan 29, 2017 at 3:16 PM, Adam Borowski <kilobyte@angband.pl> wrote:
> > On Sun, Jan 29, 2017 at 08:12:56AM -0500, Subscription Account wrote:
> >> [ 2506.816795] BTRFS: missing devices(1) exceeds the limit(0),
> >> writeable mount is not allowed
> >>
> >> In the read-only mode I am not able to add a new device or replace :(.
> >> Please help.
> >
> > A known problem; you can mount rw degraded only once, if you don't fix the
> > degradation somehow (by adding a device or converting down), you can't mount
> > rw again.
> 
> Uh oh! I wish I know that I only had one shot :(.

No data is actually lost, it was raid1, it's raid1+single now.  The mount
check is naive and in this case wrong.

> > If you know how to build a kernel, here's a crude patch.
> 
> I am feeling a little lucky because I still have the other disk and I
> am assuming if I remove my current disk and put the other disk in, I
> would be able to mount it once again?

Not sure if that's a good idea if the copies are out of sync.

> Also, since I have couple times already tried btrfs repair etc, can I
> trust the current disk anyways?
> 
> if I get into issue again, I would definitely use the patch to
> recompile the kernel and thanks a lot for the same.

No, the patch doesn't prevent the issue.  It lets you recover -- ie, it
should help in the situation you're in right now.


Meow!
-- 
Autotools hint: to do a zx-spectrum build on a pdp11 host, type:
  ./configure --host=zx-spectrum --build=pdp11

Re: Raid1 won't mount degraded

[Subscription Account]

On Sun, Jan 29, 2017 at 8:49 PM, Adam Borowski <kilobyte@angband.pl> wrote:
> On Sun, Jan 29, 2017 at 08:26:54PM -0500, Subscription Account wrote:
>> On Sun, Jan 29, 2017 at 3:16 PM, Adam Borowski <kilobyte@angband.pl> wrote:
>> > On Sun, Jan 29, 2017 at 08:12:56AM -0500, Subscription Account wrote:
>> >> [ 2506.816795] BTRFS: missing devices(1) exceeds the limit(0),
>> >> writeable mount is not allowed
>> >>
>> >> In the read-only mode I am not able to add a new device or replace :(.
>> >> Please help.
>> >
>> > A known problem; you can mount rw degraded only once, if you don't fix the
>> > degradation somehow (by adding a device or converting down), you can't mount
>> > rw again.
>>
>> Uh oh! I wish I know that I only had one shot :(.
>
> No data is actually lost, it was raid1, it's raid1+single now.  The mount
> check is naive and in this case wrong.
>
>> > If you know how to build a kernel, here's a crude patch.
>>
>> I am feeling a little lucky because I still have the other disk and I
>> am assuming if I remove my current disk and put the other disk in, I
>> would be able to mount it once again?
>
> Not sure if that's a good idea if the copies are out of sync.
>

Not sure I follow, since I will be removing the current disk which may
have some changes, but I don't really care for those changes. When I
put the other disk in, I would be removing the first one so there is
only one disk at one time, it is just that I will go back to a few
days ago status? Am I missing something here?

Thanks,

--
Raj

>> Also, since I have couple times already tried btrfs repair etc, can I
>> trust the current disk anyways?
>>
>> if I get into issue again, I would definitely use the patch to
>> recompile the kernel and thanks a lot for the same.
>
> No, the patch doesn't prevent the issue.  It lets you recover -- ie, it
> should help in the situation you're in right now.
>
>
> Meow!
> --
> Autotools hint: to do a zx-spectrum build on a pdp11 host, type:
>   ./configure --host=zx-spectrum --build=pdp11