unable to mount btrfs after upgrading from 4.16.1 to 4.19.1

unable to mount btrfs after upgrading from 4.16.1 to 4.19.1

[Tomasz Chmielewski]

btrfs sits on md RAID-5:

/dev/md2 /data btrfs noatime,compress-force=zstd,space_cache=v2,noauto 0 
0


After upgrading from kernel 4.16.1 to 4.19.1 and a clean restart, the fs 
no longer mounts:


# mount /data
mount: wrong fs type, bad option, bad superblock on /dev/md2,
        missing codepage or helper program, or other error

        In some cases useful info is found in syslog - try
        dmesg | tail or so.

# dmesg
[  322.877321] BTRFS info (device md2): force zstd compression, level 0
[  322.877326] BTRFS info (device md2): enabling free space tree
[  322.877329] BTRFS info (device md2): using free space tree
[  322.877330] BTRFS info (device md2): has skinny extents
[  367.832058] BTRFS error (device md2): bad tree block start, want 
21120019922944 have 620757027
[  367.832116] BTRFS error (device md2): failed to read block groups: -5
[  367.891339] BTRFS error (device md2): open_ctree failed



The error (bad tree block start) in dmesg changes every time as I re-run 
"mount /data":

[  589.425362] BTRFS error (device md2): bad tree block start, want 
22008094932992 have 3378573292323635748
[  589.425423] BTRFS error (device md2): failed to read block groups: -5
[  589.469979] BTRFS error (device md2): open_ctree failed


[  680.585908] BTRFS error (device md2): bad tree block start, want 
21058406105088 have 18446616720224032488
[  680.585991] BTRFS error (device md2): failed to read block groups: -5
[  680.625419] BTRFS error (device md2): open_ctree failed


Any advice how to recover?


Tomasz Chmielewski
https://lxadm.com

Re: unable to mount btrfs after upgrading from 4.16.1 to 4.19.1

[Roman Mamedov]

On Sat, 10 Nov 2018 03:08:01 +0900
Tomasz Chmielewski <tch@virtall.com> wrote:

> After upgrading from kernel 4.16.1 to 4.19.1 and a clean restart, the fs 
> no longer mounts:

Did you try rebooting back to 4.16.1 to see if it still mounts there?

-- 
With respect,
Roman

Re: unable to mount btrfs after upgrading from 4.16.1 to 4.19.1

[Tomasz Chmielewski]

On 2018-11-10 03:20, Roman Mamedov wrote:
> On Sat, 10 Nov 2018 03:08:01 +0900
> Tomasz Chmielewski <tch@virtall.com> wrote:
> 
>> After upgrading from kernel 4.16.1 to 4.19.1 and a clean restart, the 
>> fs
>> no longer mounts:
> 
> Did you try rebooting back to 4.16.1 to see if it still mounts there?

Yes, just did.

Interestingly, it does mount when I boot back to 4.16.1 - side note - it 
takes some 50 (!) minutes and ~8 GB of reads (according to iostat -m) to 
mount... device size is 16 TB, on HDD.



Tomasz Chmielewski
https://lxadm.com

Re: unable to mount btrfs after upgrading from 4.16.1 to 4.19.1

[Tomasz Chmielewski]

On 2018-11-10 04:15, Tomasz Chmielewski wrote:
> On 2018-11-10 03:20, Roman Mamedov wrote:
>> On Sat, 10 Nov 2018 03:08:01 +0900
>> Tomasz Chmielewski <tch@virtall.com> wrote:
>> 
>>> After upgrading from kernel 4.16.1 to 4.19.1 and a clean restart, the 
>>> fs
>>> no longer mounts:
>> 
>> Did you try rebooting back to 4.16.1 to see if it still mounts there?
> 
> Yes, just did.
> 
> Interestingly, it does mount when I boot back to 4.16.1 - side note -
> it takes some 50 (!) minutes and ~8 GB of reads (according to iostat
> -m) to mount... device size is 16 TB, on HDD.

Also - it did mount with 4.18.17.

Way faster, in some 2 min.


Tomasz Chmielewski
https://lxadm.com

Re: unable to mount btrfs after upgrading from 4.16.1 to 4.19.1

[Tomasz Chmielewski]

On 2018-11-10 04:20, Tomasz Chmielewski wrote:
> On 2018-11-10 04:15, Tomasz Chmielewski wrote:
>> On 2018-11-10 03:20, Roman Mamedov wrote:
>>> On Sat, 10 Nov 2018 03:08:01 +0900
>>> Tomasz Chmielewski <tch@virtall.com> wrote:
>>> 
>>>> After upgrading from kernel 4.16.1 to 4.19.1 and a clean restart, 
>>>> the fs
>>>> no longer mounts:
>>> 
>>> Did you try rebooting back to 4.16.1 to see if it still mounts there?
>> 
>> Yes, just did.
>> 
>> Interestingly, it does mount when I boot back to 4.16.1 - side note -
>> it takes some 50 (!) minutes and ~8 GB of reads (according to iostat
>> -m) to mount... device size is 16 TB, on HDD.
> 
> Also - it did mount with 4.18.17.
> 
> Way faster, in some 2 min.

A few more clean reboot cycles with 4.18.17 and got:

[  113.677829] BTRFS error (device md2): open_ctree failed
[  113.692298] BTRFS info (device md2): force zstd compression, level 0
[  113.692302] BTRFS info (device md2): using free space tree
[  113.692304] BTRFS info (device md2): has skinny extents
[  113.897681] BTRFS error (device md2): super_total_bytes 
17920974913536 mismatch with fs_devices total_rw_bytes 35841949827072
[  113.897751] BTRFS error (device md2): failed to read chunk tree: -22
[  113.935149] BTRFS error (device md2): open_ctree failed


Another "mount /data" (without rebooting) mounted it fine.

Why are btrfs mounts so irregular here?

# btrfs device stats /data
[/dev/md2].write_io_errs    0
[/dev/md2].read_io_errs     0
[/dev/md2].flush_io_errs    0
[/dev/md2].corruption_errs  0
[/dev/md2].generation_errs  0


# btrfs fi usage /data
Overall:
     Device size:                  16.30TiB
     Device allocated:             14.26TiB
     Device unallocated:            2.04TiB
     Device missing:                  0.00B
     Used:                          7.99TiB
     Free (estimated):              8.27TiB      (min: 8.27TiB)
     Data ratio:                       1.00
     Metadata ratio:                   1.00
     Global reserve:              512.00MiB      (used: 0.00B)

Data,single: Size:14.15TiB, Used:7.92TiB
    /dev/md2       14.15TiB

Metadata,single: Size:111.00GiB, Used:74.78GiB
    /dev/md2      111.00GiB

System,single: Size:32.00MiB, Used:1.81MiB
    /dev/md2       32.00MiB

Unallocated:
    /dev/md2        2.04TiB



Tomasz Chmielewski
https://lxadm.com