Re: bad tree block start, want 705757184 have 82362368

[Stephan Olbrich <stephanolbrich@gmx.de>]

Am Sonntag, 18. November 2018, 14:31:36 CET schrieb Anand Jain:
> On 11/18/2018 03:56 PM, Stephan Olbrich wrote:
> > Am Sonntag, 18. November 2018, 01:30:14 CET schrieb Qu Wenruo:
> >>>>> Late on I got the same errors for my /home partition (on the same
> >>>>> drive)
> >>>>> as well. I have snapshots of all partitions on another drive made by
> >>>>> btrbk. To get a working system, I made new (rw) snapshots of the most
> >>>>> recent backup and setup grub and fstab, so my system would boot from
> >>>>> the
> >>>>> other drive. Unfortunately now I got the "bad tree block start" error
> >>>>> again at least once in dmesg but I didn't save it and it's not in
> >>>>> syslog
> >>>>> 
> >>>>> :-( What I remember is, that it was followed by other btrfs error
> >>>>> 
> >>>>> messages saying something about correcting something. And the
> >>>>> filesystem
> >>>>> was still read/write this time.
> >>>>> At the moment I can't reproduce it.
> > 
> > Today it happend again (sdb is my backup drive, which is my main drive at
> > the moment): [  286.325857] BTRFS error (device sdb1): bad tree block
> > start, want 787719208960 have 11268016545161247416 [  286.363245] BTRFS
> > info (device sdb1): read error corrected: ino 0 off 787719208960 (dev
> > /dev/sdb1 sector 1243815072) [  286.364087] BTRFS info (device sdb1):
> > read error corrected: ino 0 off 787719213056 (dev /dev/sdb1 sector
> > 1243815080) [  286.425946] BTRFS info (device sdb1): read error
> > corrected: ino 0 off 787719217152 (dev /dev/sdb1 sector 1243815088) [ 
> > 286.427530] BTRFS info (device sdb1): read error corrected: ino 0 off
> > 787719221248 (dev /dev/sdb1 sector 1243815096)
>   Was there any hardware issues? How about the following data from the
> system..
> 
>    btrfs fi df <mnt>
>    btrfs dev stat <mnt>

I didn't have any hardware issues (that I know of).

$ btrfs fi df /
Data, single: total=841.00GiB, used=791.92GiB                                                                                                                                                              
System, DUP: total=8.00MiB, used=112.00KiB                                                                                                                                                                 
System, single: total=4.00MiB, used=0.00B                                                                                                                                                                  
Metadata, DUP: total=10.50GiB, used=8.44GiB                                                                                                                                                                
Metadata, single: total=8.00MiB, used=0.00B                                                                                                                                                                
GlobalReserve, single: total=512.00MiB, used=0.00B

$ btrfs dev stat /                                                                                                                                                             
[/dev/sdb1].write_io_errs    0                                                                                                                                                                             
[/dev/sdb1].read_io_errs     0                                                                                                                                                                             
[/dev/sdb1].flush_io_errs    0                                                                                                                                                                             
[/dev/sdb1].corruption_errs  0                                                                                                                                                                             
[/dev/sdb1].generation_errs  0 

Also the numbers in the error messages keep changing:
[ 1577.302959] BTRFS error (device sdb1): bad tree block start, want 788086226944 have 411602213549769407
[ 1577.369083] BTRFS info (device sdb1): read error corrected: ino 0 off 788086226944 (dev /dev/sdb1 sector 1244531904)
[ 1577.428139] BTRFS info (device sdb1): read error corrected: ino 0 off 788086231040 (dev /dev/sdb1 sector 1244531912)
[ 1577.429091] BTRFS info (device sdb1): read error corrected: ino 0 off 788086235136 (dev /dev/sdb1 sector 1244531920)
[ 1577.478249] BTRFS info (device sdb1): read error corrected: ino 0 off 788086239232 (dev /dev/sdb1 sector 1244531928)

Thanks,
Stephan

> Thanks, Anand
> 
> >>>>> Is there any way to find out, which files are affected by the errors
> >>>>> above?
> >>>> 
> >>>> No files are affected, but an essential tree, extent tree, is
> >>>> corrupted.
> >>>> 
> >>>> Normally this may prevent RW mount, and even it mounts it can still
> >>>> cause problem when doing any write.
> >>>> It could even prevent RO mount if the corrupted leaf contains block
> >>>> group item.
> >>>> 
> >>>> But your data should be OK if there is no other corruption, and in that
> >>>> case btrfs-restore should work well.
> >>>> 
> >>>>> I don't really trust the data on the drive I'm using at the
> >>>>> moment, as it has shown errors as well, but I have a less current
> >>>>> backup
> >>>>> on yet another drive but at it is a few weeks old, I don't want to use
> >>>>> it
> >>>>> to setup the system on the SSD again, but just copy the relevant files
> >>>>> if
> >>>>> possible. Or is it possible to repair the original file system?
> >>>> 
> >>>> At least we need "btrfs check" output.
> >>> 
> >>> I updated btrfs-progs and run btrfs check for / and /home
> >>> No errors are found on / (sda2), but there are errors on /home ??
> >>> 
> >>> $ btrfs --version
> >>> btrfs-progs v4.19
> >>> 
> >>> $ btrfs check /dev/sda2
> >>> Opening filesystem to check...
> >>> Checking filesystem on /dev/sda2
> >>> UUID: 80368989-ffa8-463c-98fb-fe2e28ca7bf3
> >>> [1/7] checking root items
> >>> [2/7] checking extents
> >>> [3/7] checking free space cache
> >>> [4/7] checking fs roots
> >>> [5/7] checking only csums items (without verifying data)
> >>> [6/7] checking root refs
> >>> [7/7] checking quota groups skipped (not enabled on this FS)
> >>> found 64816218112 bytes used, no error found
> >>> total csum bytes: 59518732
> >>> total tree bytes: 2180268032
> >>> total fs tree bytes: 1965965312
> >>> total extent tree bytes: 123289600
> >>> btree space waste bytes: 478665777
> >>> file data blocks allocated: 151083261952
> >>> 
> >>>   referenced 76879990784
> >> 
> >> This fs is completely fine, including your data.
> >> 
> >>> $ btrfs check /dev/sda4
> >>> Opening filesystem to check...
> >>> Checking filesystem on /dev/sda4
> >>> UUID: 81c38df8-b7f9-412c-8c88-cfde8db68eb1
> >>> [1/7] checking root items
> >>> [2/7] checking extents
> >>> [3/7] checking free space cache
> >>> [4/7] checking fs roots
> >>> root 257 inode 7970563 errors 100, file extent discount
> >>> 
> >>> Found file extent holes:
> >>>          start: 0, len: 20480
> >>> 
> >>> root 257 inode 7970564 errors 100, file extent discount
> >>> 
> >>> Found file extent holes:
> >>>          start: 0, len: 77824
> >>> 
> >>> ERROR: errors found in fs roots
> >> 
> >> These are just minor errors, won't even causing any data mismatch.
> >> 
> >> So all your fses should be mostly OK.
> >> 
> >> Would you please try to use v4.19-rc* to see if it changes anything?
> > 
> > v4.19-rc1 is the only rc I found, but that is older than v4.19, right?
> > Anyway, here is the output:
> > 
> > $ btrfs --version
> > btrfs-progs v4.19-rc1
> > 
> > $ btrfs check /dev/sda2
> > Opening filesystem to check...
> > Checking filesystem on /dev/sda2
> > UUID: 80368989-ffa8-463c-98fb-fe2e28ca7bf3
> > [1/7] checking root items
> > [2/7] checking extents
> > [3/7] checking free space cache
> > [4/7] checking fs roots
> > [5/7] checking only csums items (without verifying data)
> > [6/7] checking root refs
> > [7/7] checking quota groups skipped (not enabled on this FS)
> > found 64816218112 bytes used, no error found
> > total csum bytes: 59518732
> > total tree bytes: 2180268032
> > total fs tree bytes: 1965965312
> > total extent tree bytes: 123289600
> > btree space waste bytes: 478665777
> > file data blocks allocated: 151083261952
> > 
> >   referenced 76879990784
> > 
> > $ btrfs check /dev/sda4
> > Opening filesystem to check...
> > Checking filesystem on /dev/sda4
> > UUID: 81c38df8-b7f9-412c-8c88-cfde8db68eb1
> > [1/7] checking root items
> > [2/7] checking extents
> > [3/7] checking free space cache
> > [4/7] checking fs roots
> > root 257 inode 7970563 errors 100, file extent discount
> > 
> > Found file extent holes:
> >          start: 0, len: 20480
> > 
> > root 257 inode 7970564 errors 100, file extent discount
> > 
> > Found file extent holes:
> >          start: 0, len: 77824
> > 
> > ERROR: errors found in fs roots
> > found 303386652672 bytes used, error(s) found
> > total csum bytes: 289501272
> > total tree bytes: 2336227328
> > total fs tree bytes: 1766473728
> > total extent tree bytes: 202014720
> > btree space waste bytes: 519245278
> > file data blocks allocated: 6851730792448
> > 
> >   referenced 533348069376
> > 
> > Thanks,
> > Stephan
> > 
> >> Thanks,
> >> Qu
> >> 
> >>> found 303386652672 bytes used, error(s) found
> >>> total csum bytes: 289501272
> >>> total tree bytes: 2336227328
> >>> total fs tree bytes: 1766473728
> >>> total extent tree bytes: 202014720
> >>> btree space waste bytes: 519245278
> >>> file data blocks allocated: 6851730792448
> >>> 
> >>>   referenced 533348069376
> >>> 
> >>> Thanks,
> >>> Stephan
> >>> 
> >>>>> Some information about my system:
> >>>>> Kubuntu 18.04
> >>>>> Kernel 4.19.1 when the problem occured, now 4.19.2
> >>>>> btrfs-tools 4.15.1
> >>>> 
> >>>> And "btrfs check" should be executed using latest version.
> >>>> 
> >>>> Thanks,
> >>>> Qu
> >>>> 
> >>>>> Regards,
> >>>>> Stephan