btrfs-progs 4.7, check reports many "incorrect local backref count" messages

btrfs-progs 4.7, check reports many "incorrect local backref count" messages

[Chris Murphy]

[-- Attachment #1: Type: text/plain, Size: 776 bytes --]

See attached for 'btrfs check' output.

The last check was about two weeks ago when btrfs-progs 4.7 was
released, and it came up clean. The file system has only been written
to since then with a 4.7.0 kernel, and 4.7.2 kernel for the past ~36
hours.

The workload in those two weeks has only been curl downloads of OS
ISOs. No snapshotting, or reflinks. Only clean umounts. It is a single
device volume, on dmcrypt, on a Toshiba hard drive. Metadata is DUP.
It mounts without error, so it seems to be just btrfs check noise.

If it's not just noise, or bogus messages, then there's something in
4.7.x that's causing problems silently in normal operation. But it
looks bogus

It always complains about the same backref number, but a different owner number.



-- 
Chris Murphy

[-- Attachment #2: btrfs4.7incorrectbackref.txt --]
[-- Type: text/plain, Size: 43987 bytes --]


This is a crypto LUKS volume.
cat /proc/mounts
/dev/mapper/brick1 /brick1 btrfs rw,seclabel,noatime,space_cache,subvolid=659,subvol=/everything-new 0 0


[chris@f24s ~]$ sudo btrfs check /dev/mapper/brick1 
Checking filesystem on /dev/mapper/brick1
UUID: 30f4724a-9a9f-4a5d-a37d-3e53876bf2e1
checking extents
Backref 16913485824 root 659 owner 145055 offset 0 num_refs 0 not found in extent tree
Incorrect local backref count on 16913485824 root 659 owner 145055 offset 0 found 1 wanted 0 back 0x565421c9afb0
Incorrect local backref count on 16913485824 root 659 owner 145055 offset 0 found 0 wanted 1 back 0x565421d544c0
Backref disk bytenr does not match extent record, bytenr=16913485824, ref bytenr=14100377632768
Backref bytes do not match extent backref, bytenr=16913485824, ref bytes=7577600, backref bytes=4096
Backref 16913485824 root 659 owner 144113 offset 0 num_refs 0 not found in extent tree
Incorrect local backref count on 16913485824 root 659 owner 144113 offset 0 found 1 wanted 0 back 0x565421c9ad50
Incorrect local backref count on 16913485824 root 659 owner 144113 offset 0 found 0 wanted 1 back 0x565421d54a00
Backref disk bytenr does not match extent record, bytenr=16913485824, ref bytenr=7168248739989422080
Backref bytes do not match extent backref, bytenr=16913485824, ref bytes=7577600, backref bytes=4096
Backref 16913485824 root 659 owner 144112 offset 0 num_refs 0 not found in extent tree
Incorrect local backref count on 16913485824 root 659 owner 144112 offset 0 found 1 wanted 0 back 0x565421c940d0
Incorrect local backref count on 16913485824 root 659 owner 144112 offset 0 found 0 wanted 1 back 0x565421d54f40
Backref disk bytenr does not match extent record, bytenr=16913485824, ref bytenr=0
Backref bytes do not match extent backref, bytenr=16913485824, ref bytes=7577600, backref bytes=4096
Backref 16913485824 root 659 owner 144111 offset 0 num_refs 0 not found in extent tree
Incorrect local backref count on 16913485824 root 659 owner 144111 offset 0 found 1 wanted 0 back 0x565421c93fa0
Incorrect local backref count on 16913485824 root 659 owner 144111 offset 0 found 0 wanted 1 back 0x565421d55480
Backref disk bytenr does not match extent record, bytenr=16913485824, ref bytenr=0
Backref bytes do not match extent backref, bytenr=16913485824, ref bytes=7577600, backref bytes=4096
Backref 16913485824 root 659 owner 144110 offset 0 num_refs 0 not found in extent tree
Incorrect local backref count on 16913485824 root 659 owner 144110 offset 0 found 1 wanted 0 back 0x565421c93e70
Incorrect local backref count on 16913485824 root 659 owner 144110 offset 0 found 0 wanted 1 back 0x565421d54520
Backref disk bytenr does not match extent record, bytenr=16913485824, ref bytenr=7166173136914087936
Backref bytes do not match extent backref, bytenr=16913485824, ref bytes=7577600, backref bytes=4096
Backref 16913485824 root 659 owner 144109 offset 0 num_refs 0 not found in extent tree
Incorrect local backref count on 16913485824 root 659 owner 144109 offset 0 found 1 wanted 0 back 0x565421c93d40
Incorrect local backref count on 16913485824 root 659 owner 144109 offset 0 found 0 wanted 1 back 0x565421d543a0
Backref disk bytenr does not match extent record, bytenr=16913485824, ref bytenr=33554432
Backref bytes do not match extent backref, bytenr=16913485824, ref bytes=7577600, backref bytes=4096
Backref 16913485824 root 659 owner 144108 offset 0 num_refs 0 not found in extent tree
Incorrect local backref count on 16913485824 root 659 owner 144108 offset 0 found 1 wanted 0 back 0x565421c93c10
Incorrect local backref count on 16913485824 root 659 owner 144108 offset 0 found 0 wanted 1 back 0x565421d555a0
Backref disk bytenr does not match extent record, bytenr=16913485824, ref bytenr=0
Backref bytes do not match extent backref, bytenr=16913485824, ref bytes=7577600, backref bytes=4096
Backref 16913485824 root 659 owner 144107 offset 0 num_refs 0 not found in extent tree
Incorrect local backref count on 16913485824 root 659 owner 144107 offset 0 found 1 wanted 0 back 0x565421c93ae0
Incorrect local backref count on 16913485824 root 659 owner 144107 offset 0 found 0 wanted 1 back 0x565421d546a0
Backref disk bytenr does not match extent record, bytenr=16913485824, ref bytenr=65536
Backref bytes do not match extent backref, bytenr=16913485824, ref bytes=7577600, backref bytes=4096
Backref 16913485824 root 659 owner 144106 offset 0 num_refs 0 not found in extent tree
Incorrect local backref count on 16913485824 root 659 owner 144106 offset 0 found 1 wanted 0 back 0x565421c9ac20
Incorrect local backref count on 16913485824 root 659 owner 144106 offset 0 found 0 wanted 1 back 0x565421d55300
Backref disk bytenr does not match extent record, bytenr=16913485824, ref bytenr=0
Backref bytes do not match extent backref, bytenr=16913485824, ref bytes=7577600, backref bytes=4096
Backref 16913485824 root 659 owner 144105 offset 0 num_refs 0 not found in extent tree
Incorrect local backref count on 16913485824 root 659 owner 144105 offset 0 found 1 wanted 0 back 0x565421c9aaf0
Incorrect local backref count on 16913485824 root 659 owner 144105 offset 0 found 0 wanted 1 back 0x565421d55180
Backref disk bytenr does not match extent record, bytenr=16913485824, ref bytenr=0
Backref bytes do not match extent backref, bytenr=16913485824, ref bytes=7577600, backref bytes=4096
Backref 16913485824 root 659 owner 144104 offset 0 num_refs 0 not found in extent tree
Incorrect local backref count on 16913485824 root 659 owner 144104 offset 0 found 1 wanted 0 back 0x565421c9a9c0
Incorrect local backref count on 16913485824 root 659 owner 144104 offset 0 found 0 wanted 1 back 0x565421d547c0
Backref disk bytenr does not match extent record, bytenr=16913485824, ref bytenr=128974848
Backref bytes do not match extent backref, bytenr=16913485824, ref bytes=7577600, backref bytes=4096
Backref 16913485824 root 659 owner 144103 offset 0 num_refs 0 not found in extent tree
Incorrect local backref count on 16913485824 root 659 owner 144103 offset 0 found 1 wanted 0 back 0x565421cc9180
Incorrect local backref count on 16913485824 root 659 owner 144103 offset 0 found 0 wanted 1 back 0x565421d54b20
Backref disk bytenr does not match extent record, bytenr=16913485824, ref bytenr=128974848
Backref bytes do not match extent backref, bytenr=16913485824, ref bytes=7577600, backref bytes=4096
Backref 16913485824 root 659 owner 143839 offset 1048576 num_refs 0 not found in extent tree
Incorrect local backref count on 16913485824 root 659 owner 143839 offset 1048576 found 1 wanted 0 back 0x565424bd93e0
Incorrect local backref count on 16913485824 root 659 owner 143839 offset 1048576 found 0 wanted 1 back 0x565421d55120
Backref disk bytenr does not match extent record, bytenr=16913485824, ref bytenr=0
Backref bytes do not match extent backref, bytenr=16913485824, ref bytes=7577600, backref bytes=4096
Backref 16913485824 root 659 owner 143838 offset 1048576 num_refs 0 not found in extent tree
Incorrect local backref count on 16913485824 root 659 owner 143838 offset 1048576 found 1 wanted 0 back 0x565424bc2620
Incorrect local backref count on 16913485824 root 659 owner 143838 offset 1048576 found 0 wanted 1 back 0x565421d54940
Backref disk bytenr does not match extent record, bytenr=16913485824, ref bytenr=426848714752
Backref bytes do not match extent backref, bytenr=16913485824, ref bytes=7577600, backref bytes=4096
Backref 16913485824 root 659 owner 143821 offset 0 num_refs 0 not found in extent tree
Incorrect local backref count on 16913485824 root 659 owner 143821 offset 0 found 1 wanted 0 back 0x565421d30c40
Incorrect local backref count on 16913485824 root 659 owner 143821 offset 0 found 0 wanted 1 back 0x565421d54dc0
Backref disk bytenr does not match extent record, bytenr=16913485824, ref bytenr=0
Backref bytes do not match extent backref, bytenr=16913485824, ref bytes=7577600, backref bytes=4096
Backref 16913485824 root 659 owner 143820 offset 0 num_refs 0 not found in extent tree
Incorrect local backref count on 16913485824 root 659 owner 143820 offset 0 found 1 wanted 0 back 0x565421cc9240
Incorrect local backref count on 16913485824 root 659 owner 143820 offset 0 found 0 wanted 1 back 0x565421d54c40
Backref disk bytenr does not match extent record, bytenr=16913485824, ref bytenr=4611886129543668612
Backref bytes do not match extent backref, bytenr=16913485824, ref bytes=7577600, backref bytes=4096
Backref 16913485824 root 659 owner 143819 offset 0 num_refs 0 not found in extent tree
Incorrect local backref count on 16913485824 root 659 owner 143819 offset 0 found 1 wanted 0 back 0x565421cc9ae0
Incorrect local backref count on 16913485824 root 659 owner 143819 offset 0 found 0 wanted 1 back 0x565421d55000
Backref disk bytenr does not match extent record, bytenr=16913485824, ref bytenr=0
Backref bytes do not match extent backref, bytenr=16913485824, ref bytes=7577600, backref bytes=4096
Backref 16913485824 root 659 owner 143818 offset 0 num_refs 0 not found in extent tree
Incorrect local backref count on 16913485824 root 659 owner 143818 offset 0 found 1 wanted 0 back 0x565421d3ba20
Incorrect local backref count on 16913485824 root 659 owner 143818 offset 0 found 0 wanted 1 back 0x565421d54ac0
Backref disk bytenr does not match extent record, bytenr=16913485824, ref bytenr=7
Backref bytes do not match extent backref, bytenr=16913485824, ref bytes=7577600, backref bytes=4096
Backref 16913485824 root 659 owner 143817 offset 0 num_refs 0 not found in extent tree
Incorrect local backref count on 16913485824 root 659 owner 143817 offset 0 found 1 wanted 0 back 0x565421d38fc0
Incorrect local backref count on 16913485824 root 659 owner 143817 offset 0 found 0 wanted 1 back 0x565421d548e0
Backref disk bytenr does not match extent record, bytenr=16913485824, ref bytenr=9223572147971056522
Backref bytes do not match extent backref, bytenr=16913485824, ref bytes=7577600, backref bytes=4096
Backref 16913485824 root 659 owner 143244 offset 0 num_refs 0 not found in extent tree
Incorrect local backref count on 16913485824 root 659 owner 143244 offset 0 found 1 wanted 0 back 0x565421d3ad20
Incorrect local backref count on 16913485824 root 659 owner 143244 offset 0 found 0 wanted 1 back 0x565421d545e0
Backref disk bytenr does not match extent record, bytenr=16913485824, ref bytenr=7537667604480
Backref bytes do not match extent backref, bytenr=16913485824, ref bytes=7577600, backref bytes=4096
Backref 16913485824 root 659 owner 143243 offset 0 num_refs 0 not found in extent tree
Incorrect local backref count on 16913485824 root 659 owner 143243 offset 0 found 1 wanted 0 back 0x565421d385c0
Incorrect local backref count on 16913485824 root 659 owner 143243 offset 0 found 0 wanted 1 back 0x565421d55540
Backref disk bytenr does not match extent record, bytenr=16913485824, ref bytenr=0
Backref bytes do not match extent backref, bytenr=16913485824, ref bytes=7577600, backref bytes=4096
Backref 16913485824 root 659 owner 143242 offset 0 num_refs 0 not found in extent tree
Incorrect local backref count on 16913485824 root 659 owner 143242 offset 0 found 1 wanted 0 back 0x565424bbac60
Incorrect local backref count on 16913485824 root 659 owner 143242 offset 0 found 0 wanted 1 back 0x565421d542e0
Backref disk bytenr does not match extent record, bytenr=16913485824, ref bytenr=2163838883463168
Backref bytes do not match extent backref, bytenr=16913485824, ref bytes=7577600, backref bytes=4096
Backref 16913485824 root 659 owner 143241 offset 0 num_refs 0 not found in extent tree
Incorrect local backref count on 16913485824 root 659 owner 143241 offset 0 found 1 wanted 0 back 0x565421d38ee0
Incorrect local backref count on 16913485824 root 659 owner 143241 offset 0 found 0 wanted 1 back 0x565421d54460
Backref disk bytenr does not match extent record, bytenr=16913485824, ref bytenr=450048
Backref bytes do not match extent backref, bytenr=16913485824, ref bytes=7577600, backref bytes=4096
Backref 16913485824 root 659 owner 143240 offset 0 num_refs 0 not found in extent tree
Incorrect local backref count on 16913485824 root 659 owner 143240 offset 0 found 1 wanted 0 back 0x565424bd1a20
Incorrect local backref count on 16913485824 root 659 owner 143240 offset 0 found 0 wanted 1 back 0x565421d553c0
Backref disk bytenr does not match extent record, bytenr=16913485824, ref bytenr=0
Backref bytes do not match extent backref, bytenr=16913485824, ref bytes=7577600, backref bytes=4096
Backref 16913485824 root 659 owner 143239 offset 0 num_refs 0 not found in extent tree
Incorrect local backref count on 16913485824 root 659 owner 143239 offset 0 found 1 wanted 0 back 0x565424bc9fe0
Incorrect local backref count on 16913485824 root 659 owner 143239 offset 0 found 0 wanted 1 back 0x565421d550c0
Backref disk bytenr does not match extent record, bytenr=16913485824, ref bytenr=0
Backref bytes do not match extent backref, bytenr=16913485824, ref bytes=7577600, backref bytes=4096
Backref 16913485824 root 659 owner 143238 offset 0 num_refs 0 not found in extent tree
Incorrect local backref count on 16913485824 root 659 owner 143238 offset 0 found 1 wanted 0 back 0x56542248ecd0
Incorrect local backref count on 16913485824 root 659 owner 143238 offset 0 found 0 wanted 1 back 0x565421d54880
Backref disk bytenr does not match extent record, bytenr=16913485824, ref bytenr=2
Backref bytes do not match extent backref, bytenr=16913485824, ref bytes=7577600, backref bytes=4096
Backref 16913485824 root 659 owner 143237 offset 0 num_refs 0 not found in extent tree
Incorrect local backref count on 16913485824 root 659 owner 143237 offset 0 found 1 wanted 0 back 0x565421cf3710
Incorrect local backref count on 16913485824 root 659 owner 143237 offset 0 found 0 wanted 1 back 0x565421d54a60
Backref disk bytenr does not match extent record, bytenr=16913485824, ref bytenr=1099511653218
Backref bytes do not match extent backref, bytenr=16913485824, ref bytes=7577600, backref bytes=4096
Backref 16913485824 root 659 owner 143236 offset 0 num_refs 0 not found in extent tree
Incorrect local backref count on 16913485824 root 659 owner 143236 offset 0 found 1 wanted 0 back 0x565421ceefc0
Incorrect local backref count on 16913485824 root 659 owner 143236 offset 0 found 0 wanted 1 back 0x565421d54fa0
Backref disk bytenr does not match extent record, bytenr=16913485824, ref bytenr=0
Backref bytes do not match extent backref, bytenr=16913485824, ref bytes=7577600, backref bytes=4096
Backref 16913485824 root 659 owner 143235 offset 0 num_refs 0 not found in extent tree
Incorrect local backref count on 16913485824 root 659 owner 143235 offset 0 found 1 wanted 0 back 0x565421cee010
Incorrect local backref count on 16913485824 root 659 owner 143235 offset 0 found 0 wanted 1 back 0x565421d54ca0
Backref disk bytenr does not match extent record, bytenr=16913485824, ref bytenr=426848649216
Backref bytes do not match extent backref, bytenr=16913485824, ref bytes=7577600, backref bytes=4096
Backref 16913485824 root 659 owner 143234 offset 0 num_refs 0 not found in extent tree
Incorrect local backref count on 16913485824 root 659 owner 143234 offset 0 found 1 wanted 0 back 0x565421cedf40
Incorrect local backref count on 16913485824 root 659 owner 143234 offset 0 found 0 wanted 1 back 0x565421d54e20
Backref disk bytenr does not match extent record, bytenr=16913485824, ref bytenr=0
Backref bytes do not match extent backref, bytenr=16913485824, ref bytes=7577600, backref bytes=4096
Backref 16913485824 root 659 owner 143233 offset 0 num_refs 0 not found in extent tree
Incorrect local backref count on 16913485824 root 659 owner 143233 offset 0 found 1 wanted 0 back 0x565421cf6b90
Incorrect local backref count on 16913485824 root 659 owner 143233 offset 0 found 0 wanted 1 back 0x565421d54e80
Backref disk bytenr does not match extent record, bytenr=16913485824, ref bytenr=0
Backref bytes do not match extent backref, bytenr=16913485824, ref bytes=7577600, backref bytes=4096
Backref 16913485824 root 659 owner 143232 offset 0 num_refs 0 not found in extent tree
Incorrect local backref count on 16913485824 root 659 owner 143232 offset 0 found 1 wanted 0 back 0x565421cad580
Incorrect local backref count on 16913485824 root 659 owner 143232 offset 0 found 0 wanted 1 back 0x565421d54b80
Backref disk bytenr does not match extent record, bytenr=16913485824, ref bytenr=2163838883463168
Backref bytes do not match extent backref, bytenr=16913485824, ref bytes=7577600, backref bytes=4096
Backref 16913485824 root 659 owner 143231 offset 0 num_refs 0 not found in extent tree
Incorrect local backref count on 16913485824 root 659 owner 143231 offset 0 found 1 wanted 0 back 0x565421cb0cf0
Incorrect local backref count on 16913485824 root 659 owner 143231 offset 0 found 0 wanted 1 back 0x565421d55360
Backref disk bytenr does not match extent record, bytenr=16913485824, ref bytenr=0
Backref bytes do not match extent backref, bytenr=16913485824, ref bytes=7577600, backref bytes=4096
Backref 16913485824 root 659 owner 143230 offset 0 num_refs 0 not found in extent tree
Incorrect local backref count on 16913485824 root 659 owner 143230 offset 0 found 1 wanted 0 back 0x565421cb0bc0
Incorrect local backref count on 16913485824 root 659 owner 143230 offset 0 found 0 wanted 1 back 0x565421d54400
Backref disk bytenr does not match extent record, bytenr=16913485824, ref bytenr=562949953421312
Backref bytes do not match extent backref, bytenr=16913485824, ref bytes=7577600, backref bytes=4096
Backref 16913485824 root 659 owner 143229 offset 0 num_refs 0 not found in extent tree
Incorrect local backref count on 16913485824 root 659 owner 143229 offset 0 found 1 wanted 0 back 0x565421caaf80
Incorrect local backref count on 16913485824 root 659 owner 143229 offset 0 found 0 wanted 1 back 0x565421d54280
Backref disk bytenr does not match extent record, bytenr=16913485824, ref bytenr=128974848
Backref bytes do not match extent backref, bytenr=16913485824, ref bytes=7577600, backref bytes=4096
Backref 16913485824 root 659 owner 143228 offset 0 num_refs 0 not found in extent tree
Incorrect local backref count on 16913485824 root 659 owner 143228 offset 0 found 1 wanted 0 back 0x565421ca96f0
Incorrect local backref count on 16913485824 root 659 owner 143228 offset 0 found 0 wanted 1 back 0x565421d554e0
Backref disk bytenr does not match extent record, bytenr=16913485824, ref bytenr=0
Backref bytes do not match extent backref, bytenr=16913485824, ref bytes=7577600, backref bytes=4096
Backref 16913485824 root 659 owner 143227 offset 0 num_refs 0 not found in extent tree
Incorrect local backref count on 16913485824 root 659 owner 143227 offset 0 found 1 wanted 0 back 0x565421ca9bb0
Incorrect local backref count on 16913485824 root 659 owner 143227 offset 0 found 0 wanted 1 back 0x565421d54640
Backref disk bytenr does not match extent record, bytenr=16913485824, ref bytenr=15996785876420001792
Backref bytes do not match extent backref, bytenr=16913485824, ref bytes=7577600, backref bytes=4096
Backref 16913485824 root 659 owner 143226 offset 0 num_refs 0 not found in extent tree
Incorrect local backref count on 16913485824 root 659 owner 143226 offset 0 found 1 wanted 0 back 0x565421ca8c40
Incorrect local backref count on 16913485824 root 659 owner 143226 offset 0 found 0 wanted 1 back 0x565421d55240
Backref disk bytenr does not match extent record, bytenr=16913485824, ref bytenr=0
Backref bytes do not match extent backref, bytenr=16913485824, ref bytes=7577600, backref bytes=4096
Backref 16913485824 root 659 owner 142896 offset 0 num_refs 0 not found in extent tree
Incorrect local backref count on 16913485824 root 659 owner 142896 offset 0 found 1 wanted 0 back 0x565421caffe0
Incorrect local backref count on 16913485824 root 659 owner 142896 offset 0 found 0 wanted 1 back 0x565421d54d00
Backref disk bytenr does not match extent record, bytenr=16913485824, ref bytenr=200111116280709
Backref bytes do not match extent backref, bytenr=16913485824, ref bytes=7577600, backref bytes=4096
Backref 16913485824 root 659 owner 142895 offset 0 num_refs 0 not found in extent tree
Incorrect local backref count on 16913485824 root 659 owner 142895 offset 0 found 1 wanted 0 back 0x565421d38c40
Incorrect local backref count on 16913485824 root 659 owner 142895 offset 0 found 0 wanted 1 back 0x565421d551e0
Backref disk bytenr does not match extent record, bytenr=16913485824, ref bytenr=0
Backref bytes do not match extent backref, bytenr=16913485824, ref bytes=7577600, backref bytes=4096
Backref 16913485824 root 659 owner 142894 offset 0 num_refs 0 not found in extent tree
Incorrect local backref count on 16913485824 root 659 owner 142894 offset 0 found 1 wanted 0 back 0x565421ccac80
Incorrect local backref count on 16913485824 root 659 owner 142894 offset 0 found 0 wanted 1 back 0x565421d54760
Backref disk bytenr does not match extent record, bytenr=16913485824, ref bytenr=7
Backref bytes do not match extent backref, bytenr=16913485824, ref bytes=7577600, backref bytes=4096
Backref 16913485824 root 659 owner 142893 offset 0 num_refs 0 not found in extent tree
Incorrect local backref count on 16913485824 root 659 owner 142893 offset 0 found 1 wanted 0 back 0x565421ccb640
Incorrect local backref count on 16913485824 root 659 owner 142893 offset 0 found 0 wanted 1 back 0x565421d54700
Backref disk bytenr does not match extent record, bytenr=16913485824, ref bytenr=1099511627776
Backref bytes do not match extent backref, bytenr=16913485824, ref bytes=7577600, backref bytes=4096
Backref 16913485824 root 659 owner 142892 offset 0 num_refs 0 not found in extent tree
Incorrect local backref count on 16913485824 root 659 owner 142892 offset 0 found 1 wanted 0 back 0x565421ccb2c0
Incorrect local backref count on 16913485824 root 659 owner 142892 offset 0 found 0 wanted 1 back 0x565421d552a0
Backref disk bytenr does not match extent record, bytenr=16913485824, ref bytenr=0
Backref bytes do not match extent backref, bytenr=16913485824, ref bytes=7577600, backref bytes=4096
Backref 16913485824 root 659 owner 142891 offset 0 num_refs 0 not found in extent tree
Incorrect local backref count on 16913485824 root 659 owner 142891 offset 0 found 1 wanted 0 back 0x565421d391c0
Incorrect local backref count on 16913485824 root 659 owner 142891 offset 0 found 0 wanted 1 back 0x565421d54340
Backref disk bytenr does not match extent record, bytenr=16913485824, ref bytenr=2
Backref bytes do not match extent backref, bytenr=16913485824, ref bytes=7577600, backref bytes=4096
Backref 16913485824 root 659 owner 142890 offset 0 num_refs 0 not found in extent tree
Incorrect local backref count on 16913485824 root 659 owner 142890 offset 0 found 1 wanted 0 back 0x565421cc9760
Incorrect local backref count on 16913485824 root 659 owner 142890 offset 0 found 0 wanted 1 back 0x565421d55600
Backref disk bytenr does not match extent record, bytenr=16913485824, ref bytenr=0
Backref bytes do not match extent backref, bytenr=16913485824, ref bytes=7577600, backref bytes=4096
Backref 16913485824 root 659 owner 142889 offset 0 num_refs 0 not found in extent tree
Incorrect local backref count on 16913485824 root 659 owner 142889 offset 0 found 1 wanted 0 back 0x565421d392c0
Incorrect local backref count on 16913485824 root 659 owner 142889 offset 0 found 0 wanted 1 back 0x565421d55420
Backref disk bytenr does not match extent record, bytenr=16913485824, ref bytenr=0
Backref bytes do not match extent backref, bytenr=16913485824, ref bytes=7577600, backref bytes=4096
Backref 16913485824 root 659 owner 142888 offset 0 num_refs 0 not found in extent tree
Incorrect local backref count on 16913485824 root 659 owner 142888 offset 0 found 1 wanted 0 back 0x565421d39c80
Incorrect local backref count on 16913485824 root 659 owner 142888 offset 0 found 0 wanted 1 back 0x565421d54580
Backref disk bytenr does not match extent record, bytenr=16913485824, ref bytenr=450048
Backref bytes do not match extent backref, bytenr=16913485824, ref bytes=7577600, backref bytes=4096
Backref 16913485824 root 659 owner 142887 offset 0 num_refs 0 not found in extent tree
Incorrect local backref count on 16913485824 root 659 owner 142887 offset 0 found 1 wanted 0 back 0x565421ccada0
Incorrect local backref count on 16913485824 root 659 owner 142887 offset 0 found 0 wanted 1 back 0x565421d54820
Backref disk bytenr does not match extent record, bytenr=16913485824, ref bytenr=2163838883463168
Backref bytes do not match extent backref, bytenr=16913485824, ref bytes=7577600, backref bytes=4096
Backref 16913485824 root 659 owner 142886 offset 0 num_refs 0 not found in extent tree
Incorrect local backref count on 16913485824 root 659 owner 142886 offset 0 found 1 wanted 0 back 0x565421d3b2c0
Incorrect local backref count on 16913485824 root 659 owner 142886 offset 0 found 0 wanted 1 back 0x565421d55060
Backref disk bytenr does not match extent record, bytenr=16913485824, ref bytenr=0
Backref bytes do not match extent backref, bytenr=16913485824, ref bytes=7577600, backref bytes=4096
Backref 16913485824 root 659 owner 142885 offset 0 num_refs 0 not found in extent tree
Incorrect local backref count on 16913485824 root 659 owner 142885 offset 0 found 1 wanted 0 back 0x565421ccaf80
Incorrect local backref count on 16913485824 root 659 owner 142885 offset 0 found 0 wanted 1 back 0x565421d54ee0
Backref disk bytenr does not match extent record, bytenr=16913485824, ref bytenr=0
Backref bytes do not match extent backref, bytenr=16913485824, ref bytes=7577600, backref bytes=4096
Backref 16913485824 root 659 owner 142884 offset 0 num_refs 0 not found in extent tree
Incorrect local backref count on 16913485824 root 659 owner 142884 offset 0 found 1 wanted 0 back 0x565421cca5e0
Incorrect local backref count on 16913485824 root 659 owner 142884 offset 0 found 0 wanted 1 back 0x565421d549a0
Backref disk bytenr does not match extent record, bytenr=16913485824, ref bytenr=7550552506368
Backref bytes do not match extent backref, bytenr=16913485824, ref bytes=7577600, backref bytes=4096
Backref 16913485824 root 659 owner 142883 offset 0 num_refs 0 not found in extent tree
Incorrect local backref count on 16913485824 root 659 owner 142883 offset 0 found 1 wanted 0 back 0x565421d30f80
Incorrect local backref count on 16913485824 root 659 owner 142883 offset 0 found 0 wanted 1 back 0x565421d54d60
Backref disk bytenr does not match extent record, bytenr=16913485824, ref bytenr=426848698368
Backref bytes do not match extent backref, bytenr=16913485824, ref bytes=7577600, backref bytes=4096
Backref 16913485824 root 659 owner 142882 offset 0 num_refs 0 not found in extent tree
Incorrect local backref count on 16913485824 root 659 owner 142882 offset 0 found 1 wanted 0 back 0x565421d308e0
Incorrect local backref count on 16913485824 root 659 owner 142882 offset 0 found 0 wanted 1 back 0x565421d54be0
Backref disk bytenr does not match extent record, bytenr=16913485824, ref bytenr=258
Backref bytes do not match extent backref, bytenr=16913485824, ref bytes=7577600, backref bytes=4096
backpointer mismatch on [16913485824 7577600]
Backref 246050455552 root 659 owner 188780 offset 18446744073709289472 num_refs 0 not found in extent tree
Incorrect local backref count on 246050455552 root 659 owner 188780 offset 18446744073709289472 found 1 wanted 0 back 0x565424af3d60
Incorrect local backref count on 246050455552 root 659 owner 188780 offset 18446744073709289472 found 0 wanted 1 back 0x565424159d00
Backref disk bytenr does not match extent record, bytenr=246050455552, ref bytenr=12177733394079923520
Backref bytes do not match extent backref, bytenr=246050455552, ref bytes=92831744, backref bytes=4096
Backref 246050455552 root 659 owner 188780 offset 18446744073708896256 num_refs 0 not found in extent tree
Incorrect local backref count on 246050455552 root 659 owner 188780 offset 18446744073708896256 found 1 wanted 0 back 0x565424aabde0
Incorrect local backref count on 246050455552 root 659 owner 188780 offset 18446744073708896256 found 0 wanted 1 back 0x565424159700
Backref disk bytenr does not match extent record, bytenr=246050455552, ref bytenr=9288675033350144
Backref bytes do not match extent backref, bytenr=246050455552, ref bytes=92831744, backref bytes=4096
Backref 246050455552 root 659 owner 188780 offset 2669543424 num_refs 0 not found in extent tree
Incorrect local backref count on 246050455552 root 659 owner 188780 offset 2669543424 found 1 wanted 0 back 0x565424a8cd60
Incorrect local backref count on 246050455552 root 659 owner 188780 offset 2669543424 found 0 wanted 1 back 0x565424159a00
Backref disk bytenr does not match extent record, bytenr=246050455552, ref bytenr=169
Backref bytes do not match extent backref, bytenr=246050455552, ref bytes=92831744, backref bytes=4096
Backref 246050455552 root 659 owner 188780 offset 2752512 num_refs 0 not found in extent tree
Incorrect local backref count on 246050455552 root 659 owner 188780 offset 2752512 found 1 wanted 0 back 0x565424b39ca0
Incorrect local backref count on 246050455552 root 659 owner 188780 offset 2752512 found 0 wanted 1 back 0x565424159640
Backref disk bytenr does not match extent record, bytenr=246050455552, ref bytenr=28018364110602240
Backref bytes do not match extent backref, bytenr=246050455552, ref bytes=92831744, backref bytes=4096
Backref 246050455552 root 659 owner 188780 offset 262144 num_refs 0 not found in extent tree
Incorrect local backref count on 246050455552 root 659 owner 188780 offset 262144 found 1 wanted 0 back 0x565424b32160
Incorrect local backref count on 246050455552 root 659 owner 188780 offset 262144 found 0 wanted 1 back 0x565424159b80
Backref disk bytenr does not match extent record, bytenr=246050455552, ref bytenr=141733931441
Backref bytes do not match extent backref, bytenr=246050455552, ref bytes=92831744, backref bytes=4096
Backref 246050455552 root 659 owner 188780 offset 0 num_refs 0 not found in extent tree
Incorrect local backref count on 246050455552 root 659 owner 188780 offset 0 found 1 wanted 0 back 0x565424ab46a0
Incorrect local backref count on 246050455552 root 659 owner 188780 offset 0 found 0 wanted 1 back 0x565424159820
Backref disk bytenr does not match extent record, bytenr=246050455552, ref bytenr=185817465119627
Backref bytes do not match extent backref, bytenr=246050455552, ref bytes=92831744, backref bytes=4096
Backref 246050455552 root 659 owner 188778 offset 18446744073709420544 num_refs 0 not found in extent tree
Incorrect local backref count on 246050455552 root 659 owner 188778 offset 18446744073709420544 found 2 wanted 0 back 0x565424acbde0
Incorrect local backref count on 246050455552 root 659 owner 188778 offset 18446744073709420544 found 0 wanted 2 back 0x565424159940
Backref disk bytenr does not match extent record, bytenr=246050455552, ref bytenr=36283886645504
Backref bytes do not match extent backref, bytenr=246050455552, ref bytes=92831744, backref bytes=4096
Backref 246050455552 root 659 owner 188778 offset 18446744073708371968 num_refs 0 not found in extent tree
Incorrect local backref count on 246050455552 root 659 owner 188778 offset 18446744073708371968 found 1 wanted 0 back 0x565424a84fa0
Incorrect local backref count on 246050455552 root 659 owner 188778 offset 18446744073708371968 found 0 wanted 1 back 0x565424159be0
Backref disk bytenr does not match extent record, bytenr=246050455552, ref bytenr=45273250267136
Backref bytes do not match extent backref, bytenr=246050455552, ref bytes=92831744, backref bytes=4096
Backref 246050455552 root 659 owner 188778 offset 3787325440 num_refs 0 not found in extent tree
Incorrect local backref count on 246050455552 root 659 owner 188778 offset 3787325440 found 1 wanted 0 back 0x565424b7c1a0
Incorrect local backref count on 246050455552 root 659 owner 188778 offset 3787325440 found 0 wanted 1 back 0x5654241597c0
Backref disk bytenr does not match extent record, bytenr=246050455552, ref bytenr=174336
Backref bytes do not match extent backref, bytenr=246050455552, ref bytes=92831744, backref bytes=4096
Backref 246050455552 root 659 owner 188778 offset 3276800 num_refs 0 not found in extent tree
Incorrect local backref count on 246050455552 root 659 owner 188778 offset 3276800 found 1 wanted 0 back 0x565424adb320
Incorrect local backref count on 246050455552 root 659 owner 188778 offset 3276800 found 0 wanted 1 back 0x565424159b20
Backref disk bytenr does not match extent record, bytenr=246050455552, ref bytenr=10067585860754210849
Backref bytes do not match extent backref, bytenr=246050455552, ref bytes=92831744, backref bytes=4096
Backref 246050455552 root 659 owner 188778 offset 655360 num_refs 0 not found in extent tree
Incorrect local backref count on 246050455552 root 659 owner 188778 offset 655360 found 1 wanted 0 back 0x565424ac43a0
Incorrect local backref count on 246050455552 root 659 owner 188778 offset 655360 found 0 wanted 1 back 0x565424159ca0
Backref disk bytenr does not match extent record, bytenr=246050455552, ref bytenr=2835349504
Backref bytes do not match extent backref, bytenr=246050455552, ref bytes=92831744, backref bytes=4096
Backref 246050455552 root 659 owner 188778 offset 524288 num_refs 0 not found in extent tree
Incorrect local backref count on 246050455552 root 659 owner 188778 offset 524288 found 1 wanted 0 back 0x565424aa3f20
Incorrect local backref count on 246050455552 root 659 owner 188778 offset 524288 found 0 wanted 1 back 0x565424159a60
Backref disk bytenr does not match extent record, bytenr=246050455552, ref bytenr=725849473123
Backref bytes do not match extent backref, bytenr=246050455552, ref bytes=92831744, backref bytes=4096
Backref 246050455552 root 659 owner 188778 offset 131072 num_refs 0 not found in extent tree
Incorrect local backref count on 246050455552 root 659 owner 188778 offset 131072 found 1 wanted 0 back 0x565424a9c460
Incorrect local backref count on 246050455552 root 659 owner 188778 offset 131072 found 0 wanted 1 back 0x565424159d60
Backref disk bytenr does not match extent record, bytenr=246050455552, ref bytenr=7173037113116459008
Backref bytes do not match extent backref, bytenr=246050455552, ref bytes=92831744, backref bytes=4096
Backref 246050455552 root 659 owner 188778 offset 0 num_refs 0 not found in extent tree
Incorrect local backref count on 246050455552 root 659 owner 188778 offset 0 found 1 wanted 0 back 0x565424abc760
Incorrect local backref count on 246050455552 root 659 owner 188778 offset 0 found 0 wanted 1 back 0x565424159760
Backref disk bytenr does not match extent record, bytenr=246050455552, ref bytenr=3404439843315384320
Backref bytes do not match extent backref, bytenr=246050455552, ref bytes=92831744, backref bytes=4096
Backref 246050455552 root 659 owner 188777 offset 3729260544 num_refs 0 not found in extent tree
Incorrect local backref count on 246050455552 root 659 owner 188777 offset 3729260544 found 1 wanted 0 back 0x565421d39960
Incorrect local backref count on 246050455552 root 659 owner 188777 offset 3729260544 found 0 wanted 1 back 0x565424159ee0
Backref disk bytenr does not match extent record, bytenr=246050455552, ref bytenr=11075584
Backref bytes do not match extent backref, bytenr=246050455552, ref bytes=92831744, backref bytes=4096
Backref 246050455552 root 659 owner 188777 offset 14286848 num_refs 0 not found in extent tree
Incorrect local backref count on 246050455552 root 659 owner 188777 offset 14286848 found 1 wanted 0 back 0x565424b940e0
Incorrect local backref count on 246050455552 root 659 owner 188777 offset 14286848 found 0 wanted 1 back 0x5654241599a0
Backref disk bytenr does not match extent record, bytenr=246050455552, ref bytenr=12424481393868800
Backref bytes do not match extent backref, bytenr=246050455552, ref bytes=92831744, backref bytes=4096
Backref 246050455552 root 659 owner 188777 offset 13631488 num_refs 0 not found in extent tree
Incorrect local backref count on 246050455552 root 659 owner 188777 offset 13631488 found 1 wanted 0 back 0x565424b9c120
Incorrect local backref count on 246050455552 root 659 owner 188777 offset 13631488 found 0 wanted 1 back 0x565424159dc0
Backref disk bytenr does not match extent record, bytenr=246050455552, ref bytenr=553648166
Backref bytes do not match extent backref, bytenr=246050455552, ref bytes=92831744, backref bytes=4096
Backref 246050455552 root 659 owner 188777 offset 10616832 num_refs 0 not found in extent tree
Incorrect local backref count on 246050455552 root 659 owner 188777 offset 10616832 found 1 wanted 0 back 0x565424b8bda0
Incorrect local backref count on 246050455552 root 659 owner 188777 offset 10616832 found 0 wanted 1 back 0x5654241598e0
Backref disk bytenr does not match extent record, bytenr=246050455552, ref bytenr=9277415232383232512
Backref bytes do not match extent backref, bytenr=246050455552, ref bytes=92831744, backref bytes=4096
Backref 246050455552 root 659 owner 188777 offset 10354688 num_refs 0 not found in extent tree
Incorrect local backref count on 246050455552 root 659 owner 188777 offset 10354688 found 2 wanted 0 back 0x565424b41ee0
Incorrect local backref count on 246050455552 root 659 owner 188777 offset 10354688 found 0 wanted 2 back 0x565424159e20
Backref disk bytenr does not match extent record, bytenr=246050455552, ref bytenr=2377900767366348800
Backref bytes do not match extent backref, bytenr=246050455552, ref bytes=92831744, backref bytes=4096
Backref 246050455552 root 659 owner 188777 offset 7471104 num_refs 0 not found in extent tree
Incorrect local backref count on 246050455552 root 659 owner 188777 offset 7471104 found 1 wanted 0 back 0x565424b83c60
Incorrect local backref count on 246050455552 root 659 owner 188777 offset 7471104 found 0 wanted 1 back 0x565424159c40
Backref disk bytenr does not match extent record, bytenr=246050455552, ref bytenr=0
Backref bytes do not match extent backref, bytenr=246050455552, ref bytes=92831744, backref bytes=4096
Backref 246050455552 root 659 owner 188777 offset 7340032 num_refs 0 not found in extent tree
Incorrect local backref count on 246050455552 root 659 owner 188777 offset 7340032 found 1 wanted 0 back 0x565424afb8a0
Incorrect local backref count on 246050455552 root 659 owner 188777 offset 7340032 found 0 wanted 1 back 0x565424159880
Backref disk bytenr does not match extent record, bytenr=246050455552, ref bytenr=109450383851520
Backref bytes do not match extent backref, bytenr=246050455552, ref bytes=92831744, backref bytes=4096
Backref 246050455552 root 659 owner 188777 offset 6815744 num_refs 0 not found in extent tree
Incorrect local backref count on 246050455552 root 659 owner 188777 offset 6815744 found 1 wanted 0 back 0x565424b747e0
Incorrect local backref count on 246050455552 root 659 owner 188777 offset 6815744 found 0 wanted 1 back 0x565424159e80
Backref disk bytenr does not match extent record, bytenr=246050455552, ref bytenr=12826251738751172608
Backref bytes do not match extent backref, bytenr=246050455552, ref bytes=92831744, backref bytes=4096
Backref 246050455552 root 659 owner 188777 offset 6684672 num_refs 0 not found in extent tree
Incorrect local backref count on 246050455552 root 659 owner 188777 offset 6684672 found 1 wanted 0 back 0x565424b64ae0
Incorrect local backref count on 246050455552 root 659 owner 188777 offset 6684672 found 0 wanted 1 back 0x565424159f40
Backref disk bytenr does not match extent record, bytenr=246050455552, ref bytenr=47569271070624738
Backref bytes do not match extent backref, bytenr=246050455552, ref bytes=92831744, backref bytes=4096
Backref 246050455552 root 659 owner 188777 offset 5898240 num_refs 0 not found in extent tree
Incorrect local backref count on 246050455552 root 659 owner 188777 offset 5898240 found 1 wanted 0 back 0x565421d386e0
Incorrect local backref count on 246050455552 root 659 owner 188777 offset 5898240 found 0 wanted 1 back 0x565424159ac0
Backref disk bytenr does not match extent record, bytenr=246050455552, ref bytenr=427545706496
Backref bytes do not match extent backref, bytenr=246050455552, ref bytes=92831744, backref bytes=4096
Backref 246050455552 root 659 owner 188777 offset 0 num_refs 0 not found in extent tree
Incorrect local backref count on 246050455552 root 659 owner 188777 offset 0 found 1 wanted 0 back 0x565424b6c9a0
Incorrect local backref count on 246050455552 root 659 owner 188777 offset 0 found 0 wanted 1 back 0x5654241596a0
Backref disk bytenr does not match extent record, bytenr=246050455552, ref bytenr=2162688
Backref bytes do not match extent backref, bytenr=246050455552, ref bytes=92831744, backref bytes=4096
Backref 246050455552 root 659 owner 188776 offset 0 num_refs 0 not found in extent tree
Incorrect local backref count on 246050455552 root 659 owner 188776 offset 0 found 1 wanted 0 back 0x565424b499a0
Incorrect local backref count on 246050455552 root 659 owner 188776 offset 0 found 0 wanted 1 back 0x565424159fa0
Backref disk bytenr does not match extent record, bytenr=246050455552, ref bytenr=28019832989417472
Backref bytes do not match extent backref, bytenr=246050455552, ref bytes=92831744, backref bytes=4096
backpointer mismatch on [246050455552 92831744]
Incorrect local backref count on 284858793984 root 659 owner 188824 offset 422068224 found 2 wanted 3 back 0x5654241add70
Incorrect global backref count on 284858793984 found 8 wanted 9
backpointer mismatch on [284858793984 134217728]
Incorrect local backref count on 285446881280 root 659 owner 188824 offset 1010155520 found 2 wanted 3 back 0x5654241aed30
Incorrect global backref count on 285446881280 found 8 wanted 9
backpointer mismatch on [285446881280 134217728]
Incorrect local backref count on 285913120768 root 659 owner 188824 offset 1462910976 found 2 wanted 4 back 0x5654241b0780
Incorrect global backref count on 285913120768 found 9 wanted 11
backpointer mismatch on [285913120768 134217728]
Incorrect local backref count on 286852644864 root 659 owner 188824 offset 2357133312 found 2 wanted 3 back 0x5654248a89f0
Incorrect global backref count on 286852644864 found 8 wanted 9
backpointer mismatch on [286852644864 134217728]
Incorrect local backref count on 287165235200 root 659 owner 188824 offset 2669723648 found 2 wanted 3 back 0x5654248a9640
Incorrect global backref count on 287165235200 found 14 wanted 15
backpointer mismatch on [287165235200 134217728]
Incorrect local backref count on 287433670656 root 659 owner 188824 offset 2938159104 found 2 wanted 4 back 0x5654248a9c00
Incorrect global backref count on 287433670656 found 9 wanted 11
backpointer mismatch on [287433670656 134217728]
Incorrect local backref count on 287792168960 root 659 owner 188824 offset 3212296192 found 2 wanted 3 back 0x5654248aae40
Incorrect global backref count on 287792168960 found 10 wanted 11
backpointer mismatch on [287792168960 134217728]
Incorrect local backref count on 287926386688 root 659 owner 188824 offset 3346513920 found 2 wanted 3 back 0x5654248ab220
Incorrect global backref count on 287926386688 found 7 wanted 8
backpointer mismatch on [287926386688 134217728]
Incorrect local backref count on 454773628928 root 659 owner 204210 offset 0 found 2 wanted 4 back 0x56542404ad80
Incorrect global backref count on 454773628928 found 2 wanted 4
backpointer mismatch on [454773628928 8388608]
Incorrect local backref count on 455505809408 root 659 owner 204212 offset 0 found 2 wanted 77 back 0x565423cf4650
Incorrect global backref count on 455505809408 found 79 wanted 154
backpointer mismatch on [455505809408 8388608]
Incorrect local backref count on 455564529664 root 659 owner 204222 offset 0 found 2 wanted 4 back 0x565423cf51a0
Incorrect global backref count on 455564529664 found 6 wanted 8
backpointer mismatch on [455564529664 8388608]
checking free space cache
checking fs roots
checking csums
checking root refs
found 423405654016 bytes used err is 0
total csum bytes: 412769428
total tree bytes: 626212864
total fs tree bytes: 160776192
total extent tree bytes: 18202624
btree space waste bytes: 68135217
file data blocks allocated: 1030486196224
 referenced 1005660291072

Re: btrfs-progs 4.7, check reports many "incorrect local backref count" messages

[Chris Murphy]

[chris@f24s ~]$ sudo btrfs dev stats /brick1
[/dev/mapper/brick1].write_io_errs   0
[/dev/mapper/brick1].read_io_errs    0
[/dev/mapper/brick1].flush_io_errs   0
[/dev/mapper/brick1].corruption_errs 0
[/dev/mapper/brick1].generation_errs 0

[chris@f24s ~]$ sudo btrfs scrub status /brick1
scrub status for 30f4724a-9a9f-4a5d-a37d-3e53876bf2e1
    scrub started at Wed Aug 24 10:07:10 2016 and finished after 01:22:15
    total bytes scrubbed: 394.91GiB with 0 errors

And metadata is DUP.

Re: btrfs-progs 4.7, check reports many "incorrect local backref count" messages

[Chris Murphy]

Plugging in the 'btrfs check' reporte backrefs into btrfs-debug-tree

[chris@f24s ~]$ sudo btrfs-debug-tree -b 16913485824 /dev/mapper/brick1
btrfs-progs v4.7
checksum verify failed on 16913485824 found FEAA7A13 wanted 20000000
checksum verify failed on 16913485824 found FEAA7A13 wanted 20000000
bytenr mismatch, want=16913485824, have=14276644976086482944
ERROR: failed to read 16913485824


[chris@f24s ~]$ sudo btrfs-debug-tree -b 246050455552 /dev/mapper/brick1
btrfs-progs v4.7
checksum verify failed on 246050455552 found E4E3BDB6 wanted 00000000
checksum verify failed on 246050455552 found E4E3BDB6 wanted 00000000
bytenr mismatch, want=246050455552, have=0
ERROR: failed to read 246050455552


[chris@f24s ~]$ sudo btrfs-debug-tree -b 285446881280 /dev/mapper/brick1
btrfs-progs v4.7
checksum verify failed on 285446881280 found 6BB60B47 wanted C9310000
checksum verify failed on 285446881280 found 6BB60B47 wanted C9310000
bytenr mismatch, want=285446881280, have=5188146780436697072
ERROR: failed to read 285446881280


So, all I can say is these must be stale, and that's why their
checksums are wrong; or there's a pretty big big in Btrfs scrub right
now because it came up clean. So at the moment I'm going with the idea
that there's a bug in btrfs check 4.7 that's reporting non-real
problems.



Chris Murphy

Re: btrfs-progs 4.7, check reports many "incorrect local backref count" messages

[Chris Murphy]

On Wed, Aug 24, 2016 at 11:59 AM, Chris Murphy <lists@colorremedies.com> wrote:
> Plugging in the 'btrfs check' reporte backrefs into btrfs-debug-tree
>
> [chris@f24s ~]$ sudo btrfs-debug-tree -b 16913485824 /dev/mapper/brick1
> btrfs-progs v4.7
> checksum verify failed on 16913485824 found FEAA7A13 wanted 20000000
> checksum verify failed on 16913485824 found FEAA7A13 wanted 20000000
> bytenr mismatch, want=16913485824, have=14276644976086482944
> ERROR: failed to read 16913485824
>
>
> [chris@f24s ~]$ sudo btrfs-debug-tree -b 246050455552 /dev/mapper/brick1
> btrfs-progs v4.7
> checksum verify failed on 246050455552 found E4E3BDB6 wanted 00000000
> checksum verify failed on 246050455552 found E4E3BDB6 wanted 00000000
> bytenr mismatch, want=246050455552, have=0
> ERROR: failed to read 246050455552
>
>
> [chris@f24s ~]$ sudo btrfs-debug-tree -b 285446881280 /dev/mapper/brick1
> btrfs-progs v4.7
> checksum verify failed on 285446881280 found 6BB60B47 wanted C9310000
> checksum verify failed on 285446881280 found 6BB60B47 wanted C9310000
> bytenr mismatch, want=285446881280, have=5188146780436697072
> ERROR: failed to read 285446881280


Looks like the above is simply invalid usage on my part. The backref
value doesn't apply to btrfs-debug-tree -b and must be some other
thing. Those values don't show up as either nodes or leaves, only as a
reference to extent data. But a complete btrfs-debug-tree has no
complaints about the file system, so I'm still not sure the btrfs
check complaints are valid.


-- 
Chris Murphy

Re: btrfs-progs 4.7, check reports many "incorrect local backref count" messages

[Chris Murphy]

[chris@f24s ~]$ btrfs --version
btrfs-progs v4.5.3
[chris@f24s ~]$ sudo btrfs check /dev/mapper/brick1
Checking filesystem on /dev/mapper/brick1
UUID: 30f4724a-9a9f-4a5d-a37d-3e53876bf2e1
checking extents
checking free space cache
checking fs roots
checking csums
checking root refs
found 423302123916 bytes used err is 0
total csum bytes: 412769428
total tree bytes: 626229248
total fs tree bytes: 160776192
total extent tree bytes: 18219008
btree space waste bytes: 68151409
file data blocks allocated: 1030486196224
 referenced 1005660291072


[chris@f24s ~]$ btrfs --version
btrfs-progs v4.6.1
[chris@f24s ~]$ sudo btrfs check /dev/mapper/brick1
Checking filesystem on /dev/mapper/brick1
UUID: 30f4724a-9a9f-4a5d-a37d-3e53876bf2e1
checking extents
checking free space cache
checking fs roots
checking csums
checking root refs
found 423405670400 bytes used err is 0
total csum bytes: 412769428
total tree bytes: 626229248
total fs tree bytes: 160776192
total extent tree bytes: 18219008
btree space waste bytes: 68151409
file data blocks allocated: 1030486196224
 referenced 1005660291072


Looks like it's some kind of regression, or just confusing messages
meant for developers and not the user. It's sort of annoying that
these messages don't come with any advice what the user is supposed to
do about them.

Curious the found bytes between progs 4.5.3 and 4.6.1 differ... file
system was not mounted in between these checks.


----
Chris Murphy

Re: btrfs-progs 4.7, check reports many "incorrect local backref count" messages

[Chris Murphy]

This is still happening with btrfs-progs 4.7.1 and there is zero
information in the long result what to do about the problem, and
whether it's sane to try have --repair fix it, let alone what the
original cause of the problem was.



-- 
Chris Murphy

RE: btrfs-progs 4.7, check reports many "incorrect local backref count" messages

[Paul Jones]

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1: Type: text/plain; charset="utf-8", Size: 1110 bytes --]

> -----Original Message-----
> From: linux-btrfs-owner@vger.kernel.org [mailto:linux-btrfs-
> owner@vger.kernel.org] On Behalf Of Chris Murphy
> Sent: Thursday, 1 September 2016 7:59 AM
> To: Btrfs BTRFS <linux-btrfs@vger.kernel.org>
> Subject: Re: btrfs-progs 4.7, check reports many "incorrect local backref
> count" messages
> 
> This is still happening with btrfs-progs 4.7.1 and there is zero information in
> the long result what to do about the problem, and whether it's sane to try
> have --repair fix it, let alone what the original cause of the problem was.

I just potentially damaged a perfectly good filesystem because of this. I was getting hundreds of "Incorrect local backref count" so I decided to try repair, which seemed to complete ok. I then rescanned without repair and btrfs check eventually crashed with an assertion. That's when I figured something may be wrong.

Thankfully I have an offsite backup (this was the onsite backup), but what a pain....

Paul.

ÿôèº{.nÇ+‰·Ÿ®‰­†+%ŠËÿ±éݶ\x17¥Šwÿº{.nÇ+‰·¥Š{±ý»k~ÏâžØ^n‡r¡ö¦zË\x1aëh™¨è­Ú&£ûàz¿äz¹Þ—ú+€Ê+zf£¢·hšˆ§~†­†Ûiÿÿïêÿ‘êçz_è®\x0fæj:+v‰¨þ)ߣøm

Re: btrfs-progs 4.7, check reports many "incorrect local backref count" messages

[Chris Murphy]

On Thu, Sep 1, 2016 at 12:51 AM, Paul Jones <paul@pauljones.id.au> wrote:
>> -----Original Message-----
>> From: linux-btrfs-owner@vger.kernel.org [mailto:linux-btrfs-
>> owner@vger.kernel.org] On Behalf Of Chris Murphy
>> Sent: Thursday, 1 September 2016 7:59 AM
>> To: Btrfs BTRFS <linux-btrfs@vger.kernel.org>
>> Subject: Re: btrfs-progs 4.7, check reports many "incorrect local backref
>> count" messages
>>
>> This is still happening with btrfs-progs 4.7.1 and there is zero information in
>> the long result what to do about the problem, and whether it's sane to try
>> have --repair fix it, let alone what the original cause of the problem was.
>
> I just potentially damaged a perfectly good filesystem because of this. I was getting hundreds of "Incorrect local backref count" so I decided to try repair, which seemed to complete ok. I then rescanned without repair and btrfs check eventually crashed with an assertion. That's when I figured something may be wrong.

Wait, so you did a --repair and then the following-up check crashes?
But does the file system mount and does it still work? The first is
bad enough, but if it won't mount this is terrible.



-- 
Chris Murphy

Re: btrfs-progs 4.7, check reports many "incorrect local backref count" messages

[Chris Murphy]

OK so I have a 2nd volume, which is only ever used to 'btrfs receive'
from the 1st volume. The 2nd volume is never persistently mounted. But
it too has bunches of these incorrect backref messages, where
btrfs-progs 4.6.1 comes up clean. A 3rd volume, is only ever used to
receive rsync from the 1st and it comes up clean with 4.6.1, and
4.7.1, i.e. no such incorrect backref messages.

The 1st volume says:
found 438074621952 bytes used err is 0

So if "err is 0" then there should be nothing to fix.

I'm going to btrfs check and btrfs-image the 1st volume to capture the
current state, and then try a --repair and if it breaks the file
system I'm probably going to go ballistic...


---
Chris Murphy

Re: btrfs-progs 4.7, check reports many "incorrect local backref count" messages

[Chris Murphy]

On Thu, Sep 1, 2016 at 11:59 AM, Chris Murphy <lists@colorremedies.com> wrote:
> OK so I have a 2nd volume, which is only ever used to 'btrfs receive'
> from the 1st volume. The 2nd volume is never persistently mounted. But
> it too has bunches of these incorrect backref messages, where
> btrfs-progs 4.6.1 comes up clean. A 3rd volume, is only ever used to
> receive rsync from the 1st and it comes up clean with 4.6.1, and
> 4.7.1, i.e. no such incorrect backref messages.
>
> The 1st volume says:
> found 438074621952 bytes used err is 0
>
> So if "err is 0" then there should be nothing to fix.
>
> I'm going to btrfs check and btrfs-image the 1st volume to capture the
> current state, and then try a --repair and if it breaks the file
> system I'm probably going to go ballistic...

OK so I did a repair on the primary volume, it claim to fix a bunch of
stuff, without errors. It mounts fine afterward, just like it did
before repair. And a subsequent check still finds all kinds of backref
problems that were in fact not fixed even though repair claimed to fix
them. But it doesn't crash at least.

I'll file a bug on the Ux of this, which is pretty bad, aside from the
fact it appears to not repair what it claims to repair.

Example from repair:

adding new data backref on 16913485824 root 659 owner 143234 offset 0 found 1

and from the after check
Backref 16913485824 root 659 owner 143234 offset 0 num_refs 0 not
found in extent tree
Incorrect local backref count on 16913485824 root 659 owner 143234
offset 0 found 1 wanted 0 back 0x55e44e85ccd0
Incorrect local backref count on 16913485824 root 659 owner 143234
offset 0 found 0 wanted 1 back 0x55e44eb83910
Backref disk bytenr does not match extent record, bytenr=16913485824,
ref bytenr=79553
Backref bytes do not match extent backref, bytenr=16913485824, ref
bytes=7577600, backref bytes=4096
Backref 16913485824 root 659 owner 143233 offset 0 num_refs 0 not
found in extent tree

Which is identical to what's found in the before check.




-- 
Chris Murphy

RE: btrfs-progs 4.7, check reports many "incorrect local backref count" messages

[Paul Jones]

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1: Type: text/plain; charset="utf-8", Size: 2150 bytes --]

> -----Original Message-----
> From: chris@colorremedies.com [mailto:chris@colorremedies.com] On
> 
> On Thu, Sep 1, 2016 at 12:51 AM, Paul Jones <paul@pauljones.id.au> wrote:
> >> -----Original Message-----
> >> From: linux-btrfs-owner@vger.kernel.org [mailto:linux-btrfs-
> >> owner@vger.kernel.org] On Behalf Of Chris Murphy
> >> Sent: Thursday, 1 September 2016 7:59 AM
> >> To: Btrfs BTRFS <linux-btrfs@vger.kernel.org>
> >> Subject: Re: btrfs-progs 4.7, check reports many "incorrect local
> >> backref count" messages
> >>
> >> This is still happening with btrfs-progs 4.7.1 and there is zero
> >> information in the long result what to do about the problem, and
> >> whether it's sane to try have --repair fix it, let alone what the original
> cause of the problem was.
> >
> > I just potentially damaged a perfectly good filesystem because of this. I was
> getting hundreds of "Incorrect local backref count" so I decided to try repair,
> which seemed to complete ok. I then rescanned without repair and btrfs
> check eventually crashed with an assertion. That's when I figured something
> may be wrong.
> 
> Wait, so you did a --repair and then the following-up check crashes?
> But does the file system mount and does it still work? The first is bad enough,
> but if it won't mount this is terrible.

Correct. I forgot to mention the crash was with btrfs-progs 4.7.1 and kernel 4.7.2. After that I reverted to btrfs-progs 4.6.1 and kernel 4.6.7
The filesystem still mounted, so I tried to run scrub but that got stuck at 0 bytes (with no errors). I reset the system (reboot didn't finish) and run repair a few more times, and even though there were still errors every time I noticed the last few errors appeared to be the same.
I mounted and tried scrub again, which worked the second time. I let it run overnight and it didn't find any errors. I'm just about to rsync (with checksums) from the backup to verify the data is still good.

I'd suggest btrfs-progs 4.7.x is withdrawn until this can be fixed.

Paul.
ÿôèº{.nÇ+‰·Ÿ®‰­†+%ŠËÿ±éݶ\x17¥Šwÿº{.nÇ+‰·¥Š{±ý»k~ÏâžØ^n‡r¡ö¦zË\x1aëh™¨è­Ú&£ûàz¿äz¹Þ—ú+€Ê+zf£¢·hšˆ§~†­†Ûiÿÿïêÿ‘êçz_è®\x0fæj:+v‰¨þ)ߣøm

Re: btrfs-progs 4.7, check reports many "incorrect local backref count" messages

[Chris Murphy]

OK I filed a bug:

progs 4.7.x, numerous incorrect backrefs are not fixed with --repair
https://bugzilla.kernel.org/show_bug.cgi?id=155791

Re: btrfs-progs 4.7, check reports many "incorrect local backref count" messages

[Chris Murphy]

On Thu, Sep 1, 2016 at 5:51 PM, Paul Jones <paul@pauljones.id.au> wrote:
>> -----Original Message-----
>> From: chris@colorremedies.com [mailto:chris@colorremedies.com] On
>>
>> On Thu, Sep 1, 2016 at 12:51 AM, Paul Jones <paul@pauljones.id.au> wrote:
>> >> -----Original Message-----
>> >> From: linux-btrfs-owner@vger.kernel.org [mailto:linux-btrfs-
>> >> owner@vger.kernel.org] On Behalf Of Chris Murphy
>> >> Sent: Thursday, 1 September 2016 7:59 AM
>> >> To: Btrfs BTRFS <linux-btrfs@vger.kernel.org>
>> >> Subject: Re: btrfs-progs 4.7, check reports many "incorrect local
>> >> backref count" messages
>> >>
>> >> This is still happening with btrfs-progs 4.7.1 and there is zero
>> >> information in the long result what to do about the problem, and
>> >> whether it's sane to try have --repair fix it, let alone what the original
>> cause of the problem was.
>> >
>> > I just potentially damaged a perfectly good filesystem because of this. I was
>> getting hundreds of "Incorrect local backref count" so I decided to try repair,
>> which seemed to complete ok. I then rescanned without repair and btrfs
>> check eventually crashed with an assertion. That's when I figured something
>> may be wrong.
>>
>> Wait, so you did a --repair and then the following-up check crashes?
>> But does the file system mount and does it still work? The first is bad enough,
>> but if it won't mount this is terrible.
>
> Correct. I forgot to mention the crash was with btrfs-progs 4.7.1 and kernel 4.7.2. After that I reverted to btrfs-progs 4.6.1 and kernel 4.6.7
> The filesystem still mounted, so I tried to run scrub but that got stuck at 0 bytes (with no errors). I reset the system (reboot didn't finish) and run repair a few more times, and even though there were still errors every time I noticed the last few errors appeared to be the same.
> I mounted and tried scrub again, which worked the second time. I let it run overnight and it didn't find any errors. I'm just about to rsync (with checksums) from the backup to verify the data is still good.
>
> I'd suggest btrfs-progs 4.7.x is withdrawn until this can be fixed.

Yeah upstream can't really do that, it's up to the distros. The main
thing though is that it didn't actually make the file system worse
off, it just sounds like it wasted some time on your end.

My complaint about this is I have no idea what the messages mean, how
big of a problem they are, whether they should be repaired, and what
to do about the fact they claim to be repaired but aren't.



-- 
Chris Murphy