From: Zach Aller <ZAller@iteris.com>
To: Zach Aller <ZAller@iteris.com>, Chris Murphy <lists@colorremedies.com>
Cc: "linux-btrfs@vger.kernel.org" <linux-btrfs@vger.kernel.org>
Subject: Re: How to fix these btrfs errors
Date: Mon, 1 May 2017 02:35:39 +0000 [thread overview]
Message-ID: <D52C0859.486C4%zaller@iteris.com> (raw)
In-Reply-To: <D52BCC47.486A8%zaller@iteris.com>
One more while running a scrub dmesg logs showed this
[98761.912449] scrub_handle_errored_block: 22 callbacks suppressed
[98761.912472] BTRFS warning (device sda): checksum error at logical
5614914584576 on dev /dev/sda, sector 10979229344: metadata leaf (level 0)
in tree 7
[98761.912476] BTRFS warning (device sda): checksum error at logical
5614914584576 on dev /dev/sda, sector 10979229344: metadata leaf (level 0)
in tree 7
[98761.912479] btrfs_dev_stat_print_on_error: 22 callbacks suppressed
[98761.912484] BTRFS error (device sda): bdev /dev/sda errs: wr 0, rd 0,
flush 0, corrupt 270, gen 22
[98761.921028] scrub_handle_errored_block: 22 callbacks suppressed
[98761.921034] BTRFS error (device sda): unable to fixup (regular) error
at logical 5614914584576 on dev /dev/sda
[98761.925274] BTRFS warning (device sda): checksum error at logical
5614914600960 on dev /dev/sda, sector 10979229376: metadata leaf (level 0)
in tree 7
[98761.925278] BTRFS warning (device sda): checksum error at logical
5614914600960 on dev /dev/sda, sector 10979229376: metadata leaf (level 0)
in tree 7
[98761.925284] BTRFS error (device sda): bdev /dev/sda errs: wr 0, rd 0,
flush 0, corrupt 271, gen 22
[98761.929502] BTRFS error (device sda): unable to fixup (regular) error
at logical 5614914600960 on dev /dev/sda
[98761.933715] BTRFS warning (device sda): checksum error at logical
5614914617344 on dev /dev/sda, sector 10979229408: metadata leaf (level 0)
in tree 7
[98761.933717] BTRFS warning (device sda): checksum error at logical
5614914617344 on dev /dev/sda, sector 10979229408: metadata leaf (level 0)
in tree 7
[98761.933722] BTRFS error (device sda): bdev /dev/sda errs: wr 0, rd 0,
flush 0, corrupt 272, gen 22
[98761.937951] BTRFS error (device sda): unable to fixup (regular) error
at logical 5614914617344 on dev /dev/sda
[98763.775452] BTRFS warning (device sda): checksum/header error at
logical 5614914584576 on dev /dev/sda, sector 10980277920: metadata leaf
(level 0) in tree 7
[98763.775457] BTRFS warning (device sda): checksum/header error at
logical 5614914584576 on dev /dev/sda, sector 10980277920: metadata leaf
(level 0) in tree 7
[98763.775463] BTRFS error (device sda): bdev /dev/sda errs: wr 0, rd 0,
flush 0, corrupt 272, gen 23
[98763.784019] BTRFS error (device sda): unable to fixup (regular) error
at logical 5614914584576 on dev /dev/sda
[98763.788323] BTRFS warning (device sda): checksum/header error at
logical 5614914600960 on dev /dev/sda, sector 10980277952: metadata leaf
(level 0) in tree 7
[98763.788328] BTRFS warning (device sda): checksum/header error at
logical 5614914600960 on dev /dev/sda, sector 10980277952: metadata leaf
(level 0) in tree 7
[98763.788334] BTRFS error (device sda): bdev /dev/sda errs: wr 0, rd 0,
flush 0, corrupt 272, gen 24
[98763.792722] BTRFS error (device sda): unable to fixup (regular) error
at logical 5614914600960 on dev /dev/sda
[98763.797106] BTRFS warning (device sda): checksum/header error at
logical 5614914617344 on dev /dev/sda, sector 10980277984: metadata leaf
(level 0) in tree 7
[98763.797111] BTRFS warning (device sda): checksum/header error at
logical 5614914617344 on dev /dev/sda, sector 10980277984: metadata leaf
(level 0) in tree 7
[98763.797116] BTRFS error (device sda): bdev /dev/sda errs: wr 0, rd 0,
flush 0, corrupt 272, gen 25
[98763.801401] BTRFS error (device sda): unable to fixup (regular) error
at logical 5614914617344 on dev /dev/sda
[100173.104082] BTRFS error (device sda): bad tree block start 65536
5614914584576
[100173.833202] BTRFS error (device sda): bad tree block start 65536
5614914584576
[100173.845121] BTRFS error (device sda): bad tree block start
499762348032 5614914584576
And then it was aborted
scrub status for bdd89c26-038d-49fd-b895-52b8deb989cc
scrub started at Sun Apr 30 16:27:12 2017 and was aborted after 03:29:08
total bytes scrubbed: 5.79TiB with 55 errors
error details: verify=3 csum=52
corrected errors: 0, uncorrectable errors: 55, unverified errors: 0
I also tried to run this with no luck
./btrfs inspect-internal logical-resolve 5614914584576 /mnt/btrfs/
ERROR: logical ino ioctl: No such file or directory
/btrfs inspect-internal logical-resolve 5614914600960 /mnt/btrfs/
ERROR: logical ino ioctl: No such file or directory
On 4/30/17, 5:20 PM, "Zach Aller" <ZAller@iteris.com> wrote:
>Output of one more command.
>-------
>./btrfs inspect-internal dump-tree /dev/sda > dump.txt
>parent transid verify failed on 15732050345984 wanted 73879 found 73881
>parent transid verify failed on 15732050345984 wanted 73879 found 73881
>parent transid verify failed on 15732050345984 wanted 73879 found 73881
>parent transid verify failed on 15732050345984 wanted 73879 found 73881
>Ignoring transid failure
>WARNING: eb corrupted: item 0 eb level 3 next level 3, skipping the rest
>-------
>
>
>
>Here is the end of dump.txt after the above command failed.
>-------
>
>checksum tree key (CSUM_TREE ROOT_ITEM 0)
>node 15732050329600 level 3 items 15 free 478 generation 73879 owner 7
>fs uuid bdd89c26-038d-49fd-b895-52b8deb989cc
>chunk uuid 2cf80f39-d64e-489f-acec-8411f5c1bb33
> key (EXTENT_CSUM EXTENT_CSUM 12582912) block 15732050345984 (960208151)
>gen 73879
> key (EXTENT_CSUM EXTENT_CSUM 1036259938304) block 15732062109696
>(960208869) gen 73879
> key (EXTENT_CSUM EXTENT_CSUM 2055219736576) block 22061012254720
>(1346497330) gen 73796
> key (EXTENT_CSUM EXTENT_CSUM 3067616186368) block 15732273971200
>(960221800) gen 72465
> key (EXTENT_CSUM EXTENT_CSUM 4125595508736) block 19670583918592
>(1200597163) gen 71747
> key (EXTENT_CSUM EXTENT_CSUM 5160724217856) block 22060767379456
>(1346482384) gen 73047
> key (EXTENT_CSUM EXTENT_CSUM 6225035022336) block 18786519695360
>(1146638165) gen 71390
> key (EXTENT_CSUM EXTENT_CSUM 7317037834240) block 22060823707648
>(1346485822) gen 73791
> key (EXTENT_CSUM EXTENT_CSUM 8520881180672) block 22061229129728
>(1346510567) gen 72304
> key (EXTENT_CSUM EXTENT_CSUM 9556147200000) block 21773927448576
>(1328975064) gen 72764
> key (EXTENT_CSUM EXTENT_CSUM 10643353645056) block 22061034897408
>(1346498712) gen 73797
> key (EXTENT_CSUM EXTENT_CSUM 11886318727168) block 21773509066752
>(1328949528) gen 73741
> key (EXTENT_CSUM EXTENT_CSUM 13041959669760) block 2532038983680
>(154543395) gen 71958
> key (EXTENT_CSUM EXTENT_CSUM 14315004522496) block 21773773946880
>(1328965695) gen 57898
> key (EXTENT_CSUM EXTENT_CSUM 16297194516480) block 22061036748800
>(1346498825) gen 72864
>uuid tree key (UUID_TREE ROOT_ITEM 0)
>leaf 29376512 items 0 free space 16283 generation 6 owner 9
>fs uuid bdd89c26-038d-49fd-b895-52b8deb989cc
>chunk uuid 2cf80f39-d64e-489f-acec-8411f5c1bb33
>data reloc tree key (DATA_RELOC_TREE ROOT_ITEM 0)
>leaf 29442048 items 2 free space 16061 generation 4 owner
>18446744073709551607
>fs uuid bdd89c26-038d-49fd-b895-52b8deb989cc
>chunk uuid 2cf80f39-d64e-489f-acec-8411f5c1bb33
> item 0 key (256 INODE_ITEM 0) itemoff 16123 itemsize 160
> inode generation 3 transid 0 size 0 nbytes 16384
> block group 0 mode 40755 links 1 uid 0 gid 0 rdev 0
> sequence 0 flags 0x0(none)
> atime 1490407706.0 (2017-03-24 21:08:26)
> ctime 1490407706.0 (2017-03-24 21:08:26)
> mtime 1490407706.0 (2017-03-24 21:08:26)
> otime 1490407706.0 (2017-03-24 21:08:26)
> item 1 key (256 INODE_REF 256) itemoff 16111 itemsize 12
> inode ref index 0 namelen 2 name: ..
>total bytes 24004303781888
>bytes used 18737354412032
>uuid bdd89c26-038d-49fd-b895-52b8deb989cc
>
>-------
>
>
>
>
>On 4/30/17, 4:39 PM, "linux-btrfs-owner@vger.kernel.org on behalf of Zach
>Aller" <linux-btrfs-owner@vger.kernel.org on behalf of ZAller@iteris.com>
>wrote:
>
>>It is a recent filesystem the data was written with kernel 4.10, today I
>>upgraded to 4.11rc8 to see if it helped anything which it did not.
>>
>>On 4/30/17, 4:35 PM, "chris@colorremedies.com on behalf of Chris Murphy"
>><chris@colorremedies.com on behalf of lists@colorremedies.com> wrote:
>>
>>>On Sun, Apr 30, 2017 at 3:08 PM, Zach Aller <ZAller@iteris.com> wrote:
>>>
>>>> uname -a
>>>> Linux server 4.11.0-041100rc8-generic #201704232131 SMP Mon Apr 24
>>>> 01:32:55 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
>>>>
>>>> ./btrfs --version
>>>> btrfs-progs v4.10
>>>>
>>>> ./btrfs fi show
>>>> Label: none uuid: bdd89c26-038d-49fd-b895-52b8deb989cc
>>>> Total devices 1 FS bytes used 17.04TiB
>>>> devid 1 size 21.83TiB used 17.28TiB path /dev/sda
>>>
>>>
>>>How old is the file system? Is this a recent problem with just
>>>4.11rc8? Is most of the 17TB written with a particular kernel version,
>>>which?
>>>
>>>
>>>>
>>>> Here is a dmesg snippet
>>>>
>>>>
>>>> [ 3.633295] BTRFS: device fsid bdd89c26-038d-49fd-b895-52b8deb989cc
>>>> devid 1 transid 72387 /dev/sda
>>>> [ 12.907658] BTRFS info (device sda): disk space caching is enabled
>>>> [ 12.907659] BTRFS info (device sda): has skinny extents
>>>> [ 13.129140] BTRFS info (device sda): bdev /dev/sda errs: wr 0, rd
>>>>0,
>>>> flush 0, corrupt 217, gen 19
>>>> [20956.415076] BTRFS info (device sda): The free space cache file
>>>> (9804365955072) is invalid. skip it
>>>> [36292.358558] BTRFS warning (device sda): checksum error at logical
>>>> 5614914584576 on dev /dev/sda, sector 10979229344: metadata leaf
>>>>(level
>>>>0)
>>>> in tree 7
>>>> [36292.358563] BTRFS warning (device sda): checksum error at logical
>>>> 5614914584576 on dev /dev/sda, sector 10979229344: metadata leaf
>>>>(level
>>>>0)
>>>> in tree 7
>>>> [36292.358569] BTRFS error (device sda): bdev /dev/sda errs: wr 0, rd
>>>>0,
>>>> flush 0, corrupt 218, gen 19
>>>> [36292.364717] BTRFS error (device sda): unable to fixup (regular)
>>>>error
>>>> at logical 5614914584576 on dev /dev/sda
>>>
>>>
>>>Both copies of metadata are failing checksum, so it can't be fixed. It
>>>suggests there's a hardware problem (memory or storage), or maybe a
>>>new bug.
>>>
>>>Have there been any crashes while writing to the file system?
>>>What is the storage stack configuration? 22TB for a single block
>>>device means it's built up from something else.
>>>
>>>I'd dig around for any non-btrfs storage stack related errors in the
>>>meantime, maybe a dev will have some idea what's going on from the
>>>call traces, I'm not sure what they mean.
>>>
>>>--
>>>Chris Murphy
>>
>>--
>>To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
>>the body of a message to majordomo@vger.kernel.org
>>More majordomo info at http://vger.kernel.org/majordomo-info.html
>
prev parent reply other threads:[~2017-05-01 2:35 UTC|newest]
Thread overview: 5+ messages / expand[flat|nested] mbox.gz Atom feed top
2017-04-30 21:08 How to fix these btrfs errors Zach Aller
2017-04-30 21:35 ` Chris Murphy
2017-04-30 21:39 ` Zach Aller
2017-04-30 22:20 ` Zach Aller
2017-05-01 2:35 ` Zach Aller [this message]
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=D52C0859.486C4%zaller@iteris.com \
--to=zaller@iteris.com \
--cc=linux-btrfs@vger.kernel.org \
--cc=lists@colorremedies.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox