btrfs scrub failing

btrfs scrub failing

[John Center]

Hi,

I run a weekly scrub, using Marc Merlin's btrfs-scrub script.
Usually, it completes without a problem, but this week it failed.  I
ran the scrub manually & it stops shortly:

john@mariposa:~$ sudo /sbin/btrfs scrub start -BdR /dev/md124p2
ERROR: scrubbing /dev/md124p2 failed for device id 1: ret=-1, errno=5
(Input/output error)
scrub device /dev/md124p2 (id 1) canceled
scrub started at Thu Dec 31 00:26:34 2015 and was aborted after 00:01:29
data_extents_scrubbed: 110967
tree_extents_scrubbed: 99638
data_bytes_scrubbed: 2548817920
tree_bytes_scrubbed: 1632468992
read_errors: 0
csum_errors: 0
verify_errors: 0
no_csum: 1573
csum_discards: 74371
super_errors: 0
malloc_errors: 0
uncorrectable_errors: 0
unverified_errors: 0
corrected_errors: 0
last_physical: 4729667584

john@mariposa:~$ sudo /sbin/btrfs scrub status /dev/md124p2
scrub status for 9b5a6959-7df1-4455-a643-d369487d24aa
scrub started at Thu Dec 31 00:29:06 2015, running for 00:01:15
total bytes scrubbed: 3.46GiB with 0 errors

My Ubuntu 14.04 workstation is using the 4.2 kernel (Wily).  I'm using
btrfs-tools v4.3.1.  Btrfs is on top of mdadm raid1 (imsm).
Autodefrag is enabled. Both drives have checked out ok on smart tests.
Some directories are set up with nodatacow for VMs, etc.

john@mariposa:~$ sudo btrfs fi show
Label: none  uuid: 9b5a6959-7df1-4455-a643-d369487d24aa
Total devices 1 FS bytes used 961.46GiB
devid    1 size 1.76TiB used 978.04GiB path /dev/md124p2

Funny thing is, if the scrub hadn't failed, I wouldn't know there were
any problems!  I've rebooted twice since the original scrub that
failed w/o a problem.  I've backed up all my files to an ext4
partition, again w/o a problem.

I've been searching for a clue on the wiki, mailing list, etc. on how
to fix this, but I'm at a loss.  From what I read, I shouldn't be able
to boot my workstation.  How should I go about repairing this?  Any
help would be greatly appreciated.

Thanks.

    -John


BTW, I did run btrfs-find-root at one point & got the following:

john@mariposa:~$ sudo btrfs-find-root /dev/md124p2
Superblock thinks the generation is 1031315
Superblock thinks the level is 1
Found tree root at 1039015591936 gen 1031315 level 1
Well block 1039013101568(gen: 1031314 level: 1) seems good, but
generation/level doesn't match, want gen: 1031315 level: 1
Well block 1039003533312(gen: 1031313 level: 1) seems good, but
generation/level doesn't match, want gen: 1031315 level: 1
Well block 1039006171136(gen: 1031311 level: 0) seems good, but
generation/level doesn't match, want gen: 1031315 level: 1

... 500+ lines skipped...

Well block 519183810560(gen: 163422 level: 0) seems good, but
generation/level doesn't match, want gen: 1031315 level: 1
Well block 143915876352(gen: 38834 level: 0) seems good, but
generation/level doesn't match, want gen: 1031315 level: 1
Well block 4243456(gen: 3 level: 0) seems good, but generation/level
doesn't match, want gen: 1031315 level: 1
Well block 4194304(gen: 2 level: 0) seems good, but generation/level
doesn't match, want gen: 1031315 level: 1

Re: btrfs scrub failing

[Duncan]

John Center posted on Thu, 31 Dec 2015 11:20:28 -0500 as excerpted:

> I run a weekly scrub, using Marc Merlin's btrfs-scrub script.
> Usually, it completes without a problem, but this week it failed.  I ran
> the scrub manually & it stops shortly:
> 
> john@mariposa:~$ sudo /sbin/btrfs scrub start -BdR /dev/md124p2
> ERROR: scrubbing /dev/md124p2 failed for device id 1:
> ret=-1, errno=5 (Input/output error)
> scrub device /dev/md124p2 (id 1) canceled
> scrub started at Thu Dec 31 00:26:34 2015
> and was aborted after 00:01:29 [...]

> My Ubuntu 14.04 workstation is using the 4.2 kernel (Wily).
> I'm using btrfs-tools v4.3.1. [...]

A couple months ago, which would have made it around the 4.2 kernel 
you're running (with 4.3 being current and 4.4 nearly out), there were a 
number of similar scrub aborted reports on the list.

I don't recall seeing any directly related patches, but the reports died 
down, whether because everybody having them had reported already, or 
because a newer kernel fixed the problem, I'm not sure, as I never had 
the problem myself[1].

So I'd suggest upgrading to either the current 4.3 kernel or the latest 
4.4-rc, and hopefully the problem will be gone.  If I'd had the problem 
myself I could tell you for sure whether it went away for me with 4.3, 
but as I didn't...

In any event, the 4.2 kernel wasn't a long-term-support kernel anyway.  
4.1 was and 4.4 is scheduled to be.  So upstream 4.2 updates should be 
ended or coming to an end in any case, and an upgrade (or possibly 
downgrade to the LTS 4.1) is recommended anyway, tho of course Ubuntu 
could be doing its own support, independent of upstream, but then you'd 
need to get support from them as upstream won't be tracking patches 
they've backported and thus can't provide proper support.

---
[1]  All my btrfs are relatively small, under 50 GiB per device, and on 
SSD, so scrubs normally complete in under a minute, while your report of 
scrub aborting after a minute and a half was typical, so it's likely my 
scrubs were simply done before whatever the problem was could trigger.


-- 
Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman

Re: btrfs scrub failing

[John Center]

Hi Duncan,

> On Jan 1, 2016, at 12:55 AM, Duncan <1i5t5.duncan@cox.net> wrote:
> 
> John Center posted on Thu, 31 Dec 2015 11:20:28 -0500 as excerpted:
> 
>> I run a weekly scrub, using Marc Merlin's btrfs-scrub script.
>> Usually, it completes without a problem, but this week it failed.  I ran
>> the scrub manually & it stops shortly:
>> 
>> john@mariposa:~$ sudo /sbin/btrfs scrub start -BdR /dev/md124p2
>> ERROR: scrubbing /dev/md124p2 failed for device id 1:
>> ret=-1, errno=5 (Input/output error)
>> scrub device /dev/md124p2 (id 1) canceled
>> scrub started at Thu Dec 31 00:26:34 2015
>> and was aborted after 00:01:29 [...]
> 
>> My Ubuntu 14.04 workstation is using the 4.2 kernel (Wily).
>> I'm using btrfs-tools v4.3.1. [...]
> 
> A couple months ago, which would have made it around the 4.2 kernel 
> you're running (with 4.3 being current and 4.4 nearly out), there were a 
> number of similar scrub aborted reports on the list.
> 
I must have missed that, I'll check the list again to try & understand the issue better. 

> I don't recall seeing any directly related patches, but the reports died 
> down, whether because everybody having them had reported already, or 
> because a newer kernel fixed the problem, I'm not sure, as I never had 
> the problem myself[1].
> 
> So I'd suggest upgrading to either the current 4.3 kernel or the latest 
> 4.4-rc, and hopefully the problem will be gone.  If I'd had the problem 
> myself I could tell you for sure whether it went away for me with 4.3, 
> but as I didn't...
> 
Ok, I'll upgrade to 4.3 & see if that resolves the problem with scrubbing. I was wondering when I compiled the btrfs-tools if there would be a problem with them not being in sync with the major kernel version. 

If this doesn't resolve the problem, what would you recommend my next steps should be?  I've been hesitant to run too many of the btrfs-tools, mainly because I don't want to accidentally screw things up & I don't always know how to interpret the results. (I ran btrfs-debug-tree, hoping something obvious would show up.  Big mistake. 😋)

Thanks for your help!

    -John

Re: btrfs scrub failing

[Duncan]

John Center posted on Fri, 01 Jan 2016 11:41:20 -0500 as excerpted:

> Ok, I'll upgrade to 4.3 & see if that resolves the problem with
> scrubbing.
> I was wondering when I compiled the btrfs-tools if there would be a
> problem with them not being in sync with the major kernel version.

FWIW newer (or older either, as long as not too old and you don't need 
any of the features not in the older, and you're not trying to fix 
problems only the newer can deal with) versions of btrfs-progs should be 
fine.  As a rule of thumb I recommend staying at least current to kernel 
version, but that's a rule of thumb, primarily to prevent getting /too/ 
old, only.  Both the btrfs-progs userspace and the kernel itself are 
normally designed to be able to work with both older and newer versions 
of the other one.

So userspace not being in sync with the kernel version shouldn't be a 
problem.

> If this doesn't resolve the problem, what would you recommend my next
> steps should be?  I've been hesitant to run too many of the btrfs-tools,
> mainly because I don't want to accidentally screw things up & I don't
> always know how to interpret the results. (I ran btrfs-debug-tree,
> hoping something obvious would show up.  Big mistake. 😋)

LOLed at that debug-tree remark.  Been there (with other tools) myself.

Well, I'm hoping someone who had the problem can confirm whether it's 
fixed in current kernels (scrub is one of those userspace commands that's 
mostly just a front-end to the kernel code which does the real work, so 
kernel version is the important thing for scrub).  I'm guessing so, and 
that you'll find the problem gone in 4.3.

We'll cross the not-gone bridge if we get to it, but again, if the other 
people who had the similar problem can confirm whether it disappeared for 
them with the new kernel, it would help a lot, as there were enough such 
reports that if it's the same problem and still there for everyone (which 
I doubt as I expect there'd still be way more posts about it if so, but 
confirmation's always good), nothing to do but wait for a fix, while if 
not, and you still have your problem, then it's a different issue and the 
devs will need to work with you on a fix specific to your problem.

-- 
Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman

Re: btrfs scrub failing

[Martin Steigerwald]

Am Freitag, 1. Januar 2016, 11:41:20 CET schrieb John Center:

Happy New Year!

> > On Jan 1, 2016, at 12:55 AM, Duncan <1i5t5.duncan@cox.net> wrote:
> > 
> >
> > John Center posted on Thu, 31 Dec 2015 11:20:28 -0500 as excerpted:
> > 
> >
> >> I run a weekly scrub, using Marc Merlin's btrfs-scrub script.
> >> Usually, it completes without a problem, but this week it failed.  I ran
> >>
> >> the scrub manually & it stops shortly:
> >> 
> >>
> >> john@mariposa:~$ sudo /sbin/btrfs scrub start -BdR /dev/md124p2
> >> ERROR: scrubbing /dev/md124p2 failed for device id 1:
> >> ret=-1, errno=5 (Input/output error)
> >> scrub device /dev/md124p2 (id 1) canceled
> >> scrub started at Thu Dec 31 00:26:34 2015
> >> and was aborted after 00:01:29 [...]
> >
> > 
> >
> >> My Ubuntu 14.04 workstation is using the 4.2 kernel (Wily).
> >> I'm using btrfs-tools v4.3.1. [...]
> >
> > 
> >
> > A couple months ago, which would have made it around the 4.2 kernel 
> > you're running (with 4.3 being current and 4.4 nearly out), there were a 
> > number of similar scrub aborted reports on the list.
> >
> > 
> 
> I must have missed that, I'll check the list again to try & understand the
> issue better. 

I had repeatedly failing scrubs as mentioned in another thread here, until I 
used 4.4 kernel. With 4.3 kernel scrub also didn´t work. I didn´t use the 
debug options you used above and I am not sure whether I had this scrub issue 
with 4.2 already, so I am not sure it has been the same issue. But you may 
need to run 4.4 kernel in order to get scrub working again.

See my thread "[4.3-rc4] scrubbing aborts before finishing" for details.

Thanks,
-- 
Martin

Re: btrfs scrub failing

[John Center]

Hi Martin,

Happy New Year!

> On Jan 1, 2016, at 12:41 PM, Martin Steigerwald <martin@lichtvoll.de> wrote:
> 
> Am Freitag, 1. Januar 2016, 11:41:20 CET schrieb John Center:
> 
> Happy New Year!
> 
>>> On Jan 1, 2016, at 12:55 AM, Duncan <1i5t5.duncan@cox.net> wrote:
>>> 
>>> A couple months ago, which would have made it around the 4.2 kernel 
>>> you're running (with 4.3 being current and 4.4 nearly out), there were a 
>>> number of similar scrub aborted reports on the list.
>> 
>> I must have missed that, I'll check the list again to try & understand the
>> issue better.
> 
> I had repeatedly failing scrubs as mentioned in another thread here, until I 
> used 4.4 kernel. With 4.3 kernel scrub also didn´t work. I didn´t use the 
> debug options you used above and I am not sure whether I had this scrub issue 
> with 4.2 already, so I am not sure it has been the same issue. But you may 
> need to run 4.4 kernel in order to get scrub working again.
> 
> See my thread "[4.3-rc4] scrubbing aborts before finishing" for details.
> 
I was afraid of this. I just read your thread. I generally try to stay away from kernels so new, but I may have to try it. Was there any reason you didn't go to 4.1 instead?  (I run win8.1 in VirtualBox 5.0.12, when I need to run somethings under Windows. I'd have to wait until 4.4 is released & supported to do that.)

Thanks. 

    -John

Re: btrfs scrub failing

[John Center]

Hi Duncan,

> On Jan 1, 2016, at 12:05 PM, Duncan <1i5t5.duncan@cox.net> wrote:
> 
> John Center posted on Fri, 01 Jan 2016 11:41:20 -0500 as excerpted:
> 
>> Ok, I'll upgrade to 4.3 & see if that resolves the problem with
>> scrubbing.
>> I was wondering when I compiled the btrfs-tools if there would be a
>> problem with them not being in sync with the major kernel version.
> 
> FWIW newer (or older either, as long as not too old and you don't need 
> any of the features not in the older, and you're not trying to fix 
> problems only the newer can deal with) versions of btrfs-progs should be 
> fine.  As a rule of thumb I recommend staying at least current to kernel 
> version, but that's a rule of thumb, primarily to prevent getting /too/ 
> old, only.  Both the btrfs-progs userspace and the kernel itself are 
> normally designed to be able to work with both older and newer versions 
> of the other one.
> 
> So userspace not being in sync with the kernel version shouldn't be a 
> problem.
> 
Ok, good to know. 

> Well, I'm hoping someone who had the problem can confirm whether it's 
> fixed in current kernels (scrub is one of those userspace commands that's 
> mostly just a front-end to the kernel code which does the real work, so 
> kernel version is the important thing for scrub).  I'm guessing so, and 
> that you'll find the problem gone in 4.3.
> 
I wasn't aware of this. Good to know. 

> We'll cross the not-gone bridge if we get to it, but again, if the other 
> people who had the similar problem can confirm whether it disappeared for 
> them with the new kernel, it would help a lot, as there were enough such 
> reports that if it's the same problem and still there for everyone (which 
> I doubt as I expect there'd still be way more posts about it if so, but 
> confirmation's always good), nothing to do but wait for a fix, while if 
> not, and you still have your problem, then it's a different issue and the 
> devs will need to work with you on a fix specific to your problem.
> 
Ok, understood. 

Thanks & Happy New Year!

    -John

Re: btrfs scrub failing

[Martin Steigerwald]

Am Freitag, 1. Januar 2016, 13:20:49 CET schrieb John Center:
> > On Jan 1, 2016, at 12:41 PM, Martin Steigerwald <martin@lichtvoll.de>
> > wrote:
> > Am Freitag, 1. Januar 2016, 11:41:20 CET schrieb John Center:
[…]
> >>> On Jan 1, 2016, at 12:55 AM, Duncan <1i5t5.duncan@cox.net> wrote:
> >>> 
> >>> A couple months ago, which would have made it around the 4.2 kernel
> >>> you're running (with 4.3 being current and 4.4 nearly out), there were a
> >>> number of similar scrub aborted reports on the list.
> >> 
> >> I must have missed that, I'll check the list again to try & understand
> >> the
> >> issue better.
> > 
> > I had repeatedly failing scrubs as mentioned in another thread here, until
> > I used 4.4 kernel. With 4.3 kernel scrub also didn´t work. I didn´t use
> > the debug options you used above and I am not sure whether I had this
> > scrub issue with 4.2 already, so I am not sure it has been the same
> > issue. But you may need to run 4.4 kernel in order to get scrub working
> > again.
> > 
> > See my thread "[4.3-rc4] scrubbing aborts before finishing" for details.
> 
> I was afraid of this. I just read your thread. I generally try to stay away
> from kernels so new, but I may have to try it. Was there any reason you
> didn't go to 4.1 instead?  (I run win8.1 in VirtualBox 5.0.12, when I need
> to run somethings under Windows. I'd have to wait until 4.4 is released &
> supported to do that.)

So far 4.4-rc6 is pretty stable for me. And I think its almost before release 
as rc7 is out already.

Reason for not going with 4.1? Ey, that would be downgrading, wouldn´t it? But 
sure it is also an option.

Virtualbox 5.0.12-dfsg-2 as packaged by Debian runs fine here with 4.4-rc6.

Thanks,
-- 
Martin

Re: btrfs scrub failing

[John Center]

Hi Duncan,

On Fri, Jan 1, 2016 at 12:05 PM, Duncan <1i5t5.duncan@cox.net> wrote:
> John Center posted on Fri, 01 Jan 2016 11:41:20 -0500 as excerpted:
>
>> If this doesn't resolve the problem, what would you recommend my next
>> steps should be?  I've been hesitant to run too many of the btrfs-tools,
>> mainly because I don't want to accidentally screw things up & I don't
>> always know how to interpret the results. (I ran btrfs-debug-tree,
>> hoping something obvious would show up.  Big mistake. 😋)
>
> LOLed at that debug-tree remark.  Been there (with other tools) myself.
>
> Well, I'm hoping someone who had the problem can confirm whether it's
> fixed in current kernels (scrub is one of those userspace commands that's
> mostly just a front-end to the kernel code which does the real work, so
> kernel version is the important thing for scrub).  I'm guessing so, and
> that you'll find the problem gone in 4.3.
>
> We'll cross the not-gone bridge if we get to it, but again, if the other
> people who had the similar problem can confirm whether it disappeared for
> them with the new kernel, it would help a lot, as there were enough such
> reports that if it's the same problem and still there for everyone (which
> I doubt as I expect there'd still be way more posts about it if so, but
> confirmation's always good), nothing to do but wait for a fix, while if
> not, and you still have your problem, then it's a different issue and the
> devs will need to work with you on a fix specific to your problem.
>
Ok, I'm at the next bridge. :-(  I upgraded the kernel to 4.4rc7 from
the Ubuntu Mainline archive & I just ran the scrub:

john@mariposa:~$ sudo /sbin/btrfs scrub start -BdR /dev/md125p2
ERROR: scrubbing /dev/md125p2 failed for device id 1: ret=-1, errno=5
(Input/output error)
scrub device /dev/md125p2 (id 1) canceled
scrub started at Fri Jan  1 19:38:21 2016 and was aborted after 00:02:34
data_extents_scrubbed: 111031
tree_extents_scrubbed: 104061
data_bytes_scrubbed: 2549907456
tree_bytes_scrubbed: 1704935424
read_errors: 0
csum_errors: 0
verify_errors: 0
no_csum: 1573
csum_discards: 0
super_errors: 0
malloc_errors: 0
uncorrectable_errors: 0
unverified_errors: 0
corrected_errors: 0
last_physical: 4729667584

I checked dmesg & this appeared:

[11428.983355] BTRFS error (device md125p2): parent transid verify
failed on 241287168 wanted 33554449 found 17
[11431.028399] BTRFS error (device md125p2): parent transid verify
failed on 241287168 wanted 33554449 found 17

Where do I go from here?

Thanks for your help.

    -John

Re: btrfs scrub failing

[John Center]

Hi Duncan,

Doing some more digging, I ran btrfs-image & found the following
errors.  I'm not sure how useful this is, or what this means in terms
of the other btrfs-tools messages.  Maybe more clues?

Thanks.

    -John

john@mariposa:~$ sudo btrfs-image -c9 -t4 /dev/md125p2
/media/data1/btrfs.image-01012016
WARNING: The device is mounted. Make sure the filesystem is quiescent.
parent transid verify failed on 337676075008 wanted 1036368 found 1036377
parent transid verify failed on 337676075008 wanted 1036368 found 1036377
parent transid verify failed on 337676075008 wanted 1036368 found 1036377
parent transid verify failed on 337676075008 wanted 1036368 found 1036377
Ignoring transid failure
parent transid verify failed on 337674846208 wanted 1036370 found 1036377
parent transid verify failed on 337674846208 wanted 1036370 found 1036377
parent transid verify failed on 337674846208 wanted 1036370 found 1036377
parent transid verify failed on 337674846208 wanted 1036370 found 1036377
Ignoring transid failure
parent transid verify failed on 337675403264 wanted 1036370 found 1036377
parent transid verify failed on 337675403264 wanted 1036370 found 1036377
parent transid verify failed on 337675403264 wanted 1036370 found 1036377
parent transid verify failed on 337675403264 wanted 1036370 found 1036377
Ignoring transid failure
parent transid verify failed on 337681907712 wanted 1036375 found 1036377
parent transid verify failed on 337681907712 wanted 1036375 found 1036377
parent transid verify failed on 337681907712 wanted 1036375 found 1036377
parent transid verify failed on 337681907712 wanted 1036375 found 1036377
Ignoring transid failure
parent transid verify failed on 337646354432 wanted 1036368 found 1036377
parent transid verify failed on 337646354432 wanted 1036368 found 1036377
parent transid verify failed on 337646354432 wanted 1036368 found 1036377
parent transid verify failed on 337646354432 wanted 1036368 found 1036377
Ignoring transid failure
parent transid verify failed on 337679597568 wanted 1036373 found 1036377
parent transid verify failed on 337679597568 wanted 1036373 found 1036377
parent transid verify failed on 337679597568 wanted 1036373 found 1036377
parent transid verify failed on 337679597568 wanted 1036373 found 1036377
Ignoring transid failure
parent transid verify failed on 337679613952 wanted 1036373 found 1036377
parent transid verify failed on 337679613952 wanted 1036373 found 1036377
parent transid verify failed on 337679613952 wanted 1036373 found 1036377
parent transid verify failed on 337679613952 wanted 1036373 found 1036377
Ignoring transid failure
parent transid verify failed on 337679745024 wanted 1036372 found 1036377
parent transid verify failed on 337679745024 wanted 1036372 found 1036377
parent transid verify failed on 337679745024 wanted 1036372 found 1036377
parent transid verify failed on 337679745024 wanted 1036372 found 1036377
Ignoring transid failure
parent transid verify failed on 337682022400 wanted 1036369 found 1036377
parent transid verify failed on 337682022400 wanted 1036369 found 1036377
parent transid verify failed on 337682022400 wanted 1036369 found 1036377
parent transid verify failed on 337682022400 wanted 1036369 found 1036377
Ignoring transid failure
parent transid verify failed on 337679712256 wanted 1036373 found 1036377
parent transid verify failed on 337679712256 wanted 1036373 found 1036377
parent transid verify failed on 337679712256 wanted 1036373 found 1036377
parent transid verify failed on 337679712256 wanted 1036373 found 1036377
Ignoring transid failure
parent transid verify failed on 337647927296 wanted 1036368 found 1036377
parent transid verify failed on 337647927296 wanted 1036368 found 1036377
parent transid verify failed on 337647927296 wanted 1036368 found 1036377
parent transid verify failed on 337647927296 wanted 1036368 found 1036377
Ignoring transid failure
parent transid verify failed on 337683251200 wanted 1036372 found 1036377
parent transid verify failed on 337683251200 wanted 1036372 found 1036377
parent transid verify failed on 337683251200 wanted 1036372 found 1036377
parent transid verify failed on 337683251200 wanted 1036372 found 1036377
Ignoring transid failure
parent transid verify failed on 337683398656 wanted 1036372 found 1036377
parent transid verify failed on 337683398656 wanted 1036372 found 1036377
parent transid verify failed on 337683398656 wanted 1036372 found 1036377
parent transid verify failed on 337683398656 wanted 1036372 found 1036377
Ignoring transid failure
parent transid verify failed on 337682432000 wanted 1036369 found 1036377
parent transid verify failed on 337682432000 wanted 1036369 found 1036377
parent transid verify failed on 337682432000 wanted 1036369 found 1036377
parent transid verify failed on 337682432000 wanted 1036369 found 1036377
Ignoring transid failure
parent transid verify failed on 337633558528 wanted 1036368 found 1036377
parent transid verify failed on 337633558528 wanted 1036368 found 1036377
parent transid verify failed on 337633558528 wanted 1036368 found 1036377
parent transid verify failed on 337633558528 wanted 1036368 found 1036377
Ignoring transid failure
parent transid verify failed on 337682513920 wanted 1036371 found 1036377
parent transid verify failed on 337682513920 wanted 1036371 found 1036377
parent transid verify failed on 337682513920 wanted 1036371 found 1036377
parent transid verify failed on 337682513920 wanted 1036371 found 1036377
Ignoring transid failure
parent transid verify failed on 337687429120 wanted 1036374 found 1036377
parent transid verify failed on 337687429120 wanted 1036374 found 1036377
parent transid verify failed on 337687429120 wanted 1036374 found 1036377
parent transid verify failed on 337687429120 wanted 1036374 found 1036377
Ignoring transid failure
parent transid verify failed on 337682300928 wanted 1036375 found 1036377
parent transid verify failed on 337682300928 wanted 1036375 found 1036377
parent transid verify failed on 337682300928 wanted 1036375 found 1036377
parent transid verify failed on 337682300928 wanted 1036375 found 1036377
Ignoring transid failure
parent transid verify failed on 337683775488 wanted 1036372 found 1036377
parent transid verify failed on 337683775488 wanted 1036372 found 1036377
parent transid verify failed on 337683775488 wanted 1036372 found 1036377
parent transid verify failed on 337683775488 wanted 1036372 found 1036377
Ignoring transid failure
parent transid verify failed on 337680465920 wanted 1036373 found 1036377
parent transid verify failed on 337680465920 wanted 1036373 found 1036377
parent transid verify failed on 337680465920 wanted 1036373 found 1036377
parent transid verify failed on 337680465920 wanted 1036373 found 1036377
Ignoring transid failure
parent transid verify failed on 337680531456 wanted 1036373 found 1036377
parent transid verify failed on 337680531456 wanted 1036373 found 1036377
parent transid verify failed on 337680531456 wanted 1036373 found 1036377
parent transid verify failed on 337680531456 wanted 1036373 found 1036377
Ignoring transid failure
parent transid verify failed on 337642536960 wanted 1036370 found 1036377
parent transid verify failed on 337642536960 wanted 1036370 found 1036377
parent transid verify failed on 337642536960 wanted 1036370 found 1036377
parent transid verify failed on 337642536960 wanted 1036370 found 1036377
Ignoring transid failure
parent transid verify failed on 337677352960 wanted 1036371 found 1036377
parent transid verify failed on 337677352960 wanted 1036371 found 1036377
parent transid verify failed on 337677352960 wanted 1036371 found 1036377
parent transid verify failed on 337677352960 wanted 1036371 found 1036377
Ignoring transid failure
parent transid verify failed on 337682448384 wanted 1036375 found 1036377
parent transid verify failed on 337682448384 wanted 1036375 found 1036377
parent transid verify failed on 337682448384 wanted 1036375 found 1036377
parent transid verify failed on 337682448384 wanted 1036375 found 1036377
Ignoring transid failure
parent transid verify failed on 337682530304 wanted 1036375 found 1036377
parent transid verify failed on 337682530304 wanted 1036375 found 1036377
parent transid verify failed on 337682530304 wanted 1036375 found 1036377
parent transid verify failed on 337682530304 wanted 1036375 found 1036377
Ignoring transid failure
parent transid verify failed on 337682546688 wanted 1036375 found 1036377
parent transid verify failed on 337682546688 wanted 1036375 found 1036377
parent transid verify failed on 337682546688 wanted 1036375 found 1036377
parent transid verify failed on 337682546688 wanted 1036375 found 1036377
Ignoring transid failure
parent transid verify failed on 337682563072 wanted 1036375 found 1036377
parent transid verify failed on 337682563072 wanted 1036375 found 1036377
parent transid verify failed on 337682563072 wanted 1036375 found 1036377
parent transid verify failed on 337682563072 wanted 1036375 found 1036377
Ignoring transid failure
parent transid verify failed on 337682579456 wanted 1036375 found 1036377
parent transid verify failed on 337682579456 wanted 1036375 found 1036377
parent transid verify failed on 337682579456 wanted 1036375 found 1036377
parent transid verify failed on 337682579456 wanted 1036375 found 1036377
Ignoring transid failure
parent transid verify failed on 337682595840 wanted 1036375 found 1036377
parent transid verify failed on 337682595840 wanted 1036375 found 1036377
parent transid verify failed on 337682595840 wanted 1036375 found 1036377
parent transid verify failed on 337682595840 wanted 1036375 found 1036377
Ignoring transid failure
parent transid verify failed on 337682612224 wanted 1036375 found 1036377
parent transid verify failed on 337682612224 wanted 1036375 found 1036377
parent transid verify failed on 337682612224 wanted 1036375 found 1036377
parent transid verify failed on 337682612224 wanted 1036375 found 1036377
Ignoring transid failure
parent transid verify failed on 337682759680 wanted 1036375 found 1036377
parent transid verify failed on 337682759680 wanted 1036375 found 1036377
parent transid verify failed on 337682759680 wanted 1036375 found 1036377
parent transid verify failed on 337682759680 wanted 1036375 found 1036377
Ignoring transid failure
parent transid verify failed on 337682808832 wanted 1036375 found 1036377
parent transid verify failed on 337682808832 wanted 1036375 found 1036377
parent transid verify failed on 337682808832 wanted 1036375 found 1036377
parent transid verify failed on 337682808832 wanted 1036375 found 1036377
Ignoring transid failure
parent transid verify failed on 337682874368 wanted 1036375 found 1036377
parent transid verify failed on 337682874368 wanted 1036375 found 1036377
parent transid verify failed on 337682874368 wanted 1036375 found 1036377
parent transid verify failed on 337682874368 wanted 1036375 found 1036377
Ignoring transid failure
parent transid verify failed on 337682907136 wanted 1036375 found 1036377
parent transid verify failed on 337682907136 wanted 1036375 found 1036377
parent transid verify failed on 337682907136 wanted 1036375 found 1036377
parent transid verify failed on 337682907136 wanted 1036375 found 1036377
Ignoring transid failure
parent transid verify failed on 337682956288 wanted 1036375 found 1036377
parent transid verify failed on 337682956288 wanted 1036375 found 1036377
parent transid verify failed on 337682956288 wanted 1036375 found 1036377
parent transid verify failed on 337682956288 wanted 1036375 found 1036377
Ignoring transid failure
parent transid verify failed on 337682972672 wanted 1036375 found 1036377
parent transid verify failed on 337682972672 wanted 1036375 found 1036377
parent transid verify failed on 337682972672 wanted 1036375 found 1036377
parent transid verify failed on 337682972672 wanted 1036375 found 1036377
Ignoring transid failure
parent transid verify failed on 337682989056 wanted 1036375 found 1036377
parent transid verify failed on 337682989056 wanted 1036375 found 1036377
parent transid verify failed on 337682989056 wanted 1036375 found 1036377
parent transid verify failed on 337682989056 wanted 1036375 found 1036377
Ignoring transid failure
parent transid verify failed on 337683005440 wanted 1036375 found 1036377
parent transid verify failed on 337683005440 wanted 1036375 found 1036377
parent transid verify failed on 337683005440 wanted 1036375 found 1036377
parent transid verify failed on 337683005440 wanted 1036375 found 1036377
Ignoring transid failure
parent transid verify failed on 337683054592 wanted 1036375 found 1036377
parent transid verify failed on 337683054592 wanted 1036375 found 1036377
parent transid verify failed on 337683054592 wanted 1036375 found 1036377
parent transid verify failed on 337683054592 wanted 1036375 found 1036377
Ignoring transid failure
parent transid verify failed on 337683070976 wanted 1036375 found 1036377
parent transid verify failed on 337683070976 wanted 1036375 found 1036377
parent transid verify failed on 337683070976 wanted 1036375 found 1036377
parent transid verify failed on 337683070976 wanted 1036375 found 1036377
Ignoring transid failure
parent transid verify failed on 337683087360 wanted 1036375 found 1036377
parent transid verify failed on 337683087360 wanted 1036375 found 1036377
parent transid verify failed on 337683087360 wanted 1036375 found 1036377
parent transid verify failed on 337683087360 wanted 1036375 found 1036377
Ignoring transid failure
parent transid verify failed on 337683136512 wanted 1036375 found 1036377
parent transid verify failed on 337683136512 wanted 1036375 found 1036377
parent transid verify failed on 337683136512 wanted 1036375 found 1036377
parent transid verify failed on 337683136512 wanted 1036375 found 1036377
Ignoring transid failure
parent transid verify failed on 337683333120 wanted 1036375 found 1036377
parent transid verify failed on 337683333120 wanted 1036375 found 1036377
parent transid verify failed on 337683333120 wanted 1036375 found 1036377
parent transid verify failed on 337683333120 wanted 1036375 found 1036377
Ignoring transid failure
parent transid verify failed on 337684938752 wanted 1036375 found 1036377
parent transid verify failed on 337684938752 wanted 1036375 found 1036377
parent transid verify failed on 337684938752 wanted 1036375 found 1036377
parent transid verify failed on 337684938752 wanted 1036375 found 1036377
Ignoring transid failure
parent transid verify failed on 337685790720 wanted 1036375 found 1036377
parent transid verify failed on 337685790720 wanted 1036375 found 1036377
parent transid verify failed on 337685790720 wanted 1036375 found 1036377
parent transid verify failed on 337685790720 wanted 1036375 found 1036377
Ignoring transid failure
parent transid verify failed on 337685823488 wanted 1036375 found 1036377
parent transid verify failed on 337685823488 wanted 1036375 found 1036377
parent transid verify failed on 337685823488 wanted 1036375 found 1036377
parent transid verify failed on 337685823488 wanted 1036375 found 1036377
Ignoring transid failure
parent transid verify failed on 337686315008 wanted 1036375 found 1036377
parent transid verify failed on 337686315008 wanted 1036375 found 1036377
parent transid verify failed on 337686315008 wanted 1036375 found 1036377
parent transid verify failed on 337686315008 wanted 1036375 found 1036377
Ignoring transid failure
parent transid verify failed on 337684987904 wanted 1036369 found 1036377
parent transid verify failed on 337684987904 wanted 1036369 found 1036377
parent transid verify failed on 337684987904 wanted 1036369 found 1036377
parent transid verify failed on 337684987904 wanted 1036369 found 1036377
Ignoring transid failure
parent transid verify failed on 337690689536 wanted 1036375 found 1036377
parent transid verify failed on 337690689536 wanted 1036375 found 1036377
parent transid verify failed on 337690689536 wanted 1036375 found 1036377
parent transid verify failed on 337690689536 wanted 1036375 found 1036377
Ignoring transid failure
parent transid verify failed on 337687707648 wanted 1036372 found 1036377
parent transid verify failed on 337687707648 wanted 1036372 found 1036377
parent transid verify failed on 337687707648 wanted 1036372 found 1036377
parent transid verify failed on 337687707648 wanted 1036372 found 1036377
Ignoring transid failure



On Fri, Jan 1, 2016 at 8:04 PM, John Center <jlcenter15@gmail.com> wrote:
> Hi Duncan,
>
> On Fri, Jan 1, 2016 at 12:05 PM, Duncan <1i5t5.duncan@cox.net> wrote:
>> John Center posted on Fri, 01 Jan 2016 11:41:20 -0500 as excerpted:
>>
>>> If this doesn't resolve the problem, what would you recommend my next
>>> steps should be?  I've been hesitant to run too many of the btrfs-tools,
>>> mainly because I don't want to accidentally screw things up & I don't
>>> always know how to interpret the results. (I ran btrfs-debug-tree,
>>> hoping something obvious would show up.  Big mistake. 😋)
>>
>> LOLed at that debug-tree remark.  Been there (with other tools) myself.
>>
>> Well, I'm hoping someone who had the problem can confirm whether it's
>> fixed in current kernels (scrub is one of those userspace commands that's
>> mostly just a front-end to the kernel code which does the real work, so
>> kernel version is the important thing for scrub).  I'm guessing so, and
>> that you'll find the problem gone in 4.3.
>>
>> We'll cross the not-gone bridge if we get to it, but again, if the other
>> people who had the similar problem can confirm whether it disappeared for
>> them with the new kernel, it would help a lot, as there were enough such
>> reports that if it's the same problem and still there for everyone (which
>> I doubt as I expect there'd still be way more posts about it if so, but
>> confirmation's always good), nothing to do but wait for a fix, while if
>> not, and you still have your problem, then it's a different issue and the
>> devs will need to work with you on a fix specific to your problem.
>>
> Ok, I'm at the next bridge. :-(  I upgraded the kernel to 4.4rc7 from
> the Ubuntu Mainline archive & I just ran the scrub:
>
> john@mariposa:~$ sudo /sbin/btrfs scrub start -BdR /dev/md125p2
> ERROR: scrubbing /dev/md125p2 failed for device id 1: ret=-1, errno=5
> (Input/output error)
> scrub device /dev/md125p2 (id 1) canceled
> scrub started at Fri Jan  1 19:38:21 2016 and was aborted after 00:02:34
> data_extents_scrubbed: 111031
> tree_extents_scrubbed: 104061
> data_bytes_scrubbed: 2549907456
> tree_bytes_scrubbed: 1704935424
> read_errors: 0
> csum_errors: 0
> verify_errors: 0
> no_csum: 1573
> csum_discards: 0
> super_errors: 0
> malloc_errors: 0
> uncorrectable_errors: 0
> unverified_errors: 0
> corrected_errors: 0
> last_physical: 4729667584
>
> I checked dmesg & this appeared:
>
> [11428.983355] BTRFS error (device md125p2): parent transid verify
> failed on 241287168 wanted 33554449 found 17
> [11431.028399] BTRFS error (device md125p2): parent transid verify
> failed on 241287168 wanted 33554449 found 17
>
> Where do I go from here?
>
> Thanks for your help.
>
>     -John

Re: btrfs scrub failing

[John Center]

Last thing for tonight, I tried to run btrfs-debug-tree & direct the
output to a file.  It crashed during the run with the following
errors:

john@mariposa:~$ sudo btrfs-debug-tree /dev/md125p2 >>
/media/data1/btrfs-debug-tree-01012016.txt
parent transid verify failed on 519273742336 wanted 1036426 found 1036428
parent transid verify failed on 519273742336 wanted 1036426 found 1036428
parent transid verify failed on 519273742336 wanted 1036426 found 1036428
parent transid verify failed on 519273742336 wanted 1036426 found 1036428
Ignoring transid failure
parent transid verify failed on 519271792640 wanted 1036425 found 1036428
parent transid verify failed on 519271792640 wanted 1036425 found 1036428
parent transid verify failed on 519271792640 wanted 1036425 found 1036428
parent transid verify failed on 519271792640 wanted 1036425 found 1036428
Ignoring transid failure
parent transid verify failed on 519274119168 wanted 1036426 found 1036428
parent transid verify failed on 519274119168 wanted 1036426 found 1036428
parent transid verify failed on 519274119168 wanted 1036426 found 1036428
parent transid verify failed on 519274119168 wanted 1036426 found 1036428
Ignoring transid failure
parent transid verify failed on 519274135552 wanted 1036426 found 1036428
parent transid verify failed on 519274135552 wanted 1036426 found 1036428
parent transid verify failed on 519274135552 wanted 1036426 found 1036428
parent transid verify failed on 519274135552 wanted 1036426 found 1036428
Ignoring transid failure
print-tree.c:1108: btrfs_print_tree: Assertion failed.
btrfs-debug-tree[0x418d99]
btrfs-debug-tree(btrfs_print_tree+0x2c0)[0x41ad4c]
btrfs-debug-tree(btrfs_print_tree+0x2dc)[0x41ad68]
btrfs-debug-tree(main+0x9a5)[0x432589]
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf5)[0x7ff0629ccec5]
btrfs-debug-tree[0x4070e9]

Hopefully this will help the developers.

    -John


On Fri, Jan 1, 2016 at 11:04 PM, John Center <jlcenter15@gmail.com> wrote:
> Hi Duncan,
>
> Doing some more digging, I ran btrfs-image & found the following
> errors.  I'm not sure how useful this is, or what this means in terms
> of the other btrfs-tools messages.  Maybe more clues?
>
> Thanks.
>
>     -John
>
> john@mariposa:~$ sudo btrfs-image -c9 -t4 /dev/md125p2
> /media/data1/btrfs.image-01012016
> WARNING: The device is mounted. Make sure the filesystem is quiescent.
> parent transid verify failed on 337676075008 wanted 1036368 found 1036377
> parent transid verify failed on 337676075008 wanted 1036368 found 1036377
> parent transid verify failed on 337676075008 wanted 1036368 found 1036377
> parent transid verify failed on 337676075008 wanted 1036368 found 1036377
> Ignoring transid failure
> parent transid verify failed on 337674846208 wanted 1036370 found 1036377
> parent transid verify failed on 337674846208 wanted 1036370 found 1036377
> parent transid verify failed on 337674846208 wanted 1036370 found 1036377
> parent transid verify failed on 337674846208 wanted 1036370 found 1036377
> Ignoring transid failure
> parent transid verify failed on 337675403264 wanted 1036370 found 1036377
> parent transid verify failed on 337675403264 wanted 1036370 found 1036377
> parent transid verify failed on 337675403264 wanted 1036370 found 1036377
> parent transid verify failed on 337675403264 wanted 1036370 found 1036377
> Ignoring transid failure
> parent transid verify failed on 337681907712 wanted 1036375 found 1036377
> parent transid verify failed on 337681907712 wanted 1036375 found 1036377
> parent transid verify failed on 337681907712 wanted 1036375 found 1036377
> parent transid verify failed on 337681907712 wanted 1036375 found 1036377
> Ignoring transid failure
> parent transid verify failed on 337646354432 wanted 1036368 found 1036377
> parent transid verify failed on 337646354432 wanted 1036368 found 1036377
> parent transid verify failed on 337646354432 wanted 1036368 found 1036377
> parent transid verify failed on 337646354432 wanted 1036368 found 1036377
> Ignoring transid failure
> parent transid verify failed on 337679597568 wanted 1036373 found 1036377
> parent transid verify failed on 337679597568 wanted 1036373 found 1036377
> parent transid verify failed on 337679597568 wanted 1036373 found 1036377
> parent transid verify failed on 337679597568 wanted 1036373 found 1036377
> Ignoring transid failure
> parent transid verify failed on 337679613952 wanted 1036373 found 1036377
> parent transid verify failed on 337679613952 wanted 1036373 found 1036377
> parent transid verify failed on 337679613952 wanted 1036373 found 1036377
> parent transid verify failed on 337679613952 wanted 1036373 found 1036377
> Ignoring transid failure
> parent transid verify failed on 337679745024 wanted 1036372 found 1036377
> parent transid verify failed on 337679745024 wanted 1036372 found 1036377
> parent transid verify failed on 337679745024 wanted 1036372 found 1036377
> parent transid verify failed on 337679745024 wanted 1036372 found 1036377
> Ignoring transid failure
> parent transid verify failed on 337682022400 wanted 1036369 found 1036377
> parent transid verify failed on 337682022400 wanted 1036369 found 1036377
> parent transid verify failed on 337682022400 wanted 1036369 found 1036377
> parent transid verify failed on 337682022400 wanted 1036369 found 1036377
> Ignoring transid failure
> parent transid verify failed on 337679712256 wanted 1036373 found 1036377
> parent transid verify failed on 337679712256 wanted 1036373 found 1036377
> parent transid verify failed on 337679712256 wanted 1036373 found 1036377
> parent transid verify failed on 337679712256 wanted 1036373 found 1036377
> Ignoring transid failure
> parent transid verify failed on 337647927296 wanted 1036368 found 1036377
> parent transid verify failed on 337647927296 wanted 1036368 found 1036377
> parent transid verify failed on 337647927296 wanted 1036368 found 1036377
> parent transid verify failed on 337647927296 wanted 1036368 found 1036377
> Ignoring transid failure
> parent transid verify failed on 337683251200 wanted 1036372 found 1036377
> parent transid verify failed on 337683251200 wanted 1036372 found 1036377
> parent transid verify failed on 337683251200 wanted 1036372 found 1036377
> parent transid verify failed on 337683251200 wanted 1036372 found 1036377
> Ignoring transid failure
> parent transid verify failed on 337683398656 wanted 1036372 found 1036377
> parent transid verify failed on 337683398656 wanted 1036372 found 1036377
> parent transid verify failed on 337683398656 wanted 1036372 found 1036377
> parent transid verify failed on 337683398656 wanted 1036372 found 1036377
> Ignoring transid failure
> parent transid verify failed on 337682432000 wanted 1036369 found 1036377
> parent transid verify failed on 337682432000 wanted 1036369 found 1036377
> parent transid verify failed on 337682432000 wanted 1036369 found 1036377
> parent transid verify failed on 337682432000 wanted 1036369 found 1036377
> Ignoring transid failure
> parent transid verify failed on 337633558528 wanted 1036368 found 1036377
> parent transid verify failed on 337633558528 wanted 1036368 found 1036377
> parent transid verify failed on 337633558528 wanted 1036368 found 1036377
> parent transid verify failed on 337633558528 wanted 1036368 found 1036377
> Ignoring transid failure
> parent transid verify failed on 337682513920 wanted 1036371 found 1036377
> parent transid verify failed on 337682513920 wanted 1036371 found 1036377
> parent transid verify failed on 337682513920 wanted 1036371 found 1036377
> parent transid verify failed on 337682513920 wanted 1036371 found 1036377
> Ignoring transid failure
> parent transid verify failed on 337687429120 wanted 1036374 found 1036377
> parent transid verify failed on 337687429120 wanted 1036374 found 1036377
> parent transid verify failed on 337687429120 wanted 1036374 found 1036377
> parent transid verify failed on 337687429120 wanted 1036374 found 1036377
> Ignoring transid failure
> parent transid verify failed on 337682300928 wanted 1036375 found 1036377
> parent transid verify failed on 337682300928 wanted 1036375 found 1036377
> parent transid verify failed on 337682300928 wanted 1036375 found 1036377
> parent transid verify failed on 337682300928 wanted 1036375 found 1036377
> Ignoring transid failure
> parent transid verify failed on 337683775488 wanted 1036372 found 1036377
> parent transid verify failed on 337683775488 wanted 1036372 found 1036377
> parent transid verify failed on 337683775488 wanted 1036372 found 1036377
> parent transid verify failed on 337683775488 wanted 1036372 found 1036377
> Ignoring transid failure
> parent transid verify failed on 337680465920 wanted 1036373 found 1036377
> parent transid verify failed on 337680465920 wanted 1036373 found 1036377
> parent transid verify failed on 337680465920 wanted 1036373 found 1036377
> parent transid verify failed on 337680465920 wanted 1036373 found 1036377
> Ignoring transid failure
> parent transid verify failed on 337680531456 wanted 1036373 found 1036377
> parent transid verify failed on 337680531456 wanted 1036373 found 1036377
> parent transid verify failed on 337680531456 wanted 1036373 found 1036377
> parent transid verify failed on 337680531456 wanted 1036373 found 1036377
> Ignoring transid failure
> parent transid verify failed on 337642536960 wanted 1036370 found 1036377
> parent transid verify failed on 337642536960 wanted 1036370 found 1036377
> parent transid verify failed on 337642536960 wanted 1036370 found 1036377
> parent transid verify failed on 337642536960 wanted 1036370 found 1036377
> Ignoring transid failure
> parent transid verify failed on 337677352960 wanted 1036371 found 1036377
> parent transid verify failed on 337677352960 wanted 1036371 found 1036377
> parent transid verify failed on 337677352960 wanted 1036371 found 1036377
> parent transid verify failed on 337677352960 wanted 1036371 found 1036377
> Ignoring transid failure
> parent transid verify failed on 337682448384 wanted 1036375 found 1036377
> parent transid verify failed on 337682448384 wanted 1036375 found 1036377
> parent transid verify failed on 337682448384 wanted 1036375 found 1036377
> parent transid verify failed on 337682448384 wanted 1036375 found 1036377
> Ignoring transid failure
> parent transid verify failed on 337682530304 wanted 1036375 found 1036377
> parent transid verify failed on 337682530304 wanted 1036375 found 1036377
> parent transid verify failed on 337682530304 wanted 1036375 found 1036377
> parent transid verify failed on 337682530304 wanted 1036375 found 1036377
> Ignoring transid failure
> parent transid verify failed on 337682546688 wanted 1036375 found 1036377
> parent transid verify failed on 337682546688 wanted 1036375 found 1036377
> parent transid verify failed on 337682546688 wanted 1036375 found 1036377
> parent transid verify failed on 337682546688 wanted 1036375 found 1036377
> Ignoring transid failure
> parent transid verify failed on 337682563072 wanted 1036375 found 1036377
> parent transid verify failed on 337682563072 wanted 1036375 found 1036377
> parent transid verify failed on 337682563072 wanted 1036375 found 1036377
> parent transid verify failed on 337682563072 wanted 1036375 found 1036377
> Ignoring transid failure
> parent transid verify failed on 337682579456 wanted 1036375 found 1036377
> parent transid verify failed on 337682579456 wanted 1036375 found 1036377
> parent transid verify failed on 337682579456 wanted 1036375 found 1036377
> parent transid verify failed on 337682579456 wanted 1036375 found 1036377
> Ignoring transid failure
> parent transid verify failed on 337682595840 wanted 1036375 found 1036377
> parent transid verify failed on 337682595840 wanted 1036375 found 1036377
> parent transid verify failed on 337682595840 wanted 1036375 found 1036377
> parent transid verify failed on 337682595840 wanted 1036375 found 1036377
> Ignoring transid failure
> parent transid verify failed on 337682612224 wanted 1036375 found 1036377
> parent transid verify failed on 337682612224 wanted 1036375 found 1036377
> parent transid verify failed on 337682612224 wanted 1036375 found 1036377
> parent transid verify failed on 337682612224 wanted 1036375 found 1036377
> Ignoring transid failure
> parent transid verify failed on 337682759680 wanted 1036375 found 1036377
> parent transid verify failed on 337682759680 wanted 1036375 found 1036377
> parent transid verify failed on 337682759680 wanted 1036375 found 1036377
> parent transid verify failed on 337682759680 wanted 1036375 found 1036377
> Ignoring transid failure
> parent transid verify failed on 337682808832 wanted 1036375 found 1036377
> parent transid verify failed on 337682808832 wanted 1036375 found 1036377
> parent transid verify failed on 337682808832 wanted 1036375 found 1036377
> parent transid verify failed on 337682808832 wanted 1036375 found 1036377
> Ignoring transid failure
> parent transid verify failed on 337682874368 wanted 1036375 found 1036377
> parent transid verify failed on 337682874368 wanted 1036375 found 1036377
> parent transid verify failed on 337682874368 wanted 1036375 found 1036377
> parent transid verify failed on 337682874368 wanted 1036375 found 1036377
> Ignoring transid failure
> parent transid verify failed on 337682907136 wanted 1036375 found 1036377
> parent transid verify failed on 337682907136 wanted 1036375 found 1036377
> parent transid verify failed on 337682907136 wanted 1036375 found 1036377
> parent transid verify failed on 337682907136 wanted 1036375 found 1036377
> Ignoring transid failure
> parent transid verify failed on 337682956288 wanted 1036375 found 1036377
> parent transid verify failed on 337682956288 wanted 1036375 found 1036377
> parent transid verify failed on 337682956288 wanted 1036375 found 1036377
> parent transid verify failed on 337682956288 wanted 1036375 found 1036377
> Ignoring transid failure
> parent transid verify failed on 337682972672 wanted 1036375 found 1036377
> parent transid verify failed on 337682972672 wanted 1036375 found 1036377
> parent transid verify failed on 337682972672 wanted 1036375 found 1036377
> parent transid verify failed on 337682972672 wanted 1036375 found 1036377
> Ignoring transid failure
> parent transid verify failed on 337682989056 wanted 1036375 found 1036377
> parent transid verify failed on 337682989056 wanted 1036375 found 1036377
> parent transid verify failed on 337682989056 wanted 1036375 found 1036377
> parent transid verify failed on 337682989056 wanted 1036375 found 1036377
> Ignoring transid failure
> parent transid verify failed on 337683005440 wanted 1036375 found 1036377
> parent transid verify failed on 337683005440 wanted 1036375 found 1036377
> parent transid verify failed on 337683005440 wanted 1036375 found 1036377
> parent transid verify failed on 337683005440 wanted 1036375 found 1036377
> Ignoring transid failure
> parent transid verify failed on 337683054592 wanted 1036375 found 1036377
> parent transid verify failed on 337683054592 wanted 1036375 found 1036377
> parent transid verify failed on 337683054592 wanted 1036375 found 1036377
> parent transid verify failed on 337683054592 wanted 1036375 found 1036377
> Ignoring transid failure
> parent transid verify failed on 337683070976 wanted 1036375 found 1036377
> parent transid verify failed on 337683070976 wanted 1036375 found 1036377
> parent transid verify failed on 337683070976 wanted 1036375 found 1036377
> parent transid verify failed on 337683070976 wanted 1036375 found 1036377
> Ignoring transid failure
> parent transid verify failed on 337683087360 wanted 1036375 found 1036377
> parent transid verify failed on 337683087360 wanted 1036375 found 1036377
> parent transid verify failed on 337683087360 wanted 1036375 found 1036377
> parent transid verify failed on 337683087360 wanted 1036375 found 1036377
> Ignoring transid failure
> parent transid verify failed on 337683136512 wanted 1036375 found 1036377
> parent transid verify failed on 337683136512 wanted 1036375 found 1036377
> parent transid verify failed on 337683136512 wanted 1036375 found 1036377
> parent transid verify failed on 337683136512 wanted 1036375 found 1036377
> Ignoring transid failure
> parent transid verify failed on 337683333120 wanted 1036375 found 1036377
> parent transid verify failed on 337683333120 wanted 1036375 found 1036377
> parent transid verify failed on 337683333120 wanted 1036375 found 1036377
> parent transid verify failed on 337683333120 wanted 1036375 found 1036377
> Ignoring transid failure
> parent transid verify failed on 337684938752 wanted 1036375 found 1036377
> parent transid verify failed on 337684938752 wanted 1036375 found 1036377
> parent transid verify failed on 337684938752 wanted 1036375 found 1036377
> parent transid verify failed on 337684938752 wanted 1036375 found 1036377
> Ignoring transid failure
> parent transid verify failed on 337685790720 wanted 1036375 found 1036377
> parent transid verify failed on 337685790720 wanted 1036375 found 1036377
> parent transid verify failed on 337685790720 wanted 1036375 found 1036377
> parent transid verify failed on 337685790720 wanted 1036375 found 1036377
> Ignoring transid failure
> parent transid verify failed on 337685823488 wanted 1036375 found 1036377
> parent transid verify failed on 337685823488 wanted 1036375 found 1036377
> parent transid verify failed on 337685823488 wanted 1036375 found 1036377
> parent transid verify failed on 337685823488 wanted 1036375 found 1036377
> Ignoring transid failure
> parent transid verify failed on 337686315008 wanted 1036375 found 1036377
> parent transid verify failed on 337686315008 wanted 1036375 found 1036377
> parent transid verify failed on 337686315008 wanted 1036375 found 1036377
> parent transid verify failed on 337686315008 wanted 1036375 found 1036377
> Ignoring transid failure
> parent transid verify failed on 337684987904 wanted 1036369 found 1036377
> parent transid verify failed on 337684987904 wanted 1036369 found 1036377
> parent transid verify failed on 337684987904 wanted 1036369 found 1036377
> parent transid verify failed on 337684987904 wanted 1036369 found 1036377
> Ignoring transid failure
> parent transid verify failed on 337690689536 wanted 1036375 found 1036377
> parent transid verify failed on 337690689536 wanted 1036375 found 1036377
> parent transid verify failed on 337690689536 wanted 1036375 found 1036377
> parent transid verify failed on 337690689536 wanted 1036375 found 1036377
> Ignoring transid failure
> parent transid verify failed on 337687707648 wanted 1036372 found 1036377
> parent transid verify failed on 337687707648 wanted 1036372 found 1036377
> parent transid verify failed on 337687707648 wanted 1036372 found 1036377
> parent transid verify failed on 337687707648 wanted 1036372 found 1036377
> Ignoring transid failure
>
>
>
> On Fri, Jan 1, 2016 at 8:04 PM, John Center <jlcenter15@gmail.com> wrote:
>> Hi Duncan,
>>
>> On Fri, Jan 1, 2016 at 12:05 PM, Duncan <1i5t5.duncan@cox.net> wrote:
>>> John Center posted on Fri, 01 Jan 2016 11:41:20 -0500 as excerpted:
>>>
>>>> If this doesn't resolve the problem, what would you recommend my next
>>>> steps should be?  I've been hesitant to run too many of the btrfs-tools,
>>>> mainly because I don't want to accidentally screw things up & I don't
>>>> always know how to interpret the results. (I ran btrfs-debug-tree,
>>>> hoping something obvious would show up.  Big mistake. 😋)
>>>
>>> LOLed at that debug-tree remark.  Been there (with other tools) myself.
>>>
>>> Well, I'm hoping someone who had the problem can confirm whether it's
>>> fixed in current kernels (scrub is one of those userspace commands that's
>>> mostly just a front-end to the kernel code which does the real work, so
>>> kernel version is the important thing for scrub).  I'm guessing so, and
>>> that you'll find the problem gone in 4.3.
>>>
>>> We'll cross the not-gone bridge if we get to it, but again, if the other
>>> people who had the similar problem can confirm whether it disappeared for
>>> them with the new kernel, it would help a lot, as there were enough such
>>> reports that if it's the same problem and still there for everyone (which
>>> I doubt as I expect there'd still be way more posts about it if so, but
>>> confirmation's always good), nothing to do but wait for a fix, while if
>>> not, and you still have your problem, then it's a different issue and the
>>> devs will need to work with you on a fix specific to your problem.
>>>
>> Ok, I'm at the next bridge. :-(  I upgraded the kernel to 4.4rc7 from
>> the Ubuntu Mainline archive & I just ran the scrub:
>>
>> john@mariposa:~$ sudo /sbin/btrfs scrub start -BdR /dev/md125p2
>> ERROR: scrubbing /dev/md125p2 failed for device id 1: ret=-1, errno=5
>> (Input/output error)
>> scrub device /dev/md125p2 (id 1) canceled
>> scrub started at Fri Jan  1 19:38:21 2016 and was aborted after 00:02:34
>> data_extents_scrubbed: 111031
>> tree_extents_scrubbed: 104061
>> data_bytes_scrubbed: 2549907456
>> tree_bytes_scrubbed: 1704935424
>> read_errors: 0
>> csum_errors: 0
>> verify_errors: 0
>> no_csum: 1573
>> csum_discards: 0
>> super_errors: 0
>> malloc_errors: 0
>> uncorrectable_errors: 0
>> unverified_errors: 0
>> corrected_errors: 0
>> last_physical: 4729667584
>>
>> I checked dmesg & this appeared:
>>
>> [11428.983355] BTRFS error (device md125p2): parent transid verify
>> failed on 241287168 wanted 33554449 found 17
>> [11431.028399] BTRFS error (device md125p2): parent transid verify
>> failed on 241287168 wanted 33554449 found 17
>>
>> Where do I go from here?
>>
>> Thanks for your help.
>>
>>     -John

Re: btrfs scrub failing

[Martin Steigerwald]

Am Freitag, 1. Januar 2016, 20:04:43 CET schrieb John Center:
> Hi Duncan,
> 
> On Fri, Jan 1, 2016 at 12:05 PM, Duncan <1i5t5.duncan@cox.net> wrote:
> > John Center posted on Fri, 01 Jan 2016 11:41:20 -0500 as excerpted:
> >> If this doesn't resolve the problem, what would you recommend my next
> >> steps should be?  I've been hesitant to run too many of the btrfs-tools,
> >> mainly because I don't want to accidentally screw things up & I don't
> >> always know how to interpret the results. (I ran btrfs-debug-tree,
> >> hoping something obvious would show up.  Big mistake. 😋)
> > 
> > LOLed at that debug-tree remark.  Been there (with other tools) myself.
> > 
> > Well, I'm hoping someone who had the problem can confirm whether it's
> > fixed in current kernels (scrub is one of those userspace commands that's
> > mostly just a front-end to the kernel code which does the real work, so
> > kernel version is the important thing for scrub).  I'm guessing so, and
> > that you'll find the problem gone in 4.3.
> > 
> > We'll cross the not-gone bridge if we get to it, but again, if the other
> > people who had the similar problem can confirm whether it disappeared for
> > them with the new kernel, it would help a lot, as there were enough such
> > reports that if it's the same problem and still there for everyone (which
> > I doubt as I expect there'd still be way more posts about it if so, but
> > confirmation's always good), nothing to do but wait for a fix, while if
> > not, and you still have your problem, then it's a different issue and the
> > devs will need to work with you on a fix specific to your problem.
> 
> Ok, I'm at the next bridge. :-(  I upgraded the kernel to 4.4rc7 from
> the Ubuntu Mainline archive & I just ran the scrub:
> 
> john@mariposa:~$ sudo /sbin/btrfs scrub start -BdR /dev/md125p2
> ERROR: scrubbing /dev/md125p2 failed for device id 1: ret=-1, errno=5
> (Input/output error)
> scrub device /dev/md125p2 (id 1) canceled
> scrub started at Fri Jan  1 19:38:21 2016 and was aborted after 00:02:34
> data_extents_scrubbed: 111031
> tree_extents_scrubbed: 104061
> data_bytes_scrubbed: 2549907456
> tree_bytes_scrubbed: 1704935424
> read_errors: 0
> csum_errors: 0
> verify_errors: 0
> no_csum: 1573
> csum_discards: 0
> super_errors: 0
> malloc_errors: 0
> uncorrectable_errors: 0
> unverified_errors: 0
> corrected_errors: 0
> last_physical: 4729667584
> 
> I checked dmesg & this appeared:
> 
> [11428.983355] BTRFS error (device md125p2): parent transid verify
> failed on 241287168 wanted 33554449 found 17
> [11431.028399] BTRFS error (device md125p2): parent transid verify
> failed on 241287168 wanted 33554449 found 17
> 
> Where do I go from here?

These and the other errors point at an issue with the filesystem structure.

As I never had to deal with that, I can only give generic advice:

1) Use latest stable btrfs-progs.

2) Umount the filesystem and run 

btrfs check (maybe with -p)

When it finds some errors, proceed with the following steps:

Without --repair or some of the other options that modify things it is read 
only.

3) If you can still access all the files, first thing to do is: rsync or 
otherwise backup them all to a different location, before attempting anything 
to repair the issue.

4) If you can´t access some files, you may try to use btrfs restore for 
restoring them.

5) Then, if you made sure you have an up-to-date backup run

btrfs --repair


Also watch out for other guidance you may receive her. My approach is based on 
what I would do. I never had the need to repair a BTRFS filesystem so far.

Thanks,
-- 
Martin

Re: btrfs scrub failing

[Martin Steigerwald]

Am Samstag, 2. Januar 2016, 11:35:51 CET schrieb Martin Steigerwald:
> Am Freitag, 1. Januar 2016, 20:04:43 CET schrieb John Center:
> > Hi Duncan,
> > 
> > On Fri, Jan 1, 2016 at 12:05 PM, Duncan <1i5t5.duncan@cox.net> wrote:
> > > John Center posted on Fri, 01 Jan 2016 11:41:20 -0500 as excerpted:
> > >> If this doesn't resolve the problem, what would you recommend my next
> > >> steps should be?  I've been hesitant to run too many of the
> > >> btrfs-tools,
> > >> mainly because I don't want to accidentally screw things up & I don't
> > >> always know how to interpret the results. (I ran btrfs-debug-tree,
> > >> hoping something obvious would show up.  Big mistake. 😋)
> > > 
> > > LOLed at that debug-tree remark.  Been there (with other tools) myself.
> > > 
> > > Well, I'm hoping someone who had the problem can confirm whether it's
> > > fixed in current kernels (scrub is one of those userspace commands
> > > that's
> > > mostly just a front-end to the kernel code which does the real work, so
> > > kernel version is the important thing for scrub).  I'm guessing so, and
> > > that you'll find the problem gone in 4.3.
> > > 
> > > We'll cross the not-gone bridge if we get to it, but again, if the other
> > > people who had the similar problem can confirm whether it disappeared
> > > for
> > > them with the new kernel, it would help a lot, as there were enough such
> > > reports that if it's the same problem and still there for everyone
> > > (which
> > > I doubt as I expect there'd still be way more posts about it if so, but
> > > confirmation's always good), nothing to do but wait for a fix, while if
> > > not, and you still have your problem, then it's a different issue and
> > > the
> > > devs will need to work with you on a fix specific to your problem.
> > 
> > Ok, I'm at the next bridge. :-(  I upgraded the kernel to 4.4rc7 from
> > the Ubuntu Mainline archive & I just ran the scrub:
> > 
> > john@mariposa:~$ sudo /sbin/btrfs scrub start -BdR /dev/md125p2
> > ERROR: scrubbing /dev/md125p2 failed for device id 1: ret=-1, errno=5
> > (Input/output error)
> > scrub device /dev/md125p2 (id 1) canceled
> > scrub started at Fri Jan  1 19:38:21 2016 and was aborted after 00:02:34
> > data_extents_scrubbed: 111031
> > tree_extents_scrubbed: 104061
> > data_bytes_scrubbed: 2549907456
> > tree_bytes_scrubbed: 1704935424
> > read_errors: 0
> > csum_errors: 0
> > verify_errors: 0
> > no_csum: 1573
> > csum_discards: 0
> > super_errors: 0
> > malloc_errors: 0
> > uncorrectable_errors: 0
> > unverified_errors: 0
> > corrected_errors: 0
> > last_physical: 4729667584
> > 
> > I checked dmesg & this appeared:
> > 
> > [11428.983355] BTRFS error (device md125p2): parent transid verify
> > failed on 241287168 wanted 33554449 found 17
> > [11431.028399] BTRFS error (device md125p2): parent transid verify
> > failed on 241287168 wanted 33554449 found 17
> > 
> > Where do I go from here?
> 
> These and the other errors point at an issue with the filesystem structure.
> 
> As I never had to deal with that, I can only give generic advice:
> 
> 1) Use latest stable btrfs-progs.
> 
> 2) Umount the filesystem and run
> 
> btrfs check (maybe with -p)
> 
> When it finds some errors, proceed with the following steps:
> 
> Without --repair or some of the other options that modify things it is read
> only.
> 
> 3) If you can still access all the files, first thing to do is: rsync or
> otherwise backup them all to a different location, before attempting
> anything to repair the issue.
> 
> 4) If you can´t access some files, you may try to use btrfs restore for
> restoring them.
> 
> 5) Then, if you made sure you have an up-to-date backup run
> 
> btrfs --repair

Before doing that, review:

https://btrfs.wiki.kernel.org/index.php/Btrfsck

to learn about other options.

Thanks,
-- 
Martin

Re: btrfs scrub failing

[Martin Steigerwald]

Am Samstag, 2. Januar 2016, 18:27:16 CET schrieb John Center:
> 😂Hi Martin,
> 
> > On Jan 2, 2016, at 6:41 AM, Martin Steigerwald <martin@lichtvoll.de>
> 
> wrote:
> > Am Samstag, 2. Januar 2016, 11:35:51 CET schrieb Martin Steigerwald:
> >> Am Freitag, 1. Januar 2016, 20:04:43 CET schrieb John Center:
> >>> Hi Duncan,
> >>> 
> >>>> On Fri, Jan 1, 2016 at 12:05 PM, Duncan <1i5t5.duncan@cox.net> wrote:
> >>> 
> >>>> John Center posted on Fri, 01 Jan 2016 11:41:20 -0500 as excerpted:
> >>> Where do I go from here?
> >> 
> >> These and the other errors point at an issue with the filesystem
> 
> structure.
> 
> >> As I never had to deal with that, I can only give generic advice:
> >> 
> >> 1) Use latest stable btrfs-progs.
> 
> I'm in the process of creating a live USB to boot with.  Since I'm running
> mdadm (imsm) I need to purge dmraid & install mdadm to assemble the drives
> first.  I also need to put the latest version of btrfs-progs on it, too.
>  (As a side note, things have been getting flaky with my workstation, so I
> guess I'm either going to fix this or rebuild it.  I have the data files
> backed up, it's just a pain to have to recreate the system again.)

I think you could even just run a GRML from USB stick and grab the sources via 
git clone and compile them there. Shouldn´t take long. I use:

git://git.kernel.org/pub/scm/linux/kernel/git/kdave/btrfs-progs.git

which has 4.3.1.

> >> 2) Umount the filesystem and run
> >> 
> >> btrfs check (maybe with -p)
> >> 
> >> When it finds some errors, proceed with the following steps:
> >> 
> >> Without --repair or some of the other options that modify things it is
> 
> read
> 
> >> only.
> >> 
> >> 3) If you can still access all the files, first thing to do is: rsync or
> >> otherwise backup them all to a different location, before attempting
> >> anything to repair the issue.
> >> 
> >> 4) If you can´t access some files, you may try to use btrfs restore for
> >> restoring them.
> >> 
> >> 5) Then, if you made sure you have an up-to-date backup run
> >> 
> >> btrfs --repair
> > 
> > Before doing that, review:
> > 
> > https://btrfs.wiki.kernel.org/index.php/Btrfsck
> > 
> > to learn about other options.
> 
> Ok, so "btrfs check -p" first to understand how bad the filesystem is
> corrupted.
> Should I then try to do a recovery mount, or should I run "btrfs check
> --repair -p" to try & fix it?  I'm not sure what a "mount -o ro,recovery"
> does.

Well, if "btrfs check -p" confirms that something needs fixing, you make sure 
you have a working backup *first*, either by rsync or btrfs restore if some 
files are inaccesible, but from what I remember you are able to access all 
files?

Then I´d say give btrfs check --repair a try.

I don´t know much about that mount -o ro,recovery does but from what I 
gathered so far I thought it is only needed when you *can´t* mount the 
filesystem anymore.

> If I have to reformat & reinstall Ubuntu, are there any recommended
> mkfs.btrfs options I should use?  Something that might help prevent
> problems in the future? 😂

Lol. :)

As far as I see mkfs.btrfs from btrfs-tools 4.3.1 already sets extref and 
skinny-metadata as well as 16 KiB node and leaf size by default, so as I 
recreated my /daten BTRFS filesystem due to the extent type mismatch errors 
(see other thread) I used use mkfs.btrfs -L daten to set a label.

Thanks,
-- 
Martin

Re: btrfs scrub failing

[Martin Steigerwald]

Am Sonntag, 3. Januar 2016, 02:02:12 CET schrieb John Center:
> Hi Martin & Duncan,

Hi John,

> Since I had a backup of my data, I first ran "btrfs check -p" on the
> unmounted array.  It first found 3 parent transid errors:
> 
> root@ubuntu:~# btrfs check -p /dev/md126p2
> Checking filesystem on /dev/md126p2
> UUID: 9b5a6959-7df1-4455-a643-d369487d24aa
> parent transid verify failed on 97763328 wanted 33736296 found 181864
> ...
> Ignoring transid failure
> parent transid verify failed on 241287168 wanted 33554449 found 17
> ...
> Ignoring transid failure
> parent transid verify failed on 1016217600 wanted 33556071 found 1639
> ...
> Ignoring transid failure
> 
> Then a huge number of bad extent mismatches:
> 
> bad extent [29360128, 29376512), type mismatch with chunk
> bad extent [29376512, 29392896), type mismatch with chunk
> ...
> bad extent [1039947448320, 1039947464704), type mismatch with chunk
> bad extent [1039948005376, 1039948021760), type mismatch with chunk

Due to these I recommend you redo the BTRFS filesystem using your backup. See 
the other thread where Duncan explained the situation that this may be a sign 
of a filesystem corruption introduced by a faulty mkfs.btrfs version.

I had this yesterday with one of my BTRFS filesystems and these type mismatch 
things didn´t go away with btrfs check --repair from btrfs-tools 4.3.1.

Also

> Next:
> 
> Couldn't find free space inode 1
> checking free space cache [o]
> parent transid verify failed on 241287168 wanted 33554449 found 17
> Ignoring transid failure
> checkingunresolved ref dir 418890 index 0 namelen 15 name umq-onetouch.ko
> filetype 1 errors 6, no dir index, no inode ref
>     unresolved ref dir 418890 index 8 namelen 15 name ums-onetouch.ko
> filetype 1 errors 1, no dir item

the further errors and

[…]
> Once it finished, I tried a recovery mount, which went ok.  Since I already
> had a backup of my data, I tried to run btrfs repair:
> […]
> Then it got stuck on the same error as before.  It appears to be a loop:
> 
> parent transid verify failed on 1016217600 wanted 33556071 found 1639
> Ignoring transid failure
> parent transid verify failed on 1016217600 wanted 33556071 found 1639
> Ignoring transid failure
> ...
[…]
> It's been running this way for over an hour now, never moving on from the
> same errors & the same couple of files.  I'm going to let it run overnight,
> but I don't have a lot of confidence that it will ever exit this loop.  Any
> recommendations as what I should do next?

is a clear sign to me that it likely is more effective to just redo the 
filesystem from scratch than trying to repair it with the limited capabilities 
of current btrfs check command.

So when you have a good backup of your data and want to be confident of a 
sound structure of the filesytem, redo it from scratch with latest btrfs-tools 
4.3.1.

Thats at least my take on this.

Thanks,
-- 
Martin

Re: btrfs scrub failing

[John Center]

Hi Martin,

> On Jan 3, 2016, at 5:06 AM, Martin Steigerwald <martin@lichtvoll.de> wrote:
> 
> Am Sonntag, 3. Januar 2016, 02:02:12 CET schrieb John Center:
>> Hi Martin & Duncan,
> 
> Hi John,
> 
>> Since I had a backup of my data, I first ran "btrfs check -p" on the
>> unmounted array.  It first found 3 parent transid errors:
>> 
>> root@ubuntu:~# btrfs check -p /dev/md126p2
>> Checking filesystem on /dev/md126p2
>> UUID: 9b5a6959-7df1-4455-a643-d369487d24aa
>> parent transid verify failed on 97763328 wanted 33736296 found 181864
>> ...
>> Ignoring transid failure
>> parent transid verify failed on 241287168 wanted 33554449 found 17
>> ...
>> Ignoring transid failure
>> parent transid verify failed on 1016217600 wanted 33556071 found 1639
>> ...
>> Ignoring transid failure
>> 
>> Then a huge number of bad extent mismatches:
>> 
>> bad extent [29360128, 29376512), type mismatch with chunk
>> bad extent [29376512, 29392896), type mismatch with chunk
>> ...
>> bad extent [1039947448320, 1039947464704), type mismatch with chunk
>> bad extent [1039948005376, 1039948021760), type mismatch with chunk
> 
> Due to these I recommend you redo the BTRFS filesystem using your backup. See 
> the other thread where Duncan explained the situation that this may be a sign 
> of a filesystem corruption introduced by a faulty mkfs.btrfs version.
> 
> I had this yesterday with one of my BTRFS filesystems and these type mismatch 
> things didn´t go away with btrfs check --repair from btrfs-tools 4.3.1.
> 
> Also
> 
>> Next:
>> 
>> Couldn't find free space inode 1
>> checking free space cache [o]
>> parent transid verify failed on 241287168 wanted 33554449 found 17
>> Ignoring transid failure
>> checkingunresolved ref dir 418890 index 0 namelen 15 name umq-onetouch.ko
>> filetype 1 errors 6, no dir index, no inode ref
>>    unresolved ref dir 418890 index 8 namelen 15 name ums-onetouch.ko
>> filetype 1 errors 1, no dir item
> 
> the further errors and
> 
> […]
>> Once it finished, I tried a recovery mount, which went ok.  Since I already
>> had a backup of my data, I tried to run btrfs repair:
>> […]
>> Then it got stuck on the same error as before.  It appears to be a loop:
>> 
>> parent transid verify failed on 1016217600 wanted 33556071 found 1639
>> Ignoring transid failure
>> parent transid verify failed on 1016217600 wanted 33556071 found 1639
>> Ignoring transid failure
>> ...
> […]
>> It's been running this way for over an hour now, never moving on from the
>> same errors & the same couple of files.  I'm going to let it run overnight,
>> but I don't have a lot of confidence that it will ever exit this loop.  Any
>> recommendations as what I should do next?
> 
> is a clear sign to me that it likely is more effective to just redo the 
> filesystem from scratch than trying to repair it with the limited capabilities 
> of current btrfs check command.
> 
> So when you have a good backup of your data and want to be confident of a 
> sound structure of the filesytem, redo it from scratch with latest btrfs-tools 
> 4.3.1.
> 
> Thats at least my take on this.
> 
Yeah, unfortunately I came to the same conclusion. I eventually killed the repair loop & ran check again with no luck - same errors, no repairs. I'm going to rebuild it, hopefully next weekend. 

Thanks again for your help!

    -John

Re: btrfs scrub failing

[John Center]

Hi Martin,

One thing I forgot, I did run btrfs-image & it appears to have successfully completed afaict. Do you think it would be useful to someone for future troubleshooting?

Thanks.

    -John

Sent from my iPhone

> On Jan 3, 2016, at 5:06 AM, Martin Steigerwald <martin@lichtvoll.de> wrote:
> 
> Am Sonntag, 3. Januar 2016, 02:02:12 CET schrieb John Center:
>> Hi Martin & Duncan,
> 
> Hi John,
> 
>> Since I had a backup of my data, I first ran "btrfs check -p" on the
>> unmounted array.  It first found 3 parent transid errors:
>> 
>> root@ubuntu:~# btrfs check -p /dev/md126p2
>> Checking filesystem on /dev/md126p2
>> UUID: 9b5a6959-7df1-4455-a643-d369487d24aa
>> parent transid verify failed on 97763328 wanted 33736296 found 181864
>> ...
>> Ignoring transid failure
>> parent transid verify failed on 241287168 wanted 33554449 found 17
>> ...
>> Ignoring transid failure
>> parent transid verify failed on 1016217600 wanted 33556071 found 1639
>> ...
>> Ignoring transid failure
>> 
>> Then a huge number of bad extent mismatches:
>> 
>> bad extent [29360128, 29376512), type mismatch with chunk
>> bad extent [29376512, 29392896), type mismatch with chunk
>> ...
>> bad extent [1039947448320, 1039947464704), type mismatch with chunk
>> bad extent [1039948005376, 1039948021760), type mismatch with chunk
> 
> Due to these I recommend you redo the BTRFS filesystem using your backup. See 
> the other thread where Duncan explained the situation that this may be a sign 
> of a filesystem corruption introduced by a faulty mkfs.btrfs version.
> 
> I had this yesterday with one of my BTRFS filesystems and these type mismatch 
> things didn´t go away with btrfs check --repair from btrfs-tools 4.3.1.
> 
> Also
> 
>> Next:
>> 
>> Couldn't find free space inode 1
>> checking free space cache [o]
>> parent transid verify failed on 241287168 wanted 33554449 found 17
>> Ignoring transid failure
>> checkingunresolved ref dir 418890 index 0 namelen 15 name umq-onetouch.ko
>> filetype 1 errors 6, no dir index, no inode ref
>>    unresolved ref dir 418890 index 8 namelen 15 name ums-onetouch.ko
>> filetype 1 errors 1, no dir item
> 
> the further errors and
> 
> […]
>> Once it finished, I tried a recovery mount, which went ok.  Since I already
>> had a backup of my data, I tried to run btrfs repair:
>> […]
>> Then it got stuck on the same error as before.  It appears to be a loop:
>> 
>> parent transid verify failed on 1016217600 wanted 33556071 found 1639
>> Ignoring transid failure
>> parent transid verify failed on 1016217600 wanted 33556071 found 1639
>> Ignoring transid failure
>> ...
> […]
>> It's been running this way for over an hour now, never moving on from the
>> same errors & the same couple of files.  I'm going to let it run overnight,
>> but I don't have a lot of confidence that it will ever exit this loop.  Any
>> recommendations as what I should do next?
> 
> is a clear sign to me that it likely is more effective to just redo the 
> filesystem from scratch than trying to repair it with the limited capabilities 
> of current btrfs check command.
> 
> So when you have a good backup of your data and want to be confident of a 
> sound structure of the filesytem, redo it from scratch with latest btrfs-tools 
> 4.3.1.
> 
> Thats at least my take on this.
> 
> Thanks,
> -- 
> Martin

Re: btrfs scrub failing

[Martin Steigerwald]

Am Sonntag, 3. Januar 2016, 17:33:03 CET schrieb John Center:
> Hi Martin,

Hi John,

> One thing I forgot, I did run btrfs-image & it appears to have successfully
> completed afaict. Do you think it would be useful to someone for future
> troubleshooting?

I leave that to the devs to decide. Maybe, if its not too large you can keep 
it for a while and ask the devs whether they want it. But, as it contains the 
type mismatch thing that they already know and fixed in mkfs.btrfs, if may not 
be of much use to them.

Thank you,
Martin

> 
> Thanks.
> 
>     -John
> 
> Sent from my iPhone
> 
> > On Jan 3, 2016, at 5:06 AM, Martin Steigerwald <martin@lichtvoll.de>
> > wrote:
> > 
> > Am Sonntag, 3. Januar 2016, 02:02:12 CET schrieb John Center:
> >> Hi Martin & Duncan,
> > 
> > Hi John,
> > 
> >> Since I had a backup of my data, I first ran "btrfs check -p" on the
> >> unmounted array.  It first found 3 parent transid errors:
> >> 
> >> root@ubuntu:~# btrfs check -p /dev/md126p2
> >> Checking filesystem on /dev/md126p2
> >> UUID: 9b5a6959-7df1-4455-a643-d369487d24aa
> >> parent transid verify failed on 97763328 wanted 33736296 found 181864
> >> ...
> >> Ignoring transid failure
> >> parent transid verify failed on 241287168 wanted 33554449 found 17
> >> ...
> >> Ignoring transid failure
> >> parent transid verify failed on 1016217600 wanted 33556071 found 1639
> >> ...
> >> Ignoring transid failure
> >> 
> >> Then a huge number of bad extent mismatches:
> >> 
> >> bad extent [29360128, 29376512), type mismatch with chunk
> >> bad extent [29376512, 29392896), type mismatch with chunk
> >> ...
> >> bad extent [1039947448320, 1039947464704), type mismatch with chunk
> >> bad extent [1039948005376, 1039948021760), type mismatch with chunk
> > 
> > Due to these I recommend you redo the BTRFS filesystem using your backup.
> > See the other thread where Duncan explained the situation that this may
> > be a sign of a filesystem corruption introduced by a faulty mkfs.btrfs
> > version.
> > 
> > I had this yesterday with one of my BTRFS filesystems and these type
> > mismatch things didn´t go away with btrfs check --repair from btrfs-tools
> > 4.3.1.
> > 
> > Also
> > 
> >> Next:
> >> 
> >> Couldn't find free space inode 1
> >> checking free space cache [o]
> >> parent transid verify failed on 241287168 wanted 33554449 found 17
> >> Ignoring transid failure
> >> checkingunresolved ref dir 418890 index 0 namelen 15 name umq-onetouch.ko
> >> filetype 1 errors 6, no dir index, no inode ref
> >> 
> >>    unresolved ref dir 418890 index 8 namelen 15 name ums-onetouch.ko
> >> 
> >> filetype 1 errors 1, no dir item
> > 
> > the further errors and
> > 
> > […]
> > 
> >> Once it finished, I tried a recovery mount, which went ok.  Since I
> >> already
> >> had a backup of my data, I tried to run btrfs repair:
> >> […]
> >> Then it got stuck on the same error as before.  It appears to be a loop:
> >> 
> >> parent transid verify failed on 1016217600 wanted 33556071 found 1639
> >> Ignoring transid failure
> >> parent transid verify failed on 1016217600 wanted 33556071 found 1639
> >> Ignoring transid failure
> >> ...
> > 
> > […]
> > 
> >> It's been running this way for over an hour now, never moving on from the
> >> same errors & the same couple of files.  I'm going to let it run
> >> overnight,
> >> but I don't have a lot of confidence that it will ever exit this loop. 
> >> Any
> >> recommendations as what I should do next?
> > 
> > is a clear sign to me that it likely is more effective to just redo the
> > filesystem from scratch than trying to repair it with the limited
> > capabilities of current btrfs check command.
> > 
> > So when you have a good backup of your data and want to be confident of a
> > sound structure of the filesytem, redo it from scratch with latest
> > btrfs-tools 4.3.1.
> > 
> > Thats at least my take on this.
> > 
> > Thanks,


-- 
Martin