utils version and convert crash

utils version and convert crash

[Gareth Pye]

I'm getting some crashes when converting from RAID1 to RAID5, I know
that there was some issues recently but was lead to believe that there
were fixes that should have been in 4.3. (I'm using the ubuntu kernel
ppa teams 4.3 kernel)

One of the first things I checked was that I was using an up to date
btrfs util and it keeps reporting that I'm using version 4.0 (btrfs
--version). This is after confirming that my git clone is up to date,
the last commit to my master is talking about v4.3.1 and the
version.sh tells me it is 4.3.1 as well. Why the different reported
versions?

i'm converting in small chunks so I can then do a balance to recover
the partial blocks, the following command locks the pc up
occassionally. Mostly the balance completes happily on next boot but
I've seen it cause a reboot once.

btrfs fi balance start -dprofiles=raid1,convert=raid5,limit=1 /data

After running that several times I'm using the following to clean up
the partial RAID5 chunks:

btrfs fi balance start -dprofiles=raid5,usage=21 /data

I haven't noticed that clean up balance ever crashing.  Is this a
known bug or should I try and discover the cause better. Dmesg and
syslog don't have anything obvious in them.

-- 
Gareth Pye - blog.cerberos.id.au
Level 2 MTG Judge, Melbourne, Australia
"Dear God, I would like to file a bug report"

Re: utils version and convert crash

[Gareth Pye]

Poking around I just noticed that btrfs de stats /data points out that
3 of my drives have some read_io_errors. I'm guessing that is a bad
thing. I assume this would indicate bad hardware and would be a likely
cause of system crashes.

:(

On Tue, Dec 1, 2015 at 11:38 PM, Gareth Pye <gareth@cerberos.id.au> wrote:
> I'm getting some crashes when converting from RAID1 to RAID5, I know
> that there was some issues recently but was lead to believe that there
> were fixes that should have been in 4.3. (I'm using the ubuntu kernel
> ppa teams 4.3 kernel)
>
> One of the first things I checked was that I was using an up to date
> btrfs util and it keeps reporting that I'm using version 4.0 (btrfs
> --version). This is after confirming that my git clone is up to date,
> the last commit to my master is talking about v4.3.1 and the
> version.sh tells me it is 4.3.1 as well. Why the different reported
> versions?
>
> i'm converting in small chunks so I can then do a balance to recover
> the partial blocks, the following command locks the pc up
> occassionally. Mostly the balance completes happily on next boot but
> I've seen it cause a reboot once.
>
> btrfs fi balance start -dprofiles=raid1,convert=raid5,limit=1 /data
>
> After running that several times I'm using the following to clean up
> the partial RAID5 chunks:
>
> btrfs fi balance start -dprofiles=raid5,usage=21 /data
>
> I haven't noticed that clean up balance ever crashing.  Is this a
> known bug or should I try and discover the cause better. Dmesg and
> syslog don't have anything obvious in them.
>
> --
> Gareth Pye - blog.cerberos.id.au
> Level 2 MTG Judge, Melbourne, Australia
> "Dear God, I would like to file a bug report"



-- 
Gareth Pye - blog.cerberos.id.au
Level 2 MTG Judge, Melbourne, Australia
"Dear God, I would like to file a bug report"

Re: utils version and convert crash

[Duncan]

Gareth Pye posted on Tue, 01 Dec 2015 23:57:28 +1100 as excerpted:

> Poking around I just noticed that btrfs de stats /data points out that 3
> of my drives have some read_io_errors. I'm guessing that is a bad thing.
> I assume this would indicate bad hardware and would be a likely cause of
> system crashes.

Yes it's a bad thing and can indicate hardware issues.

But something that has tricked a number of posters in the past... the 
stats numbers don't reset on reboot, they're meant to indicate errors 
over time and thus continue to increase until manually reset (stats -z 
displays and resets).

In particular, if you had a system crash or hard power-down, it likely 
left some errors (fixable by scrub if you're running dup/raid other than 
raid0) that will show up when btrfs comes across them, that don't 
necessarily mean hardware problems (other than likely lack of a UPS).  
Because the numbers don't reset unless you do so manually, that crash/
hard-power-down could have been months ago...

So yes it's a bad thing, but potentially not as alarming as you might 
have first thought, if you had incorrectly assumed those error counts 
were since last boot. =:^)

-- 
Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman

Re: utils version and convert crash

[Austin S Hemmelgarn]

[-- Attachment #1: Type: text/plain, Size: 2160 bytes --]

On 2015-12-01 07:57, Gareth Pye wrote:
> Poking around I just noticed that btrfs de stats /data points out that
> 3 of my drives have some read_io_errors. I'm guessing that is a bad
> thing. I assume this would indicate bad hardware and would be a likely
> cause of system crashes.
In general, given that info, I would suggest that you do the following:
1. Run btrfs device stats -z to reset the counters (they're running 
counts stored on disk, not counts of recent errors or errors since last 
boot, so the numbers are probably over the lifetime of the filesystem 
right now).
2. Run a scrub on the filesystem (if you add -Bd, you get stats 
per-device when it's done, although it runs in the foreground).  If the 
scrub reports no errors, it's less likely that the issue is hardware 
than software (or just the system having crashed).
3. Regardless of the scrub results, use smartctl (usually found in a 
package called smartmontools or something similar) to check what the 
disk firmware thinks about how healthy the disk hardware is. 
Interpreting anything beyond the SMART attributes and the SMART health 
status is somewhat difficult without a lot of experience and some 
significant low-level knowledge of the hardware and software, but if the 
disk says it's healthy (check smartctl -H, and possibly smartctl -A), 
then it's _probably_ OK.
4. Check your kernel logs for messages about ATA link resets.  If you 
see a number of these, check your cables.  If the cables are fine 
(securely connected, don't appear damaged), then this may be an early 
indication of failing hardware (although there are other non-failure 
hardware issues this can be indicative of).

In general, read-errors are not a huge issue as long as you scrub the 
filesystem regularly (unless you get a lot in a short period of time, in 
which case you should be worried).  When you start getting write errors 
or link resets (like mentioned in step 4 above), or when the SMART 
pre-failure attributes hit their thresholds is when you should be 
getting worried and start actively looking for a replacement disk (and 
verifying your backups).


[-- Attachment #2: S/MIME Cryptographic Signature --]
[-- Type: application/pkcs7-signature, Size: 3019 bytes --]

Re: utils version and convert crash

[Duncan]

Gareth Pye posted on Tue, 01 Dec 2015 23:38:52 +1100 as excerpted:

> One of the first things I checked was that I was using an up to date
> btrfs util and it keeps reporting that I'm using version 4.0 (btrfs
> --version). This is after confirming that my git clone is up to date,
> the last commit to my master is talking about v4.3.1 and the version.sh
> tells me it is 4.3.1 as well. Why the different reported versions?

The gentoo btrfs-progs live-git version had a similar problem a month or 
two ago.  Turns out the main repo had changed policy.  As explained on 
the wiki, the master branch updates to latest release (so is basically no 
different than building from the latest release tarball), and the devel 
branch (which gentoo had been following) was no longer updated.  The devel 
branch on github (kdave) and repo.or.cz, however, should be current.

So if you're running into the same problem gentoo's live-git build did, 
it's likely because you're building the devel branch cloned from 
kernel.org, which is no longer updated.  Either switch to the github or 
repo.or.cz mirrors where devel is updated, or switch back to master 
branch if you prefer to stick with released versions but still want to 
fetch from git for bisect flexibility, etc, instead of sticking with the 
release tarballs.

Here's the gentoo bug I filed:

https://bugs.gentoo.org/show_bug.cgi?id=562636

... and the btrfs wiki page describing policy and listing the repos
(referenced in the bug as well):

https://btrfs.wiki.kernel.org/index.php/Btrfs_source_repositories


The other possibility is that you're running a stale binary, either a 
stale distro package binary, or a stale self-built, as there's currently 
posted patch saying the default-install switched to /usr/ with the switch 
to autotools, and switches it back to the traditional self-build default-
install to /usr/local/. 

-- 
Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman

Re: utils version and convert crash

[Gareth Pye]

On Wed, Dec 2, 2015 at 2:14 AM, Duncan <1i5t5.duncan@cox.net> wrote:
> So if you're running into the same problem gentoo's live-git build did,
> it's likely because you're building the devel branch cloned from
> kernel.org, which is no longer updated.


Woah, kernel.org is making a log that looks like it's up to date but
isn't that's awkward :(

Building now from the github you mentioned.

Also running a scrub, but I'm starting to suspect something else is
responsible. It ran fine overnight but crashed in less than a minute
after I logged back in on ssh this morning . . .

-- 
Gareth Pye - blog.cerberos.id.au
Level 2 MTG Judge, Melbourne, Australia
"Dear God, I would like to file a bug report"

Re: utils version and convert crash

[Austin S Hemmelgarn]

[-- Attachment #1: Type: text/plain, Size: 1256 bytes --]

On 2015-12-01 15:12, Gareth Pye wrote:
> On Wed, Dec 2, 2015 at 2:14 AM, Duncan <1i5t5.duncan@cox.net> wrote:
>> So if you're running into the same problem gentoo's live-git build did,
>> it's likely because you're building the devel branch cloned from
>> kernel.org, which is no longer updated.
>
>
> Woah, kernel.org is making a log that looks like it's up to date but
> isn't that's awkward :(
>
> Building now from the github you mentioned.
>
> Also running a scrub, but I'm starting to suspect something else is
> responsible. It ran fine overnight but crashed in less than a minute
> after I logged back in on ssh this morning . . .
>
Hmm, the fact that it's intermittent is the most concerning part IMHO. 
It means it's a lot harder to track down.  If your hard drives aren't 
any noisier than normal (most traditional hard disks get noticeably 
noisier when they're failing), then I'd suggest running something like 
memtest86+ for at least a full cycle with default options to verify if 
your RAM is working correctly.  Usually, when I see intermittent crashes 
like this it's either a race condition in software somewhere, or bad 
RAM, and it's a lot easier to test for bad RAM than it is to test for 
race conditions.


[-- Attachment #2: S/MIME Cryptographic Signature --]
[-- Type: application/pkcs7-signature, Size: 3019 bytes --]

Re: utils version and convert crash

[Gareth Pye]

Will do that once the scrub finishes/I get home from work.

On Wed, Dec 2, 2015 at 7:30 AM, Austin S Hemmelgarn
<ahferroin7@gmail.com> wrote:
> On 2015-12-01 15:12, Gareth Pye wrote:
>>
>> On Wed, Dec 2, 2015 at 2:14 AM, Duncan <1i5t5.duncan@cox.net> wrote:
>>>
>>> So if you're running into the same problem gentoo's live-git build did,
>>> it's likely because you're building the devel branch cloned from
>>> kernel.org, which is no longer updated.
>>
>>
>>
>> Woah, kernel.org is making a log that looks like it's up to date but
>> isn't that's awkward :(
>>
>> Building now from the github you mentioned.
>>
>> Also running a scrub, but I'm starting to suspect something else is
>> responsible. It ran fine overnight but crashed in less than a minute
>> after I logged back in on ssh this morning . . .
>>
> Hmm, the fact that it's intermittent is the most concerning part IMHO. It
> means it's a lot harder to track down.  If your hard drives aren't any
> noisier than normal (most traditional hard disks get noticeably noisier when
> they're failing), then I'd suggest running something like memtest86+ for at
> least a full cycle with default options to verify if your RAM is working
> correctly.  Usually, when I see intermittent crashes like this it's either a
> race condition in software somewhere, or bad RAM, and it's a lot easier to
> test for bad RAM than it is to test for race conditions.
>



-- 
Gareth Pye - blog.cerberos.id.au
Level 2 MTG Judge, Melbourne, Australia
"Dear God, I would like to file a bug report"

Re: utils version and convert crash

[Gareth Pye]

Output from scrub:
sudo btrfs scrub start -Bd /data
scrub device /dev/sdd (id 2) done
        scrub started at Wed Dec  2 07:04:08 2015 and finished after 08:53:53
        total bytes scrubbed: 2.47TiB with 0 errors
scrub device /dev/sde (id 3) done
        scrub started at Wed Dec  2 07:04:08 2015 and finished after 07:22:02
        total bytes scrubbed: 2.47TiB with 0 errors
scrub device /dev/sdf (id 4) done
        scrub started at Wed Dec  2 07:04:08 2015 and finished after 06:48:31
        total bytes scrubbed: 1.09TiB with 0 errors
scrub device /dev/sdg (id 5) done
        scrub started at Wed Dec  2 07:04:08 2015 and finished after 06:52:07
        total bytes scrubbed: 1.10TiB with 0 errors
scrub device /dev/sdh (id 6) done
        scrub started at Wed Dec  2 07:04:08 2015 and finished after 06:47:22
        total bytes scrubbed: 1.09TiB with 2 errors
        error details: read=2
        corrected errors: 2, uncorrectable errors: 0, unverified errors: 30
scrub device /dev/sdc (id 7) done
        scrub started at Wed Dec  2 07:04:08 2015 and finished after 04:11:59
        total bytes scrubbed: 430.52GiB with 0 errors
WARNING: errors detected during scrubbing, corrected

Looks like I have some issues. Going to confirm cables are all secure
and run a memtest.

On Wed, Dec 2, 2015 at 9:22 AM, Gareth Pye <gareth@cerberos.id.au> wrote:
> Will do that once the scrub finishes/I get home from work.
>
> On Wed, Dec 2, 2015 at 7:30 AM, Austin S Hemmelgarn
> <ahferroin7@gmail.com> wrote:
>> On 2015-12-01 15:12, Gareth Pye wrote:
>>>
>>> On Wed, Dec 2, 2015 at 2:14 AM, Duncan <1i5t5.duncan@cox.net> wrote:
>>>>
>>>> So if you're running into the same problem gentoo's live-git build did,
>>>> it's likely because you're building the devel branch cloned from
>>>> kernel.org, which is no longer updated.
>>>
>>>
>>>
>>> Woah, kernel.org is making a log that looks like it's up to date but
>>> isn't that's awkward :(
>>>
>>> Building now from the github you mentioned.
>>>
>>> Also running a scrub, but I'm starting to suspect something else is
>>> responsible. It ran fine overnight but crashed in less than a minute
>>> after I logged back in on ssh this morning . . .
>>>
>> Hmm, the fact that it's intermittent is the most concerning part IMHO. It
>> means it's a lot harder to track down.  If your hard drives aren't any
>> noisier than normal (most traditional hard disks get noticeably noisier when
>> they're failing), then I'd suggest running something like memtest86+ for at
>> least a full cycle with default options to verify if your RAM is working
>> correctly.  Usually, when I see intermittent crashes like this it's either a
>> race condition in software somewhere, or bad RAM, and it's a lot easier to
>> test for bad RAM than it is to test for race conditions.
>>
>
>
>
> --
> Gareth Pye - blog.cerberos.id.au
> Level 2 MTG Judge, Melbourne, Australia
> "Dear God, I would like to file a bug report"



-- 
Gareth Pye - blog.cerberos.id.au
Level 2 MTG Judge, Melbourne, Australia
"Dear God, I would like to file a bug report"

Re: utils version and convert crash

[Duncan]

Gareth Pye posted on Wed, 02 Dec 2015 18:07:48 +1100 as excerpted:

> Output from scrub:
> sudo btrfs scrub start -Bd /data

[Omitted no-error device reports.]

> scrub device /dev/sdh (id 6) done
>    scrub started at Wed Dec  2 07:04:08 2015 and finished after 06:47:22
>    total bytes scrubbed: 1.09TiB with 2 errors
>    error details: read=2
>    corrected errors: 2, uncorrectable errors: 0, unverified errors: 30

Also note those unverified errors...

I have quite a bit of experience with btrfs scrub as I ran with a failing 
ssd for awhile, using btrfs scrub on the multiple btrfs raid1 filesystems 
on parallel partitions on the failing ssd and another good one to correct 
the errors and continue operations.

Unverified errors are, I believe[1], errors where a metadata block 
holding checksums itself has an error, so the blocks its checksums in 
turn covered are not checksum-verified.

What that means in practice is that once the first metadata block error 
has been corrected in a first scrub run, a second scrub run can now check 
the blocks that were recorded as unverified errors in the first run, 
potentially finding and hopefully fixing additional errors, tho unless 
the problem's extreme, most of the unverifieds should end up being 
correct once they can be verified, with only a few possible further 
errors found.

Of course if some of these previously unverified blocks are themselves 
metadata blocks with further checksums, yet another run may be required.

Fortunately, these trees are quite wide (121 items according to an old 
post from Hugo I found myself rereading a few hours ago) and thus don't 
tend to be very deep -- I think I ended up rerunning scrub four times at 
one point, before both read and unverified errors went to zero, tho 
that's on relatively small partitioned-up ssd filesystems of under 50 gig 
usable capacity (pair-raid1, 50 gig per device), so I could see terabyte-
scale filesystems going to 6-7 levels.

And, again on a btrfs raid1 with a known failing device -- several 
thousand redirected sectors by the time I gave up and btrfs replaced -- 
generally each successive scrub run would return an order of magnitude or 
so fewer errors (corrected and unverified both) than the previous run, 
tho occasionally I'd hit a bad spot and the number would go up a bit in 
one run, before dropping an order of magnitude or so again on the next 
run.

So with only two corrected read-errors and 30 unverified, I'd expect 
maybe another one or two corrected read-errors on a second run, and 
probably no unverifieds, in which case a third run shouldn't be necessary 
unless you just want the peace of mind of seeing that no errors found 
message.  Tho of course if you're unlucky, one of those 30 will turn out 
to be a a read error on a full 121-item metadata block, so your 
unverifieds will go up for that run, before going down again in 
subsequent runs.

Of course with filesystems of under 50 gig capacity on fast ssds, a 
typical scrub ran in under a minute, so repeated scrubs to find and 
correct all errors wasn't a big deal, generally under 10 minutes 
including human response time.  On terabyte-scale spinning rust with 
scrubs taking hours, multiple scrubs could easily take a full 24-hour day 
or more! =:^(

So now that you did one scrub and did find errors, you do probably want 
to trace them down and correct the problem if possible, before running 
further scrubs to find and exterminate any errors still hiding behind 
unverified in the first run.  But once you're reasonably confident you're 
running a reliable system again, you probably do want to run further 
scrubs until that unverified count goes to zero (assuming no 
uncorrectable errors in the mean time).

---
[1] I'm not a dev and am not absolutely sure of the technical accuracy of 
this description, but from an admin's viewpoint it seems to be correct at 
least in practice, based on the fact that further scrubs as long as there 
were unverified errors often did find additional errors, while once the 
unverified count dropped to zero and the last read errors were corrected, 
further scrubs turned up no further errors.

-- 
Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman

Re: utils version and convert crash

[Gareth Pye]

Thanks for that info, ram appears to be checking out fine and smartctl
reported that the drives are old but one had some form of elevated
error. Looks like I might be buying a new drive.

On Wed, Dec 2, 2015 at 9:01 PM, Duncan <1i5t5.duncan@cox.net> wrote:
> Gareth Pye posted on Wed, 02 Dec 2015 18:07:48 +1100 as excerpted:
>
>> Output from scrub:
>> sudo btrfs scrub start -Bd /data
>
> [Omitted no-error device reports.]
>
>> scrub device /dev/sdh (id 6) done
>>    scrub started at Wed Dec  2 07:04:08 2015 and finished after 06:47:22
>>    total bytes scrubbed: 1.09TiB with 2 errors
>>    error details: read=2
>>    corrected errors: 2, uncorrectable errors: 0, unverified errors: 30
>
> Also note those unverified errors...
>
> I have quite a bit of experience with btrfs scrub as I ran with a failing
> ssd for awhile, using btrfs scrub on the multiple btrfs raid1 filesystems
> on parallel partitions on the failing ssd and another good one to correct
> the errors and continue operations.
>
> Unverified errors are, I believe[1], errors where a metadata block
> holding checksums itself has an error, so the blocks its checksums in
> turn covered are not checksum-verified.
>
> What that means in practice is that once the first metadata block error
> has been corrected in a first scrub run, a second scrub run can now check
> the blocks that were recorded as unverified errors in the first run,
> potentially finding and hopefully fixing additional errors, tho unless
> the problem's extreme, most of the unverifieds should end up being
> correct once they can be verified, with only a few possible further
> errors found.
>
> Of course if some of these previously unverified blocks are themselves
> metadata blocks with further checksums, yet another run may be required.
>
> Fortunately, these trees are quite wide (121 items according to an old
> post from Hugo I found myself rereading a few hours ago) and thus don't
> tend to be very deep -- I think I ended up rerunning scrub four times at
> one point, before both read and unverified errors went to zero, tho
> that's on relatively small partitioned-up ssd filesystems of under 50 gig
> usable capacity (pair-raid1, 50 gig per device), so I could see terabyte-
> scale filesystems going to 6-7 levels.
>
> And, again on a btrfs raid1 with a known failing device -- several
> thousand redirected sectors by the time I gave up and btrfs replaced --
> generally each successive scrub run would return an order of magnitude or
> so fewer errors (corrected and unverified both) than the previous run,
> tho occasionally I'd hit a bad spot and the number would go up a bit in
> one run, before dropping an order of magnitude or so again on the next
> run.
>
> So with only two corrected read-errors and 30 unverified, I'd expect
> maybe another one or two corrected read-errors on a second run, and
> probably no unverifieds, in which case a third run shouldn't be necessary
> unless you just want the peace of mind of seeing that no errors found
> message.  Tho of course if you're unlucky, one of those 30 will turn out
> to be a a read error on a full 121-item metadata block, so your
> unverifieds will go up for that run, before going down again in
> subsequent runs.
>
> Of course with filesystems of under 50 gig capacity on fast ssds, a
> typical scrub ran in under a minute, so repeated scrubs to find and
> correct all errors wasn't a big deal, generally under 10 minutes
> including human response time.  On terabyte-scale spinning rust with
> scrubs taking hours, multiple scrubs could easily take a full 24-hour day
> or more! =:^(
>
> So now that you did one scrub and did find errors, you do probably want
> to trace them down and correct the problem if possible, before running
> further scrubs to find and exterminate any errors still hiding behind
> unverified in the first run.  But once you're reasonably confident you're
> running a reliable system again, you probably do want to run further
> scrubs until that unverified count goes to zero (assuming no
> uncorrectable errors in the mean time).
>
> ---
> [1] I'm not a dev and am not absolutely sure of the technical accuracy of
> this description, but from an admin's viewpoint it seems to be correct at
> least in practice, based on the fact that further scrubs as long as there
> were unverified errors often did find additional errors, while once the
> unverified count dropped to zero and the last read errors were corrected,
> further scrubs turned up no further errors.
>
> --
> Duncan - List replies preferred.   No HTML msgs.
> "Every nonfree program has a lord, a master --
> and if you use the program, he is your master."  Richard Stallman
>
> --
> To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html



-- 
Gareth Pye - blog.cerberos.id.au
Level 2 MTG Judge, Melbourne, Australia
"Dear God, I would like to file a bug report"

Re: utils version and convert crash

[Austin S Hemmelgarn]

[-- Attachment #1: Type: text/plain, Size: 4584 bytes --]

On 2015-12-02 05:01, Duncan wrote:
> Gareth Pye posted on Wed, 02 Dec 2015 18:07:48 +1100 as excerpted:
>
>> Output from scrub:
>> sudo btrfs scrub start -Bd /data
>
> [Omitted no-error device reports.]
>
>> scrub device /dev/sdh (id 6) done
>>     scrub started at Wed Dec  2 07:04:08 2015 and finished after 06:47:22
>>     total bytes scrubbed: 1.09TiB with 2 errors
>>     error details: read=2
>>     corrected errors: 2, uncorrectable errors: 0, unverified errors: 30
>
> Also note those unverified errors...
>
> I have quite a bit of experience with btrfs scrub as I ran with a failing
> ssd for awhile, using btrfs scrub on the multiple btrfs raid1 filesystems
> on parallel partitions on the failing ssd and another good one to correct
> the errors and continue operations.
>
> Unverified errors are, I believe[1], errors where a metadata block
> holding checksums itself has an error, so the blocks its checksums in
> turn covered are not checksum-verified.
>
> What that means in practice is that once the first metadata block error
> has been corrected in a first scrub run, a second scrub run can now check
> the blocks that were recorded as unverified errors in the first run,
> potentially finding and hopefully fixing additional errors, tho unless
> the problem's extreme, most of the unverifieds should end up being
> correct once they can be verified, with only a few possible further
> errors found.
>
> Of course if some of these previously unverified blocks are themselves
> metadata blocks with further checksums, yet another run may be required.
>
> Fortunately, these trees are quite wide (121 items according to an old
> post from Hugo I found myself rereading a few hours ago) and thus don't
> tend to be very deep -- I think I ended up rerunning scrub four times at
> one point, before both read and unverified errors went to zero, tho
> that's on relatively small partitioned-up ssd filesystems of under 50 gig
> usable capacity (pair-raid1, 50 gig per device), so I could see terabyte-
> scale filesystems going to 6-7 levels.
>
> And, again on a btrfs raid1 with a known failing device -- several
> thousand redirected sectors by the time I gave up and btrfs replaced --
> generally each successive scrub run would return an order of magnitude or
> so fewer errors (corrected and unverified both) than the previous run,
> tho occasionally I'd hit a bad spot and the number would go up a bit in
> one run, before dropping an order of magnitude or so again on the next
> run.
>
> So with only two corrected read-errors and 30 unverified, I'd expect
> maybe another one or two corrected read-errors on a second run, and
> probably no unverifieds, in which case a third run shouldn't be necessary
> unless you just want the peace of mind of seeing that no errors found
> message.  Tho of course if you're unlucky, one of those 30 will turn out
> to be a a read error on a full 121-item metadata block, so your
> unverifieds will go up for that run, before going down again in
> subsequent runs.
>
> Of course with filesystems of under 50 gig capacity on fast ssds, a
> typical scrub ran in under a minute, so repeated scrubs to find and
> correct all errors wasn't a big deal, generally under 10 minutes
> including human response time.  On terabyte-scale spinning rust with
> scrubs taking hours, multiple scrubs could easily take a full 24-hour day
> or more! =:^(
>
> So now that you did one scrub and did find errors, you do probably want
> to trace them down and correct the problem if possible, before running
> further scrubs to find and exterminate any errors still hiding behind
> unverified in the first run.  But once you're reasonably confident you're
> running a reliable system again, you probably do want to run further
> scrubs until that unverified count goes to zero (assuming no
> uncorrectable errors in the mean time).
>
> ---
> [1] I'm not a dev and am not absolutely sure of the technical accuracy of
> this description, but from an admin's viewpoint it seems to be correct at
> least in practice, based on the fact that further scrubs as long as there
> were unverified errors often did find additional errors, while once the
> unverified count dropped to zero and the last read errors were corrected,
> further scrubs turned up no further errors.
>
AFAICT from reading the code, that is a correct assessment.  It would be 
kind of nice though if there was some way to tell scrub to recheck up to 
X many times if there are unverified errors...


[-- Attachment #2: S/MIME Cryptographic Signature --]
[-- Type: application/pkcs7-signature, Size: 3019 bytes --]

Re: utils version and convert crash

[Duncan]

Austin S Hemmelgarn posted on Wed, 02 Dec 2015 07:25:13 -0500 as
excerpted:

> On 2015-12-02 05:01, Duncan wrote:

[on unverified errors returned by scrub]
>>
>> Unverified errors are, I believe[1], errors where a metadata block
>> holding checksums itself has an error, so the blocks its checksums in
>> turn covered are not checksum-verified.
>>
>> What that means in practice is that once the first metadata block error
>> has been corrected in a first scrub run, a second scrub run can now
>> check the blocks that were recorded as unverified errors in the first
>> run, potentially finding and hopefully fixing additional errors[.]

>> ---
>> [1] I'm not a dev and am not absolutely sure of the technical accuracy
>> of this description, but from an admin's viewpoint it seems to be
>> correct at least in practice, based on the fact that further scrubs as
>> long as there were unverified errors often did find additional errors,
>> while once the unverified count dropped to zero and the last read
>> errors were corrected, further scrubs turned up no further errors.
>>
> AFAICT from reading the code, that is a correct assessment.  It would be
> kind of nice though if there was some way to tell scrub to recheck up to
> X many times if there are unverified errors...

Yes.  For me as explained it wasn't that big a deal as another scrub was 
another minute or less, but definitely on terabyte-scale filesystems on 
spinning rust, where scrubs take hours, having scrub be able to 
automatically track just the corrected errors along with their 
unverifieds, and rescan just those, should only take a matter of a few 
minutes more, while a full rescan of /everything/ would take the same 
number of hours yet again... and again if there's a third scan required, 
etc.

I'd say just make it automatic on corrected metadata errors as I can't 
think of a reason people wouldn't want it, given the time it would save 
over rerunning a full scrub over and over again, but making it an option 
would be fine with me too.

-- 
Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman

Re: utils version and convert crash

[Austin S Hemmelgarn]

[-- Attachment #1: Type: text/plain, Size: 2336 bytes --]

On 2015-12-02 08:45, Duncan wrote:
> Austin S Hemmelgarn posted on Wed, 02 Dec 2015 07:25:13 -0500 as
> excerpted:
>
>> On 2015-12-02 05:01, Duncan wrote:
>
> [on unverified errors returned by scrub]
>>>
>>> Unverified errors are, I believe[1], errors where a metadata block
>>> holding checksums itself has an error, so the blocks its checksums in
>>> turn covered are not checksum-verified.
>>>
>>> What that means in practice is that once the first metadata block error
>>> has been corrected in a first scrub run, a second scrub run can now
>>> check the blocks that were recorded as unverified errors in the first
>>> run, potentially finding and hopefully fixing additional errors[.]
>
>>> ---
>>> [1] I'm not a dev and am not absolutely sure of the technical accuracy
>>> of this description, but from an admin's viewpoint it seems to be
>>> correct at least in practice, based on the fact that further scrubs as
>>> long as there were unverified errors often did find additional errors,
>>> while once the unverified count dropped to zero and the last read
>>> errors were corrected, further scrubs turned up no further errors.
>>>
>> AFAICT from reading the code, that is a correct assessment.  It would be
>> kind of nice though if there was some way to tell scrub to recheck up to
>> X many times if there are unverified errors...
>
> Yes.  For me as explained it wasn't that big a deal as another scrub was
> another minute or less, but definitely on terabyte-scale filesystems on
> spinning rust, where scrubs take hours, having scrub be able to
> automatically track just the corrected errors along with their
> unverifieds, and rescan just those, should only take a matter of a few
> minutes more, while a full rescan of /everything/ would take the same
> number of hours yet again... and again if there's a third scan required,
> etc.
>
> I'd say just make it automatic on corrected metadata errors as I can't
> think of a reason people wouldn't want it, given the time it would save
> over rerunning a full scrub over and over again, but making it an option
> would be fine with me too.
>
I was thinking an option to do a full re-scrub, but having an automatic 
reparse of the metadata in a fixed metadata block would be a lot more 
efficient that what I was thinking :)


[-- Attachment #2: S/MIME Cryptographic Signature --]
[-- Type: application/pkcs7-signature, Size: 3019 bytes --]

Re: utils version and convert crash

[Gareth Pye]

Yeah having a scrub take 9 hours instead of 24 (+ latency of human
involvement) would be really nice.

On Thu, Dec 3, 2015 at 1:32 AM, Austin S Hemmelgarn
<ahferroin7@gmail.com> wrote:
> On 2015-12-02 08:45, Duncan wrote:
>>
>> Austin S Hemmelgarn posted on Wed, 02 Dec 2015 07:25:13 -0500 as
>> excerpted:
>>
>>> On 2015-12-02 05:01, Duncan wrote:
>>
>>
>> [on unverified errors returned by scrub]
>>>>
>>>>
>>>> Unverified errors are, I believe[1], errors where a metadata block
>>>> holding checksums itself has an error, so the blocks its checksums in
>>>> turn covered are not checksum-verified.
>>>>
>>>> What that means in practice is that once the first metadata block error
>>>> has been corrected in a first scrub run, a second scrub run can now
>>>> check the blocks that were recorded as unverified errors in the first
>>>> run, potentially finding and hopefully fixing additional errors[.]
>>
>>
>>>> ---
>>>> [1] I'm not a dev and am not absolutely sure of the technical accuracy
>>>> of this description, but from an admin's viewpoint it seems to be
>>>> correct at least in practice, based on the fact that further scrubs as
>>>> long as there were unverified errors often did find additional errors,
>>>> while once the unverified count dropped to zero and the last read
>>>> errors were corrected, further scrubs turned up no further errors.
>>>>
>>> AFAICT from reading the code, that is a correct assessment.  It would be
>>> kind of nice though if there was some way to tell scrub to recheck up to
>>> X many times if there are unverified errors...
>>
>>
>> Yes.  For me as explained it wasn't that big a deal as another scrub was
>> another minute or less, but definitely on terabyte-scale filesystems on
>> spinning rust, where scrubs take hours, having scrub be able to
>> automatically track just the corrected errors along with their
>> unverifieds, and rescan just those, should only take a matter of a few
>> minutes more, while a full rescan of /everything/ would take the same
>> number of hours yet again... and again if there's a third scan required,
>> etc.
>>
>> I'd say just make it automatic on corrected metadata errors as I can't
>> think of a reason people wouldn't want it, given the time it would save
>> over rerunning a full scrub over and over again, but making it an option
>> would be fine with me too.
>>
> I was thinking an option to do a full re-scrub, but having an automatic
> reparse of the metadata in a fixed metadata block would be a lot more
> efficient that what I was thinking :)
>



-- 
Gareth Pye - blog.cerberos.id.au
Level 2 MTG Judge, Melbourne, Australia
"Dear God, I would like to file a bug report"

Re: utils version and convert crash

[Gareth Pye]

Just noting that I left things till I put a 4.4 kernel on (4.4.3 as it
turns out) and now convert is going much nicer.

Well it's still got some silly thing where the newly allocated blocks
are mostly empty. It appears that the convert likes to take the 1Gig
RAID1 block and write it to a new RAID5 block (6x1Gig=5Gig capacity).
Meaning that the disks are likely to hit full during the convert. To
avoid that I'm looping a convert with a block limit with a balance to
target those blocks. The balance is pretty quick, but it does slow the
process down.

On Thu, Dec 3, 2015 at 9:14 AM, Gareth Pye <gareth@cerberos.id.au> wrote:
> Yeah having a scrub take 9 hours instead of 24 (+ latency of human
> involvement) would be really nice.
>
> On Thu, Dec 3, 2015 at 1:32 AM, Austin S Hemmelgarn
> <ahferroin7@gmail.com> wrote:
>> On 2015-12-02 08:45, Duncan wrote:
>>>
>>> Austin S Hemmelgarn posted on Wed, 02 Dec 2015 07:25:13 -0500 as
>>> excerpted:
>>>
>>>> On 2015-12-02 05:01, Duncan wrote:
>>>
>>>
>>> [on unverified errors returned by scrub]
>>>>>
>>>>>
>>>>> Unverified errors are, I believe[1], errors where a metadata block
>>>>> holding checksums itself has an error, so the blocks its checksums in
>>>>> turn covered are not checksum-verified.
>>>>>
>>>>> What that means in practice is that once the first metadata block error
>>>>> has been corrected in a first scrub run, a second scrub run can now
>>>>> check the blocks that were recorded as unverified errors in the first
>>>>> run, potentially finding and hopefully fixing additional errors[.]
>>>
>>>
>>>>> ---
>>>>> [1] I'm not a dev and am not absolutely sure of the technical accuracy
>>>>> of this description, but from an admin's viewpoint it seems to be
>>>>> correct at least in practice, based on the fact that further scrubs as
>>>>> long as there were unverified errors often did find additional errors,
>>>>> while once the unverified count dropped to zero and the last read
>>>>> errors were corrected, further scrubs turned up no further errors.
>>>>>
>>>> AFAICT from reading the code, that is a correct assessment.  It would be
>>>> kind of nice though if there was some way to tell scrub to recheck up to
>>>> X many times if there are unverified errors...
>>>
>>>
>>> Yes.  For me as explained it wasn't that big a deal as another scrub was
>>> another minute or less, but definitely on terabyte-scale filesystems on
>>> spinning rust, where scrubs take hours, having scrub be able to
>>> automatically track just the corrected errors along with their
>>> unverifieds, and rescan just those, should only take a matter of a few
>>> minutes more, while a full rescan of /everything/ would take the same
>>> number of hours yet again... and again if there's a third scan required,
>>> etc.
>>>
>>> I'd say just make it automatic on corrected metadata errors as I can't
>>> think of a reason people wouldn't want it, given the time it would save
>>> over rerunning a full scrub over and over again, but making it an option
>>> would be fine with me too.
>>>
>> I was thinking an option to do a full re-scrub, but having an automatic
>> reparse of the metadata in a fixed metadata block would be a lot more
>> efficient that what I was thinking :)
>>
>
>
>
> --
> Gareth Pye - blog.cerberos.id.au
> Level 2 MTG Judge, Melbourne, Australia
> "Dear God, I would like to file a bug report"



-- 
Gareth Pye - blog.cerberos.id.au
Level 2 MTG Judge, Melbourne, Australia