btrfs scrub crashes OS

btrfs scrub crashes OS

[Lukas Pirl]

Dear all,

I experience reproducible OS crashes when scrubbing a btrfs file system.
Apart from that, the file system mounts rw and is usable without any
problems (including modifying snapshots and all that).

When the system crashes (i.e., freezes), there are no errors printed to
the system logs or via `dmesg` (had a display connected).

Recovery is only possible via power-cycling the machine.

The host experienced a lot of crashes and ATA errors due to hardware
failures in the past.
To the best of my knowledge, the hardware is stable now.

`btrfs device stats` outputs zeros for all counters.

`btrfsck --readonly --mode lowmem` outputs a bunch of
  referencer count mismatch …
and
  ERROR: data extent[… …] backref lost
see https://pastebin.com/seC4fReP for the full log.

System info:
btrfs RAID 1 (~1.5 years old), 7 SATA HDDs
  MIXED_BACKREF, BIG_METADATA, EXTENDED_IREF, SKINNY_METADATA, NO_HOLES
  no quotas in use
  see also https://pastebin.com/4me6zDsN for more details
btrfs-progs v4.12
GNU/Linux 4.12.0-0.bpo.1-amd64 #1 SMP Debian 4.12.6-1~bpo9+1 x86_64

The question, obviously, is how can I make this fs "scrubable" again?
Are the errors found by btrfsck safe to repair using btrfsck or some
other tool?

Thank you so much in advance,

Lukas

Re: btrfs scrub crashes OS

[Qu Wenruo]

On 2017年09月25日 17:50, Lukas Pirl wrote:
> Dear all,
> 
> I experience reproducible OS crashes when scrubbing a btrfs file system.
> Apart from that, the file system mounts rw and is usable without any
> problems (including modifying snapshots and all that).
> 
> When the system crashes (i.e., freezes), there are no errors printed to
> the system logs or via `dmesg` (had a display connected).

Even no dmesg output using tty or netconsole?

That's strange.
Normally it should be kernel BUG_ON() to cause such problem.

And if the system is still responsible (either from TTY or ssh), is 
there anything strange like tons of IO or CPU usage?

> 
> Recovery is only possible via power-cycling the machine.
> 
> The host experienced a lot of crashes and ATA errors due to hardware
> failures in the past.
> To the best of my knowledge, the hardware is stable now.
> 
> `btrfs device stats` outputs zeros for all counters.
> 
> `btrfsck --readonly --mode lowmem` outputs a bunch of
>    referencer count mismatch …
> and
>    ERROR: data extent[… …] backref lost
> see https://pastebin.com/seC4fReP for the full log.

There is a known bug for lowmem mode to report such false alert.

Btrfs-progs v4.13 should have fixed it.

As long as v4.13 btrfs check reports no error, its metadata should be good.

> 
> System info:
> btrfs RAID 1 (~1.5 years old), 7 SATA HDDs

Oh, RAID1, so normal "btrfs check --check-data-csum" can't really check 
all data checksum. (It will pass 2nd mirror if 1st one matches the csum)

You could try the out-of-tree offline scrub to do a full scrub of your 
fs unmounted, so it won't crash your system (if nothing wrong happened)
https://github.com/gujx2017/btrfs-progs/tree/offline_scrub

>    MIXED_BACKREF, BIG_METADATA, EXTENDED_IREF, SKINNY_METADATA, NO_HOLES

Only NO_HOLES is not ordinary, but shouldn't cause a problem.


Without kernel backtrace, it's tricky to locate the problem.
So I would recommend to use netconsole (IIRC more reliable, as I use it 
on my test VM to capture the dying message) or TTY output to verify 
there is no kernel message/backtrace.

Thanks,
Qu

>    no quotas in use
>    see also https://pastebin.com/4me6zDsN for more details
> btrfs-progs v4.12
> GNU/Linux 4.12.0-0.bpo.1-amd64 #1 SMP Debian 4.12.6-1~bpo9+1 x86_64
> 
> The question, obviously, is how can I make this fs "scrubable" again?
> Are the errors found by btrfsck safe to repair using btrfsck or some
> other tool?
> 
> Thank you so much in advance,
> 
> Lukas
> --
> To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 

Re: btrfs scrub crashes OS

[Lukas Pirl]

Dear Qu,

thanks for your reply.

On 09/25/2017 12:19 PM, Qu Wenruo wrote as excerpted:
> Even no dmesg output using tty or netconsole?

And thanks for the pointer to netconsole, I tried that one.
No success. I set netconsole up, verified it worked, started a scrub,
the machine went away after a couple of hours, netconsole empty.

> That's strange.
> Normally it should be kernel BUG_ON() to cause such problem.
>  
> And if the system is still responsible (either from TTY or ssh), is
> there anything strange like tons of IO or CPU usage?

I can't tell, the machine just disappears from the network. Dead. IIRC,
it was also all dead when I sat in front of it.

> Btrfs-progs v4.13 should have fixed it.
> As long as v4.13 btrfs check reports no error, its metadata should be
> good.

I can try that one, if helpful.

> You could try the out-of-tree offline scrub to do a full scrub of your
> fs unmounted, so it won't crash your system (if nothing wrong happened)
> https://github.com/gujx2017/btrfs-progs/tree/offline_scrub

Did that, machine crashed again.

>>    MIXED_BACKREF, BIG_METADATA, EXTENDED_IREF, SKINNY_METADATA, NO_HOLES
> 
> Only NO_HOLES is not ordinary, but shouldn't cause a problem.

Would it be sensible to turn that feature off using `btrfstune` (if
possible at all)?

> Without kernel backtrace, it's tricky to locate the problem.
> So I would recommend to use netconsole (IIRC more reliable, as I use it
> on my test VM to capture the dying message) or TTY output to verify
> there is no kernel message/backtrace.

Yeah I see we are in a tricky situation here.

I will try to scrub with autodefrag and compression deactivated.

Could a full balance be of any help? At least to find out if it crashes
the machine as well?

Cheers,

Lukas

> Thanks,
> Qu
> 
>>    no quotas in use
>>    see also https://pastebin.com/4me6zDsN for more details
>> btrfs-progs v4.12
>> GNU/Linux 4.12.0-0.bpo.1-amd64 #1 SMP Debian 4.12.6-1~bpo9+1 x86_64
>>
>> The question, obviously, is how can I make this fs "scrubable" again?
>> Are the errors found by btrfsck safe to repair using btrfsck or some
>> other tool?
>>
>> Thank you so much in advance,
>>
>> Lukas
>> -- 
>> To unsubscribe from this list: send the line "unsubscribe
>> linux-btrfs" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>>

Re: btrfs scrub crashes OS

[Qu Wenruo]

On 2017年09月26日 16:34, Lukas Pirl wrote:
> Dear Qu,
> 
> thanks for your reply.
> 
> On 09/25/2017 12:19 PM, Qu Wenruo wrote as excerpted:
>> Even no dmesg output using tty or netconsole?
> 
> And thanks for the pointer to netconsole, I tried that one.
> No success. I set netconsole up, verified it worked, started a scrub,
> the machine went away after a couple of hours, netconsole empty.

Sad to hear that.
This means we have nothing to refer to, so it's really hard to continue 
investigating (if not impossible).

> 
>> That's strange.
>> Normally it should be kernel BUG_ON() to cause such problem.
>>   
>> And if the system is still responsible (either from TTY or ssh), is
>> there anything strange like tons of IO or CPU usage?
> 
> I can't tell, the machine just disappears from the network. Dead. IIRC,
> it was also all dead when I sat in front of it.
> 
>> Btrfs-progs v4.13 should have fixed it.
>> As long as v4.13 btrfs check reports no error, its metadata should be
>> good.
> 
> I can try that one, if helpful.
> 
>> You could try the out-of-tree offline scrub to do a full scrub of your
>> fs unmounted, so it won't crash your system (if nothing wrong happened)
>> https://github.com/gujx2017/btrfs-progs/tree/offline_scrub
> 
> Did that, machine crashed again.

This make things more weird.

Just in case, are you executing offline scrub by "btrfs scrub start 
--offline <device>"

If so, I think there may be some problem outside the btrfs territory.

Offline scrub has nothing to do with btrfs kernel module, it just reads 
out on-disk data and verify checksum in *user* space.

So if offline scrub can also screw up the system, it means there is 
something wrong in the disk IO routine, not btrfs.

And scrub can trigger it because normal btrfs IO won't try to read that 
part/mirror.

> 
>>>     MIXED_BACKREF, BIG_METADATA, EXTENDED_IREF, SKINNY_METADATA, NO_HOLES
>>
>> Only NO_HOLES is not ordinary, but shouldn't cause a problem.
> 
> Would it be sensible to turn that feature off using `btrfstune` (if
> possible at all)?
> 

Not possible, and I don't believe it's related to that feature.

>> Without kernel backtrace, it's tricky to locate the problem.
>> So I would recommend to use netconsole (IIRC more reliable, as I use it
>> on my test VM to capture the dying message) or TTY output to verify
>> there is no kernel message/backtrace.
> 
> Yeah I see we are in a tricky situation here.
> 
> I will try to scrub with autodefrag and compression deactivated. >
> Could a full balance be of any help? At least to find out if it crashes
> the machine as well?

According to your report, I think full balance may also crash your 
system, and may further crash your system every time you try to mount it.

So I won't recommend to do it.


What about trying to read all data out of your raw disk?
If offline crashes the system, reading the disk may crash it also.
Using dd to read each of your disk (with btrfs unmounted) may expose 
which disk caused the problem.

Thanks,
Qu

> 
> Cheers,
> 
> Lukas
> 
>> Thanks,
>> Qu
>>
>>>     no quotas in use
>>>     see also https://pastebin.com/4me6zDsN for more details
>>> btrfs-progs v4.12
>>> GNU/Linux 4.12.0-0.bpo.1-amd64 #1 SMP Debian 4.12.6-1~bpo9+1 x86_64
>>>
>>> The question, obviously, is how can I make this fs "scrubable" again?
>>> Are the errors found by btrfsck safe to repair using btrfsck or some
>>> other tool?
>>>
>>> Thank you so much in advance,
>>>
>>> Lukas
>>> -- 
>>> To unsubscribe from this list: send the line "unsubscribe
>>> linux-btrfs" in
>>> the body of a message to majordomo@vger.kernel.org
>>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>>>
> 
> --
> To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 

Re: btrfs scrub crashes OS

[Lukas Pirl]

Hi Qu,

On 09/26/2017 10:51 AM, Qu Wenruo wrote as excerpted:
> This make things more weird.
> Just in case, are you executing offline scrub by "btrfs scrub start
> --offline <device>"

Yes. I even got some output (pretty sure the last lines are missing due
to the crash):

WARNING: Offline scrub doesn't support extra options other than -r
[I gave -d as well]
Invalid mapping for 644337258496-644337332224, got
645348196352-646421938176
Couldn't map the block 644337258496
ERROR: failed to read out data at bytenr 644337258496 mirror 1
Invalid mapping for 653402148864-653402152960, got
653938130944-655011872768
Couldn't map the block 653402148864
ERROR: failed to read out data at bytenr 653402148864 mirror 1
Invalid mapping for 717315420160-717315526656, got
718362640384-719436382208
Couldn't map the block 717315420160
ERROR: failed to read out data at bytenr 717315420160 mirror 1
Invalid mapping for 875072008192-875072040960, got
875128946688-876202688512
Couldn't map the block 875072008192
ERROR: failed to read tree block 875072008192 mirror 1
ERROR: extent 875072008192 len 32768 CORRUPTED: all mirror(s)
corrupted, can't be recovered

Can I find out on which disk a mirror of a block is?

> If so, I think there may be some problem outside the btrfs territory.

Of course, that is a possibility…

> Offline scrub has nothing to do with btrfs kernel module, it just reads
> out on-disk data and verify checksum in *user* space.
> 
> So if offline scrub can also screw up the system, it means there is
> something wrong in the disk IO routine, not btrfs.
> 
> And scrub can trigger it because normal btrfs IO won't try to read that
> part/mirror.

…especially when considering this.

> What about trying to read all data out of your raw disk?
> If offline crashes the system, reading the disk may crash it also.
> Using dd to read each of your disk (with btrfs unmounted) may expose
> which disk caused the problem.

That it is good idea! Will go ahead.

Thanks for your help so far.

Re: btrfs scrub crashes OS

[Qu Wenruo]

On 2017年09月26日 17:26, Lukas Pirl wrote:
> Hi Qu,
> 
> On 09/26/2017 10:51 AM, Qu Wenruo wrote as excerpted:
>> This make things more weird.
>> Just in case, are you executing offline scrub by "btrfs scrub start
>> --offline <device>"
> 
> Yes. I even got some output (pretty sure the last lines are missing due
> to the crash):
> 
> WARNING: Offline scrub doesn't support extra options other than -r
> [I gave -d as well]
> Invalid mapping for 644337258496-644337332224, got
> 645348196352-646421938176
> Couldn't map the block 644337258496

This is strange, this means that we can't find a chunk map for a 72K 
length data extent.

Either the new mapper code has some bug, or it's a big problem.
But I think it's more possible for former case.

Would you please try to dump the chunk tree (which should be quite 
small) using the following command?

$ btrfs inspect-internal dump-tree -t chunk <device>

> ERROR: failed to read out data at bytenr 644337258496 mirror 1
> Invalid mapping for 653402148864-653402152960, got
> 653938130944-655011872768
> Couldn't map the block 653402148864
> ERROR: failed to read out data at bytenr 653402148864 mirror 1
> Invalid mapping for 717315420160-717315526656, got
> 718362640384-719436382208
> Couldn't map the block 717315420160
> ERROR: failed to read out data at bytenr 717315420160 mirror 1
> Invalid mapping for 875072008192-875072040960, got
> 875128946688-876202688512
> Couldn't map the block 875072008192
> ERROR: failed to read tree block 875072008192 mirror 1
> ERROR: extent 875072008192 len 32768 CORRUPTED: all mirror(s)
> corrupted, can't be recovered
> 
> Can I find out on which disk a mirror of a block is?

btrfs-map-logical can help you.
But I doubt if the offline scrub code, especially the new 
btrfs_map_block_v2() has hidden bug which caused the problem.

Withouth chunk tree dump, I am not which if it's a real bug or missing 
device.

Thanks,
Qu
> 
>> If so, I think there may be some problem outside the btrfs territory.
> 
> Of course, that is a possibility…
> 
>> Offline scrub has nothing to do with btrfs kernel module, it just reads
>> out on-disk data and verify checksum in *user* space.
>>
>> So if offline scrub can also screw up the system, it means there is
>> something wrong in the disk IO routine, not btrfs.
>>
>> And scrub can trigger it because normal btrfs IO won't try to read that
>> part/mirror.
> 
> …especially when considering this.
> 
>> What about trying to read all data out of your raw disk?
>> If offline crashes the system, reading the disk may crash it also.
>> Using dd to read each of your disk (with btrfs unmounted) may expose
>> which disk caused the problem.
> 
> That it is good idea! Will go ahead.
> 
> Thanks for your help so far.
> --
> To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 

Re: btrfs scrub crashes OS

[Lukas Pirl]

On 09/26/2017 11:36 AM, Qu Wenruo wrote as excerpted:
> This is strange, this means that we can't find a chunk map for a 72K
> length data extent.
> 
> Either the new mapper code has some bug, or it's a big problem.
> But I think it's more possible for former case.
> 
> Would you please try to dump the chunk tree (which should be quite
> small) using the following command?
> 
> $ btrfs inspect-internal dump-tree -t chunk <device>

Sure, happy to provide that:
  https://static.lukas-pirl.de/dump-chunk-tree.txt
(too large for Pastebin, file will probably go away in a couple of weeks).

Cheers,

Lukas

Re: btrfs scrub crashes OS

[Qu Wenruo]

On 2017年09月26日 19:50, Lukas Pirl wrote:
> On 09/26/2017 11:36 AM, Qu Wenruo wrote as excerpted:
>> This is strange, this means that we can't find a chunk map for a 72K
>> length data extent.
>>
>> Either the new mapper code has some bug, or it's a big problem.
>> But I think it's more possible for former case.
>>
>> Would you please try to dump the chunk tree (which should be quite
>> small) using the following command?
>>
>> $ btrfs inspect-internal dump-tree -t chunk <device>
> 
> Sure, happy to provide that:
>    https://static.lukas-pirl.de/dump-chunk-tree.txt
> (too large for Pastebin, file will probably go away in a couple of weeks).
> 

Found the needed info.

Your original data extent range is [644337258496, +72K].
Which is just in the range of the following 2 chunks:

---
	item 5 key (FIRST_CHUNK_TREE CHUNK_ITEM 643200712704) itemoff 15611 
itemsize 112
		length 1073741824 owner 2 stripe_len 65536 type DATA|RAID1
                 <snip>

	item 6 key (FIRST_CHUNK_TREE CHUNK_ITEM 645348196352) itemoff 15499 
itemsize 112
		length 1073741824 owner 2 stripe_len 65536 type DATA|RAID1
		io_align 65536 io_width 65536 sector_size 4096
                 <snip>
---
Those two chunks are covering the ranges of:
[643200712704, +1G)
[645348196352, +1G)

And no other chunk covers the hole between them.
But your original data extent range is in that hole.

So offline scrub output that error messages.
At least the chunk mapping code is correct.

Maybe something is wrong in the extent tree.
But anyway, it shouldn't cause too much trouble in offline scrub as you 
can see, it's a user space program and handles the problem quite well.
(Outputs error message and continue, without panic out)

So reading all your disk may still be needed to wipe out or confirm the 
possibility of the disk IO routine.

Thanks,
Qu

> Cheers,
> 
> Lukas
>