Uncorrectable errors after rebooting with Magic Sysrq Keys

Uncorrectable errors after rebooting with Magic Sysrq Keys

[Niccolò Belli]

Hi,
Unfortunately because of buggy upstream support for my hardware (Dell XPS 
13 9343) I often have to force reboot using Magic Sysrq Keys (REISUB). In 
fact I have quite a few hangs, also the majority of times I am not able to 
shutdown without relying on REISUB. There are obviously times when even 
REISUB do not work (kernel is completely unresponsive), but the vast 
majority of times it works. What I do not understand is why Magic Sysrq 
Keys leave me with a damaged filesystem: shouldn't an emergency SYNC + read 
only remount be enough to secure my data? After rebooting with REISUB my 
system often complains about "read only" files and if I "stat" them I get 
"weird file". I often loose some of my desktop settings like the plasmoids 
I had on the desktop or my favourite applicatios I had in the menu, but 
what's even stranger is that I often magically recover them later, while 
doing exactly NOTHING to recover them. This behaviour scares me so much 
that I'm thinking about switching to another fs if I will not find a 
solution very soon.

The disk seems fine: https://bpaste.net/show/822d4b4ff902

dmesg: http://paste.pound-python.org/show/wVyHXXOw4emWmWFfVJHQ/

$ sudo btrfs scrub status /
[sudo] password di niko: 
scrub status for 28443ff1-5325-45f6-b879-dad895fcdcfb
        scrub started at Fri Apr 15 09:38:09 2016 and finished after 
00:08:41
        total bytes scrubbed: 133.94GiB with 10 errors
        error details: csum=10
        corrected errors: 0, uncorrectable errors: 10, unverified errors: 0

(yesterday there were 4 uncorrectable errors, but after today's reboot with 
Magic Sysrq Keys it is now 10)

Distro is Arch Linux, kernel is 4.6.0-rc3.
$ btrfs --version
btrfs-progs v4.4.1

Greetings,
Niccolò

Re: Uncorrectable errors after rebooting with Magic Sysrq Keys

[Qu Wenruo]

Would you please run "btrfs check --readonly <your dm0>" and paste the 
output?

The dmesg seems very impossible:

 > BTRFS error (device dm-0): bad tree block start 245497856 245498111

The later one is not even aligned to 2.

But you system still seems mountable as you succeeded in running btrfs 
scrub.

So I assume either the tree block is not a critical one or the copy 
saved you.

Thanks,
Qu

Niccolò Belli wrote on Fri Apr 15 2016 16:21:00 GMT+0800 (CST):
> Hi,
> Unfortunately because of buggy upstream support for my hardware (Dell
> XPS 13 9343) I often have to force reboot using Magic Sysrq Keys
> (REISUB). In fact I have quite a few hangs, also the majority of times I
> am not able to shutdown without relying on REISUB. There are obviously
> times when even REISUB do not work (kernel is completely unresponsive),
> but the vast majority of times it works. What I do not understand is why
> Magic Sysrq Keys leave me with a damaged filesystem: shouldn't an
> emergency SYNC + read only remount be enough to secure my data? After
> rebooting with REISUB my system often complains about "read only" files
> and if I "stat" them I get "weird file". I often loose some of my
> desktop settings like the plasmoids I had on the desktop or my favourite
> applicatios I had in the menu, but what's even stranger is that I often
> magically recover them later, while doing exactly NOTHING to recover
> them. This behaviour scares me so much that I'm thinking about switching
> to another fs if I will not find a solution very soon.
>
> The disk seems fine: https://bpaste.net/show/822d4b4ff902
>
> dmesg: http://paste.pound-python.org/show/wVyHXXOw4emWmWFfVJHQ/
>
> $ sudo btrfs scrub status /
> [sudo] password di niko: scrub status for
> 28443ff1-5325-45f6-b879-dad895fcdcfb
>         scrub started at Fri Apr 15 09:38:09 2016 and finished after
> 00:08:41
>         total bytes scrubbed: 133.94GiB with 10 errors
>         error details: csum=10
>         corrected errors: 0, uncorrectable errors: 10, unverified errors: 0
>
> (yesterday there were 4 uncorrectable errors, but after today's reboot
> with Magic Sysrq Keys it is now 10)
>
> Distro is Arch Linux, kernel is 4.6.0-rc3.
> $ btrfs --version
> btrfs-progs v4.4.1
>
> Greetings,
> Niccolò
> --
> To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>
>

Re: Uncorrectable errors after rebooting with Magic Sysrq Keys

[Niccolò Belli]

Hi,
Is it 100% safe to run a btrfs check without --repair?
Because otherwise I will have to wait for my new external drive to arrive 
and make a backup first.

Thanks,
Niccolò

On venerdì 15 aprile 2016 11:30:32 CEST, Qu Wenruo wrote:
> Would you please run "btrfs check --readonly <your dm0>" and 
> paste the output?
>
> The dmesg seems very impossible:
>
>> BTRFS error (device dm-0): bad tree block start 245497856 245498111
>
> The later one is not even aligned to 2.
>
> But you system still seems mountable as you succeeded in 
> running btrfs scrub.
>
> So I assume either the tree block is not a critical one or the 
> copy saved you.
>
> Thanks,
> Qu

Re: Uncorrectable errors after rebooting with Magic Sysrq Keys

[Hugo Mills]

[-- Attachment #1: Type: text/plain, Size: 1231 bytes --]

On Fri, Apr 15, 2016 at 05:42:21PM +0200, Niccolò Belli wrote:
> Hi,
> Is it 100% safe to run a btrfs check without --repair?
> Because otherwise I will have to wait for my new external drive to
> arrive and make a backup first.

   Yes, it's safe, unless there's some underlying hardware problem
that would cause the hardware to break even more by exercising it.

   For added peace of mind, use --readonly, which does nothing itself,
but conflicts with all of the options that might change things.

   Hugo.

> Thanks,
> Niccolò
> 
> On venerdì 15 aprile 2016 11:30:32 CEST, Qu Wenruo wrote:
> >Would you please run "btrfs check --readonly <your dm0>" and paste
> >the output?
> >
> >The dmesg seems very impossible:
> >
> >>BTRFS error (device dm-0): bad tree block start 245497856 245498111
> >
> >The later one is not even aligned to 2.
> >
> >But you system still seems mountable as you succeeded in running
> >btrfs scrub.
> >
> >So I assume either the tree block is not a critical one or the
> >copy saved you.
> >
> >Thanks,
> >Qu

-- 
Hugo Mills             | What part of "gestalt" don't you understand?
hugo@... carfax.org.uk |
http://carfax.org.uk/  |
PGP: E2AB1DE4          |

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 836 bytes --]

Re: Uncorrectable errors after rebooting with Magic Sysrq Keys

[Niccolò Belli]

https://bpaste.net/show/df9cc097c1da

This fs is *completely* FUCKED. Can't wait to get my hands on the external 
drive to be able to make a full backup.
Is it possible it is a kernel 4.6 regression? I had problems before, but 
nothing like this :(

Niccolò

Re: Uncorrectable errors after rebooting with Magic Sysrq Keys

[Roman Mamedov]

[-- Attachment #1: Type: text/plain, Size: 546 bytes --]

On Fri, 15 Apr 2016 17:42:21 +0200
Niccolò Belli <darkbasic@linuxsystems.it> wrote:

> Hi,
> Is it 100% safe to run a btrfs check without --repair?
> Because otherwise I will have to wait for my new external drive to arrive 
> and make a backup first.

In cases like these remember that you can use "blockdev --setro /dev/sdX", to
ensure that any program (even the root user with 'dd') can't write anything
to the block device anymore even if tried.

Use "blockdev --setrw /dev/sdX" after you finished.

-- 
With respect,
Roman

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 181 bytes --]

Re: Uncorrectable errors after rebooting with Magic Sysrq Keys

[Niccolò Belli]

I finally run a btrfs check --readonly on my fs, sorry if it took so long 
but it complained about the fs being mounted even if it was readonly, so I 
had to download a Fedora 24 alpha livecd to be able to run it.
Here it is (8.5MB): 
https://drive.google.com/open?id=0Bwe9Wtc-5xF1blJGMTNHaDdUQjg

In the meantime, since I suspected it may be a 4.6 regression, I switched 
back to 4.5.

P.S.
Scrub's uncorrectable errors went down from 10 to 4 by itself, without any 
apparent reason.

Niccolò

Re: Uncorrectable errors after rebooting with Magic Sysrq Keys

[Niccolò Belli]

Finally my external drive arrived and I've been able to make a backup and 
try btrfs check --repair.
Unfortunately btrfs check --repair got stuck in an infinite loop like this 
one (https://www.spinics.net/lists/linux-btrfs/msg54146.html) and after 
several hours of looping and several Gigabytes of logs I had to kill it, 
which gave me a completely fucked fs.
I still have backup images, so I can restore the old state and try again 
with updated tools (I used latest btrfs-progs 4.5.1, but I also tried 
4.4.1).
For those who didn't read the whole thread I can mount the fs, but it hangs 
while trying to read certain files and sometimes it remounts read-only. I'm 
pretty sure the culprit was a bug in 4.6-rc because problems started 
roughly after upgrading. Disk (an SSD) is fine. The fs is on top of 
dm-crypt and I always mounted it with 
"rw,relatime,ssd,space_cache,discard,compress=lzo,autodefrag".

You can find the whole logs here: 
https://drive.google.com/open?id=0Bwe9Wtc-5xF1Z2YwN1Y4U0ROSUU

01_scrub is the scrub output
02_check is the btrfs check output (14MB)
03_repair_short is the btrfs check --repair output truncated to 14MB

I hope someone will be able to help me recover my data, otherwise I will 
have to backup just the most important files and reinstall the whole system 
from scratch. Mounting the fs and doing a backup with cp -a wasn't a viable 
solution because it got stuck after several GBs.

Niccolò

P.S.
I changed my spf/dkim/dmarc settings, this email should no longer go into 
the spam folder, if it does please let me know. Thanks.