F21 fails to mount root part, btrfs check: Couldn't open file system

F21 fails to mount root part, btrfs check: Couldn't open file system

[Martin Langhoff]

Hi BTRFS folks,

one of my dev boxes is a Thinkpad x220, with a single hybrid
(HDD+Flash) disk, running F21 with BTRFS partitions for /home and / .

After losing power (ran out of battery, possibly while trying to
hibernate) -- the system will not boot. The initrd breaks out to a
shell where I find that the partition holding / is failing to mount.

 - journalctl/dmesg doesn't show anything of interest, though it says
there is no hibernate image to restore, and that it cannot mount
sysroot
 - btrfs rescue super-recover says there's no problem, all superblocks ok
 - btrfs rescue chunk-recover -vy takes a while, and says there's
nothing to recover
 - btrfs check (tried all options) -- says Couldn't open file system
 - kernel is 3.1.9-200.fc21
 - btrfs tools 3.18.1
 - disk is a Seagate 500GB Thin SSHD SATA 6Gb/s 64MB Cache 2.5-Inch
(ST500LM000) -- cost ~$100 about 2y ago

Is there anything I can do here? Anything to recover the partition?
Anything to diagnose things in a way that is useful to btrfs
development, and understanding of dataloss crashes? A failure of the
HDD cannot be ruled out, low power conditions, cheap consumer part...

There is no valuable data here, just have to reinstall F21.

cheers,


martin [ I am not on the list, CC appreciated... ]
-- 
 martin.langhoff@gmail.com
 -  ask interesting questions
 - don't get distracted with shiny stuff  - working code first
 ~ http://docs.moodle.org/en/User:Martin_Langhoff

Re: F21 fails to mount root part, btrfs check: Couldn't open file system

[Chris Murphy]

On Tue, Mar 31, 2015 at 12:55 PM, Martin Langhoff
<martin.langhoff@gmail.com> wrote:
> Hi BTRFS folks,
>
> one of my dev boxes is a Thinkpad x220, with a single hybrid
> (HDD+Flash) disk, running F21 with BTRFS partitions for /home and / .
>
> After losing power (ran out of battery, possibly while trying to
> hibernate) -- the system will not boot. The initrd breaks out to a
> shell where I find that the partition holding / is failing to mount.

There should be a reference to an rdsosreport.txt in /run/... so find
a way to get that posted somewhere.

>  - kernel is 3.1.9-200.fc21

This is probably 3.18.9, which is the current F21 kernel.

>A failure of the
> HDD cannot be ruled out, low power conditions, cheap consumer part...

Well you have to rule that out before anyone on this list can really
help. Try booting Fedora 21 install media, and using smartctl -x on
the drive.


-- 
Chris Murphy

Re: F21 fails to mount root part, btrfs check: Couldn't open file system

[Chris Murphy]

On Tue, Mar 31, 2015 at 2:09 PM, Chris Murphy <lists@colorremedies.com> wrote:

> Well you have to rule that out before anyone on this list can really
> help. Try booting Fedora 21 install media, and using smartctl -x on
> the drive.

While you're at it, try to mount the Btrfs volume in question normally
and report kernel messages. If mount fails, try it with -o recovery
mount option, and also report kernel messages and whether that fails.

-- 
Chris Murphy

Re: F21 fails to mount root part, btrfs check: Couldn't open file system

[Kai Krakow]

Chris Murphy <lists@colorremedies.com> schrieb:

> On Tue, Mar 31, 2015 at 2:09 PM, Chris Murphy <lists@colorremedies.com>
> wrote:
> 
>> Well you have to rule that out before anyone on this list can really
>> help. Try booting Fedora 21 install media, and using smartctl -x on
>> the drive.
> 
> While you're at it, try to mount the Btrfs volume in question normally
> and report kernel messages. If mount fails, try it with -o recovery
> mount option, and also report kernel messages and whether that fails.

I had this happen, too, lately. It's quite often happening after an unclean 
shutdown (which currently quite often happend to me due to the xorg intel 
driver having GPU freezes). SysRq+W shows that the mount process is locked 
somewhere in the btrfs code path and won't quit if Ctrl+C'd...

Only way to fix it was to btrfs-zero-log. But it still took some reboots 
from initramfs until it successfully mounted again (I could mount it in 
initramfs right after zero-log but upon reboot it hung again though at a 
different stage probably).

So I guess there's some race on the one hand (happens from time to time non-
related to fixing it with zero-log), and a deadlock on the other hand after 
some unclean shutdowns (more or less random).

My setup is 3-device btrfs-mraid1-draid0 on bcache. Bcache wasn't involved 
in the backtrace of SysRq+W, however. Apparently I don't have a screenshot 
of it because my smart phone is currently fried...

-- 
Replies to list only preferred.

Re: F21 fails to mount root part, btrfs check: Couldn't open file system

[Kai Krakow]

Kai Krakow <hurikhan77@gmail.com> schrieb:

> Chris Murphy <lists@colorremedies.com> schrieb:
> 
>> On Tue, Mar 31, 2015 at 2:09 PM, Chris Murphy <lists@colorremedies.com>
>> wrote:
>> 
>>> Well you have to rule that out before anyone on this list can really
>>> help. Try booting Fedora 21 install media, and using smartctl -x on
>>> the drive.
>> 
>> While you're at it, try to mount the Btrfs volume in question normally
>> and report kernel messages. If mount fails, try it with -o recovery
>> mount option, and also report kernel messages and whether that fails.
> 
> I had this happen, too, lately. It's quite often happening after an
> unclean shutdown (which currently quite often happend to me due to the
> xorg intel driver having GPU freezes). SysRq+W shows that the mount
> process is locked somewhere in the btrfs code path and won't quit if
> Ctrl+C'd...
> 
> Only way to fix it was to btrfs-zero-log. But it still took some reboots
> from initramfs until it successfully mounted again (I could mount it in
> initramfs right after zero-log but upon reboot it hung again though at a
> different stage probably).
> 
> So I guess there's some race on the one hand (happens from time to time
> non- related to fixing it with zero-log), and a deadlock on the other hand
> after some unclean shutdowns (more or less random).
> 
> My setup is 3-device btrfs-mraid1-draid0 on bcache. Bcache wasn't involved
> in the backtrace of SysRq+W, however. Apparently I don't have a screenshot
> of it because my smart phone is currently fried...

BTW: I tried all kernels from current 3.19.x back to 3.18.0 which still live 
on my boot partition - each with the same result and very similar backtrace 
(SysRq+W)...

-- 
Replies to list only preferred.

Re: F21 fails to mount root part, btrfs check: Couldn't open file system

[Chris Murphy]

On Tue, Mar 31, 2015 at 3:45 PM, Kai Krakow <hurikhan77@gmail.com> wrote:

> I had this happen, too, lately. It's quite often happening after an unclean
> shutdown (which currently quite often happend to me due to the xorg intel
> driver having GPU freezes). SysRq+W shows that the mount process is locked
> somewhere in the btrfs code path and won't quit if Ctrl+C'd...
>
> Only way to fix it was to btrfs-zero-log.

The i915 regression right now is really annoying. With a Samsung 840
EVO I've  had inexplicable and non-deterministic boot failures. When
running btrfs check from the initramfs  (booting with
rd.break=pre-mount) I get a very long pile of complaints... minutes of
scrolling text of horrible sounding problems. Yet the same btrfs-progs
and the same kernel from Fedora 22 install media, zero complaints, and
mounts fine.

So I have no idea what's going on right now. It even corrupts the EFI
System partition, these crashes.

-- 
Chris Murphy

Re: F21 fails to mount root part, btrfs check: Couldn't open file system

[Chris Murphy]

On Tue, Mar 31, 2015 at 4:15 PM, Chris Murphy <lists@colorremedies.com> wrote:

> The i915 regression right now is really annoying. With a Samsung 840
> EVO I've  had inexplicable and non-deterministic boot failures.

Clarification: the boot failures happen following the i915 panic and
subsequent forced power off.


-- 
Chris Murphy

Re: F21 fails to mount root part, btrfs check: Couldn't open file system

[Kai Krakow]

Chris Murphy <lists@colorremedies.com> schrieb:

> On Tue, Mar 31, 2015 at 4:15 PM, Chris Murphy <lists@colorremedies.com>
> wrote:
> 
>> The i915 regression right now is really annoying. With a Samsung 840
>> EVO I've  had inexplicable and non-deterministic boot failures.
> 
> Clarification: the boot failures happen following the i915 panic and
> subsequent forced power off.

Yeah I thought that, too, because after hitting "reset" it looked like one 
hard disk didn't appear in dmesg and thus btrfs didn't mount (btrfs-raid). 
So I turned the machine off completely because I had similar issues with 
i915 freezes and strange boot issues during the following boot before. It 
looks like the GPU is not necessarily completely reset when hitting the 
reset button. But that's another story.

In my case the hard disk was there - I didn't just scan hard enough through 
the huge pile of logs. I had to btrfs-zero-log, wrote "reboot" into the 
rescue shell, kernel came back, mount still locking up and sitting there 
until systemd decided to throw me to emergency after 5 minutes of waiting or 
so. I've rebooted again, machine came up. This was a few reboots after the 
machine was powered off, so I'd rule any GPU freeze artifacts out here. I 
just needed multiple reboots to arrange myself with my dracut/systemd combo 
super hero voodoo abilities (read: I cumbersome tried everything until one 
thing worked while swearing at my innocent monitor, well sort of, it's 
powered by the GPU).

On every reboot it felt like bcache was replaying cache transactions - but I 
think this is by design (read: bcache is always dirty, even after a clean 
shutdown, if using write-back mode) and not part of the problem.

-- 
Replies to list only preferred.

Re: F21 fails to mount root part, btrfs check: Couldn't open file system

[Martin Langhoff]

On Tue, Mar 31, 2015 at 4:11 PM, Chris Murphy <lists@colorremedies.com> wrote:
> While you're at it, try to mount the Btrfs volume in question normally
> and report kernel messages. If mount fails, try it with -o recovery
> mount option, and also report kernel messages and whether that fails.

Oh, I should have mentioned this -- in the context of the
initramfs/systemd diagnostic shell (which is single-user), it just
hangs. No messages.

I'll get a bootable usb going and try under that.

cheers,



m
-- 
 martin.langhoff@gmail.com
 -  ask interesting questions
 - don't get distracted with shiny stuff  - working code first
 ~ http://docs.moodle.org/en/User:Martin_Langhoff

Re: F21 fails to mount root part, btrfs check: Couldn't open file system

[Chris Murphy]

On Tue, Mar 31, 2015 at 2:39 PM, Matt Grant
<matt.grant@foodstuffs-si.co.nz> wrote:
> Seen this before at home.
>
> You have to mount -o recovery off a 3.19 kernel to fix it...
>
> If you can get the SSD out, attach it to a desktop, as there will be no Install CDs using 3.19 yet.

Fedora 22 Workstation alpha has 4.0.0 (rc1 I think), and the current
TC6 beta has 4.0.0-rc4. It's possible to use the netinstall, which is
much smaller, and use boot param single or rescue to avoid the
installer launching.

And actually 3.19.2 is the stable kernel for Fedora 21, with 3.19.3
just pushed today (take mirrors a day or two to catch up), not 3.18.9
as I reported earlier.

-- 
Chris Murphy

Re: F21 fails to mount root part, btrfs check: Couldn't open file system

[Martin Langhoff]

On Tue, Mar 31, 2015 at 4:09 PM, Chris Murphy <lists@colorremedies.com> wrote:
> There should be a reference to an rdsosreport.txt in /run/... so find
> a way to get that posted somewhere.

I'll try, but it truly says nothing of interest from a block device /
btrfs PoV. I have ample background debugging boot issues, disk
corruption, etc from years of work w OLPC.

>>  - kernel is 3.1.9-200.fc21
>
> This is probably 3.18.9, which is the current F21 kernel.

Correct, thanks. I typo'd that.

>>A failure of the
>> HDD cannot be ruled out, low power conditions, cheap consumer part...
>
> Well you have to rule that out before anyone on this list can really
> help. Try booting Fedora 21 install media, and using smartctl -x on
> the drive.

That's a good idea! I was referring to something else -- I guess what
I'm trying to say is: I'm not sure if this scrambled disk partition is
a btrfs/kernel bug, or the cheap HDD lied about flushing a write to
disk.

cheers,


m
-- 
 martin.langhoff@gmail.com
 -  ask interesting questions
 - don't get distracted with shiny stuff  - working code first
 ~ http://docs.moodle.org/en/User:Martin_Langhoff

Re: F21 fails to mount root part, btrfs check: Couldn't open file system

[Chris Murphy]

On Tue, Mar 31, 2015 at 5:54 PM, Martin Langhoff
<martin.langhoff@gmail.com> wrote:
> On Tue, Mar 31, 2015 at 4:09 PM, Chris Murphy <lists@colorremedies.com> wrote:
>> There should be a reference to an rdsosreport.txt in /run/... so find
>> a way to get that posted somewhere.
>
> I'll try, but it truly says nothing of interest from a block device /
> btrfs PoV. I have ample background debugging boot issues, disk
> corruption, etc from years of work w OLPC.

If there is no reference in this dracut shell to rdsosreport.txt, then use:
journalctl -b -l -o short-monotonic

You can mount anything at /sysroot including the boot partition if you
want, or a USB stick. The usual directories aren't available in the
initramfs before switchroot happens.


> That's a good idea! I was referring to something else -- I guess what
> I'm trying to say is: I'm not sure if this scrambled disk partition is
> a btrfs/kernel bug, or the cheap HDD lied about flushing a write to
> disk.

This is the realm of both esoteric knowledge and an active area of
research how to get reliable information about "what happened when the
power cut out." So you're not the only one not sure.


-- 
Chris Murphy

Re: F21 fails to mount root part, btrfs check: Couldn't open file system

[Martin Langhoff]

Hi Chris, list,

thanks for your debugging ideas so far. Now this gets interesting. I
booted off a LiveUSB disk, and it just mounted sysroot. WTH?

See below. Perhaps the newer kernel (in latest F21) has regressed in
handling some kinds of errors during mount, or the dracut/systemd
mounting process is less resilient than mounting under a fully booted
system?


[root@localhost liveuser]# uname -a
Linux localhost 3.17.4-301.fc21.x86_64 #1 SMP Thu Nov 27 19:09:10 UTC
2014 x86_64 x86_64 x86_64 GNU/Linux

## Before booting into liveUSB, I made a copy of
## rdsosreport.txt in the /home partition
## which is a separate btrfs fs, and seems
## to not be affected by the problem at all
[root@localhost liveuser]# mkdir /myhome
[root@localhost liveuser]# mkdir /mysysroot
[root@localhost liveuser]# mount /dev/sda2 /myhome
[root@localhost liveuser]# ls /myhome
home  rdsosreport.txt

[root@localhost liveuser]# fpaste < /myhome/rdsosreport.txt
Uploading (93.4KiB)...
http://ur1.ca/k2zue -> http://paste.fedoraproject.org/205971/01928142

#### Strange, on first book from live USB F21 image, it just mounts
## (I tried about half a dozen cold boots earlier -- all resulting in
##  the same initramfs/dracut/systemd emergency shell...)
[root@localhost liveuser]# mount /dev/sda6 /mysysroot

Apr 01 11:26:51 localhost kernel: BTRFS info (device sda6): disk space
caching is enabled
Apr 01 11:26:56 localhost kernel: BTRFS: checking UUID tree
Apr 01 11:26:56 localhost kernel: SELinux: initialized (dev sda6, type
btrfs), uses xattr

[root@localhost liveuser]# ls /mysysroot
root
[root@localhost liveuser]# ls /mysysroot/root
bin  boot  dev  etc  home  lib  lib64  media  mnt  opt  proc  root
run  sbin  srv  sys  sysroot  tmp  usr  var

[root@localhost liveuser]# umount /dev/sda6
[root@localhost liveuser]# btrfs check /dev/sda6
Checking filesystem on /dev/sda6
UUID: 94637b35-a294-4be2-aa47-82c52d6d53ef
checking extents
checking free space cache
checking fs roots
root 256 inode 39841 errors 400, nbytes wrong
found 7747100703 bytes used err is 1
total csum bytes: 11912932
total tree bytes: 476725248
total fs tree bytes: 434733056
total extent tree bytes: 22986752
btree space waste bytes: 83962424
file data blocks allocated: 30820143104
 referenced 11997040640
Btrfs v3.17

[root@localhost liveuser]# btrfs check --repair /dev/sda6
enabling repair mode
Fixed 0 roots.
Checking filesystem on /dev/sda6
UUID: 94637b35-a294-4be2-aa47-82c52d6d53ef
checking extents
checking free space cache
cache and super generation don't match, space cache will be invalidated
checking fs roots
root 256 inode 39841 errors 400, nbytes wrong
found 7747100703 bytes used err is 1
total csum bytes: 11912932
total tree bytes: 476725248
total fs tree bytes: 434733056
total extent tree bytes: 22986752
btree space waste bytes: 83962424
file data blocks allocated: 30820143104
 referenced 11997040640
Btrfs v3.17



EOM

cheers,



martin
-- 
 martin.langhoff@gmail.com
 -  ask interesting questions
 - don't get distracted with shiny stuff  - working code first
 ~ http://docs.moodle.org/en/User:Martin_Langhoff

Re: F21 fails to mount root part, btrfs check: Couldn't open file system

[Martin Langhoff]

On Wed, Apr 1, 2015 at 11:42 AM, Martin Langhoff
<martin.langhoff@gmail.com> wrote:
> See below. Perhaps the newer kernel (in latest F21) has regressed in
> handling some kinds of errors during mount, or the dracut/systemd
> mounting process is less resilient than mounting under a fully booted
> system?

This is getting even more interesting.

Under 3.17.4-301.fc21.x86 from LiveUSB, I could mount, even repair the disk.

Since the repair, the on-disk latest kernel (3.18.9-200.fc21) tries to
boot, but dracut/systemd time out on mounting sysroot after waiting
for quite a while. I don't get a dracut shell anymore so the failure
mode has changed. I may try to set a breakpoint to force a shell.

I do have an earlier F21 kernel on disk-- 3.18.7-200.fc21 -- and this
boots the system without a glitch. After a complete boot with
3.18.7.200, clean shutdown and booting into 3.18.9-200 is still
broken, same failure mode.

Will try to capture some info from a dracut breakpoint  (I'll try
mount). At this point this really looks like a regression.

cheers,



martin
-- 
 martin.langhoff@gmail.com
 -  ask interesting questions
 - don't get distracted with shiny stuff  - working code first
 ~ http://docs.moodle.org/en/User:Martin_Langhoff

Re: F21 fails to mount root part, btrfs check: Couldn't open file system

[Martin Langhoff]

On Wed, Apr 1, 2015 at 12:15 PM, Martin Langhoff
<martin.langhoff@gmail.com> wrote:
> Will try to capture some info from a dracut breakpoint  (I'll try
> mount). At this point this really looks like a regression.

After a couple alternating boots, the problem vanished :-/:

 - 3.18.7-200.fc21 - success
 - 3.18.9-200.fc21 - timeout
 - 3.18.7-200.fc21 - success
 - 3.18.9-200.fc21 - timeout
 - 3.18.7-200.fc21 - success
 - 3.18.9-200.fc21 - with rd.shell rd.break=mount rd.debug --> success
 - 3.18.9-200.fc21 - with no special params -- success

(all boots had rhgb and quiet params removed)

So problem is gone, yet I feel like this
https://c2.staticflickr.com/4/3438/4593531893_f67a757fa1.jpg



m
-- 
 martin.langhoff@gmail.com
 -  ask interesting questions
 - don't get distracted with shiny stuff  - working code first
 ~ http://docs.moodle.org/en/User:Martin_Langhoff

Re: F21 fails to mount root part, btrfs check: Couldn't open file system

[Chris Murphy]

On Wed, Apr 1, 2015 at 10:15 AM, Martin Langhoff
<martin.langhoff@gmail.com> wrote:
> On Wed, Apr 1, 2015 at 11:42 AM, Martin Langhoff
> <martin.langhoff@gmail.com> wrote:
>> See below. Perhaps the newer kernel (in latest F21) has regressed in
>> handling some kinds of errors during mount, or the dracut/systemd
>> mounting process is less resilient than mounting under a fully booted
>> system?
>
> This is getting even more interesting.
>
> Under 3.17.4-301.fc21.x86 from LiveUSB, I could mount, even repair the disk.
>
> Since the repair, the on-disk latest kernel (3.18.9-200.fc21) tries to
> boot, but dracut/systemd time out on mounting sysroot after waiting
> for quite a while. I don't get a dracut shell anymore so the failure
> mode has changed. I may try to set a breakpoint to force a shell.
>
> I do have an earlier F21 kernel on disk-- 3.18.7-200.fc21 -- and this
> boots the system without a glitch. After a complete boot with
> 3.18.7.200, clean shutdown and booting into 3.18.9-200 is still
> broken, same failure mode.
>
> Will try to capture some info from a dracut breakpoint  (I'll try
> mount). At this point this really looks like a regression.

Yeah I don't know what's going on, but with a new file system, and
disabled i915 to avoid crashes, and thus no crashes since the new fs
was created, I get boot failure with 3.19.3 but not 3.19.2, and I
can't figure out why. I get the systemd cylon eye with 5 services
pending so I can't actually tell which one it's hung up on, but one of
them is looking for the fs volume UUID and apparently can't find it
which is completely bogus.

-- 
Chris Murphy

Re: F21 fails to mount root part, btrfs check: Couldn't open file system

[Chris Murphy]

On Wed, Apr 1, 2015 at 9:42 AM, Martin Langhoff
<martin.langhoff@gmail.com> wrote:
# mount /dev/sda6 /mysysroot
>
> Apr 01 11:26:51 localhost kernel: BTRFS info (device sda6): disk space
> caching is enabled
> Apr 01 11:26:56 localhost kernel: BTRFS: checking UUID tree
> Apr 01 11:26:56 localhost kernel: SELinux: initialized (dev sda6, type
> btrfs), uses xattr

Right so it mounts fine with no errors from live media, but won't
mount at boot time. Same problem I was having.


# btrfs check /dev/sda6
> Checking filesystem on /dev/sda6
> UUID: 94637b35-a294-4be2-aa47-82c52d6d53ef
> checking extents
> checking free space cache
> checking fs roots
> root 256 inode 39841 errors 400, nbytes wrong

mount /dev/sda6 /mnt
btrfs inspect-internal inode-resolve 39841 /mnt

It should resolve a path to file for that inode. Chances are you can
just use cp to make a new copy of it, delete the original, and rename
the copy to match the original file name. Unmount. And now the btrfs
check error won't happen.



> [root@localhost liveuser]# btrfs check --repair /dev/sda6
> enabling repair mode
> Fixed 0 roots.
> Checking filesystem on /dev/sda6
> UUID: 94637b35-a294-4be2-aa47-82c52d6d53ef
> checking extents
> checking free space cache
> cache and super generation don't match, space cache will be invalidated
> checking fs roots
> root 256 inode 39841 errors 400, nbytes wrong
> found 7747100703 bytes used err is 1
> total csum bytes: 11912932
> total tree bytes: 476725248
> total fs tree bytes: 434733056
> total extent tree bytes: 22986752
> btree space waste bytes: 83962424
> file data blocks allocated: 30820143104
>  referenced 11997040640
> Btrfs v3.17

Yeah I don't know what this errors 400 nbytes wrong means, but at the
moment btrfs-progs doesn't fix it.


-- 
Chris Murphy

Re: F21 fails to mount root part, btrfs check: Couldn't open file system

[Martin Langhoff]

On Wed, Apr 1, 2015 at 1:03 PM, Chris Murphy <lists@colorremedies.com> wrote:
> mount /dev/sda6 /mnt
> btrfs inspect-internal inode-resolve 39841 /mnt

on the booted system...
# uname -a
Linux tp-martin.remote-learner.net 3.18.9-200.fc21.x86_64 #1 SMP Mon
Mar 9 15:10:50 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux
# btrfs inspect-internal inode-resolve 39841 /
//etc/shadow-
# diff -u /etc/shadow{,-}
--- /etc/shadow 2015-03-04 02:26:59.478255332 -0500
+++ /etc/shadow-        2015-03-04 02:26:59.000000000 -0500
@@ -42,4 +42,3 @@
 systemd-timesync:!!:16498::::::
 systemd-network:!!:16498::::::
 systemd-resolve:!!:16498::::::
-systemd-bus-proxy:!!:16498::::::

Bizarre.

cheers,



m
-- 
 martin.langhoff@gmail.com
 -  ask interesting questions
 - don't get distracted with shiny stuff  - working code first
 ~ http://docs.moodle.org/en/User:Martin_Langhoff

Re: F21 fails to mount root part, btrfs check: Couldn't open file system

[Chris Murphy]

On Wed, Apr 1, 2015 at 11:26 AM, Martin Langhoff
<martin.langhoff@gmail.com> wrote:
> On Wed, Apr 1, 2015 at 1:03 PM, Chris Murphy <lists@colorremedies.com> wrote:
>> mount /dev/sda6 /mnt
>> btrfs inspect-internal inode-resolve 39841 /mnt
>
> on the booted system...
> # uname -a
> Linux tp-martin.remote-learner.net 3.18.9-200.fc21.x86_64 #1 SMP Mon
> Mar 9 15:10:50 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux
> # btrfs inspect-internal inode-resolve 39841 /
> //etc/shadow-
> # diff -u /etc/shadow{,-}
> --- /etc/shadow 2015-03-04 02:26:59.478255332 -0500
> +++ /etc/shadow-        2015-03-04 02:26:59.000000000 -0500
> @@ -42,4 +42,3 @@
>  systemd-timesync:!!:16498::::::
>  systemd-network:!!:16498::::::
>  systemd-resolve:!!:16498::::::
> -systemd-bus-proxy:!!:16498::::::
>
> Bizarre.

When I had this same btrfs check error, it was the exact inode number
and same /etc/shadow file. I didn't diff the two shadow files, but I
the the cp mv rm routine, and then the system booted. Goofy cakes.
It's almost like an April Fools joke.

-- 
Chris Murphy

Re: F21 fails to mount root part, btrfs check: Couldn't open file system

[Martin Langhoff]

On Wed, Apr 1, 2015 at 2:04 PM, Chris Murphy <lists@colorremedies.com> wrote:
> When I had this same btrfs check error, it was the exact inode number
> and same /etc/shadow file. I didn't diff the two shadow files, but I

That's too bizarre for words. Two folks, on two different systems,
getting btrfs problems on similar kernels on the exact same filepath.
In my case, the file was last frobbed by yum/rpm. Do we have a strange
interaction between a kernel regression and yum/rpm rubbing the
filesystem the wrong way?

BTW, I did not change/touch the file at all. My only "fix" action was
the btrfs check --repair mentioned earlier. Right now, on the booted
system I did

# uname -a
Linux tp-martin.remote-learner.net 3.18.9-200.fc21.x86_64 #1 SMP Mon
Mar 9 15:10:50 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux
# btrfs scrub start -BrR   /
scrub done for 94637b35-a294-4be2-aa47-82c52d6d53ef
        scrub started at Wed Apr  1 13:46:20 2015 and finished after 266 seconds
        data_extents_scrubbed: 344155
        tree_extents_scrubbed: 58048
        data_bytes_scrubbed: 11896840192
        tree_bytes_scrubbed: 951058432
        read_errors: 0
        csum_errors: 0
        verify_errors: 0
        no_csum: 20268
        csum_discards: 254459
        super_errors: 0
        malloc_errors: 0
        uncorrectable_errors: 0
        unverified_errors: 0
        corrected_errors: 0
        last_physical: 23928504320

cheers,



m
-- 
 martin.langhoff@gmail.com
 -  ask interesting questions
 - don't get distracted with shiny stuff  - working code first
 ~ http://docs.moodle.org/en/User:Martin_Langhoff

Re: F21 fails to mount root part, btrfs check: Couldn't open file system

[Chris Murphy]

On Wed, Apr 1, 2015 at 12:16 PM, Martin Langhoff
<martin.langhoff@gmail.com> wrote:
> On Wed, Apr 1, 2015 at 2:04 PM, Chris Murphy <lists@colorremedies.com> wrote:
>> When I had this same btrfs check error, it was the exact inode number
>> and same /etc/shadow file. I didn't diff the two shadow files, but I
>
> That's too bizarre for words. Two folks, on two different systems,
> getting btrfs problems on similar kernels on the exact same filepath.
> In my case, the file was last frobbed by yum/rpm. Do we have a strange
> interaction between a kernel regression and yum/rpm rubbing the
> filesystem the wrong way?

No idea, but it happened to me more than once, same inode number, same file.



> BTW, I did not change/touch the file at all. My only "fix" action was
> the btrfs check --repair mentioned earlier.

That won't fix it. Once errors 400 appears, at this point you have to
replace the affected file.





-- 
Chris Murphy

Re: F21 fails to mount root part, btrfs check: Couldn't open file system

[Martin Langhoff]

On Wed, Apr 1, 2015 at 2:20 PM, Chris Murphy <lists@colorremedies.com> wrote:
> That won't fix it. Once errors 400 appears, at this point you have to
> replace the affected file.

Interesting.

Right now I am booting without problems. I have no evidence of
continued problems. What would I do to check whether I see an error
similar to yours on this fs?

Trying to ascertain whether my fs is cured, and whether we can learn
something else about this oddity...

cheers,


m
-- 
 martin.langhoff@gmail.com
 -  ask interesting questions
 - don't get distracted with shiny stuff  - working code first
 ~ http://docs.moodle.org/en/User:Martin_Langhoff

Re: F21 fails to mount root part, btrfs check: Couldn't open file system

[Chris Murphy]

On Wed, Apr 1, 2015 at 12:29 PM, Martin Langhoff
<martin.langhoff@gmail.com> wrote:
> On Wed, Apr 1, 2015 at 2:20 PM, Chris Murphy <lists@colorremedies.com> wrote:
>> That won't fix it. Once errors 400 appears, at this point you have to
>> replace the affected file.
>
> Interesting.
>
> Right now I am booting without problems. I have no evidence of
> continued problems. What would I do to check whether I see an error
> similar to yours on this fs?
>
> Trying to ascertain whether my fs is cured, and whether we can learn
> something else about this oddity...

Re-run the btrfs check. The error is still there even after a --repair.

-- 
Chris Murphy

Re: F21 fails to mount root part, btrfs check: Couldn't open file system

[Martin Langhoff]

On Wed, Apr 1, 2015 at 2:54 PM, Chris Murphy <lists@colorremedies.com> wrote:
> Re-run the btrfs check. The error is still there even after a --repair.

Bingo! You are right the error persists.

It has no effect on my use of the system right now. Is anyone
interested in debugging this further?

cheers,



martin
-- 
 martin.langhoff@gmail.com
 -  ask interesting questions
 - don't get distracted with shiny stuff  - working code first
 ~ http://docs.moodle.org/en/User:Martin_Langhoff

Re: F21 fails to mount root part, btrfs check: Couldn't open file system

[Chris Murphy]

On Wed, Apr 1, 2015 at 1:23 PM, Martin Langhoff
<martin.langhoff@gmail.com> wrote:
> On Wed, Apr 1, 2015 at 2:54 PM, Chris Murphy <lists@colorremedies.com> wrote:
>> Re-run the btrfs check. The error is still there even after a --repair.
>
> Bingo! You are right the error persists.
>
> It has no effect on my use of the system right now. Is anyone
> interested in debugging this further?

400 errors, nbytes wrong, isn't repaired by current btrfs check
https://bugzilla.kernel.org/show_bug.cgi?id=90071

What's interesting in that bug report that I'd forgotten about?

># btrfs inspect inode 804 /mnt/root
>/mnt/root/etc/shadow-

Different inode number, but the shadow file is affected. In every
single case I've had now (about 1/2 dozen) with this errors 400
message, it's involved the shadow file. I have no idea what's going on
between Btrfs and the shadow file, but something seems to be. Or it's
quite a coincidence.




-- 
Chris Murphy

Re: F21 fails to mount root part, btrfs check: Couldn't open file system

[Chris Murphy]

Related bugs:

https://bugzilla.kernel.org/show_bug.cgi?id=68411
https://bugzilla.redhat.com/show_bug.cgi?id=1037963

The RHBZ one also mentioned the shadow file.

Anyway, it seems to be a somewhat known problem, but it's just not
known yet what causes it.

Chris Murphy

Re: F21 fails to mount root part, btrfs check: Couldn't open file system

[Chris Murphy]

Whenever I have these boot problems, I'm noticing that sometimes the
device, /dev/sda5, is showing up with lsblk (libblkid) as
/dev/block/8:5 while everything else (not-Btrfs) on that device shows
up as /dev/sdaX. Does anyone know what that might mean?


Chris Murphy

Re: F21 fails to mount root part, btrfs check: Couldn't open file system

[Martin Langhoff]

On Tue, Mar 31, 2015 at 4:09 PM, Chris Murphy <lists@colorremedies.com> wrote:
>>A failure of the
>> HDD cannot be ruled out, low power conditions, cheap consumer part...
>
> Well you have to rule that out before anyone on this list can really
> help. Try booting Fedora 21 install media, and using smartctl -x on
> the drive.

smartctl thinks the drive is ok. Unfortunately, it doesn't have a
truth serum to distinguish whether this drive lies about writes or
not...

[root@localhost liveuser]# smartctl -x /dev/sda
smartctl 6.2 2014-07-16 r3952 [x86_64-linux-3.17.4-301.fc21.x86_64]
(local build)
Copyright (C) 2002-13, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Model Family:     Seagate Laptop SSHD
Device Model:     ST500LM000-1EJ162
Serial Number:    W3709VQD
LU WWN Device Id: 5 000c50 069f901e9
Firmware Version: SM14
User Capacity:    500,107,862,016 bytes [500 GB]
Sector Sizes:     512 bytes logical, 4096 bytes physical
Rotation Rate:    5400 rpm
Device is:        In smartctl database [for details use: -P show]
ATA Version is:   ACS-2, ACS-3 T13/2161-D revision 3b
SATA Version is:  SATA 3.1, 6.0 Gb/s (current: 6.0 Gb/s)
Local Time is:    Wed Apr  1 11:44:43 2015 EDT
SMART support is: Available - device has SMART capability.
SMART support is: Enabled
AAM feature is:   Unavailable
APM level is:     128 (minimum power consumption without standby)
Rd look-ahead is: Enabled
Write cache is:   Enabled
ATA Security is:  Disabled, frozen [SEC2]
Write SCT (Get) XXX Error Recovery Control Command failed: scsi error
aborted command
Wt Cache Reorder: N/A

=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

General SMART Values:
Offline data collection status:  (0x00)    Offline data collection activity
                    was never started.
                    Auto Offline Data Collection: Disabled.
Self-test execution status:      (   0)    The previous self-test
routine completed
                    without error or no self-test has ever
                    been run.
Total time to complete Offline
data collection:         (  139) seconds.
Offline data collection
capabilities:              (0x73) SMART execute Offline immediate.
                    Auto Offline data collection on/off support.
                    Suspend Offline collection upon new
                    command.
                    No Offline surface scan supported.
                    Self-test supported.
                    Conveyance Self-test supported.
                    Selective Self-test supported.
SMART capabilities:            (0x0003)    Saves SMART data before entering
                    power-saving mode.
                    Supports SMART auto save timer.
Error logging capability:        (0x01)    Error logging supported.
                    General Purpose Logging supported.
Short self-test routine
recommended polling time:      (   1) minutes.
Extended self-test routine
recommended polling time:      (  98) minutes.
Conveyance self-test routine
recommended polling time:      (   2) minutes.
SCT capabilities:            (0x1081)    SCT Status supported.

SMART Attributes Data Structure revision number: 10
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME          FLAGS    VALUE WORST THRESH FAIL RAW_VALUE
  1 Raw_Read_Error_Rate     POSR--   112   099   006    -    46707576
  3 Spin_Up_Time            PO----   099   099   000    -    0
  4 Start_Stop_Count        -O--CK   100   100   020    -    147
  5 Reallocated_Sector_Ct   PO--CK   100   100   010    -    0
  7 Seek_Error_Rate         POSR--   078   060   030    -    65832005
  9 Power_On_Hours          -O--CK   092   092   000    -    7775
 10 Spin_Retry_Count        PO--C-   100   100   097    -    0
 12 Power_Cycle_Count       -O--CK   100   100   020    -    159
184 End-to-End_Error        -O--CK   100   100   099    -    0
187 Reported_Uncorrect      -O--CK   100   100   000    -    0
188 Command_Timeout         -O--CK   100   100   000    -    1
189 High_Fly_Writes         -O-RCK   095   095   000    -    5
190 Airflow_Temperature_Cel -O---K   070   058   045    -    30 (Min/Max 27/31)
191 G-Sense_Error_Rate      -O--CK   100   100   000    -    0
192 Power-Off_Retract_Count -O--CK   100   100   000    -    25
193 Load_Cycle_Count        -O--CK   066   066   000    -    68484
194 Temperature_Celsius     -O---K   030   042   000    -    30 (0 16 0 0 0)
197 Current_Pending_Sector  -O--C-   100   100   000    -    0
198 Offline_Uncorrectable   ----C-   100   100   000    -    0
199 UDMA_CRC_Error_Count    -OSRCK   200   200   000    -    0
254 Free_Fall_Sensor        -O--CK   100   100   000    -    0
                            ||||||_ K auto-keep
                            |||||__ C event count
                            ||||___ R error rate
                            |||____ S speed/performance
                            ||_____ O updated online
                            |______ P prefailure warning

General Purpose Log Directory Version 1
SMART           Log Directory Version 1 [multi-sector log support]
Address    Access  R/W   Size  Description
0x00       GPL,SL  R/O      1  Log Directory
0x01           SL  R/O      1  Summary SMART error log
0x02           SL  R/O      5  Comprehensive SMART error log
0x03       GPL     R/O      5  Ext. Comprehensive SMART error log
0x06           SL  R/O      1  SMART self-test log
0x07       GPL     R/O      1  Extended self-test log
0x09           SL  R/W      1  Selective self-test log
0x10       GPL     R/O      1  NCQ Command Error log
0x11       GPL     R/O      1  SATA Phy Event Counters
0x21       GPL     R/O      1  Write stream error log
0x22       GPL     R/O      1  Read stream error log
0x24       GPL     R/O   1223  Current Device Internal Status Data log
0x25       GPL     R/O   1223  Saved Device Internal Status Data log
0x30       GPL,SL  R/O      9  IDENTIFY DEVICE data log
0x80-0x9f  GPL,SL  R/W     16  Host vendor specific log
0xa1       GPL,SL  VS      20  Device vendor specific log
0xa2       GPL     VS    3900  Device vendor specific log
0xa8       GPL,SL  VS     129  Device vendor specific log
0xa9       GPL,SL  VS       1  Device vendor specific log
0xab       GPL     VS       1  Device vendor specific log
0xae       GPL     VS       1  Device vendor specific log
0xb0       GPL     VS    4580  Device vendor specific log
0xbe-0xbf  GPL     VS   65535  Device vendor specific log
0xc1       GPL,SL  VS      10  Device vendor specific log
0xc2       GPL,SL  VS      50  Device vendor specific log
0xc4       GPL,SL  VS       5  Device vendor specific log
0xe0       GPL,SL  R/W      1  SCT Command/Status
0xe1       GPL,SL  R/W      1  SCT Data Transfer

SMART Extended Comprehensive Error Log Version: 1 (5 sectors)
No Errors Logged

SMART Extended Self-test Log Version: 1 (1 sectors)
No self-tests have been logged.  [To run self-tests, use: smartctl -t]

SMART Selective self-test log data structure revision number 1
 SPAN  MIN_LBA  MAX_LBA  CURRENT_TEST_STATUS
    1        0        0  Not_testing
    2        0        0  Not_testing
    3        0        0  Not_testing
    4        0        0  Not_testing
    5        0        0  Not_testing
Selective self-test flags (0x0):
  After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.

SCT Data Table command not supported

SCT Error Recovery Control command not supported

Device Statistics (GP Log 0x04) not supported

SATA Phy Event Counters (GP Log 0x11)
ID      Size     Value  Description
0x000a  2            4  Device-to-host register FISes sent due to a COMRESET
0x0001  2            0  Command failed due to ICRC error
0x0003  2            0  R_ERR response for device-to-host data FIS
0x0004  2            0  R_ERR response for host-to-device data FIS
0x0006  2            0  R_ERR response for device-to-host non-data FIS
0x0007  2            0  R_ERR response for host-to-device non-data FIS



-- 
 martin.langhoff@gmail.com
 -  ask interesting questions
 - don't get distracted with shiny stuff  - working code first
 ~ http://docs.moodle.org/en/User:Martin_Langhoff