[!] Unable to mount nilfs2 filesystem after power outage - error loading super root.

[!] Unable to mount nilfs2 filesystem after power outage - error loading super root.

[KARBOWSKI Piotr]

Hi,

Yesterday I had an power outage. After booting my notebook again, I was 
no longer able to mount /home that was on the top of nilfs2 filesystem.

I've come across a post from 2011 about fsck in git.nilfs.org but that 
repo is no longer accessable. The repo from githab does not have any 
trace of the experimental fsck tool thus I've dumped the filesystem to 
image, recreated ext4 and restored backup.

The symptoms:

# mount -o loop ./kurogane_ssd-home.17.03.2016.img /mnt/tmp
mount.nilfs2: Error while mounting /dev/loop/0 on /mnt/tmp: No such file 
or directory

# dmesg
[ 9591.626210] NILFS warning: mounting unchecked fs
[ 9591.824166] NILFS: error loading super root.

No nilfs-utils's tools works beside the tune with -l

# nilfs-tune -l /dev/loop1
nilfs-tune 2.2.2
Filesystem volume name:	  home
Filesystem UUID:	  6a915104-61d5-4214-bed4-2538d1bdfaa5
Filesystem magic number:  0x3434
Filesystem revision #:	  2.0
Filesystem features:      (none)
Filesystem state:	  invalid or mounted
Filesystem OS type:	  Linux
Block size:		  4096
Filesystem created:	  Sat Nov 14 14:40:18 2015
Last mount time:	  Thu Mar 17 17:41:23 2016
Last write time:	  Thu Mar 17 22:46:08 2016
Mount count:		  154
Maximum mount count:	  50
Reserve blocks uid:	  0 (user root)
Reserve blocks gid:	  0 (group root)
First inode:		  11
Inode size:		  128
DAT entry size:		  32
Checkpoint size:	  192
Segment usage size:	  16
Number of segments:	  11294
Device size:		  94745133056
First data block:	  1
# of blocks per segment:  2048
Reserved segments %:	  5
Last checkpoint #:	  1016585
Last block address:	  17023836
Last sequence #:	  80736
Free blocks count:	  6764544
Commit interval:	  0
# of blks to create seg:  0
CRC seed:		  0x60cb4676
CRC check sum:		  0x68371ded
CRC check data size:	  0x00000118

This is something I woudn't expect from a log structured filesystem. I 
am running kernel 4.5.0.

Any comments and help with restoring access to this fs much appriecieate

-- Piotr.
--
To unsubscribe from this list: send the line "unsubscribe linux-nilfs" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [!] Unable to mount nilfs2 filesystem after power outage - error loading super root.

[Viacheslav Dubeyko]

On Fri, 2016-03-18 at 19:16 +0100, KARBOWSKI Piotr wrote:
> Hi,
> 
> Yesterday I had an power outage. After booting my notebook again, I was 
> no longer able to mount /home that was on the top of nilfs2 filesystem.
> 
> I've come across a post from 2011 about fsck in git.nilfs.org but that 
> repo is no longer accessable. The repo from githab does not have any 
> trace of the experimental fsck tool thus I've dumped the filesystem to 
> image, recreated ext4 and restored backup.
> 

NILFS2 hasn't fsck utility. And no implementation efforts exist.

> The symptoms:
> 
> # mount -o loop ./kurogane_ssd-home.17.03.2016.img /mnt/tmp
> mount.nilfs2: Error while mounting /dev/loop/0 on /mnt/tmp: No such file 
> or directory
> 
> # dmesg
> [ 9591.626210] NILFS warning: mounting unchecked fs
> [ 9591.824166] NILFS: error loading super root.
> 

Do you have any NILFS2 related errors before mount operation in your
syslog? Does strace show something interesting for mount operation?

> No nilfs-utils's tools works beside the tune with -l
> 
> # nilfs-tune -l /dev/loop1
> nilfs-tune 2.2.2
> Filesystem volume name:	  home
> Filesystem UUID:	  6a915104-61d5-4214-bed4-2538d1bdfaa5
> Filesystem magic number:  0x3434
> Filesystem revision #:	  2.0
> Filesystem features:      (none)
> Filesystem state:	  invalid or mounted
> Filesystem OS type:	  Linux
> Block size:		  4096
> Filesystem created:	  Sat Nov 14 14:40:18 2015
> Last mount time:	  Thu Mar 17 17:41:23 2016
> Last write time:	  Thu Mar 17 22:46:08 2016
> Mount count:		  154
> Maximum mount count:	  50
> Reserve blocks uid:	  0 (user root)
> Reserve blocks gid:	  0 (group root)
> First inode:		  11
> Inode size:		  128
> DAT entry size:		  32
> Checkpoint size:	  192
> Segment usage size:	  16
> Number of segments:	  11294
> Device size:		  94745133056
> First data block:	  1
> # of blocks per segment:  2048
> Reserved segments %:	  5
> Last checkpoint #:	  1016585
> Last block address:	  17023836
> Last sequence #:	  80736
> Free blocks count:	  6764544
> Commit interval:	  0
> # of blks to create seg:  0
> CRC seed:		  0x60cb4676
> CRC check sum:		  0x68371ded
> CRC check data size:	  0x00000118
> 
> This is something I woudn't expect from a log structured filesystem. I 
> am running kernel 4.5.0.
> 

Any software has bugs. NILFS2 needs in testing and stabilization yet.
So, even if NILFS2 is LFS then it's software that contains bugs.

> Any comments and help with restoring access to this fs much appriecieate
> 

It's possible to try to investigate your issue but no promise about
recovering corrupted NILFS2 volume. We haven't any tools for it. But if
you are ready then you could implement such tool(s).

Thanks,
Vyacheslav Dubeyko.


--
To unsubscribe from this list: send the line "unsubscribe linux-nilfs" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [!] Unable to mount nilfs2 filesystem after power outage - error loading super root.

[KARBOWSKI Piotr]

Hi,

On 03/18/2016 09:13 PM, Viacheslav Dubeyko wrote:
> On Fri, 2016-03-18 at 19:16 +0100, KARBOWSKI Piotr wrote:
>> Hi,
>>
>> Yesterday I had an power outage. After booting my notebook again, I was
>> no longer able to mount /home that was on the top of nilfs2 filesystem.
>>
>> I've come across a post from 2011 about fsck in git.nilfs.org but that
>> repo is no longer accessable. The repo from githab does not have any
>> trace of the experimental fsck tool thus I've dumped the filesystem to
>> image, recreated ext4 and restored backup.
>>
>
> NILFS2 hasn't fsck utility. And no implementation efforts exist.

I was refering to http://marc.info/?l=linux-nilfs&m=130799952525081&w=2

>> The symptoms:
>>
>> # mount -o loop ./kurogane_ssd-home.17.03.2016.img /mnt/tmp
>> mount.nilfs2: Error while mounting /dev/loop/0 on /mnt/tmp: No such file
>> or directory
>>
>> # dmesg
>> [ 9591.626210] NILFS warning: mounting unchecked fs
>> [ 9591.824166] NILFS: error loading super root.
>>
>
> Do you have any NILFS2 related errors before mount operation in your
> syslog? Does strace show something interesting for mount operation?

Nothing special. Right now all I have is a image of the filesystem, and 
just the two lines are printed when I try to mount it. Strace does not 
provide anything useful

     mount("/dev/loop/1", "/mnt/tmp", "nilfs2", MS_MGC_VAL, NULL) = -1 
ENOENT

>> No nilfs-utils's tools works beside the tune with -l
>>
>> # nilfs-tune -l /dev/loop1
>> nilfs-tune 2.2.2
>> Filesystem volume name:	  home
>> Filesystem UUID:	  6a915104-61d5-4214-bed4-2538d1bdfaa5
>> Filesystem magic number:  0x3434
>> Filesystem revision #:	  2.0
>> Filesystem features:      (none)
>> Filesystem state:	  invalid or mounted
>> Filesystem OS type:	  Linux
>> Block size:		  4096
>> Filesystem created:	  Sat Nov 14 14:40:18 2015
>> Last mount time:	  Thu Mar 17 17:41:23 2016
>> Last write time:	  Thu Mar 17 22:46:08 2016
>> Mount count:		  154
>> Maximum mount count:	  50
>> Reserve blocks uid:	  0 (user root)
>> Reserve blocks gid:	  0 (group root)
>> First inode:		  11
>> Inode size:		  128
>> DAT entry size:		  32
>> Checkpoint size:	  192
>> Segment usage size:	  16
>> Number of segments:	  11294
>> Device size:		  94745133056
>> First data block:	  1
>> # of blocks per segment:  2048
>> Reserved segments %:	  5
>> Last checkpoint #:	  1016585
>> Last block address:	  17023836
>> Last sequence #:	  80736
>> Free blocks count:	  6764544
>> Commit interval:	  0
>> # of blks to create seg:  0
>> CRC seed:		  0x60cb4676
>> CRC check sum:		  0x68371ded
>> CRC check data size:	  0x00000118
>>
>> This is something I woudn't expect from a log structured filesystem. I
>> am running kernel 4.5.0.
>>
>
> Any software has bugs. NILFS2 needs in testing and stabilization yet.
> So, even if NILFS2 is LFS then it's software that contains bugs.
>> Any comments and help with restoring access to this fs much appriecieate
>>
>
> It's possible to try to investigate your issue but no promise about
> recovering corrupted NILFS2 volume. We haven't any tools for it. But if
> you are ready then you could implement such tool(s).



--
To unsubscribe from this list: send the line "unsubscribe linux-nilfs" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [!] Unable to mount nilfs2 filesystem after power outage - error loading super root.

[Viacheslav Dubeyko]

On Fri, 2016-03-18 at 21:40 +0100, KARBOWSKI Piotr wrote:
> Hi,
> 
> On 03/18/2016 09:13 PM, Viacheslav Dubeyko wrote:
> > On Fri, 2016-03-18 at 19:16 +0100, KARBOWSKI Piotr wrote:
> >> Hi,
> >>
> >> Yesterday I had an power outage. After booting my notebook again, I was
> >> no longer able to mount /home that was on the top of nilfs2 filesystem.
> >>
> >> I've come across a post from 2011 about fsck in git.nilfs.org but that
> >> repo is no longer accessable. The repo from githab does not have any
> >> trace of the experimental fsck tool thus I've dumped the filesystem to
> >> image, recreated ext4 and restored backup.
> >>
> >
> > NILFS2 hasn't fsck utility. And no implementation efforts exist.
> 
> I was refering to http://marc.info/?l=linux-nilfs&m=130799952525081&w=2
> 

You could check this link
http://nilfs.sourceforge.net/en/current_status.html
The fsck tool is still in TODO list.

> >> The symptoms:
> >>
> >> # mount -o loop ./kurogane_ssd-home.17.03.2016.img /mnt/tmp
> >> mount.nilfs2: Error while mounting /dev/loop/0 on /mnt/tmp: No such file
> >> or directory
> >>
> >> # dmesg
> >> [ 9591.626210] NILFS warning: mounting unchecked fs
> >> [ 9591.824166] NILFS: error loading super root.
> >>
> >
> > Do you have any NILFS2 related errors before mount operation in your
> > syslog? Does strace show something interesting for mount operation?
> 
> Nothing special. Right now all I have is a image of the filesystem, and 
> just the two lines are printed when I try to mount it. Strace does not 
> provide anything useful
> 
>      mount("/dev/loop/1", "/mnt/tmp", "nilfs2", MS_MGC_VAL, NULL) = -1 
> ENOENT
> 

It needs to dive into state of the volume. You can use lscp and lssu
tools with the goal to retrieve segments and checkpoints state. But,
again, this path could provide opportunity for understanding the bug
essence and environment. I could see only one way for "recovering"
corrupted volume: (1) understand the current state of volume's segments;
(2) localize erroneous segment; (3) try to zero erroneous segment(s) by
means of dd utility; (4) try to mount again. But it's really dangerous
way, you could loose your data.

Thanks,
Vyacheslav Dubeyko.



--
To unsubscribe from this list: send the line "unsubscribe linux-nilfs" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [!] Unable to mount nilfs2 filesystem after power outage - error loading super root.

[KARBOWSKI Piotr]

Hi,

On 03/18/2016 10:36 PM, Viacheslav Dubeyko wrote:
> It needs to dive into state of the volume. You can use lscp and lssu
> tools with the goal to retrieve segments and checkpoints state. But,
> again, this path could provide opportunity for understanding the bug
> essence and environment. I could see only one way for "recovering"
> corrupted volume: (1) understand the current state of volume's segments;
> (2) localize erroneous segment; (3) try to zero erroneous segment(s) by
> means of dd utility; (4) try to mount again. But it's really dangerous
> way, you could loose your data.

I could try, but lscp and lssu does not work with this filesystem since 
crash. Only 'nilfs-tune -l' display anything.

# strace lssu /dev/loop/1 2>&1 | grep loop
execve("/usr/bin/lssu", ["lssu", "/dev/loop/1"], [/* 61 vars */]) = 0
readlink("/dev/loop", 0x7ffe6a073ac0, 4096) = -1 EINVAL (Invalid argument)
readlink("/dev/loop/1", 0x7ffe6a073ac0, 4096) = -1 EINVAL (Invalid argument)
readlink("/dev/loop", 0x7ffe6a073ac0, 4096) = -1 EINVAL (Invalid argument)
readlink("/dev/loop/1", 0x7ffe6a073ac0, 4096) = -1 EINVAL (Invalid argument)
write(2, "lssu: cannot open NILFS on /dev/"..., 66lssu: cannot open 
NILFS on /dev/loop/1: No such file or directory

Does lscp and lssu needs actually mounted filesystem? It does run 
readlink() and check /proc/mounts...

-- Piotr.

--
To unsubscribe from this list: send the line "unsubscribe linux-nilfs" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [!] Unable to mount nilfs2 filesystem after power outage - error loading super root.

[Viacheslav Dubeyko]

On Fri, 2016-03-18 at 23:14 +0100, KARBOWSKI Piotr wrote:
> Hi,
> 
> On 03/18/2016 10:36 PM, Viacheslav Dubeyko wrote:
> > It needs to dive into state of the volume. You can use lscp and lssu
> > tools with the goal to retrieve segments and checkpoints state. But,
> > again, this path could provide opportunity for understanding the bug
> > essence and environment. I could see only one way for "recovering"
> > corrupted volume: (1) understand the current state of volume's segments;
> > (2) localize erroneous segment; (3) try to zero erroneous segment(s) by
> > means of dd utility; (4) try to mount again. But it's really dangerous
> > way, you could loose your data.
> 
> I could try, but lscp and lssu does not work with this filesystem since 
> crash. Only 'nilfs-tune -l' display anything.
> 
> # strace lssu /dev/loop/1 2>&1 | grep loop
> execve("/usr/bin/lssu", ["lssu", "/dev/loop/1"], [/* 61 vars */]) = 0
> readlink("/dev/loop", 0x7ffe6a073ac0, 4096) = -1 EINVAL (Invalid argument)
> readlink("/dev/loop/1", 0x7ffe6a073ac0, 4096) = -1 EINVAL (Invalid argument)
> readlink("/dev/loop", 0x7ffe6a073ac0, 4096) = -1 EINVAL (Invalid argument)
> readlink("/dev/loop/1", 0x7ffe6a073ac0, 4096) = -1 EINVAL (Invalid argument)
> write(2, "lssu: cannot open NILFS on /dev/"..., 66lssu: cannot open 
> NILFS on /dev/loop/1: No such file or directory
> 
> Does lscp and lssu needs actually mounted filesystem? It does run 
> readlink() and check /proc/mounts...
> 

Ahhh, I've forgot that you cannot mount your volume. Yes, you can use
lscp and lssu for mounted volume only. So, it needs to analyze raw
volume's content in such situation. No another way. Welcome to the real
world. Sorry..

If you have such issue after power outage then the most probable reason
of such issue is corruption of the last segment. Theoretically, NILFS2
has all necessary for recovering from such issue (because NILFS2 has
checkpoints for all metadata in previous logs). But current state of
NILFS2 code is unable to overcome such situation. Another possible
reason could be inconsistent state of superblock. NILFS2 has two
superblocks (in the beginning and in the end of the volume). So, it's
possible that one of the superblock (or both) has inconsistent state. It
means that some of superblock's field will contain some invalid state.
So, you need to elaborate vision what's wrong on your volume and to
modify the volume state manually.

Thanks,
Vyacheslav Dubeyko.


--
To unsubscribe from this list: send the line "unsubscribe linux-nilfs" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html