Fail to mount NILFS.

Fail to mount NILFS.

[Yuri Chislov]

Hi,

 I get error when mount nilfs2 partition.

mount.nilfs2 /dev/cciss/c0d1p1 /home/
mount.nilfs2: WARNING! - The NILFS on-disk format may change at any time.
mount.nilfs2: WARNING! - Do not place critical data on a NILFS filesystem.
mount.nilfs2: Error while mounting /dev/cciss/c0d1p1 on /home/: Invalid 
argument

dmesg output:
segctord starting. Construction interval = 5 seconds, CP frequency < 30 
seconds
NILFS: corrupt root inode.

Used kernel: 2.6.30.4
nilfs-utils-2.0.12-3.i386


Is it possible to fix the partition?
Is it some tools exists, like fsck?

Thank you.
Yuri.

Re: Fail to mount NILFS.

[Ryusuke Konishi]

[-- Attachment #1: Type: Text/Plain, Size: 1341 bytes --]

Hi,
On Fri, 21 Aug 2009 19:18:34 +0300, Yuri Chislov wrote:
> Hi,
> 
>  I get error when mount nilfs2 partition.
> 
> mount.nilfs2 /dev/cciss/c0d1p1 /home/
> mount.nilfs2: WARNING! - The NILFS on-disk format may change at any time.
> mount.nilfs2: WARNING! - Do not place critical data on a NILFS filesystem.
> mount.nilfs2: Error while mounting /dev/cciss/c0d1p1 on /home/: Invalid 
> argument
> 
> dmesg output:
> segctord starting. Construction interval = 5 seconds, CP frequency < 30 
> seconds
> NILFS: corrupt root inode.
> 
> Used kernel: 2.6.30.4
> nilfs-utils-2.0.12-3.i386
> 
> 
> Is it possible to fix the partition?
> Is it some tools exists, like fsck?
> 
> Thank you.
> Yuri.

Sorry for the inconvenience.
I'm afraid to say fsck has not been developed yet.

Are there any warnings or error messages of nilfs in syslog before the
problem ?

I'll attach a patch to allow nilfs to mount plain checkpoints.

If you made the kernel from source, try the patch as follows:

 $ cd linux-2.6.30.4
 $ patch -p1 < path-to/0001-nilfs2-remove-snapshot-check-on-mount.patch
 $ make modules
 $ sudo rmmod nilfs2
 $ sudo insmod fs/nilfs2/nilfs2.ko
 $ sudo mount -t nilfs2 -r -o cp=xxx /dev/yyy /mnt

If you succeed to mount a valid past checkpoint, you will be able to
do backup with rsync or so from the checkpoint.

Regards,
Ryusuke Konishi

[-- Attachment #2: 0001-nilfs2-remove-snapshot-check-on-mount.patch.bz2 --]
[-- Type: Application/Octet-Stream, Size: 951 bytes --]

[-- Attachment #3: Type: text/plain, Size: 158 bytes --]

_______________________________________________
users mailing list
users-JrjvKiOkagjYtjvyW6yDsg@public.gmane.org
https://www.nilfs.org/mailman/listinfo/users

Re: Fail to mount NILFS.

[Yuri Chislov]

On Saturday 22 August 2009 05:25:33 Ryusuke Konishi wrote:
> f you made the kernel from source, try the patch as follows:
>
>  $ cd linux-2.6.30.4
>  $ patch -p1 < path-to/0001-nilfs2-remove-snapshot-check-on-mount.patch
>  $ make modules
>  $ sudo rmmod nilfs2
>  $ sudo insmod fs/nilfs2/nilfs2.ko
>  $ sudo mount -t nilfs2 -r -o cp=xxx /dev/yyy /mnt
>
> If you succeed to mount a valid past checkpoint, you will be able to
> do backup with rsync or so from the checkpoint.
>
> Regards,
> Ryusuke Konishi

Hi, 
  
  Thank you for patch.
 I have try to mount as you propose, but receive following errors:
/sbin/mount.nilfs2 -r -o cp=30778 /dev/cciss/c0d1p1 /mnt/

NILFS: latest checkpoint=30778
NILFS: The specified checkpoint is not a snapshot (checkpoint number=30778).
NILFS: corrupt root inode.


If I try to mount older snapshot 
/sbin/mount.nilfs2 -r -o cp=27252 /dev/cciss/c0d1p1 /mnt/

then 

NILFS: latest checkpoint=30778
NILFS: corrupt root inode.

If try to convert 30778 to snapshot I get
chcp: /dev/cciss/c0d1p1: cannot open NILFS
 
Thank you.

Yuri.

Re: Fail to mount NILFS.

[Ryusuke Konishi]

Hi,
On Sat, 22 Aug 2009 11:46:28 +0300, Yuri Chislov wrote:
> On Saturday 22 August 2009 05:25:33 Ryusuke Konishi wrote:
> > f you made the kernel from source, try the patch as follows:
> >
> >  $ cd linux-2.6.30.4
> >  $ patch -p1 < path-to/0001-nilfs2-remove-snapshot-check-on-mount.patch
> >  $ make modules
> >  $ sudo rmmod nilfs2
> >  $ sudo insmod fs/nilfs2/nilfs2.ko
> >  $ sudo mount -t nilfs2 -r -o cp=xxx /dev/yyy /mnt
> >
> > If you succeed to mount a valid past checkpoint, you will be able to
> > do backup with rsync or so from the checkpoint.
> >
> > Regards,
> > Ryusuke Konishi
> 
> Hi, 
>   
>   Thank you for patch.
>  I have try to mount as you propose, but receive following errors:
> /sbin/mount.nilfs2 -r -o cp=30778 /dev/cciss/c0d1p1 /mnt/
> 
> NILFS: latest checkpoint=30778
> NILFS: The specified checkpoint is not a snapshot (checkpoint number=30778).

Ignore this message. The patch actually changed nilfs to accept all
checkpoints, but I left this error message.

> NILFS: corrupt root inode.

This _is_ the problem. This shows the inode of root directory is
broken or empty.  I think you cannot mount generations which you see
this message.
 
> If I try to mount older snapshot 
> /sbin/mount.nilfs2 -r -o cp=27252 /dev/cciss/c0d1p1 /mnt/
> 
> then 
> 
> NILFS: latest checkpoint=30778
> NILFS: corrupt root inode.

The first message is just information I added in the patch.
But, uum, the root inode of checkpoint 27252 seems to be broken, too.
 
Can you go back to much older checkpoints ?

> If try to convert 30778 to snapshot I get
> chcp: /dev/cciss/c0d1p1: cannot open NILFS

Yeah, chcp is not possible if there is no rw-mount.
You don't have to do chcp because my workaround patch makes
existent checkpoints mountable.

BTW, your nilfs-utils seems a bit old.  I believe there was no problem
causing filesystem corruption in nilfs-utils-2.0.12.  But, I recommend
you to use nilfs-utils-2.0.14 if available.

Regards,
Ryusuke Konishi

Re: Fail to mount NILFS.

[Yuri Chislov]

Hi,

On Saturday 22 August 2009 12:50:13 Ryusuke Konishi wrote:
> The first message is just information I added in the patch.
> But, uum, the root inode of checkpoint 27252 seems to be broken, too.
>  
> Can you go back to much older checkpoints ?

I try to use older snapshot, that was successfully used before:

Aug 16 07:10:32 filer filer: Running: /sbin/mount.nilfs2 -r -o cp=5063 
/dev/cciss/c0d1p1 /snaps/2009-08-16_07_10_07
Aug 16 07:10:32 filer filer: Status: [OK]  Ouput:

But currently I get the same error.

The stored data is not critical for me so, as I have full backups, but I
prefer to check possibility of the fs recovery, if possible.

I have update utils to 2.0.14 and kernel to 2.6.30.5 + your patch.
Is it possible to add some printk to get additional information?

Thank you.
Yuri.

Re: Fail to mount NILFS.

[Ryusuke Konishi]

On Sat, 22 Aug 2009 13:36:13 +0300, Yuri Chislov wrote:
> Hi,
> 
> On Saturday 22 August 2009 12:50:13 Ryusuke Konishi wrote:
> > The first message is just information I added in the patch.
> > But, uum, the root inode of checkpoint 27252 seems to be broken, too.
> >  
> > Can you go back to much older checkpoints ?
> 
> I try to use older snapshot, that was successfully used before:
> 
> Aug 16 07:10:32 filer filer: Running: /sbin/mount.nilfs2 -r -o cp=5063 
> /dev/cciss/c0d1p1 /snaps/2009-08-16_07_10_07
> Aug 16 07:10:32 filer filer: Status: [OK]  Ouput:
> 
> But currently I get the same error.

Hmm, this seems to be caused by corruption of DAT file.  Maybe garbage
collection failed to update the DAT file due to some bug.

> The stored data is not critical for me so, as I have full backups, but I
> prefer to check possibility of the fs recovery, if possible.
> 
> I have update utils to 2.0.14 and kernel to 2.6.30.5 + your patch.
> Is it possible to add some printk to get additional information?
> 
> Thank you.
> Yuri.

Thanks.  I will send you an additional patch for analysis later.

Thanks,
Ryusuke Konishi

Re: Fail to mount NILFS.

[Ryusuke Konishi]

[-- Attachment #1: Type: Text/Plain, Size: 1335 bytes --]

On Sat, 22 Aug 2009 20:43:16 +0900 (JST), Ryusuke Konishi wrote:
> On Sat, 22 Aug 2009 13:36:13 +0300, Yuri Chislov wrote:
> > I have update utils to 2.0.14 and kernel to 2.6.30.5 + your patch.
> > Is it possible to add some printk to get additional information?
> > 
> > Thank you.
> > Yuri.
> 
> Thanks.  I will send you an additional patch for analysis later.
> 
> Thanks,
> Ryusuke Konishi

Yuri, I will attach a patch to log the latest segment.

If you apply this, you will see messages like:

segctord starting. Construction interval = 5 seconds, CP frequency < 30 seconds
NILFS: looking segment (seg_start=1382400, seg_end=1384447, segnum=675, seg_seq=675)
NILFS: found super root: segnum=675, seq=675, pseg_start=1382558, pseg_offset=174
NILFS: latest checkpoint=26461


Then, you can dump summary of the segment with dumpseg command.


 # dumpseg 675
           ~~~ specify the segnum reported in the log.
 # dumpseg 674
 # dumpseg 673
 ..

Could you send me the dump log around the segment?

If you find an entry like:

    finfo
      ino = 6, cno = xxxxx, nblocks = x, ndatblk = x
        vblocknr = yyyyyy, blkoff = 2, blocknr = xxxxxx
      ~~~~~~~              ~~~~~~~~~~
   ino = 6 indicates       The root inode is included in this block.
   inode metadata file

This writeback is suspicious.

Thanks,
Ryusuke Konishi

[-- Attachment #2: 0001-nilfs2-insert-some-printks-in-recovery-logic.patch.bz2 --]
[-- Type: Application/Octet-Stream, Size: 1143 bytes --]

[-- Attachment #3: Type: text/plain, Size: 158 bytes --]

_______________________________________________
users mailing list
users-JrjvKiOkagjYtjvyW6yDsg@public.gmane.org
https://www.nilfs.org/mailman/listinfo/users

Re: Fail to mount NILFS.

[Yuri Chislov]

[-- Attachment #1: Type: text/plain, Size: 574 bytes --]

On Saturday 22 August 2009 15:34:38 Ryusuke Konishi wrote:
> Could you send me the dump log around the segment?
>
> If you find an entry like:
>
>     finfo
>       ino = 6, cno = xxxxx, nblocks = x, ndatblk = x
>         vblocknr = yyyyyy, blkoff = 2, blocknr = xxxxxx
>       ~~~~~~~              ~~~~~~~~~~
>    ino = 6 indicates       The root inode is included in this block.
>    inode metadata file
>
> This writeback is suspicious.
>
> Thanks,
> Ryusuke Konishi
Hi, 

Attached file with dumpseg output.
It's look, that root inode in  segnum=10938.

Thank you.
Yuri.

[-- Attachment #2: root_inode_issue.bz2 --]
[-- Type: application/x-bzip, Size: 35060 bytes --]

[-- Attachment #3: Type: text/plain, Size: 158 bytes --]

_______________________________________________
users mailing list
users-JrjvKiOkagjYtjvyW6yDsg@public.gmane.org
https://www.nilfs.org/mailman/listinfo/users

Re: Fail to mount NILFS.

[Ryusuke Konishi]

[-- Attachment #1: Type: Text/Plain, Size: 987 bytes --]

On Sat, 22 Aug 2009 17:36:42 +0300, Yuri Chislov wrote:
> On Saturday 22 August 2009 15:34:38 Ryusuke Konishi wrote:
> > Could you send me the dump log around the segment?
> >
> > If you find an entry like:
> >
> >     finfo
> >       ino = 6, cno = xxxxx, nblocks = x, ndatblk = x
> >         vblocknr = yyyyyy, blkoff = 2, blocknr = xxxxxx
> >       ~~~~~~~              ~~~~~~~~~~
> >    ino = 6 indicates       The root inode is included in this block.
> >    inode metadata file
> >
> > This writeback is suspicious.
> >
> > Thanks,
> > Ryusuke Konishi
> Hi, 
> 
> Attached file with dumpseg output.
> It's look, that root inode in  segnum=10938.
> 
> Thank you.
> Yuri.

The segment in the attached log did not include the root inode.
( The entry with ino = 6, did not include the block with blkoff = 2 )

What will happen if you run the attached script?

 $ chmod +x find-root.sh
 $ sudo ./find-root.sh <device> 10938 | tee -a segment-with-root-inode.log

Thanks,
Ryusuke Konishi

[-- Attachment #2: find-root.sh --]
[-- Type: Text/Plain, Size: 394 bytes --]

#!/bin/bash

cmd=`basename $0`

function die() { echo "$1" 1>&2; exit 1; }

test -n "$1" -a -n "$2" -a -b "$1" || die "usage: $cmd <device> <segnum>"

dev=$1

for (( i=$2; i >= 0; i-- )); do
    if dumpseg $dev $i | sed -ne '/ino = 6/,/finfo/{p}' | grep -q "blkoff = 2,";  then
	echo "$cmd: root inode found in segnum=$i"
	dumpseg $dev $i
	exit 0
    fi
done

die "$cmd: root inode not found"


[-- Attachment #3: Type: text/plain, Size: 158 bytes --]

_______________________________________________
users mailing list
users-JrjvKiOkagjYtjvyW6yDsg@public.gmane.org
https://www.nilfs.org/mailman/listinfo/users

Re: Fail to mount NILFS.

[Yuri Chislov]

[-- Attachment #1: Type: text/plain, Size: 257 bytes --]

On Saturday 22 August 2009 19:21:39 Ryusuke Konishi wrote:
> What will happen if you run the attached script?
>
>  $ chmod +x find-root.sh
>  $ sudo ./find-root.sh <device> 10938 | tee -a segment-with-root-inode.log
Done.

Log file attached.

Thanks.
Yuri.

[-- Attachment #2: segment-with-root-inode.log.bz2 --]
[-- Type: application/x-bzip, Size: 11295 bytes --]

[-- Attachment #3: Type: text/plain, Size: 158 bytes --]

_______________________________________________
users mailing list
users-JrjvKiOkagjYtjvyW6yDsg@public.gmane.org
https://www.nilfs.org/mailman/listinfo/users

Re: Fail to mount NILFS.

[Ryusuke Konishi]

[-- Attachment #1: Type: Text/Plain, Size: 761 bytes --]

On Sat, 22 Aug 2009 19:36:12 +0300, Yuri Chislov wrote:
> On Saturday 22 August 2009 19:21:39 Ryusuke Konishi wrote:
> > What will happen if you run the attached script?
> >
> >  $ chmod +x find-root.sh
> >  $ sudo ./find-root.sh <device> 10938 | tee -a segment-with-root-inode.log
> Done.
> 
> Log file attached.
> 
> Thanks.
> Yuri.

Ah, sorry.  Pattern matching of the script was incomplete.

Could you retry with the revised script?

If the new script find the blocks storing the root inode, it will also
dump them to files like:

ifile-2-seg649-1329203
ifile-2-seg649-1329259
ifile-2-seg649-1329315
ifile-2-seg649-1329372
ifile-2-seg649-1329428
ifile-2-seg649-1329455

Then, we can see if the root inode is actually broken or not.

Thanks,
Ryuske Konishi


[-- Attachment #2: find-root.sh --]
[-- Type: Text/Plain, Size: 619 bytes --]

#!/bin/bash

cmd=`basename $0`
blocksize=4k

function die() { echo "$1" 1>&2; exit 1; }

test -n "$1" -a -n "$2" -a -b "$1" || die "usage: $cmd <device> <segnum>"

dev=$1

for (( i=$2; i >= 0; i-- )); do
    hit=`dumpseg $dev $i | sed -ne '/ino = 6,/,/finfo/{p}' |  sed -ne "/blkoff = 2,/{s/^.* blocknr = \([0-9]\+\).*$/\1/p}"`
    if [ -n "$hit" ]; then
	echo "$cmd: root inode found in segnum=$i"
	dumpseg $dev $i

	for b in $hit; do
	    echo "$cmd: dump root inode block: blocknr=$b"
	    dd if=$dev of=ifile-2-seg${i}-${b} bs=$blocksize skip=$b count=1
	done
	exit 0
    fi
done

die "$cmd: root inode not found"


[-- Attachment #3: Type: text/plain, Size: 158 bytes --]

_______________________________________________
users mailing list
users-JrjvKiOkagjYtjvyW6yDsg@public.gmane.org
https://www.nilfs.org/mailman/listinfo/users

Re: Fail to mount NILFS.

[Yuri Chislov]

On Sunday 23 August 2009 06:23:22 Ryusuke Konishi wrote:
> If the new script find the blocks storing the root inode, it will also
> dump them to files like:
>
> ifile-2-seg649-1329203
> ifile-2-seg649-1329259
> ifile-2-seg649-1329315
> ifile-2-seg649-1329372
> ifile-2-seg649-1329428
> ifile-2-seg649-1329455
>
> Then, we can see if the root inode is actually broken or not.
Hi,

 There is output from script.
sudo ./find_root.sh /dev/cciss/c0d1p1 10938 | tee -a segment-with-root-
inode.log
find_root.sh: root inode not found

Thanks.
Yuri.

Re: Fail to mount NILFS.

[Ryusuke Konishi]

[-- Attachment #1: Type: Text/Plain, Size: 1375 bytes --]

Hi,
On Sun, 23 Aug 2009 07:55:02 +0300, Yuri Chislov wrote:
> On Sunday 23 August 2009 06:23:22 Ryusuke Konishi wrote:
> > If the new script find the blocks storing the root inode, it will also
> > dump them to files like:
> >
> > ifile-2-seg649-1329203
> > ifile-2-seg649-1329259
> > ifile-2-seg649-1329315
> > ifile-2-seg649-1329372
> > ifile-2-seg649-1329428
> > ifile-2-seg649-1329455
> >
> > Then, we can see if the root inode is actually broken or not.
> Hi,
> 
>  There is output from script.
> sudo ./find_root.sh /dev/cciss/c0d1p1 10938 | tee -a segment-with-root-
> inode.log
> find_root.sh: root inode not found
> 
> Thanks.
> Yuri.

Thank you for following this analysis.

It reached the beginning of the partition.

What will happen if you start the script from the last segment number?

The last segment number is calculated from the block count which
appears in /proc/partitions:

 $ cat /proc/partitions
major minor  #blocks  name

   8       16   71687325 sdb
   8       17   35840983 sdb1
   8       18   35841015 sdb2

 $ expr 35840983 / 4 / 2048 - 1
 4374
 $ sudo ./find-root.sh /dev/sdb1 4374


And, the attached patch will output some information about how disk
block numbers are converted with DAT file.

I think this will give another hint on which metadata broke in nilfs.

Could you apply it and try mount again?

Thanks in advance,
Ryusuke Konishi

[-- Attachment #2: 0001-nilfs2-add-debug-messages-on-bmap-lookup-and-dat-tra.patch.bz2 --]
[-- Type: Application/Octet-Stream, Size: 1143 bytes --]

[-- Attachment #3: Type: text/plain, Size: 158 bytes --]

_______________________________________________
users mailing list
users-JrjvKiOkagjYtjvyW6yDsg@public.gmane.org
https://www.nilfs.org/mailman/listinfo/users

Re: Fail to mount NILFS.

[Yuri Chislov]

[-- Attachment #1: Type: text/plain, Size: 496 bytes --]

Resend to mailing list.
On Sunday 23 August 2009 11:20:53 you wrote:
> And, the attached patch will output some information about how disk
> block numbers are converted with DAT file.
>
> I think this will give another hint on which metadata broke in nilfs.
>
> Could you apply it and try mount again?
Done.

104       17  209720511 cciss/c0d1p1
expr 209720511 / 4 / 2048 - 1
25599
./find_root.sh /dev/cciss/c0d1p1 25599
find_root.sh: root inode not found

Attached dmesg output.

Thanks.
Yuri.


[-- Attachment #2: dmesg.output --]
[-- Type: text/plain, Size: 7273 bytes --]

NILFS: corrupt root inode.
NILFS: looking segment (seg_start=22401024, seg_end=22403071, segnum=10938, seg_seq=62136)
NILFS: found super root: segnum=10938, seq=62136, pseg_start=22401723, pseg_offset=730
NILFS: latest checkpoint=30778
nilfs_btnode_submit_block: read ino=3 vblocknr=22401752 blocknr=22401752
nilfs_btnode_submit_block: read ino=3 vblocknr=22401750 blocknr=22401750
nilfs_mdt_submit_block lookup: ino=3 blkoff=25589 blknum=22401741
nilfs_mdt_submit_block lookup: ino=3 blkoff=25590 blknum=20023970
nilfs_mdt_submit_block lookup: ino=3 blkoff=25591 blknum=20015776
nilfs_mdt_submit_block lookup: ino=3 blkoff=25592 blknum=20077247
nilfs_mdt_submit_block lookup: ino=3 blkoff=25593 blknum=21061324
nilfs_mdt_submit_block lookup: ino=3 blkoff=25594 blknum=21061325
nilfs_mdt_submit_block lookup: ino=3 blkoff=25595 blknum=20417338
nilfs_mdt_submit_block lookup: ino=3 blkoff=25596 blknum=20392792
nilfs_mdt_submit_block lookup: ino=3 blkoff=25597 blknum=20602217
nilfs_mdt_submit_block lookup: ino=3 blkoff=25598 blknum=21061326
nilfs_mdt_submit_block lookup: ino=3 blkoff=25599 blknum=21061327
nilfs_mdt_submit_block lookup: ino=3 blkoff=25600 blknum=20975305
nilfs_mdt_submit_block lookup: ino=3 blkoff=25601 blknum=20967082
nilfs_mdt_submit_block lookup: ino=3 blkoff=25602 blknum=21126881
nilfs_mdt_submit_block lookup: ino=3 blkoff=25603 blknum=20634953
nilfs_mdt_submit_block lookup: ino=3 blkoff=25604 blknum=20593993
nilfs_dat_translate: vblocknr=3262527 -> blocknr=22401729
nilfs_btnode_submit_block: read ino=4 vblocknr=3262527 blocknr=22401729
nilfs_btnode_submit_block: read ino=3 vblocknr=22401749 blocknr=22401749
nilfs_mdt_submit_block lookup: ino=3 blkoff=25446 blknum=20606318
nilfs_mdt_submit_block lookup: ino=3 blkoff=25447 blknum=20593987
nilfs_mdt_submit_block lookup: ino=3 blkoff=25448 blknum=21028552
nilfs_mdt_submit_block lookup: ino=3 blkoff=25449 blknum=21335832
nilfs_mdt_submit_block lookup: ino=3 blkoff=25450 blknum=20921992
nilfs_mdt_submit_block lookup: ino=3 blkoff=25451 blknum=20921993
nilfs_mdt_submit_block lookup: ino=3 blkoff=25452 blknum=19983010
nilfs_mdt_submit_block lookup: ino=3 blkoff=25453 blknum=21869087
nilfs_mdt_submit_block lookup: ino=3 blkoff=25454 blknum=20323108
nilfs_mdt_submit_block lookup: ino=3 blkoff=25455 blknum=20495207
nilfs_mdt_submit_block lookup: ino=3 blkoff=25456 blknum=20581411
nilfs_mdt_submit_block lookup: ino=3 blkoff=25457 blknum=20745683
nilfs_mdt_submit_block lookup: ino=3 blkoff=25458 blknum=22397612
nilfs_mdt_submit_block lookup: ino=3 blkoff=25459 blknum=21003983
nilfs_mdt_submit_block lookup: ino=3 blkoff=25460 blknum=21176067
nilfs_mdt_submit_block lookup: ino=3 blkoff=25461 blknum=21352218
nilfs_dat_translate: vblocknr=3244247 -> blocknr=20603353
nilfs_mdt_submit_block lookup: ino=4 blkoff=241 blknum=20603353
nilfs_btnode_submit_block: read ino=3 vblocknr=22401751 blocknr=22401751
nilfs_mdt_submit_block lookup: ino=3 blkoff=25826 blknum=22401745
nilfs_mdt_submit_block lookup: ino=3 blkoff=25827 blknum=20634960
nilfs_mdt_submit_block lookup: ino=3 blkoff=25828 blknum=21155593
nilfs_mdt_submit_block lookup: ino=3 blkoff=25829 blknum=21680660
nilfs_mdt_submit_block lookup: ino=3 blkoff=25830 blknum=22205035
nilfs_mdt_submit_block lookup: ino=3 blkoff=25831 blknum=19942046
nilfs_mdt_submit_block lookup: ino=3 blkoff=25832 blknum=20470634
nilfs_mdt_submit_block lookup: ino=3 blkoff=25833 blknum=20999888
nilfs_mdt_submit_block lookup: ino=3 blkoff=25834 blknum=21524975
nilfs_mdt_submit_block lookup: ino=3 blkoff=25835 blknum=22049363
nilfs_mdt_submit_block lookup: ino=3 blkoff=25836 blknum=19786311
nilfs_mdt_submit_block lookup: ino=3 blkoff=25837 blknum=20314919
nilfs_mdt_submit_block lookup: ino=3 blkoff=25838 blknum=20840030
nilfs_mdt_submit_block lookup: ino=3 blkoff=25839 blknum=21368604
nilfs_mdt_submit_block lookup: ino=3 blkoff=25840 blknum=21893675
nilfs_mdt_submit_block lookup: ino=3 blkoff=25841 blknum=19634717
nilfs_dat_translate: vblocknr=3292780 -> blocknr=22401727
nilfs_mdt_submit_block lookup: ino=4 blkoff=0 blknum=22401727
nilfs_mdt_submit_block lookup: ino=3 blkoff=25417 blknum=22254189
nilfs_mdt_submit_block lookup: ino=3 blkoff=25418 blknum=21611014
nilfs_mdt_submit_block lookup: ino=3 blkoff=25419 blknum=21459426
nilfs_mdt_submit_block lookup: ino=3 blkoff=25420 blknum=21167869
nilfs_mdt_submit_block lookup: ino=3 blkoff=25421 blknum=20515697
nilfs_mdt_submit_block lookup: ino=3 blkoff=25422 blknum=20499307
nilfs_mdt_submit_block lookup: ino=3 blkoff=25423 blknum=20864598
nilfs_mdt_submit_block lookup: ino=3 blkoff=25424 blknum=20844113
nilfs_mdt_submit_block lookup: ino=3 blkoff=25425 blknum=20864599
nilfs_mdt_submit_block lookup: ino=3 blkoff=25426 blknum=20864600
nilfs_mdt_submit_block lookup: ino=3 blkoff=25427 blknum=22401739
nilfs_mdt_submit_block lookup: ino=3 blkoff=25428 blknum=20614524
nilfs_mdt_submit_block lookup: ino=3 blkoff=25429 blknum=21298959
nilfs_mdt_submit_block lookup: ino=3 blkoff=25430 blknum=21664292
nilfs_mdt_submit_block lookup: ino=3 blkoff=25431 blknum=21611016
nilfs_mdt_submit_block lookup: ino=3 blkoff=25432 blknum=21910050
nilfs_dat_translate: vblocknr=3240647 -> blocknr=20617314
nilfs_btnode_submit_block: read ino=6 vblocknr=3240647 blocknr=20617314
nilfs_mdt_submit_block lookup: ino=3 blkoff=25416 blknum=22241917
nilfs_dat_translate: vblocknr=3240498 -> blocknr=3280927
nilfs_mdt_submit_block lookup: ino=6 blkoff=2 blknum=3280927
nilfs_dat_translate: vblocknr=3240490 -> blocknr=20843792
nilfs_mdt_submit_block lookup: ino=6 blkoff=3 blknum=20843792
nilfs_dat_translate: vblocknr=3240646 -> blocknr=20617313
nilfs_mdt_submit_block lookup: ino=6 blkoff=4 blknum=20617313
nilfs_dat_translate: vblocknr=3240678 -> blocknr=20621795
nilfs_mdt_submit_block lookup: ino=6 blkoff=5 blknum=20621795
nilfs_dat_translate: vblocknr=3240679 -> blocknr=20621796
nilfs_mdt_submit_block lookup: ino=6 blkoff=6 blknum=20621796
nilfs_dat_translate: vblocknr=3240697 -> blocknr=20621797
nilfs_mdt_submit_block lookup: ino=6 blkoff=7 blknum=20621797
nilfs_dat_translate: vblocknr=3240652 -> blocknr=20621776
nilfs_mdt_submit_block lookup: ino=6 blkoff=8 blknum=20621776
nilfs_dat_translate: vblocknr=3240653 -> blocknr=20621777
nilfs_mdt_submit_block lookup: ino=6 blkoff=9 blknum=20621777
nilfs_dat_translate: vblocknr=3240654 -> blocknr=20621778
nilfs_mdt_submit_block lookup: ino=6 blkoff=10 blknum=20621778
nilfs_dat_translate: vblocknr=3240655 -> blocknr=20621779
nilfs_mdt_submit_block lookup: ino=6 blkoff=11 blknum=20621779
nilfs_dat_translate: vblocknr=3240656 -> blocknr=20621780
nilfs_mdt_submit_block lookup: ino=6 blkoff=12 blknum=20621780
nilfs_dat_translate: vblocknr=3240657 -> blocknr=20621781
nilfs_mdt_submit_block lookup: ino=6 blkoff=13 blknum=20621781
nilfs_dat_translate: vblocknr=3240658 -> blocknr=20621782
nilfs_mdt_submit_block lookup: ino=6 blkoff=14 blknum=20621782
nilfs_dat_translate: vblocknr=3240659 -> blocknr=20621783
nilfs_mdt_submit_block lookup: ino=6 blkoff=15 blknum=20621783
nilfs_dat_translate: vblocknr=3240660 -> blocknr=20621784
nilfs_mdt_submit_block lookup: ino=6 blkoff=16 blknum=20621784
nilfs_dat_translate: vblocknr=3240661 -> blocknr=20621785
nilfs_mdt_submit_block lookup: ino=6 blkoff=17 blknum=20621785
NILFS: corrupt root inode.

[-- Attachment #3: Type: text/plain, Size: 158 bytes --]

_______________________________________________
users mailing list
users-JrjvKiOkagjYtjvyW6yDsg@public.gmane.org
https://www.nilfs.org/mailman/listinfo/users

Re: Fail to mount NILFS.

[Ryusuke Konishi]

I'll resend to the list, too.
On Sun, 23 Aug 2009 16:06:17 +0300, Yuri Chislov wrote:
> Resend to mailing list.
> On Sunday 23 August 2009 11:20:53 you wrote:
> > And, the attached patch will output some information about how disk
> > block numbers are converted with DAT file.
> >
> > I think this will give another hint on which metadata broke in nilfs.
> >
> > Could you apply it and try mount again?
> Done.
> 
> 104       17  209720511 cciss/c0d1p1
> expr 209720511 / 4 / 2048 - 1
> 25599
> ./find_root.sh /dev/cciss/c0d1p1 25599
> find_root.sh: root inode not found
> 
> Attached dmesg output.
> 
> Thanks.
> Yuri.
> 

Thank you for your cooperation.
I'll look into the dmesg output.

Thanks,
Ryusuke Konishi

Re: Fail to mount NILFS.

[Ryusuke Konishi]

Hi Yuri,
On Sun, 23 Aug 2009 22:09:41 +0900 (JST), Ryusuke Konishi wrote:
> I'll resend to the list, too.
> On Sun, 23 Aug 2009 16:06:17 +0300, Yuri Chislov wrote:
> > Resend to mailing list.
> > On Sunday 23 August 2009 11:20:53 you wrote:
> > > And, the attached patch will output some information about how disk
> > > block numbers are converted with DAT file.
> > >
> > > I think this will give another hint on which metadata broke in nilfs.
> > >
> > > Could you apply it and try mount again?
> > Done.
> > 
> > 104       17  209720511 cciss/c0d1p1
> > expr 209720511 / 4 / 2048 - 1
> > 25599
> > ./find_root.sh /dev/cciss/c0d1p1 25599
> > find_root.sh: root inode not found
> > 
> > Attached dmesg output.
> > 
> > Thanks.
> > Yuri.
> > 
> 
> Thank you for your cooperation.
> I'll look into the dmesg output.
> 
> Thanks,
> Ryusuke Konishi

I found the location of the root inode in your log.

> nilfs_dat_translate: vblocknr=3240647 -> blocknr=20617314
> nilfs_btnode_submit_block: read ino=6 vblocknr=3240647 blocknr=20617314
> nilfs_mdt_submit_block lookup: ino=3 blkoff=25416 blknum=22241917
> nilfs_dat_translate: vblocknr=3240498 -> blocknr=3280927
> nilfs_mdt_submit_block lookup: ino=6 blkoff=2 blknum=3280927
                                 ^^^^^       ^^        ^^^^^^^
                                 ifile       |
                                             `-- Offset of the block in
                                                 which the root inode is
                                                 written.

Could you send me the two dump files taken by the following steps ?

 # dd if=/dev/cciss/c0d1p1 bs=4k count=1 skip=3280927 | hd > blk3280927.log
 # expr 3280927 / 2048
 1602
 # dumpseg /dev/cciss/c0d1p1 1602 > seg1602.log


Thanks,
Ryusuke Konishi

Re: Fail to mount NILFS.

[Yuri Chislov]

[-- Attachment #1: Type: text/plain, Size: 366 bytes --]

On Monday 24 August 2009 19:08:47 you wrote:
> Could you send me the two dump files taken by the following steps ?
>
>  # dd if=/dev/cciss/c0d1p1 bs=4k count=1 skip=3280927 | hd > blk3280927.log
>  # expr 3280927 / 2048
>  1602
>  # dumpseg /dev/cciss/c0d1p1 1602 > seg1602.log
Hi,

 As I understand hd is hexdump alias.
Archive with files attached.

Thanks.
Yuri. 

[-- Attachment #2: seg1602.tbz2 --]
[-- Type: application/x-bzip-compressed-tar, Size: 12414 bytes --]

[-- Attachment #3: Type: text/plain, Size: 158 bytes --]

_______________________________________________
users mailing list
users-JrjvKiOkagjYtjvyW6yDsg@public.gmane.org
https://www.nilfs.org/mailman/listinfo/users

Re: Fail to mount NILFS.

[Ryusuke Konishi]

Hi,
On Mon, 24 Aug 2009 19:21:35 +0300, Yuri Chislov wrote:
> On Monday 24 August 2009 19:08:47 you wrote:
> > Could you send me the two dump files taken by the following steps ?
> >
> >  # dd if=/dev/cciss/c0d1p1 bs=4k count=1 skip=3280927 | hd > blk3280927.log
> >  # expr 3280927 / 2048
> >  1602
> >  # dumpseg /dev/cciss/c0d1p1 1602 > seg1602.log
> Hi,
> 
>  As I understand hd is hexdump alias.
> Archive with files attached.
> 
> Thanks.
> Yuri. 

Thank you for your very quick response ;)

The segment was unfortunately overwritten with irrelevant files, and
the block was not that of the root inode.

According to the log, it happened on 20th August 12:22:24.


I guess garbage collector mistakenly reclaimed the segment without
moving the root inode block to a new place.

For debugging, I want to know the content of the DAT block located at
#22241917.

Could you give me a hand one more time, please?

  # dd if=/dev/cciss/c0d1p1 of=blk22241917 bs=4k count=1 skip=22241917
  # expr 22241917 / 2048
  10860
  # dumpseg /dev/cciss/c0d1p1 10860 > seg10860.log

Thanks,
Ryusuke Konishi

Re: Fail to mount NILFS.

[Yuri Chislov]

[-- Attachment #1: Type: text/plain, Size: 310 bytes --]

On Monday 24 August 2009 20:33:49 Ryusuke Konishi wrote:
> Could you give me a hand one more time, please?
>
>   # dd if=/dev/cciss/c0d1p1 of=blk22241917 bs=4k count=1 skip=22241917
>   # expr 22241917 / 2048
>   10860
>   # dumpseg /dev/cciss/c0d1p1 10860 > seg10860.log
Done.

 tbz2 attached.

Thanks.
Yuri.

[-- Attachment #2: seg10860.tbz2 --]
[-- Type: application/x-bzip-compressed-tar, Size: 9750 bytes --]

[-- Attachment #3: Type: text/plain, Size: 158 bytes --]

_______________________________________________
users mailing list
users-JrjvKiOkagjYtjvyW6yDsg@public.gmane.org
https://www.nilfs.org/mailman/listinfo/users