Re: NILFS: bad btree node

[Vyacheslav Dubeyko <slava-yeENwD64cLxBDgjK7y7TUQ@public.gmane.org>]

On Thu, 2012-12-20 at 17:08 +0800, 张 磊 wrote:
> Hi,
> 
> I remounted the filesystem, and started the MySQLs. The filesytstem became readonly again.
> 
> Dec 20 16:03:31 localhost kernel: segctord starting. Construction interval = 60 seconds, CP frequency < 30 seconds
> Dec 20 16:03:31 localhost kernel: NILFS warning: mounting fs with errors
> Dec 20 16:03:31 localhost nilfs_cleanerd[29120]: start
> Dec 20 16:03:55 localhost kernel: NILFS: bad btree node (blocknr=710153406): level = 0, flags = 0x2, nchildren = 25088
> Dec 20 16:03:55 localhost kernel: NILFS error (device sdb2): nilfs_bmap_lookup_contig: broken bmap (inode number=321775)
> Dec 20 16:03:55 localhost kernel:
> Dec 20 16:03:55 localhost kernel: Remounting filesystem read-only
> Dec 20 16:03:55 localhost kernel: NILFS: bad btree node (blocknr=710153406): level = 0, flags = 0x2, nchildren = 25088
> Dec 20 16:03:55 localhost kernel: NILFS error (device sdb2): nilfs_bmap_lookup_contig: broken bmap (inode number=321775)
> Dec 20 16:03:55 localhost kernel:
> Dec 20 16:03:57 localhost nilfs_cleanerd[29120]: cannot clean segments: Read-only file system
> Dec 20 16:03:57 localhost nilfs_cleanerd[29120]: shutdown
> 
> I remounted the filesystem again, and tried to delete the bad files, but delete failed.
> 
> Dec 20 16:04:02 localhost kernel: segctord starting. Construction interval = 60 seconds, CP frequency < 30 seconds
> Dec 20 16:04:02 localhost kernel: NILFS warning: mounting fs with errors
> Dec 20 16:04:02 localhost nilfs_cleanerd[30054]: start
> Dec 20 16:12:08 localhost kernel: NILFS: bad btree node (blocknr=710153406): level = 0, flags = 0x2, nchildren = 25088
> Dec 20 16:12:08 localhost kernel: NILFS error (device sdb2): nilfs_bmap_last_key: broken bmap (inode number=321775)
> Dec 20 16:12:08 localhost kernel:
> Dec 20 16:12:08 localhost kernel: Remounting filesystem read-only
> Dec 20 16:12:08 localhost kernel: NILFS warning (device sdb2): nilfs_truncate_bmap: failed to truncate bmap (ino=321775, err=-5)
> Dec 20 16:12:08 localhost nilfs_cleanerd[30054]: cannot clean segments: Read-only file system
> Dec 20 16:12:08 localhost nilfs_cleanerd[30054]: shutdown
> 
> I tried a third remount, but failed. The server was down, and restarted.
> 
> Dec 20 16:12:42 localhost kernel: NILFS warning (device sdb2): nilfs_detach_log_writer: Hit dirty file after stopped log writer
> Dec 20 16:12:42 localhost kernel:
> 

Yes, it is bad. The remount solves the trouble earlier.

As a result, do you have NILFS2 volume mounted as read-only?

Could you share more details about your environment? It needs for
understanding situation and trying to reproduce. I need to know:
1. Linux kernel version.
2. nilfs-utils version.
3. "mount" output.
4. "df -h" output.
5. "lscp" output.
6. "lssu" output.
7. "nilfs-tune -l" output (superblock content)

> I found that fsck.nilfs2 was added into nilfs-utils v4. Can I try it? Where can I download nilfs-utils v4?
> 

Last version of nilfs-utils is 2.1.4. Currently, fsck.nilfs2 is on early
stage of development. The v4 is a fsck.nilfs2 patchset version. You can
try fsck.nilfs2 after applying this patchset on source code of
nilfs-utils of 2.1.4 version. But fsck.nilfs2 can check only superblocks
and segment summary headers and can't recover completely. So, I think
that it will be useless for you.

With the best regards,
Vyacheslav Dubeyko.

> 在 2012-12-20，14:08，Vyacheslav Dubeyko <slava@dubeyko.com> 写道：
> 
> > Hi,
> > 
> > On Thu, 2012-12-20 at 10:46 +0800, 张 磊 wrote:
> >> Hello.
> >> 	My nilfs suddenly become read-only. I saw these logs in /var/log/messages:
> >> 
> >> Dec 19 11:20:05 localhost kernel: NILFS: bad btree node (blocknr=710153406): level = 0, flags = 0x2, nchildren = 25088
> >> Dec 19 11:20:05 localhost kernel: NILFS error (device sdb2): nilfs_bmap_lookup_contig: broken bmap (inode number=321775)
> >> Dec 19 11:20:05 localhost kernel:
> >> Dec 19 11:20:05 localhost kernel: Remounting filesystem read-only
> >> Dec 19 11:20:05 localhost kernel: NILFS: bad btree node (blocknr=710153406): level = 0, flags = 0x2, nchildren = 25088
> >> Dec 19 11:20:05 localhost kernel: NILFS error (device sdb2): nilfs_bmap_lookup_contig: broken bmap (inode number=321775)
> >> Dec 19 11:20:05 localhost kernel:
> >> Dec 19 11:20:05 localhost kernel: NILFS: bad btree node (blocknr=710153406): level = 0, flags = 0x2, nchildren = 25088
> >> Dec 19 11:20:05 localhost kernel: NILFS error (device sdb2): nilfs_bmap_lookup_contig: broken bmap (inode number=321775)
> >> Dec 19 11:20:05 localhost kernel:
> >> Dec 19 11:20:05 localhost kernel: NILFS: bad btree node (blocknr=710153406): level = 0, flags = 0x2, nchildren = 25088
> >> Dec 19 11:20:05 localhost kernel: NILFS error (device sdb2): nilfs_bmap_lookup_contig: broken bmap (inode number=321775)
> >> Dec 19 11:20:05 localhost kernel:
> >> ……………………………………………………
> >> 
> >> 	How can I fix this? There is 6TiB data on my disk, I don't want to format the disk.
> >> 	I found that a lot of people have encountered the same problem. Is this a bug of nilfs? How can I avoid this problem? When it happens, I was running multiple MySQL and rsync, and nilfs_cleanerd was cleaning segments.
> >> 
> > 
> > Yes, this issue was reported earlier. As I understand, you can simply
> > remount your filesystem in read-write mode and to continue using your
> > NILFS2 filesystem.
> > 
> > If you will encounter any troubles with remounting, please, report about
> > it.
> > 
> > With the best regards,
> > Vyacheslav Dubeyko.
> > 
> > 
> >> Elmer Zhang--
> >> To unsubscribe from this list: send the line "unsubscribe linux-nilfs" in
> >> the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
> >> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> > 
> > 
> > --
> > To unsubscribe from this list: send the line "unsubscribe linux-nilfs" in
> > the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
> > More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 


--
To unsubscribe from this list: send the line "unsubscribe linux-nilfs" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html