nilfs error

nilfs error

[David Arendt]

  Hi,

yesterday I had the following error:

Sep 18 18:03:24 server kernel: NILFS: bad btree node 
(blocknr=226497086): level
= 37, flags = 0x18, nchildren = 22092
Sep 18 18:03:24 server kernel: NILFS error (device sdb1): 
nilfs_truncate_bmap: b
map is broken (ino=69)
Sep 18 18:03:24 server kernel: Remounting filesystem read-only
Sep 18 18:03:24 server kernel: nilfs_palloc_commit_free_entry: entry 
number 69 already freed

Remounting the filesytem worked correctly. What could be the cause of 
this error ? Should I preventively reformat the disk and recreate the 
filesystem ?

Thanks in advance
Bye,
David Arendt
--
To unsubscribe from this list: send the line "unsubscribe linux-nilfs" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: nilfs error

[Ryusuke Konishi]

Hi,
On Sun, 19 Sep 2010 07:36:56 +0200, David Arendt wrote:
>   Hi,
> 
> yesterday I had the following error:
> 
> Sep 18 18:03:24 server kernel: NILFS: bad btree node 
> (blocknr=226497086): level
> = 37, flags = 0x18, nchildren = 22092
> Sep 18 18:03:24 server kernel: NILFS error (device sdb1): 
> nilfs_truncate_bmap: b
> map is broken (ino=69)
> Sep 18 18:03:24 server kernel: Remounting filesystem read-only
> Sep 18 18:03:24 server kernel: nilfs_palloc_commit_free_entry: entry 
> number 69 already freed
> 
> Remounting the filesytem worked correctly. What could be the cause of 
> this error ?

Thank you for the report.

According to the log, nilfs detected btree corruption during file
deletion; the message "bad btree node ..." shows a btree node block
that nilfs read was broken.

One possible cause is garbage collection, or there may be a problem
with exclusion control on delete operation.  Or, the block might be
actually broken if the underlying device is not reliable.


Do you think nilfs_cleanerd worked before this trouble?

How much free space was left on the device?

> Should I preventively reformat the disk and recreate the 
> filesystem ?

Yes, it's recommended for safety.  If you haven't yet reformatted the
device, try the following command for sanity check.

 # find /mount-dir -type f -exec cat {} \; >/dev/null


Regards,
Ryusuke Konishi
--
To unsubscribe from this list: send the line "unsubscribe linux-nilfs" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: nilfs error

[David Arendt]

  Hi,

there is one unimportant corrupted file. Is it safe to use the file 
system after removing this file or should I go the backup/restore route ?

Garbage collection was running when the problem appeared.

The disk is a 1tb disk and 16gb have been free.

Thanks in advance,
Bye,
David Arendt

On 09/19/10 14:37, Ryusuke Konishi wrote:
> Hi,
> On Sun, 19 Sep 2010 07:36:56 +0200, David Arendt wrote:
>>    Hi,
>>
>> yesterday I had the following error:
>>
>> Sep 18 18:03:24 server kernel: NILFS: bad btree node
>> (blocknr=226497086): level
>> = 37, flags = 0x18, nchildren = 22092
>> Sep 18 18:03:24 server kernel: NILFS error (device sdb1):
>> nilfs_truncate_bmap: b
>> map is broken (ino=69)
>> Sep 18 18:03:24 server kernel: Remounting filesystem read-only
>> Sep 18 18:03:24 server kernel: nilfs_palloc_commit_free_entry: entry
>> number 69 already freed
>>
>> Remounting the filesytem worked correctly. What could be the cause of
>> this error ?
> Thank you for the report.
>
> According to the log, nilfs detected btree corruption during file
> deletion; the message "bad btree node ..." shows a btree node block
> that nilfs read was broken.
>
> One possible cause is garbage collection, or there may be a problem
> with exclusion control on delete operation.  Or, the block might be
> actually broken if the underlying device is not reliable.
>
>
> Do you think nilfs_cleanerd worked before this trouble?
>
> How much free space was left on the device?
>
>> Should I preventively reformat the disk and recreate the
>> filesystem ?
> Yes, it's recommended for safety.  If you haven't yet reformatted the
> device, try the following command for sanity check.
>
>   # find /mount-dir -type f -exec cat {} \;>/dev/null
>
>
> Regards,
> Ryusuke Konishi
> --
> To unsubscribe from this list: send the line "unsubscribe linux-nilfs" in
> the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

--
To unsubscribe from this list: send the line "unsubscribe linux-nilfs" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: nilfs error

[Ryusuke Konishi]

Hi,
On Sun, 19 Sep 2010 19:22:46 +0200, David Arendt wrote:
>   Hi,
> 
> there is one unimportant corrupted file. Is it safe to use the file 
> system after removing this file or should I go the backup/restore route ?

Making backup is of course recommended.

But if you don't mind, continue using it.  It may give us sort of a
clue, for instance, whether the corruption is restrictive (or
temporary, etc).

> Garbage collection was running when the problem appeared.
> 
> The disk is a 1tb disk and 16gb have been free.

Well, there may be something missed between garbage collection and
file deletion.  Will recheck.

Thanks,
Ryusuke Konishi


> Thanks in advance,
> Bye,
> David Arendt
> 
> On 09/19/10 14:37, Ryusuke Konishi wrote:
> > Hi,
> > On Sun, 19 Sep 2010 07:36:56 +0200, David Arendt wrote:
> >>    Hi,
> >>
> >> yesterday I had the following error:
> >>
> >> Sep 18 18:03:24 server kernel: NILFS: bad btree node
> >> (blocknr=226497086): level
> >> = 37, flags = 0x18, nchildren = 22092
> >> Sep 18 18:03:24 server kernel: NILFS error (device sdb1):
> >> nilfs_truncate_bmap: b
> >> map is broken (ino=69)
> >> Sep 18 18:03:24 server kernel: Remounting filesystem read-only
> >> Sep 18 18:03:24 server kernel: nilfs_palloc_commit_free_entry: entry
> >> number 69 already freed
> >>
> >> Remounting the filesytem worked correctly. What could be the cause of
> >> this error ?
> > Thank you for the report.
> >
> > According to the log, nilfs detected btree corruption during file
> > deletion; the message "bad btree node ..." shows a btree node block
> > that nilfs read was broken.
> >
> > One possible cause is garbage collection, or there may be a problem
> > with exclusion control on delete operation.  Or, the block might be
> > actually broken if the underlying device is not reliable.
> >
> >
> > Do you think nilfs_cleanerd worked before this trouble?
> >
> > How much free space was left on the device?
> >
> >> Should I preventively reformat the disk and recreate the
> >> filesystem ?
> > Yes, it's recommended for safety.  If you haven't yet reformatted the
> > device, try the following command for sanity check.
> >
> >   # find /mount-dir -type f -exec cat {} \;>/dev/null
> >
> >
> > Regards,
> > Ryusuke Konishi
> > --
> > To unsubscribe from this list: send the line "unsubscribe linux-nilfs" in
> > the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
> > More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 
> --
> To unsubscribe from this list: send the line "unsubscribe linux-nilfs" in
> the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
--
To unsubscribe from this list: send the line "unsubscribe linux-nilfs" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

nilfs error

[Jon Watte]

First, let me describe my usage pattern, which is boring -- the
interesting technical stuff comes later down.

I have a mini-ATX computer with a small (32 GB) SSD, which I have
partitioned with two small partitions for /boot and swap, and one
large partition for nilfs2 root-and-everything-else.
This system has been running fine, using Arch Linux distribution x64
flavor for over a year.
I update it to the latest Arch kernel weekly, which means I'm running
fairly new code. Gotta live dangerously!
The usage pattern for this system is light; mostly web browsing (using
chromium) and software development (using vim, gcc, make, and
LXTerminal.)

Two days ago, after the latest kernel update, the drive turned
read-only in the middle of usage. I could not use "shutdown" because
"sudo" could not write its logfiles.
This has happened before when I've filled the disk too full without
noticing, so I just turned off power without shutdown (I often do this
anyway -- a "sync" in terminal and a toggle switch power-off a few
seconds later.) Typically, the checkpoint cleanup on boot would clear
this up if this was the problem.

However, yesterday when booting up agian, it refused to mount the root
disk, and booting was halted, with the messages shown in the below
photograph. Note that, because I can't boot, I can't get this output
in a copy-and-paste-able form!

https://pbs.twimg.com/media/BaxOmptCYAABQuW.jpg:large

The two repeated messages are (typed in from the photo):
NILFS: bad bree node (blocknr=1364234) ; level = 100; flags = 0x73;
nchildren = 12336
NILFS error (device sda1): nilfs_bmap_lookup_contig: broken bmap
(inode number=3935)

I cloned the sda1 partition (the main partition) to my file server for
posterity, and re-formatted the hard drive using btrfs for now until I
can get clarity on this problem.

Interesting observation: That disk image CAN be mounted if I boot the
Arch-ISO distribution image from March 2013, but can NOT be mounted if
I try to boot from it with a later kernel. Maybe someone better with
"pacman" than I am can compare kernel versions from March 2013 vs Now?

I don't really want to put this image up for FTP because it contains
my SSH private keys (one of which is not pass-phrased) but if there is
something I can do with it to help debug this occurrence, I'd be happy
to try it.


This is very sad, because I've had nothing but good performance from
nilfs since I installed it 18 months ago.


Sincerely,

Jon Watte


--
"I pledge allegiance to the flag of the United States of America, and
to the republic for which it stands, one nation indivisible, with
liberty and justice for all."
~ Adopted by U.S. Congress, June 22, 1942
--
To unsubscribe from this list: send the line "unsubscribe linux-nilfs" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: nilfs error

[Vyacheslav Dubeyko]

On Dec 7, 2013, at 5:17 AM, Jon Watte wrote:

[snip]
> However, yesterday when booting up agian, it refused to mount the root
> disk, and booting was halted, with the messages shown in the below
> photograph. Note that, because I can't boot, I can't get this output
> in a copy-and-paste-able form!
> 
> https://pbs.twimg.com/media/BaxOmptCYAABQuW.jpg:large
> 
> The two repeated messages are (typed in from the photo):
> NILFS: bad bree node (blocknr=1364234) ; level = 100; flags = 0x73;
> nchildren = 12336
> NILFS error (device sda1): nilfs_bmap_lookup_contig: broken bmap
> (inode number=3935)

What linux kernel version do you have?

I had fixed the reason of such problem by patch [1]. So, please check and
share your kernel version, firstly. I suppose that you are simply using old
kernel version.

With the best regards,
Vyacheslav Dubeyko.

[1] http://comments.gmane.org/gmane.comp.file-systems.nilfs.user/3082

--
To unsubscribe from this list: send the line "unsubscribe linux-nilfs" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: nilfs error

[Jon Watte]

I am using Linux robot1 3.12.3-1-ARCH #1 SMP PREEMPT Wed Dec 4
21:45:42 CET 2013 x86_64 GNU/Linux

Is that an "old" kernel?

Also, note that an "old" kernel (from February this year) could mount
this image, whereas this newer kernel hung. Is that consistent with
the changes you have done?

Sincerely,

jw




Sincerely,

Jon Watte


--
"I pledge allegiance to the flag of the United States of America, and
to the republic for which it stands, one nation indivisible, with
liberty and justice for all."
~ Adopted by U.S. Congress, June 22, 1942


On Sat, Dec 7, 2013 at 2:34 AM, Vyacheslav Dubeyko <slava-yeENwD64cLxBDgjK7y7TUQ@public.gmane.org> wrote:
>
> On Dec 7, 2013, at 5:17 AM, Jon Watte wrote:
>
> [snip]
>> However, yesterday when booting up agian, it refused to mount the root
>> disk, and booting was halted, with the messages shown in the below
>> photograph. Note that, because I can't boot, I can't get this output
>> in a copy-and-paste-able form!
>>
>> https://pbs.twimg.com/media/BaxOmptCYAABQuW.jpg:large
>>
>> The two repeated messages are (typed in from the photo):
>> NILFS: bad bree node (blocknr=1364234) ; level = 100; flags = 0x73;
>> nchildren = 12336
>> NILFS error (device sda1): nilfs_bmap_lookup_contig: broken bmap
>> (inode number=3935)
>
> What linux kernel version do you have?
>
> I had fixed the reason of such problem by patch [1]. So, please check and
> share your kernel version, firstly. I suppose that you are simply using old
> kernel version.
>
> With the best regards,
> Vyacheslav Dubeyko.
>
> [1] http://comments.gmane.org/gmane.comp.file-systems.nilfs.user/3082
>
--
To unsubscribe from this list: send the line "unsubscribe linux-nilfs" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: nilfs error

[Vyacheslav Dubeyko]

On Mon, 2013-12-09 at 08:33 -0800, Jon Watte wrote:
> I am using Linux robot1 3.12.3-1-ARCH #1 SMP PREEMPT Wed Dec 4
> 21:45:42 CET 2013 x86_64 GNU/Linux
> 
> Is that an "old" kernel?
> 

I can see that v3.12.2 tag contains the patch is mentioned by me.

> Also, note that an "old" kernel (from February this year) could mount
> this image, whereas this newer kernel hung. Is that consistent with
> the changes you have done?
> 

> NILFS: bad bree node (blocknr=1364234) ; level = 100; flags = 0x73; nchildren = 12336
> NILFS error (device sda1): nilfs_bmap_lookup_contig: broken bmap (inode number=3935)

Such error messages mean that you have corrupted NILFS2 volume. My patch
doesn't recover NILFS2 volume. It fixes the reason of such corruption.
And corruption of NILFS2 volume takes place without any visible
symptoms. So, you had volume corruption more earlier than you have
discovered the issue. I have such vision of the situation on your side.
Your NILFS2 volume was corrupted on kernel version that it doesn't
contain my patch. And you had discovered the issue after kernel update
only by chance. It is simply coincidence, from my point of view.

I suppose that error messages are emitted by nilfs-cleanerd. So, you can
mount volume without any troubles if nilfs-cleanerd doesn't work.

Of course, I can be not fully correct in treating of your situation. The
error message about broken bmap is very common and there are many
possible reason of it. But, anyway, this message means corruption of
volume. As a result, I need in reproducing path of the issue. I can't
investigate and fix the issue without opportunity of reproducing the
issue on my side. Can you reproduce the issue from the state of valid
volume to corrupted state stably? And could you share reproducing path
of the issue?

If I am right in understanding of your issue then I suppose that
possible solution can be: (1) create new NILFS2 volume; (2) copy valid
data from corrupted NILFS2 volume; (3) continue using new NILFS2 volume
under last version of kernel version. Unfortunately, we haven't fsck
tool for NILFS2 yet.

Thanks,
Vyacheslav Dubeyko.


--
To unsubscribe from this list: send the line "unsubscribe linux-nilfs" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: nilfs error

[Jon Watte]

Thanks for your suggestions. I have no idea about how to reproduce the
issue -- this is a light-to-medium-use desktop system.
I took an image of that volume and can mount it on an older kernel if
I need to copy out any files, although all of the important stuff was
in git or dropbox off-site, so no data really lost.
Is there anything I can do to this volume that would give you more
information about what the cause might be (i e, whether the cause is
patched or not)?
Some debug utility that will create a dump of some sort that will
help? Else I will probably just wipe that image and recover the 30G of
space.

Sincerely,

jw




Sincerely,

Jon Watte


--
"I pledge allegiance to the flag of the United States of America, and
to the republic for which it stands, one nation indivisible, with
liberty and justice for all."
~ Adopted by U.S. Congress, June 22, 1942


On Mon, Dec 9, 2013 at 10:45 PM, Vyacheslav Dubeyko <slava-yeENwD64cLxBDgjK7y7TUQ@public.gmane.org> wrote:
> On Mon, 2013-12-09 at 08:33 -0800, Jon Watte wrote:
>> I am using Linux robot1 3.12.3-1-ARCH #1 SMP PREEMPT Wed Dec 4
>> 21:45:42 CET 2013 x86_64 GNU/Linux
>>
>> Is that an "old" kernel?
>>
>
> I can see that v3.12.2 tag contains the patch is mentioned by me.
>
>> Also, note that an "old" kernel (from February this year) could mount
>> this image, whereas this newer kernel hung. Is that consistent with
>> the changes you have done?
>>
>
>> NILFS: bad bree node (blocknr=1364234) ; level = 100; flags = 0x73; nchildren = 12336
>> NILFS error (device sda1): nilfs_bmap_lookup_contig: broken bmap (inode number=3935)
>
> Such error messages mean that you have corrupted NILFS2 volume. My patch
> doesn't recover NILFS2 volume. It fixes the reason of such corruption.
> And corruption of NILFS2 volume takes place without any visible
> symptoms. So, you had volume corruption more earlier than you have
> discovered the issue. I have such vision of the situation on your side.
> Your NILFS2 volume was corrupted on kernel version that it doesn't
> contain my patch. And you had discovered the issue after kernel update
> only by chance. It is simply coincidence, from my point of view.
>
> I suppose that error messages are emitted by nilfs-cleanerd. So, you can
> mount volume without any troubles if nilfs-cleanerd doesn't work.
>
> Of course, I can be not fully correct in treating of your situation. The
> error message about broken bmap is very common and there are many
> possible reason of it. But, anyway, this message means corruption of
> volume. As a result, I need in reproducing path of the issue. I can't
> investigate and fix the issue without opportunity of reproducing the
> issue on my side. Can you reproduce the issue from the state of valid
> volume to corrupted state stably? And could you share reproducing path
> of the issue?
>
> If I am right in understanding of your issue then I suppose that
> possible solution can be: (1) create new NILFS2 volume; (2) copy valid
> data from corrupted NILFS2 volume; (3) continue using new NILFS2 volume
> under last version of kernel version. Unfortunately, we haven't fsck
> tool for NILFS2 yet.
>
> Thanks,
> Vyacheslav Dubeyko.
>
>
--
To unsubscribe from this list: send the line "unsubscribe linux-nilfs" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: nilfs error

[Vyacheslav Dubeyko]

On Tue, 2013-12-10 at 08:47 -0800, Jon Watte wrote:
> Thanks for your suggestions. I have no idea about how to reproduce the
> issue -- this is a light-to-medium-use desktop system.

I think that it is important to understand a nature of workload on your
system. Maybe do you use some application frequently or process some
data mainly on this system? Is it possible to distinguish frequent and
repeatable workload?

> I took an image of that volume and can mount it on an older kernel if
> I need to copy out any files, although all of the important stuff was
> in git or dropbox off-site, so no data really lost.
> Is there anything I can do to this volume that would give you more
> information about what the cause might be (i e, whether the cause is
> patched or not)?
> Some debug utility that will create a dump of some sort that will
> help? Else I will probably just wipe that image and recover the 30G of
> space.
> 

I think that I have a request to you. It is very important to understand
that my patch fixes the issue really. So, please, take attention to the
situation on newly created NILFS2 volume under Linux kernel with the
patch. If you will encounter the issue again then it means that the
reason of the issue doesn't be fixed.

I believe that I had fixed the issue's reason. But there is probability
to have another reason of such error messages.

Thanks,
Vyacheslav Dubeyko.


--
To unsubscribe from this list: send the line "unsubscribe linux-nilfs" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html