nilfs_cleanerd from nilfs-utils shutdown on version 2.0 and 2.1 does not fail but says nothing and does not clean the old checkpoints nor newer (actually older) ones.

nilfs_cleanerd from nilfs-utils shutdown on version 2.0 and 2.1 does not fail but says nothing and does not clean the old checkpoints nor newer (actually older) ones.

[Zahid Chowdhury]

Hello,
  If I move the system date forward, have some checkpoints created and then move the date backward a 2.0 cleanerd daemon fails on this error:
    Nov 30 14:39:37 nilfs_cleanerd[5789]: start
    Nov 30 14:39:38 kernel: nilfs_ioctl_move_inode_block: conflicting data             
        buffer: ino=4, cno=0, offset=0, blocknr=665655, vblocknr=566462
    Nov 30 14:39:38 kernel: NILFS: GC failed during preparation: cannot read 
        source blocks: err=-17
    Nov 30 14:39:38 nilfs_cleanerd[5789]: cannot clean segments: File exists
    Nov 30 14:39:38 nilfs_cleanerd[5789]: shutdown

I cannot ever start up the daemon. If I move to a 2.1 daemon, then it logs no errors, but it cleans no old or newer (really older) checkpoints - it just sits in a do-nothing mode (strace(1) shows he is hung on a mq_timedreceive syscall).

On one date change I did receive this message from the 2.0 daemon (the 2.1 daemon never says anything):
    Nov 30 13:45:30 nilfs_cleanerd[1642]: protection time rewinded: old
      period >= 1322747210, new period >= 1322689515

The fix around this is to umount the nilfs partition, move the date to after the lscp(1) future checkpoints, mount nilfs with the cleanerd, get the checkpoints cleaned up, umount nilfs, move the date back, and then mount nilfs.

Has anybody seen this error or are there any other workarounds to this error. Thanks a lot.

Zahid


--
To unsubscribe from this list: send the line "unsubscribe linux-nilfs" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: nilfs_cleanerd from nilfs-utils shutdown on version 2.0 and 2.1 does not fail but says nothing and does not clean the old checkpoints nor newer (actually older) ones.

[dexen deVries]

Hi Zahid,


On Saturday 03 December 2011 01:33:09 you wrote:
> (...)
> I cannot ever start up the daemon. If I move to a 2.1 daemon, then it logs
> no errors, but it cleans no old or newer (really older) checkpoints - it
> just sits in a do-nothing mode (strace(1) shows he is hung on a
> mq_timedreceive syscall).
> (...)

nilfs_cleanerd creates sort of a lock file in /dev/shm, named `sem.nilfs-
cleanerd-$PID'. nilfs_cleanerd version 2.1 refuses to process a filesystem if 
it has an associated /dev/shm/sem.nilfs-cleanerd-$PID file -- to protect from 
corruption occuring when multiple cleanerds accessed same filesystem. This 
looks in strace as being stuck at mq_timedreceive syscall.

All files in /dev/shm/ disappear after reboot (it's a temporary filesystem) so 
you don't usually see this behavior. However, when you start a new 
nilfs_cleanerd (v2.1) process without reboot, you need to clean relevant file 
by hand. Do ensure the old cleanerd process is dead before deleting the file. 
Otherwise corruption will happen when multiple cleanerd access same 
filesystem.


On Saturday 03 December 2011 01:33:09 you wrote:
>   If I move the system date forward, have some checkpoints created and then
> move the date backward a 2.0 cleanerd daemon fails on this error: Nov 30
> 14:39:37 nilfs_cleanerd[5789]: start
>     Nov 30 14:39:38 kernel: nilfs_ioctl_move_inode_block: conflicting data
>         buffer: ino=4, cno=0, offset=0, blocknr=665655, vblocknr=566462
>     Nov 30 14:39:38 kernel: NILFS: GC failed during preparation: cannot
> read source blocks: err=-17
>     Nov 30 14:39:38 nilfs_cleanerd[5789]: cannot clean segments: File
> exists Nov 30 14:39:38 nilfs_cleanerd[5789]: shutdown
> (...)

I got similar (or same) error with older kernel. Removing all checkpoints with 
rmcp helped -- but that doesn't seem like a 100% reliable solution to me. 
Right now I'm using kernels v3.1 and 3.2-rc3; seem rock-solid.


Regards,
-- 
dexen deVries

> Gresham’s Law for Computing:
>   The Fast drives out the Slow even if the Fast is Wrong.

William Kahan in
http://www.cs.berkeley.edu/~wkahan/Stnfrd50.pdf
--
To unsubscribe from this list: send the line "unsubscribe linux-nilfs" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

RE: nilfs_cleanerd from nilfs-utils shutdown on version 2.0 and 2.1 does not fail but says nothing and does not clean the old checkpoints nor newer (actually older) ones.

[Zahid Chowdhury]

Hi Dexen,
  I did have old cleanerd lock files from the crashed 2.0.23 in /dev/shm. The upgrade to 2.1 was hung on the lock files as you surmised on lock file removal 2.1 cleaned all the checkpoints. My box is using kernel 3.0.4, but I do have quite a few boxes still running kernel 2.6.18 from Centos 5.5 with the nilfs2 kernel mod back-ported to it (i.e. a builtin), so I will need to check if those boxes come out of the wedge of rewinded dates. Thanks a lot.

Zahid

-----Original Message-----
From: dexen deVries [mailto:dexen.devries-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org] 
Sent: Saturday, December 03, 2011 4:34 AM
To: linux-nilfs-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
Cc: Zahid Chowdhury
Subject: Re: nilfs_cleanerd from nilfs-utils shutdown on version 2.0 and 2.1 does not fail but says nothing and does not clean the old checkpoints nor newer (actually older) ones.

Hi Zahid,


On Saturday 03 December 2011 01:33:09 you wrote:
> (...)
> I cannot ever start up the daemon. If I move to a 2.1 daemon, then it logs
> no errors, but it cleans no old or newer (really older) checkpoints - it
> just sits in a do-nothing mode (strace(1) shows he is hung on a
> mq_timedreceive syscall).
> (...)

nilfs_cleanerd creates sort of a lock file in /dev/shm, named `sem.nilfs-
cleanerd-$PID'. nilfs_cleanerd version 2.1 refuses to process a filesystem if 
it has an associated /dev/shm/sem.nilfs-cleanerd-$PID file -- to protect from 
corruption occuring when multiple cleanerds accessed same filesystem. This 
looks in strace as being stuck at mq_timedreceive syscall.

All files in /dev/shm/ disappear after reboot (it's a temporary filesystem) so 
you don't usually see this behavior. However, when you start a new 
nilfs_cleanerd (v2.1) process without reboot, you need to clean relevant file 
by hand. Do ensure the old cleanerd process is dead before deleting the file. 
Otherwise corruption will happen when multiple cleanerd access same 
filesystem.


On Saturday 03 December 2011 01:33:09 you wrote:
>   If I move the system date forward, have some checkpoints created and then
> move the date backward a 2.0 cleanerd daemon fails on this error: Nov 30
> 14:39:37 nilfs_cleanerd[5789]: start
>     Nov 30 14:39:38 kernel: nilfs_ioctl_move_inode_block: conflicting data
>         buffer: ino=4, cno=0, offset=0, blocknr=665655, vblocknr=566462
>     Nov 30 14:39:38 kernel: NILFS: GC failed during preparation: cannot
> read source blocks: err=-17
>     Nov 30 14:39:38 nilfs_cleanerd[5789]: cannot clean segments: File
> exists Nov 30 14:39:38 nilfs_cleanerd[5789]: shutdown
> (...)

I got similar (or same) error with older kernel. Removing all checkpoints with 
rmcp helped -- but that doesn't seem like a 100% reliable solution to me. 
Right now I'm using kernels v3.1 and 3.2-rc3; seem rock-solid.


Regards,
-- 
dexen deVries

> Gresham's Law for Computing:
>   The Fast drives out the Slow even if the Fast is Wrong.

William Kahan in
http://www.cs.berkeley.edu/~wkahan/Stnfrd50.pdf
--
To unsubscribe from this list: send the line "unsubscribe linux-nilfs" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: nilfs_cleanerd from nilfs-utils shutdown on version 2.0 and 2.1 does not fail but says nothing and does not clean the old checkpoints nor newer (actually older) ones.

[Ryusuke Konishi]

Hi,
On Fri, 2 Dec 2011 16:33:09 -0800, Zahid Chowdhury wrote:
> Hello,
>   If I move the system date forward, have some checkpoints created and then move the date backward a 2.0 cleanerd daemon fails on this error:
>     Nov 30 14:39:37 nilfs_cleanerd[5789]: start
>     Nov 30 14:39:38 kernel: nilfs_ioctl_move_inode_block: conflicting data             
>         buffer: ino=4, cno=0, offset=0, blocknr=665655, vblocknr=566462
>     Nov 30 14:39:38 kernel: NILFS: GC failed during preparation: cannot read 
>         source blocks: err=-17
>     Nov 30 14:39:38 nilfs_cleanerd[5789]: cannot clean segments: File exists
>     Nov 30 14:39:38 nilfs_cleanerd[5789]: shutdown
> 
> I cannot ever start up the daemon. If I move to a 2.1 daemon, then it logs no errors, but it cleans no old or newer (really older) checkpoints - it just sits in a do-nothing mode (strace(1) shows he is hung on a mq_timedreceive syscall).

Hmm, this error seems to be caused by a known bug which was already
fixed on nilfs-utils 2.1 with the following patch.

It might be an actual corruption by the kernel code of nilfs2 if you
were using old kernels, but it's most likely due to the bug.

I will backport the fix to nilfs-utils 2.0 series and make another
release of it.

Regards,
Ryusuke Konishi

---
From: Ryusuke Konishi <konishi.ryusuke-Zyj7fXuS5i5L9jVzuh4AOg@public.gmane.org>

nilfs_cleanerd: fix move block errors with cpfile and sufile

This fixes the following gc error related to cpfile and sufile:

 nilfs_ioctl_move_inode_block: conflicting data buffer: ino=4, cno=0,
 offset=0, blocknr=78648, vblocknr=62283

Blocks of cpfile and sufile should be judged live only if they are
latest, and should not depends on the protection period.

Signed-off-by: Ryusuke Konishi <konishi.ryusuke-Zyj7fXuS5i5L9jVzuh4AOg@public.gmane.org>
---
 sbin/cleanerd/cleanerd.c |   10 ++++++++++
 1 files changed, 10 insertions(+), 0 deletions(-)

diff --git a/sbin/cleanerd/cleanerd.c b/sbin/cleanerd/cleanerd.c
index 45a0be0..138a444 100644
--- a/sbin/cleanerd/cleanerd.c
+++ b/sbin/cleanerd/cleanerd.c
@@ -748,6 +748,16 @@ static int nilfs_vdesc_is_live(const struct nilfs_vdesc *vdesc,
 	long low, high, index;
 	int s;
 
+	if (vdesc->vd_cno == 0) {
+		/*
+		 * live/dead judge for sufile and cpfile should not
+		 * depend on protection period and snapshots.  Without
+		 * this check, gc will cause buffer conflict error
+		 * because their checkpoint number is always zero.
+		 */
+		return vdesc->vd_period.p_end == NILFS_CNO_MAX;
+	}
+
 	if (vdesc->vd_period.p_end == NILFS_CNO_MAX ||
 	    vdesc->vd_period.p_end > protect)
 		return 1;
-- 
1.7.7.4

--
To unsubscribe from this list: send the line "unsubscribe linux-nilfs" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: nilfs_cleanerd from nilfs-utils shutdown on version 2.0 and 2.1 does not fail but says nothing and does not clean the old checkpoints nor newer (actually older) ones.

[Ryusuke Konishi]

Hi,
On Sun, 04 Dec 2011 23:56:40 +0900 (JST), Ryusuke Konishi wrote:
> Hi,
> On Fri, 2 Dec 2011 16:33:09 -0800, Zahid Chowdhury wrote:
> > Hello,
> >   If I move the system date forward, have some checkpoints created and then move the date backward a 2.0 cleanerd daemon fails on this error:
> >     Nov 30 14:39:37 nilfs_cleanerd[5789]: start
> >     Nov 30 14:39:38 kernel: nilfs_ioctl_move_inode_block: conflicting data             
> >         buffer: ino=4, cno=0, offset=0, blocknr=665655, vblocknr=566462
> >     Nov 30 14:39:38 kernel: NILFS: GC failed during preparation: cannot read 
> >         source blocks: err=-17
> >     Nov 30 14:39:38 nilfs_cleanerd[5789]: cannot clean segments: File exists
> >     Nov 30 14:39:38 nilfs_cleanerd[5789]: shutdown
> > 
> > I cannot ever start up the daemon. If I move to a 2.1 daemon, then it logs no errors, but it cleans no old or newer (really older) checkpoints - it just sits in a do-nothing mode (strace(1) shows he is hung on a mq_timedreceive syscall).
> 
> Hmm, this error seems to be caused by a known bug which was already
> fixed on nilfs-utils 2.1 with the following patch.
> 
> It might be an actual corruption by the kernel code of nilfs2 if you
> were using old kernels, but it's most likely due to the bug.
> 
> I will backport the fix to nilfs-utils 2.0 series and make another
> release of it.
> 
> Regards,
> Ryusuke Konishi

I've just released nilfs-utils-2.0.24 on nilfs.org.  Please try it.

Thanks,
Ryusuke Konishi
--
To unsubscribe from this list: send the line "unsubscribe linux-nilfs" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

RE: nilfs_cleanerd from nilfs-utils shutdown on version 2.0 and 2.1 does not fail but says nothing and does not clean the old checkpoints nor newer (actually older) ones.

[Zahid Chowdhury]

Hello Ryusuke,
  I have successfully run the nilfs utils 2.1 with a Centos 5.5 kernel with a nilfs module builtin and cleaned up all checkpoints with no issues whatsoever. Thus, no kernel bug is caused even in the old 2.6.18 kernel from the time rewind. I suggest everybody upgrade to nilfs utils 2.1 wherever possible. Thanks everybody for your help.

Zahid

-----Original Message-----
From: Ryusuke Konishi [mailto:konishi.ryusuke-Zyj7fXuS5i5L9jVzuh4AOg@public.gmane.org] 
Sent: Sunday, December 04, 2011 6:57 AM
To: Zahid Chowdhury
Cc: linux-nilfs-u79uwXL29TY76Z2rM5mHXA@public.gmane.org; dexen deVries
Subject: Re: nilfs_cleanerd from nilfs-utils shutdown on version 2.0 and 2.1 does not fail but says nothing and does not clean the old checkpoints nor newer (actually older) ones.

Hi,
On Fri, 2 Dec 2011 16:33:09 -0800, Zahid Chowdhury wrote:
> Hello,
>   If I move the system date forward, have some checkpoints created and then move the date backward a 2.0 cleanerd daemon fails on this error:
>     Nov 30 14:39:37 nilfs_cleanerd[5789]: start
>     Nov 30 14:39:38 kernel: nilfs_ioctl_move_inode_block: conflicting data             
>         buffer: ino=4, cno=0, offset=0, blocknr=665655, vblocknr=566462
>     Nov 30 14:39:38 kernel: NILFS: GC failed during preparation: cannot read 
>         source blocks: err=-17
>     Nov 30 14:39:38 nilfs_cleanerd[5789]: cannot clean segments: File exists
>     Nov 30 14:39:38 nilfs_cleanerd[5789]: shutdown
> 
> I cannot ever start up the daemon. If I move to a 2.1 daemon, then it logs no errors, but it cleans no old or newer (really older) checkpoints - it just sits in a do-nothing mode (strace(1) shows he is hung on a mq_timedreceive syscall).

Hmm, this error seems to be caused by a known bug which was already
fixed on nilfs-utils 2.1 with the following patch.

It might be an actual corruption by the kernel code of nilfs2 if you
were using old kernels, but it's most likely due to the bug.

I will backport the fix to nilfs-utils 2.0 series and make another
release of it.

Regards,
Ryusuke Konishi

---
From: Ryusuke Konishi <konishi.ryusuke-Zyj7fXuS5i5L9jVzuh4AOg@public.gmane.org>

nilfs_cleanerd: fix move block errors with cpfile and sufile

This fixes the following gc error related to cpfile and sufile:

 nilfs_ioctl_move_inode_block: conflicting data buffer: ino=4, cno=0,
 offset=0, blocknr=78648, vblocknr=62283

Blocks of cpfile and sufile should be judged live only if they are
latest, and should not depends on the protection period.

Signed-off-by: Ryusuke Konishi <konishi.ryusuke-Zyj7fXuS5i5L9jVzuh4AOg@public.gmane.org>
---
 sbin/cleanerd/cleanerd.c |   10 ++++++++++
 1 files changed, 10 insertions(+), 0 deletions(-)

diff --git a/sbin/cleanerd/cleanerd.c b/sbin/cleanerd/cleanerd.c
index 45a0be0..138a444 100644
--- a/sbin/cleanerd/cleanerd.c
+++ b/sbin/cleanerd/cleanerd.c
@@ -748,6 +748,16 @@ static int nilfs_vdesc_is_live(const struct nilfs_vdesc *vdesc,
 	long low, high, index;
 	int s;
 
+	if (vdesc->vd_cno == 0) {
+		/*
+		 * live/dead judge for sufile and cpfile should not
+		 * depend on protection period and snapshots.  Without
+		 * this check, gc will cause buffer conflict error
+		 * because their checkpoint number is always zero.
+		 */
+		return vdesc->vd_period.p_end == NILFS_CNO_MAX;
+	}
+
 	if (vdesc->vd_period.p_end == NILFS_CNO_MAX ||
 	    vdesc->vd_period.p_end > protect)
 		return 1;
-- 
1.7.7.4

--
To unsubscribe from this list: send the line "unsubscribe linux-nilfs" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

RE: nilfs_cleanerd from nilfs-utils shutdown on version 2.0 and 2.1 does not fail but says nothing and does not clean the old checkpoints nor newer (actually older) ones.

[Zahid Chowdhury]

Hello,
  On further testing with more iterations with nilfs 2.1 with a only a date rewind (across reboots also) the old dates do not get cleaned up and the new checkpoints with the rewound dates are not cleaned up (my previous testing with 2.1 daemon was on a loopback mount and involved 2 iterations - all testing now was on live systems) - though the 2.1 daemon does not crash it is a do nothing process. A few times we had nilfs_cleanerd 2.1 crashes (reboots fixed that - no crashes, but no checkpoint cleanup after the reboot). Here is the stacktrace from the crash for a 3.0.4 kernel:

Dec  2 13:53:22  kernel: Pid: 1717, comm: nilfs_cleanerd Not tainted 3.0.4 #4
Dec  2 13:53:22  kernel: Call Trace:
Dec  2 13:53:22  kernel:  [<c043e1d0>] ? warn_slowpath_common+0x65/0x7a
Dec  2 13:53:22  kernel:  [<c043e1f9>] ? warn_slowpath_null+0x14/0x18
Dec  2 13:53:22  kernel:  [<f84c000d>] ? nilfs_ioctl_move_blocks+0x11d/0x199 [nilfs2]
Dec  2 13:53:22  kernel:  [<f84c02cd>] ? nilfs_ioctl_clean_segments+0x236/0x2b6 [nilfs2]
Dec  2 13:53:22  kernel:  [<f84bfd23>] ? nilfs_ioctl_get_bdescs+0x68/0x7b [nilfs2]
Dec  2 13:53:22  kernel:  [<f84c06f2>] ? nilfs_ioctl+0x192/0x1bb [nilfs2]
Dec  2 13:53:22  kernel:  [<f84c0560>] ? nilfs_ioctl_set_alloc_range+0x12b/0x12b [nilfs2]
Dec  2 13:53:22  kernel:  [<c04fffdc>] ? vfs_ioctl+0x1e/0x38
Dec  2 13:53:22  kernel:  [<c050061c>] ? do_vfs_ioctl+0x164/0x16b
Dec  2 13:53:22  kernel:  [<c0500668>] ? sys_ioctl+0x45/0x5c
Dec  2 13:53:22  kernel:  [<c07311df>] ? sysenter_do_call+0x12/0x28
Dec  2 13:53:22  kernel:  [<c0720000>] ? ab8500_regulator_probe+0x147/0x1af
Dec  2 13:53:22  kernel: ---[ end trace 34bfcccc859adad2 ]---
Dec  2 13:53:22  kernel: NILFS: GC failed during preparation: cannot read source blocks: err=-17
Dec  2 13:53:22  nilfs_cleanerd[1717]: cannot clean segments: File exists
Dec  2 13:53:22  nilfs_cleanerd[1717]: shutdown
Dec  2 14:06:10  nilfs_cleanerd[15310]: start
Dec  2 14:06:12  kernel: ------------[ cut here ]------------
Dec  2 14:06:12  kernel: WARNING: at fs/nilfs2/ioctl.c:449 nilfs_ioctl_move_blocks+0x11d/0x199 [nilfs2]()
Dec  2 14:06:12  kernel: Pid: 15310, comm: nilfs_cleanerd Tainted: G        W   3.0.4 #4
Dec  2 14:06:12  kernel: Call Trace:
Dec  2 14:06:12  kernel:  [<c043e1d0>] ? warn_slowpath_common+0x65/0x7a
Dec  2 14:06:12  kernel:  [<c043e1f9>] ? warn_slowpath_null+0x14/0x18
Dec  2 14:06:12  kernel:  [<f84c000d>] ? nilfs_ioctl_move_blocks+0x11d/0x199 [nilfs2]
Dec  2 14:06:12  kernel:  [<f84c02cd>] ? nilfs_ioctl_clean_segments+0x236/0x2b6 [nilfs2]
Dec  2 14:06:12  kernel:  [<f84bfd23>] ? nilfs_ioctl_get_bdescs+0x68/0x7b [nilfs2]
Dec  2 14:06:12  kernel:  [<f84c06f2>] ? nilfs_ioctl+0x192/0x1bb [nilfs2]
Dec  2 14:06:12  kernel:  [<f84c0560>] ? nilfs_ioctl_set_alloc_range+0x12b/0x12b [nilfs2]
Dec  2 14:06:12  kernel:  [<c04fffdc>] ? vfs_ioctl+0x1e/0x38
Dec  2 14:06:12  kernel:  [<c050061c>] ? do_vfs_ioctl+0x164/0x16b
Dec  2 14:06:12  kernel:  [<c0500668>] ? sys_ioctl+0x45/0x5c
Dec  2 14:06:12  kernel:  [<c07311df>] ? sysenter_do_call+0x12/0x28
Dec  2 14:06:12  kernel: ---[ end trace 34bfcccc859adad3 ]---
Dec  2 14:06:12  kernel: NILFS: GC failed during preparation: cannot read source blocks: err=-17
Dec  2 14:06:12  nilfs_cleanerd[15310]: cannot clean segments: File exists
Dec  2 14:06:12  nilfs_cleanerd[15310]: shutdown
Dec  2 14:08:11  nilfs_cleanerd[15574]: start
Dec  2 14:08:13  kernel: ------------[ cut here ]------------
Dec  2 14:08:13  kernel: WARNING: at fs/nilfs2/ioctl.c:449 nilfs_ioctl_move_blocks+0x11d/0x199 [nilfs2]()

Note, that moving the date forward from to the most forward checkpoint in the future cleans all checkpoints in both 2.0 & 2.1 daemons.

Zahid

-----Original Message-----
From: linux-nilfs-owner-u79uwXL29TY76Z2rM5mHXA@public.gmane.org [mailto:linux-nilfs-owner-u79uwXL29TY76Z2rM5mHXA@public.gmane.org] On Behalf Of Zahid Chowdhury
Sent: Monday, December 05, 2011 2:16 PM
To: Ryusuke Konishi
Cc: linux-nilfs-u79uwXL29TY76Z2rM5mHXA@public.gmane.org; dexen deVries
Subject: RE: nilfs_cleanerd from nilfs-utils shutdown on version 2.0 and 2.1 does not fail but says nothing and does not clean the old checkpoints nor newer (actually older) ones.

Hello Ryusuke,
  I have successfully run the nilfs utils 2.1 with a Centos 5.5 kernel with a nilfs module builtin and cleaned up all checkpoints with no issues whatsoever. Thus, no kernel bug is caused even in the old 2.6.18 kernel from the time rewind. I suggest everybody upgrade to nilfs utils 2.1 wherever possible. Thanks everybody for your help.

Zahid

-----Original Message-----
From: Ryusuke Konishi [mailto:konishi.ryusuke-Zyj7fXuS5i5L9jVzuh4AOg@public.gmane.org] 
Sent: Sunday, December 04, 2011 6:57 AM
To: Zahid Chowdhury
Cc: linux-nilfs-u79uwXL29TY76Z2rM5mHXA@public.gmane.org; dexen deVries
Subject: Re: nilfs_cleanerd from nilfs-utils shutdown on version 2.0 and 2.1 does not fail but says nothing and does not clean the old checkpoints nor newer (actually older) ones.

Hi,
On Fri, 2 Dec 2011 16:33:09 -0800, Zahid Chowdhury wrote:
> Hello,
>   If I move the system date forward, have some checkpoints created and then move the date backward a 2.0 cleanerd daemon fails on this error:
>     Nov 30 14:39:37 nilfs_cleanerd[5789]: start
>     Nov 30 14:39:38 kernel: nilfs_ioctl_move_inode_block: conflicting data             
>         buffer: ino=4, cno=0, offset=0, blocknr=665655, vblocknr=566462
>     Nov 30 14:39:38 kernel: NILFS: GC failed during preparation: cannot read 
>         source blocks: err=-17
>     Nov 30 14:39:38 nilfs_cleanerd[5789]: cannot clean segments: File exists
>     Nov 30 14:39:38 nilfs_cleanerd[5789]: shutdown
> 
> I cannot ever start up the daemon. If I move to a 2.1 daemon, then it logs no errors, but it cleans no old or newer (really older) checkpoints - it just sits in a do-nothing mode (strace(1) shows he is hung on a mq_timedreceive syscall).

Hmm, this error seems to be caused by a known bug which was already
fixed on nilfs-utils 2.1 with the following patch.

It might be an actual corruption by the kernel code of nilfs2 if you
were using old kernels, but it's most likely due to the bug.

I will backport the fix to nilfs-utils 2.0 series and make another
release of it.

Regards,
Ryusuke Konishi

---
From: Ryusuke Konishi <konishi.ryusuke-Zyj7fXuS5i5L9jVzuh4AOg@public.gmane.org>

nilfs_cleanerd: fix move block errors with cpfile and sufile

This fixes the following gc error related to cpfile and sufile:

 nilfs_ioctl_move_inode_block: conflicting data buffer: ino=4, cno=0,
 offset=0, blocknr=78648, vblocknr=62283

Blocks of cpfile and sufile should be judged live only if they are
latest, and should not depends on the protection period.

Signed-off-by: Ryusuke Konishi <konishi.ryusuke-Zyj7fXuS5i5L9jVzuh4AOg@public.gmane.org>
---
 sbin/cleanerd/cleanerd.c |   10 ++++++++++
 1 files changed, 10 insertions(+), 0 deletions(-)

diff --git a/sbin/cleanerd/cleanerd.c b/sbin/cleanerd/cleanerd.c
index 45a0be0..138a444 100644
--- a/sbin/cleanerd/cleanerd.c
+++ b/sbin/cleanerd/cleanerd.c
@@ -748,6 +748,16 @@ static int nilfs_vdesc_is_live(const struct nilfs_vdesc *vdesc,
 	long low, high, index;
 	int s;
 
+	if (vdesc->vd_cno == 0) {
+		/*
+		 * live/dead judge for sufile and cpfile should not
+		 * depend on protection period and snapshots.  Without
+		 * this check, gc will cause buffer conflict error
+		 * because their checkpoint number is always zero.
+		 */
+		return vdesc->vd_period.p_end == NILFS_CNO_MAX;
+	}
+
 	if (vdesc->vd_period.p_end == NILFS_CNO_MAX ||
 	    vdesc->vd_period.p_end > protect)
 		return 1;
-- 
1.7.7.4

--
To unsubscribe from this list: send the line "unsubscribe linux-nilfs" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
--
To unsubscribe from this list: send the line "unsubscribe linux-nilfs" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: nilfs_cleanerd from nilfs-utils shutdown on version 2.0 and 2.1 does not fail but says nothing and does not clean the old checkpoints nor newer (actually older) ones.

[Ryusuke Konishi]

On Wed, 7 Dec 2011 17:43:13 -0800, Zahid Chowdhury wrote:
> Hello,
>   On further testing with more iterations with nilfs 2.1 with a only
>   a date rewind (across reboots also) the old dates do not get
>   cleaned up and the new checkpoints with the rewound dates are not
>   cleaned up (my previous testing with 2.1 daemon was on a loopback
>   mount and involved 2 iterations - all testing now was on live
>   systems) - though the 2.1 daemon does not crash it is a do nothing
>   process. A few times we had nilfs_cleanerd 2.1 crashes (reboots
>   fixed that - no crashes, but no checkpoint cleanup after the
>   reboot). Here is the stacktrace from the crash for a 3.0.4 kernel:
> 
> Dec  2 13:53:22  kernel: Pid: 1717, comm: nilfs_cleanerd Not tainted 3.0.4 #4
> Dec  2 13:53:22  kernel: Call Trace:
> Dec  2 13:53:22  kernel:  [<c043e1d0>] ? warn_slowpath_common+0x65/0x7a
> Dec  2 13:53:22  kernel:  [<c043e1f9>] ? warn_slowpath_null+0x14/0x18
> Dec  2 13:53:22  kernel:  [<f84c000d>] ? nilfs_ioctl_move_blocks+0x11d/0x199 [nilfs2]
> Dec  2 13:53:22  kernel:  [<f84c02cd>] ? nilfs_ioctl_clean_segments+0x236/0x2b6 [nilfs2]
> Dec  2 13:53:22  kernel:  [<f84bfd23>] ? nilfs_ioctl_get_bdescs+0x68/0x7b [nilfs2]
> Dec  2 13:53:22  kernel:  [<f84c06f2>] ? nilfs_ioctl+0x192/0x1bb [nilfs2]
> Dec  2 13:53:22  kernel:  [<f84c0560>] ? nilfs_ioctl_set_alloc_range+0x12b/0x12b [nilfs2]
> Dec  2 13:53:22  kernel:  [<c04fffdc>] ? vfs_ioctl+0x1e/0x38
> Dec  2 13:53:22  kernel:  [<c050061c>] ? do_vfs_ioctl+0x164/0x16b
> Dec  2 13:53:22  kernel:  [<c0500668>] ? sys_ioctl+0x45/0x5c
> Dec  2 13:53:22  kernel:  [<c07311df>] ? sysenter_do_call+0x12/0x28
> Dec  2 13:53:22  kernel:  [<c0720000>] ? ab8500_regulator_probe+0x147/0x1af
> Dec  2 13:53:22  kernel: ---[ end trace 34bfcccc859adad2 ]---
> Dec  2 13:53:22  kernel: NILFS: GC failed during preparation: cannot read source blocks: err=-17
> Dec  2 13:53:22  nilfs_cleanerd[1717]: cannot clean segments: File exists
> Dec  2 13:53:22  nilfs_cleanerd[1717]: shutdown
> Dec  2 14:06:10  nilfs_cleanerd[15310]: start
> Dec  2 14:06:12  kernel: ------------[ cut here ]------------
> Dec  2 14:06:12  kernel: WARNING: at fs/nilfs2/ioctl.c:449 nilfs_ioctl_move_blocks+0x11d/0x199 [nilfs2]()
> Dec  2 14:06:12  kernel: Pid: 15310, comm: nilfs_cleanerd Tainted: G        W   3.0.4 #4
> Dec  2 14:06:12  kernel: Call Trace:
> Dec  2 14:06:12  kernel:  [<c043e1d0>] ? warn_slowpath_common+0x65/0x7a
> Dec  2 14:06:12  kernel:  [<c043e1f9>] ? warn_slowpath_null+0x14/0x18
> Dec  2 14:06:12  kernel:  [<f84c000d>] ? nilfs_ioctl_move_blocks+0x11d/0x199 [nilfs2]
> Dec  2 14:06:12  kernel:  [<f84c02cd>] ? nilfs_ioctl_clean_segments+0x236/0x2b6 [nilfs2]
> Dec  2 14:06:12  kernel:  [<f84bfd23>] ? nilfs_ioctl_get_bdescs+0x68/0x7b [nilfs2]
> Dec  2 14:06:12  kernel:  [<f84c06f2>] ? nilfs_ioctl+0x192/0x1bb [nilfs2]
> Dec  2 14:06:12  kernel:  [<f84c0560>] ? nilfs_ioctl_set_alloc_range+0x12b/0x12b [nilfs2]
> Dec  2 14:06:12  kernel:  [<c04fffdc>] ? vfs_ioctl+0x1e/0x38
> Dec  2 14:06:12  kernel:  [<c050061c>] ? do_vfs_ioctl+0x164/0x16b
> Dec  2 14:06:12  kernel:  [<c0500668>] ? sys_ioctl+0x45/0x5c
> Dec  2 14:06:12  kernel:  [<c07311df>] ? sysenter_do_call+0x12/0x28
> Dec  2 14:06:12  kernel: ---[ end trace 34bfcccc859adad3 ]---
> Dec  2 14:06:12  kernel: NILFS: GC failed during preparation: cannot read source blocks: err=-17
> Dec  2 14:06:12  nilfs_cleanerd[15310]: cannot clean segments: File exists
> Dec  2 14:06:12  nilfs_cleanerd[15310]: shutdown
> Dec  2 14:08:11  nilfs_cleanerd[15574]: start
> Dec  2 14:08:13  kernel: ------------[ cut here ]------------
> Dec  2 14:08:13  kernel: WARNING: at fs/nilfs2/ioctl.c:449 nilfs_ioctl_move_blocks+0x11d/0x199 [nilfs2]()
> 
> Note, that moving the date forward from to the most forward
> checkpoint in the future cleans all checkpoints in both 2.0 & 2.1
> daemons.
> 
> Zahid

Thank you for the test.

"Dec 2" lines are rewound dates, right ? 
So you mean the cleaner crash was different problem ?

Regards,
Ryusuke Konishi
--
To unsubscribe from this list: send the line "unsubscribe linux-nilfs" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

RE: nilfs_cleanerd from nilfs-utils shutdown on version 2.0 and 2.1 does not fail but says nothing and does not clean the old checkpoints nor newer (actually older) ones.

[Zahid Chowdhury]

Hello Ryusuke,
  Yes, the Dec. 2 dates are the rewound dates - no cleanup happens of any of the checkpoints created on Dec. 2 after this (nor the Dec. 8 ones) - thus the disk will just fill-up unless you end up reaching Dec. 8 before the fill-up. One time on testing with the 2.1 gc daemon we got crashes on each startup (i.e. the stacktrace below), none of the other times did get a 2.1 crash - 3 - 4 times we tried - i.e. the cleaner crash was a different problem, cleared up on a reboot. Thanks, any help is much appreciated.

Zahid

-----Original Message-----
From: Ryusuke Konishi [mailto:konishi.ryusuke-Zyj7fXuS5i5L9jVzuh4AOg@public.gmane.org] 
Sent: Thursday, December 08, 2011 8:22 PM
To: Zahid Chowdhury
Cc: linux-nilfs-u79uwXL29TY76Z2rM5mHXA@public.gmane.org; dexen.devries-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org
Subject: Re: nilfs_cleanerd from nilfs-utils shutdown on version 2.0 and 2.1 does not fail but says nothing and does not clean the old checkpoints nor newer (actually older) ones.

On Wed, 7 Dec 2011 17:43:13 -0800, Zahid Chowdhury wrote:
> Hello,
>   On further testing with more iterations with nilfs 2.1 with a only
>   a date rewind (across reboots also) the old dates do not get
>   cleaned up and the new checkpoints with the rewound dates are not
>   cleaned up (my previous testing with 2.1 daemon was on a loopback
>   mount and involved 2 iterations - all testing now was on live
>   systems) - though the 2.1 daemon does not crash it is a do nothing
>   process. A few times we had nilfs_cleanerd 2.1 crashes (reboots
>   fixed that - no crashes, but no checkpoint cleanup after the
>   reboot). Here is the stacktrace from the crash for a 3.0.4 kernel:
> 
> Dec  2 13:53:22  kernel: Pid: 1717, comm: nilfs_cleanerd Not tainted 3.0.4 #4
> Dec  2 13:53:22  kernel: Call Trace:
> Dec  2 13:53:22  kernel:  [<c043e1d0>] ? warn_slowpath_common+0x65/0x7a
> Dec  2 13:53:22  kernel:  [<c043e1f9>] ? warn_slowpath_null+0x14/0x18
> Dec  2 13:53:22  kernel:  [<f84c000d>] ? nilfs_ioctl_move_blocks+0x11d/0x199 [nilfs2]
> Dec  2 13:53:22  kernel:  [<f84c02cd>] ? nilfs_ioctl_clean_segments+0x236/0x2b6 [nilfs2]
> Dec  2 13:53:22  kernel:  [<f84bfd23>] ? nilfs_ioctl_get_bdescs+0x68/0x7b [nilfs2]
> Dec  2 13:53:22  kernel:  [<f84c06f2>] ? nilfs_ioctl+0x192/0x1bb [nilfs2]
> Dec  2 13:53:22  kernel:  [<f84c0560>] ? nilfs_ioctl_set_alloc_range+0x12b/0x12b [nilfs2]
> Dec  2 13:53:22  kernel:  [<c04fffdc>] ? vfs_ioctl+0x1e/0x38
> Dec  2 13:53:22  kernel:  [<c050061c>] ? do_vfs_ioctl+0x164/0x16b
> Dec  2 13:53:22  kernel:  [<c0500668>] ? sys_ioctl+0x45/0x5c
> Dec  2 13:53:22  kernel:  [<c07311df>] ? sysenter_do_call+0x12/0x28
> Dec  2 13:53:22  kernel:  [<c0720000>] ? ab8500_regulator_probe+0x147/0x1af
> Dec  2 13:53:22  kernel: ---[ end trace 34bfcccc859adad2 ]---
> Dec  2 13:53:22  kernel: NILFS: GC failed during preparation: cannot read source blocks: err=-17
> Dec  2 13:53:22  nilfs_cleanerd[1717]: cannot clean segments: File exists
> Dec  2 13:53:22  nilfs_cleanerd[1717]: shutdown
> Dec  2 14:06:10  nilfs_cleanerd[15310]: start
> Dec  2 14:06:12  kernel: ------------[ cut here ]------------
> Dec  2 14:06:12  kernel: WARNING: at fs/nilfs2/ioctl.c:449 nilfs_ioctl_move_blocks+0x11d/0x199 [nilfs2]()
> Dec  2 14:06:12  kernel: Pid: 15310, comm: nilfs_cleanerd Tainted: G        W   3.0.4 #4
> Dec  2 14:06:12  kernel: Call Trace:
> Dec  2 14:06:12  kernel:  [<c043e1d0>] ? warn_slowpath_common+0x65/0x7a
> Dec  2 14:06:12  kernel:  [<c043e1f9>] ? warn_slowpath_null+0x14/0x18
> Dec  2 14:06:12  kernel:  [<f84c000d>] ? nilfs_ioctl_move_blocks+0x11d/0x199 [nilfs2]
> Dec  2 14:06:12  kernel:  [<f84c02cd>] ? nilfs_ioctl_clean_segments+0x236/0x2b6 [nilfs2]
> Dec  2 14:06:12  kernel:  [<f84bfd23>] ? nilfs_ioctl_get_bdescs+0x68/0x7b [nilfs2]
> Dec  2 14:06:12  kernel:  [<f84c06f2>] ? nilfs_ioctl+0x192/0x1bb [nilfs2]
> Dec  2 14:06:12  kernel:  [<f84c0560>] ? nilfs_ioctl_set_alloc_range+0x12b/0x12b [nilfs2]
> Dec  2 14:06:12  kernel:  [<c04fffdc>] ? vfs_ioctl+0x1e/0x38
> Dec  2 14:06:12  kernel:  [<c050061c>] ? do_vfs_ioctl+0x164/0x16b
> Dec  2 14:06:12  kernel:  [<c0500668>] ? sys_ioctl+0x45/0x5c
> Dec  2 14:06:12  kernel:  [<c07311df>] ? sysenter_do_call+0x12/0x28
> Dec  2 14:06:12  kernel: ---[ end trace 34bfcccc859adad3 ]---
> Dec  2 14:06:12  kernel: NILFS: GC failed during preparation: cannot read source blocks: err=-17
> Dec  2 14:06:12  nilfs_cleanerd[15310]: cannot clean segments: File exists
> Dec  2 14:06:12  nilfs_cleanerd[15310]: shutdown
> Dec  2 14:08:11  nilfs_cleanerd[15574]: start
> Dec  2 14:08:13  kernel: ------------[ cut here ]------------
> Dec  2 14:08:13  kernel: WARNING: at fs/nilfs2/ioctl.c:449 nilfs_ioctl_move_blocks+0x11d/0x199 [nilfs2]()
> 
> Note, that moving the date forward from to the most forward
> checkpoint in the future cleans all checkpoints in both 2.0 & 2.1
> daemons.
> 
> Zahid

Thank you for the test.

"Dec 2" lines are rewound dates, right ? 
So you mean the cleaner crash was different problem ?

Regards,
Ryusuke Konishi
--
To unsubscribe from this list: send the line "unsubscribe linux-nilfs" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html