df reports used space gradually growing.

df reports used space gradually growing.

[Eric A]

[-- Attachment #1.1: Type: text/plain, Size: 537 bytes --]

I installed nilfs2 on my 32GB SSD for my rootfs about a week ago and have been very impressed with the improvements in write speed.  However, I just noticed that df reports I am using 25GB when there is only about 13GB of data on the disk.  The 13GB size agrees with du and the size of my daily ext2 backup.  I monitored the situation over night, and the size reported by df appears to be growing by about 1-2GB a day.

Is the size reported by df incorrect?  What will happen when it reports that the entire disk is full?

Eric



      

[-- Attachment #1.2: Type: text/html, Size: 767 bytes --]

[-- Attachment #2: Type: text/plain, Size: 158 bytes --]

_______________________________________________
users mailing list
users-JrjvKiOkagjYtjvyW6yDsg@public.gmane.org
https://www.nilfs.org/mailman/listinfo/users

Re: df reports used space gradually growing.

[Jérôme Poulin]

On Sun, Jul 26, 2009 at 11:18 AM, Eric A<eric225125-/E1597aS9LQAvxtiuMwx3w@public.gmane.org> wrote:
> I installed nilfs2 on my 32GB SSD for my rootfs about a week ago and have
> been very impressed with the improvements in write speed.  However, I just
> noticed that df reports I am using 25GB when there is only about 13GB of
> data on the disk.  The 13GB size agrees with du and the size of my daily
> ext2 backup.  I monitored the situation over night, and the size reported by
> df appears to be growing by about 1-2GB a day.
>
This sounds like the nilfs_cleanerd is not running, could you check if
it appears in ps output?

> Is the size reported by df incorrect?  What will happen when it reports that
> the entire disk is full?
>
Depending on the version of NILFS you're using, you may be in trouble,
else, just starting nilfs_cleanerd manually should be able to free up
all the space.

Re: df reports used space gradually growing.

[Eric A]

[-- Attachment #1.1: Type: text/plain, Size: 1965 bytes --]

You're right; nilfs_cleanered wasn't running.  I've tried rebooting and it still isn't automatically starting.  I'm running 2.6.31-rc4 and nilfs-utils-2.0.14.  The following is the relevant fstab line:

/dev/sda2               /               nilfs2          rw,noatime              0 0
 

I manually started nilfs_cleanered and have been running it for about an hour.  Conky shows disk read/writes of 1MiB  every 2 seconds or so, but I don't see any space being freed up.  In fact, df still reports a gradual filling of the disk.

Eric




________________________________
From: Jérôme Poulin <jeromepoulin-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org>
To: eric225125-/E1597aS9LQAvxtiuMwx3w@public.gmane.org
Cc: NILFS Users mailing list <users-JrjvKiOkagjYtjvyW6yDsg@public.gmane.org>
Sent: Sunday, July 26, 2009 12:05:56 PM
Subject: Re: [NILFS users] df reports used space gradually growing.

On Sun, Jul 26, 2009 at 11:18 AM, Eric A<eric225125@yahoo.com> wrote:
> I installed nilfs2 on my 32GB SSD for my rootfs about a week ago and have
> been very impressed with the improvements in write speed..  However, I just
> noticed that df reports I am using 25GB when there is only about 13GB of
> data on the disk.  The 13GB size agrees with du and the size of my daily
> ext2 backup.  I monitored the situation over night, and the size reported by
> df appears to be growing by about 1-2GB a day.
>
This sounds like the nilfs_cleanerd is not running, could you check if
it appears in ps output?

> Is the size reported by df incorrect?  What will happen when it reports that
> the entire disk is full?
>
Depending on the version of NILFS you're using, you may be in trouble,
else, just starting nilfs_cleanerd manually should be able to free up
all the space.
_______________________________________________
users mailing list
users-JrjvKiOkagjYtjvyW6yDsg@public.gmane.org
https://www.nilfs.org/mailman/listinfo/users



      

[-- Attachment #1.2: Type: text/html, Size: 3315 bytes --]

[-- Attachment #2: Type: text/plain, Size: 158 bytes --]

_______________________________________________
users mailing list
users-JrjvKiOkagjYtjvyW6yDsg@public.gmane.org
https://www.nilfs.org/mailman/listinfo/users

Re: df reports used space gradually growing.

[Ryusuke Konishi]

Hi,
On Sun, 26 Jul 2009 10:40:08 -0700 (PDT), Eric A wrote:
> You're right; nilfs_cleanered wasn't running.  I've tried rebooting
> and it still isn't automatically starting.  I'm running 2.6.31-rc4
> and nilfs-utils-2.0.14.  The following is the relevant fstab line:
> 
> /dev/sda2               /               nilfs2          rw,noatime              0 0

Oh, I thought the problem would be fixed with nilfs-utils-2.0.14.

How does the relevant line in /proc/mounts look like?

I think you can find the error which terminated cleanerd in
/var/log/syslog or /var/log/messages.

> I manually started nilfs_cleanered and have been running it for
>about an hour.  Conky shows disk read/writes of 1MiB every 2 seconds
>or so, but I don't see any space being freed up.  In fact, df still
>reports a gradual filling of the disk.  Eric

Is is the same thing if you shorten the protection period or release
snapshots ?

Thanks,
Ryusuke Konishi

> ________________________________
> From: Jérôme Poulin <jeromepoulin-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org>
> To: eric225125-/E1597aS9LQAvxtiuMwx3w@public.gmane.org
> Cc: NILFS Users mailing list <users-JrjvKiOkagjYtjvyW6yDsg@public.gmane.org>
> Sent: Sunday, July 26, 2009 12:05:56 PM
> Subject: Re: [NILFS users] df reports used space gradually growing.
> 
> On Sun, Jul 26, 2009 at 11:18 AM, Eric A<eric225125-/E1597aS9LQAvxtiuMwx3w@public.gmane.org> wrote:
> > I installed nilfs2 on my 32GB SSD for my rootfs about a week ago and have
> > been very impressed with the improvements in write speed..  However, I just
> > noticed that df reports I am using 25GB when there is only about 13GB of
> > data on the disk.  The 13GB size agrees with du and the size of my daily
> > ext2 backup.  I monitored the situation over night, and the size reported by
> > df appears to be growing by about 1-2GB a day.
> >
> This sounds like the nilfs_cleanerd is not running, could you check if
> it appears in ps output?
> 
> > Is the size reported by df incorrect?  What will happen when it reports that
> > the entire disk is full?
> >
> Depending on the version of NILFS you're using, you may be in trouble,
> else, just starting nilfs_cleanerd manually should be able to free up
> all the space.
> _______________________________________________
> users mailing list
> users-JrjvKiOkagjYtjvyW6yDsg@public.gmane.org
> https://www.nilfs.org/mailman/listinfo/users

Re: df reports used space gradually growing.

[Eric A]

[-- Attachment #1.1: Type: text/plain, Size: 1971 bytes --]

/proc/mounts:
rootfs / rootfs rw 0 0                                                                                                        
/dev/root / nilfs2 rw,noatime 0 0

The only thing I see in syslog related to nilfs:
Jul 26 12:23:50 gentoo klogd: VFS: Mounted root (nilfs2 filesystem) readonly on device 8:2.
Jul 26 12:31:44 gentoo nilfs_cleanerd[2690]: start
Jul 26 12:31:44 gentoo nilfs_cleanerd[2690]: override protection period to 0

Note that the nilfs_cleanerd message is from when I started it manually (with -p0).

Using lscp, I see no snapshots, but several thousand checkpoints.  I tried deleting the checkpoints with rmcp, but that didn't free up any space.  The disk is still slowly filling.

Eric




________________________________
From: Ryusuke Konishi <ryusuke-sG5X7nlA6pw@public.gmane.org>
To: users-JrjvKiOkagjYtjvyW6yDsg@public.gmane.org; eric225125-/E1597aS9LQAvxtiuMwx3w@public.gmane.org
Sent: Sunday, July 26, 2009 2:57:44 PM
Subject: Re: [NILFS users] df reports used space gradually growing.

Hi,
On Sun, 26 Jul 2009 10:40:08 -0700 (PDT), Eric A wrote:
> You're right; nilfs_cleanered wasn't running.  I've tried rebooting
> and it still isn't automatically starting.  I'm running 2.6.31-rc4
> and nilfs-utils-2.0.14.  The following is the relevant fstab line:
> 
> /dev/sda2               /               nilfs2          rw,noatime              0 0

Oh, I thought the problem would be fixed with nilfs-utils-2.0.14.

How does the relevant line in /proc/mounts look like?

I think you can find the error which terminated cleanerd in
/var/log/syslog or /var/log/messages.

> I manually started nilfs_cleanered and have been running it for
>about an hour.  Conky shows disk read/writes of 1MiB every 2 seconds
>or so, but I don't see any space being freed up.  In fact, df still
>reports a gradual filling of the disk.  Eric

Is is the same thing if you shorten the protection period or release
snapshots ?

Thanks,
Ryusuke Konishi


      

[-- Attachment #1.2: Type: text/html, Size: 3399 bytes --]

[-- Attachment #2: Type: text/plain, Size: 158 bytes --]

_______________________________________________
users mailing list
users-JrjvKiOkagjYtjvyW6yDsg@public.gmane.org
https://www.nilfs.org/mailman/listinfo/users

Re: df reports used space gradually growing.

[Jiro SEKIBA]

Hi,

> Using lscp, I see no snapshots, but several thousand checkpoints.  I tried
> deleting the checkpoints with rmcp, but that didn't free up any space.  The
> disk is still slowly filling.

Unfortunately that's the way rmcp works for now.
rmcp only tells the cleanerd the checkpoint is ready to remove even
in the protection period and nilfs_cleanerd is the only process that
collect garage from the storage and create free space for new logs.

thanks,

regards,
-- 
Jiro SEKIBA <jir-hfpbi5WX9J54Eiagz67IpQ@public.gmane.org>

Re: df reports used space gradually growing.

[Eric A]

[-- Attachment #1.1: Type: text/plain, Size: 1367 bytes --]

Ok, thanks that makes sense.  Well now that nilfs_cleanerd has been running for about 9 hours, it is actually starting to free up some space!  It's freed up about 1GB so far of the 12GB that should be freed.  So, I suppose it is working correctly, just rather slowly.  So, the only real problem is that it is not being started on mount, but for now I can just manually start it. 

Eric




________________________________
From: Jiro SEKIBA <jir-hfpbi5WX9J54Eiagz67IpQ@public.gmane.org>
To: NILFS Users mailing list <users-JrjvKiOkagjYtjvyW6yDsg@public.gmane.org>
Sent: Sunday, July 26, 2009 8:35:42 PM
Subject: Re: [NILFS users] df reports used space gradually growing.

Hi,

> Using lscp, I see no snapshots, but several thousand checkpoints.  I tried
> deleting the checkpoints with rmcp, but that didn't free up any space.  The
> disk is still slowly filling.

Unfortunately that's the way rmcp works for now.
rmcp only tells the cleanerd the checkpoint is ready to remove even
in the protection period and nilfs_cleanerd is the only process that
collect garage from the storage and create free space for new logs.

thanks,

regards,
-- 
Jiro SEKIBA <jir-hfpbi5WX9J54Eiagz67IpQ@public.gmane.org>
_______________________________________________
users mailing list
users-JrjvKiOkagjYtjvyW6yDsg@public.gmane.org
https://www.nilfs.org/mailman/listinfo/users



      

[-- Attachment #1.2: Type: text/html, Size: 2406 bytes --]

[-- Attachment #2: Type: text/plain, Size: 158 bytes --]

_______________________________________________
users mailing list
users-JrjvKiOkagjYtjvyW6yDsg@public.gmane.org
https://www.nilfs.org/mailman/listinfo/users

Re: df reports used space gradually growing.

[Ryusuke Konishi]

Hi,
On Sun, 26 Jul 2009 19:00:09 -0700 (PDT), Eric A wrote:
> Ok, thanks that makes sense.  Well now that nilfs_cleanerd has been
> running for about 9 hours, it is actually starting to free up some
> space!  It's freed up about 1GB so far of the 12GB that should be
> freed.  So, I suppose it is working correctly, just rather slowly.

Well, this should get fixed in the course of improvement of the
garbage collector.

> So, the only real problem is that it is not being started on mount,
> but for now I can just manually start it.
>
> Eric

Yes, I'd like to know why cleanerd fails or is not invoked in your
(gentoo?) boot environment.

> /proc/mounts:
> rootfs / rootfs rw 0 0
> /dev/root / nilfs2 rw,noatime 0 0

looks good to me.

> The only thing I see in syslog related to nilfs:
> Jul 26 12:23:50 gentoo klogd: VFS: Mounted root (nilfs2 filesystem) readonly on device 8:2.
> Jul 26 12:31:44 gentoo nilfs_cleanerd[2690]: start
> Jul 26 12:31:44 gentoo nilfs_cleanerd[2690]: override protection period to 0

After nilfs was mounted readonly as above, it must have been remounted
read-write, and normally cleanerd should be invoked at the time.

Could you see the following points to be checked ?

 * if the symbolic link /dev/root points to an existent device /dev/sdxx
   (or /dev/hdyy) or not.
 * if the root entry is written in /etc/fstab as below:

  /dev/sdxx     /    nilfs2   defaults            0    0       

 * if mount.nilfs2 is placed in /sbin near ``/sbin/mount''.
 * /proc was mounted before the root nilfs is remounted read-write.

Thanks,
Ryusuke Konishi

> ________________________________
> From: Jiro SEKIBA <jir-hfpbi5WX9J54Eiagz67IpQ@public.gmane.org>
> To: NILFS Users mailing list <users-JrjvKiOkagjYtjvyW6yDsg@public.gmane.org>
> Sent: Sunday, July 26, 2009 8:35:42 PM
> Subject: Re: [NILFS users] df reports used space gradually growing.
> 
> Hi,
> 
> > Using lscp, I see no snapshots, but several thousand checkpoints.  I tried
> > deleting the checkpoints with rmcp, but that didn't free up any space.  The
> > disk is still slowly filling.
> 
> Unfortunately that's the way rmcp works for now.
> rmcp only tells the cleanerd the checkpoint is ready to remove even
> in the protection period and nilfs_cleanerd is the only process that
> collect garage from the storage and create free space for new logs.
> 
> thanks,
> 
> regards,
> -- 
> Jiro SEKIBA <jir-hfpbi5WX9J54Eiagz67IpQ@public.gmane.org>
> _______________________________________________
> users mailing list
> users-JrjvKiOkagjYtjvyW6yDsg@public.gmane.org
> https://www.nilfs.org/mailman/listinfo/users