Problem report: cannot run nilfs_cleanerd on full filesystem

Problem report: cannot run nilfs_cleanerd on full filesystem

[dexen deVries]

Hi,

It seems on a completely filled filesystem nilfs_cleanerd cannot be started
(nor other utilities that rely on the same libnilfs nilfs_open() function).

This is seriously troubling because removing some files does not free up the space
(per NILFS2 design), so it looks like there is no way to use this filesystem ever again,
at least without manual intervetion into on-disk metadata.


strace shows the following:

blitz!root!/tmp# strace -f /sbin/nilfs_cleanerd /dev/sda1 /mnt/x -p 2
(...lotsa normal output...)
open("/mnt/x/.nilfs", O_RDWR|O_CREAT, 0666) = -1 ENOSPC (No space left on device)
close(4)                                = 0
sendto(3, "<27>Mar 19 17:27:42 nilfs_cleane"..., 97, MSG_NOSIGNAL, NULL, 0) = 97
sendto(3, "<27>Mar 19 17:27:42 nilfs_cleane"..., 102, MSG_NOSIGNAL, NULL, 0) = 102
sendto(3, "<30>Mar 19 17:27:42 nilfs_cleane"..., 50, MSG_NOSIGNAL, NULL, 0) = 50
close(3)                                = 0
exit_group(1)                           = ?
Process 8111 detached


The nilfs_cleanerd exits quietly, without any indication of trouble.
I guess solution ought to be two-prong: open NILFS before daemonize()
and make it possible for nilfs_cleanerd to open filesystem at all when full.


To make it clear: if the nilfs_cleanerd is *running*, it will work *OK*. But
any attempt to start nilfs_cleanerd on full filesystem will fail -- for example
on system boot.


-- 
dexen deVries

``One can't proceed from the informal to the formal by formal means.''
--
To unsubscribe from this list: send the line "unsubscribe linux-nilfs" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Problem report: cannot run nilfs_cleanerd on full filesystem

[Reinoud Zandijk]

On Sat, Mar 19, 2011 at 05:37:59PM +0100, dexen deVries wrote:
> It seems on a completely filled filesystem nilfs_cleanerd cannot be started
> (nor other utilities that rely on the same libnilfs nilfs_open() function).
> 
> This is seriously troubling because removing some files does not free up the space
> (per NILFS2 design), so it looks like there is no way to use this filesystem ever again,
> at least without manual intervetion into on-disk metadata.
> 
> strace shows the following:
> 
> blitz!root!/tmp# strace -f /sbin/nilfs_cleanerd /dev/sda1 /mnt/x -p 2
> (...lotsa normal output...)
> open("/mnt/x/.nilfs", O_RDWR|O_CREAT, 0666) = -1 ENOSPC (No space left on device)
> close(4)                                = 0

I presume that /mnt/x/.nilfs is a node on the mounted FS? Shouldn't that node
be present allways and thus doesn't need to be created? or is it deleted first
and then recreated resulting in an error since the FS is full? ;)

BSD traditionally save some inodes/blocks for the superuser. Maybe some blocks
could be reserved for the nilfs_cleanerd to work with? even if only a few?
Depending on the uid/guid of the program?

just some thougts,
Reinoud

--
To unsubscribe from this list: send the line "unsubscribe linux-nilfs" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Problem report: cannot run nilfs_cleanerd on full filesystem

[dexen deVries]

Hi,

On Saturday 19 of March 2011 23:01:57 you wrote:
> I presume that /mnt/x/.nilfs is a node on the mounted FS? Shouldn't that
> node be present allways and thus doesn't need to be created? or is it
> deleted first and then recreated resulting in an error since the FS is
> full? ;)
> 
> BSD traditionally save some inodes/blocks for the superuser. Maybe some
> blocks could be reserved for the nilfs_cleanerd to work with? even if only
> a few? Depending on the uid/guid of the program?

You are right about the `.nilfs'.

I botched testing; I've removed the .nilfs file by a mistake. *that* caused 
nilfs_cleanerd not to start on a full filesystem -- because it cannot create 
the `.nilfs' file in its root directory.

The problem will thus appear when both:
1) filesystem is full
2) the `.nilfs' file is not present
3) nilfs_cleanerd is re-started (for example reboot).

Not very likely to happen in practice, but not very cool either.

As for reserved blocks, that's an ugly solution IMHO. And so is delete-proofing 
the on-disk `.nilfs' file. But perhaps the `.nilfs' could be made a virtual 
file, maintained by the NILFS2 driver rathre than a plain on-disk object?

Another possibility would be to represent each mounted filesystem somewhere in 
/sys hierarchy, so an read-write filedescriptor could be obtained without use 
of a magic file on the filesystem proper.


Regards,
-- 
dexen deVries

``One can't proceed from the informal to the formal by formal means.''
--
To unsubscribe from this list: send the line "unsubscribe linux-nilfs" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Problem report: cannot run nilfs_cleanerd on full filesystem

[Reinoud Zandijk]

On Sat, Mar 19, 2011 at 11:12:37PM +0100, dexen deVries wrote:
> Hi,
> 
> On Saturday 19 of March 2011 23:01:57 you wrote:
> > BSD traditionally save some inodes/blocks for the superuser. Maybe some
> > blocks could be reserved for the nilfs_cleanerd to work with? even if only
> > a few? Depending on the uid/guid of the program?
> 
> As for reserved blocks, that's an ugly solution IMHO.

It is, but its also a service to `root'. It gives a rooted user the
possibility to use > 100% of reported disc space thus enabling him to clean up
stuff, move stuff around and making more free space. It may not be the best
solution but at least `root' is able to fix problems on a full disc!

> And so is delete-proofing the on-disk `.nilfs' file. But perhaps the
> `.nilfs' could be made a virtual file, maintained by the NILFS2 driver
> rather than a plain on-disk object?  You are right about the `.nilfs'.

The problem with looking up /mnt/ is that it (might) return the node /mnt on
the root FS and not the NiLFS root `/mnt/.' . I don't know how linux handles
such cases but if `/mnt/.' does return the NiLFS root node then why not let
NiLFS use that node instead of the /mnt/.nilfs node?

So IMHO the .nilfs file is a hack in itself :-D

With regards,
Reinoud

--
To unsubscribe from this list: send the line "unsubscribe linux-nilfs" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Problem report: cannot run nilfs_cleanerd on full filesystem

[Ryusuke Konishi]

Hi,
On Sun, 20 Mar 2011 13:50:40 +0100, Reinoud Zandijk wrote:
> On Sat, Mar 19, 2011 at 11:12:37PM +0100, dexen deVries wrote:
> > And so is delete-proofing the on-disk `.nilfs' file. But perhaps the
> > `.nilfs' could be made a virtual file, maintained by the NILFS2 driver
> > rather than a plain on-disk object?  You are right about the `.nilfs'.
> 
> The problem with looking up /mnt/ is that it (might) return the node /mnt on
> the root FS and not the NiLFS root `/mnt/.' . I don't know how linux handles
> such cases but if `/mnt/.' does return the NiLFS root node then why not let
> NiLFS use that node instead of the /mnt/.nilfs node?
> 
> So IMHO the .nilfs file is a hack in itself :-D

The ".nilfs" file is used for two purposes:

 1) As the file on which the nilfs library issues ioctls.

 2) For advisory locks (i.e. fcntl(F_GETLK/F_SETLK/FSETLKW)).  This
    works as a mutex between the cleaner and other nilfs-tools.

Note that the nilfs2 kernel code never uses the .nilfs file; it's just
a regular file for nilfs2.

As for the item (1), we can use the root directory of each filesystem
instead.  And for (2), a Posix semaphore or other lock primitives may
be available.

In Linux, the advisory lock may be applicable against directories
though I haven't tried yet.  If so, we can simply use the filesystem
root directory for both purposes.

IMHO, the .nilfs file should be displaced sooner or later.

Thanks,
Ryusuke Konishi
--
To unsubscribe from this list: send the line "unsubscribe linux-nilfs" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Problem report: cannot run nilfs_cleanerd on full filesystem

[dexen deVries]

Hi,


On Monday 21 of March 2011 12:07:06 you wrote:
> The ".nilfs" file is used for two purposes:
> 
>  1) As the file on which the nilfs library issues ioctls.
> 
>  2) For advisory locks (i.e. fcntl(F_GETLK/F_SETLK/FSETLKW)).  This
>     works as a mutex between the cleaner and other nilfs-tools.
> 
> Note that the nilfs2 kernel code never uses the .nilfs file; it's just
> a regular file for nilfs2.
> 
> As for the item (1), we can use the root directory of each filesystem
> instead.  And for (2), a Posix semaphore or other lock primitives may
> be available.


I tried but failed; it seems Linux doesn't allow opening directory O_RDWR or 
O_WRONLY. And a modyfying ioctl() on directory opened O_RDONLY also returned 
some error, I think it was EBADF. I did not dig any further, so I dunno if 
that's done by kernel's general code or NILFS' ioctl() handler.

I believe nilfs_cleanerd could, on principle, use file descriptor of any file on 
the filesystem in case the user runs into a full fylesystem. Kernel code 
handles that OK, only nilfs_cleanerd would need an extra argument to open a 
different file. What's your oppinion on that?


Regards,
-- 
dexen deVries

[[[↓][→]]]

``In other news, STFU and hack.''
mahmud, in response to Erann Gat's ``How I lost my faith in Lisp''
http://news.ycombinator.com/item?id=2308816
--
To unsubscribe from this list: send the line "unsubscribe linux-nilfs" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Problem report: cannot run nilfs_cleanerd on full filesystem

[Ryusuke Konishi]

On Mon, 21 Mar 2011 13:00:47 +0100, dexen deVries <dexen.devries-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote:
> Hi,
> 
> 
> On Monday 21 of March 2011 12:07:06 you wrote:
> > The ".nilfs" file is used for two purposes:
> > 
> >  1) As the file on which the nilfs library issues ioctls.
> > 
> >  2) For advisory locks (i.e. fcntl(F_GETLK/F_SETLK/FSETLKW)).  This
> >     works as a mutex between the cleaner and other nilfs-tools.
> > 
> > Note that the nilfs2 kernel code never uses the .nilfs file; it's just
> > a regular file for nilfs2.
> > 
> > As for the item (1), we can use the root directory of each filesystem
> > instead.  And for (2), a Posix semaphore or other lock primitives may
> > be available.
> 
> 
> I tried but failed; it seems Linux doesn't allow opening directory O_RDWR or 
> O_WRONLY. And a modyfying ioctl() on directory opened O_RDONLY also returned 
> some error, I think it was EBADF. I did not dig any further, so I dunno if 
> that's done by kernel's general code or NILFS' ioctl() handler.

Actually we can issue ioctls even for modyfying ioctls with the
following patch.

But, the advisory lock fails for write locks (F_WRLCK) since the file
descriptor is opened O_RDONLY, and fcntl returns EBADF in that case.
Cleneard and modifying commands are using this type of lock.

So, we can remove .nilfs if we replace the advisory lock with an
alternative way.

Also, we can make .nilfs immutable on the latest kernel in which nilfs
supported lsattr and chattr:

 # chattr +i .nilfs
 # rm .nilfs
 rm: cannot remove `.nilfs': Operation not permitted

This could be another work around though I don't prefer it.

Thanks,
Ryusuke Konishi
--
From: Ryusuke Konishi <konishi.ryusuke-Zyj7fXuS5i5L9jVzuh4AOg@public.gmane.org>

nilfs-utils: use root directory instead of .nilfs

Signed-off-by: Ryusuke Konishi <konishi.ryusuke-Zyj7fXuS5i5L9jVzuh4AOg@public.gmane.org>
---
 include/nilfs.h |    1 -
 lib/nilfs.c     |   19 ++-----------------
 2 files changed, 2 insertions(+), 18 deletions(-)

diff --git a/include/nilfs.h b/include/nilfs.h
index af2584e..0485648 100644
--- a/include/nilfs.h
+++ b/include/nilfs.h
@@ -86,7 +86,6 @@ typedef __u64 sector_t;	// XXX: __u64 ??
 typedef sector_t nilfs_blkoff_t;
 typedef __u64 nilfs_cno_t;
 
-#define NILFS_IOC	".nilfs"
 #define NILFS_FSTYPE	"nilfs2"
 
 #define NILFS_CNO_MIN	((nilfs_cno_t)1)
diff --git a/lib/nilfs.c b/lib/nilfs.c
index 3ba5571..ff1c689 100644
--- a/lib/nilfs.c
+++ b/lib/nilfs.c
@@ -144,7 +144,6 @@ static int nilfs_find_fs(struct nilfs *nilfs, const char *dev, const char *dir,
 {
 	FILE *fp;
 	char line[LINE_MAX], *mntent[NMNTFLDS];
-	size_t len;
 	int ret, n;
 	char canonical[PATH_MAX + 2];
 	char *cdev = NULL, *cdir = NULL;
@@ -196,16 +195,12 @@ static int nilfs_find_fs(struct nilfs *nilfs, const char *dev, const char *dir,
 			nilfs->n_dev = strdup(mntent[MNTFLD_FS]);
 			if (nilfs->n_dev == NULL)
 				goto failed_proc_mounts;
-			len = strlen(mntent[MNTFLD_DIR]) +
-				strlen(NILFS_IOC) + 2;
-			nilfs->n_ioc = malloc(sizeof(char) * len);
+			nilfs->n_ioc = strdup(mntent[MNTFLD_DIR]);
 			if (nilfs->n_ioc == NULL) {
 				free(nilfs->n_dev);
 				nilfs->n_dev = NULL;
 				goto failed_proc_mounts;
 			}
-			snprintf(nilfs->n_ioc, len, "%s/%s",
-				 mntent[MNTFLD_DIR], NILFS_IOC);
 			ret = 0;
 			break;
 		}
@@ -280,7 +275,6 @@ struct nilfs *nilfs_open(const char *dev, const char *dir, int flags)
 {
 	struct nilfs *nilfs;
 	__u64 features;
-	int oflags;
 
 	if (!(flags & (NILFS_OPEN_RAW | NILFS_OPEN_RDONLY |
 		       NILFS_OPEN_WRONLY | NILFS_OPEN_RDWR))) {
@@ -330,16 +324,7 @@ struct nilfs *nilfs_open(const char *dev, const char *dir, int flags)
 			if (nilfs_find_fs(nilfs, dev, dir, MNTOPT_RO) < 0)
 				goto out_nilfs;
 		}
-		oflags = O_CREAT;
-		if (flags & NILFS_OPEN_RDONLY)
-			oflags |= O_RDONLY;
-		else if (flags & NILFS_OPEN_WRONLY)
-			oflags |= O_WRONLY;
-		else if (flags & NILFS_OPEN_RDWR)
-			oflags |= O_RDWR;
-		nilfs->n_iocfd = open(nilfs->n_ioc, oflags,
-				      S_IRUSR | S_IWUSR | S_IRGRP | S_IWGRP |
-				      S_IROTH | S_IWOTH);
+		nilfs->n_iocfd = open(nilfs->n_ioc, O_RDONLY);
 		if (nilfs->n_iocfd < 0)
 			goto out_fd;
 	}
-- 
1.7.3.5

--
To unsubscribe from this list: send the line "unsubscribe linux-nilfs" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Problem report: cannot run nilfs_cleanerd on full filesystem

[Ryusuke Konishi]

On Tue, 22 Mar 2011 00:07:02 +0900 (JST), Ryusuke Konishi wrote:
> Also, we can make .nilfs immutable on the latest kernel in which nilfs
> supported lsattr and chattr:
> 
>  # chattr +i .nilfs
>  # rm .nilfs
>  rm: cannot remove `.nilfs': Operation not permitted
> 
> This could be another work around though I don't prefer it.

Unfortunately, this turned out to go wrong since immutable file is not
writable. 

Using a Posix semaphore looks shorter now.

Ryusuke Konishi
--
To unsubscribe from this list: send the line "unsubscribe linux-nilfs" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Problem report: cannot run nilfs_cleanerd on full filesystem

[dexen deVries]

Hi again,


On Monday 21 of March 2011 12:07:06 you wrote:
> The ".nilfs" file is used for two purposes:
> 
>  1) As the file on which the nilfs library issues ioctls.
> 
>  2) For advisory locks (i.e. fcntl(F_GETLK/F_SETLK/FSETLKW)).  This
>     works as a mutex between the cleaner and other nilfs-tools.


Regarding 2), it seems multiple instances of nilfs_cleanerd can be started for 
one filesystem (which leads to metadata corruption, as per my earlier 
``Regarding problem with nilfs_cleanerd - part 2 and Nilfs_cleanerd err=-17'' 
mail).

Is that intentional? Should we guard on nilfs_cleanerd level, or on kernel 
level?


Regards,
-- 
dexen deVries

[[[↓][→]]]

``In other news, STFU and hack.''
mahmud, in response to Erann Gat's ``How I lost my faith in Lisp''
http://news.ycombinator.com/item?id=2308816
--
To unsubscribe from this list: send the line "unsubscribe linux-nilfs" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Problem report: cannot run nilfs_cleanerd on full filesystem

[Ryusuke Konishi]

On Mon, 21 Mar 2011 13:05:48 +0100, dexen deVries wrote:
> Hi again,
> 
> 
> On Monday 21 of March 2011 12:07:06 you wrote:
> > The ".nilfs" file is used for two purposes:
> > 
> >  1) As the file on which the nilfs library issues ioctls.
> > 
> >  2) For advisory locks (i.e. fcntl(F_GETLK/F_SETLK/FSETLKW)).  This
> >     works as a mutex between the cleaner and other nilfs-tools.
> 
> 
> Regarding 2), it seems multiple instances of nilfs_cleanerd can be started for 
> one filesystem (which leads to metadata corruption, as per my earlier 
> ``Regarding problem with nilfs_cleanerd - part 2 and Nilfs_cleanerd err=-17'' 
> mail).
> 
> Is that intentional? Should we guard on nilfs_cleanerd level, or on kernel 
> level?

Even if multiple nilfs_cleanerd were invoked on a nilfs volume, they
work exclusively and corruption will not occur.  Of course,
mount.nilfs2 does not invoke multiple cleaners unless user
intentionally do so.

I guess the corruption problem is caused by some sort of
difficult-to-find bug at this time.

Thanks,
Ryusuke Konishi
--
To unsubscribe from this list: send the line "unsubscribe linux-nilfs" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Problem report: cannot run nilfs_cleanerd on full filesystem

[Reinoud Zandijk]

On Mon, Mar 21, 2011 at 08:07:06PM +0900, Ryusuke Konishi wrote:
> > So IMHO the .nilfs file is a hack in itself :-D
> 
> The ".nilfs" file is used for two purposes:
> 
>  1) As the file on which the nilfs library issues ioctls.
> 
>  2) For advisory locks (i.e. fcntl(F_GETLK/F_SETLK/FSETLKW)).  This
>     works as a mutex between the cleaner and other nilfs-tools.
> 
> Note that the nilfs2 kernel code never uses the .nilfs file; it's just
> a regular file for nilfs2.
> 
> As for the item (1), we can use the root directory of each filesystem
> instead.  And for (2), a Posix semaphore or other lock primitives may
> be available.

In NetBSD this is normally resolved by looking up the device the directory is
residing and then opening/locking the device file etc. I think this is a
better way of operating!

It also doesn't allow two cleanerds to be started on one device; otherwise one
could be started on the head and the other on a snapshot, or not?

With regards,
Reinoud

--
To unsubscribe from this list: send the line "unsubscribe linux-nilfs" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Problem report: cannot run nilfs_cleanerd on full filesystem

[Reinoud Zandijk]

On Mon, Mar 21, 2011 at 02:01:51PM +0100, Reinoud Zandijk wrote:
> On Mon, Mar 21, 2011 at 08:07:06PM +0900, Ryusuke Konishi wrote:
> In NetBSD this is normally resolved by looking up the device the directory is
> residing and then opening/locking the device file etc. I think this is a
> better way of operating!

I looked it up for certainty and no, i was wrong; the ones i see are using
system calls or ioctl's on the directory as you have done Ryusuke... so not
that different :)

Cheers,
Reinoud

--
To unsubscribe from this list: send the line "unsubscribe linux-nilfs" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html