overlayfs.v8 testing on a Live system

overlayfs.v8 testing on a Live system

[Jordi Pujol]

Hello,

Next I expose some comments about the test of the overlayfs.v8 union filesystem 
driver in the LneT live system,

COMPILE THE KERNEL.

some modifications are needed to compile:

- rename fs/overlayfs/overlayfs.c to fs/overlayfs/module.c

- modify fs/overlayfs/Makefile
#
# Makefile for the overlay filesystem.
#

obj-$(CONFIG_OVERLAYFS_FS) += overlayfs.o

overlayfs-objs := module.o \
	inode.o \
	dir.o \
	readdir.o \
	copy_up.o

- add this line in fs/overlayfs/copy_up.c
 #include <linux/security.h>
+#include <asm/uaccess.h>
 #include "overlayfs.h"

TESTING.

The live system is started in two different modes,
- non persistent mode, upperdir on tmpfs
- persisten mode, upperdir on a disk partition directory, ext4 formatted

Changes to the underlying filesystems while part of a mounted overlay filesystem 
are not allowed.  This is not a problem for Live systems that normally are 
based on static files of squashfs type.

Non-persistent mode: We know that the upperdir can not reside on tmpfs because 
overlayfs requires xattr support on the upper filesystem.
Now trying, it does not work on a live system when the upperdir is on tmpfs.
Has a problem with whiteouts.
mv /root/etc/rc0.d/K01fuse /root/etc/rc0.d/k01fuse
"overlayfs: ERROR - failed to whiteout 'K01fuse'"
Analyzing this operation, looking in the upperdir directories, the new link 
'/root/etc/rc0.d/k01fuse' has been created in the upperdir, but the whiteout 
link '/root/etc/rc0.d/K01fuse' is not created, xattrs can not be created on 
this filesystem type.

Persistent mode: Works on a live system when the upperdir is on disk.
It's a good development, functionally easy and straight, seems very fast, 
although I don't have measured how much,

But some failure has been detected.

DETECTED FAILURE.

Scenario:
- lowerdir is an squashfs mounted on a tmpfs dir.
- upperdir is on an ext4 formatted partition.

An script modifies a file that initially is in lowerdir, issuing consecutive 
commands:

sed -i -r -e 's|\[X-\*-Core\]|&\nServerTimeout=120|' /etc/kde4/kdm/kdmrc
sed -i -e 's/\(ServerArgsLocal\=.*\)$/\1 \-dpi 120/' /etc/kde4/kdm/kdmrc
sed -i -e 's/#\(ServerArgsRemote\=.*\)$/\1 \-dpi 120/' /etc/kde4/kdm/kdmrc

Sometimes first or second changes are lost, only remain the changes done by the 
last commands.

REFERENCES.

release v8:
https://lkml.org/lkml/2011/4/18/247

git:
http://git.kernel.org/?p=linux/kernel/git/mszeredi/vfs.git;a=shortlog;h=refs/heads/overlayfs.v8

LneT custom kernel (Debian source):
http://livenet.selfip.com/ftp/debian/linux-2.6/linux-2.6.38-3.jpp.2-
lnet_2.6.38-14.tar.bz2

IMPROVEMENTS and WHISHLIST.

- upperdir on tmpfs, TODO feature already mentioned in the announcing message.

- xattrs optionally disabled, implement mount option: "nouser_xattr" (option 
name is taken from man mount). Actually this is the default (AFAIK the only) 
behaviour for a Live system, Standard Linux Security is used in the root 
filesystem, it's fast and a desktop system does not need complicated security 
configurations.

- lowerdir may be a list of directories separated by colon (:), they must 
appear in the options string from upper to lower.
That is very interesting, using this feature is easy to create different 
configurations that use the same base Live OS.
example:
mount -t overlayfs upperdir=/upper,lowerdir=/lower1:/lower2:...:/lowerN 
overlayfs /rootdir

- after the previous improvement has been done then we can think about 
changing interactively a Live OS to another configuration, that is done adding 
and removing branches to the union filesystem. Only some Live systems make it, 
others do not have an interactive configuration but require a reboot for 
changing to another configuration,

I hope this review can help,

Jordi Pujol

Live never ending Tale
GNU/Linux Live forever!
http://livenet.selfip.com

Re: overlayfs.v8 testing on a Live system

[Miklos Szeredi]

Jordi Pujol <jordipujolp@gmail.com> writes:

> Hello,
>
> Next I expose some comments about the test of the overlayfs.v8 union
> filesystem driver in the LneT live system,
>
> COMPILE THE KERNEL.
>
> some modifications are needed to compile:
>
> - rename fs/overlayfs/overlayfs.c to fs/overlayfs/module.c
>
> - modify fs/overlayfs/Makefile
> #
> # Makefile for the overlay filesystem.
> #
>
> obj-$(CONFIG_OVERLAYFS_FS) += overlayfs.o
>
> overlayfs-objs := module.o \
> 	inode.o \
> 	dir.o \
> 	readdir.o \
> 	copy_up.o
>
> - add this line in fs/overlayfs/copy_up.c
>  #include <linux/security.h>
> +#include <asm/uaccess.h>
>  #include "overlayfs.h"

Thanks for reporting these.  These are already fixed them in the
latest version of the overlayfs.v8 tree.

> Persistent mode: Works on a live system when the upperdir is on disk.
> It's a good development, functionally easy and straight, seems very fast, 
> although I don't have measured how much,
>
> But some failure has been detected.
>
> DETECTED FAILURE.
>
> Scenario:
> - lowerdir is an squashfs mounted on a tmpfs dir.
> - upperdir is on an ext4 formatted partition.
>
> An script modifies a file that initially is in lowerdir, issuing consecutive 
> commands:
>
> sed -i -r -e 's|\[X-\*-Core\]|&\nServerTimeout=120|' /etc/kde4/kdm/kdmrc
> sed -i -e 's/\(ServerArgsLocal\=.*\)$/\1 \-dpi 120/' /etc/kde4/kdm/kdmrc
> sed -i -e 's/#\(ServerArgsRemote\=.*\)$/\1 \-dpi 120/' /etc/kde4/kdm/kdmrc
>
> Sometimes first or second changes are lost, only remain the changes
> done by the last commands.

Hmm, can you please send a minimal script and necessary files with which
I can reproduce this?

> IMPROVEMENTS and WHISHLIST.
>
> - upperdir on tmpfs, TODO feature already mentioned in the announcing message.
>
> - xattrs optionally disabled, implement mount option: "nouser_xattr" (option 
> name is taken from man mount). Actually this is the default (AFAIK the only) 
> behaviour for a Live system, Standard Linux Security is used in the root 
> filesystem, it's fast and a desktop system does not need complicated security 
> configurations.

You can add nouser_xattr option for the upper filesystem and then things
should work as expected.

> - lowerdir may be a list of directories separated by colon (:), they must 
> appear in the options string from upper to lower.
> That is very interesting, using this feature is easy to create different 
> configurations that use the same base Live OS.
> example:
> mount -t overlayfs upperdir=/upper,lowerdir=/lower1:/lower2:...:/lowerN 
> overlayfs /rootdir

It shoulnd't be too difficult to implement multipe layers, but this is
not a feature I want to implement before the current functinality is
accepted and has had some time to mature.

> - after the previous improvement has been done then we can think about
> changing interactively a Live OS to another configuration, that is
> done adding and removing branches to the union filesystem. Only some
> Live systems make it, others do not have an interactive configuration
> but require a reboot for changing to another configuration,

Okay, see above comment.

Thanks,
Miklos

Re: overlayfs.v8 testing on a Live system

[Jordi Pujol]

[-- Attachment #1: Type: Text/Plain, Size: 453 bytes --]

A Dimecres 20 Abril 2011 15:37:07, vàreu escriure:
> Hmm, can you please send a minimal script and necessary files with which
> I can reproduce this?

Here is attached an script that tries to reproduce that, but the problem does 
not happen, some conditions are not met, I will continue studying what is the 
situation to cause this problem,

Thanks,

Jordi Pujol

Live never ending Tale
GNU/Linux Live forever!
http://livenet.selfip.com

[-- Attachment #2: overlayfs-test.sh --]
[-- Type: application/x-shellscript, Size: 700 bytes --]