Re: storage considerations

Re: storage considerations

[Daniel Lezcano]

Dietmar Maurer wrote:
> Hi Daniel,
>
> I think we should have several options for the root storage:
>
> 1.) simply use the host filesystem (like Openvz)
>
> 	- special quota support is needed (simfs?)
>
> 	- quota support depends on ext3 fs, so this only works for
> 	  local attached storage (does not work on NFS)
>
> 	- LVM snapshots are slow, because the snapshot includes all
> container
> 	  on that filesystem.
>
> 	- it is easy to expand/shrink storage
>
> 	- best storage consolidation (storage overcommit possible)
>   


> 2.) loop mounting an image (dm-loop)
>
> 	- quota works
>
> 	- works on any filesystem (even NFS)
>
> 	- don't know how to make snapshots?
>   
Perhaps we can use the future union mount or unionfs in order to make 
incremental snapshot ?
> 	- difficult to resize (but manageable)
>
> 	- slow
>
> 3.) use a block device
>
> 	- works great with LVM2 volumes 
>
> 	- quota works
>
> 	- best performance
>
> 	- works with shared block devices (incredible fast migrations)
>
> 	- snapshots works (at least when using an LVM2 volume)
>
> 	- it is difficult to resize block devices (but manageable)
>
> The question is: when do we mount the root block device?
>
> OpenVZ has one additional state, called 'mounted'. For example you can
> do:
>
> # vzctl mount VMID
> # vzctl umount VMID
>
> Or we simply mount the root when we create the container?
>
> What do you think?
>   
It is very interesting. Thanks for giving an overview of the available 
storage solutions.
You have a lot of idea and I need some time to compile them :)

At present lxc provides a rootfs which is symlink to the directory to 
chroot. It could be nice is to replace this by a directory and bind 
mount something on top of it. By "something" I mean one of the solutions 
you are proposing or something else.
I would prefer to find a generic solution, that is,  a configuration 
format which can be interpreted in a generic way by lxc.
The fstab file format is a good candidate :)
We can:
 1 - replace the fstab file by a new keyword lxc.mount = "a fstab line"
 2 - each time this keyword is found add a mntentry in a mount list of 
the configuration
 3 - automatically add the bind mount of the rootfs in this list when 
the lxc.rootfs is found

 From the point of view of the configuration / setup:

the lxc-create creates the fstab file (do not mount anything otherwise 
that will be lost at reboot).
the lxc-start/lxc-execute mount everything (it is up to the user to 
define the right order in the configuration file).

For example:

lxc.mount = myhost:/linux /linux nfs defaults 0 0
lxc.rootfs = /root/debian4

That will create the fstab file in the container configuration tree:

myhost:/mydistro /root/mydistro nfs defaults 0 0
/root/mydistro  /var/lxc/debian4/rootfs bind 0 0

This fstab can be used with a script for example to unshare the mount 
namespace, bind mount the fstab on top of /etc/fstab and call mount -a :)

In any case, the configuration file stays generic and a new fs support 
can be added without changing the lxc version.

Shall this fit your needs ?

RE: storage considerations

[Dietmar Maurer]

> I would prefer to find a generic solution, that is,  a configuration
> format which can be interpreted in a generic way by lxc.
> The fstab file format is a good candidate :)
> We can:
>  1 - replace the fstab file by a new keyword lxc.mount = "a fstab
line"
>  2 - each time this keyword is found add a mntentry in a mount list of
> the configuration
>  3 - automatically add the bind mount of the rootfs in this list when
> the lxc.rootfs is found
> 
>  From the point of view of the configuration / setup:
> 
> the lxc-create creates the fstab file (do not mount anything otherwise
> that will be lost at reboot).
> the lxc-start/lxc-execute mount everything (it is up to the user to
> define the right order in the configuration file).
> 
> For example:
> 
> lxc.mount = myhost:/linux /linux nfs defaults 0 0
> lxc.rootfs = /root/debian4
> 
> That will create the fstab file in the container configuration tree:
> 
> myhost:/mydistro /root/mydistro nfs defaults 0 0
> /root/mydistro  /var/lxc/debian4/rootfs bind 0 0
> 
> This fstab can be used with a script for example to unshare the mount
> namespace, bind mount the fstab on top of /etc/fstab and call mount -a
> :)
> 
> In any case, the configuration file stays generic and a new fs support
> can be added without changing the lxc version.
> 
> Shall this fit your needs ?

Yes, I guess that would work for me.

- Dietmar

Re: storage considerations

[Serge E. Hallyn]

Quoting Daniel Lezcano (daniel.lezcano-GANU6spQydw@public.gmane.org):
> Dietmar Maurer wrote:
> > Hi Daniel,
> >
> > I think we should have several options for the root storage:
> >
> > 1.) simply use the host filesystem (like Openvz)
> >
> > 	- special quota support is needed (simfs?)
> >
> > 	- quota support depends on ext3 fs, so this only works for
> > 	  local attached storage (does not work on NFS)
> >
> > 	- LVM snapshots are slow, because the snapshot includes all
> > container
> > 	  on that filesystem.

I'm looking forward to the btrfs, tux3, and nilfs snapshotting
capabilities.

-serge

Re: storage considerations

[Daniel Lezcano]

Serge E. Hallyn wrote:
> Quoting Daniel Lezcano (daniel.lezcano-GANU6spQydw@public.gmane.org):
>   
>> Dietmar Maurer wrote:
>>     
>>> Hi Daniel,
>>>
>>> I think we should have several options for the root storage:
>>>
>>> 1.) simply use the host filesystem (like Openvz)
>>>
>>> 	- special quota support is needed (simfs?)
>>>
>>> 	- quota support depends on ext3 fs, so this only works for
>>> 	  local attached storage (does not work on NFS)
>>>
>>> 	- LVM snapshots are slow, because the snapshot includes all
>>> container
>>> 	  on that filesystem.
>>>       
>
> I'm looking forward to the btrfs, tux3, and nilfs snapshotting
> capabilities.
>   
Interesting, especially the nilfs.
The COW file systems is missing in the linux kernel. Do you think the 
btrfs has a chance to go to mainline ?

Re: storage considerations

[Serge E. Hallyn]

Quoting Daniel Lezcano (daniel.lezcano-GANU6spQydw@public.gmane.org):
> Serge E. Hallyn wrote:
>> Quoting Daniel Lezcano (daniel.lezcano-GANU6spQydw@public.gmane.org):
>>   
>>> Dietmar Maurer wrote:
>>>     
>>>> Hi Daniel,
>>>>
>>>> I think we should have several options for the root storage:
>>>>
>>>> 1.) simply use the host filesystem (like Openvz)
>>>>
>>>> 	- special quota support is needed (simfs?)
>>>>
>>>> 	- quota support depends on ext3 fs, so this only works for
>>>> 	  local attached storage (does not work on NFS)
>>>>
>>>> 	- LVM snapshots are slow, because the snapshot includes all
>>>> container
>>>> 	  on that filesystem.
>>>>       
>>
>> I'm looking forward to the btrfs, tux3, and nilfs snapshotting
>> capabilities.
>>   
> Interesting, especially the nilfs.
> The COW file systems is missing in the linux kernel. Do you think the  
> btrfs has a chance to go to mainline ?

It's in Linus' tree at the moment :)  Just needs to stabilize...

-serge

Re: storage considerations

[Oren Laadan]

Daniel Lezcano wrote:
> Serge E. Hallyn wrote:
>> Quoting Daniel Lezcano (daniel.lezcano-GANU6spQydw@public.gmane.org):
>>   
>>> Dietmar Maurer wrote:
>>>     
>>>> Hi Daniel,
>>>>
>>>> I think we should have several options for the root storage:
>>>>
>>>> 1.) simply use the host filesystem (like Openvz)
>>>>
>>>> 	- special quota support is needed (simfs?)
>>>>
>>>> 	- quota support depends on ext3 fs, so this only works for
>>>> 	  local attached storage (does not work on NFS)
>>>>
>>>> 	- LVM snapshots are slow, because the snapshot includes all
>>>> container
>>>> 	  on that filesystem.
>>>>       
>> I'm looking forward to the btrfs, tux3, and nilfs snapshotting
>> capabilities.
>>   
> Interesting, especially the nilfs.
> The COW file systems is missing in the linux kernel. Do you think the 
> btrfs has a chance to go to mainline ?

We've been using NILFS (v1) for 2 years already with Zap, and it works
quite well. We had to make slight (trivial) changes to be able to
control it nicely for c/r purposes.

See our DejaView paper at
	http://www.ncl.cs.columbia.edu/publications/sosp2007_dejaview.pdf

Oren.

RE: storage considerations

[Dietmar Maurer]

> >>> Dietmar Maurer wrote:
> >>>
> >>>> Hi Daniel,
> >>>>
> >>>> I think we should have several options for the root storage:
> >>>>
> >>>> 1.) simply use the host filesystem (like Openvz)
> >>>>
> >>>> 	- special quota support is needed (simfs?)
> >>>>
> >>>> 	- quota support depends on ext3 fs, so this only works for
> >>>> 	  local attached storage (does not work on NFS)
> >>>>
> >>>> 	- LVM snapshots are slow, because the snapshot includes all
> >>>> container
> >>>> 	  on that filesystem.
> >>>>
> >> I'm looking forward to the btrfs, tux3, and nilfs snapshotting
> >> capabilities.
> >>
> > Interesting, especially the nilfs.
> > The COW file systems is missing in the linux kernel. Do you think
the
> > btrfs has a chance to go to mainline ?
> 
> We've been using NILFS (v1) for 2 years already with Zap, and it works
> quite well. We had to make slight (trivial) changes to be able to
> control it nicely for c/r purposes.

NILFS, BTRFS, ... maybe provide solution for snapshots, but quota
support
is still missing (we need quotas on subdirectories)?

And a FS independent solution would be great, so that it works on top of
NFS too. 

So UnionFS looks good, but it does not provide real snapshots (using 
base + incremental change looks clumsy for our purpose). I am also not
sure
if quota works on UnionFS?

- Dietmar

RE: storage considerations

[Dietmar Maurer]

> > 2.) loop mounting an image (dm-loop)
> >
> > 	- quota works
> >
> > 	- works on any filesystem (even NFS)
> >
> > 	- don't know how to make snapshots?
> >
> Perhaps we can use the future union mount or unionfs in order to make
> incremental snapshot ?

Or maybe it is possible to make an LVM snapshot of a dm-loop device? (I
already
asked that on the Linux-lvm mailing list but got no answer).

- Dietmar

RE: [Devel] RE: storage considerations

[Dietmar Maurer]

> And a FS independent solution would be great, so that it works on top
> of NFS too.

Replying to myself. I just noticed that this is not possible, because
NFS is a shared file system. So the only option for NFS would be a loop
mount.

- Dietmar

Re: storage considerations

[Serge E. Hallyn]

Quoting Dietmar Maurer (dietmar-YTcQvvOqK21BDgjK7y7TUQ@public.gmane.org):
> > >>> Dietmar Maurer wrote:
> > >>>
> > >>>> Hi Daniel,
> > >>>>
> > >>>> I think we should have several options for the root storage:
> > >>>>
> > >>>> 1.) simply use the host filesystem (like Openvz)
> > >>>>
> > >>>> 	- special quota support is needed (simfs?)
> > >>>>
> > >>>> 	- quota support depends on ext3 fs, so this only works for
> > >>>> 	  local attached storage (does not work on NFS)
> > >>>>
> > >>>> 	- LVM snapshots are slow, because the snapshot includes all
> > >>>> container
> > >>>> 	  on that filesystem.
> > >>>>
> > >> I'm looking forward to the btrfs, tux3, and nilfs snapshotting
> > >> capabilities.
> > >>
> > > Interesting, especially the nilfs.
> > > The COW file systems is missing in the linux kernel. Do you think
> the
> > > btrfs has a chance to go to mainline ?
> > 
> > We've been using NILFS (v1) for 2 years already with Zap, and it works
> > quite well. We had to make slight (trivial) changes to be able to
> > control it nicely for c/r purposes.
> 
> NILFS, BTRFS, ... maybe provide solution for snapshots, but quota
> support
> is still missing (we need quotas on subdirectories)?

Haven't thought it through 100%, but does it not suffice to be able
to specify quotas per user-namespace?  It won't help with NFS...

> And a FS independent solution would be great, so that it works on top of
> NFS too. 
> 
> So UnionFS looks good, but it does not provide real snapshots (using 
> base + incremental change looks clumsy for our purpose). I am also not
> sure
> if quota works on UnionFS?
> 
> - Dietmar

RE: storage considerations

[Dietmar Maurer]

> > NILFS, BTRFS, ... maybe provide solution for snapshots, but quota
> > support
> > is still missing (we need quotas on subdirectories)?
> 
> Haven't thought it through 100%, but does it not suffice to be able
> to specify quotas per user-namespace?  It won't help with NFS...

I was not aware that this is possible. How do I enable per
user-namespace quotas?

- Dietmar

Re: storage considerations

[Serge E. Hallyn]

Quoting Dietmar Maurer (dietmar-YTcQvvOqK21BDgjK7y7TUQ@public.gmane.org):
> > > NILFS, BTRFS, ... maybe provide solution for snapshots, but quota
> > > support
> > > is still missing (we need quotas on subdirectories)?
> > 
> > Haven't thought it through 100%, but does it not suffice to be able
> > to specify quotas per user-namespace?  It won't help with NFS...
> 
> I was not aware that this is possible. How do I enable per
> user-namespace quotas?

Oh you don't yet  :)  Just thinking ahead.  But I'm not clear on
your requirements - do you want per-uid quotas inside a container,
or just a gross quota for the whole container?

thanks,
-serge

RE: storage considerations

[Dietmar Maurer]

> Oh you don't yet  :)  Just thinking ahead.  But I'm not clear on
> your requirements - do you want per-uid quotas inside a container,
> or just a gross quota for the whole container?

both