NFS Kernel server inside a container

NFS Kernel server inside a container

[Martin Fick]

Hello,

I was wondering if it is possible to run an NFS kernel server inside a linux container?

I tried setting one up on a debian (vserver enabled) kernel, and it seems to start the portmap, rpc.statd, rpc.idmapd rpc.mountd daemons inside the container, but I cannot seem to mount the filesystem from a client.  I do get the following error message on server startup:

  FATAL: Could not load /lib/modules/2.6.32-trunk-vserver-686/modules.dep: No such file or directory


When trying to mount on the client, after blocking on the mount for a while, I get:

  mount.nfs: mount system call failed

Any thoughts?  Has anyone else done this?  Should this be possible in the first place?  Thanks,

-Martin

Re: NFS Kernel server inside a container

[Michael Tokarev]

18.05.2010 22:51, Martin Fick wrote:
> Hello,
>
> I was wondering if it is possible to run an NFS kernel server inside a linux container?
>
> I tried setting one up on a debian (vserver enabled) kernel, and it seems to start the portmap, rpc.statd, rpc.idmapd rpc.mountd daemons inside the container, but I cannot seem to mount the filesystem from a client.  I do get the following error message on server startup:
>
>    FATAL: Could not load /lib/modules/2.6.32-trunk-vserver-686/modules.dep: No such file or directory

Obviously you have to have necessary modules in the
container in order to _load_ them.  But it is simpler
to pre-load all required modules from the host system
instead.

> When trying to mount on the client, after blocking on the mount for a while, I get:
>
>    mount.nfs: mount system call failed

Sure: because nfs services aren't started.

> Any thoughts?  Has anyone else done this?  Should this be possible in the first place?  Thanks,

Actually it does not work even after solving
module issue.  I asked about this very issue
at the end of last year (I think), but no one
answered.  It looks like knfsd threads are
working in "global" (i.e host) namespace, not
in the container namespace, or the said
namespace isn't being set up correctly in
the kernel.  In any way, nfs service needs
to be "containerized" properly before being
useful inside a container, which is not done
currently, it seems.

/mjt

Re: NFS Kernel server inside a container

[Daniel Lezcano]

On 05/18/2010 08:51 PM, Martin Fick wrote:
> Hello,
>
> I was wondering if it is possible to run an NFS kernel server inside a linux container?
>
> I tried setting one up on a debian (vserver enabled) kernel, and it seems to start the portmap, rpc.statd, rpc.idmapd rpc.mountd daemons inside the container, but I cannot seem to mount the filesystem from a client.  I do get the following error message on server startup:
>
>    FATAL: Could not load /lib/modules/2.6.32-trunk-vserver-686/modules.dep: No such file or directory
>
>
> When trying to mount on the client, after blocking on the mount for a while, I get:
>
>    mount.nfs: mount system call failed
>    

It may be possible your network configuration is not correct regarding 
the nfs server access. Can you ping the nfs server from the container ?

> Any thoughts?  Has anyone else done this?  Should this be possible in the first place?  Thanks,
>    

I thought NFS was isolated through the mount namespace.

I have a nfs server on 172.20.0.1 exporting "/home".

On my host (IP 172.20.0.166), I mounted /home via nfs

I created a debian system container with its own rootfs and network. 
Started it. As expected, the nfs mount point is unmounted as it does not 
belong to the rootfs, and then I remounted /home from my container (IP 
172.20.0.42). This mount point is private to the container and not 
accessible from the other containers.

This is what you want to do ? Or did I miss something ?

Thanks
   -- Daniel

Re: NFS Kernel server inside a container

[Martin Fick]

--- On Wed, 5/19/10, Daniel Lezcano <daniel.lezcano-GANU6spQydw@public.gmane.org> wrote:

> > I was wondering if it is possible to run an NFS kernel
> server inside a linux container?
> >
> > I tried setting one up on a debian (vserver enabled)
> kernel, and it seems to start the portmap, rpc.statd,
> rpc.idmapd rpc.mountd daemons inside the container, but I
> cannot seem to mount the filesystem from a client.  I
> do get the following error message on server startup:
> >
> >    FATAL: Could not load
> /lib/modules/2.6.32-trunk-vserver-686/modules.dep: No such
> file or directory
> >
> >
> > When trying to mount on the client, after blocking on
> the mount for a while, I get:
> >
> >    mount.nfs: mount system call failed
> >    
> 
> It may be possible your network configuration is not
> correct regarding 
> the nfs server access. Can you ping the nfs server from the
> container ?
> 
> > Any thoughts?  Has anyone else done this? 
> Should this be possible in the first place?  Thanks,
> >    
> 
> I thought NFS was isolated through the mount namespace.
> 
> I have a nfs server on 172.20.0.1 exporting "/home".
> 
> On my host (IP 172.20.0.166), I mounted /home via nfs
> 
> I created a debian system container with its own rootfs and
> network. 
> Started it. As expected, the nfs mount point is unmounted
> as it does not 
> belong to the rootfs, and then I remounted /home from my
> container (IP 
> 172.20.0.42). This mount point is private to the container
> and not 
> accessible from the other containers.
> 
> This is what you want to do ? Or did I miss something ?

It sounds like you did an NFS client mount inside a
container. I am actually trying to do the reverse, I
would like to do kernel server exports from within
a container.  Specifically, I would like to have
several data partitions replicated with drbd and to
be able to export these partitions via NFS 
independently from different containers with
different IPs.  

So, for example, from 2 hosts, I might have 6 NFS 
partitions to export and during normal operation
I would expect each host to make 3 of the 
partitions primary via drbd and to then to
each launch three containers each with separate 
IPs which will individually export the 3 different 
drbd partitions via NFS.  For failover or load 
balancing, it should then be able to shut down
any individual container on one host and bring it
up on the other host without affecting the other
exports.

Thanks,

-Martin

Re: NFS Kernel server inside a container

[Serge E. Hallyn]

Quoting Martin Fick (mogulguy-/E1597aS9LQAvxtiuMwx3w@public.gmane.org):
> --- On Wed, 5/19/10, Daniel Lezcano <daniel.lezcano-GANU6spQydw@public.gmane.org> wrote:
> It sounds like you did an NFS client mount inside a
> container. I am actually trying to do the reverse, I
> would like to do kernel server exports from within
> a container.  Specifically, I would like to have
> several data partitions replicated with drbd and to
> be able to export these partitions via NFS 
> independently from different containers with
> different IPs.  
> 
> So, for example, from 2 hosts, I might have 6 NFS 
> partitions to export and during normal operation
> I would expect each host to make 3 of the 
> partitions primary via drbd and to then to
> each launch three containers each with separate 
> IPs which will individually export the 3 different 
> drbd partitions via NFS.  For failover or load 
> balancing, it should then be able to shut down
> any individual container on one host and bring it
> up on the other host without affecting the other
> exports.
> 
> Thanks,
> 
> -Martin

You'll need to use a userspace NFS server.  Something like
http://unfs3.sourceforge.net/.  Certainly looks...  venerable.

-serge

Re: NFS Kernel server inside a container

[Daniel Lezcano]

On 05/19/2010 05:26 PM, Martin Fick wrote:
> --- On Wed, 5/19/10, Daniel Lezcano<daniel.lezcano-GANU6spQydw@public.gmane.org>  wrote:
>
>    
>>> I was wondering if it is possible to run an NFS kernel
>>>        
>> server inside a linux container?
>>      
>>> I tried setting one up on a debian (vserver enabled)
>>>        
>> kernel, and it seems to start the portmap, rpc.statd,
>> rpc.idmapd rpc.mountd daemons inside the container, but I
>> cannot seem to mount the filesystem from a client.  I
>> do get the following error message on server startup:
>>      
>>>     FATAL: Could not load
>>>        
>> /lib/modules/2.6.32-trunk-vserver-686/modules.dep: No such
>> file or directory
>>      
>>>
>>> When trying to mount on the client, after blocking on
>>>        
>> the mount for a while, I get:
>>      
>>>     mount.nfs: mount system call failed
>>>
>>>        
>> It may be possible your network configuration is not
>> correct regarding
>> the nfs server access. Can you ping the nfs server from the
>> container ?
>>
>>      
>>> Any thoughts?  Has anyone else done this?
>>>        
>> Should this be possible in the first place?  Thanks,
>>      
>>>
>>>        
>> I thought NFS was isolated through the mount namespace.
>>
>> I have a nfs server on 172.20.0.1 exporting "/home".
>>
>> On my host (IP 172.20.0.166), I mounted /home via nfs
>>
>> I created a debian system container with its own rootfs and
>> network.
>> Started it. As expected, the nfs mount point is unmounted
>> as it does not
>> belong to the rootfs, and then I remounted /home from my
>> container (IP
>> 172.20.0.42). This mount point is private to the container
>> and not
>> accessible from the other containers.
>>
>> This is what you want to do ? Or did I miss something ?
>>      
> It sounds like you did an NFS client mount inside a
> container. I am actually trying to do the reverse, I
> would like to do kernel server exports from within
> a container.  Specifically, I would like to have
> several data partitions replicated with drbd and to
> be able to export these partitions via NFS
> independently from different containers with
> different IPs.
>    
Oh, ok. Right. I misunderstood. As Michael mentioned, that will need 
some kernel work.
I think nobody is working on that but I recall Denis Lunev from, OpenVZ, 
studied it, a couple of years ago, and said that was a big deal.

> So, for example, from 2 hosts, I might have 6 NFS
> partitions to export and during normal operation
> I would expect each host to make 3 of the
> partitions primary via drbd and to then to
> each launch three containers each with separate
> IPs which will individually export the 3 different
> drbd partitions via NFS.  For failover or load
> balancing, it should then be able to shut down
> any individual container on one host and bring it
> up on the other host without affecting the other
> exports.
>    

Sounds a good idea.

Thanks
   -- Daniel

Re: NFS Kernel server inside a container

[Martin Fick]

--- On Tue, 5/18/10, Michael Tokarev <mjt-XAri/EZa3C4vJsYlp49lxw@public.gmane.org> wrote:
> 18.05.2010 22:51, Martin Fick wrote:
> >    FATAL: Could not load
> /lib/modules/2.6.32-trunk-vserver-686/modules.dep: No such
> file or directory
> 
> Obviously you have to have necessary modules in the
> container in order to _load_ them.  But it is simpler
> to pre-load all required modules from the host system
> instead.

Right, I actually ran the kernel server on the host 
too, just to make sure the modules were loaded.

> Actually it does not work even after solving
> module issue.  I asked about this very issue
> at the end of last year (I think), but no one
> answered.  It looks like knfsd threads are
> working in "global" (i.e host) namespace, not
> in the container namespace, or the said
> namespace isn't being set up correctly in
> the kernel.  In any way, nfs service needs
> to be "containerized" properly before being
> useful inside a container, which is not done
> currently, it seems.

Is anyone working on this?  It certainly
would be a very useful feature. :)  If it
is only a matter of setting up the 
container appropriately, any ideas how to
do that? Thanks,

-Martin

Re: NFS Kernel server inside a container

[Eric W. Biederman]

Martin Fick <mogulguy-/E1597aS9LQAvxtiuMwx3w@public.gmane.org> writes:

> --- On Tue, 5/18/10, Michael Tokarev <mjt-XAri/EZa3C4vJsYlp49lxw@public.gmane.org> wrote:
>> 18.05.2010 22:51, Martin Fick wrote:
>> >    FATAL: Could not load
>> /lib/modules/2.6.32-trunk-vserver-686/modules.dep: No such
>> file or directory
>> 
>> Obviously you have to have necessary modules in the
>> container in order to _load_ them.  But it is simpler
>> to pre-load all required modules from the host system
>> instead.
>
> Right, I actually ran the kernel server on the host 
> too, just to make sure the modules were loaded.
>
>> Actually it does not work even after solving
>> module issue.  I asked about this very issue
>> at the end of last year (I think), but no one
>> answered.  It looks like knfsd threads are
>> working in "global" (i.e host) namespace, not
>> in the container namespace, or the said
>> namespace isn't being set up correctly in
>> the kernel.  In any way, nfs service needs
>> to be "containerized" properly before being
>> useful inside a container, which is not done
>> currently, it seems.
>
> Is anyone working on this?  It certainly
> would be a very useful feature. :)  If it
> is only a matter of setting up the 
> container appropriately, any ideas how to
> do that? Thanks,

I remember doing some work a while ago and to ensure it fails
properly.  Getting knfsd updated to work in a network namespace should
be possible, but there are a lot of interactions, and isn't immediately
clear how the struct net would be passed to the functions that need it.

In short it looked doable, but I don't have any immediate plans to make it
happen but I would like to see it happen.

Eric