Problem with NFS and XEN

Problem with NFS and XEN

[Roger Marcus]

Hi,

I am having a problem with NFS and I thought that you could help me or
you could steer me to
someone who might be able to.

The problem is this:

An NFS Server mounts an image on a local directory, containing a XEN
virtual machine configuration file and a XEN image.

An NFS Client computer sees the directory that was mounted remotely
and starts the XEN Virtual machine on the mounted directory.
The NFS Client computer then stops the XEN Virtual machine. So far, so
good. The machine ran. It stopped. The NFS Client
leaves the directory. No process is accessing that directory from the
client side.

Problem: The NFS Server cannot umount the local directory.
fuser returns no usage of the directory.
lsof returns no usage of the directory.
umount -f won't force umount the directory.

What does work is exportfs -ua.

This is not a good way to go, since it is planned that many clients
will be using the server; indeed many
clients will be using multiple subdirectories on the server.

1. Is this a XEN bug or an NFS bug?
2. Is there some other command like exportfs in which you can just
disable the one local directory but maintain an
active mount on the parent directory?

exports:
/adir =A0=A0=A0 192.168.11.129(rw,sync,no_subtree_check,crossmnt,no_roo=
t_squash)
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D
/adir/image.iso (contains xen vm stuff)
server executes: mount -o loop /adir/image.iso /adir/virtualstuff

client mounts adir with mount -t nfs 192.168.11.22:/adir /adir
xm create /adir/virtualstuff/config.cfg
xm destroy virtual

on the nfs-server:
umount /adir/virtualstuff (won't)

only exportfs -ua works, but this defeats my purpose.

Can you suggest anything?

thanks in advance,

Roger Marcus

Re: Problem with NFS and XEN

[Roger Marcus]

Hi Rajesh/linux-nfs,

I am sorry but I didn't understand your comment, Rajesh. ("you don't
do umount from NFS client of NFS mounted filesystem.")
Does this mean "I should do an umount from an NFS client, or does it
mean I shouldn't do an umount from an NFS client."

In any case I did not nor do I want to umount the directory from the
NFS-client side.

I mounted an iso image on the nfs-server side, so that it would be
visible to the client.
After I use it, I want to umount it from the nfs-server side. It doesn'=
t umount.

(When I try to umount the parent directory from the client - I also
get "device is busy".)

=46rom the server if I type:
exportfs -u 192.168.11.129:/adir              (releasing the parent dir=
ectory).
this fails, and the subsequent umount /adir/virtualstuff        fails

However if I type:
exportfs -ua

then umount /adir/virtualstuff    succeeds.

Is it a bug that
exportfs -u address:/path_to_directory     fails to 'unbusy the directo=
ry',
but
exportfs -ua                                                  succeeds
and a subsequent umount /adir/virtualstuff succeeds.


I am using as a server, opensuse
Linux msi 2.6.27.21-0.1-default #1 SMP 2009-03-31 14:50:44 +0200
x86_64 x86_64 x86_64 GNU/Linux

regards and thanks in advance for your help,

Roger Marcus



On Fri, May 8, 2009 at 9:15 AM, Rajesh Ghanekar
<rajesh_ghanekar@symantec.com> wrote:
> you don't do umount from NFS client of NFS mounted filesystem.
>
> Roger Marcus wrote:
>>
>> Hi,
>>
>> I am having a problem with NFS and I thought that you could help me =
or
>> you could steer me to
>> someone who might be able to.
>>
>> The problem is this:
>>
>> An NFS Server mounts an image on a local directory, containing a XEN
>> virtual machine configuration file and a XEN image.
>>
>> An NFS Client computer sees the directory that was mounted remotely
>> and starts the XEN Virtual machine on the mounted directory.
>> The NFS Client computer then stops the XEN Virtual machine. So far, =
so
>> good. The machine ran. It stopped. The NFS Client
>> leaves the directory. No process is accessing that directory from th=
e
>> client side.
>>
>> Problem: The NFS Server cannot umount the local directory.
>> fuser returns no usage of the directory.
>> lsof returns no usage of the directory.
>> umount -f won't force umount the directory.
>>
>> What does work is exportfs -ua.
>>
>> This is not a good way to go, since it is planned that many clients
>> will be using the server; indeed many
>> clients will be using multiple subdirectories on the server.
>>
>> 1. Is this a XEN bug or an NFS bug?
>> 2. Is there some other command like exportfs in which you can just
>> disable the one local directory but maintain an
>> active mount on the parent directory?
>>
>> exports:
>> /adir =A0 =A0 192.168.11.129(rw,sync,no_subtree_check,crossmnt,no_ro=
ot_squash)
>> =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D
>> /adir/image.iso (contains xen vm stuff)
>> server executes: mount -o loop /adir/image.iso /adir/virtualstuff
>>
>> client mounts adir with mount -t nfs 192.168.11.22:/adir /adir
>> xm create /adir/virtualstuff/config.cfg
>> xm destroy virtual
>>
>> on the nfs-server:
>> umount /adir/virtualstuff (won't)
>>
>> only exportfs -ua works, but this defeats my purpose.
>>
>> Can you suggest anything?
>>
>> thanks in advance,
>>
>> Roger Marcus
>> --
>> To unsubscribe from this list: send the line "unsubscribe linux-nfs"=
 in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at =A0http://vger.kernel.org/majordomo-info.html
>>
>
>
>

Re: Problem with NFS and XEN

[Trond Myklebust]

On Fri, 2009-05-08 at 08:10 +0200, Roger Marcus wrote:
> Hi,
> 
> I am having a problem with NFS and I thought that you could help me or
> you could steer me to
> someone who might be able to.
> 
> The problem is this:
> 
> An NFS Server mounts an image on a local directory, containing a XEN
> virtual machine configuration file and a XEN image.
> 
> An NFS Client computer sees the directory that was mounted remotely
> and starts the XEN Virtual machine on the mounted directory.
> The NFS Client computer then stops the XEN Virtual machine. So far, so
> good. The machine ran. It stopped. The NFS Client
> leaves the directory. No process is accessing that directory from the
> client side.

It is still exported, so the NFS server still has a reference to it.

> Problem: The NFS Server cannot umount the local directory.
> fuser returns no usage of the directory.
> lsof returns no usage of the directory.

correction: lsof and fuser don't show any _user processes_ that
reference the directory. NFSd runs in the kernel...

> umount -f won't force umount the directory.
> 
> What does work is exportfs -ua.

or 'exportfs -u client:/directory'. See 'man exportfs'

> This is not a good way to go, since it is planned that many clients
> will be using the server; indeed many
> clients will be using multiple subdirectories on the server.
> 
> 1. Is this a XEN bug or an NFS bug?
> 2. Is there some other command like exportfs in which you can just
> disable the one local directory but maintain an
> active mount on the parent directory?

See above.

Note that if you want to unexport all export entries for a particular
directory '/mydir', then something like this simple script will do it:

exportfs -u $(exportfs | awk '/^\/mydir/ { print $2 ":" $1; }')

Trond

Re: Problem with NFS and XEN

[Roger Marcus]

Hi Trond,

Thank you for your answer.

exportfs does not report the individual directories that are mounted
on the exported directory, so I cannot use your script.

on the nfsserver exports file:
/adir     192.168.11.129(rw,sync,no_subtree_check,crossmnt,no_root_squash)

on the same machine:
mount -o loop virtual.iso virtualstuff
so now I have /adir/virtualstuff

(which by the way, I can:  umount virtualstuff immediately without
problem and it releases.
however, if I run XEN on the client, then the directory is permanently
locked, ie, umount virtualstuff
DOESN'T work again unless I run exportfs -ua killing all connections
to the server.)

the nfs-client sees /adir/virtualstuff, since parent /adir is mounted.
When I run the virtualmachine within the virtualstuff directory (xm
create /adir/virtualstuff/config.cfg)
and then I destroy the virtualmachine (xm destroy /adir/virtualstuff/config.cfg)
I can never 'umount virtualstuff' from the nfsserver, even though no
one is accessing the directory
any more.

So on the nfsserver I cannot umount virtualstuff. this is the bug.

exportfs -ua    kills nfs for everybody, and then I can run 'umount
virtualstuff', but this is not what I want.
exportfs -u nfsclient:/adir       doesn't even work. I don't want this
variation because I might be running 2 virtual machines
                                                 in two subdirectories.

question: is the fact that ;exportfs -u nfsclient:/adir' doesn't work a bug?

What I want to be able to do is from the nfsserver side, 'umount
virtualstuff', which I can do if I
1) never run XEN, or
2) exportfs -ua. The latter command kills all clients attached to the server
so I cannot use this second command.
3) the command 'exportfs -u nfsclient:/adir' doesn't work and this
might be a bug. I do not
want this variant since if I am running 2 virtual machines on the
nfsclient I would kill the other machine.

In /proc/net/rpc/nfsd.export/content   I see my virtualstuff
subdirectory referenced. Perhaps this
tells you something.

Thank you in advance,

Roger Marcus




On Fri, May 8, 2009 at 2:24 PM, Trond Myklebust
<trond.myklebust@fys.uio.no> wrote:
> On Fri, 2009-05-08 at 08:10 +0200, Roger Marcus wrote:
>> Hi,
>>
>> I am having a problem with NFS and I thought that you could help me or
>> you could steer me to
>> someone who might be able to.
>>
>> The problem is this:
>>
>> An NFS Server mounts an image on a local directory, containing a XEN
>> virtual machine configuration file and a XEN image.
>>
>> An NFS Client computer sees the directory that was mounted remotely
>> and starts the XEN Virtual machine on the mounted directory.
>> The NFS Client computer then stops the XEN Virtual machine. So far, so
>> good. The machine ran. It stopped. The NFS Client
>> leaves the directory. No process is accessing that directory from the
>> client side.
>
> It is still exported, so the NFS server still has a reference to it.
>
>> Problem: The NFS Server cannot umount the local directory.
>> fuser returns no usage of the directory.
>> lsof returns no usage of the directory.
>
> correction: lsof and fuser don't show any _user processes_ that
> reference the directory. NFSd runs in the kernel...
>
>> umount -f won't force umount the directory.
>>
>> What does work is exportfs -ua.
>
> or 'exportfs -u client:/directory'. See 'man exportfs'
>
>> This is not a good way to go, since it is planned that many clients
>> will be using the server; indeed many
>> clients will be using multiple subdirectories on the server.
>>
>> 1. Is this a XEN bug or an NFS bug?
>> 2. Is there some other command like exportfs in which you can just
>> disable the one local directory but maintain an
>> active mount on the parent directory?
>
> See above.
>
> Note that if you want to unexport all export entries for a particular
> directory '/mydir', then something like this simple script will do it:
>
> exportfs -u $(exportfs | awk '/^\/mydir/ { print $2 ":" $1; }')
>
> Trond
>
>

Re: Problem with NFS and XEN

[Trond Myklebust]

On Fri, 2009-05-08 at 15:11 +0200, Roger Marcus wrote:
> Hi Trond,
> 
> Thank you for your answer.
> 
> exportfs does not report the individual directories that are mounted
> on the exported directory, so I cannot use your script.
> 
> on the nfsserver exports file:
> /adir     192.168.11.129(rw,sync,no_subtree_check,crossmnt,no_root_squash)
> 
> on the same machine:
> mount -o loop virtual.iso virtualstuff
> so now I have /adir/virtualstuff
> 
> (which by the way, I can:  umount virtualstuff immediately without
> problem and it releases.
> however, if I run XEN on the client, then the directory is permanently
> locked, ie, umount virtualstuff
> DOESN'T work again unless I run exportfs -ua killing all connections
> to the server.)
> 
> the nfs-client sees /adir/virtualstuff, since parent /adir is mounted.
> When I run the virtualmachine within the virtualstuff directory (xm
> create /adir/virtualstuff/config.cfg)
> and then I destroy the virtualmachine (xm destroy /adir/virtualstuff/config.cfg)
> I can never 'umount virtualstuff' from the nfsserver, even though no
> one is accessing the directory
> any more.
> 
> So on the nfsserver I cannot umount virtualstuff. this is the bug.
> 
> exportfs -ua    kills nfs for everybody, and then I can run 'umount
> virtualstuff', but this is not what I want.
> exportfs -u nfsclient:/adir       doesn't even work. I don't want this
> variation because I might be running 2 virtual machines
>                                                  in two subdirectories.
> 
> question: is the fact that ;exportfs -u nfsclient:/adir' doesn't work a bug?
> 
> What I want to be able to do is from the nfsserver side, 'umount
> virtualstuff', which I can do if I
> 1) never run XEN, or
> 2) exportfs -ua. The latter command kills all clients attached to the server
> so I cannot use this second command.
> 3) the command 'exportfs -u nfsclient:/adir' doesn't work and this
> might be a bug. I do not
> want this variant since if I am running 2 virtual machines on the
> nfsclient I would kill the other machine.
> 
> In /proc/net/rpc/nfsd.export/content   I see my virtualstuff
> subdirectory referenced. Perhaps this
> tells you something.
> 
> Thank you in advance,
> 
> Roger Marcus
> 

Oh, you're using crossmnt to export the subdirectories? In that case,
why doesn't just 'exportfs -f' suffice to allow you to unmount them?

Trond

Re: Problem with NFS and XEN

[Roger Marcus]

Trond:

BINGO!!!!
That was good!
Thank you thank you.
Does this mean that if I recreate the subdirectory, I have to do someth=
ing to
re-exportit? like exportfs -a nfsclient:/adir

I will keep going, but that is great so far!
Thank you very much,

Roger Marcus

trond:
Oh, you're using crossmnt to export the subdirectories? In that case,
why doesn't just 'exportfs -f' suffice to allow you to unmount them?


On Fri, May 8, 2009 at 3:43 PM, Trond Myklebust
<trond.myklebust@fys.uio.no> wrote:
> On Fri, 2009-05-08 at 15:11 +0200, Roger Marcus wrote:
>> Hi Trond,
>>
>> Thank you for your answer.
>>
>> exportfs does not report the individual directories that are mounted
>> on the exported directory, so I cannot use your script.
>>
>> on the nfsserver exports file:
>> /adir =A0 =A0 192.168.11.129(rw,sync,no_subtree_check,crossmnt,no_ro=
ot_squash)
>>
>> on the same machine:
>> mount -o loop virtual.iso virtualstuff
>> so now I have /adir/virtualstuff
>>
>> (which by the way, I can: =A0umount virtualstuff immediately without
>> problem and it releases.
>> however, if I run XEN on the client, then the directory is permanent=
ly
>> locked, ie, umount virtualstuff
>> DOESN'T work again unless I run exportfs -ua killing all connections
>> to the server.)
>>
>> the nfs-client sees /adir/virtualstuff, since parent /adir is mounte=
d.
>> When I run the virtualmachine within the virtualstuff directory (xm
>> create /adir/virtualstuff/config.cfg)
>> and then I destroy the virtualmachine (xm destroy /adir/virtualstuff=
/config.cfg)
>> I can never 'umount virtualstuff' from the nfsserver, even though no
>> one is accessing the directory
>> any more.
>>
>> So on the nfsserver I cannot umount virtualstuff. this is the bug.
>>
>> exportfs -ua =A0 =A0kills nfs for everybody, and then I can run 'umo=
unt
>> virtualstuff', but this is not what I want.
>> exportfs -u nfsclient:/adir =A0 =A0 =A0 doesn't even work. I don't w=
ant this
>> variation because I might be running 2 virtual machines
>> =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =
=A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0in two subdirectories.
>>
>> question: is the fact that ;exportfs -u nfsclient:/adir' doesn't wor=
k a bug?
>>
>> What I want to be able to do is from the nfsserver side, 'umount
>> virtualstuff', which I can do if I
>> 1) never run XEN, or
>> 2) exportfs -ua. The latter command kills all clients attached to th=
e server
>> so I cannot use this second command.
>> 3) the command 'exportfs -u nfsclient:/adir' doesn't work and this
>> might be a bug. I do not
>> want this variant since if I am running 2 virtual machines on the
>> nfsclient I would kill the other machine.
>>
>> In /proc/net/rpc/nfsd.export/content =A0 I see my virtualstuff
>> subdirectory referenced. Perhaps this
>> tells you something.
>>
>> Thank you in advance,
>>
>> Roger Marcus
>>
>
> Oh, you're using crossmnt to export the subdirectories? In that case,
> why doesn't just 'exportfs -f' suffice to allow you to unmount them?
>
> Trond
>
>

Re: Problem with NFS and XEN

[J. Bruce Fields]

On Fri, May 08, 2009 at 04:32:18PM +0200, Roger Marcus wrote:
> Trond:
>=20
> BINGO!!!!
> That was good!
> Thank you thank you.
> Does this mean that if I recreate the subdirectory, I have to do some=
thing to
> re-exportit? like exportfs -a nfsclient:/adir

No, no need.

The kernel has its own cache of the export table, and asks mountd when =
a
client tries to access a filesystem that it doesn't have a cached expor=
t
for.  The "exportfs -f" just flushes that kernels cache, without
changing mountd's idea of which filesystems are exported.

So the next time a client tries to access that filesystem, the kernel
will get the export information back from mountd (and the filesystem
will become busy again).

--b.

>=20
> I will keep going, but that is great so far!
> Thank you very much,
>=20
> Roger Marcus
>=20
> trond:
> Oh, you're using crossmnt to export the subdirectories? In that case,
> why doesn't just 'exportfs -f' suffice to allow you to unmount them?
>=20
>=20
> On Fri, May 8, 2009 at 3:43 PM, Trond Myklebust
> <trond.myklebust@fys.uio.no> wrote:
> > On Fri, 2009-05-08 at 15:11 +0200, Roger Marcus wrote:
> >> Hi Trond,
> >>
> >> Thank you for your answer.
> >>
> >> exportfs does not report the individual directories that are mount=
ed
> >> on the exported directory, so I cannot use your script.
> >>
> >> on the nfsserver exports file:
> >> /adir =C2=A0 =C2=A0 192.168.11.129(rw,sync,no_subtree_check,crossm=
nt,no_root_squash)
> >>
> >> on the same machine:
> >> mount -o loop virtual.iso virtualstuff
> >> so now I have /adir/virtualstuff
> >>
> >> (which by the way, I can: =C2=A0umount virtualstuff immediately wi=
thout
> >> problem and it releases.
> >> however, if I run XEN on the client, then the directory is permane=
ntly
> >> locked, ie, umount virtualstuff
> >> DOESN'T work again unless I run exportfs -ua killing all connectio=
ns
> >> to the server.)
> >>
> >> the nfs-client sees /adir/virtualstuff, since parent /adir is moun=
ted.
> >> When I run the virtualmachine within the virtualstuff directory (x=
m
> >> create /adir/virtualstuff/config.cfg)
> >> and then I destroy the virtualmachine (xm destroy /adir/virtualstu=
ff/config.cfg)
> >> I can never 'umount virtualstuff' from the nfsserver, even though =
no
> >> one is accessing the directory
> >> any more.
> >>
> >> So on the nfsserver I cannot umount virtualstuff. this is the bug.
> >>
> >> exportfs -ua =C2=A0 =C2=A0kills nfs for everybody, and then I can =
run 'umount
> >> virtualstuff', but this is not what I want.
> >> exportfs -u nfsclient:/adir =C2=A0 =C2=A0 =C2=A0 doesn't even work=
=2E I don't want this
> >> variation because I might be running 2 virtual machines
> >> =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=
=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0=
 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0in two subdirectories.
> >>
> >> question: is the fact that ;exportfs -u nfsclient:/adir' doesn't w=
ork a bug?
> >>
> >> What I want to be able to do is from the nfsserver side, 'umount
> >> virtualstuff', which I can do if I
> >> 1) never run XEN, or
> >> 2) exportfs -ua. The latter command kills all clients attached to =
the server
> >> so I cannot use this second command.
> >> 3) the command 'exportfs -u nfsclient:/adir' doesn't work and this
> >> might be a bug. I do not
> >> want this variant since if I am running 2 virtual machines on the
> >> nfsclient I would kill the other machine.
> >>
> >> In /proc/net/rpc/nfsd.export/content =C2=A0 I see my virtualstuff
> >> subdirectory referenced. Perhaps this
> >> tells you something.
> >>
> >> Thank you in advance,
> >>
> >> Roger Marcus
> >>
> >
> > Oh, you're using crossmnt to export the subdirectories? In that cas=
e,
> > why doesn't just 'exportfs -f' suffice to allow you to unmount them=
?
> >
> > Trond
> >
> >
> --
> To unsubscribe from this list: send the line "unsubscribe linux-nfs" =
in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html