NFS synchronization problems on Beowulf cluster

NFS synchronization problems on Beowulf cluster

[Mario Storti]

Hi all, 

We have a Beowulf class cluster built on Linux Fedora Core 3 (kernel
2.6.15). The cluster is disk-less (nodes don't have hard disks) based
on the Warewulf package. NFS traffic is reduced by using VNFS
filesystems at the nodes.

We found something strange related with NFS. For some files in user
accounts if we make some modifications to the file in the server, this
changes are not seen in the compute nodes. The NFS server is NFS3 and
with the standard configuration (8 instances of the server and default
parameters). The cluster has 20 nodes at this time, but we have made
experiments with a `cloned' cluster and even with only two nodes the
problem persist. 

The experiment is as follows: We change a text file in a user account
with Emacs and checking whether the change is seen in the compute
nodes. Sometimes the change is immediately seen in the compute nodes,
but many times some nodes don't see the change. 

This happens even when the cluster is idle (no computing or network
load in the slave nodes, neither on the server), so it's not a problem
of performance. It is plainly working bad.

Below are the `rpcinfo -p' and `nfsstat' on the server and on the
client. The version of nfs-utils are 1.0.6. 

We tried several combinations for the export and mount options
(sync/async...) with no success. 

Any help is appreciated,

TIA, Mario

http://www.cimec.org.ar/mstorti

======================= SERVER =================================
[root@aquiles ~]# rpcinfo -p
   program vers proto   port
    100000    2   tcp    111  portmapper
    100000    2   udp    111  portmapper
    100024    1   udp    947  status
    100024    1   tcp    950  status
    100011    1   udp    777  rquotad
    100011    2   udp    777  rquotad
    100003    2   udp   2049  nfs
    100003    3   udp   2049  nfs
    100003    4   udp   2049  nfs
    100003    2   tcp   2049  nfs
    100003    3   tcp   2049  nfs
    100003    4   tcp   2049  nfs
    100021    1   udp  32775  nlockmgr
    100021    3   udp  32775  nlockmgr
    100021    4   udp  32775  nlockmgr
    100021    1   tcp  32769  nlockmgr
    100021    3   tcp  32769  nlockmgr
    100021    4   tcp  32769  nlockmgr
    100005    1   udp    795  mountd
    100005    1   tcp    798  mountd
    100005    3   udp    795  mountd
    100005    3   tcp    798  mountd
[root@aquiles ~]# nfsstat
Server rpc stats:
calls      badcalls   badauth    badclnt    xdrcall
3734126    0          0          0          0       
Server nfs v3:
null       getattr    setattr    lookup     access     readlink   
240     0% 1375308 36% 86317   2% 135591  3% 605405 16% 1719    0% 
read       write      create     mkdir      symlink    mknod      
72626   1% 1352149 36% 9697    0% 0       0% 0       0% 160     0% 
remove     rmdir      rename     link       readdir    readdirplus
4909    0% 0       0% 64      0% 30      0% 18      0% 60812   1% 
fsstat     fsinfo     pathconf   commit     
0       0% 122     0% 0       0% 28958   0% 

======================= COMPUTE NODE =============================

[root@node1 ~]# rpcinfo -p
   programa vers proto   puerto
    100000    2   tcp    111  portmapper
    100000    2   udp    111  portmapper
    100024    1   udp    892  status
    100024    1   tcp    895  status
    100021    1   udp  32768  nlockmgr
    100021    3   udp  32768  nlockmgr
    100021    4   udp  32768  nlockmgr
    100021    1   tcp  53111  nlockmgr
    100021    3   tcp  53111  nlockmgr
    100021    4   tcp  53111  nlockmgr
[root@node1 ~]# nfsstat
Server rpc stats:
calls      badcalls   badauth    badclnt    xdrcall
0          0          0          0          0       
Client rpc stats:
calls      retrans    authrefrsh
2950634    0          0       
Client nfs v3:
null       getattr    setattr    lookup     access     readlink   
0       0% 988154 33% 86986   2% 44612   1% 353215 11% 102     0% 
read       write      create     mkdir      symlink    mknod      
6821    0% 1367868 46% 9227    0% 0       0% 0       0% 8       0% 
remove     rmdir      rename     link       readdir    readdirplus
4577    0% 0       0% 7       0% 18      0% 18      0% 59879   2% 
fsstat     fsinfo     pathconf   commit     
0       0% 2       0% 0       0% 29138   0% 





-------------------------------------------------------
All the advantages of Linux Managed Hosting--Without the Cost and Risk!
Fully trained technicians. The highest number of Red Hat certifications in
the hosting industry. Fanatical Support. Click to learn more
http://sel.as-us.falkag.net/sel?cmd=lnk&kid=107521&bid=248729&dat=121642
_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs

Re: NFS synchronization problems on Beowulf cluster

[Mario Storti]

Mario Storti <mstorti <at> intec.unl.edu.ar> writes:

> We have a Beowulf class cluster built on Linux Fedora Core 3 (kernel
> 2.6.15). The cluster is disk-less (nodes don't have hard disks) based
> on the Warewulf package. NFS traffic is reduced by using VNFS
> filesystems at the nodes.
> 
> We found something strange related with NFS. For some files in user
> accounts if we make some modifications to the file in the server, this
> changes are not seen in the compute nodes. The NFS server is NFS3 and
> with the standard configuration (8 instances of the server and default
> parameters). The cluster has 20 nodes at this time, but we have made
> experiments with a `cloned' cluster and even with only two nodes the
> problem persist. 

I see now that perhaps my reference to VNFS is somewhat obscure. VNFS
is basically a RAM disk at the nodes (in our case of size 50MB
approx.) with typically `light' directories /etc, /bin, /lib... 

http://www.warewulf-cluster.org/cgi-bin/trac.cgi/wiki/VNFS

The `heavy' directories like /usr and /home are exported via standard
NFS, so that the VNFS issue is not directly related with the problem. 

On the other hand we tried to switch to NFS version 2 and the problem 
persists. 

Thanks, Mario




-------------------------------------------------------
All the advantages of Linux Managed Hosting--Without the Cost and Risk!
Fully trained technicians. The highest number of Red Hat certifications in
the hosting industry. Fanatical Support. Click to learn more
http://sel.as-us.falkag.net/sel?cmd=lnk&kid=107521&bid=248729&dat=121642
_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs

Re: Re: NFS synchronization problems on Beowulf cluster

[Peter Staubach]

Mario Storti wrote:

>Mario Storti <mstorti <at> intec.unl.edu.ar> writes:
>
>  
>
>>We have a Beowulf class cluster built on Linux Fedora Core 3 (kernel
>>2.6.15). The cluster is disk-less (nodes don't have hard disks) based
>>on the Warewulf package. NFS traffic is reduced by using VNFS
>>filesystems at the nodes.
>>
>>We found something strange related with NFS. For some files in user
>>accounts if we make some modifications to the file in the server, this
>>changes are not seen in the compute nodes. The NFS server is NFS3 and
>>with the standard configuration (8 instances of the server and default
>>parameters). The cluster has 20 nodes at this time, but we have made
>>experiments with a `cloned' cluster and even with only two nodes the
>>problem persist. 
>>    
>>
>
>I see now that perhaps my reference to VNFS is somewhat obscure. VNFS
>is basically a RAM disk at the nodes (in our case of size 50MB
>approx.) with typically `light' directories /etc, /bin, /lib... 
>
>http://www.warewulf-cluster.org/cgi-bin/trac.cgi/wiki/VNFS
>
>The `heavy' directories like /usr and /home are exported via standard
>NFS, so that the VNFS issue is not directly related with the problem. 
>
>On the other hand we tried to switch to NFS version 2 and the problem 
>persists. 
>

What is the local file system type being exported on the NFS servers?

    Thanx...

       ps


-------------------------------------------------------
All the advantages of Linux Managed Hosting--Without the Cost and Risk!
Fully trained technicians. The highest number of Red Hat certifications in
the hosting industry. Fanatical Support. Click to learn more
http://sel.as-us.falkag.net/sel?cmd=lnk&kid=107521&bid=248729&dat=121642
_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs

Re: NFS synchronization problems on Beowulf cluster

[Mario Storti]

Peter Staubach <staubach <at> redhat.com> writes:

> What is the local file system type being exported on the NFS servers?

It is software RAID mounted on top of XFS. At first time we suspected
of this combination, and the first thing we do is to clone the system
on a plain ext3 filesystem without RAID. The cloned cluster with two
nodes exhibited exactly the same behaviour, so we don't think now that
the problem is in the combination of NFS with RAID+XFS. It ssems to be
something in the NFS setup/configuration itself. 

Thanks, Mario

-------------------------
Mario Alberto Storti     [cel. +54-342-156144983]
CIMEC (INTEC/CONICET-UNL), Guemes 3450 - 3000 Santa Fe, Argentina
Tel: +54-342-4511594 (ext 1015), Tel/Fax: +54-342-4511169
e-mail: mstorti at intec dot unl dot edu dot ar
http://www.cimec.org.ar/mstorti
-------------------------




-------------------------------------------------------
All the advantages of Linux Managed Hosting--Without the Cost and Risk!
Fully trained technicians. The highest number of Red Hat certifications in
the hosting industry. Fanatical Support. Click to learn more
http://sel.as-us.falkag.net/sel?cmd=lnk&kid=107521&bid=248729&dat=121642
_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs

Re: Re: NFS synchronization problems on Beowulf cluster

[Peter Staubach]

Mario Storti wrote:

>Peter Staubach <staubach <at> redhat.com> writes:
>
>  
>
>>What is the local file system type being exported on the NFS servers?
>>    
>>
>
>It is software RAID mounted on top of XFS. At first time we suspected
>of this combination, and the first thing we do is to clone the system
>on a plain ext3 filesystem without RAID. The cloned cluster with two
>nodes exhibited exactly the same behaviour, so we don't think now that
>the problem is in the combination of NFS with RAID+XFS. It ssems to be
>something in the NFS setup/configuration itself.
>

I see.

I see from the nfsstat output that this job mix is _very_ heavily write
intensive.  I might suspect that this may be contributing to the issue
being seen.  The consistency for files is very relaxed when those files
are being written to.

Are any special mount parameters being used?

    Thanx...

       ps


-------------------------------------------------------
All the advantages of Linux Managed Hosting--Without the Cost and Risk!
Fully trained technicians. The highest number of Red Hat certifications in
the hosting industry. Fanatical Support. Click to learn more
http://sel.as-us.falkag.net/sel?cmd=lnk&kid=107521&bid=248729&dat=121642
_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs

Re: NFS synchronization problems on Beowulf cluster

[Mario Storti]

Peter Staubach <staubach <at> redhat.com> writes:

> I see from the `nfsstat' output that this job mix is _very_ heavily write
> intensive.  I might suspect that this may be contributing to the issue
> being seen.  The consistency for files is very relaxed when those files
> are being written to.

The `nfsstat' results that I sent were cumulative. I learn now that in
theory we can zero them with `-z', but it seems not to work. Perhaps
there is some intense traffic when the slaves boot but not at the time
of making the `experiment'.  At that time the NFS traffic es
_extremely_ low. We change only a small text file (for reference it's
my ~/.bashrc) and check if the changes are reflected in the
nodes. There are no applications running at the nodes. 
 
> Are any special mount parameters being used?

No, we tried vers=2 for forcing version 2, also tried `noac'. Also
tried `async/sync' at the server side. None had effect on the
problem. `noac' of course reflected in a lack of efficiency, but the
problem persists. 

Thanks, Mario




-------------------------------------------------------
All the advantages of Linux Managed Hosting--Without the Cost and Risk!
Fully trained technicians. The highest number of Red Hat certifications in
the hosting industry. Fanatical Support. Click to learn more
http://sel.as-us.falkag.net/sel?cmd=lnk&kid=107521&bid=248729&dat=121642
_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs

Re: Re: NFS synchronization problems on Beowulf cluster

[Peter Staubach]

Mario Storti wrote:

>Peter Staubach <staubach <at> redhat.com> writes:
>
>  
>
>>I see from the `nfsstat' output that this job mix is _very_ heavily write
>>intensive.  I might suspect that this may be contributing to the issue
>>being seen.  The consistency for files is very relaxed when those files
>>are being written to.
>>    
>>
>
>The `nfsstat' results that I sent were cumulative. I learn now that in
>theory we can zero them with `-z', but it seems not to work. Perhaps
>there is some intense traffic when the slaves boot but not at the time
>of making the `experiment'.  At that time the NFS traffic es
>_extremely_ low. We change only a small text file (for reference it's
>my ~/.bashrc) and check if the changes are reflected in the
>nodes. There are no applications running at the nodes. 
>
>  
>

I wouldn't have thought that there would be intense write activity when
the systems are booting.  I could see lots of read activity, but the
stats didn't show that much compared with the number of writes.

>>Are any special mount parameters being used?
>>    
>>
>
>No, we tried vers=2 for forcing version 2, also tried `noac'. Also
>tried `async/sync' at the server side. None had effect on the
>problem. `noac' of course reflected in a lack of efficiency, but the
>problem persists. 
>

I wouldn't think that sync/async would have any effect on this sort
of consistency.  Those options more control when the data/metadata is
actually flushed to stable storage, whether before the server responds
to the specific request or sometime after.

Can you get a raw tethereal capture file which shows the issue?  Perhaps
a capture which includes reading the original contents of the file and
then reading it again after the file is modified on the server or by
another client?  Perhaps from that we can get some clues as to where the
problem might be.

    Thanx...

       ps


-------------------------------------------------------
All the advantages of Linux Managed Hosting--Without the Cost and Risk!
Fully trained technicians. The highest number of Red Hat certifications in
the hosting industry. Fanatical Support. Click to learn more
http://sel.as-us.falkag.net/sel?cmd=lnk&kid=107521&bid=248729&dat=121642
_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs

Re: NFS synchronization problems on Beowulf cluster

[Mario Storti]

Peter Staubach <staubach <at> redhat.com> writes:

> Can you get a raw tethereal capture file which shows the issue?  Perhaps
> a capture which includes reading the original contents of the file and
> then reading it again after the file is modified on the server or by
> another client?  Perhaps from that we can get some clues as to where the
> problem might be.

OK. Good idea. I added a line with `#jaja12' to my file `~/.bashrc'
and verified that all nodes see the change. Then I modify the line
with Emacs in the server to `#jaja13' and svae, then to `#jaja14' and
save, and to `#jaja15'. Some nodes acknowledge the change but others
not. If I do a `parallel grep' I obtain the following

================================================================
[mstorti@aquiles ~]$ pdsh -a 'grep jaja ~/.bashrc'
node2: # jaja12
node1: # jaja15
node9: # jaja15
node10: # jaja14
node11: # jaja12
node13: # jaja15
node20: # jaja15
node3: # jaja15
node17: # jaja12
node4: # jaja15
node7: # jaja12
node8: # jaja15
node18: # jaja15
node6: # jaja12
node14: # jaja14
node12: # jaja15
node16: # jaja14
node5: # jaja15
node19: # jaja14
node15: # jaja15
[mstorti@aquiles ~]$ 
================================================================

So I choose for instance `node2' which has still the "# jaja12"
string. Then 

* I open a shell in node2,  
* Launch a `tethereal' filtering the
        communication between the server (192.168.0.201) and node2
        (192.168.0.2).
* I do a grep in node2

    [mstorti@node2 ~]$ grep jaja .bashrc
    # jaja12
    [mstorti@node2 ~]$ 

* and I close the `tethereal'. 

What I obtain is the following:

================================================================
[root@aquiles ~]# tethereal -i eth0 host aquiles and host node2
Capturing on eth0
  0.000000  192.168.0.2 -> 192.168.0.201 UDP Source port: 32769  Destination
port: 8649
  2.490445 192.168.0.201 -> 192.168.0.2  Rlogin Data: g
  2.490548  192.168.0.2 -> 192.168.0.201 Rlogin Data: g
  2.490567 192.168.0.201 -> 192.168.0.2  Rlogin Data: rep jaja .bashrc\n
  2.490794  192.168.0.2 -> 192.168.0.201 Rlogin Data: rep jaja .bashrc\r\n
  2.491546  192.168.0.2 -> 192.168.0.201 NFS V3 GETATTR Call, FH:0x76120002
  2.491597 192.168.0.201 -> 192.168.0.2  NFS V3 GETATTR Reply (Call In 6)
  2.491669  192.168.0.2 -> 192.168.0.201 TCP 1023 > nfs [ACK] Seq=108 Ack=116
Win=32645 Len=0 TSV=612612 TSER=27245753
  2.491676  192.168.0.2 -> 192.168.0.201 NFS V3 GETATTR Call, FH:0xf8ea8c0a
  2.491701 192.168.0.201 -> 192.168.0.2  NFS V3 GETATTR Reply (Call In 9)
  2.529519  192.168.0.2 -> 192.168.0.201 TCP 1023 > nfs [ACK] Seq=216 Ack=232
Win=32674 Len=0 TSV=612616 TSER=27245753
  2.536043 192.168.0.201 -> 192.168.0.2  TCP 1023 > login [ACK] Seq=18 Ack=19
Win=1460 Len=0 TSV=27245757 TSER=612612
  2.536144  192.168.0.2 -> 192.168.0.201 Rlogin Data: # jaja12\r\n[mstorti@node2
~]$ 
  2.539333 192.168.0.201 -> 192.168.0.2  TCP 1023 > login [ACK] Seq=18 Ack=48
Win=1460 Len=0 TSV=27245757 TSER=612616
  7.549413  192.168.0.2 -> 192.168.0.201 UDP Source port: 32769  Destination
port: 8649
  7.549424  192.168.0.2 -> 192.168.0.201 UDP Source port: 32769  Destination
port: 8649
  7.549428  192.168.0.2 -> 192.168.0.201 UDP Source port: 32769  Destination
port: 8649
  7.549431  192.168.0.2 -> 192.168.0.201 UDP Source port: 32769  Destination
port: 8649
  7.549435  192.168.0.2 -> 192.168.0.201 UDP Source port: 32769  Destination
port: 8649
  7.549438  192.168.0.2 -> 192.168.0.201 UDP Source port: 32769  Destination
port: 8649
20 packets captured
[root@aquiles ~]# 
================================================================

Do you see something in there? I am studying it, but I didn't know
enough of NFS for understaing it yet. 

Thank for your help, Mario




-------------------------------------------------------
All the advantages of Linux Managed Hosting--Without the Cost and Risk!
Fully trained technicians. The highest number of Red Hat certifications in
the hosting industry. Fanatical Support. Click to learn more
http://sel.as-us.falkag.net/sel?cmd=lnk&kid=107521&bid=248729&dat=121642
_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs

Re: Re: NFS synchronization problems on Beowulf cluster

[Peter Staubach]

Mario Storti wrote:

>Peter Staubach <staubach <at> redhat.com> writes:
>
>  
>
>>Can you get a raw tethereal capture file which shows the issue?  Perhaps
>>a capture which includes reading the original contents of the file and
>>then reading it again after the file is modified on the server or by
>>another client?  Perhaps from that we can get some clues as to where the
>>problem might be.
>>    
>>
>
>OK. Good idea. I added a line with `#jaja12' to my file `~/.bashrc'
>and verified that all nodes see the change. Then I modify the line
>with Emacs in the server to `#jaja13' and svae, then to `#jaja14' and
>save, and to `#jaja15'. Some nodes acknowledge the change but others
>not. If I do a `parallel grep' I obtain the following
>
>================================================================
>[mstorti@aquiles ~]$ pdsh -a 'grep jaja ~/.bashrc'
>node2: # jaja12
>node1: # jaja15
>node9: # jaja15
>node10: # jaja14
>node11: # jaja12
>node13: # jaja15
>node20: # jaja15
>node3: # jaja15
>node17: # jaja12
>node4: # jaja15
>node7: # jaja12
>node8: # jaja15
>node18: # jaja15
>node6: # jaja12
>node14: # jaja14
>node12: # jaja15
>node16: # jaja14
>node5: # jaja15
>node19: # jaja14
>node15: # jaja15
>[mstorti@aquiles ~]$ 
>================================================================
>
>So I choose for instance `node2' which has still the "# jaja12"
>string. Then 
>
>* I open a shell in node2,  
>* Launch a `tethereal' filtering the
>        communication between the server (192.168.0.201) and node2
>        (192.168.0.2).
>* I do a grep in node2
>
>    [mstorti@node2 ~]$ grep jaja .bashrc
>    # jaja12
>    [mstorti@node2 ~]$ 
>
>* and I close the `tethereal'. 
>
>What I obtain is the following:
>
>================================================================
>[root@aquiles ~]# tethereal -i eth0 host aquiles and host node2
>Capturing on eth0
>  0.000000  192.168.0.2 -> 192.168.0.201 UDP Source port: 32769  Destination
>port: 8649
>  2.490445 192.168.0.201 -> 192.168.0.2  Rlogin Data: g
>  2.490548  192.168.0.2 -> 192.168.0.201 Rlogin Data: g
>  2.490567 192.168.0.201 -> 192.168.0.2  Rlogin Data: rep jaja .bashrc\n
>  2.490794  192.168.0.2 -> 192.168.0.201 Rlogin Data: rep jaja .bashrc\r\n
>  2.491546  192.168.0.2 -> 192.168.0.201 NFS V3 GETATTR Call, FH:0x76120002
>  2.491597 192.168.0.201 -> 192.168.0.2  NFS V3 GETATTR Reply (Call In 6)
>  2.491669  192.168.0.2 -> 192.168.0.201 TCP 1023 > nfs [ACK] Seq=108 Ack=116
>Win=32645 Len=0 TSV=612612 TSER=27245753
>  2.491676  192.168.0.2 -> 192.168.0.201 NFS V3 GETATTR Call, FH:0xf8ea8c0a
>  2.491701 192.168.0.201 -> 192.168.0.2  NFS V3 GETATTR Reply (Call In 9)
>  2.529519  192.168.0.2 -> 192.168.0.201 TCP 1023 > nfs [ACK] Seq=216 Ack=232
>Win=32674 Len=0 TSV=612616 TSER=27245753
>  2.536043 192.168.0.201 -> 192.168.0.2  TCP 1023 > login [ACK] Seq=18 Ack=19
>Win=1460 Len=0 TSV=27245757 TSER=612612
>  2.536144  192.168.0.2 -> 192.168.0.201 Rlogin Data: # jaja12\r\n[mstorti@node2
>~]$ 
>  2.539333 192.168.0.201 -> 192.168.0.2  TCP 1023 > login [ACK] Seq=18 Ack=48
>Win=1460 Len=0 TSV=27245757 TSER=612616
>  7.549413  192.168.0.2 -> 192.168.0.201 UDP Source port: 32769  Destination
>port: 8649
>  7.549424  192.168.0.2 -> 192.168.0.201 UDP Source port: 32769  Destination
>port: 8649
>  7.549428  192.168.0.2 -> 192.168.0.201 UDP Source port: 32769  Destination
>port: 8649
>  7.549431  192.168.0.2 -> 192.168.0.201 UDP Source port: 32769  Destination
>port: 8649
>  7.549435  192.168.0.2 -> 192.168.0.201 UDP Source port: 32769  Destination
>port: 8649
>  7.549438  192.168.0.2 -> 192.168.0.201 UDP Source port: 32769  Destination
>port: 8649
>20 packets captured
>[root@aquiles ~]# 
>================================================================
>
>Do you see something in there? I am studying it, but I didn't know
>enough of NFS for understaing it yet.
>

Actually, what I asked about was a "raw capture file".  I think that
the values returned in the GETATTR calls may be interesting.  To see
them, we need more then just the summary lines.

Can you do this again, but have tethereal capture into a file (via
the -w option) and then attach that file to a note to me or this list,
please?

    Thanx...

       ps


-------------------------------------------------------
All the advantages of Linux Managed Hosting--Without the Cost and Risk!
Fully trained technicians. The highest number of Red Hat certifications in
the hosting industry. Fanatical Support. Click to learn more
http://sel.as-us.falkag.net/sel?cmd=lnk&kid=107521&bid=248729&dat=121642
_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs