Re: NFS synchronization problems on Beowulf cluster

Re: NFS synchronization problems on Beowulf cluster

[Mario Storti]

Hi all, 

We have still problems with NFS in our Beowulf cluster. The original
post is here

http://thread.gmane.org/gmane.linux.nfs/9564/focus=9566

The news are that we reproduced the problem on Scientific Linux 4.2
and also without VNFS. In addition we have taken a tethereal capture
that may help in understanding the problem. 

In brief we have a Beowulf class cluster built on Scientific Linux
(Beryllium) 4.2, (kernel 2.6.9-22.0.1). The cluster is disk-less
(nodes don't have hard disks) based on the Warewulf package. NFS
traffic is reduced by using VNFS filesystems at the nodes. However we
reproduced the problem in a configuration with disks at the nodes. 

The problem is that for some files in user accounts if we make some
modifications to the file in the server, this changes are not seen in
the compute nodes. The NFS server is NFS3 and with the standard
configuration (8 instances of the server and default parameters). The
cluster has 20 nodes at this time, but we have made experiments with a
`cloned' cluster and even with only two nodes the problem persist.

The experiment is as follows: We change a text file in a user account
with Emacs and checking whether the change is seen in the compute
nodes. Sometimes the change is immediately seen in the compute nodes,
but many times some nodes don't see the change. 

This happens even when the cluster is idle (no computing or network
load in the slave nodes, neither on the server), so it's not a problem
of performance. It is plainly working bad.

`rpcinfo -p' and `nfsstat' on the server and on the client can be
found in the riginal post.

We tried several combinations for the export and mount options
(sync/async...) with no success. 

We made en experiment and you can find the tethereal log here

http://venus.ceride.gov.ar/~mstorti/tempo/tethereal-raw10.log.gz

The process is as follows: we start with a file `foo5' and we verify
that two compute nodes (node1 and node11) see the same file. After
that we modify the file in the server and look what the compute nodes
see. I verify that node1 sees (correctly) the change, while `node11'
doesn't see the change.

These are the operations 

* Modify file foo5 in server, adding a line with 'jaja27'
* grep jaja in node1 -> reports `jaja27'   (OK)
* grep jaja in node11 -> reports `jaja27'   (OK)
* Modify file foo5 in server, replace 'jaja27' by 'jaja28'
* grep jaja in node1 -> reports `jaja28'   (OK)
* grep jaja in node11 -> reports `jaja27'   (NOT OK)


These are the shell commands:
================================================================
                            [[turn on tethereal]]
                            [[Emacs writes #jaja27 to foo5 on server]]
[mstorti@aquiles ~]$ rsh node1 grep jaja foo5
# jaja27
[mstorti@aquiles ~]$ rsh node11 grep jaja foo5
# jaja27
                            [[Emacs changes 27 to 28 in foo5 on server]]
[mstorti@aquiles ~]$ rsh node1 grep jaja foo5
# jaja28
[mstorti@aquiles ~]$ rsh node11 grep jaja foo5
# jaja27
[mstorti@aquiles ~]$
                            [[turn off tethereal]]
================================================================

I examined the frames in the tethereal log and I found the following. 

first rsh from node1: frame 183: GETATTR reply mtime:
21:36:21.975422000

first rsh from node11: frame 378: GETATTR reply mtime:
21:36:21.975422000

second rsh from node1: frame 564: GETATTR reply mtime:
21:37:37.515422000
                       frame 566: READ

second rsh from node11: frame 713: LOOKUP reply mtime:
21:37:37.515422000

Note: the inode for the file /u/mstorti/foo  is 11590453. 

So, during the second rsh from both nodes the clients receives an
mtime for the file that is posterior to the one during the first
read. The client at node1 correctly requires the new file contents
with a READ, while the client from node11 doesn't. I assume that the
client uses the contents in the cache (which is wrong). 

Sorry for the way long post. Any help is appreciated,

TIA, Mario



-------------------------------------------------------------------------
Using Tomcat but need to do more? Need to support web services, security?
Get stuff done quickly with pre-integrated technology to make your job easier
Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo
http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642
_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs

Re: NFS synchronization problems on Beowulf cluster

[Trond Myklebust]

On Mon, 2006-07-17 at 15:17 -0300, Mario Storti wrote:
> Hi all, 
> 
> We have still problems with NFS in our Beowulf cluster. The original
> post is here
> 
> http://thread.gmane.org/gmane.linux.nfs/9564/focus=9566
> 
> The news are that we reproduced the problem on Scientific Linux 4.2
> and also without VNFS. In addition we have taken a tethereal capture
> that may help in understanding the problem. 
> 
> In brief we have a Beowulf class cluster built on Scientific Linux
> (Beryllium) 4.2, (kernel 2.6.9-22.0.1). The cluster is disk-less
> (nodes don't have hard disks) based on the Warewulf package. NFS
> traffic is reduced by using VNFS filesystems at the nodes. However we
> reproduced the problem in a configuration with disks at the nodes. 
> 
> The problem is that for some files in user accounts if we make some
> modifications to the file in the server, this changes are not seen in
> the compute nodes. The NFS server is NFS3 and with the standard
> configuration (8 instances of the server and default parameters). The
> cluster has 20 nodes at this time, but we have made experiments with a
> `cloned' cluster and even with only two nodes the problem persist.
> 
> The experiment is as follows: We change a text file in a user account
> with Emacs and checking whether the change is seen in the compute
> nodes. Sometimes the change is immediately seen in the compute nodes,
> but many times some nodes don't see the change. 
> 

This is a FAQ that has been well-publicised and discussed to death on
this list:
  http://nfs.sourceforge.net/#faq_a8

Trond


-------------------------------------------------------------------------
Take Surveys. Earn Cash. Influence the Future of IT
Join SourceForge.net's Techsay panel and you'll get the chance to share your
opinions on IT & business topics through brief surveys -- and earn cash
http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV
_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs

Re: NFS synchronization problems on Beowulf cluster

[Mario Storti]

>>>>> On Mon, 17 Jul 2006 14:49:18 -0400, 
>>>>>      Trond Myklebust <trond.myklebust@fys.uio.no> said:

> On Mon, 2006-07-17 at 15:17 -0300, Mario Storti wrote:
>> Hi all, 
>> 
>> We have still problems with NFS in our Beowulf cluster. The original
>> post is here
>> 
>> http://thread.gmane.org/gmane.linux.nfs/9564/focus=9566
>> 
>> The news are that we reproduced the problem on Scientific Linux 4.2
>> and also without VNFS. In addition we have taken a tethereal capture
>> that may help in understanding the problem. 
>> 
>> In brief we have a Beowulf class cluster built on Scientific Linux
>> (Beryllium) 4.2, (kernel 2.6.9-22.0.1). The cluster is disk-less
>> (nodes don't have hard disks) based on the Warewulf package. NFS
>> traffic is reduced by using VNFS filesystems at the nodes. However we
>> reproduced the problem in a configuration with disks at the nodes. 
>> 
>> The problem is that for some files in user accounts if we make some
>> modifications to the file in the server, this changes are not seen in
>> the compute nodes. The NFS server is NFS3 and with the standard
>> configuration (8 instances of the server and default parameters). The
>> cluster has 20 nodes at this time, but we have made experiments with a
>> `cloned' cluster and even with only two nodes the problem persist.
>> 
>> The experiment is as follows: We change a text file in a user account
>> with Emacs and checking whether the change is seen in the compute
>> nodes. Sometimes the change is immediately seen in the compute nodes,
>> but many times some nodes don't see the change. 
>> 

> This is a FAQ that has been well-publicised and discussed to death on
> this list:
>   http://nfs.sourceforge.net/#faq_a8

I tried to mount with `nocto' and `noac' options without success. Also
tried acregmax=0,acdirmax=0. We are trying to determine if there is
some mistake in our configuration or if this is a `normal'
behavior. Your post seems to suggest that this is a `normal' behavior.
However, previously we had a cluster with RedHat 7.1 (kernel
2.2.16-22, NFS version 2) and we didn't had problems like this.  We
tried to use NFS version 2 on the new cluster and the problem
persists. So I think that this is problem with the configuration and we
are still trying to solve the problem tuning the mount/server options.

Thanks for your help, 

Mario

-------------------------
Mario Alberto Storti
e-mail: mstorti at intec dot unl dot edu dot ar
http://www.cimec.org.ar/mstorti
-------------------------


-------------------------------------------------------------------------
Take Surveys. Earn Cash. Influence the Future of IT
Join SourceForge.net's Techsay panel and you'll get the chance to share your
opinions on IT & business topics through brief surveys -- and earn cash
http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV
_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs

Re: NFS synchronization problems on Beowulf cluster

[Trond Myklebust]

On Tue, 2006-07-18 at 08:19 -0300, Mario Storti wrote:
> > This is a FAQ that has been well-publicised and discussed to death on
> > this list:
> >   http://nfs.sourceforge.net/#faq_a8
> 
> I tried to mount with `nocto' and `noac' options without success. Also
> tried acregmax=0,acdirmax=0. We are trying to determine if there is
> some mistake in our configuration or if this is a `normal'
> behavior. Your post seems to suggest that this is a `normal' behavior.
> However, previously we had a cluster with RedHat 7.1 (kernel
> 2.2.16-22, NFS version 2) and we didn't had problems like this.  We
> tried to use NFS version 2 on the new cluster and the problem
> persists. So I think that this is problem with the configuration and we
> are still trying to solve the problem tuning the mount/server options.

Please reread the article. This is _normal_ behaviour. NFS has never
been guaranteed to be cache coherent in this way.

NFS checks its data cache on calls to open()/opendir(), and on taking a
lock. That is all.

Cheers,
  Trond


-------------------------------------------------------------------------
Take Surveys. Earn Cash. Influence the Future of IT
Join SourceForge.net's Techsay panel and you'll get the chance to share your
opinions on IT & business topics through brief surveys -- and earn cash
http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV
_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs

Re: NFS synchronization problems on Beowulf cluster

[Mario Storti]

>>>>> On Tue, 18 Jul 2006 09:48:51 -0400, 
>>>>>      Trond Myklebust <trond.myklebust@fys.uio.no> said:

> On Tue, 2006-07-18 at 08:19 -0300, Mario Storti wrote:
>> > This is a FAQ that has been well-publicised and discussed to death on
>> > this list:
>> >   http://nfs.sourceforge.net/#faq_a8
>> 
>> I tried to mount with `nocto' and `noac' options without success. Also
>> tried acregmax=0,acdirmax=0. We are trying to determine if there is
>> some mistake in our configuration or if this is a `normal'
>> behavior. Your post seems to suggest that this is a `normal' behavior.
>> However, previously we had a cluster with RedHat 7.1 (kernel
>> 2.2.16-22, NFS version 2) and we didn't had problems like this.  We
>> tried to use NFS version 2 on the new cluster and the problem
>> persists. So I think that this is problem with the configuration and we
>> are still trying to solve the problem tuning the mount/server options.

> Please reread the article. This is _normal_ behaviour. NFS has never
> been guaranteed to be cache coherent in this way.

> NFS checks its data cache on calls to open()/opendir(), and on taking a
> lock. That is all.

I tried to do a `ls -l' of the directory and the contents of the files
are still out of sync. (I assume that `ls -l' does an `opendir()' call.)

Perhaps these questions are too basic for this list. In that case,
please redirect me to the appropriated list or newsgroup. 

Thanks again, Mario


-------------------------------------------------------------------------
Take Surveys. Earn Cash. Influence the Future of IT
Join SourceForge.net's Techsay panel and you'll get the chance to share your
opinions on IT & business topics through brief surveys -- and earn cash
http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV
_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs

Re: NFS synchronization problems on Beowulf cluster

[Trond Myklebust]

On Thu, 2006-07-20 at 07:49 -0300, Mario Storti wrote:
> I tried to do a `ls -l' of the directory and the contents of the files
> are still out of sync. (I assume that `ls -l' does an `opendir()' call.)
> 
> Perhaps these questions are too basic for this list. In that case,
> please redirect me to the appropriated list or newsgroup. 

'ls' does an opendir(), and should normally check that the cache is up
to date. If that is not working, then something is indeed broken.

Which mount options are you seeing this with? What kind of server are
you using (if it is a Linux server, please could you specify the kernel
version and the filesystem).

Also please could you tell us if you can reproduce the problem on a more
recent kernel version (2.6.9 is several years old).

Cheers,
  Trond


-------------------------------------------------------------------------
Take Surveys. Earn Cash. Influence the Future of IT
Join SourceForge.net's Techsay panel and you'll get the chance to share your
opinions on IT & business topics through brief surveys -- and earn cash
http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV
_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs

Re: NFS synchronization problems on Beowulf cluster

[Mario Storti]

>>>>> On Thu, 20 Jul 2006 07:04:24 -0400, 
>>>>>      Trond Myklebust <trond.myklebust@fys.uio.no> said:

> On Thu, 2006-07-20 at 07:49 -0300, Mario Storti wrote:
>> I tried to do a `ls -l' of the directory and the contents of the files
>> are still out of sync. (I assume that `ls -l' does an `opendir()' call.)
>> 
>> Perhaps these questions are too basic for this list. In that case,
>> please redirect me to the appropriated list or newsgroup. 

> 'ls' does an opendir(), and should normally check that the cache is up
> to date. If that is not working, then something is indeed broken.

> Which mount options are you seeing this with? What kind of server are
> you using (if it is a Linux server, please could you specify the kernel
> version and the filesystem).

> Also please could you tell us if you can reproduce the problem on a more
> recent kernel version (2.6.9 is several years old).

Initially we built a Warewulf cluster built on Linux Fedora Core
3 (kernel 2.6.15). The cluster is disk-less (nodes don't have hard
disks). On the server the filesystem was XFS with Raid. On Warewulf
systems NFS traffic is reduced by using VNFS filesystems (RAM disks
containing /bin, /lib, etc...) at the nodes. 

As we detected the problem, we first suspected from XFS and Raid, so
we built a second system with FC3 but plain ext3 instead of XFS and no
RAID. The problem persisted. 

Then we switched to Scientific Linux 4.2 (It is equivalent to Red Hat
Enterprise Linux (RHEL) 4.2). Error persisted. 

Also we suspected of the Warewulf VNFS system (the RAM disks) and
plainly installed one server and two slave nodes but with disks at the
nodes, and the problem persisted. 

Also tried NFS version 2, and version 3 with TCP only, and UDP only. 

At this moment we are not using any mount options but we tried before
to use `acregmax=0,acdirmax=0,noac,nocto' at the nodes with no
effect. Also tried `async/sync' at the server side (we don't think
this would work, though). 

The cluster is for intensive numerical computations but all the
problems persist even when there is no computing load, neither network
traffic. Also we tried with only two slave nodes. 

There are no errors on the network, as reported by `ifconfig'. The
network is Gigabit Ethernet with 3Com switches and cards. The driver
is `sk98lin' version 8-31. 

Thanks for your time, 

Mario


-------------------------------------------------------------------------
Take Surveys. Earn Cash. Influence the Future of IT
Join SourceForge.net's Techsay panel and you'll get the chance to share your
opinions on IT & business topics through brief surveys -- and earn cash
http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV
_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs

NFS synchronization problems on Beowulf cluster

[Mario Storti]

Hi all, 

We have a Beowulf class cluster built on Linux Fedora Core 3 (kernel
2.6.15). The cluster is disk-less (nodes don't have hard disks) based
on the Warewulf package. NFS traffic is reduced by using VNFS
filesystems at the nodes.

We found something strange related with NFS. For some files in user
accounts if we make some modifications to the file in the server, this
changes are not seen in the compute nodes. The NFS server is NFS3 and
with the standard configuration (8 instances of the server and default
parameters). The cluster has 20 nodes at this time, but we have made
experiments with a `cloned' cluster and even with only two nodes the
problem persist. 

The experiment is as follows: We change a text file in a user account
with Emacs and checking whether the change is seen in the compute
nodes. Sometimes the change is immediately seen in the compute nodes,
but many times some nodes don't see the change. 

This happens even when the cluster is idle (no computing or network
load in the slave nodes, neither on the server), so it's not a problem
of performance. It is plainly working bad.

Below are the `rpcinfo -p' and `nfsstat' on the server and on the
client. The version of nfs-utils are 1.0.6. 

We tried several combinations for the export and mount options
(sync/async...) with no success. 

Any help is appreciated,

TIA, Mario

http://www.cimec.org.ar/mstorti

======================= SERVER =================================
[root@aquiles ~]# rpcinfo -p
   program vers proto   port
    100000    2   tcp    111  portmapper
    100000    2   udp    111  portmapper
    100024    1   udp    947  status
    100024    1   tcp    950  status
    100011    1   udp    777  rquotad
    100011    2   udp    777  rquotad
    100003    2   udp   2049  nfs
    100003    3   udp   2049  nfs
    100003    4   udp   2049  nfs
    100003    2   tcp   2049  nfs
    100003    3   tcp   2049  nfs
    100003    4   tcp   2049  nfs
    100021    1   udp  32775  nlockmgr
    100021    3   udp  32775  nlockmgr
    100021    4   udp  32775  nlockmgr
    100021    1   tcp  32769  nlockmgr
    100021    3   tcp  32769  nlockmgr
    100021    4   tcp  32769  nlockmgr
    100005    1   udp    795  mountd
    100005    1   tcp    798  mountd
    100005    3   udp    795  mountd
    100005    3   tcp    798  mountd
[root@aquiles ~]# nfsstat
Server rpc stats:
calls      badcalls   badauth    badclnt    xdrcall
3734126    0          0          0          0       
Server nfs v3:
null       getattr    setattr    lookup     access     readlink   
240     0% 1375308 36% 86317   2% 135591  3% 605405 16% 1719    0% 
read       write      create     mkdir      symlink    mknod      
72626   1% 1352149 36% 9697    0% 0       0% 0       0% 160     0% 
remove     rmdir      rename     link       readdir    readdirplus
4909    0% 0       0% 64      0% 30      0% 18      0% 60812   1% 
fsstat     fsinfo     pathconf   commit     
0       0% 122     0% 0       0% 28958   0% 

======================= COMPUTE NODE =============================

[root@node1 ~]# rpcinfo -p
   programa vers proto   puerto
    100000    2   tcp    111  portmapper
    100000    2   udp    111  portmapper
    100024    1   udp    892  status
    100024    1   tcp    895  status
    100021    1   udp  32768  nlockmgr
    100021    3   udp  32768  nlockmgr
    100021    4   udp  32768  nlockmgr
    100021    1   tcp  53111  nlockmgr
    100021    3   tcp  53111  nlockmgr
    100021    4   tcp  53111  nlockmgr
[root@node1 ~]# nfsstat
Server rpc stats:
calls      badcalls   badauth    badclnt    xdrcall
0          0          0          0          0       
Client rpc stats:
calls      retrans    authrefrsh
2950634    0          0       
Client nfs v3:
null       getattr    setattr    lookup     access     readlink   
0       0% 988154 33% 86986   2% 44612   1% 353215 11% 102     0% 
read       write      create     mkdir      symlink    mknod      
6821    0% 1367868 46% 9227    0% 0       0% 0       0% 8       0% 
remove     rmdir      rename     link       readdir    readdirplus
4577    0% 0       0% 7       0% 18      0% 18      0% 59879   2% 
fsstat     fsinfo     pathconf   commit     
0       0% 2       0% 0       0% 29138   0% 





-------------------------------------------------------
All the advantages of Linux Managed Hosting--Without the Cost and Risk!
Fully trained technicians. The highest number of Red Hat certifications in
the hosting industry. Fanatical Support. Click to learn more
http://sel.as-us.falkag.net/sel?cmd=lnk&kid=107521&bid=248729&dat=121642
_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs

Re: NFS synchronization problems on Beowulf cluster

[Mario Storti]

Mario Storti <mstorti <at> intec.unl.edu.ar> writes:

> We have a Beowulf class cluster built on Linux Fedora Core 3 (kernel
> 2.6.15). The cluster is disk-less (nodes don't have hard disks) based
> on the Warewulf package. NFS traffic is reduced by using VNFS
> filesystems at the nodes.
> 
> We found something strange related with NFS. For some files in user
> accounts if we make some modifications to the file in the server, this
> changes are not seen in the compute nodes. The NFS server is NFS3 and
> with the standard configuration (8 instances of the server and default
> parameters). The cluster has 20 nodes at this time, but we have made
> experiments with a `cloned' cluster and even with only two nodes the
> problem persist. 

I see now that perhaps my reference to VNFS is somewhat obscure. VNFS
is basically a RAM disk at the nodes (in our case of size 50MB
approx.) with typically `light' directories /etc, /bin, /lib... 

http://www.warewulf-cluster.org/cgi-bin/trac.cgi/wiki/VNFS

The `heavy' directories like /usr and /home are exported via standard
NFS, so that the VNFS issue is not directly related with the problem. 

On the other hand we tried to switch to NFS version 2 and the problem 
persists. 

Thanks, Mario




-------------------------------------------------------
All the advantages of Linux Managed Hosting--Without the Cost and Risk!
Fully trained technicians. The highest number of Red Hat certifications in
the hosting industry. Fanatical Support. Click to learn more
http://sel.as-us.falkag.net/sel?cmd=lnk&kid=107521&bid=248729&dat=121642
_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs

Re: Re: NFS synchronization problems on Beowulf cluster

[Peter Staubach]

Mario Storti wrote:

>Mario Storti <mstorti <at> intec.unl.edu.ar> writes:
>
>  
>
>>We have a Beowulf class cluster built on Linux Fedora Core 3 (kernel
>>2.6.15). The cluster is disk-less (nodes don't have hard disks) based
>>on the Warewulf package. NFS traffic is reduced by using VNFS
>>filesystems at the nodes.
>>
>>We found something strange related with NFS. For some files in user
>>accounts if we make some modifications to the file in the server, this
>>changes are not seen in the compute nodes. The NFS server is NFS3 and
>>with the standard configuration (8 instances of the server and default
>>parameters). The cluster has 20 nodes at this time, but we have made
>>experiments with a `cloned' cluster and even with only two nodes the
>>problem persist. 
>>    
>>
>
>I see now that perhaps my reference to VNFS is somewhat obscure. VNFS
>is basically a RAM disk at the nodes (in our case of size 50MB
>approx.) with typically `light' directories /etc, /bin, /lib... 
>
>http://www.warewulf-cluster.org/cgi-bin/trac.cgi/wiki/VNFS
>
>The `heavy' directories like /usr and /home are exported via standard
>NFS, so that the VNFS issue is not directly related with the problem. 
>
>On the other hand we tried to switch to NFS version 2 and the problem 
>persists. 
>

What is the local file system type being exported on the NFS servers?

    Thanx...

       ps


-------------------------------------------------------
All the advantages of Linux Managed Hosting--Without the Cost and Risk!
Fully trained technicians. The highest number of Red Hat certifications in
the hosting industry. Fanatical Support. Click to learn more
http://sel.as-us.falkag.net/sel?cmd=lnk&kid=107521&bid=248729&dat=121642
_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs

Re: NFS synchronization problems on Beowulf cluster

[Mario Storti]

Peter Staubach <staubach <at> redhat.com> writes:

> What is the local file system type being exported on the NFS servers?

It is software RAID mounted on top of XFS. At first time we suspected
of this combination, and the first thing we do is to clone the system
on a plain ext3 filesystem without RAID. The cloned cluster with two
nodes exhibited exactly the same behaviour, so we don't think now that
the problem is in the combination of NFS with RAID+XFS. It ssems to be
something in the NFS setup/configuration itself. 

Thanks, Mario

-------------------------
Mario Alberto Storti     [cel. +54-342-156144983]
CIMEC (INTEC/CONICET-UNL), Guemes 3450 - 3000 Santa Fe, Argentina
Tel: +54-342-4511594 (ext 1015), Tel/Fax: +54-342-4511169
e-mail: mstorti at intec dot unl dot edu dot ar
http://www.cimec.org.ar/mstorti
-------------------------




-------------------------------------------------------
All the advantages of Linux Managed Hosting--Without the Cost and Risk!
Fully trained technicians. The highest number of Red Hat certifications in
the hosting industry. Fanatical Support. Click to learn more
http://sel.as-us.falkag.net/sel?cmd=lnk&kid=107521&bid=248729&dat=121642
_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs

Re: Re: NFS synchronization problems on Beowulf cluster

[Peter Staubach]

Mario Storti wrote:

>Peter Staubach <staubach <at> redhat.com> writes:
>
>  
>
>>What is the local file system type being exported on the NFS servers?
>>    
>>
>
>It is software RAID mounted on top of XFS. At first time we suspected
>of this combination, and the first thing we do is to clone the system
>on a plain ext3 filesystem without RAID. The cloned cluster with two
>nodes exhibited exactly the same behaviour, so we don't think now that
>the problem is in the combination of NFS with RAID+XFS. It ssems to be
>something in the NFS setup/configuration itself.
>

I see.

I see from the nfsstat output that this job mix is _very_ heavily write
intensive.  I might suspect that this may be contributing to the issue
being seen.  The consistency for files is very relaxed when those files
are being written to.

Are any special mount parameters being used?

    Thanx...

       ps


-------------------------------------------------------
All the advantages of Linux Managed Hosting--Without the Cost and Risk!
Fully trained technicians. The highest number of Red Hat certifications in
the hosting industry. Fanatical Support. Click to learn more
http://sel.as-us.falkag.net/sel?cmd=lnk&kid=107521&bid=248729&dat=121642
_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs

Re: NFS synchronization problems on Beowulf cluster

[Mario Storti]

Peter Staubach <staubach <at> redhat.com> writes:

> I see from the `nfsstat' output that this job mix is _very_ heavily write
> intensive.  I might suspect that this may be contributing to the issue
> being seen.  The consistency for files is very relaxed when those files
> are being written to.

The `nfsstat' results that I sent were cumulative. I learn now that in
theory we can zero them with `-z', but it seems not to work. Perhaps
there is some intense traffic when the slaves boot but not at the time
of making the `experiment'.  At that time the NFS traffic es
_extremely_ low. We change only a small text file (for reference it's
my ~/.bashrc) and check if the changes are reflected in the
nodes. There are no applications running at the nodes. 
 
> Are any special mount parameters being used?

No, we tried vers=2 for forcing version 2, also tried `noac'. Also
tried `async/sync' at the server side. None had effect on the
problem. `noac' of course reflected in a lack of efficiency, but the
problem persists. 

Thanks, Mario




-------------------------------------------------------
All the advantages of Linux Managed Hosting--Without the Cost and Risk!
Fully trained technicians. The highest number of Red Hat certifications in
the hosting industry. Fanatical Support. Click to learn more
http://sel.as-us.falkag.net/sel?cmd=lnk&kid=107521&bid=248729&dat=121642
_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs

Re: Re: NFS synchronization problems on Beowulf cluster

[Peter Staubach]

Mario Storti wrote:

>Peter Staubach <staubach <at> redhat.com> writes:
>
>  
>
>>I see from the `nfsstat' output that this job mix is _very_ heavily write
>>intensive.  I might suspect that this may be contributing to the issue
>>being seen.  The consistency for files is very relaxed when those files
>>are being written to.
>>    
>>
>
>The `nfsstat' results that I sent were cumulative. I learn now that in
>theory we can zero them with `-z', but it seems not to work. Perhaps
>there is some intense traffic when the slaves boot but not at the time
>of making the `experiment'.  At that time the NFS traffic es
>_extremely_ low. We change only a small text file (for reference it's
>my ~/.bashrc) and check if the changes are reflected in the
>nodes. There are no applications running at the nodes. 
>
>  
>

I wouldn't have thought that there would be intense write activity when
the systems are booting.  I could see lots of read activity, but the
stats didn't show that much compared with the number of writes.

>>Are any special mount parameters being used?
>>    
>>
>
>No, we tried vers=2 for forcing version 2, also tried `noac'. Also
>tried `async/sync' at the server side. None had effect on the
>problem. `noac' of course reflected in a lack of efficiency, but the
>problem persists. 
>

I wouldn't think that sync/async would have any effect on this sort
of consistency.  Those options more control when the data/metadata is
actually flushed to stable storage, whether before the server responds
to the specific request or sometime after.

Can you get a raw tethereal capture file which shows the issue?  Perhaps
a capture which includes reading the original contents of the file and
then reading it again after the file is modified on the server or by
another client?  Perhaps from that we can get some clues as to where the
problem might be.

    Thanx...

       ps


-------------------------------------------------------
All the advantages of Linux Managed Hosting--Without the Cost and Risk!
Fully trained technicians. The highest number of Red Hat certifications in
the hosting industry. Fanatical Support. Click to learn more
http://sel.as-us.falkag.net/sel?cmd=lnk&kid=107521&bid=248729&dat=121642
_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs

Re: NFS synchronization problems on Beowulf cluster

[Mario Storti]

Peter Staubach <staubach <at> redhat.com> writes:

> Can you get a raw tethereal capture file which shows the issue?  Perhaps
> a capture which includes reading the original contents of the file and
> then reading it again after the file is modified on the server or by
> another client?  Perhaps from that we can get some clues as to where the
> problem might be.

OK. Good idea. I added a line with `#jaja12' to my file `~/.bashrc'
and verified that all nodes see the change. Then I modify the line
with Emacs in the server to `#jaja13' and svae, then to `#jaja14' and
save, and to `#jaja15'. Some nodes acknowledge the change but others
not. If I do a `parallel grep' I obtain the following

================================================================
[mstorti@aquiles ~]$ pdsh -a 'grep jaja ~/.bashrc'
node2: # jaja12
node1: # jaja15
node9: # jaja15
node10: # jaja14
node11: # jaja12
node13: # jaja15
node20: # jaja15
node3: # jaja15
node17: # jaja12
node4: # jaja15
node7: # jaja12
node8: # jaja15
node18: # jaja15
node6: # jaja12
node14: # jaja14
node12: # jaja15
node16: # jaja14
node5: # jaja15
node19: # jaja14
node15: # jaja15
[mstorti@aquiles ~]$ 
================================================================

So I choose for instance `node2' which has still the "# jaja12"
string. Then 

* I open a shell in node2,  
* Launch a `tethereal' filtering the
        communication between the server (192.168.0.201) and node2
        (192.168.0.2).
* I do a grep in node2

    [mstorti@node2 ~]$ grep jaja .bashrc
    # jaja12
    [mstorti@node2 ~]$ 

* and I close the `tethereal'. 

What I obtain is the following:

================================================================
[root@aquiles ~]# tethereal -i eth0 host aquiles and host node2
Capturing on eth0
  0.000000  192.168.0.2 -> 192.168.0.201 UDP Source port: 32769  Destination
port: 8649
  2.490445 192.168.0.201 -> 192.168.0.2  Rlogin Data: g
  2.490548  192.168.0.2 -> 192.168.0.201 Rlogin Data: g
  2.490567 192.168.0.201 -> 192.168.0.2  Rlogin Data: rep jaja .bashrc\n
  2.490794  192.168.0.2 -> 192.168.0.201 Rlogin Data: rep jaja .bashrc\r\n
  2.491546  192.168.0.2 -> 192.168.0.201 NFS V3 GETATTR Call, FH:0x76120002
  2.491597 192.168.0.201 -> 192.168.0.2  NFS V3 GETATTR Reply (Call In 6)
  2.491669  192.168.0.2 -> 192.168.0.201 TCP 1023 > nfs [ACK] Seq=108 Ack=116
Win=32645 Len=0 TSV=612612 TSER=27245753
  2.491676  192.168.0.2 -> 192.168.0.201 NFS V3 GETATTR Call, FH:0xf8ea8c0a
  2.491701 192.168.0.201 -> 192.168.0.2  NFS V3 GETATTR Reply (Call In 9)
  2.529519  192.168.0.2 -> 192.168.0.201 TCP 1023 > nfs [ACK] Seq=216 Ack=232
Win=32674 Len=0 TSV=612616 TSER=27245753
  2.536043 192.168.0.201 -> 192.168.0.2  TCP 1023 > login [ACK] Seq=18 Ack=19
Win=1460 Len=0 TSV=27245757 TSER=612612
  2.536144  192.168.0.2 -> 192.168.0.201 Rlogin Data: # jaja12\r\n[mstorti@node2
~]$ 
  2.539333 192.168.0.201 -> 192.168.0.2  TCP 1023 > login [ACK] Seq=18 Ack=48
Win=1460 Len=0 TSV=27245757 TSER=612616
  7.549413  192.168.0.2 -> 192.168.0.201 UDP Source port: 32769  Destination
port: 8649
  7.549424  192.168.0.2 -> 192.168.0.201 UDP Source port: 32769  Destination
port: 8649
  7.549428  192.168.0.2 -> 192.168.0.201 UDP Source port: 32769  Destination
port: 8649
  7.549431  192.168.0.2 -> 192.168.0.201 UDP Source port: 32769  Destination
port: 8649
  7.549435  192.168.0.2 -> 192.168.0.201 UDP Source port: 32769  Destination
port: 8649
  7.549438  192.168.0.2 -> 192.168.0.201 UDP Source port: 32769  Destination
port: 8649
20 packets captured
[root@aquiles ~]# 
================================================================

Do you see something in there? I am studying it, but I didn't know
enough of NFS for understaing it yet. 

Thank for your help, Mario




-------------------------------------------------------
All the advantages of Linux Managed Hosting--Without the Cost and Risk!
Fully trained technicians. The highest number of Red Hat certifications in
the hosting industry. Fanatical Support. Click to learn more
http://sel.as-us.falkag.net/sel?cmd=lnk&kid=107521&bid=248729&dat=121642
_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs