* NFS caching problem
@ 2005-09-26 19:13 David Warren
2005-09-26 19:36 ` Peter Staubach
2005-09-26 20:01 ` Trond Myklebust
0 siblings, 2 replies; 6+ messages in thread
From: David Warren @ 2005-09-26 19:13 UTC (permalink / raw)
To: nfs
I have discovered a wierd problem with NFSv3 on linux.
I have 3 machines
machine A and B both mount a disk D from machine C
The options are tcp,rw,hard and intr.
Program test runs on machine A writing to D:
(fortran)
program test
do i=1,10
call system("/bin/rm t")
open (10, file='t', status='new')
write(10,*)i
write(6,*)i
close(10)
call sleep(1)
enddo
end
program t2 runs on machine B reading from D:
(c, but doesn't have to be)
#include <stdlib.h>
#include <fcntl.h>
main(){
char in[80];
int file;
int len;
while(1){
file=open("t", O_RDONLY);
len=read(file,in,79);
in[len]='\0';
printf("%s\n",in);
close(file);
}
}
while machine A is counting 1 - 10 and placing these numbers into file
t, machine B is continually reading 1 from file t, then after a while it
will switch to another number and read it for a while. In my first
version of this, I was opening and rewriting the same file. In that
version, machine B always read 1's. Now that I am creating new inodes
all the time, it changes every few minutes while I repeatedly rerun test
on machine A.
Now for the other interresting facts:
Reading this file from an unrelated sun during this produces the same
result as machine B.
The same thing under NFSv4 does not do this. It works exactly as one
would expect it to. As soon as the file is writen, the reader sees the
new data.
Any ideas what I could have done wrong in my NFSv3 set up? Is there some
kernel parameter that need tweaking? is there some mount option I should
have???
Thanks.
--
David Warren INTERNET: warren@atmos.washington.edu
(206) 543-0945 Fax: (206) 543-0308
University of Washington
Dept of Atmospheric Sciences, Box 351640
Seattle, WA 98195-1640
-------------------------------------------------------
SF.Net email is sponsored by:
Tame your development challenges with Apache's Geronimo App Server.
Download it for free - -and be entered to win a 42" plasma tv or your very
own Sony(tm)PSP. Click here to play: http://sourceforge.net/geronimo.php
_______________________________________________
NFS maillist - NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs
^ permalink raw reply [flat|nested] 6+ messages in thread* Re: NFS caching problem
2005-09-26 19:13 NFS caching problem David Warren
@ 2005-09-26 19:36 ` Peter Staubach
2005-09-26 20:01 ` Trond Myklebust
1 sibling, 0 replies; 6+ messages in thread
From: Peter Staubach @ 2005-09-26 19:36 UTC (permalink / raw)
To: David Warren; +Cc: nfs
David Warren wrote:
> I have discovered a wierd problem with NFSv3 on linux.
> I have 3 machines
> machine A and B both mount a disk D from machine C
> The options are tcp,rw,hard and intr.
>
> Program test runs on machine A writing to D:
> (fortran)
> program test
> do i=1,10
> call system("/bin/rm t")
> open (10, file='t', status='new')
> write(10,*)i
> write(6,*)i
> close(10)
> call sleep(1)
> enddo
> end
>
> program t2 runs on machine B reading from D:
> (c, but doesn't have to be)
> #include <stdlib.h>
> #include <fcntl.h>
> main(){
> char in[80];
> int file;
> int len;
>
> while(1){
> file=open("t", O_RDONLY);
> len=read(file,in,79);
> in[len]='\0';
> printf("%s\n",in);
> close(file);
> }
> }
>
> while machine A is counting 1 - 10 and placing these numbers into file
> t, machine B is continually reading 1 from file t, then after a while
> it will switch to another number and read it for a while. In my first
> version of this, I was opening and rewriting the same file. In that
> version, machine B always read 1's. Now that I am creating new inodes
> all the time, it changes every few minutes while I repeatedly rerun
> test on machine A.
>
> Now for the other interresting facts:
> Reading this file from an unrelated sun during this produces the same
> result as machine B.
> The same thing under NFSv4 does not do this. It works exactly as one
> would expect it to. As soon as the file is writen, the reader sees the
> new data.
>
> Any ideas what I could have done wrong in my NFSv3 set up? Is there
> some kernel parameter that need tweaking? is there some mount option I
> should have???
What version(s) of Linux are on machines, A, B, C? What is the local file
system on machine, C, which is on disk, D?
Thanx...
ps
-------------------------------------------------------
SF.Net email is sponsored by:
Tame your development challenges with Apache's Geronimo App Server.
Download it for free - -and be entered to win a 42" plasma tv or your very
own Sony(tm)PSP. Click here to play: http://sourceforge.net/geronimo.php
_______________________________________________
NFS maillist - NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs
^ permalink raw reply [flat|nested] 6+ messages in thread* Re: NFS caching problem
2005-09-26 19:13 NFS caching problem David Warren
2005-09-26 19:36 ` Peter Staubach
@ 2005-09-26 20:01 ` Trond Myklebust
2005-09-26 20:11 ` Peter Staubach
1 sibling, 1 reply; 6+ messages in thread
From: Trond Myklebust @ 2005-09-26 20:01 UTC (permalink / raw)
To: David Warren; +Cc: nfs
m=C3=A5 den 26.09.2005 Klokka 12:13 (-0700) skreiv David Warren:
> I have discovered a wierd problem with NFSv3 on linux.
> I have 3 machines
> machine A and B both mount a disk D from machine C
> The options are tcp,rw,hard and intr.
>=20
> Program test runs on machine A writing to D:
> (fortran)
> program test
> do i=3D1,10
> call system("/bin/rm t")
> open (10, file=3D't', status=3D'new')
> write(10,*)i
> write(6,*)i
> close(10)
> call sleep(1)
> enddo
> end
>=20
> program t2 runs on machine B reading from D:
> (c, but doesn't have to be)
> #include <stdlib.h>
> #include <fcntl.h>
> main(){
> char in[80];
> int file;
> int len;
>=20
> while(1){
> file=3Dopen("t", O_RDONLY);
> len=3Dread(file,in,79);
> in[len]=3D'\0';
> printf("%s\n",in);
> close(file);
> }
> }
>=20
> while machine A is counting 1 - 10 and placing these numbers into file=20
> t, machine B is continually reading 1 from file t, then after a while it=20
> will switch to another number and read it for a while. In my first=20
> version of this, I was opening and rewriting the same file. In that=20
> version, machine B always read 1's. Now that I am creating new inodes=20
> all the time, it changes every few minutes while I repeatedly rerun test=20
> on machine A.
>=20
> Now for the other interresting facts:
> Reading this file from an unrelated sun during this produces the same=20
> result as machine B.
> The same thing under NFSv4 does not do this. It works exactly as one=20
> would expect it to. As soon as the file is writen, the reader sees the=20
> new data.
>=20
> Any ideas what I could have done wrong in my NFSv3 set up? Is there some=20
> kernel parameter that need tweaking? is there some mount option I should=20
> have???
> Thanks.
When I see that problem on my test-rig, it appears to be due to the
reuse of inode numbers by my server. IOW: the file created by the
fortran program always ends up having the same inode number (check this
using 'ls -i t').
In that case, the client is indeed expected to have problems recognising
that the file has changed w.r.t. the cache.
Cheers,
Trond
-------------------------------------------------------
SF.Net email is sponsored by:
Tame your development challenges with Apache's Geronimo App Server.
Download it for free - -and be entered to win a 42" plasma tv or your very
own Sony(tm)PSP. Click here to play: http://sourceforge.net/geronimo.php
_______________________________________________
NFS maillist - NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs
^ permalink raw reply [flat|nested] 6+ messages in thread* Re: NFS caching problem
2005-09-26 20:01 ` Trond Myklebust
@ 2005-09-26 20:11 ` Peter Staubach
0 siblings, 0 replies; 6+ messages in thread
From: Peter Staubach @ 2005-09-26 20:11 UTC (permalink / raw)
To: Trond Myklebust; +Cc: David Warren, nfs
Trond Myklebust wrote:
>m=E5 den 26.09.2005 Klokka 12:13 (-0700) skreiv David Warren:
> =20
>
>>I have discovered a wierd problem with NFSv3 on linux.
>>I have 3 machines
>>machine A and B both mount a disk D from machine C
>>The options are tcp,rw,hard and intr.
>>
>>Program test runs on machine A writing to D:
>>(fortran)
>> program test
>> do i=3D1,10
>> call system("/bin/rm t")
>> open (10, file=3D't', status=3D'new')
>> write(10,*)i
>> write(6,*)i
>> close(10)
>> call sleep(1)
>> enddo
>> end
>>
>>program t2 runs on machine B reading from D:
>>(c, but doesn't have to be)
>>#include <stdlib.h>
>>#include <fcntl.h>
>>main(){
>> char in[80];
>> int file;
>> int len;
>>
>> while(1){
>> file=3Dopen("t", O_RDONLY);
>> len=3Dread(file,in,79);
>> in[len]=3D'\0';
>> printf("%s\n",in);
>> close(file);
>> }
>>}
>>
>>while machine A is counting 1 - 10 and placing these numbers into file=20
>>t, machine B is continually reading 1 from file t, then after a while i=
t=20
>>will switch to another number and read it for a while. In my first=20
>>version of this, I was opening and rewriting the same file. In that=20
>>version, machine B always read 1's. Now that I am creating new inodes=20
>>all the time, it changes every few minutes while I repeatedly rerun tes=
t=20
>>on machine A.
>>
>>Now for the other interresting facts:
>>Reading this file from an unrelated sun during this produces the same=20
>>result as machine B.
>>The same thing under NFSv4 does not do this. It works exactly as one=20
>>would expect it to. As soon as the file is writen, the reader sees the=20
>>new data.
>>
>>Any ideas what I could have done wrong in my NFSv3 set up? Is there som=
e=20
>>kernel parameter that need tweaking? is there some mount option I shoul=
d=20
>>have???
>>Thanks.
>> =20
>>
>
>When I see that problem on my test-rig, it appears to be due to the
>reuse of inode numbers by my server. IOW: the file created by the
>fortran program always ends up having the same inode number (check this
>using 'ls -i t').
>
>In that case, the client is indeed expected to have problems recognising
>that the file has changed w.r.t. the cache.
>
Actually, the server is responsible for changing the file handle when the
inode is reused. There is a semantic called a "generation count" which i=
s
used to differentiate between instances of files (re)using the same inode
number. If the server does not do this, then it is broken and relays thi=
s
breakage on to clients as demonstrated here.
ps
-------------------------------------------------------
SF.Net email is sponsored by:
Tame your development challenges with Apache's Geronimo App Server.
Download it for free - -and be entered to win a 42" plasma tv or your very
own Sony(tm)PSP. Click here to play: http://sourceforge.net/geronimo.php
_______________________________________________
NFS maillist - NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs
^ permalink raw reply [flat|nested] 6+ messages in thread
* nfs caching problem
@ 2006-11-13 15:47 Bernd Schubert
0 siblings, 0 replies; 6+ messages in thread
From: Bernd Schubert @ 2006-11-13 15:47 UTC (permalink / raw)
To: Trond Myklebust; +Cc: nfs
Hi Trond,
I already postet this problem a couple of months before to this list, but =
never got an answer. I think there's a serious problem with nfs caching in =
2.6.16 (don't know about other kernel versions yet, we never had this probl=
em =
with 2.6.11 and below).
Here the stat output for the same directory, just on two different systems:
bernd@zuse test-wine>stat /usr/local/wine
File: `/usr/local/wine'
Size: 344 Blocks: 0 IO Block: 8192 directory
Device: 18h/24d Inode: 168810 Links: 4
Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root)
Access: 2006-11-13 15:48:50.000000000 +0100
Modify: 2006-04-24 13:40:00.000000000 +0200
Change: 2006-11-13 16:24:48.000000000 +0100
bernd@gauss bernd>stat /usr/local/wine
File: `/usr/local/wine'
Size: 400 Blocks: 0 IO Block: 8192 directory
Device: 17h/23d Inode: 181641 Links: 4
Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root)
Access: 2006-11-13 16:35:12.000000000 +0100
Modify: 2006-11-13 15:58:29.000000000 +0100
Change: 2006-11-13 16:25:20.000000000 +0100
On sytem zuse stat is actually reading the information =
of /usr/local/wine.old-2006-11-13. I had just moved wine to =
wine.old-2006-11-13. =
bernd@zuse test-wine>stat /usr/local/wine.old-2006-11-13/
File: `/usr/local/wine.old-2006-11-13/'
Size: 344 Blocks: 0 IO Block: 8192 directory
Device: 18h/24d Inode: 168810 Links: 4
Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root)
Access: 2006-11-13 15:48:50.000000000 +0100
Modify: 2006-04-24 13:40:00.000000000 +0200
Change: 2006-11-13 16:24:48.000000000 +0100
Please, this is a serious issue which already happend many time during the =
past months. Any idea how to debug this?
Thanks,
Bernd
-- =
Bernd Schubert
Physikalisch Chemisches Institut / Theoretische Chemie
Universit=E4t Heidelberg
INF 229
69120 Heidelberg
e-mail: bernd.schubert@pci.uni-heidelberg.de
-------------------------------------------------------------------------
Using Tomcat but need to do more? Need to support web services, security?
Get stuff done quickly with pre-integrated technology to make your job easi=
er
Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo
http://sel.as-us.falkag.net/sel?cmd=3Dlnk&kid=3D120709&bid=3D263057&dat=3D1=
21642
_______________________________________________
NFS maillist - NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: NFS caching problem
@ 2007-04-20 17:35 David Warren
0 siblings, 0 replies; 6+ messages in thread
From: David Warren @ 2007-04-20 17:35 UTC (permalink / raw)
To: nfs
I was a little too quick to blame it on gfs. I can now replicate it on
systems that have never been in the presence of a gfs or dlm module. It
is very consistent under V4. The only catch is you do have to get a new
inode when you recreate the file. The client does seem to pick up
changes in the same inode number. However, I have still not been able to
replicate it under V3 on these systems.
It seems like this is something that everyone should be seeing, so I am
guessing that we must have something odd in our setup. Any suggestions?
Here are all the NFS options from my config:
CONFIG_NFS_FS=m
CONFIG_NFS_V3=y
CONFIG_NFS_V3_ACL=y
CONFIG_NFS_V4=y
CONFIG_NFS_DIRECTIO=y
CONFIG_NFSD=m
CONFIG_NFSD_V2_ACL=y
CONFIG_NFSD_V3=y
CONFIG_NFSD_V3_ACL=y
CONFIG_NFSD_V4=y
CONFIG_NFSD_TCP=y
CONFIG_NFS_ACL_SUPPORT=m
CONFIG_NFS_COMMON=y
--
David Warren INTERNET: warren@atmos.washington.edu
(206) 543-0945 Fax: (206) 543-0308
University of Washington
Dept of Atmospheric Sciences, Box 351640
Seattle, WA 98195-1640
-------------------------------------------------------------------------------
DECUS E-PUBS Library Committee representative
SeaLUG DECUS Chair
-------------------------------------------------------------------------
This SF.net email is sponsored by DB2 Express
Download DB2 Express C - the FREE version of DB2 express and take
control of your XML. No limits. Just data. Click to get it now.
http://sourceforge.net/powerbar/db2/
_______________________________________________
NFS maillist - NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs
^ permalink raw reply [flat|nested] 6+ messages in thread
end of thread, other threads:[~2007-04-20 17:35 UTC | newest]
Thread overview: 6+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2005-09-26 19:13 NFS caching problem David Warren
2005-09-26 19:36 ` Peter Staubach
2005-09-26 20:01 ` Trond Myklebust
2005-09-26 20:11 ` Peter Staubach
-- strict thread matches above, loose matches on Subject: below --
2006-11-13 15:47 nfs " Bernd Schubert
2007-04-20 17:35 NFS " David Warren
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox