RE: async vs. sync

RE: async vs. sync

[Lever, Charles]

> ty den 16.11.2004 Klokka 08:15 (-0800) skreiv Lever, Charles:
> > is the effect of the "sync" export option limited to NFSv3=20
> COMMIT, or is
> > it limited to both NFSv3 COMMIT and NFSv3 FILE_SYNC/DATA_SYNC WRITE?
> >=20
> > what are the effects on NFSv4 writes and commits?
>=20
> No! It is clearly not just limited to writes and commits.
>=20
> Look at the code in fs/nfs/vfs.c: there are EX_ISSYNC()=20
> exceptions that
> wrap calls to nfsd_sync_dir() in nfsd*_create(), nfsd_symlink(),
> nfsd_link(), nfsd_rename(), and nfsd_unlink().

ulp.  wow.  this clearly affects all versions of NFS on the Linux
server, then.

i'm just looking for clarification so i can provide a good explanation
in the Linux NFS FAQ about the evils of using "async."  i'll cruise
through the server code.



-------------------------------------------------------
This SF.Net email is sponsored by: InterSystems CACHE
FREE OODBMS DOWNLOAD - A multidimensional database that combines
robust object and relational technologies, making it a perfect match
for Java, C++,COM, XML, ODBC and JDBC. www.intersystems.com/match8
_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs

Re: async vs. sync

[Olaf Kirch]

On Tue, Nov 16, 2004 at 10:48:12AM -0800, Lever, Charles wrote:
> i'm just looking for clarification so i can provide a good explanation
> in the Linux NFS FAQ about the evils of using "async."  i'll cruise
> through the server code.

Just about the only reason for async I can think of is if you have an
incoming data stream you need to write at a constant rate (think of a
diskless set top box writing an mpeg2 stream)

Olaf
-- 
Olaf Kirch     | Things that make Monday morning interesting, #2:
okir@suse.de   |        "We have 8,000 NFS mount points, why do we keep
---------------+ 	 running out of privileged ports?"


-------------------------------------------------------
SF email is sponsored by - The IT Product Guide
Read honest & candid reviews on hundreds of IT Products from real users.
Discover which products truly live up to the hype. Start reading now. 
http://productguide.itmanagersjournal.com/
_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs

Re: async vs. sync

[jehan.procaccia]

Olaf Kirch wrote:

>On Tue, Nov 16, 2004 at 10:48:12AM -0800, Lever, Charles wrote:
>  
>
>>i'm just looking for clarification so i can provide a good explanation
>>in the Linux NFS FAQ about the evils of using "async."  i'll cruise
>>through the server code.
>>    
>>
>
>Just about the only reason for async I can think of is if you have an
>incoming data stream you need to write at a constant rate (think of a
>diskless set top box writing an mpeg2 stream)
>
>Olaf
>  
>
OK, but using sync at my site is really really slow ... compare to async 
! here's a detailed (options printed) demonstration for an untar 
operation that takes 13 minutes in async mode and only 14 secondes in 
sync mode !!

1) Export in sync mode
NFS server  (RedHat ES3 kernel 2.4.21-4.ELsmp) options for that export:
$ cat /proc/fs/nfs/exports | grep arvouin
/p2v5f1 arvouin.int-evry.fr(rw,no_root_squash,sync,wdelay,acl) # 
157.159.21.55
$ cat /var/lib/nfs/xtab  | grep arvouin
/p2v5f1 
arvouin.int-evry.fr(rw,sync,wdelay,hide,nocrossmnt,secure,no_root_squash,no_all_squash,subtree_check,secure_locks,acl,mapping=identity,anonuid=-2,anongid=-2)

Client running Fedora Core 2, kernel 2.6.8-1.521
[root@arvouin /mnt/cobra3/mci/test/Test-sync]
$cat /proc/mounts
cobra3:/p2v5f1 /mnt/cobra3 nfs 
rw,v3,rsize=8192,wsize=8192,hard,tcp,lock,addr=cobra3 0 0
$time tar xvfz /usr/src/redhat/SOURCES/httpd-2.0.51.tar.gz
real    13m3.686s
user    0m1.055s
sys     0m4.354s

2) Export in async mode:
Same NFS server, options for that export:
$ cat /proc/fs/nfs/exports | grep arvouin
/p2v5f1 arvouin.int-evry.fr(rw,no_root_squash,async,wdelay,acl) # 
157.159.21.55
$ cat /var/lib/nfs/xtab  | grep arvouin
/p2v5f2 
arvouin.int-evry.fr(rw,async,wdelay,hide,nocrossmnt,secure,no_root_squash,no_all_squash,subtree_check,secure_locks,acl,mapping=identity,anonuid=-2,anongid=-2)

Same client running Fedora Core 2, kernel 2.6.8-1.521
cobra3:/p2v5f1 /mnt/cobra3 nfs 
rw,v3,rsize=8192,wsize=8192,hard,tcp,lock,addr=cobra3 0 0
[root@arvouin /mnt/cobra3/mci/test/Test-sync]
$time tar xvfz /usr/src/redhat/SOURCES/httpd-2.0.51.tar.gz
real    0m14.802s
user    0m0.867s
sys     0m2.886s

My users won't accept the sync performances ! . I have no choice, but is 
running in async mode is really evil as you mentioned it ? is there a 
way to have better performances in sync in my case ?  As anyone had the 
same gap in performance as me ( here 55 times longer in sync mode !)  ?

Thanks.


-------------------------------------------------------
SF email is sponsored by - The IT Product Guide
Read honest & candid reviews on hundreds of IT Products from real users.
Discover which products truly live up to the hype. Start reading now. 
http://productguide.itmanagersjournal.com/
_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs

RE: async vs. sync

[Roger Heflin]

This might be the issue, (and someone correct this if I am incorrect), 
I know I ran into it a couple of years ago, and it is not the easiest
to understand exactly what is actually going on.

There are 2 places where you can put sync and async, one is the exports
and one is on the mount command.  They are different.

You want sync on the exports, this will allow a client to survive without
data loss if the server reboots.   You want async on the client mount end
and
this will generally give you the speed.   With async on the client end
the client is keeping track of what is outstanding if the server crashes,
so you won't lose data.  With async on both ends the server tells one that
the data is safe when it is not, and if the server crashes the client things
that the data was safe when it really was not.  

If you put sync in both locations then your NFS disk is fully synced and 
the application won't even start another write until the last one is
confirmed
finished and on the actual disk.   With async on the client end the next
write 
will start before the client has received an ack from the server, and this
will
be reasonably fast.

So basically:

exports          mount
sync             sync      -> really safe and really slow
sync             async     -> Safe and fast
async            either    -> unsafe and fast.

Running async exports and async mount did not appear (under my testing) to
be faster under a sustained load than did sync exports and async mount.
When
the initial test was started async/async was faster but that quick changed
once
the buffer cache filled up.

                       Roger


> -----Original Message-----
> From: nfs-admin@lists.sourceforge.net 
> [mailto:nfs-admin@lists.sourceforge.net] On Behalf Of jehan.procaccia
> Sent: Monday, November 22, 2004 11:55 AM
> To: Olaf Kirch
> Cc: nfs@lists.sourceforge.net; trond.myklebust@fys.uio.no; 
> mci-unix@int-evry.fr
> Subject: Re: [NFS] async vs. sync
> 
> Olaf Kirch wrote:
> 
> >On Tue, Nov 16, 2004 at 10:48:12AM -0800, Lever, Charles wrote:
> >  
> >
> >>i'm just looking for clarification so i can provide a good 
> explanation 
> >>in the Linux NFS FAQ about the evils of using "async."  i'll cruise 
> >>through the server code.
> >>    
> >>
> >
> >Just about the only reason for async I can think of is if 
> you have an 
> >incoming data stream you need to write at a constant rate 
> (think of a 
> >diskless set top box writing an mpeg2 stream)
> >
> >Olaf
> >  
> >
> OK, but using sync at my site is really really slow ... 
> compare to async ! here's a detailed (options printed) 
> demonstration for an untar operation that takes 13 minutes in 
> async mode and only 14 secondes in sync mode !!
> 
> 1) Export in sync mode
> NFS server  (RedHat ES3 kernel 2.4.21-4.ELsmp) options for 
> that export:
> $ cat /proc/fs/nfs/exports | grep arvouin
> /p2v5f1 arvouin.int-evry.fr(rw,no_root_squash,sync,wdelay,acl) #
> 157.159.21.55
> $ cat /var/lib/nfs/xtab  | grep arvouin
> /p2v5f1
> arvouin.int-evry.fr(rw,sync,wdelay,hide,nocrossmnt,secure,no_r
oot_squash,no_all_squash,subtree_check,secure_locks,acl,mapping=identity,ano
nuid=-2,anongid=-2)
> 
> Client running Fedora Core 2, kernel 2.6.8-1.521 
> [root@arvouin /mnt/cobra3/mci/test/Test-sync] $cat /proc/mounts
> cobra3:/p2v5f1 /mnt/cobra3 nfs
> rw,v3,rsize=8192,wsize=8192,hard,tcp,lock,addr=cobra3 0 0 
> $time tar xvfz /usr/src/redhat/SOURCES/httpd-2.0.51.tar.gz
> real    13m3.686s
> user    0m1.055s
> sys     0m4.354s
> 
> 2) Export in async mode:
> Same NFS server, options for that export:
> $ cat /proc/fs/nfs/exports | grep arvouin
> /p2v5f1 arvouin.int-evry.fr(rw,no_root_squash,async,wdelay,acl) #
> 157.159.21.55
> $ cat /var/lib/nfs/xtab  | grep arvouin
> /p2v5f2
> arvouin.int-evry.fr(rw,async,wdelay,hide,nocrossmnt,secure,no_
root_squash,no_all_squash,subtree_check,secure_locks,acl,mapping=identity,an
onuid=-2,anongid=-2)
> 
> Same client running Fedora Core 2, kernel 2.6.8-1.521
> cobra3:/p2v5f1 /mnt/cobra3 nfs
> rw,v3,rsize=8192,wsize=8192,hard,tcp,lock,addr=cobra3 0 0 
> [root@arvouin /mnt/cobra3/mci/test/Test-sync] $time tar xvfz 
> /usr/src/redhat/SOURCES/httpd-2.0.51.tar.gz
> real    0m14.802s
> user    0m0.867s
> sys     0m2.886s
> 
> My users won't accept the sync performances ! . I have no 
> choice, but is running in async mode is really evil as you 
> mentioned it ? is there a way to have better performances in 
> sync in my case ?  As anyone had the same gap in performance 
> as me ( here 55 times longer in sync mode !)  ?
> 
> Thanks.
> 
> 
> -------------------------------------------------------
> SF email is sponsored by - The IT Product Guide Read honest & 
> candid reviews on hundreds of IT Products from real users.
> Discover which products truly live up to the hype. Start reading now. 
> http://productguide.itmanagersjournal.com/
> _______________________________________________
> NFS maillist  -  NFS@lists.sourceforge.net 
> https://lists.sourceforge.net/lists/listinfo/nfs
> 



-------------------------------------------------------
SF email is sponsored by - The IT Product Guide
Read honest & candid reviews on hundreds of IT Products from real users.
Discover which products truly live up to the hype. Start reading now. 
http://productguide.itmanagersjournal.com/
_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs

Re: async vs. sync

[Trond Myklebust]

m=E5 den 22.11.2004 Klokka 18:55 (+0100) skreiv jehan.procaccia:

> My users won't accept the sync performances !

Will they accept data loss in case of a server crash? If they don't
care, then hey - you're safe...

BTW: I've never seen a slowdown as bad as that. Just out of curiosity,
what happens if you turn off ACL support and/or use a client from
ftp.kernel.org, with no ACL support.

(NOTE: the reason I've rejected ACLs so far in the mainline client is
the lack of support for caching in the patches I've seen so far)

Cheers,
  Trond
--=20
Trond Myklebust <trond.myklebust@fys.uio.no>



-------------------------------------------------------
SF email is sponsored by - The IT Product Guide
Read honest & candid reviews on hundreds of IT Products from real users.
Discover which products truly live up to the hype. Start reading now. 
http://productguide.itmanagersjournal.com/
_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs

Re: async vs. sync

[jehan.procaccia]

OK , thanks for the explanation . So now I try:

exports          mount
sync             async     -> Safe and fast

[root@arvouin ~]
$mount cobra3:/p2v5f1/ -o async /mnt/cobra3

unfortunatly the async mount option  doesn't shows up in /proc/mounts, 
so I am not sure my client is using async, how can I check that ?

[root@arvouin ~]
$cat /proc/mounts
cobra3:/p2v5f1/ /mnt/cobra3 nfs 
rw,v3,rsize=8192,wsize=8192,hard,tcp,lock,addr=cobra3 0 0

Anyway, I untar again my httpd, I'am late so I didn't wait 13 minute or 
so but it started to be as long as it was :-(

Now on the server I removed ACL (nerver asked for it though !?)
$ cat /proc/fs/nfs/exports | grep arvouin
/p2v5f1 arvouin.int-evry.fr(rw,no_root_squash,sync,wdelay,no_acl) # 
157.159.21.55
again I cannot check on the client that this option is removed  ?, 
anyway again I untar my httpd,  I'am still late ... but seeing each file 
(tar -v !) shouwing up every second or so in the tty tells me that it 
will take at least 10 minutes ... :-(

Any other idea ? what is wrong in my config ?

thanks.




Roger Heflin wrote:

>This might be the issue, (and someone correct this if I am incorrect), 
>I know I ran into it a couple of years ago, and it is not the easiest
>to understand exactly what is actually going on.
>
>There are 2 places where you can put sync and async, one is the exports
>and one is on the mount command.  They are different.
>
>You want sync on the exports, this will allow a client to survive without
>data loss if the server reboots.   You want async on the client mount end
>and
>this will generally give you the speed.   With async on the client end
>the client is keeping track of what is outstanding if the server crashes,
>so you won't lose data.  With async on both ends the server tells one that
>the data is safe when it is not, and if the server crashes the client things
>that the data was safe when it really was not.  
>
>If you put sync in both locations then your NFS disk is fully synced and 
>the application won't even start another write until the last one is
>confirmed
>finished and on the actual disk.   With async on the client end the next
>write 
>will start before the client has received an ack from the server, and this
>will
>be reasonably fast.
>
>So basically:
>
>exports          mount
>sync             sync      -> really safe and really slow
>sync             async     -> Safe and fast
>async            either    -> unsafe and fast.
>
>Running async exports and async mount did not appear (under my testing) to
>be faster under a sustained load than did sync exports and async mount.
>When
>the initial test was started async/async was faster but that quick changed
>once
>the buffer cache filled up.
>
>                       Roger
>
>
>  
>
>>-----Original Message-----
>>From: nfs-admin@lists.sourceforge.net 
>>[mailto:nfs-admin@lists.sourceforge.net] On Behalf Of jehan.procaccia
>>Sent: Monday, November 22, 2004 11:55 AM
>>To: Olaf Kirch
>>Cc: nfs@lists.sourceforge.net; trond.myklebust@fys.uio.no; 
>>mci-unix@int-evry.fr
>>Subject: Re: [NFS] async vs. sync
>>
>>Olaf Kirch wrote:
>>
>>    
>>
>>>On Tue, Nov 16, 2004 at 10:48:12AM -0800, Lever, Charles wrote:
>>> 
>>>
>>>      
>>>
>>>>i'm just looking for clarification so i can provide a good 
>>>>        
>>>>
>>explanation 
>>    
>>
>>>>in the Linux NFS FAQ about the evils of using "async."  i'll cruise 
>>>>through the server code.
>>>>   
>>>>
>>>>        
>>>>
>>>Just about the only reason for async I can think of is if 
>>>      
>>>
>>you have an 
>>    
>>
>>>incoming data stream you need to write at a constant rate 
>>>      
>>>
>>(think of a 
>>    
>>
>>>diskless set top box writing an mpeg2 stream)
>>>
>>>Olaf
>>> 
>>>
>>>      
>>>
>>OK, but using sync at my site is really really slow ... 
>>compare to async ! here's a detailed (options printed) 
>>demonstration for an untar operation that takes 13 minutes in 
>>async mode and only 14 secondes in sync mode !!
>>
>>1) Export in sync mode
>>NFS server  (RedHat ES3 kernel 2.4.21-4.ELsmp) options for 
>>that export:
>>$ cat /proc/fs/nfs/exports | grep arvouin
>>/p2v5f1 arvouin.int-evry.fr(rw,no_root_squash,sync,wdelay,acl) #
>>157.159.21.55
>>$ cat /var/lib/nfs/xtab  | grep arvouin
>>/p2v5f1
>>arvouin.int-evry.fr(rw,sync,wdelay,hide,nocrossmnt,secure,no_r
>>    
>>
>oot_squash,no_all_squash,subtree_check,secure_locks,acl,mapping=identity,ano
>nuid=-2,anongid=-2)
>  
>
>>Client running Fedora Core 2, kernel 2.6.8-1.521 
>>[root@arvouin /mnt/cobra3/mci/test/Test-sync] $cat /proc/mounts
>>cobra3:/p2v5f1 /mnt/cobra3 nfs
>>rw,v3,rsize=8192,wsize=8192,hard,tcp,lock,addr=cobra3 0 0 
>>$time tar xvfz /usr/src/redhat/SOURCES/httpd-2.0.51.tar.gz
>>real    13m3.686s
>>user    0m1.055s
>>sys     0m4.354s
>>
>>2) Export in async mode:
>>Same NFS server, options for that export:
>>$ cat /proc/fs/nfs/exports | grep arvouin
>>/p2v5f1 arvouin.int-evry.fr(rw,no_root_squash,async,wdelay,acl) #
>>157.159.21.55
>>$ cat /var/lib/nfs/xtab  | grep arvouin
>>/p2v5f2
>>arvouin.int-evry.fr(rw,async,wdelay,hide,nocrossmnt,secure,no_
>>    
>>
>root_squash,no_all_squash,subtree_check,secure_locks,acl,mapping=identity,an
>onuid=-2,anongid=-2)
>  
>
>>Same client running Fedora Core 2, kernel 2.6.8-1.521
>>cobra3:/p2v5f1 /mnt/cobra3 nfs
>>rw,v3,rsize=8192,wsize=8192,hard,tcp,lock,addr=cobra3 0 0 
>>[root@arvouin /mnt/cobra3/mci/test/Test-sync] $time tar xvfz 
>>/usr/src/redhat/SOURCES/httpd-2.0.51.tar.gz
>>real    0m14.802s
>>user    0m0.867s
>>sys     0m2.886s
>>
>>My users won't accept the sync performances ! . I have no 
>>choice, but is running in async mode is really evil as you 
>>mentioned it ? is there a way to have better performances in 
>>sync in my case ?  As anyone had the same gap in performance 
>>as me ( here 55 times longer in sync mode !)  ?
>>
>>Thanks.
>>
>>
>>-------------------------------------------------------
>>SF email is sponsored by - The IT Product Guide Read honest & 
>>candid reviews on hundreds of IT Products from real users.
>>Discover which products truly live up to the hype. Start reading now. 
>>http://productguide.itmanagersjournal.com/
>>_______________________________________________
>>NFS maillist  -  NFS@lists.sourceforge.net 
>>https://lists.sourceforge.net/lists/listinfo/nfs
>>
>>    
>>
>
>  
>



-------------------------------------------------------
SF email is sponsored by - The IT Product Guide
Read honest & candid reviews on hundreds of IT Products from real users.
Discover which products truly live up to the hype. Start reading now. 
http://productguide.itmanagersjournal.com/
_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs

Re: async vs. sync

[jehan.procaccia]

Trond Myklebust wrote:

>m=E5 den 22.11.2004 Klokka 18:55 (+0100) skreiv jehan.procaccia:
>
> =20
>
>>My users won't accept the sync performances !
>>   =20
>>
>
>Will they accept data loss in case of a server crash? If they don't
>care, then hey - you're safe...
> =20
>
in my old solaris 7 nfs it was async I suppose ! although we didn't lose=20
data, maybe we where risking it ... ? anyway now I move from and old=20
solaris NFS server to a brand new linux one with a SAN (AX100) Storage=20
Processor in Raid 5 an Fiber Channel attachement, how can I support=20
performances more than 50 times longer :-( with that config,  user and=20
manager wil tell me that I  spoild money on that new server !, there=20
must be a misconfiguration somewhere ?.

>BTW: I've never seen a slowdown as bad as that. Just out of curiosity,
>what happens if you turn off ACL support and/or use a client from
>ftp.kernel.org, with no ACL support.
> =20
>
I did it on the server export option (no_acl) , no better performance.=20
However I don't know why I have ACLs,  I nerver ask for it !?
/etc/fstab on the server
/dev/emcpowerp1         /p2v5f1         ext3   =20
defaults,usrquota,grpquota     1 2

>(NOTE: the reason I've rejected ACLs so far in the mainline client is
>the lack of support for caching in the patches I've seen so far)
>
>Cheers,
>  Trond
> =20
>



-------------------------------------------------------
SF email is sponsored by - The IT Product Guide
Read honest & candid reviews on hundreds of IT Products from real users.
Discover which products truly live up to the hype. Start reading now. 
http://productguide.itmanagersjournal.com/
_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs

RE: async vs. sync

[Roger Heflin]

Solaris has always been sync on the export, async on the mount, that
is unless you had a prestoserve type module (which made some kernel
adjustments) which used to be available on the servers, and would backup
the memory that nfs was using as cache with a battery.  Without
said hardware there was no way to make Solaris async.

                               Roger=20

> -----Original Message-----
> From: nfs-admin@lists.sourceforge.net=20
> [mailto:nfs-admin@lists.sourceforge.net] On Behalf Of jehan.procaccia
> Sent: Monday, November 22, 2004 12:58 PM
> To: Trond Myklebust
> Cc: Olaf Kirch; nfs@lists.sourceforge.net; mci-unix@int-evry.fr
> Subject: Re: [NFS] async vs. sync
>=20
> Trond Myklebust wrote:
>=20
> >m=E5 den 22.11.2004 Klokka 18:55 (+0100) skreiv jehan.procaccia:
> >
> > =20
> >
> >>My users won't accept the sync performances !
> >>   =20
> >>
> >
> >Will they accept data loss in case of a server crash? If they don't=20
> >care, then hey - you're safe...
> > =20
> >
> in my old solaris 7 nfs it was async I suppose ! although we=20
> didn't lose data, maybe we where risking it ... ? anyway now=20
> I move from and old solaris NFS server to a brand new linux=20
> one with a SAN (AX100) Storage Processor in Raid 5 an Fiber=20
> Channel attachement, how can I support performances more than=20
> 50 times longer :-( with that config,  user and manager wil=20
> tell me that I  spoild money on that new server !, there must=20
> be a misconfiguration somewhere ?.
>=20
> >BTW: I've never seen a slowdown as bad as that. Just out of=20
> curiosity,=20
> >what happens if you turn off ACL support and/or use a client from=20
> >ftp.kernel.org, with no ACL support.
> > =20
> >
> I did it on the server export option (no_acl) , no better=20
> performance.=20
> However I don't know why I have ACLs,  I nerver ask for it !?
> /etc/fstab on the server
> /dev/emcpowerp1         /p2v5f1         ext3   =20
> defaults,usrquota,grpquota     1 2
>=20
> >(NOTE: the reason I've rejected ACLs so far in the mainline=20
> client is=20
> >the lack of support for caching in the patches I've seen so far)
> >
> >Cheers,
> >  Trond
> > =20
> >
>=20
>=20
>=20
> -------------------------------------------------------
> SF email is sponsored by - The IT Product Guide Read honest &=20
> candid reviews on hundreds of IT Products from real users.
> Discover which products truly live up to the hype. Start reading now.=20
> http://productguide.itmanagersjournal.com/
> _______________________________________________
> NFS maillist  -  NFS@lists.sourceforge.net=20
> https://lists.sourceforge.net/lists/listinfo/nfs
>=20



-------------------------------------------------------
SF email is sponsored by - The IT Product Guide
Read honest & candid reviews on hundreds of IT Products from real users.
Discover which products truly live up to the hype. Start reading now. 
http://productguide.itmanagersjournal.com/
_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs

RE: async vs. sync

[Roger Heflin]

 
After looking a bit closer, I am not sure that there is anything
wrong with your mount options.   The one thing I would suggest
is changing your rsize/wsize blocksize and seeing what that
does to your performance.   You should be able to go as high
as 32k.

Your test may also be somewhat misleading as it is performing
a lot of file creates and those take a lot of time no matter
how you do it, do the normal user applications do a lot of 
file creates/opens or do the normal users do a lot of large
file writes?   How many files are in that tar file? And what
is their average size?    File creates and file opens are one
place that NFS tends to have a large difference in speed.

                                 Roger

> -----Original Message-----
> From: jehan.procaccia [mailto:jehan.procaccia@int-evry.fr] 
> Sent: Monday, November 22, 2004 12:47 PM
> To: Roger Heflin
> Cc: 'Olaf Kirch'; nfs@lists.sourceforge.net; 
> trond.myklebust@fys.uio.no; mci-unix@int-evry.fr
> Subject: Re: [NFS] async vs. sync
> 
> OK , thanks for the explanation . So now I try:
> 
> exports          mount
> sync             async     -> Safe and fast
> 
> [root@arvouin ~]
> $mount cobra3:/p2v5f1/ -o async /mnt/cobra3
> 
> unfortunatly the async mount option  doesn't shows up in 
> /proc/mounts, so I am not sure my client is using async, how 
> can I check that ?
> 
> [root@arvouin ~]
> $cat /proc/mounts
> cobra3:/p2v5f1/ /mnt/cobra3 nfs
> rw,v3,rsize=8192,wsize=8192,hard,tcp,lock,addr=cobra3 0 0
> 
> Anyway, I untar again my httpd, I'am late so I didn't wait 13 
> minute or so but it started to be as long as it was :-(
> 
> Now on the server I removed ACL (nerver asked for it though 
> !?) $ cat /proc/fs/nfs/exports | grep arvouin
> /p2v5f1 arvouin.int-evry.fr(rw,no_root_squash,sync,wdelay,no_acl) #
> 157.159.21.55
> again I cannot check on the client that this option is 
> removed  ?, anyway again I untar my httpd,  I'am still late 
> ... but seeing each file (tar -v !) shouwing up every second 
> or so in the tty tells me that it will take at least 10 
> minutes ... :-(
> 
> Any other idea ? what is wrong in my config ?
> 
> thanks.
> 
> 
> 
> 
> Roger Heflin wrote:
> 
> >This might be the issue, (and someone correct this if I am 
> incorrect), 
> >I know I ran into it a couple of years ago, and it is not 
> the easiest 
> >to understand exactly what is actually going on.
> >
> >There are 2 places where you can put sync and async, one is 
> the exports 
> >and one is on the mount command.  They are different.
> >
> >You want sync on the exports, this will allow a client to 
> survive without
> >data loss if the server reboots.   You want async on the 
> client mount end
> >and
> >this will generally give you the speed.   With async on the 
> client end
> >the client is keeping track of what is outstanding if the server 
> >crashes, so you won't lose data.  With async on both ends the server 
> >tells one that the data is safe when it is not, and if the server 
> >crashes the client things that the data was safe when it 
> really was not.
> >
> >If you put sync in both locations then your NFS disk is fully synced 
> >and the application won't even start another write until the 
> last one 
> >is confirmed
> >finished and on the actual disk.   With async on the client 
> end the next
> >write
> >will start before the client has received an ack from the 
> server, and 
> >this will be reasonably fast.
> >
> >So basically:
> >
> >exports          mount
> >sync             sync      -> really safe and really slow
> >sync             async     -> Safe and fast
> >async            either    -> unsafe and fast.
> >
> >Running async exports and async mount did not appear (under 
> my testing) 
> >to be faster under a sustained load than did sync exports 
> and async mount.
> >When
> >the initial test was started async/async was faster but that quick 
> >changed once the buffer cache filled up.
> >
> >                       Roger
> >
> >
> >  
> >
> >>-----Original Message-----
> >>From: nfs-admin@lists.sourceforge.net 
> >>[mailto:nfs-admin@lists.sourceforge.net] On Behalf Of 
> jehan.procaccia
> >>Sent: Monday, November 22, 2004 11:55 AM
> >>To: Olaf Kirch
> >>Cc: nfs@lists.sourceforge.net; trond.myklebust@fys.uio.no; 
> >>mci-unix@int-evry.fr
> >>Subject: Re: [NFS] async vs. sync
> >>
> >>Olaf Kirch wrote:
> >>
> >>    
> >>
> >>>On Tue, Nov 16, 2004 at 10:48:12AM -0800, Lever, Charles wrote:
> >>> 
> >>>
> >>>      
> >>>
> >>>>i'm just looking for clarification so i can provide a good
> >>>>        
> >>>>
> >>explanation
> >>    
> >>
> >>>>in the Linux NFS FAQ about the evils of using "async."  
> i'll cruise 
> >>>>through the server code.
> >>>>   
> >>>>
> >>>>        
> >>>>
> >>>Just about the only reason for async I can think of is if
> >>>      
> >>>
> >>you have an
> >>    
> >>
> >>>incoming data stream you need to write at a constant rate
> >>>      
> >>>
> >>(think of a
> >>    
> >>
> >>>diskless set top box writing an mpeg2 stream)
> >>>
> >>>Olaf
> >>> 
> >>>
> >>>      
> >>>
> >>OK, but using sync at my site is really really slow ... 
> >>compare to async ! here's a detailed (options printed) 
> demonstration 
> >>for an untar operation that takes 13 minutes in async mode 
> and only 14 
> >>secondes in sync mode !!
> >>
> >>1) Export in sync mode
> >>NFS server  (RedHat ES3 kernel 2.4.21-4.ELsmp) options for that 
> >>export:
> >>$ cat /proc/fs/nfs/exports | grep arvouin
> >>/p2v5f1 arvouin.int-evry.fr(rw,no_root_squash,sync,wdelay,acl) #
> >>157.159.21.55
> >>$ cat /var/lib/nfs/xtab  | grep arvouin
> >>/p2v5f1
> >>arvouin.int-evry.fr(rw,sync,wdelay,hide,nocrossmnt,secure,no_r
> >>    
> >>
> >oot_squash,no_all_squash,subtree_check,secure_locks,acl,mappi
> ng=identit
> >y,ano
> >nuid=-2,anongid=-2)
> >  
> >
> >>Client running Fedora Core 2, kernel 2.6.8-1.521 [root@arvouin 
> >>/mnt/cobra3/mci/test/Test-sync] $cat /proc/mounts
> >>cobra3:/p2v5f1 /mnt/cobra3 nfs
> >>rw,v3,rsize=8192,wsize=8192,hard,tcp,lock,addr=cobra3 0 0 $time tar 
> >>xvfz /usr/src/redhat/SOURCES/httpd-2.0.51.tar.gz
> >>real    13m3.686s
> >>user    0m1.055s
> >>sys     0m4.354s
> >>
> >>2) Export in async mode:
> >>Same NFS server, options for that export:
> >>$ cat /proc/fs/nfs/exports | grep arvouin
> >>/p2v5f1 arvouin.int-evry.fr(rw,no_root_squash,async,wdelay,acl) #
> >>157.159.21.55
> >>$ cat /var/lib/nfs/xtab  | grep arvouin
> >>/p2v5f2
> >>arvouin.int-evry.fr(rw,async,wdelay,hide,nocrossmnt,secure,no_
> >>    
> >>
> >root_squash,no_all_squash,subtree_check,secure_locks,acl,mapp
> ing=identi
> >ty,an
> >onuid=-2,anongid=-2)
> >  
> >
> >>Same client running Fedora Core 2, kernel 2.6.8-1.521
> >>cobra3:/p2v5f1 /mnt/cobra3 nfs
> >>rw,v3,rsize=8192,wsize=8192,hard,tcp,lock,addr=cobra3 0 0 
> >>[root@arvouin /mnt/cobra3/mci/test/Test-sync] $time tar xvfz 
> >>/usr/src/redhat/SOURCES/httpd-2.0.51.tar.gz
> >>real    0m14.802s
> >>user    0m0.867s
> >>sys     0m2.886s
> >>
> >>My users won't accept the sync performances ! . I have no 
> choice, but 
> >>is running in async mode is really evil as you mentioned it 
> ? is there 
> >>a way to have better performances in sync in my case ?  As 
> anyone had 
> >>the same gap in performance as me ( here 55 times longer in 
> sync mode 
> >>!)  ?
> >>
> >>Thanks.
> >>
> >>
> >>-------------------------------------------------------
> >>SF email is sponsored by - The IT Product Guide Read honest 
> & candid 
> >>reviews on hundreds of IT Products from real users.
> >>Discover which products truly live up to the hype. Start 
> reading now. 
> >>http://productguide.itmanagersjournal.com/
> >>_______________________________________________
> >>NFS maillist  -  NFS@lists.sourceforge.net 
> >>https://lists.sourceforge.net/lists/listinfo/nfs
> >>
> >>    
> >>
> >
> >  
> >
> 



-------------------------------------------------------
SF email is sponsored by - The IT Product Guide
Read honest & candid reviews on hundreds of IT Products from real users.
Discover which products truly live up to the hype. Start reading now. 
http://productguide.itmanagersjournal.com/
_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs

Re: async vs. sync

[Trond Myklebust]

m=E5 den 22.11.2004 Klokka 19:57 (+0100) skreiv jehan.procaccia:
> >
> in my old solaris 7 nfs it was async I suppose ! although we didn't lose=20
> data, maybe we where risking it ... ? anyway now I move from and old=20
> solaris NFS server to a brand new linux one with a SAN (AX100) Storage=20
> Processor in Raid 5 an Fiber Channel attachement, how can I support=20
> performances more than 50 times longer :-( with that config,  user and=20
> manager wil tell me that I  spoild money on that new server !, there=20
> must be a misconfiguration somewhere ?.
>=20

Solaris has no equivalent to the "async" export option. AFAIK that is a
Linux creation.

Cheers,
  Trond
--=20
Trond Myklebust <trond.myklebust@fys.uio.no>



-------------------------------------------------------
SF email is sponsored by - The IT Product Guide
Read honest & candid reviews on hundreds of IT Products from real users.
Discover which products truly live up to the hype. Start reading now. 
http://productguide.itmanagersjournal.com/
_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs

Re: async vs. sync

[Paul Cunningham]

It has been a few years, but I remember some of the async details.  I always 
used async for performance reasons, and much testing was performed to assure 
no data would be lost.  If a client sent an async write to the server, the 
call could return prior to data being flushed to disk.  The data would make 
it to disk once the server decided to write it or the client sends in a 
COMMIT.  At some point in time the client will attempt to close the file, 
this is when a COMMIT must be sent, the hope is that the server has already 
written the dirty pages to disk while the client was busy doing other 
things, and will respond with an OK quickly.  If any dirty pages remain, 
they must be flushed prior to responding OK.  Data should never be lost as 
long as the NFSPROC3_COMMIT procedure is adhered to.

This is my recollection of how NFS v3 works (former kernel programmer), not 
necessarily how Linux implements the protocol (but I hope these rules are 
followed).

Paul Cunningham

----- Original Message ----- 
From: "Trond Myklebust" <trond.myklebust@fys.uio.no>
To: "jehan.procaccia" <jehan.procaccia@int-evry.fr>
Cc: "Olaf Kirch" <okir@suse.de>; <nfs@lists.sourceforge.net>; 
<mci-unix@int-evry.fr>
Sent: Monday, November 22, 2004 3:14 PM
Subject: Re: [NFS] async vs. sync


må den 22.11.2004 Klokka 19:57 (+0100) skreiv jehan.procaccia:
> >
> in my old solaris 7 nfs it was async I suppose ! although we didn't lose
> data, maybe we where risking it ... ? anyway now I move from and old
> solaris NFS server to a brand new linux one with a SAN (AX100) Storage
> Processor in Raid 5 an Fiber Channel attachement, how can I support
> performances more than 50 times longer :-( with that config,  user and
> manager wil tell me that I  spoild money on that new server !, there
> must be a misconfiguration somewhere ?.
>

Solaris has no equivalent to the "async" export option. AFAIK that is a
Linux creation.

Cheers,
  Trond
-- 
Trond Myklebust <trond.myklebust@fys.uio.no>



-------------------------------------------------------
SF email is sponsored by - The IT Product Guide
Read honest & candid reviews on hundreds of IT Products from real users.
Discover which products truly live up to the hype. Start reading now.
http://productguide.itmanagersjournal.com/
_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs




-------------------------------------------------------
SF email is sponsored by - The IT Product Guide
Read honest & candid reviews on hundreds of IT Products from real users.
Discover which products truly live up to the hype. Start reading now. 
http://productguide.itmanagersjournal.com/
_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs

Re: async vs. sync

[Trond Myklebust]

m=E5 den 22.11.2004 Klokka 16:04 (-0500) skreiv Paul Cunningham:

> It has been a few years, but I remember some of the async details.  I
> always=20
> used async for performance reasons, and much testing was performed to
> assure=20
> no data would be lost.  If a client sent an async write to the server,
> the=20
> call could return prior to data being flushed to disk.  The data would
> make=20
> it to disk once the server decided to write it or the client sends in
> a=20
> COMMIT.  At some point in time the client will attempt to close the
> file,=20
> this is when a COMMIT must be sent, the hope is that the server has
> already=20
> written the dirty pages to disk while the client was busy doing other=20
> things, and will respond with an OK quickly.  If any dirty pages
> remain,=20
> they must be flushed prior to responding OK.  Data should never be
> lost as=20
> long as the NFSPROC3_COMMIT procedure is adhered to.
>=20

Sure. This is how the Linux client works. The problem is the "async"
*export* option on the server.

--=20
Trond Myklebust <trond.myklebust@fys.uio.no>



-------------------------------------------------------
SF email is sponsored by - The IT Product Guide
Read honest & candid reviews on hundreds of IT Products from real users.
Discover which products truly live up to the hype. Start reading now. 
http://productguide.itmanagersjournal.com/
_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs

Re: async vs. sync

[jehan procaccia]

Roger Heflin wrote:

> 
>After looking a bit closer, I am not sure that there is anything
>wrong with your mount options.   The one thing I would suggest
>is changing your rsize/wsize blocksize and seeing what that
>does to your performance.   You should be able to go as high
>as 32k.
>  
>
[root@arvouin tmp]# mount cobra3:/p2v5f1 -o 
async,wsize=32768,rsize=32768,soft /mnt/cobra3
[root@arvouin /mnt/cobra3/mci/test/Test-sync]
$time tar xvfz /usr/src/redhat/SOURCES/httpd-2.0.51.tar.gz

sorry I don't want to wait more than 10 minutes to send that mail, but 
again seeing the files apperaing very slowly on the tty it seems not to 
be the solution :-( .

>Your test may also be somewhat misleading as it is performing
>a lot of file creates and those take a lot of time no matter
>how you do it, do the normal user applications do a lot of 
>file creates/opens or do the normal users do a lot of large
>file writes?   How many files are in that tar file? And what
>is their average size?    File creates and file opens are one
>place that NFS tends to have a large difference in speed.
>  
>
3000 files with often less than 100Ko or even 10Ko in size  in the httpd 
source tarball. Sure it won't be the day to day usage of our students, 
although they are teach computer science in our school and might have 
that kind of usage , anyway "untar, gunzip, make, make install" is one 
of Mine favorite usage ;-) , maybe I'am too selfish !.

>                                 Roger
>
>  
>
>>-----Original Message-----
>>From: jehan.procaccia [mailto:jehan.procaccia@int-evry.fr] 
>>Sent: Monday, November 22, 2004 12:47 PM
>>To: Roger Heflin
>>Cc: 'Olaf Kirch'; nfs@lists.sourceforge.net; 
>>trond.myklebust@fys.uio.no; mci-unix@int-evry.fr
>>Subject: Re: [NFS] async vs. sync
>>
>>OK , thanks for the explanation . So now I try:
>>
>>exports          mount
>>sync             async     -> Safe and fast
>>
>>[root@arvouin ~]
>>$mount cobra3:/p2v5f1/ -o async /mnt/cobra3
>>
>>unfortunatly the async mount option  doesn't shows up in 
>>/proc/mounts, so I am not sure my client is using async, how 
>>can I check that ?
>>
>>[root@arvouin ~]
>>$cat /proc/mounts
>>cobra3:/p2v5f1/ /mnt/cobra3 nfs
>>rw,v3,rsize=8192,wsize=8192,hard,tcp,lock,addr=cobra3 0 0
>>
>>Anyway, I untar again my httpd, I'am late so I didn't wait 13 
>>minute or so but it started to be as long as it was :-(
>>
>>Now on the server I removed ACL (nerver asked for it though 
>>!?) $ cat /proc/fs/nfs/exports | grep arvouin
>>/p2v5f1 arvouin.int-evry.fr(rw,no_root_squash,sync,wdelay,no_acl) #
>>157.159.21.55
>>again I cannot check on the client that this option is 
>>removed  ?, anyway again I untar my httpd,  I'am still late 
>>... but seeing each file (tar -v !) shouwing up every second 
>>or so in the tty tells me that it will take at least 10 
>>minutes ... :-(
>>
>>Any other idea ? what is wrong in my config ?
>>
>>thanks.
>>
>>
>>
>>
>>Roger Heflin wrote:
>>
>>    
>>
>>>This might be the issue, (and someone correct this if I am 
>>>      
>>>
>>incorrect), 
>>    
>>
>>>I know I ran into it a couple of years ago, and it is not 
>>>      
>>>
>>the easiest 
>>    
>>
>>>to understand exactly what is actually going on.
>>>
>>>There are 2 places where you can put sync and async, one is 
>>>      
>>>
>>the exports 
>>    
>>
>>>and one is on the mount command.  They are different.
>>>
>>>You want sync on the exports, this will allow a client to 
>>>      
>>>
>>survive without
>>    
>>
>>>data loss if the server reboots.   You want async on the 
>>>      
>>>
>>client mount end
>>    
>>
>>>and
>>>this will generally give you the speed.   With async on the 
>>>      
>>>
>>client end
>>    
>>
>>>the client is keeping track of what is outstanding if the server 
>>>crashes, so you won't lose data.  With async on both ends the server 
>>>tells one that the data is safe when it is not, and if the server 
>>>crashes the client things that the data was safe when it 
>>>      
>>>
>>really was not.
>>    
>>
>>>If you put sync in both locations then your NFS disk is fully synced 
>>>and the application won't even start another write until the 
>>>      
>>>
>>last one 
>>    
>>
>>>is confirmed
>>>finished and on the actual disk.   With async on the client 
>>>      
>>>
>>end the next
>>    
>>
>>>write
>>>will start before the client has received an ack from the 
>>>      
>>>
>>server, and 
>>    
>>
>>>this will be reasonably fast.
>>>
>>>So basically:
>>>
>>>exports          mount
>>>sync             sync      -> really safe and really slow
>>>sync             async     -> Safe and fast
>>>async            either    -> unsafe and fast.
>>>
>>>Running async exports and async mount did not appear (under 
>>>      
>>>
>>my testing) 
>>    
>>
>>>to be faster under a sustained load than did sync exports 
>>>      
>>>
>>and async mount.
>>    
>>
>>>When
>>>the initial test was started async/async was faster but that quick 
>>>changed once the buffer cache filled up.
>>>
>>>                      Roger
>>>
>>>
>>> 
>>>
>>>      
>>>
>>>>-----Original Message-----
>>>>From: nfs-admin@lists.sourceforge.net 
>>>>[mailto:nfs-admin@lists.sourceforge.net] On Behalf Of 
>>>>        
>>>>
>>jehan.procaccia
>>    
>>
>>>>Sent: Monday, November 22, 2004 11:55 AM
>>>>To: Olaf Kirch
>>>>Cc: nfs@lists.sourceforge.net; trond.myklebust@fys.uio.no; 
>>>>mci-unix@int-evry.fr
>>>>Subject: Re: [NFS] async vs. sync
>>>>
>>>>Olaf Kirch wrote:
>>>>
>>>>   
>>>>
>>>>        
>>>>
>>>>>On Tue, Nov 16, 2004 at 10:48:12AM -0800, Lever, Charles wrote:
>>>>>
>>>>>
>>>>>     
>>>>>
>>>>>          
>>>>>
>>>>>>i'm just looking for clarification so i can provide a good
>>>>>>       
>>>>>>
>>>>>>            
>>>>>>
>>>>explanation
>>>>   
>>>>
>>>>        
>>>>
>>>>>>in the Linux NFS FAQ about the evils of using "async."  
>>>>>>            
>>>>>>
>>i'll cruise 
>>    
>>
>>>>>>through the server code.
>>>>>>  
>>>>>>
>>>>>>       
>>>>>>
>>>>>>            
>>>>>>
>>>>>Just about the only reason for async I can think of is if
>>>>>     
>>>>>
>>>>>          
>>>>>
>>>>you have an
>>>>   
>>>>
>>>>        
>>>>
>>>>>incoming data stream you need to write at a constant rate
>>>>>     
>>>>>
>>>>>          
>>>>>
>>>>(think of a
>>>>   
>>>>
>>>>        
>>>>
>>>>>diskless set top box writing an mpeg2 stream)
>>>>>
>>>>>Olaf
>>>>>
>>>>>
>>>>>     
>>>>>
>>>>>          
>>>>>
>>>>OK, but using sync at my site is really really slow ... 
>>>>compare to async ! here's a detailed (options printed) 
>>>>        
>>>>
>>demonstration 
>>    
>>
>>>>for an untar operation that takes 13 minutes in async mode 
>>>>        
>>>>
>>and only 14 
>>    
>>
>>>>secondes in sync mode !!
>>>>
>>>>1) Export in sync mode
>>>>NFS server  (RedHat ES3 kernel 2.4.21-4.ELsmp) options for that 
>>>>export:
>>>>$ cat /proc/fs/nfs/exports | grep arvouin
>>>>/p2v5f1 arvouin.int-evry.fr(rw,no_root_squash,sync,wdelay,acl) #
>>>>157.159.21.55
>>>>$ cat /var/lib/nfs/xtab  | grep arvouin
>>>>/p2v5f1
>>>>arvouin.int-evry.fr(rw,sync,wdelay,hide,nocrossmnt,secure,no_r
>>>>   
>>>>
>>>>        
>>>>
>>>oot_squash,no_all_squash,subtree_check,secure_locks,acl,mappi
>>>      
>>>
>>ng=identit
>>    
>>
>>>y,ano
>>>nuid=-2,anongid=-2)
>>> 
>>>
>>>      
>>>
>>>>Client running Fedora Core 2, kernel 2.6.8-1.521 [root@arvouin 
>>>>/mnt/cobra3/mci/test/Test-sync] $cat /proc/mounts
>>>>cobra3:/p2v5f1 /mnt/cobra3 nfs
>>>>rw,v3,rsize=8192,wsize=8192,hard,tcp,lock,addr=cobra3 0 0 $time tar 
>>>>xvfz /usr/src/redhat/SOURCES/httpd-2.0.51.tar.gz
>>>>real    13m3.686s
>>>>user    0m1.055s
>>>>sys     0m4.354s
>>>>
>>>>2) Export in async mode:
>>>>Same NFS server, options for that export:
>>>>$ cat /proc/fs/nfs/exports | grep arvouin
>>>>/p2v5f1 arvouin.int-evry.fr(rw,no_root_squash,async,wdelay,acl) #
>>>>157.159.21.55
>>>>$ cat /var/lib/nfs/xtab  | grep arvouin
>>>>/p2v5f2
>>>>arvouin.int-evry.fr(rw,async,wdelay,hide,nocrossmnt,secure,no_
>>>>   
>>>>
>>>>        
>>>>
>>>root_squash,no_all_squash,subtree_check,secure_locks,acl,mapp
>>>      
>>>
>>ing=identi
>>    
>>
>>>ty,an
>>>onuid=-2,anongid=-2)
>>> 
>>>
>>>      
>>>
>>>>Same client running Fedora Core 2, kernel 2.6.8-1.521
>>>>cobra3:/p2v5f1 /mnt/cobra3 nfs
>>>>rw,v3,rsize=8192,wsize=8192,hard,tcp,lock,addr=cobra3 0 0 
>>>>[root@arvouin /mnt/cobra3/mci/test/Test-sync] $time tar xvfz 
>>>>/usr/src/redhat/SOURCES/httpd-2.0.51.tar.gz
>>>>real    0m14.802s
>>>>user    0m0.867s
>>>>sys     0m2.886s
>>>>
>>>>My users won't accept the sync performances ! . I have no 
>>>>        
>>>>
>>choice, but 
>>    
>>
>>>>is running in async mode is really evil as you mentioned it 
>>>>        
>>>>
>>? is there 
>>    
>>
>>>>a way to have better performances in sync in my case ?  As 
>>>>        
>>>>
>>anyone had 
>>    
>>
>>>>the same gap in performance as me ( here 55 times longer in 
>>>>        
>>>>
>>sync mode 
>>    
>>
>>>>!)  ?
>>>>
>>>>Thanks.
>>>>
>>>>
>>>>-------------------------------------------------------
>>>>SF email is sponsored by - The IT Product Guide Read honest 
>>>>        
>>>>
>>& candid 
>>    
>>
>>>>reviews on hundreds of IT Products from real users.
>>>>Discover which products truly live up to the hype. Start 
>>>>        
>>>>
>>reading now. 
>>    
>>
>>>>http://productguide.itmanagersjournal.com/
>>>>_______________________________________________
>>>>NFS maillist  -  NFS@lists.sourceforge.net 
>>>>https://lists.sourceforge.net/lists/listinfo/nfs
>>>>
>>>>   
>>>>
>>>>        
>>>>
>>> 
>>>
>>>      
>>>
>
>  
>



-------------------------------------------------------
SF email is sponsored by - The IT Product Guide
Read honest & candid reviews on hundreds of IT Products from real users.
Discover which products truly live up to the hype. Start reading now. 
http://productguide.itmanagersjournal.com/
_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs

Re: async vs. sync

[jehan procaccia]

jehan procaccia wrote:

> [root@arvouin tmp]# mount cobra3:/p2v5f1 -o 
> async,wsize=32768,rsize=32768,soft /mnt/cobra3
> [root@arvouin /mnt/cobra3/mci/test/Test-sync]
> $time tar xvfz /usr/src/redhat/SOURCES/httpd-2.0.51.tar.gz
>
> sorry I don't want to wait more than 10 minutes to send that mail, but 
> again seeing the files apperaing very slowly on the tty it seems not 
> to be the solution :-( .

For the record ... it finnaly ended, not very better than the original 
13 minutes !
real    12m53.520s
user    0m1.070s
sys     0m4.268s

>
>> Your test may also be somewhat misleading as it is performing
>> a lot of file creates and those take a lot of time no matter
>> how you do it, do the normal user applications do a lot of file 
>> creates/opens or do the normal users do a lot of large
>> file writes?   How many files are in that tar file? And what
>> is their average size?    File creates and file opens are one
>> place that NFS tends to have a large difference in speed.
>>  
>>
> 3000 files with often less than 100Ko or even 10Ko in size  in the 
> httpd source tarball. Sure it won't be the day to day usage of our 
> students, although they are teach computer science in our school and 
> might have that kind of usage , anyway "untar, gunzip, make, make 
> install" is one of Mine favorite usage ;-) , maybe I'am too selfish !.
>
>>                                 Roger
>>
>>  
>>
>>> -----Original Message-----
>>> From: jehan.procaccia [mailto:jehan.procaccia@int-evry.fr] Sent: 
>>> Monday, November 22, 2004 12:47 PM
>>> To: Roger Heflin
>>> Cc: 'Olaf Kirch'; nfs@lists.sourceforge.net; 
>>> trond.myklebust@fys.uio.no; mci-unix@int-evry.fr
>>> Subject: Re: [NFS] async vs. sync
>>>
>>> OK , thanks for the explanation . So now I try:
>>>
>>> exports          mount
>>> sync             async     -> Safe and fast
>>>
>>> [root@arvouin ~]
>>> $mount cobra3:/p2v5f1/ -o async /mnt/cobra3
>>>
>>> unfortunatly the async mount option  doesn't shows up in 
>>> /proc/mounts, so I am not sure my client is using async, how can I 
>>> check that ?
>>>
>>> [root@arvouin ~]
>>> $cat /proc/mounts
>>> cobra3:/p2v5f1/ /mnt/cobra3 nfs
>>> rw,v3,rsize=8192,wsize=8192,hard,tcp,lock,addr=cobra3 0 0
>>>
>>> Anyway, I untar again my httpd, I'am late so I didn't wait 13 minute 
>>> or so but it started to be as long as it was :-(
>>>
>>> Now on the server I removed ACL (nerver asked for it though !?) $ 
>>> cat /proc/fs/nfs/exports | grep arvouin
>>> /p2v5f1 arvouin.int-evry.fr(rw,no_root_squash,sync,wdelay,no_acl) #
>>> 157.159.21.55
>>> again I cannot check on the client that this option is removed  ?, 
>>> anyway again I untar my httpd,  I'am still late ... but seeing each 
>>> file (tar -v !) shouwing up every second or so in the tty tells me 
>>> that it will take at least 10 minutes ... :-(
>>>
>>> Any other idea ? what is wrong in my config ?
>>>
>>> thanks.
>>>
>>>
>>>
>>>
>>> Roger Heflin wrote:
>>>
>>>   
>>>
>>>> This might be the issue, (and someone correct this if I am     
>>>
>>> incorrect),   
>>>
>>>> I know I ran into it a couple of years ago, and it is not     
>>>
>>> the easiest   
>>>
>>>> to understand exactly what is actually going on.
>>>>
>>>> There are 2 places where you can put sync and async, one is     
>>>
>>> the exports   
>>>
>>>> and one is on the mount command.  They are different.
>>>>
>>>> You want sync on the exports, this will allow a client to     
>>>
>>> survive without
>>>   
>>>
>>>> data loss if the server reboots.   You want async on the     
>>>
>>> client mount end
>>>   
>>>
>>>> and
>>>> this will generally give you the speed.   With async on the     
>>>
>>> client end
>>>   
>>>
>>>> the client is keeping track of what is outstanding if the server 
>>>> crashes, so you won't lose data.  With async on both ends the 
>>>> server tells one that the data is safe when it is not, and if the 
>>>> server crashes the client things that the data was safe when it     
>>>
>>> really was not.
>>>   
>>>
>>>> If you put sync in both locations then your NFS disk is fully 
>>>> synced and the application won't even start another write until the 
>>>>     
>>>
>>> last one   
>>>
>>>> is confirmed
>>>> finished and on the actual disk.   With async on the client     
>>>
>>> end the next
>>>   
>>>
>>>> write
>>>> will start before the client has received an ack from the     
>>>
>>> server, and   
>>>
>>>> this will be reasonably fast.
>>>>
>>>> So basically:
>>>>
>>>> exports          mount
>>>> sync             sync      -> really safe and really slow
>>>> sync             async     -> Safe and fast
>>>> async            either    -> unsafe and fast.
>>>>
>>>> Running async exports and async mount did not appear (under     
>>>
>>> my testing)   
>>>
>>>> to be faster under a sustained load than did sync exports     
>>>
>>> and async mount.
>>>   
>>>
>>>> When
>>>> the initial test was started async/async was faster but that quick 
>>>> changed once the buffer cache filled up.
>>>>
>>>>                      Roger
>>>>
>>>>
>>>>
>>>>
>>>>     
>>>>
>>>>> -----Original Message-----
>>>>> From: nfs-admin@lists.sourceforge.net 
>>>>> [mailto:nfs-admin@lists.sourceforge.net] On Behalf Of       
>>>>
>>> jehan.procaccia
>>>   
>>>
>>>>> Sent: Monday, November 22, 2004 11:55 AM
>>>>> To: Olaf Kirch
>>>>> Cc: nfs@lists.sourceforge.net; trond.myklebust@fys.uio.no; 
>>>>> mci-unix@int-evry.fr
>>>>> Subject: Re: [NFS] async vs. sync
>>>>>
>>>>> Olaf Kirch wrote:
>>>>>
>>>>>  
>>>>>       
>>>>>
>>>>>> On Tue, Nov 16, 2004 at 10:48:12AM -0800, Lever, Charles wrote:
>>>>>>
>>>>>>
>>>>>>    
>>>>>>         
>>>>>>
>>>>>>> i'm just looking for clarification so i can provide a good
>>>>>>>      
>>>>>>>           
>>>>>>
>>>>> explanation
>>>>>  
>>>>>       
>>>>>
>>>>>>> in the Linux NFS FAQ about the evils of using "async."            
>>>>>>
>>> i'll cruise   
>>>
>>>>>>> through the server code.
>>>>>>>  
>>>>>>>
>>>>>>>      
>>>>>>>           
>>>>>>
>>>>>> Just about the only reason for async I can think of is if
>>>>>>    
>>>>>>         
>>>>>
>>>>> you have an
>>>>>  
>>>>>       
>>>>>
>>>>>> incoming data stream you need to write at a constant rate
>>>>>>    
>>>>>>         
>>>>>
>>>>> (think of a
>>>>>  
>>>>>       
>>>>>
>>>>>> diskless set top box writing an mpeg2 stream)
>>>>>>
>>>>>> Olaf
>>>>>>
>>>>>>
>>>>>>    
>>>>>>         
>>>>>
>>>>> OK, but using sync at my site is really really slow ... compare to 
>>>>> async ! here's a detailed (options printed)       
>>>>
>>> demonstration   
>>>
>>>>> for an untar operation that takes 13 minutes in async mode       
>>>>
>>> and only 14   
>>>
>>>>> secondes in sync mode !!
>>>>>
>>>>> 1) Export in sync mode
>>>>> NFS server  (RedHat ES3 kernel 2.4.21-4.ELsmp) options for that 
>>>>> export:
>>>>> $ cat /proc/fs/nfs/exports | grep arvouin
>>>>> /p2v5f1 arvouin.int-evry.fr(rw,no_root_squash,sync,wdelay,acl) #
>>>>> 157.159.21.55
>>>>> $ cat /var/lib/nfs/xtab  | grep arvouin
>>>>> /p2v5f1
>>>>> arvouin.int-evry.fr(rw,sync,wdelay,hide,nocrossmnt,secure,no_r
>>>>>  
>>>>>       
>>>>
>>>> oot_squash,no_all_squash,subtree_check,secure_locks,acl,mappi
>>>>     
>>>
>>> ng=identit
>>>   
>>>
>>>> y,ano
>>>> nuid=-2,anongid=-2)
>>>>
>>>>
>>>>     
>>>>
>>>>> Client running Fedora Core 2, kernel 2.6.8-1.521 [root@arvouin 
>>>>> /mnt/cobra3/mci/test/Test-sync] $cat /proc/mounts
>>>>> cobra3:/p2v5f1 /mnt/cobra3 nfs
>>>>> rw,v3,rsize=8192,wsize=8192,hard,tcp,lock,addr=cobra3 0 0 $time 
>>>>> tar xvfz /usr/src/redhat/SOURCES/httpd-2.0.51.tar.gz
>>>>> real    13m3.686s
>>>>> user    0m1.055s
>>>>> sys     0m4.354s
>>>>>
>>>>> 2) Export in async mode:
>>>>> Same NFS server, options for that export:
>>>>> $ cat /proc/fs/nfs/exports | grep arvouin
>>>>> /p2v5f1 arvouin.int-evry.fr(rw,no_root_squash,async,wdelay,acl) #
>>>>> 157.159.21.55
>>>>> $ cat /var/lib/nfs/xtab  | grep arvouin
>>>>> /p2v5f2
>>>>> arvouin.int-evry.fr(rw,async,wdelay,hide,nocrossmnt,secure,no_
>>>>>  
>>>>>       
>>>>
>>>> root_squash,no_all_squash,subtree_check,secure_locks,acl,mapp
>>>>     
>>>
>>> ing=identi
>>>   
>>>
>>>> ty,an
>>>> onuid=-2,anongid=-2)
>>>>
>>>>
>>>>     
>>>>
>>>>> Same client running Fedora Core 2, kernel 2.6.8-1.521
>>>>> cobra3:/p2v5f1 /mnt/cobra3 nfs
>>>>> rw,v3,rsize=8192,wsize=8192,hard,tcp,lock,addr=cobra3 0 0 
>>>>> [root@arvouin /mnt/cobra3/mci/test/Test-sync] $time tar xvfz 
>>>>> /usr/src/redhat/SOURCES/httpd-2.0.51.tar.gz
>>>>> real    0m14.802s
>>>>> user    0m0.867s
>>>>> sys     0m2.886s
>>>>>
>>>>> My users won't accept the sync performances ! . I have no       
>>>>
>>> choice, but   
>>>
>>>>> is running in async mode is really evil as you mentioned it       
>>>>
>>> ? is there   
>>>
>>>>> a way to have better performances in sync in my case ?  As       
>>>>
>>> anyone had   
>>>
>>>>> the same gap in performance as me ( here 55 times longer in       
>>>>
>>> sync mode   
>>>
>>>>> !)  ?
>>>>>
>>>>> Thanks.
>>>>>
>>>>>
>>>>> -------------------------------------------------------
>>>>> SF email is sponsored by - The IT Product Guide Read honest       
>>>>
>>> & candid   
>>>
>>>>> reviews on hundreds of IT Products from real users.
>>>>> Discover which products truly live up to the hype. Start       
>>>>
>>> reading now.   
>>>
>>>>> http://productguide.itmanagersjournal.com/
>>>>> _______________________________________________
>>>>> NFS maillist  -  NFS@lists.sourceforge.net 
>>>>> https://lists.sourceforge.net/lists/listinfo/nfs
>>>>>
>>>>>  
>>>>>       
>>>>
>>>>
>>>>
>>>>     
>>>
>>
>>  
>>
>
>
>
> -------------------------------------------------------
> SF email is sponsored by - The IT Product Guide
> Read honest & candid reviews on hundreds of IT Products from real users.
> Discover which products truly live up to the hype. Start reading now. 
> http://productguide.itmanagersjournal.com/
> _______________________________________________
> NFS maillist  -  NFS@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/nfs




-------------------------------------------------------
SF email is sponsored by - The IT Product Guide
Read honest & candid reviews on hundreds of IT Products from real users.
Discover which products truly live up to the hype. Start reading now. 
http://productguide.itmanagersjournal.com/
_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs

Re: async vs. sync

[Paul Cunningham]

Sorry, I must have missed the export option in a previous message. As I 
shake the cobwebs, I cannot remember an async export option used to increase 
performance with a warning from the man page "allows the NFS server to 
violate  the  NFS  protocol".

Sounds like a dangerous option.  Perhaps I'm too far removed from this now, 
but the applicable mount options were async and v3.  The server then should 
adhere to the mount request from the client regardless of an export option. 
This all assumes v3 is available on the server.  Scanning the man page on my 
various Linux boxes I cannot find a v3 mount option, the issue is becoming 
much clearer now!

So how does a client mount NFS v3 with Linux?

Paul Cunningham


----- Original Message ----- 
From: "Trond Myklebust" <trond.myklebust@fys.uio.no>
To: "Paul Cunningham" <paulcunningham@sprintmail.com>
Cc: <nfs@lists.sourceforge.net>
Sent: Monday, November 22, 2004 4:14 PM
Subject: Re: [NFS] async vs. sync


må den 22.11.2004 Klokka 16:04 (-0500) skreiv Paul Cunningham:

> It has been a few years, but I remember some of the async details.  I
> always
> used async for performance reasons, and much testing was performed to
> assure
> no data would be lost.  If a client sent an async write to the server,
> the
> call could return prior to data being flushed to disk.  The data would
> make
> it to disk once the server decided to write it or the client sends in
> a
> COMMIT.  At some point in time the client will attempt to close the
> file,
> this is when a COMMIT must be sent, the hope is that the server has
> already
> written the dirty pages to disk while the client was busy doing other
> things, and will respond with an OK quickly.  If any dirty pages
> remain,
> they must be flushed prior to responding OK.  Data should never be
> lost as
> long as the NFSPROC3_COMMIT procedure is adhered to.
>

Sure. This is how the Linux client works. The problem is the "async"
*export* option on the server.

-- 
Trond Myklebust <trond.myklebust@fys.uio.no>



-------------------------------------------------------
SF email is sponsored by - The IT Product Guide
Read honest & candid reviews on hundreds of IT Products from real users.
Discover which products truly live up to the hype. Start reading now.
http://productguide.itmanagersjournal.com/
_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs




-------------------------------------------------------
SF email is sponsored by - The IT Product Guide
Read honest & candid reviews on hundreds of IT Products from real users.
Discover which products truly live up to the hype. Start reading now. 
http://productguide.itmanagersjournal.com/
_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs

Re: async vs. sync

[Trond Myklebust]

[-- Attachment #1: Type: text/plain, Size: 1355 bytes --]

må den 22.11.2004 Klokka 22:52 (+0100) skreiv jehan procaccia:
> > [root@arvouin tmp]# mount cobra3:/p2v5f1 -o 
> > async,wsize=32768,rsize=32768,soft /mnt/cobra3
> > [root@arvouin /mnt/cobra3/mci/test/Test-sync]
> > $time tar xvfz /usr/src/redhat/SOURCES/httpd-2.0.51.tar.gz
> >
> > sorry I don't want to wait more than 10 minutes to send that mail, but 
> > again seeing the files apperaing very slowly on the tty it seems not 
> > to be the solution :-( .

The following script may help you understand why things are slower in
the case of "async" on an untar. It basically just creates a bunch of
files: in the first test it does not sync the directory to disk after
each file creation, in the second case it does. The test does no
reads/writes to the file.

Run it on the server and you will see  a clear difference in time
between test1 and test2. Run it on the client, and there should be
little difference between test1 and test2 (but there will be a heavy
dependency on the "async" vs "sync" export flag on the server).

NFSv3 mandates that all directory-related operations should behave as in
test 2. Only writes to ordinary files may be cached by the server, and
when the client sends a COMMIT request, the server should do an fsync()
on that file.

Cheers,
  Trond
-- 
Trond Myklebust <trond.myklebust@fys.uio.no>

[-- Attachment #2: script.sh --]
[-- Type: application/x-shellscript, Size: 447 bytes --]

Re: async vs. sync

[Trond Myklebust]

m=E5 den 22.11.2004 Klokka 17:07 (-0500) skreiv Paul Cunningham:

> So how does a client mount NFS v3 with Linux?
>=20

The default is to do just as you described: writes are asynchronous
using the caching model described in RFC1813 (NFSv3 unstable writes are
the default, and a COMMIT is sent on close and/or if memory pressure
forces the client to release the cached writes early).

There is also a "sync" mount option available that forces the client to
use NFSv3 stable writes.

The problem is the server side "async" option which basically causes
knfsd to ignore COMMIT requests, as well as causing it to fail to
fsync() the directory after file CREATE/LINK/UNLINK/RENAME/...
i.e. it turns of all consistency guarantess on the server.

Cheers,
  Trond

--=20
Trond Myklebust <trond.myklebust@fys.uio.no>



-------------------------------------------------------
SF email is sponsored by - The IT Product Guide
Read honest & candid reviews on hundreds of IT Products from real users.
Discover which products truly live up to the hype. Start reading now. 
http://productguide.itmanagersjournal.com/
_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs

Configuring NFS service for speed - Re: async vs. sync

[Neil Brown]

On Monday November 22, jehan.procaccia@int-evry.fr wrote:
> 
> My users won't accept the sync performances ! . I have no choice, but is 
> running in async mode is really evil as you mentioned it ? is there a 
> way to have better performances in sync in my case ?  As anyone had the 
> same gap in performance as me ( here 55 times longer in sync mode !)  ?

The best way to get good nfs performance with a sync export is to use
a data-journalling filesystem, and have the journal somewhere fast.

I use ext3.  I believe that some other filesystems support data
journalling, but I don't know the details.

An ext3 filesystem exported with "data=journal" is significantly
faster that with the default "data=ordered".

With ext3, it also helps to export with the "no_wdelay" export
option.  I'm not entirely sure why this helps, and I haven't confirmed
that it is still the case (I discovered this at least 2 years ago) but
it is worth checking in your environment.

Ext3 supports having the journal on a separate device.  If you put it
on a fast device, you will get faster response.

One option is to buy an NV-RAM card and put the journal on that.
www.umem.com is one supplier that I know of.  There are doubtlessly
others.

Another option is to put it on a separate drive, or preferably a
separate mirrored pair.  As journal writes are sequential, you won't
suffer seek time on the journal device and it will be quite fast.

My typical fileserver configuration is to have a pair of small (36gig
is the smallest I can easily get) 15K SCSI drives as a mirrored pair,
and to put root, swap, and a journal partition on this.
Then a separate box contains a dozen or so drives of whatever size I
want in a raid5 configuration.  The ext3 filesystem is on the raid5
array, the journal is on the mirrored pair with root and swap.  There
is almost no IO to root and swap, so the journal gets the drives
almost to itself.
This configuration is adequately fast.

NeilBrown


-------------------------------------------------------
SF email is sponsored by - The IT Product Guide
Read honest & candid reviews on hundreds of IT Products from real users.
Discover which products truly live up to the hype. Start reading now. 
http://productguide.itmanagersjournal.com/
_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs

Re: async vs. sync

[jehan procaccia]

Trond Myklebust wrote:

>m=E5 den 22.11.2004 Klokka 22:52 (+0100) skreiv jehan procaccia:
> =20
>
>>>[root@arvouin tmp]# mount cobra3:/p2v5f1 -o=20
>>>async,wsize=3D32768,rsize=3D32768,soft /mnt/cobra3
>>>[root@arvouin /mnt/cobra3/mci/test/Test-sync]
>>>$time tar xvfz /usr/src/redhat/SOURCES/httpd-2.0.51.tar.gz
>>>
>>>sorry I don't want to wait more than 10 minutes to send that mail, but=
=20
>>>again seeing the files apperaing very slowly on the tty it seems not=20
>>>to be the solution :-( .
>>>     =20
>>>
>
>The following script may help you understand why things are slower in
>the case of "async" on an untar. It basically just creates a bunch of
>files: in the first test it does not sync the directory to disk after
>each file creation, in the second case it does. The test does no
>reads/writes to the file.
>
>Run it on the server and you will see  a clear difference in time
>between test1 and test2. Run it on the client, and there should be
>little difference between test1 and test2 (but there will be a heavy
>dependency on the "async" vs "sync" export flag on the server).
> =20
>
These prediction are perfect, this is exaclty  what happened:
I reduce the loop from 1000 to 100, it was too long on the client in=20
sync mode ....

[root@cobra3 /p2v5f1/mci/test/Test-sync]
$ ./test2.sh
Test without directory sync after file creation

real    0m0.037s
user    0m0.010s
sys     0m0.000s
Test2 with directory sync after file creation

real    0m6.040s
user    0m0.000s
sys     0m0.000s

NFS client, while server export in sync mode
cobra3:/p2v5f1 /mnt/cobra3 nfs=20
rw,v3,rsize=3D8192,wsize=3D8192,soft,tcp,lock,addr=3Dcobra3 0 0
$./test2.sh
Test without directory sync after file creation

real    0m31.144s
user    0m0.042s
sys     0m0.373s
Test2 with directory sync after file creation

real    0m49.030s
user    0m0.073s
sys     0m0.694s

Now NFS server exports in async mode to the client, performances are far=20
better , even better than direclty on the server when forcing a sync=20
call after every creation !
$./test2.sh
Test without directory sync after file creation

real    0m0.446s
user    0m0.026s
sys     0m0.078s
Test2 with directory sync after file creation

real    0m0.785s
user    0m0.076s
sys     0m0.305s

Hopefully I'll try to capture traffic next wednesday when I'll be back=20
at work ..

Thanks.

>NFSv3 mandates that all directory-related operations should behave as in
>test 2. Only writes to ordinary files may be cached by the server, and
>when the client sends a COMMIT request, the server should do an fsync()
>on that file.
>
>Cheers,
>  Trond
> =20
>



-------------------------------------------------------
SF email is sponsored by - The IT Product Guide
Read honest & candid reviews on hundreds of IT Products from real users.
Discover which products truly live up to the hype. Start reading now. 
http://productguide.itmanagersjournal.com/
_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs

Re: Configuring NFS service for speed - Re: async vs. sync

[jehan procaccia]

Neil Brown wrote:

>On Monday November 22, jehan.procaccia@int-evry.fr wrote:
>  
>
>>My users won't accept the sync performances ! . I have no choice, but is 
>>running in async mode is really evil as you mentioned it ? is there a 
>>way to have better performances in sync in my case ?  As anyone had the 
>>same gap in performance as me ( here 55 times longer in sync mode !)  ?
>>    
>>
>
>The best way to get good nfs performance with a sync export is to use
>a data-journalling filesystem, and have the journal somewhere fast.
>
>I use ext3.  I believe that some other filesystems support data
>journalling, but I don't know the details.
>
>An ext3 filesystem exported with "data=journal" is significantly
>faster that with the default "data=ordered".
>
>With ext3, it also helps to export with the "no_wdelay" export
>option.  I'm not entirely sure why this helps, and I haven't confirmed
>that it is still the case (I discovered this at least 2 years ago) but
>it is worth checking in your environment.
>  
>
OK, I've just tested with a FS mounted with data=journal in /etc/fstab 
(but journal isn't somewhere else ..) and no_wdelay in exports

NFS server FS and exports:
[root@cobra3 ]
$ grep emc /var/lib/nfs/xtab
/mnt/emcpowerl1 
arvouin.int-evry.fr(rw,sync,no_wdelay,hide,nocrossmnt,secure,no_root_squash,no_all_squash,subtree_check,secure_locks,no_acl,mapping=identity,anonuid=-2,anongid=-2)
$ grep emcpowerl1 /etc/fstab
/dev/emcpowerl1         /mnt/emcpowerl1         ext3    
defaults,data=journal,usrquota,grpquota 1 2

NFS client:
cobra3:/mnt/emcpowerl1 /mnt/cobra3journal nfs 
rw,v3,rsize=8192,wsize=8192,hard,tcp,lock,addr=cobra3 0 0
Trond Myklebust 's test script apparently goes faster :-)
[root@arvouin /mnt/cobra3journal/test/Nfs]
$./test2.sh
Test without directory sync after file creation

real    0m12.243s
user    0m0.016s
sys     0m0.094s
Test2 with directory sync after file creation

real    0m12.195s
user    0m0.090s
sys     0m0.342s


It took 31s for test1 and 49s for test2 before data=journal and 
no_wdelay, now it takes 12s in both cases :-) ,  finally, that's my 
first performance enhencement tonight !.

However tar xvfz is still very long :-(
[root@arvouin /mnt/cobra3journal/test/Nfs]
$time tar xvfz /usr/src/redhat/SOURCES/httpd-2.0.51.tar.gz
real    12m5.968s
user    0m1.026s
sys     0m4.362s

I still miss something ...

>Ext3 supports having the journal on a separate device.  If you put it
>on a fast device, you will get faster response.
>
>One option is to buy an NV-RAM card and put the journal on that.
>www.umem.com is one supplier that I know of.  There are doubtlessly
>others.
>
>Another option is to put it on a separate drive, or preferably a
>separate mirrored pair.  As journal writes are sequential, you won't
>suffer seek time on the journal device and it will be quite fast.
>
>My typical fileserver configuration is to have a pair of small (36gig
>is the smallest I can easily get) 15K SCSI drives as a mirrored pair,
>and to put root, swap, and a journal partition on this.
>Then a separate box contains a dozen or so drives of whatever size I
>want in a raid5 configuration.  The ext3 filesystem is on the raid5
>array, the journal is on the mirrored pair with root and swap.  There
>is almost no IO to root and swap, so the journal gets the drives
>almost to itself.
>This configuration is adequately fast.
>
>NeilBrown
>  
>



-------------------------------------------------------
SF email is sponsored by - The IT Product Guide
Read honest & candid reviews on hundreds of IT Products from real users.
Discover which products truly live up to the hype. Start reading now. 
http://productguide.itmanagersjournal.com/
_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs

Re: async vs. sync

[jehan procaccia]

I found something interesting ! If I export an internal scsi disk of my=20
NFS server (/home) I get good performances,  My problem is with exports=20
from the Dell/EMC Storage Processor , I will open a ticket about that on=20
the hotline ...

Nfs servor exporting an Internal scsi disk in sync mode
$ cat /var/lib/nfs/xtab
/home  =20
arvouin.int-evry.fr(rw,sync,no_wdelay,hide,nocrossmnt,secure,no_root_squa=
sh,no_all_squash,subtree_check,secure_locks,no_acl,mapping=3Didentity,ano=
nuid=3D-2,anongid=3D-2)

Client mount in sync an NFS export in sync also !:
cobra3:/home /mnt/cobra3home nfs=20
rw,sync,v3,rsize=3D8192,wsize=3D8192,hard,tcp,lock,addr=3Dcobra3 0 0
[root@arvouin /mnt/cobra3home/Nfs-test]
$time tar xvfz /usr/src/redhat/SOURCES/httpd-2.0.51.tar.gz
real    0m27.719s
user    0m1.002s
sys     0m4.296s

[root@arvouin /mnt/cobra3home/Nfs-test]
$./test2.sh
Test without directory sync after file creation
=20
real    0m0.568s
user    0m0.031s
sys     0m0.114s
Test2 with directory sync after file creation
=20
real    0m0.904s
user    0m0.094s
sys     0m0.266s


This is what I expect in term of performances . I will continue my=20
requests on the DEll/EMC hotline , but maybe the security of that AX100=20
storage Processor (raid5, spare disk, double fiber attachement, UPS)=20
allows me to use async export mode in such a case ?

thanks .

jehan procaccia wrote:

> Trond Myklebust wrote:
>
>> m=E5 den 22.11.2004 Klokka 22:52 (+0100) skreiv jehan procaccia:
>> =20
>>
>>>> [root@arvouin tmp]# mount cobra3:/p2v5f1 -o=20
>>>> async,wsize=3D32768,rsize=3D32768,soft /mnt/cobra3
>>>> [root@arvouin /mnt/cobra3/mci/test/Test-sync]
>>>> $time tar xvfz /usr/src/redhat/SOURCES/httpd-2.0.51.tar.gz
>>>>
>>>> sorry I don't want to wait more than 10 minutes to send that mail,=20
>>>> but again seeing the files apperaing very slowly on the tty it=20
>>>> seems not to be the solution :-( .
>>>>    =20
>>>
>>
>> The following script may help you understand why things are slower in
>> the case of "async" on an untar. It basically just creates a bunch of
>> files: in the first test it does not sync the directory to disk after
>> each file creation, in the second case it does. The test does no
>> reads/writes to the file.
>>
>> Run it on the server and you will see  a clear difference in time
>> between test1 and test2. Run it on the client, and there should be
>> little difference between test1 and test2 (but there will be a heavy
>> dependency on the "async" vs "sync" export flag on the server).
>> =20
>>
> These prediction are perfect, this is exaclty  what happened:
> I reduce the loop from 1000 to 100, it was too long on the client in=20
> sync mode ....
>
> [root@cobra3 /p2v5f1/mci/test/Test-sync]
> $ ./test2.sh
> Test without directory sync after file creation
>
> real    0m0.037s
> user    0m0.010s
> sys     0m0.000s
> Test2 with directory sync after file creation
>
> real    0m6.040s
> user    0m0.000s
> sys     0m0.000s
>
> NFS client, while server export in sync mode
> cobra3:/p2v5f1 /mnt/cobra3 nfs=20
> rw,v3,rsize=3D8192,wsize=3D8192,soft,tcp,lock,addr=3Dcobra3 0 0
> $./test2.sh
> Test without directory sync after file creation
>
> real    0m31.144s
> user    0m0.042s
> sys     0m0.373s
> Test2 with directory sync after file creation
>
> real    0m49.030s
> user    0m0.073s
> sys     0m0.694s
>
> Now NFS server exports in async mode to the client, performances are=20
> far better , even better than direclty on the server when forcing a=20
> sync call after every creation !
> $./test2.sh
> Test without directory sync after file creation
>
> real    0m0.446s
> user    0m0.026s
> sys     0m0.078s
> Test2 with directory sync after file creation
>
> real    0m0.785s
> user    0m0.076s
> sys     0m0.305s
>
> Hopefully I'll try to capture traffic next wednesday when I'll be back=20
> at work ..
>
> Thanks.
>
>> NFSv3 mandates that all directory-related operations should behave as =
in
>> test 2. Only writes to ordinary files may be cached by the server, and
>> when the client sends a COMMIT request, the server should do an fsync(=
)
>> on that file.
>>
>> Cheers,
>>  Trond
>> =20
>>
>
>
>
> -------------------------------------------------------
> SF email is sponsored by - The IT Product Guide
> Read honest & candid reviews on hundreds of IT Products from real users=
.
> Discover which products truly live up to the hype. Start reading now.=20
> http://productguide.itmanagersjournal.com/
> _______________________________________________
> NFS maillist  -  NFS@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/nf
> s




-------------------------------------------------------
SF email is sponsored by - The IT Product Guide
Read honest & candid reviews on hundreds of IT Products from real users.
Discover which products truly live up to the hype. Start reading now. 
http://productguide.itmanagersjournal.com/
_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs

Re: async vs. sync

[J. Bruce Fields]

On Tue, Nov 23, 2004 at 10:50:55AM +0100, jehan procaccia wrote:
> This is what I expect in term of performances . I will continue my 
> requests on the DEll/EMC hotline , but maybe the security of that AX100 
> storage Processor (raid5, spare disk, double fiber attachement, UPS) 
> allows me to use async export mode in such a case ?

No.  The problem with the async export option is that it allows the
server to tell the client that data is safely committed to disk when it
hasn't actually been.  So the risk is that the nfs server will fail
while valuable data is still sitting in its memory.  It's the server
itself that's the weak point, not the storage.

--Bruce Fields


-------------------------------------------------------
SF email is sponsored by - The IT Product Guide
Read honest & candid reviews on hundreds of IT Products from real users.
Discover which products truly live up to the hype. Start reading now. 
http://productguide.itmanagersjournal.com/
_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs