Performance: NFS v3 mounting with 'noac' option

Performance: NFS v3 mounting with 'noac' option

[AK]

Hi All,

Trying to understand what performance impact could
result if we mount the file system with 'noac' option.

We have multiple file systems begin mounted from the
same NFS server. One of our user's application
requires to mount the file system with 'noac' option. 

We have user directories under /scratch file system,
Now we would like to mount this file system with
'noac' option so that we can make one of our user
happy. But this would afftect every user. Isn't it
true?

So Now if we just decide to create a new FS
/newScratchUserA and mount it with 'noac' option;
WILL THIS AFFECT the other files systems mounted from
the same server, if so how bad could it be. 

Are there any other alternatives to reduce the impact
of using this option on other mounted File systems.

really appreciate any advise .... 

~ thanks!


 
____________________________________________________________________________________
Never miss an email again!
Yahoo! Toolbar alerts you the instant new Mail arrives.
http://tools.search.yahoo.com/toolbar/features/mail/

-------------------------------------------------------------------------
Take Surveys. Earn Cash. Influence the Future of IT
Join SourceForge.net's Techsay panel and you'll get the chance to share your
opinions on IT & business topics through brief surveys-and earn cash
http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV
_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs

Re: Performance: NFS v3 mounting with 'noac' option

[Peter Staubach]

AK wrote:
> Hi All,
>
> Trying to understand what performance impact could
> result if we mount the file system with 'noac' option.
>
> We have multiple file systems begin mounted from the
> same NFS server. One of our user's application
> requires to mount the file system with 'noac' option. 
>
>   

What about this application requires that the "noac" option be used?

> We have user directories under /scratch file system,
> Now we would like to mount this file system with
> 'noac' option so that we can make one of our user
> happy. But this would afftect every user. Isn't it
> true?
>
>   

Well, you could try mounting each user's home directory individually.
The "noac" option should only apply to a specific mounted instance.

> So Now if we just decide to create a new FS
> /newScratchUserA and mount it with 'noac' option;
> WILL THIS AFFECT the other files systems mounted from
> the same server, if so how bad could it be. 
>
> Are there any other alternatives to reduce the impact
> of using this option on other mounted File systems.

One of the impacts of using the "noac" option is much more network
traffic and it will be composed of small packets.  If the server is
busy handling all of the additional GETATTR requests, then it won't
be handling other requests.  The client will also be a bit busier
with generating the additional requests, instead of just doing what
might be considered more useful work.

So, there should be no direct impact to other mounted file systems.
However, there may be indirect impacts.

       ps

-------------------------------------------------------------------------
Take Surveys. Earn Cash. Influence the Future of IT
Join SourceForge.net's Techsay panel and you'll get the chance to share your
opinions on IT & business topics through brief surveys-and earn cash
http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV
_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs

Re: Performance: NFS v3 mounting with 'noac' option

[AK]

--- Peter Staubach <staubach@redhat.com> wrote:

> > We have multiple file systems begin mounted from
> the
> > same NFS server. One of our user's application
> > requires to mount the file system with 'noac'
> option. 
> >
> >   
> 
> What about this application requires that the "noac"
> option be used?
  application is romio with mpich2 ...
 
> 
> One of the impacts of using the "noac" option is
> much more network
> traffic and it will be composed of small packets. 
> If the server is
> busy handling all of the additional GETATTR
> requests, then it won't
> be handling other requests.  The client will also be
> a bit busier
> with generating the additional requests, instead of
> just doing what
> might be considered more useful work.
> 
> So, there should be no direct impact to other
> mounted file systems.
> However, there may be indirect impacts.

indirect impacts: can you please elaborate on this.

~ thanks!


 
____________________________________________________________________________________
It's here! Your new message!  
Get new email alerts with the free Yahoo! Toolbar.
http://tools.search.yahoo.com/toolbar/features/mail/

-------------------------------------------------------------------------
Take Surveys. Earn Cash. Influence the Future of IT
Join SourceForge.net's Techsay panel and you'll get the chance to share your
opinions on IT & business topics through brief surveys-and earn cash
http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV
_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs

Re: Performance: NFS v3 mounting with 'noac' option

[Peter Staubach]

AK wrote:
> --- Peter Staubach <staubach@redhat.com> wrote:
>
>   
>>> We have multiple file systems begin mounted from
>>>       
>> the
>>     
>>> same NFS server. One of our user's application
>>> requires to mount the file system with 'noac'
>>>       
>> option. 
>>     
>>>   
>>>       
>> What about this application requires that the "noac"
>> option be used?
>>     
>   application is romio with mpich2 ...
>  
>   

Sorry, I'm not familiar with those.  What about either one of those
require "noac" semantics?

>> One of the impacts of using the "noac" option is
>> much more network
>> traffic and it will be composed of small packets. 
>> If the server is
>> busy handling all of the additional GETATTR
>> requests, then it won't
>> be handling other requests.  The client will also be
>> a bit busier
>> with generating the additional requests, instead of
>> just doing what
>> might be considered more useful work.
>>
>> So, there should be no direct impact to other
>> mounted file systems.
>> However, there may be indirect impacts.
>>     
>
> indirect impacts: can you please elaborate on this.

I thought that I did.  :-)

If the system is busy handling the "noac" aspects of one file system,
then it isn't doing something else on one of the other file systems.
System resources are limited and using up more of them for one thing
leaves less for others.

       ps

-------------------------------------------------------------------------
Take Surveys. Earn Cash. Influence the Future of IT
Join SourceForge.net's Techsay panel and you'll get the chance to share your
opinions on IT & business topics through brief surveys-and earn cash
http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV
_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs

Re: Performance: NFS v3 mounting with 'noac' option

[AK]

--- Peter Staubach <staubach@redhat.com> wrote:

> >>>       
> >> What about this application requires that the
> "noac"
> >> option be used?
> >>     
> >   application is romio with mpich2 ...
> 
> Sorry, I'm not familiar with those.  What about
> either one of those
> require "noac" semantics?

"To use ROMIO on NFS, file locking with fcntl must
work correctly on the NFS installation. On some
installations, fcntl locks don't work. To get them to
work, you need to use Version 3 of NFS, ensure that
the lockd daemon is running on all the machines, and
have the system administrator mount the NFS file
system with the "noac" option (no attribute caching).
Turning off attribute caching may reduce performance,
but it is necessary for correct behavior."

> 
> >> One of the impacts of using the "noac" option is
> >> much more network
> >> traffic and it will be composed of small packets.
> 
> >> If the server is
> >> busy handling all of the additional GETATTR
> >> requests, then it won't
> >> be handling other requests.  The client will also
> be
> >> a bit busier
> >> with generating the additional requests, instead
> of
> >> just doing what
> >> might be considered more useful work.
> >>
> >> So, there should be no direct impact to other
> >> mounted file systems.
> >> However, there may be indirect impacts.
> >>     
> >
> > indirect impacts: can you please elaborate on
> this.
> 
> I thought that I did.  :-)
> 
> If the system is busy handling the "noac" aspects of
> one file system,
> then it isn't doing something else on one of the
> other file systems.
> System resources are limited and using up more of
> them for one thing
> leaves less for others.

Got it!!! when you mentioned "However, there may be
indirect impacts." in the last line, i was like there
could be more, and hence curious :-) 

> 
>        ps
> 



 
____________________________________________________________________________________
The fish are biting. 
Get more visitors on your site using Yahoo! Search Marketing.
http://searchmarketing.yahoo.com/arp/sponsoredsearch_v2.php

-------------------------------------------------------------------------
Take Surveys. Earn Cash. Influence the Future of IT
Join SourceForge.net's Techsay panel and you'll get the chance to share your
opinions on IT & business topics through brief surveys-and earn cash
http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV
_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs

Re: Performance: NFS v3 mounting with 'noac' option

[Peter Staubach]

AK wrote:
> --- Peter Staubach <staubach@redhat.com> wrote:
>
>   
>>>>>       
>>>>>           
>>>> What about this application requires that the
>>>>         
>> "noac"
>>     
>>>> option be used?
>>>>     
>>>>         
>>>   application is romio with mpich2 ...
>>>       
>> Sorry, I'm not familiar with those.  What about
>> either one of those
>> require "noac" semantics?
>>     
>
> "To use ROMIO on NFS, file locking with fcntl must
> work correctly on the NFS installation. On some
> installations, fcntl locks don't work. To get them to
> work, you need to use Version 3 of NFS, ensure that
> the lockd daemon is running on all the machines, and
> have the system administrator mount the NFS file
> system with the "noac" option (no attribute caching).
> Turning off attribute caching may reduce performance,
> but it is necessary for correct behavior."

I think that I would assume that fcntl style locking works on all current
implementations, that this is not NFS version dependent, and also that
"noac" is not required in combination with the locking in order to get
correct behavior.

       ps

-------------------------------------------------------------------------
Take Surveys. Earn Cash. Influence the Future of IT
Join SourceForge.net's Techsay panel and you'll get the chance to share your
opinions on IT & business topics through brief surveys-and earn cash
http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV
_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs

Re: Performance: NFS v3 mounting with 'noac' option

[Talpey, Thomas]

At 11:13 AM 3/8/2007, Peter Staubach wrote:
>AK wrote:
>> "To use ROMIO on NFS, file locking with fcntl must
>> work correctly on the NFS installation. On some
>> installations, fcntl locks don't work. To get them to
>> work, you need to use Version 3 of NFS, ensure that
>> the lockd daemon is running on all the machines, and
>> have the system administrator mount the NFS file
>> system with the "noac" option (no attribute caching).
>> Turning off attribute caching may reduce performance,
>> but it is necessary for correct behavior."
>
>I think that I would assume that fcntl style locking works on all current
>implementations, that this is not NFS version dependent, and also that
>"noac" is not required in combination with the locking in order to get
>correct behavior.

Ouch - I agree with Peter, this advice is way, way out of date.
<http://www-unix.mcs.anl.gov/romio/>
<http://www-unix.mcs.anl.gov/romio/nfs-instr.txt>

>"The following are some instructions we received from Ian Wells of HP
>for setting the noac option on NFS. We have not tried them
>ourselves. We are including them here because you may find 
>them useful. Note that some of the steps may be specific to HP
>systems, and ..."

Leave noac alone, the Linux client will do the right thing with
its cache when locking is available.

Tom.


-------------------------------------------------------------------------
Take Surveys. Earn Cash. Influence the Future of IT
Join SourceForge.net's Techsay panel and you'll get the chance to share your
opinions on IT & business topics through brief surveys-and earn cash
http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV
_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs

Re: Performance: NFS v3 mounting with 'noac' option

[AK]

Hmmm..
Tom & Peter, thank you for feedbacks ...

I agree, but when I read through the web and other
doucmentation ... I found that if NFS's local caching
is enable, doing a parallel IO on a file is a problem
some, because there is no global synchronization.
Please correct me if I am wrong. I believe GFS works
that way and we don't need to turn off caching there. 

With the current setup when the user runs the MPI I/O
program, he runs into following errors.

File locking failed in ADIOI_Set_lock. If the file
system is NFS, you need to use NFS version 3, ensure
that the lockd daemon is running on all the machines,
and mount the directory with the 'noac' option (no
attribute caching).

Is there an alternative to this ?? One more thing
since we have nfs on our rocks cluster, the nfsd,
rpc.statd etc are not started on the compute nodes
...will starting this help the file locking ??

~thank you!


--- "Talpey, Thomas" <Thomas.Talpey@netapp.com> wrote:

> At 11:13 AM 3/8/2007, Peter Staubach wrote:
> >AK wrote:
> >> "To use ROMIO on NFS, file locking with fcntl
> must
> >> work correctly on the NFS installation. On some
> >> installations, fcntl locks don't work. To get
> them to
> >> work, you need to use Version 3 of NFS, ensure
> that
> >> the lockd daemon is running on all the machines,
> and
> >> have the system administrator mount the NFS file
> >> system with the "noac" option (no attribute
> caching).
> >> Turning off attribute caching may reduce
> performance,
> >> but it is necessary for correct behavior."
> >
> >I think that I would assume that fcntl style
> locking works on all current
> >implementations, that this is not NFS version
> dependent, and also that
> >"noac" is not required in combination with the
> locking in order to get
> >correct behavior.
> 
> Ouch - I agree with Peter, this advice is way, way
> out of date.
> <http://www-unix.mcs.anl.gov/romio/>
> <http://www-unix.mcs.anl.gov/romio/nfs-instr.txt>
> 
> >"The following are some instructions we received
> from Ian Wells of HP
> >for setting the noac option on NFS. We have not
> tried them
> >ourselves. We are including them here because you
> may find 
> >them useful. Note that some of the steps may be
> specific to HP
> >systems, and ..."
> 
> Leave noac alone, the Linux client will do the right
> thing with
> its cache when locking is available.
> 
> Tom.
> 
> 



 
____________________________________________________________________________________
Now that's room service!  Choose from over 150,000 hotels
in 45,000 destinations on Yahoo! Travel to find your fit.
http://farechase.yahoo.com/promo-generic-14795097

-------------------------------------------------------------------------
Take Surveys. Earn Cash. Influence the Future of IT
Join SourceForge.net's Techsay panel and you'll get the chance to share your
opinions on IT & business topics through brief surveys-and earn cash
http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV
_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs

Re: Performance: NFS v3 mounting with 'noac' option

[Talpey, Thomas]

At 11:52 AM 3/8/2007, AK wrote:
>Hmmm..
>Tom & Peter, thank you for feedbacks ...
>
>I agree, but when I read through the web and other
>doucmentation ... I found that if NFS's local caching
>is enable, doing a parallel IO on a file is a problem
>some, because there is no global synchronization.

I'm sure Peter will jump in too - the point is you *do* have global
synchronization, from the locks. And in addition, modern clients
know that locking requires them to treat their caches differently,
and they handle these cases.

This did not used to be the case in all NFS clients, but any Linux
client has for quite some time handled this correctly. The only
reason to use noac today is to force clients to do no caching
when they are performing shared writes and are incapable of,
or refuse to do, locking. And even then, it's risky in the extreme.
In addition to being slow.

>Please correct me if I am wrong. I believe GFS works
>that way and we don't need to turn off caching there. 
>
>With the current setup when the user runs the MPI I/O
>program, he runs into following errors.
>
>File locking failed in ADIOI_Set_lock. If the file
>system is NFS, you need to use NFS version 3, ensure
>that the lockd daemon is running on all the machines,
>and mount the directory with the 'noac' option (no
>attribute caching).

If the app is failing to take a lock, yes fix it. But don't take
the second part of the advice.

Remind us again - what client are you using? Some of them do
not start the client's locking automatically. Perhaps that's the
case with yours.

>
>Is there an alternative to this ?? One more thing
>since we have nfs on our rocks cluster, the nfsd,
>rpc.statd etc are not started on the compute nodes
>...will starting this help the file locking ??

You do not want or need the nfsd on the compute nodes. But you
do need the rpc.statd. If it wasn't started automatically, then either
you are running a version that has it integrated in the kernel, or you
have to perform additional client setup. Let us know that client
version...

Tom.

>
>~thank you!
>
>
>--- "Talpey, Thomas" <Thomas.Talpey@netapp.com> wrote:
>
>> At 11:13 AM 3/8/2007, Peter Staubach wrote:
>> >AK wrote:
>> >> "To use ROMIO on NFS, file locking with fcntl
>> must
>> >> work correctly on the NFS installation. On some
>> >> installations, fcntl locks don't work. To get
>> them to
>> >> work, you need to use Version 3 of NFS, ensure
>> that
>> >> the lockd daemon is running on all the machines,
>> and
>> >> have the system administrator mount the NFS file
>> >> system with the "noac" option (no attribute
>> caching).
>> >> Turning off attribute caching may reduce
>> performance,
>> >> but it is necessary for correct behavior."
>> >
>> >I think that I would assume that fcntl style
>> locking works on all current
>> >implementations, that this is not NFS version
>> dependent, and also that
>> >"noac" is not required in combination with the
>> locking in order to get
>> >correct behavior.
>> 
>> Ouch - I agree with Peter, this advice is way, way
>> out of date.
>> <http://www-unix.mcs.anl.gov/romio/>
>> <http://www-unix.mcs.anl.gov/romio/nfs-instr.txt>
>> 
>> >"The following are some instructions we received
>> from Ian Wells of HP
>> >for setting the noac option on NFS. We have not
>> tried them
>> >ourselves. We are including them here because you
>> may find 
>> >them useful. Note that some of the steps may be
>> specific to HP
>> >systems, and ..."
>> 
>> Leave noac alone, the Linux client will do the right
>> thing with
>> its cache when locking is available.
>> 
>> Tom.
>> 
>> 
>
>
>
> 
>______________________________________________________________________
>______________
>Now that's room service!  Choose from over 150,000 hotels
>in 45,000 destinations on Yahoo! Travel to find your fit.
>http://farechase.yahoo.com/promo-generic-14795097


-------------------------------------------------------------------------
Take Surveys. Earn Cash. Influence the Future of IT
Join SourceForge.net's Techsay panel and you'll get the chance to share your
opinions on IT & business topics through brief surveys-and earn cash
http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV
_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs