RE: [NFS] Disabling Symbolic Link Content Caching in NFS Client

RE: [NFS] Disabling Symbolic Link Content Caching in NFS Client

[Edward Hibbert]

-----Original Message-----
From: Vivek Goyal [mailto:vivek.goyal@wipro.com]
Sent: Tuesday, June 03, 2003 2:21 PM
To: trond.myklebust@fys.uio.no; Ion Badulescu
Cc: viro@math.psu.edu; davem@redhat.com; ezk@cs.sunysb.edu;
indou.takao@jp.fujitsu.com; nfs@lists.sourceforge.net;
linux-kernel@vger.kernel.org; Vivek Goyal
Subject: RE: [NFS] Disabling Symbolic Link Content Caching in NFS Client

<snip>

You are right. But our idea is to provide an option to disable/enable
caching based on the nature of intended application.

[EH] Hear hear :-).  

Our application consists of a number of machines collaborating on a shared
database over NFS.  We therefore require the ability to force data to be
sync'd from the client to the backend - and at the moment we do this by
disabling caching completely, via the noac option and acquiring and
releasing non-exclusive locks round io calls.

Any improvements in the granularity of control over NFS client-side caching
would be very valuable to us.

Regards,

Edward Hibbert
Internet Applications Group
Data Connection Ltd
Tel:	+44 131 662 1212		Fax:	+44 131 662 1345
Email:	eh@dataconnection.com	Web:	http://www.dataconnection.com

Re: [NFS] Disabling Symbolic Link Content Caching in NFS Client

[Frank van Maarseveen]

On Tue, Jun 03, 2003 at 02:33:45PM +0100, Edward Hibbert wrote:
> 
> Our application consists of a number of machines collaborating on a shared
> database over NFS.  We therefore require the ability to force data to be
> sync'd from the client to the backend - and at the moment we do this by
> disabling caching completely, via the noac option and acquiring and
> releasing non-exclusive locks round io calls.
> 
> Any improvements in the granularity of control over NFS client-side caching
> would be very valuable to us.

To disable ac for a single directory for a certain amount of time, see

	http://web.inter.nl.net/users/fvm/nfs-noac/2.4.20-noac-timeout.patch
	http://web.inter.nl.net/users/fvm/nfs-noac/readme:

This patch implements /proc/sys/net/nfs/noac-timeout

When a nonzero value is written, suspend atribute caching for the current
working directory and one level of files inside for the specified number
of seconds. Attribute caching will automatically be enabled when the time
elapses. Writing a zero re-enables attribute caching as well. Reading
yields the number of remaining seconds attribute caching will be disabled.


Maybe an NFS specific ioctl would be cleaner. But the above patch can be
used even from simple scripts.

-- 
Frank

Re: [NFS] Disabling Symbolic Link Content Caching in NFS Client

[Trond Myklebust]

>>>>> " " == Edward Hibbert <EH@dataconnection.com> writes:

     > Our application consists of a number of machines collaborating
     > on a shared database over NFS.  We therefore require the
     > ability to force data to be sync'd from the client to the
     > backend - and at the moment we do this by disabling caching
     > completely, via the noac option and acquiring and releasing
     > non-exclusive locks round io calls.

What does this have to do with symlinks?

...and why can't you use DIRECTIO? The above sort of application is
exactly what it is being introduced for.

Cheers,
  Trond

RE: [NFS] Disabling Symbolic Link Content Caching in NFS Client

[Vivek Goyal]

> >>>>> " " == Ion Badulescu <ionut@badula.org> writes:
>     >> 1. Make nfs_symlink_caching dynamically tunable using /proc and
>     >> sysctl interface.
>      > No. Do it on a per-mount basis, like the other OS's do.
> 
> As I said to Vivek in a private mail, it would be very nice to see
> if this could be done by replacing hlfsd with namespace groups.
> 
> Al Viro has already done all the VFS layer work, which should be ready
> and working in existing 2.4.20 and 2.5.x kernels. What is missing is
> userland support for doing a CLONE_NEWNS, and then mounting the user's
> home directory, mailspool,.... in the appropriate locations at login
> time.

Yes, this could be a better way to tackle hlfsd issues.

<snip from Trond's mail>

>I'm not overly keen on this proposal. IMHO we want to *maximize*
caching >rather than reduce it. This is particularly important as the
speed of >networks etc. starts to approach that of the PCI bus on the
server.

<end of snip>

You are right. But our idea is to provide an option to disable/enable
caching based on the nature of intended application.

With this, Linux will have enhanced flexibility in NFS client like other
operating systems. For example.

1. Solaris provides dynamically tunable parameter "nfs_do_symlink_cache"
for enabling/disabling symlink caching.

2. UNIX and IRIX provide 'symttl' mount option for enabling/disabling
symbolic link content caching in NFS client.

Kindly let me know your comments.

Regards,
Vivek

RE: [NFS] Disabling Symbolic Link Content Caching in NFS Client

[Trond Myklebust]

>>>>> " " == Vivek Goyal <vivek.goyal@wipro.com> writes:

     > You are right. But our idea is to provide an option to
     > disable/enable caching based on the nature of intended
     > application.

Apart from hlfsd, which applications actually depend on symlinks not
being cached?

     > With this, Linux will have enhanced flexibility in NFS client
     > like other operating systems. For example.

Again: Do you have any concrete examples of programs that need this
functionality, and that cannot be replaced?

Cheers,
  Trond

Disabling Symbolic Link Content Caching in NFS Client

[Vivek Goyal]

Hi,

We are planning to implement a set of Kernel Tunable Parameters and one
of these parameters is nfs_symlink_caching for disabling/enabling
symbolic link content caching in NFS client. Unices like Solaris have
this feature implemented.

Existing NFS client implementation in Linux performs symbolic link
content caching by default. There is no provision for disabling this
caching either at mount time or through kernel tuning mechanism.

This caching mechanism leads to problems in following two conditions.

a. File is modified in server without updating the modification
time-stamp.
b. Granularity of the time-stamp is too large.

I was following previous discussions in the mailing list and found that
caching mechanism affected hlfsd behavior. It looks like the problem was
resolved by updating mtime on every access. But still caching will lead
to a problem if two accesses are taking place in same jiffy.

We are considering following design strategy to implement the parameter.

1. Make nfs_symlink_caching dynamically tunable using /proc and sysctl
interface.

2. Change NFS client code as follows.

A. If caching is enabled.
There are no changes. Existing caching mechanism remains intact.

B. If caching is disabled.

	i. While serving a symlink read request (nfs_getlink ()) don't
look for the requested page in page cache (read_cache_page()). Instead
always allocate a page frame (page_cache_alloc_cold()) and send a
READLINK request to the server. This allocated page frame is not added
to the cache and hence remains local to the thread of execution and is
not visible to other execution threads.

	ii. Upon receiving the response from the server, pass the data
to the user space and free the page frame. 

Please let us know your comments/opinion on this design.

Thanks and Regards
Vivek

Re: [NFS] Disabling Symbolic Link Content Caching in NFS Client

[Ion Badulescu]

On Wed, 28 May 2003 16:39:36 +0530, Vivek Goyal <vivek.goyal@wipro.com> wrote:

> I was following previous discussions in the mailing list and found that
> caching mechanism affected hlfsd behavior. It looks like the problem was
> resolved by updating mtime on every access. But still caching will lead
> to a problem if two accesses are taking place in same jiffy.

Yes, because the the nfs client is broken in this respect. noac 
should mean noac, but the nfs client will still cache attributes if the 
second access happens within the same jiffy as the previous access which 
updated the attribute cache.

Replacing a couple of time_after() calls with time_after_eq() calls 
fixes the issue, at least for hlfsd. Of course, running with noac might 
be very much unacceptable if all you want is to avoid caching symlinks.

> We are considering following design strategy to implement the parameter.
> 
> 1. Make nfs_symlink_caching dynamically tunable using /proc and sysctl
> interface.

No. Do it on a per-mount basis, like the other OS's do.

Ion

-- 
  It is better to keep your mouth shut and be thought a fool,
            than to open it and remove all doubt.

Re: [NFS] Disabling Symbolic Link Content Caching in NFS Client

[Trond Myklebust]

>>>>> " " == Ion Badulescu <ionut@badula.org> writes:

    >> 1. Make nfs_symlink_caching dynamically tunable using /proc and
    >> sysctl interface.

     > No. Do it on a per-mount basis, like the other OS's do.

As I said to Vivek in a private mail, it would be very nice to see
if this could be done by replacing hlfsd with namespace groups.

Al Viro has already done all the VFS layer work, which should be ready
and working in existing 2.4.20 and 2.5.x kernels. What is missing is
userland support for doing a CLONE_NEWNS, and then mounting the user's
home directory, mailspool,.... in the appropriate locations at login
time.

Cheers,
  Trond

Re: [NFS] Disabling Symbolic Link Content Caching in NFS Client

[Trond Myklebust]

>>>>> " " == Ion Badulescu <ionut@badula.org> writes:

     > Replacing a couple of time_after() calls with time_after_eq()
     > calls fixes the issue, at least for hlfsd.

BTW: the above does not suffice to eliminate all races. Two processes
owned by different users may still end up waiting on the same call to
nfs_symlink_filler() no matter how often you choose to update the
metadata.

Cheers,
 Trond