Re: NFSv4/pNFS possible POSIX I/O API standards

[Rob Ross <rross@mcs.anl.gov>]

Matthew Wilcox wrote:
> On Tue, Dec 05, 2006 at 10:07:48AM +0000, Christoph Hellwig wrote:
>> The filehandle idiocy on the other hand is way of into crackpipe land.
> 
> Right, and it needs to be discarded.  Of course, there was a real
> problem that it addressed, so we need to come up with an acceptable
> alternative.
 >
> The scenario is a cluster-wide application doing simultaneous opens of
> the same file.  So thousands of nodes all hitting the same DLM locks
> (for read) all at once.  The openg() non-solution implies that all
> nodes in the cluster share the same filehandle space, so I think a
> reasonable solution can be implemented entirely within the clusterfs,
> with an extra flag to open(), say O_CLUSTER_WIDE.  When the clusterfs
> sees this flag set (in ->lookup), it can treat it as a hint that this
> pathname component is likely to be opened again on other nodes and
> broadcast that fact to the other nodes within the cluster.  Other nodes
> on seeing that hint (which could be structured as "The child "bin"
> of filehandle e62438630ca37539c8cc1553710bbfaa3cf960a7 has filehandle
> ff51a98799931256b555446b2f5675db08de6229") can keep a record of that fact.
> When they see their own open, they can populate the path to that file
> without asking the server for extra metadata.
> 
> There's obviously security issues there (why I say 'hint' rather than
> 'command'), but there's also security problems with open-by-filehandle.
> Note that this solution requires no syscall changes, no application
> changes, and also helps a scenario where each node opens a different
> file in the same directory.
> 
> I've never worked on a clusterfs, so there may be some gotchas (eg, how
> do you invalidate the caches of nodes when you do a rename).  But this
> has to be preferable to open-by-fh.

The openg() solution has the following advantages to what you propose. 
First, it places the burden of the communication of the file handle on 
the application process, not the file system. That means less work for 
the file system. Second, it does not require that clients respond to 
unexpected network traffic. Third, the network traffic is deterministic 
-- one client interacts with the file system and then explicitly 
performs the broadcast. Fourth, it does not require that the file system 
store additional state on clients.

In the O_CLUSTER_WIDE approach, a naive implementation (everyone passing 
the flag) would likely cause a storm of network traffic if clients were 
closely synchronized (which they are likely to be). We could work around 
this by having one application open early, then barrier, then have 
everyone else open, but then we might as well have just sent the handle 
as the barrier operation, and we've made the use of the O_CLUSTER_WIDE 
open() significantly more complicated for the application.

However, the application change issue is actually moot; we will make 
whatever changes inside our MPI-IO implementation, and many users will 
get the benefits for free.

The readdirplus(), readx()/writex(), and openg()/openfh() were all 
designed to allow our applications to explain exactly what they wanted 
and to allow for explicit communication. I understand that there is a 
tendency toward solutions where the FS guesses what the app is going to 
do or is passed a hint (e.g. fadvise) about what is going to happen, 
because these things don't require interface changes. But these 
solutions just aren't as effective as actually spelling out what the 
application wants.

Regards,

Rob