Re: NFSv4/pNFS possible POSIX I/O API standards

[Rob Ross <rross@mcs.anl.gov>]

Latchesar Ionkov wrote:
> On 12/5/06, Rob Ross <rross@mcs.anl.gov> wrote:
>>
>> I agree that it is not feasible to add new system calls every time
>> somebody has a problem, and we don't take adding system calls lightly.
>> However, in this case we're talking about an entire *community* of
>> people (high-end computing), not just one or two people. Of course it
>> may still be the case that that community is not important enough to
>> justify the addition of system calls; that's obviously not my call to 
>> make!
> 
> I have the feeling that openg stuff is rushed without looking into all
> solutions, that don't require changes to the current interface. I
> don't see any numbers showing where exactly the time is spent? Is
> opening too slow because of the number of requests that the file
> server suddently has to respond to? Does having an operation that
> looks up multiple names instead of a single name good enough? How much
> time is spent on opening the file once you have resolved the name?

Thanks for looking at the graph.

To clarify the workload, we do not expect that application processes 
will be opening a large number of files all at once; that was just how 
the test was run to get a reasonable average value. So I don't think 
that something that looked up multiple file names would help for this case.

I unfortunately don't have data to show exactly where the time was 
spent, but it's a good guess that it is all the network traffic in the 
open() case.

>> I'm sure that you meant more than just to rename openg() to lookup(),
>> but I don't understand what you are proposing. We still need a second
>> call to take the results of the lookup (by whatever name) and convert
>> that into a file descriptor. That's all the openfh() (previously named
>> sutoc()) is for.
> 
> The idea is that lookup doesn't open the file, just does to name
> resolution. The actual opening is done by openfh (or whatever you call
> it next :). I don't think it is a good idea to introduce another way
> of addressing files on the file system at all, but if you still decide
> to do it, it makes more sense to separate the name resolution from the
> operations (at the moment only open operation, but who knows what'll
> somebody think of next;) you want to do on the file.

I really think that we're saying the same thing here?

I think of the open() call as doing two (maybe three) things. First, 
performs name resolution and permission checking. Second, creates the 
file descriptor that allows the user process to do subsequent I/O. 
Third, creates a context for access, if the FS keeps track of "open" 
files (not all do).

The openg() really just does the lookup and permission checking). The 
openfh() creates the file descriptor and starts that context if the 
particular FS tracks that sort of thing.

>> I think the subject line might be a little misleading; we're not just
>> talking about NFS here. There are a number of different file systems
>> that might benefit from these enhancements (e.g. GPFS, Lustre, PVFS,
>> PanFS, etc.).
> 
> I think that the main problem is that all these file systems resove a
> path name, one directory at a time bringing the server to its knees by
> the huge amount of requests. I would like to see what the performance
> is if you a) cache the last few hundred lookups on the server side,
> and b) modify VFS and the file systems to support multi-name lookups.
> Just assume for a moment that there is no any way to get these new
> operations in (which is probaly going to be true anyway :). What other
> solutions can you think of? :)

Well you've caught me. I don't want to cache the values, because I 
fundamentally believe that sharing state between clients and servers is 
braindead (to use Christoph's phrase) in systems of this scale 
(thousands to tens of thousands of clients). So I don't want locks, so I 
can't keep the cache consistent, ... So someone else will have to run 
the tests you propose :)...

Also, to address Christoph's snipe while we're here; I don't care one 
way or another whether the Linux community wants to help GPFS or not. I 
do care that I'm arguing for something that is useful to more than just 
my own pet project, and that was the point that I was trying to make. 
I'll be sure not to mention GPFS again.

What's the etiquette on changing subject lines here? It might be useful 
to separate the openg() etc. discussion from the readdirplus() etc. 
discussion.

Thanks again for the comments,

Rob