Re: NFS server QoS ideas

[Mark Hills <mark.hills@framestore.com>]

On Wed, 24 Sep 2014, Benjamin Coddington wrote:

> Hi Mark, I've often thought about working on a project like this - 
> usually when I've been trying to keep a number of systems responsive 
> when there's been one very aggressive client, which typically would be 
> identified by a single uid, or a uid:ip pair.  I think your second 
> criteria would be the most useful.

Thanks Ben.

To do uid:ip paid would be similarly possible, but actually we're also 
interested in uid only -- to balance load between users on a cluster.
 
> I've done absolutely no research on this, but I had thought offhandedly 
> that a solution might integrate with cgroups somehow.

I'm not sure, as this is a server-side thing, there's no real processes to 
partition in cgroups.
 
> That's not much (less than 2 cents), but I think this feature would be
> welcomed warmly by sysadmins running shared NFS systems.

Thanks, good to hear.

I'd appreciate any input or caveats from anyone on the ideas I posted for 
implementing this.

Thanks

-- 
Mark


> On Wed, 24 Sep 2014, Mark Hills wrote:
> 
> > I am looking at possibilities to implement QoS (quality of service) in an
> > NFS server -- for a multiuser HPC environment, on NFSv3.
> > 
> > The desire is for a fully loaded server to process requests in some kind
> > of "fair share" (like round robin) or priorities based on some attribute;
> > eg. uid at the client, request type + origin, or even directory etc.
> > 
> > Criteria could be fairly blunt and specific to our use case to begin with.
> > 
> > I've looked at the code with the following ideas for where to start.
> > 
> > It seems the implementation depends a lot on which attributes to
> > share/schedule on:
> > 
> > 1) by client IP:
> > 
> >   make use the existing structures
> > 
> >   enhance the svc_xprt_enqueue/dequeue process to schedule svc_xprt
> >   as these are already per-client
> > 
> > 2) by client uid or other RPC attribute:
> > 
> >   have svc_recv make multiple calls to svc_handle_xprt
> > 
> >   buffer the requests into multiple queues held at svc_pool (re-use
> >   xpt_deferred?)
> > 
> > 3) by NFS operation, file handle etc.:
> > 
> >   most awkward, as there is not buffering of requests at the NFS level
> >   or shared between threads
> > 
> >   perhaps do as (2) but with a function in svc_program to return
> >   scheduling criteria
> > 
> > It looks like I need to consider the behaviour when there are multiple
> > svc_pool (ie. NUMA)
> > 
> > This is the first time I've looked into this code, I'm interested in any
> > comments/criticisms or alternatives.
> > 
> > Thanks
> > 
> > -- 
> > Mark