Re: Fwd: Prioritizing readdirplus/getattr/lookup

["J. Bruce Fields" <bfields@fieldses.org>]

On Tue, Apr 05, 2011 at 09:32:38AM -0700, Andrew Klaassen wrote:
> I've done some more benchmarking, and in my case writes appear to *not* be the culprit.
> 
> Having the HPC farm only reading files (with noatime set everywhere, of course) actually makes "ls -l" over NFS slightly ~slower~.

Do you know which part of 'ls -l' is taking longer?  The readdir, or the
stats?  (Would an strace determine it?)  Probably the latter, I guess.
(Apologies if you already said.)

> Having the HPC farm only reading files that all fit in the server's cache makes "ls -l" over NFS yet slower.  (I watched iostat while this was running to make sure that nothing was being written to or read from disk.)
> 
> So I've eliminated the disk as a bottleneck, and (as per my earlier emails) I've eliminated the filesystem and VM system.
> 
> It really does look at this point like nfsd is the choke point.

So I suppose the replies are likely waiting to be sent back over the tcp
connection to the client, and waiting behind big read replies?

As a check I wonder if there's any cheesy way we could prioritize
threads waiting to send back getattr requests over threads waiting to
send back reads.

Such threads are waiting on the xpt_mutex.  Is there some mutex or
semaphore variant that gives us control over the order things are woken
up in, or would we need our own?

--b.