Re: NFS client latencies

Ingo Molnar
2005-03-30T08:02:43Z

* Trond Myklebust <trond.myklebust@fys.uio.no> wrote:

> ty den 29.03.2005 Klokka 18:32 (-0500) skreiv Lee Revell:
> > On Tue, 2005-03-29 at 18:18 -0500, Trond Myklebust wrote:
> > > ty den 29.03.2005 Klokka 18:04 (-0500) skreiv Lee Revell:
> > > > I am seeing long latencies in the NFS client code. Attached is a ~1.9
> > > > ms latency trace.
> > >
> > > What kind of workload are you using to produce these numbers?
> > >
> >
> > Just a kernel compile over NFS.
>
> In other words a workload consisting mainly of mmap()ed writes?

new files created during a kernel compile are done via open()/write().

looking at the trace it seems that the NFS client code is doing list
walks over ~7000 entries (!), in nfs_list_add_request(). Whatever
protocol/server-side gain there might be due to the sorting and
coalescing, this CPU overhead seems extremely high - more than 1 msec
for this single insertion!

the comment suggests that this is optimized for append writes (which is
quite common, but by far not the only write workload) - but the
worst-case behavior of this code is very bad. How about disabling this
sorting altogether and benchmarking the result? Maybe it would get
comparable coalescing (higher levels do coalesce after all), but wastly
improved CPU utilization on the client side. (Note that the server
itself will do sorting of any write IO anyway, if this is to hit any
persistent storage - and if not then sorting so agressively on the
client side makes little sense.)

i think normal NFS benchmarks would not show this effect, as writes are
typically streamed in benchmarks. But once you have lots of outstanding
requests and a write comes out of order, CPU utilization (and latency)
skyrockets.

Ingo