From: jens kusch <jens.kusch@oracle.com>
To: linux-nfs@vger.kernel.org
Subject: strange nfsd scheduling in 2.6.32
Date: Fri, 31 May 2013 16:15:40 +0200 [thread overview]
Message-ID: <51A8B08C.3090309@oracle.com> (raw)
Hi all,
we have a problem with nfsd performance in 2.6.32. They don't seem to
able to cope with the load. This is different in 2.6.18. Anybody seen
this before?
On Linux 2.6.32:
- IOs are often processed by nfsd processes in a delayed fashion, as if
they have been queued before (seen from application traces).
- NFS pool statistics show only a smaller fraction processed immediately
(10..20%). The rest is queued or delayed.
- On the other hand there are lots of nfsd processes that sit idle at
the same time!
- CPU usage is very unevenly distributed among the nfsd servers, many
are never used
I'd just like to emphasize one detail: note the output from
/proc/fs/nfsd/pool_stats below:
# pool packets-arrived sockets-enqueued threads-woken overloads-avoided
threads-timedout
0 7740103 1837083 885771 1837081 480
The stat overloads-avoided always gets incremented in our runs. Here is
a brief description:
Counts how many times the sunrpc server layer chose not to wake an nfsd
thread, despite the presence of idle nfsd threads, because too many nfsd
threads had been recently woken but could not get enough CPU time to
actually run. In our runs, CPU utilization never gets close to 100%, so
I wonder why NFS decided not to wake up one of the idle threads we see.
In our runs, CPU utilization never gets close to 100%, so I wonder why
NFS decided not to wake up one of the idle threads we see.
On Linux 2.6.18
- Performance via NFS is better
- CPU usage is more evenly distributed among the nfsd processes, all
nfsd processes are really used
We would appreciate any hint about what could be wrong in 2.6.32.
Best regards,
Jens
next reply other threads:[~2013-05-31 14:15 UTC|newest]
Thread overview: 3+ messages / expand[flat|nested] mbox.gz Atom feed top
2013-05-31 14:15 jens kusch [this message]
2013-06-03 19:37 ` strange nfsd scheduling in 2.6.32 J. Bruce Fields
2013-06-04 16:16 ` jens kusch
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=51A8B08C.3090309@oracle.com \
--to=jens.kusch@oracle.com \
--cc=linux-nfs@vger.kernel.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).