From: Wu Fengguang <fengguang.wu@intel.com>
To: Trond Myklebust <Trond.Myklebust@netapp.com>, linux-nfs@vger.kernel.org
Cc: Peter Zijlstra <a.p.zijlstra@chello.nl>,
"linux-fsdevel@vger.kernel.org" <linux-fsdevel@vger.kernel.org>,
Andrew Morton <akpm@linux-foundation.org>,
Jan Kara <jack@suse.cz>, Christoph Hellwig <hch@lst.de>,
Dave Chinner <david@fromorbit.com>,
Greg Thelen <gthelen@google.com>,
Minchan Kim <minchan.kim@gmail.com>,
Vivek Goyal <vgoyal@redhat.com>,
Andrea Righi <arighi@develer.com>, linux-mm <linux-mm@kvack.org>,
LKML <linux-kernel@vger.kernel.org>
Subject: Re: [PATCH 1/2] nfs: writeback pages wait queue
Date: Wed, 26 Oct 2011 05:08:29 +0800 [thread overview]
Message-ID: <20111025210829.GA5064@localhost> (raw)
In-Reply-To: <20111023155439.GA7286@localhost>
[-- Attachment #1: Type: text/plain, Size: 7283 bytes --]
On Sun, Oct 23, 2011 at 11:54:39PM +0800, Wu Fengguang wrote:
> On Fri, Oct 21, 2011 at 12:05:30AM +0800, Wu Fengguang wrote:
> > Trond,
> >
> > After applying these two patches, the IO-less patchset performances
> > 45% better than the vanilla kernel and the average commit size only
> > decreases by -16% in the common NFS-thresh=1G/nfs-1dd case :)
>
> To better understand how the NFS writeback wait queue helps, I
> visualized the network traffic over time. Attached are the graphs for
> the vanilla kernel and the one with the IO-less + NFS wait queue
> patches.
>
> nfs-1dd-4k-32p-32016M-1024M:10-3.1.0-rc8-vanilla+/dstat-bw.png
> nfs-1dd-4k-32p-31951M-1024M:10-3.1.0-rc8-nfs-wq4+/dstat-bw.png
>
> The obvious difference is, the network traffic become now more
> distributed and the "zero traffic" periods are mostly reduced.
>
> The other 2dd, 10dd cases have similar results.
To better explore the internal NFS server states and to make it clear
where the performance gain comes from, I collected the dstat data in
the NFS server and visualized the network/disk throughputs over time.
The results are attached for the following test cases.
nfs-1dd-4k-32p-32016M-1024M:10-3.1.0-rc10-vanilla+
nfs-1dd-4k-32p-31951M-1024M:10-3.1.0-rc9-ioless-full-nfs-wq5-next-20111014+
In the above cases, the vanilla kernel is shown to have roughly
_interleaved_ disk/net activities. After the patches, the disk/net
pipeline is well established, hence the impressive performance improvements.
3.1.0-rc10-vanilla+ 3.1.0-rc9-ioless-full-nfs-wq5-next-20111014+
------------------------ ------------------------
57.48 +25.5% 72.11 NFS-thresh=1G/nfs-10dd-4k-32p-32768M-1024M:10-X
63.86 +27.1% 81.15 NFS-thresh=1G/nfs-1dd-4k-32p-32768M-1024M:10-X
47.51 +80.1% 85.54 NFS-thresh=1G/nfs-2dd-4k-32p-32768M-1024M:10-X
168.85 +41.4% 238.80 TOTAL write_bw
3.1.0-vanilla+ 3.1.0-rc9-ioless-full-nfs-wq5-next-20111014+
------------------------ ------------------------
21.93 +97.0% 43.21 NFS-thresh=100M/nfs-10dd-4k-32p-32768M-100M:10-X
55.91 +27.4% 71.23 NFS-thresh=100M/nfs-1dd-4k-32p-32768M-100M:10-X
26.73 +129.2% 61.25 NFS-thresh=100M/nfs-2dd-4k-32p-32768M-100M:10-X
15.66 +40.3% 21.96 NFS-thresh=10M/nfs-10dd-4k-32p-32768M-10M:10-X
29.01 +0.7% 29.20 NFS-thresh=10M/nfs-1dd-4k-32p-32768M-10M:10-X
20.52 +7.9% 22.13 NFS-thresh=10M/nfs-2dd-4k-32p-32768M-10M:10-X
56.30 +28.1% 72.11 NFS-thresh=1G/nfs-10dd-4k-32p-32768M-1024M:10-X
63.33 +28.1% 81.15 NFS-thresh=1G/nfs-1dd-4k-32p-32768M-1024M:10-X
51.95 +64.6% 85.54 NFS-thresh=1G/nfs-2dd-4k-32p-32768M-1024M:10-X
9.87 -48.2% 5.11 NFS-thresh=1M/nfs-10dd-4k-32p-32768M-1M:10-X
14.52 -24.0% 11.03 NFS-thresh=1M/nfs-1dd-4k-32p-32768M-1M:10-X
15.69 -67.4% 5.12 NFS-thresh=1M/nfs-2dd-4k-32p-32768M-1M:10-X
381.42 +33.5% 509.05 TOTAL write_bw
The vanilla kernel's disk idle periods indicate that the background
flusher work is not running. This is true. The NFS server has 12GB
memory and hence 1.2GB background dirty thresh. So when the client
side has dirty thresh <= 1GB, it won't send enough data to trigger
the background work in NFS server. So the server disk IOs will be
triggered mostly by NFS commits. The patched kernel seems to generate
smoother commit pattern to prevent most disk idles.
To explore how it performs when the NFS server's background writeback
kicks in, I further tried lowering the NFS server's dirty thresholds
to
dirty_bytes = 1GB, background threshold = 512MB
-----------------------------------------------
graphs attached for cases:
nfs-1dd-1M-32p-32016M-1024M:10-3.1.0-vanilla+
nfs-1dd-1M-32p-31951M-1024M:10-3.1.0-rc9-ioless-full-nfs-wq5-next-20111014+
3.1.0-vanilla+ 3.1.0-rc9-ioless-full-nfs-wq5-next-20111014+
------------------------ ------------------------
25.23 +71.2% 43.21 NFS-thresh=100M/nfs-10dd-1M-32p-32768M-100M:10-X
60.21 +15.1% 69.29 NFS-thresh=100M/nfs-1dd-1M-32p-32768M-100M:10-X
33.46 +82.8% 61.18 NFS-thresh=100M/nfs-2dd-1M-32p-32768M-100M:10-X
21.81 +2.8% 22.42 NFS-thresh=10M/nfs-10dd-1M-32p-32768M-10M:10-X
28.71 -11.0% 25.55 NFS-thresh=10M/nfs-1dd-1M-32p-32768M-10M:10-X
28.11 -13.0% 24.46 NFS-thresh=10M/nfs-2dd-1M-32p-32768M-10M:10-X
54.10 +42.3% 76.97 NFS-thresh=1G/nfs-10dd-1M-32p-32768M-1024M:10-X
64.88 +29.1% 83.76 NFS-thresh=1G/nfs-1dd-1M-32p-32768M-1024M:10-X
53.64 +56.2% 83.79 NFS-thresh=1G/nfs-2dd-1M-32p-32768M-1024M:10-X
13.00 -61.0% 5.06 NFS-thresh=1M/nfs-10dd-1M-32p-32768M-1M:10-X
13.53 -33.9% 8.95 NFS-thresh=1M/nfs-1dd-1M-32p-32768M-1M:10-X
16.51 -72.5% 4.54 NFS-thresh=1M/nfs-2dd-1M-32p-32768M-1M:10-X
413.20 +23.2% 509.18 TOTAL write_bw
dirty_bytes = 100MB, background threshold = 50MB
------------------------------------------------
graphs attached for cases:
nfs-1dd-100k-32p-32016M-1024M:10-3.1.0-vanilla+
nfs-1dd-100k-32p-31951M-1024M:10-3.1.0-rc9-ioless-full-nfs-wq5-next-20111014+
3.1.0-vanilla+ 3.1.0-rc9-ioless-full-nfs-wq5-next-20111014+
------------------------ ------------------------
24.93 +71.1% 42.67 NFS-thresh=100M/nfs-10dd-100k-32p-32768M-100M:10-X
58.35 +23.0% 71.78 NFS-thresh=100M/nfs-1dd-100k-32p-32768M-100M:10-X
28.47 +123.0% 63.48 NFS-thresh=100M/nfs-2dd-100k-32p-32768M-100M:10-X
18.44 +22.5% 22.60 NFS-thresh=10M/nfs-10dd-100k-32p-32768M-10M:10-X
28.75 +0.1% 28.78 NFS-thresh=10M/nfs-1dd-100k-32p-32768M-10M:10-X
20.84 +12.0% 23.34 NFS-thresh=10M/nfs-2dd-100k-32p-32768M-10M:10-X
71.78 +6.4% 76.38 NFS-thresh=1G/nfs-10dd-100k-32p-32768M-1024M:10-X
91.02 -4.7% 86.72 NFS-thresh=1G/nfs-1dd-100k-32p-32768M-1024M:10-X
78.84 +9.9% 86.64 NFS-thresh=1G/nfs-2dd-100k-32p-32768M-1024M:10-X
421.42 +19.2% 502.39 TOTAL write_bw
The conclusion is,
- it helps the vanilla kernel performance to lower the NFS server's
dirty threshold
- when NFS server's background writeback is not running, the patches
will help establish good net/disk pipeline and improve performance;
otherwise it maintains roughly the same performance
Thanks,
Fengguang
[-- Attachment #2: dstat-nfss-bw.png --]
[-- Type: image/png, Size: 58596 bytes --]
[-- Attachment #3: dstat-nfss-bw.png --]
[-- Type: image/png, Size: 80812 bytes --]
[-- Attachment #4: dstat-nfss-bw.png --]
[-- Type: image/png, Size: 57489 bytes --]
[-- Attachment #5: dstat-nfss-bw.png --]
[-- Type: image/png, Size: 72627 bytes --]
[-- Attachment #6: dstat-nfss-bw.png --]
[-- Type: image/png, Size: 63146 bytes --]
[-- Attachment #7: dstat-nfss-bw.png --]
[-- Type: image/png, Size: 70136 bytes --]
prev parent reply other threads:[~2011-10-25 21:08 UTC|newest]
Thread overview: 6+ messages / expand[flat|nested] mbox.gz Atom feed top
2011-10-20 15:55 [PATCH 1/2] nfs: writeback pages wait queue Wu Fengguang
2011-10-20 15:56 ` [PATCH 2/2] nfs: scale writeback threshold proportional to dirty threshold Wu Fengguang
2011-10-20 16:05 ` [PATCH 1/2] nfs: writeback pages wait queue Wu Fengguang
2011-10-22 7:34 ` Wu Fengguang
2011-10-23 15:54 ` Wu Fengguang
2011-10-25 21:08 ` Wu Fengguang [this message]
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20111025210829.GA5064@localhost \
--to=fengguang.wu@intel.com \
--cc=Trond.Myklebust@netapp.com \
--cc=a.p.zijlstra@chello.nl \
--cc=akpm@linux-foundation.org \
--cc=arighi@develer.com \
--cc=david@fromorbit.com \
--cc=gthelen@google.com \
--cc=hch@lst.de \
--cc=jack@suse.cz \
--cc=linux-fsdevel@vger.kernel.org \
--cc=linux-kernel@vger.kernel.org \
--cc=linux-mm@kvack.org \
--cc=linux-nfs@vger.kernel.org \
--cc=minchan.kim@gmail.com \
--cc=vgoyal@redhat.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).