From: "Darrick J. Wong" <djwong@kernel.org>
To: Jan Kara <jack@suse.cz>
Cc: linux-xfs@vger.kernel.org, Dave Chinner <dchinner@redhat.com>
Subject: Re: Performance regression with async inode inactivation
Date: Mon, 4 Oct 2021 11:17:56 -0700 [thread overview]
Message-ID: <20211004181756.GA24307@magnolia> (raw)
In-Reply-To: <20211004100653.GD2255@quack2.suse.cz>
On Mon, Oct 04, 2021 at 12:06:53PM +0200, Jan Kara wrote:
> Hello,
>
> our performance testing grid has detected a performance regression caused
> by commit ab23a77687 ("xfs: per-cpu deferred inode inactivation queues")
> with reaim benchmark running 'disk' and 'disk-large' workloads. The
> regression has been so far detected on two machines - marvin7 (48 cpus, 64
> GB ram, SATA SSD), dobby (64 cpus, 192 GB ram, rotating disk behind
> megaraid_sas controller).
I'll think about this some more, but for now my immediate reaction is to
point out the previous discussion about aim7 reporting a similar
(percentage point) regression:
https://lore.kernel.org/lkml/20210909142355.GA6270@xsang-OptiPlex-9020/
--D
>
> The regression reports from mmtests on marvin7 (which from my experience I
> find more reliable than dobby) look like:
>
> reaim-io-disk-xfs
> Hmean disk-1 3597.12 ( 0.00%) 3636.37 ( 1.09%)
> Hmean disk-25 125000.00 ( 0.00%) 121753.25 ( -2.60%)
> Hmean disk-49 218424.96 ( 0.00%) 193421.05 * -11.45%*
> Hmean disk-73 286649.22 ( 0.00%) 240131.58 * -16.23%*
> Hmean disk-97 349339.74 ( 0.00%) 285854.62 * -18.17%*
> Hmean disk-121 373456.79 ( 0.00%) 309199.32 * -17.21%*
> Hmean disk-145 399449.04 ( 0.00%) 330547.11 * -17.25%*
> Hmean disk-169 420049.71 ( 0.00%) 345132.74 * -17.84%*
> Hmean disk-193 458795.56 ( 0.00%) 375243.03 * -18.21%*
> Stddev disk-1 102.32 ( 0.00%) 117.93 ( -15.25%)
> Stddev disk-25 6981.08 ( 0.00%) 4285.17 ( 38.62%)
> Stddev disk-49 9523.63 ( 0.00%) 10723.44 ( -12.60%)
> Stddev disk-73 9704.22 ( 0.00%) 7946.73 ( 18.11%)
> Stddev disk-97 10059.91 ( 0.00%) 6111.28 ( 39.25%)
> Stddev disk-121 5730.56 ( 0.00%) 11714.34 (-104.42%)
> Stddev disk-145 11154.40 ( 0.00%) 8129.06 ( 27.12%)
> Stddev disk-169 4477.30 ( 0.00%) 3558.86 ( 20.51%)
> Stddev disk-193 8785.70 ( 0.00%) 13258.89 ( -50.91%)
>
> reaim-io-disk-large-xfs
> Hmean disk-1 722.72 ( 0.00%) 721.85 ( -0.12%)
> Hmean disk-25 24177.95 ( 0.00%) 24319.06 ( 0.58%)
> Hmean disk-49 35294.12 ( 0.00%) 34361.85 ( -2.64%)
> Hmean disk-73 43042.45 ( 0.00%) 40896.36 * -4.99%*
> Hmean disk-97 48403.19 ( 0.00%) 46044.30 * -4.87%*
> Hmean disk-121 52230.22 ( 0.00%) 49347.47 * -5.52%*
> Hmean disk-145 54613.94 ( 0.00%) 52333.98 * -4.17%*
> Hmean disk-169 57178.30 ( 0.00%) 54745.71 * -4.25%*
> Hmean disk-193 60230.94 ( 0.00%) 57106.22 * -5.19%*
> Stddev disk-1 18.74 ( 0.00%) 30.19 ( -61.11%)
> Stddev disk-25 439.49 ( 0.00%) 809.58 ( -84.21%)
> Stddev disk-49 1416.65 ( 0.00%) 603.55 ( 57.40%)
> Stddev disk-73 949.45 ( 0.00%) 584.61 ( 38.43%)
> Stddev disk-97 689.51 ( 0.00%) 602.76 ( 12.58%)
> Stddev disk-121 485.22 ( 0.00%) 612.79 ( -26.29%)
> Stddev disk-145 147.37 ( 0.00%) 442.99 (-200.60%)
> Stddev disk-169 282.25 ( 0.00%) 613.42 (-117.33%)
> Stddev disk-193 970.05 ( 0.00%) 572.59 ( 40.97%)
>
> Note that numbers behind dash (disk-xx) denote the number of reaim
> "clients" - i.e., the number of processes reaim runs in parallel.
>
> This reaim workload will create quite some small files, fsync them, do a
> few operations on them like read, write, etc. and quickly delete them. This
> happens in many processes in parallel so I can imagine the unlink workload
> is rather heavy.
>
> To reproduce the workload the easiest is probably to clone mmtests [1],
> there's README.md and docs/Tutorial.md about how to run tests. Relevant
> configurations are in configs/config-reaim-io-disk-large and
> configs/config-reaim-io-disk, you will need to edit these files to set
> appropriate test partition (will get wiped) and filesystem. Note that
> mmtests do somewhat modify standard reaim benchmark so that it does not
> call system("sync") after each operation (you can see modifications we
> apply in [2] if you're interested).
>
> I can probably find some time to better understand what's going on on these
> machines later this week but if you have some suggestions what to look for,
> you're welcome.
>
> Honza
>
> [1] https://github.com/gormanm/mmtests
> [2] https://github.com/gormanm/mmtests/blob/master/shellpack_src/src/reaim/reaim-install
>
> --
> Jan Kara <jack@suse.com>
> SUSE Labs, CR
next prev parent reply other threads:[~2021-10-04 18:17 UTC|newest]
Thread overview: 12+ messages / expand[flat|nested] mbox.gz Atom feed top
2021-10-04 10:06 Performance regression with async inode inactivation Jan Kara
2021-10-04 18:17 ` Darrick J. Wong [this message]
2021-10-04 21:15 ` Dave Chinner
2021-10-05 8:11 ` Jan Kara
2021-10-05 21:26 ` Dave Chinner
2021-10-06 18:10 ` Jan Kara
2021-10-06 21:58 ` Dave Chinner
2021-10-07 12:03 ` Jan Kara
2021-10-07 23:44 ` Dave Chinner
2021-10-12 13:42 ` Jan Kara
2021-10-12 21:23 ` Dave Chinner
2021-10-13 9:51 ` Jan Kara
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20211004181756.GA24307@magnolia \
--to=djwong@kernel.org \
--cc=dchinner@redhat.com \
--cc=jack@suse.cz \
--cc=linux-xfs@vger.kernel.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.