Re: IO queueing and complete affinity w/ threads: Some results

All of lore.kernel.org
 help / color / mirror / Atom feed

From: "Alan D. Brunelle" <Alan.Brunelle@hp.com>
To: linux-kernel@vger.kernel.org
Cc: Jens Axboe <jens.axboe@oracle.com>,
	npiggin@suse.de, dgc@sgi.com, arjan@linux.intel.com
Subject: Re: IO queueing and complete affinity w/ threads: Some results
Date: Tue, 12 Feb 2008 15:56:11 -0500	[thread overview]
Message-ID: <47B207EB.8090204@hp.com> (raw)
In-Reply-To: <47B0B69B.1050807@hp.com>

Whilst running a series of file system related loads on our 32-way*, I dropped down to a 16-way w/ only 24 disks, and ran two kernels: the original set of Jens' patches and then his subsequent kthreads-based set. Here are the results:

Original:
A Q C |  MBPS   Avg Lat StdDev |  Q-local Q-remote | C-local C-remote
----- | ------ -------- ------ | -------- -------- | ------- --------
X X X | 1850.4 0.413880 0.0109 |      0.0  55860.8 |     0.0  27946.9
X X A | 1850.6 0.413848 0.0106 |      0.0  55859.2 |     0.0  27946.1
X X I | 1850.6 0.413830 0.0107 |      0.0  55858.5 | 27945.8      0.0
----- | ------ -------- ------ | -------- -------- | ------- --------
X A X | 1850.0 0.413949 0.0106 |  55843.7      0.0 |     0.0  27938.3
X A A | 1850.2 0.413931 0.0107 |  55844.2      0.0 |     0.0  27938.6
X A I | 1850.4 0.413862 0.0107 |  55854.3      0.0 | 27943.7      0.0
----- | ------ -------- ------ | -------- -------- | ------- --------
X I X | 1850.9 0.413764 0.0107 |      0.0  55866.2 |     0.0  27949.6
X I A | 1850.5 0.413854 0.0108 |      0.0  55855.0 |     0.0  27944.0
X I I | 1850.4 0.413848 0.0105 |      0.0  55854.6 | 27943.8      0.0
===== | ====== ======== ====== | ======== ======== | ======= ========
I X X | 1570.7 0.487686 0.0142 |      0.0  47406.1 |     0.0  23719.5
I X A | 1570.8 0.487666 0.0143 |      0.0  47409.3 | 23721.2      0.0
I X I | 1570.8 0.487664 0.0142 |      0.0  47410.7 | 23721.8      0.0
----- | ------ -------- ------ | -------- -------- | ------- --------
I A X | 1570.9 0.487642 0.0144 |  47412.2      0.0 |     0.0  23722.6
I A A | 1570.8 0.487647 0.0141 |  47411.2      0.0 | 23722.1      0.0
I A I | 1570.8 0.487651 0.0143 |  47410.8      0.0 | 23721.9      0.0
----- | ------ -------- ------ | -------- -------- | ------- --------
I I X | 1570.8 0.487683 0.0142 |  47410.2      0.0 |     0.0  23721.6
I I A | 1571.1 0.487591 0.0146 |  47415.0      0.0 | 23724.0      0.0
I I I | 1571.0 0.487623 0.0143 |  47412.5      0.0 | 23722.8      0.0
===== | ====== ======== ====== | ======== ======== | ======= ========
rq=0  | 1726.7 0.443562 0.0120 |  52118.6      0.0 |  2138.6  23937.2
rq=1  | 1820.5 0.420729 0.0110 |  54938.2      0.0 |     0.0  27485.6
----- | ------ -------- ------ | -------- -------- | ------- --------


kthreads-based:
A Q C |  MBPS   Avg Lat StdDev |  Q-local Q-remote | C-local C-remote
----- | ------ -------- ------ | -------- -------- | ------- --------
X X X | 1850.5 0.413867 0.0107 |      0.0  55854.7 |     0.0  27943.8
X X A | 1850.9 0.413763 0.0107 |      0.0  55867.0 |     0.0  27950.0
X X I | 1850.3 0.413911 0.0109 |      0.0  55849.0 | 27941.0      0.0
----- | ------ -------- ------ | -------- -------- | ------- --------
X A X | 1851.0 0.413730 0.0107 |  55871.4      0.0 |     0.0  27952.2
X A A | 1850.1 0.413919 0.0107 |  55845.5      0.0 |     0.0  27939.2
X A I | 1850.8 0.413789 0.0108 |  55864.8      0.0 | 27948.9      0.0
----- | ------ -------- ------ | -------- -------- | ------- --------
X I X | 1850.5 0.413849 0.0107 |      0.0  55856.5 |     0.0  27944.8
X I A | 1850.6 0.413818 0.0108 |      0.0  55860.2 |     0.0  27946.6
X I I | 1850.8 0.413764 0.0108 |      0.0  55866.7 | 27949.8      0.0
===== | ====== ======== ====== | ======== ======== | ======= ========
I X X | 1570.9 0.487662 0.0145 |      0.0  47410.1 |     0.0  23721.6
I X A | 1570.7 0.487691 0.0142 |      0.0  47406.9 | 23720.0      0.0
I X I | 1570.7 0.487688 0.0141 |      0.0  47406.5 | 23719.8      0.0
----- | ------ -------- ------ | -------- -------- | ------- --------
I A X | 1570.9 0.487661 0.0144 |  47415.4      0.0 |     0.0  23724.2
I A A | 1570.8 0.487648 0.0141 |  47409.1      0.0 | 23721.0      0.0
I A I | 1570.7 0.487667 0.0141 |  47406.1      0.0 | 23719.5      0.0
----- | ------ -------- ------ | -------- -------- | ------- --------
I I X | 1570.8 0.487691 0.0142 |  47409.3      0.0 |     0.0  23721.2
I I A | 1570.9 0.487644 0.0142 |  47408.8      0.0 | 23720.9      0.0
I I I | 1570.6 0.487671 0.0141 |  47412.5      0.0 | 23722.8      0.0
===== | ====== ======== ====== | ======== ======== | ======= ========
rq=0  | 1742.1 0.439676 0.0118 |  52578.1      0.0 |  3602.6  22703.0
rq=1  | 1745.0 0.438918 0.0115 |  52666.3      0.0 |  3473.0  22876.6
----- | ------ -------- ------ | -------- -------- | ------- --------

For the first 18 sets on both kernels the results are very similar, the last two rq=0/1 sets are perturbed too much by application placement (I would guess). Have to think about that some more.

Alan
* What I'm doing on the 32-way is to compare and contrast mkfs, untar, kernel make & kernel clean times with different combinations of Q, C and RQ. [[This is currently with the "Jens original" patch, if things go well, I can do an overnight run with the kthreads-based patch.]]

next prev parent reply	other threads:[~2008-02-12 20:56 UTC|newest]

Thread overview: 13+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2008-02-11 20:56 IO queueing and complete affinity w/ threads: Some results Alan D. Brunelle
2008-02-12 20:56 ` Alan D. Brunelle [this message]
2008-02-12 22:08 ` Alan D. Brunelle
2008-02-12 22:26   ` Alan D. Brunelle
2008-02-13 15:35 ` Alan D. Brunelle
2008-02-14 15:36 ` Alan D. Brunelle
2008-02-18 12:37   ` Jens Axboe
2008-02-18 13:33     ` Andi Kleen
2008-02-18 14:16       ` Jens Axboe
2008-02-19  1:49       ` Nick Piggin
2008-02-19 21:14     ` Paul Jackson
2008-02-19 21:31       ` Mike Travis
2008-02-20  8:08         ` Jens Axboe

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=47B207EB.8090204@hp.com \
    --to=alan.brunelle@hp.com \
    --cc=arjan@linux.intel.com \
    --cc=dgc@sgi.com \
    --cc=jens.axboe@oracle.com \
    --cc=linux-kernel@vger.kernel.org \
    --cc=npiggin@suse.de \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link

Be sure your reply has a Subject: header at the top and a blank line before the message body.

This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.