From: Justin Piszcz <jpiszcz@lucidpixels.com>
To: Pallai Roland <dap@mail.index.hu>
Cc: Linux RAID Mailing List <linux-raid@vger.kernel.org>
Subject: Re: major performance drop on raid5 due to context switches caused by small max_hw_sectors [partially resolved]
Date: Sun, 22 Apr 2007 06:23:12 -0400 (EDT) [thread overview]
Message-ID: <Pine.LNX.4.64.0704220619390.14170@p34.internal.lan> (raw)
In-Reply-To: <200704221152.18747.dap@mail.index.hu>
On Sun, 22 Apr 2007, Pallai Roland wrote:
>
> On Sunday 22 April 2007 10:47:59 Justin Piszcz wrote:
>> On Sun, 22 Apr 2007, Pallai Roland wrote:
>>> On Sunday 22 April 2007 02:18:09 Justin Piszcz wrote:
>>>>
>>>> How did you run your read test?
>>>>
>>>
>>> I did run 100 parallel reader process (dd) top of XFS file system, try
>>> this: for i in `seq 1 100`; do dd of=$i if=/dev/zero bs=64k 2>/dev/null;
>>> done for i in `seq 1 100`; do dd if=$i of=/dev/zero bs=64k 2>/dev/null &
>>> done
>>>
>>> and don't forget to set max_sectors_kb below chunk size (eg. 64/128Kb)
>>> /sys/block# for i in sd*; do echo 64 >$i/queue/max_sectors_kb; done
>>>
>>> I also set 2048/4096 readahead sectors with blockdev --setra
>>>
>>> You need 50-100 reader processes for this issue, I think so. My kernel
>>> version is 2.6.20.3
>>>
>>
>> In one xterm:
>> for i in `seq 1 100`; do dd of=$i if=/dev/zero bs=64k 2>/dev/null; done
>>
>> In another:
>> for i in `seq 1 100`; do dd if=/dev/md3 of=$i.out bs=64k & done
>
> Write and read files top of XFS, not on the block device. $i isn't a typo, you
> should write into 100 files and read back by 100 threads in parallel when
> done. I've 1Gb of RAM, maybe you should use mem= kernel parameter on boot.
>
> 1. for i in `seq 1 100`; do dd of=$i if=/dev/zero bs=1M count=100 2>/dev/null;
> done
> 2. for i in `seq 1 100`; do dd if=$i of=/dev/zero bs=64k 2>/dev/null & done
>
>
> --
> d
>
I use a combination of 4 Silicon Image controllers (SiI) and the Intel 965
chipset. My max_sectors_kb is 128kb, my chunk size is 128kb, why do you
set the max_sectors_kb less than the chunk size? For read-ahead, there
are some good benchmarks by SGI(?) I believe and some others that states
16MB is the best value, over that, you lose on reads/writes or the other,
16MB appears to be optimal for best overall value. Do these values look
good to you, or?
Read 100 files on XFS simultaneously:
procs -----------memory---------- ---swap-- -----io---- -system-- ----cpu----
r b swpd free buff cache si so bi bo in cs us sy id wa
0 0 104 146680 460 3179400 0 0 138 17 77 161 1 0 99 0
0 0 104 146680 460 3179400 0 0 0 0 1095 190 0 0 100 0
16 84 104 149024 460 3162880 0 0 57492 25 1481 1284 2 10 88 1
0 88 104 148728 396 2982628 0 0 296904 8 2816 4728 1 48 0 51
0 88 104 147796 40 2894752 0 0 204080 0 2815 3507 0 8 0 92
0 87 104 149272 40 2845688 0 0 116200 0 2781 2836 0 6 0 94
0 87 104 147872 40 2770888 0 0 188260 0 2948 3043 0 9 0 91
0 87 104 148244 40 2583700 0 0 218440 0 2856 3218 0 12 0 88
1 87 104 147528 40 2385668 0 0 181352 0 3225 4211 1 9 0 90
0 87 104 147784 40 2343868 0 0 225716 0 3205 3725 0 9 0 91
0 88 104 146112 40 2233272 0 0 202968 8 3956 5669 0 9 0 91
0 88 104 146452 40 2325332 0 0 241196 0 4374 4927 0 7 0 93
0 88 104 147792 40 2463600 0 0 239788 0 3216 1139 1 5 0 94
1 88 104 148600 40 2468352 0 0 187488 0 3107 2883 0 6 0 94
0 88 104 147240 40 2553940 0 0 168236 0 3559 2795 0 5 0 95
0 87 104 147772 40 2682932 0 0 278092 0 5531 7507 0 9 0 91
0 87 104 148200 40 2813376 0 0 201324 0 4346 5726 0 6 0 94
1 87 104 148196 40 2774932 0 0 193620 0 5433 11595 0 10 0 90
0 87 104 148488 40 2785256 0 0 227568 0 4081 6121 0 11 0 89
0 87 104 148240 40 2798024 0 0 201800 0 5064 10857 1 11 0 89
0 87 104 148684 40 2825632 0 0 206564 20 5244 9451 0 10 0 90
2 86 104 149060 40 2899464 0 0 182584 0 6610 15984 0 10 0 90
0 86 104 148004 40 2925044 0 0 167044 0 5725 13011 0 13 0 87
0 84 104 148344 40 3025508 0 0 149640 0 6860 18340 0 9 0 91
0 81 104 146152 40 3074460 0 0 167780 0 7572 19434 1 16 0 84
0 79 104 149172 40 3127748 0 0 191064 0 5373 11928 0 10 0 90
0 77 104 146972 40 3101440 0 0 135968 0 9232 28208 1 18 0 82
0 70 104 148896 40 3081860 0 0 144372 0 7790 22857 0 16 0 84
0 69 104 148492 40 3173532 0 0 154024 0 8002 22211 0 10 0 90
1 68 104 145804 40 3192788 0 0 111412 0 9612 30604 0 14 0 86
0 66 104 147324 40 3173004 0 0 137800 0 9858 31132 0 13 0 87
0 65 104 145456 40 3121436 0 0 141348 0 6521 17443 1 13 0 87
1 64 104 148408 40 3154056 0 0 149308 0 9575 29755 0 13 0 87
0 61 104 145396 40 3187312 0 0 161976 0 9709 29021 0 13 0 87
0 61 104 147860 40 3189972 0 0 154844 0 7345 20033 0 10 0 90
2 61 104 148264 40 3162276 0 0 135804 0 10865 33643 0 15 1 8
5
0 59 104 147264 40 3203068 0 0 70464 0 9340 29345 0 11 0 89
1 58 104 147344 40 3203468 0 0 108072 48 12206 39011 0 15 0 8
5
0 58 104 146696 40 3204660 0 0 102492 32 12800 43578 0 15 0 8
5
1 58 104 145712 40 3201916 0 0 198728 8 10537 34621 0 15 0 8
5
2 56 104 148732 40 3195160 0 0 119700 0 10647 35077 1 15 0 8
4
0 54 104 148628 40 3203856 0 0 92588 0 8757 26257 0 12 0 88
0 54 104 147128 40 3203956 0 0 207224 0 9706 29997 0 14 0 86
1 54 104 146788 40 3165980 0 0 150120 36 10611 36721 0 17 0 8
3
1 52 104 146984 40 3121696 0 0 160080 8 8714 28264 0 14 0 86
0 51 104 147556 40 3123932 0 0 223940 0 6088 15013 0 13 0 87
0 50 104 147268 40 3172724 0 0 154592 0 8949 26645 0 13 0 87
0 46 104 148692 40 3135296 0 0 153708 0 7146 19844 1 15 0 85
0 45 104 148116 40 3168180 0 0 165740 20 7884 22111 0 14 0 86
0 43 104 147388 40 3209160 0 0 179588 8 9244 27501 0 14 0 86
0 41 104 147760 40 3208860 0 0 131216 24 8205 24867 0 10 0 90
0 41 104 148868 40 3122048 0 0 188260 0 9812 30872 0 14 0 86
1 40 104 146832 40 3210452 0 0 168388 0 8798 27378 0 12 0 88
1 39 104 146388 40 3211436 0 0 96044 0 13197 45162 0 15 2 83
0 37 104 144716 40 3213412 0 0 151044 12 9877 30263 1 14 0 85
0 37 104 146056 40 3212088 0 0 95988 27 13179 44933 0 15 5 80
3 36 104 146776 40 3211552 0 0 66580 0 13651 46636 0 16 9 76
1 37 104 146504 40 3212192 0 0 77196 0 16353 57091 0 18 0 82
1 37 104 146864 40 3211648 0 0 135312 0 13350 45859 0 17 1 8
3
2 35 104 148924 40 3209796 0 0 81228 0 13459 43748 0 15 0 85
1 36 104 147792 40 3211332 0 0 107248 0 15935 57007 0 21 0 7
9
2 35 104 146916 40 3212444 0 0 84328 0 16471 57611 0 18 0 82
0 35 104 146500 40 3212944 0 0 98524 0 16049 55775 0 19 0 81
0 34 104 148448 40 3211220 0 0 114116 0 14166 51140 1 17 2 8
1
0 32 104 145220 40 3215024 0 0 96068 0 16008 58012 0 21 0 79
1 29 104 145376 40 3215416 0 0 102604 0 12294 41704 0 16 1 8
4
2 27 104 146764 40 3214384 0 0 132004 0 13076 44026 1 15 3 8
2
0 27 104 148888 40 3212460 0 0 118908 0 16100 57546 0 19 0 8
1
0 26 104 144932 40 3216260 0 0 101220 24 15687 55236 0 20 2 7
8
1 24 104 147848 40 3214344 0 0 125152 8 13737 48996 0 19 0 8
1
2 21 104 147024 40 3215676 0 0 71300 0 18828 65151 0 22 0 78
1 22 104 145124 40 3217852 0 0 85228 0 19031 65807 1 20 0 80
2 21 104 147472 40 3215532 0 0 96188 0 16644 60505 0 20 0 80
2 18 104 146064 40 3217544 0 0 83316 0 16788 60762 1 21 0 79
6 13 104 145824 40 3218832 0 0 56188 0 15100 51145 0 16 1 83
0 6 104 147988 40 3218048 0 0 75836 20 19925 69456 0 22 0 78
0 0 104 148592 40 3218292 0 0 20328 0 6190 17793 0 6 70 24
0 0 104 148796 40 3218392 0 0 36 0 1199 372 1 0 98 1
0 0 104 148796 40 3218392 0 0 0 0 1090 192 0 0 100 0
0 0 104 149044 40 3218420 0 0 16 0 1110 349 0 0 100 0
0 0 104 149044 40 3218420 0 0 0 17 1105 212 0 0 100 0
0 0 104 149308 40 3218436 0 0 0 0 1143 317 0 0 100 0
next prev parent reply other threads:[~2007-04-22 10:23 UTC|newest]
Thread overview: 15+ messages / expand[flat|nested] mbox.gz Atom feed top
2007-04-20 21:06 major performance drop on raid5 due to context switches caused by small max_hw_sectors Pallai Roland
[not found] ` <5d96567b0704202247s60e4f2f1x19511f790f597ea0@mail.gmail.com>
2007-04-21 19:32 ` major performance drop on raid5 due to context switches caused by small max_hw_sectors [partially resolved] Pallai Roland
2007-04-22 0:18 ` Justin Piszcz
2007-04-22 0:42 ` Pallai Roland
2007-04-22 8:47 ` Justin Piszcz
2007-04-22 9:52 ` Pallai Roland
2007-04-22 10:23 ` Justin Piszcz [this message]
2007-04-22 11:38 ` Pallai Roland
2007-04-22 11:42 ` Justin Piszcz
2007-04-22 14:38 ` Pallai Roland
2007-04-22 14:48 ` Justin Piszcz
2007-04-22 15:09 ` Pallai Roland
2007-04-22 15:53 ` Justin Piszcz
2007-04-22 19:01 ` Mr. James W. Laferriere
2007-04-22 20:35 ` Justin Piszcz
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=Pine.LNX.4.64.0704220619390.14170@p34.internal.lan \
--to=jpiszcz@lucidpixels.com \
--cc=dap@mail.index.hu \
--cc=linux-raid@vger.kernel.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).