From: Spelic <spelic@shiftmail.org>
To: Jens Axboe <JAxboe@fusionio.com>
Cc: "fio@vger.kernel.org" <fio@vger.kernel.org>
Subject: Re: Again on IOPS higher than expected in randwrite 4k
Date: Wed, 5 Jan 2011 04:45:48 -0700 [thread overview]
Message-ID: <4D2459EC.1030308@shiftmail.org> (raw)
In-Reply-To: <4D21D8BE.1000403@fusionio.com>
On 01/03/2011 03:10 PM, Jens Axboe wrote:
> On 2011-01-03 12:26, Spelic wrote:
>
>> On 01/03/2011 12:05 PM, Jens Axboe wrote:
>>
>>> On 2011-01-02 05:12, Spelic wrote:
>>>
>> Oh I see.
>> But if I add fsync=1 I still get 300 IOPS per disk, or even 500 on
>> very short seeks, so again I'd say these disks are cheating. Do you
>> agree?
>>
> Did you verify that the fsync gets turned into a flush with eg blktrace?
> If it indeed is, then yes your number seems too high for that disk. With
> a SYNC_CACHE after each write, not even NCQ should be helping you (since
> each request will effectively be sync).
>
I'm not sure... I'm no expert of blktrace... can you have a look?
I am pasting part of the output below here.
(kernel 2.6.36.2)
I was issuing FIO to a lvm linear volume over a md raid10, with 20
threads doing 4K random writes in an 80MB file direct=1 and sync=1, I am
seeing about 450 IOPS per drive.
I captured one drive here.
There are many requests with "S", sync, but the S never reaches the "D"
driver. Does that mean there are no flushes?
I thought that DM linear and MD and raid10 in kernel 2.6.36.2 were
passing barriers to the layer below, so I thought that would work, or
anyway that the OS would work around that with some other technique so
to provide a reliable fsync (which is very important for data
consistency, isn't it).
Filesystem is ext4 mounted with defaults and it did not say nobarrier:
dmesg
[141306.496251] EXT4-fs (dm-2): mounted filesystem with ordered data
mode. Opts: (null)
cat /proc/mounts
/dev/mapper/datavg1-try2 /mnt/tmp2 ext4
rw,relatime,barrier=1,stripe=512,data=ordered 0 0
Another problem is: how come I see [swapper] operating... I had
specified direct=1 in fio!
[random-write]
rw=randwrite
rwmixcycle=1 #old settings...
rwmixread=50 #ignored in randwrite
numjobs=20
blocksize=4k
size=80m
directory=/mnt/tmp2/fio-data
direct=1
fsync=1
;iodepth=100 #this is commented out
end_fsync=1
blktrace --> blkparse:
66,144 5 5009 5.256475644 0 D W 20323192 + 8 [swapper]
66,144 7 3180 5.256676923 6134 U N [fio] 17
66,144 5 5010 5.256681988 701 A WS 33072392 + 8 <-
(66,145) 33070344
66,144 5 5011 5.256682210 701 Q WS 33072392 + 8 [md14_raid10]
66,144 5 5012 5.256682992 701 G WS 33072392 + 8 [md14_raid10]
66,144 5 5013 5.256683642 701 I W 33072392 + 8 [md14_raid10]
66,144 5 5014 5.256685095 701 U N [md14_raid10] 18
66,144 5 5015 5.256927083 6129 U N [fio] 18
66,144 7 3181 5.256933947 701 U N [md14_raid10] 18
66,144 5 5016 5.257087648 0 C W 19603416 + 8 [0]
66,144 5 5017 5.257092870 0 D W 21862904 + 8 [swapper]
66,144 5 5018 5.257238735 6116 U N [fio] 17
66,144 7 3182 5.257243081 701 A WS 19605560 + 8 <-
(66,145) 19603512
66,144 7 3183 5.257243235 701 Q WS 19605560 + 8 [md14_raid10]
66,144 7 3184 5.257243779 701 G WS 19605560 + 8 [md14_raid10]
66,144 7 3185 5.257244203 701 I W 19605560 + 8 [md14_raid10]
66,144 7 3186 5.257245545 701 U N [md14_raid10] 18
66,144 5 5019 5.257546967 6118 U N [fio] 18
66,144 7 3187 5.257552944 701 U N [md14_raid10] 18
66,144 5 5020 5.257663684 0 C W 20323192 + 8 [0]
66,144 5 5021 5.257668596 0 D W 22581240 + 8 [swapper]
66,144 7 3188 5.257816612 6117 U N [fio] 17
66,144 5 5022 5.257823723 701 U N [md14_raid10] 17
66,144 5 5023 5.258130728 6129 U N [fio] 17
66,144 7 3189 5.258137688 701 U N [md14_raid10] 17
66,144 5 5024 5.258242581 0 C W 21862904 + 8 [0]
66,144 5 5025 5.258247884 0 D W 23433784 + 8 [swapper]
66,144 5 5026 5.258398322 6119 U N [fio] 16
66,144 7 3190 5.258402511 701 A WS 21913520 + 8 <-
(66,145) 21911472
66,144 7 3191 5.258402650 701 Q WS 21913520 + 8 [md14_raid10]
66,144 7 3192 5.258403128 701 G WS 21913520 + 8 [md14_raid10]
66,144 7 3193 5.258403567 701 I W 21913520 + 8 [md14_raid10]
66,144 7 3194 5.258404755 701 U N [md14_raid10] 17
66,144 5 5027 5.258653030 6118 U N [fio] 17
66,144 7 3195 5.258657124 701 A WS 21150720 + 8 <-
(66,145) 21148672
66,144 7 3196 5.258657262 701 Q WS 21150720 + 8 [md14_raid10]
66,144 7 3197 5.258657659 701 G WS 21150720 + 8 [md14_raid10]
66,144 7 3198 5.258658017 701 I W 21150720 + 8 [md14_raid10]
66,144 7 3199 5.258659238 701 U N [md14_raid10] 18
66,144 5 5028 5.258769609 0 C W 22581240 + 8 [0]
66,144 5 5029 5.258775005 0 D W 24115736 + 8 [swapper]
66,144 7 3200 5.258927319 6120 U N [fio] 17
66,144 5 5030 5.258932228 701 A WS 22591840 + 8 <-
(66,145) 22589792
66,144 5 5031 5.258932562 701 Q WS 22591840 + 8 [md14_raid10]
66,144 5 5032 5.258933374 701 G WS 22591840 + 8 [md14_raid10]
66,144 5 5033 5.258933985 701 I W 22591840 + 8 [md14_raid10]
66,144 5 5034 5.258935275 701 U N [md14_raid10] 18
66,144 5 5035 5.259234667 6117 U N [fio] 18
66,144 7 3201 5.259241744 701 U N [md14_raid10] 18
66,144 5 5036 5.259377131 0 C W 23433784 + 8 [0]
66,144 5 5037 5.259382301 0 D W 24833912 + 8 [swapper]
66,144 5 5038 5.259536908 6121 U N [fio] 17
66,144 7 3202 5.259541161 701 A WS 23405256 + 8 <-
(66,145) 23403208
66,144 7 3203 5.259541335 701 Q WS 23405256 + 8 [md14_raid10]
66,144 7 3204 5.259541945 701 G WS 23405256 + 8 [md14_raid10]
66,144 7 3205 5.259542385 701 I W 23405256 + 8 [md14_raid10]
66,144 7 3206 5.259543684 701 U N [md14_raid10] 18
66,144 5 5039 5.259788494 6129 U N [fio] 18
66,144 7 3207 5.259794434 701 U N [md14_raid10] 18
66,144 5 5040 5.259858786 0 C W 24115736 + 8 [0]
66,144 5 5041 5.259863634 0 D W 25651296 + 8 [swapper]
66,144 7 3208 5.260001478 6122 U N [fio] 17
66,144 5 5042 5.260008880 701 U N [md14_raid10] 17
66,144 5 5043 5.260339045 6117 U N [fio] 17
66,144 7 3209 5.260346008 701 U N [md14_raid10] 17
66,144 5 5044 5.260485907 0 C W 24833912 + 8 [0]
66,144 5 5045 5.260490978 0 D W 26391696 + 8 [swapper]
66,144 5 5046 5.260643566 6123 U N [fio] 16
66,144 7 3210 5.260647837 701 A WS 24809912 + 8 <-
(66,145) 24807864
66,144 7 3211 5.260648008 701 Q WS 24809912 + 8 [md14_raid10]
66,144 7 3212 5.260648565 701 G WS 24809912 + 8 [md14_raid10]
66,144 7 3213 5.260648914 701 I W 24809912 + 8 [md14_raid10]
66,144 7 3214 5.260650315 701 U N [md14_raid10] 17
66,144 5 5047 5.260905234 6129 U N [fio] 17
66,144 7 3215 5.260910790 701 U N [md14_raid10] 17
66,144 5 5048 5.260967679 0 C W 25651296 + 8 [0]
66,144 5 5049 5.260973084 0 D W 27054336 + 8 [swapper]
66,144 7 3216 5.261122944 6124 U N [fio] 16
66,144 5 5050 5.261127859 701 A WS 25649792 + 8 <-
(66,145) 25647744
66,144 5 5051 5.261128084 701 Q WS 25649792 + 8 [md14_raid10]
66,144 5 5052 5.261128845 701 G WS 25649792 + 8 [md14_raid10]
66,144 5 5053 5.261129450 701 I W 25649792 + 8 [md14_raid10]
66,144 5 5054 5.261131041 701 U N [md14_raid10] 17
66,144 5 5055 5.261409090 6122 U N [fio] 17
66,144 7 3217 5.261413722 701 A WS 24147272 + 8 <-
(66,145) 24145224
66,144 7 3218 5.261414176 701 Q WS 24147272 + 8 [md14_raid10]
66,144 7 3219 5.261415006 701 G WS 24147272 + 8 [md14_raid10]
66,144 7 3220 5.261415614 701 I W 24147272 + 8 [md14_raid10]
66,144 7 3221 5.261417027 701 U N [md14_raid10] 18
66,144 5 5056 5.261520670 0 C W 26391696 + 8 [0]
66,144 5 5057 5.261525858 0 D W 27879824 + 8 [swapper]
66,144 5 5058 5.261678537 6125 U N [fio] 17
66,144 7 3222 5.261682847 701 A WS 26366992 + 8 <-
(66,145) 26364944
66,144 7 3223 5.261683021 701 Q WS 26366992 + 8 [md14_raid10]
66,144 7 3224 5.261683451 701 G WS 26366992 + 8 [md14_raid10]
66,144 7 3225 5.261683818 701 I W 26366992 + 8 [md14_raid10]
66,144 7 3226 5.261685265 701 U N [md14_raid10] 18
66,144 5 5059 5.261922669 6117 U N [fio] 18
66,144 7 3227 5.261926032 701 A WS 20358736 + 8 <-
(66,145) 20356688
66,144 7 3228 5.261926140 701 Q WS 20358736 + 8 [md14_raid10]
66,144 7 3229 5.261926489 701 G WS 20358736 + 8 [md14_raid10]
66,144 7 3230 5.261926748 701 I W 20358736 + 8 [md14_raid10]
66,144 7 3231 5.261927696 701 U N [md14_raid10] 19
66,144 5 5060 5.262033970 0 C W 27054336 + 8 [0]
66,144 5 5061 5.262038854 0 D W 28595752 + 8 [swapper]
66,144 7 3232 5.262194327 6126 U N [fio] 18
66,144 5 5062 5.262202033 701 U N [md14_raid10] 18
66,144 5 5063 5.262461921 6129 U N [fio] 18
66,144 7 3233 5.262469612 701 U N [md14_raid10] 18
66,144 5 5064 5.262623725 0 C W 27879824 + 8 [0]
66,144 5 5065 5.262628847 0 D W 30164048 + 8 [swapper]
66,144 5 5066 5.262829611 6127 U N [fio] 17
66,144 7 3234 5.262834041 701 A WS 27938424 + 8 <-
(66,145) 27936376
66,144 7 3235 5.262834192 701 Q WS 27938424 + 8 [md14_raid10]
66,144 7 3236 5.262834883 701 G WS 27938424 + 8 [md14_raid10]
66,144 7 3237 5.262835340 701 I W 27938424 + 8 [md14_raid10]
66,144 7 3238 5.262836673 701 U N [md14_raid10] 18
66,144 5 5067 5.263133683 6126 U N [fio] 18
66,144 7 3239 5.263137858 701 A WS 27061872 + 8 <-
(66,145) 27059824
66,144 7 3240 5.263137999 701 Q WS 27061872 + 8 [md14_raid10]
66,144 7 3241 5.263138426 701 G WS 27061872 + 8 [md14_raid10]
66,144 7 3242 5.263138826 701 I W 27061872 + 8 [md14_raid10]
66,144 7 3243 5.263139975 701 U N [md14_raid10] 19
66,144 5 5068 5.263253802 0 C W 28595752 + 8 [0]
66,144 5 5069 5.263259327 0 D W 30872560 + 8 [swapper]
66,144 7 3244 5.263456187 6128 U N [fio] 18
66,144 5 5070 5.263461225 701 A WS 28609536 + 8 <-
(66,145) 28607488
66,144 5 5071 5.263461496 701 Q WS 28609536 + 8 [md14_raid10]
66,144 5 5072 5.263462308 701 G WS 28609536 + 8 [md14_raid10]
66,144 5 5073 5.263462876 701 I W 28609536 + 8 [md14_raid10]
66,144 5 5074 5.263464227 701 U N [md14_raid10] 19
66,144 5 5075 5.263610357 6129 U N [fio] 19
66,144 7 3245 5.263617058 701 U N [md14_raid10] 19
66,144 5 5076 5.263726740 0 C W 30164048 + 8 [0]
66,144 5 5077 5.263731763 0 D W 31648664 + 8 [swapper]
66,144 5 5078 5.263910048 6130 U N [fio] 18
66,144 7 3246 5.263916821 701 U N [md14_raid10] 18
66,144 5 5079 5.264179263 6129 U N [fio] 18
66,144 7 3247 5.264185501 701 U N [md14_raid10] 18
66,144 5 5080 5.264291934 0 C W 30872560 + 8 [0]
66,144 5 5081 5.264296909 0 D W 32393312 + 8 [swapper]
66,144 7 3248 5.264396840 6131 U N [fio] 17
66,144 5 5082 5.264404167 701 U N [md14_raid10] 17
66,144 5 5083 5.264712968 6130 U N [fio] 17
66,144 7 3249 5.264719768 701 U N [md14_raid10] 17
66,144 5 5084 5.264821254 0 C W 31648664 + 8 [0]
66,144 5 5085 5.264826506 0 D W 33072392 + 8 [swapper]
66,144 5 5086 5.265026755 6132 U N [fio] 16
66,144 7 3250 5.265032611 701 U N [md14_raid10] 16
66,144 5 5087 5.265294126 6129 U N [fio] 16
66,144 7 3251 5.265299691 701 U N [md14_raid10] 16
66,144 5 5088 5.265406558 0 C W 32393312 + 8 [0]
66,144 5 5089 5.265411617 0 D W 18769952 + 8 [swapper]
66,144 7 3252 5.265569929 6133 U N [fio] 15
66,144 5 5090 5.265577207 701 U N [md14_raid10] 15
66,144 5 5091 5.265923350 6131 U N [fio] 15
66,144 7 3253 5.265930496 701 U N [md14_raid10] 15
66,144 5 5092 5.266041020 0 C W 33072392 + 8 [0]
66,144 5 5093 5.266046233 0 D W 19605560 + 8 [swapper]
66,144 5 5094 5.266200776 6134 U N [fio] 14
66,144 7 3254 5.266206557 701 U N [md14_raid10] 14
66,144 5 5095 5.266454790 6130 U N [fio] 14
66,144 7 3255 5.266459012 701 A WS 30133304 + 8 <-
(66,145) 30131256
66,144 7 3256 5.266459253 701 Q WS 30133304 + 8 [md14_raid10]
66,144 7 3257 5.266459930 701 G WS 30133304 + 8 [md14_raid10]
66,144 7 3258 5.266460309 701 I W 30133304 + 8 [md14_raid10]
66,144 7 3259 5.266461473 701 U N [md14_raid10] 15
66,144 5 5096 5.266572863 0 C W 18769952 + 8 [0]
66,144 5 5097 5.266578124 0 D W 20358736 + 8 [swapper]
66,144 7 3260 5.266732219 6115 U N [fio] 14
66,144 5 5098 5.266737163 701 A WS 18758136 + 8 <-
(66,145) 18756088
66,144 5 5099 5.266737386 701 Q WS 18758136 + 8 [md14_raid10]
66,144 5 5100 5.266738168 701 G WS 18758136 + 8 [md14_raid10]
66,144 5 5101 5.266738776 701 I W 18758136 + 8 [md14_raid10]
66,144 5 5102 5.266740268 701 U N [md14_raid10] 15
66,144 5 5103 5.266997433 6132 U N [fio] 15
66,144 7 3261 5.267004469 701 U N [md14_raid10] 15
66,144 5 5104 5.267107097 0 C W 19605560 + 8 [0]
66,144 5 5105 5.267112123 0 D W 21150720 + 8 [swapper]
66,144 5 5106 5.267261824 6116 U N [fio] 14
66,144 7 3262 5.267267689 701 U N [md14_raid10] 14
66,144 5 5107 5.267565634 6133 U N [fio] 14
Thank you
prev parent reply other threads:[~2011-01-05 11:45 UTC|newest]
Thread overview: 5+ messages / expand[flat|nested] mbox.gz Atom feed top
2011-01-02 4:12 Again on IOPS higher than expected in randwrite 4k Spelic
2011-01-03 11:05 ` Jens Axboe
2011-01-03 11:26 ` Spelic
2011-01-03 14:10 ` Jens Axboe
2011-01-05 11:45 ` Spelic [this message]
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=4D2459EC.1030308@shiftmail.org \
--to=spelic@shiftmail.org \
--cc=JAxboe@fusionio.com \
--cc=fio@vger.kernel.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox