* IOZone Performance is very Strange
@ 2011-05-17 3:53 AnnyRen
2011-05-17 13:23 ` Jeff Wu
0 siblings, 1 reply; 13+ messages in thread
From: AnnyRen @ 2011-05-17 3:53 UTC (permalink / raw)
To: ceph-devel
[-- Attachment #1: Type: text/plain, Size: 3777 bytes --]
I'm running iozone on EXT4 with Ceph v0.26.
But I got the weird result, most write performance exceed 1GB/s, even
up to 3GB/s
I think it's not normal to get the performance outpupt.
Command line I used is: /usr/bin/iozone -azcR -f
/mnt/ceph/test0516.dbf -g 1G -b /exceloutput0516.xls
Attachment is the output file...
and my environment is composed of 15 physical machines with
3 MON, 2MDS (1 active, 1 standby), 10 OSD (1 osd daemon (3T) /host)
EXT4 format
data replication size: 3
"Writer report"
"4" "8" "16" "32" "64" "128" "256" "512" "1024"
"2048" "4096" "8192" "16384"
"64" 983980 1204796 1210227 1143223 1357066
"128" 1007629 1269781 1406136 1391557 1436229 1521718
"256" 1112909 1430119 1523457 1652404 1514860 1639786 1729594
"512" 1150351 1475116 1605228 1723770 1797349 1712772 1783912 1854787
"1024" 1213334 1471481 1679160 1828574 1888889 1899750 1885572
1865912 1875690
"2048" 1229274 1540849 1708146 1843410 1903457 1980705 1930406
1913634 1906837 1815744
"4096" 1213284 1528348 1674646 1762434 1872096 1882352 1881528
1903416 1897949 1835102 1731177
"8192" 204560 155186 572387 238548 186597 429036 187327
157205 553771 416512 299810 405842
"16384" 699749 559255 687450 541030 828776 555296 742561
525483 604910 452423 564557 670539 970616
"32768" 532414 829610 812215 879441 863159 864794 865938
804951 916352 879582 608132 860732 1239475
"65536" 994824 1096543 1095791 1317968 1280277 1390267 1259868
1205214 1339111 1346927 1267888 863234 1190221
"131072" 1063429 1165115 1102650 1554828 1182128 1185731
1190752 1195792 1277441 1211063 1237567 1226999 1336961
"262144" 1280619 1368554 1497837 1633397 1598255 1609212
1607504 1665019 1590515 1548307 1591258 1505267 1625679
"524288" 1519583 1767928 1738523 1883151 2011216 1993877
2023543 1867440 2106124 2055064 1906668 1778645 1838988
"1048576" 1580851 1887530 2044131 2166133 2236379 2283578
2257454 2296612 2271066 2101274 1905829 1605923 2158238
"Reader report"
"4" "8" "16" "32" "64" "128" "256" "512" "1024"
"2048" "4096" "8192" "16384"
"64" 1933893 2801873 3057153 3363612 3958892
"128" 2286447 3053774 2727923 3468030 4104338 4557257
"256" 2903529 3236056 3245838 3705040 3654598 4496299 5117791
"512" 2906696 3437042 3628697 3431550 4871723 4296637 6246213 6395018
"1024" 3229770 3483896 4609294 3791442 4614246 5536137 4550690
5048117 4966395
"2048" 3554251 4310472 3885431 4096676 6401772 4842658 5080379
5184636 5596757 5735012
"4096" 3416292 4691638 4321103 5728903 5475122 5171846 4819300
5258919 6408472 5044289 4079948
"8192" 3233004 4615263 4536055 5618186 5414558 5025700 5553712
4926264 5634770 5281396 4659702 3652258
"16384" 3141058 3704193 4567654 4395850 4568869 5387732
4436432 5808029 5578420 4675810 3913007 3911225 3961277
"32768" 3704273 4598957 4088278 5133719 5896692 5537024
5234412 5398271 4942992 4118662 3729099 3511757 3481511
"65536" 4131091 4210184 5341188 4647619 6077765 5852474
5379762 5259330 5488249 5246682 4342682 3549202 3286487
"131072" 3582876 5251082 5332216 5269908 5303512 5574440
5635064 5796372 5406363 4958839 4435918 3673443 3647874
"262144" 3659283 4551414 5746231 5433824 5876196 6011650
5552000 5629260 5298830 4982226 4628902 4065823 3421924
"524288" 3905973 5488778 5219963 6047356 5916811 6180455
5495733 5925628 5637512 5537123 3517132 3550861 3047013
"1048576" 3855595 5634571 5410298 6001809 6464713 6299610
5894249 5516031 5800161 5209203 4295840 3724983 3641623
[-- Attachment #2: outputlog0516.log --]
[-- Type: application/octet-stream, Size: 45342 bytes --]
Iozone: Performance Test of File I/O
Version $Revision: 3.373 $
Compiled for 64 bit mode.
Build: linux-AMD64
Contributors:William Norcott, Don Capps, Isom Crawford, Kirby Collins
Al Slater, Scott Rhine, Mike Wisner, Ken Goss
Steve Landherr, Brad Smith, Mark Kelly, Dr. Alain CYR,
Randy Dunlap, Mark Montague, Dan Million, Gavin Brebner,
Jean-Marc Zucconi, Jeff Blomberg, Benny Halevy, Dave Boone,
Erik Habbinga, Kris Strecker, Walter Wong, Joshua Root,
Fabrice Bacchella, Zhenghua Xue, Qin Li, Darren Sawyer.
Run began: Mon May 16 12:02:37 2011
Auto Mode
Cross over of record size disabled.
Include close in write timing
Excel chart generation enabled
Using maximum file size of 1048576 kilobytes.
Command line used: /usr/bin/iozone -azcR -f /mnt/ceph/test0516.dbf -g 1G -b /exceloutput0516.xls
Output is in Kbytes/sec
Time Resolution = 0.000001 seconds.
Processor cache size set to 1024 Kbytes.
Processor cache line size set to 32 bytes.
File stride size set to 17 * record size.
random random bkwd record stride
KB reclen write rewrite read reread read write read rewrite read fwrite frewrite fread freread
64 4 983980 969761 1933893 2772930 2006158 1310683 1996 1609270 1828508 1232453 969761 1638743 19459
64 8 1204796 1143223 2801873 3541098 4274062 2772930 1879725 3203069 3057153 1828508 1279447 2006158 3588436
64 16 1210227 1599680 3057153 3363612 2689580 1828508 1750 2222043 2662899 1357066 1232453 2467108 2662899
64 32 1143223 1279447 3363612 5735102 4018152 4988978 2067979 3057153 2801873 1690338 1526887 2561267 4274062
64 64 1357066 1599680 3958892 4018152 3057153 2203800 1865 2222043 2662899 2067979 1722886 2662899 3203069
128 4 1007629 1162547 2286447 4012317 3380677 2132084 1858934 2558895 2166499 1705404 1145188 1967960 2464907
128 8 1269781 1391557 3053774 3560017 2905056 2098744 3522 2409592 3199360 1939522 1377278 2727923 3124872
128 16 1406136 1487977 2727923 4934216 7176872 4759253 2248149 4557257 3277486 2843510 1406136 2248149 3657016
128 32 1391557 1557024 3468030 4012317 3536566 2464907 3463 2714132 3124872 2035099 1561553 2905056 3536566
128 64 1436229 1802755 4104338 6406138 8548124 5545860 2727923 5603747 3359523 1802755 1584598 2660335 4407601
128 128 1521718 2035099 4557257 4759253 3867787 2727923 3762 2784517 3468030 2673584 2202044 3560017 3895854
256 4 1112909 1506359 2903529 4009406 3410808 2187712 2097948 1952947 2392442 2783115 1334162 2533571 3654598
256 8 1430119 1817419 3236056 3823789 3326279 2242541 6561 3009318 2911402 2192179 1649865 3052087 3557725
256 16 1523457 1842366 3245838 3879045 3454704 2665657 6402 2872459 3326279 2305128 1855098 2943325 4332998
256 32 1652404 2065659 3705040 4350555 3994490 2719671 6638 3410808 3770085 2419396 2205688 3935921 4054829
256 64 1514860 1952947 3654598 9114515 8534922 5810112 3087188 6761355 3994490 3511189 1763685 2880164 4404088
256 128 1639786 2210228 4496299 5117791 4477549 3078337 7874 3378611 3994490 1778290 2187712 3705040 4496299
256 256 1729594 2205688 5117791 5687020 5347168 3410808 3511189 4197489 3465855 3935921 2371308 3454704 7735574
512 4 1150351 1766304 2906696 3325278 2962840 2099600 13813 2024389 2438090 1723770 1446306 2668325 2811557
512 8 1475116 1827947 3437042 5214798 4701087 2991733 2991733 4099771 3239989 3710197 1707325 2753870 4262523
512 16 1605228 2370799 3628697 3815675 3678421 2860238 14750 3259661 3580298 2497638 2073249 3064306 3659616
512 32 1723770 2097550 3431550 5384786 5177083 3304808 3140488 4340054 3934520 4411377 1985091 3182373 6174377
512 64 1797349 2205248 4871723 4784885 4494470 3239989 12748 3604335 4494470 2782414 2796910 4571003 4650188
512 128 1712772 2327124 4296637 5699180 5509113 3634839 3580298 7211022 4571003 3304808 2178404 4000485 6319739
512 256 1783912 2297251 6246213 5384786 4927617 3372274 12852 3653390 4701087 1933266 2521095 4305250 4927617
512 512 1854787 2753870 6395018 6174377 6472112 4195896 3884698 4262523 3736016 4195896 2910635 3736016 6491676
1024 4 1213334 1524224 3229770 3012308 2516377 1786736 28483 2039583 2491561 1815435 1695735 2774874 2877110
1024 8 1471481 1924436 3483896 4374559 4146499 2701567 3029305 3469823 3379719 2316837 2090206 3849207 6830356
1024 16 1679160 2414523 4609294 4634161 4493556 3335105 29687 3151562 3544270 2619192 2353657 4033569 4110780
1024 32 1828574 3010197 3791442 5417427 5303701 3618930 3794792 5120336 5144870 4550690 2461573 3923040 6172653
1024 64 1888889 2353657 4614246 5630486 4805258 3469823 29659 3866533 5336651 2993413 2679655 4720752 5169641
1024 128 1899750 3261656 5536137 8593274 8326715 6093831 4300102 5682634 5983467 4550690 2586074 4995276 8542002
1024 256 1885572 2842832 4550690 5444898 5226256 3495237 29877 3999762 6317933 2089189 2586074 4674510 6402699
1024 512 1865912 2918161 5048117 5593820 4199201 2892612 4526709 4392454 4943530 2414523 3048658 4470172 6093831
1024 1024 1875690 2908281 4966395 5690162 5593820 3696803 25736 3738636 4995276 3684119 3220084 4995276 5564829
2048 4 1229274 2072957 3554251 5389574 4674016 2836079 3038742 3710866 3931669 2798201 1741742 2929912 3835124
2048 8 1540849 2265394 4310472 5489457 4960102 3219850 3848872 4223576 5159722 3937075 2206617 3777778 4481379
2048 16 1708146 2275596 3885431 5937172 5655716 3777778 3235617 4796687 4581773 3487865 2611870 4697018 5769681
2048 32 1843410 2485653 4096676 5689430 5489457 3717289 3922692 4643695 6185123 4572018 3240499 5095447 8758039
2048 64 1903457 3430751 6401772 9389900 9189005 6225466 6058611 7969835 6738230 4923146 2968386 4653758 5937172
2048 128 1980705 2809182 4842658 6807652 6649556 4730646 4633675 6058611 8190205 4674016 3246623 5507054 8949659
2048 256 1930406 2942962 5080379 6649556 6588354 4493100 4842658 5416763 5884299 3386122 2734946 4889518 6588354
2048 512 1913634 2872113 5184636 6690992 7165410 5719737 4728042 5068389 5884299 3245396 2522144 4603874 5674397
2048 1024 1906837 2848305 5596757 6711905 6649556 4312636 4663865 4775354 5596757 2029850 2233004 4751580 6588354
2048 2048 1815744 2715922 5735012 5937172 5868220 3857514 4453499 3878414 4572018 5172149 3251539 4762117 6225466
4096 4 1213284 1785514 3416292 3799338 3393349 1901520 2577699 2345757 2694963 2508081 1584010 2929815 3098371
4096 8 1528348 2264141 4691638 6493246 6300364 3909130 3849566 4257917 4320017 2860067 2289792 4230655 4900405
4096 16 1674646 2274332 4321103 4819300 4757904 3107900 3961410 4043458 4244242 2737912 2221111 4226492 4723889
4096 32 1762434 2471992 5728903 7954358 8173861 5265366 5404504 7038605 7351860 4526013 2576153 4681411 5634950
4096 64 1872096 2854365 5475122 6522830 6585338 4418911 5107271 5859825 7342434 4740836 3391339 6408472 8360824
4096 128 1882352 2925823 5171846 6370451 6459069 4581535 5786794 8096815 7447471 4423462 3139133 4653513 5431844
4096 256 1881528 2807257 4819300 6151473 6351609 4266376 4703198 5490871 6703539 3287506 2652931 4725189 6041154
4096 512 1903416 2923832 5258919 6112079 6370451 4140919 5000244 5153230 6254490 3370712 2840678 5019235 6007355
4096 1024 1897949 2771033 6408472 7756829 7834651 5190597 5026578 5056165 6254490 2491711 2114496 5657217 7434579
4096 2048 1835102 2715838 5044289 5490871 5306022 3512686 5340661 5551194 6351609 2116580 2355405 5145513 6555186
4096 4096 1731177 2610602 4079948 4171080 4125010 2886496 4141917 2921346 4176149 3381993 2987904 4196552 4240052
8192 4 204560 1723951 3233004 4216019 3996338 2351344 2993059 3245524 3723500 2464511 1733081 3123099 3910378
8192 8 155186 2656378 4615263 5143053 4926264 3155510 3899284 5756551 4943985 3106440 2238301 3636025 3775461
8192 16 572387 2281401 4536055 4844308 4809724 3164228 5091232 6253130 5664497 2623119 2390441 5512724 6145757
8192 32 238548 3054523 5618186 5944791 6345516 3893540 5385702 7008178 6229323 3593060 2579403 6045199 6439465
8192 64 186597 2793973 5414558 5897851 4903066 3047749 5357152 6814966 6233844 3661989 3229054 5361332 6234975
8192 128 429036 2908931 5025700 5375591 5472336 3596068 5720134 8104083 6720329 3703433 2713438 5700206 6527546
8192 256 187327 2790116 5553712 6262248 6580049 4250966 5728718 7320230 6307078 3517291 3268680 5708730 6085887
8192 512 157205 3371310 4926264 5211701 5323125 3488721 5457560 6895660 8577485 3558822 2677075 5400092 6040947
8192 1024 553771 2777710 5634770 6224809 6334986 4158362 5443725 6781341 8144423 2691123 2166056 4352746 4636437
8192 2048 416512 2739834 5281396 5684175 5319828 3804725 4478702 4483963 5454094 2091432 1989238 5155399 5565406
8192 4096 299810 2844630 4659702 4844991 4617744 3465148 4435342 3897073 4242568 1774247 1923535 3972772 4273171
8192 8192 405842 2333935 3652258 3876846 3880349 2975434 3725519 2477126 3760997 3027341 2790116 3508313 3588557
16384 4 699749 1721179 3141058 4280039 3751717 2203903 2688155 3690071 3582721 2270831 2003144 3880525 4145769
16384 8 559255 2281234 3704193 4454549 4740972 2944556 3788324 5730536 4366522 2911126 2721153 4580441 4787877
16384 16 687450 2252373 4567654 4672948 5170063 3284743 4920199 6463575 5033728 3084384 2917924 5496324 5628681
16384 32 541030 2414021 4395850 4437578 5137595 3597726 5317690 7440550 4667870 2969751 2613921 4650496 5919089
16384 64 828776 3238001 4568869 6236563 6289648 3758488 5777267 8027220 4779552 2524891 2867396 5657411 6145651
16384 128 555296 3118395 5387732 5614884 5643936 3935641 4794893 6930296 5628220 3376744 3199408 5589764 5811959
16384 256 742561 3322701 4436432 5298828 6021260 3949893 5378876 4863096 4969299 3098849 3010077 5647647 5915013
16384 512 525483 3063620 5808029 5944691 6131941 3990259 5639767 7025952 4638880 3278162 3176338 5432453 5655548
16384 1024 604910 3149552 5578420 5822300 5882607 3909445 5421738 7001613 8183027 2608463 2364190 5153778 5342079
16384 2048 452423 2752213 4675810 4827227 5497643 3740689 4549812 5754529 7071501 2325311 2254516 5189585 5343741
16384 4096 564557 2282826 3913007 4070623 4084657 3105992 3600931 3792506 4198966 2065520 2027972 3503092 3869600
16384 8192 670539 2312167 3911225 3915906 3916798 3011792 3725277 3262443 3402660 1661503 1999007 3814611 3893497
16384 16384 970616 2745725 3961277 3953756 4007713 3068134 3927993 3081341 3983551 3099548 2597716 3938574 4005844
32768 4 532414 1976833 3704273 3793113 3297257 2125307 3158005 3609993 3188187 2214371 1861186 3628003 3700982
32768 8 829610 2431050 4598957 4976281 4593270 2807179 4422130 5794514 3802348 2724973 2703426 4577818 4906815
32768 16 812215 2849136 4088278 4733266 4984583 3129957 4281746 6442610 5409107 2938562 2889570 4318475 5573625
32768 32 879441 2634050 5133719 4420992 5356822 3437709 4383765 7275441 4831606 3148600 3097159 5713813 5824472
32768 64 863159 2657170 5896692 5943613 5934117 3702577 5899982 8051111 4793522 3310521 3251776 5905306 5929765
32768 128 864794 3323972 5537024 5944898 5913691 3783403 5428763 8049225 5835849 3276818 3063334 5616675 5662493
32768 256 865938 2878917 5234412 5871500 5866989 3868163 5131036 7425196 4967109 3152862 2989505 4954932 5840064
32768 512 804951 3338586 5398271 5700541 5752796 3844682 4754553 6960060 4858250 2993347 2874882 5345571 5412515
32768 1024 916352 2706300 4942992 5489695 5494304 3819466 5549096 6657636 4525960 2495677 2179432 5418063 5537917
32768 2048 879582 2826054 4118662 4496787 4821606 3498170 4755211 4998906 4748967 2153952 1923358 4448610 3986302
32768 4096 608132 2414225 3729099 4183858 4145245 3160474 4324726 3859581 4393013 2059078 1931494 4227095 4392591
32768 8192 860732 2395333 3511757 3808881 3843392 2994651 3458731 3206260 3573387 2060869 1977800 3584010 3542076
32768 16384 1239475 2363857 3481511 3963311 3958517 3083747 3922366 3216691 3468945 1763896 1903591 3903760 3927073
65536 4 994824 2155282 4131091 4267508 3577518 2103606 3438217 3687171 3279899 2216224 1912116 4093685 4131898
65536 8 1096543 2518285 4210184 4885606 4396582 2759297 4276072 5667725 4177996 2738542 2706961 4207799 4660226
65536 16 1095791 2756530 5341188 5880772 5398357 3074944 4816774 6435190 4859178 2931599 2906398 4972753 5754339
65536 32 1317968 2995656 4647619 5834215 5778654 3398849 5775133 7234427 4286206 3004169 2787926 5930764 5851478
65536 64 1280277 2927914 6077765 6161136 5949248 3604445 5523096 7997889 5834338 3237483 3219359 4799448 5672637
65536 128 1390267 3278295 5852474 5923989 5852474 3766234 5852973 8104714 5701819 3218039 3170273 5669127 5990477
65536 256 1259868 3256121 5379762 5640394 5598464 3788190 5052920 7407749 4414092 3138240 3028897 5170492 5655015
65536 512 1205214 3284838 5259330 5668777 5717593 3829353 4700952 7100989 5141671 3141720 3066300 5334140 5594703
65536 1024 1339111 3358033 5488249 5666673 5691430 3807552 5528539 6607196 5296624 2496715 2423843 5529985 5587539
65536 2048 1346927 3060258 5246682 5286031 5230309 3604870 4815592 5390734 4891518 2160415 2092270 4860209 5176529
65536 4096 1267888 2841825 4342682 4525051 4459132 3279077 4225133 3792371 4563439 2130278 2065356 4014468 4254232
65536 8192 863234 2672848 3549202 3641934 3629000 2881960 3494293 3132090 3457070 1983463 2043203 3859137 3866193
65536 16384 1190221 2410558 3286487 3361524 3400405 2764069 3543848 3019680 3486669 1990587 1953328 3470294 3581993
131072 4 1063429 2122209 3582876 3725319 3141770 2079251 3427259 3733820 3508539 2219757 2205119 3585610 3648794
131072 8 1165115 2590770 5251082 5532311 4792370 2737339 4392220 5686415 4147322 2734534 2726979 4687997 4767434
131072 16 1102650 2938506 5332216 5369348 4971620 3075577 5205636 6565047 4872952 2933833 2927474 4953031 5555063
131072 32 1554828 3095179 5269908 5975749 5826473 3460025 5111211 7370980 5505607 3109956 3020506 5863511 5928960
131072 64 1182128 3209244 5303512 5886426 5685650 3632328 5604960 8189444 5566200 3258066 3242578 5789657 5839222
131072 128 1185731 3292506 5574440 6134311 5992946 3724688 5294115 8202151 5500043 3252169 3245008 5492680 5822092
131072 256 1190752 3282206 5635064 5671397 5713897 3789307 5474793 7460000 5366937 3161027 2965020 5453179 5563327
131072 512 1195792 3103478 5796372 5816672 5775910 3844209 5744226 7094190 5310839 3175965 3115208 5608047 5813535
131072 1024 1277441 3233309 5406363 5636566 5638011 3818655 5420222 6876090 5451990 2514378 2504139 5381964 5524139
131072 2048 1211063 3086819 4958839 5012279 4961300 3540853 4918247 5155887 4556473 2214580 2138871 4919567 4951336
131072 4096 1237567 2989317 4435918 4518164 4430948 3266022 4095356 3876028 4223763 2102736 2088429 4382940 4412172
131072 8192 1226999 2657804 3673443 3724789 3824233 3000935 3597341 3327299 3742258 1996101 2051757 4058082 4099724
131072 16384 1336961 2773717 3647874 3659895 3646882 2924547 3625071 3131730 3551031 2018668 2072909 3660723 3666779
262144 4 1280619 2154638 3659283 3787438 3389413 2031115 3379277 3677618 3114492 2189661 2179085 3790624 3841225
262144 8 1368554 2698845 4551414 5529885 4674718 2650889 3966351 5575763 4010886 2743156 2677426 4951443 5017878
262144 16 1497837 2913263 5746231 5926685 5440465 3039770 4665158 6368154 4585256 2906348 2892075 5802208 5845704
262144 32 1633397 3067441 5433824 5513497 5162225 3394813 5388922 7357801 5409949 3136812 3120910 5909611 6135194
262144 64 1598255 3192247 5876196 5935451 5682970 3612242 6136050 7885432 5954319 3270177 3214957 5854295 6242408
262144 128 1609212 3368944 6011650 6169690 5951870 3742613 5662221 7936316 5499708 3225946 3171575 5310321 5567829
262144 256 1607504 3452895 5552000 5637746 5559440 3821054 5383724 7232972 5307527 3176000 3184647 5417786 5459754
262144 512 1665019 3345551 5629260 5681472 5892351 3839535 5568647 6938347 5853017 3155428 3136561 5645620 5711072
262144 1024 1590515 3421392 5298830 5589766 5564166 3779847 5001990 7043464 5180100 2468400 2498397 5398474 5579441
262144 2048 1548307 3208933 4982226 5026067 4975215 3514885 4973324 6185830 5160407 2226729 2172927 4839446 5212319
262144 4096 1591258 2883678 4628902 4822550 4736970 3432639 4322033 3858980 4171677 2046304 2117170 4631027 4787045
262144 8192 1505267 2604713 4065823 4086996 4058589 3128244 3660099 3184831 2566267 1673587 2016927 4104465 4146474
262144 16384 1625679 2807556 3421924 3602159 3750106 3081471 3910086 3261795 3809286 2083948 1981766 3389758 3600425
524288 4 1519583 2170921 3905973 3935472 3072605 1982306 3319101 3649426 3348842 2204333 2216599 3801120 3847871
524288 8 1767928 2649566 5488778 5648314 4649384 2627956 4915272 5384934 4528001 2747608 2732901 5448726 5525270
524288 16 1738523 2993261 5219963 5319259 4815810 3010241 5518115 6244425 5083027 2933752 2884790 5220272 5259338
524288 32 1883151 3202894 6047356 6251508 5751925 3382458 6005802 7324710 5809870 3139314 3069658 5740019 5868713
524288 64 2011216 3325706 5916811 6050717 5724538 3616530 5902613 8245467 6050301 3287483 3265204 6323742 6491048
524288 128 1993877 3409223 6180455 6307146 6074468 3768329 5830156 8346142 5436307 3268567 3162766 5922356 6110149
524288 256 2023543 3473554 5495733 5510814 5451103 3837229 5726924 7445915 5780775 3222164 3218009 5491533 5730043
524288 512 1867440 3498683 5925628 6008411 5972883 3866259 5948357 6682482 5820234 3196199 3148046 5751489 6003801
524288 1024 2106124 3411402 5637512 5837415 5835153 3848295 5184812 7057226 5791768 2549344 2484825 5559788 5582824
524288 2048 2055064 3292893 5537123 5576298 5522023 3735872 5510054 6329075 5471489 2297191 2273059 5526089 5558383
524288 4096 1906668 2453451 3517132 3374490 3091191 2429103 3543275 3078972 3298652 1537975 1524114 3452666 3326365
524288 8192 1778645 2479136 3550861 3597691 3703619 2825756 3720311 2850958 3536391 1739883 1729039 3713639 3791937
524288 16384 1838988 2496003 3047013 3192612 3302581 2680601 3033040 2615945 3241949 1699933 1683459 3361147 3375858
1048576 4 1580851 2122536 3855595 4287055 3251396 1894535 3948544 3489010 3204978 2120718 2127533 4155346 4039790
1048576 8 1887530 2606050 5634571 5587369 4319819 2576234 5005995 5264544 4801634 2689787 2708155 5442286 5583893
1048576 16 2044131 2986791 5410298 5664915 5430754 2999412 5547493 6453453 4985572 2917500 2909656 5220067 4931690
1048576 32 2166133 3136933 6001809 6308212 5818793 3335027 6116572 6982823 5352851 3111538 3089954 6351118 6375525
1048576 64 2236379 3300325 6464713 6522786 6205367 3516420 6296652 8104565 5809577 3242070 3241587 6326897 6390328
1048576 128 2283578 3342971 6299610 6376218 6170014 3774107 6287326 8333550 5832451 3260285 3282739 6371922 6412410
1048576 256 2257454 3332929 5894249 5769936 5534899 3769223 5906940 6947470 5552809 3172772 3157808 5641936 5979971
1048576 512 2296612 3273964 5516031 5480658 5451967 3816526 5973124 7071305 5681373 3183696 3180790 5993989 5952405
1048576 1024 2271066 3433190 5800161 5931260 5926776 3844937 5902112 6670449 5914661 2298445 2277048 5695469 5967986
1048576 2048 2101274 3235126 5209203 5240812 4815192 3204557 5393465 6053431 5615255 1961164 1943185 5327395 5580853
1048576 4096 1905829 2833751 4295840 4320409 4266995 3145297 4249238 3475502 4041891 1629435 1732427 4292251 4334853
1048576 8192 1605923 2475877 3724983 3732703 3508456 2635944 3465509 2669428 3502831 1705151 1588722 3261796 3286443
1048576 16384 2158238 2523236 3641623 3654465 3633608 2897521 3563566 3349418 3545228 2029194 2077713 3681899 3987039
iozone test complete.
Excel output is below:
"Writer report"
"4" "8" "16" "32" "64" "128" "256" "512" "1024" "2048" "4096" "8192" "16384"
"64" 983980 1204796 1210227 1143223 1357066
"128" 1007629 1269781 1406136 1391557 1436229 1521718
"256" 1112909 1430119 1523457 1652404 1514860 1639786 1729594
"512" 1150351 1475116 1605228 1723770 1797349 1712772 1783912 1854787
"1024" 1213334 1471481 1679160 1828574 1888889 1899750 1885572 1865912 1875690
"2048" 1229274 1540849 1708146 1843410 1903457 1980705 1930406 1913634 1906837 1815744
"4096" 1213284 1528348 1674646 1762434 1872096 1882352 1881528 1903416 1897949 1835102 1731177
"8192" 204560 155186 572387 238548 186597 429036 187327 157205 553771 416512 299810 405842
"16384" 699749 559255 687450 541030 828776 555296 742561 525483 604910 452423 564557 670539 970616
"32768" 532414 829610 812215 879441 863159 864794 865938 804951 916352 879582 608132 860732 1239475
"65536" 994824 1096543 1095791 1317968 1280277 1390267 1259868 1205214 1339111 1346927 1267888 863234 1190221
"131072" 1063429 1165115 1102650 1554828 1182128 1185731 1190752 1195792 1277441 1211063 1237567 1226999 1336961
"262144" 1280619 1368554 1497837 1633397 1598255 1609212 1607504 1665019 1590515 1548307 1591258 1505267 1625679
"524288" 1519583 1767928 1738523 1883151 2011216 1993877 2023543 1867440 2106124 2055064 1906668 1778645 1838988
"1048576" 1580851 1887530 2044131 2166133 2236379 2283578 2257454 2296612 2271066 2101274 1905829 1605923 2158238
"Re-writer report"
"4" "8" "16" "32" "64" "128" "256" "512" "1024" "2048" "4096" "8192" "16384"
"64" 969761 1143223 1599680 1279447 1599680
"128" 1162547 1391557 1487977 1557024 1802755 2035099
"256" 1506359 1817419 1842366 2065659 1952947 2210228 2205688
"512" 1766304 1827947 2370799 2097550 2205248 2327124 2297251 2753870
"1024" 1524224 1924436 2414523 3010197 2353657 3261656 2842832 2918161 2908281
"2048" 2072957 2265394 2275596 2485653 3430751 2809182 2942962 2872113 2848305 2715922
"4096" 1785514 2264141 2274332 2471992 2854365 2925823 2807257 2923832 2771033 2715838 2610602
"8192" 1723951 2656378 2281401 3054523 2793973 2908931 2790116 3371310 2777710 2739834 2844630 2333935
"16384" 1721179 2281234 2252373 2414021 3238001 3118395 3322701 3063620 3149552 2752213 2282826 2312167 2745725
"32768" 1976833 2431050 2849136 2634050 2657170 3323972 2878917 3338586 2706300 2826054 2414225 2395333 2363857
"65536" 2155282 2518285 2756530 2995656 2927914 3278295 3256121 3284838 3358033 3060258 2841825 2672848 2410558
"131072" 2122209 2590770 2938506 3095179 3209244 3292506 3282206 3103478 3233309 3086819 2989317 2657804 2773717
"262144" 2154638 2698845 2913263 3067441 3192247 3368944 3452895 3345551 3421392 3208933 2883678 2604713 2807556
"524288" 2170921 2649566 2993261 3202894 3325706 3409223 3473554 3498683 3411402 3292893 2453451 2479136 2496003
"1048576" 2122536 2606050 2986791 3136933 3300325 3342971 3332929 3273964 3433190 3235126 2833751 2475877 2523236
"Reader report"
"4" "8" "16" "32" "64" "128" "256" "512" "1024" "2048" "4096" "8192" "16384"
"64" 1933893 2801873 3057153 3363612 3958892
"128" 2286447 3053774 2727923 3468030 4104338 4557257
"256" 2903529 3236056 3245838 3705040 3654598 4496299 5117791
"512" 2906696 3437042 3628697 3431550 4871723 4296637 6246213 6395018
"1024" 3229770 3483896 4609294 3791442 4614246 5536137 4550690 5048117 4966395
"2048" 3554251 4310472 3885431 4096676 6401772 4842658 5080379 5184636 5596757 5735012
"4096" 3416292 4691638 4321103 5728903 5475122 5171846 4819300 5258919 6408472 5044289 4079948
"8192" 3233004 4615263 4536055 5618186 5414558 5025700 5553712 4926264 5634770 5281396 4659702 3652258
"16384" 3141058 3704193 4567654 4395850 4568869 5387732 4436432 5808029 5578420 4675810 3913007 3911225 3961277
"32768" 3704273 4598957 4088278 5133719 5896692 5537024 5234412 5398271 4942992 4118662 3729099 3511757 3481511
"65536" 4131091 4210184 5341188 4647619 6077765 5852474 5379762 5259330 5488249 5246682 4342682 3549202 3286487
"131072" 3582876 5251082 5332216 5269908 5303512 5574440 5635064 5796372 5406363 4958839 4435918 3673443 3647874
"262144" 3659283 4551414 5746231 5433824 5876196 6011650 5552000 5629260 5298830 4982226 4628902 4065823 3421924
"524288" 3905973 5488778 5219963 6047356 5916811 6180455 5495733 5925628 5637512 5537123 3517132 3550861 3047013
"1048576" 3855595 5634571 5410298 6001809 6464713 6299610 5894249 5516031 5800161 5209203 4295840 3724983 3641623
"Re-Reader report"
"4" "8" "16" "32" "64" "128" "256" "512" "1024" "2048" "4096" "8192" "16384"
"64" 2772930 3541098 3363612 5735102 4018152
"128" 4012317 3560017 4934216 4012317 6406138 4759253
"256" 4009406 3823789 3879045 4350555 9114515 5117791 5687020
"512" 3325278 5214798 3815675 5384786 4784885 5699180 5384786 6174377
"1024" 3012308 4374559 4634161 5417427 5630486 8593274 5444898 5593820 5690162
"2048" 5389574 5489457 5937172 5689430 9389900 6807652 6649556 6690992 6711905 5937172
"4096" 3799338 6493246 4819300 7954358 6522830 6370451 6151473 6112079 7756829 5490871 4171080
"8192" 4216019 5143053 4844308 5944791 5897851 5375591 6262248 5211701 6224809 5684175 4844991 3876846
"16384" 4280039 4454549 4672948 4437578 6236563 5614884 5298828 5944691 5822300 4827227 4070623 3915906 3953756
"32768" 3793113 4976281 4733266 4420992 5943613 5944898 5871500 5700541 5489695 4496787 4183858 3808881 3963311
"65536" 4267508 4885606 5880772 5834215 6161136 5923989 5640394 5668777 5666673 5286031 4525051 3641934 3361524
"131072" 3725319 5532311 5369348 5975749 5886426 6134311 5671397 5816672 5636566 5012279 4518164 3724789 3659895
"262144" 3787438 5529885 5926685 5513497 5935451 6169690 5637746 5681472 5589766 5026067 4822550 4086996 3602159
"524288" 3935472 5648314 5319259 6251508 6050717 6307146 5510814 6008411 5837415 5576298 3374490 3597691 3192612
"1048576" 4287055 5587369 5664915 6308212 6522786 6376218 5769936 5480658 5931260 5240812 4320409 3732703 3654465
"Random read report"
"4" "8" "16" "32" "64" "128" "256" "512" "1024" "2048" "4096" "8192" "16384"
"64" 2006158 4274062 2689580 4018152 3057153
"128" 3380677 2905056 7176872 3536566 8548124 3867787
"256" 3410808 3326279 3454704 3994490 8534922 4477549 5347168
"512" 2962840 4701087 3678421 5177083 4494470 5509113 4927617 6472112
"1024" 2516377 4146499 4493556 5303701 4805258 8326715 5226256 4199201 5593820
"2048" 4674016 4960102 5655716 5489457 9189005 6649556 6588354 7165410 6649556 5868220
"4096" 3393349 6300364 4757904 8173861 6585338 6459069 6351609 6370451 7834651 5306022 4125010
"8192" 3996338 4926264 4809724 6345516 4903066 5472336 6580049 5323125 6334986 5319828 4617744 3880349
"16384" 3751717 4740972 5170063 5137595 6289648 5643936 6021260 6131941 5882607 5497643 4084657 3916798 4007713
"32768" 3297257 4593270 4984583 5356822 5934117 5913691 5866989 5752796 5494304 4821606 4145245 3843392 3958517
"65536" 3577518 4396582 5398357 5778654 5949248 5852474 5598464 5717593 5691430 5230309 4459132 3629000 3400405
"131072" 3141770 4792370 4971620 5826473 5685650 5992946 5713897 5775910 5638011 4961300 4430948 3824233 3646882
"262144" 3389413 4674718 5440465 5162225 5682970 5951870 5559440 5892351 5564166 4975215 4736970 4058589 3750106
"524288" 3072605 4649384 4815810 5751925 5724538 6074468 5451103 5972883 5835153 5522023 3091191 3703619 3302581
"1048576" 3251396 4319819 5430754 5818793 6205367 6170014 5534899 5451967 5926776 4815192 4266995 3508456 3633608
"Random write report"
"4" "8" "16" "32" "64" "128" "256" "512" "1024" "2048" "4096" "8192" "16384"
"64" 1310683 2772930 1828508 4988978 2203800
"128" 2132084 2098744 4759253 2464907 5545860 2727923
"256" 2187712 2242541 2665657 2719671 5810112 3078337 3410808
"512" 2099600 2991733 2860238 3304808 3239989 3634839 3372274 4195896
"1024" 1786736 2701567 3335105 3618930 3469823 6093831 3495237 2892612 3696803
"2048" 2836079 3219850 3777778 3717289 6225466 4730646 4493100 5719737 4312636 3857514
"4096" 1901520 3909130 3107900 5265366 4418911 4581535 4266376 4140919 5190597 3512686 2886496
"8192" 2351344 3155510 3164228 3893540 3047749 3596068 4250966 3488721 4158362 3804725 3465148 2975434
"16384" 2203903 2944556 3284743 3597726 3758488 3935641 3949893 3990259 3909445 3740689 3105992 3011792 3068134
"32768" 2125307 2807179 3129957 3437709 3702577 3783403 3868163 3844682 3819466 3498170 3160474 2994651 3083747
"65536" 2103606 2759297 3074944 3398849 3604445 3766234 3788190 3829353 3807552 3604870 3279077 2881960 2764069
"131072" 2079251 2737339 3075577 3460025 3632328 3724688 3789307 3844209 3818655 3540853 3266022 3000935 2924547
"262144" 2031115 2650889 3039770 3394813 3612242 3742613 3821054 3839535 3779847 3514885 3432639 3128244 3081471
"524288" 1982306 2627956 3010241 3382458 3616530 3768329 3837229 3866259 3848295 3735872 2429103 2825756 2680601
"1048576" 1894535 2576234 2999412 3335027 3516420 3774107 3769223 3816526 3844937 3204557 3145297 2635944 2897521
"Backward read report"
"4" "8" "16" "32" "64" "128" "256" "512" "1024" "2048" "4096" "8192" "16384"
"64" 1996 1879725 1750 2067979 1865
"128" 1858934 3522 2248149 3463 2727923 3762
"256" 2097948 6561 6402 6638 3087188 7874 3511189
"512" 13813 2991733 14750 3140488 12748 3580298 12852 3884698
"1024" 28483 3029305 29687 3794792 29659 4300102 29877 4526709 25736
"2048" 3038742 3848872 3235617 3922692 6058611 4633675 4842658 4728042 4663865 4453499
"4096" 2577699 3849566 3961410 5404504 5107271 5786794 4703198 5000244 5026578 5340661 4141917
"8192" 2993059 3899284 5091232 5385702 5357152 5720134 5728718 5457560 5443725 4478702 4435342 3725519
"16384" 2688155 3788324 4920199 5317690 5777267 4794893 5378876 5639767 5421738 4549812 3600931 3725277 3927993
"32768" 3158005 4422130 4281746 4383765 5899982 5428763 5131036 4754553 5549096 4755211 4324726 3458731 3922366
"65536" 3438217 4276072 4816774 5775133 5523096 5852973 5052920 4700952 5528539 4815592 4225133 3494293 3543848
"131072" 3427259 4392220 5205636 5111211 5604960 5294115 5474793 5744226 5420222 4918247 4095356 3597341 3625071
"262144" 3379277 3966351 4665158 5388922 6136050 5662221 5383724 5568647 5001990 4973324 4322033 3660099 3910086
"524288" 3319101 4915272 5518115 6005802 5902613 5830156 5726924 5948357 5184812 5510054 3543275 3720311 3033040
"1048576" 3948544 5005995 5547493 6116572 6296652 6287326 5906940 5973124 5902112 5393465 4249238 3465509 3563566
"Record rewrite report"
"4" "8" "16" "32" "64" "128" "256" "512" "1024" "2048" "4096" "8192" "16384"
"64" 1609270 3203069 2222043 3057153 2222043
"128" 2558895 2409592 4557257 2714132 5603747 2784517
"256" 1952947 3009318 2872459 3410808 6761355 3378611 4197489
"512" 2024389 4099771 3259661 4340054 3604335 7211022 3653390 4262523
"1024" 2039583 3469823 3151562 5120336 3866533 5682634 3999762 4392454 3738636
"2048" 3710866 4223576 4796687 4643695 7969835 6058611 5416763 5068389 4775354 3878414
"4096" 2345757 4257917 4043458 7038605 5859825 8096815 5490871 5153230 5056165 5551194 2921346
"8192" 3245524 5756551 6253130 7008178 6814966 8104083 7320230 6895660 6781341 4483963 3897073 2477126
"16384" 3690071 5730536 6463575 7440550 8027220 6930296 4863096 7025952 7001613 5754529 3792506 3262443 3081341
"32768" 3609993 5794514 6442610 7275441 8051111 8049225 7425196 6960060 6657636 4998906 3859581 3206260 3216691
"65536" 3687171 5667725 6435190 7234427 7997889 8104714 7407749 7100989 6607196 5390734 3792371 3132090 3019680
"131072" 3733820 5686415 6565047 7370980 8189444 8202151 7460000 7094190 6876090 5155887 3876028 3327299 3131730
"262144" 3677618 5575763 6368154 7357801 7885432 7936316 7232972 6938347 7043464 6185830 3858980 3184831 3261795
"524288" 3649426 5384934 6244425 7324710 8245467 8346142 7445915 6682482 7057226 6329075 3078972 2850958 2615945
"1048576" 3489010 5264544 6453453 6982823 8104565 8333550 6947470 7071305 6670449 6053431 3475502 2669428 3349418
"Stride read report"
"4" "8" "16" "32" "64" "128" "256" "512" "1024" "2048" "4096" "8192" "16384"
"64" 1828508 3057153 2662899 2801873 2662899
"128" 2166499 3199360 3277486 3124872 3359523 3468030
"256" 2392442 2911402 3326279 3770085 3994490 3994490 3465855
"512" 2438090 3239989 3580298 3934520 4494470 4571003 4701087 3736016
"1024" 2491561 3379719 3544270 5144870 5336651 5983467 6317933 4943530 4995276
"2048" 3931669 5159722 4581773 6185123 6738230 8190205 5884299 5884299 5596757 4572018
"4096" 2694963 4320017 4244242 7351860 7342434 7447471 6703539 6254490 6254490 6351609 4176149
"8192" 3723500 4943985 5664497 6229323 6233844 6720329 6307078 8577485 8144423 5454094 4242568 3760997
"16384" 3582721 4366522 5033728 4667870 4779552 5628220 4969299 4638880 8183027 7071501 4198966 3402660 3983551
"32768" 3188187 3802348 5409107 4831606 4793522 5835849 4967109 4858250 4525960 4748967 4393013 3573387 3468945
"65536" 3279899 4177996 4859178 4286206 5834338 5701819 4414092 5141671 5296624 4891518 4563439 3457070 3486669
"131072" 3508539 4147322 4872952 5505607 5566200 5500043 5366937 5310839 5451990 4556473 4223763 3742258 3551031
"262144" 3114492 4010886 4585256 5409949 5954319 5499708 5307527 5853017 5180100 5160407 4171677 2566267 3809286
"524288" 3348842 4528001 5083027 5809870 6050301 5436307 5780775 5820234 5791768 5471489 3298652 3536391 3241949
"1048576" 3204978 4801634 4985572 5352851 5809577 5832451 5552809 5681373 5914661 5615255 4041891 3502831 3545228
"Fwrite report"
"4" "8" "16" "32" "64" "128" "256" "512" "1024" "2048" "4096" "8192" "16384"
"64" 1232453 1828508 1357066 1690338 2067979
"128" 1705404 1939522 2843510 2035099 1802755 2673584
"256" 2783115 2192179 2305128 2419396 3511189 1778290 3935921
"512" 1723770 3710197 2497638 4411377 2782414 3304808 1933266 4195896
"1024" 1815435 2316837 2619192 4550690 2993413 4550690 2089189 2414523 3684119
"2048" 2798201 3937075 3487865 4572018 4923146 4674016 3386122 3245396 2029850 5172149
"4096" 2508081 2860067 2737912 4526013 4740836 4423462 3287506 3370712 2491711 2116580 3381993
"8192" 2464511 3106440 2623119 3593060 3661989 3703433 3517291 3558822 2691123 2091432 1774247 3027341
"16384" 2270831 2911126 3084384 2969751 2524891 3376744 3098849 3278162 2608463 2325311 2065520 1661503 3099548
"32768" 2214371 2724973 2938562 3148600 3310521 3276818 3152862 2993347 2495677 2153952 2059078 2060869 1763896
"65536" 2216224 2738542 2931599 3004169 3237483 3218039 3138240 3141720 2496715 2160415 2130278 1983463 1990587
"131072" 2219757 2734534 2933833 3109956 3258066 3252169 3161027 3175965 2514378 2214580 2102736 1996101 2018668
"262144" 2189661 2743156 2906348 3136812 3270177 3225946 3176000 3155428 2468400 2226729 2046304 1673587 2083948
"524288" 2204333 2747608 2933752 3139314 3287483 3268567 3222164 3196199 2549344 2297191 1537975 1739883 1699933
"1048576" 2120718 2689787 2917500 3111538 3242070 3260285 3172772 3183696 2298445 1961164 1629435 1705151 2029194
"Re-Fwrite report"
"4" "8" "16" "32" "64" "128" "256" "512" "1024" "2048" "4096" "8192" "16384"
"64" 969761 1279447 1232453 1526887 1722886
"128" 1145188 1377278 1406136 1561553 1584598 2202044
"256" 1334162 1649865 1855098 2205688 1763685 2187712 2371308
"512" 1446306 1707325 2073249 1985091 2796910 2178404 2521095 2910635
"1024" 1695735 2090206 2353657 2461573 2679655 2586074 2586074 3048658 3220084
"2048" 1741742 2206617 2611870 3240499 2968386 3246623 2734946 2522144 2233004 3251539
"4096" 1584010 2289792 2221111 2576153 3391339 3139133 2652931 2840678 2114496 2355405 2987904
"8192" 1733081 2238301 2390441 2579403 3229054 2713438 3268680 2677075 2166056 1989238 1923535 2790116
"16384" 2003144 2721153 2917924 2613921 2867396 3199408 3010077 3176338 2364190 2254516 2027972 1999007 2597716
"32768" 1861186 2703426 2889570 3097159 3251776 3063334 2989505 2874882 2179432 1923358 1931494 1977800 1903591
"65536" 1912116 2706961 2906398 2787926 3219359 3170273 3028897 3066300 2423843 2092270 2065356 2043203 1953328
"131072" 2205119 2726979 2927474 3020506 3242578 3245008 2965020 3115208 2504139 2138871 2088429 2051757 2072909
"262144" 2179085 2677426 2892075 3120910 3214957 3171575 3184647 3136561 2498397 2172927 2117170 2016927 1981766
"524288" 2216599 2732901 2884790 3069658 3265204 3162766 3218009 3148046 2484825 2273059 1524114 1729039 1683459
"1048576" 2127533 2708155 2909656 3089954 3241587 3282739 3157808 3180790 2277048 1943185 1732427 1588722 2077713
"Fread report"
"4" "8" "16" "32" "64" "128" "256" "512" "1024" "2048" "4096" "8192" "16384"
"64" 1638743 2006158 2467108 2561267 2662899
"128" 1967960 2727923 2248149 2905056 2660335 3560017
"256" 2533571 3052087 2943325 3935921 2880164 3705040 3454704
"512" 2668325 2753870 3064306 3182373 4571003 4000485 4305250 3736016
"1024" 2774874 3849207 4033569 3923040 4720752 4995276 4674510 4470172 4995276
"2048" 2929912 3777778 4697018 5095447 4653758 5507054 4889518 4603874 4751580 4762117
"4096" 2929815 4230655 4226492 4681411 6408472 4653513 4725189 5019235 5657217 5145513 4196552
"8192" 3123099 3636025 5512724 6045199 5361332 5700206 5708730 5400092 4352746 5155399 3972772 3508313
"16384" 3880525 4580441 5496324 4650496 5657411 5589764 5647647 5432453 5153778 5189585 3503092 3814611 3938574
"32768" 3628003 4577818 4318475 5713813 5905306 5616675 4954932 5345571 5418063 4448610 4227095 3584010 3903760
"65536" 4093685 4207799 4972753 5930764 4799448 5669127 5170492 5334140 5529985 4860209 4014468 3859137 3470294
"131072" 3585610 4687997 4953031 5863511 5789657 5492680 5453179 5608047 5381964 4919567 4382940 4058082 3660723
"262144" 3790624 4951443 5802208 5909611 5854295 5310321 5417786 5645620 5398474 4839446 4631027 4104465 3389758
"524288" 3801120 5448726 5220272 5740019 6323742 5922356 5491533 5751489 5559788 5526089 3452666 3713639 3361147
"1048576" 4155346 5442286 5220067 6351118 6326897 6371922 5641936 5993989 5695469 5327395 4292251 3261796 3681899
"Re-Fread report"
"4" "8" "16" "32" "64" "128" "256" "512" "1024" "2048" "4096" "8192" "16384"
"64" 19459 3588436 2662899 4274062 3203069
"128" 2464907 3124872 3657016 3536566 4407601 3895854
"256" 3654598 3557725 4332998 4054829 4404088 4496299 7735574
"512" 2811557 4262523 3659616 6174377 4650188 6319739 4927617 6491676
"1024" 2877110 6830356 4110780 6172653 5169641 8542002 6402699 6093831 5564829
"2048" 3835124 4481379 5769681 8758039 5937172 8949659 6588354 5674397 6588354 6225466
"4096" 3098371 4900405 4723889 5634950 8360824 5431844 6041154 6007355 7434579 6555186 4240052
"8192" 3910378 3775461 6145757 6439465 6234975 6527546 6085887 6040947 4636437 5565406 4273171 3588557
"16384" 4145769 4787877 5628681 5919089 6145651 5811959 5915013 5655548 5342079 5343741 3869600 3893497 4005844
"32768" 3700982 4906815 5573625 5824472 5929765 5662493 5840064 5412515 5537917 3986302 4392591 3542076 3927073
"65536" 4131898 4660226 5754339 5851478 5672637 5990477 5655015 5594703 5587539 5176529 4254232 3866193 3581993
"131072" 3648794 4767434 5555063 5928960 5839222 5822092 5563327 5813535 5524139 4951336 4412172 4099724 3666779
"262144" 3841225 5017878 5845704 6135194 6242408 5567829 5459754 5711072 5579441 5212319 4787045 4146474 3600425
"524288" 3847871 5525270 5259338 5868713 6491048 6110149 5730043 6003801 5582824 5558383 3326365 3791937 3375858
"1048576" 4039790 5583893 4931690 6375525 6390328 6412410 5979971 5952405 5967986 5580853 4334853 3286443 3987039
^ permalink raw reply [flat|nested] 13+ messages in thread* Re: IOZone Performance is very Strange 2011-05-17 3:53 IOZone Performance is very Strange AnnyRen @ 2011-05-17 13:23 ` Jeff Wu 2011-05-17 7:34 ` AnnyRen 0 siblings, 1 reply; 13+ messages in thread From: Jeff Wu @ 2011-05-17 13:23 UTC (permalink / raw) To: AnnyRen; +Cc: ceph-devel@vger.kernel.org Hi AnnyRen Could you run the following commands and give us the test results? $ceph osd tell OSD-N bench // OSD-N : osd number : 0,1,2 .... $ceph -w $rados bench 60 write -p data // refer to "rados -h " Jeff On Tue, 2011-05-17 at 11:53 +0800, AnnyRen wrote: > I'm running iozone on EXT4 with Ceph v0.26. > But I got the weird result, most write performance exceed 1GB/s, even > up to 3GB/s > I think it's not normal to get the performance outpupt. > > Command line I used is: /usr/bin/iozone -azcR -f > /mnt/ceph/test0516.dbf -g 1G -b /exceloutput0516.xls > Attachment is the output file... > > and my environment is composed of 15 physical machines with > > 3 MON, 2MDS (1 active, 1 standby), 10 OSD (1 osd daemon (3T) /host) > EXT4 format > data replication size: 3 > > > > "Writer report" > "4" "8" "16" "32" "64" "128" "256" "512" "1024" > "2048" "4096" "8192" "16384" > "64" 983980 1204796 1210227 1143223 1357066 > "128" 1007629 1269781 1406136 1391557 1436229 1521718 > "256" 1112909 1430119 1523457 1652404 1514860 1639786 1729594 > "512" 1150351 1475116 1605228 1723770 1797349 1712772 1783912 1854787 > "1024" 1213334 1471481 1679160 1828574 1888889 1899750 1885572 > 1865912 1875690 > "2048" 1229274 1540849 1708146 1843410 1903457 1980705 1930406 > 1913634 1906837 1815744 > "4096" 1213284 1528348 1674646 1762434 1872096 1882352 1881528 > 1903416 1897949 1835102 1731177 > "8192" 204560 155186 572387 238548 186597 429036 187327 > 157205 553771 416512 299810 405842 > "16384" 699749 559255 687450 541030 828776 555296 742561 > 525483 604910 452423 564557 670539 970616 > "32768" 532414 829610 812215 879441 863159 864794 865938 > 804951 916352 879582 608132 860732 1239475 > "65536" 994824 1096543 1095791 1317968 1280277 1390267 1259868 > 1205214 1339111 1346927 1267888 863234 1190221 > "131072" 1063429 1165115 1102650 1554828 1182128 1185731 > 1190752 1195792 1277441 1211063 1237567 1226999 1336961 > "262144" 1280619 1368554 1497837 1633397 1598255 1609212 > 1607504 1665019 1590515 1548307 1591258 1505267 1625679 > "524288" 1519583 1767928 1738523 1883151 2011216 1993877 > 2023543 1867440 2106124 2055064 1906668 1778645 1838988 > "1048576" 1580851 1887530 2044131 2166133 2236379 2283578 > 2257454 2296612 2271066 2101274 1905829 1605923 2158238 > > > > "Reader report" > "4" "8" "16" "32" "64" "128" "256" "512" "1024" > "2048" "4096" "8192" "16384" > "64" 1933893 2801873 3057153 3363612 3958892 > "128" 2286447 3053774 2727923 3468030 4104338 4557257 > "256" 2903529 3236056 3245838 3705040 3654598 4496299 5117791 > "512" 2906696 3437042 3628697 3431550 4871723 4296637 6246213 6395018 > "1024" 3229770 3483896 4609294 3791442 4614246 5536137 4550690 > 5048117 4966395 > "2048" 3554251 4310472 3885431 4096676 6401772 4842658 5080379 > 5184636 5596757 5735012 > "4096" 3416292 4691638 4321103 5728903 5475122 5171846 4819300 > 5258919 6408472 5044289 4079948 > "8192" 3233004 4615263 4536055 5618186 5414558 5025700 5553712 > 4926264 5634770 5281396 4659702 3652258 > "16384" 3141058 3704193 4567654 4395850 4568869 5387732 > 4436432 5808029 5578420 4675810 3913007 3911225 3961277 > "32768" 3704273 4598957 4088278 5133719 5896692 5537024 > 5234412 5398271 4942992 4118662 3729099 3511757 3481511 > "65536" 4131091 4210184 5341188 4647619 6077765 5852474 > 5379762 5259330 5488249 5246682 4342682 3549202 3286487 > "131072" 3582876 5251082 5332216 5269908 5303512 5574440 > 5635064 5796372 5406363 4958839 4435918 3673443 3647874 > "262144" 3659283 4551414 5746231 5433824 5876196 6011650 > 5552000 5629260 5298830 4982226 4628902 4065823 3421924 > "524288" 3905973 5488778 5219963 6047356 5916811 6180455 > 5495733 5925628 5637512 5537123 3517132 3550861 3047013 > "1048576" 3855595 5634571 5410298 6001809 6464713 6299610 > 5894249 5516031 5800161 5209203 4295840 3724983 3641623 ^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: IOZone Performance is very Strange 2011-05-17 13:23 ` Jeff Wu @ 2011-05-17 7:34 ` AnnyRen 2011-05-17 8:19 ` Simon Tian 2011-05-17 17:05 ` Jeff Wu 0 siblings, 2 replies; 13+ messages in thread From: AnnyRen @ 2011-05-17 7:34 UTC (permalink / raw) To: cpwu; +Cc: ceph-devel@vger.kernel.org Hi, Jeff: I run "ceph osd tell osd_num bench" with 1 times per osd and use ceph -w to observe every osd performance, osd0: [INF] bench: wrote 1024 MB in blocks of 4096 KB in 10.875844 sec at 96413 KB/sec osd1: [INF] bench: wrote 1024 MB in blocks of 4096 KB in 11.784985 sec at 88975 KB/sec osd2: [INF] bench: wrote 1024 MB in blocks of 4096 KB in 11.161067 sec at 93949 KB/sec osd3: [INF] bench: wrote 1024 MB in blocks of 4096 KB in 10.798796 sec at 97101 KB/sec osd4: [INF] bench: wrote 1024 MB in blocks of 4096 KB in 14.437141 sec at 72630 KB/sec osd5: [INF] bench: wrote 1024 MB in blocks of 4096 KB in 14.451444 sec at 72558 KB/sec osd6: [INF] bench: wrote 1024 MB in blocks of 4096 KB in 11.083872 sec at 94603 KB/sec osd7: [INF] bench: wrote 1024 MB in blocks of 4096 KB in 11.062728 sec at 94784 KB/sec osd8: [INF] bench: wrote 1024 MB in blocks of 4096 KB in 14.137312 sec at 74170 KB/sec osd9: [INF] bench: wrote 1024 MB in blocks of 4096 KB in 13.489992 sec at 77729 KB/sec and I run root@MDS2:/mnt/ceph# rados bench 60 write -p data the result is Total time run: 60.553247 Total writes made: 1689 Write size: 4194304 Bandwidth (MB/sec): 111.571 Average Latency: 0.573209 Max latency: 2.25691 Min latency: 0.218494 2011/5/17 Jeff Wu <cpwu@tnsoft.com.cn>: > Hi AnnyRen > > Could you run the following commands and give us the test results? > > $ceph osd tell OSD-N bench // OSD-N : osd number : 0,1,2 .... > $ceph -w > > $rados bench 60 write -p data // refer to "rados -h " > > Jeff > > > > On Tue, 2011-05-17 at 11:53 +0800, AnnyRen wrote: >> I'm running iozone on EXT4 with Ceph v0.26. >> But I got the weird result, most write performance exceed 1GB/s, even >> up to 3GB/s >> I think it's not normal to get the performance outpupt. >> >> Command line I used is: /usr/bin/iozone -azcR -f >> /mnt/ceph/test0516.dbf -g 1G -b /exceloutput0516.xls >> Attachment is the output file... >> >> and my environment is composed of 15 physical machines with >> >> 3 MON, 2MDS (1 active, 1 standby), 10 OSD (1 osd daemon (3T) /host) >> EXT4 format >> data replication size: 3 >> >> >> >> "Writer report" >> "4" "8" "16" "32" "64" "128" "256" "512" "1024" >> "2048" "4096" "8192" "16384" >> "64" 983980 1204796 1210227 1143223 1357066 >> "128" 1007629 1269781 1406136 1391557 1436229 1521718 >> "256" 1112909 1430119 1523457 1652404 1514860 1639786 1729594 >> "512" 1150351 1475116 1605228 1723770 1797349 1712772 1783912 1854787 >> "1024" 1213334 1471481 1679160 1828574 1888889 1899750 1885572 >> 1865912 1875690 >> "2048" 1229274 1540849 1708146 1843410 1903457 1980705 1930406 >> 1913634 1906837 1815744 >> "4096" 1213284 1528348 1674646 1762434 1872096 1882352 1881528 >> 1903416 1897949 1835102 1731177 >> "8192" 204560 155186 572387 238548 186597 429036 187327 >> 157205 553771 416512 299810 405842 >> "16384" 699749 559255 687450 541030 828776 555296 742561 >> 525483 604910 452423 564557 670539 970616 >> "32768" 532414 829610 812215 879441 863159 864794 865938 >> 804951 916352 879582 608132 860732 1239475 >> "65536" 994824 1096543 1095791 1317968 1280277 1390267 1259868 >> 1205214 1339111 1346927 1267888 863234 1190221 >> "131072" 1063429 1165115 1102650 1554828 1182128 1185731 >> 1190752 1195792 1277441 1211063 1237567 1226999 1336961 >> "262144" 1280619 1368554 1497837 1633397 1598255 1609212 >> 1607504 1665019 1590515 1548307 1591258 1505267 1625679 >> "524288" 1519583 1767928 1738523 1883151 2011216 1993877 >> 2023543 1867440 2106124 2055064 1906668 1778645 1838988 >> "1048576" 1580851 1887530 2044131 2166133 2236379 2283578 >> 2257454 2296612 2271066 2101274 1905829 1605923 2158238 >> >> >> >> "Reader report" >> "4" "8" "16" "32" "64" "128" "256" "512" "1024" >> "2048" "4096" "8192" "16384" >> "64" 1933893 2801873 3057153 3363612 3958892 >> "128" 2286447 3053774 2727923 3468030 4104338 4557257 >> "256" 2903529 3236056 3245838 3705040 3654598 4496299 5117791 >> "512" 2906696 3437042 3628697 3431550 4871723 4296637 6246213 6395018 >> "1024" 3229770 3483896 4609294 3791442 4614246 5536137 4550690 >> 5048117 4966395 >> "2048" 3554251 4310472 3885431 4096676 6401772 4842658 5080379 >> 5184636 5596757 5735012 >> "4096" 3416292 4691638 4321103 5728903 5475122 5171846 4819300 >> 5258919 6408472 5044289 4079948 >> "8192" 3233004 4615263 4536055 5618186 5414558 5025700 5553712 >> 4926264 5634770 5281396 4659702 3652258 >> "16384" 3141058 3704193 4567654 4395850 4568869 5387732 >> 4436432 5808029 5578420 4675810 3913007 3911225 3961277 >> "32768" 3704273 4598957 4088278 5133719 5896692 5537024 >> 5234412 5398271 4942992 4118662 3729099 3511757 3481511 >> "65536" 4131091 4210184 5341188 4647619 6077765 5852474 >> 5379762 5259330 5488249 5246682 4342682 3549202 3286487 >> "131072" 3582876 5251082 5332216 5269908 5303512 5574440 >> 5635064 5796372 5406363 4958839 4435918 3673443 3647874 >> "262144" 3659283 4551414 5746231 5433824 5876196 6011650 >> 5552000 5629260 5298830 4982226 4628902 4065823 3421924 >> "524288" 3905973 5488778 5219963 6047356 5916811 6180455 >> 5495733 5925628 5637512 5537123 3517132 3550861 3047013 >> "1048576" 3855595 5634571 5410298 6001809 6464713 6299610 >> 5894249 5516031 5800161 5209203 4295840 3724983 3641623 > > -- To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html ^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: IOZone Performance is very Strange 2011-05-17 7:34 ` AnnyRen @ 2011-05-17 8:19 ` Simon Tian 2011-05-17 9:07 ` AnnyRen 2011-05-17 17:05 ` Jeff Wu 1 sibling, 1 reply; 13+ messages in thread From: Simon Tian @ 2011-05-17 8:19 UTC (permalink / raw) To: AnnyRen; +Cc: cpwu, ceph-devel@vger.kernel.org Hi, The throughput of EXT4 is nice.... How about btrfs? Thx! Simon 2011/5/17 AnnyRen <annyren6@gmail.com>: > Hi, Jeff: > > I run "ceph osd tell osd_num bench" with 1 times per osd > > and use ceph -w to observe every osd performance, > > osd0: [INF] bench: wrote 1024 MB in blocks of 4096 KB in 10.875844 > sec at 96413 KB/sec > osd1: [INF] bench: wrote 1024 MB in blocks of 4096 KB in 11.784985 > sec at 88975 KB/sec > osd2: [INF] bench: wrote 1024 MB in blocks of 4096 KB in 11.161067 > sec at 93949 KB/sec > osd3: [INF] bench: wrote 1024 MB in blocks of 4096 KB in 10.798796 > sec at 97101 KB/sec > osd4: [INF] bench: wrote 1024 MB in blocks of 4096 KB in 14.437141 > sec at 72630 KB/sec > osd5: [INF] bench: wrote 1024 MB in blocks of 4096 KB in 14.451444 > sec at 72558 KB/sec > osd6: [INF] bench: wrote 1024 MB in blocks of 4096 KB in 11.083872 > sec at 94603 KB/sec > osd7: [INF] bench: wrote 1024 MB in blocks of 4096 KB in 11.062728 > sec at 94784 KB/sec > osd8: [INF] bench: wrote 1024 MB in blocks of 4096 KB in 14.137312 > sec at 74170 KB/sec > osd9: [INF] bench: wrote 1024 MB in blocks of 4096 KB in 13.489992 > sec at 77729 KB/sec > > > and I run > root@MDS2:/mnt/ceph# rados bench 60 write -p data > > the result is > > Total time run: 60.553247 > Total writes made: 1689 > Write size: 4194304 > Bandwidth (MB/sec): 111.571 > > Average Latency: 0.573209 > Max latency: 2.25691 > Min latency: 0.218494 > > > > 2011/5/17 Jeff Wu <cpwu@tnsoft.com.cn>: >> Hi AnnyRen >> >> Could you run the following commands and give us the test results? >> >> $ceph osd tell OSD-N bench // OSD-N : osd number : 0,1,2 .... >> $ceph -w >> >> $rados bench 60 write -p data // refer to "rados -h " >> >> Jeff >> >> >> >> On Tue, 2011-05-17 at 11:53 +0800, AnnyRen wrote: >>> I'm running iozone on EXT4 with Ceph v0.26. >>> But I got the weird result, most write performance exceed 1GB/s, even >>> up to 3GB/s >>> I think it's not normal to get the performance outpupt. >>> >>> Command line I used is: /usr/bin/iozone -azcR -f >>> /mnt/ceph/test0516.dbf -g 1G -b /exceloutput0516.xls >>> Attachment is the output file... >>> >>> and my environment is composed of 15 physical machines with >>> >>> 3 MON, 2MDS (1 active, 1 standby), 10 OSD (1 osd daemon (3T) /host) >>> EXT4 format >>> data replication size: 3 >>> >>> >>> >>> "Writer report" >>> "4" "8" "16" "32" "64" "128" "256" "512" "1024" >>> "2048" "4096" "8192" "16384" >>> "64" 983980 1204796 1210227 1143223 1357066 >>> "128" 1007629 1269781 1406136 1391557 1436229 1521718 >>> "256" 1112909 1430119 1523457 1652404 1514860 1639786 1729594 >>> "512" 1150351 1475116 1605228 1723770 1797349 1712772 1783912 1854787 >>> "1024" 1213334 1471481 1679160 1828574 1888889 1899750 1885572 >>> 1865912 1875690 >>> "2048" 1229274 1540849 1708146 1843410 1903457 1980705 1930406 >>> 1913634 1906837 1815744 >>> "4096" 1213284 1528348 1674646 1762434 1872096 1882352 1881528 >>> 1903416 1897949 1835102 1731177 >>> "8192" 204560 155186 572387 238548 186597 429036 187327 >>> 157205 553771 416512 299810 405842 >>> "16384" 699749 559255 687450 541030 828776 555296 742561 >>> 525483 604910 452423 564557 670539 970616 >>> "32768" 532414 829610 812215 879441 863159 864794 865938 >>> 804951 916352 879582 608132 860732 1239475 >>> "65536" 994824 1096543 1095791 1317968 1280277 1390267 1259868 >>> 1205214 1339111 1346927 1267888 863234 1190221 >>> "131072" 1063429 1165115 1102650 1554828 1182128 1185731 >>> 1190752 1195792 1277441 1211063 1237567 1226999 1336961 >>> "262144" 1280619 1368554 1497837 1633397 1598255 1609212 >>> 1607504 1665019 1590515 1548307 1591258 1505267 1625679 >>> "524288" 1519583 1767928 1738523 1883151 2011216 1993877 >>> 2023543 1867440 2106124 2055064 1906668 1778645 1838988 >>> "1048576" 1580851 1887530 2044131 2166133 2236379 2283578 >>> 2257454 2296612 2271066 2101274 1905829 1605923 2158238 >>> >>> >>> >>> "Reader report" >>> "4" "8" "16" "32" "64" "128" "256" "512" "1024" >>> "2048" "4096" "8192" "16384" >>> "64" 1933893 2801873 3057153 3363612 3958892 >>> "128" 2286447 3053774 2727923 3468030 4104338 4557257 >>> "256" 2903529 3236056 3245838 3705040 3654598 4496299 5117791 >>> "512" 2906696 3437042 3628697 3431550 4871723 4296637 6246213 6395018 >>> "1024" 3229770 3483896 4609294 3791442 4614246 5536137 4550690 >>> 5048117 4966395 >>> "2048" 3554251 4310472 3885431 4096676 6401772 4842658 5080379 >>> 5184636 5596757 5735012 >>> "4096" 3416292 4691638 4321103 5728903 5475122 5171846 4819300 >>> 5258919 6408472 5044289 4079948 >>> "8192" 3233004 4615263 4536055 5618186 5414558 5025700 5553712 >>> 4926264 5634770 5281396 4659702 3652258 >>> "16384" 3141058 3704193 4567654 4395850 4568869 5387732 >>> 4436432 5808029 5578420 4675810 3913007 3911225 3961277 >>> "32768" 3704273 4598957 4088278 5133719 5896692 5537024 >>> 5234412 5398271 4942992 4118662 3729099 3511757 3481511 >>> "65536" 4131091 4210184 5341188 4647619 6077765 5852474 >>> 5379762 5259330 5488249 5246682 4342682 3549202 3286487 >>> "131072" 3582876 5251082 5332216 5269908 5303512 5574440 >>> 5635064 5796372 5406363 4958839 4435918 3673443 3647874 >>> "262144" 3659283 4551414 5746231 5433824 5876196 6011650 >>> 5552000 5629260 5298830 4982226 4628902 4065823 3421924 >>> "524288" 3905973 5488778 5219963 6047356 5916811 6180455 >>> 5495733 5925628 5637512 5537123 3517132 3550861 3047013 >>> "1048576" 3855595 5634571 5410298 6001809 6464713 6299610 >>> 5894249 5516031 5800161 5209203 4295840 3724983 3641623 >> >> > -- > To unsubscribe from this list: send the line "unsubscribe ceph-devel" in > the body of a message to majordomo@vger.kernel.org > More majordomo info at http://vger.kernel.org/majordomo-info.html > -- To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html ^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: IOZone Performance is very Strange 2011-05-17 8:19 ` Simon Tian @ 2011-05-17 9:07 ` AnnyRen 0 siblings, 0 replies; 13+ messages in thread From: AnnyRen @ 2011-05-17 9:07 UTC (permalink / raw) To: Simon Tian; +Cc: cpwu, ceph-devel@vger.kernel.org [-- Attachment #1: Type: text/plain, Size: 6751 bytes --] Hi, We choose EXT4 for stability instead of btrfs. If we try btrfs later, I'll provide the throughput for you. :) In fact, I still think the iozone result is unusual, is anyone has idea with the read/write performance? (write: 1~2 GB/s) Thanks a lot~ 2011/5/17 Simon Tian <aixt2006@gmail.com>: > Hi, > The throughput of EXT4 is nice.... How about btrfs? > > Thx! > Simon > > 2011/5/17 AnnyRen <annyren6@gmail.com>: >> Hi, Jeff: >> >> I run "ceph osd tell osd_num bench" with 1 times per osd >> >> and use ceph -w to observe every osd performance, >> >> osd0: [INF] bench: wrote 1024 MB in blocks of 4096 KB in 10.875844 >> sec at 96413 KB/sec >> osd1: [INF] bench: wrote 1024 MB in blocks of 4096 KB in 11.784985 >> sec at 88975 KB/sec >> osd2: [INF] bench: wrote 1024 MB in blocks of 4096 KB in 11.161067 >> sec at 93949 KB/sec >> osd3: [INF] bench: wrote 1024 MB in blocks of 4096 KB in 10.798796 >> sec at 97101 KB/sec >> osd4: [INF] bench: wrote 1024 MB in blocks of 4096 KB in 14.437141 >> sec at 72630 KB/sec >> osd5: [INF] bench: wrote 1024 MB in blocks of 4096 KB in 14.451444 >> sec at 72558 KB/sec >> osd6: [INF] bench: wrote 1024 MB in blocks of 4096 KB in 11.083872 >> sec at 94603 KB/sec >> osd7: [INF] bench: wrote 1024 MB in blocks of 4096 KB in 11.062728 >> sec at 94784 KB/sec >> osd8: [INF] bench: wrote 1024 MB in blocks of 4096 KB in 14.137312 >> sec at 74170 KB/sec >> osd9: [INF] bench: wrote 1024 MB in blocks of 4096 KB in 13.489992 >> sec at 77729 KB/sec >> >> >> and I run >> root@MDS2:/mnt/ceph# rados bench 60 write -p data >> >> the result is >> >> Total time run: 60.553247 >> Total writes made: 1689 >> Write size: 4194304 >> Bandwidth (MB/sec): 111.571 >> >> Average Latency: 0.573209 >> Max latency: 2.25691 >> Min latency: 0.218494 >> >> >> >> 2011/5/17 Jeff Wu <cpwu@tnsoft.com.cn>: >>> Hi AnnyRen >>> >>> Could you run the following commands and give us the test results? >>> >>> $ceph osd tell OSD-N bench // OSD-N : osd number : 0,1,2 .... >>> $ceph -w >>> >>> $rados bench 60 write -p data // refer to "rados -h " >>> >>> Jeff >>> >>> >>> >>> On Tue, 2011-05-17 at 11:53 +0800, AnnyRen wrote: >>>> I'm running iozone on EXT4 with Ceph v0.26. >>>> But I got the weird result, most write performance exceed 1GB/s, even >>>> up to 3GB/s >>>> I think it's not normal to get the performance outpupt. >>>> >>>> Command line I used is: /usr/bin/iozone -azcR -f >>>> /mnt/ceph/test0516.dbf -g 1G -b /exceloutput0516.xls >>>> Attachment is the output file... >>>> >>>> and my environment is composed of 15 physical machines with >>>> >>>> 3 MON, 2MDS (1 active, 1 standby), 10 OSD (1 osd daemon (3T) /host) >>>> EXT4 format >>>> data replication size: 3 >>>> >>>> >>>> >>>> "Writer report" >>>> "4" "8" "16" "32" "64" "128" "256" "512" "1024" >>>> "2048" "4096" "8192" "16384" >>>> "64" 983980 1204796 1210227 1143223 1357066 >>>> "128" 1007629 1269781 1406136 1391557 1436229 1521718 >>>> "256" 1112909 1430119 1523457 1652404 1514860 1639786 1729594 >>>> "512" 1150351 1475116 1605228 1723770 1797349 1712772 1783912 1854787 >>>> "1024" 1213334 1471481 1679160 1828574 1888889 1899750 1885572 >>>> 1865912 1875690 >>>> "2048" 1229274 1540849 1708146 1843410 1903457 1980705 1930406 >>>> 1913634 1906837 1815744 >>>> "4096" 1213284 1528348 1674646 1762434 1872096 1882352 1881528 >>>> 1903416 1897949 1835102 1731177 >>>> "8192" 204560 155186 572387 238548 186597 429036 187327 >>>> 157205 553771 416512 299810 405842 >>>> "16384" 699749 559255 687450 541030 828776 555296 742561 >>>> 525483 604910 452423 564557 670539 970616 >>>> "32768" 532414 829610 812215 879441 863159 864794 865938 >>>> 804951 916352 879582 608132 860732 1239475 >>>> "65536" 994824 1096543 1095791 1317968 1280277 1390267 1259868 >>>> 1205214 1339111 1346927 1267888 863234 1190221 >>>> "131072" 1063429 1165115 1102650 1554828 1182128 1185731 >>>> 1190752 1195792 1277441 1211063 1237567 1226999 1336961 >>>> "262144" 1280619 1368554 1497837 1633397 1598255 1609212 >>>> 1607504 1665019 1590515 1548307 1591258 1505267 1625679 >>>> "524288" 1519583 1767928 1738523 1883151 2011216 1993877 >>>> 2023543 1867440 2106124 2055064 1906668 1778645 1838988 >>>> "1048576" 1580851 1887530 2044131 2166133 2236379 2283578 >>>> 2257454 2296612 2271066 2101274 1905829 1605923 2158238 >>>> >>>> >>>> >>>> "Reader report" >>>> "4" "8" "16" "32" "64" "128" "256" "512" "1024" >>>> "2048" "4096" "8192" "16384" >>>> "64" 1933893 2801873 3057153 3363612 3958892 >>>> "128" 2286447 3053774 2727923 3468030 4104338 4557257 >>>> "256" 2903529 3236056 3245838 3705040 3654598 4496299 5117791 >>>> "512" 2906696 3437042 3628697 3431550 4871723 4296637 6246213 6395018 >>>> "1024" 3229770 3483896 4609294 3791442 4614246 5536137 4550690 >>>> 5048117 4966395 >>>> "2048" 3554251 4310472 3885431 4096676 6401772 4842658 5080379 >>>> 5184636 5596757 5735012 >>>> "4096" 3416292 4691638 4321103 5728903 5475122 5171846 4819300 >>>> 5258919 6408472 5044289 4079948 >>>> "8192" 3233004 4615263 4536055 5618186 5414558 5025700 5553712 >>>> 4926264 5634770 5281396 4659702 3652258 >>>> "16384" 3141058 3704193 4567654 4395850 4568869 5387732 >>>> 4436432 5808029 5578420 4675810 3913007 3911225 3961277 >>>> "32768" 3704273 4598957 4088278 5133719 5896692 5537024 >>>> 5234412 5398271 4942992 4118662 3729099 3511757 3481511 >>>> "65536" 4131091 4210184 5341188 4647619 6077765 5852474 >>>> 5379762 5259330 5488249 5246682 4342682 3549202 3286487 >>>> "131072" 3582876 5251082 5332216 5269908 5303512 5574440 >>>> 5635064 5796372 5406363 4958839 4435918 3673443 3647874 >>>> "262144" 3659283 4551414 5746231 5433824 5876196 6011650 >>>> 5552000 5629260 5298830 4982226 4628902 4065823 3421924 >>>> "524288" 3905973 5488778 5219963 6047356 5916811 6180455 >>>> 5495733 5925628 5637512 5537123 3517132 3550861 3047013 >>>> "1048576" 3855595 5634571 5410298 6001809 6464713 6299610 >>>> 5894249 5516031 5800161 5209203 4295840 3724983 3641623 >>> >>> >> -- >> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in >> the body of a message to majordomo@vger.kernel.org >> More majordomo info at http://vger.kernel.org/majordomo-info.html >> > [-- Attachment #2: outputlog0516.log --] [-- Type: application/octet-stream, Size: 45342 bytes --] Iozone: Performance Test of File I/O Version $Revision: 3.373 $ Compiled for 64 bit mode. Build: linux-AMD64 Contributors:William Norcott, Don Capps, Isom Crawford, Kirby Collins Al Slater, Scott Rhine, Mike Wisner, Ken Goss Steve Landherr, Brad Smith, Mark Kelly, Dr. Alain CYR, Randy Dunlap, Mark Montague, Dan Million, Gavin Brebner, Jean-Marc Zucconi, Jeff Blomberg, Benny Halevy, Dave Boone, Erik Habbinga, Kris Strecker, Walter Wong, Joshua Root, Fabrice Bacchella, Zhenghua Xue, Qin Li, Darren Sawyer. Run began: Mon May 16 12:02:37 2011 Auto Mode Cross over of record size disabled. Include close in write timing Excel chart generation enabled Using maximum file size of 1048576 kilobytes. Command line used: /usr/bin/iozone -azcR -f /mnt/ceph/test0516.dbf -g 1G -b /exceloutput0516.xls Output is in Kbytes/sec Time Resolution = 0.000001 seconds. Processor cache size set to 1024 Kbytes. Processor cache line size set to 32 bytes. File stride size set to 17 * record size. random random bkwd record stride KB reclen write rewrite read reread read write read rewrite read fwrite frewrite fread freread 64 4 983980 969761 1933893 2772930 2006158 1310683 1996 1609270 1828508 1232453 969761 1638743 19459 64 8 1204796 1143223 2801873 3541098 4274062 2772930 1879725 3203069 3057153 1828508 1279447 2006158 3588436 64 16 1210227 1599680 3057153 3363612 2689580 1828508 1750 2222043 2662899 1357066 1232453 2467108 2662899 64 32 1143223 1279447 3363612 5735102 4018152 4988978 2067979 3057153 2801873 1690338 1526887 2561267 4274062 64 64 1357066 1599680 3958892 4018152 3057153 2203800 1865 2222043 2662899 2067979 1722886 2662899 3203069 128 4 1007629 1162547 2286447 4012317 3380677 2132084 1858934 2558895 2166499 1705404 1145188 1967960 2464907 128 8 1269781 1391557 3053774 3560017 2905056 2098744 3522 2409592 3199360 1939522 1377278 2727923 3124872 128 16 1406136 1487977 2727923 4934216 7176872 4759253 2248149 4557257 3277486 2843510 1406136 2248149 3657016 128 32 1391557 1557024 3468030 4012317 3536566 2464907 3463 2714132 3124872 2035099 1561553 2905056 3536566 128 64 1436229 1802755 4104338 6406138 8548124 5545860 2727923 5603747 3359523 1802755 1584598 2660335 4407601 128 128 1521718 2035099 4557257 4759253 3867787 2727923 3762 2784517 3468030 2673584 2202044 3560017 3895854 256 4 1112909 1506359 2903529 4009406 3410808 2187712 2097948 1952947 2392442 2783115 1334162 2533571 3654598 256 8 1430119 1817419 3236056 3823789 3326279 2242541 6561 3009318 2911402 2192179 1649865 3052087 3557725 256 16 1523457 1842366 3245838 3879045 3454704 2665657 6402 2872459 3326279 2305128 1855098 2943325 4332998 256 32 1652404 2065659 3705040 4350555 3994490 2719671 6638 3410808 3770085 2419396 2205688 3935921 4054829 256 64 1514860 1952947 3654598 9114515 8534922 5810112 3087188 6761355 3994490 3511189 1763685 2880164 4404088 256 128 1639786 2210228 4496299 5117791 4477549 3078337 7874 3378611 3994490 1778290 2187712 3705040 4496299 256 256 1729594 2205688 5117791 5687020 5347168 3410808 3511189 4197489 3465855 3935921 2371308 3454704 7735574 512 4 1150351 1766304 2906696 3325278 2962840 2099600 13813 2024389 2438090 1723770 1446306 2668325 2811557 512 8 1475116 1827947 3437042 5214798 4701087 2991733 2991733 4099771 3239989 3710197 1707325 2753870 4262523 512 16 1605228 2370799 3628697 3815675 3678421 2860238 14750 3259661 3580298 2497638 2073249 3064306 3659616 512 32 1723770 2097550 3431550 5384786 5177083 3304808 3140488 4340054 3934520 4411377 1985091 3182373 6174377 512 64 1797349 2205248 4871723 4784885 4494470 3239989 12748 3604335 4494470 2782414 2796910 4571003 4650188 512 128 1712772 2327124 4296637 5699180 5509113 3634839 3580298 7211022 4571003 3304808 2178404 4000485 6319739 512 256 1783912 2297251 6246213 5384786 4927617 3372274 12852 3653390 4701087 1933266 2521095 4305250 4927617 512 512 1854787 2753870 6395018 6174377 6472112 4195896 3884698 4262523 3736016 4195896 2910635 3736016 6491676 1024 4 1213334 1524224 3229770 3012308 2516377 1786736 28483 2039583 2491561 1815435 1695735 2774874 2877110 1024 8 1471481 1924436 3483896 4374559 4146499 2701567 3029305 3469823 3379719 2316837 2090206 3849207 6830356 1024 16 1679160 2414523 4609294 4634161 4493556 3335105 29687 3151562 3544270 2619192 2353657 4033569 4110780 1024 32 1828574 3010197 3791442 5417427 5303701 3618930 3794792 5120336 5144870 4550690 2461573 3923040 6172653 1024 64 1888889 2353657 4614246 5630486 4805258 3469823 29659 3866533 5336651 2993413 2679655 4720752 5169641 1024 128 1899750 3261656 5536137 8593274 8326715 6093831 4300102 5682634 5983467 4550690 2586074 4995276 8542002 1024 256 1885572 2842832 4550690 5444898 5226256 3495237 29877 3999762 6317933 2089189 2586074 4674510 6402699 1024 512 1865912 2918161 5048117 5593820 4199201 2892612 4526709 4392454 4943530 2414523 3048658 4470172 6093831 1024 1024 1875690 2908281 4966395 5690162 5593820 3696803 25736 3738636 4995276 3684119 3220084 4995276 5564829 2048 4 1229274 2072957 3554251 5389574 4674016 2836079 3038742 3710866 3931669 2798201 1741742 2929912 3835124 2048 8 1540849 2265394 4310472 5489457 4960102 3219850 3848872 4223576 5159722 3937075 2206617 3777778 4481379 2048 16 1708146 2275596 3885431 5937172 5655716 3777778 3235617 4796687 4581773 3487865 2611870 4697018 5769681 2048 32 1843410 2485653 4096676 5689430 5489457 3717289 3922692 4643695 6185123 4572018 3240499 5095447 8758039 2048 64 1903457 3430751 6401772 9389900 9189005 6225466 6058611 7969835 6738230 4923146 2968386 4653758 5937172 2048 128 1980705 2809182 4842658 6807652 6649556 4730646 4633675 6058611 8190205 4674016 3246623 5507054 8949659 2048 256 1930406 2942962 5080379 6649556 6588354 4493100 4842658 5416763 5884299 3386122 2734946 4889518 6588354 2048 512 1913634 2872113 5184636 6690992 7165410 5719737 4728042 5068389 5884299 3245396 2522144 4603874 5674397 2048 1024 1906837 2848305 5596757 6711905 6649556 4312636 4663865 4775354 5596757 2029850 2233004 4751580 6588354 2048 2048 1815744 2715922 5735012 5937172 5868220 3857514 4453499 3878414 4572018 5172149 3251539 4762117 6225466 4096 4 1213284 1785514 3416292 3799338 3393349 1901520 2577699 2345757 2694963 2508081 1584010 2929815 3098371 4096 8 1528348 2264141 4691638 6493246 6300364 3909130 3849566 4257917 4320017 2860067 2289792 4230655 4900405 4096 16 1674646 2274332 4321103 4819300 4757904 3107900 3961410 4043458 4244242 2737912 2221111 4226492 4723889 4096 32 1762434 2471992 5728903 7954358 8173861 5265366 5404504 7038605 7351860 4526013 2576153 4681411 5634950 4096 64 1872096 2854365 5475122 6522830 6585338 4418911 5107271 5859825 7342434 4740836 3391339 6408472 8360824 4096 128 1882352 2925823 5171846 6370451 6459069 4581535 5786794 8096815 7447471 4423462 3139133 4653513 5431844 4096 256 1881528 2807257 4819300 6151473 6351609 4266376 4703198 5490871 6703539 3287506 2652931 4725189 6041154 4096 512 1903416 2923832 5258919 6112079 6370451 4140919 5000244 5153230 6254490 3370712 2840678 5019235 6007355 4096 1024 1897949 2771033 6408472 7756829 7834651 5190597 5026578 5056165 6254490 2491711 2114496 5657217 7434579 4096 2048 1835102 2715838 5044289 5490871 5306022 3512686 5340661 5551194 6351609 2116580 2355405 5145513 6555186 4096 4096 1731177 2610602 4079948 4171080 4125010 2886496 4141917 2921346 4176149 3381993 2987904 4196552 4240052 8192 4 204560 1723951 3233004 4216019 3996338 2351344 2993059 3245524 3723500 2464511 1733081 3123099 3910378 8192 8 155186 2656378 4615263 5143053 4926264 3155510 3899284 5756551 4943985 3106440 2238301 3636025 3775461 8192 16 572387 2281401 4536055 4844308 4809724 3164228 5091232 6253130 5664497 2623119 2390441 5512724 6145757 8192 32 238548 3054523 5618186 5944791 6345516 3893540 5385702 7008178 6229323 3593060 2579403 6045199 6439465 8192 64 186597 2793973 5414558 5897851 4903066 3047749 5357152 6814966 6233844 3661989 3229054 5361332 6234975 8192 128 429036 2908931 5025700 5375591 5472336 3596068 5720134 8104083 6720329 3703433 2713438 5700206 6527546 8192 256 187327 2790116 5553712 6262248 6580049 4250966 5728718 7320230 6307078 3517291 3268680 5708730 6085887 8192 512 157205 3371310 4926264 5211701 5323125 3488721 5457560 6895660 8577485 3558822 2677075 5400092 6040947 8192 1024 553771 2777710 5634770 6224809 6334986 4158362 5443725 6781341 8144423 2691123 2166056 4352746 4636437 8192 2048 416512 2739834 5281396 5684175 5319828 3804725 4478702 4483963 5454094 2091432 1989238 5155399 5565406 8192 4096 299810 2844630 4659702 4844991 4617744 3465148 4435342 3897073 4242568 1774247 1923535 3972772 4273171 8192 8192 405842 2333935 3652258 3876846 3880349 2975434 3725519 2477126 3760997 3027341 2790116 3508313 3588557 16384 4 699749 1721179 3141058 4280039 3751717 2203903 2688155 3690071 3582721 2270831 2003144 3880525 4145769 16384 8 559255 2281234 3704193 4454549 4740972 2944556 3788324 5730536 4366522 2911126 2721153 4580441 4787877 16384 16 687450 2252373 4567654 4672948 5170063 3284743 4920199 6463575 5033728 3084384 2917924 5496324 5628681 16384 32 541030 2414021 4395850 4437578 5137595 3597726 5317690 7440550 4667870 2969751 2613921 4650496 5919089 16384 64 828776 3238001 4568869 6236563 6289648 3758488 5777267 8027220 4779552 2524891 2867396 5657411 6145651 16384 128 555296 3118395 5387732 5614884 5643936 3935641 4794893 6930296 5628220 3376744 3199408 5589764 5811959 16384 256 742561 3322701 4436432 5298828 6021260 3949893 5378876 4863096 4969299 3098849 3010077 5647647 5915013 16384 512 525483 3063620 5808029 5944691 6131941 3990259 5639767 7025952 4638880 3278162 3176338 5432453 5655548 16384 1024 604910 3149552 5578420 5822300 5882607 3909445 5421738 7001613 8183027 2608463 2364190 5153778 5342079 16384 2048 452423 2752213 4675810 4827227 5497643 3740689 4549812 5754529 7071501 2325311 2254516 5189585 5343741 16384 4096 564557 2282826 3913007 4070623 4084657 3105992 3600931 3792506 4198966 2065520 2027972 3503092 3869600 16384 8192 670539 2312167 3911225 3915906 3916798 3011792 3725277 3262443 3402660 1661503 1999007 3814611 3893497 16384 16384 970616 2745725 3961277 3953756 4007713 3068134 3927993 3081341 3983551 3099548 2597716 3938574 4005844 32768 4 532414 1976833 3704273 3793113 3297257 2125307 3158005 3609993 3188187 2214371 1861186 3628003 3700982 32768 8 829610 2431050 4598957 4976281 4593270 2807179 4422130 5794514 3802348 2724973 2703426 4577818 4906815 32768 16 812215 2849136 4088278 4733266 4984583 3129957 4281746 6442610 5409107 2938562 2889570 4318475 5573625 32768 32 879441 2634050 5133719 4420992 5356822 3437709 4383765 7275441 4831606 3148600 3097159 5713813 5824472 32768 64 863159 2657170 5896692 5943613 5934117 3702577 5899982 8051111 4793522 3310521 3251776 5905306 5929765 32768 128 864794 3323972 5537024 5944898 5913691 3783403 5428763 8049225 5835849 3276818 3063334 5616675 5662493 32768 256 865938 2878917 5234412 5871500 5866989 3868163 5131036 7425196 4967109 3152862 2989505 4954932 5840064 32768 512 804951 3338586 5398271 5700541 5752796 3844682 4754553 6960060 4858250 2993347 2874882 5345571 5412515 32768 1024 916352 2706300 4942992 5489695 5494304 3819466 5549096 6657636 4525960 2495677 2179432 5418063 5537917 32768 2048 879582 2826054 4118662 4496787 4821606 3498170 4755211 4998906 4748967 2153952 1923358 4448610 3986302 32768 4096 608132 2414225 3729099 4183858 4145245 3160474 4324726 3859581 4393013 2059078 1931494 4227095 4392591 32768 8192 860732 2395333 3511757 3808881 3843392 2994651 3458731 3206260 3573387 2060869 1977800 3584010 3542076 32768 16384 1239475 2363857 3481511 3963311 3958517 3083747 3922366 3216691 3468945 1763896 1903591 3903760 3927073 65536 4 994824 2155282 4131091 4267508 3577518 2103606 3438217 3687171 3279899 2216224 1912116 4093685 4131898 65536 8 1096543 2518285 4210184 4885606 4396582 2759297 4276072 5667725 4177996 2738542 2706961 4207799 4660226 65536 16 1095791 2756530 5341188 5880772 5398357 3074944 4816774 6435190 4859178 2931599 2906398 4972753 5754339 65536 32 1317968 2995656 4647619 5834215 5778654 3398849 5775133 7234427 4286206 3004169 2787926 5930764 5851478 65536 64 1280277 2927914 6077765 6161136 5949248 3604445 5523096 7997889 5834338 3237483 3219359 4799448 5672637 65536 128 1390267 3278295 5852474 5923989 5852474 3766234 5852973 8104714 5701819 3218039 3170273 5669127 5990477 65536 256 1259868 3256121 5379762 5640394 5598464 3788190 5052920 7407749 4414092 3138240 3028897 5170492 5655015 65536 512 1205214 3284838 5259330 5668777 5717593 3829353 4700952 7100989 5141671 3141720 3066300 5334140 5594703 65536 1024 1339111 3358033 5488249 5666673 5691430 3807552 5528539 6607196 5296624 2496715 2423843 5529985 5587539 65536 2048 1346927 3060258 5246682 5286031 5230309 3604870 4815592 5390734 4891518 2160415 2092270 4860209 5176529 65536 4096 1267888 2841825 4342682 4525051 4459132 3279077 4225133 3792371 4563439 2130278 2065356 4014468 4254232 65536 8192 863234 2672848 3549202 3641934 3629000 2881960 3494293 3132090 3457070 1983463 2043203 3859137 3866193 65536 16384 1190221 2410558 3286487 3361524 3400405 2764069 3543848 3019680 3486669 1990587 1953328 3470294 3581993 131072 4 1063429 2122209 3582876 3725319 3141770 2079251 3427259 3733820 3508539 2219757 2205119 3585610 3648794 131072 8 1165115 2590770 5251082 5532311 4792370 2737339 4392220 5686415 4147322 2734534 2726979 4687997 4767434 131072 16 1102650 2938506 5332216 5369348 4971620 3075577 5205636 6565047 4872952 2933833 2927474 4953031 5555063 131072 32 1554828 3095179 5269908 5975749 5826473 3460025 5111211 7370980 5505607 3109956 3020506 5863511 5928960 131072 64 1182128 3209244 5303512 5886426 5685650 3632328 5604960 8189444 5566200 3258066 3242578 5789657 5839222 131072 128 1185731 3292506 5574440 6134311 5992946 3724688 5294115 8202151 5500043 3252169 3245008 5492680 5822092 131072 256 1190752 3282206 5635064 5671397 5713897 3789307 5474793 7460000 5366937 3161027 2965020 5453179 5563327 131072 512 1195792 3103478 5796372 5816672 5775910 3844209 5744226 7094190 5310839 3175965 3115208 5608047 5813535 131072 1024 1277441 3233309 5406363 5636566 5638011 3818655 5420222 6876090 5451990 2514378 2504139 5381964 5524139 131072 2048 1211063 3086819 4958839 5012279 4961300 3540853 4918247 5155887 4556473 2214580 2138871 4919567 4951336 131072 4096 1237567 2989317 4435918 4518164 4430948 3266022 4095356 3876028 4223763 2102736 2088429 4382940 4412172 131072 8192 1226999 2657804 3673443 3724789 3824233 3000935 3597341 3327299 3742258 1996101 2051757 4058082 4099724 131072 16384 1336961 2773717 3647874 3659895 3646882 2924547 3625071 3131730 3551031 2018668 2072909 3660723 3666779 262144 4 1280619 2154638 3659283 3787438 3389413 2031115 3379277 3677618 3114492 2189661 2179085 3790624 3841225 262144 8 1368554 2698845 4551414 5529885 4674718 2650889 3966351 5575763 4010886 2743156 2677426 4951443 5017878 262144 16 1497837 2913263 5746231 5926685 5440465 3039770 4665158 6368154 4585256 2906348 2892075 5802208 5845704 262144 32 1633397 3067441 5433824 5513497 5162225 3394813 5388922 7357801 5409949 3136812 3120910 5909611 6135194 262144 64 1598255 3192247 5876196 5935451 5682970 3612242 6136050 7885432 5954319 3270177 3214957 5854295 6242408 262144 128 1609212 3368944 6011650 6169690 5951870 3742613 5662221 7936316 5499708 3225946 3171575 5310321 5567829 262144 256 1607504 3452895 5552000 5637746 5559440 3821054 5383724 7232972 5307527 3176000 3184647 5417786 5459754 262144 512 1665019 3345551 5629260 5681472 5892351 3839535 5568647 6938347 5853017 3155428 3136561 5645620 5711072 262144 1024 1590515 3421392 5298830 5589766 5564166 3779847 5001990 7043464 5180100 2468400 2498397 5398474 5579441 262144 2048 1548307 3208933 4982226 5026067 4975215 3514885 4973324 6185830 5160407 2226729 2172927 4839446 5212319 262144 4096 1591258 2883678 4628902 4822550 4736970 3432639 4322033 3858980 4171677 2046304 2117170 4631027 4787045 262144 8192 1505267 2604713 4065823 4086996 4058589 3128244 3660099 3184831 2566267 1673587 2016927 4104465 4146474 262144 16384 1625679 2807556 3421924 3602159 3750106 3081471 3910086 3261795 3809286 2083948 1981766 3389758 3600425 524288 4 1519583 2170921 3905973 3935472 3072605 1982306 3319101 3649426 3348842 2204333 2216599 3801120 3847871 524288 8 1767928 2649566 5488778 5648314 4649384 2627956 4915272 5384934 4528001 2747608 2732901 5448726 5525270 524288 16 1738523 2993261 5219963 5319259 4815810 3010241 5518115 6244425 5083027 2933752 2884790 5220272 5259338 524288 32 1883151 3202894 6047356 6251508 5751925 3382458 6005802 7324710 5809870 3139314 3069658 5740019 5868713 524288 64 2011216 3325706 5916811 6050717 5724538 3616530 5902613 8245467 6050301 3287483 3265204 6323742 6491048 524288 128 1993877 3409223 6180455 6307146 6074468 3768329 5830156 8346142 5436307 3268567 3162766 5922356 6110149 524288 256 2023543 3473554 5495733 5510814 5451103 3837229 5726924 7445915 5780775 3222164 3218009 5491533 5730043 524288 512 1867440 3498683 5925628 6008411 5972883 3866259 5948357 6682482 5820234 3196199 3148046 5751489 6003801 524288 1024 2106124 3411402 5637512 5837415 5835153 3848295 5184812 7057226 5791768 2549344 2484825 5559788 5582824 524288 2048 2055064 3292893 5537123 5576298 5522023 3735872 5510054 6329075 5471489 2297191 2273059 5526089 5558383 524288 4096 1906668 2453451 3517132 3374490 3091191 2429103 3543275 3078972 3298652 1537975 1524114 3452666 3326365 524288 8192 1778645 2479136 3550861 3597691 3703619 2825756 3720311 2850958 3536391 1739883 1729039 3713639 3791937 524288 16384 1838988 2496003 3047013 3192612 3302581 2680601 3033040 2615945 3241949 1699933 1683459 3361147 3375858 1048576 4 1580851 2122536 3855595 4287055 3251396 1894535 3948544 3489010 3204978 2120718 2127533 4155346 4039790 1048576 8 1887530 2606050 5634571 5587369 4319819 2576234 5005995 5264544 4801634 2689787 2708155 5442286 5583893 1048576 16 2044131 2986791 5410298 5664915 5430754 2999412 5547493 6453453 4985572 2917500 2909656 5220067 4931690 1048576 32 2166133 3136933 6001809 6308212 5818793 3335027 6116572 6982823 5352851 3111538 3089954 6351118 6375525 1048576 64 2236379 3300325 6464713 6522786 6205367 3516420 6296652 8104565 5809577 3242070 3241587 6326897 6390328 1048576 128 2283578 3342971 6299610 6376218 6170014 3774107 6287326 8333550 5832451 3260285 3282739 6371922 6412410 1048576 256 2257454 3332929 5894249 5769936 5534899 3769223 5906940 6947470 5552809 3172772 3157808 5641936 5979971 1048576 512 2296612 3273964 5516031 5480658 5451967 3816526 5973124 7071305 5681373 3183696 3180790 5993989 5952405 1048576 1024 2271066 3433190 5800161 5931260 5926776 3844937 5902112 6670449 5914661 2298445 2277048 5695469 5967986 1048576 2048 2101274 3235126 5209203 5240812 4815192 3204557 5393465 6053431 5615255 1961164 1943185 5327395 5580853 1048576 4096 1905829 2833751 4295840 4320409 4266995 3145297 4249238 3475502 4041891 1629435 1732427 4292251 4334853 1048576 8192 1605923 2475877 3724983 3732703 3508456 2635944 3465509 2669428 3502831 1705151 1588722 3261796 3286443 1048576 16384 2158238 2523236 3641623 3654465 3633608 2897521 3563566 3349418 3545228 2029194 2077713 3681899 3987039 iozone test complete. Excel output is below: "Writer report" "4" "8" "16" "32" "64" "128" "256" "512" "1024" "2048" "4096" "8192" "16384" "64" 983980 1204796 1210227 1143223 1357066 "128" 1007629 1269781 1406136 1391557 1436229 1521718 "256" 1112909 1430119 1523457 1652404 1514860 1639786 1729594 "512" 1150351 1475116 1605228 1723770 1797349 1712772 1783912 1854787 "1024" 1213334 1471481 1679160 1828574 1888889 1899750 1885572 1865912 1875690 "2048" 1229274 1540849 1708146 1843410 1903457 1980705 1930406 1913634 1906837 1815744 "4096" 1213284 1528348 1674646 1762434 1872096 1882352 1881528 1903416 1897949 1835102 1731177 "8192" 204560 155186 572387 238548 186597 429036 187327 157205 553771 416512 299810 405842 "16384" 699749 559255 687450 541030 828776 555296 742561 525483 604910 452423 564557 670539 970616 "32768" 532414 829610 812215 879441 863159 864794 865938 804951 916352 879582 608132 860732 1239475 "65536" 994824 1096543 1095791 1317968 1280277 1390267 1259868 1205214 1339111 1346927 1267888 863234 1190221 "131072" 1063429 1165115 1102650 1554828 1182128 1185731 1190752 1195792 1277441 1211063 1237567 1226999 1336961 "262144" 1280619 1368554 1497837 1633397 1598255 1609212 1607504 1665019 1590515 1548307 1591258 1505267 1625679 "524288" 1519583 1767928 1738523 1883151 2011216 1993877 2023543 1867440 2106124 2055064 1906668 1778645 1838988 "1048576" 1580851 1887530 2044131 2166133 2236379 2283578 2257454 2296612 2271066 2101274 1905829 1605923 2158238 "Re-writer report" "4" "8" "16" "32" "64" "128" "256" "512" "1024" "2048" "4096" "8192" "16384" "64" 969761 1143223 1599680 1279447 1599680 "128" 1162547 1391557 1487977 1557024 1802755 2035099 "256" 1506359 1817419 1842366 2065659 1952947 2210228 2205688 "512" 1766304 1827947 2370799 2097550 2205248 2327124 2297251 2753870 "1024" 1524224 1924436 2414523 3010197 2353657 3261656 2842832 2918161 2908281 "2048" 2072957 2265394 2275596 2485653 3430751 2809182 2942962 2872113 2848305 2715922 "4096" 1785514 2264141 2274332 2471992 2854365 2925823 2807257 2923832 2771033 2715838 2610602 "8192" 1723951 2656378 2281401 3054523 2793973 2908931 2790116 3371310 2777710 2739834 2844630 2333935 "16384" 1721179 2281234 2252373 2414021 3238001 3118395 3322701 3063620 3149552 2752213 2282826 2312167 2745725 "32768" 1976833 2431050 2849136 2634050 2657170 3323972 2878917 3338586 2706300 2826054 2414225 2395333 2363857 "65536" 2155282 2518285 2756530 2995656 2927914 3278295 3256121 3284838 3358033 3060258 2841825 2672848 2410558 "131072" 2122209 2590770 2938506 3095179 3209244 3292506 3282206 3103478 3233309 3086819 2989317 2657804 2773717 "262144" 2154638 2698845 2913263 3067441 3192247 3368944 3452895 3345551 3421392 3208933 2883678 2604713 2807556 "524288" 2170921 2649566 2993261 3202894 3325706 3409223 3473554 3498683 3411402 3292893 2453451 2479136 2496003 "1048576" 2122536 2606050 2986791 3136933 3300325 3342971 3332929 3273964 3433190 3235126 2833751 2475877 2523236 "Reader report" "4" "8" "16" "32" "64" "128" "256" "512" "1024" "2048" "4096" "8192" "16384" "64" 1933893 2801873 3057153 3363612 3958892 "128" 2286447 3053774 2727923 3468030 4104338 4557257 "256" 2903529 3236056 3245838 3705040 3654598 4496299 5117791 "512" 2906696 3437042 3628697 3431550 4871723 4296637 6246213 6395018 "1024" 3229770 3483896 4609294 3791442 4614246 5536137 4550690 5048117 4966395 "2048" 3554251 4310472 3885431 4096676 6401772 4842658 5080379 5184636 5596757 5735012 "4096" 3416292 4691638 4321103 5728903 5475122 5171846 4819300 5258919 6408472 5044289 4079948 "8192" 3233004 4615263 4536055 5618186 5414558 5025700 5553712 4926264 5634770 5281396 4659702 3652258 "16384" 3141058 3704193 4567654 4395850 4568869 5387732 4436432 5808029 5578420 4675810 3913007 3911225 3961277 "32768" 3704273 4598957 4088278 5133719 5896692 5537024 5234412 5398271 4942992 4118662 3729099 3511757 3481511 "65536" 4131091 4210184 5341188 4647619 6077765 5852474 5379762 5259330 5488249 5246682 4342682 3549202 3286487 "131072" 3582876 5251082 5332216 5269908 5303512 5574440 5635064 5796372 5406363 4958839 4435918 3673443 3647874 "262144" 3659283 4551414 5746231 5433824 5876196 6011650 5552000 5629260 5298830 4982226 4628902 4065823 3421924 "524288" 3905973 5488778 5219963 6047356 5916811 6180455 5495733 5925628 5637512 5537123 3517132 3550861 3047013 "1048576" 3855595 5634571 5410298 6001809 6464713 6299610 5894249 5516031 5800161 5209203 4295840 3724983 3641623 "Re-Reader report" "4" "8" "16" "32" "64" "128" "256" "512" "1024" "2048" "4096" "8192" "16384" "64" 2772930 3541098 3363612 5735102 4018152 "128" 4012317 3560017 4934216 4012317 6406138 4759253 "256" 4009406 3823789 3879045 4350555 9114515 5117791 5687020 "512" 3325278 5214798 3815675 5384786 4784885 5699180 5384786 6174377 "1024" 3012308 4374559 4634161 5417427 5630486 8593274 5444898 5593820 5690162 "2048" 5389574 5489457 5937172 5689430 9389900 6807652 6649556 6690992 6711905 5937172 "4096" 3799338 6493246 4819300 7954358 6522830 6370451 6151473 6112079 7756829 5490871 4171080 "8192" 4216019 5143053 4844308 5944791 5897851 5375591 6262248 5211701 6224809 5684175 4844991 3876846 "16384" 4280039 4454549 4672948 4437578 6236563 5614884 5298828 5944691 5822300 4827227 4070623 3915906 3953756 "32768" 3793113 4976281 4733266 4420992 5943613 5944898 5871500 5700541 5489695 4496787 4183858 3808881 3963311 "65536" 4267508 4885606 5880772 5834215 6161136 5923989 5640394 5668777 5666673 5286031 4525051 3641934 3361524 "131072" 3725319 5532311 5369348 5975749 5886426 6134311 5671397 5816672 5636566 5012279 4518164 3724789 3659895 "262144" 3787438 5529885 5926685 5513497 5935451 6169690 5637746 5681472 5589766 5026067 4822550 4086996 3602159 "524288" 3935472 5648314 5319259 6251508 6050717 6307146 5510814 6008411 5837415 5576298 3374490 3597691 3192612 "1048576" 4287055 5587369 5664915 6308212 6522786 6376218 5769936 5480658 5931260 5240812 4320409 3732703 3654465 "Random read report" "4" "8" "16" "32" "64" "128" "256" "512" "1024" "2048" "4096" "8192" "16384" "64" 2006158 4274062 2689580 4018152 3057153 "128" 3380677 2905056 7176872 3536566 8548124 3867787 "256" 3410808 3326279 3454704 3994490 8534922 4477549 5347168 "512" 2962840 4701087 3678421 5177083 4494470 5509113 4927617 6472112 "1024" 2516377 4146499 4493556 5303701 4805258 8326715 5226256 4199201 5593820 "2048" 4674016 4960102 5655716 5489457 9189005 6649556 6588354 7165410 6649556 5868220 "4096" 3393349 6300364 4757904 8173861 6585338 6459069 6351609 6370451 7834651 5306022 4125010 "8192" 3996338 4926264 4809724 6345516 4903066 5472336 6580049 5323125 6334986 5319828 4617744 3880349 "16384" 3751717 4740972 5170063 5137595 6289648 5643936 6021260 6131941 5882607 5497643 4084657 3916798 4007713 "32768" 3297257 4593270 4984583 5356822 5934117 5913691 5866989 5752796 5494304 4821606 4145245 3843392 3958517 "65536" 3577518 4396582 5398357 5778654 5949248 5852474 5598464 5717593 5691430 5230309 4459132 3629000 3400405 "131072" 3141770 4792370 4971620 5826473 5685650 5992946 5713897 5775910 5638011 4961300 4430948 3824233 3646882 "262144" 3389413 4674718 5440465 5162225 5682970 5951870 5559440 5892351 5564166 4975215 4736970 4058589 3750106 "524288" 3072605 4649384 4815810 5751925 5724538 6074468 5451103 5972883 5835153 5522023 3091191 3703619 3302581 "1048576" 3251396 4319819 5430754 5818793 6205367 6170014 5534899 5451967 5926776 4815192 4266995 3508456 3633608 "Random write report" "4" "8" "16" "32" "64" "128" "256" "512" "1024" "2048" "4096" "8192" "16384" "64" 1310683 2772930 1828508 4988978 2203800 "128" 2132084 2098744 4759253 2464907 5545860 2727923 "256" 2187712 2242541 2665657 2719671 5810112 3078337 3410808 "512" 2099600 2991733 2860238 3304808 3239989 3634839 3372274 4195896 "1024" 1786736 2701567 3335105 3618930 3469823 6093831 3495237 2892612 3696803 "2048" 2836079 3219850 3777778 3717289 6225466 4730646 4493100 5719737 4312636 3857514 "4096" 1901520 3909130 3107900 5265366 4418911 4581535 4266376 4140919 5190597 3512686 2886496 "8192" 2351344 3155510 3164228 3893540 3047749 3596068 4250966 3488721 4158362 3804725 3465148 2975434 "16384" 2203903 2944556 3284743 3597726 3758488 3935641 3949893 3990259 3909445 3740689 3105992 3011792 3068134 "32768" 2125307 2807179 3129957 3437709 3702577 3783403 3868163 3844682 3819466 3498170 3160474 2994651 3083747 "65536" 2103606 2759297 3074944 3398849 3604445 3766234 3788190 3829353 3807552 3604870 3279077 2881960 2764069 "131072" 2079251 2737339 3075577 3460025 3632328 3724688 3789307 3844209 3818655 3540853 3266022 3000935 2924547 "262144" 2031115 2650889 3039770 3394813 3612242 3742613 3821054 3839535 3779847 3514885 3432639 3128244 3081471 "524288" 1982306 2627956 3010241 3382458 3616530 3768329 3837229 3866259 3848295 3735872 2429103 2825756 2680601 "1048576" 1894535 2576234 2999412 3335027 3516420 3774107 3769223 3816526 3844937 3204557 3145297 2635944 2897521 "Backward read report" "4" "8" "16" "32" "64" "128" "256" "512" "1024" "2048" "4096" "8192" "16384" "64" 1996 1879725 1750 2067979 1865 "128" 1858934 3522 2248149 3463 2727923 3762 "256" 2097948 6561 6402 6638 3087188 7874 3511189 "512" 13813 2991733 14750 3140488 12748 3580298 12852 3884698 "1024" 28483 3029305 29687 3794792 29659 4300102 29877 4526709 25736 "2048" 3038742 3848872 3235617 3922692 6058611 4633675 4842658 4728042 4663865 4453499 "4096" 2577699 3849566 3961410 5404504 5107271 5786794 4703198 5000244 5026578 5340661 4141917 "8192" 2993059 3899284 5091232 5385702 5357152 5720134 5728718 5457560 5443725 4478702 4435342 3725519 "16384" 2688155 3788324 4920199 5317690 5777267 4794893 5378876 5639767 5421738 4549812 3600931 3725277 3927993 "32768" 3158005 4422130 4281746 4383765 5899982 5428763 5131036 4754553 5549096 4755211 4324726 3458731 3922366 "65536" 3438217 4276072 4816774 5775133 5523096 5852973 5052920 4700952 5528539 4815592 4225133 3494293 3543848 "131072" 3427259 4392220 5205636 5111211 5604960 5294115 5474793 5744226 5420222 4918247 4095356 3597341 3625071 "262144" 3379277 3966351 4665158 5388922 6136050 5662221 5383724 5568647 5001990 4973324 4322033 3660099 3910086 "524288" 3319101 4915272 5518115 6005802 5902613 5830156 5726924 5948357 5184812 5510054 3543275 3720311 3033040 "1048576" 3948544 5005995 5547493 6116572 6296652 6287326 5906940 5973124 5902112 5393465 4249238 3465509 3563566 "Record rewrite report" "4" "8" "16" "32" "64" "128" "256" "512" "1024" "2048" "4096" "8192" "16384" "64" 1609270 3203069 2222043 3057153 2222043 "128" 2558895 2409592 4557257 2714132 5603747 2784517 "256" 1952947 3009318 2872459 3410808 6761355 3378611 4197489 "512" 2024389 4099771 3259661 4340054 3604335 7211022 3653390 4262523 "1024" 2039583 3469823 3151562 5120336 3866533 5682634 3999762 4392454 3738636 "2048" 3710866 4223576 4796687 4643695 7969835 6058611 5416763 5068389 4775354 3878414 "4096" 2345757 4257917 4043458 7038605 5859825 8096815 5490871 5153230 5056165 5551194 2921346 "8192" 3245524 5756551 6253130 7008178 6814966 8104083 7320230 6895660 6781341 4483963 3897073 2477126 "16384" 3690071 5730536 6463575 7440550 8027220 6930296 4863096 7025952 7001613 5754529 3792506 3262443 3081341 "32768" 3609993 5794514 6442610 7275441 8051111 8049225 7425196 6960060 6657636 4998906 3859581 3206260 3216691 "65536" 3687171 5667725 6435190 7234427 7997889 8104714 7407749 7100989 6607196 5390734 3792371 3132090 3019680 "131072" 3733820 5686415 6565047 7370980 8189444 8202151 7460000 7094190 6876090 5155887 3876028 3327299 3131730 "262144" 3677618 5575763 6368154 7357801 7885432 7936316 7232972 6938347 7043464 6185830 3858980 3184831 3261795 "524288" 3649426 5384934 6244425 7324710 8245467 8346142 7445915 6682482 7057226 6329075 3078972 2850958 2615945 "1048576" 3489010 5264544 6453453 6982823 8104565 8333550 6947470 7071305 6670449 6053431 3475502 2669428 3349418 "Stride read report" "4" "8" "16" "32" "64" "128" "256" "512" "1024" "2048" "4096" "8192" "16384" "64" 1828508 3057153 2662899 2801873 2662899 "128" 2166499 3199360 3277486 3124872 3359523 3468030 "256" 2392442 2911402 3326279 3770085 3994490 3994490 3465855 "512" 2438090 3239989 3580298 3934520 4494470 4571003 4701087 3736016 "1024" 2491561 3379719 3544270 5144870 5336651 5983467 6317933 4943530 4995276 "2048" 3931669 5159722 4581773 6185123 6738230 8190205 5884299 5884299 5596757 4572018 "4096" 2694963 4320017 4244242 7351860 7342434 7447471 6703539 6254490 6254490 6351609 4176149 "8192" 3723500 4943985 5664497 6229323 6233844 6720329 6307078 8577485 8144423 5454094 4242568 3760997 "16384" 3582721 4366522 5033728 4667870 4779552 5628220 4969299 4638880 8183027 7071501 4198966 3402660 3983551 "32768" 3188187 3802348 5409107 4831606 4793522 5835849 4967109 4858250 4525960 4748967 4393013 3573387 3468945 "65536" 3279899 4177996 4859178 4286206 5834338 5701819 4414092 5141671 5296624 4891518 4563439 3457070 3486669 "131072" 3508539 4147322 4872952 5505607 5566200 5500043 5366937 5310839 5451990 4556473 4223763 3742258 3551031 "262144" 3114492 4010886 4585256 5409949 5954319 5499708 5307527 5853017 5180100 5160407 4171677 2566267 3809286 "524288" 3348842 4528001 5083027 5809870 6050301 5436307 5780775 5820234 5791768 5471489 3298652 3536391 3241949 "1048576" 3204978 4801634 4985572 5352851 5809577 5832451 5552809 5681373 5914661 5615255 4041891 3502831 3545228 "Fwrite report" "4" "8" "16" "32" "64" "128" "256" "512" "1024" "2048" "4096" "8192" "16384" "64" 1232453 1828508 1357066 1690338 2067979 "128" 1705404 1939522 2843510 2035099 1802755 2673584 "256" 2783115 2192179 2305128 2419396 3511189 1778290 3935921 "512" 1723770 3710197 2497638 4411377 2782414 3304808 1933266 4195896 "1024" 1815435 2316837 2619192 4550690 2993413 4550690 2089189 2414523 3684119 "2048" 2798201 3937075 3487865 4572018 4923146 4674016 3386122 3245396 2029850 5172149 "4096" 2508081 2860067 2737912 4526013 4740836 4423462 3287506 3370712 2491711 2116580 3381993 "8192" 2464511 3106440 2623119 3593060 3661989 3703433 3517291 3558822 2691123 2091432 1774247 3027341 "16384" 2270831 2911126 3084384 2969751 2524891 3376744 3098849 3278162 2608463 2325311 2065520 1661503 3099548 "32768" 2214371 2724973 2938562 3148600 3310521 3276818 3152862 2993347 2495677 2153952 2059078 2060869 1763896 "65536" 2216224 2738542 2931599 3004169 3237483 3218039 3138240 3141720 2496715 2160415 2130278 1983463 1990587 "131072" 2219757 2734534 2933833 3109956 3258066 3252169 3161027 3175965 2514378 2214580 2102736 1996101 2018668 "262144" 2189661 2743156 2906348 3136812 3270177 3225946 3176000 3155428 2468400 2226729 2046304 1673587 2083948 "524288" 2204333 2747608 2933752 3139314 3287483 3268567 3222164 3196199 2549344 2297191 1537975 1739883 1699933 "1048576" 2120718 2689787 2917500 3111538 3242070 3260285 3172772 3183696 2298445 1961164 1629435 1705151 2029194 "Re-Fwrite report" "4" "8" "16" "32" "64" "128" "256" "512" "1024" "2048" "4096" "8192" "16384" "64" 969761 1279447 1232453 1526887 1722886 "128" 1145188 1377278 1406136 1561553 1584598 2202044 "256" 1334162 1649865 1855098 2205688 1763685 2187712 2371308 "512" 1446306 1707325 2073249 1985091 2796910 2178404 2521095 2910635 "1024" 1695735 2090206 2353657 2461573 2679655 2586074 2586074 3048658 3220084 "2048" 1741742 2206617 2611870 3240499 2968386 3246623 2734946 2522144 2233004 3251539 "4096" 1584010 2289792 2221111 2576153 3391339 3139133 2652931 2840678 2114496 2355405 2987904 "8192" 1733081 2238301 2390441 2579403 3229054 2713438 3268680 2677075 2166056 1989238 1923535 2790116 "16384" 2003144 2721153 2917924 2613921 2867396 3199408 3010077 3176338 2364190 2254516 2027972 1999007 2597716 "32768" 1861186 2703426 2889570 3097159 3251776 3063334 2989505 2874882 2179432 1923358 1931494 1977800 1903591 "65536" 1912116 2706961 2906398 2787926 3219359 3170273 3028897 3066300 2423843 2092270 2065356 2043203 1953328 "131072" 2205119 2726979 2927474 3020506 3242578 3245008 2965020 3115208 2504139 2138871 2088429 2051757 2072909 "262144" 2179085 2677426 2892075 3120910 3214957 3171575 3184647 3136561 2498397 2172927 2117170 2016927 1981766 "524288" 2216599 2732901 2884790 3069658 3265204 3162766 3218009 3148046 2484825 2273059 1524114 1729039 1683459 "1048576" 2127533 2708155 2909656 3089954 3241587 3282739 3157808 3180790 2277048 1943185 1732427 1588722 2077713 "Fread report" "4" "8" "16" "32" "64" "128" "256" "512" "1024" "2048" "4096" "8192" "16384" "64" 1638743 2006158 2467108 2561267 2662899 "128" 1967960 2727923 2248149 2905056 2660335 3560017 "256" 2533571 3052087 2943325 3935921 2880164 3705040 3454704 "512" 2668325 2753870 3064306 3182373 4571003 4000485 4305250 3736016 "1024" 2774874 3849207 4033569 3923040 4720752 4995276 4674510 4470172 4995276 "2048" 2929912 3777778 4697018 5095447 4653758 5507054 4889518 4603874 4751580 4762117 "4096" 2929815 4230655 4226492 4681411 6408472 4653513 4725189 5019235 5657217 5145513 4196552 "8192" 3123099 3636025 5512724 6045199 5361332 5700206 5708730 5400092 4352746 5155399 3972772 3508313 "16384" 3880525 4580441 5496324 4650496 5657411 5589764 5647647 5432453 5153778 5189585 3503092 3814611 3938574 "32768" 3628003 4577818 4318475 5713813 5905306 5616675 4954932 5345571 5418063 4448610 4227095 3584010 3903760 "65536" 4093685 4207799 4972753 5930764 4799448 5669127 5170492 5334140 5529985 4860209 4014468 3859137 3470294 "131072" 3585610 4687997 4953031 5863511 5789657 5492680 5453179 5608047 5381964 4919567 4382940 4058082 3660723 "262144" 3790624 4951443 5802208 5909611 5854295 5310321 5417786 5645620 5398474 4839446 4631027 4104465 3389758 "524288" 3801120 5448726 5220272 5740019 6323742 5922356 5491533 5751489 5559788 5526089 3452666 3713639 3361147 "1048576" 4155346 5442286 5220067 6351118 6326897 6371922 5641936 5993989 5695469 5327395 4292251 3261796 3681899 "Re-Fread report" "4" "8" "16" "32" "64" "128" "256" "512" "1024" "2048" "4096" "8192" "16384" "64" 19459 3588436 2662899 4274062 3203069 "128" 2464907 3124872 3657016 3536566 4407601 3895854 "256" 3654598 3557725 4332998 4054829 4404088 4496299 7735574 "512" 2811557 4262523 3659616 6174377 4650188 6319739 4927617 6491676 "1024" 2877110 6830356 4110780 6172653 5169641 8542002 6402699 6093831 5564829 "2048" 3835124 4481379 5769681 8758039 5937172 8949659 6588354 5674397 6588354 6225466 "4096" 3098371 4900405 4723889 5634950 8360824 5431844 6041154 6007355 7434579 6555186 4240052 "8192" 3910378 3775461 6145757 6439465 6234975 6527546 6085887 6040947 4636437 5565406 4273171 3588557 "16384" 4145769 4787877 5628681 5919089 6145651 5811959 5915013 5655548 5342079 5343741 3869600 3893497 4005844 "32768" 3700982 4906815 5573625 5824472 5929765 5662493 5840064 5412515 5537917 3986302 4392591 3542076 3927073 "65536" 4131898 4660226 5754339 5851478 5672637 5990477 5655015 5594703 5587539 5176529 4254232 3866193 3581993 "131072" 3648794 4767434 5555063 5928960 5839222 5822092 5563327 5813535 5524139 4951336 4412172 4099724 3666779 "262144" 3841225 5017878 5845704 6135194 6242408 5567829 5459754 5711072 5579441 5212319 4787045 4146474 3600425 "524288" 3847871 5525270 5259338 5868713 6491048 6110149 5730043 6003801 5582824 5558383 3326365 3791937 3375858 "1048576" 4039790 5583893 4931690 6375525 6390328 6412410 5979971 5952405 5967986 5580853 4334853 3286443 3987039 ^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: IOZone Performance is very Strange 2011-05-17 7:34 ` AnnyRen 2011-05-17 8:19 ` Simon Tian @ 2011-05-17 17:05 ` Jeff Wu 2011-05-17 9:14 ` Henry C Chang ` (2 more replies) 1 sibling, 3 replies; 13+ messages in thread From: Jeff Wu @ 2011-05-17 17:05 UTC (permalink / raw) To: AnnyRen; +Cc: ceph-devel@vger.kernel.org Hi AnnyRen, with the test results , run a 1G data at your ceph cluster , should be about 100M/sec. maybe,iozone test results didn't include iozone flush time. Could you list your hardware platform infos ? network:1G,4G,8G,FC ...?,cpu,memory:size ? ,disk:PATA,SATA,SAS,SSD ?? and could you try other iozone commands ,for instance : 1) add "-e" param to include flush(fsync,fflush) in the timing calculations. /usr/bin/iozone -azcR -e -f /mnt/ceph/test0516.dbf \ -g 1G -b /exceloutput0516.xls 2)run a large data which size is your host memory size*2: $./iozone -z -c -e -a -n 512M -g {memory_size}*2M -i 0 -i 1 -i 2 \ -f /mnt/ceph/fio -Rb ./iozone.xls or 2) !/bin/sh for i in 32 64 128 256 do ./iozone -r ${i}k -t 10 -s 4096M -i 0 -i 1 -i 2 \ -F /mnt/ceph/F1 /mnt/ceph/F2 /mnt/ceph/F3 /mnt/ceph/F4 /mnt/ceph/F5 /mnt/ceph/F6 /mnt/ceph/F7 /mnt/ceph/F8 /mnt/ceph/F9 /mnt/ceph/F10 done Jeff On Tue, 2011-05-17 at 15:34 +0800, AnnyRen wrote: > Hi, Jeff: > > I run "ceph osd tell osd_num bench" with 1 times per osd > > and use ceph -w to observe every osd performance, > > osd0: [INF] bench: wrote 1024 MB in blocks of 4096 KB in 10.875844 > sec at 96413 KB/sec > osd1: [INF] bench: wrote 1024 MB in blocks of 4096 KB in 11.784985 > sec at 88975 KB/sec > osd2: [INF] bench: wrote 1024 MB in blocks of 4096 KB in 11.161067 > sec at 93949 KB/sec > osd3: [INF] bench: wrote 1024 MB in blocks of 4096 KB in 10.798796 > sec at 97101 KB/sec > osd4: [INF] bench: wrote 1024 MB in blocks of 4096 KB in 14.437141 > sec at 72630 KB/sec > osd5: [INF] bench: wrote 1024 MB in blocks of 4096 KB in 14.451444 > sec at 72558 KB/sec > osd6: [INF] bench: wrote 1024 MB in blocks of 4096 KB in 11.083872 > sec at 94603 KB/sec > osd7: [INF] bench: wrote 1024 MB in blocks of 4096 KB in 11.062728 > sec at 94784 KB/sec > osd8: [INF] bench: wrote 1024 MB in blocks of 4096 KB in 14.137312 > sec at 74170 KB/sec > osd9: [INF] bench: wrote 1024 MB in blocks of 4096 KB in 13.489992 > sec at 77729 KB/sec > > > and I run > root@MDS2:/mnt/ceph# rados bench 60 write -p data > > the result is > > Total time run: 60.553247 > Total writes made: 1689 > Write size: 4194304 > Bandwidth (MB/sec): 111.571 > > Average Latency: 0.573209 > Max latency: 2.25691 > Min latency: 0.218494 > > > > 2011/5/17 Jeff Wu <cpwu@tnsoft.com.cn>: > > Hi AnnyRen > > > > Could you run the following commands and give us the test results? > > > > $ceph osd tell OSD-N bench // OSD-N : osd number : 0,1,2 .... > > $ceph -w > > > > $rados bench 60 write -p data // refer to "rados -h " > > > > Jeff > > > > > > > > On Tue, 2011-05-17 at 11:53 +0800, AnnyRen wrote: > >> I'm running iozone on EXT4 with Ceph v0.26. > >> But I got the weird result, most write performance exceed 1GB/s, even > >> up to 3GB/s > >> I think it's not normal to get the performance outpupt. > >> > >> Command line I used is: /usr/bin/iozone -azcR -f > >> /mnt/ceph/test0516.dbf -g 1G -b /exceloutput0516.xls > >> Attachment is the output file... > >> > >> and my environment is composed of 15 physical machines with > >> > >> 3 MON, 2MDS (1 active, 1 standby), 10 OSD (1 osd daemon (3T) /host) > >> EXT4 format > >> data replication size: 3 > >> > >> > >> > >> "Writer report" > >> "4" "8" "16" "32" "64" "128" "256" "512" "1024" > >> "2048" "4096" "8192" "16384" > >> "64" 983980 1204796 1210227 1143223 1357066 > >> "128" 1007629 1269781 1406136 1391557 1436229 1521718 > >> "256" 1112909 1430119 1523457 1652404 1514860 1639786 1729594 > >> "512" 1150351 1475116 1605228 1723770 1797349 1712772 1783912 1854787 > >> "1024" 1213334 1471481 1679160 1828574 1888889 1899750 1885572 > >> 1865912 1875690 > >> "2048" 1229274 1540849 1708146 1843410 1903457 1980705 1930406 > >> 1913634 1906837 1815744 > >> "4096" 1213284 1528348 1674646 1762434 1872096 1882352 1881528 > >> 1903416 1897949 1835102 1731177 > >> "8192" 204560 155186 572387 238548 186597 429036 187327 > >> 157205 553771 416512 299810 405842 > >> "16384" 699749 559255 687450 541030 828776 555296 742561 > >> 525483 604910 452423 564557 670539 970616 > >> "32768" 532414 829610 812215 879441 863159 864794 865938 > >> 804951 916352 879582 608132 860732 1239475 > >> "65536" 994824 1096543 1095791 1317968 1280277 1390267 1259868 > >> 1205214 1339111 1346927 1267888 863234 1190221 > >> "131072" 1063429 1165115 1102650 1554828 1182128 1185731 > >> 1190752 1195792 1277441 1211063 1237567 1226999 1336961 > >> "262144" 1280619 1368554 1497837 1633397 1598255 1609212 > >> 1607504 1665019 1590515 1548307 1591258 1505267 1625679 > >> "524288" 1519583 1767928 1738523 1883151 2011216 1993877 > >> 2023543 1867440 2106124 2055064 1906668 1778645 1838988 > >> "1048576" 1580851 1887530 2044131 2166133 2236379 2283578 > >> 2257454 2296612 2271066 2101274 1905829 1605923 2158238 > >> > >> > >> > >> "Reader report" > >> "4" "8" "16" "32" "64" "128" "256" "512" "1024" > >> "2048" "4096" "8192" "16384" > >> "64" 1933893 2801873 3057153 3363612 3958892 > >> "128" 2286447 3053774 2727923 3468030 4104338 4557257 > >> "256" 2903529 3236056 3245838 3705040 3654598 4496299 5117791 > >> "512" 2906696 3437042 3628697 3431550 4871723 4296637 6246213 6395018 > >> "1024" 3229770 3483896 4609294 3791442 4614246 5536137 4550690 > >> 5048117 4966395 > >> "2048" 3554251 4310472 3885431 4096676 6401772 4842658 5080379 > >> 5184636 5596757 5735012 > >> "4096" 3416292 4691638 4321103 5728903 5475122 5171846 4819300 > >> 5258919 6408472 5044289 4079948 > >> "8192" 3233004 4615263 4536055 5618186 5414558 5025700 5553712 > >> 4926264 5634770 5281396 4659702 3652258 > >> "16384" 3141058 3704193 4567654 4395850 4568869 5387732 > >> 4436432 5808029 5578420 4675810 3913007 3911225 3961277 > >> "32768" 3704273 4598957 4088278 5133719 5896692 5537024 > >> 5234412 5398271 4942992 4118662 3729099 3511757 3481511 > >> "65536" 4131091 4210184 5341188 4647619 6077765 5852474 > >> 5379762 5259330 5488249 5246682 4342682 3549202 3286487 > >> "131072" 3582876 5251082 5332216 5269908 5303512 5574440 > >> 5635064 5796372 5406363 4958839 4435918 3673443 3647874 > >> "262144" 3659283 4551414 5746231 5433824 5876196 6011650 > >> 5552000 5629260 5298830 4982226 4628902 4065823 3421924 > >> "524288" 3905973 5488778 5219963 6047356 5916811 6180455 > >> 5495733 5925628 5637512 5537123 3517132 3550861 3047013 > >> "1048576" 3855595 5634571 5410298 6001809 6464713 6299610 > >> 5894249 5516031 5800161 5209203 4295840 3724983 3641623 > > > > ^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: IOZone Performance is very Strange 2011-05-17 17:05 ` Jeff Wu @ 2011-05-17 9:14 ` Henry C Chang 2011-05-17 9:44 ` AnnyRen 2011-05-17 9:37 ` AnnyRen 2011-05-18 5:54 ` AnnyRen 2 siblings, 1 reply; 13+ messages in thread From: Henry C Chang @ 2011-05-17 9:14 UTC (permalink / raw) To: cpwu; +Cc: AnnyRen, ceph-devel@vger.kernel.org Hi Anny, As Jeff mentioned, you need to add the following option for iozone testing: -e: include flush in the timing calculation to remedy the cache effect and may add: -n: set the minimum size for auto mode (e.g., -n 256M) to skip testing of small file sizes and drop -z if you want to omit testing small record sizes. (starting from 64K instead of 4K) -- Henry 2011/5/18 Jeff Wu <cpwu@tnsoft.com.cn>: > > Hi AnnyRen, > > with the test results , run a 1G data at your ceph cluster , > should be about 100M/sec. > maybe,iozone test results didn't include iozone flush time. > > Could you list your hardware platform infos ? > network:1G,4G,8G,FC ...?,cpu,memory:size ? ,disk:PATA,SATA,SAS,SSD ?? > and > could you try other iozone commands ,for instance : > > 1) > add "-e" param to include flush(fsync,fflush) in the timing > calculations. > > /usr/bin/iozone -azcR -e -f /mnt/ceph/test0516.dbf \ > -g 1G -b /exceloutput0516.xls > > 2)run a large data which size is your host memory size*2: > > $./iozone -z -c -e -a -n 512M -g {memory_size}*2M -i 0 -i 1 -i 2 \ > -f /mnt/ceph/fio -Rb ./iozone.xls > > or > 2) > !/bin/sh > > for i in 32 64 128 256 > do > ./iozone -r ${i}k -t 10 -s 4096M -i 0 -i 1 -i 2 \ > -F /mnt/ceph/F1 /mnt/ceph/F2 /mnt/ceph/F3 /mnt/ceph/F4 /mnt/ceph/F5 /mnt/ceph/F6 /mnt/ceph/F7 /mnt/ceph/F8 /mnt/ceph/F9 /mnt/ceph/F10 > done > > > > > Jeff > > > > > > > On Tue, 2011-05-17 at 15:34 +0800, AnnyRen wrote: >> Hi, Jeff: >> >> I run "ceph osd tell osd_num bench" with 1 times per osd >> >> and use ceph -w to observe every osd performance, >> >> osd0: [INF] bench: wrote 1024 MB in blocks of 4096 KB in 10.875844 >> sec at 96413 KB/sec >> osd1: [INF] bench: wrote 1024 MB in blocks of 4096 KB in 11.784985 >> sec at 88975 KB/sec >> osd2: [INF] bench: wrote 1024 MB in blocks of 4096 KB in 11.161067 >> sec at 93949 KB/sec >> osd3: [INF] bench: wrote 1024 MB in blocks of 4096 KB in 10.798796 >> sec at 97101 KB/sec >> osd4: [INF] bench: wrote 1024 MB in blocks of 4096 KB in 14.437141 >> sec at 72630 KB/sec >> osd5: [INF] bench: wrote 1024 MB in blocks of 4096 KB in 14.451444 >> sec at 72558 KB/sec >> osd6: [INF] bench: wrote 1024 MB in blocks of 4096 KB in 11.083872 >> sec at 94603 KB/sec >> osd7: [INF] bench: wrote 1024 MB in blocks of 4096 KB in 11.062728 >> sec at 94784 KB/sec >> osd8: [INF] bench: wrote 1024 MB in blocks of 4096 KB in 14.137312 >> sec at 74170 KB/sec >> osd9: [INF] bench: wrote 1024 MB in blocks of 4096 KB in 13.489992 >> sec at 77729 KB/sec >> >> >> and I run >> root@MDS2:/mnt/ceph# rados bench 60 write -p data >> >> the result is >> >> Total time run: 60.553247 >> Total writes made: 1689 >> Write size: 4194304 >> Bandwidth (MB/sec): 111.571 >> >> Average Latency: 0.573209 >> Max latency: 2.25691 >> Min latency: 0.218494 >> >> >> >> 2011/5/17 Jeff Wu <cpwu@tnsoft.com.cn>: >> > Hi AnnyRen >> > >> > Could you run the following commands and give us the test results? >> > >> > $ceph osd tell OSD-N bench // OSD-N : osd number : 0,1,2 .... >> > $ceph -w >> > >> > $rados bench 60 write -p data // refer to "rados -h " >> > >> > Jeff >> > >> > >> > >> > On Tue, 2011-05-17 at 11:53 +0800, AnnyRen wrote: >> >> I'm running iozone on EXT4 with Ceph v0.26. >> >> But I got the weird result, most write performance exceed 1GB/s, even >> >> up to 3GB/s >> >> I think it's not normal to get the performance outpupt. >> >> >> >> Command line I used is: /usr/bin/iozone -azcR -f >> >> /mnt/ceph/test0516.dbf -g 1G -b /exceloutput0516.xls >> >> Attachment is the output file... >> >> >> >> and my environment is composed of 15 physical machines with >> >> >> >> 3 MON, 2MDS (1 active, 1 standby), 10 OSD (1 osd daemon (3T) /host) >> >> EXT4 format >> >> data replication size: 3 >> >> >> >> >> >> >> >> "Writer report" >> >> "4" "8" "16" "32" "64" "128" "256" "512" "1024" >> >> "2048" "4096" "8192" "16384" >> >> "64" 983980 1204796 1210227 1143223 1357066 >> >> "128" 1007629 1269781 1406136 1391557 1436229 1521718 >> >> "256" 1112909 1430119 1523457 1652404 1514860 1639786 1729594 >> >> "512" 1150351 1475116 1605228 1723770 1797349 1712772 1783912 1854787 >> >> "1024" 1213334 1471481 1679160 1828574 1888889 1899750 1885572 >> >> 1865912 1875690 >> >> "2048" 1229274 1540849 1708146 1843410 1903457 1980705 1930406 >> >> 1913634 1906837 1815744 >> >> "4096" 1213284 1528348 1674646 1762434 1872096 1882352 1881528 >> >> 1903416 1897949 1835102 1731177 >> >> "8192" 204560 155186 572387 238548 186597 429036 187327 >> >> 157205 553771 416512 299810 405842 >> >> "16384" 699749 559255 687450 541030 828776 555296 742561 >> >> 525483 604910 452423 564557 670539 970616 >> >> "32768" 532414 829610 812215 879441 863159 864794 865938 >> >> 804951 916352 879582 608132 860732 1239475 >> >> "65536" 994824 1096543 1095791 1317968 1280277 1390267 1259868 >> >> 1205214 1339111 1346927 1267888 863234 1190221 >> >> "131072" 1063429 1165115 1102650 1554828 1182128 1185731 >> >> 1190752 1195792 1277441 1211063 1237567 1226999 1336961 >> >> "262144" 1280619 1368554 1497837 1633397 1598255 1609212 >> >> 1607504 1665019 1590515 1548307 1591258 1505267 1625679 >> >> "524288" 1519583 1767928 1738523 1883151 2011216 1993877 >> >> 2023543 1867440 2106124 2055064 1906668 1778645 1838988 >> >> "1048576" 1580851 1887530 2044131 2166133 2236379 2283578 >> >> 2257454 2296612 2271066 2101274 1905829 1605923 2158238 >> >> >> >> >> >> >> >> "Reader report" >> >> "4" "8" "16" "32" "64" "128" "256" "512" "1024" >> >> "2048" "4096" "8192" "16384" >> >> "64" 1933893 2801873 3057153 3363612 3958892 >> >> "128" 2286447 3053774 2727923 3468030 4104338 4557257 >> >> "256" 2903529 3236056 3245838 3705040 3654598 4496299 5117791 >> >> "512" 2906696 3437042 3628697 3431550 4871723 4296637 6246213 6395018 >> >> "1024" 3229770 3483896 4609294 3791442 4614246 5536137 4550690 >> >> 5048117 4966395 >> >> "2048" 3554251 4310472 3885431 4096676 6401772 4842658 5080379 >> >> 5184636 5596757 5735012 >> >> "4096" 3416292 4691638 4321103 5728903 5475122 5171846 4819300 >> >> 5258919 6408472 5044289 4079948 >> >> "8192" 3233004 4615263 4536055 5618186 5414558 5025700 5553712 >> >> 4926264 5634770 5281396 4659702 3652258 >> >> "16384" 3141058 3704193 4567654 4395850 4568869 5387732 >> >> 4436432 5808029 5578420 4675810 3913007 3911225 3961277 >> >> "32768" 3704273 4598957 4088278 5133719 5896692 5537024 >> >> 5234412 5398271 4942992 4118662 3729099 3511757 3481511 >> >> "65536" 4131091 4210184 5341188 4647619 6077765 5852474 >> >> 5379762 5259330 5488249 5246682 4342682 3549202 3286487 >> >> "131072" 3582876 5251082 5332216 5269908 5303512 5574440 >> >> 5635064 5796372 5406363 4958839 4435918 3673443 3647874 >> >> "262144" 3659283 4551414 5746231 5433824 5876196 6011650 >> >> 5552000 5629260 5298830 4982226 4628902 4065823 3421924 >> >> "524288" 3905973 5488778 5219963 6047356 5916811 6180455 >> >> 5495733 5925628 5637512 5537123 3517132 3550861 3047013 >> >> "1048576" 3855595 5634571 5410298 6001809 6464713 6299610 >> >> 5894249 5516031 5800161 5209203 4295840 3724983 3641623 >> > >> > > > -- > To unsubscribe from this list: send the line "unsubscribe ceph-devel" in > the body of a message to majordomo@vger.kernel.org > More majordomo info at http://vger.kernel.org/majordomo-info.html > -- To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html ^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: IOZone Performance is very Strange 2011-05-17 9:14 ` Henry C Chang @ 2011-05-17 9:44 ` AnnyRen 0 siblings, 0 replies; 13+ messages in thread From: AnnyRen @ 2011-05-17 9:44 UTC (permalink / raw) To: Henry C Chang; +Cc: cpwu, ceph-devel@vger.kernel.org Hi Henry, Thanks for your suggestion! We'll try it. :) Hopefully come out with good result! Thank you. Best Regards, Anny 2011/5/17 Henry C Chang <henry.cy.chang@gmail.com>: > Hi Anny, > > As Jeff mentioned, you need to add the following option for iozone testing: > -e: include flush in the timing calculation to remedy the cache effect > > and may add: > -n: set the minimum size for auto mode (e.g., -n 256M) to skip testing > of small file sizes > > and drop -z if you want to omit testing small record sizes. (starting > from 64K instead of 4K) > > -- > Henry > > > 2011/5/18 Jeff Wu <cpwu@tnsoft.com.cn>: >> >> Hi AnnyRen, >> >> with the test results , run a 1G data at your ceph cluster , >> should be about 100M/sec. >> maybe,iozone test results didn't include iozone flush time. >> >> Could you list your hardware platform infos ? >> network:1G,4G,8G,FC ...?,cpu,memory:size ? ,disk:PATA,SATA,SAS,SSD ?? >> and >> could you try other iozone commands ,for instance : >> >> 1) >> add "-e" param to include flush(fsync,fflush) in the timing >> calculations. >> >> /usr/bin/iozone -azcR -e -f /mnt/ceph/test0516.dbf \ >> -g 1G -b /exceloutput0516.xls >> >> 2)run a large data which size is your host memory size*2: >> >> $./iozone -z -c -e -a -n 512M -g {memory_size}*2M -i 0 -i 1 -i 2 \ >> -f /mnt/ceph/fio -Rb ./iozone.xls >> >> or >> 2) >> !/bin/sh >> >> for i in 32 64 128 256 >> do >> ./iozone -r ${i}k -t 10 -s 4096M -i 0 -i 1 -i 2 \ >> -F /mnt/ceph/F1 /mnt/ceph/F2 /mnt/ceph/F3 /mnt/ceph/F4 /mnt/ceph/F5 /mnt/ceph/F6 /mnt/ceph/F7 /mnt/ceph/F8 /mnt/ceph/F9 /mnt/ceph/F10 >> done >> >> >> >> >> Jeff >> >> >> >> >> >> >> On Tue, 2011-05-17 at 15:34 +0800, AnnyRen wrote: >>> Hi, Jeff: >>> >>> I run "ceph osd tell osd_num bench" with 1 times per osd >>> >>> and use ceph -w to observe every osd performance, >>> >>> osd0: [INF] bench: wrote 1024 MB in blocks of 4096 KB in 10.875844 >>> sec at 96413 KB/sec >>> osd1: [INF] bench: wrote 1024 MB in blocks of 4096 KB in 11.784985 >>> sec at 88975 KB/sec >>> osd2: [INF] bench: wrote 1024 MB in blocks of 4096 KB in 11.161067 >>> sec at 93949 KB/sec >>> osd3: [INF] bench: wrote 1024 MB in blocks of 4096 KB in 10.798796 >>> sec at 97101 KB/sec >>> osd4: [INF] bench: wrote 1024 MB in blocks of 4096 KB in 14.437141 >>> sec at 72630 KB/sec >>> osd5: [INF] bench: wrote 1024 MB in blocks of 4096 KB in 14.451444 >>> sec at 72558 KB/sec >>> osd6: [INF] bench: wrote 1024 MB in blocks of 4096 KB in 11.083872 >>> sec at 94603 KB/sec >>> osd7: [INF] bench: wrote 1024 MB in blocks of 4096 KB in 11.062728 >>> sec at 94784 KB/sec >>> osd8: [INF] bench: wrote 1024 MB in blocks of 4096 KB in 14.137312 >>> sec at 74170 KB/sec >>> osd9: [INF] bench: wrote 1024 MB in blocks of 4096 KB in 13.489992 >>> sec at 77729 KB/sec >>> >>> >>> and I run >>> root@MDS2:/mnt/ceph# rados bench 60 write -p data >>> >>> the result is >>> >>> Total time run: 60.553247 >>> Total writes made: 1689 >>> Write size: 4194304 >>> Bandwidth (MB/sec): 111.571 >>> >>> Average Latency: 0.573209 >>> Max latency: 2.25691 >>> Min latency: 0.218494 >>> >>> >>> >>> 2011/5/17 Jeff Wu <cpwu@tnsoft.com.cn>: >>> > Hi AnnyRen >>> > >>> > Could you run the following commands and give us the test results? >>> > >>> > $ceph osd tell OSD-N bench // OSD-N : osd number : 0,1,2 .... >>> > $ceph -w >>> > >>> > $rados bench 60 write -p data // refer to "rados -h " >>> > >>> > Jeff >>> > >>> > >>> > >>> > On Tue, 2011-05-17 at 11:53 +0800, AnnyRen wrote: >>> >> I'm running iozone on EXT4 with Ceph v0.26. >>> >> But I got the weird result, most write performance exceed 1GB/s, even >>> >> up to 3GB/s >>> >> I think it's not normal to get the performance outpupt. >>> >> >>> >> Command line I used is: /usr/bin/iozone -azcR -f >>> >> /mnt/ceph/test0516.dbf -g 1G -b /exceloutput0516.xls >>> >> Attachment is the output file... >>> >> >>> >> and my environment is composed of 15 physical machines with >>> >> >>> >> 3 MON, 2MDS (1 active, 1 standby), 10 OSD (1 osd daemon (3T) /host) >>> >> EXT4 format >>> >> data replication size: 3 >>> >> >>> >> >>> >> >>> >> "Writer report" >>> >> "4" "8" "16" "32" "64" "128" "256" "512" "1024" >>> >> "2048" "4096" "8192" "16384" >>> >> "64" 983980 1204796 1210227 1143223 1357066 >>> >> "128" 1007629 1269781 1406136 1391557 1436229 1521718 >>> >> "256" 1112909 1430119 1523457 1652404 1514860 1639786 1729594 >>> >> "512" 1150351 1475116 1605228 1723770 1797349 1712772 1783912 1854787 >>> >> "1024" 1213334 1471481 1679160 1828574 1888889 1899750 1885572 >>> >> 1865912 1875690 >>> >> "2048" 1229274 1540849 1708146 1843410 1903457 1980705 1930406 >>> >> 1913634 1906837 1815744 >>> >> "4096" 1213284 1528348 1674646 1762434 1872096 1882352 1881528 >>> >> 1903416 1897949 1835102 1731177 >>> >> "8192" 204560 155186 572387 238548 186597 429036 187327 >>> >> 157205 553771 416512 299810 405842 >>> >> "16384" 699749 559255 687450 541030 828776 555296 742561 >>> >> 525483 604910 452423 564557 670539 970616 >>> >> "32768" 532414 829610 812215 879441 863159 864794 865938 >>> >> 804951 916352 879582 608132 860732 1239475 >>> >> "65536" 994824 1096543 1095791 1317968 1280277 1390267 1259868 >>> >> 1205214 1339111 1346927 1267888 863234 1190221 >>> >> "131072" 1063429 1165115 1102650 1554828 1182128 1185731 >>> >> 1190752 1195792 1277441 1211063 1237567 1226999 1336961 >>> >> "262144" 1280619 1368554 1497837 1633397 1598255 1609212 >>> >> 1607504 1665019 1590515 1548307 1591258 1505267 1625679 >>> >> "524288" 1519583 1767928 1738523 1883151 2011216 1993877 >>> >> 2023543 1867440 2106124 2055064 1906668 1778645 1838988 >>> >> "1048576" 1580851 1887530 2044131 2166133 2236379 2283578 >>> >> 2257454 2296612 2271066 2101274 1905829 1605923 2158238 >>> >> >>> >> >>> >> >>> >> "Reader report" >>> >> "4" "8" "16" "32" "64" "128" "256" "512" "1024" >>> >> "2048" "4096" "8192" "16384" >>> >> "64" 1933893 2801873 3057153 3363612 3958892 >>> >> "128" 2286447 3053774 2727923 3468030 4104338 4557257 >>> >> "256" 2903529 3236056 3245838 3705040 3654598 4496299 5117791 >>> >> "512" 2906696 3437042 3628697 3431550 4871723 4296637 6246213 6395018 >>> >> "1024" 3229770 3483896 4609294 3791442 4614246 5536137 4550690 >>> >> 5048117 4966395 >>> >> "2048" 3554251 4310472 3885431 4096676 6401772 4842658 5080379 >>> >> 5184636 5596757 5735012 >>> >> "4096" 3416292 4691638 4321103 5728903 5475122 5171846 4819300 >>> >> 5258919 6408472 5044289 4079948 >>> >> "8192" 3233004 4615263 4536055 5618186 5414558 5025700 5553712 >>> >> 4926264 5634770 5281396 4659702 3652258 >>> >> "16384" 3141058 3704193 4567654 4395850 4568869 5387732 >>> >> 4436432 5808029 5578420 4675810 3913007 3911225 3961277 >>> >> "32768" 3704273 4598957 4088278 5133719 5896692 5537024 >>> >> 5234412 5398271 4942992 4118662 3729099 3511757 3481511 >>> >> "65536" 4131091 4210184 5341188 4647619 6077765 5852474 >>> >> 5379762 5259330 5488249 5246682 4342682 3549202 3286487 >>> >> "131072" 3582876 5251082 5332216 5269908 5303512 5574440 >>> >> 5635064 5796372 5406363 4958839 4435918 3673443 3647874 >>> >> "262144" 3659283 4551414 5746231 5433824 5876196 6011650 >>> >> 5552000 5629260 5298830 4982226 4628902 4065823 3421924 >>> >> "524288" 3905973 5488778 5219963 6047356 5916811 6180455 >>> >> 5495733 5925628 5637512 5537123 3517132 3550861 3047013 >>> >> "1048576" 3855595 5634571 5410298 6001809 6464713 6299610 >>> >> 5894249 5516031 5800161 5209203 4295840 3724983 3641623 >>> > >>> > >> >> -- >> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in >> the body of a message to majordomo@vger.kernel.org >> More majordomo info at http://vger.kernel.org/majordomo-info.html >> > -- To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html ^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: IOZone Performance is very Strange 2011-05-17 17:05 ` Jeff Wu 2011-05-17 9:14 ` Henry C Chang @ 2011-05-17 9:37 ` AnnyRen 2011-05-18 5:54 ` AnnyRen 2 siblings, 0 replies; 13+ messages in thread From: AnnyRen @ 2011-05-17 9:37 UTC (permalink / raw) To: cpwu; +Cc: ceph-devel@vger.kernel.org Hi, Jeff: Thanks for your quickly suggestion and response~ We will add -e param to include flush time in the calculation, and write a larger file to avoid using the machine's memory. Thank you very much! :) Best Regards, Anny 2011/5/18 Jeff Wu <cpwu@tnsoft.com.cn>: > > Hi AnnyRen, > > with the test results , run a 1G data at your ceph cluster , > should be about 100M/sec. > maybe,iozone test results didn't include iozone flush time. > > Could you list your hardware platform infos ? > network:1G,4G,8G,FC ...?,cpu,memory:size ? ,disk:PATA,SATA,SAS,SSD ?? > and > could you try other iozone commands ,for instance : > > 1) > add "-e" param to include flush(fsync,fflush) in the timing > calculations. > > /usr/bin/iozone -azcR -e -f /mnt/ceph/test0516.dbf \ > -g 1G -b /exceloutput0516.xls > > 2)run a large data which size is your host memory size*2: > > $./iozone -z -c -e -a -n 512M -g {memory_size}*2M -i 0 -i 1 -i 2 \ > -f /mnt/ceph/fio -Rb ./iozone.xls > > or > 2) > !/bin/sh > > for i in 32 64 128 256 > do > ./iozone -r ${i}k -t 10 -s 4096M -i 0 -i 1 -i 2 \ > -F /mnt/ceph/F1 /mnt/ceph/F2 /mnt/ceph/F3 /mnt/ceph/F4 /mnt/ceph/F5 /mnt/ceph/F6 /mnt/ceph/F7 /mnt/ceph/F8 /mnt/ceph/F9 /mnt/ceph/F10 > done > > > > > Jeff > > > > > > > On Tue, 2011-05-17 at 15:34 +0800, AnnyRen wrote: >> Hi, Jeff: >> >> I run "ceph osd tell osd_num bench" with 1 times per osd >> >> and use ceph -w to observe every osd performance, >> >> osd0: [INF] bench: wrote 1024 MB in blocks of 4096 KB in 10.875844 >> sec at 96413 KB/sec >> osd1: [INF] bench: wrote 1024 MB in blocks of 4096 KB in 11.784985 >> sec at 88975 KB/sec >> osd2: [INF] bench: wrote 1024 MB in blocks of 4096 KB in 11.161067 >> sec at 93949 KB/sec >> osd3: [INF] bench: wrote 1024 MB in blocks of 4096 KB in 10.798796 >> sec at 97101 KB/sec >> osd4: [INF] bench: wrote 1024 MB in blocks of 4096 KB in 14.437141 >> sec at 72630 KB/sec >> osd5: [INF] bench: wrote 1024 MB in blocks of 4096 KB in 14.451444 >> sec at 72558 KB/sec >> osd6: [INF] bench: wrote 1024 MB in blocks of 4096 KB in 11.083872 >> sec at 94603 KB/sec >> osd7: [INF] bench: wrote 1024 MB in blocks of 4096 KB in 11.062728 >> sec at 94784 KB/sec >> osd8: [INF] bench: wrote 1024 MB in blocks of 4096 KB in 14.137312 >> sec at 74170 KB/sec >> osd9: [INF] bench: wrote 1024 MB in blocks of 4096 KB in 13.489992 >> sec at 77729 KB/sec >> >> >> and I run >> root@MDS2:/mnt/ceph# rados bench 60 write -p data >> >> the result is >> >> Total time run: 60.553247 >> Total writes made: 1689 >> Write size: 4194304 >> Bandwidth (MB/sec): 111.571 >> >> Average Latency: 0.573209 >> Max latency: 2.25691 >> Min latency: 0.218494 >> >> >> >> 2011/5/17 Jeff Wu <cpwu@tnsoft.com.cn>: >> > Hi AnnyRen >> > >> > Could you run the following commands and give us the test results? >> > >> > $ceph osd tell OSD-N bench // OSD-N : osd number : 0,1,2 .... >> > $ceph -w >> > >> > $rados bench 60 write -p data // refer to "rados -h " >> > >> > Jeff >> > >> > >> > >> > On Tue, 2011-05-17 at 11:53 +0800, AnnyRen wrote: >> >> I'm running iozone on EXT4 with Ceph v0.26. >> >> But I got the weird result, most write performance exceed 1GB/s, even >> >> up to 3GB/s >> >> I think it's not normal to get the performance outpupt. >> >> >> >> Command line I used is: /usr/bin/iozone -azcR -f >> >> /mnt/ceph/test0516.dbf -g 1G -b /exceloutput0516.xls >> >> Attachment is the output file... >> >> >> >> and my environment is composed of 15 physical machines with >> >> >> >> 3 MON, 2MDS (1 active, 1 standby), 10 OSD (1 osd daemon (3T) /host) >> >> EXT4 format >> >> data replication size: 3 >> >> >> >> >> >> >> >> "Writer report" >> >> "4" "8" "16" "32" "64" "128" "256" "512" "1024" >> >> "2048" "4096" "8192" "16384" >> >> "64" 983980 1204796 1210227 1143223 1357066 >> >> "128" 1007629 1269781 1406136 1391557 1436229 1521718 >> >> "256" 1112909 1430119 1523457 1652404 1514860 1639786 1729594 >> >> "512" 1150351 1475116 1605228 1723770 1797349 1712772 1783912 1854787 >> >> "1024" 1213334 1471481 1679160 1828574 1888889 1899750 1885572 >> >> 1865912 1875690 >> >> "2048" 1229274 1540849 1708146 1843410 1903457 1980705 1930406 >> >> 1913634 1906837 1815744 >> >> "4096" 1213284 1528348 1674646 1762434 1872096 1882352 1881528 >> >> 1903416 1897949 1835102 1731177 >> >> "8192" 204560 155186 572387 238548 186597 429036 187327 >> >> 157205 553771 416512 299810 405842 >> >> "16384" 699749 559255 687450 541030 828776 555296 742561 >> >> 525483 604910 452423 564557 670539 970616 >> >> "32768" 532414 829610 812215 879441 863159 864794 865938 >> >> 804951 916352 879582 608132 860732 1239475 >> >> "65536" 994824 1096543 1095791 1317968 1280277 1390267 1259868 >> >> 1205214 1339111 1346927 1267888 863234 1190221 >> >> "131072" 1063429 1165115 1102650 1554828 1182128 1185731 >> >> 1190752 1195792 1277441 1211063 1237567 1226999 1336961 >> >> "262144" 1280619 1368554 1497837 1633397 1598255 1609212 >> >> 1607504 1665019 1590515 1548307 1591258 1505267 1625679 >> >> "524288" 1519583 1767928 1738523 1883151 2011216 1993877 >> >> 2023543 1867440 2106124 2055064 1906668 1778645 1838988 >> >> "1048576" 1580851 1887530 2044131 2166133 2236379 2283578 >> >> 2257454 2296612 2271066 2101274 1905829 1605923 2158238 >> >> >> >> >> >> >> >> "Reader report" >> >> "4" "8" "16" "32" "64" "128" "256" "512" "1024" >> >> "2048" "4096" "8192" "16384" >> >> "64" 1933893 2801873 3057153 3363612 3958892 >> >> "128" 2286447 3053774 2727923 3468030 4104338 4557257 >> >> "256" 2903529 3236056 3245838 3705040 3654598 4496299 5117791 >> >> "512" 2906696 3437042 3628697 3431550 4871723 4296637 6246213 6395018 >> >> "1024" 3229770 3483896 4609294 3791442 4614246 5536137 4550690 >> >> 5048117 4966395 >> >> "2048" 3554251 4310472 3885431 4096676 6401772 4842658 5080379 >> >> 5184636 5596757 5735012 >> >> "4096" 3416292 4691638 4321103 5728903 5475122 5171846 4819300 >> >> 5258919 6408472 5044289 4079948 >> >> "8192" 3233004 4615263 4536055 5618186 5414558 5025700 5553712 >> >> 4926264 5634770 5281396 4659702 3652258 >> >> "16384" 3141058 3704193 4567654 4395850 4568869 5387732 >> >> 4436432 5808029 5578420 4675810 3913007 3911225 3961277 >> >> "32768" 3704273 4598957 4088278 5133719 5896692 5537024 >> >> 5234412 5398271 4942992 4118662 3729099 3511757 3481511 >> >> "65536" 4131091 4210184 5341188 4647619 6077765 5852474 >> >> 5379762 5259330 5488249 5246682 4342682 3549202 3286487 >> >> "131072" 3582876 5251082 5332216 5269908 5303512 5574440 >> >> 5635064 5796372 5406363 4958839 4435918 3673443 3647874 >> >> "262144" 3659283 4551414 5746231 5433824 5876196 6011650 >> >> 5552000 5629260 5298830 4982226 4628902 4065823 3421924 >> >> "524288" 3905973 5488778 5219963 6047356 5916811 6180455 >> >> 5495733 5925628 5637512 5537123 3517132 3550861 3047013 >> >> "1048576" 3855595 5634571 5410298 6001809 6464713 6299610 >> >> 5894249 5516031 5800161 5209203 4295840 3724983 3641623 >> > >> > > > -- To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html ^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: IOZone Performance is very Strange 2011-05-17 17:05 ` Jeff Wu 2011-05-17 9:14 ` Henry C Chang 2011-05-17 9:37 ` AnnyRen @ 2011-05-18 5:54 ` AnnyRen 2011-05-18 14:08 ` Jeff Wu 2 siblings, 1 reply; 13+ messages in thread From: AnnyRen @ 2011-05-18 5:54 UTC (permalink / raw) To: cpwu, jerry1209; +Cc: ceph-devel@vger.kernel.org Hi Jeff & Henry, I took your advice and used the following command to run iozone test /usr/bin/iozone -z -c -e -a -n 512M -g 2G -i 0 -i 1 -i 2 -f /mnt/ceph/file0517.dbf -Rb /excel0517.xls Attachment is the output result. Writer Report 4 8 16 32 64 128 256 512 1024 2048 4096 8192 16384 524288 97920 101778 97050 103121 103445 102137 101082 101993 98835 103674 102043 99424 102679 1048576 104306 104841 103616 105884 106044 102221 105666 105890 105811 106409 103957 104062 102115 2097152 106896 107989 108428 107500 108453 108444 109584 108618 109413 108102 107529 108357 107545 Reader Report 4 8 16 32 64 128 256 512 1024 2048 4096 8192 16384 524288 3756286 5029629 5049864 5810960 5577953 5554606 5388867 5636832 5709141 5253596 3805560 3640738 3442332 1048576 3709316 4733745 5400857 5655874 5788923 5565162 5865768 5546521 5886345 5167771 4485693 3531487 3473877 2097152 4305329 5103282 5713626 6399318 6180508 6392487 5804140 4621404 5873338 5837715 4181626 3420274 3506328 the file size grows from 512 MB to 2 GB, write performance is about 94 MB/s~ 106 MB/s and read performance still exceed 3 GB~6 GB, is it too fast ? Or we used the cache or buffer but we don't know? if yes, how could we avoid to use the cache to read ? Thanks very much. :-) Best Regards, Anny 2011/5/18 Jeff Wu <cpwu@tnsoft.com.cn>: > > Hi AnnyRen, > > with the test results , run a 1G data at your ceph cluster , > should be about 100M/sec. > maybe,iozone test results didn't include iozone flush time. > > Could you list your hardware platform infos ? > network:1G,4G,8G,FC ...?,cpu,memory:size ? ,disk:PATA,SATA,SAS,SSD ?? > and > could you try other iozone commands ,for instance : > > 1) > add "-e" param to include flush(fsync,fflush) in the timing > calculations. > > /usr/bin/iozone -azcR -e -f /mnt/ceph/test0516.dbf \ > -g 1G -b /exceloutput0516.xls > > 2)run a large data which size is your host memory size*2: > > $./iozone -z -c -e -a -n 512M -g {memory_size}*2M -i 0 -i 1 -i 2 \ > -f /mnt/ceph/fio -Rb ./iozone.xls > > or > 2) > !/bin/sh > > for i in 32 64 128 256 > do > ./iozone -r ${i}k -t 10 -s 4096M -i 0 -i 1 -i 2 \ > -F /mnt/ceph/F1 /mnt/ceph/F2 /mnt/ceph/F3 /mnt/ceph/F4 /mnt/ceph/F5 /mnt/ceph/F6 /mnt/ceph/F7 /mnt/ceph/F8 /mnt/ceph/F9 /mnt/ceph/F10 > done > > > > > Jeff > > > > > > > On Tue, 2011-05-17 at 15:34 +0800, AnnyRen wrote: >> Hi, Jeff: >> >> I run "ceph osd tell osd_num bench" with 1 times per osd >> >> and use ceph -w to observe every osd performance, >> >> osd0: [INF] bench: wrote 1024 MB in blocks of 4096 KB in 10.875844 >> sec at 96413 KB/sec >> osd1: [INF] bench: wrote 1024 MB in blocks of 4096 KB in 11.784985 >> sec at 88975 KB/sec >> osd2: [INF] bench: wrote 1024 MB in blocks of 4096 KB in 11.161067 >> sec at 93949 KB/sec >> osd3: [INF] bench: wrote 1024 MB in blocks of 4096 KB in 10.798796 >> sec at 97101 KB/sec >> osd4: [INF] bench: wrote 1024 MB in blocks of 4096 KB in 14.437141 >> sec at 72630 KB/sec >> osd5: [INF] bench: wrote 1024 MB in blocks of 4096 KB in 14.451444 >> sec at 72558 KB/sec >> osd6: [INF] bench: wrote 1024 MB in blocks of 4096 KB in 11.083872 >> sec at 94603 KB/sec >> osd7: [INF] bench: wrote 1024 MB in blocks of 4096 KB in 11.062728 >> sec at 94784 KB/sec >> osd8: [INF] bench: wrote 1024 MB in blocks of 4096 KB in 14.137312 >> sec at 74170 KB/sec >> osd9: [INF] bench: wrote 1024 MB in blocks of 4096 KB in 13.489992 >> sec at 77729 KB/sec >> >> >> and I run >> root@MDS2:/mnt/ceph# rados bench 60 write -p data >> >> the result is >> >> Total time run: 60.553247 >> Total writes made: 1689 >> Write size: 4194304 >> Bandwidth (MB/sec): 111.571 >> >> Average Latency: 0.573209 >> Max latency: 2.25691 >> Min latency: 0.218494 >> >> >> >> 2011/5/17 Jeff Wu <cpwu@tnsoft.com.cn>: >> > Hi AnnyRen >> > >> > Could you run the following commands and give us the test results? >> > >> > $ceph osd tell OSD-N bench // OSD-N : osd number : 0,1,2 .... >> > $ceph -w >> > >> > $rados bench 60 write -p data // refer to "rados -h " >> > >> > Jeff >> > >> > >> > >> > On Tue, 2011-05-17 at 11:53 +0800, AnnyRen wrote: >> >> I'm running iozone on EXT4 with Ceph v0.26. >> >> But I got the weird result, most write performance exceed 1GB/s, even >> >> up to 3GB/s >> >> I think it's not normal to get the performance outpupt. >> >> >> >> Command line I used is: /usr/bin/iozone -azcR -f >> >> /mnt/ceph/test0516.dbf -g 1G -b /exceloutput0516.xls >> >> Attachment is the output file... >> >> >> >> and my environment is composed of 15 physical machines with >> >> >> >> 3 MON, 2MDS (1 active, 1 standby), 10 OSD (1 osd daemon (3T) /host) >> >> EXT4 format >> >> data replication size: 3 >> >> >> >> >> >> >> >> "Writer report" >> >> "4" "8" "16" "32" "64" "128" "256" "512" "1024" >> >> "2048" "4096" "8192" "16384" >> >> "64" 983980 1204796 1210227 1143223 1357066 >> >> "128" 1007629 1269781 1406136 1391557 1436229 1521718 >> >> "256" 1112909 1430119 1523457 1652404 1514860 1639786 1729594 >> >> "512" 1150351 1475116 1605228 1723770 1797349 1712772 1783912 1854787 >> >> "1024" 1213334 1471481 1679160 1828574 1888889 1899750 1885572 >> >> 1865912 1875690 >> >> "2048" 1229274 1540849 1708146 1843410 1903457 1980705 1930406 >> >> 1913634 1906837 1815744 >> >> "4096" 1213284 1528348 1674646 1762434 1872096 1882352 1881528 >> >> 1903416 1897949 1835102 1731177 >> >> "8192" 204560 155186 572387 238548 186597 429036 187327 >> >> 157205 553771 416512 299810 405842 >> >> "16384" 699749 559255 687450 541030 828776 555296 742561 >> >> 525483 604910 452423 564557 670539 970616 >> >> "32768" 532414 829610 812215 879441 863159 864794 865938 >> >> 804951 916352 879582 608132 860732 1239475 >> >> "65536" 994824 1096543 1095791 1317968 1280277 1390267 1259868 >> >> 1205214 1339111 1346927 1267888 863234 1190221 >> >> "131072" 1063429 1165115 1102650 1554828 1182128 1185731 >> >> 1190752 1195792 1277441 1211063 1237567 1226999 1336961 >> >> "262144" 1280619 1368554 1497837 1633397 1598255 1609212 >> >> 1607504 1665019 1590515 1548307 1591258 1505267 1625679 >> >> "524288" 1519583 1767928 1738523 1883151 2011216 1993877 >> >> 2023543 1867440 2106124 2055064 1906668 1778645 1838988 >> >> "1048576" 1580851 1887530 2044131 2166133 2236379 2283578 >> >> 2257454 2296612 2271066 2101274 1905829 1605923 2158238 >> >> >> >> >> >> >> >> "Reader report" >> >> "4" "8" "16" "32" "64" "128" "256" "512" "1024" >> >> "2048" "4096" "8192" "16384" >> >> "64" 1933893 2801873 3057153 3363612 3958892 >> >> "128" 2286447 3053774 2727923 3468030 4104338 4557257 >> >> "256" 2903529 3236056 3245838 3705040 3654598 4496299 5117791 >> >> "512" 2906696 3437042 3628697 3431550 4871723 4296637 6246213 6395018 >> >> "1024" 3229770 3483896 4609294 3791442 4614246 5536137 4550690 >> >> 5048117 4966395 >> >> "2048" 3554251 4310472 3885431 4096676 6401772 4842658 5080379 >> >> 5184636 5596757 5735012 >> >> "4096" 3416292 4691638 4321103 5728903 5475122 5171846 4819300 >> >> 5258919 6408472 5044289 4079948 >> >> "8192" 3233004 4615263 4536055 5618186 5414558 5025700 5553712 >> >> 4926264 5634770 5281396 4659702 3652258 >> >> "16384" 3141058 3704193 4567654 4395850 4568869 5387732 >> >> 4436432 5808029 5578420 4675810 3913007 3911225 3961277 >> >> "32768" 3704273 4598957 4088278 5133719 5896692 5537024 >> >> 5234412 5398271 4942992 4118662 3729099 3511757 3481511 >> >> "65536" 4131091 4210184 5341188 4647619 6077765 5852474 >> >> 5379762 5259330 5488249 5246682 4342682 3549202 3286487 >> >> "131072" 3582876 5251082 5332216 5269908 5303512 5574440 >> >> 5635064 5796372 5406363 4958839 4435918 3673443 3647874 >> >> "262144" 3659283 4551414 5746231 5433824 5876196 6011650 >> >> 5552000 5629260 5298830 4982226 4628902 4065823 3421924 >> >> "524288" 3905973 5488778 5219963 6047356 5916811 6180455 >> >> 5495733 5925628 5637512 5537123 3517132 3550861 3047013 >> >> "1048576" 3855595 5634571 5410298 6001809 6464713 6299610 >> >> 5894249 5516031 5800161 5209203 4295840 3724983 3641623 >> > >> > > > -- To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html ^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: IOZone Performance is very Strange 2011-05-18 5:54 ` AnnyRen @ 2011-05-18 14:08 ` Jeff Wu 2011-05-18 6:24 ` AnnyRen 0 siblings, 1 reply; 13+ messages in thread From: Jeff Wu @ 2011-05-18 14:08 UTC (permalink / raw) To: AnnyRen; +Cc: jerry1209@cht.com.tw, ceph-devel@vger.kernel.org Hi , Normally , read performance should be more than write performance ,about 20% ~ 30% . Could you run the following commands and attach all of the results ? $rados bench 30 write -p data //write performance $rados bench 30 seq -p data // read performance // On Wed, 2011-05-18 at 13:54 +0800, AnnyRen wrote: > Hi Jeff & Henry, > > I took your advice and used the following command to run iozone test > > /usr/bin/iozone -z -c -e -a -n 512M -g 2G -i 0 -i 1 -i 2 -f > /mnt/ceph/file0517.dbf -Rb /excel0517.xls > > Attachment is the output result. > > Writer Report > 4 8 16 32 64 128 256 512 1024 2048 4096 8192 16384 > 524288 97920 101778 97050 103121 103445 102137 101082 101993 98835 103674 102043 99424 102679 > 1048576 104306 104841 103616 105884 106044 102221 105666 105890 105811 106409 103957 104062 102115 > 2097152 106896 107989 108428 107500 108453 108444 109584 108618 109413 108102 107529 108357 107545 > > Reader Report > 4 8 16 32 64 128 256 512 1024 2048 4096 8192 16384 > 524288 3756286 5029629 5049864 5810960 5577953 5554606 5388867 5636832 5709141 5253596 3805560 3640738 3442332 > 1048576 3709316 4733745 5400857 5655874 5788923 5565162 5865768 5546521 5886345 5167771 4485693 3531487 3473877 > 2097152 4305329 5103282 5713626 6399318 6180508 6392487 5804140 4621404 5873338 5837715 4181626 3420274 3506328 > > > the file size grows from 512 MB to 2 GB, write performance is about 94 > MB/s~ 106 MB/s > and read performance still exceed 3 GB~6 GB, is it too fast ? > > Or we used the cache or buffer but we don't know? if yes, how could we > avoid to use the cache to read ? > Thanks very much. :-) > > Best Regards, > Anny > > > > > 2011/5/18 Jeff Wu <cpwu@tnsoft.com.cn>: > > > > Hi AnnyRen, > > > > with the test results , run a 1G data at your ceph cluster , > > should be about 100M/sec. > > maybe,iozone test results didn't include iozone flush time. > > > > Could you list your hardware platform infos ? > > network:1G,4G,8G,FC ...?,cpu,memory:size ? ,disk:PATA,SATA,SAS,SSD ?? > > and > > could you try other iozone commands ,for instance : > > > > 1) > > add "-e" param to include flush(fsync,fflush) in the timing > > calculations. > > > > /usr/bin/iozone -azcR -e -f /mnt/ceph/test0516.dbf \ > > -g 1G -b /exceloutput0516.xls > > > > 2)run a large data which size is your host memory size*2: > > > > $./iozone -z -c -e -a -n 512M -g {memory_size}*2M -i 0 -i 1 -i 2 \ > > -f /mnt/ceph/fio -Rb ./iozone.xls > > > > or > > 2) > > !/bin/sh > > > > for i in 32 64 128 256 > > do > > ./iozone -r ${i}k -t 10 -s 4096M -i 0 -i 1 -i 2 \ > > -F /mnt/ceph/F1 /mnt/ceph/F2 /mnt/ceph/F3 /mnt/ceph/F4 /mnt/ceph/F5 /mnt/ceph/F6 /mnt/ceph/F7 /mnt/ceph/F8 /mnt/ceph/F9 /mnt/ceph/F10 > > done > > > > > > > > > > Jeff > > > > > > > > > > > > > > On Tue, 2011-05-17 at 15:34 +0800, AnnyRen wrote: > >> Hi, Jeff: > >> > >> I run "ceph osd tell osd_num bench" with 1 times per osd > >> > >> and use ceph -w to observe every osd performance, > >> > >> osd0: [INF] bench: wrote 1024 MB in blocks of 4096 KB in 10.875844 > >> sec at 96413 KB/sec > >> osd1: [INF] bench: wrote 1024 MB in blocks of 4096 KB in 11.784985 > >> sec at 88975 KB/sec > >> osd2: [INF] bench: wrote 1024 MB in blocks of 4096 KB in 11.161067 > >> sec at 93949 KB/sec > >> osd3: [INF] bench: wrote 1024 MB in blocks of 4096 KB in 10.798796 > >> sec at 97101 KB/sec > >> osd4: [INF] bench: wrote 1024 MB in blocks of 4096 KB in 14.437141 > >> sec at 72630 KB/sec > >> osd5: [INF] bench: wrote 1024 MB in blocks of 4096 KB in 14.451444 > >> sec at 72558 KB/sec > >> osd6: [INF] bench: wrote 1024 MB in blocks of 4096 KB in 11.083872 > >> sec at 94603 KB/sec > >> osd7: [INF] bench: wrote 1024 MB in blocks of 4096 KB in 11.062728 > >> sec at 94784 KB/sec > >> osd8: [INF] bench: wrote 1024 MB in blocks of 4096 KB in 14.137312 > >> sec at 74170 KB/sec > >> osd9: [INF] bench: wrote 1024 MB in blocks of 4096 KB in 13.489992 > >> sec at 77729 KB/sec > >> > >> > >> and I run > >> root@MDS2:/mnt/ceph# rados bench 60 write -p data > >> > >> the result is > >> > >> Total time run: 60.553247 > >> Total writes made: 1689 > >> Write size: 4194304 > >> Bandwidth (MB/sec): 111.571 > >> > >> Average Latency: 0.573209 > >> Max latency: 2.25691 > >> Min latency: 0.218494 > >> > >> > >> > >> 2011/5/17 Jeff Wu <cpwu@tnsoft.com.cn>: > >> > Hi AnnyRen > >> > > >> > Could you run the following commands and give us the test results? > >> > > >> > $ceph osd tell OSD-N bench // OSD-N : osd number : 0,1,2 .... > >> > $ceph -w > >> > > >> > $rados bench 60 write -p data // refer to "rados -h " > >> > > >> > Jeff > >> > > >> > > >> > > >> > On Tue, 2011-05-17 at 11:53 +0800, AnnyRen wrote: > >> >> I'm running iozone on EXT4 with Ceph v0.26. > >> >> But I got the weird result, most write performance exceed 1GB/s, even > >> >> up to 3GB/s > >> >> I think it's not normal to get the performance outpupt. > >> >> > >> >> Command line I used is: /usr/bin/iozone -azcR -f > >> >> /mnt/ceph/test0516.dbf -g 1G -b /exceloutput0516.xls > >> >> Attachment is the output file... > >> >> > >> >> and my environment is composed of 15 physical machines with > >> >> > >> >> 3 MON, 2MDS (1 active, 1 standby), 10 OSD (1 osd daemon (3T) /host) > >> >> EXT4 format > >> >> data replication size: 3 > >> >> > >> >> > >> >> > >> >> "Writer report" > >> >> "4" "8" "16" "32" "64" "128" "256" "512" "1024" > >> >> "2048" "4096" "8192" "16384" > >> >> "64" 983980 1204796 1210227 1143223 1357066 > >> >> "128" 1007629 1269781 1406136 1391557 1436229 1521718 > >> >> "256" 1112909 1430119 1523457 1652404 1514860 1639786 1729594 > >> >> "512" 1150351 1475116 1605228 1723770 1797349 1712772 1783912 1854787 > >> >> "1024" 1213334 1471481 1679160 1828574 1888889 1899750 1885572 > >> >> 1865912 1875690 > >> >> "2048" 1229274 1540849 1708146 1843410 1903457 1980705 1930406 > >> >> 1913634 1906837 1815744 > >> >> "4096" 1213284 1528348 1674646 1762434 1872096 1882352 1881528 > >> >> 1903416 1897949 1835102 1731177 > >> >> "8192" 204560 155186 572387 238548 186597 429036 187327 > >> >> 157205 553771 416512 299810 405842 > >> >> "16384" 699749 559255 687450 541030 828776 555296 742561 > >> >> 525483 604910 452423 564557 670539 970616 > >> >> "32768" 532414 829610 812215 879441 863159 864794 865938 > >> >> 804951 916352 879582 608132 860732 1239475 > >> >> "65536" 994824 1096543 1095791 1317968 1280277 1390267 1259868 > >> >> 1205214 1339111 1346927 1267888 863234 1190221 > >> >> "131072" 1063429 1165115 1102650 1554828 1182128 1185731 > >> >> 1190752 1195792 1277441 1211063 1237567 1226999 1336961 > >> >> "262144" 1280619 1368554 1497837 1633397 1598255 1609212 > >> >> 1607504 1665019 1590515 1548307 1591258 1505267 1625679 > >> >> "524288" 1519583 1767928 1738523 1883151 2011216 1993877 > >> >> 2023543 1867440 2106124 2055064 1906668 1778645 1838988 > >> >> "1048576" 1580851 1887530 2044131 2166133 2236379 2283578 > >> >> 2257454 2296612 2271066 2101274 1905829 1605923 2158238 > >> >> > >> >> > >> >> > >> >> "Reader report" > >> >> "4" "8" "16" "32" "64" "128" "256" "512" "1024" > >> >> "2048" "4096" "8192" "16384" > >> >> "64" 1933893 2801873 3057153 3363612 3958892 > >> >> "128" 2286447 3053774 2727923 3468030 4104338 4557257 > >> >> "256" 2903529 3236056 3245838 3705040 3654598 4496299 5117791 > >> >> "512" 2906696 3437042 3628697 3431550 4871723 4296637 6246213 6395018 > >> >> "1024" 3229770 3483896 4609294 3791442 4614246 5536137 4550690 > >> >> 5048117 4966395 > >> >> "2048" 3554251 4310472 3885431 4096676 6401772 4842658 5080379 > >> >> 5184636 5596757 5735012 > >> >> "4096" 3416292 4691638 4321103 5728903 5475122 5171846 4819300 > >> >> 5258919 6408472 5044289 4079948 > >> >> "8192" 3233004 4615263 4536055 5618186 5414558 5025700 5553712 > >> >> 4926264 5634770 5281396 4659702 3652258 > >> >> "16384" 3141058 3704193 4567654 4395850 4568869 5387732 > >> >> 4436432 5808029 5578420 4675810 3913007 3911225 3961277 > >> >> "32768" 3704273 4598957 4088278 5133719 5896692 5537024 > >> >> 5234412 5398271 4942992 4118662 3729099 3511757 3481511 > >> >> "65536" 4131091 4210184 5341188 4647619 6077765 5852474 > >> >> 5379762 5259330 5488249 5246682 4342682 3549202 3286487 > >> >> "131072" 3582876 5251082 5332216 5269908 5303512 5574440 > >> >> 5635064 5796372 5406363 4958839 4435918 3673443 3647874 > >> >> "262144" 3659283 4551414 5746231 5433824 5876196 6011650 > >> >> 5552000 5629260 5298830 4982226 4628902 4065823 3421924 > >> >> "524288" 3905973 5488778 5219963 6047356 5916811 6180455 > >> >> 5495733 5925628 5637512 5537123 3517132 3550861 3047013 > >> >> "1048576" 3855595 5634571 5410298 6001809 6464713 6299610 > >> >> 5894249 5516031 5800161 5209203 4295840 3724983 3641623 > >> > > >> > > > > > ^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: IOZone Performance is very Strange 2011-05-18 14:08 ` Jeff Wu @ 2011-05-18 6:24 ` AnnyRen 2011-05-18 14:48 ` Jeff Wu 0 siblings, 1 reply; 13+ messages in thread From: AnnyRen @ 2011-05-18 6:24 UTC (permalink / raw) To: cpwu; +Cc: jerry1209@cht.com.tw, ceph-devel@vger.kernel.org root@MDS2:/mnt/ceph# rados bench 30 write -p data Maintaining 16 concurrent writes of 4194304 bytes for at least 30 seconds. sec Cur ops started finished avg MB/s cur MB/s last lat avg lat 0 0 0 0 0 0 - 0 1 16 37 21 83.9856 84 0.993284 0.567195 2 16 64 48 95.9862 108 0.470228 0.562306 3 16 92 76 101.314 112 0.320282 0.573153 4 16 119 103 102.983 108 0.645115 0.575708 5 16 147 131 104.782 112 0.595362 0.57243 6 16 176 160 106.649 116 0.320091 0.566513 7 16 204 188 107.412 112 0.764551 0.568444 8 16 230 214 106.983 104 0.519568 0.570681 9 16 261 245 108.872 124 0.572544 0.570245 10 16 287 271 108.383 104 1.06776 0.569834 11 16 317 301 109.437 120 0.450807 0.569928 12 16 342 326 108.649 100 0.336045 0.565149 13 16 371 355 109.213 116 0.846668 0.570469 14 16 400 384 109.696 116 0.758495 0.570053 15 16 429 413 110.115 116 0.325192 0.569528 16 16 455 439 109.732 104 0.454317 0.568965 17 16 485 469 110.335 120 1.36841 0.570114 18 16 513 497 110.427 112 1.26513 0.56956 19 16 541 525 110.509 112 0.650765 0.57068 min lat: 0.237473 max lat: 1.6513 avg lat: 0.570027 sec Cur ops started finished avg MB/s cur MB/s last lat avg lat 20 16 565 549 109.782 96 0.561031 0.570027 21 16 593 577 109.886 112 0.619525 0.570106 22 16 624 608 110.527 124 0.505926 0.572197 23 16 650 634 110.243 104 0.609987 0.570785 24 16 679 663 110.482 116 0.475714 0.569979 25 16 708 692 110.702 116 0.444257 0.568121 26 16 737 721 110.905 116 0.288582 0.569373 27 16 766 750 111.093 116 1.15793 0.567702 28 16 793 777 110.982 108 0.287232 0.56945 29 16 821 805 111.016 112 0.324029 0.571677 30 16 847 831 110.781 104 0.386532 0.569997 Total time run: 30.413070 Total writes made: 848 Write size: 4194304 Bandwidth (MB/sec): 111.531 Average Latency: 0.573732 Max latency: 1.6513 Min latency: 0.237473 root@MDS2:/mnt/ceph# rados bench 30 seq -p data sec Cur ops started finished avg MB/s cur MB/s last lat avg lat 0 0 0 0 0 0 - 0 1 16 39 23 91.9819 92 0.913858 0.441642 2 16 68 52 103.984 116 0.259619 0.472126 3 16 96 80 106.652 112 0.34457 0.511364 4 16 125 109 108.985 116 0.408573 0.533846 5 16 153 137 109.584 112 0.28345 0.550811 6 16 179 163 108.65 104 0.659187 0.549494 7 16 208 192 109.698 116 0.981796 0.553188 8 16 234 218 108.984 104 0.539604 0.55086 9 16 264 248 110.206 120 0.479113 0.547496 10 16 294 278 111.184 120 1.27273 0.552185 11 16 320 304 110.529 104 1.14992 0.557991 12 16 349 333 110.984 116 0.933907 0.559665 13 16 376 360 110.753 108 0.26945 0.559205 14 16 403 387 110.555 108 1.65993 0.559534 15 16 432 416 110.916 116 0.895407 0.563238 16 16 460 444 110.983 112 0.872572 0.560972 17 16 488 472 111.041 112 0.678441 0.561167 18 16 516 500 111.094 112 0.737577 0.563033 19 16 544 528 111.14 112 0.304587 0.565015 min lat: 0.138238 max lat: 2.30221 avg lat: 0.564114 sec Cur ops started finished avg MB/s cur MB/s last lat avg lat 20 16 573 557 111.381 116 0.413558 0.564114 21 16 602 586 111.601 116 1.0044 0.561393 22 16 627 611 111.072 100 1.33495 0.562511 23 16 658 642 111.633 124 1.28495 0.564387 24 16 685 669 111.482 108 0.255379 0.565283 25 16 714 698 111.662 116 0.979657 0.565571 26 16 742 726 111.674 112 0.296857 0.561789 27 16 770 754 111.685 112 0.205119 0.55878 28 16 796 780 111.41 104 0.44966 0.559254 29 16 826 810 111.706 120 0.432128 0.559298 read got -2 error during benchmark: -5 error 5: Input/output error Do you know What happened to the final output of read performance test~? Thank you, Jeff!! :-) Anny 2011/5/18 Jeff Wu <cpwu@tnsoft.com.cn>: > Hi , > > Normally , read performance should be more than write performance ,about > 20% ~ 30% . > > Could you run the following commands and attach all of the results ? > $rados bench 30 write -p data //write performance > $rados bench 30 seq -p data // read performance // > > > > On Wed, 2011-05-18 at 13:54 +0800, AnnyRen wrote: >> Hi Jeff & Henry, >> >> I took your advice and used the following command to run iozone test >> >> /usr/bin/iozone -z -c -e -a -n 512M -g 2G -i 0 -i 1 -i 2 -f >> /mnt/ceph/file0517.dbf -Rb /excel0517.xls >> >> Attachment is the output result. >> >> Writer Report >> 4 8 16 32 64 128 256 512 1024 2048 4096 8192 16384 >> 524288 97920 101778 97050 103121 103445 102137 101082 101993 98835 103674 102043 99424 102679 >> 1048576 104306 104841 103616 105884 106044 102221 105666 105890 105811 106409 103957 104062 102115 >> 2097152 106896 107989 108428 107500 108453 108444 109584 108618 109413 108102 107529 108357 107545 >> >> Reader Report >> 4 8 16 32 64 128 256 512 1024 2048 4096 8192 16384 >> 524288 3756286 5029629 5049864 5810960 5577953 5554606 5388867 5636832 5709141 5253596 3805560 3640738 3442332 >> 1048576 3709316 4733745 5400857 5655874 5788923 5565162 5865768 5546521 5886345 5167771 4485693 3531487 3473877 >> 2097152 4305329 5103282 5713626 6399318 6180508 6392487 5804140 4621404 5873338 5837715 4181626 3420274 3506328 >> >> >> the file size grows from 512 MB to 2 GB, write performance is about 94 >> MB/s~ 106 MB/s >> and read performance still exceed 3 GB~6 GB, is it too fast ? >> >> Or we used the cache or buffer but we don't know? if yes, how could we >> avoid to use the cache to read ? >> Thanks very much. :-) >> >> Best Regards, >> Anny >> >> >> >> >> 2011/5/18 Jeff Wu <cpwu@tnsoft.com.cn>: >> > >> > Hi AnnyRen, >> > >> > with the test results , run a 1G data at your ceph cluster , >> > should be about 100M/sec. >> > maybe,iozone test results didn't include iozone flush time. >> > >> > Could you list your hardware platform infos ? >> > network:1G,4G,8G,FC ...?,cpu,memory:size ? ,disk:PATA,SATA,SAS,SSD ?? >> > and >> > could you try other iozone commands ,for instance : >> > >> > 1) >> > add "-e" param to include flush(fsync,fflush) in the timing >> > calculations. >> > >> > /usr/bin/iozone -azcR -e -f /mnt/ceph/test0516.dbf \ >> > -g 1G -b /exceloutput0516.xls >> > >> > 2)run a large data which size is your host memory size*2: >> > >> > $./iozone -z -c -e -a -n 512M -g {memory_size}*2M -i 0 -i 1 -i 2 \ >> > -f /mnt/ceph/fio -Rb ./iozone.xls >> > >> > or >> > 2) >> > !/bin/sh >> > >> > for i in 32 64 128 256 >> > do >> > ./iozone -r ${i}k -t 10 -s 4096M -i 0 -i 1 -i 2 \ >> > -F /mnt/ceph/F1 /mnt/ceph/F2 /mnt/ceph/F3 /mnt/ceph/F4 /mnt/ceph/F5 /mnt/ceph/F6 /mnt/ceph/F7 /mnt/ceph/F8 /mnt/ceph/F9 /mnt/ceph/F10 >> > done >> > >> > >> > >> > >> > Jeff >> > >> > >> > >> > >> > >> > >> > On Tue, 2011-05-17 at 15:34 +0800, AnnyRen wrote: >> >> Hi, Jeff: >> >> >> >> I run "ceph osd tell osd_num bench" with 1 times per osd >> >> >> >> and use ceph -w to observe every osd performance, >> >> >> >> osd0: [INF] bench: wrote 1024 MB in blocks of 4096 KB in 10.875844 >> >> sec at 96413 KB/sec >> >> osd1: [INF] bench: wrote 1024 MB in blocks of 4096 KB in 11.784985 >> >> sec at 88975 KB/sec >> >> osd2: [INF] bench: wrote 1024 MB in blocks of 4096 KB in 11.161067 >> >> sec at 93949 KB/sec >> >> osd3: [INF] bench: wrote 1024 MB in blocks of 4096 KB in 10.798796 >> >> sec at 97101 KB/sec >> >> osd4: [INF] bench: wrote 1024 MB in blocks of 4096 KB in 14.437141 >> >> sec at 72630 KB/sec >> >> osd5: [INF] bench: wrote 1024 MB in blocks of 4096 KB in 14.451444 >> >> sec at 72558 KB/sec >> >> osd6: [INF] bench: wrote 1024 MB in blocks of 4096 KB in 11.083872 >> >> sec at 94603 KB/sec >> >> osd7: [INF] bench: wrote 1024 MB in blocks of 4096 KB in 11.062728 >> >> sec at 94784 KB/sec >> >> osd8: [INF] bench: wrote 1024 MB in blocks of 4096 KB in 14.137312 >> >> sec at 74170 KB/sec >> >> osd9: [INF] bench: wrote 1024 MB in blocks of 4096 KB in 13.489992 >> >> sec at 77729 KB/sec >> >> >> >> >> >> and I run >> >> root@MDS2:/mnt/ceph# rados bench 60 write -p data >> >> >> >> the result is >> >> >> >> Total time run: 60.553247 >> >> Total writes made: 1689 >> >> Write size: 4194304 >> >> Bandwidth (MB/sec): 111.571 >> >> >> >> Average Latency: 0.573209 >> >> Max latency: 2.25691 >> >> Min latency: 0.218494 >> >> >> >> >> >> >> >> 2011/5/17 Jeff Wu <cpwu@tnsoft.com.cn>: >> >> > Hi AnnyRen >> >> > >> >> > Could you run the following commands and give us the test results? >> >> > >> >> > $ceph osd tell OSD-N bench // OSD-N : osd number : 0,1,2 .... >> >> > $ceph -w >> >> > >> >> > $rados bench 60 write -p data // refer to "rados -h " >> >> > >> >> > Jeff >> >> > >> >> > >> >> > >> >> > On Tue, 2011-05-17 at 11:53 +0800, AnnyRen wrote: >> >> >> I'm running iozone on EXT4 with Ceph v0.26. >> >> >> But I got the weird result, most write performance exceed 1GB/s, even >> >> >> up to 3GB/s >> >> >> I think it's not normal to get the performance outpupt. >> >> >> >> >> >> Command line I used is: /usr/bin/iozone -azcR -f >> >> >> /mnt/ceph/test0516.dbf -g 1G -b /exceloutput0516.xls >> >> >> Attachment is the output file... >> >> >> >> >> >> and my environment is composed of 15 physical machines with >> >> >> >> >> >> 3 MON, 2MDS (1 active, 1 standby), 10 OSD (1 osd daemon (3T) /host) >> >> >> EXT4 format >> >> >> data replication size: 3 >> >> >> >> >> >> >> >> >> >> >> >> "Writer report" >> >> >> "4" "8" "16" "32" "64" "128" "256" "512" "1024" >> >> >> "2048" "4096" "8192" "16384" >> >> >> "64" 983980 1204796 1210227 1143223 1357066 >> >> >> "128" 1007629 1269781 1406136 1391557 1436229 1521718 >> >> >> "256" 1112909 1430119 1523457 1652404 1514860 1639786 1729594 >> >> >> "512" 1150351 1475116 1605228 1723770 1797349 1712772 1783912 1854787 >> >> >> "1024" 1213334 1471481 1679160 1828574 1888889 1899750 1885572 >> >> >> 1865912 1875690 >> >> >> "2048" 1229274 1540849 1708146 1843410 1903457 1980705 1930406 >> >> >> 1913634 1906837 1815744 >> >> >> "4096" 1213284 1528348 1674646 1762434 1872096 1882352 1881528 >> >> >> 1903416 1897949 1835102 1731177 >> >> >> "8192" 204560 155186 572387 238548 186597 429036 187327 >> >> >> 157205 553771 416512 299810 405842 >> >> >> "16384" 699749 559255 687450 541030 828776 555296 742561 >> >> >> 525483 604910 452423 564557 670539 970616 >> >> >> "32768" 532414 829610 812215 879441 863159 864794 865938 >> >> >> 804951 916352 879582 608132 860732 1239475 >> >> >> "65536" 994824 1096543 1095791 1317968 1280277 1390267 1259868 >> >> >> 1205214 1339111 1346927 1267888 863234 1190221 >> >> >> "131072" 1063429 1165115 1102650 1554828 1182128 1185731 >> >> >> 1190752 1195792 1277441 1211063 1237567 1226999 1336961 >> >> >> "262144" 1280619 1368554 1497837 1633397 1598255 1609212 >> >> >> 1607504 1665019 1590515 1548307 1591258 1505267 1625679 >> >> >> "524288" 1519583 1767928 1738523 1883151 2011216 1993877 >> >> >> 2023543 1867440 2106124 2055064 1906668 1778645 1838988 >> >> >> "1048576" 1580851 1887530 2044131 2166133 2236379 2283578 >> >> >> 2257454 2296612 2271066 2101274 1905829 1605923 2158238 >> >> >> >> >> >> >> >> >> >> >> >> "Reader report" >> >> >> "4" "8" "16" "32" "64" "128" "256" "512" "1024" >> >> >> "2048" "4096" "8192" "16384" >> >> >> "64" 1933893 2801873 3057153 3363612 3958892 >> >> >> "128" 2286447 3053774 2727923 3468030 4104338 4557257 >> >> >> "256" 2903529 3236056 3245838 3705040 3654598 4496299 5117791 >> >> >> "512" 2906696 3437042 3628697 3431550 4871723 4296637 6246213 6395018 >> >> >> "1024" 3229770 3483896 4609294 3791442 4614246 5536137 4550690 >> >> >> 5048117 4966395 >> >> >> "2048" 3554251 4310472 3885431 4096676 6401772 4842658 5080379 >> >> >> 5184636 5596757 5735012 >> >> >> "4096" 3416292 4691638 4321103 5728903 5475122 5171846 4819300 >> >> >> 5258919 6408472 5044289 4079948 >> >> >> "8192" 3233004 4615263 4536055 5618186 5414558 5025700 5553712 >> >> >> 4926264 5634770 5281396 4659702 3652258 >> >> >> "16384" 3141058 3704193 4567654 4395850 4568869 5387732 >> >> >> 4436432 5808029 5578420 4675810 3913007 3911225 3961277 >> >> >> "32768" 3704273 4598957 4088278 5133719 5896692 5537024 >> >> >> 5234412 5398271 4942992 4118662 3729099 3511757 3481511 >> >> >> "65536" 4131091 4210184 5341188 4647619 6077765 5852474 >> >> >> 5379762 5259330 5488249 5246682 4342682 3549202 3286487 >> >> >> "131072" 3582876 5251082 5332216 5269908 5303512 5574440 >> >> >> 5635064 5796372 5406363 4958839 4435918 3673443 3647874 >> >> >> "262144" 3659283 4551414 5746231 5433824 5876196 6011650 >> >> >> 5552000 5629260 5298830 4982226 4628902 4065823 3421924 >> >> >> "524288" 3905973 5488778 5219963 6047356 5916811 6180455 >> >> >> 5495733 5925628 5637512 5537123 3517132 3550861 3047013 >> >> >> "1048576" 3855595 5634571 5410298 6001809 6464713 6299610 >> >> >> 5894249 5516031 5800161 5209203 4295840 3724983 3641623 >> >> > >> >> > >> > >> > > > -- To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html ^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: IOZone Performance is very Strange 2011-05-18 6:24 ` AnnyRen @ 2011-05-18 14:48 ` Jeff Wu 0 siblings, 0 replies; 13+ messages in thread From: Jeff Wu @ 2011-05-18 14:48 UTC (permalink / raw) To: AnnyRen; +Cc: jerry1209@cht.com.tw, ceph-devel@vger.kernel.org On Wed, 2011-05-18 at 14:24 +0800, AnnyRen wrote: > root@MDS2:/mnt/ceph# rados bench 30 write -p data > Maintaining 16 concurrent writes of 4194304 bytes for at least 30 seconds. > sec Cur ops started finished avg MB/s cur MB/s last lat avg lat > 0 0 0 0 0 0 - 0 > 1 16 37 21 83.9856 84 0.993284 0.567195 > 2 16 64 48 95.9862 108 0.470228 0.562306 > 3 16 92 76 101.314 112 0.320282 0.573153 > 4 16 119 103 102.983 108 0.645115 0.575708 > 5 16 147 131 104.782 112 0.595362 0.57243 > 6 16 176 160 106.649 116 0.320091 0.566513 > 7 16 204 188 107.412 112 0.764551 0.568444 > 8 16 230 214 106.983 104 0.519568 0.570681 > 9 16 261 245 108.872 124 0.572544 0.570245 > 10 16 287 271 108.383 104 1.06776 0.569834 > 11 16 317 301 109.437 120 0.450807 0.569928 > 12 16 342 326 108.649 100 0.336045 0.565149 > 13 16 371 355 109.213 116 0.846668 0.570469 > 14 16 400 384 109.696 116 0.758495 0.570053 > 15 16 429 413 110.115 116 0.325192 0.569528 > 16 16 455 439 109.732 104 0.454317 0.568965 > 17 16 485 469 110.335 120 1.36841 0.570114 > 18 16 513 497 110.427 112 1.26513 0.56956 > 19 16 541 525 110.509 112 0.650765 0.57068 > min lat: 0.237473 max lat: 1.6513 avg lat: 0.570027 > sec Cur ops started finished avg MB/s cur MB/s last lat avg lat > 20 16 565 549 109.782 96 0.561031 0.570027 > 21 16 593 577 109.886 112 0.619525 0.570106 > 22 16 624 608 110.527 124 0.505926 0.572197 > 23 16 650 634 110.243 104 0.609987 0.570785 > 24 16 679 663 110.482 116 0.475714 0.569979 > 25 16 708 692 110.702 116 0.444257 0.568121 > 26 16 737 721 110.905 116 0.288582 0.569373 > 27 16 766 750 111.093 116 1.15793 0.567702 > 28 16 793 777 110.982 108 0.287232 0.56945 > 29 16 821 805 111.016 112 0.324029 0.571677 > 30 16 847 831 110.781 104 0.386532 0.569997 > Total time run: 30.413070 > Total writes made: 848 > Write size: 4194304 > Bandwidth (MB/sec): 111.531 > > Average Latency: 0.573732 > Max latency: 1.6513 > Min latency: 0.237473 > > > > root@MDS2:/mnt/ceph# rados bench 30 seq -p data > sec Cur ops started finished avg MB/s cur MB/s last lat avg lat > 0 0 0 0 0 0 - 0 > 1 16 39 23 91.9819 92 0.913858 0.441642 > 2 16 68 52 103.984 116 0.259619 0.472126 > 3 16 96 80 106.652 112 0.34457 0.511364 > 4 16 125 109 108.985 116 0.408573 0.533846 > 5 16 153 137 109.584 112 0.28345 0.550811 > 6 16 179 163 108.65 104 0.659187 0.549494 > 7 16 208 192 109.698 116 0.981796 0.553188 > 8 16 234 218 108.984 104 0.539604 0.55086 > 9 16 264 248 110.206 120 0.479113 0.547496 > 10 16 294 278 111.184 120 1.27273 0.552185 > 11 16 320 304 110.529 104 1.14992 0.557991 > 12 16 349 333 110.984 116 0.933907 0.559665 > 13 16 376 360 110.753 108 0.26945 0.559205 > 14 16 403 387 110.555 108 1.65993 0.559534 > 15 16 432 416 110.916 116 0.895407 0.563238 > 16 16 460 444 110.983 112 0.872572 0.560972 > 17 16 488 472 111.041 112 0.678441 0.561167 > 18 16 516 500 111.094 112 0.737577 0.563033 > 19 16 544 528 111.14 112 0.304587 0.565015 > min lat: 0.138238 max lat: 2.30221 avg lat: 0.564114 > sec Cur ops started finished avg MB/s cur MB/s last lat avg lat > 20 16 573 557 111.381 116 0.413558 0.564114 > 21 16 602 586 111.601 116 1.0044 0.561393 > 22 16 627 611 111.072 100 1.33495 0.562511 > 23 16 658 642 111.633 124 1.28495 0.564387 > 24 16 685 669 111.482 108 0.255379 0.565283 > 25 16 714 698 111.662 116 0.979657 0.565571 > 26 16 742 726 111.674 112 0.296857 0.561789 > 27 16 770 754 111.685 112 0.205119 0.55878 > 28 16 796 780 111.41 104 0.44966 0.559254 > 29 16 826 810 111.706 120 0.432128 0.559298 > read got -2 > error during benchmark: -5 > error 5: Input/output error > ~~ should be a bug ,i also run into this issue. filed it at http://tracker.newdream.net/issues/1095 with the log, your cluster platform read performance should be about 110M /sec. i'm not sure that your iozone read performance what's wrong.maybe , read data from cache. > > Do you know What happened to the final output of read performance test~? > > Thank you, Jeff!! :-) > > > Anny > > > > > > > 2011/5/18 Jeff Wu <cpwu@tnsoft.com.cn>: > > Hi , > > > > Normally , read performance should be more than write performance ,about > > 20% ~ 30% . > > > > Could you run the following commands and attach all of the results ? > > $rados bench 30 write -p data //write performance > > $rados bench 30 seq -p data // read performance // > > > > > > > > On Wed, 2011-05-18 at 13:54 +0800, AnnyRen wrote: > >> Hi Jeff & Henry, > >> > >> I took your advice and used the following command to run iozone test > >> > >> /usr/bin/iozone -z -c -e -a -n 512M -g 2G -i 0 -i 1 -i 2 -f > >> /mnt/ceph/file0517.dbf -Rb /excel0517.xls > >> > >> Attachment is the output result. > >> > >> Writer Report > >> 4 8 16 32 64 128 256 512 1024 2048 4096 8192 16384 > >> 524288 97920 101778 97050 103121 103445 102137 101082 101993 98835 103674 102043 99424 102679 > >> 1048576 104306 104841 103616 105884 106044 102221 105666 105890 105811 106409 103957 104062 102115 > >> 2097152 106896 107989 108428 107500 108453 108444 109584 108618 109413 108102 107529 108357 107545 > >> > >> Reader Report > >> 4 8 16 32 64 128 256 512 1024 2048 4096 8192 16384 > >> 524288 3756286 5029629 5049864 5810960 5577953 5554606 5388867 5636832 5709141 5253596 3805560 3640738 3442332 > >> 1048576 3709316 4733745 5400857 5655874 5788923 5565162 5865768 5546521 5886345 5167771 4485693 3531487 3473877 > >> 2097152 4305329 5103282 5713626 6399318 6180508 6392487 5804140 4621404 5873338 5837715 4181626 3420274 3506328 > >> > >> > >> the file size grows from 512 MB to 2 GB, write performance is about 94 > >> MB/s~ 106 MB/s > >> and read performance still exceed 3 GB~6 GB, is it too fast ? > >> > >> Or we used the cache or buffer but we don't know? if yes, how could we > >> avoid to use the cache to read ? > >> Thanks very much. :-) > >> > >> Best Regards, > >> Anny > >> > >> > >> > >> > >> 2011/5/18 Jeff Wu <cpwu@tnsoft.com.cn>: > >> > > >> > Hi AnnyRen, > >> > > >> > with the test results , run a 1G data at your ceph cluster , > >> > should be about 100M/sec. > >> > maybe,iozone test results didn't include iozone flush time. > >> > > >> > Could you list your hardware platform infos ? > >> > network:1G,4G,8G,FC ...?,cpu,memory:size ? ,disk:PATA,SATA,SAS,SSD ?? > >> > and > >> > could you try other iozone commands ,for instance : > >> > > >> > 1) > >> > add "-e" param to include flush(fsync,fflush) in the timing > >> > calculations. > >> > > >> > /usr/bin/iozone -azcR -e -f /mnt/ceph/test0516.dbf \ > >> > -g 1G -b /exceloutput0516.xls > >> > > >> > 2)run a large data which size is your host memory size*2: > >> > > >> > $./iozone -z -c -e -a -n 512M -g {memory_size}*2M -i 0 -i 1 -i 2 \ > >> > -f /mnt/ceph/fio -Rb ./iozone.xls > >> > > >> > or > >> > 2) > >> > !/bin/sh > >> > > >> > for i in 32 64 128 256 > >> > do > >> > ./iozone -r ${i}k -t 10 -s 4096M -i 0 -i 1 -i 2 \ > >> > -F /mnt/ceph/F1 /mnt/ceph/F2 /mnt/ceph/F3 /mnt/ceph/F4 /mnt/ceph/F5 /mnt/ceph/F6 /mnt/ceph/F7 /mnt/ceph/F8 /mnt/ceph/F9 /mnt/ceph/F10 > >> > done > >> > > >> > > >> > > >> > > >> > Jeff > >> > > >> > > >> > > >> > > >> > > >> > > >> > On Tue, 2011-05-17 at 15:34 +0800, AnnyRen wrote: > >> >> Hi, Jeff: > >> >> > >> >> I run "ceph osd tell osd_num bench" with 1 times per osd > >> >> > >> >> and use ceph -w to observe every osd performance, > >> >> > >> >> osd0: [INF] bench: wrote 1024 MB in blocks of 4096 KB in 10.875844 > >> >> sec at 96413 KB/sec > >> >> osd1: [INF] bench: wrote 1024 MB in blocks of 4096 KB in 11.784985 > >> >> sec at 88975 KB/sec > >> >> osd2: [INF] bench: wrote 1024 MB in blocks of 4096 KB in 11.161067 > >> >> sec at 93949 KB/sec > >> >> osd3: [INF] bench: wrote 1024 MB in blocks of 4096 KB in 10.798796 > >> >> sec at 97101 KB/sec > >> >> osd4: [INF] bench: wrote 1024 MB in blocks of 4096 KB in 14.437141 > >> >> sec at 72630 KB/sec > >> >> osd5: [INF] bench: wrote 1024 MB in blocks of 4096 KB in 14.451444 > >> >> sec at 72558 KB/sec > >> >> osd6: [INF] bench: wrote 1024 MB in blocks of 4096 KB in 11.083872 > >> >> sec at 94603 KB/sec > >> >> osd7: [INF] bench: wrote 1024 MB in blocks of 4096 KB in 11.062728 > >> >> sec at 94784 KB/sec > >> >> osd8: [INF] bench: wrote 1024 MB in blocks of 4096 KB in 14.137312 > >> >> sec at 74170 KB/sec > >> >> osd9: [INF] bench: wrote 1024 MB in blocks of 4096 KB in 13.489992 > >> >> sec at 77729 KB/sec > >> >> > >> >> > >> >> and I run > >> >> root@MDS2:/mnt/ceph# rados bench 60 write -p data > >> >> > >> >> the result is > >> >> > >> >> Total time run: 60.553247 > >> >> Total writes made: 1689 > >> >> Write size: 4194304 > >> >> Bandwidth (MB/sec): 111.571 > >> >> > >> >> Average Latency: 0.573209 > >> >> Max latency: 2.25691 > >> >> Min latency: 0.218494 > >> >> > >> >> > >> >> > >> >> 2011/5/17 Jeff Wu <cpwu@tnsoft.com.cn>: > >> >> > Hi AnnyRen > >> >> > > >> >> > Could you run the following commands and give us the test results? > >> >> > > >> >> > $ceph osd tell OSD-N bench // OSD-N : osd number : 0,1,2 .... > >> >> > $ceph -w > >> >> > > >> >> > $rados bench 60 write -p data // refer to "rados -h " > >> >> > > >> >> > Jeff > >> >> > > >> >> > > >> >> > > >> >> > On Tue, 2011-05-17 at 11:53 +0800, AnnyRen wrote: > >> >> >> I'm running iozone on EXT4 with Ceph v0.26. > >> >> >> But I got the weird result, most write performance exceed 1GB/s, even > >> >> >> up to 3GB/s > >> >> >> I think it's not normal to get the performance outpupt. > >> >> >> > >> >> >> Command line I used is: /usr/bin/iozone -azcR -f > >> >> >> /mnt/ceph/test0516.dbf -g 1G -b /exceloutput0516.xls > >> >> >> Attachment is the output file... > >> >> >> > >> >> >> and my environment is composed of 15 physical machines with > >> >> >> > >> >> >> 3 MON, 2MDS (1 active, 1 standby), 10 OSD (1 osd daemon (3T) /host) > >> >> >> EXT4 format > >> >> >> data replication size: 3 > >> >> >> > >> >> >> > >> >> >> > >> >> >> "Writer report" > >> >> >> "4" "8" "16" "32" "64" "128" "256" "512" "1024" > >> >> >> "2048" "4096" "8192" "16384" > >> >> >> "64" 983980 1204796 1210227 1143223 1357066 > >> >> >> "128" 1007629 1269781 1406136 1391557 1436229 1521718 > >> >> >> "256" 1112909 1430119 1523457 1652404 1514860 1639786 1729594 > >> >> >> "512" 1150351 1475116 1605228 1723770 1797349 1712772 1783912 1854787 > >> >> >> "1024" 1213334 1471481 1679160 1828574 1888889 1899750 1885572 > >> >> >> 1865912 1875690 > >> >> >> "2048" 1229274 1540849 1708146 1843410 1903457 1980705 1930406 > >> >> >> 1913634 1906837 1815744 > >> >> >> "4096" 1213284 1528348 1674646 1762434 1872096 1882352 1881528 > >> >> >> 1903416 1897949 1835102 1731177 > >> >> >> "8192" 204560 155186 572387 238548 186597 429036 187327 > >> >> >> 157205 553771 416512 299810 405842 > >> >> >> "16384" 699749 559255 687450 541030 828776 555296 742561 > >> >> >> 525483 604910 452423 564557 670539 970616 > >> >> >> "32768" 532414 829610 812215 879441 863159 864794 865938 > >> >> >> 804951 916352 879582 608132 860732 1239475 > >> >> >> "65536" 994824 1096543 1095791 1317968 1280277 1390267 1259868 > >> >> >> 1205214 1339111 1346927 1267888 863234 1190221 > >> >> >> "131072" 1063429 1165115 1102650 1554828 1182128 1185731 > >> >> >> 1190752 1195792 1277441 1211063 1237567 1226999 1336961 > >> >> >> "262144" 1280619 1368554 1497837 1633397 1598255 1609212 > >> >> >> 1607504 1665019 1590515 1548307 1591258 1505267 1625679 > >> >> >> "524288" 1519583 1767928 1738523 1883151 2011216 1993877 > >> >> >> 2023543 1867440 2106124 2055064 1906668 1778645 1838988 > >> >> >> "1048576" 1580851 1887530 2044131 2166133 2236379 2283578 > >> >> >> 2257454 2296612 2271066 2101274 1905829 1605923 2158238 > >> >> >> > >> >> >> > >> >> >> > >> >> >> "Reader report" > >> >> >> "4" "8" "16" "32" "64" "128" "256" "512" "1024" > >> >> >> "2048" "4096" "8192" "16384" > >> >> >> "64" 1933893 2801873 3057153 3363612 3958892 > >> >> >> "128" 2286447 3053774 2727923 3468030 4104338 4557257 > >> >> >> "256" 2903529 3236056 3245838 3705040 3654598 4496299 5117791 > >> >> >> "512" 2906696 3437042 3628697 3431550 4871723 4296637 6246213 6395018 > >> >> >> "1024" 3229770 3483896 4609294 3791442 4614246 5536137 4550690 > >> >> >> 5048117 4966395 > >> >> >> "2048" 3554251 4310472 3885431 4096676 6401772 4842658 5080379 > >> >> >> 5184636 5596757 5735012 > >> >> >> "4096" 3416292 4691638 4321103 5728903 5475122 5171846 4819300 > >> >> >> 5258919 6408472 5044289 4079948 > >> >> >> "8192" 3233004 4615263 4536055 5618186 5414558 5025700 5553712 > >> >> >> 4926264 5634770 5281396 4659702 3652258 > >> >> >> "16384" 3141058 3704193 4567654 4395850 4568869 5387732 > >> >> >> 4436432 5808029 5578420 4675810 3913007 3911225 3961277 > >> >> >> "32768" 3704273 4598957 4088278 5133719 5896692 5537024 > >> >> >> 5234412 5398271 4942992 4118662 3729099 3511757 3481511 > >> >> >> "65536" 4131091 4210184 5341188 4647619 6077765 5852474 > >> >> >> 5379762 5259330 5488249 5246682 4342682 3549202 3286487 > >> >> >> "131072" 3582876 5251082 5332216 5269908 5303512 5574440 > >> >> >> 5635064 5796372 5406363 4958839 4435918 3673443 3647874 > >> >> >> "262144" 3659283 4551414 5746231 5433824 5876196 6011650 > >> >> >> 5552000 5629260 5298830 4982226 4628902 4065823 3421924 > >> >> >> "524288" 3905973 5488778 5219963 6047356 5916811 6180455 > >> >> >> 5495733 5925628 5637512 5537123 3517132 3550861 3047013 > >> >> >> "1048576" 3855595 5634571 5410298 6001809 6464713 6299610 > >> >> >> 5894249 5516031 5800161 5209203 4295840 3724983 3641623 > >> >> > > >> >> > > >> > > >> > > > > > ^ permalink raw reply [flat|nested] 13+ messages in thread
end of thread, other threads:[~2011-05-18 6:48 UTC | newest] Thread overview: 13+ messages (download: mbox.gz follow: Atom feed -- links below jump to the message on this page -- 2011-05-17 3:53 IOZone Performance is very Strange AnnyRen 2011-05-17 13:23 ` Jeff Wu 2011-05-17 7:34 ` AnnyRen 2011-05-17 8:19 ` Simon Tian 2011-05-17 9:07 ` AnnyRen 2011-05-17 17:05 ` Jeff Wu 2011-05-17 9:14 ` Henry C Chang 2011-05-17 9:44 ` AnnyRen 2011-05-17 9:37 ` AnnyRen 2011-05-18 5:54 ` AnnyRen 2011-05-18 14:08 ` Jeff Wu 2011-05-18 6:24 ` AnnyRen 2011-05-18 14:48 ` Jeff Wu
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.