From mboxrd@z Thu Jan 1 00:00:00 1970 From: "hzwulibin-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org" Subject: Re: [performance] why rbd_aio_write latency increase from 4ms to 7.3ms after the same test Date: Tue, 3 Nov 2015 08:43:47 +0800 Message-ID: <201511030843428328313@gmail.com> References: <5636F376.3090104@gmail.com>, <6F3FA899187F0043BA1827A69DA2F7CC03639832@shsmsx102.ccr.corp.intel.com> Mime-Version: 1.0 Content-Type: multipart/mixed; boundary="===============1794768861==" Return-path: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: ceph-users-bounces-idqoXFIVOFJgJs9I8MT0rw@public.gmane.org Sender: "ceph-users" To: =?UTF-8?B?Q2hlbiwgWGlhb3hp?= , "ceph-devel-u79uwXL29TY76Z2rM5mHXA@public.gmane.org" , "ceph-users-idqoXFIVOFJgJs9I8MT0rw@public.gmane.org" List-Id: ceph-devel.vger.kernel.org This is a multi-part message in MIME format. --===============1794768861== Content-Type: multipart/alternative; boundary="----=_001_NextPart741106821217_=----" This is a multi-part message in MIME format. ------=_001_NextPart741106821217_=---- Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: base64 SGksDQoNClRoYW5rIHlvdSwgdGhhdCBtYWtlIHNlbnNlIGZvciB0ZXN0aW5nLCBidXQgaSdtIGFm cmFpZCBub3QgaW4gbXkgY2FzZS4NCkV2ZW4gaSB0ZXN0IG9uIHRoZSB2b2x1bWUgdGhhdCBhbHJl YWR5IHRlc3QgbWFueSB0aW1lcywgdGhlIElPUFMgd2lsbCBub3QgZ3Jvd2luZyB1cCANCmFnYWlu LiBZZWFoLCBpIG1lYW4sIHRoaXMgVk0gaXMgYnJva2VuLCBJT1BTIG9mIHRoZSBWTSB3aWxsIG5l dmVyIGdyb3dpbmcgdXAuLg0KDQpUaGFua3MhDQoNCg0KDQpoend1bGliaW5AZ21haWwuY29tDQog DQpGcm9tOiBDaGVuLCBYaWFveGkNCkRhdGU6IDIwMTUtMTEtMDIgMTQ6MTENClRvOiBoend1bGli aW47IGNlcGgtZGV2ZWw7IGNlcGgtdXNlcnMNClN1YmplY3Q6IFJFOiBbcGVyZm9ybWFuY2VdIHdo eSByYmRfYWlvX3dyaXRlIGxhdGVuY3kgaW5jcmVhc2UgZnJvbSA0bXMgdG8gNy4zbXMgYWZ0ZXIg dGhlIHNhbWUgdGVzdA0KUHJlLWFsbG9jYXRlZCB0aGUgdm9sdW1lIGJ5ICJERCIgYWNyb3NzIHRo ZSBlbnRpcmUgUkJEIGJlZm9yZSB5b3UgZG8gYW55IHBlcmZvcm1hbmNlIHRlc3Q6KS4NCiANCklu IHRoaXMgY2FzZSwgeW91IG1heSB3YW50IHRvIHJlLWNyZWF0ZSB0aGUgUkJELCBwcmUtYWxsb2Nh dGUgYW5kIHRyeSBhZ2Fpbi4NCiANCj4gLS0tLS1PcmlnaW5hbCBNZXNzYWdlLS0tLS0NCj4gRnJv bTogY2VwaC1kZXZlbC1vd25lckB2Z2VyLmtlcm5lbC5vcmcgW21haWx0bzpjZXBoLWRldmVsLQ0K PiBvd25lckB2Z2VyLmtlcm5lbC5vcmddIE9uIEJlaGFsZiBPZiBoend1bGliaW4NCj4gU2VudDog TW9uZGF5LCBOb3ZlbWJlciAyLCAyMDE1IDE6MjQgUE0NCj4gVG86IGNlcGgtZGV2ZWw7IGNlcGgt dXNlcnMNCj4gU3ViamVjdDogW3BlcmZvcm1hbmNlXSB3aHkgcmJkX2Fpb193cml0ZSBsYXRlbmN5 IGluY3JlYXNlIGZyb20gNG1zIHRvDQo+IDcuM21zIGFmdGVyIHRoZSBzYW1lIHRlc3QNCj4gDQo+ IEhpLA0KPiBzYW1lIGVudmlyb25tZW50LCBhZnRlciBhIHRlc3Qgc2NyaXB0LCB0aGUgaW8gbGF0 ZW5jeShnZXQgZnJvbSBzdWRvIGNlcGggLS0NCj4gYWRtaW4tZGFlbW9uIC9ydW4vY2VwaC9ndWVz dHMvY2VwaC1jbGllbnQuKi5hc29rIHBlciBkdW1wKSBpbmNyZWFzZQ0KPiBmcm9tIGFib3V0IDRt cyB0byA3LjNtcw0KPiANCj4gcWVtdSB2ZXJzaW9uOiBkZWJpYW4gMi4xLjINCj4ga2VybmVsOjMu MTAuNDUtb3BlbnN0YWNrLWFtZDY0DQo+IHN5c3RlbTogZGViaWFuIDcuOA0KPiBjZXBoOiAwLjk0 LjUNCj4gVk0gQ1BVIG51bWJlcjogNCAgKGNwdSBNSHogOiAyNTk5Ljk5OCkNCj4gVk0gbWVtb3J5 IHNpemU6IDE2R0INCj4gOSBPU0Qgc3RvcmFnZSBzZXJ2ZXJzLCB3aXRoIDQgU1NEIE9TRCBvbiBl YWNoLCB0b3RhbCAzNiBPU0RzLg0KPiANCj4gVGVzdCBzY3JpcHRzIGluIFZNOg0KPiAjIGNhdCBy ZXByb2R1Y2Uuc2gNCj4gIyEvYmluL2Jhc2gNCj4gDQo+IHRpbWVzPTIwDQo+IGZvcigoaT0xO2k8 PSR0aW1lcztpKyspKQ0KPiBkbw0KPiAgICAgdG1wZGF0ZT1gZGF0ZSAiKyVGLSVUImANCj4gICAg IGVjaG8NCj4gIj09PT09PT09PT09PT09PT09PT09PT09JHRtcGRhdGUoJGkvJHRpbWVzKT09PT09 PT09PT09PT09PT09PT0NCj4gPT09PSINCj4gICAgIHRtcD0kKChpJTIpKQ0KPiAgICAgaWYgW1sg JHRtcCAtZXEgMCBdXTt0aGVuDQo+ICAgICAgICAgZWNobyAiIyMjIyMjIyMjIyMjIyMjIGZpbyAv cm9vdC92ZGIuY2ZnICMjIyMjIyMjIyMjIyMjIyINCj4gICAgICAgICBmaW8gL3Jvb3QvdmRiLmNm Zw0KPiAgICAgZWxzZQ0KPiAgICAgICAgIGVjaG8gIiMjIyMjIyMjIyMjIyMjIyBmaW8gL3Jvb3Qv dmRjLmNmZyAjIyMjIyMjIyMjIyMjIyMiDQo+ICAgICAgICAgZmlvIC9yb290L3ZkYy5jZmcNCj4g ICAgIGZpDQo+IGRvbmUNCj4gDQo+IA0KPiB0bXBkYXRlPWBkYXRlICIrJUYtJVQiYA0KPiBlY2hv ICIjIyMjIyMjIyMjIyMjIyMgWyR0bXBkYXRlXSBmaW8gL3Jvb3QvdmRlLmNmZyAjIyMjIyMjIyMj IyMjIyMiDQo+IGZpbyAvcm9vdC92ZGUuY2ZnDQo+IA0KPiANCj4gIyBjYXQgdmRiLmNmZw0KPiBb Z2xvYmFsXQ0KPiBydz1yYW5kd3JpdGUNCj4gZGlyZWN0PTENCj4gbnVtam9icz02NA0KPiBpb2Vu Z2luZT1zeW5jDQo+IGJzcmFuZ2U9NGstNGsNCj4gcnVudGltZT0xODANCj4gZ3JvdXBfcmVwb3J0 aW5nDQo+IA0KPiBbZGlzazAxXQ0KPiBmaWxlbmFtZT0vZGV2L3ZkYg0KPiANCj4gDQo+ICMgY2F0 IHZkYy5jZmcNCj4gW2dsb2JhbF0NCj4gcnc9cmFuZHdyaXRlDQo+IGRpcmVjdD0xDQo+IG51bWpv YnM9NjQNCj4gaW9lbmdpbmU9c3luYw0KPiBic3JhbmdlPTRrLTRrDQo+IHJ1bnRpbWU9MTgwDQo+ IGdyb3VwX3JlcG9ydGluZw0KPiANCj4gW2Rpc2swMV0NCj4gZmlsZW5hbWU9L2Rldi92ZGMNCj4g DQo+ICMgY2F0IHZkZC5jZmcNCj4gW2dsb2JhbF0NCj4gcnc9cmFuZHdyaXRlDQo+IGRpcmVjdD0x DQo+IG51bWpvYnM9NjQNCj4gaW9lbmdpbmU9c3luYw0KPiBic3JhbmdlPTRrLTRrDQo+IHJ1bnRp bWU9MTgwDQo+IGdyb3VwX3JlcG9ydGluZw0KPiANCj4gW2Rpc2swMV0NCj4gZmlsZW5hbWU9L2Rl di92ZGQNCj4gDQo+ICMgY2F0IHZkZS5jZmcNCj4gW2dsb2JhbF0NCj4gcnc9cmFuZHdyaXRlDQo+ IGRpcmVjdD0xDQo+IG51bWpvYnM9NjQNCj4gaW9lbmdpbmU9c3luYw0KPiBic3JhbmdlPTRrLTRr DQo+IHJ1bnRpbWU9MTgwDQo+IGdyb3VwX3JlcG9ydGluZw0KPiANCj4gW2Rpc2swMV0NCj4gZmls ZW5hbWU9L2Rldi92ZGUNCj4gDQo+IEFmdGVyIHJ1biB0aGUgc2NyaXB0cyByZXByb2R1Y2Uuc2gs IHRoZSBkaXNrcyBpbiB0aGUgVk0ncyBJT1BTIGN1dGRvd24gZnJvbQ0KPiAxMmsgdG8gNWssIHRo ZSBsYXRlbmN5IGluY3JlYXNlIGZyb20gNG1zIHRvIDcuM21zLg0KPiANCj4gcnVuIHN0ZXBzOg0K PiAxLiBjcmVhdGUgYSBWTQ0KPiAyLiBjcmVhdGUgZm91ciB2b2x1bWVzIGFuZCBhdHRhdGNoIHRv IHRoZSBWTSAzLiBzaCByZXByb2R1Y2Uuc2ggNC4gaW4gdGhlDQo+IHJ1bnRpbWUgb2YgIHJlcHJv ZHVjZS5zaCwgcnVuICJmaW8gdmRkLmNmZyIgb3IgImZpbyB2ZGUuY2ZnIiB0byBjaGVja3QgdGhl DQo+IHBlcmZvcm1hbmNlDQo+IA0KPiBBZnRlciByZXByb2R1Y2Uuc2ggZmluaXNoZWQsIHBlcmZv cm1hbmNlIGRvd24uDQo+IA0KPiANCj4gQW55b25lIGhhcyB0aGUgc2FtZSBwcm9ibGVtIG9yIGhh cyBzb21lIGlkZWFzIGFib3V0IHRoaXM/DQo+IA0KPiBUaGFua3MhDQo+IC0tLS0tLS0tLS0tLS0t DQo+IGh6d3VsaWJpbg0KPiAyMDE1LTExLTAyDQo+IHsubiArICAgICAgICslICBsendtICBiIOun siAgciAgecepINevelggICAg3Kh9ICAgxqB6ICZqOit2ICAgICAgICB6WisgICt6ZiAgIGggICB+ ICAgIGkgICB6IHcgICA/DQo+ICYgKd+iIGYNCg== ------=_001_NextPart741106821217_=---- Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable =0A
Hi,

Thank you, that make sense for testing, but i'm afraid not in my ca= se.
Even i test on the volume that already test many times, the = IOPS will not growing up 
again. Yeah, i mean, this VM is b= roken, IOPS of the VM will never growing up..

Tha= nks!
=0A

=0A
hzwulibin-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org
=0A
 
Subject: RE: [perf= ormance] why rbd_aio_write latency increase from 4ms to 7.3ms after the sa= me test
Pre-allocated the volume by "DD" across= the entire RBD before you do any performance test:).
=0A
 <= /div>=0A
In this case, you may want to re-create the RBD, pre-allocate= and try again.
=0A
 
=0A
> -----Original Messag= e-----
=0A
> From: ceph-devel-owner-u79uwXL29TY76Z2rM5mHXA@public.gmane.org [mailto:ce= ph-devel-
=0A
> owner-u79uwXL29TY76Z2rM5mHXA@public.gmane.org] On Behalf Of hzwulibin<= /div>=0A
> Sent: Monday, November 2, 2015 1:24 PM
=0A
>= To: ceph-devel; ceph-users
=0A
> Subject: [performance] why r= bd_aio_write latency increase from 4ms to
=0A
> 7.3ms after th= e same test
=0A
>
=0A
> Hi,
=0A
> sam= e environment, after a test script, the io latency(get from sudo ceph --=0A
> admin-daemon /run/ceph/guests/ceph-client.*.asok per dump= ) increase
=0A
> from about 4ms to 7.3ms
=0A
> =0A
> qemu version: debian 2.1.2
=0A
> kernel:3.10.4= 5-openstack-amd64
=0A
> system: debian 7.8
=0A
> c= eph: 0.94.5
=0A
> VM CPU number: 4  (cpu MHz : 2599.998)<= /div>=0A
> VM memory size: 16GB
=0A
> 9 OSD storage ser= vers, with 4 SSD OSD on each, total 36 OSDs.
=0A
>
=0A> Test scripts in VM:
=0A
> # cat reproduce.sh
=0A<= div>> #!/bin/bash
=0A
>
=0A
> times=3D20
= =0A
> for((i=3D1;i<=3D$times;i++))
=0A
> do
=0A=
>     tmpdate=3D`date "+%F-%T"`
=0A
&= gt;     echo
=0A
> "=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D$tmpdate($i/$times)=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D
=0A
> =3D= =3D=3D=3D"
=0A
>     tmp=3D$((i%2))
= =0A
>     if [[ $tmp -eq 0 ]];then
=0A>         echo "##############= # fio /root/vdb.cfg ###############"
=0A
>   &n= bsp;     fio /root/vdb.cfg
=0A
> &nbs= p;   else
=0A
>      &= nbsp;  echo "############### fio /root/vdc.cfg ###############"
= =0A
>         fio /root/vdc= .cfg
=0A
>     fi
=0A
> done=0A
>
=0A
>
=0A
> tmpdate=3D`date "+%= F-%T"`
=0A
> echo "############### [$tmpdate] fio /root/vde.cf= g ###############"
=0A
> fio /root/vde.cfg
=0A
> <= /div>=0A
>
=0A
> # cat vdb.cfg
=0A
> [glob= al]
=0A
> rw=3Drandwrite
=0A
> direct=3D1
=0A=
> numjobs=3D64
=0A
> ioengine=3Dsync
=0A
>= bsrange=3D4k-4k
=0A
> runtime=3D180
=0A
> group_r= eporting
=0A
>
=0A
> [disk01]
=0A
> f= ilename=3D/dev/vdb
=0A
>
=0A
>
=0A
>= # cat vdc.cfg
=0A
> [global]
=0A
> rw=3Drandwrite=
=0A
> direct=3D1
=0A
> numjobs=3D64
=0A
= > ioengine=3Dsync
=0A
> bsrange=3D4k-4k
=0A
> r= untime=3D180
=0A
> group_reporting
=0A
>
=0A=
> [disk01]
=0A
> filename=3D/dev/vdc
=0A
>=
=0A
> # cat vdd.cfg
=0A
> [global]
=0A
= > rw=3Drandwrite
=0A
> direct=3D1
=0A
> numjobs= =3D64
=0A
> ioengine=3Dsync
=0A
> bsrange=3D4k-4k<= /div>=0A
> runtime=3D180
=0A
> group_reporting
=0A=
>
=0A
> [disk01]
=0A
> filename=3D/dev/vd= d
=0A
>
=0A
> # cat vde.cfg
=0A
> [gl= obal]
=0A
> rw=3Drandwrite
=0A
> direct=3D1
= =0A
> numjobs=3D64
=0A
> ioengine=3Dsync
=0A
&= gt; bsrange=3D4k-4k
=0A
> runtime=3D180
=0A
> grou= p_reporting
=0A
>
=0A
> [disk01]
=0A
>= ; filename=3D/dev/vde
=0A
>
=0A
> After run the s= cripts reproduce.sh, the disks in the VM's IOPS cutdown from
=0A
= > 12k to 5k, the latency increase from 4ms to 7.3ms.
=0A
> =
=0A
> run steps:
=0A
> 1. create a VM
=0A> 2. create four volumes and attatch to the VM 3. sh reproduce.sh 4. = in the
=0A
> runtime of  reproduce.sh, run "fio vdd.cfg" = or "fio vde.cfg" to checkt the
=0A
> performance
=0A
= >
=0A
> After reproduce.sh finished, performance down.=0A
>
=0A
>
=0A
> Anyone has the same p= roblem or has some ideas about this?
=0A
>
=0A
> = Thanks!
=0A
> --------------
=0A
> hzwulibin
= =0A
> 2015-11-02
=0A
> =04 {.n +    = ;   +%  lzwm  b =EB=A7=B2  r  y=C7=A9 =D7=AF= zX  =17  =DC=A8}   =C6=A0z &j:+v   =      zZ+  +zf   h   ~ &n= bsp;  i   z =1E w   ?
=0A
> & )= =DF=A2=1Bf
=0A
=0A ------=_001_NextPart741106821217_=------ --===============1794768861== Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Disposition: inline _______________________________________________ ceph-users mailing list ceph-users-idqoXFIVOFJgJs9I8MT0rw@public.gmane.org http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com --===============1794768861==--