From mboxrd@z Thu Jan 1 00:00:00 1970 From: Alexandre DERUMIER Subject: Re: filestore flusher = false , correct my problem of constant write (need info on this parameter) Date: Sun, 24 Jun 2012 10:10:48 +0200 (CEST) Message-ID: References: Mime-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: QUOTED-PRINTABLE Return-path: Received: from mailpro.odiso.net ([89.248.209.98]:46253 "EHLO mailpro.odiso.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1752493Ab2FXILN convert rfc822-to-8bit (ORCPT ); Sun, 24 Jun 2012 04:11:13 -0400 In-Reply-To: Sender: ceph-devel-owner@vger.kernel.org List-ID: To: Sage Weil Cc: Mark Nelson , ceph-devel@vger.kernel.org, Stefan Priebe ok, I have done tests with more than 1 client.=20 3 kvm guest on 3 differents kvm host server and 3 kvm guest on same ser= ver. I have the same result , around 2000 io/s shared between clients. So it= doesn't scale. I have also tried with 3 nodes x 5 osd 15k drive + 5 tmpfs journal=20 and 3 nodes x 1 osd (hardware raid0 with 5 x 15kdisk) + 1 tmpfs journal the results are same, around 2000 io/s But if I try with 3 nodes x 1 osd with 1x15k drive, I got around 500 io/s I also known that stefan priebe has achieve around 12000io/s with ssds = in osd. So it's seem related to osd drive speed So, are we sure that journal is acking to client before flushing to dis= k ?=20 benchmark used is fio, with directio, write 100MB of 4K block. (so jour= nal is big enough to handle all the write) fio --filename=3D[disk] --direct=3D1 --rw=3Drandwrite --bs=3D4k --size=3D= 100M --numjobs=3D50 --runtime=3D30 --group_reporting --name=3Dfile1 ----- Mail original -----=20 De: "Alexandre DERUMIER" =20 =C3=80: "Sage Weil" =20 Cc: "Mark Nelson" , ceph-devel@vger.kernel.org= , "Stefan Priebe" =20 Envoy=C3=A9: Samedi 23 Juin 2012 20:21:05=20 Objet: Re: filestore flusher =3D false , correct my problem of constant= write (need info on this parameter)=20 >>Is that 2000 ios from a single client? You might try multiple clients= and=20 >>see if the sum of the ios will scale any higher.=20 yes from a single client. (qemu-kvm guest).=20 Tomorrow,I'll retry with 3 qemu-kvm guest, on same host and 3 different= s hosts.=20 I'll also try on bigger cpu machine to compra.(I see a lot of cpu on my= kvm guest process, more than with iscsi)=20 I'll keep you in touch.=20 Thanks=20 Alexandre=20 ----- Mail original -----=20 De: "Sage Weil" =20 =C3=80: "Alexandre DERUMIER" =20 Cc: "Mark Nelson" , ceph-devel@vger.kernel.org= , "Stefan Priebe" =20 Envoy=C3=A9: Samedi 23 Juin 2012 20:12:49=20 Objet: Re: filestore flusher =3D false , correct my problem of constant= write (need info on this parameter)=20 On Sat, 23 Jun 2012, Alexandre DERUMIER wrote:=20 > >>I was just talking with Elder on IRC yesterday about looking into h= ow=20 > >>much small network transfers are hurting us in cases like these. Ev= en=20 > >>with SSD based OSDs I haven't seen a very dramatic improvement in s= mall=20 > >>request performance. How tough would it be to aggregate requests in= to=20 > >>larger network transactions? There would be a latency penalty of=20 > >>course, but we could flush a client side dirty cache pretty quickly= and=20 > >>still benefit if we are getting bombarded with lots of tiny request= s.=20 >=20 > Yes, I see no improvement with journal on tmpfs ...this is strange..=20 >=20 > Also, I have tried with rbd_cache=3Dtrue, so ios should be already ag= gregate in bigger transaction.=20 > But I didnt't have see any improvement.=20 >=20 > I'm around 2000 ios.=20 >=20 > Do you know what is the bottleneck ? rbd protocol (some kind of overh= ead=20 > for each io ?)=20 Is that 2000 ios from a single client? You might try multiple clients a= nd=20 see if the sum of the ios will scale any higher. That will tell us=20 whether it is in the messenger or osd request pipeline. The latter=20 definitely needs some work, although there may be a quick fix to the ms= gr=20 that will buy us some too.=20 sage=20 >=20 >=20 > ----- Mail original -----=20 >=20 > De: "Mark Nelson" =20 > =C3=80: "Sage Weil" =20 > Cc: "Alexandre DERUMIER" , ceph-devel@vger.kerne= l.org, "Stefan Priebe" =20 > Envoy=C3=A9: Samedi 23 Juin 2012 18:40:28=20 > Objet: Re: filestore flusher =3D false , correct my problem of consta= nt write (need info on this parameter)=20 >=20 > On 6/23/12 10:38 AM, Sage Weil wrote:=20 > > On Fri, 22 Jun 2012, Alexandre DERUMIER wrote:=20 > >> Hi Sage,=20 > >> thanks for your response.=20 > >>=20 > >>>> If you turn off the journal compeletely, you will see bursty wri= te commits=20 > >>> >from the perspective of the client, because the OSD is periodica= lly doing=20 > >>>> a sync or snapshot and only acking the writes then.=20 > >>>> If you enable the journal, the OSD will reply with a commit as s= oon as the=20 > >>>> write is stable in the journal. That's one reason why it is ther= e--file=20 > >>>> system commits of heavyweight and slow.=20 > >>=20 > >> Yes of course, I don't wan't to desactivate journal, using a journ= al on a fast ssd or nvram is the right way.=20 > >>=20 > >>>> If we left the file system to its own devices and did a sync eve= ry 10=20 > >>>> seconds, the disk would sit idle while a bunch of dirty data acc= umulated=20 > >>>> in cache, and then the sync/snapshot would take a really long ti= me. This=20 > >>>> is horribly inefficient (the disk is idle half the time), and us= eless (the=20 > >>>> delayed write behavior makes sense for local workloads, but not = servers=20 > >>>> where there is a client on the other end batching its writes). T= o prevent=20 > >>>> this, 'filestore flusher' will prod the kernel to flush out any = written=20 > >>>> data to the disk quickly. Then, when we get around to doing the=20 > >>>> sync/snapshot it is pretty quick, because only fs metadata and=20 > >>>> just-written data needs to be flushed.=20 > >>=20 > >> mmm, I disagree.=20 > >>=20 > >> If you flush quickly, it's works fine with sequential write worklo= ad.=20 > >>=20 > >> But if you have a lot of random write with 4k block by exemple, yo= u are=20 > >> going to have a lot of disk seeks. The way zfs works or netapp san= =20 > >> storage works, they take random writes in a fast journal then flus= h them=20 > >> sequentially each 20s to slow storage.=20 > >=20 > > Oh, I see what you're getting at. Yes, that is not ideal for small = random=20 > > writes. There is a branch in ceph.git called wip-flushmin that just= sets=20 > > a minimum write size for the flush that will probably do a decent j= ob of=20 > > dealing with this: small writes won't get flushed, large ones will.= =20 > > Picking the right value will depend on how expensive seeks are for = your=20 > > storage system.=20 > >=20 > > You'll want to cherry-pick just the top commit on top of whatever i= t is=20 > > you're running...=20 >=20 > I was just talking with Elder on IRC yesterday about looking into how= =20 > much small network transfers are hurting us in cases like these. Even= =20 > with SSD based OSDs I haven't seen a very dramatic improvement in sma= ll=20 > request performance. How tough would it be to aggregate requests into= =20 > larger network transactions? There would be a latency penalty of=20 > course, but we could flush a client side dirty cache pretty quickly a= nd=20 > still benefit if we are getting bombarded with lots of tiny requests.= =20 >=20 > Mark=20 > --=20 > To unsubscribe from this list: send the line "unsubscribe ceph-devel"= in=20 > the body of a message to majordomo@vger.kernel.org=20 > More majordomo info at http://vger.kernel.org/majordomo-info.html=20 >=20 >=20 >=20 > --=20 >=20 > --=20 >=20 >=20 >=20 >=20 >=20 > Alexandre D e rumier=20 >=20 > Ing=C3=A9nieur Syst=C3=A8mes et R=C3=A9seaux=20 >=20 >=20 > Fixe : 03 20 68 88 85=20 >=20 > Fax : 03 20 68 90 88=20 >=20 >=20 > 45 Bvd du G=C3=A9n=C3=A9ral Leclerc 59100 Roubaix=20 > 12 rue Marivaux 75002 Paris=20 > --=20 > To unsubscribe from this list: send the line "unsubscribe ceph-devel"= in=20 > the body of a message to majordomo@vger.kernel.org=20 > More majordomo info at http://vger.kernel.org/majordomo-info.html=20 >=20 >=20 --=20 --=20 Alexandre D e rumier=20 Ing=C3=A9nieur Syst=C3=A8mes et R=C3=A9seaux=20 =46ixe : 03 20 68 88 85=20 =46ax : 03 20 68 90 88=20 45 Bvd du G=C3=A9n=C3=A9ral Leclerc 59100 Roubaix=20 12 rue Marivaux 75002 Paris=20 --=20 To unsubscribe from this list: send the line "unsubscribe ceph-devel" i= n=20 the body of a message to majordomo@vger.kernel.org=20 More majordomo info at http://vger.kernel.org/majordomo-info.html=20 --=20 --=20 =09 Alexandre D e rumier=20 Ing=C3=A9nieur Syst=C3=A8mes et R=C3=A9seaux=20 =46ixe : 03 20 68 88 85=20 =46ax : 03 20 68 90 88=20 45 Bvd du G=C3=A9n=C3=A9ral Leclerc 59100 Roubaix=20 12 rue Marivaux 75002 Paris=20 -- To unsubscribe from this list: send the line "unsubscribe ceph-devel" i= n the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html