From mboxrd@z Thu Jan 1 00:00:00 1970 From: Alexandre DERUMIER Subject: Re: wip-auth Date: Fri, 6 Feb 2015 08:45:04 +0100 (CET) Message-ID: <999603781.7886589.1423208704771.JavaMail.zimbra@oxygem.tv> References: <3649A15A2562B54294DE14BCE5AC79120AB43257@ORSMSX152.amr.corp.intel.com> <3649A15A2562B54294DE14BCE5AC79120AB4E6A6@FMSMSX106.amr.corp.intel.com> <20150130081013.3f615803@doppio> <54D2AA3A.1070800@redhat.com> Mime-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: QUOTED-PRINTABLE Return-path: Received: from mailpro.odiso.net ([89.248.209.98]:42128 "EHLO mailpro.odiso.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1751404AbbBFHpH convert rfc822-to-8bit (ORCPT ); Fri, 6 Feb 2015 02:45:07 -0500 In-Reply-To: <1597425172.7886580.1423208685818.JavaMail.zimbra@oxygem.tv> Sender: ceph-devel-owner@vger.kernel.org List-ID: To: Mark Nelson Cc: Sage Weil , Andreas Bluemle , Stephen L Blinick , ceph-devel Hi Mark, do you known why rhel7 is faster than ubuntu ? (the difference seem to = be quite huge) Not sure it's related, but I have found a bug report on ubuntu, about irqbalance not working correctly on ubuntu 14.04 https://bugs.launchpad.net/ubuntu/+source/irqbalance/+bug/1379065 ----- Mail original ----- De: "Mark Nelson" =C3=80: "Sage Weil" , "Andreas Bluemle" Cc: "Stephen L Blinick" , "ceph-devel" Envoy=C3=A9: Jeudi 5 F=C3=A9vrier 2015 00:24:42 Objet: Re: wip-auth Hi All,=20 I completed some tests with wip-auth to the memstore on ubuntu earlier=20 today. Basic gist of it is that the improvements in wip-auth help but=20 don't quite get us to what can be achieved with auth disabled. RHEL7=20 (without auth) continues to do very well in latency bound situations.=20 Next up will be to see if how much this matters when testing against on= =20 SSDs.=20 Here are the results:=20 sync 4k object writes=20 =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=20 Ceph OS Auth Avg Lat (ms) Avg IOPS=20 ----------------------------------------------------------------=20 master Ubuntu 14.04 Yes 0.99 1007=20 Wip-auth Ubuntu 14.04 Yes 0.81 1237=20 master Ubuntu 14.04 No 0.64 1549=20 Wip-auth Ubuntu 14.04 No 0.65 1563=20 master RHEL7 No 0.32 3158=20 sync 4k object reads=20 =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=20 Ceph OS Auth Avg Lat (ms) Avg IOPS=20 ----------------------------------------------------------------=20 master Ubuntu 14.04 Yes 0.59 1695=20 Wip-auth Ubuntu 14.04 Yes 0.41 2409=20 master Ubuntu 14.04 No 0.29 3425=20 Wip-auth Ubuntu 14.04 No 0.29 3474=20 master RHEL7 No 0.17 5853=20 256 concurrent 4k object writes=20 =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=20 Ceph OS Auth Avg Lat (ms) Avg IOPS=20 ----------------------------------------------------------------=20 master Ubuntu 14.04 Yes 40.39 6339=20 Wip-auth Ubuntu 14.04 Yes 26.22 9763=20 master Ubuntu 14.04 No 17.46 14662=20 Wip-auth Ubuntu 14.04 No 17.34 14759=20 master RHEL7 No 14.93 17139=20 256 concurrent 4k object reads=20 =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=20 Ceph OS Auth Avg Lat (ms) Avg IOPS=20 ----------------------------------------------------------------=20 master Ubuntu 14.04 Yes 31.47 8134=20 Wip-auth Ubuntu 14.04 Yes 19.81 12922=20 master Ubuntu 14.04 No 12.82 19968=20 Wip-auth Ubuntu 14.04 No 12.75 20080=20 master RHEL7 No 12.04 21257=20 Mark=20 On 01/30/2015 03:08 PM, Sage Weil wrote:=20 > Hi Andreas,=20 >=20 > It looks like that was a stale sha1, but the newer one was also=20 > broken. I've retested and it's working for me now. See latest wip-aut= h,=20 > sha1 0c21a7875059bef80842756dfb003f47cc2d66a6.=20 >=20 > Thanks!=20 > sage=20 >=20 > On Fri, 30 Jan 2015, Andreas Bluemle wrote:=20 >=20 >> Hi Sage,=20 >>=20 >> I tried to integrate wip-auth into my 0.91 build environment.=20 >>=20 >> I had not been able to start the cluster successfully: ceph-mon=20 >> crashes with a segmentation fault in CryptoKey::encrypt()=20 >> (see attachment).=20 >>=20 >> This happens when linking with libnss or libcryptopp (version 5.6.2)= =2E=20 >>=20 >> I created the patch to add wip-auth based on github=20 >> pull request 3523 and was able to use this patch directly=20 >> with 0.91 with only a minor adaptation for common/ceph_context.h:=20 >> the 0.91 version of ceph_context.h did not know anything about=20 >> the experimental "class CephContextObs".=20 >>=20 >> wip-auth commit ID is 1a0507a2940f6edcc2bf9533cfa6c210b0b41933.=20 >>=20 >> As my build environment is rpm, I had to modify the invocation=20 >> of the configure script in the spec file instead of the do_autogen.s= h=20 >> script.=20 >>=20 >>=20 >> Best Regards=20 >>=20 >> Andreas Bluemle=20 >>=20 >>=20 >> On Tue, 27 Jan 2015 09:18:45 -0800 (PST)=20 >> Sage Weil wrote:=20 >>=20 >>> I spent some time focusing on just CryptoKey::encrypt(). I=20 >>> benchmarked 100,000 encrypts of 128 bytes and got, at baseline,=20 >>>=20 >>> cryptopp: 100000 encoded in 0.655651=20 >>> libnss : 100000 encoded in 1.288786=20 >>>=20 >>> Ouch! With a (fixed) version of my earlier patch that avoids=20 >>> uselessly copying the input buffer:=20 >>>=20 >>> 100000 encoded in 1.231977=20 >>>=20 >>> With a patch that puts the key structures in CryptoKey instead of=20 >>> recreating them each time:=20 >>>=20 >>> 100000 encoded in 0.396208 -- ~70% improvement over original=20 >>>=20 >>> This is pushed to wip-auth. There's also a patch that caches key=20 >>> structs for crypopp.. it now takes=20 >>>=20 >>> 100000 encoded in 0.440758 -- ~33% improvement over original=20 >>>=20 >>> (Not that almost anybody will ever care; we use libnss by default f= or=20 >>> both rpm and deb distros.)=20 >>>=20 >>> So, yay, nss is now a bit faster. What I'm not completely certain=20 >>> about is whether the structures I've preserved are truly stateless=20 >>> (and can be shared across threads, etc.). They encrypt/decrypt=20 >>> methods are const so, if the libraries are const-correct, it should= =20 >>> be fine... but perhaps someone familiar with nss and/or crypto++ ca= n=20 >>> review this?=20 >>>=20 >>> This is pushed to the latest wip-auth branch:=20 >>>=20 >>> https://github.com/ceph/ceph/commits/wip-auth=20 >>>=20 >>> Andreas and Stephen, what effect does this have on your numbers?=20 >>>=20 >>> Thanks!=20 >>> sage=20 >>>=20 >>>=20 >>> On Mon, 26 Jan 2015, Blinick, Stephen L wrote:=20 >>>=20 >>>> Good to know, I was wondering why the spec file defaulted to=20 >>>> lib-nss.. the dpkg-build for debian packages just uses whatever=20 >>>> configuration you had built, and I believe that will use=20 >>>> libcryptopp if the dependency is installed on the build machine=20 >>>> (last I looked).=20 >>>>=20 >>>> I forgot to mention the numbers below were based on v.91.=20 >>>>=20 >>>> Thanks,=20 >>>>=20 >>>> Stephen=20 >>>>=20 >>>> -----Original Message-----=20 >>>> From: ceph-devel-owner@vger.kernel.org=20 >>>> [mailto:ceph-devel-owner@vger.kernel.org] On Behalf Of Sage Weil=20 >>>> Sent: Monday, January 26, 2015 10:24 AM To: Blinick, Stephen L=20 >>>> Cc: andreas.bluemle@itxperts.de; ceph-devel@vger.kernel.org=20 >>>> Subject: RE: wip-auth=20 >>>>=20 >>>> On Mon, 26 Jan 2015, Blinick, Stephen L wrote:=20 >>>>> I noticed that the spec file for building RPM's defaults to=20 >>>>> building with libnss, instead of libcrypto++. Since the=20 >>>>> measurements I'd done so far were from those RPM's I rebuilt with= =20 >>>>> libcrypto++.. so FWIW here is the difference between those two on= =20 >>>>> my system, memstore backend with a single OSD, and single=20 >>>>> client.=20 >>>>>=20 >>>>> Dual socket Xeon E5 2620v3, 64GB Memory, RHEL7=20 >>>>> Kernel: 3.10.0-123.13.2.el7=20 >>>>>=20 >>>>> 100% 4K Writes, 1xOSD w/ Rados Bench=20 >>>>> libnss |=20 >>>>> Cryptopp # QD IOPS Latency(ms) |=20 >>>>> IOPS Latency(ms) IOPS Improvement % 16=20 >>>>> 14432.57 1.11 | 18896.60 0.85=20 >>>>> 30.93% 100% 4K Reads, 1xOSD w/ Rados=20 >>>>> Bench libnss | Cryptopp # QD IOPS Latency(ms) | IOPS Latency(ms)=20 >>>>> IOPS Improvement % 16 19532.53 0.82 | 25708.70 0.62 31.62%=20 >>>>=20 >>>> Yikes, 30%! I think this definitely worth some effort. We=20 >>>> switched to libnss because it has the weird government=20 >>>> certfiications that everyone wants and is more prevalent. crypto++= =20 >>>> is also not packaged for Red Hat distros at all (presumably for=20 >>>> that reason).=20 >>>>=20 >>>> I suspect that most of the overhead is in the encryption context=20 >>>> setup and can be avoided with a bit of effort..=20 >>>>=20 >>>> sage=20 >>>>=20 >>>>=20 >>>>>=20 >>>>>=20 >>>>> Thanks,=20 >>>>>=20 >>>>> Stephen=20 >>>>>=20 >>>>> -----Original Message-----=20 >>>>> From: ceph-devel-owner@vger.kernel.org=20 >>>>> [mailto:ceph-devel-owner@vger.kernel.org] On Behalf Of Sage Weil=20 >>>>> Sent: Thursday, January 22, 2015 4:56 PM=20 >>>>> To: andreas.bluemle@itxperts.de=20 >>>>> Cc: ceph-devel@vger.kernel.org=20 >>>>> Subject: wip-auth=20 >>>>>=20 >>>>> Hi Andreas,=20 >>>>>=20 >>>>> I took a look at the wip-auth I mentioned in the security call=20 >>>>> last week... and the patch didn't work at all. Sorry if you=20 >>>>> wasted any time trying it.=20 >>>>>=20 >>>>> Anyway, I fixed it up so that it actually worked and made one=20 >>>>> other optimization. It would be great to hear what latencies you=20 >>>>> measure with the changes in place.=20 >>>>>=20 >>>>> Also, it might be worth trying --with-cryptopp (or --with-nss if=20 >>>>> you built cryptopp by default) to see if there is a difference.=20 >>>>> There is a ton of boilerplate setting up encryption contexts and=20 >>>>> key structures and so on that I suspect could be cached (perhaps=20 >>>>> stashed in the CryptoKey struct?) with a bit of effort. See=20 >>>>>=20 >>>>> https://github.com/ceph/ceph/blob/master/src/auth/Crypto.cc#L99-L= 213=20 >>>>>=20 >>>>> sage=20 >>>>> --=20 >>>>> To unsubscribe from this list: send the line "unsubscribe=20 >>>>> ceph-devel" in the body of a message to majordomo@vger.kernel.org= =20 >>>>> More majordomo info at http://vger.kernel.org/majordomo-info.html= =20 >>>>>=20 >>>>>=20 >>>> --=20 >>>> To unsubscribe from this list: send the line "unsubscribe=20 >>>> ceph-devel" in the body of a message to majordomo@vger.kernel.org=20 >>>> More majordomo info at http://vger.kernel.org/majordomo-info.html=20 >>>> -- To unsubscribe from this list: send the line "unsubscribe=20 >>>> ceph-devel" in the body of a message to majordomo@vger.kernel.org=20 >>>> More majordomo info at http://vger.kernel.org/majordomo-info.html=20 >>>>=20 >>>>=20 >>>=20 >>>=20 >>=20 >>=20 >>=20 >> --=20 >> Andreas Bluemle mailto:Andreas.Bluemle@itxperts.de=20 >> ITXperts GmbH http://www.itxperts.de=20 >> Balanstrasse 73, Geb. 08 Phone: (+49) 89 89044917=20 >> D-81541 Muenchen (Germany) Fax: (+49) 89 89044910=20 >>=20 >> Company details: http://www.itxperts.de/imprint.htm=20 >>=20 > --=20 > To unsubscribe from this list: send the line "unsubscribe ceph-devel"= in=20 > the body of a message to majordomo@vger.kernel.org=20 > More majordomo info at http://vger.kernel.org/majordomo-info.html=20 >=20 --=20 To unsubscribe from this list: send the line "unsubscribe ceph-devel" i= n=20 the body of a message to majordomo@vger.kernel.org=20 More majordomo info at http://vger.kernel.org/majordomo-info.html=20 -- To unsubscribe from this list: send the line "unsubscribe ceph-devel" i= n the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html