Re: Memstore performance improvements v0.90 vs v0.87

[Mark Nelson <mnelson@redhat.com>]

I think paying attention to NUMA is good advice.  One of the things that 
apparently changed in RHEL7 is that they are now doing automatic NUMA 
tuning:

http://rhelblog.redhat.com/2015/01/12/mysteries-of-numa-memory-management-revealed/

It's possible that this could be having an effect on the results.

Mark

On 02/20/2015 03:49 AM, Blair Bethwaite wrote:
> Hi James,
>
> Interesting results, but did you do any tests with a NUMA system? IIUC
> the original report was from a dual socket setup, and that'd
> presumably be the standard setup for most folks (both OSD server and
> client side).
>
> Cheers,
>
> On 20 February 2015 at 20:07, James Page <james.page@ubuntu.com> wrote:
>> -----BEGIN PGP SIGNED MESSAGE-----
>> Hash: SHA256
>>
>> Hi All
>>
>> The Ubuntu Kernel team have spent the last few weeks investigating the
>> apparent performance disparity between RHEL 7 and Ubuntu 14.04; we've
>> focussed efforts in a few ways (see below).
>>
>> All testing has been done using the latest Firefly release.
>>
>> 1) Base network latency
>>
>> Jay Vosburgh looked at the base network latencies between RHEL 7 and
>> Ubuntu 14.04; under default install, RHEL actually had slightly worse
>> latency than Ubuntu due to the default enablement of a firewall;
>> disabling this brought latency back inline between the two distributions:
>>
>> OS                      rtt min/avg/max/mdev
>> Ubuntu 14.04 (3.13)     0.013/0.016/0.018/0.005 ms
>> RHEL7 (3.10)            0.010/0.018/0.025/0.005 ms
>>
>> ...base network latency is pretty much the same.
>>
>> This testing was performed on a matched pair of Dell Poweredge R610's,
>> configured with a single 4 core CPU and 8G of RAM.
>>
>> 2) Latency and performance in Ceph using Rados bench
>>
>> Colin King spent a number of days testing and analysing results using
>> rados bench against a single node ceph deployment, configured with a
>> single memory backed OSD, to see if we could reproduce the disparities
>> reported.
>>
>> He ran 120 second OSD benchmarks on RHEL 7 as well as Ubuntu 14.04 LTS
>> with a selection of kernels including 3.10 vanilla, 3.13.0-44 (release
>> kernel), 3.16.0-30 (utopic HWE kernel), 3.18.0-12 (vivid HWE kernel)
>> and 3.19-rc6 with 1, 16 and 128 client threads.  The data collected is
>> available at [0].
>>
>> Each round of tests consisted of 15 runs, from which we computed
>> average latency, latency deviation and latency distribution:
>>
>>> 120 second x 1 thread
>>
>> Results all seem to cluster around 0.04->0.05ms, with RHEL 7 averaging
>> at 0.044 and recent Ubuntu kernels at 0.036-0.037ms.  The older 3.10
>> kernel in RHEL 7 does have some slightly higher average latency.
>>
>>> 120 second x 16 threads
>>
>> Results all seem to cluster around 0.6-0.7ms.  3.19.0-rc6 had a couple
>> of 1.4ms outliers which pushed it out to be worse than RHEL 7. On the
>> whole Ubuntu 3.10-3.18 kernels are better than RHEL 7 by ~0.1ms.  RHEL
>> shows a far higher standard deviation, due to the bimodal latency
>> distribution, which from the casual observer may appear to be more
>> "jittery".
>>
>>> 120 second x 128 threads
>>
>> Later kernels show up to have less standard deviation than RHEL 7, so
>> that shows perhaps less jitter in the stats than RHEL 7's 3.10 kernel.
>> With this many threads pounding the test, we get a wider spread of
>> latencies and it is hard to tell any kind of latency distribution
>> patterns with just 15 rounds because of the large amount of latency
>> jitter.  All systems show a latency of ~ 5ms.  Taking into
>> consideration the amount of jitter, we think these results do not make
>> much sense unless we repeat these tests with say 100 samples.
>>
>> 3) Conclusion
>>
>> We’ve have not been able to show any major anomalies in Ceph on Ubuntu
>> compared to RHEL 7 when using memstore.  Our current hypothesis is that
>> one needs to run the OSD bench stressor many times to get a fair capture
>> of system latency stats.  The reason for this is:
>>
>> * Latencies are very low with memstore, so any small jitter in
>> scheduling etc will show up as a large distortion (as shown by the large
>> standard deviations in the samples).
>>
>> * When memstore is heavily utilized, memory pressure causes the system
>> to page heavily and so we are subject to the nature of perhaps delays on
>> paging that cause some latency jitters.  Latency differences may be just
>> down to where a random page is in memory or in swap, and with memstore
>> these may cause the large perturbations we see when running just a
>> single test.
>>
>> * We needed to make *many* tens of measurements to get a typical idea of
>> average latency and the latency distributions. Don't trust the results
>> from just one test
>>
>> * We ran the tests with a pool configured to 100 pgs and 100 pgps [1].
>> One can get different results with different placement group configs.
>>
>> I've CC'ed both Colin and Jay on this mail - so if anyone has any
>> specific questions about the testing they can chime in with responses.
>>
>> Regards
>>
>> James
>>
>> [0] http://kernel.ubuntu.com/~cking/.ceph/ceph-benchmarks.ods
>> [1] http://ceph.com/docs/master/rados/configuration/pool-pg-config-ref/
>>
>> - --
>> James Page
>> Ubuntu and Debian Developer
>> james.page@ubuntu.com
>> jamespage@debian.org
>> -----BEGIN PGP SIGNATURE-----
>> Version: GnuPG v1
>>
>> iQIcBAEBCAAGBQJU5vlrAAoJEL/srsug59jDMvAQAIhSR4GFTXNc4RLpHtLT6h/X
>> K5uyauKZGtL+wqtPKRfsXqbbUw9I5AZDifQuOEJ0APccLIPbgqxEN3d2uht/qygH
>> G8q2Ax+M8OyZz07yqTitnD4JV3RmL8wNHUveWPLV0gs2TzBBYwP1ywExbRPed3PY
>> cfDrszgkQszA/JwT5W5YNf1LZc+5VpOEFrTiLIaRzUDoxg7mm6Hwr3XT8OFjZhjm
>> LSenKREHtrKKWoBh+OKTvuCUnHzEemK+CiwwRbNQ8l7xbp71wLyS08NpSB5C1y70
>> 7uft+kP6XOGE9AKLvsdEL1PIXHfeKNonBEN5mO6nsXIW+MQzou01zHgDtne7AxDA
>> 5OebQJfJtArmKt78WHuVg7h8gPcIRTRSW43LqJiADnIHL8fnZxj2v5yDiUQj7isw
>> nYWXEJ3rR7mlVgydN34KQ7gpVWmGjhrVb8N01+zYOMAaTBnekldHdueEAXR07eU0
>> PXiP9aOZiAxbEnDiJmreehjCuNFTagQqNeECRIHssSacfQXPxVljaImvuSfrxf8i
>> myQLzftiObINTIHSN4TVDKMyveYrU2hILCKfYuxnSJh29j35wsRSeftjntOEyHai
>> RDnrLD3fCPk4h3hCY6l60nqu9MQfbgdSB/FItvhiBGYqXvGb4+wuBeU9RT9SwG8N
>> XPih7nLNvqDNw38IkkDN
>> =qcvG
>> -----END PGP SIGNATURE-----
>> --
>> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>
>
>
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html