Re: [Qemu-devel] [PATCH] configure: Add support for jemalloc

[Alexandre DERUMIER <aderumier@odiso.com>]

@cc Fam Zheng <famz@redhat.com>,

as he's the author of tcmalloc support patch

----- Mail original -----
De: "aderumier" <aderumier@odiso.com>
À: "qemu-devel" <qemu-devel@nongnu.org>
Cc: "aderumier" <aderumier@odiso.com>
Envoyé: Vendredi 19 Juin 2015 12:56:58
Objet: [PATCH] configure: Add support for jemalloc

This adds "--enable-jemalloc" and "--disable-jemalloc" to allow linking 
to jemalloc memory allocator. 

We have already tcmalloc support, 
but it seem to not working well with a lot of iothreads/disks. 

The main problem is that tcmalloc use a shared thread cache of 16MB 
by default. 
With more threads, this cache is shared, and some bad garbage collections 
can occur if the cache is too low. 

It's possible to tcmalloc cache increase it with a env var: 
TCMALLOC_MAX_TOTAL_THREAD_CACHE_BYTES=256MB 

With default 16MB, performances are really bad with more than 2 disks. 
Increasing to 256MB, it's helping but still have problem with 16 disks/iothreads. 

Jemalloc don't have performance problem with default configuration. 

Here the benchmark results in iops of 1 qemu vm randread 4K iodepth=32, 
with rbd block backend (librbd is doing a lot of memory allocation), 
1 iothread by disk 

glibc malloc 
------------ 

1 disk 29052 
2 disks 55878 
4 disks 127899 
8 disks 240566 
15 disks 269976 

jemalloc 
-------- 

1 disk 41278 
2 disks 75781 
4 disks 195351 
8 disks 294241 
15 disks 298199 

tcmalloc 2.2.1 default 16M cache 
-------------------------------- 

1 disk 37911 
2 disks 67698 
4 disks 41076 
8 disks 43312 
15 disks 37569 

tcmalloc : 256M cache 
--------------------------- 

1 disk 33914 
2 disks 58839 
4 disks 148205 
8 disks 213298 
15 disks 218383 

Signed-off-by: Alexandre Derumier <aderumier@odiso.com> 
--- 
configure | 29 +++++++++++++++++++++++++++++ 
1 file changed, 29 insertions(+) 

diff --git a/configure b/configure 
index 222694f..2fe1e05 100755 
--- a/configure 
+++ b/configure 
@@ -336,6 +336,7 @@ vhdx="" 
quorum="" 
numa="" 
tcmalloc="no" 
+jemalloc="no" 

# parse CC options first 
for opt do 
@@ -1147,6 +1148,10 @@ for opt do 
;; 
--enable-tcmalloc) tcmalloc="yes" 
;; 
+ --disable-jemalloc) jemalloc="no" 
+ ;; 
+ --enable-jemalloc) jemalloc="yes" 
+ ;; 
*) 
echo "ERROR: unknown option $opt" 
echo "Try '$0 --help' for more information" 
@@ -1420,6 +1425,8 @@ Advanced options (experts only): 
--enable-numa enable libnuma support 
--disable-tcmalloc disable tcmalloc support 
--enable-tcmalloc enable tcmalloc support 
+ --disable-jemalloc disable jemalloc support 
+ --enable-jemalloc enable jemalloc support 

NOTE: The object files are built at the place where configure is launched 
EOF 
@@ -3344,6 +3351,11 @@ EOF 
fi 
fi 

+if test "$tcmalloc" = "yes" && test "$jemalloc" = "yes" ; then 
+ echo "ERROR: tcmalloc && jemalloc can't be used at the same time" 
+ exit 1 
+fi 
+ 
########################################## 
# tcmalloc probe 

@@ -3361,6 +3373,22 @@ EOF 
fi 

########################################## 
+# jemalloc probe 
+ 
+if test "$jemalloc" = "yes" ; then 
+ cat > $TMPC << EOF 
+#include <stdlib.h> 
+int main(void) { malloc(1); return 0; } 
+EOF 
+ 
+ if compile_prog "" "-ljemalloc" ; then 
+ LIBS="-ljemalloc $LIBS" 
+ else 
+ feature_not_found "jemalloc" "install jemalloc devel" 
+ fi 
+fi 
+ 
+########################################## 
# signalfd probe 
signalfd="no" 
cat > $TMPC << EOF 
@@ -4499,6 +4527,7 @@ echo "snappy support $snappy" 
echo "bzip2 support $bzip2" 
echo "NUMA host support $numa" 
echo "tcmalloc support $tcmalloc" 
+echo "jemalloc support $jemalloc" 

if test "$sdl_too_old" = "yes"; then 
echo "-> Your SDL version is too old - please upgrade to have SDL support" 
-- 
2.1.4