* Memory leak in migration on Xen 4.4
@ 2015-11-19 14:51 Ian Campbell
2015-11-20 15:22 ` Matthew Vernon
0 siblings, 1 reply; 2+ messages in thread
From: Ian Campbell @ 2015-11-19 14:51 UTC (permalink / raw)
To: xen-devel; +Cc: Ian Jackson, Matthew Vernon
[-- Attachment #1: Type: text/plain, Size: 2170 bytes --]
Following on from IRC conversation regarding an issue similar to [0] (which
was reported by someone else).
I've implemented the missing hypercall support in valgrind, see attached
which applies to SVN r15732. This is migrating cleanly (i.e. no
"README_MISSING_SYSCALL_OR_IOCTL" spew) for me with the attached
suppression file which knocks out one known (benign) issue.
With this things appear pretty clean using the Jessie packages for
everything. Just a few k lost or remaining reachable at the end (see
attached logs).
I'm somewhat concerned by the restore log though, it contains:
==3670== HEAP SUMMARY:
==3670== in use at exit: 3,288 bytes in 4 blocks
==3670== total heap usage: 1,343 allocs, 1,339 frees, 542,238,933 bytes allocated
That's 517M allocated, which is approximately the size of my test VM. It's
not clear though if this is cumulative usage or maximum concurrent usage.
Trying with a domain with 3500M (which is the most I can manage for a live
migration on this host) I get:
==4521== HEAP SUMMARY:
==4521== in use at exit: 15,240 bytes in 4 blocks
==4521== total heap usage: 8,847 allocs, 8,843 frees, 3,698,460,199 bytes allocated
So it does seem to related to the domain size.
However my dom0 has 512M RAM and 2G of swap, so 3500 should be well into
swap if this was max concurrent memory use, but I'm not seeing it getting
much into swap at all i.e. when the migration is about 80% done:
root@st40:~# free
total used free shared buffers cached
Mem: 431092 411448 19644 408 4488 198944
-/+ buffers/cache: 208016 223076
Swap: 1949692 1036 1948656
So I conclude that the "bytes allocated" is cumulative and not maximum
concurrent (and that I can still not reproduce this issue).
For completeness I've attached my libxl-save-helper wrapper script.
I'm running everything from the Jessie packages.
Ian.
[0] https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=797205
[-- Attachment #2: 0001-xen-Implement-__HYPERVISOR_mmu_update.patch --]
[-- Type: text/x-patch, Size: 2248 bytes --]
From b81cb76e34ac27441d270a655e48f741bf6b4b37 Mon Sep 17 00:00:00 2001
From: Ian Campbell <ian.campbell@citrix.com>
Date: Thu, 19 Nov 2015 13:07:07 +0000
Subject: [PATCH] xen: Implement __HYPERVISOR_mmu_update
Signed-off-by: Ian Campbell <ian.campbell@citrix.com>
---
coregrind/m_syswrap/syswrap-xen.c | 16 +++++++++++++++-
include/vki/vki-xen.h | 5 +++++
2 files changed, 20 insertions(+), 1 deletion(-)
diff --git a/coregrind/m_syswrap/syswrap-xen.c b/coregrind/m_syswrap/syswrap-xen.c
index 20341c4..5c0e77b 100644
--- a/coregrind/m_syswrap/syswrap-xen.c
+++ b/coregrind/m_syswrap/syswrap-xen.c
@@ -109,6 +109,15 @@ static void bad_subop ( ThreadId tid,
SET_STATUS_Failure(VKI_ENOSYS);
}
+PRE(mmu_update)
+{
+ PRINT("__HYPERVISOR_mmu_update ( %#lx, %#lx, %#lx, %#lx )",
+ ARG1, ARG2, ARG3, ARG4);
+
+ PRE_MEM_READ("__HYPERVISOR_mmu_update",
+ (Addr)ARG1, sizeof(struct vki_xen_mmu_update)*ARG2);
+}
+
PRE(memory_op)
{
PRINT("__HYPERVISOR_memory_op ( %lu, %#lx )", ARG1, ARG2);
@@ -1327,6 +1336,11 @@ PRE(tmem_op)
#undef __PRE_XEN_TMEMOP_READ
}
+POST(mmu_update)
+{
+ POST_MEM_WRITE(ARG3, sizeof(unsigned int));
+}
+
POST(memory_op)
{
switch (ARG1) {
@@ -2035,7 +2049,7 @@ typedef
static XenHypercallTableEntry hypercall_table[] = {
// __VKI_XEN_set_trap_table // 0
- // __VKI_XEN_mmu_update // 1
+ HYPXY(__VKI_XEN_mmu_update, mmu_update, 4), // 1
// __VKI_XEN_set_gdt // 2
// __VKI_XEN_stack_switch // 3
// __VKI_XEN_set_callbacks // 4
diff --git a/include/vki/vki-xen.h b/include/vki/vki-xen.h
index 924a1e9..0d6ed0c 100644
--- a/include/vki/vki-xen.h
+++ b/include/vki/vki-xen.h
@@ -104,6 +104,11 @@ struct vki_xenctl_bitmap {
vki_uint32_t nr_bits;
};
+struct vki_xen_mmu_update {
+ vki_uint64_t ptr; /* Machine address of PTE. */
+ vki_uint64_t val; /* New contents of PTE. */
+};
+
#include <vki/vki-xen-domctl.h>
#include <vki/vki-xen-sysctl.h>
#include <vki/vki-xen-mmuext.h>
--
2.1.4
[-- Attachment #3: xen-migration.supp --]
[-- Type: text/plain, Size: 215 bytes --]
{
xc_domain_save
Memcheck:Param
XEN_DOMCTL_getvcpuextstate u.vcpuextstate.buffer
fun:ioctl
fun:xc_domctl
fun:xc_domain_save
obj:/usr/lib/xen-4.4/bin/libxl-save-helper.orig
fun:(below main)
}
[-- Attachment #4: libxl-save-helper--restore-domain.valgrind.log --]
[-- Type: text/x-log, Size: 2699 bytes --]
==3670== Memcheck, a memory error detector
==3670== Copyright (C) 2002-2015, and GNU GPL'd, by Julian Seward et al.
==3670== Using Valgrind-3.12.0.SVN and LibVEX; rerun with -h for copyright info
==3670== Command: /usr/lib/xen-4.4/bin/libxl-save-helper.orig --restore-domain 6 8 1 0 2 0 0 1 0 1 0 0
==3670== Parent PID: 3657
==3670==
==3670==
==3670== HEAP SUMMARY:
==3670== in use at exit: 3,288 bytes in 4 blocks
==3670== total heap usage: 1,343 allocs, 1,339 frees, 542,238,933 bytes allocated
==3670==
==3670== 8 bytes in 1 blocks are definitely lost in loss record 1 of 4
==3670== at 0x4C27C0F: malloc (vg_replace_malloc.c:298)
==3670== by 0x4C29BDB: realloc (vg_replace_malloc.c:785)
==3670== by 0x5062405: ??? (in /usr/lib/x86_64-linux-gnu/libxenguest-4.4.so)
==3670== by 0x5061E02: ??? (in /usr/lib/x86_64-linux-gnu/libxenguest-4.4.so)
==3670== by 0x5063ADB: xc_domain_restore (in /usr/lib/x86_64-linux-gnu/libxenguest-4.4.so)
==3670== by 0x4014AF: ??? (in /usr/lib/xen-4.4/bin/libxl-save-helper.orig)
==3670== by 0x54C4B44: (below main) (libc-start.c:287)
==3670==
==3670== 24 bytes in 1 blocks are still reachable in loss record 2 of 4
==3670== at 0x4C27CC5: malloc (vg_replace_malloc.c:299)
==3670== by 0x401774: ??? (in /usr/lib/xen-4.4/bin/libxl-save-helper.orig)
==3670== by 0x401456: ??? (in /usr/lib/xen-4.4/bin/libxl-save-helper.orig)
==3670== by 0x54C4B44: (below main) (libc-start.c:287)
==3670==
==3670== 1,208 bytes in 1 blocks are still reachable in loss record 3 of 4
==3670== at 0x4C27CC5: malloc (vg_replace_malloc.c:299)
==3670== by 0x4E4808A: ??? (in /usr/lib/x86_64-linux-gnu/libxenctrl-4.4.so)
==3670== by 0x4017CC: ??? (in /usr/lib/xen-4.4/bin/libxl-save-helper.orig)
==3670== by 0x401456: ??? (in /usr/lib/xen-4.4/bin/libxl-save-helper.orig)
==3670== by 0x54C4B44: (below main) (libc-start.c:287)
==3670==
==3670== 2,048 bytes in 1 blocks are definitely lost in loss record 4 of 4
==3670== at 0x4C27CC5: malloc (vg_replace_malloc.c:299)
==3670== by 0x5063806: xc_domain_restore (in /usr/lib/x86_64-linux-gnu/libxenguest-4.4.so)
==3670== by 0x4014AF: ??? (in /usr/lib/xen-4.4/bin/libxl-save-helper.orig)
==3670== by 0x54C4B44: (below main) (libc-start.c:287)
==3670==
==3670== LEAK SUMMARY:
==3670== definitely lost: 2,056 bytes in 2 blocks
==3670== indirectly lost: 0 bytes in 0 blocks
==3670== possibly lost: 0 bytes in 0 blocks
==3670== still reachable: 1,232 bytes in 2 blocks
==3670== suppressed: 0 bytes in 0 blocks
==3670==
==3670== For counts of detected and suppressed errors, rerun with: -v
==3670== ERROR SUMMARY: 2 errors from 2 contexts (suppressed: 0 from 0)
[-- Attachment #5: libxl-save-helper--save-domain.valgrind.log --]
[-- Type: text/x-log, Size: 1721 bytes --]
==3668== Memcheck, a memory error detector
==3668== Copyright (C) 2002-2015, and GNU GPL'd, by Julian Seward et al.
==3668== Using Valgrind-3.12.0.SVN and LibVEX; rerun with -h for copyright info
==3668== Command: /usr/lib/xen-4.4/bin/libxl-save-helper.orig --save-domain 8 7 0 0 1 0 0 12 8 72
==3668== Parent PID: 3643
==3668==
==3668==
==3668== HEAP SUMMARY:
==3668== in use at exit: 1,232 bytes in 2 blocks
==3668== total heap usage: 203 allocs, 201 frees, 16,891,376 bytes allocated
==3668==
==3668== 24 bytes in 1 blocks are still reachable in loss record 1 of 2
==3668== at 0x4C27CC5: malloc (vg_replace_malloc.c:299)
==3668== by 0x401774: ??? (in /usr/lib/xen-4.4/bin/libxl-save-helper.orig)
==3668== by 0x40128C: ??? (in /usr/lib/xen-4.4/bin/libxl-save-helper.orig)
==3668== by 0x54C4B44: (below main) (libc-start.c:287)
==3668==
==3668== 1,208 bytes in 1 blocks are still reachable in loss record 2 of 2
==3668== at 0x4C27CC5: malloc (vg_replace_malloc.c:299)
==3668== by 0x4E4808A: ??? (in /usr/lib/x86_64-linux-gnu/libxenctrl-4.4.so)
==3668== by 0x4017CC: ??? (in /usr/lib/xen-4.4/bin/libxl-save-helper.orig)
==3668== by 0x40128C: ??? (in /usr/lib/xen-4.4/bin/libxl-save-helper.orig)
==3668== by 0x54C4B44: (below main) (libc-start.c:287)
==3668==
==3668== LEAK SUMMARY:
==3668== definitely lost: 0 bytes in 0 blocks
==3668== indirectly lost: 0 bytes in 0 blocks
==3668== possibly lost: 0 bytes in 0 blocks
==3668== still reachable: 1,232 bytes in 2 blocks
==3668== suppressed: 0 bytes in 0 blocks
==3668==
==3668== For counts of detected and suppressed errors, rerun with: -v
==3668== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 1 from 1)
[-- Attachment #6: libxl-save-helper --]
[-- Type: application/x-shellscript, Size: 283 bytes --]
[-- Attachment #7: Type: text/plain, Size: 126 bytes --]
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel
^ permalink raw reply related [flat|nested] 2+ messages in thread
* Re: Memory leak in migration on Xen 4.4
2015-11-19 14:51 Memory leak in migration on Xen 4.4 Ian Campbell
@ 2015-11-20 15:22 ` Matthew Vernon
0 siblings, 0 replies; 2+ messages in thread
From: Matthew Vernon @ 2015-11-20 15:22 UTC (permalink / raw)
To: Ian Campbell, xen-devel; +Cc: Ian Jackson
[-- Attachment #1: Type: text/plain, Size: 1007 bytes --]
Hello,
On 19/11/15 14:51, Ian Campbell wrote:
> Following on from IRC conversation regarding an issue similar to [0] (which
> was reported by someone else).
>
> I've implemented the missing hypercall support in valgrind, see attached
> which applies to SVN r15732. This is migrating cleanly (i.e. no
> "README_MISSING_SYSCALL_OR_IOCTL" spew) for me with the attached
> suppression file which knocks out one known (benign) issue.
I applied this patch to unstable's valgrind, and have now performed a
migration in both directions (see attached).
> With this things appear pretty clean using the Jessie packages for
> everything. Just a few k lost or remaining reachable at the end (see
> attached logs).
Can I suggest you try the kernel from jessie-backports? That's what I'm
using; I have kernel 4.2.0 (I needed this because 3.6 from jessie
doesn't support some of my hardware when running Xen).
This is a 1G guest again, and you'll see that receiving version gets
through over 4G.
Regards,
Matthew
[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: xsl_receive.log --]
[-- Type: text/x-log; name="xsl_receive.log", Size: 3550 bytes --]
==17035== Memcheck, a memory error detector
==17035== Copyright (C) 2002-2015, and GNU GPL'd, by Julian Seward et al.
==17035== Using Valgrind-3.11.0 and LibVEX; rerun with -h for copyright info
==17035== Command: /usr/lib/xen-4.4/bin/libxl-save-helper.orig --restore-domain 6 9 1 0 2 0 0 1 0 1 0 0
==17035== Parent PID: 17034
==17035==
==17035== Warning: set address range perms: large range [0x395dc040, 0x795dc040) (defined)
==17035== Warning: set address range perms: large range [0x795dd040, 0xb95dd040) (defined)
==17035== Warning: set address range perms: large range [0xb95de040, 0xf93de040) (undefined)
==17035== Warning: set address range perms: large range [0xf93df000, 0x1393df000) (defined)
==17035== Warning: set address range perms: large range [0xf93df000, 0x1393df000) (noaccess)
==17035== Warning: set address range perms: large range [0x395dc028, 0x795dc058) (noaccess)
==17035== Warning: set address range perms: large range [0x795dd028, 0xb95dd058) (noaccess)
==17035== Warning: set address range perms: large range [0xb95de028, 0xf93de058) (noaccess)
==17035==
==17035== HEAP SUMMARY:
==17035== in use at exit: 2,098,392 bytes in 4 blocks
==17035== total heap usage: 2,822 allocs, 2,818 frees, 4,342,797,896 bytes allocated
==17035==
==17035== 8 bytes in 1 blocks are definitely lost in loss record 1 of 4
==17035== at 0x4C28CA0: malloc (vg_replace_malloc.c:299)
==17035== by 0x4C2B04F: realloc (vg_replace_malloc.c:785)
==17035== by 0x5064405: ??? (in /usr/lib/x86_64-linux-gnu/libxenguest-4.4.so)
==17035== by 0x5063E02: ??? (in /usr/lib/x86_64-linux-gnu/libxenguest-4.4.so)
==17035== by 0x5065ADB: xc_domain_restore (in /usr/lib/x86_64-linux-gnu/libxenguest-4.4.so)
==17035== by 0x4014AF: ??? (in /usr/lib/xen-4.4/bin/libxl-save-helper.orig)
==17035== by 0x54C6B44: (below main) (libc-start.c:287)
==17035==
==17035== 24 bytes in 1 blocks are still reachable in loss record 2 of 4
==17035== at 0x4C28CA0: malloc (vg_replace_malloc.c:299)
==17035== by 0x401774: ??? (in /usr/lib/xen-4.4/bin/libxl-save-helper.orig)
==17035== by 0x401456: ??? (in /usr/lib/xen-4.4/bin/libxl-save-helper.orig)
==17035== by 0x54C6B44: (below main) (libc-start.c:287)
==17035==
==17035== 1,208 bytes in 1 blocks are still reachable in loss record 3 of 4
==17035== at 0x4C28CA0: malloc (vg_replace_malloc.c:299)
==17035== by 0x4E4A08A: ??? (in /usr/lib/x86_64-linux-gnu/libxenctrl-4.4.so)
==17035== by 0x4017CC: ??? (in /usr/lib/xen-4.4/bin/libxl-save-helper.orig)
==17035== by 0x401456: ??? (in /usr/lib/xen-4.4/bin/libxl-save-helper.orig)
==17035== by 0x54C6B44: (below main) (libc-start.c:287)
==17035==
==17035== 2,097,152 bytes in 1 blocks are definitely lost in loss record 4 of 4
==17035== at 0x4C28CA0: malloc (vg_replace_malloc.c:299)
==17035== by 0x5065806: xc_domain_restore (in /usr/lib/x86_64-linux-gnu/libxenguest-4.4.so)
==17035== by 0x4014AF: ??? (in /usr/lib/xen-4.4/bin/libxl-save-helper.orig)
==17035== by 0x54C6B44: (below main) (libc-start.c:287)
==17035==
==17035== LEAK SUMMARY:
==17035== definitely lost: 2,097,160 bytes in 2 blocks
==17035== indirectly lost: 0 bytes in 0 blocks
==17035== possibly lost: 0 bytes in 0 blocks
==17035== still reachable: 1,232 bytes in 2 blocks
==17035== suppressed: 0 bytes in 0 blocks
==17035==
==17035== For counts of detected and suppressed errors, rerun with: -v
==17035== ERROR SUMMARY: 2 errors from 2 contexts (suppressed: 0 from 0)
[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #3: xsl_send.log --]
[-- Type: text/x-log; name="xsl_send.log", Size: 2114 bytes --]
==6331== Memcheck, a memory error detector
==6331== Copyright (C) 2002-2015, and GNU GPL'd, by Julian Seward et al.
==6331== Using Valgrind-3.11.0 and LibVEX; rerun with -h for copyright info
==6331== Command: /usr/lib/xen-4.4/bin/libxl-save-helper.orig --save-domain 8 9 0 0 1 0 0 12 8 72
==6331== Parent PID: 6330
==6331==
==6331== Warning: set address range perms: large range [0xa292000, 0x2a692000) (defined)
==6331== Warning: set address range perms: large range [0x395dc000, 0x795dc000) (defined)
==6331== Warning: set address range perms: large range [0x395dc000, 0x795dc000) (noaccess)
==6331== Warning: set address range perms: large range [0xa292000, 0x2a692000) (noaccess)
==6331==
==6331== HEAP SUMMARY:
==6331== in use at exit: 1,232 bytes in 2 blocks
==6331== total heap usage: 697 allocs, 695 frees, 38,370,651 bytes allocated
==6331==
==6331== 24 bytes in 1 blocks are still reachable in loss record 1 of 2
==6331== at 0x4C28CA0: malloc (vg_replace_malloc.c:299)
==6331== by 0x401774: ??? (in /usr/lib/xen-4.4/bin/libxl-save-helper.orig)
==6331== by 0x40128C: ??? (in /usr/lib/xen-4.4/bin/libxl-save-helper.orig)
==6331== by 0x54C6B44: (below main) (libc-start.c:287)
==6331==
==6331== 1,208 bytes in 1 blocks are still reachable in loss record 2 of 2
==6331== at 0x4C28CA0: malloc (vg_replace_malloc.c:299)
==6331== by 0x4E4A08A: ??? (in /usr/lib/x86_64-linux-gnu/libxenctrl-4.4.so)
==6331== by 0x4017CC: ??? (in /usr/lib/xen-4.4/bin/libxl-save-helper.orig)
==6331== by 0x40128C: ??? (in /usr/lib/xen-4.4/bin/libxl-save-helper.orig)
==6331== by 0x54C6B44: (below main) (libc-start.c:287)
==6331==
==6331== LEAK SUMMARY:
==6331== definitely lost: 0 bytes in 0 blocks
==6331== indirectly lost: 0 bytes in 0 blocks
==6331== possibly lost: 0 bytes in 0 blocks
==6331== still reachable: 1,232 bytes in 2 blocks
==6331== suppressed: 0 bytes in 0 blocks
==6331==
==6331== For counts of detected and suppressed errors, rerun with: -v
==6331== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 1 from 1)
[-- Attachment #4: Type: text/plain, Size: 126 bytes --]
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 2+ messages in thread
end of thread, other threads:[~2015-11-20 15:22 UTC | newest]
Thread overview: 2+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2015-11-19 14:51 Memory leak in migration on Xen 4.4 Ian Campbell
2015-11-20 15:22 ` Matthew Vernon
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).