Found memory leak in libxenstore

Found memory leak in libxenstore

[Sunguodong]

[-- Attachment #1.1: Type: text/plain, Size: 2415 bytes --]

Hi,
I found this memory leak on xen-4.5.0:
linux-byXjTX:~ # virsh version
Compiled against library: libvirt 1.2.17
Using library: libvirt 1.2.17
Using API: Xen 1.2.17
Running hypervisor: Xen 4.5.0

Steps to produce this leak:

1.       Start a vm, win7_64_2U_vhd, for example.

2.       Stop libvirtd and start it with valgrind, using the following command:

valgrind --tool=memcheck --log-file=xs.log --leak-check=full --show-reachable=yes --track-origins=yes --trace-children=yes --verbose libvirtd -d -l

3.       Wait until libvirtd started, then reboot the vm:

virsh reboot win7_64_2U_vhd

4.       After vm restarted, kill valgind with signal 3:

kill -3 27483 (27483 is the pid of valgrind)

5.       Open xs.log, search 'definitely', we will always find this leak:

==28989== 40 bytes in 1 blocks are definitely lost in loss record 327 of 585

==28989==    at 0x4C27B9B: malloc (vg_replace_malloc.c:263)

==28989==    by 0x7EBD618: read_message (xs.c:1146)

==28989==    by 0x7EBE8C6: read_thread (xs.c:1222)

==28989==    by 0x74F8805: start_thread (in /lib64/libpthread-2.11.3.so)

==28989==    by 0x77EC66C: clone (in /lib64/libc-2.11.3.so)

When we reboot vm twice, this leak would happen twice either. Could anyone verify this and fix it?

VM XML config attached here, just for reference:
<domain type='xen'>
   <name>win7_64_2U_vhd</name>
   <memory>2097152</memory>
   <currentMemory>2097152</currentMemory>
   <vcpu>2</vcpu>
   <hotplug vcpu='disable'/>
   <on_poweroff>destroy</on_poweroff>
   <on_reboot>restart</on_reboot>
   <on_crash>destroy</on_crash>
   <viridian>1</viridian>
   <rtc_timeoffset>3600</rtc_timeoffset>
   <clock offset='utc'/>
   <keymap>en-us</keymap>
  <os>
   <type>hvm</type>
   <loader>/usr/lib/xen/boot/hvmloader</loader>
   <boot dev='hd'/>
  </os>
  <devices>
    <emulator>/usr/lib/xen/bin/qemu-system-i386 </emulator>
    <disk type='file' device='disk'>
      <driver name='file'/>
      <source file='/sdb/sgd/win7_64_2U_vhd'/>
      <target dev='xvda' bus='xen'/>
    </disk>
    <disk type='file' device='cdrom'>
     <driver name='file'/>
     <source file='/usr/bin/pvdriver_upgrade/null.iso'/>
     <target dev='xvdd'/>
     <readonly/>
    </disk>
    <graphics type='vnc' listen='0.0.0.0'/>
    <video>
      <model type='cirrus' vram='8092'/>
    </video>
</devices>
</domain>

[-- Attachment #1.2: Type: text/html, Size: 9453 bytes --]

[-- Attachment #2: Type: text/plain, Size: 126 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: Found memory leak in libxenstore

[Wei Liu]

I only had a brief look at this.

On Mon, Sep 07, 2015 at 09:52:08AM +0000, Sunguodong wrote:
> Hi,
> I found this memory leak on xen-4.5.0:
> linux-byXjTX:~ # virsh version
> Compiled against library: libvirt 1.2.17
> Using library: libvirt 1.2.17
> Using API: Xen 1.2.17
> Running hypervisor: Xen 4.5.0
> 
> Steps to produce this leak:
> 
> 1.       Start a vm, win7_64_2U_vhd, for example.
> 
> 2.       Stop libvirtd and start it with valgrind, using the following command:
> 
> valgrind --tool=memcheck --log-file=xs.log --leak-check=full --show-reachable=yes --track-origins=yes --trace-children=yes --verbose libvirtd -d -l
> 
> 3.       Wait until libvirtd started, then reboot the vm:
> 
> virsh reboot win7_64_2U_vhd
> 
> 4.       After vm restarted, kill valgind with signal 3:
> 
> kill -3 27483 (27483 is the pid of valgrind)
> 
> 5.       Open xs.log, search 'definitely', we will always find this leak:
> 
> ==28989== 40 bytes in 1 blocks are definitely lost in loss record 327 of 585
> 
> ==28989==    at 0x4C27B9B: malloc (vg_replace_malloc.c:263)
> 
> ==28989==    by 0x7EBD618: read_message (xs.c:1146)
> 
> ==28989==    by 0x7EBE8C6: read_thread (xs.c:1222)
> 
> ==28989==    by 0x74F8805: start_thread (in /lib64/libpthread-2.11.3.so)
> 
> ==28989==    by 0x77EC66C: clone (in /lib64/libc-2.11.3.so)
> 
> When we reboot vm twice, this leak would happen twice either. Could anyone verify this and fix it?

Those messages are freed after the connection is closed. Are you sure
they are still there after you close the connection?

Wei.

Re: Found memory leak in libxenstore

[Wei Liu]

On Mon, Sep 07, 2015 at 11:37:48AM +0100, Wei Liu wrote:
> I only had a brief look at this.
> 
> On Mon, Sep 07, 2015 at 09:52:08AM +0000, Sunguodong wrote:
> > Hi,
> > I found this memory leak on xen-4.5.0:
> > linux-byXjTX:~ # virsh version
> > Compiled against library: libvirt 1.2.17
> > Using library: libvirt 1.2.17
> > Using API: Xen 1.2.17
> > Running hypervisor: Xen 4.5.0
> > 
> > Steps to produce this leak:
> > 
> > 1.       Start a vm, win7_64_2U_vhd, for example.
> > 
> > 2.       Stop libvirtd and start it with valgrind, using the following command:
> > 
> > valgrind --tool=memcheck --log-file=xs.log --leak-check=full --show-reachable=yes --track-origins=yes --trace-children=yes --verbose libvirtd -d -l
> > 
> > 3.       Wait until libvirtd started, then reboot the vm:
> > 
> > virsh reboot win7_64_2U_vhd
> > 
> > 4.       After vm restarted, kill valgind with signal 3:
> > 
> > kill -3 27483 (27483 is the pid of valgrind)
> > 
> > 5.       Open xs.log, search 'definitely', we will always find this leak:
> > 
> > ==28989== 40 bytes in 1 blocks are definitely lost in loss record 327 of 585
> > 
> > ==28989==    at 0x4C27B9B: malloc (vg_replace_malloc.c:263)
> > 
> > ==28989==    by 0x7EBD618: read_message (xs.c:1146)
> > 
> > ==28989==    by 0x7EBE8C6: read_thread (xs.c:1222)
> > 
> > ==28989==    by 0x74F8805: start_thread (in /lib64/libpthread-2.11.3.so)
> > 
> > ==28989==    by 0x77EC66C: clone (in /lib64/libc-2.11.3.so)
> > 
> > When we reboot vm twice, this leak would happen twice either. Could anyone verify this and fix it?
> 
> Those messages are freed after the connection is closed. Are you sure
> they are still there after you close the connection?
> 

Another theory is that libvirt never closes the connection. I do think
this is a bit undesirable (that, messages are not freed until connection
is closed), but unfortunately I don't have time for this until 4.6 is
out.

Wei.

> Wei.

Re: Found memory leak in libxenstore

[Ian Campbell]

On Mon, 2015-09-07 at 11:37 +0100, Wei Liu wrote:
> I only had a brief look at this.
> 
> On Mon, Sep 07, 2015 at 09:52:08AM +0000, Sunguodong wrote:
> > Hi,
> > I found this memory leak on xen-4.5.0:
> > linux-byXjTX:~ # virsh version
> > Compiled against library: libvirt 1.2.17
> > Using library: libvirt 1.2.17
> > Using API: Xen 1.2.17
> > Running hypervisor: Xen 4.5.0
> > 
> > Steps to produce this leak:
> > 
> > 1.       Start a vm, win7_64_2U_vhd, for example.
> > 
> > 2.       Stop libvirtd and start it with valgrind, using the following 
> > command:
> > 
> > valgrind --tool=memcheck --log-file=xs.log --leak-check=full --show
> > -reachable=yes --track-origins=yes --trace-children=yes --verbose 
> > libvirtd -d -l
> > 
> > 3.       Wait until libvirtd started, then reboot the vm:
> > 
> > virsh reboot win7_64_2U_vhd
> > 
> > 4.       After vm restarted, kill valgind with signal 3:
> > 
> > kill -3 27483 (27483 is the pid of valgrind)
> > 
> > 5.       Open xs.log, search 'definitely', we will always find this 
> > leak:
> > 
> > ==28989== 40 bytes in 1 blocks are definitely lost in loss record 327 
> > of 585
> > 
> > ==28989==    at 0x4C27B9B: malloc (vg_replace_malloc.c:263)
> > 
> > ==28989==    by 0x7EBD618: read_message (xs.c:1146)
> > 
> > ==28989==    by 0x7EBE8C6: read_thread (xs.c:1222)
> > 
> > ==28989==    by 0x74F8805: start_thread (in /lib64/libpthread
> > -2.11.3.so)
> > 
> > ==28989==    by 0x77EC66C: clone (in /lib64/libc-2.11.3.so)
> > 
> > When we reboot vm twice, this leak would happen twice either. Could 
> > anyone verify this and fix it?
> 
> Those messages are freed after the connection is closed. Are you sure
> they are still there after you close the connection?

It seems like libvirtd would be killed ungraciously by killing valgrind in
step 4 above, so it would never close down these connections and cleanup
the memory used by them.

A better test would be to send libvirtd whatever signal causes it to exit
gracefully.

Ian.