* [Qemu-devel] [Bug?]When close VM the hugepage not freed
@ 2014-10-14 12:02 Linhaifeng
2014-10-14 12:08 ` Daniel P. Berrange
2014-10-15 7:39 ` Linhaifeng
0 siblings, 2 replies; 6+ messages in thread
From: Linhaifeng @ 2014-10-14 12:02 UTC (permalink / raw)
To: qemu-devel; +Cc: lilijun, Michael S. Tsirkin
Hi,all
I was trying to use hugepage with VM and found that the hugepage not freed when close VM.
1.Before start VM the /proc/meminfo is:
AnonHugePages: 124928 kB
HugePages_Total: 4096
HugePages_Free: 3072
HugePages_Rsvd: 0
HugePages_Surp: 0
Hugepagesize: 2048 kB
2.Start VM the /proc/meminfo is:
AnonHugePages: 139264 kB
HugePages_Total: 4096
HugePages_Free: 2048
HugePages_Rsvd: 0
HugePages_Surp: 0
Hugepagesize: 2048 kB
3.Close VM the /proc/meminfo is:
AnonHugePages: 124928 kB
HugePages_Total: 4096
HugePages_Free: 2048
HugePages_Rsvd: 0
HugePages_Surp: 0
Hugepagesize: 2048 kB
We can see there are 1024 hugepage leak!
I try to found which function used to free hugepage but i'm not sure where the qemu_ram_free is the function to free hugepage.
I found that the qemu_ram_free function not call unlink and we know unlink is used to free hugepage(see example of hugepage-mmap.c in kernel source).
void qemu_ram_free(ram_addr_t addr)
{
RAMBlock *block;
/* This assumes the iothread lock is taken here too. */
qemu_mutex_lock_ramlist();
QTAILQ_FOREACH(block, &ram_list.blocks, next) {
if (addr == block->offset) {
QTAILQ_REMOVE(&ram_list.blocks, block, next);
ram_list.mru_block = NULL;
ram_list.version++;
if (block->flags & RAM_PREALLOC) {
;
} else if (xen_enabled()) {
xen_invalidate_map_cache_entry(block->host);
#ifndef _WIN32
} else if (block->fd >= 0) {
munmap(block->host, block->length);
close(block->fd);
// should we add unlink here to free hugepage?
#endif
} else {
qemu_anon_ram_free(block->host, block->length);
}
g_free(block);
break;
}
}
qemu_mutex_unlock_ramlist();
}
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [Qemu-devel] [Bug?]When close VM the hugepage not freed
2014-10-14 12:02 [Qemu-devel] [Bug?]When close VM the hugepage not freed Linhaifeng
@ 2014-10-14 12:08 ` Daniel P. Berrange
2014-10-14 16:28 ` Michael S. Tsirkin
2014-10-15 1:32 ` Linhaifeng
2014-10-15 7:39 ` Linhaifeng
1 sibling, 2 replies; 6+ messages in thread
From: Daniel P. Berrange @ 2014-10-14 12:08 UTC (permalink / raw)
To: Linhaifeng; +Cc: Michael S. Tsirkin, qemu-devel, lilijun
On Tue, Oct 14, 2014 at 08:02:38PM +0800, Linhaifeng wrote:
> Hi,all
>
> I was trying to use hugepage with VM and found that the hugepage not freed when close VM.
>
>
> 1.Before start VM the /proc/meminfo is:
> AnonHugePages: 124928 kB
> HugePages_Total: 4096
> HugePages_Free: 3072
> HugePages_Rsvd: 0
> HugePages_Surp: 0
> Hugepagesize: 2048 kB
>
> 2.Start VM the /proc/meminfo is:
> AnonHugePages: 139264 kB
> HugePages_Total: 4096
> HugePages_Free: 2048
> HugePages_Rsvd: 0
> HugePages_Surp: 0
> Hugepagesize: 2048 kB
>
> 3.Close VM the /proc/meminfo is:
> AnonHugePages: 124928 kB
> HugePages_Total: 4096
> HugePages_Free: 2048
> HugePages_Rsvd: 0
> HugePages_Surp: 0
> Hugepagesize: 2048 kB
>
> We can see there are 1024 hugepage leak!
>
> I try to found which function used to free hugepage but i'm not sure
> where the qemu_ram_free is the function to free hugepage.
> I found that the qemu_ram_free function not call unlink and we know
> unlink is used to free hugepage(see example of hugepage-mmap.c in
> kernel source).
We can't rely on 'qemu_ram_free' ever executing because we must
ensure hugepages are freed upon QEMU crash.
It seems we should rely on UNIX filesytstem semantics and simply
unlink the memory segment the moment we create it & open the FD.
That way the kernel will automatically free it when the FD is
closed when QEMU process exits.
Regards,
Daniel
--
|: http://berrange.com -o- http://www.flickr.com/photos/dberrange/ :|
|: http://libvirt.org -o- http://virt-manager.org :|
|: http://autobuild.org -o- http://search.cpan.org/~danberr/ :|
|: http://entangle-photo.org -o- http://live.gnome.org/gtk-vnc :|
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [Qemu-devel] [Bug?]When close VM the hugepage not freed
2014-10-14 12:08 ` Daniel P. Berrange
@ 2014-10-14 16:28 ` Michael S. Tsirkin
2014-10-15 7:03 ` Daniel P. Berrange
2014-10-15 1:32 ` Linhaifeng
1 sibling, 1 reply; 6+ messages in thread
From: Michael S. Tsirkin @ 2014-10-14 16:28 UTC (permalink / raw)
To: Daniel P. Berrange; +Cc: lilijun, Linhaifeng, qemu-devel
On Tue, Oct 14, 2014 at 01:08:15PM +0100, Daniel P. Berrange wrote:
> On Tue, Oct 14, 2014 at 08:02:38PM +0800, Linhaifeng wrote:
> > Hi,all
> >
> > I was trying to use hugepage with VM and found that the hugepage not freed when close VM.
> >
> >
> > 1.Before start VM the /proc/meminfo is:
> > AnonHugePages: 124928 kB
> > HugePages_Total: 4096
> > HugePages_Free: 3072
> > HugePages_Rsvd: 0
> > HugePages_Surp: 0
> > Hugepagesize: 2048 kB
> >
> > 2.Start VM the /proc/meminfo is:
> > AnonHugePages: 139264 kB
> > HugePages_Total: 4096
> > HugePages_Free: 2048
> > HugePages_Rsvd: 0
> > HugePages_Surp: 0
> > Hugepagesize: 2048 kB
> >
> > 3.Close VM the /proc/meminfo is:
> > AnonHugePages: 124928 kB
> > HugePages_Total: 4096
> > HugePages_Free: 2048
> > HugePages_Rsvd: 0
> > HugePages_Surp: 0
> > Hugepagesize: 2048 kB
> >
> > We can see there are 1024 hugepage leak!
> >
> > I try to found which function used to free hugepage but i'm not sure
> > where the qemu_ram_free is the function to free hugepage.
> > I found that the qemu_ram_free function not call unlink and we know
> > unlink is used to free hugepage(see example of hugepage-mmap.c in
> > kernel source).
>
> We can't rely on 'qemu_ram_free' ever executing because we must
> ensure hugepages are freed upon QEMU crash.
>
> It seems we should rely on UNIX filesytstem semantics and simply
> unlink the memory segment the moment we create it & open the FD.
> That way the kernel will automatically free it when the FD is
> closed when QEMU process exits.
>
>
> Regards,
> Daniel
We being libvirt?
> --
> |: http://berrange.com -o- http://www.flickr.com/photos/dberrange/ :|
> |: http://libvirt.org -o- http://virt-manager.org :|
> |: http://autobuild.org -o- http://search.cpan.org/~danberr/ :|
> |: http://entangle-photo.org -o- http://live.gnome.org/gtk-vnc :|
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [Qemu-devel] [Bug?]When close VM the hugepage not freed
2014-10-14 12:08 ` Daniel P. Berrange
2014-10-14 16:28 ` Michael S. Tsirkin
@ 2014-10-15 1:32 ` Linhaifeng
1 sibling, 0 replies; 6+ messages in thread
From: Linhaifeng @ 2014-10-15 1:32 UTC (permalink / raw)
To: Daniel P. Berrange; +Cc: Michael S. Tsirkin, qemu-devel, lilijun
On 2014/10/14 20:08, Daniel P. Berrange wrote:
> On Tue, Oct 14, 2014 at 08:02:38PM +0800, Linhaifeng wrote:
>> Hi,all
>>
>> I was trying to use hugepage with VM and found that the hugepage not freed when close VM.
>>
>>
>> 1.Before start VM the /proc/meminfo is:
>> AnonHugePages: 124928 kB
>> HugePages_Total: 4096
>> HugePages_Free: 3072
>> HugePages_Rsvd: 0
>> HugePages_Surp: 0
>> Hugepagesize: 2048 kB
>>
>> 2.Start VM the /proc/meminfo is:
>> AnonHugePages: 139264 kB
>> HugePages_Total: 4096
>> HugePages_Free: 2048
>> HugePages_Rsvd: 0
>> HugePages_Surp: 0
>> Hugepagesize: 2048 kB
>>
>> 3.Close VM the /proc/meminfo is:
>> AnonHugePages: 124928 kB
>> HugePages_Total: 4096
>> HugePages_Free: 2048
>> HugePages_Rsvd: 0
>> HugePages_Surp: 0
>> Hugepagesize: 2048 kB
>>
>> We can see there are 1024 hugepage leak!
>>
>> I try to found which function used to free hugepage but i'm not sure
>> where the qemu_ram_free is the function to free hugepage.
>> I found that the qemu_ram_free function not call unlink and we know
>> unlink is used to free hugepage(see example of hugepage-mmap.c in
>> kernel source).
>
> We can't rely on 'qemu_ram_free' ever executing because we must
> ensure hugepages are freed upon QEMU crash.
>
> It seems we should rely on UNIX filesytstem semantics and simply
> unlink the memory segment the moment we create it & open the FD.
> That way the kernel will automatically free it when the FD is
> closed when QEMU process exits.
>
>
> Regards,
> Daniel
>
Hi, daniel
Thank you for your answer.
Does it means libvirt should free the hugepage?
QEMU create the hugepage with template file and unlink it before mmap.
Do you know why to unlink the hugepage before mmap?
When unlink the hugepage before mmap libvirt cannot found the hugepage.
How does libvirt to free the hugepage ?
Regards,
Haifeng
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [Qemu-devel] [Bug?]When close VM the hugepage not freed
2014-10-14 16:28 ` Michael S. Tsirkin
@ 2014-10-15 7:03 ` Daniel P. Berrange
0 siblings, 0 replies; 6+ messages in thread
From: Daniel P. Berrange @ 2014-10-15 7:03 UTC (permalink / raw)
To: Michael S. Tsirkin; +Cc: lilijun, Linhaifeng, qemu-devel
On Tue, Oct 14, 2014 at 07:28:39PM +0300, Michael S. Tsirkin wrote:
> On Tue, Oct 14, 2014 at 01:08:15PM +0100, Daniel P. Berrange wrote:
> > On Tue, Oct 14, 2014 at 08:02:38PM +0800, Linhaifeng wrote:
> > > Hi,all
> > >
> > > I was trying to use hugepage with VM and found that the hugepage not freed when close VM.
> > >
> > >
> > > 1.Before start VM the /proc/meminfo is:
> > > AnonHugePages: 124928 kB
> > > HugePages_Total: 4096
> > > HugePages_Free: 3072
> > > HugePages_Rsvd: 0
> > > HugePages_Surp: 0
> > > Hugepagesize: 2048 kB
> > >
> > > 2.Start VM the /proc/meminfo is:
> > > AnonHugePages: 139264 kB
> > > HugePages_Total: 4096
> > > HugePages_Free: 2048
> > > HugePages_Rsvd: 0
> > > HugePages_Surp: 0
> > > Hugepagesize: 2048 kB
> > >
> > > 3.Close VM the /proc/meminfo is:
> > > AnonHugePages: 124928 kB
> > > HugePages_Total: 4096
> > > HugePages_Free: 2048
> > > HugePages_Rsvd: 0
> > > HugePages_Surp: 0
> > > Hugepagesize: 2048 kB
> > >
> > > We can see there are 1024 hugepage leak!
> > >
> > > I try to found which function used to free hugepage but i'm not sure
> > > where the qemu_ram_free is the function to free hugepage.
> > > I found that the qemu_ram_free function not call unlink and we know
> > > unlink is used to free hugepage(see example of hugepage-mmap.c in
> > > kernel source).
> >
> > We can't rely on 'qemu_ram_free' ever executing because we must
> > ensure hugepages are freed upon QEMU crash.
> >
> > It seems we should rely on UNIX filesytstem semantics and simply
> > unlink the memory segment the moment we create it & open the FD.
> > That way the kernel will automatically free it when the FD is
> > closed when QEMU process exits.
> >
> We being libvirt?
No, QEMU is responsible for creating the mmap file, so QEMU should
take care of unlinking it after opening it.
Regards,
Daniel
--
|: http://berrange.com -o- http://www.flickr.com/photos/dberrange/ :|
|: http://libvirt.org -o- http://virt-manager.org :|
|: http://autobuild.org -o- http://search.cpan.org/~danberr/ :|
|: http://entangle-photo.org -o- http://live.gnome.org/gtk-vnc :|
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [Qemu-devel] [Bug?]When close VM the hugepage not freed
2014-10-14 12:02 [Qemu-devel] [Bug?]When close VM the hugepage not freed Linhaifeng
2014-10-14 12:08 ` Daniel P. Berrange
@ 2014-10-15 7:39 ` Linhaifeng
1 sibling, 0 replies; 6+ messages in thread
From: Linhaifeng @ 2014-10-15 7:39 UTC (permalink / raw)
To: qemu-devel; +Cc: lilijun, Michael S. Tsirkin
On 2014/10/14 20:02, Linhaifeng wrote:
> Hi,all
>
> I was trying to use hugepage with VM and found that the hugepage not freed when close VM.
>
>
> 1.Before start VM the /proc/meminfo is:
> AnonHugePages: 124928 kB
> HugePages_Total: 4096
> HugePages_Free: 3072
> HugePages_Rsvd: 0
> HugePages_Surp: 0
> Hugepagesize: 2048 kB
>
> 2.Start VM the /proc/meminfo is:
> AnonHugePages: 139264 kB
> HugePages_Total: 4096
> HugePages_Free: 2048
> HugePages_Rsvd: 0
> HugePages_Surp: 0
> Hugepagesize: 2048 kB
>
> 3.Close VM the /proc/meminfo is:
> AnonHugePages: 124928 kB
> HugePages_Total: 4096
> HugePages_Free: 2048
> HugePages_Rsvd: 0
> HugePages_Surp: 0
> Hugepagesize: 2048 kB
>
> We can see there are 1024 hugepage leak!
>
> I try to found which function used to free hugepage but i'm not sure where the qemu_ram_free is the function to free hugepage.
> I found that the qemu_ram_free function not call unlink and we know unlink is used to free hugepage(see example of hugepage-mmap.c in kernel source).
>
> void qemu_ram_free(ram_addr_t addr)
> {
> RAMBlock *block;
>
> /* This assumes the iothread lock is taken here too. */
> qemu_mutex_lock_ramlist();
> QTAILQ_FOREACH(block, &ram_list.blocks, next) {
> if (addr == block->offset) {
> QTAILQ_REMOVE(&ram_list.blocks, block, next);
> ram_list.mru_block = NULL;
> ram_list.version++;
> if (block->flags & RAM_PREALLOC) {
> ;
> } else if (xen_enabled()) {
> xen_invalidate_map_cache_entry(block->host);
> #ifndef _WIN32
> } else if (block->fd >= 0) {
> munmap(block->host, block->length);
> close(block->fd);
> // should we add unlink here to free hugepage?
> #endif
> } else {
> qemu_anon_ram_free(block->host, block->length);
> }
> g_free(block);
> break;
> }
> }
> qemu_mutex_unlock_ramlist();
>
> }
>
>
>
>
When i run the QEMU with tap backend the hugepage will be freed but not the vhost-user backend.
Maybe the vhost-user process should close the hugefile.
--
Regards,
Haifeng
^ permalink raw reply [flat|nested] 6+ messages in thread
end of thread, other threads:[~2014-10-15 7:40 UTC | newest]
Thread overview: 6+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2014-10-14 12:02 [Qemu-devel] [Bug?]When close VM the hugepage not freed Linhaifeng
2014-10-14 12:08 ` Daniel P. Berrange
2014-10-14 16:28 ` Michael S. Tsirkin
2014-10-15 7:03 ` Daniel P. Berrange
2014-10-15 1:32 ` Linhaifeng
2014-10-15 7:39 ` Linhaifeng
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).