* [Qemu-devel] [PATCH] fix the memory leak for share hugepage @ 2014-10-17 8:27 haifeng.lin 2014-10-17 8:33 ` Daniel P. Berrange 2014-10-17 8:43 ` zhanghailiang 0 siblings, 2 replies; 17+ messages in thread From: haifeng.lin @ 2014-10-17 8:27 UTC (permalink / raw) To: qemu-devel; +Cc: jerry.lilijun, mst From: linhaifeng <haifeng.lin@huawei.com> The VM start with share hugepage should close the hugefile fd when exit.Because the hugepage fd may be send to other process e.g vhost-user If qemu not close the fd the other process can not free the hugepage otherwise exit process,this is ugly,so qemu should close all shared fd when exit. Signed-off-by: linhaifeng <haifeng.lin@huawei.com> --- exec.c | 12 ++++++++++++ vl.c | 7 +++++++ 2 files changed, 19 insertions(+) diff --git a/exec.c b/exec.c index 759055d..d120b73 100644 --- a/exec.c +++ b/exec.c @@ -1535,6 +1535,18 @@ void qemu_ram_remap(ram_addr_t addr, ram_addr_t length) } } } + +void qemu_close_all_ram_fd(void) +{ + RAMBlock *block; + + qemu_mutex_lock_ramlist(); + QTAILQ_FOREACH(block, &ram_list.blocks, next) { + close(block->fd); + } + qemu_mutex_unlock_ramlist(); +} + #endif /* !_WIN32 */ int qemu_get_ram_fd(ram_addr_t addr) diff --git a/vl.c b/vl.c index aee73e1..0b78f3f 100644 --- a/vl.c +++ b/vl.c @@ -1658,6 +1658,7 @@ static int qemu_shutdown_requested(void) return r; } +extern void qemu_close_all_ram_fd(void); static void qemu_kill_report(void) { if (!qtest_driver() && shutdown_signal != -1) { @@ -1671,6 +1672,12 @@ static void qemu_kill_report(void) fprintf(stderr, " from pid " FMT_pid "\n", shutdown_pid); } shutdown_signal = -1; + + /* Close all ram fd when exit. If the ram is shared by othter process + * e.g vhost-user, it can free the hugepage by close fd after qemu exit, + * otherwise the process have to exit to free hugepage. + */ + qemu_close_all_ram_fd(); } } -- 1.9.0 ^ permalink raw reply related [flat|nested] 17+ messages in thread
* Re: [Qemu-devel] [PATCH] fix the memory leak for share hugepage 2014-10-17 8:27 [Qemu-devel] [PATCH] fix the memory leak for share hugepage haifeng.lin @ 2014-10-17 8:33 ` Daniel P. Berrange 2014-10-17 8:56 ` Gonglei 2014-10-17 8:57 ` Linhaifeng 2014-10-17 8:43 ` zhanghailiang 1 sibling, 2 replies; 17+ messages in thread From: Daniel P. Berrange @ 2014-10-17 8:33 UTC (permalink / raw) To: haifeng.lin; +Cc: mst, qemu-devel, jerry.lilijun On Fri, Oct 17, 2014 at 04:27:17PM +0800, haifeng.lin@huawei.com wrote: > From: linhaifeng <haifeng.lin@huawei.com> > > The VM start with share hugepage should close the hugefile fd > when exit.Because the hugepage fd may be send to other process > e.g vhost-user If qemu not close the fd the other process can > not free the hugepage otherwise exit process,this is ugly,so > qemu should close all shared fd when exit. > > Signed-off-by: linhaifeng <haifeng.lin@huawei.com> Err, all file descriptors are closed automatically when a process exits. So manually calling close(fd) before exit can't have any functional effect on a resource leak. If QEMU has sent the FD to another process, that process has a completely separate copy of the FD. Closing the FD in QEMU will not close the FD in the other process. You need the other process to exit for the copy to be closed. Regards, Daniel -- |: http://berrange.com -o- http://www.flickr.com/photos/dberrange/ :| |: http://libvirt.org -o- http://virt-manager.org :| |: http://autobuild.org -o- http://search.cpan.org/~danberr/ :| |: http://entangle-photo.org -o- http://live.gnome.org/gtk-vnc :| ^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: [Qemu-devel] [PATCH] fix the memory leak for share hugepage 2014-10-17 8:33 ` Daniel P. Berrange @ 2014-10-17 8:56 ` Gonglei 2014-10-17 9:13 ` Linhaifeng 2014-10-17 8:57 ` Linhaifeng 1 sibling, 1 reply; 17+ messages in thread From: Gonglei @ 2014-10-17 8:56 UTC (permalink / raw) To: Daniel P. Berrange; +Cc: qemu-devel, jerry.lilijun, haifeng.lin, mst On 2014/10/17 16:33, Daniel P. Berrange wrote: > On Fri, Oct 17, 2014 at 04:27:17PM +0800, haifeng.lin@huawei.com wrote: >> From: linhaifeng <haifeng.lin@huawei.com> >> >> The VM start with share hugepage should close the hugefile fd >> when exit.Because the hugepage fd may be send to other process >> e.g vhost-user If qemu not close the fd the other process can >> not free the hugepage otherwise exit process,this is ugly,so >> qemu should close all shared fd when exit. >> >> Signed-off-by: linhaifeng <haifeng.lin@huawei.com> > > Err, all file descriptors are closed automatically when a process > exits. So manually calling close(fd) before exit can't have any > functional effect on a resource leak. > > If QEMU has sent the FD to another process, that process has a > completely separate copy of the FD. Closing the FD in QEMU will > not close the FD in the other process. You need the other process > to exit for the copy to be closed. > Actually, when vhost-user close the FD manually, the hugepage leak too unless the vhost-user process exit. So, maybe the FD is not a separate copy IMHO, but simply add the ref-count of FD. When QEMU exit, because the ref is not zero, the operate system will not free the FD automatically, and when vhost-user close the FD, because of the same reason, OS will not free FD resource. BTW, I don't think this patch is good. When Qemu exit exceptionally, sush as 'by kill -9', this problem of memory leak still exist. Best Regards, -Gonglei ^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: [Qemu-devel] [PATCH] fix the memory leak for share hugepage 2014-10-17 8:56 ` Gonglei @ 2014-10-17 9:13 ` Linhaifeng 0 siblings, 0 replies; 17+ messages in thread From: Linhaifeng @ 2014-10-17 9:13 UTC (permalink / raw) To: Gonglei, Daniel P. Berrange; +Cc: jerry.lilijun, qemu-devel, mst On 2014/10/17 16:56, Gonglei wrote: > On 2014/10/17 16:33, Daniel P. Berrange wrote: > >> On Fri, Oct 17, 2014 at 04:27:17PM +0800, haifeng.lin@huawei.com wrote: >>> From: linhaifeng <haifeng.lin@huawei.com> >>> >>> The VM start with share hugepage should close the hugefile fd >>> when exit.Because the hugepage fd may be send to other process >>> e.g vhost-user If qemu not close the fd the other process can >>> not free the hugepage otherwise exit process,this is ugly,so >>> qemu should close all shared fd when exit. >>> >>> Signed-off-by: linhaifeng <haifeng.lin@huawei.com> >> >> Err, all file descriptors are closed automatically when a process >> exits. So manually calling close(fd) before exit can't have any >> functional effect on a resource leak. >> >> If QEMU has sent the FD to another process, that process has a >> completely separate copy of the FD. Closing the FD in QEMU will >> not close the FD in the other process. You need the other process >> to exit for the copy to be closed. >> > > Actually, when vhost-user close the FD manually, the hugepage leak too > unless the vhost-user process exit. So, maybe the FD is not a separate > copy IMHO, but simply add the ref-count of FD. When QEMU exit, > because the ref is not zero, the operate system will not free the FD > automatically, and when vhost-user close the FD, because of the same > reason, OS will not free FD resource. > > BTW, I don't think this patch is good. When Qemu exit exceptionally, > sush as 'by kill -9', this problem of memory leak still exist. > So,we should close qemu by 'kill -15' or close with virsh. > Best Regards, > -Gonglei > > > > > . > -- Regards, Haifeng ^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: [Qemu-devel] [PATCH] fix the memory leak for share hugepage 2014-10-17 8:33 ` Daniel P. Berrange 2014-10-17 8:56 ` Gonglei @ 2014-10-17 8:57 ` Linhaifeng 2014-10-17 9:21 ` Linhaifeng 2014-10-17 13:26 ` Daniel P. Berrange 1 sibling, 2 replies; 17+ messages in thread From: Linhaifeng @ 2014-10-17 8:57 UTC (permalink / raw) To: Daniel P. Berrange; +Cc: mst, qemu-devel, jerry.lilijun On 2014/10/17 16:33, Daniel P. Berrange wrote: > On Fri, Oct 17, 2014 at 04:27:17PM +0800, haifeng.lin@huawei.com wrote: >> From: linhaifeng <haifeng.lin@huawei.com> >> >> The VM start with share hugepage should close the hugefile fd >> when exit.Because the hugepage fd may be send to other process >> e.g vhost-user If qemu not close the fd the other process can >> not free the hugepage otherwise exit process,this is ugly,so >> qemu should close all shared fd when exit. >> >> Signed-off-by: linhaifeng <haifeng.lin@huawei.com> > > Err, all file descriptors are closed automatically when a process > exits. So manually calling close(fd) before exit can't have any > functional effect on a resource leak. > > If QEMU has sent the FD to another process, that process has a > completely separate copy of the FD. Closing the FD in QEMU will > not close the FD in the other process. You need the other process > to exit for the copy to be closed. > > Regards, > Daniel > Hi,daniel QEMU send the fd by unix domain socket.unix domain socket just install the fd to other process and inc the f_count,if qemu not close the fd the f_count is not dec. Then the other process even close the fd the hugepage would not freed whise the other process exit. I have test it for many times. -- Regards, Haifeng ^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: [Qemu-devel] [PATCH] fix the memory leak for share hugepage 2014-10-17 8:57 ` Linhaifeng @ 2014-10-17 9:21 ` Linhaifeng 2014-10-17 13:26 ` Daniel P. Berrange 1 sibling, 0 replies; 17+ messages in thread From: Linhaifeng @ 2014-10-17 9:21 UTC (permalink / raw) To: Daniel P. Berrange; +Cc: jerry.lilijun, qemu-devel, mst On 2014/10/17 16:57, Linhaifeng wrote: > > > On 2014/10/17 16:33, Daniel P. Berrange wrote: >> On Fri, Oct 17, 2014 at 04:27:17PM +0800, haifeng.lin@huawei.com wrote: >>> From: linhaifeng <haifeng.lin@huawei.com> >>> >>> The VM start with share hugepage should close the hugefile fd >>> when exit.Because the hugepage fd may be send to other process >>> e.g vhost-user If qemu not close the fd the other process can >>> not free the hugepage otherwise exit process,this is ugly,so >>> qemu should close all shared fd when exit. >>> >>> Signed-off-by: linhaifeng <haifeng.lin@huawei.com> >> >> Err, all file descriptors are closed automatically when a process >> exits. So manually calling close(fd) before exit can't have any >> functional effect on a resource leak. >> >> If QEMU has sent the FD to another process, that process has a >> completely separate copy of the FD. Closing the FD in QEMU will >> not close the FD in the other process. You need the other process >> to exit for the copy to be closed. >> >> Regards, >> Daniel >> > Hi,daniel > > QEMU send the fd by unix domain socket.unix domain socket just install the fd to > other process and inc the f_count,if qemu not close the fd the f_count is not dec. > Then the other process even close the fd the hugepage would not freed whise the other process exit. > > I have test it for many times. > The point is I want to free the hugepage when close port in vhost-user process and not exit. -- Regards, Haifeng ^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: [Qemu-devel] [PATCH] fix the memory leak for share hugepage 2014-10-17 8:57 ` Linhaifeng 2014-10-17 9:21 ` Linhaifeng @ 2014-10-17 13:26 ` Daniel P. Berrange 2014-10-18 3:20 ` Linhaifeng 1 sibling, 1 reply; 17+ messages in thread From: Daniel P. Berrange @ 2014-10-17 13:26 UTC (permalink / raw) To: Linhaifeng; +Cc: mst, qemu-devel, jerry.lilijun On Fri, Oct 17, 2014 at 04:57:27PM +0800, Linhaifeng wrote: > > > On 2014/10/17 16:33, Daniel P. Berrange wrote: > > On Fri, Oct 17, 2014 at 04:27:17PM +0800, haifeng.lin@huawei.com wrote: > >> From: linhaifeng <haifeng.lin@huawei.com> > >> > >> The VM start with share hugepage should close the hugefile fd > >> when exit.Because the hugepage fd may be send to other process > >> e.g vhost-user If qemu not close the fd the other process can > >> not free the hugepage otherwise exit process,this is ugly,so > >> qemu should close all shared fd when exit. > >> > >> Signed-off-by: linhaifeng <haifeng.lin@huawei.com> > > > > Err, all file descriptors are closed automatically when a process > > exits. So manually calling close(fd) before exit can't have any > > functional effect on a resource leak. > > > > If QEMU has sent the FD to another process, that process has a > > completely separate copy of the FD. Closing the FD in QEMU will > > not close the FD in the other process. You need the other process > > to exit for the copy to be closed. > > > > Regards, > > Daniel > > > Hi,daniel > > QEMU send the fd by unix domain socket.unix domain socket just install the fd to > other process and inc the f_count,if qemu not close the fd the f_count is not dec. > Then the other process even close the fd the hugepage would not freed whise the other process exit. The kernel always closes all FDs when a process exits. So if this FD is not being correctly closed then it is a kernel bug. There should never be any reason for an application to do close(fd) before exiting. Regards, Daniel -- |: http://berrange.com -o- http://www.flickr.com/photos/dberrange/ :| |: http://libvirt.org -o- http://virt-manager.org :| |: http://autobuild.org -o- http://search.cpan.org/~danberr/ :| |: http://entangle-photo.org -o- http://live.gnome.org/gtk-vnc :| ^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: [Qemu-devel] [PATCH] fix the memory leak for share hugepage 2014-10-17 13:26 ` Daniel P. Berrange @ 2014-10-18 3:20 ` Linhaifeng 2014-10-20 2:12 ` Wen Congyang 2014-10-20 7:54 ` Daniel P. Berrange 0 siblings, 2 replies; 17+ messages in thread From: Linhaifeng @ 2014-10-18 3:20 UTC (permalink / raw) To: Daniel P. Berrange; +Cc: mst, qemu-devel, jerry.lilijun On 2014/10/17 21:26, Daniel P. Berrange wrote: > On Fri, Oct 17, 2014 at 04:57:27PM +0800, Linhaifeng wrote: >> >> >> On 2014/10/17 16:33, Daniel P. Berrange wrote: >>> On Fri, Oct 17, 2014 at 04:27:17PM +0800, haifeng.lin@huawei.com wrote: >>>> From: linhaifeng <haifeng.lin@huawei.com> >>>> >>>> The VM start with share hugepage should close the hugefile fd >>>> when exit.Because the hugepage fd may be send to other process >>>> e.g vhost-user If qemu not close the fd the other process can >>>> not free the hugepage otherwise exit process,this is ugly,so >>>> qemu should close all shared fd when exit. >>>> >>>> Signed-off-by: linhaifeng <haifeng.lin@huawei.com> >>> >>> Err, all file descriptors are closed automatically when a process >>> exits. So manually calling close(fd) before exit can't have any >>> functional effect on a resource leak. >>> >>> If QEMU has sent the FD to another process, that process has a >>> completely separate copy of the FD. Closing the FD in QEMU will >>> not close the FD in the other process. You need the other process >>> to exit for the copy to be closed. >>> >>> Regards, >>> Daniel >>> >> Hi,daniel >> >> QEMU send the fd by unix domain socket.unix domain socket just install the fd to >> other process and inc the f_count,if qemu not close the fd the f_count is not dec. >> Then the other process even close the fd the hugepage would not freed whise the other process exit. > > The kernel always closes all FDs when a process exits. So if this FD is > not being correctly closed then it is a kernel bug. There should never > be any reason for an application to do close(fd) before exiting. > > Regards, > Daniel > Hi,daniel I don't think this is kernel's bug.May be this a problem about usage. If you open a file you should close it too. This is <<linux man page>>about how to free resource of file. http://linux.die.net/man/2/close I'm trying to describe my problem. For example, there are 2 VMs run with hugepage and the hugepage only for QEMU to use. Before run VM the meminfo is : HugePages_Total: 4096 HugePages_Free: 4096 HugePages_Rsvd: 0 HugePages_Surp: 0 Hugepagesize: 2048 kB Run the two VMs.QEMU deal with hugepage as follow steps: 1.open 2.unlink 3.mmap 4.use memory of hugepage.After this step the meminfo is : HugePages_Total: 4096 HugePages_Free: 0 HugePages_Rsvd: 0 HugePages_Surp: 0 Hugepagesize: 2048 kB 5.shutdown VM with signal 15 without close(fd).After this step the meminfo is : HugePages_Total: 4096 HugePages_Free: 4096 HugePages_Rsvd: 0 HugePages_Surp: 0 Hugepagesize: 2048 kB Yes,it works well,like you said the kernel recycle all resources. For another example,there are 2 VMs run with hugepage and share the hugepage with vapp(a vhost-user application). Before run VM the meminfo is : HugePages_Total: 4096 HugePages_Free: 4096 HugePages_Rsvd: 0 HugePages_Surp: 0 Hugepagesize: 2048 kB Run the first VM.QEMU deal with hugepage as follow steps: 1.open 2.unlink 3.mmap 4.use memory of hugepage and send the fd to vapp with unix domain socket.After this step the meminfo is: HugePages_Total: 4096 HugePages_Free: 2048 HugePages_Rsvd: 0 HugePages_Surp: 0 Hugepagesize: 2048 kB Run the second VM.After this step the meminfo is: HugePages_Total: 4096 HugePages_Free: 0 HugePages_Rsvd: 0 HugePages_Surp: 0 Hugepagesize: 2048 kB Then I want to close the first VM and run another VM.After close the first VM and close the fd in vapp the meminfo is : HugePages_Total: 4096 HugePages_Free: 0 HugePages_Rsvd: 0 HugePages_Surp: 0 Hugepagesize: 2048 kB So failed to run the third VM because the first VM have not free the hugepage.After apply this patch the meminfo is: HugePages_Total: 4096 HugePages_Free: 2048 HugePages_Rsvd: 0 HugePages_Surp: 0 Hugepagesize: 2048 kB So i can run the third VM success. -- Regards, Haifeng ^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: [Qemu-devel] [PATCH] fix the memory leak for share hugepage 2014-10-18 3:20 ` Linhaifeng @ 2014-10-20 2:12 ` Wen Congyang 2014-10-20 4:48 ` Linhaifeng 2014-10-20 7:54 ` Daniel P. Berrange 1 sibling, 1 reply; 17+ messages in thread From: Wen Congyang @ 2014-10-20 2:12 UTC (permalink / raw) To: Linhaifeng, Daniel P. Berrange; +Cc: jerry.lilijun, qemu-devel, mst On 10/18/2014 11:20 AM, Linhaifeng wrote: > > > On 2014/10/17 21:26, Daniel P. Berrange wrote: >> On Fri, Oct 17, 2014 at 04:57:27PM +0800, Linhaifeng wrote: >>> >>> >>> On 2014/10/17 16:33, Daniel P. Berrange wrote: >>>> On Fri, Oct 17, 2014 at 04:27:17PM +0800, haifeng.lin@huawei.com wrote: >>>>> From: linhaifeng <haifeng.lin@huawei.com> >>>>> >>>>> The VM start with share hugepage should close the hugefile fd >>>>> when exit.Because the hugepage fd may be send to other process >>>>> e.g vhost-user If qemu not close the fd the other process can >>>>> not free the hugepage otherwise exit process,this is ugly,so >>>>> qemu should close all shared fd when exit. >>>>> >>>>> Signed-off-by: linhaifeng <haifeng.lin@huawei.com> >>>> >>>> Err, all file descriptors are closed automatically when a process >>>> exits. So manually calling close(fd) before exit can't have any >>>> functional effect on a resource leak. >>>> >>>> If QEMU has sent the FD to another process, that process has a >>>> completely separate copy of the FD. Closing the FD in QEMU will >>>> not close the FD in the other process. You need the other process >>>> to exit for the copy to be closed. >>>> >>>> Regards, >>>> Daniel >>>> >>> Hi,daniel >>> >>> QEMU send the fd by unix domain socket.unix domain socket just install the fd to >>> other process and inc the f_count,if qemu not close the fd the f_count is not dec. >>> Then the other process even close the fd the hugepage would not freed whise the other process exit. >> >> The kernel always closes all FDs when a process exits. So if this FD is >> not being correctly closed then it is a kernel bug. There should never >> be any reason for an application to do close(fd) before exiting. >> >> Regards, >> Daniel >> > Hi,daniel > > I don't think this is kernel's bug.May be this a problem about usage. > If you open a file you should close it too. If you don't close it, the kernel will help you when the program exits. > > This is <<linux man page>>about how to free resource of file. > http://linux.die.net/man/2/close > > > I'm trying to describe my problem. > > For example, there are 2 VMs run with hugepage and the hugepage only for QEMU to use. > > Before run VM the meminfo is : > HugePages_Total: 4096 > HugePages_Free: 4096 > HugePages_Rsvd: 0 > HugePages_Surp: 0 > Hugepagesize: 2048 kB > > Run the two VMs.QEMU deal with hugepage as follow steps: > 1.open > 2.unlink > 3.mmap > 4.use memory of hugepage.After this step the meminfo is : > HugePages_Total: 4096 > HugePages_Free: 0 > HugePages_Rsvd: 0 > HugePages_Surp: 0 > Hugepagesize: 2048 kB > 5.shutdown VM with signal 15 without close(fd).After this step the meminfo is : > HugePages_Total: 4096 > HugePages_Free: 4096 > HugePages_Rsvd: 0 > HugePages_Surp: 0 > Hugepagesize: 2048 kB > > Yes,it works well,like you said the kernel recycle all resources. > > For another example,there are 2 VMs run with hugepage and share the hugepage with vapp(a vhost-user application). The vapp is your internal application? > > Before run VM the meminfo is : > HugePages_Total: 4096 > HugePages_Free: 4096 > HugePages_Rsvd: 0 > HugePages_Surp: 0 > Hugepagesize: 2048 kB > > Run the first VM.QEMU deal with hugepage as follow steps: > 1.open > 2.unlink > 3.mmap > 4.use memory of hugepage and send the fd to vapp with unix domain socket.After this step the meminfo is: > HugePages_Total: 4096 > HugePages_Free: 2048 > HugePages_Rsvd: 0 > HugePages_Surp: 0 > Hugepagesize: 2048 kB > > Run the second VM.After this step the meminfo is: > HugePages_Total: 4096 > HugePages_Free: 0 > HugePages_Rsvd: 0 > HugePages_Surp: 0 > Hugepagesize: 2048 kB > > Then I want to close the first VM and run another VM.After close the first VM and close the fd in vapp the meminfo is : > HugePages_Total: 4096 > HugePages_Free: 0 > HugePages_Rsvd: 0 > HugePages_Surp: 0 > Hugepagesize: 2048 kB Does the qemu still run after you close the first VM? If the qemu exits, the fd will be closed by the kernel, so this bug is very strange. > > So failed to run the third VM because the first VM have not free the hugepage.After apply this patch the meminfo is: > HugePages_Total: 4096 > HugePages_Free: 2048 > HugePages_Rsvd: 0 > HugePages_Surp: 0 > Hugepagesize: 2048 kB > So i can run the third VM success. > ^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: [Qemu-devel] [PATCH] fix the memory leak for share hugepage 2014-10-20 2:12 ` Wen Congyang @ 2014-10-20 4:48 ` Linhaifeng 2014-10-20 5:32 ` Wen Congyang 0 siblings, 1 reply; 17+ messages in thread From: Linhaifeng @ 2014-10-20 4:48 UTC (permalink / raw) To: Wen Congyang, Daniel P. Berrange Cc: pbonzini, qemu-devel, aliguori@amazon.com >> Anthony Liguori, jerry.lilijun On 2014/10/20 10:12, Wen Congyang wrote: > On 10/18/2014 11:20 AM, Linhaifeng wrote: >> >> >> On 2014/10/17 21:26, Daniel P. Berrange wrote: >>> On Fri, Oct 17, 2014 at 04:57:27PM +0800, Linhaifeng wrote: >>>> >>>> >>>> On 2014/10/17 16:33, Daniel P. Berrange wrote: >>>>> On Fri, Oct 17, 2014 at 04:27:17PM +0800, haifeng.lin@huawei.com wrote: >>>>>> From: linhaifeng <haifeng.lin@huawei.com> >>>>>> >>>>>> The VM start with share hugepage should close the hugefile fd >>>>>> when exit.Because the hugepage fd may be send to other process >>>>>> e.g vhost-user If qemu not close the fd the other process can >>>>>> not free the hugepage otherwise exit process,this is ugly,so >>>>>> qemu should close all shared fd when exit. >>>>>> >>>>>> Signed-off-by: linhaifeng <haifeng.lin@huawei.com> >>>>> >>>>> Err, all file descriptors are closed automatically when a process >>>>> exits. So manually calling close(fd) before exit can't have any >>>>> functional effect on a resource leak. >>>>> >>>>> If QEMU has sent the FD to another process, that process has a >>>>> completely separate copy of the FD. Closing the FD in QEMU will >>>>> not close the FD in the other process. You need the other process >>>>> to exit for the copy to be closed. >>>>> >>>>> Regards, >>>>> Daniel >>>>> >>>> Hi,daniel >>>> >>>> QEMU send the fd by unix domain socket.unix domain socket just install the fd to >>>> other process and inc the f_count,if qemu not close the fd the f_count is not dec. >>>> Then the other process even close the fd the hugepage would not freed whise the other process exit. >>> >>> The kernel always closes all FDs when a process exits. So if this FD is >>> not being correctly closed then it is a kernel bug. There should never >>> be any reason for an application to do close(fd) before exiting. >>> >>> Regards, >>> Daniel >>> >> Hi,daniel >> >> I don't think this is kernel's bug.May be this a problem about usage. >> If you open a file you should close it too. > > If you don't close it, the kernel will help you when the program exits. > Yes,when the hugepage is only used for qemu,the kernel will free the file object.If the hugepage shared for other process,when qemu exit the kernel will not free the file. >> >> This is <<linux man page>>about how to free resource of file. >> http://linux.die.net/man/2/close >> >> >> I'm trying to describe my problem. >> >> For example, there are 2 VMs run with hugepage and the hugepage only for QEMU to use. >> >> Before run VM the meminfo is : >> HugePages_Total: 4096 >> HugePages_Free: 4096 >> HugePages_Rsvd: 0 >> HugePages_Surp: 0 >> Hugepagesize: 2048 kB >> >> Run the two VMs.QEMU deal with hugepage as follow steps: >> 1.open >> 2.unlink >> 3.mmap >> 4.use memory of hugepage.After this step the meminfo is : >> HugePages_Total: 4096 >> HugePages_Free: 0 >> HugePages_Rsvd: 0 >> HugePages_Surp: 0 >> Hugepagesize: 2048 kB >> 5.shutdown VM with signal 15 without close(fd).After this step the meminfo is : >> HugePages_Total: 4096 >> HugePages_Free: 4096 >> HugePages_Rsvd: 0 >> HugePages_Surp: 0 >> Hugepagesize: 2048 kB >> >> Yes,it works well,like you said the kernel recycle all resources. >> >> For another example,there are 2 VMs run with hugepage and share the hugepage with vapp(a vhost-user application). > > The vapp is your internal application? > Yes vapp is a application to share the QEMU's hugepage.So threr are two process use the hugepage. >> >> Before run VM the meminfo is : >> HugePages_Total: 4096 >> HugePages_Free: 4096 >> HugePages_Rsvd: 0 >> HugePages_Surp: 0 >> Hugepagesize: 2048 kB >> >> Run the first VM.QEMU deal with hugepage as follow steps: >> 1.open >> 2.unlink >> 3.mmap >> 4.use memory of hugepage and send the fd to vapp with unix domain socket.After this step the meminfo is: >> HugePages_Total: 4096 >> HugePages_Free: 2048 >> HugePages_Rsvd: 0 >> HugePages_Surp: 0 >> Hugepagesize: 2048 kB >> >> Run the second VM.After this step the meminfo is: >> HugePages_Total: 4096 >> HugePages_Free: 0 >> HugePages_Rsvd: 0 >> HugePages_Surp: 0 >> Hugepagesize: 2048 kB >> >> Then I want to close the first VM and run another VM.After close the first VM and close the fd in vapp the meminfo is : >> HugePages_Total: 4096 >> HugePages_Free: 0 >> HugePages_Rsvd: 0 >> HugePages_Surp: 0 >> Hugepagesize: 2048 kB > > Does the qemu still run after you close the first VM? If the qemu exits, the fd will be closed by the kernel, so this > bug is very strange. > qemu is not run when close the first VM.If other process used the file will be closed by kernel too? >> >> So failed to run the third VM because the first VM have not free the hugepage.After apply this patch the meminfo is: >> HugePages_Total: 4096 >> HugePages_Free: 2048 >> HugePages_Rsvd: 0 >> HugePages_Surp: 0 >> Hugepagesize: 2048 kB >> So i can run the third VM success. >> > > > . > -- Regards, Haifeng ^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: [Qemu-devel] [PATCH] fix the memory leak for share hugepage 2014-10-20 4:48 ` Linhaifeng @ 2014-10-20 5:32 ` Wen Congyang 2014-10-20 6:17 ` Linhaifeng 0 siblings, 1 reply; 17+ messages in thread From: Wen Congyang @ 2014-10-20 5:32 UTC (permalink / raw) To: Linhaifeng, Daniel P. Berrange Cc: pbonzini, qemu-devel, aliguori@amazon.com >> Anthony Liguori, jerry.lilijun On 10/20/2014 12:48 PM, Linhaifeng wrote: > > > On 2014/10/20 10:12, Wen Congyang wrote: >> On 10/18/2014 11:20 AM, Linhaifeng wrote: >>> >>> >>> On 2014/10/17 21:26, Daniel P. Berrange wrote: >>>> On Fri, Oct 17, 2014 at 04:57:27PM +0800, Linhaifeng wrote: >>>>> >>>>> >>>>> On 2014/10/17 16:33, Daniel P. Berrange wrote: >>>>>> On Fri, Oct 17, 2014 at 04:27:17PM +0800, haifeng.lin@huawei.com wrote: >>>>>>> From: linhaifeng <haifeng.lin@huawei.com> >>>>>>> >>>>>>> The VM start with share hugepage should close the hugefile fd >>>>>>> when exit.Because the hugepage fd may be send to other process >>>>>>> e.g vhost-user If qemu not close the fd the other process can >>>>>>> not free the hugepage otherwise exit process,this is ugly,so >>>>>>> qemu should close all shared fd when exit. >>>>>>> >>>>>>> Signed-off-by: linhaifeng <haifeng.lin@huawei.com> >>>>>> >>>>>> Err, all file descriptors are closed automatically when a process >>>>>> exits. So manually calling close(fd) before exit can't have any >>>>>> functional effect on a resource leak. >>>>>> >>>>>> If QEMU has sent the FD to another process, that process has a >>>>>> completely separate copy of the FD. Closing the FD in QEMU will >>>>>> not close the FD in the other process. You need the other process >>>>>> to exit for the copy to be closed. >>>>>> >>>>>> Regards, >>>>>> Daniel >>>>>> >>>>> Hi,daniel >>>>> >>>>> QEMU send the fd by unix domain socket.unix domain socket just install the fd to >>>>> other process and inc the f_count,if qemu not close the fd the f_count is not dec. >>>>> Then the other process even close the fd the hugepage would not freed whise the other process exit. >>>> >>>> The kernel always closes all FDs when a process exits. So if this FD is >>>> not being correctly closed then it is a kernel bug. There should never >>>> be any reason for an application to do close(fd) before exiting. >>>> >>>> Regards, >>>> Daniel >>>> >>> Hi,daniel >>> >>> I don't think this is kernel's bug.May be this a problem about usage. >>> If you open a file you should close it too. >> >> If you don't close it, the kernel will help you when the program exits. >> > Yes,when the hugepage is only used for qemu,the kernel will free the file object.If the hugepage shared for other process,when qemu exit the kernel will not free the file. Even if the hugepage is shared with the other process, the kernel will auto close the fd when qemu exits. If the kernel doesn't do it, it is a kernel bug. >>> >>> This is <<linux man page>>about how to free resource of file. >>> http://linux.die.net/man/2/close >>> >>> >>> I'm trying to describe my problem. >>> >>> For example, there are 2 VMs run with hugepage and the hugepage only for QEMU to use. >>> >>> Before run VM the meminfo is : >>> HugePages_Total: 4096 >>> HugePages_Free: 4096 >>> HugePages_Rsvd: 0 >>> HugePages_Surp: 0 >>> Hugepagesize: 2048 kB >>> >>> Run the two VMs.QEMU deal with hugepage as follow steps: >>> 1.open >>> 2.unlink >>> 3.mmap >>> 4.use memory of hugepage.After this step the meminfo is : >>> HugePages_Total: 4096 >>> HugePages_Free: 0 >>> HugePages_Rsvd: 0 >>> HugePages_Surp: 0 >>> Hugepagesize: 2048 kB >>> 5.shutdown VM with signal 15 without close(fd).After this step the meminfo is : >>> HugePages_Total: 4096 >>> HugePages_Free: 4096 >>> HugePages_Rsvd: 0 >>> HugePages_Surp: 0 >>> Hugepagesize: 2048 kB >>> >>> Yes,it works well,like you said the kernel recycle all resources. >>> >>> For another example,there are 2 VMs run with hugepage and share the hugepage with vapp(a vhost-user application). >> >> The vapp is your internal application? >> > Yes vapp is a application to share the QEMU's hugepage.So threr are two process use the hugepage. > >>> >>> Before run VM the meminfo is : >>> HugePages_Total: 4096 >>> HugePages_Free: 4096 >>> HugePages_Rsvd: 0 >>> HugePages_Surp: 0 >>> Hugepagesize: 2048 kB >>> >>> Run the first VM.QEMU deal with hugepage as follow steps: >>> 1.open >>> 2.unlink >>> 3.mmap >>> 4.use memory of hugepage and send the fd to vapp with unix domain socket.After this step the meminfo is: Do you modify qemu? >>> HugePages_Total: 4096 >>> HugePages_Free: 2048 >>> HugePages_Rsvd: 0 >>> HugePages_Surp: 0 >>> Hugepagesize: 2048 kB >>> >>> Run the second VM.After this step the meminfo is: >>> HugePages_Total: 4096 >>> HugePages_Free: 0 >>> HugePages_Rsvd: 0 >>> HugePages_Surp: 0 >>> Hugepagesize: 2048 kB >>> >>> Then I want to close the first VM and run another VM.After close the first VM and close the fd in vapp the meminfo is : >>> HugePages_Total: 4096 >>> HugePages_Free: 0 >>> HugePages_Rsvd: 0 >>> HugePages_Surp: 0 >>> Hugepagesize: 2048 kB >> >> Does the qemu still run after you close the first VM? If the qemu exits, the fd will be closed by the kernel, so this >> bug is very strange. >> > qemu is not run when close the first VM.If other process used the file will be closed by kernel too? If qeum doesn't run after the first vm is closed, the fd should be closed even if another process uses the file. > >>> >>> So failed to run the third VM because the first VM have not free the hugepage.After apply this patch the meminfo is: >>> HugePages_Total: 4096 >>> HugePages_Free: 2048 >>> HugePages_Rsvd: 0 >>> HugePages_Surp: 0 >>> Hugepagesize: 2048 kB >>> So i can run the third VM success. >>> >> >> >> . >> > ^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: [Qemu-devel] [PATCH] fix the memory leak for share hugepage 2014-10-20 5:32 ` Wen Congyang @ 2014-10-20 6:17 ` Linhaifeng 2014-10-20 6:26 ` Wen Congyang 0 siblings, 1 reply; 17+ messages in thread From: Linhaifeng @ 2014-10-20 6:17 UTC (permalink / raw) To: Wen Congyang, Daniel P. Berrange Cc: lucho, ericvh, qemu-devel, jerry.lilijun, aliguori@amazon.com >> Anthony Liguori, rminnich, pbonzini On 2014/10/20 13:32, Wen Congyang wrote: > On 10/20/2014 12:48 PM, Linhaifeng wrote: >> >> >> On 2014/10/20 10:12, Wen Congyang wrote: >>> On 10/18/2014 11:20 AM, Linhaifeng wrote: >>>> >>>> >>>> On 2014/10/17 21:26, Daniel P. Berrange wrote: >>>>> On Fri, Oct 17, 2014 at 04:57:27PM +0800, Linhaifeng wrote: >>>>>> >>>>>> >>>>>> On 2014/10/17 16:33, Daniel P. Berrange wrote: >>>>>>> On Fri, Oct 17, 2014 at 04:27:17PM +0800, haifeng.lin@huawei.com wrote: >>>>>>>> From: linhaifeng <haifeng.lin@huawei.com> >>>>>>>> >>>>>>>> The VM start with share hugepage should close the hugefile fd >>>>>>>> when exit.Because the hugepage fd may be send to other process >>>>>>>> e.g vhost-user If qemu not close the fd the other process can >>>>>>>> not free the hugepage otherwise exit process,this is ugly,so >>>>>>>> qemu should close all shared fd when exit. >>>>>>>> >>>>>>>> Signed-off-by: linhaifeng <haifeng.lin@huawei.com> >>>>>>> >>>>>>> Err, all file descriptors are closed automatically when a process >>>>>>> exits. So manually calling close(fd) before exit can't have any >>>>>>> functional effect on a resource leak. >>>>>>> >>>>>>> If QEMU has sent the FD to another process, that process has a >>>>>>> completely separate copy of the FD. Closing the FD in QEMU will >>>>>>> not close the FD in the other process. You need the other process >>>>>>> to exit for the copy to be closed. >>>>>>> >>>>>>> Regards, >>>>>>> Daniel >>>>>>> >>>>>> Hi,daniel >>>>>> >>>>>> QEMU send the fd by unix domain socket.unix domain socket just install the fd to >>>>>> other process and inc the f_count,if qemu not close the fd the f_count is not dec. >>>>>> Then the other process even close the fd the hugepage would not freed whise the other process exit. >>>>> >>>>> The kernel always closes all FDs when a process exits. So if this FD is >>>>> not being correctly closed then it is a kernel bug. There should never >>>>> be any reason for an application to do close(fd) before exiting. >>>>> >>>>> Regards, >>>>> Daniel >>>>> >>>> Hi,daniel >>>> >>>> I don't think this is kernel's bug.May be this a problem about usage. >>>> If you open a file you should close it too. >>> >>> If you don't close it, the kernel will help you when the program exits. >>> >> Yes,when the hugepage is only used for qemu,the kernel will free the file object.If the hugepage shared for other process,when qemu exit the kernel will not free the file. > > Even if the hugepage is shared with the other process, the kernel will auto close the fd when qemu > exits. If the kernel doesn't do it, it is a kernel bug. > Kernel supply close to fix this bug.If you call open you must call close. If not, the result is unpredictability. >>>> >>>> This is <<linux man page>>about how to free resource of file. >>>> http://linux.die.net/man/2/close >>>> >>>> >>>> I'm trying to describe my problem. >>>> >>>> For example, there are 2 VMs run with hugepage and the hugepage only for QEMU to use. >>>> >>>> Before run VM the meminfo is : >>>> HugePages_Total: 4096 >>>> HugePages_Free: 4096 >>>> HugePages_Rsvd: 0 >>>> HugePages_Surp: 0 >>>> Hugepagesize: 2048 kB >>>> >>>> Run the two VMs.QEMU deal with hugepage as follow steps: >>>> 1.open >>>> 2.unlink >>>> 3.mmap >>>> 4.use memory of hugepage.After this step the meminfo is : >>>> HugePages_Total: 4096 >>>> HugePages_Free: 0 >>>> HugePages_Rsvd: 0 >>>> HugePages_Surp: 0 >>>> Hugepagesize: 2048 kB >>>> 5.shutdown VM with signal 15 without close(fd).After this step the meminfo is : >>>> HugePages_Total: 4096 >>>> HugePages_Free: 4096 >>>> HugePages_Rsvd: 0 >>>> HugePages_Surp: 0 >>>> Hugepagesize: 2048 kB >>>> >>>> Yes,it works well,like you said the kernel recycle all resources. >>>> >>>> For another example,there are 2 VMs run with hugepage and share the hugepage with vapp(a vhost-user application). >>> >>> The vapp is your internal application? >>> >> Yes vapp is a application to share the QEMU's hugepage.So threr are two process use the hugepage. >> >>>> >>>> Before run VM the meminfo is : >>>> HugePages_Total: 4096 >>>> HugePages_Free: 4096 >>>> HugePages_Rsvd: 0 >>>> HugePages_Surp: 0 >>>> Hugepagesize: 2048 kB >>>> >>>> Run the first VM.QEMU deal with hugepage as follow steps: >>>> 1.open >>>> 2.unlink >>>> 3.mmap >>>> 4.use memory of hugepage and send the fd to vapp with unix domain socket.After this step the meminfo is: > > Do you modify qemu? > >>>> HugePages_Total: 4096 >>>> HugePages_Free: 2048 >>>> HugePages_Rsvd: 0 >>>> HugePages_Surp: 0 >>>> Hugepagesize: 2048 kB >>>> >>>> Run the second VM.After this step the meminfo is: >>>> HugePages_Total: 4096 >>>> HugePages_Free: 0 >>>> HugePages_Rsvd: 0 >>>> HugePages_Surp: 0 >>>> Hugepagesize: 2048 kB >>>> >>>> Then I want to close the first VM and run another VM.After close the first VM and close the fd in vapp the meminfo is : >>>> HugePages_Total: 4096 >>>> HugePages_Free: 0 >>>> HugePages_Rsvd: 0 >>>> HugePages_Surp: 0 >>>> Hugepagesize: 2048 kB >>> >>> Does the qemu still run after you close the first VM? If the qemu exits, the fd will be closed by the kernel, so this >>> bug is very strange. >>> >> qemu is not run when close the first VM.If other process used the file will be closed by kernel too? > > If qeum doesn't run after the first vm is closed, the fd should be closed even if another process uses the file. > >> >>>> >>>> So failed to run the third VM because the first VM have not free the hugepage.After apply this patch the meminfo is: >>>> HugePages_Total: 4096 >>>> HugePages_Free: 2048 >>>> HugePages_Rsvd: 0 >>>> HugePages_Surp: 0 >>>> Hugepagesize: 2048 kB >>>> So i can run the third VM success. >>>> >>> >>> >>> . >>> >> > > > . > -- Regards, Haifeng ^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: [Qemu-devel] [PATCH] fix the memory leak for share hugepage 2014-10-20 6:17 ` Linhaifeng @ 2014-10-20 6:26 ` Wen Congyang 2014-10-20 7:50 ` Linhaifeng 0 siblings, 1 reply; 17+ messages in thread From: Wen Congyang @ 2014-10-20 6:26 UTC (permalink / raw) To: Linhaifeng, Daniel P. Berrange Cc: lucho, ericvh, qemu-devel, jerry.lilijun, aliguori@amazon.com >> Anthony Liguori, rminnich, pbonzini On 10/20/2014 02:17 PM, Linhaifeng wrote: > > > On 2014/10/20 13:32, Wen Congyang wrote: >> On 10/20/2014 12:48 PM, Linhaifeng wrote: >>> >>> >>> On 2014/10/20 10:12, Wen Congyang wrote: >>>> On 10/18/2014 11:20 AM, Linhaifeng wrote: >>>>> >>>>> >>>>> On 2014/10/17 21:26, Daniel P. Berrange wrote: >>>>>> On Fri, Oct 17, 2014 at 04:57:27PM +0800, Linhaifeng wrote: >>>>>>> >>>>>>> >>>>>>> On 2014/10/17 16:33, Daniel P. Berrange wrote: >>>>>>>> On Fri, Oct 17, 2014 at 04:27:17PM +0800, haifeng.lin@huawei.com wrote: >>>>>>>>> From: linhaifeng <haifeng.lin@huawei.com> >>>>>>>>> >>>>>>>>> The VM start with share hugepage should close the hugefile fd >>>>>>>>> when exit.Because the hugepage fd may be send to other process >>>>>>>>> e.g vhost-user If qemu not close the fd the other process can >>>>>>>>> not free the hugepage otherwise exit process,this is ugly,so >>>>>>>>> qemu should close all shared fd when exit. >>>>>>>>> >>>>>>>>> Signed-off-by: linhaifeng <haifeng.lin@huawei.com> >>>>>>>> >>>>>>>> Err, all file descriptors are closed automatically when a process >>>>>>>> exits. So manually calling close(fd) before exit can't have any >>>>>>>> functional effect on a resource leak. >>>>>>>> >>>>>>>> If QEMU has sent the FD to another process, that process has a >>>>>>>> completely separate copy of the FD. Closing the FD in QEMU will >>>>>>>> not close the FD in the other process. You need the other process >>>>>>>> to exit for the copy to be closed. >>>>>>>> >>>>>>>> Regards, >>>>>>>> Daniel >>>>>>>> >>>>>>> Hi,daniel >>>>>>> >>>>>>> QEMU send the fd by unix domain socket.unix domain socket just install the fd to >>>>>>> other process and inc the f_count,if qemu not close the fd the f_count is not dec. >>>>>>> Then the other process even close the fd the hugepage would not freed whise the other process exit. >>>>>> >>>>>> The kernel always closes all FDs when a process exits. So if this FD is >>>>>> not being correctly closed then it is a kernel bug. There should never >>>>>> be any reason for an application to do close(fd) before exiting. >>>>>> >>>>>> Regards, >>>>>> Daniel >>>>>> >>>>> Hi,daniel >>>>> >>>>> I don't think this is kernel's bug.May be this a problem about usage. >>>>> If you open a file you should close it too. >>>> >>>> If you don't close it, the kernel will help you when the program exits. >>>> >>> Yes,when the hugepage is only used for qemu,the kernel will free the file object.If the hugepage shared for other process,when qemu exit the kernel will not free the file. >> >> Even if the hugepage is shared with the other process, the kernel will auto close the fd when qemu >> exits. If the kernel doesn't do it, it is a kernel bug. >> > Kernel supply close to fix this bug.If you call open you must call close. > If not, the result is unpredictability. No, if the program exists, the kernel must close all fd used by the program. So, there is no need to close fd before program exists. Thanks Wen Congyang >>>>> >>>>> This is <<linux man page>>about how to free resource of file. >>>>> http://linux.die.net/man/2/close >>>>> >>>>> >>>>> I'm trying to describe my problem. >>>>> >>>>> For example, there are 2 VMs run with hugepage and the hugepage only for QEMU to use. >>>>> >>>>> Before run VM the meminfo is : >>>>> HugePages_Total: 4096 >>>>> HugePages_Free: 4096 >>>>> HugePages_Rsvd: 0 >>>>> HugePages_Surp: 0 >>>>> Hugepagesize: 2048 kB >>>>> >>>>> Run the two VMs.QEMU deal with hugepage as follow steps: >>>>> 1.open >>>>> 2.unlink >>>>> 3.mmap >>>>> 4.use memory of hugepage.After this step the meminfo is : >>>>> HugePages_Total: 4096 >>>>> HugePages_Free: 0 >>>>> HugePages_Rsvd: 0 >>>>> HugePages_Surp: 0 >>>>> Hugepagesize: 2048 kB >>>>> 5.shutdown VM with signal 15 without close(fd).After this step the meminfo is : >>>>> HugePages_Total: 4096 >>>>> HugePages_Free: 4096 >>>>> HugePages_Rsvd: 0 >>>>> HugePages_Surp: 0 >>>>> Hugepagesize: 2048 kB >>>>> >>>>> Yes,it works well,like you said the kernel recycle all resources. >>>>> >>>>> For another example,there are 2 VMs run with hugepage and share the hugepage with vapp(a vhost-user application). >>>> >>>> The vapp is your internal application? >>>> >>> Yes vapp is a application to share the QEMU's hugepage.So threr are two process use the hugepage. >>> >>>>> >>>>> Before run VM the meminfo is : >>>>> HugePages_Total: 4096 >>>>> HugePages_Free: 4096 >>>>> HugePages_Rsvd: 0 >>>>> HugePages_Surp: 0 >>>>> Hugepagesize: 2048 kB >>>>> >>>>> Run the first VM.QEMU deal with hugepage as follow steps: >>>>> 1.open >>>>> 2.unlink >>>>> 3.mmap >>>>> 4.use memory of hugepage and send the fd to vapp with unix domain socket.After this step the meminfo is: >> >> Do you modify qemu? >> >>>>> HugePages_Total: 4096 >>>>> HugePages_Free: 2048 >>>>> HugePages_Rsvd: 0 >>>>> HugePages_Surp: 0 >>>>> Hugepagesize: 2048 kB >>>>> >>>>> Run the second VM.After this step the meminfo is: >>>>> HugePages_Total: 4096 >>>>> HugePages_Free: 0 >>>>> HugePages_Rsvd: 0 >>>>> HugePages_Surp: 0 >>>>> Hugepagesize: 2048 kB >>>>> >>>>> Then I want to close the first VM and run another VM.After close the first VM and close the fd in vapp the meminfo is : >>>>> HugePages_Total: 4096 >>>>> HugePages_Free: 0 >>>>> HugePages_Rsvd: 0 >>>>> HugePages_Surp: 0 >>>>> Hugepagesize: 2048 kB >>>> >>>> Does the qemu still run after you close the first VM? If the qemu exits, the fd will be closed by the kernel, so this >>>> bug is very strange. >>>> >>> qemu is not run when close the first VM.If other process used the file will be closed by kernel too? >> >> If qeum doesn't run after the first vm is closed, the fd should be closed even if another process uses the file. >> >>> >>>>> >>>>> So failed to run the third VM because the first VM have not free the hugepage.After apply this patch the meminfo is: >>>>> HugePages_Total: 4096 >>>>> HugePages_Free: 2048 >>>>> HugePages_Rsvd: 0 >>>>> HugePages_Surp: 0 >>>>> Hugepagesize: 2048 kB >>>>> So i can run the third VM success. >>>>> >>>> >>>> >>>> . >>>> >>> >> >> >> . >> > ^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: [Qemu-devel] [PATCH] fix the memory leak for share hugepage 2014-10-20 6:26 ` Wen Congyang @ 2014-10-20 7:50 ` Linhaifeng 0 siblings, 0 replies; 17+ messages in thread From: Linhaifeng @ 2014-10-20 7:50 UTC (permalink / raw) To: Wen Congyang, Daniel P. Berrange Cc: lucho, ericvh, qemu-devel, jerry.lilijun, aliguori@amazon.com >> Anthony Liguori, rminnich, pbonzini Hi,all Maybe this is unix domain socket's bug.I found that qemu send the fd to vapp the fd's f_count inc twice in kernel. 1.kernel calls when we call send. unix_stream_sendmsg -> unix_scm_to_skb -> unix_attach_fds -> scm_fp_dup -> get_file -> atomic_long_inc(&f->f_count) Maybe should't inc the f_count when call send. 2.kernel calls when we call recv unix_stream_recvmsg -> scm_fp_dup -> get_file -> atomic_long_inc(&f->f_count) On 2014/10/20 14:26, Wen Congyang wrote: > On 10/20/2014 02:17 PM, Linhaifeng wrote: >> >> >> On 2014/10/20 13:32, Wen Congyang wrote: >>> On 10/20/2014 12:48 PM, Linhaifeng wrote: >>>> >>>> >>>> On 2014/10/20 10:12, Wen Congyang wrote: >>>>> On 10/18/2014 11:20 AM, Linhaifeng wrote: >>>>>> >>>>>> >>>>>> On 2014/10/17 21:26, Daniel P. Berrange wrote: >>>>>>> On Fri, Oct 17, 2014 at 04:57:27PM +0800, Linhaifeng wrote: >>>>>>>> >>>>>>>> >>>>>>>> On 2014/10/17 16:33, Daniel P. Berrange wrote: >>>>>>>>> On Fri, Oct 17, 2014 at 04:27:17PM +0800, haifeng.lin@huawei.com wrote: >>>>>>>>>> From: linhaifeng <haifeng.lin@huawei.com> >>>>>>>>>> >>>>>>>>>> The VM start with share hugepage should close the hugefile fd >>>>>>>>>> when exit.Because the hugepage fd may be send to other process >>>>>>>>>> e.g vhost-user If qemu not close the fd the other process can >>>>>>>>>> not free the hugepage otherwise exit process,this is ugly,so >>>>>>>>>> qemu should close all shared fd when exit. >>>>>>>>>> >>>>>>>>>> Signed-off-by: linhaifeng <haifeng.lin@huawei.com> >>>>>>>>> >>>>>>>>> Err, all file descriptors are closed automatically when a process >>>>>>>>> exits. So manually calling close(fd) before exit can't have any >>>>>>>>> functional effect on a resource leak. >>>>>>>>> >>>>>>>>> If QEMU has sent the FD to another process, that process has a >>>>>>>>> completely separate copy of the FD. Closing the FD in QEMU will >>>>>>>>> not close the FD in the other process. You need the other process >>>>>>>>> to exit for the copy to be closed. >>>>>>>>> >>>>>>>>> Regards, >>>>>>>>> Daniel >>>>>>>>> >>>>>>>> Hi,daniel >>>>>>>> >>>>>>>> QEMU send the fd by unix domain socket.unix domain socket just install the fd to >>>>>>>> other process and inc the f_count,if qemu not close the fd the f_count is not dec. >>>>>>>> Then the other process even close the fd the hugepage would not freed whise the other process exit. >>>>>>> >>>>>>> The kernel always closes all FDs when a process exits. So if this FD is >>>>>>> not being correctly closed then it is a kernel bug. There should never >>>>>>> be any reason for an application to do close(fd) before exiting. >>>>>>> >>>>>>> Regards, >>>>>>> Daniel >>>>>>> >>>>>> Hi,daniel >>>>>> >>>>>> I don't think this is kernel's bug.May be this a problem about usage. >>>>>> If you open a file you should close it too. >>>>> >>>>> If you don't close it, the kernel will help you when the program exits. >>>>> >>>> Yes,when the hugepage is only used for qemu,the kernel will free the file object.If the hugepage shared for other process,when qemu exit the kernel will not free the file. >>> >>> Even if the hugepage is shared with the other process, the kernel will auto close the fd when qemu >>> exits. If the kernel doesn't do it, it is a kernel bug. >>> >> Kernel supply close to fix this bug.If you call open you must call close. >> If not, the result is unpredictability. > > No, if the program exists, the kernel must close all fd used by the program. > So, there is no need to close fd before program exists. > > Thanks > Wen Congyang > >>>>>> >>>>>> This is <<linux man page>>about how to free resource of file. >>>>>> http://linux.die.net/man/2/close >>>>>> >>>>>> >>>>>> I'm trying to describe my problem. >>>>>> >>>>>> For example, there are 2 VMs run with hugepage and the hugepage only for QEMU to use. >>>>>> >>>>>> Before run VM the meminfo is : >>>>>> HugePages_Total: 4096 >>>>>> HugePages_Free: 4096 >>>>>> HugePages_Rsvd: 0 >>>>>> HugePages_Surp: 0 >>>>>> Hugepagesize: 2048 kB >>>>>> >>>>>> Run the two VMs.QEMU deal with hugepage as follow steps: >>>>>> 1.open >>>>>> 2.unlink >>>>>> 3.mmap >>>>>> 4.use memory of hugepage.After this step the meminfo is : >>>>>> HugePages_Total: 4096 >>>>>> HugePages_Free: 0 >>>>>> HugePages_Rsvd: 0 >>>>>> HugePages_Surp: 0 >>>>>> Hugepagesize: 2048 kB >>>>>> 5.shutdown VM with signal 15 without close(fd).After this step the meminfo is : >>>>>> HugePages_Total: 4096 >>>>>> HugePages_Free: 4096 >>>>>> HugePages_Rsvd: 0 >>>>>> HugePages_Surp: 0 >>>>>> Hugepagesize: 2048 kB >>>>>> >>>>>> Yes,it works well,like you said the kernel recycle all resources. >>>>>> >>>>>> For another example,there are 2 VMs run with hugepage and share the hugepage with vapp(a vhost-user application). >>>>> >>>>> The vapp is your internal application? >>>>> >>>> Yes vapp is a application to share the QEMU's hugepage.So threr are two process use the hugepage. >>>> >>>>>> >>>>>> Before run VM the meminfo is : >>>>>> HugePages_Total: 4096 >>>>>> HugePages_Free: 4096 >>>>>> HugePages_Rsvd: 0 >>>>>> HugePages_Surp: 0 >>>>>> Hugepagesize: 2048 kB >>>>>> >>>>>> Run the first VM.QEMU deal with hugepage as follow steps: >>>>>> 1.open >>>>>> 2.unlink >>>>>> 3.mmap >>>>>> 4.use memory of hugepage and send the fd to vapp with unix domain socket.After this step the meminfo is: >>> >>> Do you modify qemu? >>> >>>>>> HugePages_Total: 4096 >>>>>> HugePages_Free: 2048 >>>>>> HugePages_Rsvd: 0 >>>>>> HugePages_Surp: 0 >>>>>> Hugepagesize: 2048 kB >>>>>> >>>>>> Run the second VM.After this step the meminfo is: >>>>>> HugePages_Total: 4096 >>>>>> HugePages_Free: 0 >>>>>> HugePages_Rsvd: 0 >>>>>> HugePages_Surp: 0 >>>>>> Hugepagesize: 2048 kB >>>>>> >>>>>> Then I want to close the first VM and run another VM.After close the first VM and close the fd in vapp the meminfo is : >>>>>> HugePages_Total: 4096 >>>>>> HugePages_Free: 0 >>>>>> HugePages_Rsvd: 0 >>>>>> HugePages_Surp: 0 >>>>>> Hugepagesize: 2048 kB >>>>> >>>>> Does the qemu still run after you close the first VM? If the qemu exits, the fd will be closed by the kernel, so this >>>>> bug is very strange. >>>>> >>>> qemu is not run when close the first VM.If other process used the file will be closed by kernel too? >>> >>> If qeum doesn't run after the first vm is closed, the fd should be closed even if another process uses the file. >>> >>>> >>>>>> >>>>>> So failed to run the third VM because the first VM have not free the hugepage.After apply this patch the meminfo is: >>>>>> HugePages_Total: 4096 >>>>>> HugePages_Free: 2048 >>>>>> HugePages_Rsvd: 0 >>>>>> HugePages_Surp: 0 >>>>>> Hugepagesize: 2048 kB >>>>>> So i can run the third VM success. >>>>>> >>>>> >>>>> >>>>> . >>>>> >>>> >>> >>> >>> . >>> >> > > > . > -- Regards, Haifeng ^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: [Qemu-devel] [PATCH] fix the memory leak for share hugepage 2014-10-18 3:20 ` Linhaifeng 2014-10-20 2:12 ` Wen Congyang @ 2014-10-20 7:54 ` Daniel P. Berrange 1 sibling, 0 replies; 17+ messages in thread From: Daniel P. Berrange @ 2014-10-20 7:54 UTC (permalink / raw) To: Linhaifeng; +Cc: mst, qemu-devel, jerry.lilijun On Sat, Oct 18, 2014 at 11:20:13AM +0800, Linhaifeng wrote: > > > On 2014/10/17 21:26, Daniel P. Berrange wrote: > > On Fri, Oct 17, 2014 at 04:57:27PM +0800, Linhaifeng wrote: > >> > >> > >> On 2014/10/17 16:33, Daniel P. Berrange wrote: > >>> On Fri, Oct 17, 2014 at 04:27:17PM +0800, haifeng.lin@huawei.com wrote: > >>>> From: linhaifeng <haifeng.lin@huawei.com> > >>>> > >>>> The VM start with share hugepage should close the hugefile fd > >>>> when exit.Because the hugepage fd may be send to other process > >>>> e.g vhost-user If qemu not close the fd the other process can > >>>> not free the hugepage otherwise exit process,this is ugly,so > >>>> qemu should close all shared fd when exit. > >>>> > >>>> Signed-off-by: linhaifeng <haifeng.lin@huawei.com> > >>> > >>> Err, all file descriptors are closed automatically when a process > >>> exits. So manually calling close(fd) before exit can't have any > >>> functional effect on a resource leak. > >>> > >>> If QEMU has sent the FD to another process, that process has a > >>> completely separate copy of the FD. Closing the FD in QEMU will > >>> not close the FD in the other process. You need the other process > >>> to exit for the copy to be closed. > >>> > >>> Regards, > >>> Daniel > >>> > >> Hi,daniel > >> > >> QEMU send the fd by unix domain socket.unix domain socket just install the fd to > >> other process and inc the f_count,if qemu not close the fd the f_count is not dec. > >> Then the other process even close the fd the hugepage would not freed whise the other process exit. > > > > The kernel always closes all FDs when a process exits. So if this FD is > > not being correctly closed then it is a kernel bug. There should never > > be any reason for an application to do close(fd) before exiting. > > > > Regards, > > Daniel > > > Hi,daniel > > I don't think this is kernel's bug.May be this a problem about usage. > If you open a file you should close it too. No, the standard UNIX semantics are that the kernel will close all file descriptors when a process exits. There is *no* requirement to manually close any file descriptors before exiting. The only time you need to close files is if you are using a higher level API (eg C library "FILE *" ) and you need to explicitly flush any buffered I/O operations before exiting. This doesn't apply in the case of huge pages. > Run the first VM.QEMU deal with hugepage as follow steps: > 1.open > 2.unlink > 3.mmap > 4.use memory of hugepage and send the fd to vapp with unix domain socket.After this step the meminfo is: If the QEMU process has exited then the kernel has closed the FD that QEMU had open. The logical implication is that the 'vapp' process still has its copy of the file descriptor open even after QEMU has exited. Another possibility is that whatever launched QEMU has not done a waitpid and thus leaving QEMU in a zombie state where file descriptors are not yet cleaned up. Regards, Daniel -- |: http://berrange.com -o- http://www.flickr.com/photos/dberrange/ :| |: http://libvirt.org -o- http://virt-manager.org :| |: http://autobuild.org -o- http://search.cpan.org/~danberr/ :| |: http://entangle-photo.org -o- http://live.gnome.org/gtk-vnc :| ^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: [Qemu-devel] [PATCH] fix the memory leak for share hugepage 2014-10-17 8:27 [Qemu-devel] [PATCH] fix the memory leak for share hugepage haifeng.lin 2014-10-17 8:33 ` Daniel P. Berrange @ 2014-10-17 8:43 ` zhanghailiang 2014-10-18 3:22 ` Linhaifeng 1 sibling, 1 reply; 17+ messages in thread From: zhanghailiang @ 2014-10-17 8:43 UTC (permalink / raw) To: haifeng.lin, qemu-devel; +Cc: jerry.lilijun, mst On 2014/10/17 16:27, haifeng.lin@huawei.com wrote: > From: linhaifeng <haifeng.lin@huawei.com> > > The VM start with share hugepage should close the hugefile fd > when exit.Because the hugepage fd may be send to other process > e.g vhost-user If qemu not close the fd the other process can > not free the hugepage otherwise exit process,this is ugly,so > qemu should close all shared fd when exit. > > Signed-off-by: linhaifeng <haifeng.lin@huawei.com> > --- > exec.c | 12 ++++++++++++ > vl.c | 7 +++++++ > 2 files changed, 19 insertions(+) > > diff --git a/exec.c b/exec.c > index 759055d..d120b73 100644 > --- a/exec.c > +++ b/exec.c > @@ -1535,6 +1535,18 @@ void qemu_ram_remap(ram_addr_t addr, ram_addr_t length) > } > } > } > + > +void qemu_close_all_ram_fd(void) > +{ > + RAMBlock *block; > + > + qemu_mutex_lock_ramlist(); > + QTAILQ_FOREACH(block, &ram_list.blocks, next) { > + close(block->fd); > + } > + qemu_mutex_unlock_ramlist(); > +} > + > #endif /* !_WIN32 */ > > int qemu_get_ram_fd(ram_addr_t addr) > diff --git a/vl.c b/vl.c > index aee73e1..0b78f3f 100644 > --- a/vl.c > +++ b/vl.c > @@ -1658,6 +1658,7 @@ static int qemu_shutdown_requested(void) > return r; > } > > +extern void qemu_close_all_ram_fd(void); > static void qemu_kill_report(void) > { > if (!qtest_driver() && shutdown_signal != -1) { > @@ -1671,6 +1672,12 @@ static void qemu_kill_report(void) > fprintf(stderr, " from pid " FMT_pid "\n", shutdown_pid); > } > shutdown_signal = -1; > + > + /* Close all ram fd when exit. If the ram is shared by othter process s/othter/other/ > + * e.g vhost-user, it can free the hugepage by close fd after qemu exit, > + * otherwise the process have to exit to free hugepage. > + */ > + qemu_close_all_ram_fd(); > } > } > > ^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: [Qemu-devel] [PATCH] fix the memory leak for share hugepage 2014-10-17 8:43 ` zhanghailiang @ 2014-10-18 3:22 ` Linhaifeng 0 siblings, 0 replies; 17+ messages in thread From: Linhaifeng @ 2014-10-18 3:22 UTC (permalink / raw) To: zhanghailiang, qemu-devel; +Cc: jerry.lilijun, mst On 2014/10/17 16:43, zhanghailiang wrote: > On 2014/10/17 16:27, haifeng.lin@huawei.com wrote: >> From: linhaifeng <haifeng.lin@huawei.com> >> >> The VM start with share hugepage should close the hugefile fd >> when exit.Because the hugepage fd may be send to other process >> e.g vhost-user If qemu not close the fd the other process can >> not free the hugepage otherwise exit process,this is ugly,so >> qemu should close all shared fd when exit. >> >> Signed-off-by: linhaifeng <haifeng.lin@huawei.com> >> --- >> exec.c | 12 ++++++++++++ >> vl.c | 7 +++++++ >> 2 files changed, 19 insertions(+) >> >> diff --git a/exec.c b/exec.c >> index 759055d..d120b73 100644 >> --- a/exec.c >> +++ b/exec.c >> @@ -1535,6 +1535,18 @@ void qemu_ram_remap(ram_addr_t addr, ram_addr_t length) >> } >> } >> } >> + >> +void qemu_close_all_ram_fd(void) >> +{ >> + RAMBlock *block; >> + >> + qemu_mutex_lock_ramlist(); >> + QTAILQ_FOREACH(block, &ram_list.blocks, next) { >> + close(block->fd); >> + } >> + qemu_mutex_unlock_ramlist(); >> +} >> + >> #endif /* !_WIN32 */ >> >> int qemu_get_ram_fd(ram_addr_t addr) >> diff --git a/vl.c b/vl.c >> index aee73e1..0b78f3f 100644 >> --- a/vl.c >> +++ b/vl.c >> @@ -1658,6 +1658,7 @@ static int qemu_shutdown_requested(void) >> return r; >> } >> >> +extern void qemu_close_all_ram_fd(void); >> static void qemu_kill_report(void) >> { >> if (!qtest_driver() && shutdown_signal != -1) { >> @@ -1671,6 +1672,12 @@ static void qemu_kill_report(void) >> fprintf(stderr, " from pid " FMT_pid "\n", shutdown_pid); >> } >> shutdown_signal = -1; >> + >> + /* Close all ram fd when exit. If the ram is shared by othter process > > s/othter/other/ > OK.thank you. >> + * e.g vhost-user, it can free the hugepage by close fd after qemu exit, >> + * otherwise the process have to exit to free hugepage. >> + */ >> + qemu_close_all_ram_fd(); >> } >> } >> >> > > > > . > -- Regards, Haifeng ^ permalink raw reply [flat|nested] 17+ messages in thread
end of thread, other threads:[~2014-10-20 7:55 UTC | newest] Thread overview: 17+ messages (download: mbox.gz follow: Atom feed -- links below jump to the message on this page -- 2014-10-17 8:27 [Qemu-devel] [PATCH] fix the memory leak for share hugepage haifeng.lin 2014-10-17 8:33 ` Daniel P. Berrange 2014-10-17 8:56 ` Gonglei 2014-10-17 9:13 ` Linhaifeng 2014-10-17 8:57 ` Linhaifeng 2014-10-17 9:21 ` Linhaifeng 2014-10-17 13:26 ` Daniel P. Berrange 2014-10-18 3:20 ` Linhaifeng 2014-10-20 2:12 ` Wen Congyang 2014-10-20 4:48 ` Linhaifeng 2014-10-20 5:32 ` Wen Congyang 2014-10-20 6:17 ` Linhaifeng 2014-10-20 6:26 ` Wen Congyang 2014-10-20 7:50 ` Linhaifeng 2014-10-20 7:54 ` Daniel P. Berrange 2014-10-17 8:43 ` zhanghailiang 2014-10-18 3:22 ` Linhaifeng
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for NNTP newsgroup(s).