From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from eggs.gnu.org ([2001:4830:134:3::10]:39068) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1Xg7kH-00053E-Ez for qemu-devel@nongnu.org; Mon, 20 Oct 2014 03:51:17 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1Xg7kC-0008Lx-VY for qemu-devel@nongnu.org; Mon, 20 Oct 2014 03:51:13 -0400 Received: from szxga02-in.huawei.com ([119.145.14.65]:27248) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1Xg7kB-0008LZ-UC for qemu-devel@nongnu.org; Mon, 20 Oct 2014 03:51:08 -0400 Message-ID: <5444BEBD.1020405@huawei.com> Date: Mon, 20 Oct 2014 15:50:21 +0800 From: Linhaifeng MIME-Version: 1.0 References: <1413534437-13516-1-git-send-email-haifeng.lin@huawei.com> <20141017083358.GB5874@redhat.com> <5440D9F7.5070708@huawei.com> <20141017132626.GA6628@redhat.com> <5441DC6D.2060602@huawei.com> <54446F78.8020505@cn.fujitsu.com> <5444940A.1080909@huawei.com> <54449E76.4090703@cn.fujitsu.com> <5444A8E8.8000903@huawei.com> <5444AB19.3040405@cn.fujitsu.com> In-Reply-To: <5444AB19.3040405@cn.fujitsu.com> Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: 7bit Subject: Re: [Qemu-devel] [PATCH] fix the memory leak for share hugepage List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , To: Wen Congyang , "Daniel P. Berrange" Cc: lucho@ionkov.net, ericvh@gmail.com, qemu-devel@nongnu.org, jerry.lilijun@huawei.com, "aliguori@amazon.com >> Anthony Liguori" , rminnich@sandia.gov, pbonzini@redhat.com Hi,all Maybe this is unix domain socket's bug.I found that qemu send the fd to vapp the fd's f_count inc twice in kernel. 1.kernel calls when we call send. unix_stream_sendmsg -> unix_scm_to_skb -> unix_attach_fds -> scm_fp_dup -> get_file -> atomic_long_inc(&f->f_count) Maybe should't inc the f_count when call send. 2.kernel calls when we call recv unix_stream_recvmsg -> scm_fp_dup -> get_file -> atomic_long_inc(&f->f_count) On 2014/10/20 14:26, Wen Congyang wrote: > On 10/20/2014 02:17 PM, Linhaifeng wrote: >> >> >> On 2014/10/20 13:32, Wen Congyang wrote: >>> On 10/20/2014 12:48 PM, Linhaifeng wrote: >>>> >>>> >>>> On 2014/10/20 10:12, Wen Congyang wrote: >>>>> On 10/18/2014 11:20 AM, Linhaifeng wrote: >>>>>> >>>>>> >>>>>> On 2014/10/17 21:26, Daniel P. Berrange wrote: >>>>>>> On Fri, Oct 17, 2014 at 04:57:27PM +0800, Linhaifeng wrote: >>>>>>>> >>>>>>>> >>>>>>>> On 2014/10/17 16:33, Daniel P. Berrange wrote: >>>>>>>>> On Fri, Oct 17, 2014 at 04:27:17PM +0800, haifeng.lin@huawei.com wrote: >>>>>>>>>> From: linhaifeng >>>>>>>>>> >>>>>>>>>> The VM start with share hugepage should close the hugefile fd >>>>>>>>>> when exit.Because the hugepage fd may be send to other process >>>>>>>>>> e.g vhost-user If qemu not close the fd the other process can >>>>>>>>>> not free the hugepage otherwise exit process,this is ugly,so >>>>>>>>>> qemu should close all shared fd when exit. >>>>>>>>>> >>>>>>>>>> Signed-off-by: linhaifeng >>>>>>>>> >>>>>>>>> Err, all file descriptors are closed automatically when a process >>>>>>>>> exits. So manually calling close(fd) before exit can't have any >>>>>>>>> functional effect on a resource leak. >>>>>>>>> >>>>>>>>> If QEMU has sent the FD to another process, that process has a >>>>>>>>> completely separate copy of the FD. Closing the FD in QEMU will >>>>>>>>> not close the FD in the other process. You need the other process >>>>>>>>> to exit for the copy to be closed. >>>>>>>>> >>>>>>>>> Regards, >>>>>>>>> Daniel >>>>>>>>> >>>>>>>> Hi,daniel >>>>>>>> >>>>>>>> QEMU send the fd by unix domain socket.unix domain socket just install the fd to >>>>>>>> other process and inc the f_count,if qemu not close the fd the f_count is not dec. >>>>>>>> Then the other process even close the fd the hugepage would not freed whise the other process exit. >>>>>>> >>>>>>> The kernel always closes all FDs when a process exits. So if this FD is >>>>>>> not being correctly closed then it is a kernel bug. There should never >>>>>>> be any reason for an application to do close(fd) before exiting. >>>>>>> >>>>>>> Regards, >>>>>>> Daniel >>>>>>> >>>>>> Hi,daniel >>>>>> >>>>>> I don't think this is kernel's bug.May be this a problem about usage. >>>>>> If you open a file you should close it too. >>>>> >>>>> If you don't close it, the kernel will help you when the program exits. >>>>> >>>> Yes,when the hugepage is only used for qemu,the kernel will free the file object.If the hugepage shared for other process,when qemu exit the kernel will not free the file. >>> >>> Even if the hugepage is shared with the other process, the kernel will auto close the fd when qemu >>> exits. If the kernel doesn't do it, it is a kernel bug. >>> >> Kernel supply close to fix this bug.If you call open you must call close. >> If not, the result is unpredictability. > > No, if the program exists, the kernel must close all fd used by the program. > So, there is no need to close fd before program exists. > > Thanks > Wen Congyang > >>>>>> >>>>>> This is <>about how to free resource of file. >>>>>> http://linux.die.net/man/2/close >>>>>> >>>>>> >>>>>> I'm trying to describe my problem. >>>>>> >>>>>> For example, there are 2 VMs run with hugepage and the hugepage only for QEMU to use. >>>>>> >>>>>> Before run VM the meminfo is : >>>>>> HugePages_Total: 4096 >>>>>> HugePages_Free: 4096 >>>>>> HugePages_Rsvd: 0 >>>>>> HugePages_Surp: 0 >>>>>> Hugepagesize: 2048 kB >>>>>> >>>>>> Run the two VMs.QEMU deal with hugepage as follow steps: >>>>>> 1.open >>>>>> 2.unlink >>>>>> 3.mmap >>>>>> 4.use memory of hugepage.After this step the meminfo is : >>>>>> HugePages_Total: 4096 >>>>>> HugePages_Free: 0 >>>>>> HugePages_Rsvd: 0 >>>>>> HugePages_Surp: 0 >>>>>> Hugepagesize: 2048 kB >>>>>> 5.shutdown VM with signal 15 without close(fd).After this step the meminfo is : >>>>>> HugePages_Total: 4096 >>>>>> HugePages_Free: 4096 >>>>>> HugePages_Rsvd: 0 >>>>>> HugePages_Surp: 0 >>>>>> Hugepagesize: 2048 kB >>>>>> >>>>>> Yes,it works well,like you said the kernel recycle all resources. >>>>>> >>>>>> For another example,there are 2 VMs run with hugepage and share the hugepage with vapp(a vhost-user application). >>>>> >>>>> The vapp is your internal application? >>>>> >>>> Yes vapp is a application to share the QEMU's hugepage.So threr are two process use the hugepage. >>>> >>>>>> >>>>>> Before run VM the meminfo is : >>>>>> HugePages_Total: 4096 >>>>>> HugePages_Free: 4096 >>>>>> HugePages_Rsvd: 0 >>>>>> HugePages_Surp: 0 >>>>>> Hugepagesize: 2048 kB >>>>>> >>>>>> Run the first VM.QEMU deal with hugepage as follow steps: >>>>>> 1.open >>>>>> 2.unlink >>>>>> 3.mmap >>>>>> 4.use memory of hugepage and send the fd to vapp with unix domain socket.After this step the meminfo is: >>> >>> Do you modify qemu? >>> >>>>>> HugePages_Total: 4096 >>>>>> HugePages_Free: 2048 >>>>>> HugePages_Rsvd: 0 >>>>>> HugePages_Surp: 0 >>>>>> Hugepagesize: 2048 kB >>>>>> >>>>>> Run the second VM.After this step the meminfo is: >>>>>> HugePages_Total: 4096 >>>>>> HugePages_Free: 0 >>>>>> HugePages_Rsvd: 0 >>>>>> HugePages_Surp: 0 >>>>>> Hugepagesize: 2048 kB >>>>>> >>>>>> Then I want to close the first VM and run another VM.After close the first VM and close the fd in vapp the meminfo is : >>>>>> HugePages_Total: 4096 >>>>>> HugePages_Free: 0 >>>>>> HugePages_Rsvd: 0 >>>>>> HugePages_Surp: 0 >>>>>> Hugepagesize: 2048 kB >>>>> >>>>> Does the qemu still run after you close the first VM? If the qemu exits, the fd will be closed by the kernel, so this >>>>> bug is very strange. >>>>> >>>> qemu is not run when close the first VM.If other process used the file will be closed by kernel too? >>> >>> If qeum doesn't run after the first vm is closed, the fd should be closed even if another process uses the file. >>> >>>> >>>>>> >>>>>> So failed to run the third VM because the first VM have not free the hugepage.After apply this patch the meminfo is: >>>>>> HugePages_Total: 4096 >>>>>> HugePages_Free: 2048 >>>>>> HugePages_Rsvd: 0 >>>>>> HugePages_Surp: 0 >>>>>> Hugepagesize: 2048 kB >>>>>> So i can run the third VM success. >>>>>> >>>>> >>>>> >>>>> . >>>>> >>>> >>> >>> >>> . >>> >> > > > . > -- Regards, Haifeng