From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1760788AbdEVRvU (ORCPT ); Mon, 22 May 2017 13:51:20 -0400 Received: from mx0b-001b2d01.pphosted.com ([148.163.158.5]:49740 "EHLO mx0a-001b2d01.pphosted.com" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S1760768AbdEVRvS (ORCPT ); Mon, 22 May 2017 13:51:18 -0400 Date: Mon, 22 May 2017 20:51:07 +0300 From: Mike Rapoport To: Vlastimil Babka Cc: Michal Hocko , "Kirill A. Shutemov" , Andrew Morton , Arnd Bergmann , "Kirill A. Shutemov" , Andrea Arcangeli , Pavel Emelyanov , linux-mm , lkml Subject: Re: [PATCH] mm: introduce MADV_CLR_HUGEPAGE References: <1495433562-26625-1-git-send-email-rppt@linux.vnet.ibm.com> <20170522114243.2wrdbncilozygbpl@node.shutemov.name> <20170522133559.GE27382@rapoport-lnx> <20170522135548.GA8514@dhcp22.suse.cz> <20170522142927.GG27382@rapoport-lnx> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.5.24 (2015-08-30) X-TM-AS-GCONF: 00 x-cbid: 17052217-0040-0000-0000-000003B30290 X-IBM-AV-DETECTION: SAVI=unused REMOTE=unused XFE=unused x-cbparentid: 17052217-0041-0000-0000-0000203AD24D Message-Id: <20170522175106.GA15644@rapoport-lnx> X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10432:,, definitions=2017-05-22_09:,, signatures=0 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 spamscore=0 suspectscore=2 malwarescore=0 phishscore=0 adultscore=0 bulkscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.0.1-1703280000 definitions=main-1705220094 Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On Mon, May 22, 2017 at 05:52:47PM +0200, Vlastimil Babka wrote: > On 05/22/2017 04:29 PM, Mike Rapoport wrote: > > On Mon, May 22, 2017 at 03:55:48PM +0200, Michal Hocko wrote: > >> On Mon 22-05-17 16:36:00, Mike Rapoport wrote: > >>> On Mon, May 22, 2017 at 02:42:43PM +0300, Kirill A. Shutemov wrote: > >>>> On Mon, May 22, 2017 at 09:12:42AM +0300, Mike Rapoport wrote: > >>>>> Currently applications can explicitly enable or disable THP for a memory > >>>>> region using MADV_HUGEPAGE or MADV_NOHUGEPAGE. However, once either of > >>>>> these advises is used, the region will always have > >>>>> VM_HUGEPAGE/VM_NOHUGEPAGE flag set in vma->vm_flags. > >>>>> The MADV_CLR_HUGEPAGE resets both these flags and allows managing THP in > >>>>> the region according to system-wide settings. > >>>> > >>>> Seems reasonable. But could you describe an use-case when it's useful in > >>>> real world. > >>> > >>> My use-case was combination of pre- and post-copy migration of containers > >>> with CRIU. > >>> In this case we populate a part of a memory region with data that was saved > >>> during the pre-copy stage. Afterwards, the region is registered with > >>> userfaultfd and we expect to get page faults for the parts of the region > >>> that were not yet populated. However, khugepaged collapses the pages and > >>> the page faults we would expect do not occur. > >> > >> I am not sure I undestand the problem. Do I get it right that the > >> khugepaged will effectivelly corrupt the memory by collapsing a range > >> which is not yet fully populated? If yes shouldn't that be fixed in > >> khugepaged rather than adding yet another madvise command? Also how do > >> you prevent on races? (say you VM_NOHUGEPAGE, khugepaged would be in the > >> middle of the operation and sees a collapsable vma and you get the same > >> result) > > > > Probably I didn't explained it too well. > > > > The range is intentionally not populated. When we combine pre- and > > post-copy for process migration, we create memory pre-dump without stopping > > the process, then we freeze the process without dumping the pages it has > > dirtied between pre-dump and freeze, and then, during restore, we populate > > the dirtied pages using userfaultfd. > > > > When CRIU restores a process in such scenario, it does something like: > > > > * mmap() memory region > > * fill in the pages that were collected during the pre-dump > > * do some other stuff > > * register memory region with userfaultfd > > * populate the missing memory on demand > > > > khugepaged collapses the pages in the partially populated regions before we > > have a chance to register these regions with userfaultfd, which would > > prevent the collapse. > > > > We could have used MADV_NOHUGEPAGE right after the mmap() call, and then > > there would be no race because there would be nothing for khugepaged to > > collapse at that point. But the problem is that we have no way to reset > > *HUGEPAGE flags after the memory restore is complete. > > Hmm, I wouldn't be that sure if this is indeed race-free. Check that > this scenario is indeed impossible? > > - you do the mmap > - khugepaged will choose the process' mm to scan > - khugepaged will get to the vma in question, it doesn't have > MADV_NOHUGEPAGE yet > - you set MADV_NOHUGEPAGE on the vma > - you start populating the vma > - khugepaged sees the vma is non-empty, collapses > > unless I'm wrong, the racers will have mmap_sem for reading only when > setting/checking the MADV_NOHUGEPAGE? Might be actually considered a bug. madvise(MADV_*HUGEPAGE) takes mmap_sem for writing, so it is safe. > However, can't you use prctl(PR_SET_THP_DISABLE) instead? "If arg2 has a > nonzero value, the flag is set, otherwise it is cleared." says the > manpage. Do it before the mmap and you avoid the race as well? I've missed that one, thanks Vlastimil!