From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S967891AbdEXOZU (ORCPT ); Wed, 24 May 2017 10:25:20 -0400 Received: from mail-db5eur01on0114.outbound.protection.outlook.com ([104.47.2.114]:13376 "EHLO EUR01-DB5-obe.outbound.protection.outlook.com" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S934677AbdEXOZP (ORCPT ); Wed, 24 May 2017 10:25:15 -0400 Authentication-Results: vger.kernel.org; dkim=none (message not signed) header.d=none;vger.kernel.org; dmarc=none action=none header.from=virtuozzo.com; Subject: Re: [PATCH] mm: introduce MADV_CLR_HUGEPAGE To: Michal Hocko , Mike Rapoport References: <1495433562-26625-1-git-send-email-rppt@linux.vnet.ibm.com> <20170522114243.2wrdbncilozygbpl@node.shutemov.name> <20170522133559.GE27382@rapoport-lnx> <20170522135548.GA8514@dhcp22.suse.cz> <20170522142927.GG27382@rapoport-lnx> <20170524075043.GB3063@rapoport-lnx> <20170524103947.GC3063@rapoport-lnx> <20170524111800.GD14733@dhcp22.suse.cz> CC: Vlastimil Babka , "Kirill A. Shutemov" , Andrew Morton , Arnd Bergmann , "Kirill A. Shutemov" , Andrea Arcangeli , linux-mm , lkml , Linux API From: Pavel Emelyanov Message-ID: <45c88ac2-9fc3-2740-c54d-82b0be2d1c9f@virtuozzo.com> Date: Wed, 24 May 2017 17:25:05 +0300 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:45.0) Gecko/20100101 Thunderbird/45.8.0 MIME-Version: 1.0 In-Reply-To: <20170524111800.GD14733@dhcp22.suse.cz> Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: 7bit X-Originating-IP: [83.220.239.69] X-ClientProxiedBy: VI1PR06CA0036.eurprd06.prod.outlook.com (10.162.116.174) To VI1PR0802MB2142.eurprd08.prod.outlook.com (10.172.12.11) X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: VI1PR0802MB2142: X-MS-Office365-Filtering-Correlation-Id: a58e920d-715a-47e2-e27b-08d4a2b0b081 X-Microsoft-Antispam: UriScan:;BCL:0;PCL:0;RULEID:(22001)(201703131423075)(201703031133081);SRVR:VI1PR0802MB2142; X-Microsoft-Exchange-Diagnostics: 1;VI1PR0802MB2142;3:glgA8aypz99HcNwItrN3hLUCZWVxbiQIpiwokkPLvYUTe/muQ43bQa781Px0Qn5YrGEi2FGqfxBFXroVj/PAPkqQiqid52WKwsOZw41oRWcb5EkcDozX5IU2GQVoeSIeP9dwg86d2877LjIcW+rI4LWAN9lODaxLbUd8jQYOKc2ozQ/Z8TAhjwkL55sVCrinC6vCozGXnKf0hQsCIZ1o932PW22EooRJtwhDZT24ZjnVyOs4eGjdcaMZuFQwoGId/48ZxDRgyNKYkBBFRxvlhwdEA5TKE1dufwpvEiVARk2bpFZNhxwdZYK7d7FdgC5FDalS5U9e7u8Zqfk5a/WqyA==;25:2tbg6zz3ASbGi7sgigelPUsGc5lm1shuYbQd+WqKnfMeAZZ+pWD2Fjc2MZWszFuhyfAhj0xwm3oGBy20e/5/gX/ouauG/yZNqS6nLQdY5AiyfSqjfWTdKDUW1bSkCJq9HjbdVoklc6cha1kOAn9OyQqnpcC9bLmUt4EM2FzX8pYs+KTLgx9laqTlxgrGRtFj6HEPRMzvKd5lTCdrHXav9rYHEPFuE57PonI0Cx26J4l9d980HHD59TwAWXDtgoT0LpJnqGLLTgDHKmeWkau5hvcoUijkXV9581qnR3sJLDYgDL3NXr2UOXqkAsjen4SEseWCdQWptiNRgrL8UdREDeLUUI2FEsJV/j28cqJClJPfx91TXFLBQqcRrDw3HdTneJIL7NnPfsX0uysk/mTkYUaM2+z8fA2Uw4rCgHhpyFmhOaNiCsmKuAEs2NGvdVhGXVS90lIWi9DLHJOlPPzKZRCWdu5Lsb5isekSBZw4SwM= X-Microsoft-Exchange-Diagnostics: 1;VI1PR0802MB2142;31:x4CMNSlWw8G1o1VE2M+JrZdv9OfdLQUfvbLky4hvyc0eNp68/XsLc3Ww6KHpQKOEEcBZQa9/Pnti1G/gDNDVYdgmM80sbflMhWVEYHwFEZGFqUFqQEA7yl8n//0LxBS9h9t8+8x7Yj8wERAyuqVc8TozoVa6OeGKUZFtPMCoYDsQoZLdYP7z7oduuz2Q/qI/K7Qek9GPiiC+GeI3I0r6jzUV1MnlhZNMzdC+Ol6PZpePkCXBWU7gwZ9+Eyd+g770brBDy63PH2ybRbDRntemQA==;20:fFyh4lphLqhIWyOVSgCvqNMlKp8jkhAQPMJpmDNAQnFeq38nA+jra9HlU0/pFk2Gr3YkdEadP0yMUHgGn2I4yC5J+MniSVdQaGXq+9QTPUW5H/lAJodcpBS9R7oCJiM2dXHmcCTc2rksuO+Dt5ztbIZ8ln0PAfljCko6yg37/suGfufyPhTEWy6qg/d3F0UwGxWmH+oZ3KGwmykiZI/n8mpwbupp6cN9Xyctnow00tdIdSBiarZDbOsz/mqNuu/Sa/fd9OZxzf2FUXgbK9AQtq2EZSbf7T7ZricijDP6wqdcEPgiekn/xAJlXDafFNffvlF0hL1BFC7IyGeIG4AHkRaJlCH+zH6v7XY3FeLgsQCAfkYcwlNtT8b2J30xkR+Xo2SGsFxFyoi9AwTgRlT1UOf8v4CgsrIdjoiXNwihRIE= X-Microsoft-Antispam-PRVS: X-Exchange-Antispam-Report-Test: UriScan:; X-Exchange-Antispam-Report-CFA-Test: BCL:0;PCL:0;RULEID:(6040450)(601004)(2401047)(5005006)(8121501046)(3002001)(93006095)(93001095)(10201501046)(6041248)(20161123564025)(20161123555025)(201703131423075)(201702281528075)(201703061421075)(201703061406153)(20161123558100)(20161123562025)(20161123560025)(6072148);SRVR:VI1PR0802MB2142;BCL:0;PCL:0;RULEID:;SRVR:VI1PR0802MB2142; X-Microsoft-Exchange-Diagnostics: 1;VI1PR0802MB2142;4:XIr+qNpMymr7KC9xrJr+OKxezaYmrM6Q22pcBDZRtwngL2D/K7lNsTyiBuMbd/8rjz1EYLYFh4x/Y6CJOxQ2WM9GFDBhiUMdKQY8Br1wEr5LUE+AdGCBKUl3KB8W47BOLnLX0/bz45IX8kvglDmMkhCieUzDFyA8Jg/fmRIK4SGvtwWyc4GP1pLYrYTnHzxXjAWSUM39oWDbf2/aH3wNzavOVlPHqebXGpwU5yMhuHqy3WqJgpeOQTiH8PJXtj5MedXVR+jDqYTWJ7Rmw5cYnlysz2sVsHqnv/0q86C2v21UQIs0ZsL3meg6WgEiHhJW0egKp28dt8ueBaWFUXdvMv79GXK3W+Ot0/nBb1vr+LgZ5ypXi0yIHKe7uKAYyEf3E9fZEL3W681WcGgD1NFSq+QNpC0jpEHhuRfLOLYmg3LGphoTu/yoIzlSsl64fuq9v5MZFdZIUGUDXkYE2wszCMfmgMj1Gzxi+O16YQH5WX73HhR+gwK0Yny4BSsqZ2AhRIJpYh5fu8Dm7CJZ5HqPKXCn+sr20mt5pJ6M4d9lrh7qgLB9E4kea6MPeWAgpsTT/iC4ZU0ZmG+CrvhpgoyasHOCqEhS60LbKzkx+/VhjdGc9CBrT6X9ZRPLOv2T+yEKZhY8cTtO+3SXIM0CjwlzNOSdV0081IVaN5aw0B0l3s36oTWco+8no18TKFaAmF3KvuvC2UlaiHzBYivbrQ/itz7Alg6JUHcjV4ST/2KBzooavtfO/L5bSIiAnyZttR3Ur0Xl7Wdv11x6MEI+xRQNDvTJB31klh9UZ/nqWipKfU8= X-Forefront-PRVS: 031763BCAF X-Forefront-Antispam-Report: SFV:NSPM;SFS:(10019020)(4630300001)(6049001)(6009001)(39840400002)(39450400003)(39400400002)(39410400002)(377454003)(377424004)(24454002)(2906002)(229853002)(7736002)(8676002)(66066001)(65956001)(65806001)(4001350100001)(3846002)(81166006)(36756003)(189998001)(47776003)(6666003)(7416002)(478600001)(305945005)(2950100002)(5660300001)(23676002)(65826007)(6116002)(83506001)(77096006)(90366009)(6486002)(31696002)(230700001)(50466002)(4326008)(53936002)(25786009)(53546009)(54906002)(50986999)(76176999)(54356999)(33646002)(6246003)(117156002)(31686004)(86362001)(38730400002)(42186005)(42262002);DIR:OUT;SFP:1102;SCL:1;SRVR:VI1PR0802MB2142;H:[192.168.43.229];FPR:;SPF:None;MLV:sfv;LANG:en; X-Microsoft-Exchange-Diagnostics: =?utf-8?B?MTtWSTFQUjA4MDJNQjIxNDI7MjM6WDE3QkRQekhsK1kzWkhuSFZjNjBIMUZo?= =?utf-8?B?U2JiS2diMW44SFMycW1QKzdPNVo3ajM5Q0lhQ2FzTGUrNlRrTGY5YzBYbTBo?= =?utf-8?B?K1hIcHVMZ280WjYvemFlc2ZYdnoxVkFMR1hvTG10RklkQjFEaHh0MVB4Vnkz?= =?utf-8?B?SkxyM2U3VGgwbmM4MkttV1cwcDMzV0ROS3o2bi9PWDVneXY1amJ2SWlQN1BY?= =?utf-8?B?VTlaclFmVmJjNnUweDV1ejBJQnhpT3hISzVJMzhuTFdhSmdMSUM2N0g1enRk?= =?utf-8?B?czN2REY2bm9QYktTTExwSGxnS2lKYjlJOUZqT2hwc053WmlrMmgycEtuUlVS?= =?utf-8?B?NlJOT1dUNDNGZG5oZU52dnVoWG1WT0JpWDhZeHZHcHMxYlE4TGFDcmpoNnhO?= =?utf-8?B?eGo5TjZpSU1xd2F4NTl5aStVSmxzNUg0TW0wUmt1ZVYwL3RLQytjQnFtZGVw?= =?utf-8?B?MFAzN2x6V0xEMk1pNnQ1RXl5bDNUTkFRMnpxS0FsYndhMGtNdkx4azBEZm5D?= =?utf-8?B?YllPaXZKRUJORFFTN1ppaHFhNWdxaU16Tjl3TjVucTMwVHA1NnZRMXNTZmpH?= =?utf-8?B?NHhHMWFZaWp1dk9Va1h1OE94KzZZOUQwUmd0NmhFUVZBZFZHQlNhSGhHMGE4?= =?utf-8?B?ck9UYkRxQU8rVW1uZEQyWVRXVHY4cnJvMGRIWFVEeU5mWlhIbG1adzhWZ3ZN?= =?utf-8?B?UWhlMjhOb0NJUzZOL0ZSb2tVSENPcWN5VUVmNjljc3ZoaVdrZUI5SWFKTmRL?= =?utf-8?B?T3VyL3VsTzV0L003ekJXZHdyeHFhM0pYWHRsTVBQaHJqTFJDUlkreFFrSldu?= =?utf-8?B?YjA2NGNHam9Ma2ttNUZ4d1J2bGlKWjZnSTg3STlXKzE5UnFPak9xbHZXSnVH?= =?utf-8?B?YkdxbVZiMDNkZlN1eXVxanBWVEtZaHMzMllHUVJSaUpBY1o0NVVVZ1JiQkl6?= =?utf-8?B?a0h4V25CcVo3MUwrK0Zac25qdVhTWVRMS0RqblVJNEZEY0hDOTdBL1BOR1VR?= =?utf-8?B?RnhmaG1qWTFmV3JwdXhMU2xSUE5aNm1oQ0JtSngyKzk0YW9QbDcrbGtydUVJ?= =?utf-8?B?WDdiYSs0NGtzUDR6OG5YbDBaL29VbmVHTEg4bTQ0N3d1c3BrejdyWnhDU0RI?= =?utf-8?B?VTBSUFZub0RCaGRQT0JCY05Qc2NSNjFKdE1sc3dwZEc0M2MvcStVdEM3OFhW?= =?utf-8?B?dGJmRG5HUFJkMFc0eURXYkhSTWE5UVVpQjRGQzV0dFZvQW80cDA5RDM3T3Fa?= =?utf-8?B?UVZFOXNWY0MxYTZIUE9yNjhmVFB0LzdxTEorRVpiMmZMMko1UjdON2ZrS1c0?= =?utf-8?B?NGl2Z3RuT05SNlJaVGhFb2t3MVd3R2VmMmE5TS9KQnFIZHJzU3FWaHFod1Z4?= =?utf-8?B?YUliRkZjRlZLZjN4ZnpuaEYyZG5ER09WS0ZXSlVQSFQ5R2RxUGJ6TDFoUXdl?= =?utf-8?B?RTlUT1Z4bXYvVCtNb2hIME1oVk80M1Z6L1JHQW84Y3ZKa3MxQ3ZhamRCZXVW?= =?utf-8?B?bHNMcmZtNk4xK0Y5SzMxM3JBUmRrVzRQODNITUZWS1g3WWEzRGJ6VFUwZDQw?= =?utf-8?B?TEk5Sk50aGU5enRxcytqcUw3MHJXMEVyaE9pcExyWjlTMkJhSERDTkc4TnEr?= =?utf-8?B?SjBBTnhLRDRxM0FXUmFITnVqeFcxT1ZWSFdrU0lXSTE4MXB0aU4weVZPek0x?= =?utf-8?B?L3lGREdVRG1kY0lWYUhpZzg5ZHVDVTY1M3loWjVIb0JDTXlZOGIyS0ZneHRq?= =?utf-8?B?UEM1dHVPeVZvR3lSTHlIM01XdCsrRG84K3JKWFFhV29SMWNiUVdnVVAvbjlm?= =?utf-8?B?Q2ZxOThmMDEyZjZPb3BzeU1NZjVaT3VaNlhCeWl6QkVpMmVMUT09?= X-Microsoft-Exchange-Diagnostics: 1;VI1PR0802MB2142;6:TWVZK65CPr790kc+k+nglyQzdPbifgABLUVdntRe+TNHq0BGgUT045t+20BJOl4D1EIymgrXw4RDJ8T4ZJToCE246fH5U0ZQdZyi1wHccZdnk1OxF3Apb8j2tS2Bu8JNtkqgRHhen9qdxkbQvOeltBM0dicQ5NT7GEGLDbrBDueArWXvR6OmglVhB9uvM4yztCUmEw6YFE39t+DYGu1BKmDsh2hGrAtX1AucGDaMcFjo3OOGLsIPT79Hvv4Iq7tGdJ1tIajLleGEHepqEsjq4BCLp7a1gr3Mt9HJ+kujEMobnKTHjlpx8JhyGKmqeD1UkuuKemlRIywd5M74fyZcuf7gPdu/aTuYo0RxR+6Z1b7DI2n+RiZDnS6WOGHwGHU9yksQYCHpbDl7yy/KRt5pZZ8McZpjhhb136anrs+dMdauEr4zrf1r8x7u9Ae3Yr02TyWBI4V2I1YIMARROKAQnVwi+utKQ/UNbjiog2Kz6gseVq0anpOZ6bKe3kGJvYHJodU3+faukOoPaqSslR1mGw==;5:Hwg/DC2i6OwqMTlVvb7I2+zUIySWA0CDLAg9V9MTbPHEBejAawOw0zyQhSojy/6xyd2yE1jG5iBNYM7kohPDBl9qrsFitqPMvEAwHnt4IJ0aEz3+JPoy5BVT/mgVqcg8d1LLv4goLlqgYT4uuNlCik5iyG2r8mYl7r0nDv/xhPg=;24:QgG/qGK8U8y7BzT5C++XNAUyantVWlQxDGOe1U94c2QqfcY5j4ooMYxULUr4kPo0ozUJd7l588DZH77rQ2q65xBmRiuEt4oX/aXqkQ+Jr3w= SpamDiagnosticOutput: 1:99 SpamDiagnosticMetadata: NSPM X-Microsoft-Exchange-Diagnostics: 1;VI1PR0802MB2142;7:4G5JUl1+ALg2UBf4IAKEERp3h0WdzV+abFHIHoShslENK7ksZuc14Y2SQuzLaCHh7L/CJ8xTpccx3M/sRONn4IBH9R6PQRLpVf4n/7X1GAXw9NC7htILk+g044imtlR9cwTLcVqoe5Wr7PNWIrodKUe86XA8hlX4xlYnBiadeJSJ02azJTpq1dgCumQdsZMxWatEzBqDinmsGbJIjSDaaPYP/0J9Op9R3c/D0N29Zk125ecTiQOFeixyII4by4iNXR+67G3AoCsSdKjBCJVg/JzX2OPi8D9vdKDzRSEGTzpygTAWYDrv47VUPKNEp8LEGGVRRBoulUbA48E0tXZx8w==;20:yJcIqr1tTxUOg35Rdc/v3GMGfmw5kZLthP/NulmqwtRizcw5z+P6pXt1xKQuLbU7sSa0zDX3uwLgSUQrZU0YHyIP5NmcU04vUtvD3o3kmCAYhm7a+US+nu+VTkqYhHPJ4zm0wGRaUugKYvOx0J5qUtGCBGS/6mco3yOWkfgofMA= X-OriginatorOrg: virtuozzo.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 24 May 2017 14:25:11.2287 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-Transport-CrossTenantHeadersStamped: VI1PR0802MB2142 Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On 05/24/2017 02:18 PM, Michal Hocko wrote: > On Wed 24-05-17 13:39:48, Mike Rapoport wrote: >> On Wed, May 24, 2017 at 09:58:06AM +0200, Vlastimil Babka wrote: >>> On 05/24/2017 09:50 AM, Mike Rapoport wrote: >>>> On Mon, May 22, 2017 at 05:52:47PM +0200, Vlastimil Babka wrote: >>>>> On 05/22/2017 04:29 PM, Mike Rapoport wrote: >>>>>> >>>>>> Probably I didn't explained it too well. >>>>>> >>>>>> The range is intentionally not populated. When we combine pre- and >>>>>> post-copy for process migration, we create memory pre-dump without stopping >>>>>> the process, then we freeze the process without dumping the pages it has >>>>>> dirtied between pre-dump and freeze, and then, during restore, we populate >>>>>> the dirtied pages using userfaultfd. >>>>>> >>>>>> When CRIU restores a process in such scenario, it does something like: >>>>>> >>>>>> * mmap() memory region >>>>>> * fill in the pages that were collected during the pre-dump >>>>>> * do some other stuff >>>>>> * register memory region with userfaultfd >>>>>> * populate the missing memory on demand >>>>>> >>>>>> khugepaged collapses the pages in the partially populated regions before we >>>>>> have a chance to register these regions with userfaultfd, which would >>>>>> prevent the collapse. >>>>>> >>>>>> We could have used MADV_NOHUGEPAGE right after the mmap() call, and then >>>>>> there would be no race because there would be nothing for khugepaged to >>>>>> collapse at that point. But the problem is that we have no way to reset >>>>>> *HUGEPAGE flags after the memory restore is complete. >>>>> >>>>> Hmm, I wouldn't be that sure if this is indeed race-free. Check that >>>>> this scenario is indeed impossible? >>>>> >>>>> - you do the mmap >>>>> - khugepaged will choose the process' mm to scan >>>>> - khugepaged will get to the vma in question, it doesn't have >>>>> MADV_NOHUGEPAGE yet >>>>> - you set MADV_NOHUGEPAGE on the vma >>>>> - you start populating the vma >>>>> - khugepaged sees the vma is non-empty, collapses >>>>> >>>>> unless I'm wrong, the racers will have mmap_sem for reading only when >>>>> setting/checking the MADV_NOHUGEPAGE? Might be actually considered a bug. >>>>> >>>>> However, can't you use prctl(PR_SET_THP_DISABLE) instead? "If arg2 has a >>>>> nonzero value, the flag is set, otherwise it is cleared." says the >>>>> manpage. Do it before the mmap and you avoid the race as well? >>>> >>>> Unfortunately, prctl(PR_SET_THP_DISABLE) didn't help :( >>>> When I've tried to use it, I've ended up with VM_NOHUGEPAGE set on all VMAs >>>> created after prctl(). This returns me to the state when checkpoint-restore >>>> alters the application vma->vm_flags although it shouldn't and I do not see >>>> a way to fix it using existing interfaces. >>> >>> [CC linux-api, should have been done in the initial posting already] >> >> Sorry, missed that. >> >>> Hm so the prctl does: >>> >>> if (arg2) >>> me->mm->def_flags |= VM_NOHUGEPAGE; >>> else >>> me->mm->def_flags &= ~VM_NOHUGEPAGE; >>> >>> That's rather lazy implementation IMHO. Could we change it so the flag >>> is stored elsewhere in the mm, and the code that decides to (not) use >>> THP will check both the per-vma flag and the per-mm flag? >> >> I afraid I don't understand how that can help. >> What we need is an ability to temporarily disable collapse of the pages in >> VMAs that do not have VM_*HUGEPAGE flags set and that after we re-enable >> THP, the vma->vm_flags for those VMAs will remain intact. > > Why cannot khugepaged simply skip over all VMAs which have userfault > regions registered? This would sound like a less error prone approach to > me. It already does so. The problem is that there's a race window. We first populate VMA with pages, then register it in UFFD. Between these two actions khugepaged comes and generates a huge page out of populated pages and holes. And the holes in question are not, well, holes -- they should be populated later via the UFFD, while the generated huge page prevents this from happening. -- Pavel