From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1752210AbdAaUQI (ORCPT ); Tue, 31 Jan 2017 15:16:08 -0500 Received: from mx0a-00082601.pphosted.com ([67.231.145.42]:40378 "EHLO mx0a-00082601.pphosted.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1751025AbdAaUP4 (ORCPT ); Tue, 31 Jan 2017 15:15:56 -0500 Date: Tue, 31 Jan 2017 11:45:47 -0800 From: Shaohua Li To: Johannes Weiner CC: , , , , , , , Subject: Re: [RFC 0/6]mm: add new LRU list for MADV_FREE pages Message-ID: <20170131194546.GA70126@shli-mbp.local> References: <20170131185949.GA5037@cmpxchg.org> MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Disposition: inline In-Reply-To: <20170131185949.GA5037@cmpxchg.org> User-Agent: Mutt/1.6.1 (2016-04-27) X-Originating-IP: [2620:10d:c090:200::5:a8fc] X-ClientProxiedBy: DB6PR0301CA0019.eurprd03.prod.outlook.com (10.168.49.29) To MWHPR15MB1648.namprd15.prod.outlook.com (10.175.139.151) X-MS-Office365-Filtering-Correlation-Id: bb8f2a32-af03-436e-ca5a-08d44a11cd95 X-Microsoft-Antispam: UriScan:;BCL:0;PCL:0;RULEID:(22001);SRVR:MWHPR15MB1648; X-Microsoft-Exchange-Diagnostics: 1;MWHPR15MB1648;3:mOxJgqq18GCK9HTmdDhgtJSS6I4fxgysC2t3doqz1roYKG4ttR3l051xc084MP3D8laR0xJUI7w5yxJpqXToxJ/PrhI/BPGvHrDF8i+DXZe9Q/WnUcXJ5X9Gkg0azyQ+xViebkPf95Ytm2zRIco9kTJEabQQQEZXW2HaaquJcUoHtf6sjsAHeDC/wR8FBw9nyBIM+EmgL5M0x647mqN3P39s64Ox1sUDOQYNvSR4H/+cMZsJWOtCzgapTgQmd/XMKYIx0WzgNK7XlQ8oebuaew== X-Microsoft-Exchange-Diagnostics: 1;MWHPR15MB1648;25:j3BGBSljxIyQNDWxnl7+qddRmdlRF4t6JqjcznCaJCTm5pLuB92AFLirkXHf7n+oxX2kvvpMZNaZq1UDoEmKrbTJbp2vVKmEK67Yf8oJv4IHpqS3vVTrqPejogItTMop4tUXTlDmsaaSHQ3AaQz4m1KcBgPneutILA+B69QM0mKxDh+aS71qvTVhzqokPAwN+eDktq3aFTRid7SXa6jdpLGyCIwzvW8nkU3RjSBZA/uCecTyJd2rWK6vc1ZQ3ryDWdbGN4n1wI8Oc2fLlL+phdi7bniVpcL3n3C7HKZLJdfrSKxmn0RjRYCIEI2OUleLkpsYOqynoEuva0t727SlzwUMHApK/fGM/7JvqUeF/nVHbastIm9eZh8FVORLQ7iRY9ZkCyaDUbdpK1RZLpn4QjztBsb6Tfkhm3gIKfElAJ1d8BriuX1babSpu1avE7ryRSVYNRMONX48zyJBwYC/1ru3BUEGHgRpEAMz1OwSv9gTdkmzYjymASdlz6gI3c+wKi03+P0Z1X/ZsQg+FKJH7zv/arcPJNMR3FiE5Z32k56U5b74T2lqGY2FNgP8FV+584S97pZ/Sgixiif8C77JbUy3DJ+cAkQsMWs5FEKuAaOCUmndccBo1L7x7scLDvnVtirM282YKGqDD7A+bOqP57YLLHg39NnbRhEeGCl9n+2q2yi90vJxmwaqWDruw/4DtnpTrSDEgEkpjtHh+dx9bg== X-Microsoft-Exchange-Diagnostics: 1;MWHPR15MB1648;31:gLxxuC6Ft4axFUpCAu0MR8c0oU/bQM4FtkJZ2myzhPgo93zIWG59pcC3n3nkeRFv+oEg4h+30iTXsuvHaMC+i2tlcdMybYXCjBSmrUpEzJHFASNmmfooYAvK5VH7svxKlnz2hLQEziHj4dDV2VACcvzBngFxxbpJo1fyEDlA+rED0OA31/mnRF9V/IH7TLZQfXVtOnp0uVQdh0/l7rJYkjbiAln0TGUrXYQ6lR/MGW8H5mCg35SjQ6KOFDC5rl6xz1p92Vil0JtoPAaPOqrtFQ==;20:QFoOhYVDJvqGpVdbCndrGb0YJAZ/deEXCxrYdZ/L03hsDcmYH/XowjQ9N1Ott8fcfSY/X9dpVuplovOG3MpicjpuvHZyWzTs5EBap2pNKkLfP1SFPjuV721AYZJnu2KJH6qazn/NwFuTIik4kbVtOfIWLfRYdyzGbcbFSMawrXLx43lZxzHWuSQBIGmukBjuJGDZxNNan9DR07kZKKCL3YWMWssMUXi6nEz1xcEiUFsleOvk4WQZESNkdQwJG4NZJDqSiwLPndrQbQ2acbykOHLLjnQeC9dk9NotAKoomt3+dmMHFl8bQShuPv4WLiE+UVr1cjfO2lu6ICWYKBBxqaIFyowfBZmrnZo4+ryeAdR8uSv3rhakz9D+HliKKCGMQCcWdK8ARU81YRySL56EYwnRt7Pu39uEQYPLElzBBBu9mob2zfPVBlsJw+UnEP+zsbYxLpQxur0I8OW7DpSk0nnzqDE0Ds5vSYMK0hzgKTB7WEda3BacsEGorUDEVABx X-Microsoft-Antispam-PRVS: X-Exchange-Antispam-Report-Test: UriScan:; X-Exchange-Antispam-Report-CFA-Test: BCL:0;PCL:0;RULEID:(6040375)(601004)(2401047)(8121501046)(5005006)(10201501046)(3002001)(6041248)(20161123555025)(20161123562025)(20161123560025)(20161123564025)(6072148);SRVR:MWHPR15MB1648;BCL:0;PCL:0;RULEID:;SRVR:MWHPR15MB1648; X-Microsoft-Exchange-Diagnostics: 1;MWHPR15MB1648;4:n47vxFkFRL39n6vpOM3Fg56NBb91LYrZ80k4tPmMFHyU0Rc3As7UtHtz3SmdO83APiai7wbLukaRWGJVYPUghXBC78LsOZa3vExaPeb/QTbVbiq0u/ef3FnF6eK+8MTo03+QQ3hO8OTZwzv/h3ZlLPbhqUlwzfdJVqVvWWXE+l879wdsP1qW4zVbxy4o4O5PUMrq+kkQVIKXLbTTu948127oOax6NPFCWbkdTYO1dmlvFYCck4nOmaEmyHvuN59WO8AQLz4Mm+P23Fpk2vcT/SK4YxFYGYKrVcG2sUBe0vqnd6QXBSssuOoqkc+LCnBFufVNR1gzmjrJAJ0arxaLmKy5IICvl9olP2QQ5+qiBDiJ1+i+Sxj8sgqxgooVZR6Bz2NLKllex0Nr2FS8kDPm0I5LYGBWUavwdsAOg9Vb10xD1jIF3fNWZK+OF8d90z6D3Pc2xRlmC+cKD+HGkwXPzLdHlBWhe/mtjfW7iiTvU14eBJ3Nrg6z7SZFW0Ap1/gkSi8RMXzKLhO6siWaNGQAaxw8qKWVIigObxo+0UPrKb2eJ7+BCOhL0Xqyko2a7ouCMFgboVoSxqVthfjIbx06Jg== X-Forefront-PRVS: 0204F0BDE2 X-Forefront-Antispam-Report: SFV:NSPM;SFS:(10019020)(4630300001)(6009001)(7916002)(39450400003)(199003)(189002)(24454002)(31014005)(92566002)(5660300001)(50986999)(54356999)(76176999)(6116002)(4326007)(305945005)(9686003)(8676002)(2906002)(1076002)(23726003)(101416001)(54906002)(7736002)(46406003)(53936002)(83506001)(2950100002)(50466002)(6666003)(42186005)(110136003)(97736004)(38730400001)(4001350100001)(55016002)(25786008)(105586002)(106356001)(81156014)(47776003)(81166006)(189998001)(6506006)(98436002)(33656002)(68736007)(229853002)(86362001)(97756001)(6916009)(18370500001);DIR:OUT;SFP:1102;SCL:1;SRVR:MWHPR15MB1648;H:shli-mbp.local;FPR:;SPF:None;PTR:InfoNoRecords;A:1;MX:1;LANG:en; X-Microsoft-Exchange-Diagnostics: =?us-ascii?Q?1;MWHPR15MB1648;23:hJsSloIhBWWrdQskFNUCzroiQskHhpjP+w3/oH9S7?= =?us-ascii?Q?tw5WRrTP2st7YT4pYdjqxvcbhkRbkcUwj4pouD3nrgM5LYNV58cD6ZdmpBye?= =?us-ascii?Q?BrqDe0tN7dBHVZuaA9YjYqC5xtCqvSLI4ngPjZML2cMlEV1RnN1YBBxx1BeA?= =?us-ascii?Q?+P4nkEsqfuAWDa6tx3LbxcK3tLUBYQwhoRPNxH6kakKQXluvTbZtihjX85ta?= =?us-ascii?Q?ihWej+MzpPHDLfxPVQeY+AXhjfNE+vLuXZ+V2y9tR49GiZQQJVFElJRSDmNI?= =?us-ascii?Q?nZrYdWh/f5KO/eA8jZBnY8aTVZ/J3JMYxAKjHuc4sx8T6rg+pUGqCREq49U7?= =?us-ascii?Q?NViQ4WMMsU/07YOBaqMzxV1sWiAhoo617ch0PuxWkzrWXwkR2T/23q2SSzvA?= =?us-ascii?Q?wFogszJbdgOmTK+g9vdSnxq+vPE9YmZ771Wqko6+yz6w+I2OyQ68fmxLn+Uf?= =?us-ascii?Q?Myeq9UC8p2P6VNr03/nJubcinTPoW3dM6quhJj6/o723SImrBUJyKp9AKr/g?= =?us-ascii?Q?X2j3nOTPqml7FR+v9nEN8qoQ9UEuAtDbMqOijB+UyzZTQM4jzZbcnQ3FK9FV?= =?us-ascii?Q?tyEfJ94IieChct0O2G5aE58/06yDhDNKsTKIKA6qEin/ay0kU+RP1+ee0guG?= =?us-ascii?Q?3reTOEyKoUuWimeU7iOsHsv0NtkytahVq2szQKw1AZwH/0FTM27aOxW8IzcZ?= =?us-ascii?Q?kLAqUZFB1aOPg9GCtuITiyK16+IUI137EkkZZIboOLQVkmE6B4FEFVzZ4l4T?= =?us-ascii?Q?GnSFemcpCqg3Wqvd/xW4YcldpyKUnRmyG8xK3K1ENInLf6LlSQ0zRwjXa1fh?= =?us-ascii?Q?fYxNIKMvh4qOP8dw/LLREoUPbbjCTy6+sZSjSRV+4j77M+zmvEVIg/Z4mscP?= =?us-ascii?Q?3F/335vVJU5hSJE7TvnP4ISqnwSMDPRrNygCaEjHj+xSv9TT8ullTpzuHVW7?= =?us-ascii?Q?06/E0g7vwmhe4NOqwSvzJf6kyjtX2vTJewM5gCVCvacpj94YN4TQ862jjXg/?= =?us-ascii?Q?NKDHvKvch1Y1rtMAv6beRzJilvrM/346MoG0qjmjQTJYS88qKguj7G9tntft?= =?us-ascii?Q?MUPD3ybH0cknmkucAKO/dtS3xkec6ipZBQWb0gdP9/a/a6uoXPekJ4f6VXSH?= =?us-ascii?Q?RwcGRxDPY4YiFMsVmKnd9PZQZonqItMcUehEZNCxOe44dqvao9r4TM9pbQ3y?= =?us-ascii?Q?Sh6GirbhixMLrzc+2IRRHxjcKwOiiLQzH2i32uBgnerD7/WDt2ula8DeKYSw?= =?us-ascii?Q?x2V0gWDghtH0do0I3I=3D?= X-Microsoft-Exchange-Diagnostics: 1;MWHPR15MB1648;6:0zt2UGnoo1MNEdhYXBE0YPZ+hJqmBF1QFizn7nN0EQPw1TGC71bGBcFB5cWBG8Zh8AARTTGACwhzaaf1dTfJDcva0Tfcq7tPfbgAfRL67fynaniHCrDX0OBRyRdlORXklN2DdLiF2KTUrI+Gpnw9VIux23cGMTcCQUvjqgO4StRRg1Mu0YLJvNXKSH2lEEjjbT06oKl+xvoH3yWx+nql3bj63aSYNAH7tbZP/kLwEJEso/bKm5QEGIoGoW1Mar+tTB4r/oNBwA61EjvR7ssiL9S94H9YqUBVJmN203qNmx6WwQFoZ6Z/wuE+nxnDQnRBkCAWxeJ91CHiIwROSKcFOM5puUwJz8M8qPeJJ71X1kDQFnHxDxAyrubc6i3wFS9Gsg/epr377r0zY5ln5Y76eWeq/cbtWbF4OvP+FS2grlA=;5:T96ZU5pjkgoSOpeTSqzf1PGyk0WbTB8n5PM1RpDSVMe92TGnTEwpeFlsI/koLj0hNl6D328EvEAJrfx+3nm7Q4j17cZNdnTN8uOMuLyLEfliW/IeR+6xMPO+2AxgnPEmmZWm6o4F4M5Uy01nivqHqF+dynUaI7UorbdW68N+76g=;24:QzOAzho8uy06nTjbQZCUeKltuQ807zc9x3Cckx66ERLLMXs8zuD2C/+zOl8OD4VvPTN0w1UPQ41ks9CYrb0Eb5w3QwSyhTqpE7USSQ4HOKI= SpamDiagnosticOutput: 1:99 SpamDiagnosticMetadata: NSPM X-Microsoft-Exchange-Diagnostics: 1;MWHPR15MB1648;7:xhXpmGGXiXGwqg1RpIekaREspJVxnBVJ/8aR9YI+rx6lLMK2WuStqMuhPMSx6l6mWuWQpVxSHaMoX+Z8GgF6VvMLqc9h33WT1O+K59/yP0Vp9zseaYp7rMJpzB9Wjw7pAcYE1KDjAMcrbvcS5vOpiODbspUuwniHkH9/fMLExHu9ZmKuVEgXraUEhK5kfdAIq68vvvweOuE38m7Ee/anWX0yngZnzDBpfRJHKtUHjcyBsS5SID3GOz6KsNbck8XXXcCqlFEB/TWF4laLfv4Uq8Pi00LV2LSE/AyWszvJs4V6QKDdxzu9DgxjFIqK7HbtfP1VPVq0OL4aqaISt+ANIhyNsEcHaXGiYCiIVA1/E5iRjgJBRC9xA1TEcC3XN9gdif3/JB4n2XAEWhIzT2Usk+wDTJuPgrpKC0TMBc0o3jVjKIV3IIsJt+X78Oa1z4ZE2NwyHzKxP/agn0/OgjW9foJB/eVnLMw7r7yFuEIjMX0auLTHAOTpMhmiDv8Kw/HojtdjMxSnzMtRPmwffAfHZSIYgJtklk0d6eLJISfgGV52ZoV03wtLxMMrWYc0xkoe;20:qZtmdARGyTdYXE4vCG9KaxjD60jtb7asniK//DZN6merTXjkkqXJLak6KHQyxPVYDlBtmVIxA1+Hk2dUU1pzU6FGYU3WR5UpX+Tw3PH3qoK63HQDocfY5wFvNgvptj8I7+hF8wcbRnlzpu2o20rhirrG4qH/BrcduhFpL2rETdM= X-MS-Exchange-CrossTenant-OriginalArrivalTime: 31 Jan 2017 19:46:05.6840 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-Transport-CrossTenantHeadersStamped: MWHPR15MB1648 X-OriginatorOrg: fb.com X-Proofpoint-Spam-Reason: safe X-FB-Internal: Safe X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10432:,, definitions=2017-01-31_09:,, signatures=0 Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On Tue, Jan 31, 2017 at 01:59:49PM -0500, Johannes Weiner wrote: > Hi Shaohua, > > On Sun, Jan 29, 2017 at 09:51:17PM -0800, Shaohua Li wrote: > > We are trying to use MADV_FREE in jemalloc. Several issues are found. Without > > solving the issues, jemalloc can't use the MADV_FREE feature. > > - Doesn't support system without swap enabled. Because if swap is off, we can't > > or can't efficiently age anonymous pages. And since MADV_FREE pages are mixed > > with other anonymous pages, we can't reclaim MADV_FREE pages. In current > > implementation, MADV_FREE will fallback to MADV_DONTNEED without swap enabled. > > But in our environment, a lot of machines don't enable swap. This will prevent > > our setup using MADV_FREE. > > - Increases memory pressure. page reclaim bias file pages reclaim against > > anonymous pages. This doesn't make sense for MADV_FREE pages, because those > > pages could be freed easily and refilled with very slight penality. Even page > > reclaim doesn't bias file pages, there is still an issue, because MADV_FREE > > pages and other anonymous pages are mixed together. To reclaim a MADV_FREE > > page, we probably must scan a lot of other anonymous pages, which is > > inefficient. In our test, we usually see oom with MADV_FREE enabled and nothing > > without it. > > Fully agreed, the anon LRU is a bad place for these pages. > > > For the first two issues, introducing a new LRU list for MADV_FREE pages could > > solve the issues. We can directly reclaim MADV_FREE pages without writting them > > out to swap, so the first issue could be fixed. If only MADV_FREE pages are in > > the new list, page reclaim can easily reclaim such pages without interference > > of file or anonymous pages. The memory pressure issue will disappear. > > Do we actually need a new page flag and a special LRU for them? These > pages are basically like clean cache pages at that point. What do you > think about clearing their PG_swapbacked flag on MADV_FREE and moving > them to the inactive file list? The way isolate+putback works should > not even need much modification, something like clear_page_mlock(). > > When the reclaim scanner finds anon && dirty && !swapbacked, it can > again set PG_swapbacked and goto keep_locked to move the page back > into the anon LRU to get reclaimed according to swapping rules. Interesting idea! Not sure though, the MADV_FREE pages are actually anonymous pages, this will introduce confusion. On the other hand, if the MADV_FREE pages are mixed with inactive file pages, page reclaim need to reclaim a lot of file pages first before reclaim the MADV_FREE pages. This doesn't look good. The point of a separate LRU is to avoid scan other anon/file pages. > > For the third issue, we can add a separate RSS count for MADV_FREE pages. The > > count will be increased in madvise syscall and decreased in page reclaim (eg, > > unmap). One issue is activate_page(). A MADV_FREE page can be promoted to > > active page there. But there isn't mm_struct context at that place. Iterating > > vma there sounds too silly. The patchset don't fix this issue yet. Hopefully > > somebody can share a hint how to fix this issue. > > This problem also goes away if we use the file LRUs. Can you elaborate this please? Maybe you mean charge them to MM_FILEPAGES? But that doesn't solve the problem. 'statm' proc file will still report a big RSS. Thanks, Shaohua