From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 874E3C2FC0E for ; Thu, 17 Aug 2023 08:19:34 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id EBFF328003A; Thu, 17 Aug 2023 04:19:33 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id E6EC4280039; Thu, 17 Aug 2023 04:19:33 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id D5DCF28003A; Thu, 17 Aug 2023 04:19:33 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id C6C7F280039 for ; Thu, 17 Aug 2023 04:19:33 -0400 (EDT) Received: from smtpin15.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id 8DCFEC068D for ; Thu, 17 Aug 2023 08:19:33 +0000 (UTC) X-FDA: 81132897426.15.F4C2E2B Received: from invmail4.hynix.com (exvmail4.hynix.com [166.125.252.92]) by imf07.hostedemail.com (Postfix) with ESMTP id EECC840010 for ; Thu, 17 Aug 2023 08:19:30 +0000 (UTC) Authentication-Results: imf07.hostedemail.com; dkim=none; dmarc=none; spf=pass (imf07.hostedemail.com: domain of byungchul@sk.com designates 166.125.252.92 as permitted sender) smtp.mailfrom=byungchul@sk.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1692260371; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=Ih3IiA7fxORHLk0DeojofQ22JWOaPoNpYnsGVDWkHug=; b=H2w2TMt6ykLHQUAA+gS2+tVwV8ExCpXKtkdcG+pGihunBQ4fmWmSKtyfj0qGg5XVbsQfJr 0uXwLbuEX8sEpG7qTwYh/j4i3uTu3au7RfkyIE6vQfe03lgWHbAIwg4R0KA3iVArSFrbBq s5dLBsBj3npUu+enKPItTsUk5El0eXs= ARC-Authentication-Results: i=1; imf07.hostedemail.com; dkim=none; dmarc=none; spf=pass (imf07.hostedemail.com: domain of byungchul@sk.com designates 166.125.252.92 as permitted sender) smtp.mailfrom=byungchul@sk.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1692260371; a=rsa-sha256; cv=none; b=uqBuUa7r+qbIU3kfCZW7sn22HbonI5X9k93C0CQjLvz0opPw1cAgYNxVs6Z7eadJJ1Pbeq gEbouVvS38tJ4fBg7bvLvRFTWcs8lj5Zd5uhb83waO61W+iVfCV6EN6JAN4wzmqdE2CaTC 64nVHMA+q0Ga/03NCHRM806XjHi0Duk= X-AuditID: a67dfc5b-d6dff70000001748-8b-64ddd81085dc Date: Thu, 17 Aug 2023 17:16:31 +0900 From: Byungchul Park To: "Huang, Ying" Cc: linux-kernel@vger.kernel.org, linux-mm@kvack.org, kernel_team@skhynix.com, akpm@linux-foundation.org, namit@vmware.com, xhao@linux.alibaba.com, mgorman@techsingularity.net, hughd@google.com, willy@infradead.org, david@redhat.com Subject: Re: [RFC 2/2] mm: Defer TLB flush by keeping both src and dst folios at migration Message-ID: <20230817081631.GA48308@system.software.com> References: <20230804061850.21498-1-byungchul@sk.com> <20230804061850.21498-3-byungchul@sk.com> <877cpx9jsx.fsf@yhuang6-desk2.ccr.corp.intel.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <877cpx9jsx.fsf@yhuang6-desk2.ccr.corp.intel.com> User-Agent: Mutt/1.9.4 (2018-02-28) X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFvrJLMWRmVeSWpSXmKPExsXC9ZZnka7AjbspBr9eGFrMWb+GzeLr+l/M Fk8/9bFYXN41h83i3pr/rBY7lu5jsri+6yGjxe8fQLE5U6wsTs6azOLA5bFgU6nH5hVaHov3 vGTy2PRpErvHiRm/WTx2PrT0eL/vKpvH1l92Hp83yXm8m/+WLYArissmJTUnsyy1SN8ugSuj 7dQhxoLbAhWTn89hbmB8xNPFyMEhIWAisekLkMkJZt5eeo8FJMwioCpx/VY0SJhNQF3ixo2f zCC2iICGxKeFy9m7GLk4mAU+MEq82PqEDSQhLBAjcWLLZyYQm1fAQmLuxFVsIEVCAlMZJebM 28IMkRCUODnzCQuIzSygJXHj30smkGXMAtISy/9xgIQ5Bewk1i68AFYuKqAscWDbcSaQORIC z9kkZtzZzwZxqKTEwRU3WCYwCsxCMnYWkrGzEMYuYGRexSiUmVeWm5iZY6KXUZmXWaGXnJ+7 iREYIctq/0TvYPx0IfgQowAHoxIPr8OuOylCrIllxZW5hxglOJiVRHh7eG+lCPGmJFZWpRbl xxeV5qQWH2KU5mBREuc1+laeIiSQnliSmp2aWpBaBJNl4uCUamB083L4LLjC2PrizCiZsMy5 jvP65s9W1bknpyavo3q6cVv0pr93veWt/d4pyu+ocxERW3Tz/KEDnDPVWz1WFqzSuDFF5lFi deOkP4e/pAtcy5fKDzwXuWF390/XjXpF2uqVE75t66041WrMXLRC4fD26R+MZz5ImnHDQ1mB O+TA/b+PJT9c2NutxFKckWioxVxUnAgA2aTwrIwCAAA= X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFnrILMWRmVeSWpSXmKPExsXC5WfdrCtw426KwcJWJYs569ewWXxd/4vZ 4umnPhaLw3NPslpc3jWHzeLemv+sFjuW7mOyuL7rIaPF7x9AsTlTrCxOzprM4sDtsWBTqcfm FVoei/e8ZPLY9GkSu8eJGb9ZPHY+tPR4v+8qm8fiFx+YPLb+svP4vEnO4938t2wB3FFcNimp OZllqUX6dglcGW2nDjEW3BaomPx8DnMD4yOeLkZODgkBE4nbS++xdDFycLAIqEpcvxUNEmYT UJe4ceMnM4gtIqAh8WnhcvYuRi4OZoEPjBIvtj5hA0kIC8RInNjymQnE5hWwkJg7cRUbSJGQ wFRGiTnztjBDJAQlTs58wgJiMwtoSdz495IJZBmzgLTE8n8cIGFOATuJtQsvgJWLCihLHNh2 nGkCI+8sJN2zkHTPQuhewMi8ilEkM68sNzEzx1SvODujMi+zQi85P3cTIzDgl9X+mbiD8ctl 90OMAhyMSjy8DrvupAixJpYVV+YeYpTgYFYS4e3hvZUixJuSWFmVWpQfX1Sak1p8iFGag0VJ nNcrPDVBSCA9sSQ1OzW1ILUIJsvEwSnVwLh+EUdknsKNS94x2yXnNlubcRbcbdLJeV19UV0t VXVawoe7VXdStDLnP3hskG31/OO8k1GdBstPLfK43qe3ML8rPz6kudfyhxMbw3axN4rVBVFc D1QT88NkVk5mSMlrsni358rCew8WGJm4XNr570Zt7K/EEwWs/eJv6lYxbdm6z+zGz09qfUos xRmJhlrMRcWJAGr5CbV0AgAA X-CFilter-Loop: Reflected X-Rspam-User: X-Rspamd-Server: rspam12 X-Rspamd-Queue-Id: EECC840010 X-Stat-Signature: sj3funsko6wr576sucktn1rayrx11487 X-HE-Tag: 1692260370-816569 X-HE-Meta: U2FsdGVkX1+3PKRlgtswlXvGgCPvrMWgtPjD+hKm1gSBh07wkEPlHkt9W2GUPR6PV/2FBAF8SQs+r1qkj31MRmq8k7uTdHiFmV6d6EaDa/ChB415WNKIBq0uWAsUiJMGZtmZwCPZkRDcvKcqtQ3zOYNXRSawrbwkS2bzEML4c/QiLQpbqi7DejmXEopfeVOST7uvPquqEmpIYO1LP1OpTpAO40XcXbeEW9VmFZJ9JyyufENvzN+f8kuZYpGtsy61SYimnGaQoVU/pxVT5zLbxF/MI+G0JEn+b35VGVW7uOQAYUuu0fbB+Nrlx8Qm6ADOOv26iX94BNiq+i6kjMvsfbKUeqiDLjdN+HzOtPOaowW0gDcYiHuIeJ0V5pv2DOECDUNQQhZX3jIirlTOvPKHj1wETvBj1vLlLZTLXNTTUs9OSwEDRCh/kDw7+y+juzXSq2vaWEJIHL2hOLMxPGX8R7NkBE9o1w7C0PyGH1/1B4p9gHEOctcApYeuMT/rOa0O+5buFLSwgDD4CP2uhsOdCO2H+v264RdSrVu8+piKWozlr5bqyBHkuYFHgICntWFvbwtWHCV2ahrwrmNwZb5VdohcY1NFG3/5JulI+RHDs+mN7DDTzCOz4ev3IMYyQ304gJOtZbEsp2OqGqs9HLQ9S3Gs0JxHXKUODf1YoSVrNBo5XlZBFRiGF4rq0QAW/76Ivc2WmdiOVVnXh2M+hlzhoJd8aw5Ra/PBZ2Yxs/Y/1CbDRlwkyfAeXC4tAMXvnyyNmomTGVxjPIspIFvjVVgaHpDWrvhMZfve6zncRxopzsDoO0PGQEQtuO8iJdKi32qzD6rgeSp/UQBOJFomjxSRGC6zPw/nhnBI/Ixs9/Mi2IIa5PCS4IXoLnHVvSa5wCBDHw9Ud9Bc1o6Lo6chuluE8FSnmOlhgunIUZNQh0xx1fXY31js+LQrl013opkCmfANccMh5ge6V6NXPFQnkUD iyR3VrdE EelS9 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000006, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: On Tue, Aug 15, 2023 at 09:27:26AM +0800, Huang, Ying wrote: > Byungchul Park writes: > > > Implementation of CONFIG_MIGRC that stands for 'Migration Read Copy'. > > > > We always face the migration overhead at either promotion or demotion, > > while working with tiered memory e.g. CXL memory and found out TLB > > shootdown is a quite big one that is needed to get rid of if possible. > > > > Fortunately, TLB flush can be defered or even skipped if both source and > > destination of folios during migration are kept until all TLB flushes > > required will have been done, of course, only if the target PTE entries > > have read only permission, more precisely speaking, don't have write > > permission. Otherwise, no doubt the folio might get messed up. > > > > To achieve that: > > > > 1. For the folios that have only non-writable TLB entries, prevent > > TLB flush by keeping both source and destination of folios during > > migration, which will be handled later at a better time. > > > > 2. When any non-writable TLB entry changes to writable e.g. through > > fault handler, give up CONFIG_MIGRC mechanism so as to perform > > TLB flush required right away. > > > > 3. TLB flushes can be skipped if all TLB flushes required to free the > > duplicated folios have been done by any reason, which doesn't have > > to be done from migrations. > > > > 4. Adjust watermark check routine, __zone_watermark_ok(), with the > > number of duplicated folios because those folios can be freed > > and obtained right away through appropreate TLB flushes. > > > > 5. Perform TLB flushes and free the duplicated folios pending the > > flushes if page allocation routine is in trouble due to memory > > pressure, even more aggresively for high order allocation. > > Is the optimization restricted for page migration only? Can it be used > for other places? Like page reclaiming? I don't get you but if I'm missing something here and your idea is something good, then it can be a future work. Byungchul