From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 6F3E9C4345F for ; Fri, 19 Apr 2024 06:02:50 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id B77956B007B; Fri, 19 Apr 2024 02:02:49 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id B27E56B0082; Fri, 19 Apr 2024 02:02:49 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id A15D86B0083; Fri, 19 Apr 2024 02:02:49 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id 859BF6B007B for ; Fri, 19 Apr 2024 02:02:49 -0400 (EDT) Received: from smtpin13.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id 39F711C08E5 for ; Fri, 19 Apr 2024 06:02:49 +0000 (UTC) X-FDA: 82025237658.13.4024CC9 Received: from invmail4.hynix.com (exvmail4.skhynix.com [166.125.252.92]) by imf13.hostedemail.com (Postfix) with ESMTP id 34A462001A for ; Fri, 19 Apr 2024 06:02:44 +0000 (UTC) Authentication-Results: imf13.hostedemail.com; dkim=none; dmarc=none; spf=pass (imf13.hostedemail.com: domain of byungchul@sk.com designates 166.125.252.92 as permitted sender) smtp.mailfrom=byungchul@sk.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1713506567; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=K1KGEyYOZvjeSaa6ujaQODrjEH1Ss6gNrWKXJAkPa7I=; b=A06kAmUM6PpxtpdROY9MP6xl27iAsgiy7C4NR0lqv6xooaumYEdf563HjPNC1Q0UxNJ6IY 10lLzFY4To61dr4jN5t4rwLnp/9xmmcBS0j845pLs9tuVn1r2Bl5mZekZaUycZe+IGRQmO HpWjxVzDdRNvitLvmlRPy3zIHZFl/bw= ARC-Authentication-Results: i=1; imf13.hostedemail.com; dkim=none; dmarc=none; spf=pass (imf13.hostedemail.com: domain of byungchul@sk.com designates 166.125.252.92 as permitted sender) smtp.mailfrom=byungchul@sk.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1713506567; a=rsa-sha256; cv=none; b=XmpL1bR86MRCLeUQbulk9y+MdTDHux5T5IMSekIZcqN8BEbz4A9meHSxQKxt8YdnIJY+1i rwM+pQTCOXGBJDt3zRYNiIUGnXaEVz3f70EvBc6fgX9NaxBu/KfiiEXq4qTmYQnL2Asl7i 8Owhix3yDIHr8NAPDobwV4RhaXBDDcs= X-AuditID: a67dfc5b-d85ff70000001748-6c-662208ff0789 Date: Fri, 19 Apr 2024 15:02:34 +0900 From: Byungchul Park To: Andrew Morton Cc: linux-kernel@vger.kernel.org, linux-mm@kvack.org, kernel_team@skhynix.com, ying.huang@intel.com, vernhao@tencent.com, mgorman@techsingularity.net, hughd@google.com, willy@infradead.org, david@redhat.com, peterz@infradead.org, luto@kernel.org, tglx@linutronix.de, mingo@redhat.com, bp@alien8.de, dave.hansen@linux.intel.com, rjgolo@gmail.com Subject: Re: [PATCH v9 rebase on mm-unstable 0/8] Reduce tlb and interrupt numbers over 90% by improving folio migration Message-ID: <20240419060234.GA48027@system.software.com> References: <20240418061536.11645-1-byungchul@sk.com> <20240418131757.cd05a8e7c7e66b3e90508c11@linux-foundation.org> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20240418131757.cd05a8e7c7e66b3e90508c11@linux-foundation.org> User-Agent: Mutt/1.9.4 (2018-02-28) X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFlrDIsWRmVeSWpSXmKPExsXC9ZZnke5/DqU0g7+HtC3mrF/DZvF5wz82 ixcb2hktvq7/xWzx9FMfi8XlXXPYLO6t+c9qcX7XWlaLHUv3MVlcOrCAyeJ47wEmi/n3PrNZ bN40ldni+JSpjBa/fwAVn5w1mcVBwON7ax+Lx85Zd9k9Fmwq9di8Qstj8Z6XTB6bVnWyeWz6 NInd4925c+weJ2b8ZvGYdzLQ4/2+q2weW3/ZeTROvcbm8XmTXABfFJdNSmpOZllqkb5dAlfG s/8rWQveCVe8mLqCvYFxEn8XIyeHhICJxOvVe1lg7EUtT9lAbBYBVYkTq2cygdhsAuoSN278 ZAaxRQR0JVY93wVkc3EwCzxiklj+cRVYg7BAlcS623fBbF4BC4n9s96D2UICZRI3etZAxQUl Ts58AraMWUBL4sa/l0ALOIBsaYnl/zhATE4Bb4mOeTkgFaICyhIHth1nAlklIbCMXeL08n+M EHdKShxccYNlAqPALCRTZyGZOgth6gJG5lWMQpl5ZbmJmTkmehmVeZkVesn5uZsYgXG4rPZP 9A7GTxeCDzEKcDAq8fAuiFdME2JNLCuuzD3EKMHBrCTCa8YBFOJNSaysSi3Kjy8qzUktPsQo zcGiJM5r9K08RUggPbEkNTs1tSC1CCbLxMEp1cAoG3zjxz/TW1s8zrb83XH+xrynPH2hM3d9 6eKqlfgW+3JXxVF2lpgXZ5hjZULPe6j13IwOrvvTOC9hU/Eug/JfgqtM935+GnGhWmXj64T/ i18qJDS5zXTf7Xbk2nRum7Q+fbUbB1t+Wf3Z9KM1Vrrr9L2n1y/yaeyYvyUp6pN5yHu7N6Wh 5081KbEUZyQaajEXFScCAAADjaC/AgAA X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFprMIsWRmVeSWpSXmKPExsXC5WfdrPufQynN4MYOKYs569ewWXze8I/N 4sWGdkaLr+t/MVs8/dTHYnF47klWi8u75rBZ3Fvzn9Xi/K61rBY7lu5jsrh0YAGTxfHeA0wW 8+99ZrPYvGkqs8XxKVMZLX7/ACo+OWsyi4Ogx/fWPhaPnbPusnss2FTqsXmFlsfiPS+ZPDat 6mTz2PRpErvHu3Pn2D1OzPjN4jHvZKDH+31X2TwWv/jA5LH1l51H49RrbB6fN8kF8Edx2aSk 5mSWpRbp2yVwZTz7v5K14J1wxYupK9gbGCfxdzFyckgImEgsannKBmKzCKhKnFg9kwnEZhNQ l7hx4ycziC0ioCux6vkuIJuLg1ngEZPE8o+rwBqEBaok1t2+C2bzClhI7J/1HswWEiiTuNGz BiouKHFy5hMWEJtZQEvixr+XQAs4gGxpieX/OEBMTgFviY55OSAVogLKEge2HWeawMg7C0nz LCTNsxCaFzAyr2IUycwry03MzDHVK87OqMzLrNBLzs/dxAiMqmW1fybuYPxy2f0QowAHoxIP 74J4xTQh1sSy4srcQ4wSHMxKIrxmHEAh3pTEyqrUovz4otKc1OJDjNIcLErivF7hqQlCAumJ JanZqakFqUUwWSYOTqkGxqaDf7Z4dMdJ7q5nKDX/sYxjY75vwtlw9/y5Ew9vmeUi1DLLU8HS xILveAGPxoNVzxY+F2meER7r56XZyeBoITFFeopT7fO8U9ei8vNzDBj2OTl8ZlY8OH/T+Ryb 5F2LdqzYzB+/b238lGpvt0OqLUFPZX95mh4JUaqRXDqhUNup0Thm/qFOJZbijERDLeai4kQA xvmT7KYCAAA= X-CFilter-Loop: Reflected X-Rspam-User: X-Rspamd-Server: rspam11 X-Rspamd-Queue-Id: 34A462001A X-Stat-Signature: kae55zstye7n9mi1kmqdjz55kfinqczw X-HE-Tag: 1713506564-948186 X-HE-Meta: U2FsdGVkX1+TPjIXArYGnUq4DiHG/LCUHDY7oRVZ/gs4S3IXDvd1dszL1QjsphkI+eEk/6aSu1+iiJo5QJPRXfR9gsMjlTRUFAEultuMd0b/M4aTRrXPFqtjEMX+2T8f0KLiu9w3fbfd7iitJXAsxv5aSsI1STw2/7I43YE/rkX7FR5WTo67b4/ZjqWzQh2e8BqhOLV377HBW+QJJTGN02uW5zNYoQWdaZ4XbxvPIR6o2r7A/u1GKgY8RHEiTGKqWnrhY5GN9xUC8EKu4xgrFxhmBQuLvyssrlCeoigrmmu0C6n3kzK+gtRmePWw2d8I6atx7f06Ym+/Te04dM+WCcX99xF5j6+P22srheEBdXlwnSadS5k6gnLxfWNIpIsIysWMFcXpMThQcrRgyiR6aSf+nvpYQZGJ6tOiEZO5IMLTbZVjNgtYyST/Xj2rm2FwtqKM4kVR9Tr9LFTCmr7WyV6S1wPcgUf2CQQbsGQDqAd9durgxBUXN+HlRZ1/8pU7Mrb3HGBkb/KZ/uyEzEe7N8ldrj+zZS/GCRZhNFksE9PKmyiPrfGD4SJtI97Muf4nBdnPW/Hs9FHMiW5KdHxgBQj9OWWGKeywf8LIxhOCsnX2hvS4ViQHpkJzlAXqbEeTXN0ODEs91eU+n1ED6piMi9Vn8vjinnvjv1cpg5DSypk= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: On Thu, Apr 18, 2024 at 01:17:57PM -0700, Andrew Morton wrote: > On Thu, 18 Apr 2024 15:15:28 +0900 Byungchul Park wrote: > > > $ time XSBench -t 16 -p 50000000 > > > > BEFORE > > ------ > > Threads: 16 > > Runtime: 968.783 seconds > > Lookups: 1,700,000,000 > > Lookups/s: 1,754,778 > > > > 15208.91s user 141.44s system 1564% cpu 16:20.98 total > > > > AFTER > > ----- > > Threads: 16 > > Runtime: 913.210 seconds > > Lookups: 1,700,000,000 > > Lookups/s: 1,861,565 > > > > 14351.69s user 138.23s system 1565% cpu 15:25.47 total > > Well that's nice. What exactly is XSBench doing in this situation? As far as I know, it's frequently and continuously accessing annon areas with addresses ranged within 6GB, by multi threads. Thus, it triggers a lot of promotions by hinting fault of numa balancing tiering and a lot of demotions by kswapd as well, resulting in a ton of tlb flushes. All I need is a system suffering from memory reclaim or any type of folio migration since migrc mechanism is one for mitigating the overhead of folio migration. To see the benefits of migrc, it doesn't have to be XSBench but any workload suffering from reclaim. > What sort of improvements can we expect to see in useful workloads? Increase throughput(= runtime reduction of each work in the system). 1. Because migrc removes the CPU time that would've been spent in IPI handler due to tlb shootdown, by skipping a lot of tlb shootdowns. 2. Becasue migrc reduces tlb misses so as to utilize tlb cache better, by skipping a lof of tlb flushes. Besides, I expect overall scheduler latencies can be enhanced, the worst latencies measured using some tracters of ftrace showed no change though. > I see it no longer consumes an additional page flag, good. > > The patches show no evidence of review activity and I'm not seeing much > on the mailing list (patchset title was changed. Previous title > "Reduce TLB flushes under some specific conditions"). Perhaps a better I changed the title because it was supposed to work with only numa balancing tiering like promotion and demotion but, for now, migrc works with any type of folio migration. Thus, I can tell migrc demonstrates its benefits as long as a system is under the control of reclaim and folio migration. Byungchul > description of the overall benefit to our users would help to motivate > reviewers.