From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-3.5 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI,SPF_HELO_NONE, SPF_PASS,URIBL_BLOCKED autolearn=no autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 52F42C4363D for ; Fri, 2 Oct 2020 14:00:57 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 68035206DB for ; Fri, 2 Oct 2020 14:00:56 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (2048-bit key) header.d=infradead.org header.i=@infradead.org header.b="SbK0nYmP" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 68035206DB Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=infradead.org Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 01AF26B0068; Fri, 2 Oct 2020 10:00:56 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id F0C80900002; Fri, 2 Oct 2020 10:00:55 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id E2C226B0068; Fri, 2 Oct 2020 10:00:55 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0180.hostedemail.com [216.40.44.180]) by kanga.kvack.org (Postfix) with ESMTP id B47096B0068 for ; Fri, 2 Oct 2020 10:00:55 -0400 (EDT) Received: from smtpin02.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay02.hostedemail.com (Postfix) with ESMTP id 247083623 for ; Fri, 2 Oct 2020 14:00:55 +0000 (UTC) X-FDA: 77327146470.02.thing42_3f02bae271a4 Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin02.hostedemail.com (Postfix) with ESMTP id 095901004F5B6 for ; Fri, 2 Oct 2020 14:00:55 +0000 (UTC) X-HE-Tag: thing42_3f02bae271a4 X-Filterd-Recvd-Size: 4234 Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) by imf19.hostedemail.com (Postfix) with ESMTP for ; Fri, 2 Oct 2020 14:00:54 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=In-Reply-To:Content-Type:MIME-Version: References:Message-ID:Subject:Cc:To:From:Date:Sender:Reply-To: Content-Transfer-Encoding:Content-ID:Content-Description; bh=CUDGsfaX06RV82lRVdfunkb9Yk0TwCX6+kZd1kTshAQ=; b=SbK0nYmPxa1e4eYN/337RPAU/6 Zi933T8DX436+uLM5QIy3XFcYxzc6NRdPhln+IwhSJ4FW2sa2SB0HYpJoGcsVgH4Mv2sAukCge/+W b+jwTuKpJiGxQKdVcSX57y2xBCWcetkbMy7nDy9mYN60KMC2fKlk6svRGKCSUfMs7CDJVXvq/exZH ddLlgLu92q/O2REzhz5Pwb2DHQnnPM3DxibKoHH1dTZbyRLxE0m7jqtP6P9GNVSeBDi2iIXz3LbPL pwUoKdFDML2Kfto6bcTDe/XEz62FkTxLHFao/r5MpBavVmfvELx/qJbR1PxIl8Sszp4kGvr81Xd1A +M5fghZQ==; Received: from willy by casper.infradead.org with local (Exim 4.92.3 #3 (Red Hat Linux)) id 1kOLc2-0001FE-L6; Fri, 02 Oct 2020 14:00:42 +0000 Date: Fri, 2 Oct 2020 15:00:42 +0100 From: Matthew Wilcox To: Rik van Riel Cc: Michal Hocko , Sebastiaan Meijer , akpm@linux-foundation.org, buddy.lumpkin@oracle.com, hannes@cmpxchg.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, mgorman@suse.de Subject: Re: [RFC PATCH 1/1] vmscan: Support multiple kswapd threads per node Message-ID: <20201002140042.GB20115@casper.infradead.org> References: <20201001123032.GC22560@dhcp22.suse.cz> <20201002070333.GA21871@dhcp22.suse.cz> <656725362af9bd757a281f0799a0bb9c9b2487bd.camel@surriel.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <656725362af9bd757a281f0799a0bb9c9b2487bd.camel@surriel.com> X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: On Fri, Oct 02, 2020 at 09:53:05AM -0400, Rik van Riel wrote: > On Fri, 2020-10-02 at 09:03 +0200, Michal Hocko wrote: > > On Thu 01-10-20 18:18:10, Sebastiaan Meijer wrote: > > > (Apologies for messing up the mailing list thread, Gmail had fooled > > > me into > > > believing that it properly picked up the thread) > > > > > > On Thu, 1 Oct 2020 at 14:30, Michal Hocko wrote: > > > > On Wed 30-09-20 21:27:12, Sebastiaan Meijer wrote: > > > > > > yes it shows the bottleneck but it is quite artificial. Read > > > > > > data is > > > > > > usually processed and/or written back and that changes the > > > > > > picture a > > > > > > lot. > > > > > Apologies for reviving an ancient thread (and apologies in > > > > > advance for my lack > > > > > of knowledge on how mailing lists work), but I'd like to offer > > > > > up another > > > > > reason why merging this might be a good idea. > > > > > > > > > > From what I understand, zswap runs its compression on the same > > > > > kswapd thread, > > > > > limiting it to a single thread for compression. Given enough > > > > > processing power, > > > > > zswap can get great throughput using heavier compression > > > > > algorithms like zstd, > > > > > but this is currently greatly limited by the lack of threading. > > > > > > > > Isn't this a problem of the zswap implementation rather than > > > > general > > > > kswapd reclaim? Why zswap doesn't do the same as normal swap out > > > > in a > > > > context outside of the reclaim? > > On systems with lots of very fast IO devices, we have > also seen kswapd take 100% CPU time without any zswap > in use. > > This seems like a generic issue, though zswap does > manage to bring it out on lower end systems. Then, given Mel's observation about contention on the LRU lock, what's the solution? Partition the LRU list? Batch removals from the LRU list by kswapd and hand off to per-?node?cpu? worker threads? Rik, if you have access to one of those systems, I'd be interested to know whether using file THPs would help with your workload. Tracking only one THP instead of, say, 16 regular size pages is going to reduce the amount of time taken to pull things off the LRU list.