From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from mail-pl1-f173.google.com (mail-pl1-f173.google.com [209.85.214.173]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id AD2173DA7E4 for ; Tue, 14 Jul 2026 06:29:32 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.214.173 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1784010574; cv=none; b=G4bCoYsgjP3a/jgDP3OktrPxLcWexrDY4owj33vaEdJtx55lf9DBnB867rcovQT9G/UWrNuUlentpBWiBRdVJdaluuk+Fj8egbnb01K+khjNFa9ap2lJg1BHKALEmc/nL6HXDFWvzcbgYkX69izs1g677/tOG2SI9aC2jCHYOvc= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1784010574; c=relaxed/simple; bh=W6zvz+m+18iPbQMMOQV2DLS4LOC/QcuNiJ+cFuraoTM=; h=Message-ID:Date:MIME-Version:Subject:To:Cc:References:From: In-Reply-To:Content-Type; b=EQIJ2Wd8atOCgLoJ7FTt6nDGY1BDHMEAYb0blkSJrmCLmx/AbcDrDryaxxd9Q1OizBH3KkziTrZY8Lw5xXCiggoSjGUb3wM8qg1mGat/ud5xGlFDDFQAy9mdgLRdTbhnxntbdQzMAI3lQ243ktjyfQwoQMTRZyz6ktTMUrD4yWY= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=Ky2SDqSD; arc=none smtp.client-ip=209.85.214.173 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="Ky2SDqSD" Received: by mail-pl1-f173.google.com with SMTP id d9443c01a7336-2ceed7018c8so2563615ad.1 for ; Mon, 13 Jul 2026 23:29:32 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20251104; t=1784010572; x=1784615372; darn=vger.kernel.org; h=content-transfer-encoding:content-type:in-reply-to:from:references :cc:to:subject:user-agent:mime-version:date:message-id:from:to:cc :subject:date:message-id:reply-to:content-type; bh=w5q6gfBBfOjUS6i3fiduNPrbsndG/x1Ykm2ieMz81Cg=; b=Ky2SDqSDgoNpo6ZGEyNcdI2pXX2mtVL6nHFf3MFDzw8zuT6ugt+O5OX2wf14hYplQ2 ARg7UJuuT5d06wKgiDnl8GRFUfThV+mwQVjDE3y/3quBBd2Sl/rL/gwOLO8W6og5oosO eZIy9crhbcTivCptBnA3mpgBMa48jNjafTTK7/kTAQLGdBOBNbB7ZXW+VHdmlgd51UHJ vW36x7HnG2NrKgjpbM/4N3SM33YTjnQ2J74Wrx06lGUwhwQF/+awuMAcKstv7u4O7m1j gXPc6IX8j1GdwujFykr5iEPdNQX7ncYTIWBgo+Z6D3zF/ZI0mTyhkNjeKdrvShanHAuC 4E+Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1784010572; x=1784615372; h=content-transfer-encoding:content-type:in-reply-to:from:references :cc:to:subject:user-agent:mime-version:date:message-id:x-gm-gg :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to :content-type; bh=w5q6gfBBfOjUS6i3fiduNPrbsndG/x1Ykm2ieMz81Cg=; b=jZmreWUV31OG4hIE+e/Zm4acfBZdlCz0xzjHfSEI3kXqF++Az4YEwwkJVMaQRb8GN9 NGFUWeWZ6rFY+S5eVkPtSobTSeWh7fy0gYOqnpLirS8Bhe0os9EDqqS7a85A/2Wck+FK bnaV1huDRfqwXxBiDB4f3QYhAdWGl9MLP4CR7HO2NFEcHspNbmhqFG3wg78E7+CWy3Qe /orxcZTbmN6UTeZtpF1Omv9rfzOns5GDJINKXLKpqi71wVmzqKCnaiqoDUO1SAZBBFhE n/dJNA0JMP8ZMaNRtcAa4+9UinaDK+hYvaywJIqoNDmHtRwXAx0MTAxmd2w9BIzruFaN ZnQA== X-Forwarded-Encrypted: i=1; AHgh+RpQI7s9NqfzMOLWx0VDYC2GJQEaQeGHx6yP3KSeyfZnGEYmdSfEtfVFc+3zk1B4lmU0wRo1X2/cJV3c/QU=@vger.kernel.org X-Gm-Message-State: AOJu0YyLJwdZJ4SStpvXnwHjMpPCqhx9TR/coAfk2Tyf9zs2HRMauTX+ XLcU+dUJJbHq2Rfa5yH1f1hOJmsJZh9N4f5Ii6L3cI+RM494KGay5YuJ X-Gm-Gg: AfdE7cmQVABJ24KG8ysngI9zF3iiVMBJ77HIfS1dAMOcybMkrmGlNIlL4qVNMEXkl+M tibuTIDcQQuc5g4Jo3Ly1nRnrZNYtk8oHsY0rfkke46NkpmYNOAK0wkM9E4C0XclsW6MiRws1qq 5AHxwavV4z3zRQ4bOygICdxFTfz4iB6487W0xf6rm7sGr0VO9a1cBD8b5Bbg2xZzPmeh8ubcgl9 NyCTewD6iTJxOccAyWprxg5eMA4CoPUfz6LKXV619hIHW7WMzYSa3fx2hUKf4dJNvmUP0OqhyOR pbmTqrGkxIA63hla0Jr8zncVNlFHNSTN6FLXEJ6OhdK32F7v37dkU3A1pZKmVo7c/vupV0ogq1C lwM8dIRAdGlyAF+5qUy+KFKCqF5/BaS7qphgU9vQf50aU7GVXpCtVok0b1ZXaqpCP67wRoYEf7L MeGki3r0IOmAXCpzpc95K2Vh+8yULP1bk67g== X-Received: by 2002:a17:902:f683:b0:2c9:97a7:f543 with SMTP id d9443c01a7336-2ce9f1789d4mr105975955ad.41.1784010571982; Mon, 13 Jul 2026 23:29:31 -0700 (PDT) Received: from [10.125.192.114] ([210.184.73.204]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-2ccdf808379sm98300195ad.48.2026.07.13.23.29.10 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Mon, 13 Jul 2026 23:29:31 -0700 (PDT) Message-ID: <4b85fb73-73fb-d08d-e218-0eb34c6f90e9@gmail.com> Date: Tue, 14 Jul 2026 14:29:06 +0800 Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:102.0) Gecko/20100101 Thunderbird/102.15.0 Subject: Re: [PATCH v5 4/6] mm/zswap: Implement proactive writeback To: Yosry Ahmed Cc: akpm@linux-foundation.org, tj@kernel.org, hannes@cmpxchg.org, shakeel.butt@linux.dev, mhocko@kernel.org, mkoutny@suse.com, nphamcs@gmail.com, chengming.zhou@linux.dev, muchun.song@linux.dev, roman.gushchin@linux.dev, linux-mm@kvack.org, linux-kernel@vger.kernel.org, linux-doc@vger.kernel.org, Hao Jia References: <20260629112032.20423-1-jiahao.kernel@gmail.com> <20260629112032.20423-5-jiahao.kernel@gmail.com> <4ec2bd64-af40-8ebf-b8a8-2dd7421a1100@gmail.com> <5ce4035b-7f56-d1d2-2d2a-668446d870e8@gmail.com> <30c8df43-9464-8fa0-3614-0ca06b97862e@gmail.com> From: Hao Jia In-Reply-To: Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit On 2026/7/13 23:53, Yosry Ahmed wrote: > On Fri, Jul 10, 2026 at 3:04 AM Hao Jia wrote: >> >> >> >> On 2026/7/10 04:44, Yosry Ahmed wrote: >>> On Wed, Jul 8, 2026 at 7:15 AM Hao Jia wrote: >>>> >>>> >>>> >>>> On 2026/7/7 03:33, Yosry Ahmed wrote: >>>>> On Thu, Jul 2, 2026 at 5:32 AM Hao Jia wrote: >>>>>> >>>>>> >>>>>> >>>>>> On 2026/7/1 19:45, Hao Jia wrote: >>>>>>> >>>>>>> >>>>>>> On 2026/7/1 00:10, Yosry Ahmed wrote: >>>>>>>>>> Before going through more versions we need to figure out if this will >>>>>>>>>> pivot to be a proactive demotion interfcae for swap tiering. >>>>>>>>>> >>>>>>>>> >>>>>>>>> Yes. Should I drop patches 4-6 in the next version and wait for swap >>>>>>>>> tiering to be finalized? >>>>>>>>> We can try to get the non-memcg parts (patches 1-3) merged upstream >>>>>>>>> first. This would also give them plenty of time to bake and catch any >>>>>>>>> potential regressions. Thoughts? >>>>>>>> >>>>>>>> Patches 1-2 can be sent and merged separately, yes. For patch 2, >>>>>>>> please include some numbers for the writeback performance before and >>>>>>>> after batching. >>>>>>> >>>>>>> I'd love to collect some performance data. Do you have any recommended >>>>>>> benchmarks for this? >>>>>>> >>>>>> >>>>>> Perhaps the following test case could work? >>>>>> >>>>>> Test Setup: >>>>>> - Total memory: 32 GB >>>>>> - zswap settings: max_pool_percent=1, accept_threshold_percent=50, >>>>>> shrinker_enabled=N >>>>>> - cgroup constraint: memory.max=1G >>>>>> - Workload: Run the following stress-ng command inside the cgroup for >>>>>> 120s to >>>>>> continuously force zswap store failures and trigger shrink_worker(): >>>>>> >>>>>> bash -c 'echo $$ > /sys/fs/cgroup/zswaptest/cgroup.procs ; \ >>>>>> exec stress-ng --vm 4 --vm-bytes 4G --vm-keep --vm-method rand-set -t >>>>>> 120s -q' >>>>>> >>>>>> The following comparison results were collected over multiple runs via >>>>>> bpftrace >>>>>> and the 'written_back_pages' sysfs interface: >>>>>> >>>>>> Baseline Patched >>>>>> --------------------------------------------------- >>>>>> shrink_worker wakeups 5,587 878 >>>>>> shrink_memcg calls 7,823,853 2,347,320 >>>>>> written_back 257 781,214 >>>>>> >>>>>> Conclusion: >>>>>> Under the same workload and duration, the patched kernel shows a >>>>>> significant reduction >>>>>> in both shrink_worker wakeups and shrink_memcg calls, while successfully >>>>>> executing a >>>>>> much higher volume of page writebacks. >>>>> >>>>> Hmm this is actually a bit concerning. Yes, we are invoking the >>>>> shrinker less, but we're writing back *a lot* more memory, orders of >>>>> magnitude more. We are using a batch size of 64, and making ~1/3 of >>>>> the calls to shrink_memcg(), so the number of written back pages >>>>> should be ~20x more, not 3000x more? I think I am missing something. >>>>> >>>>> Also, ideally, the batching wouldn't result in significantly more >>>>> writeback, but a similar amount of writeback over less shrinker >>>>> invocations. If we are writing back significantly more pages then the >>>>> batching logic is probably too aggressive? >>>> >>>> Apologies, I think the test I constructed has a bit of a problem. This >>>> test has very, very heavy memory pressure and is already a very abnormal >>>> case. >>>> >>>> The zswap entry returns the first time because of "second chance" after >>>> setting referenced to false. For the baseline, it scans 1 page per node >>>> each time for 16 loops. During the test, shrink_worker() basically exits >>>> at about 16 pages each time. >>>> >>>> Since stress-ng periodically and randomly writes to this 4G memory, it >>>> keeps triggering zswapin and then waiting to zswapout new zswap entries >>>> after falling below the pool threshold. When the speed of zswapin/out is >>>> far greater than the scanning speed of shrink_worker(), a large number >>>> of zswap entries cannot wait until the second scan for writeback. New >>>> entries are stored on the zswap LRU list again, and the referenced of >>>> the new zswap entries is set to true again. During the test, it was >>>> found that 99.21% of the return values of shrink_memcg_cb() in the >>>> baseline kernel were LRU_ROTATE. >>> >>> Hmm if I understand correctly, you are saying that the current >>> upstream code is actually failing to writeback when it should in the >>> previous test case with very high memory pressure, but it is with >>> batching? If that's the case, I think it's actually really good data >>> to include. However, we should make sure that's what's actually >>> happening. If the current shrinker is not keeping up and failing to >>> writeback, we should observe: >>> 1. shrink_worker() hitting MAX_RECLAIM_RETRIES continuously and bailing. >>> 2. zswap usage consistently remains at/near the limit, and not going >>> down to the acceptance threshold. >>> 3. zswap_store() failing to accept pages and the pages going directly >>> to disk, causing an LRU inversion (hotter pages on disk, colder pages >>> in zswap). >>> >>> Can you confirm that this is what's observed with the high pressure test case? >>> >> >> Apologies, my previous explanation might not have been very clear. >> >> For an entry to be written back, the shrinker must scan the *same* entry >> twice: the first scan sets referenced to false and returns ROTATE, and >> only during the second scan can it be written back. >> >> If a swap entry is zswapin'd between the first and second scan (meaning >> the entry is no longer on the zswap LRU), then this swap entry will not >> be written back by the shrinker. Therefore, the second scan must occur >> before this entry is zswapin'd for it to be possible to be written back. >> So, if the baseline scanning speed is far slower than the lifecycle >> speed of the swap entries, it results in only scanning once. In the >> baseline kernel, 99.21% of the return values of shrink_memcg_cb() are >> LRU_ROTATE, while the patched kernel's shrink_worker() scans at least 64 >> * 16 entries in a single pass, resulting in only 58.7% of the return >> values of shrink_memcg_cb() being LRU_ROTATE. > > Right, my question is, is the high rate of LRU_ROTATE leading to > failure to writeback in a way that causes zswap store failures (and > pages skipping zswap and going directly to swap)? Yes, I observed that the number of zswap_store() failures (returning false) is very high. Thanks， Hao