From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 6C0BDC07545 for ; Tue, 24 Oct 2023 16:09:09 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id F30946B01F6; Tue, 24 Oct 2023 12:09:08 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id EE1416B01FE; Tue, 24 Oct 2023 12:09:08 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id DA7D06B025F; Tue, 24 Oct 2023 12:09:08 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id CDC7E6B01F6 for ; Tue, 24 Oct 2023 12:09:08 -0400 (EDT) Received: from smtpin20.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id 9C817C041B for ; Tue, 24 Oct 2023 16:09:08 +0000 (UTC) X-FDA: 81380839176.20.6E4E09F Received: from mail-qk1-f181.google.com (mail-qk1-f181.google.com [209.85.222.181]) by imf25.hostedemail.com (Postfix) with ESMTP id 725F1A001F for ; Tue, 24 Oct 2023 16:09:06 +0000 (UTC) Authentication-Results: imf25.hostedemail.com; dkim=pass header.d=cmpxchg-org.20230601.gappssmtp.com header.s=20230601 header.b="f/OGyAIV"; dmarc=pass (policy=none) header.from=cmpxchg.org; spf=pass (imf25.hostedemail.com: domain of hannes@cmpxchg.org designates 209.85.222.181 as permitted sender) smtp.mailfrom=hannes@cmpxchg.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1698163746; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=VOgTqaWod2GIRMlQnZ21kgeA7fU4We2gZHXmH2gakWs=; b=Q2vMyd+hxG4N9tTlExOIb0V/dyFgfE2OnWrWzRWQVdSpd46OcbAdmZ68wUR45V57r7EvZY MnrBk33moAyuBy/8sK6kJui/FDdy3GDxM6rtZXsdl8zhAPSuySOX8T7h6SPn+YSkTg70nO FTCle12bGbGAJi6Q1+gd/HYYfA3MI04= ARC-Authentication-Results: i=1; imf25.hostedemail.com; dkim=pass header.d=cmpxchg-org.20230601.gappssmtp.com header.s=20230601 header.b="f/OGyAIV"; dmarc=pass (policy=none) header.from=cmpxchg.org; spf=pass (imf25.hostedemail.com: domain of hannes@cmpxchg.org designates 209.85.222.181 as permitted sender) smtp.mailfrom=hannes@cmpxchg.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1698163746; a=rsa-sha256; cv=none; b=jQfqlOM8VqN7J4dj6RvBCtwUgWr/VmBxNoMX8Yg6VJL73hfj9BAzDatyCBlZZEjW9zgaVy xn8AtK51UuQe5czqAf8KDlkzZImUND71qR1uWGt9MAvM1PxCzIey1OAFGSs2yu2hau95o3 EmQ8nJM000MlTFHUD2Sy9ChjNfNWBKE= Received: by mail-qk1-f181.google.com with SMTP id af79cd13be357-77063481352so453081085a.1 for ; Tue, 24 Oct 2023 09:09:05 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=cmpxchg-org.20230601.gappssmtp.com; s=20230601; t=1698163745; x=1698768545; darn=kvack.org; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:from:date:from:to:cc:subject:date:message-id:reply-to; bh=VOgTqaWod2GIRMlQnZ21kgeA7fU4We2gZHXmH2gakWs=; b=f/OGyAIVtH7MNUYSKP6PNphCLUKgCjy/BaB4/tECENfl3mo7vr80aEyuLrOPlccRy2 sZQRGNCEnAACadYGxRxQO0rubuUhYaiDFOQhipOyKLAnMVfJ70MWvcNog7FNAMZ989Sb o8l1BuZ9vDiNDDfnGGvf+K6692M7qjb6cRXeNXymQUuDURwD+4ODHFj/sSsrC92sOnuM N8qkcjk6OURsiM66a5S5VoXvVMRvYcs0mXleQ2wz+lxvd/BW/BDNPZD0+fSiouE2SF0n 4wIL87lwZTKaxoSs4fjyZxkF0o7ec43ZhHIeDvx9U19pFFtkebGnzVb37VYzbCBgJLkz V36Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1698163745; x=1698768545; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:from:date:x-gm-message-state:from:to:cc:subject:date :message-id:reply-to; bh=VOgTqaWod2GIRMlQnZ21kgeA7fU4We2gZHXmH2gakWs=; b=gclViufrO1FmBYBa5XmBpP+d4ObTFb2yZZLF6eSBRFy8o+4uhfVKNT5Gzw+quCbMj5 bPpfmINTm/vKLbrbkUU5DatN4fWPBpYai2CO2zr0VgBb3NLbBR9K3WLAWdk7smJeKGdH lqMUz8tS4TSY0DzdfatAgdK18FShoBXQkXLTnZWxSNU0zrXnA8JCD2fZB4DYs243LSX2 Rrkp3PWnS22OqdwSmUYD7DgfOZh1pqR0LZ5eqLZbZJtTANjxt1uNM6RKPVCI3kCJSA/M We0JwC3GzW9PDUIK36ru/SUbztf3uDXUCzv5OkUd3maTDN41t1LykpuysFq7nFlQQ7og i9gw== X-Gm-Message-State: AOJu0YwpOcF8yOzrK+uoCPQ+YwGdq4mFxFXS/4cLYRqjhOLrqTVz+pgO bwswlqHFxPsERNl3bQJfOi7LCA== X-Google-Smtp-Source: AGHT+IE4iOSeMFxlVe/rUGgAvUrIKN/iHb2FxiHseAUx3vVefnmIsABNiiFtr0ZTnpRstzIYxb6ysQ== X-Received: by 2002:a05:620a:448a:b0:773:ac84:3f57 with SMTP id x10-20020a05620a448a00b00773ac843f57mr17278800qkp.5.1698163745326; Tue, 24 Oct 2023 09:09:05 -0700 (PDT) Received: from localhost (2603-7000-0c01-2716-3012-16a2-6bc2-2937.res6.spectrum.com. [2603:7000:c01:2716:3012:16a2:6bc2:2937]) by smtp.gmail.com with ESMTPSA id r16-20020a05620a299000b007756c8ce8f5sm3505539qkp.59.2023.10.24.09.09.04 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 24 Oct 2023 09:09:05 -0700 (PDT) Date: Tue, 24 Oct 2023 12:09:04 -0400 From: Johannes Weiner To: Nhat Pham Cc: akpm@linux-foundation.org, cerasuolodomenico@gmail.com, yosryahmed@google.com, sjenning@redhat.com, ddstreet@ieee.org, vitaly.wool@konsulko.com, mhocko@kernel.org, roman.gushchin@linux.dev, shakeelb@google.com, muchun.song@linux.dev, linux-mm@kvack.org, kernel-team@meta.com, linux-kernel@vger.kernel.org, cgroups@vger.kernel.org Subject: Re: [PATCH] zswap: export more zswap store failure stats Message-ID: <20231024160904.GA1971738@cmpxchg.org> References: <20231024000702.1387130-1-nphamcs@gmail.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20231024000702.1387130-1-nphamcs@gmail.com> X-Rspamd-Queue-Id: 725F1A001F X-Rspam-User: X-Rspamd-Server: rspam02 X-Stat-Signature: stbfqgq6mrxkju9hnwudh7hy6gepdtmp X-HE-Tag: 1698163746-800813 X-HE-Meta: U2FsdGVkX1+eKxVwKHj6FkkBvWN8kwPnZE4884kicto8iZoI5G35+vFB1mz1Gvff5oz9P1E9cSbRt+51QSeFa1PZrqvdrwf+xcj55uIbU9qqL3VbpkvgdjYz8Ip9b+8iiyQ1jExNuylC7ag67N1EBlAPS1xJLs2yDNPX+R3sDuZNhmOfGrCWHA6DAnA+x2w2GKHHnouNWe0gcjlAkbMOABzWJqY2wLUW073tjG7iYKcSnkTGP9BUliWxYhlBULPHBjgju2VFyccDUcb+xPYQcy5+Hpt2RQleYDdIbyQjQnCRTNZGKVbubUkOji3uS40di/tL/43SYX6Psrv81jfRACMMpv00ApnfK28F9ha9KSOnNutmYrjDb/gagO6oHh1bzpCimZumji2v+gmdiHeefeNgw8+KOTDJWgsfUaG+628db8naZKvuxG0dI5J3PXAM76TSSnJgA8EAPm2fzDYf9xX3ccx+NLcyBLgvFIQAgBn7d0XV4tMI8S/OPbY4Kd68ByS1orueyWbmpR3eOw4DO9fRmO5J6bdQVoPewmiIs53+ZBTK/Ad3ejUZFyXoCNhqCWzhfuNy5cntHaU0/tuCPT42zhvL3YB9LWunkxxwCyrm/DwC4VxATOpXFlaNJGNzNG1AYW1ZNHf6fIZVz5598oKD0iWoZzWfWb7gn1qBLSXtn0QvTR6QS60z0JHk0r2f2C1oPyZgCrScnej9+Vot/2woQ5UBx2o9ftJRACfhZgHQSxczQ7sD9Y20W+E4tLmX9i5TON20O83TVCFXZlAGxVP44AAkUbLyESwKz6X1t92c3yu39zWrB14u6LKnKMgx6WeczICvrCeuP0iwch3DYttMzUeZf1rm+tXMuRfVjh4sRsgvOWP63ZYDqBPrE9FrXcfkHtZZ6tSJFNIzyn7Sthl57sAiFXwH7PJWOnnMTtV3CP3pBH0zOZecmyhgtXnjEpfaJVHzsORvzILukPI JLCJ0UzH dtEXMqCAXh/TOUk1jz8xJ/tqdWh+sFL45dwdaIz+i3XxkLcC3reSJdAtp554IiqQbpyVUXSeDd+c0DmvD/kY4ggLrKjBExopJ56lic48mBiJcDL6520zpiim/FVg2nc5JwCTrFO2Bo7404cPZgsiyBX3RwgqUoXaXSSy2/d8V3FzLs6go9M9r2pzsG7aeghgGW49ltEoO+16rFmMkfPnfXYJ5absQNM9Ylb48nJ66zICUGWK5ZTs30IqBURuXBv+vO1TFGQcI9EOBd0XFJ4gjiDAhPUelDkydOh4pdQs5pZfbIfvuRJMRsYMNRLMoR8vtr0eamipbk/KLcWnXnkX9R13GevMLbjz9PibzBxF6q4kp62DU1Y0t5w8QGnnp9KNUGG2G24N3823hdj48zPQzbcyZM88yKSA1WkRe45zdPhDCoJw= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: On Mon, Oct 23, 2023 at 05:07:02PM -0700, Nhat Pham wrote: > Since: > > "42c06a0e8ebe mm: kill frontswap" > > we no longer have a counter to tracks the number of zswap store > failures. This makes it hard to investigate and monitor for zswap > issues. > > This patch adds a global and a per-cgroup zswap store failure counter, > as well as a dedicated debugfs counter for compression algorithm failure > (which can happen for e.g when random data are passed to zswap). > > Signed-off-by: Nhat Pham I agree this is an issue. > --- > include/linux/vm_event_item.h | 1 + > mm/memcontrol.c | 1 + > mm/vmstat.c | 1 + > mm/zswap.c | 18 ++++++++++++++---- > 4 files changed, 17 insertions(+), 4 deletions(-) > > diff --git a/include/linux/vm_event_item.h b/include/linux/vm_event_item.h > index 8abfa1240040..7b2b117b193d 100644 > --- a/include/linux/vm_event_item.h > +++ b/include/linux/vm_event_item.h > @@ -145,6 +145,7 @@ enum vm_event_item { PGPGIN, PGPGOUT, PSWPIN, PSWPOUT, > #ifdef CONFIG_ZSWAP > ZSWPIN, > ZSWPOUT, > + ZSWPOUT_FAIL, Would the writeback stat be sufficient to determine this? Hear me out. We already have pswpout that shows when we're hitting disk swap. Right now we can't tell if this is because of a rejection or because of writeback. With a writeback counter we could. And I think we want the writeback counter anyway going forward in order to monitor and understand the dynamic shrinker's performance. Either way we go, one of the metrics needs to be derived from the other(s). But I think subtle and not so subtle shrinker issues are more concerning than outright configuration problems where zswap doesn't work at all. The latter is easier to catch before or during early deployment with simple functionality tests. Plus, rejections should be rare. They are now, and they should become even more rare or cease to exist going forward. Because every time they happen at scale, they represent problematic LRU inversions. We have patched, have pending patches, or discussed changes to reduce every single one of them: /* Store failed due to a reclaim failure after pool limit was reached */ static u64 zswap_reject_reclaim_fail; With the shrinker this becomes less relevant. There was also the proposal to disable the bypass to swap and just keep the page. /* Compressed page was too big for the allocator to (optimally) store */ static u64 zswap_reject_compress_poor; You were working on eradicating this (with zsmalloc at least). /* Store failed because underlying allocator could not get memory */ static u64 zswap_reject_alloc_fail; /* Store failed because the entry metadata could not be allocated (rare) */ static u64 zswap_reject_kmemcache_fail; These shouldn't happen at all due to PF_MEMALLOC. IOW, the fail counter is expected to stay zero in healthy, well-configured systems. Rather than an MM event that needs counting, this strikes me as something that could be a WARN down the line... I agree with adding the debugfs counter though.