From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.subspace.kernel.org (Postfix) with ESMTP id D408B606A32A for ; Sat, 21 Nov 2026 12:11:44 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=chromium.org header.i=@chromium.org header.a=rsa-sha256 header.s=google header.b=EpowrZdN Received: by kanga.kvack.org (Postfix) id CE83F6B05B1; Sat, 21 Nov 2026 07:11:43 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id C98F66B05B2; Sat, 21 Nov 2026 07:11:43 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id B88406B05B3; Sat, 21 Nov 2026 07:11:43 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id 8E2086B05B1 for ; Sat, 21 Nov 2026 07:11:43 -0500 (EST) Received: from smtpin06.hostedemail.com (lb01a-stub [10.200.18.249]) by unirelay09.hostedemail.com (Postfix) with ESMTP id 208298049D for ; Sat, 21 Nov 2026 12:11:43 +0000 (UTC) X-FDA: 85459012086.06.5CB6B46 Received: from mail-pj1-f42.google.com (mail-pj1-f42.google.com [209.85.216.42]) by imf29.hostedemail.com (Postfix) with ESMTP id 3AAA212000B for ; Sat, 21 Nov 2026 12:11:41 +0000 (UTC) Authentication-Results: imf29.hostedemail.com; dkim=pass header.d=chromium.org header.s=google header.b=EpowrZdN; dmarc=pass (policy=none) header.from=chromium.org; spf=pass (imf29.hostedemail.com: domain of senozhatsky@chromium.org designates 209.85.216.42 as permitted sender) smtp.mailfrom=senozhatsky@chromium.org ARC-Seal: i=1; a=rsa-sha256; d=hostedemail.com; s=arc-20220608; cv=none; t=1795263101; b=nnU/zNHBXlfvXo4F6rPRCoTcFohF2gAcdVrsshvu15Z3O7CP8Iuyk2/r7UNaTl5eVtjWkx IYpD7wByyECWAveFJ+rDwBJj5+Sa9pwFP5cGHlwg1IH0t7eNLZK25A8GlR2SRrdBkOhP9x 6g9S78Lc9N/CQLMJbbIGgmygStkH6uU= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1795263101; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=F6qkXouGRbsp6kvGJBQv90f1f75Gxd//qB1z237SRcc=; b=d2y6GF+hFhf0m0/DwJD/sD7Bp8YjgZYKcmrwnLxPn+wAmKq/Ar9qI3VIvxSQ3ZyLTGMW3a +F2QqGvpnrX1hEafQBX3+CFGmfAv6qjZnwLxpBMZFisNEor5AlBPBgCc4nlf+MgTcQrxTP loWUT4YkG5m+u4a9afxQ+XchS0+JOuk= ARC-Authentication-Results: i=1; imf29.hostedemail.com; dkim=pass header.d=chromium.org header.s=google header.b=EpowrZdN; dmarc=pass (policy=none) header.from=chromium.org; spf=pass (imf29.hostedemail.com: domain of senozhatsky@chromium.org designates 209.85.216.42 as permitted sender) smtp.mailfrom=senozhatsky@chromium.org Received: by mail-pj1-f42.google.com with SMTP id 98e67ed59e1d1-3b0e0e65adbso816341a91.3 for ; Sat, 21 Nov 2026 04:11:40 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=chromium.org; s=google; t=1795263100; x=1795867900; darn=kvack.org; h=in-reply-to:content-disposition:content-type:mime-version :references:message-id:subject:cc:to:from:date:from:to:cc:subject :date:message-id:reply-to:content-type; bh=F6qkXouGRbsp6kvGJBQv90f1f75Gxd//qB1z237SRcc=; b=EpowrZdNq97VvOq7u+7yfByN2pApFcrWFYAAa8og6OYzvlJWWvZtQvCHkbCKIxycER 7a3GO4y7SDzyHigK11fLhW3lYIKhvqemYDVPBVn9q9yixNoFxO7EiYPWFBFhmiM5Qws/ w9HluKpjUef6bfuqnJ4YYruumlb+NNJjDf8Do= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20260707; t=1795263100; x=1795867900; h=in-reply-to:content-disposition:content-type:mime-version :references:message-id:subject:cc:to:from:date:x-gm-gg :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to :content-type; bh=F6qkXouGRbsp6kvGJBQv90f1f75Gxd//qB1z237SRcc=; b=DMLXapImIiVorIcg+nSO7s+RGUuKhswJA0q0VObEyssGAr5DxjB9asfHyTOeSUUjNv jQOM+WDXmU7SEEPTd/M2U0TsH6l6cD9udFnfywPO9LePi3fdAT9GNpmwcwq+631tsZHk Oy6TUgdQ5KwS5QVu3mMAc7BL2Q2z4SXmm+9L1IB0gD0vrJLfls0jnZqp1nIdxMcddZ1H k/urFqhtLxq/E6+ue8Pw9yOzpnAQ6EURNa00AMmaORSszoBfInqhWgh4FvgGIQaG8QpP od9l2V3v6DlqgD5H2pOO2Puy7UtpBMQXklgOMpstiZX3dv2DUfBrE+fPXouoOnGWvS5d VJXA== X-Forwarded-Encrypted: i=1; AL+hErEG1qqFIrlzXv+fvaiwRCqFQfQCUrzJ341HWBIxzUFMZAYt6NqWAFY0NYtbNEIn1pRHnl3OlLcwXQ==@kvack.org X-Gm-Message-State: APuU4Z/yI5Rb6efjKUhnZw9jwR7LJwdo3TpwloMkqK/3eq7OXjzb0UUA eOtAkKtvWRDzF3Pkx1UqXvFJrxc3Zjg6K3iJWFS07QivViz3Yy/09SEdZSBXFk4hZA== X-Gm-Gg: AZFdTZcz3JnTEp6n8MvKuTJ4oDV8klRnc7Ziea+9Ki3uF8VgdGGBh0q1to8krKTTIZI AC1HokeNkeWHBbQpQl9GpALRyoskhnka6jqW54/NFmwnYlBE37FFmsXnxb1doUsflRGNPVT3sJZ lCbMDJrLVZI8+c4SIBIQuu7ChKSPuHXsKUYucVOAcDDMri+jwFYa2Jy6lW8V1q6cYNr6IY/bEeM acL3AJ9fPw+u8VbkOL7t3NgdnCaUUO1k+Tx+NDF6/b9EkYQ81TylgzrligGbpCGUXbaavkzPAdh YrmSrff0MR/quoRdvZqzqasK9qQJifd9DrLVSbUv6QvotuKV+mAN13RfODRFuMNWYmHMKYtHrBf xRucCqWd6FXWKU2qegz2CeO9lcAt3WLXIrq8K4f9gBvqb7iCd1iEBIjFWKlJBappcIQgr1aWl18 CG1/nKJeI8P7WASoRmjeHZPRLcDsBNr6dMCkhvED21PNygnPa0T9Kex2PO/QrhjhivWf5zRUVE0 Mj7jFB3sOEpmSc9SHsW4yQYyxGbjkExwWSgejAbkLZXM7whViqPSXYEGlArUXGa38CUzocvIpmU S/MtoeYYSSN0MAtGC04IQypg+9vrw+JSsDrQrBOA2+NWJVLx9LhjFl/DpScg92r8qigKJ7JjsRe j9UeUmyohGQQavFI= X-Received: by 2002:a17:90b:1fcc:b0:3ab:b39f:41c8 with SMTP id 98e67ed59e1d1-3b41c9481b7mr5245450a91.46.1795263099805; Sat, 21 Nov 2026 04:11:39 -0800 (PST) Received: from google.com ([2a00:79e0:2031:6:2f43:ed99:516b:8dcd]) by smtp.gmail.com with ESMTPSA id 41be03b00d2f7-cde6d03adafsm2653866a12.23.2026.11.21.04.11.35 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sat, 21 Nov 2026 04:11:38 -0800 (PST) Date: Sat, 21 Nov 2026 21:11:33 +0900 From: Sergey Senozhatsky To: Sergey Senozhatsky Cc: Hongfeng Chen , Andrew Morton , Barry Song , Chengming Zhou , Jens Axboe , Johannes Weiner , Minchan Kim , Nhat Pham , Yosry Ahmed , Wenchao Hao , Wenchao Hao , linux-mm@kvack.org, linux-kernel@vger.kernel.org, linux-block@vger.kernel.org, chenhongfeng0212@gmail.com Subject: Re: [RFC PATCH v4 2/2] zram: keep the hot statistics per cpu Message-ID: References: <20261016120927.3191690-1-hongfengchen@xiaomi.com> <20261017031542.3504947-1-hongfengchen@xiaomi.com> X-Archived-By: MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: X-Rspam-User: X-Rspamd-Server: rspam12 X-Spam-Status: No, score=-10.8 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_PASS,SPF_PASS autolearn=ham autolearn_force=no version=4.0.1 X-Rspamd-Queue-Id: 3AAA212000B X-Stat-Signature: gqfzzbeuoe61b7p1zdczc34xnx1jpxbn X-HE-Tag: 1795263101-132789 X-HE-Meta: U2FsdGVkX1/alYhDaJPmJj0CyFgB3WEl4zlAFn1LPiQT4QGZeoRGZaPpiCb3SsX2s2kxoCZRwEDalV1830jpToWaNOUQl1ny5Jz5NFiT3GobYWljOShFzuQaARbT8ij7kvGBcQQ85zYVq0HmfCLSx0WnP7d7HPDWPhLEFXFouu4v+hT9lQT7JZFLBQ/g/6Ka68nH+e6bKAvEWK8EfwtNJIBGH22SUraA9ChfpTEtLv4Hl5iBXMcKv1syke/Fam5UuwtntmxaHyUZ/IB3Jr9EO5n5kfhCEAuz7B9n/qxxlzKNYU40HLronY8EIL1MtPm+0lJBefGdsfVikggfWBCMqmM1GoIkqiskU+e5+0IK+aeiQ2M431cwc1Qo/2T7fNpgfOZC4tfGKr65O04VnVhrcB/yznnLxwsnmnVdjIewuQuNKhMsGSqiJotoRWsQIYQLJm1wAs/0KX2AXugt6+FxC2FovUdpStg4nXWxdlbzdcf5/E919z+JtzPkNNTc/uEXSVoqeTDc9Nr28y4gt4sZL+v1RK0ddLUL7vHBtiFPaH4C98UfEGaDJ+BuknMZ/846t9evDb7uCbiRYjaPrPfSr1BtZkeupjYi9/fJWc330aY7OglEkpQKEj343S6/ru9d23H36wcwN9bhD7UXYqrBVxVayI8xIhc6R6CG+oClvDSLzOa8lQXBeSmWILl2UQyRf+scF5hCCMxc4i/FtBTzaQ+vLx2qsOBX05klsUHODbSS4isBJ0O/Rcy3GY0WJWKSXBvbu3j6PKsatZ9GnxUbsOLAjiyNgQrzDF7wukufEt/W39GMXmJ9UtxVsNHP8uPFToyAJyjyt22CG9/OK8NqttuBRRlQKRPQ8NdjGPUmVR+L4eAgKjwpSumb/Ca93lSPLv8Mk2UhGcApF1crqXRoIB12zbTna6HCHaJ+hqXBCq79WoVL1vcES8rKIbmTK/2PZDcdlJNDOFjRTcCHXjv /7zeqGJj ScBZfK6+FUvbqGp7/okcTPiWRf2Z679q9lnlOWxNsA5O/O3KHwAYM2dlaqYGZj2AdbKbxa4R//6YxYF6VCpDYppGQXzfkQs0vRL/7uQudCVLfsQV+wa1h55JAIb00DwYRxYLOOSWaYo9nWlR/xLifAYOiimGU4aTxFLE5eYjtGooMEp0uF5dRXVNCdIwR+RAXxKzQGz9D3MHv0W/tmuN1uizp/PtSy4qkq4FRmDgG6nsmc5mKpViV8cpLnooINGPfmSvPEb4EUh6Huo6zugKpkht1/VZOqt+3IdpGzr2ZqKaGoFpqMHixgARfpiVk8ADPF4p3kn2XylHGqMLvueBnZrV9AtATXQrJwra8UEZ4LTCpcqVsPXZXm80cMqxbaktMtD5HXQn2FPoTwikU9iX5nFl1xj+ewMD9/UBHD6fdG1Oqpn2rP391u4ulmq3by/hySOK4cMhsQXBScfVOllesKncSTx/S8OcWdFhPbl3vG8jJUQb+dsi5nRVJ5LAbeBKFpBlgD5UYAeZvEdk1MTAN0p6IDLaiJB7A/q3+JmL107eSVFpJaJcwWxEYb6H1svqKwKiCXsjmJRBzydQ= Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: X-Spam-Checker-Version: SpamAssassin 4.0.1 (2024-03-26) on aws-us-west-2-korg-listserv-alma10-1 On (26/11/21 17:35), Sergey Senozhatsky wrote: [..] > > clear_slot_flag(zram, index, ZRAM_IDLE); > > - if (test_slot_flag(zram, index, ZRAM_HUGE)) > > - atomic64_dec(&zram->stats.huge_pages); > > - atomic64_sub(get_slot_size(zram, index), &zram->stats.compr_data_size); > > + huge = test_slot_flag(zram, index, ZRAM_HUGE) ? -1 : 0; > > + size = get_slot_size(zram, index); > > + stats = get_cpu_ptr(zram->pcpu_stats); > > + u64_stats_update_begin(&stats->syncp); > > + WRITE_ONCE(stats->huge_pages, stats->huge_pages + huge); > > + WRITE_ONCE(stats->compr_data_size, stats->compr_data_size - size); > > + u64_stats_update_end(&stats->syncp); > > + put_cpu_ptr(zram->pcpu_stats); > > Thanks for reminding me that I actually looked into per-CPU stats a while > ago. And I had much less code, IIRC, let me find and rebase that patch, > these WRITE_ONCE()-s, etc. are hard to maintain. Let me find my patch. So this is what I had, basically replicating zsmalloc stats array. --- drivers/block/zram/zram_drv.c | 108 ++++++++++++++++++++++++++-------- drivers/block/zram/zram_drv.h | 26 ++++++-- 2 files changed, 102 insertions(+), 32 deletions(-) diff --git a/drivers/block/zram/zram_drv.c b/drivers/block/zram/zram_drv.c index 54eb691f1409..951a38caeb93 100644 --- a/drivers/block/zram/zram_drv.c +++ b/drivers/block/zram/zram_drv.c @@ -95,6 +95,47 @@ static void slot_unlock(struct zram *zram, unsigned long index) clear_and_wake_up_bit(ZRAM_ENTRY_LOCK_BIT, lock); } +static void zram_stat_add(struct zram *zram, enum zram_pcpu_stat item, + s64 amount) +{ + struct zram_pcpu_stats *stats = get_cpu_ptr(zram->pcpu_stats); + + local64_add(amount, &stats->stat[item]); + put_cpu_ptr(zram->pcpu_stats); +} + +static void zram_stat_sub(struct zram *zram, enum zram_pcpu_stat item, + s64 amount) +{ + struct zram_pcpu_stats *stats = get_cpu_ptr(zram->pcpu_stats); + + local64_sub(amount, &stats->stat[item]); + put_cpu_ptr(zram->pcpu_stats); +} + +static u64 zram_stat_read(struct zram *zram, enum zram_pcpu_stat item) +{ + struct zram_pcpu_stats *stats; + s64 sum; + int cpu; + + /* + * Other CPUs keep updating while the values are added up: a page + * stored on a CPU already summed and freed on one not yet summed + * can make the sum negative. Sum them up again then. + */ + for (;;) { + sum = 0; + for_each_possible_cpu(cpu) { + stats = per_cpu_ptr(zram->pcpu_stats, cpu); + sum += local64_read(&stats->stat[item]); + } + if (sum >= 0) + return sum; + cond_resched(); + } +} + static inline bool init_done(struct zram *zram) { return zram->disksize; @@ -944,8 +985,9 @@ static int zram_writeback_complete(struct zram *zram, struct zram_wb_req *req) clear_slot_flag(zram, index, ZRAM_IDLE); if (test_slot_flag(zram, index, ZRAM_HUGE)) - atomic64_dec(&zram->stats.huge_pages); - atomic64_sub(get_slot_size(zram, index), &zram->stats.compr_data_size); + zram_stat_sub(zram, ZRAM_STAT_HUGE_PAGES, 1); + zram_stat_sub(zram, ZRAM_STAT_COMPR_DATA_SIZE, + get_slot_size(zram, index)); zs_free(zram->mem_pool, &zram->table[index].handle); zram->table[index].blk_idx = req->blk_idx; set_slot_flag(zram, index, ZRAM_WB); @@ -1927,7 +1969,7 @@ static ssize_t io_stat_show(struct device *dev, struct device_attribute *attr, "%8llu %8llu 0 %8llu\n", (u64)atomic64_read(&zram->stats.failed_reads), (u64)atomic64_read(&zram->stats.failed_writes), - (u64)atomic64_read(&zram->stats.notify_free)); + zram_stat_read(zram, ZRAM_STAT_NOTIFY_FREE)); return ret; } @@ -1949,20 +1991,20 @@ static ssize_t mm_stat_show(struct device *dev, struct device_attribute *attr, zs_pool_stats(zram->mem_pool, &pool_stats); } - orig_size = atomic64_read(&zram->stats.pages_stored); + orig_size = zram_stat_read(zram, ZRAM_STAT_PAGES_STORED); max_used = atomic_long_read(&zram->stats.max_used_pages); ret = sysfs_emit(buf, "%8llu %8llu %8llu %8lu %8ld %8llu %8lu %8llu %8llu\n", orig_size << PAGE_SHIFT, - (u64)atomic64_read(&zram->stats.compr_data_size), + zram_stat_read(zram, ZRAM_STAT_COMPR_DATA_SIZE), mem_used << PAGE_SHIFT, zram->limit_pages << PAGE_SHIFT, max_used << PAGE_SHIFT, - (u64)atomic64_read(&zram->stats.same_pages), + zram_stat_read(zram, ZRAM_STAT_SAME_PAGES), atomic_long_read(&pool_stats.pages_compacted), - (u64)atomic64_read(&zram->stats.huge_pages), - (u64)atomic64_read(&zram->stats.huge_pages_since)); + zram_stat_read(zram, ZRAM_STAT_HUGE_PAGES), + zram_stat_read(zram, ZRAM_STAT_HUGE_PAGES_SINCE)); return ret; } @@ -2045,7 +2087,7 @@ static void slot_free(struct zram *zram, unsigned long index) * ZRAM_HUGE flag for deferred decompression path. */ if (!test_slot_flag(zram, index, ZRAM_WB)) - atomic64_dec(&zram->stats.huge_pages); + zram_stat_sub(zram, ZRAM_STAT_HUGE_PAGES, 1); clear_slot_flag(zram, index, ZRAM_HUGE); } @@ -2061,7 +2103,7 @@ static void slot_free(struct zram *zram, unsigned long index) */ if (test_slot_flag(zram, index, ZRAM_SAME)) { clear_slot_flag(zram, index, ZRAM_SAME); - atomic64_dec(&zram->stats.same_pages); + zram_stat_sub(zram, ZRAM_STAT_SAME_PAGES, 1); goto out; } @@ -2071,10 +2113,10 @@ static void slot_free(struct zram *zram, unsigned long index) zs_free(zram->mem_pool, handle); - atomic64_sub(get_slot_size(zram, index), - &zram->stats.compr_data_size); + zram_stat_sub(zram, ZRAM_STAT_COMPR_DATA_SIZE, + get_slot_size(zram, index)); out: - atomic64_dec(&zram->stats.pages_stored); + zram_stat_sub(zram, ZRAM_STAT_PAGES_STORED, 1); zram->table[index].element = 0; set_slot_size(zram, index, 0); } @@ -2246,8 +2288,8 @@ static int write_same_filled_page(struct zram *zram, unsigned long fill, mark_slot_accessed(zram, index); slot_unlock(zram, index); - atomic64_inc(&zram->stats.same_pages); - atomic64_inc(&zram->stats.pages_stored); + zram_stat_add(zram, ZRAM_STAT_SAME_PAGES, 1); + zram_stat_add(zram, ZRAM_STAT_PAGES_STORED, 1); return 0; } @@ -2277,10 +2319,10 @@ static int write_incompressible_page(struct zram *zram, struct page *page, mark_slot_accessed(zram, index); slot_unlock(zram, index); - atomic64_add(PAGE_SIZE, &zram->stats.compr_data_size); - atomic64_inc(&zram->stats.huge_pages); - atomic64_inc(&zram->stats.huge_pages_since); - atomic64_inc(&zram->stats.pages_stored); + zram_stat_add(zram, ZRAM_STAT_COMPR_DATA_SIZE, PAGE_SIZE); + zram_stat_add(zram, ZRAM_STAT_HUGE_PAGES, 1); + zram_stat_add(zram, ZRAM_STAT_HUGE_PAGES_SINCE, 1); + zram_stat_add(zram, ZRAM_STAT_PAGES_STORED, 1); return 0; } @@ -2340,8 +2382,8 @@ static int zram_write_page(struct zram *zram, struct page *page, slot_unlock(zram, index); /* Update stats */ - atomic64_inc(&zram->stats.pages_stored); - atomic64_add(comp_len, &zram->stats.compr_data_size); + zram_stat_add(zram, ZRAM_STAT_PAGES_STORED, 1); + zram_stat_add(zram, ZRAM_STAT_COMPR_DATA_SIZE, comp_len); return ret; } @@ -2532,8 +2574,8 @@ static int recompress_slot(struct zram *zram, unsigned long index, set_slot_size(zram, index, comp_len_new); set_slot_comp_priority(zram, index, prio); - atomic64_add(comp_len_new, &zram->stats.compr_data_size); - atomic64_inc(&zram->stats.pages_stored); + zram_stat_add(zram, ZRAM_STAT_COMPR_DATA_SIZE, comp_len_new); + zram_stat_add(zram, ZRAM_STAT_PAGES_STORED, 1); return 0; } @@ -2711,7 +2753,7 @@ static void zram_bio_discard(struct zram *zram, struct bio *bio) slot_lock(zram, index); slot_free(zram, index); slot_unlock(zram, index); - atomic64_inc(&zram->stats.notify_free); + zram_stat_add(zram, ZRAM_STAT_NOTIFY_FREE, 1); index++; n -= PAGE_SIZE; } @@ -2804,7 +2846,7 @@ static void zram_slot_free_notify(struct block_device *bdev, zram = bdev->bd_disk->private_data; - atomic64_inc(&zram->stats.notify_free); + zram_stat_add(zram, ZRAM_STAT_NOTIFY_FREE, 1); if (!slot_trylock(zram, index)) { atomic64_inc(&zram->stats.miss_free); return; @@ -2845,6 +2887,8 @@ static void zram_destroy_comps(struct zram *zram) static void zram_reset_device(struct zram *zram) { + int cpu; + guard(rwsem_write)(&zram->dev_lock); zram->limit_pages = 0; @@ -2857,6 +2901,9 @@ static void zram_reset_device(struct zram *zram) zram->disksize = 0; zram_destroy_comps(zram); memset(&zram->stats, 0, sizeof(zram->stats)); + for_each_possible_cpu(cpu) + memset(per_cpu_ptr(zram->pcpu_stats, cpu), 0, + sizeof(struct zram_pcpu_stats)); reset_bdev(zram); #ifdef CONFIG_ZRAM_WRITEBACK zram->wb_limit_enable = false; @@ -3075,9 +3122,15 @@ static int zram_add(void) if (!zram) return -ENOMEM; + zram->pcpu_stats = alloc_percpu(struct zram_pcpu_stats); + if (!zram->pcpu_stats) { + ret = -ENOMEM; + goto out_free_dev; + } + ret = idr_alloc(&zram_index_idr, zram, 0, 0, GFP_KERNEL); if (ret < 0) - goto out_free_dev; + goto out_free_stats; device_id = ret; init_rwsem(&zram->dev_lock); @@ -3119,6 +3172,8 @@ static int zram_add(void) put_disk(zram->disk); out_free_idr: idr_remove(&zram_index_idr, device_id); +out_free_stats: + free_percpu(zram->pcpu_stats); out_free_dev: kfree(zram); return ret; @@ -3168,6 +3223,7 @@ static int zram_remove(struct zram *zram) zram_reset_device(zram); put_disk(zram->disk); + free_percpu(zram->pcpu_stats); kfree(zram); return 0; } diff --git a/drivers/block/zram/zram_drv.h b/drivers/block/zram/zram_drv.h index 724f8ce6709b..a3fa6e3d007f 100644 --- a/drivers/block/zram/zram_drv.h +++ b/drivers/block/zram/zram_drv.h @@ -10,6 +10,7 @@ #define _ZRAM_DRV_H_ #include +#include #include #include @@ -88,15 +89,27 @@ struct zram_table_entry { }; }; +/* + * Updated for every page that is stored or freed, so kept per CPU: with + * shared counters, concurrent swap frees bounce their cacheline. + */ +enum zram_pcpu_stat { + ZRAM_STAT_PAGES_STORED, /* no. of pages currently stored */ + ZRAM_STAT_COMPR_DATA_SIZE, /* compressed size of pages stored */ + ZRAM_STAT_SAME_PAGES, /* no. of same element filled pages */ + ZRAM_STAT_HUGE_PAGES, /* no. of huge pages */ + ZRAM_STAT_HUGE_PAGES_SINCE, /* no. of huge pages since set up */ + ZRAM_STAT_NOTIFY_FREE, /* no. of slot free notifications */ + NR_ZRAM_PCPU_STATS, +}; + +struct zram_pcpu_stats { + local64_t stat[NR_ZRAM_PCPU_STATS]; +}; + struct zram_stats { - atomic64_t compr_data_size; /* compressed size of pages stored */ atomic64_t failed_reads; /* can happen when memory is too low */ atomic64_t failed_writes; /* can happen when memory is too low */ - atomic64_t notify_free; /* no. of swap slot free notifications */ - atomic64_t same_pages; /* no. of same element filled pages */ - atomic64_t huge_pages; /* no. of huge pages */ - atomic64_t huge_pages_since; /* no. of huge pages since zram set up */ - atomic64_t pages_stored; /* no. of pages currently stored */ atomic_long_t max_used_pages; /* no. of maximum pages stored */ atomic64_t miss_free; /* no. of missed free */ #ifdef CONFIG_ZRAM_WRITEBACK @@ -132,6 +145,7 @@ struct zram { unsigned long limit_pages; struct zram_stats stats; + struct zram_pcpu_stats __percpu *pcpu_stats; /* * This is the limit on amount of *uncompressed* worth of data * we can store in a disk. -- 2.56.1.750.gf70dc682cf-goog