From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 6E89AC4332F for ; Fri, 3 Nov 2023 16:04:33 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Cc:To:From:Subject:Message-ID: References:Mime-Version:In-Reply-To:Date:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=Af3+vnvbnoMoWymAwIPvXwFjMxM3a/OSl+earHXXMWk=; b=CHzAFSdEAlTzWUuGLTgnu7EVrb D725ACaSeGaYFhl0u+Ed3Ly8va867+OgUgIGeBkCLCEB+iD6KGJtR9H7Q1gohcHjPhADwJ3dF+ye4 KzAtrlZPrH/NOHCOzM51U0gZTFgw6iCZ1L06oyDe6M407r8cVhfrLOVxdulUwiRHnp8UnFWpLRqoD /0+HMDb+dbrrLwsosgoV90xZg8+5Armhgan06R7ZsgplrXHnF3mJ3YZAWNNXJf4dhmOgfPqjSfR9R FKUVMxDOWa1uF15i8eimGwCsw7kAOOykWYtGaq+fnOnIp1LNQMN2HeP2CLOgy/1KHmEtUlzGNeNOp HwMc2Gaw==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1qyweG-00Bksc-0F; Fri, 03 Nov 2023 16:03:52 +0000 Received: from mail-yb1-xb4a.google.com ([2607:f8b0:4864:20::b4a]) by bombadil.infradead.org with esmtps (Exim 4.96 #2 (Red Hat Linux)) id 1qyweD-00BkqH-1A for linux-arm-kernel@lists.infradead.org; Fri, 03 Nov 2023 16:03:50 +0000 Received: by mail-yb1-xb4a.google.com with SMTP id 3f1490d57ef6-d86dac81f8fso2654153276.1 for ; Fri, 03 Nov 2023 09:03:48 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1699027428; x=1699632228; darn=lists.infradead.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=kakEavv2Bx4kC7ui8vSjgo++eh6M+uQZM7WF3AfKH1U=; b=vAV1pN9k0GJ4SaxA/KrkgtDrkfUREk3sTaxZUgFQCOqicZG/XRDQpPN2Ps49hcnW/Z TUq4bxNwftPncKThYwAcyf+xJ266zbsPWdYtPBvKppZb2JNQbz0R9eYanwRyfHBz++1U z3rIR+eX+0rZVRjpc8LPpDPxdnfCnqxnj+2uNPWxWe7cI/xUJkC3ekjhFAgOariIPotd +Jf2sl2X9pKj+RpMp6JzGkJFam/eE/OkkgC29sgWBmLi9kvAAwBtu98Fyam/dq6ztxXa QdLQ947l8mJ6ztRR+xz2b962ZC1poGH+Vlk99NHaok3/yeBWwDH6cKSX3lmTgiKBUu72 yPGQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1699027428; x=1699632228; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=kakEavv2Bx4kC7ui8vSjgo++eh6M+uQZM7WF3AfKH1U=; b=jiUQgO3ZmyVh+qEPCD05Nhwd8X1QgSTqC30CKYcQKUWQcbQvG9MOIZ7Tqegm+k3KLa DdnburkvK4pgiBFZTu/w99bAh7Iz7dWs/V7ZMgdLU4n9EGQPsBHgalT5l0bB9EIdYJki ozyp3Zet3Q6FuR+2uuOKTO/R458WQAXkCIZWIJb22mCj33L+eftGbUvXUvN9YEkyo2IM 4PswzESIh6tpqye9RbxG7Nh9V/A0AdP3cNAeeXor5QfXX6GLux7iopaWoW9RwxhHODRv wqaiLRmKaUMhK2GWBSlr6xN3Z23yIkp/OaljTBDkQWF6TRdDnn/4JUi1L1q5l8/1XrUS Lb/w== X-Gm-Message-State: AOJu0YygURvwqJLuiVP5pfYllmvIXCWq0XBKt6icyRuGEAhkRbC36D99 KT0GvdOp0SQlsjg43L9oRYKZ1fM5ghU= X-Google-Smtp-Source: AGHT+IHR9RZ3AGRYHVoGIfBerEYYQF7lrQOl7qGnkNHzxbQ4hBmSQ407bgV+B4b0VM3Hx1nwqixE0RPesMI= X-Received: from glider.muc.corp.google.com ([2a00:79e0:9c:201:74c:1f8e:4661:7aaa]) (user=glider job=sendgmr) by 2002:a25:770f:0:b0:da0:73c2:db78 with SMTP id s15-20020a25770f000000b00da073c2db78mr452259ybc.9.1699027428174; Fri, 03 Nov 2023 09:03:48 -0700 (PDT) Date: Fri, 3 Nov 2023 17:03:35 +0100 In-Reply-To: <20231103160335.2464561-1-glider@google.com> Mime-Version: 1.0 References: <20231103160335.2464561-1-glider@google.com> X-Mailer: git-send-email 2.42.0.869.gea05f2083d-goog Message-ID: <20231103160335.2464561-4-glider@google.com> Subject: [PATCH v8 3/3] arm64: mte: add compression support to mteswap.c From: Alexander Potapenko To: glider@google.com, catalin.marinas@arm.com, will@kernel.org, pcc@google.com, andreyknvl@gmail.com, andriy.shevchenko@linux.intel.com, aleksander.lobakin@intel.com, linux@rasmusvillemoes.dk, yury.norov@gmail.com, alexandru.elisei@arm.com Cc: linux-kernel@vger.kernel.org, linux-arm-kernel@lists.infradead.org, eugenis@google.com, syednwaris@gmail.com, william.gray@linaro.org X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20231103_090349_401583_A7400638 X-CRM114-Status: GOOD ( 21.34 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org Update mteswap.c to perform inline compression of memory tags when possible. If CONFIG_ARM64_MTE_COMP is enabled, mteswap.c will attemt to compress saved tags for a struct page and store them directly in Xarray entry instead of wasting heap space. Soon after booting Android, tag compression saves ~2x memory previously spent by mteswap.c on tag allocations. On a moderately loaded device with ~20% tagged pages, this leads to saving several megabytes of kernel heap: # cat /sys/kernel/debug/mteswap/stats 8 bytes: 102496 allocations, 67302 deallocations 128 bytes: 212234 allocations, 178278 deallocations uncompressed tag storage size: 8851200 compressed tag storage size: 4346368 Signed-off-by: Alexander Potapenko --- v8: - adapt to the new compression API, abandon mteswap_{no,}comp.c - move stats collection to mteswap.c v5: - drop a dead variable from _mte_free_saved_tags() in mteswap_comp.c - ensure MTE compression works with arbitrary page sizes - update patch description v4: - minor code simplifications suggested by Andy Shevchenko, added missing header dependencies - changed compression API names to reflect modifications made to memcomp.h (as suggested by Yury Norov) v3: - Addressed comments by Andy Shevchenko in another patch: - fixed includes order - replaced u64 with unsigned long - added MODULE_IMPORT_NS(MTECOMP) --- arch/arm64/mm/mteswap.c | 88 +++++++++++++++++++++++++++++++++++++++-- 1 file changed, 84 insertions(+), 4 deletions(-) diff --git a/arch/arm64/mm/mteswap.c b/arch/arm64/mm/mteswap.c index a31833e3ddc54..0f558942d88b8 100644 --- a/arch/arm64/mm/mteswap.c +++ b/arch/arm64/mm/mteswap.c @@ -1,28 +1,48 @@ // SPDX-License-Identifier: GPL-2.0-only +#include #include #include #include #include #include #include +#include +#include "mtecomp.h" + +enum mteswap_counters { + MTESWAP_CTR_INLINE = 0, + MTESWAP_CTR_NOINLINE, + MTESWAP_CTR_SIZE +}; +static atomic_long_t alloc_counters[MTESWAP_CTR_SIZE]; +static atomic_long_t dealloc_counters[MTESWAP_CTR_SIZE]; static DEFINE_XARRAY(mte_pages); void *mte_allocate_tag_storage(void) { - /* tags granule is 16 bytes, 2 tags stored per byte */ - return kmalloc(MTE_PAGE_TAG_STORAGE, GFP_KERNEL); + void *ret; + + ret = kmalloc(MTE_PAGE_TAG_STORAGE, GFP_KERNEL); + if (ret) + atomic_long_inc(&alloc_counters[MTESWAP_CTR_NOINLINE]); + return ret; } void mte_free_tag_storage(char *storage) { - kfree(storage); + if (!mte_is_compressed(storage)) { + kfree(storage); + atomic_long_dec(&alloc_counters[MTESWAP_CTR_NOINLINE]); + } else { + atomic_long_dec(&alloc_counters[MTESWAP_CTR_INLINE]); + } } int mte_save_tags(struct page *page) { - void *tag_storage, *ret; + void *tag_storage, *ret, *compressed; if (!page_mte_tagged(page)) return 0; @@ -32,6 +52,12 @@ int mte_save_tags(struct page *page) return -ENOMEM; mte_save_page_tags(page_address(page), tag_storage); + compressed = mte_compress(tag_storage); + if (compressed) { + mte_free_tag_storage(tag_storage); + tag_storage = (void *)compressed; + atomic_long_inc(&alloc_counters[MTESWAP_CTR_INLINE]); + } /* lookup the swap entry.val from the page */ ret = xa_store(&mte_pages, page_swap_entry(page).val, tag_storage, @@ -50,13 +76,20 @@ int mte_save_tags(struct page *page) void mte_restore_tags(swp_entry_t entry, struct page *page) { void *tags = xa_load(&mte_pages, entry.val); + void *tag_storage = NULL; if (!tags) return; if (try_page_mte_tagging(page)) { + if (mte_is_compressed(tags)) { + tag_storage = mte_allocate_tag_storage(); + mte_decompress(tags, tag_storage); + tags = tag_storage; + } mte_restore_page_tags(page_address(page), tags); set_page_mte_tagged(page); + mte_free_tag_storage(tag_storage); } } @@ -83,3 +116,50 @@ void mte_invalidate_tags_area(int type) } xa_unlock(&mte_pages); } + +/* DebugFS interface. */ +static int stats_show(struct seq_file *seq, void *v) +{ + unsigned long total_mem_alloc = 0, total_mem_dealloc = 0; + unsigned long total_num_alloc = 0, total_num_dealloc = 0; + unsigned long sizes[2] = { 8, MTE_PAGE_TAG_STORAGE }; + long alloc, dealloc; + unsigned long size; + int i; + + for (i = 0; i < MTESWAP_CTR_SIZE; i++) { + alloc = atomic_long_read(&alloc_counters[i]); + dealloc = atomic_long_read(&dealloc_counters[i]); + total_num_alloc += alloc; + total_num_dealloc += dealloc; + size = sizes[i]; + /* + * Do not count 8-byte buffers towards compressed tag storage + * size. + */ + if (i) { + total_mem_alloc += (size * alloc); + total_mem_dealloc += (size * dealloc); + } + seq_printf(seq, + "%lu bytes: %lu allocations, %lu deallocations\n", + size, alloc, dealloc); + } + seq_printf(seq, "uncompressed tag storage size: %lu\n", + (total_num_alloc - total_num_dealloc) * + MTE_PAGE_TAG_STORAGE); + seq_printf(seq, "compressed tag storage size: %lu\n", + total_mem_alloc - total_mem_dealloc); + return 0; +} +DEFINE_SHOW_ATTRIBUTE(stats); + +static int mteswap_init(void) +{ + struct dentry *mteswap_dir; + + mteswap_dir = debugfs_create_dir("mteswap", NULL); + debugfs_create_file("stats", 0444, mteswap_dir, NULL, &stats_fops); + return 0; +} +module_init(mteswap_init); -- 2.42.0.869.gea05f2083d-goog _______________________________________________ linux-arm-kernel mailing list linux-arm-kernel@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-arm-kernel