From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 277A1C4332F for ; Thu, 14 Dec 2023 11:07:34 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Cc:To:From:Subject:Message-ID: References:Mime-Version:In-Reply-To:Date:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=bBCH/MBNqhTlvz+1jqTjXLIaYZKzQNTUQzf0JxEPrjU=; b=t5VRxsOPouZPufwFWV6we+buoi WR8W7gIivpf7taoM0Rbgzr+vnKe7+bg/WFYEyDYIZAISDOtLD8DvY9YctG8SOHgYn4mClXFTv3mwL PUpoUldIv7G/5VhJU+JOUnlbPWrBHTPDJNz5brw2Xeg8eIhmibXUw5HLAd3RpvAcePfOhiaCJrvKZ RikZSmL/bfMVJrLZ2sQTIDfs2X68/qr+121YfdXyz1Pog9k3B7Bpz4YAlva5C8IEvisRf/fvwuJtN J0BMZGTjQZU5Q98tdqdqT2cX/gg4DVhYx//G+I95xUJcLyBfZ2sdMQRgKIoI2clFyybsZshxYggH6 qpIy8+SQ==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1rDjYa-00HZz8-0w; Thu, 14 Dec 2023 11:07:08 +0000 Received: from mail-yw1-x1149.google.com ([2607:f8b0:4864:20::1149]) by bombadil.infradead.org with esmtps (Exim 4.96 #2 (Red Hat Linux)) id 1rDjYW-00HZwf-0P for linux-arm-kernel@lists.infradead.org; Thu, 14 Dec 2023 11:07:05 +0000 Received: by mail-yw1-x1149.google.com with SMTP id 00721157ae682-5caf86963ecso89075347b3.3 for ; Thu, 14 Dec 2023 03:07:03 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1702552022; x=1703156822; darn=lists.infradead.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=qETeNVW4T3lC++QM3orD9LTmmtp3iyUET5aUKhJnmac=; b=z/VW9h1ICU6rwIJgm5MryOUXf01tl6tug/JGuXerbDZjH1ACB2j8TCP8U5AzfQKRqD m5EVOWZyAcn53GG/vn8CbdCOzo6HS6yGnuEGSWQb4GPt+4XEhQbKuc11qzIQAqvpvbDJ MuVpFz47IhhmW+daAhLwe8wfLbcwo4UooymycIvs/Qdbtnv7dDKy0M7uLXyvfj5RUGA+ wxzkbGDULaT5cmr+MzAqwcyYCHEzaKvh6+t+WLCW5geaHIaTFrqQmwQE+SRS94cdgdzL MFW2vFugZ1Kxdyvsf6WPewAMjvU7avkI7OjhaWufhFQ2ZFQ6si0w7eR2PUYJbu3gsGxx 9IEg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1702552022; x=1703156822; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=qETeNVW4T3lC++QM3orD9LTmmtp3iyUET5aUKhJnmac=; b=cFQVhAWbUbUpuFi88UHAg0tLohDlLCg0cVjnZzBmB8G4qvtmIdLb9zIVmZN9bCZZjj WjTeITmbQ/JVySgwuXIBXcVMM+EEknjXnIsX5vwRShlBsJleFDOmAdhx6iaNwB+GmdU0 nzm0HfvUPmMeMOr3Qr8UQqUoZh4hqQ5HChLUJzPMEEqXC/YZZpdZX+S6t2uRLIIhc9HV +Z6mhtGaa8U+Ht01UUp9L5bOYzo+Pvdc0lnggQccV96QwZhUN4tgZfEHnXjLAubXqF/f 19ITinemozlrFxoeXsls9zEYYRZG0bhaSMWQfW5WAQcH0kBoLn6XZwnC08z83FLz/wuS B1zQ== X-Gm-Message-State: AOJu0YzZUFpm112jhvyaQ23iGg1gaenGX7Rofuui8G5M0QBC4PuQXNAp mIWAk1A54BxVEVf3OAJ+DhDHGsLQL7Q= X-Google-Smtp-Source: AGHT+IHFo4Qd+bUyHXV7uDnWTfmpROKcqEvV1HIH+ht9/jDqhLREekiPTUW/I5ttnI06QXPQH14EnN1Qqvg= X-Received: from glider.muc.corp.google.com ([2a00:79e0:9c:201:8447:3e89:1f77:ff8a]) (user=glider job=sendgmr) by 2002:a05:690c:a81:b0:5df:4a9b:fb6c with SMTP id ci1-20020a05690c0a8100b005df4a9bfb6cmr109550ywb.3.1702552022462; Thu, 14 Dec 2023 03:07:02 -0800 (PST) Date: Thu, 14 Dec 2023 12:06:38 +0100 In-Reply-To: <20231214110639.2294687-1-glider@google.com> Mime-Version: 1.0 References: <20231214110639.2294687-1-glider@google.com> X-Mailer: git-send-email 2.43.0.472.g3155946c3a-goog Message-ID: <20231214110639.2294687-7-glider@google.com> Subject: [PATCH v10-mte 6/7] arm64: mte: add compression support to mteswap.c From: Alexander Potapenko To: glider@google.com, catalin.marinas@arm.com, will@kernel.org, pcc@google.com, andreyknvl@gmail.com, andriy.shevchenko@linux.intel.com, aleksander.lobakin@intel.com, linux@rasmusvillemoes.dk, yury.norov@gmail.com, alexandru.elisei@arm.com Cc: linux-kernel@vger.kernel.org, linux-arm-kernel@lists.infradead.org, eugenis@google.com, syednwaris@gmail.com, william.gray@linaro.org X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20231214_030704_166966_6DDEDCD9 X-CRM114-Status: GOOD ( 17.25 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org Update mteswap.c to perform inline compression of memory tags when possible. If CONFIG_ARM64_MTE_COMP is enabled, mteswap.c will attempt to compress saved tags for a struct page and store them directly in Xarray entry instead of wasting heap space. Soon after booting Android, tag compression saves ~2x memory previously spent by mteswap.c on tag allocations. On a moderately loaded device with ~20% tagged pages, this leads to saving several megabytes of kernel heap: # cat /sys/kernel/debug/mteswap/stats 8 bytes: 102496 allocations, 67302 deallocations 128 bytes: 212234 allocations, 178278 deallocations uncompressed tag storage size: 8851200 compressed tag storage size: 4346368 (statistics collection is introduced in the following patch) Signed-off-by: Alexander Potapenko Reviewed-by: Catalin Marinas --- v10-mte: - added Catalin's Reviewed-by: v9: - as requested by Yury Norov, split off statistics collection into a separate patch - minor fixes v8: - adapt to the new compression API, abandon mteswap_{no,}comp.c - move stats collection to mteswap.c v5: - drop a dead variable from _mte_free_saved_tags() in mteswap_comp.c - ensure MTE compression works with arbitrary page sizes - update patch description v4: - minor code simplifications suggested by Andy Shevchenko, added missing header dependencies - changed compression API names to reflect modifications made to memcomp.h (as suggested by Yury Norov) v3: - Addressed comments by Andy Shevchenko in another patch: - fixed includes order - replaced u64 with unsigned long - added MODULE_IMPORT_NS(MTECOMP) --- arch/arm64/mm/mteswap.c | 19 +++++++++++++++++-- 1 file changed, 17 insertions(+), 2 deletions(-) diff --git a/arch/arm64/mm/mteswap.c b/arch/arm64/mm/mteswap.c index a31833e3ddc54..70f5c8ecd640d 100644 --- a/arch/arm64/mm/mteswap.c +++ b/arch/arm64/mm/mteswap.c @@ -6,6 +6,8 @@ #include #include #include +#include +#include "mtecomp.h" static DEFINE_XARRAY(mte_pages); @@ -17,12 +19,13 @@ void *mte_allocate_tag_storage(void) void mte_free_tag_storage(char *storage) { - kfree(storage); + if (!mte_is_compressed(storage)) + kfree(storage); } int mte_save_tags(struct page *page) { - void *tag_storage, *ret; + void *tag_storage, *compressed_storage, *ret; if (!page_mte_tagged(page)) return 0; @@ -32,6 +35,11 @@ int mte_save_tags(struct page *page) return -ENOMEM; mte_save_page_tags(page_address(page), tag_storage); + compressed_storage = mte_compress(tag_storage); + if (compressed_storage) { + mte_free_tag_storage(tag_storage); + tag_storage = compressed_storage; + } /* lookup the swap entry.val from the page */ ret = xa_store(&mte_pages, page_swap_entry(page).val, tag_storage, @@ -50,13 +58,20 @@ int mte_save_tags(struct page *page) void mte_restore_tags(swp_entry_t entry, struct page *page) { void *tags = xa_load(&mte_pages, entry.val); + void *tag_storage = NULL; if (!tags) return; if (try_page_mte_tagging(page)) { + if (mte_is_compressed(tags)) { + tag_storage = mte_allocate_tag_storage(); + mte_decompress(tags, tag_storage); + tags = tag_storage; + } mte_restore_page_tags(page_address(page), tags); set_page_mte_tagged(page); + mte_free_tag_storage(tag_storage); } } -- 2.43.0.472.g3155946c3a-goog _______________________________________________ linux-arm-kernel mailing list linux-arm-kernel@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-arm-kernel