From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 927CBC4167B for ; Wed, 29 Nov 2023 11:19:48 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Cc:To:From:Subject:Message-ID: References:Mime-Version:In-Reply-To:Date:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=KaVggeRfvW0GGnSb3e5H8vAZGoG/ggF11MEbgiqU/7w=; b=B5/SIqeh2Yr5P//yWQNSpWWT4e y0Z/j6SrRuV0wGveruM56RkXJpc192mYPT266kvzLVPE6yyOjYfjQ3ZxJUuhMd2MbsPKwmsQzNP9i exCGDqx/w1GKlx7s/+2v1RvBurA9yD98GL/RTy/DaSMUCRF0TSPziBKDT9jEbIzwB6/UNY9McS3J1 y6CM2mOTImy/VRp7IRL0ejuwAWSjY+RuZw4c84oxXJP1zYu3cy5I/ktEo956Py6BISXrP77ezNMAq GJr11BgUx4WHokF9wUSlQR9VglLKEw0WmcoKhfDI5ku1ri9/gdRqTqLiH/J9sitlhOwU/uEJfk+Ua oVsBtbMA==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1r8Ib1-0080XQ-2E; Wed, 29 Nov 2023 11:19:11 +0000 Received: from casper.infradead.org ([2001:8b0:10b:1236::1]) by bombadil.infradead.org with esmtps (Exim 4.96 #2 (Red Hat Linux)) id 1r8IaY-00808e-14 for linux-arm-kernel@bombadil.infradead.org; Wed, 29 Nov 2023 11:18:42 +0000 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Type:Cc:To:From:Subject: Message-ID:References:Mime-Version:In-Reply-To:Date:Sender:Reply-To: Content-Transfer-Encoding:Content-ID:Content-Description; bh=4dtJd4Vj6J3++Ly9vtEw4LExLz6deq0l4YHI2Xx/LP8=; b=eYya/r5EWKqz2ncOGbnsOwTo6I zEgQv6qfYeXbzipTrjCbDHxW1kIbw3JouE1VA4WkQKHPKXN/cscuJ8rNgG6OJaJuMo4iY3uVJwJvE pysvops/Dp/5SIFd9wzgYlwMVQ2z0wQLj0S3ePzB+YevmU3hDukbT3xxIXZR8l7RbhBFogBbKi0cG FuhUy5fKpfg8yxDeQesOTsIwihTw9CB5Bz9eOYrlJ50tpJHvDVhe8ZuwJ2zMOIoyzKXu+yaZCdI/h KZJqqkucPfVranVrmSzahichoc1pAuHhaZoqpbAdrUeEQ5ekzcmSE7re9xgeMLFrAkUDRRpdNAFEV lym3cBcw==; Received: from mail-wm1-x349.google.com ([2a00:1450:4864:20::349]) by casper.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1r8IaT-00DKS0-6U for linux-arm-kernel@lists.infradead.org; Wed, 29 Nov 2023 11:18:41 +0000 Received: by mail-wm1-x349.google.com with SMTP id 5b1f17b1804b1-40b346a11cdso43552475e9.3 for ; Wed, 29 Nov 2023 03:18:36 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1701256714; x=1701861514; darn=lists.infradead.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=4dtJd4Vj6J3++Ly9vtEw4LExLz6deq0l4YHI2Xx/LP8=; b=tzGX8vGBlizDoU9TW1mDciNku4wzaB7Y7etTHBMTt2HGMZGwanjifNOrabhVir1u3r BUAXiB0bdIFkyShGffOG4XzZK6IfutXMVfGtdAhJXfjRiI1aj9rLb/uYyZBnpb8xSnwn Ikn3qx2h0/BFI6DSmTNdBJz+pqRSSf/cZ4WH6oP0Bgo4JNoGkjz/+rXs7cyzWeGCx7dL pYfhR9bDtNgn1dCNliziVo3E6iG/ZyVw+vo6aMbTadY9oyB/D6cZi4GZIa5TJ9imAlEq qAxTfVNey+uZpioK37bWUVcrSWCSr10ydynefCozPvGgCfAZ+EZM95xJ0LviayEFvEKw 4a8A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1701256714; x=1701861514; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=4dtJd4Vj6J3++Ly9vtEw4LExLz6deq0l4YHI2Xx/LP8=; b=dAZQCgoVD9ktk6tCmCJ8kF8jlR5ZsuXc1Se75OU1fTxnCnqFRyZSZVNF5wlrBRyoci l38+UGikpH7Ohk0MbYY886s+rQ8txp9pur+hVUAYNPTtDfbYV4elhrnsjO8PNA88Vepe 9IbOa+KV1qhFIHd7pLuMBy4uzqt4wNBKUb0xQHCA+Di/St9PAHnQEGXAmqPEF8+KHEYO BMhQQYDzLzF6vYhpLmSMh8Q0ynv5uQUyyg1B2pcjDg0PAYx/sd1dsjxPNOENpaHu4Dwk GhFS7bO46P7yTQOIVDqLQZVLIMgkJM03SvkvTXGiW/92yOaKvAddHRjBL8I6pFPt/bUn WiCA== X-Gm-Message-State: AOJu0YzWNO0AkfQeuK9vgYSlQEX3Ab1rtL+5fQx1BMi/FFlRW+nFKpnX J4fMex2hjqr+xY2Wj5wAoZIW6DB7Xrf+YJpq3WYwcz/LkB6TXEv2+pijykJQmWCNVdgiR6P9FZY HqNdPd/IRLuU7Q04fa2wnj0emVn19aqCDIJKXEel319XORccPlPtHioGZVvYg5t/+wsHNvkhweo A= X-Google-Smtp-Source: AGHT+IHyYoNd+mQJPpHwXPnFrrfRkXyuwqiW9eA2jpWifCRed7NePPtWcRqe1Z+L9UJRED1BIpFzwuxU X-Received: from palermo.c.googlers.com ([fda3:e722:ac3:cc00:28:9cb1:c0a8:118a]) (user=ardb job=sendgmr) by 2002:adf:fb12:0:b0:333:1775:939f with SMTP id c18-20020adffb12000000b003331775939fmr14366wrr.8.1701256713718; Wed, 29 Nov 2023 03:18:33 -0800 (PST) Date: Wed, 29 Nov 2023 12:16:30 +0100 In-Reply-To: <20231129111555.3594833-43-ardb@google.com> Mime-Version: 1.0 References: <20231129111555.3594833-43-ardb@google.com> X-Developer-Key: i=ardb@kernel.org; a=openpgp; fpr=F43D03328115A198C90016883D200E9CA6329909 X-Developer-Signature: v=1; a=openpgp-sha256; l=3648; i=ardb@kernel.org; h=from:subject; bh=39VHkrLocnVcTq9rDf/brURimnNLw0Ir6dXWX/R1/Rg=; b=owGbwMvMwCFmkMcZplerG8N4Wi2JITVddtKTcNu5TnO5bi2csrh15cmUZ+V5+2okXlRO+bVXl S/oRv3CjlIWBjEOBlkxRRaB2X/f7Tw9UarWeZYszBxWJpAhDFycAjCR02cZ/hkrfJv9f8sCb6Vz Pfc9Q/+7M3dOYdi0/5jM5fTV+zk7j/1lZNjFtMXKUy2XrYb3TZtztFdU0xLW3vPvyrPtbhz+z6i 8lgUA X-Mailer: git-send-email 2.43.0.rc1.413.gea7ed67945-goog Message-ID: <20231129111555.3594833-77-ardb@google.com> Subject: [PATCH v6 34/41] arm64: mmu: Make __cpu_replace_ttbr1() out of line From: Ard Biesheuvel To: linux-arm-kernel@lists.infradead.org Cc: Ard Biesheuvel , Catalin Marinas , Will Deacon , Marc Zyngier , Mark Rutland , Ryan Roberts , Anshuman Khandual , Kees Cook X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20231129_111837_266667_EE7AE3F4 X-CRM114-Status: GOOD ( 15.43 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org From: Ard Biesheuvel __cpu_replace_ttbr1() is a static inline, and so it gets instantiated wherever it is used. This is not really necessary, as it is never called on a hot path. It also has the unfortunate side effect that the symbol idmap_cpu_replace_ttbr1 may never be referenced from kCFI enabled C code, and this means the type id symbol may not exist either. This will result in a build error once we start referring to this symbol from asm code as well. (Note that this problem only occurs when CnP, KAsan and suspend/resume are all disabled in the Kconfig but that is a valid config, if unusual). So let's just move it out of line so all callers will share the same implementation, which will reference idmap_cpu_replace_ttbr1 unconditionally. Signed-off-by: Ard Biesheuvel --- arch/arm64/include/asm/mmu_context.h | 32 +------------------- arch/arm64/mm/mmu.c | 32 ++++++++++++++++++++ 2 files changed, 33 insertions(+), 31 deletions(-) diff --git a/arch/arm64/include/asm/mmu_context.h b/arch/arm64/include/asm/mmu_context.h index 9ce4200508b1..926fbbcecbe0 100644 --- a/arch/arm64/include/asm/mmu_context.h +++ b/arch/arm64/include/asm/mmu_context.h @@ -148,37 +148,7 @@ static inline void cpu_install_ttbr0(phys_addr_t ttbr0, unsigned long t0sz) isb(); } -/* - * Atomically replaces the active TTBR1_EL1 PGD with a new VA-compatible PGD, - * avoiding the possibility of conflicting TLB entries being allocated. - */ -static inline void __cpu_replace_ttbr1(pgd_t *pgdp, pgd_t *idmap, bool cnp) -{ - typedef void (ttbr_replace_func)(phys_addr_t); - extern ttbr_replace_func idmap_cpu_replace_ttbr1; - ttbr_replace_func *replace_phys; - unsigned long daif; - - /* phys_to_ttbr() zeros lower 2 bits of ttbr with 52-bit PA */ - phys_addr_t ttbr1 = phys_to_ttbr(virt_to_phys(pgdp)); - - if (cnp) - ttbr1 |= TTBR_CNP_BIT; - - replace_phys = (void *)__pa_symbol(idmap_cpu_replace_ttbr1); - - __cpu_install_idmap(idmap); - - /* - * We really don't want to take *any* exceptions while TTBR1 is - * in the process of being replaced so mask everything. - */ - daif = local_daif_save(); - replace_phys(ttbr1); - local_daif_restore(daif); - - cpu_uninstall_idmap(); -} +void __cpu_replace_ttbr1(pgd_t *pgdp, pgd_t *idmap, bool cnp); static inline void cpu_enable_swapper_cnp(void) { diff --git a/arch/arm64/mm/mmu.c b/arch/arm64/mm/mmu.c index 03c73e9197ac..22104febebf2 100644 --- a/arch/arm64/mm/mmu.c +++ b/arch/arm64/mm/mmu.c @@ -1483,3 +1483,35 @@ void ptep_modify_prot_commit(struct vm_area_struct *vma, unsigned long addr, pte { set_pte_at(vma->vm_mm, addr, ptep, pte); } + +/* + * Atomically replaces the active TTBR1_EL1 PGD with a new VA-compatible PGD, + * avoiding the possibility of conflicting TLB entries being allocated. + */ +void __cpu_replace_ttbr1(pgd_t *pgdp, pgd_t *idmap, bool cnp) +{ + typedef void (ttbr_replace_func)(phys_addr_t); + extern ttbr_replace_func idmap_cpu_replace_ttbr1; + ttbr_replace_func *replace_phys; + unsigned long daif; + + /* phys_to_ttbr() zeros lower 2 bits of ttbr with 52-bit PA */ + phys_addr_t ttbr1 = phys_to_ttbr(virt_to_phys(pgdp)); + + if (cnp) + ttbr1 |= TTBR_CNP_BIT; + + replace_phys = (void *)__pa_symbol(idmap_cpu_replace_ttbr1); + + __cpu_install_idmap(idmap); + + /* + * We really don't want to take *any* exceptions while TTBR1 is + * in the process of being replaced so mask everything. + */ + daif = local_daif_save(); + replace_phys(ttbr1); + local_daif_restore(daif); + + cpu_uninstall_idmap(); +} -- 2.43.0.rc1.413.gea7ed67945-goog _______________________________________________ linux-arm-kernel mailing list linux-arm-kernel@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-arm-kernel