From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 79E4BC48BEE for ; Wed, 14 Feb 2024 12:32:26 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Cc:To:From:Subject:Message-ID: References:Mime-Version:In-Reply-To:Date:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=DgVORo34ERPsEt0hfNH0SP9DbPj+JjPG4g9Q3dWhsXs=; b=dDE7vPyn7daLrGVFwb8qjemn9R 31E4iZG5lIPoTcaD6ymJ6NL35sP2j1aHC+I/sWQf8HlIVhV8JaGxNVUAuAjHcn8xaE+5gkaBObTGX 2L/EzFxuTapnP6P5JdAt8iEAlGhUDWGpE5F4XyTvehZxqGycSYgL3e1JKIiL8eo/wtTTptUL/llT4 OR40SLk+i4/o8SWeuoBoV51xY6mxyq+MGtB9ybwtQZIS7/oJO5+d3rCvltEHWNQSQKI68F/Ff5vN2 19x0tU55ZFf8u4NbL6abU+jbPtpJ131odHfihC51pxk2QZYb+T6/8dWKwTHIzLWMTTyTdZo9tmb7C A6LP7naA==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.97.1 #2 (Red Hat Linux)) id 1raEQq-0000000CpPL-2llI; Wed, 14 Feb 2024 12:32:08 +0000 Received: from mail-yw1-x1149.google.com ([2607:f8b0:4864:20::1149]) by bombadil.infradead.org with esmtps (Exim 4.97.1 #2 (Red Hat Linux)) id 1raEP5-0000000CoWO-0ogf for linux-arm-kernel@lists.infradead.org; Wed, 14 Feb 2024 12:30:43 +0000 Received: by mail-yw1-x1149.google.com with SMTP id 00721157ae682-6077ec913f0so42263777b3.0 for ; Wed, 14 Feb 2024 04:30:18 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1707913818; x=1708518618; darn=lists.infradead.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=unR/pVecox9ouLHJTW1/MJ7pGY21RIdyDZtkCV8+fVo=; b=GjVY1HSfjG8bUdlyJuGW1JyarUq2MX5nhWrUhCy8/ougoatm9LsE1wd3Vcf/78bMlr Cwg+AOqB8uFTtVdrmqybETmXO35FmfretwBce/N5CPN0ky9mpx5mlAVTQHjjkJQbRb4A Q5RffWKb9xJnqTSoQF2qUmypCI5p+/V7sm1cwD6Y5TEW+OZ3Vung8g2HI9y18fLIaLuZ ruSozPKiPS7bnwxpcShktrvdYpijYc8j0nDH6guAH0vInjti1z0G6CnMmD6XWCPxQxqt yzC+D2O1VlbGha2HstO3xjozN0AhA+wyw+YxgzC4mIGnTgqbe2wAlKfclxVoFfwEkjrt HvpQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1707913818; x=1708518618; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=unR/pVecox9ouLHJTW1/MJ7pGY21RIdyDZtkCV8+fVo=; b=GPZkWwDpY2IMsTfcKBBl82ej8ndhwEHyRMtuWPm1MTsU+H9fSS3JIic7bIpjEYAQC1 4l5FM6FH2Qj24k65oBuAYGRGX3A/4XDeArl8syIYGhLNmMiFMCTRGtRrpYtJ7MkJk+dN eOE76HiqcOLJN6wfkIKwBuHTN+4h7+M85k+lG8pktiooRAzlt8bz99laYXdg+yneexZK bfQOnz5QbzlVVSTi1987SOwrAXKP6aWAdjudM9eoMPphUhOb+Ep4svA7RwLw6y6VCs0t nI/Cm4nlivFNvDERQDVaKZR/wz2x7W6+YeimsZEPo//EAyBswEFnL+8eHypCC59LwU/b aEnA== X-Gm-Message-State: AOJu0YwDWU9EThSjguUg0rC2hvc9cE9Vwo9pP96/vAgToB4PA/u/0UbN c+l4J3FnE7p81DEyOIEOl8iUWTrVpATEfIYCpQNGLJ7ySiERGhABXOvOJKnMpGZITU/8NDVenPi GNlH+QNvqJ9NV9e6SL5BenOOw1/CLPgcjz0+wuhl33m8vHID7OdbxhYVTZrYR6VmmG3zLryMA99 nPAbtigF3EvZFSUoPOcqeRkdaEe7UeonhqjKg6YDJ1 X-Google-Smtp-Source: AGHT+IHN3zejx7M9FiR6GcYODwOwKE72XRoZO7fvo1jy4v0X6jmIXBIjGAI/kZeCHLqHeyBp/dD3tBJF X-Received: from palermo.c.googlers.com ([fda3:e722:ac3:cc00:28:9cb1:c0a8:118a]) (user=ardb job=sendgmr) by 2002:a05:6902:1007:b0:dc9:5ef8:2b2d with SMTP id w7-20020a056902100700b00dc95ef82b2dmr554101ybt.4.1707913817612; Wed, 14 Feb 2024 04:30:17 -0800 (PST) Date: Wed, 14 Feb 2024 13:29:03 +0100 In-Reply-To: <20240214122845.2033971-45-ardb+git@google.com> Mime-Version: 1.0 References: <20240214122845.2033971-45-ardb+git@google.com> X-Developer-Key: i=ardb@kernel.org; a=openpgp; fpr=F43D03328115A198C90016883D200E9CA6329909 X-Developer-Signature: v=1; a=openpgp-sha256; l=3644; i=ardb@kernel.org; h=from:subject; bh=fwxVCFIDdoHP5fUkExYxnLXpMatgoL0Rf/pPEVWwwN0=; b=owGbwMvMwCFmkMcZplerG8N4Wi2JIfXMJoFir2w5J/f2/RdWLFXm9TohcrrkalfI1an78t+9f mW9cKd0RykLgxgHg6yYIovA7L/vdp6eKFXrPEsWZg4rE8gQBi5OAZjIDTGGf7qe2+8GVttsPWo4 8dyeep7XX1m+evH+/Xy9i4/LWCPgkjLDXzllu2t/dndeDLQ+nLP5iE3j9Lns3zdHW/82YZ6iviR oIRMA X-Mailer: git-send-email 2.43.0.687.g38aa6559b0-goog Message-ID: <20240214122845.2033971-62-ardb+git@google.com> Subject: [PATCH v8 17/43] arm64: mmu: Make __cpu_replace_ttbr1() out of line From: Ard Biesheuvel To: linux-arm-kernel@lists.infradead.org Cc: Ard Biesheuvel , Catalin Marinas , Will Deacon , Marc Zyngier , Mark Rutland , Ryan Roberts , Anshuman Khandual , Kees Cook X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20240214_043019_549592_6DD9727B X-CRM114-Status: GOOD ( 16.37 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org From: Ard Biesheuvel __cpu_replace_ttbr1() is a static inline, and so it gets instantiated wherever it is used. This is not really necessary, as it is never called on a hot path. It also has the unfortunate side effect that the symbol idmap_cpu_replace_ttbr1 may never be referenced from kCFI enabled C code, and this means the type id symbol may not exist either. This will result in a build error once we start referring to this symbol from asm code as well. (Note that this problem only occurs when CnP, KAsan and suspend/resume are all disabled in the Kconfig but that is a valid config, if unusual). So let's just move it out of line so all callers will share the same implementation, which will reference idmap_cpu_replace_ttbr1 unconditionally. Signed-off-by: Ard Biesheuvel --- arch/arm64/include/asm/mmu_context.h | 32 +------------------- arch/arm64/mm/mmu.c | 32 ++++++++++++++++++++ 2 files changed, 33 insertions(+), 31 deletions(-) diff --git a/arch/arm64/include/asm/mmu_context.h b/arch/arm64/include/asm/mmu_context.h index 9ce4200508b1..926fbbcecbe0 100644 --- a/arch/arm64/include/asm/mmu_context.h +++ b/arch/arm64/include/asm/mmu_context.h @@ -148,37 +148,7 @@ static inline void cpu_install_ttbr0(phys_addr_t ttbr0, unsigned long t0sz) isb(); } -/* - * Atomically replaces the active TTBR1_EL1 PGD with a new VA-compatible PGD, - * avoiding the possibility of conflicting TLB entries being allocated. - */ -static inline void __cpu_replace_ttbr1(pgd_t *pgdp, pgd_t *idmap, bool cnp) -{ - typedef void (ttbr_replace_func)(phys_addr_t); - extern ttbr_replace_func idmap_cpu_replace_ttbr1; - ttbr_replace_func *replace_phys; - unsigned long daif; - - /* phys_to_ttbr() zeros lower 2 bits of ttbr with 52-bit PA */ - phys_addr_t ttbr1 = phys_to_ttbr(virt_to_phys(pgdp)); - - if (cnp) - ttbr1 |= TTBR_CNP_BIT; - - replace_phys = (void *)__pa_symbol(idmap_cpu_replace_ttbr1); - - __cpu_install_idmap(idmap); - - /* - * We really don't want to take *any* exceptions while TTBR1 is - * in the process of being replaced so mask everything. - */ - daif = local_daif_save(); - replace_phys(ttbr1); - local_daif_restore(daif); - - cpu_uninstall_idmap(); -} +void __cpu_replace_ttbr1(pgd_t *pgdp, pgd_t *idmap, bool cnp); static inline void cpu_enable_swapper_cnp(void) { diff --git a/arch/arm64/mm/mmu.c b/arch/arm64/mm/mmu.c index 1ac7467d34c9..f9332eea318f 100644 --- a/arch/arm64/mm/mmu.c +++ b/arch/arm64/mm/mmu.c @@ -1486,3 +1486,35 @@ void ptep_modify_prot_commit(struct vm_area_struct *vma, unsigned long addr, pte { set_pte_at(vma->vm_mm, addr, ptep, pte); } + +/* + * Atomically replaces the active TTBR1_EL1 PGD with a new VA-compatible PGD, + * avoiding the possibility of conflicting TLB entries being allocated. + */ +void __cpu_replace_ttbr1(pgd_t *pgdp, pgd_t *idmap, bool cnp) +{ + typedef void (ttbr_replace_func)(phys_addr_t); + extern ttbr_replace_func idmap_cpu_replace_ttbr1; + ttbr_replace_func *replace_phys; + unsigned long daif; + + /* phys_to_ttbr() zeros lower 2 bits of ttbr with 52-bit PA */ + phys_addr_t ttbr1 = phys_to_ttbr(virt_to_phys(pgdp)); + + if (cnp) + ttbr1 |= TTBR_CNP_BIT; + + replace_phys = (void *)__pa_symbol(idmap_cpu_replace_ttbr1); + + __cpu_install_idmap(idmap); + + /* + * We really don't want to take *any* exceptions while TTBR1 is + * in the process of being replaced so mask everything. + */ + daif = local_daif_save(); + replace_phys(ttbr1); + local_daif_restore(daif); + + cpu_uninstall_idmap(); +} -- 2.43.0.687.g38aa6559b0-goog _______________________________________________ linux-arm-kernel mailing list linux-arm-kernel@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-arm-kernel