From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from mail-yb1-f201.google.com (mail-yb1-f201.google.com [209.85.219.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id C73F8A55 for ; Fri, 19 Apr 2024 08:12:13 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.219.201 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1713514335; cv=none; b=g+mMHVs6BjmSVoawHTzeo2+2P092q7O8KVh6tlLD+tKuga006Z8zhY301wC9VOGb2jdrYVboaP1XZJ3cKZxhZuKYfl8GSR8gD9vDBI+T+oD0Xci7TN5ZJTwdcO3ljf1Lm+7PUabw4J9zdG9YvDvf9XbCvnBcl+yXkYN9lgFrYFM= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1713514335; c=relaxed/simple; bh=rDaUFDG4ciHlh6JyBJ3+1oh56yKlyX+9SWZ3ow1pFm8=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Content-Type; b=XDrHRRxIUTqUQXNSk4pB0ziGms3/YZQSUPnN2EmnbgelOzba1Q0DTTpMpu9/B4pNFjEdLon+tDf0OBbAMihLeygW8FL//8MN2Kss8wSHck5jX/8D4xONxfoIF9gkNXdcWdLwUkBphmmh8W2T4LxtxLBydiG+DtFDNZtZn2VgZvE= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--ardb.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=Ti1B+uhc; arc=none smtp.client-ip=209.85.219.201 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--ardb.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="Ti1B+uhc" Received: by mail-yb1-f201.google.com with SMTP id 3f1490d57ef6-de4c4a2d370so245682276.0 for ; Fri, 19 Apr 2024 01:12:13 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1713514333; x=1714119133; darn=vger.kernel.org; h=to:from:subject:message-id:references:mime-version:in-reply-to:date :from:to:cc:subject:date:message-id:reply-to; bh=UYDDRLkDFDzmTpuH/PRqmxMdfFVhDQwY+25jrFbakQY=; b=Ti1B+uhcVz4CdS+pmUqXnRJvSQRUu50KTZ3dFnoCgjocD0wQbahyP9sQERVCRtY8nG lzzINmxmfG8msnE1bST9rLIuPnFCoNsjJeMBSShcZEBPOFUjzDLTRYjvm7W+DHn1z+B9 sZs9diuoceXrMnLXNkGpE29P9jsQSKlPfY/j5VN7ndp8eouQb5vX5akv1pwXodNM5iZb 6jAUq4fTYM4FySeNey8/O1agniE+8hyI7gvYi9vfugFkKkRciiAsnSPvDtsbszswvRy8 bLbUvYFAOq+2lar4NuKXuOTQhQIKLJNbGYVe+sIZsZFEU/kM2HFESjI37iGyMLa6t3QS 5DUg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1713514333; x=1714119133; h=to:from:subject:message-id:references:mime-version:in-reply-to:date :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=UYDDRLkDFDzmTpuH/PRqmxMdfFVhDQwY+25jrFbakQY=; b=iTiHvxoIvWGPpHTlQriOQTkL+afmTLEZCI9nRYjdinrPjEdhF8A/Kma/aVo252u33F cuFnaJYf3U7DUNS3WA0j5RWRjhzrnFef59crC+5WbQUjEGq8XZOKej5czlG8hpR9YvVO Fo6IVOZY53AJkiOALPsbol2tLA96nsprmTPeEQeoPWNvcinA2q9Cin4LRar1q/ePPVlk YkNN5MTvDZLLEz1jkrElrcr6G9gzv3XqcM/4fxF4/l/RIEJ1isVYobUvZvOiQ0HDaarT P8CUjvos3bmeD2tp5KxkhhaIRhZKGc/iDPKYME0w3N2BIprA3qe59qtEysqIx2fw/Hu9 zS+A== X-Gm-Message-State: AOJu0YynkpGN1V3DdzRcTZYdQrnTiN2bflA3QJLX9AHrLpHNF2UaUZ9C EOPomDC2CezJizgteqErN+uVG323hLu5cXrIVqhU6XHoKP2fBK4kXw+cZIEz7XoFsNsKLys48en WXrmrsuSQxGl3/aZVfQB64By9C8Po/RzkRV/a9jDztPgXiMw0aXke9Dgk7n6pFpx3jV5B+FPmH5 fZemIj5iGU0O4wyYhzeTXiLg== X-Google-Smtp-Source: AGHT+IE/qrFyfxlUU6OUjHPOWn6vyK8x68B5aPB3MA07wEdkChoLfzqe/IyK9JsHu2UQWdAV60Fs6agN X-Received: from palermo.c.googlers.com ([fda3:e722:ac3:cc00:28:9cb1:c0a8:118a]) (user=ardb job=sendgmr) by 2002:a25:d8d5:0:b0:dcc:50ca:e153 with SMTP id p204-20020a25d8d5000000b00dcc50cae153mr413907ybg.7.1713514332918; Fri, 19 Apr 2024 01:12:12 -0700 (PDT) Date: Fri, 19 Apr 2024 10:11:27 +0200 In-Reply-To: <20240419081105.3817596-25-ardb+git@google.com> Precedence: bulk X-Mailing-List: stable@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20240419081105.3817596-25-ardb+git@google.com> X-Developer-Key: i=ardb@kernel.org; a=openpgp; fpr=F43D03328115A198C90016883D200E9CA6329909 X-Developer-Signature: v=1; a=openpgp-sha256; l=8441; i=ardb@kernel.org; h=from:subject; bh=fpRx+Y0bidlKRW0+Q73sYj/tLmZfUhLMEO+/3e52HS8=; b=owGbwMvMwCFmkMcZplerG8N4Wi2JIU1JXet870f7928iZwatv3rc93BicFxLbkrUZffET3lWO VsnVHl0lLIwiHEwyIopsgjM/vtu5+mJUrXOs2Rh5rAygQxh4OIUgIkcOMzIsFR5+QrugOVxPz9Y rpxSFSB37G6ge1nGMsbnW4yFfjOURDIytIZ5rE44tk6k8mfXG3snee2prTJhPDdLrtySPHGYt4S DDQA= X-Mailer: git-send-email 2.44.0.769.g3c40516874-goog Message-ID: <20240419081105.3817596-46-ardb+git@google.com> Subject: [PATCH for-stable-6.1 21/23] x86/sme: Move early SME kernel encryption handling into .head.text From: Ard Biesheuvel To: stable@vger.kernel.org Content-Type: text/plain; charset="UTF-8" From: Ard Biesheuvel [ Commit 48204aba801f1b512b3abed10b8e1a63e03f3dd1 upstream ] The .head.text section is the initial primary entrypoint of the core kernel, and is entered with the CPU executing from a 1:1 mapping of memory. Such code must never access global variables using absolute references, as these are based on the kernel virtual mapping which is not active yet at this point. Given that the SME startup code is also called from this early execution context, move it into .head.text as well. This will allow more thorough build time checks in the future to ensure that early startup code only uses RIP-relative references to global variables. Also replace some occurrences of __pa_symbol() [which relies on the compiler generating an absolute reference, which is not guaranteed] and an open coded RIP-relative access with RIP_REL_REF(). Signed-off-by: Ard Biesheuvel Signed-off-by: Borislav Petkov (AMD) Tested-by: Tom Lendacky Link: https://lore.kernel.org/r/20240227151907.387873-18-ardb+git@google.com Signed-off-by: Ard Biesheuvel --- arch/x86/include/asm/mem_encrypt.h | 8 ++-- arch/x86/mm/mem_encrypt_identity.c | 42 ++++++++------------ 2 files changed, 21 insertions(+), 29 deletions(-) diff --git a/arch/x86/include/asm/mem_encrypt.h b/arch/x86/include/asm/mem_encrypt.h index 41d06822bc8c..853f423b1d13 100644 --- a/arch/x86/include/asm/mem_encrypt.h +++ b/arch/x86/include/asm/mem_encrypt.h @@ -46,8 +46,8 @@ void __init sme_unmap_bootdata(char *real_mode_data); void __init sme_early_init(void); void __init sev_setup_arch(void); -void __init sme_encrypt_kernel(struct boot_params *bp); -void __init sme_enable(struct boot_params *bp); +void sme_encrypt_kernel(struct boot_params *bp); +void sme_enable(struct boot_params *bp); int __init early_set_memory_decrypted(unsigned long vaddr, unsigned long size); int __init early_set_memory_encrypted(unsigned long vaddr, unsigned long size); @@ -80,8 +80,8 @@ static inline void __init sme_unmap_bootdata(char *real_mode_data) { } static inline void __init sme_early_init(void) { } static inline void __init sev_setup_arch(void) { } -static inline void __init sme_encrypt_kernel(struct boot_params *bp) { } -static inline void __init sme_enable(struct boot_params *bp) { } +static inline void sme_encrypt_kernel(struct boot_params *bp) { } +static inline void sme_enable(struct boot_params *bp) { } static inline void sev_es_init_vc_handling(void) { } diff --git a/arch/x86/mm/mem_encrypt_identity.c b/arch/x86/mm/mem_encrypt_identity.c index f7dfc3f89921..f17609884874 100644 --- a/arch/x86/mm/mem_encrypt_identity.c +++ b/arch/x86/mm/mem_encrypt_identity.c @@ -41,6 +41,7 @@ #include #include +#include #include #include #include @@ -98,7 +99,7 @@ static char sme_workarea[2 * PMD_SIZE] __section(".init.scratch"); static char sme_cmdline_arg[] __initdata = "mem_encrypt"; static char sme_cmdline_on[] __initdata = "on"; -static void __init sme_clear_pgd(struct sme_populate_pgd_data *ppd) +static void __head sme_clear_pgd(struct sme_populate_pgd_data *ppd) { unsigned long pgd_start, pgd_end, pgd_size; pgd_t *pgd_p; @@ -113,7 +114,7 @@ static void __init sme_clear_pgd(struct sme_populate_pgd_data *ppd) memset(pgd_p, 0, pgd_size); } -static pud_t __init *sme_prepare_pgd(struct sme_populate_pgd_data *ppd) +static pud_t __head *sme_prepare_pgd(struct sme_populate_pgd_data *ppd) { pgd_t *pgd; p4d_t *p4d; @@ -150,7 +151,7 @@ static pud_t __init *sme_prepare_pgd(struct sme_populate_pgd_data *ppd) return pud; } -static void __init sme_populate_pgd_large(struct sme_populate_pgd_data *ppd) +static void __head sme_populate_pgd_large(struct sme_populate_pgd_data *ppd) { pud_t *pud; pmd_t *pmd; @@ -166,7 +167,7 @@ static void __init sme_populate_pgd_large(struct sme_populate_pgd_data *ppd) set_pmd(pmd, __pmd(ppd->paddr | ppd->pmd_flags)); } -static void __init sme_populate_pgd(struct sme_populate_pgd_data *ppd) +static void __head sme_populate_pgd(struct sme_populate_pgd_data *ppd) { pud_t *pud; pmd_t *pmd; @@ -192,7 +193,7 @@ static void __init sme_populate_pgd(struct sme_populate_pgd_data *ppd) set_pte(pte, __pte(ppd->paddr | ppd->pte_flags)); } -static void __init __sme_map_range_pmd(struct sme_populate_pgd_data *ppd) +static void __head __sme_map_range_pmd(struct sme_populate_pgd_data *ppd) { while (ppd->vaddr < ppd->vaddr_end) { sme_populate_pgd_large(ppd); @@ -202,7 +203,7 @@ static void __init __sme_map_range_pmd(struct sme_populate_pgd_data *ppd) } } -static void __init __sme_map_range_pte(struct sme_populate_pgd_data *ppd) +static void __head __sme_map_range_pte(struct sme_populate_pgd_data *ppd) { while (ppd->vaddr < ppd->vaddr_end) { sme_populate_pgd(ppd); @@ -212,7 +213,7 @@ static void __init __sme_map_range_pte(struct sme_populate_pgd_data *ppd) } } -static void __init __sme_map_range(struct sme_populate_pgd_data *ppd, +static void __head __sme_map_range(struct sme_populate_pgd_data *ppd, pmdval_t pmd_flags, pteval_t pte_flags) { unsigned long vaddr_end; @@ -236,22 +237,22 @@ static void __init __sme_map_range(struct sme_populate_pgd_data *ppd, __sme_map_range_pte(ppd); } -static void __init sme_map_range_encrypted(struct sme_populate_pgd_data *ppd) +static void __head sme_map_range_encrypted(struct sme_populate_pgd_data *ppd) { __sme_map_range(ppd, PMD_FLAGS_ENC, PTE_FLAGS_ENC); } -static void __init sme_map_range_decrypted(struct sme_populate_pgd_data *ppd) +static void __head sme_map_range_decrypted(struct sme_populate_pgd_data *ppd) { __sme_map_range(ppd, PMD_FLAGS_DEC, PTE_FLAGS_DEC); } -static void __init sme_map_range_decrypted_wp(struct sme_populate_pgd_data *ppd) +static void __head sme_map_range_decrypted_wp(struct sme_populate_pgd_data *ppd) { __sme_map_range(ppd, PMD_FLAGS_DEC_WP, PTE_FLAGS_DEC_WP); } -static unsigned long __init sme_pgtable_calc(unsigned long len) +static unsigned long __head sme_pgtable_calc(unsigned long len) { unsigned long entries = 0, tables = 0; @@ -288,7 +289,7 @@ static unsigned long __init sme_pgtable_calc(unsigned long len) return entries + tables; } -void __init sme_encrypt_kernel(struct boot_params *bp) +void __head sme_encrypt_kernel(struct boot_params *bp) { unsigned long workarea_start, workarea_end, workarea_len; unsigned long execute_start, execute_end, execute_len; @@ -323,9 +324,8 @@ void __init sme_encrypt_kernel(struct boot_params *bp) * memory from being cached. */ - /* Physical addresses gives us the identity mapped virtual addresses */ - kernel_start = __pa_symbol(_text); - kernel_end = ALIGN(__pa_symbol(_end), PMD_SIZE); + kernel_start = (unsigned long)RIP_REL_REF(_text); + kernel_end = ALIGN((unsigned long)RIP_REL_REF(_end), PMD_SIZE); kernel_len = kernel_end - kernel_start; initrd_start = 0; @@ -342,14 +342,6 @@ void __init sme_encrypt_kernel(struct boot_params *bp) } #endif - /* - * We're running identity mapped, so we must obtain the address to the - * SME encryption workarea using rip-relative addressing. - */ - asm ("lea sme_workarea(%%rip), %0" - : "=r" (workarea_start) - : "p" (sme_workarea)); - /* * Calculate required number of workarea bytes needed: * executable encryption area size: @@ -359,7 +351,7 @@ void __init sme_encrypt_kernel(struct boot_params *bp) * pagetable structures for the encryption of the kernel * pagetable structures for workarea (in case not currently mapped) */ - execute_start = workarea_start; + execute_start = workarea_start = (unsigned long)RIP_REL_REF(sme_workarea); execute_end = execute_start + (PAGE_SIZE * 2) + PMD_SIZE; execute_len = execute_end - execute_start; @@ -502,7 +494,7 @@ void __init sme_encrypt_kernel(struct boot_params *bp) native_write_cr3(__native_read_cr3()); } -void __init sme_enable(struct boot_params *bp) +void __head sme_enable(struct boot_params *bp) { const char *cmdline_ptr, *cmdline_arg, *cmdline_on; unsigned int eax, ebx, ecx, edx; -- 2.44.0.769.g3c40516874-goog