From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-20.4 required=3.0 tests=BAYES_00,DKIMWL_WL_MED, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_PATCH,MAILING_LIST_MULTI,SIGNED_OFF_BY,SPF_HELO_NONE,SPF_PASS, USER_AGENT_GIT,USER_IN_DEF_DKIM_WL autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id ED9D8C55179 for ; Tue, 3 Nov 2020 17:59:41 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 8FE1F22226 for ; Tue, 3 Nov 2020 17:59:41 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="rAGTbC3O" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1729080AbgKCR7L (ORCPT ); Tue, 3 Nov 2020 12:59:11 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:52066 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1729061AbgKCR7D (ORCPT ); Tue, 3 Nov 2020 12:59:03 -0500 Received: from mail-ed1-x54a.google.com (mail-ed1-x54a.google.com [IPv6:2a00:1450:4864:20::54a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id DDA2DC0617A6 for ; Tue, 3 Nov 2020 09:59:01 -0800 (PST) Received: by mail-ed1-x54a.google.com with SMTP id y8so7841450edj.5 for ; Tue, 03 Nov 2020 09:59:01 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=sender:date:in-reply-to:message-id:mime-version:references:subject :from:to:cc; bh=CjOul1VqaRkscFJ+dD4H59GaMNHUJMUL0L/eCnbi6QE=; b=rAGTbC3OB8Gr5wn9dTgBQq5E9lSZTCXB6/V/73NZWHmvm9QPQp4pGG4Ep3y8Bp8b4Q 7qA+PrzO/03REN+nljLqGkHfPuFk66L40kt6Qw4C+nOxQhAlbQT7zkEuXkA+SR7RHNQ8 DfNqS8wXgnUhoFbBJNTvBvTlp/lqdlvZ2PVJybT4H/8fjq3O6+rsf1PYLiQykW/wtIid WSSnY4RgZPwDx6OnF2NNsvNtxgt6dvNKuLwSRiYpzYkPNlxIr8TirE67iKv/Wy11oiGi FoV8KJQz2AyxYIha+sGd0U23UVWf2Q2nc384ZXDRY6LDA9oVhfwWkOVc9xwKs7u3/5L4 XGdA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=CjOul1VqaRkscFJ+dD4H59GaMNHUJMUL0L/eCnbi6QE=; b=KWevvcetzODdzBOSYVzFn5G3DcYAvyjgiOkXCu+kDIqcOJRTRzq7vudrOLEPDOKBiG zGOxK99b99jLOiJfEwn6VXMAZAL8vKMlin0dHEs5XD/jZC7C7jK6qehvBFQD/pDPcrGI L1FYLR+3FIFUrA6GcRL+VhdvxJSMmsb9nyyzWsEo91tWWMW05LP29oaYiddwEJ3fa2tY OxR0eBmJccXGTHNRZdMQEeFpxjgmvVRgPluu54fyXYkyIkjz7J5+tOeLOx2rjfFhL1Xl 1TORymNVfohV9d8QEibxsH2xk41oSwnmdyd/umvmbv0dBjTAdT2YXnyMF9YtkltAxyv8 9+5A== X-Gm-Message-State: AOAM531HvgfjS6mmAWbPc6nV94AYsG7d6I/88eTrzY8O91+WMRM1rCXx rwb3Tgwzs6k2znD40oZ3NDPrA6grWQ== X-Google-Smtp-Source: ABdhPJw7jn9Aw9p+ba9azss7oDHFoiQHb4Di3vC9AcakPZBAocSHGKh5dLfJKGFn7yyrZXAvfw6COGNsEA== Sender: "elver via sendgmr" X-Received: from elver.muc.corp.google.com ([2a00:79e0:15:13:f693:9fff:fef4:2449]) (user=elver job=sendgmr) by 2002:a05:6402:b6e:: with SMTP id cb14mr9832146edb.308.1604426340371; Tue, 03 Nov 2020 09:59:00 -0800 (PST) Date: Tue, 3 Nov 2020 18:58:34 +0100 In-Reply-To: <20201103175841.3495947-1-elver@google.com> Message-Id: <20201103175841.3495947-3-elver@google.com> Mime-Version: 1.0 References: <20201103175841.3495947-1-elver@google.com> X-Mailer: git-send-email 2.29.1.341.ge80a0c044ae-goog Subject: [PATCH v7 2/9] x86, kfence: enable KFENCE for x86 From: Marco Elver To: elver@google.com, akpm@linux-foundation.org, glider@google.com Cc: hpa@zytor.com, paulmck@kernel.org, andreyknvl@google.com, aryabinin@virtuozzo.com, luto@kernel.org, bp@alien8.de, catalin.marinas@arm.com, cl@linux.com, dave.hansen@linux.intel.com, rientjes@google.com, dvyukov@google.com, edumazet@google.com, gregkh@linuxfoundation.org, hdanton@sina.com, mingo@redhat.com, jannh@google.com, Jonathan.Cameron@huawei.com, corbet@lwn.net, iamjoonsoo.kim@lge.com, joern@purestorage.com, keescook@chromium.org, mark.rutland@arm.com, penberg@kernel.org, peterz@infradead.org, sjpark@amazon.com, tglx@linutronix.de, vbabka@suse.cz, will@kernel.org, x86@kernel.org, linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, kasan-dev@googlegroups.com, linux-arm-kernel@lists.infradead.org, linux-mm@kvack.org Content-Type: text/plain; charset="UTF-8" Precedence: bulk List-ID: X-Mailing-List: linux-doc@vger.kernel.org From: Alexander Potapenko Add architecture specific implementation details for KFENCE and enable KFENCE for the x86 architecture. In particular, this implements the required interface in for setting up the pool and providing helper functions for protecting and unprotecting pages. For x86, we need to ensure that the pool uses 4K pages, which is done using the set_memory_4k() helper function. Reviewed-by: Dmitry Vyukov Co-developed-by: Marco Elver Signed-off-by: Marco Elver Signed-off-by: Alexander Potapenko --- v7: * Only not-present faults should be handled by kfence [reported by Jann Horn]. v5: * MAJOR CHANGE: Switch to the memblock_alloc'd pool. Running benchmarks with the newly optimized is_kfence_address(), no difference between baseline and KFENCE is observed. * Suggested by Jann Horn: * Move x86 kfence_handle_page_fault before oops handling. * WARN_ON in kfence_protect_page if non-4K pages. * Better comments for x86 kfence_protect_page. v4: * Define __kfence_pool_attrs. --- arch/x86/Kconfig | 1 + arch/x86/include/asm/kfence.h | 65 +++++++++++++++++++++++++++++++++++ arch/x86/mm/fault.c | 5 +++ 3 files changed, 71 insertions(+) create mode 100644 arch/x86/include/asm/kfence.h diff --git a/arch/x86/Kconfig b/arch/x86/Kconfig index f6946b81f74a..c9ec6b5ba358 100644 --- a/arch/x86/Kconfig +++ b/arch/x86/Kconfig @@ -144,6 +144,7 @@ config X86 select HAVE_ARCH_JUMP_LABEL_RELATIVE select HAVE_ARCH_KASAN if X86_64 select HAVE_ARCH_KASAN_VMALLOC if X86_64 + select HAVE_ARCH_KFENCE select HAVE_ARCH_KGDB select HAVE_ARCH_MMAP_RND_BITS if MMU select HAVE_ARCH_MMAP_RND_COMPAT_BITS if MMU && COMPAT diff --git a/arch/x86/include/asm/kfence.h b/arch/x86/include/asm/kfence.h new file mode 100644 index 000000000000..beeac105dae7 --- /dev/null +++ b/arch/x86/include/asm/kfence.h @@ -0,0 +1,65 @@ +/* SPDX-License-Identifier: GPL-2.0 */ + +#ifndef _ASM_X86_KFENCE_H +#define _ASM_X86_KFENCE_H + +#include +#include + +#include +#include +#include +#include + +/* + * The page fault handler entry function, up to which the stack trace is + * truncated in reports. + */ +#define KFENCE_SKIP_ARCH_FAULT_HANDLER "asm_exc_page_fault" + +/* Force 4K pages for __kfence_pool. */ +static inline bool arch_kfence_init_pool(void) +{ + unsigned long addr; + + for (addr = (unsigned long)__kfence_pool; is_kfence_address((void *)addr); + addr += PAGE_SIZE) { + unsigned int level; + + if (!lookup_address(addr, &level)) + return false; + + if (level != PG_LEVEL_4K) + set_memory_4k(addr, 1); + } + + return true; +} + +/* Protect the given page and flush TLB. */ +static inline bool kfence_protect_page(unsigned long addr, bool protect) +{ + unsigned int level; + pte_t *pte = lookup_address(addr, &level); + + if (WARN_ON(!pte || level != PG_LEVEL_4K)) + return false; + + /* + * We need to avoid IPIs, as we may get KFENCE allocations or faults + * with interrupts disabled. Therefore, the below is best-effort, and + * does not flush TLBs on all CPUs. We can tolerate some inaccuracy; + * lazy fault handling takes care of faults after the page is PRESENT. + */ + + if (protect) + set_pte(pte, __pte(pte_val(*pte) & ~_PAGE_PRESENT)); + else + set_pte(pte, __pte(pte_val(*pte) | _PAGE_PRESENT)); + + /* Flush this CPU's TLB. */ + flush_tlb_one_kernel(addr); + return true; +} + +#endif /* _ASM_X86_KFENCE_H */ diff --git a/arch/x86/mm/fault.c b/arch/x86/mm/fault.c index 82bf37a5c9ec..e42db2836438 100644 --- a/arch/x86/mm/fault.c +++ b/arch/x86/mm/fault.c @@ -9,6 +9,7 @@ #include /* oops_begin/end, ... */ #include /* search_exception_tables */ #include /* max_low_pfn */ +#include /* kfence_handle_page_fault */ #include /* NOKPROBE_SYMBOL, ... */ #include /* kmmio_handler, ... */ #include /* perf_sw_event */ @@ -725,6 +726,10 @@ no_context(struct pt_regs *regs, unsigned long error_code, if (IS_ENABLED(CONFIG_EFI)) efi_recover_from_page_fault(address); + /* Only not-present faults should be handled by KFENCE. */ + if (!(error_code & X86_PF_PROT) && kfence_handle_page_fault(address)) + return; + oops: /* * Oops. The kernel tried to access some bad page. We'll have to -- 2.29.1.341.ge80a0c044ae-goog