From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 910E6C433F5 for ; Tue, 26 Oct 2021 13:14:28 +0000 (UTC) Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 5E73A61039 for ; Tue, 26 Oct 2021 13:14:28 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.4.1 mail.kernel.org 5E73A61039 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=kernel.org Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:Message-Id:Date:Subject:Cc :To:From:Reply-To:Content-ID:Content-Description:Resent-Date:Resent-From: Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:In-Reply-To:References: List-Owner; bh=TkDoMaiRd8Yc92VDum3HQZB+WdijsV9Gr+lXjBeYHUE=; b=ejvAtugtozo2lp u0Yfi6c8s/bkYbYoORNX9ahC7shIIQPAHNfs6DtNGe7hrw8vyAsRLsIGcNePbH2tTVI49bu/FgXpj YV5MTScTcGGp+GCrwp6id46Dsr2YFZpK47N4I+69oi1q7U+B1b75er9JHXcd+ZADoNUlxhC46yCwX ml4fMzxuNIz/Mdn5dEGhtDI0Cr9QfRh95SdmTaweDcgJt2c7Kjx5F22KNIdHB5oWcIUoTSRAdkCeL ZzUfkOPK5Qv6K9ulLi9BzqOjl4dXzSjnLt0yM8ni3P/0eByTWadrilBE2BF4jNhCf9lnhReVxFx5z jN41+sVfLQ6W1IZkSICw==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1mfMGI-001qbz-DT; Tue, 26 Oct 2021 13:13:06 +0000 Received: from mail.kernel.org ([198.145.29.99]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1mfMGC-001qaF-88 for linux-arm-kernel@lists.infradead.org; Tue, 26 Oct 2021 13:13:04 +0000 Received: by mail.kernel.org (Postfix) with ESMTPSA id 41E3160E96; Tue, 26 Oct 2021 13:12:58 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1635253979; bh=DzYIcyyysl0ZCevNgoF10+gTWz69EQy1FhtVoPf63g0=; h=From:To:Cc:Subject:Date:From; b=f02OkbT4Pe92WC44T3u6Z7fMy6a9ZA5i5QkEs3BMbJbJeZHW7AdZsthSG/KZo1CSb gWRQBsv33nXHZgOQutm/nppdBjrGuAunyXwqYqXQQXxPtmaNUj1ny3dD0uIHGl2Sol v6JJvS/SU8NaygXuFlgIzJdOqxlAM2YXkqCQD7gtkCa28ie+N1g+nFpppQGFzp52PC asl9GRc03iCObEfkcFDAIknBUm5yd6Tlhy/D5OL3sM33GU22xNZhY+9H2Bv+UAVSBY +DM5BOgSSpRU7RYa9f01rzB8D6exM20NscKWB8uSphh6uxoKpGdAdqK40K5AEnp4d0 7efR94Eqe0JFQ== From: Ard Biesheuvel To: linux@armlinux.org.uk Cc: linus.walleij@linaro.org, arnd@arndb.de, linux-arm-kernel@lists.infradead.org, tglx@linutronix.de, quanyang.wang@windriver.com, Ard Biesheuvel Subject: [PATCH] kmap_local: don't assume kmap PTEs are linear arrays in memory Date: Tue, 26 Oct 2021 15:12:49 +0200 Message-Id: <20211026131249.3731275-1-ardb@kernel.org> X-Mailer: git-send-email 2.30.2 MIME-Version: 1.0 X-Developer-Signature: v=1; a=openpgp-sha256; l=5367; h=from:subject; bh=DzYIcyyysl0ZCevNgoF10+gTWz69EQy1FhtVoPf63g0=; b=owEB7QES/pANAwAKAcNPIjmS2Y8kAcsmYgBhd/7Qi8Yc675fO3Va489kJ/9piPzG6DVdJYLdk3sr cLNbHqaJAbMEAAEKAB0WIQT72WJ8QGnJQhU3VynDTyI5ktmPJAUCYXf+0AAKCRDDTyI5ktmPJGAhC/ 9jOAlCS2ntd81XY5Re6tOzgIIUfXJ/j27gxcXOT/dnrKSkij46gTb32ngnoEF51imDMjKrJcsXZS4a hyZIXZNjJJGAoLzCc5SfGpY5aYRN3C1eEvIkVfPQ9UhJG40VYpkBsZU1X9pKqG0JaWkH+zoz7egEYr HyRmFGvytEzh84l2II9VkBgHsfVx2HN0diPyGhMpEKZhvwdhyyhvfvgKs4nUJaAd0Is7s5h1nTOvLf C9zHnfUUdCF8L8LF1KOSG6DDkGEZuslzzasS9y7Hq14PzS9Uz1OGp9j6d2D0Ktt9a21w6pnv02hang hDxtL7xM2aLCqU4/rpG+3FDzjQP9Q6z10vv+kfBziYn5vJ2StDHaG4nRelEFoGAKZnvfM4RkZLBz7X SVd4je/pwNPIR7my6QkDw6lC7u5w3wo9zTRZozxj8kElpdWi1Y0OL4Oki/MpE46RapptlXcrsR9Zwt Ypm0B8UI0BibHmg3DDDJ47/BVKstpUPc6fyYE16AHuvj4= X-Developer-Key: i=ardb@kernel.org; a=openpgp; fpr=F43D03328115A198C90016883D200E9CA6329909 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20211026_061300_352442_02009A8B X-CRM114-Status: GOOD ( 16.66 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org The kmap_local conversion broke the ARM architecture, because the new code assumes that all PTEs used for creating kmaps form a linear array in memory, and uses array indexing to look up the kmap PTE belonging to a certain kmap index. On ARM, this cannot work, not only because the PTE pages may be non-adjacent in memory, but also because ARM/!LPAE interleaves hardware entries and extended entries (carrying software-only bits) in a way that is not compatible with array indexing. Fortunately, this only seems to affect configurations with more than 8 CPUs, due to the way the per-CPU kmap slots are organized in memory. Work around this by permitting an architecture to set a Kconfig symbol that signifies that the kmap PTEs do not form a lineary array in memory, and so the only way to locate the appropriate one is to walk the page tables. Reported-by: Quanyang Wang Signed-off-by: Ard Biesheuvel --- arch/arm/Kconfig | 1 + mm/Kconfig | 3 +++ mm/highmem.c | 32 +++++++++++++++++++++----------- 3 files changed, 25 insertions(+), 11 deletions(-) diff --git a/arch/arm/Kconfig b/arch/arm/Kconfig index 727c00c7d616..9aa0528f85de 100644 --- a/arch/arm/Kconfig +++ b/arch/arm/Kconfig @@ -1467,6 +1467,7 @@ config HIGHMEM bool "High Memory Support" depends on MMU select KMAP_LOCAL + select KMAP_LOCAL_NON_LINEAR_PTE_ARRAY help The address space of ARM processors is only 4 Gigabytes large and it has to accommodate user address space, kernel address diff --git a/mm/Kconfig b/mm/Kconfig index d16ba9249bc5..c048dea7e342 100644 --- a/mm/Kconfig +++ b/mm/Kconfig @@ -887,6 +887,9 @@ config MAPPING_DIRTY_HELPERS config KMAP_LOCAL bool +config KMAP_LOCAL_NON_LINEAR_PTE_ARRAY + bool + # struct io_mapping based helper. Selected by drivers that need them config IO_MAPPING bool diff --git a/mm/highmem.c b/mm/highmem.c index 4212ad0e4a19..1f0c8a52fd80 100644 --- a/mm/highmem.c +++ b/mm/highmem.c @@ -504,16 +504,22 @@ static inline int kmap_local_calc_idx(int idx) static pte_t *__kmap_pte; -static pte_t *kmap_get_pte(void) +static pte_t *kmap_get_pte(unsigned long vaddr, int idx) { + if (IS_ENABLED(CONFIG_KMAP_LOCAL_NON_LINEAR_PTE_ARRAY)) + /* + * Set by the arch if __kmap_pte[-idx] does not produce + * the correct entry. + */ + return virt_to_kpte(vaddr); if (!__kmap_pte) __kmap_pte = virt_to_kpte(__fix_to_virt(FIX_KMAP_BEGIN)); - return __kmap_pte; + return &__kmap_pte[-idx]; } void *__kmap_local_pfn_prot(unsigned long pfn, pgprot_t prot) { - pte_t pteval, *kmap_pte = kmap_get_pte(); + pte_t pteval, *kmap_pte; unsigned long vaddr; int idx; @@ -525,9 +531,10 @@ void *__kmap_local_pfn_prot(unsigned long pfn, pgprot_t prot) preempt_disable(); idx = arch_kmap_local_map_idx(kmap_local_idx_push(), pfn); vaddr = __fix_to_virt(FIX_KMAP_BEGIN + idx); - BUG_ON(!pte_none(*(kmap_pte - idx))); + kmap_pte = kmap_get_pte(vaddr, idx); + BUG_ON(!pte_none(*kmap_pte)); pteval = pfn_pte(pfn, prot); - arch_kmap_local_set_pte(&init_mm, vaddr, kmap_pte - idx, pteval); + arch_kmap_local_set_pte(&init_mm, vaddr, kmap_pte, pteval); arch_kmap_local_post_map(vaddr, pteval); current->kmap_ctrl.pteval[kmap_local_idx()] = pteval; preempt_enable(); @@ -560,7 +567,7 @@ EXPORT_SYMBOL(__kmap_local_page_prot); void kunmap_local_indexed(void *vaddr) { unsigned long addr = (unsigned long) vaddr & PAGE_MASK; - pte_t *kmap_pte = kmap_get_pte(); + pte_t *kmap_pte; int idx; if (addr < __fix_to_virt(FIX_KMAP_END) || @@ -585,8 +592,9 @@ void kunmap_local_indexed(void *vaddr) idx = arch_kmap_local_unmap_idx(kmap_local_idx(), addr); WARN_ON_ONCE(addr != __fix_to_virt(FIX_KMAP_BEGIN + idx)); + kmap_pte = kmap_get_pte(addr, idx); arch_kmap_local_pre_unmap(addr); - pte_clear(&init_mm, addr, kmap_pte - idx); + pte_clear(&init_mm, addr, kmap_pte); arch_kmap_local_post_unmap(addr); current->kmap_ctrl.pteval[kmap_local_idx()] = __pte(0); kmap_local_idx_pop(); @@ -608,7 +616,7 @@ EXPORT_SYMBOL(kunmap_local_indexed); void __kmap_local_sched_out(void) { struct task_struct *tsk = current; - pte_t *kmap_pte = kmap_get_pte(); + pte_t *kmap_pte; int i; /* Clear kmaps */ @@ -635,8 +643,9 @@ void __kmap_local_sched_out(void) idx = arch_kmap_local_map_idx(i, pte_pfn(pteval)); addr = __fix_to_virt(FIX_KMAP_BEGIN + idx); + kmap_pte = kmap_get_pte(addr, idx); arch_kmap_local_pre_unmap(addr); - pte_clear(&init_mm, addr, kmap_pte - idx); + pte_clear(&init_mm, addr, kmap_pte); arch_kmap_local_post_unmap(addr); } } @@ -644,7 +653,7 @@ void __kmap_local_sched_out(void) void __kmap_local_sched_in(void) { struct task_struct *tsk = current; - pte_t *kmap_pte = kmap_get_pte(); + pte_t *kmap_pte; int i; /* Restore kmaps */ @@ -664,7 +673,8 @@ void __kmap_local_sched_in(void) /* See comment in __kmap_local_sched_out() */ idx = arch_kmap_local_map_idx(i, pte_pfn(pteval)); addr = __fix_to_virt(FIX_KMAP_BEGIN + idx); - set_pte_at(&init_mm, addr, kmap_pte - idx, pteval); + kmap_pte = kmap_get_pte(addr, idx); + set_pte_at(&init_mm, addr, kmap_pte, pteval); arch_kmap_local_post_map(addr, pteval); } } -- 2.30.2 _______________________________________________ linux-arm-kernel mailing list linux-arm-kernel@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-arm-kernel