* Re: [PATCH 11/11 v2.2] ftrace: Add recording of functions that caused recursion
From: Petr Mladek @ 2020-11-03 14:10 UTC (permalink / raw)
To: Steven Rostedt
Cc: Anton Vorontsov, linux-doc, Peter Zijlstra,
Sebastian Andrzej Siewior, Kamalesh Babulal, James E.J. Bottomley,
Guo Ren, H. Peter Anvin, live-patching, Miroslav Benes,
Ingo Molnar, linux-s390, Joe Lawrence, Jonathan Corbet,
Mauro Carvalho Chehab, Helge Deller, x86, linux-csky,
Christian Borntraeger, Kees Cook, Vasily Gorbik, Heiko Carstens,
Jiri Kosina, Borislav Petkov, Josh Poimboeuf, Thomas Gleixner,
Tony Luck, linux-parisc, linux-kernel, Masami Hiramatsu,
Colin Cross, Paul Mackerras, Andrew Morton, linuxppc-dev
In-Reply-To: <20201102142254.7e148f8a@gandalf.local.home>
On Mon 2020-11-02 14:23:14, Steven Rostedt wrote:
> From c532ff6b048dd4a12943b05c7b8ce30666c587c8 Mon Sep 17 00:00:00 2001
> From: "Steven Rostedt (VMware)" <rostedt@goodmis.org>
> Date: Thu, 29 Oct 2020 15:27:06 -0400
> Subject: [PATCH] ftrace: Add recording of functions that caused recursion
>
> This adds CONFIG_FTRACE_RECORD_RECURSION that will record to a file
> "recursed_functions" all the functions that caused recursion while a
> callback to the function tracer was running.
>
> diff --git a/include/linux/trace_recursion.h b/include/linux/trace_recursion.h
> index ac3d73484cb2..1cba5fe8777a 100644
> --- a/include/linux/trace_recursion.h
> +++ b/include/linux/trace_recursion.h
> @@ -142,7 +142,28 @@ static __always_inline int trace_get_context_bit(void)
> pc & HARDIRQ_MASK ? TRACE_CTX_IRQ : TRACE_CTX_SOFTIRQ;
> }
>
> -static __always_inline int trace_test_and_set_recursion(int start, int max)
> +#ifdef CONFIG_FTRACE_RECORD_RECURSION
> +extern void ftrace_record_recursion(unsigned long ip, unsigned long parent_ip);
> +/*
> +* The paranoid_test check can cause dropped reports (unlikely), but
> +* if the recursion is common, it will likely still be recorded later.
> +* But the paranoid_test is needed to make sure we don't crash.
> +*/
> +# define do_ftrace_record_recursion(ip, pip) \
> + do { \
> + static atomic_t paranoid_test; \
> + if (!atomic_read(¶noid_test)) { \
> + atomic_inc(¶noid_test); \
> + ftrace_record_recursion(ip, pip); \
> + atomic_dec(¶noid_test); \
BTW: What is actually the purpose of paranoid_test, please?
It prevents nested ftrace_record_recursion() calls on the same CPU
(recursion, nesting from IRQ, NMI context).
Parallel calls from different CPUs are still possible:
CPU0 CPU1
if (!atomic_read(¶noid_test)) if (!atomic_read(¶noid_test))
// passes // passes
atomic_inc(¶noid_test); atomic_inc(¶noid_test);
I do not see how a nested call could cause crash while a parallel
one would be OK.
> --- /dev/null
> +++ b/kernel/trace/trace_recursion_record.c
> @@ -0,0 +1,236 @@
> +// SPDX-License-Identifier: GPL-2.0
> +
> +#include <linux/seq_file.h>
> +#include <linux/kallsyms.h>
> +#include <linux/module.h>
> +#include <linux/ftrace.h>
> +#include <linux/fs.h>
> +
> +#include "trace_output.h"
> +
> +struct recursed_functions {
> + unsigned long ip;
> + unsigned long parent_ip;
> +};
> +
> +static struct recursed_functions recursed_functions[CONFIG_FTRACE_RECORD_RECURSION_SIZE];
> +static atomic_t nr_records;
> +
> +/*
> + * Cache the last found function. Yes, updates to this is racey, but
> + * so is memory cache ;-)
> + */
> +static unsigned long cached_function;
> +
> +void ftrace_record_recursion(unsigned long ip, unsigned long parent_ip)
> +{
> + int index = 0;
> + int i;
> + unsigned long old;
> +
> + again:
> + /* First check the last one recorded */
> + if (ip == cached_function)
> + return;
> +
> + i = atomic_read(&nr_records);
> + /* nr_records is -1 when clearing records */
> + smp_mb__after_atomic();
> + if (i < 0)
> + return;
> +
> + /*
> + * If there's two writers and this writer comes in second,
> + * the cmpxchg() below to update the ip will fail. Then this
> + * writer will try again. It is possible that index will now
> + * be greater than nr_records. This is because the writer
> + * that succeeded has not updated the nr_records yet.
> + * This writer could keep trying again until the other writer
> + * updates nr_records. But if the other writer takes an
> + * interrupt, and that interrupt locks up that CPU, we do
> + * not want this CPU to lock up due to the recursion protection,
> + * and have a bug report showing this CPU as the cause of
> + * locking up the computer. To not lose this record, this
> + * writer will simply use the next position to update the
> + * recursed_functions, and it will update the nr_records
> + * accordingly.
> + */
> + if (index < i)
> + index = i;
> + if (index >= CONFIG_FTRACE_RECORD_RECURSION_SIZE)
> + return;
> +
> + for (i = index - 1; i >= 0; i--) {
> + if (recursed_functions[i].ip == ip) {
> + cached_function = ip;
> + return;
> + }
> + }
> +
> + cached_function = ip;
> +
> + /*
> + * We only want to add a function if it hasn't been added before.
> + * Add to the current location before incrementing the count.
> + * If it fails to add, then increment the index (save in i)
> + * and try again.
> + */
> + old = cmpxchg(&recursed_functions[index].ip, 0, ip);
> + if (old != 0) {
> + /* Did something else already added this for us? */
> + if (old == ip)
> + return;
> + /* Try the next location (use i for the next index) */
> + index++;
> + goto again;
> + }
> +
> + recursed_functions[index].parent_ip = parent_ip;
> +
> + /*
> + * It's still possible that we could race with the clearing
> + * CPU0 CPU1
> + * ---- ----
> + * ip = func
> + * nr_records = -1;
> + * recursed_functions[0] = 0;
> + * i = -1
> + * if (i < 0)
> + * nr_records = 0;
> + * (new recursion detected)
> + * recursed_functions[0] = func
> + * cmpxchg(recursed_functions[0],
> + * func, 0)
> + *
> + * But the worse that could happen is that we get a zero in
> + * the recursed_functions array, and it's likely that "func" will
> + * be recorded again.
> + */
> + i = atomic_read(&nr_records);
> + smp_mb__after_atomic();
> + if (i < 0)
> + cmpxchg(&recursed_functions[index].ip, ip, 0);
> + else if (i <= index)
> + atomic_cmpxchg(&nr_records, i, index + 1);
Are you aware of the following race, please?
CPU0 CPU1
ftrace_record_recursion()
i = atomic_read(&nr_records);
// i = 20 (for example)
if (i < index)
index = i;
// index = 20;
recursed_function_open()
atomic_set(&nr_records, -1);
memset(recursed_functions, 0, );
atomic_set(&nr_records, 0);
// successfully store ip at index == 20
cmpxchg(&recursed_functions[index].ip, 0, ip);
recursed_functions[index].parent_ip = parent_ip;
// check race with clearing
i = atomic_read(&nr_records);
// i == 0
if (i < 0)
// no
else
atomic_cmpxchg(&nr_records, i, index + 1);
RESULT:
+ nr_records == 21
+ and slots 0..19 are zeroed
I played with the code and ended with head entangled by chicken & egg
like problems.
I believe that a solution might be a combined atomic variable from
nr_records + cleanup_count.
ftrace_record_recursion() would be allowed to increase nr_records
only when cleanup_count is still the same. cleanup_count would
be incremented together with clearing nr_records.
Well, I am not sure if it is worth the effort. The race is rather
theoretical. In the worst case, the cache might contain many
zero values.
Anyway, it is yet another experience for me that lockless algorithms
are more tricky that one would expect.
Best Regards,
Petr
^ permalink raw reply
* [PATCH] powerpc: Don't use asm goto for put_user() with GCC 4.9
From: Michael Ellerman @ 2020-11-03 13:29 UTC (permalink / raw)
To: linuxppc-dev; +Cc: schwab
Andreas reported that commit ee0a49a6870e ("powerpc/uaccess: Switch
__put_user_size_allowed() to __put_user_asm_goto()") broke
CLONE_CHILD_SETTID.
Further inspection showed that the put_user() in schedule_tail() was
missing entirely, the store not emitted by the compiler.
<.schedule_tail>:
mflr r0
std r0,16(r1)
stdu r1,-112(r1)
bl <.finish_task_switch>
ld r9,2496(r3)
cmpdi cr7,r9,0
bne cr7,<.schedule_tail+0x60>
ld r3,392(r13)
ld r9,1392(r3)
cmpdi cr7,r9,0
beq cr7,<.schedule_tail+0x3c>
li r4,0
li r5,0
bl <.__task_pid_nr_ns>
nop
bl <.calculate_sigpending>
nop
addi r1,r1,112
ld r0,16(r1)
mtlr r0
blr
nop
nop
nop
bl <.__balance_callback>
b <.schedule_tail+0x1c>
Notice there are no stores other than to the stack. There should be a
stw in there for the store to current->set_child_tid.
This is only seen with GCC 4.9 era compilers (tested with 4.9.3 and
4.9.4), and only when CONFIG_PPC_KUAP is disabled.
When CONFIG_PPC_KUAP=y, the memory clobber that's part of the isync()
and mtspr() inlined via allow_user_access() seems to be enough to
avoid the bug.
For now though let's just not use asm goto with GCC 4.9, to avoid this
bug and any other issues we haven't noticed yet. Possibly in future we
can find a smaller workaround.
This is basically a revert of commit ee0a49a6870e ("powerpc/uaccess:
Switch __put_user_size_allowed() to __put_user_asm_goto()") and commit
7fdf966bed15 ("powerpc/uaccess: Remove __put_user_asm() and
__put_user_asm2()"), but only for GCC 4.9.
With this applied the code generation looks more like it will work:
<.schedule_tail>:
mflr r0
std r31,-8(r1)
std r0,16(r1)
stdu r1,-144(r1)
std r3,112(r1)
bl <._mcount>
nop
ld r3,112(r1)
bl <.finish_task_switch>
ld r9,2624(r3)
cmpdi cr7,r9,0
bne cr7,<.schedule_tail+0xa0>
ld r3,2408(r13)
ld r31,1856(r3)
cmpdi cr7,r31,0
beq cr7,<.schedule_tail+0x80>
li r4,0
li r5,0
bl <.__task_pid_nr_ns>
nop
li r9,-1
clrldi r9,r9,12
cmpld cr7,r31,r9
bgt cr7,<.schedule_tail+0x80>
lis r9,16
rldicr r9,r9,32,31
subf r9,r31,r9
cmpldi cr7,r9,3
ble cr7,<.schedule_tail+0x80>
li r9,0
stw r3,0(r31) <-- stw
nop
bl <.calculate_sigpending>
nop
addi r1,r1,144
ld r0,16(r1)
ld r31,-8(r1)
mtlr r0
blr
nop
bl <.__balance_callback>
b <.schedule_tail+0x30>
Fixes: ee0a49a6870e ("powerpc/uaccess: Switch __put_user_size_allowed() to __put_user_asm_goto()")
Reported-by: Andreas Schwab <schwab@linux-m68k.org>
Signed-off-by: Michael Ellerman <mpe@ellerman.id.au>
---
arch/powerpc/include/asm/uaccess.h | 48 ++++++++++++++++++++++++++++++
1 file changed, 48 insertions(+)
diff --git a/arch/powerpc/include/asm/uaccess.h b/arch/powerpc/include/asm/uaccess.h
index ef5bbb705c08..526a6658946b 100644
--- a/arch/powerpc/include/asm/uaccess.h
+++ b/arch/powerpc/include/asm/uaccess.h
@@ -110,6 +110,52 @@ static inline bool __access_ok(unsigned long addr, unsigned long size)
extern long __put_user_bad(void);
+#if defined(GCC_VERSION) && GCC_VERSION < 50000
+#define __put_user_asm(x, addr, err, op) \
+ __asm__ __volatile__( \
+ "1: " op "%U2%X2 %1,%2 # put_user\n" \
+ "2:\n" \
+ ".section .fixup,\"ax\"\n" \
+ "3: li %0,%3\n" \
+ " b 2b\n" \
+ ".previous\n" \
+ EX_TABLE(1b, 3b) \
+ : "=r" (err) \
+ : "r" (x), "m<>" (*addr), "i" (-EFAULT), "0" (err))
+
+#ifdef __powerpc64__
+#define __put_user_asm2(x, ptr, retval) \
+ __put_user_asm(x, ptr, retval, "std")
+#else /* __powerpc64__ */
+#define __put_user_asm2(x, addr, err) \
+ __asm__ __volatile__( \
+ "1: stw%X2 %1,%2\n" \
+ "2: stw%X2 %L1,%L2\n" \
+ "3:\n" \
+ ".section .fixup,\"ax\"\n" \
+ "4: li %0,%3\n" \
+ " b 3b\n" \
+ ".previous\n" \
+ EX_TABLE(1b, 4b) \
+ EX_TABLE(2b, 4b) \
+ : "=r" (err) \
+ : "r" (x), "m" (*addr), "i" (-EFAULT), "0" (err))
+#endif /* __powerpc64__ */
+
+#define __put_user_size_allowed(x, ptr, size, retval) \
+do { \
+ retval = 0; \
+ switch (size) { \
+ case 1: __put_user_asm(x, ptr, retval, "stb"); break; \
+ case 2: __put_user_asm(x, ptr, retval, "sth"); break; \
+ case 4: __put_user_asm(x, ptr, retval, "stw"); break; \
+ case 8: __put_user_asm2(x, ptr, retval); break; \
+ default: __put_user_bad(); \
+ } \
+} while (0)
+
+#else
+
#define __put_user_size_allowed(x, ptr, size, retval) \
do { \
__label__ __pu_failed; \
@@ -122,6 +168,8 @@ __pu_failed: \
retval = -EFAULT; \
} while (0)
+#endif /* GCC_VERSION */
+
#define __put_user_size(x, ptr, size, retval) \
do { \
allow_write_to_user(ptr, size); \
--
2.25.1
^ permalink raw reply related
* Re: [PATCH 1/3] powerpc/uaccess: Switch __put_user_size_allowed() to __put_user_asm_goto()
From: Michael Ellerman @ 2020-11-03 13:21 UTC (permalink / raw)
To: Andreas Schwab; +Cc: Paul Mackerras, linuxppc-dev, linux-kernel
In-Reply-To: <871rhgwzcg.fsf@igel.home>
Andreas Schwab <schwab@linux-m68k.org> writes:
> #
> # Automatically generated file; DO NOT EDIT.
> # Linux/powerpc 5.10.0-rc1 Kernel Configuration
> #
> CONFIG_CC_VERSION_TEXT="gcc-4.9 (SUSE Linux) 4.9.3"
So it seems to be a combination of GCC 4.9 and ...
> # CONFIG_PPC_RADIX_MMU is not set
That ^, which specifically causes PPC_KUAP=n.
When PPC_KUAP=y allow_user_access() inlines an isync and mtspr, both of
which contain a memory clobber, and that seems to hide the bug.
I think for now we just have to stop using asm goto for put_user() on
GCC 4.9.
I'll send a patch for that.
cheers
^ permalink raw reply
* Re: [PATCH v3 2/4] PM: hibernate: make direct map manipulations more explicit
From: Mike Rapoport @ 2020-11-03 12:13 UTC (permalink / raw)
To: Kirill A. Shutemov
Cc: Rafael J . Wysocki, David Hildenbrand, Peter Zijlstra,
Dave Hansen, linux-mm, Paul Mackerras, Pavel Machek,
H. Peter Anvin, sparclinux, Christoph Lameter, Will Deacon,
linux-riscv, linux-s390, x86, Mike Rapoport,
Christian Borntraeger, Ingo Molnar, Catalin Marinas, Len Brown,
Albert Ou, Vasily Gorbik, linux-pm, Heiko Carstens,
David Rientjes, Borislav Petkov, Andy Lutomirski, Paul Walmsley,
Thomas Gleixner, Joonsoo Kim, linux-arm-kernel, Rafael J. Wysocki,
linux-kernel, Pekka Enberg, Palmer Dabbelt, Andrew Morton,
Edgecombe, Rick P, linuxppc-dev, David S. Miller
In-Reply-To: <20201103110816.t6a3ebtgcm7mfogy@box>
On Tue, Nov 03, 2020 at 02:08:16PM +0300, Kirill A. Shutemov wrote:
> On Sun, Nov 01, 2020 at 07:08:13PM +0200, Mike Rapoport wrote:
> > diff --git a/kernel/power/snapshot.c b/kernel/power/snapshot.c
> > index 46b1804c1ddf..054c8cce4236 100644
> > --- a/kernel/power/snapshot.c
> > +++ b/kernel/power/snapshot.c
> > @@ -76,6 +76,32 @@ static inline void hibernate_restore_protect_page(void *page_address) {}
> > static inline void hibernate_restore_unprotect_page(void *page_address) {}
> > #endif /* CONFIG_STRICT_KERNEL_RWX && CONFIG_ARCH_HAS_SET_MEMORY */
> >
> > +static inline void hibernate_map_page(struct page *page, int enable)
> > +{
> > + if (IS_ENABLED(CONFIG_ARCH_HAS_SET_DIRECT_MAP)) {
> > + unsigned long addr = (unsigned long)page_address(page);
> > + int ret;
> > +
> > + /*
> > + * This should not fail because remapping a page here means
> > + * that we only update protection bits in an existing PTE.
> > + * It is still worth to have WARN_ON() here if something
> > + * changes and this will no longer be the case.
> > + */
> > + if (enable)
> > + ret = set_direct_map_default_noflush(page);
> > + else
> > + ret = set_direct_map_invalid_noflush(page);
> > +
> > + if (WARN_ON(ret))
>
> _ONCE?
I've changed it to pr_warn() after David said people enable panic on
warn in production kernels.
> > + return;
> > +
> > + flush_tlb_kernel_range(addr, addr + PAGE_SIZE);
> > + } else {
> > + debug_pagealloc_map_pages(page, 1, enable);
> > + }
> > +}
> > +
> > static int swsusp_page_is_free(struct page *);
> > static void swsusp_set_page_forbidden(struct page *);
> > static void swsusp_unset_page_forbidden(struct page *);
>
> --
> Kirill A. Shutemov
--
Sincerely yours,
Mike.
^ permalink raw reply
* Re: [PATCH 05/11 v2] kprobes/ftrace: Add recursion protection to the ftrace callback
From: Masami Hiramatsu @ 2020-11-03 11:22 UTC (permalink / raw)
To: Steven Rostedt
Cc: Peter Zijlstra, James E.J. Bottomley, Guo Ren, linux-csky,
H. Peter Anvin, Miroslav Benes, Ingo Molnar, linux-s390,
Helge Deller, x86, Anil S Keshavamurthy, Christian Borntraeger,
Naveen N. Rao, Petr Mladek, Vasily Gorbik, Heiko Carstens,
Jiri Kosina, Borislav Petkov, Josh Poimboeuf, Thomas Gleixner,
linux-parisc, linux-kernel, Masami Hiramatsu, Paul Mackerras,
Andrew Morton, linuxppc-dev, David S. Miller
In-Reply-To: <20201030214013.824581418@goodmis.org>
On Fri, 30 Oct 2020 17:31:47 -0400
Steven Rostedt <rostedt@goodmis.org> wrote:
> From: "Steven Rostedt (VMware)" <rostedt@goodmis.org>
>
> If a ftrace callback does not supply its own recursion protection and
> does not set the RECURSION_SAFE flag in its ftrace_ops, then ftrace will
> make a helper trampoline to do so before calling the callback instead of
> just calling the callback directly.
>
> The default for ftrace_ops is going to change. It will expect that handlers
> provide their own recursion protection, unless its ftrace_ops states
> otherwise.
>
> Link: https://lkml.kernel.org/r/20201028115613.140212174@goodmis.org
>
> Cc: Andrew Morton <akpm@linux-foundation.org>
> Cc: Masami Hiramatsu <mhiramat@kernel.org>
> Cc: Guo Ren <guoren@kernel.org>
> Cc: "James E.J. Bottomley" <James.Bottomley@HansenPartnership.com>
> Cc: Helge Deller <deller@gmx.de>
> Cc: Michael Ellerman <mpe@ellerman.id.au>
> Cc: Benjamin Herrenschmidt <benh@kernel.crashing.org>
> Cc: Paul Mackerras <paulus@samba.org>
> Cc: Heiko Carstens <hca@linux.ibm.com>
> Cc: Vasily Gorbik <gor@linux.ibm.com>
> Cc: Christian Borntraeger <borntraeger@de.ibm.com>
> Cc: Thomas Gleixner <tglx@linutronix.de>
> Cc: Borislav Petkov <bp@alien8.de>
> Cc: x86@kernel.org
> Cc: "H. Peter Anvin" <hpa@zytor.com>
> Cc: "Naveen N. Rao" <naveen.n.rao@linux.ibm.com>
> Cc: Anil S Keshavamurthy <anil.s.keshavamurthy@intel.com>
> Cc: "David S. Miller" <davem@davemloft.net>
> Cc: linux-csky@vger.kernel.org
> Cc: linux-parisc@vger.kernel.org
> Cc: linuxppc-dev@lists.ozlabs.org
> Cc: linux-s390@vger.kernel.org
> Signed-off-by: Steven Rostedt (VMware) <rostedt@goodmis.org>
> ---
> arch/csky/kernel/probes/ftrace.c | 12 ++++++++++--
> arch/parisc/kernel/ftrace.c | 13 +++++++++++--
> arch/powerpc/kernel/kprobes-ftrace.c | 11 ++++++++++-
> arch/s390/kernel/ftrace.c | 13 +++++++++++--
> arch/x86/kernel/kprobes/ftrace.c | 12 ++++++++++--
> 5 files changed, 52 insertions(+), 9 deletions(-)
>
> diff --git a/arch/csky/kernel/probes/ftrace.c b/arch/csky/kernel/probes/ftrace.c
> index 5264763d05be..5eb2604fdf71 100644
> --- a/arch/csky/kernel/probes/ftrace.c
> +++ b/arch/csky/kernel/probes/ftrace.c
> @@ -13,16 +13,21 @@ int arch_check_ftrace_location(struct kprobe *p)
> void kprobe_ftrace_handler(unsigned long ip, unsigned long parent_ip,
> struct ftrace_ops *ops, struct pt_regs *regs)
> {
> + int bit;
> bool lr_saver = false;
> struct kprobe *p;
> struct kprobe_ctlblk *kcb;
>
> - /* Preempt is disabled by ftrace */
> + bit = ftrace_test_recursion_trylock();
> + if (bit < 0)
> + return;
> +
> + preempt_disable_notrace();
> p = get_kprobe((kprobe_opcode_t *)ip);
> if (!p) {
> p = get_kprobe((kprobe_opcode_t *)(ip - MCOUNT_INSN_SIZE));
> if (unlikely(!p) || kprobe_disabled(p))
> - return;
> + goto out;
> lr_saver = true;
> }
>
> @@ -56,6 +61,9 @@ void kprobe_ftrace_handler(unsigned long ip, unsigned long parent_ip,
> */
> __this_cpu_write(current_kprobe, NULL);
> }
> +out:
> + preempt_enable_notrace();
> + ftrace_test_recursion_unlock(bit);
> }
> NOKPROBE_SYMBOL(kprobe_ftrace_handler);
>
> diff --git a/arch/parisc/kernel/ftrace.c b/arch/parisc/kernel/ftrace.c
> index 63e3ecb9da81..4b1fdf15662c 100644
> --- a/arch/parisc/kernel/ftrace.c
> +++ b/arch/parisc/kernel/ftrace.c
> @@ -208,13 +208,19 @@ void kprobe_ftrace_handler(unsigned long ip, unsigned long parent_ip,
> {
> struct kprobe_ctlblk *kcb;
> struct kprobe *p = get_kprobe((kprobe_opcode_t *)ip);
> + int bit;
>
> - if (unlikely(!p) || kprobe_disabled(p))
> + bit = ftrace_test_recursion_trylock();
> + if (bit < 0)
> return;
>
> + preempt_disable_notrace();
If we disable preempt here, we also move the get_kprobe() here as below.
(get_kprobe() accesses percpu variable)
p = get_kprobe((kprobe_opcode_t *)ip);
> + if (unlikely(!p) || kprobe_disabled(p))
> + goto out;
> +
> if (kprobe_running()) {
> kprobes_inc_nmissed_count(p);
> - return;
> + goto out;
> }
>
> __this_cpu_write(current_kprobe, p);
> @@ -235,6 +241,9 @@ void kprobe_ftrace_handler(unsigned long ip, unsigned long parent_ip,
> }
> }
> __this_cpu_write(current_kprobe, NULL);
> +out:
> + preempt_enable_notrace();
> + ftrace_test_recursion_unlock(bit);
> }
> NOKPROBE_SYMBOL(kprobe_ftrace_handler);
>
> diff --git a/arch/powerpc/kernel/kprobes-ftrace.c b/arch/powerpc/kernel/kprobes-ftrace.c
> index 972cb28174b2..5df8d50c65ae 100644
> --- a/arch/powerpc/kernel/kprobes-ftrace.c
> +++ b/arch/powerpc/kernel/kprobes-ftrace.c
> @@ -18,10 +18,16 @@ void kprobe_ftrace_handler(unsigned long nip, unsigned long parent_nip,
> {
> struct kprobe *p;
> struct kprobe_ctlblk *kcb;
> + int bit;
>
> + bit = ftrace_test_recursion_trylock();
> + if (bit < 0)
> + return;
> +
> + preempt_disable_notrace();
> p = get_kprobe((kprobe_opcode_t *)nip);
> if (unlikely(!p) || kprobe_disabled(p))
> - return;
> + goto out;
>
> kcb = get_kprobe_ctlblk();
> if (kprobe_running()) {
> @@ -52,6 +58,9 @@ void kprobe_ftrace_handler(unsigned long nip, unsigned long parent_nip,
> */
> __this_cpu_write(current_kprobe, NULL);
> }
> +out:
> + preempt_enable_notrace();
> + ftrace_test_recursion_unlock(bit);
> }
> NOKPROBE_SYMBOL(kprobe_ftrace_handler);
>
> diff --git a/arch/s390/kernel/ftrace.c b/arch/s390/kernel/ftrace.c
> index b388e87a08bf..88466d7fb6b2 100644
> --- a/arch/s390/kernel/ftrace.c
> +++ b/arch/s390/kernel/ftrace.c
> @@ -202,13 +202,19 @@ void kprobe_ftrace_handler(unsigned long ip, unsigned long parent_ip,
> {
> struct kprobe_ctlblk *kcb;
> struct kprobe *p = get_kprobe((kprobe_opcode_t *)ip);
> + int bit;
>
> - if (unlikely(!p) || kprobe_disabled(p))
> + bit = ftrace_test_recursion_trylock();
> + if (bit < 0)
> return;
>
> + preempt_disable_notrace();
Ditto.
Others look good to me.
Thank you,
> + if (unlikely(!p) || kprobe_disabled(p))
> + goto out;
> +
> if (kprobe_running()) {
> kprobes_inc_nmissed_count(p);
> - return;
> + goto out;
> }
>
> __this_cpu_write(current_kprobe, p);
> @@ -228,6 +234,9 @@ void kprobe_ftrace_handler(unsigned long ip, unsigned long parent_ip,
> }
> }
> __this_cpu_write(current_kprobe, NULL);
> +out:
> + preempt_enable_notrace();
> + ftrace_test_recursion_unlock(bit);
> }
> NOKPROBE_SYMBOL(kprobe_ftrace_handler);
>
> diff --git a/arch/x86/kernel/kprobes/ftrace.c b/arch/x86/kernel/kprobes/ftrace.c
> index 681a4b36e9bb..a40a6cdfcca3 100644
> --- a/arch/x86/kernel/kprobes/ftrace.c
> +++ b/arch/x86/kernel/kprobes/ftrace.c
> @@ -18,11 +18,16 @@ void kprobe_ftrace_handler(unsigned long ip, unsigned long parent_ip,
> {
> struct kprobe *p;
> struct kprobe_ctlblk *kcb;
> + int bit;
>
> - /* Preempt is disabled by ftrace */
> + bit = ftrace_test_recursion_trylock();
> + if (bit < 0)
> + return;
> +
> + preempt_disable_notrace();
> p = get_kprobe((kprobe_opcode_t *)ip);
> if (unlikely(!p) || kprobe_disabled(p))
> - return;
> + goto out;
>
> kcb = get_kprobe_ctlblk();
> if (kprobe_running()) {
> @@ -52,6 +57,9 @@ void kprobe_ftrace_handler(unsigned long ip, unsigned long parent_ip,
> */
> __this_cpu_write(current_kprobe, NULL);
> }
> +out:
> + preempt_enable_notrace();
> + ftrace_test_recursion_unlock(bit);
> }
> NOKPROBE_SYMBOL(kprobe_ftrace_handler);
>
> --
> 2.28.0
>
>
--
Masami Hiramatsu <mhiramat@kernel.org>
^ permalink raw reply
* Re: [PATCH v3 0/4] arch, mm: improve robustness of direct map manipulation
From: Kirill A. Shutemov @ 2020-11-03 11:15 UTC (permalink / raw)
To: Mike Rapoport
Cc: David Hildenbrand, Peter Zijlstra, Dave Hansen, linux-mm,
Paul Mackerras, Pavel Machek, H. Peter Anvin, sparclinux,
Christoph Lameter, Will Deacon, linux-riscv, linux-s390, x86,
Mike Rapoport, Christian Borntraeger, Ingo Molnar,
Catalin Marinas, Len Brown, Albert Ou, Vasily Gorbik, linux-pm,
Heiko Carstens, David Rientjes, Borislav Petkov, Andy Lutomirski,
Paul Walmsley, Thomas Gleixner, Joonsoo Kim, linux-arm-kernel,
Rafael J. Wysocki, linux-kernel, Pekka Enberg, Palmer Dabbelt,
Andrew Morton, Edgecombe, Rick P, linuxppc-dev, David S. Miller
In-Reply-To: <20201101170815.9795-1-rppt@kernel.org>
On Sun, Nov 01, 2020 at 07:08:11PM +0200, Mike Rapoport wrote:
> Mike Rapoport (4):
> mm: introduce debug_pagealloc_map_pages() helper
> PM: hibernate: make direct map manipulations more explicit
> arch, mm: restore dependency of __kernel_map_pages() of DEBUG_PAGEALLOC
> arch, mm: make kernel_page_present() always available
The series looks good to me (apart from the minor nit):
Acked-by: Kirill A. Shutemov <kirill.shutemov@linux.intel.com>
--
Kirill A. Shutemov
^ permalink raw reply
* Re: [PATCH v3 2/4] PM: hibernate: make direct map manipulations more explicit
From: Kirill A. Shutemov @ 2020-11-03 11:08 UTC (permalink / raw)
To: Mike Rapoport
Cc: Rafael J . Wysocki, David Hildenbrand, Peter Zijlstra,
Dave Hansen, linux-mm, Paul Mackerras, Pavel Machek,
H. Peter Anvin, sparclinux, Christoph Lameter, Will Deacon,
linux-riscv, linux-s390, x86, Mike Rapoport,
Christian Borntraeger, Ingo Molnar, Catalin Marinas, Len Brown,
Albert Ou, Vasily Gorbik, linux-pm, Heiko Carstens,
David Rientjes, Borislav Petkov, Andy Lutomirski, Paul Walmsley,
Thomas Gleixner, Joonsoo Kim, linux-arm-kernel, Rafael J. Wysocki,
linux-kernel, Pekka Enberg, Palmer Dabbelt, Andrew Morton,
Edgecombe, Rick P, linuxppc-dev, David S. Miller
In-Reply-To: <20201101170815.9795-3-rppt@kernel.org>
On Sun, Nov 01, 2020 at 07:08:13PM +0200, Mike Rapoport wrote:
> diff --git a/kernel/power/snapshot.c b/kernel/power/snapshot.c
> index 46b1804c1ddf..054c8cce4236 100644
> --- a/kernel/power/snapshot.c
> +++ b/kernel/power/snapshot.c
> @@ -76,6 +76,32 @@ static inline void hibernate_restore_protect_page(void *page_address) {}
> static inline void hibernate_restore_unprotect_page(void *page_address) {}
> #endif /* CONFIG_STRICT_KERNEL_RWX && CONFIG_ARCH_HAS_SET_MEMORY */
>
> +static inline void hibernate_map_page(struct page *page, int enable)
> +{
> + if (IS_ENABLED(CONFIG_ARCH_HAS_SET_DIRECT_MAP)) {
> + unsigned long addr = (unsigned long)page_address(page);
> + int ret;
> +
> + /*
> + * This should not fail because remapping a page here means
> + * that we only update protection bits in an existing PTE.
> + * It is still worth to have WARN_ON() here if something
> + * changes and this will no longer be the case.
> + */
> + if (enable)
> + ret = set_direct_map_default_noflush(page);
> + else
> + ret = set_direct_map_invalid_noflush(page);
> +
> + if (WARN_ON(ret))
_ONCE?
> + return;
> +
> + flush_tlb_kernel_range(addr, addr + PAGE_SIZE);
> + } else {
> + debug_pagealloc_map_pages(page, 1, enable);
> + }
> +}
> +
> static int swsusp_page_is_free(struct page *);
> static void swsusp_set_page_forbidden(struct page *);
> static void swsusp_unset_page_forbidden(struct page *);
--
Kirill A. Shutemov
^ permalink raw reply
* [Bug 209869] Kernel 5.10-rc1 fails to boot on a PowerMac G4 3,6 at an early stage
From: bugzilla-daemon @ 2020-11-03 11:06 UTC (permalink / raw)
To: linuxppc-dev
In-Reply-To: <bug-209869-206035@https.bugzilla.kernel.org/>
https://bugzilla.kernel.org/show_bug.cgi?id=209869
--- Comment #2 from Erhard F. (erhard_f@mailbox.org) ---
Yes, reverting this commit (on top of -rc1) makes the G4 boot again. Did not
try -rc2 though.
--
You are receiving this mail because:
You are watching the assignee of the bug.
^ permalink raw reply
* [patch V3 37/37] io-mapping: Remove io_mapping_map_atomic_wc()
From: Thomas Gleixner @ 2020-11-03 9:27 UTC (permalink / raw)
To: LKML
Cc: Juri Lelli, linux-aio, Peter Zijlstra, Sebastian Andrzej Siewior,
Joonas Lahtinen, dri-devel, virtualization, Ben Segall,
Chris Mason, Huang Rui, Paul Mackerras, Gerd Hoffmann,
Daniel Bristot de Oliveira, sparclinux, Vincent Chen,
Christoph Hellwig, Vincent Guittot, Paul McKenney, Max Filippov,
x86, Russell King, linux-csky, Ingo Molnar, David Airlie,
VMware Graphics, Mel Gorman, nouveau, Dave Airlie, linux-snps-arc,
Ben Skeggs, linux-xtensa, Arnd Bergmann, intel-gfx,
Roland Scheidegger, Josef Bacik, Steven Rostedt, Linus Torvalds,
Alexander Viro, spice-devel, David Sterba, Rodrigo Vivi,
Dietmar Eggemann, linux-arm-kernel, Jani Nikula, Chris Zankel,
Michal Simek, Thomas Bogendoerfer, Nick Hu, linux-mm,
Vineet Gupta, linux-mips, Christian Koenig, Benjamin LaHaise,
Daniel Vetter, linux-fsdevel, Andrew Morton, linuxppc-dev,
David S. Miller, linux-btrfs, Greentime Hu
In-Reply-To: <20201103092712.714480842@linutronix.de>
No more users. Get rid of it and remove the traces in documentation.
Signed-off-by: Thomas Gleixner <tglx@linutronix.de>
---
V3: New patch
---
Documentation/driver-api/io-mapping.rst | 22 +++++-----------
include/linux/io-mapping.h | 42 +-------------------------------
2 files changed, 9 insertions(+), 55 deletions(-)
--- a/Documentation/driver-api/io-mapping.rst
+++ b/Documentation/driver-api/io-mapping.rst
@@ -21,19 +21,15 @@ mappable, while 'size' indicates how lar
enable. Both are in bytes.
This _wc variant provides a mapping which may only be used with
-io_mapping_map_atomic_wc(), io_mapping_map_local_wc() or
-io_mapping_map_wc().
+io_mapping_map_local_wc() or io_mapping_map_wc().
With this mapping object, individual pages can be mapped either temporarily
or long term, depending on the requirements. Of course, temporary maps are
-more efficient. They come in two flavours::
+more efficient.
void *io_mapping_map_local_wc(struct io_mapping *mapping,
unsigned long offset)
- void *io_mapping_map_atomic_wc(struct io_mapping *mapping,
- unsigned long offset)
-
'offset' is the offset within the defined mapping region. Accessing
addresses beyond the region specified in the creation function yields
undefined results. Using an offset which is not page aligned yields an
@@ -50,9 +46,6 @@ io_mapping_map_local_wc() has a side eff
migration to make the mapping code work. No caller can rely on this side
effect.
-io_mapping_map_atomic_wc() has the side effect of disabling preemption and
-pagefaults. Don't use in new code. Use io_mapping_map_local_wc() instead.
-
Nested mappings need to be undone in reverse order because the mapping
code uses a stack for keeping track of them::
@@ -65,11 +58,10 @@ Nested mappings need to be undone in rev
The mappings are released with::
void io_mapping_unmap_local(void *vaddr)
- void io_mapping_unmap_atomic(void *vaddr)
-'vaddr' must be the value returned by the last io_mapping_map_local_wc() or
-io_mapping_map_atomic_wc() call. This unmaps the specified mapping and
-undoes the side effects of the mapping functions.
+'vaddr' must be the value returned by the last io_mapping_map_local_wc()
+call. This unmaps the specified mapping and undoes eventual side effects of
+the mapping function.
If you need to sleep while holding a mapping, you can use the regular
variant, although this may be significantly slower::
@@ -77,8 +69,8 @@ If you need to sleep while holding a map
void *io_mapping_map_wc(struct io_mapping *mapping,
unsigned long offset)
-This works like io_mapping_map_atomic/local_wc() except it has no side
-effects and the pointer is globaly visible.
+This works like io_mapping_map_local_wc() except it has no side effects and
+the pointer is globaly visible.
The mappings are released with::
--- a/include/linux/io-mapping.h
+++ b/include/linux/io-mapping.h
@@ -60,28 +60,7 @@ io_mapping_fini(struct io_mapping *mappi
iomap_free(mapping->base, mapping->size);
}
-/* Atomic map/unmap */
-static inline void __iomem *
-io_mapping_map_atomic_wc(struct io_mapping *mapping,
- unsigned long offset)
-{
- resource_size_t phys_addr;
-
- BUG_ON(offset >= mapping->size);
- phys_addr = mapping->base + offset;
- preempt_disable();
- pagefault_disable();
- return __iomap_local_pfn_prot(PHYS_PFN(phys_addr), mapping->prot);
-}
-
-static inline void
-io_mapping_unmap_atomic(void __iomem *vaddr)
-{
- kunmap_local_indexed((void __force *)vaddr);
- pagefault_enable();
- preempt_enable();
-}
-
+/* Temporary mappings which are only valid in the current context */
static inline void __iomem *
io_mapping_map_local_wc(struct io_mapping *mapping, unsigned long offset)
{
@@ -163,24 +142,7 @@ io_mapping_unmap(void __iomem *vaddr)
{
}
-/* Atomic map/unmap */
-static inline void __iomem *
-io_mapping_map_atomic_wc(struct io_mapping *mapping,
- unsigned long offset)
-{
- preempt_disable();
- pagefault_disable();
- return io_mapping_map_wc(mapping, offset, PAGE_SIZE);
-}
-
-static inline void
-io_mapping_unmap_atomic(void __iomem *vaddr)
-{
- io_mapping_unmap(vaddr);
- pagefault_enable();
- preempt_enable();
-}
-
+/* Temporary mappings which are only valid in the current context */
static inline void __iomem *
io_mapping_map_local_wc(struct io_mapping *mapping, unsigned long offset)
{
^ permalink raw reply
* [patch V3 36/37] drm/i915: Replace io_mapping_map_atomic_wc()
From: Thomas Gleixner @ 2020-11-03 9:27 UTC (permalink / raw)
To: LKML
Cc: Juri Lelli, linux-aio, Peter Zijlstra, Sebastian Andrzej Siewior,
Joonas Lahtinen, dri-devel, virtualization, Ben Segall, linux-mm,
Huang Rui, Paul Mackerras, Gerd Hoffmann,
Daniel Bristot de Oliveira, sparclinux, Dave Airlie, Vincent Chen,
Christoph Hellwig, Vincent Guittot, Arnd Bergmann, Max Filippov,
x86, Russell King, linux-csky, Ingo Molnar, David Airlie,
Steven Rostedt, Mel Gorman, nouveau, VMware Graphics,
linux-snps-arc, Ben Skeggs, linux-xtensa, Paul McKenney,
intel-gfx, Roland Scheidegger, Josef Bacik, Jani Nikula,
Linus Torvalds, Alexander Viro, Rodrigo Vivi, David Sterba,
Dietmar Eggemann, linux-arm-kernel, Chris Zankel, Michal Simek,
Thomas Bogendoerfer, Nick Hu, Chris Mason, Vineet Gupta,
linux-mips, Christian Koenig, Benjamin LaHaise, spice-devel,
Daniel Vetter, linux-fsdevel, Andrew Morton, linuxppc-dev,
David S. Miller, linux-btrfs, Greentime Hu
In-Reply-To: <20201103092712.714480842@linutronix.de>
None of these mapping requires the side effect of disabling pagefaults and
preemption.
Use io_mapping_map_local_wc() instead, and clean up gtt_user_read() and
gtt_user_write() to use a plain copy_from_user() as the local maps are not
disabling pagefaults.
Signed-off-by: Thomas Gleixner <tglx@linutronix.de>
Cc: Jani Nikula <jani.nikula@linux.intel.com>
Cc: Joonas Lahtinen <joonas.lahtinen@linux.intel.com>
Cc: Rodrigo Vivi <rodrigo.vivi@intel.com>
Cc: David Airlie <airlied@linux.ie>
Cc: Daniel Vetter <daniel@ffwll.ch>
Cc: intel-gfx@lists.freedesktop.org
Cc: dri-devel@lists.freedesktop.org
---
V3: New patch
---
drivers/gpu/drm/i915/gem/i915_gem_execbuffer.c | 7 +---
drivers/gpu/drm/i915/i915_gem.c | 40 ++++++++-----------------
drivers/gpu/drm/i915/selftests/i915_gem.c | 4 +-
drivers/gpu/drm/i915/selftests/i915_gem_gtt.c | 8 ++---
4 files changed, 22 insertions(+), 37 deletions(-)
--- a/drivers/gpu/drm/i915/gem/i915_gem_execbuffer.c
+++ b/drivers/gpu/drm/i915/gem/i915_gem_execbuffer.c
@@ -1081,7 +1081,7 @@ static void reloc_cache_reset(struct rel
struct i915_ggtt *ggtt = cache_to_ggtt(cache);
intel_gt_flush_ggtt_writes(ggtt->vm.gt);
- io_mapping_unmap_atomic((void __iomem *)vaddr);
+ io_mapping_unmap_local((void __iomem *)vaddr);
if (drm_mm_node_allocated(&cache->node)) {
ggtt->vm.clear_range(&ggtt->vm,
@@ -1147,7 +1147,7 @@ static void *reloc_iomap(struct drm_i915
if (cache->vaddr) {
intel_gt_flush_ggtt_writes(ggtt->vm.gt);
- io_mapping_unmap_atomic((void __force __iomem *) unmask_page(cache->vaddr));
+ io_mapping_unmap_local((void __force __iomem *) unmask_page(cache->vaddr));
} else {
struct i915_vma *vma;
int err;
@@ -1195,8 +1195,7 @@ static void *reloc_iomap(struct drm_i915
offset += page << PAGE_SHIFT;
}
- vaddr = (void __force *)io_mapping_map_atomic_wc(&ggtt->iomap,
- offset);
+ vaddr = (void __force *)io_mapping_map_local_wc(&ggtt->iomap, offset);
cache->page = page;
cache->vaddr = (unsigned long)vaddr;
--- a/drivers/gpu/drm/i915/i915_gem.c
+++ b/drivers/gpu/drm/i915/i915_gem.c
@@ -379,22 +379,15 @@ gtt_user_read(struct io_mapping *mapping
char __user *user_data, int length)
{
void __iomem *vaddr;
- unsigned long unwritten;
+ bool fail = false;
/* We can use the cpu mem copy function because this is X86. */
- vaddr = io_mapping_map_atomic_wc(mapping, base);
- unwritten = __copy_to_user_inatomic(user_data,
- (void __force *)vaddr + offset,
- length);
- io_mapping_unmap_atomic(vaddr);
- if (unwritten) {
- vaddr = io_mapping_map_wc(mapping, base, PAGE_SIZE);
- unwritten = copy_to_user(user_data,
- (void __force *)vaddr + offset,
- length);
- io_mapping_unmap(vaddr);
- }
- return unwritten;
+ vaddr = io_mapping_map_local_wc(mapping, base);
+ if (copy_to_user(user_data, (void __force *)vaddr + offset, length))
+ fail = true;
+ io_mapping_unmap_local(vaddr);
+
+ return fail;
}
static int
@@ -557,21 +550,14 @@ ggtt_write(struct io_mapping *mapping,
char __user *user_data, int length)
{
void __iomem *vaddr;
- unsigned long unwritten;
+ bool fail = false;
/* We can use the cpu mem copy function because this is X86. */
- vaddr = io_mapping_map_atomic_wc(mapping, base);
- unwritten = __copy_from_user_inatomic_nocache((void __force *)vaddr + offset,
- user_data, length);
- io_mapping_unmap_atomic(vaddr);
- if (unwritten) {
- vaddr = io_mapping_map_wc(mapping, base, PAGE_SIZE);
- unwritten = copy_from_user((void __force *)vaddr + offset,
- user_data, length);
- io_mapping_unmap(vaddr);
- }
-
- return unwritten;
+ vaddr = io_mapping_map_local_wc(mapping, base);
+ if (copy_from_user((void __force *)vaddr + offset, user_data, length))
+ fail = true;
+ io_mapping_unmap_local(vaddr);
+ return fail;
}
/**
--- a/drivers/gpu/drm/i915/selftests/i915_gem.c
+++ b/drivers/gpu/drm/i915/selftests/i915_gem.c
@@ -57,12 +57,12 @@ static void trash_stolen(struct drm_i915
ggtt->vm.insert_page(&ggtt->vm, dma, slot, I915_CACHE_NONE, 0);
- s = io_mapping_map_atomic_wc(&ggtt->iomap, slot);
+ s = io_mapping_map_local_wc(&ggtt->iomap, slot);
for (x = 0; x < PAGE_SIZE / sizeof(u32); x++) {
prng = next_pseudo_random32(prng);
iowrite32(prng, &s[x]);
}
- io_mapping_unmap_atomic(s);
+ io_mapping_unmap_local(s);
}
ggtt->vm.clear_range(&ggtt->vm, slot, PAGE_SIZE);
--- a/drivers/gpu/drm/i915/selftests/i915_gem_gtt.c
+++ b/drivers/gpu/drm/i915/selftests/i915_gem_gtt.c
@@ -1200,9 +1200,9 @@ static int igt_ggtt_page(void *arg)
u64 offset = tmp.start + order[n] * PAGE_SIZE;
u32 __iomem *vaddr;
- vaddr = io_mapping_map_atomic_wc(&ggtt->iomap, offset);
+ vaddr = io_mapping_map_local_wc(&ggtt->iomap, offset);
iowrite32(n, vaddr + n);
- io_mapping_unmap_atomic(vaddr);
+ io_mapping_unmap_local(vaddr);
}
intel_gt_flush_ggtt_writes(ggtt->vm.gt);
@@ -1212,9 +1212,9 @@ static int igt_ggtt_page(void *arg)
u32 __iomem *vaddr;
u32 val;
- vaddr = io_mapping_map_atomic_wc(&ggtt->iomap, offset);
+ vaddr = io_mapping_map_local_wc(&ggtt->iomap, offset);
val = ioread32(vaddr + n);
- io_mapping_unmap_atomic(vaddr);
+ io_mapping_unmap_local(vaddr);
if (val != n) {
pr_err("insert page failed: found %d, expected %d\n",
^ permalink raw reply
* [patch V3 35/37] drm/nouveau/device: Replace io_mapping_map_atomic_wc()
From: Thomas Gleixner @ 2020-11-03 9:27 UTC (permalink / raw)
To: LKML
Cc: Juri Lelli, linux-aio, Peter Zijlstra, nouveau,
Sebastian Andrzej Siewior, Joonas Lahtinen, dri-devel,
virtualization, Ben Segall, linux-mm, Huang Rui, Paul Mackerras,
Gerd Hoffmann, Daniel Bristot de Oliveira, sparclinux,
Vincent Chen, Christoph Hellwig, Vincent Guittot, Arnd Bergmann,
Max Filippov, x86, Russell King, linux-csky, Ingo Molnar,
David Airlie, VMware Graphics, Ben Skeggs, Dave Airlie,
linux-snps-arc, Mel Gorman, linux-xtensa, Paul McKenney,
intel-gfx, Roland Scheidegger, Josef Bacik, Steven Rostedt,
Linus Torvalds, Alexander Viro, spice-devel, David Sterba,
Rodrigo Vivi, Dietmar Eggemann, linux-arm-kernel, Jani Nikula,
Chris Zankel, Michal Simek, Thomas Bogendoerfer, Nick Hu,
Chris Mason, Vineet Gupta, linux-mips, Christian Koenig,
Benjamin LaHaise, Daniel Vetter, linux-fsdevel, Andrew Morton,
linuxppc-dev, David S. Miller, linux-btrfs, Greentime Hu
In-Reply-To: <20201103092712.714480842@linutronix.de>
Neither fbmem_peek() nor fbmem_poke() require to disable pagefaults and
preemption as a side effect of io_mapping_map_atomic_wc().
Use io_mapping_map_local_wc() instead.
Signed-off-by: Thomas Gleixner <tglx@linutronix.de>
Cc: Ben Skeggs <bskeggs@redhat.com>
Cc: David Airlie <airlied@linux.ie>
Cc: Daniel Vetter <daniel@ffwll.ch>
Cc: dri-devel@lists.freedesktop.org
Cc: nouveau@lists.freedesktop.org
---
V3: New patch
---
drivers/gpu/drm/nouveau/nvkm/subdev/devinit/fbmem.h | 8 ++++----
1 file changed, 4 insertions(+), 4 deletions(-)
--- a/drivers/gpu/drm/nouveau/nvkm/subdev/devinit/fbmem.h
+++ b/drivers/gpu/drm/nouveau/nvkm/subdev/devinit/fbmem.h
@@ -60,19 +60,19 @@ fbmem_fini(struct io_mapping *fb)
static inline u32
fbmem_peek(struct io_mapping *fb, u32 off)
{
- u8 __iomem *p = io_mapping_map_atomic_wc(fb, off & PAGE_MASK);
+ u8 __iomem *p = io_mapping_map_local_wc(fb, off & PAGE_MASK);
u32 val = ioread32(p + (off & ~PAGE_MASK));
- io_mapping_unmap_atomic(p);
+ io_mapping_unmap_local(p);
return val;
}
static inline void
fbmem_poke(struct io_mapping *fb, u32 off, u32 val)
{
- u8 __iomem *p = io_mapping_map_atomic_wc(fb, off & PAGE_MASK);
+ u8 __iomem *p = io_mapping_map_local_wc(fb, off & PAGE_MASK);
iowrite32(val, p + (off & ~PAGE_MASK));
wmb();
- io_mapping_unmap_atomic(p);
+ io_mapping_unmap_local(p);
}
static inline bool
^ permalink raw reply
* [patch V3 34/37] drm/qxl: Replace io_mapping_map_atomic_wc()
From: Thomas Gleixner @ 2020-11-03 9:27 UTC (permalink / raw)
To: LKML
Cc: Juri Lelli, linux-aio, Peter Zijlstra, Sebastian Andrzej Siewior,
Joonas Lahtinen, dri-devel, virtualization, Ben Segall, linux-mm,
Huang Rui, Paul Mackerras, Gerd Hoffmann,
Daniel Bristot de Oliveira, sparclinux, Vincent Chen,
Christoph Hellwig, Vincent Guittot, Arnd Bergmann, Max Filippov,
x86, Russell King, linux-csky, Ingo Molnar, David Airlie,
VMware Graphics, Mel Gorman, nouveau, spice-devel, linux-snps-arc,
Ben Skeggs, linux-xtensa, Paul McKenney, intel-gfx,
Roland Scheidegger, Josef Bacik, Steven Rostedt, Linus Torvalds,
Alexander Viro, Dave Airlie, David Sterba, Rodrigo Vivi,
Dietmar Eggemann, linux-arm-kernel, Jani Nikula, Chris Zankel,
Michal Simek, Thomas Bogendoerfer, Nick Hu, Chris Mason,
Vineet Gupta, linux-mips, Christian Koenig, Benjamin LaHaise,
Daniel Vetter, linux-fsdevel, Andrew Morton, linuxppc-dev,
David S. Miller, linux-btrfs, Greentime Hu
In-Reply-To: <20201103092712.714480842@linutronix.de>
None of these mapping requires the side effect of disabling pagefaults and
preemption.
Use io_mapping_map_local_wc() instead, rename the related functions
accordingly and clean up qxl_process_single_command() to use a plain
copy_from_user() as the local maps are not disabling pagefaults.
Signed-off-by: Thomas Gleixner <tglx@linutronix.de>
Cc: Dave Airlie <airlied@redhat.com>
Cc: Gerd Hoffmann <kraxel@redhat.com>
Cc: David Airlie <airlied@linux.ie>
Cc: Daniel Vetter <daniel@ffwll.ch>
Cc: virtualization@lists.linux-foundation.org
Cc: spice-devel@lists.freedesktop.org
---
V3: New patch
---
drivers/gpu/drm/qxl/qxl_image.c | 18 +++++++++---------
drivers/gpu/drm/qxl/qxl_ioctl.c | 27 +++++++++++++--------------
drivers/gpu/drm/qxl/qxl_object.c | 12 ++++++------
drivers/gpu/drm/qxl/qxl_object.h | 4 ++--
drivers/gpu/drm/qxl/qxl_release.c | 4 ++--
5 files changed, 32 insertions(+), 33 deletions(-)
--- a/drivers/gpu/drm/qxl/qxl_image.c
+++ b/drivers/gpu/drm/qxl/qxl_image.c
@@ -124,12 +124,12 @@ qxl_image_init_helper(struct qxl_device
wrong (check the bitmaps are sent correctly
first) */
- ptr = qxl_bo_kmap_atomic_page(qdev, chunk_bo, 0);
+ ptr = qxl_bo_kmap_local_page(qdev, chunk_bo, 0);
chunk = ptr;
chunk->data_size = height * chunk_stride;
chunk->prev_chunk = 0;
chunk->next_chunk = 0;
- qxl_bo_kunmap_atomic_page(qdev, chunk_bo, ptr);
+ qxl_bo_kunmap_local_page(qdev, chunk_bo, ptr);
{
void *k_data, *i_data;
@@ -143,7 +143,7 @@ qxl_image_init_helper(struct qxl_device
i_data = (void *)data;
while (remain > 0) {
- ptr = qxl_bo_kmap_atomic_page(qdev, chunk_bo, page << PAGE_SHIFT);
+ ptr = qxl_bo_kmap_local_page(qdev, chunk_bo, page << PAGE_SHIFT);
if (page == 0) {
chunk = ptr;
@@ -157,7 +157,7 @@ qxl_image_init_helper(struct qxl_device
memcpy(k_data, i_data, size);
- qxl_bo_kunmap_atomic_page(qdev, chunk_bo, ptr);
+ qxl_bo_kunmap_local_page(qdev, chunk_bo, ptr);
i_data += size;
remain -= size;
page++;
@@ -175,10 +175,10 @@ qxl_image_init_helper(struct qxl_device
page_offset = offset_in_page(out_offset);
size = min((int)(PAGE_SIZE - page_offset), remain);
- ptr = qxl_bo_kmap_atomic_page(qdev, chunk_bo, page_base);
+ ptr = qxl_bo_kmap_local_page(qdev, chunk_bo, page_base);
k_data = ptr + page_offset;
memcpy(k_data, i_data, size);
- qxl_bo_kunmap_atomic_page(qdev, chunk_bo, ptr);
+ qxl_bo_kunmap_local_page(qdev, chunk_bo, ptr);
remain -= size;
i_data += size;
out_offset += size;
@@ -189,7 +189,7 @@ qxl_image_init_helper(struct qxl_device
qxl_bo_kunmap(chunk_bo);
image_bo = dimage->bo;
- ptr = qxl_bo_kmap_atomic_page(qdev, image_bo, 0);
+ ptr = qxl_bo_kmap_local_page(qdev, image_bo, 0);
image = ptr;
image->descriptor.id = 0;
@@ -212,7 +212,7 @@ qxl_image_init_helper(struct qxl_device
break;
default:
DRM_ERROR("unsupported image bit depth\n");
- qxl_bo_kunmap_atomic_page(qdev, image_bo, ptr);
+ qxl_bo_kunmap_local_page(qdev, image_bo, ptr);
return -EINVAL;
}
image->u.bitmap.flags = QXL_BITMAP_TOP_DOWN;
@@ -222,7 +222,7 @@ qxl_image_init_helper(struct qxl_device
image->u.bitmap.palette = 0;
image->u.bitmap.data = qxl_bo_physical_address(qdev, chunk_bo, 0);
- qxl_bo_kunmap_atomic_page(qdev, image_bo, ptr);
+ qxl_bo_kunmap_local_page(qdev, image_bo, ptr);
return 0;
}
--- a/drivers/gpu/drm/qxl/qxl_ioctl.c
+++ b/drivers/gpu/drm/qxl/qxl_ioctl.c
@@ -89,11 +89,11 @@ apply_reloc(struct qxl_device *qdev, str
{
void *reloc_page;
- reloc_page = qxl_bo_kmap_atomic_page(qdev, info->dst_bo, info->dst_offset & PAGE_MASK);
+ reloc_page = qxl_bo_kmap_local_page(qdev, info->dst_bo, info->dst_offset & PAGE_MASK);
*(uint64_t *)(reloc_page + (info->dst_offset & ~PAGE_MASK)) = qxl_bo_physical_address(qdev,
info->src_bo,
info->src_offset);
- qxl_bo_kunmap_atomic_page(qdev, info->dst_bo, reloc_page);
+ qxl_bo_kunmap_local_page(qdev, info->dst_bo, reloc_page);
}
static void
@@ -105,9 +105,9 @@ apply_surf_reloc(struct qxl_device *qdev
if (info->src_bo && !info->src_bo->is_primary)
id = info->src_bo->surface_id;
- reloc_page = qxl_bo_kmap_atomic_page(qdev, info->dst_bo, info->dst_offset & PAGE_MASK);
+ reloc_page = qxl_bo_kmap_local_page(qdev, info->dst_bo, info->dst_offset & PAGE_MASK);
*(uint32_t *)(reloc_page + (info->dst_offset & ~PAGE_MASK)) = id;
- qxl_bo_kunmap_atomic_page(qdev, info->dst_bo, reloc_page);
+ qxl_bo_kunmap_local_page(qdev, info->dst_bo, reloc_page);
}
/* return holding the reference to this object */
@@ -149,7 +149,6 @@ static int qxl_process_single_command(st
struct qxl_bo *cmd_bo;
void *fb_cmd;
int i, ret, num_relocs;
- int unwritten;
switch (cmd->type) {
case QXL_CMD_DRAW:
@@ -185,21 +184,21 @@ static int qxl_process_single_command(st
goto out_free_reloc;
/* TODO copy slow path code from i915 */
- fb_cmd = qxl_bo_kmap_atomic_page(qdev, cmd_bo, (release->release_offset & PAGE_MASK));
- unwritten = __copy_from_user_inatomic_nocache
- (fb_cmd + sizeof(union qxl_release_info) + (release->release_offset & ~PAGE_MASK),
- u64_to_user_ptr(cmd->command), cmd->command_size);
+ fb_cmd = qxl_bo_kmap_local_page(qdev, cmd_bo, (release->release_offset & PAGE_MASK));
- {
+ if (copy_from_user(fb_cmd + sizeof(union qxl_release_info) +
+ (release->release_offset & ~PAGE_MASK),
+ u64_to_user_ptr(cmd->command), cmd->command_size)) {
+ ret = -EFAULT;
+ } else {
struct qxl_drawable *draw = fb_cmd;
draw->mm_time = qdev->rom->mm_clock;
}
- qxl_bo_kunmap_atomic_page(qdev, cmd_bo, fb_cmd);
- if (unwritten) {
- DRM_ERROR("got unwritten %d\n", unwritten);
- ret = -EFAULT;
+ qxl_bo_kunmap_local_page(qdev, cmd_bo, fb_cmd);
+ if (ret) {
+ DRM_ERROR("copy from user failed %d\n", ret);
goto out_free_release;
}
--- a/drivers/gpu/drm/qxl/qxl_object.c
+++ b/drivers/gpu/drm/qxl/qxl_object.c
@@ -172,8 +172,8 @@ int qxl_bo_kmap(struct qxl_bo *bo, void
return 0;
}
-void *qxl_bo_kmap_atomic_page(struct qxl_device *qdev,
- struct qxl_bo *bo, int page_offset)
+void *qxl_bo_kmap_local_page(struct qxl_device *qdev,
+ struct qxl_bo *bo, int page_offset)
{
unsigned long offset;
void *rptr;
@@ -188,7 +188,7 @@ void *qxl_bo_kmap_atomic_page(struct qxl
goto fallback;
offset = bo->tbo.mem.start << PAGE_SHIFT;
- return io_mapping_map_atomic_wc(map, offset + page_offset);
+ return io_mapping_map_local_wc(map, offset + page_offset);
fallback:
if (bo->kptr) {
rptr = bo->kptr + (page_offset * PAGE_SIZE);
@@ -214,14 +214,14 @@ void qxl_bo_kunmap(struct qxl_bo *bo)
ttm_bo_kunmap(&bo->kmap);
}
-void qxl_bo_kunmap_atomic_page(struct qxl_device *qdev,
- struct qxl_bo *bo, void *pmap)
+void qxl_bo_kunmap_local_page(struct qxl_device *qdev,
+ struct qxl_bo *bo, void *pmap)
{
if ((bo->tbo.mem.mem_type != TTM_PL_VRAM) &&
(bo->tbo.mem.mem_type != TTM_PL_PRIV))
goto fallback;
- io_mapping_unmap_atomic(pmap);
+ io_mapping_unmap_local(pmap);
return;
fallback:
qxl_bo_kunmap(bo);
--- a/drivers/gpu/drm/qxl/qxl_object.h
+++ b/drivers/gpu/drm/qxl/qxl_object.h
@@ -88,8 +88,8 @@ extern int qxl_bo_create(struct qxl_devi
struct qxl_bo **bo_ptr);
extern int qxl_bo_kmap(struct qxl_bo *bo, void **ptr);
extern void qxl_bo_kunmap(struct qxl_bo *bo);
-void *qxl_bo_kmap_atomic_page(struct qxl_device *qdev, struct qxl_bo *bo, int page_offset);
-void qxl_bo_kunmap_atomic_page(struct qxl_device *qdev, struct qxl_bo *bo, void *map);
+void *qxl_bo_kmap_local_page(struct qxl_device *qdev, struct qxl_bo *bo, int page_offset);
+void qxl_bo_kunmap_local_page(struct qxl_device *qdev, struct qxl_bo *bo, void *map);
extern struct qxl_bo *qxl_bo_ref(struct qxl_bo *bo);
extern void qxl_bo_unref(struct qxl_bo **bo);
extern int qxl_bo_pin(struct qxl_bo *bo);
--- a/drivers/gpu/drm/qxl/qxl_release.c
+++ b/drivers/gpu/drm/qxl/qxl_release.c
@@ -408,7 +408,7 @@ union qxl_release_info *qxl_release_map(
union qxl_release_info *info;
struct qxl_bo *bo = release->release_bo;
- ptr = qxl_bo_kmap_atomic_page(qdev, bo, release->release_offset & PAGE_MASK);
+ ptr = qxl_bo_kmap_local_page(qdev, bo, release->release_offset & PAGE_MASK);
if (!ptr)
return NULL;
info = ptr + (release->release_offset & ~PAGE_MASK);
@@ -423,7 +423,7 @@ void qxl_release_unmap(struct qxl_device
void *ptr;
ptr = ((void *)info) - (release->release_offset & ~PAGE_MASK);
- qxl_bo_kunmap_atomic_page(qdev, bo, ptr);
+ qxl_bo_kunmap_local_page(qdev, bo, ptr);
}
void qxl_release_fence_buffer_objects(struct qxl_release *release)
^ permalink raw reply
* [patch V3 32/37] drm/vmgfx: Replace kmap_atomic()
From: Thomas Gleixner @ 2020-11-03 9:27 UTC (permalink / raw)
To: LKML
Cc: Juri Lelli, linux-aio, Peter Zijlstra, Sebastian Andrzej Siewior,
Joonas Lahtinen, dri-devel, virtualization, Ben Segall, linux-mm,
Huang Rui, Paul Mackerras, Gerd Hoffmann,
Daniel Bristot de Oliveira, sparclinux, Vincent Chen,
Christoph Hellwig, Vincent Guittot, Arnd Bergmann, Max Filippov,
x86, Russell King, linux-csky, Ingo Molnar, David Airlie,
VMware Graphics, Mel Gorman, nouveau, Dave Airlie, linux-snps-arc,
Ben Skeggs, linux-xtensa, Paul McKenney, intel-gfx,
Roland Scheidegger, Josef Bacik, Steven Rostedt, Linus Torvalds,
Alexander Viro, spice-devel, David Sterba, Rodrigo Vivi,
Dietmar Eggemann, linux-arm-kernel, Jani Nikula, Chris Zankel,
Michal Simek, Thomas Bogendoerfer, Nick Hu, Chris Mason,
Vineet Gupta, linux-mips, Christian Koenig, Benjamin LaHaise,
Daniel Vetter, linux-fsdevel, Andrew Morton, linuxppc-dev,
David S. Miller, linux-btrfs, Greentime Hu
In-Reply-To: <20201103092712.714480842@linutronix.de>
There is no reason to disable pagefaults and preemption as a side effect of
kmap_atomic_prot().
Use kmap_local_page_prot() instead and document the reasoning for the
mapping usage with the given pgprot.
Remove the NULL pointer check for the map. These functions return a valid
address for valid pages and the return was bogus anyway as it would have
left preemption and pagefaults disabled.
Signed-off-by: Thomas Gleixner <tglx@linutronix.de>
Cc: VMware Graphics <linux-graphics-maintainer@vmware.com>
Cc: Roland Scheidegger <sroland@vmware.com>
Cc: David Airlie <airlied@linux.ie>
Cc: Daniel Vetter <daniel@ffwll.ch>
Cc: dri-devel@lists.freedesktop.org
---
V3: New patch
---
drivers/gpu/drm/vmwgfx/vmwgfx_blit.c | 30 ++++++++++++------------------
1 file changed, 12 insertions(+), 18 deletions(-)
--- a/drivers/gpu/drm/vmwgfx/vmwgfx_blit.c
+++ b/drivers/gpu/drm/vmwgfx/vmwgfx_blit.c
@@ -375,12 +375,12 @@ static int vmw_bo_cpu_blit_line(struct v
copy_size = min_t(u32, copy_size, PAGE_SIZE - src_page_offset);
if (unmap_src) {
- kunmap_atomic(d->src_addr);
+ kunmap_local(d->src_addr);
d->src_addr = NULL;
}
if (unmap_dst) {
- kunmap_atomic(d->dst_addr);
+ kunmap_local(d->dst_addr);
d->dst_addr = NULL;
}
@@ -388,12 +388,8 @@ static int vmw_bo_cpu_blit_line(struct v
if (WARN_ON_ONCE(dst_page >= d->dst_num_pages))
return -EINVAL;
- d->dst_addr =
- kmap_atomic_prot(d->dst_pages[dst_page],
- d->dst_prot);
- if (!d->dst_addr)
- return -ENOMEM;
-
+ d->dst_addr = kmap_local_page_prot(d->dst_pages[dst_page],
+ d->dst_prot);
d->mapped_dst = dst_page;
}
@@ -401,12 +397,8 @@ static int vmw_bo_cpu_blit_line(struct v
if (WARN_ON_ONCE(src_page >= d->src_num_pages))
return -EINVAL;
- d->src_addr =
- kmap_atomic_prot(d->src_pages[src_page],
- d->src_prot);
- if (!d->src_addr)
- return -ENOMEM;
-
+ d->src_addr = kmap_local_page_prot(d->src_pages[src_page],
+ d->src_prot);
d->mapped_src = src_page;
}
diff->do_cpy(diff, d->dst_addr + dst_page_offset,
@@ -436,8 +428,10 @@ static int vmw_bo_cpu_blit_line(struct v
*
* Performs a CPU blit from one buffer object to another avoiding a full
* bo vmap which may exhaust- or fragment vmalloc space.
- * On supported architectures (x86), we're using kmap_atomic which avoids
- * cross-processor TLB- and cache flushes and may, on non-HIGHMEM systems
+ *
+ * On supported architectures (x86), we're using kmap_local_prot() which
+ * avoids cross-processor TLB- and cache flushes. kmap_local_prot() will
+ * either map a highmem page with the proper pgprot on HIGHMEM=y systems or
* reference already set-up mappings.
*
* Neither of the buffer objects may be placed in PCI memory
@@ -500,9 +494,9 @@ int vmw_bo_cpu_blit(struct ttm_buffer_ob
}
out:
if (d.src_addr)
- kunmap_atomic(d.src_addr);
+ kunmap_local(d.src_addr);
if (d.dst_addr)
- kunmap_atomic(d.dst_addr);
+ kunmap_local(d.dst_addr);
return ret;
}
^ permalink raw reply
* [patch V3 33/37] highmem: Remove kmap_atomic_prot()
From: Thomas Gleixner @ 2020-11-03 9:27 UTC (permalink / raw)
To: LKML
Cc: Juri Lelli, linux-aio, Peter Zijlstra, Sebastian Andrzej Siewior,
Joonas Lahtinen, dri-devel, virtualization, Ben Segall,
Chris Mason, Huang Rui, Paul Mackerras, Gerd Hoffmann,
Daniel Bristot de Oliveira, sparclinux, Vincent Chen,
Christoph Hellwig, Vincent Guittot, Paul McKenney, Max Filippov,
x86, Russell King, linux-csky, Ingo Molnar, David Airlie,
VMware Graphics, Mel Gorman, nouveau, Dave Airlie, linux-snps-arc,
Ben Skeggs, linux-xtensa, Arnd Bergmann, intel-gfx,
Roland Scheidegger, Josef Bacik, Steven Rostedt, Linus Torvalds,
Alexander Viro, spice-devel, David Sterba, Rodrigo Vivi,
Dietmar Eggemann, linux-arm-kernel, Jani Nikula, Chris Zankel,
Michal Simek, Thomas Bogendoerfer, Nick Hu, linux-mm,
Vineet Gupta, linux-mips, Christian Koenig, Benjamin LaHaise,
Daniel Vetter, linux-fsdevel, Andrew Morton, linuxppc-dev,
David S. Miller, linux-btrfs, Greentime Hu
In-Reply-To: <20201103092712.714480842@linutronix.de>
No more users.
Signed-off-by: Thomas Gleixner <tglx@linutronix.de>
---
V3: New patch
---
include/linux/highmem-internal.h | 14 ++------------
1 file changed, 2 insertions(+), 12 deletions(-)
--- a/include/linux/highmem-internal.h
+++ b/include/linux/highmem-internal.h
@@ -87,16 +87,11 @@ static inline void __kunmap_local(void *
kunmap_local_indexed(vaddr);
}
-static inline void *kmap_atomic_prot(struct page *page, pgprot_t prot)
+static inline void *kmap_atomic(struct page *page)
{
preempt_disable();
pagefault_disable();
- return __kmap_local_page_prot(page, prot);
-}
-
-static inline void *kmap_atomic(struct page *page)
-{
- return kmap_atomic_prot(page, kmap_prot);
+ return __kmap_local_page_prot(page, kmap_prot);
}
static inline void __kunmap_atomic(void *addr)
@@ -181,11 +176,6 @@ static inline void *kmap_atomic(struct p
return page_address(page);
}
-static inline void *kmap_atomic_prot(struct page *page, pgprot_t prot)
-{
- return kmap_atomic(page);
-}
-
static inline void __kunmap_atomic(void *addr)
{
#ifdef ARCH_HAS_FLUSH_ON_KUNMAP
^ permalink raw reply
* [patch V3 31/37] drm/ttm: Replace kmap_atomic() usage
From: Thomas Gleixner @ 2020-11-03 9:27 UTC (permalink / raw)
To: LKML
Cc: Juri Lelli, linux-aio, Peter Zijlstra, Sebastian Andrzej Siewior,
Joonas Lahtinen, dri-devel, virtualization, Ben Segall, linux-mm,
Huang Rui, Paul Mackerras, Gerd Hoffmann,
Daniel Bristot de Oliveira, sparclinux, Vincent Chen,
Christoph Hellwig, Vincent Guittot, Arnd Bergmann, Max Filippov,
x86, Russell King, linux-csky, Ingo Molnar, David Airlie,
VMware Graphics, Mel Gorman, nouveau, Dave Airlie, linux-snps-arc,
Ben Skeggs, linux-xtensa, Paul McKenney, intel-gfx,
Roland Scheidegger, Josef Bacik, Steven Rostedt, Linus Torvalds,
Alexander Viro, spice-devel, David Sterba, Rodrigo Vivi,
Dietmar Eggemann, linux-arm-kernel, Jani Nikula, Chris Zankel,
Michal Simek, Thomas Bogendoerfer, Nick Hu, Chris Mason,
Vineet Gupta, linux-mips, David S. Miller, Benjamin LaHaise,
Daniel Vetter, linux-fsdevel, Andrew Morton, linuxppc-dev,
Christian Koenig, linux-btrfs, Greentime Hu
In-Reply-To: <20201103092712.714480842@linutronix.de>
There is no reason to disable pagefaults and preemption as a side effect of
kmap_atomic_prot().
Use kmap_local_page_prot() instead and document the reasoning for the
mapping usage with the given pgprot.
Remove the NULL pointer check for the map. These functions return a valid
address for valid pages and the return was bogus anyway as it would have
left preemption and pagefaults disabled.
Signed-off-by: Thomas Gleixner <tglx@linutronix.de>
Cc: Christian Koenig <christian.koenig@amd.com>
Cc: Huang Rui <ray.huang@amd.com>
Cc: David Airlie <airlied@linux.ie>
Cc: Daniel Vetter <daniel@ffwll.ch>
Cc: dri-devel@lists.freedesktop.org
---
V3: New patch
---
drivers/gpu/drm/ttm/ttm_bo_util.c | 20 ++++++++++++--------
1 file changed, 12 insertions(+), 8 deletions(-)
--- a/drivers/gpu/drm/ttm/ttm_bo_util.c
+++ b/drivers/gpu/drm/ttm/ttm_bo_util.c
@@ -181,13 +181,15 @@ static int ttm_copy_io_ttm_page(struct t
return -ENOMEM;
src = (void *)((unsigned long)src + (page << PAGE_SHIFT));
- dst = kmap_atomic_prot(d, prot);
- if (!dst)
- return -ENOMEM;
+ /*
+ * Ensure that a highmem page is mapped with the correct
+ * pgprot. For non highmem the mapping is already there.
+ */
+ dst = kmap_local_page_prot(d, prot);
memcpy_fromio(dst, src, PAGE_SIZE);
- kunmap_atomic(dst);
+ kunmap_local(dst);
return 0;
}
@@ -203,13 +205,15 @@ static int ttm_copy_ttm_io_page(struct t
return -ENOMEM;
dst = (void *)((unsigned long)dst + (page << PAGE_SHIFT));
- src = kmap_atomic_prot(s, prot);
- if (!src)
- return -ENOMEM;
+ /*
+ * Ensure that a highmem page is mapped with the correct
+ * pgprot. For non highmem the mapping is already there.
+ */
+ src = kmap_local_page_prot(s, prot);
memcpy_toio(dst, src, PAGE_SIZE);
- kunmap_atomic(src);
+ kunmap_local(src);
return 0;
}
^ permalink raw reply
* [patch V3 30/37] highmem: Remove kmap_atomic_pfn()
From: Thomas Gleixner @ 2020-11-03 9:27 UTC (permalink / raw)
To: LKML
Cc: Juri Lelli, linux-aio, Peter Zijlstra, Sebastian Andrzej Siewior,
Joonas Lahtinen, dri-devel, virtualization, Ben Segall,
Chris Mason, Huang Rui, Paul Mackerras, Gerd Hoffmann,
Daniel Bristot de Oliveira, sparclinux, Vincent Chen,
Christoph Hellwig, Vincent Guittot, Paul McKenney, Max Filippov,
x86, Russell King, linux-csky, Ingo Molnar, David Airlie,
VMware Graphics, Mel Gorman, nouveau, Dave Airlie, linux-snps-arc,
Ben Skeggs, linux-xtensa, Arnd Bergmann, intel-gfx,
Roland Scheidegger, Josef Bacik, Steven Rostedt, Linus Torvalds,
Alexander Viro, spice-devel, David Sterba, Rodrigo Vivi,
Dietmar Eggemann, linux-arm-kernel, Jani Nikula, Chris Zankel,
Michal Simek, Thomas Bogendoerfer, Nick Hu, linux-mm,
Vineet Gupta, linux-mips, Christian Koenig, Benjamin LaHaise,
Daniel Vetter, linux-fsdevel, Andrew Morton, linuxppc-dev,
David S. Miller, linux-btrfs, Greentime Hu
In-Reply-To: <20201103092712.714480842@linutronix.de>
No more users.
Signed-off-by: Thomas Gleixner <tglx@linutronix.de>
---
V3: New patch
---
include/linux/highmem-internal.h | 12 ------------
1 file changed, 12 deletions(-)
--- a/include/linux/highmem-internal.h
+++ b/include/linux/highmem-internal.h
@@ -99,13 +99,6 @@ static inline void *kmap_atomic(struct p
return kmap_atomic_prot(page, kmap_prot);
}
-static inline void *kmap_atomic_pfn(unsigned long pfn)
-{
- preempt_disable();
- pagefault_disable();
- return __kmap_local_pfn_prot(pfn, kmap_prot);
-}
-
static inline void __kunmap_atomic(void *addr)
{
kunmap_local_indexed(addr);
@@ -193,11 +186,6 @@ static inline void *kmap_atomic_prot(str
return kmap_atomic(page);
}
-static inline void *kmap_atomic_pfn(unsigned long pfn)
-{
- return kmap_atomic(pfn_to_page(pfn));
-}
-
static inline void __kunmap_atomic(void *addr)
{
#ifdef ARCH_HAS_FLUSH_ON_KUNMAP
^ permalink raw reply
* [patch V3 29/37] ARM: mm: Replace kmap_atomic_pfn()
From: Thomas Gleixner @ 2020-11-03 9:27 UTC (permalink / raw)
To: LKML
Cc: Juri Lelli, linux-aio, Peter Zijlstra, Sebastian Andrzej Siewior,
Joonas Lahtinen, dri-devel, virtualization, Ben Segall,
Chris Mason, Huang Rui, Paul Mackerras, Gerd Hoffmann,
Daniel Bristot de Oliveira, sparclinux, Vincent Chen,
Christoph Hellwig, Vincent Guittot, Paul McKenney, Max Filippov,
x86, Russell King, linux-csky, Ingo Molnar, David Airlie,
VMware Graphics, Mel Gorman, nouveau, Dave Airlie, linux-snps-arc,
Ben Skeggs, linux-xtensa, Arnd Bergmann, intel-gfx,
Roland Scheidegger, Josef Bacik, Steven Rostedt, Linus Torvalds,
Alexander Viro, spice-devel, David Sterba, Rodrigo Vivi,
Dietmar Eggemann, linux-arm-kernel, Jani Nikula, Chris Zankel,
Michal Simek, Thomas Bogendoerfer, Nick Hu, linux-mm,
Vineet Gupta, linux-mips, Christian Koenig, Benjamin LaHaise,
Daniel Vetter, linux-fsdevel, Andrew Morton, linuxppc-dev,
David S. Miller, linux-btrfs, Greentime Hu
In-Reply-To: <20201103092712.714480842@linutronix.de>
There is no requirement to disable pagefaults and preemption for these
cache management mappings.
Replace kmap_atomic_pfn() with kmap_local_pfn(). This allows to remove
kmap_atomic_pfn() in the next step.
Signed-off-by: Thomas Gleixner <tglx@linutronix.de>
Cc: Russell King <linux@armlinux.org.uk>
Cc: linux-arm-kernel@lists.infradead.org
---
V3: New patch
---
arch/arm/mm/cache-feroceon-l2.c | 6 +++---
arch/arm/mm/cache-xsc3l2.c | 4 ++--
2 files changed, 5 insertions(+), 5 deletions(-)
--- a/arch/arm/mm/cache-feroceon-l2.c
+++ b/arch/arm/mm/cache-feroceon-l2.c
@@ -49,9 +49,9 @@ static inline unsigned long l2_get_va(un
* we simply install a virtual mapping for it only for the
* TLB lookup to occur, hence no need to flush the untouched
* memory mapping afterwards (note: a cache flush may happen
- * in some circumstances depending on the path taken in kunmap_atomic).
+ * in some circumstances depending on the path taken in kunmap_local).
*/
- void *vaddr = kmap_atomic_pfn(paddr >> PAGE_SHIFT);
+ void *vaddr = kmap_local_pfn(paddr >> PAGE_SHIFT);
return (unsigned long)vaddr + (paddr & ~PAGE_MASK);
#else
return __phys_to_virt(paddr);
@@ -61,7 +61,7 @@ static inline unsigned long l2_get_va(un
static inline void l2_put_va(unsigned long vaddr)
{
#ifdef CONFIG_HIGHMEM
- kunmap_atomic((void *)vaddr);
+ kunmap_local((void *)vaddr);
#endif
}
--- a/arch/arm/mm/cache-xsc3l2.c
+++ b/arch/arm/mm/cache-xsc3l2.c
@@ -59,7 +59,7 @@ static inline void l2_unmap_va(unsigned
{
#ifdef CONFIG_HIGHMEM
if (va != -1)
- kunmap_atomic((void *)va);
+ kunmap_local((void *)va);
#endif
}
@@ -75,7 +75,7 @@ static inline unsigned long l2_map_va(un
* in place for it.
*/
l2_unmap_va(prev_va);
- va = (unsigned long)kmap_atomic_pfn(pa >> PAGE_SHIFT);
+ va = (unsigned long)kmap_local_pfn(pa >> PAGE_SHIFT);
}
return va + (pa_offset >> (32 - PAGE_SHIFT));
#else
^ permalink raw reply
* [patch V3 28/37] mips/crashdump: Simplify copy_oldmem_page()
From: Thomas Gleixner @ 2020-11-03 9:27 UTC (permalink / raw)
To: LKML
Cc: Juri Lelli, linux-aio, Peter Zijlstra, Sebastian Andrzej Siewior,
Joonas Lahtinen, dri-devel, virtualization, Ben Segall,
Chris Mason, Huang Rui, Paul Mackerras, Gerd Hoffmann,
Daniel Bristot de Oliveira, sparclinux, Vincent Chen,
Christoph Hellwig, Vincent Guittot, Paul McKenney, Max Filippov,
x86, Russell King, linux-csky, Ingo Molnar, David Airlie,
VMware Graphics, Mel Gorman, nouveau, Dave Airlie, linux-snps-arc,
Ben Skeggs, linux-xtensa, Arnd Bergmann, intel-gfx,
Roland Scheidegger, Josef Bacik, Steven Rostedt, Linus Torvalds,
Alexander Viro, spice-devel, David Sterba, Rodrigo Vivi,
Dietmar Eggemann, linux-arm-kernel, Jani Nikula, Chris Zankel,
Michal Simek, Thomas Bogendoerfer, Nick Hu, linux-mm,
Vineet Gupta, linux-mips, Christian Koenig, Benjamin LaHaise,
Daniel Vetter, linux-fsdevel, Andrew Morton, linuxppc-dev,
David S. Miller, linux-btrfs, Greentime Hu
In-Reply-To: <20201103092712.714480842@linutronix.de>
Replace kmap_atomic_pfn() with kmap_local_pfn() which is preemptible and
can take page faults.
Remove the indirection of the dump page and the related cruft which is not
longer required.
Signed-off-by: Thomas Gleixner <tglx@linutronix.de>
Cc: Thomas Bogendoerfer <tsbogend@alpha.franken.de>
Cc: linux-mips@vger.kernel.org
---
V3: New patch
---
arch/mips/kernel/crash_dump.c | 42 +++++++-----------------------------------
1 file changed, 7 insertions(+), 35 deletions(-)
--- a/arch/mips/kernel/crash_dump.c
+++ b/arch/mips/kernel/crash_dump.c
@@ -5,8 +5,6 @@
#include <linux/uaccess.h>
#include <linux/slab.h>
-static void *kdump_buf_page;
-
/**
* copy_oldmem_page - copy one page from "oldmem"
* @pfn: page frame number to be copied
@@ -17,51 +15,25 @@ static void *kdump_buf_page;
* @userbuf: if set, @buf is in user address space, use copy_to_user(),
* otherwise @buf is in kernel address space, use memcpy().
*
- * Copy a page from "oldmem". For this page, there is no pte mapped
+ * Copy a page from "oldmem". For this page, there might be no pte mapped
* in the current kernel.
- *
- * Calling copy_to_user() in atomic context is not desirable. Hence first
- * copying the data to a pre-allocated kernel page and then copying to user
- * space in non-atomic context.
*/
-ssize_t copy_oldmem_page(unsigned long pfn, char *buf,
- size_t csize, unsigned long offset, int userbuf)
+ssize_t copy_oldmem_page(unsigned long pfn, char *buf, size_t csize,
+ unsigned long offset, int userbuf)
{
void *vaddr;
if (!csize)
return 0;
- vaddr = kmap_atomic_pfn(pfn);
+ vaddr = kmap_local_pfn(pfn);
if (!userbuf) {
- memcpy(buf, (vaddr + offset), csize);
- kunmap_atomic(vaddr);
+ memcpy(buf, vaddr + offset, csize);
} else {
- if (!kdump_buf_page) {
- pr_warn("Kdump: Kdump buffer page not allocated\n");
-
- return -EFAULT;
- }
- copy_page(kdump_buf_page, vaddr);
- kunmap_atomic(vaddr);
- if (copy_to_user(buf, (kdump_buf_page + offset), csize))
- return -EFAULT;
+ if (copy_to_user(buf, vaddr + offset, csize))
+ csize = -EFAULT;
}
return csize;
}
-
-static int __init kdump_buf_page_init(void)
-{
- int ret = 0;
-
- kdump_buf_page = kmalloc(PAGE_SIZE, GFP_KERNEL);
- if (!kdump_buf_page) {
- pr_warn("Kdump: Failed to allocate kdump buffer page\n");
- ret = -ENOMEM;
- }
-
- return ret;
-}
-arch_initcall(kdump_buf_page_init);
^ permalink raw reply
* [patch V3 26/37] io-mapping: Provide iomap_local variant
From: Thomas Gleixner @ 2020-11-03 9:27 UTC (permalink / raw)
To: LKML
Cc: Juri Lelli, linux-aio, Peter Zijlstra, Sebastian Andrzej Siewior,
Joonas Lahtinen, dri-devel, virtualization, Ben Segall,
Chris Mason, Huang Rui, Paul Mackerras, Gerd Hoffmann,
Daniel Bristot de Oliveira, sparclinux, Vincent Chen,
Christoph Hellwig, Vincent Guittot, Paul McKenney, Max Filippov,
x86, Russell King, linux-csky, Ingo Molnar, David Airlie,
VMware Graphics, Mel Gorman, nouveau, Dave Airlie, linux-snps-arc,
Ben Skeggs, linux-xtensa, Arnd Bergmann, intel-gfx,
Roland Scheidegger, Josef Bacik, Steven Rostedt, Linus Torvalds,
Alexander Viro, spice-devel, David Sterba, Rodrigo Vivi,
Dietmar Eggemann, linux-arm-kernel, Jani Nikula, Chris Zankel,
Michal Simek, Thomas Bogendoerfer, Nick Hu, linux-mm,
Vineet Gupta, linux-mips, Christian Koenig, Benjamin LaHaise,
Daniel Vetter, linux-fsdevel, Andrew Morton, linuxppc-dev,
David S. Miller, linux-btrfs, Greentime Hu
In-Reply-To: <20201103092712.714480842@linutronix.de>
Similar to kmap local provide a iomap local variant which only disables
migration, but neither disables pagefaults nor preemption.
Signed-off-by: Thomas Gleixner <tglx@linutronix.de>
---
V3: Restrict migrate disable to the 32bit mapping case and update documentation.
V2: Split out from the large combo patch and add the !IOMAP_ATOMIC variants
---
Documentation/driver-api/io-mapping.rst | 76 +++++++++++++++++++-------------
include/linux/io-mapping.h | 30 +++++++++++-
2 files changed, 74 insertions(+), 32 deletions(-)
--- a/Documentation/driver-api/io-mapping.rst
+++ b/Documentation/driver-api/io-mapping.rst
@@ -20,55 +20,71 @@ as it would consume too much of the kern
mappable, while 'size' indicates how large a mapping region to
enable. Both are in bytes.
-This _wc variant provides a mapping which may only be used
-with the io_mapping_map_atomic_wc or io_mapping_map_wc.
+This _wc variant provides a mapping which may only be used with
+io_mapping_map_atomic_wc(), io_mapping_map_local_wc() or
+io_mapping_map_wc().
+
+With this mapping object, individual pages can be mapped either temporarily
+or long term, depending on the requirements. Of course, temporary maps are
+more efficient. They come in two flavours::
-With this mapping object, individual pages can be mapped either atomically
-or not, depending on the necessary scheduling environment. Of course, atomic
-maps are more efficient::
+ void *io_mapping_map_local_wc(struct io_mapping *mapping,
+ unsigned long offset)
void *io_mapping_map_atomic_wc(struct io_mapping *mapping,
unsigned long offset)
-'offset' is the offset within the defined mapping region.
-Accessing addresses beyond the region specified in the
-creation function yields undefined results. Using an offset
-which is not page aligned yields an undefined result. The
-return value points to a single page in CPU address space.
-
-This _wc variant returns a write-combining map to the
-page and may only be used with mappings created by
-io_mapping_create_wc
+'offset' is the offset within the defined mapping region. Accessing
+addresses beyond the region specified in the creation function yields
+undefined results. Using an offset which is not page aligned yields an
+undefined result. The return value points to a single page in CPU address
+space.
-Note that the task may not sleep while holding this page
-mapped.
+This _wc variant returns a write-combining map to the page and may only be
+used with mappings created by io_mapping_create_wc()
-::
+Temporary mappings are only valid in the context of the caller. The mapping
+is not guaranteed to be globaly visible.
- void io_mapping_unmap_atomic(void *vaddr)
+io_mapping_map_local_wc() has a side effect on X86 32bit as it disables
+migration to make the mapping code work. No caller can rely on this side
+effect.
+
+io_mapping_map_atomic_wc() has the side effect of disabling preemption and
+pagefaults. Don't use in new code. Use io_mapping_map_local_wc() instead.
-'vaddr' must be the value returned by the last
-io_mapping_map_atomic_wc call. This unmaps the specified
-page and allows the task to sleep once again.
+Nested mappings need to be undone in reverse order because the mapping
+code uses a stack for keeping track of them::
-If you need to sleep while holding the lock, you can use the non-atomic
-variant, although they may be significantly slower.
+ addr1 = io_mapping_map_local_wc(map1, offset1);
+ addr2 = io_mapping_map_local_wc(map2, offset2);
+ ...
+ io_mapping_unmap_local(addr2);
+ io_mapping_unmap_local(addr1);
-::
+The mappings are released with::
+
+ void io_mapping_unmap_local(void *vaddr)
+ void io_mapping_unmap_atomic(void *vaddr)
+
+'vaddr' must be the value returned by the last io_mapping_map_local_wc() or
+io_mapping_map_atomic_wc() call. This unmaps the specified mapping and
+undoes the side effects of the mapping functions.
+
+If you need to sleep while holding a mapping, you can use the regular
+variant, although this may be significantly slower::
void *io_mapping_map_wc(struct io_mapping *mapping,
unsigned long offset)
-This works like io_mapping_map_atomic_wc except it allows
-the task to sleep while holding the page mapped.
-
+This works like io_mapping_map_atomic/local_wc() except it has no side
+effects and the pointer is globaly visible.
-::
+The mappings are released with::
void io_mapping_unmap(void *vaddr)
-This works like io_mapping_unmap_atomic, except it is used
-for pages mapped with io_mapping_map_wc.
+Use for pages mapped with io_mapping_map_wc().
At driver close time, the io_mapping object must be freed::
--- a/include/linux/io-mapping.h
+++ b/include/linux/io-mapping.h
@@ -83,6 +83,21 @@ io_mapping_unmap_atomic(void __iomem *va
}
static inline void __iomem *
+io_mapping_map_local_wc(struct io_mapping *mapping, unsigned long offset)
+{
+ resource_size_t phys_addr;
+
+ BUG_ON(offset >= mapping->size);
+ phys_addr = mapping->base + offset;
+ return __iomap_local_pfn_prot(PHYS_PFN(phys_addr), mapping->prot);
+}
+
+static inline void io_mapping_unmap_local(void __iomem *vaddr)
+{
+ kunmap_local_indexed((void __force *)vaddr);
+}
+
+static inline void __iomem *
io_mapping_map_wc(struct io_mapping *mapping,
unsigned long offset,
unsigned long size)
@@ -101,7 +116,7 @@ io_mapping_unmap(void __iomem *vaddr)
iounmap(vaddr);
}
-#else
+#else /* HAVE_ATOMIC_IOMAP */
#include <linux/uaccess.h>
@@ -166,7 +181,18 @@ io_mapping_unmap_atomic(void __iomem *va
preempt_enable();
}
-#endif /* HAVE_ATOMIC_IOMAP */
+static inline void __iomem *
+io_mapping_map_local_wc(struct io_mapping *mapping, unsigned long offset)
+{
+ return io_mapping_map_wc(mapping, offset, PAGE_SIZE);
+}
+
+static inline void io_mapping_unmap_local(void __iomem *vaddr)
+{
+ io_mapping_unmap(vaddr);
+}
+
+#endif /* !HAVE_ATOMIC_IOMAP */
static inline struct io_mapping *
io_mapping_create_wc(resource_size_t base,
^ permalink raw reply
* [patch V3 25/37] mm/highmem: Provide kmap_local*
From: Thomas Gleixner @ 2020-11-03 9:27 UTC (permalink / raw)
To: LKML
Cc: Juri Lelli, linux-aio, Peter Zijlstra, Sebastian Andrzej Siewior,
Joonas Lahtinen, dri-devel, virtualization, Ben Segall,
Chris Mason, Huang Rui, Paul Mackerras, Gerd Hoffmann,
Daniel Bristot de Oliveira, sparclinux, Vincent Chen,
Christoph Hellwig, Vincent Guittot, Paul McKenney, Max Filippov,
x86, Russell King, linux-csky, Ingo Molnar, David Airlie,
VMware Graphics, Mel Gorman, nouveau, Dave Airlie, linux-snps-arc,
Ben Skeggs, linux-xtensa, Arnd Bergmann, intel-gfx,
Roland Scheidegger, Josef Bacik, Steven Rostedt, Linus Torvalds,
Alexander Viro, spice-devel, David Sterba, Rodrigo Vivi,
Dietmar Eggemann, linux-arm-kernel, Jani Nikula, Chris Zankel,
Michal Simek, Thomas Bogendoerfer, Nick Hu, linux-mm,
Vineet Gupta, linux-mips, Christian Koenig, Benjamin LaHaise,
Daniel Vetter, linux-fsdevel, Andrew Morton, linuxppc-dev,
David S. Miller, linux-btrfs, Greentime Hu
In-Reply-To: <20201103092712.714480842@linutronix.de>
Now that the kmap atomic index is stored in task struct provide a
preemptible variant. On context switch the maps of an outgoing task are
removed and the map of the incoming task are restored. That's obviously
slow, but highmem is slow anyway.
The kmap_local.*() functions can be invoked from both preemptible and
atomic context. kmap local sections disable migration to keep the resulting
virtual mapping address correct, but disable neither pagefaults nor
preemption.
A wholesale conversion of kmap_atomic to be fully preemptible is not
possible because some of the usage sites might rely on the preemption
disable for serialization or on the implicit pagefault disable. Needs to be
done on a case by case basis.
Signed-off-by: Thomas Gleixner <tglx@linutronix.de>
---
V3: Move migrate disable into the actual highmem mapping code so it only
affects real highmem mappings.
V2: Make it more consistent and add commentry
---
include/linux/highmem-internal.h | 48 +++++++++++++++++++++++++++++++++++++++
include/linux/highmem.h | 43 +++++++++++++++++++++-------------
mm/highmem.c | 6 ++++
3 files changed, 81 insertions(+), 16 deletions(-)
--- a/include/linux/highmem-internal.h
+++ b/include/linux/highmem-internal.h
@@ -69,6 +69,26 @@ static inline void kmap_flush_unused(voi
__kmap_flush_unused();
}
+static inline void *kmap_local_page(struct page *page)
+{
+ return __kmap_local_page_prot(page, kmap_prot);
+}
+
+static inline void *kmap_local_page_prot(struct page *page, pgprot_t prot)
+{
+ return __kmap_local_page_prot(page, prot);
+}
+
+static inline void *kmap_local_pfn(unsigned long pfn)
+{
+ return __kmap_local_pfn_prot(pfn, kmap_prot);
+}
+
+static inline void __kunmap_local(void *vaddr)
+{
+ kunmap_local_indexed(vaddr);
+}
+
static inline void *kmap_atomic_prot(struct page *page, pgprot_t prot)
{
preempt_disable();
@@ -141,6 +161,28 @@ static inline void kunmap(struct page *p
#endif
}
+static inline void *kmap_local_page(struct page *page)
+{
+ return page_address(page);
+}
+
+static inline void *kmap_local_page_prot(struct page *page, pgprot_t prot)
+{
+ return kmap_local_page(page);
+}
+
+static inline void *kmap_local_pfn(unsigned long pfn)
+{
+ return kmap_local_page(pfn_to_page(pfn));
+}
+
+static inline void __kunmap_local(void *addr)
+{
+#ifdef ARCH_HAS_FLUSH_ON_KUNMAP
+ kunmap_flush_on_unmap(addr);
+#endif
+}
+
static inline void *kmap_atomic(struct page *page)
{
preempt_disable();
@@ -182,4 +224,10 @@ do { \
__kunmap_atomic(__addr); \
} while (0)
+#define kunmap_local(__addr) \
+do { \
+ BUILD_BUG_ON(__same_type((__addr), struct page *)); \
+ __kunmap_local(__addr); \
+} while (0)
+
#endif
--- a/include/linux/highmem.h
+++ b/include/linux/highmem.h
@@ -60,24 +60,22 @@ static inline struct page *kmap_to_page(
static inline void kmap_flush_unused(void);
/**
- * kmap_atomic - Atomically map a page for temporary usage
+ * kmap_local_page - Map a page for temporary usage
* @page: Pointer to the page to be mapped
*
* Returns: The virtual address of the mapping
*
- * Side effect: On return pagefaults and preemption are disabled.
- *
* Can be invoked from any context.
*
* Requires careful handling when nesting multiple mappings because the map
* management is stack based. The unmap has to be in the reverse order of
* the map operation:
*
- * addr1 = kmap_atomic(page1);
- * addr2 = kmap_atomic(page2);
+ * addr1 = kmap_local_page(page1);
+ * addr2 = kmap_local_page(page2);
* ...
- * kunmap_atomic(addr2);
- * kunmap_atomic(addr1);
+ * kunmap_local(addr2);
+ * kunmap_local(addr1);
*
* Unmapping addr1 before addr2 is invalid and causes malfunction.
*
@@ -88,10 +86,26 @@ static inline void kmap_flush_unused(voi
* virtual address of the direct mapping. Only real highmem pages are
* temporarily mapped.
*
- * While it is significantly faster than kmap() it comes with restrictions
- * about the pointer validity and the side effects of disabling page faults
- * and preemption. Use it only when absolutely necessary, e.g. from non
- * preemptible contexts.
+ * While it is significantly faster than kmap() for the higmem case it
+ * comes with restrictions about the pointer validity. Only use when really
+ * necessary.
+ *
+ * On HIGHMEM enabled systems mapping a highmem page has the side effect of
+ * disabling migration in order to keep the virtual address stable across
+ * preemption. No caller of kmap_local_page() can rely on this side effect.
+ */
+static inline void *kmap_local_page(struct page *page);
+
+/**
+ * kmap_atomic - Atomically map a page for temporary usage - Deprecated!
+ * @page: Pointer to the page to be mapped
+ *
+ * Returns: The virtual address of the mapping
+ *
+ * Effectively a wrapper around kmap_local_page() which disables pagefaults
+ * and preemption.
+ *
+ * Do not use in new code. Use kmap_local_page() instead.
*/
static inline void *kmap_atomic(struct page *page);
@@ -101,12 +115,9 @@ static inline void *kmap_atomic(struct p
*
* Counterpart to kmap_atomic().
*
- * Undoes the side effects of kmap_atomic(), i.e. reenabling pagefaults and
+ * Effectively a wrapper around kunmap_local() which additionally undoes
+ * the side effects of kmap_atomic(), i.e. reenabling pagefaults and
* preemption.
- *
- * Other than that a NOOP for CONFIG_HIGHMEM=n and for mappings of pages
- * in the low memory area. For real highmen pages the mapping which was
- * established with kmap_atomic() is destroyed.
*/
/* Highmem related interfaces for management code */
--- a/mm/highmem.c
+++ b/mm/highmem.c
@@ -450,6 +450,11 @@ void *__kmap_local_pfn_prot(unsigned lon
unsigned long vaddr;
int idx;
+ /*
+ * Disable migration so resulting virtual address is stable
+ * accross preemption.
+ */
+ migrate_disable();
preempt_disable();
idx = arch_kmap_local_map_idx(kmap_local_idx_push(), pfn);
vaddr = __fix_to_virt(FIX_KMAP_BEGIN + idx);
@@ -505,6 +510,7 @@ void kunmap_local_indexed(void *vaddr)
current->kmap_ctrl.pteval[kmap_local_idx()] = __pte(0);
kmap_local_idx_pop();
preempt_enable();
+ migrate_enable();
}
EXPORT_SYMBOL(kunmap_local_indexed);
^ permalink raw reply
* [patch V3 27/37] x86/crashdump/32: Simplify copy_oldmem_page()
From: Thomas Gleixner @ 2020-11-03 9:27 UTC (permalink / raw)
To: LKML
Cc: Juri Lelli, linux-aio, Peter Zijlstra, Sebastian Andrzej Siewior,
Joonas Lahtinen, dri-devel, virtualization, Ben Segall,
Chris Mason, Huang Rui, Paul Mackerras, Gerd Hoffmann,
Daniel Bristot de Oliveira, sparclinux, Vincent Chen,
Christoph Hellwig, Vincent Guittot, Paul McKenney, Max Filippov,
x86, Russell King, linux-csky, Ingo Molnar, David Airlie,
VMware Graphics, Mel Gorman, nouveau, Dave Airlie, linux-snps-arc,
Ben Skeggs, linux-xtensa, Arnd Bergmann, intel-gfx,
Roland Scheidegger, Josef Bacik, Steven Rostedt, Linus Torvalds,
Alexander Viro, spice-devel, David Sterba, Rodrigo Vivi,
Dietmar Eggemann, linux-arm-kernel, Jani Nikula, Chris Zankel,
Michal Simek, Thomas Bogendoerfer, Nick Hu, linux-mm,
Vineet Gupta, linux-mips, Christian Koenig, Benjamin LaHaise,
Daniel Vetter, linux-fsdevel, Andrew Morton, linuxppc-dev,
David S. Miller, linux-btrfs, Greentime Hu
In-Reply-To: <20201103092712.714480842@linutronix.de>
Replace kmap_atomic_pfn() with kmap_local_pfn() which is preemptible and
can take page faults.
Remove the indirection of the dump page and the related cruft which is not
longer required.
Signed-off-by: Thomas Gleixner <tglx@linutronix.de>
---
V3: New patch
---
arch/x86/kernel/crash_dump_32.c | 48 ++++++++--------------------------------
1 file changed, 10 insertions(+), 38 deletions(-)
--- a/arch/x86/kernel/crash_dump_32.c
+++ b/arch/x86/kernel/crash_dump_32.c
@@ -13,8 +13,6 @@
#include <linux/uaccess.h>
-static void *kdump_buf_page;
-
static inline bool is_crashed_pfn_valid(unsigned long pfn)
{
#ifndef CONFIG_X86_PAE
@@ -41,15 +39,11 @@ static inline bool is_crashed_pfn_valid(
* @userbuf: if set, @buf is in user address space, use copy_to_user(),
* otherwise @buf is in kernel address space, use memcpy().
*
- * Copy a page from "oldmem". For this page, there is no pte mapped
- * in the current kernel. We stitch up a pte, similar to kmap_atomic.
- *
- * Calling copy_to_user() in atomic context is not desirable. Hence first
- * copying the data to a pre-allocated kernel page and then copying to user
- * space in non-atomic context.
+ * Copy a page from "oldmem". For this page, there might be no pte mapped
+ * in the current kernel.
*/
-ssize_t copy_oldmem_page(unsigned long pfn, char *buf,
- size_t csize, unsigned long offset, int userbuf)
+ssize_t copy_oldmem_page(unsigned long pfn, char *buf, size_t csize,
+ unsigned long offset, int userbuf)
{
void *vaddr;
@@ -59,38 +53,16 @@ ssize_t copy_oldmem_page(unsigned long p
if (!is_crashed_pfn_valid(pfn))
return -EFAULT;
- vaddr = kmap_atomic_pfn(pfn);
+ vaddr = kmap_local_pfn(pfn);
if (!userbuf) {
- memcpy(buf, (vaddr + offset), csize);
- kunmap_atomic(vaddr);
+ memcpy(buf, vaddr + offset, csize);
} else {
- if (!kdump_buf_page) {
- printk(KERN_WARNING "Kdump: Kdump buffer page not"
- " allocated\n");
- kunmap_atomic(vaddr);
- return -EFAULT;
- }
- copy_page(kdump_buf_page, vaddr);
- kunmap_atomic(vaddr);
- if (copy_to_user(buf, (kdump_buf_page + offset), csize))
- return -EFAULT;
+ if (copy_to_user(buf, vaddr + offset, csize))
+ csize = -EFAULT;
}
- return csize;
-}
+ kunmap_local(vaddr);
-static int __init kdump_buf_page_init(void)
-{
- int ret = 0;
-
- kdump_buf_page = kmalloc(PAGE_SIZE, GFP_KERNEL);
- if (!kdump_buf_page) {
- printk(KERN_WARNING "Kdump: Failed to allocate kdump buffer"
- " page\n");
- ret = -ENOMEM;
- }
-
- return ret;
+ return csize;
}
-arch_initcall(kdump_buf_page_init);
^ permalink raw reply
* [patch V3 24/37] sched: highmem: Store local kmaps in task struct
From: Thomas Gleixner @ 2020-11-03 9:27 UTC (permalink / raw)
To: LKML
Cc: Juri Lelli, linux-aio, Peter Zijlstra, Sebastian Andrzej Siewior,
Joonas Lahtinen, dri-devel, virtualization, Ben Segall,
Chris Mason, Huang Rui, Paul Mackerras, Gerd Hoffmann,
Daniel Bristot de Oliveira, sparclinux, Vincent Chen,
Christoph Hellwig, Vincent Guittot, Paul McKenney, Max Filippov,
x86, Russell King, linux-csky, Ingo Molnar, David Airlie,
VMware Graphics, Mel Gorman, nouveau, Dave Airlie, linux-snps-arc,
Ben Skeggs, linux-xtensa, Arnd Bergmann, intel-gfx,
Roland Scheidegger, Josef Bacik, Steven Rostedt, Linus Torvalds,
Alexander Viro, spice-devel, David Sterba, Rodrigo Vivi,
Dietmar Eggemann, linux-arm-kernel, Jani Nikula, Chris Zankel,
Michal Simek, Thomas Bogendoerfer, Nick Hu, linux-mm,
Vineet Gupta, linux-mips, Christian Koenig, Benjamin LaHaise,
Daniel Vetter, linux-fsdevel, Andrew Morton, linuxppc-dev,
David S. Miller, linux-btrfs, Greentime Hu
In-Reply-To: <20201103092712.714480842@linutronix.de>
Instead of storing the map per CPU provide and use per task storage. That
prepares for local kmaps which are preemptible.
The context switch code is preparatory and not yet in use because
kmap_atomic() runs with preemption disabled. Will be made usable in the
next step.
The context switch logic is safe even when an interrupt happens after
clearing or before restoring the kmaps. The kmap index in task struct is
not modified so any nesting kmap in an interrupt will use unused indices
and on return the counter is the same as before.
Also add an assert into the return to user space code. Going back to user
space with an active kmap local is a nono.
Signed-off-by: Thomas Gleixner <tglx@linutronix.de>
---
V3: Handle the debug case correctly
---
include/linux/highmem-internal.h | 10 +++
include/linux/sched.h | 9 +++
kernel/entry/common.c | 2
kernel/fork.c | 1
kernel/sched/core.c | 18 +++++++
mm/highmem.c | 99 +++++++++++++++++++++++++++++++++++----
6 files changed, 129 insertions(+), 10 deletions(-)
--- a/include/linux/highmem-internal.h
+++ b/include/linux/highmem-internal.h
@@ -9,6 +9,16 @@
void *__kmap_local_pfn_prot(unsigned long pfn, pgprot_t prot);
void *__kmap_local_page_prot(struct page *page, pgprot_t prot);
void kunmap_local_indexed(void *vaddr);
+void kmap_local_fork(struct task_struct *tsk);
+void __kmap_local_sched_out(void);
+void __kmap_local_sched_in(void);
+static inline void kmap_assert_nomap(void)
+{
+ DEBUG_LOCKS_WARN_ON(current->kmap_ctrl.idx);
+}
+#else
+static inline void kmap_local_fork(struct task_struct *tsk) { }
+static inline void kmap_assert_nomap(void) { }
#endif
#ifdef CONFIG_HIGHMEM
--- a/include/linux/sched.h
+++ b/include/linux/sched.h
@@ -34,6 +34,7 @@
#include <linux/rseq.h>
#include <linux/seqlock.h>
#include <linux/kcsan.h>
+#include <asm/kmap_size.h>
/* task_struct member predeclarations (sorted alphabetically): */
struct audit_context;
@@ -629,6 +630,13 @@ struct wake_q_node {
struct wake_q_node *next;
};
+struct kmap_ctrl {
+#ifdef CONFIG_KMAP_LOCAL
+ int idx;
+ pte_t pteval[KM_TYPE_NR];
+#endif
+};
+
struct task_struct {
#ifdef CONFIG_THREAD_INFO_IN_TASK
/*
@@ -1294,6 +1302,7 @@ struct task_struct {
unsigned int sequential_io;
unsigned int sequential_io_avg;
#endif
+ struct kmap_ctrl kmap_ctrl;
#ifdef CONFIG_DEBUG_ATOMIC_SLEEP
unsigned long task_state_change;
#endif
--- a/kernel/entry/common.c
+++ b/kernel/entry/common.c
@@ -2,6 +2,7 @@
#include <linux/context_tracking.h>
#include <linux/entry-common.h>
+#include <linux/highmem.h>
#include <linux/livepatch.h>
#include <linux/audit.h>
@@ -194,6 +195,7 @@ static void exit_to_user_mode_prepare(st
/* Ensure that the address limit is intact and no locks are held */
addr_limit_user_check();
+ kmap_assert_nomap();
lockdep_assert_irqs_disabled();
lockdep_sys_exit();
}
--- a/kernel/fork.c
+++ b/kernel/fork.c
@@ -930,6 +930,7 @@ static struct task_struct *dup_task_stru
account_kernel_stack(tsk, 1);
kcov_task_init(tsk);
+ kmap_local_fork(tsk);
#ifdef CONFIG_FAULT_INJECTION
tsk->fail_nth = 0;
--- a/kernel/sched/core.c
+++ b/kernel/sched/core.c
@@ -4053,6 +4053,22 @@ static inline void finish_lock_switch(st
# define finish_arch_post_lock_switch() do { } while (0)
#endif
+static inline void kmap_local_sched_out(void)
+{
+#ifdef CONFIG_KMAP_LOCAL
+ if (unlikely(current->kmap_ctrl.idx))
+ __kmap_local_sched_out();
+#endif
+}
+
+static inline void kmap_local_sched_in(void)
+{
+#ifdef CONFIG_KMAP_LOCAL
+ if (unlikely(current->kmap_ctrl.idx))
+ __kmap_local_sched_in();
+#endif
+}
+
/**
* prepare_task_switch - prepare to switch tasks
* @rq: the runqueue preparing to switch
@@ -4075,6 +4091,7 @@ prepare_task_switch(struct rq *rq, struc
perf_event_task_sched_out(prev, next);
rseq_preempt(prev);
fire_sched_out_preempt_notifiers(prev, next);
+ kmap_local_sched_out();
prepare_task(next);
prepare_arch_switch(next);
}
@@ -4141,6 +4158,7 @@ static struct rq *finish_task_switch(str
finish_lock_switch(rq);
finish_arch_post_lock_switch();
kcov_finish_switch(current);
+ kmap_local_sched_in();
fire_sched_in_preempt_notifiers(current);
/*
--- a/mm/highmem.c
+++ b/mm/highmem.c
@@ -365,8 +365,6 @@ EXPORT_SYMBOL(kunmap_high);
#include <asm/kmap_size.h>
-static DEFINE_PER_CPU(int, __kmap_local_idx);
-
/*
* With DEBUG_HIGHMEM the stack depth is doubled and every second
* slot is unused which acts as a guard page
@@ -379,23 +377,21 @@ static DEFINE_PER_CPU(int, __kmap_local_
static inline int kmap_local_idx_push(void)
{
- int idx = __this_cpu_add_return(__kmap_local_idx, KM_INCR) - 1;
-
WARN_ON_ONCE(in_irq() && !irqs_disabled());
- BUG_ON(idx >= KM_MAX_IDX);
- return idx;
+ current->kmap_ctrl.idx += KM_INCR;
+ BUG_ON(current->kmap_ctrl.idx >= KM_TYPE_NR);
+ return current->kmap_ctrl.idx - 1;
}
static inline int kmap_local_idx(void)
{
- return __this_cpu_read(__kmap_local_idx) - 1;
+ return current->kmap_ctrl.idx - 1;
}
static inline void kmap_local_idx_pop(void)
{
- int idx = __this_cpu_sub_return(__kmap_local_idx, KM_INCR);
-
- BUG_ON(idx < 0);
+ current->kmap_ctrl.idx -= KM_INCR;
+ BUG_ON(current->kmap_ctrl.idx < 0);
}
#ifndef arch_kmap_local_post_map
@@ -461,6 +457,7 @@ void *__kmap_local_pfn_prot(unsigned lon
pteval = pfn_pte(pfn, prot);
set_pte_at(&init_mm, vaddr, kmap_pte - idx, pteval);
arch_kmap_local_post_map(vaddr, pteval);
+ current->kmap_ctrl.pteval[kmap_local_idx()] = pteval;
preempt_enable();
return (void *)vaddr;
@@ -505,10 +502,92 @@ void kunmap_local_indexed(void *vaddr)
arch_kmap_local_pre_unmap(addr);
pte_clear(&init_mm, addr, kmap_pte - idx);
arch_kmap_local_post_unmap(addr);
+ current->kmap_ctrl.pteval[kmap_local_idx()] = __pte(0);
kmap_local_idx_pop();
preempt_enable();
}
EXPORT_SYMBOL(kunmap_local_indexed);
+
+/*
+ * Invoked before switch_to(). This is safe even when during or after
+ * clearing the maps an interrupt which needs a kmap_local happens because
+ * the task::kmap_ctrl.idx is not modified by the unmapping code so a
+ * nested kmap_local will use the next unused index and restore the index
+ * on unmap. The already cleared kmaps of the outgoing task are irrelevant
+ * because the interrupt context does not know about them. The same applies
+ * when scheduling back in for an interrupt which happens before the
+ * restore is complete.
+ */
+void __kmap_local_sched_out(void)
+{
+ struct task_struct *tsk = current;
+ pte_t *kmap_pte = kmap_get_pte();
+ int i;
+
+ /* Clear kmaps */
+ for (i = 0; i < tsk->kmap_ctrl.idx; i++) {
+ pte_t pteval = tsk->kmap_ctrl.pteval[i];
+ unsigned long addr;
+ int idx;
+
+ /* With debug all even slots are unmapped and act as guard */
+ if (IS_ENABLED(CONFIG_DEBUG_HIGHMEM) && !(i & 0x01)) {
+ WARN_ON_ONCE(!pte_none(pteval));
+ continue;
+ }
+ if (WARN_ON_ONCE(pte_none(pteval)))
+ continue;
+
+ /*
+ * This is a horrible hack for XTENSA to calculate the
+ * coloured PTE index. Uses the PFN encoded into the pteval
+ * and the map index calculation because the actual mapped
+ * virtual address is not stored in task::kmap_ctrl.
+ * For any sane architecture this is optimized out.
+ */
+ idx = arch_kmap_local_map_idx(i, pte_pfn(pteval));
+
+ addr = __fix_to_virt(FIX_KMAP_BEGIN + idx);
+ arch_kmap_local_pre_unmap(addr);
+ pte_clear(&init_mm, addr, kmap_pte - idx);
+ arch_kmap_local_post_unmap(addr);
+ }
+}
+
+void __kmap_local_sched_in(void)
+{
+ struct task_struct *tsk = current;
+ pte_t *kmap_pte = kmap_get_pte();
+ int i;
+
+ /* Restore kmaps */
+ for (i = 0; i < tsk->kmap_ctrl.idx; i++) {
+ pte_t pteval = tsk->kmap_ctrl.pteval[i];
+ unsigned long addr;
+ int idx;
+
+ /* With debug all even slots are unmapped and act as guard */
+ if (IS_ENABLED(CONFIG_DEBUG_HIGHMEM) && !(i & 0x01)) {
+ WARN_ON_ONCE(!pte_none(pteval));
+ continue;
+ }
+ if (WARN_ON_ONCE(pte_none(pteval)))
+ continue;
+
+ /* See comment in __kmap_local_sched_out() */
+ idx = arch_kmap_local_map_idx(i, pte_pfn(pteval));
+ addr = __fix_to_virt(FIX_KMAP_BEGIN + idx);
+ set_pte_at(&init_mm, addr, kmap_pte - idx, pteval);
+ arch_kmap_local_post_map(addr, pteval);
+ }
+}
+
+void kmap_local_fork(struct task_struct *tsk)
+{
+ if (WARN_ON_ONCE(tsk->kmap_ctrl.idx))
+ memset(&tsk->kmap_ctrl, 0, sizeof(tsk->kmap_ctrl));
+}
+
#endif
#if defined(HASHED_PAGE_VIRTUAL)
^ permalink raw reply
* [patch V3 23/37] sched: Make migrate_disable/enable() independent of RT
From: Thomas Gleixner @ 2020-11-03 9:27 UTC (permalink / raw)
To: LKML
Cc: Juri Lelli, linux-aio, Peter Zijlstra, Sebastian Andrzej Siewior,
Joonas Lahtinen, dri-devel, virtualization, Ben Segall,
Chris Mason, Huang Rui, Paul Mackerras, Gerd Hoffmann, sparclinux,
Vincent Chen, Ingo Molnar, Vincent Guittot, Arnd Bergmann,
Max Filippov, x86, Russell King, linux-csky, Christoph Hellwig,
David Airlie, VMware Graphics, Mel Gorman, nouveau, Dave Airlie,
linux-snps-arc, Ben Skeggs, linux-xtensa, Paul McKenney,
intel-gfx, Roland Scheidegger, Josef Bacik, Steven Rostedt,
Linus Torvalds, Alexander Viro, spice-devel, David Sterba,
Rodrigo Vivi, Dietmar Eggemann, linux-arm-kernel, Jani Nikula,
Chris Zankel, Michal Simek, Thomas Bogendoerfer, Nick Hu,
linux-mm, Vineet Gupta, linuxppc-dev, linux-mips,
Christian Koenig, Benjamin LaHaise, Daniel Vetter, linux-fsdevel,
Andrew Morton, Daniel Bristot de Oliveira, David S. Miller,
linux-btrfs, Greentime Hu
In-Reply-To: <20201103092712.714480842@linutronix.de>
Now that the scheduler can deal with migrate disable properly, there is no
real compelling reason to make it only available for RT.
There are quite some code pathes which needlessly disable preemption in
order to prevent migration and some constructs like kmap_atomic() enforce
it implicitly.
Making it available independent of RT allows to provide a preemptible
variant of kmap_atomic() and makes the code more consistent in general.
FIXME: Rework the comment in preempt.h
Signed-off-by: Thomas Gleixner <tglx@linutronix.de>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Ingo Molnar <mingo@kernel.org>
Cc: Juri Lelli <juri.lelli@redhat.com>
Cc: Vincent Guittot <vincent.guittot@linaro.org>
Cc: Dietmar Eggemann <dietmar.eggemann@arm.com>
Cc: Steven Rostedt <rostedt@goodmis.org>
Cc: Ben Segall <bsegall@google.com>
Cc: Mel Gorman <mgorman@suse.de>
Cc: Daniel Bristot de Oliveira <bristot@redhat.com>
---
include/linux/kernel.h | 21 ++++++++++++++-------
include/linux/preempt.h | 38 +++-----------------------------------
include/linux/sched.h | 2 +-
kernel/sched/core.c | 45 +++++++++++++++++++++++++++++++++++----------
kernel/sched/sched.h | 4 ++--
lib/smp_processor_id.c | 2 +-
6 files changed, 56 insertions(+), 56 deletions(-)
--- a/include/linux/kernel.h
+++ b/include/linux/kernel.h
@@ -204,6 +204,7 @@ extern int _cond_resched(void);
extern void ___might_sleep(const char *file, int line, int preempt_offset);
extern void __might_sleep(const char *file, int line, int preempt_offset);
extern void __cant_sleep(const char *file, int line, int preempt_offset);
+extern void __cant_migrate(const char *file, int line);
/**
* might_sleep - annotation for functions that can sleep
@@ -227,6 +228,18 @@ extern void __cant_sleep(const char *fil
# define cant_sleep() \
do { __cant_sleep(__FILE__, __LINE__, 0); } while (0)
# define sched_annotate_sleep() (current->task_state_change = 0)
+
+/**
+ * cant_migrate - annotation for functions that cannot migrate
+ *
+ * Will print a stack trace if executed in code which is migratable
+ */
+# define cant_migrate() \
+ do { \
+ if (IS_ENABLED(CONFIG_SMP)) \
+ __cant_migrate(__FILE__, __LINE__); \
+ } while (0)
+
/**
* non_block_start - annotate the start of section where sleeping is prohibited
*
@@ -251,6 +264,7 @@ extern void __cant_sleep(const char *fil
int preempt_offset) { }
# define might_sleep() do { might_resched(); } while (0)
# define cant_sleep() do { } while (0)
+# define cant_migrate() do { } while (0)
# define sched_annotate_sleep() do { } while (0)
# define non_block_start() do { } while (0)
# define non_block_end() do { } while (0)
@@ -258,13 +272,6 @@ extern void __cant_sleep(const char *fil
#define might_sleep_if(cond) do { if (cond) might_sleep(); } while (0)
-#ifndef CONFIG_PREEMPT_RT
-# define cant_migrate() cant_sleep()
-#else
- /* Placeholder for now */
-# define cant_migrate() do { } while (0)
-#endif
-
/**
* abs - return absolute value of an argument
* @x: the value. If it is unsigned type, it is converted to signed type first.
--- a/include/linux/preempt.h
+++ b/include/linux/preempt.h
@@ -322,7 +322,7 @@ static inline void preempt_notifier_init
#endif
-#if defined(CONFIG_SMP) && defined(CONFIG_PREEMPT_RT)
+#ifdef CONFIG_SMP
/*
* Migrate-Disable and why it is undesired.
@@ -382,43 +382,11 @@ static inline void preempt_notifier_init
extern void migrate_disable(void);
extern void migrate_enable(void);
-#elif defined(CONFIG_PREEMPT_RT)
+#else
static inline void migrate_disable(void) { }
static inline void migrate_enable(void) { }
-#else /* !CONFIG_PREEMPT_RT */
-
-/**
- * migrate_disable - Prevent migration of the current task
- *
- * Maps to preempt_disable() which also disables preemption. Use
- * migrate_disable() to annotate that the intent is to prevent migration,
- * but not necessarily preemption.
- *
- * Can be invoked nested like preempt_disable() and needs the corresponding
- * number of migrate_enable() invocations.
- */
-static __always_inline void migrate_disable(void)
-{
- preempt_disable();
-}
-
-/**
- * migrate_enable - Allow migration of the current task
- *
- * Counterpart to migrate_disable().
- *
- * As migrate_disable() can be invoked nested, only the outermost invocation
- * reenables migration.
- *
- * Currently mapped to preempt_enable().
- */
-static __always_inline void migrate_enable(void)
-{
- preempt_enable();
-}
-
-#endif /* CONFIG_SMP && CONFIG_PREEMPT_RT */
+#endif /* CONFIG_SMP */
#endif /* __LINUX_PREEMPT_H */
--- a/include/linux/sched.h
+++ b/include/linux/sched.h
@@ -715,7 +715,7 @@ struct task_struct {
const cpumask_t *cpus_ptr;
cpumask_t cpus_mask;
void *migration_pending;
-#if defined(CONFIG_SMP) && defined(CONFIG_PREEMPT_RT)
+#ifdef CONFIG_SMP
unsigned short migration_disabled;
#endif
unsigned short migration_flags;
--- a/kernel/sched/core.c
+++ b/kernel/sched/core.c
@@ -1696,8 +1696,6 @@ void check_preempt_curr(struct rq *rq, s
#ifdef CONFIG_SMP
-#ifdef CONFIG_PREEMPT_RT
-
static void
__do_set_cpus_allowed(struct task_struct *p, const struct cpumask *new_mask, u32 flags);
@@ -1772,8 +1770,6 @@ static inline bool rq_has_pinned_tasks(s
return rq->nr_pinned;
}
-#endif
-
/*
* Per-CPU kthreads are allowed to run on !active && online CPUs, see
* __set_cpus_allowed_ptr() and select_fallback_rq().
@@ -2841,7 +2837,7 @@ void sched_set_stop_task(int cpu, struct
}
}
-#else
+#else /* CONFIG_SMP */
static inline int __set_cpus_allowed_ptr(struct task_struct *p,
const struct cpumask *new_mask,
@@ -2850,10 +2846,6 @@ static inline int __set_cpus_allowed_ptr
return set_cpus_allowed_ptr(p, new_mask);
}
-#endif /* CONFIG_SMP */
-
-#if !defined(CONFIG_SMP) || !defined(CONFIG_PREEMPT_RT)
-
static inline void migrate_disable_switch(struct rq *rq, struct task_struct *p) { }
static inline bool rq_has_pinned_tasks(struct rq *rq)
@@ -2861,7 +2853,7 @@ static inline bool rq_has_pinned_tasks(s
return false;
}
-#endif
+#endif /* !CONFIG_SMP */
static void
ttwu_stat(struct task_struct *p, int cpu, int wake_flags)
@@ -7883,6 +7875,39 @@ void __cant_sleep(const char *file, int
add_taint(TAINT_WARN, LOCKDEP_STILL_OK);
}
EXPORT_SYMBOL_GPL(__cant_sleep);
+
+#ifdef CONFIG_SMP
+void __cant_migrate(const char *file, int line)
+{
+ static unsigned long prev_jiffy;
+
+ if (irqs_disabled())
+ return;
+
+ if (is_migration_disabled(current))
+ return;
+
+ if (!IS_ENABLED(CONFIG_PREEMPT_COUNT))
+ return;
+
+ if (preempt_count() > 0)
+ return;
+
+ if (time_before(jiffies, prev_jiffy + HZ) && prev_jiffy)
+ return;
+ prev_jiffy = jiffies;
+
+ pr_err("BUG: assuming non migratable context at %s:%d\n", file, line);
+ pr_err("in_atomic(): %d, irqs_disabled(): %d, migration_disabled() %u pid: %d, name: %s\n",
+ in_atomic(), irqs_disabled(), is_migration_disabled(current),
+ current->pid, current->comm);
+
+ debug_show_held_locks(current);
+ dump_stack();
+ add_taint(TAINT_WARN, LOCKDEP_STILL_OK);
+}
+EXPORT_SYMBOL_GPL(__cant_migrate);
+#endif
#endif
#ifdef CONFIG_MAGIC_SYSRQ
--- a/kernel/sched/sched.h
+++ b/kernel/sched/sched.h
@@ -1056,7 +1056,7 @@ struct rq {
struct cpuidle_state *idle_state;
#endif
-#if defined(CONFIG_PREEMPT_RT) && defined(CONFIG_SMP)
+#ifdef CONFIG_SMP
unsigned int nr_pinned;
#endif
unsigned int push_busy;
@@ -1092,7 +1092,7 @@ static inline int cpu_of(struct rq *rq)
static inline bool is_migration_disabled(struct task_struct *p)
{
-#if defined(CONFIG_SMP) && defined(CONFIG_PREEMPT_RT)
+#ifdef CONFIG_SMP
return p->migration_disabled;
#else
return false;
--- a/lib/smp_processor_id.c
+++ b/lib/smp_processor_id.c
@@ -26,7 +26,7 @@ unsigned int check_preemption_disabled(c
if (current->nr_cpus_allowed == 1)
goto out;
-#if defined(CONFIG_SMP) && defined(CONFIG_PREEMPT_RT)
+#ifdef CONFIG_SMP
if (current->migration_disabled)
goto out;
#endif
^ permalink raw reply
* [patch V3 22/37] highmem: High implementation details and document API
From: Thomas Gleixner @ 2020-11-03 9:27 UTC (permalink / raw)
To: LKML
Cc: Juri Lelli, linux-aio, Peter Zijlstra, Sebastian Andrzej Siewior,
Joonas Lahtinen, dri-devel, virtualization, Ben Segall,
Chris Mason, Huang Rui, Paul Mackerras, Gerd Hoffmann,
Daniel Bristot de Oliveira, sparclinux, Vincent Chen,
Christoph Hellwig, Vincent Guittot, Paul McKenney, Max Filippov,
x86, Russell King, linux-csky, Ingo Molnar, David Airlie,
VMware Graphics, Mel Gorman, nouveau, Dave Airlie, linux-snps-arc,
Ben Skeggs, linux-xtensa, Arnd Bergmann, intel-gfx,
Roland Scheidegger, Josef Bacik, Steven Rostedt, Linus Torvalds,
Alexander Viro, spice-devel, David Sterba, Rodrigo Vivi,
Dietmar Eggemann, linux-arm-kernel, Jani Nikula, Chris Zankel,
Michal Simek, Thomas Bogendoerfer, Nick Hu, linux-mm,
Vineet Gupta, linux-mips, Christian Koenig, Benjamin LaHaise,
Daniel Vetter, linux-fsdevel, Andrew Morton, linuxppc-dev,
David S. Miller, linux-btrfs, Greentime Hu
In-Reply-To: <20201103092712.714480842@linutronix.de>
Move the gory details of kmap & al into a private header and only document
the interfaces which are usable by drivers.
Signed-off-by: Thomas Gleixner <tglx@linutronix.de>
---
V3: New patch
---
include/linux/highmem-internal.h | 174 +++++++++++++++++++++++++
include/linux/highmem.h | 270 ++++++++++++++-------------------------
mm/highmem.c | 11 -
3 files changed, 276 insertions(+), 179 deletions(-)
--- /dev/null
+++ b/include/linux/highmem-internal.h
@@ -0,0 +1,174 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+#ifndef _LINUX_HIGHMEM_INTERNAL_H
+#define _LINUX_HIGHMEM_INTERNAL_H
+
+/*
+ * Outside of CONFIG_HIGHMEM to support X86 32bit iomap_atomic() cruft.
+ */
+#ifdef CONFIG_KMAP_LOCAL
+void *__kmap_local_pfn_prot(unsigned long pfn, pgprot_t prot);
+void *__kmap_local_page_prot(struct page *page, pgprot_t prot);
+void kunmap_local_indexed(void *vaddr);
+#endif
+
+#ifdef CONFIG_HIGHMEM
+#include <asm/highmem.h>
+
+#ifndef ARCH_HAS_KMAP_FLUSH_TLB
+static inline void kmap_flush_tlb(unsigned long addr) { }
+#endif
+
+#ifndef kmap_prot
+#define kmap_prot PAGE_KERNEL
+#endif
+
+void *kmap_high(struct page *page);
+void kunmap_high(struct page *page);
+void __kmap_flush_unused(void);
+struct page *__kmap_to_page(void *addr);
+
+static inline void *kmap(struct page *page)
+{
+ void *addr;
+
+ might_sleep();
+ if (!PageHighMem(page))
+ addr = page_address(page);
+ else
+ addr = kmap_high(page);
+ kmap_flush_tlb((unsigned long)addr);
+ return addr;
+}
+
+static inline void kunmap(struct page *page)
+{
+ might_sleep();
+ if (!PageHighMem(page))
+ return;
+ kunmap_high(page);
+}
+
+static inline struct page *kmap_to_page(void *addr)
+{
+ return __kmap_to_page(addr);
+}
+
+static inline void kmap_flush_unused(void)
+{
+ __kmap_flush_unused();
+}
+
+static inline void *kmap_atomic_prot(struct page *page, pgprot_t prot)
+{
+ preempt_disable();
+ pagefault_disable();
+ return __kmap_local_page_prot(page, prot);
+}
+
+static inline void *kmap_atomic(struct page *page)
+{
+ return kmap_atomic_prot(page, kmap_prot);
+}
+
+static inline void *kmap_atomic_pfn(unsigned long pfn)
+{
+ preempt_disable();
+ pagefault_disable();
+ return __kmap_local_pfn_prot(pfn, kmap_prot);
+}
+
+static inline void __kunmap_atomic(void *addr)
+{
+ kunmap_local_indexed(addr);
+ pagefault_enable();
+ preempt_enable();
+}
+
+unsigned int __nr_free_highpages(void);
+extern atomic_long_t _totalhigh_pages;
+
+static inline unsigned int nr_free_highpages(void)
+{
+ return __nr_free_highpages();
+}
+
+static inline unsigned long totalhigh_pages(void)
+{
+ return (unsigned long)atomic_long_read(&_totalhigh_pages);
+}
+
+static inline void totalhigh_pages_inc(void)
+{
+ atomic_long_inc(&_totalhigh_pages);
+}
+
+static inline void totalhigh_pages_add(long count)
+{
+ atomic_long_add(count, &_totalhigh_pages);
+}
+
+#else /* CONFIG_HIGHMEM */
+
+static inline struct page *kmap_to_page(void *addr)
+{
+ return virt_to_page(addr);
+}
+
+static inline void *kmap(struct page *page)
+{
+ might_sleep();
+ return page_address(page);
+}
+
+static inline void kunmap_high(struct page *page) { }
+static inline void kmap_flush_unused(void) { }
+
+static inline void kunmap(struct page *page)
+{
+#ifdef ARCH_HAS_FLUSH_ON_KUNMAP
+ kunmap_flush_on_unmap(page_address(page));
+#endif
+}
+
+static inline void *kmap_atomic(struct page *page)
+{
+ preempt_disable();
+ pagefault_disable();
+ return page_address(page);
+}
+
+static inline void *kmap_atomic_prot(struct page *page, pgprot_t prot)
+{
+ return kmap_atomic(page);
+}
+
+static inline void *kmap_atomic_pfn(unsigned long pfn)
+{
+ return kmap_atomic(pfn_to_page(pfn));
+}
+
+static inline void __kunmap_atomic(void *addr)
+{
+#ifdef ARCH_HAS_FLUSH_ON_KUNMAP
+ kunmap_flush_on_unmap(addr);
+#endif
+ pagefault_enable();
+ preempt_enable();
+}
+
+static inline unsigned int nr_free_highpages(void) { return 0; }
+static inline unsigned long totalhigh_pages(void) { return 0UL; }
+
+#endif /* CONFIG_HIGHMEM */
+
+/*
+ * Prevent people trying to call kunmap_atomic() as if it were kunmap()
+ * kunmap_atomic() should get the return value of kmap_atomic, not the page.
+ */
+#define kunmap_atomic(__addr) \
+do { \
+ BUILD_BUG_ON(__same_type((__addr), struct page *)); \
+ __kunmap_atomic(__addr); \
+} while (0)
+
+#endif
--- a/include/linux/highmem.h
+++ b/include/linux/highmem.h
@@ -11,199 +11,125 @@
#include <asm/cacheflush.h>
-#ifndef ARCH_HAS_FLUSH_ANON_PAGE
-static inline void flush_anon_page(struct vm_area_struct *vma, struct page *page, unsigned long vmaddr)
-{
-}
-#endif
+#include "highmem-internal.h"
-#ifndef ARCH_HAS_FLUSH_KERNEL_DCACHE_PAGE
-static inline void flush_kernel_dcache_page(struct page *page)
-{
-}
-static inline void flush_kernel_vmap_range(void *vaddr, int size)
-{
-}
-static inline void invalidate_kernel_vmap_range(void *vaddr, int size)
-{
-}
-#endif
-
-/*
- * Outside of CONFIG_HIGHMEM to support X86 32bit iomap_atomic() cruft.
+/**
+ * kmap - Map a page for long term usage
+ * @page: Pointer to the page to be mapped
+ *
+ * Returns: The virtual address of the mapping
+ *
+ * Can only be invoked from preemptible task context because on 32bit
+ * systems with CONFIG_HIGHMEM enabled this function might sleep.
+ *
+ * For systems with CONFIG_HIGHMEM=n and for pages in the low memory area
+ * this returns the virtual address of the direct kernel mapping.
+ *
+ * The returned virtual address is globally visible and valid up to the
+ * point where it is unmapped via kunmap(). The pointer can be handed to
+ * other contexts.
+ *
+ * For highmem pages on 32bit systems this can be slow as the mapping space
+ * is limited and protected by a global lock. In case that there is no
+ * mapping slot available the function blocks until a slot is released via
+ * kunmap().
*/
-#ifdef CONFIG_KMAP_LOCAL
-void *__kmap_local_pfn_prot(unsigned long pfn, pgprot_t prot);
-void *__kmap_local_page_prot(struct page *page, pgprot_t prot);
-void kunmap_local_indexed(void *vaddr);
-#endif
-
-#ifdef CONFIG_HIGHMEM
-#include <asm/highmem.h>
+static inline void *kmap(struct page *page);
-#ifndef ARCH_HAS_KMAP_FLUSH_TLB
-static inline void kmap_flush_tlb(unsigned long addr) { }
-#endif
-
-#ifndef kmap_prot
-#define kmap_prot PAGE_KERNEL
-#endif
-
-void *kmap_high(struct page *page);
-static inline void *kmap(struct page *page)
-{
- void *addr;
-
- might_sleep();
- if (!PageHighMem(page))
- addr = page_address(page);
- else
- addr = kmap_high(page);
- kmap_flush_tlb((unsigned long)addr);
- return addr;
-}
-
-void kunmap_high(struct page *page);
-
-static inline void kunmap(struct page *page)
-{
- might_sleep();
- if (!PageHighMem(page))
- return;
- kunmap_high(page);
-}
-
-/*
- * kmap_atomic/kunmap_atomic is significantly faster than kmap/kunmap because
- * no global lock is needed and because the kmap code must perform a global TLB
- * invalidation when the kmap pool wraps.
- *
- * However when holding an atomic kmap it is not legal to sleep, so atomic
- * kmaps are appropriate for short, tight code paths only.
- *
- * The use of kmap_atomic/kunmap_atomic is discouraged - kmap/kunmap
- * gives a more generic (and caching) interface. But kmap_atomic can
- * be used in IRQ contexts, so in some (very limited) cases we need
- * it.
+/**
+ * kunmap - Unmap the virtual address mapped by kmap()
+ * @addr: Virtual address to be unmapped
+ *
+ * Counterpart to kmap(). A NOOP for CONFIG_HIGHMEM=n and for mappings of
+ * pages in the low memory area.
*/
-static inline void *kmap_atomic_prot(struct page *page, pgprot_t prot)
-{
- preempt_disable();
- pagefault_disable();
- return __kmap_local_page_prot(page, prot);
-}
-
-static inline void *kmap_atomic(struct page *page)
-{
- return kmap_atomic_prot(page, kmap_prot);
-}
-
-static inline void *kmap_atomic_pfn(unsigned long pfn)
-{
- preempt_disable();
- pagefault_disable();
- return __kmap_local_pfn_prot(pfn, kmap_prot);
-}
-
-static inline void __kunmap_atomic(void *addr)
-{
- kunmap_local_indexed(addr);
-}
-
-/* declarations for linux/mm/highmem.c */
-unsigned int nr_free_highpages(void);
-extern atomic_long_t _totalhigh_pages;
-static inline unsigned long totalhigh_pages(void)
-{
- return (unsigned long)atomic_long_read(&_totalhigh_pages);
-}
+static inline void kunmap(struct page *page);
-static inline void totalhigh_pages_inc(void)
-{
- atomic_long_inc(&_totalhigh_pages);
-}
-
-static inline void totalhigh_pages_add(long count)
-{
- atomic_long_add(count, &_totalhigh_pages);
-}
-
-void kmap_flush_unused(void);
-
-struct page *kmap_to_page(void *addr);
-
-#else /* CONFIG_HIGHMEM */
+/**
+ * kmap_to_page - Get the page for a kmap'ed address
+ * @addr: The address to look up
+ *
+ * Returns: The page which is mapped to @addr.
+ */
+static inline struct page *kmap_to_page(void *addr);
-static inline unsigned int nr_free_highpages(void) { return 0; }
+/**
+ * kmap_flush_unused - Flush all unused kmap mappings in order to
+ * remove stray mappings
+ */
+static inline void kmap_flush_unused(void);
-static inline struct page *kmap_to_page(void *addr)
-{
- return virt_to_page(addr);
-}
+/**
+ * kmap_atomic - Atomically map a page for temporary usage
+ * @page: Pointer to the page to be mapped
+ *
+ * Returns: The virtual address of the mapping
+ *
+ * Side effect: On return pagefaults and preemption are disabled.
+ *
+ * Can be invoked from any context.
+ *
+ * Requires careful handling when nesting multiple mappings because the map
+ * management is stack based. The unmap has to be in the reverse order of
+ * the map operation:
+ *
+ * addr1 = kmap_atomic(page1);
+ * addr2 = kmap_atomic(page2);
+ * ...
+ * kunmap_atomic(addr2);
+ * kunmap_atomic(addr1);
+ *
+ * Unmapping addr1 before addr2 is invalid and causes malfunction.
+ *
+ * Contrary to kmap() mappings the mapping is only valid in the context of
+ * the caller and cannot be handed to other contexts.
+ *
+ * On CONFIG_HIGHMEM=n kernels and for low memory pages this returns the
+ * virtual address of the direct mapping. Only real highmem pages are
+ * temporarily mapped.
+ *
+ * While it is significantly faster than kmap() it comes with restrictions
+ * about the pointer validity and the side effects of disabling page faults
+ * and preemption. Use it only when absolutely necessary, e.g. from non
+ * preemptible contexts.
+ */
+static inline void *kmap_atomic(struct page *page);
-static inline unsigned long totalhigh_pages(void) { return 0UL; }
+/**
+ * kunmap_atomic - Unmap the virtual address mapped by kmap_atomic()
+ * @addr: Virtual address to be unmapped
+ *
+ * Counterpart to kmap_atomic().
+ *
+ * Undoes the side effects of kmap_atomic(), i.e. reenabling pagefaults and
+ * preemption.
+ *
+ * Other than that a NOOP for CONFIG_HIGHMEM=n and for mappings of pages
+ * in the low memory area. For real highmen pages the mapping which was
+ * established with kmap_atomic() is destroyed.
+ */
-static inline void *kmap(struct page *page)
-{
- might_sleep();
- return page_address(page);
-}
+/* Highmem related interfaces for management code */
+static inline unsigned int nr_free_highpages(void);
+static inline unsigned long totalhigh_pages(void);
-static inline void kunmap_high(struct page *page)
+#ifndef ARCH_HAS_FLUSH_ANON_PAGE
+static inline void flush_anon_page(struct vm_area_struct *vma, struct page *page, unsigned long vmaddr)
{
}
-
-static inline void kunmap(struct page *page)
-{
-#ifdef ARCH_HAS_FLUSH_ON_KUNMAP
- kunmap_flush_on_unmap(page_address(page));
#endif
-}
-static inline void *kmap_atomic(struct page *page)
+#ifndef ARCH_HAS_FLUSH_KERNEL_DCACHE_PAGE
+static inline void flush_kernel_dcache_page(struct page *page)
{
- preempt_disable();
- pagefault_disable();
- return page_address(page);
}
-
-static inline void *kmap_atomic_prot(struct page *page, pgprot_t prot)
+static inline void flush_kernel_vmap_range(void *vaddr, int size)
{
- return kmap_atomic(page);
}
-
-static inline void *kmap_atomic_pfn(unsigned long pfn)
+static inline void invalidate_kernel_vmap_range(void *vaddr, int size)
{
- return kmap_atomic(pfn_to_page(pfn));
}
-
-static inline void __kunmap_atomic(void *addr)
-{
- /*
- * Mostly nothing to do in the CONFIG_HIGHMEM=n case as kunmap_atomic()
- * handles re-enabling faults and preemption
- */
-#ifdef ARCH_HAS_FLUSH_ON_KUNMAP
- kunmap_flush_on_unmap(addr);
#endif
-}
-
-#define kmap_flush_unused() do {} while(0)
-
-
-#endif /* CONFIG_HIGHMEM */
-
-/*
- * Prevent people trying to call kunmap_atomic() as if it were kunmap()
- * kunmap_atomic() should get the return value of kmap_atomic, not the page.
- */
-#define kunmap_atomic(__addr) \
-do { \
- BUILD_BUG_ON(__same_type((__addr), struct page *)); \
- __kunmap_atomic(__addr); \
- pagefault_enable(); \
- preempt_enable(); \
-} while (0)
/* when CONFIG_HIGHMEM is not set these will be plain clear/copy_page */
#ifndef clear_user_highpage
--- a/mm/highmem.c
+++ b/mm/highmem.c
@@ -104,7 +104,7 @@ static inline wait_queue_head_t *get_pkm
atomic_long_t _totalhigh_pages __read_mostly;
EXPORT_SYMBOL(_totalhigh_pages);
-unsigned int nr_free_highpages (void)
+unsigned int __nr_free_highpages (void)
{
struct zone *zone;
unsigned int pages = 0;
@@ -141,7 +141,7 @@ pte_t * pkmap_page_table;
do { spin_unlock(&kmap_lock); (void)(flags); } while (0)
#endif
-struct page *kmap_to_page(void *vaddr)
+struct page *__kmap_to_page(void *vaddr)
{
unsigned long addr = (unsigned long)vaddr;
@@ -152,7 +152,7 @@ struct page *kmap_to_page(void *vaddr)
return virt_to_page(addr);
}
-EXPORT_SYMBOL(kmap_to_page);
+EXPORT_SYMBOL(__kmap_to_page);
static void flush_all_zero_pkmaps(void)
{
@@ -194,10 +194,7 @@ static void flush_all_zero_pkmaps(void)
flush_tlb_kernel_range(PKMAP_ADDR(0), PKMAP_ADDR(LAST_PKMAP));
}
-/**
- * kmap_flush_unused - flush all unused kmap mappings in order to remove stray mappings
- */
-void kmap_flush_unused(void)
+void __kmap_flush_unused(void)
{
lock_kmap();
flush_all_zero_pkmaps();
^ permalink raw reply
* [patch V3 21/37] Documentation/io-mapping: Remove outdated blurb
From: Thomas Gleixner @ 2020-11-03 9:27 UTC (permalink / raw)
To: LKML
Cc: Juri Lelli, linux-aio, Peter Zijlstra, Sebastian Andrzej Siewior,
Joonas Lahtinen, dri-devel, virtualization, Ben Segall,
Chris Mason, Huang Rui, Paul Mackerras, Gerd Hoffmann,
Daniel Bristot de Oliveira, sparclinux, Vincent Chen,
Christoph Hellwig, Vincent Guittot, Paul McKenney, Max Filippov,
x86, Russell King, linux-csky, Ingo Molnar, David Airlie,
VMware Graphics, Mel Gorman, nouveau, Dave Airlie, linux-snps-arc,
Ben Skeggs, linux-xtensa, Arnd Bergmann, intel-gfx,
Roland Scheidegger, Josef Bacik, Steven Rostedt, Linus Torvalds,
Alexander Viro, spice-devel, David Sterba, Rodrigo Vivi,
Dietmar Eggemann, linux-arm-kernel, Jani Nikula, Chris Zankel,
Michal Simek, Thomas Bogendoerfer, Nick Hu, linux-mm,
Vineet Gupta, linux-mips, Christian Koenig, Benjamin LaHaise,
Daniel Vetter, linux-fsdevel, Andrew Morton, linuxppc-dev,
David S. Miller, linux-btrfs, Greentime Hu
In-Reply-To: <20201103092712.714480842@linutronix.de>
The implementation details in the documentation are outdated and not really
helpful. Remove them.
Signed-off-by: Thomas Gleixner <tglx@linutronix.de>
---
V3: New patch
---
Documentation/driver-api/io-mapping.rst | 22 ----------------------
1 file changed, 22 deletions(-)
--- a/Documentation/driver-api/io-mapping.rst
+++ b/Documentation/driver-api/io-mapping.rst
@@ -73,25 +73,3 @@ for pages mapped with io_mapping_map_wc.
At driver close time, the io_mapping object must be freed::
void io_mapping_free(struct io_mapping *mapping)
-
-Current Implementation
-======================
-
-The initial implementation of these functions uses existing mapping
-mechanisms and so provides only an abstraction layer and no new
-functionality.
-
-On 64-bit processors, io_mapping_create_wc calls ioremap_wc for the whole
-range, creating a permanent kernel-visible mapping to the resource. The
-map_atomic and map functions add the requested offset to the base of the
-virtual address returned by ioremap_wc.
-
-On 32-bit processors with HIGHMEM defined, io_mapping_map_atomic_wc uses
-kmap_atomic_pfn to map the specified page in an atomic fashion;
-kmap_atomic_pfn isn't really supposed to be used with device pages, but it
-provides an efficient mapping for this usage.
-
-On 32-bit processors without HIGHMEM defined, io_mapping_map_atomic_wc and
-io_mapping_map_wc both use ioremap_wc, a terribly inefficient function which
-performs an IPI to inform all processors about the new mapping. This results
-in a significant performance penalty.
^ permalink raw reply
page: next (older) | prev (newer) | latest
- recent:[subjects (threaded)|topics (new)|topics (active)]
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox