* Re: [RFC PATCH 02/12] powerpc: remove arguments from interrupt handler functions
From: Christophe Leroy @ 2020-09-07 11:34 UTC (permalink / raw)
To: Nicholas Piggin; +Cc: linuxppc-dev
In-Reply-To: <e34fead9-a356-3ae6-aa33-544380230bd5@csgroup.eu>
On Mon, 2020-09-07 at 11:20 +0200, Christophe Leroy wrote:
>
> Le 05/09/2020 à 19:43, Nicholas Piggin a écrit :
> > Make interrupt handlers all just take the pt_regs * argument and load
> > DAR/DSISR etc from that. Make those that return a value return long.
>
> I like this, it will likely simplify a bit the VMAP_STACK mess.
>
> Not sure it is that easy. My board is stuck after the start of init.
>
>
> On the 8xx, on Instruction TLB Error exception, we do
>
> andis. r5,r9,DSISR_SRR1_MATCH_32S@h /* Filter relevant SRR1 bits */
>
> On book3s/32, on ISI exception we do:
> andis. r5,r9,DSISR_SRR1_MATCH_32S@h /* Filter relevant SRR1 bits */
>
> On 40x and bookE, on ISI exception we do:
> li r5,0 /* Pass zero as arg3 */
>
>
> And regs->dsisr will just contain nothing
>
> So it means we should at least write back r5 into regs->dsisr from there
> ? The performance impact should be minimal as we already write _DAR so
> the cache line should already be in the cache.
>
> A hacky 'stw r5, _DSISR(r1)' in handle_page_fault() does the trick,
> allthough we don't want to do it for both ISI and DSI at the end, so
> you'll have to do it in every head_xxx.S
To get you series build and work, I did the following hacks:
diff --git a/arch/powerpc/include/asm/interrupt.h
b/arch/powerpc/include/asm/interrupt.h
index acfcc7d5779b..c11045d3113a 100644
--- a/arch/powerpc/include/asm/interrupt.h
+++ b/arch/powerpc/include/asm/interrupt.h
@@ -93,7 +93,9 @@ static inline void interrupt_nmi_exit_prepare(struct
pt_regs *regs, struct inter
{
nmi_exit();
+#ifdef CONFIG_PPC64
this_cpu_set_ftrace_enabled(state->ftrace_enabled);
+#endif
#ifdef CONFIG_PPC_BOOK3S_64
/* Check we didn't change the pending interrupt mask. */
diff --git a/arch/powerpc/kernel/entry_32.S
b/arch/powerpc/kernel/entry_32.S
index f4d0af8e1136..66f7adbe1076 100644
--- a/arch/powerpc/kernel/entry_32.S
+++ b/arch/powerpc/kernel/entry_32.S
@@ -663,6 +663,7 @@ ppc_swapcontext:
*/
.globl handle_page_fault
handle_page_fault:
+ stw r5,_DSISR(r1)
addi r3,r1,STACK_FRAME_OVERHEAD
#ifdef CONFIG_PPC_BOOK3S_32
andis. r0,r5,DSISR_DABRMATCH@h
---
Christophe
^ permalink raw reply related
* [PATCH] scsi: ibmvfc: Fix error return in ibmvfc_probe()
From: Jing Xiangfeng @ 2020-09-07 8:39 UTC (permalink / raw)
To: tyreld, mpe, benh, paulus, jejb, martin.petersen
Cc: linuxppc-dev, linux-kernel, linux-scsi, jingxiangfeng
Fix to return error code PTR_ERR() from the error handling case instead
of 0.
Signed-off-by: Jing Xiangfeng <jingxiangfeng@huawei.com>
---
drivers/scsi/ibmvscsi/ibmvfc.c | 1 +
1 file changed, 1 insertion(+)
diff --git a/drivers/scsi/ibmvscsi/ibmvfc.c b/drivers/scsi/ibmvscsi/ibmvfc.c
index ea7c8930592d..70daa0605082 100644
--- a/drivers/scsi/ibmvscsi/ibmvfc.c
+++ b/drivers/scsi/ibmvscsi/ibmvfc.c
@@ -4928,6 +4928,7 @@ static int ibmvfc_probe(struct vio_dev *vdev, const struct vio_device_id *id)
if (IS_ERR(vhost->work_thread)) {
dev_err(dev, "Couldn't create kernel thread: %ld\n",
PTR_ERR(vhost->work_thread));
+ rc = PTR_ERR(vhost->work_thread);
goto free_host_mem;
}
--
2.17.1
^ permalink raw reply related
* Re: [PATCH] arch: vdso: add vdso linker script to 'targets' instead of extra-y
From: Masahiro Yamada @ 2020-09-07 12:44 UTC (permalink / raw)
To: Linux Kbuild mailing list
Cc: linux-s390, Catalin Marinas, Vasily Gorbik, Nick Hu,
Heiko Carstens, linuxppc-dev, Linux Kernel Mailing List,
Christian Borntraeger, Paul Mackerras, Greentime Hu, Vincent Chen,
Will Deacon, linux-arm-kernel
In-Reply-To: <20200831182239.480317-1-masahiroy@kernel.org>
On Tue, Sep 1, 2020 at 3:23 AM Masahiro Yamada <masahiroy@kernel.org> wrote:
>
> The vdso linker script is preprocessed on demand.
> Adding it to 'targets' is enough to include the .cmd file.
>
> Signed-off-by: Masahiro Yamada <masahiroy@kernel.org>
> ---
Applied to linux-kbuild.
> arch/arm64/kernel/vdso/Makefile | 2 +-
> arch/arm64/kernel/vdso32/Makefile | 2 +-
> arch/nds32/kernel/vdso/Makefile | 2 +-
> arch/powerpc/kernel/vdso32/Makefile | 2 +-
> arch/powerpc/kernel/vdso64/Makefile | 2 +-
> arch/s390/kernel/vdso64/Makefile | 2 +-
> 6 files changed, 6 insertions(+), 6 deletions(-)
>
> diff --git a/arch/arm64/kernel/vdso/Makefile b/arch/arm64/kernel/vdso/Makefile
> index 45d5cfe46429..7cd8aafbe96e 100644
> --- a/arch/arm64/kernel/vdso/Makefile
> +++ b/arch/arm64/kernel/vdso/Makefile
> @@ -54,7 +54,7 @@ endif
> GCOV_PROFILE := n
>
> obj-y += vdso.o
> -extra-y += vdso.lds
> +targets += vdso.lds
> CPPFLAGS_vdso.lds += -P -C -U$(ARCH)
>
> # Force dependency (incbin is bad)
> diff --git a/arch/arm64/kernel/vdso32/Makefile b/arch/arm64/kernel/vdso32/Makefile
> index d6adb4677c25..572475b7b7ed 100644
> --- a/arch/arm64/kernel/vdso32/Makefile
> +++ b/arch/arm64/kernel/vdso32/Makefile
> @@ -155,7 +155,7 @@ asm-obj-vdso := $(addprefix $(obj)/, $(asm-obj-vdso))
> obj-vdso := $(c-obj-vdso) $(c-obj-vdso-gettimeofday) $(asm-obj-vdso)
>
> obj-y += vdso.o
> -extra-y += vdso.lds
> +targets += vdso.lds
> CPPFLAGS_vdso.lds += -P -C -U$(ARCH)
>
> # Force dependency (vdso.s includes vdso.so through incbin)
> diff --git a/arch/nds32/kernel/vdso/Makefile b/arch/nds32/kernel/vdso/Makefile
> index 7c3c1ccb196e..55df25ef0057 100644
> --- a/arch/nds32/kernel/vdso/Makefile
> +++ b/arch/nds32/kernel/vdso/Makefile
> @@ -20,7 +20,7 @@ GCOV_PROFILE := n
>
>
> obj-y += vdso.o
> -extra-y += vdso.lds
> +targets += vdso.lds
> CPPFLAGS_vdso.lds += -P -C -U$(ARCH)
>
> # Force dependency
> diff --git a/arch/powerpc/kernel/vdso32/Makefile b/arch/powerpc/kernel/vdso32/Makefile
> index 87ab1152d5ce..fd5072a4c73c 100644
> --- a/arch/powerpc/kernel/vdso32/Makefile
> +++ b/arch/powerpc/kernel/vdso32/Makefile
> @@ -29,7 +29,7 @@ ccflags-y := -shared -fno-common -fno-builtin -nostdlib \
> asflags-y := -D__VDSO32__ -s
>
> obj-y += vdso32_wrapper.o
> -extra-y += vdso32.lds
> +targets += vdso32.lds
> CPPFLAGS_vdso32.lds += -P -C -Upowerpc
>
> # Force dependency (incbin is bad)
> diff --git a/arch/powerpc/kernel/vdso64/Makefile b/arch/powerpc/kernel/vdso64/Makefile
> index 38c317f25141..c737b3ea3207 100644
> --- a/arch/powerpc/kernel/vdso64/Makefile
> +++ b/arch/powerpc/kernel/vdso64/Makefile
> @@ -17,7 +17,7 @@ ccflags-y := -shared -fno-common -fno-builtin -nostdlib \
> asflags-y := -D__VDSO64__ -s
>
> obj-y += vdso64_wrapper.o
> -extra-y += vdso64.lds
> +targets += vdso64.lds
> CPPFLAGS_vdso64.lds += -P -C -U$(ARCH)
>
> # Force dependency (incbin is bad)
> diff --git a/arch/s390/kernel/vdso64/Makefile b/arch/s390/kernel/vdso64/Makefile
> index 4a66a1cb919b..d0d406cfffa9 100644
> --- a/arch/s390/kernel/vdso64/Makefile
> +++ b/arch/s390/kernel/vdso64/Makefile
> @@ -25,7 +25,7 @@ $(targets:%=$(obj)/%.dbg): KBUILD_CFLAGS = $(KBUILD_CFLAGS_64)
> $(targets:%=$(obj)/%.dbg): KBUILD_AFLAGS = $(KBUILD_AFLAGS_64)
>
> obj-y += vdso64_wrapper.o
> -extra-y += vdso64.lds
> +targets += vdso64.lds
> CPPFLAGS_vdso64.lds += -P -C -U$(ARCH)
>
> # Disable gcov profiling, ubsan and kasan for VDSO code
> --
> 2.25.1
>
--
Best Regards
Masahiro Yamada
^ permalink raw reply
* Re: [RFC PATCH 12/12] powerpc/64s: power4 nap fixup in C
From: Nicholas Piggin @ 2020-09-07 13:05 UTC (permalink / raw)
To: Christophe Leroy; +Cc: linuxppc-dev
In-Reply-To: <9a647445-a438-ae93-f8d5-c177b7fe9662@csgroup.eu>
Excerpts from Christophe Leroy's message of September 7, 2020 2:48 pm:
>
>
> Le 07/09/2020 à 06:02, Nicholas Piggin a écrit :
>> Excerpts from Christophe Leroy's message of September 6, 2020 5:32 pm:
>>>
>>>
>>> Le 05/09/2020 à 19:43, Nicholas Piggin a écrit :
>>>> There is no need for this to be in asm, use the new intrrupt entry wrapper.
>>>>
>>>> Signed-off-by: Nicholas Piggin <npiggin@gmail.com>
>>>> ---
>>>> arch/powerpc/include/asm/interrupt.h | 14 ++++++++
>>>> arch/powerpc/include/asm/processor.h | 1 +
>>>> arch/powerpc/include/asm/thread_info.h | 6 ++++
>>>> arch/powerpc/kernel/exceptions-64s.S | 45 --------------------------
>>>> arch/powerpc/kernel/idle_book3s.S | 4 +++
>>>> 5 files changed, 25 insertions(+), 45 deletions(-)
>>>>
>>>> diff --git a/arch/powerpc/include/asm/processor.h b/arch/powerpc/include/asm/processor.h
>>>> index ed0d633ab5aa..3da1dba91386 100644
>>>> --- a/arch/powerpc/include/asm/processor.h
>>>> +++ b/arch/powerpc/include/asm/processor.h
>>>> @@ -424,6 +424,7 @@ extern unsigned long isa300_idle_stop_mayloss(unsigned long psscr_val);
>>>> extern unsigned long isa206_idle_insn_mayloss(unsigned long type);
>>>> #ifdef CONFIG_PPC_970_NAP
>>>> extern void power4_idle_nap(void);
>>>> +extern void power4_idle_nap_return(void);
>>>
>>> Please please please, 'extern' keyword is pointless and deprecated for
>>> function prototypes. Don't add new ones.
>>>
>>> Also, put it outside the #ifdef, so that you can use IS_ENABLED()
>>> instead of #ifdef when using it.
>>
>> I just copy paste and forget to remove it. I expect someone will do a
>> "cleanup" patch to get rid of them in one go, I find a random assortment
>> of extern and not extern to be even uglier :(
>
> If we don't want to make fixes backporting a huge headache, some
> transition with random assortment is the price to pay.
>
> One day, when 'extern' have become the minority, we can get rid of the
> few last ones.
>
> But if someone believe it is not such a problem with backporting, I can
> provide a cleanup patch now.
I can't really see declarations being a big problem for backporting
or git history. But maybe Michael won't like a patch.
I will try to remember externs though.
Thanks,
Nick
^ permalink raw reply
* Re: KVM on POWER8 host lock up since 10d91611f426 ("powerpc/64s: Reimplement book3s idle code in C")
From: Nicholas Piggin @ 2020-09-07 13:13 UTC (permalink / raw)
To: Michael Ellerman, Michal Suchánek; +Cc: ro, linuxppc-dev, Hari Bathini
In-Reply-To: <87y2lv1430.fsf@mpe.ellerman.id.au>
Excerpts from Michael Ellerman's message of August 31, 2020 8:50 pm:
> Michal Suchánek <msuchanek@suse.de> writes:
>> On Mon, Aug 31, 2020 at 11:14:18AM +1000, Nicholas Piggin wrote:
>>> Excerpts from Michal Suchánek's message of August 31, 2020 6:11 am:
>>> > Hello,
>>> >
>>> > on POWER8 KVM hosts lock up since commit 10d91611f426 ("powerpc/64s:
>>> > Reimplement book3s idle code in C").
>>> >
>>> > The symptom is host locking up completely after some hours of KVM
>>> > workload with messages like
>>> >
>>> > 2020-08-30T10:51:31+00:00 obs-power8-01 kernel: KVM: couldn't grab cpu 47
>>> > 2020-08-30T10:51:31+00:00 obs-power8-01 kernel: KVM: couldn't grab cpu 71
>>> > 2020-08-30T10:51:31+00:00 obs-power8-01 kernel: KVM: couldn't grab cpu 47
>>> > 2020-08-30T10:51:31+00:00 obs-power8-01 kernel: KVM: couldn't grab cpu 71
>>> > 2020-08-30T10:51:31+00:00 obs-power8-01 kernel: KVM: couldn't grab cpu 47
>>> >
>>> > printed before the host locks up.
>>> >
>>> > The machines run sandboxed builds which is a mixed workload resulting in
>>> > IO/single core/mutiple core load over time and there are periods of no
>>> > activity and no VMS runnig as well. The VMs are shortlived so VM
>>> > setup/terdown is somewhat excercised as well.
>>> >
>>> > POWER9 with the new guest entry fast path does not seem to be affected.
>>> >
>>> > Reverted the patch and the followup idle fixes on top of 5.2.14 and
>>> > re-applied commit a3f3072db6ca ("powerpc/powernv/idle: Restore IAMR
>>> > after idle") which gives same idle code as 5.1.16 and the kernel seems
>>> > stable.
>>> >
>>> > Config is attached.
>>> >
>>> > I cannot easily revert this commit, especially if I want to use the same
>>> > kernel on POWER8 and POWER9 - many of the POWER9 fixes are applicable
>>> > only to the new idle code.
>>> >
>>> > Any idea what can be the problem?
>>>
>>> So hwthread_state is never getting back to to HWTHREAD_IN_IDLE on
>>> those threads. I wonder what they are doing. POWER8 doesn't have a good
>>> NMI IPI and I don't know if it supports pdbg dumping registers from the
>>> BMC unfortunately.
>>
>> It may be possible to set up fadump with a later kernel version that
>> supports it on powernv and dump the whole kernel.
>
> Your firmware won't support it AFAIK.
>
> You could try kdump, but if we have CPUs stuck in KVM then there's a
> good chance it won't work :/
I haven't had any luck yet reproducing this still. Testing with sub
cores of various different combinations, etc. I'll keep trying though.
I don't know if there's much we can add to debug it. Can we run pdbg
on the BMCs on these things?
Thanks,
Nick
^ permalink raw reply
* Re: KVM on POWER8 host lock up since 10d91611f426 ("powerpc/64s: Reimplement book3s idle code in C")
From: Michal Suchánek @ 2020-09-07 13:25 UTC (permalink / raw)
To: Nicholas Piggin; +Cc: ro, linuxppc-dev, Hari Bathini
In-Reply-To: <1599484062.vgmycu6q5i.astroid@bobo.none>
On Mon, Sep 07, 2020 at 11:13:47PM +1000, Nicholas Piggin wrote:
> Excerpts from Michael Ellerman's message of August 31, 2020 8:50 pm:
> > Michal Suchánek <msuchanek@suse.de> writes:
> >> On Mon, Aug 31, 2020 at 11:14:18AM +1000, Nicholas Piggin wrote:
> >>> Excerpts from Michal Suchánek's message of August 31, 2020 6:11 am:
> >>> > Hello,
> >>> >
> >>> > on POWER8 KVM hosts lock up since commit 10d91611f426 ("powerpc/64s:
> >>> > Reimplement book3s idle code in C").
> >>> >
> >>> > The symptom is host locking up completely after some hours of KVM
> >>> > workload with messages like
> >>> >
> >>> > 2020-08-30T10:51:31+00:00 obs-power8-01 kernel: KVM: couldn't grab cpu 47
> >>> > 2020-08-30T10:51:31+00:00 obs-power8-01 kernel: KVM: couldn't grab cpu 71
> >>> > 2020-08-30T10:51:31+00:00 obs-power8-01 kernel: KVM: couldn't grab cpu 47
> >>> > 2020-08-30T10:51:31+00:00 obs-power8-01 kernel: KVM: couldn't grab cpu 71
> >>> > 2020-08-30T10:51:31+00:00 obs-power8-01 kernel: KVM: couldn't grab cpu 47
> >>> >
> >>> > printed before the host locks up.
> >>> >
> >>> > The machines run sandboxed builds which is a mixed workload resulting in
> >>> > IO/single core/mutiple core load over time and there are periods of no
> >>> > activity and no VMS runnig as well. The VMs are shortlived so VM
> >>> > setup/terdown is somewhat excercised as well.
> >>> >
> >>> > POWER9 with the new guest entry fast path does not seem to be affected.
> >>> >
> >>> > Reverted the patch and the followup idle fixes on top of 5.2.14 and
> >>> > re-applied commit a3f3072db6ca ("powerpc/powernv/idle: Restore IAMR
> >>> > after idle") which gives same idle code as 5.1.16 and the kernel seems
> >>> > stable.
> >>> >
> >>> > Config is attached.
> >>> >
> >>> > I cannot easily revert this commit, especially if I want to use the same
> >>> > kernel on POWER8 and POWER9 - many of the POWER9 fixes are applicable
> >>> > only to the new idle code.
> >>> >
> >>> > Any idea what can be the problem?
> >>>
> >>> So hwthread_state is never getting back to to HWTHREAD_IN_IDLE on
> >>> those threads. I wonder what they are doing. POWER8 doesn't have a good
> >>> NMI IPI and I don't know if it supports pdbg dumping registers from the
> >>> BMC unfortunately.
> >>
> >> It may be possible to set up fadump with a later kernel version that
> >> supports it on powernv and dump the whole kernel.
> >
> > Your firmware won't support it AFAIK.
> >
> > You could try kdump, but if we have CPUs stuck in KVM then there's a
> > good chance it won't work :/
>
> I haven't had any luck yet reproducing this still. Testing with sub
> cores of various different combinations, etc. I'll keep trying though.
>
> I don't know if there's much we can add to debug it. Can we run pdbg
> on the BMCs on these things?
I suppose it depends on the machine type?
Thanks
Michal
^ permalink raw reply
* Re: [PATCH] kbuild: preprocess module linker script
From: Will Deacon @ 2020-09-07 13:30 UTC (permalink / raw)
To: Masahiro Yamada
Cc: linux-ia64, Peter Zijlstra, Catalin Marinas, Paul Mackerras,
linux-riscv, Ard Biesheuvel, Anton Ivanov, linux-arch,
Mauro Carvalho Chehab, Richard Weinberger, Russell King,
Ingo Molnar, Geert Uytterhoeven, Fenghua Yu, Albert Ou, Kees Cook,
Arnd Bergmann, linux-kbuild, Jeff Dike, Jessica Yu, linux-um,
linux-m68k, Tony Luck, Paul Walmsley, linux-arm-kernel,
Michal Marek, linux-kernel, Palmer Dabbelt, linuxppc-dev
In-Reply-To: <20200904133122.133071-1-masahiroy@kernel.org>
On Fri, Sep 04, 2020 at 10:31:21PM +0900, Masahiro Yamada wrote:
> There was a request to preprocess the module linker script like we do
> for the vmlinux one (https://lkml.org/lkml/2020/8/21/512).
>
> The difference between vmlinux.lds and module.lds is that the latter
> is needed for external module builds, thus must be cleaned up by
> 'make mrproper' instead of 'make clean' (also, it must be created by
> 'make modules_prepare').
>
> You cannot put it in arch/*/kernel/ because 'make clean' descends into
> it. I moved arch/*/kernel/module.lds to arch/*/include/asm/module.lds.h,
> which is included from scripts/module.lds.S.
>
> scripts/module.lds is fine because 'make clean' keeps all the build
> artifacts under scripts/.
>
> You can add arch-specific sections in <asm/module.lds.h>.
>
> Signed-off-by: Masahiro Yamada <masahiroy@kernel.org>
> Tested-by: Jessica Yu <jeyu@kernel.org>
> ---
For the arm64 bits:
Acked-by: Will Deacon <will@kernel.org>
Thanks,
Will
^ permalink raw reply
* [PATCH 1/2] powerpc/32: Fix vmap stack - Do not activate MMU before reading task struct
From: Christophe Leroy @ 2020-09-07 13:42 UTC (permalink / raw)
To: Benjamin Herrenschmidt, Paul Mackerras, Michael Ellerman
Cc: linuxppc-dev, linux-kernel
We need r1 to be properly set before activating MMU, so
reading task_struct->stack must be done with MMU off.
This means we need an additional register to play with MSR
bits while r11 now points to the stack. For that, move r10
back to CR (As is already done for hash MMU) and use r10.
We still don't have r1 correct yet when we activate MMU.
It is done in following patch.
Fixes: 028474876f47 ("powerpc/32: prepare for CONFIG_VMAP_STACK")
Cc: stable@vger.kernel.org
Signed-off-by: Christophe Leroy <christophe.leroy@csgroup.eu>
---
arch/powerpc/kernel/head_32.S | 6 ------
arch/powerpc/kernel/head_32.h | 31 ++++++-------------------------
2 files changed, 6 insertions(+), 31 deletions(-)
diff --git a/arch/powerpc/kernel/head_32.S b/arch/powerpc/kernel/head_32.S
index f3ab94d73936..d967266d62e8 100644
--- a/arch/powerpc/kernel/head_32.S
+++ b/arch/powerpc/kernel/head_32.S
@@ -274,14 +274,8 @@ __secondary_hold_acknowledge:
DO_KVM 0x200
MachineCheck:
EXCEPTION_PROLOG_0
-#ifdef CONFIG_VMAP_STACK
- li r11, MSR_KERNEL & ~(MSR_IR | MSR_RI) /* can take DTLB miss */
- mtmsr r11
- isync
-#endif
#ifdef CONFIG_PPC_CHRP
mfspr r11, SPRN_SPRG_THREAD
- tovirt_vmstack r11, r11
lwz r11, RTAS_SP(r11)
cmpwi cr1, r11, 0
bne cr1, 7f
diff --git a/arch/powerpc/kernel/head_32.h b/arch/powerpc/kernel/head_32.h
index 9abec6cd099c..21effebb9277 100644
--- a/arch/powerpc/kernel/head_32.h
+++ b/arch/powerpc/kernel/head_32.h
@@ -39,24 +39,13 @@
.endm
.macro EXCEPTION_PROLOG_1 for_rtas=0
-#ifdef CONFIG_VMAP_STACK
- .ifeq \for_rtas
- li r11, MSR_KERNEL & ~(MSR_IR | MSR_RI) /* can take DTLB miss */
- mtmsr r11
- isync
- .endif
subi r11, r1, INT_FRAME_SIZE /* use r1 if kernel */
-#else
- tophys(r11,r1) /* use tophys(r1) if kernel */
- subi r11, r11, INT_FRAME_SIZE /* alloc exc. frame */
-#endif
beq 1f
mfspr r11,SPRN_SPRG_THREAD
- tovirt_vmstack r11, r11
lwz r11,TASK_STACK-THREAD(r11)
addi r11, r11, THREAD_SIZE - INT_FRAME_SIZE
- tophys_novmstack r11, r11
1:
+ tophys_novmstack r11, r11
#ifdef CONFIG_VMAP_STACK
mtcrf 0x7f, r11
bt 32 - THREAD_ALIGN_SHIFT, stack_overflow
@@ -64,12 +53,11 @@
.endm
.macro EXCEPTION_PROLOG_2 handle_dar_dsisr=0
-#if defined(CONFIG_VMAP_STACK) && defined(CONFIG_PPC_BOOK3S)
-BEGIN_MMU_FTR_SECTION
+#ifdef CONFIG_VMAP_STACK
mtcr r10
-FTR_SECTION_ELSE
- stw r10, _CCR(r11)
-ALT_MMU_FTR_SECTION_END_IFSET(MMU_FTR_HPTE_TABLE)
+ li r10, MSR_KERNEL & ~(MSR_IR | MSR_RI) /* can take DTLB miss */
+ mtmsr r10
+ isync
#else
stw r10,_CCR(r11) /* save registers */
#endif
@@ -77,11 +65,9 @@ ALT_MMU_FTR_SECTION_END_IFSET(MMU_FTR_HPTE_TABLE)
stw r12,GPR12(r11)
stw r9,GPR9(r11)
stw r10,GPR10(r11)
-#if defined(CONFIG_VMAP_STACK) && defined(CONFIG_PPC_BOOK3S)
-BEGIN_MMU_FTR_SECTION
+#ifdef CONFIG_VMAP_STACK
mfcr r10
stw r10, _CCR(r11)
-END_MMU_FTR_SECTION_IFSET(MMU_FTR_HPTE_TABLE)
#endif
mfspr r12,SPRN_SPRG_SCRATCH1
stw r12,GPR11(r11)
@@ -97,11 +83,7 @@ END_MMU_FTR_SECTION_IFSET(MMU_FTR_HPTE_TABLE)
stw r10, _DSISR(r11)
.endif
lwz r9, SRR1(r12)
-#if defined(CONFIG_VMAP_STACK) && defined(CONFIG_PPC_BOOK3S)
-BEGIN_MMU_FTR_SECTION
andi. r10, r9, MSR_PR
-END_MMU_FTR_SECTION_IFSET(MMU_FTR_HPTE_TABLE)
-#endif
lwz r12, SRR0(r12)
#else
mfspr r12,SPRN_SRR0
@@ -328,7 +310,6 @@ END_MMU_FTR_SECTION_IFSET(MMU_FTR_HPTE_TABLE)
#ifdef CONFIG_VMAP_STACK
#ifdef CONFIG_SMP
mfspr r11, SPRN_SPRG_THREAD
- tovirt(r11, r11)
lwz r11, TASK_CPU - THREAD(r11)
slwi r11, r11, 3
addis r11, r11, emergency_ctx@ha
--
2.25.0
^ permalink raw reply related
* [PATCH 2/2] powerpc/32: Fix vmap stack - Properly set r1 before activating MMU
From: Christophe Leroy @ 2020-09-07 13:42 UTC (permalink / raw)
To: Benjamin Herrenschmidt, Paul Mackerras, Michael Ellerman
Cc: linuxppc-dev, linux-kernel
In-Reply-To: <a027d447022a006c9c4958ac734128e577a3c5c1.1599486108.git.christophe.leroy@csgroup.eu>
We need r1 to be properly set before activating MMU, otherwise any new
exception taken while saving registers into the stack in exception
prologs will use the user stack, which is wrong and will even lockup
or crash when KUAP is selected.
Do that by switching the meaning of r11 and r1 until we have saved r1
to the stack: copy r1 into r11 and setup the new stack pointer in r1.
To avoid complicating and impacting all generic and specific prolog
code (and more), copy back r1 into r11 once r11 is save onto
the stack.
We could get rid of copying r1 back and forth at the cost of
rewriting everything to use r1 instead of r11 all the way when
CONFIG_VMAP_STACK is set, but the effort is probably not worth it.
Fixes: 028474876f47 ("powerpc/32: prepare for CONFIG_VMAP_STACK")
Cc: stable@vger.kernel.org
Signed-off-by: Christophe Leroy <christophe.leroy@csgroup.eu>
---
arch/powerpc/kernel/head_32.h | 43 +++++++++++++++++++++++------------
1 file changed, 29 insertions(+), 14 deletions(-)
diff --git a/arch/powerpc/kernel/head_32.h b/arch/powerpc/kernel/head_32.h
index 21effebb9277..cc36998c5541 100644
--- a/arch/powerpc/kernel/head_32.h
+++ b/arch/powerpc/kernel/head_32.h
@@ -39,15 +39,24 @@
.endm
.macro EXCEPTION_PROLOG_1 for_rtas=0
+#ifdef CONFIG_VMAP_STACK
+ mr r11, r1
+ subi r1, r1, INT_FRAME_SIZE /* use r1 if kernel */
+ beq 1f
+ mfspr r1,SPRN_SPRG_THREAD
+ lwz r1,TASK_STACK-THREAD(r1)
+ addi r1, r1, THREAD_SIZE - INT_FRAME_SIZE
+#else
subi r11, r1, INT_FRAME_SIZE /* use r1 if kernel */
beq 1f
mfspr r11,SPRN_SPRG_THREAD
lwz r11,TASK_STACK-THREAD(r11)
addi r11, r11, THREAD_SIZE - INT_FRAME_SIZE
+#endif
1:
tophys_novmstack r11, r11
#ifdef CONFIG_VMAP_STACK
- mtcrf 0x7f, r11
+ mtcrf 0x7f, r1
bt 32 - THREAD_ALIGN_SHIFT, stack_overflow
#endif
.endm
@@ -62,6 +71,15 @@
stw r10,_CCR(r11) /* save registers */
#endif
mfspr r10, SPRN_SPRG_SCRATCH0
+#ifdef CONFIG_VMAP_STACK
+ stw r11,GPR1(r1)
+ stw r11,0(r1)
+ mr r11, r1
+#else
+ stw r1,GPR1(r11)
+ stw r1,0(r11)
+ tovirt(r1, r11) /* set new kernel sp */
+#endif
stw r12,GPR12(r11)
stw r9,GPR9(r11)
stw r10,GPR10(r11)
@@ -89,9 +107,6 @@
mfspr r12,SPRN_SRR0
mfspr r9,SPRN_SRR1
#endif
- stw r1,GPR1(r11)
- stw r1,0(r11)
- tovirt_novmstack r1, r11 /* set new kernel sp */
#ifdef CONFIG_40x
rlwinm r9,r9,0,14,12 /* clear MSR_WE (necessary?) */
#else
@@ -309,19 +324,19 @@
.macro vmap_stack_overflow_exception
#ifdef CONFIG_VMAP_STACK
#ifdef CONFIG_SMP
- mfspr r11, SPRN_SPRG_THREAD
- lwz r11, TASK_CPU - THREAD(r11)
- slwi r11, r11, 3
- addis r11, r11, emergency_ctx@ha
+ mfspr r1, SPRN_SPRG_THREAD
+ lwz r1, TASK_CPU - THREAD(r1)
+ slwi r1, r1, 3
+ addis r1, r1, emergency_ctx@ha
#else
- lis r11, emergency_ctx@ha
+ lis r1, emergency_ctx@ha
#endif
- lwz r11, emergency_ctx@l(r11)
- cmpwi cr1, r11, 0
+ lwz r1, emergency_ctx@l(r1)
+ cmpwi cr1, r1, 0
bne cr1, 1f
- lis r11, init_thread_union@ha
- addi r11, r11, init_thread_union@l
-1: addi r11, r11, THREAD_SIZE - INT_FRAME_SIZE
+ lis r1, init_thread_union@ha
+ addi r1, r1, init_thread_union@l
+1: addi r1, r1, THREAD_SIZE - INT_FRAME_SIZE
EXCEPTION_PROLOG_2
SAVE_NVGPRS(r11)
addi r3, r1, STACK_FRAME_OVERHEAD
--
2.25.0
^ permalink raw reply related
* Re: fsl_espi errors on v5.7.15
From: Joakim Tjernlund @ 2020-09-07 15:38 UTC (permalink / raw)
To: mpe@ellerman.id.au, broonie@kernel.org, paulus@samba.org,
npiggin@gmail.com, Chris.Packham@alliedtelesis.co.nz,
benh@kernel.crashing.org, hkallweit1@gmail.com
Cc: linuxppc-dev@lists.ozlabs.org, linux-kernel@vger.kernel.org,
linux-spi@vger.kernel.org
In-Reply-To: <23d13439e4cc1872c29db2f93e715a61f4843943.camel@infinera.com>
[SNIP]
> >
> > > Would you be able to ftrace the interrupt handler function and see if you
> > > can see a difference in number or timing of interrupts? I'm at a bit of
> > > a loss.
> >
> > I tried ftrace but I really wasn't sure what I was looking for.
> > Capturing a "bad" case was pretty tricky. But I think I've identified a
> > fix (I'll send it as a proper patch shortly). The gist is
> >
> > diff --git a/drivers/spi/spi-fsl-espi.c b/drivers/spi/spi-fsl-espi.c
> > index 7e7c92cafdbb..cb120b68c0e2 100644
> > --- a/drivers/spi/spi-fsl-espi.c
> > +++ b/drivers/spi/spi-fsl-espi.c
> > @@ -574,13 +574,14 @@ static void fsl_espi_cpu_irq(struct fsl_espi
> > *espi, u32 events)
> > static irqreturn_t fsl_espi_irq(s32 irq, void *context_data)
> > {
> > struct fsl_espi *espi = context_data;
> > - u32 events;
> > + u32 events, mask;
> >
> > spin_lock(&espi->lock);
> >
> > /* Get interrupt events(tx/rx) */
> > events = fsl_espi_read_reg(espi, ESPI_SPIE);
> > - if (!events) {
> > + mask = fsl_espi_read_reg(espi, ESPI_SPIM);
> > + if (!(events & mask)) {
> > spin_unlock(&espi->lock);
> > return IRQ_NONE;
> > }
> >
> > The SPIE register contains the TXCNT so events is pretty much always
> > going to have something set. By checking events against what we've
> > actually requested interrupts for we don't see any spurious events.
> >
> > I've tested this on the T2080RDB and on our custom hardware and it seems
> > to resolve the problem.
> >
>
> I looked at the fsl_espi_irq() too and noticed that clearing of the IRQ events
> are after processing TX/RX. That looks a bit odd to me.
I should have been more specific. I think you can loose IRQs as fsl_espi_irq() works now.
Consider this:
1) You get TX IRQ and enter fsl_espi_irq()
2) Enter fsl_espi_fill_tx_fifo() to process any chars until done.
3) Now you get one more TX IRQ
4) fsl_espi_irq() clear events -> IRQ from 3) is lost.
Jocke
^ permalink raw reply
* [PATCH AUTOSEL 5.8 14/53] ibmvnic fix NULL tx_pools and rx_tools issue at do_reset
From: Sasha Levin @ 2020-09-07 16:31 UTC (permalink / raw)
To: linux-kernel, stable
Cc: Sasha Levin, netdev, Mingming Cao, Dany Madden, linuxppc-dev,
David S . Miller
In-Reply-To: <20200907163220.1280412-1-sashal@kernel.org>
From: Mingming Cao <mmc@linux.vnet.ibm.com>
[ Upstream commit 9f13457377907fa253aef560e1a37e1ca4197f9b ]
At the time of do_rest, ibmvnic tries to re-initalize the tx_pools
and rx_pools to avoid re-allocating the long term buffer. However
there is a window inside do_reset that the tx_pools and
rx_pools were freed before re-initialized making it possible to deference
null pointers.
This patch fix this issue by always check the tx_pool
and rx_pool are not NULL after ibmvnic_login. If so, re-allocating
the pools. This will avoid getting into calling reset_tx/rx_pools with
NULL adapter tx_pools/rx_pools pointer. Also add null pointer check in
reset_tx_pools and reset_rx_pools to safe handle NULL pointer case.
Signed-off-by: Mingming Cao <mmc@linux.vnet.ibm.com>
Signed-off-by: Dany Madden <drt@linux.ibm.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
Signed-off-by: Sasha Levin <sashal@kernel.org>
---
drivers/net/ethernet/ibm/ibmvnic.c | 15 ++++++++++++++-
1 file changed, 14 insertions(+), 1 deletion(-)
diff --git a/drivers/net/ethernet/ibm/ibmvnic.c b/drivers/net/ethernet/ibm/ibmvnic.c
index 5afb3c9c52d20..d3a774331afc7 100644
--- a/drivers/net/ethernet/ibm/ibmvnic.c
+++ b/drivers/net/ethernet/ibm/ibmvnic.c
@@ -479,6 +479,9 @@ static int reset_rx_pools(struct ibmvnic_adapter *adapter)
int i, j, rc;
u64 *size_array;
+ if (!adapter->rx_pool)
+ return -1;
+
size_array = (u64 *)((u8 *)(adapter->login_rsp_buf) +
be32_to_cpu(adapter->login_rsp_buf->off_rxadd_buff_size));
@@ -649,6 +652,9 @@ static int reset_tx_pools(struct ibmvnic_adapter *adapter)
int tx_scrqs;
int i, rc;
+ if (!adapter->tx_pool)
+ return -1;
+
tx_scrqs = be32_to_cpu(adapter->login_rsp_buf->num_txsubm_subcrqs);
for (i = 0; i < tx_scrqs; i++) {
rc = reset_one_tx_pool(adapter, &adapter->tso_pool[i]);
@@ -2011,7 +2017,10 @@ static int do_reset(struct ibmvnic_adapter *adapter,
adapter->req_rx_add_entries_per_subcrq !=
old_num_rx_slots ||
adapter->req_tx_entries_per_subcrq !=
- old_num_tx_slots) {
+ old_num_tx_slots ||
+ !adapter->rx_pool ||
+ !adapter->tso_pool ||
+ !adapter->tx_pool) {
release_rx_pools(adapter);
release_tx_pools(adapter);
release_napi(adapter);
@@ -2024,10 +2033,14 @@ static int do_reset(struct ibmvnic_adapter *adapter,
} else {
rc = reset_tx_pools(adapter);
if (rc)
+ netdev_dbg(adapter->netdev, "reset tx pools failed (%d)\n",
+ rc);
goto out;
rc = reset_rx_pools(adapter);
if (rc)
+ netdev_dbg(adapter->netdev, "reset rx pools failed (%d)\n",
+ rc);
goto out;
}
ibmvnic_disable_irqs(adapter);
--
2.25.1
^ permalink raw reply related
* [PATCH AUTOSEL 5.4 11/43] ibmvnic fix NULL tx_pools and rx_tools issue at do_reset
From: Sasha Levin @ 2020-09-07 16:32 UTC (permalink / raw)
To: linux-kernel, stable
Cc: Sasha Levin, netdev, Mingming Cao, Dany Madden, linuxppc-dev,
David S . Miller
In-Reply-To: <20200907163329.1280888-1-sashal@kernel.org>
From: Mingming Cao <mmc@linux.vnet.ibm.com>
[ Upstream commit 9f13457377907fa253aef560e1a37e1ca4197f9b ]
At the time of do_rest, ibmvnic tries to re-initalize the tx_pools
and rx_pools to avoid re-allocating the long term buffer. However
there is a window inside do_reset that the tx_pools and
rx_pools were freed before re-initialized making it possible to deference
null pointers.
This patch fix this issue by always check the tx_pool
and rx_pool are not NULL after ibmvnic_login. If so, re-allocating
the pools. This will avoid getting into calling reset_tx/rx_pools with
NULL adapter tx_pools/rx_pools pointer. Also add null pointer check in
reset_tx_pools and reset_rx_pools to safe handle NULL pointer case.
Signed-off-by: Mingming Cao <mmc@linux.vnet.ibm.com>
Signed-off-by: Dany Madden <drt@linux.ibm.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
Signed-off-by: Sasha Levin <sashal@kernel.org>
---
drivers/net/ethernet/ibm/ibmvnic.c | 15 ++++++++++++++-
1 file changed, 14 insertions(+), 1 deletion(-)
diff --git a/drivers/net/ethernet/ibm/ibmvnic.c b/drivers/net/ethernet/ibm/ibmvnic.c
index 2d20a48f0ba0a..de45b3709c14e 100644
--- a/drivers/net/ethernet/ibm/ibmvnic.c
+++ b/drivers/net/ethernet/ibm/ibmvnic.c
@@ -416,6 +416,9 @@ static int reset_rx_pools(struct ibmvnic_adapter *adapter)
int i, j, rc;
u64 *size_array;
+ if (!adapter->rx_pool)
+ return -1;
+
size_array = (u64 *)((u8 *)(adapter->login_rsp_buf) +
be32_to_cpu(adapter->login_rsp_buf->off_rxadd_buff_size));
@@ -586,6 +589,9 @@ static int reset_tx_pools(struct ibmvnic_adapter *adapter)
int tx_scrqs;
int i, rc;
+ if (!adapter->tx_pool)
+ return -1;
+
tx_scrqs = be32_to_cpu(adapter->login_rsp_buf->num_txsubm_subcrqs);
for (i = 0; i < tx_scrqs; i++) {
rc = reset_one_tx_pool(adapter, &adapter->tso_pool[i]);
@@ -1918,7 +1924,10 @@ static int do_reset(struct ibmvnic_adapter *adapter,
adapter->req_rx_add_entries_per_subcrq !=
old_num_rx_slots ||
adapter->req_tx_entries_per_subcrq !=
- old_num_tx_slots) {
+ old_num_tx_slots ||
+ !adapter->rx_pool ||
+ !adapter->tso_pool ||
+ !adapter->tx_pool) {
release_rx_pools(adapter);
release_tx_pools(adapter);
release_napi(adapter);
@@ -1931,10 +1940,14 @@ static int do_reset(struct ibmvnic_adapter *adapter,
} else {
rc = reset_tx_pools(adapter);
if (rc)
+ netdev_dbg(adapter->netdev, "reset tx pools failed (%d)\n",
+ rc);
goto out;
rc = reset_rx_pools(adapter);
if (rc)
+ netdev_dbg(adapter->netdev, "reset rx pools failed (%d)\n",
+ rc);
goto out;
}
ibmvnic_disable_irqs(adapter);
--
2.25.1
^ permalink raw reply related
* [RFC PATCH v2 3/3] mm: make generic pXd_addr_end() macros inline functions
From: Gerald Schaefer @ 2020-09-07 18:00 UTC (permalink / raw)
To: Jason Gunthorpe, John Hubbard
Cc: Peter Zijlstra, Dave Hansen, linux-mm, Paul Mackerras,
linux-sparc, Alexander Gordeev, Claudio Imbrenda, Will Deacon,
linux-arch, linux-s390, Vasily Gorbik, Richard Weinberger,
linux-x86, Russell King, Christian Borntraeger, Ingo Molnar,
Catalin Marinas, Andrey Ryabinin, Heiko Carstens, Arnd Bergmann,
Jeff Dike, linux-um, Borislav Petkov, Andy Lutomirski,
Thomas Gleixner, linux-arm, linux-power, LKML, Andrew Morton,
Linus Torvalds, Mike Rapoport
In-Reply-To: <20200907180058.64880-1-gerald.schaefer@linux.ibm.com>
From: Alexander Gordeev <agordeev@linux.ibm.com>
Since pXd_addr_end() macros take pXd page-table entry as a
parameter it makes sense to check the entry type on compile.
Even though most archs do not make use of page-table entries
in pXd_addr_end() calls, checking the type in traversal code
paths could help to avoid subtle bugs.
Signed-off-by: Alexander Gordeev <agordeev@linux.ibm.com>
Signed-off-by: Gerald Schaefer <gerald.schaefer@linux.ibm.com>
---
include/linux/pgtable.h | 36 ++++++++++++++++++++----------------
1 file changed, 20 insertions(+), 16 deletions(-)
diff --git a/include/linux/pgtable.h b/include/linux/pgtable.h
index 67ebc22cf83d..d9e7d16c2263 100644
--- a/include/linux/pgtable.h
+++ b/include/linux/pgtable.h
@@ -656,31 +656,35 @@ static inline int arch_unmap_one(struct mm_struct *mm,
*/
#ifndef pgd_addr_end
-#define pgd_addr_end(pgd, addr, end) \
-({ unsigned long __boundary = ((addr) + PGDIR_SIZE) & PGDIR_MASK; \
- (__boundary - 1 < (end) - 1)? __boundary: (end); \
-})
+#define pgd_addr_end pgd_addr_end
+static inline unsigned long pgd_addr_end(pgd_t pgd, unsigned long addr, unsigned long end)
+{ unsigned long __boundary = (addr + PGDIR_SIZE) & PGDIR_MASK;
+ return (__boundary - 1 < end - 1) ? __boundary : end;
+}
#endif
#ifndef p4d_addr_end
-#define p4d_addr_end(p4d, addr, end) \
-({ unsigned long __boundary = ((addr) + P4D_SIZE) & P4D_MASK; \
- (__boundary - 1 < (end) - 1)? __boundary: (end); \
-})
+#define p4d_addr_end p4d_addr_end
+static inline unsigned long p4d_addr_end(p4d_t p4d, unsigned long addr, unsigned long end)
+{ unsigned long __boundary = (addr + P4D_SIZE) & P4D_MASK;
+ return (__boundary - 1 < end - 1) ? __boundary : end;
+}
#endif
#ifndef pud_addr_end
-#define pud_addr_end(pud, addr, end) \
-({ unsigned long __boundary = ((addr) + PUD_SIZE) & PUD_MASK; \
- (__boundary - 1 < (end) - 1)? __boundary: (end); \
-})
+#define pud_addr_end pud_addr_end
+static inline unsigned long pud_addr_end(pud_t pud, unsigned long addr, unsigned long end)
+{ unsigned long __boundary = (addr + PUD_SIZE) & PUD_MASK;
+ return (__boundary - 1 < end - 1) ? __boundary : end;
+}
#endif
#ifndef pmd_addr_end
-#define pmd_addr_end(pmd, addr, end) \
-({ unsigned long __boundary = ((addr) + PMD_SIZE) & PMD_MASK; \
- (__boundary - 1 < (end) - 1)? __boundary: (end); \
-})
+#define pmd_addr_end pmd_addr_end
+static inline unsigned long pmd_addr_end(pmd_t pmd, unsigned long addr, unsigned long end)
+{ unsigned long __boundary = (addr + PMD_SIZE) & PMD_MASK;
+ return (__boundary - 1 < end - 1) ? __boundary : end;
+}
#endif
/*
--
2.17.1
^ permalink raw reply related
* [RFC PATCH v2 1/3] mm/gup: fix gup_fast with dynamic page table folding
From: Gerald Schaefer @ 2020-09-07 18:00 UTC (permalink / raw)
To: Jason Gunthorpe, John Hubbard
Cc: Peter Zijlstra, Dave Hansen, linux-mm, Paul Mackerras,
linux-sparc, Alexander Gordeev, Claudio Imbrenda, Will Deacon,
linux-arch, linux-s390, Vasily Gorbik, Richard Weinberger,
linux-x86, Russell King, Christian Borntraeger, Ingo Molnar,
Catalin Marinas, Andrey Ryabinin, Heiko Carstens, Arnd Bergmann,
Jeff Dike, linux-um, Borislav Petkov, Andy Lutomirski,
Thomas Gleixner, linux-arm, linux-power, LKML, Andrew Morton,
Linus Torvalds, Mike Rapoport
In-Reply-To: <20200907180058.64880-1-gerald.schaefer@linux.ibm.com>
From: Alexander Gordeev <agordeev@linux.ibm.com>
Commit 1a42010cdc26 ("s390/mm: convert to the generic get_user_pages_fast
code") introduced a subtle but severe bug on s390 with gup_fast, due to
dynamic page table folding.
The question "What would it require for the generic code to work for s390"
has already been discussed here
https://lkml.kernel.org/r/20190418100218.0a4afd51@mschwideX1
and ended with a promising approach here
https://lkml.kernel.org/r/20190419153307.4f2911b5@mschwideX1
which in the end unfortunately didn't quite work completely.
We tried to mimic static level folding by changing pgd_offset to always
calculate top level page table offset, and do nothing in folded pXd_offset.
What has been overlooked is that PxD_SIZE/MASK and thus pXd_addr_end do
not reflect this dynamic behaviour, and still act like static 5-level
page tables.
Here is an example of what happens with gup_fast on s390, for a task with
3-levels paging, crossing a 2 GB pud boundary:
// addr = 0x1007ffff000, end = 0x10080001000
static int gup_pud_range(p4d_t p4d, unsigned long addr, unsigned long end,
unsigned int flags, struct page **pages, int *nr)
{
unsigned long next;
pud_t *pudp;
// pud_offset returns &p4d itself (a pointer to a value on stack)
pudp = pud_offset(&p4d, addr);
do {
// on second iteratation reading "random" stack value
pud_t pud = READ_ONCE(*pudp);
// next = 0x10080000000, due to PUD_SIZE/MASK != PGDIR_SIZE/MASK on s390
next = pud_addr_end(addr, end);
...
} while (pudp++, addr = next, addr != end); // pudp++ iterating over stack
return 1;
}
pud_addr_end = 0x10080000000 is correct, but the previous pgd/p4d_addr_end
should also have returned that limit, instead of the 5-level static
pgd/p4d limits with PUD_SIZE/MASK != PGDIR_SIZE/MASK. Then the "end"
parameter for gup_pud_range would also have been 0x10080000000, and we
would not iterate further in gup_pud_range, but rather go back and
(correctly) do it in gup_pgd_range.
So, for the second iteration in gup_pud_range, we will increase pudp,
which pointed to a stack value and not the real pud table. This new pudp
will then point to whatever lies behind the p4d stack value. In general,
this happens to be the previously read pgd, but it probably could also
be something different, depending on compiler decisions.
Most unfortunately, if it happens to be the pgd value, which is the
same as the p4d / pud due to folding, it is a valid and present entry.
So after the increment, we would still point to the same pud entry.
The addr however has been increased in the second iteration, so that we
now have different pmd/pte_index values, which will result in very wrong
behaviour for the remaining gup_pmd/pte_range calls. We will effectively
operate on an address minus 2 GB, due to missing pudp increase.
In the "good case", if nothing is mapped there, we will fall back to
the slow gup path. But if something is mapped there, and valid
for gup_fast, we will end up (silently) getting references on the wrong
pages and also add the wrong pages to the **pages result array. This
can cause data corruption.
Fix this by introducing new pXd_addr_end_folded helpers, which take an
additional pXd entry value parameter, that can be used on s390
to determine the correct page table level and return corresponding
end / boundary. With that, the pointer iteration will always
happen in gup_pgd_range for s390. No change for other architectures
introduced.
Fixes: 1a42010cdc26 ("s390/mm: convert to the generic get_user_pages_fast code")
Cc: <stable@vger.kernel.org> # 5.2+
Reviewed-by: Gerald Schaefer <gerald.schaefer@linux.ibm.com>
Signed-off-by: Alexander Gordeev <agordeev@linux.ibm.com>
Signed-off-by: Gerald Schaefer <gerald.schaefer@linux.ibm.com>
---
arch/s390/include/asm/pgtable.h | 42 +++++++++++++++++++++++++++++++++
include/linux/pgtable.h | 16 +++++++++++++
mm/gup.c | 8 +++----
3 files changed, 62 insertions(+), 4 deletions(-)
diff --git a/arch/s390/include/asm/pgtable.h b/arch/s390/include/asm/pgtable.h
index 7eb01a5459cd..027206e4959d 100644
--- a/arch/s390/include/asm/pgtable.h
+++ b/arch/s390/include/asm/pgtable.h
@@ -512,6 +512,48 @@ static inline bool mm_pmd_folded(struct mm_struct *mm)
}
#define mm_pmd_folded(mm) mm_pmd_folded(mm)
+/*
+ * With dynamic page table levels on s390, the static pXd_addr_end() functions
+ * will not return corresponding dynamic boundaries. This is no problem as long
+ * as only pXd pointers are passed down during page table walk, because
+ * pXd_offset() will simply return the given pointer for folded levels, and the
+ * pointer iteration over a range simply happens at the correct page table
+ * level.
+ * It is however a problem with gup_fast, or other places walking the page
+ * tables w/o locks using READ_ONCE(), and passing down the pXd values instead
+ * of pointers. In this case, the pointer given to pXd_offset() is a pointer to
+ * a stack variable, which cannot be used for pointer iteration at the correct
+ * level. Instead, the iteration then has to happen by going up to pgd level
+ * again. To allow this, provide pXd_addr_end_folded() functions with an
+ * additional pXd value parameter, which can be used on s390 to determine the
+ * folding level and return the corresponding boundary.
+ */
+static inline unsigned long rste_addr_end_folded(unsigned long rste, unsigned long addr, unsigned long end)
+{
+ unsigned long type = (rste & _REGION_ENTRY_TYPE_MASK) >> 2;
+ unsigned long size = 1UL << (_SEGMENT_SHIFT + type * 11);
+ unsigned long boundary = (addr + size) & ~(size - 1);
+
+ /*
+ * FIXME The below check is for internal testing only, to be removed
+ */
+ VM_BUG_ON(type < (_REGION_ENTRY_TYPE_R3 >> 2));
+
+ return (boundary - 1) < (end - 1) ? boundary : end;
+}
+
+#define pgd_addr_end_folded pgd_addr_end_folded
+static inline unsigned long pgd_addr_end_folded(pgd_t pgd, unsigned long addr, unsigned long end)
+{
+ return rste_addr_end_folded(pgd_val(pgd), addr, end);
+}
+
+#define p4d_addr_end_folded p4d_addr_end_folded
+static inline unsigned long p4d_addr_end_folded(p4d_t p4d, unsigned long addr, unsigned long end)
+{
+ return rste_addr_end_folded(p4d_val(p4d), addr, end);
+}
+
static inline int mm_has_pgste(struct mm_struct *mm)
{
#ifdef CONFIG_PGSTE
diff --git a/include/linux/pgtable.h b/include/linux/pgtable.h
index e8cbc2e795d5..981c4c2a31fe 100644
--- a/include/linux/pgtable.h
+++ b/include/linux/pgtable.h
@@ -681,6 +681,22 @@ static inline int arch_unmap_one(struct mm_struct *mm,
})
#endif
+#ifndef pgd_addr_end_folded
+#define pgd_addr_end_folded(pgd, addr, end) pgd_addr_end(addr, end)
+#endif
+
+#ifndef p4d_addr_end_folded
+#define p4d_addr_end_folded(p4d, addr, end) p4d_addr_end(addr, end)
+#endif
+
+#ifndef pud_addr_end_folded
+#define pud_addr_end_folded(pud, addr, end) pud_addr_end(addr, end)
+#endif
+
+#ifndef pmd_addr_end_folded
+#define pmd_addr_end_folded(pmd, addr, end) pmd_addr_end(addr, end)
+#endif
+
/*
* When walking page tables, we usually want to skip any p?d_none entries;
* and any p?d_bad entries - reporting the error before resetting to none.
diff --git a/mm/gup.c b/mm/gup.c
index bd883a112724..ba4aace5d0f4 100644
--- a/mm/gup.c
+++ b/mm/gup.c
@@ -2521,7 +2521,7 @@ static int gup_pmd_range(pud_t pud, unsigned long addr, unsigned long end,
do {
pmd_t pmd = READ_ONCE(*pmdp);
- next = pmd_addr_end(addr, end);
+ next = pmd_addr_end_folded(pmd, addr, end);
if (!pmd_present(pmd))
return 0;
@@ -2564,7 +2564,7 @@ static int gup_pud_range(p4d_t p4d, unsigned long addr, unsigned long end,
do {
pud_t pud = READ_ONCE(*pudp);
- next = pud_addr_end(addr, end);
+ next = pud_addr_end_folded(pud, addr, end);
if (unlikely(!pud_present(pud)))
return 0;
if (unlikely(pud_huge(pud))) {
@@ -2592,7 +2592,7 @@ static int gup_p4d_range(pgd_t pgd, unsigned long addr, unsigned long end,
do {
p4d_t p4d = READ_ONCE(*p4dp);
- next = p4d_addr_end(addr, end);
+ next = p4d_addr_end_folded(p4d, addr, end);
if (p4d_none(p4d))
return 0;
BUILD_BUG_ON(p4d_huge(p4d));
@@ -2617,7 +2617,7 @@ static void gup_pgd_range(unsigned long addr, unsigned long end,
do {
pgd_t pgd = READ_ONCE(*pgdp);
- next = pgd_addr_end(addr, end);
+ next = pgd_addr_end_folded(pgd, addr, end);
if (pgd_none(pgd))
return;
if (unlikely(pgd_huge(pgd))) {
--
2.17.1
^ permalink raw reply related
* [RFC PATCH v2 2/3] mm: make pXd_addr_end() functions page-table entry aware
From: Gerald Schaefer @ 2020-09-07 18:00 UTC (permalink / raw)
To: Jason Gunthorpe, John Hubbard
Cc: Peter Zijlstra, Dave Hansen, linux-mm, Paul Mackerras,
linux-sparc, Alexander Gordeev, Claudio Imbrenda, Will Deacon,
linux-arch, linux-s390, Vasily Gorbik, Richard Weinberger,
linux-x86, Russell King, Christian Borntraeger, Ingo Molnar,
Catalin Marinas, Andrey Ryabinin, Heiko Carstens, Arnd Bergmann,
Jeff Dike, linux-um, Borislav Petkov, Andy Lutomirski,
Thomas Gleixner, linux-arm, linux-power, LKML, Andrew Morton,
Linus Torvalds, Mike Rapoport
In-Reply-To: <20200907180058.64880-1-gerald.schaefer@linux.ibm.com>
From: Alexander Gordeev <agordeev@linux.ibm.com>
Unlike all other page-table abstractions pXd_addr_end() do not take
into account a particular table entry in which context the functions
are called. On architectures with dynamic page-tables folding that
might lead to lack of necessary information that is difficult to
obtain other than from the table entry itself. That already led to
a subtle memory corruption issue on s390.
By letting pXd_addr_end() functions know about the page-table entry
we allow archs not only make extra checks, but also optimizations.
As result of this change the pXd_addr_end_folded() functions used
in gup_fast traversal code become unnecessary and get replaced with
universal pXd_addr_end() variants.
The arch-specific updates not only add dereferencing of page-table
entry pointers, but also small changes to the code flow to make those
dereferences possible, at least for x86 and powerpc. Also for arm64,
but in way that should not have any impact.
So, even though the dereferenced page-table entries are not used on
archs other than s390, and are optimized out by the compiler, there
is a small change in kernel size and this is what bloat-o-meter reports:
x86:
add/remove: 0/0 grow/shrink: 2/0 up/down: 10/0 (10)
Function old new delta
vmemmap_populate 587 592 +5
munlock_vma_pages_range 556 561 +5
Total: Before=15534694, After=15534704, chg +0.00%
powerpc:
add/remove: 0/0 grow/shrink: 1/0 up/down: 4/0 (4)
Function old new delta
.remove_pagetable 1648 1652 +4
Total: Before=21478240, After=21478244, chg +0.00%
arm64:
add/remove: 0/0 grow/shrink: 0/0 up/down: 0/0 (0)
Function old new delta
Total: Before=20240851, After=20240851, chg +0.00%
sparc:
add/remove: 0/0 grow/shrink: 0/0 up/down: 0/0 (0)
Function old new delta
Total: Before=4907262, After=4907262, chg +0.00%
Signed-off-by: Alexander Gordeev <agordeev@linux.ibm.com>
Signed-off-by: Gerald Schaefer <gerald.schaefer@linux.ibm.com>
---
arch/arm/include/asm/pgtable-2level.h | 2 +-
arch/arm/mm/idmap.c | 6 ++--
arch/arm/mm/mmu.c | 8 ++---
arch/arm64/kernel/hibernate.c | 16 ++++++----
arch/arm64/kvm/mmu.c | 16 +++++-----
arch/arm64/mm/kasan_init.c | 8 ++---
arch/arm64/mm/mmu.c | 25 +++++++--------
arch/powerpc/mm/book3s64/radix_pgtable.c | 7 ++---
arch/powerpc/mm/hugetlbpage.c | 6 ++--
arch/s390/include/asm/pgtable.h | 8 ++---
arch/s390/mm/page-states.c | 8 ++---
arch/s390/mm/pageattr.c | 8 ++---
arch/s390/mm/vmem.c | 8 ++---
arch/sparc/mm/hugetlbpage.c | 6 ++--
arch/um/kernel/tlb.c | 8 ++---
arch/x86/mm/init_64.c | 15 ++++-----
arch/x86/mm/kasan_init_64.c | 16 +++++-----
include/asm-generic/pgtable-nop4d.h | 2 +-
include/asm-generic/pgtable-nopmd.h | 2 +-
include/asm-generic/pgtable-nopud.h | 2 +-
include/linux/pgtable.h | 26 ++++-----------
mm/gup.c | 8 ++---
mm/ioremap.c | 8 ++---
mm/kasan/init.c | 17 +++++-----
mm/madvise.c | 4 +--
mm/memory.c | 40 ++++++++++++------------
mm/mlock.c | 18 ++++++++---
mm/mprotect.c | 8 ++---
mm/pagewalk.c | 8 ++---
mm/swapfile.c | 8 ++---
mm/vmalloc.c | 16 +++++-----
31 files changed, 165 insertions(+), 173 deletions(-)
diff --git a/arch/arm/include/asm/pgtable-2level.h b/arch/arm/include/asm/pgtable-2level.h
index 3502c2f746ca..5e6416b339f4 100644
--- a/arch/arm/include/asm/pgtable-2level.h
+++ b/arch/arm/include/asm/pgtable-2level.h
@@ -209,7 +209,7 @@ static inline pmd_t *pmd_offset(pud_t *pud, unsigned long addr)
} while (0)
/* we don't need complex calculations here as the pmd is folded into the pgd */
-#define pmd_addr_end(addr,end) (end)
+#define pmd_addr_end(pmd,addr,end) (end)
#define set_pte_ext(ptep,pte,ext) cpu_set_pte_ext(ptep,pte,ext)
diff --git a/arch/arm/mm/idmap.c b/arch/arm/mm/idmap.c
index 448e57c6f653..5437f943ca8b 100644
--- a/arch/arm/mm/idmap.c
+++ b/arch/arm/mm/idmap.c
@@ -46,7 +46,7 @@ static void idmap_add_pmd(pud_t *pud, unsigned long addr, unsigned long end,
pmd = pmd_offset(pud, addr);
do {
- next = pmd_addr_end(addr, end);
+ next = pmd_addr_end(*pmd, addr, end);
*pmd = __pmd((addr & PMD_MASK) | prot);
flush_pmd_entry(pmd);
} while (pmd++, addr = next, addr != end);
@@ -73,7 +73,7 @@ static void idmap_add_pud(pgd_t *pgd, unsigned long addr, unsigned long end,
unsigned long next;
do {
- next = pud_addr_end(addr, end);
+ next = pud_addr_end(*pud, addr, end);
idmap_add_pmd(pud, addr, next, prot);
} while (pud++, addr = next, addr != end);
}
@@ -95,7 +95,7 @@ static void identity_mapping_add(pgd_t *pgd, const char *text_start,
pgd += pgd_index(addr);
do {
- next = pgd_addr_end(addr, end);
+ next = pgd_addr_end(*pgd, addr, end);
idmap_add_pud(pgd, addr, next, prot);
} while (pgd++, addr = next, addr != end);
}
diff --git a/arch/arm/mm/mmu.c b/arch/arm/mm/mmu.c
index 698cc740c6b8..4013746e4c75 100644
--- a/arch/arm/mm/mmu.c
+++ b/arch/arm/mm/mmu.c
@@ -777,7 +777,7 @@ static void __init alloc_init_pmd(pud_t *pud, unsigned long addr,
* With LPAE, we must loop over to map
* all the pmds for the given range.
*/
- next = pmd_addr_end(addr, end);
+ next = pmd_addr_end(*pmd, addr, end);
/*
* Try a section mapping - addr, next and phys must all be
@@ -805,7 +805,7 @@ static void __init alloc_init_pud(p4d_t *p4d, unsigned long addr,
unsigned long next;
do {
- next = pud_addr_end(addr, end);
+ next = pud_addr_end(*pud, addr, end);
alloc_init_pmd(pud, addr, next, phys, type, alloc, ng);
phys += next - addr;
} while (pud++, addr = next, addr != end);
@@ -820,7 +820,7 @@ static void __init alloc_init_p4d(pgd_t *pgd, unsigned long addr,
unsigned long next;
do {
- next = p4d_addr_end(addr, end);
+ next = p4d_addr_end(*p4d, addr, end);
alloc_init_pud(p4d, addr, next, phys, type, alloc, ng);
phys += next - addr;
} while (p4d++, addr = next, addr != end);
@@ -923,7 +923,7 @@ static void __init __create_mapping(struct mm_struct *mm, struct map_desc *md,
pgd = pgd_offset(mm, addr);
end = addr + length;
do {
- unsigned long next = pgd_addr_end(addr, end);
+ unsigned long next = pgd_addr_end(*pgd, addr, end);
alloc_init_p4d(pgd, addr, next, phys, type, alloc, ng);
diff --git a/arch/arm64/kernel/hibernate.c b/arch/arm64/kernel/hibernate.c
index 68e14152d6e9..7be8c9cdc5c8 100644
--- a/arch/arm64/kernel/hibernate.c
+++ b/arch/arm64/kernel/hibernate.c
@@ -412,7 +412,7 @@ static int copy_pmd(pud_t *dst_pudp, pud_t *src_pudp, unsigned long start,
do {
pmd_t pmd = READ_ONCE(*src_pmdp);
- next = pmd_addr_end(addr, end);
+ next = pmd_addr_end(pmd, addr, end);
if (pmd_none(pmd))
continue;
if (pmd_table(pmd)) {
@@ -447,7 +447,7 @@ static int copy_pud(p4d_t *dst_p4dp, p4d_t *src_p4dp, unsigned long start,
do {
pud_t pud = READ_ONCE(*src_pudp);
- next = pud_addr_end(addr, end);
+ next = pud_addr_end(pud, addr, end);
if (pud_none(pud))
continue;
if (pud_table(pud)) {
@@ -473,8 +473,10 @@ static int copy_p4d(pgd_t *dst_pgdp, pgd_t *src_pgdp, unsigned long start,
dst_p4dp = p4d_offset(dst_pgdp, start);
src_p4dp = p4d_offset(src_pgdp, start);
do {
- next = p4d_addr_end(addr, end);
- if (p4d_none(READ_ONCE(*src_p4dp)))
+ p4d_t p4d = READ_ONCE(*src_p4dp);
+
+ next = p4d_addr_end(p4d, addr, end);
+ if (p4d_none(p4d))
continue;
if (copy_pud(dst_p4dp, src_p4dp, addr, next))
return -ENOMEM;
@@ -492,8 +494,10 @@ static int copy_page_tables(pgd_t *dst_pgdp, unsigned long start,
dst_pgdp = pgd_offset_pgd(dst_pgdp, start);
do {
- next = pgd_addr_end(addr, end);
- if (pgd_none(READ_ONCE(*src_pgdp)))
+ pgd_t pgd = READ_ONCE(*src_pgdp);
+
+ next = pgd_addr_end(pgd, addr, end);
+ if (pgd_none(pgd))
continue;
if (copy_p4d(dst_pgdp, src_pgdp, addr, next))
return -ENOMEM;
diff --git a/arch/arm64/kvm/mmu.c b/arch/arm64/kvm/mmu.c
index ba00bcc0c884..8f470f93a8e9 100644
--- a/arch/arm64/kvm/mmu.c
+++ b/arch/arm64/kvm/mmu.c
@@ -547,7 +547,7 @@ static void unmap_hyp_pmds(pud_t *pud, phys_addr_t addr, phys_addr_t end)
start_pmd = pmd = pmd_offset(pud, addr);
do {
- next = pmd_addr_end(addr, end);
+ next = pmd_addr_end(*pmd, addr, end);
/* Hyp doesn't use huge pmds */
if (!pmd_none(*pmd))
unmap_hyp_ptes(pmd, addr, next);
@@ -564,7 +564,7 @@ static void unmap_hyp_puds(p4d_t *p4d, phys_addr_t addr, phys_addr_t end)
start_pud = pud = pud_offset(p4d, addr);
do {
- next = pud_addr_end(addr, end);
+ next = pud_addr_end(*pud, addr, end);
/* Hyp doesn't use huge puds */
if (!pud_none(*pud))
unmap_hyp_pmds(pud, addr, next);
@@ -581,7 +581,7 @@ static void unmap_hyp_p4ds(pgd_t *pgd, phys_addr_t addr, phys_addr_t end)
start_p4d = p4d = p4d_offset(pgd, addr);
do {
- next = p4d_addr_end(addr, end);
+ next = p4d_addr_end(*p4d, addr, end);
/* Hyp doesn't use huge p4ds */
if (!p4d_none(*p4d))
unmap_hyp_puds(p4d, addr, next);
@@ -609,7 +609,7 @@ static void __unmap_hyp_range(pgd_t *pgdp, unsigned long ptrs_per_pgd,
*/
pgd = pgdp + kvm_pgd_index(addr, ptrs_per_pgd);
do {
- next = pgd_addr_end(addr, end);
+ next = pgd_addr_end(*pgd, addr, end);
if (!pgd_none(*pgd))
unmap_hyp_p4ds(pgd, addr, next);
} while (pgd++, addr = next, addr != end);
@@ -712,7 +712,7 @@ static int create_hyp_pmd_mappings(pud_t *pud, unsigned long start,
get_page(virt_to_page(pmd));
}
- next = pmd_addr_end(addr, end);
+ next = pmd_addr_end(*pmd, addr, end);
create_hyp_pte_mappings(pmd, addr, next, pfn, prot);
pfn += (next - addr) >> PAGE_SHIFT;
@@ -744,7 +744,7 @@ static int create_hyp_pud_mappings(p4d_t *p4d, unsigned long start,
get_page(virt_to_page(pud));
}
- next = pud_addr_end(addr, end);
+ next = pud_addr_end(*pud, addr, end);
ret = create_hyp_pmd_mappings(pud, addr, next, pfn, prot);
if (ret)
return ret;
@@ -777,7 +777,7 @@ static int create_hyp_p4d_mappings(pgd_t *pgd, unsigned long start,
get_page(virt_to_page(p4d));
}
- next = p4d_addr_end(addr, end);
+ next = p4d_addr_end(*p4d, addr, end);
ret = create_hyp_pud_mappings(p4d, addr, next, pfn, prot);
if (ret)
return ret;
@@ -813,7 +813,7 @@ static int __create_hyp_mappings(pgd_t *pgdp, unsigned long ptrs_per_pgd,
get_page(virt_to_page(pgd));
}
- next = pgd_addr_end(addr, end);
+ next = pgd_addr_end(*pgd, addr, end);
err = create_hyp_p4d_mappings(pgd, addr, next, pfn, prot);
if (err)
goto out;
diff --git a/arch/arm64/mm/kasan_init.c b/arch/arm64/mm/kasan_init.c
index b24e43d20667..8d1c811fd59e 100644
--- a/arch/arm64/mm/kasan_init.c
+++ b/arch/arm64/mm/kasan_init.c
@@ -120,7 +120,7 @@ static void __init kasan_pmd_populate(pud_t *pudp, unsigned long addr,
pmd_t *pmdp = kasan_pmd_offset(pudp, addr, node, early);
do {
- next = pmd_addr_end(addr, end);
+ next = pmd_addr_end(*pmdp, addr, end);
kasan_pte_populate(pmdp, addr, next, node, early);
} while (pmdp++, addr = next, addr != end && pmd_none(READ_ONCE(*pmdp)));
}
@@ -132,7 +132,7 @@ static void __init kasan_pud_populate(p4d_t *p4dp, unsigned long addr,
pud_t *pudp = kasan_pud_offset(p4dp, addr, node, early);
do {
- next = pud_addr_end(addr, end);
+ next = pud_addr_end(*pudp, addr, end);
kasan_pmd_populate(pudp, addr, next, node, early);
} while (pudp++, addr = next, addr != end && pud_none(READ_ONCE(*pudp)));
}
@@ -144,7 +144,7 @@ static void __init kasan_p4d_populate(pgd_t *pgdp, unsigned long addr,
p4d_t *p4dp = p4d_offset(pgdp, addr);
do {
- next = p4d_addr_end(addr, end);
+ next = p4d_addr_end(*p4dp, addr, end);
kasan_pud_populate(p4dp, addr, next, node, early);
} while (p4dp++, addr = next, addr != end);
}
@@ -157,7 +157,7 @@ static void __init kasan_pgd_populate(unsigned long addr, unsigned long end,
pgdp = pgd_offset_k(addr);
do {
- next = pgd_addr_end(addr, end);
+ next = pgd_addr_end(*pgdp, addr, end);
kasan_p4d_populate(pgdp, addr, next, node, early);
} while (pgdp++, addr = next, addr != end);
}
diff --git a/arch/arm64/mm/mmu.c b/arch/arm64/mm/mmu.c
index 64211436629d..d679cf024bc8 100644
--- a/arch/arm64/mm/mmu.c
+++ b/arch/arm64/mm/mmu.c
@@ -209,7 +209,7 @@ static void init_pmd(pud_t *pudp, unsigned long addr, unsigned long end,
do {
pmd_t old_pmd = READ_ONCE(*pmdp);
- next = pmd_addr_end(addr, end);
+ next = pmd_addr_end(old_pmd, addr, end);
/* try section mapping first */
if (((addr | next | phys) & ~SECTION_MASK) == 0 &&
@@ -307,7 +307,7 @@ static void alloc_init_pud(pgd_t *pgdp, unsigned long addr, unsigned long end,
do {
pud_t old_pud = READ_ONCE(*pudp);
- next = pud_addr_end(addr, end);
+ next = pud_addr_end(old_pud, addr, end);
/*
* For 4K granule only, attempt to put down a 1GB block
@@ -356,7 +356,7 @@ static void __create_pgd_mapping(pgd_t *pgdir, phys_addr_t phys,
end = PAGE_ALIGN(virt + size);
do {
- next = pgd_addr_end(addr, end);
+ next = pgd_addr_end(*pgdp, addr, end);
alloc_init_pud(pgdp, addr, next, phys, prot, pgtable_alloc,
flags);
phys += next - addr;
@@ -820,9 +820,9 @@ static void unmap_hotplug_pmd_range(pud_t *pudp, unsigned long addr,
pmd_t *pmdp, pmd;
do {
- next = pmd_addr_end(addr, end);
pmdp = pmd_offset(pudp, addr);
pmd = READ_ONCE(*pmdp);
+ next = pmd_addr_end(pmd, addr, end);
if (pmd_none(pmd))
continue;
@@ -853,9 +853,9 @@ static void unmap_hotplug_pud_range(p4d_t *p4dp, unsigned long addr,
pud_t *pudp, pud;
do {
- next = pud_addr_end(addr, end);
pudp = pud_offset(p4dp, addr);
pud = READ_ONCE(*pudp);
+ next = pud_addr_end(pud, addr, end);
if (pud_none(pud))
continue;
@@ -886,9 +886,9 @@ static void unmap_hotplug_p4d_range(pgd_t *pgdp, unsigned long addr,
p4d_t *p4dp, p4d;
do {
- next = p4d_addr_end(addr, end);
p4dp = p4d_offset(pgdp, addr);
p4d = READ_ONCE(*p4dp);
+ next = p4d_addr_end(p4d, addr, end);
if (p4d_none(p4d))
continue;
@@ -912,9 +912,9 @@ static void unmap_hotplug_range(unsigned long addr, unsigned long end,
WARN_ON(!free_mapped && altmap);
do {
- next = pgd_addr_end(addr, end);
pgdp = pgd_offset_k(addr);
pgd = READ_ONCE(*pgdp);
+ next = pgd_addr_end(pgd, addr, end);
if (pgd_none(pgd))
continue;
@@ -968,9 +968,9 @@ static void free_empty_pmd_table(pud_t *pudp, unsigned long addr,
unsigned long i, next, start = addr;
do {
- next = pmd_addr_end(addr, end);
pmdp = pmd_offset(pudp, addr);
pmd = READ_ONCE(*pmdp);
+ next = pmd_addr_end(pmd, addr, end);
if (pmd_none(pmd))
continue;
@@ -1008,9 +1008,9 @@ static void free_empty_pud_table(p4d_t *p4dp, unsigned long addr,
unsigned long i, next, start = addr;
do {
- next = pud_addr_end(addr, end);
pudp = pud_offset(p4dp, addr);
pud = READ_ONCE(*pudp);
+ next = pud_addr_end(pud, addr, end);
if (pud_none(pud))
continue;
@@ -1048,9 +1048,9 @@ static void free_empty_p4d_table(pgd_t *pgdp, unsigned long addr,
p4d_t *p4dp, p4d;
do {
- next = p4d_addr_end(addr, end);
p4dp = p4d_offset(pgdp, addr);
p4d = READ_ONCE(*p4dp);
+ next = p4d_addr_end(p4d, addr, end);
if (p4d_none(p4d))
continue;
@@ -1066,9 +1066,9 @@ static void free_empty_tables(unsigned long addr, unsigned long end,
pgd_t *pgdp, pgd;
do {
- next = pgd_addr_end(addr, end);
pgdp = pgd_offset_k(addr);
pgd = READ_ONCE(*pgdp);
+ next = pgd_addr_end(pgd, addr, end);
if (pgd_none(pgd))
continue;
@@ -1097,8 +1097,6 @@ int __meminit vmemmap_populate(unsigned long start, unsigned long end, int node,
pmd_t *pmdp;
do {
- next = pmd_addr_end(addr, end);
-
pgdp = vmemmap_pgd_populate(addr, node);
if (!pgdp)
return -ENOMEM;
@@ -1112,6 +1110,7 @@ int __meminit vmemmap_populate(unsigned long start, unsigned long end, int node,
return -ENOMEM;
pmdp = pmd_offset(pudp, addr);
+ next = pmd_addr_end(*pmdp, addr, end);
if (pmd_none(READ_ONCE(*pmdp))) {
void *p = NULL;
diff --git a/arch/powerpc/mm/book3s64/radix_pgtable.c b/arch/powerpc/mm/book3s64/radix_pgtable.c
index cc72666e891a..816e218df285 100644
--- a/arch/powerpc/mm/book3s64/radix_pgtable.c
+++ b/arch/powerpc/mm/book3s64/radix_pgtable.c
@@ -817,7 +817,7 @@ static void __meminit remove_pmd_table(pmd_t *pmd_start, unsigned long addr,
pmd = pmd_start + pmd_index(addr);
for (; addr < end; addr = next, pmd++) {
- next = pmd_addr_end(addr, end);
+ next = pmd_addr_end(*pmd, addr, end);
if (!pmd_present(*pmd))
continue;
@@ -847,7 +847,7 @@ static void __meminit remove_pud_table(pud_t *pud_start, unsigned long addr,
pud = pud_start + pud_index(addr);
for (; addr < end; addr = next, pud++) {
- next = pud_addr_end(addr, end);
+ next = pud_addr_end(*pud, addr, end);
if (!pud_present(*pud))
continue;
@@ -878,10 +878,9 @@ static void __meminit remove_pagetable(unsigned long start, unsigned long end)
spin_lock(&init_mm.page_table_lock);
for (addr = start; addr < end; addr = next) {
- next = pgd_addr_end(addr, end);
-
pgd = pgd_offset_k(addr);
p4d = p4d_offset(pgd, addr);
+ next = pgd_addr_end(*pgd, addr, end);
if (!p4d_present(*p4d))
continue;
diff --git a/arch/powerpc/mm/hugetlbpage.c b/arch/powerpc/mm/hugetlbpage.c
index 26292544630f..f0606d6774a4 100644
--- a/arch/powerpc/mm/hugetlbpage.c
+++ b/arch/powerpc/mm/hugetlbpage.c
@@ -352,7 +352,7 @@ static void hugetlb_free_pmd_range(struct mmu_gather *tlb, pud_t *pud,
unsigned long more;
pmd = pmd_offset(pud, addr);
- next = pmd_addr_end(addr, end);
+ next = pmd_addr_end(*pmd, addr, end);
if (!is_hugepd(__hugepd(pmd_val(*pmd)))) {
if (pmd_none_or_clear_bad(pmd))
continue;
@@ -409,7 +409,7 @@ static void hugetlb_free_pud_range(struct mmu_gather *tlb, p4d_t *p4d,
start = addr;
do {
pud = pud_offset(p4d, addr);
- next = pud_addr_end(addr, end);
+ next = pud_addr_end(*pud, addr, end);
if (!is_hugepd(__hugepd(pud_val(*pud)))) {
if (pud_none_or_clear_bad(pud))
continue;
@@ -478,9 +478,9 @@ void hugetlb_free_pgd_range(struct mmu_gather *tlb,
*/
do {
- next = pgd_addr_end(addr, end);
pgd = pgd_offset(tlb->mm, addr);
p4d = p4d_offset(pgd, addr);
+ next = pgd_addr_end(*pgd, addr, end);
if (!is_hugepd(__hugepd(pgd_val(*pgd)))) {
if (p4d_none_or_clear_bad(p4d))
continue;
diff --git a/arch/s390/include/asm/pgtable.h b/arch/s390/include/asm/pgtable.h
index 027206e4959d..6fb17ac413be 100644
--- a/arch/s390/include/asm/pgtable.h
+++ b/arch/s390/include/asm/pgtable.h
@@ -542,14 +542,14 @@ static inline unsigned long rste_addr_end_folded(unsigned long rste, unsigned lo
return (boundary - 1) < (end - 1) ? boundary : end;
}
-#define pgd_addr_end_folded pgd_addr_end_folded
-static inline unsigned long pgd_addr_end_folded(pgd_t pgd, unsigned long addr, unsigned long end)
+#define pgd_addr_end pgd_addr_end
+static inline unsigned long pgd_addr_end(pgd_t pgd, unsigned long addr, unsigned long end)
{
return rste_addr_end_folded(pgd_val(pgd), addr, end);
}
-#define p4d_addr_end_folded p4d_addr_end_folded
-static inline unsigned long p4d_addr_end_folded(p4d_t p4d, unsigned long addr, unsigned long end)
+#define p4d_addr_end p4d_addr_end
+static inline unsigned long p4d_addr_end(p4d_t p4d, unsigned long addr, unsigned long end)
{
return rste_addr_end_folded(p4d_val(p4d), addr, end);
}
diff --git a/arch/s390/mm/page-states.c b/arch/s390/mm/page-states.c
index 567c69f3069e..4aba634b4b26 100644
--- a/arch/s390/mm/page-states.c
+++ b/arch/s390/mm/page-states.c
@@ -109,7 +109,7 @@ static void mark_kernel_pmd(pud_t *pud, unsigned long addr, unsigned long end)
pmd = pmd_offset(pud, addr);
do {
- next = pmd_addr_end(addr, end);
+ next = pmd_addr_end(*pmd, addr, end);
if (pmd_none(*pmd) || pmd_large(*pmd))
continue;
page = virt_to_page(pmd_val(*pmd));
@@ -126,7 +126,7 @@ static void mark_kernel_pud(p4d_t *p4d, unsigned long addr, unsigned long end)
pud = pud_offset(p4d, addr);
do {
- next = pud_addr_end(addr, end);
+ next = pud_addr_end(*pud, addr, end);
if (pud_none(*pud) || pud_large(*pud))
continue;
if (!pud_folded(*pud)) {
@@ -147,7 +147,7 @@ static void mark_kernel_p4d(pgd_t *pgd, unsigned long addr, unsigned long end)
p4d = p4d_offset(pgd, addr);
do {
- next = p4d_addr_end(addr, end);
+ next = p4d_addr_end(*p4d, addr, end);
if (p4d_none(*p4d))
continue;
if (!p4d_folded(*p4d)) {
@@ -169,7 +169,7 @@ static void mark_kernel_pgd(void)
addr = 0;
pgd = pgd_offset_k(addr);
do {
- next = pgd_addr_end(addr, MODULES_END);
+ next = pgd_addr_end(*pgd, addr, MODULES_END);
if (pgd_none(*pgd))
continue;
if (!pgd_folded(*pgd)) {
diff --git a/arch/s390/mm/pageattr.c b/arch/s390/mm/pageattr.c
index c5c52ec2b46f..b827d758a17a 100644
--- a/arch/s390/mm/pageattr.c
+++ b/arch/s390/mm/pageattr.c
@@ -162,7 +162,7 @@ static int walk_pmd_level(pud_t *pudp, unsigned long addr, unsigned long end,
do {
if (pmd_none(*pmdp))
return -EINVAL;
- next = pmd_addr_end(addr, end);
+ next = pmd_addr_end(*pmdp, addr, end);
if (pmd_large(*pmdp)) {
if (addr & ~PMD_MASK || addr + PMD_SIZE > next) {
rc = split_pmd_page(pmdp, addr);
@@ -239,7 +239,7 @@ static int walk_pud_level(p4d_t *p4d, unsigned long addr, unsigned long end,
do {
if (pud_none(*pudp))
return -EINVAL;
- next = pud_addr_end(addr, end);
+ next = pud_addr_end(*pudp, addr, end);
if (pud_large(*pudp)) {
if (addr & ~PUD_MASK || addr + PUD_SIZE > next) {
rc = split_pud_page(pudp, addr);
@@ -269,7 +269,7 @@ static int walk_p4d_level(pgd_t *pgd, unsigned long addr, unsigned long end,
do {
if (p4d_none(*p4dp))
return -EINVAL;
- next = p4d_addr_end(addr, end);
+ next = p4d_addr_end(*p4dp, addr, end);
rc = walk_pud_level(p4dp, addr, next, flags);
p4dp++;
addr = next;
@@ -296,7 +296,7 @@ static int change_page_attr(unsigned long addr, unsigned long end,
do {
if (pgd_none(*pgdp))
break;
- next = pgd_addr_end(addr, end);
+ next = pgd_addr_end(*pgdp, addr, end);
rc = walk_p4d_level(pgdp, addr, next, flags);
if (rc)
break;
diff --git a/arch/s390/mm/vmem.c b/arch/s390/mm/vmem.c
index b239f2ba93b0..672bc89f13e7 100644
--- a/arch/s390/mm/vmem.c
+++ b/arch/s390/mm/vmem.c
@@ -219,7 +219,7 @@ static int __ref modify_pmd_table(pud_t *pud, unsigned long addr,
pmd = pmd_offset(pud, addr);
for (; addr < end; addr = next, pmd++) {
- next = pmd_addr_end(addr, end);
+ next = pmd_addr_end(*pmd, addr, end);
if (!add) {
if (pmd_none(*pmd))
continue;
@@ -320,7 +320,7 @@ static int modify_pud_table(p4d_t *p4d, unsigned long addr, unsigned long end,
prot &= ~_REGION_ENTRY_NOEXEC;
pud = pud_offset(p4d, addr);
for (; addr < end; addr = next, pud++) {
- next = pud_addr_end(addr, end);
+ next = pud_addr_end(*pud, addr, end);
if (!add) {
if (pud_none(*pud))
continue;
@@ -394,7 +394,7 @@ static int modify_p4d_table(pgd_t *pgd, unsigned long addr, unsigned long end,
p4d = p4d_offset(pgd, addr);
for (; addr < end; addr = next, p4d++) {
- next = p4d_addr_end(addr, end);
+ next = p4d_addr_end(*p4d, addr, end);
if (!add) {
if (p4d_none(*p4d))
continue;
@@ -449,8 +449,8 @@ static int modify_pagetable(unsigned long start, unsigned long end, bool add,
if (WARN_ON_ONCE(!PAGE_ALIGNED(start | end)))
return -EINVAL;
for (addr = start; addr < end; addr = next) {
- next = pgd_addr_end(addr, end);
pgd = pgd_offset_k(addr);
+ next = pgd_addr_end(*pgd, addr, end);
if (!add) {
if (pgd_none(*pgd))
diff --git a/arch/sparc/mm/hugetlbpage.c b/arch/sparc/mm/hugetlbpage.c
index ec423b5f17dd..341c2ff8d31a 100644
--- a/arch/sparc/mm/hugetlbpage.c
+++ b/arch/sparc/mm/hugetlbpage.c
@@ -428,7 +428,7 @@ static void hugetlb_free_pmd_range(struct mmu_gather *tlb, pud_t *pud,
start = addr;
pmd = pmd_offset(pud, addr);
do {
- next = pmd_addr_end(addr, end);
+ next = pmd_addr_end(*pmd, addr, end);
if (pmd_none(*pmd))
continue;
if (is_hugetlb_pmd(*pmd))
@@ -465,7 +465,7 @@ static void hugetlb_free_pud_range(struct mmu_gather *tlb, p4d_t *p4d,
start = addr;
pud = pud_offset(p4d, addr);
do {
- next = pud_addr_end(addr, end);
+ next = pud_addr_end(*pud, addr, end);
if (pud_none_or_clear_bad(pud))
continue;
if (is_hugetlb_pud(*pud))
@@ -519,7 +519,7 @@ void hugetlb_free_pgd_range(struct mmu_gather *tlb,
pgd = pgd_offset(tlb->mm, addr);
p4d = p4d_offset(pgd, addr);
do {
- next = p4d_addr_end(addr, end);
+ next = p4d_addr_end(*p4d, addr, end);
if (p4d_none_or_clear_bad(p4d))
continue;
hugetlb_free_pud_range(tlb, p4d, addr, next, floor, ceiling);
diff --git a/arch/um/kernel/tlb.c b/arch/um/kernel/tlb.c
index 61776790cd67..7b4fe31c8df2 100644
--- a/arch/um/kernel/tlb.c
+++ b/arch/um/kernel/tlb.c
@@ -264,7 +264,7 @@ static inline int update_pmd_range(pud_t *pud, unsigned long addr,
pmd = pmd_offset(pud, addr);
do {
- next = pmd_addr_end(addr, end);
+ next = pmd_addr_end(*pmd, addr, end);
if (!pmd_present(*pmd)) {
if (hvc->force || pmd_newpage(*pmd)) {
ret = add_munmap(addr, next - addr, hvc);
@@ -286,7 +286,7 @@ static inline int update_pud_range(p4d_t *p4d, unsigned long addr,
pud = pud_offset(p4d, addr);
do {
- next = pud_addr_end(addr, end);
+ next = pud_addr_end(*pud, addr, end);
if (!pud_present(*pud)) {
if (hvc->force || pud_newpage(*pud)) {
ret = add_munmap(addr, next - addr, hvc);
@@ -308,7 +308,7 @@ static inline int update_p4d_range(pgd_t *pgd, unsigned long addr,
p4d = p4d_offset(pgd, addr);
do {
- next = p4d_addr_end(addr, end);
+ next = p4d_addr_end(*p4d, addr, end);
if (!p4d_present(*p4d)) {
if (hvc->force || p4d_newpage(*p4d)) {
ret = add_munmap(addr, next - addr, hvc);
@@ -331,7 +331,7 @@ void fix_range_common(struct mm_struct *mm, unsigned long start_addr,
hvc = INIT_HVC(mm, force, userspace);
pgd = pgd_offset(mm, addr);
do {
- next = pgd_addr_end(addr, end_addr);
+ next = pgd_addr_end(*pgd, addr, end_addr);
if (!pgd_present(*pgd)) {
if (force || pgd_newpage(*pgd)) {
ret = add_munmap(addr, next - addr, &hvc);
diff --git a/arch/x86/mm/init_64.c b/arch/x86/mm/init_64.c
index a4ac13cc3fdc..e2cb9316a104 100644
--- a/arch/x86/mm/init_64.c
+++ b/arch/x86/mm/init_64.c
@@ -1043,7 +1043,7 @@ remove_pmd_table(pmd_t *pmd_start, unsigned long addr, unsigned long end,
pmd = pmd_start + pmd_index(addr);
for (; addr < end; addr = next, pmd++) {
- next = pmd_addr_end(addr, end);
+ next = pmd_addr_end(*pmd, addr, end);
if (!pmd_present(*pmd))
continue;
@@ -1099,7 +1099,7 @@ remove_pud_table(pud_t *pud_start, unsigned long addr, unsigned long end,
pud = pud_start + pud_index(addr);
for (; addr < end; addr = next, pud++) {
- next = pud_addr_end(addr, end);
+ next = pud_addr_end(*pud, addr, end);
if (!pud_present(*pud))
continue;
@@ -1153,7 +1153,7 @@ remove_p4d_table(p4d_t *p4d_start, unsigned long addr, unsigned long end,
p4d = p4d_start + p4d_index(addr);
for (; addr < end; addr = next, p4d++) {
- next = p4d_addr_end(addr, end);
+ next = p4d_addr_end(*p4d, addr, end);
if (!p4d_present(*p4d))
continue;
@@ -1186,9 +1186,8 @@ remove_pagetable(unsigned long start, unsigned long end, bool direct,
p4d_t *p4d;
for (addr = start; addr < end; addr = next) {
- next = pgd_addr_end(addr, end);
-
pgd = pgd_offset_k(addr);
+ next = pgd_addr_end(*pgd, addr, end);
if (!pgd_present(*pgd))
continue;
@@ -1500,8 +1499,6 @@ static int __meminit vmemmap_populate_hugepages(unsigned long start,
pmd_t *pmd;
for (addr = start; addr < end; addr = next) {
- next = pmd_addr_end(addr, end);
-
pgd = vmemmap_pgd_populate(addr, node);
if (!pgd)
return -ENOMEM;
@@ -1515,6 +1512,7 @@ static int __meminit vmemmap_populate_hugepages(unsigned long start,
return -ENOMEM;
pmd = pmd_offset(pud, addr);
+ next = pmd_addr_end(*pmd, addr, end);
if (pmd_none(*pmd)) {
void *p;
@@ -1623,9 +1621,8 @@ void register_page_bootmem_memmap(unsigned long section_nr,
get_page_bootmem(section_nr, pte_page(*pte),
SECTION_INFO);
} else {
- next = pmd_addr_end(addr, end);
-
pmd = pmd_offset(pud, addr);
+ next = pmd_addr_end(*pmd, addr, end);
if (pmd_none(*pmd))
continue;
diff --git a/arch/x86/mm/kasan_init_64.c b/arch/x86/mm/kasan_init_64.c
index 1a50434c8a4d..2c105b5154ba 100644
--- a/arch/x86/mm/kasan_init_64.c
+++ b/arch/x86/mm/kasan_init_64.c
@@ -96,7 +96,7 @@ static void __init kasan_populate_pud(pud_t *pud, unsigned long addr,
pmd = pmd_offset(pud, addr);
do {
- next = pmd_addr_end(addr, end);
+ next = pmd_addr_end(*pmd, addr, end);
if (!pmd_large(*pmd))
kasan_populate_pmd(pmd, addr, next, nid);
} while (pmd++, addr = next, addr != end);
@@ -116,7 +116,7 @@ static void __init kasan_populate_p4d(p4d_t *p4d, unsigned long addr,
pud = pud_offset(p4d, addr);
do {
- next = pud_addr_end(addr, end);
+ next = pud_addr_end(*pud, addr, end);
if (!pud_large(*pud))
kasan_populate_pud(pud, addr, next, nid);
} while (pud++, addr = next, addr != end);
@@ -136,7 +136,7 @@ static void __init kasan_populate_pgd(pgd_t *pgd, unsigned long addr,
p4d = p4d_offset(pgd, addr);
do {
- next = p4d_addr_end(addr, end);
+ next = p4d_addr_end(*p4d, addr, end);
kasan_populate_p4d(p4d, addr, next, nid);
} while (p4d++, addr = next, addr != end);
}
@@ -151,7 +151,7 @@ static void __init kasan_populate_shadow(unsigned long addr, unsigned long end,
end = round_up(end, PAGE_SIZE);
pgd = pgd_offset_k(addr);
do {
- next = pgd_addr_end(addr, end);
+ next = pgd_addr_end(*pgd, addr, end);
kasan_populate_pgd(pgd, addr, next, nid);
} while (pgd++, addr = next, addr != end);
}
@@ -219,7 +219,7 @@ static void __init kasan_early_p4d_populate(pgd_t *pgd,
p4d = early_p4d_offset(pgd, addr);
do {
- next = p4d_addr_end(addr, end);
+ next = p4d_addr_end(*p4d, addr, end);
if (!p4d_none(*p4d))
continue;
@@ -239,7 +239,7 @@ static void __init kasan_map_early_shadow(pgd_t *pgd)
pgd += pgd_index(addr);
do {
- next = pgd_addr_end(addr, end);
+ next = pgd_addr_end(*pgd, addr, end);
kasan_early_p4d_populate(pgd, addr, next);
} while (pgd++, addr = next, addr != end);
}
@@ -254,7 +254,7 @@ static void __init kasan_shallow_populate_p4ds(pgd_t *pgd,
p4d = p4d_offset(pgd, addr);
do {
- next = p4d_addr_end(addr, end);
+ next = p4d_addr_end(*p4d, addr, end);
if (p4d_none(*p4d)) {
p = early_alloc(PAGE_SIZE, NUMA_NO_NODE, true);
@@ -272,7 +272,7 @@ static void __init kasan_shallow_populate_pgds(void *start, void *end)
addr = (unsigned long)start;
pgd = pgd_offset_k(addr);
do {
- next = pgd_addr_end(addr, (unsigned long)end);
+ next = pgd_addr_end(*pgd, addr, (unsigned long)end);
if (pgd_none(*pgd)) {
p = early_alloc(PAGE_SIZE, NUMA_NO_NODE, true);
diff --git a/include/asm-generic/pgtable-nop4d.h b/include/asm-generic/pgtable-nop4d.h
index ce2cbb3c380f..156b42e51424 100644
--- a/include/asm-generic/pgtable-nop4d.h
+++ b/include/asm-generic/pgtable-nop4d.h
@@ -53,7 +53,7 @@ static inline p4d_t *p4d_offset(pgd_t *pgd, unsigned long address)
#define p4d_free_tlb(tlb, x, a) do { } while (0)
#undef p4d_addr_end
-#define p4d_addr_end(addr, end) (end)
+#define p4d_addr_end(p4d, addr, end) (end)
#endif /* __ASSEMBLY__ */
#endif /* _PGTABLE_NOP4D_H */
diff --git a/include/asm-generic/pgtable-nopmd.h b/include/asm-generic/pgtable-nopmd.h
index 3e13acd019ae..e988384de1c7 100644
--- a/include/asm-generic/pgtable-nopmd.h
+++ b/include/asm-generic/pgtable-nopmd.h
@@ -64,7 +64,7 @@ static inline void pmd_free(struct mm_struct *mm, pmd_t *pmd)
#define pmd_free_tlb(tlb, x, a) do { } while (0)
#undef pmd_addr_end
-#define pmd_addr_end(addr, end) (end)
+#define pmd_addr_end(pmd, addr, end) (end)
#endif /* __ASSEMBLY__ */
diff --git a/include/asm-generic/pgtable-nopud.h b/include/asm-generic/pgtable-nopud.h
index a9d751fbda9e..57a28bade9f9 100644
--- a/include/asm-generic/pgtable-nopud.h
+++ b/include/asm-generic/pgtable-nopud.h
@@ -60,7 +60,7 @@ static inline pud_t *pud_offset(p4d_t *p4d, unsigned long address)
#define pud_free_tlb(tlb, x, a) do { } while (0)
#undef pud_addr_end
-#define pud_addr_end(addr, end) (end)
+#define pud_addr_end(pud, addr, end) (end)
#endif /* __ASSEMBLY__ */
#endif /* _PGTABLE_NOPUD_H */
diff --git a/include/linux/pgtable.h b/include/linux/pgtable.h
index 981c4c2a31fe..67ebc22cf83d 100644
--- a/include/linux/pgtable.h
+++ b/include/linux/pgtable.h
@@ -655,48 +655,34 @@ static inline int arch_unmap_one(struct mm_struct *mm,
* vma end wraps to 0, rounded up __boundary may wrap to 0 throughout.
*/
-#define pgd_addr_end(addr, end) \
+#ifndef pgd_addr_end
+#define pgd_addr_end(pgd, addr, end) \
({ unsigned long __boundary = ((addr) + PGDIR_SIZE) & PGDIR_MASK; \
(__boundary - 1 < (end) - 1)? __boundary: (end); \
})
+#endif
#ifndef p4d_addr_end
-#define p4d_addr_end(addr, end) \
+#define p4d_addr_end(p4d, addr, end) \
({ unsigned long __boundary = ((addr) + P4D_SIZE) & P4D_MASK; \
(__boundary - 1 < (end) - 1)? __boundary: (end); \
})
#endif
#ifndef pud_addr_end
-#define pud_addr_end(addr, end) \
+#define pud_addr_end(pud, addr, end) \
({ unsigned long __boundary = ((addr) + PUD_SIZE) & PUD_MASK; \
(__boundary - 1 < (end) - 1)? __boundary: (end); \
})
#endif
#ifndef pmd_addr_end
-#define pmd_addr_end(addr, end) \
+#define pmd_addr_end(pmd, addr, end) \
({ unsigned long __boundary = ((addr) + PMD_SIZE) & PMD_MASK; \
(__boundary - 1 < (end) - 1)? __boundary: (end); \
})
#endif
-#ifndef pgd_addr_end_folded
-#define pgd_addr_end_folded(pgd, addr, end) pgd_addr_end(addr, end)
-#endif
-
-#ifndef p4d_addr_end_folded
-#define p4d_addr_end_folded(p4d, addr, end) p4d_addr_end(addr, end)
-#endif
-
-#ifndef pud_addr_end_folded
-#define pud_addr_end_folded(pud, addr, end) pud_addr_end(addr, end)
-#endif
-
-#ifndef pmd_addr_end_folded
-#define pmd_addr_end_folded(pmd, addr, end) pmd_addr_end(addr, end)
-#endif
-
/*
* When walking page tables, we usually want to skip any p?d_none entries;
* and any p?d_bad entries - reporting the error before resetting to none.
diff --git a/mm/gup.c b/mm/gup.c
index ba4aace5d0f4..7826876ae7e0 100644
--- a/mm/gup.c
+++ b/mm/gup.c
@@ -2521,7 +2521,7 @@ static int gup_pmd_range(pud_t pud, unsigned long addr, unsigned long end,
do {
pmd_t pmd = READ_ONCE(*pmdp);
- next = pmd_addr_end_folded(pmd, addr, end);
+ next = pmd_addr_end(pmd, addr, end);
if (!pmd_present(pmd))
return 0;
@@ -2564,7 +2564,7 @@ static int gup_pud_range(p4d_t p4d, unsigned long addr, unsigned long end,
do {
pud_t pud = READ_ONCE(*pudp);
- next = pud_addr_end_folded(pud, addr, end);
+ next = pud_addr_end(pud, addr, end);
if (unlikely(!pud_present(pud)))
return 0;
if (unlikely(pud_huge(pud))) {
@@ -2592,7 +2592,7 @@ static int gup_p4d_range(pgd_t pgd, unsigned long addr, unsigned long end,
do {
p4d_t p4d = READ_ONCE(*p4dp);
- next = p4d_addr_end_folded(p4d, addr, end);
+ next = p4d_addr_end(p4d, addr, end);
if (p4d_none(p4d))
return 0;
BUILD_BUG_ON(p4d_huge(p4d));
@@ -2617,7 +2617,7 @@ static void gup_pgd_range(unsigned long addr, unsigned long end,
do {
pgd_t pgd = READ_ONCE(*pgdp);
- next = pgd_addr_end_folded(pgd, addr, end);
+ next = pgd_addr_end(pgd, addr, end);
if (pgd_none(pgd))
return;
if (unlikely(pgd_huge(pgd))) {
diff --git a/mm/ioremap.c b/mm/ioremap.c
index 5fa1ab41d152..400fa119c09d 100644
--- a/mm/ioremap.c
+++ b/mm/ioremap.c
@@ -114,7 +114,7 @@ static inline int ioremap_pmd_range(pud_t *pud, unsigned long addr,
if (!pmd)
return -ENOMEM;
do {
- next = pmd_addr_end(addr, end);
+ next = pmd_addr_end(*pmd, addr, end);
if (ioremap_try_huge_pmd(pmd, addr, next, phys_addr, prot)) {
*mask |= PGTBL_PMD_MODIFIED;
@@ -160,7 +160,7 @@ static inline int ioremap_pud_range(p4d_t *p4d, unsigned long addr,
if (!pud)
return -ENOMEM;
do {
- next = pud_addr_end(addr, end);
+ next = pud_addr_end(*pud, addr, end);
if (ioremap_try_huge_pud(pud, addr, next, phys_addr, prot)) {
*mask |= PGTBL_PUD_MODIFIED;
@@ -206,7 +206,7 @@ static inline int ioremap_p4d_range(pgd_t *pgd, unsigned long addr,
if (!p4d)
return -ENOMEM;
do {
- next = p4d_addr_end(addr, end);
+ next = p4d_addr_end(*p4d, addr, end);
if (ioremap_try_huge_p4d(p4d, addr, next, phys_addr, prot)) {
*mask |= PGTBL_P4D_MODIFIED;
@@ -234,7 +234,7 @@ int ioremap_page_range(unsigned long addr,
start = addr;
pgd = pgd_offset_k(addr);
do {
- next = pgd_addr_end(addr, end);
+ next = pgd_addr_end(*pgd, addr, end);
err = ioremap_p4d_range(pgd, addr, next, phys_addr, prot,
&mask);
if (err)
diff --git a/mm/kasan/init.c b/mm/kasan/init.c
index fe6be0be1f76..829627a92763 100644
--- a/mm/kasan/init.c
+++ b/mm/kasan/init.c
@@ -117,7 +117,7 @@ static int __ref zero_pmd_populate(pud_t *pud, unsigned long addr,
unsigned long next;
do {
- next = pmd_addr_end(addr, end);
+ next = pmd_addr_end(*pmd, addr, end);
if (IS_ALIGNED(addr, PMD_SIZE) && end - addr >= PMD_SIZE) {
pmd_populate_kernel(&init_mm, pmd,
@@ -150,7 +150,7 @@ static int __ref zero_pud_populate(p4d_t *p4d, unsigned long addr,
unsigned long next;
do {
- next = pud_addr_end(addr, end);
+ next = pud_addr_end(*pud, addr, end);
if (IS_ALIGNED(addr, PUD_SIZE) && end - addr >= PUD_SIZE) {
pmd_t *pmd;
@@ -187,7 +187,7 @@ static int __ref zero_p4d_populate(pgd_t *pgd, unsigned long addr,
unsigned long next;
do {
- next = p4d_addr_end(addr, end);
+ next = p4d_addr_end(*p4d, addr, end);
if (IS_ALIGNED(addr, P4D_SIZE) && end - addr >= P4D_SIZE) {
pud_t *pud;
pmd_t *pmd;
@@ -236,7 +236,7 @@ int __ref kasan_populate_early_shadow(const void *shadow_start,
unsigned long next;
do {
- next = pgd_addr_end(addr, end);
+ next = pgd_addr_end(*pgd, addr, end);
if (IS_ALIGNED(addr, PGDIR_SIZE) && end - addr >= PGDIR_SIZE) {
p4d_t *p4d;
@@ -370,7 +370,7 @@ static void kasan_remove_pmd_table(pmd_t *pmd, unsigned long addr,
for (; addr < end; addr = next, pmd++) {
pte_t *pte;
- next = pmd_addr_end(addr, end);
+ next = pmd_addr_end(*pmd, addr, end);
if (!pmd_present(*pmd))
continue;
@@ -395,7 +395,7 @@ static void kasan_remove_pud_table(pud_t *pud, unsigned long addr,
for (; addr < end; addr = next, pud++) {
pmd_t *pmd, *pmd_base;
- next = pud_addr_end(addr, end);
+ next = pud_addr_end(*pud, addr, end);
if (!pud_present(*pud))
continue;
@@ -421,7 +421,7 @@ static void kasan_remove_p4d_table(p4d_t *p4d, unsigned long addr,
for (; addr < end; addr = next, p4d++) {
pud_t *pud;
- next = p4d_addr_end(addr, end);
+ next = p4d_addr_end(*p4d, addr, end);
if (!p4d_present(*p4d))
continue;
@@ -454,9 +454,8 @@ void kasan_remove_zero_shadow(void *start, unsigned long size)
for (; addr < end; addr = next) {
p4d_t *p4d;
- next = pgd_addr_end(addr, end);
-
pgd = pgd_offset_k(addr);
+ next = pgd_addr_end(*pgd, addr, end);
if (!pgd_present(*pgd))
continue;
diff --git a/mm/madvise.c b/mm/madvise.c
index e32e7efbba0f..acfb3441d97e 100644
--- a/mm/madvise.c
+++ b/mm/madvise.c
@@ -326,7 +326,7 @@ static int madvise_cold_or_pageout_pte_range(pmd_t *pmd,
#ifdef CONFIG_TRANSPARENT_HUGEPAGE
if (pmd_trans_huge(*pmd)) {
pmd_t orig_pmd;
- unsigned long next = pmd_addr_end(addr, end);
+ unsigned long next = pmd_addr_end(*pmd, addr, end);
tlb_change_page_size(tlb, HPAGE_PMD_SIZE);
ptl = pmd_trans_huge_lock(pmd, vma);
@@ -587,7 +587,7 @@ static int madvise_free_pte_range(pmd_t *pmd, unsigned long addr,
int nr_swap = 0;
unsigned long next;
- next = pmd_addr_end(addr, end);
+ next = pmd_addr_end(*pmd, addr, end);
if (pmd_trans_huge(*pmd))
if (madvise_free_huge_pmd(tlb, vma, pmd, addr, next))
goto next;
diff --git a/mm/memory.c b/mm/memory.c
index fb5463153351..f95424946b0d 100644
--- a/mm/memory.c
+++ b/mm/memory.c
@@ -233,7 +233,7 @@ static inline void free_pmd_range(struct mmu_gather *tlb, pud_t *pud,
start = addr;
pmd = pmd_offset(pud, addr);
do {
- next = pmd_addr_end(addr, end);
+ next = pmd_addr_end(*pmd, addr, end);
if (pmd_none_or_clear_bad(pmd))
continue;
free_pte_range(tlb, pmd, addr);
@@ -267,7 +267,7 @@ static inline void free_pud_range(struct mmu_gather *tlb, p4d_t *p4d,
start = addr;
pud = pud_offset(p4d, addr);
do {
- next = pud_addr_end(addr, end);
+ next = pud_addr_end(*pud, addr, end);
if (pud_none_or_clear_bad(pud))
continue;
free_pmd_range(tlb, pud, addr, next, floor, ceiling);
@@ -301,7 +301,7 @@ static inline void free_p4d_range(struct mmu_gather *tlb, pgd_t *pgd,
start = addr;
p4d = p4d_offset(pgd, addr);
do {
- next = p4d_addr_end(addr, end);
+ next = p4d_addr_end(*p4d, addr, end);
if (p4d_none_or_clear_bad(p4d))
continue;
free_pud_range(tlb, p4d, addr, next, floor, ceiling);
@@ -381,7 +381,7 @@ void free_pgd_range(struct mmu_gather *tlb,
tlb_change_page_size(tlb, PAGE_SIZE);
pgd = pgd_offset(tlb->mm, addr);
do {
- next = pgd_addr_end(addr, end);
+ next = pgd_addr_end(*pgd, addr, end);
if (pgd_none_or_clear_bad(pgd))
continue;
free_p4d_range(tlb, pgd, addr, next, floor, ceiling);
@@ -887,7 +887,7 @@ static inline int copy_pmd_range(struct mm_struct *dst_mm, struct mm_struct *src
return -ENOMEM;
src_pmd = pmd_offset(src_pud, addr);
do {
- next = pmd_addr_end(addr, end);
+ next = pmd_addr_end(*src_pmd, addr, end);
if (is_swap_pmd(*src_pmd) || pmd_trans_huge(*src_pmd)
|| pmd_devmap(*src_pmd)) {
int err;
@@ -921,7 +921,7 @@ static inline int copy_pud_range(struct mm_struct *dst_mm, struct mm_struct *src
return -ENOMEM;
src_pud = pud_offset(src_p4d, addr);
do {
- next = pud_addr_end(addr, end);
+ next = pud_addr_end(*src_pud, addr, end);
if (pud_trans_huge(*src_pud) || pud_devmap(*src_pud)) {
int err;
@@ -955,7 +955,7 @@ static inline int copy_p4d_range(struct mm_struct *dst_mm, struct mm_struct *src
return -ENOMEM;
src_p4d = p4d_offset(src_pgd, addr);
do {
- next = p4d_addr_end(addr, end);
+ next = p4d_addr_end(*src_p4d, addr, end);
if (p4d_none_or_clear_bad(src_p4d))
continue;
if (copy_pud_range(dst_mm, src_mm, dst_p4d, src_p4d,
@@ -1017,7 +1017,7 @@ int copy_page_range(struct mm_struct *dst_mm, struct mm_struct *src_mm,
dst_pgd = pgd_offset(dst_mm, addr);
src_pgd = pgd_offset(src_mm, addr);
do {
- next = pgd_addr_end(addr, end);
+ next = pgd_addr_end(*src_pgd, addr, end);
if (pgd_none_or_clear_bad(src_pgd))
continue;
if (unlikely(copy_p4d_range(dst_mm, src_mm, dst_pgd, src_pgd,
@@ -1177,7 +1177,7 @@ static inline unsigned long zap_pmd_range(struct mmu_gather *tlb,
pmd = pmd_offset(pud, addr);
do {
- next = pmd_addr_end(addr, end);
+ next = pmd_addr_end(*pmd, addr, end);
if (is_swap_pmd(*pmd) || pmd_trans_huge(*pmd) || pmd_devmap(*pmd)) {
if (next - addr != HPAGE_PMD_SIZE)
__split_huge_pmd(vma, pmd, addr, false, NULL);
@@ -1212,7 +1212,7 @@ static inline unsigned long zap_pud_range(struct mmu_gather *tlb,
pud = pud_offset(p4d, addr);
do {
- next = pud_addr_end(addr, end);
+ next = pud_addr_end(*pud, addr, end);
if (pud_trans_huge(*pud) || pud_devmap(*pud)) {
if (next - addr != HPAGE_PUD_SIZE) {
mmap_assert_locked(tlb->mm);
@@ -1241,7 +1241,7 @@ static inline unsigned long zap_p4d_range(struct mmu_gather *tlb,
p4d = p4d_offset(pgd, addr);
do {
- next = p4d_addr_end(addr, end);
+ next = p4d_addr_end(*p4d, addr, end);
if (p4d_none_or_clear_bad(p4d))
continue;
next = zap_pud_range(tlb, vma, p4d, addr, next, details);
@@ -1262,7 +1262,7 @@ void unmap_page_range(struct mmu_gather *tlb,
tlb_start_vma(tlb, vma);
pgd = pgd_offset(vma->vm_mm, addr);
do {
- next = pgd_addr_end(addr, end);
+ next = pgd_addr_end(*pgd, addr, end);
if (pgd_none_or_clear_bad(pgd))
continue;
next = zap_p4d_range(tlb, vma, pgd, addr, next, details);
@@ -2030,7 +2030,7 @@ static inline int remap_pmd_range(struct mm_struct *mm, pud_t *pud,
return -ENOMEM;
VM_BUG_ON(pmd_trans_huge(*pmd));
do {
- next = pmd_addr_end(addr, end);
+ next = pmd_addr_end(*pmd, addr, end);
err = remap_pte_range(mm, pmd, addr, next,
pfn + (addr >> PAGE_SHIFT), prot);
if (err)
@@ -2052,7 +2052,7 @@ static inline int remap_pud_range(struct mm_struct *mm, p4d_t *p4d,
if (!pud)
return -ENOMEM;
do {
- next = pud_addr_end(addr, end);
+ next = pud_addr_end(*pud, addr, end);
err = remap_pmd_range(mm, pud, addr, next,
pfn + (addr >> PAGE_SHIFT), prot);
if (err)
@@ -2074,7 +2074,7 @@ static inline int remap_p4d_range(struct mm_struct *mm, pgd_t *pgd,
if (!p4d)
return -ENOMEM;
do {
- next = p4d_addr_end(addr, end);
+ next = p4d_addr_end(*p4d, addr, end);
err = remap_pud_range(mm, p4d, addr, next,
pfn + (addr >> PAGE_SHIFT), prot);
if (err)
@@ -2143,7 +2143,7 @@ int remap_pfn_range(struct vm_area_struct *vma, unsigned long addr,
pgd = pgd_offset(mm, addr);
flush_cache_range(vma, addr, end);
do {
- next = pgd_addr_end(addr, end);
+ next = pgd_addr_end(*pgd, addr, end);
err = remap_p4d_range(mm, pgd, addr, next,
pfn + (addr >> PAGE_SHIFT), prot);
if (err)
@@ -2266,7 +2266,7 @@ static int apply_to_pmd_range(struct mm_struct *mm, pud_t *pud,
pmd = pmd_offset(pud, addr);
}
do {
- next = pmd_addr_end(addr, end);
+ next = pmd_addr_end(*pmd, addr, end);
if (create || !pmd_none_or_clear_bad(pmd)) {
err = apply_to_pte_range(mm, pmd, addr, next, fn, data,
create, mask);
@@ -2294,7 +2294,7 @@ static int apply_to_pud_range(struct mm_struct *mm, p4d_t *p4d,
pud = pud_offset(p4d, addr);
}
do {
- next = pud_addr_end(addr, end);
+ next = pud_addr_end(*pud, addr, end);
if (create || !pud_none_or_clear_bad(pud)) {
err = apply_to_pmd_range(mm, pud, addr, next, fn, data,
create, mask);
@@ -2322,7 +2322,7 @@ static int apply_to_p4d_range(struct mm_struct *mm, pgd_t *pgd,
p4d = p4d_offset(pgd, addr);
}
do {
- next = p4d_addr_end(addr, end);
+ next = p4d_addr_end(*p4d, addr, end);
if (create || !p4d_none_or_clear_bad(p4d)) {
err = apply_to_pud_range(mm, p4d, addr, next, fn, data,
create, mask);
@@ -2348,7 +2348,7 @@ static int __apply_to_page_range(struct mm_struct *mm, unsigned long addr,
pgd = pgd_offset(mm, addr);
do {
- next = pgd_addr_end(addr, end);
+ next = pgd_addr_end(*pgd, addr, end);
if (!create && pgd_none_or_clear_bad(pgd))
continue;
err = apply_to_p4d_range(mm, pgd, addr, next, fn, data, create, &mask);
diff --git a/mm/mlock.c b/mm/mlock.c
index 93ca2bf30b4f..5898e8fe2288 100644
--- a/mm/mlock.c
+++ b/mm/mlock.c
@@ -374,8 +374,12 @@ static unsigned long __munlock_pagevec_fill(struct pagevec *pvec,
struct vm_area_struct *vma, struct zone *zone,
unsigned long start, unsigned long end)
{
- pte_t *pte;
spinlock_t *ptl;
+ pgd_t *pgd;
+ p4d_t *p4d;
+ pud_t *pud;
+ pmd_t *pmd;
+ pte_t *pte;
/*
* Initialize pte walk starting at the already pinned page where we
@@ -384,10 +388,14 @@ static unsigned long __munlock_pagevec_fill(struct pagevec *pvec,
*/
pte = get_locked_pte(vma->vm_mm, start, &ptl);
/* Make sure we do not cross the page table boundary */
- end = pgd_addr_end(start, end);
- end = p4d_addr_end(start, end);
- end = pud_addr_end(start, end);
- end = pmd_addr_end(start, end);
+ pgd = pgd_offset(vma->vm_mm, start);
+ end = pgd_addr_end(*pgd, start, end);
+ p4d = p4d_offset(pgd, start);
+ end = p4d_addr_end(*p4d, start, end);
+ pud = pud_offset(p4d, start);
+ end = pud_addr_end(*pud, start, end);
+ pmd = pmd_offset(pud, start);
+ end = pmd_addr_end(*pmd, start, end);
/* The page next to the pinned page is the first we will try to get */
start += PAGE_SIZE;
diff --git a/mm/mprotect.c b/mm/mprotect.c
index ce8b8a5eacbb..278f2dbd1f20 100644
--- a/mm/mprotect.c
+++ b/mm/mprotect.c
@@ -225,7 +225,7 @@ static inline unsigned long change_pmd_range(struct vm_area_struct *vma,
do {
unsigned long this_pages;
- next = pmd_addr_end(addr, end);
+ next = pmd_addr_end(*pmd, addr, end);
/*
* Automatic NUMA balancing walks the tables with mmap_lock
@@ -291,7 +291,7 @@ static inline unsigned long change_pud_range(struct vm_area_struct *vma,
pud = pud_offset(p4d, addr);
do {
- next = pud_addr_end(addr, end);
+ next = pud_addr_end(*pud, addr, end);
if (pud_none_or_clear_bad(pud))
continue;
pages += change_pmd_range(vma, pud, addr, next, newprot,
@@ -311,7 +311,7 @@ static inline unsigned long change_p4d_range(struct vm_area_struct *vma,
p4d = p4d_offset(pgd, addr);
do {
- next = p4d_addr_end(addr, end);
+ next = p4d_addr_end(*p4d, addr, end);
if (p4d_none_or_clear_bad(p4d))
continue;
pages += change_pud_range(vma, p4d, addr, next, newprot,
@@ -336,7 +336,7 @@ static unsigned long change_protection_range(struct vm_area_struct *vma,
flush_cache_range(vma, addr, end);
inc_tlb_flush_pending(mm);
do {
- next = pgd_addr_end(addr, end);
+ next = pgd_addr_end(*pgd, addr, end);
if (pgd_none_or_clear_bad(pgd))
continue;
pages += change_p4d_range(vma, pgd, addr, next, newprot,
diff --git a/mm/pagewalk.c b/mm/pagewalk.c
index e81640d9f177..a5b9f61b5d45 100644
--- a/mm/pagewalk.c
+++ b/mm/pagewalk.c
@@ -70,7 +70,7 @@ static int walk_pmd_range(pud_t *pud, unsigned long addr, unsigned long end,
pmd = pmd_offset(pud, addr);
do {
again:
- next = pmd_addr_end(addr, end);
+ next = pmd_addr_end(*pmd, addr, end);
if (pmd_none(*pmd) || (!walk->vma && !walk->no_vma)) {
if (ops->pte_hole)
err = ops->pte_hole(addr, next, depth, walk);
@@ -128,7 +128,7 @@ static int walk_pud_range(p4d_t *p4d, unsigned long addr, unsigned long end,
pud = pud_offset(p4d, addr);
do {
again:
- next = pud_addr_end(addr, end);
+ next = pud_addr_end(*pud, addr, end);
if (pud_none(*pud) || (!walk->vma && !walk->no_vma)) {
if (ops->pte_hole)
err = ops->pte_hole(addr, next, depth, walk);
@@ -176,7 +176,7 @@ static int walk_p4d_range(pgd_t *pgd, unsigned long addr, unsigned long end,
p4d = p4d_offset(pgd, addr);
do {
- next = p4d_addr_end(addr, end);
+ next = p4d_addr_end(*p4d, addr, end);
if (p4d_none_or_clear_bad(p4d)) {
if (ops->pte_hole)
err = ops->pte_hole(addr, next, depth, walk);
@@ -211,7 +211,7 @@ static int walk_pgd_range(unsigned long addr, unsigned long end,
else
pgd = pgd_offset(walk->mm, addr);
do {
- next = pgd_addr_end(addr, end);
+ next = pgd_addr_end(*pgd, addr, end);
if (pgd_none_or_clear_bad(pgd)) {
if (ops->pte_hole)
err = ops->pte_hole(addr, next, 0, walk);
diff --git a/mm/swapfile.c b/mm/swapfile.c
index 20012c0c0252..b1dd815aee6b 100644
--- a/mm/swapfile.c
+++ b/mm/swapfile.c
@@ -2018,7 +2018,7 @@ static inline int unuse_pmd_range(struct vm_area_struct *vma, pud_t *pud,
pmd = pmd_offset(pud, addr);
do {
cond_resched();
- next = pmd_addr_end(addr, end);
+ next = pmd_addr_end(*pmd, addr, end);
if (pmd_none_or_trans_huge_or_clear_bad(pmd))
continue;
ret = unuse_pte_range(vma, pmd, addr, next, type,
@@ -2040,7 +2040,7 @@ static inline int unuse_pud_range(struct vm_area_struct *vma, p4d_t *p4d,
pud = pud_offset(p4d, addr);
do {
- next = pud_addr_end(addr, end);
+ next = pud_addr_end(*pud, addr, end);
if (pud_none_or_clear_bad(pud))
continue;
ret = unuse_pmd_range(vma, pud, addr, next, type,
@@ -2062,7 +2062,7 @@ static inline int unuse_p4d_range(struct vm_area_struct *vma, pgd_t *pgd,
p4d = p4d_offset(pgd, addr);
do {
- next = p4d_addr_end(addr, end);
+ next = p4d_addr_end(*p4d, addr, end);
if (p4d_none_or_clear_bad(p4d))
continue;
ret = unuse_pud_range(vma, p4d, addr, next, type,
@@ -2085,7 +2085,7 @@ static int unuse_vma(struct vm_area_struct *vma, unsigned int type,
pgd = pgd_offset(vma->vm_mm, addr);
do {
- next = pgd_addr_end(addr, end);
+ next = pgd_addr_end(*pgd, addr, end);
if (pgd_none_or_clear_bad(pgd))
continue;
ret = unuse_p4d_range(vma, pgd, addr, next, type,
diff --git a/mm/vmalloc.c b/mm/vmalloc.c
index be4724b916b3..09ff0d5ecbc1 100644
--- a/mm/vmalloc.c
+++ b/mm/vmalloc.c
@@ -93,7 +93,7 @@ static void vunmap_pmd_range(pud_t *pud, unsigned long addr, unsigned long end,
pmd = pmd_offset(pud, addr);
do {
- next = pmd_addr_end(addr, end);
+ next = pmd_addr_end(*pmd, addr, end);
cleared = pmd_clear_huge(pmd);
if (cleared || pmd_bad(*pmd))
@@ -118,7 +118,7 @@ static void vunmap_pud_range(p4d_t *p4d, unsigned long addr, unsigned long end,
pud = pud_offset(p4d, addr);
do {
- next = pud_addr_end(addr, end);
+ next = pud_addr_end(*pud, addr, end);
cleared = pud_clear_huge(pud);
if (cleared || pud_bad(*pud))
@@ -141,7 +141,7 @@ static void vunmap_p4d_range(pgd_t *pgd, unsigned long addr, unsigned long end,
p4d = p4d_offset(pgd, addr);
do {
- next = p4d_addr_end(addr, end);
+ next = p4d_addr_end(*p4d, addr, end);
cleared = p4d_clear_huge(p4d);
if (cleared || p4d_bad(*p4d))
@@ -179,7 +179,7 @@ void unmap_kernel_range_noflush(unsigned long start, unsigned long size)
BUG_ON(addr >= end);
pgd = pgd_offset_k(addr);
do {
- next = pgd_addr_end(addr, end);
+ next = pgd_addr_end(*pgd, addr, end);
if (pgd_bad(*pgd))
mask |= PGTBL_PGD_MODIFIED;
if (pgd_none_or_clear_bad(pgd))
@@ -230,7 +230,7 @@ static int vmap_pmd_range(pud_t *pud, unsigned long addr,
if (!pmd)
return -ENOMEM;
do {
- next = pmd_addr_end(addr, end);
+ next = pmd_addr_end(*pmd, addr, end);
if (vmap_pte_range(pmd, addr, next, prot, pages, nr, mask))
return -ENOMEM;
} while (pmd++, addr = next, addr != end);
@@ -248,7 +248,7 @@ static int vmap_pud_range(p4d_t *p4d, unsigned long addr,
if (!pud)
return -ENOMEM;
do {
- next = pud_addr_end(addr, end);
+ next = pud_addr_end(*pud, addr, end);
if (vmap_pmd_range(pud, addr, next, prot, pages, nr, mask))
return -ENOMEM;
} while (pud++, addr = next, addr != end);
@@ -266,7 +266,7 @@ static int vmap_p4d_range(pgd_t *pgd, unsigned long addr,
if (!p4d)
return -ENOMEM;
do {
- next = p4d_addr_end(addr, end);
+ next = p4d_addr_end(*p4d, addr, end);
if (vmap_pud_range(p4d, addr, next, prot, pages, nr, mask))
return -ENOMEM;
} while (p4d++, addr = next, addr != end);
@@ -305,7 +305,7 @@ int map_kernel_range_noflush(unsigned long addr, unsigned long size,
BUG_ON(addr >= end);
pgd = pgd_offset_k(addr);
do {
- next = pgd_addr_end(addr, end);
+ next = pgd_addr_end(*pgd, addr, end);
if (pgd_bad(*pgd))
mask |= PGTBL_PGD_MODIFIED;
err = vmap_p4d_range(pgd, addr, next, prot, pages, &nr, &mask);
--
2.17.1
^ permalink raw reply related
* [RFC PATCH v2 0/3] mm/gup: fix gup_fast with dynamic page table folding
From: Gerald Schaefer @ 2020-09-07 18:00 UTC (permalink / raw)
To: Jason Gunthorpe, John Hubbard
Cc: Peter Zijlstra, Dave Hansen, linux-mm, Paul Mackerras,
linux-sparc, Alexander Gordeev, Claudio Imbrenda, Will Deacon,
linux-arch, linux-s390, Vasily Gorbik, Richard Weinberger,
linux-x86, Russell King, Christian Borntraeger, Ingo Molnar,
Catalin Marinas, Andrey Ryabinin, Heiko Carstens, Arnd Bergmann,
Jeff Dike, linux-um, Borislav Petkov, Andy Lutomirski,
Thomas Gleixner, linux-arm, linux-power, LKML, Andrew Morton,
Linus Torvalds, Mike Rapoport
This is v2 of an RFC previously discussed here:
https://lore.kernel.org/lkml/20200828140314.8556-1-gerald.schaefer@linux.ibm.com/
Patch 1 is a fix for a regression in gup_fast on s390, after our conversion
to common gup_fast code. It will introduce special helper functions
pXd_addr_end_folded(), which have to be used in places where pagetable walk
is done w/o lock and with READ_ONCE, so currently only in gup_fast.
Patch 2 is an attempt to make that more generic, i.e. change pXd_addr_end()
themselves by adding an extra pXd value parameter. That was suggested by
Jason during v1 discussion, because he is already thinking of some other
places where he might want to switch to the READ_ONCE logic for pagetable
walks. In general, that would be the cleanest / safest solution, but there
is some impact on other architectures and common code, hence the new and
greatly enlarged recipient list.
Patch 3 is a "nice to have" add-on, which makes pXd_addr_end() inline
functions instead of #defines, so that we get some type checking for the
new pXd value parameter.
Not sure about Fixes/stable tags for the generic solution. Only patch 1
fixes a real bug on s390, and has Fixes/stable tags. Patches 2 + 3 might
still be nice to have in stable, to ease future backports, but I guess
"nice to have" does not really qualify for stable backports.
Changes in v2:
- Pick option 2 from v1 discussion (pXd_addr_end_folded helpers)
- Add patch 2 + 3 for more generic approach
Alexander Gordeev (3):
mm/gup: fix gup_fast with dynamic page table folding
mm: make pXd_addr_end() functions page-table entry aware
mm: make generic pXd_addr_end() macros inline functions
arch/arm/include/asm/pgtable-2level.h | 2 +-
arch/arm/mm/idmap.c | 6 ++--
arch/arm/mm/mmu.c | 8 ++---
arch/arm64/kernel/hibernate.c | 16 +++++----
arch/arm64/kvm/mmu.c | 16 ++++-----
arch/arm64/mm/kasan_init.c | 8 ++---
arch/arm64/mm/mmu.c | 25 +++++++-------
arch/powerpc/mm/book3s64/radix_pgtable.c | 7 ++--
arch/powerpc/mm/hugetlbpage.c | 6 ++--
arch/s390/include/asm/pgtable.h | 42 ++++++++++++++++++++++++
arch/s390/mm/page-states.c | 8 ++---
arch/s390/mm/pageattr.c | 8 ++---
arch/s390/mm/vmem.c | 8 ++---
arch/sparc/mm/hugetlbpage.c | 6 ++--
arch/um/kernel/tlb.c | 8 ++---
arch/x86/mm/init_64.c | 15 ++++-----
arch/x86/mm/kasan_init_64.c | 16 ++++-----
include/asm-generic/pgtable-nop4d.h | 2 +-
include/asm-generic/pgtable-nopmd.h | 2 +-
include/asm-generic/pgtable-nopud.h | 2 +-
include/linux/pgtable.h | 38 ++++++++++++---------
mm/gup.c | 8 ++---
mm/ioremap.c | 8 ++---
mm/kasan/init.c | 17 +++++-----
mm/madvise.c | 4 +--
mm/memory.c | 40 +++++++++++-----------
mm/mlock.c | 18 +++++++---
mm/mprotect.c | 8 ++---
mm/pagewalk.c | 8 ++---
mm/swapfile.c | 8 ++---
mm/vmalloc.c | 16 ++++-----
31 files changed, 219 insertions(+), 165 deletions(-)
--
2.17.1
^ permalink raw reply
* Re: [RFC PATCH v2 0/3] mm/gup: fix gup_fast with dynamic page table folding
From: Mike Rapoport @ 2020-09-07 20:12 UTC (permalink / raw)
To: Gerald Schaefer
Cc: Peter Zijlstra, Dave Hansen, linux-mm, Paul Mackerras,
linux-sparc, Alexander Gordeev, Claudio Imbrenda, Will Deacon,
linux-arch, linux-s390, Vasily Gorbik, Christian Borntraeger,
Richard Weinberger, linux-x86, Russell King, Jason Gunthorpe,
Ingo Molnar, Catalin Marinas, Andrey Ryabinin, Heiko Carstens,
Arnd Bergmann, John Hubbard, Jeff Dike, linux-um, Borislav Petkov,
Andy Lutomirski, Thomas Gleixner, linux-arm, linux-power, LKML,
Andrew Morton, Linus Torvalds
In-Reply-To: <20200907180058.64880-1-gerald.schaefer@linux.ibm.com>
On Mon, Sep 07, 2020 at 08:00:55PM +0200, Gerald Schaefer wrote:
> This is v2 of an RFC previously discussed here:
> https://lore.kernel.org/lkml/20200828140314.8556-1-gerald.schaefer@linux.ibm.com/
>
> Patch 1 is a fix for a regression in gup_fast on s390, after our conversion
> to common gup_fast code. It will introduce special helper functions
> pXd_addr_end_folded(), which have to be used in places where pagetable walk
> is done w/o lock and with READ_ONCE, so currently only in gup_fast.
>
> Patch 2 is an attempt to make that more generic, i.e. change pXd_addr_end()
> themselves by adding an extra pXd value parameter. That was suggested by
> Jason during v1 discussion, because he is already thinking of some other
> places where he might want to switch to the READ_ONCE logic for pagetable
> walks. In general, that would be the cleanest / safest solution, but there
> is some impact on other architectures and common code, hence the new and
> greatly enlarged recipient list.
>
> Patch 3 is a "nice to have" add-on, which makes pXd_addr_end() inline
> functions instead of #defines, so that we get some type checking for the
> new pXd value parameter.
>
> Not sure about Fixes/stable tags for the generic solution. Only patch 1
> fixes a real bug on s390, and has Fixes/stable tags. Patches 2 + 3 might
> still be nice to have in stable, to ease future backports, but I guess
> "nice to have" does not really qualify for stable backports.
I also think that adding pXd parameter to pXd_addr_end() is a cleaner
way and with this patch 1 is not really required. I would even merge
patches 2 and 3 into a single patch and use only it as the fix.
[ /me apologises to stable@ team :-) ]
> Changes in v2:
> - Pick option 2 from v1 discussion (pXd_addr_end_folded helpers)
> - Add patch 2 + 3 for more generic approach
>
> Alexander Gordeev (3):
> mm/gup: fix gup_fast with dynamic page table folding
> mm: make pXd_addr_end() functions page-table entry aware
> mm: make generic pXd_addr_end() macros inline functions
>
> arch/arm/include/asm/pgtable-2level.h | 2 +-
> arch/arm/mm/idmap.c | 6 ++--
> arch/arm/mm/mmu.c | 8 ++---
> arch/arm64/kernel/hibernate.c | 16 +++++----
> arch/arm64/kvm/mmu.c | 16 ++++-----
> arch/arm64/mm/kasan_init.c | 8 ++---
> arch/arm64/mm/mmu.c | 25 +++++++-------
> arch/powerpc/mm/book3s64/radix_pgtable.c | 7 ++--
> arch/powerpc/mm/hugetlbpage.c | 6 ++--
> arch/s390/include/asm/pgtable.h | 42 ++++++++++++++++++++++++
> arch/s390/mm/page-states.c | 8 ++---
> arch/s390/mm/pageattr.c | 8 ++---
> arch/s390/mm/vmem.c | 8 ++---
> arch/sparc/mm/hugetlbpage.c | 6 ++--
> arch/um/kernel/tlb.c | 8 ++---
> arch/x86/mm/init_64.c | 15 ++++-----
> arch/x86/mm/kasan_init_64.c | 16 ++++-----
> include/asm-generic/pgtable-nop4d.h | 2 +-
> include/asm-generic/pgtable-nopmd.h | 2 +-
> include/asm-generic/pgtable-nopud.h | 2 +-
> include/linux/pgtable.h | 38 ++++++++++++---------
> mm/gup.c | 8 ++---
> mm/ioremap.c | 8 ++---
> mm/kasan/init.c | 17 +++++-----
> mm/madvise.c | 4 +--
> mm/memory.c | 40 +++++++++++-----------
> mm/mlock.c | 18 +++++++---
> mm/mprotect.c | 8 ++---
> mm/pagewalk.c | 8 ++---
> mm/swapfile.c | 8 ++---
> mm/vmalloc.c | 16 ++++-----
> 31 files changed, 219 insertions(+), 165 deletions(-)
>
> --
> 2.17.1
>
--
Sincerely yours,
Mike.
^ permalink raw reply
* Re: [RFC PATCH v2 3/3] mm: make generic pXd_addr_end() macros inline functions
From: Mike Rapoport @ 2020-09-07 20:15 UTC (permalink / raw)
To: Gerald Schaefer
Cc: Peter Zijlstra, Dave Hansen, linux-mm, Paul Mackerras,
linux-sparc, Alexander Gordeev, Claudio Imbrenda, Will Deacon,
linux-arch, linux-s390, Vasily Gorbik, Christian Borntraeger,
Richard Weinberger, linux-x86, Russell King, Jason Gunthorpe,
Ingo Molnar, Catalin Marinas, Andrey Ryabinin, Heiko Carstens,
Arnd Bergmann, John Hubbard, Jeff Dike, linux-um, Borislav Petkov,
Andy Lutomirski, Thomas Gleixner, linux-arm, linux-power, LKML,
Andrew Morton, Linus Torvalds
In-Reply-To: <20200907180058.64880-4-gerald.schaefer@linux.ibm.com>
Hi,
Some style comments below.
On Mon, Sep 07, 2020 at 08:00:58PM +0200, Gerald Schaefer wrote:
> From: Alexander Gordeev <agordeev@linux.ibm.com>
>
> Since pXd_addr_end() macros take pXd page-table entry as a
> parameter it makes sense to check the entry type on compile.
> Even though most archs do not make use of page-table entries
> in pXd_addr_end() calls, checking the type in traversal code
> paths could help to avoid subtle bugs.
>
> Signed-off-by: Alexander Gordeev <agordeev@linux.ibm.com>
> Signed-off-by: Gerald Schaefer <gerald.schaefer@linux.ibm.com>
> ---
> include/linux/pgtable.h | 36 ++++++++++++++++++++----------------
> 1 file changed, 20 insertions(+), 16 deletions(-)
>
> diff --git a/include/linux/pgtable.h b/include/linux/pgtable.h
> index 67ebc22cf83d..d9e7d16c2263 100644
> --- a/include/linux/pgtable.h
> +++ b/include/linux/pgtable.h
> @@ -656,31 +656,35 @@ static inline int arch_unmap_one(struct mm_struct *mm,
> */
>
> #ifndef pgd_addr_end
> -#define pgd_addr_end(pgd, addr, end) \
> -({ unsigned long __boundary = ((addr) + PGDIR_SIZE) & PGDIR_MASK; \
> - (__boundary - 1 < (end) - 1)? __boundary: (end); \
> -})
> +#define pgd_addr_end pgd_addr_end
> +static inline unsigned long pgd_addr_end(pgd_t pgd, unsigned long addr, unsigned long end)
> +{ unsigned long __boundary = (addr + PGDIR_SIZE) & PGDIR_MASK;
The code should be on a separate line from the curly brace.
Besides, since this is not a macro anymore, I think it would be nicer to
use 'boundary' without underscores.
This applies to the changes below as well.
> + return (__boundary - 1 < end - 1) ? __boundary : end;
> +}
> #endif
>
> #ifndef p4d_addr_end
> -#define p4d_addr_end(p4d, addr, end) \
> -({ unsigned long __boundary = ((addr) + P4D_SIZE) & P4D_MASK; \
> - (__boundary - 1 < (end) - 1)? __boundary: (end); \
> -})
> +#define p4d_addr_end p4d_addr_end
> +static inline unsigned long p4d_addr_end(p4d_t p4d, unsigned long addr, unsigned long end)
> +{ unsigned long __boundary = (addr + P4D_SIZE) & P4D_MASK;
> + return (__boundary - 1 < end - 1) ? __boundary : end;
> +}
> #endif
>
> #ifndef pud_addr_end
> -#define pud_addr_end(pud, addr, end) \
> -({ unsigned long __boundary = ((addr) + PUD_SIZE) & PUD_MASK; \
> - (__boundary - 1 < (end) - 1)? __boundary: (end); \
> -})
> +#define pud_addr_end pud_addr_end
> +static inline unsigned long pud_addr_end(pud_t pud, unsigned long addr, unsigned long end)
> +{ unsigned long __boundary = (addr + PUD_SIZE) & PUD_MASK;
> + return (__boundary - 1 < end - 1) ? __boundary : end;
> +}
> #endif
>
> #ifndef pmd_addr_end
> -#define pmd_addr_end(pmd, addr, end) \
> -({ unsigned long __boundary = ((addr) + PMD_SIZE) & PMD_MASK; \
> - (__boundary - 1 < (end) - 1)? __boundary: (end); \
> -})
> +#define pmd_addr_end pmd_addr_end
> +static inline unsigned long pmd_addr_end(pmd_t pmd, unsigned long addr, unsigned long end)
> +{ unsigned long __boundary = (addr + PMD_SIZE) & PMD_MASK;
> + return (__boundary - 1 < end - 1) ? __boundary : end;
> +}
> #endif
>
> /*
> --
> 2.17.1
>
--
Sincerely yours,
Mike.
^ permalink raw reply
* Re: [PATCH AUTOSEL 5.8 14/53] ibmvnic fix NULL tx_pools and rx_tools issue at do_reset
From: Jakub Kicinski @ 2020-09-07 21:10 UTC (permalink / raw)
To: Sasha Levin
Cc: netdev, linux-kernel, stable, Mingming Cao, Dany Madden,
linuxppc-dev, David S . Miller
In-Reply-To: <20200907163220.1280412-14-sashal@kernel.org>
On Mon, 7 Sep 2020 12:31:40 -0400 Sasha Levin wrote:
> [ Upstream commit 9f13457377907fa253aef560e1a37e1ca4197f9b ]
> @@ -2024,10 +2033,14 @@ static int do_reset(struct ibmvnic_adapter *adapter,
> } else {
> rc = reset_tx_pools(adapter);
> if (rc)
> + netdev_dbg(adapter->netdev, "reset tx pools failed (%d)\n",
> + rc);
> goto out;
>
> rc = reset_rx_pools(adapter);
> if (rc)
> + netdev_dbg(adapter->netdev, "reset rx pools failed (%d)\n",
> + rc);
> goto out;
> }
> ibmvnic_disable_irqs(adapter);
Hi Sasha!
I just pushed this to net:
8ae4dff882eb ("ibmvnic: add missing parenthesis in do_reset()")
You definitely want to pull that in if you decide to backport this one.
^ permalink raw reply
* Re: [PATCH AUTOSEL 5.8 14/53] ibmvnic fix NULL tx_pools and rx_tools issue at do_reset
From: Sasha Levin @ 2020-09-07 22:24 UTC (permalink / raw)
To: Jakub Kicinski
Cc: netdev, linux-kernel, stable, Mingming Cao, Dany Madden,
linuxppc-dev, David S . Miller
In-Reply-To: <20200907141026.093fc160@kicinski-fedora-pc1c0hjn.dhcp.thefacebook.com>
On Mon, Sep 07, 2020 at 02:10:26PM -0700, Jakub Kicinski wrote:
>On Mon, 7 Sep 2020 12:31:40 -0400 Sasha Levin wrote:
>> [ Upstream commit 9f13457377907fa253aef560e1a37e1ca4197f9b ]
>
>> @@ -2024,10 +2033,14 @@ static int do_reset(struct ibmvnic_adapter *adapter,
>> } else {
>> rc = reset_tx_pools(adapter);
>> if (rc)
>> + netdev_dbg(adapter->netdev, "reset tx pools failed (%d)\n",
>> + rc);
>> goto out;
>>
>> rc = reset_rx_pools(adapter);
>> if (rc)
>> + netdev_dbg(adapter->netdev, "reset rx pools failed (%d)\n",
>> + rc);
>> goto out;
>> }
>> ibmvnic_disable_irqs(adapter);
>
>Hi Sasha!
>
>I just pushed this to net:
>
>8ae4dff882eb ("ibmvnic: add missing parenthesis in do_reset()")
>
>You definitely want to pull that in if you decide to backport this one.
Will do, thanks!
--
Thanks,
Sasha
^ permalink raw reply
* [PATCH kernel] powerpc/dma: Fix dma_map_ops::get_required_mask
From: Alexey Kardashevskiy @ 2020-09-08 1:51 UTC (permalink / raw)
To: linuxppc-dev
Cc: Alexey Kardashevskiy, Oliver O'Halloran, Christoph Hellwig,
Cédric Le Goater
There are 2 problems with it:
1. "<" vs expected "<<"
2. the shift number is an IOMMU page number mask, not an address mask
as the IOMMU page shift is missing.
This did not hit us before f1565c24b596 ("powerpc: use the generic
dma_ops_bypass mode") because we had there additional code to handle
bypass mask so this chunk (almost?) never executed. However there
were reports that aacraid does not work with "iommu=nobypass".
After f1565c24b596, aacraid (and probably others which call
dma_get_required_mask() before setting the mask) was unable to
enable 64bit DMA and fall back to using IOMMU which was known not to work,
one of the problems is double free of an IOMMU page.
This fixes DMA for aacraid, both with and without "iommu=nobypass"
in the kernel command line. Verified with "stress-ng -d 4".
Fixes: f1565c24b596 ("powerpc: use the generic dma_ops_bypass mode")
Signed-off-by: Alexey Kardashevskiy <aik@ozlabs.ru>
---
The original code came Jun 24 2011:
6a5c7be5e484 ("powerpc: Override dma_get_required_mask by platform hook and ops")
What is dma_get_required_mask() for anyway? What "requires" what here?
Even though it works for now (due to huge - >4GB - default DMA window),
I am still not convinced we do not want this chunk here
(this is what f1565c24b596 removed):
if (dev_is_pci(dev)) {
u64 bypass_mask = dma_direct_get_required_mask(dev);
if (dma_iommu_bypass_supported(dev, bypass_mask))
return bypass_mask;
}
---
arch/powerpc/kernel/dma-iommu.c | 3 ++-
1 file changed, 2 insertions(+), 1 deletion(-)
diff --git a/arch/powerpc/kernel/dma-iommu.c b/arch/powerpc/kernel/dma-iommu.c
index 569fecd7b5b2..9053fc9d20c7 100644
--- a/arch/powerpc/kernel/dma-iommu.c
+++ b/arch/powerpc/kernel/dma-iommu.c
@@ -120,7 +120,8 @@ u64 dma_iommu_get_required_mask(struct device *dev)
if (!tbl)
return 0;
- mask = 1ULL < (fls_long(tbl->it_offset + tbl->it_size) - 1);
+ mask = 1ULL << (fls_long(tbl->it_offset + tbl->it_size) +
+ tbl->it_page_shift - 1);
mask += mask - 1;
return mask;
--
2.17.1
^ permalink raw reply related
* Re: [PATCH v1 02/10] powerpc/kernel/iommu: Align size for IOMMU_PAGE_SIZE on iommu_*_coherent()
From: Alexey Kardashevskiy @ 2020-09-08 3:18 UTC (permalink / raw)
To: Leonardo Bras, Michael Ellerman, Benjamin Herrenschmidt,
Paul Mackerras, Christophe Leroy, Joel Stanley,
Thiago Jung Bauermann, Ram Pai, Brian King,
Murilo Fossa Vicentini, David Dai
Cc: linuxppc-dev, linux-kernel
In-Reply-To: <ef7e80b0a7399bad607324301a604bfb46c2de05.camel@gmail.com>
On 04/09/2020 16:04, Leonardo Bras wrote:
> On Thu, 2020-09-03 at 14:41 +1000, Alexey Kardashevskiy wrote:
>> I am new to this, so I am trying to understand how a memory page mapped
>>> as DMA, and used for something else could be a problem.
>>
>> From the device prospective, there is PCI space and everything from 0
>> till 1<<64 is accessible and what is that mapped to - the device does
>> not know. PHB's IOMMU is the thing to notice invalid access and raise
>> EEH but PHB only knows about PCI->physical memory mapping (with IOMMU
>> pages) but nothing about the host kernel pages. Does this help? Thanks,
>
> According to our conversation on Slack:
> 1- There is a problem if a hypervisor gives to it's VMs contiguous
> memory blocks that are not aligned to IOMMU pages, because then an
> iommu_map_page() could map some memory in this VM and some memory in
> other VM / process.
> 2- To guarantee this, we should have system pagesize >= iommu_pagesize
>
> One way to get (2) is by doing this in enable_ddw():
> if ((query.page_size & 4) && PAGE_SHIFT >= 24) {
You won't ever (well, soon) see PAGE_SHIFT==24, it is either 4K or 64K.
However 16MB IOMMU pages is fine - if hypervisor uses huge pages for VMs
RAM, it also then advertises huge IOMMU pages in ddw-query. So for the
1:1 case there must be no "PAGE_SHIFT >= 24".
> page_shift = 24; /* 16MB */
> } else if ((query.page_size & 2) && PAGE_SHIFT >= 16 ) {
> page_shift = 16; /* 64kB */
> } else if (query.page_size & 1 && PAGE_SHIFT >= 12) {
> page_shift = 12; /* 4kB */
> [...]
>
> Another way of solving this, would be adding in LoPAR documentation
> that the blocksize of contiguous memory the hypervisor gives a VM
> should always be aligned to IOMMU pagesize offered.
I think this is assumed already by the design of the DDW API.
>
> I think the best approach would be first sending the above patch, which
> is faster, and then get working into adding that to documentation, so
> hypervisors guarantee this.
>
> If this gets into the docs, we can revert the patch.
>
> What do you think?
I think we diverted from the original patch :) I am not quite sure what
you were fixing there. Thanks,
--
Alexey
^ permalink raw reply
* [PATCH v2] kbuild: preprocess module linker script
From: Masahiro Yamada @ 2020-09-08 4:27 UTC (permalink / raw)
To: linux-kbuild
Cc: linux-ia64, Catalin Marinas, Paul Mackerras, linux-riscv,
Will Deacon, Anton Ivanov, linux-arch, Richard Weinberger,
Masahiro Yamada, Russell King, Geert Uytterhoeven, Fenghua Yu,
Albert Ou, Arnd Bergmann, Jeff Dike, Jessica Yu, linux-um,
linux-m68k, Michal Marek, Paul Walmsley, linux-arm-kernel,
Tony Luck, linux-kernel, Palmer Dabbelt, linuxppc-dev
There was a request to preprocess the module linker script like we
do for the vmlinux one. (https://lkml.org/lkml/2020/8/21/512)
The difference between vmlinux.lds and module.lds is that the latter
is needed for external module builds, thus must be cleaned up by
'make mrproper' instead of 'make clean'. Also, it must be created
by 'make modules_prepare'.
You cannot put it in arch/$(SRCARCH)/kernel/, which is cleaned up by
'make clean'. I moved arch/$(SRCARCH)/kernel/module.lds to
arch/$(SRCARCH)/include/asm/module.lds.h, which is included from
scripts/module.lds.S.
scripts/module.lds is fine because 'make clean' keeps all the
build artifacts under scripts/.
You can add arch-specific sections in <asm/module.lds.h>.
Signed-off-by: Masahiro Yamada <masahiroy@kernel.org>
Tested-by: Jessica Yu <jeyu@kernel.org>
Acked-by: Will Deacon <will@kernel.org>
---
Changes in v2:
- Fix the race between the two targets 'scripts' and 'asm-generic'
Makefile | 10 ++++++----
arch/arm/Makefile | 4 ----
.../{kernel/module.lds => include/asm/module.lds.h} | 2 ++
arch/arm64/Makefile | 4 ----
.../{kernel/module.lds => include/asm/module.lds.h} | 2 ++
arch/ia64/Makefile | 1 -
arch/ia64/{module.lds => include/asm/module.lds.h} | 0
arch/m68k/Makefile | 1 -
.../{kernel/module.lds => include/asm/module.lds.h} | 0
arch/powerpc/Makefile | 1 -
.../{kernel/module.lds => include/asm/module.lds.h} | 0
arch/riscv/Makefile | 3 ---
.../{kernel/module.lds => include/asm/module.lds.h} | 3 ++-
arch/um/include/asm/Kbuild | 1 +
include/asm-generic/Kbuild | 1 +
include/asm-generic/module.lds.h | 10 ++++++++++
scripts/.gitignore | 1 +
scripts/Makefile | 3 +++
scripts/Makefile.modfinal | 5 ++---
scripts/{module-common.lds => module.lds.S} | 3 +++
scripts/package/builddeb | 2 +-
21 files changed, 34 insertions(+), 23 deletions(-)
rename arch/arm/{kernel/module.lds => include/asm/module.lds.h} (72%)
rename arch/arm64/{kernel/module.lds => include/asm/module.lds.h} (76%)
rename arch/ia64/{module.lds => include/asm/module.lds.h} (100%)
rename arch/m68k/{kernel/module.lds => include/asm/module.lds.h} (100%)
rename arch/powerpc/{kernel/module.lds => include/asm/module.lds.h} (100%)
rename arch/riscv/{kernel/module.lds => include/asm/module.lds.h} (84%)
create mode 100644 include/asm-generic/module.lds.h
rename scripts/{module-common.lds => module.lds.S} (93%)
diff --git a/Makefile b/Makefile
index 37739ee53f27..97b1dae1783b 100644
--- a/Makefile
+++ b/Makefile
@@ -505,7 +505,6 @@ KBUILD_CFLAGS_KERNEL :=
KBUILD_AFLAGS_MODULE := -DMODULE
KBUILD_CFLAGS_MODULE := -DMODULE
KBUILD_LDFLAGS_MODULE :=
-export KBUILD_LDS_MODULE := $(srctree)/scripts/module-common.lds
KBUILD_LDFLAGS :=
CLANG_FLAGS :=
@@ -1395,7 +1394,7 @@ endif
# using awk while concatenating to the final file.
PHONY += modules
-modules: $(if $(KBUILD_BUILTIN),vmlinux) modules_check
+modules: $(if $(KBUILD_BUILTIN),vmlinux) modules_check modules_prepare
$(Q)$(MAKE) -f $(srctree)/scripts/Makefile.modpost
PHONY += modules_check
@@ -1412,6 +1411,7 @@ targets += modules.order
# Target to prepare building external modules
PHONY += modules_prepare
modules_prepare: prepare
+ $(Q)$(MAKE) $(build)=scripts scripts/module.lds
# Target to install modules
PHONY += modules_install
@@ -1743,7 +1743,9 @@ help:
@echo ' clean - remove generated files in module directory only'
@echo ''
-PHONY += prepare
+# no-op for external module builds
+PHONY += prepare modules_prepare
+
endif # KBUILD_EXTMOD
# Single targets
@@ -1776,7 +1778,7 @@ MODORDER := .modules.tmp
endif
PHONY += single_modpost
-single_modpost: $(single-no-ko)
+single_modpost: $(single-no-ko) modules_prepare
$(Q){ $(foreach m, $(single-ko), echo $(extmod-prefix)$m;) } > $(MODORDER)
$(Q)$(MAKE) -f $(srctree)/scripts/Makefile.modpost
diff --git a/arch/arm/Makefile b/arch/arm/Makefile
index 4e877354515f..a0cb15de9677 100644
--- a/arch/arm/Makefile
+++ b/arch/arm/Makefile
@@ -16,10 +16,6 @@ LDFLAGS_vmlinux += --be8
KBUILD_LDFLAGS_MODULE += --be8
endif
-ifeq ($(CONFIG_ARM_MODULE_PLTS),y)
-KBUILD_LDS_MODULE += $(srctree)/arch/arm/kernel/module.lds
-endif
-
GZFLAGS :=-9
#KBUILD_CFLAGS +=-pipe
diff --git a/arch/arm/kernel/module.lds b/arch/arm/include/asm/module.lds.h
similarity index 72%
rename from arch/arm/kernel/module.lds
rename to arch/arm/include/asm/module.lds.h
index 79cb6af565e5..0e7cb4e314b4 100644
--- a/arch/arm/kernel/module.lds
+++ b/arch/arm/include/asm/module.lds.h
@@ -1,5 +1,7 @@
/* SPDX-License-Identifier: GPL-2.0 */
+#ifdef CONFIG_ARM_MODULE_PLTS
SECTIONS {
.plt : { BYTE(0) }
.init.plt : { BYTE(0) }
}
+#endif
diff --git a/arch/arm64/Makefile b/arch/arm64/Makefile
index b45f0124cc16..76667ad47980 100644
--- a/arch/arm64/Makefile
+++ b/arch/arm64/Makefile
@@ -115,10 +115,6 @@ endif
CHECKFLAGS += -D__aarch64__
-ifeq ($(CONFIG_ARM64_MODULE_PLTS),y)
-KBUILD_LDS_MODULE += $(srctree)/arch/arm64/kernel/module.lds
-endif
-
ifeq ($(CONFIG_DYNAMIC_FTRACE_WITH_REGS),y)
KBUILD_CPPFLAGS += -DCC_USING_PATCHABLE_FUNCTION_ENTRY
CC_FLAGS_FTRACE := -fpatchable-function-entry=2
diff --git a/arch/arm64/kernel/module.lds b/arch/arm64/include/asm/module.lds.h
similarity index 76%
rename from arch/arm64/kernel/module.lds
rename to arch/arm64/include/asm/module.lds.h
index 22e36a21c113..691f15af788e 100644
--- a/arch/arm64/kernel/module.lds
+++ b/arch/arm64/include/asm/module.lds.h
@@ -1,5 +1,7 @@
+#ifdef CONFIG_ARM64_MODULE_PLTS
SECTIONS {
.plt (NOLOAD) : { BYTE(0) }
.init.plt (NOLOAD) : { BYTE(0) }
.text.ftrace_trampoline (NOLOAD) : { BYTE(0) }
}
+#endif
diff --git a/arch/ia64/Makefile b/arch/ia64/Makefile
index 2876a7df1b0a..703b1c4f6d12 100644
--- a/arch/ia64/Makefile
+++ b/arch/ia64/Makefile
@@ -20,7 +20,6 @@ CHECKFLAGS += -D__ia64=1 -D__ia64__=1 -D_LP64 -D__LP64__
OBJCOPYFLAGS := --strip-all
LDFLAGS_vmlinux := -static
-KBUILD_LDS_MODULE += $(srctree)/arch/ia64/module.lds
KBUILD_AFLAGS_KERNEL := -mconstant-gp
EXTRA :=
diff --git a/arch/ia64/module.lds b/arch/ia64/include/asm/module.lds.h
similarity index 100%
rename from arch/ia64/module.lds
rename to arch/ia64/include/asm/module.lds.h
diff --git a/arch/m68k/Makefile b/arch/m68k/Makefile
index 4438ffb4bbe1..ea14f2046fb4 100644
--- a/arch/m68k/Makefile
+++ b/arch/m68k/Makefile
@@ -75,7 +75,6 @@ KBUILD_CPPFLAGS += -D__uClinux__
endif
KBUILD_LDFLAGS := -m m68kelf
-KBUILD_LDS_MODULE += $(srctree)/arch/m68k/kernel/module.lds
ifdef CONFIG_SUN3
LDFLAGS_vmlinux = -N
diff --git a/arch/m68k/kernel/module.lds b/arch/m68k/include/asm/module.lds.h
similarity index 100%
rename from arch/m68k/kernel/module.lds
rename to arch/m68k/include/asm/module.lds.h
diff --git a/arch/powerpc/Makefile b/arch/powerpc/Makefile
index 3e8da9cf2eb9..8935658fcd06 100644
--- a/arch/powerpc/Makefile
+++ b/arch/powerpc/Makefile
@@ -65,7 +65,6 @@ UTS_MACHINE := $(subst $(space),,$(machine-y))
ifdef CONFIG_PPC32
KBUILD_LDFLAGS_MODULE += arch/powerpc/lib/crtsavres.o
else
-KBUILD_LDS_MODULE += $(srctree)/arch/powerpc/kernel/module.lds
ifeq ($(call ld-ifversion, -ge, 225000000, y),y)
# Have the linker provide sfpr if possible.
# There is a corresponding test in arch/powerpc/lib/Makefile
diff --git a/arch/powerpc/kernel/module.lds b/arch/powerpc/include/asm/module.lds.h
similarity index 100%
rename from arch/powerpc/kernel/module.lds
rename to arch/powerpc/include/asm/module.lds.h
diff --git a/arch/riscv/Makefile b/arch/riscv/Makefile
index fb6e37db836d..8edaa8bd86d6 100644
--- a/arch/riscv/Makefile
+++ b/arch/riscv/Makefile
@@ -53,9 +53,6 @@ endif
ifeq ($(CONFIG_CMODEL_MEDANY),y)
KBUILD_CFLAGS += -mcmodel=medany
endif
-ifeq ($(CONFIG_MODULE_SECTIONS),y)
- KBUILD_LDS_MODULE += $(srctree)/arch/riscv/kernel/module.lds
-endif
ifeq ($(CONFIG_PERF_EVENTS),y)
KBUILD_CFLAGS += -fno-omit-frame-pointer
endif
diff --git a/arch/riscv/kernel/module.lds b/arch/riscv/include/asm/module.lds.h
similarity index 84%
rename from arch/riscv/kernel/module.lds
rename to arch/riscv/include/asm/module.lds.h
index 295ecfb341a2..4254ff2ff049 100644
--- a/arch/riscv/kernel/module.lds
+++ b/arch/riscv/include/asm/module.lds.h
@@ -1,8 +1,9 @@
/* SPDX-License-Identifier: GPL-2.0 */
/* Copyright (C) 2017 Andes Technology Corporation */
-
+#ifdef CONFIG_MODULE_SECTIONS
SECTIONS {
.plt (NOLOAD) : { BYTE(0) }
.got (NOLOAD) : { BYTE(0) }
.got.plt (NOLOAD) : { BYTE(0) }
}
+#endif
diff --git a/arch/um/include/asm/Kbuild b/arch/um/include/asm/Kbuild
index 8d435f8a6dec..1c63b260ecc4 100644
--- a/arch/um/include/asm/Kbuild
+++ b/arch/um/include/asm/Kbuild
@@ -16,6 +16,7 @@ generic-y += kdebug.h
generic-y += mcs_spinlock.h
generic-y += mm-arch-hooks.h
generic-y += mmiowb.h
+generic-y += module.lds.h
generic-y += param.h
generic-y += pci.h
generic-y += percpu.h
diff --git a/include/asm-generic/Kbuild b/include/asm-generic/Kbuild
index 74b0612601dd..7cd4e627e00e 100644
--- a/include/asm-generic/Kbuild
+++ b/include/asm-generic/Kbuild
@@ -40,6 +40,7 @@ mandatory-y += mmiowb.h
mandatory-y += mmu.h
mandatory-y += mmu_context.h
mandatory-y += module.h
+mandatory-y += module.lds.h
mandatory-y += msi.h
mandatory-y += pci.h
mandatory-y += percpu.h
diff --git a/include/asm-generic/module.lds.h b/include/asm-generic/module.lds.h
new file mode 100644
index 000000000000..f210d5c1b78b
--- /dev/null
+++ b/include/asm-generic/module.lds.h
@@ -0,0 +1,10 @@
+/* SPDX-License-Identifier: GPL-2.0-only */
+#ifndef __ASM_GENERIC_MODULE_LDS_H
+#define __ASM_GENERIC_MODULE_LDS_H
+
+/*
+ * <asm/module.lds.h> can specify arch-specific sections for linking modules.
+ * Empty for the asm-generic header.
+ */
+
+#endif /* __ASM_GENERIC_MODULE_LDS_H */
diff --git a/scripts/.gitignore b/scripts/.gitignore
index 0d1c8e217cd7..a6c11316c969 100644
--- a/scripts/.gitignore
+++ b/scripts/.gitignore
@@ -8,3 +8,4 @@ asn1_compiler
extract-cert
sign-file
insert-sys-cert
+/module.lds
diff --git a/scripts/Makefile b/scripts/Makefile
index bc018e4b733e..b5418ec587fb 100644
--- a/scripts/Makefile
+++ b/scripts/Makefile
@@ -29,6 +29,9 @@ endif
# The following programs are only built on demand
hostprogs += unifdef
+# The module linker script is preprocessed on demand
+targets += module.lds
+
subdir-$(CONFIG_GCC_PLUGINS) += gcc-plugins
subdir-$(CONFIG_MODVERSIONS) += genksyms
subdir-$(CONFIG_SECURITY_SELINUX) += selinux
diff --git a/scripts/Makefile.modfinal b/scripts/Makefile.modfinal
index 411c1e600e7d..ae01baf96f4e 100644
--- a/scripts/Makefile.modfinal
+++ b/scripts/Makefile.modfinal
@@ -33,11 +33,10 @@ quiet_cmd_ld_ko_o = LD [M] $@
cmd_ld_ko_o = \
$(LD) -r $(KBUILD_LDFLAGS) \
$(KBUILD_LDFLAGS_MODULE) $(LDFLAGS_MODULE) \
- $(addprefix -T , $(KBUILD_LDS_MODULE)) \
- -o $@ $(filter %.o, $^); \
+ -T scripts/module.lds -o $@ $(filter %.o, $^); \
$(if $(ARCH_POSTLINK), $(MAKE) -f $(ARCH_POSTLINK) $@, true)
-$(modules): %.ko: %.o %.mod.o $(KBUILD_LDS_MODULE) FORCE
+$(modules): %.ko: %.o %.mod.o scripts/module.lds FORCE
+$(call if_changed,ld_ko_o)
targets += $(modules) $(modules:.ko=.mod.o)
diff --git a/scripts/module-common.lds b/scripts/module.lds.S
similarity index 93%
rename from scripts/module-common.lds
rename to scripts/module.lds.S
index d61b9e8678e8..69b9b71a6a47 100644
--- a/scripts/module-common.lds
+++ b/scripts/module.lds.S
@@ -24,3 +24,6 @@ SECTIONS {
__jump_table 0 : ALIGN(8) { KEEP(*(__jump_table)) }
}
+
+/* bring in arch-specific sections */
+#include <asm/module.lds.h>
diff --git a/scripts/package/builddeb b/scripts/package/builddeb
index 6df3c9f8b2da..44f212e37935 100755
--- a/scripts/package/builddeb
+++ b/scripts/package/builddeb
@@ -55,7 +55,7 @@ deploy_kernel_headers () {
cd $srctree
find . arch/$SRCARCH -maxdepth 1 -name Makefile\*
find include scripts -type f -o -type l
- find arch/$SRCARCH -name module.lds -o -name Kbuild.platforms -o -name Platform
+ find arch/$SRCARCH -name Kbuild.platforms -o -name Platform
find $(find arch/$SRCARCH -name include -o -name scripts -type d) -type f
) > debian/hdrsrcfiles
--
2.25.1
^ permalink raw reply related
* Re: [RFC PATCH v2 0/3] mm/gup: fix gup_fast with dynamic page table folding
From: Christophe Leroy @ 2020-09-08 4:42 UTC (permalink / raw)
To: Gerald Schaefer, Jason Gunthorpe, John Hubbard
Cc: Peter Zijlstra, Catalin Marinas, Dave Hansen, linux-mm,
Paul Mackerras, linux-sparc, Alexander Gordeev, Claudio Imbrenda,
Will Deacon, linux-arch, linux-s390, Vasily Gorbik,
Richard Weinberger, linux-x86, Russell King,
Christian Borntraeger, Ingo Molnar, Andrey Ryabinin, Jeff Dike,
Arnd Bergmann, Heiko Carstens, linux-um, Borislav Petkov,
Andy Lutomirski, Thomas Gleixner, linux-arm, Linus Torvalds, LKML,
Andrew Morton, linux-power, Mike Rapoport
In-Reply-To: <20200907180058.64880-1-gerald.schaefer@linux.ibm.com>
Le 07/09/2020 à 20:00, Gerald Schaefer a écrit :
> This is v2 of an RFC previously discussed here:
> https://lore.kernel.org/lkml/20200828140314.8556-1-gerald.schaefer@linux.ibm.com/
>
> Patch 1 is a fix for a regression in gup_fast on s390, after our conversion
> to common gup_fast code. It will introduce special helper functions
> pXd_addr_end_folded(), which have to be used in places where pagetable walk
> is done w/o lock and with READ_ONCE, so currently only in gup_fast.
>
> Patch 2 is an attempt to make that more generic, i.e. change pXd_addr_end()
> themselves by adding an extra pXd value parameter. That was suggested by
> Jason during v1 discussion, because he is already thinking of some other
> places where he might want to switch to the READ_ONCE logic for pagetable
> walks. In general, that would be the cleanest / safest solution, but there
> is some impact on other architectures and common code, hence the new and
> greatly enlarged recipient list.
>
> Patch 3 is a "nice to have" add-on, which makes pXd_addr_end() inline
> functions instead of #defines, so that we get some type checking for the
> new pXd value parameter.
>
> Not sure about Fixes/stable tags for the generic solution. Only patch 1
> fixes a real bug on s390, and has Fixes/stable tags. Patches 2 + 3 might
> still be nice to have in stable, to ease future backports, but I guess
> "nice to have" does not really qualify for stable backports.
If one day you have to backport a fix that requires patch 2 and/or 3,
just mark it "depends-on:" and the patches will go in stable at the
relevant time.
Christophe
^ permalink raw reply
* Re: [RFC PATCH v2 1/3] mm/gup: fix gup_fast with dynamic page table folding
From: Christophe Leroy @ 2020-09-08 5:06 UTC (permalink / raw)
To: Gerald Schaefer, Jason Gunthorpe, John Hubbard
Cc: Peter Zijlstra, Catalin Marinas, Dave Hansen, linux-mm,
Paul Mackerras, linux-sparc, Alexander Gordeev, Claudio Imbrenda,
Will Deacon, linux-arch, linux-s390, Vasily Gorbik,
Richard Weinberger, linux-x86, Russell King,
Christian Borntraeger, Ingo Molnar, Andrey Ryabinin, Jeff Dike,
Arnd Bergmann, Heiko Carstens, linux-um, Borislav Petkov,
Andy Lutomirski, Thomas Gleixner, linux-arm, Linus Torvalds, LKML,
Andrew Morton, linux-power, Mike Rapoport
In-Reply-To: <20200907180058.64880-2-gerald.schaefer@linux.ibm.com>
Le 07/09/2020 à 20:00, Gerald Schaefer a écrit :
> From: Alexander Gordeev <agordeev@linux.ibm.com>
>
> Commit 1a42010cdc26 ("s390/mm: convert to the generic get_user_pages_fast
> code") introduced a subtle but severe bug on s390 with gup_fast, due to
> dynamic page table folding.
>
> The question "What would it require for the generic code to work for s390"
> has already been discussed here
> https://lkml.kernel.org/r/20190418100218.0a4afd51@mschwideX1
> and ended with a promising approach here
> https://lkml.kernel.org/r/20190419153307.4f2911b5@mschwideX1
> which in the end unfortunately didn't quite work completely.
>
> We tried to mimic static level folding by changing pgd_offset to always
> calculate top level page table offset, and do nothing in folded pXd_offset.
> What has been overlooked is that PxD_SIZE/MASK and thus pXd_addr_end do
> not reflect this dynamic behaviour, and still act like static 5-level
> page tables.
>
[...]
>
> Fix this by introducing new pXd_addr_end_folded helpers, which take an
> additional pXd entry value parameter, that can be used on s390
> to determine the correct page table level and return corresponding
> end / boundary. With that, the pointer iteration will always
> happen in gup_pgd_range for s390. No change for other architectures
> introduced.
Not sure pXd_addr_end_folded() is the best understandable name,
allthough I don't have any alternative suggestion at the moment.
Maybe could be something like pXd_addr_end_fixup() as it will disappear
in the next patch, or pXd_addr_end_gup() ?
Also, if it happens to be acceptable to get patch 2 in stable, I think
you should switch patch 1 and patch 2 to avoid the step through
pXd_addr_end_folded()
>
> Fixes: 1a42010cdc26 ("s390/mm: convert to the generic get_user_pages_fast code")
> Cc: <stable@vger.kernel.org> # 5.2+
> Reviewed-by: Gerald Schaefer <gerald.schaefer@linux.ibm.com>
> Signed-off-by: Alexander Gordeev <agordeev@linux.ibm.com>
> Signed-off-by: Gerald Schaefer <gerald.schaefer@linux.ibm.com>
> ---
> arch/s390/include/asm/pgtable.h | 42 +++++++++++++++++++++++++++++++++
> include/linux/pgtable.h | 16 +++++++++++++
> mm/gup.c | 8 +++----
> 3 files changed, 62 insertions(+), 4 deletions(-)
>
> diff --git a/arch/s390/include/asm/pgtable.h b/arch/s390/include/asm/pgtable.h
> index 7eb01a5459cd..027206e4959d 100644
> --- a/arch/s390/include/asm/pgtable.h
> +++ b/arch/s390/include/asm/pgtable.h
> @@ -512,6 +512,48 @@ static inline bool mm_pmd_folded(struct mm_struct *mm)
> }
> #define mm_pmd_folded(mm) mm_pmd_folded(mm)
>
> +/*
> + * With dynamic page table levels on s390, the static pXd_addr_end() functions
> + * will not return corresponding dynamic boundaries. This is no problem as long
> + * as only pXd pointers are passed down during page table walk, because
> + * pXd_offset() will simply return the given pointer for folded levels, and the
> + * pointer iteration over a range simply happens at the correct page table
> + * level.
> + * It is however a problem with gup_fast, or other places walking the page
> + * tables w/o locks using READ_ONCE(), and passing down the pXd values instead
> + * of pointers. In this case, the pointer given to pXd_offset() is a pointer to
> + * a stack variable, which cannot be used for pointer iteration at the correct
> + * level. Instead, the iteration then has to happen by going up to pgd level
> + * again. To allow this, provide pXd_addr_end_folded() functions with an
> + * additional pXd value parameter, which can be used on s390 to determine the
> + * folding level and return the corresponding boundary.
> + */
> +static inline unsigned long rste_addr_end_folded(unsigned long rste, unsigned long addr, unsigned long end)
What does 'rste' stands for ?
Isn't this line a bit long ?
> +{
> + unsigned long type = (rste & _REGION_ENTRY_TYPE_MASK) >> 2;
> + unsigned long size = 1UL << (_SEGMENT_SHIFT + type * 11);
> + unsigned long boundary = (addr + size) & ~(size - 1);
> +
> + /*
> + * FIXME The below check is for internal testing only, to be removed
> + */
> + VM_BUG_ON(type < (_REGION_ENTRY_TYPE_R3 >> 2));
> +
> + return (boundary - 1) < (end - 1) ? boundary : end;
> +}
> +
> +#define pgd_addr_end_folded pgd_addr_end_folded
> +static inline unsigned long pgd_addr_end_folded(pgd_t pgd, unsigned long addr, unsigned long end)
> +{
> + return rste_addr_end_folded(pgd_val(pgd), addr, end);
> +}
> +
> +#define p4d_addr_end_folded p4d_addr_end_folded
> +static inline unsigned long p4d_addr_end_folded(p4d_t p4d, unsigned long addr, unsigned long end)
> +{
> + return rste_addr_end_folded(p4d_val(p4d), addr, end);
> +}
> +
> static inline int mm_has_pgste(struct mm_struct *mm)
> {
> #ifdef CONFIG_PGSTE
> diff --git a/include/linux/pgtable.h b/include/linux/pgtable.h
> index e8cbc2e795d5..981c4c2a31fe 100644
> --- a/include/linux/pgtable.h
> +++ b/include/linux/pgtable.h
> @@ -681,6 +681,22 @@ static inline int arch_unmap_one(struct mm_struct *mm,
> })
> #endif
>
> +#ifndef pgd_addr_end_folded
> +#define pgd_addr_end_folded(pgd, addr, end) pgd_addr_end(addr, end)
> +#endif
> +
> +#ifndef p4d_addr_end_folded
> +#define p4d_addr_end_folded(p4d, addr, end) p4d_addr_end(addr, end)
> +#endif
> +
> +#ifndef pud_addr_end_folded
> +#define pud_addr_end_folded(pud, addr, end) pud_addr_end(addr, end)
> +#endif
> +
> +#ifndef pmd_addr_end_folded
> +#define pmd_addr_end_folded(pmd, addr, end) pmd_addr_end(addr, end)
> +#endif
> +
> /*
> * When walking page tables, we usually want to skip any p?d_none entries;
> * and any p?d_bad entries - reporting the error before resetting to none.
> diff --git a/mm/gup.c b/mm/gup.c
> index bd883a112724..ba4aace5d0f4 100644
> --- a/mm/gup.c
> +++ b/mm/gup.c
> @@ -2521,7 +2521,7 @@ static int gup_pmd_range(pud_t pud, unsigned long addr, unsigned long end,
> do {
> pmd_t pmd = READ_ONCE(*pmdp);
>
> - next = pmd_addr_end(addr, end);
> + next = pmd_addr_end_folded(pmd, addr, end);
> if (!pmd_present(pmd))
> return 0;
>
> @@ -2564,7 +2564,7 @@ static int gup_pud_range(p4d_t p4d, unsigned long addr, unsigned long end,
> do {
> pud_t pud = READ_ONCE(*pudp);
>
> - next = pud_addr_end(addr, end);
> + next = pud_addr_end_folded(pud, addr, end);
> if (unlikely(!pud_present(pud)))
> return 0;
> if (unlikely(pud_huge(pud))) {
> @@ -2592,7 +2592,7 @@ static int gup_p4d_range(pgd_t pgd, unsigned long addr, unsigned long end,
> do {
> p4d_t p4d = READ_ONCE(*p4dp);
>
> - next = p4d_addr_end(addr, end);
> + next = p4d_addr_end_folded(p4d, addr, end);
> if (p4d_none(p4d))
> return 0;
> BUILD_BUG_ON(p4d_huge(p4d));
> @@ -2617,7 +2617,7 @@ static void gup_pgd_range(unsigned long addr, unsigned long end,
> do {
> pgd_t pgd = READ_ONCE(*pgdp);
>
> - next = pgd_addr_end(addr, end);
> + next = pgd_addr_end_folded(pgd, addr, end);
> if (pgd_none(pgd))
> return;
> if (unlikely(pgd_huge(pgd))) {
>
Christophe
^ permalink raw reply
page: next (older) | prev (newer) | latest
- recent:[subjects (threaded)|topics (new)|topics (active)]
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox