* Re: [RFC PATCH v2 2/3] mm: make pXd_addr_end() functions page-table entry aware
From: Alexander Gordeev @ 2020-09-08 7:46 UTC (permalink / raw)
To: Christophe Leroy, Michael Ellerman
Cc: Peter Zijlstra, Catalin Marinas, Dave Hansen, linux-mm,
Paul Mackerras, linux-sparc, Claudio Imbrenda, Will Deacon,
linux-arch, linux-s390, Vasily Gorbik, Christian Borntraeger,
Richard Weinberger, linux-x86, Russell King, Jason Gunthorpe,
Ingo Molnar, Andrey Ryabinin, Gerald Schaefer, Jeff Dike,
Arnd Bergmann, John Hubbard, Heiko Carstens, linux-um,
Borislav Petkov, Andy Lutomirski, Thomas Gleixner, linux-arm,
Linus Torvalds, LKML, Andrew Morton, linux-power, Mike Rapoport
In-Reply-To: <31dfb3ed-a0cc-3024-d389-ab9bd19e881f@csgroup.eu>
On Tue, Sep 08, 2020 at 07:14:38AM +0200, Christophe Leroy wrote:
> You forgot arch/powerpc/mm/book3s64/subpage_prot.c it seems.
Yes, and also two more sources :/
arch/powerpc/mm/kasan/8xx.c
arch/powerpc/mm/kasan/kasan_init_32.c
But these two are not quite obvious wrt pgd_addr_end() used
while traversing pmds. Could you please clarify a bit?
diff --git a/arch/powerpc/mm/kasan/8xx.c b/arch/powerpc/mm/kasan/8xx.c
index 2784224..89c5053 100644
--- a/arch/powerpc/mm/kasan/8xx.c
+++ b/arch/powerpc/mm/kasan/8xx.c
@@ -15,8 +15,8 @@
for (k_cur = k_start; k_cur != k_end; k_cur = k_next, pmd += 2, block += SZ_8M) {
pte_basic_t *new;
- k_next = pgd_addr_end(k_cur, k_end);
- k_next = pgd_addr_end(k_next, k_end);
+ k_next = pmd_addr_end(k_cur, k_end);
+ k_next = pmd_addr_end(k_next, k_end);
if ((void *)pmd_page_vaddr(*pmd) != kasan_early_shadow_pte)
continue;
diff --git a/arch/powerpc/mm/kasan/kasan_init_32.c b/arch/powerpc/mm/kasan/kasan_init_32.c
index fb29404..3f7d6dc6 100644
--- a/arch/powerpc/mm/kasan/kasan_init_32.c
+++ b/arch/powerpc/mm/kasan/kasan_init_32.c
@@ -38,7 +38,7 @@ int __init kasan_init_shadow_page_tables(unsigned long k_start, unsigned long k_
for (k_cur = k_start; k_cur != k_end; k_cur = k_next, pmd++) {
pte_t *new;
- k_next = pgd_addr_end(k_cur, k_end);
+ k_next = pmd_addr_end(k_cur, k_end);
if ((void *)pmd_page_vaddr(*pmd) != kasan_early_shadow_pte)
continue;
@@ -196,7 +196,7 @@ void __init kasan_early_init(void)
kasan_populate_pte(kasan_early_shadow_pte, PAGE_KERNEL);
do {
- next = pgd_addr_end(addr, end);
+ next = pmd_addr_end(addr, end);
pmd_populate_kernel(&init_mm, pmd, kasan_early_shadow_pte);
} while (pmd++, addr = next, addr != end);
> Christophe
^ permalink raw reply related
* Re: [RFC PATCH 02/12] powerpc: remove arguments from interrupt handler functions
From: Nicholas Piggin @ 2020-09-08 7:46 UTC (permalink / raw)
To: Christophe Leroy; +Cc: linuxppc-dev
In-Reply-To: <e34fead9-a356-3ae6-aa33-544380230bd5@csgroup.eu>
Excerpts from Christophe Leroy's message of September 7, 2020 7:20 pm:
>
>
> Le 05/09/2020 à 19:43, Nicholas Piggin a écrit :
>> Make interrupt handlers all just take the pt_regs * argument and load
>> DAR/DSISR etc from that. Make those that return a value return long.
>
> I like this, it will likely simplify a bit the VMAP_STACK mess.
>
> Not sure it is that easy. My board is stuck after the start of init.
>
>
> On the 8xx, on Instruction TLB Error exception, we do
>
> andis. r5,r9,DSISR_SRR1_MATCH_32S@h /* Filter relevant SRR1 bits */
>
> On book3s/32, on ISI exception we do:
> andis. r5,r9,DSISR_SRR1_MATCH_32S@h /* Filter relevant SRR1 bits */
>
> On 40x and bookE, on ISI exception we do:
> li r5,0 /* Pass zero as arg3 */
>
>
> And regs->dsisr will just contain nothing
>
> So it means we should at least write back r5 into regs->dsisr from there
> ? The performance impact should be minimal as we already write _DAR so
> the cache line should already be in the cache.
Yes, I think that would be required. Sorry I didn't look closely at
32 bit.
> A hacky 'stw r5, _DSISR(r1)' in handle_page_fault() does the trick,
> allthough we don't want to do it for both ISI and DSI at the end, so
> you'll have to do it in every head_xxx.S
>
>
> While you are at it, it would probably also make sense to do remove the
> address param of bad_page_fault(), there is no point in loading back
> regs->dar in handle_page_fault() and machine_check_8xx() and
> alignment_exception(), just read regs->dar in bad_page_fault()
>
> The case of do_break() should also be looked at.
Yeah that's valid, I didn't do that because bad_page_fault was also
being called from asm, but an incremental patch should be quite easy.
> Why changing return code from int to long ?
Oh it's to make the next patch work without any changes to function
prototypes. Some handlers are returning int, others long. There is
no reason not to just return long AFAIKS so that's what I changed to.
Thanks,
Nick
^ permalink raw reply
* Re: [PATCH] selftests/powerpc: Skip PROT_SAO test in guests/LPARS
From: Michael Ellerman @ 2020-09-08 7:22 UTC (permalink / raw)
To: Sachin Sant; +Cc: linuxppc-dev
In-Reply-To: <5E12FA66-4147-46E6-9D8A-358FFAFDC3FB@linux.vnet.ibm.com>
Sachin Sant <sachinp@linux.vnet.ibm.com> writes:
>> On 01-Sep-2020, at 6:16 PM, Michael Ellerman <mpe@ellerman.id.au> wrote:
>>
>> In commit 9b725a90a8f1 ("powerpc/64s: Disallow PROT_SAO in LPARs by
>> default") PROT_SAO was disabled in guests/LPARs by default. So skip
>> the test if we are running in a guest to avoid a spurious failure.
>>
>> Signed-off-by: Michael Ellerman <mpe@ellerman.id.au>
>> —
>
> Tested-by: Sachin Sant <sachinp@linux.vnet.ibm.com>
>
> With the fix test is skipped while running in a guest
>
> # ./prot_sao
> test: prot-sao
> tags: git_version:unknown
> [SKIP] Test skipped on line 25
> skip: prot-sao
> #
Thanks. Sorry I missed adding your Tested-by tag.
cheers
^ permalink raw reply
* Re: [PATCH] powerpc/boot/dts: Fix dtc "pciex" warnings
From: Michael Ellerman @ 2020-09-08 7:11 UTC (permalink / raw)
To: Christian Lamparter, linuxppc-dev
Cc: sfr, Chris Blake,
open list:OPEN FIRMWARE AND FLATTENED DEVICE TREE BINDINGS
In-Reply-To: <d2652e63-b136-a805-fd6d-00584b64c772@gmail.com>
Christian Lamparter <chunkeey@gmail.com> writes:
> On 2020-06-23 15:03, Michael Ellerman wrote:
>> With CONFIG_OF_ALL_DTBS=y, as set by eg. allmodconfig, we see lots of
>> warnings about our dts files, such as:
>>
>> arch/powerpc/boot/dts/glacier.dts:492.26-532.5:
>> Warning (pci_bridge): /plb/pciex@d00000000: node name is not "pci"
>> or "pcie"
>>
>> The node name should not particularly matter, it's just a name, and
>> AFAICS there's no kernel code that cares whether nodes are *named*
>> "pciex" or "pcie". So shutup these warnings by converting to the name
>> dtc wants.
>>
>> As always there's some risk this could break something obscure that
>> does rely on the name, in which case we can revert.
>
> Hmm, I noticed this when I was looking up why nobody commented
> on my series of adding more devices to the APM82181/bluestone series:
>
> <https://lore.kernel.org/linuxppc-dev/cover.1598124791.git.chunkeey@gmail.com/>
> (I'll post a v3 "soonish".)
>
>
> Unfortunately yes. This patch will break uboot code in Meraki MX60(W) / MX60.
>
> > https://github.com/riptidewave93/meraki-uboot/blob/mx60w-20180413/board/amcc/bluestone/bluestone.c#L1178
>
> | if (!pci_available()) {
> | fdt_find_and_setprop(blob, "/plb/pciex@d00000000", "status",
> | "disabled", sizeof("disabled"), 1);
> | }
>
>
> Backstory: There are two version of the Meraki MX60. The MX60
> and the MX60W. The difference is that the MX60W has a populated
> mini-pcie slot on the PCB for a >W<ireless card.
>
> That said, this is not earth shattering.
I'm happy to revert that hunk if you think any one is actually booting
mainline on those.
cheers
> (In theory, this can also cause problems for the bluestone and canyonlands
> dev boards that have the option to be configured as either dual sata or
> pcie+sata.... But this is probably not a problem for customer boards)
>
> OT: Please note that the plb, opb and ebc node paths (/plb/opb/ebc) are
> hardcoded too :(. Amending the proper unit-addresses will lead to no-longer
> working DTBs as the "ranges" are missing.
>
> Cheers,
> Christian
>> Signed-off-by: Michael Ellerman <mpe@ellerman.id.au>
>> ---
>>
>> diff --git a/arch/powerpc/boot/dts/bluestone.dts b/arch/powerpc/boot/dts/bluestone.dts
>> index cc965a1816b6..aa1ae94cd776 100644
>> --- a/arch/powerpc/boot/dts/bluestone.dts
>> +++ b/arch/powerpc/boot/dts/bluestone.dts
>> @@ -325,7 +325,7 @@ EMAC0: ethernet@ef600c00 {
>> };
>> };
>>
>> - PCIE0: pciex@d00000000 {
>> + PCIE0: pcie@d00000000 {
>> device_type = "pci";
>> #interrupt-cells = <1>;
>> #size-cells = <2>;
^ permalink raw reply
* Re: [PATCH v2] kbuild: preprocess module linker script
From: Geert Uytterhoeven @ 2020-09-08 6:59 UTC (permalink / raw)
To: Masahiro Yamada
Cc: linux-ia64@vger.kernel.org, Catalin Marinas, Paul Mackerras,
linux-riscv, Will Deacon, Anton Ivanov, Linux-Arch,
Richard Weinberger, Russell King, Fenghua Yu, Albert Ou,
Arnd Bergmann, linux-kbuild, Jeff Dike, linux-um, linux-m68k,
Tony Luck, Paul Walmsley, Linux ARM, Michal Marek,
Linux Kernel Mailing List, Palmer Dabbelt, Jessica Yu,
linuxppc-dev
In-Reply-To: <20200908042708.2511528-1-masahiroy@kernel.org>
On Tue, Sep 8, 2020 at 6:29 AM Masahiro Yamada <masahiroy@kernel.org> wrote:
> There was a request to preprocess the module linker script like we
> do for the vmlinux one. (https://lkml.org/lkml/2020/8/21/512)
>
> The difference between vmlinux.lds and module.lds is that the latter
> is needed for external module builds, thus must be cleaned up by
> 'make mrproper' instead of 'make clean'. Also, it must be created
> by 'make modules_prepare'.
>
> You cannot put it in arch/$(SRCARCH)/kernel/, which is cleaned up by
> 'make clean'. I moved arch/$(SRCARCH)/kernel/module.lds to
> arch/$(SRCARCH)/include/asm/module.lds.h, which is included from
> scripts/module.lds.S.
>
> scripts/module.lds is fine because 'make clean' keeps all the
> build artifacts under scripts/.
>
> You can add arch-specific sections in <asm/module.lds.h>.
>
> Signed-off-by: Masahiro Yamada <masahiroy@kernel.org>
> Tested-by: Jessica Yu <jeyu@kernel.org>
> Acked-by: Will Deacon <will@kernel.org>
> arch/m68k/Makefile | 1 -
> .../{kernel/module.lds => include/asm/module.lds.h} | 0
Acked-by: Geert Uytterhoeven <geert@linux-m68k.org>
Gr{oetje,eeting}s,
Geert
--
Geert Uytterhoeven -- There's lots of Linux beyond ia32 -- geert@linux-m68k.org
In personal conversations with technical people, I call myself a hacker. But
when I'm talking to journalists I just say "programmer" or something like that.
-- Linus Torvalds
^ permalink raw reply
* Re: [PATCH] mm: check for memory's node later during boot
From: Laurent Dufour @ 2020-09-08 6:56 UTC (permalink / raw)
To: Andrew Morton
Cc: nathanl, Rafael J. Wysocki, Greg Kroah-Hartman, cheloha,
linux-kernel, linux-mm, linuxppc-dev
In-Reply-To: <20200903143523.02e163bf06be3b48bac7f967@linux-foundation.org>
Le 03/09/2020 à 23:35, Andrew Morton a écrit :
> On Wed, 2 Sep 2020 11:09:11 +0200 Laurent Dufour <ldufour@linux.ibm.com> wrote:
>
>> register_mem_sect_under_nodem() is checking the memory block's node id only
>> if the system state is "SYSTEM_BOOTING". On PowerPC, the memory blocks are
>> registered while the system state is "SYSTEM_SCHEDULING", the one before
>> SYSTEM_RUNNING.
>>
>> The consequence on PowerPC guest with interleaved memory node's ranges is
>> that some memory block could be assigned to multiple nodes on sysfs. This
>> lately prevents some memory hot-plug and hot-unplug to succeed because
>> links are remaining. Such a panic is then displayed:
>>
>> ------------[ cut here ]------------
>> kernel BUG at /Users/laurent/src/linux-ppc/mm/memory_hotplug.c:1084!
>> Oops: Exception in kernel mode, sig: 5 [#1]
>> LE PAGE_SIZE=64K MMU=Hash SMP NR_CPUS=2048 NUMA pSeries
>> Modules linked in: rpadlpar_io rpaphp pseries_rng rng_core vmx_crypto gf128mul binfmt_misc ip_tables x_tables xfs libcrc32c crc32c_vpmsum autofs4
>> CPU: 8 PID: 10256 Comm: drmgr Not tainted 5.9.0-rc1+ #25
>> NIP: c000000000403f34 LR: c000000000403f2c CTR: 0000000000000000
>> REGS: c0000004876e3660 TRAP: 0700 Not tainted (5.9.0-rc1+)
>> MSR: 800000000282b033 <SF,VEC,VSX,EE,FP,ME,IR,DR,RI,LE> CR: 24000448 XER: 20040000
>> CFAR: c000000000846d20 IRQMASK: 0
>> GPR00: c000000000403f2c c0000004876e38f0 c0000000012f6f00 ffffffffffffffef
>> GPR04: 0000000000000227 c0000004805ae680 0000000000000000 00000004886f0000
>> GPR08: 0000000000000226 0000000000000003 0000000000000002 fffffffffffffffd
>> GPR12: 0000000088000484 c00000001ec96280 0000000000000000 0000000000000000
>> GPR16: 0000000000000000 0000000000000000 0000000000000004 0000000000000003
>> GPR20: c00000047814ffe0 c0000007ffff7c08 0000000000000010 c0000000013332c8
>> GPR24: 0000000000000000 c0000000011f6cc0 0000000000000000 0000000000000000
>> GPR28: ffffffffffffffef 0000000000000001 0000000150000000 0000000010000000
>> NIP [c000000000403f34] add_memory_resource+0x244/0x340
>> LR [c000000000403f2c] add_memory_resource+0x23c/0x340
>> Call Trace:
>> [c0000004876e38f0] [c000000000403f2c] add_memory_resource+0x23c/0x340 (unreliable)
>> [c0000004876e39c0] [c00000000040408c] __add_memory+0x5c/0xf0
>> [c0000004876e39f0] [c0000000000e2b94] dlpar_add_lmb+0x1b4/0x500
>> [c0000004876e3ad0] [c0000000000e3888] dlpar_memory+0x1f8/0xb80
>> [c0000004876e3b60] [c0000000000dc0d0] handle_dlpar_errorlog+0xc0/0x190
>> [c0000004876e3bd0] [c0000000000dc398] dlpar_store+0x198/0x4a0
>> [c0000004876e3c90] [c00000000072e630] kobj_attr_store+0x30/0x50
>> [c0000004876e3cb0] [c00000000051f954] sysfs_kf_write+0x64/0x90
>> [c0000004876e3cd0] [c00000000051ee40] kernfs_fop_write+0x1b0/0x290
>> [c0000004876e3d20] [c000000000438dd8] vfs_write+0xe8/0x290
>> [c0000004876e3d70] [c0000000004391ac] ksys_write+0xdc/0x130
>> [c0000004876e3dc0] [c000000000034e40] system_call_exception+0x160/0x270
>> [c0000004876e3e20] [c00000000000d740] system_call_common+0xf0/0x27c
>> Instruction dump:
>> 48442e35 60000000 0b030000 3cbe0001 7fa3eb78 7bc48402 38a5fffe 7ca5fa14
>> 78a58402 48442db1 60000000 7c7c1b78 <0b030000> 7f23cb78 4bda371d 60000000
>> ---[ end trace 562fd6c109cd0fb2 ]---
>>
>> To prevent this multiple links, make the node checking done for states
>> prior to SYSTEM_RUNNING.
>
> Did you consider adding a cc:stable to this fix?
I should have, but now I've to review the fix based on David's comment.
^ permalink raw reply
* Re: [PATCH kernel] powerpc/dma: Fix dma_map_ops::get_required_mask
From: Michael Ellerman @ 2020-09-08 6:45 UTC (permalink / raw)
To: Alexey Kardashevskiy, linuxppc-dev
Cc: Alexey Kardashevskiy, Oliver O'Halloran, Christoph Hellwig,
Cédric Le Goater
In-Reply-To: <20200908015106.79661-1-aik@ozlabs.ru>
Alexey Kardashevskiy <aik@ozlabs.ru> writes:
> There are 2 problems with it:
> 1. "<" vs expected "<<"
> 2. the shift number is an IOMMU page number mask, not an address mask
> as the IOMMU page shift is missing.
>
> This did not hit us before f1565c24b596 ("powerpc: use the generic
> dma_ops_bypass mode") because we had there additional code to handle
> bypass mask so this chunk (almost?) never executed. However there
> were reports that aacraid does not work with "iommu=nobypass".
> After f1565c24b596, aacraid (and probably others which call
> dma_get_required_mask() before setting the mask) was unable to
> enable 64bit DMA and fall back to using IOMMU which was known not to work,
> one of the problems is double free of an IOMMU page.
>
> This fixes DMA for aacraid, both with and without "iommu=nobypass"
> in the kernel command line. Verified with "stress-ng -d 4".
>
> Fixes: f1565c24b596 ("powerpc: use the generic dma_ops_bypass mode")
I think it'd be better to point the Fixes tag at 6a5c7be5e484, which
originally introduced the bug, even if we didn't notice it until
f1565c24b596 exposed it (or made it more likely).
cheers
> Signed-off-by: Alexey Kardashevskiy <aik@ozlabs.ru>
> ---
>
> The original code came Jun 24 2011:
> 6a5c7be5e484 ("powerpc: Override dma_get_required_mask by platform hook and ops")
>
>
> What is dma_get_required_mask() for anyway? What "requires" what here?
>
> Even though it works for now (due to huge - >4GB - default DMA window),
> I am still not convinced we do not want this chunk here
> (this is what f1565c24b596 removed):
>
> if (dev_is_pci(dev)) {
> u64 bypass_mask = dma_direct_get_required_mask(dev);
>
> if (dma_iommu_bypass_supported(dev, bypass_mask))
> return bypass_mask;
> }
> ---
> arch/powerpc/kernel/dma-iommu.c | 3 ++-
> 1 file changed, 2 insertions(+), 1 deletion(-)
>
> diff --git a/arch/powerpc/kernel/dma-iommu.c b/arch/powerpc/kernel/dma-iommu.c
> index 569fecd7b5b2..9053fc9d20c7 100644
> --- a/arch/powerpc/kernel/dma-iommu.c
> +++ b/arch/powerpc/kernel/dma-iommu.c
> @@ -120,7 +120,8 @@ u64 dma_iommu_get_required_mask(struct device *dev)
> if (!tbl)
> return 0;
>
> - mask = 1ULL < (fls_long(tbl->it_offset + tbl->it_size) - 1);
> + mask = 1ULL << (fls_long(tbl->it_offset + tbl->it_size) +
> + tbl->it_page_shift - 1);
> mask += mask - 1;
>
> return mask;
> --
> 2.17.1
^ permalink raw reply
* Re: [PATCH kernel] powerpc/dma: Fix dma_map_ops::get_required_mask
From: Christoph Hellwig @ 2020-09-08 5:44 UTC (permalink / raw)
To: Alexey Kardashevskiy
Cc: Oliver O'Halloran, linuxppc-dev, Christoph Hellwig,
Cédric Le Goater
In-Reply-To: <20200908015106.79661-1-aik@ozlabs.ru>
On Tue, Sep 08, 2020 at 11:51:06AM +1000, Alexey Kardashevskiy wrote:
> What is dma_get_required_mask() for anyway? What "requires" what here?
Yes, it is a really odd API. It comes from classic old PCI where
64-bit addressing required an additional bus cycle, and various devices
had different addressing schemes, with the smaller addresses beeing
more efficient. So this allows the driver to request the "required"
addressing mode to address all memory. "preferred" might be a better
name as we'll bounce buffer if it isn't met. I also don't really see
why a driver would ever want to use it for a modern PCIe device.
^ permalink raw reply
* Re: [RFC PATCH v2 0/3] mm/gup: fix gup_fast with dynamic page table folding
From: Christophe Leroy @ 2020-09-08 5:22 UTC (permalink / raw)
To: Mike Rapoport, Gerald Schaefer
Cc: Peter Zijlstra, Catalin Marinas, Dave Hansen, linux-mm,
Paul Mackerras, linux-sparc, Alexander Gordeev, Claudio Imbrenda,
Will Deacon, linux-arch, linux-s390, Vasily Gorbik,
Jason Gunthorpe, Richard Weinberger, linux-x86, Russell King,
Christian Borntraeger, Ingo Molnar, Andrey Ryabinin, Jeff Dike,
Arnd Bergmann, John Hubbard, Heiko Carstens, linux-um,
Borislav Petkov, Andy Lutomirski, Thomas Gleixner, linux-arm,
Linus Torvalds, LKML, Andrew Morton, linux-power
In-Reply-To: <20200907201256.GC1976319@kernel.org>
Le 07/09/2020 à 22:12, Mike Rapoport a écrit :
> On Mon, Sep 07, 2020 at 08:00:55PM +0200, Gerald Schaefer wrote:
>> This is v2 of an RFC previously discussed here:
>> https://lore.kernel.org/lkml/20200828140314.8556-1-gerald.schaefer@linux.ibm.com/
>>
>> Patch 1 is a fix for a regression in gup_fast on s390, after our conversion
>> to common gup_fast code. It will introduce special helper functions
>> pXd_addr_end_folded(), which have to be used in places where pagetable walk
>> is done w/o lock and with READ_ONCE, so currently only in gup_fast.
>>
>> Patch 2 is an attempt to make that more generic, i.e. change pXd_addr_end()
>> themselves by adding an extra pXd value parameter. That was suggested by
>> Jason during v1 discussion, because he is already thinking of some other
>> places where he might want to switch to the READ_ONCE logic for pagetable
>> walks. In general, that would be the cleanest / safest solution, but there
>> is some impact on other architectures and common code, hence the new and
>> greatly enlarged recipient list.
>>
>> Patch 3 is a "nice to have" add-on, which makes pXd_addr_end() inline
>> functions instead of #defines, so that we get some type checking for the
>> new pXd value parameter.
>>
>> Not sure about Fixes/stable tags for the generic solution. Only patch 1
>> fixes a real bug on s390, and has Fixes/stable tags. Patches 2 + 3 might
>> still be nice to have in stable, to ease future backports, but I guess
>> "nice to have" does not really qualify for stable backports.
>
> I also think that adding pXd parameter to pXd_addr_end() is a cleaner
> way and with this patch 1 is not really required. I would even merge
> patches 2 and 3 into a single patch and use only it as the fix.
Why not merging patches 2 and 3, but I would keep patch 1 separate but
after the generic changes, so that we first do the generic changes, then
we do the specific S390 use of it.
Christophe
^ permalink raw reply
* Re: [RFC PATCH v2 3/3] mm: make generic pXd_addr_end() macros inline functions
From: Christophe Leroy @ 2020-09-08 5:19 UTC (permalink / raw)
To: Gerald Schaefer, Jason Gunthorpe, John Hubbard
Cc: Peter Zijlstra, Catalin Marinas, Dave Hansen, linux-mm,
Paul Mackerras, linux-sparc, Alexander Gordeev, Claudio Imbrenda,
Will Deacon, linux-arch, linux-s390, Vasily Gorbik,
Richard Weinberger, linux-x86, Russell King,
Christian Borntraeger, Ingo Molnar, Andrey Ryabinin, Jeff Dike,
Arnd Bergmann, Heiko Carstens, linux-um, Borislav Petkov,
Andy Lutomirski, Thomas Gleixner, linux-arm, Linus Torvalds, LKML,
Andrew Morton, linux-power, Mike Rapoport
In-Reply-To: <20200907180058.64880-4-gerald.schaefer@linux.ibm.com>
Le 07/09/2020 à 20:00, Gerald Schaefer a écrit :
> From: Alexander Gordeev <agordeev@linux.ibm.com>
>
> Since pXd_addr_end() macros take pXd page-table entry as a
> parameter it makes sense to check the entry type on compile.
> Even though most archs do not make use of page-table entries
> in pXd_addr_end() calls, checking the type in traversal code
> paths could help to avoid subtle bugs.
>
> Signed-off-by: Alexander Gordeev <agordeev@linux.ibm.com>
> Signed-off-by: Gerald Schaefer <gerald.schaefer@linux.ibm.com>
> ---
> include/linux/pgtable.h | 36 ++++++++++++++++++++----------------
> 1 file changed, 20 insertions(+), 16 deletions(-)
>
> diff --git a/include/linux/pgtable.h b/include/linux/pgtable.h
> index 67ebc22cf83d..d9e7d16c2263 100644
> --- a/include/linux/pgtable.h
> +++ b/include/linux/pgtable.h
> @@ -656,31 +656,35 @@ static inline int arch_unmap_one(struct mm_struct *mm,
> */
>
> #ifndef pgd_addr_end
> -#define pgd_addr_end(pgd, addr, end) \
> -({ unsigned long __boundary = ((addr) + PGDIR_SIZE) & PGDIR_MASK; \
> - (__boundary - 1 < (end) - 1)? __boundary: (end); \
> -})
> +#define pgd_addr_end pgd_addr_end
I think that #define is pointless, usually there is no such #define for
the default case.
> +static inline unsigned long pgd_addr_end(pgd_t pgd, unsigned long addr, unsigned long end)
> +{ unsigned long __boundary = (addr + PGDIR_SIZE) & PGDIR_MASK;
> + return (__boundary - 1 < end - 1) ? __boundary : end;
> +}
Please use the standard layout, ie entry { and exit } alone on their
line, and space between local vars declaration and the rest.
Also remove the leading __ in front of var names as it's not needed once
it is not macros anymore.
f_name()
{
some_local_var;
do_something();
}
> #endif
>
> #ifndef p4d_addr_end
> -#define p4d_addr_end(p4d, addr, end) \
> -({ unsigned long __boundary = ((addr) + P4D_SIZE) & P4D_MASK; \
> - (__boundary - 1 < (end) - 1)? __boundary: (end); \
> -})
> +#define p4d_addr_end p4d_addr_end
> +static inline unsigned long p4d_addr_end(p4d_t p4d, unsigned long addr, unsigned long end)
> +{ unsigned long __boundary = (addr + P4D_SIZE) & P4D_MASK;
> + return (__boundary - 1 < end - 1) ? __boundary : end;
> +}
> #endif
>
> #ifndef pud_addr_end
> -#define pud_addr_end(pud, addr, end) \
> -({ unsigned long __boundary = ((addr) + PUD_SIZE) & PUD_MASK; \
> - (__boundary - 1 < (end) - 1)? __boundary: (end); \
> -})
> +#define pud_addr_end pud_addr_end
> +static inline unsigned long pud_addr_end(pud_t pud, unsigned long addr, unsigned long end)
> +{ unsigned long __boundary = (addr + PUD_SIZE) & PUD_MASK;
> + return (__boundary - 1 < end - 1) ? __boundary : end;
> +}
> #endif
>
> #ifndef pmd_addr_end
> -#define pmd_addr_end(pmd, addr, end) \
> -({ unsigned long __boundary = ((addr) + PMD_SIZE) & PMD_MASK; \
> - (__boundary - 1 < (end) - 1)? __boundary: (end); \
> -})
> +#define pmd_addr_end pmd_addr_end
> +static inline unsigned long pmd_addr_end(pmd_t pmd, unsigned long addr, unsigned long end)
> +{ unsigned long __boundary = (addr + PMD_SIZE) & PMD_MASK;
> + return (__boundary - 1 < end - 1) ? __boundary : end;
> +}
> #endif
>
> /*
>
^ permalink raw reply
* Re: [RFC PATCH v2 2/3] mm: make pXd_addr_end() functions page-table entry aware
From: Christophe Leroy @ 2020-09-08 5:14 UTC (permalink / raw)
To: Gerald Schaefer, Jason Gunthorpe, John Hubbard
Cc: Peter Zijlstra, Catalin Marinas, Dave Hansen, linux-mm,
Paul Mackerras, linux-sparc, Alexander Gordeev, Claudio Imbrenda,
Will Deacon, linux-arch, linux-s390, Vasily Gorbik,
Richard Weinberger, linux-x86, Russell King,
Christian Borntraeger, Ingo Molnar, Andrey Ryabinin, Jeff Dike,
Arnd Bergmann, Heiko Carstens, linux-um, Borislav Petkov,
Andy Lutomirski, Thomas Gleixner, linux-arm, Linus Torvalds, LKML,
Andrew Morton, linux-power, Mike Rapoport
In-Reply-To: <20200907180058.64880-3-gerald.schaefer@linux.ibm.com>
Le 07/09/2020 à 20:00, Gerald Schaefer a écrit :
> From: Alexander Gordeev <agordeev@linux.ibm.com>
>
> Unlike all other page-table abstractions pXd_addr_end() do not take
> into account a particular table entry in which context the functions
> are called. On architectures with dynamic page-tables folding that
> might lead to lack of necessary information that is difficult to
> obtain other than from the table entry itself. That already led to
> a subtle memory corruption issue on s390.
>
> By letting pXd_addr_end() functions know about the page-table entry
> we allow archs not only make extra checks, but also optimizations.
>
> As result of this change the pXd_addr_end_folded() functions used
> in gup_fast traversal code become unnecessary and get replaced with
> universal pXd_addr_end() variants.
>
> The arch-specific updates not only add dereferencing of page-table
> entry pointers, but also small changes to the code flow to make those
> dereferences possible, at least for x86 and powerpc. Also for arm64,
> but in way that should not have any impact.
>
[...]
>
> Signed-off-by: Alexander Gordeev <agordeev@linux.ibm.com>
> Signed-off-by: Gerald Schaefer <gerald.schaefer@linux.ibm.com>
> ---
> arch/arm/include/asm/pgtable-2level.h | 2 +-
> arch/arm/mm/idmap.c | 6 ++--
> arch/arm/mm/mmu.c | 8 ++---
> arch/arm64/kernel/hibernate.c | 16 ++++++----
> arch/arm64/kvm/mmu.c | 16 +++++-----
> arch/arm64/mm/kasan_init.c | 8 ++---
> arch/arm64/mm/mmu.c | 25 +++++++--------
> arch/powerpc/mm/book3s64/radix_pgtable.c | 7 ++---
> arch/powerpc/mm/hugetlbpage.c | 6 ++--
You forgot arch/powerpc/mm/book3s64/subpage_prot.c it seems.
> arch/s390/include/asm/pgtable.h | 8 ++---
> arch/s390/mm/page-states.c | 8 ++---
> arch/s390/mm/pageattr.c | 8 ++---
> arch/s390/mm/vmem.c | 8 ++---
> arch/sparc/mm/hugetlbpage.c | 6 ++--
> arch/um/kernel/tlb.c | 8 ++---
> arch/x86/mm/init_64.c | 15 ++++-----
> arch/x86/mm/kasan_init_64.c | 16 +++++-----
> include/asm-generic/pgtable-nop4d.h | 2 +-
> include/asm-generic/pgtable-nopmd.h | 2 +-
> include/asm-generic/pgtable-nopud.h | 2 +-
> include/linux/pgtable.h | 26 ++++-----------
> mm/gup.c | 8 ++---
> mm/ioremap.c | 8 ++---
> mm/kasan/init.c | 17 +++++-----
> mm/madvise.c | 4 +--
> mm/memory.c | 40 ++++++++++++------------
> mm/mlock.c | 18 ++++++++---
> mm/mprotect.c | 8 ++---
> mm/pagewalk.c | 8 ++---
> mm/swapfile.c | 8 ++---
> mm/vmalloc.c | 16 +++++-----
> 31 files changed, 165 insertions(+), 173 deletions(-)
Christophe
^ permalink raw reply
* Re: [RFC PATCH v2 1/3] mm/gup: fix gup_fast with dynamic page table folding
From: Christophe Leroy @ 2020-09-08 5:06 UTC (permalink / raw)
To: Gerald Schaefer, Jason Gunthorpe, John Hubbard
Cc: Peter Zijlstra, Catalin Marinas, Dave Hansen, linux-mm,
Paul Mackerras, linux-sparc, Alexander Gordeev, Claudio Imbrenda,
Will Deacon, linux-arch, linux-s390, Vasily Gorbik,
Richard Weinberger, linux-x86, Russell King,
Christian Borntraeger, Ingo Molnar, Andrey Ryabinin, Jeff Dike,
Arnd Bergmann, Heiko Carstens, linux-um, Borislav Petkov,
Andy Lutomirski, Thomas Gleixner, linux-arm, Linus Torvalds, LKML,
Andrew Morton, linux-power, Mike Rapoport
In-Reply-To: <20200907180058.64880-2-gerald.schaefer@linux.ibm.com>
Le 07/09/2020 à 20:00, Gerald Schaefer a écrit :
> From: Alexander Gordeev <agordeev@linux.ibm.com>
>
> Commit 1a42010cdc26 ("s390/mm: convert to the generic get_user_pages_fast
> code") introduced a subtle but severe bug on s390 with gup_fast, due to
> dynamic page table folding.
>
> The question "What would it require for the generic code to work for s390"
> has already been discussed here
> https://lkml.kernel.org/r/20190418100218.0a4afd51@mschwideX1
> and ended with a promising approach here
> https://lkml.kernel.org/r/20190419153307.4f2911b5@mschwideX1
> which in the end unfortunately didn't quite work completely.
>
> We tried to mimic static level folding by changing pgd_offset to always
> calculate top level page table offset, and do nothing in folded pXd_offset.
> What has been overlooked is that PxD_SIZE/MASK and thus pXd_addr_end do
> not reflect this dynamic behaviour, and still act like static 5-level
> page tables.
>
[...]
>
> Fix this by introducing new pXd_addr_end_folded helpers, which take an
> additional pXd entry value parameter, that can be used on s390
> to determine the correct page table level and return corresponding
> end / boundary. With that, the pointer iteration will always
> happen in gup_pgd_range for s390. No change for other architectures
> introduced.
Not sure pXd_addr_end_folded() is the best understandable name,
allthough I don't have any alternative suggestion at the moment.
Maybe could be something like pXd_addr_end_fixup() as it will disappear
in the next patch, or pXd_addr_end_gup() ?
Also, if it happens to be acceptable to get patch 2 in stable, I think
you should switch patch 1 and patch 2 to avoid the step through
pXd_addr_end_folded()
>
> Fixes: 1a42010cdc26 ("s390/mm: convert to the generic get_user_pages_fast code")
> Cc: <stable@vger.kernel.org> # 5.2+
> Reviewed-by: Gerald Schaefer <gerald.schaefer@linux.ibm.com>
> Signed-off-by: Alexander Gordeev <agordeev@linux.ibm.com>
> Signed-off-by: Gerald Schaefer <gerald.schaefer@linux.ibm.com>
> ---
> arch/s390/include/asm/pgtable.h | 42 +++++++++++++++++++++++++++++++++
> include/linux/pgtable.h | 16 +++++++++++++
> mm/gup.c | 8 +++----
> 3 files changed, 62 insertions(+), 4 deletions(-)
>
> diff --git a/arch/s390/include/asm/pgtable.h b/arch/s390/include/asm/pgtable.h
> index 7eb01a5459cd..027206e4959d 100644
> --- a/arch/s390/include/asm/pgtable.h
> +++ b/arch/s390/include/asm/pgtable.h
> @@ -512,6 +512,48 @@ static inline bool mm_pmd_folded(struct mm_struct *mm)
> }
> #define mm_pmd_folded(mm) mm_pmd_folded(mm)
>
> +/*
> + * With dynamic page table levels on s390, the static pXd_addr_end() functions
> + * will not return corresponding dynamic boundaries. This is no problem as long
> + * as only pXd pointers are passed down during page table walk, because
> + * pXd_offset() will simply return the given pointer for folded levels, and the
> + * pointer iteration over a range simply happens at the correct page table
> + * level.
> + * It is however a problem with gup_fast, or other places walking the page
> + * tables w/o locks using READ_ONCE(), and passing down the pXd values instead
> + * of pointers. In this case, the pointer given to pXd_offset() is a pointer to
> + * a stack variable, which cannot be used for pointer iteration at the correct
> + * level. Instead, the iteration then has to happen by going up to pgd level
> + * again. To allow this, provide pXd_addr_end_folded() functions with an
> + * additional pXd value parameter, which can be used on s390 to determine the
> + * folding level and return the corresponding boundary.
> + */
> +static inline unsigned long rste_addr_end_folded(unsigned long rste, unsigned long addr, unsigned long end)
What does 'rste' stands for ?
Isn't this line a bit long ?
> +{
> + unsigned long type = (rste & _REGION_ENTRY_TYPE_MASK) >> 2;
> + unsigned long size = 1UL << (_SEGMENT_SHIFT + type * 11);
> + unsigned long boundary = (addr + size) & ~(size - 1);
> +
> + /*
> + * FIXME The below check is for internal testing only, to be removed
> + */
> + VM_BUG_ON(type < (_REGION_ENTRY_TYPE_R3 >> 2));
> +
> + return (boundary - 1) < (end - 1) ? boundary : end;
> +}
> +
> +#define pgd_addr_end_folded pgd_addr_end_folded
> +static inline unsigned long pgd_addr_end_folded(pgd_t pgd, unsigned long addr, unsigned long end)
> +{
> + return rste_addr_end_folded(pgd_val(pgd), addr, end);
> +}
> +
> +#define p4d_addr_end_folded p4d_addr_end_folded
> +static inline unsigned long p4d_addr_end_folded(p4d_t p4d, unsigned long addr, unsigned long end)
> +{
> + return rste_addr_end_folded(p4d_val(p4d), addr, end);
> +}
> +
> static inline int mm_has_pgste(struct mm_struct *mm)
> {
> #ifdef CONFIG_PGSTE
> diff --git a/include/linux/pgtable.h b/include/linux/pgtable.h
> index e8cbc2e795d5..981c4c2a31fe 100644
> --- a/include/linux/pgtable.h
> +++ b/include/linux/pgtable.h
> @@ -681,6 +681,22 @@ static inline int arch_unmap_one(struct mm_struct *mm,
> })
> #endif
>
> +#ifndef pgd_addr_end_folded
> +#define pgd_addr_end_folded(pgd, addr, end) pgd_addr_end(addr, end)
> +#endif
> +
> +#ifndef p4d_addr_end_folded
> +#define p4d_addr_end_folded(p4d, addr, end) p4d_addr_end(addr, end)
> +#endif
> +
> +#ifndef pud_addr_end_folded
> +#define pud_addr_end_folded(pud, addr, end) pud_addr_end(addr, end)
> +#endif
> +
> +#ifndef pmd_addr_end_folded
> +#define pmd_addr_end_folded(pmd, addr, end) pmd_addr_end(addr, end)
> +#endif
> +
> /*
> * When walking page tables, we usually want to skip any p?d_none entries;
> * and any p?d_bad entries - reporting the error before resetting to none.
> diff --git a/mm/gup.c b/mm/gup.c
> index bd883a112724..ba4aace5d0f4 100644
> --- a/mm/gup.c
> +++ b/mm/gup.c
> @@ -2521,7 +2521,7 @@ static int gup_pmd_range(pud_t pud, unsigned long addr, unsigned long end,
> do {
> pmd_t pmd = READ_ONCE(*pmdp);
>
> - next = pmd_addr_end(addr, end);
> + next = pmd_addr_end_folded(pmd, addr, end);
> if (!pmd_present(pmd))
> return 0;
>
> @@ -2564,7 +2564,7 @@ static int gup_pud_range(p4d_t p4d, unsigned long addr, unsigned long end,
> do {
> pud_t pud = READ_ONCE(*pudp);
>
> - next = pud_addr_end(addr, end);
> + next = pud_addr_end_folded(pud, addr, end);
> if (unlikely(!pud_present(pud)))
> return 0;
> if (unlikely(pud_huge(pud))) {
> @@ -2592,7 +2592,7 @@ static int gup_p4d_range(pgd_t pgd, unsigned long addr, unsigned long end,
> do {
> p4d_t p4d = READ_ONCE(*p4dp);
>
> - next = p4d_addr_end(addr, end);
> + next = p4d_addr_end_folded(p4d, addr, end);
> if (p4d_none(p4d))
> return 0;
> BUILD_BUG_ON(p4d_huge(p4d));
> @@ -2617,7 +2617,7 @@ static void gup_pgd_range(unsigned long addr, unsigned long end,
> do {
> pgd_t pgd = READ_ONCE(*pgdp);
>
> - next = pgd_addr_end(addr, end);
> + next = pgd_addr_end_folded(pgd, addr, end);
> if (pgd_none(pgd))
> return;
> if (unlikely(pgd_huge(pgd))) {
>
Christophe
^ permalink raw reply
* Re: [RFC PATCH v2 0/3] mm/gup: fix gup_fast with dynamic page table folding
From: Christophe Leroy @ 2020-09-08 4:42 UTC (permalink / raw)
To: Gerald Schaefer, Jason Gunthorpe, John Hubbard
Cc: Peter Zijlstra, Catalin Marinas, Dave Hansen, linux-mm,
Paul Mackerras, linux-sparc, Alexander Gordeev, Claudio Imbrenda,
Will Deacon, linux-arch, linux-s390, Vasily Gorbik,
Richard Weinberger, linux-x86, Russell King,
Christian Borntraeger, Ingo Molnar, Andrey Ryabinin, Jeff Dike,
Arnd Bergmann, Heiko Carstens, linux-um, Borislav Petkov,
Andy Lutomirski, Thomas Gleixner, linux-arm, Linus Torvalds, LKML,
Andrew Morton, linux-power, Mike Rapoport
In-Reply-To: <20200907180058.64880-1-gerald.schaefer@linux.ibm.com>
Le 07/09/2020 à 20:00, Gerald Schaefer a écrit :
> This is v2 of an RFC previously discussed here:
> https://lore.kernel.org/lkml/20200828140314.8556-1-gerald.schaefer@linux.ibm.com/
>
> Patch 1 is a fix for a regression in gup_fast on s390, after our conversion
> to common gup_fast code. It will introduce special helper functions
> pXd_addr_end_folded(), which have to be used in places where pagetable walk
> is done w/o lock and with READ_ONCE, so currently only in gup_fast.
>
> Patch 2 is an attempt to make that more generic, i.e. change pXd_addr_end()
> themselves by adding an extra pXd value parameter. That was suggested by
> Jason during v1 discussion, because he is already thinking of some other
> places where he might want to switch to the READ_ONCE logic for pagetable
> walks. In general, that would be the cleanest / safest solution, but there
> is some impact on other architectures and common code, hence the new and
> greatly enlarged recipient list.
>
> Patch 3 is a "nice to have" add-on, which makes pXd_addr_end() inline
> functions instead of #defines, so that we get some type checking for the
> new pXd value parameter.
>
> Not sure about Fixes/stable tags for the generic solution. Only patch 1
> fixes a real bug on s390, and has Fixes/stable tags. Patches 2 + 3 might
> still be nice to have in stable, to ease future backports, but I guess
> "nice to have" does not really qualify for stable backports.
If one day you have to backport a fix that requires patch 2 and/or 3,
just mark it "depends-on:" and the patches will go in stable at the
relevant time.
Christophe
^ permalink raw reply
* [PATCH v2] kbuild: preprocess module linker script
From: Masahiro Yamada @ 2020-09-08 4:27 UTC (permalink / raw)
To: linux-kbuild
Cc: linux-ia64, Catalin Marinas, Paul Mackerras, linux-riscv,
Will Deacon, Anton Ivanov, linux-arch, Richard Weinberger,
Masahiro Yamada, Russell King, Geert Uytterhoeven, Fenghua Yu,
Albert Ou, Arnd Bergmann, Jeff Dike, Jessica Yu, linux-um,
linux-m68k, Michal Marek, Paul Walmsley, linux-arm-kernel,
Tony Luck, linux-kernel, Palmer Dabbelt, linuxppc-dev
There was a request to preprocess the module linker script like we
do for the vmlinux one. (https://lkml.org/lkml/2020/8/21/512)
The difference between vmlinux.lds and module.lds is that the latter
is needed for external module builds, thus must be cleaned up by
'make mrproper' instead of 'make clean'. Also, it must be created
by 'make modules_prepare'.
You cannot put it in arch/$(SRCARCH)/kernel/, which is cleaned up by
'make clean'. I moved arch/$(SRCARCH)/kernel/module.lds to
arch/$(SRCARCH)/include/asm/module.lds.h, which is included from
scripts/module.lds.S.
scripts/module.lds is fine because 'make clean' keeps all the
build artifacts under scripts/.
You can add arch-specific sections in <asm/module.lds.h>.
Signed-off-by: Masahiro Yamada <masahiroy@kernel.org>
Tested-by: Jessica Yu <jeyu@kernel.org>
Acked-by: Will Deacon <will@kernel.org>
---
Changes in v2:
- Fix the race between the two targets 'scripts' and 'asm-generic'
Makefile | 10 ++++++----
arch/arm/Makefile | 4 ----
.../{kernel/module.lds => include/asm/module.lds.h} | 2 ++
arch/arm64/Makefile | 4 ----
.../{kernel/module.lds => include/asm/module.lds.h} | 2 ++
arch/ia64/Makefile | 1 -
arch/ia64/{module.lds => include/asm/module.lds.h} | 0
arch/m68k/Makefile | 1 -
.../{kernel/module.lds => include/asm/module.lds.h} | 0
arch/powerpc/Makefile | 1 -
.../{kernel/module.lds => include/asm/module.lds.h} | 0
arch/riscv/Makefile | 3 ---
.../{kernel/module.lds => include/asm/module.lds.h} | 3 ++-
arch/um/include/asm/Kbuild | 1 +
include/asm-generic/Kbuild | 1 +
include/asm-generic/module.lds.h | 10 ++++++++++
scripts/.gitignore | 1 +
scripts/Makefile | 3 +++
scripts/Makefile.modfinal | 5 ++---
scripts/{module-common.lds => module.lds.S} | 3 +++
scripts/package/builddeb | 2 +-
21 files changed, 34 insertions(+), 23 deletions(-)
rename arch/arm/{kernel/module.lds => include/asm/module.lds.h} (72%)
rename arch/arm64/{kernel/module.lds => include/asm/module.lds.h} (76%)
rename arch/ia64/{module.lds => include/asm/module.lds.h} (100%)
rename arch/m68k/{kernel/module.lds => include/asm/module.lds.h} (100%)
rename arch/powerpc/{kernel/module.lds => include/asm/module.lds.h} (100%)
rename arch/riscv/{kernel/module.lds => include/asm/module.lds.h} (84%)
create mode 100644 include/asm-generic/module.lds.h
rename scripts/{module-common.lds => module.lds.S} (93%)
diff --git a/Makefile b/Makefile
index 37739ee53f27..97b1dae1783b 100644
--- a/Makefile
+++ b/Makefile
@@ -505,7 +505,6 @@ KBUILD_CFLAGS_KERNEL :=
KBUILD_AFLAGS_MODULE := -DMODULE
KBUILD_CFLAGS_MODULE := -DMODULE
KBUILD_LDFLAGS_MODULE :=
-export KBUILD_LDS_MODULE := $(srctree)/scripts/module-common.lds
KBUILD_LDFLAGS :=
CLANG_FLAGS :=
@@ -1395,7 +1394,7 @@ endif
# using awk while concatenating to the final file.
PHONY += modules
-modules: $(if $(KBUILD_BUILTIN),vmlinux) modules_check
+modules: $(if $(KBUILD_BUILTIN),vmlinux) modules_check modules_prepare
$(Q)$(MAKE) -f $(srctree)/scripts/Makefile.modpost
PHONY += modules_check
@@ -1412,6 +1411,7 @@ targets += modules.order
# Target to prepare building external modules
PHONY += modules_prepare
modules_prepare: prepare
+ $(Q)$(MAKE) $(build)=scripts scripts/module.lds
# Target to install modules
PHONY += modules_install
@@ -1743,7 +1743,9 @@ help:
@echo ' clean - remove generated files in module directory only'
@echo ''
-PHONY += prepare
+# no-op for external module builds
+PHONY += prepare modules_prepare
+
endif # KBUILD_EXTMOD
# Single targets
@@ -1776,7 +1778,7 @@ MODORDER := .modules.tmp
endif
PHONY += single_modpost
-single_modpost: $(single-no-ko)
+single_modpost: $(single-no-ko) modules_prepare
$(Q){ $(foreach m, $(single-ko), echo $(extmod-prefix)$m;) } > $(MODORDER)
$(Q)$(MAKE) -f $(srctree)/scripts/Makefile.modpost
diff --git a/arch/arm/Makefile b/arch/arm/Makefile
index 4e877354515f..a0cb15de9677 100644
--- a/arch/arm/Makefile
+++ b/arch/arm/Makefile
@@ -16,10 +16,6 @@ LDFLAGS_vmlinux += --be8
KBUILD_LDFLAGS_MODULE += --be8
endif
-ifeq ($(CONFIG_ARM_MODULE_PLTS),y)
-KBUILD_LDS_MODULE += $(srctree)/arch/arm/kernel/module.lds
-endif
-
GZFLAGS :=-9
#KBUILD_CFLAGS +=-pipe
diff --git a/arch/arm/kernel/module.lds b/arch/arm/include/asm/module.lds.h
similarity index 72%
rename from arch/arm/kernel/module.lds
rename to arch/arm/include/asm/module.lds.h
index 79cb6af565e5..0e7cb4e314b4 100644
--- a/arch/arm/kernel/module.lds
+++ b/arch/arm/include/asm/module.lds.h
@@ -1,5 +1,7 @@
/* SPDX-License-Identifier: GPL-2.0 */
+#ifdef CONFIG_ARM_MODULE_PLTS
SECTIONS {
.plt : { BYTE(0) }
.init.plt : { BYTE(0) }
}
+#endif
diff --git a/arch/arm64/Makefile b/arch/arm64/Makefile
index b45f0124cc16..76667ad47980 100644
--- a/arch/arm64/Makefile
+++ b/arch/arm64/Makefile
@@ -115,10 +115,6 @@ endif
CHECKFLAGS += -D__aarch64__
-ifeq ($(CONFIG_ARM64_MODULE_PLTS),y)
-KBUILD_LDS_MODULE += $(srctree)/arch/arm64/kernel/module.lds
-endif
-
ifeq ($(CONFIG_DYNAMIC_FTRACE_WITH_REGS),y)
KBUILD_CPPFLAGS += -DCC_USING_PATCHABLE_FUNCTION_ENTRY
CC_FLAGS_FTRACE := -fpatchable-function-entry=2
diff --git a/arch/arm64/kernel/module.lds b/arch/arm64/include/asm/module.lds.h
similarity index 76%
rename from arch/arm64/kernel/module.lds
rename to arch/arm64/include/asm/module.lds.h
index 22e36a21c113..691f15af788e 100644
--- a/arch/arm64/kernel/module.lds
+++ b/arch/arm64/include/asm/module.lds.h
@@ -1,5 +1,7 @@
+#ifdef CONFIG_ARM64_MODULE_PLTS
SECTIONS {
.plt (NOLOAD) : { BYTE(0) }
.init.plt (NOLOAD) : { BYTE(0) }
.text.ftrace_trampoline (NOLOAD) : { BYTE(0) }
}
+#endif
diff --git a/arch/ia64/Makefile b/arch/ia64/Makefile
index 2876a7df1b0a..703b1c4f6d12 100644
--- a/arch/ia64/Makefile
+++ b/arch/ia64/Makefile
@@ -20,7 +20,6 @@ CHECKFLAGS += -D__ia64=1 -D__ia64__=1 -D_LP64 -D__LP64__
OBJCOPYFLAGS := --strip-all
LDFLAGS_vmlinux := -static
-KBUILD_LDS_MODULE += $(srctree)/arch/ia64/module.lds
KBUILD_AFLAGS_KERNEL := -mconstant-gp
EXTRA :=
diff --git a/arch/ia64/module.lds b/arch/ia64/include/asm/module.lds.h
similarity index 100%
rename from arch/ia64/module.lds
rename to arch/ia64/include/asm/module.lds.h
diff --git a/arch/m68k/Makefile b/arch/m68k/Makefile
index 4438ffb4bbe1..ea14f2046fb4 100644
--- a/arch/m68k/Makefile
+++ b/arch/m68k/Makefile
@@ -75,7 +75,6 @@ KBUILD_CPPFLAGS += -D__uClinux__
endif
KBUILD_LDFLAGS := -m m68kelf
-KBUILD_LDS_MODULE += $(srctree)/arch/m68k/kernel/module.lds
ifdef CONFIG_SUN3
LDFLAGS_vmlinux = -N
diff --git a/arch/m68k/kernel/module.lds b/arch/m68k/include/asm/module.lds.h
similarity index 100%
rename from arch/m68k/kernel/module.lds
rename to arch/m68k/include/asm/module.lds.h
diff --git a/arch/powerpc/Makefile b/arch/powerpc/Makefile
index 3e8da9cf2eb9..8935658fcd06 100644
--- a/arch/powerpc/Makefile
+++ b/arch/powerpc/Makefile
@@ -65,7 +65,6 @@ UTS_MACHINE := $(subst $(space),,$(machine-y))
ifdef CONFIG_PPC32
KBUILD_LDFLAGS_MODULE += arch/powerpc/lib/crtsavres.o
else
-KBUILD_LDS_MODULE += $(srctree)/arch/powerpc/kernel/module.lds
ifeq ($(call ld-ifversion, -ge, 225000000, y),y)
# Have the linker provide sfpr if possible.
# There is a corresponding test in arch/powerpc/lib/Makefile
diff --git a/arch/powerpc/kernel/module.lds b/arch/powerpc/include/asm/module.lds.h
similarity index 100%
rename from arch/powerpc/kernel/module.lds
rename to arch/powerpc/include/asm/module.lds.h
diff --git a/arch/riscv/Makefile b/arch/riscv/Makefile
index fb6e37db836d..8edaa8bd86d6 100644
--- a/arch/riscv/Makefile
+++ b/arch/riscv/Makefile
@@ -53,9 +53,6 @@ endif
ifeq ($(CONFIG_CMODEL_MEDANY),y)
KBUILD_CFLAGS += -mcmodel=medany
endif
-ifeq ($(CONFIG_MODULE_SECTIONS),y)
- KBUILD_LDS_MODULE += $(srctree)/arch/riscv/kernel/module.lds
-endif
ifeq ($(CONFIG_PERF_EVENTS),y)
KBUILD_CFLAGS += -fno-omit-frame-pointer
endif
diff --git a/arch/riscv/kernel/module.lds b/arch/riscv/include/asm/module.lds.h
similarity index 84%
rename from arch/riscv/kernel/module.lds
rename to arch/riscv/include/asm/module.lds.h
index 295ecfb341a2..4254ff2ff049 100644
--- a/arch/riscv/kernel/module.lds
+++ b/arch/riscv/include/asm/module.lds.h
@@ -1,8 +1,9 @@
/* SPDX-License-Identifier: GPL-2.0 */
/* Copyright (C) 2017 Andes Technology Corporation */
-
+#ifdef CONFIG_MODULE_SECTIONS
SECTIONS {
.plt (NOLOAD) : { BYTE(0) }
.got (NOLOAD) : { BYTE(0) }
.got.plt (NOLOAD) : { BYTE(0) }
}
+#endif
diff --git a/arch/um/include/asm/Kbuild b/arch/um/include/asm/Kbuild
index 8d435f8a6dec..1c63b260ecc4 100644
--- a/arch/um/include/asm/Kbuild
+++ b/arch/um/include/asm/Kbuild
@@ -16,6 +16,7 @@ generic-y += kdebug.h
generic-y += mcs_spinlock.h
generic-y += mm-arch-hooks.h
generic-y += mmiowb.h
+generic-y += module.lds.h
generic-y += param.h
generic-y += pci.h
generic-y += percpu.h
diff --git a/include/asm-generic/Kbuild b/include/asm-generic/Kbuild
index 74b0612601dd..7cd4e627e00e 100644
--- a/include/asm-generic/Kbuild
+++ b/include/asm-generic/Kbuild
@@ -40,6 +40,7 @@ mandatory-y += mmiowb.h
mandatory-y += mmu.h
mandatory-y += mmu_context.h
mandatory-y += module.h
+mandatory-y += module.lds.h
mandatory-y += msi.h
mandatory-y += pci.h
mandatory-y += percpu.h
diff --git a/include/asm-generic/module.lds.h b/include/asm-generic/module.lds.h
new file mode 100644
index 000000000000..f210d5c1b78b
--- /dev/null
+++ b/include/asm-generic/module.lds.h
@@ -0,0 +1,10 @@
+/* SPDX-License-Identifier: GPL-2.0-only */
+#ifndef __ASM_GENERIC_MODULE_LDS_H
+#define __ASM_GENERIC_MODULE_LDS_H
+
+/*
+ * <asm/module.lds.h> can specify arch-specific sections for linking modules.
+ * Empty for the asm-generic header.
+ */
+
+#endif /* __ASM_GENERIC_MODULE_LDS_H */
diff --git a/scripts/.gitignore b/scripts/.gitignore
index 0d1c8e217cd7..a6c11316c969 100644
--- a/scripts/.gitignore
+++ b/scripts/.gitignore
@@ -8,3 +8,4 @@ asn1_compiler
extract-cert
sign-file
insert-sys-cert
+/module.lds
diff --git a/scripts/Makefile b/scripts/Makefile
index bc018e4b733e..b5418ec587fb 100644
--- a/scripts/Makefile
+++ b/scripts/Makefile
@@ -29,6 +29,9 @@ endif
# The following programs are only built on demand
hostprogs += unifdef
+# The module linker script is preprocessed on demand
+targets += module.lds
+
subdir-$(CONFIG_GCC_PLUGINS) += gcc-plugins
subdir-$(CONFIG_MODVERSIONS) += genksyms
subdir-$(CONFIG_SECURITY_SELINUX) += selinux
diff --git a/scripts/Makefile.modfinal b/scripts/Makefile.modfinal
index 411c1e600e7d..ae01baf96f4e 100644
--- a/scripts/Makefile.modfinal
+++ b/scripts/Makefile.modfinal
@@ -33,11 +33,10 @@ quiet_cmd_ld_ko_o = LD [M] $@
cmd_ld_ko_o = \
$(LD) -r $(KBUILD_LDFLAGS) \
$(KBUILD_LDFLAGS_MODULE) $(LDFLAGS_MODULE) \
- $(addprefix -T , $(KBUILD_LDS_MODULE)) \
- -o $@ $(filter %.o, $^); \
+ -T scripts/module.lds -o $@ $(filter %.o, $^); \
$(if $(ARCH_POSTLINK), $(MAKE) -f $(ARCH_POSTLINK) $@, true)
-$(modules): %.ko: %.o %.mod.o $(KBUILD_LDS_MODULE) FORCE
+$(modules): %.ko: %.o %.mod.o scripts/module.lds FORCE
+$(call if_changed,ld_ko_o)
targets += $(modules) $(modules:.ko=.mod.o)
diff --git a/scripts/module-common.lds b/scripts/module.lds.S
similarity index 93%
rename from scripts/module-common.lds
rename to scripts/module.lds.S
index d61b9e8678e8..69b9b71a6a47 100644
--- a/scripts/module-common.lds
+++ b/scripts/module.lds.S
@@ -24,3 +24,6 @@ SECTIONS {
__jump_table 0 : ALIGN(8) { KEEP(*(__jump_table)) }
}
+
+/* bring in arch-specific sections */
+#include <asm/module.lds.h>
diff --git a/scripts/package/builddeb b/scripts/package/builddeb
index 6df3c9f8b2da..44f212e37935 100755
--- a/scripts/package/builddeb
+++ b/scripts/package/builddeb
@@ -55,7 +55,7 @@ deploy_kernel_headers () {
cd $srctree
find . arch/$SRCARCH -maxdepth 1 -name Makefile\*
find include scripts -type f -o -type l
- find arch/$SRCARCH -name module.lds -o -name Kbuild.platforms -o -name Platform
+ find arch/$SRCARCH -name Kbuild.platforms -o -name Platform
find $(find arch/$SRCARCH -name include -o -name scripts -type d) -type f
) > debian/hdrsrcfiles
--
2.25.1
^ permalink raw reply related
* Re: [PATCH v1 02/10] powerpc/kernel/iommu: Align size for IOMMU_PAGE_SIZE on iommu_*_coherent()
From: Alexey Kardashevskiy @ 2020-09-08 3:18 UTC (permalink / raw)
To: Leonardo Bras, Michael Ellerman, Benjamin Herrenschmidt,
Paul Mackerras, Christophe Leroy, Joel Stanley,
Thiago Jung Bauermann, Ram Pai, Brian King,
Murilo Fossa Vicentini, David Dai
Cc: linuxppc-dev, linux-kernel
In-Reply-To: <ef7e80b0a7399bad607324301a604bfb46c2de05.camel@gmail.com>
On 04/09/2020 16:04, Leonardo Bras wrote:
> On Thu, 2020-09-03 at 14:41 +1000, Alexey Kardashevskiy wrote:
>> I am new to this, so I am trying to understand how a memory page mapped
>>> as DMA, and used for something else could be a problem.
>>
>> From the device prospective, there is PCI space and everything from 0
>> till 1<<64 is accessible and what is that mapped to - the device does
>> not know. PHB's IOMMU is the thing to notice invalid access and raise
>> EEH but PHB only knows about PCI->physical memory mapping (with IOMMU
>> pages) but nothing about the host kernel pages. Does this help? Thanks,
>
> According to our conversation on Slack:
> 1- There is a problem if a hypervisor gives to it's VMs contiguous
> memory blocks that are not aligned to IOMMU pages, because then an
> iommu_map_page() could map some memory in this VM and some memory in
> other VM / process.
> 2- To guarantee this, we should have system pagesize >= iommu_pagesize
>
> One way to get (2) is by doing this in enable_ddw():
> if ((query.page_size & 4) && PAGE_SHIFT >= 24) {
You won't ever (well, soon) see PAGE_SHIFT==24, it is either 4K or 64K.
However 16MB IOMMU pages is fine - if hypervisor uses huge pages for VMs
RAM, it also then advertises huge IOMMU pages in ddw-query. So for the
1:1 case there must be no "PAGE_SHIFT >= 24".
> page_shift = 24; /* 16MB */
> } else if ((query.page_size & 2) && PAGE_SHIFT >= 16 ) {
> page_shift = 16; /* 64kB */
> } else if (query.page_size & 1 && PAGE_SHIFT >= 12) {
> page_shift = 12; /* 4kB */
> [...]
>
> Another way of solving this, would be adding in LoPAR documentation
> that the blocksize of contiguous memory the hypervisor gives a VM
> should always be aligned to IOMMU pagesize offered.
I think this is assumed already by the design of the DDW API.
>
> I think the best approach would be first sending the above patch, which
> is faster, and then get working into adding that to documentation, so
> hypervisors guarantee this.
>
> If this gets into the docs, we can revert the patch.
>
> What do you think?
I think we diverted from the original patch :) I am not quite sure what
you were fixing there. Thanks,
--
Alexey
^ permalink raw reply
* [PATCH kernel] powerpc/dma: Fix dma_map_ops::get_required_mask
From: Alexey Kardashevskiy @ 2020-09-08 1:51 UTC (permalink / raw)
To: linuxppc-dev
Cc: Alexey Kardashevskiy, Oliver O'Halloran, Christoph Hellwig,
Cédric Le Goater
There are 2 problems with it:
1. "<" vs expected "<<"
2. the shift number is an IOMMU page number mask, not an address mask
as the IOMMU page shift is missing.
This did not hit us before f1565c24b596 ("powerpc: use the generic
dma_ops_bypass mode") because we had there additional code to handle
bypass mask so this chunk (almost?) never executed. However there
were reports that aacraid does not work with "iommu=nobypass".
After f1565c24b596, aacraid (and probably others which call
dma_get_required_mask() before setting the mask) was unable to
enable 64bit DMA and fall back to using IOMMU which was known not to work,
one of the problems is double free of an IOMMU page.
This fixes DMA for aacraid, both with and without "iommu=nobypass"
in the kernel command line. Verified with "stress-ng -d 4".
Fixes: f1565c24b596 ("powerpc: use the generic dma_ops_bypass mode")
Signed-off-by: Alexey Kardashevskiy <aik@ozlabs.ru>
---
The original code came Jun 24 2011:
6a5c7be5e484 ("powerpc: Override dma_get_required_mask by platform hook and ops")
What is dma_get_required_mask() for anyway? What "requires" what here?
Even though it works for now (due to huge - >4GB - default DMA window),
I am still not convinced we do not want this chunk here
(this is what f1565c24b596 removed):
if (dev_is_pci(dev)) {
u64 bypass_mask = dma_direct_get_required_mask(dev);
if (dma_iommu_bypass_supported(dev, bypass_mask))
return bypass_mask;
}
---
arch/powerpc/kernel/dma-iommu.c | 3 ++-
1 file changed, 2 insertions(+), 1 deletion(-)
diff --git a/arch/powerpc/kernel/dma-iommu.c b/arch/powerpc/kernel/dma-iommu.c
index 569fecd7b5b2..9053fc9d20c7 100644
--- a/arch/powerpc/kernel/dma-iommu.c
+++ b/arch/powerpc/kernel/dma-iommu.c
@@ -120,7 +120,8 @@ u64 dma_iommu_get_required_mask(struct device *dev)
if (!tbl)
return 0;
- mask = 1ULL < (fls_long(tbl->it_offset + tbl->it_size) - 1);
+ mask = 1ULL << (fls_long(tbl->it_offset + tbl->it_size) +
+ tbl->it_page_shift - 1);
mask += mask - 1;
return mask;
--
2.17.1
^ permalink raw reply related
* Re: [PATCH AUTOSEL 5.8 14/53] ibmvnic fix NULL tx_pools and rx_tools issue at do_reset
From: Sasha Levin @ 2020-09-07 22:24 UTC (permalink / raw)
To: Jakub Kicinski
Cc: netdev, linux-kernel, stable, Mingming Cao, Dany Madden,
linuxppc-dev, David S . Miller
In-Reply-To: <20200907141026.093fc160@kicinski-fedora-pc1c0hjn.dhcp.thefacebook.com>
On Mon, Sep 07, 2020 at 02:10:26PM -0700, Jakub Kicinski wrote:
>On Mon, 7 Sep 2020 12:31:40 -0400 Sasha Levin wrote:
>> [ Upstream commit 9f13457377907fa253aef560e1a37e1ca4197f9b ]
>
>> @@ -2024,10 +2033,14 @@ static int do_reset(struct ibmvnic_adapter *adapter,
>> } else {
>> rc = reset_tx_pools(adapter);
>> if (rc)
>> + netdev_dbg(adapter->netdev, "reset tx pools failed (%d)\n",
>> + rc);
>> goto out;
>>
>> rc = reset_rx_pools(adapter);
>> if (rc)
>> + netdev_dbg(adapter->netdev, "reset rx pools failed (%d)\n",
>> + rc);
>> goto out;
>> }
>> ibmvnic_disable_irqs(adapter);
>
>Hi Sasha!
>
>I just pushed this to net:
>
>8ae4dff882eb ("ibmvnic: add missing parenthesis in do_reset()")
>
>You definitely want to pull that in if you decide to backport this one.
Will do, thanks!
--
Thanks,
Sasha
^ permalink raw reply
* Re: [PATCH AUTOSEL 5.8 14/53] ibmvnic fix NULL tx_pools and rx_tools issue at do_reset
From: Jakub Kicinski @ 2020-09-07 21:10 UTC (permalink / raw)
To: Sasha Levin
Cc: netdev, linux-kernel, stable, Mingming Cao, Dany Madden,
linuxppc-dev, David S . Miller
In-Reply-To: <20200907163220.1280412-14-sashal@kernel.org>
On Mon, 7 Sep 2020 12:31:40 -0400 Sasha Levin wrote:
> [ Upstream commit 9f13457377907fa253aef560e1a37e1ca4197f9b ]
> @@ -2024,10 +2033,14 @@ static int do_reset(struct ibmvnic_adapter *adapter,
> } else {
> rc = reset_tx_pools(adapter);
> if (rc)
> + netdev_dbg(adapter->netdev, "reset tx pools failed (%d)\n",
> + rc);
> goto out;
>
> rc = reset_rx_pools(adapter);
> if (rc)
> + netdev_dbg(adapter->netdev, "reset rx pools failed (%d)\n",
> + rc);
> goto out;
> }
> ibmvnic_disable_irqs(adapter);
Hi Sasha!
I just pushed this to net:
8ae4dff882eb ("ibmvnic: add missing parenthesis in do_reset()")
You definitely want to pull that in if you decide to backport this one.
^ permalink raw reply
* Re: [RFC PATCH v2 3/3] mm: make generic pXd_addr_end() macros inline functions
From: Mike Rapoport @ 2020-09-07 20:15 UTC (permalink / raw)
To: Gerald Schaefer
Cc: Peter Zijlstra, Dave Hansen, linux-mm, Paul Mackerras,
linux-sparc, Alexander Gordeev, Claudio Imbrenda, Will Deacon,
linux-arch, linux-s390, Vasily Gorbik, Christian Borntraeger,
Richard Weinberger, linux-x86, Russell King, Jason Gunthorpe,
Ingo Molnar, Catalin Marinas, Andrey Ryabinin, Heiko Carstens,
Arnd Bergmann, John Hubbard, Jeff Dike, linux-um, Borislav Petkov,
Andy Lutomirski, Thomas Gleixner, linux-arm, linux-power, LKML,
Andrew Morton, Linus Torvalds
In-Reply-To: <20200907180058.64880-4-gerald.schaefer@linux.ibm.com>
Hi,
Some style comments below.
On Mon, Sep 07, 2020 at 08:00:58PM +0200, Gerald Schaefer wrote:
> From: Alexander Gordeev <agordeev@linux.ibm.com>
>
> Since pXd_addr_end() macros take pXd page-table entry as a
> parameter it makes sense to check the entry type on compile.
> Even though most archs do not make use of page-table entries
> in pXd_addr_end() calls, checking the type in traversal code
> paths could help to avoid subtle bugs.
>
> Signed-off-by: Alexander Gordeev <agordeev@linux.ibm.com>
> Signed-off-by: Gerald Schaefer <gerald.schaefer@linux.ibm.com>
> ---
> include/linux/pgtable.h | 36 ++++++++++++++++++++----------------
> 1 file changed, 20 insertions(+), 16 deletions(-)
>
> diff --git a/include/linux/pgtable.h b/include/linux/pgtable.h
> index 67ebc22cf83d..d9e7d16c2263 100644
> --- a/include/linux/pgtable.h
> +++ b/include/linux/pgtable.h
> @@ -656,31 +656,35 @@ static inline int arch_unmap_one(struct mm_struct *mm,
> */
>
> #ifndef pgd_addr_end
> -#define pgd_addr_end(pgd, addr, end) \
> -({ unsigned long __boundary = ((addr) + PGDIR_SIZE) & PGDIR_MASK; \
> - (__boundary - 1 < (end) - 1)? __boundary: (end); \
> -})
> +#define pgd_addr_end pgd_addr_end
> +static inline unsigned long pgd_addr_end(pgd_t pgd, unsigned long addr, unsigned long end)
> +{ unsigned long __boundary = (addr + PGDIR_SIZE) & PGDIR_MASK;
The code should be on a separate line from the curly brace.
Besides, since this is not a macro anymore, I think it would be nicer to
use 'boundary' without underscores.
This applies to the changes below as well.
> + return (__boundary - 1 < end - 1) ? __boundary : end;
> +}
> #endif
>
> #ifndef p4d_addr_end
> -#define p4d_addr_end(p4d, addr, end) \
> -({ unsigned long __boundary = ((addr) + P4D_SIZE) & P4D_MASK; \
> - (__boundary - 1 < (end) - 1)? __boundary: (end); \
> -})
> +#define p4d_addr_end p4d_addr_end
> +static inline unsigned long p4d_addr_end(p4d_t p4d, unsigned long addr, unsigned long end)
> +{ unsigned long __boundary = (addr + P4D_SIZE) & P4D_MASK;
> + return (__boundary - 1 < end - 1) ? __boundary : end;
> +}
> #endif
>
> #ifndef pud_addr_end
> -#define pud_addr_end(pud, addr, end) \
> -({ unsigned long __boundary = ((addr) + PUD_SIZE) & PUD_MASK; \
> - (__boundary - 1 < (end) - 1)? __boundary: (end); \
> -})
> +#define pud_addr_end pud_addr_end
> +static inline unsigned long pud_addr_end(pud_t pud, unsigned long addr, unsigned long end)
> +{ unsigned long __boundary = (addr + PUD_SIZE) & PUD_MASK;
> + return (__boundary - 1 < end - 1) ? __boundary : end;
> +}
> #endif
>
> #ifndef pmd_addr_end
> -#define pmd_addr_end(pmd, addr, end) \
> -({ unsigned long __boundary = ((addr) + PMD_SIZE) & PMD_MASK; \
> - (__boundary - 1 < (end) - 1)? __boundary: (end); \
> -})
> +#define pmd_addr_end pmd_addr_end
> +static inline unsigned long pmd_addr_end(pmd_t pmd, unsigned long addr, unsigned long end)
> +{ unsigned long __boundary = (addr + PMD_SIZE) & PMD_MASK;
> + return (__boundary - 1 < end - 1) ? __boundary : end;
> +}
> #endif
>
> /*
> --
> 2.17.1
>
--
Sincerely yours,
Mike.
^ permalink raw reply
* Re: [RFC PATCH v2 0/3] mm/gup: fix gup_fast with dynamic page table folding
From: Mike Rapoport @ 2020-09-07 20:12 UTC (permalink / raw)
To: Gerald Schaefer
Cc: Peter Zijlstra, Dave Hansen, linux-mm, Paul Mackerras,
linux-sparc, Alexander Gordeev, Claudio Imbrenda, Will Deacon,
linux-arch, linux-s390, Vasily Gorbik, Christian Borntraeger,
Richard Weinberger, linux-x86, Russell King, Jason Gunthorpe,
Ingo Molnar, Catalin Marinas, Andrey Ryabinin, Heiko Carstens,
Arnd Bergmann, John Hubbard, Jeff Dike, linux-um, Borislav Petkov,
Andy Lutomirski, Thomas Gleixner, linux-arm, linux-power, LKML,
Andrew Morton, Linus Torvalds
In-Reply-To: <20200907180058.64880-1-gerald.schaefer@linux.ibm.com>
On Mon, Sep 07, 2020 at 08:00:55PM +0200, Gerald Schaefer wrote:
> This is v2 of an RFC previously discussed here:
> https://lore.kernel.org/lkml/20200828140314.8556-1-gerald.schaefer@linux.ibm.com/
>
> Patch 1 is a fix for a regression in gup_fast on s390, after our conversion
> to common gup_fast code. It will introduce special helper functions
> pXd_addr_end_folded(), which have to be used in places where pagetable walk
> is done w/o lock and with READ_ONCE, so currently only in gup_fast.
>
> Patch 2 is an attempt to make that more generic, i.e. change pXd_addr_end()
> themselves by adding an extra pXd value parameter. That was suggested by
> Jason during v1 discussion, because he is already thinking of some other
> places where he might want to switch to the READ_ONCE logic for pagetable
> walks. In general, that would be the cleanest / safest solution, but there
> is some impact on other architectures and common code, hence the new and
> greatly enlarged recipient list.
>
> Patch 3 is a "nice to have" add-on, which makes pXd_addr_end() inline
> functions instead of #defines, so that we get some type checking for the
> new pXd value parameter.
>
> Not sure about Fixes/stable tags for the generic solution. Only patch 1
> fixes a real bug on s390, and has Fixes/stable tags. Patches 2 + 3 might
> still be nice to have in stable, to ease future backports, but I guess
> "nice to have" does not really qualify for stable backports.
I also think that adding pXd parameter to pXd_addr_end() is a cleaner
way and with this patch 1 is not really required. I would even merge
patches 2 and 3 into a single patch and use only it as the fix.
[ /me apologises to stable@ team :-) ]
> Changes in v2:
> - Pick option 2 from v1 discussion (pXd_addr_end_folded helpers)
> - Add patch 2 + 3 for more generic approach
>
> Alexander Gordeev (3):
> mm/gup: fix gup_fast with dynamic page table folding
> mm: make pXd_addr_end() functions page-table entry aware
> mm: make generic pXd_addr_end() macros inline functions
>
> arch/arm/include/asm/pgtable-2level.h | 2 +-
> arch/arm/mm/idmap.c | 6 ++--
> arch/arm/mm/mmu.c | 8 ++---
> arch/arm64/kernel/hibernate.c | 16 +++++----
> arch/arm64/kvm/mmu.c | 16 ++++-----
> arch/arm64/mm/kasan_init.c | 8 ++---
> arch/arm64/mm/mmu.c | 25 +++++++-------
> arch/powerpc/mm/book3s64/radix_pgtable.c | 7 ++--
> arch/powerpc/mm/hugetlbpage.c | 6 ++--
> arch/s390/include/asm/pgtable.h | 42 ++++++++++++++++++++++++
> arch/s390/mm/page-states.c | 8 ++---
> arch/s390/mm/pageattr.c | 8 ++---
> arch/s390/mm/vmem.c | 8 ++---
> arch/sparc/mm/hugetlbpage.c | 6 ++--
> arch/um/kernel/tlb.c | 8 ++---
> arch/x86/mm/init_64.c | 15 ++++-----
> arch/x86/mm/kasan_init_64.c | 16 ++++-----
> include/asm-generic/pgtable-nop4d.h | 2 +-
> include/asm-generic/pgtable-nopmd.h | 2 +-
> include/asm-generic/pgtable-nopud.h | 2 +-
> include/linux/pgtable.h | 38 ++++++++++++---------
> mm/gup.c | 8 ++---
> mm/ioremap.c | 8 ++---
> mm/kasan/init.c | 17 +++++-----
> mm/madvise.c | 4 +--
> mm/memory.c | 40 +++++++++++-----------
> mm/mlock.c | 18 +++++++---
> mm/mprotect.c | 8 ++---
> mm/pagewalk.c | 8 ++---
> mm/swapfile.c | 8 ++---
> mm/vmalloc.c | 16 ++++-----
> 31 files changed, 219 insertions(+), 165 deletions(-)
>
> --
> 2.17.1
>
--
Sincerely yours,
Mike.
^ permalink raw reply
* [RFC PATCH v2 0/3] mm/gup: fix gup_fast with dynamic page table folding
From: Gerald Schaefer @ 2020-09-07 18:00 UTC (permalink / raw)
To: Jason Gunthorpe, John Hubbard
Cc: Peter Zijlstra, Dave Hansen, linux-mm, Paul Mackerras,
linux-sparc, Alexander Gordeev, Claudio Imbrenda, Will Deacon,
linux-arch, linux-s390, Vasily Gorbik, Richard Weinberger,
linux-x86, Russell King, Christian Borntraeger, Ingo Molnar,
Catalin Marinas, Andrey Ryabinin, Heiko Carstens, Arnd Bergmann,
Jeff Dike, linux-um, Borislav Petkov, Andy Lutomirski,
Thomas Gleixner, linux-arm, linux-power, LKML, Andrew Morton,
Linus Torvalds, Mike Rapoport
This is v2 of an RFC previously discussed here:
https://lore.kernel.org/lkml/20200828140314.8556-1-gerald.schaefer@linux.ibm.com/
Patch 1 is a fix for a regression in gup_fast on s390, after our conversion
to common gup_fast code. It will introduce special helper functions
pXd_addr_end_folded(), which have to be used in places where pagetable walk
is done w/o lock and with READ_ONCE, so currently only in gup_fast.
Patch 2 is an attempt to make that more generic, i.e. change pXd_addr_end()
themselves by adding an extra pXd value parameter. That was suggested by
Jason during v1 discussion, because he is already thinking of some other
places where he might want to switch to the READ_ONCE logic for pagetable
walks. In general, that would be the cleanest / safest solution, but there
is some impact on other architectures and common code, hence the new and
greatly enlarged recipient list.
Patch 3 is a "nice to have" add-on, which makes pXd_addr_end() inline
functions instead of #defines, so that we get some type checking for the
new pXd value parameter.
Not sure about Fixes/stable tags for the generic solution. Only patch 1
fixes a real bug on s390, and has Fixes/stable tags. Patches 2 + 3 might
still be nice to have in stable, to ease future backports, but I guess
"nice to have" does not really qualify for stable backports.
Changes in v2:
- Pick option 2 from v1 discussion (pXd_addr_end_folded helpers)
- Add patch 2 + 3 for more generic approach
Alexander Gordeev (3):
mm/gup: fix gup_fast with dynamic page table folding
mm: make pXd_addr_end() functions page-table entry aware
mm: make generic pXd_addr_end() macros inline functions
arch/arm/include/asm/pgtable-2level.h | 2 +-
arch/arm/mm/idmap.c | 6 ++--
arch/arm/mm/mmu.c | 8 ++---
arch/arm64/kernel/hibernate.c | 16 +++++----
arch/arm64/kvm/mmu.c | 16 ++++-----
arch/arm64/mm/kasan_init.c | 8 ++---
arch/arm64/mm/mmu.c | 25 +++++++-------
arch/powerpc/mm/book3s64/radix_pgtable.c | 7 ++--
arch/powerpc/mm/hugetlbpage.c | 6 ++--
arch/s390/include/asm/pgtable.h | 42 ++++++++++++++++++++++++
arch/s390/mm/page-states.c | 8 ++---
arch/s390/mm/pageattr.c | 8 ++---
arch/s390/mm/vmem.c | 8 ++---
arch/sparc/mm/hugetlbpage.c | 6 ++--
arch/um/kernel/tlb.c | 8 ++---
arch/x86/mm/init_64.c | 15 ++++-----
arch/x86/mm/kasan_init_64.c | 16 ++++-----
include/asm-generic/pgtable-nop4d.h | 2 +-
include/asm-generic/pgtable-nopmd.h | 2 +-
include/asm-generic/pgtable-nopud.h | 2 +-
include/linux/pgtable.h | 38 ++++++++++++---------
mm/gup.c | 8 ++---
mm/ioremap.c | 8 ++---
mm/kasan/init.c | 17 +++++-----
mm/madvise.c | 4 +--
mm/memory.c | 40 +++++++++++-----------
mm/mlock.c | 18 +++++++---
mm/mprotect.c | 8 ++---
mm/pagewalk.c | 8 ++---
mm/swapfile.c | 8 ++---
mm/vmalloc.c | 16 ++++-----
31 files changed, 219 insertions(+), 165 deletions(-)
--
2.17.1
^ permalink raw reply
* [RFC PATCH v2 2/3] mm: make pXd_addr_end() functions page-table entry aware
From: Gerald Schaefer @ 2020-09-07 18:00 UTC (permalink / raw)
To: Jason Gunthorpe, John Hubbard
Cc: Peter Zijlstra, Dave Hansen, linux-mm, Paul Mackerras,
linux-sparc, Alexander Gordeev, Claudio Imbrenda, Will Deacon,
linux-arch, linux-s390, Vasily Gorbik, Richard Weinberger,
linux-x86, Russell King, Christian Borntraeger, Ingo Molnar,
Catalin Marinas, Andrey Ryabinin, Heiko Carstens, Arnd Bergmann,
Jeff Dike, linux-um, Borislav Petkov, Andy Lutomirski,
Thomas Gleixner, linux-arm, linux-power, LKML, Andrew Morton,
Linus Torvalds, Mike Rapoport
In-Reply-To: <20200907180058.64880-1-gerald.schaefer@linux.ibm.com>
From: Alexander Gordeev <agordeev@linux.ibm.com>
Unlike all other page-table abstractions pXd_addr_end() do not take
into account a particular table entry in which context the functions
are called. On architectures with dynamic page-tables folding that
might lead to lack of necessary information that is difficult to
obtain other than from the table entry itself. That already led to
a subtle memory corruption issue on s390.
By letting pXd_addr_end() functions know about the page-table entry
we allow archs not only make extra checks, but also optimizations.
As result of this change the pXd_addr_end_folded() functions used
in gup_fast traversal code become unnecessary and get replaced with
universal pXd_addr_end() variants.
The arch-specific updates not only add dereferencing of page-table
entry pointers, but also small changes to the code flow to make those
dereferences possible, at least for x86 and powerpc. Also for arm64,
but in way that should not have any impact.
So, even though the dereferenced page-table entries are not used on
archs other than s390, and are optimized out by the compiler, there
is a small change in kernel size and this is what bloat-o-meter reports:
x86:
add/remove: 0/0 grow/shrink: 2/0 up/down: 10/0 (10)
Function old new delta
vmemmap_populate 587 592 +5
munlock_vma_pages_range 556 561 +5
Total: Before=15534694, After=15534704, chg +0.00%
powerpc:
add/remove: 0/0 grow/shrink: 1/0 up/down: 4/0 (4)
Function old new delta
.remove_pagetable 1648 1652 +4
Total: Before=21478240, After=21478244, chg +0.00%
arm64:
add/remove: 0/0 grow/shrink: 0/0 up/down: 0/0 (0)
Function old new delta
Total: Before=20240851, After=20240851, chg +0.00%
sparc:
add/remove: 0/0 grow/shrink: 0/0 up/down: 0/0 (0)
Function old new delta
Total: Before=4907262, After=4907262, chg +0.00%
Signed-off-by: Alexander Gordeev <agordeev@linux.ibm.com>
Signed-off-by: Gerald Schaefer <gerald.schaefer@linux.ibm.com>
---
arch/arm/include/asm/pgtable-2level.h | 2 +-
arch/arm/mm/idmap.c | 6 ++--
arch/arm/mm/mmu.c | 8 ++---
arch/arm64/kernel/hibernate.c | 16 ++++++----
arch/arm64/kvm/mmu.c | 16 +++++-----
arch/arm64/mm/kasan_init.c | 8 ++---
arch/arm64/mm/mmu.c | 25 +++++++--------
arch/powerpc/mm/book3s64/radix_pgtable.c | 7 ++---
arch/powerpc/mm/hugetlbpage.c | 6 ++--
arch/s390/include/asm/pgtable.h | 8 ++---
arch/s390/mm/page-states.c | 8 ++---
arch/s390/mm/pageattr.c | 8 ++---
arch/s390/mm/vmem.c | 8 ++---
arch/sparc/mm/hugetlbpage.c | 6 ++--
arch/um/kernel/tlb.c | 8 ++---
arch/x86/mm/init_64.c | 15 ++++-----
arch/x86/mm/kasan_init_64.c | 16 +++++-----
include/asm-generic/pgtable-nop4d.h | 2 +-
include/asm-generic/pgtable-nopmd.h | 2 +-
include/asm-generic/pgtable-nopud.h | 2 +-
include/linux/pgtable.h | 26 ++++-----------
mm/gup.c | 8 ++---
mm/ioremap.c | 8 ++---
mm/kasan/init.c | 17 +++++-----
mm/madvise.c | 4 +--
mm/memory.c | 40 ++++++++++++------------
mm/mlock.c | 18 ++++++++---
mm/mprotect.c | 8 ++---
mm/pagewalk.c | 8 ++---
mm/swapfile.c | 8 ++---
mm/vmalloc.c | 16 +++++-----
31 files changed, 165 insertions(+), 173 deletions(-)
diff --git a/arch/arm/include/asm/pgtable-2level.h b/arch/arm/include/asm/pgtable-2level.h
index 3502c2f746ca..5e6416b339f4 100644
--- a/arch/arm/include/asm/pgtable-2level.h
+++ b/arch/arm/include/asm/pgtable-2level.h
@@ -209,7 +209,7 @@ static inline pmd_t *pmd_offset(pud_t *pud, unsigned long addr)
} while (0)
/* we don't need complex calculations here as the pmd is folded into the pgd */
-#define pmd_addr_end(addr,end) (end)
+#define pmd_addr_end(pmd,addr,end) (end)
#define set_pte_ext(ptep,pte,ext) cpu_set_pte_ext(ptep,pte,ext)
diff --git a/arch/arm/mm/idmap.c b/arch/arm/mm/idmap.c
index 448e57c6f653..5437f943ca8b 100644
--- a/arch/arm/mm/idmap.c
+++ b/arch/arm/mm/idmap.c
@@ -46,7 +46,7 @@ static void idmap_add_pmd(pud_t *pud, unsigned long addr, unsigned long end,
pmd = pmd_offset(pud, addr);
do {
- next = pmd_addr_end(addr, end);
+ next = pmd_addr_end(*pmd, addr, end);
*pmd = __pmd((addr & PMD_MASK) | prot);
flush_pmd_entry(pmd);
} while (pmd++, addr = next, addr != end);
@@ -73,7 +73,7 @@ static void idmap_add_pud(pgd_t *pgd, unsigned long addr, unsigned long end,
unsigned long next;
do {
- next = pud_addr_end(addr, end);
+ next = pud_addr_end(*pud, addr, end);
idmap_add_pmd(pud, addr, next, prot);
} while (pud++, addr = next, addr != end);
}
@@ -95,7 +95,7 @@ static void identity_mapping_add(pgd_t *pgd, const char *text_start,
pgd += pgd_index(addr);
do {
- next = pgd_addr_end(addr, end);
+ next = pgd_addr_end(*pgd, addr, end);
idmap_add_pud(pgd, addr, next, prot);
} while (pgd++, addr = next, addr != end);
}
diff --git a/arch/arm/mm/mmu.c b/arch/arm/mm/mmu.c
index 698cc740c6b8..4013746e4c75 100644
--- a/arch/arm/mm/mmu.c
+++ b/arch/arm/mm/mmu.c
@@ -777,7 +777,7 @@ static void __init alloc_init_pmd(pud_t *pud, unsigned long addr,
* With LPAE, we must loop over to map
* all the pmds for the given range.
*/
- next = pmd_addr_end(addr, end);
+ next = pmd_addr_end(*pmd, addr, end);
/*
* Try a section mapping - addr, next and phys must all be
@@ -805,7 +805,7 @@ static void __init alloc_init_pud(p4d_t *p4d, unsigned long addr,
unsigned long next;
do {
- next = pud_addr_end(addr, end);
+ next = pud_addr_end(*pud, addr, end);
alloc_init_pmd(pud, addr, next, phys, type, alloc, ng);
phys += next - addr;
} while (pud++, addr = next, addr != end);
@@ -820,7 +820,7 @@ static void __init alloc_init_p4d(pgd_t *pgd, unsigned long addr,
unsigned long next;
do {
- next = p4d_addr_end(addr, end);
+ next = p4d_addr_end(*p4d, addr, end);
alloc_init_pud(p4d, addr, next, phys, type, alloc, ng);
phys += next - addr;
} while (p4d++, addr = next, addr != end);
@@ -923,7 +923,7 @@ static void __init __create_mapping(struct mm_struct *mm, struct map_desc *md,
pgd = pgd_offset(mm, addr);
end = addr + length;
do {
- unsigned long next = pgd_addr_end(addr, end);
+ unsigned long next = pgd_addr_end(*pgd, addr, end);
alloc_init_p4d(pgd, addr, next, phys, type, alloc, ng);
diff --git a/arch/arm64/kernel/hibernate.c b/arch/arm64/kernel/hibernate.c
index 68e14152d6e9..7be8c9cdc5c8 100644
--- a/arch/arm64/kernel/hibernate.c
+++ b/arch/arm64/kernel/hibernate.c
@@ -412,7 +412,7 @@ static int copy_pmd(pud_t *dst_pudp, pud_t *src_pudp, unsigned long start,
do {
pmd_t pmd = READ_ONCE(*src_pmdp);
- next = pmd_addr_end(addr, end);
+ next = pmd_addr_end(pmd, addr, end);
if (pmd_none(pmd))
continue;
if (pmd_table(pmd)) {
@@ -447,7 +447,7 @@ static int copy_pud(p4d_t *dst_p4dp, p4d_t *src_p4dp, unsigned long start,
do {
pud_t pud = READ_ONCE(*src_pudp);
- next = pud_addr_end(addr, end);
+ next = pud_addr_end(pud, addr, end);
if (pud_none(pud))
continue;
if (pud_table(pud)) {
@@ -473,8 +473,10 @@ static int copy_p4d(pgd_t *dst_pgdp, pgd_t *src_pgdp, unsigned long start,
dst_p4dp = p4d_offset(dst_pgdp, start);
src_p4dp = p4d_offset(src_pgdp, start);
do {
- next = p4d_addr_end(addr, end);
- if (p4d_none(READ_ONCE(*src_p4dp)))
+ p4d_t p4d = READ_ONCE(*src_p4dp);
+
+ next = p4d_addr_end(p4d, addr, end);
+ if (p4d_none(p4d))
continue;
if (copy_pud(dst_p4dp, src_p4dp, addr, next))
return -ENOMEM;
@@ -492,8 +494,10 @@ static int copy_page_tables(pgd_t *dst_pgdp, unsigned long start,
dst_pgdp = pgd_offset_pgd(dst_pgdp, start);
do {
- next = pgd_addr_end(addr, end);
- if (pgd_none(READ_ONCE(*src_pgdp)))
+ pgd_t pgd = READ_ONCE(*src_pgdp);
+
+ next = pgd_addr_end(pgd, addr, end);
+ if (pgd_none(pgd))
continue;
if (copy_p4d(dst_pgdp, src_pgdp, addr, next))
return -ENOMEM;
diff --git a/arch/arm64/kvm/mmu.c b/arch/arm64/kvm/mmu.c
index ba00bcc0c884..8f470f93a8e9 100644
--- a/arch/arm64/kvm/mmu.c
+++ b/arch/arm64/kvm/mmu.c
@@ -547,7 +547,7 @@ static void unmap_hyp_pmds(pud_t *pud, phys_addr_t addr, phys_addr_t end)
start_pmd = pmd = pmd_offset(pud, addr);
do {
- next = pmd_addr_end(addr, end);
+ next = pmd_addr_end(*pmd, addr, end);
/* Hyp doesn't use huge pmds */
if (!pmd_none(*pmd))
unmap_hyp_ptes(pmd, addr, next);
@@ -564,7 +564,7 @@ static void unmap_hyp_puds(p4d_t *p4d, phys_addr_t addr, phys_addr_t end)
start_pud = pud = pud_offset(p4d, addr);
do {
- next = pud_addr_end(addr, end);
+ next = pud_addr_end(*pud, addr, end);
/* Hyp doesn't use huge puds */
if (!pud_none(*pud))
unmap_hyp_pmds(pud, addr, next);
@@ -581,7 +581,7 @@ static void unmap_hyp_p4ds(pgd_t *pgd, phys_addr_t addr, phys_addr_t end)
start_p4d = p4d = p4d_offset(pgd, addr);
do {
- next = p4d_addr_end(addr, end);
+ next = p4d_addr_end(*p4d, addr, end);
/* Hyp doesn't use huge p4ds */
if (!p4d_none(*p4d))
unmap_hyp_puds(p4d, addr, next);
@@ -609,7 +609,7 @@ static void __unmap_hyp_range(pgd_t *pgdp, unsigned long ptrs_per_pgd,
*/
pgd = pgdp + kvm_pgd_index(addr, ptrs_per_pgd);
do {
- next = pgd_addr_end(addr, end);
+ next = pgd_addr_end(*pgd, addr, end);
if (!pgd_none(*pgd))
unmap_hyp_p4ds(pgd, addr, next);
} while (pgd++, addr = next, addr != end);
@@ -712,7 +712,7 @@ static int create_hyp_pmd_mappings(pud_t *pud, unsigned long start,
get_page(virt_to_page(pmd));
}
- next = pmd_addr_end(addr, end);
+ next = pmd_addr_end(*pmd, addr, end);
create_hyp_pte_mappings(pmd, addr, next, pfn, prot);
pfn += (next - addr) >> PAGE_SHIFT;
@@ -744,7 +744,7 @@ static int create_hyp_pud_mappings(p4d_t *p4d, unsigned long start,
get_page(virt_to_page(pud));
}
- next = pud_addr_end(addr, end);
+ next = pud_addr_end(*pud, addr, end);
ret = create_hyp_pmd_mappings(pud, addr, next, pfn, prot);
if (ret)
return ret;
@@ -777,7 +777,7 @@ static int create_hyp_p4d_mappings(pgd_t *pgd, unsigned long start,
get_page(virt_to_page(p4d));
}
- next = p4d_addr_end(addr, end);
+ next = p4d_addr_end(*p4d, addr, end);
ret = create_hyp_pud_mappings(p4d, addr, next, pfn, prot);
if (ret)
return ret;
@@ -813,7 +813,7 @@ static int __create_hyp_mappings(pgd_t *pgdp, unsigned long ptrs_per_pgd,
get_page(virt_to_page(pgd));
}
- next = pgd_addr_end(addr, end);
+ next = pgd_addr_end(*pgd, addr, end);
err = create_hyp_p4d_mappings(pgd, addr, next, pfn, prot);
if (err)
goto out;
diff --git a/arch/arm64/mm/kasan_init.c b/arch/arm64/mm/kasan_init.c
index b24e43d20667..8d1c811fd59e 100644
--- a/arch/arm64/mm/kasan_init.c
+++ b/arch/arm64/mm/kasan_init.c
@@ -120,7 +120,7 @@ static void __init kasan_pmd_populate(pud_t *pudp, unsigned long addr,
pmd_t *pmdp = kasan_pmd_offset(pudp, addr, node, early);
do {
- next = pmd_addr_end(addr, end);
+ next = pmd_addr_end(*pmdp, addr, end);
kasan_pte_populate(pmdp, addr, next, node, early);
} while (pmdp++, addr = next, addr != end && pmd_none(READ_ONCE(*pmdp)));
}
@@ -132,7 +132,7 @@ static void __init kasan_pud_populate(p4d_t *p4dp, unsigned long addr,
pud_t *pudp = kasan_pud_offset(p4dp, addr, node, early);
do {
- next = pud_addr_end(addr, end);
+ next = pud_addr_end(*pudp, addr, end);
kasan_pmd_populate(pudp, addr, next, node, early);
} while (pudp++, addr = next, addr != end && pud_none(READ_ONCE(*pudp)));
}
@@ -144,7 +144,7 @@ static void __init kasan_p4d_populate(pgd_t *pgdp, unsigned long addr,
p4d_t *p4dp = p4d_offset(pgdp, addr);
do {
- next = p4d_addr_end(addr, end);
+ next = p4d_addr_end(*p4dp, addr, end);
kasan_pud_populate(p4dp, addr, next, node, early);
} while (p4dp++, addr = next, addr != end);
}
@@ -157,7 +157,7 @@ static void __init kasan_pgd_populate(unsigned long addr, unsigned long end,
pgdp = pgd_offset_k(addr);
do {
- next = pgd_addr_end(addr, end);
+ next = pgd_addr_end(*pgdp, addr, end);
kasan_p4d_populate(pgdp, addr, next, node, early);
} while (pgdp++, addr = next, addr != end);
}
diff --git a/arch/arm64/mm/mmu.c b/arch/arm64/mm/mmu.c
index 64211436629d..d679cf024bc8 100644
--- a/arch/arm64/mm/mmu.c
+++ b/arch/arm64/mm/mmu.c
@@ -209,7 +209,7 @@ static void init_pmd(pud_t *pudp, unsigned long addr, unsigned long end,
do {
pmd_t old_pmd = READ_ONCE(*pmdp);
- next = pmd_addr_end(addr, end);
+ next = pmd_addr_end(old_pmd, addr, end);
/* try section mapping first */
if (((addr | next | phys) & ~SECTION_MASK) == 0 &&
@@ -307,7 +307,7 @@ static void alloc_init_pud(pgd_t *pgdp, unsigned long addr, unsigned long end,
do {
pud_t old_pud = READ_ONCE(*pudp);
- next = pud_addr_end(addr, end);
+ next = pud_addr_end(old_pud, addr, end);
/*
* For 4K granule only, attempt to put down a 1GB block
@@ -356,7 +356,7 @@ static void __create_pgd_mapping(pgd_t *pgdir, phys_addr_t phys,
end = PAGE_ALIGN(virt + size);
do {
- next = pgd_addr_end(addr, end);
+ next = pgd_addr_end(*pgdp, addr, end);
alloc_init_pud(pgdp, addr, next, phys, prot, pgtable_alloc,
flags);
phys += next - addr;
@@ -820,9 +820,9 @@ static void unmap_hotplug_pmd_range(pud_t *pudp, unsigned long addr,
pmd_t *pmdp, pmd;
do {
- next = pmd_addr_end(addr, end);
pmdp = pmd_offset(pudp, addr);
pmd = READ_ONCE(*pmdp);
+ next = pmd_addr_end(pmd, addr, end);
if (pmd_none(pmd))
continue;
@@ -853,9 +853,9 @@ static void unmap_hotplug_pud_range(p4d_t *p4dp, unsigned long addr,
pud_t *pudp, pud;
do {
- next = pud_addr_end(addr, end);
pudp = pud_offset(p4dp, addr);
pud = READ_ONCE(*pudp);
+ next = pud_addr_end(pud, addr, end);
if (pud_none(pud))
continue;
@@ -886,9 +886,9 @@ static void unmap_hotplug_p4d_range(pgd_t *pgdp, unsigned long addr,
p4d_t *p4dp, p4d;
do {
- next = p4d_addr_end(addr, end);
p4dp = p4d_offset(pgdp, addr);
p4d = READ_ONCE(*p4dp);
+ next = p4d_addr_end(p4d, addr, end);
if (p4d_none(p4d))
continue;
@@ -912,9 +912,9 @@ static void unmap_hotplug_range(unsigned long addr, unsigned long end,
WARN_ON(!free_mapped && altmap);
do {
- next = pgd_addr_end(addr, end);
pgdp = pgd_offset_k(addr);
pgd = READ_ONCE(*pgdp);
+ next = pgd_addr_end(pgd, addr, end);
if (pgd_none(pgd))
continue;
@@ -968,9 +968,9 @@ static void free_empty_pmd_table(pud_t *pudp, unsigned long addr,
unsigned long i, next, start = addr;
do {
- next = pmd_addr_end(addr, end);
pmdp = pmd_offset(pudp, addr);
pmd = READ_ONCE(*pmdp);
+ next = pmd_addr_end(pmd, addr, end);
if (pmd_none(pmd))
continue;
@@ -1008,9 +1008,9 @@ static void free_empty_pud_table(p4d_t *p4dp, unsigned long addr,
unsigned long i, next, start = addr;
do {
- next = pud_addr_end(addr, end);
pudp = pud_offset(p4dp, addr);
pud = READ_ONCE(*pudp);
+ next = pud_addr_end(pud, addr, end);
if (pud_none(pud))
continue;
@@ -1048,9 +1048,9 @@ static void free_empty_p4d_table(pgd_t *pgdp, unsigned long addr,
p4d_t *p4dp, p4d;
do {
- next = p4d_addr_end(addr, end);
p4dp = p4d_offset(pgdp, addr);
p4d = READ_ONCE(*p4dp);
+ next = p4d_addr_end(p4d, addr, end);
if (p4d_none(p4d))
continue;
@@ -1066,9 +1066,9 @@ static void free_empty_tables(unsigned long addr, unsigned long end,
pgd_t *pgdp, pgd;
do {
- next = pgd_addr_end(addr, end);
pgdp = pgd_offset_k(addr);
pgd = READ_ONCE(*pgdp);
+ next = pgd_addr_end(pgd, addr, end);
if (pgd_none(pgd))
continue;
@@ -1097,8 +1097,6 @@ int __meminit vmemmap_populate(unsigned long start, unsigned long end, int node,
pmd_t *pmdp;
do {
- next = pmd_addr_end(addr, end);
-
pgdp = vmemmap_pgd_populate(addr, node);
if (!pgdp)
return -ENOMEM;
@@ -1112,6 +1110,7 @@ int __meminit vmemmap_populate(unsigned long start, unsigned long end, int node,
return -ENOMEM;
pmdp = pmd_offset(pudp, addr);
+ next = pmd_addr_end(*pmdp, addr, end);
if (pmd_none(READ_ONCE(*pmdp))) {
void *p = NULL;
diff --git a/arch/powerpc/mm/book3s64/radix_pgtable.c b/arch/powerpc/mm/book3s64/radix_pgtable.c
index cc72666e891a..816e218df285 100644
--- a/arch/powerpc/mm/book3s64/radix_pgtable.c
+++ b/arch/powerpc/mm/book3s64/radix_pgtable.c
@@ -817,7 +817,7 @@ static void __meminit remove_pmd_table(pmd_t *pmd_start, unsigned long addr,
pmd = pmd_start + pmd_index(addr);
for (; addr < end; addr = next, pmd++) {
- next = pmd_addr_end(addr, end);
+ next = pmd_addr_end(*pmd, addr, end);
if (!pmd_present(*pmd))
continue;
@@ -847,7 +847,7 @@ static void __meminit remove_pud_table(pud_t *pud_start, unsigned long addr,
pud = pud_start + pud_index(addr);
for (; addr < end; addr = next, pud++) {
- next = pud_addr_end(addr, end);
+ next = pud_addr_end(*pud, addr, end);
if (!pud_present(*pud))
continue;
@@ -878,10 +878,9 @@ static void __meminit remove_pagetable(unsigned long start, unsigned long end)
spin_lock(&init_mm.page_table_lock);
for (addr = start; addr < end; addr = next) {
- next = pgd_addr_end(addr, end);
-
pgd = pgd_offset_k(addr);
p4d = p4d_offset(pgd, addr);
+ next = pgd_addr_end(*pgd, addr, end);
if (!p4d_present(*p4d))
continue;
diff --git a/arch/powerpc/mm/hugetlbpage.c b/arch/powerpc/mm/hugetlbpage.c
index 26292544630f..f0606d6774a4 100644
--- a/arch/powerpc/mm/hugetlbpage.c
+++ b/arch/powerpc/mm/hugetlbpage.c
@@ -352,7 +352,7 @@ static void hugetlb_free_pmd_range(struct mmu_gather *tlb, pud_t *pud,
unsigned long more;
pmd = pmd_offset(pud, addr);
- next = pmd_addr_end(addr, end);
+ next = pmd_addr_end(*pmd, addr, end);
if (!is_hugepd(__hugepd(pmd_val(*pmd)))) {
if (pmd_none_or_clear_bad(pmd))
continue;
@@ -409,7 +409,7 @@ static void hugetlb_free_pud_range(struct mmu_gather *tlb, p4d_t *p4d,
start = addr;
do {
pud = pud_offset(p4d, addr);
- next = pud_addr_end(addr, end);
+ next = pud_addr_end(*pud, addr, end);
if (!is_hugepd(__hugepd(pud_val(*pud)))) {
if (pud_none_or_clear_bad(pud))
continue;
@@ -478,9 +478,9 @@ void hugetlb_free_pgd_range(struct mmu_gather *tlb,
*/
do {
- next = pgd_addr_end(addr, end);
pgd = pgd_offset(tlb->mm, addr);
p4d = p4d_offset(pgd, addr);
+ next = pgd_addr_end(*pgd, addr, end);
if (!is_hugepd(__hugepd(pgd_val(*pgd)))) {
if (p4d_none_or_clear_bad(p4d))
continue;
diff --git a/arch/s390/include/asm/pgtable.h b/arch/s390/include/asm/pgtable.h
index 027206e4959d..6fb17ac413be 100644
--- a/arch/s390/include/asm/pgtable.h
+++ b/arch/s390/include/asm/pgtable.h
@@ -542,14 +542,14 @@ static inline unsigned long rste_addr_end_folded(unsigned long rste, unsigned lo
return (boundary - 1) < (end - 1) ? boundary : end;
}
-#define pgd_addr_end_folded pgd_addr_end_folded
-static inline unsigned long pgd_addr_end_folded(pgd_t pgd, unsigned long addr, unsigned long end)
+#define pgd_addr_end pgd_addr_end
+static inline unsigned long pgd_addr_end(pgd_t pgd, unsigned long addr, unsigned long end)
{
return rste_addr_end_folded(pgd_val(pgd), addr, end);
}
-#define p4d_addr_end_folded p4d_addr_end_folded
-static inline unsigned long p4d_addr_end_folded(p4d_t p4d, unsigned long addr, unsigned long end)
+#define p4d_addr_end p4d_addr_end
+static inline unsigned long p4d_addr_end(p4d_t p4d, unsigned long addr, unsigned long end)
{
return rste_addr_end_folded(p4d_val(p4d), addr, end);
}
diff --git a/arch/s390/mm/page-states.c b/arch/s390/mm/page-states.c
index 567c69f3069e..4aba634b4b26 100644
--- a/arch/s390/mm/page-states.c
+++ b/arch/s390/mm/page-states.c
@@ -109,7 +109,7 @@ static void mark_kernel_pmd(pud_t *pud, unsigned long addr, unsigned long end)
pmd = pmd_offset(pud, addr);
do {
- next = pmd_addr_end(addr, end);
+ next = pmd_addr_end(*pmd, addr, end);
if (pmd_none(*pmd) || pmd_large(*pmd))
continue;
page = virt_to_page(pmd_val(*pmd));
@@ -126,7 +126,7 @@ static void mark_kernel_pud(p4d_t *p4d, unsigned long addr, unsigned long end)
pud = pud_offset(p4d, addr);
do {
- next = pud_addr_end(addr, end);
+ next = pud_addr_end(*pud, addr, end);
if (pud_none(*pud) || pud_large(*pud))
continue;
if (!pud_folded(*pud)) {
@@ -147,7 +147,7 @@ static void mark_kernel_p4d(pgd_t *pgd, unsigned long addr, unsigned long end)
p4d = p4d_offset(pgd, addr);
do {
- next = p4d_addr_end(addr, end);
+ next = p4d_addr_end(*p4d, addr, end);
if (p4d_none(*p4d))
continue;
if (!p4d_folded(*p4d)) {
@@ -169,7 +169,7 @@ static void mark_kernel_pgd(void)
addr = 0;
pgd = pgd_offset_k(addr);
do {
- next = pgd_addr_end(addr, MODULES_END);
+ next = pgd_addr_end(*pgd, addr, MODULES_END);
if (pgd_none(*pgd))
continue;
if (!pgd_folded(*pgd)) {
diff --git a/arch/s390/mm/pageattr.c b/arch/s390/mm/pageattr.c
index c5c52ec2b46f..b827d758a17a 100644
--- a/arch/s390/mm/pageattr.c
+++ b/arch/s390/mm/pageattr.c
@@ -162,7 +162,7 @@ static int walk_pmd_level(pud_t *pudp, unsigned long addr, unsigned long end,
do {
if (pmd_none(*pmdp))
return -EINVAL;
- next = pmd_addr_end(addr, end);
+ next = pmd_addr_end(*pmdp, addr, end);
if (pmd_large(*pmdp)) {
if (addr & ~PMD_MASK || addr + PMD_SIZE > next) {
rc = split_pmd_page(pmdp, addr);
@@ -239,7 +239,7 @@ static int walk_pud_level(p4d_t *p4d, unsigned long addr, unsigned long end,
do {
if (pud_none(*pudp))
return -EINVAL;
- next = pud_addr_end(addr, end);
+ next = pud_addr_end(*pudp, addr, end);
if (pud_large(*pudp)) {
if (addr & ~PUD_MASK || addr + PUD_SIZE > next) {
rc = split_pud_page(pudp, addr);
@@ -269,7 +269,7 @@ static int walk_p4d_level(pgd_t *pgd, unsigned long addr, unsigned long end,
do {
if (p4d_none(*p4dp))
return -EINVAL;
- next = p4d_addr_end(addr, end);
+ next = p4d_addr_end(*p4dp, addr, end);
rc = walk_pud_level(p4dp, addr, next, flags);
p4dp++;
addr = next;
@@ -296,7 +296,7 @@ static int change_page_attr(unsigned long addr, unsigned long end,
do {
if (pgd_none(*pgdp))
break;
- next = pgd_addr_end(addr, end);
+ next = pgd_addr_end(*pgdp, addr, end);
rc = walk_p4d_level(pgdp, addr, next, flags);
if (rc)
break;
diff --git a/arch/s390/mm/vmem.c b/arch/s390/mm/vmem.c
index b239f2ba93b0..672bc89f13e7 100644
--- a/arch/s390/mm/vmem.c
+++ b/arch/s390/mm/vmem.c
@@ -219,7 +219,7 @@ static int __ref modify_pmd_table(pud_t *pud, unsigned long addr,
pmd = pmd_offset(pud, addr);
for (; addr < end; addr = next, pmd++) {
- next = pmd_addr_end(addr, end);
+ next = pmd_addr_end(*pmd, addr, end);
if (!add) {
if (pmd_none(*pmd))
continue;
@@ -320,7 +320,7 @@ static int modify_pud_table(p4d_t *p4d, unsigned long addr, unsigned long end,
prot &= ~_REGION_ENTRY_NOEXEC;
pud = pud_offset(p4d, addr);
for (; addr < end; addr = next, pud++) {
- next = pud_addr_end(addr, end);
+ next = pud_addr_end(*pud, addr, end);
if (!add) {
if (pud_none(*pud))
continue;
@@ -394,7 +394,7 @@ static int modify_p4d_table(pgd_t *pgd, unsigned long addr, unsigned long end,
p4d = p4d_offset(pgd, addr);
for (; addr < end; addr = next, p4d++) {
- next = p4d_addr_end(addr, end);
+ next = p4d_addr_end(*p4d, addr, end);
if (!add) {
if (p4d_none(*p4d))
continue;
@@ -449,8 +449,8 @@ static int modify_pagetable(unsigned long start, unsigned long end, bool add,
if (WARN_ON_ONCE(!PAGE_ALIGNED(start | end)))
return -EINVAL;
for (addr = start; addr < end; addr = next) {
- next = pgd_addr_end(addr, end);
pgd = pgd_offset_k(addr);
+ next = pgd_addr_end(*pgd, addr, end);
if (!add) {
if (pgd_none(*pgd))
diff --git a/arch/sparc/mm/hugetlbpage.c b/arch/sparc/mm/hugetlbpage.c
index ec423b5f17dd..341c2ff8d31a 100644
--- a/arch/sparc/mm/hugetlbpage.c
+++ b/arch/sparc/mm/hugetlbpage.c
@@ -428,7 +428,7 @@ static void hugetlb_free_pmd_range(struct mmu_gather *tlb, pud_t *pud,
start = addr;
pmd = pmd_offset(pud, addr);
do {
- next = pmd_addr_end(addr, end);
+ next = pmd_addr_end(*pmd, addr, end);
if (pmd_none(*pmd))
continue;
if (is_hugetlb_pmd(*pmd))
@@ -465,7 +465,7 @@ static void hugetlb_free_pud_range(struct mmu_gather *tlb, p4d_t *p4d,
start = addr;
pud = pud_offset(p4d, addr);
do {
- next = pud_addr_end(addr, end);
+ next = pud_addr_end(*pud, addr, end);
if (pud_none_or_clear_bad(pud))
continue;
if (is_hugetlb_pud(*pud))
@@ -519,7 +519,7 @@ void hugetlb_free_pgd_range(struct mmu_gather *tlb,
pgd = pgd_offset(tlb->mm, addr);
p4d = p4d_offset(pgd, addr);
do {
- next = p4d_addr_end(addr, end);
+ next = p4d_addr_end(*p4d, addr, end);
if (p4d_none_or_clear_bad(p4d))
continue;
hugetlb_free_pud_range(tlb, p4d, addr, next, floor, ceiling);
diff --git a/arch/um/kernel/tlb.c b/arch/um/kernel/tlb.c
index 61776790cd67..7b4fe31c8df2 100644
--- a/arch/um/kernel/tlb.c
+++ b/arch/um/kernel/tlb.c
@@ -264,7 +264,7 @@ static inline int update_pmd_range(pud_t *pud, unsigned long addr,
pmd = pmd_offset(pud, addr);
do {
- next = pmd_addr_end(addr, end);
+ next = pmd_addr_end(*pmd, addr, end);
if (!pmd_present(*pmd)) {
if (hvc->force || pmd_newpage(*pmd)) {
ret = add_munmap(addr, next - addr, hvc);
@@ -286,7 +286,7 @@ static inline int update_pud_range(p4d_t *p4d, unsigned long addr,
pud = pud_offset(p4d, addr);
do {
- next = pud_addr_end(addr, end);
+ next = pud_addr_end(*pud, addr, end);
if (!pud_present(*pud)) {
if (hvc->force || pud_newpage(*pud)) {
ret = add_munmap(addr, next - addr, hvc);
@@ -308,7 +308,7 @@ static inline int update_p4d_range(pgd_t *pgd, unsigned long addr,
p4d = p4d_offset(pgd, addr);
do {
- next = p4d_addr_end(addr, end);
+ next = p4d_addr_end(*p4d, addr, end);
if (!p4d_present(*p4d)) {
if (hvc->force || p4d_newpage(*p4d)) {
ret = add_munmap(addr, next - addr, hvc);
@@ -331,7 +331,7 @@ void fix_range_common(struct mm_struct *mm, unsigned long start_addr,
hvc = INIT_HVC(mm, force, userspace);
pgd = pgd_offset(mm, addr);
do {
- next = pgd_addr_end(addr, end_addr);
+ next = pgd_addr_end(*pgd, addr, end_addr);
if (!pgd_present(*pgd)) {
if (force || pgd_newpage(*pgd)) {
ret = add_munmap(addr, next - addr, &hvc);
diff --git a/arch/x86/mm/init_64.c b/arch/x86/mm/init_64.c
index a4ac13cc3fdc..e2cb9316a104 100644
--- a/arch/x86/mm/init_64.c
+++ b/arch/x86/mm/init_64.c
@@ -1043,7 +1043,7 @@ remove_pmd_table(pmd_t *pmd_start, unsigned long addr, unsigned long end,
pmd = pmd_start + pmd_index(addr);
for (; addr < end; addr = next, pmd++) {
- next = pmd_addr_end(addr, end);
+ next = pmd_addr_end(*pmd, addr, end);
if (!pmd_present(*pmd))
continue;
@@ -1099,7 +1099,7 @@ remove_pud_table(pud_t *pud_start, unsigned long addr, unsigned long end,
pud = pud_start + pud_index(addr);
for (; addr < end; addr = next, pud++) {
- next = pud_addr_end(addr, end);
+ next = pud_addr_end(*pud, addr, end);
if (!pud_present(*pud))
continue;
@@ -1153,7 +1153,7 @@ remove_p4d_table(p4d_t *p4d_start, unsigned long addr, unsigned long end,
p4d = p4d_start + p4d_index(addr);
for (; addr < end; addr = next, p4d++) {
- next = p4d_addr_end(addr, end);
+ next = p4d_addr_end(*p4d, addr, end);
if (!p4d_present(*p4d))
continue;
@@ -1186,9 +1186,8 @@ remove_pagetable(unsigned long start, unsigned long end, bool direct,
p4d_t *p4d;
for (addr = start; addr < end; addr = next) {
- next = pgd_addr_end(addr, end);
-
pgd = pgd_offset_k(addr);
+ next = pgd_addr_end(*pgd, addr, end);
if (!pgd_present(*pgd))
continue;
@@ -1500,8 +1499,6 @@ static int __meminit vmemmap_populate_hugepages(unsigned long start,
pmd_t *pmd;
for (addr = start; addr < end; addr = next) {
- next = pmd_addr_end(addr, end);
-
pgd = vmemmap_pgd_populate(addr, node);
if (!pgd)
return -ENOMEM;
@@ -1515,6 +1512,7 @@ static int __meminit vmemmap_populate_hugepages(unsigned long start,
return -ENOMEM;
pmd = pmd_offset(pud, addr);
+ next = pmd_addr_end(*pmd, addr, end);
if (pmd_none(*pmd)) {
void *p;
@@ -1623,9 +1621,8 @@ void register_page_bootmem_memmap(unsigned long section_nr,
get_page_bootmem(section_nr, pte_page(*pte),
SECTION_INFO);
} else {
- next = pmd_addr_end(addr, end);
-
pmd = pmd_offset(pud, addr);
+ next = pmd_addr_end(*pmd, addr, end);
if (pmd_none(*pmd))
continue;
diff --git a/arch/x86/mm/kasan_init_64.c b/arch/x86/mm/kasan_init_64.c
index 1a50434c8a4d..2c105b5154ba 100644
--- a/arch/x86/mm/kasan_init_64.c
+++ b/arch/x86/mm/kasan_init_64.c
@@ -96,7 +96,7 @@ static void __init kasan_populate_pud(pud_t *pud, unsigned long addr,
pmd = pmd_offset(pud, addr);
do {
- next = pmd_addr_end(addr, end);
+ next = pmd_addr_end(*pmd, addr, end);
if (!pmd_large(*pmd))
kasan_populate_pmd(pmd, addr, next, nid);
} while (pmd++, addr = next, addr != end);
@@ -116,7 +116,7 @@ static void __init kasan_populate_p4d(p4d_t *p4d, unsigned long addr,
pud = pud_offset(p4d, addr);
do {
- next = pud_addr_end(addr, end);
+ next = pud_addr_end(*pud, addr, end);
if (!pud_large(*pud))
kasan_populate_pud(pud, addr, next, nid);
} while (pud++, addr = next, addr != end);
@@ -136,7 +136,7 @@ static void __init kasan_populate_pgd(pgd_t *pgd, unsigned long addr,
p4d = p4d_offset(pgd, addr);
do {
- next = p4d_addr_end(addr, end);
+ next = p4d_addr_end(*p4d, addr, end);
kasan_populate_p4d(p4d, addr, next, nid);
} while (p4d++, addr = next, addr != end);
}
@@ -151,7 +151,7 @@ static void __init kasan_populate_shadow(unsigned long addr, unsigned long end,
end = round_up(end, PAGE_SIZE);
pgd = pgd_offset_k(addr);
do {
- next = pgd_addr_end(addr, end);
+ next = pgd_addr_end(*pgd, addr, end);
kasan_populate_pgd(pgd, addr, next, nid);
} while (pgd++, addr = next, addr != end);
}
@@ -219,7 +219,7 @@ static void __init kasan_early_p4d_populate(pgd_t *pgd,
p4d = early_p4d_offset(pgd, addr);
do {
- next = p4d_addr_end(addr, end);
+ next = p4d_addr_end(*p4d, addr, end);
if (!p4d_none(*p4d))
continue;
@@ -239,7 +239,7 @@ static void __init kasan_map_early_shadow(pgd_t *pgd)
pgd += pgd_index(addr);
do {
- next = pgd_addr_end(addr, end);
+ next = pgd_addr_end(*pgd, addr, end);
kasan_early_p4d_populate(pgd, addr, next);
} while (pgd++, addr = next, addr != end);
}
@@ -254,7 +254,7 @@ static void __init kasan_shallow_populate_p4ds(pgd_t *pgd,
p4d = p4d_offset(pgd, addr);
do {
- next = p4d_addr_end(addr, end);
+ next = p4d_addr_end(*p4d, addr, end);
if (p4d_none(*p4d)) {
p = early_alloc(PAGE_SIZE, NUMA_NO_NODE, true);
@@ -272,7 +272,7 @@ static void __init kasan_shallow_populate_pgds(void *start, void *end)
addr = (unsigned long)start;
pgd = pgd_offset_k(addr);
do {
- next = pgd_addr_end(addr, (unsigned long)end);
+ next = pgd_addr_end(*pgd, addr, (unsigned long)end);
if (pgd_none(*pgd)) {
p = early_alloc(PAGE_SIZE, NUMA_NO_NODE, true);
diff --git a/include/asm-generic/pgtable-nop4d.h b/include/asm-generic/pgtable-nop4d.h
index ce2cbb3c380f..156b42e51424 100644
--- a/include/asm-generic/pgtable-nop4d.h
+++ b/include/asm-generic/pgtable-nop4d.h
@@ -53,7 +53,7 @@ static inline p4d_t *p4d_offset(pgd_t *pgd, unsigned long address)
#define p4d_free_tlb(tlb, x, a) do { } while (0)
#undef p4d_addr_end
-#define p4d_addr_end(addr, end) (end)
+#define p4d_addr_end(p4d, addr, end) (end)
#endif /* __ASSEMBLY__ */
#endif /* _PGTABLE_NOP4D_H */
diff --git a/include/asm-generic/pgtable-nopmd.h b/include/asm-generic/pgtable-nopmd.h
index 3e13acd019ae..e988384de1c7 100644
--- a/include/asm-generic/pgtable-nopmd.h
+++ b/include/asm-generic/pgtable-nopmd.h
@@ -64,7 +64,7 @@ static inline void pmd_free(struct mm_struct *mm, pmd_t *pmd)
#define pmd_free_tlb(tlb, x, a) do { } while (0)
#undef pmd_addr_end
-#define pmd_addr_end(addr, end) (end)
+#define pmd_addr_end(pmd, addr, end) (end)
#endif /* __ASSEMBLY__ */
diff --git a/include/asm-generic/pgtable-nopud.h b/include/asm-generic/pgtable-nopud.h
index a9d751fbda9e..57a28bade9f9 100644
--- a/include/asm-generic/pgtable-nopud.h
+++ b/include/asm-generic/pgtable-nopud.h
@@ -60,7 +60,7 @@ static inline pud_t *pud_offset(p4d_t *p4d, unsigned long address)
#define pud_free_tlb(tlb, x, a) do { } while (0)
#undef pud_addr_end
-#define pud_addr_end(addr, end) (end)
+#define pud_addr_end(pud, addr, end) (end)
#endif /* __ASSEMBLY__ */
#endif /* _PGTABLE_NOPUD_H */
diff --git a/include/linux/pgtable.h b/include/linux/pgtable.h
index 981c4c2a31fe..67ebc22cf83d 100644
--- a/include/linux/pgtable.h
+++ b/include/linux/pgtable.h
@@ -655,48 +655,34 @@ static inline int arch_unmap_one(struct mm_struct *mm,
* vma end wraps to 0, rounded up __boundary may wrap to 0 throughout.
*/
-#define pgd_addr_end(addr, end) \
+#ifndef pgd_addr_end
+#define pgd_addr_end(pgd, addr, end) \
({ unsigned long __boundary = ((addr) + PGDIR_SIZE) & PGDIR_MASK; \
(__boundary - 1 < (end) - 1)? __boundary: (end); \
})
+#endif
#ifndef p4d_addr_end
-#define p4d_addr_end(addr, end) \
+#define p4d_addr_end(p4d, addr, end) \
({ unsigned long __boundary = ((addr) + P4D_SIZE) & P4D_MASK; \
(__boundary - 1 < (end) - 1)? __boundary: (end); \
})
#endif
#ifndef pud_addr_end
-#define pud_addr_end(addr, end) \
+#define pud_addr_end(pud, addr, end) \
({ unsigned long __boundary = ((addr) + PUD_SIZE) & PUD_MASK; \
(__boundary - 1 < (end) - 1)? __boundary: (end); \
})
#endif
#ifndef pmd_addr_end
-#define pmd_addr_end(addr, end) \
+#define pmd_addr_end(pmd, addr, end) \
({ unsigned long __boundary = ((addr) + PMD_SIZE) & PMD_MASK; \
(__boundary - 1 < (end) - 1)? __boundary: (end); \
})
#endif
-#ifndef pgd_addr_end_folded
-#define pgd_addr_end_folded(pgd, addr, end) pgd_addr_end(addr, end)
-#endif
-
-#ifndef p4d_addr_end_folded
-#define p4d_addr_end_folded(p4d, addr, end) p4d_addr_end(addr, end)
-#endif
-
-#ifndef pud_addr_end_folded
-#define pud_addr_end_folded(pud, addr, end) pud_addr_end(addr, end)
-#endif
-
-#ifndef pmd_addr_end_folded
-#define pmd_addr_end_folded(pmd, addr, end) pmd_addr_end(addr, end)
-#endif
-
/*
* When walking page tables, we usually want to skip any p?d_none entries;
* and any p?d_bad entries - reporting the error before resetting to none.
diff --git a/mm/gup.c b/mm/gup.c
index ba4aace5d0f4..7826876ae7e0 100644
--- a/mm/gup.c
+++ b/mm/gup.c
@@ -2521,7 +2521,7 @@ static int gup_pmd_range(pud_t pud, unsigned long addr, unsigned long end,
do {
pmd_t pmd = READ_ONCE(*pmdp);
- next = pmd_addr_end_folded(pmd, addr, end);
+ next = pmd_addr_end(pmd, addr, end);
if (!pmd_present(pmd))
return 0;
@@ -2564,7 +2564,7 @@ static int gup_pud_range(p4d_t p4d, unsigned long addr, unsigned long end,
do {
pud_t pud = READ_ONCE(*pudp);
- next = pud_addr_end_folded(pud, addr, end);
+ next = pud_addr_end(pud, addr, end);
if (unlikely(!pud_present(pud)))
return 0;
if (unlikely(pud_huge(pud))) {
@@ -2592,7 +2592,7 @@ static int gup_p4d_range(pgd_t pgd, unsigned long addr, unsigned long end,
do {
p4d_t p4d = READ_ONCE(*p4dp);
- next = p4d_addr_end_folded(p4d, addr, end);
+ next = p4d_addr_end(p4d, addr, end);
if (p4d_none(p4d))
return 0;
BUILD_BUG_ON(p4d_huge(p4d));
@@ -2617,7 +2617,7 @@ static void gup_pgd_range(unsigned long addr, unsigned long end,
do {
pgd_t pgd = READ_ONCE(*pgdp);
- next = pgd_addr_end_folded(pgd, addr, end);
+ next = pgd_addr_end(pgd, addr, end);
if (pgd_none(pgd))
return;
if (unlikely(pgd_huge(pgd))) {
diff --git a/mm/ioremap.c b/mm/ioremap.c
index 5fa1ab41d152..400fa119c09d 100644
--- a/mm/ioremap.c
+++ b/mm/ioremap.c
@@ -114,7 +114,7 @@ static inline int ioremap_pmd_range(pud_t *pud, unsigned long addr,
if (!pmd)
return -ENOMEM;
do {
- next = pmd_addr_end(addr, end);
+ next = pmd_addr_end(*pmd, addr, end);
if (ioremap_try_huge_pmd(pmd, addr, next, phys_addr, prot)) {
*mask |= PGTBL_PMD_MODIFIED;
@@ -160,7 +160,7 @@ static inline int ioremap_pud_range(p4d_t *p4d, unsigned long addr,
if (!pud)
return -ENOMEM;
do {
- next = pud_addr_end(addr, end);
+ next = pud_addr_end(*pud, addr, end);
if (ioremap_try_huge_pud(pud, addr, next, phys_addr, prot)) {
*mask |= PGTBL_PUD_MODIFIED;
@@ -206,7 +206,7 @@ static inline int ioremap_p4d_range(pgd_t *pgd, unsigned long addr,
if (!p4d)
return -ENOMEM;
do {
- next = p4d_addr_end(addr, end);
+ next = p4d_addr_end(*p4d, addr, end);
if (ioremap_try_huge_p4d(p4d, addr, next, phys_addr, prot)) {
*mask |= PGTBL_P4D_MODIFIED;
@@ -234,7 +234,7 @@ int ioremap_page_range(unsigned long addr,
start = addr;
pgd = pgd_offset_k(addr);
do {
- next = pgd_addr_end(addr, end);
+ next = pgd_addr_end(*pgd, addr, end);
err = ioremap_p4d_range(pgd, addr, next, phys_addr, prot,
&mask);
if (err)
diff --git a/mm/kasan/init.c b/mm/kasan/init.c
index fe6be0be1f76..829627a92763 100644
--- a/mm/kasan/init.c
+++ b/mm/kasan/init.c
@@ -117,7 +117,7 @@ static int __ref zero_pmd_populate(pud_t *pud, unsigned long addr,
unsigned long next;
do {
- next = pmd_addr_end(addr, end);
+ next = pmd_addr_end(*pmd, addr, end);
if (IS_ALIGNED(addr, PMD_SIZE) && end - addr >= PMD_SIZE) {
pmd_populate_kernel(&init_mm, pmd,
@@ -150,7 +150,7 @@ static int __ref zero_pud_populate(p4d_t *p4d, unsigned long addr,
unsigned long next;
do {
- next = pud_addr_end(addr, end);
+ next = pud_addr_end(*pud, addr, end);
if (IS_ALIGNED(addr, PUD_SIZE) && end - addr >= PUD_SIZE) {
pmd_t *pmd;
@@ -187,7 +187,7 @@ static int __ref zero_p4d_populate(pgd_t *pgd, unsigned long addr,
unsigned long next;
do {
- next = p4d_addr_end(addr, end);
+ next = p4d_addr_end(*p4d, addr, end);
if (IS_ALIGNED(addr, P4D_SIZE) && end - addr >= P4D_SIZE) {
pud_t *pud;
pmd_t *pmd;
@@ -236,7 +236,7 @@ int __ref kasan_populate_early_shadow(const void *shadow_start,
unsigned long next;
do {
- next = pgd_addr_end(addr, end);
+ next = pgd_addr_end(*pgd, addr, end);
if (IS_ALIGNED(addr, PGDIR_SIZE) && end - addr >= PGDIR_SIZE) {
p4d_t *p4d;
@@ -370,7 +370,7 @@ static void kasan_remove_pmd_table(pmd_t *pmd, unsigned long addr,
for (; addr < end; addr = next, pmd++) {
pte_t *pte;
- next = pmd_addr_end(addr, end);
+ next = pmd_addr_end(*pmd, addr, end);
if (!pmd_present(*pmd))
continue;
@@ -395,7 +395,7 @@ static void kasan_remove_pud_table(pud_t *pud, unsigned long addr,
for (; addr < end; addr = next, pud++) {
pmd_t *pmd, *pmd_base;
- next = pud_addr_end(addr, end);
+ next = pud_addr_end(*pud, addr, end);
if (!pud_present(*pud))
continue;
@@ -421,7 +421,7 @@ static void kasan_remove_p4d_table(p4d_t *p4d, unsigned long addr,
for (; addr < end; addr = next, p4d++) {
pud_t *pud;
- next = p4d_addr_end(addr, end);
+ next = p4d_addr_end(*p4d, addr, end);
if (!p4d_present(*p4d))
continue;
@@ -454,9 +454,8 @@ void kasan_remove_zero_shadow(void *start, unsigned long size)
for (; addr < end; addr = next) {
p4d_t *p4d;
- next = pgd_addr_end(addr, end);
-
pgd = pgd_offset_k(addr);
+ next = pgd_addr_end(*pgd, addr, end);
if (!pgd_present(*pgd))
continue;
diff --git a/mm/madvise.c b/mm/madvise.c
index e32e7efbba0f..acfb3441d97e 100644
--- a/mm/madvise.c
+++ b/mm/madvise.c
@@ -326,7 +326,7 @@ static int madvise_cold_or_pageout_pte_range(pmd_t *pmd,
#ifdef CONFIG_TRANSPARENT_HUGEPAGE
if (pmd_trans_huge(*pmd)) {
pmd_t orig_pmd;
- unsigned long next = pmd_addr_end(addr, end);
+ unsigned long next = pmd_addr_end(*pmd, addr, end);
tlb_change_page_size(tlb, HPAGE_PMD_SIZE);
ptl = pmd_trans_huge_lock(pmd, vma);
@@ -587,7 +587,7 @@ static int madvise_free_pte_range(pmd_t *pmd, unsigned long addr,
int nr_swap = 0;
unsigned long next;
- next = pmd_addr_end(addr, end);
+ next = pmd_addr_end(*pmd, addr, end);
if (pmd_trans_huge(*pmd))
if (madvise_free_huge_pmd(tlb, vma, pmd, addr, next))
goto next;
diff --git a/mm/memory.c b/mm/memory.c
index fb5463153351..f95424946b0d 100644
--- a/mm/memory.c
+++ b/mm/memory.c
@@ -233,7 +233,7 @@ static inline void free_pmd_range(struct mmu_gather *tlb, pud_t *pud,
start = addr;
pmd = pmd_offset(pud, addr);
do {
- next = pmd_addr_end(addr, end);
+ next = pmd_addr_end(*pmd, addr, end);
if (pmd_none_or_clear_bad(pmd))
continue;
free_pte_range(tlb, pmd, addr);
@@ -267,7 +267,7 @@ static inline void free_pud_range(struct mmu_gather *tlb, p4d_t *p4d,
start = addr;
pud = pud_offset(p4d, addr);
do {
- next = pud_addr_end(addr, end);
+ next = pud_addr_end(*pud, addr, end);
if (pud_none_or_clear_bad(pud))
continue;
free_pmd_range(tlb, pud, addr, next, floor, ceiling);
@@ -301,7 +301,7 @@ static inline void free_p4d_range(struct mmu_gather *tlb, pgd_t *pgd,
start = addr;
p4d = p4d_offset(pgd, addr);
do {
- next = p4d_addr_end(addr, end);
+ next = p4d_addr_end(*p4d, addr, end);
if (p4d_none_or_clear_bad(p4d))
continue;
free_pud_range(tlb, p4d, addr, next, floor, ceiling);
@@ -381,7 +381,7 @@ void free_pgd_range(struct mmu_gather *tlb,
tlb_change_page_size(tlb, PAGE_SIZE);
pgd = pgd_offset(tlb->mm, addr);
do {
- next = pgd_addr_end(addr, end);
+ next = pgd_addr_end(*pgd, addr, end);
if (pgd_none_or_clear_bad(pgd))
continue;
free_p4d_range(tlb, pgd, addr, next, floor, ceiling);
@@ -887,7 +887,7 @@ static inline int copy_pmd_range(struct mm_struct *dst_mm, struct mm_struct *src
return -ENOMEM;
src_pmd = pmd_offset(src_pud, addr);
do {
- next = pmd_addr_end(addr, end);
+ next = pmd_addr_end(*src_pmd, addr, end);
if (is_swap_pmd(*src_pmd) || pmd_trans_huge(*src_pmd)
|| pmd_devmap(*src_pmd)) {
int err;
@@ -921,7 +921,7 @@ static inline int copy_pud_range(struct mm_struct *dst_mm, struct mm_struct *src
return -ENOMEM;
src_pud = pud_offset(src_p4d, addr);
do {
- next = pud_addr_end(addr, end);
+ next = pud_addr_end(*src_pud, addr, end);
if (pud_trans_huge(*src_pud) || pud_devmap(*src_pud)) {
int err;
@@ -955,7 +955,7 @@ static inline int copy_p4d_range(struct mm_struct *dst_mm, struct mm_struct *src
return -ENOMEM;
src_p4d = p4d_offset(src_pgd, addr);
do {
- next = p4d_addr_end(addr, end);
+ next = p4d_addr_end(*src_p4d, addr, end);
if (p4d_none_or_clear_bad(src_p4d))
continue;
if (copy_pud_range(dst_mm, src_mm, dst_p4d, src_p4d,
@@ -1017,7 +1017,7 @@ int copy_page_range(struct mm_struct *dst_mm, struct mm_struct *src_mm,
dst_pgd = pgd_offset(dst_mm, addr);
src_pgd = pgd_offset(src_mm, addr);
do {
- next = pgd_addr_end(addr, end);
+ next = pgd_addr_end(*src_pgd, addr, end);
if (pgd_none_or_clear_bad(src_pgd))
continue;
if (unlikely(copy_p4d_range(dst_mm, src_mm, dst_pgd, src_pgd,
@@ -1177,7 +1177,7 @@ static inline unsigned long zap_pmd_range(struct mmu_gather *tlb,
pmd = pmd_offset(pud, addr);
do {
- next = pmd_addr_end(addr, end);
+ next = pmd_addr_end(*pmd, addr, end);
if (is_swap_pmd(*pmd) || pmd_trans_huge(*pmd) || pmd_devmap(*pmd)) {
if (next - addr != HPAGE_PMD_SIZE)
__split_huge_pmd(vma, pmd, addr, false, NULL);
@@ -1212,7 +1212,7 @@ static inline unsigned long zap_pud_range(struct mmu_gather *tlb,
pud = pud_offset(p4d, addr);
do {
- next = pud_addr_end(addr, end);
+ next = pud_addr_end(*pud, addr, end);
if (pud_trans_huge(*pud) || pud_devmap(*pud)) {
if (next - addr != HPAGE_PUD_SIZE) {
mmap_assert_locked(tlb->mm);
@@ -1241,7 +1241,7 @@ static inline unsigned long zap_p4d_range(struct mmu_gather *tlb,
p4d = p4d_offset(pgd, addr);
do {
- next = p4d_addr_end(addr, end);
+ next = p4d_addr_end(*p4d, addr, end);
if (p4d_none_or_clear_bad(p4d))
continue;
next = zap_pud_range(tlb, vma, p4d, addr, next, details);
@@ -1262,7 +1262,7 @@ void unmap_page_range(struct mmu_gather *tlb,
tlb_start_vma(tlb, vma);
pgd = pgd_offset(vma->vm_mm, addr);
do {
- next = pgd_addr_end(addr, end);
+ next = pgd_addr_end(*pgd, addr, end);
if (pgd_none_or_clear_bad(pgd))
continue;
next = zap_p4d_range(tlb, vma, pgd, addr, next, details);
@@ -2030,7 +2030,7 @@ static inline int remap_pmd_range(struct mm_struct *mm, pud_t *pud,
return -ENOMEM;
VM_BUG_ON(pmd_trans_huge(*pmd));
do {
- next = pmd_addr_end(addr, end);
+ next = pmd_addr_end(*pmd, addr, end);
err = remap_pte_range(mm, pmd, addr, next,
pfn + (addr >> PAGE_SHIFT), prot);
if (err)
@@ -2052,7 +2052,7 @@ static inline int remap_pud_range(struct mm_struct *mm, p4d_t *p4d,
if (!pud)
return -ENOMEM;
do {
- next = pud_addr_end(addr, end);
+ next = pud_addr_end(*pud, addr, end);
err = remap_pmd_range(mm, pud, addr, next,
pfn + (addr >> PAGE_SHIFT), prot);
if (err)
@@ -2074,7 +2074,7 @@ static inline int remap_p4d_range(struct mm_struct *mm, pgd_t *pgd,
if (!p4d)
return -ENOMEM;
do {
- next = p4d_addr_end(addr, end);
+ next = p4d_addr_end(*p4d, addr, end);
err = remap_pud_range(mm, p4d, addr, next,
pfn + (addr >> PAGE_SHIFT), prot);
if (err)
@@ -2143,7 +2143,7 @@ int remap_pfn_range(struct vm_area_struct *vma, unsigned long addr,
pgd = pgd_offset(mm, addr);
flush_cache_range(vma, addr, end);
do {
- next = pgd_addr_end(addr, end);
+ next = pgd_addr_end(*pgd, addr, end);
err = remap_p4d_range(mm, pgd, addr, next,
pfn + (addr >> PAGE_SHIFT), prot);
if (err)
@@ -2266,7 +2266,7 @@ static int apply_to_pmd_range(struct mm_struct *mm, pud_t *pud,
pmd = pmd_offset(pud, addr);
}
do {
- next = pmd_addr_end(addr, end);
+ next = pmd_addr_end(*pmd, addr, end);
if (create || !pmd_none_or_clear_bad(pmd)) {
err = apply_to_pte_range(mm, pmd, addr, next, fn, data,
create, mask);
@@ -2294,7 +2294,7 @@ static int apply_to_pud_range(struct mm_struct *mm, p4d_t *p4d,
pud = pud_offset(p4d, addr);
}
do {
- next = pud_addr_end(addr, end);
+ next = pud_addr_end(*pud, addr, end);
if (create || !pud_none_or_clear_bad(pud)) {
err = apply_to_pmd_range(mm, pud, addr, next, fn, data,
create, mask);
@@ -2322,7 +2322,7 @@ static int apply_to_p4d_range(struct mm_struct *mm, pgd_t *pgd,
p4d = p4d_offset(pgd, addr);
}
do {
- next = p4d_addr_end(addr, end);
+ next = p4d_addr_end(*p4d, addr, end);
if (create || !p4d_none_or_clear_bad(p4d)) {
err = apply_to_pud_range(mm, p4d, addr, next, fn, data,
create, mask);
@@ -2348,7 +2348,7 @@ static int __apply_to_page_range(struct mm_struct *mm, unsigned long addr,
pgd = pgd_offset(mm, addr);
do {
- next = pgd_addr_end(addr, end);
+ next = pgd_addr_end(*pgd, addr, end);
if (!create && pgd_none_or_clear_bad(pgd))
continue;
err = apply_to_p4d_range(mm, pgd, addr, next, fn, data, create, &mask);
diff --git a/mm/mlock.c b/mm/mlock.c
index 93ca2bf30b4f..5898e8fe2288 100644
--- a/mm/mlock.c
+++ b/mm/mlock.c
@@ -374,8 +374,12 @@ static unsigned long __munlock_pagevec_fill(struct pagevec *pvec,
struct vm_area_struct *vma, struct zone *zone,
unsigned long start, unsigned long end)
{
- pte_t *pte;
spinlock_t *ptl;
+ pgd_t *pgd;
+ p4d_t *p4d;
+ pud_t *pud;
+ pmd_t *pmd;
+ pte_t *pte;
/*
* Initialize pte walk starting at the already pinned page where we
@@ -384,10 +388,14 @@ static unsigned long __munlock_pagevec_fill(struct pagevec *pvec,
*/
pte = get_locked_pte(vma->vm_mm, start, &ptl);
/* Make sure we do not cross the page table boundary */
- end = pgd_addr_end(start, end);
- end = p4d_addr_end(start, end);
- end = pud_addr_end(start, end);
- end = pmd_addr_end(start, end);
+ pgd = pgd_offset(vma->vm_mm, start);
+ end = pgd_addr_end(*pgd, start, end);
+ p4d = p4d_offset(pgd, start);
+ end = p4d_addr_end(*p4d, start, end);
+ pud = pud_offset(p4d, start);
+ end = pud_addr_end(*pud, start, end);
+ pmd = pmd_offset(pud, start);
+ end = pmd_addr_end(*pmd, start, end);
/* The page next to the pinned page is the first we will try to get */
start += PAGE_SIZE;
diff --git a/mm/mprotect.c b/mm/mprotect.c
index ce8b8a5eacbb..278f2dbd1f20 100644
--- a/mm/mprotect.c
+++ b/mm/mprotect.c
@@ -225,7 +225,7 @@ static inline unsigned long change_pmd_range(struct vm_area_struct *vma,
do {
unsigned long this_pages;
- next = pmd_addr_end(addr, end);
+ next = pmd_addr_end(*pmd, addr, end);
/*
* Automatic NUMA balancing walks the tables with mmap_lock
@@ -291,7 +291,7 @@ static inline unsigned long change_pud_range(struct vm_area_struct *vma,
pud = pud_offset(p4d, addr);
do {
- next = pud_addr_end(addr, end);
+ next = pud_addr_end(*pud, addr, end);
if (pud_none_or_clear_bad(pud))
continue;
pages += change_pmd_range(vma, pud, addr, next, newprot,
@@ -311,7 +311,7 @@ static inline unsigned long change_p4d_range(struct vm_area_struct *vma,
p4d = p4d_offset(pgd, addr);
do {
- next = p4d_addr_end(addr, end);
+ next = p4d_addr_end(*p4d, addr, end);
if (p4d_none_or_clear_bad(p4d))
continue;
pages += change_pud_range(vma, p4d, addr, next, newprot,
@@ -336,7 +336,7 @@ static unsigned long change_protection_range(struct vm_area_struct *vma,
flush_cache_range(vma, addr, end);
inc_tlb_flush_pending(mm);
do {
- next = pgd_addr_end(addr, end);
+ next = pgd_addr_end(*pgd, addr, end);
if (pgd_none_or_clear_bad(pgd))
continue;
pages += change_p4d_range(vma, pgd, addr, next, newprot,
diff --git a/mm/pagewalk.c b/mm/pagewalk.c
index e81640d9f177..a5b9f61b5d45 100644
--- a/mm/pagewalk.c
+++ b/mm/pagewalk.c
@@ -70,7 +70,7 @@ static int walk_pmd_range(pud_t *pud, unsigned long addr, unsigned long end,
pmd = pmd_offset(pud, addr);
do {
again:
- next = pmd_addr_end(addr, end);
+ next = pmd_addr_end(*pmd, addr, end);
if (pmd_none(*pmd) || (!walk->vma && !walk->no_vma)) {
if (ops->pte_hole)
err = ops->pte_hole(addr, next, depth, walk);
@@ -128,7 +128,7 @@ static int walk_pud_range(p4d_t *p4d, unsigned long addr, unsigned long end,
pud = pud_offset(p4d, addr);
do {
again:
- next = pud_addr_end(addr, end);
+ next = pud_addr_end(*pud, addr, end);
if (pud_none(*pud) || (!walk->vma && !walk->no_vma)) {
if (ops->pte_hole)
err = ops->pte_hole(addr, next, depth, walk);
@@ -176,7 +176,7 @@ static int walk_p4d_range(pgd_t *pgd, unsigned long addr, unsigned long end,
p4d = p4d_offset(pgd, addr);
do {
- next = p4d_addr_end(addr, end);
+ next = p4d_addr_end(*p4d, addr, end);
if (p4d_none_or_clear_bad(p4d)) {
if (ops->pte_hole)
err = ops->pte_hole(addr, next, depth, walk);
@@ -211,7 +211,7 @@ static int walk_pgd_range(unsigned long addr, unsigned long end,
else
pgd = pgd_offset(walk->mm, addr);
do {
- next = pgd_addr_end(addr, end);
+ next = pgd_addr_end(*pgd, addr, end);
if (pgd_none_or_clear_bad(pgd)) {
if (ops->pte_hole)
err = ops->pte_hole(addr, next, 0, walk);
diff --git a/mm/swapfile.c b/mm/swapfile.c
index 20012c0c0252..b1dd815aee6b 100644
--- a/mm/swapfile.c
+++ b/mm/swapfile.c
@@ -2018,7 +2018,7 @@ static inline int unuse_pmd_range(struct vm_area_struct *vma, pud_t *pud,
pmd = pmd_offset(pud, addr);
do {
cond_resched();
- next = pmd_addr_end(addr, end);
+ next = pmd_addr_end(*pmd, addr, end);
if (pmd_none_or_trans_huge_or_clear_bad(pmd))
continue;
ret = unuse_pte_range(vma, pmd, addr, next, type,
@@ -2040,7 +2040,7 @@ static inline int unuse_pud_range(struct vm_area_struct *vma, p4d_t *p4d,
pud = pud_offset(p4d, addr);
do {
- next = pud_addr_end(addr, end);
+ next = pud_addr_end(*pud, addr, end);
if (pud_none_or_clear_bad(pud))
continue;
ret = unuse_pmd_range(vma, pud, addr, next, type,
@@ -2062,7 +2062,7 @@ static inline int unuse_p4d_range(struct vm_area_struct *vma, pgd_t *pgd,
p4d = p4d_offset(pgd, addr);
do {
- next = p4d_addr_end(addr, end);
+ next = p4d_addr_end(*p4d, addr, end);
if (p4d_none_or_clear_bad(p4d))
continue;
ret = unuse_pud_range(vma, p4d, addr, next, type,
@@ -2085,7 +2085,7 @@ static int unuse_vma(struct vm_area_struct *vma, unsigned int type,
pgd = pgd_offset(vma->vm_mm, addr);
do {
- next = pgd_addr_end(addr, end);
+ next = pgd_addr_end(*pgd, addr, end);
if (pgd_none_or_clear_bad(pgd))
continue;
ret = unuse_p4d_range(vma, pgd, addr, next, type,
diff --git a/mm/vmalloc.c b/mm/vmalloc.c
index be4724b916b3..09ff0d5ecbc1 100644
--- a/mm/vmalloc.c
+++ b/mm/vmalloc.c
@@ -93,7 +93,7 @@ static void vunmap_pmd_range(pud_t *pud, unsigned long addr, unsigned long end,
pmd = pmd_offset(pud, addr);
do {
- next = pmd_addr_end(addr, end);
+ next = pmd_addr_end(*pmd, addr, end);
cleared = pmd_clear_huge(pmd);
if (cleared || pmd_bad(*pmd))
@@ -118,7 +118,7 @@ static void vunmap_pud_range(p4d_t *p4d, unsigned long addr, unsigned long end,
pud = pud_offset(p4d, addr);
do {
- next = pud_addr_end(addr, end);
+ next = pud_addr_end(*pud, addr, end);
cleared = pud_clear_huge(pud);
if (cleared || pud_bad(*pud))
@@ -141,7 +141,7 @@ static void vunmap_p4d_range(pgd_t *pgd, unsigned long addr, unsigned long end,
p4d = p4d_offset(pgd, addr);
do {
- next = p4d_addr_end(addr, end);
+ next = p4d_addr_end(*p4d, addr, end);
cleared = p4d_clear_huge(p4d);
if (cleared || p4d_bad(*p4d))
@@ -179,7 +179,7 @@ void unmap_kernel_range_noflush(unsigned long start, unsigned long size)
BUG_ON(addr >= end);
pgd = pgd_offset_k(addr);
do {
- next = pgd_addr_end(addr, end);
+ next = pgd_addr_end(*pgd, addr, end);
if (pgd_bad(*pgd))
mask |= PGTBL_PGD_MODIFIED;
if (pgd_none_or_clear_bad(pgd))
@@ -230,7 +230,7 @@ static int vmap_pmd_range(pud_t *pud, unsigned long addr,
if (!pmd)
return -ENOMEM;
do {
- next = pmd_addr_end(addr, end);
+ next = pmd_addr_end(*pmd, addr, end);
if (vmap_pte_range(pmd, addr, next, prot, pages, nr, mask))
return -ENOMEM;
} while (pmd++, addr = next, addr != end);
@@ -248,7 +248,7 @@ static int vmap_pud_range(p4d_t *p4d, unsigned long addr,
if (!pud)
return -ENOMEM;
do {
- next = pud_addr_end(addr, end);
+ next = pud_addr_end(*pud, addr, end);
if (vmap_pmd_range(pud, addr, next, prot, pages, nr, mask))
return -ENOMEM;
} while (pud++, addr = next, addr != end);
@@ -266,7 +266,7 @@ static int vmap_p4d_range(pgd_t *pgd, unsigned long addr,
if (!p4d)
return -ENOMEM;
do {
- next = p4d_addr_end(addr, end);
+ next = p4d_addr_end(*p4d, addr, end);
if (vmap_pud_range(p4d, addr, next, prot, pages, nr, mask))
return -ENOMEM;
} while (p4d++, addr = next, addr != end);
@@ -305,7 +305,7 @@ int map_kernel_range_noflush(unsigned long addr, unsigned long size,
BUG_ON(addr >= end);
pgd = pgd_offset_k(addr);
do {
- next = pgd_addr_end(addr, end);
+ next = pgd_addr_end(*pgd, addr, end);
if (pgd_bad(*pgd))
mask |= PGTBL_PGD_MODIFIED;
err = vmap_p4d_range(pgd, addr, next, prot, pages, &nr, &mask);
--
2.17.1
^ permalink raw reply related
* [RFC PATCH v2 1/3] mm/gup: fix gup_fast with dynamic page table folding
From: Gerald Schaefer @ 2020-09-07 18:00 UTC (permalink / raw)
To: Jason Gunthorpe, John Hubbard
Cc: Peter Zijlstra, Dave Hansen, linux-mm, Paul Mackerras,
linux-sparc, Alexander Gordeev, Claudio Imbrenda, Will Deacon,
linux-arch, linux-s390, Vasily Gorbik, Richard Weinberger,
linux-x86, Russell King, Christian Borntraeger, Ingo Molnar,
Catalin Marinas, Andrey Ryabinin, Heiko Carstens, Arnd Bergmann,
Jeff Dike, linux-um, Borislav Petkov, Andy Lutomirski,
Thomas Gleixner, linux-arm, linux-power, LKML, Andrew Morton,
Linus Torvalds, Mike Rapoport
In-Reply-To: <20200907180058.64880-1-gerald.schaefer@linux.ibm.com>
From: Alexander Gordeev <agordeev@linux.ibm.com>
Commit 1a42010cdc26 ("s390/mm: convert to the generic get_user_pages_fast
code") introduced a subtle but severe bug on s390 with gup_fast, due to
dynamic page table folding.
The question "What would it require for the generic code to work for s390"
has already been discussed here
https://lkml.kernel.org/r/20190418100218.0a4afd51@mschwideX1
and ended with a promising approach here
https://lkml.kernel.org/r/20190419153307.4f2911b5@mschwideX1
which in the end unfortunately didn't quite work completely.
We tried to mimic static level folding by changing pgd_offset to always
calculate top level page table offset, and do nothing in folded pXd_offset.
What has been overlooked is that PxD_SIZE/MASK and thus pXd_addr_end do
not reflect this dynamic behaviour, and still act like static 5-level
page tables.
Here is an example of what happens with gup_fast on s390, for a task with
3-levels paging, crossing a 2 GB pud boundary:
// addr = 0x1007ffff000, end = 0x10080001000
static int gup_pud_range(p4d_t p4d, unsigned long addr, unsigned long end,
unsigned int flags, struct page **pages, int *nr)
{
unsigned long next;
pud_t *pudp;
// pud_offset returns &p4d itself (a pointer to a value on stack)
pudp = pud_offset(&p4d, addr);
do {
// on second iteratation reading "random" stack value
pud_t pud = READ_ONCE(*pudp);
// next = 0x10080000000, due to PUD_SIZE/MASK != PGDIR_SIZE/MASK on s390
next = pud_addr_end(addr, end);
...
} while (pudp++, addr = next, addr != end); // pudp++ iterating over stack
return 1;
}
pud_addr_end = 0x10080000000 is correct, but the previous pgd/p4d_addr_end
should also have returned that limit, instead of the 5-level static
pgd/p4d limits with PUD_SIZE/MASK != PGDIR_SIZE/MASK. Then the "end"
parameter for gup_pud_range would also have been 0x10080000000, and we
would not iterate further in gup_pud_range, but rather go back and
(correctly) do it in gup_pgd_range.
So, for the second iteration in gup_pud_range, we will increase pudp,
which pointed to a stack value and not the real pud table. This new pudp
will then point to whatever lies behind the p4d stack value. In general,
this happens to be the previously read pgd, but it probably could also
be something different, depending on compiler decisions.
Most unfortunately, if it happens to be the pgd value, which is the
same as the p4d / pud due to folding, it is a valid and present entry.
So after the increment, we would still point to the same pud entry.
The addr however has been increased in the second iteration, so that we
now have different pmd/pte_index values, which will result in very wrong
behaviour for the remaining gup_pmd/pte_range calls. We will effectively
operate on an address minus 2 GB, due to missing pudp increase.
In the "good case", if nothing is mapped there, we will fall back to
the slow gup path. But if something is mapped there, and valid
for gup_fast, we will end up (silently) getting references on the wrong
pages and also add the wrong pages to the **pages result array. This
can cause data corruption.
Fix this by introducing new pXd_addr_end_folded helpers, which take an
additional pXd entry value parameter, that can be used on s390
to determine the correct page table level and return corresponding
end / boundary. With that, the pointer iteration will always
happen in gup_pgd_range for s390. No change for other architectures
introduced.
Fixes: 1a42010cdc26 ("s390/mm: convert to the generic get_user_pages_fast code")
Cc: <stable@vger.kernel.org> # 5.2+
Reviewed-by: Gerald Schaefer <gerald.schaefer@linux.ibm.com>
Signed-off-by: Alexander Gordeev <agordeev@linux.ibm.com>
Signed-off-by: Gerald Schaefer <gerald.schaefer@linux.ibm.com>
---
arch/s390/include/asm/pgtable.h | 42 +++++++++++++++++++++++++++++++++
include/linux/pgtable.h | 16 +++++++++++++
mm/gup.c | 8 +++----
3 files changed, 62 insertions(+), 4 deletions(-)
diff --git a/arch/s390/include/asm/pgtable.h b/arch/s390/include/asm/pgtable.h
index 7eb01a5459cd..027206e4959d 100644
--- a/arch/s390/include/asm/pgtable.h
+++ b/arch/s390/include/asm/pgtable.h
@@ -512,6 +512,48 @@ static inline bool mm_pmd_folded(struct mm_struct *mm)
}
#define mm_pmd_folded(mm) mm_pmd_folded(mm)
+/*
+ * With dynamic page table levels on s390, the static pXd_addr_end() functions
+ * will not return corresponding dynamic boundaries. This is no problem as long
+ * as only pXd pointers are passed down during page table walk, because
+ * pXd_offset() will simply return the given pointer for folded levels, and the
+ * pointer iteration over a range simply happens at the correct page table
+ * level.
+ * It is however a problem with gup_fast, or other places walking the page
+ * tables w/o locks using READ_ONCE(), and passing down the pXd values instead
+ * of pointers. In this case, the pointer given to pXd_offset() is a pointer to
+ * a stack variable, which cannot be used for pointer iteration at the correct
+ * level. Instead, the iteration then has to happen by going up to pgd level
+ * again. To allow this, provide pXd_addr_end_folded() functions with an
+ * additional pXd value parameter, which can be used on s390 to determine the
+ * folding level and return the corresponding boundary.
+ */
+static inline unsigned long rste_addr_end_folded(unsigned long rste, unsigned long addr, unsigned long end)
+{
+ unsigned long type = (rste & _REGION_ENTRY_TYPE_MASK) >> 2;
+ unsigned long size = 1UL << (_SEGMENT_SHIFT + type * 11);
+ unsigned long boundary = (addr + size) & ~(size - 1);
+
+ /*
+ * FIXME The below check is for internal testing only, to be removed
+ */
+ VM_BUG_ON(type < (_REGION_ENTRY_TYPE_R3 >> 2));
+
+ return (boundary - 1) < (end - 1) ? boundary : end;
+}
+
+#define pgd_addr_end_folded pgd_addr_end_folded
+static inline unsigned long pgd_addr_end_folded(pgd_t pgd, unsigned long addr, unsigned long end)
+{
+ return rste_addr_end_folded(pgd_val(pgd), addr, end);
+}
+
+#define p4d_addr_end_folded p4d_addr_end_folded
+static inline unsigned long p4d_addr_end_folded(p4d_t p4d, unsigned long addr, unsigned long end)
+{
+ return rste_addr_end_folded(p4d_val(p4d), addr, end);
+}
+
static inline int mm_has_pgste(struct mm_struct *mm)
{
#ifdef CONFIG_PGSTE
diff --git a/include/linux/pgtable.h b/include/linux/pgtable.h
index e8cbc2e795d5..981c4c2a31fe 100644
--- a/include/linux/pgtable.h
+++ b/include/linux/pgtable.h
@@ -681,6 +681,22 @@ static inline int arch_unmap_one(struct mm_struct *mm,
})
#endif
+#ifndef pgd_addr_end_folded
+#define pgd_addr_end_folded(pgd, addr, end) pgd_addr_end(addr, end)
+#endif
+
+#ifndef p4d_addr_end_folded
+#define p4d_addr_end_folded(p4d, addr, end) p4d_addr_end(addr, end)
+#endif
+
+#ifndef pud_addr_end_folded
+#define pud_addr_end_folded(pud, addr, end) pud_addr_end(addr, end)
+#endif
+
+#ifndef pmd_addr_end_folded
+#define pmd_addr_end_folded(pmd, addr, end) pmd_addr_end(addr, end)
+#endif
+
/*
* When walking page tables, we usually want to skip any p?d_none entries;
* and any p?d_bad entries - reporting the error before resetting to none.
diff --git a/mm/gup.c b/mm/gup.c
index bd883a112724..ba4aace5d0f4 100644
--- a/mm/gup.c
+++ b/mm/gup.c
@@ -2521,7 +2521,7 @@ static int gup_pmd_range(pud_t pud, unsigned long addr, unsigned long end,
do {
pmd_t pmd = READ_ONCE(*pmdp);
- next = pmd_addr_end(addr, end);
+ next = pmd_addr_end_folded(pmd, addr, end);
if (!pmd_present(pmd))
return 0;
@@ -2564,7 +2564,7 @@ static int gup_pud_range(p4d_t p4d, unsigned long addr, unsigned long end,
do {
pud_t pud = READ_ONCE(*pudp);
- next = pud_addr_end(addr, end);
+ next = pud_addr_end_folded(pud, addr, end);
if (unlikely(!pud_present(pud)))
return 0;
if (unlikely(pud_huge(pud))) {
@@ -2592,7 +2592,7 @@ static int gup_p4d_range(pgd_t pgd, unsigned long addr, unsigned long end,
do {
p4d_t p4d = READ_ONCE(*p4dp);
- next = p4d_addr_end(addr, end);
+ next = p4d_addr_end_folded(p4d, addr, end);
if (p4d_none(p4d))
return 0;
BUILD_BUG_ON(p4d_huge(p4d));
@@ -2617,7 +2617,7 @@ static void gup_pgd_range(unsigned long addr, unsigned long end,
do {
pgd_t pgd = READ_ONCE(*pgdp);
- next = pgd_addr_end(addr, end);
+ next = pgd_addr_end_folded(pgd, addr, end);
if (pgd_none(pgd))
return;
if (unlikely(pgd_huge(pgd))) {
--
2.17.1
^ permalink raw reply related
* [RFC PATCH v2 3/3] mm: make generic pXd_addr_end() macros inline functions
From: Gerald Schaefer @ 2020-09-07 18:00 UTC (permalink / raw)
To: Jason Gunthorpe, John Hubbard
Cc: Peter Zijlstra, Dave Hansen, linux-mm, Paul Mackerras,
linux-sparc, Alexander Gordeev, Claudio Imbrenda, Will Deacon,
linux-arch, linux-s390, Vasily Gorbik, Richard Weinberger,
linux-x86, Russell King, Christian Borntraeger, Ingo Molnar,
Catalin Marinas, Andrey Ryabinin, Heiko Carstens, Arnd Bergmann,
Jeff Dike, linux-um, Borislav Petkov, Andy Lutomirski,
Thomas Gleixner, linux-arm, linux-power, LKML, Andrew Morton,
Linus Torvalds, Mike Rapoport
In-Reply-To: <20200907180058.64880-1-gerald.schaefer@linux.ibm.com>
From: Alexander Gordeev <agordeev@linux.ibm.com>
Since pXd_addr_end() macros take pXd page-table entry as a
parameter it makes sense to check the entry type on compile.
Even though most archs do not make use of page-table entries
in pXd_addr_end() calls, checking the type in traversal code
paths could help to avoid subtle bugs.
Signed-off-by: Alexander Gordeev <agordeev@linux.ibm.com>
Signed-off-by: Gerald Schaefer <gerald.schaefer@linux.ibm.com>
---
include/linux/pgtable.h | 36 ++++++++++++++++++++----------------
1 file changed, 20 insertions(+), 16 deletions(-)
diff --git a/include/linux/pgtable.h b/include/linux/pgtable.h
index 67ebc22cf83d..d9e7d16c2263 100644
--- a/include/linux/pgtable.h
+++ b/include/linux/pgtable.h
@@ -656,31 +656,35 @@ static inline int arch_unmap_one(struct mm_struct *mm,
*/
#ifndef pgd_addr_end
-#define pgd_addr_end(pgd, addr, end) \
-({ unsigned long __boundary = ((addr) + PGDIR_SIZE) & PGDIR_MASK; \
- (__boundary - 1 < (end) - 1)? __boundary: (end); \
-})
+#define pgd_addr_end pgd_addr_end
+static inline unsigned long pgd_addr_end(pgd_t pgd, unsigned long addr, unsigned long end)
+{ unsigned long __boundary = (addr + PGDIR_SIZE) & PGDIR_MASK;
+ return (__boundary - 1 < end - 1) ? __boundary : end;
+}
#endif
#ifndef p4d_addr_end
-#define p4d_addr_end(p4d, addr, end) \
-({ unsigned long __boundary = ((addr) + P4D_SIZE) & P4D_MASK; \
- (__boundary - 1 < (end) - 1)? __boundary: (end); \
-})
+#define p4d_addr_end p4d_addr_end
+static inline unsigned long p4d_addr_end(p4d_t p4d, unsigned long addr, unsigned long end)
+{ unsigned long __boundary = (addr + P4D_SIZE) & P4D_MASK;
+ return (__boundary - 1 < end - 1) ? __boundary : end;
+}
#endif
#ifndef pud_addr_end
-#define pud_addr_end(pud, addr, end) \
-({ unsigned long __boundary = ((addr) + PUD_SIZE) & PUD_MASK; \
- (__boundary - 1 < (end) - 1)? __boundary: (end); \
-})
+#define pud_addr_end pud_addr_end
+static inline unsigned long pud_addr_end(pud_t pud, unsigned long addr, unsigned long end)
+{ unsigned long __boundary = (addr + PUD_SIZE) & PUD_MASK;
+ return (__boundary - 1 < end - 1) ? __boundary : end;
+}
#endif
#ifndef pmd_addr_end
-#define pmd_addr_end(pmd, addr, end) \
-({ unsigned long __boundary = ((addr) + PMD_SIZE) & PMD_MASK; \
- (__boundary - 1 < (end) - 1)? __boundary: (end); \
-})
+#define pmd_addr_end pmd_addr_end
+static inline unsigned long pmd_addr_end(pmd_t pmd, unsigned long addr, unsigned long end)
+{ unsigned long __boundary = (addr + PMD_SIZE) & PMD_MASK;
+ return (__boundary - 1 < end - 1) ? __boundary : end;
+}
#endif
/*
--
2.17.1
^ permalink raw reply related
* [PATCH AUTOSEL 5.4 11/43] ibmvnic fix NULL tx_pools and rx_tools issue at do_reset
From: Sasha Levin @ 2020-09-07 16:32 UTC (permalink / raw)
To: linux-kernel, stable
Cc: Sasha Levin, netdev, Mingming Cao, Dany Madden, linuxppc-dev,
David S . Miller
In-Reply-To: <20200907163329.1280888-1-sashal@kernel.org>
From: Mingming Cao <mmc@linux.vnet.ibm.com>
[ Upstream commit 9f13457377907fa253aef560e1a37e1ca4197f9b ]
At the time of do_rest, ibmvnic tries to re-initalize the tx_pools
and rx_pools to avoid re-allocating the long term buffer. However
there is a window inside do_reset that the tx_pools and
rx_pools were freed before re-initialized making it possible to deference
null pointers.
This patch fix this issue by always check the tx_pool
and rx_pool are not NULL after ibmvnic_login. If so, re-allocating
the pools. This will avoid getting into calling reset_tx/rx_pools with
NULL adapter tx_pools/rx_pools pointer. Also add null pointer check in
reset_tx_pools and reset_rx_pools to safe handle NULL pointer case.
Signed-off-by: Mingming Cao <mmc@linux.vnet.ibm.com>
Signed-off-by: Dany Madden <drt@linux.ibm.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
Signed-off-by: Sasha Levin <sashal@kernel.org>
---
drivers/net/ethernet/ibm/ibmvnic.c | 15 ++++++++++++++-
1 file changed, 14 insertions(+), 1 deletion(-)
diff --git a/drivers/net/ethernet/ibm/ibmvnic.c b/drivers/net/ethernet/ibm/ibmvnic.c
index 2d20a48f0ba0a..de45b3709c14e 100644
--- a/drivers/net/ethernet/ibm/ibmvnic.c
+++ b/drivers/net/ethernet/ibm/ibmvnic.c
@@ -416,6 +416,9 @@ static int reset_rx_pools(struct ibmvnic_adapter *adapter)
int i, j, rc;
u64 *size_array;
+ if (!adapter->rx_pool)
+ return -1;
+
size_array = (u64 *)((u8 *)(adapter->login_rsp_buf) +
be32_to_cpu(adapter->login_rsp_buf->off_rxadd_buff_size));
@@ -586,6 +589,9 @@ static int reset_tx_pools(struct ibmvnic_adapter *adapter)
int tx_scrqs;
int i, rc;
+ if (!adapter->tx_pool)
+ return -1;
+
tx_scrqs = be32_to_cpu(adapter->login_rsp_buf->num_txsubm_subcrqs);
for (i = 0; i < tx_scrqs; i++) {
rc = reset_one_tx_pool(adapter, &adapter->tso_pool[i]);
@@ -1918,7 +1924,10 @@ static int do_reset(struct ibmvnic_adapter *adapter,
adapter->req_rx_add_entries_per_subcrq !=
old_num_rx_slots ||
adapter->req_tx_entries_per_subcrq !=
- old_num_tx_slots) {
+ old_num_tx_slots ||
+ !adapter->rx_pool ||
+ !adapter->tso_pool ||
+ !adapter->tx_pool) {
release_rx_pools(adapter);
release_tx_pools(adapter);
release_napi(adapter);
@@ -1931,10 +1940,14 @@ static int do_reset(struct ibmvnic_adapter *adapter,
} else {
rc = reset_tx_pools(adapter);
if (rc)
+ netdev_dbg(adapter->netdev, "reset tx pools failed (%d)\n",
+ rc);
goto out;
rc = reset_rx_pools(adapter);
if (rc)
+ netdev_dbg(adapter->netdev, "reset rx pools failed (%d)\n",
+ rc);
goto out;
}
ibmvnic_disable_irqs(adapter);
--
2.25.1
^ permalink raw reply related
page: next (older) | prev (newer) | latest
- recent:[subjects (threaded)|topics (new)|topics (active)]
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox