* Re: [PATCH v2 1/2] treewide: remove unused address argument from pte_alloc functions
From: Julia Lawall @ 2018-10-12 16:38 UTC (permalink / raw)
To: Joel Fernandes
Cc: linux-mips, linux-m68k, Rich Felker, linux-ia64, linux-sh,
Peter Zijlstra, Catalin Marinas, Dave Hansen, Will Deacon,
Michal Hocko, linux-mm, lokeshgidra, sparclinux, linux-riscv,
elfring, Jonas Bonn, linux-s390, dancol, Yoshinori Sato,
Max Filippov, linux-hexagon, Helge Deller,
maintainer:X86 ARCHITECTURE 32-BIT AND 64-BIT, hughd,
James E.J. Bottomley, kasan-dev, Anton Ivanov, Ingo Molnar,
kvmarm, Geert Uytterhoeven, linux-snps-arc, kernel-team,
Sam Creasey, linux-xtensa, Jeff Dike, linux-alpha, linux-um,
Stefan Kristiansson, Julia Lawall, openrisc, Borislav Petkov,
Andy Lutomirski, Andrey Ryabinin, kirill, Stafford Horne,
Guan Xuetao, linux-arm-kernel, Chris Zankel, Tony Luck,
linux-parisc, pantin, linux-kernel, Fenghua Yu, minchan,
Thomas Gleixner, Richard Weinberger, nios2-dev, akpm,
linuxppc-dev, David S. Miller, Ley Foon Tan
In-Reply-To: <20181012163433.GA223066@joelaf.mtv.corp.google.com>
> I wrote something like this as below but it failed to compile, Julia any
> suggestions on how to express this?
>
> @pte_alloc_func_proto depends on patch exists@
> type T1, T2, T3, T4;
> identifier fn =~
> "^(__pte_alloc|pte_alloc_one|pte_alloc|__pte_alloc_kernel|pte_alloc_one_kernel)$";
> @@
>
> (
> - T3 fn(T1, T2);
> + T3 fn(T1);
> |
> - T3 fn(T1, T2, T4);
> + T3 fn(T1, T2);
> )
What goes wrong? It seems fine to me.
julia
^ permalink raw reply
* Re: [PATCH v2 2/2] mm: speed up mremap by 500x on large regions
From: Kirill A. Shutemov @ 2018-10-12 13:19 UTC (permalink / raw)
To: Joel Fernandes
Cc: linux-mips, Rich Felker, linux-ia64, linux-sh, Peter Zijlstra,
Catalin Marinas, Dave Hansen, Will Deacon, mhocko, linux-mm,
lokeshgidra, sparclinux, linux-riscv, elfring, Jonas Bonn,
linux-s390, dancol, Yoshinori Sato, linux-xtensa, linux-hexagon,
Helge Deller, maintainer:X86 ARCHITECTURE (32-BIT AND 64-BIT),
hughd, James E.J. Bottomley, kasan-dev, kvmarm, Ingo Molnar,
Geert Uytterhoeven, Andrey Ryabinin, linux-snps-arc, kernel-team,
Sam Creasey, Fenghua Yu, Jeff Dike, linux-um, Stefan Kristiansson,
Julia Lawall, linux-m68k, openrisc, Borislav Petkov,
Andy Lutomirski, nios2-dev, Stafford Horne, Guan Xuetao,
linux-arm-kernel, Chris Zankel, Tony Luck, Richard Weinberger,
linux-parisc, pantin, Max Filippov, linux-kernel, minchan,
Thomas Gleixner, linux-alpha, Ley Foon Tan, akpm, linuxppc-dev,
David S. Miller
In-Reply-To: <20181012125046.GA170912@joelaf.mtv.corp.google.com>
On Fri, Oct 12, 2018 at 05:50:46AM -0700, Joel Fernandes wrote:
> On Fri, Oct 12, 2018 at 02:30:56PM +0300, Kirill A. Shutemov wrote:
> > On Thu, Oct 11, 2018 at 06:37:56PM -0700, Joel Fernandes (Google) wrote:
> > > Android needs to mremap large regions of memory during memory management
> > > related operations. The mremap system call can be really slow if THP is
> > > not enabled. The bottleneck is move_page_tables, which is copying each
> > > pte at a time, and can be really slow across a large map. Turning on THP
> > > may not be a viable option, and is not for us. This patch speeds up the
> > > performance for non-THP system by copying at the PMD level when possible.
> > >
> > > The speed up is three orders of magnitude. On a 1GB mremap, the mremap
> > > completion times drops from 160-250 millesconds to 380-400 microseconds.
> > >
> > > Before:
> > > Total mremap time for 1GB data: 242321014 nanoseconds.
> > > Total mremap time for 1GB data: 196842467 nanoseconds.
> > > Total mremap time for 1GB data: 167051162 nanoseconds.
> > >
> > > After:
> > > Total mremap time for 1GB data: 385781 nanoseconds.
> > > Total mremap time for 1GB data: 388959 nanoseconds.
> > > Total mremap time for 1GB data: 402813 nanoseconds.
> > >
> > > Incase THP is enabled, the optimization is skipped. I also flush the
> > > tlb every time we do this optimization since I couldn't find a way to
> > > determine if the low-level PTEs are dirty. It is seen that the cost of
> > > doing so is not much compared the improvement, on both x86-64 and arm64.
> >
> > I looked into the code more and noticed move_pte() helper called from
> > move_ptes(). It changes PTE entry to suite new address.
> >
> > It is only defined in non-trivial way on Sparc. I don't know much about
> > Sparc and it's hard for me to say if the optimization will break anything
> > there.
>
> Sparc's move_pte seems to be flushing the D-cache to prevent aliasing. It is
> not modifying the PTE itself AFAICS:
>
> #ifdef DCACHE_ALIASING_POSSIBLE
> #define __HAVE_ARCH_MOVE_PTE
> #define move_pte(pte, prot, old_addr, new_addr) \
> ({ \
> pte_t newpte = (pte); \
> if (tlb_type != hypervisor && pte_present(pte)) { \
> unsigned long this_pfn = pte_pfn(pte); \
> \
> if (pfn_valid(this_pfn) && \
> (((old_addr) ^ (new_addr)) & (1 << 13))) \
> flush_dcache_page_all(current->mm, \
> pfn_to_page(this_pfn)); \
> } \
> newpte; \
> })
> #endif
>
> If its an issue, then how do transparent huge pages work on Sparc? I don't
> see the huge page code (move_huge_pages) during mremap doing anything special
> for Sparc architecture when moving PMDs..
My *guess* is that it will work fine on Sparc as it apprarently it only
cares about change in bit 13 of virtual address. It will never happen for
huge pages or when PTE page tables move.
But I just realized that the problem is bigger: since we pass new_addr to
the set_pte_at() we would need to audit all implementations that they are
safe with just moving PTE page table.
I would rather go with per-architecture enabling. It's much safer.
> Also, do we not flush the caches from any path when we munmap address space?
> We do call do_munmap on the old mapping from mremap after moving to the new one.
Are you sure about that? It can be hided deeper in architecture-specific
code.
--
Kirill A. Shutemov
^ permalink raw reply
* Re: [PATCH v2 2/2] mm: speed up mremap by 500x on large regions
From: Kirill A. Shutemov @ 2018-10-12 14:37 UTC (permalink / raw)
To: Anton Ivanov
Cc: linux-mips, Rich Felker, linux-ia64, linux-sh, Peter Zijlstra,
Catalin Marinas, Dave Hansen, Will Deacon, mhocko, linux-mm,
lokeshgidra, Joel Fernandes (Google), linux-riscv, elfring,
Jonas Bonn, linux-s390, dancol, Yoshinori Sato, Max Filippov,
linux-hexagon, Helge Deller,
maintainer:X86 ARCHITECTURE 32-BIT AND 64-BIT, hughd,
James E.J. Bottomley, kasan-dev, kvmarm, Ingo Molnar,
Geert Uytterhoeven, Andrey Ryabinin, linux-snps-arc, kernel-team,
Sam Creasey, linux-xtensa, Jeff Dike, linux-alpha, linux-um,
Stefan Kristiansson, Julia Lawall, linux-m68k, openrisc,
Borislav Petkov, Andy Lutomirski, nios2-dev, Stafford Horne,
Guan Xuetao, linux-arm-kernel, Chris Zankel, Tony Luck,
linux-parisc, pantin, linux-kernel, Fenghua Yu, minchan,
Thomas Gleixner, Richard Weinberger, sparclinux, akpm,
linuxppc-dev, David S. Miller, Ley Foon Tan
In-Reply-To: <9ed82f9e-88c4-8e4f-8c45-3ef153469603@kot-begemot.co.uk>
On Fri, Oct 12, 2018 at 03:09:49PM +0100, Anton Ivanov wrote:
> On 10/12/18 2:37 AM, Joel Fernandes (Google) wrote:
> > Android needs to mremap large regions of memory during memory management
> > related operations. The mremap system call can be really slow if THP is
> > not enabled. The bottleneck is move_page_tables, which is copying each
> > pte at a time, and can be really slow across a large map. Turning on THP
> > may not be a viable option, and is not for us. This patch speeds up the
> > performance for non-THP system by copying at the PMD level when possible.
> >
> > The speed up is three orders of magnitude. On a 1GB mremap, the mremap
> > completion times drops from 160-250 millesconds to 380-400 microseconds.
> >
> > Before:
> > Total mremap time for 1GB data: 242321014 nanoseconds.
> > Total mremap time for 1GB data: 196842467 nanoseconds.
> > Total mremap time for 1GB data: 167051162 nanoseconds.
> >
> > After:
> > Total mremap time for 1GB data: 385781 nanoseconds.
> > Total mremap time for 1GB data: 388959 nanoseconds.
> > Total mremap time for 1GB data: 402813 nanoseconds.
> >
> > Incase THP is enabled, the optimization is skipped. I also flush the
> > tlb every time we do this optimization since I couldn't find a way to
> > determine if the low-level PTEs are dirty. It is seen that the cost of
> > doing so is not much compared the improvement, on both x86-64 and arm64.
> >
> > Cc: minchan@kernel.org
> > Cc: pantin@google.com
> > Cc: hughd@google.com
> > Cc: lokeshgidra@google.com
> > Cc: dancol@google.com
> > Cc: mhocko@kernel.org
> > Cc: kirill@shutemov.name
> > Cc: akpm@linux-foundation.org
> > Signed-off-by: Joel Fernandes (Google) <joel@joelfernandes.org>
> > ---
> > mm/mremap.c | 62 +++++++++++++++++++++++++++++++++++++++++++++++++++++
> > 1 file changed, 62 insertions(+)
> >
> > diff --git a/mm/mremap.c b/mm/mremap.c
> > index 9e68a02a52b1..d82c485822ef 100644
> > --- a/mm/mremap.c
> > +++ b/mm/mremap.c
> > @@ -191,6 +191,54 @@ static void move_ptes(struct vm_area_struct *vma, pmd_t *old_pmd,
> > drop_rmap_locks(vma);
> > }
> > +static bool move_normal_pmd(struct vm_area_struct *vma, unsigned long old_addr,
> > + unsigned long new_addr, unsigned long old_end,
> > + pmd_t *old_pmd, pmd_t *new_pmd, bool *need_flush)
> > +{
> > + spinlock_t *old_ptl, *new_ptl;
> > + struct mm_struct *mm = vma->vm_mm;
> > +
> > + if ((old_addr & ~PMD_MASK) || (new_addr & ~PMD_MASK)
> > + || old_end - old_addr < PMD_SIZE)
> > + return false;
> > +
> > + /*
> > + * The destination pmd shouldn't be established, free_pgtables()
> > + * should have release it.
> > + */
> > + if (WARN_ON(!pmd_none(*new_pmd)))
> > + return false;
> > +
> > + /*
> > + * We don't have to worry about the ordering of src and dst
> > + * ptlocks because exclusive mmap_sem prevents deadlock.
> > + */
> > + old_ptl = pmd_lock(vma->vm_mm, old_pmd);
> > + if (old_ptl) {
> > + pmd_t pmd;
> > +
> > + new_ptl = pmd_lockptr(mm, new_pmd);
> > + if (new_ptl != old_ptl)
> > + spin_lock_nested(new_ptl, SINGLE_DEPTH_NESTING);
> > +
> > + /* Clear the pmd */
> > + pmd = *old_pmd;
> > + pmd_clear(old_pmd);
> > +
> > + VM_BUG_ON(!pmd_none(*new_pmd));
> > +
> > + /* Set the new pmd */
> > + set_pmd_at(mm, new_addr, new_pmd, pmd);
>
> UML does not have set_pmd_at at all
Every architecture does. :)
But it may come not from the arch code.
> If I read the code right, MIPS completely ignores the address argument so
> set_pmd_at there may not have the effect which this patch is trying to
> achieve.
Ignoring address is fine. Most architectures do that..
The ideas is to move page table to the new pmd slot. It's nothing to do
with the address passed to set_pmd_at().
--
Kirill A. Shutemov
^ permalink raw reply
* Re: [PATCH v2 2/2] mm: speed up mremap by 500x on large regions
From: Joel Fernandes @ 2018-10-12 16:50 UTC (permalink / raw)
To: Anton Ivanov
Cc: linux-mips, Rich Felker, linux-ia64, linux-sh, Peter Zijlstra,
Catalin Marinas, Dave Hansen, Will Deacon, mhocko, linux-mm,
lokeshgidra, sparclinux, linux-riscv, elfring, Jonas Bonn,
linux-s390, dancol, Yoshinori Sato, Max Filippov, linux-hexagon,
Helge Deller, maintainer:X86 ARCHITECTURE 32-BIT AND 64-BIT,
hughd, James E.J. Bottomley, kasan-dev, kvmarm, Ingo Molnar,
Geert Uytterhoeven, Andrey Ryabinin, linux-snps-arc, kernel-team,
Sam Creasey, linux-xtensa, Jeff Dike, linux-alpha, linux-um,
Stefan Kristiansson, Julia Lawall, linux-m68k, openrisc,
Borislav Petkov, Andy Lutomirski, Ley Foon Tan,
Kirill A. Shutemov, Stafford Horne, Guan Xuetao, linux-arm-kernel,
Chris Zankel, Tony Luck, linux-parisc, pantin, linux-kernel,
Fenghua Yu, minchan, Thomas Gleixner, Richard Weinberger,
nios2-dev, akpm, linuxppc-dev, David S. Miller
In-Reply-To: <97cb3fe1-7bc1-12ff-d602-56c72a5496c5@kot-begemot.co.uk>
On Fri, Oct 12, 2018 at 05:42:24PM +0100, Anton Ivanov wrote:
>
> On 10/12/18 3:48 PM, Anton Ivanov wrote:
> > On 12/10/2018 15:37, Kirill A. Shutemov wrote:
> > > On Fri, Oct 12, 2018 at 03:09:49PM +0100, Anton Ivanov wrote:
> > > > On 10/12/18 2:37 AM, Joel Fernandes (Google) wrote:
> > > > > Android needs to mremap large regions of memory during
> > > > > memory management
> > > > > related operations. The mremap system call can be really
> > > > > slow if THP is
> > > > > not enabled. The bottleneck is move_page_tables, which is copying each
> > > > > pte at a time, and can be really slow across a large map.
> > > > > Turning on THP
> > > > > may not be a viable option, and is not for us. This patch
> > > > > speeds up the
> > > > > performance for non-THP system by copying at the PMD level
> > > > > when possible.
> > > > >
> > > > > The speed up is three orders of magnitude. On a 1GB mremap, the mremap
> > > > > completion times drops from 160-250 millesconds to 380-400
> > > > > microseconds.
> > > > >
> > > > > Before:
> > > > > Total mremap time for 1GB data: 242321014 nanoseconds.
> > > > > Total mremap time for 1GB data: 196842467 nanoseconds.
> > > > > Total mremap time for 1GB data: 167051162 nanoseconds.
> > > > >
> > > > > After:
> > > > > Total mremap time for 1GB data: 385781 nanoseconds.
> > > > > Total mremap time for 1GB data: 388959 nanoseconds.
> > > > > Total mremap time for 1GB data: 402813 nanoseconds.
> > > > >
> > > > > Incase THP is enabled, the optimization is skipped. I also flush the
> > > > > tlb every time we do this optimization since I couldn't find a way to
> > > > > determine if the low-level PTEs are dirty. It is seen that the cost of
> > > > > doing so is not much compared the improvement, on both
> > > > > x86-64 and arm64.
> > > > >
> > > > > Cc: minchan@kernel.org
> > > > > Cc: pantin@google.com
> > > > > Cc: hughd@google.com
> > > > > Cc: lokeshgidra@google.com
> > > > > Cc: dancol@google.com
> > > > > Cc: mhocko@kernel.org
> > > > > Cc: kirill@shutemov.name
> > > > > Cc: akpm@linux-foundation.org
> > > > > Signed-off-by: Joel Fernandes (Google) <joel@joelfernandes.org>
> > > > > ---
> > > > > mm/mremap.c | 62
> > > > > +++++++++++++++++++++++++++++++++++++++++++++++++++++
> > > > > 1 file changed, 62 insertions(+)
> > > > >
> > > > > diff --git a/mm/mremap.c b/mm/mremap.c
> > > > > index 9e68a02a52b1..d82c485822ef 100644
> > > > > --- a/mm/mremap.c
> > > > > +++ b/mm/mremap.c
> > > > > @@ -191,6 +191,54 @@ static void move_ptes(struct
> > > > > vm_area_struct *vma, pmd_t *old_pmd,
> > > > > drop_rmap_locks(vma);
> > > > > }
> > > > > +static bool move_normal_pmd(struct vm_area_struct *vma,
> > > > > unsigned long old_addr,
> > > > > + unsigned long new_addr, unsigned long old_end,
> > > > > + pmd_t *old_pmd, pmd_t *new_pmd, bool *need_flush)
> > > > > +{
> > > > > + spinlock_t *old_ptl, *new_ptl;
> > > > > + struct mm_struct *mm = vma->vm_mm;
> > > > > +
> > > > > + if ((old_addr & ~PMD_MASK) || (new_addr & ~PMD_MASK)
> > > > > + || old_end - old_addr < PMD_SIZE)
> > > > > + return false;
> > > > > +
> > > > > + /*
> > > > > + * The destination pmd shouldn't be established, free_pgtables()
> > > > > + * should have release it.
> > > > > + */
> > > > > + if (WARN_ON(!pmd_none(*new_pmd)))
> > > > > + return false;
> > > > > +
> > > > > + /*
> > > > > + * We don't have to worry about the ordering of src and dst
> > > > > + * ptlocks because exclusive mmap_sem prevents deadlock.
> > > > > + */
> > > > > + old_ptl = pmd_lock(vma->vm_mm, old_pmd);
> > > > > + if (old_ptl) {
> > > > > + pmd_t pmd;
> > > > > +
> > > > > + new_ptl = pmd_lockptr(mm, new_pmd);
> > > > > + if (new_ptl != old_ptl)
> > > > > + spin_lock_nested(new_ptl, SINGLE_DEPTH_NESTING);
> > > > > +
> > > > > + /* Clear the pmd */
> > > > > + pmd = *old_pmd;
> > > > > + pmd_clear(old_pmd);
> > > > > +
> > > > > + VM_BUG_ON(!pmd_none(*new_pmd));
> > > > > +
> > > > > + /* Set the new pmd */
> > > > > + set_pmd_at(mm, new_addr, new_pmd, pmd);
> > > > UML does not have set_pmd_at at all
> > > Every architecture does. :)
> >
> > I tried to build it patching vs 4.19-rc before I made this statement and
> > ran into that.
> >
> > Presently it does not.
> >
> > https://elixir.bootlin.com/linux/v4.19-rc7/ident/set_pmd_at - UML is not
> > on the list.
>
> Once this problem as well as the omissions in the include changes for UML in
> patch one have been fixed it appears to be working.
>
> What it needs is attached.
>
>
> >
> > >
> > > But it may come not from the arch code.
> >
> > There is no generic definition as far as I can see. All 12 defines in
> > 4.19 are in arch specific code. Unless i am missing something...
> >
> > >
> > > > If I read the code right, MIPS completely ignores the address
> > > > argument so
> > > > set_pmd_at there may not have the effect which this patch is trying to
> > > > achieve.
> > > Ignoring address is fine. Most architectures do that..
> > > The ideas is to move page table to the new pmd slot. It's nothing to do
> > > with the address passed to set_pmd_at().
> >
> > If that is it's only function, then I am going to appropriate the code
> > out of the MIPS tree for further uml testing. It does exactly that -
> > just move the pmd the new slot.
> >
> > >
> > A.
>
>
> A.
>
> From ac265d96897a346b05646fce91784ed4922c7f8d Mon Sep 17 00:00:00 2001
> From: Anton Ivanov <anton.ivanov@cambridgegreys.com>
> Date: Fri, 12 Oct 2018 17:24:10 +0100
> Subject: [PATCH] Incremental fixes to the mmremap patch
>
> Signed-off-by: Anton Ivanov <anton.ivanov@cambridgegreys.com>
> ---
> arch/um/include/asm/pgalloc.h | 4 ++--
> arch/um/include/asm/pgtable.h | 3 +++
> arch/um/kernel/tlb.c | 6 ++++++
> 3 files changed, 11 insertions(+), 2 deletions(-)
>
> diff --git a/arch/um/include/asm/pgalloc.h b/arch/um/include/asm/pgalloc.h
> index bf90b2aa2002..99eb5682792a 100644
> --- a/arch/um/include/asm/pgalloc.h
> +++ b/arch/um/include/asm/pgalloc.h
> @@ -25,8 +25,8 @@
> extern pgd_t *pgd_alloc(struct mm_struct *);
> extern void pgd_free(struct mm_struct *mm, pgd_t *pgd);
>
> -extern pte_t *pte_alloc_one_kernel(struct mm_struct *, unsigned long);
> -extern pgtable_t pte_alloc_one(struct mm_struct *, unsigned long);
> +extern pte_t *pte_alloc_one_kernel(struct mm_struct *);
> +extern pgtable_t pte_alloc_one(struct mm_struct *);
If its Ok, let me handle this bit since otherwise it complicates things for
me.
> static inline void pte_free_kernel(struct mm_struct *mm, pte_t *pte)
> {
> diff --git a/arch/um/include/asm/pgtable.h b/arch/um/include/asm/pgtable.h
> index 7485398d0737..1692da55e63a 100644
> --- a/arch/um/include/asm/pgtable.h
> +++ b/arch/um/include/asm/pgtable.h
> @@ -359,4 +359,7 @@ do { \
> __flush_tlb_one((vaddr)); \
> } while (0)
>
> +extern void set_pmd_at(struct mm_struct *mm, unsigned long addr,
> + pmd_t *pmdp, pmd_t pmd);
> +
> #endif
> diff --git a/arch/um/kernel/tlb.c b/arch/um/kernel/tlb.c
> index 763d35bdda01..d17b74184ba0 100644
> --- a/arch/um/kernel/tlb.c
> +++ b/arch/um/kernel/tlb.c
> @@ -647,3 +647,9 @@ void force_flush_all(void)
> vma = vma->vm_next;
> }
> }
> +void set_pmd_at(struct mm_struct *mm, unsigned long addr,
> + pmd_t *pmdp, pmd_t pmd)
> +{
> + *pmdp = pmd;
> +}
> +
I believe this should be included in a separate patch since it is not related
specifically to pte_alloc argument removal. If you want, I could split it
into a separate patch for my series with you as author.
thanks,
- Joel
^ permalink raw reply
* Re: [PATCH v2 2/2] mm: speed up mremap by 500x on large regions
From: Joel Fernandes @ 2018-10-12 16:57 UTC (permalink / raw)
To: Kirill A. Shutemov
Cc: linux-mips, Rich Felker, linux-ia64, linux-sh, Peter Zijlstra,
Catalin Marinas, Dave Hansen, Will Deacon, mhocko, linux-mm,
lokeshgidra, sparclinux, linux-riscv, elfring, Jonas Bonn,
linux-s390, dancol, Yoshinori Sato, linux-xtensa, linux-hexagon,
Helge Deller, maintainer:X86 ARCHITECTURE (32-BIT AND 64-BIT),
hughd, James E.J. Bottomley, kasan-dev, kvmarm, Ingo Molnar,
Geert Uytterhoeven, Andrey Ryabinin, linux-snps-arc, kernel-team,
Sam Creasey, Fenghua Yu, Jeff Dike, linux-um, Stefan Kristiansson,
Julia Lawall, linux-m68k, openrisc, Borislav Petkov,
Andy Lutomirski, nios2-dev, Stafford Horne, Guan Xuetao,
linux-arm-kernel, Chris Zankel, Tony Luck, Richard Weinberger,
linux-parisc, pantin, Max Filippov, linux-kernel, minchan,
Thomas Gleixner, linux-alpha, Ley Foon Tan, akpm, linuxppc-dev,
David S. Miller
In-Reply-To: <20181012131946.zoab2lpfmrycmuju@kshutemo-mobl1>
On Fri, Oct 12, 2018 at 04:19:46PM +0300, Kirill A. Shutemov wrote:
> On Fri, Oct 12, 2018 at 05:50:46AM -0700, Joel Fernandes wrote:
> > On Fri, Oct 12, 2018 at 02:30:56PM +0300, Kirill A. Shutemov wrote:
> > > On Thu, Oct 11, 2018 at 06:37:56PM -0700, Joel Fernandes (Google) wrote:
> > > > Android needs to mremap large regions of memory during memory management
> > > > related operations. The mremap system call can be really slow if THP is
> > > > not enabled. The bottleneck is move_page_tables, which is copying each
> > > > pte at a time, and can be really slow across a large map. Turning on THP
> > > > may not be a viable option, and is not for us. This patch speeds up the
> > > > performance for non-THP system by copying at the PMD level when possible.
> > > >
> > > > The speed up is three orders of magnitude. On a 1GB mremap, the mremap
> > > > completion times drops from 160-250 millesconds to 380-400 microseconds.
> > > >
> > > > Before:
> > > > Total mremap time for 1GB data: 242321014 nanoseconds.
> > > > Total mremap time for 1GB data: 196842467 nanoseconds.
> > > > Total mremap time for 1GB data: 167051162 nanoseconds.
> > > >
> > > > After:
> > > > Total mremap time for 1GB data: 385781 nanoseconds.
> > > > Total mremap time for 1GB data: 388959 nanoseconds.
> > > > Total mremap time for 1GB data: 402813 nanoseconds.
> > > >
> > > > Incase THP is enabled, the optimization is skipped. I also flush the
> > > > tlb every time we do this optimization since I couldn't find a way to
> > > > determine if the low-level PTEs are dirty. It is seen that the cost of
> > > > doing so is not much compared the improvement, on both x86-64 and arm64.
> > >
> > > I looked into the code more and noticed move_pte() helper called from
> > > move_ptes(). It changes PTE entry to suite new address.
> > >
> > > It is only defined in non-trivial way on Sparc. I don't know much about
> > > Sparc and it's hard for me to say if the optimization will break anything
> > > there.
> >
> > Sparc's move_pte seems to be flushing the D-cache to prevent aliasing. It is
> > not modifying the PTE itself AFAICS:
> >
> > #ifdef DCACHE_ALIASING_POSSIBLE
> > #define __HAVE_ARCH_MOVE_PTE
> > #define move_pte(pte, prot, old_addr, new_addr) \
> > ({ \
> > pte_t newpte = (pte); \
> > if (tlb_type != hypervisor && pte_present(pte)) { \
> > unsigned long this_pfn = pte_pfn(pte); \
> > \
> > if (pfn_valid(this_pfn) && \
> > (((old_addr) ^ (new_addr)) & (1 << 13))) \
> > flush_dcache_page_all(current->mm, \
> > pfn_to_page(this_pfn)); \
> > } \
> > newpte; \
> > })
> > #endif
> >
> > If its an issue, then how do transparent huge pages work on Sparc? I don't
> > see the huge page code (move_huge_pages) during mremap doing anything special
> > for Sparc architecture when moving PMDs..
>
> My *guess* is that it will work fine on Sparc as it apprarently it only
> cares about change in bit 13 of virtual address. It will never happen for
> huge pages or when PTE page tables move.
>
> But I just realized that the problem is bigger: since we pass new_addr to
> the set_pte_at() we would need to audit all implementations that they are
> safe with just moving PTE page table.
>
> I would rather go with per-architecture enabling. It's much safer.
I'm Ok with the per-arch enabling, I agree its safer. So I should be adding a
a new __HAVE_ARCH_MOVE_PMD right, or did you have a better name for that?
Also, do you feel we should still need to remove the address argument from
set_pte_alloc? Or should we leave that alone if we do per-arch?
I figure I spent a bunch of time on that already anyway, and its a clean up
anyway, so may as well do it. But perhaps that "pte_alloc cleanup" can then
be a separate patch independent of this series?
> > Also, do we not flush the caches from any path when we munmap address space?
> > We do call do_munmap on the old mapping from mremap after moving to the new one.
>
> Are you sure about that? It can be hided deeper in architecture-specific
> code.
I am sure we do call do_munmap, I was asking if we flush the caches as well.
If we're enabling this per architecture, then I guess it does not matter for
the purposes of this patch.
thanks,
- Joel
^ permalink raw reply
* Re: [PATCH v2 2/2] mm: speed up mremap by 500x on large regions
From: Joel Fernandes @ 2018-10-12 17:06 UTC (permalink / raw)
To: Anton Ivanov
Cc: linux-mips, Rich Felker, linux-ia64, linux-sh, Peter Zijlstra,
Catalin Marinas, Dave Hansen, Will Deacon, mhocko, linux-mm,
lokeshgidra, sparclinux, linux-riscv, elfring, Jonas Bonn,
linux-s390, dancol, Yoshinori Sato, Max Filippov, linux-hexagon,
Helge Deller, maintainer:X86 ARCHITECTURE 32-BIT AND 64-BIT,
hughd, James E.J. Bottomley, kasan-dev, kvmarm, Ingo Molnar,
Geert Uytterhoeven, Andrey Ryabinin, linux-snps-arc, kernel-team,
Sam Creasey, linux-xtensa, Jeff Dike, linux-alpha, linux-um,
Stefan Kristiansson, Julia Lawall, linux-m68k, openrisc,
Borislav Petkov, Andy Lutomirski, Ley Foon Tan,
Kirill A. Shutemov, Stafford Horne, Guan Xuetao, linux-arm-kernel,
Chris Zankel, Tony Luck, linux-parisc, pantin, linux-kernel,
Fenghua Yu, minchan, Thomas Gleixner, Richard Weinberger,
nios2-dev, akpm, linuxppc-dev, David S. Miller
In-Reply-To: <4f969958-913e-cb9f-48fb-e3a88e1d288c@kot-begemot.co.uk>
On Fri, Oct 12, 2018 at 05:58:40PM +0100, Anton Ivanov wrote:
[...]
> > > > > > If I read the code right, MIPS completely ignores the address
> > > > > > argument so
> > > > > > set_pmd_at there may not have the effect which this patch is trying to
> > > > > > achieve.
> > > > > Ignoring address is fine. Most architectures do that..
> > > > > The ideas is to move page table to the new pmd slot. It's nothing to do
> > > > > with the address passed to set_pmd_at().
> > > > If that is it's only function, then I am going to appropriate the code
> > > > out of the MIPS tree for further uml testing. It does exactly that -
> > > > just move the pmd the new slot.
> > > >
> > > > A.
> > >
> > > A.
> > >
> > > From ac265d96897a346b05646fce91784ed4922c7f8d Mon Sep 17 00:00:00 2001
> > > From: Anton Ivanov <anton.ivanov@cambridgegreys.com>
> > > Date: Fri, 12 Oct 2018 17:24:10 +0100
> > > Subject: [PATCH] Incremental fixes to the mmremap patch
> > >
> > > Signed-off-by: Anton Ivanov <anton.ivanov@cambridgegreys.com>
> > > ---
> > > arch/um/include/asm/pgalloc.h | 4 ++--
> > > arch/um/include/asm/pgtable.h | 3 +++
> > > arch/um/kernel/tlb.c | 6 ++++++
> > > 3 files changed, 11 insertions(+), 2 deletions(-)
> > >
> > > diff --git a/arch/um/include/asm/pgalloc.h b/arch/um/include/asm/pgalloc.h
> > > index bf90b2aa2002..99eb5682792a 100644
> > > --- a/arch/um/include/asm/pgalloc.h
> > > +++ b/arch/um/include/asm/pgalloc.h
> > > @@ -25,8 +25,8 @@
> > > extern pgd_t *pgd_alloc(struct mm_struct *);
> > > extern void pgd_free(struct mm_struct *mm, pgd_t *pgd);
> > > -extern pte_t *pte_alloc_one_kernel(struct mm_struct *, unsigned long);
> > > -extern pgtable_t pte_alloc_one(struct mm_struct *, unsigned long);
> > > +extern pte_t *pte_alloc_one_kernel(struct mm_struct *);
> > > +extern pgtable_t pte_alloc_one(struct mm_struct *);
> > If its Ok, let me handle this bit since otherwise it complicates things for
> > me.
> >
> > > static inline void pte_free_kernel(struct mm_struct *mm, pte_t *pte)
> > > {
> > > diff --git a/arch/um/include/asm/pgtable.h b/arch/um/include/asm/pgtable.h
> > > index 7485398d0737..1692da55e63a 100644
> > > --- a/arch/um/include/asm/pgtable.h
> > > +++ b/arch/um/include/asm/pgtable.h
> > > @@ -359,4 +359,7 @@ do { \
> > > __flush_tlb_one((vaddr)); \
> > > } while (0)
> > > +extern void set_pmd_at(struct mm_struct *mm, unsigned long addr,
> > > + pmd_t *pmdp, pmd_t pmd);
> > > +
> > > #endif
> > > diff --git a/arch/um/kernel/tlb.c b/arch/um/kernel/tlb.c
> > > index 763d35bdda01..d17b74184ba0 100644
> > > --- a/arch/um/kernel/tlb.c
> > > +++ b/arch/um/kernel/tlb.c
> > > @@ -647,3 +647,9 @@ void force_flush_all(void)
> > > vma = vma->vm_next;
> > > }
> > > }
> > > +void set_pmd_at(struct mm_struct *mm, unsigned long addr,
> > > + pmd_t *pmdp, pmd_t pmd)
> > > +{
> > > + *pmdp = pmd;
> > > +}
> > > +
> > I believe this should be included in a separate patch since it is not related
> > specifically to pte_alloc argument removal. If you want, I could split it
> > into a separate patch for my series with you as author.
>
>
> Whichever is more convenient for you.
Ok.
> One thing to note - tlb flush is extremely expensive on uml.
>
> I have lifted the definition of set_pmd_at from the mips tree and removed
> the tlb_flush_all from it for this exact reason.
>
> If I read the original patch correctly, it does its own flush control so
> set_pmd_at does not need to do a force flush every time. It is done further
> up the chain.
That is correct. It is not done during the optimization, but is done later
after the pmds have moved.
thanks,
- Joel
^ permalink raw reply
* Re: [PATCH v2 2/2] mm: speed up mremap by 500x on large regions
From: Anton Ivanov @ 2018-10-12 16:58 UTC (permalink / raw)
To: Joel Fernandes
Cc: linux-mips, Rich Felker, linux-ia64, linux-sh, Peter Zijlstra,
Catalin Marinas, Dave Hansen, Will Deacon, mhocko, linux-mm,
lokeshgidra, sparclinux, linux-riscv, elfring, Jonas Bonn,
linux-s390, dancol, Yoshinori Sato, Max Filippov, linux-hexagon,
Helge Deller, maintainer:X86 ARCHITECTURE 32-BIT AND 64-BIT,
hughd, James E.J. Bottomley, kasan-dev, kvmarm, Ingo Molnar,
Geert Uytterhoeven, Andrey Ryabinin, linux-snps-arc, kernel-team,
Sam Creasey, linux-xtensa, Jeff Dike, linux-alpha, linux-um,
Stefan Kristiansson, Julia Lawall, linux-m68k, openrisc,
Borislav Petkov, Andy Lutomirski, Ley Foon Tan,
Kirill A. Shutemov, Stafford Horne, Guan Xuetao, linux-arm-kernel,
Chris Zankel, Tony Luck, linux-parisc, pantin, linux-kernel,
Fenghua Yu, minchan, Thomas Gleixner, Richard Weinberger,
nios2-dev, akpm, linuxppc-dev, David S. Miller
In-Reply-To: <20181012165012.GD223066@joelaf.mtv.corp.google.com>
On 10/12/18 5:50 PM, Joel Fernandes wrote:
> On Fri, Oct 12, 2018 at 05:42:24PM +0100, Anton Ivanov wrote:
>> On 10/12/18 3:48 PM, Anton Ivanov wrote:
>>> On 12/10/2018 15:37, Kirill A. Shutemov wrote:
>>>> On Fri, Oct 12, 2018 at 03:09:49PM +0100, Anton Ivanov wrote:
>>>>> On 10/12/18 2:37 AM, Joel Fernandes (Google) wrote:
>>>>>> Android needs to mremap large regions of memory during
>>>>>> memory management
>>>>>> related operations. The mremap system call can be really
>>>>>> slow if THP is
>>>>>> not enabled. The bottleneck is move_page_tables, which is copying each
>>>>>> pte at a time, and can be really slow across a large map.
>>>>>> Turning on THP
>>>>>> may not be a viable option, and is not for us. This patch
>>>>>> speeds up the
>>>>>> performance for non-THP system by copying at the PMD level
>>>>>> when possible.
>>>>>>
>>>>>> The speed up is three orders of magnitude. On a 1GB mremap, the mremap
>>>>>> completion times drops from 160-250 millesconds to 380-400
>>>>>> microseconds.
>>>>>>
>>>>>> Before:
>>>>>> Total mremap time for 1GB data: 242321014 nanoseconds.
>>>>>> Total mremap time for 1GB data: 196842467 nanoseconds.
>>>>>> Total mremap time for 1GB data: 167051162 nanoseconds.
>>>>>>
>>>>>> After:
>>>>>> Total mremap time for 1GB data: 385781 nanoseconds.
>>>>>> Total mremap time for 1GB data: 388959 nanoseconds.
>>>>>> Total mremap time for 1GB data: 402813 nanoseconds.
>>>>>>
>>>>>> Incase THP is enabled, the optimization is skipped. I also flush the
>>>>>> tlb every time we do this optimization since I couldn't find a way to
>>>>>> determine if the low-level PTEs are dirty. It is seen that the cost of
>>>>>> doing so is not much compared the improvement, on both
>>>>>> x86-64 and arm64.
>>>>>>
>>>>>> Cc: minchan@kernel.org
>>>>>> Cc: pantin@google.com
>>>>>> Cc: hughd@google.com
>>>>>> Cc: lokeshgidra@google.com
>>>>>> Cc: dancol@google.com
>>>>>> Cc: mhocko@kernel.org
>>>>>> Cc: kirill@shutemov.name
>>>>>> Cc: akpm@linux-foundation.org
>>>>>> Signed-off-by: Joel Fernandes (Google) <joel@joelfernandes.org>
>>>>>> ---
>>>>>> mm/mremap.c | 62
>>>>>> +++++++++++++++++++++++++++++++++++++++++++++++++++++
>>>>>> 1 file changed, 62 insertions(+)
>>>>>>
>>>>>> diff --git a/mm/mremap.c b/mm/mremap.c
>>>>>> index 9e68a02a52b1..d82c485822ef 100644
>>>>>> --- a/mm/mremap.c
>>>>>> +++ b/mm/mremap.c
>>>>>> @@ -191,6 +191,54 @@ static void move_ptes(struct
>>>>>> vm_area_struct *vma, pmd_t *old_pmd,
>>>>>> drop_rmap_locks(vma);
>>>>>> }
>>>>>> +static bool move_normal_pmd(struct vm_area_struct *vma,
>>>>>> unsigned long old_addr,
>>>>>> + unsigned long new_addr, unsigned long old_end,
>>>>>> + pmd_t *old_pmd, pmd_t *new_pmd, bool *need_flush)
>>>>>> +{
>>>>>> + spinlock_t *old_ptl, *new_ptl;
>>>>>> + struct mm_struct *mm = vma->vm_mm;
>>>>>> +
>>>>>> + if ((old_addr & ~PMD_MASK) || (new_addr & ~PMD_MASK)
>>>>>> + || old_end - old_addr < PMD_SIZE)
>>>>>> + return false;
>>>>>> +
>>>>>> + /*
>>>>>> + * The destination pmd shouldn't be established, free_pgtables()
>>>>>> + * should have release it.
>>>>>> + */
>>>>>> + if (WARN_ON(!pmd_none(*new_pmd)))
>>>>>> + return false;
>>>>>> +
>>>>>> + /*
>>>>>> + * We don't have to worry about the ordering of src and dst
>>>>>> + * ptlocks because exclusive mmap_sem prevents deadlock.
>>>>>> + */
>>>>>> + old_ptl = pmd_lock(vma->vm_mm, old_pmd);
>>>>>> + if (old_ptl) {
>>>>>> + pmd_t pmd;
>>>>>> +
>>>>>> + new_ptl = pmd_lockptr(mm, new_pmd);
>>>>>> + if (new_ptl != old_ptl)
>>>>>> + spin_lock_nested(new_ptl, SINGLE_DEPTH_NESTING);
>>>>>> +
>>>>>> + /* Clear the pmd */
>>>>>> + pmd = *old_pmd;
>>>>>> + pmd_clear(old_pmd);
>>>>>> +
>>>>>> + VM_BUG_ON(!pmd_none(*new_pmd));
>>>>>> +
>>>>>> + /* Set the new pmd */
>>>>>> + set_pmd_at(mm, new_addr, new_pmd, pmd);
>>>>> UML does not have set_pmd_at at all
>>>> Every architecture does. :)
>>> I tried to build it patching vs 4.19-rc before I made this statement and
>>> ran into that.
>>>
>>> Presently it does not.
>>>
>>> https://elixir.bootlin.com/linux/v4.19-rc7/ident/set_pmd_at - UML is not
>>> on the list.
>> Once this problem as well as the omissions in the include changes for UML in
>> patch one have been fixed it appears to be working.
>>
>> What it needs is attached.
>>
>>
>>>> But it may come not from the arch code.
>>> There is no generic definition as far as I can see. All 12 defines in
>>> 4.19 are in arch specific code. Unless i am missing something...
>>>
>>>>> If I read the code right, MIPS completely ignores the address
>>>>> argument so
>>>>> set_pmd_at there may not have the effect which this patch is trying to
>>>>> achieve.
>>>> Ignoring address is fine. Most architectures do that..
>>>> The ideas is to move page table to the new pmd slot. It's nothing to do
>>>> with the address passed to set_pmd_at().
>>> If that is it's only function, then I am going to appropriate the code
>>> out of the MIPS tree for further uml testing. It does exactly that -
>>> just move the pmd the new slot.
>>>
>>> A.
>>
>> A.
>>
>> From ac265d96897a346b05646fce91784ed4922c7f8d Mon Sep 17 00:00:00 2001
>> From: Anton Ivanov <anton.ivanov@cambridgegreys.com>
>> Date: Fri, 12 Oct 2018 17:24:10 +0100
>> Subject: [PATCH] Incremental fixes to the mmremap patch
>>
>> Signed-off-by: Anton Ivanov <anton.ivanov@cambridgegreys.com>
>> ---
>> arch/um/include/asm/pgalloc.h | 4 ++--
>> arch/um/include/asm/pgtable.h | 3 +++
>> arch/um/kernel/tlb.c | 6 ++++++
>> 3 files changed, 11 insertions(+), 2 deletions(-)
>>
>> diff --git a/arch/um/include/asm/pgalloc.h b/arch/um/include/asm/pgalloc.h
>> index bf90b2aa2002..99eb5682792a 100644
>> --- a/arch/um/include/asm/pgalloc.h
>> +++ b/arch/um/include/asm/pgalloc.h
>> @@ -25,8 +25,8 @@
>> extern pgd_t *pgd_alloc(struct mm_struct *);
>> extern void pgd_free(struct mm_struct *mm, pgd_t *pgd);
>>
>> -extern pte_t *pte_alloc_one_kernel(struct mm_struct *, unsigned long);
>> -extern pgtable_t pte_alloc_one(struct mm_struct *, unsigned long);
>> +extern pte_t *pte_alloc_one_kernel(struct mm_struct *);
>> +extern pgtable_t pte_alloc_one(struct mm_struct *);
> If its Ok, let me handle this bit since otherwise it complicates things for
> me.
>
>> static inline void pte_free_kernel(struct mm_struct *mm, pte_t *pte)
>> {
>> diff --git a/arch/um/include/asm/pgtable.h b/arch/um/include/asm/pgtable.h
>> index 7485398d0737..1692da55e63a 100644
>> --- a/arch/um/include/asm/pgtable.h
>> +++ b/arch/um/include/asm/pgtable.h
>> @@ -359,4 +359,7 @@ do { \
>> __flush_tlb_one((vaddr)); \
>> } while (0)
>>
>> +extern void set_pmd_at(struct mm_struct *mm, unsigned long addr,
>> + pmd_t *pmdp, pmd_t pmd);
>> +
>> #endif
>> diff --git a/arch/um/kernel/tlb.c b/arch/um/kernel/tlb.c
>> index 763d35bdda01..d17b74184ba0 100644
>> --- a/arch/um/kernel/tlb.c
>> +++ b/arch/um/kernel/tlb.c
>> @@ -647,3 +647,9 @@ void force_flush_all(void)
>> vma = vma->vm_next;
>> }
>> }
>> +void set_pmd_at(struct mm_struct *mm, unsigned long addr,
>> + pmd_t *pmdp, pmd_t pmd)
>> +{
>> + *pmdp = pmd;
>> +}
>> +
> I believe this should be included in a separate patch since it is not related
> specifically to pte_alloc argument removal. If you want, I could split it
> into a separate patch for my series with you as author.
Whichever is more convenient for you.
One thing to note - tlb flush is extremely expensive on uml.
I have lifted the definition of set_pmd_at from the mips tree and
removed the tlb_flush_all from it for this exact reason.
If I read the original patch correctly, it does its own flush control so
set_pmd_at does not need to do a force flush every time. It is done
further up the chain.
Brgds,
A.
>
> thanks,
>
> - Joel
>
>
^ permalink raw reply
* Re: [PATCH v2 2/2] mm: speed up mremap by 500x on large regions
From: Anton Ivanov @ 2018-10-12 16:42 UTC (permalink / raw)
To: Kirill A. Shutemov
Cc: linux-mips, Rich Felker, linux-ia64, linux-sh, Peter Zijlstra,
Catalin Marinas, Dave Hansen, Will Deacon, mhocko, linux-mm,
lokeshgidra, Joel Fernandes (Google), linux-riscv, elfring,
Jonas Bonn, linux-s390, dancol, Yoshinori Sato, Max Filippov,
linux-hexagon, Helge Deller,
maintainer:X86 ARCHITECTURE 32-BIT AND 64-BIT, hughd,
James E.J. Bottomley, kasan-dev, kvmarm, Ingo Molnar,
Geert Uytterhoeven, Andrey Ryabinin, linux-snps-arc, kernel-team,
Sam Creasey, linux-xtensa, Jeff Dike, linux-alpha, linux-um,
Stefan Kristiansson, Julia Lawall, linux-m68k, openrisc,
Borislav Petkov, Andy Lutomirski, nios2-dev, Stafford Horne,
Guan Xuetao, linux-arm-kernel, Chris Zankel, Tony Luck,
linux-parisc, pantin, linux-kernel, Fenghua Yu, minchan,
Thomas Gleixner, Richard Weinberger, sparclinux, akpm,
linuxppc-dev, David S. Miller, Ley Foon Tan
In-Reply-To: <4dd52e22-5b51-9b30-7178-fde603a08f88@kot-begemot.co.uk>
[-- Attachment #1: Type: text/plain, Size: 5032 bytes --]
On 10/12/18 3:48 PM, Anton Ivanov wrote:
> On 12/10/2018 15:37, Kirill A. Shutemov wrote:
>> On Fri, Oct 12, 2018 at 03:09:49PM +0100, Anton Ivanov wrote:
>>> On 10/12/18 2:37 AM, Joel Fernandes (Google) wrote:
>>>> Android needs to mremap large regions of memory during memory
>>>> management
>>>> related operations. The mremap system call can be really slow if
>>>> THP is
>>>> not enabled. The bottleneck is move_page_tables, which is copying each
>>>> pte at a time, and can be really slow across a large map. Turning
>>>> on THP
>>>> may not be a viable option, and is not for us. This patch speeds up
>>>> the
>>>> performance for non-THP system by copying at the PMD level when
>>>> possible.
>>>>
>>>> The speed up is three orders of magnitude. On a 1GB mremap, the mremap
>>>> completion times drops from 160-250 millesconds to 380-400
>>>> microseconds.
>>>>
>>>> Before:
>>>> Total mremap time for 1GB data: 242321014 nanoseconds.
>>>> Total mremap time for 1GB data: 196842467 nanoseconds.
>>>> Total mremap time for 1GB data: 167051162 nanoseconds.
>>>>
>>>> After:
>>>> Total mremap time for 1GB data: 385781 nanoseconds.
>>>> Total mremap time for 1GB data: 388959 nanoseconds.
>>>> Total mremap time for 1GB data: 402813 nanoseconds.
>>>>
>>>> Incase THP is enabled, the optimization is skipped. I also flush the
>>>> tlb every time we do this optimization since I couldn't find a way to
>>>> determine if the low-level PTEs are dirty. It is seen that the cost of
>>>> doing so is not much compared the improvement, on both x86-64 and
>>>> arm64.
>>>>
>>>> Cc: minchan@kernel.org
>>>> Cc: pantin@google.com
>>>> Cc: hughd@google.com
>>>> Cc: lokeshgidra@google.com
>>>> Cc: dancol@google.com
>>>> Cc: mhocko@kernel.org
>>>> Cc: kirill@shutemov.name
>>>> Cc: akpm@linux-foundation.org
>>>> Signed-off-by: Joel Fernandes (Google) <joel@joelfernandes.org>
>>>> ---
>>>> mm/mremap.c | 62
>>>> +++++++++++++++++++++++++++++++++++++++++++++++++++++
>>>> 1 file changed, 62 insertions(+)
>>>>
>>>> diff --git a/mm/mremap.c b/mm/mremap.c
>>>> index 9e68a02a52b1..d82c485822ef 100644
>>>> --- a/mm/mremap.c
>>>> +++ b/mm/mremap.c
>>>> @@ -191,6 +191,54 @@ static void move_ptes(struct vm_area_struct
>>>> *vma, pmd_t *old_pmd,
>>>> drop_rmap_locks(vma);
>>>> }
>>>> +static bool move_normal_pmd(struct vm_area_struct *vma, unsigned
>>>> long old_addr,
>>>> + unsigned long new_addr, unsigned long old_end,
>>>> + pmd_t *old_pmd, pmd_t *new_pmd, bool *need_flush)
>>>> +{
>>>> + spinlock_t *old_ptl, *new_ptl;
>>>> + struct mm_struct *mm = vma->vm_mm;
>>>> +
>>>> + if ((old_addr & ~PMD_MASK) || (new_addr & ~PMD_MASK)
>>>> + || old_end - old_addr < PMD_SIZE)
>>>> + return false;
>>>> +
>>>> + /*
>>>> + * The destination pmd shouldn't be established, free_pgtables()
>>>> + * should have release it.
>>>> + */
>>>> + if (WARN_ON(!pmd_none(*new_pmd)))
>>>> + return false;
>>>> +
>>>> + /*
>>>> + * We don't have to worry about the ordering of src and dst
>>>> + * ptlocks because exclusive mmap_sem prevents deadlock.
>>>> + */
>>>> + old_ptl = pmd_lock(vma->vm_mm, old_pmd);
>>>> + if (old_ptl) {
>>>> + pmd_t pmd;
>>>> +
>>>> + new_ptl = pmd_lockptr(mm, new_pmd);
>>>> + if (new_ptl != old_ptl)
>>>> + spin_lock_nested(new_ptl, SINGLE_DEPTH_NESTING);
>>>> +
>>>> + /* Clear the pmd */
>>>> + pmd = *old_pmd;
>>>> + pmd_clear(old_pmd);
>>>> +
>>>> + VM_BUG_ON(!pmd_none(*new_pmd));
>>>> +
>>>> + /* Set the new pmd */
>>>> + set_pmd_at(mm, new_addr, new_pmd, pmd);
>>> UML does not have set_pmd_at at all
>> Every architecture does. :)
>
> I tried to build it patching vs 4.19-rc before I made this statement
> and ran into that.
>
> Presently it does not.
>
> https://elixir.bootlin.com/linux/v4.19-rc7/ident/set_pmd_at - UML is
> not on the list.
Once this problem as well as the omissions in the include changes for
UML in patch one have been fixed it appears to be working.
What it needs is attached.
>
>>
>> But it may come not from the arch code.
>
> There is no generic definition as far as I can see. All 12 defines in
> 4.19 are in arch specific code. Unless i am missing something...
>
>>
>>> If I read the code right, MIPS completely ignores the address
>>> argument so
>>> set_pmd_at there may not have the effect which this patch is trying to
>>> achieve.
>> Ignoring address is fine. Most architectures do that..
>> The ideas is to move page table to the new pmd slot. It's nothing to do
>> with the address passed to set_pmd_at().
>
> If that is it's only function, then I am going to appropriate the code
> out of the MIPS tree for further uml testing. It does exactly that -
> just move the pmd the new slot.
>
>>
> A.
A.
[-- Attachment #2: 0001-Incremental-fixes-to-the-mmremap-patch.patch --]
[-- Type: text/x-patch, Size: 1811 bytes --]
From ac265d96897a346b05646fce91784ed4922c7f8d Mon Sep 17 00:00:00 2001
From: Anton Ivanov <anton.ivanov@cambridgegreys.com>
Date: Fri, 12 Oct 2018 17:24:10 +0100
Subject: [PATCH] Incremental fixes to the mmremap patch
Signed-off-by: Anton Ivanov <anton.ivanov@cambridgegreys.com>
---
arch/um/include/asm/pgalloc.h | 4 ++--
arch/um/include/asm/pgtable.h | 3 +++
arch/um/kernel/tlb.c | 6 ++++++
3 files changed, 11 insertions(+), 2 deletions(-)
diff --git a/arch/um/include/asm/pgalloc.h b/arch/um/include/asm/pgalloc.h
index bf90b2aa2002..99eb5682792a 100644
--- a/arch/um/include/asm/pgalloc.h
+++ b/arch/um/include/asm/pgalloc.h
@@ -25,8 +25,8 @@
extern pgd_t *pgd_alloc(struct mm_struct *);
extern void pgd_free(struct mm_struct *mm, pgd_t *pgd);
-extern pte_t *pte_alloc_one_kernel(struct mm_struct *, unsigned long);
-extern pgtable_t pte_alloc_one(struct mm_struct *, unsigned long);
+extern pte_t *pte_alloc_one_kernel(struct mm_struct *);
+extern pgtable_t pte_alloc_one(struct mm_struct *);
static inline void pte_free_kernel(struct mm_struct *mm, pte_t *pte)
{
diff --git a/arch/um/include/asm/pgtable.h b/arch/um/include/asm/pgtable.h
index 7485398d0737..1692da55e63a 100644
--- a/arch/um/include/asm/pgtable.h
+++ b/arch/um/include/asm/pgtable.h
@@ -359,4 +359,7 @@ do { \
__flush_tlb_one((vaddr)); \
} while (0)
+extern void set_pmd_at(struct mm_struct *mm, unsigned long addr,
+ pmd_t *pmdp, pmd_t pmd);
+
#endif
diff --git a/arch/um/kernel/tlb.c b/arch/um/kernel/tlb.c
index 763d35bdda01..d17b74184ba0 100644
--- a/arch/um/kernel/tlb.c
+++ b/arch/um/kernel/tlb.c
@@ -647,3 +647,9 @@ void force_flush_all(void)
vma = vma->vm_next;
}
}
+void set_pmd_at(struct mm_struct *mm, unsigned long addr,
+ pmd_t *pmdp, pmd_t pmd)
+{
+ *pmdp = pmd;
+}
+
--
2.11.0
^ permalink raw reply related
* Re: [PATCH v2 2/2] mm: speed up mremap by 500x on large regions
From: Anton Ivanov @ 2018-10-12 14:09 UTC (permalink / raw)
To: Joel Fernandes (Google), linux-kernel
Cc: linux-mips, Rich Felker, linux-ia64, linux-sh, Peter Zijlstra,
Catalin Marinas, Dave Hansen, Will Deacon, mhocko, linux-mm,
lokeshgidra, sparclinux, linux-riscv, elfring, Jonas Bonn,
linux-s390, dancol, Yoshinori Sato, Max Filippov, linux-hexagon,
Helge Deller, maintainer:X86 ARCHITECTURE 32-BIT AND 64-BIT,
hughd, James E.J. Bottomley, kasan-dev, kvmarm, Ingo Molnar,
Geert Uytterhoeven, Andrey Ryabinin, linux-snps-arc, kernel-team,
Sam Creasey, linux-xtensa, Jeff Dike, linux-alpha, linux-um,
Stefan Kristiansson, Julia Lawall, linux-m68k, openrisc,
Borislav Petkov, Andy Lutomirski, Ley Foon Tan, kirill,
Stafford Horne, Guan Xuetao, linux-arm-kernel, Chris Zankel,
Tony Luck, linux-parisc, pantin, Fenghua Yu, minchan,
Thomas Gleixner, Richard Weinberger, nios2-dev, akpm,
linuxppc-dev, David S. Miller
In-Reply-To: <20181012013756.11285-2-joel@joelfernandes.org>
On 10/12/18 2:37 AM, Joel Fernandes (Google) wrote:
> Android needs to mremap large regions of memory during memory management
> related operations. The mremap system call can be really slow if THP is
> not enabled. The bottleneck is move_page_tables, which is copying each
> pte at a time, and can be really slow across a large map. Turning on THP
> may not be a viable option, and is not for us. This patch speeds up the
> performance for non-THP system by copying at the PMD level when possible.
>
> The speed up is three orders of magnitude. On a 1GB mremap, the mremap
> completion times drops from 160-250 millesconds to 380-400 microseconds.
>
> Before:
> Total mremap time for 1GB data: 242321014 nanoseconds.
> Total mremap time for 1GB data: 196842467 nanoseconds.
> Total mremap time for 1GB data: 167051162 nanoseconds.
>
> After:
> Total mremap time for 1GB data: 385781 nanoseconds.
> Total mremap time for 1GB data: 388959 nanoseconds.
> Total mremap time for 1GB data: 402813 nanoseconds.
>
> Incase THP is enabled, the optimization is skipped. I also flush the
> tlb every time we do this optimization since I couldn't find a way to
> determine if the low-level PTEs are dirty. It is seen that the cost of
> doing so is not much compared the improvement, on both x86-64 and arm64.
>
> Cc: minchan@kernel.org
> Cc: pantin@google.com
> Cc: hughd@google.com
> Cc: lokeshgidra@google.com
> Cc: dancol@google.com
> Cc: mhocko@kernel.org
> Cc: kirill@shutemov.name
> Cc: akpm@linux-foundation.org
> Signed-off-by: Joel Fernandes (Google) <joel@joelfernandes.org>
> ---
> mm/mremap.c | 62 +++++++++++++++++++++++++++++++++++++++++++++++++++++
> 1 file changed, 62 insertions(+)
>
> diff --git a/mm/mremap.c b/mm/mremap.c
> index 9e68a02a52b1..d82c485822ef 100644
> --- a/mm/mremap.c
> +++ b/mm/mremap.c
> @@ -191,6 +191,54 @@ static void move_ptes(struct vm_area_struct *vma, pmd_t *old_pmd,
> drop_rmap_locks(vma);
> }
>
> +static bool move_normal_pmd(struct vm_area_struct *vma, unsigned long old_addr,
> + unsigned long new_addr, unsigned long old_end,
> + pmd_t *old_pmd, pmd_t *new_pmd, bool *need_flush)
> +{
> + spinlock_t *old_ptl, *new_ptl;
> + struct mm_struct *mm = vma->vm_mm;
> +
> + if ((old_addr & ~PMD_MASK) || (new_addr & ~PMD_MASK)
> + || old_end - old_addr < PMD_SIZE)
> + return false;
> +
> + /*
> + * The destination pmd shouldn't be established, free_pgtables()
> + * should have release it.
> + */
> + if (WARN_ON(!pmd_none(*new_pmd)))
> + return false;
> +
> + /*
> + * We don't have to worry about the ordering of src and dst
> + * ptlocks because exclusive mmap_sem prevents deadlock.
> + */
> + old_ptl = pmd_lock(vma->vm_mm, old_pmd);
> + if (old_ptl) {
> + pmd_t pmd;
> +
> + new_ptl = pmd_lockptr(mm, new_pmd);
> + if (new_ptl != old_ptl)
> + spin_lock_nested(new_ptl, SINGLE_DEPTH_NESTING);
> +
> + /* Clear the pmd */
> + pmd = *old_pmd;
> + pmd_clear(old_pmd);
> +
> + VM_BUG_ON(!pmd_none(*new_pmd));
> +
> + /* Set the new pmd */
> + set_pmd_at(mm, new_addr, new_pmd, pmd);
UML does not have set_pmd_at at all
If I read the code right, MIPS completely ignores the address argument
so set_pmd_at there may not have the effect which this patch is trying
to achieve.
IMHO, this needs to be a per-architecture, not across full tree.
> + if (new_ptl != old_ptl)
> + spin_unlock(new_ptl);
> + spin_unlock(old_ptl);
> +
> + *need_flush = true;
> + return true;
> + }
> + return false;
> +}
> +
> unsigned long move_page_tables(struct vm_area_struct *vma,
> unsigned long old_addr, struct vm_area_struct *new_vma,
> unsigned long new_addr, unsigned long len,
> @@ -239,7 +287,21 @@ unsigned long move_page_tables(struct vm_area_struct *vma,
> split_huge_pmd(vma, old_pmd, old_addr);
> if (pmd_trans_unstable(old_pmd))
> continue;
> + } else if (extent == PMD_SIZE) {
> + bool moved;
> +
> + /* See comment in move_ptes() */
> + if (need_rmap_locks)
> + take_rmap_locks(vma);
> + moved = move_normal_pmd(vma, old_addr, new_addr,
> + old_end, old_pmd, new_pmd,
> + &need_flush);
> + if (need_rmap_locks)
> + drop_rmap_locks(vma);
> + if (moved)
> + continue;
> }
> +
> if (pte_alloc(new_vma->vm_mm, new_pmd))
> break;
> next = (new_addr + PMD_SIZE) & PMD_MASK;
Brgds,
A.
^ permalink raw reply
* Re: [PATCH v2 2/2] mm: speed up mremap by 500x on large regions
From: Anton Ivanov @ 2018-10-12 14:48 UTC (permalink / raw)
To: Kirill A. Shutemov
Cc: linux-mips, Rich Felker, linux-ia64, linux-sh, Peter Zijlstra,
Catalin Marinas, Dave Hansen, Will Deacon, mhocko, linux-mm,
lokeshgidra, Joel Fernandes (Google), linux-riscv, elfring,
Jonas Bonn, linux-s390, dancol, Yoshinori Sato, Max Filippov,
linux-hexagon, Helge Deller,
maintainer:X86 ARCHITECTURE 32-BIT AND 64-BIT, hughd,
James E.J. Bottomley, kasan-dev, kvmarm, Ingo Molnar,
Geert Uytterhoeven, Andrey Ryabinin, linux-snps-arc, kernel-team,
Sam Creasey, linux-xtensa, Jeff Dike, linux-alpha, linux-um,
Stefan Kristiansson, Julia Lawall, linux-m68k, openrisc,
Borislav Petkov, Andy Lutomirski, nios2-dev, Stafford Horne,
Guan Xuetao, linux-arm-kernel, Chris Zankel, Tony Luck,
linux-parisc, pantin, linux-kernel, Fenghua Yu, minchan,
Thomas Gleixner, Richard Weinberger, sparclinux, akpm,
linuxppc-dev, David S. Miller, Ley Foon Tan
In-Reply-To: <20181012143728.t42uvr6etg7gp7fh@kshutemo-mobl1>
On 12/10/2018 15:37, Kirill A. Shutemov wrote:
> On Fri, Oct 12, 2018 at 03:09:49PM +0100, Anton Ivanov wrote:
>> On 10/12/18 2:37 AM, Joel Fernandes (Google) wrote:
>>> Android needs to mremap large regions of memory during memory management
>>> related operations. The mremap system call can be really slow if THP is
>>> not enabled. The bottleneck is move_page_tables, which is copying each
>>> pte at a time, and can be really slow across a large map. Turning on THP
>>> may not be a viable option, and is not for us. This patch speeds up the
>>> performance for non-THP system by copying at the PMD level when possible.
>>>
>>> The speed up is three orders of magnitude. On a 1GB mremap, the mremap
>>> completion times drops from 160-250 millesconds to 380-400 microseconds.
>>>
>>> Before:
>>> Total mremap time for 1GB data: 242321014 nanoseconds.
>>> Total mremap time for 1GB data: 196842467 nanoseconds.
>>> Total mremap time for 1GB data: 167051162 nanoseconds.
>>>
>>> After:
>>> Total mremap time for 1GB data: 385781 nanoseconds.
>>> Total mremap time for 1GB data: 388959 nanoseconds.
>>> Total mremap time for 1GB data: 402813 nanoseconds.
>>>
>>> Incase THP is enabled, the optimization is skipped. I also flush the
>>> tlb every time we do this optimization since I couldn't find a way to
>>> determine if the low-level PTEs are dirty. It is seen that the cost of
>>> doing so is not much compared the improvement, on both x86-64 and arm64.
>>>
>>> Cc: minchan@kernel.org
>>> Cc: pantin@google.com
>>> Cc: hughd@google.com
>>> Cc: lokeshgidra@google.com
>>> Cc: dancol@google.com
>>> Cc: mhocko@kernel.org
>>> Cc: kirill@shutemov.name
>>> Cc: akpm@linux-foundation.org
>>> Signed-off-by: Joel Fernandes (Google) <joel@joelfernandes.org>
>>> ---
>>> mm/mremap.c | 62 +++++++++++++++++++++++++++++++++++++++++++++++++++++
>>> 1 file changed, 62 insertions(+)
>>>
>>> diff --git a/mm/mremap.c b/mm/mremap.c
>>> index 9e68a02a52b1..d82c485822ef 100644
>>> --- a/mm/mremap.c
>>> +++ b/mm/mremap.c
>>> @@ -191,6 +191,54 @@ static void move_ptes(struct vm_area_struct *vma, pmd_t *old_pmd,
>>> drop_rmap_locks(vma);
>>> }
>>> +static bool move_normal_pmd(struct vm_area_struct *vma, unsigned long old_addr,
>>> + unsigned long new_addr, unsigned long old_end,
>>> + pmd_t *old_pmd, pmd_t *new_pmd, bool *need_flush)
>>> +{
>>> + spinlock_t *old_ptl, *new_ptl;
>>> + struct mm_struct *mm = vma->vm_mm;
>>> +
>>> + if ((old_addr & ~PMD_MASK) || (new_addr & ~PMD_MASK)
>>> + || old_end - old_addr < PMD_SIZE)
>>> + return false;
>>> +
>>> + /*
>>> + * The destination pmd shouldn't be established, free_pgtables()
>>> + * should have release it.
>>> + */
>>> + if (WARN_ON(!pmd_none(*new_pmd)))
>>> + return false;
>>> +
>>> + /*
>>> + * We don't have to worry about the ordering of src and dst
>>> + * ptlocks because exclusive mmap_sem prevents deadlock.
>>> + */
>>> + old_ptl = pmd_lock(vma->vm_mm, old_pmd);
>>> + if (old_ptl) {
>>> + pmd_t pmd;
>>> +
>>> + new_ptl = pmd_lockptr(mm, new_pmd);
>>> + if (new_ptl != old_ptl)
>>> + spin_lock_nested(new_ptl, SINGLE_DEPTH_NESTING);
>>> +
>>> + /* Clear the pmd */
>>> + pmd = *old_pmd;
>>> + pmd_clear(old_pmd);
>>> +
>>> + VM_BUG_ON(!pmd_none(*new_pmd));
>>> +
>>> + /* Set the new pmd */
>>> + set_pmd_at(mm, new_addr, new_pmd, pmd);
>> UML does not have set_pmd_at at all
> Every architecture does. :)
I tried to build it patching vs 4.19-rc before I made this statement and
ran into that.
Presently it does not.
https://elixir.bootlin.com/linux/v4.19-rc7/ident/set_pmd_at - UML is not
on the list.
>
> But it may come not from the arch code.
There is no generic definition as far as I can see. All 12 defines in
4.19 are in arch specific code. Unless i am missing something...
>
>> If I read the code right, MIPS completely ignores the address argument so
>> set_pmd_at there may not have the effect which this patch is trying to
>> achieve.
> Ignoring address is fine. Most architectures do that..
> The ideas is to move page table to the new pmd slot. It's nothing to do
> with the address passed to set_pmd_at().
If that is it's only function, then I am going to appropriate the code
out of the MIPS tree for further uml testing. It does exactly that -
just move the pmd the new slot.
>
A.
^ permalink raw reply
* Re: [PATCH v2 1/2] treewide: remove unused address argument from pte_alloc functions
From: Anton Ivanov @ 2018-10-12 13:56 UTC (permalink / raw)
To: Joel Fernandes (Google), linux-kernel
Cc: linux-mips, Rich Felker, linux-ia64, linux-sh, Peter Zijlstra,
Catalin Marinas, Dave Hansen, Will Deacon, Michal Hocko, linux-mm,
lokeshgidra, sparclinux, linux-riscv, elfring, Jonas Bonn,
linux-s390, dancol, Yoshinori Sato, Max Filippov, linux-hexagon,
Helge Deller, maintainer:X86 ARCHITECTURE 32-BIT AND 64-BIT,
hughd, James E.J. Bottomley, kasan-dev, kvmarm, Ingo Molnar,
Geert Uytterhoeven, Andrey Ryabinin, linux-snps-arc, kernel-team,
Sam Creasey, linux-xtensa, Jeff Dike, linux-alpha, linux-um,
Stefan Kristiansson, Julia Lawall, linux-m68k, openrisc,
Borislav Petkov, Andy Lutomirski, Ley Foon Tan, kirill,
Stafford Horne, Guan Xuetao, linux-arm-kernel, Chris Zankel,
Tony Luck, linux-parisc, pantin, Fenghua Yu, minchan,
Thomas Gleixner, Richard Weinberger, nios2-dev, akpm,
linuxppc-dev, David S. Miller
In-Reply-To: <20181012013756.11285-1-joel@joelfernandes.org>
On 10/12/18 2:37 AM, Joel Fernandes (Google) wrote:
> This series speeds up mremap(2) syscall by copying page tables at the
> PMD level even for non-THP systems. There is concern that the extra
> 'address' argument that mremap passes to pte_alloc may do something
> subtle architecture related in the future, that makes the scheme not
> work. Also we find that there is no point in passing the 'address' to
> pte_alloc since its unused.
>
> This patch therefore removes this argument tree-wide resulting in a nice
> negative diff as well. Also ensuring along the way that the architecture
> does not do anything funky with 'address' argument that goes unnoticed.
>
> Build and boot tested on x86-64. Build tested on arm64.
>
> The changes were obtained by applying the following Coccinelle script.
> The pte_fragment_alloc was manually fixed up since it was only 2
> occurences and could not be easily generalized (and thanks Julia for
> answering all my silly and not-silly Coccinelle questions!).
>
> // Options: --include-headers --no-includes
> // Note: I split the 'identifier fn' line, so if you are manually
> // running it, please unsplit it so it runs for you.
>
> virtual patch
>
> @pte_alloc_func_def depends on patch exists@
> identifier E2;
> identifier fn =~
> "^(__pte_alloc|pte_alloc_one|pte_alloc|__pte_alloc_kernel|pte_alloc_one_kernel)$";
> type T2;
> @@
>
> fn(...
> - , T2 E2
> )
> { ... }
>
> @pte_alloc_func_proto depends on patch exists@
> identifier E1, E2, E4;
> type T1, T2, T3, T4;
> identifier fn =~
> "^(__pte_alloc|pte_alloc_one|pte_alloc|__pte_alloc_kernel|pte_alloc_one_kernel)$";
> @@
>
> (
> - T3 fn(T1 E1, T2 E2);
> + T3 fn(T1 E1);
> |
> - T3 fn(T1 E1, T2 E2, T4 E4);
> + T3 fn(T1 E1, T2 E2);
> )
>
> @pte_alloc_func_call depends on patch exists@
> expression E2;
> identifier fn =~
> "^(__pte_alloc|pte_alloc_one|pte_alloc|__pte_alloc_kernel|pte_alloc_one_kernel)$";
> @@
>
> fn(...
> -, E2
> )
>
> @pte_alloc_macro depends on patch exists@
> identifier fn =~
> "^(__pte_alloc|pte_alloc_one|pte_alloc|__pte_alloc_kernel|pte_alloc_one_kernel)$";
> identifier a, b, c;
> expression e;
> position p;
> @@
>
> (
> - #define fn(a, b, c)@p e
> + #define fn(a, b) e
> |
> - #define fn(a, b)@p e
> + #define fn(a) e
> )
>
> Suggested-by: Kirill A. Shutemov <kirill@shutemov.name>
> Cc: Michal Hocko <mhocko@kernel.org>
> Cc: Julia Lawall <Julia.Lawall@lip6.fr>
> Cc: elfring@users.sourceforge.net
> Signed-off-by: Joel Fernandes (Google) <joel@joelfernandes.org>
> ---
> arch/alpha/include/asm/pgalloc.h | 6 +++---
> arch/arc/include/asm/pgalloc.h | 5 ++---
> arch/arm/include/asm/pgalloc.h | 4 ++--
> arch/arm64/include/asm/pgalloc.h | 4 ++--
> arch/hexagon/include/asm/pgalloc.h | 6 ++----
> arch/ia64/include/asm/pgalloc.h | 5 ++---
> arch/m68k/include/asm/mcf_pgalloc.h | 8 ++------
> arch/m68k/include/asm/motorola_pgalloc.h | 4 ++--
> arch/m68k/include/asm/sun3_pgalloc.h | 6 ++----
> arch/microblaze/include/asm/pgalloc.h | 19 ++-----------------
> arch/microblaze/mm/pgtable.c | 3 +--
> arch/mips/include/asm/pgalloc.h | 6 ++----
> arch/nds32/include/asm/pgalloc.h | 5 ++---
> arch/nios2/include/asm/pgalloc.h | 6 ++----
> arch/openrisc/include/asm/pgalloc.h | 5 ++---
> arch/openrisc/mm/ioremap.c | 3 +--
> arch/parisc/include/asm/pgalloc.h | 4 ++--
> arch/powerpc/include/asm/book3s/32/pgalloc.h | 4 ++--
> arch/powerpc/include/asm/book3s/64/pgalloc.h | 12 +++++-------
> arch/powerpc/include/asm/nohash/32/pgalloc.h | 4 ++--
> arch/powerpc/include/asm/nohash/64/pgalloc.h | 6 ++----
> arch/powerpc/mm/pgtable-book3s64.c | 2 +-
> arch/powerpc/mm/pgtable_32.c | 4 ++--
> arch/riscv/include/asm/pgalloc.h | 6 ++----
> arch/s390/include/asm/pgalloc.h | 4 ++--
> arch/sh/include/asm/pgalloc.h | 6 ++----
> arch/sparc/include/asm/pgalloc_32.h | 5 ++---
> arch/sparc/include/asm/pgalloc_64.h | 6 ++----
> arch/sparc/mm/init_64.c | 6 ++----
> arch/sparc/mm/srmmu.c | 4 ++--
> arch/um/kernel/mem.c | 4 ++--
There is a declaration of pte_alloc_one in arch/um/include/asm/pgalloc.h
This patch missed it.
> arch/unicore32/include/asm/pgalloc.h | 4 ++--
> arch/x86/include/asm/pgalloc.h | 4 ++--
> arch/x86/mm/pgtable.c | 4 ++--
> arch/xtensa/include/asm/pgalloc.h | 8 +++-----
> include/linux/mm.h | 13 ++++++-------
> mm/huge_memory.c | 8 ++++----
> mm/kasan/kasan_init.c | 2 +-
> mm/memory.c | 17 ++++++++---------
> mm/migrate.c | 2 +-
> mm/mremap.c | 2 +-
> mm/userfaultfd.c | 2 +-
> virt/kvm/arm/mmu.c | 2 +-
> 43 files changed, 95 insertions(+), 145 deletions(-)
>
> diff --git a/arch/alpha/include/asm/pgalloc.h b/arch/alpha/include/asm/pgalloc.h
> index ab3e3a8638fb..02f9f91bb4f0 100644
> --- a/arch/alpha/include/asm/pgalloc.h
> +++ b/arch/alpha/include/asm/pgalloc.h
> @@ -52,7 +52,7 @@ pmd_free(struct mm_struct *mm, pmd_t *pmd)
> }
>
> static inline pte_t *
> -pte_alloc_one_kernel(struct mm_struct *mm, unsigned long address)
> +pte_alloc_one_kernel(struct mm_struct *mm)
> {
> pte_t *pte = (pte_t *)__get_free_page(GFP_KERNEL|__GFP_ZERO);
> return pte;
> @@ -65,9 +65,9 @@ pte_free_kernel(struct mm_struct *mm, pte_t *pte)
> }
>
> static inline pgtable_t
> -pte_alloc_one(struct mm_struct *mm, unsigned long address)
> +pte_alloc_one(struct mm_struct *mm)
> {
> - pte_t *pte = pte_alloc_one_kernel(mm, address);
> + pte_t *pte = pte_alloc_one_kernel(mm);
> struct page *page;
>
> if (!pte)
> diff --git a/arch/arc/include/asm/pgalloc.h b/arch/arc/include/asm/pgalloc.h
> index 3749234b7419..9c9b5a5ebf2e 100644
> --- a/arch/arc/include/asm/pgalloc.h
> +++ b/arch/arc/include/asm/pgalloc.h
> @@ -90,8 +90,7 @@ static inline int __get_order_pte(void)
> return get_order(PTRS_PER_PTE * sizeof(pte_t));
> }
>
> -static inline pte_t *pte_alloc_one_kernel(struct mm_struct *mm,
> - unsigned long address)
> +static inline pte_t *pte_alloc_one_kernel(struct mm_struct *mm)
> {
> pte_t *pte;
>
> @@ -102,7 +101,7 @@ static inline pte_t *pte_alloc_one_kernel(struct mm_struct *mm,
> }
>
> static inline pgtable_t
> -pte_alloc_one(struct mm_struct *mm, unsigned long address)
> +pte_alloc_one(struct mm_struct *mm)
> {
> pgtable_t pte_pg;
> struct page *page;
> diff --git a/arch/arm/include/asm/pgalloc.h b/arch/arm/include/asm/pgalloc.h
> index 2d7344f0e208..17ab72f0cc4e 100644
> --- a/arch/arm/include/asm/pgalloc.h
> +++ b/arch/arm/include/asm/pgalloc.h
> @@ -81,7 +81,7 @@ static inline void clean_pte_table(pte_t *pte)
> * +------------+
> */
> static inline pte_t *
> -pte_alloc_one_kernel(struct mm_struct *mm, unsigned long addr)
> +pte_alloc_one_kernel(struct mm_struct *mm)
> {
> pte_t *pte;
>
> @@ -93,7 +93,7 @@ pte_alloc_one_kernel(struct mm_struct *mm, unsigned long addr)
> }
>
> static inline pgtable_t
> -pte_alloc_one(struct mm_struct *mm, unsigned long addr)
> +pte_alloc_one(struct mm_struct *mm)
> {
> struct page *pte;
>
> diff --git a/arch/arm64/include/asm/pgalloc.h b/arch/arm64/include/asm/pgalloc.h
> index 2e05bcd944c8..52fa47c73bf0 100644
> --- a/arch/arm64/include/asm/pgalloc.h
> +++ b/arch/arm64/include/asm/pgalloc.h
> @@ -91,13 +91,13 @@ extern pgd_t *pgd_alloc(struct mm_struct *mm);
> extern void pgd_free(struct mm_struct *mm, pgd_t *pgdp);
>
> static inline pte_t *
> -pte_alloc_one_kernel(struct mm_struct *mm, unsigned long addr)
> +pte_alloc_one_kernel(struct mm_struct *mm)
> {
> return (pte_t *)__get_free_page(PGALLOC_GFP);
> }
>
> static inline pgtable_t
> -pte_alloc_one(struct mm_struct *mm, unsigned long addr)
> +pte_alloc_one(struct mm_struct *mm)
> {
> struct page *pte;
>
> diff --git a/arch/hexagon/include/asm/pgalloc.h b/arch/hexagon/include/asm/pgalloc.h
> index eeebf862c46c..d36183887b60 100644
> --- a/arch/hexagon/include/asm/pgalloc.h
> +++ b/arch/hexagon/include/asm/pgalloc.h
> @@ -59,8 +59,7 @@ static inline void pgd_free(struct mm_struct *mm, pgd_t *pgd)
> free_page((unsigned long) pgd);
> }
>
> -static inline struct page *pte_alloc_one(struct mm_struct *mm,
> - unsigned long address)
> +static inline struct page *pte_alloc_one(struct mm_struct *mm)
> {
> struct page *pte;
>
> @@ -75,8 +74,7 @@ static inline struct page *pte_alloc_one(struct mm_struct *mm,
> }
>
> /* _kernel variant gets to use a different allocator */
> -static inline pte_t *pte_alloc_one_kernel(struct mm_struct *mm,
> - unsigned long address)
> +static inline pte_t *pte_alloc_one_kernel(struct mm_struct *mm)
> {
> gfp_t flags = GFP_KERNEL | __GFP_ZERO;
> return (pte_t *) __get_free_page(flags);
> diff --git a/arch/ia64/include/asm/pgalloc.h b/arch/ia64/include/asm/pgalloc.h
> index 3ee5362f2661..c9e481023c25 100644
> --- a/arch/ia64/include/asm/pgalloc.h
> +++ b/arch/ia64/include/asm/pgalloc.h
> @@ -83,7 +83,7 @@ pmd_populate_kernel(struct mm_struct *mm, pmd_t * pmd_entry, pte_t * pte)
> pmd_val(*pmd_entry) = __pa(pte);
> }
>
> -static inline pgtable_t pte_alloc_one(struct mm_struct *mm, unsigned long addr)
> +static inline pgtable_t pte_alloc_one(struct mm_struct *mm)
> {
> struct page *page;
> void *pg;
> @@ -99,8 +99,7 @@ static inline pgtable_t pte_alloc_one(struct mm_struct *mm, unsigned long addr)
> return page;
> }
>
> -static inline pte_t *pte_alloc_one_kernel(struct mm_struct *mm,
> - unsigned long addr)
> +static inline pte_t *pte_alloc_one_kernel(struct mm_struct *mm)
> {
> return quicklist_alloc(0, GFP_KERNEL, NULL);
> }
> diff --git a/arch/m68k/include/asm/mcf_pgalloc.h b/arch/m68k/include/asm/mcf_pgalloc.h
> index 12fe700632f4..4399d712f6db 100644
> --- a/arch/m68k/include/asm/mcf_pgalloc.h
> +++ b/arch/m68k/include/asm/mcf_pgalloc.h
> @@ -12,8 +12,7 @@ extern inline void pte_free_kernel(struct mm_struct *mm, pte_t *pte)
>
> extern const char bad_pmd_string[];
>
> -extern inline pte_t *pte_alloc_one_kernel(struct mm_struct *mm,
> - unsigned long address)
> +extern inline pte_t *pte_alloc_one_kernel(struct mm_struct *mm)
> {
> unsigned long page = __get_free_page(GFP_DMA);
>
> @@ -32,8 +31,6 @@ extern inline pmd_t *pmd_alloc_kernel(pgd_t *pgd, unsigned long address)
> #define pmd_alloc_one_fast(mm, address) ({ BUG(); ((pmd_t *)1); })
> #define pmd_alloc_one(mm, address) ({ BUG(); ((pmd_t *)2); })
>
> -#define pte_alloc_one_fast(mm, addr) pte_alloc_one(mm, addr)
> -
> #define pmd_populate(mm, pmd, page) (pmd_val(*pmd) = \
> (unsigned long)(page_address(page)))
>
> @@ -50,8 +47,7 @@ static inline void __pte_free_tlb(struct mmu_gather *tlb, pgtable_t page,
>
> #define __pmd_free_tlb(tlb, pmd, address) do { } while (0)
>
> -static inline struct page *pte_alloc_one(struct mm_struct *mm,
> - unsigned long address)
> +static inline struct page *pte_alloc_one(struct mm_struct *mm)
> {
> struct page *page = alloc_pages(GFP_DMA, 0);
> pte_t *pte;
> diff --git a/arch/m68k/include/asm/motorola_pgalloc.h b/arch/m68k/include/asm/motorola_pgalloc.h
> index 7859a86319cf..d04d9ba9b976 100644
> --- a/arch/m68k/include/asm/motorola_pgalloc.h
> +++ b/arch/m68k/include/asm/motorola_pgalloc.h
> @@ -8,7 +8,7 @@
> extern pmd_t *get_pointer_table(void);
> extern int free_pointer_table(pmd_t *);
>
> -static inline pte_t *pte_alloc_one_kernel(struct mm_struct *mm, unsigned long address)
> +static inline pte_t *pte_alloc_one_kernel(struct mm_struct *mm)
> {
> pte_t *pte;
>
> @@ -28,7 +28,7 @@ static inline void pte_free_kernel(struct mm_struct *mm, pte_t *pte)
> free_page((unsigned long) pte);
> }
>
> -static inline pgtable_t pte_alloc_one(struct mm_struct *mm, unsigned long address)
> +static inline pgtable_t pte_alloc_one(struct mm_struct *mm)
> {
> struct page *page;
> pte_t *pte;
> diff --git a/arch/m68k/include/asm/sun3_pgalloc.h b/arch/m68k/include/asm/sun3_pgalloc.h
> index 11485d38de4e..1456c5eecbd9 100644
> --- a/arch/m68k/include/asm/sun3_pgalloc.h
> +++ b/arch/m68k/include/asm/sun3_pgalloc.h
> @@ -35,8 +35,7 @@ do { \
> tlb_remove_page((tlb), pte); \
> } while (0)
>
> -static inline pte_t *pte_alloc_one_kernel(struct mm_struct *mm,
> - unsigned long address)
> +static inline pte_t *pte_alloc_one_kernel(struct mm_struct *mm)
> {
> unsigned long page = __get_free_page(GFP_KERNEL);
>
> @@ -47,8 +46,7 @@ static inline pte_t *pte_alloc_one_kernel(struct mm_struct *mm,
> return (pte_t *) (page);
> }
>
> -static inline pgtable_t pte_alloc_one(struct mm_struct *mm,
> - unsigned long address)
> +static inline pgtable_t pte_alloc_one(struct mm_struct *mm)
> {
> struct page *page = alloc_pages(GFP_KERNEL, 0);
>
> diff --git a/arch/microblaze/include/asm/pgalloc.h b/arch/microblaze/include/asm/pgalloc.h
> index 7c89390c0c13..f4cc9ffc449e 100644
> --- a/arch/microblaze/include/asm/pgalloc.h
> +++ b/arch/microblaze/include/asm/pgalloc.h
> @@ -108,10 +108,9 @@ static inline void free_pgd_slow(pgd_t *pgd)
> #define pmd_alloc_one_fast(mm, address) ({ BUG(); ((pmd_t *)1); })
> #define pmd_alloc_one(mm, address) ({ BUG(); ((pmd_t *)2); })
>
> -extern pte_t *pte_alloc_one_kernel(struct mm_struct *mm, unsigned long addr);
> +extern pte_t *pte_alloc_one_kernel(struct mm_struct *mm);
>
> -static inline struct page *pte_alloc_one(struct mm_struct *mm,
> - unsigned long address)
> +static inline struct page *pte_alloc_one(struct mm_struct *mm)
> {
> struct page *ptepage;
>
> @@ -132,20 +131,6 @@ static inline struct page *pte_alloc_one(struct mm_struct *mm,
> return ptepage;
> }
>
> -static inline pte_t *pte_alloc_one_fast(struct mm_struct *mm,
> - unsigned long address)
> -{
> - unsigned long *ret;
> -
> - ret = pte_quicklist;
> - if (ret != NULL) {
> - pte_quicklist = (unsigned long *)(*ret);
> - ret[0] = 0;
> - pgtable_cache_size--;
> - }
> - return (pte_t *)ret;
> -}
> -
> static inline void pte_free_fast(pte_t *pte)
> {
> *(unsigned long **)pte = pte_quicklist;
> diff --git a/arch/microblaze/mm/pgtable.c b/arch/microblaze/mm/pgtable.c
> index 7f525962cdfa..c2ce1e42b888 100644
> --- a/arch/microblaze/mm/pgtable.c
> +++ b/arch/microblaze/mm/pgtable.c
> @@ -235,8 +235,7 @@ unsigned long iopa(unsigned long addr)
> return pa;
> }
>
> -__ref pte_t *pte_alloc_one_kernel(struct mm_struct *mm,
> - unsigned long address)
> +__ref pte_t *pte_alloc_one_kernel(struct mm_struct *mm)
> {
> pte_t *pte;
> if (mem_init_done) {
> diff --git a/arch/mips/include/asm/pgalloc.h b/arch/mips/include/asm/pgalloc.h
> index 39b9f311c4ef..27808d9461f4 100644
> --- a/arch/mips/include/asm/pgalloc.h
> +++ b/arch/mips/include/asm/pgalloc.h
> @@ -50,14 +50,12 @@ static inline void pgd_free(struct mm_struct *mm, pgd_t *pgd)
> free_pages((unsigned long)pgd, PGD_ORDER);
> }
>
> -static inline pte_t *pte_alloc_one_kernel(struct mm_struct *mm,
> - unsigned long address)
> +static inline pte_t *pte_alloc_one_kernel(struct mm_struct *mm)
> {
> return (pte_t *)__get_free_pages(GFP_KERNEL | __GFP_ZERO, PTE_ORDER);
> }
>
> -static inline struct page *pte_alloc_one(struct mm_struct *mm,
> - unsigned long address)
> +static inline struct page *pte_alloc_one(struct mm_struct *mm)
> {
> struct page *pte;
>
> diff --git a/arch/nds32/include/asm/pgalloc.h b/arch/nds32/include/asm/pgalloc.h
> index 27448869131a..3c5fee5b5759 100644
> --- a/arch/nds32/include/asm/pgalloc.h
> +++ b/arch/nds32/include/asm/pgalloc.h
> @@ -22,8 +22,7 @@ extern void pgd_free(struct mm_struct *mm, pgd_t * pgd);
>
> #define check_pgt_cache() do { } while (0)
>
> -static inline pte_t *pte_alloc_one_kernel(struct mm_struct *mm,
> - unsigned long addr)
> +static inline pte_t *pte_alloc_one_kernel(struct mm_struct *mm)
> {
> pte_t *pte;
>
> @@ -34,7 +33,7 @@ static inline pte_t *pte_alloc_one_kernel(struct mm_struct *mm,
> return pte;
> }
>
> -static inline pgtable_t pte_alloc_one(struct mm_struct *mm, unsigned long addr)
> +static inline pgtable_t pte_alloc_one(struct mm_struct *mm)
> {
> pgtable_t pte;
>
> diff --git a/arch/nios2/include/asm/pgalloc.h b/arch/nios2/include/asm/pgalloc.h
> index bb47d08c8ef7..3a149ead1207 100644
> --- a/arch/nios2/include/asm/pgalloc.h
> +++ b/arch/nios2/include/asm/pgalloc.h
> @@ -37,8 +37,7 @@ static inline void pgd_free(struct mm_struct *mm, pgd_t *pgd)
> free_pages((unsigned long)pgd, PGD_ORDER);
> }
>
> -static inline pte_t *pte_alloc_one_kernel(struct mm_struct *mm,
> - unsigned long address)
> +static inline pte_t *pte_alloc_one_kernel(struct mm_struct *mm)
> {
> pte_t *pte;
>
> @@ -47,8 +46,7 @@ static inline pte_t *pte_alloc_one_kernel(struct mm_struct *mm,
> return pte;
> }
>
> -static inline pgtable_t pte_alloc_one(struct mm_struct *mm,
> - unsigned long address)
> +static inline pgtable_t pte_alloc_one(struct mm_struct *mm)
> {
> struct page *pte;
>
> diff --git a/arch/openrisc/include/asm/pgalloc.h b/arch/openrisc/include/asm/pgalloc.h
> index 8999b9226512..149c82ee4b8b 100644
> --- a/arch/openrisc/include/asm/pgalloc.h
> +++ b/arch/openrisc/include/asm/pgalloc.h
> @@ -70,10 +70,9 @@ static inline void pgd_free(struct mm_struct *mm, pgd_t *pgd)
> free_page((unsigned long)pgd);
> }
>
> -extern pte_t *pte_alloc_one_kernel(struct mm_struct *mm, unsigned long address);
> +extern pte_t *pte_alloc_one_kernel(struct mm_struct *mm);
>
> -static inline struct page *pte_alloc_one(struct mm_struct *mm,
> - unsigned long address)
> +static inline struct page *pte_alloc_one(struct mm_struct *mm)
> {
> struct page *pte;
> pte = alloc_pages(GFP_KERNEL, 0);
> diff --git a/arch/openrisc/mm/ioremap.c b/arch/openrisc/mm/ioremap.c
> index 2175e4bfd9fc..24fb1021c75a 100644
> --- a/arch/openrisc/mm/ioremap.c
> +++ b/arch/openrisc/mm/ioremap.c
> @@ -118,8 +118,7 @@ EXPORT_SYMBOL(iounmap);
> * the memblock infrastructure.
> */
>
> -pte_t __ref *pte_alloc_one_kernel(struct mm_struct *mm,
> - unsigned long address)
> +pte_t __ref *pte_alloc_one_kernel(struct mm_struct *mm)
> {
> pte_t *pte;
>
> diff --git a/arch/parisc/include/asm/pgalloc.h b/arch/parisc/include/asm/pgalloc.h
> index cf13275f7c6d..d05c678c77c4 100644
> --- a/arch/parisc/include/asm/pgalloc.h
> +++ b/arch/parisc/include/asm/pgalloc.h
> @@ -122,7 +122,7 @@ pmd_populate_kernel(struct mm_struct *mm, pmd_t *pmd, pte_t *pte)
> #define pmd_pgtable(pmd) pmd_page(pmd)
>
> static inline pgtable_t
> -pte_alloc_one(struct mm_struct *mm, unsigned long address)
> +pte_alloc_one(struct mm_struct *mm)
> {
> struct page *page = alloc_page(GFP_KERNEL|__GFP_ZERO);
> if (!page)
> @@ -135,7 +135,7 @@ pte_alloc_one(struct mm_struct *mm, unsigned long address)
> }
>
> static inline pte_t *
> -pte_alloc_one_kernel(struct mm_struct *mm, unsigned long addr)
> +pte_alloc_one_kernel(struct mm_struct *mm)
> {
> pte_t *pte = (pte_t *)__get_free_page(GFP_KERNEL|__GFP_ZERO);
> return pte;
> diff --git a/arch/powerpc/include/asm/book3s/32/pgalloc.h b/arch/powerpc/include/asm/book3s/32/pgalloc.h
> index 82e44b1a00ae..af9e13555d95 100644
> --- a/arch/powerpc/include/asm/book3s/32/pgalloc.h
> +++ b/arch/powerpc/include/asm/book3s/32/pgalloc.h
> @@ -82,8 +82,8 @@ static inline void pmd_populate(struct mm_struct *mm, pmd_t *pmdp,
> #define pmd_pgtable(pmd) pmd_page(pmd)
> #endif
>
> -extern pte_t *pte_alloc_one_kernel(struct mm_struct *mm, unsigned long addr);
> -extern pgtable_t pte_alloc_one(struct mm_struct *mm, unsigned long addr);
> +extern pte_t *pte_alloc_one_kernel(struct mm_struct *mm);
> +extern pgtable_t pte_alloc_one(struct mm_struct *mm);
>
> static inline void pte_free_kernel(struct mm_struct *mm, pte_t *pte)
> {
> diff --git a/arch/powerpc/include/asm/book3s/64/pgalloc.h b/arch/powerpc/include/asm/book3s/64/pgalloc.h
> index 391ed2c3b697..8f1d92e99fe5 100644
> --- a/arch/powerpc/include/asm/book3s/64/pgalloc.h
> +++ b/arch/powerpc/include/asm/book3s/64/pgalloc.h
> @@ -42,7 +42,7 @@ extern struct kmem_cache *pgtable_cache[];
> pgtable_cache[(shift) - 1]; \
> })
>
> -extern pte_t *pte_fragment_alloc(struct mm_struct *, unsigned long, int);
> +extern pte_t *pte_fragment_alloc(struct mm_struct *, int);
> extern pmd_t *pmd_fragment_alloc(struct mm_struct *, unsigned long);
> extern void pte_fragment_free(unsigned long *, int);
> extern void pmd_fragment_free(unsigned long *);
> @@ -192,16 +192,14 @@ static inline pgtable_t pmd_pgtable(pmd_t pmd)
> return (pgtable_t)pmd_page_vaddr(pmd);
> }
>
> -static inline pte_t *pte_alloc_one_kernel(struct mm_struct *mm,
> - unsigned long address)
> +static inline pte_t *pte_alloc_one_kernel(struct mm_struct *mm)
> {
> - return (pte_t *)pte_fragment_alloc(mm, address, 1);
> + return (pte_t *)pte_fragment_alloc(mm, 1);
> }
>
> -static inline pgtable_t pte_alloc_one(struct mm_struct *mm,
> - unsigned long address)
> +static inline pgtable_t pte_alloc_one(struct mm_struct *mm)
> {
> - return (pgtable_t)pte_fragment_alloc(mm, address, 0);
> + return (pgtable_t)pte_fragment_alloc(mm, 0);
> }
>
> static inline void pte_free_kernel(struct mm_struct *mm, pte_t *pte)
> diff --git a/arch/powerpc/include/asm/nohash/32/pgalloc.h b/arch/powerpc/include/asm/nohash/32/pgalloc.h
> index 8825953c225b..16623f53f0d4 100644
> --- a/arch/powerpc/include/asm/nohash/32/pgalloc.h
> +++ b/arch/powerpc/include/asm/nohash/32/pgalloc.h
> @@ -83,8 +83,8 @@ static inline void pmd_populate(struct mm_struct *mm, pmd_t *pmdp,
> #define pmd_pgtable(pmd) pmd_page(pmd)
> #endif
>
> -extern pte_t *pte_alloc_one_kernel(struct mm_struct *mm, unsigned long addr);
> -extern pgtable_t pte_alloc_one(struct mm_struct *mm, unsigned long addr);
> +extern pte_t *pte_alloc_one_kernel(struct mm_struct *mm);
> +extern pgtable_t pte_alloc_one(struct mm_struct *mm);
>
> static inline void pte_free_kernel(struct mm_struct *mm, pte_t *pte)
> {
> diff --git a/arch/powerpc/include/asm/nohash/64/pgalloc.h b/arch/powerpc/include/asm/nohash/64/pgalloc.h
> index e2d62d033708..2e7e0230edf4 100644
> --- a/arch/powerpc/include/asm/nohash/64/pgalloc.h
> +++ b/arch/powerpc/include/asm/nohash/64/pgalloc.h
> @@ -96,14 +96,12 @@ static inline void pmd_free(struct mm_struct *mm, pmd_t *pmd)
> }
>
>
> -static inline pte_t *pte_alloc_one_kernel(struct mm_struct *mm,
> - unsigned long address)
> +static inline pte_t *pte_alloc_one_kernel(struct mm_struct *mm)
> {
> return (pte_t *)__get_free_page(GFP_KERNEL | __GFP_ZERO);
> }
>
> -static inline pgtable_t pte_alloc_one(struct mm_struct *mm,
> - unsigned long address)
> +static inline pgtable_t pte_alloc_one(struct mm_struct *mm)
> {
> struct page *page;
> pte_t *pte;
> diff --git a/arch/powerpc/mm/pgtable-book3s64.c b/arch/powerpc/mm/pgtable-book3s64.c
> index 01d7c0f7c4f0..cff1d426ca6a 100644
> --- a/arch/powerpc/mm/pgtable-book3s64.c
> +++ b/arch/powerpc/mm/pgtable-book3s64.c
> @@ -379,7 +379,7 @@ static pte_t *__alloc_for_ptecache(struct mm_struct *mm, int kernel)
> return (pte_t *)ret;
> }
>
> -pte_t *pte_fragment_alloc(struct mm_struct *mm, unsigned long vmaddr, int kernel)
> +pte_t *pte_fragment_alloc(struct mm_struct *mm, int kernel)
> {
> pte_t *pte;
>
> diff --git a/arch/powerpc/mm/pgtable_32.c b/arch/powerpc/mm/pgtable_32.c
> index 120a49bfb9c6..b99a89cdcc5e 100644
> --- a/arch/powerpc/mm/pgtable_32.c
> +++ b/arch/powerpc/mm/pgtable_32.c
> @@ -43,7 +43,7 @@ EXPORT_SYMBOL(ioremap_bot); /* aka VMALLOC_END */
>
> extern char etext[], _stext[], _sinittext[], _einittext[];
>
> -__ref pte_t *pte_alloc_one_kernel(struct mm_struct *mm, unsigned long address)
> +__ref pte_t *pte_alloc_one_kernel(struct mm_struct *mm)
> {
> pte_t *pte;
>
> @@ -57,7 +57,7 @@ __ref pte_t *pte_alloc_one_kernel(struct mm_struct *mm, unsigned long address)
> return pte;
> }
>
> -pgtable_t pte_alloc_one(struct mm_struct *mm, unsigned long address)
> +pgtable_t pte_alloc_one(struct mm_struct *mm)
> {
> struct page *ptepage;
>
> diff --git a/arch/riscv/include/asm/pgalloc.h b/arch/riscv/include/asm/pgalloc.h
> index a79ed5faff3a..94043cf83c90 100644
> --- a/arch/riscv/include/asm/pgalloc.h
> +++ b/arch/riscv/include/asm/pgalloc.h
> @@ -82,15 +82,13 @@ static inline void pmd_free(struct mm_struct *mm, pmd_t *pmd)
>
> #endif /* __PAGETABLE_PMD_FOLDED */
>
> -static inline pte_t *pte_alloc_one_kernel(struct mm_struct *mm,
> - unsigned long address)
> +static inline pte_t *pte_alloc_one_kernel(struct mm_struct *mm)
> {
> return (pte_t *)__get_free_page(
> GFP_KERNEL | __GFP_RETRY_MAYFAIL | __GFP_ZERO);
> }
>
> -static inline struct page *pte_alloc_one(struct mm_struct *mm,
> - unsigned long address)
> +static inline struct page *pte_alloc_one(struct mm_struct *mm)
> {
> struct page *pte;
>
> diff --git a/arch/s390/include/asm/pgalloc.h b/arch/s390/include/asm/pgalloc.h
> index f0f9bcf94c03..ce2ca8cbd2ec 100644
> --- a/arch/s390/include/asm/pgalloc.h
> +++ b/arch/s390/include/asm/pgalloc.h
> @@ -139,8 +139,8 @@ static inline void pmd_populate(struct mm_struct *mm,
> /*
> * page table entry allocation/free routines.
> */
> -#define pte_alloc_one_kernel(mm, vmaddr) ((pte_t *) page_table_alloc(mm))
> -#define pte_alloc_one(mm, vmaddr) ((pte_t *) page_table_alloc(mm))
> +#define pte_alloc_one_kernel(mm) ((pte_t *)page_table_alloc(mm))
> +#define pte_alloc_one(mm) ((pte_t *)page_table_alloc(mm))
>
> #define pte_free_kernel(mm, pte) page_table_free(mm, (unsigned long *) pte)
> #define pte_free(mm, pte) page_table_free(mm, (unsigned long *) pte)
> diff --git a/arch/sh/include/asm/pgalloc.h b/arch/sh/include/asm/pgalloc.h
> index ed053a359ab7..8ad73cb31121 100644
> --- a/arch/sh/include/asm/pgalloc.h
> +++ b/arch/sh/include/asm/pgalloc.h
> @@ -32,14 +32,12 @@ static inline void pmd_populate(struct mm_struct *mm, pmd_t *pmd,
> /*
> * Allocate and free page tables.
> */
> -static inline pte_t *pte_alloc_one_kernel(struct mm_struct *mm,
> - unsigned long address)
> +static inline pte_t *pte_alloc_one_kernel(struct mm_struct *mm)
> {
> return quicklist_alloc(QUICK_PT, GFP_KERNEL, NULL);
> }
>
> -static inline pgtable_t pte_alloc_one(struct mm_struct *mm,
> - unsigned long address)
> +static inline pgtable_t pte_alloc_one(struct mm_struct *mm)
> {
> struct page *page;
> void *pg;
> diff --git a/arch/sparc/include/asm/pgalloc_32.h b/arch/sparc/include/asm/pgalloc_32.h
> index 90459481c6c7..282be50a4adf 100644
> --- a/arch/sparc/include/asm/pgalloc_32.h
> +++ b/arch/sparc/include/asm/pgalloc_32.h
> @@ -58,10 +58,9 @@ void pmd_populate(struct mm_struct *mm, pmd_t *pmdp, struct page *ptep);
> void pmd_set(pmd_t *pmdp, pte_t *ptep);
> #define pmd_populate_kernel(MM, PMD, PTE) pmd_set(PMD, PTE)
>
> -pgtable_t pte_alloc_one(struct mm_struct *mm, unsigned long address);
> +pgtable_t pte_alloc_one(struct mm_struct *mm);
>
> -static inline pte_t *pte_alloc_one_kernel(struct mm_struct *mm,
> - unsigned long address)
> +static inline pte_t *pte_alloc_one_kernel(struct mm_struct *mm)
> {
> return srmmu_get_nocache(PTE_SIZE, PTE_SIZE);
> }
> diff --git a/arch/sparc/include/asm/pgalloc_64.h b/arch/sparc/include/asm/pgalloc_64.h
> index 874632f34f62..48abccba4991 100644
> --- a/arch/sparc/include/asm/pgalloc_64.h
> +++ b/arch/sparc/include/asm/pgalloc_64.h
> @@ -60,10 +60,8 @@ static inline void pmd_free(struct mm_struct *mm, pmd_t *pmd)
> kmem_cache_free(pgtable_cache, pmd);
> }
>
> -pte_t *pte_alloc_one_kernel(struct mm_struct *mm,
> - unsigned long address);
> -pgtable_t pte_alloc_one(struct mm_struct *mm,
> - unsigned long address);
> +pte_t *pte_alloc_one_kernel(struct mm_struct *mm);
> +pgtable_t pte_alloc_one(struct mm_struct *mm);
> void pte_free_kernel(struct mm_struct *mm, pte_t *pte);
> void pte_free(struct mm_struct *mm, pgtable_t ptepage);
>
> diff --git a/arch/sparc/mm/init_64.c b/arch/sparc/mm/init_64.c
> index f396048a0d68..6133f21811e9 100644
> --- a/arch/sparc/mm/init_64.c
> +++ b/arch/sparc/mm/init_64.c
> @@ -2921,8 +2921,7 @@ void __flush_tlb_all(void)
> : : "r" (pstate));
> }
>
> -pte_t *pte_alloc_one_kernel(struct mm_struct *mm,
> - unsigned long address)
> +pte_t *pte_alloc_one_kernel(struct mm_struct *mm)
> {
> struct page *page = alloc_page(GFP_KERNEL | __GFP_ZERO);
> pte_t *pte = NULL;
> @@ -2933,8 +2932,7 @@ pte_t *pte_alloc_one_kernel(struct mm_struct *mm,
> return pte;
> }
>
> -pgtable_t pte_alloc_one(struct mm_struct *mm,
> - unsigned long address)
> +pgtable_t pte_alloc_one(struct mm_struct *mm)
> {
> struct page *page = alloc_page(GFP_KERNEL | __GFP_ZERO);
> if (!page)
> diff --git a/arch/sparc/mm/srmmu.c b/arch/sparc/mm/srmmu.c
> index be9cb0065179..ce67a96e70c3 100644
> --- a/arch/sparc/mm/srmmu.c
> +++ b/arch/sparc/mm/srmmu.c
> @@ -364,12 +364,12 @@ pgd_t *get_pgd_fast(void)
> * Alignments up to the page size are the same for physical and virtual
> * addresses of the nocache area.
> */
> -pgtable_t pte_alloc_one(struct mm_struct *mm, unsigned long address)
> +pgtable_t pte_alloc_one(struct mm_struct *mm)
> {
> unsigned long pte;
> struct page *page;
>
> - if ((pte = (unsigned long)pte_alloc_one_kernel(mm, address)) == 0)
> + if ((pte = (unsigned long)pte_alloc_one_kernel(mm)) == 0)
> return NULL;
> page = pfn_to_page(__nocache_pa(pte) >> PAGE_SHIFT);
> if (!pgtable_page_ctor(page)) {
> diff --git a/arch/um/kernel/mem.c b/arch/um/kernel/mem.c
> index 3c0e470ea646..1f277191fbf3 100644
> --- a/arch/um/kernel/mem.c
> +++ b/arch/um/kernel/mem.c
> @@ -197,7 +197,7 @@ void pgd_free(struct mm_struct *mm, pgd_t *pgd)
> free_page((unsigned long) pgd);
> }
>
> -pte_t *pte_alloc_one_kernel(struct mm_struct *mm, unsigned long address)
> +pte_t *pte_alloc_one_kernel(struct mm_struct *mm)
> {
> pte_t *pte;
>
> @@ -205,7 +205,7 @@ pte_t *pte_alloc_one_kernel(struct mm_struct *mm, unsigned long address)
> return pte;
> }
>
> -pgtable_t pte_alloc_one(struct mm_struct *mm, unsigned long address)
> +pgtable_t pte_alloc_one(struct mm_struct *mm)
> {
> struct page *pte;
>
> diff --git a/arch/unicore32/include/asm/pgalloc.h b/arch/unicore32/include/asm/pgalloc.h
> index f0fdb268f8f2..7cceabecf4e3 100644
> --- a/arch/unicore32/include/asm/pgalloc.h
> +++ b/arch/unicore32/include/asm/pgalloc.h
> @@ -34,7 +34,7 @@ extern void free_pgd_slow(struct mm_struct *mm, pgd_t *pgd);
> * Allocate one PTE table.
> */
> static inline pte_t *
> -pte_alloc_one_kernel(struct mm_struct *mm, unsigned long addr)
> +pte_alloc_one_kernel(struct mm_struct *mm)
> {
> pte_t *pte;
>
> @@ -46,7 +46,7 @@ pte_alloc_one_kernel(struct mm_struct *mm, unsigned long addr)
> }
>
> static inline pgtable_t
> -pte_alloc_one(struct mm_struct *mm, unsigned long addr)
> +pte_alloc_one(struct mm_struct *mm)
> {
> struct page *pte;
>
> diff --git a/arch/x86/include/asm/pgalloc.h b/arch/x86/include/asm/pgalloc.h
> index fbd578daa66e..5068e85165b2 100644
> --- a/arch/x86/include/asm/pgalloc.h
> +++ b/arch/x86/include/asm/pgalloc.h
> @@ -47,8 +47,8 @@ extern gfp_t __userpte_alloc_gfp;
> extern pgd_t *pgd_alloc(struct mm_struct *);
> extern void pgd_free(struct mm_struct *mm, pgd_t *pgd);
>
> -extern pte_t *pte_alloc_one_kernel(struct mm_struct *, unsigned long);
> -extern pgtable_t pte_alloc_one(struct mm_struct *, unsigned long);
> +extern pte_t *pte_alloc_one_kernel(struct mm_struct *);
> +extern pgtable_t pte_alloc_one(struct mm_struct *);
>
> /* Should really implement gc for free page table pages. This could be
> done with a reference count in struct page. */
> diff --git a/arch/x86/mm/pgtable.c b/arch/x86/mm/pgtable.c
> index 089e78c4effd..a2eff247377b 100644
> --- a/arch/x86/mm/pgtable.c
> +++ b/arch/x86/mm/pgtable.c
> @@ -23,12 +23,12 @@ EXPORT_SYMBOL(physical_mask);
>
> gfp_t __userpte_alloc_gfp = PGALLOC_GFP | PGALLOC_USER_GFP;
>
> -pte_t *pte_alloc_one_kernel(struct mm_struct *mm, unsigned long address)
> +pte_t *pte_alloc_one_kernel(struct mm_struct *mm)
> {
> return (pte_t *)__get_free_page(PGALLOC_GFP & ~__GFP_ACCOUNT);
> }
>
> -pgtable_t pte_alloc_one(struct mm_struct *mm, unsigned long address)
> +pgtable_t pte_alloc_one(struct mm_struct *mm)
> {
> struct page *pte;
>
> diff --git a/arch/xtensa/include/asm/pgalloc.h b/arch/xtensa/include/asm/pgalloc.h
> index 1065bc8bcae5..b3b388ff2f01 100644
> --- a/arch/xtensa/include/asm/pgalloc.h
> +++ b/arch/xtensa/include/asm/pgalloc.h
> @@ -38,8 +38,7 @@ static inline void pgd_free(struct mm_struct *mm, pgd_t *pgd)
> free_page((unsigned long)pgd);
> }
>
> -static inline pte_t *pte_alloc_one_kernel(struct mm_struct *mm,
> - unsigned long address)
> +static inline pte_t *pte_alloc_one_kernel(struct mm_struct *mm)
> {
> pte_t *ptep;
> int i;
> @@ -52,13 +51,12 @@ static inline pte_t *pte_alloc_one_kernel(struct mm_struct *mm,
> return ptep;
> }
>
> -static inline pgtable_t pte_alloc_one(struct mm_struct *mm,
> - unsigned long addr)
> +static inline pgtable_t pte_alloc_one(struct mm_struct *mm)
> {
> pte_t *pte;
> struct page *page;
>
> - pte = pte_alloc_one_kernel(mm, addr);
> + pte = pte_alloc_one_kernel(mm);
> if (!pte)
> return NULL;
> page = virt_to_page(pte);
> diff --git a/include/linux/mm.h b/include/linux/mm.h
> index 0416a7204be3..89c2b1739a69 100644
> --- a/include/linux/mm.h
> +++ b/include/linux/mm.h
> @@ -1789,8 +1789,8 @@ static inline void mm_inc_nr_ptes(struct mm_struct *mm) {}
> static inline void mm_dec_nr_ptes(struct mm_struct *mm) {}
> #endif
>
> -int __pte_alloc(struct mm_struct *mm, pmd_t *pmd, unsigned long address);
> -int __pte_alloc_kernel(pmd_t *pmd, unsigned long address);
> +int __pte_alloc(struct mm_struct *mm, pmd_t *pmd);
> +int __pte_alloc_kernel(pmd_t *pmd);
>
> /*
> * The following ifdef needed to get the 4level-fixup.h header to work.
> @@ -1928,18 +1928,17 @@ static inline void pgtable_page_dtor(struct page *page)
> pte_unmap(pte); \
> } while (0)
>
> -#define pte_alloc(mm, pmd, address) \
> - (unlikely(pmd_none(*(pmd))) && __pte_alloc(mm, pmd, address))
> +#define pte_alloc(mm, pmd) (unlikely(pmd_none(*(pmd))) && __pte_alloc(mm, pmd))
>
> #define pte_alloc_map(mm, pmd, address) \
> - (pte_alloc(mm, pmd, address) ? NULL : pte_offset_map(pmd, address))
> + (pte_alloc(mm, pmd) ? NULL : pte_offset_map(pmd, address))
>
> #define pte_alloc_map_lock(mm, pmd, address, ptlp) \
> - (pte_alloc(mm, pmd, address) ? \
> + (pte_alloc(mm, pmd) ? \
> NULL : pte_offset_map_lock(mm, pmd, address, ptlp))
>
> #define pte_alloc_kernel(pmd, address) \
> - ((unlikely(pmd_none(*(pmd))) && __pte_alloc_kernel(pmd, address))? \
> + ((unlikely(pmd_none(*(pmd))) && __pte_alloc_kernel(pmd))? \
> NULL: pte_offset_kernel(pmd, address))
>
> #if USE_SPLIT_PMD_PTLOCKS
> diff --git a/mm/huge_memory.c b/mm/huge_memory.c
> index 00704060b7f7..fd7e8714e5a1 100644
> --- a/mm/huge_memory.c
> +++ b/mm/huge_memory.c
> @@ -558,7 +558,7 @@ static vm_fault_t __do_huge_pmd_anonymous_page(struct vm_fault *vmf,
> return VM_FAULT_FALLBACK;
> }
>
> - pgtable = pte_alloc_one(vma->vm_mm, haddr);
> + pgtable = pte_alloc_one(vma->vm_mm);
> if (unlikely(!pgtable)) {
> ret = VM_FAULT_OOM;
> goto release;
> @@ -683,7 +683,7 @@ vm_fault_t do_huge_pmd_anonymous_page(struct vm_fault *vmf)
> struct page *zero_page;
> bool set;
> vm_fault_t ret;
> - pgtable = pte_alloc_one(vma->vm_mm, haddr);
> + pgtable = pte_alloc_one(vma->vm_mm);
> if (unlikely(!pgtable))
> return VM_FAULT_OOM;
> zero_page = mm_get_huge_zero_page(vma->vm_mm);
> @@ -772,7 +772,7 @@ vm_fault_t vmf_insert_pfn_pmd(struct vm_area_struct *vma, unsigned long addr,
> return VM_FAULT_SIGBUS;
>
> if (arch_needs_pgtable_deposit()) {
> - pgtable = pte_alloc_one(vma->vm_mm, addr);
> + pgtable = pte_alloc_one(vma->vm_mm);
> if (!pgtable)
> return VM_FAULT_OOM;
> }
> @@ -910,7 +910,7 @@ int copy_huge_pmd(struct mm_struct *dst_mm, struct mm_struct *src_mm,
> if (!vma_is_anonymous(vma))
> return 0;
>
> - pgtable = pte_alloc_one(dst_mm, addr);
> + pgtable = pte_alloc_one(dst_mm);
> if (unlikely(!pgtable))
> goto out;
>
> diff --git a/mm/kasan/kasan_init.c b/mm/kasan/kasan_init.c
> index 7a2a2f13f86f..272849cd2007 100644
> --- a/mm/kasan/kasan_init.c
> +++ b/mm/kasan/kasan_init.c
> @@ -121,7 +121,7 @@ static int __ref zero_pmd_populate(pud_t *pud, unsigned long addr,
> pte_t *p;
>
> if (slab_is_available())
> - p = pte_alloc_one_kernel(&init_mm, addr);
> + p = pte_alloc_one_kernel(&init_mm);
> else
> p = early_alloc(PAGE_SIZE, NUMA_NO_NODE);
> if (!p)
> diff --git a/mm/memory.c b/mm/memory.c
> index c467102a5cbc..3afdcf38993d 100644
> --- a/mm/memory.c
> +++ b/mm/memory.c
> @@ -647,10 +647,10 @@ void free_pgtables(struct mmu_gather *tlb, struct vm_area_struct *vma,
> }
> }
>
> -int __pte_alloc(struct mm_struct *mm, pmd_t *pmd, unsigned long address)
> +int __pte_alloc(struct mm_struct *mm, pmd_t *pmd)
> {
> spinlock_t *ptl;
> - pgtable_t new = pte_alloc_one(mm, address);
> + pgtable_t new = pte_alloc_one(mm);
> if (!new)
> return -ENOMEM;
>
> @@ -681,9 +681,9 @@ int __pte_alloc(struct mm_struct *mm, pmd_t *pmd, unsigned long address)
> return 0;
> }
>
> -int __pte_alloc_kernel(pmd_t *pmd, unsigned long address)
> +int __pte_alloc_kernel(pmd_t *pmd)
> {
> - pte_t *new = pte_alloc_one_kernel(&init_mm, address);
> + pte_t *new = pte_alloc_one_kernel(&init_mm);
> if (!new)
> return -ENOMEM;
>
> @@ -3139,7 +3139,7 @@ static vm_fault_t do_anonymous_page(struct vm_fault *vmf)
> *
> * Here we only have down_read(mmap_sem).
> */
> - if (pte_alloc(vma->vm_mm, vmf->pmd, vmf->address))
> + if (pte_alloc(vma->vm_mm, vmf->pmd))
> return VM_FAULT_OOM;
>
> /* See the comment in pte_alloc_one_map() */
> @@ -3286,7 +3286,7 @@ static vm_fault_t pte_alloc_one_map(struct vm_fault *vmf)
> pmd_populate(vma->vm_mm, vmf->pmd, vmf->prealloc_pte);
> spin_unlock(vmf->ptl);
> vmf->prealloc_pte = NULL;
> - } else if (unlikely(pte_alloc(vma->vm_mm, vmf->pmd, vmf->address))) {
> + } else if (unlikely(pte_alloc(vma->vm_mm, vmf->pmd))) {
> return VM_FAULT_OOM;
> }
> map_pte:
> @@ -3365,7 +3365,7 @@ static vm_fault_t do_set_pmd(struct vm_fault *vmf, struct page *page)
> * related to pte entry. Use the preallocated table for that.
> */
> if (arch_needs_pgtable_deposit() && !vmf->prealloc_pte) {
> - vmf->prealloc_pte = pte_alloc_one(vma->vm_mm, vmf->address);
> + vmf->prealloc_pte = pte_alloc_one(vma->vm_mm);
> if (!vmf->prealloc_pte)
> return VM_FAULT_OOM;
> smp_wmb(); /* See comment in __pte_alloc() */
> @@ -3603,8 +3603,7 @@ static vm_fault_t do_fault_around(struct vm_fault *vmf)
> start_pgoff + nr_pages - 1);
>
> if (pmd_none(*vmf->pmd)) {
> - vmf->prealloc_pte = pte_alloc_one(vmf->vma->vm_mm,
> - vmf->address);
> + vmf->prealloc_pte = pte_alloc_one(vmf->vma->vm_mm);
> if (!vmf->prealloc_pte)
> goto out;
> smp_wmb(); /* See comment in __pte_alloc() */
> diff --git a/mm/migrate.c b/mm/migrate.c
> index 84381b55b2bd..3080b0626026 100644
> --- a/mm/migrate.c
> +++ b/mm/migrate.c
> @@ -2605,7 +2605,7 @@ static void migrate_vma_insert_page(struct migrate_vma *migrate,
> *
> * Here we only have down_read(mmap_sem).
> */
> - if (pte_alloc(mm, pmdp, addr))
> + if (pte_alloc(mm, pmdp))
> goto abort;
>
> /* See the comment in pte_alloc_one_map() */
> diff --git a/mm/mremap.c b/mm/mremap.c
> index 5c2e18505f75..9e68a02a52b1 100644
> --- a/mm/mremap.c
> +++ b/mm/mremap.c
> @@ -240,7 +240,7 @@ unsigned long move_page_tables(struct vm_area_struct *vma,
> if (pmd_trans_unstable(old_pmd))
> continue;
> }
> - if (pte_alloc(new_vma->vm_mm, new_pmd, new_addr))
> + if (pte_alloc(new_vma->vm_mm, new_pmd))
> break;
> next = (new_addr + PMD_SIZE) & PMD_MASK;
> if (extent > next - new_addr)
> diff --git a/mm/userfaultfd.c b/mm/userfaultfd.c
> index 5029f241908f..f05c8bc38ca5 100644
> --- a/mm/userfaultfd.c
> +++ b/mm/userfaultfd.c
> @@ -513,7 +513,7 @@ static __always_inline ssize_t __mcopy_atomic(struct mm_struct *dst_mm,
> break;
> }
> if (unlikely(pmd_none(dst_pmdval)) &&
> - unlikely(__pte_alloc(dst_mm, dst_pmd, dst_addr))) {
> + unlikely(__pte_alloc(dst_mm, dst_pmd))) {
> err = -ENOMEM;
> break;
> }
> diff --git a/virt/kvm/arm/mmu.c b/virt/kvm/arm/mmu.c
> index ed162a6c57c5..3f8180414301 100644
> --- a/virt/kvm/arm/mmu.c
> +++ b/virt/kvm/arm/mmu.c
> @@ -628,7 +628,7 @@ static int create_hyp_pmd_mappings(pud_t *pud, unsigned long start,
> BUG_ON(pmd_sect(*pmd));
>
> if (pmd_none(*pmd)) {
> - pte = pte_alloc_one_kernel(NULL, addr);
> + pte = pte_alloc_one_kernel(NULL);
> if (!pte) {
> kvm_err("Cannot allocate Hyp pte\n");
> return -ENOMEM;
^ permalink raw reply
* Re: [PATCH v2 1/2] treewide: remove unused address argument from pte_alloc functions
From: SF Markus Elfring @ 2018-10-12 18:51 UTC (permalink / raw)
To: Joel Fernandes, kernel-janitors
Cc: linux-mips, Rich Felker, linux-ia64, linux-sh, Peter Zijlstra,
Catalin Marinas, Dave Hansen, Will Deacon, Michal Hocko, linux-mm,
Lokesh Gidra, sparclinux, linux-riscv, kvmarm, Jonas Bonn,
linux-s390, Daniel Colascione, Yoshinori Sato, Max Filippov,
linux-hexagon, Helge Deller,
maintainer:X86 ARCHITECTURE (32-BIT AND 64-BIT), Hugh Dickins,
James E. J. Bottomley, kasan-dev, Ingo Molnar, Geert Uytterhoeven,
Andrey Ryabinin, linux-snps-arc, kernel-team, Sam Creasey,
Fenghua Yu, Jeff Dike, linux-um, Stefan Kristiansson,
Julia Lawall, linux-m68k, openrisc, Borislav Petkov,
Andy Lutomirski, nios2-dev, Kirill A. Shutemov, Stafford Horne,
Guan Xuetao, linux-arm-kernel, Chris Zankel, Tony Luck,
Richard Weinberger, linux-parisc, pantin, linux-kernel,
Minchan Kim, Thomas Gleixner, linux-alpha, Ley Foon Tan,
Andrew Morton, linuxppc-dev, David S. Miller
In-Reply-To: <20181012013756.11285-1-joel@joelfernandes.org>
> The changes were obtained by applying the following Coccinelle script.
A bit of clarification happened for its implementation details.
https://systeme.lip6.fr/pipermail/cocci/2018-October/005374.html
I have taken also another look at the following SmPL code.
> identifier fn =~
> "^(__pte_alloc|pte_alloc_one|pte_alloc|__pte_alloc_kernel|pte_alloc_one_kernel)$";
I suggest to adjust the regular expression for this constraint
and in subsequent SmPL rules.
"^(?:pte_alloc(?:_one(?:_kernel)?)?|__pte_alloc(?:_kernel)?)$";
> (
> - T3 fn(T1 E1, T2 E2);
> + T3 fn(T1 E1);
> |
> - T3 fn(T1 E1, T2 E2, T4 E4);
> + T3 fn(T1 E1, T2 E2);
> )
I propose to take an other SmPL disjunction into account here.
T3 fn(T1 E1,
(
- T2 E2
| T2 E2,
- T4 E4
) );
> (
> - #define fn(a, b, c)@p e
> + #define fn(a, b) e
> |
> - #define fn(a, b)@p e
> + #define fn(a) e
> )
How do you think about to omit the metavariable “position p” here?
Regards,
Markus
^ permalink raw reply
* Re: [PATCH v2 1/2] treewide: remove unused address argument from pte_alloc functions
From: Joel Fernandes @ 2018-10-12 19:42 UTC (permalink / raw)
To: SF Markus Elfring
Cc: linux-mips, Rich Felker, linux-ia64, linux-sh, Peter Zijlstra,
Catalin Marinas, Dave Hansen, kernel-janitors, Michal Hocko,
linux-mm, Lokesh Gidra, sparclinux, linux-riscv, kvmarm,
Jonas Bonn, linux-s390, Daniel Colascione, Yoshinori Sato,
Max Filippov, linux-hexagon, Helge Deller,
maintainer:X86 ARCHITECTURE (32-BIT AND 64-BIT), Hugh Dickins,
James E. J. Bottomley, kasan-dev, Ingo Molnar, Geert Uytterhoeven,
Andrey Ryabinin, linux-snps-arc, kernel-team, Sam Creasey,
Fenghua Yu, Will Deacon, Jeff Dike, linux-um, Stefan Kristiansson,
Julia Lawall, linux-m68k, openrisc, Borislav Petkov,
Andy Lutomirski, nios2-dev, Kirill A. Shutemov, Stafford Horne,
Guan Xuetao, linux-arm-kernel, Chris Zankel, Tony Luck,
Richard Weinberger, linux-parisc, pantin, linux-kernel,
Minchan Kim, Thomas Gleixner, linux-alpha, Ley Foon Tan,
Andrew Morton, linuxppc-dev, David S. Miller
In-Reply-To: <03b524f3-5f3a-baa0-2254-9c588103d2d6@users.sourceforge.net>
On Fri, Oct 12, 2018 at 08:51:45PM +0200, SF Markus Elfring wrote:
> > The changes were obtained by applying the following Coccinelle script.
>
> A bit of clarification happened for its implementation details.
> https://systeme.lip6.fr/pipermail/cocci/2018-October/005374.html
>
> I have taken also another look at the following SmPL code.
>
>
> > identifier fn =~
> > "^(__pte_alloc|pte_alloc_one|pte_alloc|__pte_alloc_kernel|pte_alloc_one_kernel)$";
>
> I suggest to adjust the regular expression for this constraint
> and in subsequent SmPL rules.
> "^(?:pte_alloc(?:_one(?:_kernel)?)?|__pte_alloc(?:_kernel)?)$";
Sure it looks more clever, but why? Ugh that's harder to read and confusing.
> > (
> > - T3 fn(T1 E1, T2 E2);
> > + T3 fn(T1 E1);
> > |
> > - T3 fn(T1 E1, T2 E2, T4 E4);
> > + T3 fn(T1 E1, T2 E2);
> > )
>
> I propose to take an other SmPL disjunction into account here.
>
> T3 fn(T1 E1,
> (
> - T2 E2
> | T2 E2,
> - T4 E4
> ) );
Again this is confusing. It makes one think that maybe the second argument
can also be removed and requires careful observation that the ");" follows.
> > (
> > - #define fn(a, b, c)@p e
> > + #define fn(a, b) e
> > |
> > - #define fn(a, b)@p e
> > + #define fn(a) e
> > )
>
> How do you think about to omit the metavariable “position p” here?
Right, I don't need it in this case. But the script works either way.
I like to take more of a problem solving approach that makes sense, than
aiming for perfection, after all this is a useful script that we do not
need to check in once we finish with it.
- Joel
^ permalink raw reply
* Re: [PATCH v2 2/2] mm: speed up mremap by 500x on large regions
From: Kirill A. Shutemov @ 2018-10-12 21:33 UTC (permalink / raw)
To: Joel Fernandes
Cc: linux-mips, Rich Felker, linux-ia64, linux-sh, Peter Zijlstra,
Catalin Marinas, Dave Hansen, Will Deacon, mhocko, linux-mm,
lokeshgidra, sparclinux, linux-riscv, elfring, Jonas Bonn,
linux-s390, dancol, Yoshinori Sato, linux-xtensa, linux-hexagon,
Helge Deller, maintainer:X86 ARCHITECTURE (32-BIT AND 64-BIT),
hughd, James E.J. Bottomley, kasan-dev, kvmarm, Ingo Molnar,
Geert Uytterhoeven, Andrey Ryabinin, linux-snps-arc, kernel-team,
Sam Creasey, Fenghua Yu, Jeff Dike, linux-um, Stefan Kristiansson,
Julia Lawall, linux-m68k, openrisc, Borislav Petkov,
Andy Lutomirski, nios2-dev, Stafford Horne, Guan Xuetao,
linux-arm-kernel, Chris Zankel, Tony Luck, Richard Weinberger,
linux-parisc, pantin, Max Filippov, linux-kernel, minchan,
Thomas Gleixner, linux-alpha, Ley Foon Tan, akpm, linuxppc-dev,
David S. Miller
In-Reply-To: <20181012165719.GE223066@joelaf.mtv.corp.google.com>
On Fri, Oct 12, 2018 at 09:57:19AM -0700, Joel Fernandes wrote:
> On Fri, Oct 12, 2018 at 04:19:46PM +0300, Kirill A. Shutemov wrote:
> > On Fri, Oct 12, 2018 at 05:50:46AM -0700, Joel Fernandes wrote:
> > > On Fri, Oct 12, 2018 at 02:30:56PM +0300, Kirill A. Shutemov wrote:
> > > > On Thu, Oct 11, 2018 at 06:37:56PM -0700, Joel Fernandes (Google) wrote:
> > > > > Android needs to mremap large regions of memory during memory management
> > > > > related operations. The mremap system call can be really slow if THP is
> > > > > not enabled. The bottleneck is move_page_tables, which is copying each
> > > > > pte at a time, and can be really slow across a large map. Turning on THP
> > > > > may not be a viable option, and is not for us. This patch speeds up the
> > > > > performance for non-THP system by copying at the PMD level when possible.
> > > > >
> > > > > The speed up is three orders of magnitude. On a 1GB mremap, the mremap
> > > > > completion times drops from 160-250 millesconds to 380-400 microseconds.
> > > > >
> > > > > Before:
> > > > > Total mremap time for 1GB data: 242321014 nanoseconds.
> > > > > Total mremap time for 1GB data: 196842467 nanoseconds.
> > > > > Total mremap time for 1GB data: 167051162 nanoseconds.
> > > > >
> > > > > After:
> > > > > Total mremap time for 1GB data: 385781 nanoseconds.
> > > > > Total mremap time for 1GB data: 388959 nanoseconds.
> > > > > Total mremap time for 1GB data: 402813 nanoseconds.
> > > > >
> > > > > Incase THP is enabled, the optimization is skipped. I also flush the
> > > > > tlb every time we do this optimization since I couldn't find a way to
> > > > > determine if the low-level PTEs are dirty. It is seen that the cost of
> > > > > doing so is not much compared the improvement, on both x86-64 and arm64.
> > > >
> > > > I looked into the code more and noticed move_pte() helper called from
> > > > move_ptes(). It changes PTE entry to suite new address.
> > > >
> > > > It is only defined in non-trivial way on Sparc. I don't know much about
> > > > Sparc and it's hard for me to say if the optimization will break anything
> > > > there.
> > >
> > > Sparc's move_pte seems to be flushing the D-cache to prevent aliasing. It is
> > > not modifying the PTE itself AFAICS:
> > >
> > > #ifdef DCACHE_ALIASING_POSSIBLE
> > > #define __HAVE_ARCH_MOVE_PTE
> > > #define move_pte(pte, prot, old_addr, new_addr) \
> > > ({ \
> > > pte_t newpte = (pte); \
> > > if (tlb_type != hypervisor && pte_present(pte)) { \
> > > unsigned long this_pfn = pte_pfn(pte); \
> > > \
> > > if (pfn_valid(this_pfn) && \
> > > (((old_addr) ^ (new_addr)) & (1 << 13))) \
> > > flush_dcache_page_all(current->mm, \
> > > pfn_to_page(this_pfn)); \
> > > } \
> > > newpte; \
> > > })
> > > #endif
> > >
> > > If its an issue, then how do transparent huge pages work on Sparc? I don't
> > > see the huge page code (move_huge_pages) during mremap doing anything special
> > > for Sparc architecture when moving PMDs..
> >
> > My *guess* is that it will work fine on Sparc as it apprarently it only
> > cares about change in bit 13 of virtual address. It will never happen for
> > huge pages or when PTE page tables move.
> >
> > But I just realized that the problem is bigger: since we pass new_addr to
> > the set_pte_at() we would need to audit all implementations that they are
> > safe with just moving PTE page table.
> >
> > I would rather go with per-architecture enabling. It's much safer.
>
> I'm Ok with the per-arch enabling, I agree its safer. So I should be adding a
> a new __HAVE_ARCH_MOVE_PMD right, or did you have a better name for that?
I believe Kconfig option is more cononical way to do this nowadays.
So CONFIG_HAVE_ARCH_MOVE_PMD, I guess. Or CONFIG_HAVE_MOVE_PMD.
An arch that supports it would select the option.
> Also, do you feel we should still need to remove the address argument from
> set_pte_alloc? Or should we leave that alone if we do per-arch?
> I figure I spent a bunch of time on that already anyway, and its a clean up
> anyway, so may as well do it. But perhaps that "pte_alloc cleanup" can then
> be a separate patch independent of this series?
Yeah. The cleanup makes sense anyway.
--
Kirill A. Shutemov
^ permalink raw reply
* Re: [PATCH v2 2/2] mm: speed up mremap by 500x on large regions
From: Kirill A. Shutemov @ 2018-10-12 21:40 UTC (permalink / raw)
To: Anton Ivanov
Cc: linux-mips, Rich Felker, linux-ia64, linux-sh, Peter Zijlstra,
Catalin Marinas, Dave Hansen, Will Deacon, mhocko, linux-mm,
lokeshgidra, Joel Fernandes (Google), linux-riscv, elfring,
Jonas Bonn, linux-s390, dancol, Yoshinori Sato, Max Filippov,
linux-hexagon, Helge Deller,
maintainer:X86 ARCHITECTURE 32-BIT AND 64-BIT, hughd,
James E.J. Bottomley, kasan-dev, kvmarm, Ingo Molnar,
Geert Uytterhoeven, Andrey Ryabinin, linux-snps-arc, kernel-team,
Sam Creasey, linux-xtensa, Jeff Dike, linux-alpha, linux-um,
Stefan Kristiansson, Julia Lawall, linux-m68k, openrisc,
Borislav Petkov, Andy Lutomirski, nios2-dev, Stafford Horne,
Guan Xuetao, linux-arm-kernel, Chris Zankel, Tony Luck,
linux-parisc, pantin, linux-kernel, Fenghua Yu, minchan,
Thomas Gleixner, Richard Weinberger, sparclinux, akpm,
linuxppc-dev, David S. Miller, Ley Foon Tan
In-Reply-To: <97cb3fe1-7bc1-12ff-d602-56c72a5496c5@kot-begemot.co.uk>
On Fri, Oct 12, 2018 at 05:42:24PM +0100, Anton Ivanov wrote:
>
> On 10/12/18 3:48 PM, Anton Ivanov wrote:
> > On 12/10/2018 15:37, Kirill A. Shutemov wrote:
> > > On Fri, Oct 12, 2018 at 03:09:49PM +0100, Anton Ivanov wrote:
> > > > On 10/12/18 2:37 AM, Joel Fernandes (Google) wrote:
> > > > > Android needs to mremap large regions of memory during
> > > > > memory management
> > > > > related operations. The mremap system call can be really
> > > > > slow if THP is
> > > > > not enabled. The bottleneck is move_page_tables, which is copying each
> > > > > pte at a time, and can be really slow across a large map.
> > > > > Turning on THP
> > > > > may not be a viable option, and is not for us. This patch
> > > > > speeds up the
> > > > > performance for non-THP system by copying at the PMD level
> > > > > when possible.
> > > > >
> > > > > The speed up is three orders of magnitude. On a 1GB mremap, the mremap
> > > > > completion times drops from 160-250 millesconds to 380-400
> > > > > microseconds.
> > > > >
> > > > > Before:
> > > > > Total mremap time for 1GB data: 242321014 nanoseconds.
> > > > > Total mremap time for 1GB data: 196842467 nanoseconds.
> > > > > Total mremap time for 1GB data: 167051162 nanoseconds.
> > > > >
> > > > > After:
> > > > > Total mremap time for 1GB data: 385781 nanoseconds.
> > > > > Total mremap time for 1GB data: 388959 nanoseconds.
> > > > > Total mremap time for 1GB data: 402813 nanoseconds.
> > > > >
> > > > > Incase THP is enabled, the optimization is skipped. I also flush the
> > > > > tlb every time we do this optimization since I couldn't find a way to
> > > > > determine if the low-level PTEs are dirty. It is seen that the cost of
> > > > > doing so is not much compared the improvement, on both
> > > > > x86-64 and arm64.
> > > > >
> > > > > Cc: minchan@kernel.org
> > > > > Cc: pantin@google.com
> > > > > Cc: hughd@google.com
> > > > > Cc: lokeshgidra@google.com
> > > > > Cc: dancol@google.com
> > > > > Cc: mhocko@kernel.org
> > > > > Cc: kirill@shutemov.name
> > > > > Cc: akpm@linux-foundation.org
> > > > > Signed-off-by: Joel Fernandes (Google) <joel@joelfernandes.org>
> > > > > ---
> > > > > mm/mremap.c | 62
> > > > > +++++++++++++++++++++++++++++++++++++++++++++++++++++
> > > > > 1 file changed, 62 insertions(+)
> > > > >
> > > > > diff --git a/mm/mremap.c b/mm/mremap.c
> > > > > index 9e68a02a52b1..d82c485822ef 100644
> > > > > --- a/mm/mremap.c
> > > > > +++ b/mm/mremap.c
> > > > > @@ -191,6 +191,54 @@ static void move_ptes(struct
> > > > > vm_area_struct *vma, pmd_t *old_pmd,
> > > > > drop_rmap_locks(vma);
> > > > > }
> > > > > +static bool move_normal_pmd(struct vm_area_struct *vma,
> > > > > unsigned long old_addr,
> > > > > + unsigned long new_addr, unsigned long old_end,
> > > > > + pmd_t *old_pmd, pmd_t *new_pmd, bool *need_flush)
> > > > > +{
> > > > > + spinlock_t *old_ptl, *new_ptl;
> > > > > + struct mm_struct *mm = vma->vm_mm;
> > > > > +
> > > > > + if ((old_addr & ~PMD_MASK) || (new_addr & ~PMD_MASK)
> > > > > + || old_end - old_addr < PMD_SIZE)
> > > > > + return false;
> > > > > +
> > > > > + /*
> > > > > + * The destination pmd shouldn't be established, free_pgtables()
> > > > > + * should have release it.
> > > > > + */
> > > > > + if (WARN_ON(!pmd_none(*new_pmd)))
> > > > > + return false;
> > > > > +
> > > > > + /*
> > > > > + * We don't have to worry about the ordering of src and dst
> > > > > + * ptlocks because exclusive mmap_sem prevents deadlock.
> > > > > + */
> > > > > + old_ptl = pmd_lock(vma->vm_mm, old_pmd);
> > > > > + if (old_ptl) {
> > > > > + pmd_t pmd;
> > > > > +
> > > > > + new_ptl = pmd_lockptr(mm, new_pmd);
> > > > > + if (new_ptl != old_ptl)
> > > > > + spin_lock_nested(new_ptl, SINGLE_DEPTH_NESTING);
> > > > > +
> > > > > + /* Clear the pmd */
> > > > > + pmd = *old_pmd;
> > > > > + pmd_clear(old_pmd);
> > > > > +
> > > > > + VM_BUG_ON(!pmd_none(*new_pmd));
> > > > > +
> > > > > + /* Set the new pmd */
> > > > > + set_pmd_at(mm, new_addr, new_pmd, pmd);
> > > > UML does not have set_pmd_at at all
> > > Every architecture does. :)
> >
> > I tried to build it patching vs 4.19-rc before I made this statement and
> > ran into that.
> >
> > Presently it does not.
> >
> > https://elixir.bootlin.com/linux/v4.19-rc7/ident/set_pmd_at - UML is not
> > on the list.
>
> Once this problem as well as the omissions in the include changes for UML in
> patch one have been fixed it appears to be working.
>
> What it needs is attached.
Well, the optization is only suitable for arch that has 3 or more levels
of page tables. Otherwise it will not have [non-folded] pmd.
And in this case arch/um already should have set_pmd_at(), see
3_LEVEL_PGTABLES.
To port on 2-level paging, it has to be handled on pgd level. It
complicates the code and will not bring much value.
--
Kirill A. Shutemov
^ permalink raw reply
* Re: [PATCH 2/2] powerpc/pseries: Add driver for PAPR SCM regions
From: Dan Williams @ 2018-10-12 22:36 UTC (permalink / raw)
To: Oliver O'Halloran; +Cc: Nathan Fontenot, linuxppc-dev, linux-nvdimm
In-Reply-To: <20181010060812.20068-3-oohall@gmail.com>
On Tue, Oct 9, 2018 at 11:21 PM Oliver O'Halloran <oohall@gmail.com> wrote:
>
> Adds a driver that implements support for enabling and accessing PAPR
> SCM regions. Unfortunately due to how the PAPR interface works we can't
> use the existing of_pmem driver (yet) because:
>
> a) The guest is required to use the H_SCM_BIND_MEM h-call to add
> add the SCM region to it's physical address space, and
> b) There is currently no mechanism for relating a bare of_pmem region
> to the backing DIMM (or not-a-DIMM for our case).
>
> Both of these are easily handled by rolling the functionality into a
> seperate driver so here we are...
>
> Signed-off-by: Oliver O'Halloran <oohall@gmail.com>
> ---
> The alternative implementation here is that we have the pseries code
> do the h-calls and craft a pmem-region@<addr> node based on that.
> However, that doesn't solve b) and mpe has expressed his dislike of
> adding new stuff to the DT at runtime so i'd say that's a non-starter.
> ---
> arch/powerpc/platforms/pseries/Kconfig | 7 +
> arch/powerpc/platforms/pseries/Makefile | 1 +
> arch/powerpc/platforms/pseries/papr_scm.c | 340 ++++++++++++++++++++++++++++++
> 3 files changed, 348 insertions(+)
> create mode 100644 arch/powerpc/platforms/pseries/papr_scm.c
>
> diff --git a/arch/powerpc/platforms/pseries/Kconfig b/arch/powerpc/platforms/pseries/Kconfig
> index 0c698fd6d491..4b0fcb80efe5 100644
> --- a/arch/powerpc/platforms/pseries/Kconfig
> +++ b/arch/powerpc/platforms/pseries/Kconfig
> @@ -140,3 +140,10 @@ config IBMEBUS
> bool "Support for GX bus based adapters"
> help
> Bus device driver for GX bus based adapters.
> +
> +config PAPR_SCM
> + depends on PPC_PSERIES && MEMORY_HOTPLUG
> + select LIBNVDIMM
> + tristate "Support for the PAPR Storage Class Memory interface"
> + help
> + Enable access to hypervisor provided storage class memory.
> diff --git a/arch/powerpc/platforms/pseries/Makefile b/arch/powerpc/platforms/pseries/Makefile
> index 892b27ced973..a43ec843c8e2 100644
> --- a/arch/powerpc/platforms/pseries/Makefile
> +++ b/arch/powerpc/platforms/pseries/Makefile
> @@ -24,6 +24,7 @@ obj-$(CONFIG_IO_EVENT_IRQ) += io_event_irq.o
> obj-$(CONFIG_LPARCFG) += lparcfg.o
> obj-$(CONFIG_IBMVIO) += vio.o
> obj-$(CONFIG_IBMEBUS) += ibmebus.o
> +obj-$(CONFIG_PAPR_SCM) += papr_scm.o
>
> ifdef CONFIG_PPC_PSERIES
> obj-$(CONFIG_SUSPEND) += suspend.o
> diff --git a/arch/powerpc/platforms/pseries/papr_scm.c b/arch/powerpc/platforms/pseries/papr_scm.c
> new file mode 100644
> index 000000000000..87d4dbc18845
> --- /dev/null
> +++ b/arch/powerpc/platforms/pseries/papr_scm.c
> @@ -0,0 +1,340 @@
> +// SPDX-License-Identifier: GPL-2.0
> +
> +#define pr_fmt(fmt) "papr-scm: " fmt
> +
> +#include <linux/of.h>
> +#include <linux/kernel.h>
> +#include <linux/module.h>
> +#include <linux/ioport.h>
> +#include <linux/slab.h>
> +#include <linux/ndctl.h>
> +#include <linux/libnvdimm.h>
> +#include <linux/platform_device.h>
> +
> +#include <asm/plpar_wrappers.h>
> +
> +#define BIND_ANY_ADDR (~0ul)
> +
> +#define PAPR_SCM_DIMM_CMD_MASK \
> + ((1ul << ND_CMD_GET_CONFIG_SIZE) | \
> + (1ul << ND_CMD_GET_CONFIG_DATA) | \
> + (1ul << ND_CMD_SET_CONFIG_DATA))
> +
> +struct papr_scm_priv {
> + struct platform_device *pdev;
> + struct device_node *dn;
> + uint32_t drc_index;
> + uint64_t blocks;
> + uint64_t block_size;
> + int metadata_size;
> +
> + uint64_t bound_addr;
> +
> + struct nvdimm_bus_descriptor bus_desc;
> + struct nvdimm_bus *bus;
> + struct nvdimm *nvdimm;
> + struct resource res;
> + struct nd_region *region;
> + struct nd_interleave_set nd_set;
> +};
> +
> +static int drc_pmem_bind(struct papr_scm_priv *p)
> +{
> + unsigned long ret[PLPAR_HCALL_BUFSIZE];
> + uint64_t rc, token;
> +
> + /*
> + * When the hypervisor cannot map all the requested memory in a single
> + * hcall it returns H_BUSY and we call again with the token until
> + * we get H_SUCCESS. Aborting the retry loop before getting H_SUCCESS
> + * leave the system in an undefined state, so we wait.
> + */
> + token = 0;
> +
> + do {
> + rc = plpar_hcall(H_SCM_BIND_MEM, ret, p->drc_index, 0,
> + p->blocks, BIND_ANY_ADDR, token);
> + token = be64_to_cpu(ret[0]);
Does plpar_hcall sleep? Should there be a cond_resched() here if not?
> + } while (rc == H_BUSY);
> +
> + if (rc) {
> + dev_err(&p->pdev->dev, "bind err: %lld\n", rc);
> + return -ENXIO;
> + }
> +
> + p->bound_addr = be64_to_cpu(ret[1]);
> +
> + dev_dbg(&p->pdev->dev, "bound drc %x to %pR\n", p->drc_index, &p->res);
> +
> + return 0;
> +}
> +
> +static int drc_pmem_unbind(struct papr_scm_priv *p)
> +{
> + unsigned long ret[PLPAR_HCALL_BUFSIZE];
> + uint64_t rc, token;
> +
> + token = 0;
> +
> + /* NB: unbind has the same retry requirements mentioned above */
> + do {
> + rc = plpar_hcall(H_SCM_UNBIND_MEM, ret, p->drc_index,
> + p->bound_addr, p->blocks, token);
> + token = be64_to_cpu(ret);
...same busy loop comment.
> + } while (rc == H_BUSY);
> +
> + if (rc)
> + dev_err(&p->pdev->dev, "unbind error: %lld\n", rc);
> +
> + return !!rc;
> +}
> +
> +static int papr_scm_meta_get(struct papr_scm_priv *p,
> + struct nd_cmd_get_config_data_hdr *hdr)
> +{
> + unsigned long data[PLPAR_HCALL_BUFSIZE];
> + int64_t ret;
> +
> + if (hdr->in_offset >= p->metadata_size || hdr->in_length != 1)
> + return -EINVAL;
> +
> + ret = plpar_hcall(H_SCM_READ_METADATA, data, p->drc_index,
> + hdr->in_offset, 1);
> +
> + if (ret == H_PARAMETER) /* bad DRC index */
> + return -ENODEV;
> + if (ret)
> + return -EINVAL; /* other invalid parameter */
> +
> + hdr->out_buf[0] = data[0] & 0xff;
> +
> + return 0;
> +}
> +
> +static int papr_scm_meta_set(struct papr_scm_priv *p,
> + struct nd_cmd_set_config_hdr *hdr)
> +{
> + int64_t ret;
> +
> + if (hdr->in_offset >= p->metadata_size || hdr->in_length != 1)
> + return -EINVAL;
> +
> + ret = plpar_hcall_norets(H_SCM_WRITE_METADATA,
> + p->drc_index, hdr->in_offset, hdr->in_buf[0], 1);
> +
> + if (ret == H_PARAMETER) /* bad DRC index */
> + return -ENODEV;
> + if (ret)
> + return -EINVAL; /* other invalid parameter */
> +
> + return 0;
> +}
> +
> +int papr_scm_ndctl(struct nvdimm_bus_descriptor *nd_desc, struct nvdimm *nvdimm,
> + unsigned int cmd, void *buf, unsigned int buf_len, int *cmd_rc)
> +{
> + struct nd_cmd_get_config_size *get_size_hdr;
> + struct papr_scm_priv *p;
> +
> + /* Only dimm-specific calls are supported atm */
> + if (!nvdimm)
> + return -EINVAL;
> +
> + p = nvdimm_provider_data(nvdimm);
> +
> + switch (cmd) {
> + case ND_CMD_GET_CONFIG_SIZE:
> + get_size_hdr = buf;
> +
> + get_size_hdr->status = 0;
> + get_size_hdr->max_xfer = 1;
> + get_size_hdr->config_size = p->metadata_size;
> + *cmd_rc = 0;
> + break;
> +
> + case ND_CMD_GET_CONFIG_DATA:
> + *cmd_rc = papr_scm_meta_get(p, buf);
> + break;
> +
> + case ND_CMD_SET_CONFIG_DATA:
> + *cmd_rc = papr_scm_meta_set(p, buf);
> + break;
> +
> + default:
> + return -EINVAL;
> + }
> +
> + dev_dbg(&p->pdev->dev, "returned with cmd_rc = %d\n", *cmd_rc);
> +
> + return 0;
> +}
> +
> +static const struct attribute_group *region_attr_groups[] = {
> + &nd_region_attribute_group,
> + &nd_device_attribute_group,
> + &nd_mapping_attribute_group,
> + &nd_numa_attribute_group,
> + NULL,
> +};
> +
> +static const struct attribute_group *bus_attr_groups[] = {
> + &nvdimm_bus_attribute_group,
> + NULL,
> +};
> +
> +static const struct attribute_group *papr_scm_dimm_groups[] = {
> + &nvdimm_attribute_group,
> + &nd_device_attribute_group,
> + NULL,
> +};
> +
> +static int papr_scm_nvdimm_init(struct papr_scm_priv *p)
> +{
> + struct device *dev = &p->pdev->dev;
> + struct nd_mapping_desc mapping;
> + struct nd_region_desc ndr_desc;
> + unsigned long dimm_flags;
> +
> + p->bus_desc.ndctl = papr_scm_ndctl;
> + p->bus_desc.module = THIS_MODULE;
> + p->bus_desc.of_node = p->pdev->dev.of_node;
> + p->bus_desc.attr_groups = bus_attr_groups;
> + p->bus_desc.provider_name = kstrdup(p->pdev->name, GFP_KERNEL);
> +
> + if (!p->bus_desc.provider_name)
> + return -ENOMEM;
> +
> + p->bus = nvdimm_bus_register(NULL, &p->bus_desc);
> + if (!p->bus) {
> + dev_err(dev, "Error creating nvdimm bus %pOF\n", p->dn);
> + return -ENXIO;
> + }
> +
> + dimm_flags = 0;
> + set_bit(NDD_ALIASING, &dimm_flags);
> +
> + p->nvdimm = nvdimm_create(p->bus, p, papr_scm_dimm_groups,
> + dimm_flags, PAPR_SCM_DIMM_CMD_MASK, 0, NULL);
Looks good, although I'm just about to push out commits that change
this function signature to take a 'security_ops' pointer. If you need
a stable branch to base this on, let me know.
> + if (!p->nvdimm) {
> + dev_err(dev, "Error creating DIMM object for %pOF\n", p->dn);
> + goto err;
> + }
> +
> + /* now add the region */
> +
> + memset(&mapping, 0, sizeof(mapping));
> + mapping.nvdimm = p->nvdimm;
> + mapping.start = p->res.start;
> + mapping.size = p->blocks * p->block_size; // XXX: potential overflow?
Ideally, should be nvdimm relative addresses. Since you have just a
single DIMM per-region I would expect this to be 0 to
nvdimm->region_size. That said I don't think anything breaks if you
make the nvdimm address equal to the system-physical address, just
wanted to point out the intent. This only matters when you have
multiple DIMMs per region and need to record the per-DIMM contribution
in the labels on each DIMM.
Other than that looks ok to me:
Acked-by: Dan Williams <dan.j.williams@intel.com>
...just the matter of what to do about function signature change.
^ permalink raw reply
* [PATCH] powerpc: Fix stackprotector detection for non-glibc toolchains
From: Michael Ellerman @ 2018-10-12 22:58 UTC (permalink / raw)
To: linuxppc-dev
If GCC is not built with glibc support then we must explicitly tell it
which register to use for TLS mode stack protector, otherwise it will
error out and the cc-option check will fail.
Signed-off-by: Michael Ellerman <mpe@ellerman.id.au>
---
arch/powerpc/Kconfig | 3 ++-
1 file changed, 2 insertions(+), 1 deletion(-)
diff --git a/arch/powerpc/Kconfig b/arch/powerpc/Kconfig
index 1888636c9eb6..3d008115fe18 100644
--- a/arch/powerpc/Kconfig
+++ b/arch/powerpc/Kconfig
@@ -180,7 +180,8 @@ config PPC
select HAVE_ARCH_SECCOMP_FILTER
select HAVE_ARCH_TRACEHOOK
select HAVE_CBPF_JIT if !PPC64
- select HAVE_STACKPROTECTOR if $(cc-option,-mstack-protector-guard=tls)
+ select HAVE_STACKPROTECTOR if PPC64 && $(cc-option,-mstack-protector-guard=tls -mstack-protector-guard-reg=r13)
+ select HAVE_STACKPROTECTOR if PPC32 && $(cc-option,-mstack-protector-guard=tls -mstack-protector-guard-reg=r2)
select HAVE_CONTEXT_TRACKING if PPC64
select HAVE_DEBUG_KMEMLEAK
select HAVE_DEBUG_STACKOVERFLOW
--
2.17.1
^ permalink raw reply related
* Re: Looking for architecture papers
From: Raz @ 2018-10-13 0:27 UTC (permalink / raw)
To: gromero; +Cc: linuxppc-dev
In-Reply-To: <3f5ace27-7692-6d95-6e34-611d3eee3ead@linux.vnet.ibm.com>
So,if we're talk about address...
EA0 is actually bit 63 for any kernel address. This means[1] the
effective address is
the real address ( the physical address), because MSR_HV=1.
1. What does it mean PA=EA ? How does the translation work now ?
2. in interrupts, the program counter is set to EA 0c00, ==> EA=0.
So, in interrupt there's a different addressing model ?
[1]. Section 5.7.3, PowerISA 2.07B.
On Mon, Oct 8, 2018 at 10:59 PM Gustavo Romero
<gromero@linux.vnet.ibm.com> wrote:
>
> Hi Raz,
>
> On 10/04/2018 04:41 AM, Raz wrote:
> > Frankly, the more I read the more perplexed I get. For example,
> > according to BOOK III-S, chapter 3,
> > the MSR bits are differ from the ones described in
> > arch/powerpc/include/asm/reg.h.
> > Bit zero, is LE, but in the book it is 64-bit mode.
> >
> > Would someone be kind to explain what I do not understand?
>
> Yes, I know that can be confusing at the first sight when one is used to, for
> instance, x86.
>
> x86 documents use LSB 0 notation, which means (as others already pointed out)
> that the least significant bit of a value is marked as being bit 0.
>
> On the other hand Power documents use MSB 0 notation, which means that the most
> significant bit of a value is marked as being bit 0 and as a consequence the
> least significant bit in that notation in a 64-bit platform is bit 63, not bit
> 0. MSB 0 notation is also known as IBM bit notation/bit numbering.
>
> Historically LSB 0 notation tend to be used on docs about little-endian
> architectures (for instance, x86), whilst MSB 0 notation tend to be used on docs
> about big-endian architectures (for instance, Power - Power is actually a little
> different because it's now bi-endian actually).
>
> However LSB 0 and MSB 0 are only different notations, so LSB 0 can be employed
> on a big-endian architecture documentation, and vice versa.
>
> It happens that kernel code is written in C and for shifts, etc, it's convenient
> the LSB 0 notation, not the MSB 0 notation, so it's convenient to use LSB 0
> notation when creating a mask, like in arch/powerpc/include/asm/reg.h), i.e.
> it's convenient to employ bit positions as '63 - <bit position in PowerISA>'.
>
> So, as another example, in the following gcc macro '_TEXASR_EXTRACT_BITS' takes
> a bit position 'BITNUM' as found in the PowerISA documentation but then for the
> shift right it uses '63 - BITNUM':
>
> https://github.com/gcc-mirror/gcc/blob/master/gcc/config/rs6000/htmintrin.h#L44-L45
>
> I think it's also important to mention that on PowerISA the elements also follow
> the MSB 0 notation. So byte, word, and dword elements in a register found in the
> instruction descriptions when referred to 0 are the element "at the left tip",
> i.e. "the most significant elements", so to speak. For instance, take
> instruction "vperm": doc says 'index' takes bits 3:7 of a byte from [byte]
> element 'i'. So for a byte element i=0 it means the most significant byte
> ("on the left tip") of vector register operand 'VRC'. Moreover, specified bits
> in that byte element, i.e. bits 3:7, also follow the MSB 0, so for the
> little-endian addicted thought they are bits 4:0 (LSB 0 notation).
>
> Now, if bits 4:0 = 0b00011 (decimal 3), we grab byte element 3 from 'src'
> (256-bit). However byte element 3 is also in MSB 0 notation, so it means third
> byte of 'src' but starting counting bytes from 0 from the left to the right
> (which in IMO looks indeed more natural since we count, for instance, Natural
> Numbers on the 'x' axis similarly).
>
> Hence, it's like to say that 'vperm' instruction in a certain sense has a
> "big-endian semantics" for the byte indices. The 'vpermr' instruction introduced
> by PowerISA v3.0 is meant to cope with that, so 'vpermr' byte indices have a
> "little-endian semantics", so for bits 3:7 MSB 0 (or bits 4:0 in LSB 0 notation) =
> 0b00011 (decimal 3), on the 'vpermr' instruction it really means we must count
> bytes starting from right to left as in the LSB 0 notation and grab the third byte
> element from right to left.
>
> So, for instance:
>
> vr0 uint128 = 0x00000000000000000000000000000000
> vr1 uint128 = 0x00102030405060708090a0b0c0d0e0f0
> vr2 uint128 = 0x01112233445566779999aabbccddeeff
> vr3 uint128 = 0x03000000000000000000000000000000
>
> we have 'src' as:
>
> MSB 0: v--- byte 0, 1, 2, 3, ...
> LSB 0: ... 3, 2, 1, byte 0 ---v
> src = vr1 || vr2 = 00 10 20 30 40 50 60 70 80 90 A0 B0 C0 D0 E0 F0 01 11 22 33 44 55 66 77 99 99 AA BB CC DD EE FF
>
> vperm vr0, vr1, vr2, vr3 result is:
> vr0 uint128 = 0x30000000000000000000000000000000
> byte 3 in MSB 0 = 0x30 ---^ and 0x00 (byte 0 in MSB 0) copied to the remaining bytes
>
> whilst with vpermr (PowerISA v3.0 / POWER9):
> vpermr vr0, vr1, vr2, vr3 result is:
> vr0 uint128 = 0xccffffffffffffffffffffffffffffff
> byte 3 in LSB 0 = 0xCC----^ and 0xFF (byte 0 in LSB 0) copied to the remaining bytes
>
>
> Anyway, vperm/vpermr was just an example about notation not being restricted to
> bits on Power ISA. So read the docs carefully :) GDB is always useful for checking
> if one's understanding about a given Power instruction is correct.
>
> HTH.
>
>
> Regards,
> Gustavo
>
^ permalink raw reply
* Re: [PATCH 4/4] powerpc: Add -Wimplicit-fallthrough to arch CFLAGS
From: Kees Cook @ 2018-10-13 1:23 UTC (permalink / raw)
To: Gustavo A. R. Silva; +Cc: maddy, linuxppc-dev
In-Reply-To: <0de89924-ac22-b03a-b041-c79837d3c8cc@embeddedor.com>
On Fri, Oct 12, 2018 at 2:59 AM, Gustavo A. R. Silva
<gustavo@embeddedor.com> wrote:
>
>
> On 10/12/18 11:32 AM, Michael Ellerman wrote:
>>
>> Sure. The kbuild report up thread has some or most of them.
>>
>> But here's a full list:
>>
>> Failed 279/290
>> http://kisskb.ellerman.id.au/kisskb/head/1d59e2c78793d8aea9949ca71323c4583c78f488/
>> Failed: powerpc-next/ppc64_defconfig/powerpc-gcc8 (http://kisskb.ellerman.id.au/kisskb/buildresult/13542791/log/)
>> /kisskb/src/arch/powerpc/platforms/powermac/feature.c:1477:6: error: this statement may fall through [-Werror=implicit-fallthrough=]
>> /kisskb/src/include/linux/signal.h:180:22: error: this statement may fall through [-Werror=implicit-fallthrough=]
>> /kisskb/src/include/linux/compat.h:495:51: error: this statement may fall through [-Werror=implicit-fallthrough=]
>> /kisskb/src/include/linux/compat.h:496:51: error: this statement may fall through [-Werror=implicit-fallthrough=]
>> /kisskb/src/include/linux/compat.h:494:51: error: this statement may fall through [-Werror=implicit-fallthrough=]
>> Failed: powerpc-next/corenet64_smp_defconfig/powerpc-gcc8 (http://kisskb.ellerman.id.au/kisskb/buildresult/13542786/log/)
>> /kisskb/src/include/linux/compat.h:494:51: error: this statement may fall through [-Werror=implicit-fallthrough=]
>> /kisskb/src/include/linux/compat.h:495:51: error: this statement may fall through [-Werror=implicit-fallthrough=]
>> /kisskb/src/include/linux/compat.h:496:51: error: this statement may fall through [-Werror=implicit-fallthrough=]
>> Failed: powerpc-next/ppc64le_defconfig/powerpc-gcc8 (http://kisskb.ellerman.id.au/kisskb/buildresult/13542777/log/)
>> /kisskb/src/include/linux/signal.h:180:22: error: this statement may fall through [-Werror=implicit-fallthrough=]
>> Failed: powerpc-next/powernv_defconfig/powerpc-gcc8 (http://kisskb.ellerman.id.au/kisskb/buildresult/13542776/log/)
>> /kisskb/src/include/linux/signal.h:180:22: error: this statement may fall through [-Werror=implicit-fallthrough=]
>>
>
> Thanks, Michael.
>
>
> Kees, can you take the patches?
>
> Apparently, neither signal.h nor compat.h have a dedicated maintainer:
>
> $ scripts/get_maintainer.pl --nokeywords --nogit --nogit-fallback include/linux/signal.h
> linux-kernel@vger.kernel.org (open list)
>
> $ scripts/get_maintainer.pl --nokeywords --nogit --nogit-fallback include/linux/compat.h
> linux-kernel@vger.kernel.org (open list)
Normally things like that go through akpm, but I'm happy to carry them
if needed.
-Kees
--
Kees Cook
Pixel Security
^ permalink raw reply
* Re: [PATCH v2 2/2] mm: speed up mremap by 500x on large regions
From: Joel Fernandes @ 2018-10-13 1:35 UTC (permalink / raw)
To: David Miller
Cc: linux-mips, linux-ia64, linux-sh, peterz, catalin.marinas,
dave.hansen, mhocko, linux-mm, lokeshgidra, linux-riscv, elfring,
jonas, linux-s390, dancol, linux-xtensa, linux-hexagon, deller,
hughd, jejb, kasan-dev, kvmarm, mingo, geert, aryabinin,
linux-snps-arc, kernel-team, fenghua.yu, jdike, linux-um,
Julia.Lawall, linux-m68k, bp, luto, nios2-dev, kirill, gxt, chris,
richard, linux-parisc, pantin, jcmvbkbc, linux-kernel, minchan,
linux-alpha, lftan, akpm, linuxppc-dev
In-Reply-To: <20181012.111836.1569129998592378186.davem@davemloft.net>
On Fri, Oct 12, 2018 at 11:18:36AM -0700, David Miller wrote:
> From: Joel Fernandes <joel@joelfernandes.org>
[...]
> > Also, do we not flush the caches from any path when we munmap
> > address space? We do call do_munmap on the old mapping from mremap
> > after moving to the new one.
>
> Sparc makes sure that shared mapping have consistent colors. Therefore
> all that's left are private mappings and those will be initialized by
> block stores to clear the page out or similar.
>
> Also, when creating new mappings, we flush the D-cache when necessary
> in update_mmu_cache().
>
> We also maintain a bit in the page struct to track when a page which
> was potentially written to on one cpu ends up mapped into another
> address space and flush as necessary.
>
> The cache is write-through, which simplifies the preconditions we have
> to maintain.
Makes sense, thanks. For the moment I sent patches to enable this on arm64
and x86. We can enable it on sparc as well at a later time as it sounds it
could be a safe optimization to apply to that architecture as well.
thanks,
- Joel
^ permalink raw reply
* Re: [PATCH v2 2/2] mm: speed up mremap by 500x on large regions
From: Joel Fernandes @ 2018-10-13 1:44 UTC (permalink / raw)
To: Daniel Colascione
Cc: linux-mips, linux-ia64, linux-sh, Peter Zijlstra, catalin.marinas,
dave.hansen, Michal Hocko, linux-mm, Lokesh Gidra, linux-riscv,
elfring, jonas, linux-s390, linux-xtensa, linux-hexagon, deller,
hughd, jejb, kasan-dev, kvmarm, mingo, geert, aryabinin,
linux-snps-arc, kernel-team, fenghua.yu, jdike, linux-um,
Julia.Lawall, linux-m68k, bp, luto, nios2-dev, kirill, gxt, chris,
richard, linux-parisc, Ramon Pantin, jcmvbkbc, linux-kernel,
Minchan Kim, linux-alpha, lftan, Andrew Morton, linuxppc-dev,
David Miller
In-Reply-To: <CAKOZueuNvWvn18vffJWpbpg7h-uScT8gXrrudTB2pnT4M2HJ_w@mail.gmail.com>
On Fri, Oct 12, 2018 at 06:39:45PM -0700, Daniel Colascione wrote:
> Not 32-bit ARM?
Well, I didn't want to enable every possible architecture we could in a
single go. Certainly arm32 can be a follow on enablement as can be other
architectures. The point of this series is to upstream this feature and
enable a hand-picked few architectures as a first step.
thanks,
- Joel
^ permalink raw reply
* Re: [PATCH v2 2/2] mm: speed up mremap by 500x on large regions
From: Joel Fernandes @ 2018-10-13 2:10 UTC (permalink / raw)
To: Daniel Colascione
Cc: linux-mips, linux-ia64, linux-sh, Peter Zijlstra, catalin.marinas,
dave.hansen, Michal Hocko, linux-mm, Lokesh Gidra, linux-riscv,
elfring, jonas, linux-s390, linux-xtensa, linux-hexagon, deller,
hughd, jejb, kasan-dev, kvmarm, mingo, geert, aryabinin,
linux-snps-arc, kernel-team, fenghua.yu, jdike, linux-um,
Julia.Lawall, linux-m68k, bp, luto, nios2-dev, kirill, gxt, chris,
richard, linux-parisc, Ramon Pantin, jcmvbkbc, linux-kernel,
Minchan Kim, linux-alpha, lftan, Andrew Morton, linuxppc-dev,
David Miller
In-Reply-To: <CAKOZues25aaKz3_AiyfJ=r2QBd5MghgY3ky_ptg4Z8=ST4DCgw@mail.gmail.com>
On Fri, Oct 12, 2018 at 06:54:33PM -0700, Daniel Colascione wrote:
> I wonder whether it makes sense to expose to userspace somehow whether
> mremap is "fast" for a particular architecture. If a feature relies on
> fast mremap, it might be better for some userland component to disable
> that feature entirely rather than blindly use mremap and end up
> performing very poorly. If we're disabling fast mremap when THP is
> enabled, the userland component can't just rely on an architecture
> switch and some kind of runtime feature detection becomes even more
> important.
I hate to point out that its forbidden to top post on LKML :-)
https://kernelnewbies.org/mailinglistguidelines
So don't that Mr. Dan! :D
But anyway, I think this runtime detection thing is not needed. THP is
actually expected to be as fast as this anyway, so if that's available then
we should already be as fast. This is for non-THP where THP cannot be enabled
and there is still room for some improvement. Most/all architectures will be
just fine with this. This flag is more of a safety-net type of thing where in
the future if there is this one or two weird architectures that don't play
well, then they can turn it off at the architecture level by not selecting
the flag. See my latest patches for the per-architecture compile-time
controls. Ideally we'd like to blanket turn it on on all, but this is just
playing it extra safe as Kirill and me were discussing on other threads.
thanks!
- Joel
^ permalink raw reply
* [PATCH 0/4] Add support for fast mremap
From: Joel Fernandes (Google) @ 2018-10-13 1:31 UTC (permalink / raw)
To: linux-kernel
Cc: linux-mips, Rich Felker, linux-ia64, linux-sh, Peter Zijlstra,
Catalin Marinas, Dave Hansen, Will Deacon, mhocko, linux-mm,
lokeshgidra, Joel Fernandes (Google), linux-riscv, elfring,
Jonas Bonn, kvmarm, dancol, Yoshinori Sato, sparclinux,
linux-xtensa, linux-hexagon, Helge Deller,
maintainer:X86 ARCHITECTURE 32-BIT AND 64-BIT, hughd,
James E.J. Bottomley, kasan-dev, anton.ivanov, Ingo Molnar,
Geert Uytterhoeven, Andrey Ryabinin, linux-snps-arc, kernel-team,
Sam Creasey, Fenghua Yu, linux-s390, Jeff Dike, linux-um,
Stefan Kristiansson, Julia Lawall, linux-m68k, Borislav Petkov,
Andy Lutomirski, nios2-dev, Kirill A. Shutemov, Stafford Horne,
Guan Xuetao, Chris Zankel, Tony Luck, Richard Weinberger,
linux-parisc, Max Filippov, pantin, minchan, Thomas Gleixner,
linux-alpha, Ley Foon Tan, akpm, linuxppc-dev, David S. Miller
Hi,
Here is the latest "fast mremap" series. The main change in this submission is
to enable the fast mremap optimization on a per-architecture basis to prevent
possible issues with architectures that may not behave well with such change.
x86: select HAVE_MOVE_PMD for faster mremap (v1)
arm64: select HAVE_MOVE_PMD for faster mremap (v1)
mm: speed up mremap by 500x on large regions (v2)
v1->v2: Added support for per-arch enablement (Kirill Shutemov)
treewide: remove unused address argument from pte_alloc functions (v2)
v1->v2: fix arch/um/ prototype which was missed in v1 (Anton Ivanov)
update changelog with manual fixups for m68k and microblaze.
Joel Fernandes (Google) (4):
treewide: remove unused address argument from pte_alloc functions (v2)
mm: speed up mremap by 500x on large regions (v2)
arm64: select HAVE_MOVE_PMD for faster mremap (v1)
x86: select HAVE_MOVE_PMD for faster mremap (v1)
arch/Kconfig | 5 ++
arch/alpha/include/asm/pgalloc.h | 6 +-
arch/arc/include/asm/pgalloc.h | 5 +-
arch/arm/include/asm/pgalloc.h | 4 +-
arch/arm64/Kconfig | 1 +
arch/arm64/include/asm/pgalloc.h | 4 +-
arch/hexagon/include/asm/pgalloc.h | 6 +-
arch/ia64/include/asm/pgalloc.h | 5 +-
arch/m68k/include/asm/mcf_pgalloc.h | 8 +--
arch/m68k/include/asm/motorola_pgalloc.h | 4 +-
arch/m68k/include/asm/sun3_pgalloc.h | 6 +-
arch/microblaze/include/asm/pgalloc.h | 19 +-----
arch/microblaze/mm/pgtable.c | 3 +-
arch/mips/include/asm/pgalloc.h | 6 +-
arch/nds32/include/asm/pgalloc.h | 5 +-
arch/nios2/include/asm/pgalloc.h | 6 +-
arch/openrisc/include/asm/pgalloc.h | 5 +-
arch/openrisc/mm/ioremap.c | 3 +-
arch/parisc/include/asm/pgalloc.h | 4 +-
arch/powerpc/include/asm/book3s/32/pgalloc.h | 4 +-
arch/powerpc/include/asm/book3s/64/pgalloc.h | 12 ++--
arch/powerpc/include/asm/nohash/32/pgalloc.h | 4 +-
arch/powerpc/include/asm/nohash/64/pgalloc.h | 6 +-
arch/powerpc/mm/pgtable-book3s64.c | 2 +-
arch/powerpc/mm/pgtable_32.c | 4 +-
arch/riscv/include/asm/pgalloc.h | 6 +-
arch/s390/include/asm/pgalloc.h | 4 +-
arch/sh/include/asm/pgalloc.h | 6 +-
arch/sparc/include/asm/pgalloc_32.h | 5 +-
arch/sparc/include/asm/pgalloc_64.h | 6 +-
arch/sparc/mm/init_64.c | 6 +-
arch/sparc/mm/srmmu.c | 4 +-
arch/um/include/asm/pgalloc.h | 4 +-
arch/um/kernel/mem.c | 4 +-
arch/unicore32/include/asm/pgalloc.h | 4 +-
arch/x86/Kconfig | 1 +
arch/x86/include/asm/pgalloc.h | 4 +-
arch/x86/mm/pgtable.c | 4 +-
arch/xtensa/include/asm/pgalloc.h | 8 +--
include/linux/mm.h | 13 ++--
mm/huge_memory.c | 8 +--
mm/kasan/kasan_init.c | 2 +-
mm/memory.c | 17 +++--
mm/migrate.c | 2 +-
mm/mremap.c | 67 +++++++++++++++++++-
mm/userfaultfd.c | 2 +-
virt/kvm/arm/mmu.c | 2 +-
47 files changed, 169 insertions(+), 147 deletions(-)
--
2.19.0.605.g01d371f741-goog
^ permalink raw reply
* [PATCH 1/4] treewide: remove unused address argument from pte_alloc functions (v2)
From: Joel Fernandes (Google) @ 2018-10-13 1:31 UTC (permalink / raw)
To: linux-kernel
Cc: linux-mips, Rich Felker, linux-ia64, linux-sh, Peter Zijlstra,
Catalin Marinas, Dave Hansen, Will Deacon, Michal Hocko, linux-mm,
lokeshgidra, Joel Fernandes (Google), linux-riscv, elfring,
Jonas Bonn, kvmarm, dancol, Yoshinori Sato, sparclinux,
linux-xtensa, linux-hexagon, Helge Deller,
maintainer:X86 ARCHITECTURE 32-BIT AND 64-BIT, hughd,
James E.J. Bottomley, kasan-dev, anton.ivanov, Ingo Molnar,
Geert Uytterhoeven, Andrey Ryabinin, linux-snps-arc, kernel-team,
Sam Creasey, Fenghua Yu, linux-s390, Jeff Dike, linux-um,
Stefan Kristiansson, Julia Lawall, linux-m68k, Borislav Petkov,
Andy Lutomirski, nios2-dev, Kirill A . Shutemov, Stafford Horne,
Guan Xuetao, Chris Zankel, Tony Luck, Richard Weinberger,
linux-parisc, Max Filippov, pantin, minchan, Thomas Gleixner,
linux-alpha, Ley Foon Tan, akpm, linuxppc-dev, David S. Miller
In-Reply-To: <20181013013200.206928-1-joel@joelfernandes.org>
This series speeds up mremap(2) syscall by copying page tables at the
PMD level even for non-THP systems. There is concern that the extra
'address' argument that mremap passes to pte_alloc may do something
subtle architecture related in the future that may make the scheme not
work. Also we find that there is no point in passing the 'address' to
pte_alloc since its unused. So this patch therefore removes this
argument tree-wide resulting in a nice negative diff as well. Also
ensuring along the way that the enabled architectures do not do anything
funky with 'address' argument that goes unnoticed by the optimization.
Build and boot tested on x86-64. Build tested on arm64.
The changes were obtained by applying the following Coccinelle script.
(thanks Julia for answering all Coccinelle questions!).
Following fix ups were done manually:
* Removal of address argument from pte_fragment_alloc
* Removal of pte_alloc_one_fast definitions from m68k and microblaze.
// Options: --include-headers --no-includes
// Note: I split the 'identifier fn' line, so if you are manually
// running it, please unsplit it so it runs for you.
virtual patch
@pte_alloc_func_def depends on patch exists@
identifier E2;
identifier fn =~
"^(__pte_alloc|pte_alloc_one|pte_alloc|__pte_alloc_kernel|pte_alloc_one_kernel)$";
type T2;
@@
fn(...
- , T2 E2
)
{ ... }
@pte_alloc_func_proto_noarg depends on patch exists@
type T1, T2, T3, T4;
identifier fn =~ "^(__pte_alloc|pte_alloc_one|pte_alloc|__pte_alloc_kernel|pte_alloc_one_kernel)$";
@@
(
- T3 fn(T1, T2);
+ T3 fn(T1);
|
- T3 fn(T1, T2, T4);
+ T3 fn(T1, T2);
)
@pte_alloc_func_proto depends on patch exists@
identifier E1, E2, E4;
type T1, T2, T3, T4;
identifier fn =~
"^(__pte_alloc|pte_alloc_one|pte_alloc|__pte_alloc_kernel|pte_alloc_one_kernel)$";
@@
(
- T3 fn(T1 E1, T2 E2);
+ T3 fn(T1 E1);
|
- T3 fn(T1 E1, T2 E2, T4 E4);
+ T3 fn(T1 E1, T2 E2);
)
@pte_alloc_func_call depends on patch exists@
expression E2;
identifier fn =~
"^(__pte_alloc|pte_alloc_one|pte_alloc|__pte_alloc_kernel|pte_alloc_one_kernel)$";
@@
fn(...
-, E2
)
@pte_alloc_macro depends on patch exists@
identifier fn =~
"^(__pte_alloc|pte_alloc_one|pte_alloc|__pte_alloc_kernel|pte_alloc_one_kernel)$";
identifier a, b, c;
expression e;
position p;
@@
(
- #define fn(a, b, c) e
+ #define fn(a, b) e
|
- #define fn(a, b) e
+ #define fn(a) e
)
Suggested-by: Kirill A. Shutemov <kirill@shutemov.name>
Cc: Kirill A. Shutemov <kirill@shutemov.name>
Cc: Michal Hocko <mhocko@kernel.org>
Cc: Julia Lawall <Julia.Lawall@lip6.fr>
Signed-off-by: Joel Fernandes (Google) <joel@joelfernandes.org>
---
arch/alpha/include/asm/pgalloc.h | 6 +++---
arch/arc/include/asm/pgalloc.h | 5 ++---
arch/arm/include/asm/pgalloc.h | 4 ++--
arch/arm64/include/asm/pgalloc.h | 4 ++--
arch/hexagon/include/asm/pgalloc.h | 6 ++----
arch/ia64/include/asm/pgalloc.h | 5 ++---
arch/m68k/include/asm/mcf_pgalloc.h | 8 ++------
arch/m68k/include/asm/motorola_pgalloc.h | 4 ++--
arch/m68k/include/asm/sun3_pgalloc.h | 6 ++----
arch/microblaze/include/asm/pgalloc.h | 19 ++-----------------
arch/microblaze/mm/pgtable.c | 3 +--
arch/mips/include/asm/pgalloc.h | 6 ++----
arch/nds32/include/asm/pgalloc.h | 5 ++---
arch/nios2/include/asm/pgalloc.h | 6 ++----
arch/openrisc/include/asm/pgalloc.h | 5 ++---
arch/openrisc/mm/ioremap.c | 3 +--
arch/parisc/include/asm/pgalloc.h | 4 ++--
arch/powerpc/include/asm/book3s/32/pgalloc.h | 4 ++--
arch/powerpc/include/asm/book3s/64/pgalloc.h | 12 +++++-------
arch/powerpc/include/asm/nohash/32/pgalloc.h | 4 ++--
arch/powerpc/include/asm/nohash/64/pgalloc.h | 6 ++----
arch/powerpc/mm/pgtable-book3s64.c | 2 +-
arch/powerpc/mm/pgtable_32.c | 4 ++--
arch/riscv/include/asm/pgalloc.h | 6 ++----
arch/s390/include/asm/pgalloc.h | 4 ++--
arch/sh/include/asm/pgalloc.h | 6 ++----
arch/sparc/include/asm/pgalloc_32.h | 5 ++---
arch/sparc/include/asm/pgalloc_64.h | 6 ++----
arch/sparc/mm/init_64.c | 6 ++----
arch/sparc/mm/srmmu.c | 4 ++--
arch/um/include/asm/pgalloc.h | 4 ++--
arch/um/kernel/mem.c | 4 ++--
arch/unicore32/include/asm/pgalloc.h | 4 ++--
arch/x86/include/asm/pgalloc.h | 4 ++--
arch/x86/mm/pgtable.c | 4 ++--
arch/xtensa/include/asm/pgalloc.h | 8 +++-----
include/linux/mm.h | 13 ++++++-------
mm/huge_memory.c | 8 ++++----
mm/kasan/kasan_init.c | 2 +-
mm/memory.c | 17 ++++++++---------
mm/migrate.c | 2 +-
mm/mremap.c | 2 +-
mm/userfaultfd.c | 2 +-
virt/kvm/arm/mmu.c | 2 +-
44 files changed, 97 insertions(+), 147 deletions(-)
diff --git a/arch/alpha/include/asm/pgalloc.h b/arch/alpha/include/asm/pgalloc.h
index ab3e3a8638fb..02f9f91bb4f0 100644
--- a/arch/alpha/include/asm/pgalloc.h
+++ b/arch/alpha/include/asm/pgalloc.h
@@ -52,7 +52,7 @@ pmd_free(struct mm_struct *mm, pmd_t *pmd)
}
static inline pte_t *
-pte_alloc_one_kernel(struct mm_struct *mm, unsigned long address)
+pte_alloc_one_kernel(struct mm_struct *mm)
{
pte_t *pte = (pte_t *)__get_free_page(GFP_KERNEL|__GFP_ZERO);
return pte;
@@ -65,9 +65,9 @@ pte_free_kernel(struct mm_struct *mm, pte_t *pte)
}
static inline pgtable_t
-pte_alloc_one(struct mm_struct *mm, unsigned long address)
+pte_alloc_one(struct mm_struct *mm)
{
- pte_t *pte = pte_alloc_one_kernel(mm, address);
+ pte_t *pte = pte_alloc_one_kernel(mm);
struct page *page;
if (!pte)
diff --git a/arch/arc/include/asm/pgalloc.h b/arch/arc/include/asm/pgalloc.h
index 3749234b7419..9c9b5a5ebf2e 100644
--- a/arch/arc/include/asm/pgalloc.h
+++ b/arch/arc/include/asm/pgalloc.h
@@ -90,8 +90,7 @@ static inline int __get_order_pte(void)
return get_order(PTRS_PER_PTE * sizeof(pte_t));
}
-static inline pte_t *pte_alloc_one_kernel(struct mm_struct *mm,
- unsigned long address)
+static inline pte_t *pte_alloc_one_kernel(struct mm_struct *mm)
{
pte_t *pte;
@@ -102,7 +101,7 @@ static inline pte_t *pte_alloc_one_kernel(struct mm_struct *mm,
}
static inline pgtable_t
-pte_alloc_one(struct mm_struct *mm, unsigned long address)
+pte_alloc_one(struct mm_struct *mm)
{
pgtable_t pte_pg;
struct page *page;
diff --git a/arch/arm/include/asm/pgalloc.h b/arch/arm/include/asm/pgalloc.h
index 2d7344f0e208..17ab72f0cc4e 100644
--- a/arch/arm/include/asm/pgalloc.h
+++ b/arch/arm/include/asm/pgalloc.h
@@ -81,7 +81,7 @@ static inline void clean_pte_table(pte_t *pte)
* +------------+
*/
static inline pte_t *
-pte_alloc_one_kernel(struct mm_struct *mm, unsigned long addr)
+pte_alloc_one_kernel(struct mm_struct *mm)
{
pte_t *pte;
@@ -93,7 +93,7 @@ pte_alloc_one_kernel(struct mm_struct *mm, unsigned long addr)
}
static inline pgtable_t
-pte_alloc_one(struct mm_struct *mm, unsigned long addr)
+pte_alloc_one(struct mm_struct *mm)
{
struct page *pte;
diff --git a/arch/arm64/include/asm/pgalloc.h b/arch/arm64/include/asm/pgalloc.h
index 2e05bcd944c8..52fa47c73bf0 100644
--- a/arch/arm64/include/asm/pgalloc.h
+++ b/arch/arm64/include/asm/pgalloc.h
@@ -91,13 +91,13 @@ extern pgd_t *pgd_alloc(struct mm_struct *mm);
extern void pgd_free(struct mm_struct *mm, pgd_t *pgdp);
static inline pte_t *
-pte_alloc_one_kernel(struct mm_struct *mm, unsigned long addr)
+pte_alloc_one_kernel(struct mm_struct *mm)
{
return (pte_t *)__get_free_page(PGALLOC_GFP);
}
static inline pgtable_t
-pte_alloc_one(struct mm_struct *mm, unsigned long addr)
+pte_alloc_one(struct mm_struct *mm)
{
struct page *pte;
diff --git a/arch/hexagon/include/asm/pgalloc.h b/arch/hexagon/include/asm/pgalloc.h
index eeebf862c46c..d36183887b60 100644
--- a/arch/hexagon/include/asm/pgalloc.h
+++ b/arch/hexagon/include/asm/pgalloc.h
@@ -59,8 +59,7 @@ static inline void pgd_free(struct mm_struct *mm, pgd_t *pgd)
free_page((unsigned long) pgd);
}
-static inline struct page *pte_alloc_one(struct mm_struct *mm,
- unsigned long address)
+static inline struct page *pte_alloc_one(struct mm_struct *mm)
{
struct page *pte;
@@ -75,8 +74,7 @@ static inline struct page *pte_alloc_one(struct mm_struct *mm,
}
/* _kernel variant gets to use a different allocator */
-static inline pte_t *pte_alloc_one_kernel(struct mm_struct *mm,
- unsigned long address)
+static inline pte_t *pte_alloc_one_kernel(struct mm_struct *mm)
{
gfp_t flags = GFP_KERNEL | __GFP_ZERO;
return (pte_t *) __get_free_page(flags);
diff --git a/arch/ia64/include/asm/pgalloc.h b/arch/ia64/include/asm/pgalloc.h
index 3ee5362f2661..c9e481023c25 100644
--- a/arch/ia64/include/asm/pgalloc.h
+++ b/arch/ia64/include/asm/pgalloc.h
@@ -83,7 +83,7 @@ pmd_populate_kernel(struct mm_struct *mm, pmd_t * pmd_entry, pte_t * pte)
pmd_val(*pmd_entry) = __pa(pte);
}
-static inline pgtable_t pte_alloc_one(struct mm_struct *mm, unsigned long addr)
+static inline pgtable_t pte_alloc_one(struct mm_struct *mm)
{
struct page *page;
void *pg;
@@ -99,8 +99,7 @@ static inline pgtable_t pte_alloc_one(struct mm_struct *mm, unsigned long addr)
return page;
}
-static inline pte_t *pte_alloc_one_kernel(struct mm_struct *mm,
- unsigned long addr)
+static inline pte_t *pte_alloc_one_kernel(struct mm_struct *mm)
{
return quicklist_alloc(0, GFP_KERNEL, NULL);
}
diff --git a/arch/m68k/include/asm/mcf_pgalloc.h b/arch/m68k/include/asm/mcf_pgalloc.h
index 12fe700632f4..4399d712f6db 100644
--- a/arch/m68k/include/asm/mcf_pgalloc.h
+++ b/arch/m68k/include/asm/mcf_pgalloc.h
@@ -12,8 +12,7 @@ extern inline void pte_free_kernel(struct mm_struct *mm, pte_t *pte)
extern const char bad_pmd_string[];
-extern inline pte_t *pte_alloc_one_kernel(struct mm_struct *mm,
- unsigned long address)
+extern inline pte_t *pte_alloc_one_kernel(struct mm_struct *mm)
{
unsigned long page = __get_free_page(GFP_DMA);
@@ -32,8 +31,6 @@ extern inline pmd_t *pmd_alloc_kernel(pgd_t *pgd, unsigned long address)
#define pmd_alloc_one_fast(mm, address) ({ BUG(); ((pmd_t *)1); })
#define pmd_alloc_one(mm, address) ({ BUG(); ((pmd_t *)2); })
-#define pte_alloc_one_fast(mm, addr) pte_alloc_one(mm, addr)
-
#define pmd_populate(mm, pmd, page) (pmd_val(*pmd) = \
(unsigned long)(page_address(page)))
@@ -50,8 +47,7 @@ static inline void __pte_free_tlb(struct mmu_gather *tlb, pgtable_t page,
#define __pmd_free_tlb(tlb, pmd, address) do { } while (0)
-static inline struct page *pte_alloc_one(struct mm_struct *mm,
- unsigned long address)
+static inline struct page *pte_alloc_one(struct mm_struct *mm)
{
struct page *page = alloc_pages(GFP_DMA, 0);
pte_t *pte;
diff --git a/arch/m68k/include/asm/motorola_pgalloc.h b/arch/m68k/include/asm/motorola_pgalloc.h
index 7859a86319cf..d04d9ba9b976 100644
--- a/arch/m68k/include/asm/motorola_pgalloc.h
+++ b/arch/m68k/include/asm/motorola_pgalloc.h
@@ -8,7 +8,7 @@
extern pmd_t *get_pointer_table(void);
extern int free_pointer_table(pmd_t *);
-static inline pte_t *pte_alloc_one_kernel(struct mm_struct *mm, unsigned long address)
+static inline pte_t *pte_alloc_one_kernel(struct mm_struct *mm)
{
pte_t *pte;
@@ -28,7 +28,7 @@ static inline void pte_free_kernel(struct mm_struct *mm, pte_t *pte)
free_page((unsigned long) pte);
}
-static inline pgtable_t pte_alloc_one(struct mm_struct *mm, unsigned long address)
+static inline pgtable_t pte_alloc_one(struct mm_struct *mm)
{
struct page *page;
pte_t *pte;
diff --git a/arch/m68k/include/asm/sun3_pgalloc.h b/arch/m68k/include/asm/sun3_pgalloc.h
index 11485d38de4e..1456c5eecbd9 100644
--- a/arch/m68k/include/asm/sun3_pgalloc.h
+++ b/arch/m68k/include/asm/sun3_pgalloc.h
@@ -35,8 +35,7 @@ do { \
tlb_remove_page((tlb), pte); \
} while (0)
-static inline pte_t *pte_alloc_one_kernel(struct mm_struct *mm,
- unsigned long address)
+static inline pte_t *pte_alloc_one_kernel(struct mm_struct *mm)
{
unsigned long page = __get_free_page(GFP_KERNEL);
@@ -47,8 +46,7 @@ static inline pte_t *pte_alloc_one_kernel(struct mm_struct *mm,
return (pte_t *) (page);
}
-static inline pgtable_t pte_alloc_one(struct mm_struct *mm,
- unsigned long address)
+static inline pgtable_t pte_alloc_one(struct mm_struct *mm)
{
struct page *page = alloc_pages(GFP_KERNEL, 0);
diff --git a/arch/microblaze/include/asm/pgalloc.h b/arch/microblaze/include/asm/pgalloc.h
index 7c89390c0c13..f4cc9ffc449e 100644
--- a/arch/microblaze/include/asm/pgalloc.h
+++ b/arch/microblaze/include/asm/pgalloc.h
@@ -108,10 +108,9 @@ static inline void free_pgd_slow(pgd_t *pgd)
#define pmd_alloc_one_fast(mm, address) ({ BUG(); ((pmd_t *)1); })
#define pmd_alloc_one(mm, address) ({ BUG(); ((pmd_t *)2); })
-extern pte_t *pte_alloc_one_kernel(struct mm_struct *mm, unsigned long addr);
+extern pte_t *pte_alloc_one_kernel(struct mm_struct *mm);
-static inline struct page *pte_alloc_one(struct mm_struct *mm,
- unsigned long address)
+static inline struct page *pte_alloc_one(struct mm_struct *mm)
{
struct page *ptepage;
@@ -132,20 +131,6 @@ static inline struct page *pte_alloc_one(struct mm_struct *mm,
return ptepage;
}
-static inline pte_t *pte_alloc_one_fast(struct mm_struct *mm,
- unsigned long address)
-{
- unsigned long *ret;
-
- ret = pte_quicklist;
- if (ret != NULL) {
- pte_quicklist = (unsigned long *)(*ret);
- ret[0] = 0;
- pgtable_cache_size--;
- }
- return (pte_t *)ret;
-}
-
static inline void pte_free_fast(pte_t *pte)
{
*(unsigned long **)pte = pte_quicklist;
diff --git a/arch/microblaze/mm/pgtable.c b/arch/microblaze/mm/pgtable.c
index 7f525962cdfa..c2ce1e42b888 100644
--- a/arch/microblaze/mm/pgtable.c
+++ b/arch/microblaze/mm/pgtable.c
@@ -235,8 +235,7 @@ unsigned long iopa(unsigned long addr)
return pa;
}
-__ref pte_t *pte_alloc_one_kernel(struct mm_struct *mm,
- unsigned long address)
+__ref pte_t *pte_alloc_one_kernel(struct mm_struct *mm)
{
pte_t *pte;
if (mem_init_done) {
diff --git a/arch/mips/include/asm/pgalloc.h b/arch/mips/include/asm/pgalloc.h
index 39b9f311c4ef..27808d9461f4 100644
--- a/arch/mips/include/asm/pgalloc.h
+++ b/arch/mips/include/asm/pgalloc.h
@@ -50,14 +50,12 @@ static inline void pgd_free(struct mm_struct *mm, pgd_t *pgd)
free_pages((unsigned long)pgd, PGD_ORDER);
}
-static inline pte_t *pte_alloc_one_kernel(struct mm_struct *mm,
- unsigned long address)
+static inline pte_t *pte_alloc_one_kernel(struct mm_struct *mm)
{
return (pte_t *)__get_free_pages(GFP_KERNEL | __GFP_ZERO, PTE_ORDER);
}
-static inline struct page *pte_alloc_one(struct mm_struct *mm,
- unsigned long address)
+static inline struct page *pte_alloc_one(struct mm_struct *mm)
{
struct page *pte;
diff --git a/arch/nds32/include/asm/pgalloc.h b/arch/nds32/include/asm/pgalloc.h
index 27448869131a..3c5fee5b5759 100644
--- a/arch/nds32/include/asm/pgalloc.h
+++ b/arch/nds32/include/asm/pgalloc.h
@@ -22,8 +22,7 @@ extern void pgd_free(struct mm_struct *mm, pgd_t * pgd);
#define check_pgt_cache() do { } while (0)
-static inline pte_t *pte_alloc_one_kernel(struct mm_struct *mm,
- unsigned long addr)
+static inline pte_t *pte_alloc_one_kernel(struct mm_struct *mm)
{
pte_t *pte;
@@ -34,7 +33,7 @@ static inline pte_t *pte_alloc_one_kernel(struct mm_struct *mm,
return pte;
}
-static inline pgtable_t pte_alloc_one(struct mm_struct *mm, unsigned long addr)
+static inline pgtable_t pte_alloc_one(struct mm_struct *mm)
{
pgtable_t pte;
diff --git a/arch/nios2/include/asm/pgalloc.h b/arch/nios2/include/asm/pgalloc.h
index bb47d08c8ef7..3a149ead1207 100644
--- a/arch/nios2/include/asm/pgalloc.h
+++ b/arch/nios2/include/asm/pgalloc.h
@@ -37,8 +37,7 @@ static inline void pgd_free(struct mm_struct *mm, pgd_t *pgd)
free_pages((unsigned long)pgd, PGD_ORDER);
}
-static inline pte_t *pte_alloc_one_kernel(struct mm_struct *mm,
- unsigned long address)
+static inline pte_t *pte_alloc_one_kernel(struct mm_struct *mm)
{
pte_t *pte;
@@ -47,8 +46,7 @@ static inline pte_t *pte_alloc_one_kernel(struct mm_struct *mm,
return pte;
}
-static inline pgtable_t pte_alloc_one(struct mm_struct *mm,
- unsigned long address)
+static inline pgtable_t pte_alloc_one(struct mm_struct *mm)
{
struct page *pte;
diff --git a/arch/openrisc/include/asm/pgalloc.h b/arch/openrisc/include/asm/pgalloc.h
index 8999b9226512..149c82ee4b8b 100644
--- a/arch/openrisc/include/asm/pgalloc.h
+++ b/arch/openrisc/include/asm/pgalloc.h
@@ -70,10 +70,9 @@ static inline void pgd_free(struct mm_struct *mm, pgd_t *pgd)
free_page((unsigned long)pgd);
}
-extern pte_t *pte_alloc_one_kernel(struct mm_struct *mm, unsigned long address);
+extern pte_t *pte_alloc_one_kernel(struct mm_struct *mm);
-static inline struct page *pte_alloc_one(struct mm_struct *mm,
- unsigned long address)
+static inline struct page *pte_alloc_one(struct mm_struct *mm)
{
struct page *pte;
pte = alloc_pages(GFP_KERNEL, 0);
diff --git a/arch/openrisc/mm/ioremap.c b/arch/openrisc/mm/ioremap.c
index 2175e4bfd9fc..24fb1021c75a 100644
--- a/arch/openrisc/mm/ioremap.c
+++ b/arch/openrisc/mm/ioremap.c
@@ -118,8 +118,7 @@ EXPORT_SYMBOL(iounmap);
* the memblock infrastructure.
*/
-pte_t __ref *pte_alloc_one_kernel(struct mm_struct *mm,
- unsigned long address)
+pte_t __ref *pte_alloc_one_kernel(struct mm_struct *mm)
{
pte_t *pte;
diff --git a/arch/parisc/include/asm/pgalloc.h b/arch/parisc/include/asm/pgalloc.h
index cf13275f7c6d..d05c678c77c4 100644
--- a/arch/parisc/include/asm/pgalloc.h
+++ b/arch/parisc/include/asm/pgalloc.h
@@ -122,7 +122,7 @@ pmd_populate_kernel(struct mm_struct *mm, pmd_t *pmd, pte_t *pte)
#define pmd_pgtable(pmd) pmd_page(pmd)
static inline pgtable_t
-pte_alloc_one(struct mm_struct *mm, unsigned long address)
+pte_alloc_one(struct mm_struct *mm)
{
struct page *page = alloc_page(GFP_KERNEL|__GFP_ZERO);
if (!page)
@@ -135,7 +135,7 @@ pte_alloc_one(struct mm_struct *mm, unsigned long address)
}
static inline pte_t *
-pte_alloc_one_kernel(struct mm_struct *mm, unsigned long addr)
+pte_alloc_one_kernel(struct mm_struct *mm)
{
pte_t *pte = (pte_t *)__get_free_page(GFP_KERNEL|__GFP_ZERO);
return pte;
diff --git a/arch/powerpc/include/asm/book3s/32/pgalloc.h b/arch/powerpc/include/asm/book3s/32/pgalloc.h
index 82e44b1a00ae..af9e13555d95 100644
--- a/arch/powerpc/include/asm/book3s/32/pgalloc.h
+++ b/arch/powerpc/include/asm/book3s/32/pgalloc.h
@@ -82,8 +82,8 @@ static inline void pmd_populate(struct mm_struct *mm, pmd_t *pmdp,
#define pmd_pgtable(pmd) pmd_page(pmd)
#endif
-extern pte_t *pte_alloc_one_kernel(struct mm_struct *mm, unsigned long addr);
-extern pgtable_t pte_alloc_one(struct mm_struct *mm, unsigned long addr);
+extern pte_t *pte_alloc_one_kernel(struct mm_struct *mm);
+extern pgtable_t pte_alloc_one(struct mm_struct *mm);
static inline void pte_free_kernel(struct mm_struct *mm, pte_t *pte)
{
diff --git a/arch/powerpc/include/asm/book3s/64/pgalloc.h b/arch/powerpc/include/asm/book3s/64/pgalloc.h
index 391ed2c3b697..8f1d92e99fe5 100644
--- a/arch/powerpc/include/asm/book3s/64/pgalloc.h
+++ b/arch/powerpc/include/asm/book3s/64/pgalloc.h
@@ -42,7 +42,7 @@ extern struct kmem_cache *pgtable_cache[];
pgtable_cache[(shift) - 1]; \
})
-extern pte_t *pte_fragment_alloc(struct mm_struct *, unsigned long, int);
+extern pte_t *pte_fragment_alloc(struct mm_struct *, int);
extern pmd_t *pmd_fragment_alloc(struct mm_struct *, unsigned long);
extern void pte_fragment_free(unsigned long *, int);
extern void pmd_fragment_free(unsigned long *);
@@ -192,16 +192,14 @@ static inline pgtable_t pmd_pgtable(pmd_t pmd)
return (pgtable_t)pmd_page_vaddr(pmd);
}
-static inline pte_t *pte_alloc_one_kernel(struct mm_struct *mm,
- unsigned long address)
+static inline pte_t *pte_alloc_one_kernel(struct mm_struct *mm)
{
- return (pte_t *)pte_fragment_alloc(mm, address, 1);
+ return (pte_t *)pte_fragment_alloc(mm, 1);
}
-static inline pgtable_t pte_alloc_one(struct mm_struct *mm,
- unsigned long address)
+static inline pgtable_t pte_alloc_one(struct mm_struct *mm)
{
- return (pgtable_t)pte_fragment_alloc(mm, address, 0);
+ return (pgtable_t)pte_fragment_alloc(mm, 0);
}
static inline void pte_free_kernel(struct mm_struct *mm, pte_t *pte)
diff --git a/arch/powerpc/include/asm/nohash/32/pgalloc.h b/arch/powerpc/include/asm/nohash/32/pgalloc.h
index 8825953c225b..16623f53f0d4 100644
--- a/arch/powerpc/include/asm/nohash/32/pgalloc.h
+++ b/arch/powerpc/include/asm/nohash/32/pgalloc.h
@@ -83,8 +83,8 @@ static inline void pmd_populate(struct mm_struct *mm, pmd_t *pmdp,
#define pmd_pgtable(pmd) pmd_page(pmd)
#endif
-extern pte_t *pte_alloc_one_kernel(struct mm_struct *mm, unsigned long addr);
-extern pgtable_t pte_alloc_one(struct mm_struct *mm, unsigned long addr);
+extern pte_t *pte_alloc_one_kernel(struct mm_struct *mm);
+extern pgtable_t pte_alloc_one(struct mm_struct *mm);
static inline void pte_free_kernel(struct mm_struct *mm, pte_t *pte)
{
diff --git a/arch/powerpc/include/asm/nohash/64/pgalloc.h b/arch/powerpc/include/asm/nohash/64/pgalloc.h
index e2d62d033708..2e7e0230edf4 100644
--- a/arch/powerpc/include/asm/nohash/64/pgalloc.h
+++ b/arch/powerpc/include/asm/nohash/64/pgalloc.h
@@ -96,14 +96,12 @@ static inline void pmd_free(struct mm_struct *mm, pmd_t *pmd)
}
-static inline pte_t *pte_alloc_one_kernel(struct mm_struct *mm,
- unsigned long address)
+static inline pte_t *pte_alloc_one_kernel(struct mm_struct *mm)
{
return (pte_t *)__get_free_page(GFP_KERNEL | __GFP_ZERO);
}
-static inline pgtable_t pte_alloc_one(struct mm_struct *mm,
- unsigned long address)
+static inline pgtable_t pte_alloc_one(struct mm_struct *mm)
{
struct page *page;
pte_t *pte;
diff --git a/arch/powerpc/mm/pgtable-book3s64.c b/arch/powerpc/mm/pgtable-book3s64.c
index 01d7c0f7c4f0..cff1d426ca6a 100644
--- a/arch/powerpc/mm/pgtable-book3s64.c
+++ b/arch/powerpc/mm/pgtable-book3s64.c
@@ -379,7 +379,7 @@ static pte_t *__alloc_for_ptecache(struct mm_struct *mm, int kernel)
return (pte_t *)ret;
}
-pte_t *pte_fragment_alloc(struct mm_struct *mm, unsigned long vmaddr, int kernel)
+pte_t *pte_fragment_alloc(struct mm_struct *mm, int kernel)
{
pte_t *pte;
diff --git a/arch/powerpc/mm/pgtable_32.c b/arch/powerpc/mm/pgtable_32.c
index 120a49bfb9c6..b99a89cdcc5e 100644
--- a/arch/powerpc/mm/pgtable_32.c
+++ b/arch/powerpc/mm/pgtable_32.c
@@ -43,7 +43,7 @@ EXPORT_SYMBOL(ioremap_bot); /* aka VMALLOC_END */
extern char etext[], _stext[], _sinittext[], _einittext[];
-__ref pte_t *pte_alloc_one_kernel(struct mm_struct *mm, unsigned long address)
+__ref pte_t *pte_alloc_one_kernel(struct mm_struct *mm)
{
pte_t *pte;
@@ -57,7 +57,7 @@ __ref pte_t *pte_alloc_one_kernel(struct mm_struct *mm, unsigned long address)
return pte;
}
-pgtable_t pte_alloc_one(struct mm_struct *mm, unsigned long address)
+pgtable_t pte_alloc_one(struct mm_struct *mm)
{
struct page *ptepage;
diff --git a/arch/riscv/include/asm/pgalloc.h b/arch/riscv/include/asm/pgalloc.h
index a79ed5faff3a..94043cf83c90 100644
--- a/arch/riscv/include/asm/pgalloc.h
+++ b/arch/riscv/include/asm/pgalloc.h
@@ -82,15 +82,13 @@ static inline void pmd_free(struct mm_struct *mm, pmd_t *pmd)
#endif /* __PAGETABLE_PMD_FOLDED */
-static inline pte_t *pte_alloc_one_kernel(struct mm_struct *mm,
- unsigned long address)
+static inline pte_t *pte_alloc_one_kernel(struct mm_struct *mm)
{
return (pte_t *)__get_free_page(
GFP_KERNEL | __GFP_RETRY_MAYFAIL | __GFP_ZERO);
}
-static inline struct page *pte_alloc_one(struct mm_struct *mm,
- unsigned long address)
+static inline struct page *pte_alloc_one(struct mm_struct *mm)
{
struct page *pte;
diff --git a/arch/s390/include/asm/pgalloc.h b/arch/s390/include/asm/pgalloc.h
index f0f9bcf94c03..ce2ca8cbd2ec 100644
--- a/arch/s390/include/asm/pgalloc.h
+++ b/arch/s390/include/asm/pgalloc.h
@@ -139,8 +139,8 @@ static inline void pmd_populate(struct mm_struct *mm,
/*
* page table entry allocation/free routines.
*/
-#define pte_alloc_one_kernel(mm, vmaddr) ((pte_t *) page_table_alloc(mm))
-#define pte_alloc_one(mm, vmaddr) ((pte_t *) page_table_alloc(mm))
+#define pte_alloc_one_kernel(mm) ((pte_t *)page_table_alloc(mm))
+#define pte_alloc_one(mm) ((pte_t *)page_table_alloc(mm))
#define pte_free_kernel(mm, pte) page_table_free(mm, (unsigned long *) pte)
#define pte_free(mm, pte) page_table_free(mm, (unsigned long *) pte)
diff --git a/arch/sh/include/asm/pgalloc.h b/arch/sh/include/asm/pgalloc.h
index ed053a359ab7..8ad73cb31121 100644
--- a/arch/sh/include/asm/pgalloc.h
+++ b/arch/sh/include/asm/pgalloc.h
@@ -32,14 +32,12 @@ static inline void pmd_populate(struct mm_struct *mm, pmd_t *pmd,
/*
* Allocate and free page tables.
*/
-static inline pte_t *pte_alloc_one_kernel(struct mm_struct *mm,
- unsigned long address)
+static inline pte_t *pte_alloc_one_kernel(struct mm_struct *mm)
{
return quicklist_alloc(QUICK_PT, GFP_KERNEL, NULL);
}
-static inline pgtable_t pte_alloc_one(struct mm_struct *mm,
- unsigned long address)
+static inline pgtable_t pte_alloc_one(struct mm_struct *mm)
{
struct page *page;
void *pg;
diff --git a/arch/sparc/include/asm/pgalloc_32.h b/arch/sparc/include/asm/pgalloc_32.h
index 90459481c6c7..282be50a4adf 100644
--- a/arch/sparc/include/asm/pgalloc_32.h
+++ b/arch/sparc/include/asm/pgalloc_32.h
@@ -58,10 +58,9 @@ void pmd_populate(struct mm_struct *mm, pmd_t *pmdp, struct page *ptep);
void pmd_set(pmd_t *pmdp, pte_t *ptep);
#define pmd_populate_kernel(MM, PMD, PTE) pmd_set(PMD, PTE)
-pgtable_t pte_alloc_one(struct mm_struct *mm, unsigned long address);
+pgtable_t pte_alloc_one(struct mm_struct *mm);
-static inline pte_t *pte_alloc_one_kernel(struct mm_struct *mm,
- unsigned long address)
+static inline pte_t *pte_alloc_one_kernel(struct mm_struct *mm)
{
return srmmu_get_nocache(PTE_SIZE, PTE_SIZE);
}
diff --git a/arch/sparc/include/asm/pgalloc_64.h b/arch/sparc/include/asm/pgalloc_64.h
index 874632f34f62..48abccba4991 100644
--- a/arch/sparc/include/asm/pgalloc_64.h
+++ b/arch/sparc/include/asm/pgalloc_64.h
@@ -60,10 +60,8 @@ static inline void pmd_free(struct mm_struct *mm, pmd_t *pmd)
kmem_cache_free(pgtable_cache, pmd);
}
-pte_t *pte_alloc_one_kernel(struct mm_struct *mm,
- unsigned long address);
-pgtable_t pte_alloc_one(struct mm_struct *mm,
- unsigned long address);
+pte_t *pte_alloc_one_kernel(struct mm_struct *mm);
+pgtable_t pte_alloc_one(struct mm_struct *mm);
void pte_free_kernel(struct mm_struct *mm, pte_t *pte);
void pte_free(struct mm_struct *mm, pgtable_t ptepage);
diff --git a/arch/sparc/mm/init_64.c b/arch/sparc/mm/init_64.c
index f396048a0d68..6133f21811e9 100644
--- a/arch/sparc/mm/init_64.c
+++ b/arch/sparc/mm/init_64.c
@@ -2921,8 +2921,7 @@ void __flush_tlb_all(void)
: : "r" (pstate));
}
-pte_t *pte_alloc_one_kernel(struct mm_struct *mm,
- unsigned long address)
+pte_t *pte_alloc_one_kernel(struct mm_struct *mm)
{
struct page *page = alloc_page(GFP_KERNEL | __GFP_ZERO);
pte_t *pte = NULL;
@@ -2933,8 +2932,7 @@ pte_t *pte_alloc_one_kernel(struct mm_struct *mm,
return pte;
}
-pgtable_t pte_alloc_one(struct mm_struct *mm,
- unsigned long address)
+pgtable_t pte_alloc_one(struct mm_struct *mm)
{
struct page *page = alloc_page(GFP_KERNEL | __GFP_ZERO);
if (!page)
diff --git a/arch/sparc/mm/srmmu.c b/arch/sparc/mm/srmmu.c
index be9cb0065179..ce67a96e70c3 100644
--- a/arch/sparc/mm/srmmu.c
+++ b/arch/sparc/mm/srmmu.c
@@ -364,12 +364,12 @@ pgd_t *get_pgd_fast(void)
* Alignments up to the page size are the same for physical and virtual
* addresses of the nocache area.
*/
-pgtable_t pte_alloc_one(struct mm_struct *mm, unsigned long address)
+pgtable_t pte_alloc_one(struct mm_struct *mm)
{
unsigned long pte;
struct page *page;
- if ((pte = (unsigned long)pte_alloc_one_kernel(mm, address)) == 0)
+ if ((pte = (unsigned long)pte_alloc_one_kernel(mm)) == 0)
return NULL;
page = pfn_to_page(__nocache_pa(pte) >> PAGE_SHIFT);
if (!pgtable_page_ctor(page)) {
diff --git a/arch/um/include/asm/pgalloc.h b/arch/um/include/asm/pgalloc.h
index bf90b2aa2002..99eb5682792a 100644
--- a/arch/um/include/asm/pgalloc.h
+++ b/arch/um/include/asm/pgalloc.h
@@ -25,8 +25,8 @@
extern pgd_t *pgd_alloc(struct mm_struct *);
extern void pgd_free(struct mm_struct *mm, pgd_t *pgd);
-extern pte_t *pte_alloc_one_kernel(struct mm_struct *, unsigned long);
-extern pgtable_t pte_alloc_one(struct mm_struct *, unsigned long);
+extern pte_t *pte_alloc_one_kernel(struct mm_struct *);
+extern pgtable_t pte_alloc_one(struct mm_struct *);
static inline void pte_free_kernel(struct mm_struct *mm, pte_t *pte)
{
diff --git a/arch/um/kernel/mem.c b/arch/um/kernel/mem.c
index 3c0e470ea646..1f277191fbf3 100644
--- a/arch/um/kernel/mem.c
+++ b/arch/um/kernel/mem.c
@@ -197,7 +197,7 @@ void pgd_free(struct mm_struct *mm, pgd_t *pgd)
free_page((unsigned long) pgd);
}
-pte_t *pte_alloc_one_kernel(struct mm_struct *mm, unsigned long address)
+pte_t *pte_alloc_one_kernel(struct mm_struct *mm)
{
pte_t *pte;
@@ -205,7 +205,7 @@ pte_t *pte_alloc_one_kernel(struct mm_struct *mm, unsigned long address)
return pte;
}
-pgtable_t pte_alloc_one(struct mm_struct *mm, unsigned long address)
+pgtable_t pte_alloc_one(struct mm_struct *mm)
{
struct page *pte;
diff --git a/arch/unicore32/include/asm/pgalloc.h b/arch/unicore32/include/asm/pgalloc.h
index f0fdb268f8f2..7cceabecf4e3 100644
--- a/arch/unicore32/include/asm/pgalloc.h
+++ b/arch/unicore32/include/asm/pgalloc.h
@@ -34,7 +34,7 @@ extern void free_pgd_slow(struct mm_struct *mm, pgd_t *pgd);
* Allocate one PTE table.
*/
static inline pte_t *
-pte_alloc_one_kernel(struct mm_struct *mm, unsigned long addr)
+pte_alloc_one_kernel(struct mm_struct *mm)
{
pte_t *pte;
@@ -46,7 +46,7 @@ pte_alloc_one_kernel(struct mm_struct *mm, unsigned long addr)
}
static inline pgtable_t
-pte_alloc_one(struct mm_struct *mm, unsigned long addr)
+pte_alloc_one(struct mm_struct *mm)
{
struct page *pte;
diff --git a/arch/x86/include/asm/pgalloc.h b/arch/x86/include/asm/pgalloc.h
index fbd578daa66e..5068e85165b2 100644
--- a/arch/x86/include/asm/pgalloc.h
+++ b/arch/x86/include/asm/pgalloc.h
@@ -47,8 +47,8 @@ extern gfp_t __userpte_alloc_gfp;
extern pgd_t *pgd_alloc(struct mm_struct *);
extern void pgd_free(struct mm_struct *mm, pgd_t *pgd);
-extern pte_t *pte_alloc_one_kernel(struct mm_struct *, unsigned long);
-extern pgtable_t pte_alloc_one(struct mm_struct *, unsigned long);
+extern pte_t *pte_alloc_one_kernel(struct mm_struct *);
+extern pgtable_t pte_alloc_one(struct mm_struct *);
/* Should really implement gc for free page table pages. This could be
done with a reference count in struct page. */
diff --git a/arch/x86/mm/pgtable.c b/arch/x86/mm/pgtable.c
index 089e78c4effd..a2eff247377b 100644
--- a/arch/x86/mm/pgtable.c
+++ b/arch/x86/mm/pgtable.c
@@ -23,12 +23,12 @@ EXPORT_SYMBOL(physical_mask);
gfp_t __userpte_alloc_gfp = PGALLOC_GFP | PGALLOC_USER_GFP;
-pte_t *pte_alloc_one_kernel(struct mm_struct *mm, unsigned long address)
+pte_t *pte_alloc_one_kernel(struct mm_struct *mm)
{
return (pte_t *)__get_free_page(PGALLOC_GFP & ~__GFP_ACCOUNT);
}
-pgtable_t pte_alloc_one(struct mm_struct *mm, unsigned long address)
+pgtable_t pte_alloc_one(struct mm_struct *mm)
{
struct page *pte;
diff --git a/arch/xtensa/include/asm/pgalloc.h b/arch/xtensa/include/asm/pgalloc.h
index 1065bc8bcae5..b3b388ff2f01 100644
--- a/arch/xtensa/include/asm/pgalloc.h
+++ b/arch/xtensa/include/asm/pgalloc.h
@@ -38,8 +38,7 @@ static inline void pgd_free(struct mm_struct *mm, pgd_t *pgd)
free_page((unsigned long)pgd);
}
-static inline pte_t *pte_alloc_one_kernel(struct mm_struct *mm,
- unsigned long address)
+static inline pte_t *pte_alloc_one_kernel(struct mm_struct *mm)
{
pte_t *ptep;
int i;
@@ -52,13 +51,12 @@ static inline pte_t *pte_alloc_one_kernel(struct mm_struct *mm,
return ptep;
}
-static inline pgtable_t pte_alloc_one(struct mm_struct *mm,
- unsigned long addr)
+static inline pgtable_t pte_alloc_one(struct mm_struct *mm)
{
pte_t *pte;
struct page *page;
- pte = pte_alloc_one_kernel(mm, addr);
+ pte = pte_alloc_one_kernel(mm);
if (!pte)
return NULL;
page = virt_to_page(pte);
diff --git a/include/linux/mm.h b/include/linux/mm.h
index 0416a7204be3..43ce50edc499 100644
--- a/include/linux/mm.h
+++ b/include/linux/mm.h
@@ -1789,8 +1789,8 @@ static inline void mm_inc_nr_ptes(struct mm_struct *mm) {}
static inline void mm_dec_nr_ptes(struct mm_struct *mm) {}
#endif
-int __pte_alloc(struct mm_struct *mm, pmd_t *pmd, unsigned long address);
-int __pte_alloc_kernel(pmd_t *pmd, unsigned long address);
+int __pte_alloc(struct mm_struct *mm, pmd_t *pmd);
+int __pte_alloc_kernel(pmd_t *pmd);
/*
* The following ifdef needed to get the 4level-fixup.h header to work.
@@ -1928,18 +1928,17 @@ static inline void pgtable_page_dtor(struct page *page)
pte_unmap(pte); \
} while (0)
-#define pte_alloc(mm, pmd, address) \
- (unlikely(pmd_none(*(pmd))) && __pte_alloc(mm, pmd, address))
+#define pte_alloc(mm, pmd) (unlikely(pmd_none(*(pmd))) && __pte_alloc(mm, pmd))
#define pte_alloc_map(mm, pmd, address) \
- (pte_alloc(mm, pmd, address) ? NULL : pte_offset_map(pmd, address))
+ (pte_alloc(mm, pmd) ? NULL : pte_offset_map(pmd, address))
#define pte_alloc_map_lock(mm, pmd, address, ptlp) \
- (pte_alloc(mm, pmd, address) ? \
+ (pte_alloc(mm, pmd) ? \
NULL : pte_offset_map_lock(mm, pmd, address, ptlp))
#define pte_alloc_kernel(pmd, address) \
- ((unlikely(pmd_none(*(pmd))) && __pte_alloc_kernel(pmd, address))? \
+ ((unlikely(pmd_none(*(pmd))) && __pte_alloc_kernel(pmd))? \
NULL: pte_offset_kernel(pmd, address))
#if USE_SPLIT_PMD_PTLOCKS
diff --git a/mm/huge_memory.c b/mm/huge_memory.c
index 00704060b7f7..fd7e8714e5a1 100644
--- a/mm/huge_memory.c
+++ b/mm/huge_memory.c
@@ -558,7 +558,7 @@ static vm_fault_t __do_huge_pmd_anonymous_page(struct vm_fault *vmf,
return VM_FAULT_FALLBACK;
}
- pgtable = pte_alloc_one(vma->vm_mm, haddr);
+ pgtable = pte_alloc_one(vma->vm_mm);
if (unlikely(!pgtable)) {
ret = VM_FAULT_OOM;
goto release;
@@ -683,7 +683,7 @@ vm_fault_t do_huge_pmd_anonymous_page(struct vm_fault *vmf)
struct page *zero_page;
bool set;
vm_fault_t ret;
- pgtable = pte_alloc_one(vma->vm_mm, haddr);
+ pgtable = pte_alloc_one(vma->vm_mm);
if (unlikely(!pgtable))
return VM_FAULT_OOM;
zero_page = mm_get_huge_zero_page(vma->vm_mm);
@@ -772,7 +772,7 @@ vm_fault_t vmf_insert_pfn_pmd(struct vm_area_struct *vma, unsigned long addr,
return VM_FAULT_SIGBUS;
if (arch_needs_pgtable_deposit()) {
- pgtable = pte_alloc_one(vma->vm_mm, addr);
+ pgtable = pte_alloc_one(vma->vm_mm);
if (!pgtable)
return VM_FAULT_OOM;
}
@@ -910,7 +910,7 @@ int copy_huge_pmd(struct mm_struct *dst_mm, struct mm_struct *src_mm,
if (!vma_is_anonymous(vma))
return 0;
- pgtable = pte_alloc_one(dst_mm, addr);
+ pgtable = pte_alloc_one(dst_mm);
if (unlikely(!pgtable))
goto out;
diff --git a/mm/kasan/kasan_init.c b/mm/kasan/kasan_init.c
index 7a2a2f13f86f..272849cd2007 100644
--- a/mm/kasan/kasan_init.c
+++ b/mm/kasan/kasan_init.c
@@ -121,7 +121,7 @@ static int __ref zero_pmd_populate(pud_t *pud, unsigned long addr,
pte_t *p;
if (slab_is_available())
- p = pte_alloc_one_kernel(&init_mm, addr);
+ p = pte_alloc_one_kernel(&init_mm);
else
p = early_alloc(PAGE_SIZE, NUMA_NO_NODE);
if (!p)
diff --git a/mm/memory.c b/mm/memory.c
index c467102a5cbc..3afdcf38993d 100644
--- a/mm/memory.c
+++ b/mm/memory.c
@@ -647,10 +647,10 @@ void free_pgtables(struct mmu_gather *tlb, struct vm_area_struct *vma,
}
}
-int __pte_alloc(struct mm_struct *mm, pmd_t *pmd, unsigned long address)
+int __pte_alloc(struct mm_struct *mm, pmd_t *pmd)
{
spinlock_t *ptl;
- pgtable_t new = pte_alloc_one(mm, address);
+ pgtable_t new = pte_alloc_one(mm);
if (!new)
return -ENOMEM;
@@ -681,9 +681,9 @@ int __pte_alloc(struct mm_struct *mm, pmd_t *pmd, unsigned long address)
return 0;
}
-int __pte_alloc_kernel(pmd_t *pmd, unsigned long address)
+int __pte_alloc_kernel(pmd_t *pmd)
{
- pte_t *new = pte_alloc_one_kernel(&init_mm, address);
+ pte_t *new = pte_alloc_one_kernel(&init_mm);
if (!new)
return -ENOMEM;
@@ -3139,7 +3139,7 @@ static vm_fault_t do_anonymous_page(struct vm_fault *vmf)
*
* Here we only have down_read(mmap_sem).
*/
- if (pte_alloc(vma->vm_mm, vmf->pmd, vmf->address))
+ if (pte_alloc(vma->vm_mm, vmf->pmd))
return VM_FAULT_OOM;
/* See the comment in pte_alloc_one_map() */
@@ -3286,7 +3286,7 @@ static vm_fault_t pte_alloc_one_map(struct vm_fault *vmf)
pmd_populate(vma->vm_mm, vmf->pmd, vmf->prealloc_pte);
spin_unlock(vmf->ptl);
vmf->prealloc_pte = NULL;
- } else if (unlikely(pte_alloc(vma->vm_mm, vmf->pmd, vmf->address))) {
+ } else if (unlikely(pte_alloc(vma->vm_mm, vmf->pmd))) {
return VM_FAULT_OOM;
}
map_pte:
@@ -3365,7 +3365,7 @@ static vm_fault_t do_set_pmd(struct vm_fault *vmf, struct page *page)
* related to pte entry. Use the preallocated table for that.
*/
if (arch_needs_pgtable_deposit() && !vmf->prealloc_pte) {
- vmf->prealloc_pte = pte_alloc_one(vma->vm_mm, vmf->address);
+ vmf->prealloc_pte = pte_alloc_one(vma->vm_mm);
if (!vmf->prealloc_pte)
return VM_FAULT_OOM;
smp_wmb(); /* See comment in __pte_alloc() */
@@ -3603,8 +3603,7 @@ static vm_fault_t do_fault_around(struct vm_fault *vmf)
start_pgoff + nr_pages - 1);
if (pmd_none(*vmf->pmd)) {
- vmf->prealloc_pte = pte_alloc_one(vmf->vma->vm_mm,
- vmf->address);
+ vmf->prealloc_pte = pte_alloc_one(vmf->vma->vm_mm);
if (!vmf->prealloc_pte)
goto out;
smp_wmb(); /* See comment in __pte_alloc() */
diff --git a/mm/migrate.c b/mm/migrate.c
index 84381b55b2bd..3080b0626026 100644
--- a/mm/migrate.c
+++ b/mm/migrate.c
@@ -2605,7 +2605,7 @@ static void migrate_vma_insert_page(struct migrate_vma *migrate,
*
* Here we only have down_read(mmap_sem).
*/
- if (pte_alloc(mm, pmdp, addr))
+ if (pte_alloc(mm, pmdp))
goto abort;
/* See the comment in pte_alloc_one_map() */
diff --git a/mm/mremap.c b/mm/mremap.c
index 5c2e18505f75..9e68a02a52b1 100644
--- a/mm/mremap.c
+++ b/mm/mremap.c
@@ -240,7 +240,7 @@ unsigned long move_page_tables(struct vm_area_struct *vma,
if (pmd_trans_unstable(old_pmd))
continue;
}
- if (pte_alloc(new_vma->vm_mm, new_pmd, new_addr))
+ if (pte_alloc(new_vma->vm_mm, new_pmd))
break;
next = (new_addr + PMD_SIZE) & PMD_MASK;
if (extent > next - new_addr)
diff --git a/mm/userfaultfd.c b/mm/userfaultfd.c
index 5029f241908f..f05c8bc38ca5 100644
--- a/mm/userfaultfd.c
+++ b/mm/userfaultfd.c
@@ -513,7 +513,7 @@ static __always_inline ssize_t __mcopy_atomic(struct mm_struct *dst_mm,
break;
}
if (unlikely(pmd_none(dst_pmdval)) &&
- unlikely(__pte_alloc(dst_mm, dst_pmd, dst_addr))) {
+ unlikely(__pte_alloc(dst_mm, dst_pmd))) {
err = -ENOMEM;
break;
}
diff --git a/virt/kvm/arm/mmu.c b/virt/kvm/arm/mmu.c
index ed162a6c57c5..3f8180414301 100644
--- a/virt/kvm/arm/mmu.c
+++ b/virt/kvm/arm/mmu.c
@@ -628,7 +628,7 @@ static int create_hyp_pmd_mappings(pud_t *pud, unsigned long start,
BUG_ON(pmd_sect(*pmd));
if (pmd_none(*pmd)) {
- pte = pte_alloc_one_kernel(NULL, addr);
+ pte = pte_alloc_one_kernel(NULL);
if (!pte) {
kvm_err("Cannot allocate Hyp pte\n");
return -ENOMEM;
--
2.19.0.605.g01d371f741-goog
^ permalink raw reply related
* [PATCH 2/4] mm: speed up mremap by 500x on large regions (v2)
From: Joel Fernandes (Google) @ 2018-10-13 1:31 UTC (permalink / raw)
To: linux-kernel
Cc: linux-mips, Rich Felker, linux-ia64, linux-sh, Peter Zijlstra,
Catalin Marinas, Dave Hansen, Will Deacon, mhocko, linux-mm,
lokeshgidra, Joel Fernandes (Google), linux-riscv, elfring,
Jonas Bonn, kvmarm, dancol, Yoshinori Sato, sparclinux,
linux-xtensa, linux-hexagon, Helge Deller,
maintainer:X86 ARCHITECTURE 32-BIT AND 64-BIT, hughd,
James E.J. Bottomley, kasan-dev, anton.ivanov, Ingo Molnar,
Geert Uytterhoeven, Andrey Ryabinin, linux-snps-arc, kernel-team,
Sam Creasey, Fenghua Yu, linux-s390, Jeff Dike, linux-um,
Stefan Kristiansson, Julia Lawall, linux-m68k, Borislav Petkov,
Andy Lutomirski, nios2-dev, kirill, Stafford Horne, Guan Xuetao,
Chris Zankel, Tony Luck, Richard Weinberger, linux-parisc, pantin,
Max Filippov, minchan, Thomas Gleixner, linux-alpha, Ley Foon Tan,
akpm, linuxppc-dev, David S. Miller
In-Reply-To: <20181013013200.206928-1-joel@joelfernandes.org>
Android needs to mremap large regions of memory during memory management
related operations. The mremap system call can be really slow if THP is
not enabled. The bottleneck is move_page_tables, which is copying each
pte at a time, and can be really slow across a large map. Turning on THP
may not be a viable option, and is not for us. This patch speeds up the
performance for non-THP system by copying at the PMD level when possible.
The speed up is three orders of magnitude. On a 1GB mremap, the mremap
completion times drops from 160-250 millesconds to 380-400 microseconds.
Before:
Total mremap time for 1GB data: 242321014 nanoseconds.
Total mremap time for 1GB data: 196842467 nanoseconds.
Total mremap time for 1GB data: 167051162 nanoseconds.
After:
Total mremap time for 1GB data: 385781 nanoseconds.
Total mremap time for 1GB data: 388959 nanoseconds.
Total mremap time for 1GB data: 402813 nanoseconds.
Incase THP is enabled, the optimization is skipped. I also flush the
tlb every time we do this optimization since I couldn't find a way to
determine if the low-level PTEs are dirty. It is seen that the cost of
doing so is not much compared the improvement, on both x86-64 and arm64.
Cc: minchan@kernel.org
Cc: pantin@google.com
Cc: hughd@google.com
Cc: lokeshgidra@google.com
Cc: dancol@google.com
Cc: mhocko@kernel.org
Cc: kirill@shutemov.name
Cc: akpm@linux-foundation.org
Cc: kernel-team@android.com
Signed-off-by: Joel Fernandes (Google) <joel@joelfernandes.org>
---
arch/Kconfig | 5 ++++
mm/mremap.c | 65 ++++++++++++++++++++++++++++++++++++++++++++++++++++
2 files changed, 70 insertions(+)
diff --git a/arch/Kconfig b/arch/Kconfig
index 6801123932a5..9724fe39884f 100644
--- a/arch/Kconfig
+++ b/arch/Kconfig
@@ -518,6 +518,11 @@ config HAVE_IRQ_TIME_ACCOUNTING
Archs need to ensure they use a high enough resolution clock to
support irq time accounting and then call enable_sched_clock_irqtime().
+config HAVE_MOVE_PMD
+ bool
+ help
+ Archs that select this are able to move page tables at the PMD level.
+
config HAVE_ARCH_TRANSPARENT_HUGEPAGE
bool
diff --git a/mm/mremap.c b/mm/mremap.c
index 9e68a02a52b1..2fd163cff406 100644
--- a/mm/mremap.c
+++ b/mm/mremap.c
@@ -191,6 +191,54 @@ static void move_ptes(struct vm_area_struct *vma, pmd_t *old_pmd,
drop_rmap_locks(vma);
}
+static bool move_normal_pmd(struct vm_area_struct *vma, unsigned long old_addr,
+ unsigned long new_addr, unsigned long old_end,
+ pmd_t *old_pmd, pmd_t *new_pmd, bool *need_flush)
+{
+ spinlock_t *old_ptl, *new_ptl;
+ struct mm_struct *mm = vma->vm_mm;
+
+ if ((old_addr & ~PMD_MASK) || (new_addr & ~PMD_MASK)
+ || old_end - old_addr < PMD_SIZE)
+ return false;
+
+ /*
+ * The destination pmd shouldn't be established, free_pgtables()
+ * should have release it.
+ */
+ if (WARN_ON(!pmd_none(*new_pmd)))
+ return false;
+
+ /*
+ * We don't have to worry about the ordering of src and dst
+ * ptlocks because exclusive mmap_sem prevents deadlock.
+ */
+ old_ptl = pmd_lock(vma->vm_mm, old_pmd);
+ if (old_ptl) {
+ pmd_t pmd;
+
+ new_ptl = pmd_lockptr(mm, new_pmd);
+ if (new_ptl != old_ptl)
+ spin_lock_nested(new_ptl, SINGLE_DEPTH_NESTING);
+
+ /* Clear the pmd */
+ pmd = *old_pmd;
+ pmd_clear(old_pmd);
+
+ VM_BUG_ON(!pmd_none(*new_pmd));
+
+ /* Set the new pmd */
+ set_pmd_at(mm, new_addr, new_pmd, pmd);
+ if (new_ptl != old_ptl)
+ spin_unlock(new_ptl);
+ spin_unlock(old_ptl);
+
+ *need_flush = true;
+ return true;
+ }
+ return false;
+}
+
unsigned long move_page_tables(struct vm_area_struct *vma,
unsigned long old_addr, struct vm_area_struct *new_vma,
unsigned long new_addr, unsigned long len,
@@ -239,7 +287,24 @@ unsigned long move_page_tables(struct vm_area_struct *vma,
split_huge_pmd(vma, old_pmd, old_addr);
if (pmd_trans_unstable(old_pmd))
continue;
+ } else if (extent == PMD_SIZE && IS_ENABLED(CONFIG_HAVE_MOVE_PMD)) {
+ /*
+ * If the extent is PMD-sized, try to speed the move by
+ * moving at the PMD level if possible.
+ */
+ bool moved;
+
+ if (need_rmap_locks)
+ take_rmap_locks(vma);
+ moved = move_normal_pmd(vma, old_addr, new_addr,
+ old_end, old_pmd, new_pmd,
+ &need_flush);
+ if (need_rmap_locks)
+ drop_rmap_locks(vma);
+ if (moved)
+ continue;
}
+
if (pte_alloc(new_vma->vm_mm, new_pmd))
break;
next = (new_addr + PMD_SIZE) & PMD_MASK;
--
2.19.0.605.g01d371f741-goog
^ permalink raw reply related
page: next (older) | prev (newer) | latest
- recent:[subjects (threaded)|topics (new)|topics (active)]
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox