All of lore.kernel.org
 help / color / mirror / Atom feed
* [PATCH v2 00/35] Speculative page faults
@ 2022-01-28 13:09 Michel Lespinasse
  2022-01-28 13:09 ` [PATCH v2 01/35] mm: export dump_mm Michel Lespinasse
                   ` (37 more replies)
  0 siblings, 38 replies; 77+ messages in thread
From: Michel Lespinasse @ 2022-01-28 13:09 UTC (permalink / raw)
  To: Linux-MM, linux-kernel, Andrew Morton
  Cc: kernel-team, Laurent Dufour, Jerome Glisse, Peter Zijlstra,
	Michal Hocko, Vlastimil Babka, Davidlohr Bueso, Matthew Wilcox,
	Liam Howlett, Rik van Riel, Paul McKenney, Song Liu,
	Suren Baghdasaryan, Minchan Kim, Joel Fernandes, David Rientjes,
	Axel Rasmussen, Andy Lutomirski, Michel Lespinasse

This patchset is my take on speculative page faults (spf).
It builds on ideas that have been previously proposed by Laurent Dufour,
Peter Zijlstra and others before. While Laurent's previous proposal
was rejected around the time of LSF/MM 2019, I am hoping we can revisit
this now based on what I think is a simpler and more bisectable approach,
much improved scaling numbers in the anonymous vma case, and the Android
use case that has since emerged. I will expand on these points towards
the end of this message.

The patch series applies on top of linux v5.17-rc1;
a git tree is also available:
git fetch https://github.com/lespinasse/linux.git v5.17-rc1-spf-anon

I would like these patches to be considered for inclusion into v5.18.
Several android vendors are using Laurent Dufour's previous SPF work into
their kernel tree in order to improve application startup performance,
want to converge to an upstream accepted solution, and have reported good
numbers with previous versions of this patchset. Also, there is a broader
interest into reducing mmap lock dependencies in critical MM paths,
and I think this patchset would be a good first step in that direction.


This patchset follows the same overall structure as the v1 proposal,
with the following differences:
- Commit 12 (mm: separate mmap locked assertion from find_vma) is new.
- The mmu notifier lock is new; this fixes a race in v1 patchset
  between speculative COW faults and registering new MMU notifiers.
- Speculative handling of swap-cache pages has been removed.
- Commit 30 is new; this fixes build issues that showed in some configs.


In principle it would also be possible to extend this work for handling
file mapped vmas; I have pending work on such patches too but they are
not mature enough to be submitted for inclusion at this point.


Patchset summary:

Classical page fault processing takes the mmap read lock in order to
prevent races with mmap writers. In contrast, speculative fault
processing does not take the mmap read lock, and instead verifies,
when the results of the page fault are about to get committed and
become visible to other threads, that no mmap writers have been
running concurrently with the page fault. If the check fails,
speculative updates do not get committed and the fault is retried
in the usual, non-speculative way (with the mmap read lock held).

The concurrency check is implemented using a per-mm mmap sequence count.
The counter is incremented at the beginning and end of each mmap write
operation. If the counter is initially observed to have an even value,
and has the same value later on, the observer can deduce that no mmap
writers have been running concurrently with it between those two times.
This is similar to a seqlock, except that readers never spin on the
counter value (they would instead revert to taking the mmap read lock),
and writers are allowed to sleep. One benefit of this approach is that
it requires no writer side changes, just some hooks in the mmap write
lock APIs that writers already use.

The first step of a speculative page fault is to look up the vma and
read its contents (currently by making a copy of the vma, though in
principle it would be sufficient to only read the vma attributes that
are used in page faults). The mmap sequence count is used to verify
that there were no mmap writers concurrent to the lookup and copy steps.
Note that walking rbtrees while there may potentially be concurrent
writers is not an entirely new idea in linux, as latched rbtrees
are already doing this. This is safe as long as the lookup is
followed by a sequence check to verify that concurrency did not
actually occur (and abort the speculative fault if it did).

The next step is to walk down the existing page table tree to find the
current pte entry. This is done with interrupts disabled to avoid
races with munmap(). Again, not an entirely new idea, as this repeats
a pattern already present in fast GUP. Similar precautions are also
taken when taking the page table lock.

Breaking COW on an existing mapping may require firing MMU notifiers.
Some care is required to avoid racing with registering new notifiers.
This patchset adds a new per-cpu rwsem to handle this situation.


Commits 1 to 5 are preparatory cleanups.

Commits 6 and 7 introduce CONFIG_SPECULATIVE_PAGE_FAULT and lets us
enable it on x86 so we can test the new code as it gets introduced.

Commits 8 and 9 extend handle_mm_fault() so it can be used for
speculative faults; initially these always abort with VM_FAULT_RETRY.

Commits 10 to 27 progressively implement the speculative handling of
page faults. Importantly, they are structured to be bisectable:
the new code gets enabled every few commits.
- Commit 10 adds the mmap sequence count that will be used for detecting
  when writers have been running concurrently with an spf attempt
  (in which case the attempt will be aborted);
- Commit 11 adds RCU safe vma freeing;
- Commit 12 adds a version of find_vma that doesn't check for mmap locking;
- Commit 13 does a lockless VMA lookup and starts the spf handling attempt;
- Commit 14 introduces an API for preventing page table reclamation
  (using RCU or disabling interrupts depending on build config options);
- (Commit 15 is a small refactor preparing for the next commit);
- Commit 16 walks down the existing page tables, carefully avoiding
  races with potential writers (munmap in particular)
- Commit 17 introduces pte_map_lock() and pte_spinlock(), which attempt
  to (optionally map and) lock an existing page table when it's time to
  commit page fault results to it.
- Commits 18 to 21 implement SPF for the simplest cases
  (do_anonymous_page and do_numa_page). This mostly comes down to
  using the pte_map_lock() and pte_spinlock() APIs where needed,
  and making sure to abort speculation in unsupported cases
  (mostly anon_vma allocation and userfaultfd).
- Commits 22 to 25 add a new mmu_notifier_lock
- Commits 26 and 27 implement some additional SPF cases, using the new
  mmu_notifier_lock for the COW cases.

Commits 28 and 29 disable speculative handling for single threaded
userspace. This is for (minor) performance tuning and is pushed
towards the end of the series to make it easier to exercise the spf
paths as they are introduced.

Commits 30 and 31 add some extra statistics.

Commits 32 to 35 add spf support on the arm64 and powerpc architectures.


Michel Lespinasse (34):
  mm: export dump_mm
  mmap locking API: mmap_lock_is_contended returns a bool
  mmap locking API: name the return values
  do_anonymous_page: use update_mmu_tlb()
  do_anonymous_page: reduce code duplication
  mm: introduce CONFIG_SPECULATIVE_PAGE_FAULT
  x86/mm: define ARCH_SUPPORTS_SPECULATIVE_PAGE_FAULT
  mm: add FAULT_FLAG_SPECULATIVE flag
  mm: add do_handle_mm_fault()
  mm: add per-mm mmap sequence counter for speculative page fault handling.
  mm: rcu safe vma freeing
  mm: separate mmap locked assertion from find_vma
  x86/mm: attempt speculative mm faults first
  mm: add speculative_page_walk_begin() and speculative_page_walk_end()
  mm: refactor __handle_mm_fault() / handle_pte_fault()
  mm: implement speculative handling in __handle_mm_fault().
  mm: add pte_map_lock() and pte_spinlock()
  mm: implement speculative handling in do_anonymous_page()
  mm: enable speculative fault handling through do_anonymous_page()
  mm: implement speculative handling in do_numa_page()
  mm: enable speculative fault handling in do_numa_page()
  mm: add mmu_notifier_lock
  mm: write lock mmu_notifier_lock when registering mmu notifiers
  mm: add mmu_notifier_trylock() and mmu_notifier_unlock()
  mm: implement speculative handling in wp_page_copy()
  mm: implement and enable speculative fault handling in handle_pte_fault()
  mm: disable speculative faults for single threaded user space
  mm: disable rcu safe vma freeing for single threaded user space
  mm: create new include/linux/vm_event.h header file
  mm: anon spf statistics
  arm64/mm: define ARCH_SUPPORTS_SPECULATIVE_PAGE_FAULT
  arm64/mm: attempt speculative mm faults first
  powerpc/mm: define ARCH_SUPPORTS_SPECULATIVE_PAGE_FAULT
  powerpc/mm: attempt speculative mm faults first

Suren Baghdasaryan (1):
  percpu-rwsem: enable percpu_sem destruction in atomic context

 arch/arm64/Kconfig                    |   1 +
 arch/arm64/mm/fault.c                 |  62 ++++
 arch/powerpc/Kconfig                  |   1 +
 arch/powerpc/mm/fault.c               |  64 ++++
 arch/x86/Kconfig                      |   1 +
 arch/x86/mm/fault.c                   |  63 ++++
 drivers/gpu/drm/i915/i915_gpu_error.c |   4 +-
 include/linux/mm.h                    |  68 +++-
 include/linux/mm_types.h              |  33 +-
 include/linux/mmap_lock.h             | 109 ++++--
 include/linux/mmu_notifier.h          |  52 ++-
 include/linux/percpu-rwsem.h          |  13 +-
 include/linux/vm_event.h              | 111 ++++++
 include/linux/vm_event_item.h         |  25 ++
 include/linux/vmstat.h                |  95 +-----
 kernel/fork.c                         |  18 +-
 kernel/locking/percpu-rwsem.c         |  32 ++
 mm/Kconfig                            |  22 ++
 mm/Kconfig.debug                      |   7 +
 mm/debug.c                            |   1 +
 mm/memory.c                           | 474 +++++++++++++++++++-------
 mm/mmap.c                             |  13 +-
 mm/vmstat.c                           |  25 ++
 23 files changed, 1040 insertions(+), 254 deletions(-)
 create mode 100644 include/linux/vm_event.h

-- 
2.20.1



^ permalink raw reply	[flat|nested] 77+ messages in thread
* Re: [PATCH v2 18/35] mm: implement speculative handling in do_anonymous_page()
@ 2022-01-30 18:03 kernel test robot
  0 siblings, 0 replies; 77+ messages in thread
From: kernel test robot @ 2022-01-30 18:03 UTC (permalink / raw)
  To: kbuild

[-- Attachment #1: Type: text/plain, Size: 22826 bytes --]

CC: llvm(a)lists.linux.dev
CC: kbuild-all(a)lists.01.org
In-Reply-To: <20220128131006.67712-19-michel@lespinasse.org>
References: <20220128131006.67712-19-michel@lespinasse.org>
TO: Michel Lespinasse <michel@lespinasse.org>
TO: "Linux-MM" <linux-mm@kvack.org>
TO: linux-kernel(a)vger.kernel.org
TO: Andrew Morton <akpm@linux-foundation.org>
CC: kernel-team(a)fb.com
CC: Laurent Dufour <ldufour@linux.ibm.com>
CC: Jerome Glisse <jglisse@google.com>
CC: Peter Zijlstra <peterz@infradead.org>
CC: Michal Hocko <mhocko@suse.com>
CC: Vlastimil Babka <vbabka@suse.cz>
CC: Davidlohr Bueso <dave@stgolabs.net>

Hi Michel,

Thank you for the patch! Perhaps something to improve:

[auto build test WARNING on linus/master]
[also build test WARNING on v5.17-rc1 next-20220128]
[cannot apply to tip/x86/mm arm64/for-next/core powerpc/next hnaz-mm/master]
[If your patch is applied to the wrong git tree, kindly drop us a note.
And when submitting patch, we suggest to use '--base' as documented in
https://git-scm.com/docs/git-format-patch]

url:    https://github.com/0day-ci/linux/commits/Michel-Lespinasse/Speculative-page-faults/20220128-212122
base:   https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git 145d9b498fc827b79c1260b4caa29a8e59d4c2b9
:::::: branch date: 2 days ago
:::::: commit date: 2 days ago
config: x86_64-randconfig-c007-20220124 (https://download.01.org/0day-ci/archive/20220131/202201310126.IymdD4Vv-lkp(a)intel.com/config)
compiler: clang version 14.0.0 (https://github.com/llvm/llvm-project 33b45ee44b1f32ffdbc995e6fec806271b4b3ba4)
reproduce (this is a W=1 build):
        wget https://raw.githubusercontent.com/intel/lkp-tests/master/sbin/make.cross -O ~/bin/make.cross
        chmod +x ~/bin/make.cross
        # https://github.com/0day-ci/linux/commit/fa5331bae2e49ce86eff959390b451b7401f9156
        git remote add linux-review https://github.com/0day-ci/linux
        git fetch --no-tags linux-review Michel-Lespinasse/Speculative-page-faults/20220128-212122
        git checkout fa5331bae2e49ce86eff959390b451b7401f9156
        # save the config file to linux build tree
        COMPILER_INSTALL_PATH=$HOME/0day COMPILER=clang make.cross ARCH=x86_64 clang-analyzer 

If you fix the issue, kindly add following tag as appropriate
Reported-by: kernel test robot <lkp@intel.com>


clang-analyzer warnings: (new ones prefixed by >>)
                       ^~~~~~~~~~~~~~
   mm/memory.c:2576:22: note: Left side of '&&' is false
                   if (pmd_none(*pmd) && !create)
                                      ^
   mm/memory.c:2578:7: note: Assuming the condition is true
                   if (WARN_ON_ONCE(pmd_leaf(*pmd)))
                       ^
   include/asm-generic/bug.h:104:23: note: expanded from macro 'WARN_ON_ONCE'
           int __ret_warn_on = !!(condition);                      \
                                ^~~~~~~~~~~~
   mm/memory.c:2578:7: note: Taking false branch
                   if (WARN_ON_ONCE(pmd_leaf(*pmd)))
                       ^
   include/asm-generic/bug.h:105:2: note: expanded from macro 'WARN_ON_ONCE'
           if (unlikely(__ret_warn_on))                            \
           ^
   mm/memory.c:2578:3: note: Taking false branch
                   if (WARN_ON_ONCE(pmd_leaf(*pmd)))
                   ^
   mm/memory.c:2580:8: note: Calling 'pmd_none'
                   if (!pmd_none(*pmd) && WARN_ON_ONCE(pmd_bad(*pmd))) {
                        ^~~~~~~~~~~~~~
   arch/x86/include/asm/pgtable.h:797:2: note: Returning zero, which participates in a condition later
           return (val & ~_PAGE_KNL_ERRATUM_MASK) == 0;
           ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
   mm/memory.c:2580:8: note: Returning from 'pmd_none'
                   if (!pmd_none(*pmd) && WARN_ON_ONCE(pmd_bad(*pmd))) {
                        ^~~~~~~~~~~~~~
   mm/memory.c:2580:7: note: Left side of '&&' is true
                   if (!pmd_none(*pmd) && WARN_ON_ONCE(pmd_bad(*pmd))) {
                       ^
   mm/memory.c:2580:26: note: Taking false branch
                   if (!pmd_none(*pmd) && WARN_ON_ONCE(pmd_bad(*pmd))) {
                                          ^
   include/asm-generic/bug.h:105:2: note: expanded from macro 'WARN_ON_ONCE'
           if (unlikely(__ret_warn_on))                            \
           ^
   mm/memory.c:2580:3: note: Taking false branch
                   if (!pmd_none(*pmd) && WARN_ON_ONCE(pmd_bad(*pmd))) {
                   ^
   mm/memory.c:2585:9: note: Calling 'apply_to_pte_range'
                   err = apply_to_pte_range(mm, pmd, addr, next,
                         ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
   mm/memory.c:2520:2: note: 'ptl' declared without an initial value
           spinlock_t *ptl;
           ^~~~~~~~~~~~~~~
   mm/memory.c:2522:6: note: 'create' is false
           if (create) {
               ^~~~~~
   mm/memory.c:2522:2: note: Taking false branch
           if (create) {
           ^
   mm/memory.c:2529:23: note: Assuming the condition is true
                   mapped_pte = pte = (mm == &init_mm) ?
                                       ^~~~~~~~~~~~~~
   mm/memory.c:2529:22: note: '?' condition is true
                   mapped_pte = pte = (mm == &init_mm) ?
                                      ^
   mm/memory.c:2534:2: note: Taking false branch
           BUG_ON(pmd_huge(*pmd));
           ^
   include/asm-generic/bug.h:65:32: note: expanded from macro 'BUG_ON'
   #define BUG_ON(condition) do { if (unlikely(condition)) BUG(); } while (0)
                                  ^
   mm/memory.c:2534:2: note: Loop condition is false.  Exiting loop
           BUG_ON(pmd_huge(*pmd));
           ^
   include/asm-generic/bug.h:65:27: note: expanded from macro 'BUG_ON'
   #define BUG_ON(condition) do { if (unlikely(condition)) BUG(); } while (0)
                             ^
   mm/memory.c:2536:2: note: Loop condition is false.  Exiting loop
           arch_enter_lazy_mmu_mode();
           ^
   include/linux/pgtable.h:985:36: note: expanded from macro 'arch_enter_lazy_mmu_mode'
   #define arch_enter_lazy_mmu_mode()      do {} while (0)
                                           ^
   mm/memory.c:2538:6: note: Assuming 'fn' is null
           if (fn) {
               ^~
   mm/memory.c:2538:2: note: Taking false branch
           if (fn) {
           ^
   mm/memory.c:2549:2: note: Loop condition is false.  Exiting loop
           arch_leave_lazy_mmu_mode();
           ^
   include/linux/pgtable.h:986:36: note: expanded from macro 'arch_leave_lazy_mmu_mode'
   #define arch_leave_lazy_mmu_mode()      do {} while (0)
                                           ^
   mm/memory.c:2551:6: note: Assuming the condition is true
           if (mm != &init_mm)
               ^~~~~~~~~~~~~~
   mm/memory.c:2551:2: note: Taking true branch
           if (mm != &init_mm)
           ^
   mm/memory.c:2552:3: note: 1st function call argument is an uninitialized value
                   pte_unmap_unlock(mapped_pte, ptl);
                   ^
   include/linux/mm.h:2357:2: note: expanded from macro 'pte_unmap_unlock'
           spin_unlock(ptl);                               \
           ^           ~~~
>> mm/memory.c:3876:7: warning: Assigned value is garbage or undefined [clang-analyzer-core.uninitialized.Assign]
           if (!pte_map_lock(vmf)) {
                ^
   include/linux/mm.h:3418:2: note: expanded from macro 'pte_map_lock'
           struct vm_fault *vmf = __vmf;                                   \
           ^
   mm/memory.c:4940:6: note: Assuming the condition is false
           if (flags & FAULT_FLAG_SPECULATIVE)
               ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
   mm/memory.c:4940:2: note: Taking false branch
           if (flags & FAULT_FLAG_SPECULATIVE)
           ^
   mm/memory.c:4943:2: note: Taking false branch
           __set_current_state(TASK_RUNNING);
           ^
   include/linux/sched.h:204:3: note: expanded from macro '__set_current_state'
                   debug_normal_state_change((state_value));               \
                   ^
   include/linux/sched.h:137:3: note: expanded from macro 'debug_normal_state_change'
                   WARN_ON_ONCE(is_special_task_state(state_value));       \
                   ^
   include/asm-generic/bug.h:105:2: note: expanded from macro 'WARN_ON_ONCE'
           if (unlikely(__ret_warn_on))                            \
           ^
   mm/memory.c:4943:2: note: Loop condition is false.  Exiting loop
           __set_current_state(TASK_RUNNING);
           ^
   include/linux/sched.h:204:3: note: expanded from macro '__set_current_state'
                   debug_normal_state_change((state_value));               \
                   ^
   include/linux/sched.h:136:2: note: expanded from macro 'debug_normal_state_change'
           do {                                                            \
           ^
   mm/memory.c:4943:2: note: Left side of '||' is false
           __set_current_state(TASK_RUNNING);
           ^
   include/linux/sched.h:205:3: note: expanded from macro '__set_current_state'
                   WRITE_ONCE(current->__state, (state_value));            \
                   ^
   include/asm-generic/rwonce.h:60:2: note: expanded from macro 'WRITE_ONCE'
           compiletime_assert_rwonce_type(x);                              \
           ^
   include/asm-generic/rwonce.h:36:21: note: expanded from macro 'compiletime_assert_rwonce_type'
           compiletime_assert(__native_word(t) || sizeof(t) == sizeof(long long),  \
                              ^
   include/linux/compiler_types.h:313:3: note: expanded from macro '__native_word'
           (sizeof(t) == sizeof(char) || sizeof(t) == sizeof(short) || \
            ^
   mm/memory.c:4943:2: note: Left side of '||' is false
           __set_current_state(TASK_RUNNING);
           ^
   include/linux/sched.h:205:3: note: expanded from macro '__set_current_state'
                   WRITE_ONCE(current->__state, (state_value));            \
                   ^
   include/asm-generic/rwonce.h:60:2: note: expanded from macro 'WRITE_ONCE'
           compiletime_assert_rwonce_type(x);                              \
           ^
   include/asm-generic/rwonce.h:36:21: note: expanded from macro 'compiletime_assert_rwonce_type'
           compiletime_assert(__native_word(t) || sizeof(t) == sizeof(long long),  \
                              ^
   include/linux/compiler_types.h:313:3: note: expanded from macro '__native_word'
           (sizeof(t) == sizeof(char) || sizeof(t) == sizeof(short) || \
            ^
   mm/memory.c:4943:2: note: Left side of '||' is true
           __set_current_state(TASK_RUNNING);
           ^
   include/linux/sched.h:205:3: note: expanded from macro '__set_current_state'
                   WRITE_ONCE(current->__state, (state_value));            \
                   ^
   include/asm-generic/rwonce.h:60:2: note: expanded from macro 'WRITE_ONCE'
           compiletime_assert_rwonce_type(x);                              \
           ^
   include/asm-generic/rwonce.h:36:21: note: expanded from macro 'compiletime_assert_rwonce_type'
           compiletime_assert(__native_word(t) || sizeof(t) == sizeof(long long),  \
                              ^
   include/linux/compiler_types.h:314:28: note: expanded from macro '__native_word'
            sizeof(t) == sizeof(int) || sizeof(t) == sizeof(long))
                                     ^
   mm/memory.c:4943:2: note: Taking false branch
           __set_current_state(TASK_RUNNING);
           ^
   include/linux/sched.h:205:3: note: expanded from macro '__set_current_state'
                   WRITE_ONCE(current->__state, (state_value));            \
                   ^
   include/asm-generic/rwonce.h:60:2: note: expanded from macro 'WRITE_ONCE'
           compiletime_assert_rwonce_type(x);                              \
           ^
   include/asm-generic/rwonce.h:36:2: note: expanded from macro 'compiletime_assert_rwonce_type'
           compiletime_assert(__native_word(t) || sizeof(t) == sizeof(long long),  \
           ^
   include/linux/compiler_types.h:346:2: note: expanded from macro 'compiletime_assert'
           _compiletime_assert(condition, msg, __compiletime_assert_, __COUNTER__)
           ^
   include/linux/compiler_types.h:334:2: note: expanded from macro '_compiletime_assert'
           __compiletime_assert(condition, msg, prefix, suffix)
           ^
   include/linux/compiler_types.h:326:3: note: expanded from macro '__compiletime_assert'
                   if (!(condition))                                       \
                   ^
   mm/memory.c:4943:2: note: Loop condition is false.  Exiting loop
           __set_current_state(TASK_RUNNING);

vim +3876 mm/memory.c

^1da177e4c3f41 Linus Torvalds          2005-04-16  3808  
^1da177e4c3f41 Linus Torvalds          2005-04-16  3809  /*
c1e8d7c6a7a682 Michel Lespinasse       2020-06-08  3810   * We enter with non-exclusive mmap_lock (to exclude vma changes,
8f4e2101fd7df9 Hugh Dickins            2005-10-29  3811   * but allow concurrent faults), and pte mapped but not yet locked.
c1e8d7c6a7a682 Michel Lespinasse       2020-06-08  3812   * We return with mmap_lock still held, but pte unmapped and unlocked.
^1da177e4c3f41 Linus Torvalds          2005-04-16  3813   */
2b7403035459c7 Souptick Joarder        2018-08-23  3814  static vm_fault_t do_anonymous_page(struct vm_fault *vmf)
^1da177e4c3f41 Linus Torvalds          2005-04-16  3815  {
82b0f8c39a3869 Jan Kara                2016-12-14  3816  	struct vm_area_struct *vma = vmf->vma;
e2bf0c1a3180a3 Michel Lespinasse       2022-01-28  3817  	struct page *page = NULL;
2b7403035459c7 Souptick Joarder        2018-08-23  3818  	vm_fault_t ret = 0;
^1da177e4c3f41 Linus Torvalds          2005-04-16  3819  	pte_t entry;
^1da177e4c3f41 Linus Torvalds          2005-04-16  3820  
6b7339f4c31ad6 Kirill A. Shutemov      2015-07-06  3821  	/* File mapping without ->vm_ops ? */
6b7339f4c31ad6 Kirill A. Shutemov      2015-07-06  3822  	if (vma->vm_flags & VM_SHARED)
6b7339f4c31ad6 Kirill A. Shutemov      2015-07-06  3823  		return VM_FAULT_SIGBUS;
6b7339f4c31ad6 Kirill A. Shutemov      2015-07-06  3824  
7267ec008b5cd8 Kirill A. Shutemov      2016-07-26  3825  	/*
7267ec008b5cd8 Kirill A. Shutemov      2016-07-26  3826  	 * Use pte_alloc() instead of pte_alloc_map().  We can't run
7267ec008b5cd8 Kirill A. Shutemov      2016-07-26  3827  	 * pte_offset_map() on pmds where a huge pmd might be created
7267ec008b5cd8 Kirill A. Shutemov      2016-07-26  3828  	 * from a different thread.
7267ec008b5cd8 Kirill A. Shutemov      2016-07-26  3829  	 *
3e4e28c5a8f01e Michel Lespinasse       2020-06-08  3830  	 * pte_alloc_map() is safe to use under mmap_write_lock(mm) or when
7267ec008b5cd8 Kirill A. Shutemov      2016-07-26  3831  	 * parallel threads are excluded by other means.
7267ec008b5cd8 Kirill A. Shutemov      2016-07-26  3832  	 *
3e4e28c5a8f01e Michel Lespinasse       2020-06-08  3833  	 * Here we only have mmap_read_lock(mm).
7267ec008b5cd8 Kirill A. Shutemov      2016-07-26  3834  	 */
4cf58924951ef8 Joel Fernandes (Google  2019-01-03  3835) 	if (pte_alloc(vma->vm_mm, vmf->pmd))
7267ec008b5cd8 Kirill A. Shutemov      2016-07-26  3836  		return VM_FAULT_OOM;
7267ec008b5cd8 Kirill A. Shutemov      2016-07-26  3837  
2fce3f44868d85 Michel Lespinasse       2022-01-28  3838  	/* See comment in __handle_mm_fault() */
82b0f8c39a3869 Jan Kara                2016-12-14  3839  	if (unlikely(pmd_trans_unstable(vmf->pmd)))
7267ec008b5cd8 Kirill A. Shutemov      2016-07-26  3840  		return 0;
7267ec008b5cd8 Kirill A. Shutemov      2016-07-26  3841  
11ac552477e328 Linus Torvalds          2010-08-14  3842  	/* Use the zero-page for reads */
82b0f8c39a3869 Jan Kara                2016-12-14  3843  	if (!(vmf->flags & FAULT_FLAG_WRITE) &&
bae473a423f65e Kirill A. Shutemov      2016-07-26  3844  			!mm_forbids_zeropage(vma->vm_mm)) {
82b0f8c39a3869 Jan Kara                2016-12-14  3845  		entry = pte_mkspecial(pfn_pte(my_zero_pfn(vmf->address),
62eede62dafb4a Hugh Dickins            2009-09-21  3846  						vma->vm_page_prot));
e2bf0c1a3180a3 Michel Lespinasse       2022-01-28  3847  	} else {
^1da177e4c3f41 Linus Torvalds          2005-04-16  3848  		/* Allocate our own private page. */
fa5331bae2e49c Michel Lespinasse       2022-01-28  3849  		if (unlikely(!vma->anon_vma)) {
fa5331bae2e49c Michel Lespinasse       2022-01-28  3850  			if (vmf->flags & FAULT_FLAG_SPECULATIVE)
fa5331bae2e49c Michel Lespinasse       2022-01-28  3851  				return VM_FAULT_RETRY;
fa5331bae2e49c Michel Lespinasse       2022-01-28  3852  			if (__anon_vma_prepare(vma))
65500d234e74fc Hugh Dickins            2005-10-29  3853  				goto oom;
fa5331bae2e49c Michel Lespinasse       2022-01-28  3854  		}
82b0f8c39a3869 Jan Kara                2016-12-14  3855  		page = alloc_zeroed_user_highpage_movable(vma, vmf->address);
^1da177e4c3f41 Linus Torvalds          2005-04-16  3856  		if (!page)
65500d234e74fc Hugh Dickins            2005-10-29  3857  			goto oom;
eb3c24f305e56c Mel Gorman              2015-06-24  3858  
8f425e4ed0eb3e Matthew Wilcox (Oracle  2021-06-25  3859) 		if (mem_cgroup_charge(page_folio(page), vma->vm_mm, GFP_KERNEL))
eb3c24f305e56c Mel Gorman              2015-06-24  3860  			goto oom_free_page;
9d82c69438d0df Johannes Weiner         2020-06-03  3861  		cgroup_throttle_swaprate(page, GFP_KERNEL);
eb3c24f305e56c Mel Gorman              2015-06-24  3862  
52f37629fd3c7b Minchan Kim             2013-04-29  3863  		/*
52f37629fd3c7b Minchan Kim             2013-04-29  3864  		 * The memory barrier inside __SetPageUptodate makes sure that
f4f5329d453704 Wei Yang                2019-11-30  3865  		 * preceding stores to the page contents become visible before
52f37629fd3c7b Minchan Kim             2013-04-29  3866  		 * the set_pte_at() write.
52f37629fd3c7b Minchan Kim             2013-04-29  3867  		 */
0ed361dec36945 Nicholas Piggin         2008-02-04  3868  		__SetPageUptodate(page);
^1da177e4c3f41 Linus Torvalds          2005-04-16  3869  
65500d234e74fc Hugh Dickins            2005-10-29  3870  		entry = mk_pte(page, vma->vm_page_prot);
50c25ee97cf6ab Thomas Bogendoerfer     2021-06-04  3871  		entry = pte_sw_mkyoung(entry);
1ac0cb5d0e22d5 Hugh Dickins            2009-09-21  3872  		if (vma->vm_flags & VM_WRITE)
1ac0cb5d0e22d5 Hugh Dickins            2009-09-21  3873  			entry = pte_mkwrite(pte_mkdirty(entry));
e2bf0c1a3180a3 Michel Lespinasse       2022-01-28  3874  	}
8f4e2101fd7df9 Hugh Dickins            2005-10-29  3875  
fa5331bae2e49c Michel Lespinasse       2022-01-28 @3876  	if (!pte_map_lock(vmf)) {
fa5331bae2e49c Michel Lespinasse       2022-01-28  3877  		ret = VM_FAULT_RETRY;
fa5331bae2e49c Michel Lespinasse       2022-01-28  3878  		goto release;
fa5331bae2e49c Michel Lespinasse       2022-01-28  3879  	}
7df676974359f9 Bibo Mao                2020-05-27  3880  	if (!pte_none(*vmf->pte)) {
45ee1834760b3b Michel Lespinasse       2022-01-28  3881  		update_mmu_tlb(vma, vmf->address, vmf->pte);
e2bf0c1a3180a3 Michel Lespinasse       2022-01-28  3882  		goto unlock;
7df676974359f9 Bibo Mao                2020-05-27  3883  	}
9ba6929480088a Hugh Dickins            2009-09-21  3884  
6b31d5955cb29a Michal Hocko            2017-08-18  3885  	ret = check_stable_address_space(vma->vm_mm);
6b31d5955cb29a Michal Hocko            2017-08-18  3886  	if (ret)
e2bf0c1a3180a3 Michel Lespinasse       2022-01-28  3887  		goto unlock;
6b31d5955cb29a Michal Hocko            2017-08-18  3888  
6b251fc96cf2cd Andrea Arcangeli        2015-09-04  3889  	/* Deliver the page fault to userland, check inside PT lock */
6b251fc96cf2cd Andrea Arcangeli        2015-09-04  3890  	if (userfaultfd_missing(vma)) {
82b0f8c39a3869 Jan Kara                2016-12-14  3891  		pte_unmap_unlock(vmf->pte, vmf->ptl);
e2bf0c1a3180a3 Michel Lespinasse       2022-01-28  3892  		if (page)
09cbfeaf1a5a67 Kirill A. Shutemov      2016-04-01  3893  			put_page(page);
fa5331bae2e49c Michel Lespinasse       2022-01-28  3894  		if (vmf->flags & FAULT_FLAG_SPECULATIVE)
fa5331bae2e49c Michel Lespinasse       2022-01-28  3895  			return VM_FAULT_RETRY;
82b0f8c39a3869 Jan Kara                2016-12-14  3896  		return handle_userfault(vmf, VM_UFFD_MISSING);
6b251fc96cf2cd Andrea Arcangeli        2015-09-04  3897  	}
6b251fc96cf2cd Andrea Arcangeli        2015-09-04  3898  
e2bf0c1a3180a3 Michel Lespinasse       2022-01-28  3899  	if (page) {
bae473a423f65e Kirill A. Shutemov      2016-07-26  3900  		inc_mm_counter_fast(vma->vm_mm, MM_ANONPAGES);
82b0f8c39a3869 Jan Kara                2016-12-14  3901  		page_add_new_anon_rmap(page, vma, vmf->address, false);
b518154e59aab3 Joonsoo Kim             2020-08-11  3902  		lru_cache_add_inactive_or_unevictable(page, vma);
e2bf0c1a3180a3 Michel Lespinasse       2022-01-28  3903  	}
e2bf0c1a3180a3 Michel Lespinasse       2022-01-28  3904  
82b0f8c39a3869 Jan Kara                2016-12-14  3905  	set_pte_at(vma->vm_mm, vmf->address, vmf->pte, entry);
^1da177e4c3f41 Linus Torvalds          2005-04-16  3906  
^1da177e4c3f41 Linus Torvalds          2005-04-16  3907  	/* No need to invalidate - it was non-present before */
82b0f8c39a3869 Jan Kara                2016-12-14  3908  	update_mmu_cache(vma, vmf->address, vmf->pte);
e2bf0c1a3180a3 Michel Lespinasse       2022-01-28  3909  	pte_unmap_unlock(vmf->pte, vmf->ptl);
e2bf0c1a3180a3 Michel Lespinasse       2022-01-28  3910  	return 0;
65500d234e74fc Hugh Dickins            2005-10-29  3911  unlock:
82b0f8c39a3869 Jan Kara                2016-12-14  3912  	pte_unmap_unlock(vmf->pte, vmf->ptl);
fa5331bae2e49c Michel Lespinasse       2022-01-28  3913  release:
e2bf0c1a3180a3 Michel Lespinasse       2022-01-28  3914  	if (page)
09cbfeaf1a5a67 Kirill A. Shutemov      2016-04-01  3915  		put_page(page);
e2bf0c1a3180a3 Michel Lespinasse       2022-01-28  3916  	return ret;
8a9f3ccd24741b Balbir Singh            2008-02-07  3917  oom_free_page:
09cbfeaf1a5a67 Kirill A. Shutemov      2016-04-01  3918  	put_page(page);
65500d234e74fc Hugh Dickins            2005-10-29  3919  oom:
^1da177e4c3f41 Linus Torvalds          2005-04-16  3920  	return VM_FAULT_OOM;
^1da177e4c3f41 Linus Torvalds          2005-04-16  3921  }
^1da177e4c3f41 Linus Torvalds          2005-04-16  3922  

---
0-DAY CI Kernel Test Service, Intel Corporation
https://lists.01.org/hyperkitty/list/kbuild-all(a)lists.01.org

^ permalink raw reply	[flat|nested] 77+ messages in thread

end of thread, other threads:[~2022-08-25 11:24 UTC | newest]

Thread overview: 77+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2022-01-28 13:09 [PATCH v2 00/35] Speculative page faults Michel Lespinasse
2022-01-28 13:09 ` [PATCH v2 01/35] mm: export dump_mm Michel Lespinasse
2022-01-28 13:09 ` [PATCH v2 02/35] mmap locking API: mmap_lock_is_contended returns a bool Michel Lespinasse
2022-01-28 13:09 ` [PATCH v2 03/35] mmap locking API: name the return values Michel Lespinasse
2022-01-31 16:17   ` Liam Howlett
2022-02-07 17:39     ` Michel Lespinasse
2022-01-28 13:09 ` [PATCH v2 04/35] do_anonymous_page: use update_mmu_tlb() Michel Lespinasse
2022-01-28 13:09 ` [PATCH v2 05/35] do_anonymous_page: reduce code duplication Michel Lespinasse
2022-01-28 13:09 ` [PATCH v2 06/35] mm: introduce CONFIG_SPECULATIVE_PAGE_FAULT Michel Lespinasse
2022-01-28 13:09 ` [PATCH v2 07/35] x86/mm: define ARCH_SUPPORTS_SPECULATIVE_PAGE_FAULT Michel Lespinasse
2022-01-28 13:09 ` [PATCH v2 08/35] mm: add FAULT_FLAG_SPECULATIVE flag Michel Lespinasse
2022-01-28 13:09 ` [PATCH v2 09/35] mm: add do_handle_mm_fault() Michel Lespinasse
2022-01-28 13:09 ` [PATCH v2 10/35] mm: add per-mm mmap sequence counter for speculative page fault handling Michel Lespinasse
2022-08-25 11:23   ` Pavan Kondeti
2022-01-28 13:09 ` [PATCH v2 11/35] mm: rcu safe vma freeing Michel Lespinasse
2022-01-28 13:09 ` [PATCH v2 12/35] mm: separate mmap locked assertion from find_vma Michel Lespinasse
2022-01-29  0:08   ` kernel test robot
2022-01-29  0:08     ` kernel test robot
2022-01-29  0:33     ` Michel Lespinasse
2022-01-29  0:33       ` Michel Lespinasse
2022-01-31 14:44   ` Matthew Wilcox
2022-02-04 22:41     ` Michel Lespinasse
2022-01-28 13:09 ` [PATCH v2 13/35] x86/mm: attempt speculative mm faults first Michel Lespinasse
2022-02-01 17:16   ` Liam Howlett
2022-02-07 17:39     ` Michel Lespinasse
2022-01-28 13:09 ` [PATCH v2 14/35] mm: add speculative_page_walk_begin() and speculative_page_walk_end() Michel Lespinasse
2022-01-28 13:09 ` [PATCH v2 15/35] mm: refactor __handle_mm_fault() / handle_pte_fault() Michel Lespinasse
2022-01-28 13:09 ` [PATCH v2 16/35] mm: implement speculative handling in __handle_mm_fault() Michel Lespinasse
2022-01-28 13:09 ` [PATCH v2 17/35] mm: add pte_map_lock() and pte_spinlock() Michel Lespinasse
2022-01-28 13:09 ` [PATCH v2 18/35] mm: implement speculative handling in do_anonymous_page() Michel Lespinasse
2022-01-28 21:03   ` kernel test robot
2022-01-28 21:03     ` kernel test robot
2022-01-28 22:08     ` Michel Lespinasse
2022-01-28 22:08       ` Michel Lespinasse
2022-01-30  2:54   ` [mm] fa5331bae2: canonical_address#:#[##] kernel test robot
2022-01-30  2:54     ` kernel test robot
2022-01-30  5:08     ` Michel Lespinasse
2022-01-30  5:08       ` Michel Lespinasse
2022-01-28 13:09 ` [PATCH v2 19/35] mm: enable speculative fault handling through do_anonymous_page() Michel Lespinasse
2022-01-28 13:09 ` [PATCH v2 20/35] mm: implement speculative handling in do_numa_page() Michel Lespinasse
2022-01-28 13:09 ` [PATCH v2 21/35] mm: enable speculative fault " Michel Lespinasse
2022-01-28 13:09 ` [PATCH v2 22/35] percpu-rwsem: enable percpu_sem destruction in atomic context Michel Lespinasse
2022-01-29 12:13   ` Hillf Danton
2022-01-31 18:04     ` Suren Baghdasaryan
2022-02-01  2:09       ` Hillf Danton
2022-02-07 19:31         ` Suren Baghdasaryan
2022-02-08  0:20           ` Hillf Danton
2022-02-08  1:31             ` Suren Baghdasaryan
2022-01-28 13:09 ` [PATCH v2 23/35] mm: add mmu_notifier_lock Michel Lespinasse
2022-07-27  7:34   ` Pavan Kondeti
2022-07-27 20:30     ` Suren Baghdasaryan
2022-01-28 13:09 ` [PATCH v2 24/35] mm: write lock mmu_notifier_lock when registering mmu notifiers Michel Lespinasse
2022-01-28 13:09 ` [PATCH v2 25/35] mm: add mmu_notifier_trylock() and mmu_notifier_unlock() Michel Lespinasse
2022-01-28 13:09 ` [PATCH v2 26/35] mm: implement speculative handling in wp_page_copy() Michel Lespinasse
2022-01-28 13:09 ` [PATCH v2 27/35] mm: implement and enable speculative fault handling in handle_pte_fault() Michel Lespinasse
2022-01-28 13:09 ` [PATCH v2 28/35] mm: disable speculative faults for single threaded user space Michel Lespinasse
2022-01-28 13:10 ` [PATCH v2 29/35] mm: disable rcu safe vma freeing " Michel Lespinasse
2022-01-28 13:10 ` [PATCH v2 30/35] mm: create new include/linux/vm_event.h header file Michel Lespinasse
2022-01-28 13:10 ` [PATCH v2 31/35] mm: anon spf statistics Michel Lespinasse
2022-01-28 13:10 ` [PATCH v2 32/35] arm64/mm: define ARCH_SUPPORTS_SPECULATIVE_PAGE_FAULT Michel Lespinasse
2022-01-28 13:10 ` [PATCH v2 33/35] arm64/mm: attempt speculative mm faults first Michel Lespinasse
2022-01-30  9:13   ` Mike Rapoport
2022-01-31  8:07     ` Michel Lespinasse
2022-02-01  8:58       ` Mike Rapoport
2022-02-07 17:39         ` Michel Lespinasse
2022-02-08  9:07           ` Mike Rapoport
2022-01-28 13:10 ` [PATCH v2 34/35] powerpc/mm: define ARCH_SUPPORTS_SPECULATIVE_PAGE_FAULT Michel Lespinasse
2022-01-28 13:10 ` [PATCH v2 35/35] powerpc/mm: attempt speculative mm faults first Michel Lespinasse
2022-01-31  9:56 ` [PATCH v2 00/35] Speculative page faults David Hildenbrand
2022-01-31 17:00   ` Suren Baghdasaryan
2022-02-01  1:14 ` Andrew Morton
2022-02-01  2:20   ` Matthew Wilcox
2022-02-07 17:39     ` Michel Lespinasse
2022-02-01 17:17   ` Sebastian Andrzej Siewior
2022-02-23 16:11 ` Mel Gorman
2022-03-08  5:37   ` Suren Baghdasaryan
  -- strict thread matches above, loose matches on Subject: below --
2022-01-30 18:03 [PATCH v2 18/35] mm: implement speculative handling in do_anonymous_page() kernel test robot

This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.