[RFC PATCH 00/16] membarrier: fixes, selftests and documentation

[RFC PATCH 00/16] membarrier: fixes, selftests and documentation

[Mathieu Desnoyers]

This series comes from a review of the membarrier system call. The
review covered the code, its selftests and the fixes applied over
time. It also covered the barrier and core serialization requirements
of each architecture.

The series fixes four kernel issues:

 - Patch 1: update the runqueue membarrier state when the scheduler
   switches from a kernel thread or an exited task which borrowed the
   mm. MEMBARRIER_CMD_GLOBAL_EXPEDITED no longer skips a registered
   process.

 - Patch 2: report a registration in MEMBARRIER_CMD_GET_REGISTRATIONS
   only when it completed. An rseq or sync core registration is no
   longer reported as a private expedited registration as well.

 - Patch 8 and patch 9: add a full memory barrier in switch_mm() for
   registered processes, on LoongArch and on parisc. The expedited
   commands need it after the store to rq->curr.

Patch 1 and patch 2 were reproduced from user-space on v7.4-rc2.

Patch 8 and patch 9 were found by reading the code and the
architecture manuals. Feedback from the LoongArch and parisc
maintainers is welcome.

The series fixes and extends the selftests:

 - Patch 3: rebuild the tests when membarrier_test_impl.h changes.

 - Patch 4: issue the sync core command in the test which is named
   after it.

 - Patch 5: run the tests on nohz_full kernels, which only lack
   MEMBARRIER_CMD_GLOBAL.

 - Patch 6: add membarrier_test_ipi. It checks which CPUs each
   expedited command interrupts. Two of its test cases fail without
   patch 1.

 - Patch 7: add membarrier_test_cmd. It checks the command arguments
   and the registrations across fork, exec and CLONE_VM. Two of its
   tests fail without patch 2.

 - Patch 13: add litmus tests of the memory ordering scenarios, for
   the Linux kernel memory model. One variant shows the LoongArch and
   parisc issue.

 - Patch 14: add the selftests to the MEMBARRIER SUPPORT entry of
   MAINTAINERS.

 - Patch 15: add membarrier_test_barrier, a stress test for x86-64.
   BPF programs on kprobes count, for each thread, the memory
   barriers which membarrier relies on. The test also checks the
   barriers of the scheduler. Two of its test cases fail without
   patch 1. It adds no kernel code.

 - Patch 16: extend membarrier_test_barrier to the sync core
   command. It counts the core serializing instructions in the same
   way.

The series updates the documentation:

 - Patch 10: add the missing architectures to the list of barriers
   after the store to rq->curr.

 - Patch 11: describe what the sync core command relies on for x86
   with FRED and for arm64 with FEAT_ExS.

 - Patch 12: remove two stale x86 entry comments.

The sync core requirements were checked against the architecture
manuals of x86 (Intel SDM and AMD APM), arm, arm64, powerpc, s390 and
riscv. These are the architectures which support the sync core
command. No return path to user-space was found to miss its documented
serialization.

Testing:

 - The six membarrier test programs pass in an x86-64 KVM guest
   with 8 CPUs and lockdep.

 - With nohz_full=2-7, the six programs pass. Only the
   MEMBARRIER_CMD_GLOBAL test is skipped.

 - With nosmp and with maxcpus=2, membarrier_test_ipi is skipped
   because it needs 3 CPUs. membarrier_test_barrier skips the two
   test cases which need 3 CPUs. The four other programs pass.

 - With pti=on, membarrier_test_barrier passes and does not check
   the core serialization.

 - Patch 8 and patch 9 are compile-tested only: LoongArch and parisc
   (32-bit, SMP), default configuration.

 - herd7 reports the expected result for each of the 13 litmus tests.

The series is based on v7.4-rc2.

An LLM coding assistant did this work at my request: the review, the
reproducers, the fixes, the tests and the changelogs. I actively
steered it and reviewed the result.

Mathieu Desnoyers (16):
  sched/membarrier: Fix stale runqueue state after a lazy mm switch
  sched/membarrier: Report only completed registrations
  selftests/membarrier: Rebuild the tests when their header changes
  selftests/membarrier: Issue the sync core command in its test
  selftests/membarrier: Run the tests on nohz_full kernels
  selftests/membarrier: Test which CPUs the expedited commands interrupt
  selftests/membarrier: Test command arguments and registration lifetime
  LoongArch: membarrier: Add full memory barrier in switch_mm()
  parisc: membarrier: Add full memory barrier in switch_mm()
  Documentation/membarrier: Update the per-architecture barrier list
  Documentation/features: Update the membarrier sync core requirements
  x86/entry: Remove stale membarrier sync core comments
  selftests/membarrier: Add litmus tests of the memory ordering
  MAINTAINERS: Add the membarrier selftests to MEMBARRIER SUPPORT
  selftests/membarrier: Check the memory barriers of each thread
  selftests/membarrier: Check the core serialization of the sync core
    command

 .../membarrier-sync-core/arch-support.txt     |   35 +-
 Documentation/scheduler/membarrier.rst        |   14 +-
 MAINTAINERS                                   |    1 +
 arch/loongarch/Kconfig                        |    1 +
 arch/loongarch/include/asm/membarrier.h       |   33 +
 arch/loongarch/include/asm/mmu_context.h      |    4 +
 arch/parisc/Kconfig                           |    1 +
 arch/parisc/include/asm/membarrier.h          |   33 +
 arch/parisc/include/asm/mmu_context.h         |    2 +
 arch/x86/entry/entry_32.S                     |    5 -
 arch/x86/entry/entry_64.S                     |    4 -
 arch/x86/include/asm/sync_core.h              |    2 +-
 kernel/sched/core.c                           |   16 +-
 kernel/sched/membarrier.c                     |   32 +-
 kernel/sched/sched.h                          |   15 +
 tools/testing/selftests/membarrier/.gitignore |    6 +
 tools/testing/selftests/membarrier/Makefile   |   20 +
 tools/testing/selftests/membarrier/config     |    9 +
 .../selftests/membarrier/litmus/README        |   52 +
 .../litmus/ipi-after-thread-no-exit-mb.litmus |   36 +
 .../ipi-after-thread-no-handler-mb.litmus     |   36 +
 .../membarrier/litmus/ipi-after-thread.litmus |   47 +
 .../ipi-before-thread-no-entry-mb.litmus      |   37 +
 .../ipi-before-thread-no-handler-mb.litmus    |   37 +
 .../litmus/ipi-before-thread.litmus           |   46 +
 .../litmus/sched-in-no-entry-mb.litmus        |   34 +
 .../membarrier/litmus/sched-in-no-mb.litmus   |   33 +
 .../membarrier/litmus/sched-in-release.litmus |   38 +
 .../membarrier/litmus/sched-in.litmus         |   51 +
 .../litmus/sched-out-no-exit-mb.litmus        |   34 +
 .../litmus/sched-out-no-sched-mb.litmus       |   34 +
 .../membarrier/litmus/sched-out.litmus        |   49 +
 .../membarrier/membarrier_test_barrier.bpf.c  |  173 +++
 .../membarrier/membarrier_test_barrier.c      | 1347 +++++++++++++++++
 .../membarrier/membarrier_test_barrier.h      |   54 +
 .../membarrier/membarrier_test_cmd.c          |  352 +++++
 .../membarrier/membarrier_test_impl.h         |   14 +-
 .../membarrier/membarrier_test_ipi.c          |  739 +++++++++
 38 files changed, 3426 insertions(+), 50 deletions(-)
 create mode 100644 arch/loongarch/include/asm/membarrier.h
 create mode 100644 arch/parisc/include/asm/membarrier.h
 create mode 100644 tools/testing/selftests/membarrier/config
 create mode 100644 tools/testing/selftests/membarrier/litmus/README
 create mode 100644 tools/testing/selftests/membarrier/litmus/ipi-after-thread-no-exit-mb.litmus
 create mode 100644 tools/testing/selftests/membarrier/litmus/ipi-after-thread-no-handler-mb.litmus
 create mode 100644 tools/testing/selftests/membarrier/litmus/ipi-after-thread.litmus
 create mode 100644 tools/testing/selftests/membarrier/litmus/ipi-before-thread-no-entry-mb.litmus
 create mode 100644 tools/testing/selftests/membarrier/litmus/ipi-before-thread-no-handler-mb.litmus
 create mode 100644 tools/testing/selftests/membarrier/litmus/ipi-before-thread.litmus
 create mode 100644 tools/testing/selftests/membarrier/litmus/sched-in-no-entry-mb.litmus
 create mode 100644 tools/testing/selftests/membarrier/litmus/sched-in-no-mb.litmus
 create mode 100644 tools/testing/selftests/membarrier/litmus/sched-in-release.litmus
 create mode 100644 tools/testing/selftests/membarrier/litmus/sched-in.litmus
 create mode 100644 tools/testing/selftests/membarrier/litmus/sched-out-no-exit-mb.litmus
 create mode 100644 tools/testing/selftests/membarrier/litmus/sched-out-no-sched-mb.litmus
 create mode 100644 tools/testing/selftests/membarrier/litmus/sched-out.litmus
 create mode 100644 tools/testing/selftests/membarrier/membarrier_test_barrier.bpf.c
 create mode 100644 tools/testing/selftests/membarrier/membarrier_test_barrier.c
 create mode 100644 tools/testing/selftests/membarrier/membarrier_test_barrier.h
 create mode 100644 tools/testing/selftests/membarrier/membarrier_test_cmd.c
 create mode 100644 tools/testing/selftests/membarrier/membarrier_test_ipi.c


base-commit: 7475ea8b58616b33c04afb976cf0fbaad9dd63fb
-- 
2.47.3

[RFC PATCH 00/16] membarrier: fixes, selftests and documentation

[Mathieu Desnoyers]

This series comes from a review of the membarrier system call. The
review covered the code, its selftests and the fixes applied over
time. It also covered the barrier and core serialization requirements
of each architecture.

The series fixes four kernel issues:

 - Patch 1: update the runqueue membarrier state when the scheduler
   switches from a kernel thread or an exited task which borrowed the
   mm. MEMBARRIER_CMD_GLOBAL_EXPEDITED no longer skips a registered
   process.

 - Patch 2: report a registration in MEMBARRIER_CMD_GET_REGISTRATIONS
   only when it completed. An rseq or sync core registration is no
   longer reported as a private expedited registration as well.

 - Patch 8 and patch 9: add a full memory barrier in switch_mm() for
   registered processes, on LoongArch and on parisc. The expedited
   commands need it after the store to rq->curr.

Patch 1 and patch 2 were reproduced from user-space on v7.4-rc2.

Patch 8 and patch 9 were found by reading the code and the
architecture manuals. Feedback from the LoongArch and parisc
maintainers is welcome.

The series fixes and extends the selftests:

 - Patch 3: rebuild the tests when membarrier_test_impl.h changes.

 - Patch 4: issue the sync core command in the test which is named
   after it.

 - Patch 5: run the tests on nohz_full kernels, which only lack
   MEMBARRIER_CMD_GLOBAL.

 - Patch 6: add membarrier_test_ipi. It checks which CPUs each
   expedited command interrupts. Two of its test cases fail without
   patch 1.

 - Patch 7: add membarrier_test_cmd. It checks the command arguments
   and the registrations across fork, exec and CLONE_VM. Two of its
   tests fail without patch 2.

 - Patch 13: add litmus tests of the memory ordering scenarios, for
   the Linux kernel memory model. One variant shows the LoongArch and
   parisc issue.

 - Patch 14: add the selftests to the MEMBARRIER SUPPORT entry of
   MAINTAINERS.

 - Patch 15: add membarrier_test_barrier, a stress test for x86-64.
   BPF programs on kprobes count, for each thread, the memory
   barriers which membarrier relies on. The test also checks the
   barriers of the scheduler. Two of its test cases fail without
   patch 1. It adds no kernel code.

 - Patch 16: extend membarrier_test_barrier to the sync core
   command. It counts the core serializing instructions in the same
   way.

The series updates the documentation:

 - Patch 10: add the missing architectures to the list of barriers
   after the store to rq->curr.

 - Patch 11: describe what the sync core command relies on for x86
   with FRED and for arm64 with FEAT_ExS.

 - Patch 12: remove two stale x86 entry comments.

The sync core requirements were checked against the architecture
manuals of x86 (Intel SDM and AMD APM), arm, arm64, powerpc, s390 and
riscv. These are the architectures which support the sync core
command. No return path to user-space was found to miss its documented
serialization.

Testing:

 - The six membarrier test programs pass in an x86-64 KVM guest
   with 8 CPUs and lockdep.

 - With nohz_full=2-7, the six programs pass. Only the
   MEMBARRIER_CMD_GLOBAL test is skipped.

 - With nosmp and with maxcpus=2, membarrier_test_ipi is skipped
   because it needs 3 CPUs. membarrier_test_barrier skips the two
   test cases which need 3 CPUs. The four other programs pass.

 - With pti=on, membarrier_test_barrier passes and does not check
   the core serialization.

 - Patch 8 and patch 9 are compile-tested only: LoongArch and parisc
   (32-bit, SMP), default configuration.

 - herd7 reports the expected result for each of the 13 litmus tests.

The series is based on v7.4-rc2.

An LLM coding assistant did this work at my request: the review, the
reproducers, the fixes, the tests and the changelogs. I actively
steered it and reviewed the result.

Mathieu Desnoyers (16):
  sched/membarrier: Fix stale runqueue state after a lazy mm switch
  sched/membarrier: Report only completed registrations
  selftests/membarrier: Rebuild the tests when their header changes
  selftests/membarrier: Issue the sync core command in its test
  selftests/membarrier: Run the tests on nohz_full kernels
  selftests/membarrier: Test which CPUs the expedited commands interrupt
  selftests/membarrier: Test command arguments and registration lifetime
  LoongArch: membarrier: Add full memory barrier in switch_mm()
  parisc: membarrier: Add full memory barrier in switch_mm()
  Documentation/membarrier: Update the per-architecture barrier list
  Documentation/features: Update the membarrier sync core requirements
  x86/entry: Remove stale membarrier sync core comments
  selftests/membarrier: Add litmus tests of the memory ordering
  MAINTAINERS: Add the membarrier selftests to MEMBARRIER SUPPORT
  selftests/membarrier: Check the memory barriers of each thread
  selftests/membarrier: Check the core serialization of the sync core
    command

 .../membarrier-sync-core/arch-support.txt     |   35 +-
 Documentation/scheduler/membarrier.rst        |   14 +-
 MAINTAINERS                                   |    1 +
 arch/loongarch/Kconfig                        |    1 +
 arch/loongarch/include/asm/membarrier.h       |   33 +
 arch/loongarch/include/asm/mmu_context.h      |    4 +
 arch/parisc/Kconfig                           |    1 +
 arch/parisc/include/asm/membarrier.h          |   33 +
 arch/parisc/include/asm/mmu_context.h         |    2 +
 arch/x86/entry/entry_32.S                     |    5 -
 arch/x86/entry/entry_64.S                     |    4 -
 arch/x86/include/asm/sync_core.h              |    2 +-
 kernel/sched/core.c                           |   16 +-
 kernel/sched/membarrier.c                     |   32 +-
 kernel/sched/sched.h                          |   15 +
 tools/testing/selftests/membarrier/.gitignore |    6 +
 tools/testing/selftests/membarrier/Makefile   |   20 +
 tools/testing/selftests/membarrier/config     |    9 +
 .../selftests/membarrier/litmus/README        |   52 +
 .../litmus/ipi-after-thread-no-exit-mb.litmus |   36 +
 .../ipi-after-thread-no-handler-mb.litmus     |   36 +
 .../membarrier/litmus/ipi-after-thread.litmus |   47 +
 .../ipi-before-thread-no-entry-mb.litmus      |   37 +
 .../ipi-before-thread-no-handler-mb.litmus    |   37 +
 .../litmus/ipi-before-thread.litmus           |   46 +
 .../litmus/sched-in-no-entry-mb.litmus        |   34 +
 .../membarrier/litmus/sched-in-no-mb.litmus   |   33 +
 .../membarrier/litmus/sched-in-release.litmus |   38 +
 .../membarrier/litmus/sched-in.litmus         |   51 +
 .../litmus/sched-out-no-exit-mb.litmus        |   34 +
 .../litmus/sched-out-no-sched-mb.litmus       |   34 +
 .../membarrier/litmus/sched-out.litmus        |   49 +
 .../membarrier/membarrier_test_barrier.bpf.c  |  173 +++
 .../membarrier/membarrier_test_barrier.c      | 1347 +++++++++++++++++
 .../membarrier/membarrier_test_barrier.h      |   54 +
 .../membarrier/membarrier_test_cmd.c          |  352 +++++
 .../membarrier/membarrier_test_impl.h         |   14 +-
 .../membarrier/membarrier_test_ipi.c          |  739 +++++++++
 38 files changed, 3426 insertions(+), 50 deletions(-)
 create mode 100644 arch/loongarch/include/asm/membarrier.h
 create mode 100644 arch/parisc/include/asm/membarrier.h
 create mode 100644 tools/testing/selftests/membarrier/config
 create mode 100644 tools/testing/selftests/membarrier/litmus/README
 create mode 100644 tools/testing/selftests/membarrier/litmus/ipi-after-thread-no-exit-mb.litmus
 create mode 100644 tools/testing/selftests/membarrier/litmus/ipi-after-thread-no-handler-mb.litmus
 create mode 100644 tools/testing/selftests/membarrier/litmus/ipi-after-thread.litmus
 create mode 100644 tools/testing/selftests/membarrier/litmus/ipi-before-thread-no-entry-mb.litmus
 create mode 100644 tools/testing/selftests/membarrier/litmus/ipi-before-thread-no-handler-mb.litmus
 create mode 100644 tools/testing/selftests/membarrier/litmus/ipi-before-thread.litmus
 create mode 100644 tools/testing/selftests/membarrier/litmus/sched-in-no-entry-mb.litmus
 create mode 100644 tools/testing/selftests/membarrier/litmus/sched-in-no-mb.litmus
 create mode 100644 tools/testing/selftests/membarrier/litmus/sched-in-release.litmus
 create mode 100644 tools/testing/selftests/membarrier/litmus/sched-in.litmus
 create mode 100644 tools/testing/selftests/membarrier/litmus/sched-out-no-exit-mb.litmus
 create mode 100644 tools/testing/selftests/membarrier/litmus/sched-out-no-sched-mb.litmus
 create mode 100644 tools/testing/selftests/membarrier/litmus/sched-out.litmus
 create mode 100644 tools/testing/selftests/membarrier/membarrier_test_barrier.bpf.c
 create mode 100644 tools/testing/selftests/membarrier/membarrier_test_barrier.c
 create mode 100644 tools/testing/selftests/membarrier/membarrier_test_barrier.h
 create mode 100644 tools/testing/selftests/membarrier/membarrier_test_cmd.c
 create mode 100644 tools/testing/selftests/membarrier/membarrier_test_ipi.c


base-commit: 7475ea8b58616b33c04afb976cf0fbaad9dd63fb
-- 
2.47.3


_______________________________________________
linux-riscv mailing list
linux-riscv@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-riscv

[RFC PATCH 01/16] sched/membarrier: Fix stale runqueue state after a lazy mm switch

[Mathieu Desnoyers]

MEMBARRIER_CMD_GLOBAL_EXPEDITED sends an IPI to each CPU which runs
a registered process. It finds those CPUs with rq->membarrier_state,
a per-runqueue copy of the membarrier state of the mm.

Commit 227a4aadc75b ("sched/membarrier: Fix p->mm->membarrier_state
racy load") added the copy. The scheduler updates it when it switches
between two mm. A registration sends an IPI to update the runqueues
which keep running the mm.

membarrier_switch_mm() compares the previous active_mm with the next
mm. So it also skips the update when the previous task only borrowed
the mm. This is a kernel thread in lazy TLB mode, or a task past
exit_mm(). The copy can be stale in two such cases.

Case 1: a process registers while a lazy TLB kernel thread borrows
its mm on another CPU. T is a thread of a process with mm M, which is
not registered yet. S is a thread of another process.

  CPU 0                                        CPU 1

  T runs in user-space
    rq->membarrier_state == 0
  T migrates to CPU 1
  context_switch(T -> idle)
    idle borrows M as its active_mm
                                               context_switch(-> T)
                                               T: MEMBARRIER_CMD_REGISTER_GLOBAL_EXPEDITED
                                                 sets the state of M
                                                 sets rq->membarrier_state of CPU 1
                                                 CPU 0 runs idle: no IPI
                                               T migrates to CPU 0
  context_switch(idle -> T)
    membarrier_switch_mm(rq, idle->active_mm, T->mm)
      prev_mm == next_mm: no update
  T runs in user-space
    rq->membarrier_state == 0 (stale)
                                               S: MEMBARRIER_CMD_GLOBAL_EXPEDITED
                                                 CPU 0: state not set, no IPI

Case 2: a thread exits and the CPU then runs another thread of the
same process. A and B are threads of a registered process with mm M.
S is a thread of another process.

  CPU 0                                        CPU 1

  B exits
  exit_mm()
    B->mm = NULL
    membarrier_update_current_mm(NULL)
      rq->membarrier_state = 0
  context_switch(B -> A)
    membarrier_switch_mm(rq, B->active_mm, A->mm)
      prev_mm == next_mm: no update
  A runs in user-space
    rq->membarrier_state == 0 (stale)
                                               S: MEMBARRIER_CMD_GLOBAL_EXPEDITED
                                                 CPU 0: state not set, no IPI

kthread_unuse_mm() clears the copy in the same way as exit_mm().

In both cases, the registered process runs without the promised
memory barrier. This lasts until the CPU switches to the process from
another mm.

User-space sees this as a memory ordering failure. The registered
processes omit memory barriers from their fast paths. The caller
issues MEMBARRIER_CMD_GLOBAL_EXPEDITED where it needs those barriers.
An example is RCU-like synchronization over shared memory. Without the
IPI, the caller would miss a recent store of a registered process. It
would then, for example, free memory which that process still uses.
A thread exit in a registered multi-threaded process is enough to
trigger case 2.

Both cases were reproduced on v7.4-rc2: 1000 commands send no IPI to
the registered process.

Fix this by passing prev->mm rather than prev->active_mm. It is NULL
for a kernel thread and for an exited task, which forces the update.
The update is still skipped between two user tasks of the same mm.

Fixes: 227a4aadc75b ("sched/membarrier: Fix p->mm->membarrier_state racy load")
Cc: stable@vger.kernel.org
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
---
 kernel/sched/core.c  |  6 +++++-
 kernel/sched/sched.h | 15 +++++++++++++++
 2 files changed, 20 insertions(+), 1 deletion(-)

diff --git a/kernel/sched/core.c b/kernel/sched/core.c
index 1a088d4576f5..df6d15cf7109 100644
--- a/kernel/sched/core.c
+++ b/kernel/sched/core.c
@@ -5551,7 +5551,11 @@ context_switch(struct rq *rq, struct task_struct *prev,
 		else
 			prev->active_mm = NULL;
 	} else {					// to user
-		membarrier_switch_mm(rq, prev->active_mm, next->mm);
+		/*
+		 * Pass prev->mm, not prev->active_mm: see
+		 * membarrier_switch_mm().
+		 */
+		membarrier_switch_mm(rq, prev->mm, next->mm);
 		/*
 		 * sys_membarrier() requires an smp_mb() between setting
 		 * rq->curr / membarrier_switch_mm() and returning to userspace.
diff --git a/kernel/sched/sched.h b/kernel/sched/sched.h
index 1d76950dbb0a..68353796a7f9 100644
--- a/kernel/sched/sched.h
+++ b/kernel/sched/sched.h
@@ -3907,6 +3907,21 @@ static inline bool sched_energy_enabled(void) { return false; }
  * - prior user-space memory accesses and store to rq->membarrier_state,
  * - store to rq->membarrier_state and following user-space memory accesses.
  * In the same way it provides those guarantees around store to rq->curr.
+ *
+ * @prev_mm is the mm of the previous task. It is NULL for a kernel
+ * thread and for a task past exit_mm().
+ *
+ * The update is skipped only between two user tasks of the same mm.
+ * rq->membarrier_state is up to date in that case. It was set when
+ * the first of those tasks was scheduled in, or later by the
+ * registration IPI.
+ *
+ * It must not be skipped when the previous task only borrowed the mm
+ * as its active_mm. rq->membarrier_state can be stale then:
+ *
+ * - exit_mm() and kthread_unuse_mm() clear it,
+ * - a registration does not send an IPI to a CPU which runs a lazy
+ *   TLB kernel thread.
  */
 static inline void membarrier_switch_mm(struct rq *rq,
 					struct mm_struct *prev_mm,
-- 
2.47.3

[RFC PATCH 01/16] sched/membarrier: Fix stale runqueue state after a lazy mm switch

[Mathieu Desnoyers]

MEMBARRIER_CMD_GLOBAL_EXPEDITED sends an IPI to each CPU which runs
a registered process. It finds those CPUs with rq->membarrier_state,
a per-runqueue copy of the membarrier state of the mm.

Commit 227a4aadc75b ("sched/membarrier: Fix p->mm->membarrier_state
racy load") added the copy. The scheduler updates it when it switches
between two mm. A registration sends an IPI to update the runqueues
which keep running the mm.

membarrier_switch_mm() compares the previous active_mm with the next
mm. So it also skips the update when the previous task only borrowed
the mm. This is a kernel thread in lazy TLB mode, or a task past
exit_mm(). The copy can be stale in two such cases.

Case 1: a process registers while a lazy TLB kernel thread borrows
its mm on another CPU. T is a thread of a process with mm M, which is
not registered yet. S is a thread of another process.

  CPU 0                                        CPU 1

  T runs in user-space
    rq->membarrier_state == 0
  T migrates to CPU 1
  context_switch(T -> idle)
    idle borrows M as its active_mm
                                               context_switch(-> T)
                                               T: MEMBARRIER_CMD_REGISTER_GLOBAL_EXPEDITED
                                                 sets the state of M
                                                 sets rq->membarrier_state of CPU 1
                                                 CPU 0 runs idle: no IPI
                                               T migrates to CPU 0
  context_switch(idle -> T)
    membarrier_switch_mm(rq, idle->active_mm, T->mm)
      prev_mm == next_mm: no update
  T runs in user-space
    rq->membarrier_state == 0 (stale)
                                               S: MEMBARRIER_CMD_GLOBAL_EXPEDITED
                                                 CPU 0: state not set, no IPI

Case 2: a thread exits and the CPU then runs another thread of the
same process. A and B are threads of a registered process with mm M.
S is a thread of another process.

  CPU 0                                        CPU 1

  B exits
  exit_mm()
    B->mm = NULL
    membarrier_update_current_mm(NULL)
      rq->membarrier_state = 0
  context_switch(B -> A)
    membarrier_switch_mm(rq, B->active_mm, A->mm)
      prev_mm == next_mm: no update
  A runs in user-space
    rq->membarrier_state == 0 (stale)
                                               S: MEMBARRIER_CMD_GLOBAL_EXPEDITED
                                                 CPU 0: state not set, no IPI

kthread_unuse_mm() clears the copy in the same way as exit_mm().

In both cases, the registered process runs without the promised
memory barrier. This lasts until the CPU switches to the process from
another mm.

User-space sees this as a memory ordering failure. The registered
processes omit memory barriers from their fast paths. The caller
issues MEMBARRIER_CMD_GLOBAL_EXPEDITED where it needs those barriers.
An example is RCU-like synchronization over shared memory. Without the
IPI, the caller would miss a recent store of a registered process. It
would then, for example, free memory which that process still uses.
A thread exit in a registered multi-threaded process is enough to
trigger case 2.

Both cases were reproduced on v7.4-rc2: 1000 commands send no IPI to
the registered process.

Fix this by passing prev->mm rather than prev->active_mm. It is NULL
for a kernel thread and for an exited task, which forces the update.
The update is still skipped between two user tasks of the same mm.

Fixes: 227a4aadc75b ("sched/membarrier: Fix p->mm->membarrier_state racy load")
Cc: stable@vger.kernel.org
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
---
 kernel/sched/core.c  |  6 +++++-
 kernel/sched/sched.h | 15 +++++++++++++++
 2 files changed, 20 insertions(+), 1 deletion(-)

diff --git a/kernel/sched/core.c b/kernel/sched/core.c
index 1a088d4576f5..df6d15cf7109 100644
--- a/kernel/sched/core.c
+++ b/kernel/sched/core.c
@@ -5551,7 +5551,11 @@ context_switch(struct rq *rq, struct task_struct *prev,
 		else
 			prev->active_mm = NULL;
 	} else {					// to user
-		membarrier_switch_mm(rq, prev->active_mm, next->mm);
+		/*
+		 * Pass prev->mm, not prev->active_mm: see
+		 * membarrier_switch_mm().
+		 */
+		membarrier_switch_mm(rq, prev->mm, next->mm);
 		/*
 		 * sys_membarrier() requires an smp_mb() between setting
 		 * rq->curr / membarrier_switch_mm() and returning to userspace.
diff --git a/kernel/sched/sched.h b/kernel/sched/sched.h
index 1d76950dbb0a..68353796a7f9 100644
--- a/kernel/sched/sched.h
+++ b/kernel/sched/sched.h
@@ -3907,6 +3907,21 @@ static inline bool sched_energy_enabled(void) { return false; }
  * - prior user-space memory accesses and store to rq->membarrier_state,
  * - store to rq->membarrier_state and following user-space memory accesses.
  * In the same way it provides those guarantees around store to rq->curr.
+ *
+ * @prev_mm is the mm of the previous task. It is NULL for a kernel
+ * thread and for a task past exit_mm().
+ *
+ * The update is skipped only between two user tasks of the same mm.
+ * rq->membarrier_state is up to date in that case. It was set when
+ * the first of those tasks was scheduled in, or later by the
+ * registration IPI.
+ *
+ * It must not be skipped when the previous task only borrowed the mm
+ * as its active_mm. rq->membarrier_state can be stale then:
+ *
+ * - exit_mm() and kthread_unuse_mm() clear it,
+ * - a registration does not send an IPI to a CPU which runs a lazy
+ *   TLB kernel thread.
  */
 static inline void membarrier_switch_mm(struct rq *rq,
 					struct mm_struct *prev_mm,
-- 
2.47.3


_______________________________________________
linux-riscv mailing list
linux-riscv@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-riscv

[RFC PATCH 02/16] sched/membarrier: Report only completed registrations

[Mathieu Desnoyers]

MEMBARRIER_CMD_GET_REGISTRATIONS returns the registration commands
which the process issued.

Each registration has two state bits: one is set when the registration
starts and the "ready" one when it completes. The command reports a
registration when either bit is set.

The sync core and rseq registrations also set
MEMBARRIER_STATE_PRIVATE_EXPEDITED, without its "ready" bit. They set
it because their commands need the same scheduler barriers as
MEMBARRIER_CMD_PRIVATE_EXPEDITED. So a process which only issued
MEMBARRIER_CMD_REGISTER_PRIVATE_EXPEDITED_RSEQ is reported as having
issued MEMBARRIER_CMD_REGISTER_PRIVATE_EXPEDITED as well. Yet
MEMBARRIER_CMD_PRIVATE_EXPEDITED returns -EPERM for that process.

A registration which failed with -ENOMEM would be reported the same
way.

User-space gets a wrong registration mask. The command was added for
CRIU, which saves the registrations of a process at checkpoint. A
restored process would carry a registration which the original
process did not issue. A program which trusts the mask would issue
MEMBARRIER_CMD_PRIVATE_EXPEDITED and get -EPERM.

Fix this by reporting a registration only when its "ready" bit is
set.

This changes the mask which user-space reads.
MEMBARRIER_CMD_GET_REGISTRATIONS no longer reports
MEMBARRIER_CMD_REGISTER_PRIVATE_EXPEDITED for a process which only
issued the sync core or the rseq registration. The result of every
other command is unchanged.

Fixes: 544a4f2ecd45 ("sched/membarrier: Introduce MEMBARRIER_CMD_GET_REGISTRATIONS")
Cc: stable@vger.kernel.org
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
---
 kernel/sched/membarrier.c | 29 ++++++++++++++---------------
 1 file changed, 14 insertions(+), 15 deletions(-)

diff --git a/kernel/sched/membarrier.c b/kernel/sched/membarrier.c
index cb957b8f1946..46e823a3d668 100644
--- a/kernel/sched/membarrier.c
+++ b/kernel/sched/membarrier.c
@@ -566,15 +566,17 @@ static int membarrier_get_registrations(void)
 	struct task_struct *p = current;
 	struct mm_struct *mm = p->mm;
 	int registrations_mask = 0, membarrier_state, i;
-	static const int states[] = {
-		MEMBARRIER_STATE_GLOBAL_EXPEDITED |
-			MEMBARRIER_STATE_GLOBAL_EXPEDITED_READY,
-		MEMBARRIER_STATE_PRIVATE_EXPEDITED |
-			MEMBARRIER_STATE_PRIVATE_EXPEDITED_READY,
-		MEMBARRIER_STATE_PRIVATE_EXPEDITED_SYNC_CORE |
-			MEMBARRIER_STATE_PRIVATE_EXPEDITED_SYNC_CORE_READY,
-		MEMBARRIER_STATE_PRIVATE_EXPEDITED_RSEQ |
-			MEMBARRIER_STATE_PRIVATE_EXPEDITED_RSEQ_READY
+	/*
+	 * Only the "ready" state tells that a registration command
+	 * completed. The other state of each pair is not specific to
+	 * one registration command: the sync core and rseq
+	 * registrations also set MEMBARRIER_STATE_PRIVATE_EXPEDITED.
+	 */
+	static const int ready_states[] = {
+		MEMBARRIER_STATE_GLOBAL_EXPEDITED_READY,
+		MEMBARRIER_STATE_PRIVATE_EXPEDITED_READY,
+		MEMBARRIER_STATE_PRIVATE_EXPEDITED_SYNC_CORE_READY,
+		MEMBARRIER_STATE_PRIVATE_EXPEDITED_RSEQ_READY
 	};
 	static const int registration_cmds[] = {
 		MEMBARRIER_CMD_REGISTER_GLOBAL_EXPEDITED,
@@ -582,16 +584,13 @@ static int membarrier_get_registrations(void)
 		MEMBARRIER_CMD_REGISTER_PRIVATE_EXPEDITED_SYNC_CORE,
 		MEMBARRIER_CMD_REGISTER_PRIVATE_EXPEDITED_RSEQ
 	};
-	BUILD_BUG_ON(ARRAY_SIZE(states) != ARRAY_SIZE(registration_cmds));
+	BUILD_BUG_ON(ARRAY_SIZE(ready_states) != ARRAY_SIZE(registration_cmds));
 
 	membarrier_state = atomic_read(&mm->membarrier_state);
-	for (i = 0; i < ARRAY_SIZE(states); ++i) {
-		if (membarrier_state & states[i]) {
+	for (i = 0; i < ARRAY_SIZE(ready_states); ++i) {
+		if (membarrier_state & ready_states[i])
 			registrations_mask |= registration_cmds[i];
-			membarrier_state &= ~states[i];
-		}
 	}
-	WARN_ON_ONCE(membarrier_state != 0);
 	return registrations_mask;
 }
 
-- 
2.47.3

[RFC PATCH 02/16] sched/membarrier: Report only completed registrations

[Mathieu Desnoyers]

MEMBARRIER_CMD_GET_REGISTRATIONS returns the registration commands
which the process issued.

Each registration has two state bits: one is set when the registration
starts and the "ready" one when it completes. The command reports a
registration when either bit is set.

The sync core and rseq registrations also set
MEMBARRIER_STATE_PRIVATE_EXPEDITED, without its "ready" bit. They set
it because their commands need the same scheduler barriers as
MEMBARRIER_CMD_PRIVATE_EXPEDITED. So a process which only issued
MEMBARRIER_CMD_REGISTER_PRIVATE_EXPEDITED_RSEQ is reported as having
issued MEMBARRIER_CMD_REGISTER_PRIVATE_EXPEDITED as well. Yet
MEMBARRIER_CMD_PRIVATE_EXPEDITED returns -EPERM for that process.

A registration which failed with -ENOMEM would be reported the same
way.

User-space gets a wrong registration mask. The command was added for
CRIU, which saves the registrations of a process at checkpoint. A
restored process would carry a registration which the original
process did not issue. A program which trusts the mask would issue
MEMBARRIER_CMD_PRIVATE_EXPEDITED and get -EPERM.

Fix this by reporting a registration only when its "ready" bit is
set.

This changes the mask which user-space reads.
MEMBARRIER_CMD_GET_REGISTRATIONS no longer reports
MEMBARRIER_CMD_REGISTER_PRIVATE_EXPEDITED for a process which only
issued the sync core or the rseq registration. The result of every
other command is unchanged.

Fixes: 544a4f2ecd45 ("sched/membarrier: Introduce MEMBARRIER_CMD_GET_REGISTRATIONS")
Cc: stable@vger.kernel.org
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
---
 kernel/sched/membarrier.c | 29 ++++++++++++++---------------
 1 file changed, 14 insertions(+), 15 deletions(-)

diff --git a/kernel/sched/membarrier.c b/kernel/sched/membarrier.c
index cb957b8f1946..46e823a3d668 100644
--- a/kernel/sched/membarrier.c
+++ b/kernel/sched/membarrier.c
@@ -566,15 +566,17 @@ static int membarrier_get_registrations(void)
 	struct task_struct *p = current;
 	struct mm_struct *mm = p->mm;
 	int registrations_mask = 0, membarrier_state, i;
-	static const int states[] = {
-		MEMBARRIER_STATE_GLOBAL_EXPEDITED |
-			MEMBARRIER_STATE_GLOBAL_EXPEDITED_READY,
-		MEMBARRIER_STATE_PRIVATE_EXPEDITED |
-			MEMBARRIER_STATE_PRIVATE_EXPEDITED_READY,
-		MEMBARRIER_STATE_PRIVATE_EXPEDITED_SYNC_CORE |
-			MEMBARRIER_STATE_PRIVATE_EXPEDITED_SYNC_CORE_READY,
-		MEMBARRIER_STATE_PRIVATE_EXPEDITED_RSEQ |
-			MEMBARRIER_STATE_PRIVATE_EXPEDITED_RSEQ_READY
+	/*
+	 * Only the "ready" state tells that a registration command
+	 * completed. The other state of each pair is not specific to
+	 * one registration command: the sync core and rseq
+	 * registrations also set MEMBARRIER_STATE_PRIVATE_EXPEDITED.
+	 */
+	static const int ready_states[] = {
+		MEMBARRIER_STATE_GLOBAL_EXPEDITED_READY,
+		MEMBARRIER_STATE_PRIVATE_EXPEDITED_READY,
+		MEMBARRIER_STATE_PRIVATE_EXPEDITED_SYNC_CORE_READY,
+		MEMBARRIER_STATE_PRIVATE_EXPEDITED_RSEQ_READY
 	};
 	static const int registration_cmds[] = {
 		MEMBARRIER_CMD_REGISTER_GLOBAL_EXPEDITED,
@@ -582,16 +584,13 @@ static int membarrier_get_registrations(void)
 		MEMBARRIER_CMD_REGISTER_PRIVATE_EXPEDITED_SYNC_CORE,
 		MEMBARRIER_CMD_REGISTER_PRIVATE_EXPEDITED_RSEQ
 	};
-	BUILD_BUG_ON(ARRAY_SIZE(states) != ARRAY_SIZE(registration_cmds));
+	BUILD_BUG_ON(ARRAY_SIZE(ready_states) != ARRAY_SIZE(registration_cmds));
 
 	membarrier_state = atomic_read(&mm->membarrier_state);
-	for (i = 0; i < ARRAY_SIZE(states); ++i) {
-		if (membarrier_state & states[i]) {
+	for (i = 0; i < ARRAY_SIZE(ready_states); ++i) {
+		if (membarrier_state & ready_states[i])
 			registrations_mask |= registration_cmds[i];
-			membarrier_state &= ~states[i];
-		}
 	}
-	WARN_ON_ONCE(membarrier_state != 0);
 	return registrations_mask;
 }
 
-- 
2.47.3


_______________________________________________
linux-riscv mailing list
linux-riscv@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-riscv

[RFC PATCH 03/16] selftests/membarrier: Rebuild the tests when their header changes

[Mathieu Desnoyers]

membarrier_test_impl.h holds the test code of
membarrier_test_single_thread and membarrier_test_multi_thread. The
Makefile does not list it as a prerequisite. A change to the header
does not rebuild the two tests, and a stale binary is run.

Add the header to LOCAL_HDRS. lib.mk makes each test depend on these
headers and keeps them out of the compiler command line.

Also add membarrier_rseq_stress to .gitignore.

Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
---
 tools/testing/selftests/membarrier/.gitignore | 1 +
 tools/testing/selftests/membarrier/Makefile   | 2 ++
 2 files changed, 3 insertions(+)

diff --git a/tools/testing/selftests/membarrier/.gitignore b/tools/testing/selftests/membarrier/.gitignore
index f2fbba178601..031920478205 100644
--- a/tools/testing/selftests/membarrier/.gitignore
+++ b/tools/testing/selftests/membarrier/.gitignore
@@ -1,3 +1,4 @@
 # SPDX-License-Identifier: GPL-2.0-only
 membarrier_test_multi_thread
 membarrier_test_single_thread
+membarrier_rseq_stress
diff --git a/tools/testing/selftests/membarrier/Makefile b/tools/testing/selftests/membarrier/Makefile
index 829f95c83515..20bb66380090 100644
--- a/tools/testing/selftests/membarrier/Makefile
+++ b/tools/testing/selftests/membarrier/Makefile
@@ -6,4 +6,6 @@ TEST_GEN_PROGS := membarrier_test_single_thread \
 		membarrier_test_multi_thread \
 		membarrier_rseq_stress
 
+LOCAL_HDRS += membarrier_test_impl.h
+
 include ../lib.mk
-- 
2.47.3

[RFC PATCH 03/16] selftests/membarrier: Rebuild the tests when their header changes

[Mathieu Desnoyers]

membarrier_test_impl.h holds the test code of
membarrier_test_single_thread and membarrier_test_multi_thread. The
Makefile does not list it as a prerequisite. A change to the header
does not rebuild the two tests, and a stale binary is run.

Add the header to LOCAL_HDRS. lib.mk makes each test depend on these
headers and keeps them out of the compiler command line.

Also add membarrier_rseq_stress to .gitignore.

Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
---
 tools/testing/selftests/membarrier/.gitignore | 1 +
 tools/testing/selftests/membarrier/Makefile   | 2 ++
 2 files changed, 3 insertions(+)

diff --git a/tools/testing/selftests/membarrier/.gitignore b/tools/testing/selftests/membarrier/.gitignore
index f2fbba178601..031920478205 100644
--- a/tools/testing/selftests/membarrier/.gitignore
+++ b/tools/testing/selftests/membarrier/.gitignore
@@ -1,3 +1,4 @@
 # SPDX-License-Identifier: GPL-2.0-only
 membarrier_test_multi_thread
 membarrier_test_single_thread
+membarrier_rseq_stress
diff --git a/tools/testing/selftests/membarrier/Makefile b/tools/testing/selftests/membarrier/Makefile
index 829f95c83515..20bb66380090 100644
--- a/tools/testing/selftests/membarrier/Makefile
+++ b/tools/testing/selftests/membarrier/Makefile
@@ -6,4 +6,6 @@ TEST_GEN_PROGS := membarrier_test_single_thread \
 		membarrier_test_multi_thread \
 		membarrier_rseq_stress
 
+LOCAL_HDRS += membarrier_test_impl.h
+
 include ../lib.mk
-- 
2.47.3


_______________________________________________
linux-riscv mailing list
linux-riscv@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-riscv

[RFC PATCH 04/16] selftests/membarrier: Issue the sync core command in its test

[Mathieu Desnoyers]

test_membarrier_private_expedited_sync_core_success() is meant to
check that MEMBARRIER_CMD_PRIVATE_EXPEDITED_SYNC_CORE succeeds after
its registration. It issues MEMBARRIER_CMD_PRIVATE_EXPEDITED instead.
No test issues the sync core command with success.

Issue the sync core command.

While at it, pass the third system call argument. It was left
uninitialized.

Fixes: 460e8c3340a2 ("membarrier/selftest: Test private expedited sync core command")
Cc: stable@vger.kernel.org
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
---
 tools/testing/selftests/membarrier/membarrier_test_impl.h | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/tools/testing/selftests/membarrier/membarrier_test_impl.h b/tools/testing/selftests/membarrier/membarrier_test_impl.h
index b4dcbb32538d..4b9d1771916e 100644
--- a/tools/testing/selftests/membarrier/membarrier_test_impl.h
+++ b/tools/testing/selftests/membarrier/membarrier_test_impl.h
@@ -13,7 +13,7 @@ static int registrations;
 
 static int sys_membarrier(int cmd, int flags)
 {
-	return syscall(__NR_membarrier, cmd, flags);
+	return syscall(__NR_membarrier, cmd, flags, 0);
 }
 
 static int membarrier_get_registrations(void)
@@ -221,7 +221,7 @@ static int test_membarrier_register_private_expedited_sync_core_success(void)
 
 static int test_membarrier_private_expedited_sync_core_success(void)
 {
-	int cmd = MEMBARRIER_CMD_PRIVATE_EXPEDITED, flags = 0;
+	int cmd = MEMBARRIER_CMD_PRIVATE_EXPEDITED_SYNC_CORE, flags = 0;
 	const char *test_name = "sys membarrier MEMBARRIER_CMD_PRIVATE_EXPEDITED_SYNC_CORE";
 
 	if (sys_membarrier(cmd, flags) != 0) {
-- 
2.47.3

[RFC PATCH 04/16] selftests/membarrier: Issue the sync core command in its test

[Mathieu Desnoyers]

test_membarrier_private_expedited_sync_core_success() is meant to
check that MEMBARRIER_CMD_PRIVATE_EXPEDITED_SYNC_CORE succeeds after
its registration. It issues MEMBARRIER_CMD_PRIVATE_EXPEDITED instead.
No test issues the sync core command with success.

Issue the sync core command.

While at it, pass the third system call argument. It was left
uninitialized.

Fixes: 460e8c3340a2 ("membarrier/selftest: Test private expedited sync core command")
Cc: stable@vger.kernel.org
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
---
 tools/testing/selftests/membarrier/membarrier_test_impl.h | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/tools/testing/selftests/membarrier/membarrier_test_impl.h b/tools/testing/selftests/membarrier/membarrier_test_impl.h
index b4dcbb32538d..4b9d1771916e 100644
--- a/tools/testing/selftests/membarrier/membarrier_test_impl.h
+++ b/tools/testing/selftests/membarrier/membarrier_test_impl.h
@@ -13,7 +13,7 @@ static int registrations;
 
 static int sys_membarrier(int cmd, int flags)
 {
-	return syscall(__NR_membarrier, cmd, flags);
+	return syscall(__NR_membarrier, cmd, flags, 0);
 }
 
 static int membarrier_get_registrations(void)
@@ -221,7 +221,7 @@ static int test_membarrier_register_private_expedited_sync_core_success(void)
 
 static int test_membarrier_private_expedited_sync_core_success(void)
 {
-	int cmd = MEMBARRIER_CMD_PRIVATE_EXPEDITED, flags = 0;
+	int cmd = MEMBARRIER_CMD_PRIVATE_EXPEDITED_SYNC_CORE, flags = 0;
 	const char *test_name = "sys membarrier MEMBARRIER_CMD_PRIVATE_EXPEDITED_SYNC_CORE";
 
 	if (sys_membarrier(cmd, flags) != 0) {
-- 
2.47.3


_______________________________________________
linux-riscv mailing list
linux-riscv@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-riscv

[RFC PATCH 05/16] selftests/membarrier: Run the tests on nohz_full kernels

[Mathieu Desnoyers]

A nohz_full kernel does not provide MEMBARRIER_CMD_GLOBAL. The tests
skip everything when the query mask lacks this command. The other
commands are available on such a kernel and are never tested there.

Skip only the MEMBARRIER_CMD_GLOBAL test.

Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
---
 .../selftests/membarrier/membarrier_test_impl.h        | 10 +++++++---
 1 file changed, 7 insertions(+), 3 deletions(-)

diff --git a/tools/testing/selftests/membarrier/membarrier_test_impl.h b/tools/testing/selftests/membarrier/membarrier_test_impl.h
index 4b9d1771916e..49f5fdb56dab 100644
--- a/tools/testing/selftests/membarrier/membarrier_test_impl.h
+++ b/tools/testing/selftests/membarrier/membarrier_test_impl.h
@@ -97,6 +97,13 @@ static int test_membarrier_global_success(void)
 	int cmd = MEMBARRIER_CMD_GLOBAL, flags = 0;
 	const char *test_name = "sys membarrier MEMBARRIER_CMD_GLOBAL";
 
+	/* A nohz_full kernel does not provide MEMBARRIER_CMD_GLOBAL. */
+	if (!(sys_membarrier(MEMBARRIER_CMD_QUERY, 0) & cmd)) {
+		ksft_test_result_skip("%s test: Command not available\n",
+				      test_name);
+		return 0;
+	}
+
 	if (sys_membarrier(cmd, flags) != 0) {
 		ksft_exit_fail_msg(
 			"%s test: flags = %d, errno = %d\n",
@@ -356,9 +363,6 @@ static int test_membarrier_query(void)
 		}
 		ksft_exit_fail_msg("sys_membarrier() failed\n");
 	}
-	if (!(ret & MEMBARRIER_CMD_GLOBAL))
-		ksft_exit_skip(
-			"sys_membarrier unsupported: CMD_GLOBAL not found.\n");
 
 	ksft_test_result_pass("sys_membarrier available\n");
 	return 0;
-- 
2.47.3

[RFC PATCH 05/16] selftests/membarrier: Run the tests on nohz_full kernels

[Mathieu Desnoyers]

A nohz_full kernel does not provide MEMBARRIER_CMD_GLOBAL. The tests
skip everything when the query mask lacks this command. The other
commands are available on such a kernel and are never tested there.

Skip only the MEMBARRIER_CMD_GLOBAL test.

Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
---
 .../selftests/membarrier/membarrier_test_impl.h        | 10 +++++++---
 1 file changed, 7 insertions(+), 3 deletions(-)

diff --git a/tools/testing/selftests/membarrier/membarrier_test_impl.h b/tools/testing/selftests/membarrier/membarrier_test_impl.h
index 4b9d1771916e..49f5fdb56dab 100644
--- a/tools/testing/selftests/membarrier/membarrier_test_impl.h
+++ b/tools/testing/selftests/membarrier/membarrier_test_impl.h
@@ -97,6 +97,13 @@ static int test_membarrier_global_success(void)
 	int cmd = MEMBARRIER_CMD_GLOBAL, flags = 0;
 	const char *test_name = "sys membarrier MEMBARRIER_CMD_GLOBAL";
 
+	/* A nohz_full kernel does not provide MEMBARRIER_CMD_GLOBAL. */
+	if (!(sys_membarrier(MEMBARRIER_CMD_QUERY, 0) & cmd)) {
+		ksft_test_result_skip("%s test: Command not available\n",
+				      test_name);
+		return 0;
+	}
+
 	if (sys_membarrier(cmd, flags) != 0) {
 		ksft_exit_fail_msg(
 			"%s test: flags = %d, errno = %d\n",
@@ -356,9 +363,6 @@ static int test_membarrier_query(void)
 		}
 		ksft_exit_fail_msg("sys_membarrier() failed\n");
 	}
-	if (!(ret & MEMBARRIER_CMD_GLOBAL))
-		ksft_exit_skip(
-			"sys_membarrier unsupported: CMD_GLOBAL not found.\n");
 
 	ksft_test_result_pass("sys_membarrier available\n");
 	return 0;
-- 
2.47.3


_______________________________________________
linux-riscv mailing list
linux-riscv@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-riscv

[RFC PATCH 06/16] selftests/membarrier: Test which CPUs the expedited commands interrupt

[Mathieu Desnoyers]

The existing tests only check the return value of each command. They
cannot tell whether a command reached the threads it must reach.

Add membarrier_test_ipi. A test cannot observe a missing memory
barrier reliably. It can observe a missing IPI in /proc/interrupts.
The comment at the top of the file describes the method. The test
cases are:

 - global expedited reaches a registered process and skips an
   unregistered one,
 - global expedited reaches a registered process after one of its
   threads exits,
 - global expedited reaches a process which registered on another CPU
   and came back to a CPU which stayed idle meanwhile,
 - global expedited reaches each thread of a process which registered
   while multi-threaded,
 - global expedited reaches a forked child and skips a process after
   exec,
 - private expedited, sync core and rseq reach each thread,
 - rseq with MEMBARRIER_CMD_FLAG_CPU reaches only the CPU it names,
 - private expedited skips another registered process,
 - private expedited reaches a CLONE_VM task of another thread group.

Each test case runs in its own process, because a membarrier
registration cannot be undone.

The thread exit and idle CPU cases fail without commit
"sched/membarrier: Fix stale runqueue state after a lazy mm switch".

The test is skipped with fewer than 3 allowed CPUs. It is also skipped
when /proc/interrupts has no function call interrupt count.

Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
---
 tools/testing/selftests/membarrier/.gitignore |   1 +
 tools/testing/selftests/membarrier/Makefile   |   1 +
 .../membarrier/membarrier_test_ipi.c          | 739 ++++++++++++++++++
 3 files changed, 741 insertions(+)
 create mode 100644 tools/testing/selftests/membarrier/membarrier_test_ipi.c

diff --git a/tools/testing/selftests/membarrier/.gitignore b/tools/testing/selftests/membarrier/.gitignore
index 031920478205..58969a5e4613 100644
--- a/tools/testing/selftests/membarrier/.gitignore
+++ b/tools/testing/selftests/membarrier/.gitignore
@@ -2,3 +2,4 @@
 membarrier_test_multi_thread
 membarrier_test_single_thread
 membarrier_rseq_stress
+membarrier_test_ipi
diff --git a/tools/testing/selftests/membarrier/Makefile b/tools/testing/selftests/membarrier/Makefile
index 20bb66380090..ee5b9deb808b 100644
--- a/tools/testing/selftests/membarrier/Makefile
+++ b/tools/testing/selftests/membarrier/Makefile
@@ -4,6 +4,7 @@ LDLIBS += -lpthread
 
 TEST_GEN_PROGS := membarrier_test_single_thread \
 		membarrier_test_multi_thread \
+		membarrier_test_ipi \
 		membarrier_rseq_stress
 
 LOCAL_HDRS += membarrier_test_impl.h
diff --git a/tools/testing/selftests/membarrier/membarrier_test_ipi.c b/tools/testing/selftests/membarrier/membarrier_test_ipi.c
new file mode 100644
index 000000000000..49b74844c1f4
--- /dev/null
+++ b/tools/testing/selftests/membarrier/membarrier_test_ipi.c
@@ -0,0 +1,739 @@
+// SPDX-License-Identifier: GPL-2.0
+/*
+ * Check which CPUs the expedited membarrier commands interrupt.
+ *
+ * An expedited command sends an IPI to each CPU which runs a targeted
+ * thread. The IPI is the memory barrier. A test cannot observe a
+ * missing memory barrier reliably. It can observe a missing IPI: the
+ * kernel counts the function call interrupts of each CPU in
+ * /proc/interrupts.
+ *
+ * Each test case pins a "target" which spins in user-space on its own
+ * CPU. A "sender" pinned on another CPU issues the command NR_CMDS
+ * times. The test then compares the count of function call interrupts
+ * of the target CPU before and after.
+ *
+ * Each test case runs in its own process because a membarrier
+ * registration cannot be undone.
+ */
+#define _GNU_SOURCE
+#include <linux/membarrier.h>
+#include <sys/mman.h>
+#include <sys/prctl.h>
+#include <sys/wait.h>
+#include <syscall.h>
+#include <stdatomic.h>
+#include <pthread.h>
+#include <signal.h>
+#include <stdbool.h>
+#include <stdlib.h>
+#include <string.h>
+#include <unistd.h>
+#include <sched.h>
+#include <stdio.h>
+#include <errno.h>
+#include <time.h>
+
+#include "kselftest.h"
+
+#define NR_CMDS		1000
+/*
+ * A target which must be interrupted can lose a few IPIs when another
+ * task preempts it. A target which must not be interrupted can receive
+ * a few unrelated IPIs. The observed counts are either close to
+ * NR_CMDS or close to zero. Keep a wide margin on both sides.
+ */
+#define MIN_HIT		(NR_CMDS / 2)
+#define MAX_MISS	(NR_CMDS / 4)
+
+#define MAX_TARGETS	2
+/* Seconds to wait for the targets of a test case. */
+#define READY_TIMEOUT	10
+#define STACK_SIZE	(64 * 1024)
+
+/*
+ * Exit status of a test case process. Each skip status has its own
+ * reason, which run_case() reports. CASE_FAIL is a failure of
+ * membarrier. CASE_ERROR is a failure of the test setup.
+ */
+enum {
+	CASE_PASS = 0,
+	CASE_FAIL = 1,
+	CASE_SKIP_NO_COUNT = 2,
+	CASE_SKIP_NO_CMD = 3,
+	CASE_ERROR = 4,
+};
+
+struct shared {
+	atomic_int ready;
+	atomic_int stop;
+};
+
+static struct shared *shared;
+static int shared_fd;
+
+/* CPU of the sender and CPUs of the targets. */
+static int sender_cpu, target_cpu[MAX_TARGETS];
+
+static int sys_membarrier(int cmd, unsigned int flags, int cpu_id)
+{
+	return syscall(__NR_membarrier, cmd, flags, cpu_id);
+}
+
+/* Pin the task @pid on @cpu. A @pid of 0 is the calling thread. */
+static void pin_task(pid_t pid, int cpu)
+{
+	cpu_set_t set;
+
+	CPU_ZERO(&set);
+	CPU_SET(cpu, &set);
+	if (sched_setaffinity(pid, sizeof(set), &set)) {
+		perror("sched_setaffinity");
+		_exit(CASE_ERROR);
+	}
+}
+
+static void pin_to_cpu(int cpu)
+{
+	pin_task(0, cpu);
+}
+
+/* Die with the parent. A killed test must leave no spinning target. */
+static void die_with_parent(void)
+{
+	if (prctl(PR_SET_PDEATHSIG, SIGKILL)) {
+		perror("prctl");
+		_exit(CASE_ERROR);
+	}
+}
+
+static void must_join(pthread_t thread)
+{
+	errno = pthread_join(thread, NULL);
+	if (errno) {
+		perror("pthread_join");
+		_exit(CASE_ERROR);
+	}
+}
+
+/* Wait for the child @pid. Its setup failed unless it exits with 0. */
+static void must_wait(pid_t pid)
+{
+	int status;
+
+	if (waitpid(pid, &status, 0) != pid) {
+		perror("waitpid");
+		_exit(CASE_ERROR);
+	}
+	if (!WIFEXITED(status) || WEXITSTATUS(status)) {
+		fprintf(stderr, "# child %d did not exit with 0\n", pid);
+		_exit(CASE_ERROR);
+	}
+}
+
+static void must_register(int cmd)
+{
+	if (sys_membarrier(cmd, 0, 0)) {
+		perror("membarrier registration");
+		_exit(CASE_ERROR);
+	}
+}
+
+/* Announce that this target is in place, then spin in user-space. */
+static void spin(void)
+{
+	die_with_parent();
+	atomic_fetch_add(&shared->ready, 1);
+	while (!atomic_load(&shared->stop))
+		;
+}
+
+/*
+ * Wait until @nr targets spin. Report a setup error when they do not
+ * show up: a target may have died in its setup.
+ */
+static void wait_ready(int nr)
+{
+	time_t start = time(NULL);
+
+	while (atomic_load(&shared->ready) < nr) {
+		if (time(NULL) - start > READY_TIMEOUT) {
+			fprintf(stderr, "# the targets are not ready\n");
+			atomic_store(&shared->stop, 1);
+			_exit(CASE_ERROR);
+		}
+		sched_yield();
+	}
+}
+
+/*
+ * Read the count of function call interrupts of each CPU of @cpus.
+ * Return false when /proc/interrupts has no such count for one of
+ * them. An offline CPU has no count.
+ */
+static bool read_ipi_counts(const int *cpus, int nr_cpus, long *counts)
+{
+	int *col_cpu = NULL, nr_cols = 0, i, c;
+	char *line = NULL, *p, *end;
+	unsigned int seen = 0;
+	size_t len = 0;
+	FILE *f;
+
+	for (c = 0; c < nr_cpus; c++)
+		counts[c] = 0;
+	f = fopen("/proc/interrupts", "r");
+	if (!f)
+		return false;
+	/* The first line names the CPU of each column. */
+	if (getline(&line, &len, f) < 0)
+		goto out;
+	for (p = line; (p = strstr(p, "CPU")); p = end) {
+		int cpu = strtol(p + 3, &end, 10);
+
+		int *new_col_cpu;
+
+		new_col_cpu = realloc(col_cpu, (nr_cols + 1) * sizeof(*col_cpu));
+		if (!new_col_cpu)
+			goto out;
+		col_cpu = new_col_cpu;
+		col_cpu[nr_cols++] = cpu;
+	}
+	while (getline(&line, &len, f) >= 0) {
+		/* x86 uses the "CAL" label. Others describe the line. */
+		if (!strstr(line, "CAL:") && !strstr(line, "unction call"))
+			continue;
+		p = strchr(line, ':');
+		if (!p)
+			continue;
+		p++;
+		for (i = 0; i < nr_cols; i++) {
+			long val = strtol(p, &end, 10);
+
+			if (end == p)
+				break;
+			p = end;
+			for (c = 0; c < nr_cpus; c++)
+				if (cpus[c] == col_cpu[i]) {
+					counts[c] += val;
+					seen |= 1U << c;
+				}
+		}
+	}
+out:
+	free(col_cpu);
+	free(line);
+	fclose(f);
+	return seen == (1U << nr_cpus) - 1;
+}
+
+/*
+ * Issue @cmd NR_CMDS times. Store into @delta the count of function
+ * call interrupts which each target CPU received meanwhile.
+ */
+static int send_cmds(int cmd, unsigned int flags, int cpu_id, long *delta)
+{
+	long before[MAX_TARGETS], after[MAX_TARGETS];
+	int i;
+
+	if (!read_ipi_counts(target_cpu, MAX_TARGETS, before))
+		return CASE_SKIP_NO_COUNT;
+	for (i = 0; i < NR_CMDS; i++) {
+		if (sys_membarrier(cmd, flags, cpu_id)) {
+			perror("membarrier");
+			return CASE_FAIL;
+		}
+	}
+	if (!read_ipi_counts(target_cpu, MAX_TARGETS, after))
+		return CASE_SKIP_NO_COUNT;
+	for (i = 0; i < MAX_TARGETS; i++)
+		delta[i] = after[i] - before[i];
+	return CASE_PASS;
+}
+
+static int expect_hit(int target, long *delta)
+{
+	if (delta[target] >= MIN_HIT)
+		return CASE_PASS;
+	fprintf(stderr, "# CPU %d received %ld IPIs for %d commands\n",
+		target_cpu[target], delta[target], NR_CMDS);
+	return CASE_FAIL;
+}
+
+static int expect_miss(int target, long *delta)
+{
+	if (delta[target] <= MAX_MISS)
+		return CASE_PASS;
+	fprintf(stderr, "# CPU %d received %ld IPIs, expected none\n",
+		target_cpu[target], delta[target]);
+	return CASE_FAIL;
+}
+
+/*
+ * Global expedited test cases.
+ *
+ * The sender is the test case process. The target is a child process.
+ * @setup runs in the child and ends with spin().
+ */
+
+static void *thread_nop(void *arg)
+{
+	return NULL;
+}
+
+static void *thread_spin(void *arg)
+{
+	pin_to_cpu(target_cpu[(long)arg]);
+	spin();
+	return NULL;
+}
+
+static void setup_registered(void)
+{
+	must_register(MEMBARRIER_CMD_REGISTER_GLOBAL_EXPEDITED);
+	spin();
+}
+
+static void setup_unregistered(void)
+{
+	spin();
+}
+
+/*
+ * The exiting thread clears the runqueue membarrier state in
+ * exit_mm(). The scheduler then switches to the main thread, which
+ * has the same mm.
+ */
+static void setup_thread_exit(void)
+{
+	pthread_t thread;
+
+	must_register(MEMBARRIER_CMD_REGISTER_GLOBAL_EXPEDITED);
+	if (pthread_create(&thread, NULL, thread_nop, NULL))
+		_exit(CASE_ERROR);
+	must_join(thread);
+	spin();
+}
+
+/*
+ * Register on another CPU. Meanwhile the first CPU is idle and its
+ * idle thread borrows the mm of this process. Come back to the first
+ * CPU: the scheduler switches from the idle thread to this process.
+ */
+static void setup_lazy_tlb(void)
+{
+	pin_to_cpu(target_cpu[1]);
+	must_register(MEMBARRIER_CMD_REGISTER_GLOBAL_EXPEDITED);
+	pin_to_cpu(target_cpu[0]);
+	spin();
+}
+
+/* Register from a multi-threaded process. Both threads are targets. */
+static void setup_multi_thread(void)
+{
+	pthread_t thread;
+
+	if (pthread_create(&thread, NULL, thread_spin, (void *)1L))
+		_exit(CASE_ERROR);
+	wait_ready(1);
+	must_register(MEMBARRIER_CMD_REGISTER_GLOBAL_EXPEDITED);
+	spin();
+}
+
+/* A forked child inherits the registration. */
+static void setup_fork(void)
+{
+	pid_t pid;
+
+	must_register(MEMBARRIER_CMD_REGISTER_GLOBAL_EXPEDITED);
+	pid = fork();
+	if (pid < 0)
+		_exit(CASE_ERROR);
+	if (!pid) {
+		spin();
+		_exit(0);
+	}
+	must_wait(pid);
+}
+
+/* exec() drops the registration. */
+static void setup_exec(void)
+{
+	char fd_str[16];
+
+	must_register(MEMBARRIER_CMD_REGISTER_GLOBAL_EXPEDITED);
+	snprintf(fd_str, sizeof(fd_str), "%d", shared_fd);
+	execl("/proc/self/exe", "membarrier_test_ipi", "--spin", fd_str, NULL);
+	perror("execl");
+	_exit(CASE_ERROR);
+}
+
+static int test_global(void (*setup)(void), int nr_targets, bool hit)
+{
+	long delta[MAX_TARGETS];
+	int ret, i;
+	pid_t pid;
+
+	if (!(sys_membarrier(MEMBARRIER_CMD_QUERY, 0, 0) &
+	      MEMBARRIER_CMD_GLOBAL_EXPEDITED))
+		return CASE_SKIP_NO_CMD;
+	pid = fork();
+	if (pid < 0)
+		return CASE_ERROR;
+	if (!pid) {
+		die_with_parent();
+		pin_to_cpu(target_cpu[0]);
+		setup();
+		_exit(0);
+	}
+	wait_ready(nr_targets);
+	ret = send_cmds(MEMBARRIER_CMD_GLOBAL_EXPEDITED, 0, 0, delta);
+	atomic_store(&shared->stop, 1);
+	must_wait(pid);
+	for (i = 0; !ret && i < nr_targets; i++)
+		ret = hit ? expect_hit(i, delta) : expect_miss(i, delta);
+	return ret;
+}
+
+static int case_global_registered(void)
+{
+	return test_global(setup_registered, 1, true);
+}
+
+static int case_global_unregistered(void)
+{
+	return test_global(setup_unregistered, 1, false);
+}
+
+static int case_global_thread_exit(void)
+{
+	return test_global(setup_thread_exit, 1, true);
+}
+
+static int case_global_lazy_tlb(void)
+{
+	return test_global(setup_lazy_tlb, 1, true);
+}
+
+static int case_global_multi_thread(void)
+{
+	return test_global(setup_multi_thread, 2, true);
+}
+
+static int case_global_fork(void)
+{
+	return test_global(setup_fork, 1, true);
+}
+
+static int case_global_exec(void)
+{
+	return test_global(setup_exec, 1, false);
+}
+
+/*
+ * Private expedited test cases.
+ *
+ * The sender is the main thread of the test case process. The targets
+ * are threads of the same process, one per target CPU.
+ */
+
+static int start_thread_targets(pthread_t *threads)
+{
+	long i;
+
+	for (i = 0; i < MAX_TARGETS; i++)
+		if (pthread_create(&threads[i], NULL, thread_spin, (void *)i))
+			return CASE_ERROR;
+	wait_ready(MAX_TARGETS);
+	return CASE_PASS;
+}
+
+static void stop_thread_targets(pthread_t *threads)
+{
+	int i;
+
+	atomic_store(&shared->stop, 1);
+	for (i = 0; i < MAX_TARGETS; i++)
+		must_join(threads[i]);
+}
+
+static int test_private(int register_cmd, int cmd)
+{
+	pthread_t threads[MAX_TARGETS];
+	long delta[MAX_TARGETS];
+	int ret, i;
+
+	if (!(sys_membarrier(MEMBARRIER_CMD_QUERY, 0, 0) & cmd))
+		return CASE_SKIP_NO_CMD;
+	if (start_thread_targets(threads))
+		return CASE_ERROR;
+	must_register(register_cmd);
+	ret = send_cmds(cmd, 0, 0, delta);
+	stop_thread_targets(threads);
+	for (i = 0; !ret && i < MAX_TARGETS; i++)
+		ret = expect_hit(i, delta);
+	return ret;
+}
+
+static int case_private(void)
+{
+	return test_private(MEMBARRIER_CMD_REGISTER_PRIVATE_EXPEDITED,
+			    MEMBARRIER_CMD_PRIVATE_EXPEDITED);
+}
+
+static int case_private_sync_core(void)
+{
+	return test_private(MEMBARRIER_CMD_REGISTER_PRIVATE_EXPEDITED_SYNC_CORE,
+			    MEMBARRIER_CMD_PRIVATE_EXPEDITED_SYNC_CORE);
+}
+
+static int case_private_rseq(void)
+{
+	return test_private(MEMBARRIER_CMD_REGISTER_PRIVATE_EXPEDITED_RSEQ,
+			    MEMBARRIER_CMD_PRIVATE_EXPEDITED_RSEQ);
+}
+
+/* MEMBARRIER_CMD_FLAG_CPU interrupts only the CPU it names. */
+static int case_private_rseq_cpu(void)
+{
+	pthread_t threads[MAX_TARGETS];
+	long delta[MAX_TARGETS];
+	int ret;
+
+	if (!(sys_membarrier(MEMBARRIER_CMD_QUERY, 0, 0) &
+	      MEMBARRIER_CMD_PRIVATE_EXPEDITED_RSEQ))
+		return CASE_SKIP_NO_CMD;
+	if (start_thread_targets(threads))
+		return CASE_ERROR;
+	must_register(MEMBARRIER_CMD_REGISTER_PRIVATE_EXPEDITED_RSEQ);
+	ret = send_cmds(MEMBARRIER_CMD_PRIVATE_EXPEDITED_RSEQ,
+			MEMBARRIER_CMD_FLAG_CPU, target_cpu[0], delta);
+	stop_thread_targets(threads);
+	if (!ret)
+		ret = expect_hit(0, delta);
+	if (!ret)
+		ret = expect_miss(1, delta);
+	return ret;
+}
+
+/* A private command does not interrupt another registered process. */
+static int case_private_other_process(void)
+{
+	long delta[MAX_TARGETS];
+	pthread_t thread;
+	pid_t pid;
+	int ret;
+
+	if (!(sys_membarrier(MEMBARRIER_CMD_QUERY, 0, 0) &
+	      MEMBARRIER_CMD_PRIVATE_EXPEDITED))
+		return CASE_SKIP_NO_CMD;
+	pid = fork();
+	if (pid < 0)
+		return CASE_ERROR;
+	if (!pid) {
+		pin_to_cpu(target_cpu[0]);
+		must_register(MEMBARRIER_CMD_REGISTER_PRIVATE_EXPEDITED);
+		spin();
+		_exit(0);
+	}
+	if (pthread_create(&thread, NULL, thread_spin, (void *)1L))
+		return CASE_ERROR;
+	wait_ready(2);
+	must_register(MEMBARRIER_CMD_REGISTER_PRIVATE_EXPEDITED);
+	ret = send_cmds(MEMBARRIER_CMD_PRIVATE_EXPEDITED, 0, 0, delta);
+	atomic_store(&shared->stop, 1);
+	must_join(thread);
+	must_wait(pid);
+	if (!ret)
+		ret = expect_miss(0, delta);
+	if (!ret)
+		ret = expect_hit(1, delta);
+	return ret;
+}
+
+/*
+ * This task shares the TLS of its parent. Its parent pins it, so that
+ * it only spins.
+ */
+static int clone_vm_fn(void *arg)
+{
+	spin();
+	return 0;
+}
+
+/*
+ * A private command covers every task which uses the mm. This includes
+ * a task of another thread group created with CLONE_VM.
+ */
+static int case_private_clone_vm(void)
+{
+	long delta[MAX_TARGETS];
+	char *stack;
+	pid_t pid;
+	int ret;
+
+	if (!(sys_membarrier(MEMBARRIER_CMD_QUERY, 0, 0) &
+	      MEMBARRIER_CMD_PRIVATE_EXPEDITED))
+		return CASE_SKIP_NO_CMD;
+	stack = malloc(STACK_SIZE);
+	if (!stack)
+		return CASE_ERROR;
+	pid = clone(clone_vm_fn, stack + STACK_SIZE, CLONE_VM | SIGCHLD, NULL);
+	if (pid < 0) {
+		free(stack);
+		return CASE_ERROR;
+	}
+	pin_task(pid, target_cpu[0]);
+	wait_ready(1);
+	must_register(MEMBARRIER_CMD_REGISTER_PRIVATE_EXPEDITED);
+	ret = send_cmds(MEMBARRIER_CMD_PRIVATE_EXPEDITED, 0, 0, delta);
+	atomic_store(&shared->stop, 1);
+	must_wait(pid);
+	free(stack);
+	if (!ret)
+		ret = expect_hit(0, delta);
+	return ret;
+}
+
+static const struct test_case {
+	const char *name;
+	int (*fn)(void);
+} test_cases[] = {
+	{ "global expedited reaches a registered process",
+	  case_global_registered },
+	{ "global expedited skips an unregistered process",
+	  case_global_unregistered },
+	{ "global expedited reaches a process after a thread exit",
+	  case_global_thread_exit },
+	{ "global expedited reaches a process registered on another CPU",
+	  case_global_lazy_tlb },
+	{ "global expedited reaches each thread of a process",
+	  case_global_multi_thread },
+	{ "global expedited reaches a forked child",
+	  case_global_fork },
+	{ "global expedited skips a process after exec",
+	  case_global_exec },
+	{ "private expedited reaches each thread",
+	  case_private },
+	{ "private expedited sync core reaches each thread",
+	  case_private_sync_core },
+	{ "private expedited rseq reaches each thread",
+	  case_private_rseq },
+	{ "private expedited rseq with a CPU reaches only that CPU",
+	  case_private_rseq_cpu },
+	{ "private expedited skips another process",
+	  case_private_other_process },
+	{ "private expedited reaches a CLONE_VM task",
+	  case_private_clone_vm },
+};
+
+static void run_case(const struct test_case *tc)
+{
+	int status;
+	pid_t pid;
+
+	atomic_store(&shared->ready, 0);
+	atomic_store(&shared->stop, 0);
+	pid = fork();
+	if (pid < 0)
+		ksft_exit_fail_msg("fork failed\n");
+	if (!pid) {
+		/* The targets inherit this process group. */
+		if (setpgid(0, 0)) {
+			perror("setpgid");
+			_exit(CASE_ERROR);
+		}
+		die_with_parent();
+		pin_to_cpu(sender_cpu);
+		_exit(tc->fn());
+	}
+	if (waitpid(pid, &status, 0) != pid)
+		ksft_exit_fail_msg("waitpid failed\n");
+	/*
+	 * Kill the targets which a failed test case left behind. There
+	 * is usually none: kill() then fails with ESRCH.
+	 */
+	atomic_store(&shared->stop, 1);
+	if (kill(-pid, SIGKILL) && errno != ESRCH)
+		ksft_exit_fail_msg("kill failed\n");
+	if (!WIFEXITED(status))
+		status = CASE_FAIL;
+	else
+		status = WEXITSTATUS(status);
+	switch (status) {
+	case CASE_PASS:
+		ksft_test_result_pass("%s\n", tc->name);
+		break;
+	case CASE_SKIP_NO_COUNT:
+		ksft_test_result_skip("%s: Cannot read the interrupt counts\n",
+				      tc->name);
+		break;
+	case CASE_SKIP_NO_CMD:
+		ksft_test_result_skip("%s: Command not available\n",
+				      tc->name);
+		break;
+	case CASE_ERROR:
+		ksft_test_result_error("%s: Test setup failed\n", tc->name);
+		break;
+	default:
+		ksft_test_result_fail("%s\n", tc->name);
+	}
+}
+
+/* Pick the sender CPU and the target CPUs among the allowed CPUs. */
+static bool pick_cpus(void)
+{
+	int cpu, nr = 0, *dst[] = {
+		&sender_cpu, &target_cpu[0], &target_cpu[1],
+	};
+	cpu_set_t set;
+
+	if (sched_getaffinity(0, sizeof(set), &set))
+		return false;
+	for (cpu = 0; cpu < CPU_SETSIZE && nr < ARRAY_SIZE(dst); cpu++)
+		if (CPU_ISSET(cpu, &set))
+			*dst[nr++] = cpu;
+	return nr == ARRAY_SIZE(dst);
+}
+
+static struct shared *map_shared(int fd)
+{
+	void *p;
+
+	p = mmap(NULL, sizeof(struct shared), PROT_READ | PROT_WRITE,
+		 MAP_SHARED, fd, 0);
+	return p == MAP_FAILED ? NULL : p;
+}
+
+int main(int argc, char **argv)
+{
+	long counts[MAX_TARGETS];
+	int i;
+
+	/* Target of the exec test case. See setup_exec(). */
+	if (argc == 3 && !strcmp(argv[1], "--spin")) {
+		shared = map_shared(atoi(argv[2]));
+		if (!shared)
+			return CASE_ERROR;
+		spin();
+		return 0;
+	}
+
+	ksft_print_header();
+	if (sys_membarrier(MEMBARRIER_CMD_QUERY, 0, 0) < 0)
+		ksft_exit_skip("sys membarrier (CONFIG_MEMBARRIER) is disabled.\n");
+	if (!pick_cpus())
+		ksft_exit_skip("This test needs 3 allowed CPUs.\n");
+	if (!read_ipi_counts(target_cpu, MAX_TARGETS, counts))
+		ksft_exit_skip("No function call interrupt count in /proc/interrupts.\n");
+	shared_fd = memfd_create("membarrier_test_ipi", 0);
+	if (shared_fd < 0 || ftruncate(shared_fd, sizeof(struct shared)))
+		ksft_exit_fail_msg("memfd_create failed\n");
+	shared = map_shared(shared_fd);
+	if (!shared)
+		ksft_exit_fail_msg("mmap failed\n");
+
+	ksft_set_plan(ARRAY_SIZE(test_cases));
+	for (i = 0; i < ARRAY_SIZE(test_cases); i++)
+		run_case(&test_cases[i]);
+	ksft_finished();
+}
-- 
2.47.3

[RFC PATCH 06/16] selftests/membarrier: Test which CPUs the expedited commands interrupt

[Mathieu Desnoyers]

The existing tests only check the return value of each command. They
cannot tell whether a command reached the threads it must reach.

Add membarrier_test_ipi. A test cannot observe a missing memory
barrier reliably. It can observe a missing IPI in /proc/interrupts.
The comment at the top of the file describes the method. The test
cases are:

 - global expedited reaches a registered process and skips an
   unregistered one,
 - global expedited reaches a registered process after one of its
   threads exits,
 - global expedited reaches a process which registered on another CPU
   and came back to a CPU which stayed idle meanwhile,
 - global expedited reaches each thread of a process which registered
   while multi-threaded,
 - global expedited reaches a forked child and skips a process after
   exec,
 - private expedited, sync core and rseq reach each thread,
 - rseq with MEMBARRIER_CMD_FLAG_CPU reaches only the CPU it names,
 - private expedited skips another registered process,
 - private expedited reaches a CLONE_VM task of another thread group.

Each test case runs in its own process, because a membarrier
registration cannot be undone.

The thread exit and idle CPU cases fail without commit
"sched/membarrier: Fix stale runqueue state after a lazy mm switch".

The test is skipped with fewer than 3 allowed CPUs. It is also skipped
when /proc/interrupts has no function call interrupt count.

Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
---
 tools/testing/selftests/membarrier/.gitignore |   1 +
 tools/testing/selftests/membarrier/Makefile   |   1 +
 .../membarrier/membarrier_test_ipi.c          | 739 ++++++++++++++++++
 3 files changed, 741 insertions(+)
 create mode 100644 tools/testing/selftests/membarrier/membarrier_test_ipi.c

diff --git a/tools/testing/selftests/membarrier/.gitignore b/tools/testing/selftests/membarrier/.gitignore
index 031920478205..58969a5e4613 100644
--- a/tools/testing/selftests/membarrier/.gitignore
+++ b/tools/testing/selftests/membarrier/.gitignore
@@ -2,3 +2,4 @@
 membarrier_test_multi_thread
 membarrier_test_single_thread
 membarrier_rseq_stress
+membarrier_test_ipi
diff --git a/tools/testing/selftests/membarrier/Makefile b/tools/testing/selftests/membarrier/Makefile
index 20bb66380090..ee5b9deb808b 100644
--- a/tools/testing/selftests/membarrier/Makefile
+++ b/tools/testing/selftests/membarrier/Makefile
@@ -4,6 +4,7 @@ LDLIBS += -lpthread
 
 TEST_GEN_PROGS := membarrier_test_single_thread \
 		membarrier_test_multi_thread \
+		membarrier_test_ipi \
 		membarrier_rseq_stress
 
 LOCAL_HDRS += membarrier_test_impl.h
diff --git a/tools/testing/selftests/membarrier/membarrier_test_ipi.c b/tools/testing/selftests/membarrier/membarrier_test_ipi.c
new file mode 100644
index 000000000000..49b74844c1f4
--- /dev/null
+++ b/tools/testing/selftests/membarrier/membarrier_test_ipi.c
@@ -0,0 +1,739 @@
+// SPDX-License-Identifier: GPL-2.0
+/*
+ * Check which CPUs the expedited membarrier commands interrupt.
+ *
+ * An expedited command sends an IPI to each CPU which runs a targeted
+ * thread. The IPI is the memory barrier. A test cannot observe a
+ * missing memory barrier reliably. It can observe a missing IPI: the
+ * kernel counts the function call interrupts of each CPU in
+ * /proc/interrupts.
+ *
+ * Each test case pins a "target" which spins in user-space on its own
+ * CPU. A "sender" pinned on another CPU issues the command NR_CMDS
+ * times. The test then compares the count of function call interrupts
+ * of the target CPU before and after.
+ *
+ * Each test case runs in its own process because a membarrier
+ * registration cannot be undone.
+ */
+#define _GNU_SOURCE
+#include <linux/membarrier.h>
+#include <sys/mman.h>
+#include <sys/prctl.h>
+#include <sys/wait.h>
+#include <syscall.h>
+#include <stdatomic.h>
+#include <pthread.h>
+#include <signal.h>
+#include <stdbool.h>
+#include <stdlib.h>
+#include <string.h>
+#include <unistd.h>
+#include <sched.h>
+#include <stdio.h>
+#include <errno.h>
+#include <time.h>
+
+#include "kselftest.h"
+
+#define NR_CMDS		1000
+/*
+ * A target which must be interrupted can lose a few IPIs when another
+ * task preempts it. A target which must not be interrupted can receive
+ * a few unrelated IPIs. The observed counts are either close to
+ * NR_CMDS or close to zero. Keep a wide margin on both sides.
+ */
+#define MIN_HIT		(NR_CMDS / 2)
+#define MAX_MISS	(NR_CMDS / 4)
+
+#define MAX_TARGETS	2
+/* Seconds to wait for the targets of a test case. */
+#define READY_TIMEOUT	10
+#define STACK_SIZE	(64 * 1024)
+
+/*
+ * Exit status of a test case process. Each skip status has its own
+ * reason, which run_case() reports. CASE_FAIL is a failure of
+ * membarrier. CASE_ERROR is a failure of the test setup.
+ */
+enum {
+	CASE_PASS = 0,
+	CASE_FAIL = 1,
+	CASE_SKIP_NO_COUNT = 2,
+	CASE_SKIP_NO_CMD = 3,
+	CASE_ERROR = 4,
+};
+
+struct shared {
+	atomic_int ready;
+	atomic_int stop;
+};
+
+static struct shared *shared;
+static int shared_fd;
+
+/* CPU of the sender and CPUs of the targets. */
+static int sender_cpu, target_cpu[MAX_TARGETS];
+
+static int sys_membarrier(int cmd, unsigned int flags, int cpu_id)
+{
+	return syscall(__NR_membarrier, cmd, flags, cpu_id);
+}
+
+/* Pin the task @pid on @cpu. A @pid of 0 is the calling thread. */
+static void pin_task(pid_t pid, int cpu)
+{
+	cpu_set_t set;
+
+	CPU_ZERO(&set);
+	CPU_SET(cpu, &set);
+	if (sched_setaffinity(pid, sizeof(set), &set)) {
+		perror("sched_setaffinity");
+		_exit(CASE_ERROR);
+	}
+}
+
+static void pin_to_cpu(int cpu)
+{
+	pin_task(0, cpu);
+}
+
+/* Die with the parent. A killed test must leave no spinning target. */
+static void die_with_parent(void)
+{
+	if (prctl(PR_SET_PDEATHSIG, SIGKILL)) {
+		perror("prctl");
+		_exit(CASE_ERROR);
+	}
+}
+
+static void must_join(pthread_t thread)
+{
+	errno = pthread_join(thread, NULL);
+	if (errno) {
+		perror("pthread_join");
+		_exit(CASE_ERROR);
+	}
+}
+
+/* Wait for the child @pid. Its setup failed unless it exits with 0. */
+static void must_wait(pid_t pid)
+{
+	int status;
+
+	if (waitpid(pid, &status, 0) != pid) {
+		perror("waitpid");
+		_exit(CASE_ERROR);
+	}
+	if (!WIFEXITED(status) || WEXITSTATUS(status)) {
+		fprintf(stderr, "# child %d did not exit with 0\n", pid);
+		_exit(CASE_ERROR);
+	}
+}
+
+static void must_register(int cmd)
+{
+	if (sys_membarrier(cmd, 0, 0)) {
+		perror("membarrier registration");
+		_exit(CASE_ERROR);
+	}
+}
+
+/* Announce that this target is in place, then spin in user-space. */
+static void spin(void)
+{
+	die_with_parent();
+	atomic_fetch_add(&shared->ready, 1);
+	while (!atomic_load(&shared->stop))
+		;
+}
+
+/*
+ * Wait until @nr targets spin. Report a setup error when they do not
+ * show up: a target may have died in its setup.
+ */
+static void wait_ready(int nr)
+{
+	time_t start = time(NULL);
+
+	while (atomic_load(&shared->ready) < nr) {
+		if (time(NULL) - start > READY_TIMEOUT) {
+			fprintf(stderr, "# the targets are not ready\n");
+			atomic_store(&shared->stop, 1);
+			_exit(CASE_ERROR);
+		}
+		sched_yield();
+	}
+}
+
+/*
+ * Read the count of function call interrupts of each CPU of @cpus.
+ * Return false when /proc/interrupts has no such count for one of
+ * them. An offline CPU has no count.
+ */
+static bool read_ipi_counts(const int *cpus, int nr_cpus, long *counts)
+{
+	int *col_cpu = NULL, nr_cols = 0, i, c;
+	char *line = NULL, *p, *end;
+	unsigned int seen = 0;
+	size_t len = 0;
+	FILE *f;
+
+	for (c = 0; c < nr_cpus; c++)
+		counts[c] = 0;
+	f = fopen("/proc/interrupts", "r");
+	if (!f)
+		return false;
+	/* The first line names the CPU of each column. */
+	if (getline(&line, &len, f) < 0)
+		goto out;
+	for (p = line; (p = strstr(p, "CPU")); p = end) {
+		int cpu = strtol(p + 3, &end, 10);
+
+		int *new_col_cpu;
+
+		new_col_cpu = realloc(col_cpu, (nr_cols + 1) * sizeof(*col_cpu));
+		if (!new_col_cpu)
+			goto out;
+		col_cpu = new_col_cpu;
+		col_cpu[nr_cols++] = cpu;
+	}
+	while (getline(&line, &len, f) >= 0) {
+		/* x86 uses the "CAL" label. Others describe the line. */
+		if (!strstr(line, "CAL:") && !strstr(line, "unction call"))
+			continue;
+		p = strchr(line, ':');
+		if (!p)
+			continue;
+		p++;
+		for (i = 0; i < nr_cols; i++) {
+			long val = strtol(p, &end, 10);
+
+			if (end == p)
+				break;
+			p = end;
+			for (c = 0; c < nr_cpus; c++)
+				if (cpus[c] == col_cpu[i]) {
+					counts[c] += val;
+					seen |= 1U << c;
+				}
+		}
+	}
+out:
+	free(col_cpu);
+	free(line);
+	fclose(f);
+	return seen == (1U << nr_cpus) - 1;
+}
+
+/*
+ * Issue @cmd NR_CMDS times. Store into @delta the count of function
+ * call interrupts which each target CPU received meanwhile.
+ */
+static int send_cmds(int cmd, unsigned int flags, int cpu_id, long *delta)
+{
+	long before[MAX_TARGETS], after[MAX_TARGETS];
+	int i;
+
+	if (!read_ipi_counts(target_cpu, MAX_TARGETS, before))
+		return CASE_SKIP_NO_COUNT;
+	for (i = 0; i < NR_CMDS; i++) {
+		if (sys_membarrier(cmd, flags, cpu_id)) {
+			perror("membarrier");
+			return CASE_FAIL;
+		}
+	}
+	if (!read_ipi_counts(target_cpu, MAX_TARGETS, after))
+		return CASE_SKIP_NO_COUNT;
+	for (i = 0; i < MAX_TARGETS; i++)
+		delta[i] = after[i] - before[i];
+	return CASE_PASS;
+}
+
+static int expect_hit(int target, long *delta)
+{
+	if (delta[target] >= MIN_HIT)
+		return CASE_PASS;
+	fprintf(stderr, "# CPU %d received %ld IPIs for %d commands\n",
+		target_cpu[target], delta[target], NR_CMDS);
+	return CASE_FAIL;
+}
+
+static int expect_miss(int target, long *delta)
+{
+	if (delta[target] <= MAX_MISS)
+		return CASE_PASS;
+	fprintf(stderr, "# CPU %d received %ld IPIs, expected none\n",
+		target_cpu[target], delta[target]);
+	return CASE_FAIL;
+}
+
+/*
+ * Global expedited test cases.
+ *
+ * The sender is the test case process. The target is a child process.
+ * @setup runs in the child and ends with spin().
+ */
+
+static void *thread_nop(void *arg)
+{
+	return NULL;
+}
+
+static void *thread_spin(void *arg)
+{
+	pin_to_cpu(target_cpu[(long)arg]);
+	spin();
+	return NULL;
+}
+
+static void setup_registered(void)
+{
+	must_register(MEMBARRIER_CMD_REGISTER_GLOBAL_EXPEDITED);
+	spin();
+}
+
+static void setup_unregistered(void)
+{
+	spin();
+}
+
+/*
+ * The exiting thread clears the runqueue membarrier state in
+ * exit_mm(). The scheduler then switches to the main thread, which
+ * has the same mm.
+ */
+static void setup_thread_exit(void)
+{
+	pthread_t thread;
+
+	must_register(MEMBARRIER_CMD_REGISTER_GLOBAL_EXPEDITED);
+	if (pthread_create(&thread, NULL, thread_nop, NULL))
+		_exit(CASE_ERROR);
+	must_join(thread);
+	spin();
+}
+
+/*
+ * Register on another CPU. Meanwhile the first CPU is idle and its
+ * idle thread borrows the mm of this process. Come back to the first
+ * CPU: the scheduler switches from the idle thread to this process.
+ */
+static void setup_lazy_tlb(void)
+{
+	pin_to_cpu(target_cpu[1]);
+	must_register(MEMBARRIER_CMD_REGISTER_GLOBAL_EXPEDITED);
+	pin_to_cpu(target_cpu[0]);
+	spin();
+}
+
+/* Register from a multi-threaded process. Both threads are targets. */
+static void setup_multi_thread(void)
+{
+	pthread_t thread;
+
+	if (pthread_create(&thread, NULL, thread_spin, (void *)1L))
+		_exit(CASE_ERROR);
+	wait_ready(1);
+	must_register(MEMBARRIER_CMD_REGISTER_GLOBAL_EXPEDITED);
+	spin();
+}
+
+/* A forked child inherits the registration. */
+static void setup_fork(void)
+{
+	pid_t pid;
+
+	must_register(MEMBARRIER_CMD_REGISTER_GLOBAL_EXPEDITED);
+	pid = fork();
+	if (pid < 0)
+		_exit(CASE_ERROR);
+	if (!pid) {
+		spin();
+		_exit(0);
+	}
+	must_wait(pid);
+}
+
+/* exec() drops the registration. */
+static void setup_exec(void)
+{
+	char fd_str[16];
+
+	must_register(MEMBARRIER_CMD_REGISTER_GLOBAL_EXPEDITED);
+	snprintf(fd_str, sizeof(fd_str), "%d", shared_fd);
+	execl("/proc/self/exe", "membarrier_test_ipi", "--spin", fd_str, NULL);
+	perror("execl");
+	_exit(CASE_ERROR);
+}
+
+static int test_global(void (*setup)(void), int nr_targets, bool hit)
+{
+	long delta[MAX_TARGETS];
+	int ret, i;
+	pid_t pid;
+
+	if (!(sys_membarrier(MEMBARRIER_CMD_QUERY, 0, 0) &
+	      MEMBARRIER_CMD_GLOBAL_EXPEDITED))
+		return CASE_SKIP_NO_CMD;
+	pid = fork();
+	if (pid < 0)
+		return CASE_ERROR;
+	if (!pid) {
+		die_with_parent();
+		pin_to_cpu(target_cpu[0]);
+		setup();
+		_exit(0);
+	}
+	wait_ready(nr_targets);
+	ret = send_cmds(MEMBARRIER_CMD_GLOBAL_EXPEDITED, 0, 0, delta);
+	atomic_store(&shared->stop, 1);
+	must_wait(pid);
+	for (i = 0; !ret && i < nr_targets; i++)
+		ret = hit ? expect_hit(i, delta) : expect_miss(i, delta);
+	return ret;
+}
+
+static int case_global_registered(void)
+{
+	return test_global(setup_registered, 1, true);
+}
+
+static int case_global_unregistered(void)
+{
+	return test_global(setup_unregistered, 1, false);
+}
+
+static int case_global_thread_exit(void)
+{
+	return test_global(setup_thread_exit, 1, true);
+}
+
+static int case_global_lazy_tlb(void)
+{
+	return test_global(setup_lazy_tlb, 1, true);
+}
+
+static int case_global_multi_thread(void)
+{
+	return test_global(setup_multi_thread, 2, true);
+}
+
+static int case_global_fork(void)
+{
+	return test_global(setup_fork, 1, true);
+}
+
+static int case_global_exec(void)
+{
+	return test_global(setup_exec, 1, false);
+}
+
+/*
+ * Private expedited test cases.
+ *
+ * The sender is the main thread of the test case process. The targets
+ * are threads of the same process, one per target CPU.
+ */
+
+static int start_thread_targets(pthread_t *threads)
+{
+	long i;
+
+	for (i = 0; i < MAX_TARGETS; i++)
+		if (pthread_create(&threads[i], NULL, thread_spin, (void *)i))
+			return CASE_ERROR;
+	wait_ready(MAX_TARGETS);
+	return CASE_PASS;
+}
+
+static void stop_thread_targets(pthread_t *threads)
+{
+	int i;
+
+	atomic_store(&shared->stop, 1);
+	for (i = 0; i < MAX_TARGETS; i++)
+		must_join(threads[i]);
+}
+
+static int test_private(int register_cmd, int cmd)
+{
+	pthread_t threads[MAX_TARGETS];
+	long delta[MAX_TARGETS];
+	int ret, i;
+
+	if (!(sys_membarrier(MEMBARRIER_CMD_QUERY, 0, 0) & cmd))
+		return CASE_SKIP_NO_CMD;
+	if (start_thread_targets(threads))
+		return CASE_ERROR;
+	must_register(register_cmd);
+	ret = send_cmds(cmd, 0, 0, delta);
+	stop_thread_targets(threads);
+	for (i = 0; !ret && i < MAX_TARGETS; i++)
+		ret = expect_hit(i, delta);
+	return ret;
+}
+
+static int case_private(void)
+{
+	return test_private(MEMBARRIER_CMD_REGISTER_PRIVATE_EXPEDITED,
+			    MEMBARRIER_CMD_PRIVATE_EXPEDITED);
+}
+
+static int case_private_sync_core(void)
+{
+	return test_private(MEMBARRIER_CMD_REGISTER_PRIVATE_EXPEDITED_SYNC_CORE,
+			    MEMBARRIER_CMD_PRIVATE_EXPEDITED_SYNC_CORE);
+}
+
+static int case_private_rseq(void)
+{
+	return test_private(MEMBARRIER_CMD_REGISTER_PRIVATE_EXPEDITED_RSEQ,
+			    MEMBARRIER_CMD_PRIVATE_EXPEDITED_RSEQ);
+}
+
+/* MEMBARRIER_CMD_FLAG_CPU interrupts only the CPU it names. */
+static int case_private_rseq_cpu(void)
+{
+	pthread_t threads[MAX_TARGETS];
+	long delta[MAX_TARGETS];
+	int ret;
+
+	if (!(sys_membarrier(MEMBARRIER_CMD_QUERY, 0, 0) &
+	      MEMBARRIER_CMD_PRIVATE_EXPEDITED_RSEQ))
+		return CASE_SKIP_NO_CMD;
+	if (start_thread_targets(threads))
+		return CASE_ERROR;
+	must_register(MEMBARRIER_CMD_REGISTER_PRIVATE_EXPEDITED_RSEQ);
+	ret = send_cmds(MEMBARRIER_CMD_PRIVATE_EXPEDITED_RSEQ,
+			MEMBARRIER_CMD_FLAG_CPU, target_cpu[0], delta);
+	stop_thread_targets(threads);
+	if (!ret)
+		ret = expect_hit(0, delta);
+	if (!ret)
+		ret = expect_miss(1, delta);
+	return ret;
+}
+
+/* A private command does not interrupt another registered process. */
+static int case_private_other_process(void)
+{
+	long delta[MAX_TARGETS];
+	pthread_t thread;
+	pid_t pid;
+	int ret;
+
+	if (!(sys_membarrier(MEMBARRIER_CMD_QUERY, 0, 0) &
+	      MEMBARRIER_CMD_PRIVATE_EXPEDITED))
+		return CASE_SKIP_NO_CMD;
+	pid = fork();
+	if (pid < 0)
+		return CASE_ERROR;
+	if (!pid) {
+		pin_to_cpu(target_cpu[0]);
+		must_register(MEMBARRIER_CMD_REGISTER_PRIVATE_EXPEDITED);
+		spin();
+		_exit(0);
+	}
+	if (pthread_create(&thread, NULL, thread_spin, (void *)1L))
+		return CASE_ERROR;
+	wait_ready(2);
+	must_register(MEMBARRIER_CMD_REGISTER_PRIVATE_EXPEDITED);
+	ret = send_cmds(MEMBARRIER_CMD_PRIVATE_EXPEDITED, 0, 0, delta);
+	atomic_store(&shared->stop, 1);
+	must_join(thread);
+	must_wait(pid);
+	if (!ret)
+		ret = expect_miss(0, delta);
+	if (!ret)
+		ret = expect_hit(1, delta);
+	return ret;
+}
+
+/*
+ * This task shares the TLS of its parent. Its parent pins it, so that
+ * it only spins.
+ */
+static int clone_vm_fn(void *arg)
+{
+	spin();
+	return 0;
+}
+
+/*
+ * A private command covers every task which uses the mm. This includes
+ * a task of another thread group created with CLONE_VM.
+ */
+static int case_private_clone_vm(void)
+{
+	long delta[MAX_TARGETS];
+	char *stack;
+	pid_t pid;
+	int ret;
+
+	if (!(sys_membarrier(MEMBARRIER_CMD_QUERY, 0, 0) &
+	      MEMBARRIER_CMD_PRIVATE_EXPEDITED))
+		return CASE_SKIP_NO_CMD;
+	stack = malloc(STACK_SIZE);
+	if (!stack)
+		return CASE_ERROR;
+	pid = clone(clone_vm_fn, stack + STACK_SIZE, CLONE_VM | SIGCHLD, NULL);
+	if (pid < 0) {
+		free(stack);
+		return CASE_ERROR;
+	}
+	pin_task(pid, target_cpu[0]);
+	wait_ready(1);
+	must_register(MEMBARRIER_CMD_REGISTER_PRIVATE_EXPEDITED);
+	ret = send_cmds(MEMBARRIER_CMD_PRIVATE_EXPEDITED, 0, 0, delta);
+	atomic_store(&shared->stop, 1);
+	must_wait(pid);
+	free(stack);
+	if (!ret)
+		ret = expect_hit(0, delta);
+	return ret;
+}
+
+static const struct test_case {
+	const char *name;
+	int (*fn)(void);
+} test_cases[] = {
+	{ "global expedited reaches a registered process",
+	  case_global_registered },
+	{ "global expedited skips an unregistered process",
+	  case_global_unregistered },
+	{ "global expedited reaches a process after a thread exit",
+	  case_global_thread_exit },
+	{ "global expedited reaches a process registered on another CPU",
+	  case_global_lazy_tlb },
+	{ "global expedited reaches each thread of a process",
+	  case_global_multi_thread },
+	{ "global expedited reaches a forked child",
+	  case_global_fork },
+	{ "global expedited skips a process after exec",
+	  case_global_exec },
+	{ "private expedited reaches each thread",
+	  case_private },
+	{ "private expedited sync core reaches each thread",
+	  case_private_sync_core },
+	{ "private expedited rseq reaches each thread",
+	  case_private_rseq },
+	{ "private expedited rseq with a CPU reaches only that CPU",
+	  case_private_rseq_cpu },
+	{ "private expedited skips another process",
+	  case_private_other_process },
+	{ "private expedited reaches a CLONE_VM task",
+	  case_private_clone_vm },
+};
+
+static void run_case(const struct test_case *tc)
+{
+	int status;
+	pid_t pid;
+
+	atomic_store(&shared->ready, 0);
+	atomic_store(&shared->stop, 0);
+	pid = fork();
+	if (pid < 0)
+		ksft_exit_fail_msg("fork failed\n");
+	if (!pid) {
+		/* The targets inherit this process group. */
+		if (setpgid(0, 0)) {
+			perror("setpgid");
+			_exit(CASE_ERROR);
+		}
+		die_with_parent();
+		pin_to_cpu(sender_cpu);
+		_exit(tc->fn());
+	}
+	if (waitpid(pid, &status, 0) != pid)
+		ksft_exit_fail_msg("waitpid failed\n");
+	/*
+	 * Kill the targets which a failed test case left behind. There
+	 * is usually none: kill() then fails with ESRCH.
+	 */
+	atomic_store(&shared->stop, 1);
+	if (kill(-pid, SIGKILL) && errno != ESRCH)
+		ksft_exit_fail_msg("kill failed\n");
+	if (!WIFEXITED(status))
+		status = CASE_FAIL;
+	else
+		status = WEXITSTATUS(status);
+	switch (status) {
+	case CASE_PASS:
+		ksft_test_result_pass("%s\n", tc->name);
+		break;
+	case CASE_SKIP_NO_COUNT:
+		ksft_test_result_skip("%s: Cannot read the interrupt counts\n",
+				      tc->name);
+		break;
+	case CASE_SKIP_NO_CMD:
+		ksft_test_result_skip("%s: Command not available\n",
+				      tc->name);
+		break;
+	case CASE_ERROR:
+		ksft_test_result_error("%s: Test setup failed\n", tc->name);
+		break;
+	default:
+		ksft_test_result_fail("%s\n", tc->name);
+	}
+}
+
+/* Pick the sender CPU and the target CPUs among the allowed CPUs. */
+static bool pick_cpus(void)
+{
+	int cpu, nr = 0, *dst[] = {
+		&sender_cpu, &target_cpu[0], &target_cpu[1],
+	};
+	cpu_set_t set;
+
+	if (sched_getaffinity(0, sizeof(set), &set))
+		return false;
+	for (cpu = 0; cpu < CPU_SETSIZE && nr < ARRAY_SIZE(dst); cpu++)
+		if (CPU_ISSET(cpu, &set))
+			*dst[nr++] = cpu;
+	return nr == ARRAY_SIZE(dst);
+}
+
+static struct shared *map_shared(int fd)
+{
+	void *p;
+
+	p = mmap(NULL, sizeof(struct shared), PROT_READ | PROT_WRITE,
+		 MAP_SHARED, fd, 0);
+	return p == MAP_FAILED ? NULL : p;
+}
+
+int main(int argc, char **argv)
+{
+	long counts[MAX_TARGETS];
+	int i;
+
+	/* Target of the exec test case. See setup_exec(). */
+	if (argc == 3 && !strcmp(argv[1], "--spin")) {
+		shared = map_shared(atoi(argv[2]));
+		if (!shared)
+			return CASE_ERROR;
+		spin();
+		return 0;
+	}
+
+	ksft_print_header();
+	if (sys_membarrier(MEMBARRIER_CMD_QUERY, 0, 0) < 0)
+		ksft_exit_skip("sys membarrier (CONFIG_MEMBARRIER) is disabled.\n");
+	if (!pick_cpus())
+		ksft_exit_skip("This test needs 3 allowed CPUs.\n");
+	if (!read_ipi_counts(target_cpu, MAX_TARGETS, counts))
+		ksft_exit_skip("No function call interrupt count in /proc/interrupts.\n");
+	shared_fd = memfd_create("membarrier_test_ipi", 0);
+	if (shared_fd < 0 || ftruncate(shared_fd, sizeof(struct shared)))
+		ksft_exit_fail_msg("memfd_create failed\n");
+	shared = map_shared(shared_fd);
+	if (!shared)
+		ksft_exit_fail_msg("mmap failed\n");
+
+	ksft_set_plan(ARRAY_SIZE(test_cases));
+	for (i = 0; i < ARRAY_SIZE(test_cases); i++)
+		run_case(&test_cases[i]);
+	ksft_finished();
+}
-- 
2.47.3


_______________________________________________
linux-riscv mailing list
linux-riscv@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-riscv

[RFC PATCH 07/16] selftests/membarrier: Test command arguments and registration lifetime

[Mathieu Desnoyers]

The existing tests issue each command once with valid arguments, in
one fixed order of registrations. They miss the following:

 - the query mask is not compared with the commands which the kernel
   accepts. Commit 809232619f5b ("sched/membarrier: Fix
   membarrier-rseq fence command missing from query bitmask") fixed
   such a mismatch,
 - only MEMBARRIER_CMD_QUERY is tested with an invalid flag,
 - only the stress test issues the rseq command. Its registration,
   its -EPERM result and its CPU argument are not checked,
 - each registration is not tested alone. Commit fc0d77387cb5
   ("sched/membarrier: Fix private expedited registration check")
   fixed an interaction between two registrations,
 - the registrations are not tested across fork, exec and CLONE_VM.

Add membarrier_test_cmd for those. It uses the kselftest harness,
which runs each test in its own process. A registration cannot be
undone, so each test needs a new process.

The sync core and rseq registration tests fail without commit
"sched/membarrier: Report only completed registrations".

Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
---
 tools/testing/selftests/membarrier/.gitignore |   1 +
 tools/testing/selftests/membarrier/Makefile   |   1 +
 .../membarrier/membarrier_test_cmd.c          | 352 ++++++++++++++++++
 3 files changed, 354 insertions(+)
 create mode 100644 tools/testing/selftests/membarrier/membarrier_test_cmd.c

diff --git a/tools/testing/selftests/membarrier/.gitignore b/tools/testing/selftests/membarrier/.gitignore
index 58969a5e4613..45b995b884a6 100644
--- a/tools/testing/selftests/membarrier/.gitignore
+++ b/tools/testing/selftests/membarrier/.gitignore
@@ -3,3 +3,4 @@ membarrier_test_multi_thread
 membarrier_test_single_thread
 membarrier_rseq_stress
 membarrier_test_ipi
+membarrier_test_cmd
diff --git a/tools/testing/selftests/membarrier/Makefile b/tools/testing/selftests/membarrier/Makefile
index ee5b9deb808b..c7fa0573e978 100644
--- a/tools/testing/selftests/membarrier/Makefile
+++ b/tools/testing/selftests/membarrier/Makefile
@@ -5,6 +5,7 @@ LDLIBS += -lpthread
 TEST_GEN_PROGS := membarrier_test_single_thread \
 		membarrier_test_multi_thread \
 		membarrier_test_ipi \
+		membarrier_test_cmd \
 		membarrier_rseq_stress
 
 LOCAL_HDRS += membarrier_test_impl.h
diff --git a/tools/testing/selftests/membarrier/membarrier_test_cmd.c b/tools/testing/selftests/membarrier/membarrier_test_cmd.c
new file mode 100644
index 000000000000..5982861f77b6
--- /dev/null
+++ b/tools/testing/selftests/membarrier/membarrier_test_cmd.c
@@ -0,0 +1,352 @@
+// SPDX-License-Identifier: GPL-2.0
+/*
+ * Check the membarrier command arguments and the lifetime of the
+ * registrations.
+ *
+ * The test harness runs each test in its own process. This matters
+ * because a membarrier registration cannot be undone.
+ */
+#define _GNU_SOURCE
+#include <linux/membarrier.h>
+#include <sys/wait.h>
+#include <syscall.h>
+#include <limits.h>
+#include <pthread.h>
+#include <signal.h>
+#include <stdlib.h>
+#include <string.h>
+#include <unistd.h>
+#include <sched.h>
+#include <errno.h>
+
+#include "kselftest_harness.h"
+
+#define STACK_SIZE	(64 * 1024)
+
+#define REGISTER_CMDS						\
+	(MEMBARRIER_CMD_REGISTER_GLOBAL_EXPEDITED |		\
+	 MEMBARRIER_CMD_REGISTER_PRIVATE_EXPEDITED |		\
+	 MEMBARRIER_CMD_REGISTER_PRIVATE_EXPEDITED_SYNC_CORE |	\
+	 MEMBARRIER_CMD_REGISTER_PRIVATE_EXPEDITED_RSEQ)
+
+/* Pairs of registration command and command which needs it. */
+static const struct {
+	int register_cmd;
+	int cmd;
+} private_cmds[] = {
+	{ MEMBARRIER_CMD_REGISTER_PRIVATE_EXPEDITED,
+	  MEMBARRIER_CMD_PRIVATE_EXPEDITED },
+	{ MEMBARRIER_CMD_REGISTER_PRIVATE_EXPEDITED_SYNC_CORE,
+	  MEMBARRIER_CMD_PRIVATE_EXPEDITED_SYNC_CORE },
+	{ MEMBARRIER_CMD_REGISTER_PRIVATE_EXPEDITED_RSEQ,
+	  MEMBARRIER_CMD_PRIVATE_EXPEDITED_RSEQ },
+};
+
+static int sys_membarrier(int cmd, unsigned int flags, int cpu_id)
+{
+	return syscall(__NR_membarrier, cmd, flags, cpu_id);
+}
+
+static int query(void)
+{
+	return sys_membarrier(MEMBARRIER_CMD_QUERY, 0, 0);
+}
+
+static int get_registrations(void)
+{
+	return sys_membarrier(MEMBARRIER_CMD_GET_REGISTRATIONS, 0, 0);
+}
+
+/* Each command is either in the query mask or returns -EINVAL. */
+TEST(query_matches_commands)
+{
+	int mask = query(), bit, ret;
+
+	ASSERT_GE(mask, 0);
+	for (bit = 0; bit < 32; bit++) {
+		int cmd = (int)(1U << bit);
+
+		errno = 0;
+		ret = sys_membarrier(cmd, 0, 0);
+		if (mask & cmd) {
+			EXPECT_NE(errno, EINVAL) {
+				TH_LOG("command 0x%x is in the query mask", cmd);
+			}
+		} else {
+			EXPECT_EQ(ret, -1);
+			EXPECT_EQ(errno, EINVAL) {
+				TH_LOG("command 0x%x is not in the query mask", cmd);
+			}
+		}
+	}
+}
+
+/* A command is a single bit. */
+TEST(invalid_commands)
+{
+	static const int cmds[] = {
+		-1, INT_MAX,
+		MEMBARRIER_CMD_GLOBAL | MEMBARRIER_CMD_GLOBAL_EXPEDITED,
+		MEMBARRIER_CMD_PRIVATE_EXPEDITED |
+			MEMBARRIER_CMD_REGISTER_PRIVATE_EXPEDITED,
+	};
+	int i;
+
+	for (i = 0; i < ARRAY_SIZE(cmds); i++) {
+		EXPECT_EQ(sys_membarrier(cmds[i], 0, 0), -1);
+		EXPECT_EQ(errno, EINVAL);
+	}
+}
+
+/* Only the rseq command accepts a flag: MEMBARRIER_CMD_FLAG_CPU. */
+TEST(invalid_flags)
+{
+	static const unsigned int flags[] = {
+		MEMBARRIER_CMD_FLAG_CPU, 1U << 1, 1U << 31, ~0U,
+		MEMBARRIER_CMD_FLAG_CPU | (1U << 1),
+	};
+	int mask = query(), bit, i;
+
+	ASSERT_GE(mask, 0);
+	/* Bit -1 stands for MEMBARRIER_CMD_QUERY, which is 0. */
+	for (bit = -1; bit < 32; bit++) {
+		int cmd = bit < 0 ? MEMBARRIER_CMD_QUERY : (int)(1U << bit);
+
+		if (cmd && !(mask & cmd))
+			continue;
+		for (i = 0; i < ARRAY_SIZE(flags); i++) {
+			if (cmd == MEMBARRIER_CMD_PRIVATE_EXPEDITED_RSEQ &&
+			    flags[i] == MEMBARRIER_CMD_FLAG_CPU)
+				continue;
+			EXPECT_EQ(sys_membarrier(cmd, flags[i], 0), -1);
+			EXPECT_EQ(errno, EINVAL) {
+				TH_LOG("command 0x%x flags 0x%x", cmd, flags[i]);
+			}
+		}
+	}
+}
+
+/* A private command needs its own registration. */
+TEST(private_needs_registration)
+{
+	int mask = query(), registered = get_registrations(), i;
+
+	ASSERT_GE(mask, 0);
+	ASSERT_GE(registered, 0);
+	for (i = 0; i < ARRAY_SIZE(private_cmds); i++) {
+		if (!(mask & private_cmds[i].cmd))
+			continue;
+		/* The C library may have registered already. */
+		if (registered & private_cmds[i].register_cmd)
+			continue;
+		EXPECT_EQ(sys_membarrier(private_cmds[i].cmd, 0, 0), -1);
+		EXPECT_EQ(errno, EPERM);
+	}
+	if ((mask & MEMBARRIER_CMD_PRIVATE_EXPEDITED_RSEQ) &&
+	    !(registered & MEMBARRIER_CMD_REGISTER_PRIVATE_EXPEDITED_RSEQ)) {
+		EXPECT_EQ(sys_membarrier(MEMBARRIER_CMD_PRIVATE_EXPEDITED_RSEQ,
+					 MEMBARRIER_CMD_FLAG_CPU, 0), -1);
+		EXPECT_EQ(errno, EPERM);
+	}
+}
+
+/*
+ * Issue one registration command. It must add its own bit to the
+ * registrations, and no other bit. It must allow its own command, and
+ * no other command.
+ */
+static void register_one(struct __test_metadata *_metadata, int register_cmd)
+{
+	int mask = query(), before = get_registrations(), i;
+
+	ASSERT_GE(mask, 0);
+	ASSERT_GE(before, 0);
+	if (!(mask & register_cmd))
+		SKIP(return, "command 0x%x is not available", register_cmd);
+	ASSERT_EQ(sys_membarrier(register_cmd, 0, 0), 0);
+	EXPECT_EQ(get_registrations(), before | register_cmd);
+	/* A second registration succeeds and changes nothing. */
+	ASSERT_EQ(sys_membarrier(register_cmd, 0, 0), 0);
+	EXPECT_EQ(get_registrations(), before | register_cmd);
+
+	for (i = 0; i < ARRAY_SIZE(private_cmds); i++) {
+		if (!(mask & private_cmds[i].cmd))
+			continue;
+		if ((before | register_cmd) & private_cmds[i].register_cmd) {
+			EXPECT_EQ(sys_membarrier(private_cmds[i].cmd, 0, 0), 0);
+		} else {
+			EXPECT_EQ(sys_membarrier(private_cmds[i].cmd, 0, 0), -1);
+			EXPECT_EQ(errno, EPERM);
+		}
+	}
+}
+
+TEST(register_global_expedited)
+{
+	register_one(_metadata, MEMBARRIER_CMD_REGISTER_GLOBAL_EXPEDITED);
+}
+
+TEST(register_private_expedited)
+{
+	register_one(_metadata, MEMBARRIER_CMD_REGISTER_PRIVATE_EXPEDITED);
+}
+
+TEST(register_private_expedited_sync_core)
+{
+	register_one(_metadata,
+		     MEMBARRIER_CMD_REGISTER_PRIVATE_EXPEDITED_SYNC_CORE);
+}
+
+TEST(register_private_expedited_rseq)
+{
+	register_one(_metadata, MEMBARRIER_CMD_REGISTER_PRIVATE_EXPEDITED_RSEQ);
+}
+
+/* Register every command, one at a time. Return the registrations. */
+static int register_all(struct __test_metadata *_metadata)
+{
+	int mask = query(), expected = get_registrations(), bit;
+
+	for (bit = 0; bit < 31; bit++) {
+		int cmd = 1 << bit;
+
+		if (!(cmd & REGISTER_CMDS & mask))
+			continue;
+		ASSERT_EQ(sys_membarrier(cmd, 0, 0), 0);
+		expected |= cmd;
+		EXPECT_EQ(get_registrations(), expected);
+	}
+	return expected;
+}
+
+TEST(register_all_commands)
+{
+	register_all(_metadata);
+}
+
+static void *thread_pause(void *arg)
+{
+	for (;;)
+		pause();
+	return NULL;
+}
+
+/*
+ * The rseq command accepts any CPU number. It ignores a CPU which does
+ * not exist. A negative CPU number selects all CPUs.
+ *
+ * The kernel only looks at the CPU number of a multi-threaded process,
+ * so keep a second thread alive.
+ */
+TEST(rseq_cpu_argument)
+{
+	static const int cpus[] = { 0, INT_MAX, -1, INT_MIN };
+	pthread_t thread;
+	int i;
+
+	if (!(query() & MEMBARRIER_CMD_PRIVATE_EXPEDITED_RSEQ))
+		SKIP(return, "the rseq command is not available");
+	ASSERT_EQ(pthread_create(&thread, NULL, thread_pause, NULL), 0);
+	ASSERT_EQ(sys_membarrier(MEMBARRIER_CMD_REGISTER_PRIVATE_EXPEDITED_RSEQ,
+				 0, 0), 0);
+	EXPECT_EQ(sys_membarrier(MEMBARRIER_CMD_PRIVATE_EXPEDITED_RSEQ,
+				 MEMBARRIER_CMD_FLAG_CPU, sched_getcpu()), 0);
+	for (i = 0; i < ARRAY_SIZE(cpus); i++) {
+		EXPECT_EQ(sys_membarrier(MEMBARRIER_CMD_PRIVATE_EXPEDITED_RSEQ,
+					 MEMBARRIER_CMD_FLAG_CPU, cpus[i]), 0) {
+			TH_LOG("cpu_id %d", cpus[i]);
+		}
+	}
+	/* Without the flag, the CPU argument is ignored. */
+	EXPECT_EQ(sys_membarrier(MEMBARRIER_CMD_PRIVATE_EXPEDITED_RSEQ,
+				 0, INT_MAX), 0);
+}
+
+/* A forked child inherits the registrations. */
+TEST(fork_inherits_registrations)
+{
+	int expected = register_all(_metadata), status;
+	pid_t pid;
+
+	pid = fork();
+	ASSERT_GE(pid, 0);
+	if (!pid)
+		_exit(get_registrations() != expected);
+	ASSERT_EQ(waitpid(pid, &status, 0), pid);
+	EXPECT_TRUE(WIFEXITED(status));
+	EXPECT_EQ(WEXITSTATUS(status), 0);
+}
+
+/*
+ * exec() drops the registrations. The new program image starts with
+ * the registrations which this test program had when it started.
+ */
+static int initial_registrations;
+
+TEST(exec_drops_registrations)
+{
+	char expected[16];
+	int status;
+	pid_t pid;
+
+	register_all(_metadata);
+	snprintf(expected, sizeof(expected), "%d", initial_registrations);
+	pid = fork();
+	ASSERT_GE(pid, 0);
+	if (!pid) {
+		execl("/proc/self/exe", "membarrier_test_cmd",
+		      "--expect-registrations", expected, NULL);
+		_exit(2);
+	}
+	ASSERT_EQ(waitpid(pid, &status, 0), pid);
+	EXPECT_TRUE(WIFEXITED(status));
+	if (WEXITSTATUS(status) == 2)
+		SKIP(return, "cannot exec /proc/self/exe");
+	EXPECT_EQ(WEXITSTATUS(status), 0);
+}
+
+static int clone_vm_fn(void *arg)
+{
+	int cmd = *(int *)arg;
+
+	return sys_membarrier(cmd, 0, 0) ? 1 : 0;
+}
+
+/*
+ * The registrations belong to the mm. A task of another thread group
+ * which shares the mm (CLONE_VM without CLONE_THREAD) registers for
+ * both thread groups.
+ */
+TEST(clone_vm_shares_registrations)
+{
+	int cmd = MEMBARRIER_CMD_REGISTER_PRIVATE_EXPEDITED;
+	int before = get_registrations(), status;
+	char *stack;
+	pid_t pid;
+
+	ASSERT_GE(before, 0);
+	stack = malloc(STACK_SIZE);
+	ASSERT_NE(stack, NULL);
+	pid = clone(clone_vm_fn, stack + STACK_SIZE, CLONE_VM | SIGCHLD, &cmd);
+	ASSERT_GE(pid, 0);
+	ASSERT_EQ(waitpid(pid, &status, 0), pid);
+	EXPECT_TRUE(WIFEXITED(status));
+	EXPECT_EQ(WEXITSTATUS(status), 0);
+	EXPECT_EQ(get_registrations(), before | cmd);
+	EXPECT_EQ(sys_membarrier(MEMBARRIER_CMD_PRIVATE_EXPEDITED, 0, 0), 0);
+	free(stack);
+}
+
+int main(int argc, char **argv)
+{
+	/* New program image of exec_drops_registrations. */
+	if (argc == 3 && !strcmp(argv[1], "--expect-registrations"))
+		return get_registrations() != atoi(argv[2]);
+
+	if (query() < 0)
+		ksft_exit_skip("sys membarrier (CONFIG_MEMBARRIER) is disabled.\n");
+	initial_registrations = get_registrations();
+	if (initial_registrations < 0)
+		ksft_exit_skip("MEMBARRIER_CMD_GET_REGISTRATIONS is not available.\n");
+	return test_harness_run(argc, argv);
+}
-- 
2.47.3

[RFC PATCH 07/16] selftests/membarrier: Test command arguments and registration lifetime

[Mathieu Desnoyers]

The existing tests issue each command once with valid arguments, in
one fixed order of registrations. They miss the following:

 - the query mask is not compared with the commands which the kernel
   accepts. Commit 809232619f5b ("sched/membarrier: Fix
   membarrier-rseq fence command missing from query bitmask") fixed
   such a mismatch,
 - only MEMBARRIER_CMD_QUERY is tested with an invalid flag,
 - only the stress test issues the rseq command. Its registration,
   its -EPERM result and its CPU argument are not checked,
 - each registration is not tested alone. Commit fc0d77387cb5
   ("sched/membarrier: Fix private expedited registration check")
   fixed an interaction between two registrations,
 - the registrations are not tested across fork, exec and CLONE_VM.

Add membarrier_test_cmd for those. It uses the kselftest harness,
which runs each test in its own process. A registration cannot be
undone, so each test needs a new process.

The sync core and rseq registration tests fail without commit
"sched/membarrier: Report only completed registrations".

Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
---
 tools/testing/selftests/membarrier/.gitignore |   1 +
 tools/testing/selftests/membarrier/Makefile   |   1 +
 .../membarrier/membarrier_test_cmd.c          | 352 ++++++++++++++++++
 3 files changed, 354 insertions(+)
 create mode 100644 tools/testing/selftests/membarrier/membarrier_test_cmd.c

diff --git a/tools/testing/selftests/membarrier/.gitignore b/tools/testing/selftests/membarrier/.gitignore
index 58969a5e4613..45b995b884a6 100644
--- a/tools/testing/selftests/membarrier/.gitignore
+++ b/tools/testing/selftests/membarrier/.gitignore
@@ -3,3 +3,4 @@ membarrier_test_multi_thread
 membarrier_test_single_thread
 membarrier_rseq_stress
 membarrier_test_ipi
+membarrier_test_cmd
diff --git a/tools/testing/selftests/membarrier/Makefile b/tools/testing/selftests/membarrier/Makefile
index ee5b9deb808b..c7fa0573e978 100644
--- a/tools/testing/selftests/membarrier/Makefile
+++ b/tools/testing/selftests/membarrier/Makefile
@@ -5,6 +5,7 @@ LDLIBS += -lpthread
 TEST_GEN_PROGS := membarrier_test_single_thread \
 		membarrier_test_multi_thread \
 		membarrier_test_ipi \
+		membarrier_test_cmd \
 		membarrier_rseq_stress
 
 LOCAL_HDRS += membarrier_test_impl.h
diff --git a/tools/testing/selftests/membarrier/membarrier_test_cmd.c b/tools/testing/selftests/membarrier/membarrier_test_cmd.c
new file mode 100644
index 000000000000..5982861f77b6
--- /dev/null
+++ b/tools/testing/selftests/membarrier/membarrier_test_cmd.c
@@ -0,0 +1,352 @@
+// SPDX-License-Identifier: GPL-2.0
+/*
+ * Check the membarrier command arguments and the lifetime of the
+ * registrations.
+ *
+ * The test harness runs each test in its own process. This matters
+ * because a membarrier registration cannot be undone.
+ */
+#define _GNU_SOURCE
+#include <linux/membarrier.h>
+#include <sys/wait.h>
+#include <syscall.h>
+#include <limits.h>
+#include <pthread.h>
+#include <signal.h>
+#include <stdlib.h>
+#include <string.h>
+#include <unistd.h>
+#include <sched.h>
+#include <errno.h>
+
+#include "kselftest_harness.h"
+
+#define STACK_SIZE	(64 * 1024)
+
+#define REGISTER_CMDS						\
+	(MEMBARRIER_CMD_REGISTER_GLOBAL_EXPEDITED |		\
+	 MEMBARRIER_CMD_REGISTER_PRIVATE_EXPEDITED |		\
+	 MEMBARRIER_CMD_REGISTER_PRIVATE_EXPEDITED_SYNC_CORE |	\
+	 MEMBARRIER_CMD_REGISTER_PRIVATE_EXPEDITED_RSEQ)
+
+/* Pairs of registration command and command which needs it. */
+static const struct {
+	int register_cmd;
+	int cmd;
+} private_cmds[] = {
+	{ MEMBARRIER_CMD_REGISTER_PRIVATE_EXPEDITED,
+	  MEMBARRIER_CMD_PRIVATE_EXPEDITED },
+	{ MEMBARRIER_CMD_REGISTER_PRIVATE_EXPEDITED_SYNC_CORE,
+	  MEMBARRIER_CMD_PRIVATE_EXPEDITED_SYNC_CORE },
+	{ MEMBARRIER_CMD_REGISTER_PRIVATE_EXPEDITED_RSEQ,
+	  MEMBARRIER_CMD_PRIVATE_EXPEDITED_RSEQ },
+};
+
+static int sys_membarrier(int cmd, unsigned int flags, int cpu_id)
+{
+	return syscall(__NR_membarrier, cmd, flags, cpu_id);
+}
+
+static int query(void)
+{
+	return sys_membarrier(MEMBARRIER_CMD_QUERY, 0, 0);
+}
+
+static int get_registrations(void)
+{
+	return sys_membarrier(MEMBARRIER_CMD_GET_REGISTRATIONS, 0, 0);
+}
+
+/* Each command is either in the query mask or returns -EINVAL. */
+TEST(query_matches_commands)
+{
+	int mask = query(), bit, ret;
+
+	ASSERT_GE(mask, 0);
+	for (bit = 0; bit < 32; bit++) {
+		int cmd = (int)(1U << bit);
+
+		errno = 0;
+		ret = sys_membarrier(cmd, 0, 0);
+		if (mask & cmd) {
+			EXPECT_NE(errno, EINVAL) {
+				TH_LOG("command 0x%x is in the query mask", cmd);
+			}
+		} else {
+			EXPECT_EQ(ret, -1);
+			EXPECT_EQ(errno, EINVAL) {
+				TH_LOG("command 0x%x is not in the query mask", cmd);
+			}
+		}
+	}
+}
+
+/* A command is a single bit. */
+TEST(invalid_commands)
+{
+	static const int cmds[] = {
+		-1, INT_MAX,
+		MEMBARRIER_CMD_GLOBAL | MEMBARRIER_CMD_GLOBAL_EXPEDITED,
+		MEMBARRIER_CMD_PRIVATE_EXPEDITED |
+			MEMBARRIER_CMD_REGISTER_PRIVATE_EXPEDITED,
+	};
+	int i;
+
+	for (i = 0; i < ARRAY_SIZE(cmds); i++) {
+		EXPECT_EQ(sys_membarrier(cmds[i], 0, 0), -1);
+		EXPECT_EQ(errno, EINVAL);
+	}
+}
+
+/* Only the rseq command accepts a flag: MEMBARRIER_CMD_FLAG_CPU. */
+TEST(invalid_flags)
+{
+	static const unsigned int flags[] = {
+		MEMBARRIER_CMD_FLAG_CPU, 1U << 1, 1U << 31, ~0U,
+		MEMBARRIER_CMD_FLAG_CPU | (1U << 1),
+	};
+	int mask = query(), bit, i;
+
+	ASSERT_GE(mask, 0);
+	/* Bit -1 stands for MEMBARRIER_CMD_QUERY, which is 0. */
+	for (bit = -1; bit < 32; bit++) {
+		int cmd = bit < 0 ? MEMBARRIER_CMD_QUERY : (int)(1U << bit);
+
+		if (cmd && !(mask & cmd))
+			continue;
+		for (i = 0; i < ARRAY_SIZE(flags); i++) {
+			if (cmd == MEMBARRIER_CMD_PRIVATE_EXPEDITED_RSEQ &&
+			    flags[i] == MEMBARRIER_CMD_FLAG_CPU)
+				continue;
+			EXPECT_EQ(sys_membarrier(cmd, flags[i], 0), -1);
+			EXPECT_EQ(errno, EINVAL) {
+				TH_LOG("command 0x%x flags 0x%x", cmd, flags[i]);
+			}
+		}
+	}
+}
+
+/* A private command needs its own registration. */
+TEST(private_needs_registration)
+{
+	int mask = query(), registered = get_registrations(), i;
+
+	ASSERT_GE(mask, 0);
+	ASSERT_GE(registered, 0);
+	for (i = 0; i < ARRAY_SIZE(private_cmds); i++) {
+		if (!(mask & private_cmds[i].cmd))
+			continue;
+		/* The C library may have registered already. */
+		if (registered & private_cmds[i].register_cmd)
+			continue;
+		EXPECT_EQ(sys_membarrier(private_cmds[i].cmd, 0, 0), -1);
+		EXPECT_EQ(errno, EPERM);
+	}
+	if ((mask & MEMBARRIER_CMD_PRIVATE_EXPEDITED_RSEQ) &&
+	    !(registered & MEMBARRIER_CMD_REGISTER_PRIVATE_EXPEDITED_RSEQ)) {
+		EXPECT_EQ(sys_membarrier(MEMBARRIER_CMD_PRIVATE_EXPEDITED_RSEQ,
+					 MEMBARRIER_CMD_FLAG_CPU, 0), -1);
+		EXPECT_EQ(errno, EPERM);
+	}
+}
+
+/*
+ * Issue one registration command. It must add its own bit to the
+ * registrations, and no other bit. It must allow its own command, and
+ * no other command.
+ */
+static void register_one(struct __test_metadata *_metadata, int register_cmd)
+{
+	int mask = query(), before = get_registrations(), i;
+
+	ASSERT_GE(mask, 0);
+	ASSERT_GE(before, 0);
+	if (!(mask & register_cmd))
+		SKIP(return, "command 0x%x is not available", register_cmd);
+	ASSERT_EQ(sys_membarrier(register_cmd, 0, 0), 0);
+	EXPECT_EQ(get_registrations(), before | register_cmd);
+	/* A second registration succeeds and changes nothing. */
+	ASSERT_EQ(sys_membarrier(register_cmd, 0, 0), 0);
+	EXPECT_EQ(get_registrations(), before | register_cmd);
+
+	for (i = 0; i < ARRAY_SIZE(private_cmds); i++) {
+		if (!(mask & private_cmds[i].cmd))
+			continue;
+		if ((before | register_cmd) & private_cmds[i].register_cmd) {
+			EXPECT_EQ(sys_membarrier(private_cmds[i].cmd, 0, 0), 0);
+		} else {
+			EXPECT_EQ(sys_membarrier(private_cmds[i].cmd, 0, 0), -1);
+			EXPECT_EQ(errno, EPERM);
+		}
+	}
+}
+
+TEST(register_global_expedited)
+{
+	register_one(_metadata, MEMBARRIER_CMD_REGISTER_GLOBAL_EXPEDITED);
+}
+
+TEST(register_private_expedited)
+{
+	register_one(_metadata, MEMBARRIER_CMD_REGISTER_PRIVATE_EXPEDITED);
+}
+
+TEST(register_private_expedited_sync_core)
+{
+	register_one(_metadata,
+		     MEMBARRIER_CMD_REGISTER_PRIVATE_EXPEDITED_SYNC_CORE);
+}
+
+TEST(register_private_expedited_rseq)
+{
+	register_one(_metadata, MEMBARRIER_CMD_REGISTER_PRIVATE_EXPEDITED_RSEQ);
+}
+
+/* Register every command, one at a time. Return the registrations. */
+static int register_all(struct __test_metadata *_metadata)
+{
+	int mask = query(), expected = get_registrations(), bit;
+
+	for (bit = 0; bit < 31; bit++) {
+		int cmd = 1 << bit;
+
+		if (!(cmd & REGISTER_CMDS & mask))
+			continue;
+		ASSERT_EQ(sys_membarrier(cmd, 0, 0), 0);
+		expected |= cmd;
+		EXPECT_EQ(get_registrations(), expected);
+	}
+	return expected;
+}
+
+TEST(register_all_commands)
+{
+	register_all(_metadata);
+}
+
+static void *thread_pause(void *arg)
+{
+	for (;;)
+		pause();
+	return NULL;
+}
+
+/*
+ * The rseq command accepts any CPU number. It ignores a CPU which does
+ * not exist. A negative CPU number selects all CPUs.
+ *
+ * The kernel only looks at the CPU number of a multi-threaded process,
+ * so keep a second thread alive.
+ */
+TEST(rseq_cpu_argument)
+{
+	static const int cpus[] = { 0, INT_MAX, -1, INT_MIN };
+	pthread_t thread;
+	int i;
+
+	if (!(query() & MEMBARRIER_CMD_PRIVATE_EXPEDITED_RSEQ))
+		SKIP(return, "the rseq command is not available");
+	ASSERT_EQ(pthread_create(&thread, NULL, thread_pause, NULL), 0);
+	ASSERT_EQ(sys_membarrier(MEMBARRIER_CMD_REGISTER_PRIVATE_EXPEDITED_RSEQ,
+				 0, 0), 0);
+	EXPECT_EQ(sys_membarrier(MEMBARRIER_CMD_PRIVATE_EXPEDITED_RSEQ,
+				 MEMBARRIER_CMD_FLAG_CPU, sched_getcpu()), 0);
+	for (i = 0; i < ARRAY_SIZE(cpus); i++) {
+		EXPECT_EQ(sys_membarrier(MEMBARRIER_CMD_PRIVATE_EXPEDITED_RSEQ,
+					 MEMBARRIER_CMD_FLAG_CPU, cpus[i]), 0) {
+			TH_LOG("cpu_id %d", cpus[i]);
+		}
+	}
+	/* Without the flag, the CPU argument is ignored. */
+	EXPECT_EQ(sys_membarrier(MEMBARRIER_CMD_PRIVATE_EXPEDITED_RSEQ,
+				 0, INT_MAX), 0);
+}
+
+/* A forked child inherits the registrations. */
+TEST(fork_inherits_registrations)
+{
+	int expected = register_all(_metadata), status;
+	pid_t pid;
+
+	pid = fork();
+	ASSERT_GE(pid, 0);
+	if (!pid)
+		_exit(get_registrations() != expected);
+	ASSERT_EQ(waitpid(pid, &status, 0), pid);
+	EXPECT_TRUE(WIFEXITED(status));
+	EXPECT_EQ(WEXITSTATUS(status), 0);
+}
+
+/*
+ * exec() drops the registrations. The new program image starts with
+ * the registrations which this test program had when it started.
+ */
+static int initial_registrations;
+
+TEST(exec_drops_registrations)
+{
+	char expected[16];
+	int status;
+	pid_t pid;
+
+	register_all(_metadata);
+	snprintf(expected, sizeof(expected), "%d", initial_registrations);
+	pid = fork();
+	ASSERT_GE(pid, 0);
+	if (!pid) {
+		execl("/proc/self/exe", "membarrier_test_cmd",
+		      "--expect-registrations", expected, NULL);
+		_exit(2);
+	}
+	ASSERT_EQ(waitpid(pid, &status, 0), pid);
+	EXPECT_TRUE(WIFEXITED(status));
+	if (WEXITSTATUS(status) == 2)
+		SKIP(return, "cannot exec /proc/self/exe");
+	EXPECT_EQ(WEXITSTATUS(status), 0);
+}
+
+static int clone_vm_fn(void *arg)
+{
+	int cmd = *(int *)arg;
+
+	return sys_membarrier(cmd, 0, 0) ? 1 : 0;
+}
+
+/*
+ * The registrations belong to the mm. A task of another thread group
+ * which shares the mm (CLONE_VM without CLONE_THREAD) registers for
+ * both thread groups.
+ */
+TEST(clone_vm_shares_registrations)
+{
+	int cmd = MEMBARRIER_CMD_REGISTER_PRIVATE_EXPEDITED;
+	int before = get_registrations(), status;
+	char *stack;
+	pid_t pid;
+
+	ASSERT_GE(before, 0);
+	stack = malloc(STACK_SIZE);
+	ASSERT_NE(stack, NULL);
+	pid = clone(clone_vm_fn, stack + STACK_SIZE, CLONE_VM | SIGCHLD, &cmd);
+	ASSERT_GE(pid, 0);
+	ASSERT_EQ(waitpid(pid, &status, 0), pid);
+	EXPECT_TRUE(WIFEXITED(status));
+	EXPECT_EQ(WEXITSTATUS(status), 0);
+	EXPECT_EQ(get_registrations(), before | cmd);
+	EXPECT_EQ(sys_membarrier(MEMBARRIER_CMD_PRIVATE_EXPEDITED, 0, 0), 0);
+	free(stack);
+}
+
+int main(int argc, char **argv)
+{
+	/* New program image of exec_drops_registrations. */
+	if (argc == 3 && !strcmp(argv[1], "--expect-registrations"))
+		return get_registrations() != atoi(argv[2]);
+
+	if (query() < 0)
+		ksft_exit_skip("sys membarrier (CONFIG_MEMBARRIER) is disabled.\n");
+	initial_registrations = get_registrations();
+	if (initial_registrations < 0)
+		ksft_exit_skip("MEMBARRIER_CMD_GET_REGISTRATIONS is not available.\n");
+	return test_harness_run(argc, argv);
+}
-- 
2.47.3


_______________________________________________
linux-riscv mailing list
linux-riscv@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-riscv

[RFC PATCH 08/16] LoongArch: membarrier: Add full memory barrier in switch_mm()

[Mathieu Desnoyers]

The expedited membarrier commands need a full memory barrier after
the scheduler stores to rq->curr, before the CPU returns to
user-space. membarrier() skips the IPI to a CPU which it sees running
another mm. This barrier orders the user-space accesses of the
incoming thread after that check. See scenario (A) in
kernel/sched/membarrier.c.

LoongArch has no dedicated barrier. It used to get it from
spin_unlock() in finish_lock_switch(), which was a full barrier
("dbar 0").

This is no longer true. Commit e031a5f3f1ed ("LoongArch: Support dbar
with different hints") made smp_store_release() an ordering barrier
("dbar 0b10010"). It does not order a previous store with a following
load. That commit kept a full barrier in queued_spin_unlock(). Commit
9c68ece8b2a5 ("LoongArch: Define the __io_aw() hook as mmiowb()") then
removed it.

Nothing else is a full barrier on a switch between two user
processes. switch_mm_irqs_off() writes CSRs and sets a bit in the mm
cpumask with a plain "amor". finish_task() uses smp_store_release().

The LoongArch manual lists the instructions which order memory
accesses: DBAR, IBAR, the AM*_DB instructions and LL/SC pairs. A CSR
write and ERTN are not among them.

This was found by reading the code and the architecture manual. On
Loongson-3A6000 or newer, a thread which is scheduled in would run
user-space code without the barrier which membarrier promised to the
caller. User-space would see a memory ordering failure: the caller of
an expedited membarrier command would miss a recent store of that
thread. Older processors treat every dbar hint as "dbar 0" and are not
affected.

Do as powerpc and riscv: add membarrier_arch_switch_mm() and call it
from switch_mm_irqs_off() when the mm changes. It issues smp_mb()
only when the next mm registered for an expedited command.

Compile-tested only.

Fixes: 9c68ece8b2a5 ("LoongArch: Define the __io_aw() hook as mmiowb()")
Cc: stable@vger.kernel.org
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
---
 arch/loongarch/Kconfig                   |  1 +
 arch/loongarch/include/asm/membarrier.h  | 33 ++++++++++++++++++++++++
 arch/loongarch/include/asm/mmu_context.h |  4 +++
 3 files changed, 38 insertions(+)
 create mode 100644 arch/loongarch/include/asm/membarrier.h

diff --git a/arch/loongarch/Kconfig b/arch/loongarch/Kconfig
index b5ed4881d3c0..be6e7545d646 100644
--- a/arch/loongarch/Kconfig
+++ b/arch/loongarch/Kconfig
@@ -22,6 +22,7 @@ config LOONGARCH
 	select ARCH_HAS_FORTIFY_SOURCE
 	select ARCH_HAS_KCOV
 	select ARCH_HAS_KERNEL_FPU_SUPPORT if 64BIT && CPU_HAS_FPU
+	select ARCH_HAS_MEMBARRIER_CALLBACKS
 	select ARCH_HAS_NMI_SAFE_THIS_CPU_OPS
 	select ARCH_HAS_NON_OVERLAPPING_ADDRESS_SPACE
 	select ARCH_HAS_PREEMPT_LAZY
diff --git a/arch/loongarch/include/asm/membarrier.h b/arch/loongarch/include/asm/membarrier.h
new file mode 100644
index 000000000000..1789f878821f
--- /dev/null
+++ b/arch/loongarch/include/asm/membarrier.h
@@ -0,0 +1,33 @@
+/* SPDX-License-Identifier: GPL-2.0-only */
+#ifndef _ASM_LOONGARCH_MEMBARRIER_H
+#define _ASM_LOONGARCH_MEMBARRIER_H
+
+static inline void membarrier_arch_switch_mm(struct mm_struct *prev,
+					     struct mm_struct *next,
+					     struct task_struct *tsk)
+{
+	/*
+	 * Only need the full barrier when switching between processes.
+	 * Barrier when switching from kernel to userspace is not
+	 * required here, given that it is implied by mmdrop(). Barrier
+	 * when switching from userspace to kernel is not needed after
+	 * store to rq->curr.
+	 */
+	if (IS_ENABLED(CONFIG_SMP) &&
+	    likely(!(atomic_read(&next->membarrier_state) &
+		     (MEMBARRIER_STATE_PRIVATE_EXPEDITED |
+		      MEMBARRIER_STATE_GLOBAL_EXPEDITED))))
+		return;
+
+	/*
+	 * The membarrier system call requires a full memory barrier
+	 * after storing to rq->curr, before going back to user-space.
+	 *
+	 * Nothing else provides it on this path. spin_unlock() and
+	 * smp_store_release() are ordering barriers ("dbar 0b10010"):
+	 * they do not order a previous store with a following load.
+	 */
+	smp_mb();
+}
+
+#endif /* _ASM_LOONGARCH_MEMBARRIER_H */
diff --git a/arch/loongarch/include/asm/mmu_context.h b/arch/loongarch/include/asm/mmu_context.h
index 304363bd3935..dba62dd7a131 100644
--- a/arch/loongarch/include/asm/mmu_context.h
+++ b/arch/loongarch/include/asm/mmu_context.h
@@ -9,6 +9,7 @@
 
 #include <linux/errno.h>
 #include <linux/sched.h>
+#include <linux/sched/mm.h>
 #include <linux/mm_types.h>
 #include <linux/smp.h>
 #include <linux/slab.h>
@@ -91,6 +92,9 @@ static inline void switch_mm_irqs_off(struct mm_struct *prev, struct mm_struct *
 	bool need_flush = false;
 	unsigned int cpu = smp_processor_id();
 
+	if (prev != next)
+		membarrier_arch_switch_mm(prev, next, tsk);
+
 	/* Check if our ASID is of an older version and thus invalid */
 	if (!asid_valid(next, cpu))
 		get_new_mmu_context(next, cpu, &need_flush);
-- 
2.47.3

[RFC PATCH 08/16] LoongArch: membarrier: Add full memory barrier in switch_mm()

[Mathieu Desnoyers]

The expedited membarrier commands need a full memory barrier after
the scheduler stores to rq->curr, before the CPU returns to
user-space. membarrier() skips the IPI to a CPU which it sees running
another mm. This barrier orders the user-space accesses of the
incoming thread after that check. See scenario (A) in
kernel/sched/membarrier.c.

LoongArch has no dedicated barrier. It used to get it from
spin_unlock() in finish_lock_switch(), which was a full barrier
("dbar 0").

This is no longer true. Commit e031a5f3f1ed ("LoongArch: Support dbar
with different hints") made smp_store_release() an ordering barrier
("dbar 0b10010"). It does not order a previous store with a following
load. That commit kept a full barrier in queued_spin_unlock(). Commit
9c68ece8b2a5 ("LoongArch: Define the __io_aw() hook as mmiowb()") then
removed it.

Nothing else is a full barrier on a switch between two user
processes. switch_mm_irqs_off() writes CSRs and sets a bit in the mm
cpumask with a plain "amor". finish_task() uses smp_store_release().

The LoongArch manual lists the instructions which order memory
accesses: DBAR, IBAR, the AM*_DB instructions and LL/SC pairs. A CSR
write and ERTN are not among them.

This was found by reading the code and the architecture manual. On
Loongson-3A6000 or newer, a thread which is scheduled in would run
user-space code without the barrier which membarrier promised to the
caller. User-space would see a memory ordering failure: the caller of
an expedited membarrier command would miss a recent store of that
thread. Older processors treat every dbar hint as "dbar 0" and are not
affected.

Do as powerpc and riscv: add membarrier_arch_switch_mm() and call it
from switch_mm_irqs_off() when the mm changes. It issues smp_mb()
only when the next mm registered for an expedited command.

Compile-tested only.

Fixes: 9c68ece8b2a5 ("LoongArch: Define the __io_aw() hook as mmiowb()")
Cc: stable@vger.kernel.org
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
---
 arch/loongarch/Kconfig                   |  1 +
 arch/loongarch/include/asm/membarrier.h  | 33 ++++++++++++++++++++++++
 arch/loongarch/include/asm/mmu_context.h |  4 +++
 3 files changed, 38 insertions(+)
 create mode 100644 arch/loongarch/include/asm/membarrier.h

diff --git a/arch/loongarch/Kconfig b/arch/loongarch/Kconfig
index b5ed4881d3c0..be6e7545d646 100644
--- a/arch/loongarch/Kconfig
+++ b/arch/loongarch/Kconfig
@@ -22,6 +22,7 @@ config LOONGARCH
 	select ARCH_HAS_FORTIFY_SOURCE
 	select ARCH_HAS_KCOV
 	select ARCH_HAS_KERNEL_FPU_SUPPORT if 64BIT && CPU_HAS_FPU
+	select ARCH_HAS_MEMBARRIER_CALLBACKS
 	select ARCH_HAS_NMI_SAFE_THIS_CPU_OPS
 	select ARCH_HAS_NON_OVERLAPPING_ADDRESS_SPACE
 	select ARCH_HAS_PREEMPT_LAZY
diff --git a/arch/loongarch/include/asm/membarrier.h b/arch/loongarch/include/asm/membarrier.h
new file mode 100644
index 000000000000..1789f878821f
--- /dev/null
+++ b/arch/loongarch/include/asm/membarrier.h
@@ -0,0 +1,33 @@
+/* SPDX-License-Identifier: GPL-2.0-only */
+#ifndef _ASM_LOONGARCH_MEMBARRIER_H
+#define _ASM_LOONGARCH_MEMBARRIER_H
+
+static inline void membarrier_arch_switch_mm(struct mm_struct *prev,
+					     struct mm_struct *next,
+					     struct task_struct *tsk)
+{
+	/*
+	 * Only need the full barrier when switching between processes.
+	 * Barrier when switching from kernel to userspace is not
+	 * required here, given that it is implied by mmdrop(). Barrier
+	 * when switching from userspace to kernel is not needed after
+	 * store to rq->curr.
+	 */
+	if (IS_ENABLED(CONFIG_SMP) &&
+	    likely(!(atomic_read(&next->membarrier_state) &
+		     (MEMBARRIER_STATE_PRIVATE_EXPEDITED |
+		      MEMBARRIER_STATE_GLOBAL_EXPEDITED))))
+		return;
+
+	/*
+	 * The membarrier system call requires a full memory barrier
+	 * after storing to rq->curr, before going back to user-space.
+	 *
+	 * Nothing else provides it on this path. spin_unlock() and
+	 * smp_store_release() are ordering barriers ("dbar 0b10010"):
+	 * they do not order a previous store with a following load.
+	 */
+	smp_mb();
+}
+
+#endif /* _ASM_LOONGARCH_MEMBARRIER_H */
diff --git a/arch/loongarch/include/asm/mmu_context.h b/arch/loongarch/include/asm/mmu_context.h
index 304363bd3935..dba62dd7a131 100644
--- a/arch/loongarch/include/asm/mmu_context.h
+++ b/arch/loongarch/include/asm/mmu_context.h
@@ -9,6 +9,7 @@
 
 #include <linux/errno.h>
 #include <linux/sched.h>
+#include <linux/sched/mm.h>
 #include <linux/mm_types.h>
 #include <linux/smp.h>
 #include <linux/slab.h>
@@ -91,6 +92,9 @@ static inline void switch_mm_irqs_off(struct mm_struct *prev, struct mm_struct *
 	bool need_flush = false;
 	unsigned int cpu = smp_processor_id();
 
+	if (prev != next)
+		membarrier_arch_switch_mm(prev, next, tsk);
+
 	/* Check if our ASID is of an older version and thus invalid */
 	if (!asid_valid(next, cpu))
 		get_new_mmu_context(next, cpu, &need_flush);
-- 
2.47.3


_______________________________________________
linux-riscv mailing list
linux-riscv@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-riscv

[RFC PATCH 09/16] parisc: membarrier: Add full memory barrier in switch_mm()

[Mathieu Desnoyers]

The expedited membarrier commands need a full memory barrier after
the scheduler stores to rq->curr, before the CPU returns to
user-space. membarrier() skips the IPI to a CPU which it sees running
another mm. This barrier orders the user-space accesses of the
incoming thread after that check. See scenario (A) in
kernel/sched/membarrier.c.

parisc has no dedicated barrier. It used to get it from
smp_store_release() in finish_task() and from spin_unlock() in
finish_lock_switch(). The first one issued a SYNC instruction. The
second one issued an LDCW instruction, which is strongly ordered.

Both are ordered stores since v5.9:

 - commit 157e9afcc4fa ("Revert "parisc: Revert "Release spinlocks
   using ordered store"""),
 - commit e96ebd589deb ("parisc: Implement __smp_store_release and
   __smp_load_acquire barriers").

Linux runs with the PSW O bit clear. The PA-RISC 2.0 architecture
then allows plain loads and stores to be weakly ordered. It orders an
ordered store before a following ordered load, not before a following
plain load. So an ordered store is not a full barrier.

Nothing else is a full barrier on a switch between two user
processes. switch_mm_irqs_off() only writes control and space
registers.

The PA-RISC 2.0 manual lists the strongly ordered accesses: I/O
space, pages with the TLB U-bit set, LDCW, LDCD, the TLB purges and
SYNC. MTCTL, MTSP and RFI are not among them.

This was found by reading the code and the architecture manual. On a
PA-RISC 2.0 SMP machine, a thread which is scheduled in would run
user-space code without the promised barrier. User-space would see a
memory ordering failure: the caller of an expedited membarrier command
would miss a recent store of that thread. PA-RISC 1.1 is strongly
ordered and not affected.

Do as powerpc and riscv: add membarrier_arch_switch_mm() and call it
from switch_mm_irqs_off() when the mm changes. It issues smp_mb()
only when the next mm registered for an expedited command.

Compile-tested only.

Fixes: 157e9afcc4fa ("Revert "parisc: Revert "Release spinlocks using ordered store""")
Fixes: e96ebd589deb ("parisc: Implement __smp_store_release and __smp_load_acquire barriers")
Cc: stable@vger.kernel.org
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
---
 arch/parisc/Kconfig                   |  1 +
 arch/parisc/include/asm/membarrier.h  | 33 +++++++++++++++++++++++++++
 arch/parisc/include/asm/mmu_context.h |  2 ++
 3 files changed, 36 insertions(+)
 create mode 100644 arch/parisc/include/asm/membarrier.h

diff --git a/arch/parisc/Kconfig b/arch/parisc/Kconfig
index 77f67028ad89..3d6b4cc8d6a5 100644
--- a/arch/parisc/Kconfig
+++ b/arch/parisc/Kconfig
@@ -13,6 +13,7 @@ config PARISC
 	select ARCH_HAS_DMA_ALLOC if PA11
 	select ARCH_HAS_DMA_OPS
 	select ARCH_HAS_ELF_RANDOMIZE
+	select ARCH_HAS_MEMBARRIER_CALLBACKS
 	select ARCH_HAS_STRICT_KERNEL_RWX
 	select ARCH_HAS_STRICT_MODULE_RWX
 	select ARCH_HAS_UBSAN
diff --git a/arch/parisc/include/asm/membarrier.h b/arch/parisc/include/asm/membarrier.h
new file mode 100644
index 000000000000..822dcadaf9bb
--- /dev/null
+++ b/arch/parisc/include/asm/membarrier.h
@@ -0,0 +1,33 @@
+/* SPDX-License-Identifier: GPL-2.0-only */
+#ifndef _ASM_PARISC_MEMBARRIER_H
+#define _ASM_PARISC_MEMBARRIER_H
+
+static inline void membarrier_arch_switch_mm(struct mm_struct *prev,
+					     struct mm_struct *next,
+					     struct task_struct *tsk)
+{
+	/*
+	 * Only need the full barrier when switching between processes.
+	 * Barrier when switching from kernel to userspace is not
+	 * required here, given that it is implied by mmdrop(). Barrier
+	 * when switching from userspace to kernel is not needed after
+	 * store to rq->curr.
+	 */
+	if (IS_ENABLED(CONFIG_SMP) &&
+	    likely(!(atomic_read(&next->membarrier_state) &
+		     (MEMBARRIER_STATE_PRIVATE_EXPEDITED |
+		      MEMBARRIER_STATE_GLOBAL_EXPEDITED))))
+		return;
+
+	/*
+	 * The membarrier system call requires a full memory barrier
+	 * after storing to rq->curr, before going back to user-space.
+	 *
+	 * Nothing else provides it on this path. spin_unlock() and
+	 * smp_store_release() are ordered stores. PA-RISC 2.0 does not
+	 * order an ordered store with a following weakly ordered load.
+	 */
+	smp_mb();
+}
+
+#endif /* _ASM_PARISC_MEMBARRIER_H */
diff --git a/arch/parisc/include/asm/mmu_context.h b/arch/parisc/include/asm/mmu_context.h
index c9187fe836a3..42ff1b9444f1 100644
--- a/arch/parisc/include/asm/mmu_context.h
+++ b/arch/parisc/include/asm/mmu_context.h
@@ -4,6 +4,7 @@
 
 #include <linux/mm.h>
 #include <linux/sched.h>
+#include <linux/sched/mm.h>
 #include <linux/atomic.h>
 #include <linux/spinlock.h>
 #include <asm-generic/mm_hooks.h>
@@ -51,6 +52,7 @@ static inline void switch_mm_irqs_off(struct mm_struct *prev,
 		struct mm_struct *next, struct task_struct *tsk)
 {
 	if (prev != next) {
+		membarrier_arch_switch_mm(prev, next, tsk);
 #ifdef CONFIG_TLB_PTLOCK
 		/* put physical address of page_table_lock in cr28 (tr4)
 		   for TLB faults */
-- 
2.47.3

[RFC PATCH 09/16] parisc: membarrier: Add full memory barrier in switch_mm()

[Mathieu Desnoyers]

The expedited membarrier commands need a full memory barrier after
the scheduler stores to rq->curr, before the CPU returns to
user-space. membarrier() skips the IPI to a CPU which it sees running
another mm. This barrier orders the user-space accesses of the
incoming thread after that check. See scenario (A) in
kernel/sched/membarrier.c.

parisc has no dedicated barrier. It used to get it from
smp_store_release() in finish_task() and from spin_unlock() in
finish_lock_switch(). The first one issued a SYNC instruction. The
second one issued an LDCW instruction, which is strongly ordered.

Both are ordered stores since v5.9:

 - commit 157e9afcc4fa ("Revert "parisc: Revert "Release spinlocks
   using ordered store"""),
 - commit e96ebd589deb ("parisc: Implement __smp_store_release and
   __smp_load_acquire barriers").

Linux runs with the PSW O bit clear. The PA-RISC 2.0 architecture
then allows plain loads and stores to be weakly ordered. It orders an
ordered store before a following ordered load, not before a following
plain load. So an ordered store is not a full barrier.

Nothing else is a full barrier on a switch between two user
processes. switch_mm_irqs_off() only writes control and space
registers.

The PA-RISC 2.0 manual lists the strongly ordered accesses: I/O
space, pages with the TLB U-bit set, LDCW, LDCD, the TLB purges and
SYNC. MTCTL, MTSP and RFI are not among them.

This was found by reading the code and the architecture manual. On a
PA-RISC 2.0 SMP machine, a thread which is scheduled in would run
user-space code without the promised barrier. User-space would see a
memory ordering failure: the caller of an expedited membarrier command
would miss a recent store of that thread. PA-RISC 1.1 is strongly
ordered and not affected.

Do as powerpc and riscv: add membarrier_arch_switch_mm() and call it
from switch_mm_irqs_off() when the mm changes. It issues smp_mb()
only when the next mm registered for an expedited command.

Compile-tested only.

Fixes: 157e9afcc4fa ("Revert "parisc: Revert "Release spinlocks using ordered store""")
Fixes: e96ebd589deb ("parisc: Implement __smp_store_release and __smp_load_acquire barriers")
Cc: stable@vger.kernel.org
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
---
 arch/parisc/Kconfig                   |  1 +
 arch/parisc/include/asm/membarrier.h  | 33 +++++++++++++++++++++++++++
 arch/parisc/include/asm/mmu_context.h |  2 ++
 3 files changed, 36 insertions(+)
 create mode 100644 arch/parisc/include/asm/membarrier.h

diff --git a/arch/parisc/Kconfig b/arch/parisc/Kconfig
index 77f67028ad89..3d6b4cc8d6a5 100644
--- a/arch/parisc/Kconfig
+++ b/arch/parisc/Kconfig
@@ -13,6 +13,7 @@ config PARISC
 	select ARCH_HAS_DMA_ALLOC if PA11
 	select ARCH_HAS_DMA_OPS
 	select ARCH_HAS_ELF_RANDOMIZE
+	select ARCH_HAS_MEMBARRIER_CALLBACKS
 	select ARCH_HAS_STRICT_KERNEL_RWX
 	select ARCH_HAS_STRICT_MODULE_RWX
 	select ARCH_HAS_UBSAN
diff --git a/arch/parisc/include/asm/membarrier.h b/arch/parisc/include/asm/membarrier.h
new file mode 100644
index 000000000000..822dcadaf9bb
--- /dev/null
+++ b/arch/parisc/include/asm/membarrier.h
@@ -0,0 +1,33 @@
+/* SPDX-License-Identifier: GPL-2.0-only */
+#ifndef _ASM_PARISC_MEMBARRIER_H
+#define _ASM_PARISC_MEMBARRIER_H
+
+static inline void membarrier_arch_switch_mm(struct mm_struct *prev,
+					     struct mm_struct *next,
+					     struct task_struct *tsk)
+{
+	/*
+	 * Only need the full barrier when switching between processes.
+	 * Barrier when switching from kernel to userspace is not
+	 * required here, given that it is implied by mmdrop(). Barrier
+	 * when switching from userspace to kernel is not needed after
+	 * store to rq->curr.
+	 */
+	if (IS_ENABLED(CONFIG_SMP) &&
+	    likely(!(atomic_read(&next->membarrier_state) &
+		     (MEMBARRIER_STATE_PRIVATE_EXPEDITED |
+		      MEMBARRIER_STATE_GLOBAL_EXPEDITED))))
+		return;
+
+	/*
+	 * The membarrier system call requires a full memory barrier
+	 * after storing to rq->curr, before going back to user-space.
+	 *
+	 * Nothing else provides it on this path. spin_unlock() and
+	 * smp_store_release() are ordered stores. PA-RISC 2.0 does not
+	 * order an ordered store with a following weakly ordered load.
+	 */
+	smp_mb();
+}
+
+#endif /* _ASM_PARISC_MEMBARRIER_H */
diff --git a/arch/parisc/include/asm/mmu_context.h b/arch/parisc/include/asm/mmu_context.h
index c9187fe836a3..42ff1b9444f1 100644
--- a/arch/parisc/include/asm/mmu_context.h
+++ b/arch/parisc/include/asm/mmu_context.h
@@ -4,6 +4,7 @@
 
 #include <linux/mm.h>
 #include <linux/sched.h>
+#include <linux/sched/mm.h>
 #include <linux/atomic.h>
 #include <linux/spinlock.h>
 #include <asm-generic/mm_hooks.h>
@@ -51,6 +52,7 @@ static inline void switch_mm_irqs_off(struct mm_struct *prev,
 		struct mm_struct *next, struct task_struct *tsk)
 {
 	if (prev != next) {
+		membarrier_arch_switch_mm(prev, next, tsk);
 #ifdef CONFIG_TLB_PTLOCK
 		/* put physical address of page_table_lock in cr28 (tr4)
 		   for TLB faults */
-- 
2.47.3


_______________________________________________
linux-riscv mailing list
linux-riscv@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-riscv

[RFC PATCH 10/16] Documentation/membarrier: Update the per-architecture barrier list

[Mathieu Desnoyers]

membarrier.rst lists how each architecture provides the full memory
barrier after the store to rq->curr. The list misses several SMP
architectures.

Add csky, openrisc, sh and xtensa. They use the generic
smp_store_release(). It issues smp_mb() before the store.
finish_task() uses it.

LoongArch and parisc now provide the barrier with
membarrier_arch_switch_mm(). Add them next to powerpc and riscv.

State the rule which both broke: an architecture which weakens
smp_store_release() or spin_unlock() must provide the barrier another
way.

Update the matching comment in __schedule().

Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
---
 Documentation/scheduler/membarrier.rst | 14 ++++++++++----
 kernel/sched/core.c                    | 10 ++++++----
 2 files changed, 16 insertions(+), 8 deletions(-)

diff --git a/Documentation/scheduler/membarrier.rst b/Documentation/scheduler/membarrier.rst
index 2387804b1c63..c5f6f4f57331 100644
--- a/Documentation/scheduler/membarrier.rst
+++ b/Documentation/scheduler/membarrier.rst
@@ -28,12 +28,18 @@ architectures are as follows.
  - alpha, arc, arm, hexagon, mips rely on the full barrier implied by
    spin_unlock() in finish_lock_switch().
 
+ - csky, openrisc, sh, xtensa rely on the generic smp_store_release(),
+   which issues a full barrier before the store.  finish_task() uses it.
+
  - arm64 relies on the full barrier implied by switch_to().
 
- - powerpc, riscv, s390, sparc, x86 rely on the full barrier implied by
-   switch_mm(), if mm is not NULL; they rely on the full barrier implied
-   by mmdrop(), otherwise.  On powerpc and riscv, switch_mm() relies on
-   membarrier_arch_switch_mm().
+ - loongarch, parisc, powerpc, riscv, s390, sparc, x86 rely on the full
+   barrier implied by switch_mm(), if mm is not NULL; they rely on the full
+   barrier implied by mmdrop(), otherwise.  On loongarch, parisc, powerpc and
+   riscv, switch_mm() relies on membarrier_arch_switch_mm().
+
+An architecture which overrides smp_store_release() or spin_unlock() with
+something weaker than a full barrier must provide this barrier another way.
 
 The barrier matches a full barrier in the proximity of the membarrier system call
 entry, cf. membarrier_{private,global}_expedited().
diff --git a/kernel/sched/core.c b/kernel/sched/core.c
index df6d15cf7109..9985c95ecd73 100644
--- a/kernel/sched/core.c
+++ b/kernel/sched/core.c
@@ -7367,10 +7367,12 @@ static void __sched notrace __schedule(int sched_mode)
 		 * Here are the schemes providing that barrier on the
 		 * various architectures:
 		 * - mm ? switch_mm() : mmdrop() for x86, s390, sparc, PowerPC,
-		 *   RISC-V.  switch_mm() relies on membarrier_arch_switch_mm()
-		 *   on PowerPC and on RISC-V.
-		 * - finish_lock_switch() for weakly-ordered
-		 *   architectures where spin_unlock is a full barrier,
+		 *   RISC-V, LoongArch, PA-RISC.  switch_mm() relies on
+		 *   membarrier_arch_switch_mm() on PowerPC, RISC-V,
+		 *   LoongArch and PA-RISC.
+		 * - finish_task() and finish_lock_switch() for weakly-ordered
+		 *   architectures where smp_store_release() or spin_unlock
+		 *   is a full barrier,
 		 * - switch_to() for arm64 (weakly-ordered, spin_unlock
 		 *   is a RELEASE barrier),
 		 *
-- 
2.47.3

[RFC PATCH 10/16] Documentation/membarrier: Update the per-architecture barrier list

[Mathieu Desnoyers]

membarrier.rst lists how each architecture provides the full memory
barrier after the store to rq->curr. The list misses several SMP
architectures.

Add csky, openrisc, sh and xtensa. They use the generic
smp_store_release(). It issues smp_mb() before the store.
finish_task() uses it.

LoongArch and parisc now provide the barrier with
membarrier_arch_switch_mm(). Add them next to powerpc and riscv.

State the rule which both broke: an architecture which weakens
smp_store_release() or spin_unlock() must provide the barrier another
way.

Update the matching comment in __schedule().

Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
---
 Documentation/scheduler/membarrier.rst | 14 ++++++++++----
 kernel/sched/core.c                    | 10 ++++++----
 2 files changed, 16 insertions(+), 8 deletions(-)

diff --git a/Documentation/scheduler/membarrier.rst b/Documentation/scheduler/membarrier.rst
index 2387804b1c63..c5f6f4f57331 100644
--- a/Documentation/scheduler/membarrier.rst
+++ b/Documentation/scheduler/membarrier.rst
@@ -28,12 +28,18 @@ architectures are as follows.
  - alpha, arc, arm, hexagon, mips rely on the full barrier implied by
    spin_unlock() in finish_lock_switch().
 
+ - csky, openrisc, sh, xtensa rely on the generic smp_store_release(),
+   which issues a full barrier before the store.  finish_task() uses it.
+
  - arm64 relies on the full barrier implied by switch_to().
 
- - powerpc, riscv, s390, sparc, x86 rely on the full barrier implied by
-   switch_mm(), if mm is not NULL; they rely on the full barrier implied
-   by mmdrop(), otherwise.  On powerpc and riscv, switch_mm() relies on
-   membarrier_arch_switch_mm().
+ - loongarch, parisc, powerpc, riscv, s390, sparc, x86 rely on the full
+   barrier implied by switch_mm(), if mm is not NULL; they rely on the full
+   barrier implied by mmdrop(), otherwise.  On loongarch, parisc, powerpc and
+   riscv, switch_mm() relies on membarrier_arch_switch_mm().
+
+An architecture which overrides smp_store_release() or spin_unlock() with
+something weaker than a full barrier must provide this barrier another way.
 
 The barrier matches a full barrier in the proximity of the membarrier system call
 entry, cf. membarrier_{private,global}_expedited().
diff --git a/kernel/sched/core.c b/kernel/sched/core.c
index df6d15cf7109..9985c95ecd73 100644
--- a/kernel/sched/core.c
+++ b/kernel/sched/core.c
@@ -7367,10 +7367,12 @@ static void __sched notrace __schedule(int sched_mode)
 		 * Here are the schemes providing that barrier on the
 		 * various architectures:
 		 * - mm ? switch_mm() : mmdrop() for x86, s390, sparc, PowerPC,
-		 *   RISC-V.  switch_mm() relies on membarrier_arch_switch_mm()
-		 *   on PowerPC and on RISC-V.
-		 * - finish_lock_switch() for weakly-ordered
-		 *   architectures where spin_unlock is a full barrier,
+		 *   RISC-V, LoongArch, PA-RISC.  switch_mm() relies on
+		 *   membarrier_arch_switch_mm() on PowerPC, RISC-V,
+		 *   LoongArch and PA-RISC.
+		 * - finish_task() and finish_lock_switch() for weakly-ordered
+		 *   architectures where smp_store_release() or spin_unlock
+		 *   is a full barrier,
 		 * - switch_to() for arm64 (weakly-ordered, spin_unlock
 		 *   is a RELEASE barrier),
 		 *
-- 
2.47.3


_______________________________________________
linux-riscv mailing list
linux-riscv@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-riscv

[RFC PATCH 11/16] Documentation/features: Update the membarrier sync core requirements

[Mathieu Desnoyers]

The x86 text says that the return from interrupt uses IRET, which
serializes the core after the IPI. This is not what the code relies
on, and it is not always true:

 - Since commit a493d1ca1a03 ("x86/membarrier: Get rid of a dubious
   optimization"), the IPI handler issues a core serializing
   instruction right away. The interrupted thread can be rescheduled
   before the return from interrupt, and another thread of the mm can
   then return to user-space through SYSRET.

 - With FRED, the kernel returns through ERETS and ERETU. The Intel
   SDM does not list them as serializing instructions. It states that
   ERETU has the instruction ordering guarantees of SYSRET.
   The AMD APM does not describe these two instructions.

Describe what x86 relies on instead: sync_core_before_usermode() and
the writes to CR3.

On arm64 with FEAT_ExS, an exception return is a context
synchronization event only when SCTLR_ELx.EOS is set. The kernel sets
it. Document that the sync core command depends on it.

Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
---
 .../membarrier-sync-core/arch-support.txt     | 35 +++++++++++++------
 1 file changed, 24 insertions(+), 11 deletions(-)

diff --git a/Documentation/features/sched/membarrier-sync-core/arch-support.txt b/Documentation/features/sched/membarrier-sync-core/arch-support.txt
index 7425d2b994a3..2782b0d25f43 100644
--- a/Documentation/features/sched/membarrier-sync-core/arch-support.txt
+++ b/Documentation/features/sched/membarrier-sync-core/arch-support.txt
@@ -10,6 +10,10 @@
 # Rely on implicit context synchronization as a result of exception return
 # when returning from IPI handler, and when returning to user-space.
 #
+# On arm64 with FEAT_ExS, an exception return is a context synchronization
+# event only when SCTLR_ELx.EOS is set. The kernel sets it, cf.
+# SCTLR_EL1_EOS in INIT_SCTLR_EL1_MMU_ON. It must stay set.
+#
 # * riscv
 #
 # riscv uses xRET as return from interrupt and to return to user-space.
@@ -28,19 +32,28 @@
 #
 # * x86
 #
-# x86-32 uses IRET as return from interrupt, which takes care of the IPI.
-# However, it uses both IRET and SYSEXIT to go back to user-space. The IRET
-# instruction is core serializing, but not SYSEXIT.
+# x86-32 returns to user-space through either IRET or SYSEXIT. x86-64 returns
+# to user-space through either SYSRETL (compat code), SYSRETQ or IRET, or
+# through ERETU when FRED is enabled. It returns from interrupt through IRET,
+# or through ERETS or ERETU when FRED is enabled.
+#
+# The IRET instruction is core serializing. SYSRET{L,Q}, SYSEXIT, ERETS and
+# ERETU are not. Therefore x86 does not rely on the return instruction:
+#
+#  - the IPI handler calls sync_core_before_usermode(), which issues a core
+#    serializing instruction right away (cf. ipi_sync_core()). The thread may
+#    otherwise be rescheduled before the return from interrupt, and a thread
+#    of the same mm may return to user-space through SYSRET,
+#
+#  - write_cr3() performed by switch_mm() provides core serialization after
+#    changing the current mm,
 #
-# x86-64 uses IRET as return from interrupt, which takes care of the IPI.
-# However, it can return to user-space through either SYSRETL (compat code),
-# SYSRETQ, or IRET.
+#  - the special case of kthread -> uthread (temporarily keeping current mm
+#    into active_mm) issues a sync_core_before_usermode().
 #
-# Given that neither SYSRET{L,Q}, nor SYSEXIT, are core serializing, we rely
-# instead on write_cr3() performed by switch_mm() to provide core serialization
-# after changing the current mm, and deal with the special case of kthread ->
-# uthread (temporarily keeping current mm into active_mm) by issuing a
-# sync_core_before_usermode() in that specific case.
+# With page table isolation, sync_core_before_usermode() does nothing: each
+# return to user-space writes to CR3, which is core serializing. Page table
+# isolation and FRED are mutually exclusive.
 #
     -----------------------
     |         arch |status|
-- 
2.47.3

[RFC PATCH 11/16] Documentation/features: Update the membarrier sync core requirements

[Mathieu Desnoyers]

The x86 text says that the return from interrupt uses IRET, which
serializes the core after the IPI. This is not what the code relies
on, and it is not always true:

 - Since commit a493d1ca1a03 ("x86/membarrier: Get rid of a dubious
   optimization"), the IPI handler issues a core serializing
   instruction right away. The interrupted thread can be rescheduled
   before the return from interrupt, and another thread of the mm can
   then return to user-space through SYSRET.

 - With FRED, the kernel returns through ERETS and ERETU. The Intel
   SDM does not list them as serializing instructions. It states that
   ERETU has the instruction ordering guarantees of SYSRET.
   The AMD APM does not describe these two instructions.

Describe what x86 relies on instead: sync_core_before_usermode() and
the writes to CR3.

On arm64 with FEAT_ExS, an exception return is a context
synchronization event only when SCTLR_ELx.EOS is set. The kernel sets
it. Document that the sync core command depends on it.

Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
---
 .../membarrier-sync-core/arch-support.txt     | 35 +++++++++++++------
 1 file changed, 24 insertions(+), 11 deletions(-)

diff --git a/Documentation/features/sched/membarrier-sync-core/arch-support.txt b/Documentation/features/sched/membarrier-sync-core/arch-support.txt
index 7425d2b994a3..2782b0d25f43 100644
--- a/Documentation/features/sched/membarrier-sync-core/arch-support.txt
+++ b/Documentation/features/sched/membarrier-sync-core/arch-support.txt
@@ -10,6 +10,10 @@
 # Rely on implicit context synchronization as a result of exception return
 # when returning from IPI handler, and when returning to user-space.
 #
+# On arm64 with FEAT_ExS, an exception return is a context synchronization
+# event only when SCTLR_ELx.EOS is set. The kernel sets it, cf.
+# SCTLR_EL1_EOS in INIT_SCTLR_EL1_MMU_ON. It must stay set.
+#
 # * riscv
 #
 # riscv uses xRET as return from interrupt and to return to user-space.
@@ -28,19 +32,28 @@
 #
 # * x86
 #
-# x86-32 uses IRET as return from interrupt, which takes care of the IPI.
-# However, it uses both IRET and SYSEXIT to go back to user-space. The IRET
-# instruction is core serializing, but not SYSEXIT.
+# x86-32 returns to user-space through either IRET or SYSEXIT. x86-64 returns
+# to user-space through either SYSRETL (compat code), SYSRETQ or IRET, or
+# through ERETU when FRED is enabled. It returns from interrupt through IRET,
+# or through ERETS or ERETU when FRED is enabled.
+#
+# The IRET instruction is core serializing. SYSRET{L,Q}, SYSEXIT, ERETS and
+# ERETU are not. Therefore x86 does not rely on the return instruction:
+#
+#  - the IPI handler calls sync_core_before_usermode(), which issues a core
+#    serializing instruction right away (cf. ipi_sync_core()). The thread may
+#    otherwise be rescheduled before the return from interrupt, and a thread
+#    of the same mm may return to user-space through SYSRET,
+#
+#  - write_cr3() performed by switch_mm() provides core serialization after
+#    changing the current mm,
 #
-# x86-64 uses IRET as return from interrupt, which takes care of the IPI.
-# However, it can return to user-space through either SYSRETL (compat code),
-# SYSRETQ, or IRET.
+#  - the special case of kthread -> uthread (temporarily keeping current mm
+#    into active_mm) issues a sync_core_before_usermode().
 #
-# Given that neither SYSRET{L,Q}, nor SYSEXIT, are core serializing, we rely
-# instead on write_cr3() performed by switch_mm() to provide core serialization
-# after changing the current mm, and deal with the special case of kthread ->
-# uthread (temporarily keeping current mm into active_mm) by issuing a
-# sync_core_before_usermode() in that specific case.
+# With page table isolation, sync_core_before_usermode() does nothing: each
+# return to user-space writes to CR3, which is core serializing. Page table
+# isolation and FRED are mutually exclusive.
 #
     -----------------------
     |         arch |status|
-- 
2.47.3


_______________________________________________
linux-riscv mailing list
linux-riscv@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-riscv

[RFC PATCH 12/16] x86/entry: Remove stale membarrier sync core comments

[Mathieu Desnoyers]

Two comments state that the membarrier sync core command relies on
the IRET which returns from the IPI handler. This is no longer true
since commit a493d1ca1a03 ("x86/membarrier: Get rid of a dubious
optimization"): sync_core_before_usermode() serializes the core in
the IPI handler.

Remove both comments.

The comment of sync_core_before_usermode() lists the instructions
which return to user-space without serializing the core. Fix a typo
in it ("sysrel") and add ERETU, which FRED uses.

Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
---
 arch/x86/entry/entry_32.S        | 5 -----
 arch/x86/entry/entry_64.S        | 4 ----
 arch/x86/include/asm/sync_core.h | 2 +-
 3 files changed, 1 insertion(+), 10 deletions(-)

diff --git a/arch/x86/entry/entry_32.S b/arch/x86/entry/entry_32.S
index 92c0b4a94e0a..d47bb08afb24 100644
--- a/arch/x86/entry/entry_32.S
+++ b/arch/x86/entry/entry_32.S
@@ -954,11 +954,6 @@ restore_all_switch_stack:
 	RESTORE_REGS pop=4			# skip orig_eax/error_code
 	CLEAR_CPU_BUFFERS
 .Lirq_return:
-	/*
-	 * ARCH_HAS_MEMBARRIER_SYNC_CORE rely on IRET core serialization
-	 * when returning from IPI handler and when returning from
-	 * scheduler to user-space.
-	 */
 	iret
 
 .Lasm_iret_error:
diff --git a/arch/x86/entry/entry_64.S b/arch/x86/entry/entry_64.S
index 253f0a585927..b8661064232c 100644
--- a/arch/x86/entry/entry_64.S
+++ b/arch/x86/entry/entry_64.S
@@ -626,10 +626,6 @@ SYM_INNER_LABEL(restore_regs_and_return_to_kernel, SYM_L_GLOBAL)
 #endif
 	POP_REGS
 	addq	$8, %rsp	/* skip regs->orig_ax */
-	/*
-	 * ARCH_HAS_MEMBARRIER_SYNC_CORE rely on IRET core serialization
-	 * when returning from IPI handler.
-	 */
 #ifdef CONFIG_XEN_PV
 SYM_INNER_LABEL(early_xen_iret_patch, SYM_L_GLOBAL)
 	ANNOTATE_NOENDBR
diff --git a/arch/x86/include/asm/sync_core.h b/arch/x86/include/asm/sync_core.h
index df22113db9cd..0ab91f7e2020 100644
--- a/arch/x86/include/asm/sync_core.h
+++ b/arch/x86/include/asm/sync_core.h
@@ -91,7 +91,7 @@ static __always_inline void sync_core(void)
 /*
  * Ensure that a core serializing instruction is issued before returning
  * to user-mode. x86 implements return to user-space through sysexit,
- * sysrel, and sysretq, which are not core serializing.
+ * sysretl, sysretq and eretu, which are not core serializing.
  */
 static inline void sync_core_before_usermode(void)
 {
-- 
2.47.3

[RFC PATCH 12/16] x86/entry: Remove stale membarrier sync core comments

[Mathieu Desnoyers]

Two comments state that the membarrier sync core command relies on
the IRET which returns from the IPI handler. This is no longer true
since commit a493d1ca1a03 ("x86/membarrier: Get rid of a dubious
optimization"): sync_core_before_usermode() serializes the core in
the IPI handler.

Remove both comments.

The comment of sync_core_before_usermode() lists the instructions
which return to user-space without serializing the core. Fix a typo
in it ("sysrel") and add ERETU, which FRED uses.

Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
---
 arch/x86/entry/entry_32.S        | 5 -----
 arch/x86/entry/entry_64.S        | 4 ----
 arch/x86/include/asm/sync_core.h | 2 +-
 3 files changed, 1 insertion(+), 10 deletions(-)

diff --git a/arch/x86/entry/entry_32.S b/arch/x86/entry/entry_32.S
index 92c0b4a94e0a..d47bb08afb24 100644
--- a/arch/x86/entry/entry_32.S
+++ b/arch/x86/entry/entry_32.S
@@ -954,11 +954,6 @@ restore_all_switch_stack:
 	RESTORE_REGS pop=4			# skip orig_eax/error_code
 	CLEAR_CPU_BUFFERS
 .Lirq_return:
-	/*
-	 * ARCH_HAS_MEMBARRIER_SYNC_CORE rely on IRET core serialization
-	 * when returning from IPI handler and when returning from
-	 * scheduler to user-space.
-	 */
 	iret
 
 .Lasm_iret_error:
diff --git a/arch/x86/entry/entry_64.S b/arch/x86/entry/entry_64.S
index 253f0a585927..b8661064232c 100644
--- a/arch/x86/entry/entry_64.S
+++ b/arch/x86/entry/entry_64.S
@@ -626,10 +626,6 @@ SYM_INNER_LABEL(restore_regs_and_return_to_kernel, SYM_L_GLOBAL)
 #endif
 	POP_REGS
 	addq	$8, %rsp	/* skip regs->orig_ax */
-	/*
-	 * ARCH_HAS_MEMBARRIER_SYNC_CORE rely on IRET core serialization
-	 * when returning from IPI handler.
-	 */
 #ifdef CONFIG_XEN_PV
 SYM_INNER_LABEL(early_xen_iret_patch, SYM_L_GLOBAL)
 	ANNOTATE_NOENDBR
diff --git a/arch/x86/include/asm/sync_core.h b/arch/x86/include/asm/sync_core.h
index df22113db9cd..0ab91f7e2020 100644
--- a/arch/x86/include/asm/sync_core.h
+++ b/arch/x86/include/asm/sync_core.h
@@ -91,7 +91,7 @@ static __always_inline void sync_core(void)
 /*
  * Ensure that a core serializing instruction is issued before returning
  * to user-mode. x86 implements return to user-space through sysexit,
- * sysrel, and sysretq, which are not core serializing.
+ * sysretl, sysretq and eretu, which are not core serializing.
  */
 static inline void sync_core_before_usermode(void)
 {
-- 
2.47.3


_______________________________________________
linux-riscv mailing list
linux-riscv@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-riscv

[RFC PATCH 13/16] selftests/membarrier: Add litmus tests of the memory ordering

[Mathieu Desnoyers]

membarrier relies on memory barriers in the system call, in its IPI
handler and in the scheduler. The comment at the top of
kernel/sched/membarrier.c describes the scenarios. A test on real
hardware can check that the kernel executes a barrier. It cannot show
that the barrier is needed: a missing barrier may never fail in
practice.

Check these scenarios with the Linux kernel memory model instead.
The litmus tests cover:

 - a thread which runs after the IPI, scenario (A),
 - a thread which ran before the IPI, scenario (B),
 - a thread which is scheduled in while membarrier() sees another
   task in rq->curr, scenarios (A) and (C),
 - a thread which is scheduled out before membarrier() loads
   rq->curr, scenarios (B) and (C).

herd7 reports "Never" for these 4 tests. 9 variants drop one barrier
each, and herd7 reports "Sometimes" for them. One of the variants
replaces the full barrier after the store to rq->curr with
smp_store_release(). This is the LoongArch and parisc case.

The README describes the model and how to run the tests.

Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
---
 kernel/sched/membarrier.c                     |  3 ++
 .../selftests/membarrier/litmus/README        | 52 +++++++++++++++++++
 .../litmus/ipi-after-thread-no-exit-mb.litmus | 36 +++++++++++++
 .../ipi-after-thread-no-handler-mb.litmus     | 36 +++++++++++++
 .../membarrier/litmus/ipi-after-thread.litmus | 47 +++++++++++++++++
 .../ipi-before-thread-no-entry-mb.litmus      | 37 +++++++++++++
 .../ipi-before-thread-no-handler-mb.litmus    | 37 +++++++++++++
 .../litmus/ipi-before-thread.litmus           | 46 ++++++++++++++++
 .../litmus/sched-in-no-entry-mb.litmus        | 34 ++++++++++++
 .../membarrier/litmus/sched-in-no-mb.litmus   | 33 ++++++++++++
 .../membarrier/litmus/sched-in-release.litmus | 38 ++++++++++++++
 .../membarrier/litmus/sched-in.litmus         | 51 ++++++++++++++++++
 .../litmus/sched-out-no-exit-mb.litmus        | 34 ++++++++++++
 .../litmus/sched-out-no-sched-mb.litmus       | 34 ++++++++++++
 .../membarrier/litmus/sched-out.litmus        | 49 +++++++++++++++++
 15 files changed, 567 insertions(+)
 create mode 100644 tools/testing/selftests/membarrier/litmus/README
 create mode 100644 tools/testing/selftests/membarrier/litmus/ipi-after-thread-no-exit-mb.litmus
 create mode 100644 tools/testing/selftests/membarrier/litmus/ipi-after-thread-no-handler-mb.litmus
 create mode 100644 tools/testing/selftests/membarrier/litmus/ipi-after-thread.litmus
 create mode 100644 tools/testing/selftests/membarrier/litmus/ipi-before-thread-no-entry-mb.litmus
 create mode 100644 tools/testing/selftests/membarrier/litmus/ipi-before-thread-no-handler-mb.litmus
 create mode 100644 tools/testing/selftests/membarrier/litmus/ipi-before-thread.litmus
 create mode 100644 tools/testing/selftests/membarrier/litmus/sched-in-no-entry-mb.litmus
 create mode 100644 tools/testing/selftests/membarrier/litmus/sched-in-no-mb.litmus
 create mode 100644 tools/testing/selftests/membarrier/litmus/sched-in-release.litmus
 create mode 100644 tools/testing/selftests/membarrier/litmus/sched-in.litmus
 create mode 100644 tools/testing/selftests/membarrier/litmus/sched-out-no-exit-mb.litmus
 create mode 100644 tools/testing/selftests/membarrier/litmus/sched-out-no-sched-mb.litmus
 create mode 100644 tools/testing/selftests/membarrier/litmus/sched-out.litmus

diff --git a/kernel/sched/membarrier.c b/kernel/sched/membarrier.c
index 46e823a3d668..5dd88ac0957b 100644
--- a/kernel/sched/membarrier.c
+++ b/kernel/sched/membarrier.c
@@ -133,6 +133,9 @@
  * Using the scenario from (A), we can show that (a) needs to be paired
  * with (g). Using the scenario from (B), we can show that (c) needs to
  * be paired with (d).
+ *
+ * The litmus tests in tools/testing/selftests/membarrier/litmus check
+ * these scenarios with the Linux kernel memory model.
  */
 
 /*
diff --git a/tools/testing/selftests/membarrier/litmus/README b/tools/testing/selftests/membarrier/litmus/README
new file mode 100644
index 000000000000..c9ae572a9d43
--- /dev/null
+++ b/tools/testing/selftests/membarrier/litmus/README
@@ -0,0 +1,52 @@
+Litmus tests of the membarrier system call. They check the memory
+ordering scenarios of kernel/sched/membarrier.c with the Linux kernel
+memory model. Each test with the result "Never" models a case of
+these scenarios. A test with the result "Sometimes" drops one
+barrier, to show that the scenario needs it.
+
+In each test, P0 is the caller of membarrier(). P1 is a CPU which runs
+a thread of the target process.
+
+The user-space accesses of the thread are WRITE_ONCE() and READ_ONCE()
+in program order, without a memory barrier between them. This is what
+barrier() gives in user-space code.
+
+The model has no interrupt. The IPI is modeled as a request and an
+acknowledgment: the caller stores "req", the handler loads it, issues
+its memory barrier and stores "ack", and the caller loads "ack". The
+"exists" clause of each test keeps the executions where this exchange
+took place.
+
+Run them from tools/memory-model, with herd7 installed:
+
+  cd tools/memory-model
+  for f in ../testing/selftests/membarrier/litmus/*.litmus; do
+	herd7 -conf linux-kernel.cfg $f | grep Observation
+  done
+
+Each observation must match the "Result" in the comment of its test.
+
+The thread is interrupted by the IPI:
+
+  ipi-before-thread.litmus, ipi-before-thread-no-entry-mb.litmus,
+  ipi-before-thread-no-handler-mb.litmus
+    Scenario (A). The thread runs after the IPI. It sees the stores
+    which the caller did before membarrier().
+
+  ipi-after-thread.litmus, ipi-after-thread-no-exit-mb.litmus,
+  ipi-after-thread-no-handler-mb.litmus
+    Scenario (B). The thread ran before the IPI. The caller sees its
+    stores after membarrier().
+
+The caller sees another task in rq->curr and sends no IPI:
+
+  sched-in.litmus, sched-in-no-entry-mb.litmus, sched-in-no-mb.litmus,
+  sched-in-release.litmus
+    Scenarios (A) and (C). The thread is scheduled in. The scheduler
+    issues a full barrier after the store to rq->curr. An
+    smp_store_release() which is not a full barrier is not enough.
+
+  sched-out.litmus, sched-out-no-exit-mb.litmus,
+  sched-out-no-sched-mb.litmus
+    Scenarios (B) and (C). The thread is scheduled out. The scheduler
+    issues a full barrier before the store to rq->curr.
diff --git a/tools/testing/selftests/membarrier/litmus/ipi-after-thread-no-exit-mb.litmus b/tools/testing/selftests/membarrier/litmus/ipi-after-thread-no-exit-mb.litmus
new file mode 100644
index 000000000000..fb67beaba996
--- /dev/null
+++ b/tools/testing/selftests/membarrier/litmus/ipi-after-thread-no-exit-mb.litmus
@@ -0,0 +1,36 @@
+C ipi-after-thread-no-exit-mb
+
+(*
+ * Result: Sometimes
+ *
+ * ipi-after-thread without the barrier (c) at the end of membarrier().
+ * The load of x by P0 can then be done before the IPI completes.
+ *)
+
+{}
+
+P0(int *x, int *y, int *req, int *ack)
+{
+	int r_ack;
+	int r1;
+	int r2;
+
+	r2 = READ_ONCE(*y);
+	smp_mb();
+	WRITE_ONCE(*req, 1);
+	r_ack = READ_ONCE(*ack);
+	r1 = READ_ONCE(*x);
+}
+
+P1(int *x, int *y, int *req, int *ack)
+{
+	int r_req;
+
+	WRITE_ONCE(*x, 1);
+	WRITE_ONCE(*y, 1);
+	r_req = READ_ONCE(*req);
+	smp_mb();
+	WRITE_ONCE(*ack, 1);
+}
+
+exists (0:r2=1 /\ 1:r_req=1 /\ 0:r_ack=1 /\ 0:r1=0)
diff --git a/tools/testing/selftests/membarrier/litmus/ipi-after-thread-no-handler-mb.litmus b/tools/testing/selftests/membarrier/litmus/ipi-after-thread-no-handler-mb.litmus
new file mode 100644
index 000000000000..687e883850a9
--- /dev/null
+++ b/tools/testing/selftests/membarrier/litmus/ipi-after-thread-no-handler-mb.litmus
@@ -0,0 +1,36 @@
+C ipi-after-thread-no-handler-mb
+
+(*
+ * Result: Sometimes
+ *
+ * ipi-after-thread without the barrier in the IPI handler. The store to
+ * x by P1 can then be delayed past the acknowledgment.
+ *)
+
+{}
+
+P0(int *x, int *y, int *req, int *ack)
+{
+	int r_ack;
+	int r1;
+	int r2;
+
+	r2 = READ_ONCE(*y);
+	smp_mb();
+	WRITE_ONCE(*req, 1);
+	r_ack = READ_ONCE(*ack);
+	smp_mb();
+	r1 = READ_ONCE(*x);
+}
+
+P1(int *x, int *y, int *req, int *ack)
+{
+	int r_req;
+
+	WRITE_ONCE(*x, 1);
+	WRITE_ONCE(*y, 1);
+	r_req = READ_ONCE(*req);
+	WRITE_ONCE(*ack, 1);
+}
+
+exists (0:r2=1 /\ 1:r_req=1 /\ 0:r_ack=1 /\ 0:r1=0)
diff --git a/tools/testing/selftests/membarrier/litmus/ipi-after-thread.litmus b/tools/testing/selftests/membarrier/litmus/ipi-after-thread.litmus
new file mode 100644
index 000000000000..1145faf62b6d
--- /dev/null
+++ b/tools/testing/selftests/membarrier/litmus/ipi-after-thread.litmus
@@ -0,0 +1,47 @@
+C ipi-after-thread
+
+(*
+ * Result: Never
+ *
+ * Scenario (B) of kernel/sched/membarrier.c.
+ *
+ * P1 is a user-space thread which runs "x = 1; y = 1" in program order,
+ * without a memory barrier. The IPI of membarrier() interrupts it after
+ * that. P0 reads y before membarrier() and x after it.
+ *
+ * req and ack model the IPI, see README.
+ *
+ * The barrier in the IPI handler orders both stores before the
+ * acknowledgment. The barrier (c) at the end of membarrier() orders the
+ * acknowledgment before "r1 = x". P0 thus sees x == 1. This holds
+ * whatever P0 read from y: y is kept to match the scenario.
+ *)
+
+{}
+
+P0(int *x, int *y, int *req, int *ack)
+{
+	int r_ack;
+	int r1;
+	int r2;
+
+	r2 = READ_ONCE(*y);
+	smp_mb();
+	WRITE_ONCE(*req, 1);
+	r_ack = READ_ONCE(*ack);
+	smp_mb();
+	r1 = READ_ONCE(*x);
+}
+
+P1(int *x, int *y, int *req, int *ack)
+{
+	int r_req;
+
+	WRITE_ONCE(*x, 1);
+	WRITE_ONCE(*y, 1);
+	r_req = READ_ONCE(*req);
+	smp_mb();
+	WRITE_ONCE(*ack, 1);
+}
+
+exists (0:r2=1 /\ 1:r_req=1 /\ 0:r_ack=1 /\ 0:r1=0)
diff --git a/tools/testing/selftests/membarrier/litmus/ipi-before-thread-no-entry-mb.litmus b/tools/testing/selftests/membarrier/litmus/ipi-before-thread-no-entry-mb.litmus
new file mode 100644
index 000000000000..9685e34e224a
--- /dev/null
+++ b/tools/testing/selftests/membarrier/litmus/ipi-before-thread-no-entry-mb.litmus
@@ -0,0 +1,37 @@
+C ipi-before-thread-no-entry-mb
+
+(*
+ * Result: Sometimes
+ *
+ * ipi-before-thread without the barrier (a) at the start of
+ * membarrier(). The store to x can be delayed past the IPI. P1 and P0
+ * then both read 0.
+ *)
+
+{}
+
+P0(int *x, int *y, int *req, int *ack)
+{
+	int r_ack;
+	int r2;
+
+	WRITE_ONCE(*x, 1);
+	WRITE_ONCE(*req, 1);
+	r_ack = READ_ONCE(*ack);
+	smp_mb();
+	r2 = READ_ONCE(*y);
+}
+
+P1(int *x, int *y, int *req, int *ack)
+{
+	int r_req;
+	int r1;
+
+	r_req = READ_ONCE(*req);
+	smp_mb();
+	WRITE_ONCE(*ack, 1);
+	WRITE_ONCE(*y, 1);
+	r1 = READ_ONCE(*x);
+}
+
+exists (1:r_req=1 /\ 0:r_ack=1 /\ 1:r1=0 /\ 0:r2=0)
diff --git a/tools/testing/selftests/membarrier/litmus/ipi-before-thread-no-handler-mb.litmus b/tools/testing/selftests/membarrier/litmus/ipi-before-thread-no-handler-mb.litmus
new file mode 100644
index 000000000000..1559bce76ea5
--- /dev/null
+++ b/tools/testing/selftests/membarrier/litmus/ipi-before-thread-no-handler-mb.litmus
@@ -0,0 +1,37 @@
+C ipi-before-thread-no-handler-mb
+
+(*
+ * Result: Sometimes
+ *
+ * ipi-before-thread without the barrier in the IPI handler. The load of
+ * x by P1 can then be done before the handler loads req. This is why
+ * ipi_mb() issues smp_mb() rather than rely on the interrupt alone.
+ *)
+
+{}
+
+P0(int *x, int *y, int *req, int *ack)
+{
+	int r_ack;
+	int r2;
+
+	WRITE_ONCE(*x, 1);
+	smp_mb();
+	WRITE_ONCE(*req, 1);
+	r_ack = READ_ONCE(*ack);
+	smp_mb();
+	r2 = READ_ONCE(*y);
+}
+
+P1(int *x, int *y, int *req, int *ack)
+{
+	int r_req;
+	int r1;
+
+	r_req = READ_ONCE(*req);
+	WRITE_ONCE(*ack, 1);
+	WRITE_ONCE(*y, 1);
+	r1 = READ_ONCE(*x);
+}
+
+exists (1:r_req=1 /\ 0:r_ack=1 /\ 1:r1=0 /\ 0:r2=0)
diff --git a/tools/testing/selftests/membarrier/litmus/ipi-before-thread.litmus b/tools/testing/selftests/membarrier/litmus/ipi-before-thread.litmus
new file mode 100644
index 000000000000..e21e79e2d2ad
--- /dev/null
+++ b/tools/testing/selftests/membarrier/litmus/ipi-before-thread.litmus
@@ -0,0 +1,46 @@
+C ipi-before-thread
+
+(*
+ * Result: Never
+ *
+ * Scenario (A) of kernel/sched/membarrier.c.
+ *
+ * P0 calls membarrier(). P1 is a user-space thread which the IPI
+ * interrupts before it runs "y = 1; r1 = x". Its two accesses are in
+ * program order, without a memory barrier.
+ *
+ * req and ack model the IPI, see README.
+ *
+ * The barrier (a) at the start of membarrier() orders "x = 1" before
+ * the IPI. P1 thus sees x == 1 after the IPI. This holds whatever P0
+ * reads from y: y is kept to match the scenario.
+ *)
+
+{}
+
+P0(int *x, int *y, int *req, int *ack)
+{
+	int r_ack;
+	int r2;
+
+	WRITE_ONCE(*x, 1);
+	smp_mb();
+	WRITE_ONCE(*req, 1);
+	r_ack = READ_ONCE(*ack);
+	smp_mb();
+	r2 = READ_ONCE(*y);
+}
+
+P1(int *x, int *y, int *req, int *ack)
+{
+	int r_req;
+	int r1;
+
+	r_req = READ_ONCE(*req);
+	smp_mb();
+	WRITE_ONCE(*ack, 1);
+	WRITE_ONCE(*y, 1);
+	r1 = READ_ONCE(*x);
+}
+
+exists (1:r_req=1 /\ 0:r_ack=1 /\ 1:r1=0 /\ 0:r2=0)
diff --git a/tools/testing/selftests/membarrier/litmus/sched-in-no-entry-mb.litmus b/tools/testing/selftests/membarrier/litmus/sched-in-no-entry-mb.litmus
new file mode 100644
index 000000000000..a40837fad005
--- /dev/null
+++ b/tools/testing/selftests/membarrier/litmus/sched-in-no-entry-mb.litmus
@@ -0,0 +1,34 @@
+C sched-in-no-entry-mb
+
+(*
+ * Result: Sometimes
+ *
+ * sched-in without the barrier (a) at the start of membarrier(). The
+ * store to x can be delayed past the load of rq->curr. P0 sees no
+ * thread and sends no IPI. The thread reads x == 0.
+ *)
+
+{}
+
+P0(int *x, int *y, int *curr)
+{
+	int r_curr;
+	int r2;
+
+	WRITE_ONCE(*x, 1);
+	r_curr = READ_ONCE(*curr);
+	smp_mb();
+	r2 = READ_ONCE(*y);
+}
+
+P1(int *x, int *y, int *curr)
+{
+	int r1;
+
+	WRITE_ONCE(*curr, 1);
+	smp_mb();
+	WRITE_ONCE(*y, 1);
+	r1 = READ_ONCE(*x);
+}
+
+exists (0:r_curr=0 /\ 1:r1=0 /\ 0:r2=0)
diff --git a/tools/testing/selftests/membarrier/litmus/sched-in-no-mb.litmus b/tools/testing/selftests/membarrier/litmus/sched-in-no-mb.litmus
new file mode 100644
index 000000000000..6840cb26e13d
--- /dev/null
+++ b/tools/testing/selftests/membarrier/litmus/sched-in-no-mb.litmus
@@ -0,0 +1,33 @@
+C sched-in-no-mb
+
+(*
+ * Result: Sometimes
+ *
+ * sched-in without the full barrier after the store to rq->curr. P0
+ * sees no thread and sends no IPI. The thread reads x == 0.
+ *)
+
+{}
+
+P0(int *x, int *y, int *curr)
+{
+	int r_curr;
+	int r2;
+
+	WRITE_ONCE(*x, 1);
+	smp_mb();
+	r_curr = READ_ONCE(*curr);
+	smp_mb();
+	r2 = READ_ONCE(*y);
+}
+
+P1(int *x, int *y, int *curr)
+{
+	int r1;
+
+	WRITE_ONCE(*curr, 1);
+	WRITE_ONCE(*y, 1);
+	r1 = READ_ONCE(*x);
+}
+
+exists (0:r_curr=0 /\ 1:r1=0 /\ 0:r2=0)
diff --git a/tools/testing/selftests/membarrier/litmus/sched-in-release.litmus b/tools/testing/selftests/membarrier/litmus/sched-in-release.litmus
new file mode 100644
index 000000000000..d2df75b388fe
--- /dev/null
+++ b/tools/testing/selftests/membarrier/litmus/sched-in-release.litmus
@@ -0,0 +1,38 @@
+C sched-in-release
+
+(*
+ * Result: Sometimes
+ *
+ * sched-in with smp_store_release() in place of the full barrier after
+ * the store to rq->curr. This is finish_task() on an architecture where
+ * smp_store_release() is weaker than a full barrier. It does not order
+ * the store to rq->curr before the loads of the thread.
+ *)
+
+{
+	int on_cpu = 1;
+}
+
+P0(int *x, int *y, int *curr)
+{
+	int r_curr;
+	int r2;
+
+	WRITE_ONCE(*x, 1);
+	smp_mb();
+	r_curr = READ_ONCE(*curr);
+	smp_mb();
+	r2 = READ_ONCE(*y);
+}
+
+P1(int *x, int *y, int *curr, int *on_cpu)
+{
+	int r1;
+
+	WRITE_ONCE(*curr, 1);
+	smp_store_release(on_cpu, 0);
+	WRITE_ONCE(*y, 1);
+	r1 = READ_ONCE(*x);
+}
+
+exists (0:r_curr=0 /\ 1:r1=0 /\ 0:r2=0)
diff --git a/tools/testing/selftests/membarrier/litmus/sched-in.litmus b/tools/testing/selftests/membarrier/litmus/sched-in.litmus
new file mode 100644
index 000000000000..c89e2c6c4554
--- /dev/null
+++ b/tools/testing/selftests/membarrier/litmus/sched-in.litmus
@@ -0,0 +1,51 @@
+C sched-in
+
+(*
+ * Result: Never
+ *
+ * Scenarios (A) and (C) of kernel/sched/membarrier.c: a thread of the
+ * target process is scheduled in while P0 calls membarrier().
+ *
+ * curr stands for rq->curr of the CPU of P1: 0 for another task, 1 for
+ * the thread. P0 loads it between its two barriers. It sends no IPI
+ * when it reads 0, which is the case the "exists" clause keeps.
+ *
+ * P1 is the scheduler, then the thread in user-space. The scheduler
+ * stores to rq->curr, then issues a full barrier before the return to
+ * user-space. Each architecture provides this barrier its own way, see
+ * Documentation/scheduler/membarrier.rst.
+ *
+ * Either P0 sees the thread and sends the IPI, or the thread sees
+ * x == 1. This holds whatever P0 reads from y: y is kept to match the
+ * scenario.
+ *
+ * The same argument holds with tsk->mm in place of rq->curr. It covers
+ * kthread_use_mm(), scenario (E), and the icache_stale_mask of the
+ * RISC-V sync core command.
+ *)
+
+{}
+
+P0(int *x, int *y, int *curr)
+{
+	int r_curr;
+	int r2;
+
+	WRITE_ONCE(*x, 1);
+	smp_mb();
+	r_curr = READ_ONCE(*curr);
+	smp_mb();
+	r2 = READ_ONCE(*y);
+}
+
+P1(int *x, int *y, int *curr)
+{
+	int r1;
+
+	WRITE_ONCE(*curr, 1);
+	smp_mb();
+	WRITE_ONCE(*y, 1);
+	r1 = READ_ONCE(*x);
+}
+
+exists (0:r_curr=0 /\ 1:r1=0 /\ 0:r2=0)
diff --git a/tools/testing/selftests/membarrier/litmus/sched-out-no-exit-mb.litmus b/tools/testing/selftests/membarrier/litmus/sched-out-no-exit-mb.litmus
new file mode 100644
index 000000000000..e4bb1ee7a935
--- /dev/null
+++ b/tools/testing/selftests/membarrier/litmus/sched-out-no-exit-mb.litmus
@@ -0,0 +1,34 @@
+C sched-out-no-exit-mb
+
+(*
+ * Result: Sometimes
+ *
+ * sched-out without the barrier (c) at the end of membarrier(). The
+ * load of x by P0 can then be done before it loads rq->curr.
+ *)
+
+{
+	int curr = 1;
+}
+
+P0(int *x, int *y, int *curr)
+{
+	int r_curr;
+	int r1;
+	int r2;
+
+	r2 = READ_ONCE(*y);
+	smp_mb();
+	r_curr = READ_ONCE(*curr);
+	r1 = READ_ONCE(*x);
+}
+
+P1(int *x, int *y, int *curr)
+{
+	WRITE_ONCE(*x, 1);
+	WRITE_ONCE(*y, 1);
+	smp_mb();
+	WRITE_ONCE(*curr, 0);
+}
+
+exists (0:r2=1 /\ 0:r_curr=0 /\ 0:r1=0)
diff --git a/tools/testing/selftests/membarrier/litmus/sched-out-no-sched-mb.litmus b/tools/testing/selftests/membarrier/litmus/sched-out-no-sched-mb.litmus
new file mode 100644
index 000000000000..7b690db2b4be
--- /dev/null
+++ b/tools/testing/selftests/membarrier/litmus/sched-out-no-sched-mb.litmus
@@ -0,0 +1,34 @@
+C sched-out-no-sched-mb
+
+(*
+ * Result: Sometimes
+ *
+ * sched-out without the full barrier before the store to rq->curr. The
+ * store to x by the thread can then be delayed past it.
+ *)
+
+{
+	int curr = 1;
+}
+
+P0(int *x, int *y, int *curr)
+{
+	int r_curr;
+	int r1;
+	int r2;
+
+	r2 = READ_ONCE(*y);
+	smp_mb();
+	r_curr = READ_ONCE(*curr);
+	smp_mb();
+	r1 = READ_ONCE(*x);
+}
+
+P1(int *x, int *y, int *curr)
+{
+	WRITE_ONCE(*x, 1);
+	WRITE_ONCE(*y, 1);
+	WRITE_ONCE(*curr, 0);
+}
+
+exists (0:r2=1 /\ 0:r_curr=0 /\ 0:r1=0)
diff --git a/tools/testing/selftests/membarrier/litmus/sched-out.litmus b/tools/testing/selftests/membarrier/litmus/sched-out.litmus
new file mode 100644
index 000000000000..8fb1532bd13c
--- /dev/null
+++ b/tools/testing/selftests/membarrier/litmus/sched-out.litmus
@@ -0,0 +1,49 @@
+C sched-out
+
+(*
+ * Result: Never
+ *
+ * Scenarios (B) and (C) of kernel/sched/membarrier.c: a thread of the
+ * target process is scheduled out while P0 calls membarrier().
+ *
+ * curr stands for rq->curr of the CPU of P1: 1 for the thread, 0 for
+ * another task. P0 sends no IPI when it reads 0, which is the case the
+ * "exists" clause keeps.
+ *
+ * P1 is the thread in user-space, which runs "x = 1; y = 1" in program
+ * order. Then the scheduler issues a full barrier before it stores to
+ * rq->curr: smp_mb__after_spinlock() in __schedule().
+ *
+ * When P0 saw y == 1 and sees that the thread is gone, it sees x == 1.
+ * This holds whatever P0 read from y: y is kept to match the scenario.
+ *
+ * The same argument holds with tsk->mm in place of rq->curr. It covers
+ * exit_mm(), scenario (D), and kthread_unuse_mm(), scenario (E).
+ *)
+
+{
+	int curr = 1;
+}
+
+P0(int *x, int *y, int *curr)
+{
+	int r_curr;
+	int r1;
+	int r2;
+
+	r2 = READ_ONCE(*y);
+	smp_mb();
+	r_curr = READ_ONCE(*curr);
+	smp_mb();
+	r1 = READ_ONCE(*x);
+}
+
+P1(int *x, int *y, int *curr)
+{
+	WRITE_ONCE(*x, 1);
+	WRITE_ONCE(*y, 1);
+	smp_mb();
+	WRITE_ONCE(*curr, 0);
+}
+
+exists (0:r2=1 /\ 0:r_curr=0 /\ 0:r1=0)
-- 
2.47.3

[RFC PATCH 13/16] selftests/membarrier: Add litmus tests of the memory ordering

[Mathieu Desnoyers]

membarrier relies on memory barriers in the system call, in its IPI
handler and in the scheduler. The comment at the top of
kernel/sched/membarrier.c describes the scenarios. A test on real
hardware can check that the kernel executes a barrier. It cannot show
that the barrier is needed: a missing barrier may never fail in
practice.

Check these scenarios with the Linux kernel memory model instead.
The litmus tests cover:

 - a thread which runs after the IPI, scenario (A),
 - a thread which ran before the IPI, scenario (B),
 - a thread which is scheduled in while membarrier() sees another
   task in rq->curr, scenarios (A) and (C),
 - a thread which is scheduled out before membarrier() loads
   rq->curr, scenarios (B) and (C).

herd7 reports "Never" for these 4 tests. 9 variants drop one barrier
each, and herd7 reports "Sometimes" for them. One of the variants
replaces the full barrier after the store to rq->curr with
smp_store_release(). This is the LoongArch and parisc case.

The README describes the model and how to run the tests.

Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
---
 kernel/sched/membarrier.c                     |  3 ++
 .../selftests/membarrier/litmus/README        | 52 +++++++++++++++++++
 .../litmus/ipi-after-thread-no-exit-mb.litmus | 36 +++++++++++++
 .../ipi-after-thread-no-handler-mb.litmus     | 36 +++++++++++++
 .../membarrier/litmus/ipi-after-thread.litmus | 47 +++++++++++++++++
 .../ipi-before-thread-no-entry-mb.litmus      | 37 +++++++++++++
 .../ipi-before-thread-no-handler-mb.litmus    | 37 +++++++++++++
 .../litmus/ipi-before-thread.litmus           | 46 ++++++++++++++++
 .../litmus/sched-in-no-entry-mb.litmus        | 34 ++++++++++++
 .../membarrier/litmus/sched-in-no-mb.litmus   | 33 ++++++++++++
 .../membarrier/litmus/sched-in-release.litmus | 38 ++++++++++++++
 .../membarrier/litmus/sched-in.litmus         | 51 ++++++++++++++++++
 .../litmus/sched-out-no-exit-mb.litmus        | 34 ++++++++++++
 .../litmus/sched-out-no-sched-mb.litmus       | 34 ++++++++++++
 .../membarrier/litmus/sched-out.litmus        | 49 +++++++++++++++++
 15 files changed, 567 insertions(+)
 create mode 100644 tools/testing/selftests/membarrier/litmus/README
 create mode 100644 tools/testing/selftests/membarrier/litmus/ipi-after-thread-no-exit-mb.litmus
 create mode 100644 tools/testing/selftests/membarrier/litmus/ipi-after-thread-no-handler-mb.litmus
 create mode 100644 tools/testing/selftests/membarrier/litmus/ipi-after-thread.litmus
 create mode 100644 tools/testing/selftests/membarrier/litmus/ipi-before-thread-no-entry-mb.litmus
 create mode 100644 tools/testing/selftests/membarrier/litmus/ipi-before-thread-no-handler-mb.litmus
 create mode 100644 tools/testing/selftests/membarrier/litmus/ipi-before-thread.litmus
 create mode 100644 tools/testing/selftests/membarrier/litmus/sched-in-no-entry-mb.litmus
 create mode 100644 tools/testing/selftests/membarrier/litmus/sched-in-no-mb.litmus
 create mode 100644 tools/testing/selftests/membarrier/litmus/sched-in-release.litmus
 create mode 100644 tools/testing/selftests/membarrier/litmus/sched-in.litmus
 create mode 100644 tools/testing/selftests/membarrier/litmus/sched-out-no-exit-mb.litmus
 create mode 100644 tools/testing/selftests/membarrier/litmus/sched-out-no-sched-mb.litmus
 create mode 100644 tools/testing/selftests/membarrier/litmus/sched-out.litmus

diff --git a/kernel/sched/membarrier.c b/kernel/sched/membarrier.c
index 46e823a3d668..5dd88ac0957b 100644
--- a/kernel/sched/membarrier.c
+++ b/kernel/sched/membarrier.c
@@ -133,6 +133,9 @@
  * Using the scenario from (A), we can show that (a) needs to be paired
  * with (g). Using the scenario from (B), we can show that (c) needs to
  * be paired with (d).
+ *
+ * The litmus tests in tools/testing/selftests/membarrier/litmus check
+ * these scenarios with the Linux kernel memory model.
  */
 
 /*
diff --git a/tools/testing/selftests/membarrier/litmus/README b/tools/testing/selftests/membarrier/litmus/README
new file mode 100644
index 000000000000..c9ae572a9d43
--- /dev/null
+++ b/tools/testing/selftests/membarrier/litmus/README
@@ -0,0 +1,52 @@
+Litmus tests of the membarrier system call. They check the memory
+ordering scenarios of kernel/sched/membarrier.c with the Linux kernel
+memory model. Each test with the result "Never" models a case of
+these scenarios. A test with the result "Sometimes" drops one
+barrier, to show that the scenario needs it.
+
+In each test, P0 is the caller of membarrier(). P1 is a CPU which runs
+a thread of the target process.
+
+The user-space accesses of the thread are WRITE_ONCE() and READ_ONCE()
+in program order, without a memory barrier between them. This is what
+barrier() gives in user-space code.
+
+The model has no interrupt. The IPI is modeled as a request and an
+acknowledgment: the caller stores "req", the handler loads it, issues
+its memory barrier and stores "ack", and the caller loads "ack". The
+"exists" clause of each test keeps the executions where this exchange
+took place.
+
+Run them from tools/memory-model, with herd7 installed:
+
+  cd tools/memory-model
+  for f in ../testing/selftests/membarrier/litmus/*.litmus; do
+	herd7 -conf linux-kernel.cfg $f | grep Observation
+  done
+
+Each observation must match the "Result" in the comment of its test.
+
+The thread is interrupted by the IPI:
+
+  ipi-before-thread.litmus, ipi-before-thread-no-entry-mb.litmus,
+  ipi-before-thread-no-handler-mb.litmus
+    Scenario (A). The thread runs after the IPI. It sees the stores
+    which the caller did before membarrier().
+
+  ipi-after-thread.litmus, ipi-after-thread-no-exit-mb.litmus,
+  ipi-after-thread-no-handler-mb.litmus
+    Scenario (B). The thread ran before the IPI. The caller sees its
+    stores after membarrier().
+
+The caller sees another task in rq->curr and sends no IPI:
+
+  sched-in.litmus, sched-in-no-entry-mb.litmus, sched-in-no-mb.litmus,
+  sched-in-release.litmus
+    Scenarios (A) and (C). The thread is scheduled in. The scheduler
+    issues a full barrier after the store to rq->curr. An
+    smp_store_release() which is not a full barrier is not enough.
+
+  sched-out.litmus, sched-out-no-exit-mb.litmus,
+  sched-out-no-sched-mb.litmus
+    Scenarios (B) and (C). The thread is scheduled out. The scheduler
+    issues a full barrier before the store to rq->curr.
diff --git a/tools/testing/selftests/membarrier/litmus/ipi-after-thread-no-exit-mb.litmus b/tools/testing/selftests/membarrier/litmus/ipi-after-thread-no-exit-mb.litmus
new file mode 100644
index 000000000000..fb67beaba996
--- /dev/null
+++ b/tools/testing/selftests/membarrier/litmus/ipi-after-thread-no-exit-mb.litmus
@@ -0,0 +1,36 @@
+C ipi-after-thread-no-exit-mb
+
+(*
+ * Result: Sometimes
+ *
+ * ipi-after-thread without the barrier (c) at the end of membarrier().
+ * The load of x by P0 can then be done before the IPI completes.
+ *)
+
+{}
+
+P0(int *x, int *y, int *req, int *ack)
+{
+	int r_ack;
+	int r1;
+	int r2;
+
+	r2 = READ_ONCE(*y);
+	smp_mb();
+	WRITE_ONCE(*req, 1);
+	r_ack = READ_ONCE(*ack);
+	r1 = READ_ONCE(*x);
+}
+
+P1(int *x, int *y, int *req, int *ack)
+{
+	int r_req;
+
+	WRITE_ONCE(*x, 1);
+	WRITE_ONCE(*y, 1);
+	r_req = READ_ONCE(*req);
+	smp_mb();
+	WRITE_ONCE(*ack, 1);
+}
+
+exists (0:r2=1 /\ 1:r_req=1 /\ 0:r_ack=1 /\ 0:r1=0)
diff --git a/tools/testing/selftests/membarrier/litmus/ipi-after-thread-no-handler-mb.litmus b/tools/testing/selftests/membarrier/litmus/ipi-after-thread-no-handler-mb.litmus
new file mode 100644
index 000000000000..687e883850a9
--- /dev/null
+++ b/tools/testing/selftests/membarrier/litmus/ipi-after-thread-no-handler-mb.litmus
@@ -0,0 +1,36 @@
+C ipi-after-thread-no-handler-mb
+
+(*
+ * Result: Sometimes
+ *
+ * ipi-after-thread without the barrier in the IPI handler. The store to
+ * x by P1 can then be delayed past the acknowledgment.
+ *)
+
+{}
+
+P0(int *x, int *y, int *req, int *ack)
+{
+	int r_ack;
+	int r1;
+	int r2;
+
+	r2 = READ_ONCE(*y);
+	smp_mb();
+	WRITE_ONCE(*req, 1);
+	r_ack = READ_ONCE(*ack);
+	smp_mb();
+	r1 = READ_ONCE(*x);
+}
+
+P1(int *x, int *y, int *req, int *ack)
+{
+	int r_req;
+
+	WRITE_ONCE(*x, 1);
+	WRITE_ONCE(*y, 1);
+	r_req = READ_ONCE(*req);
+	WRITE_ONCE(*ack, 1);
+}
+
+exists (0:r2=1 /\ 1:r_req=1 /\ 0:r_ack=1 /\ 0:r1=0)
diff --git a/tools/testing/selftests/membarrier/litmus/ipi-after-thread.litmus b/tools/testing/selftests/membarrier/litmus/ipi-after-thread.litmus
new file mode 100644
index 000000000000..1145faf62b6d
--- /dev/null
+++ b/tools/testing/selftests/membarrier/litmus/ipi-after-thread.litmus
@@ -0,0 +1,47 @@
+C ipi-after-thread
+
+(*
+ * Result: Never
+ *
+ * Scenario (B) of kernel/sched/membarrier.c.
+ *
+ * P1 is a user-space thread which runs "x = 1; y = 1" in program order,
+ * without a memory barrier. The IPI of membarrier() interrupts it after
+ * that. P0 reads y before membarrier() and x after it.
+ *
+ * req and ack model the IPI, see README.
+ *
+ * The barrier in the IPI handler orders both stores before the
+ * acknowledgment. The barrier (c) at the end of membarrier() orders the
+ * acknowledgment before "r1 = x". P0 thus sees x == 1. This holds
+ * whatever P0 read from y: y is kept to match the scenario.
+ *)
+
+{}
+
+P0(int *x, int *y, int *req, int *ack)
+{
+	int r_ack;
+	int r1;
+	int r2;
+
+	r2 = READ_ONCE(*y);
+	smp_mb();
+	WRITE_ONCE(*req, 1);
+	r_ack = READ_ONCE(*ack);
+	smp_mb();
+	r1 = READ_ONCE(*x);
+}
+
+P1(int *x, int *y, int *req, int *ack)
+{
+	int r_req;
+
+	WRITE_ONCE(*x, 1);
+	WRITE_ONCE(*y, 1);
+	r_req = READ_ONCE(*req);
+	smp_mb();
+	WRITE_ONCE(*ack, 1);
+}
+
+exists (0:r2=1 /\ 1:r_req=1 /\ 0:r_ack=1 /\ 0:r1=0)
diff --git a/tools/testing/selftests/membarrier/litmus/ipi-before-thread-no-entry-mb.litmus b/tools/testing/selftests/membarrier/litmus/ipi-before-thread-no-entry-mb.litmus
new file mode 100644
index 000000000000..9685e34e224a
--- /dev/null
+++ b/tools/testing/selftests/membarrier/litmus/ipi-before-thread-no-entry-mb.litmus
@@ -0,0 +1,37 @@
+C ipi-before-thread-no-entry-mb
+
+(*
+ * Result: Sometimes
+ *
+ * ipi-before-thread without the barrier (a) at the start of
+ * membarrier(). The store to x can be delayed past the IPI. P1 and P0
+ * then both read 0.
+ *)
+
+{}
+
+P0(int *x, int *y, int *req, int *ack)
+{
+	int r_ack;
+	int r2;
+
+	WRITE_ONCE(*x, 1);
+	WRITE_ONCE(*req, 1);
+	r_ack = READ_ONCE(*ack);
+	smp_mb();
+	r2 = READ_ONCE(*y);
+}
+
+P1(int *x, int *y, int *req, int *ack)
+{
+	int r_req;
+	int r1;
+
+	r_req = READ_ONCE(*req);
+	smp_mb();
+	WRITE_ONCE(*ack, 1);
+	WRITE_ONCE(*y, 1);
+	r1 = READ_ONCE(*x);
+}
+
+exists (1:r_req=1 /\ 0:r_ack=1 /\ 1:r1=0 /\ 0:r2=0)
diff --git a/tools/testing/selftests/membarrier/litmus/ipi-before-thread-no-handler-mb.litmus b/tools/testing/selftests/membarrier/litmus/ipi-before-thread-no-handler-mb.litmus
new file mode 100644
index 000000000000..1559bce76ea5
--- /dev/null
+++ b/tools/testing/selftests/membarrier/litmus/ipi-before-thread-no-handler-mb.litmus
@@ -0,0 +1,37 @@
+C ipi-before-thread-no-handler-mb
+
+(*
+ * Result: Sometimes
+ *
+ * ipi-before-thread without the barrier in the IPI handler. The load of
+ * x by P1 can then be done before the handler loads req. This is why
+ * ipi_mb() issues smp_mb() rather than rely on the interrupt alone.
+ *)
+
+{}
+
+P0(int *x, int *y, int *req, int *ack)
+{
+	int r_ack;
+	int r2;
+
+	WRITE_ONCE(*x, 1);
+	smp_mb();
+	WRITE_ONCE(*req, 1);
+	r_ack = READ_ONCE(*ack);
+	smp_mb();
+	r2 = READ_ONCE(*y);
+}
+
+P1(int *x, int *y, int *req, int *ack)
+{
+	int r_req;
+	int r1;
+
+	r_req = READ_ONCE(*req);
+	WRITE_ONCE(*ack, 1);
+	WRITE_ONCE(*y, 1);
+	r1 = READ_ONCE(*x);
+}
+
+exists (1:r_req=1 /\ 0:r_ack=1 /\ 1:r1=0 /\ 0:r2=0)
diff --git a/tools/testing/selftests/membarrier/litmus/ipi-before-thread.litmus b/tools/testing/selftests/membarrier/litmus/ipi-before-thread.litmus
new file mode 100644
index 000000000000..e21e79e2d2ad
--- /dev/null
+++ b/tools/testing/selftests/membarrier/litmus/ipi-before-thread.litmus
@@ -0,0 +1,46 @@
+C ipi-before-thread
+
+(*
+ * Result: Never
+ *
+ * Scenario (A) of kernel/sched/membarrier.c.
+ *
+ * P0 calls membarrier(). P1 is a user-space thread which the IPI
+ * interrupts before it runs "y = 1; r1 = x". Its two accesses are in
+ * program order, without a memory barrier.
+ *
+ * req and ack model the IPI, see README.
+ *
+ * The barrier (a) at the start of membarrier() orders "x = 1" before
+ * the IPI. P1 thus sees x == 1 after the IPI. This holds whatever P0
+ * reads from y: y is kept to match the scenario.
+ *)
+
+{}
+
+P0(int *x, int *y, int *req, int *ack)
+{
+	int r_ack;
+	int r2;
+
+	WRITE_ONCE(*x, 1);
+	smp_mb();
+	WRITE_ONCE(*req, 1);
+	r_ack = READ_ONCE(*ack);
+	smp_mb();
+	r2 = READ_ONCE(*y);
+}
+
+P1(int *x, int *y, int *req, int *ack)
+{
+	int r_req;
+	int r1;
+
+	r_req = READ_ONCE(*req);
+	smp_mb();
+	WRITE_ONCE(*ack, 1);
+	WRITE_ONCE(*y, 1);
+	r1 = READ_ONCE(*x);
+}
+
+exists (1:r_req=1 /\ 0:r_ack=1 /\ 1:r1=0 /\ 0:r2=0)
diff --git a/tools/testing/selftests/membarrier/litmus/sched-in-no-entry-mb.litmus b/tools/testing/selftests/membarrier/litmus/sched-in-no-entry-mb.litmus
new file mode 100644
index 000000000000..a40837fad005
--- /dev/null
+++ b/tools/testing/selftests/membarrier/litmus/sched-in-no-entry-mb.litmus
@@ -0,0 +1,34 @@
+C sched-in-no-entry-mb
+
+(*
+ * Result: Sometimes
+ *
+ * sched-in without the barrier (a) at the start of membarrier(). The
+ * store to x can be delayed past the load of rq->curr. P0 sees no
+ * thread and sends no IPI. The thread reads x == 0.
+ *)
+
+{}
+
+P0(int *x, int *y, int *curr)
+{
+	int r_curr;
+	int r2;
+
+	WRITE_ONCE(*x, 1);
+	r_curr = READ_ONCE(*curr);
+	smp_mb();
+	r2 = READ_ONCE(*y);
+}
+
+P1(int *x, int *y, int *curr)
+{
+	int r1;
+
+	WRITE_ONCE(*curr, 1);
+	smp_mb();
+	WRITE_ONCE(*y, 1);
+	r1 = READ_ONCE(*x);
+}
+
+exists (0:r_curr=0 /\ 1:r1=0 /\ 0:r2=0)
diff --git a/tools/testing/selftests/membarrier/litmus/sched-in-no-mb.litmus b/tools/testing/selftests/membarrier/litmus/sched-in-no-mb.litmus
new file mode 100644
index 000000000000..6840cb26e13d
--- /dev/null
+++ b/tools/testing/selftests/membarrier/litmus/sched-in-no-mb.litmus
@@ -0,0 +1,33 @@
+C sched-in-no-mb
+
+(*
+ * Result: Sometimes
+ *
+ * sched-in without the full barrier after the store to rq->curr. P0
+ * sees no thread and sends no IPI. The thread reads x == 0.
+ *)
+
+{}
+
+P0(int *x, int *y, int *curr)
+{
+	int r_curr;
+	int r2;
+
+	WRITE_ONCE(*x, 1);
+	smp_mb();
+	r_curr = READ_ONCE(*curr);
+	smp_mb();
+	r2 = READ_ONCE(*y);
+}
+
+P1(int *x, int *y, int *curr)
+{
+	int r1;
+
+	WRITE_ONCE(*curr, 1);
+	WRITE_ONCE(*y, 1);
+	r1 = READ_ONCE(*x);
+}
+
+exists (0:r_curr=0 /\ 1:r1=0 /\ 0:r2=0)
diff --git a/tools/testing/selftests/membarrier/litmus/sched-in-release.litmus b/tools/testing/selftests/membarrier/litmus/sched-in-release.litmus
new file mode 100644
index 000000000000..d2df75b388fe
--- /dev/null
+++ b/tools/testing/selftests/membarrier/litmus/sched-in-release.litmus
@@ -0,0 +1,38 @@
+C sched-in-release
+
+(*
+ * Result: Sometimes
+ *
+ * sched-in with smp_store_release() in place of the full barrier after
+ * the store to rq->curr. This is finish_task() on an architecture where
+ * smp_store_release() is weaker than a full barrier. It does not order
+ * the store to rq->curr before the loads of the thread.
+ *)
+
+{
+	int on_cpu = 1;
+}
+
+P0(int *x, int *y, int *curr)
+{
+	int r_curr;
+	int r2;
+
+	WRITE_ONCE(*x, 1);
+	smp_mb();
+	r_curr = READ_ONCE(*curr);
+	smp_mb();
+	r2 = READ_ONCE(*y);
+}
+
+P1(int *x, int *y, int *curr, int *on_cpu)
+{
+	int r1;
+
+	WRITE_ONCE(*curr, 1);
+	smp_store_release(on_cpu, 0);
+	WRITE_ONCE(*y, 1);
+	r1 = READ_ONCE(*x);
+}
+
+exists (0:r_curr=0 /\ 1:r1=0 /\ 0:r2=0)
diff --git a/tools/testing/selftests/membarrier/litmus/sched-in.litmus b/tools/testing/selftests/membarrier/litmus/sched-in.litmus
new file mode 100644
index 000000000000..c89e2c6c4554
--- /dev/null
+++ b/tools/testing/selftests/membarrier/litmus/sched-in.litmus
@@ -0,0 +1,51 @@
+C sched-in
+
+(*
+ * Result: Never
+ *
+ * Scenarios (A) and (C) of kernel/sched/membarrier.c: a thread of the
+ * target process is scheduled in while P0 calls membarrier().
+ *
+ * curr stands for rq->curr of the CPU of P1: 0 for another task, 1 for
+ * the thread. P0 loads it between its two barriers. It sends no IPI
+ * when it reads 0, which is the case the "exists" clause keeps.
+ *
+ * P1 is the scheduler, then the thread in user-space. The scheduler
+ * stores to rq->curr, then issues a full barrier before the return to
+ * user-space. Each architecture provides this barrier its own way, see
+ * Documentation/scheduler/membarrier.rst.
+ *
+ * Either P0 sees the thread and sends the IPI, or the thread sees
+ * x == 1. This holds whatever P0 reads from y: y is kept to match the
+ * scenario.
+ *
+ * The same argument holds with tsk->mm in place of rq->curr. It covers
+ * kthread_use_mm(), scenario (E), and the icache_stale_mask of the
+ * RISC-V sync core command.
+ *)
+
+{}
+
+P0(int *x, int *y, int *curr)
+{
+	int r_curr;
+	int r2;
+
+	WRITE_ONCE(*x, 1);
+	smp_mb();
+	r_curr = READ_ONCE(*curr);
+	smp_mb();
+	r2 = READ_ONCE(*y);
+}
+
+P1(int *x, int *y, int *curr)
+{
+	int r1;
+
+	WRITE_ONCE(*curr, 1);
+	smp_mb();
+	WRITE_ONCE(*y, 1);
+	r1 = READ_ONCE(*x);
+}
+
+exists (0:r_curr=0 /\ 1:r1=0 /\ 0:r2=0)
diff --git a/tools/testing/selftests/membarrier/litmus/sched-out-no-exit-mb.litmus b/tools/testing/selftests/membarrier/litmus/sched-out-no-exit-mb.litmus
new file mode 100644
index 000000000000..e4bb1ee7a935
--- /dev/null
+++ b/tools/testing/selftests/membarrier/litmus/sched-out-no-exit-mb.litmus
@@ -0,0 +1,34 @@
+C sched-out-no-exit-mb
+
+(*
+ * Result: Sometimes
+ *
+ * sched-out without the barrier (c) at the end of membarrier(). The
+ * load of x by P0 can then be done before it loads rq->curr.
+ *)
+
+{
+	int curr = 1;
+}
+
+P0(int *x, int *y, int *curr)
+{
+	int r_curr;
+	int r1;
+	int r2;
+
+	r2 = READ_ONCE(*y);
+	smp_mb();
+	r_curr = READ_ONCE(*curr);
+	r1 = READ_ONCE(*x);
+}
+
+P1(int *x, int *y, int *curr)
+{
+	WRITE_ONCE(*x, 1);
+	WRITE_ONCE(*y, 1);
+	smp_mb();
+	WRITE_ONCE(*curr, 0);
+}
+
+exists (0:r2=1 /\ 0:r_curr=0 /\ 0:r1=0)
diff --git a/tools/testing/selftests/membarrier/litmus/sched-out-no-sched-mb.litmus b/tools/testing/selftests/membarrier/litmus/sched-out-no-sched-mb.litmus
new file mode 100644
index 000000000000..7b690db2b4be
--- /dev/null
+++ b/tools/testing/selftests/membarrier/litmus/sched-out-no-sched-mb.litmus
@@ -0,0 +1,34 @@
+C sched-out-no-sched-mb
+
+(*
+ * Result: Sometimes
+ *
+ * sched-out without the full barrier before the store to rq->curr. The
+ * store to x by the thread can then be delayed past it.
+ *)
+
+{
+	int curr = 1;
+}
+
+P0(int *x, int *y, int *curr)
+{
+	int r_curr;
+	int r1;
+	int r2;
+
+	r2 = READ_ONCE(*y);
+	smp_mb();
+	r_curr = READ_ONCE(*curr);
+	smp_mb();
+	r1 = READ_ONCE(*x);
+}
+
+P1(int *x, int *y, int *curr)
+{
+	WRITE_ONCE(*x, 1);
+	WRITE_ONCE(*y, 1);
+	WRITE_ONCE(*curr, 0);
+}
+
+exists (0:r2=1 /\ 0:r_curr=0 /\ 0:r1=0)
diff --git a/tools/testing/selftests/membarrier/litmus/sched-out.litmus b/tools/testing/selftests/membarrier/litmus/sched-out.litmus
new file mode 100644
index 000000000000..8fb1532bd13c
--- /dev/null
+++ b/tools/testing/selftests/membarrier/litmus/sched-out.litmus
@@ -0,0 +1,49 @@
+C sched-out
+
+(*
+ * Result: Never
+ *
+ * Scenarios (B) and (C) of kernel/sched/membarrier.c: a thread of the
+ * target process is scheduled out while P0 calls membarrier().
+ *
+ * curr stands for rq->curr of the CPU of P1: 1 for the thread, 0 for
+ * another task. P0 sends no IPI when it reads 0, which is the case the
+ * "exists" clause keeps.
+ *
+ * P1 is the thread in user-space, which runs "x = 1; y = 1" in program
+ * order. Then the scheduler issues a full barrier before it stores to
+ * rq->curr: smp_mb__after_spinlock() in __schedule().
+ *
+ * When P0 saw y == 1 and sees that the thread is gone, it sees x == 1.
+ * This holds whatever P0 read from y: y is kept to match the scenario.
+ *
+ * The same argument holds with tsk->mm in place of rq->curr. It covers
+ * exit_mm(), scenario (D), and kthread_unuse_mm(), scenario (E).
+ *)
+
+{
+	int curr = 1;
+}
+
+P0(int *x, int *y, int *curr)
+{
+	int r_curr;
+	int r1;
+	int r2;
+
+	r2 = READ_ONCE(*y);
+	smp_mb();
+	r_curr = READ_ONCE(*curr);
+	smp_mb();
+	r1 = READ_ONCE(*x);
+}
+
+P1(int *x, int *y, int *curr)
+{
+	WRITE_ONCE(*x, 1);
+	WRITE_ONCE(*y, 1);
+	smp_mb();
+	WRITE_ONCE(*curr, 0);
+}
+
+exists (0:r2=1 /\ 0:r_curr=0 /\ 0:r1=0)
-- 
2.47.3


_______________________________________________
linux-riscv mailing list
linux-riscv@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-riscv

[RFC PATCH 14/16] MAINTAINERS: Add the membarrier selftests to MEMBARRIER SUPPORT

[Mathieu Desnoyers]

The MEMBARRIER SUPPORT entry does not list the membarrier selftests.
get_maintainer.pl does not report the membarrier maintainers for a
change to these tests.

Add tools/testing/selftests/membarrier/ to the entry.

Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
---
 MAINTAINERS | 1 +
 1 file changed, 1 insertion(+)

diff --git a/MAINTAINERS b/MAINTAINERS
index 222b7fc2c3aa..2ee220ccc5e8 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -17435,6 +17435,7 @@ F:	arch/*/include/asm/membarrier.h
 F:	arch/*/include/asm/sync_core.h
 F:	include/uapi/linux/membarrier.h
 F:	kernel/sched/membarrier.c
+F:	tools/testing/selftests/membarrier/
 
 MEMBLOCK AND MEMORY MANAGEMENT INITIALIZATION
 M:	Mike Rapoport <rppt@kernel.org>
-- 
2.47.3

[RFC PATCH 14/16] MAINTAINERS: Add the membarrier selftests to MEMBARRIER SUPPORT

[Mathieu Desnoyers]

The MEMBARRIER SUPPORT entry does not list the membarrier selftests.
get_maintainer.pl does not report the membarrier maintainers for a
change to these tests.

Add tools/testing/selftests/membarrier/ to the entry.

Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
---
 MAINTAINERS | 1 +
 1 file changed, 1 insertion(+)

diff --git a/MAINTAINERS b/MAINTAINERS
index 222b7fc2c3aa..2ee220ccc5e8 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -17435,6 +17435,7 @@ F:	arch/*/include/asm/membarrier.h
 F:	arch/*/include/asm/sync_core.h
 F:	include/uapi/linux/membarrier.h
 F:	kernel/sched/membarrier.c
+F:	tools/testing/selftests/membarrier/
 
 MEMBLOCK AND MEMORY MANAGEMENT INITIALIZATION
 M:	Mike Rapoport <rppt@kernel.org>
-- 
2.47.3


_______________________________________________
linux-riscv mailing list
linux-riscv@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-riscv

[RFC PATCH 15/16] selftests/membarrier: Check the memory barriers of each thread

[Mathieu Desnoyers]

An expedited membarrier command promises that each targeted thread
executes a memory barrier during the call. User-space relies on this
to leave the barriers out of its fast paths. The kernel keeps the
promise with memory barriers in the system call, in its IPI handlers
and in the scheduler. If one of them is missing, user-space only sees
a rare memory ordering failure.

membarrier_test_ipi checks this promise through the interrupt
counts of each CPU. It needs its target threads to stay on their CPU,
and it only sees the IPIs. It does not see what the scheduler does.

Add membarrier_test_barrier. It counts, for each thread, the memory
barriers which membarrier relies on. It then checks that an expedited
command makes each targeted thread execute one. The comment at the
top of the file describes the method. No kernel code is added.

The counting works as follows:

 - The test lists the kernel functions which hold the instructions,
   and the kind of instruction to look for in each. It finds them by
   their bytes in the kernel text, which it reads from /proc/kcore.
   The kernel refuses a kprobe which is not on the first byte of an
   instruction, so a wrong match is dropped.

 - A BPF program is attached to each instruction with a kprobe. It
   increments counters of the thread on whose behalf the instruction
   is executed. For a barrier after the store to rq->curr, this is
   the thread which is scheduled in.

 - A BPF program on the sched_switch tracepoint counts two events for
   a thread. It was scheduled out. It was scheduled in after a task
   of another mm or after a kernel thread. The scheduler then owes it
   a barrier after the store to rq->curr. The program counts a miss
   if the thread is scheduled out again without it.

 - The counters are in a BPF array map, which the test maps in
   memory.

The test lists the instructions for x86-64 only. They are the
smp_mb() of the system call and of the IPI handlers, and the barriers
of switch_mm_irqs_off() and finish_task_switch(). This list depends
on what the compiler inlines into those functions.

The kernel turns a kprobe into a jump when it can. The handler of such
a probe runs with interrupts enabled. The probe of an IPI handler
which interrupts it is then skipped. So the test writes 0 to
/proc/sys/debug/kprobes-optimization while it runs. It restores the
value when it exits.

Each check of a target works as follows:

 - The caller pauses for one microsecond between two calls.
 - The target reads its counters during two pauses. It can then tell
   that exactly one call ran in between.
 - The IPI counter of the target must have changed.
 - A check where the target was scheduled out is dropped: the
   scheduler covers that target.

The test cases are those of membarrier_test_ipi. Each of them also
checks the two memory barriers of the system call, and the barrier of
the scheduler after the store to rq->curr.

The thread exit and lazy TLB test cases fail without commit
"sched/membarrier: Fix stale runqueue state after a lazy mm switch".

The test needs root. It also needs a kernel with BPF, BTF, kprobes and
/proc/kcore. It is skipped otherwise. The Makefile builds it only when
clang is found. membarrier_test_ipi stays: it needs none of this.

Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
---
 tools/testing/selftests/membarrier/.gitignore |    3 +
 tools/testing/selftests/membarrier/Makefile   |   18 +-
 tools/testing/selftests/membarrier/config     |    9 +
 .../membarrier/membarrier_test_barrier.bpf.c  |  168 +++
 .../membarrier/membarrier_test_barrier.c      | 1253 +++++++++++++++++
 .../membarrier/membarrier_test_barrier.h      |   48 +
 6 files changed, 1498 insertions(+), 1 deletion(-)
 create mode 100644 tools/testing/selftests/membarrier/config
 create mode 100644 tools/testing/selftests/membarrier/membarrier_test_barrier.bpf.c
 create mode 100644 tools/testing/selftests/membarrier/membarrier_test_barrier.c
 create mode 100644 tools/testing/selftests/membarrier/membarrier_test_barrier.h

diff --git a/tools/testing/selftests/membarrier/.gitignore b/tools/testing/selftests/membarrier/.gitignore
index 45b995b884a6..161723de5ec5 100644
--- a/tools/testing/selftests/membarrier/.gitignore
+++ b/tools/testing/selftests/membarrier/.gitignore
@@ -4,3 +4,6 @@ membarrier_test_single_thread
 membarrier_rseq_stress
 membarrier_test_ipi
 membarrier_test_cmd
+membarrier_test_barrier
+/tools
+*.bpf.o
diff --git a/tools/testing/selftests/membarrier/Makefile b/tools/testing/selftests/membarrier/Makefile
index c7fa0573e978..ad85f5acd541 100644
--- a/tools/testing/selftests/membarrier/Makefile
+++ b/tools/testing/selftests/membarrier/Makefile
@@ -8,6 +8,22 @@ TEST_GEN_PROGS := membarrier_test_single_thread \
 		membarrier_test_cmd \
 		membarrier_rseq_stress
 
-LOCAL_HDRS += membarrier_test_impl.h
+LOCAL_HDRS += membarrier_test_impl.h membarrier_test_barrier.h
+
+# membarrier_test_barrier loads BPF programs. Build it only when clang
+# is there, so that the other tests do not need it.
+CLANG ?= clang
+ifneq ($(shell command -v $(CLANG) 2>/dev/null),)
+TEST_GEN_PROGS += membarrier_test_barrier
+TEST_GEN_FILES += membarrier_test_barrier.bpf.o
+else
+$(warning membarrier selftests: skipping membarrier_test_barrier, needs clang)
+endif
 
 include ../lib.mk
+include ../net/bpf.mk
+
+$(OUTPUT)/membarrier_test_barrier.bpf.o: membarrier_test_barrier.h
+$(OUTPUT)/membarrier_test_barrier: $(BPFOBJ)
+$(OUTPUT)/membarrier_test_barrier: CFLAGS += -I$(SCRATCH_DIR)/include
+$(OUTPUT)/membarrier_test_barrier: LDLIBS += -lelf -lz
diff --git a/tools/testing/selftests/membarrier/config b/tools/testing/selftests/membarrier/config
new file mode 100644
index 000000000000..721da75da204
--- /dev/null
+++ b/tools/testing/selftests/membarrier/config
@@ -0,0 +1,9 @@
+CONFIG_BPF=y
+CONFIG_BPF_EVENTS=y
+CONFIG_BPF_JIT=y
+CONFIG_BPF_SYSCALL=y
+CONFIG_DEBUG_INFO_BTF=y
+CONFIG_KPROBES=y
+CONFIG_KPROBE_EVENTS=y
+CONFIG_MEMBARRIER=y
+CONFIG_PROC_KCORE=y
diff --git a/tools/testing/selftests/membarrier/membarrier_test_barrier.bpf.c b/tools/testing/selftests/membarrier/membarrier_test_barrier.bpf.c
new file mode 100644
index 000000000000..46cfba13f00f
--- /dev/null
+++ b/tools/testing/selftests/membarrier/membarrier_test_barrier.bpf.c
@@ -0,0 +1,168 @@
+// SPDX-License-Identifier: GPL-2.0
+/*
+ * Count the memory barriers which membarrier relies on, for each
+ * registered thread.
+ *
+ * See membarrier_test_barrier.c.
+ */
+#include <stdbool.h>
+#include <linux/bpf.h>
+#include <bpf/bpf_helpers.h>
+#include <bpf/bpf_tracing.h>
+
+#include "membarrier_test_barrier.h"
+
+struct mm_struct;
+
+/* The fields which this program reads. CO-RE relocates them. */
+struct task_struct {
+	struct mm_struct *mm;
+} __attribute__((preserve_access_index));
+
+struct pt_regs;
+
+/* Slot of a registered thread, plus one. */
+struct {
+	__uint(type, BPF_MAP_TYPE_TASK_STORAGE);
+	__uint(map_flags, BPF_F_NO_PREALLOC);
+	__type(key, int);
+	__type(value, __u32);
+} task_slot SEC(".maps");
+
+/* The counters of each slot. User-space maps them. */
+struct {
+	__uint(type, BPF_MAP_TYPE_ARRAY);
+	__uint(map_flags, BPF_F_MMAPABLE);
+	__uint(max_entries, MBT_MAX_SLOTS);
+	__type(key, __u32);
+	__type(value, struct mbt_counters);
+} counters SEC(".maps");
+
+/* What the scheduler owes to the thread which it switched to. */
+#define OWES_BARRIER	1
+
+struct cpu_state {
+	/* Slot, plus one, of the thread which this CPU last switched to. */
+	__u32 slot;
+	/* OWES_* flags. A probe of the scheduler clears them. */
+	__u32 owed;
+};
+
+struct {
+	__uint(type, BPF_MAP_TYPE_PERCPU_ARRAY);
+	__uint(max_entries, 1);
+	__type(key, __u32);
+	__type(value, struct cpu_state);
+} cpu_state SEC(".maps");
+
+struct {
+	__uint(type, BPF_MAP_TYPE_ARRAY);
+	__uint(max_entries, MBT_MAX_SITES);
+	__type(key, __u32);
+	__type(value, __u64);
+} site_hits SEC(".maps");
+
+/*
+ * Only the CPU which runs a thread, or which switches to it, counts for
+ * that thread. So the increments need no lock.
+ */
+static __always_inline void count(__u32 slot, __u32 mask)
+{
+	struct mbt_counters *c;
+	__u32 key = slot - 1;
+	int i;
+
+	if (!slot)
+		return;
+	c = bpf_map_lookup_elem(&counters, &key);
+	if (!c)
+		return;
+	for (i = 0; i < MBT_NR_COUNTERS; i++)
+		if (mask & (1U << i))
+			c->c[i]++;
+}
+
+static __always_inline __u32 slot_of(struct task_struct *task)
+{
+	__u32 *slot = bpf_task_storage_get(&task_slot, task, 0, 0);
+
+	return slot ? *slot : 0;
+}
+
+/* The attach cookie says which counters this instruction increments. */
+SEC("kprobe")
+int barrier_site(struct pt_regs *ctx)
+{
+	__u64 cookie = bpf_get_attach_cookie(ctx);
+	__u32 site = cookie >> MBT_COOKIE_SITE_SHIFT;
+	__u32 mask = cookie & MBT_COOKIE_MASK;
+	struct cpu_state *state;
+	__u32 zero = 0, slot = 0;
+	__u64 *hits;
+
+	hits = bpf_map_lookup_elem(&site_hits, &site);
+	if (hits)
+		__sync_fetch_and_add(hits, 1);
+	if (cookie & MBT_COOKIE_NEXT) {
+		/* Executed on behalf of the thread which is scheduled in. */
+		state = bpf_map_lookup_elem(&cpu_state, &zero);
+		if (!state)
+			return 0;
+		slot = state->slot;
+		if (mask & (1U << MBT_SWITCH_MM))
+			state->owed &= ~OWES_BARRIER;
+	} else {
+		slot = slot_of(bpf_get_current_task_btf());
+	}
+	count(slot, mask);
+	return 0;
+}
+
+SEC("tp_btf/sched_switch")
+int BPF_PROG(sched_switch, bool preempt, struct task_struct *prev,
+	     struct task_struct *next)
+{
+	struct cpu_state *state;
+	__u32 zero = 0, slot;
+
+	state = bpf_map_lookup_elem(&cpu_state, &zero);
+	if (!state)
+		return 0;
+	/* The CPU leaves the thread which it last switched to. */
+	if (state->owed & OWES_BARRIER)
+		count(state->slot, 1U << MBT_SCHED_IN_MISSED);
+	count(slot_of(prev), 1U << MBT_SCHED_OUT);
+	slot = slot_of(next);
+	state->slot = slot;
+	state->owed = 0;
+	/* The cases which need a barrier after the store to rq->curr. */
+	if (slot && next->mm && prev->mm != next->mm) {
+		state->owed = OWES_BARRIER;
+		count(slot, 1U << MBT_SCHED_IN);
+	}
+	return 0;
+}
+
+/* Give a slot to the calling thread. User-space runs this program. */
+SEC("syscall")
+int register_thread(struct mbt_register *ctx)
+{
+	struct mbt_counters *c;
+	__u32 key = ctx->slot;
+	__u32 *slot;
+	int i;
+
+	c = bpf_map_lookup_elem(&counters, &key);
+	if (!c)
+		return -1;
+	for (i = 0; i < MBT_NR_COUNTERS; i++)
+		c->c[i] = 0;
+	slot = bpf_task_storage_get(&task_slot, bpf_get_current_task_btf(), 0,
+				    BPF_LOCAL_STORAGE_GET_F_CREATE);
+	if (!slot)
+		return -1;
+	*slot = key + 1;
+	return 0;
+}
+
+char _license[] SEC("license") = "GPL";
diff --git a/tools/testing/selftests/membarrier/membarrier_test_barrier.c b/tools/testing/selftests/membarrier/membarrier_test_barrier.c
new file mode 100644
index 000000000000..46502d271242
--- /dev/null
+++ b/tools/testing/selftests/membarrier/membarrier_test_barrier.c
@@ -0,0 +1,1253 @@
+// SPDX-License-Identifier: GPL-2.0
+/*
+ * Check that the expedited membarrier commands make each targeted
+ * thread execute a memory barrier.
+ *
+ * BPF programs count, for each thread, the memory barriers which
+ * membarrier relies on. See membarrier_test_barrier.bpf.c. A kprobe is
+ * attached to each of these instructions. The sched_switch tracepoint
+ * has a program which counts the context switches. The counters are in a BPF array map,
+ * which this program maps in memory.
+ *
+ * A "caller" issues the command in a loop, with a short pause between
+ * two calls. It counts the calls which it started and the calls which
+ * returned. Each "target" thread runs this check in a loop:
+ *
+ *   1. read the count of returned calls,
+ *   2. read its own counters,
+ *   3. read the count of started calls,
+ *   4. wait until one more call has returned,
+ *   5. read its own counters again,
+ *   6. read the count of started calls again.
+ *
+ * The check is exact when the caller was between two calls at step 2
+ * and at step 5. This holds when steps 1 and 3 read the same count,
+ * and step 6 reads one more. Then one call ran between step 2 and
+ * step 5, and no other call overlaps them. So the target must have
+ * executed a memory barrier for that call: its IPI counter must have
+ * changed. Otherwise membarrier missed this thread.
+ *
+ * A check where the target was scheduled out between step 2 and
+ * step 5 is dropped. The scheduler then covers the target, and the IPI
+ * is not required. A check which is not exact is dropped as well.
+ *
+ * The BPF programs also check the scheduler. A target can be scheduled
+ * in after a task of another mm or after a kernel thread. The scheduler
+ * must then execute a memory barrier after the store to rq->curr,
+ * before the target is scheduled out again.
+ *
+ * The instructions are found by their bytes in the kernel text, which
+ * is read from /proc/kcore. The list is for x86-64. It depends on what
+ * the compiler inlines into the functions it names.
+ */
+#define _GNU_SOURCE
+#include <linux/membarrier.h>
+#include <linux/types.h>
+#include <bpf/libbpf.h>
+#include <bpf/bpf.h>
+#include <elf.h>
+#include <sys/mman.h>
+#include <sys/mount.h>
+#include <sys/prctl.h>
+#include <sys/wait.h>
+#include <syscall.h>
+#include <stdatomic.h>
+#include <pthread.h>
+#include <signal.h>
+#include <stdbool.h>
+#include <stdint.h>
+#include <stdlib.h>
+#include <string.h>
+#include <unistd.h>
+#include <fcntl.h>
+#include <sched.h>
+#include <stdio.h>
+#include <errno.h>
+#include <time.h>
+
+#include "kselftest_harness.h"
+#include "membarrier_test_barrier.h"
+
+#define MAX_TARGETS	16
+/* Duration of each test case, in milliseconds. */
+#define RUN_MS		1000
+/* Duration of each round of the lazy TLB test case, in milliseconds. */
+#define ROUND_MS	20
+/* Pause of the caller between two calls, in nanoseconds. */
+#define PAUSE_NS	1000
+/* Bound on step 4, in case the caller is not running. */
+#define MAX_SPINS	1000000
+/* Seconds to wait for the targets of a test case. */
+#define READY_TIMEOUT	10
+#define STACK_SIZE	(64 * 1024)
+
+struct target_stats {
+	/* Exact checks where the IPI covered the target. */
+	atomic_ulong by_ipi;
+	/* Checks dropped because the target was scheduled out. */
+	atomic_ulong by_sched;
+	/* Checks where the target executed no barrier. */
+	atomic_ulong missed;
+	/* Membarrier IPIs which the target received. */
+	atomic_ulong ipis;
+	/* Times the target was scheduled in after a task of another mm. */
+	atomic_ulong sched_in;
+	/* Scheduled in without a barrier after the store to rq->curr. */
+	atomic_ulong sched_in_missed;
+};
+
+struct shared {
+	atomic_ulong started;
+	atomic_ulong done;
+	/* Calls issued by the caller, and their memory barriers. */
+	atomic_ulong calls;
+	atomic_ulong call_barriers;
+	atomic_int ready;
+	/* The process of the targets is done with its registration. */
+	atomic_int registered;
+	atomic_int stop;
+	atomic_int error;
+	/* Next free slot of counters. */
+	atomic_uint next_slot;
+	/* One more for the test case which adds a target of its own. */
+	struct target_stats stats[MAX_TARGETS + 1];
+};
+
+struct target {
+	int idx;
+	/* Pin the target on this CPU, unless negative. */
+	int cpu;
+	/* Create a thread which exits, every so many checks. */
+	bool thread_exit;
+	/* Run for this duration. Run until the stop flag when 0. */
+	int ms;
+};
+
+static struct shared *shared;
+static int shared_fd;
+
+static int sys_membarrier(int cmd, unsigned int flags, int cpu_id)
+{
+	return syscall(__NR_membarrier, cmd, flags, cpu_id);
+}
+
+static long now_ms(void)
+{
+	struct timespec ts;
+
+	clock_gettime(CLOCK_MONOTONIC, &ts);
+	return ts.tv_sec * 1000 + ts.tv_nsec / 1000000;
+}
+
+static long long now_ns(void)
+{
+	struct timespec ts;
+
+	clock_gettime(CLOCK_MONOTONIC, &ts);
+	return ts.tv_sec * 1000000000LL + ts.tv_nsec;
+}
+
+static int pin_to_cpu(int cpu)
+{
+	cpu_set_t set;
+
+	CPU_ZERO(&set);
+	CPU_SET(cpu, &set);
+	return sched_setaffinity(0, sizeof(set), &set);
+}
+
+/* Store the allowed CPUs in @cpus. Return their count, up to @max. */
+static int allowed_cpus(int *cpus, int max)
+{
+	cpu_set_t set;
+	int cpu, nr = 0;
+
+	if (sched_getaffinity(0, sizeof(set), &set))
+		return 0;
+	for (cpu = 0; cpu < CPU_SETSIZE && nr < max; cpu++)
+		if (CPU_ISSET(cpu, &set))
+			cpus[nr++] = cpu;
+	return nr;
+}
+
+/* One target per allowed CPU. With the caller, the CPUs are overcommitted. */
+static int nr_targets(void)
+{
+	int cpus[MAX_TARGETS];
+
+	return allowed_cpus(cpus, MAX_TARGETS);
+}
+
+#define BPF_OBJECT	"membarrier_test_barrier.bpf.o"
+#define COUNTERS_SIZE	(MBT_MAX_SLOTS * sizeof(struct mbt_counters))
+#define OPTPROBES_FILE	"/proc/sys/debug/kprobes-optimization"
+
+/* Kind of instruction to probe within a function. */
+enum match {
+	/* smp_mb(): lock addl $0, -4(%rsp) */
+	MATCH_MB,
+	/* Write to CR3. */
+	MATCH_CR3,
+	/* Instruction with a LOCK prefix. */
+	MATCH_LOCK,
+};
+
+struct site {
+	/* A compiler suffix such as ".isra.0" is accepted after this name. */
+	const char *symbol;
+	enum match match;
+	/* Mask of the counters to increment. */
+	unsigned int counters;
+	/* Count for the thread which is scheduled in, not for current. */
+	bool next;
+};
+
+#define IPI		(1U << MBT_IPI)
+#define SYSCALL		(1U << MBT_SYSCALL)
+#define SWITCH_MM	(1U << MBT_SWITCH_MM)
+
+static const struct site sites[] = {
+	{ "ipi_mb", MATCH_MB, IPI },
+	{ "ipi_sync_core", MATCH_MB, IPI },
+	{ "ipi_rseq", MATCH_MB, IPI },
+	{ "membarrier_global_expedited", MATCH_MB, SYSCALL },
+	{ "membarrier_private_expedited", MATCH_MB, SYSCALL },
+	/* Switch to another mm: write to CR3. It is a full barrier. */
+	{ "switch_mm_irqs_off", MATCH_CR3, SWITCH_MM, true },
+	/* Switch to the mm which the CPU kept in lazy TLB mode: smp_mb(). */
+	{ "switch_mm_irqs_off", MATCH_MB, SWITCH_MM, true },
+	/*
+	 * Switch from a kernel thread: mmdrop_lazy_tlb_sched(). Any other
+	 * locked instruction of this function counts as well.
+	 */
+	{ "finish_task_switch", MATCH_LOCK, SWITCH_MM, true },
+};
+
+struct probe {
+	const struct site *site;
+	struct bpf_link *link;
+	char symbol[64];
+	unsigned long offset;
+};
+
+struct ksym {
+	unsigned long addr;
+	char name[64];
+};
+
+static struct probe probes[MBT_MAX_SITES];
+static int nr_probes;
+/* Probes of each site. */
+static int site_probes[ARRAY_SIZE(sites)];
+/* The setup failed, and not because the system lacks a feature. */
+static bool setup_broken;
+static bool too_many_probes;
+static struct bpf_object *bpf_obj;
+static struct bpf_program *site_prog;
+static struct bpf_link *switch_link;
+static const struct mbt_counters *all_counters;
+static int register_fd = -1, counters_fd = -1, hits_fd = -1;
+static int kcore_fd = -1;
+static char optprobes_saved;
+static pid_t setup_pid;
+
+static int libbpf_quiet(enum libbpf_print_level level, const char *fmt,
+			va_list args)
+{
+	return 0;
+}
+
+/* Read @len bytes of kernel memory at @addr from /proc/kcore. */
+static bool read_kernel(unsigned long addr, void *buf, size_t len)
+{
+	Elf64_Ehdr ehdr;
+	Elf64_Phdr phdr;
+	int i;
+
+	if (pread(kcore_fd, &ehdr, sizeof(ehdr), 0) != sizeof(ehdr))
+		return false;
+	for (i = 0; i < ehdr.e_phnum; i++) {
+		if (pread(kcore_fd, &phdr, sizeof(phdr),
+			  ehdr.e_phoff + i * sizeof(phdr)) != sizeof(phdr))
+			return false;
+		if (phdr.p_type != PT_LOAD || addr < phdr.p_vaddr ||
+		    addr + len > phdr.p_vaddr + phdr.p_memsz)
+			continue;
+		return pread(kcore_fd, buf, len,
+			     phdr.p_offset + addr - phdr.p_vaddr) == (ssize_t)len;
+	}
+	return false;
+}
+
+/*
+ * Probe the instruction at @addr. The kernel refuses an address which
+ * is not the first byte of an instruction. Return false in that case.
+ */
+static bool probe_insn(const struct site *site, const struct ksym *sym,
+		       unsigned long addr)
+{
+	LIBBPF_OPTS(bpf_kprobe_opts, opts);
+	struct probe *probe = &probes[nr_probes];
+
+	if (nr_probes == MBT_MAX_SITES) {
+		too_many_probes = true;
+		return false;
+	}
+	opts.offset = addr;
+	opts.bpf_cookie = site->counters | (site->next ? MBT_COOKIE_NEXT : 0) |
+			  (nr_probes << MBT_COOKIE_SITE_SHIFT);
+	probe->link = bpf_program__attach_kprobe_opts(site_prog, NULL, &opts);
+	if (!probe->link)
+		return false;
+	probe->site = site;
+	site_probes[site - sites]++;
+	probe->offset = addr - sym->addr;
+	snprintf(probe->symbol, sizeof(probe->symbol), "%s", sym->name);
+	nr_probes++;
+	return true;
+}
+
+static const unsigned char smp_mb_insn[] = { 0xf0, 0x83, 0x44, 0x24, 0xfc, 0x00 };
+
+/*
+ * Find the instructions of a site by their bytes, in the @size bytes
+ * of the function @sym. A match which is not an instruction is
+ * rejected by probe_insn().
+ */
+static void scan_function(const struct site *site, const struct ksym *sym,
+			  size_t size)
+{
+	unsigned char *text;
+	size_t i, left;
+
+	text = malloc(size);
+	if (!text || !read_kernel(sym->addr, text, size)) {
+		free(text);
+		return;
+	}
+	for (i = 0; i < size; i++) {
+		unsigned long addr = sym->addr + i;
+		unsigned char *insn = text + i;
+
+		left = size - i;
+		switch (site->match) {
+		case MATCH_MB:
+			if (left >= sizeof(smp_mb_insn) &&
+			    !memcmp(insn, smp_mb_insn, sizeof(smp_mb_insn)))
+				probe_insn(site, sym, addr);
+			break;
+		case MATCH_CR3:
+			/* mov %reg, %cr3 is 0f 22 /3, after a REX prefix or not. */
+			if (left < 3 || insn[0] != 0x0f || insn[1] != 0x22 ||
+			    (insn[2] & 0xf8) != 0xd8)
+				break;
+			if (i && (insn[-1] & 0xf0) == 0x40 &&
+			    probe_insn(site, sym, addr - 1))
+				break;
+			probe_insn(site, sym, addr);
+			break;
+		case MATCH_LOCK:
+			if (insn[0] == 0xf0)
+				probe_insn(site, sym, addr);
+			break;
+		}
+	}
+	free(text);
+}
+
+static int ksym_cmp(const void *a, const void *b)
+{
+	const struct ksym *ka = a, *kb = b;
+
+	return ka->addr < kb->addr ? -1 : ka->addr > kb->addr;
+}
+
+/*
+ * Probe the instructions of each site. Return NULL, or the reason why
+ * a site has no probe.
+ */
+static const char *probe_sites(void)
+{
+	struct ksym *syms = NULL;
+	size_t nr = 0, max = 0, i, j, k;
+	char line[256], type;
+	FILE *f;
+
+	f = fopen("/proc/kallsyms", "r");
+	if (!f)
+		return "cannot read /proc/kallsyms";
+	while (fgets(line, sizeof(line), f)) {
+		struct ksym sym;
+
+		if (sscanf(line, "%lx %c %63s", &sym.addr, &type, sym.name) != 3)
+			continue;
+		if (!strchr("tTwW", type))
+			continue;
+		if (nr == max) {
+			struct ksym *more;
+
+			max = max ? 2 * max : 1 << 16;
+			more = realloc(syms, max * sizeof(*syms));
+			if (!more) {
+				free(syms);
+				fclose(f);
+				setup_broken = true;
+				return "out of memory";
+			}
+			syms = more;
+		}
+		syms[nr++] = sym;
+	}
+	fclose(f);
+	qsort(syms, nr, sizeof(*syms), ksym_cmp);
+	for (i = 0; i < nr; i++) {
+		/* The function ends where the next symbol starts. */
+		for (j = i + 1; j < nr && syms[j].addr == syms[i].addr; j++)
+			;
+		if (j == nr || !syms[i].addr)
+			continue;
+		for (k = 0; k < ARRAY_SIZE(sites); k++) {
+			size_t len = strlen(sites[k].symbol);
+
+			if (strncmp(syms[i].name, sites[k].symbol, len))
+				continue;
+			if (syms[i].name[len] && syms[i].name[len] != '.')
+				continue;
+			scan_function(&sites[k], &syms[i],
+				      syms[j].addr - syms[i].addr);
+		}
+	}
+	free(syms);
+	if (too_many_probes)
+		return "too many barrier instructions in this kernel";
+	for (k = 0; k < ARRAY_SIZE(sites); k++) {
+		static char reason[128];
+
+		if (site_probes[k])
+			continue;
+		snprintf(reason, sizeof(reason),
+			 "cannot probe the instructions of %s()",
+			 sites[k].symbol);
+		return reason;
+	}
+	return NULL;
+}
+
+/*
+ * The kernel turns a kprobe into a jump when it can. The handler of
+ * such a probe runs with interrupts enabled: the probe of an IPI
+ * handler which interrupts it is skipped. So disable this optimization
+ * while the test runs.
+ *
+ * Write @value to the sysctl file. Return false if this fails. A
+ * kernel without the file does not optimize.
+ */
+static bool set_optprobes(char value)
+{
+	int fd = open(OPTPROBES_FILE, O_RDWR);
+	bool ok;
+
+	if (fd < 0)
+		return errno == ENOENT;
+	ok = optprobes_saved || pread(fd, &optprobes_saved, 1, 0) == 1;
+	ok = ok && pwrite(fd, &value, 1, 0) == 1;
+	close(fd);
+	return ok;
+}
+
+/* Restore the sysctl when a signal ends the test. */
+static void restore_optprobes(int sig)
+{
+	if (getpid() == setup_pid && optprobes_saved)
+		set_optprobes(optprobes_saved);
+	signal(sig, SIG_DFL);
+	raise(sig);
+}
+
+static void bpf_teardown(void);
+
+/*
+ * Load the BPF programs and probe the sites. Return NULL, or the error.
+ * The error sets setup_broken unless the system lacks a feature.
+ */
+static const char *bpf_setup(void)
+{
+	struct bpf_program *prog;
+	libbpf_print_fn_t print;
+	struct bpf_map *map;
+	const char *err;
+
+	setup_pid = getpid();
+	atexit(bpf_teardown);
+	if (geteuid())
+		return "this test needs root";
+	if (access("/sys/kernel/btf/vmlinux", R_OK))
+		return "the kernel has no BTF (CONFIG_DEBUG_INFO_BTF)";
+	kcore_fd = open("/proc/kcore", O_RDONLY | O_CLOEXEC);
+	if (kcore_fd < 0)
+		return "cannot read /proc/kcore (CONFIG_PROC_KCORE)";
+	setup_broken = true;
+	bpf_obj = bpf_object__open_file(BPF_OBJECT, NULL);
+	if (!bpf_obj)
+		return "cannot open " BPF_OBJECT;
+	if (bpf_object__load(bpf_obj))
+		return "cannot load the BPF programs";
+	site_prog = bpf_object__find_program_by_name(bpf_obj, "barrier_site");
+	prog = bpf_object__find_program_by_name(bpf_obj, "register_thread");
+	map = bpf_object__find_map_by_name(bpf_obj, "counters");
+	if (!site_prog || !prog || !map)
+		return "incomplete BPF object";
+	register_fd = bpf_program__fd(prog);
+	counters_fd = bpf_map__fd(map);
+	map = bpf_object__find_map_by_name(bpf_obj, "site_hits");
+	hits_fd = map ? bpf_map__fd(map) : -1;
+	/* The exec test case passes these two to the new program. */
+	if (fcntl(register_fd, F_SETFD, 0) || fcntl(counters_fd, F_SETFD, 0))
+		return "fcntl failed";
+	signal(SIGINT, restore_optprobes);
+	signal(SIGTERM, restore_optprobes);
+	signal(SIGHUP, restore_optprobes);
+	if (!set_optprobes('0'))
+		return "cannot write " OPTPROBES_FILE;
+	prog = bpf_object__find_program_by_name(bpf_obj, "sched_switch");
+	switch_link = prog ? bpf_program__attach(prog) : NULL;
+	if (!switch_link)
+		return "cannot attach to the sched_switch tracepoint";
+	/* probe_insn() fails on purpose. Do not print these errors. */
+	setup_broken = false;
+	print = libbpf_set_print(libbpf_quiet);
+	err = probe_sites();
+	libbpf_set_print(print);
+	return err;
+}
+
+/* Remove the probes. Print how many times each one was hit. */
+static void bpf_teardown(void)
+{
+	struct bpf_prog_info prog_info = {};
+	__u32 len = sizeof(prog_info);
+	unsigned long long lost = 0;
+	int i;
+
+	/* Only the process which loaded the programs removes them. */
+	if (getpid() != setup_pid)
+		return;
+
+	for (i = 0; i < nr_probes; i++) {
+		struct bpf_link_info info = {};
+		__u32 info_len = sizeof(info), key = i;
+		__u64 hits = 0;
+
+		bpf_link_get_info_by_fd(bpf_link__fd(probes[i].link), &info,
+					&info_len);
+		bpf_map_lookup_elem(hits_fd, &key, &hits);
+		ksft_print_msg("probe %s+0x%lx counters 0x%x%s hits %llu missed %llu\n",
+			       probes[i].symbol, probes[i].offset,
+			       probes[i].site->counters,
+			       probes[i].site->next ? " (next)" : "",
+			       (unsigned long long)hits,
+			       (unsigned long long)info.perf_event.kprobe.missed);
+		lost += info.perf_event.kprobe.missed;
+		bpf_link__destroy(probes[i].link);
+	}
+	if (site_prog &&
+	    !bpf_prog_get_info_by_fd(bpf_program__fd(site_prog), &prog_info,
+				     &len))
+		lost += prog_info.recursion_misses;
+	/* A lost hit looks like a missing barrier. */
+	if (lost)
+		ksft_print_msg("%llu probe hits were lost: a failure may be spurious\n",
+			       lost);
+	bpf_link__destroy(switch_link);
+	if (optprobes_saved)
+		set_optprobes(optprobes_saved);
+}
+
+static bool map_all_counters(void)
+{
+	void *p = mmap(NULL, COUNTERS_SIZE, PROT_READ, MAP_SHARED, counters_fd,
+		       0);
+
+	all_counters = p == MAP_FAILED ? NULL : p;
+	return all_counters;
+}
+
+/* Give counters to the calling thread. Return them, or NULL. */
+static const uint64_t *map_counters(void)
+{
+	struct mbt_register arg = {
+		.slot = atomic_fetch_add(&shared->next_slot, 1),
+	};
+	LIBBPF_OPTS(bpf_test_run_opts, opts,
+		    .ctx_in = &arg, .ctx_size_in = sizeof(arg));
+
+	if (arg.slot >= MBT_MAX_SLOTS) {
+		errno = ENOSPC;
+		return NULL;
+	}
+	if (bpf_prog_test_run_opts(register_fd, &opts) || opts.retval) {
+		if (!errno)
+			errno = EIO;
+		return NULL;
+	}
+	return (const uint64_t *)all_counters[arg.slot].c;
+}
+
+/*
+ * Read a counter. The kernel updates it behind the back of the compiler.
+ * The checks rely on the order of these reads with the other atomic
+ * reads.
+ */
+static uint64_t counter(const uint64_t *counters, int idx)
+{
+	return __atomic_load_n(&counters[idx], __ATOMIC_SEQ_CST);
+}
+
+static void *thread_nop(void *arg)
+{
+	return NULL;
+}
+
+/* Report a failure of the test setup. */
+static void setup_error(const char *what)
+{
+	fprintf(stderr, "# %s: %s\n", what, strerror(errno));
+	atomic_store(&shared->error, 1);
+}
+
+/*
+ * Run one check of a target. See the comment at the top of the file.
+ * Return false if no call returned during the check.
+ */
+static bool check_call(const uint64_t *counters, struct target_stats *stats)
+{
+	unsigned long done = atomic_load(&shared->done);
+	uint64_t ipi = counter(counters, MBT_IPI);
+	uint64_t sched_out = counter(counters, MBT_SCHED_OUT);
+	unsigned long started = atomic_load(&shared->started);
+	bool by_ipi, scheduled;
+	int spins = 0;
+
+	/* A call is in flight. */
+	if (done != started)
+		return false;
+	while (atomic_load(&shared->done) == started) {
+		if (++spins == MAX_SPINS || atomic_load(&shared->stop))
+			return false;
+	}
+	by_ipi = counter(counters, MBT_IPI) != ipi;
+	scheduled = counter(counters, MBT_SCHED_OUT) != sched_out;
+	if (scheduled) {
+		atomic_fetch_add(&stats->by_sched, 1);
+		return true;
+	}
+	/* The next call has started. */
+	if (atomic_load(&shared->started) != started + 1)
+		return true;
+	if (by_ipi)
+		atomic_fetch_add(&stats->by_ipi, 1);
+	else
+		atomic_fetch_add(&stats->missed, 1);
+	return true;
+}
+
+/* Run the checks of a target. */
+static void run_target(const struct target *target)
+{
+	struct target_stats *stats = &shared->stats[target->idx];
+	const uint64_t *counters;
+	long deadline = now_ms() + target->ms;
+	unsigned long checks = 0;
+	uint64_t first_ipi, first_sched_in;
+
+	if (target->cpu >= 0 && pin_to_cpu(target->cpu))
+		setup_error("sched_setaffinity");
+	counters = map_counters();
+	if (!counters) {
+		setup_error("register_thread");
+		atomic_fetch_add(&shared->ready, 1);
+		return;
+	}
+	atomic_fetch_add(&shared->ready, 1);
+	while (!atomic_load(&shared->registered) && !atomic_load(&shared->stop))
+		sched_yield();
+	first_ipi = counter(counters, MBT_IPI);
+	first_sched_in = counter(counters, MBT_SCHED_IN);
+	while (!atomic_load(&shared->stop)) {
+		if (check_call(counters, stats) && target->thread_exit &&
+		    !(++checks % 16)) {
+			pthread_t thread;
+
+			if (pthread_create(&thread, NULL, thread_nop, NULL) ||
+			    pthread_join(thread, NULL))
+				setup_error("thread exit");
+		}
+		if (target->ms && now_ms() >= deadline)
+			break;
+	}
+	atomic_fetch_add(&stats->ipis, counter(counters, MBT_IPI) - first_ipi);
+	atomic_fetch_add(&stats->sched_in,
+			 counter(counters, MBT_SCHED_IN) - first_sched_in);
+	atomic_fetch_add(&stats->sched_in_missed,
+			 counter(counters, MBT_SCHED_IN_MISSED));
+}
+
+static void *target_thread(void *arg)
+{
+	run_target(arg);
+	return NULL;
+}
+
+/*
+ * This task shares the TLS of its parent, errno included. It only makes
+ * system calls and reads shared memory, unless it reports an error.
+ */
+static int target_clone(void *arg)
+{
+	prctl(PR_SET_PDEATHSIG, SIGKILL);
+	run_target(arg);
+	return 0;
+}
+
+static struct target targets[MAX_TARGETS + 1];
+static pthread_t threads[MAX_TARGETS + 1];
+static bool created[MAX_TARGETS + 1];
+
+/* Start @nr target threads, with the stats indexes from @first. */
+static void start_targets(int first, int nr, bool thread_exit)
+{
+	int i;
+
+	for (i = first; i < first + nr; i++) {
+		targets[i] = (struct target) {
+			.idx = i, .cpu = -1, .thread_exit = thread_exit,
+		};
+		created[i] = !pthread_create(&threads[i], NULL, target_thread,
+					     &targets[i]);
+		if (!created[i])
+			setup_error("pthread_create");
+	}
+}
+
+static void join_targets(int first, int nr)
+{
+	int i;
+
+	for (i = first; i < first + nr; i++)
+		if (created[i] && pthread_join(threads[i], NULL))
+			setup_error("pthread_join");
+}
+
+/*
+ * Wait until @nr targets run and their process is done with its
+ * registration. Return false when this does not happen.
+ */
+static bool wait_ready(int nr)
+{
+	time_t start = time(NULL);
+
+	while (atomic_load(&shared->ready) < nr ||
+	       !atomic_load(&shared->registered)) {
+		if (time(NULL) - start > READY_TIMEOUT)
+			return false;
+		sched_yield();
+	}
+	return true;
+}
+
+/*
+ * Issue the command for @ms milliseconds. Count the calls and the
+ * memory barriers which the system call executed. Return 0, or -1
+ * with errno set.
+ */
+static int run_caller(int cmd, unsigned int flags, int cpu_id, int ms)
+{
+	static const uint64_t *counters;
+	long deadline = now_ms() + ms;
+	unsigned long calls = 0;
+	uint64_t barriers;
+	long long pause_end;
+	int i;
+
+	if (!counters)
+		counters = map_counters();
+	if (!counters)
+		return -1;
+	barriers = counter(counters, MBT_SYSCALL);
+	do {
+		for (i = 0; i < 64; i++) {
+			atomic_fetch_add(&shared->started, 1);
+			if (sys_membarrier(cmd, flags, cpu_id))
+				return -1;
+			atomic_fetch_add(&shared->done, 1);
+			calls++;
+			/* Let the targets see that no call is in flight. */
+			pause_end = now_ns() + PAUSE_NS;
+			while (now_ns() < pause_end)
+				;
+		}
+	} while (now_ms() < deadline);
+	atomic_fetch_add(&shared->calls, calls);
+	atomic_fetch_add(&shared->call_barriers,
+			 counter(counters, MBT_SYSCALL) - barriers);
+	return 0;
+}
+
+/* Wait for a child. Return true if it exited with 0. */
+static bool child_ok(pid_t pid)
+{
+	int status;
+
+	return waitpid(pid, &status, 0) == pid && WIFEXITED(status) &&
+	       !WEXITSTATUS(status);
+}
+
+static void sum_stats(int first, int nr, struct target_stats *sum)
+{
+	int i;
+
+	memset(sum, 0, sizeof(*sum));
+	for (i = first; i < first + nr; i++) {
+		struct target_stats *stats = &shared->stats[i];
+
+		sum->by_ipi += atomic_load(&stats->by_ipi);
+		sum->by_sched += atomic_load(&stats->by_sched);
+		sum->missed += atomic_load(&stats->missed);
+		sum->ipis += atomic_load(&stats->ipis);
+		sum->sched_in += atomic_load(&stats->sched_in);
+		sum->sched_in_missed += atomic_load(&stats->sched_in_missed);
+	}
+}
+
+/* The command must cover each check of these targets. */
+#define EXPECT_COVERED(first, nr) do {					\
+	struct target_stats __sum;					\
+									\
+	sum_stats(first, nr, &__sum);					\
+	TH_LOG("exact checks: %lu, dropped as scheduled out: %lu, IPIs: %lu",\
+	       __sum.by_ipi, __sum.by_sched, __sum.ipis);		\
+	EXPECT_EQ(__sum.missed, 0);					\
+	EXPECT_EQ(__sum.sched_in_missed, 0);				\
+	EXPECT_GT(__sum.by_ipi + __sum.by_sched + __sum.missed, 0);			\
+} while (0)
+
+/*
+ * The command must not interrupt these targets. A CPU can still
+ * receive the IPI just after it switched from a targeted thread to one
+ * of these. So allow one IPI each time a target is scheduled in.
+ */
+#define EXPECT_NO_IPI(first, nr) do {					\
+	struct target_stats __sum;					\
+									\
+	sum_stats(first, nr, &__sum);					\
+	TH_LOG("IPIs: %lu, scheduled in: %lu", __sum.ipis,		\
+	       __sum.sched_in);						\
+	EXPECT_LE(__sum.ipis, __sum.sched_in);				\
+	EXPECT_EQ(__sum.sched_in_missed, 0);				\
+} while (0)
+
+/*
+ * The system call executes a memory barrier on entry and one on exit.
+ * It returns early when a single CPU is online.
+ */
+#define EXPECT_CALL_BARRIERS() do {					\
+	unsigned long __calls = atomic_load(&shared->calls);		\
+									\
+	EXPECT_EQ(atomic_load(&shared->error), 0);			\
+	if (sysconf(_SC_NPROCESSORS_ONLN) > 1) {			\
+		EXPECT_GE(atomic_load(&shared->call_barriers),		\
+			  2 * __calls);					\
+	}								\
+} while (0)
+
+static bool has_cmd(int cmd)
+{
+	return sys_membarrier(MEMBARRIER_CMD_QUERY, 0, 0) & cmd;
+}
+
+static void reset_shared(void)
+{
+	memset(shared, 0, sizeof(*shared));
+}
+
+/*
+ * Private expedited test cases.
+ *
+ * The caller is the main thread. The targets are threads of the same
+ * process.
+ */
+
+static void test_private(struct __test_metadata *_metadata,
+			 int register_cmd, int cmd)
+{
+	int nr = nr_targets();
+
+	if (!has_cmd(cmd))
+		SKIP(return, "the command is not available");
+	reset_shared();
+	start_targets(0, nr, false);
+	ASSERT_EQ(sys_membarrier(register_cmd, 0, 0), 0);
+	atomic_store(&shared->registered, 1);
+	ASSERT_TRUE(wait_ready(nr));
+	EXPECT_EQ(run_caller(cmd, 0, 0, RUN_MS), 0);
+	atomic_store(&shared->stop, 1);
+	join_targets(0, nr);
+	EXPECT_CALL_BARRIERS();
+	EXPECT_COVERED(0, nr);
+}
+
+TEST(private_expedited)
+{
+	test_private(_metadata, MEMBARRIER_CMD_REGISTER_PRIVATE_EXPEDITED,
+		     MEMBARRIER_CMD_PRIVATE_EXPEDITED);
+}
+
+TEST(private_expedited_sync_core)
+{
+	test_private(_metadata,
+		     MEMBARRIER_CMD_REGISTER_PRIVATE_EXPEDITED_SYNC_CORE,
+		     MEMBARRIER_CMD_PRIVATE_EXPEDITED_SYNC_CORE);
+}
+
+TEST(private_expedited_rseq)
+{
+	test_private(_metadata, MEMBARRIER_CMD_REGISTER_PRIVATE_EXPEDITED_RSEQ,
+		     MEMBARRIER_CMD_PRIVATE_EXPEDITED_RSEQ);
+}
+
+/* MEMBARRIER_CMD_FLAG_CPU interrupts only the CPU it names. */
+TEST(private_expedited_rseq_cpu)
+{
+	int cpus[3], i;
+
+	if (!has_cmd(MEMBARRIER_CMD_PRIVATE_EXPEDITED_RSEQ))
+		SKIP(return, "the command is not available");
+	if (allowed_cpus(cpus, 3) < 3)
+		SKIP(return, "this test needs 3 allowed CPUs");
+	reset_shared();
+	ASSERT_EQ(pin_to_cpu(cpus[0]), 0);
+	for (i = 0; i < 2; i++) {
+		targets[i] = (struct target) { .idx = i, .cpu = cpus[i + 1] };
+		ASSERT_EQ(pthread_create(&threads[i], NULL, target_thread,
+					 &targets[i]), 0);
+		created[i] = true;
+	}
+	ASSERT_EQ(sys_membarrier(MEMBARRIER_CMD_REGISTER_PRIVATE_EXPEDITED_RSEQ,
+				 0, 0), 0);
+	atomic_store(&shared->registered, 1);
+	ASSERT_TRUE(wait_ready(2));
+	EXPECT_EQ(run_caller(MEMBARRIER_CMD_PRIVATE_EXPEDITED_RSEQ,
+			     MEMBARRIER_CMD_FLAG_CPU, cpus[1], RUN_MS), 0);
+	atomic_store(&shared->stop, 1);
+	join_targets(0, 2);
+	EXPECT_CALL_BARRIERS();
+	EXPECT_COVERED(0, 1);
+	EXPECT_NO_IPI(1, 1);
+}
+
+/* A private command does not interrupt another registered process. */
+TEST(private_expedited_other_process)
+{
+	int nr = nr_targets();
+	pid_t pid;
+
+	if (!has_cmd(MEMBARRIER_CMD_PRIVATE_EXPEDITED))
+		SKIP(return, "the command is not available");
+	reset_shared();
+	pid = fork();
+	ASSERT_GE(pid, 0);
+	if (!pid) {
+		prctl(PR_SET_PDEATHSIG, SIGKILL);
+		if (sys_membarrier(MEMBARRIER_CMD_REGISTER_PRIVATE_EXPEDITED,
+				   0, 0))
+			_exit(1);
+		start_targets(0, nr, false);
+		join_targets(0, nr);
+		_exit(0);
+	}
+	start_targets(nr, 1, false);
+	ASSERT_EQ(sys_membarrier(MEMBARRIER_CMD_REGISTER_PRIVATE_EXPEDITED,
+				 0, 0), 0);
+	atomic_store(&shared->registered, 1);
+	ASSERT_TRUE(wait_ready(nr + 1));
+	EXPECT_EQ(run_caller(MEMBARRIER_CMD_PRIVATE_EXPEDITED, 0, 0, RUN_MS),
+		  0);
+	atomic_store(&shared->stop, 1);
+	join_targets(nr, 1);
+	EXPECT_TRUE(child_ok(pid));
+	EXPECT_CALL_BARRIERS();
+	EXPECT_NO_IPI(0, nr);
+	EXPECT_COVERED(nr, 1);
+}
+
+/*
+ * A private command covers every task which uses the mm. This includes
+ * a task of another thread group created with CLONE_VM.
+ */
+TEST(private_expedited_clone_vm)
+{
+	char *stack;
+	pid_t pid;
+
+	if (!has_cmd(MEMBARRIER_CMD_PRIVATE_EXPEDITED))
+		SKIP(return, "the command is not available");
+	reset_shared();
+	stack = malloc(STACK_SIZE);
+	ASSERT_NE(stack, NULL);
+	targets[0] = (struct target) { .idx = 0, .cpu = -1 };
+	pid = clone(target_clone, stack + STACK_SIZE, CLONE_VM | SIGCHLD,
+		    &targets[0]);
+	ASSERT_GE(pid, 0);
+	ASSERT_EQ(sys_membarrier(MEMBARRIER_CMD_REGISTER_PRIVATE_EXPEDITED,
+				 0, 0), 0);
+	atomic_store(&shared->registered, 1);
+	ASSERT_TRUE(wait_ready(1));
+	EXPECT_EQ(run_caller(MEMBARRIER_CMD_PRIVATE_EXPEDITED, 0, 0, RUN_MS),
+		  0);
+	atomic_store(&shared->stop, 1);
+	EXPECT_TRUE(child_ok(pid));
+	free(stack);
+	EXPECT_CALL_BARRIERS();
+	EXPECT_COVERED(0, 1);
+}
+
+/*
+ * Global expedited test cases.
+ *
+ * The caller is the test process. The targets are threads of a child
+ * process, which runs one of the setups below.
+ */
+
+enum global_setup {
+	/* Register, then create the target threads. */
+	SETUP_REGISTERED,
+	/* Do not register. */
+	SETUP_UNREGISTERED,
+	/* Create the target threads, then register. */
+	SETUP_MULTI_THREAD,
+	/*
+	 * Each target creates threads which exit. An exiting thread
+	 * clears the runqueue membarrier state in exit_mm(). The
+	 * scheduler then switches to a thread which has the same mm.
+	 */
+	SETUP_THREAD_EXIT,
+	/* A forked child inherits the registration. */
+	SETUP_FORK,
+	/* exec() drops the registration. */
+	SETUP_EXEC,
+};
+
+static int register_global(void)
+{
+	return sys_membarrier(MEMBARRIER_CMD_REGISTER_GLOBAL_EXPEDITED, 0, 0);
+}
+
+static void run_targets(int nr, bool thread_exit)
+{
+	start_targets(0, nr, thread_exit);
+	join_targets(0, nr);
+}
+
+static void global_child(enum global_setup setup, int nr)
+{
+	char fd_str[16], nr_str[16], reg_str[16], cnt_str[16];
+	pid_t pid;
+
+	prctl(PR_SET_PDEATHSIG, SIGKILL);
+	switch (setup) {
+	case SETUP_REGISTERED:
+	case SETUP_THREAD_EXIT:
+		if (register_global())
+			_exit(1);
+		atomic_store(&shared->registered, 1);
+		run_targets(nr, setup == SETUP_THREAD_EXIT);
+		break;
+	case SETUP_UNREGISTERED:
+		atomic_store(&shared->registered, 1);
+		run_targets(nr, false);
+		break;
+	case SETUP_MULTI_THREAD:
+		start_targets(0, nr, false);
+		while (atomic_load(&shared->ready) < nr)
+			sched_yield();
+		if (register_global())
+			_exit(1);
+		atomic_store(&shared->registered, 1);
+		join_targets(0, nr);
+		break;
+	case SETUP_FORK:
+		if (register_global())
+			_exit(1);
+		atomic_store(&shared->registered, 1);
+		pid = fork();
+		if (pid < 0)
+			_exit(1);
+		if (!pid) {
+			prctl(PR_SET_PDEATHSIG, SIGKILL);
+			run_targets(nr, false);
+			_exit(0);
+		}
+		_exit(child_ok(pid) ? 0 : 1);
+	case SETUP_EXEC:
+		if (register_global())
+			_exit(1);
+		atomic_store(&shared->registered, 1);
+		snprintf(fd_str, sizeof(fd_str), "%d", shared_fd);
+		snprintf(nr_str, sizeof(nr_str), "%d", nr);
+		snprintf(reg_str, sizeof(reg_str), "%d", register_fd);
+		snprintf(cnt_str, sizeof(cnt_str), "%d", counters_fd);
+		execl("/proc/self/exe", "membarrier_test_barrier",
+		      "--targets", fd_str, nr_str, reg_str, cnt_str, NULL);
+		_exit(1);
+	}
+	_exit(0);
+}
+
+static void test_global(struct __test_metadata *_metadata,
+			enum global_setup setup, bool covered)
+{
+	int nr = nr_targets();
+	pid_t pid;
+
+	if (!has_cmd(MEMBARRIER_CMD_GLOBAL_EXPEDITED))
+		SKIP(return, "the command is not available");
+	reset_shared();
+	pid = fork();
+	ASSERT_GE(pid, 0);
+	if (!pid)
+		global_child(setup, nr);
+	ASSERT_TRUE(wait_ready(nr));
+	EXPECT_EQ(run_caller(MEMBARRIER_CMD_GLOBAL_EXPEDITED, 0, 0, RUN_MS),
+		  0);
+	atomic_store(&shared->stop, 1);
+	EXPECT_TRUE(child_ok(pid));
+	EXPECT_CALL_BARRIERS();
+	if (covered)
+		EXPECT_COVERED(0, nr);
+	else
+		EXPECT_NO_IPI(0, nr);
+}
+
+TEST(global_expedited_registered)
+{
+	test_global(_metadata, SETUP_REGISTERED, true);
+}
+
+TEST(global_expedited_unregistered)
+{
+	test_global(_metadata, SETUP_UNREGISTERED, false);
+}
+
+TEST(global_expedited_multi_thread)
+{
+	test_global(_metadata, SETUP_MULTI_THREAD, true);
+}
+
+TEST(global_expedited_thread_exit)
+{
+	test_global(_metadata, SETUP_THREAD_EXIT, true);
+}
+
+TEST(global_expedited_fork)
+{
+	test_global(_metadata, SETUP_FORK, true);
+}
+
+TEST(global_expedited_exec)
+{
+	test_global(_metadata, SETUP_EXEC, false);
+}
+
+/*
+ * Each round starts a process on a first CPU. The process registers on
+ * a second CPU. Meanwhile the first CPU is idle and its idle thread
+ * borrows the mm of the process. The process then comes back to the
+ * first CPU: the scheduler switches from the idle thread to it.
+ */
+TEST(global_expedited_lazy_tlb)
+{
+	long deadline;
+	int cpus[3];
+	pid_t pid;
+
+	if (!has_cmd(MEMBARRIER_CMD_GLOBAL_EXPEDITED))
+		SKIP(return, "the command is not available");
+	if (allowed_cpus(cpus, 3) < 3)
+		SKIP(return, "this test needs 3 allowed CPUs");
+	reset_shared();
+	ASSERT_EQ(pin_to_cpu(cpus[0]), 0);
+	deadline = now_ms() + RUN_MS;
+	do {
+		struct target target = {
+			.idx = 0, .cpu = cpus[1], .ms = ROUND_MS,
+		};
+
+		atomic_store(&shared->ready, 0);
+		atomic_store(&shared->registered, 0);
+		pid = fork();
+		ASSERT_GE(pid, 0);
+		if (!pid) {
+			prctl(PR_SET_PDEATHSIG, SIGKILL);
+			if (pin_to_cpu(cpus[1]) || pin_to_cpu(cpus[2]) ||
+			    register_global())
+				_exit(1);
+			atomic_store(&shared->registered, 1);
+			run_target(&target);
+			_exit(0);
+		}
+		ASSERT_TRUE(wait_ready(1));
+		EXPECT_EQ(run_caller(MEMBARRIER_CMD_GLOBAL_EXPEDITED, 0, 0,
+				     ROUND_MS), 0);
+		EXPECT_TRUE(child_ok(pid));
+	} while (now_ms() < deadline);
+	EXPECT_CALL_BARRIERS();
+	EXPECT_COVERED(0, 1);
+}
+
+static struct shared *map_shared(int fd)
+{
+	void *p;
+
+	p = mmap(NULL, sizeof(struct shared), PROT_READ | PROT_WRITE,
+		 MAP_SHARED, fd, 0);
+	return p == MAP_FAILED ? NULL : p;
+}
+
+int main(int argc, char **argv)
+{
+	const char *err;
+
+	/* New program image of the exec test case. See global_child(). */
+	if (argc == 6 && !strcmp(argv[1], "--targets")) {
+		shared = map_shared(atoi(argv[2]));
+		register_fd = atoi(argv[4]);
+		counters_fd = atoi(argv[5]);
+		if (!shared || !map_all_counters())
+			return 1;
+		run_targets(atoi(argv[3]), false);
+		return 0;
+	}
+
+	if (sys_membarrier(MEMBARRIER_CMD_QUERY, 0, 0) < 0)
+		ksft_exit_skip("sys membarrier (CONFIG_MEMBARRIER) is disabled.\n");
+#ifndef __x86_64__
+	ksft_exit_skip("The barrier instructions are only listed for x86-64.\n");
+#endif
+	err = bpf_setup();
+	if (!err && !map_all_counters()) {
+		err = "cannot map the counters";
+		setup_broken = true;
+	}
+	if (err && setup_broken)
+		ksft_exit_fail_msg("%s\n", err);
+	if (err)
+		ksft_exit_skip("%s\n", err);
+	shared_fd = memfd_create("membarrier_test_barrier", 0);
+	if (shared_fd < 0 || ftruncate(shared_fd, sizeof(struct shared)))
+		ksft_exit_fail_msg("memfd_create failed\n");
+	shared = map_shared(shared_fd);
+	if (!shared)
+		ksft_exit_fail_msg("mmap failed\n");
+	/* This exits. bpf_teardown() then runs. */
+	return test_harness_run(argc, argv);
+}
diff --git a/tools/testing/selftests/membarrier/membarrier_test_barrier.h b/tools/testing/selftests/membarrier/membarrier_test_barrier.h
new file mode 100644
index 000000000000..d615ad1d9e8a
--- /dev/null
+++ b/tools/testing/selftests/membarrier/membarrier_test_barrier.h
@@ -0,0 +1,48 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+#ifndef MEMBARRIER_TEST_BARRIER_H
+#define MEMBARRIER_TEST_BARRIER_H
+
+#include <linux/types.h>
+
+/* Counters of a thread. */
+enum {
+	/* Memory barrier of the membarrier IPI handler. */
+	MBT_IPI,
+	/* Memory barriers of the membarrier system call. */
+	MBT_SYSCALL,
+	/* Memory barrier of the scheduler after the store to rq->curr. */
+	MBT_SWITCH_MM,
+	/* The thread was scheduled out. */
+	MBT_SCHED_OUT,
+	/*
+	 * The thread was scheduled in after a task of another mm, or
+	 * after a kernel thread.
+	 */
+	MBT_SCHED_IN,
+	/*
+	 * The thread was scheduled in as above, then scheduled out, and
+	 * the scheduler executed no memory barrier in between.
+	 */
+	MBT_SCHED_IN_MISSED,
+	MBT_NR_COUNTERS,
+};
+
+struct mbt_counters {
+	__u64 c[16];
+};
+
+/* Argument of the register_thread program. */
+struct mbt_register {
+	__u32 slot;
+};
+
+#define MBT_MAX_SLOTS		512
+#define MBT_MAX_SITES		64
+
+/* Attach cookie of a probe: counter mask, flags, index of the site. */
+#define MBT_COOKIE_MASK		0xff
+/* Count for the thread which is scheduled in, not for current. */
+#define MBT_COOKIE_NEXT		0x100
+#define MBT_COOKIE_SITE_SHIFT	16
+
+#endif /* MEMBARRIER_TEST_BARRIER_H */
-- 
2.47.3

[RFC PATCH 15/16] selftests/membarrier: Check the memory barriers of each thread

[Mathieu Desnoyers]

An expedited membarrier command promises that each targeted thread
executes a memory barrier during the call. User-space relies on this
to leave the barriers out of its fast paths. The kernel keeps the
promise with memory barriers in the system call, in its IPI handlers
and in the scheduler. If one of them is missing, user-space only sees
a rare memory ordering failure.

membarrier_test_ipi checks this promise through the interrupt
counts of each CPU. It needs its target threads to stay on their CPU,
and it only sees the IPIs. It does not see what the scheduler does.

Add membarrier_test_barrier. It counts, for each thread, the memory
barriers which membarrier relies on. It then checks that an expedited
command makes each targeted thread execute one. The comment at the
top of the file describes the method. No kernel code is added.

The counting works as follows:

 - The test lists the kernel functions which hold the instructions,
   and the kind of instruction to look for in each. It finds them by
   their bytes in the kernel text, which it reads from /proc/kcore.
   The kernel refuses a kprobe which is not on the first byte of an
   instruction, so a wrong match is dropped.

 - A BPF program is attached to each instruction with a kprobe. It
   increments counters of the thread on whose behalf the instruction
   is executed. For a barrier after the store to rq->curr, this is
   the thread which is scheduled in.

 - A BPF program on the sched_switch tracepoint counts two events for
   a thread. It was scheduled out. It was scheduled in after a task
   of another mm or after a kernel thread. The scheduler then owes it
   a barrier after the store to rq->curr. The program counts a miss
   if the thread is scheduled out again without it.

 - The counters are in a BPF array map, which the test maps in
   memory.

The test lists the instructions for x86-64 only. They are the
smp_mb() of the system call and of the IPI handlers, and the barriers
of switch_mm_irqs_off() and finish_task_switch(). This list depends
on what the compiler inlines into those functions.

The kernel turns a kprobe into a jump when it can. The handler of such
a probe runs with interrupts enabled. The probe of an IPI handler
which interrupts it is then skipped. So the test writes 0 to
/proc/sys/debug/kprobes-optimization while it runs. It restores the
value when it exits.

Each check of a target works as follows:

 - The caller pauses for one microsecond between two calls.
 - The target reads its counters during two pauses. It can then tell
   that exactly one call ran in between.
 - The IPI counter of the target must have changed.
 - A check where the target was scheduled out is dropped: the
   scheduler covers that target.

The test cases are those of membarrier_test_ipi. Each of them also
checks the two memory barriers of the system call, and the barrier of
the scheduler after the store to rq->curr.

The thread exit and lazy TLB test cases fail without commit
"sched/membarrier: Fix stale runqueue state after a lazy mm switch".

The test needs root. It also needs a kernel with BPF, BTF, kprobes and
/proc/kcore. It is skipped otherwise. The Makefile builds it only when
clang is found. membarrier_test_ipi stays: it needs none of this.

Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
---
 tools/testing/selftests/membarrier/.gitignore |    3 +
 tools/testing/selftests/membarrier/Makefile   |   18 +-
 tools/testing/selftests/membarrier/config     |    9 +
 .../membarrier/membarrier_test_barrier.bpf.c  |  168 +++
 .../membarrier/membarrier_test_barrier.c      | 1253 +++++++++++++++++
 .../membarrier/membarrier_test_barrier.h      |   48 +
 6 files changed, 1498 insertions(+), 1 deletion(-)
 create mode 100644 tools/testing/selftests/membarrier/config
 create mode 100644 tools/testing/selftests/membarrier/membarrier_test_barrier.bpf.c
 create mode 100644 tools/testing/selftests/membarrier/membarrier_test_barrier.c
 create mode 100644 tools/testing/selftests/membarrier/membarrier_test_barrier.h

diff --git a/tools/testing/selftests/membarrier/.gitignore b/tools/testing/selftests/membarrier/.gitignore
index 45b995b884a6..161723de5ec5 100644
--- a/tools/testing/selftests/membarrier/.gitignore
+++ b/tools/testing/selftests/membarrier/.gitignore
@@ -4,3 +4,6 @@ membarrier_test_single_thread
 membarrier_rseq_stress
 membarrier_test_ipi
 membarrier_test_cmd
+membarrier_test_barrier
+/tools
+*.bpf.o
diff --git a/tools/testing/selftests/membarrier/Makefile b/tools/testing/selftests/membarrier/Makefile
index c7fa0573e978..ad85f5acd541 100644
--- a/tools/testing/selftests/membarrier/Makefile
+++ b/tools/testing/selftests/membarrier/Makefile
@@ -8,6 +8,22 @@ TEST_GEN_PROGS := membarrier_test_single_thread \
 		membarrier_test_cmd \
 		membarrier_rseq_stress
 
-LOCAL_HDRS += membarrier_test_impl.h
+LOCAL_HDRS += membarrier_test_impl.h membarrier_test_barrier.h
+
+# membarrier_test_barrier loads BPF programs. Build it only when clang
+# is there, so that the other tests do not need it.
+CLANG ?= clang
+ifneq ($(shell command -v $(CLANG) 2>/dev/null),)
+TEST_GEN_PROGS += membarrier_test_barrier
+TEST_GEN_FILES += membarrier_test_barrier.bpf.o
+else
+$(warning membarrier selftests: skipping membarrier_test_barrier, needs clang)
+endif
 
 include ../lib.mk
+include ../net/bpf.mk
+
+$(OUTPUT)/membarrier_test_barrier.bpf.o: membarrier_test_barrier.h
+$(OUTPUT)/membarrier_test_barrier: $(BPFOBJ)
+$(OUTPUT)/membarrier_test_barrier: CFLAGS += -I$(SCRATCH_DIR)/include
+$(OUTPUT)/membarrier_test_barrier: LDLIBS += -lelf -lz
diff --git a/tools/testing/selftests/membarrier/config b/tools/testing/selftests/membarrier/config
new file mode 100644
index 000000000000..721da75da204
--- /dev/null
+++ b/tools/testing/selftests/membarrier/config
@@ -0,0 +1,9 @@
+CONFIG_BPF=y
+CONFIG_BPF_EVENTS=y
+CONFIG_BPF_JIT=y
+CONFIG_BPF_SYSCALL=y
+CONFIG_DEBUG_INFO_BTF=y
+CONFIG_KPROBES=y
+CONFIG_KPROBE_EVENTS=y
+CONFIG_MEMBARRIER=y
+CONFIG_PROC_KCORE=y
diff --git a/tools/testing/selftests/membarrier/membarrier_test_barrier.bpf.c b/tools/testing/selftests/membarrier/membarrier_test_barrier.bpf.c
new file mode 100644
index 000000000000..46cfba13f00f
--- /dev/null
+++ b/tools/testing/selftests/membarrier/membarrier_test_barrier.bpf.c
@@ -0,0 +1,168 @@
+// SPDX-License-Identifier: GPL-2.0
+/*
+ * Count the memory barriers which membarrier relies on, for each
+ * registered thread.
+ *
+ * See membarrier_test_barrier.c.
+ */
+#include <stdbool.h>
+#include <linux/bpf.h>
+#include <bpf/bpf_helpers.h>
+#include <bpf/bpf_tracing.h>
+
+#include "membarrier_test_barrier.h"
+
+struct mm_struct;
+
+/* The fields which this program reads. CO-RE relocates them. */
+struct task_struct {
+	struct mm_struct *mm;
+} __attribute__((preserve_access_index));
+
+struct pt_regs;
+
+/* Slot of a registered thread, plus one. */
+struct {
+	__uint(type, BPF_MAP_TYPE_TASK_STORAGE);
+	__uint(map_flags, BPF_F_NO_PREALLOC);
+	__type(key, int);
+	__type(value, __u32);
+} task_slot SEC(".maps");
+
+/* The counters of each slot. User-space maps them. */
+struct {
+	__uint(type, BPF_MAP_TYPE_ARRAY);
+	__uint(map_flags, BPF_F_MMAPABLE);
+	__uint(max_entries, MBT_MAX_SLOTS);
+	__type(key, __u32);
+	__type(value, struct mbt_counters);
+} counters SEC(".maps");
+
+/* What the scheduler owes to the thread which it switched to. */
+#define OWES_BARRIER	1
+
+struct cpu_state {
+	/* Slot, plus one, of the thread which this CPU last switched to. */
+	__u32 slot;
+	/* OWES_* flags. A probe of the scheduler clears them. */
+	__u32 owed;
+};
+
+struct {
+	__uint(type, BPF_MAP_TYPE_PERCPU_ARRAY);
+	__uint(max_entries, 1);
+	__type(key, __u32);
+	__type(value, struct cpu_state);
+} cpu_state SEC(".maps");
+
+struct {
+	__uint(type, BPF_MAP_TYPE_ARRAY);
+	__uint(max_entries, MBT_MAX_SITES);
+	__type(key, __u32);
+	__type(value, __u64);
+} site_hits SEC(".maps");
+
+/*
+ * Only the CPU which runs a thread, or which switches to it, counts for
+ * that thread. So the increments need no lock.
+ */
+static __always_inline void count(__u32 slot, __u32 mask)
+{
+	struct mbt_counters *c;
+	__u32 key = slot - 1;
+	int i;
+
+	if (!slot)
+		return;
+	c = bpf_map_lookup_elem(&counters, &key);
+	if (!c)
+		return;
+	for (i = 0; i < MBT_NR_COUNTERS; i++)
+		if (mask & (1U << i))
+			c->c[i]++;
+}
+
+static __always_inline __u32 slot_of(struct task_struct *task)
+{
+	__u32 *slot = bpf_task_storage_get(&task_slot, task, 0, 0);
+
+	return slot ? *slot : 0;
+}
+
+/* The attach cookie says which counters this instruction increments. */
+SEC("kprobe")
+int barrier_site(struct pt_regs *ctx)
+{
+	__u64 cookie = bpf_get_attach_cookie(ctx);
+	__u32 site = cookie >> MBT_COOKIE_SITE_SHIFT;
+	__u32 mask = cookie & MBT_COOKIE_MASK;
+	struct cpu_state *state;
+	__u32 zero = 0, slot = 0;
+	__u64 *hits;
+
+	hits = bpf_map_lookup_elem(&site_hits, &site);
+	if (hits)
+		__sync_fetch_and_add(hits, 1);
+	if (cookie & MBT_COOKIE_NEXT) {
+		/* Executed on behalf of the thread which is scheduled in. */
+		state = bpf_map_lookup_elem(&cpu_state, &zero);
+		if (!state)
+			return 0;
+		slot = state->slot;
+		if (mask & (1U << MBT_SWITCH_MM))
+			state->owed &= ~OWES_BARRIER;
+	} else {
+		slot = slot_of(bpf_get_current_task_btf());
+	}
+	count(slot, mask);
+	return 0;
+}
+
+SEC("tp_btf/sched_switch")
+int BPF_PROG(sched_switch, bool preempt, struct task_struct *prev,
+	     struct task_struct *next)
+{
+	struct cpu_state *state;
+	__u32 zero = 0, slot;
+
+	state = bpf_map_lookup_elem(&cpu_state, &zero);
+	if (!state)
+		return 0;
+	/* The CPU leaves the thread which it last switched to. */
+	if (state->owed & OWES_BARRIER)
+		count(state->slot, 1U << MBT_SCHED_IN_MISSED);
+	count(slot_of(prev), 1U << MBT_SCHED_OUT);
+	slot = slot_of(next);
+	state->slot = slot;
+	state->owed = 0;
+	/* The cases which need a barrier after the store to rq->curr. */
+	if (slot && next->mm && prev->mm != next->mm) {
+		state->owed = OWES_BARRIER;
+		count(slot, 1U << MBT_SCHED_IN);
+	}
+	return 0;
+}
+
+/* Give a slot to the calling thread. User-space runs this program. */
+SEC("syscall")
+int register_thread(struct mbt_register *ctx)
+{
+	struct mbt_counters *c;
+	__u32 key = ctx->slot;
+	__u32 *slot;
+	int i;
+
+	c = bpf_map_lookup_elem(&counters, &key);
+	if (!c)
+		return -1;
+	for (i = 0; i < MBT_NR_COUNTERS; i++)
+		c->c[i] = 0;
+	slot = bpf_task_storage_get(&task_slot, bpf_get_current_task_btf(), 0,
+				    BPF_LOCAL_STORAGE_GET_F_CREATE);
+	if (!slot)
+		return -1;
+	*slot = key + 1;
+	return 0;
+}
+
+char _license[] SEC("license") = "GPL";
diff --git a/tools/testing/selftests/membarrier/membarrier_test_barrier.c b/tools/testing/selftests/membarrier/membarrier_test_barrier.c
new file mode 100644
index 000000000000..46502d271242
--- /dev/null
+++ b/tools/testing/selftests/membarrier/membarrier_test_barrier.c
@@ -0,0 +1,1253 @@
+// SPDX-License-Identifier: GPL-2.0
+/*
+ * Check that the expedited membarrier commands make each targeted
+ * thread execute a memory barrier.
+ *
+ * BPF programs count, for each thread, the memory barriers which
+ * membarrier relies on. See membarrier_test_barrier.bpf.c. A kprobe is
+ * attached to each of these instructions. The sched_switch tracepoint
+ * has a program which counts the context switches. The counters are in a BPF array map,
+ * which this program maps in memory.
+ *
+ * A "caller" issues the command in a loop, with a short pause between
+ * two calls. It counts the calls which it started and the calls which
+ * returned. Each "target" thread runs this check in a loop:
+ *
+ *   1. read the count of returned calls,
+ *   2. read its own counters,
+ *   3. read the count of started calls,
+ *   4. wait until one more call has returned,
+ *   5. read its own counters again,
+ *   6. read the count of started calls again.
+ *
+ * The check is exact when the caller was between two calls at step 2
+ * and at step 5. This holds when steps 1 and 3 read the same count,
+ * and step 6 reads one more. Then one call ran between step 2 and
+ * step 5, and no other call overlaps them. So the target must have
+ * executed a memory barrier for that call: its IPI counter must have
+ * changed. Otherwise membarrier missed this thread.
+ *
+ * A check where the target was scheduled out between step 2 and
+ * step 5 is dropped. The scheduler then covers the target, and the IPI
+ * is not required. A check which is not exact is dropped as well.
+ *
+ * The BPF programs also check the scheduler. A target can be scheduled
+ * in after a task of another mm or after a kernel thread. The scheduler
+ * must then execute a memory barrier after the store to rq->curr,
+ * before the target is scheduled out again.
+ *
+ * The instructions are found by their bytes in the kernel text, which
+ * is read from /proc/kcore. The list is for x86-64. It depends on what
+ * the compiler inlines into the functions it names.
+ */
+#define _GNU_SOURCE
+#include <linux/membarrier.h>
+#include <linux/types.h>
+#include <bpf/libbpf.h>
+#include <bpf/bpf.h>
+#include <elf.h>
+#include <sys/mman.h>
+#include <sys/mount.h>
+#include <sys/prctl.h>
+#include <sys/wait.h>
+#include <syscall.h>
+#include <stdatomic.h>
+#include <pthread.h>
+#include <signal.h>
+#include <stdbool.h>
+#include <stdint.h>
+#include <stdlib.h>
+#include <string.h>
+#include <unistd.h>
+#include <fcntl.h>
+#include <sched.h>
+#include <stdio.h>
+#include <errno.h>
+#include <time.h>
+
+#include "kselftest_harness.h"
+#include "membarrier_test_barrier.h"
+
+#define MAX_TARGETS	16
+/* Duration of each test case, in milliseconds. */
+#define RUN_MS		1000
+/* Duration of each round of the lazy TLB test case, in milliseconds. */
+#define ROUND_MS	20
+/* Pause of the caller between two calls, in nanoseconds. */
+#define PAUSE_NS	1000
+/* Bound on step 4, in case the caller is not running. */
+#define MAX_SPINS	1000000
+/* Seconds to wait for the targets of a test case. */
+#define READY_TIMEOUT	10
+#define STACK_SIZE	(64 * 1024)
+
+struct target_stats {
+	/* Exact checks where the IPI covered the target. */
+	atomic_ulong by_ipi;
+	/* Checks dropped because the target was scheduled out. */
+	atomic_ulong by_sched;
+	/* Checks where the target executed no barrier. */
+	atomic_ulong missed;
+	/* Membarrier IPIs which the target received. */
+	atomic_ulong ipis;
+	/* Times the target was scheduled in after a task of another mm. */
+	atomic_ulong sched_in;
+	/* Scheduled in without a barrier after the store to rq->curr. */
+	atomic_ulong sched_in_missed;
+};
+
+struct shared {
+	atomic_ulong started;
+	atomic_ulong done;
+	/* Calls issued by the caller, and their memory barriers. */
+	atomic_ulong calls;
+	atomic_ulong call_barriers;
+	atomic_int ready;
+	/* The process of the targets is done with its registration. */
+	atomic_int registered;
+	atomic_int stop;
+	atomic_int error;
+	/* Next free slot of counters. */
+	atomic_uint next_slot;
+	/* One more for the test case which adds a target of its own. */
+	struct target_stats stats[MAX_TARGETS + 1];
+};
+
+struct target {
+	int idx;
+	/* Pin the target on this CPU, unless negative. */
+	int cpu;
+	/* Create a thread which exits, every so many checks. */
+	bool thread_exit;
+	/* Run for this duration. Run until the stop flag when 0. */
+	int ms;
+};
+
+static struct shared *shared;
+static int shared_fd;
+
+static int sys_membarrier(int cmd, unsigned int flags, int cpu_id)
+{
+	return syscall(__NR_membarrier, cmd, flags, cpu_id);
+}
+
+static long now_ms(void)
+{
+	struct timespec ts;
+
+	clock_gettime(CLOCK_MONOTONIC, &ts);
+	return ts.tv_sec * 1000 + ts.tv_nsec / 1000000;
+}
+
+static long long now_ns(void)
+{
+	struct timespec ts;
+
+	clock_gettime(CLOCK_MONOTONIC, &ts);
+	return ts.tv_sec * 1000000000LL + ts.tv_nsec;
+}
+
+static int pin_to_cpu(int cpu)
+{
+	cpu_set_t set;
+
+	CPU_ZERO(&set);
+	CPU_SET(cpu, &set);
+	return sched_setaffinity(0, sizeof(set), &set);
+}
+
+/* Store the allowed CPUs in @cpus. Return their count, up to @max. */
+static int allowed_cpus(int *cpus, int max)
+{
+	cpu_set_t set;
+	int cpu, nr = 0;
+
+	if (sched_getaffinity(0, sizeof(set), &set))
+		return 0;
+	for (cpu = 0; cpu < CPU_SETSIZE && nr < max; cpu++)
+		if (CPU_ISSET(cpu, &set))
+			cpus[nr++] = cpu;
+	return nr;
+}
+
+/* One target per allowed CPU. With the caller, the CPUs are overcommitted. */
+static int nr_targets(void)
+{
+	int cpus[MAX_TARGETS];
+
+	return allowed_cpus(cpus, MAX_TARGETS);
+}
+
+#define BPF_OBJECT	"membarrier_test_barrier.bpf.o"
+#define COUNTERS_SIZE	(MBT_MAX_SLOTS * sizeof(struct mbt_counters))
+#define OPTPROBES_FILE	"/proc/sys/debug/kprobes-optimization"
+
+/* Kind of instruction to probe within a function. */
+enum match {
+	/* smp_mb(): lock addl $0, -4(%rsp) */
+	MATCH_MB,
+	/* Write to CR3. */
+	MATCH_CR3,
+	/* Instruction with a LOCK prefix. */
+	MATCH_LOCK,
+};
+
+struct site {
+	/* A compiler suffix such as ".isra.0" is accepted after this name. */
+	const char *symbol;
+	enum match match;
+	/* Mask of the counters to increment. */
+	unsigned int counters;
+	/* Count for the thread which is scheduled in, not for current. */
+	bool next;
+};
+
+#define IPI		(1U << MBT_IPI)
+#define SYSCALL		(1U << MBT_SYSCALL)
+#define SWITCH_MM	(1U << MBT_SWITCH_MM)
+
+static const struct site sites[] = {
+	{ "ipi_mb", MATCH_MB, IPI },
+	{ "ipi_sync_core", MATCH_MB, IPI },
+	{ "ipi_rseq", MATCH_MB, IPI },
+	{ "membarrier_global_expedited", MATCH_MB, SYSCALL },
+	{ "membarrier_private_expedited", MATCH_MB, SYSCALL },
+	/* Switch to another mm: write to CR3. It is a full barrier. */
+	{ "switch_mm_irqs_off", MATCH_CR3, SWITCH_MM, true },
+	/* Switch to the mm which the CPU kept in lazy TLB mode: smp_mb(). */
+	{ "switch_mm_irqs_off", MATCH_MB, SWITCH_MM, true },
+	/*
+	 * Switch from a kernel thread: mmdrop_lazy_tlb_sched(). Any other
+	 * locked instruction of this function counts as well.
+	 */
+	{ "finish_task_switch", MATCH_LOCK, SWITCH_MM, true },
+};
+
+struct probe {
+	const struct site *site;
+	struct bpf_link *link;
+	char symbol[64];
+	unsigned long offset;
+};
+
+struct ksym {
+	unsigned long addr;
+	char name[64];
+};
+
+static struct probe probes[MBT_MAX_SITES];
+static int nr_probes;
+/* Probes of each site. */
+static int site_probes[ARRAY_SIZE(sites)];
+/* The setup failed, and not because the system lacks a feature. */
+static bool setup_broken;
+static bool too_many_probes;
+static struct bpf_object *bpf_obj;
+static struct bpf_program *site_prog;
+static struct bpf_link *switch_link;
+static const struct mbt_counters *all_counters;
+static int register_fd = -1, counters_fd = -1, hits_fd = -1;
+static int kcore_fd = -1;
+static char optprobes_saved;
+static pid_t setup_pid;
+
+static int libbpf_quiet(enum libbpf_print_level level, const char *fmt,
+			va_list args)
+{
+	return 0;
+}
+
+/* Read @len bytes of kernel memory at @addr from /proc/kcore. */
+static bool read_kernel(unsigned long addr, void *buf, size_t len)
+{
+	Elf64_Ehdr ehdr;
+	Elf64_Phdr phdr;
+	int i;
+
+	if (pread(kcore_fd, &ehdr, sizeof(ehdr), 0) != sizeof(ehdr))
+		return false;
+	for (i = 0; i < ehdr.e_phnum; i++) {
+		if (pread(kcore_fd, &phdr, sizeof(phdr),
+			  ehdr.e_phoff + i * sizeof(phdr)) != sizeof(phdr))
+			return false;
+		if (phdr.p_type != PT_LOAD || addr < phdr.p_vaddr ||
+		    addr + len > phdr.p_vaddr + phdr.p_memsz)
+			continue;
+		return pread(kcore_fd, buf, len,
+			     phdr.p_offset + addr - phdr.p_vaddr) == (ssize_t)len;
+	}
+	return false;
+}
+
+/*
+ * Probe the instruction at @addr. The kernel refuses an address which
+ * is not the first byte of an instruction. Return false in that case.
+ */
+static bool probe_insn(const struct site *site, const struct ksym *sym,
+		       unsigned long addr)
+{
+	LIBBPF_OPTS(bpf_kprobe_opts, opts);
+	struct probe *probe = &probes[nr_probes];
+
+	if (nr_probes == MBT_MAX_SITES) {
+		too_many_probes = true;
+		return false;
+	}
+	opts.offset = addr;
+	opts.bpf_cookie = site->counters | (site->next ? MBT_COOKIE_NEXT : 0) |
+			  (nr_probes << MBT_COOKIE_SITE_SHIFT);
+	probe->link = bpf_program__attach_kprobe_opts(site_prog, NULL, &opts);
+	if (!probe->link)
+		return false;
+	probe->site = site;
+	site_probes[site - sites]++;
+	probe->offset = addr - sym->addr;
+	snprintf(probe->symbol, sizeof(probe->symbol), "%s", sym->name);
+	nr_probes++;
+	return true;
+}
+
+static const unsigned char smp_mb_insn[] = { 0xf0, 0x83, 0x44, 0x24, 0xfc, 0x00 };
+
+/*
+ * Find the instructions of a site by their bytes, in the @size bytes
+ * of the function @sym. A match which is not an instruction is
+ * rejected by probe_insn().
+ */
+static void scan_function(const struct site *site, const struct ksym *sym,
+			  size_t size)
+{
+	unsigned char *text;
+	size_t i, left;
+
+	text = malloc(size);
+	if (!text || !read_kernel(sym->addr, text, size)) {
+		free(text);
+		return;
+	}
+	for (i = 0; i < size; i++) {
+		unsigned long addr = sym->addr + i;
+		unsigned char *insn = text + i;
+
+		left = size - i;
+		switch (site->match) {
+		case MATCH_MB:
+			if (left >= sizeof(smp_mb_insn) &&
+			    !memcmp(insn, smp_mb_insn, sizeof(smp_mb_insn)))
+				probe_insn(site, sym, addr);
+			break;
+		case MATCH_CR3:
+			/* mov %reg, %cr3 is 0f 22 /3, after a REX prefix or not. */
+			if (left < 3 || insn[0] != 0x0f || insn[1] != 0x22 ||
+			    (insn[2] & 0xf8) != 0xd8)
+				break;
+			if (i && (insn[-1] & 0xf0) == 0x40 &&
+			    probe_insn(site, sym, addr - 1))
+				break;
+			probe_insn(site, sym, addr);
+			break;
+		case MATCH_LOCK:
+			if (insn[0] == 0xf0)
+				probe_insn(site, sym, addr);
+			break;
+		}
+	}
+	free(text);
+}
+
+static int ksym_cmp(const void *a, const void *b)
+{
+	const struct ksym *ka = a, *kb = b;
+
+	return ka->addr < kb->addr ? -1 : ka->addr > kb->addr;
+}
+
+/*
+ * Probe the instructions of each site. Return NULL, or the reason why
+ * a site has no probe.
+ */
+static const char *probe_sites(void)
+{
+	struct ksym *syms = NULL;
+	size_t nr = 0, max = 0, i, j, k;
+	char line[256], type;
+	FILE *f;
+
+	f = fopen("/proc/kallsyms", "r");
+	if (!f)
+		return "cannot read /proc/kallsyms";
+	while (fgets(line, sizeof(line), f)) {
+		struct ksym sym;
+
+		if (sscanf(line, "%lx %c %63s", &sym.addr, &type, sym.name) != 3)
+			continue;
+		if (!strchr("tTwW", type))
+			continue;
+		if (nr == max) {
+			struct ksym *more;
+
+			max = max ? 2 * max : 1 << 16;
+			more = realloc(syms, max * sizeof(*syms));
+			if (!more) {
+				free(syms);
+				fclose(f);
+				setup_broken = true;
+				return "out of memory";
+			}
+			syms = more;
+		}
+		syms[nr++] = sym;
+	}
+	fclose(f);
+	qsort(syms, nr, sizeof(*syms), ksym_cmp);
+	for (i = 0; i < nr; i++) {
+		/* The function ends where the next symbol starts. */
+		for (j = i + 1; j < nr && syms[j].addr == syms[i].addr; j++)
+			;
+		if (j == nr || !syms[i].addr)
+			continue;
+		for (k = 0; k < ARRAY_SIZE(sites); k++) {
+			size_t len = strlen(sites[k].symbol);
+
+			if (strncmp(syms[i].name, sites[k].symbol, len))
+				continue;
+			if (syms[i].name[len] && syms[i].name[len] != '.')
+				continue;
+			scan_function(&sites[k], &syms[i],
+				      syms[j].addr - syms[i].addr);
+		}
+	}
+	free(syms);
+	if (too_many_probes)
+		return "too many barrier instructions in this kernel";
+	for (k = 0; k < ARRAY_SIZE(sites); k++) {
+		static char reason[128];
+
+		if (site_probes[k])
+			continue;
+		snprintf(reason, sizeof(reason),
+			 "cannot probe the instructions of %s()",
+			 sites[k].symbol);
+		return reason;
+	}
+	return NULL;
+}
+
+/*
+ * The kernel turns a kprobe into a jump when it can. The handler of
+ * such a probe runs with interrupts enabled: the probe of an IPI
+ * handler which interrupts it is skipped. So disable this optimization
+ * while the test runs.
+ *
+ * Write @value to the sysctl file. Return false if this fails. A
+ * kernel without the file does not optimize.
+ */
+static bool set_optprobes(char value)
+{
+	int fd = open(OPTPROBES_FILE, O_RDWR);
+	bool ok;
+
+	if (fd < 0)
+		return errno == ENOENT;
+	ok = optprobes_saved || pread(fd, &optprobes_saved, 1, 0) == 1;
+	ok = ok && pwrite(fd, &value, 1, 0) == 1;
+	close(fd);
+	return ok;
+}
+
+/* Restore the sysctl when a signal ends the test. */
+static void restore_optprobes(int sig)
+{
+	if (getpid() == setup_pid && optprobes_saved)
+		set_optprobes(optprobes_saved);
+	signal(sig, SIG_DFL);
+	raise(sig);
+}
+
+static void bpf_teardown(void);
+
+/*
+ * Load the BPF programs and probe the sites. Return NULL, or the error.
+ * The error sets setup_broken unless the system lacks a feature.
+ */
+static const char *bpf_setup(void)
+{
+	struct bpf_program *prog;
+	libbpf_print_fn_t print;
+	struct bpf_map *map;
+	const char *err;
+
+	setup_pid = getpid();
+	atexit(bpf_teardown);
+	if (geteuid())
+		return "this test needs root";
+	if (access("/sys/kernel/btf/vmlinux", R_OK))
+		return "the kernel has no BTF (CONFIG_DEBUG_INFO_BTF)";
+	kcore_fd = open("/proc/kcore", O_RDONLY | O_CLOEXEC);
+	if (kcore_fd < 0)
+		return "cannot read /proc/kcore (CONFIG_PROC_KCORE)";
+	setup_broken = true;
+	bpf_obj = bpf_object__open_file(BPF_OBJECT, NULL);
+	if (!bpf_obj)
+		return "cannot open " BPF_OBJECT;
+	if (bpf_object__load(bpf_obj))
+		return "cannot load the BPF programs";
+	site_prog = bpf_object__find_program_by_name(bpf_obj, "barrier_site");
+	prog = bpf_object__find_program_by_name(bpf_obj, "register_thread");
+	map = bpf_object__find_map_by_name(bpf_obj, "counters");
+	if (!site_prog || !prog || !map)
+		return "incomplete BPF object";
+	register_fd = bpf_program__fd(prog);
+	counters_fd = bpf_map__fd(map);
+	map = bpf_object__find_map_by_name(bpf_obj, "site_hits");
+	hits_fd = map ? bpf_map__fd(map) : -1;
+	/* The exec test case passes these two to the new program. */
+	if (fcntl(register_fd, F_SETFD, 0) || fcntl(counters_fd, F_SETFD, 0))
+		return "fcntl failed";
+	signal(SIGINT, restore_optprobes);
+	signal(SIGTERM, restore_optprobes);
+	signal(SIGHUP, restore_optprobes);
+	if (!set_optprobes('0'))
+		return "cannot write " OPTPROBES_FILE;
+	prog = bpf_object__find_program_by_name(bpf_obj, "sched_switch");
+	switch_link = prog ? bpf_program__attach(prog) : NULL;
+	if (!switch_link)
+		return "cannot attach to the sched_switch tracepoint";
+	/* probe_insn() fails on purpose. Do not print these errors. */
+	setup_broken = false;
+	print = libbpf_set_print(libbpf_quiet);
+	err = probe_sites();
+	libbpf_set_print(print);
+	return err;
+}
+
+/* Remove the probes. Print how many times each one was hit. */
+static void bpf_teardown(void)
+{
+	struct bpf_prog_info prog_info = {};
+	__u32 len = sizeof(prog_info);
+	unsigned long long lost = 0;
+	int i;
+
+	/* Only the process which loaded the programs removes them. */
+	if (getpid() != setup_pid)
+		return;
+
+	for (i = 0; i < nr_probes; i++) {
+		struct bpf_link_info info = {};
+		__u32 info_len = sizeof(info), key = i;
+		__u64 hits = 0;
+
+		bpf_link_get_info_by_fd(bpf_link__fd(probes[i].link), &info,
+					&info_len);
+		bpf_map_lookup_elem(hits_fd, &key, &hits);
+		ksft_print_msg("probe %s+0x%lx counters 0x%x%s hits %llu missed %llu\n",
+			       probes[i].symbol, probes[i].offset,
+			       probes[i].site->counters,
+			       probes[i].site->next ? " (next)" : "",
+			       (unsigned long long)hits,
+			       (unsigned long long)info.perf_event.kprobe.missed);
+		lost += info.perf_event.kprobe.missed;
+		bpf_link__destroy(probes[i].link);
+	}
+	if (site_prog &&
+	    !bpf_prog_get_info_by_fd(bpf_program__fd(site_prog), &prog_info,
+				     &len))
+		lost += prog_info.recursion_misses;
+	/* A lost hit looks like a missing barrier. */
+	if (lost)
+		ksft_print_msg("%llu probe hits were lost: a failure may be spurious\n",
+			       lost);
+	bpf_link__destroy(switch_link);
+	if (optprobes_saved)
+		set_optprobes(optprobes_saved);
+}
+
+static bool map_all_counters(void)
+{
+	void *p = mmap(NULL, COUNTERS_SIZE, PROT_READ, MAP_SHARED, counters_fd,
+		       0);
+
+	all_counters = p == MAP_FAILED ? NULL : p;
+	return all_counters;
+}
+
+/* Give counters to the calling thread. Return them, or NULL. */
+static const uint64_t *map_counters(void)
+{
+	struct mbt_register arg = {
+		.slot = atomic_fetch_add(&shared->next_slot, 1),
+	};
+	LIBBPF_OPTS(bpf_test_run_opts, opts,
+		    .ctx_in = &arg, .ctx_size_in = sizeof(arg));
+
+	if (arg.slot >= MBT_MAX_SLOTS) {
+		errno = ENOSPC;
+		return NULL;
+	}
+	if (bpf_prog_test_run_opts(register_fd, &opts) || opts.retval) {
+		if (!errno)
+			errno = EIO;
+		return NULL;
+	}
+	return (const uint64_t *)all_counters[arg.slot].c;
+}
+
+/*
+ * Read a counter. The kernel updates it behind the back of the compiler.
+ * The checks rely on the order of these reads with the other atomic
+ * reads.
+ */
+static uint64_t counter(const uint64_t *counters, int idx)
+{
+	return __atomic_load_n(&counters[idx], __ATOMIC_SEQ_CST);
+}
+
+static void *thread_nop(void *arg)
+{
+	return NULL;
+}
+
+/* Report a failure of the test setup. */
+static void setup_error(const char *what)
+{
+	fprintf(stderr, "# %s: %s\n", what, strerror(errno));
+	atomic_store(&shared->error, 1);
+}
+
+/*
+ * Run one check of a target. See the comment at the top of the file.
+ * Return false if no call returned during the check.
+ */
+static bool check_call(const uint64_t *counters, struct target_stats *stats)
+{
+	unsigned long done = atomic_load(&shared->done);
+	uint64_t ipi = counter(counters, MBT_IPI);
+	uint64_t sched_out = counter(counters, MBT_SCHED_OUT);
+	unsigned long started = atomic_load(&shared->started);
+	bool by_ipi, scheduled;
+	int spins = 0;
+
+	/* A call is in flight. */
+	if (done != started)
+		return false;
+	while (atomic_load(&shared->done) == started) {
+		if (++spins == MAX_SPINS || atomic_load(&shared->stop))
+			return false;
+	}
+	by_ipi = counter(counters, MBT_IPI) != ipi;
+	scheduled = counter(counters, MBT_SCHED_OUT) != sched_out;
+	if (scheduled) {
+		atomic_fetch_add(&stats->by_sched, 1);
+		return true;
+	}
+	/* The next call has started. */
+	if (atomic_load(&shared->started) != started + 1)
+		return true;
+	if (by_ipi)
+		atomic_fetch_add(&stats->by_ipi, 1);
+	else
+		atomic_fetch_add(&stats->missed, 1);
+	return true;
+}
+
+/* Run the checks of a target. */
+static void run_target(const struct target *target)
+{
+	struct target_stats *stats = &shared->stats[target->idx];
+	const uint64_t *counters;
+	long deadline = now_ms() + target->ms;
+	unsigned long checks = 0;
+	uint64_t first_ipi, first_sched_in;
+
+	if (target->cpu >= 0 && pin_to_cpu(target->cpu))
+		setup_error("sched_setaffinity");
+	counters = map_counters();
+	if (!counters) {
+		setup_error("register_thread");
+		atomic_fetch_add(&shared->ready, 1);
+		return;
+	}
+	atomic_fetch_add(&shared->ready, 1);
+	while (!atomic_load(&shared->registered) && !atomic_load(&shared->stop))
+		sched_yield();
+	first_ipi = counter(counters, MBT_IPI);
+	first_sched_in = counter(counters, MBT_SCHED_IN);
+	while (!atomic_load(&shared->stop)) {
+		if (check_call(counters, stats) && target->thread_exit &&
+		    !(++checks % 16)) {
+			pthread_t thread;
+
+			if (pthread_create(&thread, NULL, thread_nop, NULL) ||
+			    pthread_join(thread, NULL))
+				setup_error("thread exit");
+		}
+		if (target->ms && now_ms() >= deadline)
+			break;
+	}
+	atomic_fetch_add(&stats->ipis, counter(counters, MBT_IPI) - first_ipi);
+	atomic_fetch_add(&stats->sched_in,
+			 counter(counters, MBT_SCHED_IN) - first_sched_in);
+	atomic_fetch_add(&stats->sched_in_missed,
+			 counter(counters, MBT_SCHED_IN_MISSED));
+}
+
+static void *target_thread(void *arg)
+{
+	run_target(arg);
+	return NULL;
+}
+
+/*
+ * This task shares the TLS of its parent, errno included. It only makes
+ * system calls and reads shared memory, unless it reports an error.
+ */
+static int target_clone(void *arg)
+{
+	prctl(PR_SET_PDEATHSIG, SIGKILL);
+	run_target(arg);
+	return 0;
+}
+
+static struct target targets[MAX_TARGETS + 1];
+static pthread_t threads[MAX_TARGETS + 1];
+static bool created[MAX_TARGETS + 1];
+
+/* Start @nr target threads, with the stats indexes from @first. */
+static void start_targets(int first, int nr, bool thread_exit)
+{
+	int i;
+
+	for (i = first; i < first + nr; i++) {
+		targets[i] = (struct target) {
+			.idx = i, .cpu = -1, .thread_exit = thread_exit,
+		};
+		created[i] = !pthread_create(&threads[i], NULL, target_thread,
+					     &targets[i]);
+		if (!created[i])
+			setup_error("pthread_create");
+	}
+}
+
+static void join_targets(int first, int nr)
+{
+	int i;
+
+	for (i = first; i < first + nr; i++)
+		if (created[i] && pthread_join(threads[i], NULL))
+			setup_error("pthread_join");
+}
+
+/*
+ * Wait until @nr targets run and their process is done with its
+ * registration. Return false when this does not happen.
+ */
+static bool wait_ready(int nr)
+{
+	time_t start = time(NULL);
+
+	while (atomic_load(&shared->ready) < nr ||
+	       !atomic_load(&shared->registered)) {
+		if (time(NULL) - start > READY_TIMEOUT)
+			return false;
+		sched_yield();
+	}
+	return true;
+}
+
+/*
+ * Issue the command for @ms milliseconds. Count the calls and the
+ * memory barriers which the system call executed. Return 0, or -1
+ * with errno set.
+ */
+static int run_caller(int cmd, unsigned int flags, int cpu_id, int ms)
+{
+	static const uint64_t *counters;
+	long deadline = now_ms() + ms;
+	unsigned long calls = 0;
+	uint64_t barriers;
+	long long pause_end;
+	int i;
+
+	if (!counters)
+		counters = map_counters();
+	if (!counters)
+		return -1;
+	barriers = counter(counters, MBT_SYSCALL);
+	do {
+		for (i = 0; i < 64; i++) {
+			atomic_fetch_add(&shared->started, 1);
+			if (sys_membarrier(cmd, flags, cpu_id))
+				return -1;
+			atomic_fetch_add(&shared->done, 1);
+			calls++;
+			/* Let the targets see that no call is in flight. */
+			pause_end = now_ns() + PAUSE_NS;
+			while (now_ns() < pause_end)
+				;
+		}
+	} while (now_ms() < deadline);
+	atomic_fetch_add(&shared->calls, calls);
+	atomic_fetch_add(&shared->call_barriers,
+			 counter(counters, MBT_SYSCALL) - barriers);
+	return 0;
+}
+
+/* Wait for a child. Return true if it exited with 0. */
+static bool child_ok(pid_t pid)
+{
+	int status;
+
+	return waitpid(pid, &status, 0) == pid && WIFEXITED(status) &&
+	       !WEXITSTATUS(status);
+}
+
+static void sum_stats(int first, int nr, struct target_stats *sum)
+{
+	int i;
+
+	memset(sum, 0, sizeof(*sum));
+	for (i = first; i < first + nr; i++) {
+		struct target_stats *stats = &shared->stats[i];
+
+		sum->by_ipi += atomic_load(&stats->by_ipi);
+		sum->by_sched += atomic_load(&stats->by_sched);
+		sum->missed += atomic_load(&stats->missed);
+		sum->ipis += atomic_load(&stats->ipis);
+		sum->sched_in += atomic_load(&stats->sched_in);
+		sum->sched_in_missed += atomic_load(&stats->sched_in_missed);
+	}
+}
+
+/* The command must cover each check of these targets. */
+#define EXPECT_COVERED(first, nr) do {					\
+	struct target_stats __sum;					\
+									\
+	sum_stats(first, nr, &__sum);					\
+	TH_LOG("exact checks: %lu, dropped as scheduled out: %lu, IPIs: %lu",\
+	       __sum.by_ipi, __sum.by_sched, __sum.ipis);		\
+	EXPECT_EQ(__sum.missed, 0);					\
+	EXPECT_EQ(__sum.sched_in_missed, 0);				\
+	EXPECT_GT(__sum.by_ipi + __sum.by_sched + __sum.missed, 0);			\
+} while (0)
+
+/*
+ * The command must not interrupt these targets. A CPU can still
+ * receive the IPI just after it switched from a targeted thread to one
+ * of these. So allow one IPI each time a target is scheduled in.
+ */
+#define EXPECT_NO_IPI(first, nr) do {					\
+	struct target_stats __sum;					\
+									\
+	sum_stats(first, nr, &__sum);					\
+	TH_LOG("IPIs: %lu, scheduled in: %lu", __sum.ipis,		\
+	       __sum.sched_in);						\
+	EXPECT_LE(__sum.ipis, __sum.sched_in);				\
+	EXPECT_EQ(__sum.sched_in_missed, 0);				\
+} while (0)
+
+/*
+ * The system call executes a memory barrier on entry and one on exit.
+ * It returns early when a single CPU is online.
+ */
+#define EXPECT_CALL_BARRIERS() do {					\
+	unsigned long __calls = atomic_load(&shared->calls);		\
+									\
+	EXPECT_EQ(atomic_load(&shared->error), 0);			\
+	if (sysconf(_SC_NPROCESSORS_ONLN) > 1) {			\
+		EXPECT_GE(atomic_load(&shared->call_barriers),		\
+			  2 * __calls);					\
+	}								\
+} while (0)
+
+static bool has_cmd(int cmd)
+{
+	return sys_membarrier(MEMBARRIER_CMD_QUERY, 0, 0) & cmd;
+}
+
+static void reset_shared(void)
+{
+	memset(shared, 0, sizeof(*shared));
+}
+
+/*
+ * Private expedited test cases.
+ *
+ * The caller is the main thread. The targets are threads of the same
+ * process.
+ */
+
+static void test_private(struct __test_metadata *_metadata,
+			 int register_cmd, int cmd)
+{
+	int nr = nr_targets();
+
+	if (!has_cmd(cmd))
+		SKIP(return, "the command is not available");
+	reset_shared();
+	start_targets(0, nr, false);
+	ASSERT_EQ(sys_membarrier(register_cmd, 0, 0), 0);
+	atomic_store(&shared->registered, 1);
+	ASSERT_TRUE(wait_ready(nr));
+	EXPECT_EQ(run_caller(cmd, 0, 0, RUN_MS), 0);
+	atomic_store(&shared->stop, 1);
+	join_targets(0, nr);
+	EXPECT_CALL_BARRIERS();
+	EXPECT_COVERED(0, nr);
+}
+
+TEST(private_expedited)
+{
+	test_private(_metadata, MEMBARRIER_CMD_REGISTER_PRIVATE_EXPEDITED,
+		     MEMBARRIER_CMD_PRIVATE_EXPEDITED);
+}
+
+TEST(private_expedited_sync_core)
+{
+	test_private(_metadata,
+		     MEMBARRIER_CMD_REGISTER_PRIVATE_EXPEDITED_SYNC_CORE,
+		     MEMBARRIER_CMD_PRIVATE_EXPEDITED_SYNC_CORE);
+}
+
+TEST(private_expedited_rseq)
+{
+	test_private(_metadata, MEMBARRIER_CMD_REGISTER_PRIVATE_EXPEDITED_RSEQ,
+		     MEMBARRIER_CMD_PRIVATE_EXPEDITED_RSEQ);
+}
+
+/* MEMBARRIER_CMD_FLAG_CPU interrupts only the CPU it names. */
+TEST(private_expedited_rseq_cpu)
+{
+	int cpus[3], i;
+
+	if (!has_cmd(MEMBARRIER_CMD_PRIVATE_EXPEDITED_RSEQ))
+		SKIP(return, "the command is not available");
+	if (allowed_cpus(cpus, 3) < 3)
+		SKIP(return, "this test needs 3 allowed CPUs");
+	reset_shared();
+	ASSERT_EQ(pin_to_cpu(cpus[0]), 0);
+	for (i = 0; i < 2; i++) {
+		targets[i] = (struct target) { .idx = i, .cpu = cpus[i + 1] };
+		ASSERT_EQ(pthread_create(&threads[i], NULL, target_thread,
+					 &targets[i]), 0);
+		created[i] = true;
+	}
+	ASSERT_EQ(sys_membarrier(MEMBARRIER_CMD_REGISTER_PRIVATE_EXPEDITED_RSEQ,
+				 0, 0), 0);
+	atomic_store(&shared->registered, 1);
+	ASSERT_TRUE(wait_ready(2));
+	EXPECT_EQ(run_caller(MEMBARRIER_CMD_PRIVATE_EXPEDITED_RSEQ,
+			     MEMBARRIER_CMD_FLAG_CPU, cpus[1], RUN_MS), 0);
+	atomic_store(&shared->stop, 1);
+	join_targets(0, 2);
+	EXPECT_CALL_BARRIERS();
+	EXPECT_COVERED(0, 1);
+	EXPECT_NO_IPI(1, 1);
+}
+
+/* A private command does not interrupt another registered process. */
+TEST(private_expedited_other_process)
+{
+	int nr = nr_targets();
+	pid_t pid;
+
+	if (!has_cmd(MEMBARRIER_CMD_PRIVATE_EXPEDITED))
+		SKIP(return, "the command is not available");
+	reset_shared();
+	pid = fork();
+	ASSERT_GE(pid, 0);
+	if (!pid) {
+		prctl(PR_SET_PDEATHSIG, SIGKILL);
+		if (sys_membarrier(MEMBARRIER_CMD_REGISTER_PRIVATE_EXPEDITED,
+				   0, 0))
+			_exit(1);
+		start_targets(0, nr, false);
+		join_targets(0, nr);
+		_exit(0);
+	}
+	start_targets(nr, 1, false);
+	ASSERT_EQ(sys_membarrier(MEMBARRIER_CMD_REGISTER_PRIVATE_EXPEDITED,
+				 0, 0), 0);
+	atomic_store(&shared->registered, 1);
+	ASSERT_TRUE(wait_ready(nr + 1));
+	EXPECT_EQ(run_caller(MEMBARRIER_CMD_PRIVATE_EXPEDITED, 0, 0, RUN_MS),
+		  0);
+	atomic_store(&shared->stop, 1);
+	join_targets(nr, 1);
+	EXPECT_TRUE(child_ok(pid));
+	EXPECT_CALL_BARRIERS();
+	EXPECT_NO_IPI(0, nr);
+	EXPECT_COVERED(nr, 1);
+}
+
+/*
+ * A private command covers every task which uses the mm. This includes
+ * a task of another thread group created with CLONE_VM.
+ */
+TEST(private_expedited_clone_vm)
+{
+	char *stack;
+	pid_t pid;
+
+	if (!has_cmd(MEMBARRIER_CMD_PRIVATE_EXPEDITED))
+		SKIP(return, "the command is not available");
+	reset_shared();
+	stack = malloc(STACK_SIZE);
+	ASSERT_NE(stack, NULL);
+	targets[0] = (struct target) { .idx = 0, .cpu = -1 };
+	pid = clone(target_clone, stack + STACK_SIZE, CLONE_VM | SIGCHLD,
+		    &targets[0]);
+	ASSERT_GE(pid, 0);
+	ASSERT_EQ(sys_membarrier(MEMBARRIER_CMD_REGISTER_PRIVATE_EXPEDITED,
+				 0, 0), 0);
+	atomic_store(&shared->registered, 1);
+	ASSERT_TRUE(wait_ready(1));
+	EXPECT_EQ(run_caller(MEMBARRIER_CMD_PRIVATE_EXPEDITED, 0, 0, RUN_MS),
+		  0);
+	atomic_store(&shared->stop, 1);
+	EXPECT_TRUE(child_ok(pid));
+	free(stack);
+	EXPECT_CALL_BARRIERS();
+	EXPECT_COVERED(0, 1);
+}
+
+/*
+ * Global expedited test cases.
+ *
+ * The caller is the test process. The targets are threads of a child
+ * process, which runs one of the setups below.
+ */
+
+enum global_setup {
+	/* Register, then create the target threads. */
+	SETUP_REGISTERED,
+	/* Do not register. */
+	SETUP_UNREGISTERED,
+	/* Create the target threads, then register. */
+	SETUP_MULTI_THREAD,
+	/*
+	 * Each target creates threads which exit. An exiting thread
+	 * clears the runqueue membarrier state in exit_mm(). The
+	 * scheduler then switches to a thread which has the same mm.
+	 */
+	SETUP_THREAD_EXIT,
+	/* A forked child inherits the registration. */
+	SETUP_FORK,
+	/* exec() drops the registration. */
+	SETUP_EXEC,
+};
+
+static int register_global(void)
+{
+	return sys_membarrier(MEMBARRIER_CMD_REGISTER_GLOBAL_EXPEDITED, 0, 0);
+}
+
+static void run_targets(int nr, bool thread_exit)
+{
+	start_targets(0, nr, thread_exit);
+	join_targets(0, nr);
+}
+
+static void global_child(enum global_setup setup, int nr)
+{
+	char fd_str[16], nr_str[16], reg_str[16], cnt_str[16];
+	pid_t pid;
+
+	prctl(PR_SET_PDEATHSIG, SIGKILL);
+	switch (setup) {
+	case SETUP_REGISTERED:
+	case SETUP_THREAD_EXIT:
+		if (register_global())
+			_exit(1);
+		atomic_store(&shared->registered, 1);
+		run_targets(nr, setup == SETUP_THREAD_EXIT);
+		break;
+	case SETUP_UNREGISTERED:
+		atomic_store(&shared->registered, 1);
+		run_targets(nr, false);
+		break;
+	case SETUP_MULTI_THREAD:
+		start_targets(0, nr, false);
+		while (atomic_load(&shared->ready) < nr)
+			sched_yield();
+		if (register_global())
+			_exit(1);
+		atomic_store(&shared->registered, 1);
+		join_targets(0, nr);
+		break;
+	case SETUP_FORK:
+		if (register_global())
+			_exit(1);
+		atomic_store(&shared->registered, 1);
+		pid = fork();
+		if (pid < 0)
+			_exit(1);
+		if (!pid) {
+			prctl(PR_SET_PDEATHSIG, SIGKILL);
+			run_targets(nr, false);
+			_exit(0);
+		}
+		_exit(child_ok(pid) ? 0 : 1);
+	case SETUP_EXEC:
+		if (register_global())
+			_exit(1);
+		atomic_store(&shared->registered, 1);
+		snprintf(fd_str, sizeof(fd_str), "%d", shared_fd);
+		snprintf(nr_str, sizeof(nr_str), "%d", nr);
+		snprintf(reg_str, sizeof(reg_str), "%d", register_fd);
+		snprintf(cnt_str, sizeof(cnt_str), "%d", counters_fd);
+		execl("/proc/self/exe", "membarrier_test_barrier",
+		      "--targets", fd_str, nr_str, reg_str, cnt_str, NULL);
+		_exit(1);
+	}
+	_exit(0);
+}
+
+static void test_global(struct __test_metadata *_metadata,
+			enum global_setup setup, bool covered)
+{
+	int nr = nr_targets();
+	pid_t pid;
+
+	if (!has_cmd(MEMBARRIER_CMD_GLOBAL_EXPEDITED))
+		SKIP(return, "the command is not available");
+	reset_shared();
+	pid = fork();
+	ASSERT_GE(pid, 0);
+	if (!pid)
+		global_child(setup, nr);
+	ASSERT_TRUE(wait_ready(nr));
+	EXPECT_EQ(run_caller(MEMBARRIER_CMD_GLOBAL_EXPEDITED, 0, 0, RUN_MS),
+		  0);
+	atomic_store(&shared->stop, 1);
+	EXPECT_TRUE(child_ok(pid));
+	EXPECT_CALL_BARRIERS();
+	if (covered)
+		EXPECT_COVERED(0, nr);
+	else
+		EXPECT_NO_IPI(0, nr);
+}
+
+TEST(global_expedited_registered)
+{
+	test_global(_metadata, SETUP_REGISTERED, true);
+}
+
+TEST(global_expedited_unregistered)
+{
+	test_global(_metadata, SETUP_UNREGISTERED, false);
+}
+
+TEST(global_expedited_multi_thread)
+{
+	test_global(_metadata, SETUP_MULTI_THREAD, true);
+}
+
+TEST(global_expedited_thread_exit)
+{
+	test_global(_metadata, SETUP_THREAD_EXIT, true);
+}
+
+TEST(global_expedited_fork)
+{
+	test_global(_metadata, SETUP_FORK, true);
+}
+
+TEST(global_expedited_exec)
+{
+	test_global(_metadata, SETUP_EXEC, false);
+}
+
+/*
+ * Each round starts a process on a first CPU. The process registers on
+ * a second CPU. Meanwhile the first CPU is idle and its idle thread
+ * borrows the mm of the process. The process then comes back to the
+ * first CPU: the scheduler switches from the idle thread to it.
+ */
+TEST(global_expedited_lazy_tlb)
+{
+	long deadline;
+	int cpus[3];
+	pid_t pid;
+
+	if (!has_cmd(MEMBARRIER_CMD_GLOBAL_EXPEDITED))
+		SKIP(return, "the command is not available");
+	if (allowed_cpus(cpus, 3) < 3)
+		SKIP(return, "this test needs 3 allowed CPUs");
+	reset_shared();
+	ASSERT_EQ(pin_to_cpu(cpus[0]), 0);
+	deadline = now_ms() + RUN_MS;
+	do {
+		struct target target = {
+			.idx = 0, .cpu = cpus[1], .ms = ROUND_MS,
+		};
+
+		atomic_store(&shared->ready, 0);
+		atomic_store(&shared->registered, 0);
+		pid = fork();
+		ASSERT_GE(pid, 0);
+		if (!pid) {
+			prctl(PR_SET_PDEATHSIG, SIGKILL);
+			if (pin_to_cpu(cpus[1]) || pin_to_cpu(cpus[2]) ||
+			    register_global())
+				_exit(1);
+			atomic_store(&shared->registered, 1);
+			run_target(&target);
+			_exit(0);
+		}
+		ASSERT_TRUE(wait_ready(1));
+		EXPECT_EQ(run_caller(MEMBARRIER_CMD_GLOBAL_EXPEDITED, 0, 0,
+				     ROUND_MS), 0);
+		EXPECT_TRUE(child_ok(pid));
+	} while (now_ms() < deadline);
+	EXPECT_CALL_BARRIERS();
+	EXPECT_COVERED(0, 1);
+}
+
+static struct shared *map_shared(int fd)
+{
+	void *p;
+
+	p = mmap(NULL, sizeof(struct shared), PROT_READ | PROT_WRITE,
+		 MAP_SHARED, fd, 0);
+	return p == MAP_FAILED ? NULL : p;
+}
+
+int main(int argc, char **argv)
+{
+	const char *err;
+
+	/* New program image of the exec test case. See global_child(). */
+	if (argc == 6 && !strcmp(argv[1], "--targets")) {
+		shared = map_shared(atoi(argv[2]));
+		register_fd = atoi(argv[4]);
+		counters_fd = atoi(argv[5]);
+		if (!shared || !map_all_counters())
+			return 1;
+		run_targets(atoi(argv[3]), false);
+		return 0;
+	}
+
+	if (sys_membarrier(MEMBARRIER_CMD_QUERY, 0, 0) < 0)
+		ksft_exit_skip("sys membarrier (CONFIG_MEMBARRIER) is disabled.\n");
+#ifndef __x86_64__
+	ksft_exit_skip("The barrier instructions are only listed for x86-64.\n");
+#endif
+	err = bpf_setup();
+	if (!err && !map_all_counters()) {
+		err = "cannot map the counters";
+		setup_broken = true;
+	}
+	if (err && setup_broken)
+		ksft_exit_fail_msg("%s\n", err);
+	if (err)
+		ksft_exit_skip("%s\n", err);
+	shared_fd = memfd_create("membarrier_test_barrier", 0);
+	if (shared_fd < 0 || ftruncate(shared_fd, sizeof(struct shared)))
+		ksft_exit_fail_msg("memfd_create failed\n");
+	shared = map_shared(shared_fd);
+	if (!shared)
+		ksft_exit_fail_msg("mmap failed\n");
+	/* This exits. bpf_teardown() then runs. */
+	return test_harness_run(argc, argv);
+}
diff --git a/tools/testing/selftests/membarrier/membarrier_test_barrier.h b/tools/testing/selftests/membarrier/membarrier_test_barrier.h
new file mode 100644
index 000000000000..d615ad1d9e8a
--- /dev/null
+++ b/tools/testing/selftests/membarrier/membarrier_test_barrier.h
@@ -0,0 +1,48 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+#ifndef MEMBARRIER_TEST_BARRIER_H
+#define MEMBARRIER_TEST_BARRIER_H
+
+#include <linux/types.h>
+
+/* Counters of a thread. */
+enum {
+	/* Memory barrier of the membarrier IPI handler. */
+	MBT_IPI,
+	/* Memory barriers of the membarrier system call. */
+	MBT_SYSCALL,
+	/* Memory barrier of the scheduler after the store to rq->curr. */
+	MBT_SWITCH_MM,
+	/* The thread was scheduled out. */
+	MBT_SCHED_OUT,
+	/*
+	 * The thread was scheduled in after a task of another mm, or
+	 * after a kernel thread.
+	 */
+	MBT_SCHED_IN,
+	/*
+	 * The thread was scheduled in as above, then scheduled out, and
+	 * the scheduler executed no memory barrier in between.
+	 */
+	MBT_SCHED_IN_MISSED,
+	MBT_NR_COUNTERS,
+};
+
+struct mbt_counters {
+	__u64 c[16];
+};
+
+/* Argument of the register_thread program. */
+struct mbt_register {
+	__u32 slot;
+};
+
+#define MBT_MAX_SLOTS		512
+#define MBT_MAX_SITES		64
+
+/* Attach cookie of a probe: counter mask, flags, index of the site. */
+#define MBT_COOKIE_MASK		0xff
+/* Count for the thread which is scheduled in, not for current. */
+#define MBT_COOKIE_NEXT		0x100
+#define MBT_COOKIE_SITE_SHIFT	16
+
+#endif /* MEMBARRIER_TEST_BARRIER_H */
-- 
2.47.3


_______________________________________________
linux-riscv mailing list
linux-riscv@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-riscv

[RFC PATCH 16/16] selftests/membarrier: Check the core serialization of the sync core command

[Mathieu Desnoyers]

MEMBARRIER_CMD_PRIVATE_EXPEDITED_SYNC_CORE promises more than a
memory barrier. Each running sibling thread of the caller executes a
core serializing instruction after the call starts, and before it
runs user-space code again. A thread which is not running executes
one before it resumes user-space. This supports memory reclaim by a
JIT compiler.

On x86, the kernel serializes the core in the IPI handler and in the
scheduler. membarrier_test_barrier counts the memory barriers of each
thread, but not these instructions.

Count them as well. These are the instructions, for x86-64:

 - sync_core() in ipi_sync_core(),
 - the writes to CR3 in switch_mm_irqs_off(),
 - sync_core() in finish_task_switch().

sync_core() issues SERIALIZE or IRET. An IRET cannot be probed, so the
instruction before it is probed instead.

The sync core test case then checks two more things:

 - A call makes each target execute a core serializing instruction
   in the IPI handler. A check where the target was scheduled out is
   dropped.

 - A target can be scheduled in after a task of another mm or after
   a kernel thread. The scheduler then executes a core serializing
   instruction before the target is scheduled out again.

These checks are skipped with page table isolation. The kernel then
serializes the core with the write to CR3 of each return to
user-space, in entry code which cannot be probed.

Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
---
 .../membarrier/membarrier_test_barrier.bpf.c  |  11 +-
 .../membarrier/membarrier_test_barrier.c      | 102 +++++++++++++++++-
 .../membarrier/membarrier_test_barrier.h      |   6 ++
 3 files changed, 112 insertions(+), 7 deletions(-)

diff --git a/tools/testing/selftests/membarrier/membarrier_test_barrier.bpf.c b/tools/testing/selftests/membarrier/membarrier_test_barrier.bpf.c
index 46cfba13f00f..ce18089f8dab 100644
--- a/tools/testing/selftests/membarrier/membarrier_test_barrier.bpf.c
+++ b/tools/testing/selftests/membarrier/membarrier_test_barrier.bpf.c
@@ -1,7 +1,7 @@
 // SPDX-License-Identifier: GPL-2.0
 /*
- * Count the memory barriers which membarrier relies on, for each
- * registered thread.
+ * Count the memory barriers and the core serializing instructions which
+ * membarrier relies on, for each registered thread.
  *
  * See membarrier_test_barrier.c.
  */
@@ -40,6 +40,7 @@ struct {
 
 /* What the scheduler owes to the thread which it switched to. */
 #define OWES_BARRIER	1
+#define OWES_SYNC_CORE	2
 
 struct cpu_state {
 	/* Slot, plus one, of the thread which this CPU last switched to. */
@@ -111,6 +112,8 @@ int barrier_site(struct pt_regs *ctx)
 		slot = state->slot;
 		if (mask & (1U << MBT_SWITCH_MM))
 			state->owed &= ~OWES_BARRIER;
+		if (mask & (1U << MBT_SYNC_CORE_SWITCH))
+			state->owed &= ~OWES_SYNC_CORE;
 	} else {
 		slot = slot_of(bpf_get_current_task_btf());
 	}
@@ -131,13 +134,15 @@ int BPF_PROG(sched_switch, bool preempt, struct task_struct *prev,
 	/* The CPU leaves the thread which it last switched to. */
 	if (state->owed & OWES_BARRIER)
 		count(state->slot, 1U << MBT_SCHED_IN_MISSED);
+	if (state->owed & OWES_SYNC_CORE)
+		count(state->slot, 1U << MBT_SYNC_CORE_IN_MISSED);
 	count(slot_of(prev), 1U << MBT_SCHED_OUT);
 	slot = slot_of(next);
 	state->slot = slot;
 	state->owed = 0;
 	/* The cases which need a barrier after the store to rq->curr. */
 	if (slot && next->mm && prev->mm != next->mm) {
-		state->owed = OWES_BARRIER;
+		state->owed = OWES_BARRIER | OWES_SYNC_CORE;
 		count(slot, 1U << MBT_SCHED_IN);
 	}
 	return 0;
diff --git a/tools/testing/selftests/membarrier/membarrier_test_barrier.c b/tools/testing/selftests/membarrier/membarrier_test_barrier.c
index 46502d271242..a8152ec4eecd 100644
--- a/tools/testing/selftests/membarrier/membarrier_test_barrier.c
+++ b/tools/testing/selftests/membarrier/membarrier_test_barrier.c
@@ -36,6 +36,9 @@
  * must then execute a memory barrier after the store to rq->curr,
  * before the target is scheduled out again.
  *
+ * The sync core command is checked in the same way, with the core
+ * serializing instructions.
+ *
  * The instructions are found by their bytes in the kernel text, which
  * is read from /proc/kcore. The list is for x86-64. It depends on what
  * the compiler inlines into the functions it names.
@@ -94,6 +97,10 @@ struct target_stats {
 	atomic_ulong sched_in;
 	/* Scheduled in without a barrier after the store to rq->curr. */
 	atomic_ulong sched_in_missed;
+	/* Checks where the target did not serialize its core. */
+	atomic_ulong sync_core_missed;
+	/* Scheduled in without a core serializing instruction. */
+	atomic_ulong sync_core_in_missed;
 };
 
 struct shared {
@@ -126,6 +133,9 @@ struct target {
 static struct shared *shared;
 static int shared_fd;
 
+/* The targets of this process also check the sync core guarantee. */
+static bool check_sync_core;
+
 static int sys_membarrier(int cmd, unsigned int flags, int cpu_id)
 {
 	return syscall(__NR_membarrier, cmd, flags, cpu_id);
@@ -190,6 +200,11 @@ enum match {
 	MATCH_CR3,
 	/* Instruction with a LOCK prefix. */
 	MATCH_LOCK,
+	/*
+	 * SERIALIZE or IRET, as issued by sync_core(). An IRET cannot be
+	 * probed: the instruction before it is probed instead.
+	 */
+	MATCH_SYNC_CORE,
 };
 
 struct site {
@@ -205,15 +220,21 @@ struct site {
 #define IPI		(1U << MBT_IPI)
 #define SYSCALL		(1U << MBT_SYSCALL)
 #define SWITCH_MM	(1U << MBT_SWITCH_MM)
+#define SYNC_CORE	(1U << MBT_SYNC_CORE)
+#define SYNC_CORE_SWITCH (1U << MBT_SYNC_CORE_SWITCH)
 
 static const struct site sites[] = {
 	{ "ipi_mb", MATCH_MB, IPI },
 	{ "ipi_sync_core", MATCH_MB, IPI },
+	{ "ipi_sync_core", MATCH_SYNC_CORE, SYNC_CORE },
 	{ "ipi_rseq", MATCH_MB, IPI },
 	{ "membarrier_global_expedited", MATCH_MB, SYSCALL },
 	{ "membarrier_private_expedited", MATCH_MB, SYSCALL },
-	/* Switch to another mm: write to CR3. It is a full barrier. */
-	{ "switch_mm_irqs_off", MATCH_CR3, SWITCH_MM, true },
+	/*
+	 * Switch to another mm: write to CR3. It is a full barrier and
+	 * it is core serializing.
+	 */
+	{ "switch_mm_irqs_off", MATCH_CR3, SWITCH_MM | SYNC_CORE_SWITCH, true },
 	/* Switch to the mm which the CPU kept in lazy TLB mode: smp_mb(). */
 	{ "switch_mm_irqs_off", MATCH_MB, SWITCH_MM, true },
 	/*
@@ -221,6 +242,8 @@ static const struct site sites[] = {
 	 * locked instruction of this function counts as well.
 	 */
 	{ "finish_task_switch", MATCH_LOCK, SWITCH_MM, true },
+	/* membarrier_mm_sync_core_before_usermode() */
+	{ "finish_task_switch", MATCH_SYNC_CORE, SYNC_CORE_SWITCH, true },
 };
 
 struct probe {
@@ -319,6 +342,7 @@ static void scan_function(const struct site *site, const struct ksym *sym,
 {
 	unsigned char *text;
 	size_t i, left;
+	int32_t imm;
 
 	text = malloc(size);
 	if (!text || !read_kernel(sym->addr, text, size)) {
@@ -350,6 +374,24 @@ static void scan_function(const struct site *site, const struct ksym *sym,
 			if (insn[0] == 0xf0)
 				probe_insn(site, sym, addr);
 			break;
+		case MATCH_SYNC_CORE:
+			/* SERIALIZE is 0f 01 e8. */
+			if (left >= 3 && insn[0] == 0x0f && insn[1] == 0x01 &&
+			    insn[2] == 0xe8) {
+				probe_insn(site, sym, addr);
+				break;
+			}
+			/*
+			 * iret_to_self() is "pushq $1f; iretq; 1:". Probe the
+			 * push.
+			 */
+			if (left < 7 || insn[0] != 0x68 || insn[5] != 0x48 ||
+			    insn[6] != 0xcf)
+				break;
+			memcpy(&imm, insn + 1, sizeof(imm));
+			if (imm == (int32_t)(addr + 7))
+				probe_insn(site, sym, addr);
+			break;
 		}
 	}
 	free(text);
@@ -624,8 +666,9 @@ static bool check_call(const uint64_t *counters, struct target_stats *stats)
 	unsigned long done = atomic_load(&shared->done);
 	uint64_t ipi = counter(counters, MBT_IPI);
 	uint64_t sched_out = counter(counters, MBT_SCHED_OUT);
+	uint64_t sync_core = counter(counters, MBT_SYNC_CORE);
 	unsigned long started = atomic_load(&shared->started);
-	bool by_ipi, scheduled;
+	bool by_ipi, scheduled, serialized;
 	int spins = 0;
 
 	/* A call is in flight. */
@@ -637,6 +680,7 @@ static bool check_call(const uint64_t *counters, struct target_stats *stats)
 	}
 	by_ipi = counter(counters, MBT_IPI) != ipi;
 	scheduled = counter(counters, MBT_SCHED_OUT) != sched_out;
+	serialized = counter(counters, MBT_SYNC_CORE) != sync_core;
 	if (scheduled) {
 		atomic_fetch_add(&stats->by_sched, 1);
 		return true;
@@ -648,6 +692,8 @@ static bool check_call(const uint64_t *counters, struct target_stats *stats)
 		atomic_fetch_add(&stats->by_ipi, 1);
 	else
 		atomic_fetch_add(&stats->missed, 1);
+	if (check_sync_core && !serialized)
+		atomic_fetch_add(&stats->sync_core_missed, 1);
 	return true;
 }
 
@@ -658,7 +704,8 @@ static void run_target(const struct target *target)
 	const uint64_t *counters;
 	long deadline = now_ms() + target->ms;
 	unsigned long checks = 0;
-	uint64_t first_ipi, first_sched_in;
+	uint64_t first_ipi, first_sched_in, first_sync_core_missed;
+	uint64_t first_sched_out;
 
 	if (target->cpu >= 0 && pin_to_cpu(target->cpu))
 		setup_error("sched_setaffinity");
@@ -671,6 +718,18 @@ static void run_target(const struct target *target)
 	atomic_fetch_add(&shared->ready, 1);
 	while (!atomic_load(&shared->registered) && !atomic_load(&shared->stop))
 		sched_yield();
+	/*
+	 * The scheduler serializes the core of a registered process
+	 * only. The target may have been scheduled in before the
+	 * registration. So wait until it is scheduled out, and count
+	 * from there.
+	 */
+	first_sched_out = counter(counters, MBT_SCHED_OUT);
+	while (check_sync_core &&
+	       counter(counters, MBT_SCHED_OUT) == first_sched_out &&
+	       !atomic_load(&shared->stop))
+		usleep(1000);
+	first_sync_core_missed = counter(counters, MBT_SYNC_CORE_IN_MISSED);
 	first_ipi = counter(counters, MBT_IPI);
 	first_sched_in = counter(counters, MBT_SCHED_IN);
 	while (!atomic_load(&shared->stop)) {
@@ -690,6 +749,10 @@ static void run_target(const struct target *target)
 			 counter(counters, MBT_SCHED_IN) - first_sched_in);
 	atomic_fetch_add(&stats->sched_in_missed,
 			 counter(counters, MBT_SCHED_IN_MISSED));
+	if (check_sync_core)
+		atomic_fetch_add(&stats->sync_core_in_missed,
+				 counter(counters, MBT_SYNC_CORE_IN_MISSED) -
+				 first_sync_core_missed);
 }
 
 static void *target_thread(void *arg)
@@ -816,6 +879,9 @@ static void sum_stats(int first, int nr, struct target_stats *sum)
 		sum->ipis += atomic_load(&stats->ipis);
 		sum->sched_in += atomic_load(&stats->sched_in);
 		sum->sched_in_missed += atomic_load(&stats->sched_in_missed);
+		sum->sync_core_missed += atomic_load(&stats->sync_core_missed);
+		sum->sync_core_in_missed +=
+			atomic_load(&stats->sync_core_in_missed);
 	}
 }
 
@@ -828,6 +894,8 @@ static void sum_stats(int first, int nr, struct target_stats *sum)
 	       __sum.by_ipi, __sum.by_sched, __sum.ipis);		\
 	EXPECT_EQ(__sum.missed, 0);					\
 	EXPECT_EQ(__sum.sched_in_missed, 0);				\
+	EXPECT_EQ(__sum.sync_core_missed, 0);				\
+	EXPECT_EQ(__sum.sync_core_in_missed, 0);			\
 	EXPECT_GT(__sum.by_ipi + __sum.by_sched + __sum.missed, 0);			\
 } while (0)
 
@@ -902,8 +970,34 @@ TEST(private_expedited)
 		     MEMBARRIER_CMD_PRIVATE_EXPEDITED);
 }
 
+/*
+ * With page table isolation, the kernel serializes the core with the
+ * write to CR3 of each return to user-space. This write is in the entry
+ * code, which cannot be probed.
+ */
+static bool page_table_isolation(void)
+{
+	char *line = NULL;
+	bool pti = false;
+	size_t len = 0;
+	FILE *f;
+
+	f = fopen("/proc/cpuinfo", "r");
+	if (!f)
+		return false;
+	while (!pti && getline(&line, &len, f) >= 0)
+		pti = !strncmp(line, "flags", 5) &&
+		      (strstr(line, " pti ") || strstr(line, " pti\n"));
+	free(line);
+	fclose(f);
+	return pti;
+}
+
 TEST(private_expedited_sync_core)
 {
+	check_sync_core = !page_table_isolation();
+	if (!check_sync_core)
+		TH_LOG("page table isolation: core serialization not checked");
 	test_private(_metadata,
 		     MEMBARRIER_CMD_REGISTER_PRIVATE_EXPEDITED_SYNC_CORE,
 		     MEMBARRIER_CMD_PRIVATE_EXPEDITED_SYNC_CORE);
diff --git a/tools/testing/selftests/membarrier/membarrier_test_barrier.h b/tools/testing/selftests/membarrier/membarrier_test_barrier.h
index d615ad1d9e8a..7c31d8cba544 100644
--- a/tools/testing/selftests/membarrier/membarrier_test_barrier.h
+++ b/tools/testing/selftests/membarrier/membarrier_test_barrier.h
@@ -24,6 +24,12 @@ enum {
 	 * the scheduler executed no memory barrier in between.
 	 */
 	MBT_SCHED_IN_MISSED,
+	/* Core serializing instruction of the membarrier IPI handler. */
+	MBT_SYNC_CORE,
+	/* Core serializing instruction of the scheduler. */
+	MBT_SYNC_CORE_SWITCH,
+	/* Like MBT_SCHED_IN_MISSED, for a core serializing instruction. */
+	MBT_SYNC_CORE_IN_MISSED,
 	MBT_NR_COUNTERS,
 };
 
-- 
2.47.3

[RFC PATCH 16/16] selftests/membarrier: Check the core serialization of the sync core command

[Mathieu Desnoyers]

MEMBARRIER_CMD_PRIVATE_EXPEDITED_SYNC_CORE promises more than a
memory barrier. Each running sibling thread of the caller executes a
core serializing instruction after the call starts, and before it
runs user-space code again. A thread which is not running executes
one before it resumes user-space. This supports memory reclaim by a
JIT compiler.

On x86, the kernel serializes the core in the IPI handler and in the
scheduler. membarrier_test_barrier counts the memory barriers of each
thread, but not these instructions.

Count them as well. These are the instructions, for x86-64:

 - sync_core() in ipi_sync_core(),
 - the writes to CR3 in switch_mm_irqs_off(),
 - sync_core() in finish_task_switch().

sync_core() issues SERIALIZE or IRET. An IRET cannot be probed, so the
instruction before it is probed instead.

The sync core test case then checks two more things:

 - A call makes each target execute a core serializing instruction
   in the IPI handler. A check where the target was scheduled out is
   dropped.

 - A target can be scheduled in after a task of another mm or after
   a kernel thread. The scheduler then executes a core serializing
   instruction before the target is scheduled out again.

These checks are skipped with page table isolation. The kernel then
serializes the core with the write to CR3 of each return to
user-space, in entry code which cannot be probed.

Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
---
 .../membarrier/membarrier_test_barrier.bpf.c  |  11 +-
 .../membarrier/membarrier_test_barrier.c      | 102 +++++++++++++++++-
 .../membarrier/membarrier_test_barrier.h      |   6 ++
 3 files changed, 112 insertions(+), 7 deletions(-)

diff --git a/tools/testing/selftests/membarrier/membarrier_test_barrier.bpf.c b/tools/testing/selftests/membarrier/membarrier_test_barrier.bpf.c
index 46cfba13f00f..ce18089f8dab 100644
--- a/tools/testing/selftests/membarrier/membarrier_test_barrier.bpf.c
+++ b/tools/testing/selftests/membarrier/membarrier_test_barrier.bpf.c
@@ -1,7 +1,7 @@
 // SPDX-License-Identifier: GPL-2.0
 /*
- * Count the memory barriers which membarrier relies on, for each
- * registered thread.
+ * Count the memory barriers and the core serializing instructions which
+ * membarrier relies on, for each registered thread.
  *
  * See membarrier_test_barrier.c.
  */
@@ -40,6 +40,7 @@ struct {
 
 /* What the scheduler owes to the thread which it switched to. */
 #define OWES_BARRIER	1
+#define OWES_SYNC_CORE	2
 
 struct cpu_state {
 	/* Slot, plus one, of the thread which this CPU last switched to. */
@@ -111,6 +112,8 @@ int barrier_site(struct pt_regs *ctx)
 		slot = state->slot;
 		if (mask & (1U << MBT_SWITCH_MM))
 			state->owed &= ~OWES_BARRIER;
+		if (mask & (1U << MBT_SYNC_CORE_SWITCH))
+			state->owed &= ~OWES_SYNC_CORE;
 	} else {
 		slot = slot_of(bpf_get_current_task_btf());
 	}
@@ -131,13 +134,15 @@ int BPF_PROG(sched_switch, bool preempt, struct task_struct *prev,
 	/* The CPU leaves the thread which it last switched to. */
 	if (state->owed & OWES_BARRIER)
 		count(state->slot, 1U << MBT_SCHED_IN_MISSED);
+	if (state->owed & OWES_SYNC_CORE)
+		count(state->slot, 1U << MBT_SYNC_CORE_IN_MISSED);
 	count(slot_of(prev), 1U << MBT_SCHED_OUT);
 	slot = slot_of(next);
 	state->slot = slot;
 	state->owed = 0;
 	/* The cases which need a barrier after the store to rq->curr. */
 	if (slot && next->mm && prev->mm != next->mm) {
-		state->owed = OWES_BARRIER;
+		state->owed = OWES_BARRIER | OWES_SYNC_CORE;
 		count(slot, 1U << MBT_SCHED_IN);
 	}
 	return 0;
diff --git a/tools/testing/selftests/membarrier/membarrier_test_barrier.c b/tools/testing/selftests/membarrier/membarrier_test_barrier.c
index 46502d271242..a8152ec4eecd 100644
--- a/tools/testing/selftests/membarrier/membarrier_test_barrier.c
+++ b/tools/testing/selftests/membarrier/membarrier_test_barrier.c
@@ -36,6 +36,9 @@
  * must then execute a memory barrier after the store to rq->curr,
  * before the target is scheduled out again.
  *
+ * The sync core command is checked in the same way, with the core
+ * serializing instructions.
+ *
  * The instructions are found by their bytes in the kernel text, which
  * is read from /proc/kcore. The list is for x86-64. It depends on what
  * the compiler inlines into the functions it names.
@@ -94,6 +97,10 @@ struct target_stats {
 	atomic_ulong sched_in;
 	/* Scheduled in without a barrier after the store to rq->curr. */
 	atomic_ulong sched_in_missed;
+	/* Checks where the target did not serialize its core. */
+	atomic_ulong sync_core_missed;
+	/* Scheduled in without a core serializing instruction. */
+	atomic_ulong sync_core_in_missed;
 };
 
 struct shared {
@@ -126,6 +133,9 @@ struct target {
 static struct shared *shared;
 static int shared_fd;
 
+/* The targets of this process also check the sync core guarantee. */
+static bool check_sync_core;
+
 static int sys_membarrier(int cmd, unsigned int flags, int cpu_id)
 {
 	return syscall(__NR_membarrier, cmd, flags, cpu_id);
@@ -190,6 +200,11 @@ enum match {
 	MATCH_CR3,
 	/* Instruction with a LOCK prefix. */
 	MATCH_LOCK,
+	/*
+	 * SERIALIZE or IRET, as issued by sync_core(). An IRET cannot be
+	 * probed: the instruction before it is probed instead.
+	 */
+	MATCH_SYNC_CORE,
 };
 
 struct site {
@@ -205,15 +220,21 @@ struct site {
 #define IPI		(1U << MBT_IPI)
 #define SYSCALL		(1U << MBT_SYSCALL)
 #define SWITCH_MM	(1U << MBT_SWITCH_MM)
+#define SYNC_CORE	(1U << MBT_SYNC_CORE)
+#define SYNC_CORE_SWITCH (1U << MBT_SYNC_CORE_SWITCH)
 
 static const struct site sites[] = {
 	{ "ipi_mb", MATCH_MB, IPI },
 	{ "ipi_sync_core", MATCH_MB, IPI },
+	{ "ipi_sync_core", MATCH_SYNC_CORE, SYNC_CORE },
 	{ "ipi_rseq", MATCH_MB, IPI },
 	{ "membarrier_global_expedited", MATCH_MB, SYSCALL },
 	{ "membarrier_private_expedited", MATCH_MB, SYSCALL },
-	/* Switch to another mm: write to CR3. It is a full barrier. */
-	{ "switch_mm_irqs_off", MATCH_CR3, SWITCH_MM, true },
+	/*
+	 * Switch to another mm: write to CR3. It is a full barrier and
+	 * it is core serializing.
+	 */
+	{ "switch_mm_irqs_off", MATCH_CR3, SWITCH_MM | SYNC_CORE_SWITCH, true },
 	/* Switch to the mm which the CPU kept in lazy TLB mode: smp_mb(). */
 	{ "switch_mm_irqs_off", MATCH_MB, SWITCH_MM, true },
 	/*
@@ -221,6 +242,8 @@ static const struct site sites[] = {
 	 * locked instruction of this function counts as well.
 	 */
 	{ "finish_task_switch", MATCH_LOCK, SWITCH_MM, true },
+	/* membarrier_mm_sync_core_before_usermode() */
+	{ "finish_task_switch", MATCH_SYNC_CORE, SYNC_CORE_SWITCH, true },
 };
 
 struct probe {
@@ -319,6 +342,7 @@ static void scan_function(const struct site *site, const struct ksym *sym,
 {
 	unsigned char *text;
 	size_t i, left;
+	int32_t imm;
 
 	text = malloc(size);
 	if (!text || !read_kernel(sym->addr, text, size)) {
@@ -350,6 +374,24 @@ static void scan_function(const struct site *site, const struct ksym *sym,
 			if (insn[0] == 0xf0)
 				probe_insn(site, sym, addr);
 			break;
+		case MATCH_SYNC_CORE:
+			/* SERIALIZE is 0f 01 e8. */
+			if (left >= 3 && insn[0] == 0x0f && insn[1] == 0x01 &&
+			    insn[2] == 0xe8) {
+				probe_insn(site, sym, addr);
+				break;
+			}
+			/*
+			 * iret_to_self() is "pushq $1f; iretq; 1:". Probe the
+			 * push.
+			 */
+			if (left < 7 || insn[0] != 0x68 || insn[5] != 0x48 ||
+			    insn[6] != 0xcf)
+				break;
+			memcpy(&imm, insn + 1, sizeof(imm));
+			if (imm == (int32_t)(addr + 7))
+				probe_insn(site, sym, addr);
+			break;
 		}
 	}
 	free(text);
@@ -624,8 +666,9 @@ static bool check_call(const uint64_t *counters, struct target_stats *stats)
 	unsigned long done = atomic_load(&shared->done);
 	uint64_t ipi = counter(counters, MBT_IPI);
 	uint64_t sched_out = counter(counters, MBT_SCHED_OUT);
+	uint64_t sync_core = counter(counters, MBT_SYNC_CORE);
 	unsigned long started = atomic_load(&shared->started);
-	bool by_ipi, scheduled;
+	bool by_ipi, scheduled, serialized;
 	int spins = 0;
 
 	/* A call is in flight. */
@@ -637,6 +680,7 @@ static bool check_call(const uint64_t *counters, struct target_stats *stats)
 	}
 	by_ipi = counter(counters, MBT_IPI) != ipi;
 	scheduled = counter(counters, MBT_SCHED_OUT) != sched_out;
+	serialized = counter(counters, MBT_SYNC_CORE) != sync_core;
 	if (scheduled) {
 		atomic_fetch_add(&stats->by_sched, 1);
 		return true;
@@ -648,6 +692,8 @@ static bool check_call(const uint64_t *counters, struct target_stats *stats)
 		atomic_fetch_add(&stats->by_ipi, 1);
 	else
 		atomic_fetch_add(&stats->missed, 1);
+	if (check_sync_core && !serialized)
+		atomic_fetch_add(&stats->sync_core_missed, 1);
 	return true;
 }
 
@@ -658,7 +704,8 @@ static void run_target(const struct target *target)
 	const uint64_t *counters;
 	long deadline = now_ms() + target->ms;
 	unsigned long checks = 0;
-	uint64_t first_ipi, first_sched_in;
+	uint64_t first_ipi, first_sched_in, first_sync_core_missed;
+	uint64_t first_sched_out;
 
 	if (target->cpu >= 0 && pin_to_cpu(target->cpu))
 		setup_error("sched_setaffinity");
@@ -671,6 +718,18 @@ static void run_target(const struct target *target)
 	atomic_fetch_add(&shared->ready, 1);
 	while (!atomic_load(&shared->registered) && !atomic_load(&shared->stop))
 		sched_yield();
+	/*
+	 * The scheduler serializes the core of a registered process
+	 * only. The target may have been scheduled in before the
+	 * registration. So wait until it is scheduled out, and count
+	 * from there.
+	 */
+	first_sched_out = counter(counters, MBT_SCHED_OUT);
+	while (check_sync_core &&
+	       counter(counters, MBT_SCHED_OUT) == first_sched_out &&
+	       !atomic_load(&shared->stop))
+		usleep(1000);
+	first_sync_core_missed = counter(counters, MBT_SYNC_CORE_IN_MISSED);
 	first_ipi = counter(counters, MBT_IPI);
 	first_sched_in = counter(counters, MBT_SCHED_IN);
 	while (!atomic_load(&shared->stop)) {
@@ -690,6 +749,10 @@ static void run_target(const struct target *target)
 			 counter(counters, MBT_SCHED_IN) - first_sched_in);
 	atomic_fetch_add(&stats->sched_in_missed,
 			 counter(counters, MBT_SCHED_IN_MISSED));
+	if (check_sync_core)
+		atomic_fetch_add(&stats->sync_core_in_missed,
+				 counter(counters, MBT_SYNC_CORE_IN_MISSED) -
+				 first_sync_core_missed);
 }
 
 static void *target_thread(void *arg)
@@ -816,6 +879,9 @@ static void sum_stats(int first, int nr, struct target_stats *sum)
 		sum->ipis += atomic_load(&stats->ipis);
 		sum->sched_in += atomic_load(&stats->sched_in);
 		sum->sched_in_missed += atomic_load(&stats->sched_in_missed);
+		sum->sync_core_missed += atomic_load(&stats->sync_core_missed);
+		sum->sync_core_in_missed +=
+			atomic_load(&stats->sync_core_in_missed);
 	}
 }
 
@@ -828,6 +894,8 @@ static void sum_stats(int first, int nr, struct target_stats *sum)
 	       __sum.by_ipi, __sum.by_sched, __sum.ipis);		\
 	EXPECT_EQ(__sum.missed, 0);					\
 	EXPECT_EQ(__sum.sched_in_missed, 0);				\
+	EXPECT_EQ(__sum.sync_core_missed, 0);				\
+	EXPECT_EQ(__sum.sync_core_in_missed, 0);			\
 	EXPECT_GT(__sum.by_ipi + __sum.by_sched + __sum.missed, 0);			\
 } while (0)
 
@@ -902,8 +970,34 @@ TEST(private_expedited)
 		     MEMBARRIER_CMD_PRIVATE_EXPEDITED);
 }
 
+/*
+ * With page table isolation, the kernel serializes the core with the
+ * write to CR3 of each return to user-space. This write is in the entry
+ * code, which cannot be probed.
+ */
+static bool page_table_isolation(void)
+{
+	char *line = NULL;
+	bool pti = false;
+	size_t len = 0;
+	FILE *f;
+
+	f = fopen("/proc/cpuinfo", "r");
+	if (!f)
+		return false;
+	while (!pti && getline(&line, &len, f) >= 0)
+		pti = !strncmp(line, "flags", 5) &&
+		      (strstr(line, " pti ") || strstr(line, " pti\n"));
+	free(line);
+	fclose(f);
+	return pti;
+}
+
 TEST(private_expedited_sync_core)
 {
+	check_sync_core = !page_table_isolation();
+	if (!check_sync_core)
+		TH_LOG("page table isolation: core serialization not checked");
 	test_private(_metadata,
 		     MEMBARRIER_CMD_REGISTER_PRIVATE_EXPEDITED_SYNC_CORE,
 		     MEMBARRIER_CMD_PRIVATE_EXPEDITED_SYNC_CORE);
diff --git a/tools/testing/selftests/membarrier/membarrier_test_barrier.h b/tools/testing/selftests/membarrier/membarrier_test_barrier.h
index d615ad1d9e8a..7c31d8cba544 100644
--- a/tools/testing/selftests/membarrier/membarrier_test_barrier.h
+++ b/tools/testing/selftests/membarrier/membarrier_test_barrier.h
@@ -24,6 +24,12 @@ enum {
 	 * the scheduler executed no memory barrier in between.
 	 */
 	MBT_SCHED_IN_MISSED,
+	/* Core serializing instruction of the membarrier IPI handler. */
+	MBT_SYNC_CORE,
+	/* Core serializing instruction of the scheduler. */
+	MBT_SYNC_CORE_SWITCH,
+	/* Like MBT_SCHED_IN_MISSED, for a core serializing instruction. */
+	MBT_SYNC_CORE_IN_MISSED,
 	MBT_NR_COUNTERS,
 };
 
-- 
2.47.3


_______________________________________________
linux-riscv mailing list
linux-riscv@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-riscv

Re: [RFC PATCH 15/16] selftests/membarrier: Check the memory barriers of each thread

[sashiko-bot]

Thank you for your contribution! Sashiko AI review found 3 potential
issues to consider:
- [Medium] selftests/membarrier: run_target() pins CPU before
  registering BPF slot
- [Medium] selftests/membarrier: bpf_setup() fails instead of skipping
  on missing BPF configs
- [Low] selftests/membarrier: reset_shared() reuses BPF counter slots of
  exiting tasks
--

commit 3ab38ee940d58df6774d85bc12aa7c4563dfeb02
Author: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>

selftests/membarrier: Check the memory barriers of each thread

This commit adds the membarrier_test_barrier selftest, which uses BPF
kprobes and the sched_switch tracepoint on x86-64 to count per-thread
memory barriers executed during expedited membarrier commands and context
switches.

> diff --git a/tools/testing/selftests/membarrier/membarrier_test_barrier.c b/tools/testing/selftests/membarrier/membarrier_test_barrier.c
> new file mode 100644
> index 0000000000000..46502d2712425
> --- /dev/null
> +++ b/tools/testing/selftests/membarrier/membarrier_test_barrier.c
> @@ -0,0 +1,1253 @@

[ ... ]

> +static const char *bpf_setup(void)
> +{
> +	struct bpf_program *prog;
> +	libbpf_print_fn_t print;
> +	struct bpf_map *map;
> +	const char *err;
> +
> +	setup_pid = getpid();
> +	atexit(bpf_teardown);
> +	if (geteuid())
> +		return "this test needs root";
> +	if (access("/sys/kernel/btf/vmlinux", R_OK))
> +		return "the kernel has no BTF (CONFIG_DEBUG_INFO_BTF)";
> +	kcore_fd = open("/proc/kcore", O_RDONLY | O_CLOEXEC);
> +	if (kcore_fd < 0)
> +		return "cannot read /proc/kcore (CONFIG_PROC_KCORE)";
> +	setup_broken = true;
> +	bpf_obj = bpf_object__open_file(BPF_OBJECT, NULL);
> +	if (!bpf_obj)
> +		return "cannot open " BPF_OBJECT;
> +	if (bpf_object__load(bpf_obj))
> +		return "cannot load the BPF programs";
> +	site_prog = bpf_object__find_program_by_name(bpf_obj, "barrier_site");
> +	prog = bpf_object__find_program_by_name(bpf_obj, "register_thread");
> +	map = bpf_object__find_map_by_name(bpf_obj, "counters");
> +	if (!site_prog || !prog || !map)
> +		return "incomplete BPF object";
> +	register_fd = bpf_program__fd(prog);
> +	counters_fd = bpf_map__fd(map);
> +	map = bpf_object__find_map_by_name(bpf_obj, "site_hits");
> +	hits_fd = map ? bpf_map__fd(map) : -1;
> +	/* The exec test case passes these two to the new program. */
> +	if (fcntl(register_fd, F_SETFD, 0) || fcntl(counters_fd, F_SETFD, 0))
> +		return "fcntl failed";
> +	signal(SIGINT, restore_optprobes);
> +	signal(SIGTERM, restore_optprobes);
> +	signal(SIGHUP, restore_optprobes);
> +	if (!set_optprobes('0'))
> +		return "cannot write " OPTPROBES_FILE;
> +	prog = bpf_object__find_program_by_name(bpf_obj, "sched_switch");
> +	switch_link = prog ? bpf_program__attach(prog) : NULL;
> +	if (!switch_link)
> +		return "cannot attach to the sched_switch tracepoint";

[Severity: Medium]
Does setting setup_broken = true before bpf_object__load() and
bpf_program__attach() cause main() to fail instead of skipping when the
kernel lacks required BPF features?

A kernel with CONFIG_DEBUG_INFO_BTF=y and CONFIG_PROC_KCORE=y can still
lack CONFIG_BPF_EVENTS=y (needed to load kprobe/tracing programs and
BPF_MAP_TYPE_TASK_STORAGE) or have BPF JIT disabled (needed to attach
tp_btf/sched_switch).  When bpf_object__load() or bpf_program__attach()
fails for those reasons, setup_broken is already true, so main() exits via
ksft_exit_fail_msg() rather than ksft_exit_skip().

[ ... ]

> +/* Run the checks of a target. */
> +static void run_target(const struct target *target)
> +{
> +	struct target_stats *stats = &shared->stats[target->idx];
> +	const uint64_t *counters;
> +	long deadline = now_ms() + target->ms;
> +	unsigned long checks = 0;
> +	uint64_t first_ipi, first_sched_in;
> +
> +	if (target->cpu >= 0 && pin_to_cpu(target->cpu))
> +		setup_error("sched_setaffinity");
> +	counters = map_counters();

[Severity: Medium]
Does calling pin_to_cpu(target->cpu) before map_counters() miss the
lazy-TLB context switch in global_expedited_lazy_tlb?

In global_expedited_lazy_tlb, the child pins itself to cpus[1] and then
cpus[2] so that cpus[1]'s idle thread borrows its mm, and then calls
run_target() with target.cpu = cpus[1].  Because pin_to_cpu() migrates the
child back onto cpus[1] before map_counters() runs register_thread to
populate task_slot, the switch from the lazy-TLB idle thread back to the
child on cpus[1] runs while slot_of(next) is still 0:

tools/testing/selftests/membarrier/membarrier_test_barrier.bpf.c:sched_switch() {
    ...
	slot = slot_of(next);
	state->slot = slot;
	state->owed = 0;
	/* The cases which need a barrier after the store to rq->curr. */
	if (slot && next->mm && prev->mm != next->mm) {
		state->owed = OWES_BARRIER;
		count(slot, 1U << MBT_SCHED_IN);
	}
    ...
}

As a result, sched_switch never sets OWES_BARRIER or increments
MBT_SCHED_IN for that lazy-TLB switch.  Calling map_counters() before
pin_to_cpu(target->cpu) registers the thread in task_slot before it
migrates back to cpus[1].

[ ... ]

> +static void reset_shared(void)
> +{
> +	memset(shared, 0, sizeof(*shared));
> +}

[Severity: Low]
Can zeroing shared->next_slot in reset_shared() let an exiting task from a
finished test case clobber the BPF counters of a new thread in the next
test case?

pthread_join() and waitpid() return during do_exit() before the dying task
reaches its final context switch in do_task_dead(), while its task_slot
entry remains attached until free_task().

CPU0                                  CPU1
join_targets() / child_ok()
reset_shared()
  shared->next_slot = 0
start_targets()
  map_counters()
    register_thread() (slot 1)
                                      do_exit()
                                        do_task_dead()
                                          __schedule()
                                            sched_switch() (prev slot 1)
                                              count(1, 1U << MBT_SCHED_OUT)
                                              <-- increments new thread's
                                                  counters

Preserving shared->next_slot across reset_shared() avoids reusing counter
slots across test cases.

-- 
Sashiko AI review · https://sashiko.dev/#/patchset/20261113131903.157186-1-mathieu.desnoyers@efficios.com?part=15

Re: [RFC PATCH 15/16] selftests/membarrier: Check the memory barriers of each thread

[Mathieu Desnoyers]

> [Severity: Medium]
> Does setting setup_broken = true before bpf_object__load() and
> bpf_program__attach() cause main() to fail instead of skipping when the
> kernel lacks required BPF features?
>
> A kernel with CONFIG_DEBUG_INFO_BTF=y and CONFIG_PROC_KCORE=y can still
> lack CONFIG_BPF_EVENTS=y (needed to load kprobe/tracing programs and
> BPF_MAP_TYPE_TASK_STORAGE) or have BPF JIT disabled (needed to attach
> tp_btf/sched_switch).

Yes for CONFIG_BPF_EVENTS. The same holds for CONFIG_KPROBE_EVENTS,
which the kprobes of the test need. The test should skip in both cases.

The BPF JIT is not needed. The sched_switch program is attached as a raw
tracepoint, which does not use a BPF trampoline. I built a kernel
without CONFIG_BPF_JIT: the test attaches and all 13 test cases pass.

Planned fix for the next round: bpf_setup() checks both options before
it treats an error as a failure. It probes the kprobe program type with
libbpf_probe_bpf_prog_type(), and it looks for the kprobe PMU in sysfs.
I tested this on a kernel without each option: the test is skipped.

> [Severity: Medium]
> Does calling pin_to_cpu(target->cpu) before map_counters() miss the
> lazy-TLB context switch in global_expedited_lazy_tlb?

[...]

> As a result, sched_switch never sets OWES_BARRIER or increments
> MBT_SCHED_IN for that lazy-TLB switch.  Calling map_counters() before
> pin_to_cpu(target->cpu) registers the thread in task_slot before it
> migrates back to cpus[1].

Yes. The target moves back to its CPU before it has a counter slot. The
BPF program does not see that switch, so the barrier which the scheduler
owes the target there is not checked.

The checks of each call start after the registration, so they are not
affected. Without the fix of patch 1, the test case still fails.

Planned fix for the next round: run_target() calls map_counters() before
pin_to_cpu().

> [Severity: Low]
> Can zeroing shared->next_slot in reset_shared() let an exiting task from a
> finished test case clobber the BPF counters of a new thread in the next
> test case?

[...]

> Preserving shared->next_slot across reset_shared() avoids reusing counter
> slots across test cases.

Yes, this window exists. I did not reproduce it.

Planned fix for the next round: reset_shared() keeps next_slot. The 13
test cases use about 270 of the 512 slots at most.

Thanks,

Mathieu


-- 
Mathieu Desnoyers
EfficiOS Inc.
https://www.efficios.com