* [RFC PATCH 09/11] mm/mempolicy: add get_mempolicy2 syscall
From: Gregory Price @ 2023-12-07 0:27 UTC (permalink / raw)
To: linux-mm, jgroves, ravis.opensrc, sthanneeru, emirakhur,
Hasan.Maruf
Cc: linux-doc, linux-fsdevel, linux-api, linux-arch, linux-kernel,
akpm, arnd, tglx, luto, mingo, bp, dave.hansen, x86, hpa, mhocko,
tj, ying.huang, gregory.price, corbet, rakie.kim, hyeongtak.ji,
honggyu.kim, vtavarespetr, peterz, Michal Hocko
In-Reply-To: <20231207002759.51418-1-gregory.price@memverge.com>
get_mempolicy2 is an extensible get_mempolicy interface which allows
a user to retrieve the memory policy for a task or address.
Defined as:
get_mempolicy2(struct mpol_args *args, size_t size, unsigned long flags)
Input values include the following fields of mpol_args:
pol_nodes: if set, the nodemask of the policy returned here
pol_maxnodes: if pol_nodes is set, must describe max number of nodes
to be copied to pol_nodes
addr: if MPOL_F_ADDR is passed in `flags`, this address will be
used to return the mempolicy details of the vma the
address belongs to
flags: if MPOL_F_MEMS_ALLOWED, returns mems_allowed in pol_nodes
if MPOL_F_ADDR, return mempolicy info vma containing addr
else, returns per-task mempolicy information
Output values include the following fields of mpol_args:
mode: mempolicy mode
mode_flags: mempolicy mode flags
pol_nodes: if set, the nodemask for the mempolicy
policy_node: if the policy has extended node information, it will
be placed here. For example MPOL_INTERLEAVE will
return the next node which will be used for allocation
addr_node: If MPOL_F_ADDR is set, the numa node that the address
is located on will be returned.
home_node: policy home node will be returned here, or -1 if not.
MPOL_F_NODE has been dropped from get_mempolicy2 (it is ignored) in
favor or returning explicit values in `policy_node` and `addr_node`.
Suggested-by: Michal Hocko <mhocko@suse.com>
Signed-off-by: Gregory Price <gregory.price@memverge.com>
---
.../admin-guide/mm/numa_memory_policy.rst | 8 +++-
arch/alpha/kernel/syscalls/syscall.tbl | 1 +
arch/arm/tools/syscall.tbl | 1 +
arch/m68k/kernel/syscalls/syscall.tbl | 1 +
arch/microblaze/kernel/syscalls/syscall.tbl | 1 +
arch/mips/kernel/syscalls/syscall_n32.tbl | 1 +
arch/mips/kernel/syscalls/syscall_o32.tbl | 1 +
arch/parisc/kernel/syscalls/syscall.tbl | 1 +
arch/powerpc/kernel/syscalls/syscall.tbl | 1 +
arch/s390/kernel/syscalls/syscall.tbl | 1 +
arch/sh/kernel/syscalls/syscall.tbl | 1 +
arch/sparc/kernel/syscalls/syscall.tbl | 1 +
arch/x86/entry/syscalls/syscall_32.tbl | 1 +
arch/x86/entry/syscalls/syscall_64.tbl | 1 +
arch/xtensa/kernel/syscalls/syscall.tbl | 1 +
include/linux/syscalls.h | 2 +
include/uapi/asm-generic/unistd.h | 4 +-
mm/mempolicy.c | 43 +++++++++++++++++++
18 files changed, 69 insertions(+), 2 deletions(-)
diff --git a/Documentation/admin-guide/mm/numa_memory_policy.rst b/Documentation/admin-guide/mm/numa_memory_policy.rst
index 7195edaeaad9..82cdb765dd58 100644
--- a/Documentation/admin-guide/mm/numa_memory_policy.rst
+++ b/Documentation/admin-guide/mm/numa_memory_policy.rst
@@ -462,11 +462,17 @@ Get [Task] Memory Policy or Related Information::
long get_mempolicy(int *mode,
const unsigned long *nmask, unsigned long maxnode,
void *addr, int flags);
+ long get_mempolicy2(struct mpol_args args, size_t size,
+ unsigned long flags);
Queries the "task/process memory policy" of the calling task, or the
policy or location of a specified virtual address, depending on the
'flags' argument.
+get_mempolicy2() is an extended version of get_mempolicy() capable of
+acquiring extended information about a mempolicy, including those
+that can only be set via set_mempolicy2() or mbind2()..
+
See the get_mempolicy(2) man page for more details
@@ -523,7 +529,7 @@ Extended Mempolicy Arguments::
The extended mempolicy argument structure is defined to allow the mempolicy
interfaces future extensibility without the need for additional system calls.
-Extended interfaces (set_mempolicy2) use this argument structure.
+Extended interfaces (set_mempolicy2 and get_mempolicy2) use this structure.
The core arguments (mode, mode_flags, pol_nodes, and pol_maxnodes) apply to
all interfaces relative to their non-extended counterparts. Each additional
diff --git a/arch/alpha/kernel/syscalls/syscall.tbl b/arch/alpha/kernel/syscalls/syscall.tbl
index 0dc288a1118a..0301a8b0a262 100644
--- a/arch/alpha/kernel/syscalls/syscall.tbl
+++ b/arch/alpha/kernel/syscalls/syscall.tbl
@@ -497,3 +497,4 @@
565 common futex_wait sys_futex_wait
566 common futex_requeue sys_futex_requeue
567 common set_mempolicy2 sys_set_mempolicy2
+568 common get_mempolicy2 sys_get_mempolicy2
diff --git a/arch/arm/tools/syscall.tbl b/arch/arm/tools/syscall.tbl
index 50172ec0e1f5..771a33446e8e 100644
--- a/arch/arm/tools/syscall.tbl
+++ b/arch/arm/tools/syscall.tbl
@@ -471,3 +471,4 @@
455 common futex_wait sys_futex_wait
456 common futex_requeue sys_futex_requeue
457 common set_mempolicy2 sys_set_mempolicy2
+458 common get_mempolicy2 sys_get_mempolicy2
diff --git a/arch/m68k/kernel/syscalls/syscall.tbl b/arch/m68k/kernel/syscalls/syscall.tbl
index 839d90c535f2..048a409e684c 100644
--- a/arch/m68k/kernel/syscalls/syscall.tbl
+++ b/arch/m68k/kernel/syscalls/syscall.tbl
@@ -457,3 +457,4 @@
455 common futex_wait sys_futex_wait
456 common futex_requeue sys_futex_requeue
457 common set_mempolicy2 sys_set_mempolicy2
+458 common get_mempolicy2 sys_get_mempolicy2
diff --git a/arch/microblaze/kernel/syscalls/syscall.tbl b/arch/microblaze/kernel/syscalls/syscall.tbl
index 567c8b883735..327b01bd6793 100644
--- a/arch/microblaze/kernel/syscalls/syscall.tbl
+++ b/arch/microblaze/kernel/syscalls/syscall.tbl
@@ -463,3 +463,4 @@
455 common futex_wait sys_futex_wait
456 common futex_requeue sys_futex_requeue
457 common set_mempolicy2 sys_set_mempolicy2
+458 common get_mempolicy2 sys_get_mempolicy2
diff --git a/arch/mips/kernel/syscalls/syscall_n32.tbl b/arch/mips/kernel/syscalls/syscall_n32.tbl
index cc0640e16f2f..921d58e1da23 100644
--- a/arch/mips/kernel/syscalls/syscall_n32.tbl
+++ b/arch/mips/kernel/syscalls/syscall_n32.tbl
@@ -396,3 +396,4 @@
455 n32 futex_wait sys_futex_wait
456 n32 futex_requeue sys_futex_requeue
457 n32 set_mempolicy2 sys_set_mempolicy2
+458 n32 get_mempolicy2 sys_get_mempolicy2
diff --git a/arch/mips/kernel/syscalls/syscall_o32.tbl b/arch/mips/kernel/syscalls/syscall_o32.tbl
index f7262fde98d9..9271c83c9993 100644
--- a/arch/mips/kernel/syscalls/syscall_o32.tbl
+++ b/arch/mips/kernel/syscalls/syscall_o32.tbl
@@ -445,3 +445,4 @@
455 o32 futex_wait sys_futex_wait
456 o32 futex_requeue sys_futex_requeue
457 o32 set_mempolicy2 sys_set_mempolicy2
+458 o32 get_mempolicy2 sys_get_mempolicy2
diff --git a/arch/parisc/kernel/syscalls/syscall.tbl b/arch/parisc/kernel/syscalls/syscall.tbl
index e10f0e8bd064..0654f3f89fc7 100644
--- a/arch/parisc/kernel/syscalls/syscall.tbl
+++ b/arch/parisc/kernel/syscalls/syscall.tbl
@@ -456,3 +456,4 @@
455 common futex_wait sys_futex_wait
456 common futex_requeue sys_futex_requeue
457 common set_mempolicy2 sys_set_mempolicy2
+458 common get_mempolicy2 sys_get_mempolicy2
diff --git a/arch/powerpc/kernel/syscalls/syscall.tbl b/arch/powerpc/kernel/syscalls/syscall.tbl
index 4f03f5f42b78..ac11d2064e7a 100644
--- a/arch/powerpc/kernel/syscalls/syscall.tbl
+++ b/arch/powerpc/kernel/syscalls/syscall.tbl
@@ -544,3 +544,4 @@
455 common futex_wait sys_futex_wait
456 common futex_requeue sys_futex_requeue
457 common set_mempolicy2 sys_set_mempolicy2
+458 common get_mempolicy2 sys_get_mempolicy2
diff --git a/arch/s390/kernel/syscalls/syscall.tbl b/arch/s390/kernel/syscalls/syscall.tbl
index f98dadc2e9df..1cdcafe1ccca 100644
--- a/arch/s390/kernel/syscalls/syscall.tbl
+++ b/arch/s390/kernel/syscalls/syscall.tbl
@@ -460,3 +460,4 @@
455 common futex_wait sys_futex_wait sys_futex_wait
456 common futex_requeue sys_futex_requeue sys_futex_requeue
457 common set_mempolicy2 sys_set_mempolicy2 sys_set_mempolicy2
+458 common get_mempolicy2 sys_get_mempolicy2 sys_get_mempolicy2
diff --git a/arch/sh/kernel/syscalls/syscall.tbl b/arch/sh/kernel/syscalls/syscall.tbl
index f47ba9f2d05d..f71742024c29 100644
--- a/arch/sh/kernel/syscalls/syscall.tbl
+++ b/arch/sh/kernel/syscalls/syscall.tbl
@@ -460,3 +460,4 @@
455 common futex_wait sys_futex_wait
456 common futex_requeue sys_futex_requeue
457 common set_mempolicy2 sys_set_mempolicy2
+458 common get_mempolicy2 sys_get_mempolicy2
diff --git a/arch/sparc/kernel/syscalls/syscall.tbl b/arch/sparc/kernel/syscalls/syscall.tbl
index 53fb16616728..2fbf5dbe0620 100644
--- a/arch/sparc/kernel/syscalls/syscall.tbl
+++ b/arch/sparc/kernel/syscalls/syscall.tbl
@@ -503,3 +503,4 @@
455 common futex_wait sys_futex_wait
456 common futex_requeue sys_futex_requeue
457 common set_mempolicy2 sys_set_mempolicy2
+458 common get_mempolicy2 sys_get_mempolicy2
diff --git a/arch/x86/entry/syscalls/syscall_32.tbl b/arch/x86/entry/syscalls/syscall_32.tbl
index 4b4dc41b24ee..0af813b9a118 100644
--- a/arch/x86/entry/syscalls/syscall_32.tbl
+++ b/arch/x86/entry/syscalls/syscall_32.tbl
@@ -462,3 +462,4 @@
455 i386 futex_wait sys_futex_wait
456 i386 futex_requeue sys_futex_requeue
457 i386 set_mempolicy2 sys_set_mempolicy2
+458 i386 get_mempolicy2 sys_get_mempolicy2
diff --git a/arch/x86/entry/syscalls/syscall_64.tbl b/arch/x86/entry/syscalls/syscall_64.tbl
index 1bc2190bec27..0b777876fc15 100644
--- a/arch/x86/entry/syscalls/syscall_64.tbl
+++ b/arch/x86/entry/syscalls/syscall_64.tbl
@@ -379,6 +379,7 @@
455 common futex_wait sys_futex_wait
456 common futex_requeue sys_futex_requeue
457 common set_mempolicy2 sys_set_mempolicy2
+458 common get_mempolicy2 sys_get_mempolicy2
#
# Due to a historical design error, certain syscalls are numbered differently
diff --git a/arch/xtensa/kernel/syscalls/syscall.tbl b/arch/xtensa/kernel/syscalls/syscall.tbl
index e26dc89399eb..4536c9a4227d 100644
--- a/arch/xtensa/kernel/syscalls/syscall.tbl
+++ b/arch/xtensa/kernel/syscalls/syscall.tbl
@@ -428,3 +428,4 @@
455 common futex_wait sys_futex_wait
456 common futex_requeue sys_futex_requeue
457 common set_mempolicy2 sys_set_mempolicy2
+458 common get_mempolicy2 sys_get_mempolicy2
diff --git a/include/linux/syscalls.h b/include/linux/syscalls.h
index 3244cd990858..774512b7934e 100644
--- a/include/linux/syscalls.h
+++ b/include/linux/syscalls.h
@@ -820,6 +820,8 @@ asmlinkage long sys_get_mempolicy(int __user *policy,
unsigned long __user *nmask,
unsigned long maxnode,
unsigned long addr, unsigned long flags);
+asmlinkage long sys_get_mempolicy2(struct mpol_args *args, size_t size,
+ unsigned long flags);
asmlinkage long sys_set_mempolicy(int mode, const unsigned long __user *nmask,
unsigned long maxnode);
asmlinkage long sys_set_mempolicy2(struct mpol_args *args, size_t size,
diff --git a/include/uapi/asm-generic/unistd.h b/include/uapi/asm-generic/unistd.h
index 55486aba099f..719accc731db 100644
--- a/include/uapi/asm-generic/unistd.h
+++ b/include/uapi/asm-generic/unistd.h
@@ -830,9 +830,11 @@ __SYSCALL(__NR_futex_wait, sys_futex_wait)
__SYSCALL(__NR_futex_requeue, sys_futex_requeue)
#define __NR_set_mempolicy2 457
__SYSCALL(__NR_set_mempolicy2, sys_set_mempolicy2)
+#define __NR_get_mempolicy2 458
+__SYSCALL(__NR_get_mempolicy2, sys_get_mempolicy2)
#undef __NR_syscalls
-#define __NR_syscalls 458
+#define __NR_syscalls 459
/*
* 32 bit systems traditionally used different
diff --git a/mm/mempolicy.c b/mm/mempolicy.c
index fdc56798226b..d1d10b2746e3 100644
--- a/mm/mempolicy.c
+++ b/mm/mempolicy.c
@@ -1860,6 +1860,49 @@ SYSCALL_DEFINE5(get_mempolicy, int __user *, policy,
return kernel_get_mempolicy(policy, nmask, maxnode, addr, flags);
}
+SYSCALL_DEFINE3(get_mempolicy2, struct mpol_args __user *, uargs, size_t, usize,
+ unsigned long, flags)
+{
+ struct mpol_args kargs;
+ struct mempolicy_args margs;
+ int err;
+ nodemask_t policy_nodemask;
+
+ err = copy_struct_from_user(&kargs, sizeof(kargs), uargs, usize);
+ if (err)
+ return -EINVAL;
+
+ if (flags & MPOL_F_MEMS_ALLOWED) {
+ if (!margs.policy_nodes)
+ return -EINVAL;
+ err = do_get_mems_allowed(&policy_nodemask);
+ if (err)
+ return err;
+ return copy_nodes_to_user(kargs.pol_nodes, kargs.pol_maxnodes,
+ &policy_nodemask);
+ }
+
+ margs.policy_nodes = kargs.pol_nodes ? &policy_nodemask : NULL;
+ if (flags & MPOL_F_ADDR) {
+ margs.addr = kargs.addr;
+ err = do_get_vma_mempolicy(&margs);
+ } else
+ err = do_get_task_mempolicy(&margs);
+
+ if (err)
+ return err;
+
+ kargs.mode = margs.mode;
+ kargs.mode_flags = margs.mode_flags;
+ kargs.policy_node = margs.policy_node;
+ kargs.addr_node = (flags & MPOL_F_ADDR) ? margs.addr_node : -1;
+ if (kargs.pol_nodes)
+ err = copy_nodes_to_user(kargs.pol_nodes, kargs.pol_maxnodes,
+ margs.policy_nodes);
+
+ return copy_to_user(uargs, &kargs, usize) ? -EFAULT : 0;
+}
+
bool vma_migratable(struct vm_area_struct *vma)
{
if (vma->vm_flags & (VM_IO | VM_PFNMAP))
--
2.39.1
^ permalink raw reply related
* [RFC PATCH 08/11] mm/mempolicy: add set_mempolicy2 syscall
From: Gregory Price @ 2023-12-07 0:27 UTC (permalink / raw)
To: linux-mm, jgroves, ravis.opensrc, sthanneeru, emirakhur,
Hasan.Maruf
Cc: linux-doc, linux-fsdevel, linux-api, linux-arch, linux-kernel,
akpm, arnd, tglx, luto, mingo, bp, dave.hansen, x86, hpa, mhocko,
tj, ying.huang, gregory.price, corbet, rakie.kim, hyeongtak.ji,
honggyu.kim, vtavarespetr, peterz, Michal Hocko
In-Reply-To: <20231207002759.51418-1-gregory.price@memverge.com>
set_mempolicy2 is an extensible set_mempolicy interface which allows
a user to set the per-task memory policy.
Defined as:
set_mempolicy2(struct mpol_args *args, size_t size, unsigned long flags);
relevant mpol_args fields include the following:
mode: The MPOL_* policy (DEFAULT, INTERLEAVE, etc.)
mode_flags: The MPOL_F_* flags that were previously passed in or'd
into the mode. This was split to hopefully allow future
extensions additional mode/flag space.
pol_nodes: the nodemask to apply for the memory policy
pol_maxnodes: The max number of nodes described by pol_nodes
The usize arg is intended for the user to pass in sizeof(mpol_args)
to allow forward/backward compatibility whenever possible.
The flags argument is intended to future proof the syscall against
future extensions which may require interpreting the arguments in
the structure differently.
Semantics of `set_mempolicy` are otherwise the same as `set_mempolicy`
as of this patch.
Suggested-by: Michal Hocko <mhocko@suse.com>
Signed-off-by: Gregory Price <gregory.price@memverge.com>
---
.../admin-guide/mm/numa_memory_policy.rst | 10 ++++++
arch/alpha/kernel/syscalls/syscall.tbl | 1 +
arch/arm/tools/syscall.tbl | 1 +
arch/m68k/kernel/syscalls/syscall.tbl | 1 +
arch/microblaze/kernel/syscalls/syscall.tbl | 1 +
arch/mips/kernel/syscalls/syscall_n32.tbl | 1 +
arch/mips/kernel/syscalls/syscall_o32.tbl | 1 +
arch/parisc/kernel/syscalls/syscall.tbl | 1 +
arch/powerpc/kernel/syscalls/syscall.tbl | 1 +
arch/s390/kernel/syscalls/syscall.tbl | 1 +
arch/sh/kernel/syscalls/syscall.tbl | 1 +
arch/sparc/kernel/syscalls/syscall.tbl | 1 +
arch/x86/entry/syscalls/syscall_32.tbl | 1 +
arch/x86/entry/syscalls/syscall_64.tbl | 1 +
arch/xtensa/kernel/syscalls/syscall.tbl | 1 +
include/linux/syscalls.h | 2 ++
include/uapi/asm-generic/unistd.h | 4 ++-
mm/mempolicy.c | 34 +++++++++++++++++++
18 files changed, 63 insertions(+), 1 deletion(-)
diff --git a/Documentation/admin-guide/mm/numa_memory_policy.rst b/Documentation/admin-guide/mm/numa_memory_policy.rst
index 6d645519c2c1..7195edaeaad9 100644
--- a/Documentation/admin-guide/mm/numa_memory_policy.rst
+++ b/Documentation/admin-guide/mm/numa_memory_policy.rst
@@ -438,6 +438,8 @@ Set [Task] Memory Policy::
long set_mempolicy(int mode, const unsigned long *nmask,
unsigned long maxnode);
+ long set_mempolicy2(struct mpol_args args, size_t size,
+ unsigned long flags);
Set's the calling task's "task/process memory policy" to mode
specified by the 'mode' argument and the set of nodes defined by
@@ -446,6 +448,12 @@ specified by the 'mode' argument and the set of nodes defined by
'mode' argument with the flag (for example: MPOL_INTERLEAVE |
MPOL_F_STATIC_NODES).
+set_mempolicy2() is an extended version of set_mempolicy() capable
+of setting a mempolicy which requires more information than can be
+passed via get_mempolicy(). For example, weighted interleave with
+task-local weights requires a weight array to be passed via the
+'mpol_args->il_weights' argument in the 'struct mpol_args' arg.
+
See the set_mempolicy(2) man page for more details
@@ -515,6 +523,8 @@ Extended Mempolicy Arguments::
The extended mempolicy argument structure is defined to allow the mempolicy
interfaces future extensibility without the need for additional system calls.
+Extended interfaces (set_mempolicy2) use this argument structure.
+
The core arguments (mode, mode_flags, pol_nodes, and pol_maxnodes) apply to
all interfaces relative to their non-extended counterparts. Each additional
field may only apply to specific extended interfaces. See the respective
diff --git a/arch/alpha/kernel/syscalls/syscall.tbl b/arch/alpha/kernel/syscalls/syscall.tbl
index 18c842ca6c32..0dc288a1118a 100644
--- a/arch/alpha/kernel/syscalls/syscall.tbl
+++ b/arch/alpha/kernel/syscalls/syscall.tbl
@@ -496,3 +496,4 @@
564 common futex_wake sys_futex_wake
565 common futex_wait sys_futex_wait
566 common futex_requeue sys_futex_requeue
+567 common set_mempolicy2 sys_set_mempolicy2
diff --git a/arch/arm/tools/syscall.tbl b/arch/arm/tools/syscall.tbl
index 584f9528c996..50172ec0e1f5 100644
--- a/arch/arm/tools/syscall.tbl
+++ b/arch/arm/tools/syscall.tbl
@@ -470,3 +470,4 @@
454 common futex_wake sys_futex_wake
455 common futex_wait sys_futex_wait
456 common futex_requeue sys_futex_requeue
+457 common set_mempolicy2 sys_set_mempolicy2
diff --git a/arch/m68k/kernel/syscalls/syscall.tbl b/arch/m68k/kernel/syscalls/syscall.tbl
index 7a4b780e82cb..839d90c535f2 100644
--- a/arch/m68k/kernel/syscalls/syscall.tbl
+++ b/arch/m68k/kernel/syscalls/syscall.tbl
@@ -456,3 +456,4 @@
454 common futex_wake sys_futex_wake
455 common futex_wait sys_futex_wait
456 common futex_requeue sys_futex_requeue
+457 common set_mempolicy2 sys_set_mempolicy2
diff --git a/arch/microblaze/kernel/syscalls/syscall.tbl b/arch/microblaze/kernel/syscalls/syscall.tbl
index 5b6a0b02b7de..567c8b883735 100644
--- a/arch/microblaze/kernel/syscalls/syscall.tbl
+++ b/arch/microblaze/kernel/syscalls/syscall.tbl
@@ -462,3 +462,4 @@
454 common futex_wake sys_futex_wake
455 common futex_wait sys_futex_wait
456 common futex_requeue sys_futex_requeue
+457 common set_mempolicy2 sys_set_mempolicy2
diff --git a/arch/mips/kernel/syscalls/syscall_n32.tbl b/arch/mips/kernel/syscalls/syscall_n32.tbl
index a842b41c8e06..cc0640e16f2f 100644
--- a/arch/mips/kernel/syscalls/syscall_n32.tbl
+++ b/arch/mips/kernel/syscalls/syscall_n32.tbl
@@ -395,3 +395,4 @@
454 n32 futex_wake sys_futex_wake
455 n32 futex_wait sys_futex_wait
456 n32 futex_requeue sys_futex_requeue
+457 n32 set_mempolicy2 sys_set_mempolicy2
diff --git a/arch/mips/kernel/syscalls/syscall_o32.tbl b/arch/mips/kernel/syscalls/syscall_o32.tbl
index 525cc54bc63b..f7262fde98d9 100644
--- a/arch/mips/kernel/syscalls/syscall_o32.tbl
+++ b/arch/mips/kernel/syscalls/syscall_o32.tbl
@@ -444,3 +444,4 @@
454 o32 futex_wake sys_futex_wake
455 o32 futex_wait sys_futex_wait
456 o32 futex_requeue sys_futex_requeue
+457 o32 set_mempolicy2 sys_set_mempolicy2
diff --git a/arch/parisc/kernel/syscalls/syscall.tbl b/arch/parisc/kernel/syscalls/syscall.tbl
index a47798fed54e..e10f0e8bd064 100644
--- a/arch/parisc/kernel/syscalls/syscall.tbl
+++ b/arch/parisc/kernel/syscalls/syscall.tbl
@@ -455,3 +455,4 @@
454 common futex_wake sys_futex_wake
455 common futex_wait sys_futex_wait
456 common futex_requeue sys_futex_requeue
+457 common set_mempolicy2 sys_set_mempolicy2
diff --git a/arch/powerpc/kernel/syscalls/syscall.tbl b/arch/powerpc/kernel/syscalls/syscall.tbl
index 7fab411378f2..4f03f5f42b78 100644
--- a/arch/powerpc/kernel/syscalls/syscall.tbl
+++ b/arch/powerpc/kernel/syscalls/syscall.tbl
@@ -543,3 +543,4 @@
454 common futex_wake sys_futex_wake
455 common futex_wait sys_futex_wait
456 common futex_requeue sys_futex_requeue
+457 common set_mempolicy2 sys_set_mempolicy2
diff --git a/arch/s390/kernel/syscalls/syscall.tbl b/arch/s390/kernel/syscalls/syscall.tbl
index 86fec9b080f6..f98dadc2e9df 100644
--- a/arch/s390/kernel/syscalls/syscall.tbl
+++ b/arch/s390/kernel/syscalls/syscall.tbl
@@ -459,3 +459,4 @@
454 common futex_wake sys_futex_wake sys_futex_wake
455 common futex_wait sys_futex_wait sys_futex_wait
456 common futex_requeue sys_futex_requeue sys_futex_requeue
+457 common set_mempolicy2 sys_set_mempolicy2 sys_set_mempolicy2
diff --git a/arch/sh/kernel/syscalls/syscall.tbl b/arch/sh/kernel/syscalls/syscall.tbl
index 363fae0fe9bf..f47ba9f2d05d 100644
--- a/arch/sh/kernel/syscalls/syscall.tbl
+++ b/arch/sh/kernel/syscalls/syscall.tbl
@@ -459,3 +459,4 @@
454 common futex_wake sys_futex_wake
455 common futex_wait sys_futex_wait
456 common futex_requeue sys_futex_requeue
+457 common set_mempolicy2 sys_set_mempolicy2
diff --git a/arch/sparc/kernel/syscalls/syscall.tbl b/arch/sparc/kernel/syscalls/syscall.tbl
index 7bcaa3d5ea44..53fb16616728 100644
--- a/arch/sparc/kernel/syscalls/syscall.tbl
+++ b/arch/sparc/kernel/syscalls/syscall.tbl
@@ -502,3 +502,4 @@
454 common futex_wake sys_futex_wake
455 common futex_wait sys_futex_wait
456 common futex_requeue sys_futex_requeue
+457 common set_mempolicy2 sys_set_mempolicy2
diff --git a/arch/x86/entry/syscalls/syscall_32.tbl b/arch/x86/entry/syscalls/syscall_32.tbl
index c8fac5205803..4b4dc41b24ee 100644
--- a/arch/x86/entry/syscalls/syscall_32.tbl
+++ b/arch/x86/entry/syscalls/syscall_32.tbl
@@ -461,3 +461,4 @@
454 i386 futex_wake sys_futex_wake
455 i386 futex_wait sys_futex_wait
456 i386 futex_requeue sys_futex_requeue
+457 i386 set_mempolicy2 sys_set_mempolicy2
diff --git a/arch/x86/entry/syscalls/syscall_64.tbl b/arch/x86/entry/syscalls/syscall_64.tbl
index 8cb8bf68721c..1bc2190bec27 100644
--- a/arch/x86/entry/syscalls/syscall_64.tbl
+++ b/arch/x86/entry/syscalls/syscall_64.tbl
@@ -378,6 +378,7 @@
454 common futex_wake sys_futex_wake
455 common futex_wait sys_futex_wait
456 common futex_requeue sys_futex_requeue
+457 common set_mempolicy2 sys_set_mempolicy2
#
# Due to a historical design error, certain syscalls are numbered differently
diff --git a/arch/xtensa/kernel/syscalls/syscall.tbl b/arch/xtensa/kernel/syscalls/syscall.tbl
index 06eefa9c1458..e26dc89399eb 100644
--- a/arch/xtensa/kernel/syscalls/syscall.tbl
+++ b/arch/xtensa/kernel/syscalls/syscall.tbl
@@ -427,3 +427,4 @@
454 common futex_wake sys_futex_wake
455 common futex_wait sys_futex_wait
456 common futex_requeue sys_futex_requeue
+457 common set_mempolicy2 sys_set_mempolicy2
diff --git a/include/linux/syscalls.h b/include/linux/syscalls.h
index fd9d12de7e92..3244cd990858 100644
--- a/include/linux/syscalls.h
+++ b/include/linux/syscalls.h
@@ -822,6 +822,8 @@ asmlinkage long sys_get_mempolicy(int __user *policy,
unsigned long addr, unsigned long flags);
asmlinkage long sys_set_mempolicy(int mode, const unsigned long __user *nmask,
unsigned long maxnode);
+asmlinkage long sys_set_mempolicy2(struct mpol_args *args, size_t size,
+ unsigned long flags);
asmlinkage long sys_migrate_pages(pid_t pid, unsigned long maxnode,
const unsigned long __user *from,
const unsigned long __user *to);
diff --git a/include/uapi/asm-generic/unistd.h b/include/uapi/asm-generic/unistd.h
index 756b013fb832..55486aba099f 100644
--- a/include/uapi/asm-generic/unistd.h
+++ b/include/uapi/asm-generic/unistd.h
@@ -828,9 +828,11 @@ __SYSCALL(__NR_futex_wake, sys_futex_wake)
__SYSCALL(__NR_futex_wait, sys_futex_wait)
#define __NR_futex_requeue 456
__SYSCALL(__NR_futex_requeue, sys_futex_requeue)
+#define __NR_set_mempolicy2 457
+__SYSCALL(__NR_set_mempolicy2, sys_set_mempolicy2)
#undef __NR_syscalls
-#define __NR_syscalls 457
+#define __NR_syscalls 458
/*
* 32 bit systems traditionally used different
diff --git a/mm/mempolicy.c b/mm/mempolicy.c
index 4be63547a4b3..fdc56798226b 100644
--- a/mm/mempolicy.c
+++ b/mm/mempolicy.c
@@ -1636,6 +1636,40 @@ SYSCALL_DEFINE3(set_mempolicy, int, mode, const unsigned long __user *, nmask,
return kernel_set_mempolicy(mode, nmask, maxnode);
}
+SYSCALL_DEFINE3(set_mempolicy2, struct mpol_args __user *, uargs, size_t, usize,
+ unsigned long, flags)
+{
+ struct mpol_args kargs;
+ struct mempolicy_args margs;
+ int err;
+ nodemask_t policy_nodemask;
+
+ if (flags)
+ return -EINVAL;
+
+ err = copy_struct_from_user(&kargs, sizeof(kargs), uargs, usize);
+ if (err)
+ return err;
+
+ err = validate_mpol_flags(kargs.mode, &kargs.mode_flags);
+ if (err)
+ return err;
+
+ memset(&margs, '\0', sizeof(margs));
+ margs.mode = kargs.mode;
+ margs.mode_flags = kargs.mode_flags;
+ if (kargs.pol_nodes) {
+ err = get_nodes(&policy_nodemask, kargs.pol_nodes,
+ kargs.pol_maxnodes);
+ if (err)
+ return err;
+ margs.policy_nodes = &policy_nodemask;
+ } else
+ margs.policy_nodes = NULL;
+
+ return do_set_mempolicy(&margs);
+}
+
static int kernel_migrate_pages(pid_t pid, unsigned long maxnode,
const unsigned long __user *old_nodes,
const unsigned long __user *new_nodes)
--
2.39.1
^ permalink raw reply related
* [RFC PATCH 07/11] mm/mempolicy: add userland mempolicy arg structure
From: Gregory Price @ 2023-12-07 0:27 UTC (permalink / raw)
To: linux-mm, jgroves, ravis.opensrc, sthanneeru, emirakhur,
Hasan.Maruf
Cc: linux-doc, linux-fsdevel, linux-api, linux-arch, linux-kernel,
akpm, arnd, tglx, luto, mingo, bp, dave.hansen, x86, hpa, mhocko,
tj, ying.huang, gregory.price, corbet, rakie.kim, hyeongtak.ji,
honggyu.kim, vtavarespetr, peterz, Frank van der Linden
In-Reply-To: <20231207002759.51418-1-gregory.price@memverge.com>
This patch adds the new user-api argument structure intended for
set_mempolicy2 and mbind2.
struct mpol_args {
/* Basic mempolicy settings */
unsigned short mode;
unsigned short mode_flags;
unsigned long *pol_nodes;
unsigned long pol_maxnodes;
/* get_mempolicy2: policy information (e.g. next interleave node) */
int policy_node;
/* get_mempolicy2: memory range policy */
unsigned long addr;
int addr_node;
/* all operations: policy home node */
unsigned long home_node;
/* mbind2: address ranges to apply the policy */
const struct iovec __user *vec;
size_t vlen;
};
This structure is intended to be extensible as new mempolicy extensions
are added.
For example, set_mempolicy_home_node was added to allow vma mempolicies
to have a preferred/home node assigned. This structure allows the
addition of that setting at the time the mempolicy is set, rather
than requiring additional calls to modify the policy.
Another suggested extension is to allow mbind2 to operate on multiple
memory ranges with a single call. mbind presently operates on a single
(address, length) tuple. It was suggested that mbind2 should operate
on an iovec, which allows many memory ranges to have the same mempolicy
applied to it with a single system call.
Full breakdown of arguments as of this patch:
mode: Mempolicy mode (MPOL_DEFAULT, MPOL_INTERLEAVE)
mode_flags: Flags previously or'd into mode in set_mempolicy
(e.g.: MPOL_F_STATIC_NODES, MPOL_F_RELATIVE_NODES)
pol_nodes: Policy nodemask
pol_maxnodes: Max number of nodes in the policy nodemask
policy_node: for get_mempolicy2. Returns extended information
about a policy that was previously reported by
passing MPOL_F_NODE to get_mempolicy. Instead of
overriding the mode value, simply add a field.
addr: for get_mempolicy2. Used with MPOL_F_ADDR to run
get_mempolicy against the vma the address belongs
to instead of the task.
addr_node: for get_mempolicy2. Returns the node the address
belongs to. Previously get_mempolicy() would
override the output value of (mode) if MPOL_F_ADDR
and MPOL_F_NODE were set. Instead, we extend
mpol_args to do this by default if MPOL_F_ADDR is
set and do away with MPOL_F_NODE.
vec/vlen: Used by mbind2 to apply the mempolicy to all
address ranges described by the iovec.
Suggested-by: Frank van der Linden <fvdl@google.com>
Suggested-by: Vinicius Tavares Petrucci <vtavarespetr@micron.com>
Suggested-by: Hasan Al Maruf <Hasan.Maruf@amd.com>
Signed-off-by: Gregory Price <gregory.price@memverge.com>
Co-developed-by: Vinicius Tavares Petrucci <vtavarespetr@micron.com>
Signed-off-by: Vinicius Tavares Petrucci <vtavarespetr@micron.com>
---
.../admin-guide/mm/numa_memory_policy.rst | 31 +++++++++++++++++++
include/uapi/linux/mempolicy.h | 18 +++++++++++
2 files changed, 49 insertions(+)
diff --git a/Documentation/admin-guide/mm/numa_memory_policy.rst b/Documentation/admin-guide/mm/numa_memory_policy.rst
index b7b8d3dd420f..6d645519c2c1 100644
--- a/Documentation/admin-guide/mm/numa_memory_policy.rst
+++ b/Documentation/admin-guide/mm/numa_memory_policy.rst
@@ -488,6 +488,37 @@ closest to which page allocation will come from. Specifying the home node overri
the default allocation policy to allocate memory close to the local node for an
executing CPU.
+Extended Mempolicy Arguments::
+
+ struct mpol_args {
+ /* Basic mempolicy settings */
+ unsigned short mode;
+ unsigned short mode_flags;
+ unsigned long *pol_nodes;
+ unsigned long pol_maxnodes;
+
+ /* get_mempolicy2: policy node information */
+ int policy_node;
+
+ /* get_mempolicy2: memory range policy */
+ unsigned long addr;
+ int addr_node;
+
+ /* mbind2: policy home node */
+ unsigned long home_node;
+
+ /* mbind2: address ranges to apply the policy */
+ struct iovec *vec;
+ size_t vlen;
+ };
+
+The extended mempolicy argument structure is defined to allow the mempolicy
+interfaces future extensibility without the need for additional system calls.
+
+The core arguments (mode, mode_flags, pol_nodes, and pol_maxnodes) apply to
+all interfaces relative to their non-extended counterparts. Each additional
+field may only apply to specific extended interfaces. See the respective
+extended interface man page for more details.
Memory Policy Command Line Interface
====================================
diff --git a/include/uapi/linux/mempolicy.h b/include/uapi/linux/mempolicy.h
index 1f9bb10d1a47..e6b50903047c 100644
--- a/include/uapi/linux/mempolicy.h
+++ b/include/uapi/linux/mempolicy.h
@@ -27,6 +27,24 @@ enum {
MPOL_MAX, /* always last member of enum */
};
+struct mpol_args {
+ /* Basic mempolicy settings */
+ unsigned short mode;
+ unsigned short mode_flags;
+ unsigned long *pol_nodes;
+ unsigned long pol_maxnodes;
+ /* get_mempolicy: policy node information */
+ int policy_node;
+ /* get_mempolicy: memory range policy */
+ unsigned long addr;
+ int addr_node;
+ /* mbind2: policy home node */
+ int home_node;
+ /* mbind2: address ranges to apply the policy */
+ struct iovec *vec;
+ size_t vlen;
+};
+
/* Flags for set_mempolicy */
#define MPOL_F_STATIC_NODES (1 << 15)
#define MPOL_F_RELATIVE_NODES (1 << 14)
--
2.39.1
^ permalink raw reply related
* [RFC PATCH 06/11] mm/mempolicy: allow home_node to be set by mpol_new
From: Gregory Price @ 2023-12-07 0:27 UTC (permalink / raw)
To: linux-mm, jgroves, ravis.opensrc, sthanneeru, emirakhur,
Hasan.Maruf
Cc: linux-doc, linux-fsdevel, linux-api, linux-arch, linux-kernel,
akpm, arnd, tglx, luto, mingo, bp, dave.hansen, x86, hpa, mhocko,
tj, ying.huang, gregory.price, corbet, rakie.kim, hyeongtak.ji,
honggyu.kim, vtavarespetr, peterz
In-Reply-To: <20231207002759.51418-1-gregory.price@memverge.com>
This patch adds the plumbing into mpol_new() to allow the argument
structure's home_node field to be set during mempolicy creation.
The syscall sys_set_mempolicy_home_node was added to allow a home
node to be registered for a vma.
For set_mempolicy2 and mbind2 syscalls, it would be useful to add
this as an extension to allow the user to submit a fully formed
mempolicy configuration in a single call, rather than require
multiple calls to configure a mempolicy.
This will become particularly useful if/when pidfd interfaces to
change process mempolicies from outside the task appear, as each
call to change the mempolicy does an atomic swap of that policy
in the task, rather than mutate the policy.
Signed-off-by: Gregory Price <gregory.price@memverge.com>
---
mm/mempolicy.c | 8 ++++++++
1 file changed, 8 insertions(+)
diff --git a/mm/mempolicy.c b/mm/mempolicy.c
index fecdc781b6a0..4be63547a4b3 100644
--- a/mm/mempolicy.c
+++ b/mm/mempolicy.c
@@ -311,6 +311,7 @@ static struct mempolicy *mpol_new(struct mempolicy_args *args)
policy->flags = flags;
policy->home_node = NUMA_NO_NODE;
policy->wil.cur_weight = 0;
+ policy->home_node = args->home_node;
return policy;
}
@@ -1624,6 +1625,7 @@ static long kernel_set_mempolicy(int mode, const unsigned long __user *nmask,
args.mode = lmode;
args.mode_flags = mode_flags;
args.policy_nodes = &nodes;
+ args.home_node = NUMA_NO_NODE;
return do_set_mempolicy(&args);
}
@@ -2967,6 +2969,8 @@ void mpol_shared_policy_init(struct shared_policy *sp, struct mempolicy *mpol)
margs.mode = mpol->mode;
margs.mode_flags = mpol->flags;
margs.policy_nodes = &mpol->w.user_nodemask;
+ margs.home_node = NUMA_NO_NODE;
+
/* contextualize the tmpfs mount point mempolicy to this file */
npol = mpol_new(&margs);
if (IS_ERR(npol))
@@ -3125,6 +3129,7 @@ void __init numa_policy_init(void)
memset(&args, 0, sizeof(args));
args.mode = MPOL_INTERLEAVE;
args.policy_nodes = &interleave_nodes;
+ args.home_node = NUMA_NO_NODE;
if (do_set_mempolicy(&args))
pr_err("%s: interleaving failed\n", __func__);
@@ -3139,6 +3144,7 @@ void numa_default_policy(void)
memset(&args, 0, sizeof(args));
args.mode = MPOL_DEFAULT;
+ args.home_node = NUMA_NO_NODE;
do_set_mempolicy(&args);
}
@@ -3261,6 +3267,8 @@ int mpol_parse_str(char *str, struct mempolicy **mpol)
margs.mode = mode;
margs.mode_flags = mode_flags;
margs.policy_nodes = &nodes;
+ margs.home_node = NUMA_NO_NODE;
+
new = mpol_new(&margs);
if (IS_ERR(new))
goto out;
--
2.39.1
^ permalink raw reply related
* [RFC PATCH 05/11] mm/mempolicy: refactor kernel_get_mempolicy for code re-use
From: Gregory Price @ 2023-12-07 0:27 UTC (permalink / raw)
To: linux-mm, jgroves, ravis.opensrc, sthanneeru, emirakhur,
Hasan.Maruf
Cc: linux-doc, linux-fsdevel, linux-api, linux-arch, linux-kernel,
akpm, arnd, tglx, luto, mingo, bp, dave.hansen, x86, hpa, mhocko,
tj, ying.huang, gregory.price, corbet, rakie.kim, hyeongtak.ji,
honggyu.kim, vtavarespetr, peterz
In-Reply-To: <20231207002759.51418-1-gregory.price@memverge.com>
Pull operation flag checking from inside do_get_mempolicy out
to kernel_get_mempolicy. This allows us to flatten the
internal code, and break it into separate functions for future
syscalls (get_mempolicy2, process_get_mempolicy) to re-use the
code, even after additional extensions are made.
The primary change is that the flag is treated as the multiplexer
that it actually is. For get_mempolicy, the flags represents 3
different primary operations:
if (flags & MPOL_F_MEMS_ALLOWED)
return task->mems_allowed
else if (flags & MPOL_F_ADDR)
return vma mempolicy information
else
return task mempolicy information
Plus the behavior modifying flag:
if (flags & MPOL_F_NODE)
change the return value of (int __user *policy)
based on whether MPOL_F_ADDR was set.
The original behavior of get_mempolicy is retained, but we utilize
the new mempolicy_args structure to pass the operations down the
stack. This will allow us to extend the internal functions without
affecting the legacy behavior of get_mempolicy.
Signed-off-by: Gregory Price <gregory.price@memverge.com>
---
mm/mempolicy.c | 240 ++++++++++++++++++++++++++++++-------------------
1 file changed, 150 insertions(+), 90 deletions(-)
diff --git a/mm/mempolicy.c b/mm/mempolicy.c
index 4c343218c033..fecdc781b6a0 100644
--- a/mm/mempolicy.c
+++ b/mm/mempolicy.c
@@ -898,106 +898,107 @@ static int lookup_node(struct mm_struct *mm, unsigned long addr)
return ret;
}
-/* Retrieve NUMA policy */
-static long do_get_mempolicy(int *policy, nodemask_t *nmask,
- unsigned long addr, unsigned long flags)
+/* Retrieve the mems_allowed for current task */
+static inline long do_get_mems_allowed(nodemask_t *nmask)
{
- int err;
- struct mm_struct *mm = current->mm;
- struct vm_area_struct *vma = NULL;
- struct mempolicy *pol = current->mempolicy, *pol_refcount = NULL;
+ task_lock(current);
+ *nmask = cpuset_current_mems_allowed;
+ task_unlock(current);
+ return 0;
+}
- if (flags &
- ~(unsigned long)(MPOL_F_NODE|MPOL_F_ADDR|MPOL_F_MEMS_ALLOWED))
- return -EINVAL;
+/* If the policy has additional node information to retrieve, return it */
+static long do_get_policy_node(struct mempolicy *pol)
+{
+ /*
+ * For MPOL_INTERLEAVE, the extended node information is the next
+ * node that will be selected for interleave. For weighted interleave
+ * we return the next node based on the current weight.
+ */
+ if (pol == current->mempolicy && pol->mode == MPOL_INTERLEAVE)
+ return next_node_in(current->il_prev, pol->nodes);
- if (flags & MPOL_F_MEMS_ALLOWED) {
- if (flags & (MPOL_F_NODE|MPOL_F_ADDR))
- return -EINVAL;
- *policy = 0; /* just so it's initialized */
+ if (pol == current->mempolicy &&
+ pol->mode == MPOL_WEIGHTED_INTERLEAVE) {
+ if (pol->wil.cur_weight)
+ return current->il_prev;
+ else
+ return next_node_in(current->il_prev, pol->nodes);
+ }
+ return -EINVAL;
+}
+
+/* Handle user_nodemask condition when fetching nodemask for userspace */
+static void do_get_mempolicy_nodemask(struct mempolicy *pol, nodemask_t *nmask)
+{
+ if (mpol_store_user_nodemask(pol)) {
+ *nmask = pol->w.user_nodemask;
+ } else {
task_lock(current);
- *nmask = cpuset_current_mems_allowed;
+ get_policy_nodemask(pol, nmask);
task_unlock(current);
- return 0;
}
+}
- if (flags & MPOL_F_ADDR) {
- pgoff_t ilx; /* ignored here */
- /*
- * Do NOT fall back to task policy if the
- * vma/shared policy at addr is NULL. We
- * want to return MPOL_DEFAULT in this case.
- */
- mmap_read_lock(mm);
- vma = vma_lookup(mm, addr);
- if (!vma) {
- mmap_read_unlock(mm);
- return -EFAULT;
- }
- pol = __get_vma_policy(vma, addr, &ilx);
- } else if (addr)
- return -EINVAL;
+/* Retrieve NUMA policy for a VMA assocated with a given address */
+static long do_get_vma_mempolicy(struct mempolicy_args *args)
+{
+ pgoff_t ilx;
+ struct mm_struct *mm = current->mm;
+ struct vm_area_struct *vma = NULL;
+ struct mempolicy *pol = NULL;
+ mmap_read_lock(mm);
+ vma = vma_lookup(mm, args->addr);
+ if (!vma) {
+ mmap_read_unlock(mm);
+ return -EFAULT;
+ }
+ pol = __get_vma_policy(vma, args->addr, &ilx);
if (!pol)
- pol = &default_policy; /* indicates default behavior */
+ pol = &default_policy;
+ /* this may cause a double-reference, resolved by a put+cond_put */
+ mpol_get(pol);
+ mmap_read_unlock(mm);
- if (flags & MPOL_F_NODE) {
- if (flags & MPOL_F_ADDR) {
- /*
- * Take a refcount on the mpol, because we are about to
- * drop the mmap_lock, after which only "pol" remains
- * valid, "vma" is stale.
- */
- pol_refcount = pol;
- vma = NULL;
- mpol_get(pol);
- mmap_read_unlock(mm);
- err = lookup_node(mm, addr);
- if (err < 0)
- goto out;
- *policy = err;
- } else if (pol == current->mempolicy &&
- pol->mode == MPOL_INTERLEAVE) {
- *policy = next_node_in(current->il_prev, pol->nodes);
- } else if (pol == current->mempolicy &&
- (pol->mode == MPOL_WEIGHTED_INTERLEAVE)) {
- if (pol->wil.cur_weight)
- *policy = current->il_prev;
- else
- *policy = next_node_in(current->il_prev,
- pol->nodes);
- } else {
- err = -EINVAL;
- goto out;
- }
- } else {
- *policy = pol == &default_policy ? MPOL_DEFAULT :
- pol->mode;
- /*
- * Internal mempolicy flags must be masked off before exposing
- * the policy to userspace.
- */
- *policy |= (pol->flags & MPOL_MODE_FLAGS);
- }
+ /* Fetch the node for the given address */
+ args->addr_node = lookup_node(mm, args->addr);
- err = 0;
- if (nmask) {
- if (mpol_store_user_nodemask(pol)) {
- *nmask = pol->w.user_nodemask;
- } else {
- task_lock(current);
- get_policy_nodemask(pol, nmask);
- task_unlock(current);
- }
+ args->mode = pol == &default_policy ? MPOL_DEFAULT : pol->mode;
+ args->mode_flags = (pol->flags & MPOL_MODE_FLAGS);
+
+ /* If this policy has extra node info, fetch that */
+ args->policy_node = do_get_policy_node(pol);
+
+ if (args->policy_nodes)
+ do_get_mempolicy_nodemask(pol, args->policy_nodes);
+
+ if (pol != &default_policy) {
+ mpol_put(pol);
+ mpol_cond_put(pol);
}
- out:
- mpol_cond_put(pol);
- if (vma)
- mmap_read_unlock(mm);
- if (pol_refcount)
- mpol_put(pol_refcount);
- return err;
+ return 0;
+}
+
+/* Retrieve NUMA policy for the current task */
+static long do_get_task_mempolicy(struct mempolicy_args *args)
+{
+ struct mempolicy *pol = current->mempolicy;
+
+ if (!pol)
+ pol = &default_policy; /* indicates default behavior */
+
+ args->mode = pol == &default_policy ? MPOL_DEFAULT : pol->mode;
+ /* Internal flags must be masked off before exposing to userspace */
+ args->mode_flags = (pol->flags & MPOL_MODE_FLAGS);
+
+ args->policy_node = do_get_policy_node(pol);
+
+ if (args->policy_nodes)
+ do_get_mempolicy_nodemask(pol, args->policy_nodes);
+
+ return 0;
}
#ifdef CONFIG_MIGRATION
@@ -1734,16 +1735,75 @@ static int kernel_get_mempolicy(int __user *policy,
unsigned long addr,
unsigned long flags)
{
+ struct mempolicy_args args;
int err;
- int pval;
+ int pval = 0;
nodemask_t nodes;
if (nmask != NULL && maxnode < nr_node_ids)
return -EINVAL;
- addr = untagged_addr(addr);
+ if (flags &
+ ~(unsigned long)(MPOL_F_NODE|MPOL_F_ADDR|MPOL_F_MEMS_ALLOWED))
+ return -EINVAL;
- err = do_get_mempolicy(&pval, &nodes, addr, flags);
+ /* Ensure any data that may be copied to userland is initialized */
+ memset(&args, 0, sizeof(args));
+ args.policy_nodes = &nodes;
+ args.addr = untagged_addr(addr);
+
+ /*
+ * set_mempolicy was originally multiplexed based on 3 flags:
+ * MPOL_F_MEMS_ALLOWED: fetch task->mems_allowed
+ * MPOL_F_ADDR : operate on vma->mempolicy
+ * MPOL_F_NODE : change return value of *policy
+ *
+ * Split this behavior out here, rather than internal functions,
+ * so that the internal functions can be re-used by future
+ * get_mempolicy2 interfaces and the arg structure made extensible
+ */
+ if (flags & MPOL_F_MEMS_ALLOWED) {
+ if (flags & (MPOL_F_NODE|MPOL_F_ADDR))
+ return -EINVAL;
+ pval = 0; /* just so it's initialized */
+ err = do_get_mems_allowed(&nodes);
+ } else if (flags & MPOL_F_ADDR) {
+ /* If F_ADDR, we operation on a vma policy (or default) */
+ err = do_get_vma_mempolicy(&args);
+ if (err)
+ return err;
+ /* if (F_ADDR | F_NODE), *pval is the address' node */
+ if (flags & MPOL_F_NODE) {
+ /* if we failed to fetch, that's likely an EFAULT */
+ if (args.addr_node < 0)
+ return args.addr_node;
+ pval = args.addr_node;
+ } else
+ pval = args.mode | args.mode_flags;
+ } else {
+ /* if not F_ADDR and addr != null, EINVAL */
+ if (addr)
+ return -EINVAL;
+
+ err = do_get_task_mempolicy(&args);
+ if (err)
+ return err;
+ /*
+ * if F_NODE was set and mode was MPOL_INTERLEAVE
+ * *pval is equal to next interleave node.
+ *
+ * if args.policy_node < 0, this means the mode did
+ * not have a policy. This presently emulates the
+ * original behavior of (F_NODE) & (!MPOL_INTERLEAVE)
+ * producing -EINVAL
+ */
+ if (flags & MPOL_F_NODE) {
+ if (args.policy_node < 0)
+ return args.policy_node;
+ pval = args.policy_node;
+ } else
+ pval = args.mode | args.mode_flags;
+ }
if (err)
return err;
--
2.39.1
^ permalink raw reply related
* [RFC PATCH 04/11] mm/mempolicy: create struct mempolicy_args for creating new mempolicies
From: Gregory Price @ 2023-12-07 0:27 UTC (permalink / raw)
To: linux-mm, jgroves, ravis.opensrc, sthanneeru, emirakhur,
Hasan.Maruf
Cc: linux-doc, linux-fsdevel, linux-api, linux-arch, linux-kernel,
akpm, arnd, tglx, luto, mingo, bp, dave.hansen, x86, hpa, mhocko,
tj, ying.huang, gregory.price, corbet, rakie.kim, hyeongtak.ji,
honggyu.kim, vtavarespetr, peterz
In-Reply-To: <20231207002759.51418-1-gregory.price@memverge.com>
This patch adds a new kernel structure `struct mempolicy_args`,
intended to be used for an extensible get/set_mempolicy interface.
This implements the fields required to support the existing syscall
interfaces interfaces, but does not expose any user-facing arg
structure.
mpol_new is refactored to take the argument structure so that future
mempolicy extensions can all be managed in the mempolicy constructor.
The get_mempolicy and mbind syscalls are refactored to utilize the
new argument structure, as are all the callers of mpol_new() and
do_set_mempolicy.
Signed-off-by: Gregory Price <gregory.price@memverge.com>
---
include/linux/mempolicy.h | 14 ++++++++
mm/mempolicy.c | 69 +++++++++++++++++++++++++++++----------
2 files changed, 65 insertions(+), 18 deletions(-)
diff --git a/include/linux/mempolicy.h b/include/linux/mempolicy.h
index ba09167e80f7..117c5395c6eb 100644
--- a/include/linux/mempolicy.h
+++ b/include/linux/mempolicy.h
@@ -61,6 +61,20 @@ struct mempolicy {
} wil;
};
+/*
+ * Describes settings of a mempolicy during set/get syscalls and
+ * kernel internal calls to do_set_mempolicy()
+ */
+struct mempolicy_args {
+ unsigned short mode; /* policy mode */
+ unsigned short mode_flags; /* policy mode flags */
+ nodemask_t *policy_nodes; /* get/set/mbind */
+ int policy_node; /* get: policy node information */
+ unsigned long addr; /* get: vma address */
+ int addr_node; /* get: node the address belongs to */
+ int home_node; /* mbind: use MPOL_MF_HOME_NODE */
+};
+
/*
* Support for managing mempolicy data objects (clone, copy, destroy)
* The default fast path of a NULL MPOL_DEFAULT policy is always inlined.
diff --git a/mm/mempolicy.c b/mm/mempolicy.c
index eec807d0c6a1..4c343218c033 100644
--- a/mm/mempolicy.c
+++ b/mm/mempolicy.c
@@ -268,10 +268,12 @@ static int mpol_set_nodemask(struct mempolicy *pol,
* This function just creates a new policy, does some check and simple
* initialization. You must invoke mpol_set_nodemask() to set nodes.
*/
-static struct mempolicy *mpol_new(unsigned short mode, unsigned short flags,
- nodemask_t *nodes)
+static struct mempolicy *mpol_new(struct mempolicy_args *args)
{
struct mempolicy *policy;
+ unsigned short mode = args->mode;
+ unsigned short flags = args->mode_flags;
+ nodemask_t *nodes = args->policy_nodes;
if (mode == MPOL_DEFAULT) {
if (nodes && !nodes_empty(*nodes))
@@ -820,8 +822,7 @@ static int mbind_range(struct vma_iterator *vmi, struct vm_area_struct *vma,
}
/* Set the process memory policy */
-static long do_set_mempolicy(unsigned short mode, unsigned short flags,
- nodemask_t *nodes)
+static long do_set_mempolicy(struct mempolicy_args *args)
{
struct mempolicy *new, *old;
NODEMASK_SCRATCH(scratch);
@@ -830,14 +831,14 @@ static long do_set_mempolicy(unsigned short mode, unsigned short flags,
if (!scratch)
return -ENOMEM;
- new = mpol_new(mode, flags, nodes);
+ new = mpol_new(args);
if (IS_ERR(new)) {
ret = PTR_ERR(new);
goto out;
}
task_lock(current);
- ret = mpol_set_nodemask(new, nodes, scratch);
+ ret = mpol_set_nodemask(new, args->policy_nodes, scratch);
if (ret) {
task_unlock(current);
mpol_put(new);
@@ -1235,8 +1236,7 @@ static struct folio *alloc_migration_target_by_mpol(struct folio *src,
#endif
static long do_mbind(unsigned long start, unsigned long len,
- unsigned short mode, unsigned short mode_flags,
- nodemask_t *nmask, unsigned long flags)
+ struct mempolicy_args *margs, unsigned long flags)
{
struct mm_struct *mm = current->mm;
struct vm_area_struct *vma, *prev;
@@ -1256,7 +1256,7 @@ static long do_mbind(unsigned long start, unsigned long len,
if (start & ~PAGE_MASK)
return -EINVAL;
- if (mode == MPOL_DEFAULT)
+ if (margs->mode == MPOL_DEFAULT)
flags &= ~MPOL_MF_STRICT;
len = PAGE_ALIGN(len);
@@ -1267,7 +1267,7 @@ static long do_mbind(unsigned long start, unsigned long len,
if (end == start)
return 0;
- new = mpol_new(mode, mode_flags, nmask);
+ new = mpol_new(margs);
if (IS_ERR(new))
return PTR_ERR(new);
@@ -1284,7 +1284,8 @@ static long do_mbind(unsigned long start, unsigned long len,
NODEMASK_SCRATCH(scratch);
if (scratch) {
mmap_write_lock(mm);
- err = mpol_set_nodemask(new, nmask, scratch);
+ err = mpol_set_nodemask(new, margs->policy_nodes,
+ scratch);
if (err)
mmap_write_unlock(mm);
} else
@@ -1298,7 +1299,7 @@ static long do_mbind(unsigned long start, unsigned long len,
* Lock the VMAs before scanning for pages to migrate,
* to ensure we don't miss a concurrently inserted page.
*/
- nr_failed = queue_pages_range(mm, start, end, nmask,
+ nr_failed = queue_pages_range(mm, start, end, margs->policy_nodes,
flags | MPOL_MF_INVERT | MPOL_MF_WRLOCK, &pagelist);
if (nr_failed < 0) {
@@ -1503,6 +1504,7 @@ static long kernel_mbind(unsigned long start, unsigned long len,
unsigned long mode, const unsigned long __user *nmask,
unsigned long maxnode, unsigned int flags)
{
+ struct mempolicy_args margs;
unsigned short mode_flags;
nodemask_t nodes;
int lmode = mode;
@@ -1517,7 +1519,12 @@ static long kernel_mbind(unsigned long start, unsigned long len,
if (err)
return err;
- return do_mbind(start, len, lmode, mode_flags, &nodes, flags);
+ memset(&margs, 0, sizeof(margs));
+ margs.mode = lmode;
+ margs.mode_flags = mode_flags;
+ margs.policy_nodes = &nodes;
+
+ return do_mbind(start, len, &margs, flags);
}
SYSCALL_DEFINE4(set_mempolicy_home_node, unsigned long, start, unsigned long, len,
@@ -1598,6 +1605,7 @@ SYSCALL_DEFINE6(mbind, unsigned long, start, unsigned long, len,
static long kernel_set_mempolicy(int mode, const unsigned long __user *nmask,
unsigned long maxnode)
{
+ struct mempolicy_args args;
unsigned short mode_flags;
nodemask_t nodes;
int lmode = mode;
@@ -1611,7 +1619,12 @@ static long kernel_set_mempolicy(int mode, const unsigned long __user *nmask,
if (err)
return err;
- return do_set_mempolicy(lmode, mode_flags, &nodes);
+ memset(&args, 0, sizeof(args));
+ args.mode = lmode;
+ args.mode_flags = mode_flags;
+ args.policy_nodes = &nodes;
+
+ return do_set_mempolicy(&args);
}
SYSCALL_DEFINE3(set_mempolicy, int, mode, const unsigned long __user *, nmask,
@@ -2877,6 +2890,7 @@ static int shared_policy_replace(struct shared_policy *sp, pgoff_t start,
void mpol_shared_policy_init(struct shared_policy *sp, struct mempolicy *mpol)
{
int ret;
+ struct mempolicy_args margs;
sp->root = RB_ROOT; /* empty tree == default mempolicy */
rwlock_init(&sp->lock);
@@ -2889,8 +2903,12 @@ void mpol_shared_policy_init(struct shared_policy *sp, struct mempolicy *mpol)
if (!scratch)
goto put_mpol;
+ memset(&margs, 0, sizeof(margs));
+ margs.mode = mpol->mode;
+ margs.mode_flags = mpol->flags;
+ margs.policy_nodes = &mpol->w.user_nodemask;
/* contextualize the tmpfs mount point mempolicy to this file */
- npol = mpol_new(mpol->mode, mpol->flags, &mpol->w.user_nodemask);
+ npol = mpol_new(&margs);
if (IS_ERR(npol))
goto free_scratch; /* no valid nodemask intersection */
@@ -2998,6 +3016,7 @@ static inline void __init check_numabalancing_enable(void)
void __init numa_policy_init(void)
{
+ struct mempolicy_args args;
nodemask_t interleave_nodes;
unsigned long largest = 0;
int nid, prefer = 0;
@@ -3043,7 +3062,11 @@ void __init numa_policy_init(void)
if (unlikely(nodes_empty(interleave_nodes)))
node_set(prefer, interleave_nodes);
- if (do_set_mempolicy(MPOL_INTERLEAVE, 0, &interleave_nodes))
+ memset(&args, 0, sizeof(args));
+ args.mode = MPOL_INTERLEAVE;
+ args.policy_nodes = &interleave_nodes;
+
+ if (do_set_mempolicy(&args))
pr_err("%s: interleaving failed\n", __func__);
check_numabalancing_enable();
@@ -3052,7 +3075,12 @@ void __init numa_policy_init(void)
/* Reset policy of current process to default */
void numa_default_policy(void)
{
- do_set_mempolicy(MPOL_DEFAULT, 0, NULL);
+ struct mempolicy_args args;
+
+ memset(&args, 0, sizeof(args));
+ args.mode = MPOL_DEFAULT;
+
+ do_set_mempolicy(&args);
}
/*
@@ -3082,6 +3110,7 @@ static const char * const policy_modes[] =
*/
int mpol_parse_str(char *str, struct mempolicy **mpol)
{
+ struct mempolicy_args margs;
struct mempolicy *new = NULL;
unsigned short mode_flags;
nodemask_t nodes;
@@ -3168,7 +3197,11 @@ int mpol_parse_str(char *str, struct mempolicy **mpol)
goto out;
}
- new = mpol_new(mode, mode_flags, &nodes);
+ memset(&margs, 0, sizeof(margs));
+ margs.mode = mode;
+ margs.mode_flags = mode_flags;
+ margs.policy_nodes = &nodes;
+ new = mpol_new(&margs);
if (IS_ERR(new))
goto out;
--
2.39.1
^ permalink raw reply related
* [RFC PATCH 03/11] mm/mempolicy: refactor sanitize_mpol_flags for reuse
From: Gregory Price @ 2023-12-07 0:27 UTC (permalink / raw)
To: linux-mm, jgroves, ravis.opensrc, sthanneeru, emirakhur,
Hasan.Maruf
Cc: linux-doc, linux-fsdevel, linux-api, linux-arch, linux-kernel,
akpm, arnd, tglx, luto, mingo, bp, dave.hansen, x86, hpa, mhocko,
tj, ying.huang, gregory.price, corbet, rakie.kim, hyeongtak.ji,
honggyu.kim, vtavarespetr, peterz
In-Reply-To: <20231207002759.51418-1-gregory.price@memverge.com>
split sanitize_mpol_flags into sanitize and validate.
Sanitize is used by set_mempolicy to split (int mode) into mode
and mode_flags, and then validates them.
Validate validates already split flags.
Validate will be reused for new syscalls that accept already
split mode and mode_flags.
Signed-off-by: Gregory Price <gregory.price@memverge.com>
---
mm/mempolicy.c | 29 ++++++++++++++++++++++-------
1 file changed, 22 insertions(+), 7 deletions(-)
diff --git a/mm/mempolicy.c b/mm/mempolicy.c
index 65e0334a1a18..eec807d0c6a1 100644
--- a/mm/mempolicy.c
+++ b/mm/mempolicy.c
@@ -1466,24 +1466,39 @@ static int copy_nodes_to_user(unsigned long __user *mask, unsigned long maxnode,
return copy_to_user(mask, nodes_addr(*nodes), copy) ? -EFAULT : 0;
}
-/* Basic parameter sanity check used by both mbind() and set_mempolicy() */
-static inline int sanitize_mpol_flags(int *mode, unsigned short *flags)
+/*
+ * Basic parameter sanity check used by mbind/set_mempolicy
+ * May modify flags to include internal flags (e.g. MPOL_F_MOF/F_MORON)
+ */
+static inline int validate_mpol_flags(unsigned short mode, unsigned short *flags)
{
- *flags = *mode & MPOL_MODE_FLAGS;
- *mode &= ~MPOL_MODE_FLAGS;
-
- if ((unsigned int)(*mode) >= MPOL_MAX)
+ if ((unsigned int)(mode) >= MPOL_MAX)
return -EINVAL;
if ((*flags & MPOL_F_STATIC_NODES) && (*flags & MPOL_F_RELATIVE_NODES))
return -EINVAL;
if (*flags & MPOL_F_NUMA_BALANCING) {
- if (*mode != MPOL_BIND)
+ if (mode != MPOL_BIND)
return -EINVAL;
*flags |= (MPOL_F_MOF | MPOL_F_MORON);
}
return 0;
}
+/*
+ * Used by mbind/set_memplicy to split and validate mode/flags
+ * set_mempolicy combines (mode | flags), split them out into separate
+ * fields and return just the mode in mode_arg and flags in flags.
+ */
+static inline int sanitize_mpol_flags(int *mode_arg, unsigned short *flags)
+{
+ unsigned short mode = (*mode_arg & ~MPOL_MODE_FLAGS);
+
+ *flags = *mode_arg & MPOL_MODE_FLAGS;
+ *mode_arg = mode;
+
+ return validate_mpol_flags(mode, flags);
+}
+
static long kernel_mbind(unsigned long start, unsigned long len,
unsigned long mode, const unsigned long __user *nmask,
unsigned long maxnode, unsigned int flags)
--
2.39.1
^ permalink raw reply related
* [RFC PATCH 02/11] mm/mempolicy: introduce MPOL_WEIGHTED_INTERLEAVE for weighted interleaving
From: Gregory Price @ 2023-12-07 0:27 UTC (permalink / raw)
To: linux-mm, jgroves, ravis.opensrc, sthanneeru, emirakhur,
Hasan.Maruf
Cc: linux-doc, linux-fsdevel, linux-api, linux-arch, linux-kernel,
akpm, arnd, tglx, luto, mingo, bp, dave.hansen, x86, hpa, mhocko,
tj, ying.huang, gregory.price, corbet, rakie.kim, hyeongtak.ji,
honggyu.kim, vtavarespetr, peterz, Srinivasulu Thanneeru
In-Reply-To: <20231207002759.51418-1-gregory.price@memverge.com>
From: Rakie Kim <rakie.kim@sk.com>
When a system has multiple NUMA nodes and it becomes bandwidth hungry,
the current MPOL_INTERLEAVE could be an wise option.
However, if those NUMA nodes consist of different types of memory such
as having local DRAM and CXL memory together, the current round-robin
based interleaving policy doesn't maximize the overall bandwidth because
of their different bandwidth characteristics.
Instead, the interleaving can be more efficient when the allocation
policy follows each NUMA nodes' bandwidth weight rather than having 1:1
round-robin allocation.
This patch introduces a new memory policy, MPOL_WEIGHTED_INTERLEAVE, which
enables weighted interleaving between NUMA nodes. Weighted interleave
allows for a proportional distribution of memory across multiple numa
nodes, preferablly apportioned to match the bandwidth capacity of each
node from the perspective of the accessing node.
For example, if a system has 1 CPU node (0), and 2 memory nodes (0,1),
with a relative bandwidth of (100GB/s, 50GB/s) respectively, the
appropriate weight distribution is (2:1).
Weights will be acquired from the global weight matrix exposed by the
sysfs extension: /sys/kernel/mm/mempolicy/weighted_interleave/
The policy will then allocate the number of pages according to the
set weights. For example, if the weights are (2,1), then 2 pages
will be allocated on node0 for every 1 page allocated on node1.
The new flag MPOL_WEIGHTED_INTERLEAVE can be used in set_mempolicy(2)
and mbind(2).
There are 3 integration points:
weighted_interleave_nodes:
Counts the number of allocations as they occur, and applies the
weight for the current node. When the weight reaches 0, switch
to the next node. Applied by `mempolicy_slab_node()` and
`policy_nodemask()`
weighted_interleave_nid:
Gets the total weight of the nodemask as well as each individual
node weight, then calculates the node based on the given index.
Applied by `policy_nodemask()` and `mpol_misplaced()`
bulk_array_weighted_interleave:
Gets the total weight of the nodemask as well as each individual
node weight, then calculates the number of "interleave rounds" as
well as any delta ("partial round"). Calculates the number of
pages for each node and allocates them.
If a node was scheduled for interleave via interleave_nodes, the
current weight (pol->cur_weight) will be allocated first, before
the remaining bulk calculation is done. This simplifies the
calculation at the cost of an additional allocation call.
One piece of complexity is the interaction between a recent refactor
which split the logic to acquire the "ilx" (interleave index) of an
allocation and the actually application of the interleave. The
calculation of the `interleave index` is done by `get_vma_policy()`,
while the actual selection of the node will be later appliex by the
relevant weighted_interleave function.
Suggested-by: Hasan Al Maruf <Hasan.Maruf@amd.com>
Signed-off-by: Rakie Kim <rakie.kim@sk.com>
Co-developed-by: Honggyu Kim <honggyu.kim@sk.com>
Signed-off-by: Honggyu Kim <honggyu.kim@sk.com>
Co-developed-by: Hyeongtak Ji <hyeongtak.ji@sk.com>
Signed-off-by: Hyeongtak Ji <hyeongtak.ji@sk.com>
Co-developed-by: Gregory Price <gregory.price@memverge.com>
Signed-off-by: Gregory Price <gregory.price@memverge.com>
Co-developed-by: Srinivasulu Thanneeru <sthanneeru.opensrc@micron.com>
Signed-off-by: Srinivasulu Thanneeru <sthanneeru.opensrc@micron.com>
Co-developed-by: Ravi Jonnalagadda <ravis.opensrc@micron.com>
Signed-off-by: Ravi Jonnalagadda <ravis.opensrc@micron.com>
---
.../admin-guide/mm/numa_memory_policy.rst | 17 ++
include/linux/mempolicy.h | 5 +
include/uapi/linux/mempolicy.h | 1 +
mm/mempolicy.c | 181 +++++++++++++++++-
4 files changed, 201 insertions(+), 3 deletions(-)
diff --git a/Documentation/admin-guide/mm/numa_memory_policy.rst b/Documentation/admin-guide/mm/numa_memory_policy.rst
index eca38fa81e0f..b7b8d3dd420f 100644
--- a/Documentation/admin-guide/mm/numa_memory_policy.rst
+++ b/Documentation/admin-guide/mm/numa_memory_policy.rst
@@ -250,6 +250,23 @@ MPOL_PREFERRED_MANY
can fall back to all existing numa nodes. This is effectively
MPOL_PREFERRED allowed for a mask rather than a single node.
+MPOL_WEIGHTED_INTERLEAVE
+ This mode operates the same as MPOL_INTERLEAVE, except that
+ interleaving behavior is executed based on weights set in
+ /sys/kernel/mm/mempolicy/weighted_interleave/
+ rather than simple round-robin interleave (which is the default).
+
+ When utilizing global weights from the sysfs interface,
+ weights are applied in a src-node relative manner. For example
+ a task executing on node0 will use the weights from
+ /sys/kernel/mm/mempolicy/weighted_interleave/node0/
+ while a task executing on node1 will use the weights from
+ /sys/kernel/mm/mempolicy/weighted_interleave/node1/
+
+ This allows for tasks migrated between nodes (for example
+ cgroup initiated migrations) to re-weight for the optimal
+ distribution of bandwidth.
+
NUMA memory policy supports the following optional mode flags:
MPOL_F_STATIC_NODES
diff --git a/include/linux/mempolicy.h b/include/linux/mempolicy.h
index 931b118336f4..ba09167e80f7 100644
--- a/include/linux/mempolicy.h
+++ b/include/linux/mempolicy.h
@@ -54,6 +54,11 @@ struct mempolicy {
nodemask_t cpuset_mems_allowed; /* relative to these nodes */
nodemask_t user_nodemask; /* nodemask passed by user */
} w;
+
+ /* Weighted interleave settings */
+ struct {
+ unsigned char cur_weight;
+ } wil;
};
/*
diff --git a/include/uapi/linux/mempolicy.h b/include/uapi/linux/mempolicy.h
index a8963f7ef4c2..1f9bb10d1a47 100644
--- a/include/uapi/linux/mempolicy.h
+++ b/include/uapi/linux/mempolicy.h
@@ -23,6 +23,7 @@ enum {
MPOL_INTERLEAVE,
MPOL_LOCAL,
MPOL_PREFERRED_MANY,
+ MPOL_WEIGHTED_INTERLEAVE,
MPOL_MAX, /* always last member of enum */
};
diff --git a/mm/mempolicy.c b/mm/mempolicy.c
index ce332b5e7a03..65e0334a1a18 100644
--- a/mm/mempolicy.c
+++ b/mm/mempolicy.c
@@ -308,6 +308,7 @@ static struct mempolicy *mpol_new(unsigned short mode, unsigned short flags,
policy->mode = mode;
policy->flags = flags;
policy->home_node = NUMA_NO_NODE;
+ policy->wil.cur_weight = 0;
return policy;
}
@@ -420,6 +421,10 @@ static const struct mempolicy_operations mpol_ops[MPOL_MAX] = {
.create = mpol_new_nodemask,
.rebind = mpol_rebind_preferred,
},
+ [MPOL_WEIGHTED_INTERLEAVE] = {
+ .create = mpol_new_nodemask,
+ .rebind = mpol_rebind_nodemask,
+ },
};
static bool migrate_folio_add(struct folio *folio, struct list_head *foliolist,
@@ -841,7 +846,8 @@ static long do_set_mempolicy(unsigned short mode, unsigned short flags,
old = current->mempolicy;
current->mempolicy = new;
- if (new && new->mode == MPOL_INTERLEAVE)
+ if (new && (new->mode == MPOL_INTERLEAVE ||
+ new->mode == MPOL_WEIGHTED_INTERLEAVE))
current->il_prev = MAX_NUMNODES-1;
task_unlock(current);
mpol_put(old);
@@ -867,6 +873,7 @@ static void get_policy_nodemask(struct mempolicy *pol, nodemask_t *nodes)
case MPOL_INTERLEAVE:
case MPOL_PREFERRED:
case MPOL_PREFERRED_MANY:
+ case MPOL_WEIGHTED_INTERLEAVE:
*nodes = pol->nodes;
break;
case MPOL_LOCAL:
@@ -951,6 +958,13 @@ static long do_get_mempolicy(int *policy, nodemask_t *nmask,
} else if (pol == current->mempolicy &&
pol->mode == MPOL_INTERLEAVE) {
*policy = next_node_in(current->il_prev, pol->nodes);
+ } else if (pol == current->mempolicy &&
+ (pol->mode == MPOL_WEIGHTED_INTERLEAVE)) {
+ if (pol->wil.cur_weight)
+ *policy = current->il_prev;
+ else
+ *policy = next_node_in(current->il_prev,
+ pol->nodes);
} else {
err = -EINVAL;
goto out;
@@ -1780,7 +1794,8 @@ struct mempolicy *get_vma_policy(struct vm_area_struct *vma,
pol = __get_vma_policy(vma, addr, ilx);
if (!pol)
pol = get_task_policy(current);
- if (pol->mode == MPOL_INTERLEAVE) {
+ if (pol->mode == MPOL_INTERLEAVE ||
+ pol->mode == MPOL_WEIGHTED_INTERLEAVE) {
*ilx += vma->vm_pgoff >> order;
*ilx += (addr - vma->vm_start) >> (PAGE_SHIFT + order);
}
@@ -1830,6 +1845,24 @@ bool apply_policy_zone(struct mempolicy *policy, enum zone_type zone)
return zone >= dynamic_policy_zone;
}
+static unsigned int weighted_interleave_nodes(struct mempolicy *policy)
+{
+ unsigned int next;
+ struct task_struct *me = current;
+
+ if (policy->wil.cur_weight > 0) {
+ policy->wil.cur_weight--;
+ return me->il_prev;
+ }
+
+ next = next_node_in(me->il_prev, policy->nodes);
+ if (next < MAX_NUMNODES) {
+ me->il_prev = next;
+ policy->wil.cur_weight = iw_table[numa_node_id()].weights[next];
+ }
+ return next;
+}
+
/* Do dynamic interleaving for a process */
static unsigned int interleave_nodes(struct mempolicy *policy)
{
@@ -1864,6 +1897,9 @@ unsigned int mempolicy_slab_node(void)
case MPOL_INTERLEAVE:
return interleave_nodes(policy);
+ case MPOL_WEIGHTED_INTERLEAVE:
+ return weighted_interleave_nodes(policy);
+
case MPOL_BIND:
case MPOL_PREFERRED_MANY:
{
@@ -1888,6 +1924,34 @@ unsigned int mempolicy_slab_node(void)
}
}
+static unsigned int weighted_interleave_nid(struct mempolicy *pol, pgoff_t ilx)
+{
+ nodemask_t nodemask = pol->nodes;
+ unsigned int target, weight_total = 0;
+ int nid, local_node = numa_node_id();
+ unsigned char weights[MAX_NUMNODES];
+ unsigned char weight;
+
+ barrier();
+
+ /* Collect weights and save them on stack so they don't change */
+ for_each_node_mask(nid, nodemask) {
+ weight = iw_table[local_node].weights[nid];
+ weight_total += weight;
+ weights[nid] = weight;
+ }
+
+ target = (unsigned int)ilx % weight_total;
+
+ for_each_node_mask(nid, nodemask) {
+ weight = weights[nid];
+ if (target < weight)
+ return nid;
+ target -= weight;
+ }
+ return nid;
+}
+
/*
* Do static interleaving for interleave index @ilx. Returns the ilx'th
* node in pol->nodes (starting from ilx=0), wrapping around if ilx
@@ -1956,6 +2020,11 @@ static nodemask_t *policy_nodemask(gfp_t gfp, struct mempolicy *pol,
*nid = (ilx == NO_INTERLEAVE_INDEX) ?
interleave_nodes(pol) : interleave_nid(pol, ilx);
break;
+ case MPOL_WEIGHTED_INTERLEAVE:
+ *nid = (ilx == NO_INTERLEAVE_INDEX) ?
+ weighted_interleave_nodes(pol) :
+ weighted_interleave_nid(pol, ilx);
+ break;
}
return nodemask;
@@ -2017,6 +2086,7 @@ bool init_nodemask_of_mempolicy(nodemask_t *mask)
case MPOL_PREFERRED_MANY:
case MPOL_BIND:
case MPOL_INTERLEAVE:
+ case MPOL_WEIGHTED_INTERLEAVE:
*mask = mempolicy->nodes;
break;
@@ -2116,7 +2186,8 @@ struct page *alloc_pages_mpol(gfp_t gfp, unsigned int order,
* If the policy is interleave or does not allow the current
* node in its nodemask, we allocate the standard way.
*/
- if (pol->mode != MPOL_INTERLEAVE &&
+ if ((pol->mode != MPOL_INTERLEAVE &&
+ pol->mode != MPOL_WEIGHTED_INTERLEAVE) &&
(!nodemask || node_isset(nid, *nodemask))) {
/*
* First, try to allocate THP only on local node, but
@@ -2252,6 +2323,97 @@ static unsigned long alloc_pages_bulk_array_interleave(gfp_t gfp,
return total_allocated;
}
+static unsigned long alloc_pages_bulk_array_weighted_interleave(gfp_t gfp,
+ struct mempolicy *pol, unsigned long nr_pages,
+ struct page **page_array)
+{
+ struct task_struct *me = current;
+ unsigned long total_allocated = 0;
+ unsigned long nr_allocated;
+ unsigned long rounds;
+ unsigned long node_pages, delta;
+ unsigned char weight;
+ unsigned char weights[MAX_NUMNODES];
+ unsigned int weight_total;
+ nodemask_t nodes = pol->nodes;
+ int nnodes, node, prev_node;
+ int i;
+
+ /* Stabilize the nodemask on the stack */
+ barrier();
+
+ nnodes = nodes_weight(nodes);
+
+ /* Collect weights and save them on stack so they don't change */
+ for_each_node_mask(node, nodes) {
+ weight = iw_table[numa_node_id()].weights[node];
+ weight_total += weight;
+ weights[node] = weight;
+ }
+
+ /* Continue allocating from most recent node and adjust the nr_pages */
+ if (pol->wil.cur_weight) {
+ node = next_node_in(me->il_prev, nodes);
+ node_pages = pol->wil.cur_weight;
+ nr_allocated = __alloc_pages_bulk(gfp, node, NULL, node_pages,
+ NULL, page_array);
+ page_array += nr_allocated;
+ total_allocated += nr_allocated;
+ /* if that's all the pages, no need to interleave */
+ if (nr_pages <= pol->wil.cur_weight) {
+ pol->wil.cur_weight -= nr_pages;
+ return total_allocated;
+ }
+ /* Otherwise we adjust nr_pages down, and continue from there */
+ nr_pages -= pol->wil.cur_weight;
+ pol->wil.cur_weight = 0;
+ prev_node = node;
+ }
+
+ /* Now we can continue allocating as if from 0 instead of an offset */
+ rounds = nr_pages / weight_total;
+ delta = nr_pages % weight_total;
+ for (i = 0; i < nnodes; i++) {
+ node = next_node_in(prev_node, nodes);
+ weight = weights[node];
+ node_pages = weight * rounds;
+ if (delta) {
+ if (delta > weight) {
+ node_pages += weight;
+ delta -= weight;
+ } else {
+ node_pages += delta;
+ delta = 0;
+ }
+ }
+ /* We may not make it all the way around */
+ if (!node_pages)
+ break;
+ nr_allocated = __alloc_pages_bulk(gfp, node, NULL, node_pages,
+ NULL, page_array);
+ page_array += nr_allocated;
+ total_allocated += nr_allocated;
+ prev_node = node;
+ }
+
+ /*
+ * Finally, we need to update me->il_prev and pol->wil.cur_weight
+ * if there were overflow pages, but not equivalent to the node
+ * weight, set the cur_weight to node_weight - delta and the
+ * me->il_prev to the previous node. Otherwise if it was perfect
+ * we can simply set il_prev to node and cur_weight to 0
+ */
+ if (node_pages) {
+ me->il_prev = prev_node;
+ pol->wil.cur_weight = weight - node_pages;
+ } else {
+ me->il_prev = node;
+ pol->wil.cur_weight = 0;
+ }
+
+ return total_allocated;
+}
+
static unsigned long alloc_pages_bulk_array_preferred_many(gfp_t gfp, int nid,
struct mempolicy *pol, unsigned long nr_pages,
struct page **page_array)
@@ -2292,6 +2454,11 @@ unsigned long alloc_pages_bulk_array_mempolicy(gfp_t gfp,
return alloc_pages_bulk_array_interleave(gfp, pol,
nr_pages, page_array);
+ if (pol->mode == MPOL_WEIGHTED_INTERLEAVE)
+ return alloc_pages_bulk_array_weighted_interleave(gfp, pol,
+ nr_pages,
+ page_array);
+
if (pol->mode == MPOL_PREFERRED_MANY)
return alloc_pages_bulk_array_preferred_many(gfp,
numa_node_id(), pol, nr_pages, page_array);
@@ -2367,6 +2534,7 @@ bool __mpol_equal(struct mempolicy *a, struct mempolicy *b)
case MPOL_INTERLEAVE:
case MPOL_PREFERRED:
case MPOL_PREFERRED_MANY:
+ case MPOL_WEIGHTED_INTERLEAVE:
return !!nodes_equal(a->nodes, b->nodes);
case MPOL_LOCAL:
return true;
@@ -2503,6 +2671,10 @@ int mpol_misplaced(struct folio *folio, struct vm_area_struct *vma,
polnid = interleave_nid(pol, ilx);
break;
+ case MPOL_WEIGHTED_INTERLEAVE:
+ polnid = weighted_interleave_nid(pol, ilx);
+ break;
+
case MPOL_PREFERRED:
if (node_isset(curnid, pol->nodes))
goto out;
@@ -2877,6 +3049,7 @@ static const char * const policy_modes[] =
[MPOL_PREFERRED] = "prefer",
[MPOL_BIND] = "bind",
[MPOL_INTERLEAVE] = "interleave",
+ [MPOL_WEIGHTED_INTERLEAVE] = "weighted interleave",
[MPOL_LOCAL] = "local",
[MPOL_PREFERRED_MANY] = "prefer (many)",
};
@@ -2936,6 +3109,7 @@ int mpol_parse_str(char *str, struct mempolicy **mpol)
}
break;
case MPOL_INTERLEAVE:
+ case MPOL_WEIGHTED_INTERLEAVE:
/*
* Default to online nodes with memory if no nodelist
*/
@@ -3046,6 +3220,7 @@ void mpol_to_str(char *buffer, int maxlen, struct mempolicy *pol)
case MPOL_PREFERRED_MANY:
case MPOL_BIND:
case MPOL_INTERLEAVE:
+ case MPOL_WEIGHTED_INTERLEAVE:
nodes = pol->nodes;
break;
default:
--
2.39.1
^ permalink raw reply related
* [RFC PATCH 01/11] mm/mempolicy: implement the sysfs-based weighted_interleave interface
From: Gregory Price @ 2023-12-07 0:27 UTC (permalink / raw)
To: linux-mm, jgroves, ravis.opensrc, sthanneeru, emirakhur,
Hasan.Maruf
Cc: linux-doc, linux-fsdevel, linux-api, linux-arch, linux-kernel,
akpm, arnd, tglx, luto, mingo, bp, dave.hansen, x86, hpa, mhocko,
tj, ying.huang, gregory.price, corbet, rakie.kim, hyeongtak.ji,
honggyu.kim, vtavarespetr, peterz
In-Reply-To: <20231207002759.51418-1-gregory.price@memverge.com>
From: Rakie Kim <rakie.kim@sk.com>
This patch provides a way to set interleave weight information under
sysfs at /sys/kernel/mm/mempolicy/weighted_interleave/node*/node*/weight
The sysfs structure is designed as follows.
$ tree /sys/kernel/mm/mempolicy/
/sys/kernel/mm/mempolicy/ [1]
├── cpu_nodes [2]
├── possible_nodes [3]
└── weighted_interleave [4]
├── node0 [5]
│ ├── node0 [6]
│ │ └── weight [7]
│ └── node1
│ └── weight
└── node1
├── node0
│ └── weight
└── node1
└── weight
Each file above can be explained as follows.
[1] mm/mempolicy: configuration interface for mempolicy subsystem
[2] cpu_nodes: list of cpu nodes
information interface which is used to describe which nodes
may generate sub-folders under each policy interface. For example,
the weighted_interleave policy generates a nodeN folder for each
cpu node.
[3] possible_nodes: list of possible nodes
informational interface which may be used across multiple memory
policy configurations. Lists the `possible` nodes for which
configurations may be required. A `possible` node is one which has
been reserved by the kernel at boot, but may or may not be online.
For example, the weighted_interleave policy generates a nodeN/nodeM
folder for each cpu node and memory node combination [N,M].
[4] weighted_interleave/: config interface for weighted interleave policy
[5] weighted_interleave/nodeN/: initiator node configurations
Each CPU node receives its own weighting table, allowing for (src,dst)
weighting to be accomplished, where src is the cpu node the task is
running on, and dst is an index into the array of weights for that
source node.
[6] weighted_interleave/nodeN/nodeM/: memory node configurations
[7] weighted_interleave/nodeN/nodeM/weight: weight for [N,M]
The weight table for nodeN which can be programmed to weight each
target (nodeM) differently. This is important for allowing re-weight
to occur automatically on a task migration event, either via scheduler
initiated migration or a cgroup.cpusets/mems_allowed policy change.
Signed-off-by: Rakie Kim <rakie.kim@sk.com>
Signed-off-by: Honggyu Kim <honggyu.kim@sk.com>
Co-developed-by: Gregory Price <gregory.price@memverge.com>
Signed-off-by: Gregory Price <gregory.price@memverge.com>
Co-developed-by: Hyeongtak Ji <hyeongtak.ji@sk.com>
Signed-off-by: Hyeongtak Ji <hyeongtak.ji@sk.com>
---
.../ABI/testing/sysfs-kernel-mm-mempolicy | 33 +++
...fs-kernel-mm-mempolicy-weighted-interleave | 35 +++
mm/mempolicy.c | 226 ++++++++++++++++++
3 files changed, 294 insertions(+)
create mode 100644 Documentation/ABI/testing/sysfs-kernel-mm-mempolicy
create mode 100644 Documentation/ABI/testing/sysfs-kernel-mm-mempolicy-weighted-interleave
diff --git a/Documentation/ABI/testing/sysfs-kernel-mm-mempolicy b/Documentation/ABI/testing/sysfs-kernel-mm-mempolicy
new file mode 100644
index 000000000000..8dc1129d4ab1
--- /dev/null
+++ b/Documentation/ABI/testing/sysfs-kernel-mm-mempolicy
@@ -0,0 +1,33 @@
+What: /sys/kernel/mm/mempolicy/
+Date: December 2023
+Contact: Linux memory management mailing list <linux-mm@kvack.org>
+Description: Interface for Mempolicy
+
+What: /sys/kernel/mm/mempolicy/cpu_nodes
+Date: December 2023
+Contact: Linux memory management mailing list <linux-mm@kvack.org>
+Description: The numa nodes from which accesses can be generated
+
+ A cpu numa node is one which has at least 1 CPU. These nodes
+ are capable of generating accesses to memory numa nodes, and
+ will have an interleave weight table.
+
+ Example output:
+
+ ===== =================================================
+ "0,1" nodes 0 and 1 have CPUs which may generate access
+ ===== =================================================
+
+What: /sys/kernel/mm/mempolicy/possible_nodes
+Date: December 2023
+Contact: Linux memory management mailing list <linux-mm@kvack.org>
+Description: The numa nodes which are possible to come online
+
+ A possible numa node is one which has been reserved by the
+ system at boot, but may or may not be online at runtime.
+
+ Example output:
+
+ ========= ========================================
+ "0,1,2,3" nodes 0-3 are possibly online or offline
+ ========= ========================================
diff --git a/Documentation/ABI/testing/sysfs-kernel-mm-mempolicy-weighted-interleave b/Documentation/ABI/testing/sysfs-kernel-mm-mempolicy-weighted-interleave
new file mode 100644
index 000000000000..75554895ede3
--- /dev/null
+++ b/Documentation/ABI/testing/sysfs-kernel-mm-mempolicy-weighted-interleave
@@ -0,0 +1,35 @@
+What: /sys/kernel/mm/mempolicy/weighted_interleave/
+Date: December 2023
+Contact: Linux memory management mailing list <linux-mm@kvack.org>
+Description: Configuration Interface for the Weighted Interleave policy
+
+What: /sys/kernel/mm/mempolicy/weighted_interleave/nodeN/
+Date: December 2023
+Contact: Linux memory management mailing list <linux-mm@kvack.org>
+Description: Configuration interface for accesses initiated from nodeN
+
+ The directory to configure access initiator weights for nodeN.
+
+ Possible numa nodes which have not been marked as a CPU node
+ at boot will not have a nodeN directory made for them at boot.
+ Hotplug for CPU nodes is not supported.
+
+What: /sys/kernel/mm/mempolicy/weighted_interleave/nodeN/nodeM
+ /sys/kernel/mm/mempolicy/weighted_interleave/nodeN/nodeM/weight
+Date: December 2023
+Contact: Linux memory management mailing list <linux-mm@kvack.org>
+Description: Configuration interface for target nodes accessed from nodeNN
+
+ The interleave weight for a memory node (M) from initiating
+ node (N). These weights are utilized by processes which have set
+ the mempolicy to MPOL_WEIGHTED_INTERLEAVE and have opted into
+ global weights by omitting a task-local weight array.
+
+ These weights only affect new allocations, and changes at runtime
+ will not cause migrations on already allocated pages.
+
+ If the weight of 0 is desired, the appropriate way to do this is
+ by removing the node from the weighted interleave nodemask.
+
+ Minimum weight: 1
+ Maximum weight: 255
diff --git a/mm/mempolicy.c b/mm/mempolicy.c
index 10a590ee1c89..ce332b5e7a03 100644
--- a/mm/mempolicy.c
+++ b/mm/mempolicy.c
@@ -131,6 +131,11 @@ static struct mempolicy default_policy = {
static struct mempolicy preferred_node_policy[MAX_NUMNODES];
+struct interleave_weight_table {
+ unsigned char weights[MAX_NUMNODES];
+};
+static struct interleave_weight_table *iw_table;
+
/**
* numa_nearest_node - Find nearest node by state
* @node: Node id to start the search
@@ -3067,3 +3072,224 @@ void mpol_to_str(char *buffer, int maxlen, struct mempolicy *pol)
p += scnprintf(p, buffer + maxlen - p, ":%*pbl",
nodemask_pr_args(&nodes));
}
+
+struct iw_node_info {
+ struct kobject kobj;
+ int src;
+ int dst;
+};
+
+static ssize_t node_weight_show(struct kobject *kobj,
+ struct kobj_attribute *attr, char *buf)
+{
+ struct iw_node_info *node_info = container_of(kobj, struct iw_node_info,
+ kobj);
+ return sysfs_emit(buf, "%d\n",
+ iw_table[node_info->src].weights[node_info->dst]);
+}
+
+static ssize_t node_weight_store(struct kobject *kobj,
+ struct kobj_attribute *attr,
+ const char *buf, size_t count)
+{
+ unsigned char weight = 0;
+ struct iw_node_info *node_info = NULL;
+
+ node_info = container_of(kobj, struct iw_node_info, kobj);
+
+ if (kstrtou8(buf, 0, &weight) || !weight)
+ return -EINVAL;
+
+ iw_table[node_info->src].weights[node_info->dst] = weight;
+
+ return count;
+}
+
+static struct kobj_attribute node_weight =
+ __ATTR(weight, 0664, node_weight_show, node_weight_store);
+
+static struct attribute *dst_node_attrs[] = {
+ &node_weight.attr,
+ NULL,
+};
+
+static struct attribute_group dst_node_attr_group = {
+ .attrs = dst_node_attrs,
+};
+
+static const struct attribute_group *dst_node_attr_groups[] = {
+ &dst_node_attr_group,
+ NULL,
+};
+
+static const struct kobj_type dst_node_kobj_ktype = {
+ .sysfs_ops = &kobj_sysfs_ops,
+ .default_groups = dst_node_attr_groups,
+};
+
+static int add_dst_node(int src, int dst, struct kobject *src_kobj)
+{
+ struct iw_node_info *node_info = NULL;
+ int ret;
+
+ node_info = kzalloc(sizeof(struct iw_node_info), GFP_KERNEL);
+ if (!node_info)
+ return -ENOMEM;
+ node_info->src = src;
+ node_info->dst = dst;
+
+ kobject_init(&node_info->kobj, &dst_node_kobj_ktype);
+ ret = kobject_add(&node_info->kobj, src_kobj, "node%d", dst);
+ if (ret) {
+ pr_err("kobject_add error [%d-node%d]: %d", src, dst, ret);
+ kobject_put(&node_info->kobj);
+ }
+ return ret;
+}
+
+static int add_src_node(int src, struct kobject *root_kobj)
+{
+ int err, dst;
+ struct kobject *src_kobj;
+ char name[24];
+
+ snprintf(name, 24, "node%d", src);
+ src_kobj = kobject_create_and_add(name, root_kobj);
+ if (!src_kobj) {
+ pr_err("failed to create source node kobject\n");
+ return -ENOMEM;
+ }
+ for_each_node_state(dst, N_POSSIBLE) {
+ err = add_dst_node(src, dst, src_kobj);
+ if (err)
+ break;
+ }
+ if (err)
+ kobject_put(src_kobj);
+ return err;
+}
+
+static int add_weighted_interleave_group(struct kobject *root_kobj)
+{
+ struct kobject *wi_kobj;
+ int nid, err;
+
+ wi_kobj = kobject_create_and_add("weighted_interleave", root_kobj);
+ if (!wi_kobj) {
+ pr_err("failed to create node kobject\n");
+ return -ENOMEM;
+ }
+
+ for_each_node_state(nid, N_CPU) {
+ err = add_src_node(nid, wi_kobj);
+ if (err) {
+ pr_err("failed to add sysfs [node%d]\n", nid);
+ break;
+ }
+ }
+ if (err)
+ kobject_put(wi_kobj);
+ return 0;
+
+}
+
+static ssize_t cpu_nodes_show(struct kobject *kobj,
+ struct kobj_attribute *attr, char *buf)
+{
+ int nid, next_nid;
+ int len = 0;
+
+ for_each_node_state(nid, N_CPU) {
+ len += sysfs_emit_at(buf, len, "%d", nid);
+ next_nid = next_node(nid, node_states[N_CPU]);
+ if (next_nid < MAX_NUMNODES)
+ len += sysfs_emit_at(buf, len, ",");
+ }
+ len += sysfs_emit_at(buf, len, "\n");
+
+ return len;
+}
+
+static ssize_t possible_nodes_show(struct kobject *kobj,
+ struct kobj_attribute *attr, char *buf)
+{
+ int nid, next_nid;
+ int len = 0;
+
+ for_each_node_state(nid, N_POSSIBLE) {
+ len += sysfs_emit_at(buf, len, "%d", nid);
+ next_nid = next_node(nid, node_states[N_POSSIBLE]);
+ if (next_nid < MAX_NUMNODES)
+ len += sysfs_emit_at(buf, len, ",");
+ }
+ len += sysfs_emit_at(buf, len, "\n");
+
+ return len;
+}
+
+static struct kobj_attribute cpu_nodes_attr = __ATTR_RO(cpu_nodes);
+static struct kobj_attribute possible_nodes_attr = __ATTR_RO(possible_nodes);
+
+static struct attribute *mempolicy_attrs[] = {
+ &cpu_nodes_attr.attr,
+ &possible_nodes_attr.attr,
+ NULL,
+};
+
+static const struct attribute_group mempolicy_attr_group = {
+ .attrs = mempolicy_attrs,
+ NULL,
+};
+
+static void mempolicy_kobj_release(struct kobject *kobj)
+{
+ kfree(kobj);
+ kfree(iw_table);
+}
+
+static const struct kobj_type mempolicy_kobj_ktype = {
+ .release = mempolicy_kobj_release,
+ .sysfs_ops = &kobj_sysfs_ops,
+};
+
+static int __init mempolicy_sysfs_init(void)
+{
+ int err, nid;
+ int cpunodes = 0;
+ struct kobject *root_kobj;
+
+ for_each_node_state(nid, N_CPU)
+ cpunodes += 1;
+ iw_table = kmalloc_array(cpunodes, sizeof(*iw_table), GFP_KERNEL);
+ if (!iw_table) {
+ pr_err("failed to create interleave weight table\n");
+ err = -ENOMEM;
+ goto fail_obj;
+ }
+ memset(iw_table, 1, cpunodes * sizeof(*iw_table));
+
+ root_kobj = kzalloc(sizeof(struct kobject), GFP_KERNEL);
+ if (!root_kobj)
+ return -ENOMEM;
+
+ kobject_init(root_kobj, &mempolicy_kobj_ktype);
+ err = kobject_add(root_kobj, mm_kobj, "mempolicy");
+ if (err) {
+ pr_err("failed to add kobject to the system\n");
+ goto fail_obj;
+ }
+
+ err = sysfs_create_group(root_kobj, &mempolicy_attr_group);
+ if (err) {
+ pr_err("failed to register mempolicy group\n");
+ goto fail_obj;
+ }
+
+ err = add_weighted_interleave_group(root_kobj);
+fail_obj:
+ if (err)
+ kobject_put(root_kobj);
+ return err;
+
+}
+late_initcall(mempolicy_sysfs_init);
--
2.39.1
^ permalink raw reply related
* [RFC PATCH 00/11] mempolicy2, mbind2, and weighted interleave
From: Gregory Price @ 2023-12-07 0:27 UTC (permalink / raw)
To: linux-mm, jgroves, ravis.opensrc, sthanneeru, emirakhur,
Hasan.Maruf
Cc: linux-doc, linux-fsdevel, linux-api, linux-arch, linux-kernel,
akpm, arnd, tglx, luto, mingo, bp, dave.hansen, x86, hpa, mhocko,
tj, ying.huang, gregory.price, corbet, rakie.kim, hyeongtak.ji,
honggyu.kim, vtavarespetr, peterz, Johannes Weiner,
Hasan Al Maruf, Hao Wang, Dan Williams, Michal Hocko, Zhongkun He,
Frank van der Linden, John Groves, Jonathan Cameron
This patch set extends the mempolicy interface to enable new
mempolicies which may require extended data to operate.
One such policy is included with this set as an example:
MPOL_WEIGHTED_INTERLEAVE
There are 3 major "phases" in the patch set:
1) Implement a "global weight" mechanism via sysfs, which allows
set_mempolicy to implement MPOL_WEIGHTED_INTERLEAVE utilizing
weights set by the administrator (or system daemon).
2) A refactor of the mempolicy creation mechanism to accept an
extensible argument structure `struct mempolicy_args` to promote
code re-use between the original mempolicy/mbind interfaces and
the new extended mempolicy2/mbind2 interfaces.
3) Implementation of set_mempolicy2, get_mempolicy2, and mbind2,
along with the addition of task-local weights so that per-task
weights can be registered for MPOL_WEIGHTED_INTERLEAVE.
=====================================================================
(Patch 1) : sysfs addition - /sys/kernel/mm/mempolicy/
This feature provides a way to set interleave weight information under
sysfs at /sys/kernel/mm/mempolicy/weighted_interleave/nodeN/nodeM/weight
The sysfs structure is designed as follows.
$ tree /sys/kernel/mm/mempolicy/
/sys/kernel/mm/mempolicy/
├── cpu_nodes
├── possible_nodes
└── weighted_interleave
├── nodeN
│ ├── nodeM
│ │ └── weight
│ └── nodeM+X
│ └── weight
└── nodeN+X
├── nodeM
│ └── weight
└── node+X
└── weight
'cpu_nodes' and 'possible_nodes' is added to 'mm/mempolicy' to help
describe the expected structures under mempolicy directorys. For
example 'cpu_nodes' will describe what 'nodeN' directories will
exist in 'weighted_interleave', while 'possible_nodes' describes
what nodeM directories wille exist under the 'nodeN' directories.
Internally, weights are represented as a matrix of [src,dst] nodes.
struct interleave_weight_table {
unsigned char weights[MAX_NUMNODES];
};
static struct interleave_weight_table *iw_table;
"Source Nodes" are nodes which have 1 or more CPUs, while "Destination
Nodes" include any possible node. A "Possible" node is one which has
been reserved by the system, but which may or may not be online.
We present possible nodes, instead of online nodes, to simplify the
management interface, considering that a) the table of MAX_NUMNODES
size is allocated anyway to simplfy fetching of weights, and b) it
simplifies the management of hotplug events, allowing for weights to
be set prior to a node coming online which may be beneficial for
immediate use of the memory.
the 'weight' of a node (an unsigned char of value 1-255) is the number
of pages that are allocated during a "weighted interleave" round.
(See 'weighted interleave' for more details').
The [src,dst] matrix is implemented to allow for the capturing the
complexity of bandwidth distribution across a multi-socket, or
heterogeneous memory environment. For example, consider a 2-socket
Intel server with 1 CXL Memory expander attached to each socket.
From the perspective of a task on a CPU in Socket 0, the bandwidth
distribution is as follows:
Socket 0 DRAM: (# DDR Channels) * (DDR Bandwidth) ~400GB/s
Socket 0 CXL : (# CXL Lanes) * (CXL Lane Bandwidth) 128GB/s
Socket 1 DRAM + CXL: (# UPI Lanes) * (UPI Bandwidth) ~64GB/s
If the task is then migrated to Socket 1, the bandwidth distribution
flips to the following.
Socket 1 DRAM: (# DDR Channels) * (DDR Bandwidth) ~400GB/s
Socket 1 CXL : (# CXL Lanes) * (CXL Lane Bandwidth) 128GB/s
Socket 0 DRAM + CXL: (# UPI Lanes) * (UPI Bandwidth) ~64GB/s
The matrix allows for a 'source node' perspective weighting strategy,
which allows for migrated tasks to simply "re-weight" new allocations
immediately, by simply changing the [src] index they access in the
global interleave weight table.
=====================================================================
(Patch 2) set_mempolicy: MPOL_WEIGHTED_INTERLEAVE
Weighted interleave is a new memory policy that interleaves memory
across numa nodes in the provided nodemask based on the weights
described in patch 1 (sysfs global weights).
When a system has multiple NUMA nodes and it becomes bandwidth hungry,
the current MPOL_INTERLEAVE could be an wise option.
However, if those NUMA nodes consist of different types of memory such
as having local DRAM and CXL memory together, the current round-robin
based interleaving policy doesn't maximize the overall bandwidth
because of their different bandwidth characteristics.
Instead, the interleaving can be more efficient when the allocation
policy follows each NUMA nodes' bandwidth weight rather than having 1:1
round-robin allocation.
This patch introduces a new memory policy, MPOL_WEIGHTED_INTERLEAVE,
which enables weighted interleaving between NUMA nodes. Weighted
interleave allows for a proportional distribution of memory across
multiple numa nodes, preferablly apportioned to match the bandwidth
capacity of each node from the perspective of the accessing node.
For example, if a system has 1 CPU node (0), and 2 memory nodes (0,1),
with a relative bandwidth of (100GB/s, 50GB/s) respectively, the
appropriate weight distribution is (2:1).
Weights will be acquired from the global weight matrix exposed by the
sysfs extension: /sys/kernel/mm/mempolicy/weighted_interleave/
The policy will then allocate the number of pages according to the
set weights. For example, if the weights are (2,1), then 2 pages
will be allocated on node0 for every 1 page allocated on node1.
The new flag MPOL_WEIGHTED_INTERLEAVE can be used in set_mempolicy(2)
and mbind(2).
=====================================================================
(Patches 3-6) Refactoring mempolicy for code-reuse
To avoid multiple paths of mempolicy creation, we should refactor the
existing code to enable the designed extensibility, and refactor
existing users to utilize the new interface (while retaining the
existing userland interface).
This set of patches introduces a new mempolicy_args structure, which
is used to more fully describe a requested mempolicy - to include
existing and future extensions.
/*
* Describes settings of a mempolicy during set/get syscalls and
* kernel internal calls to do_set_mempolicy()
*/
struct mempolicy_args {
unsigned short mode; /* policy mode */
unsigned short mode_flags; /* policy mode flags */
nodemask_t *policy_nodes; /* get/set/mbind */
int policy_node; /* get: policy node information */
unsigned long addr; /* get: vma address */
int addr_node; /* get: node the address belongs to */
int home_node; /* mbind: use MPOL_MF_HOME_NODE */
unsigned char *il_weights; /* for mode MPOL_WEIGHTED_INTERLEAVE */
};
This arg structure will eventually be utilized by the following
interfaces:
mpol_new() - new mempolicy creation
do_get_mempolicy() - acquiring information about mempolicy
do_set_mempolicy() - setting the task mempolicy
do_mbind() - setting a vma mempolicy
do_get_mempolicy() is completely refactored to break it out into
separate functionality based on the flags provided by get_mempolicy(2)
MPOL_F_MEMS_ALLOWED: acquires task->mems_allowed
MPOL_F_ADDR: acquires information on vma policies
MPOL_F_NODE: changes the output for the policy arg to node info
We refactor the get_mempolicy syscall flatten the logic based on these
flags, and aloow for set_mempolicy2() to re-use the underlying logic.
The result of this refactor, and the new mempolicy_args structure, is
that extensions like 'sys_set_mempolicy_home_node' can now be directly
integrated into the initial call to 'set_mempolicy2', and that more
complete information about a mempolicy can be returned with a single
call to 'get_mempolicy2', rather than multiple calls to 'get_mempolicy'
=====================================================================
(Patches 7-10) set_mempolicy2, get_mempolicy2, mbind2
These interfaces are the 'extended' counterpart to their relatives.
They use the userland 'struct mpol_args' structure to communicate a
complete mempolicy configuration to the kernel. This structure
looks very much like the kernel-internal 'struct mempolicy_args':
struct mpol_args {
/* Basic mempolicy settings */
unsigned short mode;
unsigned short mode_flags;
unsigned long *pol_nodes;
unsigned long pol_maxnodes;
/* get_mempolicy: policy node information */
int policy_node;
/* get_mempolicy: memory range policy */
unsigned long addr;
int addr_node;
/* mbind2: policy home node */
int home_node;
/* mbind2: address ranges to apply the policy */
struct iovec *vec;
size_t vlen;
/* weighted interleave settings */
unsigned char *il_weights; /* of size pol_maxnodes */
};
The basic mempolicy settings which are shared across all interfaces
are captured at the top of the structure, while extensions such as
'policy_node' and 'addr' are collected beneath.
The syscalls are uniform and defined as follows:
long sys_mbind2(struct mpol_args *args,
size_t size,
unsigned long flags);
long sys_get_mempolicy2(struct mpol_args *args,
size_t size,
unsigned long flags);
long sys_set_mempolicy2(struct mpol_args *args,
size_t size,
unsigned long flags);
The 'flags' argument for mbind2 is the same as 'mbind', except with
the addition of MPOL_MF_HOME_NODE to denote whether the 'home_node'
field should be utilized.
The 'flags' argument for get_mempolicy2 is the same as get_mempolicy.
The 'flags' argument is not used by 'set_mempolicy' at this time, but
may end up allowing the use of MPOL_MF_HOME_NODE if such functionality
is desired.
The extensions can be summed up as follows:
get_mempolicy2 extensions:
'mode', 'policy_node', and 'addr_node' can now be fetched with
a single call, rather than multiple with a combination of flags.
- 'mode' will always return the policy mode
- 'policy_node' will replace the functionality of MPOL_F_NODE
- 'addr_node' will return the node for 'addr' w/ MPOL_F_ADDR
set_mempolicy2:
- task-local interleave weights can be set via 'il_weights'
(see next patch)
mbind2:
- 'home_node' field sets policy home node w/ MPOL_MF_HOME_NODE
- task-local interleave weights can be set via 'il_weights'
(see next patch)
- 'vec' and 'vlen' can be used to operate on multiple memory
ranges, rather than a single memory range per syscall.
=====================================================================
(Patch 11) set_mempolicy2/mbind2: MPOL_WEIGHTED_INTERLEAVE
This patch shows the explicit extension pattern when adding new
policies to mempolicy2/mbind2. This adds the 'il_weights' field
to mpol_args and adds the logic to fill in task-local weights.
There are now two ways to weight a mempolicy: global and local.
To denote which mode the task is in, we add the internal flag:
MPOL_F_GWEIGHT /* Utilize global weights */
When MPOL_F_GWEIGHT is set, the global weights are used, and
when it is not set, task-local weights are used.
Example logic:
if (pol->flags & MPOL_F_GWEIGHT)
pol_weights = iw_table[numa_node_id()].weights;
else
pol_weights = pol->wil.weights;
set_mempolicy is changed to always set MPOL_F_GWEIGHT, since this
syscall is incapable of passing weights via its interfaces, while
set_mempolicy2 sets MPOL_F_GWEIGHT if MPOL_F_WEIGHTED_INTERLEAVE
is required but (*il_weights) in mpol_args is null.
The operation of task-local weighted is otherwise exactly the
same - except for what occurs on task migration.
On task migration, the system presently has no way of determining
what the new weights "should be", or what the user "intended".
For this reason, we default all weights to '1' and do not allow
weights to be '0'. This means, should a migration occur where
one or more nodes appear into the nodemask - the effective weight
for that node will be '1'. This avoids a potential allocation
failure condition if a migration occurs and introduces a node
which otherwise did not have a weight.
For this reason, users should use task-local weighting when
migrations are not expected, and global weighting when migrations
are expected or possible.
Suggested-by: Gregory Price <gregory.price@memverge.com>
Suggested-by: Johannes Weiner <hannes@cmpxchg.org>
Suggested-by: Hasan Al Maruf <hasanalmaruf@fb.com>
Suggested-by: Hao Wang <haowang3@fb.com>
Suggested-by: Ying Huang <ying.huang@intel.com>
Suggested-by: Dan Williams <dan.j.williams@intel.com>
Suggested-by: Michal Hocko <mhocko@suse.com>
Suggested-by: tj <tj@kernel.org>
Suggested-by: Zhongkun He <hezhongkun.hzk@bytedance.com>
Suggested-by: Frank van der Linden <fvdl@google.com>
Suggested-by: John Groves <john@jagalactic.com>
Suggested-by: Vinicius Tavares Petrucci <vtavarespetr@micron.com>
Suggested-by: Srinivasulu Thanneeru <sthanneeru@micron.com>
Suggested-by: Ravi Jonnalagadda <ravis.opensrc@micron.com>
Suggested-by: Jonathan Cameron <Jonathan.Cameron@Huawei.com>
Signed-off-by: Gregory Price <gregory.price@memverge.com>
Gregory Price (9):
mm/mempolicy: refactor sanitize_mpol_flags for reuse
mm/mempolicy: create struct mempolicy_args for creating new
mempolicies
mm/mempolicy: refactor kernel_get_mempolicy for code re-use
mm/mempolicy: allow home_node to be set by mpol_new
mm/mempolicy: add userland mempolicy arg structure
mm/mempolicy: add set_mempolicy2 syscall
mm/mempolicy: add get_mempolicy2 syscall
mm/mempolicy: add the mbind2 syscall
mm/mempolicy: extend set_mempolicy2 and mbind2 to support weighted
interleave
Rakie Kim (2):
mm/mempolicy: implement the sysfs-based weighted_interleave interface
mm/mempolicy: introduce MPOL_WEIGHTED_INTERLEAVE for weighted
interleaving
.../ABI/testing/sysfs-kernel-mm-mempolicy | 33 +
...fs-kernel-mm-mempolicy-weighted-interleave | 35 +
.../admin-guide/mm/numa_memory_policy.rst | 85 ++
arch/alpha/kernel/syscalls/syscall.tbl | 3 +
arch/arm/tools/syscall.tbl | 3 +
arch/m68k/kernel/syscalls/syscall.tbl | 3 +
arch/microblaze/kernel/syscalls/syscall.tbl | 3 +
arch/mips/kernel/syscalls/syscall_n32.tbl | 3 +
arch/mips/kernel/syscalls/syscall_o32.tbl | 3 +
arch/parisc/kernel/syscalls/syscall.tbl | 3 +
arch/powerpc/kernel/syscalls/syscall.tbl | 3 +
arch/s390/kernel/syscalls/syscall.tbl | 3 +
arch/sh/kernel/syscalls/syscall.tbl | 3 +
arch/sparc/kernel/syscalls/syscall.tbl | 3 +
arch/x86/entry/syscalls/syscall_32.tbl | 3 +
arch/x86/entry/syscalls/syscall_64.tbl | 3 +
arch/xtensa/kernel/syscalls/syscall.tbl | 3 +
include/linux/mempolicy.h | 21 +
include/linux/syscalls.h | 6 +
include/uapi/asm-generic/unistd.h | 8 +-
include/uapi/linux/mempolicy.h | 27 +-
mm/mempolicy.c | 960 ++++++++++++++++--
22 files changed, 1103 insertions(+), 114 deletions(-)
create mode 100644 Documentation/ABI/testing/sysfs-kernel-mm-mempolicy
create mode 100644 Documentation/ABI/testing/sysfs-kernel-mm-mempolicy-weighted-interleave
--
2.39.1
^ permalink raw reply
* Re: [PATCH v2] ELF: supply userspace with available page shifts (AT_PAGE_SHIFT_MASK)
From: Kees Cook @ 2023-12-06 21:09 UTC (permalink / raw)
To: Florian Weimer
Cc: Alexey Dobriyan, Andrew Morton, linux-kernel, linux-arch,
linux-api, x86
In-Reply-To: <87edfzavof.fsf@oldenburg.str.redhat.com>
On Wed, Dec 06, 2023 at 10:05:36PM +0100, Florian Weimer wrote:
> * Kees Cook:
>
> > On Tue, Dec 05, 2023 at 07:01:34PM +0300, Alexey Dobriyan wrote:
> >> Report available page shifts in arch independent manner, so that
> >> userspace developers won't have to parse /proc/cpuinfo hunting
> >> for arch specific strings:
> >>
> >> Note!
> >>
> >> This is strictly for userspace, if some page size is shutdown due
> >> to kernel command line option or CPU bug workaround, than is must not
> >> be reported in aux vector!
> >
> > Given Florian in CC, I assume this is something glibc would like to be
> > using? Please mention this in the commit log.
>
> Nope, I just wrote a random drive-by comment on the first version.
Ah, okay. Then Alexey, who do you expect to be the consumer of this new
AT value?
--
Kees Cook
^ permalink raw reply
* Re: [PATCH v2] ELF: supply userspace with available page shifts (AT_PAGE_SHIFT_MASK)
From: Florian Weimer @ 2023-12-06 21:05 UTC (permalink / raw)
To: Kees Cook
Cc: Alexey Dobriyan, Andrew Morton, linux-kernel, linux-arch,
linux-api, x86
In-Reply-To: <202312061236.DE847C52AA@keescook>
* Kees Cook:
> On Tue, Dec 05, 2023 at 07:01:34PM +0300, Alexey Dobriyan wrote:
>> Report available page shifts in arch independent manner, so that
>> userspace developers won't have to parse /proc/cpuinfo hunting
>> for arch specific strings:
>>
>> Note!
>>
>> This is strictly for userspace, if some page size is shutdown due
>> to kernel command line option or CPU bug workaround, than is must not
>> be reported in aux vector!
>
> Given Florian in CC, I assume this is something glibc would like to be
> using? Please mention this in the commit log.
Nope, I just wrote a random drive-by comment on the first version.
>> x86_64 machine with 1 GiB pages:
>>
>> 00000030 06 00 00 00 00 00 00 00 00 10 00 00 00 00 00 00
>> 00000040 1d 00 00 00 00 00 00 00 00 10 20 40 00 00 00 00
>>
>> x86_64 machine with 2 MiB pages only:
>>
>> 00000030 06 00 00 00 00 00 00 00 00 10 00 00 00 00 00 00
>> 00000040 1d 00 00 00 00 00 00 00 00 10 20 00 00 00 00 00
>>
>> AT_PAGESZ is always 4096 which is not that interesting.
>
> That's not always true. For example, see arm64:
> arch/arm64/include/asm/elf.h:#define ELF_EXEC_PAGESIZE PAGE_SIZE
I'm pretty sure the comment refers to the x86-64 situation. 8-)
> I'm not actually sure why x86 forces it to 4096. I'd need to go look
> through the history there.
On x86-64, page size 4096 is architectural. Likewise on s390x and a few
other architectures.
Thanks,
Florian
^ permalink raw reply
* Re: [PATCH v2] ELF: supply userspace with available page shifts (AT_PAGE_SHIFT_MASK)
From: Kees Cook @ 2023-12-06 20:47 UTC (permalink / raw)
To: Alexey Dobriyan
Cc: Andrew Morton, Florian Weimer, linux-kernel, linux-arch,
linux-api, x86
In-Reply-To: <1d679805-8a82-44a4-ba14-49d4f28ff597@p183>
On Tue, Dec 05, 2023 at 07:01:34PM +0300, Alexey Dobriyan wrote:
> Report available page shifts in arch independent manner, so that
> userspace developers won't have to parse /proc/cpuinfo hunting
> for arch specific strings:
>
> Note!
>
> This is strictly for userspace, if some page size is shutdown due
> to kernel command line option or CPU bug workaround, than is must not
> be reported in aux vector!
Given Florian in CC, I assume this is something glibc would like to be
using? Please mention this in the commit log.
>
> x86_64 machine with 1 GiB pages:
>
> 00000030 06 00 00 00 00 00 00 00 00 10 00 00 00 00 00 00
> 00000040 1d 00 00 00 00 00 00 00 00 10 20 40 00 00 00 00
>
> x86_64 machine with 2 MiB pages only:
>
> 00000030 06 00 00 00 00 00 00 00 00 10 00 00 00 00 00 00
> 00000040 1d 00 00 00 00 00 00 00 00 10 20 00 00 00 00 00
>
> AT_PAGESZ is always 4096 which is not that interesting.
That's not always true. For example, see arm64:
arch/arm64/include/asm/elf.h:#define ELF_EXEC_PAGESIZE PAGE_SIZE
I'm not actually sure why x86 forces it to 4096. I'd need to go look
through the history there.
>
> Signed-off-by: Alexey Dobriyan <adobriyan@gmail.com>
> ---
>
> v2: switch to 1 bit per page shift (bitmask)
>
> arch/x86/include/asm/elf.h | 12 ++++++++++++
> fs/binfmt_elf.c | 3 +++
> include/uapi/linux/auxvec.h | 14 ++++++++++++++
> 3 files changed, 29 insertions(+)
>
> --- a/arch/x86/include/asm/elf.h
> +++ b/arch/x86/include/asm/elf.h
> @@ -358,6 +358,18 @@ else if (IS_ENABLED(CONFIG_IA32_EMULATION)) \
>
> #define COMPAT_ELF_ET_DYN_BASE (TASK_UNMAPPED_BASE + 0x1000000)
>
> +#define ARCH_AT_PAGE_SHIFT_MASK \
> + do { \
> + u32 val = 1 << 12; \
> + if (boot_cpu_has(X86_FEATURE_PSE)) { \
> + val |= 1 << 21; \
> + } \
> + if (boot_cpu_has(X86_FEATURE_GBPAGES)) { \
> + val |= 1 << 30; \
> + } \
> + NEW_AUX_ENT(AT_PAGE_SHIFT_MASK, val); \
> + } while (0)
> +
> #endif /* !CONFIG_X86_32 */
Can't we have a generic ARCH_AT_PAGE_SHIFT_MASK too? Something like:
#ifndef ARCH_AT_PAGE_SHIFT_MASK
#define ARCH_AT_PAGE_SHIFT_MASK
NEW_AUX_ENT(AT_PAGE_SHIFT_MASK, 1 << PAGE_SHIFT)
#endif
Or am I misunderstanding something here?
>
> #define VDSO_CURRENT_BASE ((unsigned long)current->mm->context.vdso)
> --- a/fs/binfmt_elf.c
> +++ b/fs/binfmt_elf.c
> @@ -240,6 +240,9 @@ create_elf_tables(struct linux_binprm *bprm, const struct elfhdr *exec,
> #endif
> NEW_AUX_ENT(AT_HWCAP, ELF_HWCAP);
> NEW_AUX_ENT(AT_PAGESZ, ELF_EXEC_PAGESIZE);
> +#ifdef ARCH_AT_PAGE_SHIFT_MASK
> + ARCH_AT_PAGE_SHIFT_MASK;
> +#endif
That way we can avoid an #ifdef in the .c file.
> NEW_AUX_ENT(AT_CLKTCK, CLOCKS_PER_SEC);
> NEW_AUX_ENT(AT_PHDR, phdr_addr);
> NEW_AUX_ENT(AT_PHENT, sizeof(struct elf_phdr));
> --- a/include/uapi/linux/auxvec.h
> +++ b/include/uapi/linux/auxvec.h
> @@ -33,6 +33,20 @@
> #define AT_RSEQ_FEATURE_SIZE 27 /* rseq supported feature size */
> #define AT_RSEQ_ALIGN 28 /* rseq allocation alignment */
>
> +/*
> + * Page sizes available for mmap(2) encoded as bitmask.
> + *
> + * Example: x86_64 system with pse, pdpe1gb /proc/cpuinfo flags reports
> + * 4 KiB, 2 MiB and 1 GiB page support.
> + *
> + * $ hexdump -C /proc/self/auxv
FWIW, a more readable form is: $ LD_SHOW_AUXV=1 /bin/true
> + * 00000030 06 00 00 00 00 00 00 00 00 10 00 00 00 00 00 00
> + * 00000040 1d 00 00 00 00 00 00 00 00 10 20 40 00 00 00 00
> + *
> + * For 2^64 hugepage support please contact your Universe sales representative.
> + */
> +#define AT_PAGE_SHIFT_MASK 29
... hmm, why is 29 unused?
> +
> #define AT_EXECFN 31 /* filename of program */
>
> #ifndef AT_MINSIGSTKSZ
This will need a man page update for "getauxval" as well...
--
Kees Cook
^ permalink raw reply
* Re: [PATCH 3/4] listmount: small changes in semantics
From: Miklos Szeredi @ 2023-12-06 20:24 UTC (permalink / raw)
To: Serge E. Hallyn
Cc: Miklos Szeredi, Christian Brauner, linux-api, linux-man,
linux-security-module, Karel Zak, linux-fsdevel, Ian Kent,
David Howells, Al Viro
In-Reply-To: <20231206195807.GA209606@mail.hallyn.com>
On Wed, 6 Dec 2023 at 20:58, Serge E. Hallyn <serge@hallyn.com> wrote:
>
> On Tue, Nov 28, 2023 at 05:03:34PM +0100, Miklos Szeredi wrote:
> > - if (!is_path_reachable(m, mnt->mnt_root, &rootmnt))
> > - return capable(CAP_SYS_ADMIN) ? 0 : -EPERM;
> > + if (!capable(CAP_SYS_ADMIN) &&
>
> Was there a reason to do the capable check first? In general,
> checking capable() when not needed is frowned upon, as it will
> set the PF_SUPERPRIV flag.
>
I synchronized the permission checking with statmount() without
thinking about the order. I guess we can change the order back in
both syscalls?
I also don't understand the reason behind the using the _noaudit()
variant. Christian?
Thanks,
Miklos
^ permalink raw reply
* Re: [PATCH 3/4] listmount: small changes in semantics
From: Serge E. Hallyn @ 2023-12-06 19:58 UTC (permalink / raw)
To: Miklos Szeredi
Cc: Christian Brauner, linux-api, linux-man, linux-security-module,
Karel Zak, linux-fsdevel, Ian Kent, David Howells, Al Viro
In-Reply-To: <20231128160337.29094-4-mszeredi@redhat.com>
On Tue, Nov 28, 2023 at 05:03:34PM +0100, Miklos Szeredi wrote:
> 1) Make permission checking consistent with statmount(2): fail if mount is
> unreachable from current root. Previously it failed if mount was
> unreachable from root->mnt->mnt_root.
>
> 2) List all submounts, even if unreachable from current root. This is
> safe, since 1) will prevent listing unreachable mounts for unprivileged
> users.
>
> 3) LSMT_ROOT is unchaged, it lists mounts under current root.
>
> Signed-off-by: Miklos Szeredi <mszeredi@redhat.com>
> ---
> fs/namespace.c | 39 ++++++++++++++-------------------------
> 1 file changed, 14 insertions(+), 25 deletions(-)
>
> diff --git a/fs/namespace.c b/fs/namespace.c
> index ad62cf7ee334..10cd651175b5 100644
> --- a/fs/namespace.c
> +++ b/fs/namespace.c
> @@ -5004,37 +5004,26 @@ static struct mount *listmnt_next(struct mount *curr)
> return node_to_mount(rb_next(&curr->mnt_node));
> }
>
> -static ssize_t do_listmount(struct mount *first, struct vfsmount *mnt,
> +static ssize_t do_listmount(struct mount *first, struct path *orig, u64 mnt_id,
> u64 __user *buf, size_t bufsize,
> const struct path *root)
> {
> - struct mount *r, *m = real_mount(mnt);
> - struct path rootmnt = {
> - .mnt = root->mnt,
> - .dentry = root->mnt->mnt_root
> - };
> - struct path orig;
> + struct mount *r;
> ssize_t ctr;
> int err;
>
> - if (!is_path_reachable(m, mnt->mnt_root, &rootmnt))
> - return capable(CAP_SYS_ADMIN) ? 0 : -EPERM;
> + if (!capable(CAP_SYS_ADMIN) &&
Was there a reason to do the capable check first? In general,
checking capable() when not needed is frowned upon, as it will
set the PF_SUPERPRIV flag.
> + !is_path_reachable(real_mount(orig->mnt), orig->dentry, root))
> + return -EPERM;
>
> - err = security_sb_statfs(mnt->mnt_root);
> + err = security_sb_statfs(orig->dentry);
> if (err)
> return err;
>
> - if (root->mnt == mnt) {
> - orig = *root;
> - } else {
> - orig.mnt = mnt;
> - orig.dentry = mnt->mnt_root;
> - }
> -
> for (ctr = 0, r = first; r; r = listmnt_next(r)) {
> - if (r == m)
> + if (r->mnt_id_unique == mnt_id)
> continue;
> - if (!is_path_reachable(r, r->mnt.mnt_root, &orig))
> + if (!is_path_reachable(r, r->mnt.mnt_root, orig))
> continue;
>
> if (ctr >= bufsize)
> @@ -5053,9 +5042,8 @@ SYSCALL_DEFINE4(listmount, const struct mnt_id_req __user *, req,
> {
> struct mnt_namespace *ns = current->nsproxy->mnt_ns;
> struct mnt_id_req kreq;
> - struct vfsmount *mnt;
> struct mount *first;
> - struct path root;
> + struct path root, orig;
> u64 mnt_id;
> ssize_t ret;
>
> @@ -5071,16 +5059,17 @@ SYSCALL_DEFINE4(listmount, const struct mnt_id_req __user *, req,
> down_read(&namespace_sem);
> get_fs_root(current->fs, &root);
> if (mnt_id == LSMT_ROOT) {
> - mnt = root.mnt;
> + orig = root;
> } else {
> ret = -ENOENT;
> - mnt = lookup_mnt_in_ns(mnt_id, ns);
> - if (!mnt)
> + orig.mnt = lookup_mnt_in_ns(mnt_id, ns);
> + if (!orig.mnt)
> goto err;
> + orig.dentry = orig.mnt->mnt_root;
> }
> first = node_to_mount(rb_first(&ns->mounts));
>
> - ret = do_listmount(first, mnt, buf, bufsize, &root);
> + ret = do_listmount(first, &orig, mnt_id, buf, bufsize, &root);
> err:
> path_put(&root);
> up_read(&namespace_sem);
> --
> 2.41.0
>
^ permalink raw reply
* Re: [PATCH RFT v4 5/5] kselftest/clone3: Test shadow stack support
From: Mark Brown @ 2023-12-06 18:42 UTC (permalink / raw)
To: Edgecombe, Rick P
Cc: dietmar.eggemann@arm.com, keescook@chromium.org,
Szabolcs.Nagy@arm.com, shuah@kernel.org,
dave.hansen@linux.intel.com, debug@rivosinc.com, mgorman@suse.de,
brauner@kernel.org, fweimer@redhat.com,
linux-kernel@vger.kernel.org, mingo@redhat.com,
hjl.tools@gmail.com, rostedt@goodmis.org,
vincent.guittot@linaro.org, tglx@linutronix.de,
vschneid@redhat.com, catalin.marinas@arm.com, bristot@redhat.com,
will@kernel.org, hpa@zytor.com, peterz@infradead.org,
jannh@google.com, bp@alien8.de, bsegall@google.com,
linux-kselftest@vger.kernel.org, linux-api@vger.kernel.org,
x86@kernel.org, juri.lelli@redhat.com
In-Reply-To: <127bba3063b19dd87ae3014f6d3bba342f7a16fb.camel@intel.com>
[-- Attachment #1: Type: text/plain, Size: 1339 bytes --]
On Tue, Dec 05, 2023 at 10:31:09PM +0000, Edgecombe, Rick P wrote:
> On Tue, 2023-12-05 at 16:43 +0000, Mark Brown wrote:
> > If the x86 toolchain/libc support is widely enough deployed (or you
> > just
> > don't mind any missing coverage) we could use the toolchain support
> > there and only have the manual enable for arm64, it'd be inconsistent
> > but not wildly so.
> I'm hoping there is not too much of a gap before the glibc support
> starts filtering out. Long term, elf bit enabling is probably the right
> thing for the generic tests. Short term, manual enabling is ok with me
> if no one else minds. Maybe we could add my "don't do" list as a
> comment if we do manual enabling?
Probably good to write it up somewhere, yes - it'd also be useful for
anyone off doing their own non-libc things. It did cross my mind to
try to make a document for the generic bit of the ABI for shadow stacks.
> I'll have to check your new series, but I also wonder if we could cram
> the manual enabling and status checking pieces into some headers and
> not have to have "if x86" "if arm" logic in the test themselves.
I did think about that but was worried that a header might encourage
more users doing the hacky thing. OTOH it would mean the arch specific
tests could share the header though so perhaps you're right, I'll take a
look.
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]
^ permalink raw reply
* Re: [PATCH RFT v4 2/5] fork: Add shadow stack support to clone3()
From: Mark Brown @ 2023-12-06 18:24 UTC (permalink / raw)
To: Edgecombe, Rick P
Cc: dietmar.eggemann@arm.com, keescook@chromium.org, shuah@kernel.org,
brauner@kernel.org, dave.hansen@linux.intel.com,
debug@rivosinc.com, mgorman@suse.de, Szabolcs.Nagy@arm.com,
fweimer@redhat.com, linux-kernel@vger.kernel.org,
mingo@redhat.com, hjl.tools@gmail.com, rostedt@goodmis.org,
vincent.guittot@linaro.org, tglx@linutronix.de,
vschneid@redhat.com, catalin.marinas@arm.com, bristot@redhat.com,
will@kernel.org, hpa@zytor.com, peterz@infradead.org,
jannh@google.com, bp@alien8.de, bsegall@google.com,
linux-kselftest@vger.kernel.org, linux-api@vger.kernel.org,
x86@kernel.org, juri.lelli@redhat.com
In-Reply-To: <47bb581c20010552615125fce63e3c5c49d1c690.camel@intel.com>
[-- Attachment #1: Type: text/plain, Size: 2538 bytes --]
On Tue, Dec 05, 2023 at 10:23:08PM +0000, Edgecombe, Rick P wrote:
> On Tue, 2023-12-05 at 15:51 +0000, Mark Brown wrote:
> > Hrm, right. And we then can't use do_mmap() either. I'd be somewhat
> > tempted to disallow that specific case for now rather than deal with
> > it
> > though that's not really in the spirit of just always following what
> > the
> > user asked for.
> Oh, yea. What a pain. It doesn't seem like we could easily even add a
> do_mmap() variant that takes an mm either.
> I did a quick logging test on a Fedora userspace. systemd (I think)
> appears to do a clone(!CLONE_VM) with a stack passed. So maybe the
> combo might actually get used with a shadow_stack_size if it used
> clone3 some day. At the same time, fixing clone to mmap() in the child
> doesn't seem straight forward at all. Checking with some of our MM
> folks, the suggestion was to look at doing the child's shadow stack
> mapping in dup_mm() to avoid tripping over complications that happen
> when a remote MM becomes more "live".
Yeah, I can't see anything that looks particularly tasteful.
> If we just punt on this combination for now, then the documented rules
> for args->shadow_stack_size would be something like:
> clone3 will use the parents shadow stack when CLONE_VM is not present.
> If CLONE_VFORK is set then it will use the parents shadow stack only
> when args->shadow_stack_size is non-zero. In the cases when the parents
> shadow stack is not used, args->shadow_stack_size is used for the size
> whenever non-zero.
> I guess it doesn't seem too overly complicated. But I'm not thinking
> any of the options seem great. I'd unhappily lean towards not
Indeed, it's all really hard to get enthusiastic about.
> supporting shadow_stack_size!=0 && !CLONE_VM for now. But it seems like
> there may be a user for the unsupported case, so this would be just
> improving things a little and kicking the can down the road. I also
> wonder if this is a sign to reconsider the earlier token consuming
> design.
In the case where we have !CLONE_VM it should actually possible to reuse
the token (since the user is in at least some sense the child process
rather than the parent) so it's less pure overhead, providing you don't
mind the children of a given parent all using the same addresses for
their initial shadow stack.
I'll have a poke at the various options and come up with something,
hopefully this month but it's getting a bit busy so might be early
next year instead.
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]
^ permalink raw reply
* Re: [RFC] proposed libc interface and man page for listmount
From: Matthew House @ 2023-12-06 16:53 UTC (permalink / raw)
To: Miklos Szeredi
Cc: libc-alpha, linux-man, Alejandro Colomar, Linux API,
Florian Weimer, linux-fsdevel, Karel Zak, Ian Kent, David Howells,
Christian Brauner, Amir Goldstein, Arnd Bergmann
In-Reply-To: <CAJfpegvUWH9uncnxWj50o7p9WGWgV3BL2=EnqKY28S=4J4ywHw@mail.gmail.com>
On Wed, Dec 6, 2023 at 4:38 AM Miklos Szeredi <miklos@szeredi.hu> wrote:
> On Tue, 5 Dec 2023 at 18:51, Matthew House <mattlloydhouse@gmail.com> wrote:
> > One use case I've been thinking of involves inspecting the mount list
> > between syscall(__NR_clone3) and _exit(), so it has to be async-signal-
> > safe. It would be nice if there were a libc wrapper that accepted a user-
> > provided buffer and was async-signal-safe, so that I wouldn't have to add
> > yet another syscall wrapper and redefine the kernel types just for this
> > use case. (I can't trust the libc not to make its own funny versions of the
> > types' layouts for its own ends.)
>
> You can just #include <linux/mount.h> directly.
The problem with including the <linux/*> headers is that they conflict with
the regular libc headers. So for instance, if I try to include both
<linux/mount.h> (for the listmount(2) kernel types) and <sys/mount.h> (for
the mount(2) and umount2(2) wrappers) on glibc, then I'll get a conflicting
definition for every single MS_* macro.
I suppose I could try to put all the listmount(2) stuff in a separate file,
but that would still require manual redefinitions of the listmount(2)
flags, unless I trusted libc to have its own identical redefinitions in
<sys/mount.h> or whatever header the wrapper would end up in, instead of
shuffling stuff around and translating it. Also, my current style in C is
to put all related code into a single file as possible, which this would
interfere with. At that point, I might as well redefine the whole thing.
Thank you,
Matthew House
^ permalink raw reply
* Re: [RFC 1/3] pidfd: allow pidfd_open() on non-thread-group leaders
From: Tycho Andersen @ 2023-12-06 15:27 UTC (permalink / raw)
To: Florian Weimer
Cc: Mathieu Desnoyers, Christian Brauner, Oleg Nesterov,
Eric W . Biederman, linux-kernel, linux-api, Tycho Andersen
In-Reply-To: <87ttp3rprd.fsf@oldenburg.str.redhat.com>
On Thu, Nov 30, 2023 at 08:43:18PM +0100, Florian Weimer wrote:
> * Mathieu Desnoyers:
>
> >>> I'd like to offer a userspace API which allows safe stashing of
> >>> unreachable file descriptors on a service thread.
>
> >> By "safe" here do you mean not accessible via pidfd_getfd()?
>
> No, unreachable by close/close_range/dup2/dup3. I expect we can do an
> intra-process transfer using /proc, but I'm hoping for something nicer.
It occurred to me that we could get the seccomp() protected-memory
functionality almost all the way via some combination of
memfd_create(MFD_ALLOW_SEALING), fcntl(F_SEAL_WRITE|F_SEAL_SEAL), and
mmap(PROT_NONE). Some other thread could come along and unmap/remap,
but perhaps with some kind of F_SEAL_NOUNMAP married to one of these
special files we could both get what we want?
I submitted a talk to FOSDEM just for grins, if anyone is planning to
attend that.
Tycho
^ permalink raw reply
* Re: [RFC PATCH 0/2] Node migration between memory tiers
From: Michal Hocko @ 2023-12-06 15:22 UTC (permalink / raw)
To: Ravi Jonnalagadda
Cc: Jonathan.Cameron, aneesh.kumar, dan.j.williams, emirakhur, fvdl,
gregory.price, hannes, haowang3, hasanalmaruf, hezhongkun.hzk,
john, linux-api, linux-cxl, linux-kernel, linux-mm,
sthanneeru.opensrc, tj, vtavarespetr, ying.huang
In-Reply-To: <20231205091958.55-1-ravis.opensrc@micron.com>
On Tue 05-12-23 14:49:58, Ravi Jonnalagadda wrote:
[...]
> There was a prior discussion on this functionality in a previous thread, where
> Huang Ying thought this might be a useful feature to overcome limitations of
> systems where nodes with different bandwidth characteristics are grouped in
> a single tier.
Please summarize all those prior discussions into the cover letter.
Usecases are really crucial for the justification.
>
--
Michal Hocko
SUSE Labs
^ permalink raw reply
* Re: [RFC] proposed libc interface and man page for listmount
From: Florian Weimer @ 2023-12-06 11:05 UTC (permalink / raw)
To: Miklos Szeredi
Cc: libc-alpha, linux-man, Alejandro Colomar, Linux API,
linux-fsdevel, Karel Zak, Ian Kent, David Howells,
Christian Brauner, Amir Goldstein, Arnd Bergmann
In-Reply-To: <CAJfpeguMViqawKfJtM7_M9=m+6WsTcPfa_18t_rM9iuMG096RA@mail.gmail.com>
* Miklos Szeredi:
> Attaching the proposed man page for listing mounts (based on the new
> listmount() syscall).
>
> The raw interface is:
>
> syscall(__NR_listmount, const struct mnt_id_req __user *, req,
> u64 __user *, buf, size_t, bufsize, unsigned int, flags);
>
> The proposed libc API is.
>
> struct listmount *listmount_start(uint64_t mnt_id, unsigned int flags);
> uint64_t listmount_next(struct listmount *lm);
> void listmount_end(struct listmount *lm);
>
> I'm on the opinion that no wrapper is needed for the raw syscall, just
> like there isn't one for getdents(2).
We do have a wrapper for getdents64. It's useful because if you modify
the directory, you care about the buffer boundary because you should
rewind after processing the current buffer. The inotify facility also
exposes a sequence of variably sized objects to applications, but does
not add a new system call for that. That's just an aside, though.
The existing functions dealing with /proc/mounts or /etc/fstab are
called setmntent, getmntent or getmntment_r (the former with a bad
implementation, the latter with a bad interface), and endmntent. This
follows the pattern of NSS enumeration interfaces, except that in the
mntent case, there is an explicit file handle, so a thread-safe
implementation is possible in principle. Your proposed interface is
similar, so that's good.
I would also like to see a comment from the Hurd folks. Presumably they
have something similar already for enumerating translators?
Thanks,
Florian
^ permalink raw reply
* Re: [RFC] proposed libc interface and man page for listmount
From: Miklos Szeredi @ 2023-12-06 9:40 UTC (permalink / raw)
To: Karel Zak
Cc: libc-alpha, linux-man, Alejandro Colomar, Linux API,
Florian Weimer, linux-fsdevel, Ian Kent, David Howells,
Christian Brauner, Amir Goldstein, Arnd Bergmann
In-Reply-To: <20231205182629.qk5s6f7m7sas4anh@ws.net.home>
On Tue, 5 Dec 2023 at 19:26, Karel Zak <kzak@redhat.com> wrote:
>
> On Tue, Dec 05, 2023 at 05:27:58PM +0100, Miklos Szeredi wrote:
> > Attaching the proposed man page for listing mounts (based on the new
> > listmount() syscall).
> >
> > The raw interface is:
> >
> > syscall(__NR_listmount, const struct mnt_id_req __user *, req,
> > u64 __user *, buf, size_t, bufsize, unsigned int, flags);
> >
> > The proposed libc API is.
> >
> > struct listmount *listmount_start(uint64_t mnt_id, unsigned int flags);
> > uint64_t listmount_next(struct listmount *lm);
> > void listmount_end(struct listmount *lm);
>
> What about:
>
> getmountlist()
> nextmountlist()
> freemountlist()
>
> For me, _start and _end() sounds strange. For example, We already use
> get+free for getaddrinfo().
Fine by me. Just wanted to get the general scheme out for comment.
Thanks,
Miklos
^ permalink raw reply
* Re: [RFC] proposed libc interface and man page for listmount
From: Miklos Szeredi @ 2023-12-06 9:38 UTC (permalink / raw)
To: Matthew House
Cc: libc-alpha, linux-man, Alejandro Colomar, Linux API,
Florian Weimer, linux-fsdevel, Karel Zak, Ian Kent, David Howells,
Christian Brauner, Amir Goldstein, Arnd Bergmann
In-Reply-To: <20231205175117.686780-1-mattlloydhouse@gmail.com>
On Tue, 5 Dec 2023 at 18:51, Matthew House <mattlloydhouse@gmail.com> wrote:
>
> On Tue, Dec 5, 2023 at 11:28 AM Miklos Szeredi <miklos@szeredi.hu> wrote:
> > Attaching the proposed man page for listing mounts (based on the new
> > listmount() syscall).
> >
> > The raw interface is:
> >
> > syscall(__NR_listmount, const struct mnt_id_req __user *, req,
> > u64 __user *, buf, size_t, bufsize, unsigned int, flags);
> >
> > The proposed libc API is.
> >
> > struct listmount *listmount_start(uint64_t mnt_id, unsigned int flags);
> > uint64_t listmount_next(struct listmount *lm);
> > void listmount_end(struct listmount *lm);
> >
> > I'm on the opinion that no wrapper is needed for the raw syscall, just
> > like there isn't one for getdents(2).
> >
> > Comments?
>
> One use case I've been thinking of involves inspecting the mount list
> between syscall(__NR_clone3) and _exit(), so it has to be async-signal-
> safe. It would be nice if there were a libc wrapper that accepted a user-
> provided buffer and was async-signal-safe, so that I wouldn't have to add
> yet another syscall wrapper and redefine the kernel types just for this
> use case. (I can't trust the libc not to make its own funny versions of the
> types' layouts for its own ends.)
You can just #include <linux/mount.h> directly.
Thanks,
Miklos
^ permalink raw reply
* Re: [PATCH RFT v4 5/5] kselftest/clone3: Test shadow stack support
From: Edgecombe, Rick P @ 2023-12-05 22:31 UTC (permalink / raw)
To: broonie@kernel.org
Cc: dietmar.eggemann@arm.com, keescook@chromium.org,
Szabolcs.Nagy@arm.com, shuah@kernel.org,
dave.hansen@linux.intel.com, debug@rivosinc.com, mgorman@suse.de,
brauner@kernel.org, fweimer@redhat.com,
linux-kernel@vger.kernel.org, mingo@redhat.com,
hjl.tools@gmail.com, rostedt@goodmis.org,
vincent.guittot@linaro.org, tglx@linutronix.de,
vschneid@redhat.com, catalin.marinas@arm.com, bristot@redhat.com,
will@kernel.org, hpa@zytor.com, peterz@infradead.org,
jannh@google.com, bp@alien8.de, bsegall@google.com,
linux-kselftest@vger.kernel.org, linux-api@vger.kernel.org,
x86@kernel.org, juri.lelli@redhat.com
In-Reply-To: <098f5d43-e093-4316-9b86-80833c2b94ec@sirena.org.uk>
On Tue, 2023-12-05 at 16:43 +0000, Mark Brown wrote:
> Right, it's a small and fairly easily auditable list - it's more
> about
> the app than the double enable which was what I thought your concern
> was. It's a bit annoying definitely and not something we want to do
> in
> general but for something like this where we're adding specific
> coverage
> for API extensions for the feature it seems like a reasonable
> tradeoff.
>
> If the x86 toolchain/libc support is widely enough deployed (or you
> just
> don't mind any missing coverage) we could use the toolchain support
> there and only have the manual enable for arm64, it'd be inconsistent
> but not wildly so.
>
>
>
I'm hoping there is not too much of a gap before the glibc support
starts filtering out. Long term, elf bit enabling is probably the right
thing for the generic tests. Short term, manual enabling is ok with me
if no one else minds. Maybe we could add my "don't do" list as a
comment if we do manual enabling?
I'll have to check your new series, but I also wonder if we could cram
the manual enabling and status checking pieces into some headers and
not have to have "if x86" "if arm" logic in the test themselves.
^ permalink raw reply
* Re: [PATCH RFT v4 2/5] fork: Add shadow stack support to clone3()
From: Edgecombe, Rick P @ 2023-12-05 22:23 UTC (permalink / raw)
To: broonie@kernel.org
Cc: dietmar.eggemann@arm.com, keescook@chromium.org, shuah@kernel.org,
brauner@kernel.org, dave.hansen@linux.intel.com,
debug@rivosinc.com, mgorman@suse.de, Szabolcs.Nagy@arm.com,
fweimer@redhat.com, linux-kernel@vger.kernel.org,
mingo@redhat.com, hjl.tools@gmail.com, rostedt@goodmis.org,
vincent.guittot@linaro.org, tglx@linutronix.de,
vschneid@redhat.com, catalin.marinas@arm.com, bristot@redhat.com,
will@kernel.org, hpa@zytor.com, peterz@infradead.org,
jannh@google.com, bp@alien8.de, bsegall@google.com,
linux-kselftest@vger.kernel.org, linux-api@vger.kernel.org,
x86@kernel.org, juri.lelli@redhat.com
In-Reply-To: <ed665d6f-66b0-4eeb-8cf8-db852e017d6a@sirena.org.uk>
On Tue, 2023-12-05 at 15:51 +0000, Mark Brown wrote:
> On Tue, Dec 05, 2023 at 12:26:57AM +0000, Edgecombe, Rick P wrote:
> > On Tue, 2023-11-28 at 18:22 +0000, Mark Brown wrote:
>
> > > - size = adjust_shstk_size(stack_size);
> > > + size = adjust_shstk_size(size);
> > > addr = alloc_shstk(0, size, 0, false);
>
> > Hmm. I didn't test this, but in the copy_process(), copy_mm()
> > happens
> > before this point. So the shadow stack would get mapped in
> > current's MM
> > (i.e. the parent). So in the !CLONE_VM case with
> > shadow_stack_size!=0
> > the SSP in the child will be updated to an area that is not mapped
> > in
> > the child. I think we need to pass tsk->mm into alloc_shstk(). But
> > such
> > an exotic clone usage does give me pause, regarding whether all of
> > this
> > is premature.
>
> Hrm, right. And we then can't use do_mmap() either. I'd be somewhat
> tempted to disallow that specific case for now rather than deal with
> it
> though that's not really in the spirit of just always following what
> the
> user asked for.
Oh, yea. What a pain. It doesn't seem like we could easily even add a
do_mmap() variant that takes an mm either.
I did a quick logging test on a Fedora userspace. systemd (I think)
appears to do a clone(!CLONE_VM) with a stack passed. So maybe the
combo might actually get used with a shadow_stack_size if it used
clone3 some day. At the same time, fixing clone to mmap() in the child
doesn't seem straight forward at all. Checking with some of our MM
folks, the suggestion was to look at doing the child's shadow stack
mapping in dup_mm() to avoid tripping over complications that happen
when a remote MM becomes more "live".
If we just punt on this combination for now, then the documented rules
for args->shadow_stack_size would be something like:
clone3 will use the parents shadow stack when CLONE_VM is not present.
If CLONE_VFORK is set then it will use the parents shadow stack only
when args->shadow_stack_size is non-zero. In the cases when the parents
shadow stack is not used, args->shadow_stack_size is used for the size
whenever non-zero.
I guess it doesn't seem too overly complicated. But I'm not thinking
any of the options seem great. I'd unhappily lean towards not
supporting shadow_stack_size!=0 && !CLONE_VM for now. But it seems like
there may be a user for the unsupported case, so this would be just
improving things a little and kicking the can down the road. I also
wonder if this is a sign to reconsider the earlier token consuming
design.
^ permalink raw reply
page: next (older) | prev (newer) | latest
- recent:[subjects (threaded)|topics (new)|topics (active)]
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox