* Re: [PATCH v5 09/11] mm/mempolicy: add get_mempolicy2 syscall
From: Geert Uytterhoeven @ 2024-01-02 14:46 UTC (permalink / raw)
To: Gregory Price
Cc: linux-mm, linux-doc, linux-fsdevel, linux-kernel, linux-api, x86,
akpm, arnd, tglx, luto, mingo, bp, dave.hansen, hpa, mhocko, tj,
ying.huang, gregory.price, corbet, rakie.kim, hyeongtak.ji,
honggyu.kim, vtavarespetr, peterz, jgroves, ravis.opensrc,
sthanneeru, emirakhur, Hasan.Maruf, seungjun.ha, Michal Hocko
In-Reply-To: <20231223181101.1954-10-gregory.price@memverge.com>
On Sat, Dec 23, 2023 at 7:14 PM Gregory Price <gourry.memverge@gmail.com> wrote:
> get_mempolicy2 is an extensible get_mempolicy interface which allows
> a user to retrieve the memory policy for a task or address.
>
> Defined as:
>
> get_mempolicy2(struct mpol_args *args, size_t size,
> unsigned long addr, unsigned long flags)
>
> Top level input values:
>
> mpol_args: The field which collects information about the mempolicy
> returned to userspace.
> addr: if MPOL_F_ADDR is passed in `flags`, this address will be
> used to return the mempolicy details of the vma the
> address belongs to
> flags: if MPOL_F_ADDR, return mempolicy info vma containing addr
> else, returns task mempolicy information
>
> Input values include the following fields of mpol_args:
>
> pol_nodes: if set, the nodemask of the policy returned here
> pol_maxnodes: if pol_nodes is set, must describe max number of nodes
> to be copied to pol_nodes
>
> Output values include the following fields of mpol_args:
>
> mode: mempolicy mode
> mode_flags: mempolicy mode flags
> home_node: policy home node will be returned here, or -1 if not.
> pol_nodes: if set, the nodemask for the mempolicy
> policy_node: if the policy has extended node information, it will
> be placed here. For example MPOL_INTERLEAVE will
> return the next node which will be used for allocation
>
> MPOL_F_NODE has been dropped from get_mempolicy2 (EINVAL).
> MPOL_F_MEMS_ALLOWED has been dropped from get_mempolicy2 (EINVAL).
>
> Suggested-by: Michal Hocko <mhocko@suse.com>
> Signed-off-by: Gregory Price <gregory.price@memverge.com>
> arch/m68k/kernel/syscalls/syscall.tbl | 1 +
Acked-by: Geert Uytterhoeven <geert@linux-m68k.org>
Gr{oetje,eeting}s,
Geert
--
Geert Uytterhoeven -- There's lots of Linux beyond ia32 -- geert@linux-m68k.org
In personal conversations with technical people, I call myself a hacker. But
when I'm talking to journalists I just say "programmer" or something like that.
-- Linus Torvalds
^ permalink raw reply
* Re: [PATCH v5 00/11] mempolicy2, mbind2, and weighted interleave
From: Gregory Price @ 2024-01-02 19:06 UTC (permalink / raw)
To: Huang, Ying
Cc: Gregory Price, linux-mm, linux-doc, linux-fsdevel, linux-kernel,
linux-api, x86, akpm, arnd, tglx, luto, mingo, bp, dave.hansen,
hpa, mhocko, tj, corbet, rakie.kim, hyeongtak.ji, honggyu.kim,
vtavarespetr, peterz, jgroves, ravis.opensrc, sthanneeru,
emirakhur, Hasan.Maruf, seungjun.ha, Johannes Weiner,
Hasan Al Maruf, Hao Wang, Dan Williams, Michal Hocko, Zhongkun He,
Frank van der Linden, John Groves, Jonathan Cameron
In-Reply-To: <87le98e4w1.fsf@yhuang6-desk2.ccr.corp.intel.com>
> >> > struct mpol_args {
> >> > /* Basic mempolicy settings */
> >> > __u16 mode;
> >> > __u16 mode_flags;
> >> > __s32 home_node;
> >> > __u64 pol_maxnodes;
> >>
> >> I understand that we want to avoid hole in struct. But I still feel
> >> uncomfortable to use __u64 for a small. But I don't have solution too.
> >> Anyone else has some idea?
> >>
> >
> > maxnode has been an `unsigned long` in every other interface for quite
> > some time. Seems better to keep this consistent rather than it suddenly
> > become `unsigned long` over here and `unsigned short` over there.
>
> I don't think that it matters. The actual maximum node number will be
> less than maximum `unsigned short`.
>
the structure will end up being
struct mpol_args {
__u16 mode;
__u16 mode_flags;
__s32 home_node;
__u16 pol_maxnodes;
__u8 rsv[6];
__aligned_u64 pol_nodes;
__aligned_u64 il_weights;
}
If you're fine with that, i'll make the change.
~Gregory
^ permalink raw reply
* Re: [PATCH v5 01/11] mm/mempolicy: implement the sysfs-based weighted_interleave interface
From: Gregory Price @ 2024-01-02 19:45 UTC (permalink / raw)
To: Huang, Ying
Cc: Gregory Price, linux-mm, linux-doc, linux-fsdevel, linux-kernel,
linux-api, x86, akpm, arnd, tglx, luto, mingo, bp, dave.hansen,
hpa, mhocko, tj, corbet, rakie.kim, hyeongtak.ji, honggyu.kim,
vtavarespetr, peterz, jgroves, ravis.opensrc, sthanneeru,
emirakhur, Hasan.Maruf, seungjun.ha
In-Reply-To: <87h6jwdvxn.fsf@yhuang6-desk2.ccr.corp.intel.com>
On Tue, Jan 02, 2024 at 03:41:08PM +0800, Huang, Ying wrote:
> Gregory Price <gregory.price@memverge.com> writes:
>
> That is, if we use "1" as default weight, we need to change weights of
> nodes frequently because we haven't a "base" weight. The best candidate
> base weight is the weight of DRAM node. For example, if we set the
> default weight of DRAM node to be "16" and use that as the base weight,
> we don't need to change it in most cases. The weight of other nodes can
> be set according to the ratio of its memory bandwidth to that of DRAM.
>
> This makes it easy to set the default weight via HMAT/CDAT too.
>
> What do you think about that?
>
You're getting a bit ahead of the patch set. There is "what is a
reasonable default weight" and "what is the minumum functionality".
The minimum functionality is everything receiving a default weight of 1,
such that weighted interleave's behavior defaults to round-robin
interleave. This gets the system off the ground.
We can then expose an internal interface to drivers for them to set the
default weight to some reasonable number during system and device
initialization. The question at that point is what system is responsible
for setting the default weights... node? cxl? anything? What happens on
hotplug? etc. That seems outside the scope of this patch set.
If you want me to add the default_iw_table with special value 0 denoting
"use default" at each layer, I can do that.
The basic change is this snippet:
```
if (pol->flags & MPOL_F_GWEIGHT)
pol_weights = iw_table;
else
pol_weights = pol->wil.weights;
for_each_node_mask(nid, nodemask) {
weight = pol_weights[nid];
weight_total += weight;
weights[nid] = weight;
}
```
changes to:
```
for_each_node_mask(nid, nodemask) {
weight = pol->wil.weights[node]
if (!weight)
weight = iw_table[node]
if (!weight)
weight = default_iw_table[node]
weight_total += weight;
weights[nid] = weight
}
```
It's a bit ugly, but it allows a 0 value to represent "use default",
and default_iw_table just ends up being initialized to `1` for now.
I think it also allows MPOL_F_GWEIGHT to be eliminated.
~Gregory
^ permalink raw reply
* Re: [PATCH v5 02/11] mm/mempolicy: introduce MPOL_WEIGHTED_INTERLEAVE for weighted interleaving
From: Gregory Price @ 2024-01-02 20:30 UTC (permalink / raw)
To: Huang, Ying
Cc: Gregory Price, linux-mm, linux-doc, linux-fsdevel, linux-kernel,
linux-api, x86, akpm, arnd, tglx, luto, mingo, bp, dave.hansen,
hpa, mhocko, tj, corbet, rakie.kim, hyeongtak.ji, honggyu.kim,
vtavarespetr, peterz, jgroves, ravis.opensrc, sthanneeru,
emirakhur, Hasan.Maruf, seungjun.ha, Srinivasulu Thanneeru
In-Reply-To: <878r58dt31.fsf@yhuang6-desk2.ccr.corp.intel.com>
On Tue, Jan 02, 2024 at 04:42:42PM +0800, Huang, Ying wrote:
> Gregory Price <gregory.price@memverge.com> writes:
>
> > On Wed, Dec 27, 2023 at 04:32:37PM +0800, Huang, Ying wrote:
> >> Gregory Price <gourry.memverge@gmail.com> writes:
> >>
> >> > +static unsigned int weighted_interleave_nid(struct mempolicy *pol, pgoff_t ilx)
> >> > +{
> >> > + nodemask_t nodemask = pol->nodes;
> >> > + unsigned int target, weight_total = 0;
> >> > + int nid;
> >> > + unsigned char weights[MAX_NUMNODES];
> >>
> >> MAX_NUMNODSE could be as large as 1024. 1KB stack space may be too
> >> large?
> >>
> >
> > I've been struggling with a good solution to this. We need a local copy
> > of weights to prevent weights from changing out from under us during
> > allocation (which may take quite some time), but it seemed unwise to
> > to allocate 1KB heap in this particular path.
> >
> > Is my concern unfounded? If so, I can go ahead and add the allocation
> > code.
>
> Please take a look at NODEMASK_ALLOC().
>
This is not my question. NODEMASK_ALLOC calls kmalloc/kfree.
Some of the allocations on the stack can be replaced with a scratch
allocation, that's no big deal.
I'm specifically concerned about:
weighted_interleave_nid
alloc_pages_bulk_array_weighted_interleave
I'm unsure whether kmalloc/kfree is safe (and non-offensive) in those
contexts. If kmalloc/kfree is safe fine, this problem is trivial.
If not, there is no good solution to this without pre-allocating a
scratch area per-task.
> >> I don't think barrier() is needed to wait for memory operations for
> >> stack. It's usually used for cross-processor memory order.
> >>
> >
> > This is present in the old interleave code. To the best of my
> > understanding, the concern is for mempolicy->nodemask rebinding that can
> > occur when cgroups.cpusets.mems_allowed changes.
> >
> > so we can't iterate over (mempolicy->nodemask), we have to take a local
> > copy.
> >
> > My *best* understanding of the barrier here is to prevent the compiler
> > from reordering operations such that it attempts to optimize out the
> > local copy (or do lazy-fetch).
> >
> > It is present in the original interleave code, so I pulled it forward to
> > this, but I have not tested whether this is a bit paranoid or not.
> >
> > from `interleave_nid`:
> >
> > /*
> > * The barrier will stabilize the nodemask in a register or on
> > * the stack so that it will stop changing under the code.
> > *
> > * Between first_node() and next_node(), pol->nodes could be changed
> > * by other threads. So we put pol->nodes in a local stack.
> > */
> > barrier();
>
> Got it. This is kind of READ_ONCE() for nodemask. To avoid to add
> comments all over the place. Can we implement a wrapper for it? For
> example, memcpy_once(). __read_once_size() in
> tools/include/linux/compiler.h can be used as reference.
>
> Because node_weights[] may be changed simultaneously too. We may need
> to consider similar issue for it too. But RCU seems more appropriate
> for node_weights[].
>
Weights are collected individually onto the stack because we have to sum
them up before we actually apply the weights.
A stale weight is not offensive. RCU is not needed and doesn't help.
The reason the barrier is needed is not weights, it's the nodemask.
So you basically just want to replace barrier() with this and drop the
copy/pasted comments:
static void read_once_policy_nodemask(struct mempolicy *pol, nodemask_t *mask)
{
/*
* The barrier will stabilize the nodemask in a register or on
* the stack so that it will stop changing under the code.
*
* Between first_node() and next_node(), pol->nodes could be changed
* by other threads. So we put pol->nodes in a local stack.
*/
barrier();
__builtin_memcpy(mask, &pol->nodes, sizeof(nodemask_t));
barrier();
}
- nodemask_t nodemask = pol->nodemask
- barrier()
+ nodemask_t nodemask;
+ read_once_policy_nodemask(pol, &nodemask)
Is that right?
~Gregory
^ permalink raw reply
* Re: [PATCH v5 03/11] mm/mempolicy: refactor sanitize_mpol_flags for reuse
From: Gregory Price @ 2024-01-02 20:32 UTC (permalink / raw)
To: Huang, Ying
Cc: Gregory Price, linux-mm, linux-doc, linux-fsdevel, linux-kernel,
linux-api, x86, akpm, arnd, tglx, luto, mingo, bp, dave.hansen,
hpa, mhocko, tj, corbet, rakie.kim, hyeongtak.ji, honggyu.kim,
vtavarespetr, peterz, jgroves, ravis.opensrc, sthanneeru,
emirakhur, Hasan.Maruf, seungjun.ha
In-Reply-To: <871qb0drto.fsf@yhuang6-desk2.ccr.corp.intel.com>
On Tue, Jan 02, 2024 at 05:09:55PM +0800, Huang, Ying wrote:
> Gregory Price <gregory.price@memverge.com> writes:
>
> > On Tue, Dec 26, 2023 at 02:05:35AM -0500, Gregory Price wrote:
> >> On Wed, Dec 27, 2023 at 04:39:29PM +0800, Huang, Ying wrote:
> >> > Gregory Price <gourry.memverge@gmail.com> writes:
> >> >
> >> > > + unsigned short mode = (*mode_arg & ~MPOL_MODE_FLAGS);
> >> > > +
> >> > > + *flags = *mode_arg & MPOL_MODE_FLAGS;
> >> > > + *mode_arg = mode;
> >> >
> >> > It appears that it's unnecessary to introduce a local variable to split
> >> > mode/flags. Just reuse the original code?
> >> >
> >
> > Revisiting during fixes: Note the change from int to short.
> >
> > I chose to make this explicit because validate_mpol_flags takes a short.
> >
> > I'm fairly sure changing it back throws a truncation warning.
>
> Why something like below doesn't work?
>
> int sanitize_mpol_flags(int *mode, unsigned short *flags)
> {
> *flags = *mode & MPOL_MODE_FLAGS;
> *mode &= ~MPOL_MODE_FLAGS;
>
> return validate_mpol_flags(*mode, flags);
> }
was concerned with silent truncation of (*mode) (int) to short.
*shrug* happy to change it
~Gregory
^ permalink raw reply
* Re: [PATCH v5 01/11] mm/mempolicy: implement the sysfs-based weighted_interleave interface
From: Gregory Price @ 2024-01-03 2:46 UTC (permalink / raw)
To: Huang, Ying
Cc: Gregory Price, linux-mm, linux-doc, linux-fsdevel, linux-kernel,
linux-api, x86, akpm, arnd, tglx, luto, mingo, bp, dave.hansen,
hpa, mhocko, tj, corbet, rakie.kim, hyeongtak.ji, honggyu.kim,
vtavarespetr, peterz, jgroves, ravis.opensrc, sthanneeru,
emirakhur, Hasan.Maruf, seungjun.ha
In-Reply-To: <87h6jwdvxn.fsf@yhuang6-desk2.ccr.corp.intel.com>
On Tue, Jan 02, 2024 at 03:41:08PM +0800, Huang, Ying wrote:
> Think about the default weight value via HMAT/CDAT again. It may be not
> a good idea to use "1" as default even for now.
>
> For example,
>
> - The memory bandwidth of DRAM is 100GB, whose default weight is "1".
>
> - We hot-plug CXL.mem A with memory bandwidth 20GB. So, we change the
> weight of DRAM to 5, and use "1" as the weight of CXL.mem A.
>
> - We hot-plug CXL.mem B with memory bandwidth 10GB. So, we change the
> weight of DRAM to 10, the weight of CXL.mem A to 2, and use "1" as the
> weight of CXL.mem B.
>
> That is, if we use "1" as default weight, we need to change weights of
> nodes frequently because we haven't a "base" weight. The best candidate
> base weight is the weight of DRAM node. For example, if we set the
> default weight of DRAM node to be "16" and use that as the base weight,
> we don't need to change it in most cases. The weight of other nodes can
> be set according to the ratio of its memory bandwidth to that of DRAM.
>
> This makes it easy to set the default weight via HMAT/CDAT too.
>
> What do you think about that?
>
Giving this more thought.
Hotplug should be an incredibly rare event. I don't think swapping defaults
"frequently" is a real problem we should handle.
It's expected that dynamic capacity devices will not cause a node to
hotplug, but instead cause a node to grow/shrink.
Seems perfectly fine to rebalance weights in response to rare events.
~Gregory
^ permalink raw reply
* Re: [PATCH v5 01/11] mm/mempolicy: implement the sysfs-based weighted_interleave interface
From: Huang, Ying @ 2024-01-03 2:45 UTC (permalink / raw)
To: Gregory Price
Cc: Gregory Price, linux-mm, linux-doc, linux-fsdevel, linux-kernel,
linux-api, x86, akpm, arnd, tglx, luto, mingo, bp, dave.hansen,
hpa, mhocko, tj, corbet, rakie.kim, hyeongtak.ji, honggyu.kim,
vtavarespetr, peterz, jgroves, ravis.opensrc, sthanneeru,
emirakhur, Hasan.Maruf, seungjun.ha
In-Reply-To: <ZZRn04IiZhet8peu@memverge.com>
Gregory Price <gregory.price@memverge.com> writes:
> On Tue, Jan 02, 2024 at 03:41:08PM +0800, Huang, Ying wrote:
>> Gregory Price <gregory.price@memverge.com> writes:
>>
>> That is, if we use "1" as default weight, we need to change weights of
>> nodes frequently because we haven't a "base" weight. The best candidate
>> base weight is the weight of DRAM node. For example, if we set the
>> default weight of DRAM node to be "16" and use that as the base weight,
>> we don't need to change it in most cases. The weight of other nodes can
>> be set according to the ratio of its memory bandwidth to that of DRAM.
>>
>> This makes it easy to set the default weight via HMAT/CDAT too.
>>
>> What do you think about that?
>>
>
> You're getting a bit ahead of the patch set. There is "what is a
> reasonable default weight" and "what is the minumum functionality".
I totally agree that we need the minimal functionality firstly.
> The minimum functionality is everything receiving a default weight of 1,
> such that weighted interleave's behavior defaults to round-robin
> interleave. This gets the system off the ground.
I don't think that we need to implement all functionalities now. But,
we may need to consider more especially if it may impact the user space
interface. The default base weight is something like that. If we
change the default base weight from "1" to "16" later, users may be
surprised. So, I think it's better to discuss it now.
> We can then expose an internal interface to drivers for them to set the
> default weight to some reasonable number during system and device
> initialization. The question at that point is what system is responsible
> for setting the default weights... node? cxl? anything? What happens on
> hotplug? etc. That seems outside the scope of this patch set.
>
>
> If you want me to add the default_iw_table with special value 0 denoting
> "use default" at each layer, I can do that.
>
> The basic change is this snippet:
> ```
> if (pol->flags & MPOL_F_GWEIGHT)
> pol_weights = iw_table;
> else
> pol_weights = pol->wil.weights;
>
> for_each_node_mask(nid, nodemask) {
> weight = pol_weights[nid];
> weight_total += weight;
> weights[nid] = weight;
> }
> ```
>
> changes to:
> ```
> for_each_node_mask(nid, nodemask) {
> weight = pol->wil.weights[node]
> if (!weight)
> weight = iw_table[node]
> if (!weight)
> weight = default_iw_table[node]
> weight_total += weight;
> weights[nid] = weight
> }
> ```
>
> It's a bit ugly,
We can use a wrapper function to hide the logic.
> but it allows a 0 value to represent "use default",
> and default_iw_table just ends up being initialized to `1` for now.
Because the contents of default_iw_table are just "default weight" for
now. We don't need it for now. We can add it later.
> I think it also allows MPOL_F_GWEIGHT to be eliminated.
Do we need a way to distinguish whether to copy the global weights to
local weights when the memory policy is created? That is, when the
global weights are changed later, will the changes be used? One
possible solution is
- If no weights are specified in set_mempolicy2(), the global weights
will be used always.
- If at least one weight is specified in set_mempolicy2(), it will be
used, and the other weights in global weights will be copied to the
local weights. That is, changes to the global weights will not be
used.
--
Best Regards,
Huang, Ying
^ permalink raw reply
* Re: [PATCH v5 01/11] mm/mempolicy: implement the sysfs-based weighted_interleave interface
From: Gregory Price @ 2024-01-03 2:59 UTC (permalink / raw)
To: Huang, Ying
Cc: Gregory Price, linux-mm, linux-doc, linux-fsdevel, linux-kernel,
linux-api, x86, akpm, arnd, tglx, luto, mingo, bp, dave.hansen,
hpa, mhocko, tj, corbet, rakie.kim, hyeongtak.ji, honggyu.kim,
vtavarespetr, peterz, jgroves, ravis.opensrc, sthanneeru,
emirakhur, Hasan.Maruf, seungjun.ha
In-Reply-To: <87wmsrcexq.fsf@yhuang6-desk2.ccr.corp.intel.com>
On Wed, Jan 03, 2024 at 10:45:53AM +0800, Huang, Ying wrote:
>
> > The minimum functionality is everything receiving a default weight of 1,
> > such that weighted interleave's behavior defaults to round-robin
> > interleave. This gets the system off the ground.
>
> I don't think that we need to implement all functionalities now. But,
> we may need to consider more especially if it may impact the user space
> interface. The default base weight is something like that. If we
> change the default base weight from "1" to "16" later, users may be
> surprised. So, I think it's better to discuss it now.
>
This is a hill I don't particularly care to die on. I think the weights
are likely to end up being set at boot and rebalanced as (rare) hotplug
events occur.
So if people think the default weight should be 3,16,24 or 123, i don't
think it's going to matter.
>
> We can use a wrapper function to hide the logic.
>
Done. I'll push a new set tomorrow.
> > I think it also allows MPOL_F_GWEIGHT to be eliminated.
>
> Do we need a way to distinguish whether to copy the global weights to
> local weights when the memory policy is created? That is, when the
> global weights are changed later, will the changes be used? One
> possible solution is
>
> - If no weights are specified in set_mempolicy2(), the global weights
> will be used always.
>
> - If at least one weight is specified in set_mempolicy2(), it will be
> used, and the other weights in global weights will be copied to the
> local weights. That is, changes to the global weights will not be
> used.
>
What's confusing about that is that if a user sets a weight to 0,
they'll get a non-0 weight - always.
In my opinion, if we want to make '0' mean 'use system default', then
it should mean 'ALWAYS use system default for this node'.
"Use the system default at the time the syscall was called, and do not
update to use a new system default if that default is changed" is
confusing.
If you say use a global value, use the global value. Simple.
> --
> Best Regards,
> Huang, Ying
^ permalink raw reply
* Re: [PATCH v5 00/11] mempolicy2, mbind2, and weighted interleave
From: Huang, Ying @ 2024-01-03 3:15 UTC (permalink / raw)
To: Gregory Price
Cc: Gregory Price, linux-mm, linux-doc, linux-fsdevel, linux-kernel,
linux-api, x86, akpm, arnd, tglx, luto, mingo, bp, dave.hansen,
hpa, mhocko, tj, corbet, rakie.kim, hyeongtak.ji, honggyu.kim,
vtavarespetr, peterz, jgroves, ravis.opensrc, sthanneeru,
emirakhur, Hasan.Maruf, seungjun.ha, Johannes Weiner,
Hasan Al Maruf, Hao Wang, Dan Williams, Michal Hocko, Zhongkun He,
Frank van der Linden, John Groves, Jonathan Cameron
In-Reply-To: <ZZRepTEFNFC17fjT@memverge.com>
Gregory Price <gregory.price@memverge.com> writes:
>> >> > struct mpol_args {
>> >> > /* Basic mempolicy settings */
>> >> > __u16 mode;
>> >> > __u16 mode_flags;
>> >> > __s32 home_node;
>> >> > __u64 pol_maxnodes;
>> >>
>> >> I understand that we want to avoid hole in struct. But I still feel
>> >> uncomfortable to use __u64 for a small. But I don't have solution too.
>> >> Anyone else has some idea?
>> >>
>> >
>> > maxnode has been an `unsigned long` in every other interface for quite
>> > some time. Seems better to keep this consistent rather than it suddenly
>> > become `unsigned long` over here and `unsigned short` over there.
>>
>> I don't think that it matters. The actual maximum node number will be
>> less than maximum `unsigned short`.
>>
>
> the structure will end up being
>
> struct mpol_args {
> __u16 mode;
> __u16 mode_flags;
> __s32 home_node;
> __u16 pol_maxnodes;
> __u8 rsv[6];
> __aligned_u64 pol_nodes;
> __aligned_u64 il_weights;
> }
>
> If you're fine with that, i'll make the change.
This looks OK for me. But, I don't know whether others think this is
better.
--
Best Regards,
Huang, Ying
^ permalink raw reply
* Re: [PATCH v5 02/11] mm/mempolicy: introduce MPOL_WEIGHTED_INTERLEAVE for weighted interleaving
From: Huang, Ying @ 2024-01-03 5:46 UTC (permalink / raw)
To: Gregory Price
Cc: Gregory Price, linux-mm, linux-doc, linux-fsdevel, linux-kernel,
linux-api, x86, akpm, arnd, tglx, luto, mingo, bp, dave.hansen,
hpa, mhocko, tj, corbet, rakie.kim, hyeongtak.ji, honggyu.kim,
vtavarespetr, peterz, jgroves, ravis.opensrc, sthanneeru,
emirakhur, Hasan.Maruf, seungjun.ha, Srinivasulu Thanneeru
In-Reply-To: <ZZRybDPSoLme8Ldh@memverge.com>
Gregory Price <gregory.price@memverge.com> writes:
> On Tue, Jan 02, 2024 at 04:42:42PM +0800, Huang, Ying wrote:
>> Gregory Price <gregory.price@memverge.com> writes:
>>
>> > On Wed, Dec 27, 2023 at 04:32:37PM +0800, Huang, Ying wrote:
>> >> Gregory Price <gourry.memverge@gmail.com> writes:
>> >>
>> >> > +static unsigned int weighted_interleave_nid(struct mempolicy *pol, pgoff_t ilx)
>> >> > +{
>> >> > + nodemask_t nodemask = pol->nodes;
>> >> > + unsigned int target, weight_total = 0;
>> >> > + int nid;
>> >> > + unsigned char weights[MAX_NUMNODES];
>> >>
>> >> MAX_NUMNODSE could be as large as 1024. 1KB stack space may be too
>> >> large?
>> >>
>> >
>> > I've been struggling with a good solution to this. We need a local copy
>> > of weights to prevent weights from changing out from under us during
>> > allocation (which may take quite some time), but it seemed unwise to
>> > to allocate 1KB heap in this particular path.
>> >
>> > Is my concern unfounded? If so, I can go ahead and add the allocation
>> > code.
>>
>> Please take a look at NODEMASK_ALLOC().
>>
>
> This is not my question. NODEMASK_ALLOC calls kmalloc/kfree.
>
> Some of the allocations on the stack can be replaced with a scratch
> allocation, that's no big deal.
>
> I'm specifically concerned about:
> weighted_interleave_nid
> alloc_pages_bulk_array_weighted_interleave
>
> I'm unsure whether kmalloc/kfree is safe (and non-offensive) in those
> contexts. If kmalloc/kfree is safe fine, this problem is trivial.
>
> If not, there is no good solution to this without pre-allocating a
> scratch area per-task.
You need to audit whether it's safe for all callers. I guess that you
need to allocate pages after calling, so you can use the same GFP flags
here.
>> >> I don't think barrier() is needed to wait for memory operations for
>> >> stack. It's usually used for cross-processor memory order.
>> >>
>> >
>> > This is present in the old interleave code. To the best of my
>> > understanding, the concern is for mempolicy->nodemask rebinding that can
>> > occur when cgroups.cpusets.mems_allowed changes.
>> >
>> > so we can't iterate over (mempolicy->nodemask), we have to take a local
>> > copy.
>> >
>> > My *best* understanding of the barrier here is to prevent the compiler
>> > from reordering operations such that it attempts to optimize out the
>> > local copy (or do lazy-fetch).
>> >
>> > It is present in the original interleave code, so I pulled it forward to
>> > this, but I have not tested whether this is a bit paranoid or not.
>> >
>> > from `interleave_nid`:
>> >
>> > /*
>> > * The barrier will stabilize the nodemask in a register or on
>> > * the stack so that it will stop changing under the code.
>> > *
>> > * Between first_node() and next_node(), pol->nodes could be changed
>> > * by other threads. So we put pol->nodes in a local stack.
>> > */
>> > barrier();
>>
>> Got it. This is kind of READ_ONCE() for nodemask. To avoid to add
>> comments all over the place. Can we implement a wrapper for it? For
>> example, memcpy_once(). __read_once_size() in
>> tools/include/linux/compiler.h can be used as reference.
>>
>> Because node_weights[] may be changed simultaneously too. We may need
>> to consider similar issue for it too. But RCU seems more appropriate
>> for node_weights[].
>>
>
> Weights are collected individually onto the stack because we have to sum
> them up before we actually apply the weights.
>
> A stale weight is not offensive. RCU is not needed and doesn't help.
When you copy weights from iw_table[] to stack, it's possible for
compiler to cache its contents in register, or merge, split the memory
operations. At the same time, iw_table[] may be changed simultaneously
via sysfs interface. So, we need a mechanism to guarantee that we read
the latest contents consistently.
> The reason the barrier is needed is not weights, it's the nodemask.
Yes. So I said that we need similar stuff for weights.
> So you basically just want to replace barrier() with this and drop the
> copy/pasted comments:
>
> static void read_once_policy_nodemask(struct mempolicy *pol, nodemask_t *mask)
> {
> /*
> * The barrier will stabilize the nodemask in a register or on
> * the stack so that it will stop changing under the code.
> *
> * Between first_node() and next_node(), pol->nodes could be changed
> * by other threads. So we put pol->nodes in a local stack.
> */
> barrier();
> __builtin_memcpy(mask, &pol->nodes, sizeof(nodemask_t));
> barrier();
> }
>
> - nodemask_t nodemask = pol->nodemask
> - barrier()
> + nodemask_t nodemask;
> + read_once_policy_nodemask(pol, &nodemask)
>
> Is that right?
Yes. Something like that. Or even more general (need to be optimized?),
static inline static void memcpy_once(void *dst, void *src, size_t n)
{
barrier();
memcpy(dst, src, n);
barrier();
}
memcpy_once(&nodemask, &pol->nodemask, sizeof(nodemask));
The comments can be based on that of READ_ONCE().
--
Best Regards,
Huang, Ying
^ permalink raw reply
* Re: [PATCH v5 01/11] mm/mempolicy: implement the sysfs-based weighted_interleave interface
From: Huang, Ying @ 2024-01-03 6:03 UTC (permalink / raw)
To: Gregory Price
Cc: Gregory Price, linux-mm, linux-doc, linux-fsdevel, linux-kernel,
linux-api, x86, akpm, arnd, tglx, luto, mingo, bp, dave.hansen,
hpa, mhocko, tj, corbet, rakie.kim, hyeongtak.ji, honggyu.kim,
vtavarespetr, peterz, jgroves, ravis.opensrc, sthanneeru,
emirakhur, Hasan.Maruf, seungjun.ha
In-Reply-To: <ZZTNpGhj8EmYBB70@memverge.com>
Gregory Price <gregory.price@memverge.com> writes:
> On Wed, Jan 03, 2024 at 10:45:53AM +0800, Huang, Ying wrote:
>>
>> > The minimum functionality is everything receiving a default weight of 1,
>> > such that weighted interleave's behavior defaults to round-robin
>> > interleave. This gets the system off the ground.
>>
>> I don't think that we need to implement all functionalities now. But,
>> we may need to consider more especially if it may impact the user space
>> interface. The default base weight is something like that. If we
>> change the default base weight from "1" to "16" later, users may be
>> surprised. So, I think it's better to discuss it now.
>>
>
> This is a hill I don't particularly care to die on. I think the weights
> are likely to end up being set at boot and rebalanced as (rare) hotplug
> events occur.
>
> So if people think the default weight should be 3,16,24 or 123, i don't
> think it's going to matter.
>
>>
>> We can use a wrapper function to hide the logic.
>>
>
> Done. I'll push a new set tomorrow.
>
>> > I think it also allows MPOL_F_GWEIGHT to be eliminated.
>>
>> Do we need a way to distinguish whether to copy the global weights to
>> local weights when the memory policy is created? That is, when the
>> global weights are changed later, will the changes be used? One
>> possible solution is
>>
>> - If no weights are specified in set_mempolicy2(), the global weights
>> will be used always.
>>
>> - If at least one weight is specified in set_mempolicy2(), it will be
>> used, and the other weights in global weights will be copied to the
>> local weights. That is, changes to the global weights will not be
>> used.
>>
>
> What's confusing about that is that if a user sets a weight to 0,
> they'll get a non-0 weight - always.
>
> In my opinion, if we want to make '0' mean 'use system default', then
> it should mean 'ALWAYS use system default for this node'.
>
> "Use the system default at the time the syscall was called, and do not
> update to use a new system default if that default is changed" is
> confusing.
>
> If you say use a global value, use the global value. Simple.
I mainly have concerns about consistency. The global weights can be
changed while the local weights are fixed. For example,
- Weights of node 0,1 is [3, 1] initially
- Process A call set_mempolicy2() to set weights to [4, 0], that is, use
default weight for node 1.
- After hotplug, the weights of node is changed to [12, 4, 1], now the
effective weights used in process A becomes [4, 4]. Which is hardly
desired.
Another choice is to disallow "0" as weight in set_mempolicy2().
--
Best Regards,
Huang, Ying
^ permalink raw reply
* Re: [PATCH v4 11/11] mm/mempolicy: extend set_mempolicy2 and mbind2 to support weighted interleave
From: Dan Carpenter @ 2024-01-03 11:16 UTC (permalink / raw)
To: oe-kbuild, Gregory Price, linux-mm
Cc: lkp, oe-kbuild-all, linux-doc, linux-fsdevel, linux-kernel,
linux-api, x86, akpm, arnd, tglx, luto, mingo, bp, dave.hansen,
hpa, mhocko, tj, ying.huang, gregory.price, corbet, rakie.kim,
hyeongtak.ji, honggyu.kim, vtavarespetr, peterz, jgroves,
ravis.opensrc, sthanneeru, emirakhur, Hasan.Maruf, seungjun.ha
In-Reply-To: <20231218194631.21667-12-gregory.price@memverge.com>
Hi Gregory,
kernel test robot noticed the following build warnings:
https://git-scm.com/docs/git-format-patch#_base_tree_information]
url: https://github.com/intel-lab-lkp/linux/commits/Gregory-Price/mm-mempolicy-implement-the-sysfs-based-weighted_interleave-interface/20231219-074837
base: https://git.kernel.org/pub/scm/linux/kernel/git/perf/perf-tools.git perf-tools
patch link: https://lore.kernel.org/r/20231218194631.21667-12-gregory.price%40memverge.com
patch subject: [PATCH v4 11/11] mm/mempolicy: extend set_mempolicy2 and mbind2 to support weighted interleave
config: x86_64-randconfig-161-20231219 (https://download.01.org/0day-ci/archive/20231220/202312200223.7X9rUFgu-lkp@intel.com/config)
compiler: clang version 16.0.4 (https://github.com/llvm/llvm-project.git ae42196bc493ffe877a7e3dff8be32035dea4d07)
If you fix the issue in a separate patch/commit (i.e. not just a new version of
the same patch/commit), kindly add following tags
| Reported-by: kernel test robot <lkp@intel.com>
| Reported-by: Dan Carpenter <dan.carpenter@linaro.org>
| Closes: https://lore.kernel.org/r/202312200223.7X9rUFgu-lkp@intel.com/
smatch warnings:
mm/mempolicy.c:2044 __do_sys_get_mempolicy2() warn: maybe return -EFAULT instead of the bytes remaining?
mm/mempolicy.c:2044 __do_sys_get_mempolicy2() warn: maybe return -EFAULT instead of the bytes remaining?
vim +2044 mm/mempolicy.c
a2af87404eb73e Gregory Price 2023-12-18 1992 SYSCALL_DEFINE4(get_mempolicy2, struct mpol_args __user *, uargs, size_t, usize,
a2af87404eb73e Gregory Price 2023-12-18 1993 unsigned long, addr, unsigned long, flags)
a2af87404eb73e Gregory Price 2023-12-18 1994 {
a2af87404eb73e Gregory Price 2023-12-18 1995 struct mpol_args kargs;
a2af87404eb73e Gregory Price 2023-12-18 1996 struct mempolicy_args margs;
a2af87404eb73e Gregory Price 2023-12-18 1997 int err;
a2af87404eb73e Gregory Price 2023-12-18 1998 nodemask_t policy_nodemask;
a2af87404eb73e Gregory Price 2023-12-18 1999 unsigned long __user *nodes_ptr;
8bfd7ddc0dd439 Gregory Price 2023-12-18 2000 unsigned char __user *weights_ptr;
8bfd7ddc0dd439 Gregory Price 2023-12-18 2001 unsigned char weights[MAX_NUMNODES];
a2af87404eb73e Gregory Price 2023-12-18 2002
a2af87404eb73e Gregory Price 2023-12-18 2003 if (flags & ~(MPOL_F_ADDR))
a2af87404eb73e Gregory Price 2023-12-18 2004 return -EINVAL;
a2af87404eb73e Gregory Price 2023-12-18 2005
a2af87404eb73e Gregory Price 2023-12-18 2006 /* initialize any memory liable to be copied to userland */
a2af87404eb73e Gregory Price 2023-12-18 2007 memset(&margs, 0, sizeof(margs));
8bfd7ddc0dd439 Gregory Price 2023-12-18 2008 memset(weights, 0, sizeof(weights));
a2af87404eb73e Gregory Price 2023-12-18 2009
a2af87404eb73e Gregory Price 2023-12-18 2010 err = copy_struct_from_user(&kargs, sizeof(kargs), uargs, usize);
a2af87404eb73e Gregory Price 2023-12-18 2011 if (err)
a2af87404eb73e Gregory Price 2023-12-18 2012 return -EINVAL;
a2af87404eb73e Gregory Price 2023-12-18 2013
8bfd7ddc0dd439 Gregory Price 2023-12-18 2014 if (kargs.il_weights)
8bfd7ddc0dd439 Gregory Price 2023-12-18 2015 margs.il_weights = weights;
8bfd7ddc0dd439 Gregory Price 2023-12-18 2016 else
8bfd7ddc0dd439 Gregory Price 2023-12-18 2017 margs.il_weights = NULL;
8bfd7ddc0dd439 Gregory Price 2023-12-18 2018
a2af87404eb73e Gregory Price 2023-12-18 2019 margs.policy_nodes = kargs.pol_nodes ? &policy_nodemask : NULL;
a2af87404eb73e Gregory Price 2023-12-18 2020 if (flags & MPOL_F_ADDR)
a2af87404eb73e Gregory Price 2023-12-18 2021 err = do_get_vma_mempolicy(untagged_addr(addr), NULL, &margs);
a2af87404eb73e Gregory Price 2023-12-18 2022 else
a2af87404eb73e Gregory Price 2023-12-18 2023 err = do_get_task_mempolicy(&margs);
a2af87404eb73e Gregory Price 2023-12-18 2024
a2af87404eb73e Gregory Price 2023-12-18 2025 if (err)
a2af87404eb73e Gregory Price 2023-12-18 2026 return err;
a2af87404eb73e Gregory Price 2023-12-18 2027
a2af87404eb73e Gregory Price 2023-12-18 2028 kargs.mode = margs.mode;
a2af87404eb73e Gregory Price 2023-12-18 2029 kargs.mode_flags = margs.mode_flags;
a2af87404eb73e Gregory Price 2023-12-18 2030 kargs.policy_node = margs.policy_node;
a2af87404eb73e Gregory Price 2023-12-18 2031 kargs.home_node = margs.home_node;
a2af87404eb73e Gregory Price 2023-12-18 2032 if (kargs.pol_nodes) {
a2af87404eb73e Gregory Price 2023-12-18 2033 nodes_ptr = u64_to_user_ptr(kargs.pol_nodes);
a2af87404eb73e Gregory Price 2023-12-18 2034 err = copy_nodes_to_user(nodes_ptr, kargs.pol_maxnodes,
a2af87404eb73e Gregory Price 2023-12-18 2035 margs.policy_nodes);
a2af87404eb73e Gregory Price 2023-12-18 2036 if (err)
a2af87404eb73e Gregory Price 2023-12-18 2037 return err;
This looks wrong as well.
a2af87404eb73e Gregory Price 2023-12-18 2038 }
a2af87404eb73e Gregory Price 2023-12-18 2039
8bfd7ddc0dd439 Gregory Price 2023-12-18 2040 if (kargs.mode == MPOL_WEIGHTED_INTERLEAVE && kargs.il_weights) {
8bfd7ddc0dd439 Gregory Price 2023-12-18 2041 weights_ptr = u64_to_user_ptr(kargs.il_weights);
8bfd7ddc0dd439 Gregory Price 2023-12-18 2042 err = copy_to_user(weights_ptr, weights, kargs.pol_maxnodes);
8bfd7ddc0dd439 Gregory Price 2023-12-18 2043 if (err)
8bfd7ddc0dd439 Gregory Price 2023-12-18 @2044 return err;
This should return -EFAULT same as the copy_to_user() on the next line.
8bfd7ddc0dd439 Gregory Price 2023-12-18 2045 }
8bfd7ddc0dd439 Gregory Price 2023-12-18 2046
a2af87404eb73e Gregory Price 2023-12-18 2047 return copy_to_user(uargs, &kargs, usize) ? -EFAULT : 0;
a2af87404eb73e Gregory Price 2023-12-18 2048 }
--
0-DAY CI Kernel Test Service
https://github.com/intel/lkp-tests/wiki
^ permalink raw reply
* Re: [PATCH v5] posix-timers: add multi_clock_gettime system call
From: Sagi Maimon @ 2024-01-03 12:59 UTC (permalink / raw)
To: Arnd Bergmann
Cc: Richard Cochran, Andy Lutomirski, datglx, Ingo Molnar,
Borislav Petkov, Dave Hansen, x86, H. Peter Anvin,
Geert Uytterhoeven, Peter Zijlstra, Johannes Weiner, Sohil Mehta,
Rick Edgecombe, Nhat Pham, Palmer Dabbelt, Kees Cook,
Alexey Gladkov, Mark Rutland, linux-kernel, linux-api, Linux-Arch,
Netdev
In-Reply-To: <86fcb951-67e0-4f1d-a441-f3b4bcce8210@app.fastmail.com>
On Tue, Jan 2, 2024 at 12:22 PM Arnd Bergmann <arnd@arndb.de> wrote:
>
> On Tue, Jan 2, 2024, at 10:18, Sagi Maimon wrote:
> > Some user space applications need to read some clocks.
> > Each read requires moving from user space to kernel space.
> > The syscall overhead causes unpredictable delay between N clocks reads
> > Removing this delay causes better synchronization between N clocks.
> >
> > Introduce a new system call multi_clock_gettime, which can be used to measure
> > the offset between multiple clocks, from variety of types: PHC, virtual PHC
> > and various system clocks (CLOCK_REALTIME, CLOCK_MONOTONIC, etc).
> > The offset includes the total time that the driver needs to read the clock
> > timestamp.
> >
> > New system call allows the reading of a list of clocks - up to PTP_MAX_CLOCKS.
> > Supported clocks IDs: PHC, virtual PHC and various system clocks.
> > Up to PTP_MAX_SAMPLES times (per clock) in a single system call read.
> > The system call returns n_clocks timestamps for each measurement:
> > - clock 0 timestamp
> > - ...
> > - clock n timestamp
> >
> > Signed-off-by: Sagi Maimon <maimon.sagi@gmail.com>
> > ---
> > Changes since version 4:
> > - fix error : 'struct __ptp_multi_clock_get' declared inside parameter list
> > will not be visible outside of this definition or declaration
>
> I usually put all the changes for previous versions in a
> list here, it helps reviewers.
>
Will be done on patch V6.
> The changes you made for previous versions all look good
> to me, but I think there is still a few things worth
> considering. I'll also follow up on the earlier threads.
>
> > +#define MULTI_PTP_MAX_CLOCKS 32 /* Max number of clocks */
> > +#define MULTI_PTP_MAX_SAMPLES 32 /* Max allowed offset measurement samples. */
> > +
> > +struct __ptp_multi_clock_get {
> > + unsigned int n_clocks; /* Desired number of clocks. */
> > + unsigned int n_samples; /* Desired number of measurements per clock. */
> > + clockid_t clkid_arr[MULTI_PTP_MAX_CLOCKS]; /* list of clock IDs */
> > + /*
> > + * Array of list of n_clocks clocks time samples n_samples times.
> > + */
> > + struct __kernel_timespec ts[MULTI_PTP_MAX_SAMPLES][MULTI_PTP_MAX_CLOCKS];
> > +};
>
> Since you now access each member individually, I think it
> makes more sense here to just pass these as four
> register arguments. It helps with argument introspection,
> avoids a couple of get_user(), and lets you remove the fixed
> array dimensions.
>
I prefer the use of get_user(), I will use it to remove the fixed
array dimensions.
which will be done on patch V6.
> > +SYSCALL_DEFINE1(multi_clock_gettime, struct __ptp_multi_clock_get
> > __user *, ptp_multi_clk_get)
> > +{
> > + const struct k_clock *kc;
> > + struct timespec64 *kernel_tp;
> > + struct timespec64 *kernel_tp_base;
> > + unsigned int n_clocks; /* Desired number of clocks. */
> > + unsigned int n_samples; /* Desired number of measurements per clock.
> > */
> > + unsigned int i, j;
> > + clockid_t clkid_arr[MULTI_PTP_MAX_CLOCKS]; /* list of clock IDs */
> > + int error = 0;
> > +
> > + if (copy_from_user(&n_clocks, &ptp_multi_clk_get->n_clocks,
> > sizeof(n_clocks)))
> > + return -EFAULT;
> > + if (copy_from_user(&n_samples, &ptp_multi_clk_get->n_samples,
> > sizeof(n_samples)))
>
> If these remain as struct members rather than register arguments,
> you should use get_user() instead of copy_from_user().
>
Will be done on patch V6
> > + kernel_tp_base = kmalloc_array(n_clocks * n_samples,
> > + sizeof(struct timespec64), GFP_KERNEL);
> > + if (!kernel_tp_base)
> > + return -ENOMEM;
>
> To be on the safe side regarding possible data leak, maybe use
> kcalloc() instead of kmalloc_array() here.
>
Will be done on patch V6.
> > + kernel_tp = kernel_tp_base;
> > + for (j = 0; j < n_samples; j++) {
> > + for (i = 0; i < n_clocks; i++) {
> > + if (put_timespec64(kernel_tp++, (struct __kernel_timespec __user *)
> > + &ptp_multi_clk_get->ts[j][i])) {
>
> I think the typecast here can be removed.
>
You are right, will be fixed on patch V6.
> Arnd
Thanks for your Notes.
^ permalink raw reply
* Re: [PATCH v5 02/11] mm/mempolicy: introduce MPOL_WEIGHTED_INTERLEAVE for weighted interleaving
From: Gregory Price @ 2024-01-03 22:09 UTC (permalink / raw)
To: Huang, Ying
Cc: Gregory Price, linux-mm, linux-doc, linux-fsdevel, linux-kernel,
linux-api, x86, akpm, arnd, tglx, luto, mingo, bp, dave.hansen,
hpa, mhocko, tj, corbet, rakie.kim, hyeongtak.ji, honggyu.kim,
vtavarespetr, peterz, jgroves, ravis.opensrc, sthanneeru,
emirakhur, Hasan.Maruf, seungjun.ha, Srinivasulu Thanneeru
In-Reply-To: <87mstnc6jz.fsf@yhuang6-desk2.ccr.corp.intel.com>
On Wed, Jan 03, 2024 at 01:46:56PM +0800, Huang, Ying wrote:
> Gregory Price <gregory.price@memverge.com> writes:
> > I'm specifically concerned about:
> > weighted_interleave_nid
> > alloc_pages_bulk_array_weighted_interleave
> >
> > I'm unsure whether kmalloc/kfree is safe (and non-offensive) in those
> > contexts. If kmalloc/kfree is safe fine, this problem is trivial.
> >
> > If not, there is no good solution to this without pre-allocating a
> > scratch area per-task.
>
> You need to audit whether it's safe for all callers. I guess that you
> need to allocate pages after calling, so you can use the same GFP flags
> here.
>
After picking away i realized that this code is usually going to get
called during page fault handling - duh. So kmalloc is almost never
safe (or can fail), and we it's nasty to try to handle those errors.
Instead of doing that, I simply chose to implement the scratch space
in the mempolicy structure
mempolicy->wil.scratch_weights[MAX_NUMNODES].
We eat an extra 1kb of memory in the mempolicy, but it gives us a safe
scratch space we can use any time the task is allocating memory, and
prevents the need for any fancy error handling. That seems like a
perfectly reasonable tradeoff.
> >
> > Weights are collected individually onto the stack because we have to sum
> > them up before we actually apply the weights.
> >
> > A stale weight is not offensive. RCU is not needed and doesn't help.
>
> When you copy weights from iw_table[] to stack, it's possible for
> compiler to cache its contents in register, or merge, split the memory
> operations. At the same time, iw_table[] may be changed simultaneously
> via sysfs interface. So, we need a mechanism to guarantee that we read
> the latest contents consistently.
>
Fair enough, I went ahead and added a similar interaction.
~Gregoryg
^ permalink raw reply
* [PATCH v6 00/12] mempolicy2, mbind2, and weighted interleave
From: Gregory Price @ 2024-01-03 22:41 UTC (permalink / raw)
To: linux-mm
Cc: linux-doc, linux-fsdevel, linux-kernel, linux-api, linux-arch,
akpm, arnd, tglx, luto, mingo, bp, dave.hansen, hpa, mhocko, tj,
ying.huang, gregory.price, corbet, rakie.kim, hyeongtak.ji,
honggyu.kim, vtavarespetr, peterz, jgroves, ravis.opensrc,
sthanneeru, emirakhur, Hasan.Maruf, seungjun.ha, Johannes Weiner,
Hasan Al Maruf, Hao Wang, Dan Williams, Michal Hocko, Zhongkun He,
Frank van der Linden, John Groves, Jonathan Cameron
Weighted interleave is a new interleave policy intended to make
use of heterogeneous memory environments appearing with CXL.
To implement weighted interleave with task-local weights, we
need new syscalls capable of passing a weight array. This is
the justification for mempolicy2/mbind2 - which are designed
to be extensible to capture future policies as well.
The existing interleave mechanism does an even round-robin
distribution of memory across all nodes in a nodemask, while
weighted interleave distributes memory across nodes according
to a provided weight. (Weight = # of page allocations per round)
Weighted interleave is intended to reduce average latency when
bandwidth is pressured - therefore increasing total throughput.
In other words: It allows greater use of the total available
bandwidth in a heterogeneous hardware environment (different
hardware provides different bandwidth capacity).
As bandwidth is pressured, latency increases - first linearly
and then exponentially. By keeping bandwidth usage distributed
according to available bandwidth, we therefore can reduce the
average latency of a cacheline fetch.
A good explanation of the bandwidth vs latency response curve:
https://mahmoudhatem.wordpress.com/2017/11/07/memory-bandwidth-vs-latency-response-curve/
From the article:
```
Constant region:
The latency response is fairly constant for the first 40%
of the sustained bandwidth.
Linear region:
In between 40% to 80% of the sustained bandwidth, the
latency response increases almost linearly with the bandwidth
demand of the system due to contention overhead by numerous
memory requests.
Exponential region:
Between 80% to 100% of the sustained bandwidth, the memory
latency is dominated by the contention latency which can be
as much as twice the idle latency or more.
Maximum sustained bandwidth :
Is 65% to 75% of the theoretical maximum bandwidth.
```
As a general rule of thumb:
* If bandwidth usage is low, latency does not increase. It is
optimal to place data in the nearest (lowest latency) device.
* If bandwidth usage is high, latency increases. It is optimal
to place data such that bandwidth use is optimized per-device.
This is the top line goal: Provide a user a mechanism to target using
the "maximum sustained bandwidth" of each hardware component in a
heterogenous memory system.
For example, the stream benchmark demonstrates that default interleave
is actively harmful, where weighted interleave is beneficial. Default
interleave distributes data such that too much pressure is placed on
devices with lower available bandwidth.
Stream Benchmark (High level results, 1 Socket + 1 CXL Device)
Default interleave : -78% (slower than DRAM)
Global weighting : -6% to +4% (workload dependant)
Targeted weights : +2.5% to +4% (consistently better than DRAM)
Global means the task-policy was set (set_mempolicy2), while targeted
means VMA policies were set (mbind2). We can see weighted interleave
is not always beneficial when applied globally, but is always
beneficial when applied to bandwidth-driving data areas. This is a
good reason to provide both mechanisms (Simplicity vs Control).
We implement sysfs entries for "system global" weights which can be
set by a daemon or administrator, and new extensible syscalls
(mempolicy2, mbind2) for task-local weights to be set by either
numactl or user-software.
We chose to implement an extensible mempolicy interface so that
future extensions can be captured, rather than adding additional
syscalls for every new mempolicy which requires new data.
MPOL_WEIGHTED_INTERLEAVE is included as an example extension.
There are 3 "phases" in the patch set that could be considered
for separate merge candidates, but are presented here as a single
line as the goal is a fully functional MPOL_WEIGHTED_INTERLEAVE.
1) Implement MPOL_WEIGHTED_INTERLEAVE with a sysfs extension for
setting system-global weights via sysfs.
(Patches 1-3)
2) Refactor mempolicy creation mechanism to use an extensible arg
struct `struct mempolicy_param` to promote code re-use between
the original mempolicy/mbind interfaces and the new interfaces.
(Patches 4-7)
3) Implementation of set_mempolicy2, get_mempolicy2, and mbind2,
along with the addition of task-local weights so that per-task
weights can be registered for MPOL_WEIGHTED_INTERLEAVE.
(Patches 8-12)
Included below is LTP test information, performance test information,
and some software / numactl branch which can be used for testing.
= Performance summary =
(tests may have different configurations, see extended info below)
1) MLC (W2) : +38% over DRAM. +264% over default interleave.
MLC (W5) : +40% over DRAM. +226% over default interleave.
2) Stream : -6% to +4% over DRAM, +430% over default interleave.
3) XSBench : +19% over DRAM. +47% over default interleave.
= LTP Testing Summary =
existing mempolicy & mbind tests: pass
mempolicy & mbind + weighted interleave (global weights): pass
mempolicy2 & mbind2 + weighted interleave (global weights): pass
mempolicy2 & mbind2 + weighted interleave (local weights): pass
= v6 notes =
- bug: resolved excessive stack usage w/ scratch area
- bug: bulk allocator uninitialized value (prev_node = NUMA_NO_NODE)
- bug: global weights should be unsigned (char -> u8)
- bug: return value in get_mempolicy (thanks dan.carpenter@linaro.org)
- refactor: refactor read_once operations into functions
- change: reduce mpol_params->pol_maxnodes size from u64 to u16
- change: add weight scratch space in mempolicy used during allocation
- change: kill MPOL_F_GWEIGHT flag
- change: 0-weight now implies "use global/default"
- change: simplify bulk allocator logic
- change: weights are now all u8 for consistency
- change: add default_iw_table (system default separate from sysfs)
- change: _args to _param in struct names
- change: sanitize_flags simplification
- documentation updates
=====================================================================
Performance tests - MLC
From - Ravi Jonnalagadda <ravis.opensrc@micron.com>
Hardware: Single-socket, multiple CXL memory expanders.
Workload: W2
Data Signature: 2:1 read:write
DRAM only bandwidth (GBps): 298.8
DRAM + CXL (default interleave) (GBps): 113.04
DRAM + CXL (weighted interleave)(GBps): 412.5
Gain over DRAM only: 1.38x
Gain over default interleave: 2.64x
Workload: W5
Data Signature: 1:1 read:write
DRAM only bandwidth (GBps): 273.2
DRAM + CXL (default interleave) (GBps): 117.23
DRAM + CXL (weighted interleave)(GBps): 382.7
Gain over DRAM only: 1.4x
Gain over default interleave: 2.26x
=====================================================================
Performance test - Stream
From - Gregory Price <gregory.price@memverge.com>
Hardware: Single socket, single CXL expander
numactl extension: https://github.com/gmprice/numactl/tree/weighted_interleave_master
Summary: 64 threads, ~18GB workload, 3GB per array, executed 100 times
Default interleave : -78% (slower than DRAM)
Global weighting : -6% to +4% (workload dependant)
mbind2 weights : +2.5% to +4% (consistently better than DRAM)
dram only:
numactl --cpunodebind=1 --membind=1 ./stream_c.exe --ntimes 100 --array-size 400M --malloc
Function Direction BestRateMBs AvgTime MinTime MaxTime
Copy: 0->0 200923.2 0.032662 0.031853 0.033301
Scale: 0->0 202123.0 0.032526 0.031664 0.032970
Add: 0->0 208873.2 0.047322 0.045961 0.047884
Triad: 0->0 208523.8 0.047262 0.046038 0.048414
CXL-only:
numactl --cpunodebind=1 -w --membind=2 ./stream_c.exe --ntimes 100 --array-size 400M --malloc
Copy: 0->0 22209.7 0.288661 0.288162 0.289342
Scale: 0->0 22288.2 0.287549 0.287147 0.288291
Add: 0->0 24419.1 0.393372 0.393135 0.393735
Triad: 0->0 24484.6 0.392337 0.392083 0.394331
Based on the above, the optimal weights are ~9:1
echo 9 > /sys/kernel/mm/mempolicy/weighted_interleave/node1
echo 1 > /sys/kernel/mm/mempolicy/weighted_interleave/node2
default interleave:
numactl --cpunodebind=1 --interleave=1,2 ./stream_c.exe --ntimes 100 --array-size 400M --malloc
Copy: 0->0 44666.2 0.143671 0.143285 0.144174
Scale: 0->0 44781.6 0.143256 0.142916 0.143713
Add: 0->0 48600.7 0.197719 0.197528 0.197858
Triad: 0->0 48727.5 0.197204 0.197014 0.197439
global weighted interleave:
numactl --cpunodebind=1 -w --interleave=1,2 ./stream_c.exe --ntimes 100 --array-size 400M --malloc
Copy: 0->0 190085.9 0.034289 0.033669 0.034645
Scale: 0->0 207677.4 0.031909 0.030817 0.033061
Add: 0->0 202036.8 0.048737 0.047516 0.053409
Triad: 0->0 217671.5 0.045819 0.044103 0.046755
targted regions w/ global weights (modified stream to mbind2 malloc'd regions))
numactl --cpunodebind=1 --membind=1 ./stream_c.exe -b --ntimes 100 --array-size 400M --malloc
Copy: 0->0 205827.0 0.031445 0.031094 0.031984
Scale: 0->0 208171.8 0.031320 0.030744 0.032505
Add: 0->0 217352.0 0.045087 0.044168 0.046515
Triad: 0->0 216884.8 0.045062 0.044263 0.046982
=====================================================================
Performance tests - XSBench
From - Hyeongtak Ji <hyeongtak.ji@sk.com>
Hardware: Single socket, Single CXL memory Expander
NUMA node 0: 56 logical cores, 128 GB memory
NUMA node 2: 96 GB CXL memory
Threads: 56
Lookups: 170,000,000
Summary: +19% over DRAM. +47% over default interleave.
Performance tests - XSBench
1. dram only
$ numactl -m 0 ./XSBench -s XL –p 5000000
Runtime: 36.235 seconds
Lookups/s: 4,691,618
2. default interleave
$ numactl –i 0,2 ./XSBench –s XL –p 5000000
Runtime: 55.243 seconds
Lookups/s: 3,077,293
3. weighted interleave
numactl –w –i 0,2 ./XSBench –s XL –p 5000000
Runtime: 29.262 seconds
Lookups/s: 5,809,513
=====================================================================
LTP Tests: https://github.com/gmprice/ltp/tree/mempolicy2
= Existing tests
set_mempolicy, get_mempolicy, mbind
MPOL_WEIGHTED_INTERLEAVE added manually to test basic functionality
but did not adjust tests for weighting. Basically the weights were
set to 1, which is the default, and it should behavior like standard
MPOL_INTERLEAVE if logic is correct.
== set_mempolicy01 : passed 18, failed 0
== set_mempolicy02 : passed 10, failed 0
== set_mempolicy03 : passed 64, failed 0
== set_mempolicy04 : passed 32, failed 0
== set_mempolicy05 - n/a on non-x86
== set_mempolicy06 : passed 10, failed 0
this is set_mempolicy02 + MPOL_WEIGHTED_INTERLEAVE
== set_mempolicy07 : passed 32, failed 0
set_mempolicy04 + MPOL_WEIGHTED_INTERLEAVE
== get_mempolicy01 : passed 12, failed 0
change: added MPOL_WEIGHTED_INTERLEAVE
== get_mempolicy02 : passed 2, failed 0
== mbind01 : passed 15, failed 0
added MPOL_WEIGHTED_INTERLEAVE
== mbind02 : passed 4, failed 0
added MPOL_WEIGHTED_INTERLEAVE
== mbind03 : passed 16, failed 0
added MPOL_WEIGHTED_INTERLEAVE
== mbind04 : passed 48, failed 0
added MPOL_WEIGHTED_INTERLEAVE
= New Tests
set_mempolicy2, get_mempolicy2, mbind2
Took the original set_mempolicy and get_mempolicy tests, and updated
them to utilize the new mempolicy2 interfaces. Added additional tests
for setting task-local weights to validate behavior.
== set_mempolicy201 : passed 18, failed 0
== set_mempolicy202 : passed 10, failed 0
== set_mempolicy203 : passed 64, failed 0
== set_mempolicy204 : passed 32, failed 0
== set_mempolicy205 : passed 10, failed 0
== set_mempolicy206 : passed 32, failed 0
== set_mempolicy207 : passed 6, failed 0
new: MPOL_WEIGHTED_INTERLEAVE with task-local weights
== get_mempolicy201 : passed 12, failed 0
== get_mempolicy202 : passed 2, failed 0
== get_mempolicy203 : passed 6, failed 0
new: fetch global and local weights
== mbind201 : passed 15, failed 0
== mbind202 : passed 4, failed 0
== mbind203 : passed 16, failed 0
== mbind204 : passed 48, failed 0
=====================================================================
Basic set_mempolicy2 test
set_mempolicy2 w/ weighted interleave, task-local weights and uses
pthread_create to demonstrate the mempolicy is overwritten by child.
Manually validating the distribution via numa_maps
007c0000 weighted interleave:0-1 heap anon=65794 dirty=65794 active=0 N0=54829 N1=10965 kernelpagesize_kB=4
7f3f2c000000 weighted interleave:0-1 anon=32768 dirty=32768 active=0 N0=5461 N1=27307 kernelpagesize_kB=4
7f3f34000000 weighted interleave:0-1 anon=16384 dirty=16384 active=0 N0=2731 N1=13653 kernelpagesize_kB=4
7f3f3bffe000 weighted interleave:0-1 anon=65538 dirty=65538 active=0 N0=10924 N1=54614 kernelpagesize_kB=4
7f3f5c000000 weighted interleave:0-1 anon=16384 dirty=16384 active=0 N0=2731 N1=13653 kernelpagesize_kB=4
7f3f60dfe000 weighted interleave:0-1 anon=65537 dirty=65537 active=0 N0=54615 N1=10922 kernelpagesize_kB=4
Expected distribution is 5:1 or 1:5 (less node should be ~16.666%)
1) 10965/65794 : 16.6656...
2) 5461/32768 : 16.6656...
3) 2731/16384 : 16.6687...
4) 10924/65538 : 16.6682...
5) 2731/16384 : 16.6687...
6) 10922/65537 : 16.6653...
#define _GNU_SOURCE
#include <stdio.h>
#include <stdlib.h>
#include <numa.h>
#include <errno.h>
#include <numaif.h>
#include <unistd.h>
#include <pthread.h>
#include <sys/uio.h>
#include <sys/types.h>
#include <stdint.h>
#define MPOL_WEIGHTED_INTERLEAVE 6
#define SET_MEMPOLICY2(a, b) syscall(457, a, b, 0)
#define M256 (1024*1024*256)
#define PAGE_SIZE (4096)
struct mpol_param {
/* Basic mempolicy settings */
uint16_t mode;
uint16_t mode_flags;
int32_t home_node;
uint16_t pol_maxnodes;
uint8_t resv[6];
uint64_t pol_nodes;
uint64_t il_weights;
};
struct mpol_param wil_param;
struct bitmask *wil_nodes;
unsigned char *weights;
int total_nodes = -1;
pthread_t tid;
void set_mempolicy_call(int which)
{
weights = (unsigned char *)calloc(total_nodes, sizeof(unsigned char));
wil_nodes = numa_allocate_nodemask();
numa_bitmask_setbit(wil_nodes, 0); weights[0] = which ? 1 : 5;
numa_bitmask_setbit(wil_nodes, 1); weights[1] = which ? 5 : 1;
memset(&wil_param, 0, sizeof(wil_param));
wil_param.mode = MPOL_WEIGHTED_INTERLEAVE;
wil_param.mode_flags = 0;
wil_param.pol_nodes = wil_nodes->maskp;
wil_param.pol_maxnodes = total_nodes;
wil_param.il_weights = weights;
int ret = SET_MEMPOLICY2(&wil_param, sizeof(wil_param));
fprintf(stderr, "set_mempolicy2 result: %d(%s)\n", ret, strerror(errno));
}
void *func(void *arg)
{
char *mainmem = malloc(M256);
int i;
set_mempolicy_call(1); /* weight 1 heavier */
mainmem = malloc(M256);
memset(mainmem, 1, M256);
for (i = 0; i < (M256/PAGE_SIZE); i++) {
mainmem = malloc(PAGE_SIZE);
mainmem[0] = 1;
}
printf("thread done %d\n", getpid());
getchar();
return arg;
}
int main()
{
char * mainmem;
int i;
total_nodes = numa_max_node() + 1;
set_mempolicy_call(0); /* weight 0 heavier */
pthread_create(&tid, NULL, func, NULL);
mainmem = malloc(M256);
memset(mainmem, 1, M256);
for (i = 0; i < (M256/PAGE_SIZE); i++) {
mainmem = malloc(PAGE_SIZE);
mainmem[0] = 1;
}
printf("main done %d\n", getpid());
getchar();
return 0;
}
=====================================================================
numactl (set_mempolicy) w/ global weighting test
numactl fork: https://github.com/gmprice/numactl/tree/weighted_interleave_master
command: numactl -w --interleave=0,1 ./eatmem
result (weights 1:1):
0176a000 weighted interleave:0-1 heap anon=65793 dirty=65793 active=0 N0=32897 N1=32896 kernelpagesize_kB=4
7fceeb9ff000 weighted interleave:0-1 anon=65537 dirty=65537 active=0 N0=32768 N1=32769 kernelpagesize_kB=4
50% distribution is correct
result (weights 5:1):
01b14000 weighted interleave:0-1 heap anon=65793 dirty=65793 active=0 N0=54828 N1=10965 kernelpagesize_kB=4
7f47a1dff000 weighted interleave:0-1 anon=65537 dirty=65537 active=0 N0=54614 N1=10923 kernelpagesize_kB=4
16.666% distribution is correct
result (weights 1:5):
01f07000 weighted interleave:0-1 heap anon=65793 dirty=65793 active=0 N0=10966 N1=54827 kernelpagesize_kB=4
7f17b1dff000 weighted interleave:0-1 anon=65537 dirty=65537 active=0 N0=10923 N1=54614 kernelpagesize_kB=4
16.666% distribution is correct
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
int main (void)
{
char* mem = malloc(1024*1024*256);
memset(mem, 1, 1024*1024*256);
for (int i = 0; i < ((1024*1024*256)/4096); i++)
{
mem = malloc(4096);
mem[0] = 1;
}
printf("done\n");
getchar();
return 0;
}
=====================================================================
Suggested-by: Gregory Price <gregory.price@memverge.com>
Suggested-by: Johannes Weiner <hannes@cmpxchg.org>
Suggested-by: Hasan Al Maruf <hasanalmaruf@fb.com>
Suggested-by: Hao Wang <haowang3@fb.com>
Suggested-by: Ying Huang <ying.huang@intel.com>
Suggested-by: Dan Williams <dan.j.williams@intel.com>
Suggested-by: Michal Hocko <mhocko@suse.com>
Suggested-by: tj <tj@kernel.org>
Suggested-by: Zhongkun He <hezhongkun.hzk@bytedance.com>
Suggested-by: Frank van der Linden <fvdl@google.com>
Suggested-by: John Groves <john@jagalactic.com>
Suggested-by: Vinicius Tavares Petrucci <vtavarespetr@micron.com>
Suggested-by: Srinivasulu Thanneeru <sthanneeru@micron.com>
Suggested-by: Ravi Jonnalagadda <ravis.opensrc@micron.com>
Suggested-by: Jonathan Cameron <Jonathan.Cameron@Huawei.com>
Suggested-by: Hyeongtak Ji <hyeongtak.ji@sk.com>
Signed-off-by: Gregory Price <gregory.price@memverge.com>
Gregory Price (11):
mm/mempolicy: refactor a read-once mechanism into a function for
re-use
mm/mempolicy: introduce MPOL_WEIGHTED_INTERLEAVE for weighted
interleaving
mm/mempolicy: refactor sanitize_mpol_flags for reuse
mm/mempolicy: create struct mempolicy_param for creating new
mempolicies
mm/mempolicy: refactor kernel_get_mempolicy for code re-use
mm/mempolicy: allow home_node to be set by mpol_new
mm/mempolicy: add userland mempolicy arg structure
mm/mempolicy: add set_mempolicy2 syscall
mm/mempolicy: add get_mempolicy2 syscall
mm/mempolicy: add the mbind2 syscall
mm/mempolicy: extend mempolicy2 and mbind2 to support weighted
interleave
Rakie Kim (1):
mm/mempolicy: implement the sysfs-based weighted_interleave interface
.../ABI/testing/sysfs-kernel-mm-mempolicy | 4 +
...fs-kernel-mm-mempolicy-weighted-interleave | 26 +
.../admin-guide/mm/numa_memory_policy.rst | 67 ++
arch/alpha/kernel/syscalls/syscall.tbl | 3 +
arch/arm/tools/syscall.tbl | 3 +
arch/arm64/include/asm/unistd.h | 2 +-
arch/arm64/include/asm/unistd32.h | 6 +
arch/m68k/kernel/syscalls/syscall.tbl | 3 +
arch/microblaze/kernel/syscalls/syscall.tbl | 3 +
arch/mips/kernel/syscalls/syscall_n32.tbl | 3 +
arch/mips/kernel/syscalls/syscall_o32.tbl | 3 +
arch/parisc/kernel/syscalls/syscall.tbl | 3 +
arch/powerpc/kernel/syscalls/syscall.tbl | 3 +
arch/s390/kernel/syscalls/syscall.tbl | 3 +
arch/sh/kernel/syscalls/syscall.tbl | 3 +
arch/sparc/kernel/syscalls/syscall.tbl | 3 +
arch/x86/entry/syscalls/syscall_32.tbl | 3 +
arch/x86/entry/syscalls/syscall_64.tbl | 3 +
arch/xtensa/kernel/syscalls/syscall.tbl | 3 +
include/linux/mempolicy.h | 19 +
include/linux/syscalls.h | 8 +
include/uapi/asm-generic/unistd.h | 8 +-
include/uapi/linux/mempolicy.h | 16 +-
kernel/sys_ni.c | 3 +
mm/mempolicy.c | 976 +++++++++++++++---
.../arch/mips/entry/syscalls/syscall_n64.tbl | 3 +
.../arch/powerpc/entry/syscalls/syscall.tbl | 3 +
.../perf/arch/s390/entry/syscalls/syscall.tbl | 3 +
.../arch/x86/entry/syscalls/syscall_64.tbl | 3 +
29 files changed, 1062 insertions(+), 127 deletions(-)
create mode 100644 Documentation/ABI/testing/sysfs-kernel-mm-mempolicy
create mode 100644 Documentation/ABI/testing/sysfs-kernel-mm-mempolicy-weighted-interleave
--
2.39.1
^ permalink raw reply
* [PATCH v6 01/12] mm/mempolicy: implement the sysfs-based weighted_interleave interface
From: Gregory Price @ 2024-01-03 22:41 UTC (permalink / raw)
To: linux-mm
Cc: linux-doc, linux-fsdevel, linux-kernel, linux-api, linux-arch,
akpm, arnd, tglx, luto, mingo, bp, dave.hansen, hpa, mhocko, tj,
ying.huang, gregory.price, corbet, rakie.kim, hyeongtak.ji,
honggyu.kim, vtavarespetr, peterz, jgroves, ravis.opensrc,
sthanneeru, emirakhur, Hasan.Maruf, seungjun.ha
In-Reply-To: <20240103224209.2541-1-gregory.price@memverge.com>
From: Rakie Kim <rakie.kim@sk.com>
This patch provides a way to set interleave weight information under
sysfs at /sys/kernel/mm/mempolicy/weighted_interleave/nodeN
The sysfs structure is designed as follows.
$ tree /sys/kernel/mm/mempolicy/
/sys/kernel/mm/mempolicy/ [1]
└── weighted_interleave [2]
├── node0 [3]
└── node1
Each file above can be explained as follows.
[1] mm/mempolicy: configuration interface for mempolicy subsystem
[2] weighted_interleave/: config interface for weighted interleave policy
[3] weighted_interleave/nodeN: weight for nodeN
Internally, there is a secondary table `default_iw_table`, which holds
kernel-internal default interleave weights for each possible node.
If the value for a node is set to `0`, the default value will be used.
If sysfs is disabled in the config, interleave weights will default
to use `default_iw_table`.
Suggested-by: Huang Ying <ying.huang@intel.com>
Signed-off-by: Rakie Kim <rakie.kim@sk.com>
Signed-off-by: Honggyu Kim <honggyu.kim@sk.com>
Co-developed-by: Gregory Price <gregory.price@memverge.com>
Signed-off-by: Gregory Price <gregory.price@memverge.com>
Co-developed-by: Hyeongtak Ji <hyeongtak.ji@sk.com>
Signed-off-by: Hyeongtak Ji <hyeongtak.ji@sk.com>
---
.../ABI/testing/sysfs-kernel-mm-mempolicy | 4 +
...fs-kernel-mm-mempolicy-weighted-interleave | 26 +++
mm/mempolicy.c | 178 ++++++++++++++++++
3 files changed, 208 insertions(+)
create mode 100644 Documentation/ABI/testing/sysfs-kernel-mm-mempolicy
create mode 100644 Documentation/ABI/testing/sysfs-kernel-mm-mempolicy-weighted-interleave
diff --git a/Documentation/ABI/testing/sysfs-kernel-mm-mempolicy b/Documentation/ABI/testing/sysfs-kernel-mm-mempolicy
new file mode 100644
index 000000000000..2dcf24f4384a
--- /dev/null
+++ b/Documentation/ABI/testing/sysfs-kernel-mm-mempolicy
@@ -0,0 +1,4 @@
+What: /sys/kernel/mm/mempolicy/
+Date: December 2023
+Contact: Linux memory management mailing list <linux-mm@kvack.org>
+Description: Interface for Mempolicy
diff --git a/Documentation/ABI/testing/sysfs-kernel-mm-mempolicy-weighted-interleave b/Documentation/ABI/testing/sysfs-kernel-mm-mempolicy-weighted-interleave
new file mode 100644
index 000000000000..e6a38139bf0f
--- /dev/null
+++ b/Documentation/ABI/testing/sysfs-kernel-mm-mempolicy-weighted-interleave
@@ -0,0 +1,26 @@
+What: /sys/kernel/mm/mempolicy/weighted_interleave/
+Date: December 2023
+Contact: Linux memory management mailing list <linux-mm@kvack.org>
+Description: Configuration Interface for the Weighted Interleave policy
+
+What: /sys/kernel/mm/mempolicy/weighted_interleave/nodeN
+Date: December 2023
+Contact: Linux memory management mailing list <linux-mm@kvack.org>
+Description: Weight configuration interface for nodeN
+
+ The interleave weight for a memory node (N). These weights are
+ utilized by processes which have set their mempolicy to
+ MPOL_WEIGHTED_INTERLEAVE and have opted into global weights by
+ omitting a task-local weight array.
+
+ These weights only affect new allocations, and changes at runtime
+ will not cause migrations on already allocated pages.
+
+ The minimum weight for a node is always 1.
+
+ Minimum weight: 1
+ Maximum weight: 255
+
+ Writing an empty string or `0` will reset the weight to the
+ system default. The system default may be set by the kernel
+ or drivers at boot or during hotplug events.
diff --git a/mm/mempolicy.c b/mm/mempolicy.c
index 10a590ee1c89..30da1a1be707 100644
--- a/mm/mempolicy.c
+++ b/mm/mempolicy.c
@@ -131,6 +131,23 @@ static struct mempolicy default_policy = {
static struct mempolicy preferred_node_policy[MAX_NUMNODES];
+/*
+ * default_iw_table is the kernel-internal default value interleave
+ * weight table. It is to be set by driver code capable of reading
+ * HMAT/CDAT information, and to provide mempolicy a sane set of
+ * default weight values for WEIGHTED_INTERLEAVE mode.
+ *
+ * By default, prior to HMAT/CDAT information being consumed, the
+ * default weight of all nodes is 1. The default weight of any
+ * node can only be in the range 1-255. A 0-weight is not allowed.
+ */
+static u8 default_iw_table[MAX_NUMNODES];
+/*
+ * iw_table is the sysfs-set interleave weight table, a value of 0
+ * denotes that the default_iw_table value should be used.
+ */
+static u8 iw_table[MAX_NUMNODES];
+
/**
* numa_nearest_node - Find nearest node by state
* @node: Node id to start the search
@@ -3067,3 +3084,164 @@ void mpol_to_str(char *buffer, int maxlen, struct mempolicy *pol)
p += scnprintf(p, buffer + maxlen - p, ":%*pbl",
nodemask_pr_args(&nodes));
}
+
+#ifdef CONFIG_SYSFS
+struct iw_node_attr {
+ struct kobj_attribute kobj_attr;
+ int nid;
+};
+
+static ssize_t node_show(struct kobject *kobj, struct kobj_attribute *attr,
+ char *buf)
+{
+ struct iw_node_attr *node_attr;
+ u8 weight;
+
+ node_attr = container_of(attr, struct iw_node_attr, kobj_attr);
+ weight = iw_table[node_attr->nid];
+ if (!weight)
+ weight = default_iw_table[node_attr->nid];
+ return sysfs_emit(buf, "%d\n", weight);
+}
+
+static ssize_t node_store(struct kobject *kobj, struct kobj_attribute *attr,
+ const char *buf, size_t count)
+{
+ struct iw_node_attr *node_attr;
+ u8 weight = 0;
+
+ node_attr = container_of(attr, struct iw_node_attr, kobj_attr);
+ /* If no input, revert to default weight */
+ if (count == 0 || sysfs_streq(buf, ""))
+ weight = 0;
+ else if (kstrtou8(buf, 0, &weight))
+ return -EINVAL;
+
+ iw_table[node_attr->nid] = weight;
+ return count;
+}
+
+static struct iw_node_attr *node_attrs[MAX_NUMNODES];
+
+static void sysfs_wi_node_release(struct iw_node_attr *node_attr,
+ struct kobject *parent)
+{
+ if (!node_attr)
+ return;
+ sysfs_remove_file(parent, &node_attr->kobj_attr.attr);
+ kfree(node_attr->kobj_attr.attr.name);
+ kfree(node_attr);
+}
+
+static void sysfs_mempolicy_release(struct kobject *mempolicy_kobj)
+{
+ int i;
+
+ for (i = 0; i < MAX_NUMNODES; i++)
+ sysfs_wi_node_release(node_attrs[i], mempolicy_kobj);
+ kobject_put(mempolicy_kobj);
+}
+
+static const struct kobj_type mempolicy_ktype = {
+ .sysfs_ops = &kobj_sysfs_ops,
+ .release = sysfs_mempolicy_release,
+};
+
+static int add_weight_node(int nid, struct kobject *wi_kobj)
+{
+ struct iw_node_attr *node_attr;
+ char *name;
+
+ node_attr = kzalloc(sizeof(*node_attr), GFP_KERNEL);
+ if (!node_attr)
+ return -ENOMEM;
+
+ name = kasprintf(GFP_KERNEL, "node%d", nid);
+ if (!name) {
+ kfree(node_attr);
+ return -ENOMEM;
+ }
+
+ sysfs_attr_init(&node_attr->kobj_attr.attr);
+ node_attr->kobj_attr.attr.name = name;
+ node_attr->kobj_attr.attr.mode = 0644;
+ node_attr->kobj_attr.show = node_show;
+ node_attr->kobj_attr.store = node_store;
+ node_attr->nid = nid;
+
+ if (sysfs_create_file(wi_kobj, &node_attr->kobj_attr.attr)) {
+ kfree(node_attr->kobj_attr.attr.name);
+ kfree(node_attr);
+ pr_err("failed to add attribute to weighted_interleave\n");
+ return -ENOMEM;
+ }
+
+ node_attrs[nid] = node_attr;
+ return 0;
+}
+
+static int add_weighted_interleave_group(struct kobject *root_kobj)
+{
+ struct kobject *wi_kobj;
+ int nid, err;
+
+ wi_kobj = kzalloc(sizeof(struct kobject), GFP_KERNEL);
+ if (!wi_kobj)
+ return -ENOMEM;
+
+ err = kobject_init_and_add(wi_kobj, &mempolicy_ktype, root_kobj,
+ "weighted_interleave");
+ if (err) {
+ kfree(wi_kobj);
+ return err;
+ }
+
+ memset(node_attrs, 0, sizeof(node_attrs));
+ for_each_node_state(nid, N_POSSIBLE) {
+ err = add_weight_node(nid, wi_kobj);
+ if (err) {
+ pr_err("failed to add sysfs [node%d]\n", nid);
+ break;
+ }
+ }
+ if (err)
+ kobject_put(wi_kobj);
+ return 0;
+}
+
+static int __init mempolicy_sysfs_init(void)
+{
+ int err;
+ struct kobject *root_kobj;
+
+ memset(&default_iw_table, 1, sizeof(default_iw_table));
+ memset(&iw_table, 0, sizeof(iw_table));
+
+ root_kobj = kobject_create_and_add("mempolicy", mm_kobj);
+ if (!root_kobj) {
+ pr_err("failed to add mempolicy kobject to the system\n");
+ return -ENOMEM;
+ }
+
+ err = add_weighted_interleave_group(root_kobj);
+
+ if (err)
+ kobject_put(root_kobj);
+ return err;
+
+}
+#else
+static int __init mempolicy_sysfs_init(void)
+{
+ /*
+ * if sysfs is not enabled MPOL_WEIGHTED_INTERLEAVE defaults to
+ * MPOL_INTERLEAVE behavior, but is still defined separately to
+ * allow task-local weighted interleave and system-defaults to
+ * operate as intended.
+ */
+ memset(&default_iw_table, 1, sizeof(default_iw_table));
+ memset(&iw_table, 0, sizeof(iw_table));
+ return 0;
+}
+#endif /* CONFIG_SYSFS */
+late_initcall(mempolicy_sysfs_init);
--
2.39.1
^ permalink raw reply related
* [PATCH v6 02/12] mm/mempolicy: refactor a read-once mechanism into a function for re-use
From: Gregory Price @ 2024-01-03 22:41 UTC (permalink / raw)
To: linux-mm
Cc: linux-doc, linux-fsdevel, linux-kernel, linux-api, linux-arch,
akpm, arnd, tglx, luto, mingo, bp, dave.hansen, hpa, mhocko, tj,
ying.huang, gregory.price, corbet, rakie.kim, hyeongtak.ji,
honggyu.kim, vtavarespetr, peterz, jgroves, ravis.opensrc,
sthanneeru, emirakhur, Hasan.Maruf, seungjun.ha
In-Reply-To: <20240103224209.2541-1-gregory.price@memverge.com>
move the use of barrier() to force policy->nodemask onto the stack into
a function read_once_policy_nodemask so that it may be re-used.
Suggested-by: Huang Ying <ying.huang@intel.com>
Signed-off-by: Gregory Price <gregory.price@memverge.com>
---
mm/mempolicy.c | 26 ++++++++++++++++----------
1 file changed, 16 insertions(+), 10 deletions(-)
diff --git a/mm/mempolicy.c b/mm/mempolicy.c
index 30da1a1be707..6cdb00acb86b 100644
--- a/mm/mempolicy.c
+++ b/mm/mempolicy.c
@@ -1900,6 +1900,20 @@ unsigned int mempolicy_slab_node(void)
}
}
+static unsigned int read_once_policy_nodemask(struct mempolicy *pol,
+ nodemask_t *mask)
+{
+ /*
+ * barrier stabilizes the nodemask locally so that it can be iterated
+ * over safely without concern for changes. Allocators validate node
+ * selection does not violate mems_allowed, so this is safe.
+ */
+ barrier();
+ __builtin_memcpy(mask, &pol->nodes, sizeof(nodemask_t));
+ barrier();
+ return nodes_weight(*mask);
+}
+
/*
* Do static interleaving for interleave index @ilx. Returns the ilx'th
* node in pol->nodes (starting from ilx=0), wrapping around if ilx
@@ -1907,20 +1921,12 @@ unsigned int mempolicy_slab_node(void)
*/
static unsigned int interleave_nid(struct mempolicy *pol, pgoff_t ilx)
{
- nodemask_t nodemask = pol->nodes;
+ nodemask_t nodemask;
unsigned int target, nnodes;
int i;
int nid;
- /*
- * The barrier will stabilize the nodemask in a register or on
- * the stack so that it will stop changing under the code.
- *
- * Between first_node() and next_node(), pol->nodes could be changed
- * by other threads. So we put pol->nodes in a local stack.
- */
- barrier();
- nnodes = nodes_weight(nodemask);
+ nnodes = read_once_policy_nodemask(pol, &nodemask);
if (!nnodes)
return numa_node_id();
target = ilx % nnodes;
--
2.39.1
^ permalink raw reply related
* [PATCH v6 03/12] mm/mempolicy: introduce MPOL_WEIGHTED_INTERLEAVE for weighted interleaving
From: Gregory Price @ 2024-01-03 22:42 UTC (permalink / raw)
To: linux-mm
Cc: linux-doc, linux-fsdevel, linux-kernel, linux-api, linux-arch,
akpm, arnd, tglx, luto, mingo, bp, dave.hansen, hpa, mhocko, tj,
ying.huang, gregory.price, corbet, rakie.kim, hyeongtak.ji,
honggyu.kim, vtavarespetr, peterz, jgroves, ravis.opensrc,
sthanneeru, emirakhur, Hasan.Maruf, seungjun.ha,
Srinivasulu Thanneeru
In-Reply-To: <20240103224209.2541-1-gregory.price@memverge.com>
When a system has multiple NUMA nodes and it becomes bandwidth hungry,
the current MPOL_INTERLEAVE could be an wise option.
However, if those NUMA nodes consist of different types of memory such
as having local DRAM and CXL memory together, the current round-robin
based interleaving policy doesn't maximize the overall bandwidth because
of their different bandwidth characteristics.
Instead, the interleaving can be more efficient when the allocation
policy follows each NUMA nodes' bandwidth weight rather than having 1:1
round-robin allocation.
This patch introduces a new memory policy, MPOL_WEIGHTED_INTERLEAVE, which
enables weighted interleaving between NUMA nodes. Weighted interleave
allows for a proportional distribution of memory across multiple numa
nodes, preferablly apportioned to match the bandwidth capacity of each
node from the perspective of the accessing node.
For example, if a system has 1 CPU node (0), and 2 memory nodes (0,1),
with a relative bandwidth of (100GB/s, 50GB/s) respectively, the
appropriate weight distribution is (2:1).
Weights will be acquired from the global weight matrix exposed by the
sysfs extension: /sys/kernel/mm/mempolicy/weighted_interleave/ or
from the `default_iw_table`, which will be extended to allow defaults
to be registered by core/cxl drivers in the future.
The policy will then allocate the number of pages according to the
set weights. For example, if the weights are (2,1), then 2 pages
will be allocated on node0 for every 1 page allocated on node1.
The new flag MPOL_WEIGHTED_INTERLEAVE can be used in set_mempolicy(2)
and mbind(2).
There are 3 integration points:
weighted_interleave_nodes:
Counts the number of allocations as they occur, and applies the
weight for the current node. When the weight reaches 0, switch
to the next node. Applied by `mempolicy_slab_node()` and
`policy_nodemask()`
weighted_interleave_nid:
Gets the total weight of the nodemask as well as each individual
node weight, then calculates the node based on the given index.
Applied by `policy_nodemask()` and `mpol_misplaced()`
bulk_array_weighted_interleave:
Gets the total weight of the nodemask as well as each individual
node weight, then calculates the number of "interleave rounds" as
well as any delta ("partial round"). Calculates the number of
pages for each node and allocates them.
If a node was scheduled for interleave via interleave_nodes, the
current weight (pol->cur_weight) will be allocated first, before
the remaining bulk calculation is done. This simplifies the
calculation at the cost of an additional allocation call.
One piece of complexity is the interaction between a recent refactor
which split the logic to acquire the "ilx" (interleave index) of an
allocation and the actually application of the interleave. The
calculation of the `interleave index` is done by `get_vma_policy()`,
while the actual selection of the node will be later appliex by the
relevant weighted_interleave function.
Suggested-by: Hasan Al Maruf <Hasan.Maruf@amd.com>
Signed-off-by: Gregory Price <gregory.price@memverge.com>
Co-developed-by: Rakie Kim <rakie.kim@sk.com>
Signed-off-by: Rakie Kim <rakie.kim@sk.com>
Co-developed-by: Honggyu Kim <honggyu.kim@sk.com>
Signed-off-by: Honggyu Kim <honggyu.kim@sk.com>
Co-developed-by: Hyeongtak Ji <hyeongtak.ji@sk.com>
Signed-off-by: Hyeongtak Ji <hyeongtak.ji@sk.com>
Co-developed-by: Srinivasulu Thanneeru <sthanneeru.opensrc@micron.com>
Signed-off-by: Srinivasulu Thanneeru <sthanneeru.opensrc@micron.com>
Co-developed-by: Ravi Jonnalagadda <ravis.opensrc@micron.com>
Signed-off-by: Ravi Jonnalagadda <ravis.opensrc@micron.com>
---
.../admin-guide/mm/numa_memory_policy.rst | 9 +
include/linux/mempolicy.h | 6 +
include/uapi/linux/mempolicy.h | 1 +
mm/mempolicy.c | 198 +++++++++++++++++-
4 files changed, 211 insertions(+), 3 deletions(-)
diff --git a/Documentation/admin-guide/mm/numa_memory_policy.rst b/Documentation/admin-guide/mm/numa_memory_policy.rst
index eca38fa81e0f..a70f20ce1ffb 100644
--- a/Documentation/admin-guide/mm/numa_memory_policy.rst
+++ b/Documentation/admin-guide/mm/numa_memory_policy.rst
@@ -250,6 +250,15 @@ MPOL_PREFERRED_MANY
can fall back to all existing numa nodes. This is effectively
MPOL_PREFERRED allowed for a mask rather than a single node.
+MPOL_WEIGHTED_INTERLEAVE
+ This mode operates the same as MPOL_INTERLEAVE, except that
+ interleaving behavior is executed based on weights set in
+ /sys/kernel/mm/mempolicy/weighted_interleave/
+
+ Weighted interleave allocates pages on nodes according to a
+ weight. For example if nodes [0,1] are weighted [5,2], 5 pages
+ will be allocated on node0 for every 2 pages allocated on node1.
+
NUMA memory policy supports the following optional mode flags:
MPOL_F_STATIC_NODES
diff --git a/include/linux/mempolicy.h b/include/linux/mempolicy.h
index 931b118336f4..fae903b1d3de 100644
--- a/include/linux/mempolicy.h
+++ b/include/linux/mempolicy.h
@@ -54,6 +54,12 @@ struct mempolicy {
nodemask_t cpuset_mems_allowed; /* relative to these nodes */
nodemask_t user_nodemask; /* nodemask passed by user */
} w;
+
+ /* Weighted interleave settings */
+ struct {
+ u8 cur_weight;
+ u8 scratch_weights[MAX_NUMNODES]; /* Used to avoid allocations */
+ } wil;
};
/*
diff --git a/include/uapi/linux/mempolicy.h b/include/uapi/linux/mempolicy.h
index a8963f7ef4c2..1f9bb10d1a47 100644
--- a/include/uapi/linux/mempolicy.h
+++ b/include/uapi/linux/mempolicy.h
@@ -23,6 +23,7 @@ enum {
MPOL_INTERLEAVE,
MPOL_LOCAL,
MPOL_PREFERRED_MANY,
+ MPOL_WEIGHTED_INTERLEAVE,
MPOL_MAX, /* always last member of enum */
};
diff --git a/mm/mempolicy.c b/mm/mempolicy.c
index 6cdb00acb86b..46e6b6f36a10 100644
--- a/mm/mempolicy.c
+++ b/mm/mempolicy.c
@@ -320,6 +320,7 @@ static struct mempolicy *mpol_new(unsigned short mode, unsigned short flags,
policy->mode = mode;
policy->flags = flags;
policy->home_node = NUMA_NO_NODE;
+ policy->wil.cur_weight = 0;
return policy;
}
@@ -432,6 +433,10 @@ static const struct mempolicy_operations mpol_ops[MPOL_MAX] = {
.create = mpol_new_nodemask,
.rebind = mpol_rebind_preferred,
},
+ [MPOL_WEIGHTED_INTERLEAVE] = {
+ .create = mpol_new_nodemask,
+ .rebind = mpol_rebind_nodemask,
+ },
};
static bool migrate_folio_add(struct folio *folio, struct list_head *foliolist,
@@ -853,7 +858,8 @@ static long do_set_mempolicy(unsigned short mode, unsigned short flags,
old = current->mempolicy;
current->mempolicy = new;
- if (new && new->mode == MPOL_INTERLEAVE)
+ if (new && (new->mode == MPOL_INTERLEAVE ||
+ new->mode == MPOL_WEIGHTED_INTERLEAVE))
current->il_prev = MAX_NUMNODES-1;
task_unlock(current);
mpol_put(old);
@@ -879,6 +885,7 @@ static void get_policy_nodemask(struct mempolicy *pol, nodemask_t *nodes)
case MPOL_INTERLEAVE:
case MPOL_PREFERRED:
case MPOL_PREFERRED_MANY:
+ case MPOL_WEIGHTED_INTERLEAVE:
*nodes = pol->nodes;
break;
case MPOL_LOCAL:
@@ -963,6 +970,13 @@ static long do_get_mempolicy(int *policy, nodemask_t *nmask,
} else if (pol == current->mempolicy &&
pol->mode == MPOL_INTERLEAVE) {
*policy = next_node_in(current->il_prev, pol->nodes);
+ } else if (pol == current->mempolicy &&
+ (pol->mode == MPOL_WEIGHTED_INTERLEAVE)) {
+ if (pol->wil.cur_weight)
+ *policy = current->il_prev;
+ else
+ *policy = next_node_in(current->il_prev,
+ pol->nodes);
} else {
err = -EINVAL;
goto out;
@@ -1792,7 +1806,8 @@ struct mempolicy *get_vma_policy(struct vm_area_struct *vma,
pol = __get_vma_policy(vma, addr, ilx);
if (!pol)
pol = get_task_policy(current);
- if (pol->mode == MPOL_INTERLEAVE) {
+ if (pol->mode == MPOL_INTERLEAVE ||
+ pol->mode == MPOL_WEIGHTED_INTERLEAVE) {
*ilx += vma->vm_pgoff >> order;
*ilx += (addr - vma->vm_start) >> (PAGE_SHIFT + order);
}
@@ -1842,6 +1857,29 @@ bool apply_policy_zone(struct mempolicy *policy, enum zone_type zone)
return zone >= dynamic_policy_zone;
}
+static unsigned int weighted_interleave_nodes(struct mempolicy *policy)
+{
+ unsigned int next;
+ struct task_struct *me = current;
+
+ next = next_node_in(me->il_prev, policy->nodes);
+ if (next == MAX_NUMNODES)
+ return next;
+
+ if (!policy->wil.cur_weight) {
+ u8 next_weight = iw_table[next];
+
+ if (!next_weight)
+ next_weight = default_iw_table[next];
+ policy->wil.cur_weight = next_weight;
+ }
+
+ policy->wil.cur_weight--;
+ if (!policy->wil.cur_weight)
+ me->il_prev = next;
+ return next;
+}
+
/* Do dynamic interleaving for a process */
static unsigned int interleave_nodes(struct mempolicy *policy)
{
@@ -1876,6 +1914,9 @@ unsigned int mempolicy_slab_node(void)
case MPOL_INTERLEAVE:
return interleave_nodes(policy);
+ case MPOL_WEIGHTED_INTERLEAVE:
+ return weighted_interleave_nodes(policy);
+
case MPOL_BIND:
case MPOL_PREFERRED_MANY:
{
@@ -1914,6 +1955,52 @@ static unsigned int read_once_policy_nodemask(struct mempolicy *pol,
return nodes_weight(*mask);
}
+/* places active weights in pol->wil.scratch_weights and return total */
+static unsigned int read_once_interleave_weights(struct mempolicy *pol,
+ nodemask_t *mask)
+{
+ int nid;
+ unsigned int weight_total = 0;
+
+ /* Similar issue to read_once_policy_nodemask */
+ barrier();
+ for_each_node_mask(nid, *mask) {
+ u8 weight = iw_table[nid];
+
+ if (!weight)
+ weight = default_iw_table[nid];
+ weight_total += weight;
+ pol->wil.scratch_weights[nid] = weight;
+ }
+ barrier();
+ return weight_total;
+}
+
+static unsigned int weighted_interleave_nid(struct mempolicy *pol, pgoff_t ilx)
+{
+ nodemask_t nodemask;
+ unsigned int target, weight_total;
+ int nid;
+ u8 weight;
+
+ read_once_policy_nodemask(pol, &nodemask);
+ weight_total = read_once_interleave_weights(pol, &nodemask);
+ if (!weight_total)
+ return numa_node_id();
+
+ /* Finally, calculate the node offset based on totals */
+ target = ilx % weight_total;
+ nid = first_node(nodemask);
+ while (target) {
+ weight = pol->wil.scratch_weights[nid];
+ if (target < weight)
+ break;
+ target -= weight;
+ nid = next_node_in(nid, nodemask);
+ }
+ return nid;
+}
+
/*
* Do static interleaving for interleave index @ilx. Returns the ilx'th
* node in pol->nodes (starting from ilx=0), wrapping around if ilx
@@ -1974,6 +2061,11 @@ static nodemask_t *policy_nodemask(gfp_t gfp, struct mempolicy *pol,
*nid = (ilx == NO_INTERLEAVE_INDEX) ?
interleave_nodes(pol) : interleave_nid(pol, ilx);
break;
+ case MPOL_WEIGHTED_INTERLEAVE:
+ *nid = (ilx == NO_INTERLEAVE_INDEX) ?
+ weighted_interleave_nodes(pol) :
+ weighted_interleave_nid(pol, ilx);
+ break;
}
return nodemask;
@@ -2035,6 +2127,7 @@ bool init_nodemask_of_mempolicy(nodemask_t *mask)
case MPOL_PREFERRED_MANY:
case MPOL_BIND:
case MPOL_INTERLEAVE:
+ case MPOL_WEIGHTED_INTERLEAVE:
*mask = mempolicy->nodes;
break;
@@ -2134,7 +2227,8 @@ struct page *alloc_pages_mpol(gfp_t gfp, unsigned int order,
* If the policy is interleave or does not allow the current
* node in its nodemask, we allocate the standard way.
*/
- if (pol->mode != MPOL_INTERLEAVE &&
+ if ((pol->mode != MPOL_INTERLEAVE &&
+ pol->mode != MPOL_WEIGHTED_INTERLEAVE) &&
(!nodemask || node_isset(nid, *nodemask))) {
/*
* First, try to allocate THP only on local node, but
@@ -2270,6 +2364,91 @@ static unsigned long alloc_pages_bulk_array_interleave(gfp_t gfp,
return total_allocated;
}
+static unsigned long alloc_pages_bulk_array_weighted_interleave(gfp_t gfp,
+ struct mempolicy *pol, unsigned long nr_pages,
+ struct page **page_array)
+{
+ struct task_struct *me = current;
+ unsigned long total_allocated = 0;
+ unsigned long nr_allocated;
+ unsigned long rounds;
+ unsigned long node_pages, delta;
+ u8 weight;
+ unsigned int weight_total = 0;
+ unsigned long rem_pages = nr_pages;
+ nodemask_t nodes;
+ int nnodes, node;
+ int prev_node = NUMA_NO_NODE;
+ int i;
+
+ nnodes = read_once_policy_nodemask(pol, &nodes);
+ weight_total = read_once_interleave_weights(pol, &nodes);
+
+ /* Continue allocating from most recent node and adjust the nr_pages */
+ if (pol->wil.cur_weight) {
+ node = next_node_in(me->il_prev, nodes);
+ node_pages = pol->wil.cur_weight;
+ if (node_pages > rem_pages)
+ node_pages = rem_pages;
+ nr_allocated = __alloc_pages_bulk(gfp, node, NULL, node_pages,
+ NULL, page_array);
+ page_array += nr_allocated;
+ total_allocated += nr_allocated;
+ /* if that's all the pages, no need to interleave */
+ if (rem_pages <= pol->wil.cur_weight) {
+ pol->wil.cur_weight -= rem_pages;
+ return total_allocated;
+ }
+ /* Otherwise we adjust nr_pages down, and continue from there */
+ rem_pages -= pol->wil.cur_weight;
+ pol->wil.cur_weight = 0;
+ prev_node = node;
+ }
+
+ /* Now we can continue allocating as if from 0 instead of an offset */
+ rounds = rem_pages / weight_total;
+ delta = rem_pages % weight_total;
+ for (i = 0; i < nnodes; i++) {
+ node = next_node_in(prev_node, nodes);
+ weight = pol->wil.scratch_weights[node];
+ node_pages = weight * rounds;
+ if (delta) {
+ if (delta > weight) {
+ node_pages += weight;
+ delta -= weight;
+ } else {
+ node_pages += delta;
+ delta = 0;
+ }
+ }
+ nr_allocated = __alloc_pages_bulk(gfp, node, NULL, node_pages,
+ NULL, page_array);
+ page_array += nr_allocated;
+ total_allocated += nr_allocated;
+ if (total_allocated == nr_pages)
+ break;
+ prev_node = node;
+ }
+
+ /*
+ * Finally, we need to update me->il_prev and pol->wil.cur_weight
+ * if there were overflow pages, but not equivalent to the node
+ * weight, set the cur_weight to node_weight - delta and the
+ * me->il_prev to the previous node. Otherwise if it was perfect
+ * we can simply set il_prev to node and cur_weight to 0
+ */
+ if (node_pages) {
+ me->il_prev = prev_node;
+ node_pages %= weight;
+ pol->wil.cur_weight = weight - node_pages;
+ } else {
+ me->il_prev = node;
+ pol->wil.cur_weight = 0;
+ }
+
+ return total_allocated;
+}
+
static unsigned long alloc_pages_bulk_array_preferred_many(gfp_t gfp, int nid,
struct mempolicy *pol, unsigned long nr_pages,
struct page **page_array)
@@ -2310,6 +2489,11 @@ unsigned long alloc_pages_bulk_array_mempolicy(gfp_t gfp,
return alloc_pages_bulk_array_interleave(gfp, pol,
nr_pages, page_array);
+ if (pol->mode == MPOL_WEIGHTED_INTERLEAVE)
+ return alloc_pages_bulk_array_weighted_interleave(gfp, pol,
+ nr_pages,
+ page_array);
+
if (pol->mode == MPOL_PREFERRED_MANY)
return alloc_pages_bulk_array_preferred_many(gfp,
numa_node_id(), pol, nr_pages, page_array);
@@ -2385,6 +2569,7 @@ bool __mpol_equal(struct mempolicy *a, struct mempolicy *b)
case MPOL_INTERLEAVE:
case MPOL_PREFERRED:
case MPOL_PREFERRED_MANY:
+ case MPOL_WEIGHTED_INTERLEAVE:
return !!nodes_equal(a->nodes, b->nodes);
case MPOL_LOCAL:
return true;
@@ -2521,6 +2706,10 @@ int mpol_misplaced(struct folio *folio, struct vm_area_struct *vma,
polnid = interleave_nid(pol, ilx);
break;
+ case MPOL_WEIGHTED_INTERLEAVE:
+ polnid = weighted_interleave_nid(pol, ilx);
+ break;
+
case MPOL_PREFERRED:
if (node_isset(curnid, pol->nodes))
goto out;
@@ -2895,6 +3084,7 @@ static const char * const policy_modes[] =
[MPOL_PREFERRED] = "prefer",
[MPOL_BIND] = "bind",
[MPOL_INTERLEAVE] = "interleave",
+ [MPOL_WEIGHTED_INTERLEAVE] = "weighted interleave",
[MPOL_LOCAL] = "local",
[MPOL_PREFERRED_MANY] = "prefer (many)",
};
@@ -2954,6 +3144,7 @@ int mpol_parse_str(char *str, struct mempolicy **mpol)
}
break;
case MPOL_INTERLEAVE:
+ case MPOL_WEIGHTED_INTERLEAVE:
/*
* Default to online nodes with memory if no nodelist
*/
@@ -3064,6 +3255,7 @@ void mpol_to_str(char *buffer, int maxlen, struct mempolicy *pol)
case MPOL_PREFERRED_MANY:
case MPOL_BIND:
case MPOL_INTERLEAVE:
+ case MPOL_WEIGHTED_INTERLEAVE:
nodes = pol->nodes;
break;
default:
--
2.39.1
^ permalink raw reply related
* [PATCH v6 04/12] mm/mempolicy: refactor sanitize_mpol_flags for reuse
From: Gregory Price @ 2024-01-03 22:42 UTC (permalink / raw)
To: linux-mm
Cc: linux-doc, linux-fsdevel, linux-kernel, linux-api, linux-arch,
akpm, arnd, tglx, luto, mingo, bp, dave.hansen, hpa, mhocko, tj,
ying.huang, gregory.price, corbet, rakie.kim, hyeongtak.ji,
honggyu.kim, vtavarespetr, peterz, jgroves, ravis.opensrc,
sthanneeru, emirakhur, Hasan.Maruf, seungjun.ha
In-Reply-To: <20240103224209.2541-1-gregory.price@memverge.com>
split sanitize_mpol_flags into sanitize and validate.
Sanitize is used by set_mempolicy to split (int mode) into mode
and mode_flags, and then validates them.
Validate validates already split flags.
Validate will be reused for new syscalls that accept already
split mode and mode_flags.
Signed-off-by: Gregory Price <gregory.price@memverge.com>
---
mm/mempolicy.c | 25 ++++++++++++++++++-------
1 file changed, 18 insertions(+), 7 deletions(-)
diff --git a/mm/mempolicy.c b/mm/mempolicy.c
index 46e6b6f36a10..6e2ea94c0f31 100644
--- a/mm/mempolicy.c
+++ b/mm/mempolicy.c
@@ -1478,24 +1478,35 @@ static int copy_nodes_to_user(unsigned long __user *mask, unsigned long maxnode,
return copy_to_user(mask, nodes_addr(*nodes), copy) ? -EFAULT : 0;
}
-/* Basic parameter sanity check used by both mbind() and set_mempolicy() */
-static inline int sanitize_mpol_flags(int *mode, unsigned short *flags)
+/*
+ * Basic parameter sanity check used by mbind/set_mempolicy
+ * May modify flags to include internal flags (e.g. MPOL_F_MOF/F_MORON)
+ */
+static inline int validate_mpol_flags(unsigned short mode, unsigned short *flags)
{
- *flags = *mode & MPOL_MODE_FLAGS;
- *mode &= ~MPOL_MODE_FLAGS;
-
- if ((unsigned int)(*mode) >= MPOL_MAX)
+ if ((unsigned int)(mode) >= MPOL_MAX)
return -EINVAL;
if ((*flags & MPOL_F_STATIC_NODES) && (*flags & MPOL_F_RELATIVE_NODES))
return -EINVAL;
if (*flags & MPOL_F_NUMA_BALANCING) {
- if (*mode != MPOL_BIND)
+ if (mode != MPOL_BIND)
return -EINVAL;
*flags |= (MPOL_F_MOF | MPOL_F_MORON);
}
return 0;
}
+/*
+ * Used by mbind/set_memplicy to split and validate mode/flags
+ */
+static inline int sanitize_mpol_flags(int *mode, unsigned short *flags)
+{
+ *flags = *mode & MPOL_MODE_FLAGS;
+ *mode = *mode & ~MPOL_MODE_FLAGS;
+
+ return validate_mpol_flags(*mode, flags);
+}
+
static long kernel_mbind(unsigned long start, unsigned long len,
unsigned long mode, const unsigned long __user *nmask,
unsigned long maxnode, unsigned int flags)
--
2.39.1
^ permalink raw reply related
* [PATCH v6 05/12] mm/mempolicy: create struct mempolicy_param for creating new mempolicies
From: Gregory Price @ 2024-01-03 22:42 UTC (permalink / raw)
To: linux-mm
Cc: linux-doc, linux-fsdevel, linux-kernel, linux-api, linux-arch,
akpm, arnd, tglx, luto, mingo, bp, dave.hansen, hpa, mhocko, tj,
ying.huang, gregory.price, corbet, rakie.kim, hyeongtak.ji,
honggyu.kim, vtavarespetr, peterz, jgroves, ravis.opensrc,
sthanneeru, emirakhur, Hasan.Maruf, seungjun.ha
In-Reply-To: <20240103224209.2541-1-gregory.price@memverge.com>
This patch adds a new kernel structure `struct mempolicy_param`,
intended to be used for an extensible get/set_mempolicy interface.
This implements the fields required to support the existing syscall
interfaces interfaces, but does not expose any user-facing arg
structure.
mpol_new is refactored to take the argument structure so that future
mempolicy extensions can all be managed in the mempolicy constructor.
The get_mempolicy and mbind syscalls are refactored to utilize the
new argument structure, as are all the callers of mpol_new() and
do_set_mempolicy.
Signed-off-by: Gregory Price <gregory.price@memverge.com>
---
include/linux/mempolicy.h | 11 +++++++
mm/mempolicy.c | 69 +++++++++++++++++++++++++++++----------
2 files changed, 62 insertions(+), 18 deletions(-)
diff --git a/include/linux/mempolicy.h b/include/linux/mempolicy.h
index fae903b1d3de..e6795e2d0cc2 100644
--- a/include/linux/mempolicy.h
+++ b/include/linux/mempolicy.h
@@ -62,6 +62,17 @@ struct mempolicy {
} wil;
};
+/*
+ * Describes settings of a mempolicy during set/get syscalls and
+ * kernel internal calls to do_set_mempolicy()
+ */
+struct mempolicy_param {
+ unsigned short mode; /* policy mode */
+ unsigned short mode_flags; /* policy mode flags */
+ int home_node; /* mbind: use MPOL_MF_HOME_NODE */
+ nodemask_t *policy_nodes; /* get/set/mbind */
+};
+
/*
* Support for managing mempolicy data objects (clone, copy, destroy)
* The default fast path of a NULL MPOL_DEFAULT policy is always inlined.
diff --git a/mm/mempolicy.c b/mm/mempolicy.c
index 6e2ea94c0f31..1f6f19b5d157 100644
--- a/mm/mempolicy.c
+++ b/mm/mempolicy.c
@@ -280,10 +280,12 @@ static int mpol_set_nodemask(struct mempolicy *pol,
* This function just creates a new policy, does some check and simple
* initialization. You must invoke mpol_set_nodemask() to set nodes.
*/
-static struct mempolicy *mpol_new(unsigned short mode, unsigned short flags,
- nodemask_t *nodes)
+static struct mempolicy *mpol_new(struct mempolicy_param *param)
{
struct mempolicy *policy;
+ unsigned short mode = param->mode;
+ unsigned short flags = param->mode_flags;
+ nodemask_t *nodes = param->policy_nodes;
if (mode == MPOL_DEFAULT) {
if (nodes && !nodes_empty(*nodes))
@@ -832,8 +834,7 @@ static int mbind_range(struct vma_iterator *vmi, struct vm_area_struct *vma,
}
/* Set the process memory policy */
-static long do_set_mempolicy(unsigned short mode, unsigned short flags,
- nodemask_t *nodes)
+static long do_set_mempolicy(struct mempolicy_param *param)
{
struct mempolicy *new, *old;
NODEMASK_SCRATCH(scratch);
@@ -842,14 +843,14 @@ static long do_set_mempolicy(unsigned short mode, unsigned short flags,
if (!scratch)
return -ENOMEM;
- new = mpol_new(mode, flags, nodes);
+ new = mpol_new(param);
if (IS_ERR(new)) {
ret = PTR_ERR(new);
goto out;
}
task_lock(current);
- ret = mpol_set_nodemask(new, nodes, scratch);
+ ret = mpol_set_nodemask(new, param->policy_nodes, scratch);
if (ret) {
task_unlock(current);
mpol_put(new);
@@ -1247,8 +1248,7 @@ static struct folio *alloc_migration_target_by_mpol(struct folio *src,
#endif
static long do_mbind(unsigned long start, unsigned long len,
- unsigned short mode, unsigned short mode_flags,
- nodemask_t *nmask, unsigned long flags)
+ struct mempolicy_param *mparam, unsigned long flags)
{
struct mm_struct *mm = current->mm;
struct vm_area_struct *vma, *prev;
@@ -1268,7 +1268,7 @@ static long do_mbind(unsigned long start, unsigned long len,
if (start & ~PAGE_MASK)
return -EINVAL;
- if (mode == MPOL_DEFAULT)
+ if (mparam->mode == MPOL_DEFAULT)
flags &= ~MPOL_MF_STRICT;
len = PAGE_ALIGN(len);
@@ -1279,7 +1279,7 @@ static long do_mbind(unsigned long start, unsigned long len,
if (end == start)
return 0;
- new = mpol_new(mode, mode_flags, nmask);
+ new = mpol_new(mparam);
if (IS_ERR(new))
return PTR_ERR(new);
@@ -1296,7 +1296,8 @@ static long do_mbind(unsigned long start, unsigned long len,
NODEMASK_SCRATCH(scratch);
if (scratch) {
mmap_write_lock(mm);
- err = mpol_set_nodemask(new, nmask, scratch);
+ err = mpol_set_nodemask(new, mparam->policy_nodes,
+ scratch);
if (err)
mmap_write_unlock(mm);
} else
@@ -1310,7 +1311,7 @@ static long do_mbind(unsigned long start, unsigned long len,
* Lock the VMAs before scanning for pages to migrate,
* to ensure we don't miss a concurrently inserted page.
*/
- nr_failed = queue_pages_range(mm, start, end, nmask,
+ nr_failed = queue_pages_range(mm, start, end, mparam->policy_nodes,
flags | MPOL_MF_INVERT | MPOL_MF_WRLOCK, &pagelist);
if (nr_failed < 0) {
@@ -1511,6 +1512,7 @@ static long kernel_mbind(unsigned long start, unsigned long len,
unsigned long mode, const unsigned long __user *nmask,
unsigned long maxnode, unsigned int flags)
{
+ struct mempolicy_param mparam;
unsigned short mode_flags;
nodemask_t nodes;
int lmode = mode;
@@ -1525,7 +1527,12 @@ static long kernel_mbind(unsigned long start, unsigned long len,
if (err)
return err;
- return do_mbind(start, len, lmode, mode_flags, &nodes, flags);
+ memset(&mparam, 0, sizeof(mparam));
+ mparam.mode = lmode;
+ mparam.mode_flags = mode_flags;
+ mparam.policy_nodes = &nodes;
+
+ return do_mbind(start, len, &mparam, flags);
}
SYSCALL_DEFINE4(set_mempolicy_home_node, unsigned long, start, unsigned long, len,
@@ -1606,6 +1613,7 @@ SYSCALL_DEFINE6(mbind, unsigned long, start, unsigned long, len,
static long kernel_set_mempolicy(int mode, const unsigned long __user *nmask,
unsigned long maxnode)
{
+ struct mempolicy_param param;
unsigned short mode_flags;
nodemask_t nodes;
int lmode = mode;
@@ -1619,7 +1627,12 @@ static long kernel_set_mempolicy(int mode, const unsigned long __user *nmask,
if (err)
return err;
- return do_set_mempolicy(lmode, mode_flags, &nodes);
+ memset(¶m, 0, sizeof(param));
+ param.mode = lmode;
+ param.mode_flags = mode_flags;
+ param.policy_nodes = &nodes;
+
+ return do_set_mempolicy(¶m);
}
SYSCALL_DEFINE3(set_mempolicy, int, mode, const unsigned long __user *, nmask,
@@ -2908,6 +2921,7 @@ static int shared_policy_replace(struct shared_policy *sp, pgoff_t start,
void mpol_shared_policy_init(struct shared_policy *sp, struct mempolicy *mpol)
{
int ret;
+ struct mempolicy_param mparam;
sp->root = RB_ROOT; /* empty tree == default mempolicy */
rwlock_init(&sp->lock);
@@ -2920,8 +2934,12 @@ void mpol_shared_policy_init(struct shared_policy *sp, struct mempolicy *mpol)
if (!scratch)
goto put_mpol;
+ memset(&mparam, 0, sizeof(mparam));
+ mparam.mode = mpol->mode;
+ mparam.mode_flags = mpol->flags;
+ mparam.policy_nodes = &mpol->w.user_nodemask;
/* contextualize the tmpfs mount point mempolicy to this file */
- npol = mpol_new(mpol->mode, mpol->flags, &mpol->w.user_nodemask);
+ npol = mpol_new(&mparam);
if (IS_ERR(npol))
goto free_scratch; /* no valid nodemask intersection */
@@ -3029,6 +3047,7 @@ static inline void __init check_numabalancing_enable(void)
void __init numa_policy_init(void)
{
+ struct mempolicy_param param;
nodemask_t interleave_nodes;
unsigned long largest = 0;
int nid, prefer = 0;
@@ -3074,7 +3093,11 @@ void __init numa_policy_init(void)
if (unlikely(nodes_empty(interleave_nodes)))
node_set(prefer, interleave_nodes);
- if (do_set_mempolicy(MPOL_INTERLEAVE, 0, &interleave_nodes))
+ memset(¶m, 0, sizeof(param));
+ param.mode = MPOL_INTERLEAVE;
+ param.policy_nodes = &interleave_nodes;
+
+ if (do_set_mempolicy(¶m))
pr_err("%s: interleaving failed\n", __func__);
check_numabalancing_enable();
@@ -3083,7 +3106,12 @@ void __init numa_policy_init(void)
/* Reset policy of current process to default */
void numa_default_policy(void)
{
- do_set_mempolicy(MPOL_DEFAULT, 0, NULL);
+ struct mempolicy_param param;
+
+ memset(¶m, 0, sizeof(param));
+ param.mode = MPOL_DEFAULT;
+
+ do_set_mempolicy(¶m);
}
/*
@@ -3113,6 +3141,7 @@ static const char * const policy_modes[] =
*/
int mpol_parse_str(char *str, struct mempolicy **mpol)
{
+ struct mempolicy_param mparam;
struct mempolicy *new = NULL;
unsigned short mode_flags;
nodemask_t nodes;
@@ -3199,7 +3228,11 @@ int mpol_parse_str(char *str, struct mempolicy **mpol)
goto out;
}
- new = mpol_new(mode, mode_flags, &nodes);
+ memset(&mparam, 0, sizeof(mparam));
+ mparam.mode = mode;
+ mparam.mode_flags = mode_flags;
+ mparam.policy_nodes = &nodes;
+ new = mpol_new(&mparam);
if (IS_ERR(new))
goto out;
--
2.39.1
^ permalink raw reply related
* [PATCH v6 06/12] mm/mempolicy: refactor kernel_get_mempolicy for code re-use
From: Gregory Price @ 2024-01-03 22:42 UTC (permalink / raw)
To: linux-mm
Cc: linux-doc, linux-fsdevel, linux-kernel, linux-api, linux-arch,
akpm, arnd, tglx, luto, mingo, bp, dave.hansen, hpa, mhocko, tj,
ying.huang, gregory.price, corbet, rakie.kim, hyeongtak.ji,
honggyu.kim, vtavarespetr, peterz, jgroves, ravis.opensrc,
sthanneeru, emirakhur, Hasan.Maruf, seungjun.ha
In-Reply-To: <20240103224209.2541-1-gregory.price@memverge.com>
Pull operation flag checking from inside do_get_mempolicy out
to kernel_get_mempolicy. This allows us to flatten the
internal code, and break it into separate functions for future
syscalls (get_mempolicy2, process_get_mempolicy) to re-use the
code, even after additional extensions are made.
The primary change is that the flag is treated as the multiplexer
that it actually is. For get_mempolicy, the flags represents 3
different primary operations:
if (flags & MPOL_F_MEMS_ALLOWED)
return task->mems_allowed
else if (flags & MPOL_F_ADDR)
return vma mempolicy information
else
return task mempolicy information
Plus the behavior modifying flag:
if (flags & MPOL_F_NODE)
change the return value of (int __user *policy)
based on whether MPOL_F_ADDR was set.
The original behavior of get_mempolicy is retained, but we utilize
the new mempolicy_param structure to pass the operations down the
stack. This will allow us to extend the internal functions without
affecting the legacy behavior of get_mempolicy.
Signed-off-by: Gregory Price <gregory.price@memverge.com>
---
mm/mempolicy.c | 244 +++++++++++++++++++++++++++++++------------------
1 file changed, 154 insertions(+), 90 deletions(-)
diff --git a/mm/mempolicy.c b/mm/mempolicy.c
index 1f6f19b5d157..db290cf540d7 100644
--- a/mm/mempolicy.c
+++ b/mm/mempolicy.c
@@ -910,106 +910,109 @@ static int lookup_node(struct mm_struct *mm, unsigned long addr)
return ret;
}
-/* Retrieve NUMA policy */
-static long do_get_mempolicy(int *policy, nodemask_t *nmask,
- unsigned long addr, unsigned long flags)
+/* Retrieve the mems_allowed for current task */
+static inline long do_get_mems_allowed(nodemask_t *nmask)
{
- int err;
- struct mm_struct *mm = current->mm;
- struct vm_area_struct *vma = NULL;
- struct mempolicy *pol = current->mempolicy, *pol_refcount = NULL;
+ task_lock(current);
+ *nmask = cpuset_current_mems_allowed;
+ task_unlock(current);
+ return 0;
+}
- if (flags &
- ~(unsigned long)(MPOL_F_NODE|MPOL_F_ADDR|MPOL_F_MEMS_ALLOWED))
- return -EINVAL;
+/* If the policy has additional node information to retrieve, return it */
+static long do_get_policy_node(struct mempolicy *pol)
+{
+ /*
+ * For MPOL_INTERLEAVE, the extended node information is the next
+ * node that will be selected for interleave. For weighted interleave
+ * we return the next node based on the current weight.
+ */
+ if (pol == current->mempolicy && pol->mode == MPOL_INTERLEAVE)
+ return next_node_in(current->il_prev, pol->nodes);
- if (flags & MPOL_F_MEMS_ALLOWED) {
- if (flags & (MPOL_F_NODE|MPOL_F_ADDR))
- return -EINVAL;
- *policy = 0; /* just so it's initialized */
+ if (pol == current->mempolicy &&
+ pol->mode == MPOL_WEIGHTED_INTERLEAVE) {
+ if (pol->wil.cur_weight)
+ return current->il_prev;
+ else
+ return next_node_in(current->il_prev, pol->nodes);
+ }
+ return -EINVAL;
+}
+
+/* Handle user_nodemask condition when fetching nodemask for userspace */
+static void do_get_mempolicy_nodemask(struct mempolicy *pol, nodemask_t *nmask)
+{
+ if (mpol_store_user_nodemask(pol)) {
+ *nmask = pol->w.user_nodemask;
+ } else {
task_lock(current);
- *nmask = cpuset_current_mems_allowed;
+ get_policy_nodemask(pol, nmask);
task_unlock(current);
- return 0;
}
+}
- if (flags & MPOL_F_ADDR) {
- pgoff_t ilx; /* ignored here */
- /*
- * Do NOT fall back to task policy if the
- * vma/shared policy at addr is NULL. We
- * want to return MPOL_DEFAULT in this case.
- */
- mmap_read_lock(mm);
- vma = vma_lookup(mm, addr);
- if (!vma) {
- mmap_read_unlock(mm);
- return -EFAULT;
- }
- pol = __get_vma_policy(vma, addr, &ilx);
- } else if (addr)
- return -EINVAL;
+/* Retrieve NUMA policy for a VMA assocated with a given address */
+static long do_get_vma_mempolicy(unsigned long addr, int *addr_node,
+ struct mempolicy_param *param)
+{
+ pgoff_t ilx;
+ struct mm_struct *mm = current->mm;
+ struct vm_area_struct *vma = NULL;
+ struct mempolicy *pol = NULL;
+ mmap_read_lock(mm);
+ vma = vma_lookup(mm, addr);
+ if (!vma) {
+ mmap_read_unlock(mm);
+ return -EFAULT;
+ }
+ pol = __get_vma_policy(vma, addr, &ilx);
if (!pol)
- pol = &default_policy; /* indicates default behavior */
+ pol = &default_policy;
+ else
+ mpol_get(pol);
+ mmap_read_unlock(mm);
- if (flags & MPOL_F_NODE) {
- if (flags & MPOL_F_ADDR) {
- /*
- * Take a refcount on the mpol, because we are about to
- * drop the mmap_lock, after which only "pol" remains
- * valid, "vma" is stale.
- */
- pol_refcount = pol;
- vma = NULL;
- mpol_get(pol);
- mmap_read_unlock(mm);
- err = lookup_node(mm, addr);
- if (err < 0)
- goto out;
- *policy = err;
- } else if (pol == current->mempolicy &&
- pol->mode == MPOL_INTERLEAVE) {
- *policy = next_node_in(current->il_prev, pol->nodes);
- } else if (pol == current->mempolicy &&
- (pol->mode == MPOL_WEIGHTED_INTERLEAVE)) {
- if (pol->wil.cur_weight)
- *policy = current->il_prev;
- else
- *policy = next_node_in(current->il_prev,
- pol->nodes);
- } else {
- err = -EINVAL;
- goto out;
- }
- } else {
- *policy = pol == &default_policy ? MPOL_DEFAULT :
- pol->mode;
- /*
- * Internal mempolicy flags must be masked off before exposing
- * the policy to userspace.
- */
- *policy |= (pol->flags & MPOL_MODE_FLAGS);
- }
+ /* Fetch the node for the given address */
+ if (addr_node)
+ *addr_node = lookup_node(mm, addr);
- err = 0;
- if (nmask) {
- if (mpol_store_user_nodemask(pol)) {
- *nmask = pol->w.user_nodemask;
- } else {
- task_lock(current);
- get_policy_nodemask(pol, nmask);
- task_unlock(current);
- }
+ param->mode = pol == &default_policy ? MPOL_DEFAULT : pol->mode;
+ param->mode_flags = (pol->flags & MPOL_MODE_FLAGS);
+ param->home_node = pol->home_node;
+
+ if (param->policy_nodes)
+ do_get_mempolicy_nodemask(pol, param->policy_nodes);
+
+ if (pol != &default_policy) {
+ mpol_put(pol);
+ mpol_cond_put(pol);
}
- out:
- mpol_cond_put(pol);
- if (vma)
- mmap_read_unlock(mm);
- if (pol_refcount)
- mpol_put(pol_refcount);
- return err;
+ return 0;
+}
+
+/* Retrieve NUMA policy for the current task */
+static long do_get_task_mempolicy(struct mempolicy_param *param, int *pol_node)
+{
+ struct mempolicy *pol = current->mempolicy;
+
+ if (!pol)
+ pol = &default_policy; /* indicates default behavior */
+
+ param->mode = pol == &default_policy ? MPOL_DEFAULT : pol->mode;
+ /* Internal flags must be masked off before exposing to userspace */
+ param->mode_flags = (pol->flags & MPOL_MODE_FLAGS);
+ param->home_node = NUMA_NO_NODE;
+
+ if (pol_node)
+ *pol_node = do_get_policy_node(pol);
+
+ if (param->policy_nodes)
+ do_get_mempolicy_nodemask(pol, param->policy_nodes);
+
+ return 0;
}
#ifdef CONFIG_MIGRATION
@@ -1742,16 +1745,77 @@ static int kernel_get_mempolicy(int __user *policy,
unsigned long addr,
unsigned long flags)
{
+ struct mempolicy_param param;
int err;
- int pval;
+ int address_node = NUMA_NO_NODE;
+ int pval = 0;
+ int pol_node = 0;
nodemask_t nodes;
if (nmask != NULL && maxnode < nr_node_ids)
return -EINVAL;
- addr = untagged_addr(addr);
+ if (flags &
+ ~(unsigned long)(MPOL_F_NODE|MPOL_F_ADDR|MPOL_F_MEMS_ALLOWED))
+ return -EINVAL;
- err = do_get_mempolicy(&pval, &nodes, addr, flags);
+ /* Ensure any data that may be copied to userland is initialized */
+ memset(¶m, 0, sizeof(param));
+ param.policy_nodes = &nodes;
+
+ /*
+ * set_mempolicy was originally multiplexed based on 3 flags:
+ * MPOL_F_MEMS_ALLOWED: fetch task->mems_allowed
+ * MPOL_F_ADDR : operate on vma->mempolicy
+ * MPOL_F_NODE : change return value of *policy
+ *
+ * Split this behavior out here, rather than internal functions,
+ * so that the internal functions can be re-used by future
+ * get_mempolicy2 interfaces and the arg structure made extensible
+ */
+ if (flags & MPOL_F_MEMS_ALLOWED) {
+ if (flags & (MPOL_F_NODE|MPOL_F_ADDR))
+ return -EINVAL;
+ pval = 0; /* just so it's initialized */
+ err = do_get_mems_allowed(&nodes);
+ } else if (flags & MPOL_F_ADDR) {
+ /* If F_ADDR, we operation on a vma policy (or default) */
+ err = do_get_vma_mempolicy(untagged_addr(addr),
+ &address_node, ¶m);
+ if (err)
+ return err;
+ /* if (F_ADDR | F_NODE), *pval is the address' node */
+ if (flags & MPOL_F_NODE) {
+ /* if we failed to fetch, that's likely an EFAULT */
+ if (address_node < 0)
+ return address_node;
+ pval = address_node;
+ } else
+ pval = param.mode | param.mode_flags;
+ } else {
+ /* if not F_ADDR and addr != null, EINVAL */
+ if (addr)
+ return -EINVAL;
+
+ err = do_get_task_mempolicy(¶m, &pol_node);
+ if (err)
+ return err;
+ /*
+ * if F_NODE was set and mode was MPOL_INTERLEAVE
+ * *pval is equal to next interleave node.
+ *
+ * if pol_node < 0, this means the mode did not have a
+ * a compatible policy. This presently emulates the
+ * original behavior of (F_NODE) & (!MPOL_INTERLEAVE)
+ * producing -EINVAL
+ */
+ if (flags & MPOL_F_NODE) {
+ if (pol_node < 0)
+ return pol_node;
+ pval = pol_node;
+ } else
+ pval = param.mode | param.mode_flags;
+ }
if (err)
return err;
--
2.39.1
^ permalink raw reply related
* [PATCH v6 07/12] mm/mempolicy: allow home_node to be set by mpol_new
From: Gregory Price @ 2024-01-03 22:42 UTC (permalink / raw)
To: linux-mm
Cc: linux-doc, linux-fsdevel, linux-kernel, linux-api, linux-arch,
akpm, arnd, tglx, luto, mingo, bp, dave.hansen, hpa, mhocko, tj,
ying.huang, gregory.price, corbet, rakie.kim, hyeongtak.ji,
honggyu.kim, vtavarespetr, peterz, jgroves, ravis.opensrc,
sthanneeru, emirakhur, Hasan.Maruf, seungjun.ha
In-Reply-To: <20240103224209.2541-1-gregory.price@memverge.com>
This patch adds the plumbing into mpol_new() to allow the argument
structure's home_node field to be set during mempolicy creation.
The syscall sys_set_mempolicy_home_node was added to allow a home
node to be registered for a vma.
For set_mempolicy2 and mbind2 syscalls, it would be useful to add
this as an extension to allow the user to submit a fully formed
mempolicy configuration in a single call, rather than require
multiple calls to configure a mempolicy.
This will become particularly useful if/when pidfd interfaces to
change process mempolicies from outside the task appear, as each
call to change the mempolicy does an atomic swap of that policy
in the task, rather than mutate the policy.
Signed-off-by: Gregory Price <gregory.price@memverge.com>
---
mm/mempolicy.c | 9 ++++++++-
1 file changed, 8 insertions(+), 1 deletion(-)
diff --git a/mm/mempolicy.c b/mm/mempolicy.c
index db290cf540d7..4a566341cf43 100644
--- a/mm/mempolicy.c
+++ b/mm/mempolicy.c
@@ -321,7 +321,7 @@ static struct mempolicy *mpol_new(struct mempolicy_param *param)
atomic_set(&policy->refcnt, 1);
policy->mode = mode;
policy->flags = flags;
- policy->home_node = NUMA_NO_NODE;
+ policy->home_node = param->home_node;
policy->wil.cur_weight = 0;
return policy;
@@ -1634,6 +1634,7 @@ static long kernel_set_mempolicy(int mode, const unsigned long __user *nmask,
param.mode = lmode;
param.mode_flags = mode_flags;
param.policy_nodes = &nodes;
+ param.home_node = NUMA_NO_NODE;
return do_set_mempolicy(¶m);
}
@@ -3002,6 +3003,8 @@ void mpol_shared_policy_init(struct shared_policy *sp, struct mempolicy *mpol)
mparam.mode = mpol->mode;
mparam.mode_flags = mpol->flags;
mparam.policy_nodes = &mpol->w.user_nodemask;
+ mparam.home_node = NUMA_NO_NODE;
+
/* contextualize the tmpfs mount point mempolicy to this file */
npol = mpol_new(&mparam);
if (IS_ERR(npol))
@@ -3160,6 +3163,7 @@ void __init numa_policy_init(void)
memset(¶m, 0, sizeof(param));
param.mode = MPOL_INTERLEAVE;
param.policy_nodes = &interleave_nodes;
+ param.home_node = NUMA_NO_NODE;
if (do_set_mempolicy(¶m))
pr_err("%s: interleaving failed\n", __func__);
@@ -3174,6 +3178,7 @@ void numa_default_policy(void)
memset(¶m, 0, sizeof(param));
param.mode = MPOL_DEFAULT;
+ param.home_node = NUMA_NO_NODE;
do_set_mempolicy(¶m);
}
@@ -3296,6 +3301,8 @@ int mpol_parse_str(char *str, struct mempolicy **mpol)
mparam.mode = mode;
mparam.mode_flags = mode_flags;
mparam.policy_nodes = &nodes;
+ mparam.home_node = NUMA_NO_NODE;
+
new = mpol_new(&mparam);
if (IS_ERR(new))
goto out;
--
2.39.1
^ permalink raw reply related
* [PATCH v6 08/12] mm/mempolicy: add userland mempolicy arg structure
From: Gregory Price @ 2024-01-03 22:42 UTC (permalink / raw)
To: linux-mm
Cc: linux-doc, linux-fsdevel, linux-kernel, linux-api, linux-arch,
akpm, arnd, tglx, luto, mingo, bp, dave.hansen, hpa, mhocko, tj,
ying.huang, gregory.price, corbet, rakie.kim, hyeongtak.ji,
honggyu.kim, vtavarespetr, peterz, jgroves, ravis.opensrc,
sthanneeru, emirakhur, Hasan.Maruf, seungjun.ha,
Frank van der Linden
In-Reply-To: <20240103224209.2541-1-gregory.price@memverge.com>
This patch adds the new user-api argument structure intended for
set_mempolicy2 and mbind2.
struct mpol_param {
__u16 mode;
__u16 mode_flags;
__s32 home_node; /* mbind2: policy home node */
__u16 pol_maxnodes;
__u8 resv[6];
__aligned_u64 *pol_nodes;
};
This structure is intended to be extensible as new mempolicy extensions
are added.
For example, set_mempolicy_home_node was added to allow vma mempolicies
to have a preferred/home node assigned. This structure allows the user
to set the home node at the time mempolicy is created, rather than
requiring an additional syscalls.
Full breakdown of arguments as of this patch:
mode: Mempolicy mode (MPOL_DEFAULT, MPOL_INTERLEAVE)
mode_flags: Flags previously or'd into mode in set_mempolicy
(e.g.: MPOL_F_STATIC_NODES, MPOL_F_RELATIVE_NODES)
home_node: for mbind2. Allows the setting of a policy's home
with the use of MPOL_MF_HOME_NODE
pol_maxnodes: Max number of nodes in the policy nodemask
pol_nodes: Policy nodemask
The reserved field accounts explicitly for a potential memory hole
in the structure.
Suggested-by: Frank van der Linden <fvdl@google.com>
Suggested-by: Vinicius Tavares Petrucci <vtavarespetr@micron.com>
Suggested-by: Hasan Al Maruf <Hasan.Maruf@amd.com>
Signed-off-by: Gregory Price <gregory.price@memverge.com>
Co-developed-by: Vinicius Tavares Petrucci <vtavarespetr@micron.com>
Signed-off-by: Vinicius Tavares Petrucci <vtavarespetr@micron.com>
---
.../admin-guide/mm/numa_memory_policy.rst | 17 +++++++++++++++++
include/linux/syscalls.h | 1 +
include/uapi/linux/mempolicy.h | 9 +++++++++
3 files changed, 27 insertions(+)
diff --git a/Documentation/admin-guide/mm/numa_memory_policy.rst b/Documentation/admin-guide/mm/numa_memory_policy.rst
index a70f20ce1ffb..cbfc5f65ed77 100644
--- a/Documentation/admin-guide/mm/numa_memory_policy.rst
+++ b/Documentation/admin-guide/mm/numa_memory_policy.rst
@@ -480,6 +480,23 @@ closest to which page allocation will come from. Specifying the home node overri
the default allocation policy to allocate memory close to the local node for an
executing CPU.
+Extended Mempolicy Arguments::
+
+ struct mpol_param {
+ __u16 mode;
+ __u16 mode_flags;
+ __s32 home_node; /* mbind2: set home node */
+ __u64 pol_maxnodes;
+ __aligned_u64 pol_nodes; /* nodemask pointer */
+ };
+
+The extended mempolicy argument structure is defined to allow the mempolicy
+interfaces future extensibility without the need for additional system calls.
+
+The core arguments (mode, mode_flags, pol_nodes, and pol_maxnodes) apply to
+all interfaces relative to their non-extended counterparts. Each additional
+field may only apply to specific extended interfaces. See the respective
+extended interface man page for more details.
Memory Policy Command Line Interface
====================================
diff --git a/include/linux/syscalls.h b/include/linux/syscalls.h
index fd9d12de7e92..fb0b4b2b9bea 100644
--- a/include/linux/syscalls.h
+++ b/include/linux/syscalls.h
@@ -74,6 +74,7 @@ struct landlock_ruleset_attr;
enum landlock_rule_type;
struct cachestat_range;
struct cachestat;
+struct mpol_param;
#include <linux/types.h>
#include <linux/aio_abi.h>
diff --git a/include/uapi/linux/mempolicy.h b/include/uapi/linux/mempolicy.h
index 1f9bb10d1a47..109788c8be92 100644
--- a/include/uapi/linux/mempolicy.h
+++ b/include/uapi/linux/mempolicy.h
@@ -27,6 +27,15 @@ enum {
MPOL_MAX, /* always last member of enum */
};
+struct mpol_param {
+ __u16 mode;
+ __u16 mode_flags;
+ __s32 home_node; /* mbind2: policy home node */
+ __u16 pol_maxnodes;
+ __u8 resv[6];
+ __aligned_u64 pol_nodes;
+};
+
/* Flags for set_mempolicy */
#define MPOL_F_STATIC_NODES (1 << 15)
#define MPOL_F_RELATIVE_NODES (1 << 14)
--
2.39.1
^ permalink raw reply related
* [PATCH v6 09/12] mm/mempolicy: add set_mempolicy2 syscall
From: Gregory Price @ 2024-01-03 22:42 UTC (permalink / raw)
To: linux-mm
Cc: linux-doc, linux-fsdevel, linux-kernel, linux-api, linux-arch,
akpm, arnd, tglx, luto, mingo, bp, dave.hansen, hpa, mhocko, tj,
ying.huang, gregory.price, corbet, rakie.kim, hyeongtak.ji,
honggyu.kim, vtavarespetr, peterz, jgroves, ravis.opensrc,
sthanneeru, emirakhur, Hasan.Maruf, seungjun.ha, Michal Hocko,
Geert Uytterhoeven
In-Reply-To: <20240103224209.2541-1-gregory.price@memverge.com>
set_mempolicy2 is an extensible set_mempolicy interface which allows
a user to set the per-task memory policy.
Defined as:
set_mempolicy2(struct mpol_param *param, size_t usize,
unsigned long flags);
relevant mpol_param fields include the following:
mode: The MPOL_* policy (DEFAULT, INTERLEAVE, etc.)
mode_flags: The MPOL_F_* flags that were previously passed in or'd
into the mode. This was split to hopefully allow future
extensions additional mode/flag space.
home_node: ignored (see note below)
pol_nodes: the nodemask to apply for the memory policy
pol_maxnodes: The max number of nodes described by pol_nodes
The usize arg is intended for the user to pass in sizeof(mpol_param)
to allow forward/backward compatibility whenever possible.
The flags argument is intended to future proof the syscall against
future extensions which may require interpreting the arguments in
the structure differently.
Semantics of `set_mempolicy` are otherwise the same as `set_mempolicy`
as of this patch.
As of this patch, setting the home node of a task-policy is not
supported, as this functionality was not supported by set_mempolicy.
Additional research should be done to determine whether adding this
functionality is safe, but doing so would only require setting
MPOL_MF_HOME_NODE and providing a valid home node value.
Suggested-by: Michal Hocko <mhocko@suse.com>
Signed-off-by: Gregory Price <gregory.price@memverge.com>
Acked-by: Geert Uytterhoeven <geert@linux-m68k.org>
---
.../admin-guide/mm/numa_memory_policy.rst | 10 ++++++
arch/alpha/kernel/syscalls/syscall.tbl | 1 +
arch/arm/tools/syscall.tbl | 1 +
arch/arm64/include/asm/unistd.h | 2 +-
arch/arm64/include/asm/unistd32.h | 2 ++
arch/m68k/kernel/syscalls/syscall.tbl | 1 +
arch/microblaze/kernel/syscalls/syscall.tbl | 1 +
arch/mips/kernel/syscalls/syscall_n32.tbl | 1 +
arch/mips/kernel/syscalls/syscall_o32.tbl | 1 +
arch/parisc/kernel/syscalls/syscall.tbl | 1 +
arch/powerpc/kernel/syscalls/syscall.tbl | 1 +
arch/s390/kernel/syscalls/syscall.tbl | 1 +
arch/sh/kernel/syscalls/syscall.tbl | 1 +
arch/sparc/kernel/syscalls/syscall.tbl | 1 +
arch/x86/entry/syscalls/syscall_32.tbl | 1 +
arch/x86/entry/syscalls/syscall_64.tbl | 1 +
arch/xtensa/kernel/syscalls/syscall.tbl | 1 +
include/linux/syscalls.h | 2 ++
include/uapi/asm-generic/unistd.h | 4 ++-
kernel/sys_ni.c | 1 +
mm/mempolicy.c | 36 +++++++++++++++++++
.../arch/mips/entry/syscalls/syscall_n64.tbl | 1 +
.../arch/powerpc/entry/syscalls/syscall.tbl | 1 +
.../perf/arch/s390/entry/syscalls/syscall.tbl | 1 +
.../arch/x86/entry/syscalls/syscall_64.tbl | 1 +
25 files changed, 73 insertions(+), 2 deletions(-)
diff --git a/Documentation/admin-guide/mm/numa_memory_policy.rst b/Documentation/admin-guide/mm/numa_memory_policy.rst
index cbfc5f65ed77..62a4ea14c646 100644
--- a/Documentation/admin-guide/mm/numa_memory_policy.rst
+++ b/Documentation/admin-guide/mm/numa_memory_policy.rst
@@ -430,6 +430,8 @@ Set [Task] Memory Policy::
long set_mempolicy(int mode, const unsigned long *nmask,
unsigned long maxnode);
+ long set_mempolicy2(struct mpol_param *param, size_t size,
+ unsigned long flags);
Set's the calling task's "task/process memory policy" to mode
specified by the 'mode' argument and the set of nodes defined by
@@ -438,6 +440,12 @@ specified by the 'mode' argument and the set of nodes defined by
'mode' argument with the flag (for example: MPOL_INTERLEAVE |
MPOL_F_STATIC_NODES).
+set_mempolicy2() is an extended version of set_mempolicy() capable
+of setting a mempolicy which requires more information than can be
+passed via get_mempolicy(). For example, weighted interleave with
+task-local weights requires a weight array to be passed via the
+'mpol_param->il_weights' parameter.
+
See the set_mempolicy(2) man page for more details
@@ -493,6 +501,8 @@ Extended Mempolicy Arguments::
The extended mempolicy argument structure is defined to allow the mempolicy
interfaces future extensibility without the need for additional system calls.
+Extended interfaces (set_mempolicy2) use this argument structure.
+
The core arguments (mode, mode_flags, pol_nodes, and pol_maxnodes) apply to
all interfaces relative to their non-extended counterparts. Each additional
field may only apply to specific extended interfaces. See the respective
diff --git a/arch/alpha/kernel/syscalls/syscall.tbl b/arch/alpha/kernel/syscalls/syscall.tbl
index 18c842ca6c32..0dc288a1118a 100644
--- a/arch/alpha/kernel/syscalls/syscall.tbl
+++ b/arch/alpha/kernel/syscalls/syscall.tbl
@@ -496,3 +496,4 @@
564 common futex_wake sys_futex_wake
565 common futex_wait sys_futex_wait
566 common futex_requeue sys_futex_requeue
+567 common set_mempolicy2 sys_set_mempolicy2
diff --git a/arch/arm/tools/syscall.tbl b/arch/arm/tools/syscall.tbl
index 584f9528c996..50172ec0e1f5 100644
--- a/arch/arm/tools/syscall.tbl
+++ b/arch/arm/tools/syscall.tbl
@@ -470,3 +470,4 @@
454 common futex_wake sys_futex_wake
455 common futex_wait sys_futex_wait
456 common futex_requeue sys_futex_requeue
+457 common set_mempolicy2 sys_set_mempolicy2
diff --git a/arch/arm64/include/asm/unistd.h b/arch/arm64/include/asm/unistd.h
index 531effca5f1f..298313d2e0af 100644
--- a/arch/arm64/include/asm/unistd.h
+++ b/arch/arm64/include/asm/unistd.h
@@ -39,7 +39,7 @@
#define __ARM_NR_compat_set_tls (__ARM_NR_COMPAT_BASE + 5)
#define __ARM_NR_COMPAT_END (__ARM_NR_COMPAT_BASE + 0x800)
-#define __NR_compat_syscalls 457
+#define __NR_compat_syscalls 458
#endif
#define __ARCH_WANT_SYS_CLONE
diff --git a/arch/arm64/include/asm/unistd32.h b/arch/arm64/include/asm/unistd32.h
index 9f7c1bf99526..cee8d669c342 100644
--- a/arch/arm64/include/asm/unistd32.h
+++ b/arch/arm64/include/asm/unistd32.h
@@ -919,6 +919,8 @@ __SYSCALL(__NR_futex_wake, sys_futex_wake)
__SYSCALL(__NR_futex_wait, sys_futex_wait)
#define __NR_futex_requeue 456
__SYSCALL(__NR_futex_requeue, sys_futex_requeue)
+#define __NR_set_mempolicy2 457
+__SYSCALL(__NR_set_mempolicy2, sys_set_mempolicy2)
/*
* Please add new compat syscalls above this comment and update
diff --git a/arch/m68k/kernel/syscalls/syscall.tbl b/arch/m68k/kernel/syscalls/syscall.tbl
index 7a4b780e82cb..839d90c535f2 100644
--- a/arch/m68k/kernel/syscalls/syscall.tbl
+++ b/arch/m68k/kernel/syscalls/syscall.tbl
@@ -456,3 +456,4 @@
454 common futex_wake sys_futex_wake
455 common futex_wait sys_futex_wait
456 common futex_requeue sys_futex_requeue
+457 common set_mempolicy2 sys_set_mempolicy2
diff --git a/arch/microblaze/kernel/syscalls/syscall.tbl b/arch/microblaze/kernel/syscalls/syscall.tbl
index 5b6a0b02b7de..567c8b883735 100644
--- a/arch/microblaze/kernel/syscalls/syscall.tbl
+++ b/arch/microblaze/kernel/syscalls/syscall.tbl
@@ -462,3 +462,4 @@
454 common futex_wake sys_futex_wake
455 common futex_wait sys_futex_wait
456 common futex_requeue sys_futex_requeue
+457 common set_mempolicy2 sys_set_mempolicy2
diff --git a/arch/mips/kernel/syscalls/syscall_n32.tbl b/arch/mips/kernel/syscalls/syscall_n32.tbl
index a842b41c8e06..cc0640e16f2f 100644
--- a/arch/mips/kernel/syscalls/syscall_n32.tbl
+++ b/arch/mips/kernel/syscalls/syscall_n32.tbl
@@ -395,3 +395,4 @@
454 n32 futex_wake sys_futex_wake
455 n32 futex_wait sys_futex_wait
456 n32 futex_requeue sys_futex_requeue
+457 n32 set_mempolicy2 sys_set_mempolicy2
diff --git a/arch/mips/kernel/syscalls/syscall_o32.tbl b/arch/mips/kernel/syscalls/syscall_o32.tbl
index 525cc54bc63b..f7262fde98d9 100644
--- a/arch/mips/kernel/syscalls/syscall_o32.tbl
+++ b/arch/mips/kernel/syscalls/syscall_o32.tbl
@@ -444,3 +444,4 @@
454 o32 futex_wake sys_futex_wake
455 o32 futex_wait sys_futex_wait
456 o32 futex_requeue sys_futex_requeue
+457 o32 set_mempolicy2 sys_set_mempolicy2
diff --git a/arch/parisc/kernel/syscalls/syscall.tbl b/arch/parisc/kernel/syscalls/syscall.tbl
index a47798fed54e..e10f0e8bd064 100644
--- a/arch/parisc/kernel/syscalls/syscall.tbl
+++ b/arch/parisc/kernel/syscalls/syscall.tbl
@@ -455,3 +455,4 @@
454 common futex_wake sys_futex_wake
455 common futex_wait sys_futex_wait
456 common futex_requeue sys_futex_requeue
+457 common set_mempolicy2 sys_set_mempolicy2
diff --git a/arch/powerpc/kernel/syscalls/syscall.tbl b/arch/powerpc/kernel/syscalls/syscall.tbl
index 7fab411378f2..4f03f5f42b78 100644
--- a/arch/powerpc/kernel/syscalls/syscall.tbl
+++ b/arch/powerpc/kernel/syscalls/syscall.tbl
@@ -543,3 +543,4 @@
454 common futex_wake sys_futex_wake
455 common futex_wait sys_futex_wait
456 common futex_requeue sys_futex_requeue
+457 common set_mempolicy2 sys_set_mempolicy2
diff --git a/arch/s390/kernel/syscalls/syscall.tbl b/arch/s390/kernel/syscalls/syscall.tbl
index 86fec9b080f6..f98dadc2e9df 100644
--- a/arch/s390/kernel/syscalls/syscall.tbl
+++ b/arch/s390/kernel/syscalls/syscall.tbl
@@ -459,3 +459,4 @@
454 common futex_wake sys_futex_wake sys_futex_wake
455 common futex_wait sys_futex_wait sys_futex_wait
456 common futex_requeue sys_futex_requeue sys_futex_requeue
+457 common set_mempolicy2 sys_set_mempolicy2 sys_set_mempolicy2
diff --git a/arch/sh/kernel/syscalls/syscall.tbl b/arch/sh/kernel/syscalls/syscall.tbl
index 363fae0fe9bf..f47ba9f2d05d 100644
--- a/arch/sh/kernel/syscalls/syscall.tbl
+++ b/arch/sh/kernel/syscalls/syscall.tbl
@@ -459,3 +459,4 @@
454 common futex_wake sys_futex_wake
455 common futex_wait sys_futex_wait
456 common futex_requeue sys_futex_requeue
+457 common set_mempolicy2 sys_set_mempolicy2
diff --git a/arch/sparc/kernel/syscalls/syscall.tbl b/arch/sparc/kernel/syscalls/syscall.tbl
index 7bcaa3d5ea44..53fb16616728 100644
--- a/arch/sparc/kernel/syscalls/syscall.tbl
+++ b/arch/sparc/kernel/syscalls/syscall.tbl
@@ -502,3 +502,4 @@
454 common futex_wake sys_futex_wake
455 common futex_wait sys_futex_wait
456 common futex_requeue sys_futex_requeue
+457 common set_mempolicy2 sys_set_mempolicy2
diff --git a/arch/x86/entry/syscalls/syscall_32.tbl b/arch/x86/entry/syscalls/syscall_32.tbl
index c8fac5205803..4b4dc41b24ee 100644
--- a/arch/x86/entry/syscalls/syscall_32.tbl
+++ b/arch/x86/entry/syscalls/syscall_32.tbl
@@ -461,3 +461,4 @@
454 i386 futex_wake sys_futex_wake
455 i386 futex_wait sys_futex_wait
456 i386 futex_requeue sys_futex_requeue
+457 i386 set_mempolicy2 sys_set_mempolicy2
diff --git a/arch/x86/entry/syscalls/syscall_64.tbl b/arch/x86/entry/syscalls/syscall_64.tbl
index 8cb8bf68721c..1bc2190bec27 100644
--- a/arch/x86/entry/syscalls/syscall_64.tbl
+++ b/arch/x86/entry/syscalls/syscall_64.tbl
@@ -378,6 +378,7 @@
454 common futex_wake sys_futex_wake
455 common futex_wait sys_futex_wait
456 common futex_requeue sys_futex_requeue
+457 common set_mempolicy2 sys_set_mempolicy2
#
# Due to a historical design error, certain syscalls are numbered differently
diff --git a/arch/xtensa/kernel/syscalls/syscall.tbl b/arch/xtensa/kernel/syscalls/syscall.tbl
index 06eefa9c1458..e26dc89399eb 100644
--- a/arch/xtensa/kernel/syscalls/syscall.tbl
+++ b/arch/xtensa/kernel/syscalls/syscall.tbl
@@ -427,3 +427,4 @@
454 common futex_wake sys_futex_wake
455 common futex_wait sys_futex_wait
456 common futex_requeue sys_futex_requeue
+457 common set_mempolicy2 sys_set_mempolicy2
diff --git a/include/linux/syscalls.h b/include/linux/syscalls.h
index fb0b4b2b9bea..b37ea6715456 100644
--- a/include/linux/syscalls.h
+++ b/include/linux/syscalls.h
@@ -823,6 +823,8 @@ asmlinkage long sys_get_mempolicy(int __user *policy,
unsigned long addr, unsigned long flags);
asmlinkage long sys_set_mempolicy(int mode, const unsigned long __user *nmask,
unsigned long maxnode);
+asmlinkage long sys_set_mempolicy2(struct mpol_param __user *param, size_t size,
+ unsigned long flags);
asmlinkage long sys_migrate_pages(pid_t pid, unsigned long maxnode,
const unsigned long __user *from,
const unsigned long __user *to);
diff --git a/include/uapi/asm-generic/unistd.h b/include/uapi/asm-generic/unistd.h
index 756b013fb832..55486aba099f 100644
--- a/include/uapi/asm-generic/unistd.h
+++ b/include/uapi/asm-generic/unistd.h
@@ -828,9 +828,11 @@ __SYSCALL(__NR_futex_wake, sys_futex_wake)
__SYSCALL(__NR_futex_wait, sys_futex_wait)
#define __NR_futex_requeue 456
__SYSCALL(__NR_futex_requeue, sys_futex_requeue)
+#define __NR_set_mempolicy2 457
+__SYSCALL(__NR_set_mempolicy2, sys_set_mempolicy2)
#undef __NR_syscalls
-#define __NR_syscalls 457
+#define __NR_syscalls 458
/*
* 32 bit systems traditionally used different
diff --git a/kernel/sys_ni.c b/kernel/sys_ni.c
index 9a846439b36a..fa1373c8bff8 100644
--- a/kernel/sys_ni.c
+++ b/kernel/sys_ni.c
@@ -189,6 +189,7 @@ COND_SYSCALL(remap_file_pages);
COND_SYSCALL(mbind);
COND_SYSCALL(get_mempolicy);
COND_SYSCALL(set_mempolicy);
+COND_SYSCALL(set_mempolicy2);
COND_SYSCALL(migrate_pages);
COND_SYSCALL(move_pages);
COND_SYSCALL(set_mempolicy_home_node);
diff --git a/mm/mempolicy.c b/mm/mempolicy.c
index 4a566341cf43..84d877195deb 100644
--- a/mm/mempolicy.c
+++ b/mm/mempolicy.c
@@ -1645,6 +1645,42 @@ SYSCALL_DEFINE3(set_mempolicy, int, mode, const unsigned long __user *, nmask,
return kernel_set_mempolicy(mode, nmask, maxnode);
}
+SYSCALL_DEFINE3(set_mempolicy2, struct mpol_param __user *, uparam,
+ size_t, usize, unsigned long, flags)
+{
+ struct mpol_param kparam;
+ struct mempolicy_param mparam;
+ int err;
+ nodemask_t policy_nodemask;
+ unsigned long __user *nodes_ptr;
+
+ if (flags)
+ return -EINVAL;
+
+ err = copy_struct_from_user(&kparam, sizeof(kparam), uparam, usize);
+ if (err)
+ return err;
+
+ err = validate_mpol_flags(kparam.mode, &kparam.mode_flags);
+ if (err)
+ return err;
+
+ memset(&mparam, 0, sizeof(mparam));
+ mparam.mode = kparam.mode;
+ mparam.mode_flags = kparam.mode_flags;
+ if (kparam.pol_nodes) {
+ nodes_ptr = u64_to_user_ptr(kparam.pol_nodes);
+ err = get_nodes(&policy_nodemask, nodes_ptr,
+ kparam.pol_maxnodes);
+ if (err)
+ return err;
+ mparam.policy_nodes = &policy_nodemask;
+ } else
+ mparam.policy_nodes = NULL;
+
+ return do_set_mempolicy(&mparam);
+}
+
static int kernel_migrate_pages(pid_t pid, unsigned long maxnode,
const unsigned long __user *old_nodes,
const unsigned long __user *new_nodes)
diff --git a/tools/perf/arch/mips/entry/syscalls/syscall_n64.tbl b/tools/perf/arch/mips/entry/syscalls/syscall_n64.tbl
index 116ff501bf92..bb1351df51d9 100644
--- a/tools/perf/arch/mips/entry/syscalls/syscall_n64.tbl
+++ b/tools/perf/arch/mips/entry/syscalls/syscall_n64.tbl
@@ -371,3 +371,4 @@
454 n64 futex_wake sys_futex_wake
455 n64 futex_wait sys_futex_wait
456 n64 futex_requeue sys_futex_requeue
+457 n64 set_mempolicy2 sys_set_mempolicy2
diff --git a/tools/perf/arch/powerpc/entry/syscalls/syscall.tbl b/tools/perf/arch/powerpc/entry/syscalls/syscall.tbl
index 7fab411378f2..4f03f5f42b78 100644
--- a/tools/perf/arch/powerpc/entry/syscalls/syscall.tbl
+++ b/tools/perf/arch/powerpc/entry/syscalls/syscall.tbl
@@ -543,3 +543,4 @@
454 common futex_wake sys_futex_wake
455 common futex_wait sys_futex_wait
456 common futex_requeue sys_futex_requeue
+457 common set_mempolicy2 sys_set_mempolicy2
diff --git a/tools/perf/arch/s390/entry/syscalls/syscall.tbl b/tools/perf/arch/s390/entry/syscalls/syscall.tbl
index 86fec9b080f6..f98dadc2e9df 100644
--- a/tools/perf/arch/s390/entry/syscalls/syscall.tbl
+++ b/tools/perf/arch/s390/entry/syscalls/syscall.tbl
@@ -459,3 +459,4 @@
454 common futex_wake sys_futex_wake sys_futex_wake
455 common futex_wait sys_futex_wait sys_futex_wait
456 common futex_requeue sys_futex_requeue sys_futex_requeue
+457 common set_mempolicy2 sys_set_mempolicy2 sys_set_mempolicy2
diff --git a/tools/perf/arch/x86/entry/syscalls/syscall_64.tbl b/tools/perf/arch/x86/entry/syscalls/syscall_64.tbl
index 8cb8bf68721c..21f2579679d4 100644
--- a/tools/perf/arch/x86/entry/syscalls/syscall_64.tbl
+++ b/tools/perf/arch/x86/entry/syscalls/syscall_64.tbl
@@ -378,6 +378,7 @@
454 common futex_wake sys_futex_wake
455 common futex_wait sys_futex_wait
456 common futex_requeue sys_futex_requeue
+457 common set_mempolicy2 sys_set_mempolicy2
#
# Due to a historical design error, certain syscalls are numbered differently
--
2.39.1
^ permalink raw reply related
* [PATCH v6 10/12] mm/mempolicy: add get_mempolicy2 syscall
From: Gregory Price @ 2024-01-03 22:42 UTC (permalink / raw)
To: linux-mm
Cc: linux-doc, linux-fsdevel, linux-kernel, linux-api, linux-arch,
akpm, arnd, tglx, luto, mingo, bp, dave.hansen, hpa, mhocko, tj,
ying.huang, gregory.price, corbet, rakie.kim, hyeongtak.ji,
honggyu.kim, vtavarespetr, peterz, jgroves, ravis.opensrc,
sthanneeru, emirakhur, Hasan.Maruf, seungjun.ha, Michal Hocko,
Geert Uytterhoeven
In-Reply-To: <20240103224209.2541-1-gregory.price@memverge.com>
get_mempolicy2 is an extensible get_mempolicy interface which allows
a user to retrieve the memory policy for a task or address.
Defined as:
get_mempolicy2(struct mpol_param *param, size_t size,
unsigned long addr, unsigned long flags)
Top level input values:
mpol_param: The field which collects information about the mempolicy
returned to userspace.
addr: if MPOL_F_ADDR is passed in `flags`, this address will be
used to return the mempolicy details of the vma the
address belongs to
flags: if MPOL_F_ADDR, return mempolicy info vma containing addr
else, returns task mempolicy information
Input values include the following fields of mpol_param:
pol_maxnodes: if pol_nodes is set, must describe max number of nodes
to be copied to pol_nodes
pol_nodes: if set, the nodemask of the policy returned here
Output values include the following fields of mpol_param:
mode: mempolicy mode
mode_flags: mempolicy mode flags
home_node: policy home node will be returned here, or -1 if not.
pol_nodes: if set, the nodemask for the mempolicy
MPOL_F_NODE has been dropped from get_mempolicy2 (EINVAL).
MPOL_F_NODE originally allowed for either the ability to acquire
the node the page of `addr` is presently allocated on - or, if
addr is not provided and the policy mode is MPOL_INTERLEAVE - it
would return the node that would be used for the next allocation.
Neither of these capabilities were pulled forward into get_mempolicy2
a) both are still possible to acquire via get_mempolicy()
b) both of pieces of data are racey by definition and have not
proven useful.
c) Should such a use be identified, it can be easily added back
into mempolicy2 as an extension.
MPOL_F_MEMS_ALLOWED has been dropped from get_mempolicy2 (EINVAL).
MPOL_F_MEMS_ALLOWED originally returned the task->mems_allowed
nodemask in the nodemask parameter instead of the task or vma
nodemask.
a) this is still accessible from get_mempolicy()
b) task->mems_allowed is not technically part of mempolicy,
though it is related.
c) should this warrant bringing forward (in the scenario
where get_mempolicy is deprecated), it can be added as
an explicit extension. Or more smartly: implemented as
its own syscall.
Suggested-by: Michal Hocko <mhocko@suse.com>
Signed-off-by: Gregory Price <gregory.price@memverge.com>
Acked-by: Geert Uytterhoeven <geert@linux-m68k.org>
---
.../admin-guide/mm/numa_memory_policy.rst | 11 ++++-
arch/alpha/kernel/syscalls/syscall.tbl | 1 +
arch/arm/tools/syscall.tbl | 1 +
arch/arm64/include/asm/unistd.h | 2 +-
arch/arm64/include/asm/unistd32.h | 2 +
arch/m68k/kernel/syscalls/syscall.tbl | 1 +
arch/microblaze/kernel/syscalls/syscall.tbl | 1 +
arch/mips/kernel/syscalls/syscall_n32.tbl | 1 +
arch/mips/kernel/syscalls/syscall_o32.tbl | 1 +
arch/parisc/kernel/syscalls/syscall.tbl | 1 +
arch/powerpc/kernel/syscalls/syscall.tbl | 1 +
arch/s390/kernel/syscalls/syscall.tbl | 1 +
arch/sh/kernel/syscalls/syscall.tbl | 1 +
arch/sparc/kernel/syscalls/syscall.tbl | 1 +
arch/x86/entry/syscalls/syscall_32.tbl | 1 +
arch/x86/entry/syscalls/syscall_64.tbl | 1 +
arch/xtensa/kernel/syscalls/syscall.tbl | 1 +
include/linux/syscalls.h | 2 +
include/uapi/asm-generic/unistd.h | 4 +-
kernel/sys_ni.c | 1 +
mm/mempolicy.c | 42 +++++++++++++++++++
.../arch/mips/entry/syscalls/syscall_n64.tbl | 1 +
.../arch/powerpc/entry/syscalls/syscall.tbl | 1 +
.../perf/arch/s390/entry/syscalls/syscall.tbl | 1 +
.../arch/x86/entry/syscalls/syscall_64.tbl | 1 +
25 files changed, 79 insertions(+), 3 deletions(-)
diff --git a/Documentation/admin-guide/mm/numa_memory_policy.rst b/Documentation/admin-guide/mm/numa_memory_policy.rst
index 62a4ea14c646..a2ff6e89e48b 100644
--- a/Documentation/admin-guide/mm/numa_memory_policy.rst
+++ b/Documentation/admin-guide/mm/numa_memory_policy.rst
@@ -454,11 +454,20 @@ Get [Task] Memory Policy or Related Information::
long get_mempolicy(int *mode,
const unsigned long *nmask, unsigned long maxnode,
void *addr, int flags);
+ long get_mempolicy2(struct mpol_param *param, size_t size,
+ unsigned long addr, unsigned long flags);
Queries the "task/process memory policy" of the calling task, or the
policy or location of a specified virtual address, depending on the
'flags' argument.
+get_mempolicy2() is an extended version of get_mempolicy() capable of
+acquiring extended information about a mempolicy, including those
+that can only be set via set_mempolicy2() or mbind2().
+
+MPOL_F_NODE functionality has been removed from get_mempolicy2(),
+but can still be accessed via get_mempolicy().
+
See the get_mempolicy(2) man page for more details
@@ -501,7 +510,7 @@ Extended Mempolicy Arguments::
The extended mempolicy argument structure is defined to allow the mempolicy
interfaces future extensibility without the need for additional system calls.
-Extended interfaces (set_mempolicy2) use this argument structure.
+Extended interfaces (set_mempolicy2 and get_mempolicy2) use this structure.
The core arguments (mode, mode_flags, pol_nodes, and pol_maxnodes) apply to
all interfaces relative to their non-extended counterparts. Each additional
diff --git a/arch/alpha/kernel/syscalls/syscall.tbl b/arch/alpha/kernel/syscalls/syscall.tbl
index 0dc288a1118a..0301a8b0a262 100644
--- a/arch/alpha/kernel/syscalls/syscall.tbl
+++ b/arch/alpha/kernel/syscalls/syscall.tbl
@@ -497,3 +497,4 @@
565 common futex_wait sys_futex_wait
566 common futex_requeue sys_futex_requeue
567 common set_mempolicy2 sys_set_mempolicy2
+568 common get_mempolicy2 sys_get_mempolicy2
diff --git a/arch/arm/tools/syscall.tbl b/arch/arm/tools/syscall.tbl
index 50172ec0e1f5..771a33446e8e 100644
--- a/arch/arm/tools/syscall.tbl
+++ b/arch/arm/tools/syscall.tbl
@@ -471,3 +471,4 @@
455 common futex_wait sys_futex_wait
456 common futex_requeue sys_futex_requeue
457 common set_mempolicy2 sys_set_mempolicy2
+458 common get_mempolicy2 sys_get_mempolicy2
diff --git a/arch/arm64/include/asm/unistd.h b/arch/arm64/include/asm/unistd.h
index 298313d2e0af..b63f870debaf 100644
--- a/arch/arm64/include/asm/unistd.h
+++ b/arch/arm64/include/asm/unistd.h
@@ -39,7 +39,7 @@
#define __ARM_NR_compat_set_tls (__ARM_NR_COMPAT_BASE + 5)
#define __ARM_NR_COMPAT_END (__ARM_NR_COMPAT_BASE + 0x800)
-#define __NR_compat_syscalls 458
+#define __NR_compat_syscalls 459
#endif
#define __ARCH_WANT_SYS_CLONE
diff --git a/arch/arm64/include/asm/unistd32.h b/arch/arm64/include/asm/unistd32.h
index cee8d669c342..f8d01007aee0 100644
--- a/arch/arm64/include/asm/unistd32.h
+++ b/arch/arm64/include/asm/unistd32.h
@@ -921,6 +921,8 @@ __SYSCALL(__NR_futex_wait, sys_futex_wait)
__SYSCALL(__NR_futex_requeue, sys_futex_requeue)
#define __NR_set_mempolicy2 457
__SYSCALL(__NR_set_mempolicy2, sys_set_mempolicy2)
+#define __NR_get_mempolicy2 458
+__SYSCALL(__NR_get_mempolicy2, sys_get_mempolicy2)
/*
* Please add new compat syscalls above this comment and update
diff --git a/arch/m68k/kernel/syscalls/syscall.tbl b/arch/m68k/kernel/syscalls/syscall.tbl
index 839d90c535f2..048a409e684c 100644
--- a/arch/m68k/kernel/syscalls/syscall.tbl
+++ b/arch/m68k/kernel/syscalls/syscall.tbl
@@ -457,3 +457,4 @@
455 common futex_wait sys_futex_wait
456 common futex_requeue sys_futex_requeue
457 common set_mempolicy2 sys_set_mempolicy2
+458 common get_mempolicy2 sys_get_mempolicy2
diff --git a/arch/microblaze/kernel/syscalls/syscall.tbl b/arch/microblaze/kernel/syscalls/syscall.tbl
index 567c8b883735..327b01bd6793 100644
--- a/arch/microblaze/kernel/syscalls/syscall.tbl
+++ b/arch/microblaze/kernel/syscalls/syscall.tbl
@@ -463,3 +463,4 @@
455 common futex_wait sys_futex_wait
456 common futex_requeue sys_futex_requeue
457 common set_mempolicy2 sys_set_mempolicy2
+458 common get_mempolicy2 sys_get_mempolicy2
diff --git a/arch/mips/kernel/syscalls/syscall_n32.tbl b/arch/mips/kernel/syscalls/syscall_n32.tbl
index cc0640e16f2f..921d58e1da23 100644
--- a/arch/mips/kernel/syscalls/syscall_n32.tbl
+++ b/arch/mips/kernel/syscalls/syscall_n32.tbl
@@ -396,3 +396,4 @@
455 n32 futex_wait sys_futex_wait
456 n32 futex_requeue sys_futex_requeue
457 n32 set_mempolicy2 sys_set_mempolicy2
+458 n32 get_mempolicy2 sys_get_mempolicy2
diff --git a/arch/mips/kernel/syscalls/syscall_o32.tbl b/arch/mips/kernel/syscalls/syscall_o32.tbl
index f7262fde98d9..9271c83c9993 100644
--- a/arch/mips/kernel/syscalls/syscall_o32.tbl
+++ b/arch/mips/kernel/syscalls/syscall_o32.tbl
@@ -445,3 +445,4 @@
455 o32 futex_wait sys_futex_wait
456 o32 futex_requeue sys_futex_requeue
457 o32 set_mempolicy2 sys_set_mempolicy2
+458 o32 get_mempolicy2 sys_get_mempolicy2
diff --git a/arch/parisc/kernel/syscalls/syscall.tbl b/arch/parisc/kernel/syscalls/syscall.tbl
index e10f0e8bd064..0654f3f89fc7 100644
--- a/arch/parisc/kernel/syscalls/syscall.tbl
+++ b/arch/parisc/kernel/syscalls/syscall.tbl
@@ -456,3 +456,4 @@
455 common futex_wait sys_futex_wait
456 common futex_requeue sys_futex_requeue
457 common set_mempolicy2 sys_set_mempolicy2
+458 common get_mempolicy2 sys_get_mempolicy2
diff --git a/arch/powerpc/kernel/syscalls/syscall.tbl b/arch/powerpc/kernel/syscalls/syscall.tbl
index 4f03f5f42b78..ac11d2064e7a 100644
--- a/arch/powerpc/kernel/syscalls/syscall.tbl
+++ b/arch/powerpc/kernel/syscalls/syscall.tbl
@@ -544,3 +544,4 @@
455 common futex_wait sys_futex_wait
456 common futex_requeue sys_futex_requeue
457 common set_mempolicy2 sys_set_mempolicy2
+458 common get_mempolicy2 sys_get_mempolicy2
diff --git a/arch/s390/kernel/syscalls/syscall.tbl b/arch/s390/kernel/syscalls/syscall.tbl
index f98dadc2e9df..1cdcafe1ccca 100644
--- a/arch/s390/kernel/syscalls/syscall.tbl
+++ b/arch/s390/kernel/syscalls/syscall.tbl
@@ -460,3 +460,4 @@
455 common futex_wait sys_futex_wait sys_futex_wait
456 common futex_requeue sys_futex_requeue sys_futex_requeue
457 common set_mempolicy2 sys_set_mempolicy2 sys_set_mempolicy2
+458 common get_mempolicy2 sys_get_mempolicy2 sys_get_mempolicy2
diff --git a/arch/sh/kernel/syscalls/syscall.tbl b/arch/sh/kernel/syscalls/syscall.tbl
index f47ba9f2d05d..f71742024c29 100644
--- a/arch/sh/kernel/syscalls/syscall.tbl
+++ b/arch/sh/kernel/syscalls/syscall.tbl
@@ -460,3 +460,4 @@
455 common futex_wait sys_futex_wait
456 common futex_requeue sys_futex_requeue
457 common set_mempolicy2 sys_set_mempolicy2
+458 common get_mempolicy2 sys_get_mempolicy2
diff --git a/arch/sparc/kernel/syscalls/syscall.tbl b/arch/sparc/kernel/syscalls/syscall.tbl
index 53fb16616728..2fbf5dbe0620 100644
--- a/arch/sparc/kernel/syscalls/syscall.tbl
+++ b/arch/sparc/kernel/syscalls/syscall.tbl
@@ -503,3 +503,4 @@
455 common futex_wait sys_futex_wait
456 common futex_requeue sys_futex_requeue
457 common set_mempolicy2 sys_set_mempolicy2
+458 common get_mempolicy2 sys_get_mempolicy2
diff --git a/arch/x86/entry/syscalls/syscall_32.tbl b/arch/x86/entry/syscalls/syscall_32.tbl
index 4b4dc41b24ee..0af813b9a118 100644
--- a/arch/x86/entry/syscalls/syscall_32.tbl
+++ b/arch/x86/entry/syscalls/syscall_32.tbl
@@ -462,3 +462,4 @@
455 i386 futex_wait sys_futex_wait
456 i386 futex_requeue sys_futex_requeue
457 i386 set_mempolicy2 sys_set_mempolicy2
+458 i386 get_mempolicy2 sys_get_mempolicy2
diff --git a/arch/x86/entry/syscalls/syscall_64.tbl b/arch/x86/entry/syscalls/syscall_64.tbl
index 1bc2190bec27..0b777876fc15 100644
--- a/arch/x86/entry/syscalls/syscall_64.tbl
+++ b/arch/x86/entry/syscalls/syscall_64.tbl
@@ -379,6 +379,7 @@
455 common futex_wait sys_futex_wait
456 common futex_requeue sys_futex_requeue
457 common set_mempolicy2 sys_set_mempolicy2
+458 common get_mempolicy2 sys_get_mempolicy2
#
# Due to a historical design error, certain syscalls are numbered differently
diff --git a/arch/xtensa/kernel/syscalls/syscall.tbl b/arch/xtensa/kernel/syscalls/syscall.tbl
index e26dc89399eb..4536c9a4227d 100644
--- a/arch/xtensa/kernel/syscalls/syscall.tbl
+++ b/arch/xtensa/kernel/syscalls/syscall.tbl
@@ -428,3 +428,4 @@
455 common futex_wait sys_futex_wait
456 common futex_requeue sys_futex_requeue
457 common set_mempolicy2 sys_set_mempolicy2
+458 common get_mempolicy2 sys_get_mempolicy2
diff --git a/include/linux/syscalls.h b/include/linux/syscalls.h
index b37ea6715456..c4dc5069bae7 100644
--- a/include/linux/syscalls.h
+++ b/include/linux/syscalls.h
@@ -821,6 +821,8 @@ asmlinkage long sys_get_mempolicy(int __user *policy,
unsigned long __user *nmask,
unsigned long maxnode,
unsigned long addr, unsigned long flags);
+asmlinkage long sys_get_mempolicy2(struct mpol_param __user *param, size_t size,
+ unsigned long addr, unsigned long flags);
asmlinkage long sys_set_mempolicy(int mode, const unsigned long __user *nmask,
unsigned long maxnode);
asmlinkage long sys_set_mempolicy2(struct mpol_param __user *param, size_t size,
diff --git a/include/uapi/asm-generic/unistd.h b/include/uapi/asm-generic/unistd.h
index 55486aba099f..719accc731db 100644
--- a/include/uapi/asm-generic/unistd.h
+++ b/include/uapi/asm-generic/unistd.h
@@ -830,9 +830,11 @@ __SYSCALL(__NR_futex_wait, sys_futex_wait)
__SYSCALL(__NR_futex_requeue, sys_futex_requeue)
#define __NR_set_mempolicy2 457
__SYSCALL(__NR_set_mempolicy2, sys_set_mempolicy2)
+#define __NR_get_mempolicy2 458
+__SYSCALL(__NR_get_mempolicy2, sys_get_mempolicy2)
#undef __NR_syscalls
-#define __NR_syscalls 458
+#define __NR_syscalls 459
/*
* 32 bit systems traditionally used different
diff --git a/kernel/sys_ni.c b/kernel/sys_ni.c
index fa1373c8bff8..6afbd3a41319 100644
--- a/kernel/sys_ni.c
+++ b/kernel/sys_ni.c
@@ -188,6 +188,7 @@ COND_SYSCALL(process_mrelease);
COND_SYSCALL(remap_file_pages);
COND_SYSCALL(mbind);
COND_SYSCALL(get_mempolicy);
+COND_SYSCALL(get_mempolicy2);
COND_SYSCALL(set_mempolicy);
COND_SYSCALL(set_mempolicy2);
COND_SYSCALL(migrate_pages);
diff --git a/mm/mempolicy.c b/mm/mempolicy.c
index 84d877195deb..0b2e31d8636d 100644
--- a/mm/mempolicy.c
+++ b/mm/mempolicy.c
@@ -1873,6 +1873,48 @@ SYSCALL_DEFINE5(get_mempolicy, int __user *, policy,
return kernel_get_mempolicy(policy, nmask, maxnode, addr, flags);
}
+SYSCALL_DEFINE4(get_mempolicy2, struct mpol_param __user *, uparam, size_t, usize,
+ unsigned long, addr, unsigned long, flags)
+{
+ struct mpol_param kparam;
+ struct mempolicy_param mparam;
+ int err;
+ nodemask_t policy_nodemask;
+ unsigned long __user *nodes_ptr;
+
+ if (flags & ~(MPOL_F_ADDR))
+ return -EINVAL;
+
+ /* initialize any memory liable to be copied to userland */
+ memset(&mparam, 0, sizeof(mparam));
+
+ err = copy_struct_from_user(&kparam, sizeof(kparam), uparam, usize);
+ if (err)
+ return -EINVAL;
+
+ mparam.policy_nodes = kparam.pol_nodes ? &policy_nodemask : NULL;
+ if (flags & MPOL_F_ADDR)
+ err = do_get_vma_mempolicy(untagged_addr(addr), NULL, &mparam);
+ else
+ err = do_get_task_mempolicy(&mparam, NULL);
+
+ if (err)
+ return err;
+
+ kparam.mode = mparam.mode;
+ kparam.mode_flags = mparam.mode_flags;
+ kparam.home_node = mparam.home_node;
+ if (kparam.pol_nodes) {
+ nodes_ptr = u64_to_user_ptr(kparam.pol_nodes);
+ err = copy_nodes_to_user(nodes_ptr, kparam.pol_maxnodes,
+ mparam.policy_nodes);
+ if (err)
+ return err;
+ }
+
+ return copy_to_user(uparam, &kparam, usize) ? -EFAULT : 0;
+}
+
bool vma_migratable(struct vm_area_struct *vma)
{
if (vma->vm_flags & (VM_IO | VM_PFNMAP))
diff --git a/tools/perf/arch/mips/entry/syscalls/syscall_n64.tbl b/tools/perf/arch/mips/entry/syscalls/syscall_n64.tbl
index bb1351df51d9..c34c6877379e 100644
--- a/tools/perf/arch/mips/entry/syscalls/syscall_n64.tbl
+++ b/tools/perf/arch/mips/entry/syscalls/syscall_n64.tbl
@@ -372,3 +372,4 @@
455 n64 futex_wait sys_futex_wait
456 n64 futex_requeue sys_futex_requeue
457 n64 set_mempolicy2 sys_set_mempolicy2
+458 n64 get_mempolicy2 sys_get_mempolicy2
diff --git a/tools/perf/arch/powerpc/entry/syscalls/syscall.tbl b/tools/perf/arch/powerpc/entry/syscalls/syscall.tbl
index 4f03f5f42b78..ac11d2064e7a 100644
--- a/tools/perf/arch/powerpc/entry/syscalls/syscall.tbl
+++ b/tools/perf/arch/powerpc/entry/syscalls/syscall.tbl
@@ -544,3 +544,4 @@
455 common futex_wait sys_futex_wait
456 common futex_requeue sys_futex_requeue
457 common set_mempolicy2 sys_set_mempolicy2
+458 common get_mempolicy2 sys_get_mempolicy2
diff --git a/tools/perf/arch/s390/entry/syscalls/syscall.tbl b/tools/perf/arch/s390/entry/syscalls/syscall.tbl
index f98dadc2e9df..1cdcafe1ccca 100644
--- a/tools/perf/arch/s390/entry/syscalls/syscall.tbl
+++ b/tools/perf/arch/s390/entry/syscalls/syscall.tbl
@@ -460,3 +460,4 @@
455 common futex_wait sys_futex_wait sys_futex_wait
456 common futex_requeue sys_futex_requeue sys_futex_requeue
457 common set_mempolicy2 sys_set_mempolicy2 sys_set_mempolicy2
+458 common get_mempolicy2 sys_get_mempolicy2 sys_get_mempolicy2
diff --git a/tools/perf/arch/x86/entry/syscalls/syscall_64.tbl b/tools/perf/arch/x86/entry/syscalls/syscall_64.tbl
index 21f2579679d4..edf338f32645 100644
--- a/tools/perf/arch/x86/entry/syscalls/syscall_64.tbl
+++ b/tools/perf/arch/x86/entry/syscalls/syscall_64.tbl
@@ -379,6 +379,7 @@
455 common futex_wait sys_futex_wait
456 common futex_requeue sys_futex_requeue
457 common set_mempolicy2 sys_set_mempolicy2
+458 common get_mempolicy2 sys_get_mempolicy2
#
# Due to a historical design error, certain syscalls are numbered differently
--
2.39.1
^ permalink raw reply related
page: next (older) | prev (newer) | latest
- recent:[subjects (threaded)|topics (new)|topics (active)]
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox