* Re: [PATCH v2 1/1] mm/madvise: add MADV_F_COLLAPSE_LIGHT to process_madvise()
From: Lance Yang @ 2024-01-19 1:46 UTC (permalink / raw)
To: Zach O'Keefe
Cc: Michal Hocko, akpm, david, songmuchun, shy828301, peterx,
mknyszek, minchan, linux-mm, linux-kernel, linux-api
In-Reply-To: <CAAa6QmTN2B-JAO=38A09hMtUp=srLiUfs=sDbck7Chkr=W-dCw@mail.gmail.com>
On Thu, Jan 18, 2024 at 10:59 PM Zach O'Keefe <zokeefe@google.com> wrote:
>
> On Thu, Jan 18, 2024 at 5:43 AM Michal Hocko <mhocko@suse.com> wrote:
> >
> > Dang, forgot to cc linux-api...
> >
> > On Thu 18-01-24 14:40:19, Michal Hocko wrote:
> > > On Thu 18-01-24 20:03:46, Lance Yang wrote:
> > > [...]
> > >
> > > before we discuss the semantic, let's focus on the usecase.
> > >
> > > > Use Cases
> > > >
> > > > An immediate user of this new functionality is the Go runtime heap allocator
> > > > that manages memory in hugepage-sized chunks. In the past, whether it was a
> > > > newly allocated chunk through mmap() or a reused chunk released by
> > > > madvise(MADV_DONTNEED), the allocator attempted to eagerly back memory with
> > > > huge pages using madvise(MADV_HUGEPAGE)[2] and madvise(MADV_COLLAPSE)[3]
> > > > respectively. However, both approaches resulted in performance issues; for
> > > > both scenarios, there could be entries into direct reclaim and/or compaction,
> > > > leading to unpredictable stalls[4]. Now, the allocator can confidently use
> > > > process_madvise(MADV_F_COLLAPSE_LIGHT) to attempt the allocation of huge pages.
>
> Aside: The thought was a MADV_F_COLLAPSE_LIGHT _flag_; so it'd be
> process_madvise(..., MADV_COLLAPSE, MADV_F_COLLAPSE_LIGHT)
I apologize for the misunderstanding. I will provide the correct implementation
in version 3.
BR,
Lance
>
> > > IIUC the primary reason is the cost of the huge page allocation which
> > > can be really high if the memory is heavily fragmented and it is called
> > > synchronously from the process directly, correct? Can that be worked
> > > around by process_madvise and performing the operation from a different
> > > context? Are there any other reasons to have a different mode?
> > >
> > > I mean I can think of a more relaxed (opportunistic) MADV_COLLAPSE -
> > > e.g. non blocking one to make sure that the caller doesn't really block
> > > on resource contention (be it locks or memory availability) because that
> > > matches our non-blocking interface in other areas but having a LIGHT
> > > operation sounds really vague and the exact semantic would be
> > > implementation specific and might change over time. Non-blocking has a
> > > clear semantic but it is not really clear whether that is what you
> > > really need/want.
>
> IIUC, usecase from Go is unbounded latency due to sync compaction in a
> context where the latency is unacceptable. Working w/ them to
> understand how things can be improved -- it's possible the changes can
> occur entirely on their side, w/o any additional kernel support.
>
> The non-blocking case awkwardly sits between MADV_COLLAPSE today, and
> khugepaged; esp when common case is that the allocation can probably
> be satisfied in fast path.
>
> The suggestion for something like "LIGHT" was intentionally vague
> because it could allow for other optimizations / changes down the
> line, as you point out. I think that might be a win, vs tying to a
> specific optimization (e.g. like a MADV_F_COLLAPSE_NODEFRAG). But I
> could be alone on that front, given the design of
> /sys/kernel/mm/transparent_hugepage.
>
> But circling back, I agree w/ you that the first order of business is to
> iron out a real usecase. As of right now, it's not clear something
> like this is required or helpful.
>
> Thanks,
> Zach
>
>
>
>
> > > > [1] https://github.com/torvalds/linux/commit/7d8faaf155454f8798ec56404faca29a82689c77
> > > > [2] https://github.com/golang/go/commit/8fa9e3beee8b0e6baa7333740996181268b60a3a
> > > > [3] https://github.com/golang/go/commit/9f9bb26880388c5bead158e9eca3be4b3a9bd2af
> > > > [4] https://github.com/golang/go/issues/63334
> > > >
> > > > [v1] https://lore.kernel.org/lkml/20240117050217.43610-1-ioworker0@gmail.com/
> > > --
> > > Michal Hocko
> > > SUSE Labs
> >
> > --
> > Michal Hocko
> > SUSE Labs
^ permalink raw reply
* Re: [PATCH v2 1/1] mm/madvise: add MADV_F_COLLAPSE_LIGHT to process_madvise()
From: Lance Yang @ 2024-01-19 2:37 UTC (permalink / raw)
To: Yang Shi
Cc: Zach O'Keefe, Michal Hocko, akpm, david, songmuchun, peterx,
mknyszek, minchan, linux-mm, linux-kernel, linux-api
In-Reply-To: <CAHbLzkphG+kmLvYQFB6WZuuGadPsRvsH4Os9CwnxvizKAfwuvQ@mail.gmail.com>
Hey Yang,
Thanks for taking the time to review!
On Fri, Jan 19, 2024 at 3:00 AM Yang Shi <shy828301@gmail.com> wrote:
>
> On Thu, Jan 18, 2024 at 6:59 AM Zach O'Keefe <zokeefe@google.com> wrote:
> >
> > On Thu, Jan 18, 2024 at 5:43 AM Michal Hocko <mhocko@suse.com> wrote:
> > >
> > > Dang, forgot to cc linux-api...
> > >
> > > On Thu 18-01-24 14:40:19, Michal Hocko wrote:
> > > > On Thu 18-01-24 20:03:46, Lance Yang wrote:
> > > > [...]
> > > >
> > > > before we discuss the semantic, let's focus on the usecase.
> > > >
> > > > > Use Cases
> > > > >
> > > > > An immediate user of this new functionality is the Go runtime heap allocator
> > > > > that manages memory in hugepage-sized chunks. In the past, whether it was a
> > > > > newly allocated chunk through mmap() or a reused chunk released by
> > > > > madvise(MADV_DONTNEED), the allocator attempted to eagerly back memory with
> > > > > huge pages using madvise(MADV_HUGEPAGE)[2] and madvise(MADV_COLLAPSE)[3]
> > > > > respectively. However, both approaches resulted in performance issues; for
> > > > > both scenarios, there could be entries into direct reclaim and/or compaction,
> > > > > leading to unpredictable stalls[4]. Now, the allocator can confidently use
> > > > > process_madvise(MADV_F_COLLAPSE_LIGHT) to attempt the allocation of huge pages.
> >
> > Aside: The thought was a MADV_F_COLLAPSE_LIGHT _flag_; so it'd be
> > process_madvise(..., MADV_COLLAPSE, MADV_F_COLLAPSE_LIGHT)
> >
> > > > IIUC the primary reason is the cost of the huge page allocation which
> > > > can be really high if the memory is heavily fragmented and it is called
> > > > synchronously from the process directly, correct? Can that be worked
> > > > around by process_madvise and performing the operation from a different
> > > > context? Are there any other reasons to have a different mode?
> > > >
> > > > I mean I can think of a more relaxed (opportunistic) MADV_COLLAPSE -
> > > > e.g. non blocking one to make sure that the caller doesn't really block
> > > > on resource contention (be it locks or memory availability) because that
> > > > matches our non-blocking interface in other areas but having a LIGHT
> > > > operation sounds really vague and the exact semantic would be
> > > > implementation specific and might change over time. Non-blocking has a
> > > > clear semantic but it is not really clear whether that is what you
> > > > really need/want.
> >
> > IIUC, usecase from Go is unbounded latency due to sync compaction in a
> > context where the latency is unacceptable. Working w/ them to
> > understand how things can be improved -- it's possible the changes can
> > occur entirely on their side, w/o any additional kernel support.
> >
> > The non-blocking case awkwardly sits between MADV_COLLAPSE today, and
> > khugepaged; esp when common case is that the allocation can probably
> > be satisfied in fast path.
> >
> > The suggestion for something like "LIGHT" was intentionally vague
> > because it could allow for other optimizations / changes down the
> > line, as you point out. I think that might be a win, vs tying to a
> > specific optimization (e.g. like a MADV_F_COLLAPSE_NODEFRAG). But I
> > could be alone on that front, given the design of
> > /sys/kernel/mm/transparent_hugepage.
>
> Per the description Go marks the address spaces with MADV_HUGEPAGE. It
> means the application really wants to have huge page back the address
> space so kernel will try as hard as possible to get huge page. This is
> the default behavior of MADV_HUGEPAGE. If they don't want to enter
> direct reclaim, they can configure the defrag mode to "defer", which
> means no direct reclaim and wakeup kswapd and kcompactd, and rely on
> khugepaged to install huge page later on. But this mode is not
> supported by khugepaged defrag, so MADV_COLLAPSE may not support it
> (IIRC MADV_COLLAPSE uses khugepaged defrag mode). Or they can just not
> call MADV_HUGEPAGE and leave the decision to the users, IIRC Java does
> so (specifying a flag to indicate use huge page or not by the users).
Thank you for providing insights into the Go use cases with MADV_HUGEPAGE
and the configuration options for defrag mode.
Considering the limitations with the "defer" mode, it becomes apparent
that there
is a gap in addressing scenarios where an application desires a lighter-weight
alternative to MADV_HUGEPAGE.
MADV_F_COLLAPSE_LIGHT aims to fill this gap by providing a more flexible and
opportunistic approach, catering to applications in latency-sensitive
environments
that seek performance improvements with huge pages but prefer to avoid direct
reclaim and compaction. This option can serve as a valuable addition for users
who want more control over the behavior without the constraints of existing
configurations.
In the era of cloud-native computing, it's challenging for users to be
aware of the
THP configurations on all nodes in a cluster, let alone have
fine-grained control
over them. Simply disabling the use of huge pages due to concerns
about potential
direct reclamation and compaction may be regrettable, as users are deprived of
the opportunity to experiment with large page allocations. However,
relying solely on
MADV_HUGEPAGE introduces the risk of unpredictable stalls, making it a trade-off
that users must carefully consider.
By introducing MADV_F_COLLAPSE_LIGHT, we offer users a more flexible and
controllable solution in cloud-native environments, allowing them to
better balance
performance requirements and resource management. This selectively lightweight
alternative is designed to provide users with more choices to better
meet the diverse
needs of different scenarios.
Thanks again for your review and your suggestion!
Lance
>
> >
> > But circling back, I agree w/ you that the first order of business is to
> > iron out a real usecase. As of right now, it's not clear something
> > like this is required or helpful.
> >
> > Thanks,
> > Zach
> >
> >
> >
> >
> > > > > [1] https://github.com/torvalds/linux/commit/7d8faaf155454f8798ec56404faca29a82689c77
> > > > > [2] https://github.com/golang/go/commit/8fa9e3beee8b0e6baa7333740996181268b60a3a
> > > > > [3] https://github.com/golang/go/commit/9f9bb26880388c5bead158e9eca3be4b3a9bd2af
> > > > > [4] https://github.com/golang/go/issues/63334
> > > > >
> > > > > [v1] https://lore.kernel.org/lkml/20240117050217.43610-1-ioworker0@gmail.com/
> > > > --
> > > > Michal Hocko
> > > > SUSE Labs
> > >
> > > --
> > > Michal Hocko
> > > SUSE Labs
^ permalink raw reply
* 回复: [PATCH v4] selftests/move_mount_set_group:Make tests build with old libc
From: Huyadi @ 2024-01-19 2:57 UTC (permalink / raw)
To: 'Christian Brauner'
Cc: linux-api@vger.kernel.org, linux-kernel@vger.kernel.org,
linux-security-module@vger.kernel.org,
linux-kselftest@vger.kernel.org, 514118380@qq.com,
jmorris@namei.org, serge@hallyn.com, shuah@kernel.org,
mathieu.desnoyers@efficios.com, mic@digikod.net
In-Reply-To: <20240118-sezieren-neurologie-6690110057ca@brauner>
>On Thu, 11 Jan 2024 19:32:29 +0800, Hu Yadi wrote:
>> Replace SYS_<syscall> with __NR_<syscall>. Using the __NR_<syscall>
>> notation, provided by UAPI, is useful to build tests on systems
>> without the SYS_<syscall> definitions.
>>
>> Replace SYS_move_mount with __NR_move_mount
>>
>> Similar changes: commit 87129ef13603 ("selftests/landlock: Make tests
>> build with old libc")
>>
>> [...]
>
>Applied to the vfs.misc branch of the vfs/vfs.git tree.
>Patches in the vfs.misc branch should appear in linux-next soon.
>
>Please report any outstanding bugs that were missed during review in a new review to the original patch series allowing us to drop it.
>
>It's encouraged to provide Acked-bys and Reviewed-bys even though the patch has now been applied. If possible patch trailers will be updated.
>
>Note that commit hashes shown below are subject to change due to rebase, trailer updates or similar. If in doubt, please check the listed branch.
>
>tree: https://git.kernel.org/pub/scm/linux/kernel/git/vfs/vfs.git
>branch: vfs.misc
>
May I take the liberty to ask why I don't see patch applied to above branch?
and below commit also shows error:
Bad object id: 0778b0a1a8d2
>[1/1] selftests/move_mount_set_group:Make tests build with old libc
> https://git.kernel.org/vfs/vfs/c/0778b0a1a8d2
^ permalink raw reply
* [PATCH v3 1/1] mm/madvise: add MADV_F_COLLAPSE_LIGHT flag to process_madvise()
From: Lance Yang @ 2024-01-19 11:51 UTC (permalink / raw)
To: akpm
Cc: zokeefe, david, songmuchun, shy828301, peterx, mknyszek, minchan,
mhocko, linux-mm, linux-api, linux-kernel, Lance Yang
This idea was inspired by MADV_COLLAPSE introduced by Zach O'Keefe[1].
The MADV_F_COLLAPSE_LIGHT flag provides a more relaxed (opportunistic)
and lightweight alternative to the default behavior of MADV_COLLAPSE
advice, specifically designed to avoid direct reclaim and/or compaction,
and quickly fail on allocation errors.
In the era of cloud-native computing, it's challenging for users to be
aware of the THP configurations on all nodes in a cluster, let alone have
fine-grained control over them. Simply disabling the use of huge pages due
to concerns about potential direct reclaim and/or compaction may be
regrettable, as users are deprived of the opportunity to experiment with
huge page allocations. However, relying solely on MADV_HUGEPAGE introduces
the risk of unpredictable stalls, making it a trade-off that users must
carefully consider.
With the introduction of MADV_COLLAPSE into the kernel, it is not governed
by the defrag mode. MADV_COLLAPSE offers the potential for more fine-grained
control over the huge page allocation strategy.
Semantics
This call is independent of the system-wide THP sysfs settings, but will
fail for memory marked VM_NOHUGEPAGE. If the ranges provided span
multiple VMAs, the semantics of the collapse over each VMA is independent
from the others. This implies a hugepage cannot cross a VMA boundary. If
collapse of a given hugepage-aligned/sized region fails, the operation may
continue to attempt collapsing the remainder of memory specified.
The memory ranges provided must be page-aligned, but are not required to
be hugepage-aligned. If the memory ranges are not hugepage-aligned, the
start/end of the range will be clamped to the first/last hugepage-aligned
address covered by said range. The memory ranges must span at least one
hugepage-sized region.
All non-resident pages covered by the range will first be
swapped/faulted-in, before being internally copied onto a freshly
allocated hugepage. Unmapped pages will have their data directly
initialized to 0 in the new hugepage. However, for every eligible
hugepage aligned/sized region to-be collapsed, at least one page must
currently be backed by memory (a PMD covering the address range must
already exist).
Allocation for the new hugepage will not enter direct reclaim and/or
compaction, quickly failing if allocation fails. When the system has
multiple NUMA nodes, the hugepage will be allocated from the node providing
the most native pages. This operation operates on the current state of the
specified process and makes no persistent changes or guarantees on how pages
will be mapped, constructed, or faulted in the future.
[1] https://github.com/torvalds/linux/commit/7d8faaf155454f8798ec56404faca29a82689c77
[v1] https://lore.kernel.org/lkml/20240117050217.43610-1-ioworker0@gmail.com/
[v2] https://lore.kernel.org/lkml/20240118120347.61817-1-ioworker0@gmail.com/
Signed-off-by: Lance Yang <ioworker0@gmail.com>
Suggested-by: Zach O'Keefe <zokeefe@google.com>
Suggested-by: David Hildenbrand <david@redhat.com>
---
V3 -> V2: Add MADV_F_COLLAPSE_LIGHT flag
V1 -> V2: Treat process_madvise(MADV_F_COLLAPSE_LIGHT) as the lighter-weight
alternative to madvise(MADV_COLLAPSE)
arch/alpha/include/uapi/asm/mman.h | 4 ++
arch/mips/include/uapi/asm/mman.h | 4 ++
arch/parisc/include/uapi/asm/mman.h | 4 ++
arch/xtensa/include/uapi/asm/mman.h | 4 ++
include/linux/huge_mm.h | 5 ++-
include/uapi/asm-generic/mman-common.h | 4 ++
mm/khugepaged.c | 15 +++++--
mm/madvise.c | 45 ++++++++++++++------
tools/include/uapi/asm-generic/mman-common.h | 4 ++
9 files changed, 72 insertions(+), 17 deletions(-)
diff --git a/arch/alpha/include/uapi/asm/mman.h b/arch/alpha/include/uapi/asm/mman.h
index 763929e814e9..c9f973f86585 100644
--- a/arch/alpha/include/uapi/asm/mman.h
+++ b/arch/alpha/include/uapi/asm/mman.h
@@ -78,6 +78,10 @@
#define MADV_COLLAPSE 25 /* Synchronous hugepage collapse */
+/* process_madvise flags */
+#define MADV_F_NORMAL 0 /* the default flag */
+#define MADV_F_COLLAPSE_LIGHT 0x1 /* only for MADV_COLLAPSE, avoids direct reclaim and/or compaction */
+
/* compatibility flags */
#define MAP_FILE 0
diff --git a/arch/mips/include/uapi/asm/mman.h b/arch/mips/include/uapi/asm/mman.h
index c6e1fc77c996..aa25dfe58a44 100644
--- a/arch/mips/include/uapi/asm/mman.h
+++ b/arch/mips/include/uapi/asm/mman.h
@@ -105,6 +105,10 @@
#define MADV_COLLAPSE 25 /* Synchronous hugepage collapse */
+/* process_madvise flags */
+#define MADV_F_NORMAL 0 /* the default flag */
+#define MADV_F_COLLAPSE_LIGHT 0x1 /* only for MADV_COLLAPSE, avoids direct reclaim and/or compaction */
+
/* compatibility flags */
#define MAP_FILE 0
diff --git a/arch/parisc/include/uapi/asm/mman.h b/arch/parisc/include/uapi/asm/mman.h
index 68c44f99bc93..d9f38ff86efe 100644
--- a/arch/parisc/include/uapi/asm/mman.h
+++ b/arch/parisc/include/uapi/asm/mman.h
@@ -75,6 +75,10 @@
#define MADV_HWPOISON 100 /* poison a page for testing */
#define MADV_SOFT_OFFLINE 101 /* soft offline page for testing */
+/* process_madvise flags */
+#define MADV_F_NORMAL 0 /* the default flag */
+#define MADV_F_COLLAPSE_LIGHT 0x1 /* only for MADV_COLLAPSE, avoids direct reclaim and/or compaction */
+
/* compatibility flags */
#define MAP_FILE 0
diff --git a/arch/xtensa/include/uapi/asm/mman.h b/arch/xtensa/include/uapi/asm/mman.h
index 1ff0c858544f..f36399192f75 100644
--- a/arch/xtensa/include/uapi/asm/mman.h
+++ b/arch/xtensa/include/uapi/asm/mman.h
@@ -113,6 +113,10 @@
#define MADV_COLLAPSE 25 /* Synchronous hugepage collapse */
+/* process_madvise flags */
+#define MADV_F_NORMAL 0 /* the default flag */
+#define MADV_F_COLLAPSE_LIGHT 0x1 /* only for MADV_COLLAPSE, avoids direct reclaim and/or compaction */
+
/* compatibility flags */
#define MAP_FILE 0
diff --git a/include/linux/huge_mm.h b/include/linux/huge_mm.h
index 5adb86af35fc..bfa52c7f8c25 100644
--- a/include/linux/huge_mm.h
+++ b/include/linux/huge_mm.h
@@ -303,7 +303,7 @@ int hugepage_madvise(struct vm_area_struct *vma, unsigned long *vm_flags,
int advice);
int madvise_collapse(struct vm_area_struct *vma,
struct vm_area_struct **prev,
- unsigned long start, unsigned long end);
+ unsigned long start, unsigned long end, unsigned int flags);
void vma_adjust_trans_huge(struct vm_area_struct *vma, unsigned long start,
unsigned long end, long adjust_next);
spinlock_t *__pmd_trans_huge_lock(pmd_t *pmd, struct vm_area_struct *vma);
@@ -450,7 +450,8 @@ static inline int hugepage_madvise(struct vm_area_struct *vma,
static inline int madvise_collapse(struct vm_area_struct *vma,
struct vm_area_struct **prev,
- unsigned long start, unsigned long end)
+ unsigned long start, unsigned long end,
+ unsigned int flags)
{
return -EINVAL;
}
diff --git a/include/uapi/asm-generic/mman-common.h b/include/uapi/asm-generic/mman-common.h
index 6ce1f1ceb432..0e0fac5c9219 100644
--- a/include/uapi/asm-generic/mman-common.h
+++ b/include/uapi/asm-generic/mman-common.h
@@ -79,6 +79,10 @@
#define MADV_COLLAPSE 25 /* Synchronous hugepage collapse */
+/* process_madvise flags */
+#define MADV_F_NORMAL 0 /* the default flag */
+#define MADV_F_COLLAPSE_LIGHT 0x1 /* only for MADV_COLLAPSE, avoids direct reclaim and/or compaction */
+
/* compatibility flags */
#define MAP_FILE 0
diff --git a/mm/khugepaged.c b/mm/khugepaged.c
index 2b219acb528e..90c62c57a3f9 100644
--- a/mm/khugepaged.c
+++ b/mm/khugepaged.c
@@ -97,6 +97,8 @@ static struct kmem_cache *mm_slot_cache __ro_after_init;
struct collapse_control {
bool is_khugepaged;
+ unsigned int flags;
+
/* Num pages scanned per node */
u32 node_load[MAX_NUMNODES];
@@ -1058,10 +1060,16 @@ static int __collapse_huge_page_swapin(struct mm_struct *mm,
static int alloc_charge_hpage(struct page **hpage, struct mm_struct *mm,
struct collapse_control *cc)
{
- gfp_t gfp = (cc->is_khugepaged ? alloc_hugepage_khugepaged_gfpmask() :
- GFP_TRANSHUGE);
int node = hpage_collapse_find_target_node(cc);
struct folio *folio;
+ gfp_t gfp;
+
+ if (cc->is_khugepaged)
+ gfp = alloc_hugepage_khugepaged_gfpmask();
+ else
+ gfp = (cc->flags & MADV_F_COLLAPSE_LIGHT ?
+ GFP_TRANSHUGE_LIGHT :
+ GFP_TRANSHUGE);
if (!hpage_collapse_alloc_folio(&folio, gfp, node, &cc->alloc_nmask)) {
*hpage = NULL;
@@ -2697,7 +2705,7 @@ static int madvise_collapse_errno(enum scan_result r)
}
int madvise_collapse(struct vm_area_struct *vma, struct vm_area_struct **prev,
- unsigned long start, unsigned long end)
+ unsigned long start, unsigned long end, unsigned int flags)
{
struct collapse_control *cc;
struct mm_struct *mm = vma->vm_mm;
@@ -2718,6 +2726,7 @@ int madvise_collapse(struct vm_area_struct *vma, struct vm_area_struct **prev,
if (!cc)
return -ENOMEM;
cc->is_khugepaged = false;
+ cc->flags = flags;
mmgrab(mm);
lru_add_drain_all();
diff --git a/mm/madvise.c b/mm/madvise.c
index 912155a94ed5..c41eb8656a42 100644
--- a/mm/madvise.c
+++ b/mm/madvise.c
@@ -1014,7 +1014,7 @@ static long madvise_remove(struct vm_area_struct *vma,
static int madvise_vma_behavior(struct vm_area_struct *vma,
struct vm_area_struct **prev,
unsigned long start, unsigned long end,
- unsigned long behavior)
+ unsigned long behavior, unsigned int flags)
{
int error;
struct anon_vma_name *anon_name;
@@ -1083,7 +1083,7 @@ static int madvise_vma_behavior(struct vm_area_struct *vma,
goto out;
break;
case MADV_COLLAPSE:
- return madvise_collapse(vma, prev, start, end);
+ return madvise_collapse(vma, prev, start, end, flags);
}
anon_name = anon_vma_name(vma);
@@ -1194,6 +1194,15 @@ madvise_behavior_valid(int behavior)
}
}
+
+static bool process_madvise_flags_valid(unsigned int flags)
+{
+ if (flags & ~(MADV_F_NORMAL|MADV_F_COLLAPSE_LIGHT))
+ return false;
+
+ return true;
+}
+
static bool process_madvise_behavior_valid(int behavior)
{
switch (behavior) {
@@ -1217,10 +1226,10 @@ static bool process_madvise_behavior_valid(int behavior)
*/
static
int madvise_walk_vmas(struct mm_struct *mm, unsigned long start,
- unsigned long end, unsigned long arg,
+ unsigned long end, unsigned long arg, unsigned int flags,
int (*visit)(struct vm_area_struct *vma,
struct vm_area_struct **prev, unsigned long start,
- unsigned long end, unsigned long arg))
+ unsigned long end, unsigned long arg, unsigned int flags))
{
struct vm_area_struct *vma;
struct vm_area_struct *prev;
@@ -1257,7 +1266,7 @@ int madvise_walk_vmas(struct mm_struct *mm, unsigned long start,
tmp = end;
/* Here vma->vm_start <= start < tmp <= (end|vma->vm_end). */
- error = visit(vma, &prev, start, tmp, arg);
+ error = visit(vma, &prev, start, tmp, arg, flags);
if (error)
return error;
start = tmp;
@@ -1278,7 +1287,7 @@ int madvise_walk_vmas(struct mm_struct *mm, unsigned long start,
static int madvise_vma_anon_name(struct vm_area_struct *vma,
struct vm_area_struct **prev,
unsigned long start, unsigned long end,
- unsigned long anon_name)
+ unsigned long anon_name, unsigned int flags)
{
int error;
@@ -1320,7 +1329,7 @@ int madvise_set_anon_name(struct mm_struct *mm, unsigned long start,
return 0;
return madvise_walk_vmas(mm, start, end, (unsigned long)anon_name,
- madvise_vma_anon_name);
+ MADV_F_NORMAL, madvise_vma_anon_name);
}
#endif /* CONFIG_ANON_VMA_NAME */
/*
@@ -1381,6 +1390,11 @@ int madvise_set_anon_name(struct mm_struct *mm, unsigned long start,
* MADV_POPULATE_WRITE - populate (prefault) page tables writable by
* triggering write faults if required
*
+ * flags values:
+ * MADV_F_NORMAL - the default flag.
+ * MADV_F_COLLAPSE_LIGHT - only for MADV_COLLAPSE, avoids direct reclaim and/or
+ * compaction.
+ *
* return values:
* zero - success
* -EINVAL - start + len < 0, start is not page-aligned,
@@ -1394,7 +1408,9 @@ int madvise_set_anon_name(struct mm_struct *mm, unsigned long start,
* -EBADF - map exists, but area maps something that isn't a file.
* -EAGAIN - a kernel resource was temporarily unavailable.
*/
-int do_madvise(struct mm_struct *mm, unsigned long start, size_t len_in, int behavior)
+static
+int _do_madvise(struct mm_struct *mm, unsigned long start, size_t len_in,
+ int behavior, unsigned int flags)
{
unsigned long end;
int error;
@@ -1437,7 +1453,7 @@ int do_madvise(struct mm_struct *mm, unsigned long start, size_t len_in, int beh
end = start + len;
blk_start_plug(&plug);
- error = madvise_walk_vmas(mm, start, end, behavior,
+ error = madvise_walk_vmas(mm, start, end, behavior, flags,
madvise_vma_behavior);
blk_finish_plug(&plug);
if (write)
@@ -1448,6 +1464,11 @@ int do_madvise(struct mm_struct *mm, unsigned long start, size_t len_in, int beh
return error;
}
+int do_madvise(struct mm_struct *mm, unsigned long start, size_t len_in, int behavior)
+{
+ return _do_madvise(mm, start, len_in, behavior, MADV_F_NORMAL);
+}
+
SYSCALL_DEFINE3(madvise, unsigned long, start, size_t, len_in, int, behavior)
{
return do_madvise(current->mm, start, len_in, behavior);
@@ -1465,7 +1486,7 @@ SYSCALL_DEFINE5(process_madvise, int, pidfd, const struct iovec __user *, vec,
size_t total_len;
unsigned int f_flags;
- if (flags != 0) {
+ if (!process_madvise_flags_valid(flags)) {
ret = -EINVAL;
goto out;
}
@@ -1504,8 +1525,8 @@ SYSCALL_DEFINE5(process_madvise, int, pidfd, const struct iovec __user *, vec,
total_len = iov_iter_count(&iter);
while (iov_iter_count(&iter)) {
- ret = do_madvise(mm, (unsigned long)iter_iov_addr(&iter),
- iter_iov_len(&iter), behavior);
+ ret = _do_madvise(mm, (unsigned long)iter_iov_addr(&iter),
+ iter_iov_len(&iter), behavior, flags);
if (ret < 0)
break;
iov_iter_advance(&iter, iter_iov_len(&iter));
diff --git a/tools/include/uapi/asm-generic/mman-common.h b/tools/include/uapi/asm-generic/mman-common.h
index 6ce1f1ceb432..0e0fac5c9219 100644
--- a/tools/include/uapi/asm-generic/mman-common.h
+++ b/tools/include/uapi/asm-generic/mman-common.h
@@ -79,6 +79,10 @@
#define MADV_COLLAPSE 25 /* Synchronous hugepage collapse */
+/* process_madvise flags */
+#define MADV_F_NORMAL 0 /* the default flag */
+#define MADV_F_COLLAPSE_LIGHT 0x1 /* only for MADV_COLLAPSE, avoids direct reclaim and/or compaction */
+
/* compatibility flags */
#define MAP_FILE 0
--
2.33.1
^ permalink raw reply related
* Re: [PATCH v4] selftests/landlock:Fix two build issues
From: Mickaël Salaün @ 2024-01-19 12:31 UTC (permalink / raw)
To: Hu Yadi
Cc: jmorris, serge, shuah, mathieu.desnoyers, amir73il, brauner,
avagin, linux-api, linux-kernel, linux-security-module,
linux-kselftest, 514118380, konstantin.meskhidze
In-Reply-To: <20240115102409.19799-1-hu.yadi@h3c.com>
On Mon, Jan 15, 2024 at 06:24:09PM +0800, Hu Yadi wrote:
> From: "Hu.Yadi" <hu.yadi@h3c.com>
>
> Two issues comes up while building selftest/landlock on my side
> (gcc 7.3/glibc-2.28/kernel-4.19)
>
> the first one is as to gettid
>
> net_test.c: In function ‘set_service’:
> net_test.c:91:45: warning: implicit declaration of function ‘gettid’; [-Wimplicit-function-declaration]
> "_selftests-landlock-net-tid%d-index%d", gettid(),
> ^~~~~~
> getgid
> net_test.c:(.text+0x4e0): undefined reference to `gettid'
>
> the second is compiler error
> gcc -Wall -O2 -isystem fs_test.c -lcap -o selftests/landlock/fs_test
> fs_test.c:4575:9: error: initializer element is not constant
> .mnt = mnt_tmp,
> ^~~~~~~
>
> Fixes: 04f9070e99a4 ("selftests/landlock: Add tests for pseudo filesystems")
> Fixes: a549d055a22e ("selftests/landlock: Add network tests")
Could you please create two patches as requested for v3, one per fix?
This is useful because it enables to backport these fixes when
appropriate.
>
> this patch is to fix them
>
> Signed-off-by: Hu.Yadi <hu.yadi@h3c.com>
> Suggested-by: Jiao <jiaoxupo@h3c.com>
> Reviewed-by: Berlin <berlin@h3c.com>
> ---
> Changes v4 -> v3:
> fix gettid error from kernel test robot
> https://lore.kernel.org/oe-kbuild-all/202401151147.T1s11iHJ-lkp@intel.com/
> Changes v3 -> v2:
> - add helper of gettid instead of __NR_gettid
> - add gcc/glibc version info in comments
> Changes v1 -> v2:
> - fix whitespace error
> - replace SYS_gettid with _NR_gettid
>
> tools/testing/selftests/landlock/fs_test.c | 5 ++++-
> tools/testing/selftests/landlock/net_test.c | 7 ++++++-
> 2 files changed, 10 insertions(+), 2 deletions(-)
>
> diff --git a/tools/testing/selftests/landlock/fs_test.c b/tools/testing/selftests/landlock/fs_test.c
> index 18e1f86a6234..a992cf7c0ad1 100644
> --- a/tools/testing/selftests/landlock/fs_test.c
> +++ b/tools/testing/selftests/landlock/fs_test.c
> @@ -4572,7 +4572,10 @@ FIXTURE_VARIANT(layout3_fs)
> /* clang-format off */
> FIXTURE_VARIANT_ADD(layout3_fs, tmpfs) {
> /* clang-format on */
> - .mnt = mnt_tmp,
> + .mnt = {
> + .type = "tmpfs",
> + .data = "size=4m,mode=700",
> + },
I requested some changes here.
> .file_path = file1_s1d1,
> };
>
> diff --git a/tools/testing/selftests/landlock/net_test.c b/tools/testing/selftests/landlock/net_test.c
> index 929e21c4db05..d50f2920ed82 100644
> --- a/tools/testing/selftests/landlock/net_test.c
> +++ b/tools/testing/selftests/landlock/net_test.c
> @@ -21,6 +21,11 @@
We should include sys/syscall.h
>
> #include "common.h"
>
> +static pid_t landlock_gettid(void)
Please rename to sys_gettid().
> +{
> + return syscall(__NR_gettid);
> +}
> +
> const short sock_port_start = (1 << 10);
>
> static const char loopback_ipv4[] = "127.0.0.1";
> @@ -88,7 +93,7 @@ static int set_service(struct service_fixture *const srv,
> case AF_UNIX:
> srv->unix_addr.sun_family = prot.domain;
> sprintf(srv->unix_addr.sun_path,
> - "_selftests-landlock-net-tid%d-index%d", gettid(),
> + "_selftests-landlock-net-tid%d-index%d", landlock_gettid(),
> index);
> srv->unix_addr_len = SUN_LEN(&srv->unix_addr);
> srv->unix_addr.sun_path[0] = '\0';
> --
> 2.23.0
>
^ permalink raw reply
* Re: [PATCH v3 1/1] mm/madvise: add MADV_F_COLLAPSE_LIGHT flag to process_madvise()
From: Michal Hocko @ 2024-01-19 12:37 UTC (permalink / raw)
To: Lance Yang
Cc: akpm, zokeefe, david, songmuchun, shy828301, peterx, mknyszek,
minchan, linux-mm, linux-api, linux-kernel
In-Reply-To: <20240119115104.75456-1-ioworker0@gmail.com>
Please do not start a new version until all the outstanding points are
settled. This just fragments the discussion and makes it hard to follow
it.
Thanks!
--
Michal Hocko
SUSE Labs
^ permalink raw reply
* Re: 回复: [PATCH v4] selftests/move_mount_set_group:Make tests build with old libc
From: Christian Brauner @ 2024-01-19 13:39 UTC (permalink / raw)
To: Huyadi
Cc: linux-api@vger.kernel.org, linux-kernel@vger.kernel.org,
linux-security-module@vger.kernel.org,
linux-kselftest@vger.kernel.org, 514118380@qq.com,
jmorris@namei.org, serge@hallyn.com, shuah@kernel.org,
mathieu.desnoyers@efficios.com, mic@digikod.net
In-Reply-To: <8918d5e83d54418b9db3ee9c055d675d@h3c.com>
> May I take the liberty to ask why I don't see patch applied to above branch?
Just wasn't pushed yet. It is now.
^ permalink raw reply
* Re: Several tst-robust* tests time out with recent Linux kernel
From: Stefan Liebler @ 2024-01-19 13:56 UTC (permalink / raw)
To: Edgecombe, Rick P, peterz@infradead.org
Cc: xry111@xry111.site, andrealmeid@igalia.com, fweimer@redhat.com,
linux-mm@kvack.org, libc-alpha@sourceware.org,
linux-kernel@vger.kernel.org, tglx@linutronix.de,
linux-api@vger.kernel.org, linux-arch@vger.kernel.org,
Heiko Carstens, Sven Schnelle
In-Reply-To: <158f6a47727a40c163e3fa6041a24388549c68f2.camel@intel.com>
[-- Attachment #1: Type: text/plain, Size: 2794 bytes --]
On 17.11.23 02:22, Edgecombe, Rick P wrote:
> A bit more info...
>
> The error returned to userspace is originating from:
> https://github.com/torvalds/linux/blob/master/kernel/futex/pi.c#L295
>
> 'uval' is often zero in that error case, but sometimes just a
> mismatching value like: uval=0x567, task_pid_vnr()=0x564
>
>
> Depending on the number of CPUs the VM is running on it reproduces or
> not. When it does reproduce, the newly added path here is taken:
> https://github.com/torvalds/linux/blob/master/kernel/futex/pi.c#L1185
> The path is taken a lot during the test, sometimes >400 times before
> the above linked error is generated during the syscall. When it doesn't
> reproduce, I never saw that new path taken.
>
> More print statements make the reproduction less reliable, so it does
> seem to have a race in the mix at least somewhat. Otherwise, I haven't
> tried to understand what is going on here with all this highwire
> locking.
>
> Hope it helps.
Hi,
I've also observed fails in glibc testcase nptl/tst-robust8pi with:
mutex_timedlock of 66 in thread 7 failed with 22
=> pthread_mutex_timedlock returns 22=EINVAL
I've saw it on s390x. There I've used kernel with
commit 120d99901eb288f1d21db3976df4ba347b28f9c7
s390/vfio-ap: do not reset queue removed from host config
But I also saw it on a x86_64 kvm-guest with Fedora 39 and
copr-repository with vanilla kernel:
Linux fedora 6.7.0-0.rc8.20240107gt52b1853b.366.vanilla.fc39.x86_64 #1
SMP PREEMPT_DYNAMIC Sun Jan 7 06:17:30 UTC 2024 x86_64 GNU/Linux
And reported it to libc-alpha ("FAILING nptl/tst-robust8pi"
https://sourceware.org/pipermail/libc-alpha/2024-January/154150.html)
where Florian Weimer pointed me to this thread.
I've reduced the test (see attachement) and now have only one process
with three threads. I only use one mutex with attributes like the
original testcase: PTHREAD_MUTEX_ROBUST_NP, PTHREAD_PROCESS_SHARED,
PTHREAD_PRIO_INHERIT.
Every thread is doing a loop with pthread_mutex_timedlock(abstime={0,0})
and if locked, pthread_mutex_unlock.
I've added some uprobes before and after the futex-syscall in
__futex_lock_pi64(in pthread_mutex_timedlock) and futex_unlock_pi(in
pthread_mutex_unlock). For me __ASSUME_FUTEX_LOCK_PI2 is not available,
but __ASSUME_TIME64_SYSCALLS is defined.
For me it looks like this (simplified ubprobes-trace):
<thread> <timestamp>: <probe>
t1 4309589.419744: before syscall in __futex_lock_pi64
t3 4309589.419745: before syscall in futex_unlock_pi
t2 4309589.419745: before syscall in __futex_lock_pi64
t3 4309589.419747: after syscall in futex_unlock_pi
t2 4309589.419747: after syscall in __futex_lock_pi64 ret=-22=EINVAL
t1 4309589.419748: after syscall in __futex_lock_pi64 ret=-110=ETIMEDOUT
Can you please have a look again?
Bye,
Stefan Liebler
[-- Attachment #2: tst-robust8pi-20240118.c --]
[-- Type: text/x-csrc, Size: 4161 bytes --]
//CFLAGS=-pthread
//LDFLAGS=-lpthread
#include <stdio.h>
#include <stdlib.h>
#include <pthread.h>
#include <assert.h>
#include <errno.h>
#include <unistd.h>
#define NUM_THREADS 3
#define THREAD_FUNC thr_func
#define USE_BARRIER 1
#ifndef ROUNDS
# define ROUNDS 100000000
#endif
typedef struct thr_info
{
int nr;
pthread_t thread;
} __attribute__ ((aligned (256))) thr_info_t;
#define THR_INIT() \
thr_info_t *thr = (thr_info_t *) arg;
#define THR_PRINTF(fmt, ...) \
printf ("#%d: " fmt, thr->nr, __VA_ARGS__)
#define THR_PUTS(msg) \
printf ("#%d: " msg "\n", thr->nr)
#if USE_BARRIER != 0
static pthread_barrier_t thrs_barrier;
#endif
static pthread_mutex_t mtx;
static const struct timespec before = { 0, 0 };
/* ###################################################################
thread func
############################################################### */
static void *
thr_func (void *arg)
{
THR_INIT ();
int state = 0;
int fct;
#if 0
/* 3 threads, 1xfct=0=pthread_mutex_lock, 2xfct=1=pthread_mutex_timedlock: EINVAL. */
fct = (thr->nr + 1) % 2;
#elif 0
/* 3 threads, 2xfct=0=pthread_mutex_lock, 1xfct=1=pthread_mutex_timedlock: no fails. */
fct = (thr->nr) % 2;
#elif 1
/* >3 threads, fct=1=only pthread_mutex_timedlock: EINVAL. */
fct = 1;
#endif
int round = 0;
THR_PRINTF ("started: fct=%d\n", fct);
#if USE_BARRIER != 0
pthread_barrier_wait (&thrs_barrier);
#endif
while (1)
{
if (state == 0)
{
round ++;
int e;
switch (fct)
{
case 0:
e = pthread_mutex_lock (&mtx);
if (e != 0)
{
THR_PRINTF ("mutex_lock failed with %d (round=%d)\n", e, round);
exit (1);
}
state = 1;
break;
case 1:
e = pthread_mutex_timedlock (&mtx, &before);
if (e != 0 && e != ETIMEDOUT)
{
THR_PRINTF ("mutex_timedlock failed with %d (round=%d)\n", e, round);
exit (1);
}
break;
default:
e = pthread_mutex_trylock (&mtx);
if (e != 0 && e != EBUSY)
{
THR_PRINTF ("mutex_trylock failed with %d (round=%d)\n", e, round);
exit (1);
}
break;
}
if (e == EOWNERDEAD)
pthread_mutex_consistent (&mtx);
if (e == 0 || e == EOWNERDEAD)
state = 1;
}
else
{
int e = pthread_mutex_unlock (&mtx);
if (e != 0)
{
THR_PRINTF ("mutex_unlock of failed with %d (round=%d)\n", e, round);
exit (1);
}
state = 0;
}
if (round >= ROUNDS)
{
THR_PRINTF ("REACHED round %d. => exit\n", ROUNDS);
if (state != 0)
{
int e = pthread_mutex_unlock (&mtx);
if (e != 0)
{
THR_PRINTF ("mutex_unlock@exit of failed with %d (round=%d)\n", e, round);
exit (1);
}
}
break;
}
}
return NULL;
}
int
main (void)
{
int i;
printf ("main: start %d threads.\n", NUM_THREADS);
#if USE_BARRIER != 0
pthread_barrier_init (&thrs_barrier, NULL, NUM_THREADS + 1);
#endif
pthread_mutexattr_t ma;
if (pthread_mutexattr_init (&ma) != 0)
{
puts ("mutexattr_init failed");
return 0;
}
if (pthread_mutexattr_setrobust (&ma, PTHREAD_MUTEX_ROBUST_NP) != 0)
{
puts ("mutexattr_setrobust failed");
return 1;
}
if (pthread_mutexattr_setpshared (&ma, PTHREAD_PROCESS_SHARED) != 0)
{
puts ("mutexattr_setpshared failed");
return 1;
}
if (pthread_mutexattr_setprotocol (&ma, PTHREAD_PRIO_INHERIT) != 0)
{
puts ("pthread_mutexattr_setprotocol failed");
return 1;
}
if (pthread_mutex_init (&mtx, &ma) != 0)
{
puts ("pthread_mutex_init failed");
return 1;
}
thr_info_t thrs[NUM_THREADS];
for (i = 0; i < NUM_THREADS; i++)
{
thrs[i].nr = i;
assert (pthread_create (&(thrs[i].thread), NULL, THREAD_FUNC, &(thrs[i]))
== 0);;
}
#if USE_BARRIER != 0
/* All threads start work after this barrier. */
pthread_barrier_wait (&thrs_barrier);
#endif
for (i = 0; i < NUM_THREADS; i++)
{
pthread_join (thrs[i].thread, NULL);
}
#if USE_BARRIER != 0
pthread_barrier_destroy (&thrs_barrier);
#endif
printf ("main: end.\n");
return EXIT_SUCCESS;
}
^ permalink raw reply
* Re: [PATCH v3 1/1] mm/madvise: add MADV_F_COLLAPSE_LIGHT flag to process_madvise()
From: Lance Yang @ 2024-01-19 14:10 UTC (permalink / raw)
To: Michal Hocko
Cc: akpm, zokeefe, david, songmuchun, shy828301, peterx, mknyszek,
minchan, linux-mm, linux-api, linux-kernel
In-Reply-To: <Zaps_0jnspsheP92@tiehlicka>
Sorry for that.
BR,
Lance
On Fri, Jan 19, 2024 at 8:37 PM Michal Hocko <mhocko@suse.com> wrote:
>
> Please do not start a new version until all the outstanding points are
> settled. This just fragments the discussion and makes it hard to follow
> it.
>
> Thanks!
> --
> Michal Hocko
> SUSE Labs
^ permalink raw reply
* Re: [PATCH v2] vfs: add RWF_NOAPPEND flag for pwritev2
From: Christian Brauner @ 2024-01-19 14:33 UTC (permalink / raw)
To: Rich Felker, Jens Axboe
Cc: Christian Brauner, linux-fsdevel, kernel list, Linux API,
Alexander Viro, Pavel Begunkov, Jann Horn
In-Reply-To: <20200831153207.GO3265@brightrain.aerifal.cx>
On Mon, 31 Aug 2020 11:32:08 -0400, Rich Felker wrote:
> The pwrite function, originally defined by POSIX (thus the "p"), is
> defined to ignore O_APPEND and write at the offset passed as its
> argument. However, historically Linux honored O_APPEND if set and
> ignored the offset. This cannot be changed due to stability policy,
> but is documented in the man page as a bug.
>
> Now that there's a pwritev2 syscall providing a superset of the pwrite
> functionality that has a flags argument, the conforming behavior can
> be offered to userspace via a new flag. Since pwritev2 checks flag
> validity (in kiocb_set_rw_flags) and reports unknown ones with
> EOPNOTSUPP, callers will not get wrong behavior on old kernels that
> don't support the new flag; the error is reported and the caller can
> decide how to handle it.
>
> [...]
The RWF_* and IOCB_* flags were
aligned so they could be set in one operation. So there was a merge
conflict when applying. I've resolved it. Please take a look and make
sure that it's all correct.
---
Applied to the vfs.misc branch of the vfs/vfs.git tree.
Patches in the vfs.misc branch should appear in linux-next soon.
Please report any outstanding bugs that were missed during review in a
new review to the original patch series allowing us to drop it.
It's encouraged to provide Acked-bys and Reviewed-bys even though the
patch has now been applied. If possible patch trailers will be updated.
Note that commit hashes shown below are subject to change due to rebase,
trailer updates or similar. If in doubt, please check the listed branch.
tree: https://git.kernel.org/pub/scm/linux/kernel/git/vfs/vfs.git
branch: vfs.misc
[1/1] vfs: add RWF_NOAPPEND flag for pwritev2
https://git.kernel.org/vfs/vfs/c/31081ab305a1
^ permalink raw reply
* [PATCH v2 0/3] mm/mempolicy: weighted interleave mempolicy and sysfs extension
From: Gregory Price @ 2024-01-19 17:57 UTC (permalink / raw)
To: linux-mm
Cc: linux-kernel, linux-doc, linux-fsdevel, linux-api, corbet, akpm,
gregory.price, honggyu.kim, rakie.kim, hyeongtak.ji, mhocko,
ying.huang, vtavarespetr, jgroves, ravis.opensrc, sthanneeru,
emirakhur, Hasan.Maruf, seungjun.ha, hannes, dan.j.williams,
Hasan Al Maruf, Hao Wang, Michal Hocko, Zhongkun He,
Frank van der Linden, John Groves, Jonathan Cameron, Andi Kleen
Hi Andrew,
Can you please replace the patches on mm-unstable with this line,
it has bulk-allocator bug fixes and some design changes at the
request of Ying Huang. Full v2 notes are just before the test info.
= Cover Letter
Weighted interleave is a new interleave policy intended to make
use of heterogeneous memory environments appearing with CXL.
The existing interleave mechanism does an even round-robin
distribution of memory across all nodes in a nodemask, while
weighted interleave distributes memory across nodes according
to a provided weight. (Weight = # of page allocations per round)
Weighted interleave is intended to reduce average latency when
bandwidth is pressured - therefore increasing total throughput.
In other words: It allows greater use of the total available
bandwidth in a heterogeneous hardware environment (different
hardware provides different bandwidth capacity).
As bandwidth is pressured, latency increases - first linearly
and then exponentially. By keeping bandwidth usage distributed
according to available bandwidth, we therefore can reduce the
average latency of a cacheline fetch.
A good explanation of the bandwidth vs latency response curve:
https://mahmoudhatem.wordpress.com/2017/11/07/memory-bandwidth-vs-latency-response-curve/
From the article:
```
Constant region:
The latency response is fairly constant for the first 40%
of the sustained bandwidth.
Linear region:
In between 40% to 80% of the sustained bandwidth, the
latency response increases almost linearly with the bandwidth
demand of the system due to contention overhead by numerous
memory requests.
Exponential region:
Between 80% to 100% of the sustained bandwidth, the memory
latency is dominated by the contention latency which can be
as much as twice the idle latency or more.
Maximum sustained bandwidth :
Is 65% to 75% of the theoretical maximum bandwidth.
```
As a general rule of thumb:
* If bandwidth usage is low, latency does not increase. It is
optimal to place data in the nearest (lowest latency) device.
* If bandwidth usage is high, latency increases. It is optimal
to place data such that bandwidth use is optimized per-device.
This is the top line goal: Provide a user a mechanism to target using
the "maximum sustained bandwidth" of each hardware component in a
heterogenous memory system.
For example, the stream benchmark demonstrates that 1:1 (default)
interleave is actively harmful, while weighted interleave can be
beneficial. Default interleave distributes data such that too much
pressure is placed on devices with lower available bandwidth.
Stream Benchmark (High level results, 1 Socket + 1 CXL Device)
Default interleave : -78% (slower than DRAM)
Global weighting : -6% to +4% (workload dependant)
Targeted weights : +2.5% to +4% (consistently better than DRAM)
Global means the task-policy was set (set_mempolicy), while targeted
means VMA policies were set (mbind2). We see weighted interleave
is not always beneficial when applied globally, but is always
beneficial when applied to bandwidth-driving memory regions.
We implement sysfs entries for "system global" weights which can be
set by a daemon or administrator.
There are 3 patches in this set:
1) Implement system-global interleave weights as sysfs extension
in mm/mempolicy.c. These weights are RCU protected, and the
default/minimum weight is 1 for all nodes.
In future work, we intend to expose an interface for HMAT/CDAT
information to be used during boot to set reasonable system
default values based on the memory configuration of the system
discovered at boot or during device hotplug.
2) A mild refactor of some interleave-logic for re-use in the
new weighted interleave logic.
3) MPOL_WEIGHTED_INTERLEAVE extension for set_mempolicy/mbind
Included below are some performance and LTP test information,
and a sample numactl branch which can be used for testing.
= Performance summary =
(tests may have different configurations, see extended info below)
1) MLC (W2) : +38% over DRAM. +264% over default interleave.
MLC (W5) : +40% over DRAM. +226% over default interleave.
2) Stream : -6% to +4% over DRAM, +430% over default interleave.
3) XSBench : +19% over DRAM. +47% over default interleave.
= LTP Testing Summary =
existing mempolicy & mbind tests: pass
mempolicy & mbind + weighted interleave (global weights): pass
= version history
v2:
- MAJOR: Torture tested bulk allocator, fixed edge conditions
tracking the next me->il_node. Added documentation.
Prior version was stable, but the resulting me->il_node
could be wrong under certain circumstances.
- naming: iw_table_mtx -> iw_table_lock
- RCU: use synchronize+kfree and simplify the weight structure
- default: remove default table, since it's static for now
- sysfs setup: simplify setup, if table==NULL presume 1's
- node_store: only allocate (sizeof(u8) * nr_node_ids)
- allocators: update to deal with NULL table pointer
- read_once: __builtin_memcpy -> memcpy
- formatting
v1:
- RCU: This version protects the weight array with RCU.
- ktest fix: proper include (types.h) in uapi header
- doc: make mpol_params in docs reflect definition
- doc: mempolicy.c comments in MPOL_WEIGHTED_INTERLEAVE patch
- Dropped task-local weights and syscalls from the proposal
until affirmative use cases for task-local weights appear.
Link: https://lore.kernel.org/linux-mm/20240103224209.2541-1-gregory.price@memverge.com/
=====================================================================
Performance tests - MLC
From - Ravi Jonnalagadda <ravis.opensrc@micron.com>
Hardware: Single-socket, multiple CXL memory expanders.
Workload: W2
Data Signature: 2:1 read:write
DRAM only bandwidth (GBps): 298.8
DRAM + CXL (default interleave) (GBps): 113.04
DRAM + CXL (weighted interleave)(GBps): 412.5
Gain over DRAM only: 1.38x
Gain over default interleave: 2.64x
Workload: W5
Data Signature: 1:1 read:write
DRAM only bandwidth (GBps): 273.2
DRAM + CXL (default interleave) (GBps): 117.23
DRAM + CXL (weighted interleave)(GBps): 382.7
Gain over DRAM only: 1.4x
Gain over default interleave: 2.26x
=====================================================================
Performance test - Stream
From - Gregory Price <gregory.price@memverge.com>
Hardware: Single socket, single CXL expander
numactl extension: https://github.com/gmprice/numactl/tree/weighted_interleave_master
Summary: 64 threads, ~18GB workload, 3GB per array, executed 100 times
Default interleave : -78% (slower than DRAM)
Global weighting : -6% to +4% (workload dependant)
mbind2 weights : +2.5% to +4% (consistently better than DRAM)
dram only:
numactl --cpunodebind=1 --membind=1 ./stream_c.exe --ntimes 100 --array-size 400M --malloc
Function Direction BestRateMBs AvgTime MinTime MaxTime
Copy: 0->0 200923.2 0.032662 0.031853 0.033301
Scale: 0->0 202123.0 0.032526 0.031664 0.032970
Add: 0->0 208873.2 0.047322 0.045961 0.047884
Triad: 0->0 208523.8 0.047262 0.046038 0.048414
CXL-only:
numactl --cpunodebind=1 -w --membind=2 ./stream_c.exe --ntimes 100 --array-size 400M --malloc
Copy: 0->0 22209.7 0.288661 0.288162 0.289342
Scale: 0->0 22288.2 0.287549 0.287147 0.288291
Add: 0->0 24419.1 0.393372 0.393135 0.393735
Triad: 0->0 24484.6 0.392337 0.392083 0.394331
Based on the above, the optimal weights are ~9:1
echo 9 > /sys/kernel/mm/mempolicy/weighted_interleave/node1
echo 1 > /sys/kernel/mm/mempolicy/weighted_interleave/node2
default interleave:
numactl --cpunodebind=1 --interleave=1,2 ./stream_c.exe --ntimes 100 --array-size 400M --malloc
Copy: 0->0 44666.2 0.143671 0.143285 0.144174
Scale: 0->0 44781.6 0.143256 0.142916 0.143713
Add: 0->0 48600.7 0.197719 0.197528 0.197858
Triad: 0->0 48727.5 0.197204 0.197014 0.197439
global weighted interleave:
numactl --cpunodebind=1 -w --interleave=1,2 ./stream_c.exe --ntimes 100 --array-size 400M --malloc
Copy: 0->0 190085.9 0.034289 0.033669 0.034645
Scale: 0->0 207677.4 0.031909 0.030817 0.033061
Add: 0->0 202036.8 0.048737 0.047516 0.053409
Triad: 0->0 217671.5 0.045819 0.044103 0.046755
targted regions w/ global weights (modified stream to mbind2 malloc'd regions))
numactl --cpunodebind=1 --membind=1 ./stream_c.exe -b --ntimes 100 --array-size 400M --malloc
Copy: 0->0 205827.0 0.031445 0.031094 0.031984
Scale: 0->0 208171.8 0.031320 0.030744 0.032505
Add: 0->0 217352.0 0.045087 0.044168 0.046515
Triad: 0->0 216884.8 0.045062 0.044263 0.046982
=====================================================================
Performance tests - XSBench
From - Hyeongtak Ji <hyeongtak.ji@sk.com>
Hardware: Single socket, Single CXL memory Expander
NUMA node 0: 56 logical cores, 128 GB memory
NUMA node 2: 96 GB CXL memory
Threads: 56
Lookups: 170,000,000
Summary: +19% over DRAM. +47% over default interleave.
Performance tests - XSBench
1. dram only
$ numactl -m 0 ./XSBench -s XL –p 5000000
Runtime: 36.235 seconds
Lookups/s: 4,691,618
2. default interleave
$ numactl –i 0,2 ./XSBench –s XL –p 5000000
Runtime: 55.243 seconds
Lookups/s: 3,077,293
3. weighted interleave
numactl –w –i 0,2 ./XSBench –s XL –p 5000000
Runtime: 29.262 seconds
Lookups/s: 5,809,513
=====================================================================
LTP Tests: https://github.com/gmprice/ltp/tree/mempolicy2
= Existing tests
set_mempolicy, get_mempolicy, mbind
MPOL_WEIGHTED_INTERLEAVE added manually to test basic functionality
but did not adjust tests for weighting. Basically the weights were
set to 1, which is the default, and it should behavior like standard
MPOL_INTERLEAVE if logic is correct.
== set_mempolicy01 : passed 18, failed 0
== set_mempolicy02 : passed 10, failed 0
== set_mempolicy03 : passed 64, failed 0
== set_mempolicy04 : passed 32, failed 0
== set_mempolicy05 - n/a on non-x86
== set_mempolicy06 : passed 10, failed 0
this is set_mempolicy02 + MPOL_WEIGHTED_INTERLEAVE
== set_mempolicy07 : passed 32, failed 0
set_mempolicy04 + MPOL_WEIGHTED_INTERLEAVE
== get_mempolicy01 : passed 12, failed 0
change: added MPOL_WEIGHTED_INTERLEAVE
== get_mempolicy02 : passed 2, failed 0
== mbind01 : passed 15, failed 0
added MPOL_WEIGHTED_INTERLEAVE
== mbind02 : passed 4, failed 0
added MPOL_WEIGHTED_INTERLEAVE
== mbind03 : passed 16, failed 0
added MPOL_WEIGHTED_INTERLEAVE
== mbind04 : passed 48, failed 0
added MPOL_WEIGHTED_INTERLEAVE
=====================================================================
numactl (set_mempolicy) w/ global weighting test
numactl fork: https://github.com/gmprice/numactl/tree/weighted_interleave_master
command: numactl -w --interleave=0,1 ./eatmem
result (weights 1:1):
0176a000 weighted interleave:0-1 heap anon=65793 dirty=65793 active=0 N0=32897 N1=32896 kernelpagesize_kB=4
7fceeb9ff000 weighted interleave:0-1 anon=65537 dirty=65537 active=0 N0=32768 N1=32769 kernelpagesize_kB=4
50% distribution is correct
result (weights 5:1):
01b14000 weighted interleave:0-1 heap anon=65793 dirty=65793 active=0 N0=54828 N1=10965 kernelpagesize_kB=4
7f47a1dff000 weighted interleave:0-1 anon=65537 dirty=65537 active=0 N0=54614 N1=10923 kernelpagesize_kB=4
16.666% distribution is correct
result (weights 1:5):
01f07000 weighted interleave:0-1 heap anon=65793 dirty=65793 active=0 N0=10966 N1=54827 kernelpagesize_kB=4
7f17b1dff000 weighted interleave:0-1 anon=65537 dirty=65537 active=0 N0=10923 N1=54614 kernelpagesize_kB=4
16.666% distribution is correct
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
int main (void)
{
char* mem = malloc(1024*1024*256);
memset(mem, 1, 1024*1024*256);
for (int i = 0; i < ((1024*1024*256)/4096); i++)
{
mem = malloc(4096);
mem[0] = 1;
}
printf("done\n");
getchar();
return 0;
}
=====================================================================
Suggested-by: Gregory Price <gregory.price@memverge.com>
Suggested-by: Johannes Weiner <hannes@cmpxchg.org>
Suggested-by: Hasan Al Maruf <hasanalmaruf@fb.com>
Suggested-by: Hao Wang <haowang3@fb.com>
Suggested-by: Ying Huang <ying.huang@intel.com>
Suggested-by: Dan Williams <dan.j.williams@intel.com>
Suggested-by: Michal Hocko <mhocko@suse.com>
Suggested-by: Zhongkun He <hezhongkun.hzk@bytedance.com>
Suggested-by: Frank van der Linden <fvdl@google.com>
Suggested-by: John Groves <john@jagalactic.com>
Suggested-by: Vinicius Tavares Petrucci <vtavarespetr@micron.com>
Suggested-by: Srinivasulu Thanneeru <sthanneeru@micron.com>
Suggested-by: Ravi Jonnalagadda <ravis.opensrc@micron.com>
Suggested-by: Jonathan Cameron <Jonathan.Cameron@Huawei.com>
Suggested-by: Hyeongtak Ji <hyeongtak.ji@sk.com>
Suggested-by: Andi Kleen <ak@linux.intel.com>
Signed-off-by: Gregory Price <gregory.price@memverge.com>
Gregory Price (2):
mm/mempolicy: refactor a read-once mechanism into a function for
re-use
mm/mempolicy: introduce MPOL_WEIGHTED_INTERLEAVE for weighted
interleaving
Rakie Kim (1):
mm/mempolicy: implement the sysfs-based weighted_interleave interface
.../ABI/testing/sysfs-kernel-mm-mempolicy | 4 +
...fs-kernel-mm-mempolicy-weighted-interleave | 26 +
.../admin-guide/mm/numa_memory_policy.rst | 9 +
include/linux/mempolicy.h | 5 +
include/uapi/linux/mempolicy.h | 1 +
mm/mempolicy.c | 491 +++++++++++++++++-
6 files changed, 523 insertions(+), 13 deletions(-)
create mode 100644 Documentation/ABI/testing/sysfs-kernel-mm-mempolicy
create mode 100644 Documentation/ABI/testing/sysfs-kernel-mm-mempolicy-weighted-interleave
--
2.39.1
^ permalink raw reply
* [PATCH v2 1/3] mm/mempolicy: implement the sysfs-based weighted_interleave interface
From: Gregory Price @ 2024-01-19 17:57 UTC (permalink / raw)
To: linux-mm
Cc: linux-kernel, linux-doc, linux-fsdevel, linux-api, corbet, akpm,
gregory.price, honggyu.kim, rakie.kim, hyeongtak.ji, mhocko,
ying.huang, vtavarespetr, jgroves, ravis.opensrc, sthanneeru,
emirakhur, Hasan.Maruf, seungjun.ha, hannes, dan.j.williams
In-Reply-To: <20240119175730.15484-1-gregory.price@memverge.com>
From: Rakie Kim <rakie.kim@sk.com>
This patch provides a way to set interleave weight information under
sysfs at /sys/kernel/mm/mempolicy/weighted_interleave/nodeN
The sysfs structure is designed as follows.
$ tree /sys/kernel/mm/mempolicy/
/sys/kernel/mm/mempolicy/ [1]
└── weighted_interleave [2]
├── node0 [3]
└── node1
Each file above can be explained as follows.
[1] mm/mempolicy: configuration interface for mempolicy subsystem
[2] weighted_interleave/: config interface for weighted interleave policy
[3] weighted_interleave/nodeN: weight for nodeN
If a node value is set to `0`, the system-default value will be used.
As of this patch, the system-default for all nodes is always 1.
Suggested-by: Huang Ying <ying.huang@intel.com>
Signed-off-by: Rakie Kim <rakie.kim@sk.com>
Signed-off-by: Honggyu Kim <honggyu.kim@sk.com>
Co-developed-by: Gregory Price <gregory.price@memverge.com>
Signed-off-by: Gregory Price <gregory.price@memverge.com>
Co-developed-by: Hyeongtak Ji <hyeongtak.ji@sk.com>
Signed-off-by: Hyeongtak Ji <hyeongtak.ji@sk.com>
---
.../ABI/testing/sysfs-kernel-mm-mempolicy | 4 +
...fs-kernel-mm-mempolicy-weighted-interleave | 26 ++
mm/mempolicy.c | 231 ++++++++++++++++++
3 files changed, 261 insertions(+)
create mode 100644 Documentation/ABI/testing/sysfs-kernel-mm-mempolicy
create mode 100644 Documentation/ABI/testing/sysfs-kernel-mm-mempolicy-weighted-interleave
diff --git a/Documentation/ABI/testing/sysfs-kernel-mm-mempolicy b/Documentation/ABI/testing/sysfs-kernel-mm-mempolicy
new file mode 100644
index 000000000000..2dcf24f4384a
--- /dev/null
+++ b/Documentation/ABI/testing/sysfs-kernel-mm-mempolicy
@@ -0,0 +1,4 @@
+What: /sys/kernel/mm/mempolicy/
+Date: December 2023
+Contact: Linux memory management mailing list <linux-mm@kvack.org>
+Description: Interface for Mempolicy
diff --git a/Documentation/ABI/testing/sysfs-kernel-mm-mempolicy-weighted-interleave b/Documentation/ABI/testing/sysfs-kernel-mm-mempolicy-weighted-interleave
new file mode 100644
index 000000000000..e6a38139bf0f
--- /dev/null
+++ b/Documentation/ABI/testing/sysfs-kernel-mm-mempolicy-weighted-interleave
@@ -0,0 +1,26 @@
+What: /sys/kernel/mm/mempolicy/weighted_interleave/
+Date: December 2023
+Contact: Linux memory management mailing list <linux-mm@kvack.org>
+Description: Configuration Interface for the Weighted Interleave policy
+
+What: /sys/kernel/mm/mempolicy/weighted_interleave/nodeN
+Date: December 2023
+Contact: Linux memory management mailing list <linux-mm@kvack.org>
+Description: Weight configuration interface for nodeN
+
+ The interleave weight for a memory node (N). These weights are
+ utilized by processes which have set their mempolicy to
+ MPOL_WEIGHTED_INTERLEAVE and have opted into global weights by
+ omitting a task-local weight array.
+
+ These weights only affect new allocations, and changes at runtime
+ will not cause migrations on already allocated pages.
+
+ The minimum weight for a node is always 1.
+
+ Minimum weight: 1
+ Maximum weight: 255
+
+ Writing an empty string or `0` will reset the weight to the
+ system default. The system default may be set by the kernel
+ or drivers at boot or during hotplug events.
diff --git a/mm/mempolicy.c b/mm/mempolicy.c
index 10a590ee1c89..ae925216798f 100644
--- a/mm/mempolicy.c
+++ b/mm/mempolicy.c
@@ -131,6 +131,16 @@ static struct mempolicy default_policy = {
static struct mempolicy preferred_node_policy[MAX_NUMNODES];
+/*
+ * iw_table is the sysfs-set interleave weight table, a value of 0 denotes
+ * system-default value should be used. Until system-defaults are implemented,
+ * the system-default is always 1.
+ *
+ * iw_table is RCU protected
+ */
+static u8 __rcu *iw_table;
+static DEFINE_MUTEX(iw_table_lock);
+
/**
* numa_nearest_node - Find nearest node by state
* @node: Node id to start the search
@@ -3067,3 +3077,224 @@ void mpol_to_str(char *buffer, int maxlen, struct mempolicy *pol)
p += scnprintf(p, buffer + maxlen - p, ":%*pbl",
nodemask_pr_args(&nodes));
}
+
+#ifdef CONFIG_SYSFS
+struct iw_node_attr {
+ struct kobj_attribute kobj_attr;
+ int nid;
+};
+
+static ssize_t node_show(struct kobject *kobj, struct kobj_attribute *attr,
+ char *buf)
+{
+ struct iw_node_attr *node_attr;
+ u8 weight;
+ u8 __rcu *table;
+
+ node_attr = container_of(attr, struct iw_node_attr, kobj_attr);
+
+ rcu_read_lock();
+ table = rcu_dereference(iw_table);
+ weight = table ? table[node_attr->nid] : 1;
+ rcu_read_unlock();
+
+ return sysfs_emit(buf, "%d\n", weight);
+}
+
+static ssize_t node_store(struct kobject *kobj, struct kobj_attribute *attr,
+ const char *buf, size_t count)
+{
+ struct iw_node_attr *node_attr;
+ u8 __rcu *new;
+ u8 __rcu *old;
+ u8 weight = 0;
+
+ node_attr = container_of(attr, struct iw_node_attr, kobj_attr);
+ if (count == 0 || sysfs_streq(buf, ""))
+ weight = 0;
+ else if (kstrtou8(buf, 0, &weight))
+ return -EINVAL;
+
+ /*
+ * The default weight is 1 (for now), when the kernel-internal
+ * default weight array is implemented, this should be updated to
+ * collect the system-default weight of the node if the user passes 0.
+ */
+ if (!weight)
+ weight = 1;
+
+ /* We only need to allocate up to the number of possible nodes */
+ new = kmalloc(nr_node_ids, GFP_KERNEL);
+ if (!new)
+ return -ENOMEM;
+
+ mutex_lock(&iw_table_lock);
+ old = rcu_dereference_protected(iw_table,
+ lockdep_is_held(&iw_table_lock));
+ if (old)
+ memcpy(new, old, nr_node_ids);
+ else
+ memset(new, 1, nr_node_ids);
+ new[node_attr->nid] = weight;
+ rcu_assign_pointer(iw_table, new);
+ mutex_unlock(&iw_table_lock);
+ synchronize_rcu();
+ kfree(old);
+ return count;
+}
+
+static struct iw_node_attr *node_attrs[MAX_NUMNODES];
+
+static void sysfs_wi_node_release(struct iw_node_attr *node_attr,
+ struct kobject *parent)
+{
+ if (!node_attr)
+ return;
+ sysfs_remove_file(parent, &node_attr->kobj_attr.attr);
+ kfree(node_attr->kobj_attr.attr.name);
+ kfree(node_attr);
+}
+
+static void sysfs_wi_release(struct kobject *wi_kobj)
+{
+ int i;
+
+ for (i = 0; i < MAX_NUMNODES; i++)
+ sysfs_wi_node_release(node_attrs[i], wi_kobj);
+ kobject_put(wi_kobj);
+}
+
+static const struct kobj_type wi_ktype = {
+ .sysfs_ops = &kobj_sysfs_ops,
+ .release = sysfs_wi_release,
+};
+
+static int add_weight_node(int nid, struct kobject *wi_kobj)
+{
+ struct iw_node_attr *node_attr;
+ char *name;
+
+ node_attr = kzalloc(sizeof(*node_attr), GFP_KERNEL);
+ if (!node_attr)
+ return -ENOMEM;
+
+ name = kasprintf(GFP_KERNEL, "node%d", nid);
+ if (!name) {
+ kfree(node_attr);
+ return -ENOMEM;
+ }
+
+ sysfs_attr_init(&node_attr->kobj_attr.attr);
+ node_attr->kobj_attr.attr.name = name;
+ node_attr->kobj_attr.attr.mode = 0644;
+ node_attr->kobj_attr.show = node_show;
+ node_attr->kobj_attr.store = node_store;
+ node_attr->nid = nid;
+
+ if (sysfs_create_file(wi_kobj, &node_attr->kobj_attr.attr)) {
+ kfree(node_attr->kobj_attr.attr.name);
+ kfree(node_attr);
+ pr_err("failed to add attribute to weighted_interleave\n");
+ return -ENOMEM;
+ }
+
+ node_attrs[nid] = node_attr;
+ return 0;
+}
+
+static int add_weighted_interleave_group(struct kobject *root_kobj)
+{
+ struct kobject *wi_kobj;
+ int nid, err;
+
+ wi_kobj = kzalloc(sizeof(struct kobject), GFP_KERNEL);
+ if (!wi_kobj)
+ return -ENOMEM;
+
+ err = kobject_init_and_add(wi_kobj, &wi_ktype, root_kobj,
+ "weighted_interleave");
+ if (err) {
+ kfree(wi_kobj);
+ return err;
+ }
+
+ memset(node_attrs, 0, sizeof(node_attrs));
+ for_each_node_state(nid, N_POSSIBLE) {
+ err = add_weight_node(nid, wi_kobj);
+ if (err) {
+ pr_err("failed to add sysfs [node%d]\n", nid);
+ break;
+ }
+ }
+ if (err)
+ kobject_put(wi_kobj);
+ return 0;
+}
+
+static void mempolicy_kobj_release(struct kobject *kobj)
+{
+ u8 __rcu *old;
+
+ mutex_lock(&iw_table_lock);
+ old = rcu_dereference_protected(iw_table,
+ lockdep_is_held(&iw_table_lock));
+ rcu_assign_pointer(iw_table, NULL);
+ mutex_unlock(&iw_table_lock);
+ synchronize_rcu();
+ /* Never free the default table, it's always in use */
+ kfree(old);
+ kfree(kobj);
+}
+
+static const struct kobj_type mempolicy_ktype = {
+ .release = mempolicy_kobj_release
+};
+
+static struct kobject *mempolicy_kobj;
+static int __init mempolicy_sysfs_init(void)
+{
+ int err;
+ struct kobject *mempolicy_kobj;
+
+ /* A NULL iw_table is interpreted by interleave logic as "all 1s" */
+ iw_table = NULL;
+ mempolicy_kobj = kzalloc(sizeof(*mempolicy_kobj), GFP_KERNEL);
+ if (!mempolicy_kobj) {
+ pr_err("failed to add mempolicy kobject to the system\n");
+ return -ENOMEM;
+ }
+ err = kobject_init_and_add(mempolicy_kobj, &mempolicy_ktype, mm_kobj,
+ "mempolicy");
+ if (err) {
+ kfree(mempolicy_kobj);
+ return err;
+ }
+
+ err = add_weighted_interleave_group(mempolicy_kobj);
+
+ if (err) {
+ kobject_put(mempolicy_kobj);
+ return err;
+ }
+
+ return err;
+}
+
+static void __exit mempolicy_exit(void)
+{
+ if (mempolicy_kobj)
+ kobject_put(mempolicy_kobj);
+}
+
+#else
+static int __init mempolicy_sysfs_init(void)
+{
+ /* A NULL iw_table is interpreted by interleave logic as "all 1s" */
+ iw_table = NULL;
+ return 0;
+}
+
+static void __exit mempolicy_exit(void) { }
+#endif /* CONFIG_SYSFS */
+late_initcall(mempolicy_sysfs_init);
+module_exit(mempolicy_exit);
--
2.39.1
^ permalink raw reply related
* [PATCH v2 2/3] mm/mempolicy: refactor a read-once mechanism into a function for re-use
From: Gregory Price @ 2024-01-19 17:57 UTC (permalink / raw)
To: linux-mm
Cc: linux-kernel, linux-doc, linux-fsdevel, linux-api, corbet, akpm,
gregory.price, honggyu.kim, rakie.kim, hyeongtak.ji, mhocko,
ying.huang, vtavarespetr, jgroves, ravis.opensrc, sthanneeru,
emirakhur, Hasan.Maruf, seungjun.ha, hannes, dan.j.williams
In-Reply-To: <20240119175730.15484-1-gregory.price@memverge.com>
move the use of barrier() to force policy->nodemask onto the stack into
a function `read_once_policy_nodemask` so that it may be re-used.
Suggested-by: Huang Ying <ying.huang@intel.com>
Signed-off-by: Gregory Price <gregory.price@memverge.com>
---
mm/mempolicy.c | 26 ++++++++++++++++----------
1 file changed, 16 insertions(+), 10 deletions(-)
diff --git a/mm/mempolicy.c b/mm/mempolicy.c
index ae925216798f..427bddf115df 100644
--- a/mm/mempolicy.c
+++ b/mm/mempolicy.c
@@ -1893,6 +1893,20 @@ unsigned int mempolicy_slab_node(void)
}
}
+static unsigned int read_once_policy_nodemask(struct mempolicy *pol,
+ nodemask_t *mask)
+{
+ /*
+ * barrier stabilizes the nodemask locally so that it can be iterated
+ * over safely without concern for changes. Allocators validate node
+ * selection does not violate mems_allowed, so this is safe.
+ */
+ barrier();
+ memcpy(mask, &pol->nodes, sizeof(nodemask_t));
+ barrier();
+ return nodes_weight(*mask);
+}
+
/*
* Do static interleaving for interleave index @ilx. Returns the ilx'th
* node in pol->nodes (starting from ilx=0), wrapping around if ilx
@@ -1900,20 +1914,12 @@ unsigned int mempolicy_slab_node(void)
*/
static unsigned int interleave_nid(struct mempolicy *pol, pgoff_t ilx)
{
- nodemask_t nodemask = pol->nodes;
+ nodemask_t nodemask;
unsigned int target, nnodes;
int i;
int nid;
- /*
- * The barrier will stabilize the nodemask in a register or on
- * the stack so that it will stop changing under the code.
- *
- * Between first_node() and next_node(), pol->nodes could be changed
- * by other threads. So we put pol->nodes in a local stack.
- */
- barrier();
- nnodes = nodes_weight(nodemask);
+ nnodes = read_once_policy_nodemask(pol, &nodemask);
if (!nnodes)
return numa_node_id();
target = ilx % nnodes;
--
2.39.1
^ permalink raw reply related
* [PATCH v2 3/3] mm/mempolicy: introduce MPOL_WEIGHTED_INTERLEAVE for weighted interleaving
From: Gregory Price @ 2024-01-19 17:57 UTC (permalink / raw)
To: linux-mm
Cc: linux-kernel, linux-doc, linux-fsdevel, linux-api, corbet, akpm,
gregory.price, honggyu.kim, rakie.kim, hyeongtak.ji, mhocko,
ying.huang, vtavarespetr, jgroves, ravis.opensrc, sthanneeru,
emirakhur, Hasan.Maruf, seungjun.ha, hannes, dan.j.williams,
Srinivasulu Thanneeru
In-Reply-To: <20240119175730.15484-1-gregory.price@memverge.com>
When a system has multiple NUMA nodes and it becomes bandwidth hungry,
using the current MPOL_INTERLEAVE could be an wise option.
However, if those NUMA nodes consist of different types of memory such
as socket-attached DRAM and CXL/PCIe attached DRAM, the round-robin
based interleave policy does not optimally distribute data to make use
of their different bandwidth characteristics.
Instead, interleave is more effective when the allocation policy follows
each NUMA nodes' bandwidth weight rather than a simple 1:1 distribution.
This patch introduces a new memory policy, MPOL_WEIGHTED_INTERLEAVE,
enabling weighted interleave between NUMA nodes. Weighted interleave
allows for proportional distribution of memory across multiple numa
nodes, preferably apportioned to match the bandwidth of each node.
For example, if a system has 1 CPU node (0), and 2 memory nodes (0,1),
with bandwidth of (100GB/s, 50GB/s) respectively, the appropriate
weight distribution is (2:1).
Weights for each node can be assigned via the new sysfs extension:
/sys/kernel/mm/mempolicy/weighted_interleave/
For now, the default value of all nodes will be `1`, which matches
the behavior of standard 1:1 round-robin interleave. An extension
will be added in the future to allow default values to be registered
at kernel and device bringup time.
The policy allocates a number of pages equal to the set weights. For
example, if the weights are (2,1), then 2 pages will be allocated on
node0 for every 1 page allocated on node1.
The new flag MPOL_WEIGHTED_INTERLEAVE can be used in set_mempolicy(2)
and mbind(2).
There are 3 integration points:
weighted_interleave_nodes:
Counts the number of allocations as they occur, and applies the
weight for the current node. When the weight reaches 0, switch
to the next node.
weighted_interleave_nid:
Gets the total weight of the nodemask as well as each individual
node weight, then calculates the node based on the given index.
bulk_array_weighted_interleave:
Gets the total weight of the nodemask as well as each individual
node weight, then calculates the number of "interleave rounds" as
well as any delta ("partial round"). Calculates the number of
pages for each node and allocates them.
If a node was scheduled for interleave via interleave_nodes, the
current weight (pol->cur_weight) will be allocated first, before
the remaining bulk calculation is done.
One piece of complexity is the interaction between a recent refactor
which split the logic to acquire the "ilx" (interleave index) of an
allocation and the actually application of the interleave. The
calculation of the `interleave index` is done by `get_vma_policy()`,
while the actual selection of the node will be later appliex by the
relevant weighted_interleave function.
Suggested-by: Hasan Al Maruf <Hasan.Maruf@amd.com>
Signed-off-by: Gregory Price <gregory.price@memverge.com>
Co-developed-by: Rakie Kim <rakie.kim@sk.com>
Signed-off-by: Rakie Kim <rakie.kim@sk.com>
Co-developed-by: Honggyu Kim <honggyu.kim@sk.com>
Signed-off-by: Honggyu Kim <honggyu.kim@sk.com>
Co-developed-by: Hyeongtak Ji <hyeongtak.ji@sk.com>
Signed-off-by: Hyeongtak Ji <hyeongtak.ji@sk.com>
Co-developed-by: Srinivasulu Thanneeru <sthanneeru.opensrc@micron.com>
Signed-off-by: Srinivasulu Thanneeru <sthanneeru.opensrc@micron.com>
Co-developed-by: Ravi Jonnalagadda <ravis.opensrc@micron.com>
Signed-off-by: Ravi Jonnalagadda <ravis.opensrc@micron.com>
---
.../admin-guide/mm/numa_memory_policy.rst | 9 +
include/linux/mempolicy.h | 5 +
include/uapi/linux/mempolicy.h | 1 +
mm/mempolicy.c | 234 +++++++++++++++++-
4 files changed, 246 insertions(+), 3 deletions(-)
diff --git a/Documentation/admin-guide/mm/numa_memory_policy.rst b/Documentation/admin-guide/mm/numa_memory_policy.rst
index eca38fa81e0f..a70f20ce1ffb 100644
--- a/Documentation/admin-guide/mm/numa_memory_policy.rst
+++ b/Documentation/admin-guide/mm/numa_memory_policy.rst
@@ -250,6 +250,15 @@ MPOL_PREFERRED_MANY
can fall back to all existing numa nodes. This is effectively
MPOL_PREFERRED allowed for a mask rather than a single node.
+MPOL_WEIGHTED_INTERLEAVE
+ This mode operates the same as MPOL_INTERLEAVE, except that
+ interleaving behavior is executed based on weights set in
+ /sys/kernel/mm/mempolicy/weighted_interleave/
+
+ Weighted interleave allocates pages on nodes according to a
+ weight. For example if nodes [0,1] are weighted [5,2], 5 pages
+ will be allocated on node0 for every 2 pages allocated on node1.
+
NUMA memory policy supports the following optional mode flags:
MPOL_F_STATIC_NODES
diff --git a/include/linux/mempolicy.h b/include/linux/mempolicy.h
index 931b118336f4..c1a083eb0dd5 100644
--- a/include/linux/mempolicy.h
+++ b/include/linux/mempolicy.h
@@ -54,6 +54,11 @@ struct mempolicy {
nodemask_t cpuset_mems_allowed; /* relative to these nodes */
nodemask_t user_nodemask; /* nodemask passed by user */
} w;
+
+ /* Weighted interleave settings */
+ struct {
+ u8 cur_weight;
+ } wil;
};
/*
diff --git a/include/uapi/linux/mempolicy.h b/include/uapi/linux/mempolicy.h
index a8963f7ef4c2..1f9bb10d1a47 100644
--- a/include/uapi/linux/mempolicy.h
+++ b/include/uapi/linux/mempolicy.h
@@ -23,6 +23,7 @@ enum {
MPOL_INTERLEAVE,
MPOL_LOCAL,
MPOL_PREFERRED_MANY,
+ MPOL_WEIGHTED_INTERLEAVE,
MPOL_MAX, /* always last member of enum */
};
diff --git a/mm/mempolicy.c b/mm/mempolicy.c
index 427bddf115df..aa3b2389d3e0 100644
--- a/mm/mempolicy.c
+++ b/mm/mempolicy.c
@@ -19,6 +19,13 @@
* for anonymous memory. For process policy an process counter
* is used.
*
+ * weighted interleave
+ * Allocate memory interleaved over a set of nodes based on
+ * a set of weights (per-node), with normal fallback if it
+ * fails. Otherwise operates the same as interleave.
+ * Example: nodeset(0,1) & weights (2,1) - 2 pages allocated
+ * on node 0 for every 1 page allocated on node 1.
+ *
* bind Only allocate memory on a specific set of nodes,
* no fallback.
* FIXME: memory is allocated starting with the first node
@@ -313,6 +320,7 @@ static struct mempolicy *mpol_new(unsigned short mode, unsigned short flags,
policy->mode = mode;
policy->flags = flags;
policy->home_node = NUMA_NO_NODE;
+ policy->wil.cur_weight = 0;
return policy;
}
@@ -425,6 +433,10 @@ static const struct mempolicy_operations mpol_ops[MPOL_MAX] = {
.create = mpol_new_nodemask,
.rebind = mpol_rebind_preferred,
},
+ [MPOL_WEIGHTED_INTERLEAVE] = {
+ .create = mpol_new_nodemask,
+ .rebind = mpol_rebind_nodemask,
+ },
};
static bool migrate_folio_add(struct folio *folio, struct list_head *foliolist,
@@ -846,7 +858,8 @@ static long do_set_mempolicy(unsigned short mode, unsigned short flags,
old = current->mempolicy;
current->mempolicy = new;
- if (new && new->mode == MPOL_INTERLEAVE)
+ if (new && (new->mode == MPOL_INTERLEAVE ||
+ new->mode == MPOL_WEIGHTED_INTERLEAVE))
current->il_prev = MAX_NUMNODES-1;
task_unlock(current);
mpol_put(old);
@@ -872,6 +885,7 @@ static void get_policy_nodemask(struct mempolicy *pol, nodemask_t *nodes)
case MPOL_INTERLEAVE:
case MPOL_PREFERRED:
case MPOL_PREFERRED_MANY:
+ case MPOL_WEIGHTED_INTERLEAVE:
*nodes = pol->nodes;
break;
case MPOL_LOCAL:
@@ -956,6 +970,13 @@ static long do_get_mempolicy(int *policy, nodemask_t *nmask,
} else if (pol == current->mempolicy &&
pol->mode == MPOL_INTERLEAVE) {
*policy = next_node_in(current->il_prev, pol->nodes);
+ } else if (pol == current->mempolicy &&
+ (pol->mode == MPOL_WEIGHTED_INTERLEAVE)) {
+ if (pol->wil.cur_weight)
+ *policy = current->il_prev;
+ else
+ *policy = next_node_in(current->il_prev,
+ pol->nodes);
} else {
err = -EINVAL;
goto out;
@@ -1785,7 +1806,8 @@ struct mempolicy *get_vma_policy(struct vm_area_struct *vma,
pol = __get_vma_policy(vma, addr, ilx);
if (!pol)
pol = get_task_policy(current);
- if (pol->mode == MPOL_INTERLEAVE) {
+ if (pol->mode == MPOL_INTERLEAVE ||
+ pol->mode == MPOL_WEIGHTED_INTERLEAVE) {
*ilx += vma->vm_pgoff >> order;
*ilx += (addr - vma->vm_start) >> (PAGE_SHIFT + order);
}
@@ -1835,6 +1857,28 @@ bool apply_policy_zone(struct mempolicy *policy, enum zone_type zone)
return zone >= dynamic_policy_zone;
}
+static unsigned int weighted_interleave_nodes(struct mempolicy *policy)
+{
+ unsigned int next;
+ struct task_struct *me = current;
+ u8 __rcu *table;
+
+ next = next_node_in(me->il_prev, policy->nodes);
+ if (next == MAX_NUMNODES)
+ return next;
+
+ rcu_read_lock();
+ table = rcu_dereference(iw_table);
+ if (!policy->wil.cur_weight)
+ policy->wil.cur_weight = table ? table[next] : 1;
+ rcu_read_unlock();
+
+ policy->wil.cur_weight--;
+ if (!policy->wil.cur_weight)
+ me->il_prev = next;
+ return next;
+}
+
/* Do dynamic interleaving for a process */
static unsigned int interleave_nodes(struct mempolicy *policy)
{
@@ -1869,6 +1913,9 @@ unsigned int mempolicy_slab_node(void)
case MPOL_INTERLEAVE:
return interleave_nodes(policy);
+ case MPOL_WEIGHTED_INTERLEAVE:
+ return weighted_interleave_nodes(policy);
+
case MPOL_BIND:
case MPOL_PREFERRED_MANY:
{
@@ -1907,6 +1954,39 @@ static unsigned int read_once_policy_nodemask(struct mempolicy *pol,
return nodes_weight(*mask);
}
+static unsigned int weighted_interleave_nid(struct mempolicy *pol, pgoff_t ilx)
+{
+ nodemask_t nodemask;
+ unsigned int target, nr_nodes;
+ u8 __rcu *table;
+ unsigned int weight_total = 0;
+ u8 weight;
+ int nid;
+
+ nr_nodes = read_once_policy_nodemask(pol, &nodemask);
+ if (!nr_nodes)
+ return numa_node_id();
+
+ rcu_read_lock();
+ table = rcu_dereference(iw_table);
+ /* calculate the total weight */
+ for_each_node_mask(nid, nodemask)
+ weight_total += table ? table[nid] : 1;
+
+ /* Calculate the node offset based on totals */
+ target = ilx % weight_total;
+ nid = first_node(nodemask);
+ while (target) {
+ weight = table ? table[nid] : 1;
+ if (target < weight)
+ break;
+ target -= weight;
+ nid = next_node_in(nid, nodemask);
+ }
+ rcu_read_unlock();
+ return nid;
+}
+
/*
* Do static interleaving for interleave index @ilx. Returns the ilx'th
* node in pol->nodes (starting from ilx=0), wrapping around if ilx
@@ -1967,6 +2047,11 @@ static nodemask_t *policy_nodemask(gfp_t gfp, struct mempolicy *pol,
*nid = (ilx == NO_INTERLEAVE_INDEX) ?
interleave_nodes(pol) : interleave_nid(pol, ilx);
break;
+ case MPOL_WEIGHTED_INTERLEAVE:
+ *nid = (ilx == NO_INTERLEAVE_INDEX) ?
+ weighted_interleave_nodes(pol) :
+ weighted_interleave_nid(pol, ilx);
+ break;
}
return nodemask;
@@ -2028,6 +2113,7 @@ bool init_nodemask_of_mempolicy(nodemask_t *mask)
case MPOL_PREFERRED_MANY:
case MPOL_BIND:
case MPOL_INTERLEAVE:
+ case MPOL_WEIGHTED_INTERLEAVE:
*mask = mempolicy->nodes;
break;
@@ -2127,7 +2213,8 @@ struct page *alloc_pages_mpol(gfp_t gfp, unsigned int order,
* If the policy is interleave or does not allow the current
* node in its nodemask, we allocate the standard way.
*/
- if (pol->mode != MPOL_INTERLEAVE &&
+ if ((pol->mode != MPOL_INTERLEAVE &&
+ pol->mode != MPOL_WEIGHTED_INTERLEAVE) &&
(!nodemask || node_isset(nid, *nodemask))) {
/*
* First, try to allocate THP only on local node, but
@@ -2263,6 +2350,135 @@ static unsigned long alloc_pages_bulk_array_interleave(gfp_t gfp,
return total_allocated;
}
+static unsigned long alloc_pages_bulk_array_weighted_interleave(gfp_t gfp,
+ struct mempolicy *pol, unsigned long nr_pages,
+ struct page **page_array)
+{
+ struct task_struct *me = current;
+ unsigned long total_allocated = 0;
+ unsigned long nr_allocated;
+ unsigned long rounds;
+ unsigned long node_pages, delta;
+ u8 weight, resume_weight;
+ u8 __rcu *table;
+ u8 *weights;
+ unsigned int weight_total = 0;
+ unsigned long rem_pages = nr_pages;
+ nodemask_t nodes;
+ int nnodes, node, weight_nodes, resume_node;
+ int prev_node = NUMA_NO_NODE;
+ bool delta_depleted = false;
+ int i;
+
+ if (!nr_pages)
+ return 0;
+
+ nnodes = read_once_policy_nodemask(pol, &nodes);
+ if (!nnodes)
+ return 0;
+
+ /* Continue allocating from most recent node and adjust the nr_pages */
+ if (pol->wil.cur_weight) {
+ node = next_node_in(me->il_prev, nodes);
+ node_pages = pol->wil.cur_weight;
+ if (node_pages > rem_pages)
+ node_pages = rem_pages;
+ nr_allocated = __alloc_pages_bulk(gfp, node, NULL, node_pages,
+ NULL, page_array);
+ page_array += nr_allocated;
+ total_allocated += nr_allocated;
+ /* if that's all the pages, no need to interleave */
+ if (rem_pages <= pol->wil.cur_weight) {
+ pol->wil.cur_weight -= rem_pages;
+ return total_allocated;
+ }
+ /* Otherwise we adjust nr_pages down, and continue from there */
+ rem_pages -= pol->wil.cur_weight;
+ pol->wil.cur_weight = 0;
+ prev_node = node;
+ }
+
+ /* fetch the weights for this operation and calculate total weight */
+ weights = kmalloc(nnodes, GFP_KERNEL);
+ if (!weights)
+ return total_allocated;
+
+ rcu_read_lock();
+ table = rcu_dereference(iw_table);
+ weight_nodes = 0;
+ while (weight_nodes < nnodes) {
+ node = next_node_in(prev_node, nodes);
+ weight = table ? table[node] : 1;
+ weights[weight_nodes++] = weight;
+ weight_total += weight;
+ }
+ rcu_read_unlock();
+
+ /*
+ * Now we can continue allocating from 0 instead of an offset
+ * We calculate the number of rounds and any partial rounds so
+ * that we minimize the number of calls to __alloc_pages_bulk
+ * This requires us to track which node we should resume from.
+ *
+ * if (rounds > 0) and (delta == 0), resume_node will always be
+ * the current me->il_prev
+ *
+ * if (delta > 0) and delta is depleted exactly on a node-weight
+ * boundary, resume node will be the node last allocated from when
+ * delta reached 0.
+ *
+ * if (delta > 0) and delta is not depleted on a node-weight boundary,
+ * resume node will be the node prior to the node last allocated from.
+ *
+ * (rounds == 0) and (delta == 0) is not possible (earlier exit)
+ */
+ rounds = rem_pages / weight_total;
+ delta = rem_pages % weight_total;
+ /* If no delta, we'll resume from current prev_node and first weight */
+ for (i = 0; i < nnodes; i++) {
+ node = next_node_in(prev_node, nodes);
+ weight = weights[i];
+ node_pages = weight * rounds;
+ /* If a delta exists, add this node's portion of the delta */
+ if (delta >= weight) {
+ node_pages += weight;
+ delta -= weight;
+ resume_node = node;
+ resume_weight = i < (nnodes - 1) ? weights[i+1] :
+ weights[0];
+ /* stop tracking iff (delta == weight) */
+ delta_depleted = !delta;
+ } else if (delta) { /* <= weight */
+ /* if delta depleted, resume from this node */
+ node_pages += delta;
+ delta = 0;
+ resume_node = prev_node;
+ resume_weight = weight - (node_pages % weight);
+ delta_depleted = true; /* stop tracking */
+ } else if (!delta_depleted) {
+ /* if there was no delta, track last allocated node */
+ resume_node = node;
+ resume_weight = i < (nnodes - 1) ? weights[i+1] :
+ weights[0];
+ }
+ /* node_pages can be 0 if an allocation fails and rounds == 0 */
+ if (!node_pages)
+ break;
+ nr_allocated = __alloc_pages_bulk(gfp, node, NULL, node_pages,
+ NULL, page_array);
+ page_array += nr_allocated;
+ total_allocated += nr_allocated;
+ if (total_allocated == nr_pages)
+ break;
+ prev_node = node;
+ }
+ /* resume allocating from the calculated node and weight */
+ me->il_prev = resume_node;
+ pol->wil.cur_weight = resume_weight;
+ kfree(weights);
+ return total_allocated;
+}
+
static unsigned long alloc_pages_bulk_array_preferred_many(gfp_t gfp, int nid,
struct mempolicy *pol, unsigned long nr_pages,
struct page **page_array)
@@ -2303,6 +2519,10 @@ unsigned long alloc_pages_bulk_array_mempolicy(gfp_t gfp,
return alloc_pages_bulk_array_interleave(gfp, pol,
nr_pages, page_array);
+ if (pol->mode == MPOL_WEIGHTED_INTERLEAVE)
+ return alloc_pages_bulk_array_weighted_interleave(
+ gfp, pol, nr_pages, page_array);
+
if (pol->mode == MPOL_PREFERRED_MANY)
return alloc_pages_bulk_array_preferred_many(gfp,
numa_node_id(), pol, nr_pages, page_array);
@@ -2378,6 +2598,7 @@ bool __mpol_equal(struct mempolicy *a, struct mempolicy *b)
case MPOL_INTERLEAVE:
case MPOL_PREFERRED:
case MPOL_PREFERRED_MANY:
+ case MPOL_WEIGHTED_INTERLEAVE:
return !!nodes_equal(a->nodes, b->nodes);
case MPOL_LOCAL:
return true;
@@ -2514,6 +2735,10 @@ int mpol_misplaced(struct folio *folio, struct vm_area_struct *vma,
polnid = interleave_nid(pol, ilx);
break;
+ case MPOL_WEIGHTED_INTERLEAVE:
+ polnid = weighted_interleave_nid(pol, ilx);
+ break;
+
case MPOL_PREFERRED:
if (node_isset(curnid, pol->nodes))
goto out;
@@ -2888,6 +3113,7 @@ static const char * const policy_modes[] =
[MPOL_PREFERRED] = "prefer",
[MPOL_BIND] = "bind",
[MPOL_INTERLEAVE] = "interleave",
+ [MPOL_WEIGHTED_INTERLEAVE] = "weighted interleave",
[MPOL_LOCAL] = "local",
[MPOL_PREFERRED_MANY] = "prefer (many)",
};
@@ -2947,6 +3173,7 @@ int mpol_parse_str(char *str, struct mempolicy **mpol)
}
break;
case MPOL_INTERLEAVE:
+ case MPOL_WEIGHTED_INTERLEAVE:
/*
* Default to online nodes with memory if no nodelist
*/
@@ -3057,6 +3284,7 @@ void mpol_to_str(char *buffer, int maxlen, struct mempolicy *pol)
case MPOL_PREFERRED_MANY:
case MPOL_BIND:
case MPOL_INTERLEAVE:
+ case MPOL_WEIGHTED_INTERLEAVE:
nodes = pol->nodes;
break;
default:
--
2.39.1
^ permalink raw reply related
* Re: [PATCH v2] vfs: add RWF_NOAPPEND flag for pwritev2
From: Rich Felker @ 2024-01-19 19:10 UTC (permalink / raw)
To: Christian Brauner
Cc: Jens Axboe, linux-fsdevel, kernel list, Linux API, Alexander Viro,
Pavel Begunkov, Jann Horn
In-Reply-To: <20240119-neuverfilmung-aufregend-54a5bd5929dd@brauner>
On Fri, Jan 19, 2024 at 03:33:32PM +0100, Christian Brauner wrote:
> On Mon, 31 Aug 2020 11:32:08 -0400, Rich Felker wrote:
> > The pwrite function, originally defined by POSIX (thus the "p"), is
> > defined to ignore O_APPEND and write at the offset passed as its
> > argument. However, historically Linux honored O_APPEND if set and
> > ignored the offset. This cannot be changed due to stability policy,
> > but is documented in the man page as a bug.
> >
> > Now that there's a pwritev2 syscall providing a superset of the pwrite
> > functionality that has a flags argument, the conforming behavior can
> > be offered to userspace via a new flag. Since pwritev2 checks flag
> > validity (in kiocb_set_rw_flags) and reports unknown ones with
> > EOPNOTSUPP, callers will not get wrong behavior on old kernels that
> > don't support the new flag; the error is reported and the caller can
> > decide how to handle it.
> >
> > [...]
>
> The RWF_* and IOCB_* flags were
> aligned so they could be set in one operation. So there was a merge
> conflict when applying. I've resolved it. Please take a look and make
> sure that it's all correct.
>
> ---
>
> Applied to the vfs.misc branch of the vfs/vfs.git tree.
> Patches in the vfs.misc branch should appear in linux-next soon.
>
> Please report any outstanding bugs that were missed during review in a
> new review to the original patch series allowing us to drop it.
>
> It's encouraged to provide Acked-bys and Reviewed-bys even though the
> patch has now been applied. If possible patch trailers will be updated.
>
> Note that commit hashes shown below are subject to change due to rebase,
> trailer updates or similar. If in doubt, please check the listed branch.
>
> tree: https://git.kernel.org/pub/scm/linux/kernel/git/vfs/vfs.git
> branch: vfs.misc
>
> [1/1] vfs: add RWF_NOAPPEND flag for pwritev2
> https://git.kernel.org/vfs/vfs/c/31081ab305a1
LGTM. Thanks!
^ permalink raw reply
* [RFC 0/1] RFC: Allow busy poll to be set per epoll instance
From: Joe Damato @ 2024-01-20 0:42 UTC (permalink / raw)
To: netdev, linux-kernel
Cc: chuck.lever, jlayton, linux-api, edumazet, davem, alexander.duyck,
sridhar.samudrala, kuba, Joe Damato
Greetings:
TL;DR This RFC builds on bf3b9f6372c4 ("epoll: Add busy poll support to
epoll with socket fds.") by adding two fcntl knobs for enabling
epoll-based busy poll on a per epoll basis instead of the current
system-wide sysctl. This change makes epoll-based busy poll much more
usable.
I have another implementation which uses epoll_ctl and adds a new
EPOLL_CTL_BUSY_POLL_TIMEOUT knob instead of using fcntl, but fcntl
seemed to be slightly cleaner.
I am happy to use whatever interface is desired by the kernel community
in order to allow for per-epoll instance busy poll to be supported.
Longer explanation:
Presently epoll has support for a very useful form of busy poll based on
the incoming NAPI ID (see also: SO_INCOMING_NAPI_ID [1]).
This form of busy poll allows epoll_wait to drive NAPI packet processing
which can allow for user applications to decide when it is appropriate
to process network data vs being pre-empted during less optimal times.
For example, a network application might process an entire datagram and
get better use of L2/L3 cache by deferring packet processing until all
events are processed and epoll_wait is called.
The documentation available on this is, IMHO, a bit confusing so please
allow me to explain how to use this kernel feature.
In order to use this feature, user applications must do three things:
1. Ensure each application thread has its own epoll instance mapping
1-to-1 with NIC RX queues. An n-tuple filter would likely be used to
direct connections with specific dest ports to these queues.
2. Ensure that all incoming connections added to an epoll instance
have the same NAPI ID. This can be done with a BPF filter when
SO_REUSEPORT is used or getsockopt + SO_INCOMING_NAPI_ID when a single
accept thread is used which dispatches incoming connections to threads.
3. Lastly, busy poll must be enabled via a sysctl
(/proc/sys/net/core/busy_poll).
The unfortunate part about step 3 above is that this enables busy poll
system-wide which affects all user applications on the system.
It is worth noting that setting /proc/sys/net/core/busy_poll has
different effects on different system calls:
- poll and select based applications would not be affected as busy
polling is only enabled when this sysctl is set *and* sockets have
SO_BUSY_POLL set.
- All epoll based applications on the system, however, will busy poll
when this sysctl is set.
If the user wants to run one low latency epoll-based server application with
epoll-based busy poll, but would like to run the rest of the applications on
the system (which may also use epoll) without busy poll, this
system-wide sysctl presents a significant problem.
This change preserves the system-wide sysctl, but adds a mechanism (via
fcntl) to enable or disable busy poll for epoll instances as needed.
This change is extremely useful for low latency network applications
that need to run side-by-side with other network applications where
latency is not a major concern.
As mentioned above, the epoll_ctl approach I have (which works) seemed
less clean than the fcntl approach in this RFC. I would be happy to use
whatever interface the kernel maintainers prefer to make epoll based busy
poll more convenient for user applications to use.
Thanks,
Joe
[1]: https://lore.kernel.org/lkml/20170324170836.15226.87178.stgit@localhost.localdomain/
Joe Damato (1):
eventpoll: support busy poll per epoll instance
fs/eventpoll.c | 71 ++++++++++++++++++++++++++++++--
fs/fcntl.c | 5 +++
include/linux/eventpoll.h | 2 +
include/uapi/linux/fcntl.h | 6 +++
tools/include/uapi/linux/fcntl.h | 6 +++
tools/perf/trace/beauty/fcntl.c | 3 +-
6 files changed, 88 insertions(+), 5 deletions(-)
--
2.25.1
^ permalink raw reply
* [RFC 1/1] eventpoll: support busy poll per epoll instance
From: Joe Damato @ 2024-01-20 0:42 UTC (permalink / raw)
To: netdev, linux-kernel
Cc: chuck.lever, jlayton, linux-api, edumazet, davem, alexander.duyck,
sridhar.samudrala, kuba, Joe Damato
In-Reply-To: <20240120004247.42036-1-jdamato@fastly.com>
Add F_EPOLL_{S,G}ET_BUSY_POLL_USECS to allow setting a busy poll timeout
per epoll instance so that individual applications can enable (or
disable) epoll based busy poll as needed.
Prior to this change, epoll-based busy poll could only be enabled
system-wide, which limits the usefulness of busy poll.
Signed-off-by: Joe Damato <jdamato@fastly.com>
---
fs/eventpoll.c | 71 ++++++++++++++++++++++++++++++--
fs/fcntl.c | 5 +++
include/linux/eventpoll.h | 2 +
include/uapi/linux/fcntl.h | 6 +++
tools/include/uapi/linux/fcntl.h | 6 +++
tools/perf/trace/beauty/fcntl.c | 3 +-
6 files changed, 88 insertions(+), 5 deletions(-)
diff --git a/fs/eventpoll.c b/fs/eventpoll.c
index 3534d36a1474..a8087c2b47ef 100644
--- a/fs/eventpoll.c
+++ b/fs/eventpoll.c
@@ -227,6 +227,8 @@ struct eventpoll {
#ifdef CONFIG_NET_RX_BUSY_POLL
/* used to track busy poll napi_id */
unsigned int napi_id;
+ /* busy poll timeout */
+ u64 busy_poll_usecs;
#endif
#ifdef CONFIG_DEBUG_LOCK_ALLOC
@@ -386,12 +388,39 @@ static inline int ep_events_available(struct eventpoll *ep)
READ_ONCE(ep->ovflist) != EP_UNACTIVE_PTR;
}
+/**
+ * busy_loop_ep_timeout - check if busy poll has timed out. The timeout value
+ * from the epoll instance ep is preferred, but if it is not set fallback to
+ * the system-wide global via busy_loop_timeout.
+ */
+static inline bool busy_loop_ep_timeout(unsigned long start_time, struct eventpoll *ep)
+{
#ifdef CONFIG_NET_RX_BUSY_POLL
+ unsigned long bp_usec = READ_ONCE(ep->busy_poll_usecs);
+
+ if (bp_usec) {
+ unsigned long end_time = start_time + bp_usec;
+ unsigned long now = busy_loop_current_time();
+
+ return time_after(now, end_time);
+ } else {
+ return busy_loop_timeout(start_time);
+ }
+#endif
+ return true;
+}
+
+#ifdef CONFIG_NET_RX_BUSY_POLL
+static bool ep_busy_loop_on(struct eventpoll *ep)
+{
+ return !!ep->busy_poll_usecs ^ net_busy_loop_on();
+}
+
static bool ep_busy_loop_end(void *p, unsigned long start_time)
{
struct eventpoll *ep = p;
- return ep_events_available(ep) || busy_loop_timeout(start_time);
+ return ep_events_available(ep) || busy_loop_ep_timeout(start_time, ep);
}
/*
@@ -404,7 +433,7 @@ static bool ep_busy_loop(struct eventpoll *ep, int nonblock)
{
unsigned int napi_id = READ_ONCE(ep->napi_id);
- if ((napi_id >= MIN_NAPI_ID) && net_busy_loop_on()) {
+ if ((napi_id >= MIN_NAPI_ID) && ep_busy_loop_on(ep)) {
napi_busy_loop(napi_id, nonblock ? NULL : ep_busy_loop_end, ep, false,
BUSY_POLL_BUDGET);
if (ep_events_available(ep))
@@ -430,7 +459,8 @@ static inline void ep_set_busy_poll_napi_id(struct epitem *epi)
struct socket *sock;
struct sock *sk;
- if (!net_busy_loop_on())
+ ep = epi->ep;
+ if (!ep_busy_loop_on(ep))
return;
sock = sock_from_file(epi->ffd.file);
@@ -442,7 +472,6 @@ static inline void ep_set_busy_poll_napi_id(struct epitem *epi)
return;
napi_id = READ_ONCE(sk->sk_napi_id);
- ep = epi->ep;
/* Non-NAPI IDs can be rejected
* or
@@ -466,6 +495,10 @@ static inline void ep_set_busy_poll_napi_id(struct epitem *epi)
{
}
+static inline bool ep_busy_loop_on(struct eventpoll *ep)
+{
+ return false;
+}
#endif /* CONFIG_NET_RX_BUSY_POLL */
/*
@@ -933,6 +966,33 @@ static const struct file_operations eventpoll_fops = {
.llseek = noop_llseek,
};
+unsigned long eventpoll_fcntl(struct file *file, unsigned int cmd, unsigned long arg)
+{
+ int ret;
+ struct eventpoll *ep;
+
+ if (!is_file_epoll(file))
+ return -EINVAL;
+
+ ep = file->private_data;
+
+ switch (cmd) {
+#ifdef CONFIG_NET_RX_BUSY_POLL
+ case F_EPOLL_SET_BUSY_POLL_USECS:
+ ret = ep->busy_poll_usecs = arg;
+ break;
+ case F_EPOLL_GET_BUSY_POLL_USECS:
+ ret = ep->busy_poll_usecs;
+ break;
+#endif
+ default:
+ ret = -EINVAL;
+ break;
+ }
+
+ return ret;
+}
+
/*
* This is called from eventpoll_release() to unlink files from the eventpoll
* interface. We need to have this facility to cleanup correctly files that are
@@ -2058,6 +2118,9 @@ static int do_epoll_create(int flags)
error = PTR_ERR(file);
goto out_free_fd;
}
+#ifndef CONFIG_NET_RX_BUSY_POLL
+ ep->busy_poll_usecs = 0;
+#endif
ep->file = file;
fd_install(fd, file);
return fd;
diff --git a/fs/fcntl.c b/fs/fcntl.c
index c80a6acad742..f232e7c2eb9d 100644
--- a/fs/fcntl.c
+++ b/fs/fcntl.c
@@ -9,6 +9,7 @@
#include <linux/init.h>
#include <linux/mm.h>
#include <linux/sched/task.h>
+#include <linux/eventpoll.h>
#include <linux/fs.h>
#include <linux/filelock.h>
#include <linux/file.h>
@@ -419,6 +420,10 @@ static long do_fcntl(int fd, unsigned int cmd, unsigned long arg,
case F_SET_RW_HINT:
err = fcntl_rw_hint(filp, cmd, arg);
break;
+ case F_EPOLL_GET_BUSY_POLL_USECS:
+ case F_EPOLL_SET_BUSY_POLL_USECS:
+ err = eventpoll_fcntl(filp, cmd, arg);
+ break;
default:
break;
}
diff --git a/include/linux/eventpoll.h b/include/linux/eventpoll.h
index 3337745d81bd..3e6a49d14f52 100644
--- a/include/linux/eventpoll.h
+++ b/include/linux/eventpoll.h
@@ -22,6 +22,8 @@ struct file;
struct file *get_epoll_tfile_raw_ptr(struct file *file, int tfd, unsigned long toff);
#endif
+unsigned long eventpoll_fcntl(struct file *file, unsigned int cmd, unsigned long arg);
+
/* Used to release the epoll bits inside the "struct file" */
void eventpoll_release_file(struct file *file);
diff --git a/include/uapi/linux/fcntl.h b/include/uapi/linux/fcntl.h
index 282e90aeb163..522134ab9580 100644
--- a/include/uapi/linux/fcntl.h
+++ b/include/uapi/linux/fcntl.h
@@ -56,6 +56,12 @@
#define F_GET_FILE_RW_HINT (F_LINUX_SPECIFIC_BASE + 13)
#define F_SET_FILE_RW_HINT (F_LINUX_SPECIFIC_BASE + 14)
+/*
+ * Set/Get busy poll usecs for an epoll instance.
+ */
+#define F_EPOLL_GET_BUSY_POLL_USECS (F_LINUX_SPECIFIC_BASE + 15)
+#define F_EPOLL_SET_BUSY_POLL_USECS (F_LINUX_SPECIFIC_BASE + 16)
+
/*
* Valid hint values for F_{GET,SET}_RW_HINT. 0 is "not set", or can be
* used to clear any hints previously set.
diff --git a/tools/include/uapi/linux/fcntl.h b/tools/include/uapi/linux/fcntl.h
index 6c80f96049bd..1937f8b74783 100644
--- a/tools/include/uapi/linux/fcntl.h
+++ b/tools/include/uapi/linux/fcntl.h
@@ -56,6 +56,12 @@
#define F_GET_FILE_RW_HINT (F_LINUX_SPECIFIC_BASE + 13)
#define F_SET_FILE_RW_HINT (F_LINUX_SPECIFIC_BASE + 14)
+/*
+ * Set/Get busy poll usecs for an epoll instance.
+ */
+#define F_EPOLL_GET_BUSY_POLL_USECS (F_LINUX_SPECIFIC_BASE + 15)
+#define F_EPOLL_SET_BUSY_POLL_USECS (F_LINUX_SPECIFIC_BASE + 16)
+
/*
* Valid hint values for F_{GET,SET}_RW_HINT. 0 is "not set", or can be
* used to clear any hints previously set.
diff --git a/tools/perf/trace/beauty/fcntl.c b/tools/perf/trace/beauty/fcntl.c
index 56ef83b3d130..dae5647c5c1a 100644
--- a/tools/perf/trace/beauty/fcntl.c
+++ b/tools/perf/trace/beauty/fcntl.c
@@ -94,7 +94,8 @@ size_t syscall_arg__scnprintf_fcntl_arg(char *bf, size_t size, struct syscall_ar
cmd == F_OFD_SETLK || cmd == F_OFD_SETLKW || cmd == F_OFD_GETLK ||
cmd == F_GETOWN_EX || cmd == F_SETOWN_EX ||
cmd == F_GET_RW_HINT || cmd == F_SET_RW_HINT ||
- cmd == F_GET_FILE_RW_HINT || cmd == F_SET_FILE_RW_HINT)
+ cmd == F_GET_FILE_RW_HINT || cmd == F_SET_FILE_RW_HINT ||
+ cmd == F_EPOLL_GET_BUSY_POLL_USECS || cmd == F_EPOLL_SET_BUSY_POLL_USECS)
return syscall_arg__scnprintf_hex(bf, size, arg);
return syscall_arg__scnprintf_long(bf, size, arg);
--
2.25.1
^ permalink raw reply related
* Re: [PATCH v2 1/1] mm/madvise: add MADV_F_COLLAPSE_LIGHT to process_madvise()
From: Lance Yang @ 2024-01-21 3:12 UTC (permalink / raw)
To: akpm
Cc: zokeefe, david, songmuchun, shy828301, peterx, mknyszek, minchan,
mhocko, linux-mm, linux-kernel, linux-api
In-Reply-To: <20240118120347.61817-1-ioworker0@gmail.com>
Hello Everyone,
For applications actively utilizing THP, the defrag mode may
not be a very user-friendly design. Here are the reasons:
1. Before marking the address space with
MADV_HUGEPAGE,it is necessary to check if
the current configuration of the defrag mode aligns with
their preferences.
2. Once the defrag mode configuration changes, these
applications may face the risk of unpredictable stalls.
THP is an important feature of the Linux kernel that can
significantly enhance memory access performance.
However, due to the lack of fine-grained control over
the huge page allocation strategy, many applications
default to not using huge pages and even recommend
users to disable THP. This situation is regrettable.
With the introduction of MADV_COLLAPSE into the kernel,
it is not affected by the defrag mode.
MADV_COLLAPSE offers the potential for
fine-grained synchronous control over the huge page
allocation mechanism, marking a significant enhancement
for THP.
By adding flags to MADV_COLLAPSE, different
synchronous allocation strategies can be provided to
applications. This can instill confidence in them, allowing
them to reconsider using THP and allocate huge pages
according to their desired synchronous allocation strategy,
without worrying about the defrag mode configuration.
BR,
Lance
On Thu, Jan 18, 2024 at 8:03 PM Lance Yang <ioworker0@gmail.com> wrote:
>
> This idea was inspired by MADV_COLLAPSE introduced by Zach O'Keefe[1].
>
> Allow MADV_F_COLLAPSE_LIGHT behavior for process_madvise(2) if the caller
> has CAP_SYS_ADMIN or is requesting the collapse of its own memory.
>
> The semantics of MADV_F_COLLAPSE_LIGHT are similar to MADV_COLLAPSE, but
> it avoids direct reclaim and/or compaction, quickly failing on allocation
> errors.
>
> This change enables a more flexible and efficient usage of memory collapse
> operations, providing additional control to userspace applications for
> system-wide THP optimization.
>
> Semantics
>
> This call is independent of the system-wide THP sysfs settings, but will
> fail for memory marked VM_NOHUGEPAGE. If the ranges provided span
> multiple VMAs, the semantics of the collapse over each VMA is independent
> from the others. This implies a hugepage cannot cross a VMA boundary. If
> collapse of a given hugepage-aligned/sized region fails, the operation may
> continue to attempt collapsing the remainder of memory specified.
>
> The memory ranges provided must be page-aligned, but are not required to
> be hugepage-aligned. If the memory ranges are not hugepage-aligned, the
> start/end of the range will be clamped to the first/last hugepage-aligned
> address covered by said range. The memory ranges must span at least one
> hugepage-sized region.
>
> All non-resident pages covered by the range will first be
> swapped/faulted-in, before being internally copied onto a freshly
> allocated hugepage. Unmapped pages will have their data directly
> initialized to 0 in the new hugepage. However, for every eligible
> hugepage aligned/sized region to-be collapsed, at least one page must
> currently be backed by memory (a PMD covering the address range must
> already exist).
>
> Allocation for the new hugepage will not enter direct reclaim and/or
> compaction, quickly failing if allocation fails. When the system has
> multiple NUMA nodes, the hugepage will be allocated from the node providing
> the most native pages. This operation operates on the current state of the
> specified process and makes no persistent changes or guarantees on how pages
> will be mapped, constructed, or faulted in the future.
>
> Use Cases
>
> An immediate user of this new functionality is the Go runtime heap allocator
> that manages memory in hugepage-sized chunks. In the past, whether it was a
> newly allocated chunk through mmap() or a reused chunk released by
> madvise(MADV_DONTNEED), the allocator attempted to eagerly back memory with
> huge pages using madvise(MADV_HUGEPAGE)[2] and madvise(MADV_COLLAPSE)[3]
> respectively. However, both approaches resulted in performance issues; for
> both scenarios, there could be entries into direct reclaim and/or compaction,
> leading to unpredictable stalls[4]. Now, the allocator can confidently use
> process_madvise(MADV_F_COLLAPSE_LIGHT) to attempt the allocation of huge pages.
>
> [1] https://github.com/torvalds/linux/commit/7d8faaf155454f8798ec56404faca29a82689c77
> [2] https://github.com/golang/go/commit/8fa9e3beee8b0e6baa7333740996181268b60a3a
> [3] https://github.com/golang/go/commit/9f9bb26880388c5bead158e9eca3be4b3a9bd2af
> [4] https://github.com/golang/go/issues/63334
>
> [v1] https://lore.kernel.org/lkml/20240117050217.43610-1-ioworker0@gmail.com/
>
> Signed-off-by: Lance Yang <ioworker0@gmail.com>
> Suggested-by: Zach O'Keefe <zokeefe@google.com>
> Suggested-by: David Hildenbrand <david@redhat.com>
> ---
> V1 -> V2: Treat process_madvise(MADV_F_COLLAPSE_LIGHT) as the lighter-weight alternative
> to madvise(MADV_COLLAPSE)
>
> arch/alpha/include/uapi/asm/mman.h | 1 +
> arch/mips/include/uapi/asm/mman.h | 1 +
> arch/parisc/include/uapi/asm/mman.h | 1 +
> arch/xtensa/include/uapi/asm/mman.h | 1 +
> include/linux/huge_mm.h | 5 +--
> include/uapi/asm-generic/mman-common.h | 1 +
> mm/khugepaged.c | 15 ++++++--
> mm/madvise.c | 36 +++++++++++++++++---
> tools/include/uapi/asm-generic/mman-common.h | 1 +
> 9 files changed, 52 insertions(+), 10 deletions(-)
>
> diff --git a/arch/alpha/include/uapi/asm/mman.h b/arch/alpha/include/uapi/asm/mman.h
> index 763929e814e9..22f23ca04f1a 100644
> --- a/arch/alpha/include/uapi/asm/mman.h
> +++ b/arch/alpha/include/uapi/asm/mman.h
> @@ -77,6 +77,7 @@
> #define MADV_DONTNEED_LOCKED 24 /* like DONTNEED, but drop locked pages too */
>
> #define MADV_COLLAPSE 25 /* Synchronous hugepage collapse */
> +#define MADV_F_COLLAPSE_LIGHT 26 /* Similar to COLLAPSE, but avoids direct reclaim and/or compaction */
>
> /* compatibility flags */
> #define MAP_FILE 0
> diff --git a/arch/mips/include/uapi/asm/mman.h b/arch/mips/include/uapi/asm/mman.h
> index c6e1fc77c996..acec0b643e9c 100644
> --- a/arch/mips/include/uapi/asm/mman.h
> +++ b/arch/mips/include/uapi/asm/mman.h
> @@ -104,6 +104,7 @@
> #define MADV_DONTNEED_LOCKED 24 /* like DONTNEED, but drop locked pages too */
>
> #define MADV_COLLAPSE 25 /* Synchronous hugepage collapse */
> +#define MADV_F_COLLAPSE_LIGHT 26 /* Similar to COLLAPSE, but avoids direct reclaim and/or compaction */
>
> /* compatibility flags */
> #define MAP_FILE 0
> diff --git a/arch/parisc/include/uapi/asm/mman.h b/arch/parisc/include/uapi/asm/mman.h
> index 68c44f99bc93..812029c98cd7 100644
> --- a/arch/parisc/include/uapi/asm/mman.h
> +++ b/arch/parisc/include/uapi/asm/mman.h
> @@ -71,6 +71,7 @@
> #define MADV_DONTNEED_LOCKED 24 /* like DONTNEED, but drop locked pages too */
>
> #define MADV_COLLAPSE 25 /* Synchronous hugepage collapse */
> +#define MADV_F_COLLAPSE_LIGHT 26 /* Similar to COLLAPSE, but avoids direct reclaim and/or compaction */
>
> #define MADV_HWPOISON 100 /* poison a page for testing */
> #define MADV_SOFT_OFFLINE 101 /* soft offline page for testing */
> diff --git a/arch/xtensa/include/uapi/asm/mman.h b/arch/xtensa/include/uapi/asm/mman.h
> index 1ff0c858544f..52ef463dd5b6 100644
> --- a/arch/xtensa/include/uapi/asm/mman.h
> +++ b/arch/xtensa/include/uapi/asm/mman.h
> @@ -112,6 +112,7 @@
> #define MADV_DONTNEED_LOCKED 24 /* like DONTNEED, but drop locked pages too */
>
> #define MADV_COLLAPSE 25 /* Synchronous hugepage collapse */
> +#define MADV_F_COLLAPSE_LIGHT 26 /* Similar to COLLAPSE, but avoids direct reclaim and/or compaction */
>
> /* compatibility flags */
> #define MAP_FILE 0
> diff --git a/include/linux/huge_mm.h b/include/linux/huge_mm.h
> index 5adb86af35fc..075fdb5d481a 100644
> --- a/include/linux/huge_mm.h
> +++ b/include/linux/huge_mm.h
> @@ -303,7 +303,7 @@ int hugepage_madvise(struct vm_area_struct *vma, unsigned long *vm_flags,
> int advice);
> int madvise_collapse(struct vm_area_struct *vma,
> struct vm_area_struct **prev,
> - unsigned long start, unsigned long end);
> + unsigned long start, unsigned long end, int behavior);
> void vma_adjust_trans_huge(struct vm_area_struct *vma, unsigned long start,
> unsigned long end, long adjust_next);
> spinlock_t *__pmd_trans_huge_lock(pmd_t *pmd, struct vm_area_struct *vma);
> @@ -450,7 +450,8 @@ static inline int hugepage_madvise(struct vm_area_struct *vma,
>
> static inline int madvise_collapse(struct vm_area_struct *vma,
> struct vm_area_struct **prev,
> - unsigned long start, unsigned long end)
> + unsigned long start, unsigned long end,
> + int behavior)
> {
> return -EINVAL;
> }
> diff --git a/include/uapi/asm-generic/mman-common.h b/include/uapi/asm-generic/mman-common.h
> index 6ce1f1ceb432..92c67bc755da 100644
> --- a/include/uapi/asm-generic/mman-common.h
> +++ b/include/uapi/asm-generic/mman-common.h
> @@ -78,6 +78,7 @@
> #define MADV_DONTNEED_LOCKED 24 /* like DONTNEED, but drop locked pages too */
>
> #define MADV_COLLAPSE 25 /* Synchronous hugepage collapse */
> +#define MADV_F_COLLAPSE_LIGHT 26 /* Similar to COLLAPSE, but avoids direct reclaim and/or compaction */
>
> /* compatibility flags */
> #define MAP_FILE 0
> diff --git a/mm/khugepaged.c b/mm/khugepaged.c
> index 2b219acb528e..2840051c0ae2 100644
> --- a/mm/khugepaged.c
> +++ b/mm/khugepaged.c
> @@ -97,6 +97,8 @@ static struct kmem_cache *mm_slot_cache __ro_after_init;
> struct collapse_control {
> bool is_khugepaged;
>
> + int behavior;
> +
> /* Num pages scanned per node */
> u32 node_load[MAX_NUMNODES];
>
> @@ -1058,10 +1060,16 @@ static int __collapse_huge_page_swapin(struct mm_struct *mm,
> static int alloc_charge_hpage(struct page **hpage, struct mm_struct *mm,
> struct collapse_control *cc)
> {
> - gfp_t gfp = (cc->is_khugepaged ? alloc_hugepage_khugepaged_gfpmask() :
> - GFP_TRANSHUGE);
> int node = hpage_collapse_find_target_node(cc);
> struct folio *folio;
> + gfp_t gfp;
> +
> + if (cc->is_khugepaged)
> + gfp = alloc_hugepage_khugepaged_gfpmask();
> + else
> + gfp = (cc->behavior == MADV_F_COLLAPSE_LIGHT ?
> + GFP_TRANSHUGE_LIGHT :
> + GFP_TRANSHUGE);
>
> if (!hpage_collapse_alloc_folio(&folio, gfp, node, &cc->alloc_nmask)) {
> *hpage = NULL;
> @@ -2697,7 +2705,7 @@ static int madvise_collapse_errno(enum scan_result r)
> }
>
> int madvise_collapse(struct vm_area_struct *vma, struct vm_area_struct **prev,
> - unsigned long start, unsigned long end)
> + unsigned long start, unsigned long end, int behavior)
> {
> struct collapse_control *cc;
> struct mm_struct *mm = vma->vm_mm;
> @@ -2718,6 +2726,7 @@ int madvise_collapse(struct vm_area_struct *vma, struct vm_area_struct **prev,
> if (!cc)
> return -ENOMEM;
> cc->is_khugepaged = false;
> + cc->behavior = behavior;
>
> mmgrab(mm);
> lru_add_drain_all();
> diff --git a/mm/madvise.c b/mm/madvise.c
> index 912155a94ed5..9c40226505aa 100644
> --- a/mm/madvise.c
> +++ b/mm/madvise.c
> @@ -60,6 +60,7 @@ static int madvise_need_mmap_write(int behavior)
> case MADV_POPULATE_READ:
> case MADV_POPULATE_WRITE:
> case MADV_COLLAPSE:
> + case MADV_F_COLLAPSE_LIGHT:
> return 0;
> default:
> /* be safe, default to 1. list exceptions explicitly */
> @@ -1082,8 +1083,9 @@ static int madvise_vma_behavior(struct vm_area_struct *vma,
> if (error)
> goto out;
> break;
> + case MADV_F_COLLAPSE_LIGHT:
> case MADV_COLLAPSE:
> - return madvise_collapse(vma, prev, start, end);
> + return madvise_collapse(vma, prev, start, end, behavior);
> }
>
> anon_name = anon_vma_name(vma);
> @@ -1178,6 +1180,7 @@ madvise_behavior_valid(int behavior)
> case MADV_HUGEPAGE:
> case MADV_NOHUGEPAGE:
> case MADV_COLLAPSE:
> + case MADV_F_COLLAPSE_LIGHT:
> #endif
> case MADV_DONTDUMP:
> case MADV_DODUMP:
> @@ -1194,6 +1197,17 @@ madvise_behavior_valid(int behavior)
> }
> }
>
> +
> +static bool process_madvise_behavior_only(int behavior)
> +{
> + switch (behavior) {
> + case MADV_F_COLLAPSE_LIGHT:
> + return true;
> + default:
> + return false;
> + }
> +}
> +
> static bool process_madvise_behavior_valid(int behavior)
> {
> switch (behavior) {
> @@ -1201,6 +1215,7 @@ static bool process_madvise_behavior_valid(int behavior)
> case MADV_PAGEOUT:
> case MADV_WILLNEED:
> case MADV_COLLAPSE:
> + case MADV_F_COLLAPSE_LIGHT:
> return true;
> default:
> return false;
> @@ -1368,6 +1383,8 @@ int madvise_set_anon_name(struct mm_struct *mm, unsigned long start,
> * transparent huge pages so the existing pages will not be
> * coalesced into THP and new pages will not be allocated as THP.
> * MADV_COLLAPSE - synchronously coalesce pages into new THP.
> + * MADV_F_COLLAPSE_LIGHT - only for process_madvise, avoids direct reclaim and/or
> + * compaction.
> * MADV_DONTDUMP - the application wants to prevent pages in the given range
> * from being included in its core dump.
> * MADV_DODUMP - cancel MADV_DONTDUMP: no longer exclude from core dump.
> @@ -1394,7 +1411,8 @@ int madvise_set_anon_name(struct mm_struct *mm, unsigned long start,
> * -EBADF - map exists, but area maps something that isn't a file.
> * -EAGAIN - a kernel resource was temporarily unavailable.
> */
> -int do_madvise(struct mm_struct *mm, unsigned long start, size_t len_in, int behavior)
> +int _do_madvise(struct mm_struct *mm, unsigned long start, size_t len_in,
> + int behavior, bool is_process_madvise)
> {
> unsigned long end;
> int error;
> @@ -1405,6 +1423,9 @@ int do_madvise(struct mm_struct *mm, unsigned long start, size_t len_in, int beh
> if (!madvise_behavior_valid(behavior))
> return -EINVAL;
>
> + if (!is_process_madvise && process_madvise_behavior_only(behavior))
> + return -EINVAL;
> +
> if (!PAGE_ALIGNED(start))
> return -EINVAL;
> len = PAGE_ALIGN(len_in);
> @@ -1448,9 +1469,14 @@ int do_madvise(struct mm_struct *mm, unsigned long start, size_t len_in, int beh
> return error;
> }
>
> +int do_madvise(struct mm_struct *mm, unsigned long start, size_t len_in, int behavior)
> +{
> + return _do_madvise(mm, start, len_in, behavior, false);
> +}
> +
> SYSCALL_DEFINE3(madvise, unsigned long, start, size_t, len_in, int, behavior)
> {
> - return do_madvise(current->mm, start, len_in, behavior);
> + return _do_madvise(current->mm, start, len_in, behavior, false);
> }
>
> SYSCALL_DEFINE5(process_madvise, int, pidfd, const struct iovec __user *, vec,
> @@ -1504,8 +1530,8 @@ SYSCALL_DEFINE5(process_madvise, int, pidfd, const struct iovec __user *, vec,
> total_len = iov_iter_count(&iter);
>
> while (iov_iter_count(&iter)) {
> - ret = do_madvise(mm, (unsigned long)iter_iov_addr(&iter),
> - iter_iov_len(&iter), behavior);
> + ret = _do_madvise(mm, (unsigned long)iter_iov_addr(&iter),
> + iter_iov_len(&iter), behavior, true);
> if (ret < 0)
> break;
> iov_iter_advance(&iter, iter_iov_len(&iter));
> diff --git a/tools/include/uapi/asm-generic/mman-common.h b/tools/include/uapi/asm-generic/mman-common.h
> index 6ce1f1ceb432..92c67bc755da 100644
> --- a/tools/include/uapi/asm-generic/mman-common.h
> +++ b/tools/include/uapi/asm-generic/mman-common.h
> @@ -78,6 +78,7 @@
> #define MADV_DONTNEED_LOCKED 24 /* like DONTNEED, but drop locked pages too */
>
> #define MADV_COLLAPSE 25 /* Synchronous hugepage collapse */
> +#define MADV_F_COLLAPSE_LIGHT 26 /* Similar to COLLAPSE, but avoids direct reclaim and/or compaction */
>
> /* compatibility flags */
> #define MAP_FILE 0
> --
> 2.33.1
>
^ permalink raw reply
* 回复: [PATCH v4] selftests/landlock:Fix two build issues
From: Huyadi @ 2024-01-21 11:02 UTC (permalink / raw)
To: 'Mickaël Salaün'
Cc: jmorris@namei.org, serge@hallyn.com, shuah@kernel.org,
mathieu.desnoyers@efficios.com, amir73il@gmail.com,
brauner@kernel.org, avagin@google.com, linux-api@vger.kernel.org,
linux-kernel@vger.kernel.org,
linux-security-module@vger.kernel.org,
linux-kselftest@vger.kernel.org, 514118380@qq.com,
konstantin.meskhidze@huawei.com
In-Reply-To: <20240119.Ugaehae2ze5b@digikod.net>
> On Mon, Jan 15, 2024 at 06:24:09PM +0800, Hu Yadi wrote:
>> From: "Hu.Yadi" <hu.yadi@h3c.com>
>>
>> Two issues comes up while building selftest/landlock on my side (gcc
>> 7.3/glibc-2.28/kernel-4.19)
>>
>> the first one is as to gettid
>>
>> net_test.c: In function ‘set_service’:
>> net_test.c:91:45: warning: implicit declaration of function ‘gettid’; [-Wimplicit-function-declaration]
>> "_selftests-landlock-net-tid%d-index%d", gettid(),
>> ^~~~~~
>> getgid
>> net_test.c:(.text+0x4e0): undefined reference to `gettid'
>>
>> the second is compiler error
>> gcc -Wall -O2 -isystem fs_test.c -lcap -o selftests/landlock/fs_test
>> fs_test.c:4575:9: error: initializer element is not constant
>> .mnt = mnt_tmp,
>> ^~~~~~~
>>
>> Fixes: 04f9070e99a4 ("selftests/landlock: Add tests for pseudo
>> filesystems")
>> Fixes: a549d055a22e ("selftests/landlock: Add network tests")
>
>Could you please create two patches as requested for v3, one per fix?
>This is useful because it enables to backport these fixes when appropriate.
OK, I'll resend it by two patches with your warmly instruction.
Thanks again.
>>
>> this patch is to fix them
>>
>> Signed-off-by: Hu.Yadi <hu.yadi@h3c.com>
>> Suggested-by: Jiao <jiaoxupo@h3c.com>
>> Reviewed-by: Berlin <berlin@h3c.com>
>> ---
>> Changes v4 -> v3:
>> fix gettid error from kernel test robot
>>
>> https://lore.kernel.org/oe-kbuild-all/202401151147.T1s11iHJ-lkp@intel.
>> com/
>> Changes v3 -> v2:
>> - add helper of gettid instead of __NR_gettid
>> - add gcc/glibc version info in comments Changes v1 -> v2:
>> - fix whitespace error
>> - replace SYS_gettid with _NR_gettid
>>
>> tools/testing/selftests/landlock/fs_test.c | 5 ++++-
>> tools/testing/selftests/landlock/net_test.c | 7 ++++++-
>> 2 files changed, 10 insertions(+), 2 deletions(-)
>>
>> diff --git a/tools/testing/selftests/landlock/fs_test.c
>> b/tools/testing/selftests/landlock/fs_test.c
>> index 18e1f86a6234..a992cf7c0ad1 100644
>> --- a/tools/testing/selftests/landlock/fs_test.c
>> +++ b/tools/testing/selftests/landlock/fs_test.c
>> @@ -4572,7 +4572,10 @@ FIXTURE_VARIANT(layout3_fs)
>> /* clang-format off */
>> FIXTURE_VARIANT_ADD(layout3_fs, tmpfs) {
>> /* clang-format on */
>> - .mnt = mnt_tmp,
>> + .mnt = {
>> + .type = "tmpfs",
>> + .data = "size=4m,mode=700",
>> + },
>
>I requested some changes here.
>
>> .file_path = file1_s1d1,
>> };
>>
>> diff --git a/tools/testing/selftests/landlock/net_test.c
>> b/tools/testing/selftests/landlock/net_test.c
>> index 929e21c4db05..d50f2920ed82 100644
>> --- a/tools/testing/selftests/landlock/net_test.c
>> +++ b/tools/testing/selftests/landlock/net_test.c
> @@ -21,6 +21,11 @@
>
>We should include sys/syscall.h
>
>>
>> #include "common.h"
>>
>> +static pid_t landlock_gettid(void)
>
>Please rename to sys_gettid().
>
>> +{
>> + return syscall(__NR_gettid);
>> +}
>> +
>> const short sock_port_start = (1 << 10);
>>
>> static const char loopback_ipv4[] = "127.0.0.1"; @@ -88,7 +93,7 @@
>> static int set_service(struct service_fixture *const srv,
>> case AF_UNIX:
>> srv->unix_addr.sun_family = prot.domain;
>> sprintf(srv->unix_addr.sun_path,
>> - "_selftests-landlock-net-tid%d-index%d", gettid(),
>> + "_selftests-landlock-net-tid%d-index%d", landlock_gettid(),
>> index);
>> srv->unix_addr_len = SUN_LEN(&srv->unix_addr);
>> srv->unix_addr.sun_path[0] = '\0';
>> --
>> 2.23.0
>>
^ permalink raw reply
* Re: [PATCH v2 1/3] mm/mempolicy: implement the sysfs-based weighted_interleave interface
From: Huang, Ying @ 2024-01-22 8:03 UTC (permalink / raw)
To: Gregory Price
Cc: linux-mm, linux-kernel, linux-doc, linux-fsdevel, linux-api,
corbet, akpm, gregory.price, honggyu.kim, rakie.kim, hyeongtak.ji,
mhocko, vtavarespetr, jgroves, ravis.opensrc, sthanneeru,
emirakhur, Hasan.Maruf, seungjun.ha, hannes, dan.j.williams
In-Reply-To: <20240119175730.15484-2-gregory.price@memverge.com>
Gregory Price <gourry.memverge@gmail.com> writes:
> From: Rakie Kim <rakie.kim@sk.com>
>
> This patch provides a way to set interleave weight information under
> sysfs at /sys/kernel/mm/mempolicy/weighted_interleave/nodeN
>
> The sysfs structure is designed as follows.
>
> $ tree /sys/kernel/mm/mempolicy/
> /sys/kernel/mm/mempolicy/ [1]
> └── weighted_interleave [2]
> ├── node0 [3]
> └── node1
>
> Each file above can be explained as follows.
>
> [1] mm/mempolicy: configuration interface for mempolicy subsystem
>
> [2] weighted_interleave/: config interface for weighted interleave policy
>
> [3] weighted_interleave/nodeN: weight for nodeN
>
> If a node value is set to `0`, the system-default value will be used.
> As of this patch, the system-default for all nodes is always 1.
>
> Suggested-by: Huang Ying <ying.huang@intel.com>
> Signed-off-by: Rakie Kim <rakie.kim@sk.com>
> Signed-off-by: Honggyu Kim <honggyu.kim@sk.com>
> Co-developed-by: Gregory Price <gregory.price@memverge.com>
> Signed-off-by: Gregory Price <gregory.price@memverge.com>
> Co-developed-by: Hyeongtak Ji <hyeongtak.ji@sk.com>
> Signed-off-by: Hyeongtak Ji <hyeongtak.ji@sk.com>
> ---
> .../ABI/testing/sysfs-kernel-mm-mempolicy | 4 +
> ...fs-kernel-mm-mempolicy-weighted-interleave | 26 ++
> mm/mempolicy.c | 231 ++++++++++++++++++
> 3 files changed, 261 insertions(+)
> create mode 100644 Documentation/ABI/testing/sysfs-kernel-mm-mempolicy
> create mode 100644 Documentation/ABI/testing/sysfs-kernel-mm-mempolicy-weighted-interleave
>
> diff --git a/Documentation/ABI/testing/sysfs-kernel-mm-mempolicy b/Documentation/ABI/testing/sysfs-kernel-mm-mempolicy
> new file mode 100644
> index 000000000000..2dcf24f4384a
> --- /dev/null
> +++ b/Documentation/ABI/testing/sysfs-kernel-mm-mempolicy
> @@ -0,0 +1,4 @@
> +What: /sys/kernel/mm/mempolicy/
> +Date: December 2023
> +Contact: Linux memory management mailing list <linux-mm@kvack.org>
> +Description: Interface for Mempolicy
> diff --git a/Documentation/ABI/testing/sysfs-kernel-mm-mempolicy-weighted-interleave b/Documentation/ABI/testing/sysfs-kernel-mm-mempolicy-weighted-interleave
> new file mode 100644
> index 000000000000..e6a38139bf0f
> --- /dev/null
> +++ b/Documentation/ABI/testing/sysfs-kernel-mm-mempolicy-weighted-interleave
> @@ -0,0 +1,26 @@
> +What: /sys/kernel/mm/mempolicy/weighted_interleave/
> +Date: December 2023
May be not a big deal. The date should be "January 2024"?
> +Contact: Linux memory management mailing list <linux-mm@kvack.org>
> +Description: Configuration Interface for the Weighted Interleave policy
> +
> +What: /sys/kernel/mm/mempolicy/weighted_interleave/nodeN
> +Date: December 2023
> +Contact: Linux memory management mailing list <linux-mm@kvack.org>
> +Description: Weight configuration interface for nodeN
> +
> + The interleave weight for a memory node (N). These weights are
> + utilized by processes which have set their mempolicy to
s/processes/tasks or memory areas/
> + MPOL_WEIGHTED_INTERLEAVE and have opted into global weights by
> + omitting a task-local weight array.
Now, we haven't introduced task-local weight array. So, leave this
until we introduce that?
> +
> + These weights only affect new allocations, and changes at runtime
> + will not cause migrations on already allocated pages.
> +
> + The minimum weight for a node is always 1.
> +
> + Minimum weight: 1
> + Maximum weight: 255
> +
> + Writing an empty string or `0` will reset the weight to the
> + system default. The system default may be set by the kernel
> + or drivers at boot or during hotplug events.
> diff --git a/mm/mempolicy.c b/mm/mempolicy.c
> index 10a590ee1c89..ae925216798f 100644
> --- a/mm/mempolicy.c
> +++ b/mm/mempolicy.c
> @@ -131,6 +131,16 @@ static struct mempolicy default_policy = {
>
> static struct mempolicy preferred_node_policy[MAX_NUMNODES];
>
> +/*
> + * iw_table is the sysfs-set interleave weight table, a value of 0 denotes
> + * system-default value should be used. Until system-defaults are implemented,
> + * the system-default is always 1.
> + *
> + * iw_table is RCU protected
> + */
> +static u8 __rcu *iw_table;
> +static DEFINE_MUTEX(iw_table_lock);
> +
> /**
> * numa_nearest_node - Find nearest node by state
> * @node: Node id to start the search
> @@ -3067,3 +3077,224 @@ void mpol_to_str(char *buffer, int maxlen, struct mempolicy *pol)
> p += scnprintf(p, buffer + maxlen - p, ":%*pbl",
> nodemask_pr_args(&nodes));
> }
> +
> +#ifdef CONFIG_SYSFS
> +struct iw_node_attr {
> + struct kobj_attribute kobj_attr;
> + int nid;
> +};
> +
> +static ssize_t node_show(struct kobject *kobj, struct kobj_attribute *attr,
> + char *buf)
> +{
> + struct iw_node_attr *node_attr;
> + u8 weight;
> + u8 __rcu *table;
> +
> + node_attr = container_of(attr, struct iw_node_attr, kobj_attr);
> +
> + rcu_read_lock();
> + table = rcu_dereference(iw_table);
> + weight = table ? table[node_attr->nid] : 1;
> + rcu_read_unlock();
> +
> + return sysfs_emit(buf, "%d\n", weight);
> +}
> +
> +static ssize_t node_store(struct kobject *kobj, struct kobj_attribute *attr,
> + const char *buf, size_t count)
> +{
> + struct iw_node_attr *node_attr;
> + u8 __rcu *new;
> + u8 __rcu *old;
> + u8 weight = 0;
> +
> + node_attr = container_of(attr, struct iw_node_attr, kobj_attr);
> + if (count == 0 || sysfs_streq(buf, ""))
> + weight = 0;
> + else if (kstrtou8(buf, 0, &weight))
> + return -EINVAL;
> +
> + /*
> + * The default weight is 1 (for now), when the kernel-internal
> + * default weight array is implemented, this should be updated to
> + * collect the system-default weight of the node if the user passes 0.
> + */
> + if (!weight)
> + weight = 1;
From functionality point of view, it's OK to set "weight = 1" here now.
But when we add system default weight table in the future, we need to
use "weight = 0". Otherwise, we cannot distinguish whether the default
value have been customized via sysfs. So, I suggest to use that rule.
> +
> + /* We only need to allocate up to the number of possible nodes */
This comment appears not necessary.
> + new = kmalloc(nr_node_ids, GFP_KERNEL);
> + if (!new)
> + return -ENOMEM;
> +
> + mutex_lock(&iw_table_lock);
> + old = rcu_dereference_protected(iw_table,
> + lockdep_is_held(&iw_table_lock));
> + if (old)
> + memcpy(new, old, nr_node_ids);
> + else
> + memset(new, 1, nr_node_ids);
With similar reason as above ("From functionality..."), I suggest to set
"0" here.
> + new[node_attr->nid] = weight;
> + rcu_assign_pointer(iw_table, new);
> + mutex_unlock(&iw_table_lock);
> + synchronize_rcu();
> + kfree(old);
> + return count;
> +}
> +
> +static struct iw_node_attr *node_attrs[MAX_NUMNODES];
node_attrs[] can be allocated dynamically too. Just a suggestion.
> +
> +static void sysfs_wi_node_release(struct iw_node_attr *node_attr,
> + struct kobject *parent)
> +{
> + if (!node_attr)
> + return;
> + sysfs_remove_file(parent, &node_attr->kobj_attr.attr);
> + kfree(node_attr->kobj_attr.attr.name);
> + kfree(node_attr);
> +}
> +
> +static void sysfs_wi_release(struct kobject *wi_kobj)
> +{
> + int i;
> +
> + for (i = 0; i < MAX_NUMNODES; i++)
Nitpick, nr_node_ids should be OK here.
> + sysfs_wi_node_release(node_attrs[i], wi_kobj);
> + kobject_put(wi_kobj);
> +}
> +
> +static const struct kobj_type wi_ktype = {
> + .sysfs_ops = &kobj_sysfs_ops,
> + .release = sysfs_wi_release,
> +};
> +
> +static int add_weight_node(int nid, struct kobject *wi_kobj)
> +{
> + struct iw_node_attr *node_attr;
> + char *name;
> +
> + node_attr = kzalloc(sizeof(*node_attr), GFP_KERNEL);
> + if (!node_attr)
> + return -ENOMEM;
> +
> + name = kasprintf(GFP_KERNEL, "node%d", nid);
> + if (!name) {
> + kfree(node_attr);
> + return -ENOMEM;
> + }
> +
> + sysfs_attr_init(&node_attr->kobj_attr.attr);
> + node_attr->kobj_attr.attr.name = name;
> + node_attr->kobj_attr.attr.mode = 0644;
> + node_attr->kobj_attr.show = node_show;
> + node_attr->kobj_attr.store = node_store;
> + node_attr->nid = nid;
> +
> + if (sysfs_create_file(wi_kobj, &node_attr->kobj_attr.attr)) {
> + kfree(node_attr->kobj_attr.attr.name);
> + kfree(node_attr);
> + pr_err("failed to add attribute to weighted_interleave\n");
> + return -ENOMEM;
> + }
> +
> + node_attrs[nid] = node_attr;
> + return 0;
> +}
> +
> +static int add_weighted_interleave_group(struct kobject *root_kobj)
> +{
> + struct kobject *wi_kobj;
> + int nid, err;
> +
> + wi_kobj = kzalloc(sizeof(struct kobject), GFP_KERNEL);
> + if (!wi_kobj)
> + return -ENOMEM;
> +
> + err = kobject_init_and_add(wi_kobj, &wi_ktype, root_kobj,
> + "weighted_interleave");
> + if (err) {
> + kfree(wi_kobj);
> + return err;
> + }
> +
> + memset(node_attrs, 0, sizeof(node_attrs));
> + for_each_node_state(nid, N_POSSIBLE) {
> + err = add_weight_node(nid, wi_kobj);
> + if (err) {
> + pr_err("failed to add sysfs [node%d]\n", nid);
> + break;
> + }
> + }
> + if (err)
> + kobject_put(wi_kobj);
> + return 0;
> +}
> +
> +static void mempolicy_kobj_release(struct kobject *kobj)
> +{
> + u8 __rcu *old;
> +
> + mutex_lock(&iw_table_lock);
> + old = rcu_dereference_protected(iw_table,
> + lockdep_is_held(&iw_table_lock));
> + rcu_assign_pointer(iw_table, NULL);
> + mutex_unlock(&iw_table_lock);
> + synchronize_rcu();
> + /* Never free the default table, it's always in use */
Obsolete comment?
> + kfree(old);
It appears unnecessary to free iw_table in error path. But this isn't a
big deal because error path will almost never be executed in practice.
> + kfree(kobj);
> +}
> +
> +static const struct kobj_type mempolicy_ktype = {
> + .release = mempolicy_kobj_release
> +};
> +
> +static struct kobject *mempolicy_kobj;
> +static int __init mempolicy_sysfs_init(void)
> +{
> + int err;
> + struct kobject *mempolicy_kobj;
This overrides the global "mempolicy_kobj" defined before function. But
I don't think we need the global definition.
> +
> + /* A NULL iw_table is interpreted by interleave logic as "all 1s" */
As I suggested above, it will be "all 0s", that is, use default weight.
> + iw_table = NULL;
The default value is NULL already, it appears unnecessary to do this.
> + mempolicy_kobj = kzalloc(sizeof(*mempolicy_kobj), GFP_KERNEL);
> + if (!mempolicy_kobj) {
> + pr_err("failed to add mempolicy kobject to the system\n");
> + return -ENOMEM;
> + }
> + err = kobject_init_and_add(mempolicy_kobj, &mempolicy_ktype, mm_kobj,
> + "mempolicy");
> + if (err) {
> + kfree(mempolicy_kobj);
> + return err;
> + }
> +
> + err = add_weighted_interleave_group(mempolicy_kobj);
> +
> + if (err) {
> + kobject_put(mempolicy_kobj);
> + return err;
> + }
> +
> + return err;
> +}
> +
> +static void __exit mempolicy_exit(void)
> +{
> + if (mempolicy_kobj)
> + kobject_put(mempolicy_kobj);
> +}
> +
> +#else
> +static int __init mempolicy_sysfs_init(void)
> +{
> + /* A NULL iw_table is interpreted by interleave logic as "all 1s" */
> + iw_table = NULL;
> + return 0;
> +}
> +
> +static void __exit mempolicy_exit(void) { }
> +#endif /* CONFIG_SYSFS */
> +late_initcall(mempolicy_sysfs_init);
> +module_exit(mempolicy_exit);
mempolicy.c will not be compiled as module, so we don't need
module_exit().
--
Best Regards,
Huang, Ying
^ permalink raw reply
* Re: [PATCH v2 1/1] mm/madvise: add MADV_F_COLLAPSE_LIGHT to process_madvise()
From: Lance Yang @ 2024-01-22 14:34 UTC (permalink / raw)
To: Michal Hocko, Zach O'Keefe
Cc: akpm, david, songmuchun, shy828301, peterx, mknyszek, minchan,
linux-mm, linux-kernel, linux-api
In-Reply-To: <CAK1f24n+pTw=7atKGAbgBh8qKOTBPq-BEV7ZkLcgN3P0n3h3ug@mail.gmail.com>
Hey Zach,
What do you think about the semantic?
Thanks,
Lance
On Mon, Jan 22, 2024 at 10:14 PM Lance Yang <ioworker0@gmail.com> wrote:
>
> On Mon, Jan 22, 2024 at 9:50 PM Michal Hocko <mhocko@suse.com> wrote:
> >
> > On Sat 20-01-24 10:09:32, Lance Yang wrote:
> > [...]
> > > Hey Michal,
> > >
> > > Thanks for your suggestion!
> > >
> > > It seems that the implementation should try but not too hard aligns well
> > > with my desired behavior.
> >
> > The problem I have with this semantic is that it is really hard to
> > define and then stick with. Our implementation might change over time
> > and what somebody considers good ATM might turn int "trying harder than
> > I wanted" later on.
> >
> > > Non-blocking in general is also a great idea.
> > > Perhaps in the future, we can add a MADV_F_COLLAPSE_NOBLOCK
> > > flag for scenarios where latency is extremely critical.
> >
> > Non blocking semantic is much easier to define and maintain. The actual
> > allocation/compaction implementation might change as well over time but
> > the userspace at least knows that the request will not block waiting for
> > any required resources.
>
> I appreciate your insights!
>
> It makes sense that a non-blocking semantic is easier to define and maintain,
> providing userspace with the certainty that requests won’t be blocked.
>
> Thanks,
> Lance
>
> >
> > --
> > Michal Hocko
> > SUSE Labs
^ permalink raw reply
* Re: [RFC 1/1] eventpoll: support busy poll per epoll instance
From: Christian Brauner @ 2024-01-22 15:25 UTC (permalink / raw)
To: Joe Damato
Cc: netdev, linux-kernel, chuck.lever, jlayton, linux-api, edumazet,
davem, alexander.duyck, sridhar.samudrala, kuba
In-Reply-To: <20240120004247.42036-2-jdamato@fastly.com>
On Sat, Jan 20, 2024 at 12:42:47AM +0000, Joe Damato wrote:
> Add F_EPOLL_{S,G}ET_BUSY_POLL_USECS to allow setting a busy poll timeout
> per epoll instance so that individual applications can enable (or
> disable) epoll based busy poll as needed.
>
> Prior to this change, epoll-based busy poll could only be enabled
> system-wide, which limits the usefulness of busy poll.
>
> Signed-off-by: Joe Damato <jdamato@fastly.com>
> ---
This should be an ioctl on the epoll fd, not a fcntl(). fcntl()s
should aim to be generic which this isn't. We've recently rejected a
memfd addition as a fcntl() as well for the same reason.
^ permalink raw reply
* Re: [PATCH v2 1/3] mm/mempolicy: implement the sysfs-based weighted_interleave interface
From: Gregory Price @ 2024-01-22 16:58 UTC (permalink / raw)
To: Huang, Ying
Cc: Gregory Price, linux-mm, linux-kernel, linux-doc, linux-fsdevel,
linux-api, corbet, akpm, honggyu.kim, rakie.kim, hyeongtak.ji,
mhocko, vtavarespetr, jgroves, ravis.opensrc, sthanneeru,
emirakhur, Hasan.Maruf, seungjun.ha, hannes, dan.j.williams
In-Reply-To: <875xzlx09i.fsf@yhuang6-desk2.ccr.corp.intel.com>
On Mon, Jan 22, 2024 at 04:03:53PM +0800, Huang, Ying wrote:
> Gregory Price <gourry.memverge@gmail.com> writes:
>
> > + /*
> > + * The default weight is 1 (for now), when the kernel-internal
> > + * default weight array is implemented, this should be updated to
> > + * collect the system-default weight of the node if the user passes 0.
> > + */
> > + if (!weight)
> > + weight = 1;
>
> From functionality point of view, it's OK to set "weight = 1" here now.
> But when we add system default weight table in the future, we need to
> use "weight = 0". Otherwise, we cannot distinguish whether the default
> value have been customized via sysfs. So, I suggest to use that rule.
>
[... snip ...]
> > + else
> > + memset(new, 1, nr_node_ids);
>
> With similar reason as above ("From functionality..."), I suggest to set
> "0" here.
>
blah - the comment is misleading at best. The future patch should pass
0 through to the sysfs table and the allocators updated to collect the
system-default weight of the node.
re: doing it this way right now -
I chose to do it this way for now because it ultimately simplifies the
logic in the allocators - all of which will need to be updated with the
future patch set regardless of our implementation choice now.
e.g.
rcu_read_lock();
table = rcu_dereference(iw_table);
if (!policy->wil.cur_weight)
policy->wil.cur_weight = table ? table[next] : 1;
^^^ only need single conditional now
rcu_read_unlock();
This logic will need to be updated to use default table values, so I
chose the simpler implementation and left the change to be explicit
at the time the default table is implemented.
If you prefer it the other way now, I can change it, but this seemed
cleaner and simpler for the time being.
> > + new[node_attr->nid] = weight;
> > + rcu_assign_pointer(iw_table, new);
> > + mutex_unlock(&iw_table_lock);
> > + synchronize_rcu();
> > + kfree(old);
> > + return count;
> > +}
> > +
> > +static struct iw_node_attr *node_attrs[MAX_NUMNODES];
>
> node_attrs[] can be allocated dynamically too. Just a suggestion.
>
ack to this and other references to nr_node_ids, will change.
> > + kfree(old);
>
> It appears unnecessary to free iw_table in error path. But this isn't a
> big deal because error path will almost never be executed in practice.
>
checkpatch.pl yells at you if you do null checks before kfree :]
> > + int err;
> > + struct kobject *mempolicy_kobj;
>
> This overrides the global "mempolicy_kobj" defined before function. But
> I don't think we need the global definition.
>
Assuming the exit path isn't needed then yeah the global isn't needed.
> > +static int __init mempolicy_sysfs_init(void)
> > +{
> > + /* A NULL iw_table is interpreted by interleave logic as "all 1s" */
> > + iw_table = NULL;
> > + return 0;
> > +}
> > +
> > +static void __exit mempolicy_exit(void) { }
> > +#endif /* CONFIG_SYSFS */
> > +late_initcall(mempolicy_sysfs_init);
> > +module_exit(mempolicy_exit);
>
> mempolicy.c will not be compiled as module, so we don't need
> module_exit().
>
ack
^ permalink raw reply
* Re: [RFC 1/1] eventpoll: support busy poll per epoll instance
From: Joe Damato @ 2024-01-22 21:16 UTC (permalink / raw)
To: Christian Brauner
Cc: netdev, linux-kernel, chuck.lever, jlayton, linux-api, edumazet,
davem, alexander.duyck, sridhar.samudrala, kuba
In-Reply-To: <20240122-erwidern-erleichtern-8a04080a4db3@brauner>
On Mon, Jan 22, 2024 at 04:25:01PM +0100, Christian Brauner wrote:
> On Sat, Jan 20, 2024 at 12:42:47AM +0000, Joe Damato wrote:
> > Add F_EPOLL_{S,G}ET_BUSY_POLL_USECS to allow setting a busy poll timeout
> > per epoll instance so that individual applications can enable (or
> > disable) epoll based busy poll as needed.
> >
> > Prior to this change, epoll-based busy poll could only be enabled
> > system-wide, which limits the usefulness of busy poll.
> >
> > Signed-off-by: Joe Damato <jdamato@fastly.com>
> > ---
>
> This should be an ioctl on the epoll fd, not a fcntl(). fcntl()s
> should aim to be generic which this isn't. We've recently rejected a
> memfd addition as a fcntl() as well for the same reason.
OK, thanks for the review. An ioctl makes more sense, I agree.
I'll rewrite it as you've suggested and send another RFC.
^ permalink raw reply
* Re: [PATCH v2 3/3] mm/mempolicy: introduce MPOL_WEIGHTED_INTERLEAVE for weighted interleaving
From: Huang, Ying @ 2024-01-23 3:02 UTC (permalink / raw)
To: Gregory Price
Cc: linux-mm, linux-kernel, linux-doc, linux-fsdevel, linux-api,
corbet, akpm, gregory.price, honggyu.kim, rakie.kim, hyeongtak.ji,
mhocko, vtavarespetr, jgroves, ravis.opensrc, sthanneeru,
emirakhur, Hasan.Maruf, seungjun.ha, hannes, dan.j.williams,
Srinivasulu Thanneeru
In-Reply-To: <20240119175730.15484-4-gregory.price@memverge.com>
Gregory Price <gourry.memverge@gmail.com> writes:
> When a system has multiple NUMA nodes and it becomes bandwidth hungry,
> using the current MPOL_INTERLEAVE could be an wise option.
>
> However, if those NUMA nodes consist of different types of memory such
> as socket-attached DRAM and CXL/PCIe attached DRAM, the round-robin
> based interleave policy does not optimally distribute data to make use
> of their different bandwidth characteristics.
>
> Instead, interleave is more effective when the allocation policy follows
> each NUMA nodes' bandwidth weight rather than a simple 1:1 distribution.
>
> This patch introduces a new memory policy, MPOL_WEIGHTED_INTERLEAVE,
> enabling weighted interleave between NUMA nodes. Weighted interleave
> allows for proportional distribution of memory across multiple numa
> nodes, preferably apportioned to match the bandwidth of each node.
>
> For example, if a system has 1 CPU node (0), and 2 memory nodes (0,1),
> with bandwidth of (100GB/s, 50GB/s) respectively, the appropriate
> weight distribution is (2:1).
>
> Weights for each node can be assigned via the new sysfs extension:
> /sys/kernel/mm/mempolicy/weighted_interleave/
>
> For now, the default value of all nodes will be `1`, which matches
> the behavior of standard 1:1 round-robin interleave. An extension
> will be added in the future to allow default values to be registered
> at kernel and device bringup time.
>
> The policy allocates a number of pages equal to the set weights. For
> example, if the weights are (2,1), then 2 pages will be allocated on
> node0 for every 1 page allocated on node1.
>
> The new flag MPOL_WEIGHTED_INTERLEAVE can be used in set_mempolicy(2)
> and mbind(2).
>
> There are 3 integration points:
>
> weighted_interleave_nodes:
> Counts the number of allocations as they occur, and applies the
> weight for the current node. When the weight reaches 0, switch
> to the next node.
>
> weighted_interleave_nid:
> Gets the total weight of the nodemask as well as each individual
> node weight, then calculates the node based on the given index.
>
> bulk_array_weighted_interleave:
> Gets the total weight of the nodemask as well as each individual
> node weight, then calculates the number of "interleave rounds" as
> well as any delta ("partial round"). Calculates the number of
> pages for each node and allocates them.
>
> If a node was scheduled for interleave via interleave_nodes, the
> current weight (pol->cur_weight) will be allocated first, before
> the remaining bulk calculation is done.
>
> One piece of complexity is the interaction between a recent refactor
> which split the logic to acquire the "ilx" (interleave index) of an
> allocation and the actually application of the interleave. The
> calculation of the `interleave index` is done by `get_vma_policy()`,
> while the actual selection of the node will be later appliex by the
> relevant weighted_interleave function.
>
> Suggested-by: Hasan Al Maruf <Hasan.Maruf@amd.com>
> Signed-off-by: Gregory Price <gregory.price@memverge.com>
> Co-developed-by: Rakie Kim <rakie.kim@sk.com>
> Signed-off-by: Rakie Kim <rakie.kim@sk.com>
> Co-developed-by: Honggyu Kim <honggyu.kim@sk.com>
> Signed-off-by: Honggyu Kim <honggyu.kim@sk.com>
> Co-developed-by: Hyeongtak Ji <hyeongtak.ji@sk.com>
> Signed-off-by: Hyeongtak Ji <hyeongtak.ji@sk.com>
> Co-developed-by: Srinivasulu Thanneeru <sthanneeru.opensrc@micron.com>
> Signed-off-by: Srinivasulu Thanneeru <sthanneeru.opensrc@micron.com>
> Co-developed-by: Ravi Jonnalagadda <ravis.opensrc@micron.com>
> Signed-off-by: Ravi Jonnalagadda <ravis.opensrc@micron.com>
> ---
> .../admin-guide/mm/numa_memory_policy.rst | 9 +
> include/linux/mempolicy.h | 5 +
> include/uapi/linux/mempolicy.h | 1 +
> mm/mempolicy.c | 234 +++++++++++++++++-
> 4 files changed, 246 insertions(+), 3 deletions(-)
>
> diff --git a/Documentation/admin-guide/mm/numa_memory_policy.rst b/Documentation/admin-guide/mm/numa_memory_policy.rst
> index eca38fa81e0f..a70f20ce1ffb 100644
> --- a/Documentation/admin-guide/mm/numa_memory_policy.rst
> +++ b/Documentation/admin-guide/mm/numa_memory_policy.rst
> @@ -250,6 +250,15 @@ MPOL_PREFERRED_MANY
> can fall back to all existing numa nodes. This is effectively
> MPOL_PREFERRED allowed for a mask rather than a single node.
>
> +MPOL_WEIGHTED_INTERLEAVE
> + This mode operates the same as MPOL_INTERLEAVE, except that
> + interleaving behavior is executed based on weights set in
> + /sys/kernel/mm/mempolicy/weighted_interleave/
> +
> + Weighted interleave allocates pages on nodes according to a
> + weight. For example if nodes [0,1] are weighted [5,2], 5 pages
> + will be allocated on node0 for every 2 pages allocated on node1.
> +
> NUMA memory policy supports the following optional mode flags:
>
> MPOL_F_STATIC_NODES
> diff --git a/include/linux/mempolicy.h b/include/linux/mempolicy.h
> index 931b118336f4..c1a083eb0dd5 100644
> --- a/include/linux/mempolicy.h
> +++ b/include/linux/mempolicy.h
> @@ -54,6 +54,11 @@ struct mempolicy {
> nodemask_t cpuset_mems_allowed; /* relative to these nodes */
> nodemask_t user_nodemask; /* nodemask passed by user */
> } w;
> +
> + /* Weighted interleave settings */
> + struct {
> + u8 cur_weight;
> + } wil;
> };
>
> /*
> diff --git a/include/uapi/linux/mempolicy.h b/include/uapi/linux/mempolicy.h
> index a8963f7ef4c2..1f9bb10d1a47 100644
> --- a/include/uapi/linux/mempolicy.h
> +++ b/include/uapi/linux/mempolicy.h
> @@ -23,6 +23,7 @@ enum {
> MPOL_INTERLEAVE,
> MPOL_LOCAL,
> MPOL_PREFERRED_MANY,
> + MPOL_WEIGHTED_INTERLEAVE,
> MPOL_MAX, /* always last member of enum */
> };
>
> diff --git a/mm/mempolicy.c b/mm/mempolicy.c
> index 427bddf115df..aa3b2389d3e0 100644
> --- a/mm/mempolicy.c
> +++ b/mm/mempolicy.c
> @@ -19,6 +19,13 @@
> * for anonymous memory. For process policy an process counter
> * is used.
> *
> + * weighted interleave
> + * Allocate memory interleaved over a set of nodes based on
> + * a set of weights (per-node), with normal fallback if it
> + * fails. Otherwise operates the same as interleave.
> + * Example: nodeset(0,1) & weights (2,1) - 2 pages allocated
> + * on node 0 for every 1 page allocated on node 1.
> + *
> * bind Only allocate memory on a specific set of nodes,
> * no fallback.
> * FIXME: memory is allocated starting with the first node
> @@ -313,6 +320,7 @@ static struct mempolicy *mpol_new(unsigned short mode, unsigned short flags,
> policy->mode = mode;
> policy->flags = flags;
> policy->home_node = NUMA_NO_NODE;
> + policy->wil.cur_weight = 0;
>
> return policy;
> }
> @@ -425,6 +433,10 @@ static const struct mempolicy_operations mpol_ops[MPOL_MAX] = {
> .create = mpol_new_nodemask,
> .rebind = mpol_rebind_preferred,
> },
> + [MPOL_WEIGHTED_INTERLEAVE] = {
> + .create = mpol_new_nodemask,
> + .rebind = mpol_rebind_nodemask,
> + },
> };
>
> static bool migrate_folio_add(struct folio *folio, struct list_head *foliolist,
> @@ -846,7 +858,8 @@ static long do_set_mempolicy(unsigned short mode, unsigned short flags,
>
> old = current->mempolicy;
> current->mempolicy = new;
> - if (new && new->mode == MPOL_INTERLEAVE)
> + if (new && (new->mode == MPOL_INTERLEAVE ||
> + new->mode == MPOL_WEIGHTED_INTERLEAVE))
> current->il_prev = MAX_NUMNODES-1;
> task_unlock(current);
> mpol_put(old);
> @@ -872,6 +885,7 @@ static void get_policy_nodemask(struct mempolicy *pol, nodemask_t *nodes)
> case MPOL_INTERLEAVE:
> case MPOL_PREFERRED:
> case MPOL_PREFERRED_MANY:
> + case MPOL_WEIGHTED_INTERLEAVE:
> *nodes = pol->nodes;
> break;
> case MPOL_LOCAL:
> @@ -956,6 +970,13 @@ static long do_get_mempolicy(int *policy, nodemask_t *nmask,
> } else if (pol == current->mempolicy &&
> pol->mode == MPOL_INTERLEAVE) {
> *policy = next_node_in(current->il_prev, pol->nodes);
> + } else if (pol == current->mempolicy &&
> + (pol->mode == MPOL_WEIGHTED_INTERLEAVE)) {
> + if (pol->wil.cur_weight)
> + *policy = current->il_prev;
> + else
> + *policy = next_node_in(current->il_prev,
> + pol->nodes);
> } else {
> err = -EINVAL;
> goto out;
> @@ -1785,7 +1806,8 @@ struct mempolicy *get_vma_policy(struct vm_area_struct *vma,
> pol = __get_vma_policy(vma, addr, ilx);
> if (!pol)
> pol = get_task_policy(current);
> - if (pol->mode == MPOL_INTERLEAVE) {
> + if (pol->mode == MPOL_INTERLEAVE ||
> + pol->mode == MPOL_WEIGHTED_INTERLEAVE) {
Should change the comments above get_vma_policy() definition too.
> *ilx += vma->vm_pgoff >> order;
> *ilx += (addr - vma->vm_start) >> (PAGE_SHIFT + order);
> }
> @@ -1835,6 +1857,28 @@ bool apply_policy_zone(struct mempolicy *policy, enum zone_type zone)
> return zone >= dynamic_policy_zone;
> }
>
> +static unsigned int weighted_interleave_nodes(struct mempolicy *policy)
> +{
> + unsigned int next;
> + struct task_struct *me = current;
> + u8 __rcu *table;
> +
> + next = next_node_in(me->il_prev, policy->nodes);
> + if (next == MAX_NUMNODES)
> + return next;
> +
> + rcu_read_lock();
> + table = rcu_dereference(iw_table);
> + if (!policy->wil.cur_weight)
> + policy->wil.cur_weight = table ? table[next] : 1;
> + rcu_read_unlock();
> +
> + policy->wil.cur_weight--;
> + if (!policy->wil.cur_weight)
> + me->il_prev = next;
> + return next;
> +}
> +
> /* Do dynamic interleaving for a process */
> static unsigned int interleave_nodes(struct mempolicy *policy)
> {
> @@ -1869,6 +1913,9 @@ unsigned int mempolicy_slab_node(void)
> case MPOL_INTERLEAVE:
> return interleave_nodes(policy);
>
> + case MPOL_WEIGHTED_INTERLEAVE:
> + return weighted_interleave_nodes(policy);
> +
> case MPOL_BIND:
> case MPOL_PREFERRED_MANY:
> {
> @@ -1907,6 +1954,39 @@ static unsigned int read_once_policy_nodemask(struct mempolicy *pol,
> return nodes_weight(*mask);
> }
>
> +static unsigned int weighted_interleave_nid(struct mempolicy *pol, pgoff_t ilx)
> +{
> + nodemask_t nodemask;
> + unsigned int target, nr_nodes;
> + u8 __rcu *table;
> + unsigned int weight_total = 0;
> + u8 weight;
> + int nid;
> +
> + nr_nodes = read_once_policy_nodemask(pol, &nodemask);
> + if (!nr_nodes)
> + return numa_node_id();
> +
> + rcu_read_lock();
> + table = rcu_dereference(iw_table);
> + /* calculate the total weight */
> + for_each_node_mask(nid, nodemask)
> + weight_total += table ? table[nid] : 1;
> +
> + /* Calculate the node offset based on totals */
> + target = ilx % weight_total;
> + nid = first_node(nodemask);
> + while (target) {
> + weight = table ? table[nid] : 1;
> + if (target < weight)
> + break;
> + target -= weight;
> + nid = next_node_in(nid, nodemask);
> + }
> + rcu_read_unlock();
> + return nid;
> +}
> +
> /*
> * Do static interleaving for interleave index @ilx. Returns the ilx'th
> * node in pol->nodes (starting from ilx=0), wrapping around if ilx
> @@ -1967,6 +2047,11 @@ static nodemask_t *policy_nodemask(gfp_t gfp, struct mempolicy *pol,
> *nid = (ilx == NO_INTERLEAVE_INDEX) ?
> interleave_nodes(pol) : interleave_nid(pol, ilx);
> break;
> + case MPOL_WEIGHTED_INTERLEAVE:
> + *nid = (ilx == NO_INTERLEAVE_INDEX) ?
> + weighted_interleave_nodes(pol) :
> + weighted_interleave_nid(pol, ilx);
> + break;
> }
>
> return nodemask;
> @@ -2028,6 +2113,7 @@ bool init_nodemask_of_mempolicy(nodemask_t *mask)
> case MPOL_PREFERRED_MANY:
> case MPOL_BIND:
> case MPOL_INTERLEAVE:
> + case MPOL_WEIGHTED_INTERLEAVE:
> *mask = mempolicy->nodes;
> break;
>
> @@ -2127,7 +2213,8 @@ struct page *alloc_pages_mpol(gfp_t gfp, unsigned int order,
> * If the policy is interleave or does not allow the current
> * node in its nodemask, we allocate the standard way.
> */
> - if (pol->mode != MPOL_INTERLEAVE &&
> + if ((pol->mode != MPOL_INTERLEAVE &&
> + pol->mode != MPOL_WEIGHTED_INTERLEAVE) &&
> (!nodemask || node_isset(nid, *nodemask))) {
> /*
> * First, try to allocate THP only on local node, but
> @@ -2263,6 +2350,135 @@ static unsigned long alloc_pages_bulk_array_interleave(gfp_t gfp,
> return total_allocated;
> }
>
> +static unsigned long alloc_pages_bulk_array_weighted_interleave(gfp_t gfp,
> + struct mempolicy *pol, unsigned long nr_pages,
> + struct page **page_array)
> +{
> + struct task_struct *me = current;
> + unsigned long total_allocated = 0;
> + unsigned long nr_allocated;
> + unsigned long rounds;
> + unsigned long node_pages, delta;
> + u8 weight, resume_weight;
> + u8 __rcu *table;
> + u8 *weights;
> + unsigned int weight_total = 0;
> + unsigned long rem_pages = nr_pages;
> + nodemask_t nodes;
> + int nnodes, node, weight_nodes, resume_node;
> + int prev_node = NUMA_NO_NODE;
It appears that we should initialize prev_node with me->il_prev?
Details are as below.
> + bool delta_depleted = false;
> + int i;
> +
> + if (!nr_pages)
> + return 0;
> +
> + nnodes = read_once_policy_nodemask(pol, &nodes);
> + if (!nnodes)
> + return 0;
> +
> + /* Continue allocating from most recent node and adjust the nr_pages */
> + if (pol->wil.cur_weight) {
> + node = next_node_in(me->il_prev, nodes);
> + node_pages = pol->wil.cur_weight;
> + if (node_pages > rem_pages)
> + node_pages = rem_pages;
> + nr_allocated = __alloc_pages_bulk(gfp, node, NULL, node_pages,
> + NULL, page_array);
> + page_array += nr_allocated;
> + total_allocated += nr_allocated;
> + /* if that's all the pages, no need to interleave */
> + if (rem_pages <= pol->wil.cur_weight) {
> + pol->wil.cur_weight -= rem_pages;
If "pol->wil.cur_weight == 0" here, we need to change me->il_prev?
> + return total_allocated;
> + }
> + /* Otherwise we adjust nr_pages down, and continue from there */
> + rem_pages -= pol->wil.cur_weight;
> + pol->wil.cur_weight = 0;
> + prev_node = node;
> + }
else {
prev_node = me->il_prev;
}
> +
> + /* fetch the weights for this operation and calculate total weight */
> + weights = kmalloc(nnodes, GFP_KERNEL);
> + if (!weights)
> + return total_allocated;
> +
> + rcu_read_lock();
> + table = rcu_dereference(iw_table);
> + weight_nodes = 0;
We can replace "weight_nodes" with "i" and use a "for" loop?
> + while (weight_nodes < nnodes) {
> + node = next_node_in(prev_node, nodes);
IIUC, "node" will not change in the loop, so all "weight" below will be
the same value. To keep it simple, I think we can just copy weights
from the global iw_table and consider the default value?
> + weight = table ? table[node] : 1;
> + weights[weight_nodes++] = weight;
> + weight_total += weight;
> + }
> + rcu_read_unlock();
> +
> + /*
> + * Now we can continue allocating from 0 instead of an offset
> + * We calculate the number of rounds and any partial rounds so
> + * that we minimize the number of calls to __alloc_pages_bulk
> + * This requires us to track which node we should resume from.
> + *
> + * if (rounds > 0) and (delta == 0), resume_node will always be
> + * the current me->il_prev
> + *
> + * if (delta > 0) and delta is depleted exactly on a node-weight
> + * boundary, resume node will be the node last allocated from when
> + * delta reached 0.
> + *
> + * if (delta > 0) and delta is not depleted on a node-weight boundary,
> + * resume node will be the node prior to the node last allocated from.
> + *
> + * (rounds == 0) and (delta == 0) is not possible (earlier exit)
> + */
> + rounds = rem_pages / weight_total;
> + delta = rem_pages % weight_total;
> + /* If no delta, we'll resume from current prev_node and first weight */
> + for (i = 0; i < nnodes; i++) {
> + node = next_node_in(prev_node, nodes);
> + weight = weights[i];
> + node_pages = weight * rounds;
> + /* If a delta exists, add this node's portion of the delta */
> + if (delta >= weight) {
> + node_pages += weight;
> + delta -= weight;
> + resume_node = node;
> + resume_weight = i < (nnodes - 1) ? weights[i+1] :
> + weights[0];
> + /* stop tracking iff (delta == weight) */
> + delta_depleted = !delta;
> + } else if (delta) { /* <= weight */
The comment is unnecessary and wrong.
> + /* if delta depleted, resume from this node */
> + node_pages += delta;
> + delta = 0;
> + resume_node = prev_node;
> + resume_weight = weight - (node_pages % weight);
> + delta_depleted = true; /* stop tracking */
> + } else if (!delta_depleted) {
> + /* if there was no delta, track last allocated node */
> + resume_node = node;
> + resume_weight = i < (nnodes - 1) ? weights[i+1] :
> + weights[0];
> + }
Can the above code be simplified as something like below?
resume_node = prev_node;
resume_weight = 0;
for (...) {
...
if (delta > weight) {
node_pages += weight;
delta -= weight;
} else if (delta) {
node_pages += delta;
/* if delta depleted, resume from this node */
if (delta < weight) {
resume_node = prev_node;
resume_weight = weight - delta;
} else {
resume_node = node;
}
delta = 0;
}
...
}
--
Best Regards,
Huang, Ying
> + /* node_pages can be 0 if an allocation fails and rounds == 0 */
> + if (!node_pages)
> + break;
> + nr_allocated = __alloc_pages_bulk(gfp, node, NULL, node_pages,
> + NULL, page_array);
> + page_array += nr_allocated;
> + total_allocated += nr_allocated;
> + if (total_allocated == nr_pages)
> + break;
> + prev_node = node;
> + }
> + /* resume allocating from the calculated node and weight */
> + me->il_prev = resume_node;
> + pol->wil.cur_weight = resume_weight;
> + kfree(weights);
> + return total_allocated;
> +}
> +
> static unsigned long alloc_pages_bulk_array_preferred_many(gfp_t gfp, int nid,
> struct mempolicy *pol, unsigned long nr_pages,
> struct page **page_array)
> @@ -2303,6 +2519,10 @@ unsigned long alloc_pages_bulk_array_mempolicy(gfp_t gfp,
> return alloc_pages_bulk_array_interleave(gfp, pol,
> nr_pages, page_array);
>
> + if (pol->mode == MPOL_WEIGHTED_INTERLEAVE)
> + return alloc_pages_bulk_array_weighted_interleave(
> + gfp, pol, nr_pages, page_array);
> +
> if (pol->mode == MPOL_PREFERRED_MANY)
> return alloc_pages_bulk_array_preferred_many(gfp,
> numa_node_id(), pol, nr_pages, page_array);
> @@ -2378,6 +2598,7 @@ bool __mpol_equal(struct mempolicy *a, struct mempolicy *b)
> case MPOL_INTERLEAVE:
> case MPOL_PREFERRED:
> case MPOL_PREFERRED_MANY:
> + case MPOL_WEIGHTED_INTERLEAVE:
> return !!nodes_equal(a->nodes, b->nodes);
> case MPOL_LOCAL:
> return true;
> @@ -2514,6 +2735,10 @@ int mpol_misplaced(struct folio *folio, struct vm_area_struct *vma,
> polnid = interleave_nid(pol, ilx);
> break;
>
> + case MPOL_WEIGHTED_INTERLEAVE:
> + polnid = weighted_interleave_nid(pol, ilx);
> + break;
> +
> case MPOL_PREFERRED:
> if (node_isset(curnid, pol->nodes))
> goto out;
> @@ -2888,6 +3113,7 @@ static const char * const policy_modes[] =
> [MPOL_PREFERRED] = "prefer",
> [MPOL_BIND] = "bind",
> [MPOL_INTERLEAVE] = "interleave",
> + [MPOL_WEIGHTED_INTERLEAVE] = "weighted interleave",
> [MPOL_LOCAL] = "local",
> [MPOL_PREFERRED_MANY] = "prefer (many)",
> };
> @@ -2947,6 +3173,7 @@ int mpol_parse_str(char *str, struct mempolicy **mpol)
> }
> break;
> case MPOL_INTERLEAVE:
> + case MPOL_WEIGHTED_INTERLEAVE:
> /*
> * Default to online nodes with memory if no nodelist
> */
> @@ -3057,6 +3284,7 @@ void mpol_to_str(char *buffer, int maxlen, struct mempolicy *pol)
> case MPOL_PREFERRED_MANY:
> case MPOL_BIND:
> case MPOL_INTERLEAVE:
> + case MPOL_WEIGHTED_INTERLEAVE:
> nodes = pol->nodes;
> break;
> default:
^ permalink raw reply
page: next (older) | prev (newer) | latest
- recent:[subjects (threaded)|topics (new)|topics (active)]
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox