Re: [RFC PATCH v2 4/4] mm/mempolicy: implement a weighted-interleave

[Gregory Price <gregory.price@memverge.com>]

On Wed, Dec 06, 2023 at 05:09:44PM +0900, Seungjun Ha wrote:
> > The weighted-interleave mempolicy implements weights per-node
> > which are used to distribute memory while interleaving.
> >
> > For example:
> >    nodes: 0,1,2
> >    weights: 5,3,2
> >
> > Over 10 consecutive allocations, the following nodes will be selected:
> > [0,0,0,0,0,1,1,1,2,2]
> >
> > In this example there is a 50%/30%/20% distribution of memory across
> > the enabled nodes.
> >
> > If a node is enabled, the minimum weight is expected to be 0. If an
> > enabled node ends up with a weight of 0 (as can happen if weights
> > are being recalculated due to a cgroup mask update), a minimum
> > of 1 is applied during the interleave mechanism.
> 
> I found an issue while using the RFCv2, and want to report it. 

first, thank you very much for testing! I'll run this on my latest fork.

> In my testbed, calling set_mempolicy2() causes pthread_create() failure or system hang, depending on weight combinations.
> 

I think this is likely because i did not handle __mpol_dup correctly.
The newer fork changes the way weights are stored, so this should not
but an issue, but I will use your test to validate this.

New RFC should hopefully be out this or next week.

> FYI please find my testbed where there are 3 memory-nodes. 
> 
>             Node 0  Node 1  Node 2  Result
> Weights  
>             6 >=    1       1       pthread_create error: 11(Cannot allocate memory)
>             1~5     1       1       Pass
>             1       8 >=    1       pthread_create error: 11(Cannot allocate memory)
>             1       1~7     1       Pass
>             1       1       8 >=    pthread_create error: 11(Cannot allocate memory)
>             1       1       1~7     Pass
> 
>             6       7       7       pthread_create error: 11(Cannot allocate memory)
>             5       8       7       Pass
>             5       7       8       Pass
> 
>             40      30      20      Kernel Hang
> 
> 
> Below is the test code to reproduce the issue.
> 
> #define _GNU_SOURCE
> #include <stdio.h>
> #include <stdlib.h>
> #include <numa.h>
> #include <errno.h>
> #include <numaif.h>
> #include <unistd.h>
> #include <pthread.h>
> 
> #define MPOL_WEIGHTED_INTERLEAVE MPOL_DEFAULT + 8
> #define SET_MEMPOLICY2(a, b) syscall(454, a, b)
> 
> struct mempolicy_args { on this RFC... }
> 
> struct mempolicy_args wil_args;
> struct bitmask *wil_nodes;
> unsigned char *weights;
> int total_nodes = -1;
> pthread_t tid;
> 
> void set_mempolicy_call()
> {
>         weights = (unsigned char *)calloc(total_nodes, sizeof(unsigned char));
>         wil_nodes = numa_allocate_nodemask();
> 
>         numa_bitmask_setbit(wil_nodes, 0); weights[0] = 40;
>         numa_bitmask_setbit(wil_nodes, 1); weights[1] = 30;
>         numa_bitmask_setbit(wil_nodes, 2); weights[2] = 20;
> 
>         wil_args.maxnode = total_nodes + 1;
>         wil_args.wil.weights = weights;
>         wil_args.nodemask = wil_nodes->maskp;
>         wil_args.mode = MPOL_WEIGHTED_INTERLEAVE;
>         wil_args.flags = 0;
> 
>         int ret = SET_MEMPOLICY2(&wil_args, sizeof(wil_args));
>         fprintf(stderr, "set_mempolicy2 result: %d(%s)\n", ret, strerror(errno));
> }
> 
> 
> int main()
> {
>         total_nodes = numa_max_node() + 1;
> 
>         set_mempolicy_call();
>         pthread_create(&tid, NULL, func, NULL);
> 
>         return 0;
> }