[PATCH v2 0/2] xen: credit2: flexible configuration for runqueues

[PATCH v2 0/2] xen: credit2: flexible configuration for runqueues

[Praveen Kumar]

Hello,

The idea is to give user more flexibility to configure runqueue further.
For most workloads and in most systems, using per-core means have too many
small runqueues. Using per-socket is almost always better, but it may result
in too few big runqueues.

OPTION 1 :
--------
The user can create runqueue per-cpu using Xen boot parameter like below:

 credit2_runqueue=cpu

which would mean the following:
 - pCPU 0 belong to runqueue 0
 - pCPU 1 belong to runqueue 1
 - pCPU 2 belong to runqueue 2
 and so on.

OPTION 2 :
--------
Further user can be allowed to say something shown below :

 credit2_runqueue=0,1,4,5;2,3,6,7;8,9,12,13;10,11,14,15

or (with exactly the same meaning, but a perhaps more clear syntax):

 credit2_runqueue=[[0,1,4,5][2,3,6,7][8,9,12,13][10,11,14,15]]

which would mean the following:
 - pCPUs 0, 1, 4 and 5 belong to runqueue 0
 - pCPUs 2, 3, 6 and 7 belong to runqueue 1
 - pCPUs 8, 9, 12 and 13 belong to runqueue 2
 - pCPUs 10, 11, 14 and 15 belong to runqueue 3

Thanks and Regards,
Praveen Kumar.
---
Praveen Kumar (2):
  xen: credit2: enable per cpu runqueue creation
  xen: credit2: provide flexible option to create runqueue

 docs/misc/xen-command-line.markdown |   9 +-
 xen/common/sched_credit2.c          | 185 +++++++++++++++++++++++++++++++++++-
 2 files changed, 188 insertions(+), 6 deletions(-)

-- 
2.12.0


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

[PATCH v2 1/2] xen: credit2: enable per cpu runqueue creation

[Praveen Kumar]

The patch introduces a new command line option 'cpu' that when used will
create runqueue per logical pCPU.

Signed-off-by: Praveen Kumar <kpraveen.lkml@gmail.com>
---
 docs/misc/xen-command-line.markdown |  3 ++-
 xen/common/sched_credit2.c          | 15 +++++++++++----
 2 files changed, 13 insertions(+), 5 deletions(-)

diff --git a/docs/misc/xen-command-line.markdown b/docs/misc/xen-command-line.markdown
index 9eb85d68b5..c245cfa471 100644
--- a/docs/misc/xen-command-line.markdown
+++ b/docs/misc/xen-command-line.markdown
@@ -525,7 +525,7 @@ also slow in responding to load changes.
 The default value of `1 sec` is rather long.
 
 ### credit2\_runqueue
-> `= core | socket | node | all`
+> `= cpu | core | socket | node | all`
 
 > Default: `socket`
 
@@ -536,6 +536,7 @@ balancing (for instance, it will deal better with hyperthreading),
 but also more overhead.
 
 Available alternatives, with their meaning, are:
+* `cpu`: one runqueue per each logical pCPUs of the host;
 * `core`: one runqueue per each physical core of the host;
 * `socket`: one runqueue per each physical socket (which often,
             but not always, matches a NUMA node) of the host;
diff --git a/xen/common/sched_credit2.c b/xen/common/sched_credit2.c
index bb1c657e76..ee7b443f9e 100644
--- a/xen/common/sched_credit2.c
+++ b/xen/common/sched_credit2.c
@@ -301,6 +301,9 @@ integer_param("credit2_balance_over", opt_overload_balance_tolerance);
  * want that to happen basing on topology. At the moment, it is possible
  * to choose to arrange runqueues to be:
  *
+ * - per-cpu: meaning that there will be one runqueue per logical cpu. This
+ *            will happen when if the opt_runqueue parameter is set to 'cpu'.
+ *
  * - per-core: meaning that there will be one runqueue per each physical
  *             core of the host. This will happen if the opt_runqueue
  *             parameter is set to 'core';
@@ -322,11 +325,13 @@ integer_param("credit2_balance_over", opt_overload_balance_tolerance);
  * either the same physical core, the same physical socket, the same NUMA
  * node, or just all of them, will be put together to form runqueues.
  */
-#define OPT_RUNQUEUE_CORE   0
-#define OPT_RUNQUEUE_SOCKET 1
-#define OPT_RUNQUEUE_NODE   2
-#define OPT_RUNQUEUE_ALL    3
+#define OPT_RUNQUEUE_CPU    0
+#define OPT_RUNQUEUE_CORE   1
+#define OPT_RUNQUEUE_SOCKET 2
+#define OPT_RUNQUEUE_NODE   3
+#define OPT_RUNQUEUE_ALL    4
 static const char *const opt_runqueue_str[] = {
+    [OPT_RUNQUEUE_CPU] = "cpu",
     [OPT_RUNQUEUE_CORE] = "core",
     [OPT_RUNQUEUE_SOCKET] = "socket",
     [OPT_RUNQUEUE_NODE] = "node",
@@ -682,6 +687,8 @@ cpu_to_runqueue(struct csched2_private *prv, unsigned int cpu)
         BUG_ON(cpu_to_socket(cpu) == XEN_INVALID_SOCKET_ID ||
                cpu_to_socket(peer_cpu) == XEN_INVALID_SOCKET_ID);
 
+        if (opt_runqueue == OPT_RUNQUEUE_CPU)
+            continue;
         if ( opt_runqueue == OPT_RUNQUEUE_ALL ||
              (opt_runqueue == OPT_RUNQUEUE_CORE && same_core(peer_cpu, cpu)) ||
              (opt_runqueue == OPT_RUNQUEUE_SOCKET && same_socket(peer_cpu, cpu)) ||
-- 
2.12.0


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

[PATCH v2 2/2] xen: credit2: provide custom option to create

[Praveen Kumar]

The patch introduces a new command line option 'custom' that when used will
create runqueue based upon the pCPU subset provide during bootup.

Signed-off-by: Praveen Kumar <kpraveen.lkml@gmail.com>
---
 docs/misc/xen-command-line.markdown |   8 +-
 xen/common/sched_credit2.c          | 170 +++++++++++++++++++++++++++++++++++-
 2 files changed, 176 insertions(+), 2 deletions(-)

diff --git a/docs/misc/xen-command-line.markdown b/docs/misc/xen-command-line.markdown
index c245cfa471..e65056b3d0 100644
--- a/docs/misc/xen-command-line.markdown
+++ b/docs/misc/xen-command-line.markdown
@@ -525,7 +525,7 @@ also slow in responding to load changes.
 The default value of `1 sec` is rather long.
 
 ### credit2\_runqueue
-> `= cpu | core | socket | node | all`
+> `= cpu | core | socket | node | all | custom`
 
 > Default: `socket`
 
@@ -543,6 +543,12 @@ Available alternatives, with their meaning, are:
 * `node`: one runqueue per each NUMA node of the host;
 * `all`: just one runqueue shared by all the logical pCPUs of
          the host
+* `custom`: one runqueue per subset. Example:
+            credit2_runqueue=[[0,1,4,5][2,3,6,7][8,9,12,13][10,11,14,15]]
+                - pCPUs 0, 1, 4 and 5 belong to runqueue 0
+                - pCPUs 2, 3, 6 and 7 belong to runqueue 1
+                - pCPUs 8, 9, 12 and 13 belong to runqueue 2
+                - pCPUs 10, 11, 14 and 15 belong to runqueue 3
 
 ### dbgp
 > `= ehci[ <integer> | @pci<bus>:<slot>.<func> ]`
diff --git a/xen/common/sched_credit2.c b/xen/common/sched_credit2.c
index ee7b443f9e..9234b023cb 100644
--- a/xen/common/sched_credit2.c
+++ b/xen/common/sched_credit2.c
@@ -321,6 +321,15 @@ integer_param("credit2_balance_over", opt_overload_balance_tolerance);
  *           (logical) processors of the host belong. This will happen if
  *           the opt_runqueue parameter is set to 'all'.
  *
+ * - custom: meaning that there will be one runqueue per subset being passed as
+ *           parameter to credit2_runqueue as shown in below example.
+ *           Example:
+ *           credit2_runqueue=[[cpu0,cpu1][cpu3][cpu4,cpu5]]
+ *           The example mentioned states :
+ *               cpu0 and cpu1 belongs to runqueue 0
+ *               cpu3 belongs to runqueue 1
+ *               cpu4 and cpu5 belongs to runqueue 2
+ *
  * Depending on the value of opt_runqueue, therefore, cpus that are part of
  * either the same physical core, the same physical socket, the same NUMA
  * node, or just all of them, will be put together to form runqueues.
@@ -330,18 +339,60 @@ integer_param("credit2_balance_over", opt_overload_balance_tolerance);
 #define OPT_RUNQUEUE_SOCKET 2
 #define OPT_RUNQUEUE_NODE   3
 #define OPT_RUNQUEUE_ALL    4
+#define OPT_RUNQUEUE_CUSTOM 5
 static const char *const opt_runqueue_str[] = {
     [OPT_RUNQUEUE_CPU] = "cpu",
     [OPT_RUNQUEUE_CORE] = "core",
     [OPT_RUNQUEUE_SOCKET] = "socket",
     [OPT_RUNQUEUE_NODE] = "node",
-    [OPT_RUNQUEUE_ALL] = "all"
+    [OPT_RUNQUEUE_ALL] = "all",
+    [OPT_RUNQUEUE_CUSTOM] = "custom"
 };
 static int __read_mostly opt_runqueue = OPT_RUNQUEUE_SOCKET;
 
+static int __read_mostly custom_cpu_runqueue[NR_CPUS];
+
+#define GETTOKEN( token, len, start, end )               \
+{                                                        \
+    char _tmp[len+1];                                    \
+    int _i;                                              \
+    safe_strcpy(_tmp, start);                            \
+    _tmp[len] = '\0';                                    \
+    for ( _i = 0; _tmp[_i] != '\0'; _i++ )               \
+        token = ( ( token * 10 ) + ( _tmp[_i] - '0' ) ); \
+}
+
+static inline int trim(const char *c, char *t, char elem)
+{
+    int l = strlen(c);
+    const char *x = c ;
+    int i = 0;
+    if ( !c || !t )
+        return -1;
+    while ( *x != '\0' && i < l )
+    {
+        if ( *x != elem )
+            t[i++] = *x;
+        x++;
+    }
+    t[i] = '\0';
+    return 0;
+}
+
+static inline int getlen(char *start, char *end)
+{
+    if ( ( start ) && ( end ) && ( end > start ) )
+        return end-start;
+    else
+        return -1;
+}
+
 static void parse_credit2_runqueue(const char *s)
 {
     unsigned int i;
+    const char *s_end = NULL;
+    char m[strlen(s)];
+    char *_s = NULL;
 
     for ( i = 0; i < ARRAY_SIZE(opt_runqueue_str); i++ )
     {
@@ -351,7 +402,115 @@ static void parse_credit2_runqueue(const char *s)
             return;
         }
     }
+/*
+     * At this stage we are either unknown value of credit2_runqueue or we can
+     * consider it to be custom cpu. Lets try parsing the same.
+     * Resetting the custom_cpu_runqueue for future use. Only the non-negative
+     * entries will be valid. The index 'i' in custom_cpu_runqueue will store
+     * the specific runqueue it belongs to.
+     * Example:
+     *     If custom_cpu_runqueue[3] == 2
+     *     Then, it means that cpu 3 belong to runqueue 2.
+     *     If custom_cpu_runqueue[4] == -1
+     *     Then, it means that cpu 4 doesn't belong to any runqueue.
+     */
+    for ( i = 0; i < nr_cpu_ids; i++ )
+        custom_cpu_runqueue[i] = -1;
+
+    /*
+     * Format [[0,1,4,5][2,3,6,7][8,9,12,13][10,11,14,15]]
+     */
+    i = 0;
+
+    /* In case user have spaces included in the input */
+    if ( trim(s, m, ' ') != 0 )
+    {
+        printk( "WARNING : %s[%d] trim failed.\n", __func__, __LINE__ );
+        goto errReturn;
+    }
+    /* Starting to parse and get the cpu information on trimmed data */
+    _s = m;
+    s_end = _s + strlen(_s);
+
+    /* The start and should always be in format of '[..]' */
+    if ( ( '[' == *_s ) && ( ']' == *(s_end-1)) )
+    {
+        char *start = NULL, *end = NULL;
+        int cpu_added_to_runqueue = 0;
+        _s++;
+        while ( ( _s != NULL ) && ( _s < s_end ) )
+        {
+            char *token_sub_str = NULL;
+            start = strstr(_s, "[");
+            end = strstr(_s, "]");
+            /* Validation checks for '[' and ']' to properly parse the entries
+            */
+            if ( ( !start && !end ) || ( ( end == ( s_end -1 ) ) && start ) )
+                goto errReturn;
+            /* Check for last entry */
+            else if ( !start && ( end == ( s_end -1 ) ) )
+                goto nextSet;
+
+            /* If we have [] as entry, move to nextSet */
+            if ( getlen ( start, end ) < 1 )
+                goto nextSet;
+            /* Start to parse the actual value */
+            start++;
+            /*
+             * find token within the subset
+             */
+            do
+            {
+                int token = 0;
+                int len = 0;
+                /* Get cpu ids separated by ',' within each set */
+                token_sub_str = strpbrk(start, ",");
+                if ( ( !token_sub_str && start < end ) ||
+                    ( token_sub_str > end && token_sub_str > start ) )
+                    len = getlen(start, end);
+                else
+                    len = getlen(start, token_sub_str);
+
+                if ( len <= 0 )
+                    continue;
+                /* GETTOKEN will get return the parse and populate the cpu in
+                 * token
+                 */
+                GETTOKEN(token, len, start , end );
+
+                if ( token >= nr_cpu_ids)
+                    goto errReturn;
+                /* If not set already */
+                if ( custom_cpu_runqueue[token] == -1 )
+                {
+                    custom_cpu_runqueue[token] = i;
+                    cpu_added_to_runqueue = 1;
+                }
+                else
+                    goto errReturn;
+
+                if ( !token_sub_str || token_sub_str > end )
+                    goto nextSet;
+
+                start = ++token_sub_str;
+            } while ( start < end );
+nextSet:
+            if ( cpu_added_to_runqueue )
+            {
+                i++;
+                cpu_added_to_runqueue = 0;
+            }
 
+            _s = ++end;
+        }
+        opt_runqueue = OPT_RUNQUEUE_CUSTOM;
+        return;
+    }
+errReturn:
+    /* Resetting in case of failure, so that we don't mess-up during any failure
+     * due to wrong or spurious pattern passed by user.
+     */
+    opt_runqueue = OPT_RUNQUEUE_SOCKET;
     printk("WARNING, unrecognized value of credit2_runqueue option!\n");
 }
 custom_param("credit2_runqueue", parse_credit2_runqueue);
@@ -661,6 +820,15 @@ cpu_to_runqueue(struct csched2_private *prv, unsigned int cpu)
     struct csched2_runqueue_data *rqd;
     unsigned int rqi;
 
+    if ( opt_runqueue == OPT_RUNQUEUE_CUSTOM )
+    {
+        if ( custom_cpu_runqueue[cpu] != -1 )
+        {
+            BUG_ON(custom_cpu_runqueue[cpu] >= nr_cpu_ids);
+            return custom_cpu_runqueue[cpu];
+        }
+    }
+
     for ( rqi = 0; rqi < nr_cpu_ids; rqi++ )
     {
         unsigned int peer_cpu;
-- 
2.12.0


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel