[PATCH v8 12/13] xl: enable for specifying soft-affinity in the config file

[Dario Faggioli <dario.faggioli@citrix.com>]

in a similar way to how hard-affinity is specified (i.e.,
exactly how plain vcpu-affinity was being specified before
this change).

To do so, we add a vcpu_soft_affinity array to build_info,
and treat it much like vcpu_hard_affinity. The new config
option is called "cpus_soft".

Signed-off-by: Dario Faggioli <dario.faggioli@citrix.com>
---
Changes from v7:
 * WARNING: this patch underwent quite a fundamental rework,
   given it's now building on top of Wei's "push vcpu affinity
   to libxl" patch. That's why I think it should be re-reviewed
   almost from scratch (sorry! :-P), and that's why I did not
   add IanC's ack, although he provided it to the v7 version
   of it.

Changes from v6:
 * update and improve the changelog.

Changes from v4:
 * fix typos and rephrase docs, as suggested during review;
 * more refactoring, i.e., more addressing factor of potential
   common code, as requested during review.

Changes from v3:
 * fix typos and language issues in docs and comments, as
   suggested during review;
 * common code to soft and hard affinity parsing factored
   together, as requested uring review.

Changes from v2:
 * use the new libxl API. Although the implementation changed
   only a little bit, I removed IanJ's Acked-by, although I am
   here saying that he did provided it, as requested.
---
 docs/man/xl.cfg.pod.5       |   23 ++++++++++++++++++++---
 tools/libxl/libxl.h         |   14 ++++++++++++++
 tools/libxl/libxl_dom.c     |   19 +++++++++++++------
 tools/libxl/libxl_types.idl |    1 +
 tools/libxl/xl_cmdimpl.c    |   40 +++++++++++++++++++++++++++-------------
 5 files changed, 75 insertions(+), 22 deletions(-)

diff --git a/docs/man/xl.cfg.pod.5 b/docs/man/xl.cfg.pod.5
index af48622..25a4ff7 100644
--- a/docs/man/xl.cfg.pod.5
+++ b/docs/man/xl.cfg.pod.5
@@ -152,19 +152,36 @@ run on cpus 3,4,6,7,8 of the host.
 =back
 
 If this option is not specified, no vcpu to cpu pinning is established,
-and the vcpus of the guest can run on all the cpus of the host.
+and the vcpus of the guest can run on all the cpus of the host. If this
+option is specified, the intersection of the vcpu pinning mask, provided
+here, and the soft affinity mask, provided via B<cpus\_soft=> (if any),
+is utilized to compute the domain node-affinity, for driving memory
+allocations.
 
 If we are on a NUMA machine (i.e., if the host has more than one NUMA
 node) and this option is not specified, libxl automatically tries to
 place the guest on the least possible number of nodes. That, however,
 will not affect vcpu pinning, so the guest will still be able to run on
-all the cpus, it will just prefer the ones from the node it has been
-placed on. A heuristic approach is used for choosing the best node (or
+all the cpus. A heuristic approach is used for choosing the best node (or
 set of nodes), with the goals of maximizing performance for the guest
 and, at the same time, achieving efficient utilization of host cpus
 and memory. See F<docs/misc/xl-numa-placement.markdown> for more
 details.
 
+=item B<cpus_soft="CPU-LIST">
+
+Exactly as B<cpus=>, but specifies soft affinity, rather than pinning
+(hard affinity). When using the credit scheduler, this means what cpus
+the vcpus of the domain prefer.
+
+A C<CPU-LIST> is specified exactly as above, for B<cpus=>.
+
+If this option is not specified, the vcpus of the guest will not have
+any preference regarding on what cpu to run. If this option is specified,
+the intersection of the soft affinity mask, provided here, and the vcpu
+pinning, provided via B<cpus=> (if any), is utilized to compute the
+domain node-affinity, for driving memory allocations.
+
 =back
 
 =head3 CPU Scheduling
diff --git a/tools/libxl/libxl.h b/tools/libxl/libxl.h
index 663abe2..a7c60a4 100644
--- a/tools/libxl/libxl.h
+++ b/tools/libxl/libxl.h
@@ -353,6 +353,20 @@
 #define LIBXL_HAVE_BUILDINFO_VCPU_HARD_AFFINITY_ARRAY 1
 
 /*
+ * LIBXL_HAVE_BUILDINFO_VCPU_SOFT_AFFINITY_ARRAY
+ *
+ * If this is defined, then libxl_domain_build_info structure will
+ * contain vcpu_soft_affinity, an array of libxl_bitmap that contains
+ * the necessary information to set the soft affinity of each VCPU to
+ * a set of PCPUs.
+ *
+ * The number of libxl_bitmap in the array equals to the maximum number
+ * of VCPUs. The size of each bitmap is computed basing on the maximum
+ * number of PCPUs.
+ */
+#define LIBXL_HAVE_BUILDINFO_VCPU_HARD_AFFINITY_ARRAY 1
+
+/*
  * LIBXL_HAVE_BUILDINFO_USBDEVICE_LIST
  *
  * If this is defined, then the libxl_domain_build_info structure will
diff --git a/tools/libxl/libxl_dom.c b/tools/libxl/libxl_dom.c
index 0b00470..9874dc4 100644
--- a/tools/libxl/libxl_dom.c
+++ b/tools/libxl/libxl_dom.c
@@ -262,14 +262,21 @@ int libxl__build_pre(libxl__gc *gc, uint32_t domid,
     }
     libxl_domain_set_nodeaffinity(ctx, domid, &info->nodemap);
 
-    /* If we have the vcpu hard affinity list, honour it */
-    if (d_config->b_info.num_vcpu_hard_affinity) {
+    /* If we have the vcpu hard and/or soft affinity list, honour it/them */
+    if (d_config->b_info.num_vcpu_hard_affinity ||
+        d_config->b_info.num_vcpu_soft_affinity) {
         int i;
 
-        for (i = 0; i < d_config->b_info.num_vcpu_hard_affinity; i++) {
-            if (libxl_set_vcpuaffinity(ctx, domid, i,
-                                       &d_config->b_info.vcpu_hard_affinity[i],
-                                       NULL)) {
+        for (i = 0; i < d_config->b_info.max_vcpus; i++) {
+            libxl_bitmap *hard_affinity, *soft_affinity;
+
+            hard_affinity = d_config->b_info.num_vcpu_hard_affinity ?
+                &d_config->b_info.vcpu_hard_affinity[i] : NULL;
+            soft_affinity = d_config->b_info.num_vcpu_soft_affinity ?
+                &d_config->b_info.vcpu_soft_affinity[i] : NULL;
+
+            if (libxl_set_vcpuaffinity(ctx, domid, i, hard_affinity,
+                                       soft_affinity)) {
                 LOG(ERROR, "setting affinity failed on vcpu `%d'", i);
                 return ERROR_FAIL;
             }
diff --git a/tools/libxl/libxl_types.idl b/tools/libxl/libxl_types.idl
index cd5c0d4..1744b00 100644
--- a/tools/libxl/libxl_types.idl
+++ b/tools/libxl/libxl_types.idl
@@ -304,6 +304,7 @@ libxl_domain_build_info = Struct("domain_build_info",[
     # is indeed harmless, but won't produce any actual effect on the domain.
     ("nodemap",         libxl_bitmap),
     ("vcpu_hard_affinity", Array(libxl_bitmap, "num_vcpu_hard_affinity")),
+    ("vcpu_soft_affinity", Array(libxl_bitmap, "num_vcpu_soft_affinity")),
     ("numa_placement",  libxl_defbool),
     ("tsc_mode",        libxl_tsc_mode),
     ("max_memkb",       MemKB),
diff --git a/tools/libxl/xl_cmdimpl.c b/tools/libxl/xl_cmdimpl.c
index d08a149..d0a8557 100644
--- a/tools/libxl/xl_cmdimpl.c
+++ b/tools/libxl/xl_cmdimpl.c
@@ -693,22 +693,32 @@ static void parse_top_level_sdl_options(XLU_Config *config,
 }
 
 static void parse_vcpu_affinity(XLU_Config *config, XLU_ConfigList *cpus,
-                                const char *buf, libxl_domain_build_info *info)
+                                const char *buf, libxl_domain_build_info *info,
+                                bool is_hard)
 {
-    info->num_vcpu_hard_affinity = info->max_vcpus;
     const char *buf2;
+    libxl_bitmap *vcpu_affinity_array;
     int i;
 
-    info->vcpu_hard_affinity =
-        xmalloc(info->num_vcpu_hard_affinity * sizeof(libxl_bitmap));
+    if (is_hard) {
+        info->num_vcpu_hard_affinity = info->max_vcpus;
+        info->vcpu_hard_affinity =
+            xmalloc(info->max_vcpus * sizeof(libxl_bitmap));
+        vcpu_affinity_array = info->vcpu_hard_affinity;
+    } else {
+        info->num_vcpu_soft_affinity = info->max_vcpus;
+        info->vcpu_soft_affinity =
+            xmalloc(info->max_vcpus * sizeof(libxl_bitmap));
+        vcpu_affinity_array = info->vcpu_soft_affinity;
+    }
 
-    for (i = 0; i < info->num_vcpu_hard_affinity; i++) {
-        libxl_bitmap_init(&info->vcpu_hard_affinity[i]);
-        if (libxl_cpu_bitmap_alloc(ctx, &info->vcpu_hard_affinity[i], 0)) {
+    for (i = 0; i < info->max_vcpus; i++) {
+        libxl_bitmap_init(&vcpu_affinity_array[i]);
+        if (libxl_cpu_bitmap_alloc(ctx, &vcpu_affinity_array[i], 0)) {
             fprintf(stderr, "Unable to allocate cpumap for vcpu %d\n", i);
             exit(1);
         }
-        libxl_bitmap_set_none(&info->vcpu_hard_affinity[i]);
+        libxl_bitmap_set_none(&vcpu_affinity_array[i]);
     }
 
     /*
@@ -720,13 +730,12 @@ static void parse_vcpu_affinity(XLU_Config *config, XLU_ConfigList *cpus,
      * vcpus than elements, the missing ones have their affinity masks
      * completely full).
      */
-    for (i = 0; i < info->num_vcpu_hard_affinity; i++) {
+    for (i = 0; i < info->max_vcpus; i++) {
         if (buf || ((buf2 = xlu_cfg_get_listitem(cpus, i)) != NULL)) {
-            if (vcpupin_parse(buf ? buf : buf2,
-                              &info->vcpu_hard_affinity[i]))
+            if (vcpupin_parse(buf ? buf : buf2, &vcpu_affinity_array[i]))
                 exit(1);
         } else
-            libxl_bitmap_set_any(&info->vcpu_hard_affinity[i]);
+            libxl_bitmap_set_any(&vcpu_affinity_array[i]);
     }
 
     /* We have a list of cpumaps, disable automatic placement */
@@ -867,7 +876,12 @@ static void parse_config_data(const char *config_source,
     buf = NULL;
     if (!xlu_cfg_get_list (config, "cpus", &cpus, 0, 1) ||
         !xlu_cfg_get_string (config, "cpus", &buf, 0))
-        parse_vcpu_affinity(config, cpus, buf, b_info);
+        parse_vcpu_affinity(config, cpus, buf, b_info, /* is_hard */ true);
+
+    buf = NULL;
+    if (!xlu_cfg_get_list (config, "cpus_soft", &cpus, 0, 1) ||
+        !xlu_cfg_get_string (config, "cpus_soft", &buf, 0))
+        parse_vcpu_affinity(config, cpus, buf, b_info, /* is_hard */ false);
 
     if (!xlu_cfg_get_long (config, "memory", &l, 0)) {
         b_info->max_memkb = l * 1024;