Re: [Qemu-devel] [Bug 891525] [NEW] Guest kernel crashes when booting a NUMA guest without explicitly specifying cpus= in -numa option

[Bharata B Rao <bharata@linux.vnet.ibm.com>]

The reason for guest kernel crash is because qemu enumerates the VCPUs
in a round robin fashion between the nodes. As per this comment from
vl.c, guest kernel is supposed to handle this:

/* assigning the VCPUs round-robin is easier to implement, guest OSes
 * must cope with this anyway, because there are BIOSes out there in
 * real machines which also use this scheme.
 */

I am not sure if this would be considered a bug in the guest kernel,
but I have verifed that enumerating the VCPUs in a serial manner between
nodes fixes the problem for me. I am not aware of the history behind
round robin assignment nor do I understand the full implications of
changing it, but here is a potential patch that fixes the problem for me.
---

Fix VCPU enumeration between nodes

From: Bharata B Rao <bharata@linux.vnet.ibm.com>

Currently VCPUs are assigned to nodes in round robin manner.
This is seen to break guest kernel for x86_64-softmmu. Hence assign
VCPUs serially between nodes.

Signed-off-by: Bharata B Rao <bharata@linux.vnet.ibm.com>
---

 vl.c |   15 ++++++++++-----
 1 files changed, 10 insertions(+), 5 deletions(-)


diff --git a/vl.c b/vl.c
index f5afed4..75348d0 100644
--- a/vl.c
+++ b/vl.c
@@ -3307,13 +3307,18 @@ int main(int argc, char **argv, char **envp)
             if (node_cpumask[i] != 0)
                 break;
         }
-        /* assigning the VCPUs round-robin is easier to implement, guest OSes
-         * must cope with this anyway, because there are BIOSes out there in
-         * real machines which also use this scheme.
-         */
+        /* Assign VCPUs to nodes in serial manner */
         if (i == nb_numa_nodes) {
+            int cpus_per_node = smp_cpus / nb_numa_nodes;
+
             for (i = 0; i < smp_cpus; i++) {
-                node_cpumask[i % nb_numa_nodes] |= 1 << i;
+                int nodeid = i / cpus_per_node;
+
+                /* Extra VCPUs goto Node 0 */
+                if (nodeid >= nb_numa_nodes) {
+                    nodeid = 0;
+                }
+                node_cpumask[nodeid] |= 1 << i;
             }
         }
     }