Re: Problems with merlot* AMD Opteron 6376 systems (Was Re: stable trees (was: [xen-4.2-testing test] 58584: regressions))

[Dario Faggioli <dario.faggioli@citrix.com>]

[-- Attachment #1.1: Type: text/plain, Size: 5912 bytes --]

On Wed, 2015-06-24 at 10:38 +0100, Ian Campbell wrote:
> Adding Boris+Suravee+Aravind (AMD/SVM maintainers), Dario (NUMA) and Jim
> +Anthony (libvirt) to the CC.

> Supposing that the NUMA oddities might be what is exposing this issue I
> tried an adhoc run on the merlot machines where I specified
> "dom0_max_vcpus=8 dom0_nodes=0" on the hypervisor command line:
> http://logs.test-lab.xenproject.org/osstest/logs/58853/
> 
> Again, I messed up the config for the -xsm case, so ignore.
> 
> The interesting thing is that the extra NUMA settings were
> apparently_not_ helpful. From
> http://logs.test-lab.xenproject.org/osstest/logs/58853/test-amd64-amd64-libvirt/serial-merlot0.log I can see they were applied:
> Jun 23 15:50:34.205057 (XEN) Command line: placeholder conswitch=x watchdog com1=115200,8n1 console=com1,vga gdb=com1 dom0_mem=512M,max:512M ucode=scan dom0_max_vcpus=8 dom0_nodes=0
> [...]
> Jun 23 15:50:38.309057 (XEN) Dom0 has maximum 8 VCPUs
> 
IIRC, you can drop the dom0_max_vcpus=8, as Xen would figure it out
automatically, as a consequence of dom0_nodes=0. In any case, it doesn't
hurt.

Maybe we can try running this again with dom0_nodes=2 (the other node
with memory attached). I wouldn't know what to expect, though, so, yes,
it's a shot in the dark, but since we're out of plausible
theories... :-/

> The memory info
> Jun 23 15:56:27.749008 (XEN) Memory location of each domain:
> Jun 23 15:56:27.756965 (XEN) Domain 0 (total: 131072):
> Jun 23 15:56:27.756983 (XEN)     Node 0: 126905
> Jun 23 15:56:27.756998 (XEN)     Node 1: 0
> Jun 23 15:56:27.764952 (XEN)     Node 2: 4167
> Jun 23 15:56:27.764969 (XEN)     Node 3: 0
> suggests at least a small amount of cross-node memory allocation (16M
> out of dom0s 512M total). That's probably small enough to be OK.
> 
Yeah, that is in line with what you usually get with dom0_nodes. Most of
the memory, as you noted, comes from the proper node. We're just not
(yet?) at the point where _all_ of it can come from there.

> And it seems as if the 8 dom0 vcpus are correctly pinned to the first 8
> cpus (the ones in node 0):
> Jun 23 15:56:43.797055 (XEN) VCPU information and callbacks for domain 0:
> Jun 23 15:56:43.797110 (XEN)     VCPU0: CPU4 [has=F] poll=0 upcall_pend=00 upcall_mask=00 dirty_cpus={4}
> Jun 23 15:56:43.805078 (XEN)     cpu_hard_affinity={0-7} cpu_soft_affinity={0-7}
> Jun 23 15:56:43.813121 (XEN)     pause_count=0 pause_flags=1
> Jun 23 15:56:43.813157 (XEN)     No periodic timer
> Jun 23 15:56:43.821050 (XEN)     VCPU1: CPU3 [has=F] poll=0 upcall_pend=00 upcall_mask=00 dirty_cpus={3}
> Jun 23 15:56:43.829044 (XEN)     cpu_hard_affinity={0-7} cpu_soft_affinity={0-7}
> Jun 23 15:56:43.829082 (XEN)     pause_count=0 pause_flags=1
> Jun 23 15:56:43.837051 (XEN)     No periodic timer
> Jun 23 15:56:43.837084 (XEN)     VCPU2: CPU5 [has=F] poll=0 upcall_pend=00 upcall_mask=00 dirty_cpus={5}
> Jun 23 15:56:43.845102 (XEN)     cpu_hard_affinity={0-7} cpu_soft_affinity={0-7}
> Jun 23 15:56:43.853035 (XEN)     pause_count=0 pause_flags=1
> Jun 23 15:56:43.853071 (XEN)     No periodic timer
> Jun 23 15:56:43.853099 (XEN)     VCPU3: CPU7 [has=F] poll=0 upcall_pend=00 upcall_mask=00 dirty_cpus={7}
> Jun 23 15:56:43.861102 (XEN)     cpu_hard_affinity={0-7} cpu_soft_affinity={0-7}
> Jun 23 15:56:43.869110 (XEN)     pause_count=0 pause_flags=1
> Jun 23 15:56:43.869145 (XEN)     No periodic timer
> Jun 23 15:56:43.877014 (XEN)     VCPU4: CPU0 [has=F] poll=0 upcall_pend=00 upcall_mask=00 dirty_cpus={}
> Jun 23 15:56:43.877038 (XEN)     cpu_hard_affinity={0-7} cpu_soft_affinity={0-7}
> Jun 23 15:56:43.885053 (XEN)     pause_count=0 pause_flags=1
> Jun 23 15:56:43.885088 (XEN)     No periodic timer
> Jun 23 15:56:43.893085 (XEN)     VCPU5: CPU0 [has=F] poll=0 upcall_pend=00 upcall_mask=00 dirty_cpus={}
> Jun 23 15:56:43.901075 (XEN)     cpu_hard_affinity={0-7} cpu_soft_affinity={0-7}
> Jun 23 15:56:43.901134 (XEN)     pause_count=0 pause_flags=1
> Jun 23 15:56:43.909010 (XEN)     No periodic timer
> Jun 23 15:56:43.909048 (XEN)     VCPU6: CPU2 [has=F] poll=0 upcall_pend=00 upcall_mask=00 dirty_cpus={2}
> Jun 23 15:56:43.917065 (XEN)     cpu_hard_affinity={0-7} cpu_soft_affinity={0-7}
> Jun 23 15:56:43.925055 (XEN)     pause_count=0 pause_flags=1
> Jun 23 15:56:43.925074 (XEN)     No periodic timer
> Jun 23 15:56:43.925095 (XEN)     VCPU7: CPU6 [has=F] poll=0 upcall_pend=00 upcall_mask=00 dirty_cpus={6}
> Jun 23 15:56:43.933119 (XEN)     cpu_hard_affinity={0-7} cpu_soft_affinity={0-7}
> Jun 23 15:56:43.941080 (XEN)     pause_count=0 pause_flags=1
> Jun 23 15:56:43.941129 (XEN)     No periodic timer
> 
> So whatever the issue is it doesn't seem to be particularly related to
> the strange NUMA layout.
> 
Exactly.

Inspecting the logs and looking at the dump of scheduler info, pCPUs
info and vCPUs info, everything seems completely and fully idle, at the
time the debugkeys are sent to the box.

There are no vCPUs active or waiting in any runqueue, all the host pCPUs
are in idle_loop() and all Dom0 vCPUs are in ffffffff810013aa, which
should be xen_hypercall_sched_op... So, if there was something keeping
the system busy enough to make QEMU miss the 10 secs timeout, any dead
or live lock, either in Xen or Dom0, it seems to be gone by when we
realize things have gone bad and go inspecting the system (as a further,
although of course not conclusive, proof of that, we do manage to see
the output of `xl info', `xl list', etc., performed during
ts-capture-logs, so the system is indeed able to respond).

Regards,
Dario

-- 
<<This happens because I choose it to happen!>> (Raistlin Majere)
-----------------------------------------------------------------
Dario Faggioli, Ph.D, http://about.me/dario.faggioli
Senior Software Engineer, Citrix Systems R&D Ltd., Cambridge (UK)

[-- Attachment #1.2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 181 bytes --]

[-- Attachment #2: Type: text/plain, Size: 126 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel