From mboxrd@z Thu Jan 1 00:00:00 1970 From: Dario Faggioli Subject: Re: Problems with merlot* AMD Opteron 6376 systems (Was Re: stable trees (was: [xen-4.2-testing test] 58584: regressions)) Date: Wed, 24 Jun 2015 14:29:43 +0200 Message-ID: <1435148983.25170.107.camel@citrix.com> References: <558151930200007800085E5C@mail.emea.novell.com> <21889.19282.1828.571256@mariner.uk.xensource.com> <5582C989020000780008696A@mail.emea.novell.com> <1434637353.28264.37.camel@citrix.com> <5584024B0200007800086F10@mail.emea.novell.com> <1434712049.28264.100.camel@citrix.com> <1435136770.28264.252.camel@citrix.com> <1435138723.28264.279.camel@citrix.com> Mime-Version: 1.0 Content-Type: multipart/mixed; boundary="===============0029233475466623987==" Return-path: Received: from mail6.bemta14.messagelabs.com ([193.109.254.103]) by lists.xen.org with esmtp (Exim 4.72) (envelope-from ) id 1Z7joQ-0005aE-38 for xen-devel@lists.xenproject.org; Wed, 24 Jun 2015 12:29:54 +0000 In-Reply-To: <1435138723.28264.279.camel@citrix.com> List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Sender: xen-devel-bounces@lists.xen.org Errors-To: xen-devel-bounces@lists.xen.org To: Ian Campbell Cc: Lars Kurth , Jan Beulich , Stefano Stabellini , Ian Jackson , Aravind Gopalakrishnan , Suravee Suthikulpanit , Anthony Perard , xen-devel , Boris Ostrovsky List-Id: xen-devel@lists.xenproject.org --===============0029233475466623987== Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="=-+c5j7KwAfqYQ+d1tuv6m" --=-+c5j7KwAfqYQ+d1tuv6m Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable On Wed, 2015-06-24 at 10:38 +0100, Ian Campbell wrote: > Adding Boris+Suravee+Aravind (AMD/SVM maintainers), Dario (NUMA) and Jim > +Anthony (libvirt) to the CC. > Supposing that the NUMA oddities might be what is exposing this issue I > tried an adhoc run on the merlot machines where I specified > "dom0_max_vcpus=3D8 dom0_nodes=3D0" on the hypervisor command line: > http://logs.test-lab.xenproject.org/osstest/logs/58853/ >=20 > Again, I messed up the config for the -xsm case, so ignore. >=20 > The interesting thing is that the extra NUMA settings were > apparently_not_ helpful. From > http://logs.test-lab.xenproject.org/osstest/logs/58853/test-amd64-amd64-l= ibvirt/serial-merlot0.log I can see they were applied: > Jun 23 15:50:34.205057 (XEN) Command line: placeholder conswitch=3Dx watc= hdog com1=3D115200,8n1 console=3Dcom1,vga gdb=3Dcom1 dom0_mem=3D512M,max:51= 2M ucode=3Dscan dom0_max_vcpus=3D8 dom0_nodes=3D0 > [...] > Jun 23 15:50:38.309057 (XEN) Dom0 has maximum 8 VCPUs >=20 IIRC, you can drop the dom0_max_vcpus=3D8, as Xen would figure it out automatically, as a consequence of dom0_nodes=3D0. In any case, it doesn't hurt. Maybe we can try running this again with dom0_nodes=3D2 (the other node with memory attached). I wouldn't know what to expect, though, so, yes, it's a shot in the dark, but since we're out of plausible theories... :-/ > The memory info > Jun 23 15:56:27.749008 (XEN) Memory location of each domain: > Jun 23 15:56:27.756965 (XEN) Domain 0 (total: 131072): > Jun 23 15:56:27.756983 (XEN) Node 0: 126905 > Jun 23 15:56:27.756998 (XEN) Node 1: 0 > Jun 23 15:56:27.764952 (XEN) Node 2: 4167 > Jun 23 15:56:27.764969 (XEN) Node 3: 0 > suggests at least a small amount of cross-node memory allocation (16M > out of dom0s 512M total). That's probably small enough to be OK. >=20 Yeah, that is in line with what you usually get with dom0_nodes. Most of the memory, as you noted, comes from the proper node. We're just not (yet?) at the point where _all_ of it can come from there. > And it seems as if the 8 dom0 vcpus are correctly pinned to the first 8 > cpus (the ones in node 0): > Jun 23 15:56:43.797055 (XEN) VCPU information and callbacks for domain 0: > Jun 23 15:56:43.797110 (XEN) VCPU0: CPU4 [has=3DF] poll=3D0 upcall_pe= nd=3D00 upcall_mask=3D00 dirty_cpus=3D{4} > Jun 23 15:56:43.805078 (XEN) cpu_hard_affinity=3D{0-7} cpu_soft_affin= ity=3D{0-7} > Jun 23 15:56:43.813121 (XEN) pause_count=3D0 pause_flags=3D1 > Jun 23 15:56:43.813157 (XEN) No periodic timer > Jun 23 15:56:43.821050 (XEN) VCPU1: CPU3 [has=3DF] poll=3D0 upcall_pe= nd=3D00 upcall_mask=3D00 dirty_cpus=3D{3} > Jun 23 15:56:43.829044 (XEN) cpu_hard_affinity=3D{0-7} cpu_soft_affin= ity=3D{0-7} > Jun 23 15:56:43.829082 (XEN) pause_count=3D0 pause_flags=3D1 > Jun 23 15:56:43.837051 (XEN) No periodic timer > Jun 23 15:56:43.837084 (XEN) VCPU2: CPU5 [has=3DF] poll=3D0 upcall_pe= nd=3D00 upcall_mask=3D00 dirty_cpus=3D{5} > Jun 23 15:56:43.845102 (XEN) cpu_hard_affinity=3D{0-7} cpu_soft_affin= ity=3D{0-7} > Jun 23 15:56:43.853035 (XEN) pause_count=3D0 pause_flags=3D1 > Jun 23 15:56:43.853071 (XEN) No periodic timer > Jun 23 15:56:43.853099 (XEN) VCPU3: CPU7 [has=3DF] poll=3D0 upcall_pe= nd=3D00 upcall_mask=3D00 dirty_cpus=3D{7} > Jun 23 15:56:43.861102 (XEN) cpu_hard_affinity=3D{0-7} cpu_soft_affin= ity=3D{0-7} > Jun 23 15:56:43.869110 (XEN) pause_count=3D0 pause_flags=3D1 > Jun 23 15:56:43.869145 (XEN) No periodic timer > Jun 23 15:56:43.877014 (XEN) VCPU4: CPU0 [has=3DF] poll=3D0 upcall_pe= nd=3D00 upcall_mask=3D00 dirty_cpus=3D{} > Jun 23 15:56:43.877038 (XEN) cpu_hard_affinity=3D{0-7} cpu_soft_affin= ity=3D{0-7} > Jun 23 15:56:43.885053 (XEN) pause_count=3D0 pause_flags=3D1 > Jun 23 15:56:43.885088 (XEN) No periodic timer > Jun 23 15:56:43.893085 (XEN) VCPU5: CPU0 [has=3DF] poll=3D0 upcall_pe= nd=3D00 upcall_mask=3D00 dirty_cpus=3D{} > Jun 23 15:56:43.901075 (XEN) cpu_hard_affinity=3D{0-7} cpu_soft_affin= ity=3D{0-7} > Jun 23 15:56:43.901134 (XEN) pause_count=3D0 pause_flags=3D1 > Jun 23 15:56:43.909010 (XEN) No periodic timer > Jun 23 15:56:43.909048 (XEN) VCPU6: CPU2 [has=3DF] poll=3D0 upcall_pe= nd=3D00 upcall_mask=3D00 dirty_cpus=3D{2} > Jun 23 15:56:43.917065 (XEN) cpu_hard_affinity=3D{0-7} cpu_soft_affin= ity=3D{0-7} > Jun 23 15:56:43.925055 (XEN) pause_count=3D0 pause_flags=3D1 > Jun 23 15:56:43.925074 (XEN) No periodic timer > Jun 23 15:56:43.925095 (XEN) VCPU7: CPU6 [has=3DF] poll=3D0 upcall_pe= nd=3D00 upcall_mask=3D00 dirty_cpus=3D{6} > Jun 23 15:56:43.933119 (XEN) cpu_hard_affinity=3D{0-7} cpu_soft_affin= ity=3D{0-7} > Jun 23 15:56:43.941080 (XEN) pause_count=3D0 pause_flags=3D1 > Jun 23 15:56:43.941129 (XEN) No periodic timer >=20 > So whatever the issue is it doesn't seem to be particularly related to > the strange NUMA layout. >=20 Exactly. Inspecting the logs and looking at the dump of scheduler info, pCPUs info and vCPUs info, everything seems completely and fully idle, at the time the debugkeys are sent to the box. There are no vCPUs active or waiting in any runqueue, all the host pCPUs are in idle_loop() and all Dom0 vCPUs are in ffffffff810013aa, which should be xen_hypercall_sched_op... So, if there was something keeping the system busy enough to make QEMU miss the 10 secs timeout, any dead or live lock, either in Xen or Dom0, it seems to be gone by when we realize things have gone bad and go inspecting the system (as a further, although of course not conclusive, proof of that, we do manage to see the output of `xl info', `xl list', etc., performed during ts-capture-logs, so the system is indeed able to respond). Regards, Dario --=20 <> (Raistlin Majere) ----------------------------------------------------------------- Dario Faggioli, Ph.D, http://about.me/dario.faggioli Senior Software Engineer, Citrix Systems R&D Ltd., Cambridge (UK) --=-+c5j7KwAfqYQ+d1tuv6m Content-Type: application/pgp-signature; name="signature.asc" Content-Description: This is a digitally signed message part Content-Transfer-Encoding: 7bit -----BEGIN PGP SIGNATURE----- Version: GnuPG v2 iEYEABECAAYFAlWKorcACgkQk4XaBE3IOsQmXwCfVd8fqvd54FmCz0jplr+Ql06T 9mgAoIn0ocnRnIVSjKAsP3yDv6/PLhM1 =rjN4 -----END PGP SIGNATURE----- --=-+c5j7KwAfqYQ+d1tuv6m-- --===============0029233475466623987== Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Disposition: inline _______________________________________________ Xen-devel mailing list Xen-devel@lists.xen.org http://lists.xen.org/xen-devel --===============0029233475466623987==--