Re: [PATCH v6 00/10] vnuma introduction

[Dario Faggioli <dario.faggioli@citrix.com>]

[-- Attachment #1.1: Type: text/plain, Size: 6828 bytes --]

On ven, 2014-07-18 at 12:48 +0100, Wei Liu wrote:
> On Fri, Jul 18, 2014 at 12:13:36PM +0200, Dario Faggioli wrote:
> > On ven, 2014-07-18 at 10:53 +0100, Wei Liu wrote:

> > > I've also encountered this. I suspect that even if you disble SMT with
> > > cpuid in config file, the cpu topology in guest might still be wrong.
> > >
> > Can I ask why?
> > 
> 
> Because for a PV guest (currently) the guest kernel sees the real "ID"s
> for a cpu. See those "ID"s I change in my hacky patch.
> 
Right, now I see/remember it. Well, this is, I think, something we
should try to fix _independently_ from vNUMA, isn't it?

I mean, even right now, PV guests see completely random cache-sharing
topology, and that does (at least potentially) affect performance, as
the guest scheduler will make incorrect/inconsistent assumptions.

I'm not sure what the correct fix is. Probably something similar to what
you're doing in your hack... but, indeed, I think we should do something
about this!

> > > What do hwloc-ls and lscpu show? Do you see any weird topology like one
> > > core belongs to one node while three belong to another?
> > >
> > Yep, that would be interesting to see.
> > 
> > >  (I suspect not
> > > because your vcpus are already pinned to a specific node)
> > > 
> > Sorry, I'm not sure I follow here... Are you saying that things probably
> > works ok, but that is (only) because of pinning?
> 
> Yes, given that you derive numa memory allocation from cpu pinning or
> use combination of cpu pinning, vcpu to vnode map and vnode to pnode
> map, in those cases those IDs might reflect the right topology.
> 
Well, pinning does (should?) not always happen, as a consequence of a
virtual topology being used.

So, again, I don't think we should rely on pinning to have a sane and,
more important, consistent SMT and cache sharing topology.

Linux maintainers, any ideas?


BTW, I tried a few examples, on the following host:

root@benny:~# xl info -n
...
nr_cpus                : 8
max_cpu_id             : 15
nr_nodes               : 1
cores_per_socket       : 4
threads_per_core       : 2
cpu_mhz                : 3591
...
cpu_topology           :
cpu:    core    socket     node
  0:       0        0        0
  1:       0        0        0
  2:       1        0        0
  3:       1        0        0
  4:       2        0        0
  5:       2        0        0
  6:       3        0        0
  7:       3        0        0
numa_info              :
node:    memsize    memfree    distances
   0:     34062      31029      10

With the following guest configuration, in terms of vcpu pinning:

1) 2 vCPUs ==> same pCPUs
root@benny:~# xl vcpu-list 
Name                                ID  VCPU   CPU State   Time(s) CPU Affinity
debian.guest.osstest                 9     0    0   -b-       2.7  0
debian.guest.osstest                 9     1    0   -b-       5.2  0
debian.guest.osstest                 9     2    7   -b-       2.4  7
debian.guest.osstest                 9     3    7   -b-       4.4  7

2) no SMT
root@benny:~# xl vcpu-list 
Name                                ID  VCPU   CPU State   Time(s) CPU
Affinity
debian.guest.osstest                11     0    0   -b-       0.6  0
debian.guest.osstest                11     1    2   -b-       0.4  2
debian.guest.osstest                11     2    4   -b-       1.5  4
debian.guest.osstest                11     3    6   -b-       0.5  6

3) Random
root@benny:~# xl vcpu-list 
Name                                ID  VCPU   CPU State   Time(s) CPU
Affinity
debian.guest.osstest                12     0    3   -b-       1.6  all
debian.guest.osstest                12     1    1   -b-       1.4  all
debian.guest.osstest                12     2    5   -b-       2.4  all
debian.guest.osstest                12     3    7   -b-       1.5  all

4) yes SMT
root@benny:~# xl vcpu-list
Name                                ID  VCPU   CPU State   Time(s) CPU
Affinity
debian.guest.osstest                14     0    1   -b-       1.0  1
debian.guest.osstest                14     1    2   -b-       1.8  2
debian.guest.osstest                14     2    6   -b-       1.1  6
debian.guest.osstest                14     3    7   -b-       0.8  7

And, in *all* these 4 cases, here's what I see:

root@debian:~# cat /sys/devices/system/cpu/cpu*/topology/core_siblings_list
0-3
0-3
0-3
0-3

root@debian:~# cat /sys/devices/system/cpu/cpu*/topology/thread_siblings_list
0-3
0-3
0-3
0-3

root@debian:~# lstopo
Machine (488MB) + Socket L#0 + L3 L#0 (8192KB) + L2 L#0 (256KB) + L1 L#0 (32KB) + Core L#0
  PU L#0 (P#0)
  PU L#1 (P#1)
  PU L#2 (P#2)
  PU L#3 (P#3)

root@debian:~# lscpu
Architecture:          x86_64
CPU op-mode(s):        32-bit, 64-bit
Byte Order:            Little Endian
CPU(s):                4
On-line CPU(s) list:   0-3
Thread(s) per core:    4
Core(s) per socket:    1
Socket(s):             1
Vendor ID:             GenuineIntel
CPU family:            6
Model:                 60
Stepping:              3
CPU MHz:               3591.780
BogoMIPS:              7183.56
Hypervisor vendor:     Xen
Virtualization type:   full
L1d cache:             32K
L1i cache:             32K
L2 cache:              256K
L3 cache:              8192K

I.e., no matter how I pin the vcpus, the guest sees the 4 vcpus as if
they were all SMT siblings, within the same core, sharing all cache
levels.

This is not the case for dom0 where (I booted with dom0_max_vcpus=4 on
the xen command line) I see this:

root@benny:~# lstopo
Machine (422MB)
  Socket L#0 + L3 L#0 (8192KB)
    L2 L#0 (256KB) + L1 L#0 (32KB) + Core L#0
      PU L#0 (P#0)
      PU L#1 (P#1)
    L2 L#1 (256KB) + L1 L#1 (32KB) + Core L#1
      PU L#2 (P#2)
      PU L#3 (P#3)

root@benny:~# lscpu
Architecture:          x86_64
CPU op-mode(s):        32-bit, 64-bit
Byte Order:            Little Endian
CPU(s):                4
On-line CPU(s) list:   0-3
Thread(s) per core:    2
Core(s) per socket:    2
Socket(s):             1
Vendor ID:             GenuineIntel
CPU family:            6
Model:                 60
Stepping:              3
CPU MHz:               3591.780
BogoMIPS:              7183.56
Hypervisor vendor:     Xen
Virtualization type:   none
L1d cache:             32K
L1i cache:             32K
L2 cache:              256K
L3 cache:              8192K

What am I doing wrong, or what am I missing?

Thanks and Regards,
Dario

-- 
<<This happens because I choose it to happen!>> (Raistlin Majere)
-----------------------------------------------------------------
Dario Faggioli, Ph.D, http://about.me/dario.faggioli
Senior Software Engineer, Citrix Systems R&D Ltd., Cambridge (UK)


[-- Attachment #1.2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 181 bytes --]

[-- Attachment #2: Type: text/plain, Size: 126 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel