From mboxrd@z Thu Jan 1 00:00:00 1970 From: Dario Faggioli Subject: [PATCH 0/2] Credit2: fix per-socket runqueue setup Date: Fri, 22 Aug 2014 19:15:26 +0200 Message-ID: <20140822165628.32764.15082.stgit@Solace.lan> Mime-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Return-path: Received: from mail6.bemta14.messagelabs.com ([193.109.254.103]) by lists.xen.org with esmtp (Exim 4.72) (envelope-from ) id 1XKsR9-0004YA-Oj for xen-devel@lists.xenproject.org; Fri, 22 Aug 2014 17:15:39 +0000 Received: by mail-wi0-f176.google.com with SMTP id bs8so10505148wib.15 for ; Fri, 22 Aug 2014 10:15:29 -0700 (PDT) List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Sender: xen-devel-bounces@lists.xen.org Errors-To: xen-devel-bounces@lists.xen.org To: xen-devel Cc: george.dunlap@eu.citrix.com, Andrew Cooper , keir@xen.org, Jan Beulich List-Id: xen-devel@lists.xenproject.org Hi everyone, Code is already there in the credit2 scheduler, for setting up one runqueue per each socket found in the system, despite the fact that comments in sched_credit2.c still report this (well, something like this )to be a TODO item. However, such code is not functional. In fact, on the following host: cpu_topology : cpu: core socket node 0: 0 1 0 1: 0 1 0 2: 1 1 0 3: 1 1 0 4: 2 1 0 5: 2 1 0 6: 8 1 0 7: 8 1 0 8: 9 1 0 9: 9 1 0 10: 10 1 0 11: 10 1 0 12: 0 0 1 13: 0 0 1 14: 1 0 1 15: 1 0 1 16: 2 0 1 17: 2 0 1 18: 8 0 1 19: 8 0 1 20: 9 0 1 21: 9 0 1 22: 10 0 1 23: 10 0 1 numa_info : node: memsize memfree distances 0: 18432 17488 10,20 1: 19200 17932 20,10 What I see during boot and scheduler initialization is as follows (for all the 24 pCPUs): (XEN) init_pcpu: Strange, cpu 1 already initialized! (XEN) init_pcpu: Strange, cpu 2 already initialized! ... (XEN) init_pcpu: Strange, cpu 23 already initialized! Also, once the system booted, here's what happened: root@tg03:~# xl dmesg |grep -i runqueue (XEN) Adding cpu 0 to runqueue 0 (XEN) First cpu on runqueue, activating (XEN) Adding cpu 1 to runqueue 0 (XEN) Adding cpu 2 to runqueue 0 ... (XEN) Adding cpu 23 to runqueue 0 I.e., only one runqueue was created, despite the host having 2 sockets, and all the pCPUs have been assigned to it. This series fixes that by doing the following three things: - anticipate the identification of the topology of the boot CPU to happen _before_ the scheduler's initialization. This is necessary to make sure that, when credit2 will try to figure out on which socket CPU 0 it, it will find the correct info (patch 1); - properly initialize the cpuinfo_x86 data structure, and more specifically the phys_proc_id field, that hosts the socket, to something invalid (-1). In fact, having that field in all elements set to 0 would induce credit2 to think that the pCPU have already been initialized, and that all are on socket 0 (patch 1); - use the info made available by early boot CPU identification in credit2 (patch 2). After this series is applied, no more "Strange, cpu X already initialized!" can be found in the logs. Also, here's how the scheduler is being setup: root@tg03:~# xl dmesg |grep -i runqueue (XEN) Adding cpu 0 to runqueue 1 (XEN) First cpu on runqueue, activating (XEN) Adding cpu 1 to runqueue 1 (XEN) Adding cpu 2 to runqueue 1 (XEN) Adding cpu 3 to runqueue 1 (XEN) Adding cpu 4 to runqueue 1 (XEN) Adding cpu 5 to runqueue 1 (XEN) Adding cpu 6 to runqueue 1 (XEN) Adding cpu 7 to runqueue 1 (XEN) Adding cpu 8 to runqueue 1 (XEN) Adding cpu 9 to runqueue 1 (XEN) Adding cpu 10 to runqueue 1 (XEN) Adding cpu 11 to runqueue 1 (XEN) Adding cpu 12 to runqueue 0 (XEN) First cpu on runqueue, activating (XEN) Adding cpu 13 to runqueue 0 (XEN) Adding cpu 14 to runqueue 0 (XEN) Adding cpu 15 to runqueue 0 (XEN) Adding cpu 16 to runqueue 0 (XEN) Adding cpu 17 to runqueue 0 (XEN) Adding cpu 18 to runqueue 0 (XEN) Adding cpu 19 to runqueue 0 (XEN) Adding cpu 20 to runqueue 0 (XEN) Adding cpu 21 to runqueue 0 (XEN) Adding cpu 22 to runqueue 0 (XEN) Adding cpu 23 to runqueue 0 Which makes a lot more sense. :-) Regards, Dario --- Dario Faggioli (2): x86: during boot, anticipate identifying the boot cpu sched: credit2: use boot CPU info for CPU #0 xen/arch/x86/setup.c | 8 ++++++-- xen/arch/x86/smpboot.c | 3 ++- xen/common/sched_credit2.c | 12 +++++------- xen/include/asm-x86/processor.h | 6 ++++-- 4 files changed, 17 insertions(+), 12 deletions(-) -- s happens because I choose it to happen!>> (Raistlin Majere) ----------------------------------------------------------------- Dario Faggioli, Ph.D, http://about.me/dario.faggioli Senior Software Engineer, Citrix Systems R&D Ltd., Cambridge (UK)