From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mx0a-001b2d01.pphosted.com (mx0a-001b2d01.pphosted.com [148.163.156.1]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by lists.ozlabs.org (Postfix) with ESMTPS id 421xrm10Q0zF35g for ; Fri, 31 Aug 2018 21:26:31 +1000 (AEST) Received: from pps.filterd (m0098394.ppops.net [127.0.0.1]) by mx0a-001b2d01.pphosted.com (8.16.0.22/8.16.0.22) with SMTP id w7VBOGFp119677 for ; Fri, 31 Aug 2018 07:26:29 -0400 Received: from e06smtp02.uk.ibm.com (e06smtp02.uk.ibm.com [195.75.94.98]) by mx0a-001b2d01.pphosted.com with ESMTP id 2m741v9uvp-1 (version=TLSv1.2 cipher=AES256-GCM-SHA384 bits=256 verify=NOT) for ; Fri, 31 Aug 2018 07:26:29 -0400 Received: from localhost by e06smtp02.uk.ibm.com with IBM ESMTP SMTP Gateway: Authorized Use Only! Violators will be prosecuted for from ; Fri, 31 Aug 2018 12:26:27 +0100 Date: Fri, 31 Aug 2018 16:56:18 +0530 From: Srikar Dronamraju To: Peter Zijlstra Cc: Ingo Molnar , LKML , Mel Gorman , Rik van Riel , Thomas Gleixner , Michael Ellerman , Heiko Carstens , Suravee Suthikulpanit , linuxppc-dev , Andre Wild , Benjamin Herrenschmidt Subject: Re: [PATCH] sched/topology: Use Identity node only if required Reply-To: Srikar Dronamraju References: <1533712172-11428-1-git-send-email-srikar@linux.vnet.ibm.com> <20180808075840.GO2494@hirez.programming.kicks-ass.net> <20180810164533.GB42350@linux.vnet.ibm.com> <20180829084348.GO24124@hirez.programming.kicks-ass.net> <20180831102248.GA8437@linux.vnet.ibm.com> <20180831104115.GH24124@hirez.programming.kicks-ass.net> MIME-Version: 1.0 Content-Type: text/plain; charset=iso-8859-1 In-Reply-To: <20180831104115.GH24124@hirez.programming.kicks-ass.net> Message-Id: <20180831112618.GA10911@linux.vnet.ibm.com> List-Id: Linux on PowerPC Developers Mail List List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , * Peter Zijlstra [2018-08-31 12:41:15]: > On Fri, Aug 31, 2018 at 03:22:48AM -0700, Srikar Dronamraju wrote: > > > At boot: Before topology update. > > How does that work; you do SMP bringup _before_ you know the topology !? > If you look at the other mail that I sent, the system boots to its regular state with a certain topology. The hypervisor might detect and push topology updates after the system has been booted and initialized. This topology update can happen much much later after boot. We boot with a particular topology and a later point of time, the topology update event occurs. > > After topology update. > > > > For CPU 0 > > domain-0: span=0-7 level=SMT > > groups: 0:{ span=0 }, 1:{ span=1 }, 2:{ span=2 }, 3:{ span=3 }, 4:{ span=4 }, 5:{ span=5 }, 6:{ span=6 }, 7:{ span=7 } > > domain-1: span=0-7,32-39,64-71,96-103,128-135,160-167,192-199,224-231,256-263,288-295 level=DIE > > groups: 0:{ span=0-7 cap=8192 }, 32:{ span=32-39 cap=8192 }, 64:{ span=64-71 cap=8192 }, 96:{ span=96-103 cap=8192 }, 128:{ span=128-135 cap=8192 }, 160:{ span=160-167 cap=8192 }, 192:{ span=192-199 cap=8192 }, 224:{ span=224-231 cap=8192 }, 256:{ span=256-263 cap=8192 }, 288:{ span=288-295 cap=8192 } > > domain-2: span=0-303 level=NODE > > groups: 0:{ span=0-7,32-39,64-71,96-103,128-135,160-167,192-199,224-231,256-263,288-295 cap=81920 }, 8:{ span=8-15,40-47,72-79,104-111,136-143,168-175,200-207,232-239,264-271,296-303 cap=81920 }, 16:{ span=16-23,48-55,80-87,112-119,144-151,176-183,208-215,240-247,272-279 cap=73728 }, 24:{ span=24-31,56-63,88-95,120-127,152-159,184-191,216-223,248-255,280-287 cap=73728 } > > > > For CPU 9 > > domain-0: span=8-15 level=SMT > > groups: 9:{ span=9 }, 10:{ span=10 }, 11:{ span=11 }, 12:{ span=12 }, 13:{ span=13 }, 14:{ span=14 }, 15:{ span=15 }, 8:{ span=8 } > > domain-1: span=8-15,40-47,72-79,104-111,136-143,168-175,200-207,232-239,264-271,296-303 level=DIE > > groups: 8:{ span=8-15 cap=8192 }, 40:{ span=40-47 cap=8192 }, 72:{ span=72-79 cap=8192 }, 104:{ span=104-111 cap=8192 }, 136:{ span=136-143 cap=8192 }, 168:{ span=168-175 cap=8192 }, 200:{ span=200-207 cap=8192 }, 232:{ span=232-239 cap=8192 }, 264:{ span=264-271 cap=8192 }, 296:{ span=296-303 cap=8192 } > > domain-2: span=8-15,40-47,72-79,104-111,136-143,168-175,200-207,232-239,264-271,296-303 level=NODE > > groups: 8:{ span=8-15,40-47,72-79,104-111,136-143,168-175,200-207,232-239,264-271,296-303 cap=81920 } > > domain-3: span=0-303 level=NUMA > > groups: 8:{ span=8-15,40-47,72-79,104-111,136-143,168-175,200-207,232-239,264-271,296-303 cap=81920 }, 16:{ span=16-23,48-55,80-87,112-119,144-151,176-183,208-215,240-247,272-279 cap=73728 }, 24:{ span=24-31,56-63,88-95,120-127,152-159,184-191,216-223,248-255,280-287 cap=73728 } > > ERROR: groups don't span domain->span > > This is all very confused... and does not include the error we saw > earlier. > > CPU 0 has: SMT, DIE, NODE > CPU 8 has: SMT, DIE, NODE, NUMA > This was the same in my previous posting too. Before the topology update happened, all the cpus would be in SMT, DIE. The topology updates can be disabled using a kernel parameter topology_updates=off. Its documented under https://www.kernel.org/doc/html/latest/admin-guide/kernel-parameters.html as topology_updates= [KNL, PPC, NUMA] Format: {off} Specify if the kernel should ignore (off) topology updates sent by the hypervisor to this LPAR. and is not something new in powerpc. > Something is completely buggered in your topology setup. >