From mboxrd@z Thu Jan 1 00:00:00 1970 From: gowrishankar muthukrishnan Subject: Re: [PATCH 18.05 v4] eal: add function to return number of detected sockets Date: Thu, 22 Mar 2018 10:46:09 +0530 Message-ID: References: <18deafea-5662-88ef-2ddc-3a1970d67405@linux.vnet.ibm.com> Mime-Version: 1.0 Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 8bit Cc: dev@dpdk.org, Bruce Richardson , Chao Zhu To: "Burakov, Anatoly" Return-path: Received: from mx0a-001b2d01.pphosted.com (mx0b-001b2d01.pphosted.com [148.163.158.5]) by dpdk.org (Postfix) with ESMTP id 10C065F5D for ; Thu, 22 Mar 2018 06:16:16 +0100 (CET) Received: from pps.filterd (m0098413.ppops.net [127.0.0.1]) by mx0b-001b2d01.pphosted.com (8.16.0.22/8.16.0.22) with SMTP id w2M5Dk42044028 for ; Thu, 22 Mar 2018 01:16:16 -0400 Received: from e06smtp12.uk.ibm.com (e06smtp12.uk.ibm.com [195.75.94.108]) by mx0b-001b2d01.pphosted.com with ESMTP id 2gv3t1vjer-1 (version=TLSv1.2 cipher=AES256-SHA256 bits=256 verify=NOT) for ; Thu, 22 Mar 2018 01:16:15 -0400 Received: from localhost by e06smtp12.uk.ibm.com with IBM ESMTP SMTP Gateway: Authorized Use Only! Violators will be prosecuted for from ; Thu, 22 Mar 2018 05:16:14 -0000 In-Reply-To: Content-Language: en-GB List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" On Wednesday 21 March 2018 03:54 PM, Burakov, Anatoly wrote: > >>> +    config->numa_node_count = max_socket_id + 1; >> >> In some IBM servers, socket ID number does not seem to be in >> sequence. For an instance, 0 and 8 for a 2 node server. >> >> In this case, numa_node_count would mislead users if wrongly >> understood by its variable name IMO (see below) >>> +    RTE_LOG(INFO, EAL, "Detected %u NUMA nodes\n", >>> config->numa_node_count); >> >> For an instance, reading above message would tell 'EAL detected 8 >> nodes' in my server, but actually there are only two nodes. >> >> Could its name better be 'numa_node_id_max' ?. Also, we store in >> actual count of numa nodes in _count variable. >> >> Also, there could be a case when there is no local memory available >> to a numa node too. >> >> Thanks, >> Gowrishankar > > The point of this patchset is to (pre)allocate memory only on existing > sockets. > > If we don't know how many sockets there are, we are forced to > preallocate VA space per each *possible* NUMA node - that is, reserve > e.g. 8x128G of memory, 6 of which will go unused on a 2-socket system. > We can't know if there is no memory on socket in advance, but we can > at least avoid preallocating VA space for sockets that don't exist in > the first place. > Sounds good Anatoly. May be, sysfs/ might help to confirm if a numa node has local memory ?. Anyway, for the context of this particular patch (return numa nodes), below approach you mentioned is good. > How about we store all possible socket id's instead? e.g. something like: > > static int numa_node_ids[MAX_NUMA_NODES]; > <...> > int rte_eal_cpu_init() { >     int sockets[RTE_MAX_LCORE]; >     <...> >     for (lcore_id = 0; lcore_id < RTE_MAX_LCORE; lcore_id++) { >         core_to_socket[lcore_id] = socket; sockets[lcore_id] = eal_cpu_socket_id(lcore_id); >     } >     <...> >     qsort(sockets); >     <...> >     // store all unique sockets in numa_node_ids in ascending order Just thinking that, is there a purpose of retaining a numa ID which does not have local memory attached ? but sockets[] is suppose to reflect all available nodes though (and assuming, its calling place to ensure for the existence of numa local memory). > } > <...> > > on a 2 socket system we then get: > > rte_num_sockets() => return 2 > rte_get_socket_id(int idx) => return numa_node_ids[idx] rte_get_socket_mem(idx) might help to validate for local memory existence ? > > Would that be suitable? > Thanks, Gowrishankar