From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S934475AbdKAVc3 (ORCPT ); Wed, 1 Nov 2017 17:32:29 -0400 Received: from mx1.redhat.com ([209.132.183.28]:49756 "EHLO mx1.redhat.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S933383AbdKAVc0 (ORCPT ); Wed, 1 Nov 2017 17:32:26 -0400 DMARC-Filter: OpenDMARC Filter v1.3.2 mx1.redhat.com B0EEB356CB Authentication-Results: ext-mx06.extmail.prod.ext.phx2.redhat.com; dmarc=none (p=none dis=none) header.from=redhat.com Authentication-Results: ext-mx06.extmail.prod.ext.phx2.redhat.com; spf=fail smtp.mailfrom=msnitzer@redhat.com Date: Wed, 1 Nov 2017 17:32:22 -0400 From: Mike Snitzer To: "Paul E. McKenney" Cc: dm-devel@redhat.com, Mikulas Patocka , linux-kernel@vger.kernel.org, "Alasdair G. Kergon" , Zdenek Kabelac Subject: Re: SRCU's apparent use of NR_CPUS? [was: re: dm: allocate struct mapped_device with kvzalloc] Message-ID: <20171101213222.GA27306@redhat.com> References: <20171101154844.GA25792@redhat.com> <20171101162306.GU3659@linux.vnet.ibm.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20171101162306.GU3659@linux.vnet.ibm.com> User-Agent: Mutt/1.5.21 (2010-09-15) X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.5.110.30]); Wed, 01 Nov 2017 21:32:26 +0000 (UTC) Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On Wed, Nov 01 2017 at 12:23pm -0400, Paul E. McKenney wrote: > On Wed, Nov 01, 2017 at 11:48:44AM -0400, Mike Snitzer wrote: > > [cc'ing Paul, and LKML, to get his/others' take on SRCU cpu scaling] > > > > On Tue, Oct 31 2017 at 7:33pm -0400, > > Mikulas Patocka wrote: > > > > > The structure srcu_struct can be very big, its size is proportional to the > > > value CONFIG_NR_CPUS. The Fedora kernel has CONFIG_NR_CPUS 8192, the field > > > io_barrier in the struct mapped_device has 84kB in the debugging kernel > > > and 50kB in the non-debugging kernel. The large size may result in failure > > > of the function kzalloc_node. > > > > > > In order to avoid the allocation failure, we use the function > > > kvzalloc_node, this function falls back to vmalloc if a large contiguous > > > chunk of memory is not available. This patch also moves the field > > > io_barrier to the last position of struct mapped_device - the reason is > > > that on many processor architectures, short memory offsets result in > > > smaller code than long memory offsets - on x86-64 it reduces code size by > > > 320 bytes. > > > > > > Note to stable kernel maintainers - the kernels 4.11 and older don't have > > > the function kvzalloc_node, you can use the function vzalloc_node instead. > > > > > > Signed-off-by: Mikulas Patocka > > > Cc: stable@vger.kernel.org > > > > This looks reasonable as a near-term workaround.. BUT: > > Paul has there been any discussion about how to make SRCU support > > dynamically scaling up to NR_CPUS maximum as 'nr_cpus' changes (rather > > than accounting for worst case of NR_CPUS up-front)? > > This is the first I have heard of this being a problem. > > For static instances of srcu_struct, life is hard. > > But it should not be all that difficult for SRCU to provide an allocator > for the dynamic cases, which given your kzalloc_node() above is the case > you are worried about, at least assuming that these allocations happen > after rcu_init() is invoked (which is pretty early). > > My approach would be to move the srcu_struct ->node[] array to its > own structure, with a pointer from srcu_struct, allowing short-sized > allocations to be used. (But I do need to check to make sure that there > are no gotchas, and with RCU there usually are a few.) Obviously some > -serious- testing would be required -- do you have a range of systems > to test on? If you'd like to give it a try I'd be happy to work on getting you test coverage. I do have access to a pretty wide range of systems. What type of testing would you like to see? (From where I sit as DM maintainer my testing would be DM-specific, just loading a DM device would make use of the SRCU code in question, so please let me know if there is anything more general you'd like done) > However, you would still have your potential failure case for systems > that really did have large numbers of CPUs, some of which really do > exist in the wild. > > > (But I had a quick look at scrutree.h and I'm not seeing explicit use of > > NR_CPUS, so it is likely occuring via implicit percpu through some > > member of 'struct srcu_struct', e.g. 'sda'?) > > The srcu_struct structure sees NR_CPUS via include/linux/rcu_node_tree.h, > which sizes the srcu_node array at build time. > > The sda pointer references a per-CPU allocation, which I believe already > is sized to the actual system rather than to NR_CPUS. OK, thanks for clarifying. Mike